Downtime Notice: Wednesday, 6:15 PM Eastern (-4 GMT)

[Arkat]

So awesome.

+1

[Singular]

Thank goodness it's something I can act righteous about.

DELETE YOUR #@$% PEOPLE!!
IT'S A MAILBOX NOT A WAREHOUSE!!

I was worried it was something I'd have to defensively dismiss due to personal culpability.

When you're TRing, you don't bother with the mailbox. You just check your favor totals, 'cause you've seen all those over and over. It's a chore to delete each mail one by one. If they want players to do so, they should add a mass delete button. Or an option to check 'all' before delete.

But, yeah, I feel responsible now. My bad everyone! Sorry about the lag.

[Jinxx]

What Caused This

We have an automated system that ensures the integrity of mail that we internally call a "heartbeat". Traditionally this has run every 6 hours. We found at one point that this had caused some crashes because the check got too large. We changed this heartbeat to happen every 5 minutes instead of every 6 hours so that it wouldn’t have such a backlog of data to deal with at once. On all internal servers, and public preview, this ran perfectly smoothly.

Once we hit live, however, it has become clear that the frequency of this heartbeat does not impact the effect it has on the server nearly as much as the sheer amount of mail data present. So, instead of a very noticeable hiccup every 6 hours, we were now having a slightly lessened hiccup every 5 minutes. The reason we did not see this issue on any test servers (including player preview on Lamannia) is because, even with the thousands of test case mail items created, there was not the massive backlog of mail data present that there is on a live server.

In the short term, we are reverting the heartbeat back to once every 6 hours with a patch tomorrow. In the long term we are investigating ways to clean this up and make it not an issue. This may require a combination of more aggressive (and publicly messaged ahead of time) culling of old or excessive mail as well as code changes to the heartbeart itself. Tonight, though, we don't have a timeline on those sorts of changes.

I am not sure if you (SSG) are aware but there has been a problem with the mail system for a looong time in which things keep reappearing in my mailbox after deleting them. These are not auction items but messages about favor or DDO points. It's even more prevalent on HC. Please tell me I am not the only one with this problem? I would be quite happy to never get these mailed messages if it would mean improving game performance. To me, they are like spam or junk mail...anyway to "unsubscribe"?

[Halciet]

I am not sure if you (SSG) are aware but there has been a problem with the mail system for a looong time in which things keep reappearing in my mailbox after deleting them. These are not auction items but messages about favor or DDO points. It's even more prevalent on HC. Please tell me I am not the only one with this problem? I would be quite happy to never get these mailed messages if it would mean improving game performance. To me, they are like spam or junk mail...anyway to "unsubscribe"?

A new way of teaching players about patrons could be useful as once you know that the system exists and how it works all the mail for it is spam.

[Deathlylife]

I am not sure if you (SSG) are aware but there has been a problem with the mail system for a looong time in which things keep reappearing in my mailbox after deleting them. These are not auction items but messages about favor or DDO points. It's even more prevalent on HC. Please tell me I am not the only one with this problem? I would be quite happy to never get these mailed messages if it would mean improving game performance. To me, they are like spam or junk mail...anyway to "unsubscribe"?

Not sure if it is the only reason for returning mail but I have found that deleting favor reward mail but not claiming the favor tier tends to resend the favor message at the next favor tier received. Such as getting 400 favor and deleting the message but not talking to Nyx to claim it. If I wait until at 1750 favor, I get the 400 again with the new 1750 mail. I think they also resend after a certain amount of time (3 weeks?) as a reminder to turn in the favor tier and will resend for every tier not yet claimed for every patron.

It kind of involves player upkeep to not only delete the mail but to also turn in favor rewards to avoid more mail. I think the system was nice when there was less content and still is nice for newer players as reminders that it exists in game and that they earned something but for the veteran player it is a real chore.

[phillymiket]

I'm guessing a real, permanent solution to the player base's storage issues would be better idea than just telling people to delete their sh*t.

I think game inventory is like gas diffusion and expands to fill all available space regardless of volume.

But you're right. And I was just comically overstating rather than seriously cap-lock ranting.

However, if you read between the lines of Cocomajobo's post, I'd expect a mail purge in the very near future.

[lion_mouse]

Any idea how long the servers will be down?

My vote is for 6 days if they fix, 1 day if they do a roll back.

[Atreyu]

What's Happening Now
What Caused This

We have an automated system that ensures the integrity of mail that we internally call a "heartbeat". Traditionally this has run every 6 hours. We found at one point that this had caused some crashes because the check got too large. We changed this heartbeat to happen every 5 minutes instead of every 6 hours so that it wouldn’t have such a backlog of data to deal with at once. On all internal servers, and public preview, this ran perfectly smoothly.

Once we hit live, however, it has become clear that the frequency of this heartbeat does not impact the effect it has on the server nearly as much as the sheer amount of mail data present. So, instead of a very noticeable hiccup every 6 hours, we were now having a slightly lessened hiccup every 5 minutes. The reason we did not see this issue on any test servers (including player preview on Lamannia) is because, even with the thousands of test case mail items created, there was not the massive backlog of mail data present that there is on a live server.

In the short term, we are reverting the heartbeat back to once every 6 hours with a patch tomorrow. In the long term we are investigating ways to clean this up and make it not an issue. This may require a combination of more aggressive (and publicly messaged ahead of time) culling of old or excessive mail as well as code changes to the heartbeart itself. Tonight, though, we don't have a timeline on those sorts of changes.

Would it be possible to come up with some sort of mail preference system? I'd bet most of the unread in game mail out there are favor notifications. If players that don't care about them could turn them off that'd reduce the amount of mail sitting in mailboxes quite a bit I'd imagine.

Perhaps a few levels?
Has Attachment
DDO Point Notification
Favor Item Reward Notification (Things notifications that you qualify for an extra inventory/bank slots or bags)
Favor Reward Notification (Skill boosts or enhancement tree unlocks)

You could select to receive or not, if not, no mails.

[LrdSlvrhnd]

That would be a full day. That's 33 cents of game time. In your case someone definitely needs the mail you 33 pennies so you can look at them. Than at some point (God willing) you can look at those pennies and think about what you asked for.

I'd love to have those 33 pennies so I could flick them at you one by one until you realized I wasn't actually asking for compensation, I was defining what it would look like if it did come, and you *still* managed to get it wrong.

[Nivalis]

Wait, in-game mail?

We wholeheartedly condone basically getting rid of it. Especially 1k messages in idle overflowing inboxes about points, or favor, or whatever.

I don't understand this is a problem in need of fixing, and you don't just slash by 20x the amount of mail there is, that no one wants or needs.

If it is some other mail, this has gone over my head.

The problem is that players have been using the mail system for other purpose, such as temporary storage solutions for items (scrolls, potions, etc) that aren't bound to account. So, that lag is kind of our fault. ^^; Sorry and thank you for trying to fix it

Scrollcases and Spellcomponent pouches might help solve the problem, as well as more frequent sales of inventory space upgrades?

[phillymiket]

When you're TRing, you don't bother with the mailbox. You just check your favor totals, 'cause you've seen all those over and over. It's a chore to delete each mail one by one. If they want players to do so, they should add a mass delete button. Or an option to check 'all' before delete.

But, yeah, I feel responsible now. My bad everyone! Sorry about the lag.

I doubt the favor notifications are the big issue.
And it is a PITA for sure. I leave them for a while too.
I suspect it's more the practice of mailing yourself items to store them for eternity.

iirc, it was stated by a Dev a while ago that there are people with literally like a thousand+ items in their mail box.
Like how do you find anything in that mess?

But maybe I'm wrong. I'm not a coder. That's just what I'm gathering from Coco's post.

It is funny that they bumped the "heartbeat" to 32x the previous rate and are surprised the patient is experiencing discomfort.
(I know it's just an analogy but it's a funny analogy) (and also tragic)

[TonkaBabe]

you should try and get some pictures of the foxes and post them in a new thread in a more misc based section! i'd love to see them xD

I will try, the male is kind of a brown / red mix, the female is a solid red with white tipped tail and feet and the kits are a mottled mix of dark brown, red and white. I don't see the entire family everyday but will try to get a pic to post for you if they get out tomorrow, and if my dog does not get out and run them off first. I swear he could screw up a well planned wedding or even an FBI take down, but he is a lovable bonehead and is always on watch.

Oh almost forgot, get DDO up and running again. Got to stay on message

[TonkaBabe]

Change foxes to Kangaroos and squirrels to possums and wallabies and its much the same here, a walk to calm down then cancel my vip coolly calmly and at peace with the world. Love the post by the way

You made me think ... Kangaroos hunting possums and wallabies. What am i missing about Kangaroos? Are they vicious Apex predators in the down under? are they known as the desert sharks? And lastly to stay on message, how many times has Cordovan heard people wanting to cancel VIP over the years over service outages?

[Singular]

I doubt the favor notifications are the big issue.
And it is a PITA for sure. I leave them for a while too.
I suspect it's more the practice of mailing yourself items to store them for eternity.

iirc, it was stated by a Dev a while ago that there are people with literally like a thousand+ items in their mail box.
Like how do you find anything in that mess?

But maybe I'm wrong. I'm not a coder. That's just what I'm gathering from Coco's post.

It is funny that they bumped the "heartbeat" to 32x the previous rate and are surprised the patient is experiencing discomfort.
(I know it's just an analogy but it's a funny analogy) (and also tragic)

A guildy wanted to send me something and couldn't - my mailbox was full. So, I went and deleted one mail. Then tried to buy something on the AH - couldn't, mailbox is full. I thought, huh, maybe I didn't delete that mail. Did it again, same issue, mailbox is full. I had to delete and delete to free up room. All those messages were from the server - favor stuff, patorons, etc. I suspect they overfill when you don't delete them. There's no other way to get your mailbox higher, since you won't be able to send yourself anything or use the AH.

[Zafaron]

What's Happening Now

I apologize for the poor rollout of Update 49 and the extended downtime. We believe we've located the source of the instability the live servers were seeing earlier this afternoon that prompted us to bring them offline. A larger explanation is below but just in terms of where we're at and when you can expect the game worlds to be open again:

We've got fixes in and are compiling a new game build overnight. If everything goes smoothly we'll be looking to get this deployed tomorrow morning and open the worlds before noon (eastern). We're discussing some added boosts to run alongside the planned weekend boost and will have more news on that once the game worlds are open.


What Caused This

We have an automated system that ensures the integrity of mail that we internally call a "heartbeat". Traditionally this has run every 6 hours. We found at one point that this had caused some crashes because the check got too large. We changed this heartbeat to happen every 5 minutes instead of every 6 hours so that it wouldn’t have such a backlog of data to deal with at once. On all internal servers, and public preview, this ran perfectly smoothly.

Once we hit live, however, it has become clear that the frequency of this heartbeat does not impact the effect it has on the server nearly as much as the sheer amount of mail data present. So, instead of a very noticeable hiccup every 6 hours, we were now having a slightly lessened hiccup every 5 minutes. The reason we did not see this issue on any test servers (including player preview on Lamannia) is because, even with the thousands of test case mail items created, there was not the massive backlog of mail data present that there is on a live server.

In the short term, we are reverting the heartbeat back to once every 6 hours with a patch tomorrow. In the long term we are investigating ways to clean this up and make it not an issue. This may require a combination of more aggressive (and publicly messaged ahead of time) culling of old or excessive mail as well as code changes to the heartbeart itself. Tonight, though, we don't have a timeline on those sorts of changes.

As I'm sure your engineers will point out, it is possible to iterate over partial blocks of your mail in shorter intervals. You could constantly iterate over 0.01 percent each second or minute or whatever block of time that you want and then move on to the next block until you process the whole in whatever time frame you desire before starting again. This would negate the huge spikes in CPU/resource usage and still perform the task in the same amount of time. You could even schedule the processing to only happen when your server load is below a certain threshold.

[Kmnh]

So the servers were obsessively checking all the mail every five minutes, couldn't do anything properly because of it, went into a full blown meltdown and then shutdown until the next day?

That's a perfect 2021 analogy. I'm sure a lot of the playerbase can relate.

[Stormraiser]

Ppl need to remember the terms of service. SSG has absolutely no responsibility to the player. Its up to them. simple.

Im just happy i can still play the game i love with the freinds i love and SSG gives me that platform. Thankyou

Im also stoked that there still developing the game. They can shut the servers down anytime they like.

Thankyou SSG. you may stuff it up now and then. but your still having a go.

Fin.

I must say your post has completely missed the mark and I'm not sure you understand. Let me explain this to you in the simplest terms I can. SSG, like just about every other for-profit business has a relationship with the players that goes like this:

Players provide consideration (money via subs, ddo points etc,) to SSG and in return SSG provides a service (the game).

SSG had an update that broke said game, and this service is not currently available. While mistakes do happen in businesses, there are always reactions to these mistakes. Many of the players are saying two things, first, we are unhappy with how often you break things when you update, and when we compare your service to other similar businesses, they tend to not have as many problems. Secondly, many of the players are saying that they are unhappy with the changes unveiled in this update, despite the vocal objections of many of these players.

This might be easier for you to understand in pizza terms.

Company A changes their pizza sauce recipe and makes it a lot sweeter, despite getting feedback from many surveys and demos saying the sweeter sauce is not as good as before (this is case two in case you're lost)
Then company A keeps having a problem with their oven, causing them to periodically not have pizza and sometimes in serving undercooked pizza. (Case 1)

Follow so far? So yes, you're right, company A does not owe anyone pizza. But when customers go to buy pizza (content purchases) or when customers have placed future orders for pizza (subscriptions) and they are not happy because many of them don't like the new sauce or because the pizza store is closed due to oven breakdowns or oven maintenance, they speak out. Sometimes they speak out on forums, and sometimes they speak out by buying other food or pizza from other companies.

Notes for the few very special posters -- please note I have used the words many and some in describing people who did not like the changes of U49. I understand some are neutral and some like it.
Additional note - You may notice a correlation between the same people who spoke out on the Lammania threads saying they were against the proposed changes and the people speaking out now. Wouldn't you be doubly upset if company A said, well, this sweeter sauce is a needed change to make sure our oven doesn't overheat, and we have to make the change to reduce the pizza store from closing, and then on the first day of the sauce change the oven breaks and the pizza store is closed?

Thank you for reading, I tried not to use many big words.

[Searious]

I don't delete DDO mails. Too lazy to click delete, confirm, delete, confirm so I let them expire and auto-delete. Give me a delete all button and I'll help you out. But if I wanted to sort emails I'd switch to the work laptop also on my desk cause those hundreds of emails need sorting....

[Catabree]

I think you guys are brave even digging this deep in the code. Noone has been game to for year. Thanks. Im expecting a few issues over the next few weeks/months. Im also sure you will have a nice code base eventualy.

Thanks for trying. im willing to let you have a go.

Nice to see someone with such a positive attitude. GO TEAM SSG YOU CAN DO IT.....!!!!
Now we just need +50% reaper xp over the weekend just so we know they really care right?

[longsdale]

I always hear those babbling about americans and their "never change a running system" and now?
Now we see "never change a running system" ^^.

Well i am a bit upset that this downtime is happening in this way. Wasn't there a thing called Rollback and so on?

But at least the issue is adressed, BIG positive.

And i believe that SSG will now change their "running system" to avoid such issues in the future .

They even kept us informed what happened. That was the last we denounced about their downtimes and hey, they changed it and inform us now.
I do have believe in them. Everyone can make a mistake and you only can change it if you have knowledge about it.

SO stay calm and let's give them a chance.

FunFact: update 49 (4+9=13 let the conspiracy theories go ahead ^^)