Downtime Notice: Wednesday, 6:15 PM Eastern (-4 GMT)

[DarkSkysz]

Do you live under a rock?

Terms of service are not above the law. When a company somehow make their customers lose money (vip time, ddo market time sensitive consumables and other stuff) because they didnt cared at all to test their product before releasing, they WILL have to do some sort of compensation.

And lets be honest...they already knew live servers have far more access than lamia... so testing stuff like 'heartbeats' like they call it, should be taking the real numbers in the calc, not only testing numbers.

[Mister-shadow]

Don't get on to write posts much, but i am bored atm I may have had a few prior to a two year break and a few after. I will say this has to be one of the funniest string of posts i have read in some time, i all of a sudden feel the need to go get more cheese though.

Here is my interpretation of Cordovan's post: "Mommy said we have to go to bed see you all tomorrow" and I bet it will be Friday at best for the game to be up and reasonably functioning. Wish you the best at getting it going again Cordovan, please just don't let them really screw it up
(I don't think you will).

On a side note, I plan on just going outside again tomorrow and soaking in some sun and maybe trying to get the family of Foxes to trust me by tossing them some meat since today the male Fox was stalking the squirrels out in the front of the property and that was amazing to watch until the dog ran out onto the front porch and started barking and ran everything off. The resident ground hog came almost to the porch and took some food i tossed out, by then the dog was locked up in the back room. Deer were running through the backyard and stopped to graze a bit, until the dog went busting out the back door like he was an internet tough guy (all bark and no bite) and ran them off. Good thing the husband was not home,,,, or wait, maybe i wish he would have been we might have had a pile of venison to eat, we are growing tired of not having beef. Although real life is good, please fix the game i miss my DDO friends and now I'm rambling out of boredom because i don't watch TV much, to many honey boo boo shows and i don't play shooter games, so i cant join the hubby in his nightly rantings at the screen. Doesn't seem like much fun anyhow. Enough rambling, night all, see you on hopefully sooner than later.

Change foxes to Kangaroos and squirrels to possums and wallabies and its much the same here, a walk to calm down then cancel my vip coolly calmly and at peace with the world. Love the post by the way

[SierraSilverfox]

On a side note, I plan on just going outside again tomorrow and soaking in some sun and maybe trying to get the family of Foxes to trust me by tossing them some meat since today the male Fox was stalking the squirrels out in the front of the property and that was amazing to watch until the dog ran out onto the front porch and started barking and ran everything off. The resident ground hog came almost to the porch and took some food i tossed out, by then the dog was locked up in the back room. Deer were running through the backyard and stopped to graze a bit, until the dog went busting out the back door like he was an internet tough guy (all bark and no bite) and ran them off. Good thing the husband was not home,,,, or wait, maybe i wish he would have been we might have had a pile of venison to eat, we are growing tired of not having beef. Although real life is good, please fix the game i miss my DDO friends and now I'm rambling out of boredom because i don't watch TV much, to many honey boo boo shows and i don't play shooter games, so i cant join the hubby in his nightly rantings at the screen. Doesn't seem like much fun anyhow. Enough rambling, night all, see you on hopefully sooner than later.

you should try and get some pictures of the foxes and post them in a new thread in a more misc based section! i'd love to see them xD

[phillymiket]

What Caused This

the massive backlog of mail data

Thank goodness it's something I can act righteous about.

DELETE YOUR #@$% PEOPLE!!
IT'S A MAILBOX NOT A WAREHOUSE!!

I was worried it was something I'd have to defensively dismiss due to personal culpability.

[Puddinhead]

I appreciate the explanation. Previous updates that required unexpected outages have been frustrating when we've been left in the dark. As long as I have a feeling of what's going on, I can totally be patient. Take your time, fix the issue the right way the first time, and we'll gladly see you in game soon.

[Uralte]

Thank goodness it's something I can act righteous about.

DELETE YOUR #@$% PEOPLE!!
IT'S A MAILBOX NOT A WAREHOUSE!!

I was worried it was something I'd have to defensively dismiss due to personal culpability.

I blame the AH... they keep sending me money all. the. time.

[Mister-shadow]

I think in reality you could lose the generated ddo reward mail posts about ddo points just have them pop up at the favour on your toons chat or reward box where the xp is for your quest, at time then forget it it would be easier for players; don't know about codes or whatever but it would lose a load of mail.

[Shadobeh]

I think you guys are brave even digging this deep in the code. Noone has been game to for year. Thanks. Im expecting a few issues over the next few weeks/months. Im also sure you will have a nice code base eventualy.

Thanks for trying. im willing to let you have a go.

Yes indeed. I', happy to hear that this is getting a good once over. Digging deep once in a few is good. I've benn around since 2008-9. Sometimes **** happens. Always have a roll of TP close by.

[Puddinhead]

I think in reality you could lose the generated ddo reward mail posts about ddo points just have them pop up at the favour on your toons chat or reward box where the xp is for your quest, at time then forget it it would be easier for players; don't know about codes or whatever but it would lose a load of mail.

Could also look into fixing it so that when you ETR you don't notified of favor you're already received. That would probably drop a couple % of total mail that people don't check. I bet cache maintenance mails hurt too. A better way to manage the TR cache would prevent me from mailing things like thieves tools and pots to my alts so I can empty me cache. And before you say anything, I have every pack and shared bank slot purchased, but having every single past life in the game three times, I've acquired so much gear it's hard to maintain.

[arminius]

Do people still do the thing where they mail items to themselves to essentially use the mail system as an extension of bank space? That used to be a thing. I don't know if it is now that the most important items are named and bound to account.

I have noticed that whenever I log onto a long dormant character, they have announcement emails and such going back years. If you times that by all the inactive accounts and characters going back to 2006, that seems like it would be a really outrageous amount of data clogging up the system. Maybe the mail doesn't exist until the character logs in, and there is just a flag on the character that they are to get the mail? I would hope.

[Halciet]

What's Happening Now

I apologize for the poor rollout of Update 49 and the extended downtime. We believe we've located the source of the instability the live servers were seeing earlier this afternoon that prompted us to bring them offline. A larger explanation is below but just in terms of where we're at and when you can expect the game worlds to be open again:

We've got fixes in and are compiling a new game build overnight. If everything goes smoothly we'll be looking to get this deployed tomorrow morning and open the worlds before noon (eastern). We're discussing some added boosts to run alongside the planned weekend boost and will have more news on that once the game worlds are open.


What Caused This

We have an automated system that ensures the integrity of mail that we internally call a "heartbeat". Traditionally this has run every 6 hours. We found at one point that this had caused some crashes because the check got too large. We changed this heartbeat to happen every 5 minutes instead of every 6 hours so that it wouldn’t have such a backlog of data to deal with at once. On all internal servers, and public preview, this ran perfectly smoothly.

Once we hit live, however, it has become clear that the frequency of this heartbeat does not impact the effect it has on the server nearly as much as the sheer amount of mail data present. So, instead of a very noticeable hiccup every 6 hours, we were now having a slightly lessened hiccup every 5 minutes. The reason we did not see this issue on any test servers (including player preview on Lamannia) is because, even with the thousands of test case mail items created, there was not the massive backlog of mail data present that there is on a live server.

In the short term, we are reverting the heartbeat back to once every 6 hours with a patch tomorrow. In the long term we are investigating ways to clean this up and make it not an issue. This may require a combination of more aggressive (and publicly messaged ahead of time) culling of old or excessive mail as well as code changes to the heartbeart itself. Tonight, though, we don't have a timeline on those sorts of changes.

I appreciate being told what is going on and in this situation it gives us the call to action to be more tidy with our inboxes.

[phillymiket]

they WILL have to do some sort of compensation.

I doubt SSG is worried about a class-action suit over downtime.
As has been pointed out, the actual value of the downtime is measured in pennies not dollars.
It's not really about what people legally have to do.
A restaurant doesn't *have* to comp your meal when it's terribly late to the table.
They do it because it's good business to do so.

This has happened a few times over the years.
They did compensate a little bit on those occasions.
A big-time bonus weekend.
A nicer than usual store promo code.
I'm sure they'll do something like that.
They've never done more that I can recall and I doubt they will this time either.

[fgimino]

The issues do suck! Preeetttty sure their isn't a conspiracy to screw us over. Can imagine Cord thinking "Yeah! Let's screw them all over.... Including ourselves! That will make things better for sure!"

I'm sure no one wants the worlds up and running more than Cord at this point. You will live without the game for a day...... If you can't....... Maybe you should think about some sort of Anonymous group for gaming addiction.

It's gonna be OK peeps....... It's gonna be OK!!

I could see him as a chaotic evil doppleganging wizard of the Zhentarim.

[chaosmanager]

Now that is a realistic idea. Let players apply to be break testers. Allow one character transfer to Lamania. One week before update allow them to attempt to break the server. Exploit the server. Bug the server. If none of that happens, release to general public. I'd bet many, if not most hardcore players would do it just to open the "presents" before Christmas.

Maybe even put a bounty out, 1k SSG points for every issue first reported.

Thing is, they specifically put out a call for folks to get on Lam and test things. They even went so far as to request raid teams to sign on to Lam and try to push the envelope on the server and the changes. They're also more than willing to read/discuss bugs and inconsistencies. Even if they don't respond, the see things and act. Sometimes, that act is to disagree and say it stays (see the modified crit range on the special bows in the update). The fewer procs on double strike/shot that has seen a lot of ... *vocal* opposition both here and on Lam is another example. The discussion here wasn't ignored by the devs. It was more a case of the players not liking the response, even though they had an explanation as to why that was the particular response.

The big problem on their end was that it was a short notice window that could've (should've?) been anticipated due to Preview 1 and 2 occurring two weeks and one week before, respectively.

So, the problem wasn't a request for testers or ignoring player responses. Something like this, realistically, should've been up at least a week (if not longer) with some kind of tangible compensation - say a cosmetic armor that consists of jeans and a t-shirt that reads "I broke DDO". Now, SSG has their reasons for why the schedule the Lam sessions the way that do. I even agree with some of the possible reasons. To get a good cross reference in testing takes more than 48 hours of uptime, especially on a back end change like we have here.

[GramercyRiff]

My already all time low opinion of SSG just got lower. Less than zero. First they make the changes 49 makes while giving next to nothing back, then they can't even roll this awful update out. I have less than zero trust in SSG. At every turn, on almost every front, they routinely disappoint.

[BlueLiger]

Please mail all Sunflasks to Strimtom on Argonnessen

[fgimino]

Let me roll and see *Rolls a 1*. Wait a minute all the sides say 1 on this die. lol

[voxson5]

SSG please make this an event encounter

[Arkat]

Thank goodness it's something I can act righteous about.

DELETE YOUR #@$% PEOPLE!!
IT'S A MAILBOX NOT A WAREHOUSE!!

I was worried it was something I'd have to defensively dismiss due to personal culpability.

I'm guessing a real, permanent solution to the player base's storage issues would be better idea than just telling people to delete their sh*t.

[Puddinhead]

Thing is, they specifically put out a call for folks to get on Lam and test things. They even went so far as to request raid teams to sign on to Lam and try to push the envelope on the server and the changes. They're also more than willing to read/discuss bugs and inconsistencies. Even if they don't respond, the see things and act. Sometimes, that act is to disagree and say it stays (see the modified crit range on the special bows in the update). The fewer procs on double strike/shot that has seen a lot of ... *vocal* opposition both here and on Lam is another example. The discussion here wasn't ignored by the devs. It was more a case of the players not liking the response, even though they had an explanation as to why that was the particular response.

The big problem on their end was that it was a short notice window that could've (should've?) been anticipated due to Preview 1 and 2 occurring two weeks and one week before, respectively.

So, the problem wasn't a request for testers or ignoring player responses. Something like this, realistically, should've been up at least a week (if not longer) with some kind of tangible compensation - say a cosmetic armor that consists of jeans and a t-shirt that reads "I broke DDO". Now, SSG has their reasons for why the schedule the Lam sessions the way that do. I even agree with some of the possible reasons. To get a good cross reference in testing takes more than 48 hours of uptime, especially on a back end change like we have here.

I can recall two major outages of Facebook and Chrome in the past few years. After people smarter than me researched, it came down to "We're sorry, but it's simply impossible to test EVERYTHING." This from two of the largest, best resourced operations on the planet. I feel we have a clear explanation as to why this wasn't caught in testing on Lam, and why more testing on Lam wouldn't have caught it.