Worlds Unavailable: Thursday, September 22nd

[Arkat]

Instead of the 100 EXP gem I got on my silver roll yesterday morning, I got a 400 EXP gem AND a Long Lasting Elixir of Supreme Ability today!

Best rollback ever!!





(Yes, I realize I could have gotten the 100 EXP gem yesterday AND the 400 EXP Gem + Elixir today had they not done a rollback at all. You're no fun. )

[Tanky]

The game worlds have been rolled back to an earlier state from about 7:00 AM Eastern (-4 GMT) on Thursday, September 22nd.

Closer to 4am.

[Megahurtz]

So everything I did yesterday, all the named gear and rxp I got was lost?

Yup, and without even a token apology from SSG from what I've seen.

My question for them would be why the rollback was from hours before the downtime. Surely a grown up responsible company takes backups before they do major work on their systems and doesn't just say "hey we did a snapshot a couple of hours ago, it'll be fine!"

[Urklore]

Thank you for this!

We will be working to automatically refund purchases made after the time of the rollback on Thursday to when the game worlds went down.

[LokiFrost20882]

I deleted my post because someone at SSG/DDO reached out to me and made everything right as rain.

Thank you!

[Seydlitz]

This is truly unfair - lost time, xp (18 to 20 on hardcore), gear, etc. and not one speck of remorse or compensation. Just ****ing swell - no more VIP from me, this settles it. Europa 3 comes out next month... might be time for change.

[HailBlazR]

It's definitely unfair that no compensation has been handed out whatsoever, but I'm not sure why any of you guys were expecting anything more from SSG at this point.

SSG have a long and proven track record of contempt and disdain for their players. What makes you think this would be any different?

Best you can do is speak with your wallet. Stop spending money on the game. There are hundreds of better ways you could spend it, like getting a tattoo of a pygmy marmoset on your buttocks.

[Redgob]

All game worlds except for Argonnessen and Thelanis have reopened. The game worlds have been rolled back to an earlier state from about 7:00 AM Eastern (-4 GMT) on Thursday, September 22nd. We have also completed work to correct the issue that caused some items to go missing from players' Reincarnation Caches yesterday, so no further item loss should happen when players try to move items out of their Cache.

Seriously? 7:00 AM on thursday? When the patch was on 9:00 AM? It was way more than that. It's mind boggling how there was no database backup right before the update.

[Gnominal_Aphasia]

I'm sorry you have to repeat some Hardcore questing, like many of us. But join in rejoicing, those that had perished, have now been risen. The very first unplanned second life on the hardcore servers.

Your comment worries me; can your misplaced positivity be truly so myopic?

The only reason why the Hardcore server needed to be rolled back is because of the chance that some characters would die. This is wholly the responsability of SSG for (1) implementing changes that broke systems, and (2) not providing sufficient time and warning to players to log out safely. The situation was created by SSG and their solution just compounded problems.

Again, I don't blame the Devs or Cordovan, but the people making the decisions, so poorly again and again.

[HelvanderSeries6]

So everything I did yesterday, all the named gear and rxp I got was lost?

Is there any word when Argonnessen and Thelanis will be back up.../Sadness and Despair.

[Loholt-UK]

Yup, and without even a token apology from SSG from what I've seen.

My question for them would be why the rollback was from hours before the downtime. Surely a grown up responsible company takes backups before they do major work on their systems and doesn't just say "hey we did a snapshot a couple of hours ago, it'll be fine!"

Industry standard is daily so anything less than a day is fine.

[Cordovan]

Folks are right, I got the rollback time incorrect, and it is indeed around 4:00 AM Eastern not 7:00 AM Eastern. Blame a long night and lack of sleep and multitasking...sorry! Will correct earlier copy now.

Any chance people with this TR blocking issue could be fixed on a person by person basis via tickets in the meantime? Like manually deleting them from their cache? Had no luck with tickets for two weeks but I assume that was due to the fact you guys had a global fix coming out in 56.2.

Unfortunately not due to the nature of the bug.

[Heikenmoore]

Is there any word when Argonnessen and Thelanis will be back up.../Sadness and Despair.

Unfortunately I am waiting on Thelanis to reopen. Makes me worry that the rollback point on Thelanis is several days ago and we would lose not just 2 hours of running but several days. Hopefully I am wrong.

[Loholt-UK]

Your comment worries me; can your misplaced positivity be truly so myopic?

The only reason why the Hardcore server needed to be rolled back is because of the chance that some characters would die. This is wholly the responsability of SSG for (1) implementing changes that broke systems, and (2) not providing sufficient time and warning to players to log out safely. The situation was created by SSG and their solution just compounded problems.

Again, I don't blame the Devs or Cordovan, but the people making the decisions, so poorly again and again.

The Hardcore server needed to be rolled back for the same reason as the other servers, people had lost stuff from their TR caches and rollback was the most efficient and quickest way to reinstate those items. Admittedly there were less TR caches on the HC server but there were not zero so it had to be done. That people got resurrected as a result is an undocumented bonus for those people and I'm happy for them. The point about not logging out safely is moot because the servers were shut down and therefore could not process anyone's death so nobody died due directly to the servers shutting down.

[zachmandude]

Sincerely, thank you for your hard work.

A question for Cordovan, if you have time: Recently emptied my TR cache before this previous update and I was ready to TR, but held off due to some IRL things.

Once Thelanis and such is brought back online, I planned to finally TR my character and the like. In the current state of things, is it safe to TR or is it best to hold off?

Thank you for your time and hard work, hope you guys get some rest and some sleep. I appreciate the work you guys put in to get things back to a working state.

[Handsome_Potato_86]

Yeah, so... Look i'm not trying to say any of us are entitled to anything
in any way, shape, or form, buuut...: I feel like it would be a very
kind gesture for the... Well, i don't really wanna call it "Inconvenience",
but, y'know... The little whoopsie that happened yesterday, that affected
the players (some harder than others). Something nice. Nothing big,
like i'm not saying give everyone an Ottos box or something.
Maybe a pet rock. Or some former anniversary gift.
Kobold sentient jewel would be cute...? :>
Just something to say "We're really sorry! <3" and to sweeten
the way/time back to where people were before the rollback

[Hashrabal]

It's mind boggling how there was no database backup right before the update.

I would be fired from my job for that.

This is just unacceptable. It's 2022; there are a lot of good, free clustering solutions for all kinds of databases. Take a node offline, backup from that, patch the game servers in the meantime. Re-join the node 30m or so later, let the replication take care of it. It's really not rocket science and costs you exactly zero minutes of additional downtime.

This is not hard anymore, you can automate stuff like that in under a hundred lines of scripts. It's not costly either. Worst case it costs you an additional VM for a couple hours. That is what... 10$? 15$ if you are with an extortion... I mean "popular cloud service provider"? That is the cost of one lost customer's VIP status for a single month. There is just really no excuse anymore, the tech is there, it's free and it's easy enough to use.

[jellyfish21]

Someone robbed the bank. The insurance company was called. We got our stuff back.

The End

[Ralatar]

I would be fired from my job for that.

This is just unacceptable. It's 2022; there are a lot of good, free clustering solutions for all kinds of databases. Take a node offline, backup from that, patch the game servers in the meantime. Re-join the node 30m or so later, let the replication take care of it. It's really not rocket science and costs you exactly zero minutes of additional downtime.

This is not hard anymore, you can automate stuff like that in under a hundred lines of scripts. It's not costly either. Worst case it costs you an additional VM for a couple hours. That is what... 10$? 15$ if you are with an extortion... I mean "popular cloud service provider"? That is the cost of one lost customer's VIP status for a single month. There is just really no excuse anymore, the tech is there, it's free and it's easy enough to use.

I don't remember any dev's stating that there wasn't a backup done....that was a player stating an opinion so....why are you carrying a flaming rumor??...

[merridyan]

I would be fired from my job for that.

This is just unacceptable. It's 2022; there are a lot of good, free clustering solutions for all kinds of databases. Take a node offline, backup from that, patch the game servers in the meantime. Re-join the node 30m or so later, let the replication take care of it. It's really not rocket science and costs you exactly zero minutes of additional downtime.

This is not hard anymore, you can automate stuff like that in under a hundred lines of scripts. It's not costly either. Worst case it costs you an additional VM for a couple hours. That is what... 10$? 15$ if you are with an extortion... I mean "popular cloud service provider"? That is the cost of one lost customer's VIP status for a single month. There is just really no excuse anymore, the tech is there, it's free and it's easy enough to use.

Love armchair developers, know everything about a system they have never seen.