Worlds Unavailable: Thursday, September 22nd

[jfgddo]

So, what is up with Argo and Thelanis servers? Are they coming back up or not? What is the delay with these 2 servers? Last post I seen was hopefully later this morning? It is now 1 CST and still nothing!

An update would be appreciated!!

[JBS]

Argo is up but still no love for Thelanis

[Azmodious]

Argo back up...

[JBS]

Argo is up but still no love for Thelanis

I lied, Thelanis is up (see me doing a happy dance )

[Tanky]

I don't remember any dev's stating that there wasn't a backup done.

The last backup is from 5 hours before the server was brought down for a scheduled update at 9am. It's not unreasonable to expect to be able to roll back to 9am instead of 4am.

[Cordovan]

Argonnessen and Thelanis have reopened.

[Tanky]

Love armchair developers, know everything about a system they have never seen.

It's standard business practice to be able to rollback an information system to the point in time prior to an update is applied if it goes pear shaped.

[tankhealer]

It's standard business practice to be able to rollback an information system to the point in time prior to an update is applied if it goes pear shaped.

That may be... but I doubt I will renew my sub now or ever give another dollar to this company. This was just another screw up that really ****es me off. This rollback lost me 3 epic levels and the dam sword of shadows that I have been trying to get for years that finally dropped for me yesterday!!! I was in a raid group that we had to run VoN 5 2 times because some selfish person decided to go in VoN 6 on normal and wouldn't leave even after they died inside so we had to reform and run VoN 5 all over again.

[Hashrabal]

I don't remember any dev's stating that there wasn't a backup done....that was a player stating an opinion so....why are you carrying a flaming rumor??...

Because I don't allege them of being purposefully evil? Which would be the case if they, consciously and with better options used an older backup then necessary. That would be really ludicrous thinking.
No no, they'll have used the latest unaffected backup they have, as would any even semi-competent sysadmin do in that situation. Which means they did not take a backup before the patch, if people are missing time (items, xp, etc..) between the snapshot and the patch.

Now why do you think that is, hmm? I personally in this case go with Hanlon's Razor ("Never attribute to malice that which is adequately explained by stupidity."). Meaning I assume that either their system, procedure or training is not appropriate enough to take quick and reliable backups before patches. Which is not acceptable in 2022. The tech is there, use it. This is not new tech either...

[Hashrabal]

Love armchair developers, know everything about a system they have never seen.

This particular armchair developer makes his living going to companies and fixing the messes they made, when they no longer know how to get themselves out of it (-;

But on a more serious note: it really doesn't matters in this case. There is really no DB setup that cannot be migrated to a clustered solution. It's kinda what I often do for various scaling issues. If you want an overview of how to do that in various DBMS, Wikipedia has something on it (https://en.wikipedia.org/wiki/Multi-...gement_Systems), but you can also just google multi-master clustering and your DB of choice and you'll find solutions.

Yes you need to do some work to get it to behave. Nothing is free. But it is not rocket science either and it often comes with additional benefits.

Idk much about the american market, but here where I live I'd say 90%+ of the companies are on either MySQL (and successors like MariaDB, PerconaDB...), PostgreSQL, MS or Oracle. And all of those have solutions, either integrated or plugins.

[Heledore]

@Cordovan

I spent 5 days grinding Korthos and Harbor quests to get enough DDO points to finally buy a shared bank for my F2P account that I use for my Bards.

The sale WAS for 1046 points. I had 975 points. I was 3 runs away from getting my shared bank.
The shared bank is Now 1495 points because it is now AFTER September 22nd.

Thanks to this rollback, I am now exactly 520 away from my shared bank that I spent countless hours grinding for. This is an additional 21 more mind-numbing runs.

I just can't believe my luck. How is that fair and equitable?

I can't do another 21 hours of grinding

Can you bring back the sales, pls*. ;(

[* on time with our DDO Points :P]

[Heikenmoore]

I lied, Thelanis is up (see me doing a happy dance )

Yep, thelanis is back up but every time I try to enter a portal to a quest I get stuck on the loading screen. I mean I have now tried 10 times and stuck every time. Instead of a once and a while thing it is every time. Oh well, I am tired of trying. Will try again later and hoped it is fixed by then.

[krzyysiek098]

The game worlds have been rolled back to an earlier state from about 4:00 AM Eastern (-4 GMT) on Thursday, September 22nd.

That's a BS. Just lost a +7 tome because you are so incompetent not to make a backup copy right before an update (like during downtime). This is sick and I'm already tired of you making us beta testers, because of your laziness and stupid deadlines.

[LokiFrost20882]

Love armchair developers, know everything about a system they have never seen.

This doesn't have anything to do with "armchair developers" (or even developers) at all. It has to do with general and accepted Disaster Recovery Best Practices. The guy that wrote that thread is 100% correct.

Even Windows 7, 8, 10 and 11 does a behind the scenes backup (shadowcopy) for any patch or update (before and after) for rollback purposes. One does not have to had "seen" a system in order to comment on what is known as an industry standard practice. I have been doing this for a living since 1998- Disaster Recovery and Restoration on desktop and server platforms, physical and virtual, logical and clustered. DR is agnostic.

Sorry if this sounds argumentative but I just wanted you to have all the facts before violating the Golden Rule again.

[LokiFrost20882]

This particular armchair developer makes his living going to companies and fixing the messes they made, when they no longer know how to get themselves out of it (-;

Same here. I make a good living at it as well.

[Hashrabal]

This doesn't have anything to do with "armchair developers" (or even developers) at all. It has to do with general and accepted Disaster Recovery Best Practices. The guy that wrote that thread is 100% correct.

Even Windows 7, 8 10 and 11 does a behind the scenes backup (shadowcopy) for any patch or update (before and after) for rollback purposes. One does not have to have seen a system in order to comment on what is known as an industry standard practice. I have been doing this for a living since 1998- Disaster Recovery and Restoration on desktop and server platforms, physical and virtual, logical and clustered.

Sorry if this sounds argumentative but I just wanted you to have all the facts before violating the Golden Rule again.

I mean, in this case it's even more stupid. You could do this the right way, clustered DB, isolate and backup a node, automated of course. There are proper solutions to this.

You could (not should!...) also just use the ugly hack to take a snapshot of the DB host and call it a day. When disaster strikes, you have to dig through the DBMS files on the snapshot and restore to a new backup, which is slow, true*1, but all it cost you was 20 seconds to start a snapshot and some RAM for a couple days. Heck most data centers don't even charge for a single snapshot (at least where I live). That is the very very very least I'd expect any admin to do before a risky patch.

And we knew the patch was risky because the very same system (the bank) failed just a week or two before! It is not like this is the very first time something has gone wrong. It's not even the first time this month! How you can still be so careless (or clueless...) after you already messed up last time is beyond me.


*1 Ask me how I know (-;

[DSR1DES]

Folks are right, I got the rollback time incorrect, and it is indeed around 4:00 AM Eastern not 7:00 AM Eastern. Blame a long night and lack of sleep and multitasking...sorry! Will correct earlier copy now.



Unfortunately not due to the nature of the bug.

Dang sucks going to miss out on the xp bonus days since looks like I won't be able to reincarnate for a few more weeks then but it is what it is I suppose. I figured it would at least be possible to simply delete the shadowblades from the TR cache directly by just editing the database portion that covers it for the affected character but I have no clue how you guys handle it.

[Sylvado]

This doesn't have anything to do with "armchair developers" (or even developers) at all. It has to do with general and accepted Disaster Recovery Best Practices. The guy that wrote that thread is 100% correct.

Even Windows 7, 8, 10 and 11 does a behind the scenes backup (shadowcopy) for any patch or update (before and after) for rollback purposes. One does not have to had "seen" a system in order to comment on what is known as an industry standard practice. I have been doing this for a living since 1998- Disaster Recovery and Restoration on desktop and server platforms, physical and virtual, logical and clustered. DR is agnostic.

Sorry if this sounds argumentative but I just wanted you to have all the facts before violating the Golden Rule again.

DDO is not BCA

[tankhealer]

Someone robbed the bank. The insurance company was called. We got our stuff back.

The End

Actually we didn't get our stuff back, we didn't get anything for our lost time either, and to top it off my money was taken as well because of the DDO points I lost after purchasing them during the period of time that got rolled back.

[DRoark]

That's why some of us didn't bother to do anything serious yesterday AM. When some visible issues started cascading we knew something hit Critical Derp.

Too many changes, too short a testing period. I know it's nice to crank out content, but please stop awhile and concentrate on the backlog of known issues.

.................................

A rollback was pretty much the only real choice at that point, and even that might not have been far enough.