Worlds Unavailable: Thursday, September 22nd

[ChicagoChris]

Should be in the near future, however it ends up being fixed. If we can get it in the next game update we will.

Why are shadow blades being treated any different than arrows/bolts from the Conjure Bolts/Conjure Arrows spells/SLAs?

These persist past single quests, but don't interfere with TRs.

[Cordovan]

Why are shadow blades being treated any different than arrows/bolts from the Conjure Bolts/Conjure Arrows spells/SLAs?

These persist past single quests, but don't interfere with TRs.

The Shadow Blades are a truly unique creation in the game in terms of their implementation, which is why they have been causing problems.

[Seydlitz]

The Shadow Blades are a truly unique creation in the game in terms of their implementation, which is why they have been causing problems.

And it would be a truly unique implementation if they had been tested first.
And it would be a truly unique situation if a backup had been correctly made
And it would be a truly unique update if it had actually worked correctly
And it would be a truly unique corporate stance if any dev gave a rat's ass

such a nice, truly unique turn of phrase... very versatile

[EmGreen]

All game worlds except for Argonnessen and Thelanis have reopened. The game worlds have been rolled back to an earlier state from about 4:00 AM Eastern (-4 GMT) on Thursday, September 22nd.

It was bad enough that you have to take two or three swings at every update (hotfix, hotfix, patch to fix horrendous bugs, etc) because you seem to think QA and testing prior to release isn't worth the time it takes, now it turns out it was possible for things to fall further than that.

BEFORE the update I crafted a bunch of swap items, upgraded items to legendary, and completely regeared a character, and then ran all of feywild on them. This roll back negated all of that effort. I can't tell you how mad I am that you have wasted so much of my time because you didn't take a backup right before making the update. Should I stop playing for a whole day or two before updates because I can't be confident SSG can make an update without risk? Is that a state of affairs that's good for the longevity of the game?

The boosted weekend bonuses are frustrating as an attempt to make things right because they require that we have a set period of time free, and require us to put in a chunk of effort to get the benefit proportional to what was rolled back. And now that the worlds are up there isn't even an actual apology or a statement acknowledging how badly things went, let alone promising to make changes/do better.

This is not a state of affairs in which I continue to be a paying customer.

[jfgddo]

It was like getting a call that one of your buds was in the hospital, and you get there and wait, and wait, and wait, just to find out you have to wait longer to find out if he/ she is going to be OK. You ask every 15 minutes and the same ole response and you know you will get a response eventually, but the WAIT is what sucks the most.

M2C

[LT218]

How is the game you produce doing?

Well, it's not a game, but the updates to the internal systems that my IT team releases usually go off without a hitch. They most definitely don't take our business offline for unplanned hours/days at a time.

Even more importantly, they don't cause bugs that go randomly deleting our customer, accounting, inventory, stock and shipping data.

It's amazing how much pain a proper dev-->test-->live environment setup will avoid *IF* your testers actually know the systems well enough to test it thoroughly.

Where I work, shoving a minimally/poorly tested update out the door that took our main systems offline for a day+ would raise some eyebrows the first time but would still mostly be considered a learning experience. That's mainly because we have a track record and reputation for *NOT* doing that sort of thing on a regular basis.

Shoving minimally/poorly tested updates out the door that impacted the business negatively in some way with every update would be a job ender. At some point, the loss of revenue, reputation and customer goodwill would outweigh benefit the team provides if we operated this way.

Not to mention, I would personally find it professionally embarrassing to have my name associated with a consistent stream of bugs and problems with every update.

[TheOnly_LightInDark]

Ok rerunning a quest or two I can handle but what about the astral shards I purchased disappearing ?

Can still see the purchase listed on MyAccount - History but my balance is 0

[dunklezhan]

Well, it's not a game, but the updates to the internal systems that my IT team releases usually go off without a hitch. They most definitely don't take our business offline for unplanned hours/days at a time.

Even more importantly, they don't cause bugs that go randomly deleting our customer, accounting, inventory, stock and shipping data.

It's amazing how much pain a proper dev-->test-->live environment setup will avoid *IF* your testers actually know the systems well enough to test it thoroughly.

Where I work, shoving a minimally/poorly tested update out the door that took our main systems offline for a day+ would raise some eyebrows the first time but would still mostly be considered a learning experience. That's mainly because we have a track record and reputation for *NOT* doing that sort of thing on a regular basis.

Shoving minimally/poorly tested updates out the door that impacted the business negatively in some way with every update would be a job ender. At some point, the loss of revenue, reputation and customer goodwill would outweigh benefit the team provides if we operated this way.

Not to mention, I would personally find it professionally embarrassing to have my name associated with a consistent stream of bugs and problems with every update.

Members of the community have been saying this basically the entire time I've been playing. Including me for the first few years I was here, for much the same reasons: I work in IT, this isn't how system testing is meant to work based on my understanding etc..

I still work at the same (public sector) place I have worked all along. Job's changed a bit I've gone from tech support out into projects and into skills and... anyway that's not important. The important bit is that whilst the above was all true in 2009 where i worked... it totally isn't now. Gone are the days where planning and methodical analysis are part of the picture it all about speed now. And that means accepting you'll break stuff and have to unpick and IF you test it's a cursory one somewhere close to whatever change was made to make sure there isn't a major issue with that and to hell with interoperation.

I've been waiting for probably five years for the world to fall in and it hasn't, emboldening more and more speed. I thought perhaps when our cheif officer changed and we got someone in from private sector that surely we'd get back on track, private sector must surely understand invest to save and all that... no, it's just got worse.

So I have concluded that whilst people like myself, customers and engineers might value the logic and benefit of testing... as the world gets more and more complex and faster and faster... it appears "the industry" has concluded the cost/benefit analysis for what we used to think of as "proper" testing has changed.

I suspect that world is gone, my friend. Its a brave new 21st century where gotta go fast is the only thing that matters. I strongly suspect that's not sustainable but as I say, the sky hasn't fallen in yet where I work. DDO has a couple of days unplanned downtime a year, which is less unplanned downtime than my home internet is inflicted with for which I pay a lot more. And lives depend on home internet. So it's just not that big of a disaster, is what I'm saying. It's not even necessarily indicative of bad QA or planning, it might just be that a business decision has been made that although tripling the size of the QA team might avoid this stuff, it's not worth it for the loss in revenue they see as a result.

It's frustrating and it's bad to have confidence only that if they do change something significant like banking then we're going to see problems for the next few patches/updates.

But like... we do know that's what will happen. Our group was just approaching cap when they announced the bank changes and we had to discuss whether we rushed to TR now ahead of the changes or let them go through and then waited a month maybe to see how it went. We chose the latter, and it still wasn't enough of a gap.

That's bad. But... kind of expected, so why the shock and surprise from everyone? (this is always what gets me with the humans though. The shock and surprise around perfectly predictable things. I get being mad about stuff). This is how DDO *is*. Play a different game cos I've been here since 2009 and it's always been I would say about the same level of bad when it does go wrong.

At least downtimes and rollbacks are now way shorter/still stupid-unusual.

[dunklezhan]

That's why some of us didn't bother to do anything serious yesterday AM. When some visible issues started cascading we knew something hit Critical Derp.

Too many changes, too short a testing period. I know it's nice to crank out content, but please stop awhile and concentrate on the backlog of known issues.

.................................

A rollback was pretty much the only real choice at that point, and even that might not have been far enough.

In summary, this.

[Stimulus]

Yup, and without even a token apology from SSG from what I've seen.

You might also have noticed that while server status updates were sporadic at best, the store sale announcement was up the instant the servers were back online.

lol

[merridyan]

This particular armchair developer makes his living going to companies and fixing the messes they made, when they no longer know how to get themselves out of it (-;

But on a more serious note: it really doesn't matters in this case. There is really no DB setup that cannot be migrated to a clustered solution. It's kinda what I often do for various scaling issues. If you want an overview of how to do that in various DBMS, Wikipedia has something on it (https://en.wikipedia.org/wiki/Multi-...gement_Systems), but you can also just google multi-master clustering and your DB of choice and you'll find solutions.

Yes you need to do some work to get it to behave. Nothing is free. But it is not rocket science either and it often comes with additional benefits.

Idk much about the american market, but here where I live I'd say 90%+ of the companies are on either MySQL (and successors like MariaDB, PerconaDB...), PostgreSQL, MS or Oracle. And all of those have solutions, either integrated or plugins.

DDO is on Windows servers using MS SqlServer. Probably a bit harder than linux servers, still do-able but they are a small team so possibly beyond the scope when they need to put out content to pay the bills. They are also not cloud based so adding additional compute would be a capital investment. Just guessing but I suspect each Server has it own database too, but who knows.

[Seydlitz]

Wait a minute.... let me get this straight...

After all the frustration, game time lost, in-game purchases lost, in-game gear lost, lack of transparency, failure to even concede how bad Update 56 (along with its follow up trashiness) has been...

Your idea of "making things right", of "making an apology", consists SOLELY of a whopping 25%xp bonus for two, count'em TWO! whole days, YAY! That's just sooooo AWESOME!

Gee, I can only imagine how great that is to the poor saps who have to do something else this weekend, oh wait.... I'm one of those....

****

Think that's the last straw

[moreuw]

Why HC servers affected? really few people TR there and if they lost something is much less than people lost because rollback, and even HC days will not not added?
Only people who lose life before rollback can be happy :-)

[Loholt-UK]

Ok rerunning a quest or two I can handle but what about the astral shards I purchased disappearing ?

Can still see the purchase listed on MyAccount - History but my balance is 0

The store wasn't rolled back, just the game, and it was rolled back to a point where you had zero AS. The store purchase will be refunded, Cordo has already stated this previously in this thread so don't worry, you won't lose out.

[Loholt-UK]

Wait a minute.... let me get this straight...

After all the frustration, game time lost, in-game purchases lost, in-game gear lost, lack of transparency, failure to even concede how bad Update 56 (along with its follow up trashiness) has been...

Your idea of "making things right", of "making an apology", consists SOLELY of a whopping 25%xp bonus for two, count'em TWO! whole days, YAY! That's just sooooo AWESOME!

Gee, I can only imagine how great that is to the poor saps who have to do something else this weekend, oh wait.... I'm one of those....

Me too. There is a thread in General DDO Discussions that proposes that instead of a weekly bonus being applied to a server, it is instead provided as a buff pot (I added that the buff pot should have an expiry date of 1 week) which I think is a great idea. It would mean that you and I and all of the other people who are busy at weekends would actually get some of these benefits sometimes.

[Loholt-UK]

Why HC servers affected? really few people TR there and if they lost something is much less than people lost because rollback, and even HC days will not not added?
Only people who lose life before rollback can be happy :-)

Really few is not zero.

[LT218]

Members of the community have been saying this basically the entire time I've been playing. Including me for the first few years I was here, for much the same reasons: I work in IT, this isn't how system testing is meant to work based on my understanding etc..

I still work at the same (public sector) place I have worked all along. Job's changed a bit I've gone from tech support out into projects and into skills and... anyway that's not important. The important bit is that whilst the above was all true in 2009 where i worked... it totally isn't now. Gone are the days where planning and methodical analysis are part of the picture it all about speed now. And that means accepting you'll break stuff and have to unpick and IF you test it's a cursory one somewhere close to whatever change was made to make sure there isn't a major issue with that and to hell with interoperation.

I've been waiting for probably five years for the world to fall in and it hasn't, emboldening more and more speed. I thought perhaps when our cheif officer changed and we got someone in from private sector that surely we'd get back on track, private sector must surely understand invest to save and all that... no, it's just got worse.

So I have concluded that whilst people like myself, customers and engineers might value the logic and benefit of testing... as the world gets more and more complex and faster and faster... it appears "the industry" has concluded the cost/benefit analysis for what we used to think of as "proper" testing has changed.

I suspect that world is gone, my friend. Its a brave new 21st century where gotta go fast is the only thing that matters. I strongly suspect that's not sustainable but as I say, the sky hasn't fallen in yet where I work. DDO has a couple of days unplanned downtime a year, which is less unplanned downtime than my home internet is inflicted with for which I pay a lot more. And lives depend on home internet. So it's just not that big of a disaster, is what I'm saying. It's not even necessarily indicative of bad QA or planning, it might just be that a business decision has been made that although tripling the size of the QA team might avoid this stuff, it's not worth it for the loss in revenue they see as a result.

It's frustrating and it's bad to have confidence only that if they do change something significant like banking then we're going to see problems for the next few patches/updates.

But like... we do know that's what will happen. Our group was just approaching cap when they announced the bank changes and we had to discuss whether we rushed to TR now ahead of the changes or let them go through and then waited a month maybe to see how it went. We chose the latter, and it still wasn't enough of a gap.

That's bad. But... kind of expected, so why the shock and surprise from everyone? (this is always what gets me with the humans though. The shock and surprise around perfectly predictable things. I get being mad about stuff). This is how DDO *is*. Play a different game cos I've been here since 2009 and it's always been I would say about the same level of bad when it does go wrong.

At least downtimes and rollbacks are now way shorter/still stupid-unusual.

I don't disagree with much/anything you said.

The problem is, the whole Agile development model was always intended to be "develop it fast, test it briefly, shove it out, then iterate and fix what you missed you missed *quickly*." Most places, and SSG in particular seem to miss that last bit which is really the most important piece to making Agile work well from a customer satisfaction and stability standpoint.

These days, it's "dev fast, test briefly, shove it out, fix one or two quick, easy things, then rush to the next shiny to sell more more more." The best is when they try to charge you to fix the buggy poo they rolled out in the first place.

That operation mode has cost several vendors we used to work with hundreds of thousands of dollars in lost contracts over the past 10 years. We're actually in the middle of switching to a new 7-figure ERP system contract from a different vendor because the current vendor has been doing that to us for the past few years and we're tired of it. We'd have been happy to stay with the current vendor/system and not do the significant amount of work and data conversion if that vendor had handled things differently.

Yes, the sky won't fall due to any of this. But with an aging MMO like DDO, it does accelerate the slow but steady war of attrition on the server populations. MMOs by their nature generally require healthy server populations for grouping, socializing, etc. Player/customer retention is key to any mature business model that lives or dies based on recurring income from subscriptions and repeat customers.

[thomascoolone64]

So everything I did yesterday, all the named gear and rxp I got was lost?

So im not the only one! I lost about 40k RXP!!!!
DEV INFO:

Account: Thomasniceone64

Server: Thelanis

Char Name: Titanreaper

[thomascoolone64]

Why are shadow blades being treated any different than arrows/bolts from the Conjure Bolts/Conjure Arrows spells/SLAs?

These persist past single quests, but don't interfere with TRs.

Theyre also a Physical Raid drop as well. check the named list fornthe Dryad and demigod raid.

[Nite_Prowler]

Wait a minute.... let me get this straight...

After all the frustration, game time lost, in-game purchases lost, in-game gear lost, lack of transparency, failure to even concede how bad Update 56 (along with its follow up trashiness) has been...

Your idea of "making things right", of "making an apology", consists SOLELY of a whopping 25%xp bonus for two, count'em TWO! whole days, YAY! That's just sooooo AWESOME!

The sad thing is that they expect us to pay to get some kind of compensation for something THEY Fd up.

I can tell you right now SSG is getting NONE of my hard-earned cash until they start doing a better job of managing these updates

Think that's the last straw

Indeed, I've canceled my VIP, and don't see myself getting one in the near future.
.
.
.


Well done Standing Stone Games .. .well done indeed ... /s