Weekly DDO World Restarts on Wednesday

[New-RS-God]

WOW...

After 2 days of not being able to play, I am greeted with instantly logging in and client to close for this...

AAAAAAARRRRRRRGGGGGGGGHHHHHH!!!!!!!!!!
asfsdgsdgsdgsdg
AAAAAAARRRRRRRGGGGGGGGHHHHHH!!!!!!!!!!

Going to play Battle Toads.......

Anyway, see you on Sarlona when the Worlds are back up!

[moomooprincess]

Starting on Wednesday, July 27th (tomorrow), we will be doing regular weekly restarts of the DDO game worlds. These weekly restarts will take place starting at 8:00 AM Eastern (-4 GMT) every Wednesday. This is being done to promote the general health of the game environment. Click here for your local time.

NOTE: We'll put an estimated uptime up after we work through this enough to be confident in our process, but the estimated downtime should be around an hour or so.

NOTE: When the worlds reopen, it will take up to an hour for guild airship amenities to repopulate.

As a matter of fact, that is a possibility in the future. If there are a handful of things we want to fix/update, this provides us with a weekly window. Granted, it's still a new build and all, but it may well be that we can sneak in some work between patches.

Argh, I posted this as a 9:00 AM downtime. Should be 8:00 AM.

Looks like no one knew what time the servers were supposed to go down. 9 AM EST? 8 AM EST? rebooted now.

So, 9:17 AM EST was the time.

I missed the announcements since I just logged in, I thought the time was 8 AM EST, but hey, who cares about being on time.

[Ykt]

but the point being that even simply pushing 1 fix out for the weekly restarts would be great!

What if that fix breaks something? Then you get more down time (6-24 hours) and it disrupts the schedule. They may not have the dev and testing resources to work on weekly fixes while also working on regular tri-monthly updates and monthly patches.

[SiliconScout]

Any chance of putting this in the launcher. I didn't notice the info there and that being said the server restarted around 20 after ... was 3/4 through freshen the air and didn't have time to complete but had I known about this when I logged in I would have just done some inventory cleanup and auction house stuff rather than start the quest.

That being said a weekly restart makes a lot of sense for sure to keep the system running more smoothly, hopefully this is viewed as preventative and a triage solution not the actual correction needed to resolve the core problem. I am a little surprised you didn't pick an early morning time for the restart but I suspect you no longer have anyone on site overnight so that is likely the reason. You could consider having the people needed to manage a restart simply start early on the reboot day (and go home early as well), I know I implemented that in our department and most of my team are pushing to get to be the one to manage the maintenance window as it's viewed as a quiet time where they can get a lot more accomplished uninterrupted.

Thanks.

[SiliconScout]

Just loaded the launcher again and I see that it IS there however it's the second post on the launcher under the new issue of the chronicle. I typically only look at the top most post so perhaps on the Wednesday morning you could make sure to post it so it's on the top of the launcher.

At any rate thanks again!

[moomooprincess]

Just loaded the launcher again and I see that it IS there however it's the second post on the launcher under the new issue of the chronicle. I typically only look at the top most post so perhaps on the Wednesday morning you could make sure to post it so it's on the top of the launcher.

At any rate thanks again!

It was there yesterday, then the DDO Tuesday newsletter pushed it down one spot.

But, it has always been like that. The most important thing is never on top.

[KoobTheProud]

You could consider having the people needed to manage a restart simply start early on the reboot day (and go home early as well), I know I implemented that in our department and most of my team are pushing to get to be the one to manage the maintenance window as it's viewed as a quiet time where they can get a lot more accomplished uninterrupted.

You guys must be blessed. We used to hate being the person assigned to oversee maintenance because if anything went wrong, and there was lots that could go wrong, the entire day was blown managing the recovery process.

[SiliconScout]

It was there yesterday, then the DDO Tuesday newsletter pushed it down one spot.

But, it has always been like that. The most important thing is never on top.

First of all I just want to state that I haven't played in a couple of days (which isn't unusual) so I would not have seen it at the top yesterday.

Second it is very obvious and easy to understand that the launcher displays the posts in a LIFO format (Last In First Out) so that the latest post is always on top. I am good with that, it makes sense.

What I am suggesting is that they indeed post that on the launcher the morning of a restart (EVERY restart) so that it's the top message. These restarts are going to be a regular event from the sounds of it so what I am suggesting is a simple modification of their procedure for running these that they include posting a reminder.

I will note that in all the years I have played this game (just about since launch) this is the first time that I can recall when a scheduled downtime WASN'T the top most message on the launcher so I fail to see how what I am asking for or my concern that it wasn't on top could be considered unreasonable.

At any rate the discussion feels like it's beginning to circle.

TL;DR: I am hoping that Turbine will consider that they please post the restart messages so that they appear as the topmost notification in the launcher window just like all other scheduled downtimes usually do.

[SiliconScout]

You guys must be blessed. We used to hate being the person assigned to oversee maintenance because if anything went wrong, and there was lots that could go wrong, the entire day was blown managing the recovery process.

Well a lot of that has to do with the 'sales pitch' for sure I made sure to sell it as a quiet time and "cat's away" sort of day for the staff and they ate that up.

Generally we are but we also (thank the Gods) have a robust test environment and rollback/recovery procedures.

Basically if anything will take longer than an hour to diagnose and resolve we rollback. In some cases we re-attempt the process but in the event the update didn't occur it goes back to the test environment to figure out what went wrong and make the appropriate adjustments for the next month's window. I can honestly only recall having to do this twice in the last 15 years, that being said there is quite often something that needs to be addressed that didn't rear it's head in test but generally it's manageable.

More often that not we endup with more work on the helpdesk side of the department after the maintenance window for the handful of people who seem to be unable to touch a computer without having something goofy happen.

[Flywheel]

im going to answer that right now since turbine is on east coast it makes sense to do it in the morning. no matter where they would move the time someone would complain. so it really doesn't matter honestly. its still early enough or late enough it won't effect that many people.

I guess DDo runs on Windows servers? Perhaps it's time to run it on Linux?

[dunklezhan]

So rather than actually fixing the memory leaks, they'll just implement a weekly server restart. Is Microsoft running Turbine these days?

Interesting.

Its not a solution, its a workaround. This is how you do IT Problem Management: as soon as you have a viable workaround, you deploy it. This buys you time to find the actual solution and deploy that.

Sometimes it is true that you never deploy a true solution because its impossible, too expensive, or what have you.

The tragedy really is how long its taken them to identify (or, if I'm being kind, to get to the point where they could reliably deploy and were confident in) this idea as a workaround. Switch it off and on again isn't exactly a new concept (especially if you are targeting memory leaks), and fundamentally that's what this is, just writ large on a very delicate server ecosystem.

But there's nothing in cordo's post that says they will not be still working on fixing underlying causes, or other unrelated things just because they've set up a program of restarts. In fact later on he says they may well use it as a window for deploying smaller fixes through.

So whichever way you look at it, this is a good sign for the health of the game. Progress
has been made. With current available resources, that's good.

[Cordovan]

The game worlds are open. Sorry for the confusion on the start time this week.

[gabbatastic]

Game is 10+ years old, if u haven't noticed theres not a whole lot of ppl who still play. So lower playerbase and years of code over code = probally to exspensive to fix. The restarts will help a ton , heck I notice a huge difference in LE shroud after updates or restarts.

At least they r doing something.

I hate to be the one to put fuel on this fire, but perhaps we should consider porting our characters to a shiny new world... A fresh release might really invigorate the community...

[Ebondevil]

What if that fix breaks something? Then you get more down time (6-24 hours) and it disrupts the schedule. They may not have the dev and testing resources to work on weekly fixes while also working on regular tri-monthly updates and monthly patches.

Then chances are what was broken would have been broken in the tri-monthly update or monthly patch and you'd have to wait at least a month for a potential fix, instead of potentially a week.

I'm not saying they should rush the deployment of fixes, but if the fixes are ready for deployment there's no sense holding them back, and a faster deployment on Fixes generally helps to improve the perceived health of a game by the player-base.

[Silverleafeon]

Even if the memory leak problem is identified and fixed, I would still feel much more confident in DDO if the weekly restarts continued past any said fixes on the subject.

[mudfud]

Even if the memory leak problem is identified and fixed, I would still feel much more confident in DDO if the weekly restarts continued past any said fixes on the subject.

It's been 9 years since the memory leak was first brought up. If it's not fixed by now, it won't be I'm sure.

[Gratch]

For a 10 year old game spanning millions of lines of code and using 3rd party libs for a lot of things, I'm sure memory leaks have come and gone into the server code over and over. There's probably one being added right now and probably some known ones that are either not in Turbine's code or in tough to isolate or the person who knows how that complex code left so not worth touching it.

There were times when Turbine would have the servers up for months back in the day and you wouldn't notice - though there was a lot less to do then. There was even a time right after F2P launch that seasonal game events weren't crazy laggy on the rest of the world though they have been consistently laggy the last few years.

[knightgf]

As good as it may be to help reduce lag and a tiny fraction of in-game, this solution is a sign things are going REALLY downhill with the game. Perhaps those 'maintenance mode' predictions may not be crazy after all. Maybe it is a reality that could set in months or years from now.

[011775715]

[Garix]

As good as it may be to help reduce lag and a tiny fraction of in-game, this solution is a sign things are going REALLY downhill with the game. Perhaps those 'maintenance mode' predictions may not be crazy after all. Maybe it is a reality that could set in months or years from now.

One thing to consider: Codemasters (back when they ran the European Servers) did a Weekly (IIRC) restart pretty much from the release of the game. 7-10 years ago. Things ran smoother than it appeared to on the US forums for the same game (even being 2 weeks behind in releases/fixes).

So don't get too concerned about restarts implying upcoming maintenance mode. It may happen (in fact it will happen), just not necessarily this week/month/year.