Lag after data centre move

[Holymunchkin]

stuff....

Here's the thing right...

The people experiencing this game-breaking lag are leaving. The people who aren't are staying.
The people who aren't obviously don't run 12-man raids on LE or EE.
This game has been broken for that subsection for over 2 years now.
People reach this height and then make one or two decision...

...I'll bear with it...

OR

I'm out---off to greener pastures

I've canceled my sub personally. Turbine has a month to make LE shroud playable for me or I won't bother with this game.

[Severlin]

So this morning we did optimization work to clean up network pathing problems and open up new network paths that should have alleviate, if not solve, lag related to network pathing. Thank you for all the trace routes. At this point we will be watching tonight to see who is improved, who isn't, and whether or not there is still lag that is related to server performance rather than network pathing.

Sev~

[Gauthaag]

So this morning we did optimization work to clean up network pathing problems and open up new network paths that should have alleviate, if not solve, lag related to network pathing. Thank you for all the trace routes. At this point we will be watching tonight to see who is improved, who isn't, and whether or not there is still lag that is related to server performance rather than network pathing.

Sev~

cool, thanks for information

[Mercantile_Joe]

Gah! Just ran an LN Shroud yesterday to try and just get a completion despite the lag. I'll make sure and run HoX, TS, and DoJ tonight to see if I notice any changes on my end.

[Livmo]

So this morning we did optimization work to clean up network pathing problems and open up new network paths that should have alleviate, if not solve, lag related to network pathing. Thank you for all the trace routes. At this point we will be watching tonight to see who is improved, who isn't, and whether or not there is still lag that is related to server performance rather than network pathing.

Sev~

Thanks for doing this!

I hope it helps.

Happy Friday!

[Elsbet]

So this morning we did optimization work to clean up network pathing problems and open up new network paths that should have alleviate, if not solve, lag related to network pathing. Thank you for all the trace routes. At this point we will be watching tonight to see who is improved, who isn't, and whether or not there is still lag that is related to server performance rather than network pathing.

Sev~

Thank you! This merits its own post/announcement in service news.

[Varthalos]

Did a VoN5 and several single group quests since this fix and it's no better.

* I should mention it's no better for everyone in the raid or group not just me, we're all lagging all at the same time.

[Severlin]

Did a VoN5 and several single group quests since this fix and it's no better.

* I should mention it's no better for everyone in the raid or group not just me, we're all lagging all at the same time.

What server and what time?

Sev~

[Dragavon]

What server and what time?

Sev~

Same here. Been running raids with guild on Ghallanda for the past couple of hours, bad lag in Von 5-6, LTS, LH and MoD.

If any change it is worse now. Certainly not better.

[Varthalos]

What server and what time?

Sev~

Khyber

Any time I'm in a quest with more than one person and we pull more than one or two mobs.
If I'm solo and I pull a bunch of stuff it lags. If I'm in a group and we all pull a bunch of stuff it lags a lot. If I'm in a raid and there is a lot of stuff up it locks up to the point of being unplayable. This is all quests all of the time.
Specific examples would be LE tempest spine is a slide show any time mobs are engaged, soloing to curse the sky gets very choppy if I pull more than 1 pack when solo, I could give dozens of examples if needed but what I'm saying is all quests all the time and it is directly proportional to the number of mobs and player characters interacting with each other.

[tmberwolff]

What server and what time?

Sev~

All day from 10a - 5p, CST. Argo. Every quest, without exception. Solo, group, raid.... tried them all. Even in the public areas. Actually, today has been worse, because I couldn't zone into gianthold from FoT, or Kundarak from market. Had to kill program after 3-4 minutes and restart.

Look, I don't post much, but I've stuck with DDO since beta... I've seen DDO go through some extremely rough times, but this... this is the worst. It's testing my patience.... Stop insulting your playerbase by asking us to do traceroutes and pretending that this problem might be on our side... You moved datacenters, and suddenly EVERYONE in the game has lag problems. Not DC's mind you... Not connection issues. The problem is with your server.

If you want to kill DDO, just keep doing what you're doing. Otherwise, you need to switch back or upgrade your servers. It really is just that simple.

[LT218]

What server and what time?

Sev~

Ghallanda. 8:30pm EST. LH Shroud run. Constant lag-lock deaths and we failed by not being able to find/kill Portal Keepers fast enough due to the frequent lag locking. Lag is not any better than before.

On side note, you've had multiple smart players with deep networking career backgrounds providing evidence that the main lag problems are not with your public network and routing for several days. If the lag issues are network related, it is on the private side of your network and completely internal as we're seeing none of the regular indications one would expect to see if the issues were on the public facing side.

You should find those posts and give them consideration regarding where to focus your investigations.


P.S. It would be immensely helpful to the sharp networking guys who are willing to help you troubleshoot this from the client end if you had your networking team allow pings/ICMP and traceroutes to the world servers and provided us with the list of those IPs so additional potential problems on the public facing side could be ruled out.

[Cadveen]

Please just take the game down for a week and fix the issues. The game is becoming almost unplayable.

[Seikojin]

What server and what time?

Sev~

Sarlona and from 5:30 pm pacific to 7:20 pacific, Saturday march 19th. We had consistent lag all night as we did catacombs. Every now and again the lag would get really bad, then calm down to just annoying.

Considering how it was before the move for me, this is drastic degradation. I even went through a lot of work to make sure the cable provider out here ensured the stability and quality of the connection.

Before I had almost no lag ever. And every now and again would have spikes, but nothing unexpected or uncalled for. So I would have said lag was 0 out of 10 on the annoying me scale. And when I had it, it would be a 3 at worst.

Now it is 3 as a low, and 5 or 6 when questing, and 8 to 9 every 20 minutes or so.

[Cableman]

What server and what time?

Sev~

Sarlona, 9:45PM CST doing the Pit solo, lagged in place for almost 5 minutes unable to move.

[Catclaws]

Sarlona. Gianthold slayer area. Lagging constantly. 8:17pm PDT.

[Jomee]

Howlers and shadar-kai at end fight causing studdering, relentless, unplayable LAG. The other previous quests in the chain were also laggy when the tons of mobs spawned for the end fights but it was still playable.

Server: Thelanis
Time: 11:00 Pm EST

[tmberwolff]

Running a little experiment on Argo right now. Sitting in a DW raid (10:05pm, CST), and no one has been able to move for ~10 minutes. We decided to sit and see how long it takes before someone DCs. But, we're not DC'ing. We're all chatting on the voice server and chat server. So, no connection problems or overall network issues. Just, whatever thread is running that should be letting us actually play the game is dead. It would be almost comical if it weren't for the fact that I'm paying to play an unplayable game.

Multiple people pulled a traceroute, and there were no drops and nothing over 50ms between anyone and the server. So, it is definitely server side.

Time to own up and fix this ****. Seriously.

ok, experiment over. 10:15 CST. Over 20 minutes, with no one able to move, and no DCs. We can move now, but still extremely jumpy.

[Cableman]

What server and what time?

Sev~

Sarlona, 10:27am CST, doing quest scoundrels run, froze in place for over 3 minutes.

[Deadlock]

EE Breaking the Ranks right now on Ghallanda - lagging like hell, almost a full lag wipe with the Orcs.

Lag in every single Stormhorns quest so far, but nothing as bad as this.