Lag after data centre move

[CaptainPurge]

Playing today was an exercise in frustration. We've stopped playing EE because of the lag issues. We see lag in public areas, in quests, & raids. We actually lag wiped in an EH Devil Assault, something that has never happened to us before.

I checked the traceroutes during the lag, & saw absolutely no difference from when it's not lagging. Latencies were good. Pinging Google & the Turbine address (gls.ddo.com) on a continuous basis gave exactly the same (good) results: Average of 15-16 ms with no packet losses. I'm beginning to doubt the routing theory, and think it must be something inside Turbine.

++
And the NI is less important as far as latency it seems. You can still hold a good latency while frozen in place lagorama. What is the NI pinging anyway, the forums?
WHAT IS THE NETWORK INDICATOR ACTUALLY DOING???? ???? ????? IT is Not showing actual networking data between game servers, we know this.

[mna]

I have worked for some of the largest producers of network management software in the world, installing COTS software for their customers. Very little software that is sold into a large corporate or government environment requires any system modifications. Customers simply will not put up with that kind of BS these days since it places additional burden on their IT staff to learn the idiosyncrasies of each different piece of software. Install the software, install a DB if that is required, configure them to talk to each other, and you're done. Often within just a few hours.

... I've worked in the server rooms of a producer of network management software. Any number of things did require system modifications, often fairly large too.

"Install a DB" ... heh. That was the thing that required the most modifications actually. I mean, what it needing to be high-availability and just about realtime, and heavyish traffic too... Direct-connected disks and db-level replication vs FC arrays and concurrent access, and all those other things... and then configured differently when they found an obscure bug, or the DB vendor just changed the low-level code and stopped supporting the older release.

Oh well. At least none of the "recent" versions used multi-initiator shared SCSI buses for the databases, any more.

And then there was the government side. Let's just say that they too bought stuff that required extensive modifications to existing systems.

Customization of the application itself is an entirely different matter, of course. That can take weeks or months (or longer, in some especially horrible cases) depending on the customer and their specific needs, on their ability to manage the scope of their feature requests, and especially on their central authority and the amount of hoops their various divisions might force a project to jump through prior to approval.

BTDT, also written code to deal with some of those extra hoops.

DNS updates typically take far less than a day to propagate. And any slowpokes are typically very small players like little Ma and Pa ISPs. My traces show that there was no issue, for me at least, crossing three major carrier networks, almost immediately after the servers came back up.
And as to your parenthetical at the end there, no, it does not require that.

Typically is quite far from the worst case. Now, of course the worst case is rare, but even so it sometimes does happen. And given exactly that, "does not" is incorrect - but "should never" is correct.

[Faltout]

That private IP address still times out for me. I have no idea how people are able to ping it. It is a private IP address.

My current theory is that if you are behind a router and on a private network yourself, then you cannot ping it. If you are going commando online with a public IP address on your computer's nic, then you may be able to ping it. (I still do not see how this is possible.)

Others have tried to explain that having a private IP address on the public IPv4 network is just fine. I still think it is internet breaking. That is just not allowed as I see it.

Try and ping this private IP address on my network: 192.168.5.55 .. you can't, because that's how private IP addresses work.

Any carrier that allows you to connect to a private IP address that is outside your own network is breaking the internet.

We are NOT pinging that private IP. We are simply trying to trace the route our packets take while reaching the destination. All our packets have a public IP address as destination, but the nodes within the route may say that they have private IPs.

We learn what the IPs of router 1 and router 2 are from the ICMP responses of those routers. Now, let's say that router 2 was the entrance to the turbine network and that to reach host B, we needed to pass from a router 3 inside the private turbine network. That router 3 would respond that its IP is something inside the private turbine network. Host B however, to which all our messages are being addressed, has a public IP.

[JOTMON]

Still lots of heavy lag.
either they have a lot of' tweaking' to do to get the systems to where they need to be so they are playable..
or perhaps the move to 'discountservers.com' was not to improve performance..

as it sits now.. many of the raid nights have been cancelled until further notice while we wait for lag issues to get resolved..
I guess I may go lag run meaningless solo TR lives while we wait for this to get resolved.

[Augon]

Its tough to decide where to post a comment about lag this morning. So many options....

I chose this one because the first few posts did not have the doom and gloom hyperbole or personal attacks on the devs.

This weekend my guild ran a few Raids with varying degrees of success.

We ran HOX rather successfully. There were lag patches and 2 deaths that were attributed to lag, but I didn't experience any major long lasting lag spikes personally and we finished the raid in the first round of charming the pups. So over all Id say it was ok. However, even if that was the extent of the lag, Id still say it was worse than before the move and needs addressed.

that wasn't the worst, however.

Next Raid on the list: DOJ! - need I say more?
DOJ was a joke, no doubt. Folks, that is a Raid I will be leaving off my list until the Devs have corrected this current Lag issue. I was assigned to the north of the room. We entered the main room and began fighting the first to executioners. This fight took probably 5 minutes. Yep, 5 minutes for a 12 man team with good DPS to kill TWO executioners. The lag was almost comical. I said then that if we took this long on the rest, we didn't stand a chance. We actually would never see another Executioner.

I ran to the north ramp. Well, "ran" is a bit of an overstatement. I stuttered to the north ramp. I froze some, I jumped some, I bounced back some, I finally made it to the top of the ramp. No mobs any where near me as far as I could tell, and then I died. Once we all died, the lag stopped and I bounced back to a dozen feet or so in front of the north ramp. I apparently never actually made it to the ramp. We have a pretty accommodating group for the most part. But no one really wanted to try it again. We decided to give Tempest a try.

We experienced some troubling lag at the start of tempest but it really got bad in the hall where you disable the fire traps and the Fire Eles pop up. We had a near wipe there. One of the guys managed to pop around the corner well enough to not die so he raised folks as the lag subsided. Due to that vicious lag spike and near wipe, one guy dropped out in a rage and another guy said he was dropping but ended up accepting his raise and moving on. The rest of the raid went ok. We had lag spikes occasionally but at least a few of us managed to get to a safe spot and raise folks when they passed. Again, we were able to complete the Raid, but it was way more difficult than it should have been.

I certainly do not feel this is the "Death of Raiding" or the "Death of DDO" as so many silly hyperbolic threads scream. I do think DOJ is unplayable for sure. Other raids are a pain and I will probably not pug while this is occurring, but I will continue playing. This certainly needs addressed as quickly as possible. It is probably not something our Dev team will be working on. This has the feel of a networking issue and IT admins will be trying to figure this one out.

I wish them luck cause I want to run DOJ next Saturday.

[KoobTheProud]

All I do is solo at this point. I'm getting a fair amount of lag in the process. Step lag, freeze lag, combat lag, etc. No obvious rubberbanding but I don't hammer the controls when I'm lagging either.

Prior to the data center move I lagged on a half dozen occasions over the entirety of my DDO experience spanning 10 years now. A few times in groups early on and a few times when everybody was having problems and they had to reset the servers to fix them.

The network status icon shows occasional 0.3% packet loss, the first time I've seen that. It shows that when I'm lagging sometimes, since that's the only time I look at it.

This has to be fixed because I will no longer play the game if it is not. DDO is an FRPG to me and nobody plays an FRPG that stutters and lags all the time.

Ilove the game but if it's broken beyond repair, well that's going to be it.

[donweel]

I edited my ddo settings file to setup 9000 as the port and set up the port trigger to that in my router, set up 9000-9058 as the open port and game play has improved quite a bit. I get the usual glitch here and there but things are pretty much back to normal. Normal questing, and explore areas, not raiding.

[Belgarath22]

Hello; I did several raids in last couple days in Argo; yesterday in guildie run we did Leg Normal Tempest run (really you can t handle anything harder with the lag)

We lagged and ruberbanded all raid long, from start to end... One thing we noticed is that we were all lagging when we got favor for the guild, massive lag spikes, lasting several seconds. Well, we also got ramdom lag spikes that we coudnt identify why; but I was wondering if all people are getting lag spikes in raids when you get guid renown..

I hope this will be solved soon.. it can be fun for a couple runs in normal with no fail.. but really the game is currently unplayable

[KookieKobold]

Hey folks!

Thanks for everyone who has been posting tracerts and various other information in order to help out!

That said, in order to help us help you even more, when you experience lag please assemble and post (or PM to me) the following list of information!

1. What time it happened
2. What server you are on
3. what is your ISP
4. What are you seeing lag wise (hitching, freezing, rubber banding, etc)
5. Are these being seen by everyone in your party
6. results of a tracert and pathping

Thanks!

[Wizza]

Hey folks!

Thanks for everyone who has been posting tracerts and various other information in order to help out!

That said, in order to help us help you even more, when you experience lag please assemble and post (or PM to me) the following list of information!

1. What time it happened
2. What server you are on
3. what is your ISP
4. What are you seeing lag wise (hitching, freezing, rubber banding, etc)
5. Are these being seen by everyone in your party
6. results of a tracert and pathping

Thanks!

Yo, I'm gonna help you:

Lag happens ALWAYS and IN EVERY SERVER and by EVERYONE.

[Nyata]

Hey folks!

Thanks for everyone who has been posting tracerts and various other information in order to help out!

That said, in order to help us help you even more, when you experience lag please assemble and post (or PM to me) the following list of information!

1. What time it happened
2. What server you are on
3. what is your ISP
4. What are you seeing lag wise (hitching, freezing, rubber banding, etc)
5. Are these being seen by everyone in your party
6. results of a tracert and pathping

Thanks!

so if we are getting more or less permanent lag in raids for example can we just give you the blanket information?

[Bolo_Grubb]

Hey folks!

Thanks for everyone who has been posting tracerts and various other information in order to help out!

That said, in order to help us help you even more, when you experience lag please assemble and post (or PM to me) the following list of information!

1. What time it happened
2. What server you are on
3. what is your ISP
4. What are you seeing lag wise (hitching, freezing, rubber banding, etc)
5. Are these being seen by everyone in your party
6. results of a tracert and pathping

Thanks!

Might be difficult as my Guild has stopped raiding due to the lag in every single raid we normally do since the data center move.

[Vorachtin]

Hey folks!

Thanks for everyone who has been posting tracerts and various other information in order to help out!

That said, in order to help us help you even more, when you experience lag please assemble and post (or PM to me) the following list of information!

1. What time it happened
2. What server you are on
3. what is your ISP
4. What are you seeing lag wise (hitching, freezing, rubber banding, etc)
5. Are these being seen by everyone in your party
6. results of a tracert and pathping

Thanks!

I don't know if I can answer these questions without garnering multiple infractions. So here is an emoticon

[Eryhn]

here's a thought, why dont each and every one at you at HQ go home an hour early. then when home, log into game on an epic toon via your home line, post an lfm
*turbine employe/dev XY looking for ingame experience - need full raid group - DOJ, TS, Wyrm, Peaks, all normal/hard once each*

just so we are on the same page...



as for the questions you asked, they cannot so easily be answered.

a laggy feeling is persistent almost all the time in form of some slight slugishness and movement/spell micro delay. typically, full group lag will often hit on spawn of larger mob groups in a more pronounced choppy several second freeze stutters in a row kind of way.

notably strange things:
*when clicking on quest npc to open the enter quest dialogue, there very often are long delays
this i was once in raid group with in such bad manner that it actually showed a person as having re-entered LTS, when probably all he did was spamm several clicks on the unresponsive entry dialogue, resulting in him entering the raid twice if that makes sense

*i have witnessed in group and heard several reports of spell effects like sleet storm damage, stinking cloud or burning blood clinging to a player after quest completion and persisting into public areas, these could only be removed by relogging the toon in question

*in almost all quests and raids that grant completion by defeating a final mob in a big fight (i.e. end of the road, legendary hound) the whole party will lock up immediately upon quest completion for 10-30 second, with the damage overhead floaties still scrolling down during that time, aka looks like server very often needs to catch up on combat

*in accord with that, lag very often happens not only on monster spawn, but also despawn by mob death, kill something, freeze for 2 secs. this has lead a few from my last night raid party to speculate the lag is tied to guild renown being granted, which i think however is just coincidental result of the killing of mobs ...

*when public instances lag out, most notable effect is being frozen in place several seconds, stuttering on 2-5 steps, rinse repeat, usually persisting till you exit or teleport out of the instance in question, which immediately clears up this effect. people sharing the instance with you are likewise affected by this, while people in your party in a different instance are NOT

*the same effect occasionally occurs in slayer areas, notable culprits for public: estar market, slayer: kings forest, giant hold epic

*item swaps during laggy conditions may or may not make things worse, or thats how it feels, unsure

*there was the odd stuck on loading screen, though this seems the smallest of the problems in comparison, happened to me like 3 times over the weekend and despite the conditions I was actually on a lot.


DURING ALL OF THE ABOVE, ingame netgraph shows a stable ping of 150-160 ms, which is normal for my european connection, though it was about 10ms lower on the old telia routing, which i attribute to the routing detour to kansas mentioned before. during all of this netgraph loss seems to be 0, though like in something like 10% of the case my ingame graph does show something like minimal loss of 0.3 to 0.6 %, which is however nowhere near enough to account for the amount of stutters, freeezes and sluggishness of the game at large.

meanwhile, when tracing or plotting, except for the ping fluctuations up to 250-400 ms within qwest.net hops, and the occasional packet loss on those same hops, i see nothing terribly catastrophic in my plots or traces. i think the qwest.net is worse routing then telia when telia worked ok (while when telia had issues it was bad too) and that it is only marginally impacting things. look at your gateway and your internal network config. look at ram/hdd/cpu usage. look if the bandaids for performance you did over the last year still work.

all of this you could have garnered by actually reading this thread.

p.s. my isp is telekom germany, i am getting routed to us on telekom net, then directly handed over to qwest or the same company under other name. the telekom routing over the pond has the occasional hickup once an hour or so, however not enough to expain my issues. i had ONE SINGLE DC this whole weekend, where i dced and the party worked normally, and that i could trace in pingplotter to such a hickup on the overseas backbone. ANYTHING ELSE, EVERYTHING THAT MATTERS, whole party affected, both in quests and in raids. even people who in the past were doing very well and said they never had lag now get it at the same time as everybody else.

to summarize, it is a cluster****, and it smells very serverside.

[Morroiel]

Hey folks!

Thanks for everyone who has been posting tracerts and various other information in order to help out!

That said, in order to help us help you even more, when you experience lag please assemble and post (or PM to me) the following list of information!

1. What time it happened
2. What server you are on
3. what is your ISP
4. What are you seeing lag wise (hitching, freezing, rubber banding, etc)
5. Are these being seen by everyone in your party
6. results of a tracert and pathping

Thanks!

1. Almost always - tends to be worse (instance wide lock-up type lag) when multiple event triggers are occurring (portal despawning, abbot tiles spawning/despawning, etc)
2. Thelanis
3. Cox
4. Everything from slight rubber banding/hitching, to complete freeze in place where all movement is impeded... when lag resolves usually i see everyone in party including myself go from full to dead... chat server still active during connection
5. Everyone in party is feeling lag/experiencing same thing as me
6. can't give you this as I don't run raids anymore till this problem is fixed..... tracert during idle in-game shown below (first few hops taken out for anonymity reasons):

[AtomicMew]

1. 6PM-12AM HST

2. Khyber
3. Time warner
4. All types of lag: hitching, freezing, rubber banding, etc
5. yes

[Seikojin]

In other words, still clueless?

That said, I wish you guys good luck to solve this issue as soon as possible. Unfortunately I can't help you with any info because I am not playing atm.

Well the problem is traces showed no real problems while players were experiencing effects in game. So now it comes down to establishing timestamps to reverse look up on server logs to determine what resource hits were during theses issues. Trace down to number of procs, ram, etc, if enough were being brought online in response, why not, etc.

What Eryhn said may help actually. If an event was held with prizes, you can gather players in a specific spot and monitor things on both sides.

[Qhualor]

Hey folks!

Thanks for everyone who has been posting tracerts and various other information in order to help out!

That said, in order to help us help you even more, when you experience lag please assemble and post (or PM to me) the following list of information!

1. What time it happened
2. What server you are on
3. what is your ISP
4. What are you seeing lag wise (hitching, freezing, rubber banding, etc)
5. Are these being seen by everyone in your party
6. results of a tracert and pathping

Thanks!

im going to be honest and i mean what i say with every ounce of respect.

i think it would be more helpful asking the people posting on the forums who DO NOT have lag issues. i would be sending you PMs at least 50 times a day.

[Eryhn]

im going to be honest and i mean what i say with every ounce of respect.

q.e.d ²

[mikarddo]

Such a mess.

For the first time in years I am seriously considering finding another game to play and I doubt I am alone.