Lag: DETAILS

[Thanimal]

I hold the possibly generous belief that if fixing the lag were easy, Turbine would have already done so. I further hypothesize this means it must be a subtle, evil bug. So what I want to do here is to provide as much detail as I can about an experience that ended up sucking because of lag, in hopes that this is useful in ultimately tracking down and fixing the bug. Perhaps others can do the same either here or in their own threads. Naturally I can't stop mindless "fix the freakin' lag you lazy developers" posts here or elsewhere, but I'll just remind you that's been done and doesn't seem to have helped.

I was running as Vrol on Thelanis from roughly 9p ET to 10:40p ET last night (3/8/09), doing Wizard King (I hope that's the name -- I'm a n00b in higher level content because I constantly start new characters!). Group of 6, mostly from the guild Remember Honor. Although there was random ~1 sec lag here and there, the first hour and a half was at least playable. Then pretty suddenly we all began to experience massive lag.

[Comment: This seems to eliminate the "client side" hypothesis that once or twice I've seen thrown around. The probability of 6 geographically distributed players hitting msasive lag at the same time on their local clients is negligible.]

The lag quickly ramped up to completely unplayable -- taking a step once every several seconds, somewhere around 10:30p ET. It remained like this for about 10 minutes, after which there was a slight improvement: still unplayable, but now able to take several steps at a time before lagging out.

At roughly 10:45p ET, we gave up. We then all recalled out. Once out of the instance, the lag disappeared and we could run around normally in the Sands.

[Comment: This seems incredibly important from a diagnostic standpoint, so I'm especially curious if this has been the standard experience. It seems to prove that the issue isn't hardware related, because I can't see why we would have changed hardware when we recalled out. It seems to suggest to me that something went wrong in the instance -- i.e. some kind of software bug.]

Unfortunately, since we weren't running a controlled experiment, we didn't try going back in, so I can't say if the "saved" instance remained hosed until it reset. Perhaps others have done that and can provide that information.

Good luck Devs! Godspeed to find and correct this horrible lag!

[Thanimal]

Just remembered something else that might matter: During this entire experience, there was never any noticeable voice lag -- we continued to be able to converse about how wonderful the lag was.

In my experience, whenever I've experienced obvious client-side lag, my voice has also lagged.

[MrCow]

there was never any noticeable voice lag -- we continued to be able to converse about how wonderful the lag was.

Turbine goes through a completely separate company to handle the voice transactions.

[Deadz]

One players lag can lag the whole group, can it not?

Players toons do interact with each other

[Thanimal]

One players lag can lag the whole group, can it not?

It cannot. There is no need to synchronize transactions in that manner, and DDO clearly does not: I've had a party member lag out locally before and it has no impact on anybody else. If I am lagging, it means I am getting essentially ALL of my packets on a huge delay. If somebody else is lagging, I may get updates about THEM on a delay, but this will not effect my own communications. These transactions are asynchronous.

Too bad that's not the problem: There would be a zillion examples they could turn to for how to fix THAT!

[Thanimal]

Turbine goes through a completely separate company to handle the voice transactions.

Would have guessed as much. I was just bringing it up because it provides further evidence against the client-side lag hypothesis.

[negative]

It seems pretty clear what you experianced was server side lag. However, there are still lag issues talked about on the forums (or I've seen in-game) that seem to be client side. Granted, I don't think this was one of those issues.

At roughly 10:45p ET, we gave up. We then all recalled out. Once out of the instance, the lag disappeared and we could run around normally in the Sands.

[Comment: This seems incredibly important from a diagnostic standpoint, so I'm especially curious if this has been the standard experience. It seems to prove that the issue isn't hardware related, because I can't see why we would have changed hardware when we recalled out. It seems to suggest to me that something went wrong in the instance -- i.e. some kind of software bug.]

Depending on the server setup this may not be the case. Each "server" is not a single box. I believe it's quite possible that switching instances will switch the hardware you are connecting to.

[Mattee]

It seems pretty clear what you experianced was server side lag. However, there are still lag issues talked about on the forums (or I've seen in-game) that seem to be client side. Granted, I don't think this was one of those issues.

Depending on the server setup this may not be the case. Each "server" is not a single box. I believe it's quite possible that switching instances will switch the hardware you are connecting to.

So therefore, the lag is server-based.... But wasn't there a MASSIVE downtime earlier this year to "upgrade the server hardware"? In my experience a hardware upgrade usually improves things... not makes them much much worse (I sight personal lag times of up to 2500ms over the last two months).

Also, why don't the servers carry the same hardware? and for that matter, What Is the hardware? At times it seems like they've hooked up an Atari 2600 to run the common areas! (Bit of early-morning sarcasm. sorry. hehe)

[negative]

So therefore, the lag is server-based.... But wasn't there a MASSIVE downtime earlier this year to "upgrade the server hardware"? In my experience a hardware upgrade usually improves things... not makes them much much worse (I sight personal lag times of up to 2500ms over the last two months).

Also, why don't the servers carry the same hardware? and for that matter, What Is the hardware? At times it seems like they've hooked up an Atari 2600 to run the common areas! (Bit of early-morning sarcasm. sorry. hehe)

Someone can correct me here, but I keep hearing that Turbine outsourced the servers recently. I'm guessing that was the "upgrade", if you get my drift.

Previously I think they were on blade servers, I don't know what they are on now. Probably more blade servers.

I ussually run around 60-120 fps, but I still get the server-wide lag spikes occasionally. If you really want some insight in whether you lag is more likely one or the other, just hit control-f and watch your framerate. If you are lagging with a high framerate, it's probably network or server related. If you are lagging with a bad framerate, I'd say it's your computer. My guess is those that complain about disco balls will see a bad framerate around them, unless there really is some sort of bug in the server code (which may be true as I've rarely seen what I believed to be server lag when disco balls are cast in part 1 of shroud).

I'd say a high latency is also a good indication of network or server lag. But that could be something out of turbines control. Could be your isp in that case, unless DDO is the only app with bad latency.

Bottomline is lag isn't simple.

I'm just an amateur computer nerd though so I could be way off base.

[Missing_Minds]

In your running of wizard king, how many of the mobs were you killing on sight? Or were you just zerging past them?

I'm actually starting to believe that zergers are the cause of lots of lag. My bases for this is the house J quest. The spire with the lich. (I want to call it 'and the dead shall rise. but I"m pretty certain I'm wrong on the name.) But that quest would spawn so many active NPCs that it would literally lag the party badly. Why? To many calls back and forth between all of the NPCs talking. I actually proved this when I jumped into the middle of about 40-50 odd of the skeletons that spawned and started to take them out. At the time I figured I was just going to have fun being suicidel, but after about 20 of them were taken out, we started to actually visually see lag reduction happening. Once the floor was cleared of the undead, we were moving normally again.

Now, one thing I will note, we never did lose connection or were in risk of losing connection under this circumstance.

[Thanimal]

In your running of wizard king, how many of the mobs were you killing on sight? Or were you just zerging past them?

I'm actually starting to believe that zergers are the cause of lots of lag. My bases for this is the house J quest. The spire with the lich. (I want to call it 'and the dead shall rise. but I"m pretty certain I'm wrong on the name.) But that quest would spawn so many active NPCs that it would literally lag the party badly. Why? To many calls back and forth between all of the NPCs talking. I actually proved this when I jumped into the middle of about 40-50 odd of the skeletons that spawned and started to take them out. At the time I figured I was just going to have fun being suicidel, but after about 20 of them were taken out, we started to actually visually see lag reduction happening. Once the floor was cleared of the undead, we were moving normally again.

Now, one thing I will note, we never did lose connection or were in risk of losing connection under this circumstance.

GREAT question! The information could be important to debuggers. This was actually a slow run -- my guild was teaching me the quest and were stopping to kill everything. I don't think we left anything alive or trailed any monsters around.

Last night I had a case more like what you're describing. I was zerging Dryden's Tomb and mostly not bothering to kill anything and then when I tried to read the insciption, surrounded by random baddies, I started to experience massive lag. (Just barely got out alive!)

So that may be a contributing cause, but I'm quite certain there is more to it than that, because I recently was part of a Delera's zerg in which nearly nothing was killed, and lag never became an issue at all.

[juniorpfactors]

In your running of wizard king, how many of the mobs were you killing on sight? Or were you just zerging past them?

I'm actually starting to believe that zergers are the cause of lots of lag. My bases for this is the house J quest. The spire with the lich. (I want to call it 'and the dead shall rise. but I"m pretty certain I'm wrong on the name.) But that quest would spawn so many active NPCs that it would literally lag the party badly. Why? To many calls back and forth between all of the NPCs talking. I actually proved this when I jumped into the middle of about 40-50 odd of the skeletons that spawned and started to take them out. At the time I figured I was just going to have fun being suicidel, but after about 20 of them were taken out, we started to actually visually see lag reduction happening. Once the floor was cleared of the undead, we were moving normally again.

Now, one thing I will note, we never did lose connection or were in risk of losing connection under this circumstance.

correct, and you will see the same thing in running to VOD all the teleporting devils and orthons, cause major lag, all the dps calculations the system cant handle when fighting parts 4/5 of the shroud, when zerging coalesence chamber leaving all the scorpians and trash mobs active...and then you have noob fall or player recall for mana, MASSIVE lag with so much trash mobs left alive.....
DDO cant handle all the calculations and teleporting and AI it simply cant handle the mathmatics, and especially at peek playtime ...we always say, enjoy your 9:30 Eastern shroud....

jrp

[Missing_Minds]

correct, and you will see the same thing in running to VOD all the teleporting devils and orthons, cause major lag, all the dps calculations the system cant handle when fighting parts 4/5 of the shroud, when zerging coalesence chamber leaving all the scorpians and trash mobs active...and then you have noob fall or player recall for mana, MASSIVE lag with so much trash mobs left alive.....
DDO cant handle all the calculations and teleporting and AI it simply cant handle the mathmatics, and especially at peek playtime ...we always say, enjoy your 9:30 Eastern shroud....

jrp

Haven't the dev's already admitted that teleportation was causing lag? I can't remember.

However... I wonder if the dps calculation lag.... As a player I know what you are talking about and know it is real. Suddenly all PCs are sinked up swingign at the same target, doing damage at the same time. (and missing/not getting swings also if you really analize it closely) I wonder if that lag is caused by their random number generator (RNG) not able to keep up with the requests... I mean it makes sense really if the generator only runs for X amount of time a frame, it is being called on more than what it can actually handle so either over frames or drops requests. (my guess is drops requests.)
Or even if the RNG is only framed for so much time an instance it still works out for causes of lag...

I mean I never understood why there could be lag in shroud part one, when the only thing alive was a "portal", but this.. this makes sence.

It is either that or one entity is only allowed so much calculation time. Hence why a swarm of kobolds won't lag a low level party, but one portal can.

Another thing I'll note. My hop path to gls.ddo.com has changed. 3 months back (when lag was apparently less) I had an average hop number of about 20-22. It is now an average of 18 hops and with the little I've played recently, the lag has seemed worse.

[Thanimal]

correct, and you will see the same thing in running to VOD all the teleporting devils and orthons, cause major lag, all the dps calculations the system cant handle when fighting parts 4/5 of the shroud, when zerging coalesence chamber leaving all the scorpians and trash mobs active...and then you have noob fall or player recall for mana, MASSIVE lag with so much trash mobs left alive.....
DDO cant handle all the calculations and teleporting and AI it simply cant handle the mathmatics, and especially at peek playtime ...we always say, enjoy your 9:30 Eastern shroud....

jrp

Hmm, the only computation-intensive part I can think of in all of that is pathing. And two things make me suspect this isn't the problem:

1) Throwing more server hardware at this would fix it. I know people accuse Turbine of being cheap a lot, but NOBODY could think they'd make more money in the long-run by turning off 10s of thousands of players (i.e. 100s of thousands of dollars PER MONTH) instead of buying some beefier servers. If the fix were that easy, they'd have done it.

2) Even if they were SO near-sighted that they refused to buy any more hardware, it would be simple to code around this, but just having mobs that are farthest from the players stand in place any time the pathing resources got tight. Again, this would be an easy fix.

So I'm sticking with my hypothesis that the bug is far more subtle and evil. But if I'm wrong, then: Turbine, implement one of the above solutions and let's move on!!

[juniorpfactors]

Hmm, the only computation-intensive part I can think of in all of that is pathing. And two things make me suspect this isn't the problem:

1) Throwing more server hardware at this would fix it. I know people accuse Turbine of being cheap a lot, but NOBODY could think they'd make more money in the long-run by turning off 10s of thousands of players (i.e. 100s of thousands of dollars PER MONTH) instead of buying some beefier servers. If the fix were that easy, they'd have done it.

2) Even if they were SO near-sighted that they refused to buy any more hardware, it would be simple to code around this, but just having mobs that are farthest from the players stand in place any time the pathing resources got tight. Again, this would be an easy fix.

So I'm sticking with my hypothesis that the bug is far more subtle and evil. But if I'm wrong, then: Turbine, implement one of the above solutions and let's move on!!

I brought up the devils and orthons 12 months ago, and "forum boys" called me crazy then too.... and now we know it to be true as they admitted as much

so I go with my hypothesis

jrp

[Thanimal]

I brought up the devils and orthons 12 months ago, and "forum boys" called me crazy then too.... and now we know it to be true as they admitted as much

Can you point me to that? If they KNOW what the bug is and haven't fixed it, that casts this whole thing in a VERY different (VERY ugly) light. But I reserve judgement until I read more details.

[Stormanne]

I'm not entirely sure if this makes any difference or is of any importance.

Last night, we did a guild run on Sarlona, Hound of Xoriat. No lag as the party gathered in the market. As soon as we zoned into the Subterranean, we started getting hit with lag. It wasn't random either. We did a little experiment while we were buffing for the run. The lag was persistent on an eight second interval. Every eight seconds we would get a two to three second lag spike. And it wasn't localized to one or two people, but the entire group. And it's not like we are all neighbors, this group was spread out from Canada to California.

Being on so regular an interval makes it almost seem deliberate (I know that no company would deliberately have lag as part of their product), just seems that that would make the lag easier to localize for the company responsible (either Turbine or their service provider).

[smatt]

There's so many differing possible causes for lag, some of which likely are occurring simultaneously, and even members of the same party could be experiencing lag for differing reasons. I'm sure that there are issues within the game engine itself that cause lag as well as the location of the game servers. As well as client side lag, individual ISP lag, internet hub lag, on and on and on. Also weather in certain parts to he country seem to contribute to lag. Primarily though, the worst of the overall lag seems to be related to game engine performance, in the high end content, within raid groups, with a lot of actions happening. It is likely that the game engine simply can't handle the huge amount of computations. Also, likely is that very busy instances IE: Shroud prime time with multiple groups running it at the same time play a roll, as the game engine might possible have a resource allocation limit in place.

In other words there are a lot of reasons for various kids of lag, I think it's funny that it's always Turbines fault. Yes, I think it's part of the cause, but not likely nearly all of it....

For us as players the best we can do is, if we want to lessen our lag on our side, have good well running systems with far more than the minimum requirements for the game. Have a good clean down AND uplink connection speeds with our individual ISP's. Make sure personal system resources are clean, not running extensive software,t hat places a higher demand on available RAM, clean-out internet temp files to remove those pesky tracking cookies that cause problems with uplink speeds.

Other than that, allow e can hope is that Turbine is working on further problem solving with the lag that is directly related to the game engine, and hardware on their side.

[smatt]

Can you point me to that? If they KNOW what the bug is and haven't fixed it, that casts this whole thing in a VERY different (VERY ugly) light. But I reserve judgement until I read more details.

It had to do with the way the the devils and orthons were teleporting... .. I seem to remember them saying it was hwo the mobs were deciding where to teleport to and that it was causing the engine to bog down.

So what is the point of all of this. If it's to work on individuals lag, and how to offer up some advice as to how people can mitigate it on their side, that's great. If it's to figure out what's going on Turbines side, well you will NEVER get enough information, or data to come to any reasonable conclusion IMO. Not meaning to swing a bat at you at all. We all want less lag, but Turbine is a corporation that guards things rather carefully, and rightfully so. So for you to obtain the information required to even have any real idea of how to solve the problems if it is infact something that can be remedied, well it's not going to happen....

[ArkoHighStar]

It really is unclear whether the servers were outsourced, we know the following, Turbine bought new hardware for DDO last fall, they also signed a lease for new office space. They also moved the servers into a new data center, it has never been stated that this datacenter is off site, just that the servers were moved to a new location. It is entirely possible that the datacenter is a new one inside Turbine's offices. We also do not know whether they have changed network providers, all we know is recently their vendors have been doing maintainence on their systems and that it could have caused an outage. All the recent outages had db rollbacks, and were brought down quickly to prevent further corruption, which says the staibility problems are related to db or storage issues.
The lag is more difficult to determine, as it could be internal network configuration, server configuration, thier AT&T lines to the internet (main pipes), could be code isues causing information delays.
The fact that all the server activity moves on while we get a huge lag spike tells me its is more a network issue either at the NIC card out or somewhere in between that and our PC. Given that they can easily monitor AT&T line quality my guess is somewhere in the datacenter network infratstructure things are getting bogged down both ways, the pipe is either overloaded with data because it is not wide/fast enough, or something is artificially making a big enough pipe act much slower than it should.