Reproducible Lag

[Jiirix]

I came across some situations in game that trigger heavy lag 100%:

In the Belly of the Beast: If you did the “kill Voltane” optional Rhitez will join you in the pit fight. Every time the Drow Weapon Masters start attacking Rhitez the game gets super laggy. When Rhitez is dead and the group fights the Drow Weapon Masters alone the lag disappears. The problem seems to be triggered by the Rhitez vs. hostile mobs fight, perhaps an issue with the aggro checks?

The Black Loch: Entering the cave with the pirate ships makes the game laggy, leaving the cave ends the lag.

The Shroud part 3: Even if the rest of the raid goes smoth this part is lagging.

[cru121]

Isn't the Black Loch thing a graphics issue? IIRC these forums had some fix (turn down some graphic option) that fixes the problem. My ad hoc solution is avoid looking at the ship, instead look at the ground or cave wall.

[EllisDee37]

Isn't the Black Loch thing a graphics issue? IIRC these forums had some fix (turn down some graphic option) that fixes the problem. My ad hoc solution is avoid looking at the ship, instead look at the ground or cave wall.Turn off post processing to mitigate the effect. Instead of framerate dropping to single digits it drops to the 30s or so. Both are big drops, but 30fps is much more playable than 7fps. Either way, the problem is definitely not fixed.

It's the water that causes the problem. Just keep looking up so that no water is being drawn.

[Full_Bleed]

Turn off post processing to mitigate the effect. Instead of framerate dropping to single digits it drops to the 30s or so. Both are big drops, but 30fps is much more playable than 7fps. Either way, the problem is definitely not fixed.

It's the water that causes the problem. Just keep looking up so that no water is being drawn.
The bigger picture here is that this is a LONG standing, highly reproducible issue that may be affecting other areas of the game to varying degrees. This is a proverbial smoking gun and nothing has been done about. In fact, I'd hazard a guess that it has been completely ignored (it's been around forever)... let alone use the knowledge gained from addressing it to assist with other parts of the game it might be affecting. It's not enough to say, "It's the water". There is a massive amount of water in the game that does not do this... many places, in fact, that appear to be far more complex. So there is certainly more to that particular issue than "the water". There is some interaction/design/application they have not properly identified... and the chances of it only affecting that one spot in the game is pretty slim.

Now, I admit, this probably has little (or nothing) to do the new data-center lag... but it may be key to why certain raids and mechanics consistently have issues that players simply try to "play through" on a regular basis.

Personally, I think too many darts have been thrown at the lag board in the past that probably did little to nothing to address various types of lag (i.e. remember the "exploding" shadows?) But here we have certainty. That room will frame-rate lag even the most robust computers. They should figure out exactly what's doing it, fix it, and apply that knowledge to other areas of the game.

[Gauthaag]

I have similar isue as i n Black loch in Tharask arena when i try to look into arena from the sides where those observers are - on way to drunken ogres. And i wouldnt blame my workstation for being lowend machine.

[Jiirix]

Now, I admit, this probably has little (or nothing) to do the new data-center lag... but it may be key to why certain raids and mechanics consistently have issues that players simply try to "play through" on a regular basis.



To know what lag is NOT caused by the data center is important as well. I know there are there are differnt things that cause lag but I haven't seen a "lag by catargory and what causes it" post (by a Dev idealy..)

What I can rule out is the PC I am running DDO on. There is lag that effects everyone, lag that effect only certain playersa and I know that issues of other players can have impact on my own performance. (This kind of lag is gone in the very moment the player that causes it quits the game.)

[Flavilandile]

The Black Loch: Entering the cave with the pirate ships makes the game laggy, leaving the cave ends the lag.


This one is special.
If you type ctrl-F and look at your framerate before during and after you'll understand that it's a purely client lag ( graphic lag) and that tinkering with your
settings ( like disabling video post processing ) will do a lot to limit it.
As far as I know everybody has this specific lag spot except the few that have $500+ graphic cards in SLI.

All the other cases of lag are new and came with the datacenter move.

[biggin]

Where I am seeing the biggest lag hits are when mobs spawn rather than are already there. For example, DoJ where mobs are constantly spawning and teleporting is brutal (we just ran an 8 man, almost wiped twice at last fight). Raids where mobs are already in play, like VON5, while bad aren't nearly as bad, at least IMO.

[Gauthaag]

Where I am seeing the biggest lag hits are when mobs spawn rather than are already there. For example, DoJ where mobs are constantly spawning and teleporting is brutal (we just ran an 8 man, almost wiped twice at last fight). Raids where mobs are already in play, like VON5, while bad aren't nearly as bad, at least IMO.

Exactly - in places like the very first group of mobs in Grim and Barret - when door opens and corridor of abishais spawn.

Also quite apparent in places where mobs seem to be scripted to be inactive till players approach - and then suddenly all of their AIs wake and tries to figure agro.

[lugoman]

Shroud part 2 when you kill all 4 red names and the barrier drops around the crystal the game stops. This happens in heroic and legendary and never happened to me before the move.

[SurlyYuri]

All the other cases of lag are new and came with the datacenter move.


Um, yeah, I'm gonna have to go ahead and disagree with you there.

[Belgarath22]

Other thing easily reproducible, in raids:

Every time you get guild renown.. lag like crazy

I hope this helps..


So this morning we did optimization work to clean up network pathing problems and open up new network paths that should have alleviate, if not solve, lag related to network pathing. Thank you for all the trace routes. At this point we will be watching tonight to see who is improved, who isn't, and whether or not there is still lag that is related to server performance rather than network pathing.

Sev~

[mudfud]

To know what lag is NOT caused by the data center is important as well. I know there are there are differnt things that cause lag but I haven't seen a "lag by catargory and what causes it" post (by a Dev idealy..)

What I can rule out is the PC I am running DDO on. There is lag that effects everyone, lag that effect only certain playersa and I know that issues of other players can have impact on my own performance. (This kind of lag is gone in the very moment the player that causes it quits the game.)

But that's the problem. Since lag was here before we don't know whether it's caused by the datacenter move exclusively, or just the same lag even worse after the move. Instead of "hotfixing" on our end the black loch lag, maybe that black loch lag is now the cause of all lag. Or maybe the cause of all lag is shroud part 2 after you beat all the named guys and barrier drops. Most of the "lag" issues have been taken with a grain of salt over the years and most not even looked at.
Who would have thought just using a "riposte" weapon would cause game breaking lag, but it did a few updates ago. Who would have thought just doing baudry chain would cause game breaking lag, but a few years ago it did. Maybe now the lag problem is being caused whenever anybody on the server uses a scroll. We just don't know. And instead of solutions as turning down the post processing or whatever else in graphics or not looking at the water as fixes, they need to be actually fixed.

[Flavilandile]

Um, yeah, I'm gonna have to go ahead and disagree with you there.

That's just what I see from my end.
Before the move I almost never lagged, and when I lagged everybody on the server ( according to guild chat, party chat and teamspeak ) was lagging.
After the move lag is everywhere.


To know what lag is NOT caused by the data center is important as well. I know there are there are differnt things that cause lag but I haven't seen a "lag by catargory and what causes it" post (by a Dev idealy..)


I know I'm not a dev, and I remember already posting it... Anyway :

Client side lag :
- Graphic Card generated lag ( sub par graphic card especially the integrated stuff )
- low end computer lag ( people that try to play with a computer that is not powerful enough )
- Too high graphic setting lag ( people that try to play with everything at very high, when the compute/grpahic card can't do it )
- HDD lag ( 4800/5400rpm HDD are just not good enough )
- PCI-E lag ( especially 1.0/1.1 ) ( despite what some might say there's a bottleneck here, and it can cause lag )
- software induced lag ( running BOINC/Seti@home ( or any other project ) in the background is generating lag )
- Local LAN Lag ( streaming stuff from one computer to the other in your LAN )
- DLS/Cable line lag ( we are now leaving the client side )

Network side :
- ISP lag ( the last mile can have issues )
- Carrier Lag ( backbone routers can have overloaded interfaces, usually there's way to deal with that, but in some cases, misconfiguration results in dropped packets and stuff )
- Peering lag ( when you jump from one carrier to another, the quality of service rules change, it can cause lag )
- Datacenter lag ( in the old datacenter it was seen when everybody in game froze... same as in the rest of the network an overloaded interface on a router can be the source )

Turbine side :
- Frontend network lag ( stuff that happens in the frontend network... )
- Server lag ( lag caused by the actions, or by all the stuff that needs to be calculated )
- Backend network lag ( this one is hard to see for us, sluggish interface opening for all the UIs can be it )
- Database lag ( typical example : all the cannith crafting recipe interface, the quest list pannel... this one could be easy to solve if they put a DBA on it )

Note that it's not a complete list... I typed fast and I more than probably missed a few more sources.

[Mindos]

Note that it's not a complete list... I typed fast and I more than probably missed a few more sources.

I wonder how the cannith deconstruction is going? I seem to remember all kinds of lag and wait times just trying to deconn an inv full of junk. Did it improve with the datacenter move?

Or did everyone hoard there junk, and everyone is deconstructing all at once since they made the new random loot work correctly? Is this the secret source of the lag? I know turbine was planning on changing the Cannith Crafting system already, so maybe it's possible someone put a little code in early that shouldn't be there? So this problem could not be related to the move, but instead is just a build up of Cannith deconstruction database lag?

Time to shutter the hall again! :)

[mudfud]

I wonder how the cannith deconstruction is going? I seem to remember all kinds of lag and wait times just trying to deconn an inv full of junk. Did it improve with the datacenter move?

Or did everyone hoard there junk, and everyone is deconstructing all at once since they made the new random loot work correctly? Is this the secret source of the lag? I know turbine was planning on changing the Cannith Crafting system already, so maybe it's possible someone put a little code in early that shouldn't be there? So this problem could not be related to the move, but instead is just a build up of Cannith deconstruction database lag?

Time to shutter the hall again! :)

While they did make new loot deconstructable, it's still broke. Most times you can only decon 1 item then you have to close the station, and open it again, since it refuses to do more. And last I checked the "newer" stuff was also not giving plat when you deconstruct also, that was another factor people did it, getting essences and plat. And lastly, I haven't really seen it but I haven't looked either, but people were reportedly getting wrong essences from what they selected also.

I'm still going with every time somebody uses a scroll on the server that is what causes the lag for us all. So remember no scroll using!

[Jiirix]

Thanks Flav.


That's just what I see from my end.
Before the move I almost never lagged, and when I lagged everybody on the server ( according to guild chat, party chat and teamspeak ) was lagging.
After the move lag is everywhere.



I know I'm not a dev, and I remember already posting it... Anyway :

Client side lag :
- Graphic Card generated lag ( sub par graphic card especially the integrated stuff )
- low end computer lag ( people that try to play with a computer that is not powerful enough )
- Too high graphic setting lag ( people that try to play with everything at very high, when the compute/grpahic card can't do it )
- HDD lag ( 4800/5400rpm HDD are just not good enough )
- PCI-E lag ( especially 1.0/1.1 ) ( despite what some might say there's a bottleneck here, and it can cause lag )
- software induced lag ( running BOINC/Seti@home ( or any other project ) in the background is generating lag )
- Local LAN Lag ( streaming stuff from one computer to the other in your LAN )
- DLS/Cable line lag ( we are now leaving the client side )

Network side :
- ISP lag ( the last mile can have issues )
- Carrier Lag ( backbone routers can have overloaded interfaces, usually there's way to deal with that, but in some cases, misconfiguration results in dropped packets and stuff )
- Peering lag ( when you jump from one carrier to another, the quality of service rules change, it can cause lag )
- Datacenter lag ( in the old datacenter it was seen when everybody in game froze... same as in the rest of the network an overloaded interface on a router can be the source )

Turbine side :
- Frontend network lag ( stuff that happens in the frontend network... )
- Server lag ( lag caused by the actions, or by all the stuff that needs to be calculated )
- Backend network lag ( this one is hard to see for us, sluggish interface opening for all the UIs can be it )
- Database lag ( typical example : all the cannith crafting recipe interface, the quest list pannel... this one could be easy to solve if they put a DBA on it )

Note that it's not a complete list... I typed fast and I more than probably missed a few more sources.

[bsquishwizzy]

The Black Loch: Entering the cave with the pirate ships makes the game laggy, leaving the cave ends the lag.



I can tell you with a certain degree of confidence that this is because of graphics. You can turn down a few settings (I think it is resolution or dear distance) and this gets minimized.

[Grace_ana]

Are you sure that part of the Black Loch cave lag isn't due to the archers? Whenever there are a bunch of mob archers there is a tendency to have lag. You see it at the end of Friends in Low Places too, as well as some other spots in the game.

[Belgarath22]

I can tell you with a certain degree of confidence that this is because of graphics. You can turn down a few settings (I think it is resolution or dear distance) and this gets minimized.

Yeah, this is old lag, almost every time I lagged in that cave (big cave)

[redoubt]

Are you sure that part of the Black Loch cave lag isn't due to the archers? Whenever there are a bunch of mob archers there is a tendency to have lag. You see it at the end of Friends in Low Places too, as well as some other spots in the game.

I can tell you with 99% confidence that the black loch specific lag is the water related. When I run that quest I pull my camera down so it is near the water and I am looking up at my character and thus not seeing any water. As long as I keep the water off of my screen I do not lag. If I goof and move the camera so the water is rendered the game slows.

I don't know what is special about that water. I can look at the water everywhere else and not have any issue, but there is something in THAT water...

p.s. I also see the affect you mention during Friends in Low Places, but I do not believe the Black Loch issue is caused by the archers.

[Flavilandile]

I wonder how the cannith deconstruction is going? I seem to remember all kinds of lag and wait times just trying to deconn an inv full of junk. Did it improve with the datacenter move?

From my point of view, not much change in deconstructing...
But for me, the UI lag in Cannith is caused by a broken index when the game looks up the recipe in the database.

I can't tell for crafting... because crafting is FUBAR for the time being.


Are you sure that part of the Black Loch cave lag isn't due to the archers? Whenever there are a bunch of mob archers there is a tendency to have lag. You see it at the end of Friends in Low Places too, as well as some other spots in the game.

It's definitely the water/light interraction.
Turn off video post processing and you multiply by 10 your frame rate... tinker a bit more with other parameters ( surface reflection, glow mapping, ... ) and you can see the FPS improving.
If it was the archers then changing the graphical parameters woudln't have any impact.

[Grace_ana]

It's definitely the water/light interraction.
Turn off video post processing and you multiply by 10 your frame rate... tinker a bit more with other parameters ( surface reflection, glow mapping, ... ) and you can see the FPS improving.
If it was the archers then changing the graphical parameters woudln't have any impact.

I've had video post processing off; I don't see a difference in my performance between the two in Black Loch. I do see the difference in other places, though, which is why I eventually turned it off. I'll mess with some other settings too, but for me whenever the archers go down my lag goes away in there. YMMV