The latest on lag and game performance improvements

[Belgarath22]

Not sure if this thread still matters..

For me, every day I play this game is more depressing.. good luck

[Eryhn]

to put above post into context as we shared the pain:

Argonessen
Sunday 9PM-Monday 1AM Eastern
LN TS, 10 people: constant renown lag, many 3-5 sec freezes. bigger lockups on larger groups. completed but annoying

EN Wyrm: 10 people, constant renown lag, many 3-5 sec freezes, multiple R/G deaths due to lag and or spellblock charge drain bug, much stronger freezes on rednamed at boss, 3 (!) people rubberbanded of the platform after succesfully trying to grab the ledge when dropping the kulls, completed, but yet more annoying

EN Peaks: 9 people, constant every single darn renown kill LAGs 2-5 seconds, keeping alert below yellow without locking instance up friggin PAIN on normal, several 5-10 sec float around in the air freezes, several lag death due to heals not landing. died tanking the unbuffed EN dragon when some trash wondered in - in unyielding maxed, lvl30, 1150+ HP, 150+ PRR, 80+ MRR, 70-80 saves, blured, displaced, fire shield cold up, standing against the tide, blocking twf... rly?! completed, but frustrating

EH FOT: 9 people, constant renown lag, many 3-5 sec freezes, several longer freezes, several lag induced deaths, completed. not quite as bad as peaks but no fun much.

tried 7man EN DOJ with 2 warlocks lvl28 and 30, 2 melee lvl30 1 arti lvl28, 1 rogue mech lvl30 1 necro wiz lvl30 ....had very decent dps and killed execs in like 10 sec on the few occasions we could actually move decently ... lost 3 archons within 5min, 4th within next 2, fail on reinforcements after 8 or 9 mins. 3-5 sec complete lockup on EVERY single kill. HUGE 10-12 second lockups on every spawn of 5 reinforcements + trash. un fing playable.

tonights conditions were IN NO WAY better than before your last fix. this pertains especially to raids but i likewise had issues some hours before raids running a lvl3 through solo misery's peak on korthos....

needless to say raid issues were shared by whole party. you said on twitch US based players MAY SOMETIMES also experience issues when going through faulty routing. Raid experience would suggest that over the last weeks, miracolously, all US/Canada players in our static raid group unluckily and coincidentally just so happen to ALL share the bad routing path for bi-weekly raids. yeeeeaaah right. stop hiding behind that and fix your VMs, upgrade your hardware, AND FOR CHRIS SAKE LOOK AT THE BLOODY GUILD RENOWN CODE

otherwise do us all a favor and pull the plug for good

[Eryhn]

to be fair, the otherwise overall somewhat frustrating raid night ended on a happy and satisfying note after the remainder of our group decided on "to hell with the lag, we gonna beat this ****". successfully completed an EN DOJ in a party of 5 and had great fun with it:





here's the downside though:
how can it be that our raid party of 7 lags out terribly but if i step into DOJ solo, kill a total of 40 mobs, recieve renown 28 times and run around with a pack of 20 orange nameds on me i get as good as NO lag (slightly sluggish once i built up a trail of orange) ?! and that our party of 5 got a lag spike on every single renown grant, only, that with 5 in party the spikes would last 2 sec max, or maybe 3 or 4 on a bad one, and you could still properly heal, properly hit with spells, and attacks and tactics landed, thus demonstrating absolutely clearly the correlation between party size and severity of lag? how can that please be routing related? I am a European player, according to turbine's explanation my routing should often make me miserable but not my whole party? yet, in DOJ solo it is almost flawless, in DOJ 5man noticeably laggy but manageable, and in 7 man party it goes to shambles?

How is that NOT your server, your VM, simply not being able to handle the load?

Now, I dont know if this is renown or like, whatever. BUT clearly your server is calculating *something* when mobs die, and this something bogs down the WHOLE party, and more extremely so the larger the party is. and if there is anything the spikes seem to synch with, yes, it is guild renown. when 1 person gets renown, its usually no issue, in a party of 4-6 it is noticeable, and hampering, though one can deal, in a 7+ raid group, it more often than not BREAKS the game, simple as that.

WHY can we NOT turn of guild renown for a week and see what happens? WHY is there no answer on this issue that gets mentioned by so many, time and again?

P.S.yeah sure, i DO get solo quest lag too, frequently atm, and maybe sometimes that is that mentioned routing issue. or maybe it is a party bogging down the VM that hosts BOTH our instances? ...

[Faltout]

how can it be that our raid party of 7 lags out terribly but if i step into DOJ solo, kill a total of 40 mobs, recieve renown 28 times and run around with a pack of 20 orange nameds on me i get as good as NO lag (slightly sluggish once i built up a trail of orange) ?! and that our party of 5 got a lag spike on every single renown grant, only, that with 5 in party the spikes would last 2 sec max, or maybe 3 or 4 on a bad one, and you could still properly heal, properly hit with spells, and attacks and tactics landed, thus demonstrating absolutely clearly the correlation between party size and severity of lag? how can that please be routing related? I am a European player, according to turbine's explanation my routing should often make me miserable but not my whole party? yet, in DOJ solo it is almost flawless, in DOJ 5man noticeably laggy but manageable, and in 7 man party it goes to shambles?

Each party member gets renown at the same time. It could be anything that has trouble being calculated/stored in the DB many times within a short period.

[Eryhn]

Each party member gets renown at the same time. It could be anything that has trouble being calculated/stored in the DB many times within a short period.

yes, I'm well aware that while it coincides with renown from players view it does not necessarily HAVE to be the actual renown, it might just as well be AIs recalculating each time one of their side dies i.e., that's why i wrote "renown, or idk, whatever". But there is very evidently some sort of frontend/backend interaction that locks stuff that scales exponentialy on party size, and it is beyond me how Sev can go and say "we are confident we got the server side issues down, and expect things to be back to pre-server move state when we adjusted the routing issues" NO; you don't! Look into this and if you have no clue what causes it then blimey, maybe dedicated metal that doesn't further increase the effect like the VMs seem to do would seem to be the only solution ...

[Faltout]

yes, I'm well aware that while it coincides with renown from players view it does not necessarily HAVE to be the actual renown, it might just as well be AIs recalculating each time one of their side dies i.e., that's why i wrote "renown, or idk, whatever". But there is very evidently some sort of frontend/backend interaction that locks stuff that scales exponentialy on party size, and it is beyond me how Sev can go and say "we are confident we got the server side issues down, and expect things to be back to pre-server move state when we adjusted the routing issues" NO; you don't! Look into this and if you have no clue what causes it then blimey, maybe dedicated metal that doesn't further increase the effect like the VMs seem to do would seem to be the only solution ...

My response meant to explain why you're seeing the problem with renown in larger parties and not smaller parties. In larger parties, renown is given to all players at once. If the database has trouble registering those events simultaneously, it will lag. Or it may not be the database, but something else that has trouble doing concurrent calculations. Either way, it has to do with the event of getting guild renown for many players at the same time.

[Bolo_Grubb]

1. Server: Sarlona
2. Approximate time: 1030pm Sunday 5/9/16
3. 1 rogue, 2 clerics, 2 warlock, 2 ranger, 2 druid, paladin, bard, wizard
4. 12
5. Epic Normal Shroud, due to current lag, any higher difficulty is not possible
6. General experience: extreme lag (freezing and stuttering) at every fight/spawn.

[Krelar]

Not a lag report but odd behavior.

Argonessen
Heroic quests and explorer areas lvl 13+.

Lots of mobs are having weird pathing behavior.

They really liked to jump off ledges to their death. (I especially enjoyed the archers that jumped into the mine field in undermine)

I've seen this before since the server move but it had stopped for several weeks, now it's back again.

I've also had mobs that randomly decide to run away from me. This even happens when solo so I know they're not agroed on anything else.

[Cordovan]

This past weekend had an above-normal amount of poor game performance, and is something we will be looking into. We are continuing to look into the various reported causes of lag, including renown gain. We would continue to caution that Defiler of the Just has issues outside of server-client latency, and is not a good test case for lag. We fully understand people's frustration in Defiler of the Just, and will be considering some options for that particular raid in the future.

[AnEvenNewerNoob]

This past weekend had an above-normal amount of poor game performance, and is something we will be looking into. We are continuing to look into the various reported causes of lag, including renown gain. We would continue to caution that Defiler of the Just has issues outside of server-client latency, and is not a good test case for lag. We fully understand people's frustration in Defiler of the Just, and will be considering some options for that particular raid in the future.

Glad to hear this. Hopefully those options include making the failure conditions optionals that reward a nice xp chunk and extra chests.

Hard to defend Archons when you are frozen in place.

Of course fixing the lag in there would be even better.......but I don't want to ask for the moon ya know?

[Eryhn]

This past weekend had an above-normal amount of poor game performance, and is something we will be looking into. We are continuing to look into the various reported causes of lag, including renown gain. We would continue to caution that Defiler of the Just has issues outside of server-client latency, and is not a good test case for lag. We fully understand people's frustration in Defiler of the Just, and will be considering some options for that particular raid in the future.

appreciated you are replying and acknwledging to look into the renown issue.

as for the notion of defiler being the sad black sheep, I would advise against that approach. rather, you should be seeing defiler as an exemplary case that due to its particular design highlights general, underlying issues. the characteristics of lag spikes are the same on other raids and also the same on full 6man party EE quests. there is a reason fire on thunder peaks with its amount of trash, respawn mechanic, and amount of orange named granting renown is the 2nd worst offender, closely followed by red named mobs in shadow wyrm.

AI/renown/pathing/combat effects on mobs & frontend/backend interaction peaking the VM on RAM or CPU allocation, or on database access - one of those or several at once. defiler does not have worse issues, it just SHOWS issues more clearly.

yes, there are issues with soloing sometimes, and with smaller parties, and with heroics that do not fit said pattern, and these may be either the routing or the above core issues bleeding out via the VM when instances share same machine, but the above seems to be the crucial thing rly ...

[Gratch]

This past weekend had an above-normal amount of poor game performance, and is something we will be looking into.

Was this due to some sort of Operations squeezing of servers or such? The previous week had started to make DDO feel like notSlog. Then this last weekend, especially Friday evening turned into consistent bursts of no-move 5-10s server lag. Did Turbine try to squeeze some hardware resource down this weekend or is this just some code issues on the new virtual machine platform killing off some resource after the servers were up for a week?

[Psiandron]

This past weekend had an above-normal amount of poor game performance, and is something we will be looking into. We are continuing to look into the various reported causes of lag, including renown gain. We would continue to caution that Defiler of the Just has issues outside of server-client latency, and is not a good test case for lag. We fully understand people's frustration in Defiler of the Just, and will be considering some options for that particular raid in the future.

With all due respect, "looking into it" is not going to cut it in my opinion. I understand your (Turbine's) unwillingness to expend resources and take action without a clear idea of what is going on; however, I really don't see you as having that luxury currently. Too, removing renown gain from the game at least temporarily is not likely to incurr a great deal of wrath from the player base, especially as it would likely be seen as a tangible effort to impact performance problems. My greatest fear is that it would not be as easily done as it would appear from the outside, ala the 'The' screen.

A second option, not exclusive from the first, is that in my opinion and those of many others I play with regularly, the game performance seems markedly improved after a server shut-down. As the time from the shut-down increases, so do performance issues. Given the game's historical problems with garbage collection and as I understand it, a move to virtual servers, it would make sense to me to perform regular (perhaps weekly) shutdowns of each server for the foreseeable, to help alleviate the issues the player-base are having. I understand that server shutdowns are not the easiest thing to do, given that servers are specificly designed to not be shut-down; yet it is possible and would I believe be helpful in this particular case.

Regardless what Turbine does, I sincerely hope that it is able to take action soon which at least partially remedy the current issues.

Best of luck,
Psi

[Ghwyn]

Whatever you do, don't kill DOJ by giving out loot chests like you did for CITW.

[Duana]

Yeah we had a bit o lag this weekend. I was sad. Lots of spells not going off.

One thing I did find funny, really wish I had been running fraps or something to catch it. Elminster lagged in the beyond the rift and fell off the path right after he one-shotted the bebilith. We got to go through the entire narrative at the beginning sans his presence. I have never seen that before. Oh and Elminster does NOT have feather fall!

For the first time in a long time, I was two boxing. It was different. A couple of things I noticed:

1. Char A ran would run from point a to point b with no lag. Char B ran from point a to point b with no lag. But when Char B was running to point B and I was watching it on Char A's screen it looked like Char B was laggin like a *****. Lots of snap back lag on Char A's screen that char b did not experience. And when Char A was running there was no snap back lag.

2. Running Char B up while Char A was using a shrine. When Char A stood up from the shrine, Char B lagged. This may be coinkydink, but the only time that happened was when Char A had buffs from Char B going into the shrine. It did not happen every time Char A shrined with Char B buffs, but it never happened when Char did not have Char B buffs. Again, maybe just huge unrelated event convergence.

3. Pathing, while not back to normal had improved. But this weekend mobs were running off cliffs in a loop, getting caught in corners, etc.


Something else I noticed, when I was doing stuff in ventrilo (not talking, but starting, stopping the client, logging in/out of admin) while DDO was swapping instances (loading screens) the game would hang. Closing vent enabled the game to move on. It was a random thing and I never got it reproduce on demand, but it were there.

Did guild reknown get reduced this weekend? It seemed like it from what I ran.

[alancarp]

On Thelanis last night (10-11:15pm EDT specifically), had a very difficult time with major lag spikes (a few seconds on, a few seconds off) while soloing Fleshmakers/EH with my Pally. Managed to finish, but took at least twice at long as usual.

From about 9-10pm, item deconstruction/crafting was next to impossible. Very lengthy delays in doing each item.

Noted earlier in the day that running through public areas was very choppy... similar to the problems observed right after the new data center kicked in, but through roughly the 6-7pm time frame, the Orchard quests were very playable... few if any issues then.

Note: I swapped characters several times during the afternoon/evening hours... probably 4 times each among 2 toons. Dunno if that would matter, but figured I oughta mention it.

[dunklezhan]

Been getting bad rubber banding in GH slayer of all places over the weekend. What was really weird was that I could run around areas of rubber banding. If it continues I'll bug it.

[Thar]

This past weekend had an above-normal amount of poor game performance, and is something we will be looking into. We are continuing to look into the various reported causes of lag, including renown gain. We would continue to caution that Defiler of the Just has issues outside of server-client latency, and is not a good test case for lag. We fully understand people's frustration in Defiler of the Just, and will be considering some options for that particular raid in the future.

teleporting = lag... always has.

[DagazUlf]

We are continuing to look into the various reported causes of lag, including renown gain.

I had not understood what "renown lag" meant until this past week. Essentially, as soon as the renown award was displayed in the chat window, the entire party would experience lag for a brief amount of time. It was *very* consistent throughout the week and through the weekend. This was primarily in Epic Elite quests as my guildies and I were working on True Elite Saga completions.

[Faltout]

This past weekend had an above-normal amount of poor game performance.

Wait wait wait. So, what IS the normal amount of poor game performance? I've asked the exact same question before. All this work that you've been doing has been to bring the servers back to pre-data center move conditions? Because there was plenty of lag back then as well.