Post-U9 lag

[Codeshaper]

Hi all,

We are aware of the increased lag that has cropped up post-U9 and are looking into it. There is enough anecdotal evidence to point to a few candidates for investigation, but more info could be useful, hence this thread.

What I'd like, for now, are the world you play on, the quest, the party size and composition, the time of day (with timezone, or converted to either GMT or EDT), details about what was happening when the lag hit (for example, "all the casters had just called up ice storms", or "we got up from the shrine and couldn't move"), and any other info that seems relevant. If you were chatting with other players in-game and they did or didn't see lag at the same time, that could also be useful.

Please don't post "I'm seeing horrible lag! Fix it!" or the like. As satisfying as it might be to rant and vent, it won't help us track down the problem.

Thanks!

[Cauthey]

You may have already seen this, but there was a post in the Orien section called Orien Lag that was started right around when we started experiencing bad lag.

I guess I became most aware of bad rubberbanding on Friday, May 13th (on Orien). It continued throughout the weekend. It seemed to come off and on in severe spikes. Whenever it occured, I could either confirm it with other people in general chat (Necropolis, The Orchard, Marketplace, Zawabi's Refuge, Demon Sands) or within my party.

Saturday, Metalbone played Gianthold Tor, VoN, and Demon Queen pre-raid and raid. In all quests, lag spikes were evident, but not so horrible as to make the quests unplayable.

Through the weekend and since, I've experienced a lot of bank related lag. The bank was so bad at one point that when it did finally pop open, it was far away from the bank, and instead right in front of the Orien Lightning Post in the Crafting Hall. I wish I had taken a screenshot of it (the bank was not operable at that distance despite it being open and browsable).

Not sure if it's related, but the hard-crashes that I have experienced (now that I think about it) have happened at the bank in the Crafting Hall.

Hope that helps! And thank you thank you THANK YOU for acknowledging this and looking into correcting it!

[Chai]

Thelanis AND Sarlona both exhibit these same symptoms in the same quests.

Quests: Shadow Crypt in Necropolis & Ritual Sacrifice in the Vale

Symptoms: Character will stop moving for a second or so every 7 seconds or so. Sometimes rubberbanding happens where the character bounces back to where it was a few seconds ago.

-and-

Thelanis (havent tried on Sarlona as of yet)

Quests: ADQ1 and 2 - preraid and raid. Monk, cleric, wizard, FvS. one time on high levels, and a full raid another time on my ranger where we had 2 healers, 2 casters, bard, and rest various melee.

Symptoms: Very similar to old school Shroud, where it appears I am the only one attacking opponent and everyone else appears to be standing around it. I confirmed that everyone else was experiencing the same symptoms at the same times. Seemed to happen when melee would surround and attack a mob. Also, damage numbers, healing numbers etc kept scrolling for a LONG period of time after the completion happened.

[Cam_Neely]

Orien
Typically I have no lag besides the 'loot lag' in shroud ect.

Monday About 11 EST to 1 EST
Ran a few Epics in Red Fens (6 man party) and did not have any issues myself, although guild members in the party did complain about more then normal, although nothing killer. Ran weapon shipment solo twice with no issues in the quest, but Client shut down when I entered Amarath. This happens every now and then but typically only in GH. Logging back in, I noticed huge rubber banding, and it sounded like my computer was working overtime. Left the computer alone for 5 min and came back and ran weapon shipment.

Tuesday
11PM EST: Ran shroud, 12 man group, I had a little lag, while other members of the party complained about massive lag in part 1. Cleared up a bit, with a few spikes in part II and some major issues in part five. 12:30AM EST Ran VoD. First that I noticed major lag issues. 12 man party, very choppy in the sub. Few members complained about bad lag. Issues continued in VoD, and ended up TPK in the raid, many said due to the lag, but I think that was an excuse that masked other problems we had.

Wednesday
Duoed RL, epic hangover dropped, no lag

I'm seeing horrible lag! Fix it!!

[stainer]

On the 17th at about 7:30 pm Central time. VoD 12 people. 2 Arcanes, 1 Cleric, 9 melee. Things were ok when we ported in. About half way through the Orthons lag hit. The entire party was impacted. It improved some after the first wave of devils, but was intermittent throughout the quest. Movement was as if you were chained, and I wasn't. Mass cures, recons landing late. DM Text, voice very delayed. I got the ending DM voice in the marketplace after completion.

This occurred on Khyber and I was on my character Springroll.

[Moltier]

eDQ2 - while buffing, while fighting with the first mobs, later against the queen.
ToD - same, everywhere
VoN - happy fighting, 12 dings, then few seconds later the dragon's fire breath appeared
Guild ship, solo :P - guild buffs appeared 2-4 seconds later. Also it was hard to move
Market - teleporting people everywhere, meanwhile my guildies lagged elsewhere
Sub T. - heals late ~ 5 sec
epic Last Stand, short manning - spells land 50% of the time.

Just a few from my own experience. All of this from 18:00-23:00 CET
Usually the worst is sunday night (again CET), but happens on weekdays as well.
Btw as i see, this started before update 9, but it wasnt that bad like now.

[skullzz]

Thelanis
Orchard Slayer area and Temple of Vol
5 +1 hireling: 5 include a monk/rogue/ranger, fighter (myself), 2 Wizards (1 pm, 1 am), and a bard.
Around 8-9pm EST
We were running to Temple of Vol and 2 of us just stopped running, there was only 2 or 3 mobs near the whole party on the run and no one was casting.
The monk, am and bard where not seeing any of the complete stopping lag.

Also

no specific time as it is continuously happening.
When ever I switch an item or am buffed I get about a 1 second pause while moving.

[Smitey]

Symptoms: Character will stop moving for a second or so every 7 seconds or so.

Last night (EU time) on Ghallanda I noticed framerate was 60 fps yet every 6 seconds it dropped down to 30 then jumped back up to 60.

From Myddo the time this occurred was: 05/18/2011 1:25 pm doing Ritual Sacrifice 5-man team with one cleric hireling 'lag' was unavoidable and didn't seem to be combat related nor impacted by amount of mobs; spells etc as the framerate drop also occurred simply running around.

Win7 64bit Ultra High settings - decreasing to Very high made no difference to the regular dip in FPS and loss was 0.5% latency around 115

At this point I wondered if it was the recent patch 1 shadow fix causing the hitch as my machine wasn't being taxed and normally runs smoothly.

After that quest I used the crafting hall and noticed long delays detatching items from mail/using the bank storage (not quite to this level of issue but definitely a 5-10 second pause)

This made me think it was server-side based on prior experience.

I didn't investigate further but noticed similar slowness with mailbox 6am GMT+1 when I briefly logged on this morning

[Sarisa]

Thelanis AND Sarlona both exhibit these same symptoms in the same quests.

Quests: Shadow Crypt in Necropolis & Ritual Sacrifice in the Vale

Symptoms: Character will stop moving for a second or so every 7 seconds or so. Sometimes rubberbanding happens where the character bounces back to where it was a few seconds ago.

Do you have Environmental Stencil Shadows turned off in your advanced graphics settings? These two quests, along with the Subterrane, are ones that commonly are impacted.

[knightgf]

Here's a rather interesting lag report to give you:

The lag in the crafting hall seems to have gotten worse. How so? Well, upon using the crafting system for the first time since login, the massive FPS spike I get has increased in duration by a few seconds. In addition, when crafting a shard without hitting the craft button over and over again, the lag between the button and roll has increased from 0-2 seconds to probably 4-6 seconds. There doesn't seem to be any lag when crafting the shards of potental. I've crafted at different times of the day, and haven't noticed any lag differences. In addition, im not sure if its relevant that I craft on the cannith server.

I believe you may find that the crafting hall adds quite a bit of lag. Although I must agree with you when you say there is probably more than one source.

[ProdigalGuru]

I am glad to see that Development is taking this issue seriously as it severely affects my enjoyment of the game.

Please consider my input to be that I lag MUCH more often in Raids and popular Public areas than anywhere else.

The Harbor, and the Marketplace are by far the worst offenders for public areas, with Meridia coming in 3rd.

Shroud parts 1 and 2 are by far the worst lag I encounter, and it is consistent. VoD is pretty bad, too.
I have been able to minimize lag in part one by staying away from whatever portal is being beaten on, but this does not make me popular on my Monk... and there are definitely lag spikes whenever mobs enter the zone from a portal. In part 2, the same kind of thing will happen when all the mobs swarm, and there are clouds/disco balls/etc. to add to the general DPS/Mob AI load. I tend to run it more on my Sorcerer now so that I can pike part 1 and not get yelled at, then volunteer to do the crystal on part 2, thereby avoiding the scrum.

The whole Subterrane explorer area seems to lag, but interestingly enough, I haven't had problems with it in HoX.

I honestly think there are just too many calculations going on at these times, combined with too many particle effects.

I rarely lag in 6 man groups, but have seen frequent rubber banding in public and private instances of the Sands.

People have always complained about crashes when zoning in to the Sands or GH wilderness areas.

My experiences are based on observations of Ghallanda, mostly late night Pacific time zone.

I think your prior efforts to reduce lag were good ones. DA encourages folks to kill the mobs, and when mobs are clear there is rarely lag. I do not know how much impact the TWF change actually had on this, however.

Just a random thought, but maybe it has something to do with the teleportation ability on devils? Seems the worst culprits are all devil-themed quests, because it happens pretty often in Amrath quests as well.

Thanks again for looking into this, and I wish you the best of luck.

[SirGideon]

Server Orien
Central time zone, time of incedent probably around 9-10 pm

I have witnessed lag spikes in alot of the quests already listed, this was unique in that it was major lag spikes on our ship, a group of about 4 of us getting buffs. Took us 3-4 times just to click on stat shrines and the like.

I am no expert, but this has started with the addition of crafting.

[Rasczak]

Server: Thelanis
These are the ones that effected us the most.

Raid: Tower of Despair (12 Players)
Occurence: Almost every time we go in some sort of lag happens.
Situation: Various situations including;
Healers healing the party killing Suulo, the yellow circle drops around us but nothing else happens.
Horoth will suddenly turn and run off and then tank will die due to heals not landing.

Drinking remove curse potions. Potion stack drops by 1 but curse is not removed. Drinking 3-4 potions and eventually it takes effect. *This has been confirmed for sp potions, disease, cures and poison as well.

Entire instance lags. We originally thought it was the blasephemy but we had on the anti-stun monk buff. Entire party had to wait for it to pass and we lost 3-4 characters due to the instance catching up and calculating damage done.
Time: 13:00 - 17:00 EST


Raid: Epic Zawabi (12 Players)
Occurence: Everytime we have tried it.
Situation: Entire instance slows up. Melees are swinging and healers casting but it doesn't register server side and suddenly hp bars empty in split seconds when server catches up.
In one case the queen was dead and party members would die as the server caught up.
Time: 13:00 - 18:00 EST

Raid: Hound of Xoriat (12 Players)
Situation: Puppies will drop all of a sudden dead even though they have been healed, they drop instaneously together. Players have reported killing beholders and running off, killing reavers and then dying from a FoD from a bh that was already dead.
(Confirmed as happening by myself after all bh's were dropped. Players had killed all of them and a player in front of me got FoD'd with no orbs or bh's around.)
Time: 13:00 - 18:00 EST

Raid: Shroud (12 Players, on one occasion 4 Players, All difficulties)
Situation: Part 4 we were hitting Harry and started taking damage from blades in the middle even though none of us could see or hear the blades.
Heals weren't registering although we could see the circle of heal being cast on us.
Boss's avatar will suddenly stop moving and then you take damage although being hit by them, avatar flickers to in front of you.
Times: 16:00-18:00 EST


Quest: Various quests (6 Players)

Situation: The entire instance will suddenly get the yellow connection icon and we won't be able to move. This is when ppl from all over the planet are in an instance together and all of us experience it. The mobs continue to act as normal and so it's essentially us getting hit while standing and doing nothing. Not even block works.

A dead mob will stand up again frozen and then drop dead

Jumping onto a ledge or over something and running forward only to suddenly be back in water/ hole/ with object jumped in front of you.

Running forward and being sent back a couple times before carrying on (rubberbanding)

Spells / clickies cast have no effect even though sp or charges get reduced accordingly.

Changing between areas has a massive (10 - 20 secs) delay before the blue load bar starts moving. When in parties we can still communicate through voice chat and everyone is experiencing it (Party members not in the same continent sometimes)

Time: 01:00 - 18:00 EST


There are plenty more but those are the ones that have been worse since U9 and patch. I am also ignoring the situations where 1-3 players were effected only. All of these here effected the entire party or the entire party witnessed it. In cases where the healers couldn't heal, it was all healers and not just one and there were no disconnecting players since we could all speak to each other or type in chat.

[Dragaer]

Server: Thelanis
Quest: Enter the Kobold
Time: Around 2pm CST on the 18th (+/- an hour or so)
party of 5 or 6

We had just arrived at the puzzle room. I was first in, did the initial teleport and everything froze for about 30-45 seconds. We all could talk to eachother, but no one could move at all.

After that spike, everything was fine.

[Cleanincubus]

Ghallanda around 7:30-8 pm EST last night.

"The Pit" - Solo w/o hireling.

I opened my in-game browser, clicked on a link to ddowiki from my myddo page, to see where to turn the color wheel puzzles to. The in-game browser went black and DDO immediately shut down, with no given errors (no windows pop up message asking me if I want to report an error, etc). Started game up again, and my Pale Master was no longer in Zombie form, but I was still inside the quest. Went into Zombie form again, and opened my in-game browser. Black screen again, so I closed the browser. Clicked Ctrl-Alt-Del and opened firefox to check ddowiki instead. As soon as firefox loaded, DDO error'd out and closed, giving me the windows "report error" pop-up. Started up DDO again. Again I was no longer in Zombie form, but I was still inside The Pit. Opened the in-game browser yet again. This time it started to load, but when I got to the actual ddowiki page for The Pit, it stopped loading. I clicked the "Stop" and then the "Reload" buttons on the browser, and it just showed a black screen again. DDO didn't crash again, but I was only online for another 10 minutes or so (ragequit after dying; having dealt with the crashes & wasted mana from having to change into Zombie form 3 times instead of 1.)

[BurnerD]

Server: Argo
Time: 8-11pm EST
Date: 5/18/11
Quest: Shroud - All 5 parts

12 party members. All 12 party members experiencing debilitating lag throughout the quest. Examples: massive lag spikes causing spells and melee damage to not apply. Opening a chest - character opening animation would complete, then blue bar would pop up and complete then chest would open.

Also experienced lag afterwards in 5 man party in Diplomatic Imp. and Framework. Not as bad the Shroud, but still significant.

Also one party member posted something from combat log: "you were healed by error" - wording isn't exact, but there was no toon named error so I guess the combat log wasn't able to get good data.

[GoldyGopher]

We are aware of the increased lag that has cropped up post-U9 and are looking into it. There is enough anecdotal evidence to point to a few candidates for investigation, but more info could be useful, hence this thread.

What I'd like, for now, are the world you play on, the quest, the party size and composition, the time of day (with timezone, or converted to either GMT or EDT), details about what was happening when the lag hit (for example, "all the casters had just called up ice storms", or "we got up from the shrine and couldn't move"), and any other info that seems relevant. If you were chatting with other players in-game and they did or didn't see lag at the same time, that could also be useful.

I play on Khyber, and I really see the problem in raids more so than quests, especially when a player disconnects.

I joke that when on a melee character I know when someone disconnects because suddenly my character is dead and it is followed by CHARACTER X has disconencted message. It is really annoying because the combat log will usually say something like "you been killed by something".

Healing is bad in Epic DQ using the ball method, heals and cures don't land.

Yesterday at approximately noon EDT in the shroud we had almost a party wipe in part four as 10 characters all died together leaving two melees that were fighting 'Arry. It was weird because nothing connected the other 10 characters together besides being in the raid together.

I have also noted that I am seeing a lot more of "no characters moving other than my own" in the Shroud on the portals, this is what we used to see before the fix about six seven months ago.

[hermespan]

On Sarlona I see random ping time spikes into the 700-800ms range. That seems to be when everything gets laggy. I play between 6-10pm. This did not happen before U9.

I'll see what happens tonight. If it happens I'll note the _exact_ times it happens in EST, based on my phone (since it NTPs the exact time)

Edit -- next day:

Played from 7 or so til 2 am and didn't experience the long ping times and lag.

The game worked very well. Even did a shroud that went off without a hitch.

[katz]

where: Orien
when: last night around 10-10:30ish EST
what: shroud, part 4

my husband (Ungood) was running shroud last night on his TR (Glane. ftr/rng), and after a mostly smooth and uneventful run... in part 4 they had that weird lag spike that kills the party that people have been complaining about. no damage is apparently being done, in either direction, and heals weren't hitting, most of the melees in the raid went from apparently full life to dead in an instant. it of course wiped the raid.

he turned around and went back in for another run and that went off without a hitch.


and myself, i've been experiencing a 'lag out' with far more frequency than i used to. it happened to me maybe once or twice before (ever), but after the update it has happened many times, on many different characters. what happens is this: i'll be minding my own business, going about playing, when for no reason i can see, i stop. when it FIRST manifests, my icon stays green, but i can not walk or jump. it is as if i am lagged. HOWEVER, i can chat freely, either text or voice. after a few seconds, the icon turns to yellow, and eventually red, and then i get booted out. nothing i can do stops this process, and i can freely chat all the way until i get booted. i know my messages are still getting across because i can see my text and hear my voice come from my husband's comp next to me. i have learned, painfully, that once this symptom crops up, my only recourse is exit the game and re-log in, because even if i don't log out, the game will eventually kick me. sometimes when this happens, i also cannot immediately relog, but i get the "searching for login, attempt of 1, of 2, of 3, etc, timeout" error once or twice (one time it lasted the rest of the night. i checked regularly for the next 3 hours and still couldn't log in. next morning was fine.)
it seems to happen at random, and unfortunately i have no kept track of the wheres, whens, and hows, but in the future, if it happens again, i will record it and relay the info here.

[Chai]

Do you have Environmental Stencil Shadows turned off in your advanced graphics settings? These two quests, along with the Subterrane, are ones that commonly are impacted.

I will look into that when I get home. Thanks for the tip.