Game Performance Improvements and Feedback

[Kritty]

As part of our move to the new data center we updated some architecture to modern operating systems and virtualization technologies. In some cases adjusting the old code to run in this new environment has added additional challenge.

If you are using VMWare or Hyper-V, particularly Hyper-V, as your hypervisor you may wish to try turning off dynamic memory (if you haven't already). It's a neat technology but can potentially cause hitches in the guest OS when it changes the RAM amount. Not noticeable for most kinds of services but might cause issues with something that is realtime like a game. Also make sure you haven't oversubscried the physical RAM, while it'll let you do that the penalty for paging is even more than normal.

[Elsbet]

Is it that way on all servers or some more so than others? Are there times when it is runnable? My game time is pretty static, so I don't have a basis for comparison.

[Krelar]

Argo.

Rubberbanding in house D at ~10pm eastern. Everyone that joined my group commented on it. No lag once I actually stepped into a quest.

Later on in Von3 there were little "micro" lags, the playable but annoying kind where your position hops around a bit.

[Thrudh]

If you are using VMWare or Hyper-V, particularly Hyper-V, as your hypervisor you may wish to try turning off dynamic memory (if you haven't already). It's a neat technology but can potentially cause hitches in the guest OS when it changes the RAM amount. Not noticeable for most kinds of services but might cause issues with something that is realtime like a game. Also make sure you haven't oversubscried the physical RAM, while it'll let you do that the penalty for paging is even more than normal.

One should always reserve memory for high performance VMs in VMware and never over-subscribe.

[Eryhn]

had static raid group yesterday night on argo, ran with a party of 11:

TS LN: completed pretty fast, no major lagdeaths but constant little freezes from 1 to 5 seconds duration, most notably in the room with the 3 red named, and generally as a rule of thumb whenever many mobs were killed in succesion (ambush gate, end of the bridge to inevitable)

Hound LN:completed fast, slightly smoother, but same pattern of stutter freezes on mob death, the now usual huge 15sec lag out when xizzy dies.

DOJ EN 10 people: lagged to fail on reinforcements, was terrible. lag on every aggro drawn, lag on every mob killed, on big groups freezes of up to 10 seconds, and many of those. unplayable.

DOJ EN 4 people: 2 went in to test stunning blow on orthons and devils as we were discussion if it works or not, 2 followed for the funs, as we noticed how very much quicker the lag freezes cleared we kept goin a bit for scientific purposes. the lag is still there in the exact same manner and frequency, but very noticeably shorter in duration which allows you to react/cast/dps/heal more. finished out after 10mins as the group was waiting.

EN Peaks: completed reasonably well. a lot of little lag freezes, which we managed to keep in check due to focusing very hard on killing trash asap. still same pattern of constant short freezes. unlike last time, end of fight the raid did not bug and trash properly stopped spawning.

DOJ EN 6 people. came back again as the group got smaller. as above, same style of lag, occurs as often, but less excruciating effect, noticably. less than with 10, slightly more than with 4. due to slight lack of dps went into end fight with 4 archons dead. more trash in end fight led to bigger lag spike when the worm died, which led to belated dps on boss, which led to party getting blown to pieces when she nuked from orbit. this is kinda nasty to dodge with the non working area of effect indication PLUS lag. had her at 30 % ... was very fun to get through it so far with little people, but all the more frustrating to fail so close to success due to lag despitre the halved group size ...


bottom line: im starting to grow a tinfoil hat here and believe it: CAN WE PLZ DISABLE GUILD RENOWN FOR A DAY AND SEE WHAT HAPPENS? *just* to rule that out ... cause it rly coincides with those freezes on mob death. if not that, plz look what else there is that would hang up a script or process or instance on mob death ...

[Duana]

I got lazy yesterday, wasn't really feeling well. I did not send any info in. But here is a quick recap.

Thelanis. Online off and on from 1100 to 2359 EST.

Spent the day mostly solo in Wheloon. Even solo, lag was noticeable and at times hindered gameplay. I kind of expected lag in the explorer area when I went from one realm to another (I still dislike that), but did not get it. Noticeable highlights were the two levers in Lesson in Deception that spawned 3rd density Reptilian Lizardmen. The platform at the end of Thrill of the Hunt. Again, areas with surprise spawns.

Still the constant lag all over the place, quick hitches here and there throughout the day. It did not affect game play, but it is like that constant nagging headache. Not enough to make make you quit, but makes one more and more irritable as the day wears on.

Did some party favors today when another guildmate logged on. Even with only two people we had big lag spikes today. Some each run, lasting 15-45 seconds. I have to admit, the first time, it was amusing to see the mobs doing the "running man" for once.

As has become the new, and I hope temporary, norm lag got worse as the day wore on.

Tracert, I looked it up and it looked the same as **** near every one I have sent in. By the end of the night when it was time to send it in, I felt bad enough I just could not stay on to send it in. sorry.

[Cordovan]

Thanks for the reports.

[ramseyazad]

I am suspending my VIP and leaving the game until this gets better. I simply don't have time to beat my head against the wall trying to play a game that doesn't work. Please make some sort of widespread announcement (on the FB page maybe?) when it's actually reasonable to play the game again. Honestly, I think you should shut down the servers until it's playable. It's deceptive and dishonest to leave the servers up as if they're working properly.

[EmGreen]

Thanks for the reports.

Another report - currently in a EE temple of the deathwyrm raid with 10 people on Ghallandra. Just had the second party wipe due to lag at mountain summit while fighting auraglosa. This is deeply frustrating. The whole party experiences lag and freezes and stuttering. Seems to be related to large numbers of enemies and mobs spawning.

Still getting freezes and lag even with most of the spawns dead. The monsters can still move and attack while we're frozen.

[Shadow7375]

Game is unplayable most of the time and I am paying for that with my subscription fees ... pfftt .... Party wipe after party wipe, day in - day out. It is so frustrating.

Like a buddy said today .... "Let's face it, by now it seems clear that the big "new data centers" move was a move to smaller, cheaper data center. It's Turbine's way of easing the game into maintinance mode.'"

I hope that's not the case and you guys have this issue solved in no time. Otherwise I see people leaving if this takes ages to get resolved.

[azrael4h]

Graphics are rendered on your PC. No "graphics" information is being sent across the web. This game works with real-time collision detection, when many other MMOs do not.

Not defending Turbine. They broke some stuff with their move. They need to fix it. But this game isn't "simple", and an older graphic engine doesn't mean the network code must be easier.

Not saying either side is right, but I was playing Warthunder earlier, which is basically a lobby/matchmaking based FPS (with tanks and aircraft instead of infantry). It is very intense, action based, and I spent about 4 hours playing with absolutely no lag. And it's not even installed on my SSD, it's on my Programs drive, which is mostly for old DOS games right now (too lazy to install much at the moment).

While DDO is far from simple, it also isn't as heavy on collision detection and such as a FPS. And it's almost returning to it's turn-based roots in some instances nowadays.

The thing is, I'm not certain it's even anything to do with the networking. I froze out earlier tonight in The Necropolis. No one there, nothing going on, but I was locked down, and just barely managed to turn around and return to House P. The rest of my party, who took the ship to the Orchard entrance, were lagging. In house P, both before and after that lag-filled mess, I was fine. No latency or loss according to DDO's own network icon. Other times I've lagged, it only happened when special mobs (such as orange/red named bosses) appeared, and the lag was gone when they were dead. But only on certain runs of the quests. I can reenter immediately after, and not get a bit of lag the second time through.

[Robbenklopper]

Ran TOEE P2 on EH solo, Cannith-Server +/- 18.00h GMT. I hadn´t had lag the whole evening and thought no evil, the whole dungeon was smooth. With the Spawn of Hezrous at Zuggtmoy, everything was over. 5+ secs frozen Screen and "Ting".

100Mbit Connection, SSD, 8 core cpu, 2GB VGA.

[Chacka_DDO]

there more to tell but just the most noticeable things here

Lag to very bad lag on almost every quest finish and also when quest objectives are done or monsters spawn or get active. This depends on quest and if it is a raid with more members in party.
Very bad lag foremost in Mark of Death and Defiler of Just the last times I did it. On my server Ghallanda people seem to not even dare to do this raids at the moment because of this issues.

Last time I finished Hound of Xoriath exceptional long stop of everything at finish. But in general it seems to be playable at last.

Last time I finished Temple of Elemental Evil on epic elite it was for like 10 minutes barely any moving for the whole party. We wiped several times because of this issues and that's also the reason why i can tell you in public area it was almost lag free while inside the quest everything was almost frozen. All this for everyone in my party and not only me.
This starts as we started the end fight, before it was sometimes a bit lag, but not exceptional much. The party kept trying anyhow and even waited a while until the lag got better. Someone in the party said the AI (of Zuggtmoy?) can cause this lag but this is hard to tell. But I can say after the last wave spawned it got at some point better and we where able to finish it.

Another problem here is, I got since two days almost always a latency of 500+ ms, sometimes it goes back to my normal around 130ms. This might be of course a problem on my side but i can tell you I played this game almost from start and I have this normal 130ms always and even with a different ISP (changed my ISP two times). Asked in my party and it seems like not everyone got this issue, one told me even he got 40ms (can only dream of such a nice connection).
Anyway its maybe good to mention this permanent issue of mine because the network status display is still green for me, even when 400 or more ms cause already issues like jumping up on something and then suddenly it sets you character on a completely different point as it should be without this latency issue.
It might be possible more people see lag and don't notice there latency is to high because they don't check there Network Status display (default is to not show it beside it went to yellow or even red).

Another issue with latency I have and it is weird to me: If my DDO client suddenly shuts down, witch happens by the way several times/day with no error massage or something and I restart my client I have very high latency (around 1300ms) after like 5-10 min it went back to normal (witch is normally around 130ms and at the moment at 500+ms)
If I reboot my system the latency is from start at the normal score.
The day before my latency permanent went to 500ms this issue was gone somehow for like one day.
Needless to mention but anyhow: I changed nothing on my system or ISP or anything so it is at last possible this is an issue on server side, even when it not affect everyone.

[DagazUlf]

Last night (Friday), we attempted to run just a few random raids to see how badly the lag would affect us. There were eight of us from across North America ranging from D.C. to Vancouver Island, Michigan to Southern California.

So, we jumped in the old jalopy together and started our road-trip with the Defiler of the Just experience.
Result: There was no roadway. We were basically sitting in the car on a rough frozen lake going nowhere, and actually sliding around a bit. We saw a couple of Orthons and some Green Devils, and then moved along. We were unable to even drive up to the front gate. Total failure.

Our auto-tour then continued with a trip to the Hound of Xoriat.
Result: Like driving on an older blacktop roadway that hasn't been repaired in a while. Some potholes and rough patches, but we got to the end of the road okay and collected some nice photos along the way.

Next up was a visit to the Thunderpeaks in hopes of spotting some dragons.
Result: A gravel road that has one of those "Low Maintenance Road" signs at the start of it. Some severe washboards in places, enough to rattle your teeth and make you want to get out of the car and puke. Dragons were spotted and killed; no keepsakes acquired at the gift shop.

Everyone was fairly tired of the trip at this point, but we decided to have a look at the Fall of Truth.
Result: Frustration continued as we drove along another gravel road. This one with quite a few hills and whoopie-to-dos. Coordination of kill-shots was difficult, but we made it all the way down this roadway. It was not scenic, it was not fun, and it is a trip no one is going to be taking in the future.

At this point, everyone was done with this auto-tour and left to go seek out adventure and entertainment in other venues.

Of note, the network connectivity icon showed no one with any loss at all, and latency ranged from one guy with 18ms to a couple of us with 80 to 90. We all experienced the lag, freezes, slow-downs, whatever, simultaneously. Also, the integrated DDO voice chat now seems to be of the lowest quality we have every experienced. Lots of breaking-up and "line-noise".

Overall, an incredibly frustrating and annoying experience while trying to play a game that all of us have been enjoying for the past ten years. That entertainment value is now gone, zip, nada. Not worth even logging in anymore until you guys figure this out.

Thanks for reading.

[EazyWeazy]

Last night (Friday), we attempted to run just a few random raids to see how badly the lag would affect us. There were eight of us from across North America ranging from D.C. to Vancouver Island, Michigan to Southern California.

So, we jumped in the old jalopy together and started our road-trip with the Defiler of the Just experience.
Result: There was no roadway. We were basically sitting in the car on a rough frozen lake going nowhere, and actually sliding around a bit. We saw a couple of Orthons and some Green Devils, and then moved along. We were unable to even drive up to the front gate. Total failure.

Our auto-tour then continued with a trip to the Hound of Xoriat.
Result: Like driving on an older blacktop roadway that hasn't been repaired in a while. Some potholes and rough patches, but we got to the end of the road okay and collected some nice photos along the way.

Next up was a visit to the Thunderpeaks in hopes of spotting some dragons.
Result: A gravel road that has one of those "Low Maintenance Road" signs at the start of it. Some severe washboards in places, enough to rattle your teeth and make you want to get out of the car and puke. Dragons were spotted and killed; no keepsakes acquired at the gift shop.

Everyone was fairly tired of the trip at this point, but we decided to have a look at the Fall of Truth.
Result: Frustration continued as we drove along another gravel road. This one with quite a few hills and whoopie-to-dos. Coordination of kill-shots was difficult, but we made it all the way down this roadway. It was not scenic, it was not fun, and it is a trip no one is going to be taking in the future.

At this point, everyone was done with this auto-tour and left to go seek out adventure and entertainment in other venues.

Of note, the network connectivity icon showed no one with any loss at all, and latency ranged from one guy with 18ms to a couple of us with 80 to 90. We all experienced the lag, freezes, slow-downs, whatever, simultaneously. Also, the integrated DDO voice chat now seems to be of the lowest quality we have every experienced. Lots of breaking-up and "line-noise".

Overall, an incredibly frustrating and annoying experience while trying to play a game that all of us have been enjoying for the past ten years. That entertainment value is now gone, zip, nada. Not worth even logging in anymore until you guys figure this out.

Thanks for reading.

And the most "successful" raid of the night, the Hound run, took us close to 30 mins with all the lag and stuttering. Its not that long a quest. Trying to keep positive here, but its **** hard to do. **** hard and 10 year old friends are just not logging in anymore. Really really sad.

[Faltout]

EE CitW today 3:30 to 6:00 pm EST on Khyber.

The results:


Monsters killed

Breakables smashed



Ohmega wants noted that all his deaths except 1 were by running off the edge due to lag.

Lag was horrible (hence the 8 people that left during the raid).
We lagged (froze in place) when:

• We aggroed more than 5 monsters (5 being an arbitrary number, a medium amount of monsters)
• Monsters lost line of sight to the person they had initial aggro on.
• Monsters aggroed too many players over the duration of their life
• Guild renown came in
• objectives progressed (like when ana went inside portals, when purples died, when all orbs were drained, when lolth went away)
• random times (we were fighting a single bebelith and we lagged for 5 seconds). That rarely happened, but it did happen nonetheless.

We did not lag when:
- We fought 3 or 4 monsters at a time in a tight group (whole party at the same spot)
- Ana took aggro of every trash monster in the 1st orb fight while the rest of us were mostly dead (that made us lose the 6th chest... bleh)

We did not freeze (but had lag meaning many of our attacks were lost in limbo) when:
- We were fighting many monsters that had their sights set on static targets (ie. not changing aggro). This happened in the 2nd orb fight where we had lag, but were able to keep attacking the purple bebelith. Monsters were chasing someone at that point.

Thankfully, we did not lag at the end lolth fight since we used safe tactics and lolth island was fully cleared. The hardest parts were the middle islands and the orb fights (well, the portal keepers too but at that point everyone is happy that the raid is almost over).

P.S. I'd like to thank Pipzorz and Spottedfang for sticking till the end of the raid. Especially Pipzorz that was the first to join the LFM (which was up for an hour)

[Kulothar]

Attempted an EH MoD last night with High Lords which should have been a walk in the park teaching raid.

Spawn/Dialog lag was light.
Entering the DD into the middle room to get the bag caused immediate 6 - 8 second freeze for all players zoning into the room.
Killing the Beholders and Death Knight caused about a 2 -3 second freeze.
Zoning back into main arena caused minimal but noticeable lag.
I was on cleanup. Out of 6 Beholders that I killed, I immediately lagged and died 5 times upon killing them. The 6th I had learned my lesson and ranged so that even though I lagged there was nothing around me to kill me before the lag cleared.
There was slight but noticeable lag on each mob killed, Abbot spawn and inferno. Noticeable lag on MoD and slight lag on Dragon breath.

Eventual wipe since every event listed caused deadly lag making deaths faster than rezes even though twice we were all rezed. We could not keep the Beholders under control because killing one resulted in mass lag deaths.

This is not server, connection or computer lag. It was not one player having lag or lag from multiple spawns (although this also contributed). This is GAME LAG. It is directly caused by game events and not connection. The last time I saw this was the parry/reposte lag that was fixed. Abbot was at 10% for a significant amount of time but we could not kill him since every time we killed a trash mob the lag resulted in player deaths.

Raid wiped and we had to complete on EN which had simular lag issues.

[Kristii]

Tonight, 8 pm, on Cannith server:

2 attempts at DOJ. We only took 7 people to try to prevent lag. We also reset it a few times looking the "best instance" to prevent lag. Both attempts lag-wiped. Entire party unable to move for over three minutes.

[lugoman]

I'm curious, dont devs get lag when they log into any of the live servers?

Just log onto Cannith. Start a hard deathwrym raid and enjoy.

Did one on hard recently and would take the light beam about a minute to traverse the mirrors. In red/green I was standing on a pressure plate thing not even touching the mouse/keyboard and died from moving when red light was on. We were full when we started but had maybe 7 when we got to the end due to rage quits. Took 2 hours to finish. Another group was running it on normal at the same time and reported no lag.

[dontmater]

if it helps..

i know a few times. when it was handing out guild renown while killing things, it seemed to make lag spikes. but in no way that was all the lag.

doing the two dragons on the summit bites this weekend.