Game Performance Improvements and Feedback

[Silverleafeon]

1. Server - Khyber
2. Approximate Time - 10:20- Midnight CDT, Monday evening April 11
3. Approximate location of your computer - Alabama
4. Quests/raids run - Shroud
5. Difficulties run - LN
6. Your character class - Deep Gnome Fighter 16/Pally 4 in LD blitzing
7. Number of players in the group - 12 then 1 dropped immediately, 1 DCed towards end and was kicked
8. Anything else you think we should know - Annoyingly playable, lag every 10 seconds or so in part 1, autoattack seemed to work more than button pressing, an ungly 3x killing the bosses in part 2 due to lag from lieutenants dying especially the last one, an ugly 5 round harry due to getting renown from gnolls causing 5~9 seconds of lag, part 5 went fairly smoothly

Group stuck together as much as possible which seemed to help a lot.
Computer/internet kept up with downloading foes, chests.


All in all, a very sluggish 11 man LN Shroud, late Monday right after a "spring break" type weekend is over.
Logged out afterwards, between waiting to fill, and the raid, long time.


No news on static IP yet, went and got my router today.


Pretty clear like to me with the 3rd set of gnolls generating renown several rounds in a row, and suddenly 10 seconds of lag (while harry heals up), renown is definitely a big problem atm. We took harry down by getting lucky and not getting renown from gnolls on last round.

[eterna1_drag0n]

This message is directed directly to the Developers, Executive Personnel, and Systems Protocol personnel.

There has been a huge amount of data supplied by the clientele of this game, but little to no data supplied by the staff.

I propose the following: If you let your community help you, we will. Free of charge, no fees, service charges or addendum's; we will help you fix the issues.

DDO is supposed to be based on D&D, which is; "A game of the Imagination" (Dead Ale Wives) where the only limitation is a players own creativity and knowledge. Let us, your eager and active community, help you fix the problem that has been limiting ALL of our enjoyment. I have been an active player for the better part of the last seven years, and I know for a fact that there are quite a few networking specialists that play this game, I have conversed with several about the sometimes laughable protocol issues that have bogged down the servers. I understand that EVERYONE is doing their best to resolve this issue at your headquarters, but to put it simply, it isn't working.

It is time for the paradigm to shift again. DDO and Turbine has been the innovator for in-game, micro-transactions with the Free-to-Play format. Let this be your next innovation, Community Enabled Support, CES. Let your player base help! I assure you, we are pulling out our hair in frustration as much as you are. But our frustrations are based on our inability to give constructive feedback to on an issue that we have experience with. If you don't know exactly what the issue is, that's fine, lets start with the basics. Post some specs about your servers, volume, operating: how are they running? emulated/virtualized or hardware, average population, peaks highs and lows, how is your node integrated?...

If you let us help, we will...

[donblas]

Khyber

Approx 9pm BST (4pm EST?) Monday

2 in party: level 12 sorcerer, level 10/2 wizard rogue.

Marketplace

5 minutes or so of intense lag, rubberbanding, inability to interact with teleporter.

[JOTMON]

have you ever even considered acknowledging that virtualization was a gross miscalculation and the best solution would be to buy actually proper servers? LotrO has had a lot of "small patches" which "hope" to "improve" and look what it got them.

I dont bother with typing reports here anymore, it is just tiring to spend yet more time for nothing after frustrating game experience. BUT, to sum up, over the last weeks, NONE of your tweaks and adjustments have made noticable improvements on Argo. Our static raid group still freezes and stutters through its cycle of EN LN raids every thursday and sunday night, and while with exception of xizzy which blows in ur face if bad lag, they can be completed, somehow, there is virtually NO improvement whatsoever on epic raids or EE 5 or 6man groups. While it is true that the frequency of solo questing and public instance lag has slightly decreased, as of just yesterday i still find myself hitching around midair in a lvl18 solo anniversary challenge, still lag out in various public places from necro4 to estar market to idk where on occasion etc pp.

you'll forgive me for not buying into *improvements* anymore untill i hear 14h downtime we are removing virtualization completely.

btw, i been wondering, have you done your tweak testing in pairs of servers always because now 2 servers share a physical machine on VMs?

All that is happening with virtualization is the server is running in an instance on a server.
There are more benefits then detriments running in virtual.

Virtualization is the way of the future and is very stable when setup and allocated with proper resources.
Recovery from system issues, upgrades, system improvements all are much easier when running the servers virtualized.
Bigger hosters are running multiple servers as the platform to several virtual systems simultaneously, so losing/replacing a server does not take down the system.

There is tweaking time to ensure the virtual server load demands are adequately covered by real system resources.
This falls into the $ side of it since you have to buy/allocate resources from the live server host to the VM and find that sweet spot where you are not paying for more than you use.

Where the problems happen is when the server is not allocating enough resources to the virtual server, or is hosting too many virtual instances.
Farming out to cloud servers or other hosted servers leaves you at the mercy of the hosters..


DDO has more problems than this going on.. 10 year old custom code written by guys that are no longer around.
Layers of coding to patch issues and workarounds over the years with few (if any) clean rewrites of code.
Some known issues with effects that keep reapplying themselves behind the scenes causing some overstacks of effects.
After a while too many patches and temporary fixes catch up.. spaghetti code effect...
You can only patchwork fix for so long before the cost/time of patches outweigh the value.
with dwindling resources it becomes more difficult to keep up with the spot fires and never getting to the source fixes..

starts to layer like a Jenga game.

[Eryhn]

very stable when setup and allocated with proper resources.

it does not look like ressources being sufficient for peak usage. that asside, because of this:

DDO has more problems than this going on.. 10 year old custom code written by guys that are no longer around.
Layers of coding to patch issues and workarounds over the years with few (if any) clean rewrites of code.
After a while too many patches and temporary fixes catch up..
You can only patchwork fix for so long before the cost/time of patches outweigh the value.
Then its rebuild , replace, or drive it into the ground and walk away.

having been the case for years, this:

Where the problems happen is when the server is not allocating enough resources to the virtual server, or is hosting too many virtual instances.

is not the only issue.
rather, the problem is also happening when you put the spaghetti code u described on a VM in the first place. while in theory VM is stable, its when you have a bunch of clunky things put on it that the problems created greatly multiply in effect, ya get overhead like mad and dont even know from which of your age old little ****ups, and this would be an issue even if the initial bare metal allocation behind the VMs were benevolently calculated, which i bet you a shiny festival twig it of course wasn't.

the very premise to be able to do the needed tweaks on the fly over time on life servers on a minimal cost bare metal based VM is grossly irresponsible and a huge insult to customers. the amount of messup was predictable, with code like ddo its either have so much hardware behind the VM that you might just as well go bare metal from the getgo, or do far more excessive testing and configuration BEFORE going live if you insist on this ******** being implemented. Assuming the networking team is as strongly manned as the dev, which is to say it is grossly understaffed, how could anyone think this would go smooth in a even half way decent time frame, given they been through the same **** with the 2 preceding game server migrations?

and I bet you TWO shiny festival twigs turbine is so tightlipped because they knew EXACTLY this would happen, which is btw why all the rage is quite justified and all those, *oh hey, i STILL renewed my sub despite all this, cause, THINKPOSITIVE TM* folks have my deep sympathies on the astonishing depth of their illusionary world - it's the approximate equivalent of still paying your sub fees to turbine 2 years after the game servers were turned off cause *hey, back in the day it was good times*

[Cordovan]

We are currently working on a patch, and expect to announce downtime in the coming days. This patch includes a fix to several stacking enhancement issues along with some other optimizations. We'll keep you updated.

[Kompera_Oberon]

We are currently working on a patch, and expect to announce downtime in the coming days. This patch includes a fix to several stacking enhancement issues along with some other optimizations. We'll keep you updated.

Please keep pointing to this thread. Because of your horrible communications methods it is often *ahem* lost in the web.

[MeliCat]

We are currently working on a patch, and expect to announce downtime in the coming days. This patch includes a fix to several stacking enhancement issues along with some other optimizations. We'll keep you updated.

Does "fix" mean "remove"? As in simplify so that the new HW dimensioning can cope?

[Livmo]

Last night between 8pm-9pm PST. Legendary Shroud. My first one. Posted LH, but we lagged too bad and went with 10 on LN afters.

That was my first Shroud and not a good impression due to the lag on LH. Also it seems odd that the portals have so many hit points with such bad lag. Not a good combo. LN was laggy but not as bad. We completed on LN

We ran LH Hound of Xoriat after with mostly the same peeps with 10 in party. Lag was bad in Marketplace, but not too bad in the raid. We were able to run and complete that one on LH. That was a just after 9pm PST last night.

I have not run all the raids since data center move, because I've been busy lately, but 2 raids so far unplayable beyond normal, Legendary Shroud and Fire on Thnuderpeaks.

However, with Fire the lag was so bad we didn't even get a normal compeltion the last 2 weekends in a row.

In each instance the whole party seemed to lag at the same time.

I did enjoy both the new quests, although laggy, prior to LS.

[Cruxader]

We are currently working on a patch, and expect to announce downtime in the coming days. This patch includes a fix to several stacking enhancement issues along with some other optimizations. We'll keep you updated.

Cordovan and Dev and DC teams- First, thanks for keeping focus on this, the near and long term future of the game clearly depend on solving it completely, not partially as I'm sure you are well aware from all the shouting.
Second working in the Data Center Industry for 20+ years, I'd hope to offer some helpful guidance with a couple questions I hope bring value from my experiences fixing lag for both gaming and corporate systems for decades:

1. have the network and backbone fixes your referenced at the start of the thread been implemented yet? That seems the cause of some (but not all) of the recent lag issues.

2. regarding platform portability and code fixes - I'm sure you guys have the portability issues identified by now and hopefully that patch will help a bit here as well but...

3. Regarding the fundamental Resource constraints when processing 12 person parties in raids at peak hours (the biggest threat to this game)- The server virtualization... yes as some say it is the way of the future from a corporate cost savings and resource optimization perspective, though not often from a system performance perspective which unfortunately is exactly where the problems are here today with this game.

Has anyone there yet tried upping the resources allocated from each physical server to each virtual instance? Or tried running hosted back on bare metal instead of on someone else's uncontrollable Cloud?

I assume you were running cold iron before the recent data center move and the lag, though not perfect, was much better then.
If the game worlds are hosted on shared servers (virtualized) unless there is enough resources provided to accommodate peak max usage for each instance at peak times, unless they are allocated to handle that, or burst up to that (which is usually a very fragile solution anyway) then yes there can clearly be resource constraints coming from the under-resourced virtual instances there.

I again appreciate the patches and hope they help, and think they may a little bit, but the severity and type of lag we are seeing is indicative of either massively underpowered new Data Center network, or massively under resourced virtual server instances that need to be not tweaked 30% but perhaps tripled (200-300%) to really accommodate peak load (please test case Legendary Shroud with a 12 man party on LH on each server-world alongside with 12-man DOJ, to ensure that at minimum these can run these each per server without lag in production, on all servers *simultaneously during peak hours*- *THAT* is the *minimum* benchmark the tech and DC teams needs to plan and resource for if they want the game to continue, which all of us surely do because if the servers cant even handle that parallel load at peak hours then the game is effectively a single player game until it can.

Very best regards,

[Sam1313]

Report for Tuesday 4/12/16

1. Server - Cannith
2. Approximate Time - 8:00 pm EST to 12:30 AM EST.
3. Approximate location of your computer - Lafayette, Indiana (Midwest)
4. Quests/raids run - Devils Assault.
5. Difficulties run - Epic Normal / Helping a guild member farm for Tokens so she can TR.
6. Your character class - Ranger
7. Number of players in the group - 2
8. Anything else you think we should know - See video on my Youtube channel titled Lag Or click link here: https://www.youtube.com/watch?v=NhsqcgqMVyY

[GlassCannon]

If you are continuing to see poor in-game performance, then any information you can provide is helpful.

I've been experiencing barbarian rage and frenzy effects applying negative but not positive bonuses, taking damage from sources that appear to not actually be where they show/say they are, seeing Death effects persist until logout despite being very much alive, seeing expendable items/effects not actually do anything despite being expended and other various crippling incidents. I'm reinstalling DDO today to give it another shot.

If I experience the same "My client needs to catch up to the server" problems and the server simply ignoring what my client is saying and permanently destroying items that I rely on for their described benefit without applying said benefits, I'll just go back to Ashes of the Singularity or whatever it is I find to do.

The quests I experienced these problems in are: The entire Lords of Dust chain on Heroic Elite with a Barbarian, Temple of Elemental Evil on Epic Elite with a warlock, Eveningstar chain quests on Epic Elite, Druid's Deep and Stormhorns chain quests on Epic Elite (the most problematic was What goes Up after the Mythallar has wedged in the glacier and you fight huge spawns on the way up). I actually had to just quit WGU due to being forced to watch my character die within 2 minutes (10-30 seconds of actual game-time due to lag) of being raised, more than 6 times in a row. Consistently.

Please please please figure out what's causing these problems. I freaking love this game and want to help but I'm no technomage.

[Livmo]

Between 8-9pm PST last night we ran raids on Sarlona. Heroic Elite Shroud was not too bad, but when we ran Fall of Truth afters on EH it was bad (lag).

I've run FoT countless times on EH without lag. Last night I died after after completion, because could not move or heal and eventually the boom got me. At least it was after completion.

However, its frustration to be stuck in one spot helpless watching you health go down and then die. I could not even move after death for about a minute.

The freezing and rubber banding are horrible.

I also had some rubber banding just after 9PM PST while running the High Road chain on EE for sagas.

[Cordovan]

We are currently testing a patch. Once that's done and things look good, we'll formally announce downtime. That said, we're currently targeting a Thursday 8:00 AM to 11:00 AM Eastern downtime. This patch will include several optimizations to improve game performance, and correct an issue with barbarians that is contributing to poor game performance.

[Lerincho]

barbarians that is contributing to poor game performance.

Figures, the savages causing all the trouble. Can we kick them out of Eberron?

[Cordovan]

I wanted to make sure people were aware that I chatted with Sev for the first six minutes of today's livestream about the lag, and he provides some detailed information about what we are doing and what's been happening: https://www.twitch.tv/ddostream/v/60375811

[Silverleafeon]

1. Server - Khyber
2. Approximate Time - 6:30 ~ 7:30 CDT, Monday evening April 12
3. Approximate location of your computer - Alabama
4. Quests/raids run - Shroud
5. Difficulties run - LN
6. Your character class - Deep Gnome Fighter 16/Pally 4 in US, with mostly passive bonuses.
7. Number of players in the group - 9 (I got lead was going to hold at 10, but one player toon swapped an never returned.
8. Anything else you think we should know - 100% of lag for me from renown, red name dying, objectives being met, loot calcuations, etc...
Greatly improved by my static IP address and new open router system.
2 or 3 players complained about serious lag issues.
The renown lag was definitely less with 9 players compared to the 11 players the night before, despite being a more populated server time.
The initial spawn in part 1 moving toward the bridge did have some serious lag till we killed some of those foes, but the party was able to stay with the networking.

Group stuck together as much as possible which seemed to help a lot.
I think the trash killers suffered more from lag than the melee, not sure.
Computer/internet kept up with downloading foes, chests.


9 man LN Shroud, late Tuesday right after a "spring break" type weekend is over.
Long time filling, went quicker after I got the star.
Logged out afterwards.


News on static IP & Opened up router (along with tight security subscription):
Strong improvement on overall gaming so far.

[UurlockYgmeov]

I wanted to make sure people were aware that I chatted with Sev for the first six minutes of today's livestream about the lag, and he provides some detailed information about what we are doing and what's been happening: https://www.twitch.tv/ddostream/v/60375811

Might want to make the above a separate thread.

[Captain_Wizbang]

Interesting info. Thanks Cordo & Sev

[DagazUlf]

I wanted to make sure people were aware that I chatted with Sev for the first six minutes of today's livestream about the lag, and he provides some detailed information about what we are doing and what's been happening: https://www.twitch.tv/ddostream/v/60375811

That was really great information. Thanks Cordovan.

Any chance we could just see that as a few bullet points in a tweet or forum post?