State of Lag - official statement possible?

[cadaverash]

Edit: Forget combat, cant even do a bank transfer without lagging and getting the "wait couple of decades before interacting with inventory". Ghallanda is awful, fcking awful.

If the Dev's read this, they may nerf our TR cashes and say it was causing lag.

Imagine not being able to take your stuff with you when you TR...

[Mofus]

Played a little bit yesterday, didn't get any lag, but I have experienced it often enough to know it's still there.

[Aelonwy]

If the Dev's read this, they may nerf our TR cashes and say it was causing lag.

Imagine not being able to take your stuff with you when you TR...

Careful I can just see someone at SSG like this:

[Bacon_Burger]

Kind of seems fitting that we wait for an answer, no?

Yeah, considering it took 8 days for your post to show up!

[TitusOvid]

bump

[TitusOvid]

bump for devs

[Chai]

Since no other game lags like this for this period of time unfixed I'm betting they know and fully understand what the issues are, and understand the resources to resolve do not exist due to needing to make numbers this time period. They implement one thing to address the spike behavior only, knowing the full issue will not be addressed in the same limited environment the server side architecture exists in.

The bottleneck in human decision making would need to be resolved first, before any bottleneck in hardware/throughput etc...could be addressed.

[LurkingVeteran]

Since no other game lags like this for this period of time unfixed I'm betting they know and fully understand what the issues are, and understand the resources to resolve do not exist due to needing to make numbers this time period. They implement one thing to address the spike behavior only, knowing the full issue will not be addressed in the same limited environment the server side architecture exists in.

The bottleneck in human decision making would need to be resolved first, before any bottleneck in hardware/throughput etc...could be addressed.

If it was so easy to figure out the lag issues, why would they have needed to bring in external (ex-dev) consultants to come up with the multi-hit change?

I still think they just lack investment in profiling tools for figuring this out. Considering they seem to mostly share engine devs with LotRO, the engine side of DDO running the gameplay scripts is probably a neglected pile of technical debt where content developers have stacked ever more complicated gameplay logic to satisfy content release schedules. Iirc, DDO uses some old exotic scripting engine, it might be very opaque which parts of the game causes it to lag. Basically like a browser with 100 tabs and no task manager.

[TitusOvid]

The LAG invites to all kind of speculations. I am so tired of these though. I want to know

a) if the last actions helped and they are happy with the state of lag at the moment?
b) if they do need more time and data to plan the next step?
c) if there is a next step planned already?
d) what it will be?
e) is there a way to help?
f) if they do know it is still horrible to play at times?
g) ...
...
z) did they throw in the towel in the end?

This is not some kind of big secret. It is just information that a customer is interested in because it affects the product.

And I am so tired of begging for information on this.


Titus.

[Torc]

The LAG invites to all kind of speculations. I am so tired of these though. I want to know

a) if the last actions helped and they are happy with the state of lag at the moment?
b) if they do need more time and data to plan the next step?
c) if there is a next step planned already?
d) what it will be?
e) is there a way to help?
f) if they do know it is still horrible to play at times?
g) ...
...
z) did they throw in the towel in the end?

This is not some kind of big secret. It is just information that a customer is interested in because it affects the product.

And I am so tired of begging for information on this.


Titus.

a> Some of the work, namely the double strike rework does seem to help one form of lag commonly encountered in raids. It is not the only lag issue we have, and no we are not happy with the overall situation as of yet.

b> We have many other "leads" for lack of a better term, to follow up on but we prefer to only communicate things we're relatively confident will have verifiable results. So yes, we do need time to verify some of them.

c> Several initiatives are in various states of planning or are on going

d> Until some of these things are done enough or proven out we'd rather not get into them yet. Likely one if not several more changes will appear in the U51 previews coming up. There is one exception to this I can get into right now which is "in progress". See below.

e> Giving us reproduceable steps that appear to cause the problem really helps. In some cases of lag this will be IMPOSSIBLE ...OR.. in the case of client lag impossible for us because we don't use your video card....

However things do come up you can help with. One of them is tracking down character abilities that cause consistent problems...

A good example of a report that was quickly actionable was of a set of AOE abilities that often caused noticeable hitches on use, which turned out to be a problem of bad filtering of objects in the detect logic. These guys were trying to run effect logic on well...dirt and just about everything else (I heal the dirt for 103....) Beacon of Hope was one of these abilities that got addressed back in March or so. It's still quite likely there are more of them out there. Remember the magic word here is "repro". That you can get the hitch to reoccur again and again on demand. Inconsistent stuff, well...we do chase that to when it's all we have, but it's slower going...

f> ..and... Yes, while every user is affected differently our various lag issues are much too common in the play experience right now. We are acutely aware of this. We are working on it.


z> and Not giving up, but hunting gremlins can be very time consuming and some of these issues deal with core systems that are very interconnected and must be adjusted very very carefully least we cause other problems. Sometimes it's slow going which is frustrating, for all involved....


-T

[Chai]

If it was so easy to figure out the lag issues, why would they have needed to bring in external (ex-dev) consultants to come up with the multi-hit change?

Resources.

Time consumption =/= technical difficulty of task.

I still think they just lack investment in profiling tools for figuring this out. Considering they seem to mostly share engine devs with LotRO, the engine side of DDO running the gameplay scripts is probably a neglected pile of technical debt where content developers have stacked ever more complicated gameplay logic to satisfy content release schedules. Iirc, DDO uses some old exotic scripting engine, it might be very opaque which parts of the game causes it to lag. Basically like a browser with 100 tabs and no task manager.

An example of technical debt is part of your first question. That multi-hit change was implemented back in the day as a substitute for doing a physics check on every hit, and the logic was the processing overhead was lower on procs than it is on physics checks. Now either A) they were wrong in this determination, or B) the resources were "upgraded" (downgraded) and now even that lower overhead can't be handled as well as it once was. Some suspect the data center change was a cost saving measure which fits into all this.

The way companies are run now days, the idea that something is "difficult to figure out" is usually the euphemism for "we could totally do this, but what do you want us to stop working on in order to start working on this?" There's only so many points per sprint. Allocating some to this work means reallocating it from something else. Money is made on new features. One could argue the retention of ability to continue to make money is linked to having a stable environment, but this often falls on deaf ears of managers who are not developers or QA engineers, who do not see a specific projected figure on a spreadsheet for this line item. This is highly likely the reason why we often hear of many QoL items being implemented by a dev in their spare time or off time. They understand the importance, but those in charge of time allocation on projects do not, or repeatedly prioritize some other work more directly linked to making money over this type of work.

The direct observable result is the server continues to lag, all the while there's new content to purchase periodically, and new gidgets and gadgets in the store which allow for mitigation of time consumption so you can grind PLs faster. Research stories to figure out sources of lag? Maybe next sprint, or the one after that...etc. Get those new filigrees which are +1 better than the last ones done. There needs to be more reasons to purchase the mini-expansion.

[Gniewomir]

e> Giving us reproduceable steps that appear to cause the problem really helps. In some cases of lag this will be IMPOSSIBLE ...OR.. in the case of client lag impossible for us because we don't use your video card....

Can stuff like laggy "places" cause server wide problems? I'm not talking about quests like wrath of earth, will be the best if i'll use example:

I'm pretty regularly farming wpm with my duals and i've noticed that some very specific places always cause stance wide lags. One of such places is near stalagtite in the middle corridor. If i put my duals on autorun and majority of them will get "stuck" in the same, very specific spot, suddenly whole stance becomes laggy, like really, really laggy, even for characters who are not in this spot. Leaving this spot with one or two characters is causing lags to stop. Same in corner leading to crab, near 1st door. In both cases it's very small spot near wall with no reason for real player go to, so probability of happening to party doesnt exist and for someone using duals and autorun it's still pretty small, but still - at least for me it's causing serious performance problems, just not sure if it could affect server or it's just something only on player side/serverwide meaningless, especially since it's not something that could occur often, more like really rarely. Tbh no idea why i thought about it now, so far i just ignored it, but you kinda triggered me with "reproduceable" steps to summon lags. ;P

[rabidfox]

However things do come up you can help with. One of them is tracking down character abilities that cause consistent problems...

A good example of a report that was quickly actionable was of a set of AOE abilities that often caused noticeable hitches on use, which turned out to be a problem of bad filtering of objects in the detect logic. These guys were trying to run effect logic on well...dirt and just about everything else (I heal the dirt for 103....) Beacon of Hope was one of these abilities that got addressed back in March or so. It's still quite likely there are more of them out there. Remember the magic word here is "repro". That you can get the hitch to reoccur again and again on demand. Inconsistent stuff, well...we do chase that to when it's all we have, but it's slower going...

I saw something in discord the other day about someone's warlock (aura?) buffing nearby objects like summoned bolts in inventory. Seems like that's up there with bad filtering issues.

[LT218]

Your experience is not my experience. (35 years of it.) I have seen dozens of services suffer horribly from virtualization, I have seen businesses destroy themselves with it. Now I am watching SSG do it.

ESPECIALLY high performance udp servers (such as game severs and streaming media servers). Sure you can virtualise a low traffic IIS server and you won't notice a dam thing. VMWare's sales people are not technicians, they have no idea what they are talking about when it comes to SERIOUS servers.

NEXT. (in my best possible a-hole voice)

And your analogy really sucks. It makes no sense at all, and the double spaces after your periods tell me all I need to know, Karen.

But whatever. DDO just doesn't matter anymore.

Modern hypervisors and virtualized servers run at near and in certain cases better than bare metal performance.

It's highly likely anyone who claims that modern virtualization significantly degrades performance doesn't know how to properly configure virtualized environments or is running it on the wrong hardware. There's more to configuring virtualization properly than bare metal for sure, but at the end of the day, if it's done right and it's not over provisioned, the performance impact versus bare metal is negligible and the administration and financial benefits are massive.

There's no telling exactly how SSG's virtualized environment is configured, but given how badly it performed when they first migrated to VMs and how badly it still performs, I'm guessing it is a giant pile of poo.


The fact that you resort to nitpicking a grammatical pattern typically associated with older generations (aka age discrimination) and name-calling in your attempted rebuttal tells everyone else all we need to know about you.

[Dandonk]

a> Some of the work, namely the double strike rework does seem to help one form of lag commonly encountered in raids. It is not the only lag issue we have, and no we are not happy with the overall situation as of yet.

f> ..and... Yes, while every user is affected differently our various lag issues are much too common in the play experience right now. We are acutely aware of this. We are working on it.

Personally I must admit to have, anecdotally, seen lagspikes that are a little shorter in duration than before the doublestrike change. But with all the lag still there, I'm not really happy with it. I had hoped for a much larger impact from this sizeable a change...

leading me to my point, more or less: U50 and U51 are set to make huge changes to the game, and from my point of view. From my point of view, not for the better, but let's leave that for the time being.

These are huge, sweeping changes you're set to make. I've seen you change some surface things here and there, but the basics of the system seems set, as it nearly always is, no matter what we say or think.

Yet, the sweeping changes in the had much less of an effect than I (and you?) hoped - it didn't really do what you wanted... are you sure your next set of sweeping changes will do what you want them to? Because I must admit to having not too much confidence in that.

[TitusOvid]

a> Some of the work, namely the double strike rework does seem to help one form of lag commonly encountered in raids. It is not the only lag issue we have, and no we are not happy with the overall situation as of yet.

b> We have many other "leads" for lack of a better term, to follow up on but we prefer to only communicate things we're relatively confident will have verifiable results. So yes, we do need time to verify some of them.

c> Several initiatives are in various states of planning or are on going

d> Until some of these things are done enough or proven out we'd rather not get into them yet. Likely one if not several more changes will appear in the U51 previews coming up. There is one exception to this I can get into right now which is "in progress". See below.

e> Giving us reproduceable steps that appear to cause the problem really helps. In some cases of lag this will be IMPOSSIBLE ...OR.. in the case of client lag impossible for us because we don't use your video card....

However things do come up you can help with. One of them is tracking down character abilities that cause consistent problems...

A good example of a report that was quickly actionable was of a set of AOE abilities that often caused noticeable hitches on use, which turned out to be a problem of bad filtering of objects in the detect logic. These guys were trying to run effect logic on well...dirt and just about everything else (I heal the dirt for 103....) Beacon of Hope was one of these abilities that got addressed back in March or so. It's still quite likely there are more of them out there. Remember the magic word here is "repro". That you can get the hitch to reoccur again and again on demand. Inconsistent stuff, well...we do chase that to when it's all we have, but it's slower going...

f> ..and... Yes, while every user is affected differently our various lag issues are much too common in the play experience right now. We are acutely aware of this. We are working on it.


z> and Not giving up, but hunting gremlins can be very time consuming and some of these issues deal with core systems that are very interconnected and must be adjusted very very carefully least we cause other problems. Sometimes it's slow going which is frustrating, for all involved....


-T

Thank you very much for a reaction.
Please keep us updated.

Cheers,
Titus.

[Alrik_Fassbauer]

A good example of a report that was quickly actionable was of a set of AOE abilities that often caused noticeable hitches on use, which turned out to be a problem of bad filtering of objects in the detect logic. These guys were trying to run effect logic on well...dirt and just about everything else (I heal the dirt for 103....)

I saw something in discord the other day about someone's warlock (aura?) buffing nearby objects like summoned bolts in inventory. Seems like that's up there with bad filtering issues.

A great ! Finally I have an example as to what to look for / focus on !

[Luthor_Darkhammer]

The business model at this point is to push out as much constant as possible whether the game is working or not

Easy fix for that is everyone group up and agree not to purchase any new content until existing issues are resolved. I know, a pipe dream in this community though.

[Yamato-San]

a> Some of the work, namely the double strike rework does seem to help one form of lag commonly encountered in raids. It is not the only lag issue we have, and no we are not happy with the overall situation as of yet.

b> We have many other "leads" for lack of a better term, to follow up on but we prefer to only communicate things we're relatively confident will have verifiable results. So yes, we do need time to verify some of them.

c> Several initiatives are in various states of planning or are on going

d> Until some of these things are done enough or proven out we'd rather not get into them yet. Likely one if not several more changes will appear in the U51 previews coming up. There is one exception to this I can get into right now which is "in progress". See below.

e> Giving us reproduceable steps that appear to cause the problem really helps. In some cases of lag this will be IMPOSSIBLE ...OR.. in the case of client lag impossible for us because we don't use your video card....

However things do come up you can help with. One of them is tracking down character abilities that cause consistent problems...

A good example of a report that was quickly actionable was of a set of AOE abilities that often caused noticeable hitches on use, which turned out to be a problem of bad filtering of objects in the detect logic. These guys were trying to run effect logic on well...dirt and just about everything else (I heal the dirt for 103....) Beacon of Hope was one of these abilities that got addressed back in March or so. It's still quite likely there are more of them out there. Remember the magic word here is "repro". That you can get the hitch to reoccur again and again on demand. Inconsistent stuff, well...we do chase that to when it's all we have, but it's slower going...

f> ..and... Yes, while every user is affected differently our various lag issues are much too common in the play experience right now. We are acutely aware of this. We are working on it.


z> and Not giving up, but hunting gremlins can be very time consuming and some of these issues deal with core systems that are very interconnected and must be adjusted very very carefully least we cause other problems. Sometimes it's slow going which is frustrating, for all involved....


-T

One the one hand thank you for the response, good to know that you try to fix the issues.
On the other hand your reply is not what I would have wanted to read.

I really find it strange that you only speak of fixes on the application level of the game, meaning changes to the high level scripts/code.
It is obvious that the low level core system itself has serious problems.
Typically, these are some low level functions that are not implemented optimal or design patterns that worked a long time ago but no longer.

I really wish to read that you will look in these issues too.
Sure it's not easy to change decade old legacy code from e.g. single-threaded to multi-threaded or optimize some queue algorithms.
But this is something money can fix in form of specialists who have experience with this hard stuff.
There are guys who have the knowledge how to measure/benchmark the code to find and fix the problems.
It's a one time investment to get rid of the core issues and free all of your developers from these legacy issues.
So no more tinkering with the surface (and changing the game play) but please finally get to the root of the problems.

[gravisrs]

z> (..)hunting gremlins can be very time consuming and some of these issues deal with core systems that are very interconnected and must be adjusted very very carefully least we cause other problems.

Sorry to say that but:

1. You don't have the full team that originally developed this engine (along with net code) so you may struggling from proper architectural-side approach to optimize things. I believe "spaghetti code" isn't well documented.
2. You don't have resources to fully build profiling platform, starting from unit-tests that measure every smallest bit of engine resource, through profiling server that has a real developer's build that produce proper statistics what in this code is really resource-consumig where are bottlenecks etc (I believe Mournlands was kinda this type of profiling server), ending with network tests with artificial user load to measure peak loads and find other bottlenecks on structural level.
3. You stuck with .NET C++ and technical debt this platform introduced in years. Especially at the moment of moving to 64bit applications. It's not a secret that microsoft didn't do their best when creating tools to compile in 64bit environment and every application compiled in 64bit is in fact more resource-consuming, slower and way less optimised than it's 32bit older brothers (but hey, we can waste.. I mean address more memory, right?). Not really your fault TBH but you have to carry it's burden.
4. You don't have many options to upgrade hardware, it's already top tier distributed environment. Total amount of resources this game already consumes hits the glass ceiling even for today's industry standards. You cannot go this path much further. Heck, this game engine (server) should flawlessly run on couple Raspberries PI if properly optimised...