Lag on Orien. Can we have an ETA about performance upgrade?

[Fanaval]

Greetings,

as Cordovan said in Friday streaming SSG will perform performance upgrade that will improve lag on Orien.

Can we have an ETA (estimated time on arrival) about this performance upgrade?

I think that lag should be a priority for the devs to avoid frustrating players an lose revenue.

Thanks.

[Fanaval]

It appears that the game performances will be improved soon as Cordovan said in this post:
https://forums.ddo.com/forums/showth...-for-Update-59

[holmgang]

Lag is bad yes, but the last couple of days trying to log out and switch toons has been an exercise in frustration.

It's actually quicker to quit the game and fire up the client again than it is to log out to the char select screen.

Last night I clicked log out, went to the bathroom, came back and the same toon was still on the screen.

3 minutes expired, at least.

Anyone else seeing this?

[Phoenicis]

Lag is bad yes, but the last couple of days trying to log out and switch toons has been an exercise in frustration.

It's actually quicker to quit the game and fire up the client again than it is to log out to the char select screen.

Last night I clicked log out, went to the bathroom, came back and the same toon was still on the screen.

3 minutes expired, at least.

Anyone else seeing this?

I've been seeing this for so long I stopped using the logout button years ago.

[rabidfox]

The reason Severlin gave us for not having a Megaserver was the inability to move guild and their amenities in 2019. Then he changed the reason to "lag" which was due to legacy code. Never once he said it was due to population in a server. So why is the lag so bad in Orien suddenly?

About 50:20 in with Sev. where talks about technical limitations of things last year for megaservers (it's beyond moving guild/amenities issues):

but uh so remember when i talked about mega servers
50:39
it ran into we ran into a problem where there are some limitations on what we call our master
50:45
server that um that kind of keeps track of all of the server
50:50
uh executables in a specific shard when it's a shard word uh world that
50:55
when we looked at doing the mega server technology there was significant work there and so um that that was what held us up
51:03
uh in the back end um and uh that is not an easy thing to solve so
51:08
if you're wondering what happened with that we just hit we just hit uh functional limitations
51:14
for how many how many shards could be attached to a single master server
https://www.youtube.com/live/i8SI0u2...e=share&t=3018

[Memnir]

Soon™

[Assassination]

Fix the Lag on Orien SSG, or we "the players" will be leaving. Thank you.

[draven1]

Fix the Lag on Orien SSG, or we "the players" will be leaving. Thank you.

That's it! Leaving is THE FIX!

It will reduce the lag.

[LightBear]

About 50:20 in with Sev. where talks about technical limitations of things last year for megaservers (it's beyond moving guild/amenities issues):

but uh so remember when i talked about mega servers
50:39
it ran into we ran into a problem where there are some limitations on what we call our master
50:45
server that um that kind of keeps track of all of the server
50:50
uh executables in a specific shard when it's a shard word uh world that
50:55
when we looked at doing the mega server technology there was significant work there and so um that that was what held us up
51:03
uh in the back end um and uh that is not an easy thing to solve so
51:08
if you're wondering what happened with that we just hit we just hit uh functional limitations
51:14
for how many how many shards could be attached to a single master server
https://www.youtube.com/live/i8SI0u2...e=share&t=3018

Yeah, interesting what he means there with shards. Is it a threadpool of some sort?
See this question as per example discussion:
https://stackoverflow.com/questions/...ication-create

There are limitations to I/O-, CPU- and Memory-bound things for sure.

[Valtan]

About 50:20 in with Sev. where talks about technical limitations of things last year for megaservers (it's beyond moving guild/amenities issues):

but uh so remember when i talked about mega servers
50:39
it ran into we ran into a problem where there are some limitations on what we call our master
50:45
server that um that kind of keeps track of all of the server
50:50
uh executables in a specific shard when it's a shard word uh world that
50:55
when we looked at doing the mega server technology there was significant work there and so um that that was what held us up
51:03
uh in the back end um and uh that is not an easy thing to solve so
51:08
if you're wondering what happened with that we just hit we just hit uh functional limitations
51:14
for how many how many shards could be attached to a single master server
https://www.youtube.com/live/i8SI0u2...e=share&t=3018

ELI5 - is this a coding issue or an infrastructure issue? If it is an infrastructure based, will moving to a publisher, like Amazon game services, solve it?

[magaiti]

I think that lag should be a priority for the devs to avoid frustrating players an lose revenue.

Not until they get all the revenue from paid transfers off Orien.

[DarkSkysz]

You had to see Sharn last night...

SO many people stuck near the elevator, trying to enter the quest, trying to move and none could do anything because the server was so laggy.

[Loriega]

About 50:20 in with Sev. where talks about technical limitations of things last year for megaservers (it's beyond moving guild/amenities issues):

but uh so remember when i talked about mega servers
50:39
it ran into we ran into a problem where there are some limitations on what we call our master
50:45
server that um that kind of keeps track of all of the server
50:50
uh executables in a specific shard when it's a shard word uh world that
50:55
when we looked at doing the mega server technology there was significant work there and so um that that was what held us up
51:03
uh in the back end um and uh that is not an easy thing to solve so
51:08
if you're wondering what happened with that we just hit we just hit uh functional limitations
51:14
for how many how many shards could be attached to a single master server
https://www.youtube.com/live/i8SI0u2...e=share&t=3018

That uh uhm *looks at watch* says pretty uh much uhm nothing uh, maybe uh uhm go into details uh uhm ugh

[archest]

apparently your playing the game wrong. you should be running past everything and just kill the last guy and complete the mission, if you take too long by stopping and killing other stuff you cause lag for everyone.

[Eantarus]

apparently your playing the game wrong. you should be running past everything and just kill the last guy and complete the mission, if you take too long by stopping and killing other stuff you cause lag for everyone.

Don't forget you're playing the game wrong by having items in your shared bank or on mules. Having items causes lag.

[axiom21]

ELI5 - is this a coding issue or an infrastructure issue? If it is an infrastructure based, will moving to a publisher, like Amazon game services, solve it?

80% Code, 20% Infrastructure.

Moving to AWS or otherwise will not do much unless the entire systems architecture and orchestration was re-created, which would still only do so much, DDO is already largely virtualized with one massive monolith holding virtualized game servers.

FYSA Amazon Games & Luna are largely built on AWS services, notably GameLift and Lumberyard (which would be out of scope) and GameLift is just a repackaged version of ADS/SMS/DMS. All of that AWS-specific **** aside to say it will move OS & containers to comparable virtualized hardware with all of the CPU, GPU, RAM, IOPS, and Storage you could want which doesn't do much when your underlying application is a giant mess of IO-bound and non-parallelizable CPU-bound ****.

A lot of the codebase and how it interacts with Client & Server-side DATs and how the different subsystems (from an outside-in perspective) are intertwined and how much **** needs to be persisted into memory per-character per-account per-server basis, well, that is not exactly easy to change and at times when it has changed there have been less-than-desirable effects.

This is also what crushes performance at scale. At any given second the codebase is looking up, storing, calculating, and ticking-down against likely 100s or 1000s of discrete bits of logic across all of the dependencies on a character.

Just think of how many buffs are contained, how it applies to overall stats, how it changes things like HP/SP/AC/PRR/MRR/Dodge/etc. and then how the checks on stacking stats, debuffs, ticking down damage, physics, collisions, etc. on top of persisting your Reaper trees, enhancements, feats, stances, and so on...that is a lot of calculations happening all at once across dozens of buffs before adding in things like procs, temporary effects. Now multiply that mathematical ****-show by X amount of characters/systems/mobs/instances/servers and you have a peek into the nascent issues.

[Valtan]

80% Code, 20% Infrastructure.

Just think of how many buffs are contained, how it applies to overall stats, how it changes things like HP/SP/AC/PRR/MRR/Dodge/etc. and then how the checks on stacking stats, debuffs, ticking down damage, physics, collisions, etc. on top of persisting your Reaper trees, enhancements, feats, stances, and so on...that is a lot of calculations happening all at once across dozens of buffs before adding in things like procs, temporary effects. Now multiply that mathematical ****-show by X amount of characters/systems/mobs/instances/servers and you have a peek into the nascent issues.

Thank you for the explanation. When people asked Eladrin (previous dev) when he introduced PRR/MRR and AC changes how this will impact lag he answered that it will cause a "few extra CPU cycles" but wont have major impact on lag. What a liar (or he didnt have much clue).

Seems only DDO2 will help but they dont have the skillset for that, do they?

[Chai]

This is not a population issue as those up in their schadenfreude would suggest. This is an issue of unwillingness to modernize. Still limited to just shards? No mapping independant of shard limitations? Now before the armchair developers (not the actual developers) tell us all the things we already know with all the buzzwords from previous eras of posting, the idea that the game was released in 2006 and developed in the early 2000s doesn't mean they are limited to ONLY using the tech and db types from that era to host it. EQ was developed in the mid 90s and released in 1999 but even they have instance mapping. I know DDO uses a version of this, as this is what allows the game to spawn extra duplicate maps of popular zones (marketplace1, marketplace2 etc).

Feeling like they are limited to the shard db system is a major issue. They are parent companied with Daybreak/Fippy now, a company that has gone throgh db infrastructure improvements to move closer to the modern era.

[magaiti]

Just think of how many buffs are contained, how it applies to overall stats, how it changes things like HP/SP/AC/PRR/MRR/Dodge/etc. and then how the checks on stacking stats, debuffs, ticking down damage, physics, collisions, etc. on top of persisting your Reaper trees, enhancements, feats, stances, and so on...that is a lot of calculations happening all at once across dozens of buffs before adding in things like procs, temporary effects. Now multiply that mathematical ****-show by X amount of characters/systems/mobs/instances/servers and you have a peek into the nascent issues.

If every time a game needs to know value of a skill, it has to recalculate buffs/stats/reaper enhancements, then this is just a poor, horribly unoptimised code, not a scaling/infrastructure problem.
It may be a rigid design problem, and then we are screwed unless someone invests millions into making DDO2.

[KoobTheProud]

If every time a game needs to know value of a skill, it has to recalculate buffs/stats/reaper enhancements, then this is just a poor, horribly unoptimised code, not a scaling/infrastructure problem.
It may be a rigid design problem, and then we are screwed unless someone invests millions into making DDO2.

The early Turbine devs had some very interesting gaps in combined knowledge. Pathing in Asheron's Call was a nightmare after the process had been solved at the MMO level in UO and EQ. There was also tremendous lag at times as mobs rubberbanded all over the landscape. The game itself was enjoyable on balance but it had almost amateurish gaps in the coding.

My guess would be that the DDO early efforts borrowed heavily from the "expertise" of the AC staff and that explains why pathing and lag have always been problems in DDO.