Severlin: Calculate reaper trees enhancements when we login - lag is monstrous

[Valtan]

Orien is toast. Absolute carnage yesterday and Saturday. More than 1000 players entering and exiting reaper mode, game engine constantly calculating which it cannot handle. The design of Reaper trees will go down in history of MMO as a case study for what NOT to do.
Solution: Calculate the reaper points and enhancements when we log-in and keep it across all modes. And give this task to a competent developer.

[Stradivarius]

[kmoustakas]

That gif is me everytime I jump

[Valerianus]

imho

reaper was no mistake. for sure there's something badly and hastily coded in it, but the main issue is that ssg servers (both games) can't handle some hundreds players. every time i read the word "megaserver" "orien megaserver" "players diy megaserver" i giggle and laugh, i don't understand what's mega about having about 600 players on it...it's not mega....and, uh, having problems managing such low pop it makes it not even a proper server.....there's no mega and there's no server in this megaserver thing

[magaiti]

it's BIOY megaserver

[Stradivarius]

imho

reaper was no mistake. for sure there's something badly and hastily coded in it, but the main issue is that ssg servers (both games) can't handle some hundreds players. every time i read the word "megaserver" "orien megaserver" "players diy megaserver" i giggle and laugh, i don't understand what's mega about having about 600 players on it...it's not mega....and, uh, having problems managing such low pop it makes it not even a proper server.....there's no mega and there's no server in this megaserver thing

More closer to 800 players on last night. Not sure if it was a "oh I saw the movie I wonder how my old MMO I quit long time ago is doing?" surge, but it was crazy on Orien last night and UNPLAYABLY LAGGY.

[Wahnsinnig]

It is absolutely ridiculous that DDO servers start to struggle when there is more than 400 players on a server. 400 is nothing.

An MMO game server should handle 4000 players with no problems. There is something seriously wrong with DDO servers.

[Stradivarius]

It is absolutely ridiculous that DDO servers start to struggle when there is more than 400 players on a server. 400 is nothing.

An MMO game server should handle 4000 players with no problems. There is something seriously wrong with DDO servers.

[boredGamer]

Orien is toast. Absolute carnage yesterday and Saturday. More than 1000 players entering and exiting reaper mode, game engine constantly calculating which it cannot handle. The design of Reaper trees will go down in history of MMO as a case study for what NOT to do.
Solution: Calculate the reaper points and enhancements when we log-in and keep it across all modes. And give this task to a competent developer.

I still refuse to believe this is a main source of lag.

Just the calculation part? Is it renegotiating reaper points on every action ?

Sure on zone in there is a few seconds, but that can't possibly be what's causing server wide lag, right? Is it some sort of anti-hacking thing in terms of believing the client? Because otherwise, it seems a relatively simple calculation.

[Valtan]

It is absolutely ridiculous that DDO servers start to struggle when there is more than 400 players on a server. 400 is nothing.

An MMO game server should handle 4000 players with no problems. There is something seriously wrong with DDO servers.

4000? Try something like 10 million in Diablo 4 beta weekend. Granted, I do not know how many servers were virtualised (as the only option was clicking Beta) but there was absolutely zero lag.

[Valtan]

Just the calculation part? Is it renegotiating reaper points on every action ?

This was how the reaper trees were designed initially. Every swing of your weapon and spell cast calculated your reaper enhancements. This was changed to calculate when you enter and exit quest.

Sure on zone in there is a few seconds, but that can't possibly be what's causing server wide lag, right? Is it some sort of anti-hacking thing in terms of believing the client? Because otherwise, it seems a relatively simple calculation.

This is probably exactly how it sounded when they developed this system with no load testing whatsoever. Should have worked fine in their internal test server with a population of 1 and they happily shipped it out. Now the engine is doing the same simple calculation for 1000s of players entering and exiting quests, and it cannot handle that load. How can you prioritise developing mini-xpac and Xpacs when your infrastructure is archaic and incompetent devs design a system like this!

[boredGamer]

This was how the reaper trees were designed initially. Every swing of your weapon and spell cast calculated your reaper enhancements. This was changed to calculate when you enter and exit quest.



This is probably exactly how it sounded when they developed this system with no load testing whatsoever. Should have worked fine in their internal test server with a population of 1 and they happily shipped it out. Now the engine is doing the same simple calculation for 1000s of players entering and exiting quests, and it cannot handle that load. How can you prioritise developing mini-xpac and Xpacs when your infrastructure is archaic and incompetent devs design a system like this!

If it's on every weapon swing, sure, I buy it, but I can't believe it was even designed that way at all. Even if it was, was it really communicating to the server every swing or just calculating locally? Because locally it shouldn't be a problem at all.

If it's just coming into / out of quests, computers are very good at adding 4*50+8*50, and you're not making many calls, so I don't get it still.

[ahpook]

If it's on every weapon swing, sure, I buy it, but I can't believe it was even designed that way at all. Even if it was, was it really communicating to the server every swing or just calculating locally? Because locally it shouldn't be a problem at all.

If it's just coming into / out of quests, computers are very good at adding 4*50+8*50, and you're not making many calls, so I don't get it still.

Likely it is also an issue with getting the users reaper config before doing the simple math. Since they have trouble getting data from the bank it seems likely that they may also struggle to get the reaper config. (which maybe why they are talking about a non-configurable reaper tree - Eliminate the need to fetch each users data). That is not to say that such a fetch should be an problem but ...

[Eantarus]

It is absolutely ridiculous that DDO servers start to struggle when there is more than 400 players on a server. 400 is nothing.

An MMO game server should handle 4000 players with no problems. There is something seriously wrong with DDO servers.

Its the very poorly thought-out transition from hardware to virtual servers. Game hasn't been the same since.

[o2t4f]

As previously explained many times by forum experts, this is not caused by reaper trees, server population, or anything server-side. It is clearly on the client side, as I have never lagged and haven't perceived any lag increase, as they say, since free transfers.

Please detail your machine specifications, its color, the phase of the moon when you bought it, and whether you play sitting down or standing up. If you play sitting down, please give a thorough description of what you sit on and try playing standing up to see if your lag issue resolves itself.

[Valtan]

If it's on every weapon swing, sure, I buy it, but I can't believe it was even designed that way at all.

It was confirmed by devs it was indeed designed this way initially (or Severlin in one of his FridayFour or YT calls, dont remember as it was ages ago).

If it's just coming into / out of quests, computers are very good at adding 4*50+8*50, and you're not making many calls, so I don't get it still.

It is just not what you have mentioned. It is applying enhancements (+4 stats, melee power, PRR/MRR, reflex saves, DCs, ranged power etc. based on number of points) to each player when you enter and the same calculations each player when you exit. Multiply that by 1000 players and then realise the virtual server capacity which is doing million other things at the same time. Either it is a bad coding problem or it is an infrastructure problem.

[mpetrarca]

Reaper was not a mistake, how it was designed was complete incompetence. I was looking forward to Reaper when it was first announced. After several Past Lives Elite was becoming too easy. However, when I saw the Reaper Trees I knew the power creep would offset the added difficulty, just plain stupidity.

[boredGamer]

It was confirmed by devs it was indeed designed this way initially (or Severlin in one of his FridayFour or YT calls, dont remember as it was ages ago).

It is just not what you have mentioned. It is applying enhancements (+4 stats, melee power, PRR/MRR, reflex saves, DCs, ranged power etc. based on number of points) to each player when you enter and the same calculations each player when you exit. Multiply that by 1000 players and then realise the virtual server capacity which is doing million other things at the same time. Either it is a bad coding problem or it is an infrastructure problem.

You're not saying things that make sense, so I'll just move on from this.

[Chai]

Anything that adds that much technical debt to a game is/was a mistake. First it was physics checks, which they used a proc system to mitigate (and resulted in the Dungeon Alert band-aid). Then the procs were causing issues, so they merged alot of them to mitigate (resulting in doublestrike/doubleshot being turned into multipliers as well as imbue dice consolidation).

Now having to calculate the extra credit math problem of reaper tree point allocations on a per action basis is an issue? That means it was an issue the entire time people were using it, and didn't just become an issue in the past few weeks when the ForumDDO™ armchair software experts claimed the apparent overpopulation of one server was the issue.

AKA - it was a mistake.

Since reaper has been the standard difficulty setting for the TR hamster wheel for most of the time it was around post implementation, this isn't going anywhere anytime soon, and will only get worse in its current form as more people use it.

[LurkingVeteran]

Having done my fair share of performance profiling as senior developer at an internet company, these are my 2 cents:

The only thing in DDO that should truly be computationally expensive in the conventional sense is path finding. Especially on multi-level maps. This is the first place I would look for the lag. In the worst case, to find the best path (e.g. A*), you have to enumerate over all possible positions in the map. This needs to be kept under control, and I guess dungeon alert and various hacks where mobs get stuck or ignore pathing were their somewhat ham-fisted attempts at this. Other MMOs of the era did not have these issues I recall, so probably there is a better solution (e.g. hierarchical path finding, or give mobs bows?).

The second place I would look is the database layer, since to maintain a consistent and permanent state of players, a database server is usually used, which has to be queried over a network and access storage (=expensive) every time you pick up an item or do anything permanent (and sometimes even transient things like hp changes, but that would be stupid). Often, one DB cluster serves an entire server or game, which means they become a bottleneck and can also cause cross-instance and cross-server lag for extra fun. This was the reason WoW was a slideshow for some weeks upon release. Monitor the response times over time for the DB layer. It's not enough to just check CPU usage as it can be io bound (storage, network).

The third most likely place I would look is the scripting engine. DDO has a seriously arcane one (can't remember it's name now) that they seem to use for everything. The scripts seem to be the one thing they have focused on the most with all their minor changes to doubleshot etc. However, this is just what you feed into the engine itself.

I find it a little difficult to believe that reaper scripts can be so slow that adding up a few numbers for reaper bonuses at log in is a problem. Ok, very difficult to believe. A modern CPU can do a billion operations per second, and I'm pretty sure each 400 player server is a cluster of computers. This does not add up unless they have some very weird design of their scripting layer. I wonder if they have ever tried running the scripting engine itself through a modern profiler. It wouldn't surprise me if nobody knows how the scripting engine works anymore (perhaps they don't even have the source, or nobody can code C++), so they do all their lag fixing by trial and error on the scripts instead. It wouldn't be the first company in that situation :-)

Finally, the fourth most likely place is the network load balancing or bandwidth. This can be a bit tricky to spot, and doesn't necessarily have to be between the game servers and the internet, but could also be between the database layer and the game servers. The effect of this can cascade to other parts of the system. Always monitor all parts of the network and server response times. Been there, done that.