Lamannia Update 59 Preview 1 Feedback: Performance Adjustments

[Cordovan]

We have fundamentally changed how bonuses are calculated behind the scenes to be using a more performant algorithm to reduce lag in the game. This change impacts:



Feat bonuses
Enhancement bonuses
Epic destiny bonuses
Past life bonuses
Toggled bonuses
Temporarily granted Feats
Armor Class bonuses
Blocking Armor Class bonuses
Attack bonuses
Damage bonuses
Savings throw bonuses


Please keep an eye out for discrepancies in bonuses on your character sheet between the regular Live game worlds and this Lamannia build. Adding and removing Feats, Enhancements, and acquisition of Past Lives are the primary mechanisms that might cause these bonuses to exhibit erroneous behaviors. There are more than 485 bonuses affected and not all of them are on the character sheet and easily displayable. So if character abilities are not acting as you would expect them to normally, please report your experience here!

[GoldyGopher]

We have fundamentally changed how bonuses are calculated behind the scenes to be using a more performant algorithm to reduce lag in the game. This change impacts:



Feat bonuses
Enhancement bonuses
Epic destiny bonuses
Past life bonuses
Toggled bonuses
Temporarily granted Feats
Armor Class bonuses
Blocking Armor Class bonuses
Attack bonuses
Damage bonuses
Savings throw bonuses


Please keep an eye out for discrepancies in bonuses on your character sheet between the regular Live game worlds and this Lamannia build. Adding and removing Feats, Enhancements, and acquisition of Past Lives are the primary mechanisms that might cause these bonuses to exhibit erroneous behaviors. There are more than 485 bonuses affected and not all of them are on the character sheet and easily displayable. So if character abilities are not acting as you would expect them to normally, please report your experience here!

Would it be possible to understand how this fundamentally changed and if there are potential interactions (enter or leaving a quest, shrining, buffing, other) that have a greater potential to cause discrepancies.

[LeoLionxxx]

Would it be possible to understand how this fundamentally changed and if there are potential interactions (enter or leaving a quest, shrining, buffing, other) that have a greater potential to cause discrepancies.

Indeed, if we get a technical info dump, it may help with looking for issues and thinking of and testing edge cases.

[Ereshkigal]

We have fundamentally changed how bonuses are calculated behind the scenes to be using a more performant algorithm to reduce lag in the game. This change impacts:



Feat bonuses
Enhancement bonuses
Epic destiny bonuses
Past life bonuses
Toggled bonuses
Temporarily granted Feats
Armor Class bonuses
Blocking Armor Class bonuses
Attack bonuses
Damage bonuses
Savings throw bonuses


Please keep an eye out for discrepancies in bonuses on your character sheet between the regular Live game worlds and this Lamannia build. Adding and removing Feats, Enhancements, and acquisition of Past Lives are the primary mechanisms that might cause these bonuses to exhibit erroneous behaviors. There are more than 485 bonuses affected and not all of them are on the character sheet and easily displayable. So if character abilities are not acting as you would expect them to normally, please report your experience here!

Lol.

It might be possible for you to have been more vague, but this is super vague. What have you done now? It's a fundamental change? Ok... Is there some reason you're not just telling us what you changed?

[slarden]

Would it be possible to understand how this fundamentally changed and if there are potential interactions (enter or leaving a quest, shrining, buffing, other) that have a greater potential to cause discrepancies.

The first easy test is to log on your live server, note the key stats unbuffed. Do the same going into a quest. Do the same fully buffed and maybe even with a dual box bard.

Then repeat on Lamannia.

On a complex capped character I would be impressed if it matches fully. The things listed by Cordovan can be looked at if you get exact matches, otherwise I probably wouldn't even consider a deeper dive if the starting numbers aren't right.

[Baahb3]

Dumb question I guess but are you expecting all numbers to be the same after these changes? i.e. This is just a change to the way they are calculated to get to the same total. Not a change in the actual values?

[mikarddo]

I logged in my maxed bard, in reaper with songs, on Orien and Lam.

Same stats all around except for spell points which was slightly off by 80 points. Not sure if that could be for some other reason, I didnt try to figure it out.

[Cordovan]

Dumb question I guess but are you expecting all numbers to be the same after these changes? i.e. This is just a change to the way they are calculated to get to the same total. Not a change in the actual values?

Correct.

[Aelonwy]

So? We should take snap shots of ALL our characters on live, import them to Lamannia... see if any numbers are off and try to figure out why? *sigh*

[Hawkwier]

Correct.

This presupposes all the values on live are correct.

(For example, I don't believe my HP are correctly calculated from the Heroic Durability and Combat style feats.)

It may be possible that the Lammania rework actually corrects an existing error on Live.

Hopefully, any difference will be examined objectively without presumption that Lammania values must be "wrong".

[Cordovan]

So? We should take snap shots of ALL our characters on live, import them to Lamannia... see if any numbers are off and try to figure out why? *sigh*

Mostly, just play on Lamannia and see what you recognize as different, whether it be numbers or something else.

[ahpook]

The first easy test is to log on your live server, note the key stats unbuffed. Do the same going into a quest. Do the same fully buffed and maybe even with a dual box bard.

Then repeat on Lamannia.

On a complex capped character I would be impressed if it matches fully. The things listed by Cordovan can be looked at if you get exact matches, otherwise I probably wouldn't even consider a deeper dive if the starting numbers aren't right.

If it were that easy. The challenge is taking all those numbers you can see to see if they are properly applied during your combat (dmg, tohit, strikethrough, prr, etc). You know, those numbers that we can only tell are off after playing our characters for extended periods of time due to the RNG.

[Nugaot]

This presupposes all the values on live are correct.

(For example, I don't believe my HP are correctly calculated from the Heroic Durability and Combat style feats.)

It may be possible that the Lammania rework actually corrects an existing error on Live.

Hopefully, any difference will be examined objectively without presumption that Lammania values must be "wrong".

This is my question too, there's probably more than a few edge cases where calculations aren't adding up correctly on live. Is this a "we fixed every stat calculation bug in the game" type fundamental change?

[Scrag]

This is my question too, there's probably more than a few edge cases where calculations aren't adding up correctly on live. Is this a "we fixed every stat calculation bug in the game" type fundamental change?

Fwiw, I have seen my max hp fluctuate by a constant 12 points. Zero effects, no gear swapping. Just zoning between areas, and suddenly I gain or lose 12 points. Just doing travel. I tried to figure out what was going on but ultimately said 12 hps isn't likely to slay me.

[Bjond]

Would it be possible to understand how this fundamentally changed and if there are potential interactions (enter or leaving a quest, shrining, buffing, other) that have a greater potential to cause discrepancies.

My guess is they're lifting invariant calculations as high as they could reasonably go. Worst case, they were fully recalculating every single bonus on every single attack. Best case for us is that they're changing it to precalculating as much as possible as soon as they can.

The difficulty with something like that is D&D itself with temporary buffs and effects combined with code age; ie. the more varied and different the scope of each calculation, the more work it takes to do this. This after-the-fact effort scales up a lot with older code and is a big reason why the phrase "we'll optimize it later" usually implies later = never.

Example: you can calc attribute bonus on each attack or you can precalculate a bonus and update that bonus only when things it's derived from change. It's MUCH more efficient to precalc, but to do that efficiently in code after it's already in place requires refactoring all code that makes attribute changes. All the little quick hacks and updates over the years that directly change attributes would need to be updated. You'd need to make SURE all attribute-changing code runs the "change an attribute" function, which would then also update any derived bonuses.

All that is just an example based on a guess, but it fits with typical code optimization efforts on older code.

Things to check: make sure when your character's stats & bonuses change, that it really changes. Do negative levels apply? When you remove them is your character FULLY restored or do you need to relog to fix it? My guess is they'll get gearing, feats, and trees mostly correct, because they can just run down each feat/tree slot/gear slot and test them -- there's effectively a test list for those.

Temporary and obscure bonuses would be easier to miss. Gear with unusual and/or unique effects. De/buffs applied and removed in combat. Gear-swapping in combat. Stuff like is likely not easily converted into a checklist to test. The more obscure, the more likely it's gonna slip by and either not work or require you to relog to fix. Try swapping LOTS of gear mid combat if you like to run swappy builds (my bet is gear-chaning will be slower to MUCH slower).

I sincerely hope they're doing rigorous testing on everything they can for this one because just saying "come play and let us know" is going to result in the discovery and use of bugs that benefit the players. And, this kind of patch has serious risks of accumulating either negative or positive bonuses over time. It's a good thing to do, but it badly needs VERY thorough testing.

[CodogCuisinart]

I logged in my maxed bard, in reaper with songs, on Orien and Lam.

Same stats all around except for spell points which was slightly off by 80 points. Not sure if that could be for some other reason, I didn't try to figure it out.

Please PM me with your character name and I'll take a look. This is the kind of discrepancy I'm looking for. Thank you!

[CodogCuisinart]

My guess is they're lifting invariant calculations as high as they could reasonably go. Worst case, they were fully recalculating every single bonus on every single attack. Best case for us is that they're changing it to precalculating as much as possible as soon as they can.

The difficulty with something like that is D&D itself with temporary buffs and effects combined with code age; ie. the more varied and different the scope of each calculation, the more work it takes to do this. This after-the-fact effort scales up a lot with older code and is a big reason why the phrase "we'll optimize it later" usually implies later = never.

Example: you can calc attribute bonus on each attack or you can precalculate a bonus and update that bonus only when things it's derived from change. It's MUCH more efficient to precalc, but to do that efficiently in code after it's already in place requires refactoring all code that makes attribute changes. All the little quick hacks and updates over the years that directly change attributes would need to be updated. You'd need to make SURE all attribute-changing code runs the "change an attribute" function, which would then also update any derived bonuses.

All that is just an example based on a guess, but it fits with typical code optimization efforts on older code.

Things to check: make sure when your character's stats & bonuses change, that it really changes. Do negative levels apply? When you remove them is your character FULLY restored or do you need to relog to fix it? My guess is they'll get gearing, feats, and trees mostly correct, because they can just run down each feat/tree slot/gear slot and test them -- there's effectively a test list for those.

Temporary and obscure bonuses would be easier to miss. Gear with unusual and/or unique effects. De/buffs applied and removed in combat. Gear-swapping in combat. Stuff like is likely not easily converted into a checklist to test. The more obscure, the more likely it's gonna slip by and either not work or require you to relog to fix. Try swapping LOTS of gear mid combat if you like to run swappy builds (my bet is gear-chaning will be slower to MUCH slower).

I sincerely hope they're doing rigorous testing on everything they can for this one because just saying "come play and let us know" is going to result in the discovery and use of bugs that benefit the players. And, this kind of patch has serious risks of accumulating either negative or positive bonuses over time. It's a good thing to do, but it badly needs VERY thorough testing.

I wrote the new algorithm and tested against the old in our development environment for quite a long time. We optimized how we pre-cache bonuses granted by feats. Giving too much information could tip off exploiters on where to look for vulnerabilities. When changing the underpinnings of a game that has been in development for close to 20 years, there are a lot of special corner cases in the code itself... and then there is a lot of content on top of it that sometimes will surprise you. I had many moments of "We used a feat for THAT? We gave you that feat HOW?" Being that a huge part of our game is our build meta, the sheer number of permutations and combinations of builds and interactions between bonuses makes it impossible to brute force test everything. So I appreciate all your observations and hope to find and fix any discrepancies that come up.

Best wishes!

[M.ham]

We have fundamentally changed how bonuses are calculated behind the scenes to be using a more performant algorithm to reduce lag in the game... Please keep an eye out for discrepancies in bonuses on your character sheet between the regular Live game worlds and this Lamannia build... So if character abilities are not acting as you would expect them to normally, please report your experience here!

Transferred a character to Lamannia, after Heroic TR I observed that I was missing the Universal AP point.

Cheers,
M.

[CodogCuisinart]

In other words, "We changed some stuff and we don't want to be be forthcoming with information. Hopefully you won't notice anything. But if you do, please be forthcoming with information and tell US."

Yeah, that sounds fair. :rolleyes:

We changed some algorithms to be more optimized and need feedback if anything big broke in the process. This was a fundamental rewrite to a delicate system hoping to have the exact same result minus the lag. So we "changed stuff" hoping it didn't change stuff. :P We'd like to be able to address problems before the release as opposed to turning off the lag fix post-release if we find a show stopper. So please kindly report discrepancies if you see any.

Thanks for caring about our game!

[CodogCuisinart]

https://media.giphy.com/media/eGdbckdgqg71HupmXS/giphy.gif


I'm Codog Cuisinart!

I started at Turbine in 2004 and worked on Dungeons and Dragons Online from incubation to the launch of DDO Unlimited when the game turned free to play. I recently (2 years ago) rejoined the Standing Stone Games crew to work on both DDO and Lord of the Rings Online functioning as lead engineer. It appears my title on the forums was not set up correctly since I don't tend to post on the forums. My apologies if this seemed like a clever phishing scam. :)

:cool:

[mikarddo]

I'm Codog Cuisinart!

I started at Turbine in 2004 and worked on Dungeons and Dragons Online from incubation to the launch of DDO Unlimited when the game turned free to play. I recently (2 years ago) rejoined the Standing Stone Games crew to work on both DDO and Lord of the Rings Online functioning as lead engineer. It appears my title on the forums was not set up correctly since I don't tend to post on the forums. My apologies if this seemed like a clever phishing scam. :)

:cool:

Well, we will believe you - as soon as you post using an official account :)

[CodogCuisinart]

Would it be possible to understand how this fundamentally changed and if there are potential interactions (enter or leaving a quest, shrining, buffing, other) that have a greater potential to cause discrepancies.

Reincarnation and past lives have me a little worried since the test cycle on those takes a lot of time and have a kind of funky flow behind the scenes.
Feats temporarily granted by effects is another area of worry for me especial with disconnects and logouts stewn in there.

I'd try different load outs with different enhancements and different epic destiny abilities to kind of get the maximum coverage of the types of bonuses we are looking to exercise. It sounds like we might have a bug in max spell point bonuses not giving enough bonuses in some cases, so a little focus on that would be helpful. One of the beauties of our game is that we have a robust build meta. The interactions between bonuses has so many permutations and combinations that it is impossible to brute force test it.

I'll be clear that we were just looking to fix lag, not reswizzle any stats or balance them.

[CodogCuisinart]

Well, we will believe you - as soon as you post using an official account :)

As well you should!

[Firebreed]

Happy to report that everything looks good on my main in terms of numbers (in the character tab and all its sub-tabs). What I didn't (and will not) check is if things are actually applying properly through the combat log. Also please note that I didn't Reincarnate while on Lamannia.

[Firebreed]

Transferred a character to Lamannia, after Heroic TR I observed that I was missing the Universal AP point.

Cheers,
M.

There's another post saying the same thing in another thread. (https://forums.ddo.com/forums/showthread.php/538079-Lamannia-Update-59-Preview-1-NOW-OPEN/page2?p=6579653#post6579653)

[Aelonwy]

I took pics of my main character sheet and + tab, my skills, enhancements, epic destinies, and compared to Lamannia. On Lamannia I have 80 sp points more I have no idea why and 5% more spell crit chance, ALL spell types have 5% more. Again no idea why. I did NOT change anything or equip anything else. I took pics from live and copied to Lamma and compared immediately.

[mikarddo]

As well you should!

Cheers :)

I took a closer look. The difference is 80 spell points. I have 80 more on Lam than on Orien. This is on my bard Mikarto if you want to further inspect.

I then looked at the numbers by mousing over spell points. The difference is from the line Feats.

I suspected a bugged Magical Training. I have that from the first core in the Draconic Epic Destiny. I tried to reset and reapply the points on Orien (where I have 80 spell points less). This didnt have any effect - still 80 less spell points on Orien than on Lam.

If you need me to post screen shots or test anything else just let me know. If you have the ability to simply log in my character on both servers you are welcome to do so as that would likely make for faster investigation.

https://imgur.com/a/HwscoiZ.jpg

[mikarddo]

I took pics of my main character sheet and + tab, my skills, enhancements, epic destinies, and compared to Lamannia. On Lamannia I have 80 sp points more I have no idea why and 5% more spell crit chance, ALL spell types have 5% more. Again no idea why. I did NOT change anything or equip anything else. I took pics from live and copied to Lamma and compared immediately.

I am seeing exactly the same. +80 spell points and +5% spell crit to all spells.

This fits exactly with Magical Training so maybe that is being applied twice.

[GoldyGopher]

I'm Codog Cuisinart!

I started at Turbine in 2004 and worked on Dungeons and Dragons Online from incubation to the launch of DDO Unlimited when the game turned free to play. I recently (2 years ago) rejoined the Standing Stone Games crew to work on both DDO and Lord of the Rings Online functioning as lead engineer. It appears my title on the forums was not set up correctly since I don't tend to post on the forums. My apologies if this seemed like a clever phishing scam. :)

:cool:

There used to be a developer way back when (2004 - 2011) named Codog who built this amazing Castle bed for his daughter while attempting to fix projectiles and levers.. Any relation?

[mikarddo]

I'm Codog Cuisinart!

I started at Turbine in 2004 and worked on Dungeons and Dragons Online from incubation to the launch of DDO Unlimited when the game turned free to play. I recently (2 years ago) rejoined the Standing Stone Games crew to work on both DDO and Lord of the Rings Online functioning as lead engineer. It appears my title on the forums was not set up correctly since I don't tend to post on the forums. My apologies if this seemed like a clever phishing scam. :)

:cool:

It brings a level of confidence to this endaveaur that I have not had in a while that someone who was actually there at the start and thus knows the old code (in the sense that something like that is even possible).

I am slightly more hopeful that this might improve lag significantly now.

Welcome to the forums :)

[Arkat]

We changed some algorithms to be more optimized and need feedback if anything big broke in the process. This was a fundamental rewrite to a delicate system hoping to have the exact same result minus the lag. So we "changed stuff" hoping it didn't change stuff. :P We'd like to be able to address problems before the release as opposed to turning off the lag fix post-release if we find a show stopper. So please kindly report discrepancies if you see any.

Thanks for caring about our game!

Thanks for the constructive response.

[CodogCuisinart]

There used to be a developer way back when (2004 - 2011) named Codog who built this amazing Castle bed for his daughter while attempting to fix projectiles and levers.. Any relation? That's me!

[Stradivarius]

I'm Codog Cuisinart!

I started at Turbine in 2004 and worked on Dungeons and Dragons Online from incubation to the launch of DDO Unlimited when the game turned free to play. I recently (2 years ago) rejoined the Standing Stone Games crew to work on both DDO and Lord of the Rings Online functioning as lead engineer. It appears my title on the forums was not set up correctly since I don't tend to post on the forums. My apologies if this seemed like a clever phishing scam. :)

:cool:

This was quite the risky move @CodogCuisinart; there's bound to be some unforseeables even through careful dev + player testing for extended periods of time.

But... kudos and congrats for doing it. The lag is far far better (pretty much gone! imho) on Lam which was a fairly laggy server to begin with.

Thanks for being brave! :cool:

[Captain_Wizbang]

That's me!


Well played Codog, well played.




https://media.giphy.com/media/CI6hQq7NgmCqI/giphy.gif

[GoldyGopher]

That's me!

Welcome Back!

[LightBear]

There are more than 485 bonuses affected and not all of them are on the character sheet and easily displayable.

Well, you just gave the solution for your problem there.
Eg, would be super nice if we get some way to get all those 485 bonuses to display on our + tab of the character sheet.
Even if it is just displayed under the section "Unsorted".

I mean, I'm sure some of these will just be displayed as a "You roll a 17, +56..." and that +56 might/could vary on live and on Lamania.
So just a blank comparison is a good start, sure but more tests would be needed.
Like:
Doing a ETR, TR, ITR or LTR.
Reading a tome.
And ofc the actual questing.

I do wonder what they changed, would be nice to get some interview with a dev and ask the technical details about it.
Like, less codestandards but more performance?
How did they measure this and did they try out various code structures to measure how it performed?

Edit: Read some of the responses of Codog, seems we need to test abrupt terminations of the client :(
Also, I read that you're doing C++ Templates, why do we need to run the game to get the results?
Aren't those calculated at compile time instead of runtime?

[LurkingVeteran]

I wrote the new algorithm and tested against the old in our development environment for quite a long time. We optimized how we pre-cache bonuses granted by feats. Giving too much information could tip off exploiters on where to look for vulnerabilities. When changing the underpinnings of a game that has been in development for close to 20 years, there are a lot of special corner cases in the code itself... and then there is a lot of content on top of it that sometimes will surprise you. I had many moments of "We used a feat for THAT? We gave you that feat HOW?" Being that a huge part of our game is our build meta, the sheer number of permutations and combinations of builds and interactions between bonuses makes it impossible to brute force test everything. So I appreciate all your observations and hope to find and fix any discrepancies that come up.

Best wishes!

First, thanks for looking into the lag, and talking about it on the forum! Second, it is a bit surprising that adding points to reaper trees at login can somehow have a noticeable impact on a modern server CPU (simple arithmetic). Additionally, most changes so far have focused on tweaking such scripts to be faster. However, have you tried profiling the (C++?) scripting engine itself to figure out why it is slow? Maybe nobody has remembered to grease the hamster wheel since back in 2006?

[Ordinary]

On my main Maetrim (Monk), I am missing 3 constitution from the 3 Wild Fortitude augments I have (Bracers, ring and gloves). Removing any of these 3 items and then re-adding them does not increase or decrease my Con ability.
This may be down to a bugged ring: I have Wild Fortitude in a Legendary 5 rings. Since bringing the item to Lamannia it has lost its special silver background colour, which it had on my main server.

[Krimer]

On my main Maetrim (Monk), I am missing 3 constitution from the 3 Wild Fortitude augments I have (Bracers, ring and gloves). Removing any of these 3 items and then re-adding them does not increase or decrease my Con ability.
This may be down to a bugged ring: I have Wild Fortitude in a Legendary 5 rings. Since bringing the item to Lamannia it has lost its special silver background colour, which it had on my main server.


This bug has always existed. Try removing the augments and putting the augments back in.

[FURYous]

Here are the differences I found, same toon on Orien and Lamania with the same self buffs:
Orien has less total SP:
Orien:
https://cdn.discordapp.com/attachments/1071793317467144213/1093547106427473940/image.png
Lamania:
https://cdn.discordapp.com/attachments/1071793317467144213/1093550572352839801/image.png

Spell critical chance:
Orien:
https://cdn.discordapp.com/attachments/1071793317467144213/1093547237772103730/image.png
Lamania:
https://cdn.discordapp.com/attachments/1071793317467144213/1093550882118963241/image.png

[FURYous]

Here are the differences I found, same toon on Orien and Lamania with the same self buffs]

I also noticed my will save went up one point on Lamania

[CodogCuisinart]

First, thanks for looking into the lag, and talking about it on the forum! Second, it is a bit surprising that adding points to reaper trees at login can somehow have a noticeable impact on a modern server CPU (simple arithmetic). Additionally, most changes so far have focused on tweaking such scripts to be faster. However, have you tried profiling the (C++?) scripting engine itself to figure out why it is slow? Maybe nobody has remembered to grease the hamster wheel since back in 2006?

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!

[Captain_Wizbang]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!


Good info, thanks. GRATZ on on the family front.

Those of us still here are devout fans. I built a high-end rig to get away from DDO, The irony is, it plays this game so well and almost lag-free, It kept me here. Any changes you all are making to improve playability are greatly appreciated.

My main concern for DDO is dropping the entire reaper enhancement system. (please ditch it) Yes it will irk those heavily invested in it, but IMHO it IS needed. Thoughts?

[FURYous]

Without giving too much detail

Best wishes!

I want to thank you for the communication. This helps all of us understand what is going on and stay on the same page.

Too often we have been left with no communication for days/weeks/months and in that absence, we assume the worst.

Updates and communication as to what is going on will help us to understand that you guys are really there and working to make the game we all love, better.

[FURYous]

Good info, thanks. GRATZ on on the family front.

Those of us still here are devout fans. I built a high-end rig to get away from DDO, The irony is, it plays this game so well and almost lag-free, It kept me here. Any changes you all are making to improve playability are greatly appreciated.

My main concern for DDO is dropping the entire reaper enhancement system. (please ditch it) Yes it will irk those heavily invested in it, but IMHO it IS needed. Thoughts?

I am a network engineer with over 25 years of experience and a degree in computer science.
I have a cutting edge system and when all 12 people in a raid have the exact same lag at the same time, it doesn't point to a client side problem.
When the MMORPG is the ONLY MMORPG that has to schedule weekly reboots, it doesn't point to a client side hardware issue.
When an entire server of players tell you they are having issues, this doesn't point to a client side issue.
Yes, there are some client side issues due to hardware and bandwidth, but the vast majority of the complaints are not about those issues.

Also as far as the Reaper point system, you have to create content/advancement for all your players or they become bored and quit. Getting rid of the reaper point system will gut your player base and leave the game without revenue. Some of us that spend thousands of dollars per year on the game have every past life, done every quest/raid hundreds of times, and rely on the reaper point system as a way to advance in the game. Just because you don't play in that end of the game doesn't justify taking it away from all the people who do.

This doesn't even count the tens of thousands of runes/threads I have spent.
https://cdn.discordapp.com/attachments/1071793317467144213/1093573398627635280/image.png

[GoldyGopher]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.
<SNIP>
Best wishes!

Codog,

As a long time player, I mean was Closed Beta that long ago ;) I truly have always appreciated your candor and upfront nature on the forums. You have always had a way to provide enough information from behind the curtain to satisfy many players questions and concerns. I hope your bosses allow you time to appear occasionally on the forums and peel back another curtain or two. I think that goes a long ways to making happy customers.

Best
Goldy

[Monkey_Archer]

Without giving too much detail... just gave too much detail.

Awesome post. Thank you for this.

[Monkey_Archer]

Also as far as the Reaper point system, you have to create content/advancement for all your players or they become bored and quit. Getting rid of the reaper point system will gut your player base and leave the game without revenue. Some of us that spend thousands of dollars per year on the game have every past life, done every quest/raid hundreds of times, and rely on the reaper point system as a way to advance in the game. Just because you don't play in that end of the game doesn't justify taking it away from all the people who do.

Indeed.
On the bright side, it seems that if reaper points were/are coded as feats, and the feats system gets properly isolated to only iterate on logon/advancement, then the reaper point system *shouldn't* be a cause of lag anyway. Hopefully, in theory ;)

[SavageDoom]

I want to thank you for the communication. This helps all of us understand what is going on and stay on the same page.

Too often we have been left with no communication for days/weeks/months and in that absence, we assume the worst.

Updates and communication as to what is going on will help us to understand that you guys are really there and working to make the game we all love, better.


^This
Finally someone that talks to us and explains things.
Thank you, this is very much appreciated.

[nobodynobody1426]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!

Super thanks, posts like this buy a ton of good will from the community. Those without a technical background think "wow this sounds hard", while those of us with a technical background think "holy moly, this is crazy hard".

[TPICKRELL]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.
...
Thank you so much for actually telling us what is going on!

[Captain_Wizbang]

I am a network engineer with over 25 years of experience and a degree in computer science.
I have a cutting edge system and when all 12 people in a raid have the exact same lag at the same time, it doesn't point to a client side problem.
When the MMORPG is the ONLY MMORPG that has to schedule weekly reboots, it doesn't point to a client side hardware issue.
When an entire server of players tell you they are having issues, this doesn't point to a client side issue.
Yes, there are some client side issues due to hardware and bandwidth, but the vast majority of the complaints are not about those issues.

Also as far as the Reaper point system, you have to create content/advancement for all your players or they become bored and quit. Getting rid of the reaper point system will gut your player base and leave the game without revenue. Some of us that spend thousands of dollars per year on the game have every past life, done every quest/raid hundreds of times, and rely on the reaper point system as a way to advance in the game. Just because you don't play in that end of the game doesn't justify taking it away from all the people who do.

This doesn't even count the tens of thousands of runes/threads I have spent.


Aside from players like you spending gobs of money on the hamster wheel. I to this day will defend the obvious point of a reward system introduced to offset that difficulty setting. Manyplayers fail to connect the 2, THAT is my point. I could care less about your computing quals. No offence.

[TPICKRELL]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.
...

Should have said this above, but didnt think of it right away.

As pointed out above, this post buys you and SSG a ton of good will.

But it also buys you and SSG a lot of credibility. And that's something SSG needs to get back.

This is the one of the few times in recent years that I've seen a post that I believed was forthcoming and conveying as much accurate info as you feel you are able.

Thank You.

[Xgya]

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!

The BIGGEST THANK YOUs for this.

I absolutely always loved getting a peek under DDO's hood, and this has to be one of the biggest insights I've been given yet.
Absolutely loved reading this.

I know you're busy and you can't just hand out these kinds of gifts all the time, but any level of communication from the devs on the forums has become a precious commodity.

Thanks again. Man, reading this gave me a whole other view of the scope of just how much that legacy code needs to be worked with or worked around.

[FURYous]

Aside from players like you spending gobs of money on the hamster wheel. I to this day will defend the obvious point of a reward system introduced to offset that difficulty setting. Manyplayers fail to connect the 2, THAT is my point. I could care less about your computing quals. No offence.

The "computing quals" was in response to your assertation that the lag was client side. This gave the impression that you knew anything about technology.

Secondly "spending gobs of money" is relative. To me it is a small portion of my budget for entertainment. Some people choose to spend on restaurants, some on booze, some on going to the movies, and so on, who are you to make judgement calls on my discretionary spending? Do you have an idea how much one good coder costs? How about the rest of the expenses to keep the game up? Be glad some of us are willing to pay them for the product you enjoy.

[mpetrarca]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!


CodogCuisinart you are the first person at SSG I can say I have respect for, and just for giving a real answer. Not even Cordovan with an Associate's Degree in Broadcasting communicates this well.

So my take is, if they had just stayed true to the 3.5 rule books a lot of the issues would not be around, and yes I liked DDO better when it was D&D.

[Arkat]

I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG.

If only we had been given such a detailed and artful explanation of the problem years ago. A lot of the "bad feelings" on the part of the playerbase could have been mitigated to at least some degree. I would hope that a resulting more understanding playerbase would also mitigate any bad feelings from the Devs towards the playerbase.

Thank you for taking the time to explain this stuff. MANY of us, believe it or not, have technical backgrounds in IT and software development and understand what you're saying to varying degrees.



I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!

Best wishes to you and your familiy.

We are grateful for your communication to us.

We've been waiting for a post like this for a LONG time!

+1!


Regards,

JB

[Arkat]

Not even Cordovan with an Associate's Degree in Broadcasting communicates this well.

To be fair, Cordo is VERY busy. Not only does he have to moderate the DDO Forum, put up the Store Sales, post the weekend bonuses, do all the social media posts, compiling and posting release notes, etc., etc., he also has to do it for all the similar stuff on the LotRO side.

Frankly, he needs help. He may not have the time to get explanations like Codog's post above and then post them on the forum.

I suggested LONG ago that Turbine (now SSG) should hire someone primarily responsible for liaising between the Players and the Devs as well as coordinating SSG internal communications. I called the position a "Communications Czar."

I still think this is a good idea and would allow Cordo to concentrate on "managing the community" per his job title.

[btolson]

Will all this work on feats/enhancements touch on pets as well? Currently they do not gain any benefit whatsoever from augments of any kind, as if the algo that sets their stats just doesn't even try to find them.

[Sam-u-r-eye]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

...

Best wishes!

You too dude.

[Madja]

Thank you very much for that explanation, Codog! You just doubled the reputation of SSG with that post!

One thing I've noticed creates tremendous lag in this game (at least for the individual player) is feather fall. It always makes the character stutter all over. The same thing happens when jumping out of water.
I guess it could be that isn't creating lag, but merely showing how desynced your character is due to existing lag.

Is this something that is being looked into as well?

[Arkat]

Thank you very much for that explanation, Codog! You just doubled the reputation of SSG with that post!

One thing I've noticed creates tremendous lag in this game (at least for the individual player) is feather fall. It always makes the character stutter all over. The same thing happens when jumping out of water.
I guess it could be that isn't creating lag, but merely showing how desynced your character is due to existing lag.

Is this something that is being looked into as well?

I have never experienced this and every toon I have either has a Featherfall item on or Slow Fall (Monks) on at ALL times.

[Natashaelle]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!

This is one of the all-time best forum posts in the entire history of DDO.

When I was a programmer back in the 20th Century, I always tried everything I could to try and future-proof my code for the 21st. I saw to many of my colleagues not making that effort, indeed not even understanding why they should.

As to the house rules I wrote, I realised that proper D&D scaling has to start at level zero with basic stats and end up at levels 50+ but open-ended ; and work fluidly at any possible level, even insane three figure ones.

Short-term solutions create not just long-term problems, but they also put up a brick wall in the medium-term.

I think that's what's preventing WotC developing the current D&D into the Epic and Legendary tiers ... even though the underlying game system is far more capable of it than anything since AD&D 1E.

[Hawkwier]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!

Thank you for taking the time to provide this insight. Much appreciated.

[gravisrs]

An n squared algorithm became n cubed.

I wish I could buy you a beer every time you drop a complexity in code.

N4 to N3, N3 to N2, N2 to linear

Finally someone onboard unravelling the spaghetti code :)

[ChiefVisigoth]

As a retired coder thank you for the explanation. I thought the problem was a bit deep-seated and hard to figure out but seems from your explanation the devs are on the right track. I remember spending 8 hours looking at someone else's code and then spending a half hour fixing the problem. Got to find the problems before you can develop a solution and with a code base as big as the game and as old at it is. It will take time. I started out with computers when there were no hard drives 5/1/4 floppy. No modems and the old sneaker net to get files to someone else. My have times changed. My first stab at learning programming was Fortran on a mainframe.

Never bothered by to time it would take to load that unbounded multidimensional array loop I used to load data elements for a screen in a database app I developed during an internship I did before I finish my BS in systems analysis.. Just remember the reaction of the professor when I laughed about his explanation of arrays in an algorithm course when I said 8 or 10 elements does it really matter when he was discussing a two-dimensional array in Java? I did that with the old Visual Basic language. That took me a week to figure out so I feel for your efforts. Keep up the good work.

My only real question is there a database back end or is it flat files? That is another ball of yarn to figure out besides what you outlined.

[Mindos]

When the MMORPG is the ONLY MMORPG that has to schedule weekly reboots,

Argh! I still think this is mob pathing/barrier related. Maybe a run away value with virtualization. Or maybe too fast polling... Or maybe...

[Mindos]

There used to be a developer way back when (2004 - 2011) named Codog who built this amazing Castle bed for his daughter while attempting to fix projectiles and levers.. Any relation?

It's only 177 posts.
https://forums.ddo.com/forums/member.php/7006-Codog
Click latest posts.


You folks shouldn't worry about me so much! In the morning I get the kids ready for school and walk them to the bus stop. When the kids get home, I help them with their homework, play with them, and sometimes cook dinner ( I make a mean chili! ). After dinner, we spend cuddle time with the kids and read them stories if it hasn't gotten too late. Every night I try to tuck them in to bed. My policy is that when the kids are home and awake try to focus on them.

This week, my wife has been really tired from her work and has been going to bed pretty early. This has left me with some extra time... between that and insomnia none of my girls have been neglected. :)

My folks are coming into town this weekend, so I won't be working.

We're building a castle bed for my older daughter. About nine months ago I bought the plans for it from a company called Tanglewood Designs. http://www.playhousedesigns.com/norwichcastle.html
You start with 8 sheets of MDF (medium density fiber board) and the plans are crazy! We cut the big pieces a few months ago and since then, I've been slowly assembling it, cutting the finer details, and painting it. We're done with 3 of the towers and starting to get close to putting together the 4th. A lot of the trim is complete and ready to be tacked on. Elizabeth is so excited! We _might_ get it completed and ready for final assembly this weekend if we're lucky and it doesn't rain.

Then, all we have to do is work on that puppy! :D

Regards,

Codog

[Lotoc]

When the MMORPG is the ONLY MMORPG that has to schedule weekly reboots,[/IMG]

Most MMOs in my experience schedule weekly downtime/maintenance? However it's typically scheduled for the dead of night when few people will be playing.

[CodogCuisinart]

It's only 177 posts.
https://forums.ddo.com/forums/member.php/7006-Codog
Click latest posts.

Oh wow... Memory lane! Just wow! I'm surprised those posts are still laying around after all this time. Thanks for digging those up! I read a few of those and my eyes started sweating a bit. So many bittersweet memories. It's good to be back and thanks for remembering me.

[Hawkwier]

DDO isn't the only MMO to have regular reboots.

There is one I play which does so daily.

Now, you can argue the toss about whether weekly downtime of several hours is worse than several minutes daily (and FWIW for me it probably is) but statements that DDO is somehow unique in this regard is just misleading.

[magaiti]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!

Thanks for the explanation!

It's reassuring to know there's still people left at SSG who are not afraid to dive into the old code and know a bit or two about computational complexity. May all your control paths end in linearithmic times!

[LightBear]

After falling silent for 15 minutes because what Codog wrote in his last post, looping over "So, I was right all along" until I manage to squeeze out a "I am happy this is finally getting addressed."

[Stradivarius]

--CoDog,

You're a breath of fresh air. Thank you. My only request is to take your time with this, you're delving deep into Moria's code and who knows what you'll awaken when you get all the way down there.

Best Wishes to you and yours!

--Severlin,

I was wrong. I actually think you do care up to a point. I know what your thought process was; "Oh **** this lag is getting real bad, people are (rightfully) complaining." "OK who do hire to fix this..." "Who..." "Oh I know, an old hand who was there when the code was written!" Thank you.

Best of luck for all your future projects!

[mrfantastic1]

I am responding CodogCuisinart, or has his called Shredder, by the tribes of the Eldeen.

Your recent post is honest, informative, and I think most of all HEARTFELT.

I just started a podcast about ddo, in an effort to share the great experience that this game has become.
I find myself sometimes so consumed with passion and slip into a rant that's critical for the sake of being critical.

I have been catching these before they come out. If they do escape my lips, I recognize and apologize,

I wanted to share my perspective honestly as you have done. I offer it as a cosmic payment or karmic return. KEEP IT COMING AND WE WILL TOO!!!


When can their glory fade? O the wild charge they made! All the world wondered.

Honour the charge they made! Honour the Light Brigade, Noble six hundred! (SSG DEVS)

[axiom21]

Without giving too much detail....

You can give the original tech some more credit! The first scripting language - which may or may not be hotdog related ;) - was quite smart, as was the networking stack being able to support 100s or 1000s of connections on UDP nonetheless. The VM, how the calculations were handled, how the DATs are used both client- and server-side were all very neat and ironically enough have some parallels to modern day big data systems. At least for Turbine's other MMOs - I know LOTRO/DDO differed from the 2nd MMO a bit but a lot of that pedigree is in there.

All that aside, it's great to see a post like that, even high-level here. Will it satiate many angry long-time players for longer than this Lam period? Unsure, however, no one can deny anymore that there are devs who really get the macro and the micro and give a ****.

[LurkingVeteran]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!

Thank you for the excellent answer! It restores my confidence that we might actually see lasting improvement on the lag side. I fully understand that the technical debt is hard to deal with after 15 years. I know how it is to have to support a home-grown legacy cloud architecture. The way the game seems to have ended up using feats as internal state and iterating over them, sounds truly unfortunate :-D

Regarding the profiling, you already seem to have a good grasp of the bottlenecks so this might be unnecessary, but it was not clear to me why you couldn't profile the VM engine thread(s) itself if needed.

[Bjond]

too much information could tip off exploiters

It's a serious catch-22. Exploiters and testers literally want to check the exact same things with the only difference being whether they report or use.


when all 12 people in a raid have the exact same lag at the same time, it doesn't point to a client side problem

It could be an underlying synchronization issue instead of a performance/optimization issue, though. I've noticed that most players when joining a group don't impact lag, but now and then there are players that introduce huge lag into quests and raids. They're not actually doing anything other than joining and suddenly lag becomes horrendous for everyone.

I have a tiny hint of a suspicion that a lot of the lag we experience is due to the game trying to synchronize everyone in zone rather than letting only the problem children experience sync issues. IMHO, this is a pure PvE game and thus requires absolutely no interplayer synchronization. Only PvP requires that, at least to some degree. Should be "safe" to just rip out all interplayer sync.

'Course it could also be something as simple as a bad art asset that the game's engine has horrendous difficulty rendering, thus causing everyone to experience lag at the same time due to this one player's cosmetic or play FX.

Given all the problems the game has with sound (it often can't even find sound devices that are working fine), that would be my first pick when looking for synchronized "laggy art".

[Arkat]

My only real question is there a database back end or is it flat files? That is another ball of yarn to figure out besides what you outlined.

There's gotta be a DB back-end. I'm 99% sure that part of what happens when the game goes down for maintenance on some Wednesdays is the DBs get defragged.

[Smokewolf]

There's gotta be a DB back-end. I'm 99% sure that part of what happens when the game goes down for maintenance on some Wednesdays is the DBs get defragged.

I'm pretty sure that "defragmenting" is no longer a thing, as even personal HD's are RAM based. Odds are that server-side storage is considerably more rebust / advanced than what "we" have available. Think, Amazon AWS cloud computing.

Even then the real consideration would be the operating cost. Cheap web hosting and budget friendly servers will use HHD's, as the initial costs are lower. SSD's while obviously costing more, are more reliable and consume less power.

-Smoke

[JOTMON]

~snip~
and not all of them are on the character sheet and easily displayable.
~snip~

.. the glaring question is 'why' we cant easily see them, and is this something you can resolve in future.

Being able to see what is active/not active is indispensable for analyzing and understanding what does and doesn't work, what stacks, overwrites, what bonus bucket buffs actually belong to as descriptions are notoriously inaccurate/incorrect.

[thegreatcthulhu]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!

Thanks and very informative.

And even though I personally never want to get into game development, I understand a little bit of what you are saying based on my own experiences with "regular" software development. Case in point, where you mention that the call stacks go into an external system indeed creates a big challenge. Same with the thing you described about the value caching.

[Arkat]

I'm pretty sure that "defragmenting" is no longer a thing, as even personal HD's are RAM based. Odds are that server-side storage is considerably more rebust / advanced than what "we" have available. Think, Amazon AWS cloud computing.



Defragmenting hard drives and index defragmenting (a.k.a. rebuilding the index) databases are *not* the same thing.

Any DB Admin who's worth anything knows regular index defragmenting their DBs keeps the performance up by reducing DB page counts, even if the DBs are kept on NVMe drives (or even older SSD drives) which, btw, the DDO servers do NOT run on. Their servers still likely run on SAS drives even if the game servers are virtualized.

[rabidfox]

Player Power or Cosmetics as rewards for Reaper is a false dilemma.

Cosmetics are great but you can offer other things besides power as substitutes. I'm thinking notoriety, mostly.

Leaderboards that list who who has done what difficulties for which quests (with time completions to break ties) would be a great way for the ultra-competitive players to show who's the King of the Reaper "hill."

I'm sure their are other ways to incentivize running reaper difficulties.

The speedrun challenge ( https://forums.ddo.com/forums/showthread.php/537916-r10-Duo-Speed-run-Challenge ) going on right now is some of the most fun I've had in a while where it's just about pushing oneself.

SSG could probably do seasonal contests/reward tracks with leaderboard or earn X amount of RXP during a 3-month block, or complete Y number of raids, TR Z times, etc. It could be similar to HC rewards where they reset reward tracks every few months and offer new prizes each time but with everything done on the live servers.

[Captain_Wizbang]

Pointless.
Cosmetics are one off prizes that are selectively appreciated by players, few are desirable most are fugly.
a one off cosmetic (assuming I like the look of it) is a one and done adventure.
Ultimately they are just inventory space killers and the last thing we need is more scrap to overflow an already overburdened inventory.
Cosmetics are good for periodic events like Mabar, cove, Anniversary, hardcore and expansions....


Rewards need to entice players to repeatedly replay content otherwise it is a pointless waste of programming time to to make content no one cares to play.


I am enticed to run content that gives me more power as that opens up the horizon of possibilities to tweak my build and the opportunity to successfully survive and challenge more difficult content.
..Enter the hamster wheel..

bingo

[Ganak]

LOL... just gave too much detail.

Wow, appreciate the detail, even if I couldn't follow it :D


I'm super grateful for the fans that have kept this franchise going

VIP since the start. Started playing at 29, now 46, know I will be playing and loving our game in my 50's and expect to keep going in my 60's+ thanks to you and the team. Noise comes and goes. Vocal voices come and go. My brethren and I's voice may not be the loudest, but remember we are here...appreciative and grateful. Thank you.

[LT218]

I'm pretty sure that "defragmenting" is no longer a thing, as even personal HD's are RAM based.
You'd be wrong.

Depends on the db type, but pretty much all of them require regular, recurring maintenance that is essentially "defragging" to continue running at top speed. Index rebuilds and optimizations, dump and loads, etc., etc.

At the hardware level, modern RAID controllers and enterprise drives have gotten fast enough to compensate/hide to some degree the effects of fragmentation, but they're still there. Depending on the controller and setup, defragging large arrays can be tricky, but doing so still provides benefits to performance. The larger the datasets parsed are, the larger the effect fragmentation will have.

[mikameow]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!

Just wanted to add to the chorus of voices thanking you for this transparency and technical straight talk. I don't remember the last time a forum post made me this happy. I've played this game since 2010 when I was 13 and for the first time since I was a teenager I have hope that the game's future is bright. It's easy to get caught up in the grumbled doom and gloom on here but this cut right through the usual forum dweller fog. You are the hero we don't deserve, but we certainly need right now.

[FURYous]

People seem to forget that Reaper was added because the playerbase were complaining that there was a lack of difficulty - not because of a lack of things to progress on. The enhancements has gone against the very thing that the mode was introduced for in the first place as low reaper is now easier than elite.

I agree with you, however, that if they're removed now it would remove a lot of peoples incentive to play, but they should never have been introduced it in the first place.

Had they not introduced it, the people complaining about lack of difficulty would have quit and the game would have died.

[grudgebear]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!


This was a nice read, and insight of what is going on. Thank you for your efforts and best wishes to you too!

[VinoeWhines]

I wrote the new algorithm and tested against the old in our development environment for quite a long time. We optimized how we pre-cache bonuses granted by feats. Giving too much information could tip off exploiters on where to look for vulnerabilities. When changing the underpinnings of a game that has been in development for close to 20 years, there are a lot of special corner cases in the code itself... and then there is a lot of content on top of it that sometimes will surprise you. I had many moments of "We used a feat for THAT? We gave you that feat HOW?" Being that a huge part of our game is our build meta, the sheer number of permutations and combinations of builds and interactions between bonuses makes it impossible to brute force test everything. So I appreciate all your observations and hope to find and fix any discrepancies that come up.

Best wishes!

Thanks again for putting in the workload along with the other developers in this environment.

Unfortunately stealth is still broken as well as it's playstyle as a Rogue.
So much taken away from the class: Cheat Death, Vorpal with all weapons on a 20.
Shadow Dancer Epic Destiny(Shadow Manipulation is now in Shiradi Tree) while it looses Executioner Shot/Strike, Consume and Meld Into Darkness as a separate option to attacking(Dark Imbuement), a useless Wand and Scroll Mastery, can't even choose Weird in your own tree, which is just a faster Consume(Implosion).

Would be great if they could fix that, especially since Dark Hunter seems to be better(Assassinate bonus') at sneaking and throwing useful Traps than an Assassin Rogue.

The fact you have to be in Stealth and be within Touch range to do soo many things shouldn't be soo punished as it is not an easy playstyle to play in and shouldn't of had soo many of it's abilities removed.(use to be able to Assassinate if you lined it up right 3 mobs, now your only locked to 2 if you can get it to even go off) Assassinate Cleave should be a thing, since they took away Double Strike ability. Even a "Sap" Cleave could be viable.

[VinoeWhines]

Will there be 2 more previews of Update 59 or is this going to be the only one?

[mikarddo]

Reaper the way it's set up is bad. You shouldn't get nerfed, you should increase the mobs HP and abilities and not weaken the players, you don't truly get a sense of your build honestly by making it weaker.(Give the mobs more HP and/or elemental-DR protection)



Giving mobs 7x as many hp OR lowering player damage to 1/7 has *exactly* the same effect.

[Hawkwier]

When reaper was first released there was a body of thought that supported a cosmetics only approach, as the posited rationale for reaper in the first place was "more challenge", and decidedly not "more reward", though how genuine and honest that claimed rationale was, was open to question IMO.

Anyway, the Devs went with reaper trees and that genie was let out of the bottle. A poor decision IMO, but it is what it is.

To try and re-bottle the genie now would be an even worse decision IMO. Too many have invested too much based on that initial decision. They had their chance and blew it at the time. It is what it is.

[erethizon]

So my take is, if they had just stayed true to the 3.5 rule books a lot of the issues would not be around, and yes I liked DDO better when it was D&D.

That ship sailed during Alpha testing for me when they went with Spell Points instead of spell slots. I know they said they couldn't make the game fun with X spells per rest, but that is the defining factor that makes a game D&D to me. Spell points are not D&D (even if there is some obscure rule set no one ever uses that has spell points).

[CaylisStarwynd]

So instead of getting rid of Reaper enhancements because the casuals like me appreciate them how about we just add mutations to the dungeons. Similar to the forbidden temple in Isle of Dread with the curses. we add a menu on the dungeon that can add mutations to the dungeon. Examples being like double the number of champion mobs increase the reaper ratio apply to reaper healing debuff to all healing not just self heals etc... the mutations could add an xp bonus depending on how many of them you take this could be a great way to add difficulty without taking anything away from the existing system for those of us who are not maxed out on everything.

[FURYous]

So instead of getting rid of Reaper enhancements because the casuals like me appreciate them how about we just add mutations to the dungeons. Similar to the forbidden temple in Isle of Dread with the curses. we add a menu on the dungeon that can add mutations to the dungeon. Examples being like double the number of champion mobs increase the reaper ratio apply to reaper healing debuff to all healing not just self heals etc... the mutations could add an xp bonus depending on how many of them you take this could be a great way to add difficulty without taking anything away from the existing system for those of us who are not maxed out on everything.

You pretty much summed up what reaper was all about, dial your own difficulty and get extra exp to match, the only difference is that the extra exp you got, could only be used in reaper. This helped prevent toons getting overpowered in normal content (like past lives and extra enhancement points do).

[Madja]

That ship sailed during Alpha testing for me when they went with Spell Points instead of spell slots. I know they said they couldn't make the game fun with X spells per rest, but that is the defining factor that makes a game D&D to me. Spell points are not D&D (even if there is some obscure rule set no one ever uses that has spell points).

I think spell points are a good compromise for an MMO, but now that they're virtually endless because of souls, overload of shrines and just huge pools it does feel kind of silly.

[Bjond]

Spell Points instead of spell slots. I know they said they couldn't make the game fun with X spells per rest, but that is the defining factor that makes a game D&D to me

This diverges a fair bit from the primary topic, but just can't resist .. :p

That was one of the very first rules I checked before deciding to play; ie. if it tried to use spell slots I'd have punted it. Spell slots are absolutely 100% incompatible with fun MMO play. I played DSO. It tried oh so hard to be pure (right down to spell slots, turn based, & PvP) and it was a miserable flop. It's just not fun to do nothing at all while your party carries you to the BBG where you can finally cast the couple spells you've been hoarding.

The closer the rules are to pure D&D, the less they work in an MMO. RP is why spell slots work in D&D. What would be a boring carry online becomes a lot of fun RP and perhaps very creative spell use; eg. wizard falls into a shute-trap, casts web on himself to stick to sides, gets rescue from party. That kind of ad-hoc creative application can't happen in an MMO, which seriously dilutes the class.

[VinoeWhines]

Giving mobs 7x as many hp OR lowering player damage to 1/7 has *exactly* the same effect.

Giving mobs 7x more HP is knowing what your character can do in top form, tying one arm behind you in a fight might be a challenge but it doesn't equal to the accurate authenticity of what you can do.

Remove the penalties and increase the difficulty to reflect the debuff.
This I believe will help also if Update 60 were to include it, less calculations in dungeon of constantly having to subtract effects/damages/crits/procs/saves/bonus'. Only the Stats and buffs would be added singularly once at entrance and that's it. All the other debuff calculations would be saved on the Server load-balancing as well.

As an Assassin Rogue, the Reaper Trees do almost nothing for me to Assassinate, yet I don't like that this class was neglected in Reaper, I still will defend to keep it with the incentive to improve your character, even if my character gets little other boosts, like HP, Dodge and Saves, those are still viable and useful.

I do agree with some that Reaper diff shouldn't be rewarding Spell Points only to caster, they should of given HP souls as well for melee or boost souls to do more damage, etc. but again all that should of just been removed I would say from Reaper 5 and onward, maybe give it to those that want to enter Reaper 1-4 to learn the mechanics of how Reaper is played.

I also am torn to some of how D&D is and how DDO is in it's present form, but as an MMO players can't play turn based rules so much as in a live action game.

I hope they can improve the Performance in update 59 by changing how all the feats/stats and bonus' are being calculated and hopefully they can remove the debuffs in Reaper to further increase the calculation performance in the dungeon instance and just code bump up the mobs strengths per dungeon and not add more calculations. I don't mind if the mobs can debuff you as a player but to go in and start shorthanded, I think is not a genuine difficulty to start out with. The Reapers are already doing that with their Reaper aura debuffs and that's great, let the debuffs happen from that point, just like we can debuff the mobs AC/PRR/MRR/Heal with our effects/abilities, they don't enter into our combat with getting nerfed, unless we apply it first.

[mikarddo]

Giving mobs 7x more HP is knowing what your character can do in top form, tying one arm behind you in a fight might be a challenge but it doesn't equal to the accurate authenticity of what you can do.


Oh well. For thats just the same, and I prefer the mobs keeping their regular hp. But to each his own - as it does not effect gameplay at all.

As for SP souls dropping - those are relatively worth much more for someone with low sp pool then for someone with a huge sp pool. So, for any ranged/melee that use a little spell points for heals these are worth much more than for a sorc with 7k+ spell points. HP souls for 250 hp wouldnt even matter at all anyway, so asking for those as some sort of equalizer does not seem very relevant.

[Scrag]

So... The lag is better right?

This seems pretty derailed and I would like to know more about what parts are less laggy, what parts are more, and if there is something common about what is still left over laggy and where we are in the live builds.

Looking forward to seeing less lag! Nothing sucks more than pewpewpewing with my xbow, seeing nothing happen, and then dying, or have mobs spontaneously die at my feet from massive burst, instead of safely at a distance.

[mikarddo]

With the current state of lagginess - I suggest throwing the update to the live servers now. Even if there are a few bugs just fix those on the fly.

Normally I would not suggest something like this - but lag is at a point where the game is so much less fun that it outweights the negatives of pushing an update early.

[Scrag]

With the current state of lagginess - I suggest throwing the update to the live servers now. Even if there are a few bugs just fix those on the fly.

Normally I would not suggest something like this - but lag is at a point where the game is so much less fun that it outweights the negatives of pushing an update early.

If the improvements are really that significant, Ill take some bugs in my juice!

[songswrath]

turned a post via dev into a debate. yah this will be locked soon

[momps]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!

Thanks for working to improve the spaghettii code to make the game better. After coming back to the game a few years ago i was floored at how much more complex it is now than it was at launch. I am really excited to hear that there is effort to improve things. Anyone in a development org is familiar with the problems you're talking about. My company still codes in COBOL and have code around from the 90s still so I completely relate to the issue of dealing with issues that surface whenever you make the slightest change (yay regression)

[thegreatcthulhu]

So instead of getting rid of Reaper enhancements because the casuals like me appreciate them how about we just add mutations to the dungeons. Similar to the forbidden temple in Isle of Dread with the curses. we add a menu on the dungeon that can add mutations to the dungeon. Examples being like double the number of champion mobs increase the reaper ratio apply to reaper healing debuff to all healing not just self heals etc... the mutations could add an xp bonus depending on how many of them you take this could be a great way to add difficulty without taking anything away from the existing system for those of us who are not maxed out on everything.

I have been wondering that myself... maybe its time for reaper to change how it works and mutations would be a fun twist. Maybe have a list based off things we've seen in the game and/or based off certain D&D villains:

- Vecna's Curse - same annoying thing we saw in IoD; Hand of Vecna hit everyone (including you) every number of random seconds (they probably should scale the damage down on it though)
- Strahd's "Protection" - Disables all teleport abilities, enemies and players have vampirism, Strahd randomly casts hold on monsters and players alike (not sure if people would like the full COS grip being used since we won't have a way to dispel it)
- Tiamat's Spite - Damage boost on incoming elemental damage, damage boost on outgoing elemental damage
- The Lord of Blades - The lord of blades sends random bladeforged into the dungeon; they are hostile to everyone including monsters; if you kill enough bladeforged, the LOB spawns and you have to beat him down to 10% to get him to leave, fighting off the LOB grants a temporary buff
- The Wild Hunt - Similar to the above but with weaker versions of the Hounds; the hounds are hostile to everyone including monsters; the master of the hunt we probably could leave out since LOB seems like he should stand out since this is Eberron; the hounds may drop stuff (IDK what, just saying they could to make it different)
- Asmodeus' Trickery - Random rotating status buffs and debuffs are cast on monsters and allies alike
- Zuggmoty's Spores - Toxic mushroom debuff that stacks throughout the dungeon, cleansing mushrooms will spawn within a distance from any player throughout the dungeon
- KOBOLDS! - All monsters groups have a high chance of having kobold shamen/throwers/etc added onto them

We could probably go all day thinking about fun mutators... honestly this might not be a bad idea to bring up on general.

[Belnavar]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!

Posts like this are not only illuminating for players, but reassuring. They give us insight into why something isn't an easy fix, and that SSG is indeed actually actively working on these issues. Being open and honest and humble about the problems, like you were in the above post, is also a great way to approach players in general -- and we're much more likely to be patient and understanding as a result.

Keep up the excellent work, Codog! I look forward to seeing these, and other, changes hit the live servers.

-Bel

[VinoeWhines]

Referencing LAG in Reaper 10:
One of the worst part of lag that I am experiencing is when I'm jumping away (about 20-30 feet away) from a mobs swing and when I turn to a soulstone; my soulstone(wasn't where I see I got taken out on screen) was 20 feet+ from where I was landing and by the mob's feet. I think there might be a problem with collision detection also as I see it on others to, getting hit when they were 10 feet away jumping in the air and their soulstone dropping to the floor, when I even saw they were not near the mobs swing.





Oh well. For thats just the same, and I prefer the mobs keeping their regular hp. But to each his own - as it does not effect gameplay at all.

As for SP souls dropping - those are relatively worth much more for someone with low sp pool then for someone with a huge sp pool. So, for any ranged/melee that use a little spell points for heals these are worth much more than for a sorc with 7k+ spell points. HP souls for 250 hp wouldnt even matter at all anyway, so asking for those as some sort of equalizer does not seem very relevant.


I don't think nerfing players on Reaper is a true test of difficulty as opposed to mobs being more robust, the heal nerf though not very appealing, prevents many from attempting high reaper.

SP stones shouldn't be in Reaper as it is a difficulty easy button. It's not on Elite but bump the difficulty and you get bonus cookies for participating. Grant it, I have used them and can benefit from them as I'm not a caster but melee as well but they are training wheels that maybe belong on Epic Elite or R1 but not anything over Reaper 5+.

To improve lag, like I mentioned earlier, they can try on Lamania to remove the player nerf calculations on Reaper and just bump up the mobs(non Champ) HP instead and remove all the nerf calculations per each instance of dungeon and I'm sure that will further help with less calculation on the front and back end that tend to occur on loading of gateway.

[Oliphant]

Folks, if you tested and the lag seems a lot better, maybe just post "OMG the lag seems so much better!" or something.

:D

[FURYous]

Folks, if you tested and the lag seems a lot better, maybe just post "OMG the lag seems so much better!" or something.

:D

https://www.youtube.com/watch?v=lHegMUycJyA

[Cordovan]

No fighting or dragging official discussions off topic. Captain_Wizbang and FURYous, if I see further posting in this thread from either of you that isn't specifically feedback to the OP you will be removed from the forums.

[Cordovan]

That goes for anyone else, too. Official discussion threads should not be dragged off-topic by personal vendettas and fights. If you are discussing the OP or other performance work that is fine, but anything not related to game performance must not be brought into the thread.

[FURYous]

That goes for anyone else, too. Official discussion threads should not be dragged off-topic by personal vendettas and fights. If you are discussing the OP or other performance work that is fine, but anything not related to game performance must not be brought into the thread.

I was under the impression that this was related to performance work. Captwizbang brought up that reaper trees were contributing to the lag and advocated they be removed. I disagreed with his assessment. I don't think there is any vendetta between us. Just discussing the impact of removing reaper trees to improve performance.

[Captain_Wizbang]

No fighting or dragging official discussions off topic. Captain_Wizbang and FURYous, if I see further posting in this thread from either of you that isn't specifically feedback to the OP you will be removed from the forums.


1st time I've seen admin name people specifically in a thread. :mad:

I apologized for drailing the thread, and telling furious Im not his enemy, but that post was removed.

[Oliphant]

https://www.youtube.com/watch?v=lHegMUycJyA

I watched this the other day, in fact I think I watched the link you posted on a Discord channel actually. Was gobsmacked. If this ends up being half accurate this should be the greatest update.

[Cordovan]

Closing ahead of preview 2. Thank you everyone for your feedback so far!