Lamannia Update 59 Preview 1 Feedback: Performance Adjustments

[JOTMON]

~snip~
and not all of them are on the character sheet and easily displayable.
~snip~

.. the glaring question is 'why' we cant easily see them, and is this something you can resolve in future.

Being able to see what is active/not active is indispensable for analyzing and understanding what does and doesn't work, what stacks, overwrites, what bonus bucket buffs actually belong to as descriptions are notoriously inaccurate/incorrect.

[thegreatcthulhu]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!

Thanks and very informative.

And even though I personally never want to get into game development, I understand a little bit of what you are saying based on my own experiences with "regular" software development. Case in point, where you mention that the call stacks go into an external system indeed creates a big challenge. Same with the thing you described about the value caching.

[Arkat]

I'm pretty sure that "defragmenting" is no longer a thing, as even personal HD's are RAM based. Odds are that server-side storage is considerably more rebust / advanced than what "we" have available. Think, Amazon AWS cloud computing.

Defragmenting hard drives and index defragmenting (a.k.a. rebuilding the index) databases are *not* the same thing.

Any DB Admin who's worth anything knows regular index defragmenting their DBs keeps the performance up by reducing DB page counts, even if the DBs are kept on NVMe drives (or even older SSD drives) which, btw, the DDO servers do NOT run on. Their servers still likely run on SAS drives even if the game servers are virtualized.

[rabidfox]

Player Power or Cosmetics as rewards for Reaper is a false dilemma.

Cosmetics are great but you can offer other things besides power as substitutes. I'm thinking notoriety, mostly.

Leaderboards that list who who has done what difficulties for which quests (with time completions to break ties) would be a great way for the ultra-competitive players to show who's the King of the Reaper "hill."

I'm sure their are other ways to incentivize running reaper difficulties.

The speedrun challenge ( https://forums.ddo.com/forums/showth...-run-Challenge ) going on right now is some of the most fun I've had in a while where it's just about pushing oneself.

SSG could probably do seasonal contests/reward tracks with leaderboard or earn X amount of RXP during a 3-month block, or complete Y number of raids, TR Z times, etc. It could be similar to HC rewards where they reset reward tracks every few months and offer new prizes each time but with everything done on the live servers.

[Captain_Wizbang]

Pointless.
Cosmetics are one off prizes that are selectively appreciated by players, few are desirable most are fugly.
a one off cosmetic (assuming I like the look of it) is a one and done adventure.
Ultimately they are just inventory space killers and the last thing we need is more scrap to overflow an already overburdened inventory.
Cosmetics are good for periodic events like Mabar, cove, Anniversary, hardcore and expansions....


Rewards need to entice players to repeatedly replay content otherwise it is a pointless waste of programming time to to make content no one cares to play.


I am enticed to run content that gives me more power as that opens up the horizon of possibilities to tweak my build and the opportunity to successfully survive and challenge more difficult content.
..Enter the hamster wheel..

bingo

[Ganak]

LOL... just gave too much detail.

Wow, appreciate the detail, even if I couldn't follow it

I'm super grateful for the fans that have kept this franchise going

VIP since the start. Started playing at 29, now 46, know I will be playing and loving our game in my 50's and expect to keep going in my 60's+ thanks to you and the team. Noise comes and goes. Vocal voices come and go. My brethren and I's voice may not be the loudest, but remember we are here...appreciative and grateful. Thank you.

[LT218]

I'm pretty sure that "defragmenting" is no longer a thing, as even personal HD's are RAM based.

You'd be wrong.

Depends on the db type, but pretty much all of them require regular, recurring maintenance that is essentially "defragging" to continue running at top speed. Index rebuilds and optimizations, dump and loads, etc., etc.

At the hardware level, modern RAID controllers and enterprise drives have gotten fast enough to compensate/hide to some degree the effects of fragmentation, but they're still there. Depending on the controller and setup, defragging large arrays can be tricky, but doing so still provides benefits to performance. The larger the datasets parsed are, the larger the effect fragmentation will have.

[mikameow]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!

Just wanted to add to the chorus of voices thanking you for this transparency and technical straight talk. I don't remember the last time a forum post made me this happy. I've played this game since 2010 when I was 13 and for the first time since I was a teenager I have hope that the game's future is bright. It's easy to get caught up in the grumbled doom and gloom on here but this cut right through the usual forum dweller fog. You are the hero we don't deserve, but we certainly need right now.

[FURYous]

People seem to forget that Reaper was added because the playerbase were complaining that there was a lack of difficulty - not because of a lack of things to progress on. The enhancements has gone against the very thing that the mode was introduced for in the first place as low reaper is now easier than elite.

I agree with you, however, that if they're removed now it would remove a lot of peoples incentive to play, but they should never have been introduced it in the first place.

Had they not introduced it, the people complaining about lack of difficulty would have quit and the game would have died.

[grudgebear]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!

This was a nice read, and insight of what is going on. Thank you for your efforts and best wishes to you too!

[VinoeWhines]

I wrote the new algorithm and tested against the old in our development environment for quite a long time. We optimized how we pre-cache bonuses granted by feats. Giving too much information could tip off exploiters on where to look for vulnerabilities. When changing the underpinnings of a game that has been in development for close to 20 years, there are a lot of special corner cases in the code itself... and then there is a lot of content on top of it that sometimes will surprise you. I had many moments of "We used a feat for THAT? We gave you that feat HOW?" Being that a huge part of our game is our build meta, the sheer number of permutations and combinations of builds and interactions between bonuses makes it impossible to brute force test everything. So I appreciate all your observations and hope to find and fix any discrepancies that come up.

Best wishes!

Thanks again for putting in the workload along with the other developers in this environment.

Unfortunately stealth is still broken as well as it's playstyle as a Rogue.
So much taken away from the class: Cheat Death, Vorpal with all weapons on a 20.
Shadow Dancer Epic Destiny(Shadow Manipulation is now in Shiradi Tree) while it looses Executioner Shot/Strike, Consume and Meld Into Darkness as a separate option to attacking(Dark Imbuement), a useless Wand and Scroll Mastery, can't even choose Weird in your own tree, which is just a faster Consume(Implosion).

Would be great if they could fix that, especially since Dark Hunter seems to be better(Assassinate bonus') at sneaking and throwing useful Traps than an Assassin Rogue.

The fact you have to be in Stealth and be within Touch range to do soo many things shouldn't be soo punished as it is not an easy playstyle to play in and shouldn't of had soo many of it's abilities removed.(use to be able to Assassinate if you lined it up right 3 mobs, now your only locked to 2 if you can get it to even go off) Assassinate Cleave should be a thing, since they took away Double Strike ability. Even a "Sap" Cleave could be viable.

[VinoeWhines]

Will there be 2 more previews of Update 59 or is this going to be the only one?

[mikarddo]

Reaper the way it's set up is bad. You shouldn't get nerfed, you should increase the mobs HP and abilities and not weaken the players, you don't truly get a sense of your build honestly by making it weaker.(Give the mobs more HP and/or elemental-DR protection)

Giving mobs 7x as many hp OR lowering player damage to 1/7 has *exactly* the same effect.

[Hawkwier]

When reaper was first released there was a body of thought that supported a cosmetics only approach, as the posited rationale for reaper in the first place was "more challenge", and decidedly not "more reward", though how genuine and honest that claimed rationale was, was open to question IMO.

Anyway, the Devs went with reaper trees and that genie was let out of the bottle. A poor decision IMO, but it is what it is.

To try and re-bottle the genie now would be an even worse decision IMO. Too many have invested too much based on that initial decision. They had their chance and blew it at the time. It is what it is.

[erethizon]

So my take is, if they had just stayed true to the 3.5 rule books a lot of the issues would not be around, and yes I liked DDO better when it was D&D.

That ship sailed during Alpha testing for me when they went with Spell Points instead of spell slots. I know they said they couldn't make the game fun with X spells per rest, but that is the defining factor that makes a game D&D to me. Spell points are not D&D (even if there is some obscure rule set no one ever uses that has spell points).

[CaylisStarwynd]

So instead of getting rid of Reaper enhancements because the casuals like me appreciate them how about we just add mutations to the dungeons. Similar to the forbidden temple in Isle of Dread with the curses. we add a menu on the dungeon that can add mutations to the dungeon. Examples being like double the number of champion mobs increase the reaper ratio apply to reaper healing debuff to all healing not just self heals etc... the mutations could add an xp bonus depending on how many of them you take this could be a great way to add difficulty without taking anything away from the existing system for those of us who are not maxed out on everything.

[FURYous]

So instead of getting rid of Reaper enhancements because the casuals like me appreciate them how about we just add mutations to the dungeons. Similar to the forbidden temple in Isle of Dread with the curses. we add a menu on the dungeon that can add mutations to the dungeon. Examples being like double the number of champion mobs increase the reaper ratio apply to reaper healing debuff to all healing not just self heals etc... the mutations could add an xp bonus depending on how many of them you take this could be a great way to add difficulty without taking anything away from the existing system for those of us who are not maxed out on everything.

You pretty much summed up what reaper was all about, dial your own difficulty and get extra exp to match, the only difference is that the extra exp you got, could only be used in reaper. This helped prevent toons getting overpowered in normal content (like past lives and extra enhancement points do).

[Madja]

That ship sailed during Alpha testing for me when they went with Spell Points instead of spell slots. I know they said they couldn't make the game fun with X spells per rest, but that is the defining factor that makes a game D&D to me. Spell points are not D&D (even if there is some obscure rule set no one ever uses that has spell points).

I think spell points are a good compromise for an MMO, but now that they're virtually endless because of souls, overload of shrines and just huge pools it does feel kind of silly.

[Bjond]

Spell Points instead of spell slots. I know they said they couldn't make the game fun with X spells per rest, but that is the defining factor that makes a game D&D to me

This diverges a fair bit from the primary topic, but just can't resist ..

That was one of the very first rules I checked before deciding to play; ie. if it tried to use spell slots I'd have punted it. Spell slots are absolutely 100% incompatible with fun MMO play. I played DSO. It tried oh so hard to be pure (right down to spell slots, turn based, & PvP) and it was a miserable flop. It's just not fun to do nothing at all while your party carries you to the BBG where you can finally cast the couple spells you've been hoarding.

The closer the rules are to pure D&D, the less they work in an MMO. RP is why spell slots work in D&D. What would be a boring carry online becomes a lot of fun RP and perhaps very creative spell use; eg. wizard falls into a shute-trap, casts web on himself to stick to sides, gets rescue from party. That kind of ad-hoc creative application can't happen in an MMO, which seriously dilutes the class.

[VinoeWhines]

Giving mobs 7x as many hp OR lowering player damage to 1/7 has *exactly* the same effect.

Giving mobs 7x more HP is knowing what your character can do in top form, tying one arm behind you in a fight might be a challenge but it doesn't equal to the accurate authenticity of what you can do.

Remove the penalties and increase the difficulty to reflect the debuff.
This I believe will help also if Update 60 were to include it, less calculations in dungeon of constantly having to subtract effects/damages/crits/procs/saves/bonus'. Only the Stats and buffs would be added singularly once at entrance and that's it. All the other debuff calculations would be saved on the Server load-balancing as well.

As an Assassin Rogue, the Reaper Trees do almost nothing for me to Assassinate, yet I don't like that this class was neglected in Reaper, I still will defend to keep it with the incentive to improve your character, even if my character gets little other boosts, like HP, Dodge and Saves, those are still viable and useful.

I do agree with some that Reaper diff shouldn't be rewarding Spell Points only to caster, they should of given HP souls as well for melee or boost souls to do more damage, etc. but again all that should of just been removed I would say from Reaper 5 and onward, maybe give it to those that want to enter Reaper 1-4 to learn the mechanics of how Reaper is played.

I also am torn to some of how D&D is and how DDO is in it's present form, but as an MMO players can't play turn based rules so much as in a live action game.

I hope they can improve the Performance in update 59 by changing how all the feats/stats and bonus' are being calculated and hopefully they can remove the debuffs in Reaper to further increase the calculation performance in the dungeon instance and just code bump up the mobs strengths per dungeon and not add more calculations. I don't mind if the mobs can debuff you as a player but to go in and start shorthanded, I think is not a genuine difficulty to start out with. The Reapers are already doing that with their Reaper aura debuffs and that's great, let the debuffs happen from that point, just like we can debuff the mobs AC/PRR/MRR/Heal with our effects/abilities, they don't enter into our combat with getting nerfed, unless we apply it first.