Lamannia Update 59 Preview 1 Feedback: Performance Adjustments

[mikarddo]

Giving mobs 7x more HP is knowing what your character can do in top form, tying one arm behind you in a fight might be a challenge but it doesn't equal to the accurate authenticity of what you can do.

Oh well. For thats just the same, and I prefer the mobs keeping their regular hp. But to each his own - as it does not effect gameplay at all.

As for SP souls dropping - those are relatively worth much more for someone with low sp pool then for someone with a huge sp pool. So, for any ranged/melee that use a little spell points for heals these are worth much more than for a sorc with 7k+ spell points. HP souls for 250 hp wouldnt even matter at all anyway, so asking for those as some sort of equalizer does not seem very relevant.

[Scrag]

So... The lag is better right?

This seems pretty derailed and I would like to know more about what parts are less laggy, what parts are more, and if there is something common about what is still left over laggy and where we are in the live builds.

Looking forward to seeing less lag! Nothing sucks more than pewpewpewing with my xbow, seeing nothing happen, and then dying, or have mobs spontaneously die at my feet from massive burst, instead of safely at a distance.

[mikarddo]

With the current state of lagginess - I suggest throwing the update to the live servers now. Even if there are a few bugs just fix those on the fly.

Normally I would not suggest something like this - but lag is at a point where the game is so much less fun that it outweights the negatives of pushing an update early.

[Scrag]

With the current state of lagginess - I suggest throwing the update to the live servers now. Even if there are a few bugs just fix those on the fly.

Normally I would not suggest something like this - but lag is at a point where the game is so much less fun that it outweights the negatives of pushing an update early.

If the improvements are really that significant, Ill take some bugs in my juice!

[songswrath]

turned a post via dev into a debate. yah this will be locked soon

[momps]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!

Thanks for working to improve the spaghettii code to make the game better. After coming back to the game a few years ago i was floored at how much more complex it is now than it was at launch. I am really excited to hear that there is effort to improve things. Anyone in a development org is familiar with the problems you're talking about. My company still codes in COBOL and have code around from the 90s still so I completely relate to the issue of dealing with issues that surface whenever you make the slightest change (yay regression)

[thegreatcthulhu]

So instead of getting rid of Reaper enhancements because the casuals like me appreciate them how about we just add mutations to the dungeons. Similar to the forbidden temple in Isle of Dread with the curses. we add a menu on the dungeon that can add mutations to the dungeon. Examples being like double the number of champion mobs increase the reaper ratio apply to reaper healing debuff to all healing not just self heals etc... the mutations could add an xp bonus depending on how many of them you take this could be a great way to add difficulty without taking anything away from the existing system for those of us who are not maxed out on everything.

I have been wondering that myself... maybe its time for reaper to change how it works and mutations would be a fun twist. Maybe have a list based off things we've seen in the game and/or based off certain D&D villains:

- Vecna's Curse - same annoying thing we saw in IoD; Hand of Vecna hit everyone (including you) every number of random seconds (they probably should scale the damage down on it though)
- Strahd's "Protection" - Disables all teleport abilities, enemies and players have vampirism, Strahd randomly casts hold on monsters and players alike (not sure if people would like the full COS grip being used since we won't have a way to dispel it)
- Tiamat's Spite - Damage boost on incoming elemental damage, damage boost on outgoing elemental damage
- The Lord of Blades - The lord of blades sends random bladeforged into the dungeon; they are hostile to everyone including monsters; if you kill enough bladeforged, the LOB spawns and you have to beat him down to 10% to get him to leave, fighting off the LOB grants a temporary buff
- The Wild Hunt - Similar to the above but with weaker versions of the Hounds; the hounds are hostile to everyone including monsters; the master of the hunt we probably could leave out since LOB seems like he should stand out since this is Eberron; the hounds may drop stuff (IDK what, just saying they could to make it different)
- Asmodeus' Trickery - Random rotating status buffs and debuffs are cast on monsters and allies alike
- Zuggmoty's Spores - Toxic mushroom debuff that stacks throughout the dungeon, cleansing mushrooms will spawn within a distance from any player throughout the dungeon
- KOBOLDS! - All monsters groups have a high chance of having kobold shamen/throwers/etc added onto them

We could probably go all day thinking about fun mutators... honestly this might not be a bad idea to bring up on general.

[Belnavar]

Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Best wishes!

Posts like this are not only illuminating for players, but reassuring. They give us insight into why something isn't an easy fix, and that SSG is indeed actually actively working on these issues. Being open and honest and humble about the problems, like you were in the above post, is also a great way to approach players in general -- and we're much more likely to be patient and understanding as a result.

Keep up the excellent work, Codog! I look forward to seeing these, and other, changes hit the live servers.

-Bel

[VinoeWhines]

Referencing LAG in Reaper 10:
One of the worst part of lag that I am experiencing is when I'm jumping away (about 20-30 feet away) from a mobs swing and when I turn to a soulstone; my soulstone(wasn't where I see I got taken out on screen) was 20 feet+ from where I was landing and by the mob's feet. I think there might be a problem with collision detection also as I see it on others to, getting hit when they were 10 feet away jumping in the air and their soulstone dropping to the floor, when I even saw they were not near the mobs swing.

Oh well. For thats just the same, and I prefer the mobs keeping their regular hp. But to each his own - as it does not effect gameplay at all.

As for SP souls dropping - those are relatively worth much more for someone with low sp pool then for someone with a huge sp pool. So, for any ranged/melee that use a little spell points for heals these are worth much more than for a sorc with 7k+ spell points. HP souls for 250 hp wouldnt even matter at all anyway, so asking for those as some sort of equalizer does not seem very relevant.

I don't think nerfing players on Reaper is a true test of difficulty as opposed to mobs being more robust, the heal nerf though not very appealing, prevents many from attempting high reaper.

SP stones shouldn't be in Reaper as it is a difficulty easy button. It's not on Elite but bump the difficulty and you get bonus cookies for participating. Grant it, I have used them and can benefit from them as I'm not a caster but melee as well but they are training wheels that maybe belong on Epic Elite or R1 but not anything over Reaper 5+.

To improve lag, like I mentioned earlier, they can try on Lamania to remove the player nerf calculations on Reaper and just bump up the mobs(non Champ) HP instead and remove all the nerf calculations per each instance of dungeon and I'm sure that will further help with less calculation on the front and back end that tend to occur on loading of gateway.

[Oliphant]

Folks, if you tested and the lag seems a lot better, maybe just post "OMG the lag seems so much better!" or something.

[FURYous]

Folks, if you tested and the lag seems a lot better, maybe just post "OMG the lag seems so much better!" or something.

https://www.youtube.com/watch?v=lHegMUycJyA

[Cordovan]

No fighting or dragging official discussions off topic. Captain_Wizbang and FURYous, if I see further posting in this thread from either of you that isn't specifically feedback to the OP you will be removed from the forums.

[Cordovan]

That goes for anyone else, too. Official discussion threads should not be dragged off-topic by personal vendettas and fights. If you are discussing the OP or other performance work that is fine, but anything not related to game performance must not be brought into the thread.

[FURYous]

That goes for anyone else, too. Official discussion threads should not be dragged off-topic by personal vendettas and fights. If you are discussing the OP or other performance work that is fine, but anything not related to game performance must not be brought into the thread.

I was under the impression that this was related to performance work. Captwizbang brought up that reaper trees were contributing to the lag and advocated they be removed. I disagreed with his assessment. I don't think there is any vendetta between us. Just discussing the impact of removing reaper trees to improve performance.

[Captain_Wizbang]

No fighting or dragging official discussions off topic. Captain_Wizbang and FURYous, if I see further posting in this thread from either of you that isn't specifically feedback to the OP you will be removed from the forums.

1st time I've seen admin name people specifically in a thread.

I apologized for drailing the thread, and telling furious Im not his enemy, but that post was removed.

[Oliphant]

https://www.youtube.com/watch?v=lHegMUycJyA

I watched this the other day, in fact I think I watched the link you posted on a Discord channel actually. Was gobsmacked. If this ends up being half accurate this should be the greatest update.

[Cordovan]

Closing ahead of preview 2. Thank you everyone for your feedback so far!