Page 3 of 6 FirstFirst 123456 LastLast
Results 41 to 60 of 117
  1. #41
    Community Member FURYous's Avatar
    Join Date
    Mar 2006
    Posts
    553

    Default

    Quote Originally Posted by FURYous View Post
    Here are the differences I found, same toon on Orien and Lamania with the same self buffs]
    I also noticed my will save went up one point on Lamania
    Mr Blacks - Ranged DPS - 120PL - 50RP
    Mr Blues - Main - 177PL - 98RP
    Mr Greens - Caster CC/DC - 126PL - 37RP
    Mr Purples - Healer - 43PL - 21RP
    Mr Redd - Melee DPS - 129PL - 37RP
    Mr Whites - Tank - 138PL - 58RP

  2. #42
    Developer CodogCuisinart's Avatar
    Join Date
    Sep 2022
    Posts
    11

    Default

    Quote Originally Posted by LurkingVeteran View Post
    First, thanks for looking into the lag, and talking about it on the forum! Second, it is a bit surprising that adding points to reaper trees at login can somehow have a noticeable impact on a modern server CPU (simple arithmetic). Additionally, most changes so far have focused on tweaking such scripts to be faster. However, have you tried profiling the (C++?) scripting engine itself to figure out why it is slow? Maybe nobody has remembered to grease the hamster wheel since back in 2006?
    Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

    That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

    An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

    So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

    I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

    I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

    Best wishes!
    Last edited by CodogCuisinart; 04-06-2023 at 11:40 AM.

  3. #43
    Community Member Captain_Wizbang's Avatar
    Join Date
    Jan 2008
    Posts
    0

    Default

    Quote Originally Posted by CodogCuisinart View Post
    Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

    That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

    An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

    So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

    I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

    I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

    Best wishes!

    Good info, thanks. GRATZ on on the family front.

    Those of us still here are devout fans. I built a high-end rig to get away from DDO, The irony is, it plays this game so well and almost lag-free, It kept me here. Any changes you all are making to improve playability are greatly appreciated.

    My main concern for DDO is dropping the entire reaper enhancement system. (please ditch it) Yes it will irk those heavily invested in it, but IMHO it IS needed. Thoughts?

  4. #44
    Community Member FURYous's Avatar
    Join Date
    Mar 2006
    Posts
    553

    Default

    Quote Originally Posted by CodogCuisinart View Post
    Without giving too much detail

    Best wishes!
    I want to thank you for the communication. This helps all of us understand what is going on and stay on the same page.

    Too often we have been left with no communication for days/weeks/months and in that absence, we assume the worst.

    Updates and communication as to what is going on will help us to understand that you guys are really there and working to make the game we all love, better.
    Mr Blacks - Ranged DPS - 120PL - 50RP
    Mr Blues - Main - 177PL - 98RP
    Mr Greens - Caster CC/DC - 126PL - 37RP
    Mr Purples - Healer - 43PL - 21RP
    Mr Redd - Melee DPS - 129PL - 37RP
    Mr Whites - Tank - 138PL - 58RP

  5. #45
    Community Member FURYous's Avatar
    Join Date
    Mar 2006
    Posts
    553

    Default

    Quote Originally Posted by Captain_Wizbang View Post
    Good info, thanks. GRATZ on on the family front.

    Those of us still here are devout fans. I built a high-end rig to get away from DDO, The irony is, it plays this game so well and almost lag-free, It kept me here. Any changes you all are making to improve playability are greatly appreciated.

    My main concern for DDO is dropping the entire reaper enhancement system. (please ditch it) Yes it will irk those heavily invested in it, but IMHO it IS needed. Thoughts?
    I am a network engineer with over 25 years of experience and a degree in computer science.
    I have a cutting edge system and when all 12 people in a raid have the exact same lag at the same time, it doesn't point to a client side problem.
    When the MMORPG is the ONLY MMORPG that has to schedule weekly reboots, it doesn't point to a client side hardware issue.
    When an entire server of players tell you they are having issues, this doesn't point to a client side issue.
    Yes, there are some client side issues due to hardware and bandwidth, but the vast majority of the complaints are not about those issues.

    Also as far as the Reaper point system, you have to create content/advancement for all your players or they become bored and quit. Getting rid of the reaper point system will gut your player base and leave the game without revenue. Some of us that spend thousands of dollars per year on the game have every past life, done every quest/raid hundreds of times, and rely on the reaper point system as a way to advance in the game. Just because you don't play in that end of the game doesn't justify taking it away from all the people who do.

    This doesn't even count the tens of thousands of runes/threads I have spent.
    Mr Blacks - Ranged DPS - 120PL - 50RP
    Mr Blues - Main - 177PL - 98RP
    Mr Greens - Caster CC/DC - 126PL - 37RP
    Mr Purples - Healer - 43PL - 21RP
    Mr Redd - Melee DPS - 129PL - 37RP
    Mr Whites - Tank - 138PL - 58RP

  6. #46

    Default

    Quote Originally Posted by CodogCuisinart View Post
    Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.
    <SNIP>
    Best wishes!
    Codog,

    As a long time player, I mean was Closed Beta that long ago I truly have always appreciated your candor and upfront nature on the forums. You have always had a way to provide enough information from behind the curtain to satisfy many players questions and concerns. I hope your bosses allow you time to appear occasionally on the forums and peel back another curtain or two. I think that goes a long ways to making happy customers.

    Best
    Goldy

    The Twilight Avengers are always recruiting - http://twilightavengersofeberron.yuku.com/topic/655

  7. #47
    Community Member Monkey_Archer's Avatar
    Join Date
    May 2008
    Posts
    2,416

    Default

    Quote Originally Posted by CodogCuisinart View Post
    Without giving too much detail... just gave too much detail.
    Awesome post. Thank you for this.
    Thelanis

  8. #48
    Community Member Monkey_Archer's Avatar
    Join Date
    May 2008
    Posts
    2,416

    Default

    Quote Originally Posted by FURYous View Post
    Also as far as the Reaper point system, you have to create content/advancement for all your players or they become bored and quit. Getting rid of the reaper point system will gut your player base and leave the game without revenue. Some of us that spend thousands of dollars per year on the game have every past life, done every quest/raid hundreds of times, and rely on the reaper point system as a way to advance in the game. Just because you don't play in that end of the game doesn't justify taking it away from all the people who do.
    Indeed.
    On the bright side, it seems that if reaper points were/are coded as feats, and the feats system gets properly isolated to only iterate on logon/advancement, then the reaper point system *shouldn't* be a cause of lag anyway. Hopefully, in theory
    Thelanis

  9. #49
    Phoenicis' Wife SavageDoom's Avatar
    Join Date
    Nov 2021
    Posts
    24

    Default

    Quote Originally Posted by FURYous View Post
    I want to thank you for the communication. This helps all of us understand what is going on and stay on the same page.

    Too often we have been left with no communication for days/weeks/months and in that absence, we assume the worst.

    Updates and communication as to what is going on will help us to understand that you guys are really there and working to make the game we all love, better.

    ^This
    Finally someone that talks to us and explains things.
    Thank you, this is very much appreciated.

  10. #50
    Community Member
    Join Date
    Feb 2021
    Posts
    321

    Default

    Quote Originally Posted by CodogCuisinart View Post
    Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

    That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

    An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

    So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

    I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

    I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

    Best wishes!
    Super thanks, posts like this buy a ton of good will from the community. Those without a technical background think "wow this sounds hard", while those of us with a technical background think "holy moly, this is crazy hard".

  11. #51
    Community Member TPICKRELL's Avatar
    Join Date
    Jun 2009
    Posts
    2,068

    Default

    Quote Originally Posted by CodogCuisinart View Post
    Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.
    ...
    Thank you so much for actually telling us what is going on!
    Khyber -- Grubbby, Grubonon, Gralak, and all the gang of *grubs* in the Homeboys of Stormreach.

  12. #52
    Community Member Captain_Wizbang's Avatar
    Join Date
    Jan 2008
    Posts
    0

    Default

    Quote Originally Posted by FURYous View Post
    I am a network engineer with over 25 years of experience and a degree in computer science.
    I have a cutting edge system and when all 12 people in a raid have the exact same lag at the same time, it doesn't point to a client side problem.
    When the MMORPG is the ONLY MMORPG that has to schedule weekly reboots, it doesn't point to a client side hardware issue.
    When an entire server of players tell you they are having issues, this doesn't point to a client side issue.
    Yes, there are some client side issues due to hardware and bandwidth, but the vast majority of the complaints are not about those issues.

    Also as far as the Reaper point system, you have to create content/advancement for all your players or they become bored and quit. Getting rid of the reaper point system will gut your player base and leave the game without revenue. Some of us that spend thousands of dollars per year on the game have every past life, done every quest/raid hundreds of times, and rely on the reaper point system as a way to advance in the game. Just because you don't play in that end of the game doesn't justify taking it away from all the people who do.

    This doesn't even count the tens of thousands of runes/threads I have spent.
    Aside from players like you spending gobs of money on the hamster wheel. I to this day will defend the obvious point of a reward system introduced to offset that difficulty setting. Manyplayers fail to connect the 2, THAT is my point. I could care less about your computing quals. No offence.

  13. #53
    Community Member TPICKRELL's Avatar
    Join Date
    Jun 2009
    Posts
    2,068

    Default Credibility not just good will...

    Quote Originally Posted by CodogCuisinart View Post
    Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.
    ...
    Should have said this above, but didnt think of it right away.

    As pointed out above, this post buys you and SSG a ton of good will.

    But it also buys you and SSG a lot of credibility. And that's something SSG needs to get back.

    This is the one of the few times in recent years that I've seen a post that I believed was forthcoming and conveying as much accurate info as you feel you are able.

    Thank You.
    Last edited by TPICKRELL; 04-06-2023 at 01:33 PM.
    Khyber -- Grubbby, Grubonon, Gralak, and all the gang of *grubs* in the Homeboys of Stormreach.

  14. #54
    Community Member
    Join Date
    Apr 2013
    Posts
    2,032

    Default

    Quote Originally Posted by CodogCuisinart View Post
    I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

    Best wishes!
    The BIGGEST THANK YOUs for this.

    I absolutely always loved getting a peek under DDO's hood, and this has to be one of the biggest insights I've been given yet.
    Absolutely loved reading this.

    I know you're busy and you can't just hand out these kinds of gifts all the time, but any level of communication from the devs on the forums has become a precious commodity.

    Thanks again. Man, reading this gave me a whole other view of the scope of just how much that legacy code needs to be worked with or worked around.
    Enthusiasm enthusiast enthusiast.

  15. #55
    Community Member FURYous's Avatar
    Join Date
    Mar 2006
    Posts
    553

    Default

    Quote Originally Posted by Captain_Wizbang View Post
    Aside from players like you spending gobs of money on the hamster wheel. I to this day will defend the obvious point of a reward system introduced to offset that difficulty setting. Manyplayers fail to connect the 2, THAT is my point. I could care less about your computing quals. No offence.
    The "computing quals" was in response to your assertation that the lag was client side. This gave the impression that you knew anything about technology.

    Secondly "spending gobs of money" is relative. To me it is a small portion of my budget for entertainment. Some people choose to spend on restaurants, some on booze, some on going to the movies, and so on, who are you to make judgement calls on my discretionary spending? Do you have an idea how much one good coder costs? How about the rest of the expenses to keep the game up? Be glad some of us are willing to pay them for the product you enjoy.
    Mr Blacks - Ranged DPS - 120PL - 50RP
    Mr Blues - Main - 177PL - 98RP
    Mr Greens - Caster CC/DC - 126PL - 37RP
    Mr Purples - Healer - 43PL - 21RP
    Mr Redd - Melee DPS - 129PL - 37RP
    Mr Whites - Tank - 138PL - 58RP

  16. #56
    Community Member mpetrarca's Avatar
    Join Date
    Dec 2010
    Posts
    254

    Default

    Quote Originally Posted by CodogCuisinart View Post
    Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

    That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

    An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

    So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

    I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

    I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

    Best wishes!

    CodogCuisinart you are the first person at SSG I can say I have respect for, and just for giving a real answer. Not even Cordovan with an Associate's Degree in Broadcasting communicates this well.

    So my take is, if they had just stayed true to the 3.5 rule books a lot of the issues would not be around, and yes I liked DDO better when it was D&D.

  17. #57
    Founder & Super Hero Arkat's Avatar
    Join Date
    Feb 2006
    Posts
    380

    Default

    Quote Originally Posted by CodogCuisinart View Post
    I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG.
    If only we had been given such a detailed and artful explanation of the problem years ago. A lot of the "bad feelings" on the part of the playerbase could have been mitigated to at least some degree. I would hope that a resulting more understanding playerbase would also mitigate any bad feelings from the Devs towards the playerbase.

    Thank you for taking the time to explain this stuff. MANY of us, believe it or not, have technical backgrounds in IT and software development and understand what you're saying to varying degrees.


    Quote Originally Posted by CodogCuisinart View Post
    I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

    Best wishes!
    Best wishes to you and your familiy.

    We are grateful for your communication to us.

    We've been waiting for a post like this for a LONG time!

    +1!


    Regards,

    JB
    Last edited by Arkat; 04-06-2023 at 06:28 PM.
    Quote Originally Posted by Aelonwy View Post
    Quote Originally Posted by Cordovan View Post
    The release notes themselves are essentially the same as was seen on Lamannia most recently.
    This^, in so many words, is how you say time and feedback on Lamannia are wasted.

  18. #58
    Founder & Super Hero Arkat's Avatar
    Join Date
    Feb 2006
    Posts
    380

    Default

    Quote Originally Posted by mpetrarca View Post
    Not even Cordovan with an Associate's Degree in Broadcasting communicates this well.
    To be fair, Cordo is VERY busy. Not only does he have to moderate the DDO Forum, put up the Store Sales, post the weekend bonuses, do all the social media posts, compiling and posting release notes, etc., etc., he also has to do it for all the similar stuff on the LotRO side.

    Frankly, he needs help. He may not have the time to get explanations like Codog's post above and then post them on the forum.

    I suggested LONG ago that Turbine (now SSG) should hire someone primarily responsible for liaising between the Players and the Devs as well as coordinating SSG internal communications. I called the position a "Communications Czar."

    I still think this is a good idea and would allow Cordo to concentrate on "managing the community" per his job title.
    Last edited by Arkat; 04-06-2023 at 02:33 PM.
    Quote Originally Posted by Aelonwy View Post
    Quote Originally Posted by Cordovan View Post
    The release notes themselves are essentially the same as was seen on Lamannia most recently.
    This^, in so many words, is how you say time and feedback on Lamannia are wasted.

  19. #59
    Community Member btolson's Avatar
    Join Date
    Aug 2009
    Posts
    1,291

    Default

    Will all this work on feats/enhancements touch on pets as well? Currently they do not gain any benefit whatsoever from augments of any kind, as if the algo that sets their stats just doesn't even try to find them.

  20. #60
    Barbarbarian Sam-u-r-eye's Avatar
    Join Date
    Aug 2010
    Posts
    2,024

    Default

    Quote Originally Posted by CodogCuisinart View Post
    Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

    ...

    Best wishes!
    You too dude.
    Without new players DDO will go the way of the dodo.
    Old Sorc Build Guide, Ghallanda -> Orien

Page 3 of 6 FirstFirst 123456 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

This form's session has expired. You need to reload the page.

Reload