(towards a) Gold standard in balance testing

[Jetrule]

I think that ssg has stated they have access to much better data gathering methods than players on a burntsmash test could generate. They are opaque with their data and that leaves their methods and criteria suspect, I think they do that to avoid endless criticism and argument about the relevance and accuracy of their data. Just as this thread demonstrates. So I have come to accept that hoping for a benevolent plutocracy is the best that can be realistically hoped for in this game when it comes to data driven balance.

People play the game differently and have different goals. I would make a analogy to auto racing. You seem interested in a style of play that compares to drag racing. Seeing which of the purpose built cars can hit a quarter mile fastest or a 1 mile straight track race. Others are looking for a dirt track drift race where surviving collisions and maneuvering in traffic are at least as important as raw speed. Others may be looking for endurance 500 mile races where criteria are the speed of the car and its efficiency and reliability but more than anything the quality and skill of your support team determine success.

Most players arent even competitively racing though. We just want reliable transportation that can safely get through the courses to finish. Or be a great team member that helps our team finish safely. Top performance just seems egotistical or not fun as compared to good performance and skillful or fun play.

[Karthunk]

I've made this statement several times and it is always completely ignored by our current crop of nerf ranged dps crowd:

Why is it ok for a high DC caster to completely lock down entire groups of npcs in R10 but it's outrageously unbalanced if a pure dps build is doing it's job well? Making enemies helpless is balanced but killing them fast is broken and needs to be fixed.

[Potatofasf]

...I think they do that to avoid endless criticism and argument about the relevance and accuracy of their data...

They are failing it so hard... it was proved by the nerf of Henshin Mystic and Quarterstaff based builds... and by the Optional XP Ransack.

Their "internal data" must be that flawed. Most because hard data lacks the emotional components that drives players attachment to the game. When a company don't consider how the player base (paying or not) feels... it has failed to accomplish any evaluation.

I've made this statement several times and it is always completely ignored by our current crop of nerf ranged dps crowd:

Why is it ok for a high DC caster to completely lock down entire groups of npcs in R10 but it's outrageously unbalanced if a pure dps build is doing it's job well? Making enemies helpless is balanced but killing them fast is broken and needs to be fixed.

To not ignore you... and to be clear, I don't care if Inqui or Sorc are crashing contente at R10, as you I want they carrying me over my TRs and ETRs.

Most of the complainers are teamworkers, they don't want to feel useless or worthless. They prefer a DC holding monsters to be killed by them, otherwise they are not immersed in a D&D fantasy. When a class or universal tree alone makes the work of a party of Six members, they must feel (I'm speculating) the game is broken.
And from a co-op point of view they are right. Is like in PnP when we put time and effort to create a character with backstory and the Dungeon Master forcibly input a NPC doing all the heavy lift or benefit his right hand man to the campaign stardom .

[Thrudh]

Everyone has an opinion about the state of balance in the game. However, we haven't come together to a consensus on what constitute good "tests" for balance. Devs be devs, purposely obscure with regards to what is a "good" test to convince them. So, let's build a consensus of our own (to the posters: ignore the trolling and baiting that will follow, focus on the topic).

We need good measures, IMO, for: i) boss DPS, ii) trash DPS, iii) questing "power". Let's start with boss DPS, since it is likely to be the simplest.

What constitutes a good test mob for boss DPS:

• Easy access: testing is a PITA, let's not make it a raid boss hidden after a long beatdowm (Mrs. bumbum spider goddess?).
• Relevant: gimmicky content, old bosses no one cares about, etc.
• Beefy: enough HPs to avoid the short term buffs fallacy.

How should the reports be?

• Transparent: specify build and gear, rotations; if you needed a tank / healer, report that too (they scale the instance differently).
• Video or not video: video is preferred, but other reports will be accepted (expect them to be contested).
• Multiple rounds: or the 5 beatdowns rule; a single beatdown is prone to crit fishing on martial builds (feel free to speed up the video, or provide multiple links).

The potential pitfalls:

• Racer toons: characters build specifically to "win" boss DPS racers are a problem; transparency solves this partially, but it is left the objective reader.
• Special boss mechanics: some bosses have special mechanics (teleports, cleaves, melee range), so they might alter the results wrt to kobold boss types.
• Non-comparable submissions: a player goes in with full debuffs, and the next does not.

With all this in mind, what is a good test mob? Well, no need to re-invent the wheel. Mr Bruntsmash in Cabal for One (https://ddowiki.com/page/A_Cabal_for_One).

The rules of the modified Bruntsmash test:

1. No debuffs: it is expected that someone would be applying those debuffs in a group setting. While some builds have easier access to them (fast attackers stacking then swapping), it would create a layer of complication that we cannot solve easily. The fact that does debuffs scale damage roughly equally across builds guarantees that this won't bias the test excessively. Specifically: i) always use the same weapon; ii) do not use class, PRE or ED based debuffers (jade strike, INQ vuln stacks); iii) the exception to this rule is deception and sneak attack debuff, which are allowed (SA is too innate to some builds to disallow it, and it wouldn't happen in real play).
2. No external buffs: other players are not allowed to buff you; it would make it very hard to replicate.
3. Full transparency: report your rotation, build, gear, group composition.
4. Number of skulls: 5 skulls rule of thumb; has more HPs to resist the shortest burst.

Failure to comply with the rules of the test disqualifies a submission. There are plenty of venues to post for just bragging rights, but it is of limited use in balance discussions.


My suggestion is to first make a baseline boss DPS as "fair" as possible (did I miss anything?), then move on to different bosses, and the rest of things to be tested.

Let's turn balance discussions into something constructive, and not the current toxic mess they are. I appeal to those powergamers out there that are already doing this to come out and contribute to the community. Finally, remember, do NOT feed the trolls (no matter how irritating).

This is the same **** test you proposed years ago.

Not allowing debuffs is junk. How can you compare class balance if the various classes can't use their debuff abilities?

And Chai makes an excellent point that it's not all about DPS. Being able to avoid damage or heal also matters.

It's okay for a sorc to do the most DPS if he's also more vulnerable to getting killed (glass cannon).

Balance problems comes when a class can do the most DPS AND be the best at avoiding damage.

Just like in the past, you appear to want all classes to do equal DPS. We had this conversation 5 years ago.

[nokowi]

It's facts of war.

https://www.definitions.net/definition/ranged+weapon

And if the melee made it past the front line of opposing melee then the battle was over before the first archer fell.

I am also thinking that those archers didn't put enough points into jump.

Dagg.

Cause D&D is a fantasy game based on facts????

Being an individual ranged combatant is an advantage. Having an army entirely of them is not, with a few exceptions.

Do you understand the difference?

Do you understand making ranged superior in every way is poor design, as it removes that as a good play choice?

Why not introduce tactical nuke one button magic win? Your archers have little/no defense against magic, after all.

[BigErkyKid]

I think they do that to avoid endless criticism and argument about the relevance and accuracy of their data. Just as this thread demonstrates.[....]
Most players arent even competitively racing though. We just want reliable transportation that can safely get through the courses to finish. Or be a great team member that helps our team finish safely. Top performance just seems egotistical or not fun as compared to good performance and skillful or fun play.

I see what you mean. Of course, a lot of people feel threatened by anything that approaches objectivity, for obvious reasons. The amount of misrepresentation of what I said is endless; I never argued that balance should be only about DPS, or even boss DPS. This is a first attempt to get some consensus on SOMETHING.

I want to appeal to the number crunchers, the expert builders, the metagamers, the players who believe that the community in DDO is more than a few trolls trading blows over the forums. This is the time to come out and agree on something constructive.

[BigErkyKid]

Bruntsmash, yes, thumbs up, 100% agree, great suggestion. No "reasonable observer" could disagree with beating down Bruntsmash. Bruntsmash it is. Case closed.

Suggestion, skip "ii) trash DPS" because it can be adequately incorporated into "iii) questing power", then post a thread for "questing power".

I shouldn't be so dismissive, you're passionate about this, it's cool. If DDO was 500+ quests of zoning in and killing one fat mob in 15 to 60 seconds it would seem more vital to me but I think the concept of "questing power" is going to be more useful and generate more interesting discussion when it comes to exploring the idea of build performance. It can be a part of the picture but at least for me it's the smaller part.

For what it's worth I think you can't limit debuffing the way you suggested, what about a sorcerer's awaken elemental weakness? Stricken Form? Conduction? Those are definitely inherent qualities of a build so you can't fully test a build without them. If you're testing a build that uses a weapon I'd go as far to say smart weapon selection and swapping as needed is still a quality of your build as well (and does Quick Draw make a difference), but I could see why some may disagree. Disallowing any build to use the tools at their disposal, like enhancement effects or weapon swapping is not going to give you the same result you'd get playing for real in game which should be a goal - these tests should reflect the in game reality for how players play (while solo, I presume), and no one would knowingly limit themselves that way. As long as you do want to judge solo performance which is my assumption, I agree it is important to disallow buffing from a partner (one of those videos posted earlier looked like the solo guy had the modern day equivalent of the Axer package at the start).

The debuffing part needs some care. The reasons why I suggested it: i) in a lot of scenarios were boss DPS matters, it is likely someone else will provide those debuffs, ii) debuffs typically just scale the DPS proportionately upwards, so they do not matter that much for balance (e.g. salt buffs everyone, as do vulnerability stacks). Makes more sense, now? I think it requires a fine brush and I'd like to have that discussion.

I agree. DPS is one aspect but the ultimate metric in the game is clearing quests quickly.

I think rather than looking at one encounter, ie Bruntsmash, as a benchmark merely for DPS, we should be picking one quest as a benchmark and comparing how fast builds can clear it, on a certain difficulty, while achieving certain objectives (optionals, trapping, conquest, no-death, etc.)

Comparing TTCs will give us a better sense of a build's overall efficacy in real-game scenarios. Crucible might be the best overall test - it really rewards you for overall efficacy in all facets: kill speed, move speed, trap defense, spell defense/saves, ranged, melee. Its rather long for a benchmark though and the maze part is a waste, and some of the stuff is random. I'm not sure what the best test would be since I dont have VIP...maybe one of the new pack, I've seen that they're pretty creative and not just dungeon-crawl DPS tests.

This is interesting, but I suggested starting with something simpler. A more controlled experiment, until we can get better at it. Just look at the amount of noise in the thread.

What we want I can't precisely say. But I'll say what I want, and I suspect a lot of people agree with it.
When we say balance, the exact DPS numbers aren't really what we mean. It is the holistic power of toons, including offense, defense, healing, utility, and speed that we mean.
How do you measure that? Well, you could make some sort of benchmark, but honestly, why bother? Just look at what people are actually running in the most commonly posted groups, which right now is r1 or so, with flex from elite to about r3. Everything else is kinda boutique.

What we want is for people to naturally select a reasonable variety of archetypes. Healing, blasting, shooting, melee, stealth, etc. If our groups are something like 4 inquisitors and 2 sorcerers, well, there's a problem. We don't want the rules to manufacture that meta. Why? It is boring and lame. If the meta gives us a tempest, an assassin, a sorcerer, a cleric or fvs, a warrior, and a warlock though, that's kosher. So we want diversity. Diversity is achieved by having a substantial number of tier 1 classes/builds in the meta and insuring that there's at least something fight-y, something shoot-y, and something blast-y represented there. Besides diversity at the macro level, we also would like to see combined arms---that is, a group of mixed capabilities should achieve better, by and large and on the average, to a one-dimensional group (e.g., all inquisitors). Making pale masters a bit more group-able is a recent change that helps a bit on that level.

I don't disagree, this is the ultimate goal. However, we got to start somewhere, right?

[Thrudh]

I want to appeal to the number crunchers, the expert builders, the metagamers, the players who believe that the community in DDO is more than a few trolls trading blows over the forums. This is the time to come out and agree on something constructive.

But like the other guy said

Most players arent even competitively racing though. We just want reliable transportation that can safely get through the courses to finish. Or be a great team member that helps our team finish safely. Top performance just seems egotistical or not fun as compared to good performance and skillful or fun play.

So not all of us care the same about what the number crunchers, the expert builders, the metagamers think is important.

Me, I like a balanced party...

I'm playing my ranged guy again, to get first-hand knowledge about this change, and I find I enjoy parties with melee and ranged. Easier to get sneak attack, don't have to kite as much, etc. We move plenty fast through the quests.

Maybe not 6 sorcs in one group fast, but pretty fast... Don't need absolute top performance to do well in this game.

[hit_fido]

The debuffing part needs some care. The reasons why I suggested it: i) in a lot of scenarios were boss DPS matters, it is likely someone else will provide those debuffs, ii) debuffs typically just scale the DPS proportionately upwards, so they do not matter that much for balance (e.g. salt buffs everyone, as do vulnerability stacks). Makes more sense, now? I think it requires a fine brush and I'd like to have that discussion.

In scenarios where boss DPS matters wont someone else also be providing buffs? Yet those aren't allowed.

To be consistent you should mandate a solo situation for both buffs and debuffs. Allow whatever that build brings to the table - by itself.

[redoubt]

While I applaud the attempt and transparent testing and consensus, your test itself seems to be skewed.

1. it is solo-centric

2. It is dps-centric

How do we balance a turn based cleric with necro focus? It has zero dps, but that does not mean it is not contributing.

How do we balance a spell singer bard? As a build designed to make other builds better, where does it fit in?

What about a tank?

(I'll pause for now as I don't think more examples are needed just yet.)

Ultimately, my question is: how do we balance for the non-dps builds?

[Potatofasf]

Ultimately, my question is: how do we balance for the non-dps builds?

A good question!

[BigErkyKid]

In scenarios where boss DPS matters wont someone else also be providing buffs? Yet those aren't allowed.

To be consistent you should mandate a solo situation for both buffs and debuffs. Allow whatever that build brings to the table - by itself.

Do you understand my point, though? Salt and vulnerability scale damage up for everyone by x%; hence, they do not matter so much for balance.

By allowing them, this already creates a layer of incomparability (is it a fast stacker of dust/vulnerability or not) that might not be so relevant in a group setting.

Is this point clear? or do you think I am missing something?

[BigErkyKid]

While I applaud the attempt and transparent testing and consensus, your test itself seems to be skewed.

1. it is solo-centric

2. It is dps-centric

How do we balance a turn based cleric with necro focus? It has zero dps, but that does not mean it is not contributing.

How do we balance a spell singer bard? As a build designed to make other builds better, where does it fit in?

What about a tank?

(I'll pause for now as I don't think more examples are needed just yet.)

Ultimately, my question is: how do we balance for the non-dps builds?

Again, it is not solo centric; it is meant to measure DPS output. Please do have a 2nd look at OP.

How do we measure other aspects? We'll come up with other tests! But just because something can't measure everything, it doesn't mean it is flawed to measure what it is designed to measure.

We agree on BOSS DPS rules, we move on to the next one, and so on. The community needs some consensus...just look at the frequency of troll posts.

[Renvar]

I will provide some examples of real diversity.

Scale of 1-10. you get 25 points to allocate.
---------------
Cleric
-----------------
Heal 10
damage dealing 3
damage mitigation 7
crowd control 5
---------------
Bard
--------------
Heal 7
damage dealing 5
damage mitigation 4
crowd control 9
-----------------
Ranger
---------------
Heal 6
damage dealing 10
damage mitigation 5
crowd control 4
----------------------
fighter tank
---------------------
Heal 2
damage dealing 8
damage mitigation 10
crowd control 5

This is real diversity (and why homogenization of DPS as "balance" will fail). Classes are better at different things and all are desired. The problem is in DDO, all are no longer desired, so people measure class performance largely based on DPS with a secondary emphasis on survivability, then demand sameness balance in those areas. When those same people complain about class homogenization, this is a direct contradiction in logic, because it is their very demand of class balance which created the class homogenization in the first place.

In D&D the enforcing the trade off is what keeps the balance and also keeps the diversity. In my example of someone wants their fighter to be a 10 in damage dealing, they have to move points from something else, like falling to an 8 in mitigation to get it.

I agree that this would be real balance. But that's not what we are talking about with MMO meta balance.

Balance is getting corrupted when when you min/max. Which is what meta gamers do. We need 1(2 in raids) heals and 1(2 in raids) CC and 1 damage mitigation (tank). and 3(7 in raids) damage dealing. We want whatever gives a 10 in each. A build that does 5/8/7/5 (or even 5/9/7/4) is "useless" or "unbalanced" because 8(or 9) in DPS sucks compared to 10. It doesn't matter that the build does a 7 in mitigation. We have a Fighter Tank with a 10 for that. Any class/build without a 10 in one category is sub-optimal. If only one or two classes have a 10 in a particular category then they become the meta and the game is homogenized around those 10's.

Universal Trees make it even more complex because they alter the basic class mixes which work pretty much as you describe. The add more to the table than they take off. So using Vistani or Inquisitive, you are basically playing with 26 or 27 points instead of 25.

[Avocado]

snip

Best post of year. Sums up my view of the current state of the game. Balancing to the dps metric is what people are asking and that just shows how far this game has strayed from what made it fun. This game is now all about dps, that is what matters for completions, which is what matters to most of the grinders, who are all that's left in game.
Meaningfull balance at this point comes down to shaving off the bonus that past lives give to 1/3 the current and dropping reaper tree hp bonus to 1 per ap. As well as dropping all melee ranged power enhancements to 25% of current value and spell power to 50%.

#milk the grinder whales is all that's left of ddo I give this game 2 years if it is continued to be ran in it's current state.

[Chai]

I agree that this would be real balance. But that's not what we are talking about with MMO meta balance.

Balance is getting corrupted when when you min/max. Which is what meta gamers do. We need 1(2 in raids) heals and 1(2 in raids) CC and 1 damage mitigation (tank). and 3(7 in raids) damage dealing. We want whatever gives a 10 in each. A build that does 5/8/7/5 (or even 5/9/7/4) is "useless" or "unbalanced" because 8(or 9) in DPS sucks compared to 10. It doesn't matter that the build does a 7 in mitigation. We have a Fighter Tank with a 10 for that. Any class/build without a 10 in one category is sub-optimal. If only one or two classes have a 10 in a particular category then they become the meta and the game is homogenized around those 10's.

Universal Trees make it even more complex because they alter the basic class mixes which work pretty much as you describe. The add more to the table than they take off. So using Vistani or Inquisitive, you are basically playing with 26 or 27 points instead of 25.

The example I provided allows builders to build a 10 in 2 categories. You'd be terribad at everything else though. If thats what the metagamers want, that option (specialist builds) exists in diversity based game balance.

In the example I provided if that bard wanted to be a 10 in healing they would have to lose points elsewhere to do it. They could be a 10 in DPS if they wanted, but would need to sacrifice somewhere else to get it.

It also allows for people to build jack of all trades master of none style builds, for people who prefer a cushion in survivability but also need some (but not maximum) offense. If the metagamers dont invite these to their raids, thats fine, as this crowd typically doesnt favor the same approach of cranking out specialist builds to achieve the fastest completion times. While not necessarily total flower sniffers, they are playing the game at a pace they enjoy, and have the option to create balanced builds with this system.

[DaggomaticDwarf]

Cause D&D is a fantasy game based on facts????

Being an individual ranged combatant is an advantage. Having an army entirely of them is not, with a few exceptions.

Do you understand the difference?

Do you understand making ranged superior in every way is poor design, as it removes that as a good play choice?

Why not introduce tactical nuke one button magic win? Your archers have little/no defense against magic, after all.

Are we talking about DDO or RL? An army of ranged in DDO means almost no threat of harm. We're talking baddies riddled with arrows, bolts, and Shuricans. The only superiority they have is the distance between them and the baddies. Once removed they too will take damage.

Do you understand the difference?

Ranged (Distance between two points) Combat v Melee Combat!

Dagg.

[Ralmeth]

There are too many variables to come up with one, or even a small number of tests, to define balance. Honestly, this would be better done by the Developers who can set-up test scenarios.

[DaggomaticDwarf]

There are too many variables to come up with one, or even a small number of tests, to define balance. Honestly, this would be better done by the Developers who can set-up test scenarios.

There are way too many variables to consider. In a stand up straight DPS test, can the player variable be removed due to auto-attack? No because of active attacks. And then there's Class variable, Weapon variable, Gears variable, Buffs variable, Debuff variable, etc, etc, etc, variable, variable, variable.

Dagg.

[Renvar]

The example I provided allows builders to build a 10 in 2 categories. You'd be terribad at everything else though. If thats what the metagamers want, that option (specialist builds) exists in diversity based game balance.

In the example I provided if that bard wanted to be a 10 in healing they would have to lose points elsewhere to do it. They could be a 10 in DPS if they wanted, but would need to sacrifice somewhere else to get it.

It also allows for people to build jack of all trades master of none style builds, for people who prefer a cushion in survivability but also need some (but not maximum) offense. If the metagamers dont invite these to their raids, thats fine, as this crowd typically doesnt favor the same approach of cranking out specialist builds to achieve the fastest completion times. While not necessarily total flower sniffers, they are playing the game at a pace they enjoy, and have the option to create balanced builds with this system.

OK. I think we agree on that. I think where the OP was going was:

A) How do we evaluate what is a 1-10 in DPS? What measurements can we use to distinguish an 8 from a 9 from a 10?

B) Damage dealing is too broad a description. There is single target and multi-target/AoE DPS at a minimum as sub groups under Damage dealing. A build can be a 10 at single target but be a 7 or 8 at multi-target. (i.e. rogue or insta-killer DC Caster). Conversely, a build can be an 8 at single target but a 10 at multi-target. (i.e Ranger or DPS Sorc or (theoretically) THF with glancing blows).

Once we determine how to evaluate each item, then we can find the "holes". These holes are the lack of balance. An example would be THF Barbarian or THF Kensai DPS build. They can only get to an 8 in both single target and multi-target DPS, at best. Whereas TWF Kensai or Barbarian DPS builds can get to a 9 or 10 (but typically not both simultaneously) categories. That would seem to be an imbalance in the TWF and THF styles that the same build with the same other feats (beyond the THF/TWF lines) would yield the same heals, mitigation, and CC number but a different Damage dealing number. That is a loss for no advantage. That's not at trade off. That's a hole.

The question is, how do we measure the performance with a repeatable testing methodology and thus identify the holes in an empirical way?