Suggestion for Improving Testing

[Rockcrusher99]

Premise: Better utilize your army of players and the Lamannia server for your testing cycles.

- Create a more formal bug-reporting system (outside the clumsy forum-reporting) where players can test on Lamannia, report bugs, and receive bug-bounties if they find an bug previously unreported. Many software companies successfully utilize security testers to scrutinize their software and pay them bounties as incentive. In your case, your bounties don't have to affect your bottom-line—just offer virtual store items, points, etc. depending on the severity of each issue found.

Note: Make sure "replication steps" are a required data element by the reporter as this is huge amount of a software developer's time. With this onus placed on the reporter, it will allow your developers to quickly verify, etc.

- Once a bug is able to be replicated, it can be formalized, documented, key-worded, etc. so other players can see and search the existing bugs database to avoid duplicate reporting. Keep this ongoing list public and searchable. Yes, there will likely be hundreds, but isn't that the point?

- Allow your players to prioritize the known bugs by voting on those they think are most impactful/meaningful to them. Once you capture what your fans really want fixed, you can merge your internal priorities into your developer's to-do list.

I think such a system would be more beneficial than the existing, informal system. Use the existing account authentication database so players can't cheat the voting system. Decide on a simple system for the voting. Perhaps X votes per player per month?

I would go so far as to say that given internal specs for development of such a system, I bet a few of your fans could develop such a system with placeholders where you could plug in your own authentication and database calls where needed.

[LeoLionxxx]

- Create a more formal bug-reporting system (outside the clumsy forum-reporting) where players can test on Lamannia, report bugs, and receive bug-bounties if they find an bug previously unreported. Many software companies successfully utilize security testers to scrutinize their software and pay them bounties as incentive. In your case, your bounties don't have to affect your bottom-line—just offer virtual store items, points, etc. depending on the severity of each issue found.

Not 100% sure what you mean by "clumsy forum reporting". Do you mean people just reporting bugs on the forums?

There is an offical bug reporting tool which was updated not too long ago. IMO it is rather hard to find*, and is cumbersome to fill out. I would be very happy to see a more streamlined version (and one which auto-filled with player and /loc info if they opened it in-game)

https://help.standingstonegames.com/...form_id=438007

I'm sure they can't expose a full list of known issues, lest exploits become known, but I like the idea of having a public-facing list of known issues, and players having the ability to up-vote and provide additional information on such things.




*When reporting a bug I do not want to "contact support" or "open a request" - I want to tell someone about a bug and move on with my day.

[Arkat]

My suggestions would be:

1. Hire a QA staff.

2. Pay them more than minimum wage so you can keep them.

[Eantarus]

My suggestions would be:

1. Google "software development best practices".
2. Do some of them.

This will improve the software testing and release cycle vastly.

[Drunkendex]

My suggestions would be:

1. Hire a QA staff.

2. Pay them more than minimum wage so you can keep them.

You're unreasonable...

that costs money.

Better to release untested patch and then go full crunch culture on fixing it.

Honestly i feel sorry for most of SSG team, whoever makes decisions fails utterly at their job and rest of the team gets flak for it

[Assassination]

My suggestions would be:

1. Hire a QA staff.

2. Pay them more than minimum wage so you can keep them.

+1, oh yeah and actually do some testing.

[LightBear]

Seeing that we have so many QA experts on this board.

How would you folks conduct the testing for his game?
Please be as elaborate and detailed as possible in your descriptions.

[Martininice]

Premise: Better utilize your army of players and the Lamannia server for your testing cycles.

- Create a more formal bug-reporting system (outside the clumsy forum-reporting) where players can test on Lamannia, report bugs, and receive bug-bounties if they find an bug previously unreported. Many software companies successfully utilize security testers to scrutinize their software and pay them bounties as incentive. In your case, your bounties don't have to affect your bottom-line—just offer virtual store items, points, etc. depending on the severity of each issue found.

Note: Make sure "replication steps" are a required data element by the reporter as this is huge amount of a software developer's time. With this onus placed on the reporter, it will allow your developers to quickly verify, etc.

- Once a bug is able to be replicated, it can be formalized, documented, key-worded, etc. so other players can see and search the existing bugs database to avoid duplicate reporting. Keep this ongoing list public and searchable. Yes, there will likely be hundreds, but isn't that the point?

- Allow your players to prioritize the known bugs by voting on those they think are most impactful/meaningful to them. Once you capture what your fans really want fixed, you can merge your internal priorities into your developer's to-do list.

I think such a system would be more beneficial than the existing, informal system. Use the existing account authentication database so players can't cheat the voting system. Decide on a simple system for the voting. Perhaps X votes per player per month?

I would go so far as to say that given internal specs for development of such a system, I bet a few of your fans could develop such a system with placeholders where you could plug in your own authentication and database calls where needed.

You are missing the mark. Player feedback on Lamannia is constantly ignored, even when it comes to major problems. It is not a matter of formalized processes, it is a matter of "yeah, whatever, we are shipping anyway" attitude.

[Martininice]

You're unreasonable...

that costs money.

Better to release untested patch and then go full crunch culture on fixing it.

Honestly i feel sorry for most of SSG team, whoever makes decisions fails utterly at their job and rest of the team gets flak for it

yeah, I doubt any dev is happy releasing broken updates. Management types who contribute nothing to a product rush things, do not care fixing trash will eat up more time than doing things properly, do care only for the quarter, burn out the employees who are actually important, make customers constantly unhappy. Even worse in US working culture where the boss is master I imagine.

[axel15810]

Not 100% sure what you mean by "clumsy forum reporting". Do you mean people just reporting bugs on the forums?

There is an offical bug reporting tool which was updated not too long ago. IMO it is rather hard to find*, and is cumbersome to fill out. I would be very happy to see a more streamlined version (and one which auto-filled with player and /loc info if they opened it in-game)

https://help.standingstonegames.com/...form_id=438007

I'm sure they can't expose a full list of known issues, lest exploits become known, but I like the idea of having a public-facing list of known issues, and players having the ability to up-vote and provide additional information on such things.




*When reporting a bug I do not want to "contact support" or "open a request" - I want to tell someone about a bug and move on with my day.

An easy and definitely in game bug reporting system would help a lot. I rarely report bugs that I see because it's too cumbersome to fill out the online form. An easy in game reporting tool that auto captures all your relevant info like race/build/server/location etc and maybe could also auto import CTRL + P screenshots would have me reporting a lot more and I'd imagine it'd be the same for many players.

[Tyrande]

IMHO, they need to hire automation (or SDET - Software Developers in Test) engineers or have developers actually write unit and integration tests and employ Behavior Driven Development approach. I know this is an UI game, but at least they could automate the back-end which maybe using REST or SOAP calls? Especially for the item database/bank since this is the bug-fest area?!?

Every time someone builds the code in their branch, an automated smoke would run, so developers can find out what their merge request broke immediately instead of waiting till integration.

Every time they release a tiny patch (x.x.x) , the set of mini-smoke automation should be run.

Every time they release a minor patch (x.x), the set of smoke automation would run.

Every time they release a major patch or expansion (x), the set of mini-smoke, smoke and regression automation would run.

They do not have the staff or money to do manual testing, might as well have someone already write scripts and automate the testing to save efforts.

Irish in Ireland, Polish in Poland, Ukraine, Indian engineers (from India) are probably the most affordable...

Of course, for defect tracking and management, there are affordable options like JIRA (free for commercial use under 10 people)

[LightBear]

IMHO, they need to hire automation (or SDET - Software Developers in Test) engineers or have developers actually write unit and integration tests and employ Behavior Driven Development approach. I know this is an UI game, but at least they could automate the back-end which maybe using REST or SOAP calls? Especially for the item database/bank since this is the bug-fest area?!?

Sometimes I get asked which client version I use to run the game.
I have enough knowledge to know which type of person is asking that type of question and what they are doing.

I have zero interest in running the game that way, but yeah apparently this game isn't that hard to dissect and send commands to the server in some out of the official game client kind of way.
Or at least not for the type of nerds that this game attracks.


It would be interesting to start writing some Given-When-Then style tests for this game.


Given I am <sexe> of <race>
And have <levels1> of <class1>
And have <levels2> of <class2>
And have <levels3> of <class3>
When I advance One level of <Extra-Class-Level>
Then I get <New-Class-BAB>, <New-Class-HP>, <New-Class-Feats>, <New-Class-SP>, <New-Class-Spell-Slots>, <New-Class-Spells>, <New-Skill[]>, <New-DC> etc etc etc

|INSERT PIPE TABLE HERE|

Edit:
The one thing I found that humans have trouble with understanding is the concept of null and zero, followed by one and first, to a lesser degree comparing type and value although important when transformations and/or integrations are at play as what is considered to be at place zero can all of a sudden be at place one. Simply because zero doesn't exist at the other side of the transformation. And visa-versa.

So yeah, my advice is try and find where stuff like transformations take place, and look at places where enums, lists, arrays and maps are used or created based on data from one layer in the stack passed to the next layer in the stack.

[myliftkk_v2]

Sometimes I get asked which client version I use to run the game.
I have enough knowledge to know which type of person is asking that type of question and what they are doing.

I have zero interest in running the game that way, but yeah apparently this game isn't that hard to dissect and send commands to the server in some out of the official game client kind of way.
Or at least not for the type of nerds that this game attracks.


It would be interesting to start writing some Given-When-Then style tests for this game.


Given I am <sexe> of <race>
And have <levels1> of <class1>
And have <levels2> of <class2>
And have <levels3> of <class3>
When I advance One level of <Extra-Class-Level>
Then I get <New-Class-BAB>, <New-Class-HP>, <New-Class-Feats>, <New-Class-SP>, <New-Class-Spell-Slots>, <New-Class-Spells>, <New-Skill[]>, <New-DC> etc etc etc

|INSERT PIPE TABLE HERE|

Edit:
The one thing I found that humans have trouble with understanding is the concept of null and zero, followed by one and first, to a lesser degree comparing type and value although important when transformations and/or integrations are at play as what is considered to be at place zero can all of a sudden be at place one. Simply because zero doesn't exist at the other side of the transformation. And visa-versa.

So yeah, my advice is try and find where stuff like transformations take place, and look at places where enums, lists, arrays and maps are used or created based on data from one layer in the stack passed to the next layer in the stack.

Huh? The item/bank issues have always had zero to do with clients outside of them just being a ui. Those problems were always just indicative of broken units of work because they DDO was created by developers that have zero software knowledge outside gaming. In other words, gaming being the typically ignorant backwater it is for software devs has resulted in essentially the same problems that other industries stamped out, thriving. Fun stuff if one knows how dbs work under the covers though. It's silly though, asking the vast majority of gaming devs to understand engine mechanics and database locking schemes. Might as well ask a modern graphic designer to explain how adobe illustrator works under the covers.

But, to your point. If DDO did in fact let unauthorized clients (as opposed to just the custom launchers) make successful calls to the game server (unless is was an open API situation, but arguably that has to be secured by token), that would be a much, much bigger problem than testing. We would all hold onto our credit card digits at that point because there's no telling what could happen.