Posted On:  August 6th, 2010
Posted By:  Tolero

Recently a discussion came up on my twitter feed about giving people examples of good and bad bug reports that players submit to us. After all, bug reporting is similar to actionable feedback - it’s not something that is instinctive to the average consumer. Now I could cull through our bug reports and find some great and terrible player-entered bug reports to dissect, but I didn’t think that was fair. Instead, I’m going to pull a very special bug report. Journey with me back in time…

The year is circa the winter of 2006. The quest is Threnal, with the dreaded Coyle. And whose bug report do I see? My own. Yes, that’s right, I’m going to take a bug report that I wrote back when I was a player and critique it. We’ll look at what I did wrong, what I did right, and how I would do it differently now.

Below is my bug report. Some sections have been blanked out for privacy reasons. The important part to pay attention to is the text in yellow. The yellow text is the player entered data (i.e. that I wrote myself).

Tolero’s Awesome/Awful Bug Report

 

- - - - - - - - - - - - - - - BEGIN BUG REPORT - - - - - - - - - - - - - - -

DATE: December, 2006

BUG ####: Mephits in Hold for Reinforcements bugged in Threnal

Frequency: Always
Character name: XXXXXXXX
Character race: XXXXXX
Class/Level: Bard 9, Fighter 1
Location: Inside the instance for Holding for Reinforcements in any portion of the instance.
Server: XXXXXXX

Description: I’m well aware that the mephits were ‘beefed up’ recently, but I’ve run Hold for Reinforcements literally 14+ times tonight and each time with same result, with different groups. Mephits are spawning and walking through directly to Coyle past a blanket of the following spells all applied: hypnotise, web, solid fog, glitterdust, and fascinate with a 28 CHA on my 10th level character, while blind? This was on normal, not hard or elite. Once Coyle is dead, the mephits resume being fascinated, blinded, webbed etc, but only after Coyle is dead.

Steps to Reproduce:
Charisma 28 on 10th level character
Cast web; cast glitterdust; cast solid fog; cast hypnotic pattern - repeate indefinately or until mana is gone
Herd Coyle to opposite side of where Mephits are.
Start to play Fascinate targetting mephits and targetting myself (either way same result)
Mephits get the icon for blind and proceed directly for Coyle even before Coyle has attacked them.
Mephits do their area affect and kill Coyle, then immediate become fascinated afterwards… even though fascinate was played way earlier.
Same result with multiple groups.
Tried disbanning group and reforming, same result.

- - - - - - - - - - - - - - - END BUG REPORT - - - - - - - - - - - - - - - -

Now… I pulled my bug report up last night to see what I had to work with today for this blog post. Apparently it was the only bug report I ever wrote as a player. It’s been years since I wrote that. Reading it now… I have no idea what I was even talking about. So now it’s time to pick it apart!

 

Bug Writing Tip 1: Good titles

Bug titles are very important. If someone needs to read it quickly, it should tell them what is going on in a nut shell.  Think of a bug report title like a sentence that needs to be completed, like a madlib:
The ____ in _____ is ______
or even
The [noun] in [place] is [verb/adjective/adverb]

The way I wrote it is a poor way to write a title. I only hit 2 out of 3 needed pieces of important information.

• “Mephits” - ok good, something to do with the mephit monsters
• “in Hold for Reinforcements/Threnal” - ok good, this covers where the problem is in what specific quest, I can read the bug to figure out where exactly (I hope)
• “bugged” - BZZZT! Bad Tolero!

Describing something as “broken” “bugged” “not working” “needs to be fixed” “borked” “hosed”  etc. in the title is as informative as using unrelated words like

“The Mephits in Hold for Reinforcements like bananas”
“The Mephits in Hold for Reinforcements are pretty”
“The Mephits in Hold for Reinforcements read my fortune”

If the thing I was writing about was working as intended, I wouldn’t need to write the report would I? So saying that it’s “bugged” is redundant and uninformative. Instead, I should try to summarize what the mephits are doing that caused me to write the bug report in the first place.

Bug Writing Tip 2: Location, location, location

Where a bug is happening is important. It’s not just about what the quest is, it can be what part of the quest too! I clarify in my bug report that I experienced the behavior throughout the quest instance, and what the name of the quest was. That’s good to know. Always try to describe the where as best you can. What quest was it? What part of the quest chain was it? What part of the quest itself was it, was it the boss fight, or down a certain hallway? Do your best to pin point where it’s happening. This is important for the QA folks so they can go there and try to get the behavior to happen too. They need to be able to see what you saw.

Don’t underestimate location too. Very often people will write bug reports after the issue is passed or they have logged out for the night. You may not be standing in the same place that you were when the issue was happening! Always try to provide landmarks and locs.

Bug Writing Tip 3: Good Descriptions

My first mistake with my description was writing when I’m angry. I can tell that I was writing this immediately after a frustrating time questing. This means I’m prone to over exaggeration, leaving out important details, and generalizing. Always take a deep breath, step away from the keyboard, or rant to a friend to get it out of your system first before writing a post or a bug report to avoid saying things you’ll regret, or forgetting to say things that you needed to!

My second mistake is not describing what I expected to be happening instead. This could have been a user error issue - maybe mephits are immune to all those spells? No matter how bustificated something is, or how obvious you think the “right” way of something working should be, always include what behavior you expect to be happening vs what you saw happen.

I do at least include important details like what spells were involved, what my character abilities were, what the difficulty of the quest was, and what the mephits were doing. These are good details to include… sometimes it could be a particular ability or spell that is triggering shenanigans. I include it in the repro steps, but it’s a good idea to include any actions that helped (or didn’t help as the case may be) to work around the bug.

Bug Writing Tip 4: 1, 2, 3, observe, expect!

Now this part isn’t terrible on my part. Break it down step by step - what happened, what were you doing, what was the result, what should someone look for who is trying to see what you saw? In particular, call out the points when someone should notice the wrongness happening. A good call out word is “observe” followed by a description. This tells the tester when they should see the bad behavior, and what it will look like.

What I missed was including a report for the tester of what behavior was expected to happen after the steps are completed. This is important info! It often tells the tester whether it was user error or game error. For example, let’s try a very simple real world example of “steps to reproduce” and how knowing what the user “expects” can make the difference:

Step 1: Turn faucet handle.
Observe: Water will begin to flow out of the faucet.
Step 2: Place glass under faucet.
Step 3: Fill glass with the flowing water.
Step 4: Drink from the glass full of water.
Observe: The water is hot.

Now this bug report lacks some context without the following important piece of information at the end:

Expected behavior: The water should be cool and refreshing.

So seeing a bug report like this, it’s reasonable for a tester to assume that the problem is likely that the user turned on the hot water handle instead of the cold water handle. This is of course an over exaggeration example, but always remember that no matter how obvious something is to you, it is a good idea to specify exactly what you think should have happened.

If I had To Do It Over Again

If I were to rewrite this bug of mine, I would write the following instead:

- - - - - - - - - - - - - - - BEGIN BUG REPORT - - - - - - - - - - - - - - -

Date: December 2006

BUG ####: Mephits in Hold for Reinforcements ignore crowd control spells

Frequency: Always
Character name: XXXXXXXX
Character race: XXXXXX
Class/Level: Bard 9, Fighter 1
Location: While protecting Coyle in Hold for Reinforcements anywhere in library room
Server: XXXXXXX

Description: The mephits in the quest are ignoring crowd control effects, even on normal difficulty. Spells such as hypnotism, web, solid fog, glitterdust, fascinate, and blind are being ignored by the mephits and they walk through the spells unaffected. The mephits go right for Coyle every time even though Coyle hasn’t attacked them. Once Coyle is dead, all of the crowd control effects kick in on the mephits that should have kicked in earlier. This happened no matter what class make-up was in the party. Happened in multiple quest attempts - was never able to complete the quest as a result.

Steps to Reproduce:
Step 1 - Enter quest on normal with a level 9 Bard. Boost spell save DCs as high as possible
Step 2 - Before fight begins, cast several AOE crowd control spells: web, glitterdust, solid fog, hypnotic pattern. Continue to cast these spells throughout the fight.
Step 3 - Herd Coyle to the opposite side of where Mephits are. Try to get him as far from them as possible.
Step 4 - Target a mephit and play fascinate
Observe - Mephit is not fascinated.
Observe - Mephit passes through the crowd control spells and heads directly to Coyle.
Observe - Mephits that pass through the glitterdust spell get the “blind” icon over their heads. The Mephit continues to go straight to coyle undaunted.
Step 5 - Continue to try and crowd control the mephits until they manage to kill Coyle.
Step 6 - Coyle is killed by mephits.
Observe - Mephits will now be affected by the spells that were already cast on them. Sometimes even after the spell should have expired already (fascinate).
Expected: Mephits should be hindered by the crowd control spells. If mephits are resisting the spells, they should have the “save” icon appear over their heads. 
- - - - - - - - - - - - - - - END BUG REPORT - - - - - - - - - - - - - - - -

No one expects that anyone is ever “good” at writing bug reports in the same way that not every person is “good” at giving feedback. But it’s never too late to make adjustments and improve. Back then, I had no clue how to really write a bug report. I vaguely remember going to the forums and reading about how to write bug reports. It is something that is easy to learn though, and looking at the bugs I’ve written since, it’s much better than the days of “mephits bugged!”