Something I learnt is that testers provide a service to the management. The management shall gain confidence of the status of the product. That the tester itself can learn about the product he carries out some test cases. As it is normal, the tester finds some bugs.

What is visible to the management? Often it is a percentage of passed/failed test cases. And now it’s getting interesting. The first question is: When is a test case failed? If the goal of the test case can be reached and no bugs are detected, it’s a pass – that’s mostly clear to everybody, but what status do you set, if you found a minor bug? Passed or failed? Some strict people say, if there is a bug it’s a fail. This might lead to the situation that a few bugs impact a lot of the test cases. This is bad for the visible percentage of passed/failed test cases. Quickly you get unwanted (?) management attention, being asked whether the software quality is bad. For sure it is not, but it looks like.

Asking @alanpage what to do, he suggested making something visible “…something to do with product quality…”. As this is sounding good, I was seeking for some KPIs that can be generated out of our databases (test cases and buglist). Something we can produce quickly is a report about the severity of the bugs. As expected there are no bugs with severity 1 and only a few with severity 2. The most of the bugs are in 3 and a few in 4. A pie chart made visible to the management that there is no urgent matter to take measures as there are no severe bugs.

The management was satisfied for a second. But the “real problem” of a bad passed/failed percentage is not solved. Out of our data bases (test cases and buglist) we would be able to say that one bug affects ‘a number of’ test cases. Like this an impact analysis could be done of which bugs affect which test cases. As a result a hit list (bug with most affected test cases) can be produced. Anyway the management, I guess, needs a chart to understand it.

What would such a chart look like? Suggestion appreciated. 🙂
Or do I need just words to explain the impact of a few bugs to a lot of tests. As always the management is looking for a quantitative answer. I would also definitely feel better making a real and stressable conclusion than just guessing.