Something I learnt is that testers provide a service to the management. The management shall gain confidence of the status of the product. That the tester itself can learn about the product he carries out some test cases. As it is normal, the tester finds some bugs.
What is visible to the management? Often it is a percentage of passed/failed test cases. And now it’s getting interesting. The first question is: When is a test case failed? If the goal of the test case can be reached and no bugs are detected, it’s a pass – that’s mostly clear to everybody, but what status do you set, if you found a minor bug? Passed or failed? Some strict people say, if there is a bug it’s a fail. This might lead to the situation that a few bugs impact a lot of the test cases. This is bad for the visible percentage of passed/failed test cases. Quickly you get unwanted (?) management attention, being asked whether the software quality is bad. For sure it is not, but it looks like.
Asking @alanpage what to do, he suggested making something visible “…something to do with product quality…”. As this is sounding good, I was seeking for some KPIs that can be generated out of our databases (test cases and buglist). Something we can produce quickly is a report about the severity of the bugs. As expected there are no bugs with severity 1 and only a few with severity 2. The most of the bugs are in 3 and a few in 4. A pie chart made visible to the management that there is no urgent matter to take measures as there are no severe bugs.
The management was satisfied for a second. But the “real problem” of a bad passed/failed percentage is not solved. Out of our data bases (test cases and buglist) we would be able to say that one bug affects ‘a number of’ test cases. Like this an impact analysis could be done of which bugs affect which test cases. As a result a hit list (bug with most affected test cases) can be produced. Anyway the management, I guess, needs a chart to understand it.
What would such a chart look like? Suggestion appreciated. 🙂
Or do I need just words to explain the impact of a few bugs to a lot of tests. As always the management is looking for a quantitative answer. I would also definitely feel better making a real and stressable conclusion than just guessing.
Hi, Chris…
I see a number of things that worry me here.
The management shall gain confidence of the status of the product.
That concerns me. Testers aren’t in the confidence-building business. There’s usually plenty of confidence to go around in a project. I’d argue that most of the time, we’re in the business of finding out where that confidence is misplaced by empiricially questioning the product–that is, finding out how it actually behaves, to the best of our ability.
That the tester itself can learn about the product he carries out some test cases.
The idea of “test cases” makes me nervous these days. Test cases (and their results, typically in the form of pass vs. fail results) are the widgets of testing. But testing isn’t about producing widgets. It’s about producing knowledge and awareness; it’s about producing stories about the product, about the testing, about the project, and about risk. When I see a focus on test cases, I mostly see a focus on correctness with respect to a specific question with a binary answer. There’s a place for that kind of question, but each one is like a leaf on a tree. You wouldn’t evaluate the health of the tree by counting which leaves were red and green (some leaves are naturally red); without looking at the leaves as part of a highly interconnected system; and you certainly wouldn’t look at the leaves alone. Most importantly, even if you were focusing on leaves, you would evaluate the colour of the leaves with a great deal of regard to context and to what people actually want.
What is visible to the management? Often it is a percentage of passed/failed test cases.
I notice that in these sentences, there’s no agency. That is, you’re leaving out who makes things visible to managers. One important job for the tester is to make things visible to the managers, things that the managers might not be able to see on their own (they might be busy, or relatively unskilled at a technical level, or awash in other kinds of information; any number of possible explanations). It’s our job not only to present them with information, but to frame that information in terms of the mission of testing. See http://www.developsense.com/blog/2010/09/test-framing/
This is bad for the visible percentage of passed/failed test cases. Quickly you get unwanted (?) management attention, being asked whether the software quality is bad. For sure it is not, but it looks like.
If you’re a tester, it’s not your job to determine whether the software quality is bad. Decisions about quality are the responsibility of those who make decisions about the product—the project managers and the product owners (and, to some degree, the programmers, though I would argue that they, like us, are in service to the product owners). It’s the responsibility of the tester to identify threats to the quality of the product, to the best of the tester’s ability to find those threats and frame them with respect to the product onwers’ goals. Again, see http://www.developsense.com/blog/2010/09/test-framing/
As this is sounding good, I was seeking for some KPIs that can be generated out of our databases…
Whenever I hear or see mention of KPIs, I worry that someone is driving under the influence of some kind of overstructured process model, and typically one that renders information into data, rather than the other way around. That worry may be misplaced, but in my experience it usually isn’t. Instead of thinking of KPIs, try a more natural way of putting it: things managers want to know..
Try this: managers don’t need a chart. Managers need a story that a chart might help to illustrate the story. Look at this http://www.developsense.com/presentations/2011-11-EuroSTAR-Dashboards.pdf, but also look at this: http://www.developsense.com/blog/2012/02/braiding-the-stories/ and this http://www.developsense.com/blog/2012/02/delivering-the-news-test-reporting-part-3/
—Michael B.
To start with, I don’t use test cases. When I notice a page/screen, I check & test for many things. The overall look of the page, the alignment, the consistency, the usability, branding, intuitiveness and so on. I can’t have a test case for each of it.
I have a mindmap highlighting the main pages/functions/features/scenarios as nodes. I get it reviewed by the stakeholders who matter. Then, I use the map as one of the guidelines to guide my testing but its not a rule. If I learn more about the product and value of updating the map is less for the cost of updating, I don’t update the map.
How do I report to the management?
Issue summary – how it might impact customers? Are the programmers aware of the issue & how much time they might need to fix. How much time testers might need to retest & how much to retest. I provide the above information after talking with programmers & testers.
Sometimes, everyone is in a room & the decision happens in a few minutes. So, everyone knows what is expected of each group.
Finally, what is the purpose of the management being aware of the testing progress? To make release decisions based on the information provided by testers. It should not matter how you provide as long as they are able to make the decision.
Counting test cases is like counting pieces of clothing. “I have 125 pieces of clothing” or “50% of my pieces of clothing are green” are both ridiculous sentences if you mix socks and suits together and count them each as one. Like Michael and Ajay, I don’t like the concept of test cases. Human automata come to mind.
Unfortunately, a ratio or a simple number gives the illusion of accuracy where there is none. I am very suspicious of most of the numbers/percentages/etc. that can be auto-generated from a database. Now, I am not going to repeat what Michael and Ajay have already said. Instead I am going to address a possible path for a change.
As a general rule, people don’t want to feel stupid. So don’t tell them that a passed/failed ratio is something only a simpleton would take as a valid measurement of quality. Instead, use humor and let your management discover for themselves. Now here is and idea: Keep the traditional ratios and numbers in your reporting, at least for the moment. But do two more things.
First, start telling stories about your testing activities. Tell them narratively about remarkable findings and how passionate you feel about them. This will show your management that there is more than just passed/failed figures and that your team really cares about the product.
Second – and this is the humor part – you might want to add some ridiculous figures. E.g. percentage of passed test cases whose title start with the letter “p”. It will immediately draw their attention. They will find this figure ridiculous. And you – of course – agree. That’s the start of a discussion where you can lead them to understand the inadequacy of their requested passed/failed ratio.
Being a manager myself, one of the key questions I often want to have an answer for, is this: Do we have a problem here? As soon as you can convincingly convey an answer to that question, you’re back in business.
I like the part on using humor. I’m not sure I can pull that off… but let’s see.
As the other commenters have said: providing more numbers, e.g. a hit list, will probably not solve the problem. Have you asked management what information they would like to have from the test team? And note that is not the same as what information they expect to get from the test team.
Because in my experience management expects numbers (like pass/fail), because that’s what they got in all previous projects. People can really get stuck like that in their expectations, no longer able to realize that things can be different.
Perhaps by asking them what kind of information they would *like* they have (And everything goes here. Think of it as a brainstorming session.), you might get them to think beyond their default expectations about testing.
I also find test pass rates useless – in fact, I think test pass rates are the most used usless metric around. Remember, customers don’t care how many tests passed, what the coverage rate is, or how many bugs you fixed. Customers care if the product works (for them).
With this in mind, my preference is to report test (quality) status in two ways:;
1) Scenarios. A product (or feature set of a smaller product) should have a set of customer scenarios that define how you expect customers to use the product (ideally, vetted with real users and not entirely made up). For products I work on, we have these “epics” that describe the main workflows a customer would go through. My preference is to report against the scenario. If we have 6 scenarios, I have 6 boxes colored red, yellow, or green (and sometimes shades between). Bugs, tests, coverage, code churn, and a bunch of other factors contribute to our confidence level for that scenario. If it’s green, we can explain what we tested, and what we found (and didn’t find) to justify the color-rating. These are “drillable” reports, so we can dig into bug rates or test coverage if necessary, but it’s not the default view (and frankly, uninteresting to senior management.
2) I like to report across half a dozen or so “ilities” (security, privacy, reliability, usability, globalization, etc.) using the same red-yellow-green reporting. Depending on the project, I may merge the reports.