Effective Test

It’s a sad truth about testing software that it can be done on a very, very low level. While in software development you need some basic skills such as programming languages and logic to do your job, everybody who is able to push a mouse is also able to do some kind of poor testing, i.e. following step-by-step instructions outlined in a detailed test plan, which is based on some basic requirements. But isn’t testing more than filing tests cases, counting bugs, do some statistics and push a mouse? Won’t you need more than some unskilled testers and a manager who does the administration part? I think there is much more.. and I will try to define some – surely not all – criteria for good and professional testing.

Software systems are complex – they will even be more complex if changing customer needs and market forces are taken into account. Therefore the number of possible and thinkable tests is very close to infinite, so the challenge of testing is to focus on the important ones. Good and professional testing means to identify and choose a sound and feasible selection of tests – both by sophisticated methods and by gut feeling. Furthermore testing is a learning experience: while testing you will learn and explore more about the product, the technology, the processes, the market and the stakeholders. Learning, exploring, anticipating problems and selecting a doable and efficient set of tests from an infinite amount: that’s – from my point of view - the gist of testing!

Here are some criteria to determine how a testing group performs with respect to these testing basics:

Risk Management and Confidence:

To focus testing on risky areas and to find the important bugs, you had to practise some kind of Risk Management. Testing and Risk Management are twins; they belong together. But Risk Management has to be accompanied by something which may be called Confidence Management: are there topics on which you can (hopefully) rely on, which you can take as granted? It’s as important to identify these areas of confidence than to identify the risks, otherwise you will be swamped with risks.

To do appropriate Risk Management and to spot the areas of risk, you have to examine the whole process of software development, e.g.:

• Are there specifications? Are they complete, understandable and up-to-date? Did those people who wrote them really understand the needs of the customer? Did the customer himself understand his requirements – and did he communicate it properly? Did development get the message? Have the market changed meanwhile? What are the competitors doing?
• Does development uses new technology? Do the developers have experience with this technology? Are there implications for testing? Are the developers being familiar with the code? Or is the code heavily patched and poorly documented? Are there still developers who designed or wrote it in the past?
• Does development run a reliable Configuration Management? Or will a new build contain old bugs that had been fixed some builds ago? Must there be an exhaustive regression test for each build? Or will a bug once fixed remain fixed?

There are much more topics to keep in mind and to question than bugs in the code. The testers always have to decide – based on data and their experience - whether they rely on a topic or whether they will question and test it thoroughly. This approach also requires the ability to refocus testing in a flexible way, if the test results show that a risk has been overseen or underestimated.

By identifying areas of risk and confidence the testers gain some impact on Quality Assurance, i.e. the improvement and tuning of the whole software development process. For example if the builds and handovers to the testers are chaotic, the Configuration Management may be screwed up. It’s also the job of the testers to identify and name this kind of internal problems, and contribute to its solution.

Mission:

However your mission statement may look like – a testing group’s mission may be more proactive or more defensive. Proactive testing tries to improve quality continuously, defensive testing aims at assuring a given level of quality. Proactive testing focus on anticipating problems which may occur at customer’s sites or within the development organisation itself, defensive testing tries to mitigate problems, which are already at hand. Furthermore defensive testing is often related to a defensive cover-your-back kind of corporate culture. However if there are nasty customers and litigation is impending, a defensive approach is an appropriate one. Identifying a sound and feasible selection of tests from a infinite number of possibilities is a proactive approach.

Links:

In order to anticipate problems, select the right tests, find the important bugs, identify risks and mitigate them, the testing group requires many sources of information, such as the customer, user groups, development, support, technical writers and all kinds of stakeholders. There should be a wide range of formal and informal communication channels and no restrictions by hierarchical or departmental boundaries. It’s crucial that the testers aren’t restricted to one model and one point of view about what the software is supposed to do – any requirement document will be incomplete, biased and always "under construction". A testing group which bases it’s efforts only on some specification documents can’t work in the most effective way.

Skills:

If a tester requires nearly no skills to do some poor testing, which skills will be required to be an excellent tester ? Here’s an incomplete and preliminary list:

• Testers require domain knowledge and have to be at least as skilled as an expert user, not only as a novice one. What's an expert user depends on the product, e.g. a network management program has to be tested by people that know the daily business of and have similar skills like a network administrator.
• QA people ought to have a broad and comprehensive knowledge in IT (for example different operating systems, networks, Internet, databases, programming languages, OOA/OOD …). They mustn’t be experts or gurus, but they should be able to warm up fast in many fields.
• Good writing skills are mandatory, for example for writing test cases and error reports. Writing understandable and repeatable test scripts, which also contains information about the idea and the intention of the test, the expected results and technical background, isn’t easy. There's also a fine line between too detailed test scripts, which are difficult to maintain, and too superficial ones.
• Quality Assurance is related to development, support, technical writing and field service. Therefore QA engineer should have some knowledge and experience in at least one of these areas. There shouldn't also be any borders or constraints that prevent or discourage QA people to switch to development, technical writing, field service or support - and vice versa.
• Last, but not least QA people need to be good communicators and team players. They must be diplomatic (but sometimes tough), convincing and able to present.