Menu:

Home

About Quality Assurance

Let’s assume all stakeholders (development, QA, writers, support and other field people) agreed on  specification, schedule and common processes, field bugs were analysed, quality requirements defined and some risk management was already done. The development team was put together and has started designing and coding. It’s now time to form the QA team and focus - beside reviewing the design - on one of the main tasks of QA: to make a statement about the quality of the product as soon as possible after the first release. This statement is needed for further planning and project management, one of the worst things which could happen is to discover serious quality problems too late. Therefore QA has to develop tests that are comprehensive and can be done in a limited time – a contradictory task.

To do so I suggest to write tests which imitate typical user tasks, i.e. to start with task-based, use-case and scenario testing. These kinds of tests are cross-functional, so pure functional testing – that’s what a QA uses to do most of it’s time – is neglected during this early phase of testing. To neglect functional tests may sound a little bit revolutionary, but there are two reasons why you can afford to do so: first cross-functional testing will reveal many functional bugs “incidentally” – and these bugs will be the important ones from a customer’s point of view ! Second to find functional bugs is mainly the job of the development team. If a developer checks a new function after he or she coded it, most functional bugs will be found and fixed in development – provided that development will have been granted the time to do so. This idea – to give developers the time to test their own code – is both trivial and revolutionary. I argued before that no developer, who spend a lot of time and effort to become an IT professional and who expects to get some professional satisfaction by his job, wants to deliver shoddy software. Why not giving him the chance to do so? The domain of Quality Assurance are the more complex bugs, those bugs which show’s up under stress and load, in an special environment or if there is a special sequence of actions, so let’s focus QA on this kind of bugs.

By the way: there’s another advantage of starting with task-based, use-case and scenario testing: there will soon be a software build which may be shown (but not handed over!) to customers, e.g. on fairs. This will result in an early feedback from customers and may result in further improvements. Of course there will be only some tested straight-forward ways to run the product now, but that’s sufficient to show the product by some experienced guys, who knows which functions they are allowed to call without unveiling bugs.

The idea to start with the more complex kinds of tests soon may raise the objection that there won’t be a 100% test coverage for functional tests. This leads to one of the most crucial questions in Quality Assurance is: how much testing - or how much quality - is enough? When will QA have done its job? How should quality be measured? I think the most feasible and pragmatic approach is to decide on these questions, i.e. to agree on the areas to be tested, on the depth for each test area and on the criteria for good (-enough) quality. For development, technical writers, support and field folks are involved in laying down the job and targets for QA, such a decision should be sensible and realistic – for example the support and field people are the best advocates for quality, because they would suffer themselves, if quality was shoddy. As I mentioned in the last chapter, one of the six tasks of the core team will be to determine what quality means for a given product. Based on their work and on the specification, QA will create a first plan what should be tested, which methods should be used and what should be the exit criteria. This plan will be discussed with development, tech writers, support and field people - this will be an iterative and ongoing process. By the way it's hard for members of the development project to criticise the criteria for shipment after they were involved in laying down these criteria.

To document their tests and make them repeatable QA people designs test cases (by the way: German testers use to translate "test case" by "Testfall" - a very odd and bureaucratic word). Writing good test cases is a rather demanding task, because good test cases should fulfil the following requirements:

• the idea, goal and target of the test must be clear,
• they have to be easy to maintain and to update, for example there shouldn't be descriptions of details which changes frequently,
• all requirements and prerequisites are described,
• because development and support will review them, test cases have to be understandable not only to Quality Assurance people,

A good test case looks like an onion: while the outside skins contain all information an expert tester or a reviewer from development and support will require to perform or to review the test (idea, goal, use-case, scenario, …), the inner skins give more detailed information how to run it and additional technical information. That's another argument for task-based, use-case and scenario testing - it's easier to write and to maintain test cases for these kind of tests than it is for isolated functional tests. For example a well-written task-based test will remain more or less unchanged if the GUI is redesigned, while a detailed functional test ("Open the Tools box, click on …") has to be rewritten completely.

To write sensible tests QA people require good writing and presentation skills. Here's a list of all skills Quality Assurance Engineers require to do their job:

• Testers have to be at least as skilled as an expert user, not only as a novice one. What's an expert user depends on the product, e.g. a network management program has to be tested by people that know the daily business of and have similar skills like a network administrator.
  
• Testers require domain knowledge, that means they have know the domain the software was written for. E.g. for testing accounting or banking software you need to have some accounting and banking knowledge.
  
• QA people ought to have a broad and comprehensive knowledge in IT (for example different operating systems, networks, Internet, databases, programming languages, OOA/OOD …).
  
• Good writing skills are mandatory, for example for writing test cases and error reports. Writing understandable and repeatable test scripts, which also contains information about the idea and the intention of the test, the expected results and technical background, isn’t easy. There's also a fine line between too detailed test scripts, which are difficult to maintain, and too superficial ones.
  
• Analytical skills are absolutely mandatory.
  
• QA people need to work in systematic and pragmatic way.
  
• Quality Assurance is related to development, support, technical writing and field service. Therefore QA engineer should have some knowledge and experience in at least one of these areas. There shouldn't also be any borders or constraints that prevent or discourage QA people to switch to development, technical writing, field service or support - and vice versa.
  
• Last, but not least QA people need to be good communicators and team players. They must be diplomatic (but sometimes tough), convincing and able to present.

The most important skill is the ability to choose the relevant tests from the huge bulk of tests which are thinkable and imaginable. There won't be enough time to test everything, but there is always some danger to miss important bugs. You'll need a contradictory mixture of technical and business knowledge, of experience, intuition and pragmatism to be a good tester.

Don’t establish an “up and out” scheme, i.e. you can’t stay at QA, if you don’t move up in hierarchy (many consulting companies follow such a scheme). QA shouldn't become a dead-end job for the majority of the testers. If the there’s only one career path in QA, i.e. to become a manager, joining QA or staying in QA won’t be an attractive option. You can’t expect good quality assurance if QA only is a transit camp for people on their way to become developers, managers or field people.

It’s impossible to write something about Quality Assurance without mentioning test automation, which has been a buzzword for a long time. I don’t want to discuss this topic in detail now, but I offer five statements:

• Test Automation means software development, even if you “only” intend to write some scripts. Automated tests have to be much more stable, reliable and robust than the software to be tested. Therefore you need QA people with a development background and you need methods from software development to create sound automated tests.
• Automated tests have to be maintained. If the costs of maintenance are too high, test automation won’t pay off.
• Test automation is always an investment which (hopefully) will pay off later on. You have to be granted the resources to do so - that’s a management decision. Therefore support from management is a crucial prerequisite to start test automation.
• Only simple routine tests (smoke tests, regression tests) can be automated with a reasonable effort. Therefore the main benefit of test automation is to set free resources for more demanding manual tests.
• It’s very hard to recognise error patterns automatically – and it’s expensive to write tests, which are able to do this. Human beings are much more effective in pattern recognition than computers - that’s why graphical user interfaces have become popular. Especially to evaluate the output of a GUI based product automatically is rather odd, for this product was designed to be used by a human user with his/her superior visual pattern recognition ability.

Tests automation projects usually are smaller than product development projects, but require higher flexibility and early results (i.e. useable tests). Therefore test automation may be the right kind of projects to experiment with slim and flexible development methods and processes such as XP, Scrum and Crystal.

Let me finish this chapter with some more “philosophical” remarks about what Quality Assurance is about – or what it may be about.

Quality Assurance never starts on a clean slate or happens in a cleanroom, QA reacts sensitive to the environment it’s embedded in. For example the way we do Quality Assurance today is influenced by it’s roots and history. These roots and history may be a valuable heritage or a burden. For example there are concepts of Software Development that have their origin in industrial production. Think of an assembly line with some guys at the end, holding a long checklist in their hands and checking the final product. The idea, that software may be produced like any other industrial product was popular for several years. While there are discussions in industrial production whether taylorism is up-to-date yet and whether the assembly line may be replaced by better concepts, it still survives somewhere in software development (think of the waterfall model…).

There also are different approaches to testing depending on whether the product has a command line or more modern graphic user interface, or whether the software runs on a host or a client-server environment or is internet-based. To test modern software (graphical user interface, internet-based, OO…) in an old style (e.g. command line interfaces, host, ..) won't work well. Also QA culture is a subject of change: there’s an ancient legend about the “Black Team” at IBM in the sixties, where the new QA group developed it’s own culture of being nasty and hideous destroyers and code breakers (therefore they were dressed in black). But I think, that this kind of QA culture won’t fit in our time anymore.

There may be an objection that what I suggest isn’t an independent QA, i.e. testers and developers may become too familiar with each other and so may accept bugs they shouldn’t accept. But if testers become more familiar with developers, the developers also become more familiar with testers – there is a chance that developers will adapt to the thinking and attitudes of QA. Therefore you may add “strong personality” to the skills required from QA people. Second there is still need for an independent Quality Assurance in case of large-scale projects, where software from several internal and external development teams have to be integrated and tested. But that’s another story – my point is that especially small- to medium-size development projects are difficult to manage because you can’t use the full range of clumsy methods, processes and tools described in tons of books and taught in many seminars and workshops. If speed and flexibility matters, running a software development project with developers, testers, writers and support people being integrated from the very beginning seems to me a more sensible and effective approach. The graphic on the following page suggests that there are three options to place Quality Assurance within the development process chain. The first two options – Quality assurance is close to or part of development or field organisation – doesn’t require an independent QA department.

Project and Risk Management :

As I mentioned before, I’m addressing small- to mid-size projects, which are difficult to manage because you have to find a way between organisational overkill and chaos. To do so I suggest applying a lean version of Eliyahu Goldratt’s “Critical Chain” project management approach combined with proper Risk Management (which I stole from Tom DeMarco) – both items fit together seamlessly.

The basic idea of the Critical Chain concept is to remove (hidden) time buffers from single tasks, collect them and put them in a global buffer at the end of the project. By adding buffers to each (sub-) task you will loose some time, if the buffer isn’t needed. Adding all buffers to a global one will make the whole project more transparent: especially you now are able to determine by a look on the global buffer whether the project is on track or whether the global buffer is used up too fast.

Why should this work in a better way? The more “traditional” approach is to divide the project into sub-projects and tasks, look for the critical path, define milestones and check, whether a milestone is reached in time or not. There are also plenty of project management tools that support this approach. But there’s a hidden disadvantage of this model: because the milestones have to be reached in time, the project members will add buffers to each task. If a project leader asks a team member, how long it will take to finish task X, the answer may be 5 days, although 3 days would be sufficient. The team member added a hidden buffer in case their will be unforeseen problems. The project leader may add another two days to be on the safe side. In case everything runs smoothly, 4 days were lost. This game is fostered by an environment, in which defensiveness, risk-avoidance or even intimidation are part of the corporate culture.

It’s important to be aware of the fact that there aren’t fixed milestones anymore after applying critical chain: each time a part of the global buffer is used, the dates of all remaining milestones will shift, therefore there are no fixed deadlines due to milestones. That doesn’t matter, because now the usage of the global buffer is the critical observable. This must be clearly communicated the first time this approach is applied, otherwise there will be confusion and misunderstandings.

Furthermore there’s another disadvantage of the milestone concept, which will have an unfavourable effect especially in smaller projects. Because there are often parallel tasks, a team member may be involved in more than one task at the same time (that’s less probable in large-scale projects). Frequent task switches are not for free, to avoid them is another way to increase effectiveness. By focusing on buffers instead of milestones you will gain more degrees of freedom to avoid fragmentation of your team members.

Put in a nutshell the basic ideas of a project run in a Critical Chain style are:

• Estimate the duration of each task without any buffer, i.e. the probability to finish a task in time should be 50 %;
  
• you needn’t hang your head in shame if you use some time from the buffer, that’s why the buffer was created for;
  
• avoid fragmentation of tasks, multitasking is fine for computers, but not for team members;
  
• if a task is finished earlier than estimated, please inform the team members who will need the output from this task – think of a relay race or an early warning system.

Beside the global project buffer placed at the end of the project there may be additional “feeding buffers” for all paths, which “feed” the critical path or chain.
How to determine the size of the buffer? That’s one output of risk management. The gist of risk management is

• to list each risk, there should also be an ongoing process of discovering new risks;
  
• to estimate potential impact and likelihood of each risk;
  
• to find an indicator for each risk which warns you in time that a risk is going to “materialise”;
  
• to plan proactively (buzz-word !), what you will do if a risk materialises or how you can reduce potential impact in advance. Adjusting the project buffer with respect to the risks is one very obvious action to mitigate the impact of materialised risks.

Two points are important. First risk management means planning for failure, therefore it may be difficult to sell it to stakeholders and managers – it even may misfit to the corporate culture. Nobody likes to talk about (potential) failure. Second it will cost something – i.e. the most optimistic date (no problems, project runs absolutely smoothly) will be shifted. Without risk management the project may end at the 1st of June at the earliest – that’s the blue-eyed, fairy-tale best-case scenario. But it’s more probable, that it will end in September, while the worst case will be .... summer next year. With risk management the best case will be 1st of July – so you will pay four weeks for risk management, while the worst case will be October. That’s a major improvement! So risk management means to deal with and to reduce uncertainty.

I haven’t talked about processes and metrics (how to measure progress, quality, etc.) up to now, because sometimes I have an odd feeling when talking about them. So let me do this now, starting with processes. I admit that I have three problems with processes. First “Process” has become a very popular buzzword, which meanwhile means everything, it often lost its original meaning (Who remember?). Second process thinking establishes a layer of abstraction – a process layer - between the daily business and management. As a manager you don’t deal with people reporting to you anymore, but you own and control processes. No wonder that processes becomes a matter of company politics, for example if you want to move up, you had to define and own a process. Third there is a hidden, but obvious message in process obsession: people don’t matter and are interchangeable. I believe the best way to deal with processes is to look at them as a service and as a part of the project infrastructure, which should make work more smooth, focused and successful. Effective processes are an important prerequisite in project work – they shouldn’t end up as a boring buzzword, a management fad or an obsession.

Concerning my odd feelings about metrics: if you want to measure something, you need a model describing the dependencies between the things you want to improve or to optimise and the observables you are able to measure. That’s not a trivial task, it’s very close to science (creating models and proving or disproving them by measurements is the daily business of scientists). Increasing the population of storks in an area doesn’t increase birth rate, although there’s a correlation between the number of storks and the number of births. To mix up observables and the underlying causes is one risk when dealing with metrics. Another aspect is that measurements made over a period of time will only be comparable, if the environment doesn’t change dramatically. For example if you improve one aspect of a process, you have to keep all other aspects unchanged to see the benefits or problems of this improvement. That’s not very realistic in a fast changing environment. If there are a lot of changes (new and different products and projects, new technology, changes in processes, ...), the metrics will also be subject of frequent change, which makes choosing and using metrics more difficult. On the other hand, if you run a project in an iterative way, you will be able to adjust your metrics and to learn with each iteration - a strong argument for an iterative and spiral approach. One final thought: metrics are useless as long as there are no options and actions to react on some negative or alarming measurements. During a running project, metrics only are sensible in the context of risk management. To summarize: metrics depend on models, require a stable environment to be comparable and aren’t useful without any scope for action.

To avoid misunderstandings: processes and metrics aren’t something negative, but they are demanding and there’s some chance to misinterpret or to abuse them. They won’t make project management easier, but more focused and successful. They never are an end in themselves.

Software Testing - as a part of the art of developing sound and successful software – is a vivid and living profession, which have changed and grown during the recent years. I’m sure it will continue to grow and mature in the years to come

Comments:

From dlsoo@yahoo.com [216.255.184.210] - 6/25/06 4:18 AM

Tolle Seite! Hat mir sehr gefallen. Weiter so!

From Ravindra chavan [59.95.40.52] - 6/8/06 9:37 AM

This Document is very vital to me . Thanks a million to the Writer .

From Jerri Franco [72.145.142.114] - 2/12/06 10:48 AM

WOW! This is the most comprehensive and informative description of the entrails of software quality assurance I have come across in years. You have broaden my perspective on the challenges QA testers and managers face and alternatives available in the software development field. I found myself identifying with some of the common pitfalls described, thinking it was just the "nature of the beast".  I feel rejuvenated and armed to tackle some of these obstacles. Thank you and please keep sharing.

---

Last Modified 1/15/06 2:28 PM