Menu:

Home

Prolog: Trapped in the Debugging Box

While testing system, middleware and business software for several years as a Quality Assurance Engineer (I also spent some time in software development and did implementation projects on customer sites), I frequently wrote down my experience about Quality Assurance in order to understand what I had been doing  – and why it sometimes hadn’t worked. This is my recent version. If I again become a QA Engineer in my next life (would this be a blessing or a curse?), I will write a book eventually - provided there will still be software bugs …

For QA engineers are masters in the art of negative thinking, I will start this tiny paper with describing the problems, pitfalls and contradictions of day-to-day Quality Assurance - before I will argue in favour of a comprehensive project-based approach to Quality Assurance for small to medium-size software development projects in the next chapter. I will focus on topics related to my experience and leave other cool topics (for example such as XP) to more competent and experienced authors.

So let’s start with my conviction that Quality Assurance doesn’t fit well in a departmental and hierarchical organisation. In such an environment it will get harder to take care of the cross-functional and cross-departmental aspects of Quality Assurance (for example to strive for defect prevention instead of pure defect detection). Trapped within organisational and hierarchical boundaries (i.e. trapped within a box), it's tough to practice out-of-the-box thinking. QA comes on board too late and leaves too early to have impact neither on the quality of specifications, design and schedule, nor on the field introduction. Even worse a QA department may have the negative side effect of other departments reducing their quality efforts. Often QA departments are pure debugging departments: the last barrier, the very last defence between the development suspected of producing lousy and buggy software and the annoyed customers. The mission of such a debugging department may be: find as many bugs as possible within the time left till the deadline. That’s a rather narrow, defensive and disheartening approach to Quality Assurance.

Lets have a closer look on some problems that a Quality Assurance department may face:

• As mentioned above QA hasn't any influence on the quality of the specification, the design and the schedule - it comes on board too late. If the specification, design or schedule are messed up, QA will have failed before it will start to work. The currency to pay for a unrealistic schedule is always quality: if the schedule and the deadlines aren't realistic, quality and functionality have to suffer. I'm also convinced that one of the most important tasks of QA is to do proper risk management, I will discuss this topic below.
• The existence of a QA department may has a secondary effect of reducing quality efforts in other departments. While QA hasn't the power to make a real difference with respect to quality, other departments may neglect this topic, because it seems to be the job of the QA department. For example development may skip the smoke tests and deliver untested software to QA, which already crashes during installation. Because of aggressive schedules, that's often the only chance of development to meet a deadline.
  
• If there are two or more levels of hierarchy within the QA department, communication with other departments will become slow and cumbersome. Two levels of hierarchy within QA will mean that someone responsible for the whole software project will be three levels away from daily business. Furthermore many levels of hierarchy always imply some politics, which will distract energy from the tasks at hand and may lead to bad decisions.
  
• Because of departmental boundaries QA won’t participate in the skills, knowledge and experience of development and field people. This for example will prevent QA from testing efficiently, whether some internal changes of software (new interfaces, new structure, ...) will cause problems – QA often won’t be informed about these changes. Also the quality of the bug reports depends on the insider knowledge (with respect to development and to the field) of QA. If QA people are able to recognise, that several bugs may be due to a new interface or to a screwed class definition, it will be very helpful hint for development. Without (basic) knowledge of development and some insight in the customer’s business - QA should act as the eye of the customer! -, the QA people are limited to simple functional tests, not to mention defect prevention. That's not a challenging job for QA people, even worse it's possible to run a QA with low-skilled testers, while some intelligence is put into the test plans and processes  (the QA manager may have the only interesting job within a QA department). Why is it unusual or even unthinkable that QA people change (temporarily) to development or e.g. support a pilot customer and vice versa?
  
• There are employees and managers with a Can Do and others with a Can’t Do attitude. While it’s common sense that a Can Do attitude is the preferable one (e.g. taking risks and challenges), it’s better for QA people to be of the Can’t Do type than of the Can Do one (“Sure we will meet the deadline...”, “Bugs? No, there aren’t any bugs ...”). Therefore Can’t Do people will tend to cluster at QA. What may be bad is that QA will develop a negative Can’t Do and That-will-never-work culture, which will exclude them from the world outside QA. While all other guys are running ambitious projects, QA stands grumbling aside. This negative and cynical culture will more likely develop within a QA department, than within a project team.
  
• There are no common tools and processes for development and QA, for example QA has no access to the Configuration Management tools of development to trace changes in software.
  
• People are more committed to their department than to their projects. 

Another aspect may be less obvious, but I think it’s crucial and it's worth some discussion: it's harder for a debugging department to show some output and achievements (i.e. to justify it's existence and the existence of QA managers) than it is for other departments. How can a QA department report results? Reduce the number of bugs is a fuzzy mission – it’s difficult to measure and it doesn’t sound very challenging. Furthermore as long as there are bugs in the field, the debugging department has to defend itself. A debugging department often stands in the shadow of development and may have a lower status than other parts of the organisation. Especially if QA is embedded in a cover-your-back kind of corporate culture it will be in a very defensive position.

But wait: there are ways to measure QA output and achievements (and present it to management). You may cut your tests into small pieces (called test cases), count them and measure the test coverage: the number of test cases being performed divided by the number of all test cases to be run. You may count your bugs, normalise them using something like the number of lines of code or modules, calculate the number of bugs found per day or per week, create impressive pie charts, trend charts or whatever charts. You may use sophisticated bug reporting and tracking systems, with huge databases containing thousands, millions or billions of bugs (perhaps in the future there will be some smart data mining technology available to make some sensible use of this huge amount of archaeological data). You may even enter the realm of software development by developing automated tests and test frames (by the way: who will test the code written by QA?).

As you may have realised by the irony in my sentences, I don’t recommend doing so. My concern is that there will be too many negative side effects, if there's too much emphasis on methods, processes, charts and spreadsheets introduced to improve the status of the testing department (and it’s managers). For example if test coverage matters, the testers are seduced to prefer the fast and easy test cases and leave the more time-consuming and sophisticated ones for later on. 90 % test coverage may sound impressive, but the remaining 10 % may be the sophisticated and time-consuming ones, perhaps these 10 % are the only ones worth running. If the number of found bugs matters, the testers may be seduced to report as many bugs as possible. This may be great if these bugs are all relevant, but it's more probable that the fraction of less important bugs will increase and that the testers will spend less time on preanalysis. Because every reported bug will create some administrative overhead, more reported bugs will result in more administrative work for development and QA - regardless of the relevance of the reported bugs. Finally, creating and maintaining (!) automated tests may be more expensive than running them manually again and again.

To distract QA’s focus from quality and to redirect it to some secondary targets is only one group of negative side effect. There are two other ones: first, there are elements such as intuition and experience in the testing business, which are difficult to organise, to measure and to manage. You may suppress these elements by a perfect organisation of your testing business. One tiny example: if you find a bug when running a test case, it will be sensible to create a link between the bug number and the test case, which caused the bug. So ... to retest bug x, you have to run test case y. That sounds smart. But lets assume you’ll find a bug without running a documented test case or by doing some additional steps when running a test. Maybe because of your intuition. Or maybe you have just found other bugs in this area and you know, that bugs tend to cluster. Now you really get into a mess! How to document this? Writing a new test case? Modify an existing one? Looking for a test who (by chance) matches these test steps, which you’ve just done? If there are already hundreds of tests, that will be time-consuming and interrupted fluid and flexible testing. There is no way to let the results of the previous tests influence the next one. You won’t have this problem, if you exactly follow your test plans. But do we need skilled and qualified testers to do so?

That's my second concern: each step towards better processes and formalisms may degrade the role of the testers, they become interchangeable pieces of a debugging machine. If each tiny test step is documented ("Tick the xyz button…"), skilled testers may be replaced by well-trained monkeys. This may sound a little bit extreme, but there are two causes, which may lead you towards such an extreme situation. First - if some of the testers get the feeling, that their work is mechanised and that they are becoming superfluous and interchangeable, they will quit eventually. To reduce the costs of fluctuation and to become less vulnerable to it's consequences, it's obvious to remove more intelligence and skill from the testers and put it into the processes, e.g. by investing more effort in the documentation of test cases. As a consequence, more testers may leave … a death spiral starts to turn. Second - to do testing by unskilled testers, which were controlled by smart processes and management, will deliver some benefits – for example better control, more flexibility (i.e. shifting resources on short notice) and easy career opportunities. Depending on the corporate culture of your company, these benefits may compensate or even outweigh bad software quality. The sad thing is, that there are no basic skills required to do some kind of low-level testing, while for example an unskilled software developer, who isn’t familiar with any programming language, is unthinkable. 

Well ... some of these problems may not only be typical for QA departments, they may also show up in any other part of an organisation. But because of its weak, defensive and isolated position described above, QA may be more susceptible to it. But Quality Assurance is more than finding bugs during the time left till deadline following exactly some detailed test plans (“Click the “Start” button...”). It's more than defect detection (neglecting defect prevention). It can be a challenging job for highly skilled QA people, the people who do this job really matters! Quality Assurance is more about workmanship, perhaps even about art, than it is about painting-by-numbers. It’s more about communication than about making marks in spreadsheets. And it's more about project teams and collaboration than about departments.

How can quality assurance be integrated and making a difference in the whole process of software development, starting at the specification and ending at customer sites?

The Core Team

Let's assume a new version of your successful product DoSomething should be released. Marketing has ideas about new and improved functionality, there are request from some of your major customers, the field people add ideas from projects on customer sites, support is grumbling about old bugs which after all should be fixed now, development has brilliant ideas about improving the software's infrastructure. Furthermore it's a medium-sized project, i.e. some dozen developers, testers, technical writers and support people will join it. These are the most difficult projects to manage: to apply all methods, tools and processes designed for large-scale projects would be overkill and slow down the project to stasis. Even worse you can't rely on statistics to estimate your efforts: if 30 people were estimated to work on a (sub-) project for six months, the deviation from your estimation would be very much smaller than if there were only 2 people estimated to work for 2 months. For example if somebody from the 30 left or got sick would be less disastrous than if one of the two became unavailable. On the other hand, running a project with more than 2 people requires some organisation. So there's a fine line between organisational overkill and chaos! After this emotive introduction, let's get down to business: the first steps will be
writing the specification,

• determine what quality means with respect to this new version of the product,
• analyse field bugs from the previous version,
• discuss and fix the processes,
• create the schedule and
• do some risk management.

To do so I suggest establishing a core team of four people: one developer, one tester, one technical writer and somebody from support/field. These four people will act as project and team leaders as soon as more people will come on board during the project, their task is to take care of the prerequisites for each team and to coach the team members. For larger projects it may be useful to start with more than four people at the beginning, but there have to be four named team leaders from development, QA, technical writing and field/support. The basic idea of this approach is to start with a "lean" team and to avoid early overstaffing of the project. For example as long as there aren't any specifications available developers can't start coding. The QA team leader will involve more and more testers as soon as functions are implemented. The technical writers can start after the specification will have been finished. And the support and field people will become active as soon as there will be a "prototype" to be shown on fairs or to those customers, whose requests have found their way into the specification. Even more important: the six tasks mentioned above require a lot of intense communication within the core team - with respect to communication, a small team is more efficient than a huge one.

Let's have a look on each of these six tasks:

• Specification: A specification mustn't be a ton of paper, it's much more important that the four core team members share a seamless common understanding. A specification, which nobody will read and which will end as shelf-ware, because it's to thick and cumbersome, won't have any positive impact. Furthermore the specifications must be clear and unambiguous to each new team member, which will come on board later on.
  I won’t discuss the object oriented approach (OOA and OOD) or the use of specification languages (such as UML) here. But if these techniques are applied, it will be mandatory to involve QA as soon as possible – otherwise QA is limited to do some white-box testing later on. Any smart new technique to transform specifications into tests will be useless without participation of QA people.
  
• Quality: I've mentioned above that quality is more than reducing the number of bugs. Important aspects of quality are, for example, a product's uniqueness, the positive impact it has on the business of the customers and it's usability and user-friendliness. These aspects, which may differ slightly for each product, have to find their way to the specification and will be tested by QA later on. What does quality mean with respect to the product at hand? Which aspects of quality are crucial, which are less important? What are the criteria for delivery?
  
• Analyse field bugs: To learn from old bugs is a very efficient way to improve quality and make live easier, but it’s not a trivial job. To recognise error patterns and probable causes requires experience and intuition, you will need at least a glimpse of an idea what you are looking for in order to find a pattern. And it’s a job for developers and QA and technical writers and support, because everybody has another point of view and another approach to look at errors. Development will be able to recognise, whether the bugs cluster around a module, interface or class definition, the technical writers may realise that the many user errors are due to a vague and misleading documentation or to bad usability, QA will see the holes and gaps in their testing and support will know, which errors are the most common ones.
  
• Processes: It wouldn't be a good idea to invent software development processes from the scratch for each project, but it's sensible to adjust your processes for each one. Especially in the software business even processes are subject of perpetual change. For a small project some basic processes - including configuration and change management - would be sufficient. For example if you expect 100-200 bugs, a spreadsheet will be sufficient to track your bugs, while for a huge number of expected bugs you will need a database. For process obsession is a widespread management fad, it may be difficult to start with "lean" processes, but it's worthwhile. Processes have to make a project transparent to all participants and stakeholders, too many or too sophisticated processes may mask the daily business and problems. A process is mandatory as long as it is changed by an agreement of all stakeholders. So there have to be a process to change processes - and this process is absolutely mandatory!
  
• Schedule: If you are creating plans and schedules, the underlying assumptions, prerequisites and expectations are much more important than the milestones and deadlines themselves. As soon as one assumption, prerequisite and expectation turns out to be wrong, you have to rethink and update your schedules. That’s the first step to an early-warning-system and to effective risk management. For example to determine a date by asking some guys about their estimates and taking some (weighted) mean value won’t work well - you would realise too late, that this date wasn’t met, and, even worse, you wouldn’t understand why. Therefore it’s crucial that the way a plan was made (i.e. the assumptions, prerequisites and expectations) must be clear and transparent to every stakeholder! That’s why open communication and teamwork is extremely important in the planning phase. By the way: in spite of milestones, schedules, deadlines and project plans it's important to remain open-minded: things may happen which will be in favour of you. An awkward function may be replaced by a simple one, a solution for a problem may be easier than planned, somebody may have a brilliant idea. This may happen - eventually.
• Risk Management: For risk analysis is one of the main tasks of QA, I will discuss this topic in detail below. But now I want to stress, that risk management mean to live with risks, not to avoid them: projects which make a difference are always risky. Often the most risky projects are the only ones worth doing. I've mentioned above that QA tend to play a defensive risk-avoiding role, especially if it's imbedded in a defensive corporate culture or - even worse - in a corporate culture of fear. Risk Management means a offensive approach to risks.

Why do I emphasise that software development should be team-based?  What are the benefits of this core team idea?

In the first chapter I showed some scepticism about QA departments, but some sympathy about teams and team based organisations. The reason is quite simple: teams need some common goal, some common understanding "who we are". While a sport team wants to win - no matter whether it's a world or a backyard championship - , a team in business life wants to do a good and successful job, i.e. there is a natural sense for quality. The best way to kill a team is telling them: "I know this software is still rubbish, but we have to get it out until next week. We only need to assure that it didn't crash immediately". To force professionals to do unprofessional work will only create cynicism and destroy the basis of team formation. Departments are neutral with respect to quality: it depends on the (corporate) environment whether people act as professionals or as cynics. If the existence of the Quality Assurance is only justified by the buggy and error-prone software delivered by those lousy developers (which weren't granted the time to do a better job, for example to do some testing on their own), and if the Quality Assurance is only supposed to reduce the number of bugs to a less annoying level, development and QA will end up as disillusioned cynics without any team-spirit.

The core team idea avoids departmental boundaries and corporate politics, it enforces communication between the main stakeholders and uses synergies.
For example within the core team the main task of development is to estimate the costs and problems of implementing new functions. A "simple" function may be expensive to implement or doesn't fit well into the structure of the software. Development is the only instance that can deliver this kind of information.

QA may raise the topic how to avoid the typical bugs from previous versions in the new release. If there is an error-prone part of the software, now it's the time to redesign it. If QA needs better traces, logs, test points and test hooks that will be a topic for the specification too. Furthermore being involved in the project from the very beginning enables QA to do one of it's main tasks: to make a statement about the quality of the product as soon as possible. Therefore QA has to create a short preliminary test plan a soon as the specification is finished. If QA starts testing late and leaves important topics untouched until two weeks before the deadline (this may happen if test coverage is the most important goal), you will run into avoidable risks.

The domain of the technical writers will be to check the usability and consistency of the product. If it's difficult to describe a new function in a understandable and user-friendly way, this function won't be very usable. If it's hard to summarise the basic ideas, concepts and benefits of a product at the beginning of the manual, your product may be a bunch of functions, but not a consistent product. A good specification is also a good basis for the manual.

The support and field people are the representatives of the customers. They will know the best real life war stories. Being involved in the whole development process, they also will get deep and early insight into the product itself.

After the core team has finished his final work, the 4 teams will start to grow and to evolve. I will now focus on my domain: the Quality Assurance part.

To Continue.