Structured testing in practice

Contents

1 Introduction

2 General aspects of software development processes
2.1 Software development models
2.1.1 Waterfall-Model
2.1.2 V-Model
2.1.3 W-Model
2.1.4 eXtreme Programming (XP)
2.1.5 Conclusion
2.2 Quality factors and approaches
2.2.1 Capability Maturity Model Integration® (CMMI) .
2.2.2 Testing Maturity Model (TMMsm)
2.2.3 Test Process Improvement® (TPI)
2.2.4 Comparison of TMM and TPI
2.2.5 Conclusion

3 Testing specific aspects of software development processes
3.1 The objectives and limits of testing
3.2 The Test plan
3.2.1 Strategical parts of the test plan
3.3 Test plan defined in IEEE 829
3.3.1 Test plan identifier
3.3.2 Introduction
3.3.3 Test items
3.3.4 Features to be tested
3.3.5 Features not to be tested
3.3.6 Approach
3.3.7 Item pass/fail criteria
3.3.8 Suspension criteria and resumption requirements
3.3.9 Test deliverables
3.3.10 Testing tasks
3.3.11 Environmental needs
3.3.12 Responsibilities
3.3.13 Staffing and training needs
3.3.14 Schedule
3.3.15 Risks and contingencies
3.3.16 Approvals

4 Concrete implementation of structured testing in a company
4.1 The starting point
4.2 Analysis of the current test activities
4.2.1 Test maturity matrix of testing at the starting point
4.3 The way to structured testing
4.3.1 The Test Browser
4.3.2 Storage management
4.3.3 Archiving process
4.3.4 Test automation
4.3.5 Test database
4.3.6 Build management
4.3.7 QaTraq - A web based software test case management tool
4.4 Conclusion

Abbildungsverzeichnis

Tabellenverzeichnis

Literaturverzeichnis

1 INTRODUCTION

Testing is a must! Testing is a necessary prerequisite for successfully building and implementing information technology (IT) systems. Koomen, Pol [KP99]

Software testing is often understood as an unstructured, uncontrollable process. Testing takes too much time and money without guaranteeing that the tested product does not include any defects, and is of a certain quality. Due to the fact that testing accounts between 25% to 50% of the total project budget, many companies do not spend the appropriate amount of time required for managing and implementing a valuable test process.

Introducing a structured test process can solve many problems related to testing. Structured testing means that activities, procedures, and techniques on the basis of well-known quality approaches are used, to cover all important aspects of the test process. Without fundamental knowledge of quality approaches (see [AS06, chap. 7]), test management tools, and workflows in a structured test process, putting structure on the test process may result in chaos. Quality approaches explain the steps and in addition, the order of the steps which must be taken in order to achieve a measurable improvement of the test process (see [KP99]). In a structured test process lots of activities, for instance, providing documents and measures about the quality of the product, can be automated. In addition, more time on testing the product will be spend instead of trying to manage an unstructured and maybe chaotic test project.

This thesis gives insight in well-known quality approaches. Furthermore, the objectives and limits of testing, to clear out the misconception that a product does not contain any error if the test department does not find any more defects, are discussed (see [KFN99, chap. 2]). The main focus is to illustrate the implementation of a structured test process based on a specific quality approach in detail. This process includes different test types, which are executed fully automated or manually (see [KFN99, chap. 3]), and uses a web-based presentation of test information and documents.

2 GENERAL ASPECTS OF SOFTWARE DEVELOPMENT PROCESSES

This chapter provides information on important general aspects in a software development life- cycle. Starting with well-known software development models, the chapter gives insight into quality approaches supporting companies on improving their internal processes. The main focus is to explain models used to measure and improve the quality and structure of a test process.

2.1 Software development models

According to Spillner et. al. [AS06], software process models are used to define and describe organizational activities and the link between these activities within development projects. In the last 20 years a whole suite of development models have been developed and put into practice. The target of all development models is to provide a structured and systematical approach during the development and maintenance of software systems.

Spillner [AS06] describes the following aspects as important components of software development models:

- Chronological and technical criteria to cross to the next activity (e.g. milestones, quality gates)
- Allocation of roles and responsibilities
- Documents to be created in each activity, used methods, guidelines, standards, and tools if required

Each aspect in a development model is defined in different levels of detail. Due to the fact that different models exist, this chapter provides a selection of process models where testing has a different standing only. For further details or models it is recommended to refer to books, articles, or websites, for instance, Software Engineering knowledge base [Vir] or V-Model XT [KBS]. Software development models are differentiated depending on how activities take place during the development project.

Due to the fact that software projects are built using various software process models, a classification is made reflecting the way activities are managed, which can either be sequentially or iteratively. The following list shows the different types of classification:

Sequential Models The course of activities is planned step-by-step, where each activity is passed once. All activities are linear structured. To finish a development project, each activity has to be passed. Both, Waterfall-Model [AS05, chap. 2.2] and V-Model [AS06, chap. 3] are famous representatives, for instance. Figure 2.1 illustrates the flow of activi- ties during sequential models.

illustration not visible in this excerpt

Figure 2.1: Sequential Process

Evolutionary Models The basis for the process in this model are customer defined require- ments for the software. If the requirements have been implemented, the release of the software is distributed to the customer. After a release is deployed, the customer may define requirements for the next release of the software. These steps are repeated until the software/release meets all requirements defined by the customer (see Figure 2.2).

Iterative Models Before a development project is started, all requirements for the software are specified, which is the difference to evolutionary model described above. The specifi- cation is split into logical packages. After implementation of one package is finished, a release is distributed to the customer. In the following implementation steps, the experi- ments of the customer using the software are taken into account (see Figure 2.2). Famous

illustration not visible in this excerpt

Figure 2.2: Process iteration

representatives are Rapid Application Development(RAD) [Uni] and eXtreme Programming(XP) [Bec00].

According to Spillner [Spi01], the following sections give insight into important software development models seeking to point out the main aspects of each model.

2.1.1 Waterfall-Model

First presented in 1970 by Dr. Winston W. Royce [Roy70] as an extension of the Phase-model, this model had hardly any success until B. W. Boehm extended the model with the possibility of returning to an immediate previous phase in 1981 [Boe81].

Figure 2.3: Waterfall-Model, [Spi01, chap. 2.1]

illustration not visible in this excerpt

The individual phases i.e. activities that are defined here are to be found in 4 nearly all models proposed since. In this model it is set out that each of the activities in the software development must be completed before the next can begin. A return in the development process is only possible to an immediate previous phase.

illustration not visible in this excerpt

Figure 2.3 shows the Waterfall-model with its sequential course of activities in which testing is introduced if the implementation phase is completed.

2.1.2 V-Model

Based on the graphical arrangement of the individual phases (see Figure 2.4), this model is called V-model. Starting on top of the left branch in Figure 2.4 the activities are in chronological order.

illustration not visible in this excerpt

Figure 2.4: V-Model, [Spi01, chap. 2.3]

By the ordering of activities in time sequence and with abstraction levels the connection between development and testing activities becomes clear. Oppositely laying activities complement one another i.e. serve as a base for test activities. So, for example, the system test is carried out on the basis of the results specification phase.

Although in the V-model it seems that testing follows the implementation of the software, testing is involved earlier in the project than in the Waterfall-model. The coarse division into construc- tive work to the left and destructive work to the right is the main disadvantage of the model.

2.1.3 W-Model

Based on the fundamental idea of improving the integration of testing in the development lifecycle the W-model has been established. Compared to the Waterfall-Model and the V-Model, the W-Model removes the following disadvantages from the perspective of testing:

- Testing follows the implementation
- The relationship between the several stages and their outcome for further testing activities is not clarified
- The tight link between testing, debugging and modification in the source code during the test phases is not defined

Based on the V-Model, the W-Model enhances the definition of activities with testing specific tasks. Figure 2.5 shows the relationship between constructive and destructive activities.

illustration not visible in this excerpt

Figure 2.5: W-Model, [Spi01, chap. 3.1]

Referring to Figure 2.5 the relationship between development and testing phases is clearly defined in this model. In addition, the assignment between development and testing tasks is cleared out, for instance, the test plan for the system test is based on the specification of the software. Advantages and disadvantages of the W-model are listed below:

Advantages

- Parallel to the development process, the testing process is carried out
- From the project outset onwards, testers and developers are entrusted with specific tasks
- Both, testers and developers are seen as equal-rights partner
- During the test phase, developers are responsible for removing defects by modifying the source code

Disadvantages

- In practice a stronger relationship between several activities exist
- It seems that the different activities have equal resource requirements, for instance time, and personnel, which is unrealistic in practice.

2.1.4 eXtreme Programming (XP)

In 1996, Kent Beck first used new concepts of development for a project at Daimler Crysler which has been the foundation-stone of eXtreme Programming. XP is a mixture of several successful techniques, and ideas which result in improvement software projects in four essential ways: communication, simplicity, feedback, and courage. Referring to Beck [Bec00], the important aspects of XP are:

The planning game It combines the technical estimates and business priorities to define the next milestone

Small releases A simple system should run as quickly as possible. This running solution will be enhanced in small short releases Refactorings Changing the system structure without changing its behavior. This may force to reliability and performance of the system.

Simple design The system design has to be as simple as possible. Programmers should avoid solving problems that are parts of the future releases.

Pair programming All code will be written by two programmers using one machine. Two programmers will probably find defects while modifying the code that one developer may be miss. It is more unlikely that two developers do not see a bug that .

Coding standards All developers write their code in accordance with naming and design rules. This will simplify the communication between the involved devel- opers.

Testing Programmer writes test-cases to demonstrate that a feature has been im- plemented. The tests will be written before implementing the new feature.

For each software project based on XP, user stories are created instead of specifications. Each story expresses the customer’s perspective which defines the changes in the source code, a developer has to implement in order to meet a specific predefined requirement. Time estimation and resource planning is done for this working package only. Based on priorities and the risk-management set for this working package, a decision is made on when this user story is included in the release. Furthermore, these stories build the foundation for the required test cases. It is a necessity that all unit tests written in order to test the working package are passed before a release is deployed. Further information on XP is provided in Beck [Bec00].

2.1.5 Conclusion

Within this section, well known software process models have been described. Many software projects around the world are based on these models. Despite the model used in a develop- ment project, each model has its advantages and disadvantages. In addition, critical factors when managing software projects, a company has to be aware of on how to succeed in the project exist. The following chapter deals with quality factors and approaches during a develop- ment project and during processes in a company in common. Therefore, insight into different quality approaches is given, providing the possibility to measure and improve the quality of a product or processes in a company.

2.2 Quality factors and approaches

Investing in quality is imperative. With more than half of all software projects considered failed or challenged and with support costs for defective software running as high as 50 percent of the total development cost, companies must invest in quality, even if they don’t have the resources or the culture for a full quality assurance (QA) initiative. By focusing on high- value areas like creating standard test procedures and reporting, companies can improve their production quality and reduce customer-reported defects while building the foundations for a more formal QA practice later.

Margo Visitacion A Standish Group research report [The94] evinces that over 80% of software projects are unsuccessful either because of over budget, late, missing function, or a combination, which is a very staggering result. Referring this report about 31,1% of projects will be canceled before they ever get completed, 52,7% of the projects cost 189% of their original estimates and only 16,2% are completed on-time and on-budget. Most significant reasons why projects fail are incomplete or changing requirements, lack of resources, unrealistic expectations, changing of specifications, and unrealistic time frames.

To raise the percentage of successful software projects and in addition, to improve the quality standards in a company, several models have been created. These models are related to one of the following groups:

Common models Used to measure, control and improve the quality of various processes in- side a company, for instance processes involved in manufacturing a product. Most fa- mous approaches are Total Quality Management [OM97], Kaizen [Ima86], and Six Sigma [HS00].

Software development specific models Techniques to measure and improve quality of soft- ware development projects. Famous representatives are the Capability Maturity Model Integration® [Pau93] and the ISO standard ISO/IEC 15504 [EEMD97], a result of the Software Process Improvement and Capability determination (SPICE) - project.

Test-process specific models Describing methodologies for quality measurement and im- provement of test-processes. Well known models are Testing Maturity Model (TMMsm) [BSC96] and Test Process Improvement (TPI) model [KP99].

Whereas Common models are not discussed in more detail, this chapter gives insight into a popular Software development specific model and two well-known Test-process specific models, which are the main focus of interest in this thesis. Due to the fact that models in connection with testing are often based on models created to improve quality of software development models, the next section provides insight into the Capability Maturity Model Integration® - an approach to improve quality of processes and products.

2.2.1 Capability Maturity Model Integration® (CMMI)

This section is based on the technical report CMMI for development version 1.2 [CMM06]. Due to that, further references on this report are not quoted explicitly.

Created by the Software Engineering Institute (SEI) of Carnegie Mellon University in Pittsburgh, CMMI version 1.1 has replaced the Capability Maturity Model (CMM) also published by SEI in 1993, in the year 2002. The current version of CMMI is 1.2. Already explained in chapter 2.2, CMMI is an approach to improve quality software development processes. The assumption of CMMI is that the improvement of processes connected to software development will improve the quality of the software, and furthermore result in a more precise management of time and resources. [AS06, chapter 7.2.1]

CMMI enables to process improvements and appraisals using two different representations: continuous and staged. Whereas the continuous representation defines capability levels for each process area, the staged representation depicts the maturity levels.

A maturity level is a defined evolutionary plateau for organizational process improvement. Each maturity level matures an important subset of the organization’s processes, preparing it to move to the next maturity level. The maturity levels are measured by the achievement of the specific and generic goals associated with each predefined set of process areas. [...]

Five maturity levels are used in this approach (see Figure 2.6). A short description of each level is provided as follows:

Initial Processes are insufficient undefined. Development is ad hoc and chaotic. The expec- tations on the product are fulfilled, but an excessive increase of budget and used time in conjunction with estimated time is the normal case.

illustration not visible in this excerpt

Figure 2.6: Maturity levels in CMMI

Managed Fundamental management processes are established in development projects. In this level a documented project plan exists providing the possibility to monitor and control the project’s progress.

Defined Organization-wide process standards are defined and introduced, which is the main difference to maturity level 2 Quantitatively Managed Based on collected statistical data improvement decisions are made related to the customers, end users, organizational, and process implementers require- ments.

Optimizing Systematical and continuous process improvements are established.

The advantage through maturity levels is the achievement of progressive improvements in an organizational maturity by achieving improvements on the project level and continuing to the most advanced level using both quantitative and qualitative data to make decisions.

Beside maturity levels, process areas build another main part in CMMI. A process area serves as a group of specifications related to a specific aspect.

A process area is a cluster of related practices in an area that, when implemented collectively, satisfy a set of goals considered important for making improvement in that area.

In CMMI, 22 process areas are defined. Goals are set in advance for each process area. These goals are separated in generic and specific goals, where generic goals describe the process steps or tasks to be taken in order to reach a specific goal. When all required goals for a specific process area are reached, this area achieves the corresponding maturity level.

A specific goal describes the unique characteristics that must be present to satisfy the process area. A specific goal is a required model component and is used in appraisals to help determine whether a process area is satisfied. [...]

Table 2.1 illustrates an example which describes a possible situation in a company as a combi- nation of maturity levels and process areas. The areas Validation and Verification which belong to the maturity level Defined, are strongly connected with and executed in the test department.

To gain an insight into the continuous representation provided in CMMI the understanding of capability levels is a prerequisite. Therefore these levels are explained in the following paragraphs before continuing with the description of the continuous representation, its specific process areas in combination with their corresponding capability levels.

A capability level consists of a generic goal and its related generic practices as they relate to a process area, which can improve the organization’s processes associated

illustration not visible in this excerpt

Table 2.1: Maturity levels and process areas

with that process area. [...]

In CMMI six different levels are defined, for which a description is given as follows:

Incomplete (Level 0) A process that either is not performed or partially performed. One or more specific goals are not satisfied.

Performed (Level 1) A process of this level satisfies specific goals of a process area. Activities passed in order to reach these specific goals, result in important improvements. However, these improvements can be lost over time if they are not institutionalized.

Managed (Level 2) In addition to a process of level 1, a managed process provides the basic infrastructure to support the process. This process is monitored, controlled, reviewed, and evaluated for adherence and its process description.

Defined (Level 3) A managed process which is tailored from the organization’s set of standard processes according to organization’s guidelines, contributes work products, measures, and other process improvement information to the organizational process assets.

Quantitatively Managed (Level 4) A defined process that is controlled using statistical and other quantitative techniques. Quantitative objectives for quality and process performance are established and used as criteria in managing the process.

Optimizing (Level 5) The main focus of level 5 is the continuous improvement of process per- formance through both, incremental and innovative improvements.

Whereas maturity levels in the staged representation cover the process areas as a whole and in addition, cover the maturity level of the organizational unit, the capability levels are related to one process exactly. As a result, the continuous representation is more specific than the staged representation, in which process specific assessments and descriptions are possible. Figure 2.7 reflects some specific process areas at different capability levels.

illustration not visible in this excerpt

Figure 2.7: Continuous Representation

Testing in CMMI The primary interest of test managers are the process areas in CMMI dealing with software testing. In table 2.1 the areas Verification and Validation defined as testing related processes are highlighted. In the staged representation (see Table 2.1), these two areas belong to the maturity level Defined. The following paragraphs give insight into each process area and seek to examine the relevance of testing within the CMMI approach.

Verification

The purpose of Verification (VER) is to ensure that selected work products meet their specified requirements. [...]

[...]