In response to the HP corporate-wide 10 x software quality improvement initiative, much attention has been focused on improving the quality of software products developed throughout HR The motivation for software quality improvement is most often expressed in terms of increased customer satisfaction with higher product quality, or more generally, as a need to position HP as a leader in quality software development.

A more fundamental motivation to support the initiative for higher software quality can be developed when software defect cost data is considered. The data presented in this paper is drawn from an extensive software project database maintained at the HP Waltham Division for product releases over the past five years. When software defect cost calculations are performed on this data, a very compelling bottom line" perspective on software quality emerges; software defects are very expensive and early defect prevention and removal techniques can substantially enhance the profit realized on software products. This paper will present a general model that can be used to calculate software defect cost data for any software or firmware product. Data from actual HP Waltham projects will be used to provide examples of software cost calculations.

The Need for Metrics

As an example of the need for substantive software quality cost data, consider the situation a project manager might encounter when attempting to justify the purchase and use of a new software development tool such as a static code analyzer. If the cost of the tool is $20,000 and if there is reliable data to suggest that the tool will uncover 50/o of the total number of software defects during typical use, is the project manager justified in acquiring and using the tool?

To provide answers to this type of question, it is important to have access to a reliable database of software quality metrics. Such a database is maintained by the software quality engineering group at the clinical systems business unit of HP's Waltham Division. This database has become an essential component of software quality activities at HP Waltham and is invaluable for such tasks as project scheduling, resource planning, project and product quality status reporting, and software defect cost calculations.

In addition to maintaining the metrics database, the software quality engineering group works with R&D in testing and process improvement activities.

Software Quality Metrics Database

Fig. 1 indicates the development phases of a typical software project, with the phases indicated in which metrics are collected and stored into the software quality database. Data is gathered from a variety of sources including software defect logging, product comparison studies, project post-mortem studies, code complexity and size analysis, and project schedule and resource plans. The physical data resides mainly in a standard HP STARS* database, which has been augmented with additional fields, files, and reporting utilities. All of the products represented in the metrics database are firmware-based medical devices such as critical care monitors, arrhythmia analysis computers, and clinical databases.

Figs, 2, 3, and 4 represent various types of useful data that can be extracted from the database. Fig. 2 documents the steps that are typically required to find, fix, and retest a defect discovered by the software quality engineering group during integration and system or acceptance testing. The engineering effort for this activity, which is shown as 20 hours, represents the average effort for finding and fixing one typical software defect. This value has been calculated using hundreds of data points from multiple software projects that have been tracked with the software quality database. Mg. 3 is an example of how an accurate schedule for the integration through the release phases can be developed using historical project data from the database. In this case, it is clear that a stable and linear relationship exists between product code size and resultant calendar time. Finally, Fig. 4 tabulates various software metrics from multiple software projects. This data can be very useful for developing project comparison studies.

The data presented in these figures is a small subset of the data that exists in the database. This specific data has been presented because of its applicability to software defect cost calculations.

Looking for Software Defect Costs

Software defect costs can be investigated using a variety of different approaches. For example, costs can be calculated on a prerelease or a postrelease basis, or costs can be determined per defect or per project phase, or costs can be weighted based on code size or programmer productivity. The software defect cost data developed in this paper focuses on the cost per prerelease software defect that is found and fixed during the integration through the release phases of project development. This approach is used because of the abundance of reliable data points available for study and because of the potential utility of the results.