SOFTWARE PROCESS AND PROJECT METRICS

Chapter 4

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Software metrics
Measurement can be applied to the software process with the intent of improving to assist in estimation, quality control, productivity assessment, and project control to help assess the quality of technical work products and to assist in tactical decision making as a project proceeds

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

MEASURES , METRICS , AND INDICATORS

a measure provides a quantitative indication of the extent, amount, dimensions, capacity, or size of some attribute of a product or process. a metric as " a quantitative measure of the degree to which a system, component, or process possesses a given attribute. an indicator is a metric or combination of metrics that provide insight into the software process, a software project, or the product itself A software engineer collects measures and develops metrics so that indicators will be obtained .

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

METRICS IN THE PROCESS AND PROJECT DOMAINS

Process indicators
enable a software engineering organization to gain insight into the efficacy of an existing process ( i.e. , the paradigm, software engineering tasks , work products , and milestones ) .

enable managers and practitioners to assess what works and what doesn't. Process metrics are collected across all projects and over long periods of time. Their intent is to provide indicators that lead to longterm software process improvement.

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Project indicators
enable a software project manager to assess the status of an ongoing project track potential risks uncover problem areas before they "go critical" adjust work flow or tasks evaluate the project team's ability to control quality of software engineering work products.

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Process Metrics and Software Process Improvement The only rational way to improve any process is to measure specific attributes of the process develop a set of meaningful metrics based on these attributes use the metrics to provide indicators that will lead to a strategy for improvement

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Product

Customer Characteristics

Business Conditions

Process

People

Development Environment

Technology oduct

Determinants for software quality and organizational effectiveness.

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Determinants for software quality and organizational effectiveness

process sits at the center of a triangle connecting three factors that have a profound influence on software quality and organizational performance the skill and motivation of people has most influential factor in quality and performance the complexity of the product have impact on quality and team performance the technology (the software engineering methods) the process triangle exists within a circle of environmental conditions that include the development environment, business conditions, customer characteristics (e.g., ease of communication)

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Outcomes

We measure the efficacy of a software process indirectly based on the outcomes that can be derived from the process. Outcomes :
measures of errors uncovered before release of the software efects delivered to and reported by end users work products delivered human effort expended calendar time expended schedule conformance

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

We derive process metrics by measuring the characteristics of specific software engineering tasks.
measure the effort and time spent performing the umbrella activities measure the generic software engineering activities

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Private metric
There are "private and public" uses for different types of process data : Data private to the individual
serve as an indicator for the individual only

Examples of metrics private to the individual defect rates (by individual) defect rates (by module) errors found during development

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Personal Software Process

A structured set of process descriptions , measurements , and methods that can help engineers to improve their personal performance Some process metrics are private to the software project team but public to all team members
Defects reported for major software functions Errors found during formal technical reviews Lines of code or function points per module and function

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Public metrics
Public metrics assimilate information that originally was private to individuals and teams. Project-level defect
rates , effort, calendar times, related data

are collected and evaluated in an attempt to uncover indicators that can improve organizational process performance.

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Software metrics etiquette Software process metrics can be misused.

Software metrics etiquette:
Use common sense and organizational sensitivity when interpreting metrics data. Provide regular feedback to the individuals and teams who have worked to collect measures and metries. Don't use metrics to appraise individuals. Work with practitioners and teams to set clear goals and metrics that will be used to achieve them Never use metrics to threaten individuals or teams. Metrics data that indicate a problem area should not be considered "negative." These data are merely an indicator for process improvement. Don't obsess on a single metric to the exclusion of other important metrics.

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

A simple defect distribution

Hardware interface 8% Error checking 11% Data handling 11% User interface 12% Standards 7% Specifications 25%
Causes of defects and their origin for four software projects

Software interface 6%

Logic 20%

Origin of errors/defects specification/requirem ents desig n code

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

A fishbone diagram showing the causes of one class of defects

Missing

Ambiguous

Specification defects wrong customer queried customer gave wrong info inadequate inquiries used outdated info Incorrect Changes

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Project Metrics
Used for strategic purposes
by a project manager and a software team to adapt project work flow and technical activities .

Project metrics on estimation

effort and time duration estimates

Production rates
pages of documentation, review hours, function points, delivered source lines

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

The intent of project metrics is twofold :

to minimize the development schedule, to assess product quality on an ongoing basis and when necessary, modify the technical approach to improve quality.

Every project should measure:

inputs - measures of the resources (e.g., people, environment) required to do the work, outputs - measures of the deliverables or work products created during the software engineering process, results - measures that indicate the effectiveness of the deliverables

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

SOFTWARE MEASUREMENT
Direct measures (e.g., the length of a bolt) Indirect measures (e.g., the "quality" ) Direct measures

lines of code (LOC) execution speed memory size defects reported over some set period of time

Indirect measures
functionality, quality, complexity, efficiency, reliability, maintainability and many other "abilities"

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Size-Oriented Metrics
Derived by normalizingquality and or productivity measures by considering the "size" of the software

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

10

Size Oriented Metrics

a set of simple size -oriented metrics can be developed for each project:
errors per KLOC (thousand lines of code) defects per KLOC $ per LOC pages of documentation per YLOC

other interesting metrics:

errors/person-month LOC per person-month $/page of documentation

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Lines of Code (LOC)

proponents claim
LOC can be easily counted, many existing software estimation models use LOC or KLOC as a key input, a large body of literature and data predicated on LOC already exists.

opponents claim
LOC measures are programming language dependent, they penalize well-designed but shorter programs, they cannot easily accommodate nonprocedural languages, their use in estimation requires a level of detail that may be difficult to achieve (i.e., the planner must estimate the LOC to be produced long before analysis and design have been completed).

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

11

Function-Oriented Metrics
Use a measure of the functionality delivered by the application as a normalization value. Function-oriented metrics were first proposed by Albrecht.
Function points
measurement parameter number of user inputs number of user outputs number of user inquiries number of files number of externel interfaces count = total count

Weighting Factor simple 3 4 3 7 5 average 4 5 4 10 7 complex 6 7 6 15 10 = = = = =

Computing function point metrics.

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Number of user inputs . Each user input that provides distinct applicationoriented data to the software is counted. Inputs should be distinguished from inquiries, which are counted separately. Number of user outputs. Each user output that provides application-oriented information to the user is counted. In this context output refers to reports, screens, error messages, and so on. Individual data items within a report are not counted separately. Number of user inquiries. An inquiry is defined as an on-line input that results in the generation of some immediate software response in the form of an on-line output. Each distinct inquiry is counted. Number of files . Each logical master file (i.e., a logical grouping of data that may be one part of a large database or a separate file), is counted. Number of external interfaces . All machine readable interfaces (e.g., data files on tape or disk) that are used to transmit information to another system ITU DEPARTMENT are counted. OF
COMPUTER ENGINEERING SOFTWARE ENGINEERING

12

COMPUTING FUNCTION POINTS

Once the above data have been collected, a complexity value is associated with each count. For determining whether a particular entry,
simple, average, complex.

To compute function points (FP),

FP = count-total x [0.65 + 0.01 x Fi]

The Fi ( i = 1 to 14) are " complexity adjustment values "

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

COMPUTING FUNCTION POINTS Rate each factor on a scale of 0

0 1 2 3 4 5

No influenc

Incident

Moderat

Average

Significant

Essentia

1. Does the system require reliable backup and recovery? 2. Are data communications required? 3. Are there distributed processing functions? 4. Is performance critical? 5. Will the system run in an existing, heavily utilized operational environment? 6. Does the system require on-line data entry? 7. Does the on-line data entry require the input transaction to be built over multiple screens or operations? 8. Are the master files updated on-line? 9. Are the inputs, outputs, files, or inquiries complex? 10.. Is the internal processing complex? 11. Is the code designed to be reusable? 12. Are conversion and installation included in the design? 13. Is the system designed for multiple installations in different organizations? 14. Is the application designed to facilitate change and ease o f use by the user? ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

13

Function Point

Once function points have been calculated, they are used in a manner analogous to LOC to normalize measures of software productivity, quality, and other attributes : errors per FP defects per FP $ per FP page of documentation per FP FP per person-month

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

An Example of LOC-Based Estimation

The CAD software will accept two- and three-dimensional geometric data from an engineer. The engineer will interact and control the CAD system through a user interface that will exhibit characteristics of good humanmachine interface design. All geometric data and other supporting information will be maintained in a CAD database. Design analysis modules will be developed to produce required output which will be displayed on a variety of graphics devices. The software will be designed to control and interact with peripheral devices that include a mouse, digitizer, and laser printer. user interface and control facilities (UICF) two-dimensional geometric analysis (2DGA) three-dimensional geometric analysis (3DGA) database management (DBM) computer graphics display facilities (CGDF) peripheral control (PC) design analysis modules (DAM)
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

14

Function

Estimated LOC 2,300 5,300 6,800 3,350 4,950 2,100 8,400 33,200

User interface and control facilities (UICF) Two-dimensional geometric analysis (2DGA) Three-dimensional geometric analysis (3DGA) Database management (DBM) Computer graphics display facilities (CGDF) Peripheral control (PC) Design analysis modules (DAM) Estimated lines of code

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Estimation table for LOC method.

optimistic: 4600 most likely: 6900 pessimistic: 8600 AN EXAMPLE OF FP-BASED ESTIMATION

Information Domain Value

opt.

likely pess.

est. count

weight

FP-count

Number of inputs Number of outputs Number of inquiries Number of files Number of external interfaces

20 12 16 4 2

24 15 22 4 2

30 22 28 5 3

24 16 22 4 2

4 5 4 10 7

96 80 88 40 14 318

Count-total

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

15

Estimating information domain values. Factor

Backup and recovery Data communications Distributed processing Performance critical Existing operating environment On-line data entry Input transaction over multiple screens Master files updated on-line Information domain values complex Internal processing complex Code designed for reuse Conversion/installation in design Multiple installations Application designed for change Complexity adjustment factor

Value
4 2 0 4 3 4 5 3 5 5 4 3 5 5 1.17

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

The estimated number of FP is derived: FPestimated = count-total [0.65 + 0.01 Fi ] FPestimated = 372

Historical data normalized using function points indicate that the organizational average productivity for systems of this type is 6.5 FP/p m. Burdened labor rate of $8000 per month, the cost per FP is approximately $1230. Based on the LOC estimate and the historical productivity data, the total estimated project cost is $457,000 and the estimated effort is 58 personmonths.

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

16

Extended Function Point Metrics

The function point metric was originally designed to be applied to business information systems applications. The data dimension was emphasized to the exclusion of the functional and behavioral (control) dimensions. The function point measure was inadequate for many engineering and embedded systems Feature points
a superset of the function point applications in which algorithmic complexity is high. real-time process control embedded software applications

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

To compute the feature point information domain values are again counted and weighted as described In addition, the feature point metric counts a new software characteristic, algorithms. An algorithm is defined as "a bounded computational problem that is included within a specific computer program"

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

17

3D Function Point
3D function point integrates Data dimension of software with the functional and control dimensions. Data Dimension Counts of retained data (the internal program data structure, e.g., files) External data (inputs, outputs, inquiries, and external references) Functional Dimension Measured by considering 'the number of internal operations required to transform input to output data'

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Transformation A series of processing steps that are constrained by a set of semantic statements. A transformation is accomplished with an algorithm that results in a fundamental change to input data as it is processed to become output data. Acquire data from a file Simply place that data into program memory

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

18

The level of complexity assigned to each transformation is a function of : The number of processing steps The number of semantic statements that control the processing steps.
Semantic Statements 1-5 Processing Steps 1-10 low low low 6-10 11+

11-20

low

average

high

21+

average

high

high

Determining the complexity of a transformation for 3D function points

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Control Dimension
Measured by counting the number of transitions between states

A state represents some externally observable mode of behavior, and a transition occurs as a result of some event that causes the software or system to change its mode of behavior. For example, a cellular phone auto-dial state resting state dialing state

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

19

To compute 3D function points index = I+ O +Q+F+ E+ T+ R where I, 0, Q, F, E, T, and R represent inputs, outputs, inquiries, internal data structures, external files, transformations, and transitions, respectively.. Each complexity weighted value is computed using the following relationship complexity weighted value = Nil Wil + Nia Wia + Nih Wih where Nil , Nia , and Nih represent the number of occurrences of element i (e.g., outputs) for each level of complexity (low, average, high) Wil , Wia and Wih are the corresponding weights.

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Function Point
The function point ,like the LOC, measure is controversial Proponents claim that FP is programming language independent, making it ideal for applications using conventional and nonprocedural languages It is based on data that are more likely to be known early in the evolution of a project Opponents claim that The method requires some "sleight of hand" in that computation s i based on subjective, rather than objective, data Counts of the information domain (and other dimensions) can be difficult to collect after-the fact FP has no direct physical meaning - it's just a number.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

20

Computing 3-D Function Point

Complexity Weighting measurement parameter internal data structures external data number of user inputs number of user outputs number of user inquiries transformations transitions 3D function point index

low 7 5 3 4 3 7
+

average

high

10 + 7 4 5 4
+

15 = 10 = 6 7 6 = = =

10 + n/a +

15 = n/a =

n/a +

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

RECONCILING DIFFERENT METRICS APPROACHES

The relationship between lines of code and function points depends programming language Rough estimates of the average number of lines of code required to build one function point in various programming languages Programming Language LOC/FP (average) assembly language 320 C 128 Cobol 105 Fortran 105 Pascal 90 Ada 70 object -oriented languages 30 fourth generation languages (4GLs) 20 code generators 15 spreadsheets 6 graphical languages (icons) 4 LOC and FP measures are often used to derive productivity metrics. Computing the 3D function point index
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

21

Software Productivity
Five important factors that influence software productivity: People factors. The size and expertise of the development organization. Problem factors . The complexity of the problem to be solved and the number of changes in design constraints or requirements. Process factors . Analysis and design techniques that are used, languages and CASE tools available, and review techniques. Product factors. Reliability and performance of the computer-based system. Resource factors . Availability of CASE tools and hardware and software resources.

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

METRICS FOR SOFTWARE QUALITY

The overriding goal of software engineering is to produce a high-quality system, application, or product. The quality of a system, application, or product is only as good as The requirements that describe the problem The design that models the solution The code that leads to an executable program The tests that exercise the software to uncover errors. To accomplish this real-time quality assessment, the engineer must use technical measures to evaluate quality in objective, rather than subjective, ways.

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

22

Measuring Quality
Correctness : The degree to which the software performs its required function. Common measure: Defects per KLOC , where a defect is defined as a verified lack of conformance to requirements. Maintainability: The ease with which a program can be corrected if an error is encountered, adapted if its environment changes, or enhanced if the customer desires a change in requirements. A simple time -oriented metric Mean-time -to-change (MTTC), the time it takes to analyze the change request, design an appropriate modification, implement the change, test it, and distribute the change to all users
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Measuring Quality
Integrity Measures a system's ability to withstand attacks (both accidental and intentional) on its security. Attacks programs, data, and documents. To measure integrity, two additional attributes must be defined Threat Security Threat: The probability that an attack of a specific type will occur within a given time.

Security The probability that the attack of a specific type will be repelled. integrity = [1 - threat x (1 - security)] where threat and security are summed over each type of attack.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

23

Measuring Quality
Usability User friendliness. If a program is not "user friendly," it is often doomed to failure, even if the functions that it performs are valuable User friendliness can be measured in terms of four characteristics
the physical and/or intellectual skill required to learn the system the time required to become moderately efficient in the use of the system the net increase in productivity measured when the system is used by someone who is moderately efficient a subjective assessment of users attitudes toward the system.

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

Defect Removal Efficiency

DRE = E / (E + D) where E = number of errors found before delivery of the software to the end user D = number of defects found after delivery The ideal value for DRE is 1. No defects are found in the software Realistically, D will be greater than zero, but the value of DRE can still approach 1 as E increases As E increases it is likely that the final value of D will decrease

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

24

Defect Removal Efficiency

DRE can also be used within the project to assess a team's ability to find errors before they are passed to the next framework activity

DREi = Ei / (Ei + Ei + 1 ) where Ei = number of errors found during software engineering activity i. Ei + 1 = number of errors found during software engineering activity i + 1 that are traceable to errors that were not discovered in software engineering activity i.

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

INTEGRATING METRICS WITHIN THE SOFTWARE PROCESS

The majority of software developers still do not measure. The problem is cultural. If we do not measure there is no real way of determining whether we are improving. Software project managers are concerned with more mundane issues :
developing meaningful project estimates producing higher-quality systems getting product out the door on time

by using measurement to establish a project baseline, each of these issues becomes more manageable.

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

25

INTEGRATING METRICS WITHIN THE SOFTWARE PROCESS

As the software design is completed
Which user requirements are most likely to change? Which modules in this system are most error prone? How much testing should be planned for each module? How many errors (of specific types) can I expect when testing commences?

Answers to these questions can be determined if metrics have been collected and used as a technical guide.

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

INTEGRATING METRICS WITHIN THE SOFTWARE PROCESS

Data collection requires an historical investigation of past projects to reconstruct required data. Metrics computation can span a broad range of LOC or FP metrics as well as other quality and project-oriented metrics Metrics evaluation focuses on the underlying reasons for the results obtained and produces a set of indicators that guide the project or process.

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

26

FURTHER READINGS AND OTHER INFORMATION SOURCES

On the Internet, Research reports and pointers to information on software metries are available at the Software Engineering Laboratory: http://fdd.gsfc.nasa.gov/seltext.html The U.S. Army software Metric System Web site contains a variety of useful information on process metrics: http://www.army.mil/optecpg /homepage.htm A comprehensive listing of textbooks and papers on process metrics can be obtained at: http://www.rai.com/soft_eng/sme.html

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

FURTHER READINGS AND OTHER INFORMATION SOURCES

A Listserv mailing list that addresses function point metrics has been established. To subscribe, send mail to: [email protected] SUBJECT : "none" (this field must be empty) CONTENT: SUB FUNCTION.POINT.LIST "Your name" An up-to-date list of World Wide Web references for software process metrics can be found at: http://www.rspa.com

ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING

27