New Modelling Concepts For Todays Software Process
New Modelling Concepts For Todays Software Process
New Modelling Concepts For Todays Software Process
Editorial Board
David Hutchison
Lancaster University, UK
Takeo Kanade
Carnegie Mellon University, Pittsburgh, PA, USA
Josef Kittler
University of Surrey, Guildford, UK
Jon M. Kleinberg
Cornell University, Ithaca, NY, USA
Alfred Kobsa
University of California, Irvine, CA, USA
Friedemann Mattern
ETH Zurich, Switzerland
John C. Mitchell
Stanford University, CA, USA
Moni Naor
Weizmann Institute of Science, Rehovot, Israel
Oscar Nierstrasz
University of Bern, Switzerland
C. Pandu Rangan
Indian Institute of Technology, Madras, India
Bernhard Steffen
TU Dortmund University, Germany
Madhu Sudan
Microsoft Research, Cambridge, MA, USA
Demetri Terzopoulos
University of California, Los Angeles, CA, USA
Doug Tygar
University of California, Berkeley, CA, USA
Gerhard Weikum
Max-Planck Institute of Computer Science, Saarbruecken, Germany
Jürgen Münch Ye Yang
Wilhelm Schäfer (Eds.)
13
Volume Editors
Jürgen Münch
Fraunhofer Institute for
Experimental Software Engineering
Kaiserslautern, Germany
E-mail: [email protected]
Ye Yang
Institute of Software
Chinese Academy of Sciences
Beijing, China
E-mail: [email protected]
Wilhelm Schäfer
Department of Computer Science
University of Paderborn
Paderborn, Germany
E-mail: [email protected]
ISSN 0302-9743
ISBN-10 3-642-14346-6 Springer Berlin Heidelberg New York
ISBN-13 978-3-642-14346-5 Springer Berlin Heidelberg New York
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is
concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting,
reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication
or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965,
in its current version, and permission for use must always be obtained from Springer. Violations are liable
to prosecution under the German Copyright Law.
springer.com
© Springer-Verlag Berlin Heidelberg 2010
Printed in Germany
Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India
Printed on acid-free paper 06/3180
Preface
2010 was the first time that the International Conference on Software Process was held
autonomously and not co-located with a larger conference. This was a special challenge
and we are glad that the conference gained a lot of attention, a significant number of
contributions and many highly interested participants from industry and academia.
This volume contains the papers presented at ICSP 2010 held in Paderborn, Ger-
many, during July 8-9, 2010. ICSP 2010 was the fourth conference of the ICSP series.
The conference provided a forum for researchers and industrial practitioners to ex-
change new research results, experiences, and findings in the area of software and
system process modeling and management.
The increasing distribution of development activities, new development paradigms
such as cloud computing, new classes of systems such as cyber-physical systems, and
short technology cycles are currently driving forces for the software domain. They
require appropriate answers with respect to process models and management, suitable
modeling concepts, and an understanding of the effects of the processes in specific
environments and domains. Many papers in the proceedings address these issues.
Due to the financial crisis that started in 2008, many software and IT units of or-
ganizations have been forced to show their contributions to the business goals of the
organization. If software or IT units cannot demonstrate their value with respect to
business goals, they are typically seen as cost factors and exposed to risks such as
budget cuts or personnel reduction. Therefore, one of the main themes that was ad-
dressed at ICSP 2010 was the so-called alignment of processes to an organization’s
higher-level goals. This includes the strategic alignment of processes to business
goals with methods such as GQM+Strategies, but also topics such as the evaluation of
processes with respect to their fulfilment of process requirements.
Further trends in the area of software and system processes can be seen with re-
spect to a more careful understanding of planning and re-planning processes, the
management of multiple process standards in parallel, the design of process models
for large systems of systems, the design of variant-rich processes (also referred to as
process lines or process families), the relationship between change management and
process improvement, the provision of repositories for reusable simulation model
components, and the visualization of software process data. The topics indicate that
software process modeling remains a vibrant research discipline of high interest to
industry. Emerging technologies and application domains, a paradigm shift to global
software and systems engineering in many domains, and the need for a better under-
standing of the effects of processes are reflected in these papers.
In response to the Call for Papers, 49 submissions were received by authors from
Australia, Brazil, Canada, Chile, China, Finland, France, Germany, Ireland, Japan, Mex-
ico, The Netherlands, Spain, Sweden, Switzerland, Turkey, and the USA. Three papers
were rejected due to double submissions. The other papers were rigorously reviewed
and held to very high quality standards. Due to the high quality of the overall submis-
sions, 31 papers were finally accepted as regular papers for presentation at the
conference.
VI Preface
A conference such as this one can only succeed as a team effort. All of this work
would not have been possible without the dedication and professional work of many
colleagues. We wish to express our gratitude to all contributors for submitting papers.
Their work formed the basis for the success of the conference. We would also like to
thank the Program Committee members and additional reviewers because their work
is the guarantee for the high quality of the conference. Special thanks go to the key-
note speakers for giving excellent presentations at the conference. We would also like
to thank the members of the Steering Committee, Barry Boehm, Mingshu Lee, Leon
Osterweil, David Raffo, and Wilhelm Schäfer, for their advice, encouragement, and
support.
We wish to thank the University of Paderborn, the Fraunhofer Institute for Ex-
perimental Software Engineering (IESE), the Chinese Academy of Science (ISCAS),
and all the other sponsors and supporters for their contributions and for making the
event possible. Last but not least many thanks to Jutta Haupt and Jürgen Maniera for
the local organization. For further information, please visit the conference website at
http://icsp10.upb.de/.
Paderborn, Germany
July 8–9, 2010
Steering Committee
Barry Boehm University of Southern California, USA
Mingshu Li Institute of Software, Chinese Academy of Sciences,
China
Leon J. Osterweil University of Massachusetts, USA
David M. Raffo Portland State University, USA
Wilhelm Schäfer University of Paderborn, Germany
General Chair
Wilhelm Schäfer University of Paderborn, Germany
Program Co-chairs
Jürgen Münch Fraunhofer Institute for Experimental Software
Engineering, Germany
Ye Yang Institute of Software, Chinese Academy of Sciences,
China
Secretary
Lin Shi Institute of Software, Chinese Academy of Sciences,
China
External Reviewers
Andrew Tappenden University of Alberta, Canada
Da Yang Institute of Software, Chinese Academy of Sciences,
China
Gregory Gay West Virginia University, USA
Juan Li Institute of Software, Chinese Academy of Sciences,
China
Junchao Xiao Institute of Software, Chinese Academy of Sciences,
China
Qiusong Yang Institute of Software, Chinese Academy of Sciences,
China
Jian Zhai Institute of Software, Chinese Academy of Sciences,
China
Table of Contents
Invited Talk
A Risk-Driven Decision Table for Software Process Selection . . . . . . . . . . . 1
Barry W. Boehm
Process Alignment
Using Process Definitions to Support Reasoning about Satisfaction of
Process Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Leon J. Osterweil and Alexander Wise
Process Management
Evidence-Based Software Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Barry Boehm and Jo Ann Lane
Process Models
The Incremental Commitment Model Process Patterns for
Rapid-Fielding Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Supannika Koolmanojwong and Barry Boehm
Process Representation
Visual Patterns in Issue Tracking Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
Patrick Knab, Martin Pinzger, and Harald C. Gall
Barry W. Boehm
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, p. 1, 2010.
Using Process Definitions to Support Reasoning
about Satisfaction of Process Requirements
1 Introduction
In earlier work the authors have suggested that software development processes
seem to have much in common with application software [1,2]. One similarity
was that just as applications are defined using a programming language, so
might the processes used to construct such applications also be defined using
a process programming language. This has led to the development of several
such languages, one of which (Little-JIL) is to be described later in this paper
[3]. The focus of this paper, however, is on a different similarity, namely that
both applications and the processes used to develop them should be designed so
as to demonstrably satisfy their requirements. Just as applications must satisfy
requirements (e.g. in dimensions such as functionality, speed, responsiveness,
robustness, and evolvability), so must development processes also satisfy similar
kinds of requirements. For example, software development processes must satisfy
speed requirements, namely that they complete within the time limit imposed by
customers and users. Development processes also have functional requirements
that specify such things as the artifacts they are to produce (i.e. only code, code
and design specifications, code and voluminous testing result reports, etc.).
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 2–13, 2010.
c Springer-Verlag Berlin Heidelberg 2010
Using Process Definitions to Support Reasoning about Satisfaction 3
This paper suggests the importance of devising a technology that supports the
demonstration that software development process implementations meet speci-
fied process requirements, and suggests some initial capabilities for supporting
that technology. At present the selection of development process approaches to
meet process requirements is typically done informally, generally based upon
intuition and anecdotal evidence. Thus, for example there is a general under-
standing that heavyweight processes such as those guided by the application of
such approaches as the CMMI [4] can be expected to lead to software products
that are relatively more robust, well- documented, and designed for evolvability
over a long lifespan. On the other hand such processes seem to be less well-
suited to the rapid production of software products, especially those that are
not expected to be used for an extended period of time, or evolved in an or-
derly way. Indeed the popularity of agile approaches [5] seems to have derived
largely from a sense that such development processes are more appropriate for
the rapid development of software products, especially those that are relatively
small, and destined for relatively short lifespans. While there is considerable
anecdotal evidence to suggest that these intuitions are well-supported by experi-
ence, we suggest that the selection of a software development process should not
be left entirely to intuition, and might better be done with at least the support
of engineering approaches and technologies.
An essential support for this more disciplined approach to the selection of a
software development process would seem to be technology for inferring the prop-
erties and characteristics of such processes. This, in turn, would seem to require
that these processes be defined with sufficient detail to render them amenable
to such analysis. Thus, this paper suggests how to use a process definition as
the basis for inferring the properties of a process, so that the inferred properties
might be compared to specifications of the requirements that the process is to
fulfill.
The example presented in this paper is the use of analyses of a definition of
the Scrum software development approach [6] written in the Little-JIL process
definition language. Specifically, the paper introduces the Little-JIL language in
section 2. In Section 3 Little-JIL is used to define a specific version of the Scrum
software development approach. This definition itself is shown to immediately
suggest some process properties and characteristics. Section 4 then demonstrates
how analysis of the Scrum definition can derive a property that seems suitable
for comparison to an expectable type of process requirement. The paper con-
cludes with an evaluation of this process analysis technology in section 6, and a
suggestion for future work in section 7.
product : Product
sprint backog channel: BacklogChannel
Development Iteration
Oagent: Team
Oagent: ScrumMaster
product product Oowner: ProductOwner product product
deadline: Hours = 4
Sprint Planning Meeting Sprint product: Product Sprint Retrospective
Sprint Review
might be. The Little-JIL definition that is the basis for the work described here
is based upon the informal Scrum description provided in [6].
As suggested by the preceding discussion, this process is defined hierarchically.
At the highest level, Scrum involves three concurrent activities: the management
of the product and release backlogs, and the iterative development of the prod-
uct. For our example, we elaborate only the heart of the Scrum process: the
iterative performance of “sprints.” The Development Iteration step (Figure 1)
specifies how one of these iterations is carried out. An iteration begins with a
Sprint Planning Meeting to determine the work to be performed during the cur-
rent iteration. This body of work is represented by the sprint backlog artifact
and is stored in the sprint backlog channel to prevent concurrent updates. The
Sprint Planning Meeting step is followed by the Sprint step in which the work is
actually performed. The iteration concludes with the Sprint Review step and the
Sprint Retrospective step. The Sprint Review step is a time-boxed meeting (note
the diamond annotation that specifies a fixed deadline for completion of this
step) led by the ScrumMaster agent with the support of the ProductOwner and
team as resources. This step takes as an input artifact (note downward arrow
annotation) the product artifact that was produced as the output of the Sprint
step. The purpose of this Sprint Review step is to enable the team to discuss the
results of the preceding sprint, and to close the loop between the product owner
and the team. After the Sprint Review step concludes, the Sprint Retrospective
is carried out with the team as the agent, indicating that the team meets in
private to assess its performance during the last sprint as preparation for the
next sprint.
3.1 Sprint
The Sprint subprocess (Figure 2) is the activity during which actual development
work gets done. To be more precise, the Sprint process consists of 30 (note the
30 annotation on the edge connecting the Sprint parent step to its Daily Sprint
child step) consecutive (note the right arrow step kind badge in the Sprint step)
performances of the Daily Sprint subprocess. As indicated by the = sign badge in
the Daily Sprint step, this subprocess is carried out as the parallel performance
of its three substeps, Daily Scrum, Work, and Revise Sprint Backlog. Both the
6 L.J. Osterweil and A. Wise
product: Product
Sprint
product product
product: Product
30
deadline: Days = 1
Daily Scrum and the Revise Sprint Backlog steps require both access to, and
update capability for, the sprint backlog. These accesses for these two steps are
coordinated by using the sprint backlog channel to provide the needed concurrent
access permissions.
The Daily Scrum step is a 15 minute (note the specification of this deadline
by means of the diamond annotation) progress meeting during which the team
meets to assess their progress. Note that the sprint backlog artifact is passed in
as a parameter, and then also passed out of this step, after which it is written
to the sprint backlog channel so that it is made available to the Revise Sprint
Backlog step, which may be executing in parallel.
In addition to the execution of the Daily Scrum step, there are multiple per-
formances of the Work step (note the + sign on the edge connecting the Work
step to its parent). Each instance of the Work step produces a new version of
the product artifact, presumably being comprised of more completed work items
after the execution of this step. The agent for this step is the team.
Concurrently with the performances of the Work step there may be multiple
performances of the Revise Sprint Backlog step (note the * on the edge connect-
ing this step to its parent). The agent for this step is also team, and the effect of
a performance of this step is to update the sprint backlog to reflect the addition
or removal of tasks in the backlog.
Modeling Continuous Integration. One way to build such details into the
process definition is to integrate the Continuous Integration method into the
Scrum method by replacing the step Work in the original Scrum definition, with
a sub-process Checked Work (Figure 4). In Checked Work, the original Work
step is followed by the Integrate step, whose purpose is specifically to integrate
the work just completed with prior work products. The successful integration
of this new work is verified by the performance of a post-requisite( represented
by a upward-pointing triangle on the right of a step bar) to the Integrate step
that verifies the correctness of the integrated artifact. If the verification does
not succeed, then the Rework step is performed, to make whatever modifications
are necessary in order to ensure that the required modifications are carried out.
Details of the Continuous Integration method are not provided here, as the
purpose of this section is to indicate how process rigor can support greater
assurance about the success of method integration. Details of the Continuous
Integration method definition would look very analogous to the details of the
Scrum method definition.
As shown in Figure 5, the inclusion of Continuous Integration eliminates the
single point of failure step in this definition. For this modified method, if ”Step
‘Work’ produces the wrong ‘product’ ”, then it will still be necessary for ”Excep-
tion ’BuildFailed’ is thrown by step ’integrate’ ” to fail to be thrown (note that
Checked Work
Exception "BuildFailed" is thrown Step "Work" produces wrong Step "Integrate" produces wrong
by step "Integrate" "product" "product"
Begin Sprint
the use of guided questions and disciplined natural language to aid the user in
the creation of property specification. For our property, we have used PROPEL
to specify that “Change Sprint Backlog” may not occur between a “Start Sprint”
event and an “End Sprint” event, resulting in the finite-state machine shown in
Figure 6.
Checking this property reveals that this property does not in fact hold for
Scrum, producing an example trace, which after some preamble indicates that:
As this trace shows, the team is permitted to make changes to the sprint
backlog during the sprint via the Revise Sprint Backlog step shown in Figure 2.
Indeed, an anecdote in [6] describes a situation in which the Product Owner was
bypassed and individuals were approaching team members and persuading them
to make changes during the sprint. If we assume that the team members redirect
any direct requests through the Product Owner as they are instructed to, then
we can revise our analysis to disregard changes made to the sprint backlog via
the “Revise Sprint Backlog” step, and then the property holds, showing that the
Scrum process does indeed protect the team since only they themselves are able
to make changes to the sprint backlog during the sprint.
5 Related Work
Many authors have addressed the problem of modeling, specifying, and defining
various software development processes. Prime venues for this work include the
International Conference on the Software Process (ICSP) series[15], and the jour-
nal Software Process Improvement and Practice (SPIP). Most of this literature
attempts to model processes informally or with pictorial diagrams. In partic-
ular the Scrum approach to software development is described mostly in this
Using Process Definitions to Support Reasoning about Satisfaction 11
way. Some examples are [6,7,8,9,10]. There have been a number of noteworthy
attempts to create more rigorous notations with which to define processes (e.g.
[16,17,18,19]). But these have typically not attempted to define popular software
development processes. One notable exception is the Unified Process (UP) [20],
which is defined precisely in terms of well-defined formalism. But even the UP
definition is not then analyzed in order to infer properties for comparison with
requirements. This paper seems unique in demonstrating that this is plausible
and desirable.
6 Evaluation
7 Future Work
This first demonstration of the use of process definition and analysis suggests
that further kinds of analysis might also be successful in supporting additional
demonstrations of the adherence of process definitions to additional desired re-
quirements. For example, by attaching time limits to all steps (not just the steps
in the example in section 3), and bounding the iterations (also necessary for
finite-state verification) analyzers should be able to determine the maximum
time required to carry out the specified process and thus determine if the pro-
cess must always meet speed and responsiveness requirements. Discrete event
12 L.J. Osterweil and A. Wise
simulation could also be used to study the adherence of processes to other kinds
of requirements.
Finally we note that the application of these analyses to Scrum process defini-
tions was used only as an example. These analysis approaches should be equally
applicable to other types of processes to derive various sorts of properties and
characteristics. Indeed this approach should be equally applicable to processes
defined in other languages, as long as those languages have rigorously defined
semantics. It seems particularly interesting to consider the possibility that some
set of properties and characteristics might be identified that could be used to
define the nature of such types of processes as Scrum. We suggest that a defini-
tion of what is quintessentially a “Scrum process” might well best be done by
enunciating a set of properties and characteristics, rather than some canonical
structure of steps. Analyses of the sorts suggested here might then be used to
determine which actual processes are Scrum processes by subjecting them to
analyses that determine whether or not the processes adhere to the defining set
of properties and characteristics.
Acknowledgments
The authors gratefully acknowledge the many stimulating conversations with
Aaron Cass, Bobby Simidchieva, M.S. Raunak, and Junchao Xiao, all of which
have helped shape this paper. This work also builds fundamentally on the work
of Bin Chen, Lori A. Clarke, and George S. Avrunin who have creating tools and
technological approaches that enable the Finite State Verification and Fault Tree
Analysis of process definitions. In addition Gordon Anderson’s early versions of
the Scrum process definition provided an invaluable starting point for this work.
Financial support for this work was provided by the National Science Founda-
tion under Award Nos. CCR-0427071 and CCR- 0205575, and a subcontract from
Stevens Institute of Technology under the auspices of the DoD-sponsored Sys-
tems Engineering Research Center (SERC). The U.S. Government is authorized
to reproduce and distribute reprints for Governmental purposes notwithstanding
any copyright annotation thereon. The views and conclusions contained herein
are those of the authors and should not be interpreted as necessarily representing
the official policies or endorsements, either expressed or implied of The National
Science Foundation, or the U.S. Government.
References
1. Osterweil, L.J.: Software processes are software too. In: 9th International Confer-
ence on Software Engineering (ICSE 1987), Monterey, CA, March 1987, pp. 2–13
(1987)
2. Osterweil, L.J.: Software processes are software too, revisited. In: 19th International
Conference on Software Engineering (ICSE 1997), Boston, MA, May 1987, pp. 540–
548 (1997)
3. Wise, A.: Little-JIL 1.5 language report. Technical Report UM-CS-2006-51, De-
partment of Computer Science, University of Massachusetts, Amherst, MA (2006)
Using Process Definitions to Support Reasoning about Satisfaction 13
4. CMMI Product Team: CMMI for development, version 1.2. Technical Report
CMU/SEI-2006-TR-008, Software Engineering Institute, Carnegie Mellon Univer-
sity, Pittsburg, PA (August 2006)
5. Highsmith, J., Fowler, M.: The agile manifesto. Software Development Maga-
zine 9(8), 29–30 (2001)
6. Schwaber, K., Beedle, M.: Agile Software Development with Scrum. Prentice Hall,
Upper Saddle River (2002)
7. Schwaber, K.: Agile Project Management with Scrum. Microsoft Press, Redmond
(2004)
8. Cohn, M.: Succeeding with Agile: Software Development Using Scrum. Pearson
Education, Inc., Boston (2010)
9. Scrum Alliance, Inc., http://www.scrumalliance.org/
10. Schwaber, K.: http://www.controlchaos.com/
11. Chen, B., Avrunin, G.S., Clarke, L.A., Osterweil, L.J.: Automatic fault tree
derivation from little-jil process definitions. In: Wang, Q., Pfahl, D., Raffo, D.M.,
Wernick, P. (eds.) SPW 2006 and ProSim 2006. LNCS, vol. 3966, pp. 150–158.
Springer, Heidelberg (2006)
12. Chen, B., Avrunin, G.S., Henneman, E.A., Clarke, L.A., Osterweil, L.J., Henneman,
P.L.: Analyzing medical processes. In: ACM SIGSOFT/IEEE 30th International
Conference on Software Engineering (ICSE 2008), Leipzig, Germany, May 2008, pp.
623–632 (2008)
13. Dwyer, M.B., Clarke, L.A., Cobleigh, J.M., Naumovich, G.: Flow analysis for ver-
ifying properties of concurrent software systems. ACM Transactions on Software
Engineering and Methodology (TOSEM) 13(4), 359–430 (2004)
14. Cobleigh, R.L., Avrunin, G.S., Clarke, L.A.: User guidance for creating precise and
accessible property specifications. In: ACM SIGSOFT 14th International Sympo-
sium on Foundations of Software Engineering (FSE14), Portland, OR, November
2006, pp. 208–218 (2006)
15. International Conference on Software Process,
http://www.icsp-conferences.org/
16. Dami, S., Estublier, J., Amiour, M.: Apel: A graphical yet executable formalism
for process modeling. Automated Software Engineering 5(1) (1998)
17. Katayama, T.: A hierarchical and functional software process description and its
enaction. In: Proceedings of the 11th international conference on Software engi-
neering, Pittsburgh, PA, pp. 343–352 (1989)
18. Kaiser, G., Barghouti, N., Sokolsky, M.: Experience with process modeling in the
marvel software development environment kernel. In: 23rd Annual Hawaii Interna-
tionall Conference on System Sciences, pp. 131–140 (1990)
19. OMG: Software & systems process engineering meta-model specification. Technical
Report formal/2008-04-01, Object Management Group (2008)
20. Jacobson, I., Booch, G., Rumbaugh, J.: The Unified Software Development Process.
Addison-Wesley Longman, Inc., Reading (1999)
21. Christov, S., Avrunin, G., Clarke, L.A., Osterweil, L.J., Henneman, E.: A bench-
mark for evaluating software engineering techniques for improving medical pro-
cesses. In: International Conference on Software Engineering, Workshop on Soft-
ware Engineering in Health Care (SEHC 2010), Cape Town, South Africa (May
2010)
Early Empirical Assessment of the Practical Value of
GQM Strategies
1 Introduction
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 14–25, 2010.
c Springer-Verlag Berlin Heidelberg 2010
Early Empirical Assessment of the Practical Value of GQM Strategies 15
we found it important to address the method’s practical value in our research. In our
case, practical value is defined as a capability of solving relevant practical problems
encountered by the case organization.
We carried out a case study of GQM Strategies applications within the Finnish ICT
industry. The main research goal was to evaluate the practical value of GQM Strat-
egies as perceived by the company representatives. Because the method is new and
not yet widely adopted by the industry, the only way to get immediate feedback about
it is to obtain expert opinions from the practitioners involved using the method. We
defined a five-step research approach for our case in order to improve the validity and
repeatability of the study.
For a successful adoption of any method or technology transfer in general, the un-
derstandability of foundation concepts is a prerequisite [5,6]. Therefore, we designed
a questionnaire based on the revised Bloom’s taxonomy [7] to evaluate practitioners’
cognition level of the foundation concepts of the GQM Strategies approach.
The rest of the paper is structured as follows: in Section 2, an overview of the
GQM Strategies approach and revised Bloom’s taxonomy is given; our research ap-
proach is explained in Section 3; Section 4 presents the results of the assessment of
method’s practical value; validity issues of our research are discussed in Section 5; our
final remarks and concluding statements are made in Section 6.
Strategy
INFLUENCES
LEADS TO A
Interpretation Model
SET OF
IS PART OF
Table 1. GQM Strategies goal formalization template with an example. The template specifies
eight goal elements (dimensions).
the operational level). The concept of levels is convenient for grouping and organizing
GQM Strategies elements.
The entire process of deriving business goals and GQM graphs is consolidated
through the interpretation model. During the interpretation process, measured GQM
goals and statuses of the contextassumption variables influence assessment of business
goal realization.
During the GQM Strategies grid derivation process, two parallel threads are
running: one is related to defining business goals, contextassumption elements, and
strategies for addressing goals and the other is related to defining measurable goals and
actually deriving the GQM graph. In the following paragraphs we give an overview of
the grid derivation process [10].
Elicit General Context and Assumptions. First, the organizational (business) envi-
ronment is defined by specifying context factors. Second, uncertainties are documented
using assumptions, which represent beliefs or estimates regarding relevant business
issues.
Early Empirical Assessment of the Practical Value of GQM Strategies 17
Define Top-Level Goals. First, an initial set of high-level goals is identified. Second,
the goals have to be prioritized and analyzed for potential conflicts. Third, the selected
goals are formalized using the GQM Strategies goal template (Table 1). Potential rela-
tionships with other goals are also listed.
Make Strategy Decisions. First, a list of potential strategies for achieving the business
goals is identified. Second, the most promising strategy has to be selected, considering
such factors as cost and benefit analysis.
Define Goals. First, the implications of the chosen upper-level strategies with respect to
lower-level goals have to be determined. Second, potential lower-level goals are iden-
tified based on the goal analysis. Third, the most promising goal with respect to feasi-
bility, cost, and benefit is selected. Fourth, the most promising goals are selected and
formalized using the goal template.
Define GQM Graphs. The GQM graph derivation process is well documented in the
literature2 .
For the purpose of evaluating the understandability of the GQM Strategies concepts,
we used the revised Bloom’s taxonomy as a framework.
The first version of the taxonomy of educational objectives, also known as Bloom’s
taxonomy, was introduced over 50 years ago. Since then, it has been used to classify cur-
ricular objectives on countless occasions. In addition to a measurement framework for
educational objectives, the taxonomy provides a common language for defining learn-
ing goals [7].
In 2001, a revised version of Bloom’s taxonomy was introduced by Anderson et
al. [7]. In the new version, the knowledge dimension is defined more precisely and a
taxonomy table can be used to categorize the learning objectives within the cognitive
and knowledge dimensions. The structure of the refined taxonomy is as follows:
– The first level of the taxonomy concerns remembering the learning objective. That
involves recognizing relevant concepts and recalling them.
– The second level requires understanding the relevant knowledge in di erent types
of communication. The subcategories for this level include: interpreting, exempli-
fying, classifying, summarizing, inferring, comparing and explaining the material.
– The third level in the taxonomy is applying the achieved knowledge in a given
situation. It involves executing and implementing the procedure.
– The fourth level requires that the learner can analyze the material by breaking it
into components and identifying the relationships between the parts, as well as
the overall structure. Subcategories at this level are: di erentiating, organizing and
attributing the knowledge.
– The fifth level assumes that the learner can evaluate the subject of learning based
on given criteria. This level is further divided into checking and critiquing.
2
For example, see: van Solingen, R., Berghout, E.: The Goal Question Metric Method. The
» »
– At the sixth level, the learner can create the subject. The subcategories for this level
are: generating, planning and producing.
The three lowest levels comprise the basic skills, while three upper levels are higher
order thinking skills [7].
Bloom’s taxonomy is especially useful for defining targets for competence levels. It
provides a well-known framework for planning and assessing the knowledge and skills
that will be needed to adopt a method in an organization successfully. Pfleeger [11]
states that it is important for the practitioners interested in utilizing new technologies
to understand clearly the use and benefits of the new approach. Rogers [5] emphasizes
that the understandability of a new method is a prerequisite for its successful adoption.
Understandability does not mean merely comprehensive guidelines and documentation.
3 Research Approach
The research goal was to evaluate the practical value of the GQM Strategies method
and thus provide initial empirical experiences of using the method in real-life indus-
trial settings. We carried out the research in one Finnish ICT company. Our research
approach consists of five steps, which are the following:
of the problem analysis. This phase was theory-based and done by the researchers. The
solution proposals were based on the best practices reported in literature. The main
purpose of these first two steps was to identify relevant problems to the organization
and to communicate how GQM Strategies could help in solving them.
Fig. 2. Questionnaire related to Bloom’s taxonomy. The header part of the questionnaire is not
given here in its integrated form.
Early Empirical Assessment of the Practical Value of GQM Strategies 21
The company has had defined strategic goals and objectives, which were the starting
point to derive the grid and eventually to specify what metrics to collect. The top-level
business goal was defined as: activity to decrease (focus) the time-to-market for (object)
ABC type of products for the magnitude of 10% of current time-to-market; within a
timeframe of one year in the context (scope) of Finland corporate site with constraints
of current resources availability. Relations to the corporate policy.
Further context analysis indicated that restructuring the development process could
gain a suÆcient decrease in time-to-market. The entire grid contained three levels, which
reflected the organizational structure (top-level management, mid-level management,
and the operational level). The operational level goal made it possible to generate action
points, that would be used for planning an internal project to implement the strategy.
Table 2. Descriptive statistics for the method’s cognition level (Levels 1–6) and expert opinion
questions (Q7 and Q8). N 12.
The most frequent answer for levels 2, 3, and 4 was that participants felt capable
of performing steps under the supervision of an expert. However, the distribution of
the answers shows an increased confidence in performing steps on level 2 (Q1 2
and Mode 3). Further, the confidence was gradually reduced until level 4 where the
dispersion of the answers was highest (IQR 15).
The ability to perform level 4 steps with almost the same confidence as level 3 steps
was a surprise. The probable explanation for this is in the method, which was designed
3
For example, see: C. J. Wild and G. A. F. Seber: Chance Encounters, John Wiely & Sons, Inc.,
1999.
24 V. Mandić et al.
to handle relevant context for a particular situation or case. Therefore the participants’
natural familiarity with the context made them comfortable with level 4 tasks.
The mode and median statistics indicate that level 5 and 6 responses are clearly
shifted for one point on the scale. In general, participants felt that they would not be
able to perform evaluation and creation steps.
5 Study Validity
Regarding the validity of the study, we identified and addressed following validity
issues:
Lack of practical experience. Research work with GQM Strategies is in the early
phase; there are not yet published practical experiences with the method usage. There-
fore, we designed a five-step research approach in order to assess experts’ opinions. The
research approach had twofold purpose: first, to transfer knowledge about the method
in the experts’ natural environment (analyzing practical problems and suggesting how
the method can address those); second, to confirm success of the knowledge transfer
with the help of Bloom’s taxonomy before assessing experts’ opinions.
Expert’s self-assessment. Designing a several pages long exam-like questionnaire
for Bloom’s taxonomy was not acceptable for this type of subjects (top-level managers,
chief oÆcers, and so on). Therefore, we limited the length of our questionnaire to two
A4 pages. And, presumably we trust subjects in their self-assessments. Under these
constraints, we find it important to replicate this study in order to increase the data set
and then to perform outlier analysis.
6 Conclusions
This paper provides an early empirical assessment of the practical value of GQM Strat-
egies method. The empirical study has demonstrated that the method has its practical
value for the case company. This inference has been made upon two premises: first,
practitioners’ opinions that the method can be adequately instrumented to address their
problems in a manner beneficial for the company; second, that such opinion has came
from the very practitioners who understood the method competently, according to the
revised Bloom’s taxonomy.
We refer to this study as an early empirical assessment because it was carried out
in one case company on a relatively small sample. Our future research will include
replications of this study in order to increase the data set.
Early Empirical Assessment of the Practical Value of GQM Strategies 25
Acknowledgments. Our research was funded by the Finnish Funding Agency for
Technology and Innovation (Tekes). We would also like to express special thanks to
the GQM Strategies authors Victor R. Basili, Jens Heidrich, Michael Lindvall, Jürgen
Münch, Carolyn B. Seaman, Myrna Regardie, and Adam Trendowicz.
References
1. Hall, T., Fenton, N.: Implementing eective software metrics programs. IEEE Soft-
ware 2(14), 55–65 (1997)
2. Umarji, M., Seaman, C.: Why do programmers avoid metrics? In: 2nd ACM-IEEE inter-
national symposium on Empirical software engineering and measurement (ESEM 2008),
Kaiserslautern, Germany, pp. 129–138 (2008)
3. Basili, V., Heidrich, J., Lindvall, M., Münch, J., Regardie, M., Rombach, D., et al.: Bridging
the gap between business strategy and software development. In: Twenty Eighth International
Conference on Information Systems, Montreal, Canada, pp. 1–16 (2007)
4. Basili, V., Lindvall, M., Regardie, M., Seaman, C., Heidrich, J., Münch, J., et al.: Linking
software development and business strategy through measurement. IEEE Computer 43(4),
57–65 (2010)
5. Rogers, E.: Diusion of Innovations, 3rd edn. The Free Press, New York (1995)
6. Gorman, M.: Types of knowledge and their roles in technology transfer. Journal of Technol-
ogy Transfer 27(3), 219–231 (2002)
7. Krathwohl, D.: A revision of bloom’s taxonomy: An overview. Theory Into Practice 41(4),
212–218 (2002)
8. Basili, V., Heidrich, J., Lindvall, M., Münch, J., Regardie, M., Trendowicz, A.:
GQMStrategies –aligning business strategies with software measurement. In: First Inter-
national Symposium on Empirical Software Engineering and Measurement, ESEM 2007,
Madrid, Spain, pp. 488–490 (2007)
9. Basili, V., Caldiera, G., Rombach, D.: Goal question metric paradigm. In: Marciniak, J. (ed.)
Encyclopedia of Software Engineering, vol. 1, pp. 528–532. John Wiley & Sons, Inc., New
York (1994)
10. Basili, V., Heidrich, J., Lindvall, M., Münch, J., Seaman, C., Regardie, M., et al.: Determin-
ing the impact of business strategies using principles from goal-oriented measurement. In:
Internationale Tagung Wirtschaftsinformatik 2009, Wien, Austria, vol. 9, pp. 1–10 (2009)
11. Pfleeger, S.: Understanding and improving technology transfer in software engineering. The
Journal of Systems and Software 3(47), 111–124 (1999)
12. Chrissis, M., Konrad, M., Shrum, S.: CMMI: Guidelines for Process Integration and Product
Improvement. Addison-Wesley Professional, New York (2006)
Determining Organization-Specific Process Suitability
Ove Armbrust
Abstract. Having software processes that fit technological, project, and business
demands is one important prerequisite for software-developing organizations to
operate successfully in a sustainable way. However, many such organizations
suffer from processes that do not fit their demands, either because they do not
provide the necessary support, or because they provide features that are no
longer necessary. This leads to unnecessary costs during the development cycle,
a phenomenon that worsens over time. This paper presents the SCOPE approach
for systematically determining the process demands of current and future prod-
ucts and projects, for analyzing existing processes aimed at satisfying these de-
mands, and for subsequently selecting those processes that provide the most
benefit for the organization. The validation showed that SCOPE is capable of ad-
justing an organization’s process scope in such a way that the most suitable
processes are kept and the least suitable ones can be discarded.
1 Introduction
Many facets of process technology and standards are available in industry and acade-
mia, but in practice, significant problems with processes and process management
remain. Specifically, an organization’s process landscape often does not contain the
processes that are required to support its current activities. Typically, a number of
outdated processes exist that are not or hardly used any more, yet they still are pre-
sented as a possible choice for projects, possibly even maintained. Complementary to
this, there are often conditions for which no suitable processes exist within the or-
ganization, so whenever such a condition appears, the organization’s employees need
to improvise due to a lack of guidance. Both cases are aggravated when it comes to
future projects: There is often no pro-active preparation of an organization’s processes
for future demands. This leads to the following question: How can an organization’s
processes be managed so that they support all of the organization’s activities, current
and future, while keeping the maintenance effort on an adequate level?
This paper presents the SCOPE approach for systematically determining the process
demands of current and future products and projects, for analyzing existing processes
aimed at satisfying these demands, and for subsequently selecting those processes that
provide the most benefit for the organization. The paper is structured as follows: Sec-
tion 2 presents related work. Section 3 sheds some light on current industrial practice
with respect to process scoping. Section 4 presents the SCOPE approach, and Section 5
summarizes the validation results. Finally, Section 6 discusses the approach and gives
an outlook on possible future work.
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 26–38, 2010.
© Springer-Verlag Berlin Heidelberg 2010
Determining Organization-Specific Process Suitability 27
2 Related Work
There is a variety of related work connected to identifying suitable processes for an
organization. In this section, we distinguish product scoping approaches, technique
selection approaches, and process-aware approaches.
Product Scoping Approaches. Schmid [1] describes an approach for systematically
determining the scope for a software product line. While this approach explicitly con-
siders future products, it mostly ignores projects and processes. Bayer et al. [2] trans-
fer the concept of software product line scoping to (business) workflows, which are
by their very nature somewhat similar to software processes. However, they also only
consider products, not projects or processes, and include future development only
implicitly.
Quality Function Deployment (QFD) is an approach for directing product capabili-
ties based on customer needs. Cohen [3] defines it as “…a method for structured
product planning and development that enables a development team to specify clearly
the customer’s wants and needs, and then to evaluate each proposed product or ser-
vice capability systematically in terms of its impact on meeting those needs.” This
approach explicitly considers the anticipated future of products, but again neglects
projects and processes.
To summarize, product scoping approaches assist software engineers in building
the product that best supports their customers’ requirements. However, the ones re-
viewed do not consider processes and cannot be transferred easily. For example,
while for a product, it is typically clear how to provide a certain functionality, for a
process, it is much less known whether a specific process can provide the required
features at all.
Technique Selection Approaches. Biffl and Halling [4] provide a framework for sup-
porting Fagan inspections. The approach is very detailed and provides decision mod-
els based on a literature survey; however, it does not consider the anticipated future
and is limited to Fagan inspections. Schweikhard [5] describes a framework for sup-
porting the decision-making process in inspections. It provides a classification scheme
for context and variation factors and uses historic and empirical knowledge; however,
it also does not consider the anticipated future and is limited to products.
Vegas and Basili [6] provide a characterization scheme for supporting the selection
of testing techniques. They also provide a decision model and integrate existing
knowledge; however, they neglect the anticipated future as did the previous two ap-
proaches, and support projects only, but no products or processes. Madachy et al. [7]
developed a simulation model predicting the impact of quality strategies on defect
profiles, cost, and risk, using COCOMO II [8] for cost estimation, as well as inputs on
introduced defects. It considers products in a very detailed manner; however, it also
does not consider the anticipated future, and is designed for products only, neglecting
projects and processes.
In [9], Denger et al. analyze a number of approaches to customizing quality assur-
ance techniques for different parts of the software lifecycle. They provide decision
models for quality assurance techniques, but also do not consider the anticipated fu-
ture, and they neglect projects and processes. Rus and Collofello [10] investigate the
use of an expert system for making selection decisions for a reliability engineering
28 O. Armbrust
strategy. They also provide a decision model for achieving reliability, yet again ignore
the anticipated future, products, and processes. In addition to this, they focus on
reliability only. In [11], the authors describe a vision for comprehensive software en-
gineering decision support regarding techniques. They provide decision models for
individual projects, but do not support products or processes. In addition, they also
consider the next project, but do not look any further into the future.
To summarize, the technique selection approaches described support software en-
gineers by providing help for decision-making. Strongly simplified, they assume that
a certain quality factor is important (e.g., low defect density in the final product, or
reliability of the final product) and assist decision makers in selecting appropriate
techniques for achieving this goal. However, they typically investigate only either
products or projects, but not both. In general, they also neglect processes. They also
largely ignore the anticipated future.
Process-aware Approaches. Becker-Kornstaedt [12] describes an 8-step approach to
systematic descriptive process modeling. The approach defines the scope of the proc-
ess model, but considers the anticipated future use of the process model only implic-
itly. It does not describe how scoping should be performed.
Avison and Wood-Harper [13] investigated the problem of choosing the right de-
velopment approach for information systems already very early. In the year 1991,
they stated that the number of development methodologies is very large, yet there is
no single methodology that is optimal for all contexts. Therefore, for every single
context, a suitable methodology (or, as it would be called today, process) has to be
chosen. Since an organization cannot excel at every methodology, a reduced set must
be provided from which developers can choose. They propose a contingency
approach and present Multiview, a framework representing a structure to help devel-
opers choose procedures, techniques, and tools from a fixed portfolio. Multiview
characterizes techniques based on historical knowledge and provides decision models
for some techniques, but it does not consider the anticipated future of an organization
beyond the next project. It also does not support products.
Becker et al. [14] discuss the application of Quality Function Deployment (QFD)
[3] for strategically planning software process improvement (SPI) programs to sup-
port an organization’s business goals. Their main idea is to regard SPI as the organi-
zation’s product that is to be optimized in order to support the business goals. They
use the House-of-Quality matrices subset of QFD to operationalize this idea. The ap-
proach actively considers the anticipated future through the organization’s business
goals, yet it does not investigate products or projects, but focuses on business goals
and identified problems. The recommendations for the decision model remain on a
very high level of abstraction (CMMI process areas).
In summary, the product scoping approaches focus on scoping products, i.e.,
determining the features a number of products should have. They do not consider
processes. However, they typically consider the anticipated future explicitly. The
technique selection approaches mostly focus on selecting one out of very few specific
techniques. Fagan inspections are a very popular subject in this community. The focus
of these approaches is typically very narrow, and adapting them to support other tech-
niques, possibly from other categories (e.g., extending a Fagan variant selection ap-
proach to support quality assurance techniques in general) requires enormous effort.
Determining Organization-Specific Process Suitability 29
3 Industry Approaches
This section introduces some process management approaches that can be found in
today’s industrial practice.
Fitzgerald et al. [15] report on an approach to provide a Motorola plant in Cork,
Ireland with a software development process. Unfortunately, no information is given
on how the process was constructed, apart from the reference to industry standards. In
addition, continued management of the process is not detailed. A CMMI Level 5-
certified IT supplier from India (2008: <10,000 employees) that the author of this
paper has worked with pursues a very strict process management regime. The organi-
zation’s process design team collects comments, recommendations, and requests for
changes from all employees, processes them, and provides new releases of the com-
pany standard processes every three months based on the information collected. Every
release acknowledges about 100 requests from employees. While process manage-
ment is very strictly organized and responds systematically to employee feedback,
there is no strategic process planning or suitability analysis. All modifications to the
organization’s processes are based on past experience of the employees and thus ret-
rospective. Anticipated future developments are not used when the processes are
adapted. A very similar approach has been taken by Josef Witt GmbH, a medium-
sized (2,200 employees) mail order business in the clothing domain within the Otto
group (123 companies, 55,000 employees).
ESOC (European Space Operations Centre), the European Space Agency’s (ESA)
ground segment, provides a ready-to-use implementation of the mandatory ESA proc-
ess standards (ECSS series [16]) for its suppliers, called SETG (Tailoring of ECSS
Software Engineering Standards for Ground Segments in ESA [17]). The main driver
for adapting and modifying the SETG standards are changes within the superior
ECSS standards. ESOC normally does not modify the SETG standards otherwise, for
example to reflect changed project contexts. In particular, ESOC does not utilize their
knowledge on the anticipated future when changing the SETG standards.
Except for the Motorola report, industrial case studies and the author’s experience
do not suggest that the software industry performs systematic strategic process man-
agement. Many organizations, for example the Indian IT supplier, are driven by stan-
dards such as CMMI or SPICE, which are demanded by their customers. Others, such
as Josef Witt GmbH, react to problems or events that occurred in the past, but do not
consider the anticipated future in their actions. Organizations with highly safety-
critical applications such as ESA, finally, are mostly driven by other standards and not
so much by actual problems.
All the case studies have in common that there is no systematic analysis as to
whether and how much the application of the individual standards actually contributes
to achieving the respective organization’s business goals, and how such standards
must be adapted to achieve these goals better in the future. The Indian IT supplier
example shows that even organizations with high process maturity might not manage
their processes strategically, considering the anticipated future.
30 O. Armbrust
1 n
D p (a j ) = ∑ P ( pi , a j )
n i =1 (1)
1 n
D j (a j ) = ∑ J ( ji , a j )
n i =1
The organization-wide process demand D across all products p and projects j is de-
termined by unifying their respective demands (Eqn. 2). D thereafter contains, for
every attribute, the organization’s process demand with respect to this attribute. D,
like Dp and Dj, considers how often a specific capability (represented by the appropri-
ate attribute) is required relative to all other capabilities.
D = Dp ∪ D j (2)
While D reflects what an organization needs in terms of processes for its products and
projects, our next step is to determine how suitable its processes are with respect to
these demands. This suitability of the organization’s processes is determined for each
demand, i.e., each attribute from the product and project analysis, using expert estima-
tion, empirical knowledge, or a combination of both. The result is a value S(pi, aj) for
Determining Organization-Specific Process Suitability 31
the suitability of each process for each attribute. The sum of the values of all attrib-
utes per process indicates its suitability S(pi) for the organization (Eqn. 3).
S ( pi ) =∑ S ( pi , a j ) (3)
j
Please note that S(pi) so far only reflects the processes’ general suitability for the de-
mands stated by the attributes, but does not consider how much each attribute contrib-
utes to the organization’s success, and how often the respective capability is actually
required for the organization’s business. The former can be determined by prioritizing
the attributes, e.g., through pair-wise comparison [19]; the latter is considered through
the product and project analyses. Prioritizing the attributes orders them by importance.
This order can be projected on any scale, reflecting each attribute’s relative impor-
tance. A projection on 50%...100%, for example, would mean that the least important
attribute is considered half as important as the most important attribute.
We reflect the variance in attribute importance by adjusting the generic process
suitability through the introduction of an organizational factor. The organizational
factor O(aj) for attribute j is determined by multiplying the process demand D for a
specific attribute j with the relative importance I of this attribute for the organization’s
success, as determined by the attribute prioritization (Eqn. 4).
O ( a j ) = D( a j ) ⋅ I ( a j ) (4)
The organizational factor O(aj) is then applied to the process suitability S(pi, aj) for
process i and attribute j, resulting in an organization-specific process suitability index
So(pi, aj). This adjusts the generic suitability according to the organization’s business.
The result indicates the suitability of the analyzed processes with respect to the proc-
ess demands of products and projects and each analysis attribute’s contribution to the
organization’s success (Eqn. 5).
S o ( pi , a j ) = S ( pi , a j ) ⋅ O(a j ) (5)
Finally, the organization-specific process suitability So(pi) of an individual process i
can be determined by summing up the individual organization-specific process suit-
ability values of all attributes j for this process (Eqn. 6).
S o ( pi ) =∑ S o ( pi , a j ) (6)
j
The suitability index So(pi) describes how well an organization’s processes fit all of its
demands, i.e., how well they support the product and project characteristics that were
identified in total. Fig. 1 shows a graphical rendition of the suitability index of a (fic-
tional) company for its five requirements processes and four design processes. This
information can be used, for instance, to focus training: Teaching the example organi-
zation’s employees Delphi will benefit it more than, for example, Storyboards, because
the Delphi process is far more suitable (i.e., fulfills more and more often requested
demands) than the Storyboards process.
32 O. Armbrust
Cleanroom
Leonardo
Delphi
Brainstorming
Structured
Specification
OO Design
Storyboards
Use Cases
Design
Formal
Based on the suitability index So(pi), an organization may also adjust the scope of
its processes. We will introduce two possible scenarios here: (1) an input-constrained
scenario, and (2) an output-constrained scenario.
In the input-constrained scenario, an organization has a limited amount of re-
sources (typically effort or budget) available, and wants to put them to their best use.
Using the suitability index So(pi) as an ordering criterion, the processes can be ordered
according to their suitability for the organization. Assuming that the organization can
provide a budget for maintaining and evolving n processes, it can simply choose the
first n processes from the ordered list, which provide the best support for the entirety
of the organization’s products and projects. This means that the globally best proc-
esses are selected. For example, when our example company’s budget is sufficient for
two processes each for requirements and design, it would choose Delphi and Use
Cases for the former and OO Design and Cleanroom for the latter.
In the output-constrained scenario, an organization requires processes to achieve a
certain minimum suitability in order to be utilized and maintained. Again using the
suitability index So(pi), the organization can define thresholds for the requirements
and the design processes, based on experience from past projects. Processes with a
suitability index below these thresholds will be discarded; only those with equal or
higher suitability indices will be kept. The threshold that was set in our example com-
pany is depicted by the dashed line in Fig. 1. As can be seen, from the requirements
processes, only Delphi is accepted, and from the design processes, Cleanroom and
OO Design are accepted. The other processes are deemed unsuitable and should either
be discarded or improved in order to reach the threshold.
5 Validation
The SCOPE approach was validated by means of a controlled experiment and an indus-
trial case study. The validation was aimed at showing that SCOPE (a) allows for a
greater reduction in unnecessary process variability than ad-hoc selection (hypothesis
H1); (b) allows for selecting processes that cover a broader range of demands than
Determining Organization-Specific Process Suitability 33
ad-hoc selection (H2); (c) allows for reducing process management effort compared to
ad-hoc methods (H3); and (d) is fit for industrial application (H4).
The experiment with master students at the University of Kaiserslautern evaluated
hypotheses H1 through H4. During the experiment, the students were provided with
information describing an organization’s projects, its demands with respect to proc-
esses, and the processes of the organization itself. They were asked to perform typical
process management tasks, e.g., identify suitable and unsuitable processes and iden-
tify improvement opportunities.
The application of SCOPE at the Japan Aerospace Exploration Agency (JAXA)
tested H1, H3, and H4 in an industrial environment. This included the performance of
product and project analyses, attribute prioritization, and a process analysis, and fol-
lowed the output-constrained scenario, where the goal was to fully support all pro-
jects, i.e., to keep all identified processes that were used in the analyzed projects.
Group A Group B
Session 0 Familarization
Ad-hoc Ad-hoc
Session 1
Data set 1 Data set 2
SCOPE Training
Session 2
Data Set 3
SCOPE SCOPE
Session 3
Data Set 2 Data Set 1
Session 4 Questionnaire
34 O. Armbrust
The case study evaluated hypotheses H1, H3, and H4 in JAXA’s satellite develop-
ment segment and actually went beyond pure scope determination, also taking the
first steps towards a comprehensive software process line [18]. This resulted in sets of
common and variable entities, which were used to evaluate the hypotheses. During
the course of the case study, JAXA engineers performed product and project analyses,
attribute prioritization, and a process analysis, and followed the output-constrained
scenario, where the goal was to fully support all projects. For confidentiality reasons,
we cannot disclose the detailed process suitability results here, but we can sketch the
general conclusions that were drawn.
Determining Organization-Specific Process Suitability 35
The results of the scoping activities showed that all identified project types (na-
tional/international and scientific/engineering) share 86% of their activities and 77% of
their artifacts. This means that only 14% of the activities and 23% of the artifacts must
vary between the analyzed project types – the rest is unnecessary, yet real variation. So,
by using the results of the scoping activity, JAXA could reduce the potential variation
of their processes across activities and artifacts by an average of 82%: Instead of creat-
ing, establishing, and maintaining two completely independent satellite development
process standards, they could share all common entities – effectively reducing variation
for these entities to zero, which confirms hypothesis H1. By using the scoping results,
JAXA also needs to maintain the common elements of the process standards only once,
thus reducing management effort for each by half. Assuming that maintenance effort for
all activities and artifacts is identical, SCOPE thus enabled a reduction in process man-
agement effort by 41%, confirming hypothesis H3. Finally, the feedback collected from
JAXA engineers during the application of the SCOPE approach was positive. While the
language barrier turned out to be something of an obstacle, product and project as well
as process analyses could be performed in the course of the daily work of the JAXA
process engineers. The case study results therefore support our assumption that the re-
sults from the controlled experiment with respect to hypothesis H4 can be transferred to
industrial practice. Table 3 displays an overview of the results of the performed studies.
Controlled JAXA
Hypothesis
experiment case study
H1 Process variations (9)1 46% 9 82%
H2 Process gaps 9 150%
H3 Process management effort 9 83% 9 41%
H4 Acceptance 9 4.09 2 (9) 3
To summarize, the validation showed that (a) the application of the approach in a
controlled experiment led to a 46% reduction in unnecessary process variability com-
pared to ad-hoc approaches and allowed for an 82% reduction in an industrial case
study; (b) SCOPE users identified 150% more misalignments between processes and
demands in a controlled experiment than when working ad-hoc; (c) the application of
the SCOPE approach allowed for a reduction in process management effort of 83% in
a controlled experiment and of 41% in an industrial case study; and (d) the SCOPE
approach and results were accepted by the controlled experiment participants as well
as by the engineers in the industrial case study as a means of providing adequate sup-
port for process management.
be discussed and evaluated, something that is not possible for implicit knowledge.
Another advantage is that the approach makes an organization very flexible within its
scope. Setting up a new (or modified) process based on the process landscape can be
completed very quickly, as opposed to fully tailoring a standard. For products or pro-
jects outside the scope, this is obviously not the case. However, from our experience,
this kind of flexibility on a global scale (“we’re great at everything”) is an illusion
anyway. Therefore, SCOPE assists organizations in determining their scope and then
achieving process excellence for this scope.
Both product and project analyses encourage an organization’s process engineers
to think about the product and project future of the organization. This likely leads to
identifying information that would otherwise have been neglected. The same applies
to process analysis: A thorough analysis of the currently used processes’ capabilities
with respect to the actual needs of the organization is hardly ever done. The results of
this analysis can help to rationalize otherwise sometimes rather emotional discussions
regarding advantages and disadvantages of individual processes.
The two scenarios for using the suitability index can help an organization decide
about its process future. They reflect two typical industry scenarios, where either an
existing budget should be used optimally, or past experience is used as benchmark for
process evaluation. From our experience, assistance with these types of questions is
often sought, but typical model-based SPI approaches such as CMMI or SPICE do not
provide this.
So far, the scenarios support determining what could be called the “global” suit-
ability of an organization’s processes. While this helps to determine the “value” of
individual processes for the respective organization, there may be scenarios where a
process-individual evaluation might not yield the best possible result for an organiza-
tion. For example, within some organizations, one process may score high for one
half of the analysis attributes, while yielding only low scores for the other half. An-
other process may behave vice versa. In total, these two processes would reach a me-
diocre suitability index, “losing” against a third process that is slightly better for all
6
Score
0
A B C D So
Attribute
attributes – but not as good as any of the two is for some. Fig. 2 displays this situa-
tion. It shows the analysis scores for three processes for four attributes and the result-
ing suitability index So (red box to the right, assuming equal attribute importance). It
becomes apparent that process 3 achieves the highest value for So, qualifying it for
selection. However, a combination of process 1 and process 2 might prove to be more
beneficial for the organization if their advantages with respect to the four attributes
can be combined. We plan to investigate this interesting possibility in the future.
Acknowledgments
We would like to thank Ms. Sonnhild Namingha from Fraunhofer IESE for reviewing
the first version of this article. We also thank the anonymous reviewers for their valu-
able comments on the first version of this article.
References
1. Schmid, K.: Planning Software Reuse - A Disciplined Scoping Approach for Software
Product Lines. PhD Thesis, Department of Computer Science, University of Kaiserslau-
tern, Germany (2003)
2. Bayer, J., Kose, M., Ocampo, A.: Improving the Development of e-Business Systems by
Introducing Process-Based Software Product Lines. In: Münch, J., Vierimaa, M. (eds.)
PROFES 2006. LNCS, vol. 4034, pp. 348–361. Springer, Heidelberg (2006)
3. Cohen, L.: Quality Function Deployment: How to Make QFD Work for You. Addison-
Wesley Longman, Amsterdam (1995)
4. Biffl, S., Halling, M.: Managing Software Inspection Knowledge for Decision Support of
Inspection Planning. In: Aurum, A., Jeffery, R., Wohlin, C., Handzic, M. (eds.) Managing
Software Engineering Knowledge. Springer, Berlin (2003)
5. Schweikhard, T.: Identification of inspection-variation-factors for a decision-support-tool.
Diploma Thesis, Department of Computer Science, University of Kaiserslautern, Germany
(2006)
6. Vegas, S., Basili, V.R.: A Characterization Schema for Software Testing Techniques. Em-
pirical Software Engineering 10(4), 437–466 (2005)
7. Madachy, R., Boehm, B.: Assessing Quality Processes with ODC COQUALMO. In:
Wang, Q., Pfahl, D., Raffo, D.M. (eds.) ICSP 2008. LNCS, vol. 5007, pp. 198–209.
Springer, Heidelberg (2008)
8. Boehm, B.W., Harrowitz, E.: Software Cost Estimation with Cocomo II. Prentice Hall In-
ternational, Englewood Cliffs (2000)
9. Denger, C., Elberzhager, F.: A Comprehensive Framework for Customizing Quality As-
surance Techniques, IESE-Report No. 118.06/E, Fraunhofer Institute for Experimental
Software Engineering (ISCE), Kaiserslautern, Germany (2006)
10. Rus, I., Collofello, J.S.: A Decision Support System for Software Reliability Engineering
Strategy Selection. In: Proceedings of the 23rd Annual International Computer Software
and Applications Conference, Phoenix, AZ, USA, pp. 376–381 (1999)
11. Jedlitschka, A., Pfahl, D.: Towards Comprehensive Experience-based Decision Support.
In: Dingsøyr, T. (ed.) EuroSPI 2004. LNCS, vol. 3281, pp. 34–45. Springer, Heidelberg
(2004)
38 O. Armbrust
12. Becker, U., Hamann, D., Verlage, M.: Descriptive Modeling of Software Processes,
ISERN Report 97-10, Fraunhofer Institute for Experimental Software Engineering (IESE),
Kaiserslautern, Germany (1997)
13. Avison, D.E., Wood-Harper, A.T.: Information Systems Development Research: An Ex-
ploration of Ideas in Practice. The Computer Journal 34(2), 98–112 (1991)
14. Becker, A.L., Prikladnicki, R., Audy, J.L.N.: Strategic Alignment of Software Process Im-
provement Programs Using QFD. In: Proceedings of the ICSE 2008 Workshop on Busi-
ness Impact of Process Improvements (BIPI 2008), Leipzig, Germany, May 13 (2008)
15. Fitzgerald, B., Russo, N.L., O’Kane, T.: Software Development Method Tailoring at Mo-
torola. Communications of the ACM 46(4), 65–70 (2003)
16. Collaboration website of the European Cooperation for Space Standardization,
http://www.ecss.nl/ (last visited April 25, 2009)
17. BSSC Guides and Reports,
http://www.esa.int/TEC/Software_engineering_and_
standardisation/ (last visited April 24, 2009)
18. Armbrust, O., Katahira, M., Miyamoto, Y., Münch, J., Nakao, H., Ocampo, A.: Scoping
Software Process Lines. Software Process: Improvement and Practice 14(3), 181–197
(2008)
19. David, H.A.: The Method of Paired Comparisons. Lubrecht & Cramer, Limited (1988)
20. Sheskin, D.J.: Handbook of Parametric and Nonparametric Statistical Procedures. CRC
Press, Boca Raton (1997)
21. Venkatesh, V., Morris, M.G., Davis, G.B., Davis, F.D.: User Acceptance of Information
Technology: Toward a Unified View. MIS Quarterly 27(3), 425–478 (2003)
On Scoping Stakeholders and Artifacts
in Software Process
1 Introduction
Process modeling and simulation allows individuals (e.g. project manager) or
organizations to verify and validate the correctness and to monitor or control the
operations of software processes as a generative software system [1]. Identifying
the high priority concerns of process stakeholders and process artifacts they
depend on are two critical success factors for the selection, integration and design
of effective process modeling and simulation techniques. Overlooking them often
leads to developing or selecting sub-optimal process modeling and simulation
techniques. In software engineering, stakeholders are defined as individuals or
organizations who will be affected by the system and who have a direct or indirect
influence on the system requirements [2] [3], Software artifacts, as a piece of
information that is produced, modified, or used by a process [4], are associated
with these stakeholder roles. Neither definition explicitly explains or identifies
stakeholders or artifacts associated with software process modeling itself.
Modeling Systems and Software engineering Processes (MSSP) Workshop[5]
held at University of Southern California in 2008 identified an initial set of
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 39–51, 2010.
c Springer-Verlag Berlin Heidelberg 2010
40 X. Bai, L. Huang, and H. Zhang
process modeling stakeholder roles and their top-level modeling goals. Their
dependencies on existing process modeling & simulation techniques were also
discussed.
While trying to integrate different process modeling techniques based on vari-
ous process modeling stakeholders’ perpectives to address software process trust-
worthiness [6], we found that the understanding of process stakeholder roles were
inconsistent in existing software process related literatures. One typical issue is
that process modeling stakeholders and process enactment stakeholders are not
clearly scoped or distinguished. For instance, Requirement Engineering (RE) pa-
pers often refer project stakeholders involved in RE activities as “requirement
process stakeholders” [7]. However, they actually mean project stakeholders in-
volved in RE activities instead of process modeling stakeholders involved in or
dependent on process modeling or simulation. Similarly, a majority of artifact-
related literature study software product artifacts such as requirement specifi-
cations, design documents, source codes, etc. Process artifacts produced from
or used by process modeling activities, (e.g. process guidelines, regulations and
management plans) are seldom investigated or mentioned.
In response to stakeholder-oriented software process modeling research [5],
there is an emergent need to:
– Scope, classify and define stakeholders and artifacts in software process;
– Study and analyze the understanding of proposed process modeling stake-
holder roles [5] within software process communities.
To achieve these objectives, we have followed a two step approach. We started
with a Systematic Literature Review (SLR) performed on stakeholder and arti-
fact related studies in software process related research. Then we harvested the
preliminary results and initially proposed process modeling stakeholder roles in
[5] with their associated process artifacts and undertook a questionnaire based
web survey to investigate the overall agreement of these two entities in software
process research and practice communities.
Processes are difficult to identify because their boundaries are often not defined
[8]. In software engineering, process (software process) consists of a set of log-
ically ordered tasks or activities in order to deliver or to maintain a software
product [9]. Fenton and Pfleeger distinguish three classes of elements in software
development: process, product, and resource entities [10].
– Process: collection of software-related activities.
– Product: any artifacts or deliverables that result from a process activity.
– Resource: entities required by a process activity, e.g. tools, roles, and actors.
These three classes also correspond to the entities abstracted in the three
dimensions of software development identified in the recent TRISO-Model [11],
i.e. actors (SE Human), activities (SE Process), and artifacts (SE Technology).
On Scoping Stakeholders and Artifacts in Software Process 41
R eflects
R eflects
3 Research Methods
3.1 Research Questions
Our major objective is to investigate ‘how people understand stakeholders and
artifacts based on existing software process related research’, which can be spec-
ified as following research questions:
RQ1. What is the scope where the stakeholders are identified/studied in soft-
ware process related research?
RQ2. What are the stakeholder roles in software process?
RQ3. What is the scope where the artifacts are produced or used in software
process related research?
RQ4. What are software process artifacts and what is the relationship between
these artifacts and software process?
RQ5. How are the proposed process modeling stakeholder roles [5] agreed in
software process research and practice communities?
RQ6. How are the proposed process modeling artifacts agreed in software pro-
cess research and practice communities?
Systematic Literature Review was chosen as our main research method in
this study. During the pilot of SLR, however, two questions (RQ5-6) cannot be
effectively addressed by the studies. Therefore, questionnaire-based online survey
was employed as a complementary method to seek answers to them.
Study Selection. The literature inclusion and exclusion criteria are defined
in the SLR protocol [16]. The included primary studies belong to one of the
following categories:
• Studies focusing on stakeholder interactions in software engineering;
• Studies which propose and/or apply stakeholder-oriented approaches in soft-
ware engineering;
• Studies focusing on management of software artifacts;
• Studies which propose new software artifacts in software engineering.
But the studies in following categories are excluded during review:
• Business process related studies;
• Studies focusing on tool implementation;
• Tutorials, editorials, posters, position papers, keynotes, abstract, short papers.
For any duplicate or continued publications, only the latest or the most com-
prehensive versions were included. We adopt a two-step approach to the study
selection process (described in [16]).
Due to the page limit, the quality assessment and data extraction of this SLR
are not described in this paper, but is accessible in [16].
4 Results
Systematic Literature Review. We selected totally 32 software engineering
literatures for the stakeholder study after combining both manual and automatic
search results. Among them, 24 pieces are major studies on stakeholders, while
the other 8 pieces are minor studies relevant to but not focusing on stakeholders.
These studies can be also categorized by sources of their case studies (i.e., from
industrial or academic/open source projects). Table 1 shows the statistics.
Totally 25 studies relevant to artifacts were included in our SLR. Table 2
shows 4 different categorizations of these studies. Among the included studies,
19 pieces are major studies on artifacts, while the other 6 are minor studies
relevant to but not focusing on artifacts. Another categorization shows that 3 of
the included studies are enhanced with industrial case studies, and the other 21
either use academic/open source project case studies or lack of case studies. In
addition, 40% of the studies discuss specific artifacts, while 60% of them refer
to general artifacts used in software development activities.
Web Survey. After sending out 144 invitations among software process
research communities (ICSP, ProSIM etc.), we have got 38 responses. The re-
sponse rate is 26.4%. Fig.3 shows the statistical distribution of all survey tak-
ers’ professional background with respect to the software process. Most of their
professions are related to Software Process Modeling (92.11%), Software Pro-
cess Metrics (71.05%) and Software Process Improvement (71.05%) with overlap
across various areas. Such distribution indicates that our survey results highly
represent majority opinions from researchers and/or practitioners in software
process modeling communities.
(a) (b)
Fig. 4. (a) Process Modeling Stakeholder Roles Played by Survey Participants, and
(b) Agreements on Proposed Process Modeling Stakeholder Roles
(a) (b)
Fig. 5. (a) Software Process Activities Producing or Using artifacts, and (b) Specific
Artifacts Discussed in Included Studies
6 Discussion
Process Stakeholders. Based on our observations of the available studies, the
stakeholder concepts in software engineering are varying between their appli-
cation context and different research perspectives. Process stakeholders have a
very broad coverage because: 1) the process modeling stakeholders, who concern
more in selecting the appropriate process modeling/simulation techniques, are
involved in or affected by the course of activities relating the process model;
2) the process enactment stakeholders are involved in executing the activities
defined by process models, which generate the software product. They can be
further instantiated as more specific project/organization stakeholders depend-
ing on their application scenarios.
Acknowledgements
He Zhang’s research is supported by NICTA which is funded by the Australian
Government as represented by the Department of Broadband, Communications
and the Digital Economy and the Australian Research Council through the ICT
Centre of Excellence program, and also supported, in part, by Science Foun-
dation Ireland grant 03/CE2/I303 1 to Lero - the Irish Software Engineering
Research Centre (www.lero.ie).
References
1. Osterweil, L.: Software processes are software too. In: ICSE 1987: Proceedings of
the 9th international conference on Software Engineering, pp. 2–13. IEEE Com-
puter Society Press, Los Alamitos (1987)
2. Kotonya, G., Sommerville, I.: Requirements engineering. Wiley, Chichester (1998)
3. Freeman, R.: Strategic management: A stakeholder approach. Advances in strategic
management 1, 31–60 (1983)
4. Thiel, S.: On the definition of a framework for an architecting process support-
ing product family development. Lecture notes in computer science, pp. 125–142.
Springer, Heidelberg (2002)
5. Madachy, R., Koolmanojwong, S., Osterweil, L., Huang, L., Phongpaibul, M.: Pre-
sentations on the workshop of modeling systems and software engineering processes
(2008), http://csse.usc.edu/events/2008/ARR/pages/program.html#mon
6. Bai, X., Huang, L., Zhang, H., Koolmanojwong, S.: Hybrid Modeling and Sim-
ulation for Trustworthy Software Process Management: A Stakeholder-Oriented
Approach. SPIP submission (2009)
On Scoping Stakeholders and Artifacts in Software Process 51
7. Aranda, G., Vizcaino, A., Cechich, A., Piattini, M.: How to choose groupware tools
considering stakeholders’ preferences during requirements elicitation? In: Haake,
J.M., Ochoa, S.F., Cechich, A. (eds.) CRIWG 2007. LNCS, vol. 4715, p. 319.
Springer, Heidelberg (2007)
8. Kiraka, R., Manning, K.: Managing organisations through a process-based perspec-
tive: its challenges and benefits. Business Change and Re-engineering 12, 288–298
(2005)
9. Curtis, B., Kellner, M.I., Over, J.: Process modeling. Commun. ACM 35, 75–90
(1992)
10. Fenton, N., Pfleeger, S.: Software metrics: a rigorous and practical approach. PWS
Publishing Co., Boston (1997)
11. Li, M.: Triso-model: A new approach to integrated software process assessment and
improvement: Research sections. Software Process: Improvement and Practice 12,
387–398 (2007)
12. Cohn, M.L., Sim, S.E., Lee, C.P.: What counts as software process? negotiating the
boundary of software work through artifacts and conversation. Comput. Supported
Coop. Work 18, 401–443 (2009)
13. Bai, X., Huang, L.: A stakeholder perspective in evaluating process modeling lan-
guages and hybrid process simulation (2008) Technocal Report
14. Biffl, S., Aurum, A., Boehm, B., Erdogmus, H., Grnbacher, P.: Value-Based Soft-
ware Engineering. Springer, Heidelberg (2005)
15. Kitchenham, B., Charters, S.: Guidelines for performing systematic literature re-
views in software engineering. Version 2, 2007–01 (2007)
16. Bai, X., Huang, L., Zhang, H.: Software Process Stakeholders and Artifacts: A
Systematic Review. Technical Report (2009)
17. Cotterell, M., Hughes, B.: Software project management (1995)
Critical Success Factors for Rapid, Innovative Solutions
Jo Ann Lane1, Barry Boehm1, Mark Bolas1, Azad Madni1, and Richard Turner2
1
University of Southern California
Los Angeles, California, USA
{jolane,boehm,azad.madni}@usc.edu
[email protected]
2
Stevens Institute of Technology
Hoboken, New Jersey, USA
[email protected]
1 Introduction
Most programs start up in situations where there is a high premium on early delivery,
or where they are already perceived as “behind schedule”. These programs are
quickly staffed, then jump into requirements analysis and design, continually trying to
“catch up”. This leaves little time to explore options and look for innovative solu-
tions. As a result, many organizations tend to minimize risk and protect profits by
providing “yesterday’s solutions” that use known technologies and approaches.
However, there can be limits to how far known technologies and approaches can scale
and meet the demand for new capabilities and associated attributes such as perform-
ance and security.
Many of today’s problems are in search of rapidly-developed, innovative solutions.
So, the question is, what do projects need to do to encourage the rapid development of
innovative solutions? Over the past few years, efforts have been initiated to explore
critical success factors for rapid development [1, 2, 3, 4], or for innovation [5, 6].
However, these generally did not explore how these could be achieved together. We
have done a mix of interviews and visits with a set of organizations known for provid-
ing innovative solutions while reducing development time, in order to determine these
critical success factors and how they were employed. The scope of this effort in-
cluded technical, managerial, people, and cultural aspects of the innovative environ-
ment. This paper discusses the details of these explorations.
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 52–61, 2010.
© Springer-Verlag Berlin Heidelberg 2010
Critical Success Factors for Rapid, Innovative Solutions 53
Section 2 provides background information on recent research that has been done
in the area of innovation. Section 3 describes the approach used for the identification
and analysis of critical success factors for innovation. Section 4 presents the findings
that resulted from the analysis of the data collected from interviews and site visits.
Section 5 summarizes the analysis into a set of conclusions.
2 Background
Much has been written about innovation in the commercial product development
environment. For example, Brown and Eisenhardt [7] talk about competing on the
edge by using structured chaos as a strategy. However, attempts to make software
and system development processes repeatable and manageable are based upon
concepts such as Max Weber’s bureaucratic organizational concepts and Fredrick
Taylor’s scientific management concepts which focus on compartmentalization and
division of labor [8]. Kreitman [8] refers to this as the Weber/Taylor bureaucracy and
its influence can be seen in software and system engineering process guidance such as
the Software Engineering Institute’s Capability Maturity Model [9] and the Interna-
tional Organization for Standardization (ISO) and Electronic Industries Alliance
(EIA) systems engineering standards [10, 11]. Those that are focusing on innovation
and rapid development of software-intensive systems are writing about working on
the edge of chaos [12, 13, 14] (much like Brown and Eisenhardt [7]) and the associ-
ated need for flexible, adaptable, agile development processes to deal with rapid
change and to facilitate innovative solutions in ultra-large solution spaces [15, 8]. In
addition, teams must be able to experiment and have self-adapting processes that
don’t require process committee approvals and documentation [12, 8].
Chaos theory explains and helps identify underlying order in apparently random
data or events. Behaviors thought to be random may have natural boundaries or pat-
terns. Once these boundaries or patterns are determined, the order becomes apparent.
This order is sometimes referred to as “emergent order” [16, 17, 12] and can lead to
focused innovative solutions. Key principles in this environment include:
Faste [19] has shown that problem solving across a group of people exhibits a Gaus-
sian distribution. When a group of people are asked to individually come up with a
solution, most will produce an OK solution (within the norm), a few will produce
“really good” solutions, and a few will produce “really bad” solutions. It is also inter-
esting that for every solution that someone might rate as “really good”, someone else
may perceive it as “really bad”. Therefore, it can take some additional work (explora-
tion) to determine if something is “really good” or “really bad”.
54 J.A. Lane et al.
This need to explore before deciding if something is “really good” or “really bad”
suggests the use of responsible, leveraged “play” in order to evaluate the feasibility of
candidate solutions. It is also well-known that people can’t “play” when they are
stressed which implies that innovative ideas must be identified before the team be-
comes stressed over a solution.
To better understand and further elaborate on these concepts, we contacted organi-
zations that have been successful in developing and maturing processes to support the
development of innovative technologies and systems that can be rapidly fielded when
needed.
Representatives from each of these organizations were contacted and in several cases,
visits scheduled to their innovation labs or centers. Questions in several areas were
used to guide discussions, but no rigorous attempt was made to obtain a written re-
sponse from each organization. However, several of the organizations did provide
written responses or elaborated on notes captured by the visiting researchers.
The approach used to guide the interviews started with a set of questions shown in
Figure 1. These questions were used to define the business domain for innovative
projects, how projects are scoped, and the types of products that are developed. Next,
the types of processes, methods, and tools employed in the innovation labs or centers
were explored as well as the characteristics of typical engineer and engineering teams
working in this environment and the characteristics of the associated workplace.
Lastly discussions were held to elicit the organization’s perspective on critical success
factors for rapid innovation, and to validate draft summaries of the results. The fol-
lowing sections describe in more detail the questions used to guide discussions in
each category.
Critical Success Factors for Rapid, Innovative Solutions 55
Category Question
1. What is your “home-ground” for projects?
Scope of
Project
30. How controlled is the workspace with respect to clients, potential clients,
or other visitors that may participate in collaboration?
31. What are critical success factors with respect to people, business practices,
intellectual property, and facilities?
Key Success
32. What are typical metrics you use to assess the success of your projects?
Factors
33. What aspects of your organization are critical to the success of your
projects?
34. How much of success depends upon innovation? If a significant part,
what is done to encourage innovation?
56 J.A. Lane et al.
The questions in this category captured information about the business domains or
“home-ground” for the organization’s innovation-related projects, what triggers a new
project, how projects are initially defined/scoped, and how much outside participation
there is by actual or potential clients. The goal of these questions was to discern how
new innovation projects are selected and how tightly coupled they are to business
values and goals.
3.2 Product
These questions probed the types of products that are produced in the innovation
environment. They ask about the goals of the projects, whether they are more ori-
ented towards a commercial product, technology to support future system develop-
ment work, or proof of concepts for existing projects. The questions also probed the
role of modeling, simulation, and prototyping; if they are employed, what percentage
of the system is typically modeled, simulated, or prototyped; and whether models and
prototypes can be used “as is” for an actual system or whether they need to be rede-
veloped for actual use or integration. The final set of questions asks about the mix of
projects in the innovation portfolio.
3.3 Processes
The process questions investigate the business model for innovation projects, the
formality of the innovation environment (and if there are formal processes, how are
process improvement initiatives managed), how structured the work environment is,
as well as funding sources for the projects (internal investment versus outside/sponsor
funding). In addition, questions probed typical project duration, how these durations
are set, and how classified or proprietary information is handled. Lastly, questions
were asked about reuse (components, patterns, etc.) across projects and the role this
typically has in innovation projects.
3.4 Methods
Methods can span development methods such as agile to quality methods such as
inspections and testing. They can also include “go/no-go” business reviews that
monitor results to date to determine where future investments should be made. The
questions in this category probed the types of methods used, the percentage of the
total process covered by specific methods, and the percentage of projects that use the
methods.
3.5 Tools
Typically there are both management and development tools that support project
activities. These questions probed the types of development and management tools
typically used and whether they were primarily commercial or internally developed
tools. Particular attention was paid to any types of prototyping or simulation frame-
works that were used.
Critical Success Factors for Rapid, Innovative Solutions 57
3.6 People
As pointed out by [12] and others, one can have the greatest development environ-
ment, processes, and tools, but it is having the right people that make the most differ-
ence. These questions probed both team and individual characteristics. The questions
included team size, whether or not individuals worked full time or part time on inno-
vation activities, and if not full time, how many projects each person worked concur-
rently. Questions were also asked to understand engineer backgrounds with respect to
skill types, experience, and personal work processes.
3.7 Workspace
In addition to having the right people, the workspace must be “structured” to support
both collaboration and experimentation. In addition, outsiders (clients, potential cli-
ents, venture capitalists, etc.) may want to participate in the process or view in-
progress activities. This category of questions probed these areas and focused on how
the workspaces are organized to support collaboration and innovation as well as the
extent of control or access with respect to clients, potential clients, and other visitors.
This can be a particular challenge with respect to outsiders since organizations par-
ticipating in joint ventures may find themselves competitors on related projects.
The final set of questions asked the organization representatives to comment on their
perceived critical success factors. These questions covered critical success factors
with respect to people, business practices, intellectual property, and facilities. They
also probed metrics used to assess success of projects and the aspects of the larger
organization that are critical to the success of projects. Finally, the organization was
asked to indicate how much of their success depends upon innovation and if a signifi-
cant part, what is done to encourage innovation.
4 Findings
To encourage candid responses and to protect the proprietary nature of some of the
organizations’ responses, actual responses captured are not provided. Rather, this
section describes critical success factors that were common at several sites, if not all
sites, while providing some context in distinguishing classes of solutions.
All of the organizations that provided inputs indicated the importance of early concept
exploration and feasibility assessment that often required considerable modeling and
prototyping. The level of modeling and prototyping varied, typically based upon per-
ceived risks of the technical approach or the technologies to be integrated into the
solution. In order to encourage innovation, organizations think that it is important to
establish a supportive culture and environment. The length of the “rapid innovation”
58 J.A. Lane et al.
period also reflected different balances of the risks of delay versus the risks of system
malfunction.
One organization’s objectives were to reduce the concept exploration time from 1-
2 years to 60 days. This time was used to explore several approaches for major sys-
tems with few options for fixes once they were deployed.
The commercial company’s strategy was to produce a quick prototype on Day 2,
and iterate it into an initial operational capability in 16-24 weeks. This company de-
veloped supply-chain domain applications that had lower risks of malfunctioning, as
they could be rapidly adjusted in a pilot operation.
the problem. Spending time up front to investigate the root cause of a customer’s
problem can result in significant payoffs. Sometimes the best solutions focus on
eliminating the root cause of the problem rather than developing something to deal
with the problem once it occurs.
Tools are required to succeed in this environment. However, the tools must be the
right (value-adding) tools and the users must be experienced with those tools. The
wrong tool or the right tool with no team expertise is not of value. For those organiza-
tions that periodically tap their key corporate resources (i.e. super-stars) to work on
special innovative, rapid response projects or to conduct feasibility assessments of
concept designs, it is important that the project work environment include the tools
that those team members use in their day-to-day work. Another key theme is that tools
don’t need to be the best or the most sophisticated. Sometimes it is the simple, stable
tools that work best.
Most agree that you can have the best tools and the best processes, but without the
best people, success is difficult. To achieve the desired results in an innovative, rapid
development environment, organizations need to enable the best to achieve the de-
sired task.
60 J.A. Lane et al.
Whether or not the work is classified or proprietary, the innovative, rapid develop-
ment teams tend to work in their own large, relatively unstructured open space (some-
times with cubicles) to encourage collaboration and experimentation. When the same
key people are being used frequently for intense rapid-response projects, it is impor-
tant for the organization to provide additional resources and rewards that will help
these people with their outside lives (e.g., family, external commitments). If people on
the teams are overly stressed, innovation and creativity will suffer. They could easily
end up reverting to a 9-to-5 work mode that is counter-productive to the innovation
and rapid-response goals.
5 Conclusions
This research work was motivated by the fact that certain organizations are able to
innovate consistently. In this paper, we present our investigation of such organizations
and our key findings. Successful innovative organizations share certain characteris-
tics. They are all driven by business value and they are all prepared to make the
needed investments. They exploit opportunities by taking calculated risks. They fol-
low concurrent engineering practices to accelerate cycle times. They focus on their
Critical Success Factors for Rapid, Innovative Solutions 61
core business areas and continually look for solution patterns that they can reuse and
can reuse in different and novel ways. They have proactive management that believes
in small agile teams. As a result, they provide a culture and environment that supports
innovation and arrange time for team members to investigate, play with, and learn
from candidate solutions. These findings provide considerable guidance to organiza-
tions that are striving to rapidly develop innovative solutions and will continue to
grow and evolve as more organizations employ these approaches.
References
1. Arthur, L.: Rapid Evolutionary Development. Wiley, Chichester (1991)
2. McConnell, S.: Rapid Development. Microsoft Press, Redmond (1996)
3. Beck, K.: Extreme Programming Explained. Addison-Wesley, Reading (1999)
4. Boehm, B., Turner, R.: Balancing Agility and Discipline. Addison-Wesley, Reading
(2004)
5. Christensen, C.: The Innovator’s Dilemma. Harper Collins (2000)
6. Chesbrough, H.: Open Innovation. Harvard Business School Press, Boston (2003)
7. Brown, S., Eisenhardt, K.: Competing on the Edge: Strategy as Structured Chaos. Harvard
Business School Press, Boston (1998)
8. Kreitman, K.: From the magic gig to reliable organizations: A new paradigm for the control of
complex systems. Paper presented at the symposium on complex systems engineering (1996),
http://cs.calstatela.edu/wiki/index.php/
Symposium_on_Complex_Systems_Engineering (accessed on 1/11/2007)
9. SEI. Capability maturity model integration (CMMI), CMU/SEI-2002-TR-001 (2001)
10. ISO/IEC. ISO/IEC 15288:2002(E) Systems engineering - system life cycle processes
(2002)
11. Electronic Industries Alliance, EIA Standard 632: Processes for Engineering a System
(January 1999)
12. Highsmith, J.: Adaptive software development: A collaborative approach to managing
complex systems. Dorset House Publishing, New York (2000)
13. Markus, M., Majchrzak, A., Gasser, L.: A design theory for systems that support emergent
knowledge processes. MIS Quarterly 26(3) (2002)
14. Sheard, S.A.: Practical Applications of Complexity Theory for Systems Engineers. Sys-
tems and Software Consortium, Inc. (2005)
15. Boehm, B., Lane, J.: 21st century processes for acquiring 21st century software-intensive sys-
tems of systems. CrossTalk - The Journal of Defense Software Engineering 19(5), 4–9 (2006)
16. Kauffman, S.: At Home in the Universe: The Search for the Laws of Self-Organization and
Complexity. Oxford University Press, Oxford (1995)
17. Wheatley, M.J.: Leadership and the New Science: Learning about Organization from an
Orderly Universe. Berrett-Koehler Publishers (1992)
18. Madni, A.M., Brenner, M.A., Costea, I., MacGregor, D., Meshkinpour, F.: Option Genera-
tion: Problems, Principles, and Computer-Based Aiding. In: Proceedings of 1985 IEEE In-
ternational Conference on Systems, Man, and Cybernetics, Tucson, Arizona, November
1985, pp. 757–760 (1985)
19. Faste, R.: A Visual Essay on Invention and Innovation. Design Management Review 6(2)
(1995)
20. Curtis, B., Krasner, H., Iscoe, N.: A Field Study of the Software Design Process for Large
Systems. ACM Communications 31(11), 1268–1287 (1988)
Evidence-Based Software Processes
Abstract. Many software projects fail because they commit to a set of plans
and specifications with little evidence that if these are used on the project, they
will lead to a feasible system being developed within the project’s budget and
schedule. An effective way to avoid this is to make the evidence of feasibility a
first-class developer deliverable that is reviewed by independent experts and
key decision milestones: shortfalls in evidence are risks to be considered in go-
ing forward. This further implies that the developer will create and follow
processes for evidence development. This paper provides processes for devel-
oping and reviewing feasibility evidence, and for using risk to determine how to
proceed at major milestones. It also provides quantitative result on ”how much
investment in evidence is enough,” as a function of the project’s size, criticality,
and volatility.
1 Introduction
We have found through experience, case study analysis, and cost-benefit analysis that
up-front investments in providing evidence that the specifications and plans for a
software-intensive system satisfies a set of desired properties produce positive payoffs
later. A good way to capture this evidence is in a Feasibility Evidence Description
(FED). The FED’s evidence, when validated by independent experts, can also be
used as a basis for assuring system stakeholders that it is reasonable for the project to
proceed into development.
Providing such validated evidence is generally considered to be a good practice,
but it generally fails to be done well. This is because of a lack of proof criteria, proof-
generation procedures; measures for monitoring proof generation; methods, standards,
and contractual provisions that make proof generation optional; and an appreciation of
the consequences of proceeding into development with unsubstantiated specifications
and plans. The main contributions of this paper are to provide experience-based ap-
proaches for dealing with each of these concerns, and to show the consequences of
their use via case studies and parametric analysis.
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 62–73, 2010.
© Springer-Verlag Berlin Heidelberg 2010
Evidence-Based Software Processes 63
Current software design and development methods focus strongly on the inputs and
outputs, preconditions and post-conditions that a software function, object, or service
operates by as a product. They generally lack adequate capabilities to support analy-
ses about how well the elements perform, how expensive they will be to develop, or
how compatible are their underlying assumptions. In principle, they support reason-
ing about off-nominal performance, but in practice their descriptions generally focus
on sunny-day scenarios. As a result, many software project reviews tend to focus on
exhaustive presentations of PowerPoint charts and UML diagrams. They provide
little evidence that the system they describe could handle rainy-day scenarios; per-
form adequately on throughput, response time, safety, security, usability, or other
desired quality attributes across a range of representative mission scenarios; be
buildable within the available budgets and schedules in the plan; or to generate posi-
tive returns on investment for the stakeholders.
Most current versions of model-driven development, for example, strongly focus
on expressing product capabilities and relationships, and on reasoning about their
combined incompatibilities and incompleteness. However, analyses of failed projects
such as the one shown in figure 1 of the Bank of America (BofA) Master Net project
[7, 8] find that incompatibilities among product models and other stakeholder value
models (process models, property models, success models) are at least as frequent and
important as product-product (PD-PD) model clashes.
For example, the MasterNet users specified 3.5 million source lines of code (3.5
MSLOC) worth of wish-list features that were put into the project’s requirements
specification. Even at an extremely optimistic development cost of $30/SLOC for a
project of this size, this would cost $105 million. Thus, the users’ product model was
in serious conflict with the acquirers’ property model of a $22 million budget. Also,
many of the wish-list items had no mission effectiveness rationale, conflicting with
the acquirers’ success model.
Faced with this dilemma, the acquirers searched for a supplier who could reuse a
related solution to provide useful features at a low cost. They found one in Premier
Systems, who had built similar applications for small banks. However, their product
model only worked on Prime computers, which conflicted with the BofA users’ and
maintainers’ product-model value propositions of applications compatibility and ease
of maintenance, given that BofA was an IBM mainframe operation. Also, the Prime
host could not scale up to BofA’s throughput, response time, and reliability needs,
causing a property-product evidence shortfall that should have been addressed earlier.
64 B. Boehm and J.A. Lane
Overall, the implications here, both with the BofA model clashes in red /gray in
figure 1 and with further model clashes in black from analyses of other failed projects,
is that there are many potential value-proposition conflicts just among the four main
stakeholders in a project, and that product-product evidence shortfalls are only a mod-
erate fraction of the total set of issues of concern.
Fig. 1. Stakeholder Value Model Clashes. The red or gray lines show model clashes from the
failed MasterNet system.
model clashes of the product-product form. But in general, it will still have numerous
other model clashes that are unidentified and will cause extensive project rework,
overruns, and incomplete deliveries.
Similarly, most outsourcing contracts focus on product-oriented deliverables and
reviews. These reinforce paths toward project disaster, as in the quote from a recent
large-project manager, “I’d like to have some of my systems engineers address those
software quality-factor risks, but my contract deliverables and award fees are based
on having all of the system’s functions defined by the next review.” Similar overfocus
on product definition is found in project earned-value management systems for track-
ing project progress and data item descriptions (DIDs) for deliverables. Most contract
DIDs cover function, interface, and infrastructure considerations, but place demon-
stration of their feasibility in optional appendices where, as with the project manager
above, they are the first to go when time and effort are running out.
Fig. 3. Architecture Evidence Shortfall Penalties and Resulting Architecting Investment Sweet
Spots
The analysis indicated that the amount of rework was an exponential function of
project size. A small (10 thousand equivalent source lines of code, or KSLOC) pro-
ject could fairly easily adapt its architecture to rapid change via refactoring or its
equivalent, with a rework penalty of 18% between minimal and extremely thorough
architecture and risk resolution.
However, a very large (10,000 KSLOC) project would incur a corresponding rework
penalty of 91%, covering such effort sources as integration rework due to undiscovered
large-component interface incompatibilities and critical performance shortfalls.
The effects of rapid change (volatility) and high dependability (criticality) on the
architecture evidence shortfall penalties and resulting architecting investment sweet
spots are shown in the right hand graph. Here, the solid black lines represent the
average-case cost of rework, architecting, and total cost for a 100-KSLOC project as
Evidence-Based Software Processes 67
shown at the left. The dotted red lines show the effect on the cost of architecting and
total cost if rapid change adds 50% to the cost of architecture and risk resolution.
Quantitatively, this moves the sweet spot from roughly 20% to 10% of effective archi-
tecture investment (but actually 15% due to the 50% cost penalty). Thus, high in-
vestments in architecture, feasibility analysis, and other documentation do not have a
positive return on investment for very high-volatility projects due to the high costs of
documentation rework for rapid-change adaptation.
The dashed blue lines at the right represent a conservative analysis of the external
cost effects of system failure due to unidentified architecting shortfalls. It assumes
that the costs of architecting shortfalls are not only added development project re-
work, but also losses to the organization’s operational effectiveness and productivity.
These are conservatively assumed to add 50% to the project-rework cost of architec-
ture shortfalls to the organization. In most cases for high-assurance systems, the
added cost would be considerably higher.
Quantitatively, this moves the sweet spot from roughly 20% to over 30% as the
most cost-effective investment in architecting and development of feasibility evidence
for a 100-KSLOC project. It is good to note that the “sweet spots” are actually rela-
tively flat “sweet regions” extending 5-10% to the left and right of the “sweet spots.”
However, moving to the edges of a sweet region increases the risk of significant
losses if some project assumptions turn out to be optimistic.
The bottom line for Figure 3 is that the greater the project’s size, criticality, and
stability are, the greater is the need for validated architecture feasibility evidence (i.e.,
proof-carrying specifications and plans). However, for very small, low-criticality
projects with high volatility, the proof efforts would make little difference and would
need to be continuously redone, producing a negative return on investment (the same
could be said for proof-carrying code). In such cases, agile methods such as Scrum
[15] and eXtreme Programming [2] will be more effective. Overall, evidence-based
specifications and plans will not guarantee a successful project, but in general will
eliminate many of the software delivery overruns and shortfalls experienced on cur-
rent software projects.
to reduce risks early. Feasibility evidence costs can be minimized by focusing on proto-
types that can become part of the system or early working versions of key parts of the
system. Spending time and money on early feasibility evidence should be viewed as
buying information to reduce both risk and the cost of potential rework (assuming that
the problem can be resolved later in the project) or to avoid canceling the project alto-
gether after expending considerable resources. Likewise, if risk exposure is low, then
comprehensive, detailed feasibility evidence is of less value.
Not only does the evidence need to be produced, but it needs to be validated by in-
dependent experts. These experts need to determine the realism of assumptions, the
representativeness of scenarios, the thoroughness of analysis, and the coverage of key
off-nominal conditions. The risk of validating just nominal-case scenarios is shown
in the next section.
The following is a set of criteria that cover the various dimensions of validity for
the evidence:
• Data well defined
a. What is counted
b. How it is derived (e.g., how measured, calculated, or inferred)
• Representative mission scenarios
a. Operational environment
b. Adversaries
c. Component reliability, scalability, etc.
d. Nominal and off-nominal scenarios
e. Treatment of uncertainty
f. Composability of results; interference effects
g. Scale
• Parameter values realistic
a. Based upon measured results
b. Inferred from representative projects/activities
c. Derived from relevant scenarios
• Outputs traceable to mission effectiveness
a. Directly/indirectly
b. Based on appropriate models, scenarios
• Models verified and validated
a. Via representative scenarios
b. Limiting cases, off-nominals realistic
This should be used as a checklist and not as a one-size-fits-all set of criteria, which
would generally be overkill.
Thus, event-based reviews, where the event is defined as production of the specifi-
cations and plans, need to be replaced by evidence-based reviews.
This does not mean that the project needs to spend large amounts of effort in
documenting evidence of the feasibility of a simple system. As described above, the
appropriate level of detail for the contents of the FED is based on the perceived risks
and criticality of the system to be developed. It is NOT a “one size fits all” process,
but rather a framework to help developers and stakeholders determine the appropriate
level of analysis and evaluation. As with reused software, evidence can be appropri-
ately reused. If a more complex system than the one being reviewed has been suc-
cessfully developed by the same team, a pointer to the previous project’s evidence and
results will be sufficient. Table 1 outlines a process that can be used for developing
feasibility evidence.
The steps are designated by letters rather than numbers, to indicate that they are
carried out with considerable concurrency rather than sequentially. The process
clearly depends on having the appropriate work products for the phase (Step A). As
part of the engineering work, the high-priority feasibility assurance issues are identi-
fied that are critical to the success of the system development program (Step B).
These are the issues for which options are explored, and potentially viable options
further investigated (Step C). Clearly, these and the later steps are not performed
sequentially, but concurrently.
Since the preliminary design and plans are incomplete without the FED, it becomes
a first-class project deliverable. This implies that it needs a plan for its development,
and that each task in the plan needs to be assigned an appropriate earned value. If
possible, the earned value should be based on the potential risk exposure costs, not the
perceived available budget.
Besides monitoring progress on developing the system, the project needs to moni-
tor progress on developing the feasibility evidence. This implies applying corrective
action if progress falls behind the plans, and adapting the feasibility evidence devel-
opment plans to changes in the project objectives and plans. If evidence generation is
going to be complex, it is generally a good idea to perform pilot assessments.
7 Conclusions
The most important characteristic of evidence-based software specifications and plans
is that if the evidence does not accompany the specifications and plans, the specifica-
tions and plans are incomplete.
72 B. Boehm and J.A. Lane
Thus, event-based reviews, where the event is defined as production of the specifi-
cations and plans, need to be replaced by evidence-based reviews.
The Anchor Point milestones and Feasibility Evidence Descriptions presented in
this paper provide an approach for successfully realizing the benefits of proof-
carrying software specifications and plans. Further, the Anchor Point milestones
enable synchronization and stabilization of concurrent engineering.
The parametric analysis in Section 3 concludes that the greater the project’s size,
criticality, and stability are, the greater is the need for validated architecture feasibility
evidence. However, for very small, low-criticality projects with high volatility, the
evidence generation efforts would make little difference and would need to be con-
tinuously redone, producing a negative return on investment. In such cases, agile
methods such as Scrum and eXtreme Programming will be more effective. Overall,
evidence-based specifications and plans will not guarantee a successful project, but in
general will eliminate many of the software delivery overruns and shortfalls experi-
enced on current software projects.
Some implications of defining the FED as a “first class” project deliverable are that
it needs to be planned (with resources), and made part of the project’s earned value
management system. Any shortfalls in evidence are sources of uncertainty and risk,
and should be covered by risk management plans. The main contributions of this
paper are to provide experienced-based approaches for evidence criteria, evidence-
generation procedures, and measures for monitoring evidence generation, which
support the ability to perform evidence-based software engineering. And finally, evi-
dence-based specifications and plans such as those provided by the FED can and
should be added to traditional milestone reviews.
References
1. Al Said, M.: Detecting Model Clashes During Software Systems Development. Doctoral
Thesis. Department of Computer Science, University of Southern California (2003)
2. Beck, K.: Extreme Programming Explained. Addison-Wesley, Reading (1999)
3. Boehm, B.: Anchoring the Software Process. IEEE Software, 73–82 (July 1996)
4. Boehm, B., et al.: Software Cost Estimation with COCOMO II. Prentice Hall, Englewood
Cliffs (2000)
5. Boehm, B., Lane, J.: Incremental Commitment Model Guide, version 0.5 (2009)
6. Boehm, B., Lane, J.: Using the ICM to Integrate System Acquisition, Systems Engineer-
ing, and Software Engineering. CrossTalk, 4–9 (October 2007)
7. Boehm, B., Port, D., Al Said, M.: Avoiding the Software Model-Clash Spiderweb. IEEE
Computer, 120–122 (November 2000)
8. Glass, R.: Software Runaways. Prentice Hall, Englewood Cliffs (1998)
9. Kruchten, P.: The Rational Unified Process. Addison-Wesley, Reading (1999)
10. Maranzano, J., et al.: Architecture Reviews: Practice and Experience. IEEE Software
(March/April 2005)
11. Pew, R., Mavor, A.: Human-System Integration in the System Development Process: A
New Look. National Academy Press, Washington (2007)
12. Royce, W.: Software Project Management. Addison-Wesley, Reading (1998)
Evidence-Based Software Processes 73
13. Royce, W., Bittner, K., Perrow, M.: The Economics of Iterative Software Development.
Addison-Wesley, Reading (2009)
14. Standish Group 2009. CHAOS Summary (2009), http://standishgroup.com
15. Schwaber, K., Beedle, M.: Agile Software Development with Scrum. Prentice Hall,
Englewood Cliffs (2002)
SoS Management Strategy Impacts on SoS
Engineering Effort
Jo Ann Lane
1 Introduction
Today’s need for more complex, more capable systems in a short timeframe is leading
more organizations towards the integration of new and existing systems with com-
mercial-off-the-shelf (COTS) products into network-centric, knowledge-based, soft-
ware-intensive system of systems (SoS). With this approach, system development
processes to define the new architecture, identify sources to either supply or develop
the required components, and eventually integrate and test these high level compo-
nents are evolving and are being referred to as SoS Engineering (SoSE) [1].
In 2007, the United States Department of Defense (DoD) Office of the Secretary of
Defense (OSD) Acquisition, Technology, and Logistics (AT&L) Software Engineering
and Systems Assurance (SSA) organization sponsored case study investigations to bet-
ter understand SoSE. The results of these case studies [2] and other reports [3] and [4]
have indicated that SoSE activities are considerably different from the more traditional
SE activities. These differences are primarily due to adaptations and expansions of tradi-
tional SE activities to handle the increased size and scope of SoSs as well as the interac-
tions between the SoS team and the constituent-system (CS) engineering teams.
As a result of other SoS research [5] and [6], four types of SoSE management ap-
proaches have been identified: virtual, collaborative, acknowledged, and directed.
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 74–87, 2010.
© Springer-Verlag Berlin Heidelberg 2010
SoS Management Strategy Impacts on SoS Engineering Effort 75
These categories are primarily based upon the levels of responsibility and authority
overseeing the evolution of the SoS and are described in Table 1. Many SoSE teams
interviewed as part of the SSA SoSE case studies [2] indicated that their SoS was
managed primarily as a collaborative SoS until it reached a point where it was either
too important, too complex, or not cost effective to continue managing it in this man-
ner. At this point, an SoSE team was designated to guide the evolution of the SoS
CSs. Typically, in this first evolutionary step, the SoSE team has overarching engi-
neering responsibilities and can influence the CSs, but does not have complete author-
ity over the CSs (an acknowledged SoS).
Type Description
Virtual [5] Lacks a central management authority and a clear SoS purpose. Often ad
hoc and may use a service-oriented architecture where the CSs are not
necessarily known.
Collaborative CS engineering teams work together more or less voluntarily to fulfill
[5] agreed upon central purposes. No SoSE team to guide or manage
SoS-related activities of CSs.
Acknowledged Have recognized objectives, a designated manager, and resources at the
[6] SoS level (SoSE team), but not complete authority over constituent-
systems. CSs maintain their independent ownership, objectives, funding,
and development approaches.
Directed [5] SoS centrally managed by a government, corporate, or lead system
integrator and built to fulfill specific purposes. CSs maintain ability to
operate independently, but evolution predominately controlled by SoS
management organization.
The other major differences identified in [2] that must be captured in the SoSE
model in order to compare the two management strategies are:
1. SoS Capability/Requirements: SoS requirements start with very high level
capability need statements that must be analyzed to determine a set of imple-
mentable requirements. In the case of an acknowledged SoS, the SoSE
performs this activities. In the case of a collaborative SoS, the systems engi-
neering teams from all of the CSs must work together to determine a set of
implementable requirements.
2. SoSE Capability Analysis Support: When an SoSE team performs the capabil-
ity analysis and determines an approach for providing the capability, it
depends on support from the CS SE teams in conducting the trades and evalu-
ating the feasibility of the various options.
3. Monitoring of Non-SoS Changes: In an acknowledged SoS, the SoSE team
must also monitor non-SoS changes being implemented in the CSs to ensure
that these changes do not adversely affect the SoS. If changes might adversely
affect the SoS, the SoSE team negotiates with the CS(s) to determine alterna-
tive approaches that better support SoS objectives and performance.
The next modeling step was to determine ways to capture the major differences
described above. The SoS capability/requirements issue is modeled using the CO-
SYSMO Requirements Understanding parameter. The SoSE capability analysis sup-
port issue is modeled using the work of [8] that provides a distribution of COSYSMO
effort across the various systems engineering phases. The SoSE model modifies the
system design effort for the single system to account for the additional SoSE support.
And lastly, the COSYSMO reuse extension [9] provides a framework for incorporat-
ing adjustments related to SoSE monitoring CS non-SoS changes being implemented
in parallel with SoS requirements.
One additional problem remained with the COSYSMO model: how to combine
models of the SoSE team effort with the multiple system engineering team efforts.
The initial COSYSMO model treats the system of interest as a single entity, using a
single set of parameters to characterize the system characteristics, the system engi-
neering process, and the system engineering team. In an SoS environment, one needs
to be able to characterize SoS and single system characteristics, processes, and system
engineering teams differently while capturing the diseconomy of scale as the SoS
becomes larger and more complex. The approach used in the SoSE model was the
constructive cost model for software (COCOMO) method of integrating different
effort multipliers for different parts of the system [10].
The argument that this SOSE process modeling approach is sufficient to compare
SoS management approaches for the SoSs described in [2] is based on the following:
• The COSYSMO SoSE characterization is based on the vetted findings in the
DoD SoSE guidebook [2] using parameters readily available in COSYSMO
1.0.
• The COSYSMO 1.0 model calibration data set contained several systems that
characterized themselves as an SoS and most came from the same domain
(DoD) as the SoSE case studies.
78 J.A. Lane
• Each of the systems in the COSYSMO calibration data set belonged to one or
more SoSs, as indicated by external interfaces to share data and information
with other systems.
• Given that there are significant similarities between the COSYSMO 1.0 cali-
bration data set and the DoD SoSE case studies, the accuracy of COSYSMO
for SoSE should be somewhere between the values obtained for the CO-
SYSMO 1.0 (PRED(30)=75%) and the local calibrations performed by the
COSYSMO industry affiliates (PRED(30)=85%).
However, the major limitation of the SoSE process model is that it is not sufficiently
calibrated to estimate actual SOSE effort. This is left for future research.
3 SoSE Model
The SoSE model developed for the analysis of collaborative and acknowledged SoSE
management approaches was designed to evaluate the effort required to engineer a
single SoS capability in a software-intensive, net-centric SoS. This model uses the
SoS capability size expressed in equivalent nominal requirements (the composite
COSYSMO size driver used to calculate estimated effort). There are two primary
calculations: one to calculate the associated SE effort using an acknowledged SoS
management approach and one to calculate the associated SE effort using a collabora-
tive SoS management approach. The acknowledged approach estimates the SE effort
at the SoS level and the associated SE effort at the CS level. The collaborative ap-
proach estimates the SE effort at only the CS level (since there is no SoSE team in
this approach).
The underlying method used to determine the associated effort values (in person-
months) is the COSYSMO 1.0 algorithm, Effort = 38.55*EM*(size)1.06, where 38.55
and 1.06 are calibration constants for COSYSMO 1.0 [7]. For the SoSE model, effort
multiplier (EM) values are calculated for various parts of the engineering process:
SoSE for new capability, SoSE for oversight of the non-SoS CS changes, SE at the CS
level for the SoS capability (both with and without SoSE support), and SE at the CS
level for the non-SoS-requirements being implemented in parallel with the SoS capa-
bility. The following sections describe various aspects of the model in more detail.
Several assumptions and constraints were used in the development of the SoSE proc-
ess model. In some cases, these model assumptions and constraints generate more
conservative estimates of cost savings for an acknowledged SoSE team and therefore
strengthen the resulting findings. The assumptions and constraints are:
1. All CSs currently exist. This means that all of the CSs are legacy systems un-
dergoing some level of change (very small to major upgrades). In addition,
there are no new “long-lead” CS development activities that may have extraor-
dinarily high levels of internal change and/or may not be fielded within the
timeframe of the current SoS capability change. (There could be new systems
SoS Management Strategy Impacts on SoS Engineering Effort 79
under development that may eventually be part of the SoS, but these are not
considered as part of the SoSE comparison model.)
2. The model assumes a relatively mature engineering process at both the CS and
SoS levels. This is primarily because the COSYSMO cost model has been cali-
brated using SE projects from relatively mature SE organizations. This assump-
tion is also related to the fact that successful system development has been shown
to be strongly correlated to relatively “mature engineering processes” [11]. By
limiting the CSs to existing systems, as stated in the first assumption above, one
can reasonably assume it is appropriate to model the CS engineering processes as
“mature.” As for the SoSE team processes, they are typically not as mature as the
CS due to the fact that these processes are currently evolving to meet the new
challenges that SoSs present [2]. However, the SoSE teams typically have a
strong foundation upon which to build their processes given that they leverage
the processes of the CSs and work with the CS engineers.
3. In general, each CS has its own evolutionary path based on system-level stake-
holder needs/desires. This is related to the definition of an SoS [5]. The excep-
tion for some SoSs is the SoS infrastructure that integrates the CSs together.
This is typically identified as an SoS CS, but it does not necessarily have its
own evolutionary path outside of the SoS or independent of the SoS.
4. SoS capabilities are software-intensive. Most SoSs of interest today are those
that are net-centric in nature and the CSs are interfacing each other in order to
share data or information.
5. There is no SoS capability requirements volatility. The rationale for this is that
“no SoS capability requirements volatility” simplifies the initial process model
and the impact of this volatility would similarly affect both collaborative and
acknowledged SoSs. Intuitively, the presence of an SoSE team would some-
what streamline the configuration management of changes across the CSs and
be another “value added” aspect of an acknowledged SoSE team, but this SoSE
process dynamic is left for follow-on research. Also note that this assumption
is not applicable to the CSs and in fact, the model does assume varying levels
of CS volatility.
6. 100% of the SoS capability requirements are allocated to each CS needed to
implement the capability.
7. There is no focus on schedule or the asynchronous nature of SoS CS upgrades.
The scheduling aspects of SoS capability implementation are typically driven
by the asynchronous schedules of the CSs and preliminary reports do not indi-
cate that there are significant differences in the two approaches (collaborative
and acknowledged) except for the fact that an acknowledged SoSE team may
have some leverage to negotiate different priorities for CS SE changes.
8. Management of SoS internal interfaces reduces complexity for systems. This is
based on observations [2] and heuristics [12] about the common drivers of
complexity within a system or an SoS.
9. SE effort/information provided to the SoSE team in support of SoSE must be
added to SE effort for the subset of the CS upgrade requirements that are at the
SoS level. SoS programs participating in the DoD case studies reported that it is
difficult for an SoSE team to make reasonable decisions without input from the
CS SEs [2].
80 J.A. Lane
These assumptions and constraints (or simplifiers) allowed the model to focus on
comparing the two management approaches using the COSYSMO 1.0 calibration to
determine when it is cost effective to transition an existing collaborative SoS to an
acknowledged SoS. In addition, these assumptions and constraints limit the modeled
differences between the collaborative and acknowledged approaches to those key
SoSE differences identified in [2] and therefore produce conservative estimates with
respect to each approach.
The SoSE model incorporates several effort calculations. As mentioned earlier, each
of these calculations is based upon the COSYSMO 1.0 algorithm [7], shown in Equa-
tion 1 where 38.55 and 1.06 are calibration constants for COSYSMO 1.0.
Effort (in person-months) = 38.55*EM*(size)1.06/152 . (1)
In addition, the SoSE model uses the COCOMO II approach for applying different
EM factors to different parts of the system that have different cost drivers [10]. The
general form of this approach is shown in Equation 2. In Equation 2, i ranges from
one to the number of components and A and B are the calibration factors.
Effort = A * ∑ EMi*(componenti size/total size)*(total size)B . (2)
Both the acknowledged and collaborative CS effort calculations include the effort to
engineer the non-SoS-requirements being engineered concurrently with the SoS capa-
bility. This is to ensure that the SoSE model captures the associated diseconomy of
scale that occurs as the number of requirements at the CS level increases (whether
they are SoS-related requirements or CS-only requirements). The following sections
describe each of the SoSE model calculations using the terms defined in Table 3.
SoSE Effort for Acknowledged SoS. Using the COSYSMO algorithm with different
EMs for the SoS capability requirements and the CS “monitored” requirements, the
SoSE model calculation for the SoSE team’s effort is shown in Equation 3.
SoSE Effort = 38.55*[(( SoSCR / SoSTreq)* (SoSTreq)1.06 * EMSOS-CR)+
((SoSMR / SoSTreq)*( SoSTreq)1.06 * EMSOS-MR*OSF)] /152 . (3)
CS Effort with Support from SoSE Team (for Acknowledged SoS). This equation
calculates the CS effort for engineering the SoS capability requirements allocated to it
plus engineering in parallel the non-SoS-requirements scheduled for the current up-
grade cycle. In this case, the SoS requirements engineering is led by the SoSE team
with some support from the CS. Therefore, in this calculation, one needs to include
the CS engineering effort required to support system design at the SoS level. This
SoSE “tax” is based on the findings from the DoD case studies that indicate the SoSE
team requires the support of the CSs in the design of the approach for meeting the
SoS desired capability.
SoS Management Strategy Impacts on SoS Engineering Effort 81
Term Definition
CSnonSoS Number of non-SoS CS requirements planned for upgrade cycle
CSTreqSoSE Total number of requirements planned for CS upgrade cycle with support
from an SoSE team and is equal to SoSCSalloc + CSnonSoS
CSTreqwoSoSE Total number of CS requirements that must be addressed for the upgrade
cycle with no support from an SoSE team
EMSOS-CR Effort multiplier for SoS capability requirements engineered at the SoS
level
EMSOS-MR Effort multiplier for CS non-SoS-requirements monitored by the SoSE
team
EMCS- Effort multiplier for CS capability requirements engineering with SoSE
CRwSOSE team support
EMCS-CRnSOSE Effort multiplier for CS capability requirements engineering with no SoSE
team support
EMCSnonSOS Effort multiplier for CSs engineering for non-SoS-requirements
OSF Oversight adjustment factor to capture SoSE effort associated with
monitoring CS non-SoS changes (Values used include 5%, 10%, and 15%)
SoSCR Number of SoSE capability requirements (equivalent nominal requirements),
based upon the modeled size and complexity of the SoS capability
SoSCSalloc Number of SoS capability requirements allocated to CS
SoSMR Number of SoSE monitored requirements (equivalent nominal
requirements): the sum of all non-SoS-requirements being addressed in
parallel with the SoS capability requirements in the upgrade cycle
SoSTreq Total number of SoSE nominal (weighted) requirements: SoSCR +
(SoSMR*OSF)
Using the distribution of SE effort for system design in [8], this model approxi-
mates the SoSE tax by adding on half of the typical percentage of system design effort
(half of 30%) to the CSs for those requirements allocated to the CS from the SoSE
team. This factor was based on anecdotal inputs from systems engineers with experi-
ence in the SoS environment. Thus the calculation for CS SE effort using an SoSE
team is as shown in Equation 4.
CS SE Effort with SoSE Team = 38.55*[1.15*( (SoSCSalloc / CSTreqSoSE)*
(CSTreqSoSE)1.06 * EMCS-CRwSOSE) + (CSnonSoS / CSTreqSoSE)*( CSTreqSoSE)1.06*
EMCSnonSOS] /152 . (4)
Total Systems Engineering Effort (SoSE and SE) for Acknowledged SoS. The total
concurrent systems engineering effort for the acknowledged SoS is the sum of the SoSE
effort (Equation 3) and the CS SE effort with the SoSE team support (Equation 4).
CS Effort with No SoSE Team Support (for Collaborative SoS). In the case where
there is no SoSE team to support the engineering of the SoS capability requirements,
the CSs are responsible for engineering all of the SoS capability requirements (not
just an allocated subset) as well as the non-SoS-requirements planned for the system
82 J.A. Lane
upgrade cycle. Thus the calculation for CS SE effort without SoSE team support is as
shown in Equation 5.
CS SE Effort without SoSE Team = 38.55*[(( SoSCR / CSTreqwoSoSE)*(CSTreqwoSoSE)1.06*
EMCS-CRnSOSE) + ((CSnonSoS / CSTreqwoSoSE)*( CSTreqwoSoSE)1.06* EMCSnonSOS)] /152 . (5)
As indicated above, the SoSE model uses several EMs to calculate the effort associated
with the engineering of the SoS capabilities. If all cost drivers are set to nominal, the
resulting composite EM is 1.0. If the combined set of EMs is greater than nominal, the
resulting composite EM value is greater than 1.0. Likewise, if the combined set of EMs
is less than nominal, then the resulting composite EM value is less than 1.0.
This same process is used to determine the EM values for each of the other sets of
SoSE requirements: a) SoSE “monitored” requirements (SoSE-monitored), b) CS SE
for SoS capability with SoSE support (CS SoS w/SoSE), c) CS SE for SoS capability
without SoSE support (CS SoS w/o SoSE), and d) CS SE for non-SoS requirements
(CS non-SoS). Table 5 explains the rationale for each cost driver that is non-nominal
and shows the composite EM as calculated by COSYSMO 1.0.
Table 7 presents a sample of the data sets used as inputs and a summary of the re-
sults of the associated model run.
An evaluation of these results illuminates a set of heuristics that indicate when an
SoSE will be cost-effective from an effort perspective:
1. When the SoS capability size is greater than or equal to the non-SoS changes
being implemented in parallel with the SoS changes, there is usually an SoS
size at which the SoSE team becomes cost effective.
2. For larger SoSs, there are situations where the SoS capability size is smaller
than the non-SoS changes and the SoSE team is cost effective.
3. When the SoS size is small (5 or fewer systems) there are few situations where
an SoSE team is cost effective. The situation identified in the model execu-
tions where an SoSE team would be cost effective for a small SoS is when
SoS Management Strategy Impacts on SoS Engineering Effort 85
Capability
Capability
Volatility
SoS Size
Example
Scope of
Size OSF 5% OSF 10% OSF 15%
SoS
SoS
CS
Half of SoS size 10 or SoS size 11 or SoS size 12 or
1 2-200 100 100
CSs higher higher higher
Quarter of SoS size 10 or SoS size 11 or SoS size 12 or
2 4-200 100 100
CSs higher higher higher
Half of Not cost effective Not cost effective Not cost effective
3 2-200 100 2000
CSs for any size for any size for any size
SoS size 28 or Not cost effective Not cost effective
4 2-200 100 2000 All CSs
higher
SoS change SoS change SoS change
5 10 1000 1000 1-10 affects 50% CSs affects 60% CSs affects 60% CSs
or more or more or more
SoS change SoS change SoS change
10% to
6 100 1000 1000 affects 10% or affects 14% or affects 19% or
100%
more CSs more CSs more CSs
SoS change SoS change SoS change
7 10 1000 0 1-10 affects 5 CSs or affects 5 CSs or affects 5 CSs or
more more more
Not cost effective Not cost effective Not cost effective
8 10 1 1000 1-10
for any scope for any scope for any scope
SoS change SoS change SoS change
9 5 1000 1000 1-5
affects all 5 affects all 5 affects all 5
10 5 1 1000 1-10 Not cost effective Not cost effective Not cost effective
4. When the size of the SoS capability change is extremely small compared to the
non-SoS changes being implemented in the CSs, the SoSE team is generally
not cost effective.
5. The oversight factor (OSF) which indicates the relative amount of time spent
monitoring CS non-SoS changes also impacts the cost effectiveness of the SoS
team. This is a parameter that deserves further investigation with respect to a)
the level of CS understanding required at the SoS level to adequately engineer
SoS changes, b) the potential impacts of CS non-SoS changes, and c) the ef-
fects of rework when non-SoS changes adversely impact the goals of the SoS.
It is also important to note that these heuristics are derived from a set of cases that are
based upon a set of simplifying assumptions and constraints that are described above.
86 J.A. Lane
As was noted, these model assumptions and constraints generate relatively conserva-
tive estimates of cost savings for an acknowledged SoSE team and as a result, the
actual threshold points for a specific SoS may be lower than indicated by the model.
To determine the cost effectiveness for a specific SoS implementing a specific set of
SoS changes while the CSs are implementing their specific non-SoS changes, one
needs to model those specific characteristics to determine the point where an SoSE
team is cost-effective for that SoS.
References
1. Lane, J., Valerdi, R.: Synthesizing System-of-Systems Concepts for Use in Cost Estima-
tion. Systems Engineering 10(4), 297–307 (2007)
2. Department of Defense: Systems Engineering Guide for System of Systems, Version 1.0
(2008)
3. Northrop, L., Feiler, P., Gabriel, R., Goodenough, J., Linger, R., Longstaff, T., Kazman,
R., Klein, M., Schmidt, D., Sullivan, K., Wallnau, K.: Ultra-Large-Scale Systems: The
Software Challenge of the Future. Software Engineering Institute, Pittsburgh (2006)
4. United States Air Force (USAF) Scientific Advisory Board (SAB): Report on System-of-
Systems Engineering for Air Force Capability Development; Public Release SAB-TR-05-
04 (2005)
5. Maier, M.: Architecting Principles for Systems-of-Systems. Systems Engineering 1(4),
267–284 (1998)
6. Dahmann, J., Baldwin, K.: Understanding the Current State of US Defense Systems of
Systems and the Implications for Systems Engineering. In: Proceedings of the IEEE Sys-
tems Conference, Montreal, Canada, April 7-10 (2008)
7. Valerdi, R.: Constructive Systems Engineering Cost Model. PhD. Dissertation, University
of Southern California (2005)
8. Valerdi, R., Wheaton, M.: ANSI/EIA 632 as a Standardized WBS for COSYSMO, AIAA-
2005-7373. In: Proceedings of the AIAA 5th Aviation, Technology, Integration, and Op-
erations Conference, Arlington, Virginia (2005)
9. Wang, G., Valerdi, R., Ankrum, A., Millar, C., Roedler, G.: COSYSMO Reuse Extension.
In: Proceedings of the 18th Annual International Symposium of INCOSE, The Netherlands
(2008)
10. Boehm, B., Abts, C., Brown, A., Chulani, S., Clark, B., Horowitz, E., Madachy, R., Reifer,
D., Steece, B.: Software Cost Estimation with COCOMO II. Prentice Hall, Upper Saddle
River (2000)
11. SEI: Capability Maturity Model Integration (CMMI), CMU/SEI-2002-TR-001 (2001)
12. Rechtin, E.: Systems Architecting: Creating and Building Complex Systems. Prentice Hall,
Upper Saddle River (1991)
Using Project Procedure Diagrams for
Milestone Planning
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 88–99, 2010.
c Springer-Verlag Berlin Heidelberg 2010
Using Project Procedure Diagrams for Milestone Planning 89
Please note that our approach for the support of milestone planning only
cares about the logical order of milestones. We explicitely do not address issues
like time or resource planning. Nevertheless, when designing PPDs as a process
description language, we had to keep other constraints in mind. Each of the
following goals is motivated by the fact that the V-Modell XT is a real-life
process model with a large number and a wide variety of users:
Ease of Use. The description language must be understandable for project
managers with no background in process description languages. That means
that it should have a simple, possibly graphical syntax based on few, clearly
defined concepts that should lend themselves to hands-on experimentation and
simulation. Furthermore, the specification and elaboration of simple plans should
be as easy as possible to reduce the acceptance threshold.
Practicability. The description language must support typical activities and
different planning strategies of project managers, among them forward and back-
ward planning. Furthermore, it must be possible to compose a plan from an ex-
tensible library of reusable project procedure specifications. This includes support
for abstract procedure types which can be implemented by a number of differ-
ent concrete procedure specifications. This feature allows project managers to
choose the concrete procedure based on the current project context, and it allows
process designers to extend the procedure library without having to change exist-
ing procedure specifications. Lastly, the description language should be formally
defined as a precondition for tool support.
Expressivity. The description language must offer support for the generation
of milestone plans for arbitrarily complex projects. Based on our experience,
especially adequate support for arbitrary combinations of sequential iterations
and parallel subprocesses is a critical feature. It must be possible, for example,
to derive plans where the development subprocedure of a certain subcomponent
is decoupled from the main development flow and spans an arbitrary number of
iterations, each based on its own procedure specification.
3 Existing Approaches
As set out in Sect. 1, the approaches for the direct enactment of process models
are not appropriate for high-level project processes. FUNSOFT nets as described
by [3] are a good example. Although they were once intended to model overall
software development processes, they turned out to be better suited for the
description of recurring micro-processes and standardized business processes [4].
Other approaches try to provide a formal foundation for project and resource
planning [5,6] based on Petri nets. However, these approaches also do not support
the derivation of project milestone plans from process models.
Common software development process metamodels are of course aware of
the difference of software development projects to recurring business processes.
However, their support for plan-based enactment is surprisingly weak. OMG’s
SPEM 2.0 [7] contains concepts like Milestone and Work Sequence (Kind), but
Using Project Procedure Diagrams for Milestone Planning 91
states explicitly that it “does not aim to be a generic process modeling language,
nor does it even provide its own behavior modeling concepts.” Consequently, it
only sketches the derivation of project plans. ISO’s SEMDM [8] also contains
classes like Milestone and MilestoneKind but states that it “does not include
classes to represent a sequence of events”. As both standards cannot express the
consistency of a milestone plan with a process model, they were not suitable to
express our requirements within the V-Modell XT.
SPEM 2.0 refers, amongst others, to UML activity diagrams [9] as a possible
description language. Originally, they were intended to specify the behaviour of
complex software systems. Activity diagrams thus are a very complex description
technique, compromising their understandability especially for project managers
without expert knowledge in UML. Furthermore, activity diagrams lack some
features that are necessary to fulfill the given requirements. Although they do
support the composition of diagrams via the CallBehaviour construct, the called
behavior is fixed. Thus, the project manager cannot chose it freely during the
project without altering the calling activity diagram. This seriously affects the
reusability and combinability of process building blocks. Activity diagrams also
don’t support “inverse token flow” which would be necessary to allow a backward
planning strategy. Lastly, the implementation of tools is complicated by the fact
that there is no commonly agreed formal foundation [10].
This section introduces Project Procedure Diagrams (PPDs) as being used within
the V-Modell XT since 2006. They were at first used as a graphical notion only,
but then emerged to a comprehensive process description language with formally
defined syntax and semantics. As they are an integral part of the V-Modell XT
metamodel, they were developed from scratch without any attempt to extent an
existing notation (e.g. SPEM 2.0).
The following presentation uses a board game metaphor to explain the graph-
ical syntax (the board) and the operational token flow semantics (the rules for
game moves). This approach has been proven to work for non-experts during
many training sessions with dozens of attendees from German federal govern-
ment and German industry. Thus, it demonstrates the ease of use and the un-
derstandability of the approach. In the course of the presentation, we refer to
the remaining goals of Sect. 2 as appropriate to establish the suitability of PPDs
for the given task.
To explain how a project manager may derive a milestone plan from a given set
of Project Procedure Diagrams, Fig. 2 introduces some sample Project Proce-
dure Diagrams containing all relevant concepts. Figure 3i shows a corresponding
sample milestone plan and its derivation. In the remainder of this section, meta-
model concepts are printed in italics, whereas instances within the model are
92 K. Bergner and J. Friedrich
printed in a serifless font. The complete set of rules can be found in table 1;
references to certain rules are printed in a bold font.
The Procedure SimpleProject specifies the top-level procedure for a develop-
ment project: start the project, develop something, close the project. Accord-
ing to R1, the project manager puts a token on the Initial Node of the Root
Procedure as depicted in Fig. 3a on page 94. Each token has a memory, which
contains the set of previous milestones. At the beginning, the memory set is
empty, as the token is generated afresh.
After following R1, R2 is the only applicable rule. Hence, the project manager
moves the token along the outgoing Transition Arrow. Consequently, the token
arrives at Milestone Node Project approved and R3 must be applied. To do this,
the project manager adds the new milestone 1 to the milestone plan and names it
“MyProject approved”. The token’s memory is accordingly set to the milestone
set {1}. Figure 3b illustrates the described course of action.
Again, R2 is the only applicable rule. The project manager moves the to-
ken onto the Milestone Node Project initialized and adds milestone 2 to the
milestone plan. As the token’s memory contained {1} when it arrived at the
Milestone Node, she has to add an After Relationship, pointing from milestone 1
to milestone 2 (abbreviated as 1 → 2). At the end of this procedure, the token’s
memory is reset to {2} as shown in Fig. 3c.
According to R2, the project manager moves the token onto a Subnode labeled
with :SystemDevelopment. This triggers the application of R4, and the project
manager has to decide whether she wants to do Prototyping or to start with a
ClassicalApproach, both being procedures of procedure type SystemDevelopment.
Using Project Procedure Diagrams for Milestone Planning 93
No. Description
Derivation Rules
R1 At the beginning of the plan derivation phase, a single token with an empty memory
is placed on the Initial Node of the Root Procedure.
R2 A token may be moved along a Transition Arrow. During the move, the token’s
memory remains unmodified.
R3 When a token arrives at a Milestone Node, a new milestone has to be added to
the milestone plan. The new milestone is labeled with the name of the associated
Milestone Node (or any modified phrase). Furthermore, from every milestone in
the token’s memory, an After Relationship to the new milestone has to be added.
Finally, the token’s memory is reset to the set containing only the new milestone.
R4 When a token arrives at a Subnode, it must be moved to the Initial Node of the
Procedure that fulfills the Procedure Type to which the Subnode refers to. If there is
more than one adequate Procedure, one of them may be chosen at will. The token’s
memory remains unmodified.
R5 When a token arrives at a Split-In Node, it has to be replaced with a number of
new tokens (at least one). The new tokens have to be set on the corresponding
Split-Out Nodes. For each Split-Out Node, the number of tokens placed upon must
comply with the given cardinality. Other tokens already lying on a Split-Out Node
are not considered when the rule is applied. The memory of each added token
is initialized with the token that has arrived on the Split-In Node. After that,
this (old) token must be removed. From now on, the new tokens can be moved
independently.
R6 One or more tokens lying on the Join-In Nodes of a corresponding Join-Out Node
may be replaced by a single new token, provided that the given cardinalities are
respected. The new token must be placed on the Join-Out Node. Its memory is set
to the union of all memory sets of the removed tokens.
R7 When a token arrives at the Final Node of a Procedure, it has to be moved to
a Subnode that references the Procedure Type of the Procedure. If more than one
Subnode refers to the type, any of them can be chosen at will. The token’s memory
remains unmodified. Afterwards, Rule 2 must be applied for the token.
R8 At the end of the plan derivation phase, a single token has to lie on the Final Node
of the Root Procedure, and no other token may exist.
Transformation Rules
R9 Two temporally independent milestones may be merged to a single milestone. All
existing relationships are kept during that operation.
R10 Additional After Relationships may be added, as long as they don’t lead to cycles
in the milestone plan.
R11 Redundant milestone relationships must be removed. A relationship is redundant
if it can be derived from other relationships.
R12 Additional (so-called “free”) milestones may be added at will.
R13 Milestone relationships have to be added until there is exactly one project start
milestone that lies before every other milestone and exactly one project end mile-
stone that lies after every other milestone.
94 K. Bergner and J. Friedrich
As the requirements are not yet clear, the project manager decides to move the
token to the Initial Node of the Procedure Prototyping as depicted in Fig. 3d.
Inescapably, the token arrives at a Split-In Node. In accordance with R5 the
project manager has to decide whether she needs independent milestone threads
or not. She already knows that the system will probably consist of three compo-
nents C1, C2, C3 and that she will commission a sub-contractor with the con-
struction of component C3. So she replaces the existing token with three new
tokens (each representing the development of one component) as depicted in
Fig. 3f. Note that this behavior cannot be easily modeled with a ForkNode as
provided by UML acticity diagrams, but requires extending the ForkNode with
other concepts (for example, ControlFlow.weight and MergeNodes). Afterwards,
by applying Rules 2 to 4, the project manager adds the milestones “C1 (proto-
type) finished” (3), “C2 (prototype) finished” (4) and “C3 specified” (7) as well
as the necessary relationships (2 → 3, 2 → 4, 2 → 7).
In the following, two tokens arrive at the upper Split-In Node, as depicted in
Fig. 3h. According to R6, the project manager has to decide whether she wants
to merge the two independent milestone threads. As there is only one prototype
system consisting of the two prototype implementations of C1 and C2 (note that
C3 is not included in the prototype), she decides to replace both tokens with one
and therefore to move only one token onto Milestone Node System integrated. As
the replaced tokens remember having been at milestones 3 and 4 respectively, the
new token’s memory is set to {3, 4}. As consequence, the project manager adds
milestone “Prototypical system integrated” (5), which is temporally dependent
on milestones 3 and 4, as depicted in Fig. 3i.
After this, the project manager moves the token located in Procedure Proto-
typing onto Milestone Node Architecture specified and adds milestone 6 to the
milestone plan. Inescapably, the token arrives at the subsequent Final Node and
R7 must be applied. As the procedure specification contains only one appro-
priate Subnode (:SystemDevelopment, located in Procedure SimpleProject), the
token must be moved there. Now, as two outgoing Transition Arrows start at
the Subnode, the project manager can apply R2 in two different ways: either
she moves the token to Milestone Node Project closed or she takes the other
Transition Arrow and moves the token from the Subnode to the Subnode, lead-
ing to the application of R4. She chooses the latter, as she wants to re-engineer
the prototypical versions of C1 and C2 in a second iteration. Furthermore, this
second iteration is necessary to integrate the externally developed component
C3 into the system. This time, the ClassicalApproach is used. Figure 3g depicts
the described action.
The subsequent moves can be performed according to the already introduced
rules. Thus, the project manager moves the tokens and adds milestones 8, 9,
10, 11, 12 as well as the following relations: 7 → 8, 8 → 9, 6 → 10, 10 → 11,
10 → 12. To finish the planning procedure (see R8) the project manager has
to merge all existing tokens (representing three independent milestone threads)
as depicted in Fig. 3j. As a precondition for the merge, she has to move the
token lying in Procedure ExternalDevelopment to Procedure ClassicalApproach as
96 K. Bergner and J. Friedrich
depicted in Fig. 3e. Note that this token came from Prototyping and used the
Procedure ExternalDevelopment as a kind of “bridge” (or tunnel).
After the merge, the project manager moves the one remaining token to the
Final Node of SimpleProject and adds milestones 13 and 14 (and relationships
11 → 13, 12 → 13, 9 → 13, 13 → 14) along the way. Now the plan derivation
phase is finished according to R8. This rule ensures that all parallel subprocesses
are finally joined together at some point in time. It also ensures that the token
flow game has a clearly defined end. Note that this doesn’t follow from rules R1
to R7, as the tokens created by a single application of R5 may be joined by
multiple applications of R6.
Note also that (in the general case, not in the example) the planning procedure
may not be finishable due to a “wrong” move by the project manager during
the planning game. Such a move may lead to a situation where some tokens on
Join-In Nodes cannot be moved anymore. In cases like this, the project manager
has to go back and rectify the wrong move.
(see Fig. 4f) is consistent if and only if every typed milestone is in the image
set of the mapping and if the transitive closure of the milestone relationships
(see Fig. 4e) is a superset of all relationships induced by the reduced trace. The
superset property ensures that the final milestone plan may always be “stricter”
than required by the process model.
combined planning strategies (and maybe also with the capability to check the
consistency of arbitrary milestone plans with a given set of PPDs) and adding a
library of reusable PPDs could ease the job of a project manager considerably.
Acknowledgements
We want to thank Andreas Rausch and Thomas Ternité for stimulating discus-
sions during the definition of PPDs.
References
1. Andersen, E.S.: Warning: activity planning is hazardous to your project’s health?
International Journal of Project Management 14(2), 89–94 (1996)
2. Friedrich, J., Hammerschall, U., Kuhrmann, M., Sihling, M.: Das V-Modell XT:
Für Projektleiter und QS-Verantwortliche kompakt und übersichtlich, 2nd edn.
Informatik im Fokus. Springer, Berlin (2009)
3. Deiters, W., Gruhn, V.: Process management in practice: Applying the FUNSOFT
Net approach to large-scale processes. Automated Software Engineering 5(1), 7–25
(1998)
4. Graw, G., Gruhn, V., Krumm, H.: Support of cooperating and distributed business
processes. In: Proceedings of International Conference on Parallel and Distributed
Systems, pp. 22–31 (1996)
5. Kim, J., Desrochers, A.A., Sanderson, A.C.: Task planning and project manage-
ment using Petri nets. In: Proceedings of the IEEE International Symposium on
Assembly and Task Planning, pp. 265–271 (1995)
6. Chen, Y.-L., Hsu, P.-Y., Chang, Y.-B.: A Petri net approach to support resource
assignment in project management. IEEE Transactions on Systems, Man and Cy-
bernetics, Part A: Systems and Humans 38(3), 564–574 (2008)
7. Object Management Group (OMG): Software & systems process engineering meta-
model specification (SPEM) version 2.0 (2008)
8. International Organization for Standardization (ISO): Software engineering meta-
model for development methodologies (SEMDM) (2007)
9. Object Management Group (OMG): Unified modeling language infrastructure
(2009)
10. Schattkowsky, T., Forster, A.: On the pitfalls of UML 2 activity modeling. In:
MISE 2007: Proceedings of the International Workshop on Modeling in Software
Engineering, p. 8. IEEE Computer Society, Los Alamitos (2007)
11. Bergner, K., Friedrich, J.: Modulare Spezifikation von Projektabläufen: Eine for-
male Fundierung von Syntax und Semantik der Projektdurchführungsstrategien
des V-Modell XT 1.3. Technical Report TUM-I0912, Technische Universität
München, Institut für Informatik, München (April 2009)
12. Reisig, W.: Petrinetze: Eine Einführung, 2nd edn. Springer, Berlin (1986)
A Framework for the Flexible Instantiation of
Large Scale Software Process Tailoring
1 Introduction
Traditional process driven software development approaches which are often
large, complex and formalized are widely used in industry. Examples are software
development within Siemens AG [18] and Motorola [9] with the former having a
USD 3 billions annual turnaround in software and 20,000 developers. The tasks
which must be conducted during the development are usually defined in so-called
software processes. They are comparable to reference processes that describe in
a generic way which tasks need to be considered in various software development
projects across multiple business units within a company. In order to increase
their applicability in various projects, they cover a wide range of functionality
and are therefore complex and large in size.
The task of adapting a reference process for a project is, apart from size
and complexity, further complicated by constraints. Constraints ensure that pro-
cesses follow certain correctness criteria. We considered two types of constraints
in this project, (a) syntax constraints defined by the meta model of the used
modelling language and (b) constraints defined by an organization to support
the understanding of processes by software developers, e.g., the Siemens Process
Framework [18]. Although we investigated only those types of constraints, the
presented framework supports constraints on process elements that may origi-
nate from other sources, e.g., from legal regulations like the FDA regulations for
medial devices [10] or maturity models such as CMMI.
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 100–111, 2010.
c Springer-Verlag Berlin Heidelberg 2010
A Framework for the Flexible Instantiation 101
2 Related Work
The area of project specific composition and adaptation of software processes and
methods has attracted significant attention in recent years as in, e.g., Brinkkem-
per’s Method Engineering (ME) proposal [7] as an approach for the creation of
situational methods.
However, no existing approach has established itself as a de-facto standard
in industry and provides a thorough solution for adapting Siemens processes
to project specific needs. For example, ME emphasized bottom-up assembly of
project specific methods from fragments but has failed to be widely accepted in
industry [9][14]. Contrary to ME, approaches like Little-JIL [8] regard processes
as programs, enacted by machines. Here however, we are concerned with flexi-
ble method engineering in the large and deal with semi-formal process models
offering high level guidance for humans.
An important reason for the lack of acceptance in industry is the variety
of meta-models for processes used in practice. For instance, Yoon et al. [20]
102 P. Killisperger et al.
copies the process for each possible correction again and applies the corrections
in the individual copies.
This procedure is continued until all constraints are satisfied. Processes satis-
fying all constraints are saved and presented to the user who can decide which
variation of the process to adopt after all solutions have been computed. The
algorithm for correcting a process is shown in Listing 1.
1: controlCorrection ( Process o r i g i n a l ) {
2: Container s o l u t i o n s ;
3: i f ( original . isValid () )
4: r e t u r n s o l u t i o n s . add ( o r i g i n a l ) ;
5: else {
6: Stack l i f o ;
7: l i f o . push ( o r i g i n a l ) ;
8: w h i l e ( l i f o . s i z e ( ) != 0 ) {
9: P r o c e s s p = l i f o . pop ( ) ;
10: Constraint vio = p . get V iolat ion ( ) ;
11: Container adaptedProcesses = vio . c o r r e c t ( ) ;
12: i f ( a d a p t e d P r o c e s s e s != n u l l {
13: f o r ( P r o c e s s p2 : a d a p t e d P r o c e s s e s ) {
14: i f ( p2 . i s V a l i d ( ) )
15: s o l u t i o n s . add ( p2 ) ;
16: else
17: l i f o . push ( p2 ) ;
18: }
19: }
20: }
21: }
22: return solutions ;
23:}
the process (p2) is pushed in lifo (line 17). Container solutions is eventually
returned to the caller, if there are no further processes in lifo (line 22).
When applying the algorithm as described a mechanism for preventing looping
constraints and for avoiding computation of equal resulting correct processes has
to be implemented.
Constraints loop when a correct() adapts a process leading (not necessarily
immediately) to a violation whose correct() causes a violation triggering the
first correct() again.
Duplicates in the resulting process solutions can occur when corrections due
to process violations lead to the same adapted process.
A control mechanism for preventing such situations has to find out whether
an identical process exists in lifo or solutions to a newly created and adapted
process. This can be expensive since a high number of processes might have to
be compared.
However, a loop or equal resulting processes can also be detected when a
violation occurs on an entity in the partially corrected process which has already
occurred on the same entity in an earlier version of the process during correction
(i.e. within controlCorrection). However, for it to work, correct() must solve
a violation entirely (i.e. the constraint is not violated after the execution of this
correct() any more) otherwise a loop would be detected that does not exist.
The latter approach is less expensive since only violations have to be compared
and correct() can be defined to satisfy the restriction to correct a violation
entirely. For preventing loops and duplicates in solutions we therefore check
whether the same violation has already occurred on the same entity during the
correction procedure leading to the partially corrected process at hand and if so,
the correction is no longer continued.
From this follows that certain constraints (e.g. on control flows) have to
be checked and corrected after execution of each operator of a batch.
2. Batch executions of operators can lead to obviously incorrect interactions
between possible correction choices. This is especially inefficient and cum-
bersome if the correction of the violation requires user input or is expensive
regarding computational resources.
For example, assume an activity A is deleted having an output information
flow with a workproduct W (i.e. A creates W ), and there is another activity
B using W as input information flow, causing a violation because W is
input of an activity but never created. If the activities having W as input
information flow are deleted in a later stage of the batch, the correction will
be unnecessary.
From this follows that particular constraints (e.g. on information flows)
have to be executed after all operators of the batch have been executed.
The issues of an increased number of possible correct resulting processes and
increased number of necessary corrections can be resolved by prioritizing con-
straints at two levels. Constraints with priority (1) are checked and, if necessary,
corrected after the execution of each operator, thus limiting the number of pos-
sible resulting processes. Constraints with priority (2) are executed after all
operators of the batch have been executed thus making the process adaptation
more efficient by avoiding unneeded constraint checking and correction.
4 Evaluation
A prototype of a system for instantiating a reference process as described above
has been implemented for a particular business unit of Siemens AG. The busi-
ness unit uses a reference process comprising 23 types of entities including phases
108 P. Killisperger et al.
It was used for improving the reference process of the business unit. There
had been efforts to make the reference process compliant with constraints but
this had not been successful due to the complexity of the task. For example,
after manually detecting and correcting of a violation, verification has to be
started all over again, since the correction of the violation might have caused
other violations. Siemens AG estimates that several man-months of effort were
saved due to the application of the approach.
The tool has also been used for creating a variant of the reference process for
a pure software system. Manual creation of this variant had not been carried
out, due to the effort involved resulting from dependencies of process elements
and the necessity of compliance with constraints.
5 Conclusion
We have described a current research effort to improve software process related
activities at Siemens AG. Part of these efforts is the development of a system that
supports project managers in instantiation of reference processes subject to an
explicit constraint representation. The system aims not only to execute decisions
but to restore correctness of the resulting process. Since the implementation of
such a system is organization-specific and depends on the permitted constructs in
the process, a flexible architecture has been developed supporting organization-
specific definition of constraints, their correction when violated and instantiation
of processes as required by the applying organization. Testing proved feasibility
of the approach and thus showed that the approach enables project specific
adaptations of Siemens software processes which has not been feasible due to
missing tool support until now. Future work includes further testing in real
world projects and integration of the approach in a process management tool.
Acknowledgements
This work was partially supported by a DAAD postgraduate scholarship.
References
1. Allerbach, A., Bauer, T., Reichert, M.: Managing Process Variants in the Process
Life Cycle. In: Proceedings of the Tenth International Conference on Enterprise
Information Systems, vol. ISAS-2, pp. 154–161 (2008)
2. Armbrust, O., Katahira, M., Miyamoto, Y., Münch, J., Nakao, H., Ocampo, A.:
Scoping Software Process Models - Initial Concepts and Experience from Defining
Space Standards. In: Wang, Q., Pfahl, D., Raffo, D.M. (eds.) ICSP 2008. LNCS,
vol. 5007, pp. 160–172. Springer, Heidelberg (2008)
3. Baresi, L., Guinea, S., Pasquale, L.: Self-healing BPEL processes with Dynamo
and the JBoss rule engine. In: ESSPE, pp. 11–20 (2007)
4. Becker, U., Hamann, D., Verlage, M.: Descriptive Modeling of Software Processes.
In: Proceedings of the Third Conference on Software Process Improvement, SPI
1997 (1997)
A Framework for the Flexible Instantiation 111
5. BMI. The new V-Modell XT - Development Standard for IT Systems of the Federal
Republic of Germany (2004), http://www.v-modell-xt.de (accessed 01.12.2008)
6. Boukhebouze, M., Amghar, Y., Benharkat, A.-N., Maamar, Z.: Towards Self-
healing Execution of Business Processes Based on Rules. In: ICEIS, pp. 501–512
(2009)
7. Brinkkemper, S.: Method engineering: engineering of information systems devel-
opment methods and tools. Information & Software Technology 38(4), 275–280
(1996)
8. Cass, A.G., Staudt Lerner, B., McCall, E.K., Osterweil, L.J., Sutton Jr., S.M.,
Wise, A.: Little-JIL/Juliette: A Process Definition Language and Interpreter. In:
International Conference on Software Engineering, vol. 0, p. 754 (2000)
9. Fitzgerald, B., Russo, N., O’Kane, T.: Software development method tailoring at
Motorola. Communications of the ACM 46(4), 65–70 (2003)
10. U.S. Food and Drug Administration. Code of Federal Regulations Title 21, Ch. 1,
Part 820: Medical Devices Quality System Regulation (2009),
http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/
CFRSearch.cfm?CFRPart=820&showFR=1 (accessed 11.11.2009)
11. Greiner, U., Müller, R., Rahm, E., Ramsch, J., Heller, B., Löffler, M.: AdaptFlow:
Protocol Based Medical Treatment using Adaptive Workflows. Journal of Methods
of Information in Medicine (2004)
12. Killisperger, P., Peters, G., Stumptner, M., Stückl, T.: Instantiation of Software
Processes: An Industry Approach. In: Information Systems Development - Towards
a Service Provision Society, pp. 589–597. Springer, Heidelberg (2009)
13. Killisperger, P., Stumptner, M., Peters, G., Stückl, T.: Challenges in Software
Design in Large Corporations - A Case Study at Siemens AG. In: ICEIS (3-2),
pp. 123–128 (2008)
14. Niknafs, A., Ramsin, R.: Computer-Aided Method Engineering: An Analysis of
Existing Environments. In: Bellahsène, Z., Léonard, M. (eds.) CAiSE 2008. LNCS,
vol. 5074, pp. 525–540. Springer, Heidelberg (2008)
15. Reichert, M., Dadam, P.: Enabling Adaptive Process-aware Information Systems
with ADEPT2. In: Handbook of Research on Business Process Modeling. Informa-
tion Science Reference, pp. 173–203. Hershey (2009)
16. Rombach, H.D.: Integrated Software Process and Product Lines. In: ISPW,
pp. 83–90 (2005)
17. Rosemann, M., Van der Aalst, W.: A Configurable Reference Modelling Language.
Information Systems 32(1), 1–23 (2007)
18. Schmelzer, H.J., Sesselmann, W.: Geschäftsprozessmanagement in der Praxis: Pro-
duktivität steigern - Wert erhöhen - Kunden zufrieden stellen, 4th edn. Hanser
Verlag (2004)
19. Ternité, T.: Process Lines: A product Line Approach Designed for Process Model
Development. In: Proceedings of the 35th EUROMICRO Conference on Software
Engineering and Advanced Applications, SPPI Track (2009)
20. Yoon, I.-C., Min, S.-Y., Bae, D.-H.: Tailoring and Verifying Software Process. In:
APSEC, pp. 202–209 (2001)
A Methodological Framework and Software
Infrastructure for the Construction of Software
Production Methods*
1 Introduction
Method Engineering (ME) is defined as the engineering discipline to design, con-
struct and adapt methods, techniques and tools for the development of information
systems [1]. A lot of research work has been developed in this area during the last two
decades, especially in the adaptation of Software Production Methods (here after
SPM) to a specific situation, area that is known as Situational Method Engineering
(SME). Most of the approaches developed in these two areas propose the construction
of SPMs by assembling reusable fragments (parts that compose a method). Regarding
the fragments assembly we find works focused on the description of these fragments
(as it is the case of the MEL language proposed by Brinkkemper in [1]) and on the
study of techniques for selecting ([3], [7]) and assembling ([2], [14], [15]) efficiently
the fragments stored in a repository. Nevertheless, these approaches are difficult to
*
This work has been developed with the support of MEC under the project SESAMO
TIN2007-62894 and cofinanced by FEDER.
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 112–125, 2010.
© Springer-Verlag Berlin Heidelberg 2010
A Methodological Framework and Software Infrastructure 113
put into practice, mainly because they are neither supported by software tools nor
complete solutions for the design and implementation of SPMs (generation of CASE
tool support for the designed SPM). This software support is usually provided in the
shape of CAME environments (CAME stands for Computer-Aided Method Engineer-
ing and refers to the tools supporting ME approaches) which combine theoretical
proposals with tool support for the construction of SPMs. However, to our knowl-
edge, the existing environments constitute incomplete prototypes that only cover
some parts of the ME process. This fact hinders the expected industrial adoption and
success considering the benefits that this type of tools would provide to industry.
In this paper, we provide a methodological framework that allows designing and
implementing SPMs. This is possible since the framework includes a set of engineer-
ing tools that assist during the development of the SPM. To build this framework we
apply the main ideas of the most relevant approaches developed in ME (SPM con-
struction through fragment assembly, usage of a reusable fragments repository, proc-
ess enactment, etc.). The proposed framework has been designed following the Model
Driven Engineering (MDE) principles. These principles advocate for the intensive use
of models and model transformations to perform system development. So, the SPM
design is done by specifying the model of the SPM which describes the process of the
SPM. The key concepts of this model are tasks (work done during the process) and
products (input and/or output of tasks). The construction of this model has been de-
fined in two steps. During the first step, the method engineer creates a generic SPM
model that does not include details such as how tasks have to be performed. Then, in
the second step, the method engineer instantiates the generic model by providing
information about the languages, notations or techniques used in each task or product.
This approach allows the method engineer to make different configurations of the
same generic description of the SPM according to special features of the different
projects or development teams. Once the SPM model is completed, model transforma-
tions are used to automate the generation of the CASE tool supporting the SPM. The
generated CASE tool includes: (1) all the software components that provide support
to the tasks and products of the SPM and (2) the software component that provides
support to the process of the SPM. The last component allows the software engineer
to keep the state of a SPM instance and to guide him/her during its execution. These
two aspects of the CASE tool correspond to the two parts in which a SPM is divided
(the product part and the process part).
The process followed to build the proposed framework has applied a bottom-up
strategy. It implies that the process begins by identifying the characteristics of the
CASE tools that our framework should obtain. In order to determine these characteris-
tics we have developed a CASE tool that supports a specific SPM. Once these charac-
teristics have been identified, they have been used to determine the concepts required
to specify SPM and further on to define the framework. The CASE tool has been built
using the MOSKitt tool [12], an Eclipse-based tool which includes (1) a plugin-based
architecture and (2) a set of integrated modeling tools (e.g. EMF [5]). The characteris-
tics of the MOSKitt tool turn it into a suitable software platform to support our pro-
posal, i.e. to build on it the CAME and CASE tools.
The main contribution of this work is to provide a methodological framework to help
and assist method engineers not only in the definition of SPM but also in deriving
CASE tools to support them. Among the characteristics of the framework we highlight
114 M. Cervera et al.
(1) the intensive use of standards, (2) its integrated design being all the process carried
out with the same technology (the MOSKitt tool), (3) the automation degree achieved
thanks to the application of MDE techniques, (4) the reusability degree of the SPM
models (specifying generic SPM that can be configured to suit it to a specific context of
use), and (5) the generation of CASE tools that support process enactment.
The remainder of the paper is structured as follows. Section 2 briefly presents the state
of the art of the ME and SME making emphasis in the limitations found in the existing
proposals. Then, section 3 introduces an overview of our proposal. Section 4 presents in
detail the methodological framework designed to cover the limitations found in ME/SME
areas. Section 5 presents a case study where we apply the proposed methodological
framework. Finally, section 6 provides some conclusions and outlines further work.
1
In this figure, the CAME environment has been divided in two parts, the CAME part, which
provides facilities for ME, and the CASE part, which offers means for the generation of
CASE tools and process support environment.
A Methodological Framework and Software Infrastructure 115
Trying to solve the limitations found in ME context, this work proposes a meth-
odological framework that, applying the techniques developed in the MDE area, helps
and assists the method engineer in both, the specification of SPM and the construction
of CASE tools for the support of such methods.
• Method design: The method engineer builds the Method Model by identifying all
the elements (tasks, products, etc.) involved in the process of the SPM. These
elements can be defined from scratch or selected from a method base repository,
which contains reusable elements that were specified in other SPM. This first
version of the model constitutes a generic description where no specific lan-
guages or notations are specified for the elements of the model.
116 M. Cervera et al.
To perform these three phases, we use as much as possible the available standards. To
specify SPM we use SPEM 2.0 [17], which is the standard developed by the OMG to
define software process methods, and to define both repositories we use RAS [16], the
A Methodological Framework and Software Infrastructure 117
OMG specification that provides a standard way to assembly, organize, store and
document reusable software assets.
We provide a software infrastructure, which is a set of method engineering tools, to
support the three phases of the framework. This infrastructure assists the method engi-
neer during the method design and method configuration phases, and automates a large
part of the method implementation phase. The tools provided in the software infrastruc-
ture are editors, repositories and guides, which are detailed in the next section.
This phase corresponds to the design part of Fig. 2. During this phase, the method
engineer builds the method model, which can either be built from scratch or by reus-
ing method fragments from the method base repository. This model describes the
process of the SPM by using concepts such as:
All these concepts specify who (Role), when (Flow Connector and Gateways) and
how (Task, Product and Guide) the process of the SPM is performed. In this work, to
specify the method model according to these concepts we have used SPEM 2.0, the
standard developed by the OMG for the definition of SPM.
As a result of this phase we obtain the method model which is a SPM description
that does not include specific techniques or languages/notations associated to its ele-
ments (e.g. we can specify the product business process model which does not refer to
any technique or notation). These details will be provided in the next phase (method
configuration). The main benefits of this separation are that: (1) we keep generic
definitions of SPM (which means that we can take this generic definition and perform
118 M. Cervera et al.
• An editor to build the method model: We use the EPF Composer editor
(EPFC), a SPEM 2.0 editor provided in the EPF project [6], which has been
integrated in MOSKitt. By using this editor we can build SPM models ac-
cording to the SPEM metamodel. These models describe the process of the
SPM, which can be described associating UML Activity Diagrams to the
own model, and the products involved in the process.
• A method base repository: Method fragments are stored in a method base re-
pository following the RAS standard (which implies that the fragments have
to be packaged using this specification). To reuse the fragments of this re-
pository in the method model, it is necessary to extend the EPFC editor. This
extension allows the user to retrieve fragments from the repository during the
specification of the method model. To retrieve the fragments in an efficient
way, we make use of some of the ideas proposed in [15]. According to these
ideas, our method fragments are characterized by (1) a descriptor, which
provides information about origin, objective and type of the fragment and,
(2) an interface which is a couple <situation, intention> that characterises the
situation that is the input of the fragment and the intention (goal) that the
fragment achieves.
• A guide to build the method model: A wizard is provided to guide the method
engineer through the performance of the method design phase.
Once the method design phase has been finished, the method engineer should add to
the method model some information related to its execution (configuration part of
Fig. 2). This is necessary since no information regarding the technique or lan-
guage/notation was given during the previous phase. Therefore, some of the products
and tasks included in the method model have to be associated with assets that support
them. These assets are contained in the asset base repository, which stores metamod-
els, transformations, editors, etc. With this association, the method engineer instanti-
ates the generic model into a specific one. For example, a business process model
product identified during the method design phase refers to a generic model, which
could be built in any language or notation. Later on, during the method configuration
phase, this product is associated to the BPMN metamodel (which is an asset stored in
the asset base repository). Thus the method engineer instantiates the generic business
process model into a concrete notation, which is in this case the BPMN notation.
It is possible that the asset required by the method engineer is not stored in the as-
set base repository. In this case, the flexibility of the plugin-based architecture in
which Eclipse relies on allows the method engineer to build this asset. Then, this asset
is stored in the repository so that the method engineer can use it in the configuration
of the method model.
A Methodological Framework and Software Infrastructure 119
1 2 3
Identify Software
Solve
Software Resources
Dependencies
Resources Deployment
MPS Model CASE Tool
Once the method model has been built (i.e., the method design phase is finished),
we configure it (method configuration phase). To do so, we define the links between
the elements of the model (tasks and products) and the assets that support them, which
are stored in the asset base repository. Table 1 and Table 2 show the links between the
tasks and products specified in the method model of the case study and the assets of
the asset base repository. The symbol * points out that the asset was initially available
in the repository, whereas the symbol ** points out that the asset is defined by the
method engineer.
Task Asset
Business Process Analysis Guide with the steps of the task *
System Specification Guide with the steps of the task *
Business Process Design Guide with the steps of the task **
Web specification M2M Transformation **
Enrich Web Specification Guide with the steps of the task **
Web Application Generation M2T Transformation **
Executable Business Process M2M Transformation **
122 M. Cervera et al.
Product Asset
Business Process Model BPMN Metamodel *
Conceptual Model OO-Method Metamodel *
Navigational and Presentation Models OOWS Metamodel *
Tapestry Code Text Editor *
WS-BPEL Document Text Editor *
Services Model Services Metamodel **
The links between SPM elements and the assets of the asset base repository are
established through instances of the SPEM class ReusableAsset. An example of this
link is shown in Fig. 5.
An Eclipse application (Rich Client Application) is obtained from the method model
specification. This application represents the CASE tool support (the CASE tool and
process support environment) for the SPM. The CASE tool is a MOSKitt
reconfiguration that includes all the plugins supporting the product and process parts
of the SPM. A screenshot of this tool is shown in Fig. 6. Several views can be
observed in the figure. These views are the following:
• Product Explorer: shows in a tree viewer the products consumed/produced/
modified by the ongoing tasks and the tasks that have already been executed.
A Methodological Framework and Software Infrastructure 123
This view can be filtered by role so the user only see the products he/she is
responsible for. By selecting the product element, the associated software
resource is opened to edit the product.
• Process: shows in a tree viewer the different tasks that can be executed in the
current moment. This view can be filtered in a similar way as the Product
Explorer so the user only see the tasks he/she is responsible for. When a task
is selected in this view:
o If this task generates a product, then this product is created and
added to the “Product explorer“.
o If this task consumes a product and this has not been generatead
previously in the process, the user has to specify its file system
location. On the contrary, if the product has already been created in
the process, the user has to indicate which product from the
available ones (those appearing in the Product Explorer view) is
consumed in the task.
• Guides: shows the list of the guides associated to the product or task selected
in the corresponding view. The main goal of these guides is to assist the user
during the manipulation of the product or the execution of the associated
task. There may be more than one guide associated to a product/task as
shown in Fig. 6 and even different types of guides (i.e. Eclipse cheat sheets,
text files, hyper-text files, etc.). When a guide is selected, it is opened with
the corresponding editor.
• Product Dependencies: shows the list of dependencies of the product or
task selected in the corresponding view. These dependencies are grouped by
role allowing the user to know the existing dependencies between the prod-
ucts manipulated by him and also by other roles.
References
[1] Brinkkemper, S.: Method Engineering: Engineering of Information Systems Development
Methods and Tools. Information and Software Technology 38, 275–280 (1996)
[2] Brinkkemper, S., Saeki, M., Harmsen, F.: Meta-Modelling Based Assembly Techniques
for Situational Method Engineering. Inf. Syst. 24, 209–228 (1999)
[3] Brinkkemper, S., Saeki, M., Harmsen, F.: A Method Engineering Language for the De-
scription of Systems Development Methods. In: Dittrich, K.R., Geppert, A., Norrie, M.C.
(eds.) CAiSE 2001. LNCS, vol. 2068, pp. 473–476. Springer, Heidelberg (2001)
[4] Cetina, C., Giner, P., Fons, J., Pelechano, V.: Autonomic Computing through Reuse of
Variability Models at Runtime: The Case of Smart Homes, vol. 42, pp. 37–43. IEEE
Computer Society Press, Los Alamitos (2009)
[5] Eclipse Modeling Framework Project,
http://www.eclipse.org/modeling/emf/
[6] Eclipse Process Framework Project (EPF), http://www.eclipse.org/epf/
[7] Harmsen, F., Brinkkemper, S.: Design and Implementation of a Method Base Manage-
ment System for a Situational CASE Environment. In: Asia-Pacific Software Engineering
Conference, p. 430. IEEE Computer Society, Los Alamitos (1995)
[8] Henderson-Sellers, B.: Method engineering for OO systems development. Commun.
ACM 46, 73–78 (2003)
[9] Ter Hofstede, A.H.M., Verhoef, T.F.: On the feasibility of situational method engineer-
ing. Inf. Syst. 22, 401–422 (1997)
[10] Kumar, K., Welke, R.J.: Methodology Engineering: A Proposal for Situation-Specific
Methodology Construction. In: Challenges and Strategies for Research in Systems De-
velopment, pp. 257–269. John Wiley & Sons, Inc., Chichester (1992)
[11] Mirbel, I., Ralyté, J.: Situational method engineering: combining assembly-based and
roadmap-driven approaches. Requir. Eng. 11, 58–78 (2005)
[12] MOdeling Software Kitt (MOSKitt), http://www.moskitt.org
[13] Niknafs, A., Ramsin, R.: Computer-Aided Method Engineering: An Analysis of Existing
Environments. In: Bellahsène, Z., Léonard, M. (eds.) CAiSE 2008. LNCS, vol. 5074, pp.
525–540. Springer, Heidelberg (2008)
[14] Ralyté, J., Rolland, C.: An Assembly Process Model for Method Engineering. In: Dittrich,
K.R., Geppert, A., Norrie, M.C. (eds.) CAiSE 2001. LNCS, vol. 2068, pp. 267–283.
Springer, Heidelberg (2001)
[15] Ralyté, J., Rolland, C.: An Approach for Method Reengineering. In: Kunii, H.S., Jajodia, S.,
Sølvberg, A. (eds.) ER 2001. LNCS, vol. 2224, pp. 471–484. Springer, Heidelberg (2001)
[16] Reusable Asset Specification (RAS) OMG Available Specification version 2.2. OMG
Document Number: formal/2005-11-02
[17] Software Process Engineering Meta-model (SPEM) OMG Available Specification ver-
sion 2.0. OMG Document Number: formal/2008-04-01
[18] Torres, V.: A Ph.D. thesis entitled “A Web Engineering Approach for the Development
of Business Process-Driven Web applications”. Technical University of Valencia (2008),
http://www.dsic.upv.es/mapa/ingles/desctesis.pl?tesis=etd-
04112008-140714
[19] Xpand, http://www.eclipse.org/modeling/m2t/?project=xpand
[20] Armbrust, O., Katahira, M., Miyamoto, Y., Münch, J., Nakao, H., Ocampo, A.: Scoping
Software Process Models - Initial Concepts and Experience from Defining Space Standards.
In: Wang, Q., Pfahl, D., Raffo, D.M. (eds.) ICSP 2008. LNCS, vol. 5007, pp. 160–172.
Springer, Heidelberg (2008)
Software Factories: Describing the Assembly
Process
1 Introduction
The software industry remains reliant on the craftsmanship of skilled individuals
engaged in labor intensive manual tasks. However, growing pressure to reduce
cost and time to market, and to improve software quality, may catalyze a transi-
tion to more automated methods. In this environment, Software Factories (SFs)
represent a paradigm shift with the promise to make application assembly more
cost effective through systematic reuse. This fosters the raise of supply chains
while opening the door to mass customization.
SFs capture the entire knowledge on how to produce the applications within
a specific product family. SFs offer such knowlege in the form of assets like
patterns, frameworks, models and tools. Those assets are then systematically
used to automate the delivery of family members. This impacts significanly on
reducing cost and time to market, while improving product quality over one-off
development [9]. However, these advances in software industrialization reduce the
cost of coding applications at the expense of increasing assembling complexity,
i.e., the process of coming up with the final end product (hereafter referred to
as the Assembly Process).
Advanced engineering paradigms to support SFs illustrate this point: Model
Driven Engineering (MDE) and Software Product Line Engineering (SPLE).
MDE raises the level of abstraction, defining models that capture the specifics
of the application at hand, which will then be transformed into the actual ap-
plication code implementation. As for SPLE, it aims at building a set of related
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 126–137, 2010.
c Springer-Verlag Berlin Heidelberg 2010
Software Factories: Describing the Assembly Process 127
products out of a common set of core assets. Unlike MDE, now the stress is not
so much on the abstraction level at which software is specified, but on conceiving
programs as pre-planned variations from core assets.
Both SPLE and MDE depart from one-off development to provide an in-
frastructure where different (though related) applications can be obtained. An
SPLE Assembly Process starts with the core assets where variation points are
gradually instantiated to establish the features which the end application will
eventually exhibit. Here, the SF outputs applications that mainly differ on the
extent they support a set of pre-planned functional options. By contrast, an
MDE Assembly Process is conceived as a pipeline of models and model trans-
formations from abstract to more concrete realizations of the end application in
an specific platform. Here, the SF delivers applications that mainly differ on the
underlying technological platform.
Both SPLE-based and MDE-based assembly processes are complex in their
own right [13,16]. While, the complementary nature of SPLE and MDE has
raised wide expectations among the community (e.g. variability can be more
concisely described at model level rather than at code, transformations can also
be subject to variability hence addressing non-functional requirements, etc) [12],
the combined use of SPLE and MDE puts even more stringent demands on the
assembly process. We have experienced this difficulty ourselves. In [15], we re-
ported a combined use of SPLE and MDE where facing the assembly process
resulted overall complex. Completing the assembling process for just an indivi-
dual application of the family took itself four people/day.
Based on these observations, this work advocates for a new discipline inside
the general software development process, i.e. the Assembly Plan Management .
The activities, roles and phases of this discipline are described, and illustrated
for a compound SPL/MDE-based application development. The concerns, roles
or assets needed for assembling programs in such a setting are described for
this application. Two common approaches to face complexity are abstraction
and separation of concerns. Accordingly, more abstract assembling constructs
are introduced, and distinct regards are decoupled among different activities.
SPEM is used as a notation.
2 Background
The SEI Product Line Practice Framework. A Software Product Line
(SPL) is defined as a set of software-intensive systems sharing a common, man-
aged set of features that satisfy the specific needs of a particular market seg-
ment or mission, and that are developed from a common set of core assets in a
prescribed way [4]. Unlike project-based approaches targeted to one-off develop-
ment, SPLs can have quite a long life-cycle since they provide the framework to
build distinct products along time. Therefore, SPL processes, and the associated
organizational structure are more permanent in nature, leading to a fundamental
paradigm shift compared to a project-based approach.
128 M. Azanza, O. Díaz, and S. Trujillo
rather than a possibility. So, the attention is not focused any more on a product
for a specific customer (e.g. building a website for Iberia) but on an specific ap-
plication domain, (e.g. constructing a website for airlines). Hence, the challenge
rests on establishing the scope of the SPL (a.k.a the domain), identifying the
variations to support (through the feature model [10]), and developing the infra-
structure that permits a product of the family to be developed in a cost-effective
way while keeping high quality standards.
Feature Oriented Software Development (FOSD) is one of the techniques to
realize SPLE. FOSD is a paradigm for creating software product lines where
customized programs are obtained by composing features [2]. Hence, features
not only describe increments in program functionality that serve to distinguish
between members of the product family, but are the actual software building
blocks. A product line in an FOSD includes a base that expresses the greatest
common denominator of all SPL members, and a set of meaningful increments of
functionality (hereafter referred to as deltas) that implement the features. The
SPLE Assembly Process can then be described as an equation. For instance, the
equation app = d2 • d1 • base indicates that app is obtained by composing deltas
d1 and d2 to the base. Unlike the MDE case, now delta transformations are
endogenous, i.e. the source and target models conform to the same metamodel.
3 Motivation by Example
We outline our experience on a software factory: PinkCreek, which is a product
line of portlets (building blocks of web portals) for the provision of flight reserva-
tion capabilities to travel-agency portals [6]. The challenges were two fold. First,
both the diversity and unstability of Web platforms advice to abstract away from
platform specificities into a platform-independent model. This grounds the use
of MDE. On the other hand, flight reservation is very similar among companies
but not exactly the same. Companies exhibit variations in how flight reserva-
tion is conducted. Such diversity is captured through deltas that end up being
supported using SPLE techniques. This combined setting can be characterized
by a (deltas, transformations) pair : deltas (i.e. features) stand for variations
on flight reservation whereas transformations account for mappings between the
distinct levels of abstraction at which flight booking is captured. Next para-
graphs describe the distinct artifacts that arise from both the MDE and the
SPLE perspectives (a more complete account can be found at [15]).
MDE perspective. An MDE practitioner would first strive to abstract from
the different Web platforms that can realize PinkCreek. To this end, State Charts
(SC ) are introduced to model flight-reservation control flow in a platform-
independent way. The application is described as sequence of states where each
state represents an HTML fragment (i.e. a unit of delivery during user interac-
tion). States are connected by transitions whose handlers either execute some
action, render some view, or both. A State Chart is then mapped into a model for
portlet controlers: the Ctrl metamodel. Ctrl models are in turn, mapped into Act
and View models, that define the actions to be performed and the views to be
130 M. Azanza, O. Díaz, and S. Trujillo
1. The Assembly Process is complex. PinkCreek scripts, which realize the as-
sembly process, accounted on average for 500 LOC of batch processes using
300 LOC of ANT makefiles and 2 KLOC of Java code.
2. The Assembly Process needs to be designed. We design because there are
options and tradeoffs. Design is a way to handle complexity by abstracting
away from the large number of details and focusing on the essentials. There
is not a single way to application production. Distinct assembly alternatives
may need to be contrasted and assembly counterpoints can arise.
3. The Assembly Process becomes repetitive. The paradox is that the assembly
process is typically geared towards the reuse of source code, but its realiza-
tion often lacks such reuse. This occurs in our case study where no reuse
mechanism was initially in place. Specifically, defining alternative assembly
processes involved a great deal of potential reuse but, puzzling enough in an
MDPLE setting, the unique technique available for reuse was rudimentary
“clone&own”. It did not take long to realize that this technique did not scale
up.
Wrapping it up, a need was felt for a plan that manages how the assembly
process is defined, allowing its design and capitalizing on reuse. The following
section describes such plan.
Our claim is that the combined use of MDE and SPLE increases the burden
of application development in general, and application assembly in particular.
Therefore, additional effort should be dedicated to come up with an infrastruc-
ture (i.e. core assets) that facilitates application assembly during Application
Engineering. The effort to build such infrastructure will payoff by streamlining
the assembly process.
132 M. Azanza, O. Díaz, and S. Trujillo
as stated in Figure 4. These functions will be later enacted by the Assembly Pro-
gram. It is important to highlight that this library can be reused for any portlet
SPL as long as the same metamodels are used to describe the artifacts, so this
task has to be performed only once.
Megamodel Concretization Engineering Phase. A megamodel provides a
common framework that can support different SPLs as long as they all use the
same metamodels. For instance, the megamodel depicted in Figure 4 can describe
not only PinkCreek but any SPL where SC, Ctrl and the other metamodels as well
as deltas are the mechanisms used to capture domain specification and variability.
Now, these metamodels and deltas are concretized for the domain at hand (e.g.
PinkCreek ). This basically means to define the base and the deltas for PinkCreek.
The PinkCreek Megamodel Concretization Model behaves as a catalogue of these
artifacts where information about the base and deltas are collected as well as their
location in the project directory. In this way, the Assembly Program will be able
to locate the deltas. This prevents the need to manually repeat this information
every time an Assembly Program is to be enacted.
Assembly Program Engineering Phase. During this phase the Application
Engineer declaratively describes the assembly process as an Assembly Equation.
For instance, she decides to add the bountyFees feature to the baseModel and
then, move down to code by applying transformation sc2ctrl, sc2act and act2Jak.
The app = act2jak • ctrl2act • sc2ctrl • bountyF ees • baseM odelSC reflects this
decision. Equation operands include model transformations (e.g. act2jak ) and
Software Factories: Describing the Assembly Process 135
deltas (e.g. bountyFees). The former are realized as functions of the Assembly
Machine Tool obtained during Megamodel Engineering. As for deltas, they are
obtained by referring to the Megamodel Concretization Model which is obtai-
ned during the Megamodel Concretization Phase.
Hence, an Assembly Equation is a declarative description of how a final ap-
plication is to be assembled. This equation is then mapped to an executable
Assembly Program, through a model-to-text transformation. The outcome is a
Java/Jak Assembly Program that provides flight reservation with bounty fees as
the payment mechanism.
Product Assembling. This phase is limited to enacting the Assembly Program
which delivers the Application Source, i.e. a set of code artifacts that compile
into the desired application binary.
5 Discussion
This work is based on some perceived liabilities on the what-is-to-come in appli-
cation assembly, namely: complexity (i.e. assembly programs become larger),
choiceful (i.e. assembly programs becomes fork-like), and "clonefulness” (i.e.
some assembly tasks are repetitive which leads to code snippets being repeated)
(see Section 3). The expected benefit of our approach would be the improvement
of this situation. Next, we evaluate to which extent this aim is achieved.
Handling Complexity. Before this work, an average assembly program to create
a single portlet in PinkCreek accounted for 500 LOC of batch processes using 300
LOC of ANT makefiles and 2 KLOC of Java code. More important, it took around
4 people/day to complete. Now, an assembly program to build a portlet with 5
features is specified with an equation of 15 LOC. This is providing that the Mega-
model Engineering Phase and the Megamodel Concretization Engineering Phase
have already been performed, where each took around one hour and were only
carried out once for all programs. This is an improvement of at least one order of
magnitude in the effort needed to develop the assembly program.
Handling Choices. Assembly programs are now specified at a higher abstrac-
tion level instead of being directly implemented. This allows the Application
Engineer to concentrate on the design of the assembly process, pondering the
advantages and trade-offs of each decision, without worrying about implementa-
tion details. Currently the only decision available is the order of the operations
when assembling a product (in domains where the order is unimportant). How-
ever, the equation could be enriched with more decisions related to the assembly
process . These choices in the Assembly Process can be captured as part of the
variability of the production plan [5].
Handling “Clonefulness”. As assembly gets more complex, an increase in the
number of clones, i.e. chunks of code that are repeated in distinct parts of the
program, is observed. This is what we meant when we stated that assembly
programs are cloneful. Such “clonefulness” calls for a reuse mechanism that de-
parts from traditional “clone&own” practices. This is precisely what we achieved
136 M. Azanza, O. Díaz, and S. Trujillo
by abstracting away from code and moving to a higher level where clones are
generated through transformations from the abstract model and where reuse
is accomplished in a systematic manner using model driven techniques. Model
transformations were only the first mechanism provided to foster reuse. We also
introduced the notion of Assembly Machine Tool, a reusable library that pro-
vides operations for model transformation within the scope set by a megamodel,
and which is reusable by every product family defined in such domain.
6 Related Work
7 Conclusions
The main contribution of our work is that assembly processes are no longer
implemented, but are modeled and thus designed, raising the abstraction level
in assembly processes. This makes the reuse of assembly processes possible. We
evaluated our ideas with a case study. Our future plans involve considering varia-
bility in the assembly process. According to our previous work, the production
plan is subject to variability [5]. From this perspective, the assembly process
may also have to cope with such variability.
Acknowledgments. This work was co-supported by the Spanish Ministry of Ed-
ucation, and the European Social Fund under contract MODELINE, TIN2008-
06507-C02-01 and TIN2008-06507-C02-02.
References
1. Batory, D., Azanza, M., Saraiva, J.: The Objects and Arrows of Computational De-
sign. In: Czarnecki, K., Ober, I., Bruel, J.-M., Uhl, A., Völter, M. (eds.) MODELS
2008. LNCS, vol. 5301, pp. 1–20. Springer, Heidelberg (2008)
2. Batory, D., Neal Sarvela, J., Rauschmayer, A.: Scaling Step-Wise Refinement.
IEEE Transactions on Software Engineering, TSE (2004)
3. Bézivin, J., Jouault, F., Rosenthal, P., Valduriez, P.: Modeling in the Large and
Modeling in the Small. In: MDAFA (2004)
4. Clements, P., Northrop, L.M.: Software Product Lines - Practices and Patterns.
Addison-Wesley, Reading (2001)
5. Díaz, O., Trujillo, S., Anfurrutia, F.I.: Supporting Production Strategies as Re-
finements of the Production Process. In: Obbink, H., Pohl, K. (eds.) SPLC 2005.
LNCS, vol. 3714, pp. 210–221. Springer, Heidelberg (2005)
6. Díaz, O., Trujillo, S., Perez, S.: Turning Portlets into Services: the Consumer Pro-
file. In: WWW (2007)
7. France, R., Rumpe, B.: Model-driven development of complex software: A research
roadmap. In: FOSE (2007)
8. Frankel, D.S.: Model Driven Architecture: Applying MDA to Enterprise Comput-
ing. Wiley, Chichester (2003)
9. Greenfield, J., Short, K.: Software Factories: Assembling Applications with Pat-
terns, Models, Frameworks and Tools. In: OOPSLA Companion (2003)
10. Kang, K.C., Cohen, S.G., Hess, J.A., Novak, W.E., Peterson, A.S.: Feature Ori-
ented Domain Analysis (foda) Feasability Study. Technical Report CMU/SEI-90-
TR-21, Software Engineering Institute (November 1990)
11. OMG. Software Process Engineering Metamodel Specification. Formal Specifica-
tion (April 2008), http://www.omg.org/spec/SPEM/2.0/PDF
12. AMPLE Project. Aspect-Oriented, Model-Driven Product Line Engineering
(2009), http://ample.holos.pt/
13. Rivera, J.E., Ruiz-Gonzalez, D., Lopez-Romero, F., Bautista, J., Vallecillo, A.:
Orchestrating ATL Model Transformations. In: MtATL (2009)
14. Trujillo, S., Azanza, M., Díaz, O.: Generative Metaprogramming. In: GPCE (2007)
15. Trujillo, S., Batory, D., Díaz, O.: Feature Oriented Model Driven Development: A
Case Study for Portlets. In: ICSE (2007)
16. Vanhooff, B., Ayed, D., Van Baelen, S., Joosen, W., Berbers, Y.: UniTI: A Unified
Transformation Infrastructure. In: Engels, G., Opdyke, B., Schmidt, D.C., Weil,
F. (eds.) MODELS 2007. LNCS, vol. 4735, pp. 31–45. Springer, Heidelberg (2007)
How to Welcome Software Process Improvement and
Avoid Resistance to Change
1 Introduction
Implementing a successful Software Process Improvement (SPI) is a challenging
issue that many software companies face today. Many organizations experience a
successful start on their SPI initiative and after some phases of their improvement
process they realize that the engagement to change weakens significantly after the
initial excitement [1]. Previous studies show critical factors that can be sources for
this problem e.g. resistance to change, previous negative experience, lack of the evi-
dence of the benefits, imposition, resource constraints and commercial pressures to
meet customer demands [2, 3].
Human, social and organizational factors play a decisive role for the success or
failure of a process improvement initiative and cultural issues emerge as an important
player of the changing workplace [4].
In this work, we study and evaluate the implications of one de-motivator factor for
SPI, the resistance of practitioners to adopt a new practice. This problem occurs when
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 138–149, 2010.
© Springer-Verlag Berlin Heidelberg 2010
How to Welcome Software Process Improvement and Avoid Resistance to Change 139
practitioners are not adequately encouraged to give up the old practices that they are
familiar with and that they feel comfortable with [2]. As Humphrey observed [5] this is
not a trivial problem “particularly because even intelligent people often will not do
things that common logic, experience, and even hard evidence suggests that they
should.” We will also present the impact of this factor into organization’s projects
through the analysis of an improvement practice. In addition, we provide some evalua-
tion of the benefits of this specific improvement practice, using process simulation.
The motivation of this work is to show that some barriers for SPI can be overcome
with a simple modification of team behavior, which will impact considerably in the
project results. More specifically, we will present one initiative, a simple modification
of the quality procedure, which was implemented to reduce rework. Considering the
successful results of this initiative, we believe that it can be easily incorporated in
many industrial software development processes.
In the next section we present the relevant features of the organization under study,
including a brief description of its software development process. In Section 3, we
describe the organization improvement effort that was structured as a Defect Causal
Analysis process. Section 4 provides an analysis of the results using process simula-
tion. Section 5 concludes the work and suggests some directions for further research.
2 Organization Background
Here we describe a work that was performed as a collaborative study with Synergia.
Synergia is a laboratory for software and systems engineering, hosted in the Computer
Science Department at Federal University of Minas Gerais, Brazil. Synergia is inter-
nally organized as a commercial software development organization, but it also re-
tains important academic characteristics [6]. Synergia maintains 85 people in its staff
composed by undergraduate and graduate students and non-student graduated profes-
sionals, most with a Computer Science background.
Synergia uses a tailored version of the Praxis model-driven software development
process [7] in its software and systems engineering projects, called Praxis-Synergia.
Although the Praxis process has been designed and applied primarily for education
and training in Software Engineering, it provides tailoring guidelines, which must be
interpreted according to the current situation of the development organization. The
Praxis material is available in a book and kept updated in the author’s Web site1.
One important characteristic of Praxis, which is maintained in the tailored version,
is that it models the development of each product use case as a sequence of develop-
ment states or control marks. The name of each state evokes how far the use case has
advanced towards complete implementation and acceptance. Table 1 summarizes
these states and a detailed discussion of them can be found in Pádua [6, 8]. Each state
is associated with one or more quality control procedures such as inspections, tests
and management reviews. During the execution of these activities each found defect
is recorded.
1
www.dcc.ufmg.br/~wilson/praxis/ (in Portuguese)
140 D.C.C. Peixoto et al.
The problem discussed in this article was detected while performing the quality
procedures of the Implemented state (see Table 1). Before delivering the executable
code to the Test team, the developers themselves were required to execute a list of test
procedures (a subset of the complete Manual Test Specification). The purpose of this
verification was to identify errors made by the developers themselves with the aim of
treating them before the execution of the test procedures by the Test team. For exam-
ple, a nonoperational save command or in some cases whole inoperative use case
could pass unnoticed. It is worth to mention, in previous projects (before Project
TBI), when this verification practice was adopted, the number of critical defects de-
tected by the testers dropped significantly. With these observations and some analysis
by the Software Process team, the organization decided to incorporate this practice in
its development process. Project TBI adopted it as a task required for advancing the
use case from Realized to Implemented state (Figure 1).
Fig. 1. Process Flow Chart for the Original and Modified Process
Although in Project TBI all developers were required to execute the list of test proce-
dures, they did not execute it correctly or they did not execute it at all. In a broader
evaluation, the non-execution of these test procedures could be a result of time pressure,
project cost overrun, and lack of training or experience of the team. But at Project TBI,
the developer had enough time to do their task and they knew or were trained in what
they had to inspect. Trying to find one possible cause, the developers in a DCA meeting
were asked about the reasons. It was not possible to identify one specific source of the
problem. However, the interviews drew attention to a specific team behavior: they sim-
ply neglected this task. But, why this happened? Even if the project manager had
required the developers to execute this procedure they did not execute them! One ra-
tionale behind this problem could be resistance to change, or in other words, the resis-
tance to include this practice in the developers’ list of ‘daily’ practices. Considering this
hypothesis, one proposed solution was to force the developers to execute the test proce-
dures using a specific tool, which records each step executed and the test result (whether
it succeeded or failed). In this way, the developers would provide to the Test team an
142 D.C.C. Peixoto et al.
evidence, metaphorically like signing a document, that they had really executed the test
procedures.
The action group monitored and evaluated the implementation of this action in
Project TBI to see if it would produce some positive effect. Surprisingly, the result
lived up the expectations, mainly because after 5 months the benefits of this unique
action amounted to 217% of the investments in DCA meetings. Table 2 summarizes
the problem.
Many critical defects detected during the verification task of the executable code
Problem
carried out by the testers.
Developers did not execute correctly the test procedures before delivering the use
Cause case to the Test team. Eventually, the developers did some corrections in the test
procedures when the latter had some problems or were invalid or ambiguous.
The IBM Rational Manual Tester (RMT), a testing tool that forces the developer to
record each step during the verification of the test procedures, was incorporated into
Solution the process. A log produced by the tool should be extensively evaluated by the Test
team in order to ensure that the developer performed these manual tests properly,
before the verification of the complete test procedures.
After five months of implementation, the benefit of this action amounted to 217% of
the investment in DCA meetings. At the end of Project TBI, there was an average
reduction of 5.66 hours / use case (45%) of rework to fix and check for defects. A
Benefits
total saving of 796 h. The number of use cases without defects increased from 21%
to 53%. The average number of defects reduced from 5.1 to 1.7 defects / use case
and the number of critical defects reduced in 70%.
about how Process Simulation Modeling has been implemented within industry and
government organizations to improve their processes and achieve higher levels of
process maturity. Via many examples, this work showed how process simulation
supports CMMI Process Areas from maturity level 2 through level 5 and how some
simulation results can be useful in determining financial performance measurements
such as Return on Investment and Net Present Value. In another article, Raffo [17]
showed that simulation can provide quantitative assessment of the risk and/or uncer-
tainty associated with process change and support quantitative prediction of project
level performance in terms of effort, staffing, schedule and quality.
In our work we use process simulation to forecast the impact of applying our spe-
cific process improvement practice before each inspection, as illustrated in Figure 2.
Fig. 2. Process Flow Chart for the Original and Modified Process
What is the amount of rework reduction with the adoption of the verification prac-
tice before each inspection?
What is the best configuration for the adoption of this practice?
The verification practice is also carried out using RMT. In this case, the check-list
items are documented in this tool, allowing the author’s verification before delivering
the artifact for the inspection. The objective is the same, forcing the author to check
these items. In our analysis, we observed that some items of the organization’s check-
list are subjective. For example, “A brief documentation of the use case is clear and
accurately describes its purpose”. These items can also be included in the RMT as a way
to remind the author to check them. However, in this case, we believe that the detection
of this type of error by the author may be difficult, mainly because he/she has written
the documentation as clear and as accurately as possible in his/her point of view.
The model used in this study is a model of the Praxis-Synergia process as shown in
Figure 3. The model has been calibrated with historical data from the organization
and it has been validated against the performance data from one large-project (with
the size of 585 thousand non-blank non-comment lines of code).
144 D.C.C. Peixoto et al.
A)
B)
C)
Fig. 3. Praxis-Synergia Process Simulation Model into 3 levels of abstraction. A) First level of
abstraction of the software development process. B) Detailed model of the ‘Solution Develop-
ment’ process. C) Detailed model of the ‘Test Specification’ Process.
The tool used to build the model was IBM WebSphere Business Modeler (WBM)
Advanced Edition. This tool offers process modeling, simulation and analysis capa-
bilities, with the aim of helping business users to understand, document, and deploy
business processes for continuous improvement.
How to Welcome Software Process Improvement and Avoid Resistance to Change 145
The average work for the AS-IS scenario is 325.34 hours and the average rework is
91.32 hours per use case.
In this scenario, we made changes to represent the utilization of the RMT in each
inspection of the software development process. The Praxis-Synergia development
process has 10 inspections and the average duration and rework of each inspection
was shown in Table 3.
We made changes into the model to represent this scenario, and then we ran the
model for 20 runs. Figure 5 shows the distribution of the rework probabilities for this
scenario. In this case, 10% of the process instances have a rework of 49.4 hours.
The average work for the TO-BE scenario is 410.77 hours and the average rework
is 40.45 hours per use case. Despite the increase in the number of work hours, the
rework reduced drastically (~56% of reduction). The work increased mainly because
of the amount of author’s effort required to verify the check-list items before deliver-
ing the artifact for inspection. The total effort of the AS-IS scenario, 416.66 hours, is
less than the total effort of the TO-BE scenario, 451.22 hours. However, evaluating
the cost we observed that the TO-BE scenario presents a reduction of 2.8% of the
total cost comparing with the AS-IS scenario.
How to Welcome Software Process Improvement and Avoid Resistance to Change 147
It is interesting to note that, in terms of effort, the reduction of the rework seems to
not compensate the work of implementation of the RMT in each inspection. But we
know that detecting and correcting errors in earlier stages of the process signify cost
reductions [18]. In addition, the simulations have shown that will be worthwhile to
use RMT when considering the cost parameter.
For this scenario, we examined the impact of implementing the verification practice in
some specific group of inspections, including: a) requirement inspections; b) design
inspections; and c) implementation inspections.
We made changes to the model to represent these alternative configurations, and
then we ran each model for 20 runs. We compare the results of the AS-IS and the TO-
BE processes. Table 5 summarizes the results.
Similar to scenario 1, the rework reduced in some cases more than 50%. However
the total work of using RMT for implementation inspections is higher than not using
it. The reason for this is that one specific implementation activity (see Table 4), cod-
ing, is very time-consuming and, in some cases, it takes more than 50% of the total
work to develop a use case from scratch. In this case, with an additional verification
task, the artifact will take a longer time to be delivered for the inspections. However,
at this stage the rework activities cost more.
In Table 6, we can observe that the cost of the TO-BE process is less than the cost
of the AS-IS process for all groups of inspections. This result is similar to the first
simulation result.
148 D.C.C. Peixoto et al.
Although, in some cases the total effort increased, we believe, based on the positive
results, that RMT can be used in all inspections of the whole development process.
5 Conclusion
Software development takes place within a rich cultural environment. To be successful,
a software process initiative needs to recognize this aspect and make explicit the soft-
ware practices as they are understood and applied by the software engineering team. In
this context, our work discussed one specific cultural factor, the resistance to change,
that affected a successful implementation of a software process improvement action in
a specific organization. In addition, we also showed how this organization overcame
this de-motivator factor through an adoption of a specific verification practice.
We used simulation to forecast what-if scenarios for adoption of this practice. For
that, we created a model of the organization’s software development process. Then
we run the model for two scenarios: using this practice in the whole process and using
it in specific phases of the software development process. In both cases, the results
provided evidences of a considerable reduction in terms of rework and cost.
Through a real experience, we expect that the findings of this research not only
help managers to understand the impact of cultural factors that may de-motivate prac-
titioners from supporting some improvements but also assist them in designing and
evaluating the effects of implementing improvement initiatives.
We also believe that similar work can be done in other organizations, more specifi-
cally the adoption of the verification practice in software development processes.
After this study, the organization considered process simulation an important tool to
evaluate the adoption of process changes and it plans to incorporate the verification
practice in its next projects. The future results will permit us to contrast the previous
simulations results with real ones.
Acknowledgements. We would like to thank IBM Rational for supporting this work,
within the IBM Academic Initiative. We also would like to thank Capes Foundation
(grant n. BEX 1893/09-2).
References
1. Börjesson, A., Mathiassen, L.: Making SPI Happen: The IDEAL Distribution of Effort. In:
Proceedings of the 36th Annual Hawaii International Conference on System Sciences,
HICSS 2003, January, 6–9, p. 328. IEEE Computer Society Press, Los Alamitos (2003)
How to Welcome Software Process Improvement and Avoid Resistance to Change 149
2. Baddoo, N., Hall, T.: De-motivators for software process improvement: an analysis of
practitioners’ views. Journal of Systems and Software 66(1), 23–33 (2003)
3. Nasir, M.H.N., Ahmad, R., Hassan, N.H.: Resistance factors in the implementation of
software process improvement project. Journal of Computer Science 4(3), 211–219 (2008)
4. Siakas, K.V.: What has Culture to do with SPI? In: Proceedings of the 28th Euromicro
Conference, EUROMICRO 2002, Dortmund, Germany, September, 4–6, pp. 376–381.
IEEE Computer Society Press, Los Alamitos (2002)
5. Humphrey, W.S.: Why don’t they practice what we preach? Annals of Software Engineer-
ing 6(1/4), 201–222 (1998)
6. Pádua, C., Pimentel, B., Pádua, W., Machado, F.: Transitioning model-driven development
from academia to real life. In: Proceedings of Educators’ Symposium of the ACM / IEEE
9th International Conference on Model Driven Engineering Languages and Systems,
Genova, Italy, October, 1–6, pp. 61–77 (2006)
7. Pádua, W.: A Software Process for Time-constrained Course Projects. In: Proceedings of
the 28th International Conference on Software Engineering, ICSE 2006, Shanghai, China,
May 20–28, pp. 707–710 (2006)
8. Pádua, W.: Quality Gates in Use-Case Driven Development. In: Proceedings of the Fourth
Workshop on Software Quality, 28th International Conference on Software Engineering,
WoSQ 2006, Shanghai, China, May 21, pp. 33–38 (2006)
9. Card, D.N.: Learning from Our Mistakes with Defect Causal Analysis. IEEE Soft-
ware 15(1), 56–63 (1998)
10. Boehm, B., Basili, V.R.: Software Defect Reduction Top 10 List. IEEE Computer 34(1),
135–137 (2001)
11. Siakas, K.V., Balstrup, B.: A field-study of Cultural Influences on Software Process Im-
provement in a Global Organization. In: Proceedings of European Software Process Im-
provement Conference, EuroSPI 2000, Copenhagen, Denmark, November 7–9 (2000)
12. Wong, B., Hasa, S.: Cultural Influences and Differences in Software Process Improvement
Programs. In: Proceedings of the 6th International Workshop on Software Quality, WoSQ
2008, Leipzig, Germany, May 10, pp. 3–10 (2008)
13. Hofstede, G., Hofstede, G.-J.: Cultures and Organizations: Software of the Mind, 2nd edn.
McGraw-Hill, New York (2004)
14. Kellner, M.I., Madachy, R.J., Raffo, D.M.: Software process simulation modeling: Why?
What? How? Journal of Systems and Software 46(2/3), 91–105 (1999)
15. Christie, A.M.: Simulation in support of CMM-based process improvement. Journal of
Systems and Software 46(2/3), 107–112 (1999)
16. Raffo, D.M., Wakeland, W.: Moving Up the CMMI Capability and Maturity Levels Using
Simulation. Technical Report CMU/SEI-2008-TR-002. Software Engineering Institute,
Carnegie Mellon University, Pittsburgh, PA (2008)
17. Raffo, D.M., Vandeville, J.V., Martin, R.H.: Software process simulation to achieve higher
CMM levels. Journal of Systems and Software 46(2/3), 163–172 (1999)
18. Madachy, R.J.: A software project dynamics model for process cost, schedule and risk as-
sessment. Doctoral Thesis. Department of Industrial and Systems Engineering, University
of Southern California, Los Angeles, CA (1994)
The Incremental Commitment Model Process Patterns
for Rapid-Fielding Projects
1 Introduction
The growing diversity of software systems (requirements-driven, Non-
Developmental-driven, services-driven, learning-driven, qualities-driven, systems of
systems) has made it clear that there are no one-size-fits-all processes for the full
range of software systems. Rapid Fielding is becoming an increasingly important
software system objective. To fit into market windows and respond to competition,
several possible process patterns can be used. Some process models are being devel-
oped that provide specific evidence-based and risk-based decision points. One of the
most thoroughly elaborated of these models is the Incremental Commitment Model
(ICM) [1]. To quickly select and follow the appropriate process pattern helps the
development to finish the project faster and more efficiently. Overall characteristics
of each process pattern and the ICM decision points have been defined on a general-
experience basis [2] but process guidelines and process selection decision criteria are
lacking on the ability to help produce a viable process decision early in the life cycle.
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 150–162, 2010.
© Springer-Verlag Berlin Heidelberg 2010
The Incremental Commitment Model Process Patterns for Rapid-Fielding Projects 151
Fig. 1. Net Centric Services Usage in Software Development Projects in USC Software Engi-
neering Class
This paper presents three new software development processes, for Architected
Agile, Use single NDI, and Services-Intensive, and one updated software develop-
ment process, NDI-intensive, which are extended from Yang and Boehm’s COTS-
Based Application Development guidelines (CBD) [8] by applying the risk-driven
approach of the ICM and feedback from Bhuta’s empirical analysis on COTS interop-
erability assessment [9]. This paper also reports the current rapid-fielding software
development process investigation and analysis, the decision driver of process pattern
selection, and the software development process guidelines for 4 process patterns
which are implemented and represented in terms of the electronic process guide.
This paper is organized as followed: Section 2 contains background information of
the ICM, the differences between NDI and NCS, and USC Software Engineering
course. Section 3 provides information about the nature of rapid-fielding projects,
process decision drivers, process selection graph, and the ICM EPG. Section 4 con-
tains preliminary empirical project results on process selection and on the results of
process usage. Section 5 provides conclusions and future work.
152 S. Koolmanojwong and B. Boehm
2 Background
The ICM [2] is a new generation process model. ICM covers the full system devel-
opment life cycle consisting of the Exploration phase, Valuation phase, Foundations
phase, Development phase, and Operation phase. ICM has been evaluated to be a
flexible but robust framework for system development [2]. The core concepts of the
ICM include 1) commitment and accountability of system stakeholders, 2) success-
critical stakeholder satisficing, 3) incremental growth of system definition and stake-
holder commitment, 4) concurrent engineering, 5) iterative development cycles, and
6) risk-based activity levels and milestones. One of the main focuses of the ICM is
feasibility analysis; evidence must be provided by the developer and validated by
independent experts. The ICM combines the strengths of various current process
models and limits their weaknesses. The ICM, like the V-Model [10], emphasizes
early verification and validation, but allows for multiple-incremental interpretation
and emphasizes concurrent rather than sequential engineering. Compared to the Spiral
Model [11], the ICM also focuses on risk-driven activity prioritization, but offers an
improvement by adding well-defined in-process milestones. While ICM, RUP, and
MBASE [12] perform concurrent engineering that stabilizes the software process at
anchor point milestones, ICM also supports integrated hardware-software-human
factors oriented development. Comparing with Agile methods [3], ICM embraces
adaptability to unexpected change and at the same time allows scalability. In [2], it
shows how the ICM can be applied to a broad spectrum of software-intensive systems
processes, which are categorized into 12 process patterns as follows: Use Single NDI,
Agile, Architected Agile, Formal Methods, Hardware with embedded Software com-
ponent, Indivisible Initial Operational Capability, NDI- intensive, Hybrid agile/ plan-
driven system, Multi-owner system of systems, Family of systems, Brownfield, and
Services-Intensive.
For the small e-services projects developed in our project course, four of the 12 proc-
ess patterns of the ICM [15] predominate:
• Architected Agile - For less than 80 agile-ready-people team size and a fairly ma-
ture technology project, agile methods can be scaled up using an Architected Agile
approach emphasizing early investment in a change-prescient architecture and all
success-critical-stakeholders team building [2]. The Valuation phase and Founda-
tions phases can be brief. A scrum of Scrums approach can be used in the
Development phase. For example, a database application project that majority of
the implementation tasks are building the right architecture and custom coding.
• Use Single NDI – When an appropriate NDI (COTS, open source, reuse library,
customer-furnished package) solution is available, it is an option to either use
the NDI or develop perhaps a better version by oneself or outsource such a
The Incremental Commitment Model Process Patterns for Rapid-Fielding Projects 153
development which generally incurs more expense and takes longer to begin capitaliz-
ing on its benefits. On the other hand, an NDI may come with high volatility, complex-
ity, or incompatibility. Major effort will then be spent on appraising the NDI.
For example, an accounting system that could be totally satisfied by using a single
ready-made accounting management software.
• NDI-Intensive –NDI-Intensive system is a system where 30 - 90% of end-user
functionality is provided by NDI [4]. A great deal of attention goes into appraising
the functionality and interoperability of NDI, effort spent on NDI tailoring and In-
tegration, and NDI upgrade synchronization and evolution [14]. For example, a
website development project that majority of the functionalities are provided by one
or more read-made software, but needs additional custom coding, tailoring or glue
coding.
• Services-Intensive –Net Centric Services support community service organizations
in their online information processing services such as donation, communication or
their special interest group activities such as discussion boards, file sharing, and
cloud computing. Similar to NDI-Intensive, the focus goes to appraising the
functionality of the available services and tailoring to meet needs. For example, a
website development project that some functionalities are provided by other online
services and may need additional custom coding, tailoring or glue coding.
The proposed four process patterns of software development processes in ICM are
developed by combining strengths from several development processes. The Archi-
tected Agile case balances a plan-driven approach in building up the steady architec-
ture and an agile-driven approach in iterative incremental and frequent delivery as
practice in Scrum [16] or Agile Unified Process [3]. The Software Engineering Insti-
tute (SEI) CMMI-COTS [17] and USC-CSSE COTS-Based Development (CBD)
Guidelines [8] provide strong foundations for Use Single NDI, NDI-Intensive or
COTS-Based systems (CBS). Regarding Services-Intensive, most of the processes,
including CMMI-SVC [6], cover only the process of how to develop and maintain
web services. None of them are able to pick, choose and use available online services.
Although the Services-Intensive case is similar to the NDI-intensive case, we
found that the differences between NDI and NCS shown in Table 1 below make the
CBD guidelines an imperfect fit with a Services-Based development process.
Table 2 summarizes the overview characteristics of rapid fielding projects in each
process pattern. As shown in Fig. 2, different risk patterns for each project yields
different software processes. For example, in the second case, when in the Explora-
tion phase, the developers spend a good amount of effort to explore and find a perfect
NDI that satisfied all the win conditions, the development team could skip the Valua-
tion and Foundations phase. In the third and the fourth case, if the development team
found possible NDIs/NCSs in the Exploration phase, the team could spend more ef-
fort evaluating the NDIs/NCSs, prioritizing their win-conditions or constraints. On the
other hand, since NDIs/NCSs will provide a majority of the end product features, the
development team could spend less time in Foundations and Development-related
efforts. But NDI and NCS have different risk patterns; hence they need different
process to follow.
154 S. Koolmanojwong and B. Boehm
In the keystone two-semester team project graduate software engineering course se-
quence CS577ab [5] at USC, students learn through experience how to use good
software engineering practices to develop software systems from the Exploration
Phase to the Operation Phase, all within a 24-week schedule. Six on-campus and two
off-campus students team up to develop real-client software system products. All
teams will follow the ICM Exploration Phase guidelines to determine their most ap-
propriate process pattern. Most of the clients are neighborhood non-profit organiza-
tions, small businesses or USC departments. Because of the semester break between
156 S. Koolmanojwong and B. Boehm
Fall and Spring semester, for the Software Engineering class, we added a short Re-
baselined Foundations phase to accommodate the possible changes.
As shown in Fig. 4, a rapid fielding project starts by identifying the objectives, con-
straints and priorities (OC&P) and exploring all possible software development alter-
natives. If there is no relevant NCS or NDI that would satisfy the OC&Ps and OC&Ps
cannot be adjusted, then this project will proceed to follow the Architected Agile
process. The first three phases will be focused on building a change-prescient archi-
tecture and user/developer/customer team building. In the development phase and
operation phase, with the agile techniques, the developers welcome and manage
change through a product backlog, planning capabilities for multiple future incre-
ments, based on stakeholder priorities.
On the other hand, if there are possible NDI/NCSs that satisfy the project’s
OC&Ps, the project will proceed with either Use Single NDI, Services-Intensive, or
NDI-Intensive process pattern. In the Exploration and Valuation phase, spend a good
amount of time in assessing NDI/NCS candidates. If multiple candidates are required,
one needs to analyze their interoperability. When the selected NDI/NCS provides full
or partial functionalities, the foundations phase could be skipped or shorten. In the
development phase, perform custom coding, tailoring or glue coding as required.
For all rapid-fielding projects, synchronization and stabilization are required for
each and between development cycles. Continuous verification and validation is also
required for quality assessments.
Having properly-defined software process models is essential, but the ability to effec-
tively communicate those models to the software engineers is also important. For Soft-
ware Engineering class at USC, we have been using the IBM Rational Method Composer
(RMC) to develop an Electronic Process Guide (EPG) for the ICM. The ICM EPG en-
ables process users/ students to understand the overall process as well as specific areas of
focus. The ICM EPG as shown in Fig. 4, describes the software development process by
providing guidance in multiple-view representations, which are role-based representa-
tion, activity-based representation, chronological event-based representation, and artifact-
based representation. Moreover, additional artifact templates and supplementary guides
have been demonstrated to speed up the users’ learning curve and support them in their
development process [18,19].
In order to select the appropriate process pattern, the development team will use the
16 decision drivers presented in Table 3 to evaluate the project status and map it with
the possible maximum-minimum boundary range of each possible process pattern.
The “Importance” attribute, whose values are 1-Low, 2-Medium, 3-High, will act as a
tie breaker to support decision making in selecting the best fit process pattern. The
value of project status ranges from 0-Very Low to 4-Very High.
Fig. 5 shows an example of a website development team. The team found a possi-
ble content management system NDI, but it does not satisfy all of the capability win
conditions. The team rates the project status based on 16 decision drivers, the result is
shown in the blue line. The background block diagram is the max-min boundary of
NDI-Intensive project defined in Table 3. The red underline represents High Impor-
tance level, while green dashed underline represents Low Importance level.
When the blue point lies on gray box, it shows that the project status conform to
the process pattern on that driver. As a result, the decision driver shows that this team
could follow the NDI-intensive software development process. On the other hand, as
shown in Fig. 6, if we plot the project status on the Architected Agile process pattern,
8 non-conforming points are found. As a result, comparing Fig. 5and Fig. 6, there are
less non-conforming points on Fig. 5, so that means this project would fit more to the
NDI-Intensive process pattern.
The Incremental Commitment Model Process Patterns for Rapid-Fielding Projects 159
Fig. 5. An example of using decision drivers to map project status with the NDI-Intensive
process pattern
Fig. 6. An example of using decision drivers to map project status with the Architected Agile
process pattern
4 Preliminary Results
In Fall 2009, Software Engineering students at USC have used the ICM EPG to guide
their rapid-fielding software development projects. Once the students form their teams
and select the projects, teams will follow the ICM Exploration phase process to de-
termine their most appropriate process patterns. 28% of projects are able to deliver in
one semester, instead of two semesters comparing to 12.5% from previous year. Cli-
ents are highly satisfied with the project results.
With the data from effort reporting system, Table 4 shows that with the ICM EPG,
process users and developers spent less time in learning about the process and with the
help of templates and supporting documents, they saved more than 20 hours per per-
son in documentation effort.
The result of our experiment also shows that the EPG is very effective in terms of
communicating the process model to the developers. With the role-oriented process
information and rich set of notations such as process element icons and graphical
representations, process users and developers found that it is easy for them to learn
about their roles, responsibilities, activities, and functionalities. For the process engi-
neers, IBM Rational Method Composer (RMC) supports process reuse, tailoring and
configuration. The ICM EPG can be extended to support process management be-
cause RMC is fully interoperable with various project management tools.
160 S. Koolmanojwong and B. Boehm
Table 4. Comparison of effort between paper-based guidelines and the ICM EPG
Comparing the result of the teams’ process selection by using process decision drivers
with the experts’ opinion from the architecture review board, 8 out of 14 teams se-
lected the right process pattern from beginning of the valuation phase. Three out of 14
teams selected wrong process pattern because of unclear project scope, but the teams
followed the right process after further exploration, more prototyping or found avail-
able NCS/NDI. Two out of 14 teams faced the minor changes, so they have to change
the process by the end of Valuation phase. One project followed the right process
pattern but found that the client’s requirement is infeasible within a defined budget
and schedule, hence the team proposed the feasible solution and had to switch the
process 3 times. In addition, based on the survey result, the process decision drivers
help the teams to select the most suitable process pattern. Moreover, when the scope
or status of the project changes, the development teams found it useful to reconfirm or
to re-identify the process pattern with the process decision drivers.
4.3 Limitations
This section discusses possible validity threats and ways in which the threats could be
reduced.
• Inconsistent Effort Reporting: It is possible that there might be inaccurate efforts
reported, so 2 forms of effort reporting will be used.
• Non-representativeness of subjects: On-Campus students might not represent soft-
ware engineers in the industry. However, the clients and off-campus students are
full-time working professionals.
• Learning Curve: Providing tutorials and discussion sessions in order to build the
foundations for all participants.
• Non-representativeness of projects: Although conforming to fixed semester sched-
ules, this is to some degree representative of fixed-schedule industry projects. The
projects are small e-services applications, but are developed for real clients, and
used the same NDI/NCS. Moreover, the process guidelines and decision drivers are
cross checked with experts in the field for enterprise-level compatibility.
rapid-fielding project. Four process guidelines are developed in the form of the ICM
EPG. The result of the experiment shows that, at least for small, real-client e-services
project, the EPG is very effective in terms of communicating and learning the process
model and speeding up the development process. The process pattern decision drivers
help the development teams produce a viable process decision very early in the life
cycle process with faster time to market and high customer satisfaction. Continuing
efforts are underway to evaluate the efficacy of the ICM by analyzing risk patterns,
incidence of direction changes at reviews relative to outcomes and team performance
based on team grading and compare to previous years data.
References
1. Pew, R.W., Mavor, A.S.: Human-System Integration in the System Development Process:
A New Look. National Academy Press, Washington (2007)
2. Boehm, B., Lane, J., Koolmanojwong, S.: A Risk-Driven Process Decision Table to Guide
System Development Rigor. In: Proceedings of the 19th International Conference on Sys-
tems Engineering, Singapore (July 2009)
3. Agile, Principles behind the agile manifesto,
http://agilemanifesto.org/principles.html
4. Yang, Y.: Composable Risk-Driven Processes for Developing Software Systems from
Commercial-Off-The-Shelf (COTS) Products: PhD Dissertation, Department of Computer
Science, University of Southern California (December 2006)
5. USC CSCI577ab Software Engineering Course,
http://greenbay.usc.edu/csci577/fall2009/site/index.html
6. CMMI for Services Version1.2,
ftp://ftp.sei.cmu.edu/pub/documents/09.reports/09tr001.doc
7. ProgrammableWeb, http://www.programmableweb.com/mashups (accessed on
11/9/2009)
8. Yang, Y., Boehm, B.: COTS-Based Development Process guidelines,
http://greenbay.usc.edu/csci577/spring2007/site/guidelines/
CBA-AssessmentIntensive.pdf (accessed 08/24/07)
9. Bhuta, J.: A Framework for Intelligent Assessment and Resolution of Commercial-Off-
The-Shelf Product Incompatibilities, PhD Dissertation, Department of Computer Science,
University of Southern California (August 2007)
10. V-Model Lifecycle Process Model,
http://www.v-modell.iabg.de/kurzb/vm/k_vm_e.doc
(accessed on 10/9/2009)
11. Boehm, B.: A Spiral Model of Software Development and Enhancement. IEEE Com-
puter 21(5), 61–72 (1988)
12. Boehm, B.: Anchoring the Software Process. IEEE Software 13(4), 73–82 (1996)
13. Boehm, B., Turner, R.: Balancing agility and discipline: a guide for the perplexed. Addi-
son-Wesley, Boston (2004)
14. Li, J., Bjoernson, F.O., Conradi, R., Kampenes, V.B.: An empirical study of variations in
COTS-Based Software Development Processes in Norwegian IT industry. Journal of Em-
pirical Software Engineering 11(3), 433–461 (2006)
15. Boehm, B., Lane, J., Koolmanojwong, S.: A Risk-Driven Process Decision Table to Guide
System Development Rigor. In: Proceedings of the 19th International Conference on Soft-
ware Engineering, Singapore (July 2009)
162 S. Koolmanojwong and B. Boehm
16. Rising, L., Janoff, N.: The Scrum Software Development Process for Small Teams. IEEE
Software (July/August 2000)
17. CMMI for COTS-based Systems,
http://www.sei.cmu.edu/publications/documents/03.reports/
03tr022.html
18. Koolmanojwong, S., et al.: Comparative Experiences with Software Process Modeling
Tools for the Incremental Commitment Model, In: USC CSSE Technical Report 2007-824
19. Phongpaibul, M., Koolmanojwong, S., Lam, A., Boehm, B.: Comparative Experiences
with Electronic Process Guide Generator Tools. In: Wang, Q., Pfahl, D., Raffo, D.M.
(eds.) ICSP 2007. LNCS, vol. 4470, pp. 61–72. Springer, Heidelberg (2007)
A Repository of Agile Method Fragments
1 Introduction
In recent years, agile methods for software development have seen widespread adoption
and experimentation. Agile methods are appealing partly due to their inherent character-
istics such as simplicity, and partly due to problems with traditional methods [1, 2]. The
prevalence of agile methods has provided a suitable test-bed for software researchers, to
run empirical studies, and analyze different aspects of agile methods in real trials. Every
year, a considerable number of empirical studies emerge, each addressing specific con-
cerns of deploying agile methods in various projects/organizations. These studies, which
mostly take macro-process view to the research of agile processes [3], form a large body
of knowledge about various fragments (constituent pieces) of agile methods. For in-
stance, a simple title-based search in just the IEEE Explorer for "Pair Programming"
returns over 50 results, mostly reporting on empirical studies.
Despite the abundance of empirical data for agile methods, there is no systematic
support to take advantages of that knowledge, to make it more usable and accessible.
This issue becomes more serious when we consider the way that software industry
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 163–174, 2010.
© Springer-Verlag Berlin Heidelberg 2010
164 H.C. Esfahani and E. Yu
deploys agile methods. For many project managers agile software development means
to deploy a number of agile method fragments, and then try as hard as possible to get
the works done. Based on a recent survey, around 60% of software companies do not
follow any particular method, and just deploy some techniques and tools introduced
by different methods [4]. This approach exposes a software company to the risk of
adopting inappropriate development methods, which is a serious threat to the success
of software projects [5]. Most likely, the complicated solutions of process (or method)
engineering would not be acknowledged by many agile software companies. But, a
promising solution can be a one that facilitates their access to the experience of other
practitioners in using different agile method fragments, and helps them decide on the
appropriateness of their own process.
This paper introduces a structured repository of agile method fragments (AMFs).
For each AMF, the repository entry states the objectives that the AMF aims to con-
tribute to, and a set of requisites that are needed for its success. The knowledge of this
repository has been gathered through systematic review [6] of empirical studies on
agile methods, i.e., the objectives and requisites of each AMF have been synthesized
by reviewing results from experiments or reported experiences, not from the motiva-
tional description of an agile method. The repository is evidence-based, as it provides
contextual evidences from empirical studies, which explain situations in which a
particular objective or requisite of an AMF had been met or not. In addition, this re-
pository offers a visualization of AMF data, using a goal-oriented notation.
Taking the evidence-based approach for populating this repository helped us to
present some interesting results about the contextual aspects of AMFs. There exist
numerous reports that studied the impact of different project contexts on the success
or failure of AMFs. For instance, this repository contains project contexts in which
Pair Programming (as a sample AMF) either helped or impeded a project to be on-
time to market. The evidential knowledge of this repository can be useful for project
managers, in order to come up with more appropriate sets of method fragments for
their particular projects/organizations. It can also facilitate the introduction of a new
agile method to an organization, and the reuse of its prior successful and unsuccessful
experiences. The repository is being tested at a major telecom company in a software
process improvement initiative.
2 Related Work
A number of process engineering frameworks include method repositories as compo-
nents. For instance, the Open Process Framework (OPF) [7] is a process (method)
engineering framework, which contains a repository of method fragments. Similarly,
the Eclipse Process Framework (EPF) [8] is an Eclipse-based process engineering
tool, which also provides a number of libraries for some well-known methodologies,
e.g., RUP and Scrum. These repositories are mainly aimed at describing the constitut-
ing elements of a methodology by breaking it down to a number of fragments, and
then describing each method fragment in terms of its type, functionality, needed
skills, relevant roles, etc. However, these repositories are not concerned about pre-
senting actual evidences about the success or failure of various method fragments
under different project situations.
A Repository of Agile Method Fragments 165
3 Research Method
The research method of this study is the systematic review [6] of published literature
on empirical studies of agile methods. First, to clarify the objective of systematic
review, we specified our research questions. Then, we set up a review protocol that
specified the criteria of study selection, the strategy of data extraction, and the ap-
proach of analyzing and aggregating the extracted data, and synthesizing the results.
The final stage of systematic review is the reporting of results, which we publish in
textual table format and as graphical models.
Systematic reviews should be targeted towards clear research questions [6, 9]. In this
research we were looking for answers to the following questions:
1. What are the actual beneficial outcomes achieved by an agile method fragment?
These will be treated as objectives in the repository entry so that fragments can be
retrieved according to desired outcomes. The description of a method fragment
typically comes with promises about its benefits, in terms of quality goals that it
contributes to. For instance, "enhanced quality of communication" is one of the
major promises of many agile method fragments. But, is that really the case? Is
there any evidence for this claim? Do method fragments always contribute posi-
tively to their objectives, or can they produce adverse contributions under specific
situations?
2. What unanticipated side-effects have been discovered when the method fragment
was put to use under different conditions? These issues will be treated as objectives
to which a method fragment negatively contributes. For instance, Pair Program-
ming (PP) can result in the reduction of Lines-of-Code (LOC) per programmer per
month. Thus, PP makes a negative contribution to the productivity of individuals
(with respect to the LOC factor). In this research we are looking to identify such
side-effects, as well as evidences for their appearance in empirical studies.
166 H.C. Esfahani and E. Yu
3. What are the necessary conditions for the method fragment to be used success-
fully? These will be treated as requisites in the repository entry of each AMF. The
description of every method fragment typically suggests a number of requisites for
its successful enactment. However, these requisites might not be attainable or
necessary in all project situations. For instance, "equal engagement of pair in pro-
gramming" is one of the requisites of "Pair Programming". But, is it always attain-
able? In what situations is this requisite most likely to be denied? Furthermore,
empirical experience with an AMF may have revealed additional requisites.
Here, we stress the importance of finding actual evidences for the claimed objectives
and requisites of AMFs. Such evidences will improve the reliability of the repository,
provided that they correctly represent the situation of source evidences. Thus, we had to
carefully consider those particular aspects of the reviewed empirical studies (pro-
ject/organizational factors), which could impact the enactment of their AMFs. Follow-
ing this approach, the final repository will contain a considerable amount of situational
experiences of AMFs, which can be used in process improvement initiatives.
We started the research by selecting 20 agile method fragments for consideration. For
each method fragment we looked up published empirical studies that explained the
enactment of that method fragment in a particular project/organization. We searched
the major collections of computer science publications, including IEEE Explorer,
ScienceDirect, SpringerLink, and ACM Digital Library. We used search queries that
contained the name of the method fragment, the name of the originating agile method,
and some phrases for retrieving empirical studies, such as: Case study, Experiment
Report, Ethnography, or Survey.
In addition to published empirical studies in academic journals and conferences,
we also considered articles from agile web forums as complementary sources for our
reviewed empirical studies. Every selected paper or article was then evaluated with
respect to the following items: 1- reliability of reported data; 2- reproducibility of
study; 3- relevance to the research questions; 4- quality of situation description (con-
text for enacting AMF). Only publications that meet all of the above criteria are used
as sources, and are listed as references.
While reviewing the selected empirical studies, we were looking for either explicit or
implicit answers to our research questions. For data extraction, we obtained answers
for the following questions for every reviewed paper:
• What is the type of this empirical study? (Controlled Experiment, Survey, Inter-
view, Experience report, Ethnography, Case Study)
• What agile method fragments have been investigated in this study?
• What are the distinguishing characteristics of the context of this study? (Hu-
man/organization factors, such as cultural issues, structure of development teams,
and the distribution of expertise; project related factors, such as complexity and
market pressure)
A Repository of Agile Method Fragments 167
We did not begin the systematic review with a preconceived list of objectives or req-
uisites, since one of the purposes of the research was to identify the evidential objec-
tives and requisites of AMFs. However, after reviewing and analyzing a number of
publications for each AMF, such lists began to emerge. Afterwards, we were consid-
ering those lists while reviewing a paper, and also reread some of the initially re-
viewed papers.
The extracted data from reviewed empirical studies were then analyzed and aggre-
gated as units of knowledge for every method fragment. Data analysis was performed
in order to:
comparison of method fragments, and provide insights for further empirical studies to
uncover the potential relations between agile method fragments and process objectives
or requisites. Table 4 shows a sample generic snapshot.
An appropriate modeling paradigm for visualizing the collected data of this study
should be capable of representing method objectives and requisites, also the contribu-
tion relations of AMFs to their objectives and their dependency relations to their req-
uisites. These features have been well-addressed in Goal-Oriented Requirements
Engineering (GORE), where goal models are used to express how domain objects
(such as activities and resources) contribute to domain goals [12]. In this research we
use the i* goal-oriented modeling framework for visualizing the collected knowledge
of AMFs. Table 1 describes a subset of i* modeling elements, which are used for
visualizing the objectives and requisites of method fragments. Fig. 1 shows an exam-
ple of AMF visualization. Further description of our approach in modeling method
fragments, and their application in process engineering are presented in [13].
4 Results
As mentioned before, the results of data extraction and analysis for each agile method
fragment are aggregated into an objectives table and a requisites table. Due to space
limits, we cannot present the results of our study for all of the reviewed method frag-
ments. As an example, this section presents a subset of the knowledge for two AMFs:
"Pair Programming" and "Daily Scrum Meeting". The complete set of tables are pub-
licly accessible at: http://www.cs.utoronto.ca/km/MFR/MF.htm.
Table 2 shows a subset of the objectives of the agile method fragment "Pair Pro-
gramming" (PP). It presents different types of contributions that PP might make to its
objectives, along with references to the empirical studies which have investigated
A Repository of Agile Method Fragments 169
these issues. The situations in which the stated contributions had occurred are briefly
described. For instance, it shows that Pair Programming, in general cases (represented
as "In General"), contributes strongly and positively to the objective "Reduced De-
fects in Code". It also provides evidence from a controlled experiment, which quali-
fies this contribution with more specific situational information (complex system with
junior developers).
Table 2. A subset of major and minor objectives of "Pair Programming", along with situational
evidences for contributions that this AMF can make towards them
Major Contr-
Minor Objective ibution
Study Situation
Objective
Increased Pairs with heterogeneous
+ [14, 15]
productivity personality profile
(LOC/Month/develo Pairing Professional Developers,
-- [16, 17]
per) or developers similar expertise
Improved Design
++ [18-20] In General
Quality
Improved Effectiveness
Objectives of a method fragment can be classified into two categories. The first
category are those to which the method fragment makes situation-independent contri-
butions, i.e., the change of context does not change the contribution type. For exam-
ple, in all of the reviewed empirical studies, Pair Programming is reported to have a
positive contribution to the objective "Faster Problem Solving". The second category
is for objectives to which the contribution of method fragment is situation-dependent,
i.e., a change in context can alter the type of contribution. For instance, the default
contribution of "Pair Programming" to the objective "Be on-time to market" is posi-
tive, but [23] reports that Pair Programming in the context of a large project with
limited number of developers and high market pressure can contribute negatively to
that objective. It should be noted that Table 2 and 3 only list subsets of the empirical
studies that we reviewed for the AMFs, for lack of space.
For each requisite, the table presents a number of empirical studies which had re-
ported the impact of that requisite on the success or failure of daily scrum meetings.
This table briefly describes the significant situational factors of the referenced studies,
and mentions the achievement (or denial) status of requisites in those situations. The
achievement status of requisites is specified either as: Denied (D), Partly Denied
(PD), Partly Satisfied (PS), or Satisfied (S).
Table 3. A subset of major and minor requisites of "Daily Scrum Meeting", and their achieve-
ment status in a number of referenced empirical studies
Major Achiev.
Minor Requisite Study Situation
Req. Status
Scrum Master failure in
[25,
Skilled Leadership PD organizing meetings and running
26]
them quickly
Real time Information [27] Distributed Development :
D
Passing [28] Different time zones
Meeting Be effective
One of the results of this study is a set of generic snapshots, which summarizes the
relations of all (or several) AMFs to all (or several) of the identified objectives or
requisites. These snapshots are called generic because they are not bound to any par-
ticular AMF, and typically represent a number of AMFs. For example, Table 4 shows
A Repository of Agile Method Fragments 171
Fig. 1. Visualization of the agile method fragment "Pair Programming", for a subset of its
objectives and requisites
Table 4. Generic snapshot: A list of minor objectives of five agile method fragments, all related
to the major objective "Improved Efficiency", and the contributions of the AMFs to them
Pair Prog.
fice Space
MoSCoW
Customer
Open Of-
Boxing +
Meeting
Major
On-site
Scrum
Minor Objective
Time-
Daily
Objective
Increased productivity
--
(LOC/Month/developer)
Reduced Development Cost -, ?
Improved Design (product) Quality ++ -
Improved Creativity Potential +
Improved Effectiveness
a list of minor objectives of five agile method fragments, all related to the major ob-
jective "Improved Efficiency". Due to space limits we could not present the contribu-
tion of other agile method fragments to these objectives, as well as other major objec-
tives. The online version of the repository contains similar tables for other major
objectives, e.g., "Improved Communication", "Improved Collaboration", "Higher Job
Satisfaction", and so forth. Note that an empty cell in the table does not necessarily
mean that there is no contribution relation between the corresponding agile method
fragment and objective. Further investigations in the empirical studies of software
processes might supply further contribution relations to the table.
References
1. Boehm, B., Turner, R.: Balancing Agility and Discipline, a Guide for the Perplexed. Addi-
son-Wesley, Reading (2003)
2. Cohn, M., Ford, D.: Introducing an Agile Process to an Organization. Computer 36(6),
74–78 (2003)
3. Osterweil, L.: Unifying Microprocess and Macroprocess Research. In: Li, M., Boehm, B.,
Osterweil, L.J. (eds.) SPW 2005. LNCS, vol. 3840, pp. 68–74. Springer, Heidelberg
(2006)
4. Bygstad, B., Ghinea, G., Brevik, E.: Software development methods and usability: Per-
spectives from a survey in the software industry in Norway. Interacting with Com-
puters 20(3), 375–385 (2008)
5. Linda, R., Norman, S.J.: The Scrum Software Development Process for Small Teams.
IEEE Softw. 17(4), 26–32 (2000)
6. Kitchenham, B.: Procedures for Performing Systematic Reviews, TR/SE-0401, Software
Engineering Group, Department of Computer Science, Keele University (2004)
7. Firesmith, D.: Open Process Framework (OPF) (accessible via: date accessed: November
2009)
8. Epf. Eclipse Process Framework Project (EPF) (2006),
http://www.eclipse.org/epf/ (cited June 2009)
9. Kitchenham, B.A., Dyba, T., Jorgensen, M.: Evidence-Based Software Engineering. In:
Proceedings of the 26th International Conference on Software Engineering. IEEE Com-
puter Society, Los Alamitos (2004)
10. Janzen, D.: SEED: Software Engineering Evidence Database, accessible via:
http://evidencebasedse.com/ (date accessed: January 2010)
11. Brinkkemper, S.: Method engineering: engineering of information systems development
methods and tools. Information and Software Technology 38(4), 275–280 (1996)
12. Yu, E.S.K.: Towards modelling and reasoning support for early-phase requirements engi-
neering. In: Proceedings of the Third IEEE International Symposium on Requirements En-
gineering. IEEE Computer Society, Los Alamitos (1997)
13. Chiniforooshan Esfahani, H., Yu, E.: Situational Evaluation of Method Fragments: An
Evidence-Based Goal-Oriented Approach. Submitted to Conference on Advanced Infor-
mation Systems Engineering, CAiSE 2010 (2010)
14. Sfetsos, P., Stamelos, I., Angelis, L., Deligiannis, I.: An experimental investigation of per-
sonality types impact on pair effectiveness in pair programming. Empirical Software Engi-
neering 14(2), 187–226 (2009)
15. Choi, K.S., Deek, F.P., Im, I.: Exploring the underlying aspects of pair programming: The
impact of personality. Information and Software Technology 50(11), 1114–1126 (2008)
16. Pikkarainen, M., Haikara, J., Salo, O., Abrahamsson, P., Still, J.: The impact of agile prac-
tices on communication in software development. Empirical Software Engineering 13(3),
303–337 (2008)
17. O’donnell, M.J., Richardson, I.: Problems Encountered When Implementing Agile Meth-
ods in a Very Small Company. In: Software Process Improvement, pp. 13–24. Springer,
Heidelberg (2008)
18. Cockburn, A.: Selecting a project’s methodology. IEEE Software 17(4), 64–71 (2000)
174 H.C. Esfahani and E. Yu
19. Williams, L., Mcdowell, C., Nagappan, N., Fernald, J., Werner, L.: Building pair pro-
gramming knowledge through a family of experiments. In: Proceedings of 2003 Interna-
tional Symposium on Empirical Software Engineering. IEEE Press, Los Alamitos (2003)
20. Williams, L.: Integrating pair programming into a software development process. In: Pro-
ceedings of 14th Conference on Software Engineering Education and Training Charlotte.
IEEE Computer Society, USA (2001)
21. Mujeeb-U-Rehman, M., Xiaohu, Y., Jinxiang, D., Abdul Ghafoor, M.: Heterogeneous and
homogenous pairs in pair programming: an empirical analysis. In: Canadian Conference on
Electrical and Computer Engineering. IEEE Press, Los Alamitos (2005)
22. Arisholm, E., Gallis, H., Dyba, T., Sjoberg, D.I.K.: Evaluating Pair Programming with Re-
spect to System Complexity and Programmer Expertise. IEEE Transactions on Software
Engineering 33(2), 65–86 (2007)
23. Padberg, F., Muller, M.M.: Analyzing the cost and benefit of pair programming. In: Pro-
ceedings of Ninth International Software Metrics Symposium. IEEE Press, Los Alamitos
(2003)
24. Charles, P., Jan Willem, H.: Using Extreme Programming in a Maintenance Environment.
IEEE Softw. 18(6), 42–50 (2001)
25. Begel, A., Nagappan, N.: Usage and Perceptions of Agile Software Development in an In-
dustrial Context: An Exploratory Study. In: First International Symposium on Empirical
Software Engineering and Measurement, ESEM 2007 (2007)
26. Judy, K.H., Krumins-Beens, I.: Great Scrums Need Great Product Owners: Unbounded
Collaboration  and Collective Product Ownership. In: Proceedings of the 41st An-
nual Hawaii International Conference on System Sciences (2008)
27. Sutherland, J., Viktorov, A., Blount, J., Puntikov, N.: Distributed Scrum: Agile Project
Management with Outsourced Development Teams. In: 40th Annual Hawaii International
Conference on System Sciences (2007)
28. Berczuk, S.: Back to Basics: The Role of Agile Principles in Success with an Distributed
Scrum Team. Agile (2007)
29. Larman, C., Vodde, B.: Top Ten Organizational Impediments (2009)
30. Fredrick, C.: How Douglas County, CO Cut A Project Timeline In Half. Agile Journal
(2007)
31. Lewis, J., Neher, K.: Over the Waterfall in a Barrel - MSIT Adventures in Scrum. IEEE
Publications, Agile (2007)
OAP: Toward a Process for an Open World
Yuanzhi Wang
1 Introduction
The variety, complexity, and dynamic of software-intensive systems have been
significantly increased because of advances in technologies and society in the open
environments of present era. Owing to increasing complexities, volatility, and
uncertainty involved in an open world, the well-known gap between computing
capabilities and usage demands has reached a critical level that hinders successful
engineering of complex systems in such an environment [1]. Therefore, it becomes
pressingly necessary to assess the problems of current engineering approaches
and to explore different ways to address such a great challenge.
In this paper, an engineering process called Organic Aggregation Process, or
OAP for short, is proposed. It consists of a set of inter-connected activities that
are organised in a coherent and flexible hierarchy. This structure aims to form a
coherent linkage between human intellectual efforts and practical reality. That is,
a path from human perception of complex problem situations to sensible desired
engineering ends, and to practical engineering actions that eventually improve
the situations. Such a linkage can be aggregated organically through a capacity
reuse mechanism, in support of organic agility. Moreover, a model-driven method
and some supporting tools are integrated into OAP, and facilitate management
of its engineering activities in a dynamically complex environment.
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 175–187, 2010.
c Springer-Verlag Berlin Heidelberg 2010
176 Y. Wang
The origin of a notion of processes can be traced back to ancient Greek philoso-
pher Heraclitus, who held that the essence of the world is not material substance
of things but rather a continuous natural process, or “flux” in his term [2]. White-
head’s definition of a process is “a sequentially structured sequence of successive
stages or phases” [3]. In the context of software engineering, Fuggetta defined a
process as “the coherent set of policies, organisational structures, technologies,
procedures, and artefacts that are needed to conceive, develop, deploy, and main-
tain a software product” [4]. These definitions all involve a concept of life-cycle,
and some composing elements with a structure of certain shape or format.
Similarly, Royce’s influential waterfall model [5] is a software process that
consists of a sequence of activities such as requirements analysis, design, and im-
plementation. This approach gradually evolved into more iterative, incremental,
and evolutionary models, sometimes called Iterative and Incremental Develop-
ment (IID) [6], such as spiral model, rapid prototyping, Rational Unified Process
(RUP), and more recently, agile methodologies. John Boyd’s Observation, Orien-
tation, Decision, and Action (OODA) loop, as a decision-making process model,
was designed to cope with highly dynamic and competitive environments by ac-
celerating a person’s own loop and weakening opponents’ abilities to complete
their loops at faster paces [7]. This model has been successfully adopted in critical
and dynamic environments such as military battles and business competition.
This paper, however, argues that these approaches are inadequate to deal with
great challenges of complexity, volatility, and uncertainty, which are commonly
involved in open environments. For example, OODA model and its followers,
in general, are too abstract and coarse-grained, and lack some fundamental ele-
ments in an engineering context, such as clearly defined activities at detail level,
their boundaries and relationships, and supporting tools. It provides neither
guidelines with regards to internal structures and content, nor representational
OAP: Toward a Process for an Open World 177
typical means, being generalised from proven examples, to achieve the desired
result for a type of activities. It encapsulates typical engineering capacities in
the form of service. For instance, based on certain Perception, a particular ac-
tivity of Problem Situation recognises and represents a problematic situation.
Such capacity is encapsulated in an Epitome called “Epitome 1” as depicted
in figure 2. It can be used to produce the same representation of problems un-
der the same situation, for future reference or aggregations. Such an epitome is
created by generalising a proven instance of example, which is a process called
Epitomisation. A collection of Epitomes provides a Horizontal Aggregation of
valuable engineering assets, just like a collection of Abstraction forms valuable
aggregation of knowledge about the Reality.
4 A Case Study
In order to obtain insights about the effect and value of OAP by applying its
concepts, methods, and tools in a real-world setting, a controlled case study is
conducted in the context of the First Home Saver Accounts (FHSA), a scheme in-
troduced by the Australian federal government that aims to help young residents
to purchase their first homes, by providing subsidy of government contributions
and tax benefits. The purpose is to implement a proof-of-concept, as opposed
to proof-of-performance, to demonstrate that the process meets its objectives in
general, that is, a process mechanism to deal with a problematic situation in a
complex and dynamic environment. A proof-of-concept system is produced using
IPEOAP toolkit. Apache Tomcat web server is used to host the generated web
content. Apache Axis2 Web Service Engine and ODE are used to host generated
Java web services and BPEL. JBoss provides application server that contains
some Java business logic. Derby Database is used for database implementation.
The OAP process started with Abstraction, which includes modelling of cur-
rent business domains such as structure and process of business. UML Class Di-
agrams are used to model static information structure such as Customer, Bank
Account, Clerk. UML Activity Diagrams are used to model business behaviour
and workflow such as managing FHSA account. These models are created and
managed within IPEOAP using Eclipse UML2 tools.
competition” that has two subordinate Purposes: “Satisfy customer” and “In-
crease profit”. It also has a high level Problem, “Not support FHSA”, which
violates the above Purposes. This problem is caused by a Fact : “New Gover-
ment Scheme”, that is, the government just officially released the consolidated
version of policy recently and the organisation simply does not have the resources
to react promptly in an uncertain market. This Problem is also caused by some
other Problems. For example, many customers do not recognise the benefits of
FHSA. Moreover, many of them are not eligible and the proportion of these
customers remain unknown due to lack of information. Therefore, as a strategic
management decision, no business support for FHSA was provided at the time of
FHSA announcement. This directly leads to some subordinate Problems: “Exist-
ing customers cannot create their first FHSA account”, “Cannot manage existing
fund”, and “Cannot terminate existing fund”. This scenario exemplifies a typical
complex problem situation that modern organisations are often involved in a dy-
namic marketing and legislation environment. As shown in the figure, IPEOAP
provides a graphical editing tool that enables efficient creation and manipula-
tion of PSM elements and their relationships. Moreover, using its transformation
engine, the PSM is automatically transformed into HTML documents with in-
formative details that describe the situation, which help business stakeholders
to explore, understand, and communicate the problem situations.
The value of such a PSM lies in its function of facilitating not only discovery
and communication of complex business situations and concerns at early stages
of process life cycle, but also systematic derivation of relevant and sensible en-
gineering artefacts during successive activities, such as DEM. It hence provides
184 Y. Wang
5 An Analysis of OAP
The case study shows some positive results. Firstly, the modelling mechanism in
OAP, such as PSM and DEM, provides an effective formalism to explore, rea-
son about, and rationalise problematic situations, and to form rational desired
ends to improve such situations. These important higher level intellectual efforts
justify and foster engineering design decisions and design of plans. Exploration
and clarification of problematic situation using PSM helps to identify higher
OAP: Toward a Process for an Open World 185
level purposes and problems with high priority that were not initial recognised
in a local problem scope. For example, a lack of FHSA business support is later
confirmed as a strategic problem due to its violation with high level critical pur-
poses. Secondly, the interrelationship between OAP activities are systematically
supported by linkages between various higher level models, with aid of integrated
tool environment that fosters and automates access to unified model repository
and transformation engines. Thirdly, improvement to engineering productivity is
observed although not formally measured against other approaches. The proof-
of-concept implementation enables encapsulation of knowledge and engineering
capacities, and allows automatic generation of concrete documentation and sys-
tems based on a series of interconnected OAP models. Therefore, the majority
of engineering efforts is designing and managing new abstract models, such as
PSM and DEM as illustrated in this section. With support from IPEOAP, these
modelling and transformation tasks were achieved in a short period of time.
This approach is different from Boyd’s OODA loop in the sense that it is
more concrete, specific, clearly defined, and generally better supported in an en-
gineering context. For example, the boundaries, content, relationships, and value
of involved activities are defined or clarified in the context of software-intensive
systems engineering. Various representational formalisms, either specially de-
signed, or general purpose, modelling language, are provided for OAP activities,
with support from integrated tools. Moreover, instead of simply accelerating the
pace of process, this approach provides a mechanism to reuse knowledge and
capabilities in support of system flexibility and agility, which enables formation
of coherent aggregation of engineering knowledge and process.
Its Rationalisation approach in Mind differs from traditional requirement en-
gineering in terms of scope, mechanism, and “softness”. Traditional requirement
elicitation, analysis and management usually takes a rigid and “hard” approach,
which is based on an assumption that there are requirements, clearly-definable
and stable, out there to be collected in all situations. However, we believe such
an assumption is not always valid in an open environment, where situations are
complex, volatile, and uncertain. The common concept of “requirement” does
not involve or imply “soft” intentions and desires that involve certain degree
of uncertainty in problematic situations, which, undeniably, also influence the
way engineering plans and design are derived and carried out. On the contrary,
our approach provides a specific means to enable and support discovery, under-
standing and utilisation of these aspects of engineering. That is, it incorporates
the concept of Desired End in the context of identified problematic situations,
and systematically provides sensible targets for Realisation such as engineering
planning and design. Its aim is to improve problematic situation to a satisfied
degree, as opposed to achieving a “we-must-do-THIS” type of “requirements”.
The proof-of-concept also reveals some issues and limitations. Currently not
all parts of the OAP conceptual model are supported, such as Sensation, Test and
Control. They are not incorporated appropriately in OAP at this stage because
of a lack of deep understanding of their nature and characteristics. Moreover,
process monitor and debugging activities are not supported, which makes it hard
186 Y. Wang
6 Conclusion
References
1. Baresi, L.: Toward open-world software: Issue and challenges. Computer 39, 36–43
(2006)
2. Aurobindo, S.: Heraclitus. Arya Publishing House, Calcutta (1941)
OAP: Toward a Process for an Open World 187
3. Whitehead, A.: Process and reality: corrected edition. The Free Press, New York
(1978)
4. Fuggetta, A.: Software process: a roadmap. In: Proceedings of the conference on
the future of software engineering, pp. 25–34 (2000)
5. Royce, W.: Managing the development of large software systems. Proceedings of
IEEE Wescon 26, 9 (1970)
6. Larman, C., Basili, V.: Iterative and incremental development: A brief history.
Computer, 47–56 (2003)
7. Osinga, F.: Science, strategy and war: the strategic theory of John Boyd. Routledge
Abingdon, UK (2007)
8. Beck, K., Andres, C.: Extreme programming explained: embrace change. Addison-
Wesley Professional, Reading (2004)
9. Shapiro, S.: Splitting the difference: The historical necessity of synthesis in software
engineering. IEEE Annals of the History of Computing, 20–54 (1997)
10. Checkland, P.: Systems thinking, systems practice. J. Wiley, New York (1981)
An Automatic Approach to Aid Process Integration
within a Secure Software Processes Family
1 Introduction
As software has been playing an ever-increasing role in many domains, security of
software systems has attracted more and more interest [1, 2]. Within software engi-
neering community, as an illustration, International Workshop on Software Engineer-
ing for Secure Systems (SESS) has been held annually from year 2005, in conjunction
with International Conference on Software Engineering (ICSE). Furthermore, “how to
define and support the process of building secure software” has served as a key and
constant topic ever since, which indicates the important role processes play in assur-
ing software security.
Over the years, several dedicated processes on security have been proposed, e.g.
CLASP [3], SDL [4], Touchpoints [5], SQUARE [6] , SecureXP [7], etc. Three fore-
front and widely-used representatives are CLASP, SDL, Touchpoints [8], thus in this
paper we concentrate our discussion on them. These processes each results from or-
ganizing a group of best practices on security insurance, and augments general proc-
ess frameworks (e.g. Unified Process, Microsoft Process, etc.) by adding into new
activities, artifacts, and roles. Take Touchpoints as an example. One of its seven touch
*
Corresponding author.
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 188–199, 2010.
© Springer-Verlag Berlin Heidelberg 2010
An Automatic Approach to Aid Process Integration 189
2 Analysis
As a starting point, let us consider the following motivating cases when integrating units
from different secure processes. We set our range of study to be the three high-profile
secure processes (i.e. CLASP, SDL and Touchponts) mentioned above. The original
contents of these processes are published via website (CLASP) or books (SDL and
Touchpoints), where typical process structures hide within massive text descriptions.
Therefore, for sake of simplicity and authority, we borrow the activity matrix already
established in [11] , which results from collecting all activities from the three processes
investigated. Furthermore, these activities are organized into a tree structure after adding
phases (e.g. Architecture Design) and brief titles for a group of activities (e.g. Security
response planning). An excerpt of the matrix is shown in Figure 1.
z Case A: Two activities from different processes have the same functionality,
whereas differ in implementation details. In this case, we should choose either
one from them, because having them both in the integrated process simultane-
ously will cause unnecessary function redundancy. For instance, both CLASP
and Touchpoints have their own versions of activity “Identify & describe threat
agents”. Only one of them should appear in the integrated process.
z Case B: One activity in a process has the same functionality as a group of activi-
ties from another process. In this case, we should choose either one from them as
well, because having them both in the integrated process simultaneously will
cause unnecessary function redundancy. For instance, in order to inspect poten-
tial access control risk, SDL suggests “Determine who needs access from
where”, while CLASP employs two activities, namely “Identify system entry
points” and “Map roles to entry points” as a solution. Thus, only one of them can
appear in the integrated process.
z Case C: Units from different processes can perfectly connect. In this case, we
can simply choose both of them and join them together, because the output
An Automatic Approach to Aid Process Integration 191
artifact of one unit can be passed as the input artifact of another unit. “Identify &
describe threat agents” from CLASP and “Review & revise threat agents” from
Touchpoints can be regarded as such an example.
z Case D: Units from different processes have conflicts. As an example, consider
integrating “Review & revise threat agents” from Touchpoints with any unit
from SDL. Since SDL provides no activity to create agents, it is impossible for
“Review & revise threat agents” to work with the absence of the input artifact
required. In this case, we cannot join them together.
z Case E: Units from different processes are mutually independent. In this case,
they can coexist well with each other, because they neither overlap in function,
nor require each other to produce the needed input artifact. Still consider “Iden-
tify & describe threat agents” from CLASP as an example, integrating it with
SDL’s “Make your product updatable” is feasible.
3 Approach
3.1 Formalization
In the cases listed in section 2, we judge the integration feasibility by intuition and
common sense, which cannot be supported automatically. However, if we look into
the input and output artifacts of units from case A to E, we can get several observa-
tions. To this end, we first make the following formal definitions:
For artifact a1 and a2, let a1 = a2 when a1 and a2 have the same semantics. Usu-
ally the concrete syntax of artifacts is indicated by the host process (or the organiza-
tion employing this process). Therefore, when comparing artifacts from different
processes, it’s important to focus on their semantics (what’s in them) instead of their
forms (how they are written). In our approach, we merge artifacts with the same se-
mantics into equal artifacts when extending activity matrix with artifact information.
Moreover, as some artifacts in not explicitly introduced in these processes, we judge
their semantics by the context information.
For unit u, let u.in = {a | a is an input artifact for u}, u.in = Φ when unit u receives
no artifact; let u.out = {a | a is an output artifact for u}, u.out = Φ when unit u pro-
duces no artifact.
Based on artifact information, we can define several relations between two activities.
Definition: Equivalence Relation
If u1.in = u2.in and u1.out = u2.out, then u1 and u2 satisfy the equivalence relation.
As a result, either u1 or u2 can appear in the integrated process. Case A and case B
are such examples.
Definition: Dependency Relation
If u1.out = u2.in, then u1 and u2 satisfy the dependency relation. As a result, u1 and
u2 can be integrated together. Case C is such an example.
If u1.in ≠ Φ && u2.out ≠ u1.in, or u2.in ≠ Φ && u1.out ≠ u2.in, then u1 and u2
cannot satisfy the dependency relation. As a result, u1 and u2 cannot be integrated
together. Case D is such an example.
192 J.-K. Ma et al.
Therefore, our previous analysis for these cases is complete in concept and can be
popularized to integrated processes containing various kinds of units.
Based on previous analysis, our basic idea can be summarized as follows.
1) For every process, take each of its activity as an initial unit;
2) Based on artifact information, establish equivalence, dependency and inde-
pendency relations between units, according to the predicate expressions de-
rived above.
3) Globally merge units into larger clusters according to relations among them.
For the purpose of easy and clear comparison, [11] introduced activity matrix. To
better support our approach, we extend the original activity matrix as follows:
1) Complement input and output artifact information for each activity. In case
some activities do not involve any concrete artifact, we label “null” to represent
such a condition.
2) Remove all the brief titles, which are manually added to represent semantic
means for a group of activities. Since our approach take each atomic activity
(i.e. the leaf node) as an initial unit, and leverage artifact information to deduce
the function of a unit, these titles are somewhat redundant.
An Automatic Approach to Aid Process Integration 193
3.3 Algorithm
Implementing the above basic idea onto the extended activity matrix, we have a cor-
responding algorithm, as shown in Figure 3.
The main stream of the algorithm is to decompose processes into atomic units first
(line 2 to 7); then continuously merge them into larger units, according to equivalence
relation (line 10 to 17) and dependency relation (line 18 to 26) between any two of
units. For each unit, we set its type attribute, where ‘eq’ stands for equivalence rela-
tion of its child units, ‘dep’ for dependency relation and ‘reg’ for the rest.
A notable point in the algorithm is the “late-deletion” when merging two dependent
units into a larger one (line 25 and 26), whereas equivalent units get deleted once after
merge (line 17). This is because compared with equivalence relation, dependency rela-
tion is not transitive. Formally, if (a, b) satisfies equivalence relation, and (b, c) satisfies
equivalence relation, we can deduce that (a, c) satisfies equivalence relation as well.
However, this is not the case for dependency relation. Therefore, using a larger unit to
replace two equivalent units will not lose any other equivalent unit to be merged later;
but the two dependent units have to be retained in case other dependency relations in-
volving either of them can be built. In our solution, we leverage garbageList (line 25) to
label dependency units in the first place and delete them after the iteration.
Fig. 3. An algorithm to divide processes into a group of units with certain relations
194 J.-K. Ma et al.
The end loop criteria “Until e does not change (line 27)” stands for our continuous
merge strategy. Each loop merges units into larger ones, which constitute new candi-
dates for the next composition. Since each non-skipped loop reduces at least 1 unit
(only merge two units), for input scale of n units, in the worst case the algorithm can
take no more than n-1 steps (this is merely a theoretical coarse estimation) to stop.
Therefore the algorithm is inherently convergent.
The output of algorithm is a group of units with certain relations, which constitute
a unit library. A piece of result we gain on CLASP, SDL and Touchpoints is showed
in Figure 4. Units in the same slashed rectangle (a single activity or a group of activi-
ties connected by solid arrows) have equivalence relations, and slashed arrow repre-
sents dependency relations between two units.
4 An Exploratory Experiment
We conduct an exploratory experiment to study the practical effects of our approach.
The major objective is to investigate whether the approach proposed by this paper can
promote the efficiency and effectiveness of security support in software process. To
this end, we first discuss experiment settings in section 4.1. Then, we describe the
research method employed in this experiment in section 4.2. Finally, section 4.3 pre-
sents the result and corresponding analysis.
An Automatic Approach to Aid Process Integration 195
T2: According to the design, if a user login contiguously fails for three times, the
system will prevent him from retrying in the next twenty-four hours. However, the
module records the time count in the browser’s cookie. By modifying this data from
cookie, an attacker can try the password with no times limit.
T3: The module does not guard against SQL injection. An attacker can add SQL state-
ments into the submitted form, then get or even modify confidential data in database.
T4: According to the design, only the administrator can add reply for a question.
However, the module only judge a user’s role by retrieving user type flag from
cookie. In this case, an attacker can fake as the administrator by setting his cookie.
Moreover, we choose six volunteers from our lab to act as project participants.
They are both master students majoring in software engineering.
4.3 Results
The result of our experiment is summarized in Table 1.
Time Consumed
Participant Process T1 T2 T3 T4
Total Analysis Design Implementation Testing
s1 CLASP 30h 13h 10h 4h 3h Y Y Y N
s2 SDL 41h 4h 25h 5h 7h Y N N Y
s3 Touchpoints 25h 9h 7h 2h 7h Y Y N N
s4 p1 29h 14h 9h 3h 3h Y Y Y N
s5 p2 32h 13h 12h 2h 5h Y Y Y N
s6 p3 31h 19h 7h 3h 3h Y Y N Y
An Automatic Approach to Aid Process Integration 197
For each participant, we record the time he takes when executing the assigned
process. We also label the phase of execution along with the time records. Therefore,
we have the total time in count as well as time spent on different phases (i.e. analysis,
design, implementation and testing). The time information is regarded as the cost
indicator for a secure process. For the benefit measurement, we compare the threat
identified during each process’s execution with the pre-recorded threat list (i.e. T1,
T2, T3 and T4). The “Y” and “N” in the last four columns represents whether the
given threat has been identified (Y) or not (N).
As the participants are students instead of professionals, the absolute number of
time spent on each process may have systematic deviation (longer than real cases).
However, the relative ratios among different time pieces still make sense (as partici-
pants hold a similar productivity level). For the total time, we can see SDL > p2 > p3
> CLASP > p1 > Touchpoints. In the three original secure processes, SDL seems to
be the most heavyweight. Interview with s2 confirms this, mostly for SDL requires
taking the STRIDE check for each element in the DFD. As a result, s2 substitute the
intensive check activities with units from Touchpoints when customizing p2. Similar
cases also happen between (CLASP, p1) and (Touchponts, p3), whereas s3 introduces
more time-consuming units from other processes when customizing p3. For time
spent in different phases, we can conclude that design and analysis take up large part
of secure process. It seems that the conclusion on total time also holds in most phases:
the customized process being more lightweight when its designer executes a heavy-
weight process, and vice versa.
For the identified threats, none of the cases cover the whole threats. T1 is identified
by all the 6 processes, for dealing with 3rd party component is an important aspect for
all the secure processes we study. T2, however, does not handled by SDL. According to
our interview with s2, he did not consider such an abuse usage when executing SDL,
and so as T3 (a typical attack pattern). SDL shows advantages when applying its STIDE
model to identify T4, compared with CLASP and Touchpoints. After customized, the
three new processes identify more threats (average 3) than the three original processes
(average 2.3), which can be an evidence for the effectiveness of our approach.
As a final remark, we define Threats_Indentified / TotalTime_Consumed as a sim-
ple indicator for assessing secure processes. In this respect, we have: p1 > CLASP >
p3 > p2 > Touchpoints > SDL. Generally, we can conclude that the three customized
processes shows higher efficiency than single dedicated processes.
Broadly, there are questions about the extent to which the results gained by studying a
small module would popularize, questions regarding the rationalization of our re-
search method, and questions related to the chosen of p1 to p3 in our experiment.
We choose a small module on purpose (according to experiment setting F5), but
this module is also security-critical (according to experiment setting F1). Considering
most systems valuing trustworthiness commonly employ a well designed architecture
composed of highly-coherent and loosely-coupled modules, we believe the conclusion
gained from this module can be popularized to larger systems.
The rationalization of our research method is also an important point. The basic
idea behind our experiment design is to simulate the case in industrial environment
198 J.-K. Ma et al.
with afforded cost, as close as possible. Therefore, we require the project chosen
should have complete in-process artifacts and make use of them when necessary, as if
our secure team and other project participants work together at the same time. Admit-
tedly, we have to abandon activities involving other stakeholders (e.g. customers,
developers), which may influence the power of processes we study. Moreover, we
train volunteers from our lab with the three secure processes as well as the chosen
project’s basic information, so that they can act like qualified secure team members.
Besides these compromises, an additional benefit gained from our method is the ac-
quisition of an easy, uniform and objective assessment basis for secure processes,
which results from using a completed project with all its security issue recorded.
Last but not least, the three customized processes, namely p1, p2 and p3, are wor-
thy of further discussion. The actual success of our approach highly depends on users’
ability to customize a suitable process. In our experiment, we ask s1, s2 and s3 to
customize their processes respectively, right after executing a certain dedicated secure
process assigned to him. The main reason for this is we want to simulate real cases
where processes are customized by knowledgeable managers based on their past ex-
perience with secure processes. Admittedly, the experiment can be further improved
by letting participants experience more than one secure processes before perform
customization, if resources permit.
5 Related Work
In [8], a systematic comprehension of three secure processes (CLASP, SDL and
Touchpoints) is performed. One important conclusion of [8] is that the three processes
compared are similar in many aspects, while have their own features. None of them is
absolutely better than others, for each process has its advantages as well as disadvan-
tages. Such a claim is close to the process family concept proposed by [9-10]. In [9-
10], a board and general proposal for software process family is presented. However,
how to implement the construction and application of process family is not mentioned
in detail. Our work can be seen as a concrete step towards such a direction.
To our best knowledge, the idea of automatically integrating secure processes in
the same family is new. There have been efforts on decomposing certain processes
into more fine-grained units, and organizing them [12-14]. However, these ap-
proaches are highly manual. For example, OOSP [12] summarizes and organizes a
group of patterns for object-oriented development. These patterns come from manual
extraction of expert knowledge. Compared to them, we focus on providing automatic
support, aiming to promote the usability and usefulness of the entire approach.
6 Conclusion
Supporting security is often an important requirement in many software projects.
Over the years, a wealth of dedicated secure processes has emerged, forming a so
called “Process Family”. Conceptually, such a process family can take advantage of
different security processes at a time. However, how to integrate secure processes
within a process family should be answered first.
Therefore, we proposed an automatic approach to aid the integration of the three
forefront secure processes, namely, CLASP, SDL and Touchpoints. Moreover, in
An Automatic Approach to Aid Process Integration 199
Acknowledgement
This work is founded by the National Basic Research Program of China (973) under
Grant No. 2009CB320703, the High-Tech Research and Development Program of
China under Grant No. 2007AA010301 and the Science Fund for Creative Research
Groups of China under Grant No. 60821003
References
[1] Taylor, D., McGraw, G.: Adopting a software security improvement program. IEEE Se-
curity & Privacy (2005)
[2] Byer, D., Shahmehri, N.: Design of a Process for Software Security. In: International
Conference on Availability, Reliability and Security (2007)
[3] lightweight application security process, http://www.owasp.org
[4] Steve, L., Michael., H.: The Security Development Lifecycle (SDL): A Process for De-
veloping Demonstrably More Secure Software. Microsoft Press, Redmond (2006)
[5] Gary, M.: Software Security: Building Security. Addison Wesley, Reading (2006)
[6] Mead, N.R., Houg, E.D., Stehney, T.R.: Security Quality Requirements Engineering
(Square) Methodology. Software Eng. Inst., Carnegie Mellon Univ. (2005)
[7] Boström., G., et al.: Extending XP Practices to Support Security Requirements Engineer-
ing. In: International Workshop Software Eng. for Secure Systems, SESS (2006)
[8] Bart, R.S., Koen, D., Johan, B., Wouter, G.: On the secure software development process:
CLASP, SDL and Touchpoints compared. Information and Software Technology,
1152–1171 (2008)
[9] Simidchieva, B.I., Clarke. L.A., Osterweil, L.J.: Representing Process Variation with a
Process Family. In: International Conference on Software Process (2007)
[10] Sutton, S.M., Osterweil, L.J.: Product families and process families. In: Software Process
Workshop (1996)
[11] Buyens, J.G.K., Win, B.D., Scandariato, R., Joosen, W.: Similarities and differences be-
tween CLASP, SDL, and Touchpoints: the activity-matrix, K.U. Leuven, Department of
Computer Science (2007)
[12] Ambler, S.W.: Process Patterns: Building Large-Scale Systems using Object technology.
SIGS Books/Cambridge University Press, New York (1998)
[13] Land, I.C.R., Larsson, S.: Process Patterns for Software Systems In-house Integration and
Merge – Experiences from Industry. In: Software Engineering and Advanced Applica-
tions (2005)
[14] Wang, Y., Meng, X.-x., Shi, L., Wang, F.-j.: A Process Pattern Language for Agile Meth-
ods. In: Asia-Pacific Software Engineering Conference (2007)
Engineering Adaptive IT Service Support Processes
Using Meta-modeling Technologies
1 Introduction
With the rapid evolution of Information Technologies (IT), to many organizations, the
importance of IT has been unceasingly increasing, more and more getting into the main
business of an organization in depth. As a result of its increasing role in the organiza-
tion, the focus of IT management has moved from device-oriented management to
service-oriented management [1]. Many organizations have tried to adopt IT service
support processes, techniques and tools to manage the efficient supply of IT services
with guaranteed quality. Efforts have been taken to make a progress, but new challenges
emerged: (1) When existing solutions are applied in various organizations, the customi-
zation costs are usually too large, small firms in particular [2]. (2) Dynamic changes
during IT service support process execution can’t be supported by almost all of the
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 200–210, 2010.
© Springer-Verlag Berlin Heidelberg 2010
Engineering Adaptive IT Service Support Processes 201
existed tools. (3) Even with universal understanding of IT service support concepts,
integration, reuse and exchange of process models in different process model languages
still remain challenges.
In this paper, meta-modeling of IT service support process is provided to solve the
above problems. Metamodel is an explicit model of the constructs and rules needed
to build specific models within a domain of interest. We propose a metamodel of IT
service support process, then develop a flexible IT service support process engineer-
ing platform based on it, where a process modeling tool is developed by model driven
approach, and a set of flexible IT service support process rules are designed and im-
plemented to support adaptive process management.
The rest of the paper is organized as follows: Section 2 introduces IT service support
process metamodel. Section 3 briefly describes the architecture of IT service support
process engineering platform. Section 4 explains the MDA approach applied in
development of process modeling tool. Section 5 focuses on business rule design for
adaptive process management. Section 6 presents related work, and finally section 7
concludes the paper.
There are many separate business process domains, such as software process, supply
chain, workflow, manufacturing process and etc. IT service support process is busi-
ness process too, which has five core sub-processes: incident management process,
problem management process, configuration management process, change manage-
ment process, and release management process [4]. While each domain has its own
distinct metamodels, there exist some core similarities. It may thus be interesting to
propose an IT service process metamodel inheriting a more generic business process
metamodel, as shown in Figure 1. Expected benefits are reuse and facilities for proc-
ess models translation from one metamodel to another.Using this process metamodel
architecture, the IT service support process meta-modeling method consists of two
steps:
Step1: Select an appropriate business process metamodel. We should first analyze
IT service support domain and extract its abstract process concepts based on its best
practices. Comparing these entities with concepts of existed business process meta-
models, we can select the most appropriate one.
Step2: Extend generic process metamodel to build the IT service support process
metamodel. It will be extended to include features and particular concepts of IT ser-
vice support process domain, using the Meta-Object Facility (MOF) [5] technology.
202 B. Shen et al.
Business Process
Software Process Supply Chain IT Service Support Process Workflow Other Process
IT Infrastructure Library (ITIL) [4] which was developed by the Central Computer
and Telecommunications Agency of the UK government in the middle of 1980s, pro-
vides a comprehensive, consistent and coherent set of best practices for IT and is
becoming the de facto standard for IT service support. So, we extract IT service sup-
port process concepts from ITIL specifications. From the process aspect, the key
concepts of IT service support include process, activity, role, actor, resource and so
on. Then we compare these concepts with core elements of four popular business
process metamodels -- Simple Metamodel for Business Process (SMBP) [7], UML
Activity Diagram (UML AD) [8], Event Process Chain (EPC) [9], Business Process
Definition Metamodel (BPDM) [10], as shown in Table 1. Through the above analy-
sis, we come to conclude that BPDM best supports modeling of IT service support
process.
Core Element of IT Core Element Core Element of Core Element of Core Element
Service Support of SMBP EPC UML AD of BPDM
Process Process Function (not sepecified) Process
Activity Process Function Activity Activity
Simple
Action Activity ElementaryFunction Action
Activity
Role (not sepecified) Organisation Role Swimlane Performer Role
Processor Role (not sepecified) (not sepecified) (not sepecified) Processor Role
Information
Resource Data LinkEndData Holder
Object
Engineering Adaptive IT Service Support Processes 203
VRXUFH WDUJHW
Succession
Interactive Part
VXERULJLQ
VXEGHVWLQDWLRQ
Event Part Course Part
Process Process Interaction Boundary
SURFHVVW\SH
RZQHUSURFHVV
Gateway Behavior Step
RZQHGDFWLYLW\
Normal End Error
SURFHVVXVDJH
Simple Activity
We extend BPDM for better reflecting the characteristics of IT service support proc-
ess from three aspects: activity, resource, and interaction. The extension of BPDM is
also in accordance with MOF. Fig. 3 shows an activity view of metamodel extension,
where several simple activities/ actions (e.g. Email Message, Record, and Feedback)
are defined. These activities form the atomic behavior of the process.
Fig. 4 shows a resource view of metamodel extension, where “Resource” extends
“Holder” because resource is playing an important role in IT service support process.
Like a Behavior Step indicates the state in process, the new added “State” class also
makes the lifecycle of the resource accessible. Some kinds of resources are listed in
the metamodel, such as “Incident Record” and “Problem Record”.
204 B. Shen et al.
Recover Close
Succession
Interaction
Feedback Request
“State” can be changed by certain activity. Also “State” has its succession, namely
“State Succession”, which can connect to different “State” objects. The transition
between states is inspired by activities.
Fig. 5 shows an interaction view of metamodel extension, where several kinds of
simple interactions are listed, such as “New Incident Request” and “Incident Re-
corded Notification”.
Define
Define
Use
Business Rule
Process Model
Check/Run
Event Trigger Run
Monitor/Use Generate/Update/Search/Achieve
When certain event occurs and condition is available, then action or application is
executed. Fig. 9 shows some examples of ECA rules in an incident management process.
Rule1: On (NewIncidentRequest)
If (equal (IncidentRecord.Emergency,”High”)
Do Email_Message.do (…)
Rule2: On (NewIncidentRequest)
If (equal (IncidentRecord.Type,”Hardware”)
Do Assign.do(…)
Rule3: On (IncidentClosureRequest)
If (equal (IncidentRecord.State,”Open”)
Do Transition.do(…)
Rule1: when a new incident with high emergency happens, system will inform the
staff with Email. Rule2: when a new incident typed hardware happens, system will
dispatch it to a specific expert. Rule3: when the incident closed, the state of the inci-
dent is set closed.
In ITSSP, all parameters (event, condition and action) in ECA rules use the core
elements in IT service support process metamodel. These rules can be defined and
managed dynamically during process enactment by Business Rule Definer, and
checked and run by Business Rule Engine. We use AOP technology to implement the
Business Rule Engine.
6 Related Work
Related work can be presented from three aspects: IT service support process model-
ing, meta-modeling and tool.
1) IT service support process modeling
Process modeling is a hot topic and attracts global research and practice. Lots of process
modeling languages are provided with suitable control flow features such as BPEL4WS
[10], XPDL/BPMN [11], ebXML/BPSS [12], UML [8], and Event-Process-Chain
(EPC) [9]. However, although these process modeling languages are capable of model-
ing process for IT service support, they generally lack expressiveness regarding the IT
service support domain characteristics. To our best knowledge, there is no dedicated
modeling language designed for IT service support process. Commercial organization
ARIS is also working on this ITIL Reference Model, but its target is to clarify the ITIL
concepts, and not to model related processes.
2) IT service support process metamodel
Metamodel is the base of model. So far, there are many publications and researches
about process metamodel. These metamodels construct and refine the concepts in
process domains such as software development [13], security [14], performance [15]
and broader process framework [16] and so on. However, research work on IT service
Engineering Adaptive IT Service Support Processes 209
7 Conclusions
In this paper, we propose an ITIL-based IT service support metamodel through ex-
tending a generic business process metamodel by IT service support domain charac-
teristics. Based on the proposed metamodel, we have developed a flexible IT service
support process platform — ITSSP. The platform integrates and automates IT service
support processes including incident, problem, change, release and configuration
management. It not only supports process modeling and enactment, but also supports
adaptive process management via ECA rules.
Using meta-modeling technology in IT service support processes engineering, has
gained following benefits:
1) Supporting the changes of IT service support processes on demand, both at
model level and at metamodel level.
2) Assisting model integration, exchange and reuse between IT service support
process and other business processes.
3) Generating code scaffold of IT service support process modeling tool from
metamodels, following MDA approach.
4) Facilitating the design of IT service support process rules with the core ele-
ments of metamodel, to support dynamic process logic changes.
Since March, 2009, ITSSP has been deployed in three companies: Shanghai Hardware
Fasteners Co., Ltd, China Network Co., Ltd, YunHao Machine Electronic System
Engineering Co., Ltd. All of these practices have shown that ITSSP provides flexible
approaches to static process modeling and dynamic rule definition, and it helps them
to build an IT service support system with process engineering technology, thus im-
proves the quality and efficiency of their IT services.
210 B. Shen et al.
References
1. Drago, R.B., et al.: A Proposed Web Tool for IT Environment Service Management. In: Pro-
ceedings of International Conference on Internet and Web Applications and Services (2006)
2. Aileen, C.S., Tan, W.G.: Implementation of IT Infrastructure Library (ITIL) in Australia: Pro-
gress and success factors. In: IT Governance International Conference, Auckland, NZ (2005)
3. Mi, P., Scacchi, W.: A Meta-Model for Formulating Knowledge-Based Models of Soft-
ware Development. Special issue: Decision Support Systems 17(4) (1996)
4. The Stationery Office, Office of Government Commerce, United Kingdom: ITIL and IT
Service Management (2010), http://www.itil.org.uk
5. OMG: Meta Object Facility (MOF) 2.0 Core Specification, OMG Adopted Specification
ptc/03-10-04 (2003)
6. Mendling, J., van Dongen, B.F., van der Aalst, W.M.P.: Getting Rid of the OR-Join in
Business Process Models. In: Enterprise Distributed Object Computing Conference (2007)
7. Curtis, B., Kellner, M., Over, J.: Process Modeling. Communication of the ACM 35(9)
(1992)
8. OMG: Unified Modeling Language (UML) Specification: Superstructure, V2.0 (2005),
http://www.omg.org/docs/formal/05-07-04.pdf
9. OMG: Business Process Definition Metamodel (BPDM). OMG Adopted Specification
ptc/03-10-04 (2003)
10. OASIS: Web Services Business Process Execution Language. V2.0 (2007),
http://docs.oasis-open.org/wsbpel/2.0/OS/wsbpel-v2.0-OS.html
11. OMG: BPMN Specification V2.0 (2009), http://www.bpmn.org/
12. OASIS: Business Process Specification Schema v1.01 (2001),
http://www.ebxml.org/specs/
13. Florac, W.A., Carleton, A.D.: Measuring the Software Process: statistical process control
for software process improvement. Addison-Wesley Professional, Boston (1999)
14. OMG: Enterprise Distributed Object Computing, OMG Adopted Specification ptc/03-10-
04 (2003)
15. Bézivin, J.: On the unification power of models. Software and System Modeling 4, 2
(2005)
16. Firesmith, D.G., Henderson-Sellers: The OPEN Process Framework. The OPEN Series.
Addison-Wesley, London (2002)
17. Goeken, M., Alter, S.: Representing IT Governance Frameworks as Metamodels. In: Pro-
ceedings of the 2008 International Conference on e-Learning, e-Business, Enterprise In-
formation Systems, and e-Government (2008)
18. Jäntti, M., Eerola, A.: A Conceptual Model of IT Service Problem Management. In: Pro-
ceedings of the IEEE Conference on Service Systems and Service Management (ISSSM
2006). University of Technology of Troyes, France (2006)
19. Strahonja, V.: Definition Metamodel of ITIL. Information Systems Development Chal-
lenges in Practice, Theory, and Education, vol. 2 (2009)
20. Park, C., Choi, H., et al.: Knowledge-based AOP Framework for Business Rule Aspects in
Business Process. ETRI Journal 29(4) (2007)
Modeling a Resource-Constrained Test-and-Fix Cycle
and Test Phase Duration
The Aerospace Corporation, P.O. Box 92957, Los Angeles, California 90009
{daniel.x.houston,man.v.lieu}@aero.org
Keywords: Test and fix cycle, software rework cycles, test phase duration
simulation, resource-constrained testing.
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 211–221, 2010.
© Springer-Verlag Berlin Heidelberg 2010
212 D. Houston and M. Lieu
and final testing (FT). In the first stage, each test case is developed and initially
tested. When it is ready, it goes into the TaF stage with the target software for testing
in a dedicated test facility. Once a test case runs fully in the real-time test facility
without displaying anomalous behavior, it is ready for final test. The TaF stage may
require multiple sessions in a test facility due to defects found, but the FT stage is ex-
pected to proceed without problems.
We used the models initially with designed experiments to learn about the factors
driving a resource-constrained test-and-fix cycle. In subsequent work, we developed a
procedure for applying the model to specific projects. The first part of the procedure
is interviewing project personnel to learn the specifics of their resource-constrained
testing phase so that models can be calibrated to adequately represent their testing
phase. The other part of the procedure is obtaining and analyzing defect data from
previous activities, such as peer reviews, as a basis for estimating the number of de-
fects to be discovered. Later, the learning process continues as project performance
data becomes available and that is used to improve the modeling so that it better
represents the actual process.
This paper uses an example aggregated from the class of software development
projects having a test-facility-constrained test phase. Test plans for these projects typ-
ically specify 100 to 200 test cases, so our example uses 150 test cases to illustrate
common test completion behavior for this class of projects.
2 Related Work
Software process simulation models have long included defect detection and rework
flows. To name a few, Abdel-Hamid and Madnick [3], Tvedt [4], and Zhang et al. [5]
provide examples of software development simulation models that include defect
detection and rework flows. However, the scope of these models is wider than the in-
terest of this study. Also, these models do not use cyclic flows to model testing and
fixing because, at their levels of abstraction, software was assumed to pass through
once, usually with retesting included in the fixing activity.
The process described here includes a cyclic rework process, one in which software
and test cases needed to pass through a test facility multiple times as defects were de-
tected and fixed before the test case and software went back into the test facility for
retesting. Modeling of rework cycles has been explored extensively by Cooper [6] and
by Ford and Sterman [7], which both view product development as a rework process
using a cyclic flow in a system dynamics model. In addition to the rework cycle,
Cooper’s model emphasizes the effects of various productivity-modifying factors
such as staffing characteristics (hiring, experience, turnover, morale, and overtime),
perceived progress, schedule pressure, and engineering revisions. In contrast, Ford’s
model supports representation of multiple phases using two rework loops, one for de-
fects found in-phase and another for defects found downstream.
3 Models Employed
We had developed a versatile, multiphase system dynamics model of software devel-
opment based on Ford’s model. Our model was designed to represent a wide variety
Modeling a Resource-Constrained Test-and-Fix Cycle and Test Phase Duration 213
The discrete event model (Figure 2) depicts arriving test cases receiving a value for
the attribute, Runs per Test Case. Test cases are queued for a test facility and in the
queue they are sorted by the number of runs that each has completed so far, giving
preference of test facility usage to cases further along in the testing process. The test
case at the front of the queue enters the first available facility, staying for the time re-
quired to perform the test case. After the test, the number of test sessions for the test
case is incremented and checked: if it has reached its required number of runs, it is re-
leased for final testing, otherwise it is delayed while a problem is diagnosed and
fixed. After the Fix Time, the test case is returned to the queue for another test run.
Fig. 2. Discrete Event Model of Software Testing Constrained by Test Facility Availability
The first experiment performed on the model was designed to determine the relative
importance of the modeled input factors. The arrival rate of test cases is often a con-
stant value, in this example, one new test case per day available for testing. For the
remainder of the factors, a full factorial (24) experiment was run per Table 1.
Project personnel often expect that Test Facility Availability is the dominant factor
in the test-and-fix cycle. However, experiments have shown that Runs per Test Case
is the dominant factor in software test duration, in this example solely contributing to
54% of TaF duration variation and contributing to another 22% through interactions
with other factors. In particular, this factor interacts strongly with Test Run Duration.
Factor Values
Test Facility Availability 60 and 100 hrs/week
Runs per Test Case 2, 4, 6, 8
Test Run Duration 2, 3, 4, 5 hrs
Fix Time 7 days
The influence of Runs per Test Case is evident in both regions. The planar region
reflects almost no increased duration due to Test Run Duration, but a slowly rising
duration due to Runs per Test Case. In the steeply ascending region, Runs per Test
Case interacts strongly with Test Run Duration.
The threshold represents the points at which the test facilities are fully utilized.
When all other factors are constant, the full utilization threshold is a function of test
facility availability. Thus, as Test Facility Availability decreases, shown in the con-
trast of the two Figure 4 graphs, the threshold moves out across the planar region,
adding more area to the steeply ascending region and causing it to rise to a higher
point. In practical terms, this means that the tighter constraint on test facility availabil-
ity produces full utilization sooner and the accompanying interaction between Test
Run Duration and Runs per Test Case lengthens the TaF duration.
It became clear from Figures 3 and 4 that test facility availability, though not the
dominant factor in duration of the constrained test-and-fix cycle, is active as a facilita-
tor of the dominance and interaction of other factors.
Figure 5 shows the results of 15 simulation runs, enough to exhibit both the varia-
tion and the pattern of test case completions. Despite the input variations, the comple-
tion time curves for 150 test cases produced a clear pattern comprised of two sections:
218 D. Houston and M. Lieu
the curved tails and the straight middle. The length of the tails was found to be de-
pendent on the Fix Time values: as the Fix Time distribution increases, it takes longer
to initially complete test cases, as well as to clear the last test cases out of the cycle.
The straight middle region occurs with full utilization of the test facilities. The slope
of this linear region is largely determined by test facility availability, with a limit due
to the incoming rate of test cases.
Confidence and prediction intervals are calculated for estimation purposes. For this
example, the 95% confidence interval on the mean duration is 32 to 34 weeks (error
estimate = .43), and the 95% prediction interval on duration is 29 to 36 weeks.
Project personnel often want to look at ways to reduce the test phase duration, so a
number of questions about alternatives have been posed and studied using the model.
For example, two questions were asked about increasing test facility availability,
either through taking time late in the test phase from another test facility, or by con-
structing a new test facility. Neither of these options significantly reduced TaF dura-
tion because the additional testing capacity came too late in the TaF stage.
Sometimes project personnel question the initial assumptions behind their inputs
and ask for simulation results using more optimistic assumptions. To illustrate, more
optimistic distributions for Runs per Test Case and for Fix Time were tried in this ex-
ample and they produced better results: improved Fix Time alone resulted in a 12%
duration reduction, improved Runs per Test Case alone produced a 26% improve-
ment, and improved Fix Time and Runs per Test Case together produced a 43% im-
provement. Achieving these results in practice is considered unlikely but they can
demonstrate what is required to achieve a proposed test schedule.
Modeling a Resource-Constrained Test-and-Fix Cycle and Test Phase Duration 219
6 Multiphase Model
The results of the constrained test-and-fix cycle model were taken back into the mul-
tiphase continuous model by calibrating the multiphase model to reproduce a 33 week
TaF stage. The multiphase model had been configured to reproduce an incoming rate
of one test case per day from the test case development stage. The final step in con-
figuring the multiphase model was setting the concurrency and productivity for the fi-
nal test (FT) stage.
Two cases were modeled based on resource contention. If the TaF runs are given
priority over FT runs for test facility usage, then test cases completing TaF (thick,
long dashes) proceed smoothly and FT completions (thick, short dashes) follow, more
slowly at first but then rapidly as TaF is completed. On the other hand, if management
chose to show test phase completion as soon as possible for some test cases, FT test-
ing preempts TaF testing: TaF completes more slowly (thin dashes) while FT comple-
tion is accelerated (thin dotted line). Regardless of TaF or FT priority, FT completes
very soon after TaF. However, the simulation results suggest that giving FT priority
over TaF could delay the overall completion of testing, by about 2 weeks in this ex-
ample (magnified showing that the thin dashed/dotted lines converge on 150 after the
thick dashed lines).
Fig. 6. Multiphase model results for completing development and use of test cases
220 D. Houston and M. Lieu
Though these guidelines may seem obvious to those well-versed in the economics of
software quality, many development groups have only a vague notion of quality costs
and, working under cost and schedule pressures, are inclined to choose poorly. Using
a simulation model to demonstrate the dynamics of a specific process can make a very
persuasive argument for good choices, as well as quantifying the relative importance
of factors.
In addition to general guidelines produced for projects, specific modeling results
have also been beneficial. In at least one case, modeling and simulation demon-
strated that a proposed schedule was unrealistically optimistic. In that case, our
recommendations to the test team helped them to mitigate some of the risks to the
testing schedule.
Modeling a Resource-Constrained Test-and-Fix Cycle and Test Phase Duration 221
8 Further Research
This modeling project has been very helpful in providing better planning information
to project personnel. It is currently being used to track actual results and improve test
phase forecasting. In addition, we plan to re-implement the models in a hybrid model.
References
1. Dörner, D.: The Logic of Failure: Recognizing and Avoiding Error in Complex Situations.
Perseus Publishing, Cambridge (1996)
2. Kellner, M., Madachy, R., Raffo, D.: Software Process Simulation Modeling: Why? What?
How? Journal of Systems and Software 46(2/3), 91–105 (1991)
3. Abdel-Hamid, T.K., Madnick, S.E.: Software Project Dynamics. Prentice Hall, Englewoord
Cliffs (1991)
4. Tvedt, J.D.: An Extensible Model for Evaluating the Impact of Process Improvements of
Software Development Time. PhD dissertation. Arizona State University (1996)
5. Zhang, H., Jeffrey, R., Zhu, L.: Hybrid Modeling of Test-and-Fix Processes in Incremental
Development. In: Wang, Q., Pfahl, D., Raffo, D.M. (eds.) ICSP 2008. LNCS, vol. 5007,
pp. 333–344. Springer, Heidelberg (2008)
6. Cooper, K.G., Mullen, T.W.: Swords and plowshares: The rework cycle of defense and
commercial software development projects. American Programmer 6(5), 41–51 (1993)
7. Ford, D.N., Sterman, J.D.: Dynamic Modeling of Product Development Processes. System
Dynamics Review 14(1), 31–68 (1998)
Visual Patterns in Issue Tracking Data
1 Introduction
Most software development organizations have issue tracking systems in place
where various information about the software development process is stored.
Also with all the information requirements from CMMI and ISO certifications,
there is a huge amount of data available for investigation. Applied correctly, the
insights from such an investigation help to improve the development process.
However the data gathered from software projects have some properties that
makes it hard to get meaningful results from traditional statistical analyses. The
same properties are also deterrent to more advanced data mining approaches.
One of the biggest problem for statistical approaches is the severe right skewed
distribution of the data gathered from software projects: Using problem reports
(PRs) as a starting point we usually see a huge amount of problems that are
fixed in a short amount of time, take only little effort, and affect only a small
amount of source code modules. Calculating averages, comparing distributions,
etc. does not yield the required insights. In the end, what does it mean if we can
improve the average resolution time from 1 hour to 50 minutes, or even lower.
It is very unlikely that the accuracy of the entered measures, e.g., resolution
time or effort, was that accurate to begin with. But on the other hand, finding
the reasons why certain problems took more than 100 hours to be resolved, and
present common patterns for these outliers, can help a manager to significantly
improve the process.
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 222–233, 2010.
c Springer-Verlag Berlin Heidelberg 2010
Visual Patterns in Issue Tracking Data 223
Another problem with all the data gathered in software project repositories is
quality. If the data is not used and discussed on a regular basis inside the project
team, the accuracy of the human entered data is at best questionable. Entering
accurate information is time consuming and does not yield direct results, i.e.,
no new features get implemented. It is therefore difficult to motivate developers
to dedicate enough time to this administrative tasks.
Given the difficulties stated above, we developed a visual approach that is strong
in outlier detection, supports the discovery of common patterns, and helps in pro-
moting discussions among the participating stakeholders regarding process devel-
opment metrics. It also provides facilities to detect flaws in the entered data. The
approach is based on the Micro/Macro Reading idea [1] and provides a combina-
tion of Overview and Details-on-demand where the big picture as well as detailed
problem report (PR) information are displayed at the same time. This is a very
important property of our approach and helpful for visual pattern detection.
In [2] we already presented the interactive features for data exploration. In this
paper we present and discuss generic visual patterns found in effort measurement
and problem life-cycle data. We also apply our visualization technique to an
industrial case study conducted in the EUREKA/ITEA1 project SERIOUS2 .
Discussing the results of our analysis with a group of project managers from
our industrial partner, managers and researchers from the other partners, as well
as the ITEA review board, which also consist of members from the industry, we
have strong anecdotal evidence that our approach stimulates discussion, is easy
to understand, and provides valuable insights into the data buried in software
project repositories.
The remainder of this paper is organized as follows: in the next section we
describe the data of our case study. In Section 3 we show the relevant elements
of our visualization approach. In Section 4 we present typical patterns found in
project data, and in Section 5 we present our case study and discuss the detected
patterns. In Section 6 we present related work and then conclude in Section 7.
underestimated
actualEffort
perfect estimate
color: analyser
estimatedEffort
overestimated
is determined by the value of the estimatedEffort and the height by the value of
the actualEffort. The effort measure is the sum of all efforts that were exerted
to resolve the issue described by the problem report.
With this mapping we can get a quick and effective overview over the quality
of estimates: balanced estimates (square boxes) constitute problems that were
estimated accurately with respect to the actual resolution effort needed; under-
estimated (boxes that are thin and tall), and overestimated (boxes that are short
and broad) PRs can also be spotted easily.
hours spent in
evaluation
hours spent in
resolution
To visualize the relative duration of process steps we use a Pie Chart Vi-
sualization. In Figure 3 we show a single pie with the mapping to the four
process steps: submitted, in analysis, in resolution, in evaluation. The size (i.e.,
the area) of the pie is mapped to the total time from the creation of the PR
until it was closed.
Finally our Phase View Visualization in Figure 4 is concerned with the
process life-cycle sequence. This view depicts the sequence of the process steps,
and allows an investigator to spot cycles and other exceptional situations.
226 P. Knab, M. Pinzger, and H.C. Gall
4 Visual Patterns
In this section we present some generic visual patterns that we found in the
course of our investigations. In the next section we will then show the actual
instances of these patterns in the data of our industrial case study. The first
part of this section presents the patterns for the Polymetric Views, followed
by the patterns for our Pie Chart View, as well as our Phase View in the
second part.
For all effort estimation patterns that we present in this paper, we mapped the
color of the boxes to the analyzer. With this mapping we can assess the behavior
of individual analyzers.
In Figure 5 we see the pattern that we strive for. All the estimates of the four
analyzers are perfect squares, i.e., estimated and actual effort are the same. The
problem with such a pattern, should it occur in actual data, is, that it is, most
certainly, to good to be true. Therefore if one sees only or mostly squares this
would be a strong indication that something with the data entry is wrong. If
the picture looks like in Figure 6 the perfect estimates of the analyzer that was
mapped to the blue boxes stand out. A manager should investigate the reason
for this unlikely pattern.
Another pattern that indicates a flaw in the data entry is shown in Figure 7.
In picture d) always the same, all the boxes have the same width which means,
that the entered estimate was always the same. This is again a strong indication
that the effort estimation was not done carefully or something else went wrong.
Visual Patterns in Issue Tracking Data 227
Fig. 6. Cheater
a) underestimator c) scatter
b) overestimator
Again, the manager responsible for the estimation process should talk to the
analyzer and improve the situation.
The patterns a) underestimator and b) overestimator provide probably the
best information to improve the performance of certain analyzers. These analyz-
ers show a strong tendency either towards underestimation or towards overesti-
mation. With such a clear tendency it should be easy to correct their estimation
behavior and improve the quality of the estimates. The pattern c) scatter shows
a problem that is harder to correct since the effort estimations deviate in both
directions, without a strong tendency that can easily be corrected. Therefore the
only way to improve the quality of the estimates is to train the analyzer. An-
other explanation for the inaccurate estimates could also be that this particular
analyzer just got the problems which are the hardest to analyze.
Similar considerations apply also to the patterns a) underestimator and b)
overestimator. There are always multiple possible explanations for a weak es-
timation performance and it is necessary to investigate further, and especially
discuss the initial findings with the affected engineers.
The primary goal of our visualization approach is not to asses the performance
of analyzers but to help improve the software development process and find
unwanted patterns that can then be corrected. If a correction is not possible
228 P. Knab, M. Pinzger, and H.C. Gall
d) A phase is revisited
then at least more knowledge about dependencies in the software process can be
gained, which should than lead to improvements in future projects.
5 Case Study
The case study is based on a five year multisite project in the consumer elec-
tronics domain. The issue tracking repository contains approx. 20’000 problem
reports (PRs) that were handled by 368 distinct analyzers.
Visual Patterns in Issue Tracking Data 229
a) Underestimator
c) Overestimator
e) Cheater
Fig. 10. Effort View on PRs Grouped and Colored According to the Analyzer
a)
b) c) d)
e)
Fig. 11. Pie View for Top and High Priority Problems
the problems labeled a) and b) stand out, because they stayed most of the time
in the submitted state. Again looking at their surrounding, almost no other pie
displays a similar pattern. It might therefore be worthwhile to take a closer look
at a) and b). We selected c) and d) to represent the most dominant pattern in
this visualization. There are a lot of problems that spent most of their time in
evaluation, and therefore took a long time until they were closed. And finally e)
stands out in two ways, it is the biggest pie and it is also totally blue. Consulting
the detailed log file for this problem we see that after an initial analysis that
yielded a “not repeatable” verdict, the report was not touched for a long time
until it finally was rejected. This is rather surprising, since one would expect that,
for high priority bugs, there is pressure to close problems as fast as possible.
In Figure 12, we select the biggest pies from Figure 11 and further analyze
them in our Phase View visualization. We selected the problems labeled a), b),
and c) because they look the most interesting. In the handling of problem a) a
lot of shuffling between different analyzers has happened. From the log file we
a)
b) c)
Fig. 12. Phase View for Top and High Priority Problems
Visual Patterns in Issue Tracking Data 231
can extract that every blue block represents a change in the analyzer. There
were in total seven analyzers involved and in the end the problem report was
marked as a duplicate. The first resolution attempt for the problem labeled b)
did not succeed and a second attempt was necessary. And for c) there were three
attempts necessary. Another interesting fact that we can spot by looking at c)
is, that the first resolution phase was not preceded by an analysis. This might be
acceptable behavior or something that should be corrected. Looking at the other
problems we can spot quite a few that are missing an initial analysis phase.
We can see from these examples, that by using this easy to understand visual-
izations, we have already gained insights into potential points for improvement.
It is important to keep in mind, that one of the main goals of our approach is
to improve awareness for hidden problems and stimulate discussion, and not so
much to provide ready made action plans.
6 Related Work
D’Ambros et al. used data from a release history database (RHDB) [4] to visual-
ize the life-cycle of bugs [5]. With the System Radiography and the Bug Watch
View, they provided a tool to focus on time, activity and severity/priority as-
pects of bugs. The advantage of such a visualization is the possibility to switch
easily between an overview picture and a detailed inspection of an individual
bug.
Halverson et al. [6] devised a visualization that focuses on the social aspects of
change requests and problem reports. Similar to our approach, they also visualize
state changes and reveal problematic patterns such as multiple resolve/reopen
cycles.
The main differences between these related approaches and ours is that the
state changes or the bug activity, in our approach, is only one of many possi-
bilities to look at the data. Also the relatively simple process model underlying
Bugzilla, and the missing effort measures, result in a different kind of analysis.
In [7] Weiss et al. predict the effort a problem requires to get fixed by analyz-
ing the problem description and search for similar problem reports. The average
effort it took these similar problems is than used as a prediction. This relates to
our approach in the way, that it uses only the attributes of the problem report
without relying on additional information such as source code changes.
Other approaches, such as the one from Ko et al. [8] analyze the linguistic
characteristics of problem report descriptions to improve the quality of these
reports by providing better tool support for problem report entry. Focusing
especially on the quality of bug reports, Hooimeijer et al. [9] assume that higher
quality bug reports are adressed more quickly than those with lower quality.
Based on this assumption they use various bug report attributes, e.g., severity,
readability, submitter reputation, etc., and predict if a bug report is adressed in
a given amount of time.
Who should fix this bug [10] by Anvik et al. uses data mining techniques
to suggest suitable developers to whom a bug should be assigned. The same
232 P. Knab, M. Pinzger, and H.C. Gall
authors describe in [11] the problems that arise when using open bug reposito-
ries, e.g., duplicates and irrelevant reports. From our experience with an inhouse
commercial bug repository we can generally observe similar problems, but less
frequently, since the process of bug submission and handling is controlled more
tightly.
Most of the presented additional information that researchers are extracting
from SCM systems could be integrated in a future version of our approach by
simple adding additional possibilities for coloring the polymetric views or adding
new sort orders.
7 Conclusions
Acknowledgments
This work was sponsored in part by the Eureka 2023 Programme under grants of
the ITEA project if04032. It was also supported by the Swiss National Science
Foundation under grants of the Project DiCoSA - Distributed Collaborative
Software Analysis (SNF Project No. 118063).
References
1. Tufte, E.R.: Envisioning Information. Graphics Press (May 1990)
2. Knab, P., Pinzger, M., Fluri, B., Gall, H.C.: Interactive Views for Analyzing Prob-
lem Reports. In: ICSM 2009 Proceedings of the 25th International Conference on
Software Maintenance, pp. 527–530 (2009)
3. Lanza, M., Ducasse, S.: Polymetric views — a lightweight visual approach to reverse
engineering. IEEE Transactions on Software Engineering 29(9), 782–795 (2003)
Visual Patterns in Issue Tracking Data 233
4. Fischer, M., Pinzger, M., Gall, H.: Populating a release history database from
version control and bug tracking systems. In: Proceedings of the International
Conference on Software Maintenance, Amsterdam, Netherlands, pp. 23–32. IEEE
Computer Society Press, Los Alamitos (2003)
5. D’Ambros, M., Lanza, M., Pinzger, M.: ”a bug’s life” - visualizing a bug database.
In: Proceedings of VISSOFT 2007 (4th IEEE International Workshop on Visualiz-
ing Software For Understanding and Analysis), June 2007, pp. 113–120. IEEE CS
Press, Los Alamitos (2007)
6. Halverson, C.A., Ellis, J.B., Danis, C., Kellogg, W.A.: Designing task visualiza-
tions to support the coordination of work in software development. In: CSCW
2006: Proceedings of the 2006 20th anniversary conference on Computer supported
cooperative work, pp. 39–48. ACM, New York (2006)
7. Weiss, C., Premraj, R., Zimmermann, T., Zeller, A.: How long will it take to fix this
bug? In: MSR 2007: Proceedings of the Fourth International Workshop on Mining
Software Repositories, Washington, DC, USA, p. 1. IEEE Computer Society, Los
Alamitos (2007)
8. Ko, A.J., Myers, B.A., Chau, D.H.: A linguistic analysis of how people describe
software problems. In: VLHCC 2006: Proceedings of the Visual Languages and
Human-Centric Computing, Washington, DC, USA, pp. 127–134. IEEE Computer
Society Press, Los Alamitos (2006)
9. Hooimeijer, P., Weimer, W.: Modeling bug report quality. In: ASE 2007: Pro-
ceedings of the twenty-second IEEE/ACM international conference on Automated
software engineering, pp. 34–43. ACM, New York (2007)
10. Anvik, J., Hiew, L., Murphy, G.C.: Who should fix this bug? In: ICSE 2006: Pro-
ceedings of the 28th international conference on Software engineering, pp. 361–370.
ACM, New York (2006)
11. Anvik, J., Hiew, L., Murphy, G.C.: Coping with an open bug repository. In: eclipse
2005: Proceedings of the 2005 OOPSLA workshop on Eclipse technology eXchange,
pp. 35–39. ACM, New York (2005)
Disruption-Driven Resource Rescheduling in Software
Development Processes
1 Introduction
Complex systems are typically comprised of a group of activities, each of whose
executions requires different resources that have various capabilities and availabil-
ities. Careful resource scheduling can reduce resource contention, thereby reducing
delays and inefficiencies, and in doing so, can increase the value of the system [1-3].
A lot of work has investigated different approaches to determining optimal sched-
ules of assignment of resources to system activities. One approach is static resource
scheduling [1, 2, 4-7], in which a complete schedule of resource assignment is com-
puted in advance based on advance knowledge of the sequence of activities to be
performed and the size and duration of all these activities [8, 9].
However, in disruption prone systems like those used in software development and
healthcare, activity estimates may unreliable, and uncertainties such as the sudden
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 234–247, 2010.
© Springer-Verlag Berlin Heidelberg 2010
Disruption-Driven Resource Rescheduling in Software Development Processes 235
The paper is organized as follows. Section 2 presents the TWINS framework. Sec-
tion 3 presents some details of the components and technologies used in our TWINS
implementation. Section 4 describes a simulation of a software development process
and reports on some case studies aimed at evaluating this approach. Section 5 presents
conclusions and suggests future work.
3 TWINS Implementation
3.1 Process Activity Definition
A complete, precise, and detailed system definition can improve the quality of the
resource scheduling need to support execution of the processes in that system. As in
[19], we also choose Little-JIL [21, 22] to define the system for which we will do our
scheduling. A Little-JIL process definition consists of a specification of three compo-
nents, an artifact collection (not described here due to space constraints), an activity
specification, and a resource repository. A Little-JIL activity specification is defined
as a hierarchy of steps, each of which represents an activity to be performed by an
assigned resource (referred to as its agent). Each step has a name and a set of badges
to represent control flow among its sub-steps, its interface, the exceptions it handles,
etc. A leaf step (one with no sub-steps) represents an activity to be performed by an
agent, without any guidance from the process. Each step specification also contains a
collection of resource requests. Each request in the collection is described by the
following definition.
Definition 1. Req = ( ResName, Capability1 , SkillLevel1 ,..., Capabilityr , SkillLevelr ) ,
where,
• ResName is the type of the resource being requested, (e.g. requirement analyst,
coder, tester), which indicates the kinds of capabilities that this resource has.
• Capabilityi is a capability that is being
requested.
• SkillLeveli is the minimum level of skill
in Capabilityi that is being requested.
Fig 2 shows a Little-JIL activity defini-
tion that defines a software development
process. Note that a different instance of
this process is instantiated for every new
project, and thus the work of a typical
development organization is represented
by the concurrent execution of several Fig. 2. Module development process
238 J. Xiao et al.
instances of this process. Each process needs the same types of resources, which must
be provided by one central resource repository. This sets up resource contention. Each
process instance is represented by the top step, “ProjectDevelopment”. The right
arrow in this step specifies that, in sequential order, a project is developed by re-
quirement analysis (RA), development (Develop), and testing (TST). Development is
further decomposed into design (DES), coding (COD), and writing test cases (WTC).
WTC is executed in parallel with DES and COD (represented by “=” sign).
The resource repository is the second key component of a Little-JIL process defini-
tion. It contains a set of resource instances that are available for assignment to the
tasks specified in the Little-JIL activity diagram.
Thus, ResourceRepositiory = {Res1 ,Res2 ,...,Resl } , where each element of this set has
certain capabilities and availabilities. A resource is defined as follows:
Res = ( ID , ResName,At tributes,S chedulable TimeTable, Capability 1 ,
Definition 2.
SkillLevel 1 ,Productiv ity 1 , Capability 2 , SkillLevel 2 , Productivi ty 2 , ...)
where,
• ID is a prose identification of the resource.
• ResName is the type of the resource, which indicates the kinds of capabilities that
this resource has, such as software analyst and designer.
• Attributes is a set of (name, value) pairs that describe the resource. Some example
attribute names might be Age, Experience_Level, and Salary_Rate.
• SchedulableTimeTable is a representation of the times when a resource is available
to be assigned to an activity. This consists of a set of schedulable time intervals, de-
fined by a start time (st) and end time (et). The resource can be assigned to an activ-
ity during any of the specified time intervals. Thus,
Schedulabl eTimeTable = {[ st 1 , et 1 ], [ st 2 , et 2 ],..., [ st s , et s ]}
• Capabilityi (i = 1, 2 …) is the ith kind of capability that the resource has to offer.
For example, a coder has the capability of using JAVA for coding.
• SkillLeveli (i = 1, 2 …) is the level of quality at which the resource is able to per-
form Capabilityi.
• Productivityi (i = 1, 2 …) is the average productivity that the resource is able to
achieve in performing Capabilityi.
Assume that an activity specifies that S is the quantity of Capabilityi required in order
to complete the activity. Then S/Productivityi, is the amount of time that resource R
requires in order to complete the activity. Only if this amount of time is contained
within R’s SchedulableTimeTable attribute, can R be assigned that activity.
The Scheduling Activity Set Constructor assembles all of the information needed in
order to make scheduling decisions. This function determines which upcoming activities
fall within the scheduling time window (whose size has been previously determined),
Disruption-Driven Resource Rescheduling in Software Development Processes 239
and assembles the activities into a graph called the Dynamic Flow Graph (DFG). The
DFG is derived from an analysis of another graph called the resource utilization flow
graph (RUFG). Space limitations prevent a full discussion of how these graphs are built
and used, and so we summarize descriptions of them. Details describing the DFG and
RUFG are provided in [23]
The RUFG is a directed graph, derived from a Little-JIL activity diagram, which
represents all the possible execution sequences of the process. Each node in the
RUFG represents a step in the Little-JIL process definition and is annotated with
precise and detailed information about the resource needs of the step. Thus, the RUFG
is a representation of the static structure of the process.
Our rescheduling approach, however, uses dynamic process state information to
take dynamic constraints and uncertainties into account. Thus when a rescheduling is
needed we use the static RUFG and dynamic state information to generate a dynamic
flow graph (DFG) that is then used as the basis for the rescheduling. The DFG incor-
porates both the RUFG’s static information and runtime state information such as
which activities are currently executing in parallel, which resources are allocated to
them, and the priority level assigned to each activity.
The size and shape of the DFG is determined by a specification of the time win-
dow, which dictates how many of the future execution possibilities are to be consid-
ered in the rescheduling. At present we define the size of a time window by an integer
that represents the length of the longest sequence of possible future activities that
make resource requests. Thus, specifically, if L is the integer used to define time win-
dow W and CURRACT is the set of activities that are currently being performed,
CURRACT = {activity1 , activity2 ,..., activityn } ,
then node NODE is included in W if and only if NODE requires the allocation of
resources, and, for some i, 1 ≤ i ≤ n , there is a path, P, in the RUFG
P = (activity i , n1 , n2 ,..., nk , NODE )
such that the number of nodes nj, 1 ≤ j ≤ k that make resource requests is less than L-1.
Further details about the definition of the RUFG and DFG are omitted due to space
constraints. In addition, note that other approaches may be used to define the schedul-
ing window, and the one just defined is used here as an example.
By being careful to keep time windows small we can assure that our scheduling prob-
lems remain relatively tractable. We nevertheless expect that our windows may still
be large enough to contain quantities of activities and resources that are sufficiently
large to require considerable scheduling computation. Thus in this work we have
extended the GA designed in [19] as the basis for each rescheduling.
rescheduling. For each DFG node, One DFG node (Step) Priority
N, in a time window (i.e. each node
waiting for the assignment of re- Request 1 …… Request m
Priority genes
for the node
sources), we establish a set of re- 0/1 0/1 ... 0/1 …… 0/1 0/1 ... 0/1 0/1 ... 0/1
source genes and a set of priority Size = smallest integer greater Size = smallest integer greater
Size = g
genes. Suppose N requires m than or equal to log B 2 1 than or equal to log B 2 m
• Identifying changes to a future activity: a future activity is an activity that has not
yet been executed. If during execution of one activity it is determined that changes
will cause a future activity to have insufficient resources allocated to it, then a re-
scheduling should be carried out.
• Arrival of new requirement: as soon as the need to meet a new requirement is
realized, a rescheduling to allocate resources to this new requirement must be car-
ried out.
• Un-assigned allocation of successive activities: once an activity has completed, if
any of its successor activities has not been allocated resources, a rescheduling
should be carried out.
Note that the first two of these items characterize unexpected runtime situations that
are necessitate relatively more disruptive reactive scheduling. The last two items
characterize predictable situations that trigger proactive rescheduling. This proactive
rescheduling is more desirable as its results should be expected to be less disruptive
and more accurate.
4 Evaluation
To support the analysis of the effectiveness of our approach, we used it to allocate
resources during the running of simulations of processes that define how software
project developments perform their activities and utilize their resources.
The different kinds of resources available in the case study include analyst, designer,
coder, and tester. They are described in Table 2. We use dates for SchedulableTimeTable
entries, and in each weekday, the available workload is eight person-hours, while on
Saturday and Sunday, the available workload is zero.
The trigger times of these events are the completion dates of RA. If the corresponding
changed activities have allocated resources at the trigger time, a new rescheduling
needs to be carried out.
The trigger times for these events are the planned end times of the activities.
executing activities and the future activities that are changed by these events are more
likely to be in the same window. Thus reschedulings caused by “identify the changes
of a future activity” increase as time window size increases.
• The number of “un-assigned allocation of successive activities” reschedulings
seems to decrease with increases in time window size. If the time window size in-
creases, before the start of execution of activities for which resources are not yet al-
located, there is a higher probability that new reschedulings caused by other kinds
of events may occur causing these future activities to lose their allocated resources.
Fig. 4. Number of different kinds of reschedulings and the total rescheduling time (results are
averages of 20 different simulations)
The above analysis shows that a small time window causes a lower number of re-
active reschedulings, but a larger number of proactive reschedulings. An appropri-
ately chosen time window size (3 in this example) causes a minimum number of
reschedulings. Our research is still quite preliminary, but it suggests that this optimal
window size may be context dependent and that more research is needed to under-
stand better what features and state information should be used (and how) to help
predict optimal window size.
relatively small time windows are likely to be most effective when there are a large
number of disruptions in systems, while large time windows are effective when a
system is less prone to disruptions.
16
14
Number of rescheduling
12
10
Window = 2
8
6 Window = 5
4
2
0
All projects have Six projects have Three projects No disruptions
disruptions disruptions have disruptions
Fig. 5. Comparisons of the effects of different number of disruptions (results are averages of 20
different simulations)
Acknowledgments
We would like to thank Bin Chen and Heather Conboy for their help with the trans-
formation from Little-JIL to RUFG, and Prof. Lori A. Clarke, Dr. M. S. Raunak, and
Sandy Wise for their valuable feedback about this work. This paper is supported by
the National Natural Science Foundation of China under grant Nos. 90718042,
60903051, the Hi-Tech Research and Development Program (863 Program) of China
under grant No. 2007AA010303, as well as the National Basic Research Program
(973 program) under grant No. 2007CB310802. This work was also supported by the
National Science Foundation under Awards No. CCR-0205575, CCR-0427071, CCF-
0820198, and IIS-0705772. Any opinions, findings, and conclusions or recommenda-
tions expressed in this publication are those of the author(s) and do not necessarily
reflect the views of the National Science Foundation.
References
[1] Alba, E., Chicano, J.F.: Software Project Management with GAs. Journal of Information
Sciences 177, 2380–2401 (2007)
[2] Xiao, J., Wang, Q., Li, M., Yang, Q., Xie, L., Liu, D.: Value-based Multiple Software
Projects Scheduling with Genetic Algorithm. In: Wang, Q., Garousi, V., Madachy, R.,
Pfahl, D. (eds.) ICSP 2009. LNCS, vol. 5543, pp. 50–62. Springer, Heidelberg (2009)
[3] Biffl, S., Aurum, A., Boehm, B., Erdogmus, H., Grünbacher, P.: Value-Based Software
Engineering. Springer, Heidelberg (2005)
[4] Barreto, A., Barros, M.d.O., Werner, C.M.L.: Staffing a software project: A constraint
satisfaction and optimization-based approach. Computer & Operations Research 35,
3073–3089 (2008)
[5] Pham, D.-N., Klinkert, A.: Surgical case scheduling as a generalized job shop scheduling
problem. European Journal of Operational Research 185, 1011–1025 (2008)
[6] Goncalves, J.F., Mendes, J.J.M., Resende, M.G.C.: A Genetic Algorithm for the Resource
Constrained Multi-project Scheduling Problem. European Journal of Operational Re-
search 189, 1171–1190 (2008)
[7] Peteghem, V.V., Vanhoucke, M.: A genetic algorithm for the preemptive and non-
preemptive multi-mode resource-constrained project scheduling problem. European Jour-
nal of Operational Research (2009)
[8] Fowler, J.W., Monch, L., Rose, O.: Scheduling and Simulation. In: Herrmann, J.W. (ed.)
Handbook of Production Scheduling, pp. 109–133. Springer, US (2006)
[9] Pfeiffer, A.s., Kadar, B., Monostori, L.s.: Stability-oriented evaluation of rescheduling
strategies, by using simulation. Computers in Industry 58, 630–643 (2007)
Disruption-Driven Resource Rescheduling in Software Development Processes 247
[10] Herrmann, J.W.: Rescheduling Startegies, Policies, and Methods. In: Handbook of Pro-
duction Scheduling, pp. 135–148
[11] Herroelen, W., Leus, R.: Project Scheduling under Uncertainty: Survey and Research Po-
tentials. European Journal of Operational Research 165, 289–306 (2005)
[12] Antoniol, G., Penta, M.D., Harman, M.: A Robust Search–Based Approach to Project
Management in the Presence of Abandonment, Rework, Error and Uncertainty. In: Pro-
ceedings of the 10th International Symposium on Software Metrics, pp. 172–183 (2004)
[13] Li, Z., Ierapetritou, M.G.: Robust Optimization for Process Scheduling Under Uncer-
tainty. Industrial and Engineering Chemistry Research 47, 4148–4157 (2008)
[14] Al-Fawzan, M.A., Haouari, M.: A bi-objective model for robust resource-constrained
project scheduling. International Journal of Production Economics 96, 175–187 (2005)
[15] Wang, J.: A fuzzy robust scheduling approach for product development projects. Euro-
pean Journal of Operational Research 152, 180–194 (2004)
[16] Ghezail, F., Pierreval, H., Hajri-Gabouj, S.: Analysis of robustness in proactive schedul-
ing: A graphical approach. Computers & Industrial Engineering (2009)
[17] Rangsaritratsamee, R., Ferrell Jr., W.G., Kurz, M.B.: Dynamic rescheduling that simulta-
neously considers efficiency and stability. Computers & Industrial Engineering 46, 1–15
(2004)
[18] Yang, B.: Single Machine Rescheduling with New Jobs Arrivals and Processing Time
Compression. International Journal of Advanced Manufacturing Technology 34, 378–384
(2007)
[19] Xiao, J., Osterweil, L.J., Wang, Q., Li, M.: Dynamic Resource Scheduling in Disruption-
Prone Software Development Environments. Department of Computer Science, Univer-
sity of Massachusetts, Amherst, MA 01003 UM-CS-2009-050 (2009)
[20] Holland, J.H.: Adaptation in natural and artificial systems. MIT Press, Cambridge (1992)
[21] Cass, A.G., Lerner, B.S., McCall, E.K., Osterweil, L.J., Stanley, J., Sutton, M., Wise, A.:
Little-JIL/Juliette: A Process Definition Language and Interpreter. In: Proceedings of the
22nd International Conference on Software Engineering, Limerick, Ireland, pp. 754–757
(2000)
[22] Wise, A.: Little-JIL 1.5 Language Report, Department of Computer Science, University
of Massachusetts, Amherst UM-CS-2006-51 (2006)
[23] Xiao, J., Osterweil, L.J., Wang, Q., Li, M.: Dynamic Scheduling in Systems with Com-
plex Resource Allocation Requirements. Department of Computer Science at the Univer-
sity of Massachusetts Amherst. Technical report: UM-CS-2009-049 (2009)
MODAL: A SPEM Extension to Improve
Co-design Process Models
ENSIETA
2 rue Francois Verny, 29806 Brest cedex, France
[email protected]
Abstract. Process engineering has been applied for several years in soft-
ware developments, with more or less success thanks to the use of process
languages easing analysis, improvement and even execution of software
processes. In this paper, we present a SPEM derived metamodel that
takes advantages of Model Based Engineering and innovative approaches
of process engineering. Through this work, we try to clarify the definition
of process components in order to be able to construct and execute MBE
processes “On-Demand”.
1 Introduction
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 248–259, 2010.
c Springer-Verlag Berlin Heidelberg 2010
MODAL: A SPEM Extension to Improve Co-design Process Models 249
SPEM that handle data management and retro-annotation issues. This exten-
sion allows iterative improvement of processes through automatic collect and
exploitation of process data.
Beyond continuous maturation, capitalization and reuse of process knowledge
is a key issue addressed by process engineering communities. Several works have
shown the relevancy to adapt software components to system and software pro-
cesses. The notion of process component have been introduced in the SPEM 2.0
specification. But according to [12], this definition is too basic and the author pro-
poses a clearer definition of process component that integrates some of the major
concerns of the MBE (model analysis, model transformations, etc.). In [13], the
authors discuss the specification of “MDA Tool Components” which is a process
component characterized by the services they provide to the external world.
This paper intends to make converge Process Engineering and MBE tech-
niques taking benefits of both communities. We suggest introducing notions of
intention and strategy into the SPEM metamodel through our MODAL exten-
sion. Besides, we will see that introduction of these concepts allows refinement
of process components definition which encapsulates all or any of the process
activities as well as definitions of related artifacts and constraints. We think
adopting a full MBE approach can help to automate part or possibly the whole
process through the clear identification of all design and analysis activities of
the process, including (meta-) models and transformation rules. Then, advan-
tages of MBE become clear as process models allow establishing: traceability
between requirements and system functions, critical path, relationships between
models, management of complexity through mastered transformations, etc. Fur-
thermore, an MBE approach can improve consistency of data exchanged between
involved domains through model transformations. Consequently, consistency be-
tween process development plan and its execution becomes implicit.
3 Introduced Concepts
In this section, we present the main concepts introduced to improve the definition
of the SPEM 2.0 metamodel and we illustrate them with excerpts taken from a
co-design process model targeting re-programmable platforms presented in [14].
Intention. Processes are the essence of the realization of a product. Depending
on the process community (Information System, Workflow, etc.), process def-
inition can be centered on products, activities or organization. In the context
of MBE, processes are centered on models. The notion of reason which guides
our choices is introduced under the concept of intention. Two domains are ac-
tually interested in the concept of intention: the artificial intelligence domain
(multi-agents systems) [15,16] and the information system domain [17].
The artificial intelligence is interested in the realization of programs by being
inspired by rational agents. An agent is said rational if he chooses to realize
actions based on what he knows about the world or his faiths [16]. In [15],
author proposes the BDI model (Belief, Desire and Intention) which advances
the process of reasoning which we use every day: Belief is what the agent knows
MODAL: A SPEM Extension to Improve Co-design Process Models 251
Then, constraints that guard execution of activities can use information related
to the state of the incoming artifact for a finer control. For instance, in figure 3,
we can see a use case model used in a process activity along with its lifecycle.
that their application could be relaxed to take into account specific business
needs. To this purpose, we associate a level of severity to constraints because in
some occasion, violation of a constraint can be accepted. Figure 3 shows strate-
gies that are guarded by several constraints expressed in KerMeta syntax and
the table 2 shows examples of such constraints.
Process Components. We have seen in section 2 that process components
offer several advantages to encapsulation and reuse of process knowledge. Un-
fortunately, the SPEM metamodel does not give a precise definition of process
components. Then, our idea is to take benefits from propositions presented above
to clarify the definition of process components. Figure 6 presents the abstract
syntax of the process component in our extension. A process component is a
concurrent entity communicating through connectors implementing high level
services. It is mainly characterized by its public and private parts. Whereas its
public part exposes all the services it offers to the outside world, its private part
MODAL: A SPEM Extension to Improve Co-design Process Models 255
deals with the realization of these services, even by delegation (use of services
provided by internal parts) or through activation of internal behaviors. More
precisely, its private part encapsulate definition of activities related to some
abstraction level, languages and associated tools in order to realize published
services. The use of models implies for the component the knowledge related to
corresponding metamodels.
Figure 7 shows the top level of the MoPCoM SoC / SoPC process compo-
nent which gathers the definition of the whole process through interconnected
parts. Those parts encapsulate definition of activities, constraints, models and
so on. For instance, the Abstract Modeling Level process component provides
the description of the expected level of concurrency and pipeline of the input
application model through the mapping of functional blocks onto an abstract
platform. The Execution Modeling Level process component provides the topol-
ogy of the execution platform of the application defined in terms of execution,
communication or storage nodes in order to proceed to coarse grain analysis.
4 Discussion
Using SPEM fits well to system or software process representation when captured
models are contemplative. For instance, the reference process implementation in
the EPF project is OpenUP (Open Unified Process). This model is considered
precise only because of the great number of provided details. Still, this model stay
informal and its strict application remains subject to interpretation. We have to
face the limits of SPEM when we try to achieve process enactment through
execution. Especially, we are not able to take advantages of MBE techniques to
automate processes, even partially. Of course, several propositions overcome the
shortcomings of SPEM (see section 2) but those solutions are local. Then, we
propose a more global approach that integrates those local solutions into large
processes.
Even if more experimentation is needed to validate our propositions, we can
yet make some interesting assessments. For instance, building large process mod-
els requires clear viewpoints facilitating process analysis. Such viewpoints in
SPEM can be achieved only thanks to a coarse decomposition of process activi-
ties. This approach does not facilitate analysis because an activity definition is
MODAL: A SPEM Extension to Improve Co-design Process Models 257
SPEM MODAL
Need Formalization ? Intention
Solution Formalization Activity Strategy
Work Product Usability No Yes
Constraint Usability No Yes
Activity Executability No Yes
Process Component Executability No Yes
align the definition of work product to model. This allows refinement of pro-
cess contraint and process component definition. MODAL models highlight the
strong correlation between processes, work products, languages and tools. This
extension contributes then to reinforce the semantics of process models. Proposi-
tions of the extension have been experimented on an industrial co-design project
in Thales Company, developing embedded systems. Our experiment shows that
SPEM, complemented by our proposition, can fit to processes involving several
heterogeneous businesses (system, software, platform, signals).
Future works will have to make effective the implementation of process com-
ponents in order to manage definition of tools chain which seems to be the only
way to manage the growing complexity of future development. Besides, the pro-
posed metamodel will have to provide mechanisms allowing customization of
existing process components to manage product lines.
Acknowledgements. This work is part of the MoPCoM SoC/SoPC project.
More information is available in http://www.mopcom.fr.
References
1. standishgroup: Standish group chaos report (2006)
2. Baldassarre, M.T., Boffoli, N., Bruno, G., Caivano, D.: Statistically based process
monitoring: Lessons from the trench. In: Wang, Q., Garousi, V., Madachy, R.,
Pfahl, D. (eds.) ICSP 2009. LNCS, vol. 5543, pp. 11–23. Springer, Heidelberg (2009)
3. OMG: SPEM 2.0. Technical Report ptc/05-01-06, Object Management Group
(2008)
4. Scacchi, W.: Process models in software engineering. Encyclopedia of Software
Engineering 13, 31–39 (2001)
5. Zhang, H., Kitchenham, B., Jeffery, R.: Qualitative vs. quantitative software pro-
cess simulation modeling: Conversion and comparison. In: ASWEC 2009: Proceed-
ings of the 2009 Australian Software Engineering Conference, Washington, DC,
USA, pp. 345–354. IEEE Computer Society, Los Alamitos (2009)
6. Jung, H.W., Goldenson, D.R.: Evaluating the relationship between process im-
provement and schedule deviation in software maintenance. Inf. Softw. Tech-
nol. 51(2), 351–361 (2009)
7. Osterweil, L.J.: Formalisms to support the definition of processes. J. Comput. Sci.
Technol. 24(2), 198–211 (2009)
8. Bendraou, R.: UML4SPM: Un Langage De Modélisation De Procédés De
Développement Logiciel Exécutable Et Orienté Modèle. PhD thesis, Université
Pierre et Marie Curie (2007)
9. Rolland, C.: A comprehensive view of process engineering. In: Pernici, B., Thanos,
C. (eds.) CAiSE 1998. LNCS, vol. 1413, pp. 1–24. Springer, Heidelberg (1998)
10. Bendraou, R., Combemale, B., Cregut, X., Gervais, M.P.: Definition of an exe-
cutable spem 2.0. In: APSEC 2007: Proceedings of the 14th Asia-Pacific Software
Engineering Conference (APSEC 2007), Washington, DC, USA, pp. 390–397. IEEE
Computer Society, Los Alamitos (2007)
11. Larrucea, X., Bozheva, T.: Towards an agile process pattern modeling framework.
In: SE 2007: Proceedings of the 25th conference on IASTED International Multi-
Conference, Anaheim, CA, USA, pp. 61–65. ACTA Press (2007)
MODAL: A SPEM Extension to Improve Co-design Process Models 259
12. Mekerke, F.: Structuration des modèles orientés métiers pour les systèmes embar-
qués. PhD thesis, Lisyc (2008)
13. Bézivin, J., Gérard, S., Muller, P.A., Rioux, L.: Mda components: Challenges and
opportunities. In: Workshop on Metamodelling for MDA, York, England, pp. 23–41
(2003)
14. Koudri, A., Champeau, J., Aulagnier, D., Soulard, P.: Mopcom/marte process ap-
plied to a cognitive radio system design and analysis. In: Paige, R.F., Hartman,
A., Rensink, A. (eds.) ECMDA-FA 2009. LNCS, vol. 5562, pp. 277–288. Springer,
Heidelberg (2009)
15. Bratman, M.E.: Intention, Plans, and Practical Reason. Harvard University Press,
Cambridge (1987)
16. Wooldridge, M.J.: Reasoning about Rational Agents. The MIT Press, Cambridge
(2000)
17. Rolland, C., Prakash, N., Benjamen, A.: A multi-model of process modelling. Re-
quirement Engineering 4, 169–187 (1999)
Application of Re-estimation in Re-planning of
Software Product Releases
Context: Re-planning of product releases is a very dynamic endeavor and new re-
search methods or improvements of existing methods are still required. This paper
explores the role of re-estimation in the re-planning process of product releases.
Objective: The purpose of this study is to analyze effects of defect and effort
re-estimation in the process of release re-planning. In particular, two questions
are answered: Question 1: In the absence of re-estimation, does conducting re-
planning have any advantages over not conducting re-planning? Question 2: In
the case of re-planning, does conducting re-estimation have any advantages
over not conducting re-estimation?
Method: The proposed method H2W-Pred extends the existing H2W
re-planning method by accommodating dynamic updates on defect and effort
estimates whenever re-planning takes place. Based on the updates, effort for
development of new functionality needs to be re-adjusted and balanced against
the additional effort necessary to ensure quality early. The proposed approach is
illustrated by case examples with simulated data.
Results: The simulation results show that conducting re-planning yields better
release value in terms of functionality than not conducting re-planning. Fur-
thermore, performing re-estimation when doing re-planning generates a portfo-
lio of solutions that help balance trade-offs between several aspects of release
value, e.g., between functionality and quality.
Conclusion: If the development of a product release requires balancing
between potentially conflictive aspects, such as quality vs. functionality, then
re-estimation in the re-planning process is beneficial.
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 260–272, 2010.
© Springer-Verlag Berlin Heidelberg 2010
Application of Re-estimation in Re-planning of Software Product Releases 261
2 Formalization of Re-planning
Re-planning of software product releases being in progress of implementation is
based on different types of information related to the existing plan, the change re-
quests, and the capacities and time interval of the release under investigation.
The re-planning process happens in a given release period denoted by [T1,T2]. The
implementation process of the planned release starts at t = T1. The planned release
date is t = T2. Each feature has some estimated effort for its implementation and each
release has a limited effort capacity CAP. The capacity limits the number of features
that can be implemented in the proposed plan.
This plan can be determined by any of the existing methods mentioned above. We
will later see that our greedy re-planning method can be used as well to determine an
initial plan.
Each feature f(n) for n = 1..N is characterized by a set of attributes. If attribute ef-
fort(n) describes the (estimated) effort for implementation of feature f(n), then we have
Σn=1..N x(n) effort(n) ≤ CAP (2)
In addition to effort, other attributes that we consider are risk(n) and value(n) that
represent a risk and a benefit value of the feature, respectively.
In the process of implementation of a once agreed upon plan, change request might
occur. A change request relates to changing an already accepted feature (some of the
data of its related attributes) or requesting a new feature. The set of change request is
dynamically changing, so we need to describe them in dependence of time.
We define by CR(t) the set of all change requests from the beginning of the release
period (t = T1) until point in time t ∈ (T1, T2]. The change requests themselves are
denoted by f(N+1), f(N+2), …. f(Nt) where Nt denotes the number of change requests
until point in time t. Each change request f(n), n = N+1 … Nt is characterized by four
attributes called risk(n), value(n), time(n) and effort(n).
Both risk and the feature’s value (from a stakeholder viewpoint) are classified on a
nine-point scale with values ranging from 1 (extremely low) to 9 (extremely high). A
change request f(n) is arriving at point in time t denoted by t = time(n) and is estimated
to consume effort in the amount of effort(n). Both the risk and the value estimation are
assumed to be the result of stakeholder (and expert) evaluation as described in [5].
Experts can estimate the level of risk for features from previous project history. For
estimation of effort, a variety of known methods and techniques can potentially be
applied. A method for estimation in consideration of change was studied in [6].
The “when to re-plan?” decides when to initiate re-planning. For simplicity, we use
fixed intervals for re-planning (i.e., re-planning is initiated at fixed points in time), to
study and analyze the impact of the frequency of re-planning for different scenarios.
Application of Re-estimation in Re-planning of Software Product Releases 263
For more flexible re-planning strategies we refer to Chapter 7 in [1]. The key parame-
ters for Step 1 are summarized in Table 4.
The “what to re-plan?” determines what to include in the re-planning process. Since
re-planning tries to produce up-to-date release plans by accommodating change re-
quests, the features planned but not yet implemented as well as change requests are
included in the re-planning process.
In addition to that, predictive models are applied at re-planning iterations for re-
estimating effort and defects as follows:
1. Effort re-estimation: since implementation of new features in previous itera-
tions update the system as well as the data repository, re-estimations in the
following iterations may perform better and converge towards the actual ef-
fort, which increases release plans quality.
2. Defect re-estimation: since early defect detection before system test and op-
erations is a cost-effective strategy for software quality improvement 7, we
apply defect prediction model at re-planning iterations to capture and remove
defects rather than waiting for all functional implementation to be finished.
The key parameters for Step 2 are summarized in Table 5.
Output
The set of features F1⊆ (F U CR(t)) for re-planning as well as their updated
effort estimates and number of predicted defects
The “how to re-plan?” describes the actual process of producing up-to-date release plans.
Candidate features and change requests are ranked based on their distance to the ideal
264 A. Al-Emran et al.
Output
⊆
The set of features F2 F1 representing the new up-to-date release plan as well
as resources allocated for fixing predicted defects.
To illustrate the working principle of our method, we present a simple case example
where the considered feature set is taken from real-world data. Both features and their
effort estimates were taken from the product release planning of the decision support
tool ReleasePlanner™ [10]. All the other data of this case example were generated
using synthetically.
f(n) effort(n) value(n) risk(n) f(n) effort(n) value(n) risk(n) f(n) effort(n) value(n) risk(n) f(n) effort(n) value(n) risk(n)
1 16 5 9 14 45 1 5 27 27 4 7 40 90 7 4
2 16 2 3 15 45 3 2 28 95 6 1 41 10 5 6
3 17 2 8 16 17 7 6 29 70 2 9 42 10 9 1
4 30 6 6 17 40 6 2 30 55 1 7 43 36 1 2
5 29 6 3 18 85 1 4 31 41 9 6 44 34 9 6
6 23 3 8 19 22 2 1 32 10 9 4 45 45 9 8
7 4 7 4 20 83 9 4 33 5 9 3 46 29 2 4
8 65 9 2 21 31 6 7 34 80 1 2 47 40 3 3
9 9 9 5 22 28 3 4 35 60 9 9 48 75 7 1
10 31 3 9 23 38 7 1 36 45 5 7 49 55 9 7
11 75 6 7 24 20 4 3 37 50 7 5 50 6 9 8
12 45 1 4 25 80 1 4 38 10 2 3
13 60 6 4 26 75 9 6 39 100 3 7
Application of Re-estimation in Re-planning of Software Product Releases 265
Table 5 provides the details of the baseline release planning (serving as a reference
point for subsequent re-planning). First, features are prioritized (c.f., Table 5(a))
based on the distance to the ideal point as discussed in [4]. In addition to features’
value and risk, the distance calculation (i.e., Dist(n)) is relying on their effort normal-
ized in 9-point scale (denoted by normEff(n)).
Afterwards, greedy optimization is applied that takes top-ranked features one by
one from the priority list (until T2) to determine baseline solution (c.f., Table 5(b)). In
this case, 24 features could be accommodated in the baseline plan that totals a func-
tional value of 156. This value is computed by adding up value(n) for all 24 features
that could be accommodated in the release. First half of the baseline plan i.e., until
implementation of feature f(44) has been scheduled in iteration I1 and rest of the
baseline plan has been scheduled in iteration I2.
Prioritization of Features based on Distance to Ideal Point Baseline Plan (I1) Baseline Plan (I2)
f(n) normEff(n) Dist(n) f(n) normEff(n) Dist(n) f(n) normEff(n) Dist(n) f(n) start(n) end(n) f(n) start(n) end(n)
42 1 0.00 13 6 6.56 43 4 8.60 42 0.00 1.43 37 46.00 53.14
33 1 2.00 21 3 7.00 40 9 8.78 33 1.43 2.14 4 53.14 57.43
32 1 3.00 22 3 7.00 1 2 9.00 32 2.14 3.57 48 57.43 68.14
7 1 3.61 47 4 7.00 11 7 9.00 7 3.57 4.14 41 68.14 69.57
23 4 3.61 50 1 7.00 6 3 9.43 23 4.14 9.57 13 69.57 78.14
9 1 4.00 19 2 7.07 12 5 9.43 9 9.57 10.86 21 78.14 82.57
5 3 4.12 49 5 7.21 35 6 9.43 5 10.86 15.00 22 82.57 86.57
17 4 4.36 15 5 7.28 14 5 9.80 17 15.00 20.71 47 86.57 92.29
8 6 5.10 38 1 7.28 3 2 9.95 8 20.71 30.00 50 92.29 93.14
16 2 5.48 2 2 7.35 10 3 10.20 16 30.00 32.43 19 93.14 96.29
24 2 5.48 20 8 7.62 34 8 10.68 24 32.43 35.29 38 96.29 97.71
31 4 5.83 26 7 7.81 30 5 10.77 31 35.29 41.14
Functional value: 156
44 4 5.83 46 3 7.87 18 8 11.05 44 41.14 46.00
37 5 6.00 27 3 8.06 25 8 11.05
4 3 6.16 45 5 8.06 39 9 11.66
48 7 6.32 36 5 8.25 29 7 12.21
41 1 6.40 28 9 8.54
(a) (b)
266 A. Al-Emran et al.
We apply two analogy based method for effort and defect re-estimation that uses latest
information on already developed features in the previous iterations. For effort re-
estimation, we apply the method AQUA [11] using estimation by analogy and learning.
For defect re-estimation, we apply a case-based reasoning technique [12]. The re-
estimation based re-planning approach of the paper is applicable to alternative tech-
niques as well.
Table 6 shows the evolution of effort estimates in iterations I1 and I2. Note that,
some of the estimation values are missing. This is due to the fact that either features
have been realized already before further re-estimation or they have never been realized.
For example, initial effort estimation of f(5) for iteration I1 was 29pd (i.e, person-days),
however, it took 39.88pd (which is actual effort or AE). Since f(5) is implemented
within I1, it was required to re-estimated its effort for I2. On the other hand, f(1) has
never been implemented. That is why AE could never be known and effort was esti-
mated/re-estimated in each iteration. Effort of F(50) was estimated as 6pd for I1, then
re-estimated as 4.44pd for I2, and finally implemented in I2 with 4.62pd.
We simulate the random occurrence of change requests by the customer i.e., arrival of
new features at different points in time within the release duration (T1, T2). In this
illustrative case, they are: f(51), f(52), ..., f(63). The corresponding attribute values
are also randomly generated and the value ranges for their attributes are the same as
the features in the initial feature set (e.g., 9 point scale). The arrival time of change
request, time(n), is equally distributed over the period (T1, T2]. Table 7(a) provides
all information about the simulated data for the newly arrived features as well as their
effort evolution over iterations.
The development of the release follows the realization of features in the same order as
they are in the baseline plan (c.f., Table 5 (b)). After the first iteration i.e., I1 is com-
pleted, 13 features have been implemented as shown in Table 7(b). By comparing
completed iteration I1 with the baseline iteration I1 (c.f., Table 5(b)), we can see that
the same features are implemented that were planned in the baseline since neither
re-planning took place nor re-estimation is conducted. The only differences lie on the
Application of Re-estimation in Re-planning of Software Product Releases 267
development times of the features i.e., in start(n) and end(n). This is because, actual
efforts (i.e., AE of Table 7(a)) are used since actual efforts are now known as these
features are already realized.
At the end of the first iteration, two predictive models are applied to re-estimate fea-
ture efforts for the next iteration and unknown defects generated from previously imple-
mented iterations. Afterwards, re-planning event takes place for allocating available
efforts to rest of the release based on all project updates. These updates include adjust-
ment of feature effort (by effort prediction model), additional QA effort for detected
defects (by defect prediction model) to be removed, and newly arrived features that have
been arrived until end of iteration I1 (c.f., shaded portion of Table 7(a)). Upper part of
Table 7(c) provides the updated plan for iteration I2 – first, additional QA activity is
scheduled, and then three features are planned based on distance to ideal point and avail-
able effort. The re-planning event accommodates two of the newly arrived features (i.e.,
f(58) and f(52)). f(4) is the only feature from the baseline plan that could be included in
second (also last) iteration I2. Lower part of Table 7(c) shows completed iteration I2
where f(50) could also be accommodated due to overestimation of some features’ efforts.
Change Requests and their Effort Re-estimations Completed Iteration I1 Re-planned Iteration I2
f(n) effort(n) value(n) risk(n) time(n) I1 I2 AE f(n) start(n) end(n) f(n) start(n) end(n)
51 89.97 6 4 11 89.97 62.08 42 0.00 1.88 Add.
50.76 83.76
52 45.19 9 6 15 45.19 37.98 33 1.88 2.79 QA
53 48.16 7 6 19 48.16 67.65 32 2.79 4.55 58 83.76 90.51
54 95.00 4 7 22 95.00 94.40 7 4.55 5.32 4 90.51 93.25
55 89.26 3 5 31 89.26 98.00 23 5.32 10.78 52 93.25 99.71
56 95.00 7 9 34 95.00 98.00 9 10.78 12.05 Completed Iteration I2
57 96.00 2 8 41 96.00 89.23 5 12.05 17.74 f(n) start(n) end(n)
58 47.25 9 4 43 47.25 37.80 17 17.74 23.12 Add.
50.76 83.76
59 95.25 9 9 61 95.25 95.00 8 23.12 32.63 QA
60 56.80 3 8 79 56.80 90.72 16 32.63 35.77 58 83.76 89.16
61 14.05 8 7 83 14.05 12.84 24 35.77 38.40 4 89.16 92.63
62 28.75 4 3 84 28.75 21.67 31 38.40 44.96 52 92.63 98.06
63 20.67 4 4 88 20.67 27.60 44 44.96 50.76 50 98.06 98.69
The case example summary compares the final results with the baseline plan to report
what we have gained and what we have loose as shown in Table 8.
In this particular case, 33 unknown defects could be discovered and removed early.
However, with a cost of 231 person-days effort dedicated towards additional QA activities
(other than regular QA activities like testing after functional development). Since the total
effort capacity is fixed (700 person-days), the development effort is reduced by the same
amount as the increase in QA effort. This reduction of development effort results in seven1
baseline features to be discarded from the release which in turn reduces functional value.
1
Two other baseline features are discarded in exchange of two new feature inclusions.
268 A. Al-Emran et al.
5 Simulation Experiment
In this section, we perform an analysis based on simulated data for the same project as
studied in Section 4. We performed 1000 simulation runs for different iteration dura-
tions with randomly generated data in order to explore the impact of re-estimation.
For each simulation run, 50 initial features were generated with risk and value attrib-
utes in a 9-point scale and with effort attribute up to 100 person-days. In each simula-
tion run, a total of 7 developers and a maximum of 700 efforts capacity (person-days),
and 100 days of make-spans are considered. Some of the features from the original set
with the shortest distance to the ideal point are selected for the baseline plan.
Without loss of generality, the numbers of new incoming features is assumed to be
one quarter of the number of initial features, and the new features can arrive at any-
time during the release development applying distribution UNIFORM(0, 100). As
part of the analysis, three cases (scenarios) were studied for the varying data:
CASE A: No re-planning and no re-estimation.
CASE B: Allows re-planning of features without re-estimation.
CASE C: Allows re-planning of features with re-estimation.
Based on these cases, two key questions were analyzed:
Question 1: In the absence of re-estimation, does conducting re-planning have
any advantages over not conducting re-planning?
Question 2: In the case of re-planning, does conducting re-estimation have any
advantages over not conducting re-estimation.
Answering Question 1 requires comparison between CASE A and CASE B
whereas answering Question 2 requires comparison between CASE B and CASE C.
Table 9 provides simulation results where each row represents summary of 1000
simulation runs. The first two columns are used for classification purpose whereas the
subsequent columns report average simulation outcomes. The very first column
groups simulation results corresponding to CASE A (i.e., first data row), CASE B
(i.e., middle three data rows), and CASE C (i.e., last three data rows). CASE A has
only one data row since no re-planning is considered and features are realized within
Application of Re-estimation in Re-planning of Software Product Releases 269
Three data rows of CASE B are compared here with one data row of CASE A in Ta-
ble 9. CASE B consists of three scenarios with 2, 5, and 10 iterations equally divided
within release duration (i.e., 100 working days). Re-planning event takes place at the
end of all iterations except the last one. Therefore, first, second, and third scenarios
consist 1, 4, 9 re-planning events, respectively.
By comparing CASE A with CASE B in Table 9, CASE B appears to be better in
terms of functional value. The reason behind such result is that re-planning permits
exchange of features i.e., replacing once declared features by some of the newly arriv-
ing features that are comparatively more attractive in terms of “distance to ideal point”.
discarded from baseline plan, less features to be implemented, and hence, less func-
tional value being offered.
The earlier defects are detected and removed, the less the cost relatively to the total
project effort [7]. Results of CASE C support making trade-off decisions. In depend-
ence of the relative value of new functionality and quality (reduced number of de-
fects), the product manager can suggest the most appropriate compromise decision.
methods potentially applicable for these purposes, we demonstrate the principal bene-
fit of re-planning in general and of re-planning exploiting re-estimation in particular.
Currently, no product release re-planning method is known for the same purpose.
The effectiveness of the method was tested for a re-planning problem with data
generated synthetically. Under the assumptions made, it was shown that re-planning
improves functional value of a product. In addition, it was shown that application of
re-estimation is more beneficial since this balances the total effort available between
creating new functionality and ensuring a high quality (lower number of remaining
defects) type of product. In dependence of the product quality demands and the im-
pact of delivering faulty products to the customers, the product manager can make an
informed trade-off decision between functionality and fewer defects (based on higher
effort spent on quality assurance).
The results of this paper are considered part of an ongoing effort to offer decision
support in the context of software release planning [1]. Although the preliminary
results are promising, there is outstanding future research to be done. First of all, we
look for a real-world project able to provide data for both effort and defect prediction
in an incremental development project. Second, we need to fully integrate specific
prediction methods into H2W-Pred. Third, the quality of the proposed solutions could
be improved by replacing the current greedy optimization with better, but more com-
plex optimization procedures.
Acknowledgement
Part of the work presented was financially supported by the Natural Sciences and
Engineering Research Council (NSERC) of Canada under Discovery Grant no.
250343-07. Ahmed Al-Emran would like to thank iCORE, NSERC, and the Killam
Trust for their Postgraduate/pre-doctoral scholarships and to ExtendSim for support-
ing implementation of the research prototype.
References
1. Ruhe, G.: Product Release Planning – Methods, Tools and Applications. CRC Press, Boca
Raton (appear in 2010)
2. Wiegers, K.: Software Requirements. Microsoft Press (2003)
3. Lamsweerde, V.A.: Requirements engineering in the year 00: a research perspective. In:
22nd International Conference on Software Engineering, ACM Press, New York (2000)
4. Jadallah, A., Al-Emran, A., Moussavi, M., Ruhe, G.: The How? When? and What? for the
Process of Re-Planning for Product Releases. In: Wang, Q., Garousi, V., Madachy, R.,
Pfahl, D. (eds.) ICSP 2009. LNCS, vol. 5543, pp. 24–37. Springer, Heidelberg (2009)
5. Ngo-The, A., Ruhe, G.: Optimized Resource Allocation for Software Release Planning.
IEEE Transactions on Software Engineering 35, 109–123 (2009)
6. Ramil, J.F.: Continual Resource Estimation for Evolving Software. In: International Con-
ference on Software Maintenance 2003, pp. 289–292. IEEE Press, Washington DC (2003)
7. Khoshgoftaar, T.M., Ganesan, K., Allen, E.B., Ross, F.D., Munikoti, R., Goel, N., Nandi,
A.: Predicting fault-prone modules with case-based reasoning. In: 8th International Sym-
posium on Software Reliability Engineering, pp. 27–35. IEEE Press, Los Alamitos (1997)
272 A. Al-Emran et al.
8. Steuer, R.E.: Multiple Criteria Optimization: Theory, Computation, and Application. John
Wiley, New York (1986)
9. Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms. The
MIT Press, Cambridge (2007)
10. ReleasePlannerTM, Expert Decisions Inc., http://www.releaseplanner.com
11. Li, J., Ruhe, G., Al-Emran, A., Richter, M.M.: A Flexible Method for Effort Estimation by
Analogy. Journal Empirical Software Engineering 12(1), 65–106 (2007)
12. Paikari, E.: Analogy based Defect Prediction Model. Technical Report, Software Engineer-
ing Decision Support Laboratory, University of Calgary, SEDS-TR-086/2009
13. Bagnall, A.J., Rayward-Smith, V.J., Whittley, I.M.: The Next Release Problem. Informa-
tion and Software Technology 43, 883–890 (2001)
14. Jung, H.W.: Optimizing value and cost in requirements analysis. IEEE Software 15, 74–78
(1998)
15. Van den Akker, M., Brinkkemper, S., Diepen, G., Versendaal, J.: Software product release
planning through optimization and what-if analysis. Information and Software Technol-
ogy 50, 101–111 (2008)
16. Stark, G., Skillicorn, A., Ameele, R.: An Examination of the Effects of Requirements
Changes on Software Maintenance Releases. Journal of Software Maintenance: Research
and Practice 11, 293–309 (1999)
17. Al-Emran, A., Kapur, P., Pfahl, D., Ruhe, G.: Studying the Impact of Uncertainty in Op-
erational Release Planning - an Integrated Method and its Initial Evaluation. Information
and Software Technology 52, 446–461 (2010)
18. AlBourae, T., Ruhe, G., Moussavi, M.: Lightweight Re-planning of Software Product Re-
leases. In: 14th IEEE International Requirements Engineering Conference, pp. 27–34.
IEEE Press, Washington DC (2006)
Software Process Model Blueprints
1 Introduction
There is a generalized agreement among software practitioners about the rele-
vance of counting on a well defined process model for systematizing development.
There have been several efforts in aiding organizations toward this goal: maturity
models and standards that describe which elements should be part of a software
process, notations that allow rigorous specification, and tools that support these
notations. However, having a well defined software process does not necessarily
mean having a good process.
It is not apparent how to determine if a software process is good, or if it can be
improved in any sense before it is enacted [16]. The software process metamodel
SPEM 2.0 [15] proposes wellformedness rules for software process definition but
their scope is limited. For example, SPEM 2.0 does not determine if for a given
process some of the roles are overloaded, if there are work products that are
bottlenecks, or if there is no clear termination condition for a task cycle. What
is even more difficult about these issues is that they do not always constitute
errors, but they are indicators that something may not be right [10]. Even though
there are some metrics defined for measuring software processes [2], they provide
The work of Julio Hurtado Alegrı́a has been partly funded by NIC Chile.
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 273–284, 2010.
c Springer-Verlag Berlin Heidelberg 2010
274 J.A.H. Alegrı́a et al.
little intuition about what may be wrong, where the error is, and how to find
opportunities for improvement when there is no apparent error. How can we
present a rich software process on mere flat diagrams? Software processes should
represent a complex, dynamic and multidimensional world. Moreover, available
tools are usually intended only for software process visual specification, and not
for evaluation, let alone visual evaluation.
The convenience of specifying the software architecture with multiple views
has been agreed upon [3]. Different stakeholders require different information
for evaluations, thus it seems natural to deal with multiple process architectural
views [11]. We specify our process models using the SPEM 2.0 standard. Pro-
cess architectural views are built following an architectural recovery approach
such that they allow us to evaluate different process characteristics. We propose
three process architectural view types: Role Blueprint, Task Blueprint and
Work Product Blueprint. These blueprints are process views that make use
of metrics computed from a process model for easing understanding and evalu-
ating different aspects not available in SPEM 2.0 and its associated tools. These
visualizations have been implemented in the ProcessModel tool1 based on Moose
technology and the Mondrian tool. Visualization aids in data analysis and prob-
lem detection [14]. Using different concerns about process models in a graphical
manner, we make use of the human visual capacity for interpreting the process
in context and to evaluate its coherence and suitability [13].
We have applied our approach to the process model specification of DTS, a
real world small company that develops software for retail. We have been able to
identify several problems in the process and the company’s stakeholders agreed
that they were actual problems. The process has been redefined and the changes
are currently being applied.
2 Related Work
In Process Model Testing, process models are checked against their specifica-
tions [16]. An example of process testing is a software process assessment, where
a process and its corresponding model are evaluated based on a capability matu-
rity framework. This approach is present in CMMI [19] and ISO/IEC15504 [9].
This kind of testing activities can only be carried out once the process model
has already been implemented, tailored and enacted. It checks for adherence to
a standard but it does not evaluate the appropriateness of the process for the
organizational or project needs. Process testing requires very long time cycles.
Gruhn [7] has proposed a verification technique based on simulation results and
execution trace evaluation. Simulation has a shorter cycle, but it still requires en-
actment data. This is an appropriate verification technique if the process model
is known to be suitable for the environment, but if the model is incomplete,
underspecified or it is not suitable for the organization, the process model sim-
ulation will not yield the expected results.
Cook and Wolf [4] present a formal verification and validation technique for
identifying and measuring the discrepancies between process models and actual
executions. They do not address suitability, completeness or consistency of the
process model. Pérez et al. [17] suggest to evaluate the congruence between the
process model and a given environment based on past data. However, obtaining
these measurements is hard and the results are not precise. Formal specifications
and checking based on Petri Nets in a multi-view approach are presented in [5],
but formal checking has semantic limitations. Our process model blueprints use
SPEM 2.0 models to recover and visualize them independently using some basic
metrics, and the evaluation can be delegated to process designers in an informal
but structured and practical way.
Software inspections, walkthroughs, and technical reviews, are widely used to
improve software quality, so they are promising techniques for software processes
too. The key idea is that a group of experienced software engineers, meeting
together to review a software product, can improve its quality by detecting
defects that the product designer or developer could have overseen. Our proposal
consists of a tool that supports the software process model review based on visual
mechanisms. According to [18], the review process has three main tasks: defect
discovery, defect collection, and defect discrimination. The defect discovery is the
one that demands the highest expertise. The approach and tool presented in this
paper is intended to facilitate the process model defect discovery in a visual way
inside a planned verification and validation context. Also, the specific solution
presented in this paper can be useful for a process designer to gain some insights
about the process model design and potentially find some problems.
There is no unique perfect view to visually render a software process model [10].
As for most engineering activities, defining a robust, efficient and multi-view
software process is a non-trivial activity that requires flexible and expressive
tools. As a complement to software process design tools, we propose a visual
approach to evaluate some quality criteria. We introduce the notion of blueprint
as a partial but focused graphical representation of a software process model.
when lightweight roles are expected or vice versa, an isolated role when collabo-
ration is required, and roles without training material when it is necessary for a
good performance. Therefore, a complex role could be decomposed, simple roles
could be integrated, training material could be added, collaborative techniques
and enactment constraints could be added.
areas, indicating a high coupling between main process components. The tasks
on the upper left represent the management process area, those on the upper
right correspond to engineering process area and those on the bottom belong
to testing. These process areas, interpreted as process components, should be
connected with as few dependencies as possible using process ports as SPEM2.0
suggests; they can be used with composed work products and reports gener-
ated from other work products between main or facade tasks inside each process
component. Related small tasks can be redefined as steps of a larger task.
Interpretation: this blueprint allows us to evaluate if task granularity is appro-
priate for process requirements. It enables the discovery of a complex task, or a
disconnected task or task sub graph. A complex task could be decomposed and
simple tasks could be integrated. Similarly, this blueprint can be used to find
SPI opportunities and misspecifications such as high coupling, final or initial
tasks with next or previous tasks, and tasks without assigned roles.
DTS’s development process has been defined as a SPEM 2.0 model and imple-
mented in EPF. Even though some evaluations such as CMMI appraisals were
included, there is still no information about the quality of the process model itself
or the adequacy with the company’s goals. As part of its SPI project DTS wants
to assess the quality of its process, so they used the Process Model Blueprints.
The problems found were used as input for the following improvement cycle.
282 J.A.H. Alegrı́a et al.
The evaluation process was carried out by two process engineers, one from DTS
and another one from the University of Chile, and two software engineers for
process definition. The steps followed were:
(1) Visualization: the process engineers imported the model from EPF and they
realized a short review. Some errors in the tool were identified and some aspects
were tuned. A first visualization of the blueprints was generated.
(2) Potential problem identification: for each blueprint problems were identified.
Each problem was recorded and a trivial analysis was realized in this step.
(3) Problem analysis: each blueprint was used to find a symptom of a prob-
lem and the related elements. For example, Analysis was not related to Client
and Chief Engineer roles, so it is possible that some tasks where these roles par-
ticipated were not adequately specified. Some problems required more analysis
and discussion with users of the software process.
(4) Data collection: the potential and actual problems were collected, and also
some solutions were proposed.
(5) Data analysis: the data collected in this study case was analyzed and the
main results are presented below.
Table 2 summarizes the main problems found during the evaluation. For a
total of 132 process elements, 47 actual problems were found (false positives have
been discarded). So, the process model error rate was of 0.356 errors per model
element. In general, the review found that there were few guidances associated
to model elements. This situation suggests low formality in software process
specification. Disconnected elements cause modeling problems: some connections
were omitted at modeling time and others caused serious conceptual problems
in the software process. For example, the isolated Client role was a symptom
of a problem, and actually a big problem: the client was not involved in many
tasks where his/her participation is required, specially in requirements tasks. We
suggest two kinds of improvements:
Software Process Improvements: the main suggested improvements are: in-
crease guidance for roles (guide of work) and for work products (templates,
examples and concepts); task re-design for getting tasks of medium size (neither
too large nor too small), such as Requirements Review ; big work products decom-
position such as System Requirements Specification; decrease coupling between
Software Process Model Blueprints 283
process areas, with few relationships between work products and tasks; increase
the client participation and improve the collaboration required between the An-
alyst and Engineer Manager roles. The bottleneck risks identified on the System
Requirement Specification, Alpha Component and Alpha System Cycle n work
products are normal bottlenecks because of the incremental nature of the soft-
ware process, so these work products are completed and refined in each cycle of
the software process. However, the Architecture work product and its associated
tasks are not suitable to this incremental development.
Software Process Model Specification Improvement: the main improve-
ments to software model specification are about missing nodes, edges and guid-
ances. Several problems were identified directly, others required more analysis as
collaboration between specific roles, and others were generally due to conceptual
problems of the software process.
being applied in other small companies in Chile. Also the tool is being improved,
and other blueprints can be defined based on other concepts and elements defined
in SPEM 2.0, or based on other metrics.
References
1. Bendraou, R., Jezéquél, J.-M., Fleurey, F.: Combining Aspect and Model-Driven
Engineering Approaches for Software Process Modeling and Execution. In: Wang,
Q., Garousi, V., Madachy, R., Pfahl, D. (eds.) ICSP 2009. LNCS, vol. 5543,
pp. 148–160. Springer, Heidelberg (2009)
2. Canfora, G., Garcı́a, F., Piattini, M., Ruiz, F., Visaggio, C.A.: A family of exper-
iments to validate metrics for software process models. Journal of Systems and
Software 77(2), 113–129 (2005)
3. Clements, P., Bachmann, F., Bass, L., Garlan, D., Ivers, J., Little, R., Nord, R.,
Stafford, J.: Documenting Software Architectures. Addison-Wesley, Reading (2002)
4. Cook, J.E., Wolf, A.L.: Software process validation: quantitatively measuring the
correspondence of a process to a model. ACM TOSEM 8(2), 147–176 (1999)
5. Ge, J., Hu, H., Gu, Q., Lu, J.: Modeling Multi-View Software Process with Object
Petri Nets. In: ICSEA 2006, p. 41 (2006)
6. Gı̂rba, T., Lanza, M.: Visualizing and characterizing the evolution of class hierar-
chies. In: WOOR 2004 - 5th ECOOP (2004)
7. Gruhn, V.: Validation and verification of software process models. In: Proc. of the
Software development environments and CASE technology, pp. 271–286 (1991)
8. Humphrey, W.S.: Managing the Software Process. Addison-Wesley, Reading (1989)
9. ISO./IEC 15504 : Information technology - software process assessment and im-
provement. Technical report, Int. Organization for Standardization (1998)
10. Jacobs, D., Marlin, C.: Multiple view software process support using the Multi-
View architecture. In: ISAW-2 and Viewpoints 1996, pp. 217–221. ACM, New York
(1996)
11. Jacobs, D., Marlin, C.D.: Software process representation to support multiple
views. IJSEKE 1995 5(4), 585–597 (1995)
12. Lanza, M., Ducasse, S.: Polymetric Views-A Lightweight Visual Approach to Re-
verse Engineering. TSE 2003 29(9), 782–795 (2003)
13. Larkin, J.H., Simon, H.A.: Why a Diagram is (Sometimes) Worth Ten Thousand
Words. Cognitive Science 11(1), 65–100 (1987)
14. Lee, M.D., Butavicius, M.A., Reilly, R.E.: An Empirical Evaluation of Chernoff
Faces, Star Glyphs, and Spatial Visualizations for Binary Data. In: APVis 2003,
pp. 1–10. Australian Computer Society Inc. (2003)
15. OMG. Software Process Engineering Metamodel SPEM 2.0 OMG. Technical Re-
port ptc/08-04-01, Object Managemente Group (2008)
16. Osterweil, L.: Software processes are software too. In: ICSE 1987, Los Alamitos,
CA, USA, pp. 2–13. IEEE Computer Society Press, Los Alamitos (1987)
17. Perez, G., El Emam, K., Madhavji, N.H.: Evaluating the congruence of a software
process model in a given environment. In: ICSP 1996, p. 49 (1996)
18. Sauer, C., Jeffery, D.R., Land, L., Yetton, P.: The effectiveness of software de-
velopment technical reviews: a behaviorally motivated program of research. IEEE
Transactions on Software Engineering 26(1), 1–14 (2000)
19. SEI. CMMI for Development, Version 1.2. Technical Report CMU/SEI-2006-TR-
008, Software Engineering Institute (2006)
Measurement and Analysis of Process Audit:
A Case Study
1 Introduction
Process audit, or conformance audit [1] with checklists, is a widely used software qual-
ity assurance activity for organizations conducting process management and improve-
ment. The goal of process audit is to assess the conformance of the process execution
against the process definition or standards with the direct results of Not Compatible
(NC) items. It may help individual projects in compliance with the defined process and
standards and provide insight for process improvement. Process audit has two active
modes of operation: appraisal and analysis [2]. As for the appraisal mode, auditors use
checklists to evaluate the implementation of a process to make sure it in compliance
with the process description, standards, and procedures, reflecting the visibility of the
project process in time. In the analysis mode, auditors question the process definition,
standard, etc, which are used in support of the processes.
The assessment of effectiveness and efficiency is one of the key issues for mature
and successful process audit. However, there have been few researches or reports
about the assessment of process quality assurance or assessment related processes
[1, 3, 4, 6]. J.M. Gasston et al. [3] presented a conceptual model for the assessment of
the effectiveness of software processes and reported an empirical case study; while
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 285–296, 2010.
© Springer-Verlag Berlin Heidelberg 2010
286 F. Shu et al.
they didn’t use quantitative measurement and the case study is about the configuration
management process. Timo Makinen et al. [4] introduced a project of establishing a
software process improvement network, during which the assessment process was
assessed and 20 propositions to improve the assessment process came out. R. Craig
Smith [6] introduced the experiences of using IEEE Std 1028 [5] as the basic standard
for developing the audit program and focused on the challenges and lessons learned
by using this IEEE Std., CMM, and the IS0 9001 standard from a process assurance
perspective. Alain April et al. [1] presented a perspective on measuring process
assurance and lessons learned from the assessment of the individual conformance of
projects to the Corporate Software Development Process of an organization aiming at
CMM level 2. The reasons associated with some major deviations were analyzed but
without quantitative analysis reported.
In the paper, we introduce the measurement and analysis of the process audit
practice of a Chinese software development organization, CSRD. To the best of our
knowledge, it is the first empirical report about the quantitative assessment of the
process audit in an industrial context with the process maturity of CMMI level 3.
The paper is organized as follows. Section 2 introduces the context; section 3 pre-
sents the measurement schema of process audit; section 4 illustrates the measurement
and analysis of process audit; section 5 discusses the treats to validity and lessons
learned; and section 6 concludes the paper and introduces our future work.
2 Context
CSRD began to audit process from year 2005 when moving towards CMMI ML3 and
passed the appraisal the same year. It has an independent quality assurance group and
takes process compliance as an important management goal. The core products of
CSRD are the series of SoftPM, a toolkit designed for software process management
and process improvement [7]. CSRD itself uses SoftPM to define its processes,
maintain process assets, monitor and control projects, and collect and analyze process
data for continuous process improvement, etc.
The current checklists set of CSRD covers processes classified as four categories in
accordance with CMMI v1.2 [8]. For a project, its process audit should be planned at
the planning phase, including the processes to be audited and corresponding audit
criteria and timing. A project consists of multiple processes and each process may be
audited for several times.
The execution of the audit of a specific process of a project includes following
steps. First, the auditor pinpoints detailed checking items, evaluates the expected
completion of outputs, and arranges the interviews. Second, the auditor conducts the
interviews and checks the work products of the process according to each selected
checking item. Subsequently, besides the summarized information of the audit
execution, the auditor reports the detailed attributes of each NC, such as the
description and severity; and then these NCs are verified through the communication
with the project team. Once verified, the NC becomes a problem and corresponding
corrective actions should be defined. The verification and correction of the NC are
called as the disposal of the NC and the results should also be recorded. Track records
of all NCs should be kept and the auditor closes the report when the disposal of all
NCs is cleared and let involved people know that the items are closed.
Measurement and Analysis of Process Audit: A Case Study 287
Table 1. Metrics for the Measurement of effectiveness and efficiency of process audit
Table 2 lists the top 3 most often NCs of each process, and problems due to the proc-
ess definitions and standards include:
N1: The configuration items IDs are in chaos due to no formal naming standard.
N2: There is no user requirements validation process for customization projects.
N3: Task planning documents are inconsistent with the Gantt Graphs split on
SoftPM.
N4: The practical efficiency of the unit testing standard is poor.
N5: The coding standard is infeasible.
N6: The requirements of the unit testing standard are beyond the capability of the
tester and impractical for tight schedule.
N7: It is not required that the changes of requirements must be confirmed and its
impact should be analyzed before they are approved.
N8: There is no configuration management process for periphery developing tools.
4.2 Comparison of Metrics with Respect to the Auditors and Audited Processes
and Projects
The comparisons are conducted on two sets of metrics. One consists of the metrics
from M3 to M10 at the levels of audit execution and process of a project; the other
includes the metrics about NCs and problems, M11 and M12.
Measurement and Analysis of Process Audit: A Case Study 289
Fig. 1. Prop. of the number of audit executions, NCs, problems, and serious NCs of each process
Fig. 2. The mean value of the number of NCs, problems, serious NCs and the mean value of
ProCost, ProEff, ProbPorp, and SerNcProp of the audit executions of each process
As for the first metrics set, with the significance level of 0.05, we found that there are
no significant differences among these processes and projects respectively with respect
to AuditEffort and ProbDetCost, while there are for the other cases. Subsequently, we
calculated the Spearman's rank correlation coefficient between the experiences of
auditors and the rank values set of the metrics of metrics set One resulted from above
Friedman tests. From Table 3, we can see that ProbNum, SerNcProp, and ProbDetEff
have significant positive correlation with the experiences of the auditors.
Measurement and Analysis of Process Audit: A Case Study 291
Table 3. Spearman's rank correlation coefficient between the experiences of the auditors and
the rank values set of metrics of the first metrics set resulted from the Friedman tests
As for metrics M11 and M12, based on the correlation analysis and further records
reading and interviews, we got following findings:
(1) ProbDur has no significant correlation with the experiences of the problem
solvers and testers.
(2) NCs and problems submitted later had relatively shorter time to be verified
and disposed.
(3) The more serious the NC is, the more quickly it was verified.
(4) Problems with higher serious degree took more time to deal with.
Q2, Q3: What are the NCs and why there are deviations?
Based on the NCs presented in Section 4.1, we studied why these NCs occurred
and what they would lead to and investigated the corresponding corrective actions.
Due to the limited space, only some examples are listed as follows.
Measurement and Analysis of Process Audit: A Case Study 293
For the NCs due to the process definitions and standards, the related organizational
process definitions and standards have been modified or enhanced accordingly except
N3. For N3, the latest version of SoftPM is going to generate planning documents
automatically from the defined Gantt graph since it is tedious to manually keep the
consistency between the planning documents and the Gantt Graphs split on SoftPM.
As for the NCs of missing personal task reports, through interviews, we found that
there are three main reasons: forgetting to, being too busy to fill in, and taking the
reporting as trivial. For CSRD, an organization emphasizing process management and
improvement, these reports are important for the measurement and analysis of proc-
esses and for managers to control the projects, and may provide information for proc-
ess improvement. Therefore, CSRD took several measures to eliminate such NCs,
such as emphasizing the importance of process management and reporting, adding a
functionality of automatically sending email reminders of filling the reports every
reporting day, and improving the feasibility of the reporting functionality of SoftPM.
The tracking of requirements is an important but a strenuous task and most NCs of
REQM are about it. The latest version of SoftPM will provide some new requirements
management functionalities, such as checking the completeness of requirements re-
lated information, automatically generating some information in the requirements
tracking matrix, e.g. some from the users’ requirements to the code and test cases.
In summary, the process audit in CSRD has the expected appraisal and analysis
modes of operation, and tools could not guarantee the quality of process but may aid
in decreasing some kinds of NCs.
As for the studied projects, though they all are releases of SoftPM, some of them
are major versions while the others are minor versions or customized versions. There-
fore, even for the same process, the checking items to examine and the amount of the
products and activities to check of these projects may also be different. Besides, the
difference in schedule tightness may lead to the difference in the process executions
of various projects.
In summary, there are differences in process audits at multiple levels with respect
to the auditors and audited processes and projects. The experiences of auditors, the
type of process, and the characteristics of projects all may affect the process audit.
5 Discussion
The measurement schema is the kernel of our work. We designed it using the GQM
method based on our knowledge about process and measurement [14] and consulting
the conceptual model presented in [3]. Some of the metrics are obvious, such as the
number of problems detected in an audit execution; others are defined referring some
well-know metrics, such as the problem detection cost and efficiency.
The quality of data is fundamental to the quality of our analysis. We collected data
from multiple sources, including audit reports, NCs lists, related task reports, project
reports, etc., and then verified the raw data, such as NC lists manually, consolidated
the original information, checked and resolved the inconsistency between related
sources. Through these ways, we try to assurance the quality of studied data.
Multiple analysis methods were used carefully to implement and support each
other. We applied descriptive statistics as well as statistical hypothesis test and corre-
lation analysis to explore the differences in metrics data. We paid much attention to
the usage assumptions of the statistical analysis techniques that are used in our case
study. Especially for SPC, it is still a controversial issue that whether SPC can be used
for estimating parameters for some probability model [15]. However, SPC used in our
analysis is to characterize the process behavior rather than estimating parameters. In
this sense, SPC works fine for our study, while we admit that there is still a long way
to systematically apply SPC to monitor and control the process auditing process. Fur-
thermore, we carried causal analysis on the statistical analysis results according to the
characteristics of the processes, projects, and auditors.
The application of the measurement system in CSRD illustrates its practicality and
usage in the specific context. Since we just sampled 8 projects from the 15 versions of
SoftPM, a small size population of one organization, we cannot proclaim the results
are of statistical significance nor they definitely hold in any other circumstances.
Whereas, our work, including the measurement and analysis methods and the study
results can give some references for other organizations within similar context, such
as taking process compliance as an important management quality goal, having some
organizational capability of quantitative analysis, and so on.
Measurement and Analysis of Process Audit: A Case Study 295
The effectiveness and performance of process audit consist of multiple dimensions and
factors, so does the measurement for it. For example, for an audit execution, besides
the number of the detected NCs, the proportion of NCs that are confirmed as problems
should also be taken into account. Besides the process type, the experiences of auditors
and characteristics of projects all may affect the execution of process audit.
As for the data analysis, process data usually has some characteristics which are
not ideal for statistical analysis and even make some analysis techniques unavailable,
such as small sample, abnormal distribution, and heterogeneity of variance. For ex-
ample, we thought the interaction of process type, project feature, and auditor experi-
ences have correlations with defined effectiveness and efficiency metrics. Due to the
abnormal distribution and heterogeneity of variance of the data, we cannot apply
ANOVA (Analysis of variance). Then we set new attributes to denote the interaction
of these factors, but failed to test the correlations due to too few samples. In some
cases, the violation of usage assumption will lead to unacceptably misleading results;
while in others, such as the full capability of the technique is not needed or the preci-
sion of the result is not required heavily, it is possible that the violation of assump-
tions does not hurt the intended studies. Therefore, we should choose the analysis
techniques comprehensively according to the study goal and characteristics of data as
well as the usage conditions of the analysis techniques.
Acknowledgements
This work is supported by the National Natural Science Foundation of China under
Grant Nos. 90718042, 60873072, and 60903050; the National Hi-Tech R&D Plan of
China under Grant Nos. 2007AA010303; the National Basic Research Program under
Grant No. 2007CB310802.
296 F. Shu et al.
References
1. April, A., Abran, A., Merlo, E.: Process assurance audits: lessons learned. In: 20th ICSE
1998, pp. 482–485. IEEE Computer Society, Kyoto (1998)
2. Baysinger, S.M.: The Complete Guide to the CQA. Quality Publishing Inc., Tucson (1997)
3. Gasston, J.L., Rout, T.P.: Can the effectiveness of software processes be assessed? Soft-
ware Quality Journal 3, 153–166 (1994)
4. Makinen, T., Varkoi, T., Jaakkola, H.: Assessment of a software process assessment proc-
ess. Management of Engineering and Technology 1, 437 (2001)
5. IEEE Computer Society: IEEE Standard for Software Reviews and Audits, IEEE Std
1028-1993, IEEE, New York
6. Smith, R.C.: Software development process standards: Challenges for process assurance.
In: 3rd ISESS, pp. 180–186. IEEE Computer Society, Los Alamitos (1997)
7. Wang, Q., Li, M.: Measuring and improving software process in China. In: 2005 Interna-
tional Symposium on Empirical Software Engineering, pp. 183–192 (2005)
8. CMMI Product Team: CMMI® for Development, Version 1.2. Pittsburgh, PA, Carnegie
Mellon Software Engineering Insitute (2006)
9. Basili, V.R., Caldiera, G., Rombach, H.D.: The goal question metric approach. Encyclope-
dia of software engineering, pp. 528–532 (1994)
10. SPSS: http://www.spss.com
11. Wu, X.: Statistics: From data to conclusions. China Statistics Press, Beijing (2004)
12. Florac, W.A., Carleton, A.D.: Measuring the software process: statistical process control
for software process improvement. Addison-Wesley Longman Publishing Co., Inc, Am-
sterdam (1999)
13. Baldassarre, M.T., Boffoli, N., Bruno, G., Caivano, D.: Statistically based process moni-
toring: lessons from the trench. In: Wang, Q., Garousi, V., Madachy, R., Pfahl, D. (eds.)
ICSP 2009. LNCS, vol. 5543, pp. 11–23. Springer, Heidelberg (2009)
14. Munson, J.C.: Software engineering measurement. Auerbach (2003)
15. Weller, E., Card, D.: Applying SPC to Software Development: Where and Why. IEEE
Software 25, 48–51 (2008)
A Fuzzy-Based Method for Evaluating the
Trustworthiness of Software Processes
1 Introduction
Based on the long-term experience and theoretical analyses in software engineering, it
is generally accepted that software processes to a large extent determine the corre-
sponding software products.
In our previous work [1], we proposed the concept of Process Trustworthiness as a
capability indicator to measure the relative degree of confidence for certain software
processes to deliver trustworthy software. Furthermore, we developed the software
process trustworthiness model and the corresponding trustworthiness measurement
system [2]. These models form the groundwork for evaluating the trustworthiness of
software processes with respect to metric data collected at process execution time.
Based on investigation of existing evaluation methods, we think a good process
evaluation method should:
z base on quantitative data to minimize subjectivity related to qualitative data;
z take consideration of the uncertainty situation since for some metrics it is dif-
ficult to get accurate values;
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 297–308, 2010.
© Springer-Verlag Berlin Heidelberg 2010
298 H. Zhang et al.
Considering of the metrics whose accurate values are difficult to get, we expect a
method with fuzzy-related theories. Fuzzy set [9][10] was a theory used in a wide
range of domains in which information was incomplete or imprecise, based on which
a comprehensive fuzzy evaluation method was provided, in order to evaluate the ob-
ject with uncertainty.
The procedure of the comprehensive fuzzy evaluation method can be summarized
as follows: first, an index set should be determined, whose elements are the character-
istics of the object being evaluated; second, the number of estimation scales should be
defined, which cannot be changed in the same evaluation process; third, a member-
ship degree of each scale should be assigned for each element in the index set, which
forms an evaluation matrix R
⎡ r11 " r1n ⎤
R = ( rij ) m× n = ⎢⎢ # % # ⎥⎥ (1)
⎢⎣rm1 " rmn ⎥⎦
where m is the number of the elements, n is the numbers of the evaluation scales, and
rij means the membership degree of the j th scale for the i th element; finally, a fuzzy
subset that consists of the weights of all the elements should be determined, and then
we synthesize the evaluation matrix and the fuzzy subset, and the result is a decision
set with the same number of scales as each element. The decision set is the result of
the evaluation process, each element of which means the membership degree of the
corresponding scale.
300 H. Zhang et al.
Trustworthiness Data
Data Collection
Metric System Pre-process
According to the
Tailoring
Tailored System
Trustworthiness
Metric Weights
Evaluation
Determination
Calculation
In this section we will further describe the following parts of our method in detail:
data type definitions, trustworthiness measurement system tailoring, metric weight
determination, data pre-process, evaluation calculation, and the historical database
support.
The FETP method supports various types of data, thus we could choose the most
appropriate forms to describe the metrics in order that each metric could contain as
much information as possible. In our preliminary work of the process trustworthiness
metric system, we defined three types for data: Boolean type, Level type and Numeri-
cal type. These types cannot fully reflect the characteristics of the metrics in some
cases, and thus we introduce the concept of membership degree from the fuzzy set
theory, and use numerical interval to further describe the data. Table 2 shows the four
types after the extension to our early work.
A Fuzzy-Based Method for Evaluating the Trustworthiness of Software Processes 301
Numerical Intervals {[Interval1](p1%), Scale deviation between delivered product and the
+ [Interval2](p2%),…, requirement = {[0, 0.33](0%), [0.33, 0.67](16.7%),
Membership Degrees [Intervaln](pn%)} [0.67, 1.00](83.3%)}
In this section we will further elaborate the application of AHP in the context of our
trustworthiness measurement system.
302 H. Zhang et al.
Establish
Establish the Perform pairwise Synthesize
judgment matrixes
hierarchy of the comparisons the weights in
and compute the
metric system of the metrics each level
metric weights
As shows in Fig. 2, based on our previous work [1] we first establish the hierarchy
according to the tailored trustworthiness measurement system. We regard the software
process trustworthiness as the overall objective, and then divide the metrics into dif-
ferent levels. For example, we could first group the TPAs by the definitions of TAPA,
TMPA and TEPA [1] as the level under the overall objective, and the next level are
the TPAs in each of the three groups, under which are the metrics of each TPA. Af-
terwards a tree structure is established, where every leaf is a metric.
Next we need to establish the judgment matrixes top-down with pairwise compari-
sons of the metrics in each level. The basis of the comparisons is from two aspects:
expert experience and the historical data. On one hand, a group of domain experts
give their results for every comparison respectively, and the final comparison results
are the composite opinions of the experts. On the other hand, we could also refer to
the historical data during the comparison process, which will be described later in 3.6.
The two aspects complement each other, and each importance could be adjusted ac-
cording to actual conditions, thus ensuring both the flexibility and objectivity of the
results.
Several judgment matrixes are established after the comparisons of all the levels.
When the consistency of each matrix is acceptable, we could obtain the relative
weights of the metrics. After the synthesis of the metric weights in each level, we
finally get the absolute weights of all the metrics.
negative, which means smaller value is better, the value after normalization is (emax-
e)/(emax-emin). For the data in the form of intervals, we could also use the procedures,
consider the endpoint of the interval as the value e, and perform the normalization
accordingly. Such procedures will ensure all the quantitative data have values in the
interval [0, 1], and have the consistency directions.
After the normalization of the data, we adopt the following steps to unify the form
of metric data:
1) For a metric of Boolean type, we assign 1 to the positive value, and 0 to the
negative value, and then it becomes a Numerical data, for which we assign 100% as
the membership degree to the original value. As mentioned above, the “discrete
values + membership degrees” form could also be considered as the “Numerical
Intervals + Membership Degrees”. For example, the value of a metric is {A(a%),
B(b%), C(c%)}, and after unification it should be {[A,A](a%), [B,B](b%),
[C,C](c%)}.
2) For a metric of the type “Levels + Membership Degrees”, we also unify it to
the type of “Numerical Intervals + Membership Degrees”. Assume that a metric
consists of N levels, pk means the membership degree of the k th level, and the type
could be represented as
{Class1(p1), Class2(p2), …, ClassN(pN)}
Then we divide [0, 1] to N equal intervals, and convert the original
Classk(pk)(k=1,2,…,N) to [(k-1)/N, k/N)(pk) (k=1,2,…,N), after which the metric
should be
{[0, 1/N)(p1), [1/N, 2/N)(p2), …, [(N-1)/N, 1](pN)}
So far, we have unified all the types to the “Numerical Intervals + Membership
Degrees” type.
After the preparations above, the weights of the metrics have been determined, and
the values of the metrics have been unified to the “Numerical Intervals + Membership
Degrees” type, based on which this section will give a description on the calculating
procedures of the trustworthiness evaluation, with the final result in the form of “user-
customized trustworthiness level + the membership degree of each level”.
At first we give some definitions to the variables and operations:
Definition 1. X={ x1, x2, …, xm} is an evaluation set consisting of m metrics.
Definition 2. W={w1, w2, …, wm} is a set of m metric weights, each corresponding
to the metric in X.
Definition 3. The metric xi consists of ni elements. An element contains two parts:
an interval [aij,bij] (i=1, …, m, j=1, …, ni-1), and the membership degree pij(i=1, …, m,
j=1, …, ni). Therefore,
xi={[ai1,bi1](pi1), [ai2,bi2](pi2), …, [ai,ni,bi,ni](pi,ni)} . (2)
then weight xi, that is
wixi={wi[ai1, bi1](pi1), wi[ai2, bi2](pi2), …, wi[ai,ni, bi,ni](pi,ni)} . (3)
304 H. Zhang et al.
The historical database is an essential component in our method. The content of the
database is mainly from the data accumulated in the evaluation process. Besides, the
organization could also add data to the database by other means. In the process of
trustworthiness evaluation, there are two procedures that need the support of the data-
base. On the one hand, when we make the pairwise comparisons of the metrics, the
database should provide historical data as the reference basis; on the other hand, in the
procedure of data pre-processing, the historical data used to perform the Min-Max
Normalization is from the historical database. Meanwhile, the intermediate and the
final results contain rich information, which are important experience for the software
organizations.
306 H. Zhang et al.
4 Case Study
In this section, we take the example of eight versions of SoftPM [12], which is the
lead product of a software organization rated at CMMI maturity level 3 in China. We
apply the FETP method to a certain process of these eight versions of SoftPM, and
then discuss the results with statistical analyses.
4.1 Application
We implemented trustworthiness evaluation for the Review and Test Processes of the
eight versions. According to the trustworthiness requirements and the evaluation oc-
casion, we tailor our trustworthiness measurement system and select nine metrics for
the Review and Test Processes, as shown in Table 2.
Proportion of the review defects whose severities are Levels[No, Vague, Clear] +
M02 Membership Degrees
recorded.
Levels[Poor, Good, Excellent] +
M03 Completeness of the review summary report.
Membership Degrees
M04 Average number of days for fixing one review defect. Numerical
M05 Average number of days for fixing one bug. Numerical
M06 Proportion of review and test work load. Numerical
M07 Efficiency of review and test. Numerical
M08 Coverage of test cases. Levels[Poor, Good, Excellent] +
Membership Degrees
M09 Proportion of the remaining testing bugs. Numerical
Table 3. AHP Judgment Matrix of the Metrics for the Review and Test Processes
Afterwards, we determine the weights of the metrics based on the AHP method de-
scribed in 3.2. We combine the expert experience and the historical data to make the
pairwise comparisons of the metrics, and establish the judgment matrix. We further
solve the matrix and get the weights of the metrics, as shown in Table 3.
A Fuzzy-Based Method for Evaluating the Trustworthiness of Software Processes 307
We calculate the trustworthiness of the process with the weights and collected met-
ric values, with the historical data in the database of the software organization. We
define 5 levels of trustworthiness, and get the results shown in Table 4.
Ver. Membership Degrees of Each Level (%) Ver. Membership Degrees of Each Level (%)
1 2 3 4 5 1 2 3 4 5
V1 72.5 27.5 0 0 0 V5 0 0 100.0 0 0
V2 0 0 100.0 0 0 V6 0 0 95.9 4.10 0
V3 0 100.0 0 0 0 V7 0 0 100 0 0
V4 0 13.5 86.5 0 0 V8 0 0 0 100 0
Based on the data, we further carry on a correlation analysis between the Trustwor-
thiness Median Values and the Defect Density Values at Release Time, and the Pear-
son's Correlation Coefficient is -0.559, which reflects the negative correlation between
them at a certain degree. The result is consistent with the real situation, where process
with higher trustworthiness leads to lower defect density of software product.
5 Conclusions
This paper proposed an evaluation method of software process trustworthiness which
supports various evaluation occasions and multiple data types. The AHP method was
used to compute the weight of metrics and the theory of fuzzy set was introduced thus
made the method applicable to the situations with uncertainty, based on which we
developed a procedure to perform the calculation with the values and the weights of
308 H. Zhang et al.
Acknowledgements
This work is supported by the National Natural Science Foundation of China under
Grant Nos. 90718042, 60873072, and 60903050; the National Hi-Tech R&D Plan of
China under Grant Nos. 2007AA010303; the National Basic Research Program under
Grant No. 2007CB310802.
References
1. Yang, Y., Wang, Q., Li, M.: Proceedings of the International Conference on Software
Process, Vancouver, B.C., Canada (2009)
2. Shu, F., Jiang, N., Gou, L.: Technical Report: A Trustworthiness Measurement Model.
ISCAS/iTechs Technical Report #106 (2008)
3. Saaty, T.L.: The Analytic Hierarchy Process. McGraw-Hill, New York (1990)
4. Amoroso, E., Taylor, C., et al.: A process-oriented methodology for assessing and improv-
ing software trustworthiness. In: 2nd ACM Conference on Computer and communications
security, Virginia, USA, pp. 39–50 (1994)
5. Amoroso, E., Nguyen, T., et al.: Toward an approach to measuring software trust. In: IEEE
Computer Society Symposium on Research in Security and Privacy (1991)
6. CMMI Product Team. CMMI® for Development, Version 1.2. Pittsburgh, PA, Carnegie
Mellon Software Engineering Insitute (2006)
7. Chrissis, M.B., Konrad, M., Shrum, S.: CMMI®: Guidelines for Process Integration and
Product Improvement. Addison-Wesley Publishing Company, Boston (2006)
8. Wang, Q., Yang, Y.: Technical Report: A Process-Centric Methodology to Software
Trustworthiness Assurance. ISCAS/iTechs Technical Report #105 (2008)
9. Zadeh, L.A.: Fuzzy Sets. Information Control, 338–353 (1965)
10. Klir, G.J., Yuan, B.: Fuzzy Sets and Fuzzy Logic. Theory and Applications. Prentice Hall,
New Jersey (1995)
11. Han, J., Kamber, M.: Data Mining: Concepts and Techniques. Morgan Kaufmann, San
Francisco (2006)
12. Wang, Q., Li, M.: Software Process Management: Practices in China. In: Software Process
Workshop, Beijing (2005)
13. Wang, Q., Li, M.: Measuring and improving software process in China. In: International
Symposium on Empirical Software Engineering (2005)
14. Tan, T., He, M., et al.: An Analysis to Understand Software Trustworthiness. In: The 9th
International Conference for Young Computer Scientists (2008)
15. Du, J., Tan, T., He, M., et al.: Technical Report: A Process-Centric Approach to Assure
Software Trustworthiness. ISCAS/iTechs Technical Report #106 (2008)
Software Process Simulation Modeling:
An Extended Systematic Review
1 Introduction
Software Process Simulation Modeling (SPSM) research has increased in the past
two decades, especially since the first ProSim1 Workshop held in 1998 and Kell-
ner, Madachy and Raffo’s (KMR) paper addressed the fundamental “why, what
and how ” questions of process simulation in software engineering. After 10 years
(1998-2007) progress, there is a need for a timely review of the research done in
SPSM, to update the current state-of-the-art, to summarize the experiences and
lessons, and to portray a full overview of SPSM research.
In ICSP 2008, we reported the first stage of our Systematic Literature Re-
view (SLR) on SPSM [1], which focused on the research published through
ProSim/ICSP channels over the decade (1998-2007). Nevertheless, a broader
view of SPSM research in software engineering will make this systematic lit-
erature review more valuable to the software process community, which is the
1
International Workshop on Software Process Simulation Modeling.
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 309–320, 2010.
c Springer-Verlag Berlin Heidelberg 2010
310 H. Zhang, B. Kitchenham, and D. Pfahl
motivation for the second stage of this review. During this review stage, another
relevant SLR [2] on the role of SPSM in software risk management was conducted
and reported, which focused a special use of SPSM based on the 27 studies from
a sub-scope of our review.
This paper reports the process and the preliminary results of the second stage
SLR, which searched and aggregated evidence from the publication channels out-
side ProSim/ICSP to address our original research questions. As the continuous
work of our former stage, the results derived from this stage also serves as a
validation of the ‘facts’, ‘trends’, and ‘directions’ identified in our first stage [3],
as well as the latest update and enhancement to the topics discussed in KMR’s
paper [4]. Additionally, our staged SLR enables a comparison of the research
characteristics and preferences within and outside ProSim/ICSP community.
Q1. What were the purposes or motivations for SPSM in the last decade?
Q2. Which simulation paradigms have been applied in the last decade, and how
popular were they in SPSM?
Q3. Which simulation tools are available for SPSM and have been in use in the
last decade?
Q4. On model level, what were research topics and model scopes focused by pro-
cess simulation models?
Q5. On parameter level, what were the output variables of interest when devel-
oping process simulation models?
Q6. Which simulation paradigm is the most appropriate for a specific SPSM
purpose and scope?
The three researchers involved in the first stage continued to work in their
roles during the second stage. In addition, one more researcher was invited to
join in the expert panel to ensure the review quality.
may give answers to the research questions from different points of view. Cate-
gory B studies introduce and provide effective paradigms, methods and tools for
constructing process simulation models or simulators (Category A studies). These
simulation models can be further adopted for different purposes in industrial con-
text by following the practical solutions or guidelines (Category C studies). The
experience (Category D studies) collected from modeling and adoption can be
used as feedback to iteratively improve SPSM research [3].
A. Software process simulation models or simulators;
B. Process simulation modeling paradigms, methodologies, and environments;
C. Applications, guidelines, frameworks, and solutions for adopting process sim-
ulation in software engineering practice;
D. Experience reports and empirical studies of SPSM research and practice.
Due to some minor disagreements experienced between the principal and the
secondary researchers in the classification on a small number of studies (between
Category B and C), we further specified a set of concrete criteria (questions) to
facilitate the effective identification of each study’s category (in Table 2). If a
‘yes’ answer applies to any question related to one study type, this study was
allocated the corresponding (one or more) category.
Category Question
A - Was a new process simulation model or simulator presented in the study?
- Was a process simulation model or simulator applied in a new SE domain or a new
practical context?
B - Compared with previous studies, was a new simulation modeling paradigm introduced
into SPSM?
- Was a new process simulation environment or tool developed and described?
- Was a methodology or framework proposed or developed for improving SPSM?
- Were any factors associated to SPSM discussed in the study?
C - Was a new application of SPSM introduced to SE domain?
- Was a guideline or framework of directing SPSM solution to one specific problem or
context proposed or developed?
D - Did the study report any experience (qualitative or quantitative) of applying SPSM in
industry?
- Did the study report how a process simulation model or simulator has been built or
calibrated with empirical data?
- Did the study report an empirical study related to SPSM?
3 Results
3.1 Primary Studies
After literature search and initial study selection, 79 relevant studies were up-
loaded into the online system (http://systematicreviews.org) for more care-
ful selection and data extraction. By reading the full-text, 19 studies were further
excluded because they are 1) duplicate publications; 2) not simulation studies on
software process (e.g., simulation of software systems); or 3) research proposals
without implementation.
Finally, the second stage included 60 relevant studies in addition to those
found by the first stage review. In total, 156 (S1:96/S2:60) primary studies2 on
SPSM research were identified from 1998 to 2007. They form a comprehensive
body of research of software process simulation.
Table 3 shows the number of SPSM studies published during the decade,
which are grouped by two review stages. The number of published studies per
year was between 15 and 19 after Y2K, and stabilized at 18 or 19 after 2004. This
stability applies to all papers during the period, irrespective of whether or not
they were ProSim related papers. The overall number of conference publications
(77) is very close to the journal publications (79) over the period.
1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 Sum
Stage One
- ProSim/ICSP 2 0 2 n/a n/a 11 6 7 4 8 40
- JSS/SPIP special issues 0 11 9 12 7 0 5 4 6 2 56
Stage Two
- Conference/workshop 2 3 2 3 6 1 3 7 3 7 37
- Journal 0 0 4 3 2 3 4 1 5 1 23
Total 4 14 17 18 15 15 18 19 18 18 156
Table 3 also shows that the number of SPSM studies significantly increased
after the first ProSim workshop (ProSim’98), which demonstrates the positive
effect of ProSim series workshops to software process simulation research.
Table 4 lists the top conferences and journals where the researchers published
their SPSM studies in the decade. Apart from ProSim/ICSP venues (including
SPIP and JSS), IST and PROFES also have dominated outside publications.
It is worth noticing that most primary studies cited either [7] or [4], which
are the seminal work and landmark paper on SPSM.
3.2 Classification
The second stage review confirmed the classification from Stage 1 is appropri-
ate. Each included primary study was classified into one or more of the four
categories. Figure 1 shows the distribution of study categories over the decade.
It is clear that Category A studies (simulation models) were dominant in every
single year during the period.
2
The full study list will be online for public access at http://systematicreviews.org
314 H. Zhang, B. Kitchenham, and D. Pfahl
(a) average study quality per type (b) study quality per stage and type
4 Discussion
Following up our discussion in [1], this section focuses on the preliminary answers
to the first five research questions (Q1-Q5), but provides a broader view by
synthesizing the results from the both stages.
In the first ProSim Workshop (1998), KMR presented a wide variety of reasons
for undertaking simulations of software process models [4]. Primarily, process
simulation is an aid to decision making. They identified six specific purposes
for SPSM. Our first stage SLR extended and restructured them to be ten pur-
poses [1] based on the observation of SPSM studies.
The second stage SLR confirmed that these ten purposes for SPSM research
identified in Stage 1. After minor refinements, these research purposes were
grouped in three levels (i.e. cognitive level, tactical level, strategic level ). Table 5
reports the categories and relationships.
The diversity and complexity of software processes and the richness of research
questions (concluded into simulation purposes in Table 5) determine the different
capabilities of simulation paradigms needed.
Overall, 15 simulation modeling paradigms were found in our two-staged SLR.
Table 6 shows the paradigms used by more than one applied study. The rightmost
column indicates the number of studies (not limited to Category A) using or
addressing the corresponding paradigm (the leftmost column). The first number
in the bracket denotes the study number from Stage 1 (S1), and it is followed
by the number from Stage 2 (S2) for comparison.
System Dynamics (SD, 47%) and Discrete-Event Simulation (DES, 31%)
are the most popular technologies in SPSM. Other paradigms include State-
Based Simulation (SBS), Qualitative (or semi-quantitative) Simulation (QSIM),
316 H. Zhang, B. Kitchenham, and D. Pfahl
Compared to the other paradigms, it seems that DES offers more tool options
for its modelers in SPSM. Note that though Extend provides both continuous
and discrete simulation capabilities, it was seldom used for continuous simulation
alone in SPSM studies.
It is interesting that although Vensim and Extend are two popular simulation
tools in ProSim community, their dominance was not found outside that com-
munity. Our review shows that most studies using them but published outside
are also from the active researchers in ProSim/ICSP community. Instead, the
researchers outside the ProSim/ICSP community seemed to prefer programming
their models themselves.
ev uct ct
-p se
se
e
od o j
ul ha
ha
kn m
ng n
n
Topic Sum
pr i-pr
l o ti o
u n te r
ow
m ep
t
ec
ti
(S1/S2)
-
t
gl
oj
ol
ul
n
pr
m
si
generic development 9/10 1/2 22(10/12)
software evolution 1/0 7/1 9(8/1)
software process improvement 1/0 1/0 1/0 3/1 7(6/1)
requirements engineering 2/0 1/1 0/1 1/0 1/0 7(5/2)
incremental & concurrent development 1/0 2/0 1/2 1/0 7(5/2)
inspection & testing 1/2 0/2 0/1 0/1 7(1/6)
open-source development 1/0 0/1 1/0 2/0 5(4/1)
global development 1/0 3/0 4(4/0)
agile development 1/2 1/0 4(2/2)
software maintenance 0/1 0/3 4(0/4)
software economics 1/0 1/1 1/0 4(3/1)
acquisition & outsourcing 1/0 0/1 1/0 3(2/1)
software product-line 1/0 1/0 2(2/0)
quality assurance 1/0 1/0 2(2/0)
COTS-based development 0/1 0/1 2(0/2)
software engineering education 2/0 2(2/0)
software design 1/0 1(1/0)
software services 1/0 1(1/0)
risk management 1/0 1(1/0)
productivity analysis 1/0 1(1/0)
software reliability 1/0 1(1/0)
Total 7/3 4/3 19/20 1/0 2/5 8/1 2/0 9/3 92(56/36)
92 - the number of Category A studies). Note that there are many simulation
studies (models) with multiple outputs of interest.
In terms of Table 8, it is evident that time, effort, quality, size are the most
common drivers for simulation studies of software process. There are 71% studies
(65 out of 92) including either one of them or their combination as model outputs.
This finding confirms that SPSM research focuses mainly on factors of interest
to software project managers.
Software Process Simulation Modeling: An Extended Systematic Review 319
5 Conclusion
We conducted a two-staged systematic literature review of software process sim-
ulation modeling by systematically searching and aggregating studies published
within and outside the ProSim/ICSP community from 1998 to 2007. The results
and in-depth findings from the first stage were reported in [1] and [3]. As a con-
tinuation of previous research, this paper presents the process and the updated
results of our second stage review. To be specific, this research contributes to
software process research in the following aspects.
– A two-staged SLR which identified most SPSM studies and classified them
into four categories builds a basis for future secondary studies in SPSM.
– A broad state-of-the-art of SPSM research is portrayed from diverse aspects:
purposes, paradigms, topics, scopes, outputs, and so on.
– Updates to KMR’s landmark paper based on the evolution over the decade
since ProSim’98.
– An initial comparison between the SPSM related research reported within
and outside the ProSim/ICSP community.
Some limitations still exist in the current study and need further improve-
ments: 1) the study categorization was mainly determined by the principal re-
viewer’s final judgment, which may need further examination; 2) the impact
of study quality needs to be considered in data analysis, particularly for the
inclusion of low quality studies.
As our SLR is also a kind of mapping study, which provides groups of studies
in this domain, it can be used as a precursor to future more detailed secondary
research. In particular, this work will be enhanced by including a more detailed
analysis of the studies of Categories B, C and D.
Acknowledgment
NICTA is funded by the Australian Government as represented by the Depart-
ment of Broadband, Communications and the Digital Economy and the Aus-
tralian Research Council through the ICT Centre of Excellence program.
This work was also supported, in part, by Science Foundation Ireland grant
03/CE2/I303 1 to Lero - the Irish Software Engineering Research Centre
(www.lero.ie).
References
1. Zhang, H., Kitchenham, B., Pfahl, D.: Reflections on 10 years of software process
simulation modelling: A systematic review. In: Wang, Q., Pfahl, D., Raffo, D.M.
(eds.) ICSP 2008. LNCS, vol. 5007, pp. 345–365. Springer, Heidelberg (2008)
2. Liu, D., Wang, Q., Xiao, J.: The role of software process simulation modeling in
software risk management: A systematic review. In: Proceedings of the 3rd Inter-
national Symposium on Empirical Software Engineering and Measurement (ESEM
2009), Lask Buena Vista, FL, October 2009, pp. 302–311. IEEE Computer Society,
Los Alamitos (2009)
320 H. Zhang, B. Kitchenham, and D. Pfahl
3. Zhang, H., Kitchenham, B., Pfahl, D.: Software process simulation modeling: Facts,
trends, and directions. In: Proceedings of 15th Asia-Pacific Software Engineering
Conference (APSEC 2008), Beijing, China, December 2008, pp. 59–66. IEEE Com-
puter Society, Los Alamitos (2008)
4. Kellner, M.I., Madachy, R.J., Raffo, D.M.: Software process simulation modeling:
Why? what? how? Journal of Systems and Software 46(2/3), 91–105 (1999)
5. Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews
in software engineering (version 2.3). Technical Report EBSE-2007-01, Software Engi-
neering Group, School of Computer Science and Mathematics, Keele University, and
Department of Computer Science, University of Durham (April 2007)
6. Zhang, H., Babar, M.A.: On searching relevant studies in software engineering. In:
Proceedings of 14th International Conference on Evaluation and Assessment in Soft-
ware Engineering (EASE’10), Keele, England, BCS (April 2010)
7. Abdel-Hamid, T.K., Madnick, S.E.: Software Project Dynamics: An Integrated Ap-
proach. Prentice Hall, Englewood Cliffs (1991)
SimSWE – A Library of Reusable Components
for Software Process Simulation
1 Introduction
Software engineering processes often form complex dynamic systems whose char-
acteristics and behaviors are not easily analyzed by a simple static view. Quanti-
tative process modeling and simulation have proven to be a valuable support in
this situation. From early studies [1], software process simulation has advanced
in various directions and many aspects of software engineering issues and pro-
cesses have been modeled and analyzed using simulation methodologies. See [5]
[8] [14] for an overview of this field and further references.
Based on the accomplished maturity of the domain, the time seems ripe to
consolidate existing modeling and simulation knowledge into a library of reusable
simulation components as a supporting platform for next generation software
process models. The goal is to help practitioners to develop models easily [2] and
enable researchers approach more complex modeling and simulation scenarios
by providing encapsulated building blocks for commonplace functionality. There
has been some work in this direction previously [7] [12], but these have not
produced an openly available simulation library. The SimSWE library also partly
originates from ideas presented in [9] [10].
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 321–332, 2010.
c Springer-Verlag Berlin Heidelberg 2010
322 T. Birkhölzer et al.
The development of the SimSWE library relies on four factors which are
considered of prime importance for its success:
– Open source
The development of a comprehensive library requires cooperation, contribu-
tions, improvements as well critical reviews from as many participants as
possible. The SimSWE library has been started by the authors as a joint
initiative and will be shared under the GNU Lesser Public General License
(LPGL) [6], see Section 5. All are invited to participate in this project.
– Incorporation of different simulation types
The SimSWE library contains continuous time as well as discrete event com-
ponents enabling model developers to choose the view that fits best to the
problem at hand or even a hybrid approach (see Section 2.1).
– Generic description and reference implementation
The generic description (see Section 2.3) enables the realization of the Sim-
SWE components in various simulation environments. The reference imple-
mentation (see Section 2.4) provides proof of concept as well as plug-and-play
usability.
– Evolutionary design approach
Components require interfaces, generic abstractions and common definitions
(see Section 2.2). Ideally, these are designed and corrected before any im-
plementation. Realistically, however, this architecture will evolve with the
scope and maturity of the library. Therefore, design and implementation
of the SimSWE library is seen and planned as an iterative, evolutionary
approach.
Section 2 elaborates these concepts. Section 3 provides an overview of the
components already available and Section 4 demonstrates the use by example
models. The paper concludes with an invitation for participation in Section 5,
and a summary in Section 6.
2 Library Concepts
2.1 Simulation Types
In research and practice, two main approaches are used for software engineering
process simulation: continuous time simulation (system dynamics) and discrete
event simulation [15]. However, there is no inherent separation between these ap-
proaches. Continuous time models can be be recast into discrete event models,
and similarly, discrete event simulation engines (tools) can approximate contin-
uous time simulation. From a modeler’s perspective, both approaches have their
benefits and appropriate application, often even within the same model and thus
can be combined yielding a hybrid model.
Therefore, the SimSWE library encompasses continuous time as well discrete
event components (in some cases even for similar process elements). See for
example the components WorkTestRework and WorkTestReworkD in Section 3.
The choice is left to the user which approach suits best in a given context. It is
planned for SimSWE to incorporate hybrid components as well as “glue”-utilities
to interface between continuous time and discrete event parts in the future.
SimSWE – A Library of Reusable Components 323
Most of the SimSWE components have been extracted from existing simulation
models, e.g., from [3] [4] [8] [11].
However, the design and development of an encapsulated, reusable component
poses additional requirements beyond the transfer of equations and descriptions.
Inputs, outputs, and parameters must be identified. Unfortunately there is
no clear-cut generic distinction to classify a factor as input (factors which are
expected to depend dynamically on other parts) or as parameter (factors which
are expected to be constant during a simulation). Dynamics and flows must
be enabled for repeating cycles, e.g., multiple task or project cycles. Therefore,
appropriate reset or restart functionality has to be added.
Component behavior must be (fault-)tolerant and robust to handle unusual
or unintended parameters or input values, e.g., a decrease in task size during
development. This often requires additional provisions, e.g., limiters, guards,
assertion checks, etc. Moreover, abstractions and generic measures are introduced
to enable reuse in various contexts. The most important abstractions in the
SimSWE library are:
– Generic time measure without a specific unit. Depending on the context, one
unit of time can be an hour, a day, a month, etc.
– Generic size measure without a specific unit. Depending on the context, one
unit of size can be a function point, a source line of code, a module, a test
case, etc. Moreover, the input and output signals are designed in most cases
as size rates, e.g., changes of size. This facilitates piped interfacing.
– Generic productivity. The unit of productivity depends on the size and time
units chosen for the respective component.
For the reference implementation of the SimSWE library, the MATLAB / Simu-
link environment has been chosen as tool basis. It is based on a continuous time
simulation engine but encompasses discrete event extensions as well. MATLAB
/ Simulink is a well established tool in many engineering disciplines. It contains
324 T. Birkhölzer et al.
an extensive set of predefined functions and building blocks for logic, mathemat-
ical evaluations, descriptions of dynamic systems, and signal generation, display,
and analysis. Based on that, new components and component libraries can be
easily defined and incorporated. Therefore, MATLAB / Simulink offers all the
functionalities necessary to implement the SimSWE components and provides
a comprehensive and mature development environment for model developers to
build upon.
There are other tools with similar capabilities. A detailed tool evaluation,
however, is beyond the scope of this paper. Instead, other reference implementa-
tions are encouraged to gain a more in depth understanding of the (sometimes
subtle) distinctions, advantages and disadvantages - all based on the common
grounding of the generic descriptions.
3 Component Overview
This section provides a basic overview of the status and the direction of SimSWE
development. All components listed below are already implemented. However,
the list of components as well as their systematics are expected to grow and
evolve in the future. A documentation of each component can be found at [13].
Estimators
– COCOMOIIEarlyDesign estimates a project’s cost and duration before the
entire architecture is determined. It uses 7 Cost Driver and 5 Scale Factors.
The component is based on COCOMO II [3].
– COCOMOIIPostArchitecture estimates a project’s cost and duration based
on detailed architecture parameters. It uses 17 Cost Driver and 5 Scale Fac-
tors. The component is based on COCOMO II [3].
– SLOCEstimatorBasedOnFunctionPoints estimates a project’s source lines
of code (SLOC) based on unadjusted function points using programming
language conversion factors. The component is based on COCOMO II [3].
– PerceivedQuality models the asymmetric dynamic of the perception of qual-
ity changes: In case of a decrease of quality, the perceived quality follows this
change very fast. In case of an increase, the build-up in perceived quality is
much slower. The component is based on [8].
Generic Activities
– Work models the work on a given task or artifact (e.g., development) based
on task size and productivity. The component is based on concepts of [8]
and [4].
– WorkWithCommunicationOverhead models the work on a given task or arti-
fact (e.g., development) based on task size considering communication over-
head for adjusting the effective productivity. The component is based on
concepts of [8] and [4].
– WorkWithExperience models the work on a given task or artifact (e.g., devel-
opment) based on task size considering experience for adjusting the effective
productivity. The component is based on concepts of [8] and [4].
– VerificationAndValidation models a verification and validation (V&V) ac-
tivity.
– WorkBreakdown splits a task into pieces (smaller tasks).
Generic Processes
– WorkTestRework is a generic model of the production of an artifact followed
by an associated V&V step. If the artifact does not yet meet the specified
quality criterion, it is scheduled for rework. Rework and verification are
cyclically traversed until the quality criterion is fulfilled. The model is based
on [4].
– SCRUM is a generic model of an iterative process (SCRUM). Incoming tasks
are split into assignments with a fixed time length (sprints). At the start
of each assignment (sprint), the overall amount is broken down into tasks.
Each task is performed as a Work-Test-Rework cycle. Unfinished tasks are
rescheduled for the next assignment.
326 T. Birkhölzer et al.
Management Components
– HumanResourceChain models recruitment and training of new staff using
three levels of experience. The dynamics of a level depend on the promotion
time and the quitting rate. The model is based on [8].
– DelayBasedResourceAllocation models allocation of additional resources in
case of a pending delay of a project’s end date.
– BacklogBasedResourceDistribution distributes available resources among a
set of tasks such that each task will be finished at the same time (ideally).
The model is based on [8].
– CommunicationOverhead models the communication overhead, which in-
creases with the number of persons involved. The model is based on [8].
– KnowledgeAcquisition models the dynamics of learning and aging of knowl-
edge and their influence on productivity in software development projects.
The model is based on [8].
– ExperienceAcquistion models the influence of experience from actual or sim-
ilar projects in the past on the current productivity. The productivity value
grows log-linear with the actual experience value. The model is based on [8].
– Training models the assimilation process, when new personnel is added to a
project. Adding new people to a project leads to a decrease of the produc-
tivity, because experienced developers are needed for training. The model is
based on [8].
Utilities
– LevelVariable and LevelVariableResetable model a level variable with input
and output rates.
– AsymmetricDelay allows to define different time constants for growth and
attrition.
– RateConversation synchronizes two rates.
– Size2Rate converts a size into a rate.
Generators
– TaskGeneratorD creates objects of type Task in random time intervals for
discrete event processing.
– MultiSizeTaskGeneratorD creates large projects and smaller tasks in random
time intervals for discrete event processing.
Generic Activities
– ProductionD models the work on a given task (e.g., development) based
on a productivity. While the productivity is a continuous signal, a task is
represented by a discrete object.
– VerificationD models the verification process on a given artifact or result.
The artifact or result is represented by an discrete object of type Task with
the attributes Size and Quality.
SimSWE – A Library of Reusable Components 327
Generic Activities
– WorkTestReworkD is a generic model of the production of an artifact fol-
lowed by an associated verification step using a discrete event object flow ap-
proach (instead of a time continuous representation, see WorkTestRework ).
Management Components
– KnowledgeAcquisitionExtended demonstrates the dynamics of learning and
aging of new learned knowledge. The extended version includes an knowledge
level control to hold values in an specified range.
4 Examples
It is beyond the scope of this paper to present and discuss complete models and
simulation examples, only the use of the SimSWE components are illustrated in
this section. In the models shown below, the library components are drawn with
a grey background. The other components are Simulink blocks providing glue
logic, signal generation, and graphical display.
However, this decrease is not monotonic due to the requirement volatility simu-
lated as randomized changes of value and cost. The last plot in Figure 3 shows the
changes of productivity due to familiarization with the respective requirements,
development environment, and increased development skill.
This example demonstrates how the SimSWE framework and component li-
brary eases model development and enables the exploration of more complex
simulations through reuse and extension.
As a second example, a dynamic resource allocation strategy taken from [8] is ex-
plored. The strategy has the goal to divide resources between different tasks such
SimSWE – A Library of Reusable Components 329
that all task are finished at the same time. This would be an optimal strategy in
terms of a conjoint duration (e.g., if the tasks are work packages within a project).
The strategy allocates the resources according to the estimated productivities for
the respective task and the ratio of the respective task backlogs such that tasks
with a larger expected workload get proportionally more resources. This strategy
is implemented in the SimSWE component BacklogBasedResourceDistribution.
In the case of known, constant task sizes and productivities, the resource
allocation would be constant as well. However, in the case of changing, unknown
task sizes and productivities, a continuously adaptation would be necessary. But
this is not a realistic project setting. Even in an agile management environment,
resource reassignments will occur only within finite time steps. Therefore, the
component calls for a fixed time interval between reassignments.
The model of Figure 4 is designed to test this strategy and to analyse the
effects of various lengths of such reassignment intervals in the presence of “per-
turbances” like changing task sizes and time varying productivities (e.g., due
to knowledge build-up). In this model, three SimSWE components are used:
BacklogBasedResourceDistribution, KnowledgeAcquistion, and Work. Note, that
many Simulink components (including KnowledgeAcquistion and Work ) are able
to process vectorized inputs concurrently, which is equivalent to placing instances
of the component in parallel. By this means, the model of Figure 4 processes
two tasks in parallel.
The KnowledgeAcquistion component is used to represent a knowledge build-
up for each task. It is parametrized such that task T1 starts with a relative
330 T. Birkhölzer et al.
Fig. 5. Results of a simulation run of the model depicted in Figure 4. The values for
task T1 are drawn as solid line, the values for task T2 as dashed line.
knowledge level of 50% while task T2 starts with a level of 0%. At the beginning,
both task are defined with the same size. Between time t = 5 and time t = 15,
task T1 is gradually increased by 50%.
Both effects are not taken into account in the resource allocation strategy in
the first place. Nevertheless, the dynamic adaption is able to adjust to these per-
turbances as shown by the simulation, see Figure 5: The chart at the top shows
the allocated resources. At the beginning, both tasks get equal resources due to
their equal size. However, task T2 has initially a smaller effective productivity
due to the lower initial knowledge level causing a smaller reduction in the backlog
(the variable ToProduceSize as shown in the bottom chart). Therefore, task T2
get more resources assigned in the next assignment step. After about half of the
simulation time has elapsed, the balance of resource assignments turns around,
because the knowledge build-up (note the effect of the logistic learning curve in
left half the chart ActualProductivity) saturates and the size of task T1 increases.
SimSWE – A Library of Reusable Components 331
6 Summary
SimSWE is a library of components for modeling software engineering tasks
and processes. It consists of a generic description of the underlying models and
a reference implementation as a Simulink library. At the current status, the
library already contains more than thirty components. It is available (and further
developed) as Open Source with our example simulations.
The advantages of such a library seem to be obvious: simulation models can
be developed faster by using available building blocks and more realistic, and
thus also more complex models can be build using predesigned, pretested parts
encapsulating complexity. Indeed, the library has already proven useful in quickly
replicating and refining previous simulations that were cumbersome to develop
and extend and exemplified in Section 5.
Furthermore, a library of generally recognized relations and dynamics of soft-
ware engineering related issues might also be used in the context of education
and training: the generic descriptions can serve as a basis for discussions while
332 T. Birkhölzer et al.
References
1. Abdel-Hamid, T., Madnick, S.E.: Software project dynamics: an integrated ap-
proach. Prentice-Hall, Inc., Upper Saddle River (1991)
2. Angkasaputra, N., Pfahl, D.: Making software process simulation modeling agile
and pattern-based. In: Proceedings of the 5th International Workshop on Software
Process Simulation Modeling (ProSim 2004), pp. 222–227 (2004)
3. Boehm, B., Abts, C., Winsor, A., Chulani, S., Clark, B.K., Horowitz, E., Madachy,
R., Reifer, D., Steece, B.: Software Cost Estimation with Cocomo II. Prentice Hall
International, Englewood Cliffs (2000)
4. Garousi, V., Khosrovian, K., Pfahl, D.: A customizable pattern-based software
process simulation model: design, calibration and application. Software Process:
Improvement and Practice 14(3), 165–180 (2009)
5. Kellner, M.I., Madachy, R.J., Raffo, D.: Software process simulation modeling:
Why? what? how? Journal of Systems and Software 46(2-3), 91–105 (1999)
6. GNU: lesser general public license, http://www.gnu.org/copyleft/lesser.html
(last accessed on 2010-01-05)
7. Madachy, R.J.: Reusable model structures and behaviors for software processes.
In: Wang, Q., Pfahl, D., Raffo, D.M., Wernick, P. (eds.) SPW 2006 and ProSim
2006. LNCS, vol. 3966, pp. 222–233. Springer, Heidelberg (2006)
8. Madachy, R.J.: Software Process Dynamics. John Wiley & Sons, Chichester (2008)
9. Pfahl, D.: Simkit – a software process simulation model construction kit in support
of empirical research. In: Proceedings of the 5th ACM-IEEE International Sympo-
sium on Empirical Software Engineering (Volume II: Short Papers and Posters),
ISESE 2006, pp. 3–5 (2006)
10. Pfahl, D.: Software process simulation frameworks in support of packaging and
transferring empirical evidence. In: Basili, V.R., Rombach, H.D., Schneider, K.,
Kitchenham, B., Pfahl, D., Selby, R.W. (eds.) Empirical Software Engineering Is-
sues. LNCS, vol. 4336, p. 133. Springer, Heidelberg (2007)
11. Port, D., Olkov, A., Menzies, T.: Using simulation to investigate requirements
prioritization strategies. In: ASE, pp. 268–277. IEEE, Los Alamitos (2008)
12. Raffo, D., Nayak, U., Wakeland, W.: Implementing generalized process simulation
models. In: Pfahl, D., Raffo, D., Rus, I., Wernick, P. (eds.) Proceedings of the 6th
International Workshop on Software Process Simulation Modeling (ProSim 2005),
Fraunhofer IRB, St. Louis (2005)
13. SimSWE-Wiki,
http://simswe.ei.htwg-konstanz.de/wiki_simswe/index.php/Main_Page (last
accessed on 2010-04-10)
14. Zhang, H., Kitchenham, B., Pfahl, D.: Software process simulation modeling: Facts,
trends and directions. In: APSEC, pp. 59–66. IEEE Comp. Society, Los Alamitos
(2008)
15. Zhang, H., Kitchenham, B., Pfahl, D.: Software process simulation over the past
decade: trends discovery from a systematic review. In: Rombach, H.D., Elbaum,
S.G., Münch, J. (eds.) ESEM, pp. 345–347. ACM, New York (2008)
Applications of a Generic Work-Test-Rework
Component for Software Process Simulation
1 Introduction
During the last years, quantitative process modeling and simulation has been
used to analyze various questions and issues in the domain of software engi-
neering, see, e.g., [6] and [13] and the references therein for an overview of this
field. At the beginning [1], models had to be designed from scratch developing
and implementing every part of the model by oneself. However, this is obviously
not reasonable in the long run. Therefore, research has been conducted in the
last years to develop reusable patterns [2] [5], structures [8], or models [9]. In
[5], a generic work-test-rework cycle was identified as one of the most important
reoccurring activity patterns. For a formal analysis and description of rework
activities see also [3].
Building on these results, this work goes one step further by introducing an
encapsulated work-test-rework simulation component called WorkTestRework as
a generic ready-to-use building block suitable for a wide range of simulation ap-
plications. This will help to develop models faster and to approach more complex
modeling and simulation scenarios by reusing commonplace functionality. Thus,
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 333–344, 2010.
c Springer-Verlag Berlin Heidelberg 2010
334 T. Birkhölzer, D. Pfahl, and M. Schuster
2 WorkTestRework Component
The WorkTestRework component is based on [5]. It models the production of
an artifact followed by associated verification and validation (V&V) steps. If
the artifact does not yet meet the specified quality criterion, it is scheduled for
rework. Rework and V&V are cyclically traversed until the quality criterion is
fulfilled. In this case, the artifact is released. Typical artifacts are requirement
specifications, designs, code, user stories, test cases, etc. However, any other
task, which requires effort and results in a revisable delivery, e.g., a make-or-buy
evaluation, could be modeled by this pattern as well.
The component is part of SimSWE, a library of components for modeling and
simulation of software engineering processes [12]. SimSWE provides a generic,
implementation independent description of each component and a reference im-
plementation as Simulink R
library. Therefore, the WorkTestRework component
can either be directly used as a Simulink component or realized in another sim-
ulation environment using the generic description. The SimSWE library is avail-
able under the GNU Lesser General Public License [7] at request. A description
of the component (and the models presented in Section 3 and 4) can be found
in the documentation at [12].
The outline of the component is shown in Figure 1. It has 8 inputs (values
depending dynamically an other parts of the model), 4 parameters (values which
are constant during an simulation), and 6 outputs. The parameters are not shown
in Figure 1, because in Simulink they are entered using a mask.
Inputs:
– SizeReceived : Rate of the designated size of the artifact.
– UndetectedFaultsReceived : Rate of the undetected faults in the designated
artifact caused by undetected faults in the input.
– DetectedFaultsReceived : Rate of the known faults detected or caused by pre-
ceding activities.
– DevelopmentProductivity: Productivity of the initial work phase (develop-
ment).
– VerificationProductivity: Productivity of the V&V step.
– DeliveryProductivity: Productivity of the delivery step.
– RemovalProductivity: Productivity of the rework step.
– Reset : Reset of the component to zero levels and idle state.
Parameters:
– AverageVerEffectiveness: Effectiveness of the V&V step as percentage of
faults detected.
– QualityLimit : Release threshold with respect to the number of faults pending
divided by the size of the artifact.
– AverageFaultInjectionPerSizeUnit : Number of faults injected during the ini-
tial development step.
– AverageFaultInjectionPerFaultRemoval : Number of faults injected during
the rework step.
Outputs:
– SizeDelivered : Rate of the size of the artifact delivered. The artifact is de-
livered when the specified quality limit has been achieved.
– UndetectedFaultsDelivered : Rate of the undetected faults in the artifact de-
livered.
– DetectedFaultsDelivered : Rate of the known faults in the artifact delivered.
– ToDeliverSize: Designated1 size of the artifact.
– FaultsPending: Current number of faults pending.
– StateIndex : Variable indicating the internal state of the artifact development:
0: idle (no activity); 1: the work step is active; 2: the rework step is active;
3: the V&V step is active; 4: the delivery step is active.
The first three inputs and the first three outputs are designed to connect
WorkTestRework -components in a piped, sequential chain handing on artifacts,
see Section 3 for examples. Designing these interfaces as rates simplifies the
piped operation without additional handshake signals: an input, is interpreted
as new assignment, if it is received before or after a release (StateIndex 0 or 4),
or as changing size (e.g., caused by late requirements), if it is received during a
ongoing development (StateIndex 1, 2, or 3).
The productivity inputs can be used to model dynamic resource allocation,
e.g., due to resource build-up or competing assignments, as well as time varying
productivity per resource, e.g., biased by training, experience, or motivation. In
the SimSWE library, there are components available modeling these effects.
1
In distinction to SizeDelivered, this output provides information about the (expected)
size of the artifact prior to delivery, which might be useful in certain modeling
contexts.
336 T. Birkhölzer, D. Pfahl, and M. Schuster
The interpretation of units is left to the user, e.g., depending on the context,
one unit of time can be an hour, a day, a month, or one unit of size can be a
function point, a source line of code, a module, a test case, etc. The units of
productivity obviously depend on the size and time units chosen.
All inputs, parameters, and outputs (except StateIndex ) are continuous vari-
ables assuming real values. If natural numbers are desired (e.g., for the number of
faults), the real values could easily be rounded externally. However, a real value
is maintained as output to allow to interpret 0.2 faults, e.g., as “probability of 20
percent to get one error” instead of rounding this to zero faults automatically.
Figure 2 shows a simulation demonstrating the typical behavior of the compo-
nent. At the beginning, an assignment is received. At time 150, the assignment is
enlarged by a late add-on. The areas under the rate signals (SizeReceived, SizeDe-
livered, UndetectedFaultsDelivered, DetectedFaultsDelivered ) represent the actual
sizes and fault counts. The StateIndex chart shows the cyclic traversal through
the states work, V&V, and rework until the quality limit is reached and the
artifact is delivered.
Note, that any work-test-rework activity delivers detected as well as unde-
tected faults. To avoid the delivery of undetected faults, V&V effectiveness would
have to be 100%. To avoid the delivery of detected faults, the fault injection dur-
ing removal would have to be zero (because each V&V step is always followed
by a rework step). Both are unrealistic values.
Applications of a Generic Work-Test-Rework Component 337
3 Sequential Processes
component can defined as arrays such that each index into the arrays effectively
represents an independent processing channel).
An important issue is the flow of faults within such a sequential process. The
model of Figure 3 represents a simple approach (suitable for small projects or orga-
nizations): First, the detected faults of a phase, which are below the quality limit
(i.e. which are not reworked but delivered), are handled in the next phase as left-
overs. To model this practice, the outputs DetectedFaultsDelivered are connected
to the “next” input DetectedFaultsReceived. Second, a simplified categorization
of the different test phases of the V-Modell assumes that the deliverable of the
implementation phase is tested in the module test, the deliverable of the design
phase in the integration test, and the deliverable of the requirement analysis in the
system test. Therefore, undetected faults of the implementation phase are most
likely to be found in the module test, undetected faults of the design phase in the
integration test, and undetected faults of the requirement analysis phase in the
system test. The undetected fault rates are connected accordingly in the model
of Figure 3. Third, faults detected during a test phase, i.e., during module, inte-
gration, or system test are corrected within the respective test phase as rework
without returning to a previous phase. Figure 3 represents this approach because
there is no feedback from the test phases into earlier process phases.
However, the process of Figure 3 can be refined to represent more sophisti-
cated fault handling procedures. For example, in many processes faults detected
during a test phase are reposted as a change request into the respective upstream
process phase, i.e., detected faults within a test phase trigger the development
of a new artifact (change request deliverable) in the respective upstream phase.
Subsequently, this artifact (e.g., a design change) follows the normal path, i.e,
runs through the other process phases (implementation, module test, integration
test, system test). Figure 4 includes such an extension as feedback from the test
Fig. 4. Model of a sequential process with reposting of fault corrections into upstream
process phases
Applications of a Generic Work-Test-Rework Component 339
Fig. 5. Simulation of a sequential process with reposting of fault corrections into up-
stream process phases
phases into the respective upstream phases (each test phase has a feedback into
only one upstream phase again just for simplicity).
Figure 5 shows a simulation of such an extended process. In the charts, only
the deliverable rates of the different phases are shown, the wider the peak, the
larger the deliverable. The reposting of the correction assignments (change re-
quests) and their traversing through the other process phases are the series of
small peaks following the main deliverable (wide peak). Note that due to their
small size, these change requests are processed very fast (almost appearing as
parallelism in the diagram).
Judging from the outcomes of Figure 5 only, the delay of a project due to
the re-traversing of the upstream phases seems reasonable because the change
requests traverse very fast through the process. However, this is only true, if
the required resources are always immediately available, which is not a realistic
assumption. Therefore, the model of Figure 4 contains a second set of extensions
to represent an organization in which each process phase belongs to different
organizational sub-units with own resources, but each sub-unit deals with only
one task at a time. This means, a request of a project gets all the resources of
a phase, but only on a first come first serve basis. Requests of projects arriving
after the resources are claimed, are postponed until the preceding request is
completely finished. The resulting queue is ordered according to the project
number, which reflects the project start, i.e., older projects get priority.
To model such a scenario, the execution of multiple projects can be simulated
in the model of Figure 4 using vectorized signals as mentioned above. The dimen-
sion of the vectors, i.e., the number of the projects, can be defined just by a pa-
rameter. The competing, mutually exclusive resource allocation is realized by the
340 T. Birkhölzer, D. Pfahl, and M. Schuster
4 Iterative Processes
As second context using a work-test-rework pattern, an iterative process model
is presented modeling the Scrum framework [10] [11]. In a Scrum process, the
project is divided into small tasks called user stories which are grouped into
sprints. Nevertheless, these user stories are iteratively analyzed, designed, im-
plemented (“work”), verified and validated (“test”), and corrected (“rework”).
Therefore, the work-test-rework-cycle forms the core of an iterative process as
well however processing singular pieces (user stories) instead of deliverables for
a whole project as in Section 3.
For an iteration, the implementation of the user stories (tasks) are grouped
into time-boxed sprints with a defined length, typically about 30 days. At the
start of a sprint, the content of the sprint is fixed in the sprint planning meeting
as sprint backlog. During the sprint, this backlog is worked off piecewise. Figure 7
shows the respective model of a sprint. The core element is the WorkTestRework
component, which processes the single tasks (user stories).
The subsystem ResourceManagement, see Figure 8, contains the block model-
ing the sprint backlog (component named SprintBacklog). This block is a Work-
Breakdown component, which is also part of the SimSWE library. The basic
functionality of this component is a randomized break down of a larger artifact
or task (sprint backlog) into smaller pieces (user stories). The characteristics of
this decomposition can be configured by parameters. Finished artifacts (real-
izations of the user stories including code, test cases, documentation, etc.) are
Applications of a Generic Work-Test-Rework Component 341
Undetected faults, on the other hand, are hidden in the delivered artifacts
and need to be detected before any removal action can take place. However, the
V&V effectiveness of faults within the context of a user story will be different to
the V&V effectiveness of undetected faults in the context of another user story.
Therefore, it is not appropriate to just feed back undetected errors into the next
cycle. Instead, they are accumulated within the subsystem FaultManagement. A
fraction of the resulting fault level (representing the accumulated hidden faults
in the project) is routed to any new task according to a parameter specifying
the effectiveness of the V&V steps of a user story with respect to faults in the
deliverables of another user story.
Figure 9 shows a simulation run of a single sprint. The two top-level charts de-
pict the delivery of the single tasks, the lower chart displays the cycling through
the work-test-rework states for each task. Note, that the last task is canceled and
not delivered (state 4 is not reached), because a sprint has a fixed, predefined
duration.
A Scrum process consists of an iteration of such sprints as described above,
the respective model is shown in Figure 10 containing the subsystem Sprint.
The assignment for each sprint is taken from a product backlog (block named
ProductBacklog, which is iteratively worked off similar to the sprint backlog. It
is also be modeled using the WorkBreakdown component. Unfinished parts of
the sprint backlog are (re)added to the product backlog at the end of a sprint.
Undetected faults of a sprint are accumulated as well. Again, a parameteriz-
able fraction of these undetected faults are fed back into the next sprint iteration
representing the faults, which might be detected in the next sprint representing
the cross-sprint V&V coverage as described above.
Figure 11 shows a simulation of the complete Scrum process. The upper chart
demonstrates the iterative project completion by the deliverables of the sprints.
Applications of a Generic Work-Test-Rework Component 343
The lower chart indicates that the number of undetected faults increases at the
beginning, but approaches an approximately stable level in the second half of the
simulation time. This is due to the modeling of a fixed V&V coverage across user
stories or sprints respectively as described above. It might be an interesting issue
for further research to assess, whether this model reflects real world observations,
and to devise more appropriate models otherwise.
The model of the Scrum process as shown in Figure 10 is in turn encapsulated
as a Scrum component as part of the SimSWE library. A complete description
of the component can be found in the related documentation at [12].
5 Discussion
The examples above demonstrate that the WorkTestRework component can be
used in different contexts representing different work types and artifacts (re-
quirement analysis, coding, test) as well as different scopes (project, task) just
by appropriate interfacing and parametrization. Based on this, effects of differ-
ent process settings, e.g., verification and validation intensities at various phases,
can be analyzed just by different parameter settings. Moreover, extensions could
344 T. Birkhölzer, D. Pfahl, and M. Schuster
easily be added to the model using the generic interfaces as demonstrated for
fault handling and resource allocation.
By providing essential functionality without restricting adaptability or flex-
ibility, even complex modeling and simulation scenarios (like the multi-project
simulation or the iteration workflow) can be realized with feasible effort. Thus,
the generic WorkTestRework component could be an important support for pro-
cess developers and modelers enabling to focus on the issues at hand by alle-
viating the need to (re)implement generic parts. Such reusable components are
considered an important groundwork for next generation process modeling and
simulation.
References
1. Abdel-Hamid, T., Madnick, S.E.: Software project dynamics: an integrated ap-
proach. Prentice-Hall, Inc., Upper Saddle River (1991)
2. Angkasaputra, N., Pfahl, D.: Making software process simulation modeling agile
and pattern-based. In: Proceedings of the 5th International Workshop on Software
Process Simulation Modeling (ProSim 2004), pp. 222–227 (2004)
3. Cass, A.G., Sutton Jr., S.M., Osterweil, L.J.: Formalizing rework in software pro-
cesses. In: Oquendo, F. (ed.) EWSPT 2003. LNCS, vol. 2786, pp. 16–31. Springer,
Heidelberg (2003)
4. Das V-Modell, http://v-modell.iabg.de (last accessed on 2010-01-05)
5. Garousi, V., Khosrovian, K., Pfahl, D.: A customizable pattern-based software
process simulation model: design, calibration and application. Software Process:
Improvement and Practice 14(3), 165–180 (2009)
6. Kellner, M.I., Madachy, R.J., Raffo, D.: Software process simulation modeling:
Why? what? how? Journal of Systems and Software 46(2-3), 91–105 (1999)
7. GNU lesser general public license, http://www.gnu.org/copyleft/lesser.html
(last accessed on 2010-01-05)
8. Madachy, R.J.: Reusable model structures and behaviors for software processes.
In: Wang, Q., Pfahl, D., Raffo, D.M., Wernick, P. (eds.) SPW 2006 and ProSim
2006. LNCS, vol. 3966, pp. 222–233. Springer, Heidelberg (2006)
9. Raffo, D., Nayak, U., Wakeland, W.: Implementing generalized process simulation
models. In: Pfahl, D., Raffo, D., Rus, I., Wernick, P. (eds.) Proceedings of the 6th
International Workshop on Software Process Simulation Modeling (ProSim 2005),
Fraunhofer IRB, St. Louis (2005)
10. Schwaber, K.: Scrum development process. In: Proceedings of the Conference on
Object-Oriented Programing Systems, Languages, and Applications (OOPSLA
1995) Workshop on Business Object Design and Implementation (1995)
11. ScrumAlliance, http://www.scrumalliance.org (last accessed on 2010-01-05)
12. SimSWE-Wiki,
http://simswe.ei.htwg-konstanz.de/wiki_simswe/index.php/Main_Page (last
accessed on 2010-04-10)
13. Zhang, H., Kitchenham, B., Pfahl, D.: Software process simulation modeling: Facts,
trends and directions. In: APSEC, pp. 59–66. IEEE Comp. Society, Los Alamitos
(2008)
An Empirical Study of Lead-Times in
Incremental and Agile Software Development
Kai Petersen
1 Introduction
Lead-time (also referred to as cycle-times) is the time it takes to process an order
from the request till the delivery [1]. An analysis and improvement of lead-time
is highly relevant. Not being able to deliver in short lead-times leads to a number
of disadvantages on the market, identified in the study of Bratthall et al. [2]: (1)
The risk of market lock-out is reduced [3]. Bratthall et al. [2] provided a concrete
example for that where one of the interviewees reported that they had to stall the
introduction of a new product because the competitor was introducing a similar
product one week earlier; (2) An early enrollment of a new product increase
probability of market dominance [4]. One of the participants in the study of
Bratthall et al. [2] reported that due to introducing a product three months after
a competitor the company is holding 30 % less of the world market in comparison
to the market leader; (3) Another benefit of being early on the market is that
the product conforms more to the expectations of the market [5]. This is due to
the market dynamics. Petersen et al. [6] found that a large portion (26 %) of
gathered requirements are already discarded during development. Furthermore,
the long lead-time provides a time-window for change requests and rework.
The review of literature revealed that, to the best of our knowledge, an em-
pirical analysis of lead-times in incremental and agile development has not been
conducted so far. However, as there is an increasing number of companies em-
ploying incremental and agile practices it is important to understand lead-time
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 345–356, 2010.
c Springer-Verlag Berlin Heidelberg 2010
346 K. Petersen
behavior. The studied company intended to determine target levels for lead-
times. The open question at the company was whether requirements should
have different target levels depending on the following factors:
This study investigated the effect of the three factors on lead-time. It is im-
portant to stress that existing work indicates what outcomes can be expected for
the different factors, the expected results being presented in the related work.
However, the outcome to be expected was not clear to the practitioners in the
studied company. Hence, this study sets out with formulating a set of hypothe-
ses related to the factors without assuming a specific outcome of the hypotheses
prior to analyzing them.
The research method used was an industrial case study of a company develop-
ing large-scale systems in the telecommunication domain. The quantitative data
was collected from a company proprietary system keeping track of the require-
ments flow throughout the software development lifecycle.
The remainder of the paper is structured as follows. Section 2 presents related
work. The research method is explained in Section 3. The results of the study
are shown in Section 4. Section 5 discusses the results. Section 6 concludes the
paper.
2 Related Work
Petersen et al. [6] present lead-times for waterfall development, showing that the
majority of the time (41 %) is spent on requirements engineering activities. The
remaining time was distributed as follows: 17 % in design and implementation,
19 % on verification, and 23 % on the release project. As in agile software
development the main activitis should be coding and testing [7] the literature
would suggest that those are the most time consuming activities.
Petersen and Wohlin [8] investigated issues hindering the performance of in-
cremental and agile development. When scaling agile the main issues are (1)
complex decision making in the requirements phase; (2) dependencies of com-
plex systems are not discovered early on; and (3) agile does not scale well as
complex architecture requires up-front planning. Given this qualitative result
the literature indicates that with increase of requirements impact the lead-time
should increase. For example, if a requirement can only be deliverable when parts
of it are implemented across several systems a delay in one system would lead
to prolonged lead-times for this requirement.
Harter et al. [9] identified that lines of code (size) is a predictor for cycle
time. This was confirmed by [10] who found that size was the only predictor for
An Empirical Study of Lead-Times 347
lead-time. Hence, from the related work point of view an increase of size should
lead to an increase of lead-time.
Collier [11] summarizes a number of issues in cycle time reduction and states:
(1) size prolongs lead-time, and (2) dependencies influence lead-time.
Carmel [12] investigated key success factors for achieving short lead-times.
The finding shows that team factors (small team size, cross-functional teams,
motivation) are critical. Furthermore, an awareness of lead-times is important
to choose actions specifically targeted towards lead-time reduction. However,
it is important to take quality into consideration when taking actions towards
lead-time reduction.
None of the lead-time studies focuses on agile development, and hence raising
the need for empirical studies on lead-time in an incremental and agile develop-
ment context.
3 Research Method
The research method used was a quantitative case study, the case being the
telecommunication company Ericsson AB. The systems studied were developed
in Sweden and India.
The research context is important to describe in order to know to what degree the
results of the study can be generalized [13]. Table 1 shows the context elements
for this study. The analysis focused on in total 12 systems of which 3 systems are
independent. The remaining nine systems belong to a very large communication
system and are highly dependent on each other. Thus, all requirements belonging
to the independent systems are treated as single-system requirements. The same
applies to requirements only affecting one of the nine dependent systems.
The process of the company is shown in Figure 1. Requirements from the
market are prioritized and described as high level requirements (HLR) in form
$QDWRP\
&RPSRXQG6\VWHP'HYHORSPHQW
&URVVIXQFWLRQDO
ZRUNWHDPV
$76SULQW $76SULQW
+LJK/HYHO
6\VWHP
6SHFLILFDWLRQ '56 $76SULQW $76SULQW
'HY
3ULR
'56 6\VWHP/HYHO7HVWZHHNF\FOHV
+/5
'56
+/5 $76SULQW $76SULQW $76SULQW
0DUNHW 6\VWHP
'56 $76SULQW $76SULQW 'HY
+/5
6\VWHP/HYHO7HVWZHHNF\FOHV
+/5
&RPSRXQG6\VWHP7HVWZHHNF\FOHV
3.2 Hypotheses
The hypotheses are related to differences between multi- and single-system re-
quirements, the distribution of the lead-time between phases, and the difference
between sizes of requirements. As mentioned earlier the goal is not to reject the
null hypotheses, but to determine whether the different factors lead to differ-
ences in lead-time. In the case of not rejecting the null-hypotheses the factors do
not affect the lead-time, while the rejection of the null-hypotheses implies that
the factors effect lead-time. The following hypotheses were made:
The lead-time is determined by keeping track of the duration the high level
requirements reside in different states. When a certain activity related to the
high-level requirement is executed (e.g. specification of the requirement) then
the requirement is put into that state. For the tracking of lead-times a time-
stamp was captured whenever a requirement enters a state, and leaves a state.
The lead-time data was collected from an electronic Kanban solution where
the requirements can be moved between phases to change their state. The system
can be edited over the web, showing the lists of the requirements and in which
phase they are. Whenever a person is moving a requirement from one phase to
another, a date is entered for this movement. The requirements go through the
following states:
– State Detailed Requirements Specification: The state starts with the decision
to hand over requirement to the cross-functional work-team for specification,
and ends with the hand-over to the development organization.
– State Implementation and Unit Test: The state starts with the hand-over of
the requirement to the development organization and ends with the delivery
to the system test.
– State Node/System Test: The state starts with the hand-over of the require-
ment to the system/node test and ends with the successful completion of the
compound system test. The time includes maintenance for fixing discovered
defects.
– State Ready for Release: The state starts when the requirement has success-
fully completed the compound system test and thus is ready for release to
the customer.
From the duration the requirements stay in the states the following lead-times
were calculated:
Waiting times are included in the lead-times. The accuracy of the measures
is high as the data was under regular review due to that the electronic Kanban
solution was used in daily work, and the data was subject to a monthly analysis
and review.
impact (H0,mult ). For the relationship between lead-time and system impact the
Pearson correlation was used to determine whether there is a linear relationship,
and the Spearman correlation to test whether there is a non-linear relation-
ship. For the relationship between lead-time and phase (H0,phase ) no correlation
was used as phase is a categorical variable. In order to capture whether phase
leads to variance in lead-time we test whether specific phases lead to variance in
lead-time, this is done by using stepwise regression analysis. Thereby, for each
category a dummy variable is introduced.
If there is a relationship between the variables (e.g. between system impact
and lead-time) this would mean that the system impact would be a variable
explaining some of the variance in the lead-time. The hypotheses for size (H0,size )
was only evaluated using descriptive statistics due to the limited number of data
points.
4 Results
4.1 Time Distribution Phases
Figure 2 shows the box-plots for lead-times between phases P1 (requirements
specification), P2 (implementation and unit test), P3 (node and system test),
and P4 (release projects). The box-plots do not provide a clear indication of the
differences of lead-time distribution between phases as the box-plots show high
overlaps between the phases.
/HDGWLPH
3KDVH
3333 3333 3333 3333 3333 3333 3333 3333 3333 3333 3333 3333 3333
6\VWHP $ % & ' ( ) * + , - . / /
V$
6\
0XOWL
3KDVH 35HT6SHF 3,PSO8WHVW 31RGH6\VWHVW 35HOHDVH3URM 7RWDO
/HDGWLPH
6\VWHP,PSDFW
3KDVH F VW VW M O
SH WH WH UR WD
T6 \V 3 7R
H O 8 6 V H
5 P
S H HD
3 , RG HO
3 1 5
3 3
Statistic Value p
Pearson (ϕ) -0.029 0.41
Spearman (ρ) 0.074 0.57
Expl. power (R2 ) 0.001
+/ − 1, and are not significant. Hence, this indicates that the two variables do
not seem to be related in the case of this company, leading to a rejection of
H0,multi .
Figure 5 shows the difference of lead-time between phases grouped by size, the
size being an expert estimate by requirements and system analysts. The sizes are
defined as intervals in person days, where Small(S) := [0; 300], M edium(M ) :=
[301; 699], and Large(L) := [700; ∞]. No correlation analysis was used for an-
alyzing this data as three groups only have two values, namely P2-Large, P3-
Medium, and P3-Large. The reason for the limitation was that only recently the
requirements were attributed with the size.
However, the data already shows that the difference for size seems to be small
in the phases requirements specification and node as well as compound system
testing. However, the size of requirements in the design phase shows a trend of
increased lead-time with increased size.
354 K. Petersen
/HDGWLPH
6L]H / 0 6 / 0 6 / 0 6
3KDVH 35HT6SHF 3,PSO8WHVW 31RGH6\VWHVW
5 Discussion
This section presents the practical and research implications. Furthermore, the
reasons for the results seen in the hypotheses tests are provided. The explanations
have been discussed within an analysis team at the studied companies and the
team agreed on the explanations given.
6 Conclusion
This paper evaluates software development lead-time in the context of a large-
scale organization using incremental and agile practices. The following observa-
tions were made regarding lead-time:
– Phases do not explain much of the variance in lead-time. From literature one
would expect that implementation and testing are the most time-consuming
activities in agile development. However, due to the context (large-scale)
other phases are equally time-consuming.
– There is no difference in lead-time for singe-system and multi-system re-
quirements. This finding also contradicts literature. An explanation is that
if a requirement has impact on multiple systems these systems drive each
other in implementing the requirements quickly.
– With increasing size the lead-time within the implementation phase in-
creases. This finding is in agreement with the related work.
In future work lead-times should be investigated in different contexts to pro-
vide further understanding of the behavior of lead-times in incremental and agile
development.
References
1. Carreira, B.: Lean manufacturing that works: powerful tools for dramatically re-
ducing waste and maximizing profits. American Management Association, New
York (2005)
356 K. Petersen
2. Bratthall, L., Runeson, P., Adelswärd, K., Eriksson, W.: A survey of lead-time
challenges in the development and evolution of distributed real-time systems. In-
formation & Software Technology 42(13), 947–958 (2000)
3. Schilling, M.A.: Technological lockout: an integrative model of the economic and
strategic factors driving technology success and failure. Academy of Management
Review 23(2), 267–284 (1998)
4. Urban, G.L., Carter, T., Gaskin, S., Mucha, Z.: Market share rewards to pio-
neering brands: an empirical analysis and strategic implications. Management Sci-
ence 32(6), 645–659 (1986)
5. Stalk, G.: Time - the next source of competitive advantage. Harvard Business
Review 66(4) (1988)
6. Petersen, K., Wohlin, C., Baca, D.: The waterfall model in large-scale development.
In: Proceedings of the 10th International Conference on Product-Focused Software
Process Improvement (PROFES 2009), pp. 386–400 (2009)
7. Beck, K.: Embracing change with extreme programming. IEEE Computer 32(10),
70–77 (1999)
8. Petersen, K., Wohlin, C.: A comparison of issues and advantages in agile and
incremental development between state of the art and an industrial case. Journal
of Systems and Software 82(9) (2009)
9. Harter, D.E., Krishnan, M.S., Slaughter, S.A.: Effects of process maturity on
quality, cycle time, and effort in software product development. Management Sci-
ence 46(4) (2000)
10. Agrawal, M., Chari, K.: Software effort, quality, and cycle time: A study of cmm
level 5 projects. IEEE Trans. Software Eng. 33(3), 145–156 (2007)
11. Aoyama, M.: Issues in software cycle time reduction. In: Proceedings of the 1995
IEEE Fourteenth Annual International Phoenix Conference on Computers and
Communications, pp. 302–309 (1995)
12. Carmel, E.: Cycle time in packaged software firms. Journal of Product Innovation
Management 12(2), 110–123 (1995)
13. Petersen, K., Wohlin, C.: Context in industrial software engineering research. In:
Proceedings of the 3rd International Symposium on Empirical Software Engineer-
ing and Measurement, pp. 401–404 (2010)
14. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslen, A.: Ex-
perimentation in Software Engineering: An Introduction (International Series in
Software Engineering). Springer, Heidelberg (2000)
15. Poppendieck, M., Poppendieck, T.: Lean software development: an agile toolkit.
Addison-Wesley, Boston (2003)
Improving the ROI of Software Quality Assurance
Activities: An Empirical Study
Abstract. Review, process audit, and testing are three main Quality Assurance
activities during the software development life cycle. They complement each
other to examine work products for defects and improvement opportunities to
the largest extent. Understanding the effort distribution and inter-correlation
among them will facilitate software organization project planning, improve the
software quality within the budget and schedule and make continuous process
improvement. This paper reports some empirical findings of effort distribution
pattern of the three types of QA activities from a series of incremental projects
in China. The result of the study gives us some implications on how to identify
which type of QA activity is insufficient while others might be overdone, how
to balance the effort allocation and planning for future projects, how to improve
the weak part of each QA activity and finally improve the Return On Invest-
ment (ROI) of QA activities and the whole process effectiveness under the spe-
cific organization context.
1 Introduction
For a successful project, acceptable quality must be achieved within an acceptable cost
and a reasonable schedule. Generally, if a high-quality product is required, more effort
would be put on quality assurance activities to achieve the required quality goal. IEEE’s
definition of Software Quality Assurance is: “A planned and systematic pattern of all
actions necessary to provide adequate confidence that an item or product conforms to
established technical requirements and a set of activities designed to evaluate the proc-
ess by which the products are developed or manufactured”[1]. Review, process audit
and testing are three popular quality assurance activities during the software life cycle.
Review, often referred to requirements review, formal design reviews, peer reviews
(inspection and walkthrough), is usually considered to be the most effective quality
assurance activity and the best industry practice according to many research findings[2-
5]. The maximum benefits come from review would prevent defects and save fixing
time in future work [6]. However, some studies show that one should not expect to
replace testing with reviews [6, 7]. Although the two quality assurance activities are
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 357–368, 2010.
© Springer-Verlag Berlin Heidelberg 2010
358 Q. Li et al.
overlapping in finding some common types of defects, e.g. simple programming blun-
ders and logic defects, they are fairly complementary in capturing different types of
defects. Defects of specification, missing portions, developer blind spots are easier to be
found by review, however, defects of numerical approximations and program dynamics
are harder to be found by review because reviewers can’t tell how fast and reliable the
system will be, how user-friendly the system is by just reviewing and deducing from the
abstract requirement, design documents or codes. Testing serves as a hands-on experi-
ence of the actual and operational system that review can’t achieve by imaging. It is
good at finding numerical approximations and program dynamics defects but harder to
find defects of specification, missing portions, and developer blind spots [6, 7]. Al-
though process audit has an indirect impact on the software product, it has been widely
accepted that [8] process audit can help developing team to work on the right direction,
adopt and tailor the most effective method that comes from the best industrial practices
to the specific context of the project, improve the process effectiveness and finally im-
prove the software quality.
Because the three types of quality assurance activities are complementary and can’t
be replaced by one another, how to balance the effort allocation among them to im-
prove resource allocation efficiency and maximize the Return On Investment (ROI) of
the entire quality assurance if we consider them as investments to the software devel-
opment life cycle is still an important yet challenging issue for project planning. In
this paper, based on the data from a Chinese software organization, we do an effort
distribution pattern analysis of three types of software quality assurance activities and
propose a diagnostic model to identify which type of QA activity is insufficient while
others might be overdone. This will give process implication on how to balance QA
effort allocation for future quality management plan, how to improve the weak part of
QA activities and finally improve the whole QA activities’ effectiveness under the
specific organization context.
The remainder of the paper is structured as follows: section 2 will introduce related
work, section 3 will introduce the background and data source of the study; section 4
will discuss the objective, methodology, major results of data analysis and process
implication; section 5 will discuss about the threats of validity; section 6 is the
conclusion.
2 Related Work
The efficiency of review and testing are compared in [11]. To determine the Defect
Removal Fraction (DRFs) associated with each of the six levels (i.e., Very Low, Low,
Nominal, High, Very High, Extra High) of the three profiles (i.e., automated analysis,
people reviews, execution testing and tools) for each of three types of defect artifacts
(i.e., requirement defects, design defects, and code defects), it conducted a two-round
Delphi. This study found that people review is the most efficient on removing re-
quirement and design defects, and testing is the most efficient on removing code
defects.
In [12], Capers Jones lists Defect Removal Efficiency of 16 combinations of 4 de-
fect removal methods: design inspections, code inspections, quality assurance (corre-
sponds to process audit in our study), and testing [12]. These results show that, on one
Improving the ROI of Software Quality Assurance Activities: An Empirical Study 359
side, no single defect removal method is adequate, on the other side, implies that
removal efficiency from better to worse would be design inspections, code inspec-
tions, testing and process audit.
However, both the results are generally from expert Delphi estimation and lack
quantitative verification. Second, both of them are based on the assumption that all
these defect removal strategies have been done sufficiently, but in practice, they
might be done insufficiently due to inappropriate project planning, time constraints or
market pressure. Meanwhile, from the economic aspect, the ROI might be negative if
they have been done sufficiently enough without consideration of cost. So both of the
related work doesn’t give any clue on optimal QA effort allocation guidelines to
maximize the ROI of QA activities. Although iDAVE [13] gives the decision support
of how much QA is enough, but it doesn’t address how much reviewing, process
auditing, and testing is enough respectively.
We could see in Table 1 that if we want to keep the current process maturity and also
achieve the same quality goal of 0.2defects/KLOC when it is released, the Chinese
software organization should put total around 40.13% effort on the QA activities:
review effort 4.30%, testing effort 34.47% and process audit effort 1.70% respec-
tively. The statistic description of QA effort distribution gives a baseline of required
QA effort percentage for required quality goals.
From this equation, assuming that other factors are the same, if review effort is in-
creased by 1%, testing effort is saved by 2.69%. The same analysis for process audit
is that if process audit effort is increased by 1%, testing effort is going to be saved by
362 Q. Li et al.
15.73%. The ROI of putting more effort on the review and process audit would be
(2.69%-1%)/1%=1.69 and (15.73%-1%)/1%=14.73 respectively.
Further considerations about this linear equation would be: First, under the extreme
condition that E_Re% and E_PA% are both set to 0, which means the organization
doesn’t have any activities related to review and process audit, the testing would take
up as high as 73% of the total engineering effort to satisfy the required quality goal.
This percentage of testing is far beyond the benchmark from industry at the same
level of quality which implies that the unreasonable QA effort allocation leads to a
huge waste of resources and effort. Second, if the coefficient of E_Re% or E_PA%
stands in the range from -1 to 0, which means if you add 1% more effort to review or
process audit, it will save less than 1% testing effort, so the ROI of adding more effort
on review or process audit will be negative. In such cases, you should reconsider
about reducing the effort put on review and process audit, and put more on testing. In
sum, this equation gives quality managers some insights about how to balance the
effort put on testing, process audit, and review based on history project data.
Process Implication: For this case study, testing is the most labor-intensive activities
of the QA activities and the effort put on review and process audit is insufficient, this
might give the quality manager some process improvement implication that putting
more effort on review and process audit to save the testing time, and finally shorten
the schedule and save the whole cost. Also, since the ROI of adding more effort on
process audit is larger than that of adding more effort on review, it also gives some
implications that for this software organization, improving the process audit activity
might be more cost-effective.
Process Implication: These findings from this analysis continue to support that proc-
ess audit and reviewing are more effective and efficient to detect defects than testing.
Adding more effort on early review, defects finding and process audit is more
cost-effective.
Improving the ROI of Software Quality Assurance Activities: An Empirical Study 363
Table 4. Hours per defect spent on finding and removing defects in each phase
From the analysis above, we see that it would be better to put more effort on early
review. The next question is shall we put equal effort on the requirement review,
design review and code walkthrough? A straight answer would be put the extra effort
on the weakest part would be more cost-effective. The next analysis will help quality
managers to identify which review is the weakest according to the context of the or-
ganization.
Review serves as filters, removing a percentage of the defects and let the left to en-
ter to the next phases, the definition the Filter Effectiveness (FE) of each phase:
Filter Effectiveness=defects found in this phase/defects introduced in this phase
Equation 3
364 Q. Li et al.
Process Implication: Based on the data from Table 7 and Table 8, we get the Filter
Effectiveness of each phase by reviewing in Table 5. We could see that the Filter
Effectiveness decreases as the project goes on, especially in coding phase, as low as
0.01 in V3.0. The process implication here might be that the development team relies
highly on the testing group. They should put more effort on code walkthrough and
unit testing before they release codes to the testing team.
From the analysis in section 4.2, it would be beneficial if we put more effort on process
audit too and this might even be more effective than review. The independent auditors
check the process with a set of checklists for the Process Areas that refer to CMMI v1.2
[9]. The NCs found for Process Areas are shown in Table 9. We find that Project Moni-
toring and Control (PMC) and Project Planning (PP) take most part of the NCs.
To get the further information from these NCs, we find that the main problem for
PMC is that developers often neglect submitting their periodical reports in time. Main
items of these reports include task completion ratio, planned and actual effort, prob-
lems or risks they encounter. These reports are usually used by project manager to
monitor developers’ task progress, identify risks and issues. However, missing these
reports would blindfold eyes of the project manager, some critical risks or issues
might be delayed to solve. By investigating the NCs related to PP, the most common
problem is that planned and actual effort displays a huge gap. Overestimating the task
effort might lead to project delay, shorten the time for testing and lead to low quality
product, while underestimates might lead to resource waste. The large proportion of
NCs related to PP shows the weakness of the effort estimation for project planning.
Further improvement would include creating guidelines for project effort estimation,
adopting more systematic and professional effort estimation methods or tools, such as
COCOMO tool to bridge the gap.
Improving the ROI of Software Quality Assurance Activities: An Empirical Study 365
Process Implication: Although this organization has already passed CMMI level 4,
but for this project, it is observable that it is operating at much lower CMMI levels
(e.g. lack of reporting and cost estimation). This analysis serves as a caution for this
organization to maintain their process maturity on a higher level and keep continuous
process improvement by putting more effort on process audit and improving the proc-
ess of Project Planning and Project Monitoring and Control in the future would bene-
fit the process improvement and achieve a higher ROI.
5 Threats to Validity
• Data Variance Analysis: In this case study, overall we deal with averages without
further analysis for its variance, e.g. baseline of required QA effort percentage in
Table 1, Cost of Finding and Removing Defects in Table 4 and Filter Effectiveness of
Each Phase by Reviewing in Table 5. We plan to revisit the raw data in the future to
do deeper root-cause analysis for variance, especially the large ones.
Acknowledgement
This work is cooperatively supported by the National Natural Science Foundation of
China under Grant Nos. 90718042, 60873072, and 60903050; the National Hi-Tech
R&D Plan of China under Grant Nos. 2007AA010303, 2007AA01Z186
and 2007AA01Z179; the National Basic Research Program under Grant No.
2007CB310802.
References
1. IEEE, IEEE Std 610.12-1990-IEEE Standard Glossary of Software Engineering Terminol-
ogy, Corrected Edition, in IEEE Software Engineering Standards Collection, The Institute
of Electrical and Electronics Engineers, New York (February 1991)
2. Gilb, Tom, Graham, D.: Software Inspection. Addison-Wesley, Wokingham (1993)
Improving the ROI of Software Quality Assurance Activities: An Empirical Study 367
3. Grady, R.B., Van Slack, T.: Key Lessons in Achieving Widespread Inspection Use. IEEE
Software 11(4), 46–57 (1994)
4. Holland, D.: Document Inspection as an Agent of Change. Software Quality Profes-
sional 2(1), 22–33 (1999)
5. Humphrey, W.S.: Managing the Software Process. Addison-Wesley, Reading (1989)
6. Boehm, B.W.: Software Engineering Economics. Prentice Hall, Englewood Cliffs (1981)
7. Wiegers, K.E.: Seven Truths about Peer Reviews. Cutter IT Journal (July 2002)
8. April, A., et al.: Process Assurance Audits: Lessons Learned. In: Proceedings of ICSE
1998, pp. 482–485 (1998)
9. CMMI Product Team. CMMI for Development, Version 1.2. Technical Report CMU/SEI-
2006-TR-008 (2006)
10. Li, Q., et al.: Bridge the Gap between Software Test Process and Business Value: a Case
Study. In: Wang, Q., Garousi, V., Madachy, R., Pfahl, D. (eds.) ICSP 2009. LNCS,
vol. 5543, pp. 212–223. Springer, Heidelberg (2009)
11. Boehm, B.W., et al.: Software Cost Estimation with COCOMO II, ch. 5, Table 5.37. Pren-
tice Hall, NY (2000)
12. Jones, C.: Applied Software Measurement: Global Analysis of Productivity and Quality,
3rd edn. McGraw-Hill, New York (2008)
13. Boehm, B.W., Huang, L., Jain, A., Madachy, R.J.: The ROI of Software Dependability:
The iDAVE Model. IEEE Software 21(3), 54–61 (2004)
14. Basili, V., Caldiera, G., Rombach, H.D.: The Goal Question Metric Approach, Encyclope-
dia of Software Engineering. John Wiley & Sons, Inc., Chichester (1994)
15. Madachy, R.J., Boehm, B.W.: Assessing Quality Processes with ODC COQUALMO. In:
Wang, Q., Pfahl, D., Raffo, D.M. (eds.) ICSP 2008. LNCS, vol. 5007, pp. 198–209.
Springer, Heidelberg (2008)
Appendix
Table 6. Effort (Man-Hours) and its Percentages of Each QA Activity
Slinger Jansen1 , Wouter Buts1 , Sjaak Brinkkemper1 , and André van der Hoek2
1
Department of Information and Computing Sciences, Utrecht University, Utrecht,
The Netherlands
{s.jansen,wftbuts,s.brinkkemper}@cs.uu.nl
2
Department of Informatics, University of California, Irvine, CA, USA
[email protected]
1 Introduction
J. Münch, Y. Yang, and W. Schäfer (Eds.): ICSP 2010, LNCS 6195, pp. 369–380, 2010.
c Springer-Verlag Berlin Heidelberg 2010
370 S. Jansen et al.
from twenty-six countries, participated in the survey. The respondents are prod-
uct managers. The participating companies received a custom-made benchmark
report in which their CCU process is compared to the total respondent base and
competitors in the same industry.
The contribution of this research is threefold. First, the state of the practice
of the CCU process of international product software vendors is investigated and
reported. Second, a benchmark is created for international product software ven-
dors, which gives them a clear insight in their CCU process. Third, three hypothe-
ses are investigated. Section 2 presents the CCU model. Based on this model, CCU
processes and practices are defined. In Section 3, the research design is presented,
consisting of the hypotheses and research approach. The research approach in-
cludes the creation of the survey that is designed to measure the processes and
practices defined in Section 2. In Section 4, the raw survey results are presented,
including the data for each CCU process. This data enables us to report on the
state of the practice of CCU. Section 5 presents our in-depth analysis of the hy-
potheses and section 6 touches upon validity constraints. Finally, in Section 7 the
conclusions are presented and possibilities for future research are discussed.
The survey presented in this paper, is based on Jansen and Brinkkemper’s CCU
model, displayed in Figure 1. The CCU model consists of a state diagram of
the customer on the right side and the supporting vendor functions on the left.
A specific concern for product software vendors is that every vendor has to
release, deliver and deploy its product(s) on a wide range of systems, in many
variations, and for a wide range of customers [7]. Each of the CCU processes
represents several practices. The practices consist of one or more capabilities. A
vendors CCU maturity is measured by one question per capability in the survey.
The processes and their practices used in the benchmark survey are defined
using the SPICE model for process assessment [8] [6]. The SPICE model defines a
process as “a statement of purpose and an essential set of practices that address
that purpose”. SPICE was used as a tool to help shape the survey and enabled
the researchers to take the CCU model and derive practices and capabilities from
the processes in a structured matter. The processes are defined as follows:
Release - The release process is made up of four practices. The first practice
is release frequency. The capabilities falling under the frequency practice are
that a vendor must frequently release major, minor, and bug fix releases and
that a vendor must synchronize these releases with customer convenience and
demand. The second practice is explicit release planning, which constitutes how
releases are planned within the organization. The third practice concerns release
scenarios and touches upon aspects such as scenario management and particular
policies involved. The final practice is management of (external) components
and dependencies. All dependencies between components, be they products that
have been built by the vendor or purchased externally, must be managed by
creating explicit dependencies between these products and components.
Benchmarking the CCU Process 371
Organization Customer
Uninformed
Release
Customer
SW Receive Info
Development
Advertise Update Informed
Sales
Customer
Repository
Configuration Deliver Update Customer
Product R1
Management Possesses Update
Product R2
System ..
.
Product Rn
Activation & Deployment
Deploy/Install Update
Rollback/
Deinstall
Deployment Deployment Installed
Support Feedback Customer
Configure
Activate Update
License(s) Activated
Usage
Licenses
Customer Deactivate
Usage Usage Feedback Reconfigure
Support
Delivery - With regards to the delivery process there are two practices to be
defined. The first practice prescribes that vendors must use every possible channel
for the distribution of products and updates [3]. A vendor conforms to this prac-
tice when the product is (for instance) not only available through the commercial
website but through physical media as well, such as for instance USB sticks. The
more channels the vendor uses to eliminate manual intervention for product and
update delivery, the higher its scores are. The second practice states that every
possible method for delivery must be applied for knowledge delivery. The knowl-
edge, such as product updates, product news, and customer feedback, should, for
instance, be delivered through (semi-) automatic push and pull delivery.
Deployment - There are four practices defined for the deployment process.
To begin with, a product must be removable without leaving any remnants of
data on a system. Secondly, dependency management should be implemented,
consisting of making dependencies explicit, and checking the customer’s con-
figuration before product deployment, to foresee and prevent errors. The third
deployment practice is that updates and installations must be able to deal with
customizations made by customers or third parties. A vendor supports the cus-
tomization practice when a software architecture is in place that enables cus-
tomizations. The fourth practice is deployment reliability, which is ensured by
capabilities such as validity checks, feedback reports, and externalization of cus-
tomer specific changes and data [9].
Usage - A vendors usage process is based on three practices. The first is li-
cense usage. A vendor must (semi) automatically handle all license requests and
distribute licenses. A vendor supports the license practice once customers can
372 S. Jansen et al.
explicitly manage their licenses and once licenses are generated automatically
when a sales contract is signed. Secondly, vendors must make use of feedback to
gain as much knowledge about the product in the field as possible (use usage-
and error reports). The third practice is that a vendor must be aware of its pay-
ment methods. A vendor sufficiently implements the payment practice when it
incorporates payment calculation through usage, user (name), time unit, floating
user, lump sum etc.
3 Research Design
In the area of CCU some explorative empirical work has been undertaken in the
form of case studies [5] and design research [10] [11] [2]. As the research area ma-
tures, surveys can be used to generalize theories and findings from the explorative
work. During the execution of the Dutch CCU survey comparable hypotheses were
addressed, as presented below [6]. Based on the hypothesis in the Dutch survey,
researchers found relations between change in the CCU process and success. Fur-
thermore, vendor views on the CCU process were recorded through the use of open
questions. In this research we repeat some of these questions in order to find out if
they hold on an international scale. We address the following hypotheses:
H1: Recent changes to the CCU process make the software product
more successful. When a software vendor changes its CCU process, it is likely
to improve customer experience and reduce the cost of updating per customer.
Research shows that Dutch companies experience increased product success after
adjusting the CCU process [6].
H2: The priority that the vendor gives to the CCU process has signif-
icant impact on the success of the software product. It is hypothesized
that software vendors who highly prioritize the CCU process dedicate more re-
sources to the CCU process, which should result in a more successful product.
H3: In their estimations for the future with regard to the state of
the practice of CCU, product software vendors are looking for update
tools that automate the CCU process and are customizable as well.
Based on Hall et al. [2], it is hypothesized that the CCU process will evolve
towards a more automated and customizable CCU process. At this point update
tools are available, but most of them are not generic enough to be adopted by
software vendors. Building tools (internally) from scratch, on the other hand,
is an expensive and time-consuming process. It is hypothesized that product
managers are looking for a tool combining the best of both worlds.
Respondents - Respondents organizational roles and functions matched the
following profile: the respondent is a product manager (or someone with a re-
lated function who has more than average knowledge of the CCU process) at a
product software company and completes the survey with regard to one prod-
uct in the vendor’s portfolio. Furthermore, the product is a software product
that is delivered to customers and runs at the customer site. The respondent’s
derived profile was explained in the invitation letter regarding the survey as
Benchmarking the CCU Process 373
well as at the front page of the survey website. Respondents were approached
direct, indirect and through domain specific discussion groups. The respondents
that were targeted directly were approached through University of California in-
dustry connections. These contacts consisted of business partners, alumni, and
external student advisors. From these leads, a list of people matching the profile
was generated. Potential respondents were invited by email twice. The survey
was sent out to over 200 e-mail adresses that were gathered from websites. Also,
the survey was posted on a well-visited forum. No questions concerned the way
in which the respondent heard about the survey.
Survey tool - The research question and hypotheses stated above are inves-
tigated by conducting an online benchmark survey and the analysis of the data
resulting from this survey. The survey consists of fifteen sections with four to
fourteen questions per section, adding up to a total of ninety-four questions. The
survey contained open, yes/no and multiple-choice questions1 .
An online survey tool was chosen because a large number of questions had to
be answered. We aimed for more than fifty respondents, which makes interviews
and paper surveys inefficient tools to use. The survey was accessible through a
website of the University of California, Irvine. After considering a final selection
of tools, Net Questionnaires was selected. Net Questionnaires offers the same
functions as a lot of other online survey tools but in addition offers support
for multiple types of questions and extensive SPSS export functionality. The
tool also allows respondents to pause the survey to continue at a later point in
time. The ability to pause the survey is convenient for such a lengthy survey.
The respondents are also able to browse back and forth through different survey
sections to confirm and change earlier answers. The survey was pre-tested by
two product software mangers during think aloud sessions, which led to minor
changes in use of language and terms. The majority of the questions contained
mouse over events that provided the respondents with explanations and examples
to clarify question goals. The survey’s underlying concepts were tested in five
expert interviews. A thinking aloud session with two potential respondents going
over the questionnaire resulted in small changes.
4 Results
The results can be downloaded2 . The majority of the respondents’ job functions
are closely related to the CCU process (developer, development manager, chief
executive or chief operating officer). Most of the participating companies are
relatively small: the smaller participant companies are startups or small busi-
nesses (one to ten employees). The current market is constantly bombarded with
new entrants and these companies are more eager to share their information in
an effort to improve their organization. A comparable amount of respondents
can be found in the range of fifty up to five hundred employees. The majority
1
The survey can be downloaded (in pdf format) from the following website:
http://www.ccubenchmark.com, verified on 15/10/’09.
2
http://www.softwareecosystems.org/SCR-2008-051.pdf, verified on 15/10/’09.
374 S. Jansen et al.
of the respondents produce products that are released in more than one mar-
ket. Furthermore, most commonly the respondents build products for business
productivity or system management.
To date, C++ is the main development technology for the survey respondents.
Next to C++, C, Java, Basic and C# are frequently used technologies. 66.7%
of the respondents has their headquarters in the United States. The dataset
contains companies from 26 different countries. 95% of these companies release
their product in the US, compared to 73% in Europe and around 50% in both
central and southern America as well as Asia. Moreover, 73% of the companies
build their product in the US. Europe and Asia coming (respectively 28 and 14
percent) are also favorable building locations. The US leads with 74%, Europe
comes in second place with 24% in regards to the region where intellectual prop-
erty is registered. The product sizes range from 200 to 2000 KLOC. About half
(53%) of the respondents indicate that their products architecture is based on
the client-server principle. Almost the same can be said for the products fol-
lowing the stand-alone architecture (49%). Many companies introduce parts of
their products as web-based components (33%). To complete the general data
overview: 18% of the respondents indicate they use open source components,
where 78% do not. This leaves 4% of the company products to be completely
open source.
Release - Generally, major product releases are published on a yearly basis.
In addition to these yearly major releases, minor releases are made available to
the market every three to ten months. Bug fix releases show a different pattern.
Some respondents publish these releases (almost) daily, others choose to adapt
to a reoccurring monthly schedule. Nine respondents do not use any form of pilot
testing before releasing the product or update. The rest use between one and one
hundred fifty pilot testers, with a mean of fourteen. Slightly more than half of the
product software vendors take customer convenience into account with regard
to release time (57%). Less than half of the tested companies use a formalized
release planning that contains specific dates for upcoming major, minor and bug
fix releases (45%). Of the respondents that actually perform release planning,
the largest part publishes its release planning in such a way that all internal
and product stakeholders can access the planning at all times (83%). Also, with
regard to responding vendors that indicate to use a release planning, more than
half of them use a formal publication policy concerning the release planning
document, which specifies policy decisions for a specific release (57%). Just more
than half of the respondents use a formalized release scenario that describes what
happens step by step on a release day (56%). 70% of the companies use tools
to support the CCU process. Another aspect related to explicit management of
tools is dependency management on third-party components. 69% of the software
vendors show that they explicitly manage relations between the products they
are selling and the external components that are needed to deliver an executable
package. Finally only 38% uses components-of-the-shelf. Half of the vendors
release their product when it is most convenient to the customer. More than
two thirds of the respondents explicitly manage these relations.
Benchmarking the CCU Process 375
Delivery - When one is dealing with the delivery process in a product software
company, choices have to be made regarding the information flows to customers. As
it turns out, the majority of companies informs their customers through a website
with all relevant information (84%). Sending individual emails is a popular tool as
well (81%). Interesting to note here is that only a small portion of the software ven-
dors uses unconventional methods to approach their existing and new customers.
Possibilities such as informing the customer through the product (17%), newslet-
ters (9%), general announcement lists (10%) and automatic push of updates and
information (8%) are used less frequently. Other approaches are the use of skilled
account/sales managers, blogs and user forums. 69% of the software vendors in-
form their customer between once a week and once every three months. Most of the
respondents initiate contact on a monthly basis. The majority of respondents use
e-mail as a way for customers to contact them and convey their problems (81%).
Almost half of that same group indicates to offer an online bug system for their
customers to report problems (46%). 72 companies let their customers manually
pull the releases from the vendor servers. 23 let users automatically pull releases
from their servers. 18 respondents manually push the product to the customer,
where only 10 do the push automatically. 30% of the responding companies use an
update tool that updates the product on the customer side. Respondents estimate
that, on average, 7.9% of their product installations fail at first attempt.
Deployment - For major releases, respondents indicate that 86% of their
customers install the product between one day and three to six months after
receiving news of an update. For minor releases, the same conclusion can be
drawn. Bugfix releases are generally deployed sooner. 85% of bugfix releases
are installed between one day and one month with the highest answer density
towards one day. The use of USB sticks is attractive (20%) for software delivery.
Another frequently used method is the file transfer protocol (FTP, 30%). In
56% of these software vendors the update tool is able to update the program at
runtime. All except one of the companies using an update tool are sure that their
tool is still able to deploy the product if the customer implements customizations,
extensions and/or customer specific solutions. 56% manage the relations between
proprietary and external products. Only 56% separates all data produced by the
user (documents, preferences) form the product data (components). Only thirty
percent of respondents use an update tool to deploy its product at the customer
side, even though on average eight percent of the product installations fail.
Usage - The respondents prefer payment per user (name): 46% of the re-
spondents use this licensing method. Other frequently used methods are pay
per usage (35%), lump sum (19%), pay for services (12%) and open source/no
payment (12%). Of the companies providing licenses, 17 out of 57 allow the
customer to renew, extend or expand the license without action or intervention
from the product software vendor side. Exactly half of the respondents offer
their customers licenses that expire. In 65 percent of the software vendors the
responding company is aware of the way that their customers customize prod-
ucts. 27 of the 78 participating companies send automatic error reports when
errors occur during product usage by the customer. From those 27, one does not
376 S. Jansen et al.
actually analyze the data fed back through the error reporting system. 31 com-
panies indicate that their products generate usage reports, 25 of them actually
analyze these generated results. License management is generally automated.
Other results - Respondents indicate to have made small changes to their
CCU process over the last two years (49%). 23% indicates not to have changed
anything, 22% has made large changes and 6% has completely re-designed the
CCU process. The reasons given most frequently for improving the CCU pro-
cess, are to serve customers more cost effectively, serve more customers, reduce
the number of installation problems and shorten the release cycle. Respondents
indicate to have other goals as well, such as simplifying the CCU process, in-
creasing stability of the product, better personal customer service and product
quality improvement. The respondents were asked how their product’s success
had changed over the last two years. Finally, 34 respondents see the CCU process
as a (very) high priority process, 35 are indifferent thinking CCU is neither a
high or low priority process and 9 respondents think it is a low priority process.
The following was derived from some of the open questions:
Deployment failure - According to submitters, future deployment failure will
be reduced by introducing testing procedures during the development phase,
user acceptance testing, and in depth pre-release code reviews. Submitters agree
that more planning, a controlled release process, auto updates, a more robust
setup process, and dependency checks should be used.
Custom made and commercially purchased tools - 50% of the respondents see
proprietary tools as being adequate. A small group says they would rather have
bought a solid commercial build tool to avoid a big part of current problems
and have access to support. The problem that is mentioned most often is that
of license management. A large group of submitters would like to be able to
implement a CRM module accessible by clients.
Evolving CCU process - Most experts believe that better organized firms will
make CCU a more central part of their infrastructure and procedures. The only
way to make it a more central part is to automate a significant part of the
CCU process, which will also involve support automation. Managers think that
more commercial tools will become available to support customer release man-
agement. Integration between aspects of customer release management such as
installation tools, product delivery and customer registration and bug tracking
is needed. Others think that there is no magic bullet and no significant changes,
since large corporations tend to more tightly control software deployment. These
problems will remain issues for as long as tools/environments/systems change.
When asking for the best possible solution for CCU the experts indicate that they
are looking for a customizable, stable, fully automated updating product with
sophisticated configuration options, robust logging and reporting. But, managers
are not sure if there is a ‘best’ solution that is perfect for every single case.
5 Results Analysis
H1: Recent change in the CCU process has significant impact on the
percieved success of the software product. In order to investigate H1 two
Benchmarking the CCU Process 377
specific questions were asked in the survey, being “Please indicate how your CCU
process evolved over the last two years” and “Please indicate how your product
developed over the last two years”. To the first question the respondent could
answer on a four point scale ranging from minor changes to a complete process
redesign and to the second question the respondent’s answer could range from
much less successful to much more successful. A cross-tab analysis was conducted
with a chi-square test between the variables of change and success (Pearson
Chi-Square = 37.7, p < 0.01). Figure 2 shows the cross-tab analysis results.
The horizontal axis shows the dependent success variable and the vertical axis
shows the change variable. The category: “the product is much less successful”
was omitted in the analysis because no respondents chose it. The figure shows
increasing percentages towards larger changes in the CCU process combined
with the respondents belief of a more successful product as the line in Figure 2
indicates. The software vendors are divided in four main groups in Figure 1.
Groups two and three support the hypothesis. Group two contains the software
vendors with small to no changes in the CCU process and are “less successful”
as well “as as successful as two years ago”. Group three contains the software
vendors with large and complete redesign changes in the CCU process whose
products success increased. The relationships are in line with the hypothesis.
Groups one and four do not support the hypothesis. By creating a group division
one is able to create a three dimensional table. In addition to these questions, an
extra question was asked that explicitly linked the change and success variables.
Group two and three were expected to show a higher percentage towards a strong
relation between success and changes because these groups implemented large
changes that resulted in more success or implemented small or no changes, which
resulted in no or less success. The opposite goes for group one and four. H1 holds,
with a minor proviso. For this survey, product sales, revenues, and profit were
not taken into account due to the diverse nature of respondents. As such, the
relationship is a perceived relationship and can not be validated fully.
H2: The priority that the vendor gives to the CCU process has signif-
icant impact on the success of the software product. In this hypothesis
the relation between the vendor’s CCU process priority and its effect on product
success is investigated. The same approach as in H1 is adopted. Two question
variables, being product success and CCU process priority, were compared in
a cross-tab analysis. The chi-square test was conducted (Pearson Chi-Square:
44.5, p < 0.01), which proves that there is a relation. Looking at these results it
can be concluded that when product software vendors regard their CCU process
to be of a (very) high priority, their product success is significantly higher. H2
holds, though be it with the same proviso as for H1.
H3: In their estimations for the future with regard to the state of
the practice of CCU, product software vendors are looking for cus-
tomizable update tools that automate the CCU process. Based on the
results from the open questions we accept the hypothesis. The respondents see
automation as the way to further reduce cost per customer. An interesting ad-
dition is the mention of web-based solutions by some of the respondents. Where
possible, some of the vendors hope to soon release a web based version of their
product, such that customers no longer have to bother with deploying a product
themselves. H3 holds.
There are several differences between the Dutch and international respondents.
For instance, the number of employees is 120 times bigger on average for interna-
tional respondents, ten times more end users are served, and products consist of
more KLOC. Surprising results are that the number of used programming lan-
guages is smaller for international vendors and that Dutch vendors work more
efficient with a 1:4 (employee/customer) ratio, over a 1:1.5 ratio for international
companies. The Dutch outperform their international colleagues in some areas,
such as pre-installation configuration checks.
6 Threats to Validity
The construct validity of the research is safeguarded in different ways. The
questions in the survey are based upon scientific constructs derived from liter-
ature. Respondents were asked to provide data from an existing product. To
Benchmarking the CCU Process 379
counteract guessing behavior, each question included the “I don’t know” option.
Survey heuristics developed by Fowler were applied in detail [12]. Possible prob-
lems with regard to understanding the questions were checked through expert
interviews and think-aloud sessions.
In regards to internal validity The survey respondents were treated anony-
mously, with regard to each other and could not see input of other participants.
Such a participant approach improves the truthfulness and reliability of answers.
Furthermore, the respondents were promised to receive an extensive benchmark
report including customized advice as incentive to participate in the research. An
important threat to H1 is the possible bias of providing social desirable answers.
In other words: when a respondent indicates to have made (major) changes to
its CCU process over the last two years, it might be more tempted to say that its
product was (a lot) more successful after this. There are vendors that indicate to
have a product that is less successful even though changes in the CCU process
might have been made. These results strengthen our belief that the possible bias
is of low risk to the research outcomes.
The challenge of getting respondents to participate in the research was ad-
dressed by offering them a customized benchmark report. Commercial resources
and contacts through industry and university networks were used to advertise
the survey amongst the target audience to gain a large data set to improve
external validity. After excluding incomplete- and non targeted respondent
software vendors from the dataset, seventy-eight software vendors remained.
References
1. Carzaniga, A., Fuggetta, A., Hall, R., van der Hoek, A., Heimbigner, D., Wolf, A.:
A characterization framework for software deployment technologies (1998)
2. Hall, R.S., Heimbigner, D., Wolf, A.L.: A cooperative approach to support software
deployment using the software dock. In: Proceedings of the International Confer-
ence on Software Engineering, pp. 174–183 (1999)
3. Jansen, S., Brinkkemper, S., Ballintijn, G.: A process framework and typology for
software product updaters. In: Ninth Eur. Conf. on Software Maintenance and
Reengineering, pp. 265–274. IEEE, Los Angeles (2005)
4. Brinkkemper, S., Xu, L.: Concepts for product software. European Journal of In-
formation Systems 16, 531–541
5. Jansen, S., Ballintijn, G., Brinkkemper, S., van Nieuwland, A.: Integrated develop-
ment and maintenance for the release, delivery, deployment, and customization of
product software: a case study in mass-market ERP software. Journal of Software
Maintenance and Evolution: Research and Practice 18, 133–151 (2006)
6. Jansen, S., Brinkkemper, S., Helms, R.: Benchmarking the Customer Configuration
Updating Practices of Product Software Vendors, 82–91 (2008)
7. van der Schuur, H., Jansen, S., Brinkkemper, S.: Becoming Responsive to Service
Usage and Performance Changes by Applying Service Feedback Metrics to Soft-
ware Maintenance. In: Proceedings of the 4th Intl. ERCIM Workshop on Software
Evolution and Evolvability, pp. 53–62 (2008)
8. Joint Technical Subcommittee between ISO (International Organization for Stan-
dardization) and IEC (International Electrotechnical Commission), Iso/iec 15504,
a.k.a. spice (software process improvement and capability determination)
9. Dolstra, E.: Integrating software construction and software deployment. In: West-
fechtel, B., van der Hoek, A. (eds.) SCM 2001 and SCM 2003. LNCS, vol. 2649,
pp. 102–117. Springer, Heidelberg (2003)
10. Dolstra, E., Visser, E., de Jonge, M.: Imposing a memory management discipline on
software deployment. In: Proceedings of the International Conference on Software
Engineering. IEEE, Los Alamitos (2004)
11. Jansen, S.: Pheme: An infrastructure to enable any type of communication between
a software vendor and an end-user . In: Proceedings of the International Conference
on Software Maintenance 2007, tool demonstration (2007)
12. Fowler, J.F.J.: Improving survey questions: Design and evaluation. Sage, Thousand
Oaks (1995)
Author Index