Assessing The Risk of Software Development
Assessing The Risk of Software Development
Assessing The Risk of Software Development
fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
Digital Object Identifier 10.1109/ACCESS.2017.DOI
ABSTRACT Agile methodologies aim to reduce software development risk using short iterations, feature-
driven development, continuous integration, testing automation, and other practices. However, the risk of
project failure or time and budget overruns is still a relevant problem. The aim of this paper is to present
and discuss a new approach to model some key risk factors in agile development, using software process
simulation modeling (SPSM), which can complement other approaches, and whose usage is particularly
suited for agile development. We introduce a new approach to modeling some key risk factors – namely
project duration, number of implemented issues, and key statistics of issue completion time – using a
simulator of agile development, which we developed to this purpose. The approach includes modeling the
agile process, gathering data from the tool used for project management, and performing Monte Carlo
simulations of the process, to get insights about the expected time and effort to complete the project,
and to their distributions. The model’s parameters that can cause risk are errors in effort estimation of
the features to develop, variations in developers’ assignment to these features, impediments related to
developers’ availability and work completion. To validate the simulator, and to demonstrate how the method
can be used, we analyzed three open source projects, gathering their data from JIRA repositories. We ran
Monte Carlo simulations of these projects, showing that the simulator can well approximate the progress of
the real project, then varying the identified risk factors and statistically evaluating their effects on the risk
parameters. The proposed approach is relevant for project managers, being able to quantitatively evaluate
the risks, provided that the process and the project’s data are properly modeled and gathered. As for the
future, we are working to improve our risk assessment method, evaluating it on more case studies, scaling
the model from a single team to multiple teams involved in one or more projects.
INDEX TERMS Effort estimation error, random issue allocation, risk analysis, software process develop-
ment, software process simulation.
I. INTRODUCTION so on.
Software technical risk is hard to define and there is no A broader definition is given by PMBoK (Project Manage-
agreement on a common definition. Risk is somehow defined ment Body of Knowledge book). Here, risk is defined as an
as a measure of the probability and severity of adverse "event or an uncertain condition that, if it occurs, can result
effects [29], inherent in the development of software which in positive (opportunities) or negative impacts (threats) in
does not meet its intended functions and performance re- one or more project objectives, such as scope, time, cost, and
quirements [14]. quality" [24].
The Software Engineering Institute (SEI) defines risk as Wallace et al. [50] identified and studied six dimensions of
the possibility of suffering loss [5]. In such context the loss software project risk, i.e. risks related to: Organization envi-
may describe any impact to the project, which could be in ronment, User, Requirements, Project complexity, Planning
the form of diminished quality of the end product, increased and control, Team.
costs, delayed completion, loss of market share, failure, and Traditional software development methodologies deal with
VOLUME 4, 2016 1
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
risk by performing a detailed up-front analysis with the However, how and to which extent SPSM can support
purpose of lowering risks related to poor requirements speci- software risk management is a topich that still needs to be
fication, and enforcing a strict control planning, with frequent better clarified, especially in case of ASD.
risk re-evaluations. However, they tend to integrate the vari- The goal of this work is to present a viable approach to risk
ous software modules late in the project, leaving room for management using SPSM, when using an agile development
integration risks. process, that is a process based on implementing require-
A newer approach is adopted by Agile Methodologies, ments or change requests specified as atomic "features" or
which were introduced in the late 90’s precisely to lower risks "issues". In term of research questions, this goal can be
due to changing requirements — a characteristic common specified as:
to most Internet-age software projects — but related also to RQ1: To which extent it is possible to automatically import
other issues, such as user involvement, team cooperation, late into the simulator data of real projects, from issue man-
integration, insufficient feedback [4], [33], [41], [23]. agement systems?
RQ2: How accurate can the simulator be in predicting
a: Problem statement
project completion times?
Software risk management is crucial for successful project RQ3: Can the simulator be useful to estimate project risk
management, but it is often an aspect not properly taken (induced by errors in efforts estimation, and random
into account in real projects. One of the main reasons is developer issues assignment) with a Monte Carlo ap-
that project managers often do not have effective tools and proach?
practices for software risk management.
In their work [12] Chadli and collaborators presented a
b: Contribution
research focused on discovering and classifying the various
tools mentioned in the literature which support Global soft- In this paper, we present a system to perform risk assessment
ware development (GSD) project managers, and on identi- of ASD projects using SPSM and a Monte Carlo stochastic
fying in which way they support group interaction. In this approach. Regarding Wallace’s risk dimensions [50], we
paper, the authors state that "Decision management, risk mainly consider requirement correctness and estimation. Our
management and measurement processes are not adequately approach can improve the planning and control of the project,
supported by tools when compared to the other Software and the management of its complexity. Also, team factors
Process Management processes". can be taken into account, by explicitly modeling developers’
Only recently, a few frameworks and tools have been intro- skills, impediments and turnover. The dimensions of organi-
duced for better addressing risk management in Agile Soft- zation environment and of user are out of the scope of the
ware Development (ASD). The results of a survey conducted proposed approach.
using a qualitative approach to analyze how risk management The basic tool of our system is an event-based simulator,
is carried out in Scrum software projects were presented able to simulate ASD based on the implementation of User
in [43]. De Souza Lopes et al. in [17] established a framework Stories (USs), or features, kept as independent as possible
called RIsk Management PRoduct Owner (RIMPRO), which from one another. The agile process may be iterative, as in
intends to support project teams to systematically manage Scrum, or using a continuous flow approach, as in Lean-
risks related to PO activities that may arise during the project Kanban approach.
definition proposed by PMBoK. The basic inputs to the simulator are the team composition
Risk is an ”event or an uncertain condition that, if it occurs, and skills, and the USs themselves. These inputs can be fed
can result in positive (opportunities) or negative impacts to the simulator from Issue Management tools. We have so
(threats) in one or more project objectives, such as scope, far implemented the interface to the popular JIRA tool [1].
time, cost, and quality” (PMI, 2017), in [6] intelligent agents Other inputs are parameters driving the random variation of
are used to manage risk. It does not simulate the process but team work, and of USs estimation and actual cost, as well as
only the stochastic impact of the variation [32], [38]. of possible events able to influence the project.
In such a scenario, Software Process Simulation Modeling The outputs of the simulation are key risk parameters, such
(SPSM) has emerged as a promising approach to address a as the distribution of times to complete the project, or the
variety of issues related to software engineering, including iteration, and of the forecast project cost. The simulations
risk management [35], [20]. Clearly, SPSM cannot address are performed hundreds, or thousands, of times, properly
risk dimensions such as organization environment stability varying the inputs, to get the distribution of the required
and management support, lack of User involvement, team outputs. The resulting distributions can provide hints on the
motivation and communication issues. SPSM can be useful expected outputs and of their variances, including proper risk
to establish the impact of risks on specific topics, such percentiles, useful to monitor development and to control
as requirement estimation errors, rework due to software risk. In fact, if risk assessment is not done in the initial stage,
that does not meet its intended functions and performance effort estimation will be strongly affected and the overall
requirements (software technical risk), project complexity, project may risk failure [28].
planning and control practices [49]. The main contributions of our work are:
2 VOLUME 4, 2016
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
• The development of a flexible and extensible ASD simu- The paper by de Oliveira Barros et al. [16] presents an
lator, able to simulate virtually every Agile development approach to develop, retrieve, and reuse management knowl-
process, based on incremental development. This sim- edge and experience concerned with software development
ulator models most aspects of ASD – team members, risks, using scenarios to model risk impact and resolution
USs, activities, events – and can be easily extended strategies efficacy.
and customized, due to the full object-oriented approach Shahzad and Al-Mudimigh [40] propose a model that takes
used for its development. care of the most frequently occurring risks, and provides a
• The development of a risk management model, consid- way for handling all of them. The sequence of activities for
ering team factors, requirement correctness and effort mitigating all risk factors are presented in a comprehensive
estimation. The input data, parameters and events, as model, which is useful for improving management and avoid-
well as the relevant output distributions, are analysed ance of software risk.
and discussed. Xiaosong et al. [52] use a classical matrix approach,
• The ability to feed project data from a popular issue defining 82 software project risk factors, aggregated in 14
management tools, namely JIRA, with the ability to kinds of risk, each one with a impact level on the overall
extend the simulator taking into account also other tools. project, from Negligible to Critical. Once experts evaluate
• A risk management analysis performed on three each factor, the risks are evaluated using the risk matrix.
medium-sized Open Source projects, comparing the Thereafter, high-level risks should be immediately addressed,
simulation results of different choices of the input pa- whereas medium and low risks will be monitored and kept
rameters. The results highlight both the prediction ac- under control.
curacy of our tool, and how it can be used to manage Roy et al. identified key risk factors and risk types for each
risk. of the development phases of SDLC (Software Development
Life Cycle) [37], including services and maintenance of
c: Outline software products.
Thieme et al. in their paper [45] present a study about the
The remainder of the paper is structured as follows. Section II
process for the analysis of functional software failures, their
reports the main related works on software risk management
propagation, and incorporation of the results in traditional
and on the application of SPSM to it. Section III presents
risk analysis methods (fault trees and event trees). A func-
the proposed risk-assessment method. Section IV describes
tional view on software is taken by allowing integration of
the simulation model. Section V presents how the risk-
software failure modes into risk analysis of the events and
assessment method is applied to in this paper. Section VI
effect. The proposed process can be applied during system
presents the case studies. Section VII presents the results
development and operation in order to analyses the risk level
on the case studies, and how they can be generalized and
and identify measures for system improvement.
applied to other projects. Section VIII presents the threats
to validity. Section IX eventually concludes the paper and
2) Risk Management in ASD
discusses future work.
Recently, agile software development methods have been
introduced, with the goal to be able to manage requirement
II. RELATED WORK
changes throughout project development. ASD might be
1) Risk Management in Software Projects considered also as a risk management approach, because
It is well known that several software projects suffer from errors and changes in requirements are one of the main risk
various kinds of problems, such as cost overruns, missing factors in software development. Among specific studies and
delivery deadlines and poor product quality. One of the approaches to manage project risk in ASD processes we
factors that cause these problems is the fact that the risks are already quoted the RIMPRO framework by de Souza Lopes
not handled, as shown by Charette [13]. et al. [17]. We also recall the work of Tavares et al. [44]
Risk management in software projects is a key component who propose Rm4Am, a tool which identified 5 components
of the success of a project. Software engineering researchers and 48 sub-components important in ASD, and conducted an
and professionals have proposed a number of systematic experiment with the supervision and participation of Agile
approaches and techniques for effective risk management as expert.
reported by Boehm [11].
A study conducted by the Project Management Institute 3) Simulation of Software Process
has shown that risk management is an activity not practiced Software Process Simulation Modeling (SPSM) is presented
among all the disciplines of project management in the IT as a promising approach suitable to address various kind
industry [18]. In real software projects, risks are often man- of issues in software engineering [26]. The results of the
aged using the insights of project manager, and the entire review conducted by Zhang et al. [53] showed that the risk
process of risk management is rarely followed [34]. One management is one of the several purposes for SPSM. Liu et.
of the main reasons for this is that project managers lack al. performed a systematic review on this topic, concluding
practical techniques and tools to effectively manage the risks. that the number of SPSM studies on software risk man-
VOLUME 4, 2016 3
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
agement has been gradually increasing in recent years, and threats in a software development project, to minimize risks.
that discrete-event simulation and system dynamics are two Their model, named "Apollonian", generated 20 rules that
most popular simulation paradigms, while hybrid simulation work as reference for any software development project, to
methods are more and more widely used [27]. assess the main threats to the project.
Practical examples of the use of SPSM are the works Raymond Joseph [25] propose a machine learning algo-
by Baum et al. [10], who compares pre-commit reviews rithm to generate risk prompts, based on software project
and post-commit reviews using process simulation through characteristics and other factors. His approach uses multiple
a parametric discrete event simulation model of a given multilabel artificial neural networks to label software projects
development process, and by Zhao et al. [54] who present according to the risks to which they are most exposed.
a fine-grained dynamic micro-simulation system based on an Han [22] trained a multi-layer-perceptron to assess the risk
agent model at the project level for Open Source Software level of a software project, using as learning set the OMRON
Development. dataset of 40 projects, each described by 24 risk-related
Discrete-event simulation of agile development practices parameters. The results look better than a more traditional
was first introduced by Melis et al. [30]. Anderson et al. [7] logistic regression.
proposed an event-driven, agent-based simulation model for Min et al. [31] applied fuzzy comprehensive evaluation to
Lean-Kanban process, extensible to other Agile software estimate a project’s risk level. The approach is somewhat
processes, and used it to demonstrate the effectiveness of a similar to, though much simpler than, that of ref. Fenton,
WIP-limited approach, and to optimize the WIP limits in the but instead of Bayesian nets fuzzy algebra is used to find the
various process activities. In a subsequent work, Anderson et probability of risks, given estimated risk factors.
al. [8] used an extended version of their simulator to compare Alencar at al. [6] propose a proactive and automated
Lean-Kanban with traditional and Scrum approaches on the approach based on agent technology to assist the software
data collected from a Microsoft maintenance project, show- project manager in the execution of the Risk Management
ing that the Lean-Kanban approach is superior to the others. processes.
Turner et al. worked on modeling and simulation of Kan- Abioye et al. [3] present a life-cycle approach to ontology-
ban processes in Systems Engineering [48], [47]. These two based risk management framework for software projects us-
works, though quite preliminary, propose the use of a mixed ing a dataset gathered from literature, domain experts, and
approach, merging Discrete Event and Agent-based simula- practitioners. The risks are conceptualized, modeled, and
tion approaches. In particular, Discrete Event simulation is developed using Protégé. The framework was adopted in
used to simulate the flow of high-level tasks and the accu- real-life software projects.
mulation of value, whereas Agent-based simulation is used Asif et al. [9] identify the relationship between risk factors
to model the workflow at a lower level, including working and mitigation actions automatically by using an intelli-
teams, Kanban boards, work items and activities. gent Decision Support System. The DSS is rule-based, and
Wang [51] introduces an agent-based simulation tool to identifies the rules using the Equivalence Class Clustering
compare solo and pair-programming, and other agile prac- and bottom-up Lattice Traversal (ECLAT) algorithm, starting
tices, in the context of Scrum development. The simula- from expert knowledge and literature. 26 highly cited risk
tion tool can simulate all types of Scrum context and team factors and 57 risk mitigations were identified from the
composition to test designed strategies under various what-if literature, and associated through the rules of the DSS, to help
assumptions in agent-based modelling. project managers to mitigate the risks.
4) Automated Approaches for Software Risk Management 5) SPSM for Risk Management
Another field of interest related to software risk management The literature about SPSM applied to risk management is
is the application of Machine Learning, and similar auto- still quite limited. Ghane [21] observes that ASD delivers
mated techniques. workable software in short cycles, and this helps with collect-
In this field, Fenton et al. [19] developed a complex causal ing more heuristic data as compared to traditional waterfall
model based on Bayesian networks for predicting the number methodologies. Such data can be used as quantitative metrics
of residual defects of a system. Their approach accounts for time and effort estimation. He then introduces a risk
for various phases of software development, among which management model that uses project simulation to produce
requirement analysis, design and programming, testing and risk metrics used to help with risk avoidance and mitigation.
rework. Each phase is modeled by a Bayesian net, whose These metrics are used to adjust project factors such as time,
probability values and functions are set by project managers. cost and scope during the lifespan of project.
The model allows managers to perform various types of Singh et al. [42] represent the PERT graph of a software
what-if-analyses and trade-offs, focusing on minimizing the project, where nodes are the states, and arcs represent activi-
number of defects, which is a key factor in risk management. ties. Each arc bears mean and standard deviation of its cost. A
Tinjacá Rodríguez et al. [36] present a model based on Monte Carlo simulator stimulates the model with thousands
the A.I. technique called "Rough Set" for selecting and of executions, to find the cost distribution, and the critical
prioritizing, in environments of uncertainty, a set of critical paths.
4 VOLUME 4, 2016
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
represented by a sequence of events in chronological order. 1) The simulation starts at time zero: t = 0. Time t is
Each event occurs at an instant of the simulation time, and expressed in working days. Each day involves 8 hours
causes a change in the state of the system. The simulator of work. At the beginning, the system may already hold
is also based on agents (the team members), who exhibit an initial backlog of issues to be worked out.
autonomous behavior. The agent’s actions are not fixed, but 2) The simulation proceeds by time steps of one day, until
depend on to the simulated environment. a predefined end, or until the event queue is empty.
3) Issues are entered at given days, drawn from a random
A. BASIC COMPONENTS distribution, or given as input to the system. Each issue
The basic components of the SPSM model are the issues, the is endowed with a collection of effort values for each
activities, the team members and the events: activity (in days of work), to be performed to complete
• Issues are the atomic work items in a development or the issue. These values can be drawn from a distribution,
maintenance project. They correspond to the features or obtained from real data.
described in the Lean-Kanban approach and are similar 4) Each issue passes through the activities. Each activity
to Scrum USs. Each unit of work is characterized by an takes a percentage of the whole effort to process the is-
unique identifier, a collection of effort values (in man- sue. The sum of the percentages of all the activities must
days) expressing the amount of work needed in each be 100%. When an issue enters an activity, the actual
activity to complete the issue, a priority (an integer in a effort (in man-days) needed to complete the activity is
given range, expressing the importance of the issue), and equal to the total effort of the issue multiplied by the
information of the actual effort spent in each activity. proper percentage.
When the last activity is completed, the issue becomes 5) At the beginning of each day, the developers choose the
closed. issue (and the activity) they will work on in the day.
• Activities represent the kinds of work that has to be This choice is driven by the developer’s skills, by the
done on the issues to complete them. Typically, they are: issue priority, and by the preference to keep working
Planning, Analysis, Coding and Testing, but they can be on the same issue of the preceding day, if possible. In
configured on the specific development process chosen fact, whenever an issue is processed for the first time,
by the team. or in case of developer switching from another issue, a
• Team members (Developers) hold the information on multiplicative penalty factor p, with p ≥ 1, is applied
the actual developers, which includes the skills in the to compute the time effort, in order to model time waste
various activities. If the skill is equal to one, it means due to task switching (extra time needed to study the
that the team member will perform work in that activity issue, which is proportional to the size of the task).
according to the declared effort. For instance, if the The task switching problem is well known in software
effort is one man day, the member will complete that engineering, as reported also recently by Abad et al. [2].
effort in one man day. If the skill is lower than one, 6) When the work on an issue in a given activity ends, the
for instance 0.8, it means that one-day effort will be issue is pulled to the next activity, where it can be chosen
completed in 1/0.8 = 1.25 days. A skill lower than one by a developer, or is closed in the case of last activity.
can represent an actual impairment of members, or the The developer who ended the work will choose another
fact that they have also other duties, and are not able to issue to work on, which might be the same issue, in the
work full time on the issues. If the skill for an activity is subsequent activity, or not.
zero, the member will not work in that activity. The presented simulation model can represent develop-
• Events represent what happens at a given time, which is ment, or maintenance processes, and can be customized
relevant to the development. We manage three kinds of to cater for the specific process of an organization. From
events: i) the creation of an issue; ii) the start of work on this generic model, we can derive various specific models.
an issue in a given activity; iii) the end of the work on For instance, one might introduce WIP limits, as suggested
the issue in a given activity. Each event has a time, and by the Lean-Kanban approach. For a WIP-limited process,
is executed by the simulator when its time arrives. the model has to be complemented by adding limits to the
maximum number of issues than can be processed at any
B. THE SIMULATION PROCESS given time inside an activity.
The modeled development process proceeds through a se- If a Scrum-like development has to be modeled, this can be
quence of steps. Preliminarly, one has to define the ASD accomplished by defining the length of the iteration (Sprint),
process, that is what are the activities of the process, their se- and managing the number and total effort of new issues
quence and their average relative weight on the total effort to entering at the beginning of each Sprint.
complete an issue. Secondly, the development team members
must be created, each having different skills in the various C. SIMULATOR DESIGN
activities. The simulator is implemented using Smalltalk, a language
Then, the simulator is started, executing steps with the very suited to event-driven simulation and very flexible to ac-
following characteristics: commodate any kind of changes and upgrades to the model.
6 VOLUME 4, 2016
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
pendently from other issues – and are explicitly linked to a FIGURE 4. The UML activity diagram showing how the event-driven simulator
works.
specific project.
New issues can be created as time proceeds. Each issue is
characterized by: an id, a state, the original effort estimate,
the effort actually spent to date. All efforts are in man-days. beginning of each Sprint in Scrum. The latter is where the
The possible states of the issue are shown in Fig. 3. At the completed issues are kept.
beginning, an issue is In backlog. Then, it can be chosen for When work starts on one issue within a given activity, the
development (To Do), and subsequently is pulled into the first actual effort of development of the issue in the activity is
activity, were it waits to be assigned to a developer (Waiting computed. This is accomplished by randomly increasing or
to be assigned). When a developer subscribes to it, its state decreasing it of a percentage drawn from a given distribution
becomes Under work. When work in the current activity is Of course, the average effort of all issues must be equal
finished, its state becomes Work done, and the issue waits to to the average of their initial estimates. A way to obtain
be pulled into the next activity. If the activity where the work this behavior is to multiply the estimate effort by a random
was done is the last one of the process, the status of the issue number r drawn from a log-normal distribution with mean
becomes Released, which is the final state. equal to 1 and standard deviation depending on the wished
perturbation.
Activities
Each project holds a list of all activities defining the ASD Events
process followed. These represent the specific work to be The simulator is event-driven, meaning that the simulation
done on the issues; each of them covers a given percentage proceeds by executing events, according to their time order-
of the total effort needed to complete the issue. ing and priority. When an event is executed, the time of the
Activities can be freely configured according to the process simulation is set to the time of the event. The simulator holds
steps. Each activity is characterized by its name, and by the an event queue, where events to be executed are stored sorted
typical percentage of the total estimated time effort of an by time and priority. When an event is executed, it changes
issue that pertains to the activity. For instance, if an issue has the state of the system, and it can generate new events, with
an overall estimate of 10 days, and the Testing activity has a times equal to, o greater than, the current time, inserting them
percentage of 15%, in this phase the feature will be estimated into the event queue. The simulation ends when the event
to last 1.5 days. The sum of the percentages of all the project queue is empty, or possibly if a maximum time is reached,
activities must be one. marked by an "End of simulation" event.
The first and the last activities are Backlog and Live The The time is recorded in nominal working days, from the
first and the last activities are Backlog and Live. They are start of the simulation. A day can be considered to have 8
special activities, because no work is performed on them – so nominal working hours, but developers can work more or
their effort percentage is zero. The former is a placeholder, less than 8 hours. If we want to consider calendar days, it
containing all issues put into processing, for instance at the is always possible to convert from nominal working days to
8 VOLUME 4, 2016
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
TABLE 2. The simulator inputs, highlighting the inputs imported from JIRA. "PART." means "partially".
Information on the ASD process used must also be collected Subsequently, we need to define what are the causes of
from the team. It includes: risk, which can affect the desired outcomes of the project
• the sequence of activities performed by the team; at least – in our case duration and cost. The main risk factors we
they include: Backlog, In progress, Done; identified are erroneous effort estimates of the issues, subop-
• the duration of a Sprint, and the way Story Points are timal allocation of the issues to developers, and blocks and
computed to decide how many USs to include in each impediments to the work of developers. Once the statistics of
Sprint, if Scrum is used; these factors are precisely defined, a Monte Carlo simulation
• the maximum number of USs allowed in each activity, can be used to assess the risks in a quantitative way.
if Lean-Kanban approach is used.
Starting from this information, it is possible to compute the VI. EXPERIMENTAL DATA
outputs of SPSM, which typically are: We analyzed three open source projects which comply with
the following preconditions:
• The project duration if no more issues are entered;
• The number and effort of issued forecast to be closed on 1. they are tracked on the JIRA System,
a given date; 2. they have CreationDate, OriginalTimeEstimate, Time-
• An estimate of average, standard deviation and median Spent and Assignee fields filled out for each issue; the
of the issue cycle time, that is the time needed from the last two are reported in man hours;
start of item processing to its completion. 3. they are medium size (meaning a number of issues
Fig. 5 shows the basic inputs and outputs of our model, above a few hundreds, and below one thousand).
based on JIRA issue data. Table 2 shows the parameters The medium size requirement allows us to perform a
used in the simulation, including the class they belong to, deeper analysis of the project and of the correspondent
their type, and whether they were read from JIRA, estimated outcomes when two-months intervals are examined, which
from JIRA data, set after expert consultation, or dynamically would be too time consuming for large size projects. The
computed during the simulation. size is also kept limited in order to be able to perform several
To perform a risk analysis using SPSM, first of all we need Monte Carlo runs (≥ 100).
to assess the suitability of the simulator to effectively model The selected projects have a duration of around 15-20
the software development process. This can be accomplished months, and an average time span of 330 days. The number of
by running the simulator on real data, and verifying its ability team members varies between 15 and 60, but in the latter case
to mimic the real development. team members do not work simultaneously, so on average
10 VOLUME 4, 2016
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
there are 15-20 developers active. TABLE 4. Total project duration. Mean (st. dev.) over 100 simulations.
These projects, all tracked in JIRA, are built for different
domains and purposes, so they present differences in topic,
duration, team size and composition, workflow. The simula- N. days to close project
tor is able to quite faithfully reproduce each of them. Project name N. Issues Real case Simulated case
Test Engineering (TE) 285 408 433 (31)
A. PROJECT: TEST ENGINEERING Platform 321 445 415 (26)
The first development project is TE (Test Engineering), car- CORD 205 138 154 (12)
ried on by edX (www.edx.org), a consortium of more than
160 universities offering courses and programs through an e-
learning platform completely open-source. TE is an internal and carrier business models. CORD is an open source project
project to perform testing and continuous integration of the aiming to develop a platform to build agile datacenters for the
software developed for edX, and by edX partners. network edge.
TE is an ongoing project, which we analyzed for a total
We analyzed the CORD project for a total of 192 working
of 570 working days, including 675 issues classified as: bug,
days, including 523 issues classified as: subtask, feature, bug,
epic, story and task. The team is composed by 13 developers
story and epic. Issues effort estimation approximately also
with different skills and productivity, inferred by analyzing
follows a Pareto distribution with shape value b = 1.51. The
the number and effort of issues completed by each devel-
basic statistics for the CORD project are shown in Table 3.
oper in the considered time interval. Issues effort estimation
follows a very skewed, fat-tail distribution, which is fairly
D. SIMULATOR ASSESSMENT
approximated by a Pareto distribution:
The first step to assess the validity of our simulator was
b to check whether it is able to produce time duration for
p(X) = b+1
. (2)
X completion of all issues similar to those of the real projects.
where X is expressed in man days, and the shape value is We performed simulations giving as input all issues which
b ≈ 1.35. resulted in Closed state, or with a remaining effort equal to
The basic statistics for the TE project are shown in Table 3. zero, among all issues read from JIRA repositories. The work
on each issue was performed by the same team member of
TABLE 3. Projects’ Statistics. Effort statistics in man days. JIRA data, using the developer’s productivity as estimated
from the same data.
Project name Days Issues Developers Mean Std Dev Median
For performing the simulations, we shortened their dura-
Test Engineering (TE) 570 675 15-20 2.9 4.2 1.33 tion to an interval of about 70% of the total, starting from
Platform 622 853 14-15 4.4 6.4 1.1 the date of the first completed issue. This because almost all
CORD 192 523 13 4.3 6.4 1.0
issues created afterwards were not yet completed.
Two among the selected projects have a simulation du-
ration of about 14 months (408 and 445 days). CORD was
B. PROJECT: PLATFORM tested for a shorter period of 138 days, or 4.5 months. The
The second project is also carried on by edX, and regards number of team members are around fifteen people in all
the e-learning platform of edX consortium. In fact, it is projects, remembering that this is an average team size,
called "Platform". Platform is an ongoing project, which we obtained by putting to work only the developers who are
analyzed for a total of 622 working days, including 853 issues actually active in the proper time interval, as inferred by JIRA
classified as: subtask, bug, story, and epic. data.
The team size is 65 people. At a first glance it might seem These projects are built for different domains and pur-
too big, but after a careful check we found that there was a poses, so they present differences in topic, duration, team size
high turnover. Most team members worked only for a limited and composition, workflow. The simulator is able to faith-
amount of time, so that the average team size is about 14-15 fully reproduce each of them. All projects are decomposed in
people, similar to the other projects. several issues. Each issue has a total effort estimation given
Issues effort estimation approximately follows a Pareto in man days. Different kind of issues belonging to a project
distribution with shape value b = 1.38. The basic statistics may have a different workflow. This is modeled according to
for the Platform project are shown in Table 3. the real schema including all main activities (states).
The results for the three projects are shown in Table 4,
C. PROJECT: CORD which shows the actual duration with the average duration of
The last project is called CORD (Central Office Re- the simulations. Standard deviations are shown within round
architected as a Datacenter). It is a project of the Open brackets after the means.
Networking Foundation (ONF), a non-profit operator led Besides running the simulator for the whole development
consortium driving transformation of network infrastructure time, we also refined the effectiveness tests by considering
VOLUME 4, 2016 11
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
shorter time intervals. We considered time intervals of 60 We deemed these results good enough to consider the
days, being two months the shortest time suitable to get simulator suitable for its use in risk analysis.
significant results. Trying shorter duration did not work, both
because some issues require at least 60 days to be completed, VII. RISK ASSESSMENT THROUGH SIMULATION
and because the number of issues to test in each interval We applied the Risk assessment methodology outlined in
becomes too small. Section III-A on the reported cases, to test its effectiveness.
We divided the project duration in intervals of 60 days, The key risk factors identified are: variations in efforts esti-
considering for each interval the issues actually created, and mated by developers to complete USs, and random developer
the team members actually working in the real project. The issues assignment – to be compared with assignment accord-
issues not yet completed at the end of the interval are left to ing to real data. In this preliminary study, we did not consider
the next one for the remaining part, and are considered to be variations in the skills of team members, and events stalling
completed only when their status becomes Closed. In these the development of single features, or blocking one or more
tests, the key parameter is not the duration of the project, but developers, though the simulator could account also for these
the number of issues completed in each interval, and the total factors.
number of issues completed at the end of the last interval. The effort estimation error of each issue is modeled using
random variations. The Percentage differences between esti-
TABLE 5. Results of simulations with 60 days intervals. mated and actual times to close an issue in the three projects
are very close to zero, and show a standard deviation between
Project name Days Issues Mean Std Dev Simul. Issues Diff.
0.17 and 0.26. Averaging on all closed issues of the three
Test Engineering (TE) 540 369 41 24.5 370 1 project, the standard deviation is 0.22, which we approximate
Platform 600 431 43 26.9 426 -5 to 0.2.
CORD 180 265 83.3 62.9 267 2
So, the effort estimation error is obtained by multiplying
the original issue effort value by a number that follows a log-
Table 5 shows statistics and results of the simulations normal distribution with average equal to one, and standard
for the three examined projects. Note that overall time in- deviation equal to 0.2. In this way, the efforts averaged on all
tervals considered are longer than those of the preceding issues remain equal to the average of original issues, and the
simulations, and run as long as possible, compatibly with the standard deviation of errors is very similar.
constraint to be multiple of 60 days. The reported mean and The key process outputs whose variations are checked
standard deviations are made on real issues, over the 60 days are the project total time and statistics on cycle times to
time intervals. The simulated issues are averaged over 100 implement a feature. The time from project start to the time
simulations for each project. when the last feature is released is inversely proportional
To give a further insight on these simulations, in Table 6 to the throughput, and is a very good indicator of the good
we report the number of issues closed in every 60-day in- performance of the project.
terval for project Test Engineering. The number referring to The statistics on cycle times, measuring time to give value
simulated issues is again the average over 100 simulations. to the customer, are:
The last columns reports the real values. Similar behavior can • average time to implement a feature;
be found in the issue management of the other projects. As • standard deviation of the time;
you can see, the number of closed issues varies a great deal • median time, measuring the most likely time to com-
from two months to two months. The simulation is able to plete a feature;
reproduce quite well the number of closed issues in every 60- • 95% percentile of times, measuring the limit of the
day interval. worst 5% times;
• 5%percentile of times, measuring the limit of the best
TABLE 6. Issue Management in 60-day intervals 5% times;
• maximum time, measuring the worst case.
TE (Test Engineering) Project We perform a given number of Monte Carlo simulations
Days Closed Issues (Mean) Real Closed Issues – typically one hundred, but they might be more – for each
60 56 54 choice of the tested risk factors, varying random perturba-
120 10 9 tion, or developers’ assignments to issues, recording the key
180 39 36 outputs.
240 58 60
300 44 46 From these outputs it is then possible to compute the
360 84 85 desired statistics. On each of these values it is possible to set
420 25 26 risk thresholds that, if reached or overcome, trigger proper
480 10 9
540 44 44
mitigation actions.
12 VOLUME 4, 2016
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
FIGURE 9. TE Project: N. of completed issues vs. time. Averages and FIGURE 10. Platform Project: N. of completed issues vs. time. Averages and
percentiles over 100 simulations, with random allocation of developers. percentiles over 100 simulations, with random allocation of developers.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
other kinds of risks are considered, besides error in issue Clearly, in real projects this approach should be embedded
estimation. A couple of the project managers stressed that in other risk management approaches, which include risk
this tool might be useful, but only if provided of a user identification and risk mitigation. Our SPSM-based approach
interface able to ease the tuning of the simulation parameters, can substantially help in quantitative risk analysis, but cannot
and to immediately highlight the risks to exceed time and cover different kinds of risk that cannot be modeled and
costs. simulated, such as Organizational Environment Risk, User
This leads to the answer to the last research question: Risk, and Team Risk [50]. In other words, we do not claim
that the simulation technique is better than other known
RQ3: Can the simulator be useful to estimate project
techniques to manage risk, but we claim that it should be used
risk (induced by errors in efforts estimation, and
as another tool, able to complement other approaches.
random developer issues assignment) with a Monte
For instance, in Rm4Am risk management tool for
Carlo approach?
ASD [44], our tool might be used for assessing the risk of
By varying parameters – such as variance of issue
Increments (i.e. user stories/features), in the risk analysis of
estimation errors, and developers’ availability – and
Product and Sprint backlogs. This should be done during the
performing Monte Carlo simulations, a project manager
Risk weekly meeting, a subcomponent specifically added in
can compute statistics on forecast project completion
Rm4Am to manage project risk.
times and average time to complete issues, and take
proper action to control this kind of risk. Consequently,
VIII. THREATS TO VALIDITY
also the answer to this question is positive. Clearly, The goal of this section is to discuss the threats to validity
much more causes of risk might be accounted for, as that we need to consider regarding the study of JIRA open
described below, and we are actively working to include source projects performed in order to evaluate the presented
them in the simulator. risk assessment method. The typical threats to validity taken
In this paper, for the sake of simplicity, we limited to into consideration in a software process development are:
use just issue effort perturbations, and random developers’ construct validity, internal validity and external validity [39].
assignment. However, other causes of risk can be easily • Internal Validity is the approximate truth about infer-
simulated, such as: ences regarding cause-effect or causal relationships.
• Sequentiality constraints between issues, so that de- Thus, it is only relevant in studies that try to establish
laying or stalling the development of an issue directly a causal relationship and it is not relevant in most obser-
influences other issues. vational or descriptive studies. In our case, the analysis
• Problems related to specific issues, whose development is focused to demonstrate the abilities to manage the risk
is delayed or stalled for external reasons. using a statistical analysis, and internal validity is not
• Change or deletion of existing issues. relevant to find this kind of relationship.
• Different skills of developers in the process activities, • Construct validity is focused on the relation between
and consequent preferential choices of issues to work the theory behind the experiment and the observations.
on. Even when it has been established that there is a casual
relationship between the execution of an experiment and
Other important causes of risk, which can also be sim- the observed outcome, the treatment might not corre-
ulated, and which embody random components and add spond to the cause one thinks to have controlled and
"uncertainty" to the process are shown in the followings. The altered, or the observed outcome might not correspond
distributions of the value of these factors must be studied and to the effect people think they are measuring. In our
defined in advance, typically by analyzing the past history of case, the main threat to construct validity is whether
the project, or of similar projects: our models are accurate enough to be realistic. In other
• Arrival of new issues, real or simulated to account for words, are issues, activities and developers, as modeled
forecasts of future work to be done; here the distribu- by us, enough reliable to get plausible results? Other
tion of issue effort estimates, issue priorities, and issue studies on empirical data seem to answer favourably
arrival time must be defined. Also change requests of ex- to this question [8], [15], but more research is clearly
isting issues not yet implemented might be considered. needed. Another issue related to construct validity are
• Issues which do not pass quality controls, and have to be the characteristics of feature effort variations, and of the
reworked, whose probability and extent of rework must random issues to developers allocation. Though these
in turn be specified. characteristics are common to many projects, the exact
• Changes in the availability of team members, due to var- distribution of effort variations, and the random assign-
ious causes (vacation, illness, resignation, more impor- ment performed might be improved. Moreover, there
tant commitments to carry out). This is an important risk are other possible risk factors, such as inaccurate re-
factor, able to substantially change the project schedule. quirements and team-related issues, such as resignation
Again, the probability and duration of developers’ un- of developers, introduction of new developers, intra-
availability must be defined in advance team dependencies, and so on. What we proposed is a
16 VOLUME 4, 2016
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
model able to represent some key aspects of software The proposed approach is clearly relevant for project man-
development, without trying to model every possible agers, who get a tool able to quantitatively evaluate the risks,
aspect. provided that the process and the project’s data are properly
• External validity is concerned with whether one can modeled. This is made possible by the relative simplicity of
generalize the results outside the scope of the study and Agile processes, that typically operate on atomic require-
hold them across different experimental settings, pro- ments -– the features, or issues -– through a sequence of
cedures, and participants. If a study possesses external activities.
validity, its results will generalize to a larger population Clearly, in real projects the approach should be com-
not considered in the experiment [8]. plemented with other risk management approaches, able to
In this specific study, we performed hundreds of simulation cover different kinds of risk that cannot be modeled and
runs. An important aspect to be taken into consideration is simulated, such as Organizational Environment Risk, User
that data analyzed are real but the analysis refers to just three Risk, and Team Risk [50].
simplified test cases. Although the scope of this paper would
a: Limitations
not have allowed to simulate more complex cases, this has
to be taken into account when considering external validity. In this work we limited our study considering only the effort
Also the fact that all features were available at the beginning variations and the random assignment of developers to issues.
of the simulated project, and that no more feature was added, Even though the obtained results are limited to the analysed
limits the generalization of our results. projects, our model could be customized and adapted for
other projects.
IX. CONCLUSION AND FUTURE WORK
b: Future Work
Risk management is essential to software development. Agile
In the future, we will improve our risk assessment method,
methodologies were introduced precisely to minimize the
evaluating it on many other case studies, and also exploring
risk inherent in traditional waterfall processes, with very high
the optimal parameter settings that can minimize the overall
risks related to requirement changes and final integration.
risk. An improvement we are considering is to scale the
However, only a few studies have been devoted to explicit
model from a single team to multiple teams, involved in
risk management of agile processes.
one project, or even in several projects. This would greatly
In this paper we presented a risk assessment procedure improve the utility of the tool for risk management in large
based on process modeling and simulation, and tested it on organizations.
three open source project developed using an agile approach,
with requirements collected as user stories and managed as REFERENCES
atomic unit of work, using JIRA popular issue management [1] Jira, published = https://www.atlassian.com/software/jira.
tool. The process can be organized into a sequence of basic [2] Zahra Shakeri Hossein Abad, Mohammad Noaeen, Didar Zowghi,
activities, and can thus be modeled using an event-based Behrouz H. Far, and Ken Barker. Two sides of the same coin: Soft-
ware developers’ perceptions of task switching and task interruption.
simulation. The developers are in turn modeled as agents, In Proceedings of the 22nd International Conference on Evaluation and
who decide the units of work they develop and complete. Assessment in Software Engineering 2018, EASE’18, page 175–180, New
York, NY, USA, 2018. ACM.
To be able to work with real data, we linked the simulator
[3] Temitope Elizabeth Abioye, Oluwasefunmi Tale Arogundade, Sanjay
with JIRA to collect the needed information (used process, Misra, Adio T. Akinwale, and Olusola John Adeniran. Toward ontology-
team composition, project size, number of issues, estimated based risk management framework for software projects: An empirical
study. Journal of Software: Evolution and Process, 32(12):e2269, 2020.
effort) and show the reliability of the tool.
[4] Aalaa Albadarneh, Israa Albadarneh, and Abdallah Qusef. Risk manage-
We have shown how it is possible to run the simulator in ment in agile software development: A comparative study. In 2015 IEEE
a Monte Carlo fashion, varying the identified risk factors and Jordan Conference on Applied Electrical Engineering and Computing
Technologies (AEECT), pages 1–6. IEEE, 2015.
statistically evaluating their effects on the critical outputs. In [5] Christopher J Alberts and Audrey J Dorofee. Risk management frame-
this way, a risk manager will be able to analyze the expected work. Technical report, Carnegie-Mellon Univ Pittsburgh Pa Software
parameters of the examined project, including expected opti- Engineering Inst, 2010.
[6] Thayse Alencar, Mariela I Cortés, Nécio Lima Veras, and Lui Magno. A
mal closing time of the project, and of the various issues and proactive approach to support risk management in software projects using
features, and evaluate the percentiles of their distributions, to multi-agent systems. In Proc. of the 20th International Conference on
assess the probability of adverse and favorable variations. Enterprise Information Systems (ICEIS 2018), pages 415–424, 2018.
[7] David Anderson, Giulio Concas, Maria Ilaria Lunesu, and Michele March-
We validated the simulator using three open source esi. Studying lean-kanban approach using software process simulation.
medium-sized projects, whose data are available on open In International Conference on agile software development, pages 12–26.
JIRA repositories, and considered four kind of scenarios. The Springer, 2011.
[8] David J Anderson, Giulio Concas, Maria Ilaria Lunesu, Michele Marchesi,
first two were used to test the reliability of the simulator; the and Hongyu Zhang. A comparative study of scrum and kanban approaches
third and fourth scenarios were used to make a risk analysis on a real case study using simulation. In International Conference on Agile
introducing the variation in the estimated effort to complete Software Development, pages 123–137. Springer, 2012.
[9] Muhammad Asif and Jamil Ahmed. A novel case base reasoning and
features, and when there are changes in their allocation to frequent pattern based decision support system for mitigating software risk
developers. factors. IEEE Access, 8:102278–102291, 2020.
VOLUME 4, 2016 17
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
[10] Tobias Baum, Fabian Kortum, Kurt Schneider, Arthur Brack, and Jens [32] Alok Mishra and Deepti Mishra. Software project management tools: a
Schauder. Comparing pre-commit reviews and post-commit reviews brief comparative view. ACM SIGSOFT Software Engineering Notes,
using process simulation. Journal of Software: Evolution and Process, 38(3):1–4, 2013.
29(11):e1865, 2017. [33] Jaana Nyfjord and Mira Kajko-Mattsson. Commonalities in risk manage-
[11] Barry W Boehm. Software risk management: principles and practices. ment and agile process models. In International Conference on Software
IEEE software, 8(1):32–41, 1991. Engineering Advances (ICSEA 2007), pages 18–18. IEEE, 2007.
[12] Saad Yasser Chadli, Ali Idri, Joaquín Nicolás Ros, José Luis Fernández- [34] C Ravindranath Pandian. Applied software risk management: A guide for
Alemán, Juan M Carrillo de Gea, and Ambrosio Toval. Software project software project managers. Auerbach Publications, 2006.
management tools in global software development: a systematic mapping [35] Dietmar Pfahl. Prosim/ra—software process simulation in support of
study. SpringerPlus, 5(1):1–38, 2016. risk assessment. In Value-based software engineering, pages 263–286.
[13] Robert N Charette. Why software fails [software failure]. Ieee Spectrum, Springer, 2006.
42(9):42–49, 2005. [36] Diana Leonor Tinjacá Rodríguez, Victor Hugo Medina García, and Ger-
[14] Clyde Chittister and Yacov Y Haimes. Assessment and management man Andrés Méndez Giraldo. Dynamic model to manage threats in
of software technical risk. IEEE Transactions on Systems, Man, and software development projects through artificial intelligence techniques.
Cybernetics, 24(2):187–202, 1994. In 2012 Workshop on Engineering Applications, pages 1–6. IEEE, 2012.
[15] Giulio Concas, Maria Ilaria Lunesu, Michele Marchesi, and Hongyu [37] Bibhash Roy, Ranjan Dasgupta, and Nabendu Chaki. A study on software
Zhang. Simulation of software maintenance process, with and without risk management strategies and mapping with sdlc. In Advanced Comput-
a work-in-process limit. Journal of Software: Evolution and Process, ing and Systems for Security, pages 121–138. Springer, 2016.
25(12):1225–1248, 2013. [38] Hasliza Md Sarkan, Tengku Puteri Suhilah Ahmad, and Azuraini Abu
[16] Marcio de Oliveira Barros, Claudia Maria Lima Werner, and Guil- Bakar. Using jira and redmine in requirement development for agile
herme Horta Travassos. Supporting risks in software project management. methodology. In 2011 Malaysian Conference in Software Engineering,
Journal of Systems and Software, 70(1-2):21–35, 2004. pages 408–413. IEEE, 2011.
[17] Samuel de Souza Lopes, Rogéria Cristiane Gratao de Souza, Allan [39] William R. Shadish, Thomas D. Cook, and Donald T. Campbell. Experi-
de Godoi Contessoto, André Luiz de Oliveira, and Rosana Teresinha Vac- mental and Quasi-Experimental Designs for Generalized Causal Inference.
care Braga. A risk management framework for scrum projects. In Cengage Learning, Boston, New York, 2 edition, Feb 2001.
2021 Proceedings of the 23rd International Conference on Enterprise [40] Basit Shahzad and Abdullah S Al-Mudimigh. Risk identification, mit-
Information Systems (ICEIS 2021) ISBN: 978-989-758-509-8, pages 30– igation and avoidance model for handling software risk. In 2010 2nd
40, 2021. International Conference on Computational Intelligence, Communication
[18] F Michael Dedolph. The neglected management activity: Software risk Systems and Networks, pages 191–196. IEEE, 2010.
management. Bell Labs Technical Journal, 8(3):91–95, 2003. [41] Lubna Siddique and Bassam A Hussein. Practical insight about risk
[19] Norman Fenton, Martin Neil, William Marsh, Peter Hearty, Lukasz management process in agile software projects in norway. In 2014 IEEE
Radlinski, and Paul Krause. On the effectiveness of early life cycle defect International Technology Management Conference, pages 1–4. IEEE,
prediction with bayesian nets. Empirical Software Engineering, 13:499– 2014.
537, 2008. [42] Vikram Singh, Varun Malik, and Ruchi Mittal. Risk analysis in soft-
[20] J.A. García-García, J.G. Enríquez, M. Ruiz, C. Arévalo, and A. Jiménez- ware cost estimation: A simulation-based approach. Turkish Journal of
Ramírez. Software process simulation modeling: Systematic literature Computer and Mathematics Education (TURCOMAT), 12(6):2176–2183,
review. Computer Standards Interfaces, 70:103425, 2020. 2021.
[21] Kamran Ghane. Quantitative planning and risk management of agile soft- [43] Breno Gontijo Tavares, Carlos Eduardo Sanches da Silva, and Adler Diniz
ware development. In 2017 IEEE Technology & Engineering Management de Souza. Risk management analysis in scrum software projects. Interna-
Conference (TEMSCON), pages 109–112. IEEE, 2017. tional Transactions in Operational Research, 26(5):1884–1905, 2019.
[22] Wen-Ming Han. Discriminating risky software project using neural [44] Breno Gontijo Tavares, Mark Keil, Carlos Eduardo Sanches da Silva,
networks. Computer Standards Interfaces, 40:15–22, 2015. and Adler Diniz de Souza. A risk management tool for agile software
[23] Daniela Hilčíková, Monika Dohnanská, Daniel Lajčin, and Gabriela development. Journal of Computer Information Systems, pages 1–10,
Gabrhelová. Agile project management as one of the critical success 2020.
factors in project risk management in machinery industry in slovakia. Int. [45] Christoph A Thieme, Ali Mosleh, Ingrid B Utne, and Jeevith Hegde. Incor-
J. Econ. Financ. Issues, 1(1):37–54, 2020. porating software failure in risk analysis—-part 2: Risk modeling process
[24] Project Management Institute. A Guide to the Project Management Body and case study. Reliability Engineering & System Safety, 198:106804,
of Knowledge (PMBOK Guide)., volume 2. Project Management Inst, 2020.
2017. [46] Alexey Tregubov, Barry Boehm, Natalia Rodchenko, and Jo Ann Lane.
[25] Harry Raymond Joseph. Software development risk management: Using Impact of task switching and work interruptions on software development
machine learning for generating risk prompts. In 2015 IEEE/ACM 37th processes. In Proceedings of the 2017 International Conference on
IEEE International Conference on Software Engineering, volume 2, pages Software and System Process, ICSSP 2017, page 134–138, New York, NY,
833–834, 2015. USA, 2017. ACM.
[26] Marc I Kellner, Raymond J Madachy, and David M Raffo. Software [47] Richard Turner, Dan Ingold, Jo Ann Lane, Ray Madachy, and David
process simulation modeling: why? what? how? Journal of Systems and Anderson. Effectiveness of kanban approaches in systems engineering
Software, 46(2):91–105, 1999. within rapid response environments. Procedia Computer Science, 8:309–
[27] Dapeng Liu, Qing Wang, and Junchao Xiao. The role of software process 314, 2012.
simulation modeling in software risk management: A systematic review. [48] Richard Turner, Raymond Madachy, Dan Ingold, and Jo Ann Lane.
In Empirical Software Engineering and Measurement, 2009. ESEM 2009. Modeling kanban processes in systems engineering. In Proceedings of the
3rd International Symposium on, pages 302–311. IEEE, 2009. International Conference on Software and System Process, pages 23–27.
[28] Cristina Lopez and Jose L Salmeron. Dynamic risks modelling in erp IEEE Press, 2012.
maintenance projects with fcm. Information Sciences, 256:25–45, 2014. [49] Bhawna Verma, Mamta Dhanda, B Verma, and M Dhanda. A review on
[29] William W Lowrance. Of Acceptable Risk: Science and the Determination risk management in software projects. International Journal, 2:499–503,
of Safety. Kaufmann Inc., Los Altos, CA, 1976. 2016.
[30] Marco Melis, Ivana Turnu, Alessandra Cau, and Giulio Concas. Evalu- [50] Linda Wallace, Mark Keil, and Arun Rai. How software project risk affects
ating the impact of test-first programming and pair programming through project performance: An investigation of the dimensions of risk and an
software process simulation. Software Process: Improvement and Practice, exploratory model. Decision Sciences, 35(2):289–321, 2004.
11(4):345–360, 2006. [51] Zhe Wang. Modelling and simulation of scrum team strategies: A multi-
[31] Wang Min, Zhang Jun, and Zhang Wei. The application of fuzzy com- agent approach. In Proceedings of the Computational Methods in Systems
prehensive evaluation method in the software project risk assessment. and Software, pages 32–63. Springer, 2020.
In Proceedings of the 2017 International Conference on Management [52] Li Xiaosong, Liu Shushi, Cai Wenjun, and Feng Songjiang. The applica-
Engineering, Software Engineering and Service Sciences, ICMSS ’17, tion of risk matrix to software project risk management. In 2009 Inter-
page 76–79, New York, NY, USA, 2017. Association for Computing national Forum on Information Technology and Applications, volume 2,
Machinery. pages 480–483, 2009.
18 VOLUME 4, 2016
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3115941, IEEE Access
Maria Ilaria Lunesu et al.: Assessing the Risk of Software Development in Agile Methodologies Using Simulation
[53] He Zhang, Barbara Kitchenham, and Dietmar Pfahl. Reflections on 10 MICHELE MARCHESI received the degree in
years of software process simulation modeling: a systematic review. In Electronic engineering from the University of
International Conference on Software Process, pages 345–356. Springer, Genova, in 1975. He has been a Full Professor
2008. with the Faculty of Engineering, University of
[54] Boxuan Zhao, Jian Cao, Sha Jiang, and Qing Qi. An agent based Cagliari, since 1994. Since 2016, he has been a
simulation system for open source software development. In 2020 IEEE Full Professor with the Department of Mathemat-
World Congress on Services (SERVICES), pages 164–170. IEEE, 2020. ics and Computer Science, University of Cagliari,
where he teaches software engineering course. He
has authored over 200 international publications,
including over 70 in the magazine. He has been
one of the first in Italy to deal with OOP, since 1986. He was a Founding
Member of TABOO, the Italian association on object-oriented techniques.
He has also worked on object analysis and design, UML language and
metrics for object-oriented systems since the introduction of these research
MARIA ILARIA LUNESU received the degree themes. In 1998, he was the first in Italy to deal with Extreme Programming
in Electronic Engineering from the University of (XP) and agile methodologies for software production. He organized the
Cagliari and in 2013 she holds from the same first and most important world conferences on Extreme Programming and
university the PhD in Electronic and Computer Agile Processes in Software Engineering, Sardinia, from 2000 to 2002. Since
Engineering and the title of Doctor Europaeus. She 2014, being among the first in Italy, he has extended his research interest
is currently research fellow at the Department of to blockchain technologies, obtaining significant results in the scientific
Mathematics and Computer Science at University community.
of Cagliari. Her research interests concern the
study and applications of Agile and Lean method-
ologies evaluating their potential for risk manage-
ment in software development processes, the study of startup dynamics and
ecosystem and the the study of software engineering practices for blockchain
development and distributed applications and ICOs main success factors
analysis.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/