International Conference on Computing, Communication and Automation (ICCCA2015)

A Critical Survey on Test Management in IT Projects

Sheena Kukreja 1 Abhishek Singhal2 AbhayBansal3
Department of CSE, ASET Department of CSE, ASET Department of CSE, ASET
Amity University Uttar Pradesh Amity University Uttar Pradesh Amity University Uttar Pradesh
Noida, UP, India Noida, UP, India Noida, UP, India
[email protected]
[email protected] [email protected]

Abstract— Software testing is a vital phase of software planning of test processes is highly important for the success of
development life cycle to ensure software quality. Testing and project. Early planning can be achieved by early understanding
validating a software forms necessary part of software quality. of goal of project. If early planning is not done with time, it
Different software applications require different configurations may increase difficulty in management of test processes.
and run in different hardware and software environment. So, a
Generally, test management is a broad term which comprises
rigid structure for testing cannot be set prior to the evolution of
project. It should be developed as the project evolves. Testing of managing test activities along with requirement
involves number of activities and to manage entire testing process management, defect management and configuration
effectively, need of test management arises. Test management management [2] [3].
involves control of all the testing activities in a single dedicated
environment. This paper presents a survey on test management Requirement Management: Requirement analysis and
process, its need, challenges and different tools that help in test management are important tasks throughout testing life cycle.
management. It is necessary to manage varied requirement types and provide
requirement coverage to find business risks [4]. This generally
Keywords— Testing; Software Quality; Test Management
involves storing the requirement statements, prioritizing them,
providing traceability to test cases and checking the missing
I. INTRODUCTION requirements.
A. Test Management: Managing Test Activities
Defect Management: Defects are resulted if any deviation of
actual from expected result occurs. It becomes important to
In this competitive world, software has become pervasive, track, prioritize and assign defects to individual in order to get
so the major concern for businesses is high quality software them fixed and reduce severe impact on the software. Defect
with optimum functionality. Development cycle of software management is concerned in storing and managing defect
involves requirement analysis, designing, coding and reports and the above mentioned tasks.
implementation, testing and the last, maintenance. Testing
phase of development cycle forms a crucial position, done Configuration Management: Changes cannot be avoided
mainly to find bugs in the entire software. This is done mainly during software development. But if these changes are not
by verifying the software against customer requirements. communicated in orderly manner, it may lead to
Testing is independent from improving or fixing code activities misunderstanding and wastage of time. So it becomes highly
and involves activities like test planning, design, execution, important to manage these changes in software development
evaluation and reporting, and the last, completion. Manual and hence need of configuration management arises. It helps to
testing is the most basic and traditional manner to test the identify, monitor, manage and control modifications in
software and hence, faces many difficulties. Firstly, it is time software. Versioning allows geographically dispersed teams to
consuming. Secondly, software projects become complex and collaborate and provide history of changes throughout product
larger in size with time. So the implementation of manual development [4].
testing becomes complicated. Hence automated testing is a
hopeful solution to this. For implementing automation, the The rest of the paper is structured as follows: Section II
need of test management becomes highly important. explains the test management process, Section III describes the
related work done in this area, Section IV focuses on different
Software testing is a time consuming development task and techniques, followed by section V, VI, VII exploring test
needs approximately 30-50% of development cost. Software management through different tools and the last section
companies are often faced with time and cost limitations which includes the conclusion.
hinder the effective completion of testing process [1]. So it
becomes important and difficult to perform testing with low
cost and resource and with high quality. Testing can mainly be II. TEST MANAGEMENT PROCESS:
categorized into static and dynamic. Test Planning and designing: It is a stage where in a test
With the increase in software size and testing techniques plan is developed providing answers to the questions of what,
involvement of management into testing is vital. Early where, when and why to test. A detailed test plan is prepared,

ISBN:978-1-4799-8890-7/15/$31.00 ©2015 IEEE 791

 International Conference on Computing, Communication and Automation (ICCCA2015)
providing answer of what to test by forming test cases for a 2.1 Need of test Management:
project, where to test by knowing required configuration of
project, when to test by tracking cycles and why to test by test a) To supply best quality software in competitive market
motivator. Designing is a phase in which entire testing task has within shortest period of time.
to be designed as to how the testing will be conducted in b) To provide coordination and communication between
different units [5] [6]. test team and development team to produce quality
products within tighter deadlines.
Test Authoring: It is stage of forming abstract test cases c) To provide consistency and smoothness in project
into detailed tests by constructing the steps to complete a given activities
test. These steps help the testers to test corresponding d) To provide structured approach to the test effort in
functionality and this forms a (manual or automated) script. order to keep them organized with well defined
processes.
Test Execution and tracking: In this stage, verification and e) Complexity of software increases when team is
validation of software is done to make sure that the developed geographically dispersed, so need of test management
product satisfies the customer expectations. The test cases arises to make the project successful.
produced in earlier step are executed here by providing test
input. The following guidelines should be considered:
• Test conditions and test scenarios must be reviewed. III. RELATED WORK
• Test data must be reviewed for testing each test In order to verify whether the system satisfies set of
condition. requirement, it is evaluated by manual or automated means.
• Expected result of test case must be reviewed. Sultan Aljahdali et al. [7] proposed the method, approach and
• Test input must be entered to execute test case. [7] procedure to be used to verify and validate test management
Upon execution of test, its corresponding result must be and control. Test management majorly includes 3 activities:
logged and compared with expected result. If any deviation test planning, test execution and defect management. All these
occurs, “failed test” status should be logged against the activities need to be monitored timely so as to ensure effective
corresponding test. delivery of quality product in specified time. Sultan Aljahdali
et al [7] introduced upper control limit (UCL) and lower
Test Reporting: This is done for comprehensive analysis, to control limit (LCL), and focused on the concept that test
determine the quality status of software product for its execution should be within the UCL and LCL. If any
improvement. It also helps in evaluating and reviewing the deviation from UCL and LCL occurs, need of revised plan is
status of project testing. Test reports analyze and communicate raised so as to ensure that the delivered products are not
test efforts which provide large amount of information to entire jeopardy. This methodology helps to effectively prepare test
test team. Figure 1 [8] illustrates the test management process. plan and monitor the status of execution process. To ensure
stability of product two time zones “soft freeze and hard
freeze” are available so as to ensure the minor defects are not
fixed in later stages of testing cycle, unless the criticality and
severity of these defects are high [7].
Test Planning
Yikun Lu et al. [4] proposed that integrated RTDMS
(requirement, test and defect management system) must be
Design designed to assist testing service provider in managing the
testing lifecycle from requirement management through test
execution and defect management to project closure. RTDMS
Test Authoring instance which stores automated test scripts and artifacts use
repositories to assist projects in managing newly generated
artifacts and reuse previous project’s artifacts. They conducted
a research with respect to Chinese testing service provider and
Test Execution &
used a case study to provide detailed knowledge of provision
Tracking
of testing services in respect to Esourcing. RTDMS offered
three services:
Requirement Management: By analyzing client
Defect tracking requirements and request for proposal, test requirements are
prioritized on the basis of business priority and risks. After
Test reporting prioritization RTDMS instance gather specifications from
earlier projects and maps them to respective test artifacts and
test plan. A detailed test plan is then made and further test
Fig. 1: Test Management Process [8] cases will be designed. RTDMS gather test cases from various

792
 International Conference on Computing, Communication and Automation (ICCCA2015)
projects and reuses them to reduce development time and cost. software testing. On the basis of early testing, demand can
Interdependencies between requirements and test case are then participate in discussion. Early involvement of testing and
managed and mapped bi-directionally. The test project analysis needs can help in better understanding of demands
information is then stored to the project database [4]. but it leads to production of quality specification document.
The changes in demand are inevitable and are the causes of
Test Management: It includes monitoring test process with risk in software and hence have to be managed. He also
respect to the test plan and reporting process so that project focused on introduction of test management tool.
managers can take decisions regarding the release of
applications. Execution result and faults are then reported. Randy H. Hefner [10] discussed on the issues faced while
Test management also includes version control and baseline testing large complex systems, presented a collaborative test
management in order to avoid confusion among different management approach producing number of benefits and
versions of software and test artifacts. Test management also outlines on web IT solutions that has been implemented to
includes generation of impact analysis report if client different Navy and NASA test programs. He defined a test
requirement changes and adjusting the plan accordingly [4]. management approach which mainly focuses on formal test
activities like system integration, system certification, and
Defect Management: It involves identifying defects and validation testing. The objective of this approach is to provide
tracking them till their resolution and verification. data collaboration between geographically distributed test
organization and test personnel so that co-ordination among
The major challenge that occurs in software test test activities is increased, to provide methods to confirm the
management is to efficiently handle the entire software testing validation of system by execution of test cases, to improve
task in a single dedicated environment. Ahmed Ibrahim et al. decision making by managers on the basis of test metrics.
[9] applied SpiraTeam to manage the tests on a real time
system called OBA (On-Board Automobile System). In this, Azeem Lodhi et al. [11] laid focus on importance of test
the capabilities of SpiraTeam to manage test activities have management and presented a test management framework
been focused and its importance in test management has been which should be followed by test managers and test team to
showed. This tool lays stress on supporting the management manage large IT projects. This framework helps to conduct
and control of test data produced in product development life testing from strategic to operational level. He divided test
cycle. Analysis of different test management tool was done by management framework into different parts i.e. ‘strategic part’
different test managers on the same project under varying which discusses test management at strategic level. It involves
environments. A case study of On Board Automobile System the preparation of strategic plan which describe the goal of
(OBA) was done. The specification and requirement helped in testing and prepare a plan to achieve this goal. At strategic
creating a test plan which was further used in conducting ad- level, goal of testing has to be highlighted w.r.t business
hoc test process. Requirement management was done by perspective (like cost and risk reduction). This test framework
exporting requirement into SpiraTeam using excel importer. considers test strategy as a core element since it helps in
Then these requirements were mapped with test cases to defining, controlling and organizing the processes. Second
provide test coverage in order to ensure that no requirement is part of this framework is ‘operative part’ which follows a
being missed out. Requirement Coverage displayed varying lifecycle of analysis-> plan-> design->execute-> feedback. In
status ranging from passed through fail to not run. During test analysis, business requirements are collected and translated
run, any test case result showing deviation from expected into testing requirement for verification and validation. In
result was considered as a new defect. Each defect was plan, detailed test plan as per IEEE 829 testing standard is
associated with workflow which shows different phases which prepared. In design, test catalogues and design patterns are
the defect will undergo. Status of test execution forms a vital consulted to prepare test cases. In execute, test environment is
feature of SpiraTeam Tool. Test execution result status varied set prior to execution and results are also logged which will be
from passed through failed to caution [9]. further used in ‘evaluate’ phase to determine defect, defect
detection rate. Third part of this framework i.e. ‘supportive
Lei Gao [8] introduced the concept of software testing and part’ provides help for test management processes by
its process in this paper. It focuses on how the requirement, providing templates, patterns, tools, heuristics (to define
configuration, defect and content are managed. Software budget, time), benchmarks (to evaluate testing performance).
testing process comprises of different processes ranging from The last and final part is ‘improvement’ which focuses on
planning, designing and implementation, analysis of result, improvement of test management phases by considering the
and summary. Process management is used to reduce the cost response from stakeholders. This framework was applied in
of test, test procedure and test methods. The objective of large IT projects.
process management is to control test process and manage
software development organization so as to enhance the IV. TECHNIQUES FOR TEST MANAGEMENT
capability of development lifecycle. For ordered software
testing, testing and engineering process should be efficiently As we have discussed, test management is a way to manage
managed. Demand can greatly determine the success rate of different test activities. There are different ways to manage

793
 International Conference on Computing, Communication and Automation (ICCCA2015)
entire test environment such as manual and automated test defined for parent package can be associated to the appropriate
management. Comparison between both can be illustrated by test case.
table 1. Test Session Planning and reporting: This feature enables to
define test sessions, enlist requirements required for a session,
TABLE I: Comparison between Manual and Automated Test Management logs actual output produced upon execution of test procedures.
Logged actual output in database helps in finding and
Manual Test Automated Test analyzing the validation testing status of the requirement
Management Management baseline. Advantages and disadvantages of TM2K can be
illustrated by table 2.
Traceability between Effective traceability
requirement, test and between requirements, tests TABLE II: Tabular Representation of Advantages and Disadvantages of TM2K
defects is not visible. and defects can be seen.
Advantages Disadvantages
Performance status Performance status can be
visibility is difficult. easily displayed through
tool.
Centralized Database:
Difficult to manage project Teams working at any Report Support: Generated
Storage of test data into
activities among teams geographical location can Report support for PC based
single shared database helps
which are geographically co-ordinate easily. clients is provided for
in sharing among
dispersed. Microsoft word format only.
geographically distributed
No shared database for Single shared, centralized test organizations and test
storing results. database exists. personnel. It helps to
maintain data consistency,
Graphical Representation Graphical Representation reduce information loss
of executed test cases, can be seen, thus helping to among test personnel
requirement and test monitor progress. whenever changes occur in
coverage cannot be seen. test programs.

Guaranteed solution: Since

V. EXPLORING TEST MANAGEMENT THROUGH TM2K
this tool has been applied to
various NASA test programs
TM2K (TestManager 2000) tool [10] was conjured by and activities. So it is
Randy H. Hefner [10] on the basis of knowledge gained by worldwide accepted and used
working on Department of Defense (DoD) USA and NASA since 1995.
programs. The urge to reduce struggle of coordination,
unification, management and minimization of redundant Provides integrated test
activities of worldwide distributed test organizations led to the environment: Rather than
development of an integrated tool ‘TM2K’. The basis which using various commercial off
helped in formation of tool are as follows: the shell products which
increases integration
• Requirement test baseline formulation. complexities, test managers
• Requirement distribution to test data. are seeking their attention
• Storing of test plan, test case, its result and procedure towards test environment tool
into a shared, centralized database. which helps them to minimize
difficulties in the projects
Functions supported by tool are as follows [10]:

Master Test plan Development: This feature provides

creation and modification of test package, allows allocation of VI. EXPLORING TEST MANAGEMENT THROUGH QUALITY
CENTER
requirement to test package on the basis of query selection
capability. It provides test methods for satisfying requirements
for test package association and defines data which further QC is a commercially available test management tool with
helps to create a master test plan. a single, integrated environment providing number of features
together like release, test planning, test cycle, requirement,
Detailed Test plan Development: This enables users to defect, configuration management, integrated reporting
create and modify test case and procedure. Every defined test through its dashboard feature, test scheduling and test
case is linked to its parent package. Hence, the requirements

794
 International Conference on Computing, Communication and Automation (ICCCA2015)
execution [12]. Figure 2 shows the status of mapping of a) Expensive: QC is a not an open-source, freeware tool
requirements/tests against release. to use in organizations. To use it, purchasing its license
is quite expensive. Hence, it should be utilized
effectively to bring out the best out of it.
b) Provides only VB Scripting support: The workflow
coding language of QC is VB script. No java support
feature is available.
c) Seems to crash: It is used in software companies by
thousands of multiple technology users and hence
crashes sometimes.

VII. EXPLORING TEST MANAGEMENT THROUGH SPIRATEAM

TOOL

SpiraTeam is an automated test management tool providing

requirement, release, defect and build management in a single
consolidated and integrated environment [13]. By taking the
control of entire development and testing lifecycle, it provides
excellent communication and coordination in entire project
management. Ahmed Ibrahim Safana et al. [9] applied this tool
on real time system (OBA) and illustrated requirement
mapping, test coverage and many more. Features of SpiraTeam
are as follows:
Fig. 2: Status of mapping of requirements/tests against release [12]

If the delivery of applications of any organization regularly a) Requirement Management: This enables to create,
causes delay in time and goes beyond the budget, faces delete, assign priority, assign users, categorize requirements
application breakdowns, uses high percent of development cost into varied types so to produce different workflows, map
in testing and debugging, unable to meet customer expectations requirements to test cases, link requirements to project
then it becomes important to use QC to resolve these artifacts, view test’s pass/fail status.
mentioned problems.
b) Incident Management: This enables to create, regulate,
Advantages: assign priority and users to each bug, assign status to each
bug, provide support with different tools like bugzilla and
a) World wide acceptance: QC is a worldwide accepted JIRA, provide e-mail support for bug reporting, allows
tool and is used in leading organizations to manage their creation of customized bug report fields.
projects, deliver software in time and within expected budgets.
Operating from single test management platform standardizes c) Release Management: This enables to regulate list of
different management activities. versions of tested software, provide traceability between
b) Deliver high quality products: It provides improved requirements and releases, adjust releases in hierarchical
project quality with greater efficiency and cost effective order, provide glimpse of actual vs expected progress in a
project implementation. High quality projects are produced at release.
the pace which businesses need. It ensures working of
applications as expected under different workload conditions. Advantages of SpiraTeam:
c) Drives Collaboration: It provides collaboration among
development teams, business analysts, quality analysts and a) Follows latest approach: SpiraTeam plans entire
helps them to coordinate and communicate throughout the project by using agile method instead of traditional
project lifecycle. waterfall or any other approach.
d) Implements latest Approach: It enables project
managers to apply agile approach to manage projects by b) Graphical Representation support: It helps user to
improving visibility of performance status, project status, analyze test coverage, requirement coverage, test
manage risks and improve collaboration among teams. progress rate in form of graphical charts. Figure 3 [9]
shows graphical representation of requirement
coverage of OBA.
Disadvantages:

795
 International Conference on Computing, Communication and Automation (ICCCA2015)
REFERENCES

[1] Li, Q., Y. Yang, M.S. Li, Q. Wang, B.W. Boehm, and C.Y. Hu,
“Improving Software Testing Process: Feature Prioritization to
Make Winners of Success-critical Stakeholders”, Journal of
Software Maintenance and Evolution: Research and Practice,
2010.
[2] L. Lui and D. J. Robson, “A Support Environment for the
Managing of Software Testing”, in the proceedings of Second
Symposium on Assessment of Quality Software Development
tools, IEEE, 1992, New Orleans, LA, pp 11-20, doi.
10.1109/AQSDT.1992.205830.
[3] N. Kicillof, W. Garieskamp, and V. Braberman, “Achieving
Both Model and Code Coverage with Automated Gray-box
Testing,” AMOST’07 ACM London, UK, July 2007, pp. 1–5.
Fig. 3. Requirement Coverage [9]
[4] Yikun Lu and Timo Käkölä, “An Information Systems Design
Product Theory for Integrated Requirements, Test and Defect
c) Real time support: This tool can be applied to real time Management Systems”, in the proceedings of 45th Hawaii
applications in the same manner as it was applied to International Conference on System Sciences (HICSS), IEEE,
OBA by Ahmed Ibrahim Safana et al. [9]. 2012, Maui, HI, pp 5516-5525, doi. 10.1109/HICSS.2012.102.
d) Diversity in format report: It gives customer an option [5] William e. Perry. "Effective Methods for Software Testing”, 2nd
to decide the manner in which their experimental result Edition.
is to be displayed. [6] Rex Black, "Managing the Testing Process”, 'Wiley, 2nd
Edition, 2002.
Disadvantages: [7] Sultan Aljahdali, Syed Naimatullah Hussain, Nisar Hundewale,
and Azeemsha Thacham Poyil, “Test Management and
a) Decrease in Efficiency: Efficiency of this tool Control”, in the proceedings of IEEE 3rd International
decreases as it is used on other browsers apart from Conference on Software Engineering and Service Science
Mozilla Firefox browser. (ICSESS), IEEE, 2012, Beijing, pp. 429-432, doi.
b) Less support to different operating systems: The 10.1109/ICSESSS.2012.6269496.
number of user access support decreases with [8] Lei Gao, “Research on Implementation of Software Test
different operating systems like, on windows vista OS Management”, in the proceedings of 3rd International
Conference on Computer Research and Development (ICCRD),
number of users supported are few in number(5).
IEEE, 2011, Shanghai, pp 234-237, doi.
10.1109/ICCRD.2011.5764288.
VIII. CONCLUSION [9] Ahmed Ibrahim Safana and Suhaimi Ibrahim, “Implementing
Software Test Management using SpiraTeam Tool”, in the
proceedings of 5th International Conference on Software
The increasing size and complexity of software and Engineering Advances (ICSEA), IEEE, 2010, Nice, pp 447-452,
necessary involvement of testing activity has resulted in crucial doi. 10.1109/ICSEA.2010.76.
involvement of test management into testing. Test management [10] Randy H. Hefner, “Collaborative Test Management”, in the
can be performed manually as well as automatically. But the proceedings of 26th Annual NASA Goddard on Software
necessity to deliver quality projects within tighter deadlines has Engineering Workshop, IEEE, 2001, Greenbelt, MD , pp 156-
resulted in adoption of automated test management. Different 158, doi. 10.1109/SEW.2001.992669.
tools are available among which three tools have been explored [11] Azeem Lodhi and Stefan Wind, “Test Management Framework
in this survey. for Managing IT Projects in Industry”, in the proceedings of
IEEE 10th International Conference on e-Business Engineering
(ICEBE), IEEE, 2013, Coventry, pp 509-514, doi.
IX. FUTURE WORK 10.1109/ICEBE.2013.80.
[12] HP Quality Center http://www8.hp.com/in/en/software-
The field of managing software testing tasks is growing
solutions/quality-center-quality-management retrieved on 10th
very rapidly and the focus in software testing has increased a February, 2015.
lot in order to deliver high quality end products. The current
[13] SpiraTest Management Tool Inflacter Corporation [online]
challenge is not to become an expert in all test management Available at http.www.inflactar.com retrieved on 10th January,
tools but to focus on single management tool and show how it 2015.
can lead to time and effort minimization. [14] http://www.istqb.org/downloads/viewdownload/26/101.html
retrieved on 10th February, 2015.

796