See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/344122939

A Comparative Study of Different Software Testing Techniques: A Review

Article · January 2018

CITATIONS READS

2 1,380

2 authors:

Seema Rani Deepali Gupta

Guru Jambheshwar University of Science & Technology Chitkara University
5 PUBLICATIONS 4 CITATIONS 59 PUBLICATIONS 977 CITATIONS

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Software Testing View project

All content following this page was uploaded by Seema Rani on 05 September 2020.

The user has requested enhancement of the downloaded file.

 Journal of Advances in Shell Programming
ISSN: 2395-6690 (Online)
Volume 5, Issue 1
www.stmjournals.com

A Comparative Study of Different Software Testing

Techniques: A Review
Seema Rani1,*, Deepali Gupta2
1
Department of Computer Science Engineering, Ch. Devi Lal State Institute of Engineering and
Technology, Panniwala Mota, Sirsa, Haryana, India
2
Department of Computer Science Engineering, Maharishi Markandeshwar University, Sadopur,
Ambala, Haryana, India

Abstract
Software testing is the process of identifying the correctness and quality of software program.
The purpose is to check whether the software satisfies the specific requirements, needs and
expectations of the customer. Software testing is the process to expose design, requirement and
coding errors in the program. Testing is used to identify the correctness, completeness, security
and quality of software products against the requirements of the user. It exhibits all faults,
errors and flaws in the developed software. Software testing is aimed at evaluating the
capability or usability of a program. It is a significant method of accessing the quality of
software. Though a lot of advancements have been done in formal methods and verification
techniques, still we need software to be fully tested before it could be handled to the customer
side. There are a number of testing techniques available to complete a task. Software testing is
an important area of research and a lot of development has been made in this field. In this
paper, all the software testing techniques have been described and compared. Some latest
researches have been described in brief. The concept of software testing is gaining more and
more importance in the future. There are many approaches to software testing, but effective
testing is a process of investigation, not only a matter of creating and following route procedure.
While it is not possible to detect all the faults in the software; this problem in testing thus arises
a question: what would be the strategy we should adopt for testing? In our paper, we have
described and compared the three most prevalent and commonly used software testing
techniques for detecting errors and faults. They are: white box testing, black box testing and
grey box testing.

Keywords: Black box, grey box, white box, prevention, software testing, software testing
principles, research

*Author for Correspondence E-mail: [email protected]

INTRODUCTION Software testing is an activity that is performed

Software Testing is the process of identifying for evaluating software quality and also for
the correctness and quality of software improving it. Thus, the goal of testing is
program. The purpose is to check whether the systematic and stepwise detection of different
software satisfies the specific requirements, classes of errors within a minimum amount of
needs and expectations of the customer. In other time and also with a much less amount of effort.
words, creating a program consists of the five Software testing is also an important
phases [1]: component of software quality assurance
(1) Defining a problem, (2) Designing a (SQA), and a number of software organizations
program, (3) Building a program, (4) Analyzing are spending up to 40% of their resources on
performances of a program, and (5) Final testing.
arranging of a product.
Software testing is not a “silver bullet” that can
According to this classification, software guarantee the production of high quality
testing is a component of the third phase, and software systems. While a “correct” correctness
means checking if a program for specified proof demonstrates that a software system
inputs gives correct and expected results. (which exactly meets its specification) will

JoASP (2018) 1-8 © STM Journals 2018. All Rights Reserved Page 1
Comparative Study of Software Testing Techniques Rani and Gupta

always operate in a given manner, software Demonstration

testing that is not fully exhaustive can only It shows how the system can be used with
suggest the presence of flaws and cannot prove different acceptable risks. It also demonstrates
their absence. Moreover, it is impossible to functions with special conditions and shows
completely test an application because: (1) how products are ready for integration or use.
there are too many possible input paths; (2) the
domain of program inputs is too large; and (3) Improving Quality
design and specification issues are difficult By applying effective testing on software,
tests. The first and second points present errors can be reduced and thus quality of
obvious complications and the final point software is improved.
highlights the difficulty of determining if the
specification of a problem solution and the ROLE OF SOFTWARE TESTING IN
design of its implementation are also correct
SOFTWARE DEFECT PREDICTION
[2].
PROCESS
The main objective of software testing is to The common software defect prediction
affirm the quality of software system by process follows the machine learning approach
systematically testing the software in carefully [5–7]. The first step in defect prediction process
controlled circumstances; another objective is is to find out the instances from software, an
to identify the completeness and correctness of instance can be code, function, class or method
the software, and finally it uncovers etc. These instances can be generated from the
undiscovered errors [3, 4]. different issue tracking systems, version control
systems or e-mail archives. An instance has
This paper is related to many research papers different metrics which is found out from the
published in different publications like IEEE software. These instances can be categorized in
transactions, international journals, and bugs B or number of bugs and clean C or
international conferences. This paper is number of clean. After recognizing instances
prepared as: Next part if the paper is describing with the category and metrics, the first step of
the objectives of software testing. The part after machine learning pre-processing techniques is
that provides information about the role of used on instances to create new same type of
software testing in software defect prediction instance. The pre-processing is applied to
process, software testing principles, extract the features, scaling the data and
information about software testing levels, removing the noise [8–10]. It is not compulsory
description about the software testing to apply on all types of defect prediction models
techniques and comparison between three [6, 11]. After pre-processing, the instances
forms of software testing techniques. The last generate new instances to train the defect
part provides conclusion of the paper. prediction model. The prediction model
provides the result in terms of buggy instances
OBJECTIVES and clean instances. The number of bugs in an
There are four main objectives of testing: instance is known as regression. It produces
Detection only two results for the instances bugs or
Different kinds of defects, errors and cleans; so, it is also known as binary
deficiencies are detected. Quality of all classification.
components, system capabilities and various
limitations, and the work products of the overall
SOFTWARE TESTING PRINCIPLES
system are calculated.
Different software testing principles are as
Prevention follows:
It is the process to reduce the number of errors, Test a Program so as to Make it Fail
to explain system specifications and system Software testing is the process of executing a
performance. Different ways to avoid risks and program with the aim of finding bugs and
to deal with problems in the future are errors. Testing becomes more effective when
identified. failures are exposed.

JoASP (2018) 1-8 © STM Journals 2018. All Rights Reserved Page 2
Journal of Advances in Shell Programming
Volume 5, Issue 1
ISSN: 2395-6690 (Online)

Start Testing Early which contribute to make the big picture of

Start testing early helps in finding and fixing a testing a whole system. Unit testing is generally
number of errors in the early stages of considered as a white box test class.
development, and thus reduces the rework of
finding the errors in the later stages. Integration Testing
It is done when two or more tested units are
Testing is Context Dependent combined into a larger structure. This testing is
Testing should be suitable and different for often done on the interfaces that are between the
different context at different points of time. components and the larger structure that is
being constructed, if its quality property cannot
Test Plan be properly assessed from its components.
Test plan usually describes test strategy, test
scope, test objectives, test environment, System Testing
deliverables of the test, risks and mitigation It tends to test the end-to-end quality of the
involved, schedule, levels of testing to be entire system. System test is often based on the
applied, techniques, methods and tools to be functional requirement specifications of the
used. Test plan should accurately meet the system. Non-functional quality attributes, such
needs of an organization and customer as well. as security, reliability, and maintainability, are
also checked.
Effective Test Cases
Effective test cases must be designed properly Acceptance Testing
so that test can be measured and clear test It is done when the complete system is handed
results can be produced. over to the customers or users from developer
side. The aim of acceptance testing is to assure
Test Valid as Well as Invalid Conditions that the system is working rather than to find
In addition to valid test cases, test cases for error.
invalid and unexpected inputs/conditions must
also be checked. This form of testing is SOFTWARE TESTING TECHNIQUES
sometimes specified as regression testing. The three most important techniques that are
used for finding errors are [3]:
Test at Different Levels Black Box Testing Technique
Different testing must be done at different Black box testing is technique of testing
levels of testing, so, different people can software based on output requirements and
perform testing differently using different without any knowledge of the internal structure
testing techniques at all levels. or coding in the program (Figure 1).

End of Testing
Testing has to be stopped somewhere. It is
stopped when risks are under some limit or if
there is some limitation to it.

SOFTWARE TESTING LEVELS

There are different levels during the process of
testing [12]. Levels of testing include different
methodologies that can be used while
Fig. 1: Black Box Testing.
conducting software testing. There are main
four levels of software testing: In another words, a black box is any device
Unit Testing whose workings are not understood by or
It is done at the lowest level. It tests the basic
accessible to its user. For example, in
unit of software, which can be a module or telecommunications, it is a resistor connected to
component. Unit is the smallest module i.e. a phone line that makes it impossible for the
smallest set of lines of code which can be tested.
telephone company’s equipment to detect when
Unit testing is just one of the levels of testing
a call has been answered. In data mining, a

JoASP (2018) 1-8 © STM Journals 2018. All Rights Reserved Page 3
Comparative Study of Software Testing Techniques Rani and Gupta

black box is an algorithm that does not provide 3. Rules 1 and 2 apply to output conditions.
an explanation of how it works.
If internal program data structures have
Types of Black Box Testing Techniques prescribed boundaries, then produce test cases
(Figure 2) to exercise that data structure at its boundary.
Equivalence Class Partitioning
Cause-Effect Graphing Techniques: We use
this technique when we want to translate a
Boundary Value Analysis
policy or procedure specified in a natural
Black Box Testing

language into software’s language. This

Cause-Effect Graphing Techniques technique means: Input conditions and actions
are listed for a module ⇒ an identifier is
Comparison Testing allocated for each one of them ⇒ cause-effect
graph is created ⇒ this graph is changed into a
Fuzz Testing decision table ⇒ the rules of this table are
modified to test cases.
Model-based testing
Fig. 2: Different Types of Black Box Testing Comparison Testing: In situations when
Techniques. reliability of software is critical, redundant
software is produced. In that case, we use this
Equivalence Class Partitioning: This technique technique. This technique means: Software
divides the input domain of a program onto engineering teams produce independent
equivalence classes. Equivalence classes are set versions of an application→ each version can
of valid or invalid states for input conditions, be tested with the same test data→ so the same
and can be defined in the following way: output can be ensured.
1. An input condition specifies a range: one
valid and two invalid equivalence classes Fuzz Testing: Fuzz testing is often called
are defined; robustness testing or negative testing. It was
2. An input condition needs a specific value: developed by Barton Miller at the University of
one valid and two invalid equivalence Wisconsin in 1989. This technique feeds
classes are defined; random input to application. The main
3. An input condition specifies a member of a characteristic of fuzz testing are:
set: one valid and one invalid equivalence  In fuzz testing, the input is random;
class are defined;  If the application crashes or hangs, the test
4. An input condition is Boolean: one valid is failed;
and one invalid equivalence class are  Fuzz testing can be automated to a high
defined. degree.

Well, using this technique, we can get test cases Model-Based Testing: Model-based testing is
which identify the classes of errors. automatic generation of efficient test
procedures using models of system
Boundary Value Analysis: This technique is requirements and specified functionality. In this
like the technique equivalence partitioning, method, test cases are derived in whole or in
except that for creating the test cases beside part from a model that describes some aspects
input domain use output domain. We can form of the system under test. These test cases are
the test cases in the following way: known as the abstract test suite. This testing
1. An input condition specifies a range forces detailed understanding of the system
bounded by values a and b: test cases behaviour and can do early bug detection.
should be made with values just above and
just below a and b, respectively; Advantages
2. An input condition specifies various values: 1. Testers need not to have knowledge on
test cases should be produced to exercise specific programming language.
the minimum and maximum numbers; 2. Testing is done from user’s point of view.

JoASP (2018) 1-8 © STM Journals 2018. All Rights Reserved Page 4
Journal of Advances in Shell Programming
Volume 5, Issue 1
ISSN: 2395-6690 (Online)

3. It helps to expose any ambiguities or Types of White Box Testing Techniques

inconsistencies in the requirement (Figure 4)
specifications [13]. Condition Testing
4. Programmer and tester both are

White Box Testing

independent of each other.
Data Flow Testing

Disadvantages
1. Test cases are hard to design without clear Basis Path Testing
specifications.
2. Probability of having repetition of tests that Loop Testing
are already done by programmer. Fig. 4: Different Types of White Box Testing
3. Some parts of back end are not tested at all. Techniques.

White Box Testing Condition Testing: By this technique, each

White box testing is highly effective in logical condition in a program is tested.
detecting and resolving problems, because bugs A relational expression takes a form:
(bug or fault is an appearance of an error in a E1> operator relational <E2,
software [14]) can often be found before they Where, E1 and E2 are arithmetic expressions,
cause trouble. We can shortly define this and relational operator is one of the following:
method as testing software with the knowledge <, ≤, =, ≠, >, or ≥. This technique determines
of the internal structure and coding inside the not only errors in the conditions of a program
program [15]. It is necessary for a tester to have but also errors in the whole program.
full knowledge of source code.
Data Flow Testing: Data flow testing is another
type of white box testing which looks at how
data moves within a program. In data flow
testing, the control flow graph is annoted with
the information about how the program
variables are defined and used (Control flow
graph is a diagrammatic representation of a
program and its execution). We can also define
data flow testing as testing techniques which is
based on the observation that values associated
with variables can effect program execution.

Basis Path Testing: If we use this technique, we

can evaluate logical complexity of procedural
design. After that, we can employ this
technique for description of basic set of
execution paths. For obtaining the basis set and
for presentation control flow in the program, we
use flow graphs. Main components of flow
Fig. 3: White Box Testing.
graphs are: Nodes, Edges and Regions.
White box testing is also called white box
Cyclomatic Complexity is software metric. The
analysis, clear box testing or clear box analysis.
value evaluated for cyclomatic complexity
It is a strategy for software debugging in which
defines the number of independent paths in the
the tester has excellent knowledge of how the
basis set of a program. Independent path is any
program components interact (Figure 3). This
path through a program that introduces at least
method can be used for Web services
one new set of processing statements. For the
applications. This is purely done based on
given graph G, cyclomatic complexity V(G) is
customers view point; only tester knows the set
equal to:
of inputs and predictable outputs [16, 17].
1. The number of regions in the flow graph;

JoASP (2018) 1-8 © STM Journals 2018. All Rights Reserved Page 5
Comparative Study of Software Testing Techniques Rani and Gupta

2. V(G)=E–N+2, where, E is the number of It is a technique to test the application with

edges, and N is the number of nodes; limited knowledge of the internal working of an
3. V(G)=P+1, where, P is the number of application and also has the knowledge of
predicate nodes. fundamental aspects of the system [18].
Therefore, a tester can verify both the output of
Loop Testing: Loop testing is another type of the user interface and also the process that leads
white box testing which exclusively focuses on to that user interface output. Gray-box testing
the validity of loop construct. Loops are simple can be applied to most testing phases; however
to test unless dependencies exist between the
it is mostly used in integration testing. Different
loops or among the loop and the code it
types of grey box testing techniques are briefly
contains. There are four classes of loops:
Simple loops, Nested loops, Concatenated described below in Figure 6 [19]:
Loops, and Unstructured loops. Types of Grey Box Testing Techniques
Orthogonal Array Testing
Advantages

Grey Box Testing

1. It reveals error in hidden code by removing Matrix Testing
extra lines of code.
2. Maximum coverage is attained during test Regression Testing
scenario writing [18].
3. Developer carefully gives reasons about
Pattern Testing
implementation.
Fig. 6: Different Types of Grey Box Testing
Disadvantages Techniques.
1. A skilled tester is needed to carry out this
testing because knowledge of internal Orthogonal Array Testing: This testing
structure is required. technique is used as subset of all possible
2. Many paths will remain untested as it is combinations.
very difficult to look into every nook and
corner to find out hidden errors. Matrix Testing: In this testing technique, the
status report of the project is stated.
Grey Box Testing Technique
Gray-box testing attempts, and generally
Regression Testing: If any kind of new changes
succeeds, to combine the benefits of both black-
box and white-box testing. Gray-box testing are made in the software, then regression testing
takes the straight-forward approach of black- implies running of test cases.
box testing, but also employs some limited
knowledge of the inner workings of the Pattern Testing: Pattern testing technique
application (Figure 5). verifies the good application for its architecture
and design.
White box + Black box = Grey box
Advantages
1. It provides combined benefit of black box
and white box testing techniques.
2. In grey box testing, tester can design
excellent test scenarios.
3. Unbiased testing.
4. Create an intelligent test authoring.

Disadvantages
1. Test coverage is limited as the access to
source code is not available.
2. Many program paths remain untested.
Fig. 5: Grey Box Testing. 3. The test cases can be redundant [18].

JoASP (2018) 1-8 © STM Journals 2018. All Rights Reserved Page 6
Journal of Advances in Shell Programming
Volume 5, Issue 1
ISSN: 2395-6690 (Online)

COMPARISON BETWEEN THREE errors and flaws in the developed software. It

FORMS OF TESTING TECHNIQUES executes the software with an intention of
(Table 1) finding errors in it. Software testing can provide
Table 1: Comparison between Three Forms of an independent view of the software to allow
the business to comprehend and appreciate the
Testing Techniques.
S. Black Box White Box Grey Box
risk of software implementation. So, to carry
No. Testing Testing Testing out software testing in a more effective
1. Black box In white box In grey box behaviour, this paper provides description and
testing analyses testing there is testing there is comparison of three main software testing
fundamental full partial knowledge techniques.
aspects only i.e. knowledge of of internal
there is no internal working.
knowledge of working. REFERENCES
internal working. 1. Parezanovic Nedeljko. Racunarstvo i
2. Black box It is performed It is performed by informatika. Beograd: Zavod za udzbenike
testing is by both end users and also i nastavna sredstva; 1996.
performed by developers by tester and 2. Kapfhammer Gregory M. Software
end users and and testers. developers (user
Testing: The Computer Science and
also by tester acceptance
and developers testing). Engineering Handbook. CRC Press; May
(user acceptance 2004.
testing). 3. Mohd. Ehmer Khan. Different Forms of
3. In black box Here, Here, granularity Software Testing Techniques for Finding
testing, the granularity is is medium. Errors. IJCSI. May 2010; 7(3)(1): 11–16p.
granularity is high.
low.
4. Azhar. Software Testing Methodologies.
4. Black box This testing This testing suits Available at http://azhar-paperpresentatio
testing does not does not suit the algorithm n.blogspot.com/2010/04/software-testing-
suit the the algorithm testing (suited for methodologies.html
algorithm testing. all). 5. Bacchelli A, Ambros MD, Lanza M. Are
testing. Popular Classes More Defect Prone? In
5. Here, the testing
Here, internals Here, the test
Proceedings of the 13th International
is based on only are fully design is based on
external known. high level data Conference on Fundamental Approaches to
exceptions and flow diagram, Software Engineering, FASE’10, Berlin,
internal database Heidelberg: Springer-Verlag; 2010; 59–
behaviour of the diagrams, internal 73p.
program is states, knowledge 6. Engstrom E, Runeson P, Wikstrand G. An
completely of algorithm and
ignored. architecture.
Empirical Evaluation of Regression
6. Black box White box Grey box testing Testing Based on Fix-Cache ¨
testing is least
testing is most process is Recommendations. In Software Testing,
exhaustive andexhaustive and somewhere in Verification and Validation (ICST), 2010
time consuming time between. Third International Conference on. Apr
process. consuming 2010; 75–78p.
process.
7. This testing can Here, the Here, in testing
7. Nagappan N, Ball T. Use of Relative Code
be tested only by testing can be the data domains Churn Measures to Predict System Defect
trial and error done better by and internal Density. In Proceedings of the 27th
method. testing data boundaries can be International Conference on Software
domains and tested and over Engineering, ICSE’05. 2005; 284–292p.
internal flow. 8. Nam J, Pan SJ, Kim S. Transfer Defect
boundaries.
Learning. In Proceedings of the 2013
International Conference on Software
CONCLUSION Engineering, ICSE’13, Piscataway, NJ,
Software testing is an activity or a process of USA: IEEE Press; 2013; 382–391p.
identifying the correctness, capability and 9. Kim S, Zhang H, Wu R, et al. Dealing with
quality of software program. The purpose is to Noise in Defect Prediction. In Proceeding
check whether the software satisfies the specific of the 33rd International Conference on
requirements, needs and expectations of the Software Engineering, ICSE’11. New
customer. Software testing exhibits all faults, York, NY, USA: ACM; 2011; 481–490p.

JoASP (2018) 1-8 © STM Journals 2018. All Rights Reserved Page 7
 Comparative Study of Software Testing Techniques Rani and Gupta

10. Menzies T, Greenwald J, Frank A. Data Engineering Conference, 2001

Mining Static Code Attributes to Learn Proceedings, Australia. 2001; 12–20p.
Defect Predictors. IEEE Trans Softw Eng. 17. Mitra P, Chatterjee S, Ali N. Graphical
Jan 2007; 33(1): 2–13p. Analysis of MC/DC Using Automated
11. Jiang T, Tan L, Kim S. Personalized Defect Software Testing. In Electronics Computer
Prediction. In Automated Software Technology (ICECT), 2011 3rd
Engineering (ASE), 2013 IEEE/ACM 28th International Conference on. 2011; 3: 145–
International Conference on. Nov 2013; 149p.
279–289p. 18. Khan, Mohd Ehmer, Farmeena Khan. A
12. Rasneet Kaur Chauhan, Iqbal Singh. Latest Comparative Study of White Box, Black
Research and Development on Software Box and Grey Box Testing Techniques.
Testing Techniques and Tools. IJCET. International Journal of Advanced
2014; 4(4): 2368–2372p. Computer Sciences and Applications
13. Nidhra Srinivas, Jagruthi Dondeti. Black (IJACSA). 2012; 3(6): 12–15p.
Box and White Box Testing Techniques: A 19. Grey Box Testing from Wikipedia
Literature Review. International Journal of available at http://en.wikipedia.org/wiki/G
Embedded Systems & Applications ray_box_testing
(IJESA). 2012; 2(2): 29–50p.
14. http://www.testingstandards.co.uk/living_
glossary.htm# Testing, Feb 08, 2009. Cite this Article
15. http://www.pcmag.com/encyclopedia_ter Seema Rani, Deepali Gupta. A Comparative
m/0,2542,t=white+box+testing&i=54432, Study of Different Software Testing
00.asp, Feb 08, 2009. Techniques: A Review. Journal of
16. Murnane T, Reed K. On the Effectiveness Advances in Shell Programming. 2018;
of Mutation Analysis as a Black Box 5(1): 1–8p.
Testing Technique. In Software

JoASP (2018) 1-8 © STM Journals 2018. All Rights Reserved Page 8

View publication stats