A Technical Report on

THE ROLE OF AUTOMATION IN

SOFTWARE TESTING
From
[Software Engineering - (ESC-591)]

For Continuous Assessment 2(CA 2)

Submitted by

NAME : SAYAN GHOSH

REGISTRATION NUMBER : 232690120125

ROLL NUMBER :26900123054

DEPARTMENT : CSE

YEAR : 3rd

SEMESTER : 5th

SESSION : 2023-2024
 AKNOWLEDGEMENT

I would like to express my special thanks to our ‘SOFTWARE ENGINEERIG ’ professor for
giving me this golden opportunity to do this wonderful project on the topic ‘The Role of
Automation in Software Enginering ’. I would also like to thank our college MODERN
INSTITUTE OF ENGINEERING & TECHNOLOGY and our teacher In charge “Kalyan
Adhikary” Sir. Finally, I would like to thanks my parents, and my friends who helped me a lot to
finalize this project and have been with me all the time.
 The Role of Automation in
Software Testing

Abstract - The growing demand for delivering quality software faster “Quality at Speed” requires faster and
successful execution of software testing to ensure its standard. Utilizing appropriate testing method(s) and
right test automation tools/framework are two defining factors for a successful and effective software testing
project. Using one testing method will not be sufficient to test software and ensure its standard, a
combination of some appropriate testing techniques is often required. Likewise, is no one tool that can satisfy
all automated testing needs which makes finding the right tool combination difficult. Knowing the various
testing methods and tools/frameworks is the first step towards achieving a successful and efficient software
testing. This article presents a comprehensive study of test automation tools and frameworks. Firstly,
automated testing and their categories were explained, followed by an explanation of the various test
automation frameworks. Finally, a brief explanation and comparison of some of the most commonly used
automation tools was presented.
Keywords: Software Testing, Software Automated Testing, Automation Tools Categories, Test Automation
Tools, Test Automation Frameworks
I. INTRODUCTION
Software programs has become an integral part of our everyday life as it potentially touches millions of people
in various domain of life [1], this calls for safe and reliable software. Unfortunately, humans are prone to error,
therefore the fundamental facts of humans’ core involvement in software development makes errors an
inevitable inclusion in software [2]. Software errors (bugs) can negatively affect the live operation and even
cause death [3]. It is important to treat such errors early in the development phase because they get costlier with
progress [4]. For example; Jones, in his survey [5], shows that $500 billion is lost annually due to poor software
quality, furthermore, a report released by the National Institute of Standards and Technology (NIST) shows that
Software bugs are costing the USA economy an estimated $59.5 billion annually, and about a third of the cost
($22.5 billion) could be reduced through improvements in testing, although it won’t eliminate all the bugs [6].
The eminent effects of software bugs cannot be overestimated and hence, the need for software to be tested
before delivered.
Software testing is defined as the process of evaluating a software program with the intent of finding fault or
errors. Software testing is also done to; make sure that software performs its intended purpose correctly [1],
access, achieve and preserve the quality of a software [2], and thereby verify that the software is fit for use [7].
Software testing is an integral phase in Software Development Life Cycle (SDLC) process [8], testing utilizes
around 50% of software projects’ development time and effort [1], [2], [7]. In SDLC, software is incomplete
until it has passed testing [9]. The overall purpose of testing is not to demonstrate that the system is free of
errors but to give confidence that the system is working well before installation. An error-free system shows that
either no testing or poor testing was performed on the software system [2].
Software testing can be manual or automated. Manual Testing does not require knowledge of any testing tool, it
requires lots of effort and more people. Automating manual testing is no different from a programmer using a
coding language to write programs to automate any manual process [10]. Automated testing involves the use of
automation tools and/or with frameworks in executing tests. It requires knowledge of automation tools and,
sometimes, programming skills. Automated testing increases testing accuracy and saves effort and time of tester
compared to manual testing [11], which is not only a costly and time-consuming exercise, but it is also prone to
error.
Defining factors for successful and efficient software testing projects are: (a) selecting and using an appropriate
testing method, and (b) choosing and using the right test automation tool and/or framework [12]. Software
testing methods are the various strategies or approaches used to test an application to ensure it behaves and
looks as expected. These encompass everything from front to back-end testing, including unit and system
testing. Umar [13] provides a detailed explanation on software testing methods, this paper focuses on software
test automation, specifically, on automated software testing, its various tools, and their categories as well as
testing frameworks that can be used to achieve effective, and successful testing project.
The rest of this paper is organized as follows: Section I is the Introduction, followed by Automated Software
Testing and their categories in Section II, then Section III presents Test Automation Frameworks and their
types. In Section IV, an explanation and comparison of some popular test automation tools were provided and
the last Section (V) Concludes the paper.
II. AUTOMATED SOFTWARE TESTING
Automated testing is a process using software separate from the software under test to control the execution of
tests and the comparison of actual outcomes with the expected outcomes [14]. Automation tools are used to
automate certain sections of manual testing but not all [11]. Automated testing generally saves time, the tester
can efficiently run a large number of tests in a short period and so important and repetitive tasks, as well as
testing that would be difficult to do manually, can be automated. Besides saving time, automation testing saves
money and effort, increases the quality of the testing tasks [12] and also helps in improving software accuracy.
Test Automation requires a skilled tester with knowledge of the automation tools and the software being tested
to set up the test cases and perform the testing [10].
Table 1: Advantages and Disadvantages of Test Automation

Advantages Disadvantages
Improves accuracy and quick finding of bugs Choosing the right tool requires considerable effort,
compared to manual testing time, and evolution plan.
Saves time and effort by making testing more Requires knowledge of the testing tool.
efficient
Increases test coverage because multiple testing tools Cost of buying the testing tool and, in the case of
can be used at once allowing for parallel testing of playback methods, test maintenance is a bit
different test scenarios expensive
Automation test script is repeatable Proficiency is required to write the automation test
scripts.
A. Automation Tools Categories
Software testing automation tools can be divided into different categories as follows: Unit Testing Tools,
Functional Testing Tools, Code Coverage Tools, Test Management Tools, and Performance Testing Tools.

Figure 1: Categories of Test Automation Tools

1) Unit Testing Tools

Unit testing involves testing the most basic units of code, the Unit Testing tools are used to aid unit testing
process. Practically, unit testing tools are the most used tools to automate tests [12] and are easily integrable as a
framework within a development environment such as NetBeans. Developers can write test cases in an
executable language and the test cases can be executed automatically. Unit testing tools are used to tests the
correct working of a particular unit (or method) as well as to check code structure and ensure correct observance
of programming practice [15] and also ensure correct working of individual units [14]. Some of the unit testing
frameworks are: JUnit, NUnit, JMockit, PHPUnit, and soon.
 2) Functional Testing Tools
Functional testing is testing performed to ensure that software is functioning as per users’ requirements.
Functional testing bases its test cases on the specifications of the software component under test. Functional
testing tools are tools that are used in functional testing process. Functional testing tools test functions by
feeding them input and examining the obtained output in comparison with the specified test oracle (expected
output) in the given test case. Functional testing tools evaluate the compliance of a program with specified
requirements [16]. Some of the functional testing tools are: Selenium, HP QuickTest Professional,
TestComplete, Ranorex, Watir, Tricentis Tosca Testsuite, Test Studio, and soon.
3) Code Coverage Tools
Code Coverage Tools are testing tools that are used to determine parts of software code covered by automated
tests. It measures the number of lines, statements, or blocks of code tested using automated test suites. Code
coverage testing is an essential metric to understand the quality of Quality Assurance (QA) efforts made [17].
As test cases are developed, code coverage tools highlight aspects of the code which may not be adequately
tested and which require additional testing [18], it shows how much of software code is not covered by
automated tests and is therefore vulnerable to defects. Code coverage tools typically measure code coverage in
percentage values – the closer to 100%, the lower the chance of having undetected software bugs [19]. Some of
Code Coverage Tools are: Cobertura, CodeCover, EMMA, PITest, Atlassian Clover, and soon [17].
4) Test Management Tools
Test management tools are used to automate testing activities (such as test cases creation, test plans, test
strategy, test results, test reports and soon), and help teams manage projects easily by providing a searchable and
maintainable placeholder for test activates. Various Test management tools have a diverse set of features with
distinct ways of managing testing. But they commonly offer the possibility of streamlining the testing procedure
and permit snappy access to data analysis, and simple correspondence over different venture groups [20]. The
following are some of the test management tools: Test Manager, Test Link, TETware, Test Environment Toolkit
(TET), QA Complete, and soon.
5) Performance Testing Tools
Performance testing tools are used to aid performance testing processes. Performance Testing is testing perform
to determine how the software will perform in terms of responsiveness and stability under various conditions
and workload. It can also serve to investigate, measure, validate or verify other quality attributes of software,
such as scalability, reliability, and resource usage [21]. Performance testing tools are used to evaluate the
performance of software or component in resource usage, throughput and stimulus-response time with specified
performance requirement. According to Khan [22] Load and Stress testing are the two performance testing; [23]
added Endurance and Spike testing, while [21] added: Soak, Breakpoint, Configuration, Isolation, and Internet
testing among the other performance testing types. Performance testing can be done using a wide variety of
tools such as: JMeter, Rational Performance Tester, HP LoadRunner, Silk Performer and soon.
III. TEST AUTOMATION FRAMEWORK
A test automation framework is a defined, extensible support structure within which the test automation suite is
developed and implemented [24]. Automation Framework comprises of a combination of tools and practices
that are designed to help in executing software testing more efficiently [25]. It includes the physical structures
used for test creation and implementation as well as the logical interaction between components such as coding
standards, test-data handling methods, object repositories, processes for storing test results, or information on
how to access external resources. A framework “facilitates a standard way for modifying, adding, and deleting
the [test] scripts and functions,” and provides “scalability and reliability with less effort” [26].
A. Importance of Test Automation Framework
While testers can still script or record tests using an automation tool, however, integrating it with an organized
framework typically provides additional benefits. A well-defined test automation framework will help the
testing team in; achieving higher reusability of test components, developing scripts that are easily maintainable
and, obtaining high-quality test automation scripts. Utilizing a framework for automated testing will also
increase test speed and efficiency, improve test accuracy, and reduce test maintenance costs as well as lower
risks [25]. Test automation frameworks provide support foundation to a variety of automated software tests
including: Unit testing, Functional testing, Performance testing, and soon [27].
B. Types of Test Automation Framework
There are various types of test automation framework with each having its architecture and may differ from each
other based on their support to different automation key factors like reusability, ease of maintenance and soon
[24]. It’s important to choose the right framework for automating software testing. Some of the most common
test automation frameworks are explained below.
 1) Linear Automation Framework
This is a first-generation and the simplest testing framework that is used to test a web application’s user
interface (UI). Here, the tester records each step such as navigation, user input, or checkpoints, and then plays
the script back to conduct the test in a sequential order without the need to write code to create functions. It’s
also known as a record-and-playback framework [25]
2) Modular Based Framework
This is a type of test automation framework in which software under test is divided into separate functions,
modules, or sections, each of which will be tested in isolation. A test script is created for each part and then
combined to build larger tests in a hierarchical approach. These larger sets of tests will begin to represent
various test cases. The key to achieving modularity is by decomposing the functionality and recombining the
modules [26]. The modular automation framework is also known as functional decomposition framework [27].
3) Library Architecture Framework
This framework is based on the modular framework but has some additional benefits. It identified similar tasks
or functions within the software that need to be tested and grouped them by function instead of dividing the
software under test into various independent modules or units to be tested in isolation each with its test scripts.
Thus, the software is ultimately broken down by common functions or objectives. These functions are kept in a
library which can be called upon by the test scripts whenever needed [25]. Compared to the Modular Based
Framework, this framework has a higher degree of reusability because there is a library of common functions
that can be used by multiple test scripts.
4) Data-Driven Framework
The Data-driven test framework is a prominent practice in software testing in which test data are separated from
script logic and stored externally to an external data source, such as Text Files, Excel Spreadsheets, CSV files,
SQL Tables, or ODBC repositories. Test scripts are connected to the external data source and told to read and
populate the necessary data when needed [25]. Unlike in Linear Automation, Modular-based, and Library
Architecture frameworks, this framework separates test data from the test script, thus allowing testers to test the
same function or feature of software multiple times with different sets of test data without the need to modify
the test script.
5) Keyword-Driven Framework
This framework follows a similar approach to data-driven framework, in such a way that the test data and script
logic are also separated, but this approach takes it a step further. With this approach, keywords are also stored
along with their associated objects in an external data source, making them independent from the automation
tool being used to execute the tests. Keywords are part of a script representing various actions being performed
to test the software. These keywords can be labeled as simply as ‘click,’ or ‘login,’ or with complex labels like
‘clicklink,’ or ‘verifylink’, and the objects can be a ‘Submit Button,’ or ‘Login Username.’ For this approach to
work properly, a shared object repository is needed to map objects to their associated actions [25].
6) Hybrid Testing Framework
The hybrid framework is a combination of two or more frameworks set up to get the best practices of different
frameworks suitable for automation needs. It leverages the advantages of some frameworks and avoid
weaknesses of others. Every software is different, and so should the processes used to test them. As more teams
move to an agile model, setting up a flexible framework for automated testing is crucial. A hybrid framework
can be more easily adapted to get the best test results [25].
The process of framework evaluating and selecting requires detailed planning and effort. When choosing a
framework, it is crucial to consider the right test automation tools to implement the framework with and ensure
that it can easily accommodate various automation tools and changes in the software under test thus, achieving
smooth tools integration and ultimately helping in achieving successful testing.
IV. AUTOMATION TOOLS
An automation tool is a software itself with the help of which the actual software in focus can be tested, in other
words, the automation tool help and serves as a means in doing software testing [28]. The rapid and unparalleled
change in technology affects how organizations develop, validate, deliver, and operate software products.
Meeting the demands of today’s software quality standard requires testing the software, and the success of the
testing project is largely determined by testing technique and automation tool used [24]. There are various
testing automation tools each having its strengths and weaknesses and serving for a different purpose. A detailed
analysis of those various tools should be performed before selecting any tool. Budget, application type, testing
requirements, skills required to use the tool are among the factors need to be considered [29]. The investigation
process requires a lot of effort, time, and planning. However, the time and effort spent during tool evaluation
can go a long way in ensuring a successful testing project [30]. There follows an explanation of some of the
most commonly used automation tools along with their advantages and disadvantages.
 1) JUnit
JUnit is a unit testing framework for the Java programming language. JUnit has been important in the
development of test-driven development, and it allows the developer to write an oracle for each test case, and to
automatically execute test sets. JUnit in particular permits to automatically regression test several test sets. If a
formal specification is available, it can be translated into assertions which can be checked at runtime, and thus
serve as a test oracle [31]. The JUnit framework is one of a family of unit testing frameworks that are
collectively known as xUnit that originated with SUnit [32].
Table 2: Advantages and Disadvantages of JUnit

Advantages Disadvantages
Facilitates Changes and Simplifies Requires programming skills and consume time to write
Integration and check.
Bugs can be found and resolved early It can only run tests on one JVM (Java virtual machine), as
without affecting other pieces of the code. such developers are unable to test applications that require
multiple JVMs
Makes the coding process more Agile Unit tests tools only automate the testing of functionality
at the unit level
2) Selenium
Selenium is a framework for testing web applications that is compatible with various browsers and platforms
like Windows, Mac, and Linux. Selenium helps the testers to write tests in various programming languages like
Java, PHP, C#, Python, Groovy, Ruby, and Perl. It offers record and playback features for doing tests without
the need to learn test scripting language [33]. Selenium is perhaps the most popular automation framework that
consists of many tools and plugins for Web application testing. Selenium is known for its powerful capability in
performance testing and is a popular choice in open-source test automation space, partly due to its large and
active development and user community [34].
Table 3: Advantages and Disadvantages of Selenium

Advantages Disadvantages
Open source, no licensing and maintenance fees. New teams need to invest time upfront for setup
and integration.
Large and active development and user community to Slow support from the community.
keep pace with software technologies.
Open for integration with other tools and frameworks Required good programming skills and experience
to enhance its capability to set up and integrate Selenium with other tools
and frameworks.
3) Unified Functional Testing (UFT)
UTF, formerly QuickTest Professional (QTP), is a test automation tool for functional and regression testing, it’s
probably the most popular commercial tool for functional test automation [33]. UFT offers a comprehensive set
of features that can cover most of functional automated testing needs on desktop, mobile and Web platforms.
Visual Basic Scripting Edition scripting language is used by this tool to register test processes, operate various
objects and control in testing the applications [27].
Table 4: Advantages and Disadvantages of Unified Functional Testing (UTF)

Advantages Disadvantages
Mature, comprehensive automated testing features Costly solution: license and maintenance fees are
integrated into a single system. considerably high.
Requiring only basic programming skills to get Supporting only VBScript.
started with test creation and execution.
Dedicated user support plus an established large user Possible high costs for upgrades and additional
community. modules.
 4) Katalon Studio
Katalon Studio is an automated testing platform that offers a comprehensive set of features to implement full
automated testing solutions for Web, API, and Desktop and Mobile applications. Built on top of the open-source
Selenium and Appium frameworks, Katalon Studio allows teams to get started with test automation quickly by
reducing the effort and expertise required for learning and integrating these frameworks for automated testing
needs [35].
Table 5: Advantages and Disadvantages of Katalon Studio

Advantages Disadvantages
No licensing and maintenance fees required Poor Community support
Integrating necessary frameworks and features for Support limited features
quick test case creation and execution.
Built on top of the Selenium framework but Lack of choices for scripting languages: only
eliminating the need for advanced programming Java/Groovy is supported.
skills required for Selenium.
5) TestComplete
TestComplete is also a commercial integrated platform for desktop, mobile and Web application testing. It
enables testers to build a robust testing framework that utilizes the broad spectrum of available software testing
methodologies [23]. Like UFT, TestComplete offers some key test automation features such as keyword-driven
and data-driven testing, cross-browser testing, API testing, and CI integrations. This tool supports many
languages including JavaScript, Python, VBScript, JScript, DelphiScript, C++Script, and C#Script for writing
test scripts [36].
Table 6: Advantages and Disadvantages of TestComplete

Advantages Disadvantages
Many scripting languages to choose from. Additional fees for extra modules and add-ons.
Only basic programming skills needed.
Mature, comprehensive automated testing features Like UFT, considerable licensing and maintenance
integrated into a single system. fees needed for TestComplete.
A. Comparison of Automation Tools
There are various automation tools available in the market. Identification of the right automation tool is critical
to ensure the success of the testing project. The table below presents a comparison of some of the popular
testing tools discussed.
 Table 7: Automation Tools Comparison

Features Katalon Studio Selenium UFT TestComplete

Test development Cross-platform Cross-platform Windows Windows
platform
Application Web, Mobile apps, Web apps Windows desktop, Windows desktop,
under test API/Web services Web, Mobile Web, Mobile
apps, API/Web apps, API/Web
services services
Scripting Java/Groovy Java, C#, Perl, VBScript JavaScript, Python,
languages Python, JavaScript, VBScript, JScript,
Ruby, PHP Delphi, C++ and C#
Programming Not required. Advanced skills Not required. Not required.
skills Recommended for needed to integrate Recommended for Recommended for
advanced test various tools advanced test scripts advanced test scripts
scripts
Learning curves Medium High Medium Medium
Ease of Easy to set up and Require installing Easy to set up and Easy to set up and run
installation and run and integrating run
use various tools
Script creation Quick Slow Quick Quick
time
Object storage Built-in object XPath, UI Maps Built-in object Built-in object
and maintenance repository, XPath, repository, smart repository, detecting
object re- object detection, and common objects
identification correction
Image-based Built-in support Require installing Built-in support, Built-in support
testing additional libraries image-based object
recognition
DevOps/ALM Many No (require Many Many
integrations additional libraries)
Continuous Popular CI tools Various CI tools Various CI tools Various CI tools (e.g.
integrations (e.g. Jenkins, (e.g. Jenkins, (e.g. Jenkins, HP Jenkins, HP Quality
Teamcity) Cruise Control) Quality Center) Center)
Test Analytics Katalon Analytics No No No
Product support Community, Open source Dedicated staff, Dedicated staff,
Business support community Community Community
service, Dedicated
staff
License type Freeware Open source Proprietary Proprietary
(Apache 2.0)
Cost Free Free License and License and
maintenance fees maintenance fees
B. Selection of Automation Tool
There is no good or bad automation tool; hence, the choice of automation tool largely depends on the nature of
the software to be tested. For example, Selenium may be the most popular automation tool, but if the software in
focus is desktop-based, this tool has no use here. Thus, proper understanding of the testing requirements for the
software in focus comes first, then searching for the appropriate tool or often combination of tools that match
most of those requirements. Detailed analysis based on the requirements and other certain criteria must be
conducted before selecting the tool [24]. The effort put in the tool evaluation process can determine the
successful execution of the testing project. The selection of the tool depends on various factors such as:
 The software and its technology stack which is to be tested
 Detailed testing requirements
 Skill sets available in the organization
 License cost of the tool
 V. CONCLUSION
Test automation has become an essential part of a successful software testing. The latest World
Quality Report 2018–2019 highlights that test automation is the biggest bottleneck to deliver
“Quality at Speed,” as it saves time, reduces cost, improves efficiency, and increases accuracy
[37]. Thus, effective and successful test automation cannot be achieved without the right
automation tools and framework, this report presents a detailed explanation about various test
automation tools and frameworks as well as provides insights into some of the important factors
to consider when selecting automation tool and framework.