Software Testing & Quality Assurance

Prof. Dr. Tariq Javid

Graduate School of Engineering Sciences & Information Technology
Faculty of Engineering Sciences & Technology
Hamdard University
Spring 2024
Introduction to Software Testing &
Quality Assurance

2
Outline
https://www.triad.co.uk/news/testing-vs-qa

• Introduction
• Why is Testing Necessary?
• Testing Principles
• Test Activities and Tasks

3
 Introduction

• Software Applications: Integral part of work and life

• Experience with software application that did not work as expected
• Software that does not work correctly lead to many problems
• Loss of money, time or business reputation, and even injury or death

• Software testing assesses software quality

• It helps in reducing risk of software failure while in operation
4
 Software Artifacts

• Software artifact is any item produced during software development process

• These artifacts contribute to creation of software application
• Design and Planning: Use cases, flowcharts and diagrams, system architecture
documents
• Development: Source code, unit tests
• Testing and Documentation: Test plans and scripts, test results, user manuals
• Additional: Project management plans, deployment scripts, API documentation
5
 Test Objects

• Software Testing (ST) = Set of activities to discover defects and to evaluate

quality of software artifacts
• These software artifacts are known as test objects

• Common misconception: ST only consists of executing tests (running

software and checking test results)
• ST must be aligned with software development lifecycle
6
 Verification and Validation

• Common misconception is that ST focuses entirely on verifying test object

• Verification: Checking whether system meets specified requirements

• ST involves validation
• Validation: Checking whether system meets users’ and other stakeholders’
needs in its operational environment
7
 Dynamic Testing and Static Testing

• Testing: Dynamic or static

• Dynamic testing involves execution of software
• Dynamic testing uses different types of test techniques and test approaches to
derive test cases

• Static testing does not involve execution of software

• Static testing includes reviews and static analysis
8
 Testing as an Intellectual Activity

• Testing needs to be properly planned, managed, estimated, monitored and

controlled
• Testers use tools
• Testing is largely an intellectual activity
• Testers have specialized knowledge, use analytical skills and apply critical thinking
and systems thinking
• Standard ISO/IEC/IEEE 29119-1 provides information about software testing
concepts
9
 ISO/IEC/IEEE 29119 Series

• Purpose of ISO/IEC/IEEE 29119 series of software testing standards is to

define an internationally agreed set of standards for software testing that can
be used by any organization when performing any form of software testing
• Part 1. Overall concepts and common terminology
• Part 2. Set of core processes at organizational, management, dynamic level
• Part 3. Test documentation
• Part 4. Test design techniques
10
 Typical Test Objectives

• Evaluating work products such as requirements, user stories, designs, code

• Triggering failures and finding defects
• Ensuring required coverage of a test object
• Reducing the level of risk of inadequate software quality
• Verifying whether specified requirements have been fulfilled
• Verifying that a test object complies with contractual, legal, regulatory requirements
• Providing information to stakeholders to allow them to make informed decisions
• Building confidence in the quality of the test object
• Validating whether the test object is complete and works as expected by the stakeholders
11
 Context of Test Objectives

• Objectives of testing can vary, depending upon context

• Work product being tested, test level, risks, software development lifecycle
being followed, and factors related to business context

• Corporate structure, competitive considerations, time to market

12
 Testing and Debugging

• Testing and debugging are separate activities

• Testing can trigger failures that are caused by defects in software (dynamic
testing) or can directly find defects in test object (static testing)
• When dynamic testing triggers a failure, debugging is concerned with
finding causes of this failure (defects), analyzing these causes, and
eliminating them

13
 Typical Debugging Process

• Reproduction of a failure
• Diagnosis (finding the root cause)
• Fixing the cause

14
 Confirmation and Regression Testing

• Subsequent confirmation testing checks whether fixes resolved problem

• Preferably, confirmation testing is done by same person who performed
initial test
• Subsequent regression testing can also be performed, to check whether
fixes are causing failures in other parts of test object

15
 Debugging through Static Testing

• When static testing identifies a defect, debugging is concerned with

removing it
• There is no need for reproduction or diagnosis, since static testing directly
finds defects, and cannot cause failures

16
 Why is Testing Necessary?

• Testing components, systems, and associated documentation helps to identify

defects in software
• Testing, as a form of quality control, helps in achieving agreed upon goals
within set scope, time, quality, and budget constraints
• Testing’s contribution to success should not be restricted to test team activities
• Any stakeholder can use their testing skills to bring the project closer to success

17
 Testing’s Contributions to Success, 1/2

• Testing provides a cost-effective means of detecting defects

• Defects can then be removed (by debugging – a non-testing activity), so testing
indirectly contributes to higher quality test objects
• Testing provides a means of directly evaluating quality of a test object at various
stages in software development life cycle (SDLC)
• These measures are used as part of a larger project management activity,
contributing to decisions to move to next stage of SDLC, such as release decision
18
 Testing’s Contributions to Success, 2/2

• Testing provides users with indirect representation on development project

• Testers ensure that their understanding of users’ needs are considered
throughout the development lifecycle
• Alternative is to involve a representative set of users as part of development
project, which is not usually possible due to high costs and lack of availability of
suitable users
• Testing may also be required to meet contractual or legal requirements, or to
comply with regulatory standards
19
 Testing and Quality Assurance, 1/3

• While people often use terms “testing” and “quality assurance” (QA)
interchangeably, testing and QA are not the same
• Testing is a form of quality control (QC)

20
 Testing and Quality Assurance, 2/3

• QC is a product-oriented, corrective approach that focuses on those

activities supporting achievement of appropriate levels of quality
• Testing is a major form of QC, while others include formal methods (model
checking and proof of correctness), simulation and prototyping

21
 Testing and Quality Assurance, 3/3

• QA is process-oriented, preventive approach that focuses on implementation and

improvement of processes
• It works on basis that if a good process is followed correctly, then it will generate a good
product
• QA applies to both development and testing processes, and is responsibility of everyone on a
project
• Test results are used by QA and QC
• In QC results are used to fix defects, while in QA results provide feedback on how well
development and test processes are performing
22
 Errors, Defects, Failures, and Root Causes, 1/4

• Human beings make errors (mistakes), which produce defects (faults, bugs),
which in turn may result in failures
• Humans make errors for various reasons, such as time pressure, complexity
of work products, processes, infrastructure or interactions, or simply
because they are tired or lack adequate training

23
Errors, Defects, Failures, and Root Causes, 2/4

• Defects can be found in documentation, such as requirements specification or test

script, in source code, or in a supporting artifact such as a build file
• Defects in artifacts produced earlier in the SDLC, if undetected, often lead to
defective artifacts later in the lifecycle
• If a defect in code is executed, system may fail to do what it should do, or do
something it should not, causing a failure
• Some defects will always result in failure if executed, while others will only result in
a failure in specific circumstances, and some may never result in a failure
24
 Errors, Defects, Failures, and Root Causes, 3/4

• Errors and defects are not only causes of failures

• Failures can also be caused by environmental conditions, such as when
radiation or electromagnetic field cause defects in firmware

25
 Errors, Defects, Failures, and Root Causes, 4/4

• Root cause is a fundamental reason for the occurrence of a problem (e.g., a

situation that leads to an error)
• Root causes are identified through root cause analysis, which is typically
performed when a failure occurs or a defect is identified
• It is believed that further similar failures or defects can be prevented or their
frequency reduced by addressing the root cause, such as by removing it

26
 Testing Principles

• Testing shows the presence, not the absence of defects

• Exhaustive testing is impossible
• Early testing saves time and money
• Defects cluster together
• Tests wear out
• Testing is context dependent
• Absence-of-defects fallacy
27
 Testing Shows Presence, Not Absence of
Defects

• Testing can show that defects are present in the test object, but cannot
prove that there are no defects
• Testing reduces the probability of defects remaining undiscovered in the
test object, but even if no defects are found, testing cannot prove test
object correctness

28
 Exhaustive Testing is Impossible

• Testing everything is not feasible except in trivial cases

• Rather than attempting to test exhaustively, test techniques, test case
prioritization, and risk-based testing, should be used to focus test efforts

29
 Early Testing Saves Time and Money

• Defects that are removed early in the process will not cause subsequent
defects in derived work products
• Cost of quality reduced since fewer failures occur later in SDLC
• To find defects early, both static testing and dynamic testing should be
started as early as possible

30
 Defects Cluster Together

• Small number of system components usually contain most of defects

discovered or are responsible for most of the operational failures
• This phenomenon is an illustration of the Pareto principle, e.g., 80/20
• Predicted defect clusters, and actual defect clusters observed during testing
or in operation, are an important input for risk-based testing

31
 Tests Wear Out

• If the same tests are repeated many times, they become increasingly
ineffective in detecting new defects
• To overcome this effect, existing tests and test data may need to be
modified, and new tests may need to be written
• However, in some cases, repeating the same tests can have a beneficial
outcome, e.g., in automated regression testing, to assess change impact

32
 Testing is Context Dependent

• No single universally applicable approach to testing

• Testing is done differently in different contexts

33
 Absence-of-Defects Fallacy

• It is a fallacy (i.e., a misconception) to expect that software verification will

ensure the success of a system
• Thoroughly testing all the specified requirements and fixing all the defects
found could still produce a system that does not fulfill the users’ needs and
expectations, that does not help in achieving the customer’s business goals,
and that is inferior compared to other competing systems
• In addition to verification, validation should also be carried out
34
 Test Activities, Testware, and Test Roles, 1/2

• Testing is context dependent, but, at a high level, there are common sets of
test activities without which testing is less likely to achieve test objectives
• These sets of test activities form a test process
• Test process can be tailored to a given situation based on various factors
• Which test activities are included in this test process, how they are
implemented, and when they occur is normally decided as part of the test
planning for the specific situation
35
 Test Activities, Testware, and Test Roles, 2/2

• General aspects of test process are described in terms of test activities and tasks,
impact of context, testware, traceability between test basis and testware, and
testing roles
• Standard ISO/IEC/IEEE 29119-2 provides information about test processes
• Test process usually consists of main groups of activities
• Although many of these activities may appear to follow a logical sequence, they are
often implemented iteratively or in parallel
• Testing activities usually need to be tailored to the system and the project
36
 Test Activities and Tasks

• Test planning
• Test monitoring and control
• Test analysis
• Test design
• Test implementation
• Test execution
• Test completion
37
 Test Planning

• Test planning consists of defining the test objectives and then selecting an
approach that best achieves the objectives within the constraints imposed
by the overall context

38
 Test Monitoring and Control

• Test monitoring involves the ongoing checking of all test activities and the
comparison of actual progress against the plan
• Test control involves taking actions necessary to meet objectives of testing

39
 Test Analysis

• Test analysis includes analyzing test basis to identify testable features and to
define and prioritize associated test conditions, together with related risks and
risk levels
• Test basis and test objects are also evaluated to identify defects they may contain
and to assess their testability
• Test analysis is often supported by use of test techniques
• Test analysis answers question “what to test?” in terms of measurable coverage
criteria
40
 Test Design

• Test design includes elaborating test conditions into test cases and other testware
(e.g., test charters)
• This activity often involves identification of coverage items, which serve as a guide
to specify test case inputs
• Test techniques can be used to support this activity
• Test design also includes defining test data requirements, designing test
environment and identifying any other required infrastructure and tools
• Test design answers question “how to test?”
41
 Test Implementation

• Test implementation includes creating or acquiring testware necessary for test

execution (e.g., test data)
• Test cases can be organized into test procedures and are often assembled into test
suites
• Manual and automated test scripts are created
• Test procedures are prioritized and arranged within a test execution schedule for
efficient test execution
• Test environment is built and verified to be set up correctly
42
 Test Execution

• Test execution includes running tests in accordance with test execution schedule (test runs)
• Test execution may be manual or automated
• Test execution can take many forms, including continuous testing or pair testing sessions
• Actual test results are compared with expected results
• Test results are logged
• Anomalies are analyzed to identify their likely causes
• Analysis allows us to report anomalies based on failures observed

43
 Test Completion

• Test completion activities usually occur at project milestones (e.g., release, end of iteration,
test level completion) for any unresolved defects, change requests or product backlog items
created
• Any testware that may be useful in the future is identified and archived or handed over to
appropriate teams
• Test environment is shut down to an agreed state
• Test activities are analyzed to identify lessons learned and improvements for future iterations,
releases, or projects
• Test completion report is created and communicated to stakeholders
44
 Test Process in Context

• Testing is not performed in isolation

• Test activities are an integral part of the development processes carried out
within an organization
• Testing is also funded by stakeholders and its final goal is to help fulfill
stakeholders’ business needs
• Therefore, the way the testing is carried out will depend on a number of
contextual factors
45
 Contextual Factors, 1/2

• Stakeholders (needs, expectations, requirements, willingness to cooperate, etc.)

• Team members (skills, knowledge, level of experience, availability, training needs, etc.)
• Business domain (criticality of test object, identified risks, market needs, specific legal regulations,
etc.)
• Technical factors (type of software, product architecture, technology used, etc.)
• Project constraints (scope, time, budget, resources, etc.)
• Organizational factors (organizational structure, existing policies, practices used, etc.)
• Software development lifecycle (engineering practices, development methods, etc.)
• Tools (availability, usability, compliance, etc.)
46
 Contextual Factors, 2/2

• These factors have an impact on many test-related issues, including: test

strategy, test techniques used, degree of test automation, required level of
coverage, level of detail of test documentation, reporting, etc.

47
 Testware

• Testware is created as output work products from test activities

• There is a significant variation in how different organizations produce,
shape, name, organize and manage their work products
• Proper configuration management ensures consistency and integrity of
work products

48
 Work Products, 1/3

• Test planning work products

Test plan, test schedule, risk register, and entry and exit criteria. Risk register is a list of risks together
with risk likelihood, risk impact and information about risk mitigation. Test schedule, risk register and
entry and exit criteria are often a part of the test plan
• Test monitoring and control work products
Test progress reports, documentation of control directives, and risk information
• Test analysis work products
Prioritized test conditions (e.g., acceptance criteria), and defect reports regarding defects in the test
basis (if not fixed directly)
49
 Work Products, 2/3

• Test design work products

Prioritized test cases, test charters, coverage items, test data requirements
and test environment requirements
• Test implementation work products
Test procedures, automated test scripts, test suites, test data, test execution
schedule, and test environment elements. Examples of test environment
elements include: stubs, drivers, simulators, and service virtualizations
50
 Work Products, 3/3

• Test execution work products

Test logs and defect reports
• Test completion work products
Test completion report, action items for improvement of subsequent projects
or iterations, documented lessons learned, and change requests (e.g., as
product backlog items)

51
 Traceability between Test Basis and Testware,
1/4

• Traceability provides information to assess product quality, process

capability, and project progress against business goals
• In order to implement effective test monitoring and control, it is important
to establish and maintain traceability throughout test process between test
basis elements, testware associated with these elements (e.g., test
conditions, risks, test cases), test results, and detected defects

52
 Traceability between Test Basis and Testware,
2/4

• Accurate traceability supports coverage evaluation, so it is very useful if

measurable coverage criteria are defined in test basis
• Coverage criteria can function as key performance indicators to drive
activities that show to what extent test objectives have been achieved

53
 Traceability between Test Basis and Testware,
3/4

• Traceability of test cases to requirements can verify that requirements are

covered by test cases
• Traceability of test results to risks can be used to evaluate level of residual
risk in a test object

54
 Traceability between Test Basis and Testware,
4/4

• In addition to evaluating coverage, good traceability makes it possible to

determine impact of changes, facilitates test audits, and helps meet IT
governance criteria
• Good traceability also makes test progress and completion reports more
easily understandable by including status of test basis elements
• This can also assist in communicating technical aspects of testing to
stakeholders in an understandable manner
55
 Roles in Testing, 1/4

• Two principal roles in testing: Test management role and Testing role
• Activities and tasks assigned to these roles depend on factors such as
project and product context, skills of people in roles, and organization

56
 Roles in Testing, 2/4

• Test management role takes overall responsibility for test process, test team and leadership
of test activities
• Test management role is mainly focused on activities of test planning, test monitoring and
control and test completion
• The way in which test management role is carried out varies depending on context
• In Agile software development, some of test management tasks may be handled by Agile
team
• Tasks that span multiple teams or entire organization may be performed by test managers
outside of development team
57
 Roles in Testing, 3/4

• Testing role takes overall responsibility for engineering (technical) aspect of

testing
• Testing role is mainly focused on activities of test analysis, test design, test
implementation and test execution

58
 Roles in Testing, 4/4

• Different people may take on these roles at different times

• For example, test management role can be performed by a team leader, by
a test manager, by a development manager, etc.
• It is also possible for one person to take on roles of testing and test
management at same time

59
 Essential Skills and Good Practices in Testing

• Skill is the ability to do something well that comes from one’s knowledge,
practice and aptitude
• Good testers should possess some essential skills to do their job well
• Good testers should be effective team players and should be able to
perform testing on different levels of test independence

60
 Generic Skills Required for Testing, 1/2

• Testing knowledge (to increase effectiveness of testing, e.g., by using test techniques)
• Thoroughness, carefulness, curiosity, attention to details, being methodical (to identify defects,
especially the ones that are difficult to find)
• Good communication skills, active listening, being a team player (to interact effectively with all
stakeholders, to convey information to others, to be understood, and to report and discuss defects)
• Analytical thinking, critical thinking, creativity (to increase effectiveness of testing)
• Technical knowledge (to increase efficiency of testing, e.g., by using appropriate test tools)
• Domain knowledge (to be able to understand and to communicate with end users/business
representatives)
61
 Generic Skills Required for Testing, 2/2

• Testers are often the bearers of bad news

• It is a common human trait to blame the bearer of bad news
• This makes communication skills crucial for testers
• Communicating test results may be perceived as criticism of product and of its author
• Confirmation bias can make it difficult to accept information that disagrees with currently held beliefs
• Some people may perceive testing as a destructive activity, even though it contributes greatly to project
success and product quality
• To try to improve: Information about defects and failures should be communicated in a constructive way

62
 Whole Team Approach, 1/4

• Important skill for a tester is ability to work effectively in a team context and
to contribute positively to team goals
• Whole team approach – a practice coming from Extreme Programming –
builds upon this skill

63
 Whole Team Approach, 2/4

• In whole-team approach any team member with necessary knowledge and

skills can perform any task, and everyone is responsible for quality
• Team members share the same workspace (physical or virtual), as co-
location facilitates communication and interaction
• Whole team approach improves team dynamics, enhances communication
and collaboration within team, and creates synergy by allowing various skill
sets within the team to be leveraged for the benefit of project
64
 Whole Team Approach, 3/4

• Testers work closely with other team members to ensure that desired
quality levels are achieved
• Collaborating with business representatives to help them create suitable
acceptance tests and working with developers to agree on test strategy and
decide on test automation approaches
• Testers can thus transfer testing knowledge to other team members and
influence product development
65
 Whole Team Approach, 4/4

• Depending on context, whole team approach may not always be

appropriate
• For instance, in some situations, such as safety-critical, a high level of test
independence may be needed

66
 Independence of Testing, 1/5

• A certain degree of independence makes tester more effective at finding

defects due to differences between author’s and tester’s cognitive biases
• Independence is not, however, a replacement for familiarity, e.g.,
developers can efficiently find many defects in their own code

67
 Independence of Testing, 2/5

• Work products can be tested by their author (no independence), by author's

peers from the same team (some independence), by testers from outside
author's team but within the organization (high independence), or by testers
from outside organization (very high independence)

68
 Independence of Testing, 3/5

• For most projects, it is usually best to carry out testing with multiple levels
of independence (e.g., developers performing component and component
integration testing, test team performing system and system integration
testing, and business representatives performing acceptance testing)

69
 Independence of Testing, 4/5

• Main benefit of independence of testing is that independent testers are

likely to recognize different kinds of failures and defects compared to
developers because of their different backgrounds, technical perspectives,
and biases
• Moreover, an independent tester can verify, challenge, or disprove
assumptions made by stakeholders during specification and
implementation of system
70
 Independence of Testing, 5/5

• There are some drawbacks

• Independent testers may be isolated from development team, which may
lead to a lack of collaboration, communication problems, or an adversarial
relationship with development team
• Developers may lose a sense of responsibility for quality
• Independent testers may be seen as a bottleneck or be blamed for delays in
release
71
https://www.triad.co.uk/news/testing-vs-qa 72
 Tasks

• Read the Standard: ISO/IEC/IEEE 29119-1

• https://test-design.org/practical-exercises

73