Tahira Ali: Software Testing and Quality Assurance

Software Testing and Quality Assurance
Theory and Practice

Chapter 11
System Test Design
Tahira ali
Software Testing and QA Theory and Practice (Chapter 11: System Test Design) © Naik & Tripathy 1
Outline of the Chapter
• Test Design Factors
• Requirement Identification
• Test Objective Identification
• Modeling a Test Design Process
• Modeling Test Results
• Test Design Preparedness Metrics
• Test Case Design Effectiveness
Test Design Factors
The following factors must be taken into consideration during the
design of system tests
• Coverage metrics
• Effectiveness
• Productivity
• Validation
• Maintenance
• User skills
Requirement Identification
Statistical analysis conducted by Vinter reveals that out 1000 defects:
• 23.9 % requirement issues
• 24.3 % functionality
• 20.9% component structure
• 9.6 % data
• 4.3 % implementation
• 5.2 % integration
• 0.9 % architecture
• 6.9 % testing
• 4.3 % others
• Requirements are a description of the needs or desire of users that a
system suppose to implement
• Two major challenges in defining requirements:
– Ensure that the right requirements are captured which is essential for meeting
the expectations of the users
– Ensure that requirements are communicated unambiguously to the developers
and testers so that no surprise when the system is delivered
• Essential to have an unambiguous representation of the requirements
• The requirements must be available in a centralized place so that all
the stakeholders have the same interpretation of the requirements
A stakeholder is a person or an organization who influences a
system’s behavior or who is impacted by the system
• The state diagram of a simplified requirement life-cycle starting from Submit state
to the Closed state is shown in Figure 11.1and the schema in Table 11.1
• At each of these states certain actions are taken by the owner, and the requirement
is moved to the next state after the actions are completed
• A requirement may be moved to Decline state from any of the following state:
Open, Review, Assign, Implement, and Verification for several reasons
• A marketing manager may decide that the implementation of a particular
requirement may not generate revenue and may decline a requirement.
Figure 11.1: State transition diagram of a requirement

Table 11.1: Requirement schema field summary

Definition of requirements traceability:
The requirements traceability is the ability to describe and follow the

life of a requirement, in both forward and backward direction, i.e.,
from its origins, through its development and specification, to its
subsequent deployment and use, and through periods of ongoing
refinement and iteration in any of these phase
• One can generate traceability matrix from the requirement life-cycle
system, which gives them confidence about test coverage
• A traceability matrix allows one to find a two-way mapping between

requirements and test cases as follows
– From a requirement to a functional specification to specific tests which
exercise the requirements
– From each test case back to the requirement and functional specifications
• A traceability matrix finds two applications:

– To identify and track the functional coverage of a test
– To identify which test cases must be exercised or updated when a system
evolves
Characteristics of Testable Requirements
• One way to determine the requirement description is testable is as
follows:
• Take the requirement description “The system must perform X.”
• Encapsulate the requirement description to create a test objective:

“Verify that the system performs X correctly.”
• Review this test objective and find out if it is possible to execute it

assuming that the system and the test environment are available
• If the answer to the above question is yes, then the requirement

description is clear and detailed for testing purpose
• Otherwise, more work needs to be done to revise or supplement the
requirement description
• The following items must be analyzed during the review of
requirements:
– Safety
– Security
– Completeness
– Correctness
– Consistency
– Clarity
– Relevance
– Feasibility
– Verifiable
– Traceable
• A functional specification provides
– a precise description of the major functions the system must fulfill the
requirements
– explanation of the technological risks involved
– external interfaces with other software modules
– data flow such as flowcharts, transaction sequence diagrams, and finite-state
machines describing the sequence of activities
– fault handling, memory utilization and performance estimates
– any engineering limitation
• The following are the objectives that are kept in mind while
reviewing a functional specification:
– Achieving requirements
– Correctness
– Extensible
– Comprehensive
– Necessity
– Implementable
– Efficient
– Simplicity
– Consistency with existing components
– Limitations
Table 11.4: Characteristics of testable functional specification.

Test Objective Identification
• The question “What do I test?” must be answered with another
question: “What do I expect the system to do?”
• The first step in identifying the test objective is to read, understand,
and analyze the functional specification
• It is essential to have a background familiarity with the subject area,
the goals of the system, business processes, and system users, for a
successful analysis
• One must critically analyze requirements to extract the inferred
requirements that are embedded in the requirements
• An inferred requirement is one that a system is expected to support,
but not explicitly stated
• Inferred requirements need to be tested just like the explicitly stated
requirements
• The test objectives are put together to form a test group or a
subgroup after they have been identified
• A set of (sub) groups of test cases are logically combined to form a
larger group
• A hierarchical structure of test groups as shown in Figure 11.2 is
called a test suite
• It is necessary to identify the test groups based on test categories, and
refine the test groups into sets of test objectives
• Individual test cases are created for each test objective within the
subgroups
• Test groups may be nested to an arbitrary depth
• The test grouping may be used to aid system test planning and
execution that are discussed in Chapters 12 and 13, respectively
Figure 11.2: Test Suite Structure
Modeling a Test Design Process
• One test case is created for each test objective
• Each test case is designed as a combination of modular components called test
steps
• Test cases are clearly specified so that testers can quickly understand, borrow, and
re-use the test cases
• Figure 11.6 illustrate the life-cycle model of a test case in the form of a state
transition diagram
• One can easily implement a database of test cases using the test cases schema
shown in Table 11.6
Figure 11.6: State-transition diagram of a test case.

Modeling a Test Design Process
Table 11.6: Test case schema summary.

Modeling Test Results
• A test suite schema shown in Table 11.7 can be used for testing a
particular release
• The schema requires a test suite id, a title, an objective and a list of
test cases to be managed by the test suite
• The idea is to gather a selected number of released test cases and
repackage them to form a test suite for a new project
• The results of executing those test cases are recorded in a database
for gathering and analyzing test metrics
• The result of test execution is modeled by using a state-transition
diagram as shown in Figure 11.7
• The corresponding schema is given in Table 11.8
Table 11.7: Test suite field summary.
Table 11.8: Test result schema summary.
Figure 11.7: State transition diagram of a test case results
Test Design Preparedness Metrics
• Metrics are tracked
– To know if a test project is progressing according to schedule
– Plan the next project more accurately
• The following metrics are monitored

– Preparation status of test cases (PST)
– Average time spent (ATS) in test case design
– Number of available test (NAT) cases
– Number of planned test (NPT) cases
– Coverage of a test suite (CTS)
Test Case Design Effectiveness
• The objective of the test design effectiveness metric is to
– Measure the defect revealing ability of the test suite
– Use the metric to improve the test design process
• A metric commonly used in the industry to measure test case design
effectiveness is the Test Case Design Yield (TCDY)
NPT
TDCY  100%
NPT  Number of TCE
• TCE stand for Test Case Escaped

• During testing new defects are found for which no test cases had
been planned
• For these new defects, new test cases are designed, which is known
as Test Case Escaped metric
Theory and Practice
Chapter 10
Test Generation from FSM Models
Tahira ali
Software Testing and QA Theory and Practice (Chapter 10: Test Generation from FSM Models) © Naik & Tripathy 1
• State-oriented Model • Test Architectures
• Points of Control and Observation • Testing and Test Control Notation 3
• Finite-state Machine (FSM) (TTCN-3)
• Extended Finite-state Machines
• Test Generation from an FSM
• Unique Input/Output Sequence • Test Generation from EFSM Models
• Additional Coverage Criteria for
System Testing
State-oriented Model
• Software systems
– State-oriented (Examples: Operating Systems)
– Stateless (Example: compiler)
• State-oriented system: two parts
– Control portion
– Data portion
Figure 10.1: Spectrum of software systems.

State-oriented Model
• The control portion of a system is
often modeled as an FSM.
Figure 10.4: FSM model of a dual-boot • Figure 10.5: The interactions between
laptop computer. a system and its environment are
modeled as an FSM.
Points of Control and Observation
• A PCO is a point of interaction
between a system and its users. PCO IN/OUT
Hook IN
Keypad IN
Ringer OUT
Speaker OUT
Mouthpiece IN
Figure 10.6: PCOs on a telephone Table 10.1: PCOs for testing a telephone
PBX.
Points of Control and Observation
• Figure 5.7: FSM model of a private branch exchange (PBX)
Finite-state Machine (FSM)
• A finite-state machine
M = <S, I, O, s0, δ, λ>, where
• S is a set of states.
• I is a set of inputs.
• O is a set of outputs.
• s0 is the initial state.
• δ: S x I  S (next state function)
• λ: S x I  O (output function)
Figure 10.8: FSM model of a private

branch exchange (PBX).
Test Generation from an FSM
• Let M be the FSM model of a system and I M be its implementation.
• An important testing task is to confirm if I M behaves like M.
• Conformance testing: Ensure by means of testing that I M conforms
to its spec. M.
• A general procedure for conformance testing
– Derive sequences of state-transitions from M.
• Turn each state-transition into a test sequence.
• Test IM with a test sequence to observe whether or not IM possesses the
corresponding transition sequence.
– The conformance of IM with M can be verified by choosing enough state-
transition sequences.
Testing with State Verification
• Two functions associated with a state- • State verification with
transition – Unique Input/Output (UIO) sequences
– Output function: Easy to verify in the – Distinguishing sequences
TT method.(TT: It is a sequence of
state-transitions from the initial state – Characterizing sequences
to the final state.)
– Next-state function: Ignored in the TT
method (drawback of the TT method).
Figure 10.11: Conceptual model of a test

case with state verification.
Unique Input/Output Sequence
• Let X be an input sequence applied in a state s, and Y be the
corresponding output sequence.
• X/Y is a UIO sequence for s if no other state produces output
sequence Y in response to input X. Thus, X/Y is unique to s.
• Four assumptions about an FSM
– Completely specified
– Deterministic
– Reduced
– Strongly connected
Distinguishing Sequence
• Let X be an input sequence.
• X is a distinguishing sequence for an FSM, if each state produces a
unique (i.e. different) output sequence in response to X.
• Four assumptions about an FSM
– Completely specified
– Deterministic
– Reduced
– Strongly connected
Test Architectures
• A test architecture is a certain
configuration of
– an Implementation Under Test
(IUT)
– one or more test entities,
– one or two PCOs, and
– a communication service provide.
• Common test architectures
– Local Architecture
– Distributed Architecture
– Coordinated Architecture
– Remote Architecture
Figure 10.19: Abstraction of an (N)-Entity

in the OSI reference architecture
Test Architectures
• Local Architecture • Distributed Architecture
Figure 10.22: Local architecture. Figure 10.23: Distributed architecture.
Test Architectures
• Coordinated Architecture • Remote Architecture
Figure 10.24: Coordinated architecture. Figure 10.25: Remote architecture.
Testing and Test Control Notation 3 (TTCN-3)
• TTCN-3
– A language for specifying test cases.
– Predecessors were TTCN-1 and TTCN-2 (Tree and Tabular Combined
Notation)
– Standardized by ETSI (European Telecom. Standards Institute)
• Core features of TTCN-3
– Module
– Data types
– Templates
– Ports
– Components
– Test Cases
Extended Finite-state Machines
• Two conceptual components of a software system are
– Flow of control
– Manipulation of data
• Manipulate local variables.
• Start and stop timers.
• Create instances of processes.
• Compare values and make control-flow decisions.
• Access databases.
• There is a need for modeling a software system as an EFSM.
• We consider the Specification and Description Languages (SDL).
– The basic concepts in SDL are as follows:
• System
• Behavior
• Data
• Communication
Extended Finite-state Machines
Figure 10.35: Comparison of state-transitions of an FSM and an EFSM.
Test Generation from EFSM Models
• Let E be an EFSM and PE be a program implementing E.
• Goal: Test that PE behaves as E.
• Basic idea
– Phase 1: Identify a set of state-transition sequences such that each sequence
of state-transitions represents a common use sequence.
– Phase 2: Design a test case from each state-transition sequence.
• Phase 1
– Pay attention to the following.
• Perform tests to ensure that the system under test (SUT) produces
expected sequences of outcomes in response to input sequences.
• Perform tests to ensure that the system under test takes the right actions
when a timeout occurs.
• Perform tests to ensure that the system under test has appropriately
implemented other task blocks, such as resource allocation, database
accesses, etc.
– Coverage criteria: state coverage and transition coverage.
• Phase 2
– Transform the inputs and outputs in a transition tour into outputs and
inputs, respectively.
– Augment the above core test behavior with “otherwise” events to be able to
handle exception events.
– Augment the above test behavior with timers.
Figure 10.36: Controlled access to a door.
Additional Coverage Criteria for System Testing
• PCO coverage
– Select test cases such that the SUT receives an event at each input PCO and
produces an event at each output PCO.
• Sequence of events at PCOs
– Select test cases such that common sequences of inputs and outputs occur at
the PCOs.
• Events occurring in different contexts
– An event generated at a PCO may have different meanings at different times.
• Inopportune events
– Inopportune events are normal events which occur at an inappropriate time.
Summary
• Software systems • TTCN-3
– Stateless – Data types
– State-oriented – Modules
• Control portion is modeled by an – Ports
FSM or an EFSM. – Templates
• Testing FSM based systems • Testing EFSM based systems
– Transition tour method – Identify transition sequences
– State verification method – Turn each sequence into a test case
• State verification techniques • Input/output  Output/input
– Distinguishing sequence • “Otherwise” events
– UIO sequence • Timers
– Characterizing sequence – Coverage metrics
• Coverage metrics • PCO coverage
– State coverage • Sequences of events at each PCO
– State-transition coverage • Events occurring in different contexts
• Test architectures • Inopportune events
– Local
– Distributes
– Coordinated
– Remote
Theory and Practice
Chapter 8
System Test Categories
Delivered by Tahira ali
Software Testing and QA Theory and Practice (Chapter 8: System Test Categories) © Naik & Tripathy 1
• Taxonomy of System Tests
• Basic Tests
• Functionality Tests
• Robustness Tests
• Interoperability Tests
• Performance Tests
• Scalability Tests
• Stress Tests
• Load and Stability Tests
• Regression Tests
• Documentation Tests
• Regulatory Tests
– Software Safety
– Safety Assurance
Taxonomy of System Tests
Figure 8.1: Types of system tests

• Basic tests provide an evidence that the system can be installed,
configured and be brought to an operational state
• Functionality tests provide comprehensive testing over the full

range of the requirements, within the capabilities of the system
• Robustness tests determine how well the system recovers from

various input errors and other failure situations
• Inter-operability tests determine whether the system can inter-

operate with other third party products
• Performance tests measure the performance characteristics of the

system, e.g., throughput and response time, under various conditions
• Scalability tests determine the scaling limits of the system, in terms
of user scaling, geographic scaling, and resource scaling
• Stress tests put a system under stress in order to determine the
limitations of a system and, when it fails, to determine the manner in
which the failure occurs
• Load and Stability tests provide evidence that the system remains
stable for a long period of time under full load
• Reliability tests measure the ability of the system to keep operating
for a long time without developing failures
• Regression tests determine that the system remains stable as it
cycles through the integration of other subsystems and through
maintenance tasks
• Documentation tests ensure that the system’s user guides are
accurate and usable
Basic Tests
Figure 8.2: Types of basic tests

• Boot: Boot tests are designed to verify that the system can boot up its software
image (or, build) from the supported boot options
• Upgrade/Downgrade: Upgrade/downgrade tests are designed to verify that the

system software can be upgraded or downgraded (rollback) in a graceful manner
Basic Tests
• Light Emitting Diode: The LED (Light Emitting Diode) tests are
designed to verify that the system LED status indicators functioning
as desired
• Diagnostic: Diagnostic tests are designed to verify that the hardware

components (or, modules) of the system are functioning as desired
– Power-On Self Test
– Ethernet Loop Back Test
– Bit Error Test
• Command line Interface: Command Line Interface (CLI) tests are

designed to verify that the system can be configured
Functionality Tests
Figure 8.3: Types of functionality tests

Functionality Tests
• Communication Systems Tests
– These tests are designed to verify the implementation of the communication
systems as specified in the customer requirements specification
– Four types of communication systems tests are recommended
• Basis interconnection tests
• Capability tests
• Behavior tests
• System resolution tests
• Module Tests
– Module Tests are designed to verify that all the modules function individually
as desired within the systems
– The idea here is to ensure that individual modules function correctly within the
whole system.
• For example, an Internet router contains modules such as line cards,
system controller, power supply, and fan tray. Tests are designed to verify
each of the functionalities
Functionality Tests
• Logging and Tracing Tests
– Logging and Tracing Tests are designed to verify the configurations and
operations of logging and tracing
– This also includes verification of “flight data recorder: non-volatile Flash
memory” logs when the system crashes
• Element Management Systems (EMS) Tests
– EMS tests verifies the main functionalities, which are to manage, monitor and
upgrade the communication systems network elements
– It includes both EMS client and EMS servers functionalities
Note: Read the SNMP example described in the book
• Management Information Base (MIB) Tests
– MIB tests are designed to verify
• Standard MIBs including MIB II
• Enterprise MIBs specific to the system
Note: Read the SNMP example described in the book
Functionality Tests
• Graphical User Interface Tests
– Tests are designed to look-and-feel the interface to the users of an application
system
– Tests are designed to verify different components such as icons, menu bars,
dialog boxes, scroll bars, list boxes, and radio buttons
– The GUI can be utilized to test the functionality behind the interface, such as
accurate response to database queries
– Tests the usefulness of the on-line help, error messages, tutorials, and user
manuals
– The usability characteristics of the GUI is tested, which includes the following
• Accessibility: Can users enter, navigate, and exit with relative ease?
• Responsiveness: Can users do what they want and when they want in a
way that is clear?
• Efficiency: Can users do what they want to with minimum number of
steps and time?
• Comprehensibility: Do users understand the product structure with a
minimum amount of effort?
Functionality Tests
• Security Tests
– Security tests are designed to verify that the system meets the security
requirements
• Confidentiality
– It is the requirement that data and the processes be protected from
unauthorized disclosure
• Integrity
– It is the requirement that data and process be protected from unauthorized
modification
• Availability
– It is the requirement that data and processes be protected form the denial of
service to authorized users
– Security test scenarios should include negative scenarios such as misuse and
abuse of the software system
Functionality Tests
• Feature Tests
– These tests are designed to verify any additional functionalities which are
defined in requirement specification but not covered in the functional category
discussed
– Examples
• Data conversion testing
• Cross-functionality testing
Robustness Tests
Robustness means how much sensitive a system is to erroneous input and changes
its operational environment
Tests in this category are designed to verify how gracefully the system behaves in
error situations and in a changed operational environment
• On-line insertion and removal
– On-line Insertion and Removal (OIR) tests are designed to ensure that on-line
insertion and removal of modules, incurred during both idle and heavy load
operations, are gracefully handled and recovered
• High Availability
– The concept of high availability is also known as fault tolerance
– High availability tests are designed to verify the redundancy of individual
modules, including the software that control these modules
Interoperability Tests
• Tests are designed to verify the ability of the system to inter-operate
with third party products
• The re-configuration activities during interoperability tests is known

as configuration testing
• Another kind of inter-operability tests is called (backward)

compatibility tests
– Compatibility tests verify that the system works the same way across different
platforms, operating systems, data base management systems
– Backward compatibility tests verify that the current software build flawlessly
works with older version of platforms
Performance Tests
• Tests are designed to determine the performance of the actual system
compared to the expected one
• Tests are designed to verify response time, execution time,
throughput, resource utilization and traffic rate
• For example, if the objective is to evaluate the response time, then
one needs to capture
– End-to-end response time (as seen by external user)
– CPU time
– Network connection time
– Database access time
– Network connection time
– Waiting time
Scalability Tests
• Tests are designed to verify that the system can scale up to its
engineering limits
• Scaling tests are conducted to ensure that the system response time
remains the same, or increases by a small amount, as the number of
users are increased.
• There are three major causes of these limitations:
– data storage limitations
– network bandwidth limitations
– speed limit
Stress Tests
• The goal of stress testing is to evaluate and determine the behavior of
a software component while the offered load is in excess of its
designed capacity
• The system is deliberately stressed by pushing it to and beyond its
specified limits
• Stress tests are targeted to bring out the problems associated with one
or more of the following:
– Memory leak
– Buffer allocation and memory carving
Load and Stability Tests
• Tests are designed to ensure that the system remains stable for a long
period of time under full load
• When a large number of users are introduced and applications that
run for months without restarting, a number of problems are likely to
occur:
– the system slows down
– the system encounters functionality problems
– the system crashes altogether
Reliability Tests
• Reliability tests are designed to measure the ability of the system to
remain operational for long periods of time.
• The reliability of a system is typically expressed in terms of mean
time to failure (MTTF)
• The average of all the time intervals between successive failures is
called the MTTF
• After a failure is observed, the developers analyze and fix the defects,
which consumes some time – let us call this interval the repair time.
• The average of all the repair times is known as the mean time to
repair (MTTR)
• Now we can calculate a value called mean time between failure
(MTBF) as MTBF = MTTF + MTTR
Regression Tests
• In this category, new tests are not designed, instead, test cases are
selected from the existing pool and executed
• The main idea in regression testing is to verify that no defect has
been introduced into the unchanged portion of a system due to
changes made elsewhere in the system
• During system testing, many defects are revealed, and the code is
modified to fix those defects.
Documentation Tests
• Documentation testing means verifying the technical accuracy and
readability of the user manuals, tutorials and the on-line help
• Documentation testing is performed at three levels:

– Read test: In this test a documentation is reviewed for clarity, organization,
flow, and accuracy without executing the documented instructions on the
system
– Hands-on test: Exercise the on-line help and verify the error messages to
evaluate their accuracy and usefulness.
– Functional test: Follow the instructions embodied in the documentation to

verify that the system works as it has been documented.
Software Safety
• A hazard is a state of a system or a physical situation which when
combined with certain environmental conditions, could lead to an
accident or mishap
• An accident or mishap is an unintended event or series of events that

results in death, injury, illness, damage or loss of property, or harm to
the environment
• Software safety is defined in terms of hazards
Safety Assurance
• There are two basic tasks performed by a safety assurance
engineering team:
– Provide methods for identifying, tracking, evaluating, and eliminating hazards

associated with a system
– Ensure that safety is embedded into the design and implementation in a timely
and cost effective manner, such that the risk created by the user/operator error
is minimized
Theory and Practice
Chapter 7
System Integration Testing
Delivered by
Tahira ali
Software Testing and QA Theory and Practice (Chapter 7: System Integration Testing) © Naik & Tripathy 1
• The Concept of Integration Testing
• Different Types of Interfaces
• Different Types of Interface Errors
• Granularity of System Integration Testing
• System Integration Techniques: Incremental, Top-down, Bottom-up,
and Sandwich and Big-bang
• Software and Hardware Integration
• Hardware Design Verification Tests
• Hardware and Software Compatibility Matrix
• Test Plan for System Integration
• Off-the-self Component Integration
• Off-the-shelf Component Testing
• Built-in Testing
The Concept of Integration Testing
• A software module is a self-contained element of a system
• Modules are individually tested commonly known as unit testing
• Next major task is to put the modules, i.e., pieces together to
construct the complete system
• Construction of a working system from the pieces is not a
straightforward task because of numerous interface errors
• The objective of system integration testing (SIT) is to build a
“working” version of the system
– Putting modules together in an incremental manner
– Ensuring that the additional modules work as expected without disturbing the
functionalities of the modules already put together
• Integration testing is said to be complete when
– The system is fully integrated togethr
– All the test cases have been executed
– All the severe and moderated defects found have been fixed
The Concept of Integration Testing
The major advantages of conducting SIT are as follows:
• Defects are detected early
• It is easier to fix defects detected earlier
• We get earlier feedback on the health and acceptability of the

individual modules and on the overall system
• Scheduling of defect fixes is flexible, and it can overlap with

development
Different Types of Interfaces
Three common paradigms for interfacing modules:
• Procedure call interface
• Shared memory interface
• Message passing interface
The problem arises when we “put modules together” because of interface errors
Interface errors
Interface errors are those that are associated with structures existing outside the
local environment of a module, but which the module uses
Different Types of Interface Errors
• Inadequate error processing
• Construction
• Additions to error processing

• Inadequate functionality
• Inadequate post-processing
• Location of functionality
• Inadequate interface support

• Changes in functionality
• Initialization/value errors
• Added functionality
• Validation od data constraints

• Misuse of interface
• Timing/performance problems
• Misunderstanding of interface
• Coordination changes
• Data structure alteration
• Hardware/software interfaces
Granularity of System Integration Testing
System Integration testing is performed at different levels of granularity
• Intra-system testing
– This form of testing constitutes low-level integration testing with the objective
of combining the modules together to build a cohesive system
• Inter-system testing
– It is a high-level testing phase which requires interfacing independently tested
systems
• Pairwise testing
– In pairwise integration, only two interconnected systems in an overall system
are tested at a time
– The purpose of pairwise testing is to ensure that two systems under
consideration can function together, assuming that the other systems within the
overall environment behave as expected
System Integration Techniques
Common approaches to perform system integration testing
• Incremental
• Top-down
• Bottom-up
• Sandwich
• Big-bang
Pre-requisite
A module must be available to be integrated
A module is said to available for combining with other modules when the
module’s check-in request form is ready
Incremental
• A software image is a compiled software binary
• A build is an interim software image for internal testing within an
organization
• Constructing a build is a process by which individual modules are
integrated to form am interim software image.
• The final build is a candidate for system testing
• Constructing a software image involves the following activities
– Gathering the latest unit tested, authorized versions of modules
– Compiling the source code of those modules
– Checking in the compiled code to the repository
– Linking the compiled modules into subassemblies
– Verifying that the subassemblies are correct
– Exercising version control
Incremental
• Integration testing is conducted in an incremental manner as a series
of test cycles
• In each test cycle, a few more modules are integrated with an existing
and tested build to generated larger builds
• The complete system is built, cycle by cycle until the whole system is
operational for system-level testing.
• The number of SIT cycles and the total integration time are
determined by the following parameters:
– Number of modules in the system
– Relative complexity of the module (cyclomatic complexity)
– Relative complexity of the interfaces between the modules
– Number of modules needed to be clustered together in each test cycle
– Whether the modules to be integrated have been adequately tested before
– Turnaround time for each test-debug-fix cycle
Incremental
• A release note containing the following information accompanies a
build.
– What has changed since the last build?
– What outstanding defects have been fixed?
– What are the outstanding defects in the build?
– What new modules, or features, have been added?
– What existing modules, or features, have been enhanced, modified, or deleted?
– Are there any areas where unknown changes may have occurred?
• A test strategy is created for each new build and the following issues
are addressed while planning a test strategy
– What test cases need to be selected from the SIT test plan?
– What previously failed test cases should now be re-executed in order to test the
fixes in the new build?
– How to determine the scope of a partial regression tests?
– What are the estimated time, resource demand, and cost to test this build?
Incremental
Creating a daily build is very popular among many organization
• It facilitates to a faster delivery of the system

• It puts emphasis on small incremental testing
• It steadily increases number of test cases
• The system is tested using automated, re-usable test cases
• An effort is made to fix the defects that were found within 24 hours
• Prior version of the build are retained for references and rollback
• A typical practice is to retain the past 7-10 builds
Top-down
• Module A has been decomposed into modules
B, C, and D
• Modules B, D, E, F, and G are terminal
modules
• First integrate modules A and B using stubs
C` and D` (represented by grey boxes)
• Next stub D` has been replaced with its actual
instance D
Figure 7.1: A module hierarchy with • Two kinds of tests are performed:
three levels and seven modules – Test the interface between A and D
– Regression tests to look for interface defects
between A and B in the presence of module D
Figure 7.2: Top-down integration of

modules A and B
modules A, B and D
Top-down
• Stub C` has been replaced with the
actual module C, and new stubs E`,
F`, and G`
• Perform tests as follows:
– first, test the interface between A and
C;
– second, test the combined modules A,
B, and D in the presence of C Figure 7.5: Top-down integration of
• The rest of the process depicted in the modules A, B, C, D and E
right hand side figures.

Figure 7.4: Top-down integration of modules A, B, C, D, E and F
modules A, B, D and C
Top-down
Advantages
• The SIT engineers continually observe system-level functions as the
integration process continue
• Isolation of interface errors becomes easier because of the
incremental nature of the top-down integration
• Test cases designed to test the integration of a module M are reused
during the regression tests performed after integrating other modules
Disadvantages
• It may not be possible to observe meaningful system functions
because of an absence of lower level modules and the presence of
stubs.
• Test case selection and stub design become increasingly difficult
when stubs lie far away from the top-level module.
Bottom-up
• We design a test driver to integrate
lowest-level modules E, F, and G
• Return values generated by one module
is likely to be used in another module
• The test driver mimics module C to
integrate E, F, and G in a limited way.
• The test driver is replaced with actual Figure 7.8: Bottom-up integration of
module , i.e., C. module E, F, and G
• A new test driver is used
• At this moment, more modules such as
B and D are integrated
• The new test driver mimics the behavior
of module A
• Finally, the test driver is replaced with
module A and further test are performed
Figure 7.9: Bottom-up integration of

module B, C, and D with F, F, and G
Bottom-up
Advantages
• One designs the behavior of a test driver by simplifying the behavior
of the actual module
• If the low-level modules and their combined functions are often
invoked by other modules, then it is more useful to test them first so
that meaningful effective integration of other modules can be done
Disadvantages
• Discovery of major faults are detected towards the end of the
integration process, because major design decision are embodied in
the top-level modules
• Test engineers can not observe system-level functions from a partly
integrated system. In fact, they can not observe system-level
functions until the top-level test driver is in place
Big-bang and Sandwich
Big-bang Approach
• First all the modules are individually tested
• Next all those modules are put together to construct the entire system
which is tested as a whole
Sandwich Approach
• In this approach a system is integrated using a mix of top-down,
bottom-up, and big-bang approaches
• A hierarchical system is viewed as consisting of three layers
• The bottom-up approach is applied to integrate the modules in the
bottom-layer
• The top layer modules are integrated by using top-down approach
• The middle layer is integrated by using the big-bang approach after
the top and the bottom layers have been integrated
Software and Hardware Integration
• Integration is often done in an iterative manner
• A software image with a minimal number of core modules is loaded

on a prototype hardware
• A small number of tests are performed to ensure that all the desired
software modules are present in the build
• Next, additional tests are run to verify the essential functionalities
• The process of assembling the build, loading on the target hardware,

and testing the build continues until the entire product has been
integrated
Hardware Design Verification Tests
A hardware engineering process consists of four phases
• Planning and specification
• Design, prototype implementation, and testing
• Integration with the software system
• Manufacturing, distribution and field service
A hardware Design Verification Test (DVT) plan is prepared and

executed by the hardware group before the integration with software
system
Hardware Design Verification Tests
The main hardware tests are aa follows:
• Diagnostic Test
• Electrostatic Discharge Test
• Electromagnetic Emission Test
• Electrical Test
• Thermal Test
• Environment Test
• Equipment Handling and Packaging Test
• Acoustic Test
• Safety Test
• Reliability Test
Test Plan for System Integration
Table 7.4: A framework for system integration entry criteria
Table 7.5: A framework for system integration exit criteria
Categories of System Integration Tests:
• Interface integrity
– Internal and external interfaces are tested as each module is integrated
• Functional validity
– Tests to uncover functional errors in each module after it is integrated
• End-to-end validity
– Tests are designed to ensure that a completely integrated system works together
from end-to-end
• Pairwise validity
– Tests are designed to ensure that any two systems work properly when
connected by a network
• Interface stress
– Tests are designed to ensure that the interfaces can sustain the load
• System endurance
– Tests are designed to ensure that the integrated system stay up for weeks
Off-the-self Component Integration
Organization occasionally purchase off-the-self (OTS) components
from vendors and integrate them with their own components
Useful set of components that assists in integrating actual components:
• Wrapper: It is a piece of code that one builds to isolate the

underlying components from other components of the system
• Glue: A glue component provides the functionality to combine
different components
• Tailoring: Components tailoring refers to the ability to enhance the
functionality of a component
– Tailoring is done by adding some elements to a component to enrich it with a
functionality not provided by the vendor
– Tailoring does not involve modifying the source code of the component
Off-the-shelf Component Testing
OTS components produced by the vendor organizations are known as
commercial off-the-shelf (COTS) components
A COTS component is defined as:
A unit of composition with contractually specified interfaces and
explicit context dependencies only. A software component can be
deployed independently and is subject to composition by third parties
Three types of testing techniques are use to determine the suitability of
a COTS component:
• Black-box component testing: This is used to determine the quality
of the component
• System-level fault injection testing: This is used to determine how
well a system will tolerate a failing component
• Operational system testing: This kind of tests are used to determine
the tolerance of a software system when the COTS component is
functioning correctly
Built-in Testing
• Testability is incorporated into software components
• Testing and maintenance can be self-contained

– Normal mode
• The built-in test capabilities are transparent to the component user
• The component does not differ from other non-built-in testing enabled
components
– Maintenance mode
• The component user can test the component with the help of its built-in
testing features
• The component user can invoke the respective methods of the
component, which execute the test, evaluate autonomously its results,
and output the test summary

Tahira Ali: Software Testing and Quality Assurance

Uploaded by

Tahira Ali: Software Testing and Quality Assurance

Uploaded by

Software Testing and Quality Assurance

Theory and Practice

Figure 11.1: State transition diagram of a requirement

Table 11.1: Requirement schema field summary

Definition of requirements traceability:

The requirements traceability is the ability to describe and follow the

• A traceability matrix allows one to find a two-way mapping between

• A traceability matrix finds two applications:

• Encapsulate the requirement description to create a test objective:

• Review this test objective and find out if it is possible to execute it

• If the answer to the above question is yes, then the requirement

Table 11.4: Characteristics of testable functional specification.

Figure 11.2: Test Suite Structure

Figure 11.6: State-transition diagram of a test case.

Table 11.6: Test case schema summary.

Table 11.7: Test suite field summary.

Table 11.8: Test result schema summary.

Figure 11.7: State transition diagram of a test case results

• The following metrics are monitored

• TCE stand for Test Case Escaped

Figure 10.1: Spectrum of software systems.

• Figure 5.7: FSM model of a private branch exchange (PBX)

Figure 10.8: FSM model of a private

Figure 10.11: Conceptual model of a test

Figure 10.19: Abstraction of an (N)-Entity

Figure 10.22: Local architecture. Figure 10.23: Distributed architecture.

Figure 10.24: Coordinated architecture. Figure 10.25: Remote architecture.

Figure 10.35: Comparison of state-transitions of an FSM and an EFSM.

Figure 10.36: Controlled access to a door.

Delivered by Tahira ali

Figure 8.1: Types of system tests

• Functionality tests provide comprehensive testing over the full

• Robustness tests determine how well the system recovers from

• Inter-operability tests determine whether the system can inter-

• Performance tests measure the performance characteristics of the

Figure 8.2: Types of basic tests

• Upgrade/Downgrade: Upgrade/downgrade tests are designed to verify that the

• Diagnostic: Diagnostic tests are designed to verify that the hardware

• Command line Interface: Command Line Interface (CLI) tests are

Figure 8.3: Types of functionality tests

• The re-configuration activities during interoperability tests is known

• Another kind of inter-operability tests is called (backward)

• Documentation testing is performed at three levels:

– Functional test: Follow the instructions embodied in the documentation to

• An accident or mishap is an unintended event or series of events that

• Software safety is defined in terms of hazards

– Provide methods for identifying, tracking, evaluating, and eliminating hazards

• Defects are detected early

• It is easier to fix defects detected earlier

• We get earlier feedback on the health and acceptability of the

• Scheduling of defect fixes is flexible, and it can overlap with

• Procedure call interface

• Shared memory interface

• Message passing interface

• Additions to error processing

• Inadequate interface support

• Validation od data constraints

• It facilitates to a faster delivery of the system

Figure 7.2: Top-down integration of

Figure 7.6: Top-down integration of

Figure 7.9: Bottom-up integration of

• A software image with a minimal number of core modules is loaded

• Next, additional tests are run to verify the essential functionalities

• The process of assembling the build, loading on the target hardware,

A hardware Design Verification Test (DVT) plan is prepared and

Table 7.4: A framework for system integration entry criteria

Table 7.5: A framework for system integration exit criteria

Useful set of components that assists in integrating actual components:

• Wrapper: It is a piece of code that one builds to isolate the

• Testing and maintenance can be self-contained

You might also like