Unit-Iv Validationactivities
Unit-Iv Validationactivities
Unit-Iv Validationactivities
VALIDATIONACTIVITIES
Unit testing, Integration Testing, Function testing, system testing, and acceptance testing.
Regression testing: Progressives Vs regressive testing, Regression testability, Objectives
of regression testing, when regression testing done? Regression testing types, Regression
testing techniques.
VALIDATION ACTIVITIES:
In this chapter, we will discuss the details of all validation concepts. Once we have
verified every activity/stage in SDLC as shown in Fig. 7.1, the software or module built
up till this stage should also be validated. By validation, we mean that the module that has
been prepared till now is in conformance with the requirements which were set initially in
the SRS or user manual.
As shown in Fig. 7.1, every validation testing focuses on a particular SDLC phase and
thereby focuses on a particular class of errors. We can see a one-to-one correspondence
between development and testing processes. For example, the purpose of unit validation
testing is to find discrepancies between the developed module’s functionality and its
requirements and interfaces specified in the SRS. Similarly, the purpose of system
validation testing is to explore whether the product is consistent with its original
objectives. The advantage of this structure of validation testing is that it avoids redundant
testing and prevents one from overlooking large classes of errors.
Software validation is achieved through a series of black-box tests that demonstrate
conformity with requirements [7]. A test plan outlines the classes of tests to be conducted
and a test procedure defines specific test cases that will be used to demonstrate
conformity with requirements. Both the plan and procedure are designed to ensure that all
functional requirements are satisfied, all behavioural characteristics are achieved, all
performance requirements are attained, documentation is correct and human-engineered,
and other requirements are met.
7.1 UNIT VALIDATION TESTING
Since unit is the smallest building block of the software system, it is the fi rst piece of system to
be validated. Before we validate the entire software, units or modules must be validated. Unit
testing is normally considered an adjunct to the coding step. However, it has been discussed that
testing with the code of a unit is called the verification step. Units must also be validated to
ensure that every unit of software has been built in the right manner in conformance with user
requirements. Unit tests ensure that the software meets at least a baseline level of functionality
prior to integration and system testing. While developing the software, if the developer detects
and removes the bug, it offers significant savings of time and costs. This type of testing is largely
based on black-box techniques.
Though software is divided into modules but a module is not an isolated entity. The module
under consideration might be getting some inputs from another module or the module is calling
some other module. It means that a module is not independent and cannot be tested in isolation.
While testing the module, all its interfaces must be simulated if the interfaced modules are not
ready at the time of testing the module under consideration. Discussed below are the types of
interface modules which must be simulated, if required, to test a module.
Drivers
Suppose a module is to be tested, wherein some inputs are to be received from another module.
However, this module which passes inputs to the module to be tested is not ready and under
development. In such a situation, we need to simulate the inputs required in the module to be
tested. For this purpose, a main program is prepared, wherein the required inputs are either hard-
coded or entered by the user and passed on to the module under test. This module where the
required inputs for the module under test are simulated for the purpose of module or unit testing
is known as a driver module. The driver module may print or interpret the results produced by
the module under testing.
For example, see the design hierarchy of the modules, as shown in Fig. 7.2. Suppose module B is
under test. In the hierarchy, module A is a superordinate of module B. Suppose module A is not
ready and B has to be unit tested. In this case, module B needs inputs from module A. Therefore,
a driver module is needed which will simulate module A in the sense that it passes the required
inputs to module B and acts as a main program for module B in which its being called, as shown
in Fig. 7.3.
Therefore, it can be said that a test driver is supporting the code and data used to provide an
environment for testing a part of a system in isolation. In fact, it drives the unit being tested by
creating necessary ‘inputs’ required for the unit and then invokes the unit. A test driver may take
inputs in the following form and call the unit to be tested.
Thus, a driver can be defined as a software module which is used to invoke a module under test
and provide test inputs, control and monitor execution, and report test results or most
simplistically a line of code that calls a method and passes a value to that method.
Projects where commercial drivers are not available, specialized drivers need to be developed.
This happens mainly in defence projects where projects are developed for a special application.
Stubs The module under testing may also call some other module which is not ready at the time
of testing. Therefore, these modules need to be simulated for testing. In most cases, dummy
modules instead of actual modules, which are not ready, are prepared for these subordinate
modules. These dummy modules are called stubs.
Thus, a stub can be defined as a piece of software that works similar to a unit which is referenced
by the unit being tested, but it is much simpler than the actual unit. A stub works as a ‘stand-in’
for the subordinate unit and provides the minimum required behaviour for that unit.
For example, consider Fig. 7.2 again. Module B under test needs to call module D and module E.
But they are not ready. So there must be some skeletal structure in their place so that they act as
dummy modules in place of the actual modules. Therefore, stubs are designed for module D and
module E, as shown in Fig. 7.4.
Stubs have the following characteristics:
Stub is a place holder for the actual module to be called. Therefore, it is not designed
with the functionalities performed by the actual module. It is a reduced implementation of
the actual module. „
It does not perform any action of its own and returns to the calling unit (which is being
tested). „
We may include a display instruction as a trace message in the body of stub. The idea is
that the module to be called is working fi ne by accepting the input parameters. „
A constant or null must be returned from the body of stub to the calling module. „
Stub may simulate exceptions or abnormal conditions, if required
Benefits of Designing Stubs and Drivers
The benefits of designing drivers and stubs (see Fig. 7. 5) is that a unit will work/ behave in the
simulated environment as in the actual software environment. Now a unit test case can be written
against the interface and the unit will still work properly once the drivers and stubs have been
placed correctly.
The benefits of designing stubs and drivers are:
Stubs allow the programmer to call a method in the code being developed, even if the
method does not have the desired behavior yet. „
By using stubs and drivers effectively, we can cut down our total debugging and testing
time by testing small parts of a program individually, helping us to narrow down
problems before they expand.
Stubs and drivers can also be an effective tool for demonstrating progress in a business
environment. For example, if you are to implement four specific methods in a class by
the end of the week, you can insert stubs for any method and write a short driver program
so that you can demonstrate to your manager or client that the requirement has been met.
As you continue, you can replace the stubs and drivers with the real code to finish your
program.
However, drivers and stubs represent overheads also. Overhead of designing them may increase
the time and cost of the entire software system. Therefore, they must be designed simple to keep
overheads low. Stubs and drivers are generally prepared by the developer of the module under
testing. Developers use them at the time of unit verification. But they can also be used by any
other person who is validating the unit.
Example 7.1
7.2 INTEGRATION TESTING
In the modular design of a software system where the system is composed of different modules,
integration is the activity of combining the modules together when all the modules have been
prepared. Integration of modules is according to the design of software specified earlier.
Integration aims at constructing a working software system. But a working software demands
full testing and thus, integration testing comes into the picture.
Why do we need integration testing? When all modules have been verified independently, then
why is integration testing necessary? As discussed earlier, modules are not standalone entities.
They are a part of a software system which comprises of many interfaces. Even if a single
interface is mismatched, many modules may be affected.
According to the Big-Bang theory, if all unit tested modules are integrated in this
example, then for unit testing of all the modules independently, we require four drivers
and seven stubs. This count will grow according to the size of the system.
2. Actual modules are not interfaced directly until the end of the software system.
3. It will be difficult to localize the errors since the exact location of bugs cannot be
found easily.
One method that may work is to borrow the good features of both the approaches.
According to the big-bang method, all modules should be unittested independently as
they are developed. In this way, there is parallelism. As soon as one module is ready, it
can be combined and tested again in the integrated environment according to the
incremental integration testing.
Bottom-up integration can be considered as the opposite of top-down approach. Unlike top-down
strategy, this strategy does not require the architectural design of the system to be complete.
Thus, bottom-up integration can be performed at an early stage in the developmental process. It
may be used where the system reuses and modifi es components from other systems.
Practical Approach for Integration Testing There is no single strategy adopted for industry
practice. For integrating the modules, one cannot rely on a single strategy. There are situations
depending on the project in hand which will force to integrate the modules by combining top-
down and bottom-up techniques. This combined approach is sometimes known as sandwich
integration testing.
Selection of an integration testing strategy depends on software characteristics and sometimes
project schedules. In general, sandwich testing strategy that uses top-down tests for upper levels
of the program structure with bottom-up tests for subordinate levels is the best compromise.
The practical approach for adopting sandwich testing is driven by the following factors:
Priority
There may be situations which force us to prioritize the modules to be integrated. Some of the
prioritizing guidelines [7] are given below:
„ Integration should follow the lines of first putting together those subsystems that are of
great concern or with more important requirements. „
This prioritization might dictate top-down integration if the module has a high level of
control on its subordinate modules. „
The modules with more user interfaces should be tested first, as they are more error-
prone. „
If the customer is anxious to see a running program early in the testing phase, then top-
down integration will help. „
In another situation, the machine interface and performance might be of special interest.
Here, bottom-up integration would be dictated. With this approach, we stand better
chance of experiencing a high degree of concurrency during our integration. „
The module whose cyclomatic complexity is high must be tested first.
Availability
In practice, the availability of modules matter the most. The module which is ready to be
integrated, will be integrated and tested fi rst. We should not wait according to top-down or
bottom-up sequence, but use the availability of modules. Different development schedules for
different modules of the system force us to integrate with available modules only
An effective function test cycle must have a defined set of processes and deliverables. The
primary processes/deliverables for requirements based function test are discussed below.
Test planning
During planning, the test leader with assistance from the test team defi nes the scope, schedule,
and deliverables for the function test cycle. He delivers a test plan (document) and a test
schedule (work plan)—these often undergo several revisions during the testing cycle.
Partitioning/functional decomposition
Functional decomposition of a system (or partitioning) is the breakdown of a system into its
functional components or functional areas. Another group in the organization may take
responsibility for the functional decomposition (or model) of the system, but the testing
organization should still review this deliverable for completeness before accepting it into the test
organization. If the functional decompositions or partitions have not been defined or are deemed
insufficient, then the testing organization will have to take responsibility for creating and
maintaining the partitions.
Requirement definition
The testing organization needs specified requirements in the form of proper documents to
proceed with the function test. These requirements need to be itemized under an appropriate
functional partition. Test case design
A tester designs and implements a test case to validate that the product performs in accordance
with the requirements. These test cases need to be itemized under an appropriate functional
partition and mapped/ traced to the requirements being tested.
Traceability matrix formation
Test cases need to be traced/mapped back to the appropriate requirement. A function coverage
matrix is prepared. This matrix is a table, listing specific functions to be tested, the priority for
testing each function, and test cases that contain tests for each function. Once all the aspects of a
function have been tested by one or more test cases, then the test design activity for that function
can be considered complete. This approach gives a more accurate picture of the application when
coverage analysis is done. For example, in Table 7.4, function F2 test cases must be executed, as
its priority is highest; and through the function coverage matrix, we can track which functions
are being tested through which test cases.
System testing is actually a series of different tests to test the whole system on various grounds
where bugs have the probability to occur. The ground can be performance, security, maximum
load, etc. The integrated system is passed through various tests based on these grounds and
depending on the environment and type of project. After passing through these tests, the resulting
system is a system which is ready for acceptance testing which involves the user, as shown in
Fig. 7.14.
But it is difficult to test the system on various grounds, since there is no methodology for system
testing. One solution to this problem is to think from the perspective of the user and the problem
the user is trying to solve.
Unlike function testing, the external specifications cannot be used as the basis for deriving the
system test cases, since this would weaken the purpose of system testing. On the other hand, the
objectives document cannot be used by itself to formulate test cases, since it does not by
definition, contain precise
Descriptions of the program’s external interfaces. This dilemma is solved by using the program’s
user documentation. Design the system test by analysing the objectives; formulate test cases by
analysing the user documentation. This has a useful side-effect of comparing the program with
its objectives and the user documentation, as well as comparing the user documentation with the
objectives, as shown in Fig. 7.15.
It is obvious that the central purpose of system testing is to compare the system with its stated
objectives. However, there are no test case design methodologies. The reason for this is that
objectives state what a program should do and how well the program should do it, but they do
not state the representation of the program’s functions
Given the statement of objectives, there is no identifi able methodology that would yield a set of
test cases, other than the vague but useful guideline of writing test cases to attempt to show that
the system is inconsistent with each sentence in the objectives statement. Therefore, rather than
developing a system test methodology, distinct categories of system test cases are taken. Some of
the categories are discussed below.
7.4.1 CATEGORIES OF SYSTEM TESTS
Recovery Testing
Recovery is just like the exception handling feature of a programming language. It is the ability
of a system to restart operations after the integrity of the application has been lost. It reverts to a
point where the system was functioning correctly and then, reprocesses the transactions to the
point of failure.
Some software systems (e.g. operating system, database management systems, etc.) must
recover from programming errors, hardware failures, data errors, or any disaster in the system.
So the purpose of this type of system testing is to show that these recovery functions do not work
correctly.
The main purpose of this test is to determine how good the developed software is when it faces a
disaster. Disaster can be anything from unplugging the system which is running the software
from power, network etc., also stopping the database, or crashing the developed software itself.
Thus, recovery testing is the activity of testing how well the software is able to recover from
crashes, hardware failures, and other similar problems. It is the forced failure of the software
in various ways to verify that the recovery is properly performed. Some examples of recovery
testing are given below:
While the application is running, suddenly restart the computer and thereafter, check the
validity of application’s data integrity. „
While the application receives data from the network, unplug the cable and plug-in after
awhile, and analyse the application’s ability to continue receiving data from that point,
when the network connection disappeared. „
Restart the system while the browser has a definite number of sessions and after
rebooting, check that it is able to recover all of them.
Recovery tests would determine if the system can return to a well-known state, and that no
transactions have been compromised. Systems with automated recovery are designed for this
purpose. There can be provision of multiple CPUs and/or multiple instances of devices, and
mechanisms to detect the failure of a device. A ‘checkpoint’ system can also be put that
meticulously records transactions and system states periodically to preserve them in case of
failure. This information allows the system to return to a known state after the failure.
Beizer [49] proposes that testers should work on the following areas during recovery testing:
Restart If there is a failure and we want to recover and start again, then fi rst the current
system state and transaction states are discarded. Following the criteria of checkpoints as
discussed above, the most recent checkpoint record is retrieved and the system is initialized
to the states in the checkpoint record. Thus, by using checkpoints, a system can be recovered
and started again from a new state. Testers must ensure that all transactions have been
reconstructed correctly and that all devices are in proper states. The system now is in a
position to begin to process new transactions.
Switchover
Recovery can also be done if there are standby components and in case of failure of one
component, the standby takes over the control. The ability of the system to switch to a new
component must be tested.
A good way to perform recovery testing is under maximum load. Maximum load would give
rise to transaction inaccuracies and the system would crash, resulting in defects and design
flaws.
Security Testing
Safety and security issues are gaining importance due to the proliferation of commercial
applications on the Internet and the increasing concern about privacy. Security is a protection
system that is needed to assure the customers that their data will be protected. For example, if
Internet users feel that their personal data/information is not secure, the system loses its
accountability. Security may include controlling access to data, encrypting data in
communication, ensuring secrecy of stored data, auditing security events, etc. The effects of
security breaches could be extensive and can cause loss of information, corruption of
information, misinformation, privacy violations, denial of service, etc.
Types of Security Requirements While performing security testing, the following security
requirements must be considered:
Security requirements should be associated with each functional requirement. Each
functional requirement, most likely, has a specific set of related security issues to be
addressed in the software implementation. For example, the log-on requirement in a
client-server system must specify the number of retries allowed, the action to be taken
if the log-on fails, and so on.
In addition to security concerns that are directly related to particular requirements, a
software project has security issues that are global in nature, and are therefore, related
to the application’s architecture and overall implementation. For example, a Web
application may have a global requirement that all private customer data of any kind
is stored in encrypted form in the database. In another example, a system-wide
security requirement is to use SSL to encrypt the data sent between the client browser
and the Web server. Security testing team must verify that SSL is correctly used in all
such transmissions.
Security testing is the process of attempting to devise test cases to evaluate the adequacy
of protective procedures and countermeasures.
The problem with security testing is that security-related bugs are not as obvious as other
type of bugs. It may be possible that the security system has failed and caused the loss of
information without the knowledge of loss. Thus, the tester should perform security
testing with the goal to identify the bugs that are very difficult to identify.
Security vulnerabilities
Vulnerability is an error that an attacker can exploit. Security vulnerabilities are of the
following types:
Bugs at the implementation level, such as local implementation errors or inter
procedural interface errors „
Design-level mistakes
Design-level vulnerabilities are the hardest defect category to handle, but they are also
the most prevalent and critical. Unfortunately, ascertaining whether a program has
design-level vulnerabilities requires great expertise, which makes finding not only
difficult but particularly hard to automate. Examples of design-level security flaws
include problem in error-handling, unprotected data channels, incorrect or missing access
control mechanisms, and timing errors especially in multithreaded systems.
When you are doing performance testing, it may not be possible that you are taking all
statistically representative loads for all the parameters in the system. For example, the system is
designed for n number of users, t number of transactions per user per unit time, with p unit time
response time, etc. Have you tested the performance of the system with all these representative
loads? If no, then load testing must be performed as a part of performance testing. Load is varied
from a minimum (zero) to the maximum level the system can sustain without running out of
resources. As the load is being increased, transactions may suffer (application-specific) excessive
delay. For example, when many users work simultaneously on a web server, the server responds
slow. In this way, through load testing, we are able to determine the maximum sustainable load
the system can handle.
Stress Testing
Stress testing is also a type of load testing, but the difference is that the system is put under loads
beyond the limits so that the system breaks. Thus, stress testing tries to break the system under
test by overwhelming its resources in order to fi nd the circumstances under which it will crash.
Stress testing demands the amount of time it takes to prepare for the test and the amount of
resources consumed during the actual execution of the test. Therefore, stress testing cost must be
weighed against the risk of not identifying volume-related failures. For example, many of the
defense systems which are real-time in nature, demand the systems to perform continuously in
warfare conditions. Thus, for a real-time defense system, we must stress-test the system;
otherwise, there may be loss of equipment as well as life.
Usability Testing
This type of system testing is related to a system’s presentation rather than its functionality.
System testing can be performed to find human-factors or usability problems on the system. The
idea is to adapt the software to users’ actual work styles rather than forcing the users to adapt
according to the software. Thus, the goal of usability testing is to verify that intended users of the
system are able to interact properly with the system while having a positive and convenient
experience. Usability testing identifies discrepancies between the user interfaces of a product and
the human engineering requirements of its potential users
What the user wants or expects from the system can be determined using several ways as
proposed by Dustin [12]:
Area experts The usability problems or expectations can be best understood by the subject or
area experts who have worked for years in the same area. They analyse the system’s specifi c
requirements from the user’s perspective and provide valuable suggestions.
Group meetings Group meeting is an interesting idea to elicit usability requirements from the
user. These meetings result in potential customers’ comments on what they would like to see in
an interface. Surveys Surveys are another medium to interact with the user. It can also yield
valuable information about how potential customers would use a software product to accomplish
their tasks.
Analyse similar products We can also analyse the experiences in similar kinds of projects done
previously and use it in the current project. This study will also give us clues regarding usability
requirements.
Usability characteristics against which testing is conducted are discussed below:
Ease of use
The users enter, navigate, and exit with relative ease. Each user interface must be tailored to the
intelligence, educational background, and environmental pressures of the end-user.
Interface steps
User interface steps should not be misleading. The steps should also not be very complex to
understand either
Response time The time taken in responding to the user should not be so high that the user is
frustrated or will move to some other option in the interface.
Help system A good user interface provides help to the user at every step. The help
documentation should not be redundant; it should be very precise and easily understood by every
user of the system. Error messages For every exception in the system, there must be an error
message in text form so that users can understand what has happened in the system. Error
messages should be clear and meaningful.
An effective tool in the development of a usable application is the user-interface prototype. This
kind of prototype allows interaction between potential users, requirements personnel, and
developers to determine the best approach to the system’s interface. Prototypes are far superior
because they are interactive and provide a more realistic preview of what the system will look
like.
Compatibility/Conversion/Configuration Testing
Compatibility testing is to check the compatibility of a system being developed with different
operating system, hardware and software configuration available, etc. Many software systems
interact with multiple CPUs. Some software control real-time process and embedded software
interact with devices. In many cases, users require that the devices be interchangeable,
removable, or re-configurable. Very often the software will have a set of commands or menus
that allow users to make these configuration changes. Configuration testing allows
developers/testers to evaluate system performance and availability when hardware exchanges
and reconfigurations occur. The number of possible combinations for checking the
compatibilities of available hardware and software can be too high, making this type of testing a
complex job. Discussed below are some guidelines for compatibility testing [12].
Operating systems The specifications must state all the targeted end-user operating systems on
which the system being developed will be run.
Software/Hardware The product may need to operate with certain versions of Web browsers,
with hardware devices such as printers, or with other software’s such as virus scanners or word
processors. Conversion testing Compatibility may also extend to upgrades from previous
versions of the software. Therefore, in this case, the system must be upgraded properly and all
the data and information from the previous version should also be considered. It should be
specified whether the new system will be backward compatible with the previous version. Also,
if other user’s preferences or settings are to be preserved or not.
Ranking of possible configurations
Since there will be a large set of possible configurations and compatibility concerns, the testers
must rank the possible configurations in order, from the most to the least common, for the target
system. Identification of test cases Testers must identify appropriate test cases and data for
compatibility testing. It is usually not feasible to run the entire set of possible test cases on every
possible configuration, because this would take too much time. Therefore, it is necessary to
select the most representative set of test cases that confirms the application’s proper functioning
on a particular platform.
Updating the compatibility test cases The compatibility test cases must also be continually
updated with the following:
(i) Tech-support calls provide a good source of information for updating the compatibility
test suite.
(ii) A beta-test program can also provide a large set of data regarding real end-user
configurations and compatibility issues prior to the final release of a system.
Acceptance testing is one of the most important types of testing we can conduct on a product. It
is more important to worry whether users are happy about the way a program works rather than
whether or not the program passes a bunch of tests that were created by testers in an attempt to
validate the requirements, that an analyst did their best to capture and a programmer interpreted
based on their understanding of those requirements.
Thus, acceptance testing is the formal testing conducted to determine whether a software
system satisfies its acceptance criteria and to enable buyers to determine whether to accept the
system or not.
Acceptance testing must take place at the end of the development process. It consists of tests to
determine whether the developed system meets the predetermined functionality, performance,
quality, and interface criteria acceptable to the user. Therefore, the final acceptance
acknowledges that the entire software product adequately meets the customer’s requirements.
User acceptance testing is different from system testing. System testing is invariably performed
by the development team which includes developers and testers. User acceptance testing, on the
other hand, should be carried out by end-users.
Thus, acceptance testing is designed to: „
Determine whether the software is fi t for the user. „
Making users confident about the product. „
Determine whether a software system satisfies its acceptance criteria. „
Enable the buyer to determine whether to accept the system or not.
The final acceptance marks the completion of the entire development process. It happens when
the customer and the developer has no further problems.
Acceptance test might be supported by the testers. It is very important to define the acceptance
criteria with the buyer during various phases of SDLC. A well-defined acceptance plan will help
development teams to understand users’ needs. The acceptance test plan must be created or
reviewed by the customer. The development team and the customer should work together and
make sure that they:
„ Identify interim and final products for acceptance, acceptance criteria, and schedule. „
Plan how and by whom each acceptance activities will be performed. „
Schedule adequate time for the customer to examine and review the product. „
Prepare the acceptance plan. „
Perform formal acceptance testing at delivery. „
Make a decision based on the results of acceptance testing.
Entry Criteria „
System testing is complete and defects identified are either fixed or documented. „
Acceptance plan is prepared and resources have been identified. „
Test environment for the acceptance testing is available.
Exit Criteria „
Acceptance decision is made for the software. „
In case of any warning, the development team is notified.
Types of Acceptance Testing
Acceptance testing is classified into the following two categories: „
Alpha Testing Tests are conducted at the development site by the end users. The test
environment can be controlled a little in this case. „
Beta Testing Tests are conducted at the customer site and the development team does not have
any control over the test environment.
7.5.1 ALPHA TESTING
Alpha is the test period during which the product is complete and usable in a test environment,
but not necessarily bug-free. It is the final chance to get verification from the customers that the
tradeoffs made in the final development stage are coherent.
Therefore, alpha testing is typically done for two reasons:
(i) to give confidence that the software is in a suitable state to be seen by the customers
(but not necessarily released).
(ii) to find bugs that may only be found under operational conditions. Any other major
defects or performance issues should be discovered in this stage.
Since alpha testing is performed at the development site, testers and users together perform this
testing. Therefore, the testing is in a controlled manner such that if any problem comes up, it can
be managed by the testing team.
Entry Criteria to Alpha „
All features are complete/ testable (no urgent bugs). „
High bugs on primary platforms are fixed/ verified. „
50% of medium bugs on primary platforms are fixed/ verified. „
All features have been tested on primary platforms. „
Performance has been measured/compared with previous releases (user functions). „
Usability testing and feedback (ongoing). „
Alpha sites are ready for installation.
Exit Criteria from Alpha
After alpha testing, we must: „
Get responses/feedbacks from customers. „
Prepare a report of any serious bugs being noticed. „
Notify bug-fixing issues to developers.
7.5.2 BETA TESTING
Once the alpha phase is complete, development enters the beta phase. Beta is the test period
during which the product should be complete and usable in a production environment. The
purpose of the beta ship and test period is to test the company’s ability to deliver and support the
product (and not to test the product itself). Beta also serves as a chance to get a final ‘vote of
confidence’ from a few customers to help validate our own belief that the product is now ready
for volume shipment to all customers.
Versions of the software, known as beta-versions, are released to a limited audience outside the
company. The software is released to groups of people so that further testing can ensure the
product has few or no bugs. Sometimes, beta-versions are made available to the open public to
increase the feedback field to a maximal number of future users.
Testing during the beta phase, informally called beta testing, is generally constrained to black-
box techniques, although a core of test engineers are likely to continue with white-box testing
parallel to beta tests. Thus, the term beta test can refer to a stage of the software—closer to
release than being ‘in alpha’—or it can refer to the particular group and process being done at
that stage. So a tester might be continuing to work in white-box testing while the software is ‘in
beta’ (a stage), but he or she would then not be a part of ‘the beta test’ (group/activity).
Entry Criteria to Beta „
Positive responses from alpha sites. „
Customer bugs in alpha testing have been addressed. „
There are no fatal errors which can affect the functionality of the software. „
Secondary platform compatibility testing is complete. „
Regression testing corresponding to bug fixes has been done. „
Beta sites are ready for installation.
Guidelines for Beta Testing „
Don’t expect to release new builds to beta testers more than once every two weeks. „
Don’t plan a beta with fewer than four releases. „
If you add a feature, even a small one, during the beta process, the clock goes back to the
beginning of eight weeks and you need another 3–4 releases.
Exit Criteria from Beta
After beta testing, we must: „
Get responses/feedbacks from the beta testers. „
Prepare a report of all serious bugs. „
Notify bug-fixing issues to developers.
Regression Testing
In Chapter 7, integration testing was discussed as one of the validation activity. However, when a
new module is added as a part of integration testing, the software changes. New data flow paths
are established, new I/O may occur, etc. These changes, due to addition of new modules, may
affect the functioning of earlier integrated modules which otherwise were functioning flawlessly.
Moreover, the software may also change whenever a new bug in the working system appears and
it needs some changes. The new modifications may affect other parts of the software too. It
means that whenever the software configuration changes, there is a probability of other modules
getting affected due to bug fixing. In this case, it becomes important to test the whole software
again with all the test cases so that a new modification in a certain part of the software does not
affect other parts of the software.
After a program has been modified, we must ensure that the modifications work correctly and
check that the unmodified parts of the program have not been adversely affected by these
modifications. It may be possible that small changes in one part of a program may have subtle
undesired effects on other unrelated modules of the software. It is not necessary that if you are
getting the desired outputs on the screen, then there is no problem. It may produce incorrect
outputs on other test cases on which the original software produced correct outputs earlier. Thus,
during regression testing, the modified software is executed on all tests to validate that it still
behaves the same as it did in the original software, except where a change is expected.
Thus, regression testing can be defined as the software maintenance task performed on a
modified program to instill confidence that changes are correct and have not adversely
affected the unchanged portions of the program.
Rothermel and Harrold [50,51,52] have also characterized the typical steps taken by this
technique in the following manner (see Fig. 8.2):
1. Select T’ subset of T, a set of test cases to execute on P’.
2. Test P’ with T’, establishing correctness of P’ with respect to T’.
3. If necessary, create T ‘’, a set of new functional or structural test cases for P’.
4. Test P ¢ with T’’, establishing correctness of P’ with respect to T’’.
5. Create T’’’, a new test suite and test execution profile for P’, from T, T’, and T’’.
For example, the technique of Rothermel and Harrold [55] uses controlfl ow graph
representations of P and P ¢ and test execution profiles gathered on P, to select every test case in
T that, when executed on P, exercised at least one statement that has been deleted from P, or that,
when executed on P’, will exercise at least one statement that is new or modified in P’ (when P is
executed, a statement that does not exist in P cannot be exercised).
Ad hoc/Random techniques
When time constraints prohibit the use of a retestall approach, but no test selection tool is
available, developers often select test cases based on ‘intuitions’ or loose associations of test
cases with functionality. Another simple approach is to randomly select a predetermined number
of test cases from T.
Retest-all technique The retest-all technique simply reuses all existing test cases. To test P ¢,
the technique effectively selects all test cases in T.
Evaluating Regression Test Selection Techniques
To choose a technique for practical application, Rothermel et al. [51] have recognized the
following categories for evaluating the regression test selection techniques:
Inclusiveness It measures the extent to which M chooses modificationrevealing test from T for
inclusion in T ¢, where M is a regression test selection technique.
Suppose, T contains n tests that are modification-revealing for P and P ¢, and suppose M selects
m of these tests. The inclusiveness of M relative to P and P ¢ and T is:
1. INCL(M ) = (100 ¥ (m/n)%, if n ≠ 0
2. INCL(M )% = 100%, if n = 0
For example, if T contains 100 tests of which 20 are modifi cation-revealing for P and P ¢, and
M selects 4 of these 20 tests, then M is 20% inclusive relative to P, P’, and T. If T contains no
modifi cation-revealing tests, then every test selection technique is 100% inclusive relative to P,
P’, and T. If for all P, P’, and T, M is 100% inclusive relative to P, P’, and T, then M is safe.
Precision It measures the extent to which M omits tests that are nonmodification-revealing.
Suppose T contains n tests that are non-modificationrevealing for P and P’, and suppose M omits
m of these tests. The precision of M relative to P, P’, and T is given by
Precision = 100 ¥ (m/n) % if n ≠ 0
Precision = 100 % if n = 0
For example, if T contains 50 tests of which 22 are non-modification-revealing for P and P ¢,
and M omits 11 of these 22 tests, then M is 50% precise relative to P, P’, and T. If T contains no
non-modification-revealing tests, then every test selection technique is 100% precise relative to
P, P’, and T.
Effi ciency
The effi ciency of regression test selection techniques is measured in terms of their space and
time requirements. Where time is concerned, a test selection technique is more economical than
the retest-all technique, if the cost of selecting T’ is less than the cost of running the tests in T–T
‘. Space effi ciency primarily depends on the test history and program analysis information a
technique must store. Thus, both space and time efficiency depends on the size of the test suite
that a technique selects, and on the computational cost of that technique.
8.8.2 REGRESSION TEST PRIORITIZATION
The regression test prioritization approach is different as compared to selective retest techniques.
The prioritization methods order the execution of RTS with the intention that a fault is detected
earlier. In other words, regression test prioritization attempts to reorder a regression test suite so
that those tests with the highest priority, according to some established criterion, are executed
earlier in the regression testing process than those with a lower priority. By prioritizing the
execution of a regression test suite, these methods reveal important defects in a software system
earlier in the regression testing process.
This approach can be used along with the previous selective retest technique. The steps for this
approach are given below:
1. Select T’ from T, a set of test to execute on P’.
2. Produce T’p , a permutation of T’, such that T’p will have a better rate of fault detection than
T’.
3. Test P’ with T’p in order to establish the correctness of P’ with respect to T’p .
4. If necessary, create T’’, a set of new functional or structural tests for P’’.
5. Test P’ with T’’ in order to establish the correctness of P’ with respect to T’’.
6. Create T’’’, a new test suite for P’, from T, T’p , and T’’.
Step 1 allows the tester to select T ¢such that it is a proper subset of T or that it actually contains
every test that T contains. Furthermore, step 2 could produce T’p so that it contains an execution
ordering for the regression test that is the same or different than the order provided by T’. The
intention behind prioritizing the execution of a regression test suite is to ensure that the defects
normally introduced by competent programmers are discovered earlier in the testing process.