Module IV

TEST PLANNING, MANAGEMENT,EXECUTION AND REPORTING

Topics
Scope Deciding Feature to be Tested/ Not Tested
Manageme
nt, Matrix
Test Test Infrastructure Management, Test People Management, Integrating with Product
Manageme Release
nt

Test Base Lining a Test Plan, Test Case Specification, Update of Traceability;
Process

Test Preparing a Test Plan, Setting Up Criteria for Testing, Deciding Test Approach,
Planning, Identifying Responsibilities,
Test Executing Test Cases, Preparing Test Summary Report.
Reporting
 TEST PLANNING
1.Preparing a Test Plan
While Preparing a Test plan, it should acts as execution, tracking and reporting of the entire testing
project and it should cover
• The Scope of Testing,
• How the testing is going to be performed,(Methodology)
• What Resources are needed for testing,(Requirement)
• The TimeLine(schedule)
• Risk factor

2.Scope Management

Scope Management pertains to specifying the scope of a project. For testing, Scope Management
entails,
1. Understanding what constitutes a release of a product;
2. Breaking down the release into features;
3. Prioritizing the features for testing;
4. Deciding which feature will be tested and which will not be;
5. Gathering details to prepare foe estimation of resources for testing.

The following factors drive the choice and prioritization of features to be tested.
• Feature that are new and critical for the release
• Feature whose failures can be catastrophic
• Feature that are expected to be complex to test
 • Features which are extensions of earlier features that have been defect prone

3.Deciding Test Approach

Deciding Test Approach include identifying

1. What type of testing would you use for testing the functionality?
2. What are the configurations or scenarios for testing the features?
3. What integration testing would you do to ensure these features work together?
4. What localization validations would be needed?
5. What “non-functional” tests would you need to do?

Test strategy Major types are:

• Analytical
• Model Based
• Methodological
• Process-Standard complaint
• Dynamic
• Regression

4.Setting Up Criteria for Testing

• Pass or Fail Criteria
• Suspend Criteria
• Resume Criteria
A test cycle or a test activity will not be an isolated, it is a continuous activity that can be carried out
at one go. Some of the typical criteria include,
1. Encountering more than a certain number of defect
2. Hitting a show stoppers that prevent further progress of testing
3. Developers releasing a new version which they advise should be used in lieu of the product
under test.
5.Identifying Responsibilities, Staffing and Training Needs
• Identifying group responsibilities for designing, preparing, executing, checking, and managing
• Staffing and Training Needs
• Providing environmental needs
• addresses the aspects of test planning i.e. who part of it. A testing project require different
people to play different roles, So
The different role definitions should
1. Ensure clear accountability for a given task;
2. Clearly list the responsibilities for various functions to various people;
3. Complement each other;
4. Supplement each other;
6.Resource Requirements
Project Manager should provide estimates for the various hardware and software resource required.
Some of the following factors need to be considered.
1. Machine configuration,
2. Overheads required by the test automation tool,
3. Supporting tools such as Compiler, Test data generator, Configuration management tool.
4. The different configurations of the supporting software,
5. Special requirement for running machine-intensive tests
6. Appropriate number of licenses of all the software.

7.Test Deliverables
The deliverables include the following,
1. The test plan
2. Test case Specification
3. Test design specification documents
4. Testing Strategy
 5. Testing Scripts/ procedures
6. Test data
7. Test Incident report
8. Test Traceability matrix
9. Test results/Reports
10. Install/Configuration guides
11. Test logs produced
12. Defect Report
13. Test summary report
14. Release Report
8.Testing Tasks(Size and Effort Estimation)

Estimations happens broadly in three phases,

1. Size estimation
-Number of test cases
-Number of test scenarios
-Number of configuration to be tested
2. Effort estimation
-Productivity data
-Reuse opportunities
-Robustness of processes
3. Schedule estimation
9.Risk Management
Risks are events that could potentially affect a project’s outcome. These events are beyond the control
of project manager. The risk management include the following activities:
1.identify the possible risks
2.Quantfying the risks
3.planning how to mitigate the risks
4. Responding to risks when they become a reality.

Test Management
Traditional tools used for Test management are:
• Pen and Paper
• Word Processors
• Spreadsheets
Internal Standard include,
• Simplified Communication
• Promotes consistency and uniformity
• Eliminates the need to invent another solution
• Continuity
• Presents a way of preserving proven practices
 • Bench marks and framework
1. Naming and storage conventions for test artifacts;
2. Document standards;
3. Test coding standards;
4. Test reporting standards
External Standard include:
• Customer standard
• National Standard
• International Standard
Some IEEE standards devoted for softwares :
By Different Organizations are:
✓ ISO - International Organization for Standardization
✓ SPICE - Software Process Improvement and Capability Determination
✓ NIST - National Institute of Standards and Technology
✓ DoD - Department of Defense

IEEE Standard Standard use for Purpose

610.12-1990 Standard Glossary of terminologies
730-1998 Software Quality assurance plan
828-1998 Software configuration management
829-1998 Software Test Document
830-1998 Software Requirement Specification
1008-1987 Software Unit Testing
1012-1998
1012a-1998 Software Verification and Validation
1016-1998 Software design Description
1028-1997 Standards for software Review
1044-1993 Software anomalies
1045-1992 Software Productivity Metrics(ANSI)
1058-1998 Software Project Management Plans
1058.1-1987 Software Management
1061-1998.1 Software Quality Metrics Methodology

Test Infrastructure Management/Test Management

Testing requires a robust infrastructure to be planned upfront. This infrastructure is made up of three
essential elements.
1. A test case database(TCDB) (additional)
A test case database captures all the relevant information about the test cases in an organization.
Some of the entities and the attributes are given in following table
 Sr. Test Case Purpose Attributes
No.
1 Test case Records all static information about • Test case Id
tests. • Test case name (File name)
• Test case owner
• Associated files for test case.

2 Test case Provide mapping between the tests • Test case Id

product cross and the corresponding product • Module Id
reference features, enables identification of
test cases for given feature.
3 Test case run Gives the history of when the test • Test case Id
history case was run and what was result , • Run date
provided inputs on selection of test • Time taken
for regression runs • Run status(Success/Failure)

4 Test case- Gives details of test cases • Test case Id

defect cross- introduced to test certain specific • Defect reference
reference defects detected in the product,
provides inputs on the selection of
test for regression runs.

2. A defect repository:
• Captures all the relevant details of defects reported for a product
• An important vehicle of communication that influences the work flow within a software
organization.
• Provides the base data in arriving at several of the metrics, testing defect metrics, development
defect metrics as a part of defect repository.
3. Configuration management:
• Collection of software elements that comprises a major business function.
• Concerned with labeling, tracking and controlling changes in the software elements of
system.
• To identify all the interrelated components of software and to control their evolution
throughout the various life cycles.
• That can be applied to activities like software development, document control, problem
tracking, change control and maintenance.
• Also known as Software Configuration Management repository.

Change control ensures :

• Changes to test files are made in a controlled fashion and only with proper approvals.
• Changes made by one test engineer are not accidently lost or overwritten by other changes.
• Each change produces distinct a distinct version of the file that is regrettable at any point of time.
i.e. everyone gets the access to only the most recent version of the files.
Sub elements of these may be:
✓ Test plan document
✓ Test cases (planned, designed)
 ✓ Baseline the test data
✓ A process to refresh or roll back baseline data
✓ Test Environment –Front end back end and test lab.
✓ Test case database, to track and update tests
✓ A method to prioritize test cases per test cycle
✓ Coverage analysis metrics
✓ Defect tracking database
✓ Risk management metrics
✓ Version control system
✓ Configuration management system
✓ A requirement tracking tool
✓ Metrics required to measure improvement
This helps to serve many purposes:
• Allows to run automated tests
• Provides a test environment to avoid conflicts between manual and automated testing
• Helps to track the results of test cases ,pass or fail criteria
• Gives report of test coverage levels
• Keeps expected results remain consistent across test runs.
• A Test lab to conduct multiuser and stress testing.

A defect repository captures all the relevant details of defects reported for a product. The
information that a defect repository includes is given in Table.
TCDB, defect repository and SCM repository should complement each other and work together
in an integrated fashion.
 Test People Management

People management is an integral part of any project management.

• People management also requires the ability to hire, motivate, and retain the right people.
These skills are seldom formally taught.
• Testing projects present several additional challenges. We believe that the success of a testing
organization depends vitally on judicious people management skills

Test Lead responsibilities and activities:

1. Identify how the test teams formed and aligned within organization
2. Decide the roadmap for the project
3. Identify the scope of testing using SRS documents.
4. Discuss test plan, review and approve by management/ development team.
5. Identify required metrics
6. Calculate size of project and estimate efforts and corresponding plan.
7. Identify skill gap and balance resources and need for training education.
8. Identify the tools for test reporting , test management, test automation,
9. Create healthy environment for all resources to gain maximum throughput.
10. Identify how the test teams formed and aligned within organization management/ development
team.

Test team responsibilities and activities:

1. Initiate the test plan for test case design
2. Conduct review meetings
3. Monitor test progress , check for resources, balancing and allocation
4. Check for delays in schedule discuss, resolve risks if any.
5. Intimate status to stake holders and management
6. Bridge the gap between test team and management.

Consider followings for managing test

Understand testers
Test work environment
Role of the test team

Integrating with Product Release

The success of a product depends on the effectiveness of integration of the development and
testing activities. The schedules of testing have to be linked directly to product release. The
following are some of the points to be decided for this planning-
1. Sync point between development and testing as to when different types of testing can
commence.
 2. Service level agreement/management between development and testing as to how long
it would take for the testing team to complete the testing.
3. Consistent definitions of the various priorities and severities of the defects.
4. Communication mechanisms to the documentation group to ensure that the
documentation is kept in sync with the product in terms of known defects, workarounds.

Test Process:
Fundamental test process is divided into following steps:
1.Base Lining a Test Plan
A test combines all the points into a single document that acts as an anchor point for the entire
testing project.
✓ An organization normally arrives at a template that is to be used across the board. Each
testing project puts together a test plan based on the template.
✓ The test plan is reviewed by a designated set of component people in the organization.
✓ After this, the test plan is baselined into the configuration management repository.
✓ From then on, the baselined test plan becomes the basis for running the testing project.
✓ In addition, periodically, any changes needed to the test plan templates are discussed
among the different stake holders and this is kept current and applicable to the testing
teams.

2.Test Case Specification

Using the test plan as the basis, the testing team designs test case specification, which then becomes
the basis for preparing individual test cases. Hence, a test case specification should clearly identify,
1. The purpose of the test: This lists what features or part the test is intended for.
2. Items being tested, along with their version/release numbers as appropriate.
3. Environment that needs to be set up for running the test cases: This includes the
hardware environment setup, supporting software environment setup, setup of the product
under test.
4. Input data to be used for the test case: The choice of input data will be dependent on the
test case itself and the technique followed in the test case.
5. Steps to be followed to execute the test: If automated testing is used, then, these steps ate
translated to the scripting language of the tool.
6. The expected results that are considered to be “correct result”.
7. A step to compare the actual result produced with the expected result: This step should
do an “intelligent” comparison of the expected and actual results to highlight any
discrepancies.
8. Any relationship between this test and other test: These can be in the form of
dependencies among the tests or the possibilities of reuse across the tests.
3. Update of Traceability
Black Box Testing, a requirements traceability matrix ensures that the requirements make it
through the subsequent life cycle phases and do not get orphaned mid-course.
• The traceability matrix is a tool to validate that every requirement is tested.
The traceability matrix is created during the requirement gathering phase itself by filling up
the unique identifier for each requirement.
• This ensures that there is a two-way mapping between requirements and test cases.

4.Executing Test Cases and keep the matrix updated

The prepared test cases have to be executed at the appropriate times during a project. As the test
cases are executed during a test cycle, the defect repository is updated with,
1. Defect from the earlier test cycles that are fixed in the current build;
2. New defect that get uncovered in the current run of the tests.
3. Follow test procedure to execute test suite and test cases.
4. Confirm test, re-test, re-execute to fix.
5. Log the result of test execution, status of test pass and fail.
6. Comparison of actual results with expected results.
7. Defect occurrences and differences should be updated in report
8. Decide suspension and resumption of further test cases also update traceability metrics
During test design and execution, the traceability matrix should be kept current. As and when tests
get designed and executed successfully, the traceability matrix should be updated.

5.Collecting and Analyzing Metrics

When test are executed, information about test execution gets collected in test logs and other files.
The basic measurements from running the tests are then converted to meaningful metrics by the use
of appropriate transformations and formulae.
6.Preparing Test Summary Report
At the completion of a test cycle, a test summary report is produced. This report gives insights to the
senior management about the fitness of the product for release.
7. Recommending Product Release.
✓ A Test report is any description, explanation or justification the status of a test project.

• Based on the test summary report, an organization can take a decision on whether to release the
product or not.
• Ideally, an organization would like to release a product with zero defects.
• However, market pressures may cause the product to be released with the defects provided that
the senior management is convinced that there is no major risk of customer dissatisfaction.
• Such a decision should be taken by the senior manager after consultation with customer support
team, development team, and testing team.

Test Summary Report/ Test Reporting

At the completion of a test cycle, a test summary report is produced. This report gives insights to the
senior management about the fitness of the product for release. There are two types of reports that
are required:
1. The Incident Report:
✓ Communication that happens through the testing cycle as an when defects are
 encountered.
✓ Entry made in defect repository.
✓ High impact test incidence is highlighted in test summary report.
2. Test Cycle Report:
✓ Summary of activities during test cycle
✓ Uncovered Defects based on their severity and impact
✓ Progress from previous cycle to the current cycle in terms of defect fixing
✓ Outstanding defect those yet to be fixed.
✓ Any variations observed in efforts or schedule useful for future planning
3. Test Summary Report
It is final test in test cycle.
✓ Summary of activities carried out during test cycle.
✓ Variance of the activities carried out from the activity plan includes:
o Test that were planned to be runned but could not be runned.
o Modification of test from what was in the original test specifications
o Additional tests that was run, not runned in the original test plan.
o Differences in efforts and time taken between planned and executed.
o Any other derivations from plan.
o Summary of result should include
Test that fail , with any root cause, descriptions and
Severity of impact of the defect uncovered by test.
o Comprehensive assessment and recommendation for release should include.
Fit for release , assessment and
Recommendation of release.
A summary report should present;
1. A summary of the activities carried out during the test cycle;
2. Variance of the activities carried out from the activities planned;
3. Summary of results should include tests that failed and severity of impact of defect;
4. Comprehensive assessment and recommendation for release should include “Fit for release”
assessment and Recommendation of release.
5. What kind of defects product has?
6. What is impact /severity of each of these defects?
7. What could be the risks of releasing the product with existing defects?