FAA W J. H T C T E H: Illiam Ughes Echnical Enter Est and Valuation Andbook

FAA W IL L I AM J.
H U GH E S T E C H NI C AL C E NT E R
T E ST A ND E VA L UA T I ON H AN D B O OK
DOCUMENT # VVSPT-A2-PDD-013
VERSION # VERSION 4.0
VERSION DATE MAY 16, 2017
FAA WILLIAM J. HUGHES TECHNICAL CENTER

ATLANTIC CITY INTERNATIONAL AIRPORT, NEW JERSEY 08405
T HIS D OC UM E NT B E C OME S U NC ONTR O LLE D O NC E PR INTE D

TEST & EVALUATION HANDBOOK
Intentionally Blank
VVSPT-A2-PDD-013 v4.0 ii 05/16/2017

DOCUMENT VERSION CONTROL
VERSION DESCRIPTION OF CHANGE DATE
Draft 1.0 Initial delivery to management 12/20/2006
Final 1.0 Initial delivery for T&E process implementation 08/21/2008
Incorporated comments against Draft V1.2 as detailed in

Final 2.0 08/31/2010
the V&V Repository Comments Database
Final 3.0 Release Version 3.0 09/24/2013
3.1 Incorporate comments against Final V3.0 2/24/2016
Incorporated additional items from V&V Comment

3.2 5/13/2016
Database
3.3 Incorporated comments from TSB Review 6/10/2016
3.4 Incorporating comments from External Peer Review 7/25/2016
3.5 Proofed version for Formal Branch Review 2/8/2017
Incorporated Branch Review comments. Proofed version

3.6 4/14/2017
for Division Manager Review
4.0 Final for Division Signature 5/16/2017
VVSPT-A2-PDD-013 v4.0 iv 05/16/2017

TABLE OF CONTENTS
1 INTRODUCTION ........................................................................................................................................... 1-1
1.1 VERIFICATION AND VALIDATION PRINCIPLES ...............................................................................................1-1
1.2 OBJECTIVES OF THE TEST AND EVALUATION HANDBOOK.............................................................................1-3
1.3 SCOPE ...........................................................................................................................................................1-3
1.4 ROLES AND RESPONSIBILITIES ......................................................................................................................1-4
1.5 FUNDAMENTAL PRACTICES FOR QUALITY T&E............................................................................................1-4
1.6 TAILORING OF T&E PROCESSES....................................................................................................................1-5
1.6.1 Tailoring Criteria and Processes ....................................................................................................1-5
1.6.2 Tailoring Guidelines ......................................................................................................................1-6
1.6.2.1 Types of Tailoring ..........................................................................................................1-6
1.6.2.2 Commonly Tailored Processes .......................................................................................1-6
1.7 T&E STANDARDS CONFORMANCE PROCESS .................................................................................................1-7
1.8 T&E QUALITY MANAGEMENT SYSTEM PROCESSES .....................................................................................1-7
1.9 PEER REVIEWS ..............................................................................................................................................1-8
2 RELATED DOCUMENTS AND REFERENCES ....................................................................................... 2-1
3 T&E SUPPORT ACROSS THE AMS LIFECYCLE .................................................................................. 3-1
4 T&E SUPPORT TO SERVICE ANALYSIS & STRATEGIC PLANNING AND CONCEPT &
REQUIREMENTS DEFINITION ................................................................................................................. 4-1
5 T&E SUPPORT TO INVESTMENT ANALYSIS ....................................................................................... 5-1
5.1 PROGRAM REQUIREMENTS DOCUMENT SUPPORT .........................................................................................5-1
5.2 TEST AND EVALUATION MASTER PLAN DEVELOPMENT ...............................................................................5-2
5.2.1 TEMP Objectives ...........................................................................................................................5-4
5.2.2 Test Management Using the TEMP ...............................................................................................5-5
5.2.3 Test Design Process .......................................................................................................................5-7
5.2.3.1 Identification and Assessment of Requirements .............................................................5-7
5.2.3.2 Decomposition of Critical Operational Issues ................................................................5-7
5.2.3.3 Critical Performance Requirements Evaluation Approach ...........................................5-10
5.2.3.4 Defining Test Activities ................................................................................................5-10
5.2.3.5 Formulating the TEMP Verification Requirements Traceability Matrix ......................5-12
5.2.3.6 Identifying Test Capability Requirements ....................................................................5-13
5.2.3.7 Defining Test Schedule .................................................................................................5-14
5.2.3.8 Assessing Updates and Acquisition Program Baseline Changes ..................................5-14
5.2.3.9 Documenting the Test Design in the TEMP .................................................................5-14
5.2.3.10 Interim Assessment Report Planning ..........................................................................5-14
5.3 IMPLEMENTATION STRATEGY AND PLANNING DOCUMENT SUPPORT .........................................................5-15
5.4 FAA SYSTEM SPECIFICATION SUPPORT ......................................................................................................5-15
5.5 SCREENING INFORMATION REQUEST (SIR) SUPPORT ..................................................................................5-17
5.5.1 Section L of the SIR: Instructions, Conditions, and Notices to Offerors ....................................5-17
5.5.2 Section C of the SIR: Description Of Work Requirements .........................................................5-17
5.5.2.1 Statement of Work (SOW)/Performance Work Statement (PWS) ...............................5-17
5.5.2.2 Statement of Objectives (SOO) ....................................................................................5-18
5.6 PROPOSAL EVALUATION SUPPORT ..............................................................................................................5-19
6 T&E SUPPORT TO SOLUTION IMPLEMENTATION - DT .................................................................. 6-1
6.1 DT OVERVIEW ..............................................................................................................................................6-1
VVSPT-A2-PDD-013 v4.0 v 05/16/2017

6.1.1DT Activities ..................................................................................................................................6-1

6.1.1.1 Agile Acquisition in AMS ..............................................................................................6-3
6.1.2 DT Requirements ...........................................................................................................................6-4
6.1.3 DT Support to Contract Administration .........................................................................................6-7
6.1.3.1 DT Screening Information Request Support ...................................................................6-7
6.1.3.2 Contractor Documentation Support ................................................................................6-7
6.1.3.3 Contractor Test Document Review Standards ................................................................6-8
6.1.3.4 FCA, PCA and GA Activities Support ...........................................................................6-9
6.1.4 DT Engineering Support ................................................................................................................6-9
6.2 DT PLANNING ...............................................................................................................................................6-9
6.2.1 DT Test Approach and Strategies ................................................................................................6-10
6.2.1.1 COTS/NDI Testing .......................................................................................................6-10
6.2.1.2 DT Software Testing.....................................................................................................6-11
6.2.1.3 DT Hardware Testing ...................................................................................................6-11
6.2.1.4 Factory Acceptance Testing (FAT) ..............................................................................6-12
6.2.1.5 Functional Qualification Testing (FQT) .......................................................................6-12
6.2.1.6 DT Installation and Integration Testing ........................................................................6-13
6.2.1.7 DT System Testing .......................................................................................................6-13
6.2.1.8 Production Acceptance Testing (PAT) .........................................................................6-15
6.2.1.9 Site Acceptance Testing (SAT) ....................................................................................6-15
6.2.1.10 DT Test Capability Accreditation ...............................................................................6-16
6.2.1.11 Coordination Between DT and OT Test Directors .....................................................6-16
6.2.1.12 DT Training and Familiarization ................................................................................6-16
6.2.2 Contractor Master Test Plan (CMTP) ..........................................................................................6-17
6.2.3 DT Verification Requirements Traceability Matrix (VRTM) ......................................................6-18
6.2.4 DT Integration Test Planning .......................................................................................................6-19
6.2.5 DT Test Plans ...............................................................................................................................6-19
6.2.6 DT Test Procedures......................................................................................................................6-20
6.2.7 DT Entrance Criteria ....................................................................................................................6-21
6.2.8 DT Exit Criteria ...........................................................................................................................6-21
6.3 DT TEST CONDUCT .....................................................................................................................................6-21
6.3.1 DT Debug and Dry Run Testing ..................................................................................................6-21
6.3.2 DT Test Readiness Review ..........................................................................................................6-22
6.3.3 DT Pre-Test Briefings ..................................................................................................................6-23
6.3.4 Formal DT Test Execution ...........................................................................................................6-23
6.3.5 DT Post-Test Reviews .................................................................................................................6-24
6.3.6 DT Test Status Reports ................................................................................................................6-24
6.3.7 DT Problem Reporting Processes ................................................................................................6-24
6.3.8 DT Regression Testing.................................................................................................................6-25
6.4 DT TEST REPORT ........................................................................................................................................6-26
7 T&E SUPPORT TO SOLUTION IMPLEMENTATION - OT .................................................................. 7-1
7.1 OT OVERVIEW ..............................................................................................................................................7-1
7.1.1 NAS Integration Testing ................................................................................................................7-1
7.1.2 Operational Effectiveness and Suitability Testing .........................................................................7-2
7.2 PURPOSE OF OT ............................................................................................................................................7-2
7.3 OT TEST TEAM TASKS AND FUNCTIONS .......................................................................................................7-3
VVSPT-A2-PDD-013 v4.0 vi 05/16/2017

7.3.1 OT Test Team Organization...........................................................................................................7-3

7.3.2 OT Planning in TEMP Development .............................................................................................7-3
7.3.3 OT Support to Contract Administration .........................................................................................7-5
7.3.4 OT Training ...................................................................................................................................7-5
7.3.5 Coordination with OT Field Personnel ..........................................................................................7-5
7.3.6 OT Test Capability Accreditation ..................................................................................................7-6
7.4 OT PLANNING ...............................................................................................................................................7-6
7.4.1 OT Test Plan Development ............................................................................................................7-6
7.4.2 OT Verification Requirements Traceability Matrix (VRTM) ........................................................7-7
7.4.3 OT Entrance Criteria ......................................................................................................................7-7
7.4.4 OT Exit Criteria .............................................................................................................................7-8
7.5 OT TEST PROCEDURES..................................................................................................................................7-8
7.5.1 OT Test Procedure Development ...................................................................................................7-9
7.5.2 OT Test Procedure Questionnaires ................................................................................................7-9
7.5.3 OT Test Procedure Tracking ........................................................................................................7-10
7.6 OT TEST CONDUCT .....................................................................................................................................7-10
7.6.1 Logistics for Formal Test Conduct...............................................................................................7-10
7.6.2 OT Test Readiness Review ..........................................................................................................7-11
7.6.3 Formal OT Conduct .....................................................................................................................7-12
7.6.3.1 OT Pre-Test Briefings ...................................................................................................7-12
7.6.3.2 OT Execution ................................................................................................................7-12
7.6.3.3 OT Post-Test Reviews and Test Status Reporting ........................................................7-13
7.6.4 OT Discrepancy Reporting ..........................................................................................................7-13
7.6.5 OT Caucus ...................................................................................................................................7-14
7.6.6 OT Regression Testing.................................................................................................................7-16
7.7 OT TEST REPORTING ..................................................................................................................................7-17
7.7.1 OT Interim Assessment Report ....................................................................................................7-17
7.7.2 OT Quicklook Test Report ...........................................................................................................7-18
7.7.3 OT Final Test Report ...................................................................................................................7-19
7.8 TRANSITION TO IN-SERVICE MANAGEMENT ...............................................................................................7-20
7.9 FIELD FAMILIARIZATION SUPPORT..............................................................................................................7-20
7.9.1 Field Familiarization Support Planning .......................................................................................7-20
7.9.2 Field Familiarization Support Preparation ...................................................................................7-21
7.9.3 Field Familiarization Support Conduct ........................................................................................7-22
7.9.4 Field Familiarization Reporting ...................................................................................................7-22
8 T&E SUPPORT TO IN-SERVICE MANAGEMENT ................................................................................ 8-1
8.1 MAJOR PROGRAM UPGRADE SUPPORT ..........................................................................................................8-1
8.2 MINOR PROGRAM UPGRADE SUPPORT ..........................................................................................................8-1
8.3 PROGRAM MAINTENANCE SUPPORT..............................................................................................................8-1
9 TEST PREPARATION AND DEPLOYMENT DECISION SUPPORT ................................................... 9-1
9.1 TEST CAPABILITY ACCREDITATION ..............................................................................................................9-1
9.1.1 Accreditation Definitions ...............................................................................................................9-1
9.1.2 Accreditation Testing .....................................................................................................................9-2
9.1.3 Accreditation Process .....................................................................................................................9-3
9.1.4 Accreditation Guidance..................................................................................................................9-4
VVSPT-A2-PDD-013 v4.0 vii 05/16/2017

9.2 IN-SERVICE REVIEW CHECKLIST AND DEPLOYMENT DECISION PLANNING ..................................................9-5

9.3 TEST NAS CHANGE PROPOSALS ...................................................................................................................9-5
10 ACRONYMS AND GLOSSARY................................................................................................................. 10-1
10.1 ACRONYMS .................................................................................................................................................10-1
10.2 GLOSSARY ..................................................................................................................................................10-4
APPENDIX A. TEST PLANNING PROCESS CONFORMANCE CHECKLIST..................................................................A-1
APPENDIX B. DT PROCESS CONFORMANCE CHECKLIST ....................................................................................... B-1
APPENDIX C. OT PROCESS CONFORMANCE CHECKLIST .......................................................................................C-1
APPENDIX D. T&E WORK PRODUCTS AND REVIEW PROCESS CYCLES ................................................................D-1
LIST OF FIGURES
Figure 1-1. T&E Application To The V-Model........................................................................................................ 1-2
Figure 3-1. Typical T&E Approach throughout the AMS Lifecycle ........................................................................ 3-3
Figure 3-2. T&E Phases and Requirements .............................................................................................................. 3-4
Figure 5-1. Test Category Hierarchy and Documentation ........................................................................................ 5-3
Figure 5-2. TEMP Relational Diagram..................................................................................................................... 5-6
Figure 5-3. Generic TEMP Test Design Process ...................................................................................................... 5-8
Figure 5-4. Decomposition of COIs to Test Cases ................................................................................................... 5-9
Figure 6-1. Typical DT Activities............................................................................................................................. 6-3
Figure 6-2. FAA DT Process Flow ........................................................................................................................... 6-5
Figure 6-3. SAT Process Flow.................................................................................................................................. 6-6
Figure 7-1. FAA OT Process Flow ........................................................................................................................... 7-4
Figure 7-2. From Requirements to Operational Readiness Determination ............................................................. 7-15
Figure 9-1. Test Capability Accreditation Process ................................................................................................... 9-2
Figure D-1. PRD (T&E Section) Review & Approval Cycle .................................................................................. D-3
Figure D-2. ISPD (T&E Section) Review & Approval Cycle ................................................................................. D-4
Figure D-3. pTEMP, iTEMP & fTEMP Review & Approval Cycle ....................................................................... D-5
Figure D-4. T&E Input Review Cycle for FAA System Specification, SIR Proposal Requirements and Statements
of Work ........................................................................................................................................................... D-6
Figure D-5. CMTP Review & Approval Cycle ....................................................................................................... D-7
Figure D-6. DT Test Plan Review & Approval Cycle ............................................................................................. D-8
Figure D-7. DT Test Procedures Review & Approval Cycle .................................................................................. D-9
Figure D-8. DT Test Report Review & Approval Cycle ....................................................................................... D-10
Figure D-9. DT Accreditation Plan/Report Review & Approval Cycle ................................................................ D-11
Figure D-10. OT Test Plan Review & Approval Cycle ......................................................................................... D-12
Figure D-11. OT Test Procedures and Test Capability Procedures Review & Approval Cycle............................ D-13
Figure D-12. OT Interim Assessment Report Review & Approval Cycle............................................................. D-14
Figure D-13. OT Quicklook Test Report Review & Approval Cycle ................................................................... D-15
Figure D-14. OT Final Test Report Review &Approval Cycle ............................................................................. D-16
Figure D-15. Field Familiarization Support Plan Review & Approval Cycle ....................................................... D-17
Figure D-16. OT Accreditation Plan/Report Review & Approval Cycle .............................................................. D-18
LIST OF TABLES
TABLE D-1. T&E WORK PRODUCTS ......................................................................................................................... D-1
VVSPT-A2-PDD-013 v4.0 viii 05/16/2017

1 INTRODUCTION
This Handbook provides standard processes for conducting high quality and consistent Test and
Evaluation (T&E) that supports the mission of Verification and Validation (V&V).
1.1 V ERIFICATION AND V ALI DATION P RINCIPLES
As part of the FAA mission, the William J. Hughes Technical Center (WJHTC) is actively
engaged in applying effective V&V principles and practices to T&E efforts. The intent of this
initiative is to improve the quality of T&E products and services, promoting effective planning,
reducing risks, and decreasing costs.
This initiative addresses the standards for V&V process areas that are based on the Capability
Maturity Model Integration (CMMI®) 1 standards, published by the Software Engineering
Institute of Carnegie Mellon University.
In accordance with CMMI, the purpose of verification is to ensure that a system is built right,
while validation ensures that the right system is built. Verification and validation represent
complementary process areas that are distinguished below.
• Verification - Confirmation that selected work products meet their specified
requirements. This includes evaluation of the end product (system, service or operational
change) and intermediate work products against all applicable requirements. Verification
is inherently an incremental process since it occurs throughout the development lifecycle
of the work products, beginning with initial requirements, progressing through
subsequent changes, and culminating in verification of the completed end product.
• Validation - Confirmation that an end product or end product component will fulfill its
intended purpose when placed in its intended environment. The methods employed to
accomplish validation are applied to selected work products as well as to the end product
and end product components. The work products should be selected on the basis of
which are the best predictors of how well the end product and end product components
will satisfy the intended purpose and user needs. Validation can apply to all aspects of an
end product in any of its intended environments, such as operation, training,
manufacturing, maintenance or support services.
V&V is a disciplined approach to assessing a product throughout the product lifecycle. V&V
strives to ensure that quality is built into the product and that the product satisfies operational
requirements. A strong focus on validation, an industry best practice, also helps to ensure
customer satisfaction. The T&E standards defined in this Handbook support a significant portion
of a comprehensive V&V approach. Some CMMI V&V practices are executed outside of the
T&E function, such as those that apply to the systems engineering discipline, and are therefore
not addressed in this document. The relationship of how the T&E function applies to V&V is
depicted in Figure 1-1, T&E Application to the V-Model. The V-Model illustrates the
interactions between each phase of the Acquisition Management System (AMS) lifecycle and its
1
CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.
VVSPT-A2-PDD-013 v4.0 1-1 05/16/2017

associated T&E phase (i.e., T&E Planning and Support, Development Test (DT), and
Operational Test (OT)).
F IGURE 1-1. T&E A PPLICATION T O T HE V-M O DEL

The V-Model shown in Figure 1-1 represents the idea that both Verification and Validation are
performed throughout all phases of the AMS Lifecycle, from research and concept development,
through design and implementation, integration, testing and deployment.
V&V is performed in varying degrees on a continuous basis by many entities involved in an
investment program. For instance, System Engineering has a primary role in the V&V of
requirements and concepts, while the T&E organization has an equivalent role in the V&V of the
integration and test of the system or service.
Within the context of V&V, T&E practitioners play a critical role in the AMS Lifecycle. As
depicted in the diagram, the arrows on the outside of the diagram depict the phases of T&E
described in this Handbook. The arrow on the left side of the “V” represents the T&E Planning
and Support phase. The primary V&V role of T&E during this phase is to support the
VVSPT-A2-PDD-013 v4.0 1-2 05/16/2017

development of concepts, requirements, and design. The arrows on the right side of the “V”
represent the DT and OT phases. The primary V&V role during these two phases is the conduct
of T&E itself.
The arrows in the middle of the “V” are intended to show that each side of the “V” feeds the
other. For instance, the concept and requirements definition phase drives the test cases for OT,
while OT development and conduct can provide feedback for future concepts and requirements
definition.
1.2 O BJECTI VES OF THE T EST AND E VALUATION H ANDBOOK
This Handbook describes the test and evaluation processes, methods, and standards. The
objective is to ensure that a test program is complete, well managed, and conducted in a
consistent manner by the program test team throughout a system’s lifecycle, as well as achieving
V&V objectives. The processes delineated in this Handbook may be tailored to meet the specific
needs of a given FAA program as described in Section 1.6. (The use of the word “system”
within this Handbook refers to either “system” or “service”).
This Handbook is based on the T&E Process Guidelines and the Acquisition Management
Policy. Preparation of the Handbook was guided by lessons learned from past test programs,
industry best practices, International Organization for Standardization (ISO), CMMI standards,
and future FAA T&E needs.
1.3 S COPE
This Handbook addresses the T&E preparation, planning, conduct, and reporting activities in
support of FAA programs. It is intended for use by all test personnel. This Handbook is
organized into sections that first provide an overview of the T&E preparation and support
activities throughout the AMS lifecycle phases, and then define the specific T&E activities
required for each phase.
This Handbook describes the processes associated with the AMS T&E Work Breakdown
Structure (WBS) that support the objectives of the V&V process areas. The AMS defines the
T&E WBS as follows:
Test and Evaluation (AMS WBS, 3.5) - All Government test, analysis, and evaluation
activities to verify and validate that developed or modified products and product
components meet specification, satisfy program requirements, and are operationally
suitable and effective.
The T&E Handbook, including checklists, should be followed for T&E of Non-NAS systems,
NAS systems, and NAS Enterprise Level Capabilities. NAS Enterprise Level Capabilities are
operational improvements rooted in functions that are distributed across multiple systems in a
System of Systems (SoS) environment. DT test activities may need to be tailored for programs
employing “Agile Acquisition” (see 6.1.1). Wherever the provisions of this Handbook need to
be tailored, refer to Section 1.6, Tailoring of T&E Processes.
References are made throughout the Handbook to the “V&V Repository”. The V&V Repository
is an Internet-based information source hosted on servers under FAA control. The repository
provides users of this Handbook with T&E templates, samples, and tools as standard guidance to
accomplish specific V&V practices. The V&V Repository is located within the ANG-E T&E
Portal on the Knowledge Services Network (KSN) site.
VVSPT-A2-PDD-013 v4.0 1-3 05/16/2017
1.4 R OLES AND R ESPONSIBILITI ES

In conjunction with this Handbook, the WJHTC Test Roles and Responsibilities Guide (TRRG)
provides roles and responsibilities for all test personnel, as well as supporting T&E roles. The
TRRG also includes T&E management and test team structures. The TRRG describes the roles
of the DT Test Director and the OT Test Director, however, some more complex test programs
designate one person to serve as an overall Test Director for the T&E program.
The Test Standards Board (TSB) is responsible for updating and managing this Handbook in
accordance with the TRRG. The TRRG defines the organizational roles, responsibilities, TSB
processes, and management processes for administering and managing the T&E standards and
practices.
Note: Refer to the V&V Repository for the Test Roles and Responsibilities Guide.
1.5 F UNDAMENTAL P RACTICES FOR Q UALITY T&E

All T&E programs must implement the following set of fundamental practices to ensure a quality
T&E program:
(1) T&E services are involved in all lifecycle phases of the program to support and plan
for the V&V of systems, services and associated work products.
(2) T&E activities are systematic and progressive in that they build in complexity by first
focusing on basic components and then expanding to more complex and integrated
testing.
(3) Every T&E program has one or more individuals accountable for:
a) Conduct and coordination of activities
b) Overall quality of testing
c) Accepting and reporting the test results
(4) T&E activities that support acceptance of work products or approval of program
milestones are well defined and tracked.
(5) T&E programs must include the following elements:
a) Test teams that have a thorough understanding of the program requirements, and
the knowledge, skills, and training for evaluating the system under test.
b) Documented test plans, procedures, and reports that are comprehensive, clear,
concise, objective, written in plain language, peer-reviewed, and have
incorporated end user input.
c) A stable, accredited and configuration managed test environment and tools that
enable the test cases to meet the documented test objectives without numerous or
significant deviations.
d) A complete end-to-end dry run of procedures prior to the formal or final execution
of procedures.
VVSPT-A2-PDD-013 v4.0 1-4 05/16/2017

e) Accurately documented as-run test procedures and test logs for dry runs and
formal test runs.
f) Reports that provide historical test data, results, risks, deficiencies, and
recommendations with clear analysis of the performance and limitations against
planned objectives and requirements.
g) Integration and testing in an end-state environment.
(6) Prior to requirements approval, T&E personnel participate in reviews to ensure that
all program requirements are testable and validated against the operational mission.
(7) T&E is involved in the procurement package development and source selection
process to ensure that the contract complies with the test strategy and scope
documented in acquisition planning and test strategy documents.
(8) An integrated test plan (e.g., Test and Evaluation Master Plan (TEMP), see Section
5.2) is developed and completed early in the program lifecycle, and routinely
updated to reflect the evolution of the program.
(9) Operational Testing is executed in a test environment (hardware, software,
interfaces, etc.) representative of the expected in-service operational conditions.
Operational Testing is conducted with a representative team of end users who are
expected to interface with or operate the system under test.
(10) Required modifications for defects are verified and validated under conditions
equivalent to those that existed when the defects were initially identified.
Regression testing is conducted to verify and validate that fixes and modifications
do not adversely affect other interrelated operations and functionality.
1.6 T AILORI NG OF T&E P ROCESSES
Every T&E program is unique and has its own set of challenges and differences. Tailoring is a
critical activity that allows controlled changes to processes to meet the specific needs of a project
or organization.
1.6.1 TAILORING CRITERIA AND PROCESSES
To meet the unique needs of a program, the T&E processes in this Handbook may be tailored
based on:
a) Program complexity/scope
b) Risks
c) Level of effort
d) Acquisition strategies and type (e.g., Commercial Off-the-Shelf (COTS)/Non-
Development Item (NDI), Services, Software, Hardware, procurement of systems or
equipment, modification of facilities, changes in the physical infrastructure,
development of functional interfaces, spiral development implementation, etc.)
e) Program scope changes and decisions approved by the Joint Resources Council (JRC)
f) Test strategies in the approved TEMP
VVSPT-A2-PDD-013 v4.0 1-5 05/16/2017

T&E processes and process elements that are directly related to critical standards and process
objectives and that have the word “must” in the Process Description Document (PDD) should not
be tailored without a valid programmatic or technical justification. Processes defined in this
Handbook that have been changed or omitted in a program require a clear rationale that relates to
a specific unique program element or technical variable (e.g., Integration Testing may not be
required on non-complex systems). The Fundamental T&E Practices defined in Section 1.5 must
be evident and remain intact in any tailored or developed process.
The tailored program processes are documented and approved in the Process Conformance
Checklists as defined in Section 1.7. The TSB reviews and assesses the tailoring approach and
rationale documented in the compliance checklists and determines if the T&E Handbook
processes and fundamental practices for quality T&E are maintained. If the checklists are
adequate, they are signed by the TSB. Any major deviations from test standards must be
rationalized, documented, and approved in a Request for Waiver form. This request is initiated
by the Test Director or T&E First Line Supervisor. The Quality Management Lead will identify
the approval authority.
Note: Refer to the V&V Repository for the Request for Waiver form.
1.6.2 TAILORING GUIDELINES

1.6.2.1 TYPES OF TAILORING
Tailoring of T&E processes can involve the following types of adjustments to processes:
a) Change in formality
b) Change in frequency
c) Change in format
d) Modifying a process
e) Eliminating a process
f) Combining processes
g) Renaming a process
h) Changing or consolidating process roles and responsibilities
i) Changing the order of processes
1.6.2.2 COMMONLY TAILORED PROCESSES
Candidate T&E processes in this Handbook that are commonly tailored for a test program consist
of, but are not limited to, the following major processes (as defined in Sections 6 and 7):
a) DT activities
b) Required DT contractor documents
c) DT problem reporting
d) OT Interim Assessments Reports
e) Logistics for formal OT conduct support
f) Field Familiarization Support
VVSPT-A2-PDD-013 v4.0 1-6 05/16/2017
1.7 T&E S TANDARDS C ONFORMANCE P ROCESS

The T&E standards conformance process is used to monitor conformance to the T&E processes
specified in this Handbook and to provide supporting artifacts (conformance checklists) for
quality audits. This process will support the identification of required actions to resolve non-
conformance issues. This process also identifies the specific approach that the Test Director
plans to take to address required standards.
At the onset of T&E test team involvement in a test program, the DT and OT Test Directors must
prepare and submit the initial version of the Test Planning Process Conformance Checklist to the
T&E First Line Supervisor and TSB for review and approval. Prior to submission, the DT and
OT Test Directors must jointly review the Test Planning Process Conformance Checklist to
determine which test process items, if any, need to be tailored. In a similar fashion, the DT and
OT Test Directors must review their respective DT or OT Process Conformance Checklists. As
a part of the reviews, the DT and OT Test Directors must document the tailored processes and
rationale in the checklists accordingly. The initial version of the DT and OT Process
Conformance Checklists must be prepared immediately after the program Initial Investment
Decision has been made.
Each of the three Process Conformance Checklists must be maintained by the appropriate Test
Director throughout the respective T&E phases of the program to document the test team’s
progress in and conformance to the T&E process. Any additional tailoring of the approved test
process items must be noted in the revised checklist and approved by the T&E First Line
Supervisor and the TSB. Additionally, with one week prior notification to the T&E First Line
Supervisor, the TSB will periodically review each test program’s Process Conformance
Checklists against work products and progress.
Appendices A through C of this Handbook contain the Test Planning, DT, and OT Process
Conformance Checklists, respectively. These checklists delineate the test process items derived
from this Handbook. The checklists contain a column that allows for documenting tailored test
process items. The checklists also have columns for documenting status, comments, and any
associated artifacts.
Note: Refer to the V&V Repository for current versions of the Test Planning, DT, and OT
Process Conformance Checklist templates.
1.8 T&E Q UALITY M ANAGEMENT S YSTEM P ROCESSES

The T&E Quality Management System (QMS) processes address practices that are necessary for
providing quality T&E services. These QMS processes support and enable the implementation
and execution of the standards and processes defined in this Handbook. Test personnel are
required to follow these processes in order to be compliant with the policy defined in the FAA
order titled “FAA William J. Hughes Technical Center’s Test and Evaluation Policy.” The QMS
processes are documented as Process Description Documents (PDDs) and are located in the
V&V Repository. PDDs describe the processes and work-flows for each process area. Each
PDD is defined below.
a) Project Management (PM) – provides a process to manage FAA WJHTC
organization T&E projects.
VVSPT-A2-PDD-013 v4.0 1-7 05/16/2017

b) Quality Assurance (QA) – provides processes for QA audit activities in support of the
T&E goals, and provides management with insight on the level of adherence to
applicable standards.
c) Configuration Management (CM) – provides CM processes performed by
practitioners in support of T&E-related services to ensure that Configuration Items
(CIs) are managed and controlled.
d) Document Management and Control (DMC) – provides processes to manage and
control T&E and Quality Management System (QMS) documents.
e) Peer Review – provides a process to improve the quality of T&E work products and
to help verify that work products meet requirements.
f) Calibration – provides a process to define the requirements for the calibration of test,
measurement and diagnostic equipment, diagnostic software, and test data sets.
g) Corrective and Preventive Action – provides a process to identify and correct the
cause(s) of nonconformities and/or potential nonconformities in products and
processes within the T&E organization.
h) Control of Records – provides a process for identification, collection, maintenance,
protection, and storage of records.
i) Customer Feedback – provides a process to describe how the T&E organizations
receive and process feedback from customers.
j) Management Review – provides a process by which the T&E organizations review
the QMS to ensure that it is suitable, adequate, and effective in meeting customer
requirements.
k) Nonconforming Material – provides a process for identifying, reporting, segregating,
controlling and processing nonconforming test equipment, software, and custom test
setups, to prevent unauthorized use during formal testing.
l) Special Support Activities – provides a process describing how the T&E
organizations handle special projects, studies, or provide subject matter expertise
based upon the customer’s needs.
m) Quality Manual – provides a reference of processes to ensure quality testing of
systems and services entering into the NAS.
1.9 P EER R EVIEWS
Peer reviews are a critical component of V&V for T&E work products generated throughout a
product’s AMS lifecycle. Peer reviews are an effective verification practice for the early
removal of defects from selected work products and are the first mechanism for verification prior
to having partially or fully assembled product components.
Peer reviews involve a methodical examination of work products by the producer's peers as a
quality check to identify defects and to recommend changes or improvements. Peer reviews
provide strategic technical, operational, and procedural input for specific work products. The
peer review process must be conducted in accordance with the T&E Peer Review PDD.
Note: Refer to the V&V Repository for the T&E Peer Review PDD.
VVSPT-A2-PDD-013 v4.0 1-8 05/16/2017
Peer reviews should be completed prior to the final TSB review. See Appendix D for the
recommended sequence of the peer review for specific work products.
For contract deliverables (e.g., the Contractor Master Test Plan (CMTP), DT Test Plans, DT Test
Procedures, and DT Test Reports), the DT test team should check if the deliverables have
undergone internal peer reviews by the prime contractor, prior to FAA review and approval (see
Section 6.1.3.2).
VVSPT-A2-PDD-013 v4.0 1-9 05/16/2017

Intentionally Blank
VVSPT-A2-PDD-013 v4.0 1-10 05/16/2017

2 RELATED DOCUMENTS AND REFERENCES

a) Acquisition Management Policy, FAA Acquisition Management System, Washington,
DC, updated quarterly, http://fast.faa.gov/
b) Agile Acquisition Principles and Practices, FAA/MITRE, April 2016,
http://fast.faa.gov/AMSBB_AMS_Building_Blocks.cfm#FAA_Agile
c) Agile Program Management Practices for the Federal Aviation Administration,
FAA/MITRE, October 2016,
http://fast.faa.gov/AMSBB_AMS_Building_Blocks.cfm#FAA_Agile
d) Capability Maturity Model Integration (CMMI) for Development, Version 1.3,
Carnegie Mellon University CMMI Institute, Pittsburgh, PA, November 2010,
http://cmmiinstitute.com/cmmi-models.
e) CMMI for Acquisitions, Version 1.2, Carnegie Mellon University CMMI Institute,
Pittsburgh, PA, November 2010, http://cmmiinstitute.com/cmmi-models.
f) FAA AMS Lifecycle Verification and Validation Guidelines, FAA Acquisition
Management System, Washington, DC, updated quarterly, http://fast.faa.gov/
g) FAA Order 1800.66, Configuration Management Policy with Changes 1, 2 and 3,
March 2012, Washington, DC.
h) FAA Order 6032.1D, National Airspace System Modification Program, March 2011,
Washington, DC.
i) FAA System Engineering Manual, Air Traffic Organization Operations Planning,
Washington, DC. https://sep.faa.gov/policy_and_guidance/main
j) In-Service Review Checklist,
FAA Acquisition Management System, Washington, DC, updated quarterly,
http://fast.faa.gov/
k) ISO 9001:2015, Quality Management Standard, International Organization for
Standardization (ISO), Geneva, Switzerland, http://www.iso.org/iso/home.htm.
l) Order NG 1810.8A, FAA William J. Hughes Technical Center’s Test and Evaluation
Policy, October 2015, FAA William J. Hughes Technical Center, Atlantic City
International Airport, NJ.
m) T&E Handbook Templates on WJHTC V&V Repository, (reference includes all
documents that are referenced in “Notes” within the T&E Handbook).
n) Test and Evaluation Process Flowchart, FAA Acquisition Management System,
Washington, DC, updated quarterly, http://fast.faa.gov/
o) Test and Evaluation Process Guidelines, FAA Acquisition Management System,
Washington, DC, updated quarterly, http://fast.faa.gov/
VVSPT-A2-PDD-013 v4.0 2-1 05/16/2017

Intentionally Blank
VVSPT-A2-PDD-013 v4.0 2-2 05/16/2017

3 T&E SUPPORT ACROSS THE AMS LIFECYCLE

The FAA AMS Lifecycle Phases are summarized below:
a) Service Analysis & Strategic Planning and Concept & Requirements Definition -
Service Analysis & Strategic Planning and CRD begin with the identification of new
FAA service needs. It provides a basis for long-range strategic planning and is
defined as follows:
1) Service Analysis & Strategic Planning – Service Analysis & Strategic Planning is
the process used to determine what capabilities must be in place now and in the
future to meet agency goals and the service needs of customers. Results are
captured in the “as is” and “to be” states of the enterprise architecture, as well as
the roadmaps for moving from the current state to the future state.
2) Concept & Requirements Definition (CRD) – CRD translates priority operational
needs into preliminary requirements and a solution concept of operations for the
capability needed to improve service delivery. It quantifies the service shortfall in
sufficient detail for the definition of realistic preliminary requirements and the
estimation of potential costs and benefits. Finally, CRD identifies the most
promising alternative solutions able to satisfy the service need.
In addition to the activities defined above, research projects often support and provide
information to this first phase of the AMS lifecycle.
b) Investment Analysis (IA) – IA begins after the completion of CRD with the
Investment Analysis Readiness Decision (IARD). IA is the process used to support
capital investment decisions and consists of the following two components:
1) Initial Investment Analysis – Initial Investment Analysis evaluates alternative
solutions to mission needs and provides realistic options to the JRC to support the
Initial Investment Decision (IID).
2) Final Investment Analysis – Final Investment Analysis develops detailed plans
and final requirements for the IDA-selected investment opportunity in support of
the Final Investment Decision (FID).
c) Solution Implementation (SI) – SI begins at the FID when the IDA gives approval to
the service organization to proceed with implementation of the selected investment
opportunity. The overarching goal of SI is to satisfy program requirements and
achieve the benefit targets in the Business Case Analysis (see Section 5 for a
description of the Business Case Analysis).
d) In-Service Management (ISM) – ISM begins at the In-Service Decision (ISD) when a
new system or service is commissioned for operational use. The primary goal of ISM
is to ensure that FAA systems, services, and facilities are operated, maintained,
secured, and sustained in real-time to provide the level of service required by users
and customers. In addition, ISM provides periodic monitoring, evaluation, and
improvements for fielded systems, services, and facilities.
Figure 3-1 illustrates the typical T&E approach leading to the ISD. The diagram depicts all
major AMS lifecycle phases starting with Research for Service Analysis through Service
VVSPT-A2-PDD-013 v4.0 3-1 05/16/2017
Analysis & Strategic Planning and CRD, IA, SI, and ending with ISM. The diagram also
indicates the key decisions, milestones, major test efforts, technical reviews, and most
importantly, the major T&E work products and when they would be initiated relative to the AMS
lifecycle phase. This figure depicts the typical FAA investment program lifecycle, and the
implementation of specific activities (i.e., test efforts or T&E work products) or events (i.e., key
decisions, milestones, or technical reviews).
Note: Each program is unique and may deviate from the typical lifecycle approach.
Sections 4 through 8 of this Handbook identify and define the specific T&E support required for
each of the AMS program lifecycle phases. The AMS Lifecycle phases shown in Figure 3-1 are
to be followed for any investment which has been identified by FAA as requiring executive
decision.
NAS Enterprise Level Capability programs need to be coordinated across multiple systems, each
of which has their own requirements. Enterprise Level Capability T&E teams need to be put in
place to:
a) Identify the hierarchy of requirements,
b) Allocate the requirements to appropriate organizations, and
c) Establish roles, responsibilities, and procedures for coordinating T&E activities,
including problem resolution.
Figure 3-2 illustrates test phases for both System/Solution level T&E, and NAS Enterprise Level
Capability T&E. The arrows show which set of requirements undergoes T&E in each test phase.
VVSPT-A2-PDD-013 v4.0 3-2 05/16/2017

F IGURE 3-1. T YPICAL T&E A PPROACH THRO UGHOUT THE AMS L IFECYCLE
VVSPT-A2-PDD-013 v4.0 3-3 05/16/2017
F IGURE 3-2. T&E P HASES AND R EQ UIREMENTS
VVSPT-A2-PDD-013 v4.0 3-4 05/16/2017

4 T&E SUPPORT TO SERVICE ANALYSIS & STRATEGIC

PLANNING AND CONCEPT & REQUIREMENTS DEFINITION
T&E support to Service Analysis & Strategic Planning and CRD generally consists of early
evaluations performed in support of concept feasibility determinations and analysis of alternative
solutions. Early evaluations may be used throughout the program planning process (i.e., during
both Service Analysis & Strategic Planning and CRD and IA) to minimize program risks. Early
evaluations include:
a) Prototype tests
b) User demonstrations
c) Studies
d) Modeling
e) Simulations
f) Operational Capability Demonstrations (OCDs)
g) Operational Capability Tests (OCTs)
Prototype tests, user demonstrations, studies, modeling, and simulations are used to support
system engineering processes in developing requirements, evaluating operational concepts
(including human factors), and selecting technologies. OCDs and OCTs are used to provide
functional and performance information to sponsors, to support selection from among candidate
solutions.
These evaluations may be conducted on functionally equivalent prototypes or on early
development systems, and may be used to provide field personnel with early exposure to new
systems under development. The focus is to identify product flaws or possible enhancements
prior to generating production systems (i.e., when it is least costly to implement corrective
measures or improvements). Information from early evaluations is used to develop work
products for:
a) Evaluating operational concepts
b) Supporting alternative analysis decisions
c) Developing requirements for the preliminary Program Requirements Document
(pPRD)
In keeping with best practices, early evaluations should adhere to the same fundamental T&E
practices described in Section 1.5. Early evaluations may result in changes to product
specifications, Commercial Off-the-Shelf (COTS) hardware or software, operational
requirements, or vendor selections. These evaluations can also provide information for the focus
of future test activities. The test processes from the DT and OT sections of this Handbook
(Sections 6 and 7) should be used during these evaluations to ensure a systematic approach and
properly documented results.
VVSPT-A2-PDD-013 v4.0 4-1 05/16/2017

Intentionally Blank
VVSPT-A2-PDD-013 v4.0 4-2 05/16/2017

5 T&E SUPPORT TO INVESTMENT ANALYSIS

T&E support to IA consists of the initial development of T&E program planning as documented
in the Program Requirements Document, Implementation Strategy and Planning Document, and
the Test and Evaluation Master Plan. Additionally, T&E support during this phase may include
the continuation (or the initiation) of early evaluations begun during Service Analysis &
Strategic Planning and CRD (see Section 4).
The set of acquisition planning and control documents generated during IA are as follows:
a) Program Requirements Document (PRD) – defines the operational framework and
performance baseline for an acquisition program. It is the basis for evaluating the
readiness of products and services of an acquisition program to become operational
b) Implementation Strategy and Planning Document (ISPD) – provides a summary of
the plans for SI and ISM management of the proposed acquisition to the JRC
c) Acquisition Program Baseline (APB) – defines the cost, schedule, and performance
baselines for the system or service acquisition. It is the mutual agreement between
the JRC, the providing service organization, and the operating service organization
concerning the performance and capability that the system or service will provide and
the authorized cost and schedule
d) Business Case Analysis – provides summary cost, schedule, and benefit information
for each alternative solution to the mission need for use by the JRC when making
initial and final investment decisions. (Note: The Business Case Analysis is not
directly used for T&E program planning, but may provide useful background
information for T&E strategy development)
The planning and support described in this section details the T&E program planning based on
the framework established in the program acquisition planning and control documents.
Specifically, the T&E support to the development of the PRD and ISPD, along with the
development of the TEMP, is described in Sections 5.1 through 5.6. Additionally, Appendix D
identifies the major T&E work products along with their associated review and approval cycles.
5.1 P ROGRAM R EQUIREMENTS D OCUMENT S UPPORT
The PRD is developed by the Program Management Office (PMO), and approved by the Vice
President of the organization executing the investment program during SI and the Vice President
of the operating organization. In accordance with the FAA AMS, the pPRD is developed during
the CRD phase in preparation for the Investment Analysis Readiness Decision. Program
requirements are refined in the initial PRD prior to the Initial Investment Decision and are
further refined in the final PRD prior to the Final Investment Decision. The T&E section of the
PRD is developed during the initial PRD (iPRD) in support of the Initial Investment Decision
(IID).
VVSPT-A2-PDD-013 v4.0 5-1 05/16/2017

T&E support for the development of program requirements during PRD development includes:
a) Participating in product engineering and implementation reviews
b) Ensuring that all new or modified requirements for functions or services are defined
and address the operational mission
c) Ensuring that all interfaces required for the NAS operational mission are defined
d) Reviewing and commenting on requirements for testability. (Requirements must be
precisely defined and leave no room for subjective interpretation. Parameters and
thresholds must be measurable. Functional performance requirements must be
verifiable and expressed in unambiguous terms)
e) Verifying that Critical Operational Issues (COIs) are completely described, are
operational in nature, represent observable events, and are testable
f) Verifying that program requirements essential to meeting the mission are identified as
Critical Performance Requirements (CPRs)
g) Structuring the test program to address all target system or subsystem components
and interfaces by:
1) Defining potential test strategies and seeking feedback from engineering and
implementation teams
2) Providing test strategy briefings to the PMO as required
3) Writing the T&E section of the PRD to include essential FAA and contractor tests
The T&E section of the PRD must be reviewed by the TSB, PMO, and the T&E Senior Manager,
but no formal approvals or endorsements are required. Refer to Appendix D, Figure D-1, for the
complete T&E section of the PRD review and approval cycle.
Note: Refer to the V&V Repository for a sample of the T&E section of the PRD, and for
Requirements Review Guidance, TSPAT-D3-GDE-004.
5.2 T EST AND E VALUATION M ASTER P LAN D EVELOPMENT

The TEMP is the primary test management document for the investment program throughout its
lifecycle from Investment Analysis through In-Service Management. It describes the baseline
test strategy and the scope of a test program. The TEMP delineates all activities that must be
performed to achieve the goals of V&V. It also documents the T&E methodologies that will be
used to assess safety hazard controls and security risks. All programs will be required to submit
a TEMP, unless a waiver is approved by the Technical Center Director. Figure 5-1 shows a
hierarchy of test categories and the scope of the TEMP relative to other plans and procedure
documents.
The DT and OT Test Directors are responsible for producing and managing the TEMP through
the ITT. The program follows the approved TEMP to budget for T&E resources and scheduled
activities. It is imperative that all T&E stakeholders participate early in the program and
throughout the program lifecycle to ensure that a comprehensive and effective test program is
developed and implemented.
VVSPT-A2-PDD-013 v4.0 5-2 05/16/2017

F IGURE 5-1. T EST C ATEGORY H IERARCHY AND D OCUMENTATION
VVSPT-A2-PDD-013 v4.0 5-3 05/16/2017

The preliminary TEMP (pTEMP) defines the investment program test strategy and scope. It is
developed upon the concepts and functions documented in the preliminary requirements
document prior to IID and is not expected to contain the complete level of detail required to fully
implement the T&E program. The iTEMP for each T&E program must be submitted by the DT
and OT Test Directors for approval by the Program Manager and the Technical Center Director
prior to the Final Investment Decision. The iTEMP is required for the Final Investment Decision
(FID). The iTEMP also is not expected to contain the complete level of detail required to fully
implement the T&E program. However, the iTEMP must contain estimates of the testing scope
that are sufficient to address ISPD requirements and development of T&E requirements for the
Screening Information Request (SIR).
The TEMP is a living document and is updated as the program progresses and more detailed
supporting information becomes available. Unaddressed or incomplete areas within early TEMP
versions which require refinement must be identified as “To Be Determined,” (TBD), and must
be included in the final or revised final versions as additional information becomes available.
The final TEMP (fTEMP) should be completed after design reviews, such as Critical Design
Review (CDR), are completed, and prior to delivery of the CMTP, if applicable, and is generally
revised at major program milestones (see Section 5.2.2). The pTEMP, iTEMP, fTEMP, and all
revisions to the fTEMP impacting test strategy or scope require TSB endorsement and the
approval signatures of the Program Manager and the Technical Center Director. Minor changes
to the TEMP that do not impact test strategy or scope (e.g., minor schedule changes, editorial
corrections, etc.) will be considered working drafts only and do not require these approval
signatures but are still subject to Document Management and Control. The T&E strategies and
methods described in the TEMP should be briefed to the Program Manager to obtain input and
concurrence with the general T&E approach. The briefing should occur prior to submitting the
initial and fTEMP for the Program Manager’s review and approval.
Refer to Appendix D, Figure D-3, for the complete TEMP review and approval cycle.
Note: Refer to the V&V Repository for the TEMP templates and TEMP samples.
5.2.1 TEMP OBJECTIVES
The TEMP must accomplish the following objectives to ensure a comprehensive test program:
a) Provide structure for managing the T&E effort
b) Define a test plan baseline that addresses all required test activities
c) Document the ITT’s consensus on the scope of testing that is required to evaluate the
system or service in a thorough and efficient manner
d) Define test strategies and evaluation methods for the system(s) or service(s) under test
e) Identify program requirements to be verified and validated, also noting those that will
not be evaluated.
f) Identify the CPRs from the PRD and associated criteria
g) Document traceability of program requirements and COIs to test activities in a
Verification Requirements Traceability Matrix (VRTM).
h) Decompose COIs into Measures of Effectiveness (MOEs), Measures of Suitability
(MOSs), and Measures of Performance (MOPs) in the VRTM.
i) Define a logical schedule of T&E activities
VVSPT-A2-PDD-013 v4.0 5-4 05/16/2017

j) Specify equipment and resource requirements to support testing

k) Specify T&E roles and responsibilities
l) Define an approach for specialized evaluation areas (e.g., human factors, security, and
safety)
m) Identify test tools and capabilities that require development and accreditation
n) Fully define test limitations and their respective impacts on evaluation elements
o) Define training requirements and plans
p) Define the T&E plan for support of major program decisions
q) Define the plan for making a determination of Operational Readiness
r) Define how the results of testing will be reported (e.g., Interim Assessment Reports,
DT Reports, OT Caucus Summary Report, and other OT Reports)
5.2.2 TEST MANAGEMENT USING THE TEMP

The information necessary to develop and maintain a TEMP and the process areas that are
managed through the use of a TEMP are depicted in Figure 5-2, TEMP Relational Diagram. The
following sections expand upon the relationships depicted in this figure.
The acquisition planning and control documents (see Section 5) and each program’s FAA
procurement specification are the basis of the programmatic test objectives and constraints upon
which the TEMP must be developed. The test strategies defined in the TEMP must support
those contained in the ISPD. Development of the TEMP may start prior to completing the test
section of the ISPD. The TEMP must contain more refined details that support the higher level
descriptions contained in the ISPD.
The TEMP contains strategies to be used to plan the conduct of any Operational Capability
Demonstrations (OCDs) and Operational Capability Tests (OCTs). The DT section of the TEMP
defines the test strategies and scope to be allocated to DT. Since the TEMP contains the
integrated test approach for evaluating the system or service, it must define the details of test and
support requirements to be obtained from the procurement contractor in support of both DT and
OT. These requirements are to be included in the SIR. The TEMP will be updated, as required,
based on the results and findings of the OCDs, OCTs, and DT.
The TEMP must define the scope of the OT effort. Subsequent OT plans must reflect the test
strategies and scope defined in the TEMP. The basis of the strategies will be defined in the
TEMP, including which DT activities will be needed to support OT validation of requirements.
The simulation, modeling, and/or test tools required to execute the OT strategy must be defined
so that their development and accreditation can be effectively managed.
As the system development progresses, the strategies defined in the TEMP must be reassessed.
Changes to the TEMP are required when previously developed test strategies are no longer valid,
or if new system performance risks are identified. The TEMP must be reassessed and updated as
necessary:
VVSPT-A2-PDD-013 v4.0 5-5 05/16/2017

F IGURE 5-2. TEMP R ELATIONAL D I AGRAM

a) At major milestones (e.g., Contract Award, CDR, Contractor Master Test Plan
(CMTP) delivery/update, and OT Test Plan delivery/update)
b) When the program baseline has been significantly modified
c) When the ISPD has been significantly modified
d) When the program is significantly changed or restructured
VVSPT-A2-PDD-013 v4.0 5-6 05/16/2017

5.2.3 TEST DESIGN PROCESS

The ITT develops the test design that is documented in the TEMP by researching information
from various sources including Concept of Operations (CONOPs), program requirements,
associated architectures, specifications, COTS/NDI documentation, functional descriptions, and
design documentation.
The following sections describe the test design process as depicted in Figure 5-3, Generic TEMP
Test Design Process. Early steps of this process may not be fully applicable to a pTEMP.
5.2.3.1 IDENTIFICATION AND ASSESSMENT OF REQUIREMENTS
The initial step in developing a test design is to identify a complete set of requirements within
the scope of the test effort. System requirements and operational requirements are defined in the
program’s PRD and FAA system specification(s). Each requirement must be thoroughly
analyzed to gain a comprehensive understanding of the system or service to be tested.
To perform the system requirements assessment, the ITT considers the following aspects of the
system or service to be tested:
a) Mission
b) System or service requirements
c) Available test capabilities and limitations
d) Application and operational concepts
e) Major operational changes or impacts on the NAS that may be introduced by the new
program or service
f) Acquisition strategies and issues that may impact the T&E process
g) Expected scope and size of software and hardware development efforts
h) Safety and security risks
i) Operational environments, configurations, and conditions in which the system is
intended to be placed.
5.2.3.2 DECOMPOSITION OF CRITICAL OPERATIONAL ISSUES
Critical Operational Issues (COIs) are defined in an FAA procurement program’s PRD. A COI
is a critical element or operational objective that must be examined during testing to determine
the system’s overall capability to support mission accomplishment. COIs are stated in the form
of a question at a high level and usually cannot be answered directly from a single test or
measurement. Therefore, each COI must be broken down into quantifiable measures that, when
combined, fully support resolution of the critical issue.
COIs are decomposed in the TEMP by:
a) Decomposing COIs into Measures of Effectiveness (MOEs) and Measures of
Suitability (MOSs). MOEs and MOSs are detailed sub-questions of the COI which
must be answered to fully determine whether a COI has been satisfied.
b) Further decomposing MOEs and MOSs into Measures of Performance (MOPs).
MOPs are measurable, quantitative/qualitative values that characterize and support
VVSPT-A2-PDD-013 v4.0 5-7 05/16/2017

the evaluation of the COIs, MOEs and MOSs. Decomposition must be considered
preliminary for the pTEMP and iTEMP.
c) Mapping MOEs and MOSs to PRD requirements in the VRTM. If a conflict is
discovered between the PRD and any MOEs, MOSs or MOPs the conflict must be
resolved.
F IGURE 5-3. G ENERI C TEMP T EST D ESI GN P ROCESS

VVSPT-A2-PDD-013 v4.0 5-8 05/16/2017
Figure 5-4 illustrates the decomposition of COIs down to test cases, and their relationship to
operational readiness.
F IGURE 5-4. D ECOMPOSITION OF COI S TO T EST C ASES

An MOE evaluates the degree to which a system accomplishes its mission when used by
representative personnel in the expected operational environment. Examples of Effectiveness
measures include:
a) Accuracy
b) Coverage
c) Probability of detection
d) Tracker performance
e) System capacity
f) Data timeliness
An MOS addresses system characteristics that determine whether it is suitable for use in the
NAS. Suitability measures include:
a) Reliability
VVSPT-A2-PDD-013 v4.0 5-9 05/16/2017

b) Maintainability
c) Availability
d) Compatibility
e) Transportability
f) Interoperability
g) Safety
h) Human factors
i) Logistics
j) Supportability
k) Documentation
l) Personnel
m) Training requirements
n) Site adaptability
o) Fault tolerance
p) Security
Note: Refer to the V&V Repository for the COI Decomposition Guide and a COI
Decomposition sample.
5.2.3.3 CRITICAL PERFORMANCE REQUIREMENTS EVALUATION APPROACH
CPRs are program requirements identified in the PRD as essential to the successful performance
of the system or service in meeting the program’s mission needs. Special emphasis is placed
upon the evaluation of CPRs to ensure the timely assessment of system or service capabilities
and to promote program success.
Therefore, CPRs are tracked and their status is reported throughout the test program to provide
visibility to management in support of making prudent decisions. The evaluation approach
should determine the strategy for testing the CPR and the required milestones for assessing CPR
performance.
If CPRs have not been identified in the PRD or the APB, then it is recommended that the ITT
select a PMO-approved set of CPRs from the PRD, in accordance with the PRD template
guidance. Additional CPRs may also be identified from other program work products. The full
set of CPRs is documented in the TEMP.
5.2.3.4 DEFINING TEST ACTIVITIES
Requirements are allocated to a set of DT and OT activities to support a structured evaluation of
the system. In addition the COIs and CPRs identified above, test design should consider the
following factors affecting test:
a) NAS integration requirements
b) System interface requirements testing
VVSPT-A2-PDD-013 v4.0 5-10 05/16/2017

c) Degraded mode operations

d) Stress and NAS loading
e) Failure recovery
f) Air Traffic (AT) and Technical Operations evaluations
g) System or service certification requirements
h) Transition and switchover/fallback
i) Site adaptation
j) Human Factors (HF) evaluations
k) Security evaluations
l) Supportability evaluations
m) Safety requirements and potential new safety hazards
n) Monitoring and control
o) Integrated operations
p) Stability
q) Hardware requirements
r) Software requirements
s) Functional performance
t) Reliability, Maintainability, and Availability (RMA)
u) Test NAS Change Proposals (NCPs) (see Section 9.3)
For DT, standard test activities (as defined in Section 6.1.1) are identified as DT Software
Testing, DT Hardware Testing, Factory Acceptance Testing (FAT), Functional Qualification
Testing (FQT), DT Integration Testing, DT System Testing, Production Acceptance Testing
(PAT), Site Acceptance Testing (SAT) and DT Regression Testing.
For OT, standard test activities (as defined in Section 7.1) are identified as NAS Integration
Testing, and Operational Effectiveness and Suitability Testing. For the latter, this test activity
may be further subdivided into the specific test activities of Reliability, Maintainability,
Availability, Supportability, Degraded Operations, Stress and NAS Loading, Human Factors,
Safety, Security, Site Adaptation, Transition, Certification, Training and OT Regression Testing.
As part of CMTP, DT Test Plan, and OT Test Plan development, test activities are broken down
into individually executable test cases in which functionally related requirements are tested. For
larger or more complex test programs, individual test cases within a test activity may be
organized into test case groups for execution and/or reporting purposes. Ultimately, conditions
for each test case should define the required loads, configurations, adaptations, and
environments. Allocation of test activities and their individual test cases should support the test
strategies defined in the T&E section of the ISPD.
VVSPT-A2-PDD-013 v4.0 5-11 05/16/2017

5.2.3.5 FORMULATING THE TEMP VERIFICATION REQUIREMENTS TRACEABILITY MATRIX

The TEMP VRTM provides a structured approach that ensures planned testing is comprehensive
and complete. It maps all COIs/MOEs/MOSs/MOPs as well as the program requirements into
T&E phases (DT or OT) and associated test activities (see Sections 6.1.1 and 7.1 for descriptions
of the DT and OT activities). Additionally, each MOP and program requirement contained in the
TEMP VRTM is assigned one or more of the following test verification method(s):
a) Demonstration (D): Verification that is accomplished by operation, adjustment, or
reconfiguration of items performing their designed functions under specific scenarios.
The items may be instrumented and quantitative limits of performance monitored, but
only observational data rather than actual performance data is required to be recorded
for verification. Demonstration is often used to verify compliance with requirements
in servicing, reliability, maintainability, transportability, and human factors
engineering.
Note: Demonstration does not require any actions beyond those identified in the
Test Steps section of the associated test procedures.
b) Test (T): Verification that is accomplished, with or without instrumentation, through
systematic exercising of the application item under appropriate conditions with the
collection, analysis, and evaluation of quantitative data.
Note: Acceptability of the item is determined by comparison of the data with pre-
established quantitative criteria, requirements, and occurrences.
c) Analysis (A): Verification that is accomplished through use of one or more of the
following analysis techniques to prove that an item meets specified requirements:
1) Mathematical representation such as math models, algorithms, and equations
2) Charts
3) Graphs
4) Circuit diagrams
5) Data reduction/recording
6) Representative data (may include data collected from previous or other equipment
and system verifications)
d) Inspection (I): Verification that is accomplished by a visual examination of the item,
reviewing descriptive documentation, and comparing the appropriate characteristics
with predetermined standards to determine conformance to requirements without the
use of laboratory equipment or procedures. Examples of verification by inspection
are:
1) Visual analysis of the item under test, such as displays, cables, and processors.
2) Reviewing descriptive documentation such as Contract Data Requirements List
(CDRL) items, vendor data, and engineering drawings.
3) Comparing the appropriate characteristics with a predetermined or reference
standard such as FAA and industry standards.
VVSPT-A2-PDD-013 v4.0 5-12 05/16/2017

Development of the VRTM is an iterative process. During early TEMP development, the TEMP
VRTM is generated with a mapping of COIs/MOEs/MOSs/MOPs and program requirements
down to the test activity level for OT and to the DT phase, as appropriate. Detail below the COI
level must be considered preliminary and may need to be revised and expanded as OT planning
progresses. As the program progresses, additional information such as that included in the ISPD,
the CMTP, and the FAA specification will provide further details that must be incorporated into
the VRTM.
For DT, the developing contractor will use the FAA System Specification to develop their
contractor specification(s) defining the DT requirements. The CMTP will then be developed,
containing a DT VRTM which maps the DT requirements to the DT activities. Subsequently,
the developing contractor will refine the DT VRTM with test case information during DT Test
Plan development (see Sections 6.2.2, 6.2.3, and 6.2.5). The TEMP VRTM should be updated
with the DT activity and test case information from the DT VRTM.
For OT, the TEMP VRTM will provide the basis for developing the OT VRTM. During OT
Test Plan development, the VRTM will be further refined to include COI/MOE/MOS/MOP and
program requirement mapping to the individual OT cases (see Sections 7.4.1 and 7.4.2).
Note: Refer to the V&V Repository for the VRTM template.
5.2.3.6 IDENTIFYING TEST CAPABILITY REQUIREMENTS

Test tools and environment requirements should be identified in a consolidated list for
traceability, including associated “need by” dates. Particular focus is placed on items with long
lead times or high costs. Typical examples include the preparation, procurement, or
development of laboratories and testbeds, models and simulations, test tools, telecommunications
services, and NAS interfaces that require significant coordination.
When assessing the need for test tools and environments, the following factors should be
considered:
a) Measures of Performance: Reviewing MOPs provides insight into the resources that
may be required to collect and analyze data. Instrumentation, databases, data
extraction tools, questionnaires, and analysis tools are examples of resources that may
be required.
b) Test conditions: Reviewing the required test conditions (e.g., loads, configurations,
adaptations, and environments) provides insight into the required resources.
c) Test methodology: How a set of test activities is executed also contributes to the
rationale or basis for resource requirements. Reviewing the test methodologies
provides insight into what might be needed to execute the test activities (e.g., field
testing may add costs, modeling and simulation may incur both increased costs and
potentially long lead times for development, etc.).
d) Test scenarios: Test scenarios are simulations, or scripted input developed to exercise
combinations of operational factors that influence system performance. Reviewing
all factors that have been identified may provide insight into additional assets that
may be required to approximate operationally realistic Air Traffic Control (ATC)
situations and capture “end-to-end” situational flow.
VVSPT-A2-PDD-013 v4.0 5-13 05/16/2017

5.2.3.7 DEFINING TEST SCHEDULE

Test program schedule and resources are derived from and based on the test design and
associated program constraints. Test program schedule and resources can be impacted by
program constraints which should be identified using the Risk Management process defined in
the NAS System Engineering Manual (SEM). The test team can draw on historical data from
test programs of equivalent complexity and risk to develop the test design for a program. This
data may be accessed from T&E work product archival data from other test programs. Historical
data may also be used to assess T&E tasks to establish durations and resource estimates. The
initial test schedule milestones and completion dates are developed by linking them to key
program milestones and acquisition decision points. The test schedule is further refined by
defining T&E tasks for each test milestone and completion date. Finally, the test schedule
dependencies are established by linking all associated T&E tasks, as well as other program
milestones and completion dates that relate to the T&E tasks. This final schedule becomes the
test program baseline.
All planning and management of T&E program costs and schedules should be done in
accordance with the T&E Project Management Process Description Document (PDD).
Note: Refer to the V&V Repository for the Project Management PDD.
5.2.3.8 ASSESSING UPDATES AND ACQUISITION PROGRAM BASELINE CHANGES

As additional information becomes available or changes to the APB are made, the test design
must be reassessed to ensure that the assumptions made in the test design are still valid. If
changes in assumptions impact the ability of the test design to verify and validate the system
under test, changes must be made to the test design.
5.2.3.9 DOCUMENTING THE TEST DESIGN IN THE TEMP
The TEMP documents the test design from a set of defined process steps as depicted in Figure
5-3. As described in the previous sections, the test design is based on the following factors: the
complete set of system requirements; the decomposition of COIs that must be examined during
testing; the set of CPRs that must be evaluated, tracked and assessed; a set of DT and OT
activities that support a structured evaluation of the system; a structured VRTM to ensure that
testing is comprehensive and complete; a set of necessary test capability requirements; the test
schedule, cost and resources; and an assessment of updates and APB changes.
When an investment program’s test design is completed and documented, its supporting factors
form the basis for the TEMP and any updates.
5.2.3.10 INTERIM ASSESSMENT REPORT PLANNING
The Interim Assessment Report (IAR) can play an important role in the early identification of
program issues, risks and change opportunities. It is a recommended method to provide
management with the status of critical requirements and operational issues during a program’s
design, development, and test phases. This assessment can support programmatic decisions to
help ensure that the program meets its cost, schedule or performance objectives prior to going
into operations. Test program IAR planning activities include describing how and when IARs
will be provided, and to whom the IARs will be delivered. Reporting milestones and methods
for IARs are typically based on cost, schedule and complexity of the program. The selection of
VVSPT-A2-PDD-013 v4.0 5-14 05/16/2017

milestones for when IARs are planned to be delivered should consider when critical decisions
points in the program can be best supported by the report and the availability of meaningful data.
Recommended reporting milestones include major reviews (e.g., preliminary, critical, software,
system and program design reviews), software development, start of formal DT, completion of
DT, and start of OT. The OT Test Director documents strategic plans for IARs in the TEMP and
is the responsible individual for the deliverable. For more on the IAR content, see Section 7.7.1,
OT Interim Assessment Report.
5.3 I MPLEMENTATION S TRATEGY AND P LANNI NG D OCUMENT

S UPPORT
The ISPD defines the overall strategy for an FAA program. The T&E section of the ISPD
defines the program’s T&E strategy. The ISPD T&E strategy identifies high-level aspects of
testing, including resources, roles and responsibilities, site selection, scheduling, training,
planning, and reporting. It addresses any planned product upgrades, and contractor and
Government roles and responsibilities.
T&E support for the ISPD includes supporting the development of the T&E section. System
complexity, contracting approach, project schedule, and impact on the NAS are considered when
developing the test program strategies. Both contractor and FAA test activities are addressed
with an overall structure and outline of planned testing. Any approach to streamlining testing is
identified with an explanation of how the test strategy will mitigate program and operational
risks.
The T&E section of the ISPD is developed by the DT and OT Test Directors in conjunction with
the initial TEMP (iTEMP, see 5.2 for definitions). The T&E section of the ISPD should include
the plan for delivery of the fTEMP relative to other program milestones and should be consistent
with the strategies detailed in the iTEMP. The development of the T&E section of the ISPD is
supported by the Integrated Test Team (ITT). The ITT is comprised of PMO representatives,
stakeholders, DT and OT personnel, ISM test personnel, system engineers, Subject Matter
Experts (SMEs), and is jointly chaired by the DT and OT Test Directors. (For additional
information on the ITT, refer to the TRRG.)
The T&E section of the ISPD requires endorsements from the TSB, T&E First-Line Supervisor,
and the responsible T&E Senior Manager, before final delivery to the PMO. The endorsements
must encompass review of the entire final ISPD. The approved ISPD with TSB endorsement is
delivered to the Technical Center Director. The T&E section is then ready for final
incorporation into the ISPD by the PMO. On programs designated for Independent Operational
Assessment (IOA), the Independent Safety Assessment Team also approves the ISPD section
9.2.
Refer to Appendix D, Figure D-2, for the complete T&E section of the ISPD review and
approval cycle.
Note: Refer to the V&V Repository for a sample of the T&E section of the ISPD.
5.4 FAA S YSTEM S PECIFICATIO N S UPPORT

The DT and OT Test Directors and designated test team members participate in developing the
FAA System Specification. The specification describes the physical, functional, or performance
VVSPT-A2-PDD-013 v4.0 5-15 05/16/2017
requirements of a system or service to be obtained from a contractor and contains verifiable

criteria for determining whether or not the requirements are met. The test teams evaluate the
testability of each specified requirement and assess the method of how each requirement should
be verified. The Test Directors and test teams also develop the verification section of the
specification. The following activities are performed in reviewing and developing requirements
in the FAA System Specification:
a) Review proposed software, hardware, and system designs as defined by system
engineering teams.
b) Conduct prototype evaluations (as required) to support requirements definition.
c) Attend product engineering and implementation reviews.
d) Ensure that requirements for all new and affected system components or subsystems
are defined and that they address operational requirements.
e) Ensure consistency with operational requirements in the PRD.
f) Evaluate the validity of the requirements from an operational perspective.
g) Ensure that all critical interfaces are defined.
h) Review and comment on requirements for testability:
(1) Requirements related to COTS/NDI products should only describe functional
system level requirements, since subsystem level parameters are already
documented in the vendor COTS/NDI specifications.
(2) Requirements must be clearly defined and leave no room for subjective
interpretation. Parameters and thresholds must be measurable. Functional
performance requirements must be written in unambiguous terms. Standards for
well-written requirements are provided in the Requirements Review Guidance,
TSPAT-D3-GDE-004, and in the FAA Systems Engineering Manual.
Note: Refer to the V&V Repository for the Requirements Review guidance.
i) Assess the product components, system design, and system implementation approach
for developing potential test strategies.
j) Define potential test strategies and seek feedback from implementation and
engineering teams. Provide test specification briefings to the PMO, as required, to
attain buy-in.
k) Develop test program structure to address target system components, subsystems, and
critical interfaces.
l) Write the verification section of the FAA System Specification to define essential
verification areas and methods and strategies that are required to verify the
documented requirements.
m) Ensure that operational considerations have been incorporated in the selection of
verification methods (inspection, analysis, demonstration, and test).
Note: Refer to the V&V Repository for an FAA System Specification sample.
VVSPT-A2-PDD-013 v4.0 5-16 05/16/2017

The DT Test Director must ensure that all T&E-related input to the FAA System Specification
has been peer-reviewed prior to submission (see Section 1.9). Refer to Appendix D, Figure D-4,
for the T&E Input Review Cycle for FAA System Specification.
5.5 S CREENING I NFORMATION R EQ UEST (SIR) S UPPORT
The FAA procures systems or services from contractors using agreements defined in contracts.
Before it can select a contractor to provide the system or service, the FAA issues a SIR to define
the specific efforts to be provided by the contractor. The test strategy from the TEMP is the
basis for determining what test and evaluation items belong in the SIR. The DT and OT Test
Directors must ensure that this test strategy is properly reflected in the SIR as described in the
following sections. Additionally, the DT and OT Test Directors must ensure that all T&E-
related input to the SIR has been peer-reviewed prior to submission (see Section 1.9).
5.5.1 SECTION L OF THE SIR: INSTRUCTIONS, CONDITIONS, AND NOTICES TO
OFFERORS
Section L of the SIR contains a description of the technical requirements that the contractor must
address within the proposal that they submit in response to the SIR. Contractor proposal
requirements that may be addressed in Section L of the SIR with respect to testing include:
a) Relevant experience of the contractor in testing similar systems
b) DT cost and schedule information
c) Test tools to be used
d) Test environment(s) proposed for formal tests
e) Integration and test management approach
f) Proposed test approaches and strategy to include expected conditions for CPR
evaluation
g) Specific operational conditions and loading that supports early evaluation of
operational capabilities during DT
h) Cost information for OT support
i) Test Configuration Management (CM) methods and practices to be used
j) OCD/OCT evaluation criteria (if required)
k) Risk management approach
5.5.2 SECTION C OF THE SIR: DESCRIPTION OF WORK REQUIREMENTS
Section C of the SIR contains the description of the work to be performed under the contract.
This section typically includes one of the following Government developed documents:
Statement of Work (SOW), Performance Work Statement (PWS) or Statement of Objectives
(SOO). The subsequent sections define each document purpose followed by a list of T&E areas
that should be considered when drafting the description of work requirements.
5.5.2.1 STATEMENT OF WORK (SOW)/PERFORMANCE WORK STATEMENT (PWS)
The SOW defines the specific tasks that the contractor must perform. The SOW should specify
in clear, understandable terms the work to be done in developing or producing the goods to be
VVSPT-A2-PDD-013 v4.0 5-17 05/16/2017
delivered or services to be performed by the contractor. The SOW forms the basis for successful
performance by the contractor and effective administration of the contract by the government. A
well-written SOW serves as the standard for determining if the contractor meets the stated
requirements.
A PWS is a statement of work for Performance-Based Service Acquisitions (PBSAs) that
describe the required results in clear, specific and objective terms with measureable outcomes.
The key to a PBSA is that all aspects of the acquisition are structured around the expected
outcome. This shifts the performance risk away from the government and often results in
savings by allowing the contractor to provide innovative solutions for the stated need. A PWS is
typical of a contract that engages the time and effort of a contractor whose primary purpose is to
perform an identifiable task rather than to furnish an end item.
The SOW/PWS should accomplish the following T&E goals that are common across most FAA
investment programs:
a) Describe the T&E events and activities to be accomplished by the contractor that
reflect the program T&E strategy described in the TEMP, ISPD and Integrated
Master Schedule (IMS)
b) Indicate the use of T&E processes that are critical for program success (e.g., test
program management, integrated testing, test capability accreditation, discrepancy
reporting etc.)
c) Ensure that the government has access to contractor data, test activities, and results
d) Government review and approval of contractor deliverables such as test plans,
procedures and reports
e) Contractor T&E support for government run testing (e.g., contractor personnel,
government training, meetings, DR/PTR review boards, test readiness reviews, etc.)
Refer to Appendix D, Figure D-4, for the T&E Input Review Cycle for SIR Proposal
Requirements and Statements of Work.
Note: Refer to the V&V Repository for the FAA Statement of Work (SOW) Preparation
Guide and SOW sample.
5.5.2.2 STATEMENT OF OBJECTIVES (SOO)

A SOO is used when the government intends to provide the maximum flexibility to each offeror
to propose innovative approaches to meet the government’s needs. The SOO is prepared by the
government in lieu of a SOW or PWS and provides the purpose, scope and top level objectives
of the investment program. The SOO is used by the offeror to develop a SOW or PWS as part of
their proposal. The SOO does not become part of the contract. The offeror’s SOW or PWS
should be structured for the proposed system solution and not restricted by the structure of the
government’s SOO. The government should develop a cross-reference matrix tracking the
government SOO requirements to the offeror’s proposed SOW or PWS.
Consider the following when developing the T&E content for the SOO:
a) The T&E approach documented in the government TEMP
VVSPT-A2-PDD-013 v4.0 5-18 05/16/2017

b) Integration of key T&E processes with program management and engineering

processes (e.g., risk management, requirements management, configuration
management, IMS development, status/problem reporting, etc.)
c) Cost or schedule savings associated with use of contractor T&E best practices and
processes that supports program execution
d) Use of event-driven program milestones, T&E reviews and activities (e.g., CDRs,
Test Readiness Reviews (TRRs), and entrance/exit criteria)
e) Use of commercial products/processes/standards to reduce cost or schedule
f) Government participation on contractor teams to assess program progress
g) Contractor participation in government T&E reviews and activities (e.g., test
readiness, technical reviews, data reviews, OT, etc.).
5.6 P ROPOSAL E VALUATION S UPPORT
The Test Directors (DT and OT), and designated test team members review the contractor
proposals. The Test Directors are the SMEs for reviewing the test portion of the proposals.
However, the Test Directors also review other portions of the proposals to assess potential
impacts to the test program.
For competitive solicitations, the DT and OT Test Directors evaluate the proposals for technical
content against technical evaluation criteria specified in the proposal evaluation plan. For
noncompetitive proposals (sole source procurements and Engineering Change Proposals
(ECPs)), the Test Directors assist in the evaluation of the proposed costs as well as the technical
content of the proposal.
Note: Refer to the V&V Repository for the Contractor Test Cost Proposal Review
guidance.
The DT and OT Test Directors and test team members assist in developing technical evaluation
criteria that will be used to evaluate vendor proposals to the SIR. For COTS and/or NDI
equipment procurements, the FAA may institute and conduct a “try before you buy” product
review in the technical evaluation segment. This evaluation approach is conducted within the
scope of either an OCD or an OCT. For either approach, prospective equipment vendors develop
their proposals based on evaluation criteria defined within the SIR
VVSPT-A2-PDD-013 v4.0 5-19 05/16/2017

Intentionally Blank
VVSPT-A2-PDD-013 v4.0 5-20 05/16/2017

6 T&E SUPPORT TO SOLUTION IMPLEMENTATION - DT

T&E support to SI begins with the Development Test (DT) phase of the T&E program. The
primary objective of DT is to demonstrate that all specified technical and performance
requirements for a system or service have been met. Another objective is to verify that the
system or service is fully integrated and stable, and that it has no adverse effect on the rest of the
NAS. It may include one or more demonstrations to ensure that the system meets user and
operational requirements. DT is performed by the development contractor and witnessed by the
DT Test Director and DT test team. DT activities may also be monitored by IOA personnel.
These test activities can be conducted at the contractor’s facilities, the FAA William J. Hughes
Technical Center (WJHTC), and/or FAA field sites. Note: DT may be conducted by the FAA (or
other governmental entity). The same disciplines this Handbook stipulates for contractor led DT
should apply in principle to FAA led DT.
This section of the Handbook describes the processes necessary for DT planning, conduct, and
reporting. Additionally, Appendix D identifies the major DT work products along with their
associated review and approval cycles.
6.1 DT O VERVI EW
This section of the Handbook describes the processes and best practices to be followed to help
ensure a successful DT, which may include:
a) Government or private laboratory testing, or
b) Site Acceptance Testing (SAT), whether conducted by Government personnel or a
contractor under Government scrutiny.
DT encompasses all testing necessary to verify contractual requirements and is performed at
FAA-approved contractor facilities, FAA laboratories, or FAA field sites. After DT, the FAA
will accept the portion of the system demonstrated through a formal Government Acceptance
(GA) process, which includes FCA and PCA (see Section 6.1.3.4).
As shown in Figure 3-1, there are two types of GA: the first type pertains to the acceptance of
systems at the WJHTC; the second type involves the acceptance of systems at field sites, as
illustrated in Figure 6-2.
6.1.1 DT ACTIVITIES
The DT process will differ from program to program, depending on size and complexity. It
should progress from verifying requirements at the unit level (lowest component) and move
incrementally to the fully integrated, fielded system level.
The following items define the standard test activities (which may be modified based on program
needs) for DT:
a) DT Software Testing: Verification of the specifications at the B-Level (subsystem).
DT Software Testing usually addresses new and modified software broken down to
the Computer Software Configuration Item (CSCI), Computer Software Component
(CSC), and functional design components.
VVSPT-A2-PDD-013 v4.0 6-1 05/16/2017

b) DT Hardware Testing: Verification of the specifications at the B-Level. DT

Hardware Testing usually addresses new and modified hardware broken down to the
Hardware Configuration Item (HWCI) and Hardware Design items.
c) Factory Acceptance Testing (FAT): Verification of primarily hardware, firmware,
and Commercial Off-the-Shelf (COTS)/Non-Developmental Item (NDI) subsystem
components to address B-Level and A-Level (system) specification items.
d) Functional Qualification Testing (FQT): Verification of partially integrated hardware
and software subsystem components, including COTS/NDI and modified COTS/NDI
subsystem components, to address B-Level and A-Level specification items.
e) DT Installation and Integration Testing: Verification of proper installation and
functioning of the complete system in the laboratory environment, including the
verification of system interfaces with other NAS equipment and Government
Furnished Equipment (GFE).
f) DT System Testing: Verification of integrated software and hardware components to
address A-Level specification items under conditions that emulate the projected
operational conditions.
g) Production Acceptance Testing (PAT): Verification of production line units of
developed hardware prior to installations at field sites to address B-Level and some
A-Level specification items.
h) Site Acceptance Testing (SAT): Verification of fully-integrated software and
hardware components to address A-Level specification items that could not be tested
sufficiently during DT System Testing. SAT also demonstrates that the requirements
that were verified under DT System Testing continue to remain in conformance as
installed at the operational field site by executing a subset of the DT System Tests.
i) DT Regression Testing: Verification of integrated software and hardware
components after changes to either have been made based on anomalies discovered
during previous DT activities. DT Regression Testing also ensures that the changes
made did not inadvertently result in a problem elsewhere in the system.
Figure 6-1 provides a diagram of the typical relationships of the above nine DT activities.
The typical flow shown in Fig 6-1 may be tailored for individual programs, so that some
activities may be performed in parallel, or in multiple phases of program implementation. For
some programs the traditional development cycle may be replaced by iterative methods known
as “Agile development” or “Agile Acquisition” and in these cases DT test activities may need to
be tailored.
VVSPT-A2-PDD-013 v4.0 6-2 05/16/2017

F IGURE 6-1. T YPICAL DT A CTI VITI ES
6.1.1.1 AGILE ACQUISITION IN AMS

The FAA may use Agile acquisition practices to accelerate the deployment of capabilities and to
ensure that capabilities exhibit greater user acceptance and operational value. The AMS includes
two guidance documents: FAA Agile Acquisition Principles and Practices and Agile Program
Management Practices for the FAA. For acquisitions adhering to this framework, it is important
for test practitioners to become familiar with Agile acquisition terminology and definitions.
Scrum is the iterative and incremental agile software development framework that the FAA has
adapted for use in AMS. Agile development generally refers to methods of breaking large
software releases into smaller units of working systems with limited capabilities, some of which
may be immediately deployable. These are developed in multiple “sprints” of durations of a few
weeks, at a sustainable pace (referred to as Cadence). Less important, or problematic,
capabilities are deferred (moved to a later sprint or release) in order to more quickly provide
prioritized operational capabilities to the field. Testers and developers, along with users, work
together to verify and validate each sprint, and plan the development and testing of the next
sprint.
For each sprint test event, requirements are reviewed and may be re-planned or deferred as they
are verified and validated. Agile development principles welcome changing requirements
throughout the process. This is a departure from traditional approaches employed for major
acquisitions which typically discourage frequent requirement changes
Scrum terminology includes:
a) Epics are large user stories often defined for a release that spans multiple sprints.
b) Features are program-level requirements that represent key capabilities, and are
maintained in a “Program Backlog.”
c) User stories are Sprint-level requirements
d) Daily Scrum is usually a 15-minute “stand-up” meeting to review what was done
toward the sprint goal, what needs to be done today, and what obstacles may need
help to be overcome.
VVSPT-A2-PDD-013 v4.0 6-3 05/16/2017
Test and evaluation best practices as described in this Handbook, should be maintained and
tailored to the extent feasible to ensure the integrity of the test program and thus support the
overall objective of operational readiness. Users of this Handbook should consult AMS for
updates as FAA refines its adoption of agile acquisition methods.
6.1.2 DT REQUIREMENTS
DT verification is based on contractually-required activities that the prime contractor must
perform to demonstrate conformance to the FAA-developed system specifications. The prime
contractor may develop and maintain separate specifications that are derived from the FAA
specifications and approved by the FAA as a CDRL document. The FAA- developed system
specifications or the contractor system specification (A-Level) is the “test to” document that
drives the conduct of DT. The prime contractor and associated subcontractors may develop
subsystem specifications (B-Level) and design documents that form the basis for B-Level
verification and system level procedure development.
The DT Test Director and test team review the contractor’s test CDRL documentation and
witness all tests during DT. The DT Test Director reviews the team’s comments and
recommends Government approval or disapproval to the Contracting Officer (CO) and
Contracting Officer’s Representative (COR). The DT test team must be proficient in their
particular program domain (e.g., Communications, Navigation, Surveillance, Weather, and Air
Traffic Automation) and eventually become experts on the system under test. System expertise
is necessary to ensure that the tests performed by the system contractor are valid and
comprehensive.
The DT functional flow diagram depicted in Figure 6-2, FAA DT Process Flow, identifies the
tasks and functions that support DT from the Screening Information Request (SIR) through
delivery of the DT Final Test Report. The tasks and functions that support SAT, from generation
of the SAT Plan through the start of Field Familiarization (FF), are identified in Figure 6-3, SAT
Process Flow.
VVSPT-A2-PDD-013 v4.0 6-4 05/16/2017

F IGURE 6-2. FAA DT P ROCESS F LOW
VVSPT-A2-PDD-013 v4.0 6-5 05/16/2017

F IGURE 6-3. SAT P ROCESS F LOW
VVSPT-A2-PDD-013 v4.0 6-6 05/16/2017

6.1.3 DT SUPPORT TO CONTRACT ADMINISTRATION

The DT Test Director and DT test team coordinate with the CO and the COR to provide support
in the following areas:
a) Negotiating the test cost and schedule baseline prior to contract award.
b) Tracking the contractor test cost and schedule performance measured against the
baseline. This includes reviewing documents delivered by the contractor to the FAA
(usually monthly) that detail the cost and schedule status of the program.
c) Participating with the PMO in occasional Integrated Baseline Reviews (IBRs). The
IBR is a formal meeting between the FAA and the contractor to ensure that the
program is staying within the program baseline.
d) Participating in additional negotiations when changes are made to the contract cost
and schedule baseline to determine a new test cost and schedule baseline.
e) Reporting test program status and changes in the test program.
f) Developing the contract, including modifications to criteria and contract line items
that pertain to T&E.
g) Addressing test issues that could affect the cost and schedule baseline.
h) Reporting the results of tests and the disposition of test document deliverables.
i) Evaluating acceptability of test-related contract conformance/deliverable items.
j) Deciding to award incentives based on test-related milestones.
k) Deciding GA after DT and SAT.
l) Reporting on minutes or issues from the Test Working Group (TWG) as required.
m) Conducting the Functional Configuration Audit (FCA) and Physical Configuration
Audit (PCA).
n) Assisting the FAA Quality Reliability Officer (QRO) in GA activities.
6.1.3.1 DT SCREENING INFORMATION REQUEST SUPPORT
To ensure that the contractor’s test program meets the program goals, the DT Test Director
works with the OT Test Director to develop T&E requirements for the SIR. The SIR includes a
description of work requirements (e.g., Statement of Work (SOW), Performance Work Statement
(PWS) or Statement of Objectives (SOO)) for the contractor’s proposal. See Section 5.5 for a
detailed description of DT SIR support requirements.
6.1.3.2 CONTRACTOR DOCUMENTATION SUPPORT
The FAA specifies the necessary documentation that the contractor must deliver in accordance
with the FAA AMS. The DT Test Director ensures that the test Data Item Description (DID)
requirements and the test CDRL documents are consistent with the test sections of the SOW or
PWS.
The DT Test Director recommends the schedule for delivery of test documentation by the
contractor. This schedule includes the amount of time that the FAA has to review and comment
VVSPT-A2-PDD-013 v4.0 6-7 05/16/2017

on the documents, typically 30 days. The schedule includes a time period for the contractor to
incorporate comments and redeliver the document to the FAA, typically no less than 15 days.
Prior to submitting formal draft or final CDRL documents to the FAA, the test team should
check if the contractor has conducted internal peer reviews.. These reviews help ensure that the
documents fully reflect the contractor's technical and programmatic approach for satisfying the
Government requirements. The contractor must conduct draft reviews of CDRL documents with
the FAA prior to delivering the final document. This review is conducted to ensure that all FAA
comments are clear and to verify the context in which the contractor intends to incorporate
comments.
The minimum test documentation for any procurement consists of the following:
a) CMTP
b) DT VRTM (may be contained in the CMTP, DT Test Plan(s), or in a separate CDRL
document)
c) DT Test Plan(s), Test Procedures, and Test Report
d) Integration and Test Development Plan (ITDP) and Report (as required)
e) Other contractor documentation that may be required and that impacts T&E,
including:
1) CM Plan
2) Software Development Plan
3) COTS/NDI Documentation
4) Reliability, Maintainability, and Availability (RMA) Test Report(s)
6.1.3.3 CONTRACTOR TEST DOCUMENT REVIEW STANDARDS
The DT Test Director and test team review and comment on all contractor-delivered test
documents. The Test Director provides comments and recommendations (approval/disapproval)
on the test documents to the CO and COR. The following guidelines should be used in
document review:
a) Ensure that the test team is familiar with the system design, NAS integration
environments, and general NAS architecture prior to reviewing the test plans and
procedures.
b) Ensure that the contractor has developed the DT VRTM based on the FAA System
Specification in accordance with the terms of the SOW/PWS.
c) Ensure that the approved DT VRTM has been delivered prior to the FAA reviewing
the test procedures.
d) Review the documents for technical content and compliance with the applicable DID.
e) Ensure that test documents are consistent with all relevant requirements and contain
clearly defined test objectives and success criteria.
f) Ensure that all interested parties receive CDRL documents with sufficient time for a
comprehensive review.
VVSPT-A2-PDD-013 v4.0 6-8 05/16/2017

g) Collect comments from test team document reviews. Ensure that FAA comments are
clear, detailed, specific, and based on compliance with FAA requirements.
Consolidate and submit comments by the due date.
h) Reject the deliverable if it does not meet contract requirements and standards.
Technical directions, as approved and conveyed through the Contracting Office, are
provided to the contractor to rectify the shortcomings of the deliverable.
i) Use a database, when practical, to manage comments for delivery to the contractor.
Note: Refer to the V&V Repository for the DT Comment Form template.
6.1.3.4 FCA, PCA AND GA ACTIVITIES SUPPORT

The DT test team supports the PCA and the FCA. The PCA is the formal examination of the
"as-built" configuration against its technical documentation to establish or verify the product
baseline. The FCA is the formal examination of functional characteristics, prior to GA, to verify
that the item meets the requirements specified in its functional configuration documentation. DT
results acquired from formal test documentation are used to ensure that a system or part of a
system successfully passes the PCA and FCA. The DT Test Director provides the FAA test
expertise for the PCA and FCA.
The DT test team may also support the FAA QRO in the GA process by reviewing and assessing
the accuracy of requirements verification status, system issues, and the disposition of problems
reported by the contractor.
6.1.4 DT ENGINEERING SUPPORT
The DT test team participates in system engineering meetings and activities held by the PMO to
provide test expertise. This participation includes meetings and events between the FAA and the
system contractor (e.g., System Engineering Working Group (SEWG) meetings) and internal
FAA-only activities and meetings.
The DT test team participates in hardware and software design meetings and reviews with the
contractor. This support includes reviewing contractor-developed engineering change request
documentation and hardware/software documentation. Through this activity, the DT test team
provides test engineering services for the evaluation of test results from software and hardware
redesigns.
6.2 DT P LANNING
To ensure successful test planning and execution, the DT Test Director communicates regularly
with the Contractor Test Manager. In the early part of the contract, the DT Test Director and
Contractor Test Manager work closely together to define a comprehensive test program.
Early planning tasks involve defining the strategies for:
a) Developing and maintaining the DT VRTM
b) Overall test approach and breakdown
c) Tools, simulations, and test environments
d) Developing and managing test laboratories
VVSPT-A2-PDD-013 v4.0 6-9 05/16/2017

e) Providing resources required for testing

f) System integration
g) Tailoring of standard test processes
h) Contractor test management
i) Informal and formal test conduct
To ensure that test issues are formally discussed and resolved with the contractor, a TWG is
established. This working group is co-chaired by the DT Test Director and the Contractor Test
Manager. The membership also includes the DT test team members, PMO representatives,
system users, the OT Test Director, and the OT test team members. The meeting schedule for
the TWG is established during test planning. The first meeting is held prior to the first formal
design review. The contractor, as required, records and maintains the minutes and actions items
from the TWG meetings.
6.2.1 DT TEST APPROACH AND STRATEGIES
The approach and strategies for a successful DT must apply the test activities previously
identified in Section 6.1.1 to address program needs. Not all test activities are required for every
program. The program complexity, system impacts, and scale of development will scope the
level of testing in the DT approach and strategies. A progressive and measured approach is the
best way to address T&E standards. During DT, requirements must be verified starting at unit
levels, and incrementally building through subsystem, to the integrated system(s) level.
During DT, the test environment can differ between the contractor’s facilities, a field site, or a
laboratory at the FAA WJHTC. During the test planning period, the test environment for each
test is determined. The test environment must mimic, as close as possible, the actual operational
environment of the system at the field site. Also, formal configuration control of the test
environment must be maintained throughout DT.
6.2.1.1 COTS/NDI TESTING
DT must account for the use of COTS products and NDIs in the test program. The test method
for these items is determined by the DT Test Director. Items that have undergone modification,
customization, or protocol changes require specific testing and/or demonstration of the modified
elements. The integration of these items with the other NAS subsystems must be tested or
demonstrated, or both. If the items or their integration is specialized or used for the first time in
this application, then special testing will be necessary. When planning COTS/NDI testing, it is
important to identify, consider, and report all associated risk factors (see the FAA System
Engineering Manual (SEM) for guidance on the implementation of COTS products and NDIs).
COTS/NDI hardware, software, and firmware items that are not modified or customized can be
tested through screening. Data from COTS/NDI screening is used whenever possible as the sole
means of testing requirements or in conjunction with DT activities. Screening of these items can
consist of the following FAA reviews to support verification:
a) Assessment of vendor specification, design, and production processes
b) Inspection of visual attributes
VVSPT-A2-PDD-013 v4.0 6-10 05/16/2017

c) Participation in demonstrations of performance, environmental, and suitability

characteristics
d) Evaluation of maintenance support provisions and warranties
e) Functional integration testing (as required) to assess COTS/NDI functionality as
integrated with other COTS, NDI or NAS subsystems
6.2.1.2 DT SOFTWARE TESTING
DT Software Testing is primarily conducted to verify the requirements that are at a lower level
(B-Level) than system level requirements. DT software requirements are documented in
contractor software requirement specifications or related design documents. This testing is
conducted by the developing contractor and witnessed or monitored by the FAA. Testing may
occur at a vendor facility, contractor facility, or the FAA WJHTC in accordance with the
approved CMTP.
DT Software Testing should include or address the following:
a) The contractor must report the status of testing activities and requirements
verification status at the DT TWG meetings.
b) As part of the software development, the contractor must maintain CSCI
documentation, sufficiently detailed to support independent evaluation of the
contractor’s CSCI testing plans, status, and verification results. CSCI documentation
must minimally include, in contractor format, test plans/procedures, test logs, as-run
procedures, and data recording and analysis.
c) The contractor must make CSCI documentation available to the Government for
review.
d) The verification of a subset of system level requirements may occur during DT
Software Testing. These system level requirements will be verified in either of the
following ways:
1) The system level requirement is directly verified at the non-system level by the
execution of DT Software Test steps.
2) The system level requirement is verified through inspection and analysis of the
results of the DT Software Testing for specific related non-system level
requirements.
The success criteria for the associated non-system level requirements must be included in the
delivery of system level success criteria. The FAA review of these non-system level
requirements will follow the same process as for the system level requirements.
Verification of system level requirements through CSCI testing must be documented in a DT
VRTM.
6.2.1.3 DT HARDWARE TESTING
DT Hardware Testing is conducted at the unit and/or subsystem level for programs that require
contractor-developed hardware. This testing is conducted by the developing contractor and
witnessed or monitored by the FAA. The formal DT Hardware Testing requires hardware test
plans, procedures, and reports at the unit and/or the subsystem level. In addition, the
VVSPT-A2-PDD-013 v4.0 6-11 05/16/2017

manufacturing process used to develop the hardware needs to be qualified via testing. When a
significant quantity of hardware items is to be produced, only a representative subset of the
produced hardware is subject to test.
DT Hardware Testing focuses on the following areas:
a) Verifying that the hardware conforms to applicable specifications, is free from
manufacturing defects, and is substantially identical to qualified hardware (only for
PAT)
b) Verifying hardware-related human factors and safety requirements
c) Evaluating the manufacturing process for newly developed hardware
d) Testing of COTS/NDI products
e) Testing the hardware in the racks or assemblies to be fielded
f) Testing the interfaces between Line Replaceable Units (LRUs). This interface testing
can be conducted using special test software, operational software, or both.
g) Electrical power testing
h) Thermal testing
i) Acoustic testing
j) Electromagnetic Interference (EMI)/ Electromagnetic Compatibility (EMC) testing
k) Seismic, shock, and vibration testing
6.2.1.4 FACTORY ACCEPTANCE TESTING (FAT)
FAT is performed at the subsystem or partially integrated system level to verify some system
level requirements, non-system level software requirements, and hardware subsystem
requirements. FAT may also provide for the final verification of A-level requirements that do
not require the final baseline environment. FAT is conducted in accordance with the CMTP and
the SOW/PWS. FAT may be conducted at a contractor’s facility or at the FAA WJHTC.
The following items should be considered when planning for FAT:
a) FAT is a prerequisite to DT System Testing
b) The contractor obtains Government concurrence of the success criteria
c) FAT plans, procedures, and reports are CDRL deliverables that are approved by the
FAA
d) The FAA test team will witness FAT activities
e) The test executions are conducted on a configuration-managed baseline
6.2.1.5 FUNCTIONAL QUALIFICATION TESTING (FQT)
FQT is conducted on programs that have a vendor or subcontractor under contract to the prime
contractor for delivery of a product or subsystem. FQT demonstrates capabilities to integrate
with the NAS through the use of drivers and simulators, where applicable.
VVSPT-A2-PDD-013 v4.0 6-12 05/16/2017

Prior to DT System Testing, the prime contractor directs associate subcontractors to perform
FQTs in accordance with the CMTP, the FAA contract SOW/PWS, and the prime
contractor/subcontractor SOW/PWS. FQT may be conducted at the vendor/subcontractor
facility, contractor facility, or the FAA WJHTC. The FAA will monitor the prime contractor and
subcontractor on all FQT activities.
The following items should be considered when planning FQT:
a) The prime contractor plans and approves all FQT activities
b) All FQT Test Plans, Test Procedures, and Test Reports are to be provided to the
Government for review (including formal deliverables from subcontractors)
c) FQT verifies product specification requirements associated with the subsystem
delivered by the subcontractor
d) The prime contractor utilizes the requirements from the product specification to
address and provide traceability to the FAA system specifications
e) All pertinent test data will be collected and logged in accordance with the approved
Quality Assurance (QA) processes
f) FQT may not be subjected to all of the formal FAA processes as other DT activities
since it is under the auspices of the prime contractor
6.2.1.6 DT INSTALLATION AND INTEGRATION TESTING
To ensure an efficient process, DT Installation and Integration (I&I) Testing is scheduled early in
the development of a system and made a prerequisite to DT System Testing. Early integration
testing is effective in finding low-level issues that might have been overlooked if this testing was
not conducted. For systems where the integration process is complex, specific integration
milestones are required.
The installation and integration of the hardware into laboratory environments and operational
sites to be used during system tests must be verified through DT I&I Testing prior to conducting
DT System Testing. DT I&I Testing ensures that the system is properly installed and
functioning, correctly interfaced with GFE, and ready to begin DT System Testing. It includes
hardware and NAS integration test activities. The FAA will witness all DT I&I Testing
conducted by the contractor.
6.2.1.7 DT SYSTEM TESTING
DT System Testing verifies the system’s ability to satisfy the requirements of the applicable
system specifications assigned to the contractor. The FAA will witness all DT System Testing
conducted by the contractor. System test plans, procedures, and reports are deliverables that are
approved by the FAA. System tests can include:
a) Integration and Interface Verification: The verification of subsystem integration and
NAS interfaces.
b) System Capacity and Response Time Performance Verification: The verification of
system processing time and stress thresholds.
c) Failure Mode and Failure Recovery Verification: The verification of failure mode
conditions and system recovery capabilities.
VVSPT-A2-PDD-013 v4.0 6-13 05/16/2017

d) Stability Verification: The verification of system performance under extended

continuous system use.
e) Reconfiguration Verification: The verification of system reconfiguration capabilities.
f) Functional Verification: The verification of functional performance with a fully
integrated system and realistic NAS data.
g) Site Simulation Verification: The verification of system performance under
simulated or real operational site conditions and configurations.
h) System Support Functional Verification: The verification of system support functions
with a fully integrated system required by the FAA WJHTC laboratories and the sites.
i) Monitoring and Control Verification: The verification of required system monitoring
and control functions with a fully integrated system.
j) Security Test: The verification of the security requirements. The Security Test
should comply with the program’s Security Authorization Package (SAP), and
support the Plan of Action and Milestones (POAMs), as required.
k) System-Level Acceptance Test: The verification of the fully assembled and fully
integrated system that addresses a sampling of all aspects of system functional
performance requirements. This test may provide the foundation for regression
testing and SAT.
l) Reliability Verification: The verification by analysis, consisting of reliability
predictions supplemented by data collected during system level verification.
m) Maintainability Demonstration: The demonstration of an approved subset of the
maintainability requirements.
n) Electromagnetic Interference and Electromagnetic Compatibility Verification: The
verification of EMI and EMC requirements.
o) Power Quality Testing: The verification of power quality testing requirements.
p) Simulation: Tests using simulated interfaces or subsystems are valid to verify
requirements.
System testing often requires the use of both live and simulated interfaces and subsystems. The
use of live and simulated data should be used as complementary test methods. Live data
provides for the most relevant observation of how the system performs. However, live data may
lack repeatability of test results (which may be overcome by the recording and playback of live
data). Simulations provide for repeatability and the generation of conditions that are cost
prohibitive or impractical with live data (e.g., capacity scenarios, interference, etc.). Simulations
attempt to mimic live stimuli to a system but are not an exact replication. Minor shortfalls in
simulation fidelity have been observed in practice to have a significant impact on test results
(e.g., the system works in simulation but is operationally deficient in practice). Therefore,
optimal system testing is often accomplished using a combination of live and simulated
interfaces and subsystems. Simulations must be accredited (see Section 9.1) prior to the formal
conduct of tests using the simulation.
VVSPT-A2-PDD-013 v4.0 6-14 05/16/2017

Prior to entering DT System Testing, a System Test Entrance Checklist is completed by the DT
Test Director as an internal FAA assessment of DT issues and test readiness. This checklist is
submitted to the TSB for their review and comment and must be endorsed by the T&E First Line
Supervisor prior to the Test Readiness Review (TRR) (see Section 6.3.2).
Note: Refer to the V&V Repository for the DT System Test Entrance Checklist template.
6.2.1.8 PRODUCTION ACCEPTANCE TESTING (PAT)

The contractor performs PAT on newly manufactured hardware items to verify that the hardware
items and installed software or firmware conform to applicable specifications. This testing also
verifies that the production unit is free from manufacturing defects and is substantially identical
to qualified hardware. Government personnel will witness the contractor’s PAT. The contractor
will develop PAT Test Plans, PAT Test Procedures, and PAT Test Reports.
6.2.1.9 SITE ACCEPTANCE TESTING (SAT)
SAT verifies the site installation, system integration, and operational configuration of the system
and that the system complies with contractual requirements. All requirements designated by the
DT VRTM for verification during SAT must be verified. SAT is conducted at each Government
facility where the system is installed. Ensuring the quality of installation procedures is also part
of SAT. SAT is conducted by the contractor and is witnessed by the FAA. The DT test team
reviews the SAT Test Plan and Test Procedures. SAT differs from other aspects of DT in that it
focuses on performance with respect to a specific location. SAT verification addresses
requirements that need operational site conditions and configurations not available during formal
DT System Testing activities.
The following items must be addressed when planning for SAT:
a) A specific individual is designated as the Site Test Director to be the FAA Lead for
SAT activities and formally accept test results. Other personnel may be designated as
Alternate Site Test Directors to represent the site in the absence of the Site Test
Director.
b) The DT test team coordinates with each facility prior to SAT. Meetings are held with
the Site Test Director well in advance of any test activities. This allows for logistical
coordination and helps to familiarize site personnel with the product, test process and
roles.
c) The DT test team coordinates all site test support activities with the Site Test Director
or designee.
d) The DT test team coordinates with the sites regarding the schedule and scope of the
SAT for each location.
e) SAT documentation must be delivered to the Site Test Director for review and
comment. Enough time must be provided to the Site Test Director to perform the
review and comment.
f) SAT uses a subset of DT requirements to determine fully acceptable system
implementation at each site.
VVSPT-A2-PDD-013 v4.0 6-15 05/16/2017

g) One or more “Key Sites” is/are designated as the first location(s) in the system site
deliveries. More than one Key Site may be necessary to verify system performance,
depending on the scope of the tests. Some considerations in determining a Key Site
include:
1) Amount of support available from site personnel to resolve issues found during
testing.
2) Experience of the site personnel in managing the planned testing.
3) Ability to coordinate and document facility procedures, and plan for use by
subsequent sites.
4) Location has a low risk for Air Traffic Control (ATC) safety impacts during
system integration at the site.
5) Representative field conditions and environment.
6) Available operational interfaces.
The AMS T&E Guidelines recommends that Key Site SAT be conducted following the
completion of formal OT. However, if a test program is designed to verify a significant number
of system specification requirements at the Key Site, the Key Site SAT should be planned for
completion prior to the start of formal OT.
Systems often must be adapted depending on where the system is deployed. The contractor, with
FAA approval, identifies which parameters are site-specific and ensures those parameters are
included in the software and hardware delivered to the site. Though CM is critical throughout
the test program, precautions are taken to ensure that site-specific data is correct. The contractor
briefs the site personnel on the configuration of the system prior to SAT at the TRR and Pre-Test
Briefing. These configurations are verified by the QRO and DT test team prior to formal test
conduct.
6.2.1.10 DT TEST CAPABILITY ACCREDITATION
DT test capabilities must be accredited in accordance with Section 9.1.
6.2.1.11 COORDINATION BETWEEN DT AND OT TEST DIRECTORS
The DT and the OT Test Directors coordinate on test strategies throughout the lifecycle of the
system. This coordination begins with the Integrated Test Team (ITT) when the TEMP is
developed and continues through the TWGs.
To reduce the risk of encountering major system problems during OT, the DT Test Director
should incorporate operational conditions and conduct evaluations from an operational
perspective. To accomplish this, both the DT and OT Test Directors must ensure that
operationally oriented test conditions and evaluation criteria are planned for by addressing them
appropriately in the SIR, SOW/PWS, CMTP, and system specification.
6.2.1.12 DT TRAINING AND FAMILIARIZATION
The DT Test Director is expected to be an expert in the domain for the program in which that
person leads. However, T&E specific training and qualification are required for each DT Test
Director. This training ensures that the DT Test Director understands the policies and processes
related to T&E.
VVSPT-A2-PDD-013 v4.0 6-16 05/16/2017

The DT test team receives two types of training:

a) Training on system operations and functionality, provided by the contractor as
required by the SOW/PWS. Also, the DT test team can become familiarized through
participation in the oversight of the system development by the contractor.
b) T&E Handbook and QMS process (see Section 1.8) training to ensure that T&E
practitioners fully understand what is expected to successfully perform the DT Test
Program.
6.2.2 CONTRACTOR MASTER TEST PLAN (CMTP)
The CMTP describes, at a high level, the contractor’s complete test approach for the system
investment program. Information in the CMTP includes:
a) Test Management
b) DT VRTM
c) Identification and definition of major test areas
d) Test tools and accreditation plans for test capabilities
e) Test environment
f) Integration testing
g) Schedule (based on the overall program schedule)
h) Configuration Management
i) Tracking and verification of safety requirements (includes system safety and
personnel safety during tests)
j) Security (Coordinate these tests with the development of the SCAP to ensure that
requirements are addressed)
k) Process description
l) Requirements tracking
m) COTS/NDI test strategy
n) Testing of interfaces to the NAS
o) Program Trouble Reporting and corrective action process (see Section 6.3.7)
p) Initial system certification (not applicable on all programs)
Note: Refer to the V&V Repository for the CMTP template and a CMTP sample.
The DT test team, in accordance with the contract, conducts an early informal review of the
CMTP with the contractor in order to discuss test strategies and issues in a collaborative
environment. This collaboration will ensure that the contractor’s and FAA’s test planning is in
agreement and that the CMTP is contractually adequate and compliant. Subsequently, the DT
test team and the TSB review the revised CMTP prior to the DT Test Director’s recommendation
for approval by the CO. There may be several iterations of drafts as the CMTP is worked toward
VVSPT-A2-PDD-013 v4.0 6-17 05/16/2017

acceptance by the government. The TSB can consult with the DT Test Team to determine
whether TSB review will be required for all iterations.
Refer to Appendix D, Figure D-5, for the complete CMTP review and approval cycle.
For planning purposes, the CMTP can be required as early as Contract Award to as late as 30
days after the Critical Design Review (CDR). Generally, if the product is COTS or an NDI, the
CMTP would be required sooner (i.e., towards Contract Award) rather than later. For
development products, general guidance would be based on the maturity level of the product or
the technology. For example, a development program using a proven technology may require
the CMTP to be delivered at the System Requirements Review (SRR) or no later than the
Preliminary Design Review (PDR). Consequently, a development project with an experimental
technology or other high risk may require a draft CMTP to be delivered at PDR and a final
CMTP at CDR. To summarize, using sound engineering judgment based on the product’s
maturity level is the best guidance for required delivery of the CMTP.
6.2.3 DT VERIFICATION REQUIREMENTS TRACEABILITY MATRIX (VRTM)
The DT VRTM summarizes how each requirement is verified. It includes:
a) specific requirement identifier
b) short description of the requirement
c) type of verification method that the contractor performs to show that the requirement
is satisfied (Demonstration, Test, Analysis, or Inspection, as defined in Section
5.2.3.5)
d) specific DT activity
e) unique test identifier during which the requirement is verified.
The initial DT VRTM is developed based on system level requirements contained in the FAA
System Specification in accordance with the terms of the SOW/PWS. Lower level DT VRTMs
are established and documented during the planning for hardware and software testing.
The baseline DT VRTM is contained in an appendix to the approved CMTP. Subsequent
changes to the DT VRTM do not require a resubmission of the entire CMTP. Instead, the
contractor can deliver formal updates via a revised CMTP appendix as a contract correspondence
that requires FAA approval. Additionally, during DT Test Plan development, the contractor will
refine the DT VRTM with test case information and will include the updated DT VRTM as part
of or an attachment to the DT Test Plan(s).
In addition to test case mapping, the DT VRTM includes success criteria, which describe the
condition needed to determine if the requirement is met. Success criteria are used to measure a
system task’s accomplishment and/or system characteristics, as well as measures of operational
capabilities in terms of the system’s ability to perform its mission in an operational environment.
Success criteria are developed for each contract specification requirement (per the CMTP DID)
and are generated by the contractor through a detailed analysis of requirements and provided to
the FAA for concurrence prior to the start of each final test execution. Once FAA concurrence is
provided, the contract specification requirements are assessed against the established success
criteria during DT. Any changes to the established, agreed-upon success criteria normally
require agreement in writing by the FAA and contractor, with updates incorporated into the DT
VRTM.
VVSPT-A2-PDD-013 v4.0 6-18 05/16/2017

6.2.4 DT INTEGRATION TEST PLANNING

During a complex full system development, hardware and software items are developed
separately. The integration of the hardware and software begins when both of these items reach
a sufficient level of maturity.
For large integration efforts, the prime contractor may be required to prepare and maintain an
Integration and Test Development Plan (ITDP). This plan describes the product integration
approach and criteria, the activities for developing test procedures, and the methods and
schedules for implementing and controlling the integration and test development programs.
Since this testing may not be the final or full verification of the system requirements, specific
criteria and measures must be established and documented to plan for successful DT Integration
activities. If a separate ITDP is not required, the integration testing approach can be documented
in the CMTP.
The following items must be considered when planning for DT Integration Testing:
a) Plans must measure progress and contain sufficient criteria to determine if the system
or service is ready to move on to the next DT activity.
b) Prior to an integration activity, entrance criteria must be identified and documented.
Entrance criteria provide an objective measurement of what needs to be in place, with
all dependencies clearly identified.
c) Exit criteria must be identified and documented. Exit criteria provide objective
measurements of what each activity is to accomplish, ensuring an orderly progression
of activities and demonstrating that system maturity is increasing to a successful
acceptance of the system.
d) Milestones related to the type of configuration, system loads, complexity of
configurations, duration of test run times, and variations of interfaces must be defined
to indicate system maturity and the progress of the system integration.
e) The test configurations required to perform integration testing must be defined in the
ITDP or CMTP.
6.2.5 DT TEST PLANS
DT Test Plans provide more detail than the CMTP regarding DT for the test areas identified in
the CMTP. A DT Test Plan is developed by the contractor for each of the DT activities (e.g.,
SAT Test Plan, FAT Test Plan, Integration Test Plan, etc.). DT Test Plans cover all aspects of
DT in proper detail so that the DT Test Procedures can be written from test descriptions
contained in the DT Test Plan.
A DT Test Plan contains the following key elements:
a) DT program management
b) Schedule (based on the overall program schedule)
c) Test descriptions
VVSPT-A2-PDD-013 v4.0 6-19 05/16/2017

d) Plans and itemized lists for required GFE, Government Furnished Information (GFI),
and Government Furnished Property (GFP)
e) Program Trouble Reporting and corrective action process (see Section 6.3.7)
f) Configuration Management
g) DT VRTM
h) Accreditation plans for test capabilities
Note: Refer to the V&V Repository for the DT Test Plan template and a DT Test Plan
sample.
Similar to the review and approval process described in Section 6.2.2 for the CMTP, the DT test
team conducts an early informal review of the DT Test Plan with the contractor in order to
discuss test strategies and issues in a collaborative environment. This collaboration will ensure
that the contractor’s and FAA’s test planning is in agreement. Subsequently, the DT test team
and the TSB review the revised DT Test Plan prior to the DT Test Director’s recommendation
for approval by the CO.
Refer to Appendix D, Figure D-6, for the complete DT Test Plan review and approval cycle.
6.2.6 DT TEST PROCEDURES
The DT Test Procedures are developed by the contractor based on their respective DT Test
Plan(s). The Test Procedures include all of the details for conducting a particular test to verify
requirements as specified in the respective Test Plan(s). These details include:
a) Tables of step-by-step instructions to run the test
b) Observations to be made during the test
c) Expected results, including success criteria
d) Objectives
e) Test limitations
f) GFE, GFI, and GFP required for the specific test
g) Notations of the requirements being tested by a particular test and step
h) Data collection, reduction, and analysis required
i) Test tools and equipment required
j) Configuration of the system under test and the test environment
The DT test team reviews the DT Test Procedures and provides comments to the contractor for
disposition. Once the dispositions of comments are agreed to by both the FAA and the
contractor, the DT Test Director recommends approval or disapproval of the document to the CO
and COR. The test procedures are then executed via dry run testing (see Section 6.3.1) prior to
formal test conduct.
Refer to Appendix D, Figure D-7, for the complete DT Test Procedures review and approval
cycle.
VVSPT-A2-PDD-013 v4.0 6-20 05/16/2017

Note: Refer to the V&V Repository for the DT Test Procedures template and a DT Test
Procedures sample.
6.2.7 DT ENTRANCE CRITERIA
Prior to entering any formal DT activity, the DT Test Director ensures that the following
minimum DT entrance criteria are met:
a) All entrance criteria as defined in the approved CMTP and DT Test Plan(s) are
satisfied
b) DT Test Procedures have been submitted, reviewed, and approved
c) The test configuration is known and documented
d) Dry run testing has been completed by the contractor
e) The configuration under test does not have known deficiencies that affect the
functions to be verified by the test
f) If required, test capability accreditation has been conducted and approved
6.2.8 DT EXIT CRITERIA
Prior to exiting any formal DT activity, the DT Test Director ensures that the following
minimum DT exit criteria are met:
a) All exit criteria as defined in the approved CMTP and DT Test Plan(s) are satisfied
b) Completion of all DT in accordance with approved Test Plans and Test Procedures
c) All Post-Test Reviews are complete
d) Test results are documented and accepted by the FAA
e) All DT Program Trouble Reports (PTRs) are fully documented, assessed, and status
reported
f) All contractor performance requirements are addressed in accordance with the
contract (may include PTR priority stipulations, or requirement pass-rate percentages)
6.3 DT T EST C ONDUCT
For all DT, the DT Test Director ensures that the contractor performs testing in accordance with
the SOW/PWS and approved Test Plans and Procedures. The DT Test Director also ensures that
the DT test team is prepared to witness the tests. The contractor conducts debug and dry run
testing followed by a TRR prior to each formal DT activity. The DT Test Director ensures that
the minimum DT entrance criteria as defined in Section 6.2.7 have been met.
The DT Test Director and/or the Test Lead must report and record the status of all test activities.
Following each formal DT activity, the DT Test Director ensures that the minimum DT exit
criteria as defined in Section 6.2.8 have been satisfied.
6.3.1 DT DEBUG AND DRY RUN TESTING
Prior to formal DT, debug and dry run testing of the procedures must be performed. Debug
testing is where test procedures are executed against the system under test to ensure that the test
VVSPT-A2-PDD-013 v4.0 6-21 05/16/2017

steps are correct, complete, and produce repeatable expected results. During this testing, the
procedures are refined and updated to provide a logical flow to the sequence of test steps.
Dry runs are a complete end-to-end execution of the DT Test Procedures using formal test
configurations and accredited scenarios, simulations, and/or test tools to assess the following
criteria:
a) The laboratory environment is prepared
b) The system has been properly installed and configured
c) The system has the correct versions of both software and adaptation available, and all
system parameters that need to be set have been identified
d) Procedures are mature and redline text changes are fully incorporated (i.e., test
procedures can be run start to finish, as documented, without major deviations or
anomalies)
Dry runs are executed by the contractor and witnessed by the DT test team for each test prior to
entering formal DT, with dry run test results presented at the DT TRR. For each witnessed dry
run, a Test Status Report is prepared by the DT Test Lead (see Section 6.3.6). If a dry run that is
executed as if it was a formal test run and in accordance with test plans and procedures
completes without any significant problems, the contractor may request that the FAA waive a
second execution. If the FAA concurs, the dry run execution is accepted as a formal test run. In
these cases, a Pre-Test Briefing and a Post-Test Review are conducted to capture all test
execution details.
6.3.2 DT TEST READINESS REVIEW
The TRR is presented by the contractor to the FAA as required by the SOW/PWS. The
objectives of the TRR are to officially establish that the contractor is prepared and ready to start
formal testing of the system, and not to prematurely enter into it. The TRR covers the following
items:
a) Overview of testing to be performed
b) Status of contractor development and integration milestones and checkpoints
c) Status of all applicable test documentation
d) Identification of required Government and contractor resources and personnel
e) Configuration control and accreditation of test tools and test items (both hardware and
software) and any other items necessary for the successful conduct of testing,
including Data Reduction and Analysis (DR&A) tools and equipment
f) Prior test results, including those from any dry runs of tests
g) Summary of all PTRs, including status of relevant hardware, firmware, and software
problems
h) Review of test baseline, configuration, and environment
i) System CM status
j) GFE, GFI, and GFP status (if applicable)
VVSPT-A2-PDD-013 v4.0 6-22 05/16/2017

k) Traceability between requirements and their associated tests using the DT VRTM
l) Test schedules
m) Test entrance and exit criteria
A draft TRR briefing package must be provided to the DT Test Director at a contractually
specified date prior to the planned TRR meeting date. Successful completion of the TRR
establishes the readiness for testing. The DT Test Director advises the COR on whether or not
DT may commence. A copy of the TRR briefing package must be included with the associated
DT Test Report.
6.3.3 DT PRE-TEST BRIEFINGS
Prior to each test, the contractor conducts a Pre-Test Briefing to ensure readiness to begin the
respective test. The Pre-Test Briefing covers the following items:
a) Test objectives and success criteria in accordance with the approved Test Plan and
Test Procedures
b) Proposed procedure changes (redlines)
c) Test configuration definition (e.g., hardware, adaptation, software version levels,
patches, configuration files), including any GFE
d) Test personnel assignments and Government and contractor resources
e) Test conduct walkthrough
f) Results of the CM audit
g) Test limitations
h) Review of known system anomalies that might impact testing
i) Planned deviations
j) DR&A methods
k) Results of any dependent testing that demonstrate readiness for test conduct
The DT Test Director and contractor test manager (or their designees) review and provide
signature approval of all planned deviations presented at the Pre-Test Briefing.
6.3.4 FORMAL DT TEST EXECUTION
The standard process items for formal test execution include:
a) The DT Test Director may delegate signature approval to the appropriate DT test
team member. The signature approval grants those test team members with authority
to initial changes to the formal Pre-Test Briefing and Post-Test Review packages and
redlines during formal test conduct.
b) The DT test team witnesses the formal runs of all tests. Copies of procedures are
provided to the FAA personnel witnessing formal test runs.
c) Proper test configuration is verified prior to starting formal test (including hardware,
software, firmware, adaptation parameters, test equipment, and test tools).
VVSPT-A2-PDD-013 v4.0 6-23 05/16/2017

d) A Test Status Report is prepared for each formal test run (see Section 6.3.6.).
e) The DT test team witnesses deviations and changes to the Test Procedures.
f) The DT test team ensures that anomalies are properly documented (at the time of
occurrence) in the test log and that the test log is signed at the completion of testing.
g) The contractor provides the “as-run procedures” (i.e., with mark-ups) and test log
informally after test completion.
h) The contractor performs a walkthrough review of the DR&A results of the test data
with the DT test team to verify requirements.
6.3.5 DT POST-TEST REVIEWS
The contractor conducts a Post-Test Review with the FAA for each test performed to confirm
test results and completion. The Post-Test Review consists of:
a) Overall test results, including a summary of all associated requirements
b) Status of test objectives and exit criteria as specified in the associated Test Plan and
Test Procedures
c) Test conduct details (e.g., start date and time, stop date and time, etc.)
d) Any test configuration changes since the Pre-Test Briefing
e) All problems encountered, including where and how they are documented
f) Descriptions of all deviations and anomalies encountered
g) Test procedure changes
h) Details on any failed steps and requirements
i) Review of DR&A results and walkthrough findings
j) Regression test recommendations
k) Documentation of the outstanding test issues with action plans for closure
6.3.6 DT TEST STATUS REPORTS
The DT Test Director (or designee) prepares a Test Status Report for each dry run and formal
test run. The Test Status Report is distributed to the DT test team and entered into the test status
database (which is managed by the test team). The test status database is used to document all
information pertaining to a given test execution for a particular DT activity, including: test
name, FAA and prime contractor participants, facility name where the test is conducted, software
version, lab configurations, summary and specific details of the test, issues/concerns, and the
next test activities.
Note: Refer to the V&V Repository for the Test Status Report template.
6.3.7 DT PROBLEM REPORTING PROCESSES

The contractor develops and maintains a database for submitting, tracking, reporting, and
maintaining records on PTRs. The database will support the FAA in maintaining awareness of
problems from initiation to final corrective action and archiving. The database will enable status
VVSPT-A2-PDD-013 v4.0 6-24 05/16/2017
reporting of all test problems. The contractor uses the database for tracking problems associated
with any system, equipment, software, or firmware that has been placed under formal
configuration control. The FAA reviews the overall design of the database to ensure operational
use and functionality required by the Government. The contractor will provide Government
personnel with access to the database and provide reports at the request of the Government.
PTRs entered into the database should be prioritized by the contractor (with review and
concurrence by the FAA) according to the following definitions:
Priority 1 is assigned to a problem that prevents the accomplishment of an operational,
mission-critical or mission-essential capability, or jeopardizes safety, security or other
requirements designated as Critical Performance Requirements (CPRs).
Priority 2 is assigned to a problem that adversely affects the accomplishment of an
operational, mission-critical or mission-essential capability and no work-around solution
is known.
Priority 3 is assigned to a problem that adversely affects the accomplishment of an
operational, mission-critical or mission-essential capability, but a work-around solution is
known.
Priority 4 is assigned to a problem that results in a user/operator inconvenience or
annoyance, but does not affect a required operational, mission-critical or mission-
essential capability.
Priority 5 is assigned to any other problem/defect not described above (e.g., system
documentation errors).
These definitions are also applicable to Discrepancy Reports (DRs). DRs may be generated prior
to OT for all issues discovered that impact operational requirements but are not being addressed
and need to be evaluated during OT (see Section 7.6.4).
The contractor problem-reporting system must be defined in the T&E activities section of the
Test Plan. The contractor submits the planned corrective action for each problem and identifies
the proposed regression testing or future modification(s) to the testing program required to
validate the successful corrective action. If a component fails during testing, the contractor must
perform failure analysis to identify the cause of the failure. Failed steps, with or without
associated problems, will be explained to the satisfaction of the Government. All anomalies will
be jointly analyzed by the contractor and the Government to determine a recovery plan.
The contractor is responsible for any corrective actions necessary to ensure full specification
compliance. The contractor completes repairs or procedural changes prior to submission for
regression testing. GFP-induced anomalies will be identified to determine Government
responsibilities for corrective actions. GFP anomalies do not relieve the contractor from
compliance to specification requirements.
6.3.8 DT REGRESSION TESTING
The contractor conducts regression tests when changes have been made to the hardware or
software for which the Government has previously accepted test results. The contractor
recommends and briefs the Government on the level of regression testing as part of the
corrective action. The DT Test Director will determine the extent of regression testing required.
Regression testing will not be started by the contractor until receiving Government concurrence
VVSPT-A2-PDD-013 v4.0 6-25 05/16/2017
to proceed with the regression test. In addition, regression testing and analysis will ensure that
the fix did not cause a problem elsewhere in the system.
The contractor will determine and fully document the cause of the noncompliance in the
problem-reporting database for any failed test and provide a written notification to the
Government. The contractor will conduct the regression testing using the Government-approved
test plans and test procedures. The Government reserves the right to witness all regression
testing.
6.4 DT T EST R EPORT
The DT Test Report(s) addresses test results, including test conduct, data collected, the DR&A
process, and conclusions to be drawn from the test data. The FAA utilizes the results contained
in the test report to verify that a contractor has furnished a product that conforms to all contract
requirements for acceptance. A DT Test Report includes the following:
a) A copy of the approved TRR briefing package
b) All approved deviations and waivers generated as a result of testing
c) As-run test procedures with test logs and witness signatures
d) A test summary providing the status of each requirement tested
e) A DT VRTM showing requirement verification status
f) All test results, including data analysis (graphs, pictures, etc.)
g) Evaluation of system deficiencies and performance based on the testing results
h) Pre-Test Briefing and Post-Test Review packages
i) Complete identification of the test configuration, including hardware, software,
firmware, adaptation parameters, test equipment, and test tools
The DT Test Report also includes separate sections for test activities and results from Security
and Safety requirements verification, as defined in the contract.
The objectives of the DT Test Report are to provide essential information in support of decision-
making, assessing technical and investment risks, and verifying the attainment of technical
performance specifications and objectives documented in the DT Test Plan.
Upon receipt from the contractor, the DT Test Director forwards the DT Test Report to the DT
test team and the TSB for review and comment. Subsequently, the DT test team and the TSB
review the revised DT Test Report prior to the DT Test Director’s recommendation for approval
by the CO.
Refer to Appendix D, Figure D-8, for the complete DT Test Report review and approval cycle.
Note: Refer to the V&V Repository for the DT Test Report template and a DT Test Report
sample.
VVSPT-A2-PDD-013 v4.0 6-26 05/16/2017

7 T&E SUPPORT TO SOLUTION IMPLEMENTATION - OT

Following DT and Government Acceptance (GA) of the system or service, T&E support to SI
continues with the OT phase of the T&E program. OT encompasses T&E of a system’s or
service’s operational requirements. The primary objective of OT is to examine performance and
determine operational readiness (effectiveness and suitability) of a new or modified system for
use in the NAS and that the NAS infrastructure is ready to accept the system. OT data is used to
assess the operational requirements (program requirements, COIs, MOEs, MOSs, and MOPs), to
assess operational effectiveness and operational suitability, in support of ISD and other relevant
decisions. OT is performed by the OT Test Director and OT test team with Air Traffic (AT),
Technical Operations, and second-level maintenance personnel. Other participants may include
representatives of other FAA organizations and/or external entities (e.g., airlines, cargo carriers,
military, private industry) as required. OT is conducted at a test facility, (e.g., the FAA William
J. Hughes Technical Center (WJHTC), test radar sites) but may require test activities to be
conducted at one or more operational field Key Sites in order to test configurations that are not
available in test facilities.
For designated programs, the data from OT is also used to support Independent Operational
Assessment (IOA). The results of OT are used to support the Independent Operational
Assessment Readiness Declaration (IOARD), which is prepared by the PMO in accordance with
AMS to mark the start of IOA.
This section of the Handbook describes the processes necessary for OT planning, conduct, and
reporting. Additionally, Appendix D identifies the major OT work products along with their
associated review and approval cycles.
7.1 OT O VERVIEW
OT consists of the following two major test activities:
a) NAS integration testing
b) Operational effectiveness and suitability testing
7.1.1 NAS INTEGRATION TESTING
NAS integration testing is conducted to baseline the new/modified system’s performance,
establish the NAS end-to-end performance baseline, and verify that current NAS performance
has not been degraded. Additionally, these tests verify external interface connectivity from the
system under test to other FAA and/or vendor-provided systems.
NAS integration testing may include the following:
a) Functional verification testing
b) Regression of DT in new environment test
c) Interoperability and Interface
d) Stability
e) Operational load testing
VVSPT-A2-PDD-013 v4.0 7-1 05/16/2017

7.1.2 OPERATIONAL EFFECTIVENESS AND SUITABILITY TESTING

Operational effectiveness and suitability testing includes user participation and may consist of
the following:
a) Reliability
b) Maintainability
c) Availability
d) Supportability
e) Degraded operations
f) Stress and NAS load testing of all interoperable subsystems
g) Human Factors (HF) evaluations
h) Safety requirements validation and testing to identify new safety hazards
i) Security
j) Site adaptation
k) Transition and switchover/fallback
l) Certification criteria
m) Training
n) OT Regression Testing
7.2 P URPOSE OF OT
The purpose of OT is to validate operational readiness by evaluating how well the system or
service satisfies the Critical Operational Issues (COIs), Critical Performance Requirements
(CPRs), and the requirements specified in the PRD of the system under test. All requirements in
the PRD may not be addressable in OT. The OT VRTM identifies those PRD requirements that
are addressed in OT. COIs focus on the system’s overall capability to support the operational
mission. CPRs are requirements deemed essential to the successful performance of the system or
service in meeting the program’s mission needs. (Note: If CPRs are not delineated in the PRD,
then they must be identified by the Integrated Test Team (ITT).)
A COI is decomposed into Measures of Effectiveness (MOEs) and Measures of Suitability
(MOSs), both of which are analyzed to identify Measures of Performance (MOPs). The MOPs
contain testable parameters which form the basis of the OT test case objectives. These test case
objectives form the foundation for test procedures by consolidating them into logical and
comprehensive test run components.
COIs, MOEs, MOSs, and MOPs are described in more detail in Section 5.2.3.2, Decomposition
of Critical Operational Issues. CPRs are described in more detail in Section 5.2.3.3, CPR
Evaluation Approach.
VVSPT-A2-PDD-013 v4.0 7-2 05/16/2017

7.3 OT T EST T EAM T ASKS AND F UNCTI ONS

The OT test team conducts operational testing of a new or modified system in accordance with
the tasks and functions described in the following subsections (see Figure 7-1, FAA OT Process
Flow).
7.3.1 OT TEST TEAM ORGANIZATION
The OT Test Director has overall program responsibility for OT, and works with the ITT to
define personnel and resource requirements for the OT program. In large programs, the OT Test
Director designates select test team members as Test Leads. The various Test Lead
responsibilities are selectively assigned on the basis of each member’s level of experience and
area of specialization. Test Leads are responsible for all activities related to their respective test
area. For small or non-complex programs, the OT Test Director may also serve as the Test Lead.
Additionally, for programs that have been designated for an IOA, the Independent Safety
Assessment Team will monitor OT activities.
7.3.2 OT PLANNING IN TEMP DEVELOPMENT

Planning for an OT program begins with the formation of the ITT. The OT Test Director and
OT test team work in conjunction with the DT Test Director, DT test team, and other program
stakeholders to develop the TEMP. This document incorporates DT and OT approaches, which
include an initial test design for determining system performance, effectiveness, and operational
suitability.
The OT section of the TEMP details the following:
a) OT philosophy and approach
b) All systems and configurations to be tested
c) Any required special facilities or test capabilities, including telemetry, aircraft for
flight testing, and support equipment resources
d) Roles and responsibilities for the OT Test Director, test team, field leads, and
participants
e) Test entrance and exit criteria
f) Methodologies for resolving COIs
g) OT limitations and their possible impacts on validation of OT requirements (e.g.,
incomplete or unavailable test environment(s), operational interfaces, operational
facilities, resources, training, and technical manuals)
h) Plans and methods for early, interim, and final reporting on OT issues and results
For programs using Agile development methods (see Section 6.1.1) there may be incremental
deployments to the field. For every increment that will be put into operational use the
government must verify and validate that the system is operationally ready.
VVSPT-A2-PDD-013 v4.0 7-3 05/16/2017

F IGURE 7-1. FAA OT P ROCESS F LO W
VVSPT-A2-PDD-013 v4.0 7-4 05/16/2017

7.3.3 OT SUPPORT TO CONTRACT ADMINISTRATION

To ensure that the operational perspective is taken into account during the design, development,
and testing of a new system, and to ensure that contractor support to the OT effort is made
available, the OT Test Director works with the DT Test Director to develop the T&E
requirements for the Screening Information Request (SIR). The SIR includes a description of
work requirements (e.g., Statement of Work (SOW), Performance Work Statement (PWS) or
Statement of Objectives (SOO)) for the contractor’s proposal. See Section 5.5 for a detailed
description of OT SIR support requirements.
7.3.4 OT TRAINING
The OT Test Director is expected to be an expert in the domain for the program in which that
person leads. Additionally, T&E specific training and qualification are required to ensure that
each OT Test Director understands the policies and processes related to T&E.
The OT test team receives:
a) Training on system operations and functionality, provided by the contractor as
required by the contract, along with participation in the oversight of the system
development by the contractor for system familiarization
b) T&E Handbook and QMS process training to ensure that T&E practitioners fully
understand what is expected to successfully perform the OT Test Program
c) Training through participation in prototype testbed activities and DT activities
d) Hands-on familiarization for the system under test
7.3.5 COORDINATION WITH OT FIELD PERSONNEL
Coordination with OT field participants starts during the T&E Planning and Support phase to
ensure that the OT field team is formed and their participation in the program is well-established
by the time formal OT takes place. The team is composed of field Subject Matter Experts
(SMEs), field supervisors, PMO representatives, and representatives of other organizations as
required.
It is recommended for large programs that a member of the team be selected as a liaison to serve
as a Point of Contact (POC) between the OT test team and the OT field team. This liaison assists
with:
a) Defining field personnel requirements
b) Coordination with field organizations, union representatives, and site personnel
c) Travel issues such as facility security requirements for visitors
d) OT test procedure development activities
e) OT test procedure dry runs
f) Formal OT conduct coordination
g) Field test team briefings
h) Defining operational issues
i) Collection and disposition of field test team issues
VVSPT-A2-PDD-013 v4.0 7-5 05/16/2017
7.3.6 OT TEST CAPABILITY ACCREDITATION

OT test capabilities must be accredited in accordance with Section 9.1.
7.4 OT P LANNI NG
7.4.1 OT TEST PLAN DEVELOPMENT
The OT Test Plan describes the planning and preparation activities required prior to the conduct
of OT. It includes a description of the testing to be accomplished, how test procedures will be
executed, and how test results will be reported. It provides sufficient detail to guide the
development of the OT Test Procedures.
The OT Test Plan addresses the following topics:
a) Test management
b) Identification and descriptions of tests to be conducted (providing enough detail so
that test procedures can be written based on these test descriptions)
c) Test tools
d) Test environments and facilities
e) Accreditation plans for test capabilities
f) Schedule
g) Configuration Management (CM) in accordance with the CM Plan
h) Safety (system and personnel safety)
i) Security
j) Requirements tracking through an OT VRTM for each PRD requirement allocated to
OT
k) Criteria identified to develop operational readiness determination (e.g., no Priority 1
or 2 PTRs associated with a COI)
l) Testing of interfaces to the NAS
m) Discrepancy/Program Trouble Reporting, categorization, and resolution
n) Test limitations
o) Test reporting strategy including CPR and COI status (Interim Assessment Reports,
Quicklook Test Reports, and Final Test Reports)
The OT Test Director must ensure that the OT Test Plan has been peer-reviewed (see Section
1.9) prior to submission to management for endorsement/approval. The OT Test Plan requires
endorsement by the TSB and the T&E First-Line Supervisor, and the approval of the respective
T&E Senior Manager and the Program Manager. Minor changes to the OT Test Plan that do not
impact test strategy or scope (e.g., minor schedule changes, editorial corrections, etc.) will be
considered working drafts only and do not require these approval signatures but are still subject
to Document Management and Control.
Refer to Appendix D, Figure D-10, for the complete OT Test Plan review and approval cycle.
VVSPT-A2-PDD-013 v4.0 7-6 05/16/2017

Note: Refer to the V&V Repository for the OT Test Plan template and OT Test Plan
samples.
7.4.2 OT VERIFICATION REQUIREMENTS TRACEABILITY MATRIX (VRTM)

The purpose of the OT VRTM is to address all COIs and program requirements to be verified
and validated as part of the OT test program. The OT VRTM accomplishes this by mapping the
COIs/MOEs/MOSs/MOPs and applicable program requirements from the PRD to the respective
OT activities and test cases in which the MOPs and requirements will be verified and validated.
The TEMP VRTM, which maps the COIs/MOEs/MOSs/MOPs and program requirements to the
respective test phase(s) and test activities, provides the basis for developing the OT VRTM (see
Section 5.2.3.5). During OT Test Plan development, the OT VRTM is further refined to include
COI/MOE/MOS/MOP and program requirement mapping to the individual OT cases.
7.4.3 OT ENTRANCE CRITERIA

The OT phase begins after GA of the contractor-developed system. In addition to ensuring that
the DT exit criteria have been satisfied, the OT Test Director plays a key role in ensuring that OT
entrance criteria are fully met before formal OT begins. This is accomplished through
participation in the ITT, early contractual planning, program meetings, and DT activities. The
minimum entrance criteria for formal OT include the following:
a) DT Exit criteria, as defined in the approved CMTP and the DT Plan(s), have been
satisfied
b) All Priority 1, Priority 2, and critical DT Program Trouble Reports (PTRs) that
impact OT have been fixed and incorporated into the OT baseline
c) The OT Test Plan has been reviewed and approved
d) The OT Test procedures have been reviewed and dry runs have been completed
e) All equipment and software required by the OT Test Plan and Procedures have been
installed and are available
f) All planned OT test capability accreditation has been completed and approved
g) The T&E First-Line Supervisor approves all waivers for deviations from test
standards in the T&E Handbook. Significant deviations are described in the TRR
documentation. All deviations from the OT Test Plan and approved waivers are
presented to the Program Manager or representative at TRR.
h) The OT TRR has been conducted (see Section 7.6.2). The Program Manager or
representative approves the start of formal OT with signature on the TRR package
The formal OT events may start once the above criteria have been satisfied
Note: Refer to the V&V Repository for the Request for Waiver form.
VVSPT-A2-PDD-013 v4.0 7-7 05/16/2017

7.4.4 OT EXIT CRITERIA

OT exit criteria ensure that the purpose of OT has been met and that the effectiveness and
suitability of the system has been adequately verified and validated for an operational readiness
determination.
Following formal OT conduct, the OT Test Director will facilitate a review of the OT results,
including a discussion of the OT exit criteria, with the ITT in order to determine whether OT can
be exited. Action plans must be created for any exit criteria items that have not been met. The
achievement of the OT Exit Criteria will initiate the finalization of the OT Final Test Report,
which is required for ISD entrance criteria. The minimum exit criteria for OT include the
following:
a) In accordance with the OT Test Plan and Test Procedures, the defined test case
objectives (including criteria and test case requirements), for each test must have
been addressed, met, and any support analyses completed and documented. These
include:
1) All planned OT test procedures have been satisfactorily executed
2) All MOP data has been collected and assessed
3) All COIs and CPRs are fully assessed and status reported
4) All OT artifacts from the Post-Test Reviews and the OT Caucus have been
reviewed (e.g., DRs, As-Run Test Procedures, test logs, questionnaires).
5) Completion of regression testing
b) Discrepancy Reports (DRs) and PTRs that impact the operational readiness of the
system have been fixed, verified, and regression-tested. These must be identified and
approved through the OT Caucus (see Section 7.6.5).
7.5 OT T EST P RO CEDURES

The OT test procedure development process consists of translating the test descriptions and
evaluation approaches contained in the OT Test Plan. The approach to OT data collection
includes both objective measurements of system parameters and subjective user-evaluation
questionnaires.
The OT Test Director oversees the development of the OT test procedures by the OT test team.
After the OT test procedures have been developed (including checkout and debugging), the OT
Test Director must ensure that the draft OT Test Procedures document has been peer-reviewed
(see Section 1.9). The draft OT Test Procedures should be made available to the TSB for
optional review. After receiving and incorporating any comments from the TSB, the OT Test
Director must approve the document for use in test conduct.
Refer to Appendix D, Figure D-11, for the complete OT Test Procedures review and approval
cycle.
Prior to formal OT, a dry run of the procedures must be performed. All OT test procedures must
address COIs, MOEs, MOSs, and MOPs through their respective test cases. Field site personnel
VVSPT-A2-PDD-013 v4.0 7-8 05/16/2017

should participate in the dry run process to ensure that the test environment, procedures, and
system are ready for formal testing.
7.5.1 OT TEST PROCEDURE DEVELOPMENT
OT test procedure development is comprised of the following three stages:
a) Procedure Development: The formulation of procedures that take into account
operational conditions, measures, user stimuli, and scenarios that address the OT
requirements identified in the OT VRTM. This step must also associate and
incorporate the MOEs, MOSs, and MOPs. It may also include system checkout
activities needed to support procedure development. Evaluating specific functionality
requires that the procedures contain step-by-step detailed instructions to ensure test
objectives are met and the intended data is collected. To evaluate operational
capabilities, high-level nonspecific test procedures are used to allow the participant to
realistically use the system and respond to events.
b) Procedure Debugging: The running of the test procedures against the system under
test to ensure that the test steps are correct and complete. During this stage, the
procedures are refined and updated to provide a logical flow to the sequence of test
steps.
c) Dry Runs: A complete end-to-end execution of the OT Test Procedures using formal
test configurations to assess the following criteria:
1) The laboratory environment is prepared
2) The system has been properly installed and configured
3) The system has the correct versions of both software and adaptation available, and
all system parameters that need to be set have been identified
4) Any new or modified test scenarios, simulations, or tests tools are valid
5) Procedures are mature and redline text changes are fully incorporated (i.e., test
procedures can be run start to finish, as documented, without major deviations or
anomalies)
Note: Refer to the V&V Repository for the OT Test Procedures template and an OT Test
Procedures sample.
7.5.2 OT TEST PROCEDURE QUESTIONNAIRES

Objective measures should always be the primary source of system information. Questionnaires
are used to augment objective performance measures or to provide substantive information when
objective data is unavailable or difficult to collect. Care should be taken if there is insufficient
objective data and emphasis falls on the questionnaire responses. The OT user-evaluation
questionnaires must:
a) Be effectively written to avoid the solicitation of biased replies. It is important that
the evaluation questions do not allow individual preferences of the test subject to be
reflected in the reply.
b) Be clear, concise, and directly relevant to the MOEs, MOSs, or MOPs to be assessed.
VVSPT-A2-PDD-013 v4.0 7-9 05/16/2017
c) Support the assessment of the capabilities of the system to support operational tasks.
d) Be in the form of numerical rating scales, or “yes” or “no” answers to questions. Test
subjects may provide subjective elaborations, or explanations on specific questions, to
support their responses.
Appropriate union approval from the National Air Traffic Controllers Association (NATCA),
National Association of Air Traffic Specialists (NAATS), or Professional Airways System
Specialists (PASS) is required before a questionnaire may be used. Approval may involve
significant lead time.
Note: Refer to the V&V Repository for the OT Questionnaire guidance and OT
Questionnaire samples.
7.5.3 OT TEST PROCEDURE TRACKING

The OT test team tracks and reports the status of OT test procedure development through an OT
Test Procedures Status Matrix. This status matrix will, at a minimum, indicate the progress
toward completion of the documentation, debugging, and dry runs required for each planned OT
test procedure. Additionally, a Test Status Report is generated at the conclusion of each test
activity (checkout, debug, or dry run) and provided to the ITT. Any anomalies or discrepancies
are reported and analyzed for their impact on performing formal OT.
Note: Refer to the V&V Repository for the OT Test Procedures Status Matrix and Test
Status Report templates.
7.6 OT T EST C ONDUCT

The OT test team, with the support of the field participants, conducts OT in accordance with the
approved OT Test Plan and Test Procedures. OT does not begin until the OT entrance criteria
(as documented in the TEMP and the OT Plan) are met. The ITT convenes to determine
compliance with the OT entrance criteria. An OT TRR is then conducted to ensure that all
elements are in place prior to commencing with formal OT conduct.
7.6.1 LOGISTICS FOR FORMAL TEST CONDUCT
The following list describes the logistical activities that need to be coordinated prior to the OT
TRR:
a) Approval of all required test-related NAS Change Proposals (NCPs).
b) Coordination of site access and security clearances.
c) Coordination of training for the test team and field participants.
d) Installation of the system under test is complete and complies with the documented
baseline.
e) Completion of the configuration baseline audit to identify all software, firmware, and
hardware versions in the system under test.
f) Ensuring that the OT Test Director accomplishes the following:
1) Defines roles and responsibilities of all team members.
VVSPT-A2-PDD-013 v4.0 7-10 05/16/2017
2) Coordinates all tasks required for test conduct. This includes laboratory
coordination and scheduling, data collection activities, Pre-Test Briefings and
Post-Test Reviews, observers, simulation support, and interface support.
3) Maintains all test schedules.
4) Establishes procedural checklists.
5) Conducts preliminary OT reviews of all scheduled activities with team members
and the Facility Manager.
6) Ensures that the Facility Manager is informed on all planned test activities.
7) Ensures that the Facility Manager coordinates planned test activities with adjacent
facilities.
8) Ensures that the Facility Manager coordinates and schedules required live testing
procedures, interfaces, and events.
7.6.2 OT TEST READINESS REVIEW
The OT TRR is conducted by the OT test team prior to the start of the formal OT activity. This
review verifies and approves the readiness to start formal OT and ensures participants understand
the outstanding issues. The TRR also communicates risks and limitations for formal OT. The
TRR includes:
a) Review of OT entrance criteria as documented in the OT Test Plan
b) Discussion and documentation of impacts if entry criteria are not fully met
c) OT approach, test objectives, and test structure used to determine operational
readiness
d) Test schedule
e) Hardware and software versions and configurations
f) CM process and test data management
g) Test team roles and responsibilities
h) Ensuring that all OT Test Plans and Test Procedures are approved by the
appropriate authorities
i) All OT procedure checkouts and dry runs have been conducted successfully
j) DR process
k) Reviewing known problems and workarounds (identify test impacts and criticality
of operational issues)
l) Test limitations, risks and deviations from the approved OT Test Plan are
understood and documented
m) Required test personnel are system-trained and available to support testing
n) Required technical and user documentation are current and available during
testing
VVSPT-A2-PDD-013 v4.0 7-11 05/16/2017

o) System adaptations are documented, available, and executable

p) Required Test Capability Accreditation Reports are complete
q) Data Reduction and Analysis (DR&A) tools, test tools, simulators, and emulators
are functional, configured, and certified
r) Review of OT exit criteria
s) All deviations from the OT Test Plan and approved waivers are presented to the
Program Manager or representative
t) Program Manager or representative approves/disapproves the start of formal OT
based on the TRR presentation, thereby accepting all waivers and deviations. A
signature on the TRR package is required for approval (email acceptable)
Before the start of the Test Readiness Review (TRR), a draft TRR package is provided to the
TSB at least five days prior to the event for review. The TSB may participate in the TRR event.
7.6.3 FORMAL OT CONDUCT
Formal OT conduct consists of Pre-Test Briefings, test executions, Post-Test Reviews, and status
reporting.
7.6.3.1 OT PRE-TEST BRIEFINGS
The OT Pre-Test Briefing is presented by the OT Test Lead and consists of the approach for each
test to be conducted. It includes a review of:
a) Test objectives and success criteria
b) Test configurations (hardware, adaptation, software level, patches, test tool
certification, and configuration files)
c) Test personnel assignments
d) Test conduct walkthrough
e) Proposed procedural changes or planned deviations
f) Test limitations
g) The status of all known problems and anomalies that may impact the test or its results
h) Methods of data collection
7.6.3.2 OT EXECUTION
OT case executions are the culmination of OT planning efforts resulting in the complete and
official running of the OT Test Procedures. These test cases are the primary data source for OT
requirements validation.
During formal OT, the Test Lead manages the test execution, maintains a log of all system state
changes, and records information concerning any laboratory or system issue that results in a
deviation of, or impact on, the test. Such incidents include, but are not necessarily limited to,
system reboots and hard failures. Test observer logs are used to capture issues that impact
operational effectiveness or suitability. If questionnaires are being used to capture qualitative
VVSPT-A2-PDD-013 v4.0 7-12 05/16/2017
and quantitative data, they must be completed and collected immediately following the test
execution. The Test Lead also maintains a list of DRs that identify all issues encountered during
testing (see Section 7.6.4).
7.6.3.3 OT POST-TEST REVIEWS AND TEST STATUS REPORTING
The OT Test Director should hold reviews at appropriate intervals during a test activity, such as:
a) Reviews of test case results at the end of each day
b) Reviews at the end of a test-case group
c) Reviews at the end of a test activity
d) Final Post Test Review
These reviews are all held prior to the OT Caucus. The main focus is to get field personnel to
clarify any issues that are not fully documented and to hold initial discussions on any impacts on
operational effectiveness or suitability. The development engineers responsible for
troubleshooting issues should attend to ensure that they fully understand the issues. The OT Test
Director or Test Lead generates briefing minutes and assigns action items.
The reviews may include:
a) Review of test results, including a summary of verified requirements
b) The status of test objectives and success criteria
c) Review of issues in the respective logs of the OT Test Lead and observers
d) Review of problems for accuracy, completeness, and recommended dispositions
e) Review of blue-line deviations (changes to test procedures made during formal test
conduct) from the test procedures
f) Recommendations for retesting or regression testing
Test status reports are generated following the completion of the above reviews. These reports
summarize the results of testing, highlight significant findings, and provide an assessment of
whether the objectives were met. They also record DRs and comments made against the system
documentation. The contents of the OT Post-Test Reviews and Test Status Reports are used to
support development of the OT Quicklook Test Report and Final Test Report.
Note: Refer to the V&V Repository for the Test Status Report template.
7.6.4 OT DISCREPANCY REPORTING

DRs are generated for all issues discovered during OT. DRs are also generated prior to OT for
all issues discovered that impact operational requirements but are not being addressed and need
to be evaluated during OT. A DR must contain enough detail to document the issue and
facilitate corrective action. Such details include, but are not necessarily limited to, the following:
a) Unique DR identifier
b) Project or system identification
c) Requirement(s) impacted, including reference to COI/MOE/MOS/MOP affected
VVSPT-A2-PDD-013 v4.0 7-13 05/16/2017

d) Description of the issue

e) Priority of the problem (see Section 6.3.7)
f) Impact of the problem
g) Name or number of the test procedure
h) Test step at which the discrepancy occurred
i) Test configuration and system states
j) Supporting records and data (e.g., logs, screen saves, system records, system reports,
test tool data, plots, etc.)
k) Associated PTR number (if applicable)
l) Associated resolution upon closure
Note: Refer to the V&V Repository for the Discrepancy Report form.
The OT Test Director and test team must conduct DR reviews on a regular basis to ensure
accurate descriptions, determine validity, disposition priorities, and assess proposed resolutions.
To facilitate efficient tracking of issues discovered during OT, the OT Test Director maintains a
DR database. DRs are tracked until dispositions are documented for each and are closed out
when the OT Test Director and the relevant stakeholders are satisfied that the issue has been
resolved or mitigated. All DRs that require corrective action by the contractor are raised to PTR
status. DRs that have been raised to PTRs are not closed out as DRs until corrective action has
been implemented by the contractor and verified by the FAA.
7.6.5 OT CAUCUS
The OT Caucus, chaired by the OT Test Director, is conducted to present and review all findings
and test results related to operational effectiveness and suitability of the system. The focus of the
caucus is to gain concurrence on the operational impact and criticality of all DRs and PTRs. At a
minimum participants include representatives from the OT test team, and the field team(s).
The OT Caucus is held after the completion of formal testing. In preparation for the caucus, all
OT DRs and PTRs are entered into an OT Problem Traceability Matrix. At the caucus, the
matrix is reviewed to assess the impact of DRs and PTRs on COIs, MOEs, MOSs, MOPs, and
CPRs, and to determine their operational criticality. All DRs that require corrective action by the
contractor are raised to PTR status. PTRs are assigned initial priorities to assist with resolution
plans. At the caucus, each COI must be answered based on the criticality of the problem(s)
impacting it. Figure 7-2 shows the process flow for determining the degree to which COIs and
program requirements are met.
VVSPT-A2-PDD-013 v4.0 7-14 05/16/2017

F IGURE 7-2. F ROM R EQUI REMENTS TO O PERATIONAL R EADI NESS D ETERMINATION
VVSPT-A2-PDD-013 v4.0 7-15 05/16/2017

The following standard practices and activities must be considered for the conduct of the OT
Caucus:
a) The assessment of COIs must focus on operational impacts:
1) Ensure that adverse conditions are clearly linked to operational issues.
2) Clearly identify the specific operational functions, procedures, or services that are
interrupted, delayed, or degraded.
3) Indicate suitable workarounds, or if workarounds are not feasible.
b) Every issue must be well supported by references to the established CPRs, and COIs.
The DRs are related to COIs through their constituent MOPs, MOEs and MOSs in the
OT Problem Traceability Matrix. The matrix shows whether each COI is fully met,
or not met due to significant issues. It can also indicate when a COI is limited, due to
less significant issues, but not enough to prevent the system from being operationally
effective or suitable. (See Glossary for COI status definitions of Yes, Ltd (Limited),
and No).
c) In addition to issues specifically impacting COIs, discuss and document general
issues and concerns that the OT participants found to affect operational suitability and
effectiveness. Ensure that these issues and concerns have sufficient justification to be
documented as an operational impact (i.e. impact or degradation to operational
functions or services).
d) The caucus should not make programmatic decisions (such as milestone, deployment,
scheduling, or staffing decisions).
1) The OT Caucus should only provide supporting information for programmatic
decisions.
2) The caucus can address site unique considerations that support site deployment
decision or Key site planning.
e) The OT Caucus should review the established OT Exit Criteria, as indicated in
Section 7.4.4, and report status of any outstanding items.
f) Ensure that the caucus activities provide sufficient time for stakeholders from
different functional areas to collaborate on operational issues, to provide a
comprehensive and holistic assessment of operational impacts.
An OT Caucus Summary Report is produced to document the status of each COI, MOE, MOS,
MOP, and CPR for the system under test. This report supports resolution of critical PTRs prior
to operational deployment. This report and the PTR resolution plans provide the framework for
any regression testing that may be necessary to resolve outstanding critical PTRs. The Caucus
Summary report is the artifact that provides data that will go into the Quicklook and Final
Reports.
Note: Refer to the V&V Repository for the OT Problem Traceability Matrix and OT
Caucus Summary Report templates.
7.6.6 OT REGRESSION TESTING

OT regression testing is conducted to verify the integrity of solutions to DRs/PTRs.
Additionally, regression testing is performed to ensure that these solutions have not introduced
any new problems or issues, and to support validation of the system’s readiness for operational
VVSPT-A2-PDD-013 v4.0 7-16 05/16/2017

use. Regression testing may require the participation of the OT field team(s). Regression testing
must:
a) Ensure the functionality of corrective actions
b) Demonstrate that the corrective actions do not impact other system functionality
c) Re-run OT test procedures that were impacted by the problem
d) Re-run OT test procedures that may be impacted by the fix
e) Follow the same processes required for formal OT
f) Be documented in an OT Quicklook Test Report or the OT Final Test Report
7.7 OT T EST R EPORTING
OT test reporting is comprised of the following reports:
a) Interim Assessment Report(s)
b) Quicklook Test Report(s)
c) Final Test Report
7.7.1 OT INTERIM ASSESSMENT REPORT
The OT IAR is an optional reporting mechanism that will provide management with an
assessment of the current state and maturity of the system by identifying system capabilities and
limitations as tested for that reporting period. OT IARs are developed following specific
milestones, whether issues exist or not, as defined in the TEMP. The OT IAR must be provided
to the TSB and ITT for their review and comment.
The OT IAR will provide sufficient data to support resolution plans and program decisions.
Additionally, the OT IAR will assist in the planning for future test activities and support
planning for system implementation and deployment. Specifically, the OT IAR:
a) Provides the status of critical performance criteria defined in the TEMP
b) Analyzes issues based on their impact on COIs and CPRs
c) Provides early operational reporting for:
1) DT
2) Pre-formal OT conduct
3) Formal OT conduct
d) Highlights critical system issues which may impact the following operational
milestones:
1) Initial Operational Capability (IOC)
2) ISD
3) Operational Readiness Dates at field sites
e) Provides support for programmatic decision-making (including scheduling, test
planning, site deployment, and site acceptance testing)
VVSPT-A2-PDD-013 v4.0 7-17 05/16/2017

f) Supports operations and maintenance planning

g) Provides input to the OT Quicklook Test Report and OT Final Test Report
The OT Test Director must provide the draft OT IAR to the TSB and the ITT for review and
comment prior to electronically distributing the final document to the respective T&E Senior
Manager, the Program Manager, the T&E First-Line Supervisor, and the TSB.
Refer to Appendix D, Figure D-12, for the complete OT IAR review and approval cycle.
Note: Refer to the V&V Repository for the OT IAR template.
7.7.2 OT QUICKLOOK TEST REPORT

The OT Quicklook Test Report is developed following the completion of formal OT to provide a
preliminary assessment of operational readiness. This report is provided in advance of the more
detailed Final Test Report which requires more analysis and time to complete. The report
contains preliminary results, and therefore is not intended to support programmatic or
operational milestone decisions.
The OT Quicklook Test Report documents the preliminary results of all formal testing and the
resolution plans of any issues discussed at the OT Caucus. It expands upon the content provided
through the OT Caucus Summary Report. General descriptions of the system’s performance and
functional and operational limitations are included in the Quicklook Test Report. If the
resolution of program issues delays delivery of the OT Final Test Report, an OT Quicklook Test
Report may be required after each major OT regression test.
The OT Quicklook Test Report includes:
a) An executive summary
b) Background information on the testing
c) A summary of test activities, including test article configuration
d) Results from Caucus Summary Report
e) A synopsis of preliminary test results
f) Preliminary conclusions, including a preliminary determination of operational
readiness
g) OT Issues Matrix as an appendix
It is critical that the draft OT Quicklook Test Report be provided to the ITT and the TSB for their
review and comment in time to support problem resolution and IOC/ISD preparation activities.
The approved OT Quicklook Test Report should be delivered within 15 calendar days from the
completion of the test. The approved OT Quicklook Test Report must be electronically
distributed to the respective T&E Senior Manager, the Program Manager, the T&E First-Line
Supervisor, the Technical Center Director, and the TSB.
The OT Quicklook Test Report is preliminary, and is not intended to support programmatic or
operational milestone decisions. However, if directed by the PMO to use the report in this
manner (for example, National Deployment, IOC, ORD, ISD) or in lieu of a final report, then
VVSPT-A2-PDD-013 v4.0 7-18 05/16/2017

consult the additional guidance in the OT Quick Look Report Template, which includes,
endorsement by the Test Standards Board (TSB) and signature of the WJHTC Director.
Refer to Appendix D, Figure D-13, for the complete OT Quicklook Test Report review and
approval cycle.
Note: Refer to the V&V Repository for the OT Quicklook Test Report template and an OT
Quicklook Test Report sample.
7.7.3 OT FINAL TEST REPORT

The OT Final Test Report presents specific findings related to the operational effectiveness and
suitability of the system. The report:
a) Contains an executive summary which summarizes the test results
b) Addresses the resolution of all COIs
c) Provides a determination of operational readiness of the system, (operational
effectiveness and suitability)
d) Characterizes the operational capabilities and limitations of the system based on
system performance against the CPRs
e) Documents the methods used in, and results of, the detailed analyses of test data
f) Identifies and assesses PTRs and issues that impact critical capabilities and benefits
g) Updates the status of problems highlighted in the OT Quicklook Test Report
h) Provides status of exit criteria as defined in the OT Test Plan
i) Includes test descriptions, results, conclusions, and recommendations
j) Identifies technical or operational risk areas
k) Includes all approved deviations and waivers generated as a result of testing
Conclusions regarding operational effectiveness and suitability are based on the combined results
of all operational tests. Operational readiness is determined on the basis of operational
effectiveness and suitability performance in accordance with COIs and CPRs. In addition, the
report addresses the risks or system limitations associated with shortfalls in meeting mission
needs, operational requirements, and safety requirements. Identification and assessment of any
new safety hazards should also be provided.
The OT Final Test Report is used to support an In-Service Decision (ISD) or other deployment
decision, and must include a very clear statement regarding whether or not the system is
Operationally Ready (effective and suitable). The following examples illustrate such clear
unambiguous language.
Example 1: “The OT test team found the system Not Operationally Ready because COI
3.0 was not answered satisfactorily, and CPR 5.0 was not successfully met, due to Issue
xx which was rated Priority 1.”
VVSPT-A2-PDD-013 v4.0 7-19 05/16/2017

Example 2: “The OT test team found the system Operationally Ready because all COIs
were satisfactory answered and CPRs were successfully met. All issues found during
testing were rated Priority 3 or below.”
For large or high-risk test programs, the OT Test Director must conduct an OT Final Test Report
Out-Brief to the ITT prior to delivering a draft Final Test Report for review. Additionally, the
OT Test Director must ensure that the report has been peer-reviewed (see Section 1.9) prior to
submission to management for review/endorsement/approval. The OT Final Test Report
requires:
a) Endorsement of the TSB, the T&E First Line Supervisor, and the respective T&E
Senior Manager
b) Signature of the OT Test Director
c) Approval of the Technical Center Director
The Technical Center Director will approve the final version of the report based on earlier
endorsements and approvals, and an OT Final Test Report Out-Brief by the OT Test Director (as
required).
The approved OT Final Test Report should be delivered within 60 calendar days from the
completion of the test. The approved OT Final Test Report is delivered to the TSB, and the
Program Manager. The AMS requires the OT Final Test Report as part of the entrance criteria to
the ISD. The Technical Center Director will provide an OT Final Test Report Out-Brief, as
required, to the Assistant Administrator for NextGen, ANG -1, prior to the ISD milestone.
Refer to Appendix D, Figure D-14, for the complete OT Final Test Report review and approval
cycle.
Note: Refer to the V&V Repository for the OT Final Test Report and OT Final Test
Report Out-Brief templates and samples.
7.8 T RANSI TION TO I N -S ERVI CE M ANAGEMENT

Test support is often required during the transitional period from OT to In-Service Management
(ISM) to correct PTRs and other performance issues. This test support is handled through
service-level agreements between the DT/OT test teams and the ISM organization, or via a
Project Scope Agreement with the PMO.
7.9 F IELD F AMILIARI ZATION S UPPORT
This section only addresses processes to be followed by the T&E organization in support of Field
Familiarization (FF). Site FF is the responsibility of the facility personnel at each operational
site and is the final activity to achieve facility IOC. If requested by the PMO, the test
organization provides support services for FF, including the development of plans, procedures,
and briefing materials. The guidance for FF conduct and planning is contained in the AMS T&E
guidance document. FF is a formal T&E activity. However, its main focus is to familiarize field
users, rather than providing for formal verification or validation of the system.
7.9.1 FIELD FAMILIARIZATION SUPPORT PLANNING
FF support usually consists of running test procedures that go beyond the usual site system
verification and baseline testing. The general approach is to conduct activities that will allow the
VVSPT-A2-PDD-013 v4.0 7-20 05/16/2017
facility to gain confidence in the system and attain a higher level of hands-on familiarization.
The level and type of FF support is determined through coordination with the PMO, AT site
representatives, and Technical Operations site representatives to identify specific areas that
require T&E capabilities. Once the test organization’s role is determined, an FF Support Plan is
developed that details the activities. The FF Support Plan should clearly identify who generates
and maintains the FF artifacts such as daily status reports and site final report. The plan must be
provided to the TSB for their review and comment prior to final delivery. FF support consists of
the following activities:
a) Conduct initial planning meetings with operational and in-service management
organizations to gain insight into operational concerns
b) Meet with field sites to gain the desired type of faults for system familiarization
c) Perform an assessment of familiarization needs for the site and define roles for the FF
support team
d) Develop a support schedule and status tracking matrix
e) Develop a resource allocation matrix
f) Assess engineering resources required for test case development and support
g) Define FF support team training requirements
h) Define FF test tools as required
i) Develop and document test case procedures
j) Dry run test cases
k) Develop schedules to track all FF-related activities
l) Distribute the FF Support Plan to the PMO and the site(s)
Refer to Appendix D, Figure D-15, for the complete Field Familiarization Support Plan review
and approval cycle.
Note: Refer to the V&V Repository for the FF Support Plan template.
7.9.2 FIELD FAMILIARIZATION SUPPORT PREPARATION

Test procedures are developed and a dry run is conducted based on the process specified in the
FF Support Plan. These procedures are packaged and delivered to each Site Test Director so that
they may select specific test procedures for execution.
FF support preparation is performed in the following sequence:
a) Schedule resources and tools for test case development and dry runs
b) Conduct test case dry runs on new system releases
c) Conduct site preparation meetings/briefings 30 days prior to the start of site FF
d) Participate in training activities on system overview and architecture, Computer
Based Instruction, Test Training, Operational Evaluations, etc.
VVSPT-A2-PDD-013 v4.0 7-21 05/16/2017

e) Develop and maintain FF database as central repository for FF-related documents and
data storage
Note: Refer to the V&V Repository for FF Test Procedures and FF Test Case Format
samples.
7.9.3 FIELD FAMILIARIZATION SUPPORT CONDUCT

The following standard processes must be considered for the conduct of FF support activities:
a) Ensure required test equipment, tools, and documentation are delivered to the site
b) Conduct a site kick-off meeting at the start of FF site activities
c) Conduct Pre-Test Briefings prior to each individual test
d) Conduct FF testing
e) Provide support to updating site-specific procedures, handbooks, and guidance
documents (as required)
f) Conduct Post-Test Reviews at the completion of each individual test
g) Conduct a wrap-up meeting at the completion of FF site activities
h) Perform regression testing on issues identified by the site (as required)
i) Assist in PTR verification and analysis (as required)
j) Generate Test Status Reports and submit them daily to the Program Manager, Test
Directors, and Site Test Director
7.9.4 FIELD FAMILIARIZATION REPORTING
The site FF support team tracks all site issues surfaced during the test activities and maintains
these issues in a database. A final report for each site is generated to document the activities and
summarize any issues. Reports and database information are distributed to the PMO, TSB, and
site representatives.
Note: Refer to the V&V Repository for FF Report and FF Status Tracking Matrix
samples.
VVSPT-A2-PDD-013 v4.0 7-22 05/16/2017

8 T&E SUPPORT TO IN-SERVICE MANAGEMENT

T&E support to ISM consists of various T&E activities performed by the ANG-E test teams in
support of program maintenance and upgrade activities under the responsibility of the second-
level maintenance organizations. These ANG-E T&E activities can be categorized according to
the following types of ISM program activities:
a) Major program upgrade support
b) Minor program upgrade support
c) Program maintenance support
The following sections define each of these ISM program activities and describe the T&E
support provided by the ANG-E test team if tasked by the PMO or the second-level maintenance
organization to provide T&E services. Additionally, it should be noted that the specific level of
ANG-E T&E service support may differ from program to program and tailoring may be required.
8.1 M AJOR P ROGRAM U PGRADE S UPPORT
Major program upgrades to operational systems or services are defined as those upgrades that
require new or modified program requirements, a new or modified vendor contract, and a
corresponding DT and OT test program to verify and validate the upgrade prior to its
implementation. For these types of upgrades, the test team will be responsible for developing a
TEMP, DT oversight, and OT planning, conduct, and reporting. As such, the applicable T&E
processes, methods, and standards defined in Sections 4 through 7 and 9 of this Handbook will
be followed, to include the use of the Test Planning, DT and OT Process Conformance
Checklists contained in Appendices A through C. If there is an ISD identified, then the program
must be considered as major, but not all major ISM programs have an ISD.
Note: Refer to the V&V Repository for the ISM TEMP, OT Test Plan and OT Final Test
Report templates.
8.2 M INOR P ROGRAM U PGRADE S UPPORT

Minor program upgrades to operational systems or services are defined as those upgrades
resulting from Program Trouble Report (PTR) fixes or National Change Proposals (NCPs)/case
files for which there is no need for a new or modified vendor contract. In these instances, the
second-level maintenance organization or the vendor will implement and test the upgrade, while
the ANG-E test team will only be responsible for OT planning, conduct, and reporting. As such,
in the same manner as the T&E support provided to major program upgrades as defined in
Section 8.1, the applicable T&E processes, methods, and standards defined in Sections 7 and 9 of
this Handbook will be followed.
8.3 P ROGRAM M AINTENANCE S UPPORT
In addition to the more formalized T&E support for major and minor program upgrades, the
ANG-E test team may be requested to provide T&E support, as Special Support Activities, for
program maintenance activities being performed by the second-level maintenance organization.
For these activities, the ANG-E test team will provide T&E services in a supporting role to the
second-level maintenance organization. As such, the T&E processes, methods, and standards
VVSPT-A2-PDD-013 v4.0 8-1 05/16/2017
established by the second-level maintenance organization will be followed in lieu of those

prescribed by this Handbook. However, for all T&E activities performed in support of ISM
program maintenance, it is recommended that the ANG-E test team applies, where possible, the
fundamental practices for quality T&E as defined in Section 1.5.
Note: Refer to the V&V Repository for the Special Support Activities PDD.
VVSPT-A2-PDD-013 v4.0 8-2 05/16/2017

9 TEST PREPARATION AND DEPLOYMENT DECISION SUPPORT

This section contains test preparation and deployment decision support processes that cross AMS
lifecycle phases.
9.1 T EST C APABILITY A CCREDI TATION
Accreditation is the official determination that a model or simulation (or other test capability) is
acceptable for a specific purpose. Accreditation is required for every test capability that is being
used to verify and validate requirements. DT and OT Test Directors are responsible for
identifying all DT and OT accreditation requirements, including proposed accreditation methods,
in the TEMP. These proposed accreditation requirements and methods will be approved in
conjunction with the overall TEMP approval process (see Section 5.2).
When test capability accreditation is required, accreditation plans and procedures must be
detailed in a standalone Accreditation Plan, or in DT/OT test plans. Results from accreditation
testing must be documented in an Accreditation Report prior to the start of system or service
testing.
For detailed Accreditation Plan, Procedure and Report work product approval processes see
Appendix D. Figure 9-1, Test Capability Accreditation Process, depicts the logical flow of the
accreditation process. The following sections provide additional information on the process.
9.1.1 ACCREDITATION DEFINITIONS

The following terms and definitions applicable to the accreditation process are presented for the
convenience of the reader:
a) Accreditation: The formal certification that a test capability is acceptable for a
specific application.
b) Test Capability: Test capabilities are assets that are used in conjunction with the
system/service under test or a representation of the system/service under test to
generate data to address test measures. Test capabilities include testbeds, simulated
environments (including files and interfaces), instrumentation and test tools
(including data collection and analysis), and modeling capabilities.
c) Accreditation Verification: The formal process of determining that a test capability
accurately represents the developer's conceptual description and specifications.
d) Accreditation Validation: The formal process of determining the degree to which a
test capability provides an accurate and complete representation of the test
capability’s intended purpose (e.g., real world system loading, capacity loading,
emulation of operational environments, etc.).
VVSPT-A2-PDD-013 v4.0 9-1 05/16/2017

F IGURE 9-1. T EST C APABILI TY A CCREDITATIO N P ROCESS
9.1.2 ACCREDITATION TESTING

Accreditation tests must ensure that the test capability achieves its intended purpose in
supporting a T&E activity. The level of accreditation verification and validation to be performed
VVSPT-A2-PDD-013 v4.0 9-2 05/16/2017

will be driven by the required test capability performance, functionality, and fidelity necessary to
support fulfillment of the objectives of a test activity or an individual test case. Examples of the
test capabilities typically accredited and types of accreditation are as follows:
a) Testbeds – A testbed is a stand-alone or distributed environment created for testing
purposes. It may consist of a combination of specialized hardware, software, real or
simulated environmental conditions. Verification of the testbed includes ensuring
that interfaces and components of the testbed function as designed and are free of
defects. Validation of the testbed involves ensuring that the environment and
associated stimulus provide sufficient representation of the conditions required for the
test objectives.
b) Simulated environments include:
1) Simulation Files (Scenarios/Scripts) – Simulation files or software scenarios are
used to automate procedures, system load, specific complex situations, etc.
Verification of simulations ensures that the simulation performs as scripted and is
reliable, repeatable, and free of defects. Simulations are validated to ensure that
they are realistic, comprehensive for the operational environment being simulated,
and sufficient for the intended use.
2) Simulated Interfaces – One or more interfaces are typically simulated when live
data or other system interfaces are not available or practical. Verification of the
simulated interface includes ensuring that it conforms to the Interface Control
Document (or other interface requirements). Validation of the interface should
ensure that all the appropriate messages, data, timing, and content are sufficiently
emulated by the simulated interface.
c) Instrumentation and Test Tools – Instrumentation and test tools are equipment items
that include data collection and analysis tools, drivers, oscilloscopes, meters,
analyzers, and probes. Instrumentation and test tools may be COTS or developmental
items. Verification of the instrumentation and test tools ensures that they are
performing as designed. Validation of the instrumentation and test tools ensure that
they are meeting the needs for the test. Calibration may be performed to accomplish
the accreditation. COTS items may use the vendor artifacts and documentation for
verification and validation.
d) Modeling – Modeling is a physical, mathematical, or otherwise logical representation
of a system, entity, or process. Typically, modeling is used to test and evaluate future
real-world capabilities without actually implementing them. Verification of a model
ensures that the model is fully functional, reliable, and accurately
processing/reporting data. Validation of a model focuses on ensuring that the model
encompasses all relevant variables, is representative of intended environments, and
sufficiently emulates the real-world capabilities that are identified in the test case.
9.1.3 ACCREDITATION PROCESS
Test capabilities that require accreditation are identified during test design and are documented
in the TEMP and test plans. Accreditation Plans and Procedures are developed in parallel with
the test plans. This provides the opportunity to explore alternative test approaches or
modifications to the overall test process. Additionally, Accreditation Plans and Procedures may
VVSPT-A2-PDD-013 v4.0 9-3 05/16/2017
be either standalone documents referenced in the test plans, or they may be included as part of
the test plans. The prime contractor (for DT activities) or the OT Test Director (for OT
activities) is responsible for the development of the respective Accreditation Plan(s). For DT,
the DT test team conducts an early informal review of the plan(s) to ensure that they are in
agreement with the contractor’s accreditation strategy and to provide any other initial comments.
Subsequently, the DT test team and the TSB review the revised plan(s) for technical sufficiency
prior to the DT Test Director’s recommendation of approval by the Contracting Officer (CO).
For OT, the TSB endorses the Accreditation Plan(s) prior to final approval by the respective
T&E Senior Manager.
The FAA will witness all test capability accreditation activities conducted by the prime
contractor for DT, and will conduct the test capability accreditation activities for OT. The
accreditation analysis and results from these activities should be completed and approved no
later than 30 days before the test activity, or commencement of the simulation effort, requiring
the use of the test capability(ies). If this is not possible, the prime contractor (for DT) or OT Test
Director (for OT) has the prerogative of proposing an alternative completion date. The proposal
should contain pertinent rationales and call out any associated risks.
Results of the accreditation process are described in the Accreditation Report. This report
provides information on the risks associated with using the test capability, and recommendations
on whether to proceed. For DT, the prime contractor prepares the Accreditation Report and the
DT test team conducts an early informal review of the report to ensure that they are in agreement
with the contractor’s accreditation results and to provide any other initial comments.
Subsequently, the DT test team and the TSB review the revised report prior to the DT Test
Director’s approval recommendation to the CO and COR. For OT, the OT Test Director
prepares the Accreditation Report, the TSB reviews the report, and the T&E First-Line
Supervisor endorses the report prior to final approval by the respective T&E Senior Manager.
Configuration Management (CM) of test capabilities and archiving of all Accreditation Plans,
Procedures, and Reports is the responsibility of the prime contractor (for DT) or the OT Test
Director (for OT).
Refer to Appendix D, Figure D-9, for the complete DT Accreditation Plan and Accreditation
Report review and approval cycle, and Figure D-16 for the complete OT Accreditation Plan and
Accreditation Report review and approval cycle. For OT Accreditation Procedures refer to
Figure D-11, OT Test Procedures Review and Approval Cycle, which also applies to
Accreditation Procedures.
9.1.4 ACCREDITATION GUIDANCE
Accreditation activities for new test capabilities generally consist of the following steps:
a) Describe the function and features of the test capability being accredited as well as
the inputs and outputs.
b) Determine the intended use of the capability during testing.
c) Determine the test capability accreditation criteria. This may include documents such
as the user manual, specification and design document or may include subject matter
expertise.
VVSPT-A2-PDD-013 v4.0 9-4 05/16/2017

d) Using known inputs, perform accreditation testing to exercise the capability, verifying
and validating the functions and features according to the intended use for the test.
e) Compare the outputs of the accreditation testing against the accreditation criteria for
the test capability and document the results.
In lieu of accreditation, an approved documented artifact (e.g., a prior Accreditation Report, test
logs, as-run authenticated test procedures, test data, etc.) is required for established capabilities
used in past testing or previously accredited capabilities. However, if the test capability or
environment has changed, then the test capability requires re-accreditation in the new test
environment.
The Accreditation Plan must describe the methods used to verify and validate the test capability
and must also identify any supporting artifacts. Once a test capability has been accredited, it
must be placed under CM in accordance with the CM Plan and monitored to ensure that the
accreditation standards are maintained. The Accreditation Plan must document the conditions
under which the test capability will require a re-accreditation process. Examples of such
conditions include changes to the test capability algorithms, inputs or outputs.
Note: Refer to the V&V Repository for the Test Capability Accreditation template and the
CM PDD.
9.2 I N -S ERVI CE R EVI EW C HECKLIST AND D EPLOYMENT D ECISIO N

P LANNING
Deployment planning is part of a continuous In-Service Review (ISR) process that begins early
in the lifecycle management process, usually during the development of requirements. A
tailored ISR Checklist is used to ensure that all deployment and implementation issues are
resolved prior to the In-Service Decision (ISD). This checklist will be used to: integrate
checklist issues with other emerging issues (such as PTRs from program tests and evaluation);
develop action plans for resolution of checklist and other issues; and document decisions and the
results of issue resolution and mitigation.
The service organization is responsible for developing, maintaining, and facilitating the ISR
Checklist. T&E support for the ISR checklist is provided by test POCs and FAA SMEs, who
tailor checklist items, provide action plans and status, and approve the Test Section of the ISR
Checklist from the AMS Investment Analysis phase until the ISD. An FAA SME is a
representative from the office to be contacted to provide expertise and additional information
about a checklist item's intent and will concur with the closure of each checklist item. The FAA
SME for test is the service organization test lead, which may be the DT or OT Test Director, or
both. The test POC is the individual that provides a response to the checklist items. The test
POC may also be the DT or OT Test Director, or an assigned test lead.
9.3 T EST NAS C HANGE P ROPOSALS
Test NCPs are typically required for temporary configuration changes to existing baselined
systems, including the installation of prototypes. FAA Order 1800.66 defines the Configuration
Control Board approval process for test NCPs. FAA Order 6032.1B provides additional
guidance on Test NCPs. The NCP approval process is lengthy and therefore, sufficient lead-time
VVSPT-A2-PDD-013 v4.0 9-5 05/16/2017

and preparation is necessary. In addition to the information referenced on the NCP, the case file
documentation for a test modification must include:
a) The general method for accomplishing the modification, including:
1) A description of system or system modification to be tested
2) A description of connections and interfacing systems
3) Descriptive diagrams as required
4) An installation plan
b) The applicable PRD or requirement statement(s)
c) A test plan and specific test procedures, including:
1) Test objectives
2) Evaluation plan
3) Test steps
4) Expected or desired results
5) Exit criteria
6) Anticipated duration of test
d) A description of removal plan
e) An estimate of associated costs
f) A complete schedule of all planned tasks and activities
VVSPT-A2-PDD-013 v4.0 9-6 05/16/2017

10 ACRONYMS AND GLOSSARY

10.1 A CRONYMS
AMS Acquisition Management System
APB Acquisition Program Baseline
AT Air Traffic
ATC Air Traffic Control
CDR Critical Design Review
CDRL Contract Data Requirements List
CM Configuration Management
CMMI Capability Maturity Model Integration
CMTP Contractor Master Test Plan
CO Contracting Officer
COI Critical Operational Issue
CONOPs Concept of Operations
COR Contracting Officer's Representative
COTS Commercial Off-the-Shelf
CPR Critical Performance Requirement
CRD Concept and Requirements Definition
CSC Computer Software Component
CSCI Computer Software Configuration Item
DID Data Item Description
DMC Document Management and Control
DR Discrepancy Report
DR&A Data Reduction and Analysis
DT Development Test(ing)
ECP Engineering Change Proposal
EMC Electromagnetic Compatibility
EMI Electromagnetic Interference
FAA Federal Aviation Administration
FAT Factory Acceptance Test(ing)
FCA Functional Configuration Audit
FF Field Familiarization
VVSPT-A2-PDD-013 v4.0 10-1 05/16/2017

FID Final Investment Decision

FQT Functional Qualification Test(ing)
GA Government Acceptance
GFE Government Furnished Equipment
GFI Government Furnished Information
GFP Government Furnished Property
HB Handbook
HF Human Factors
HWCI Hardware Configuration Item
I&I Installation and Integration
IA Investment Analysis
IAR Interim Assessment Report
IARD Investment Analysis Readiness Decision
IBR Integrated Baseline Review
IID Initial Investment Decision
IMS Integrated Master Schedule
IOA Independent Operational Assessment
IOARD Independent Operational Assessment Readiness Declaration
IOC Initial Operational Capability
iPRD Initial Program Requirements Document
ISD In-Service Decision
ISM In-Service Management
ISO International Organization for Standardization
ISPD Implementation Strategy and Planning Document
ISR In-Service Review
ITDP Integration and Test Development Plan
ITT Integrated Test Team
JRC Joint Resources Council
KSN Knowledge Services Network
LRU Line Replaceable Unit
MOE Measure of Effectiveness
MOP Measure of Performance
MOS Measure of Suitability
VVSPT-A2-PDD-013 v4.0 10-2 05/16/2017
NAATS National Association of Air Traffic Specialists

NAS National Airspace System
NATCA National Air Traffic Controllers Association
NCP NAS Change Proposal
NDI Non-Developmental Item
NextGen Next Generation Air Transportation System
OCD Operational Capability Demonstration
ORD Operational Readiness Date
OCT Operational Capability Test
OT Operational Test(ing)
PASS Professional Airways System Specialists
PAT Production Acceptance Test(ing)
PBSA Performance-Based Service Acquisition
PCA Physical Configuration Audit
PDD Process Description Document
PDR Preliminary Design Review
PM Project Management
PMO Program Management Office
POC Point of Contact
pPRD preliminary Program Requirements Document
PRD Program Requirements Document
PTR Program Trouble Report
PWS Performance Work Statement
QA Quality Assurance
QMS Quality Management System
QRO Quality Reliability Officer
RMA Reliability, Maintainability, and Availability
SAT Site Acceptance Testing
SAP Security Authorization Package
SEM System Engineering Manual
SEWG System Engineering Working Group
SI Solution Implementation
SIR Screening Information Request
VVSPT-A2-PDD-013 v4.0 10-3 05/16/2017
SME Subject Matter Expert

SOO Statement of Objectives
SOW Statement of Work
SRR System Requirements Review
T&E Test and Evaluation
TEMP Test and Evaluation Master Plan
TRR Test Readiness Review
TRRG Test Roles and Responsibilities Guide
TSB Test Standards Board
TWG Test Working Group
V&V Verification and Validation
VRTM Verification Requirements Traceability Matrix
WBS Work Breakdown Structure
WJHTC William J. Hughes Technical Center
10.2 G LOSSARY
Adaptation Unique site-dependent data/functions required by the
operational program to provide the flexible capability necessary
for individual site performance determined during
implementation.
A-Level Specification Another name for the System Specification Document (SSD).
The SSD must show traceability to the Product Requirements
Document (PRD).
B-Level Specification Subsystem specifications. Developed by the contractor and

must show traceability to the A-Level Specification.
Blue-line changes Changes made to test procedures during formal test conduct.
Dry Run Complete end-to-end execution of test procedures using formal

test configurations and accredited scenarios, simulations, and/or
test tools (see Section 6.3.1).
Critical Operational Issue A key operational effectiveness or operational suitability issue

(COI) that must be examined during operational test to determine the
system’s capability to perform its mission.
VVSPT-A2-PDD-013 v4.0 10-4 05/16/2017

COI status is assessed as: Yes – COI/CPR fully met under all test cases (No significant
issues impacting COI/CPR)
Ltd (Limited) – COI/CPR could not be met under all test
cases/conditions; ability to meet COI/CPR is limited (One or
more significant issues impacting COI/CPR)
No – COI/CPR not met (Serious shortfalls affecting COI/CPR)
Endorsement A recommendation for approval or disapproval of a work

product with supporting comments. Endorsement can be
accomplished via email, endorsement letter, or a written
signature on the signature page of the document by the
endorser(s). Note: The TSB endorses work products via TSB
Endorsement Position Papers.
Enterprise Level Capabilities NAS Requirements that may involve more than one system to
implement. Changes may be required in multiple FAA systems,
ATC procedures and avionics systems developed by industry.
Verification and Validation of Enterprise Level capabilities may
require multiple systems to reach specified states of
development, and may be performed by a dedicated Enterprise
Level Capability Test Team.
Field Familiarization Tests conducted at each site by Air Traffic and Technical
Operations personnel to verify that the site is ready to switch
over to the new system.
Initial Operational Capability IOC is the declaration by site personnel that the system is ready
(IOC) for conditional operational use in the NAS and denotes the end
of field familiarization at that site.
Measure of Effectiveness First-level, qualitative decomposition of an operational

(MOE) effectiveness component associated with a COI.
Measure of Performance Quantitative values that characterize MOEs or MOSs. These

(MOP) values are measurable by a test process.
Measure of Suitability (MOS) First-level, qualitative decomposition of an operational

suitability component associated with a COI.
Operational Effectiveness The degree to which a product accomplishes its mission when
used by representative personnel in the expected operational
environment.
VVSPT-A2-PDD-013 v4.0 10-5 05/16/2017

Operational Readiness Operational readiness encompasses operational effectiveness

and operational suitability. [AMS 2.6.1] Later test and
evaluation examines performance and operational readiness
(suitability and effectiveness) in support of decision-makers at
the production, deployment, and in-service decisions. [AMS
4.4.2]
Operational Readiness Date Operational readiness is attained when site operational

personnel are satisfied the solution can support full and
sustained air transportation operations. The milestone occurs
after completion of the joint acceptance and inspection process
when the site official signs the facility log designating the new
solution as the primary means for air transportation operations.
Operational Suitability The degree to which a product intended for field use satisfies its
availability, compatibility, transportability, interoperability,
reliability maintainability, safety, human factors, logistics
supportability, documentation, personnel, and training
requirements.
Prime contractor The private company or organization that has an established

contract with the government to provide a system or service.
The term prime is used to differentiate from subcontractors
whose contract is not directly with the government but rather
with the prime contractor.
Procedure Subordinate to a process. A set of activities or steps taken to

achieve a given purpose. Any specific combination of
machines, tools, methods, materials, and/or people employed to
develop a work product. Could be used by multiple persons in
one group either separately, interleaved, recursively, or
concurrently. Some activities transform inputs into outputs
needed for other activities.
Red-line changes Changes made to test procedures during dry run conduct.
Run-for-Record The execution of test cases/steps in order to collect official data

relevant to V&V of requirements.
Testbed A stand-alone or distributed environment created for testing

purposes. It may consist of a combination of specialized
hardware, software, real or simulated environmental conditions,
and may be a component of a larger test environment.
Test Activity A category of test hierarchy between Test Phase and Test Case
Group, with an identifiable title and reporting requirements.
VVSPT-A2-PDD-013 v4.0 10-6 05/16/2017

Test Capability A resource, or method, that is used to verify or validate

requirements. This may include testbed(s), simulations,
simulated interfaces, modeling, scripts, test equipment. Test
capabilities need to be accredited before they are used in test
activities.
Test Case a subset of test procedures that specify a) a set of product

requirements to be verified and validated; b) the resources
required to execute the test case; c) the specific steps that must
be taken to perform the test case. Identified by paragraph
number in a Test Procedures document.
Test Case Group A collection of test cases linked by a common purpose, such as
to verify a specific set of product requirements. A group may
consist of a similar set of test cases, performed under a range of
differing conditions. Identified by paragraph number in a Test
Procedures document.
Test Environment The hardware, software, interface, and support configuration

that are necessary to meet the test objectives (e.g., testbeds,
simulators, test instrumentation, test tools)
Test event The conduct of a Test Activity, or portion of a Test Activity,

where data is collected for the record.
Test Phase Highest level subdivision in a test program (e.g., T&E Program
Planning, DT, OT)
Test processes A general term for methods that may be used or procedures
which may be followed in order to conduct test.
Test Procedure A guiding document that details the steps to be taken
Test program All of the identified activities that are required to perform V&V
of a NAS System or a NAS Enterprise Level Capability.
Test Scenario Simulations or scripted input developed to exercise

combinations of operational factors that influence system
performance.
Test Steps A subset of a test case that directs test personnel to perform
actions, and document responses.
VVSPT-A2-PDD-013 v4.0 10-7 05/16/2017

Test Tools Automated HW/SW support equipment that allows the re-
verification of existing baseline performance and the T&E of
new functions/fixes. The thoroughness of the tools and the
amount of automation of the tools directly affect the level of
verification that can be done in a reasonable timeframe.
VVSPT-A2-PDD-013 v4.0 10-8 05/16/2017

APPENDIX A. TEST PLANNING PROCESS CONFORMANCE CHECKLIST

(See T&E Handbook, Section 1.7, T&E Standards Conformance, regarding checklists).
Test Program:
T&E Team(s):
T&E First Line Supervisor(s):
DT Test Director:
OT Test Director:
Version 1: ____ Initial ____ Revision ____ Checklist Complete
Review Signatures: Date:
_______________________________ ____________
T&E First Line Supervisor(s)
_______________________________ ____________
DT Test Director
_______________________________ ____________
OT Test Director
_______________________________ ____________
TSB Representative
Note 1: Initial – Checklist will be filled out upon notification/assignment to a new T&E project, w/anticipated action tailoring included; Dated and signed
Revision – Checklist will be updated each time a changed, newly anticipated or actual but not previously anticipated tailored action is required;
Dated and signed for each revision
Checklist Complete – Checklist will be finalized when the associated program phase is complete and all actions have been addressed; Dated and signed
VVSPT-A2-PDD-013 v4.0 A-1 05/16/2017

Test Program Name: Version Date 6 :

Status/
Action T&E HB Tailoring Change(s) Target Completion
Action TSB Role Comments/
ID Ref. Sec. w/ Justification (if used) 2 Date 3 Date 4
Artifacts 5
Participate in early product engineering and
TP1 Monitor 5.1(a)
implementation reviews
Provide test strategy briefings to the Program

TP2 Monitor 5.1(g)(2)
Office
Review the PRD for:

- Requirement completeness 5.1(b)
- Definition of NAS operational interfaces 5.1(c)
TP3 - Requirement testability Monitor 5.1(d)
- COI completeness and testability 5.1(e)
- Identification of CPRs 5.1(f)
- Test program structure 5.1(g)
Write the T&E section of the PRD, to include

essential FAA and contractor tests, and follow 5.1(g)(3),
TP4 Monitor
the review & approval cycle for submission into Fig. D-1
the PRD
Form (with the Program Office) and participate 5.3,

TP5 in the Integrated Test Team (ITT), beginning Monitor 5.2,
with ISPD and TEMP planning 7.3.2
VVSPT-A2-PDD-013 v4.0 A-2 05/16/2017


Status/
Artifacts 5
Write the T&E section of the ISPD, to include
test strategy, test schedule and test resource 5.3,
TP6 Monitor
requirements, and follow the review & approval Fig. D-2
cycle for submission into the ISPD
Review, witness and document any early

TP7 evaluations (e.g., prototype tests, user demos, Monitor 4
OCDs, OCTs)
Write the preliminary, initial and final TEMP, 5.2.3

utilizing the Test Design process:
- Identify and assess requirements
5.2.3.1
- Identify and decompose COIs into MOEs,
5.2.3.2
MOSs and MOPs
- Identify CPRs
5.2.3.3
TP8 - Define DT and OT test activities Review
5.2.3.4
- Develop the TEMP VRTM
5.2.3.5
- Identify Test Capability Requirements
5.2.3.6
- Define test schedule and costs
5.2.3.7
- Update the APB (if required)
5.2.3.8
- Document the test design in the TEMP
5.2.3.9
VVSPT-A2-PDD-013 v4.0 A-3 05/16/2017


Status/
Artifacts 5
Provide a briefing to the Program Manager for
concurrence on the T&E approach prior to
TP9 submitting the preliminary, initial and final Monitor 5.2
TEMP for Program Manager review and
approval
Tailor and approve the Test Section of the In-

TP10 Monitor 9.2
Service Review (ISR) Checklist
Identify if there is a need for test NCPs in the

TEMP. If so, provide the required NCP/case file
TP11 Monitor 9.3
documentation in accordance with FAA Orders
1800.66 and 6032.1B
VVSPT-A2-PDD-013 v4.0 A-4 05/16/2017


Status/
Artifacts 5
If required, develop test program accreditation
requirements (prime contractor for DT, OT Test
Director for OT) and document in the TEMP:
- For DT, review the prime contractor’s
Accreditation Plan(s) and follow the DT
Accreditation Plan review & approval cycle
prior to providing a final approval
recommendation to the CO and; For OT, develop
an Accreditation Plan(s) and follow the OT
Accreditation Plan review & approval cycle
9.1,
- Witness prime contractor accreditation
activities for DT, and/or conduct accreditation 9.1.3,
TP12 Review
activities for OT; Provide analysis and results for Fig. D-9,
approval no later than 30 days prior to testing
using the simulation Fig. D-16
- For DT, review the prime contractor’s

Accreditation Report and follow the DT
Accreditation Report review & approval cycle
prior to providing a final approval
recommendation to the CO. For OT, write an
Accreditation Report and follow the OT
Accreditation Report review & approval cycle
- Archive the Accreditation Plan(s), analysis, and
Accreditation Report(s)
Follow the review & approval cycle for all

5.2,
TP13 versions of the TEMP that require approval Review
Fig. D-3
signatures
VVSPT-A2-PDD-013 v4.0 A-5 05/16/2017


Status/
Artifacts 5
Update the TEMP at major milestones, when the
ISPD, or the program baseline, has been 5.2,
TP14 Review
significantly modified, or when the program is 5.2.2
significantly changed or restructured
Note 2: Tailoring – Identify any required change(s) to the Action, and provide justification for each specific change (Sec. 1.6 of this T&E Handbook can be used to provide
justifications related to tailoring the T&E process); Leave blank if no tailoring is required
Note 3: Target Date – Provide at least the month and year of the anticipated completion of the Action; Revise only when a major program schedule change has occurred; Leave
blank if the Action already occurred prior to the completion of the Initial version of the checklist
Note 4: Completion Date – Provide the actual completion date of the Action; If the Action was completed prior to the completion of the Initial version of the checklist, provide at
least the year of the Action completion or leave blank if unknown; If the Action is completed after the completion of the Initial version of the checklist, provide the full
date of the Action completion (month/day/year); For Actions that have multiple items to complete, provide completion dates for each item and explain in the
Status/Comments column
Note 5: Status/Comments – Provide any supporting information for the Action; if the Action was completed prior to the completion of the Initial version of the checklist and the
actual date of completion is unknown, enter the words “Action occurred prior to the implementation of this T&E process” in this column
Note 6: Each signed and dated version (Initial, Revisions, and Checklist Complete) of this checklist will be maintained by the DT Test Director in either a hard copy or in an
electronic copy with a scanned signature page. Each “working copy” of this checklist can be maintained electronically on the DT Test Director’s PC. All signed and
dated versions of this checklist will be maintained for a period of two (2) years beyond the completion of the OT Checklist and in accordance with the ANG-E Document
Management and Control (DMC) Process Description Document (PDD)
VVSPT-A2-PDD-013 v4.0 A-6 05/16/2017

APPENDIX B. DT PROCESS CONFORMANCE CHECKLIST

Test Program:
T&E Team(s):
DT Test Director:
_______________________________ ____________
_______________________________ ____________
DT Test Director
_______________________________ ____________
TSB Representative
Revision – Checklist will be updated each time a changed, newly anticipated or actual but not previously anticipated tailored action is required; Dated and signed for each
revision
VVSPT-A2-PDD-013 v4.0 B-1 05/16/2017


Status/
Artifacts 5
Support development of the FAA System
DT1 Specification, and ensure that all test-related Monitor 5.4
inputs are peer-reviewed prior to submission
Support development of the SIR proposal and

SOW/PWS/SOO requirements, ensuring that
DT2 adequate test requirements and test activities are Monitor 5.5,
included, and that these inputs are peer-reviewed 6.1.3.1
prior to submission
DT3 Review/evaluate the contractor proposals Monitor 5.6
Coordinate with the CO and COR to provide

DT4 N/A 6.1.3
test-related contract support
After confirming that they have undergone

internal contractor peer reviews, review and 6.1.3.2,
DT5 comment on contractor specifications (A-Level Monitor
& B-Level) and other test-related CDRL 6.1.3.3
documents
Provide status and action plans for ISR Checklist

DT6 Monitor 9.2
test-related items
Participate in and provide test expertise to

DT7 system engineering, hardware, and software Monitor 6.1.4
meetings and reviews
VVSPT-A2-PDD-013 v4.0 B-2 05/16/2017


Status/
Artifacts 5
Review the CMTP (including the DT VRTM) 6.1.3.2,

and follow the review & approval cycle prior to 6.2.2,
DT8 Review
providing an approval recommendation to the 6.2.3,
CO Fig. D-5
Participate in TWG meetings with the Contractor

DT9 Test Manager for DT test program planning and N/A 6.2
issues resolution
Determine the verification method for any

DT10 Monitor 6.2.1.1
COTS/NDI hardware, software, and firmware
6.2.1.10,
DT11 Witness accreditation of DT test capabilities N/A
9.1
Participate in system training from the contractor

DT12 and T&E process training as needed, and Monitor 6.2.1.12
maintain training records as required
Review the contractor’s DT Test Plan(s) and

follow the review & approval cycle prior to 6.2.5,
DT13 Review
providing an approval recommendation to the Fig. D-6
CO
Review the contractor’s DT Test Procedures and

follow the review & approval cycle prior to 6.2.6,
DT14 Monitor
CO
VVSPT-A2-PDD-013 v4.0 B-3 05/16/2017


Status/
Artifacts 5
Ensure that all DT entrance criteria have been
DT15 Monitor 6.2.7
met prior to entering formal DT
Witness DT dry run testing prior to each formal 6.3,

DT16 Monitor
DT test activity 6.3.1
Participate in a contractor-conducted TRR prior 6.3,

DT17 Monitor
to each formal DT test activity 6.3.2
Participate in Pre-Test Briefings prior to each

DT18 test, and review and approve any planned test Monitor 6.3.3
deviations
DT19 Witness DT Software Testing Monitor 6.2.1.2
DT20 Witness DT Hardware Testing Monitor 6.2.1.3
DT21 Witness Factory Acceptance Testing (FAT) Monitor 6.2.1.4
DT22 Monitor Functional Qualification Testing (FQT) Monitor 6.2.1.5
Witness DT Installation and Integration (I&I)

Testing
Complete a System Test Entrance Checklist to

assess DT issues and test readiness prior to
DT24 entering DT System Testing, and submit it to the Review 6.2.1.7
T&E First Line Supervisor for endorsement and
the TSB for review and comment
VVSPT-A2-PDD-013 v4.0 B-4 05/16/2017


Status/
Artifacts 5
DT25 Witness DT System Testing Monitor 6.2.1.7
DT26 Witness Production Acceptance Testing (PAT) Monitor 6.2.1.8
Witness Site Acceptance Testing (SAT)

(Coordinate with each facility prior to SAT)
During all formal DT test execution:

- Sign off/initial all test procedure
changes/redlines/deviations
DT28 - Ensure that anomalies are properly documented Monitor 6.3.4

in the contractor’s test log
- Sign off on the test log and obtain copies of the
test log and as-run test procedures at the
completion of testing
Participate in DT Post-Test Reviews after each

DT29 Monitor 6.3.5
test, and review and confirm results
Prepare, distribute to the DT test team, and enter

DT30 into the test status database a Test Status Report Monitor 6.3.6
for each dry run and formal test run
VVSPT-A2-PDD-013 v4.0 B-5 05/16/2017


Status/
Artifacts 5
Review the contractor’s PTR database,
including:
- Review and approve the PTR database design
DT31 Monitor 6.3.7
- Review and concur on PTR priorities
- Review and concur on PTR corrective actions
and proposed regression testing
Approve the extent of and witness any DT

DT32 Monitor 6.3.8
Regression Testing
Review the contractor’s DT Test Report and

follow the review & approval cycle prior to 6.4,
DT33 Review
CO
Support the FCA, PCA, and GA processes with

test expertise and verification
Ensure that all DT exit criteria have been met

DT35 Monitor 6.2.8
prior to exiting DT
VVSPT-A2-PDD-013 v4.0 B-6 05/16/2017

Note 6: Each signed and dated version (Initial, Revisions, and Checklist Complete) of this checklist will be maintained by the DT Test Director in either a hard copy or in an
electronic copy with a scanned signature page. Each “working copy” of this checklist can be maintained electronically on the DT Test Director’s PC. All signed and
VVSPT-A2-PDD-013 v4.0 B-7 05/16/2017

Intentionally Blank
VVSPT-A2-PDD-013 v4.0 B-8 05/16/2017

APPENDIX C. OT PROCESS CONFORMANCE CHECKLIST

Test Program:
T&E Team(s):
OT Test Director:
_______________________________ ____________
_______________________________ ____________
OT Test Director
_______________________________ ____________
TSB Representative
Revision – Checklist will be updated each time a changed, newly anticipated or actual but not previously anticipated tailored action is required; Dated and signed for each
revision
VVSPT-A2-PDD-013 v4.0 C-1 05/16/2017


Status/
Artifacts 5
Work with the ITT to define personnel and
OT1 resource requirements for OT, including Monitor 7.3.1
assigning test leads
Support development of the FAA System

OT2 Monitor 5.4
Specification from an operational perspective
Support development of the SIR proposal and

SOW/PWS/SOO requirements, ensuring that 5.5,
OT3 adequate OT support requirements are included, Monitor
and that these inputs are peer-reviewed prior to 7.3.3
submission
Evaluate contractor proposals from the OT

OT4 Monitor 5.6
perspective
Provide OT test-related contract support,

including inputs to the SOW/PWS/SOO and 5.5.2,
OT5 N/A
participation in specification, CDRL, and 7.3.3
program reviews
Provide status and action plans for ISR Checklist

OT6 Monitor 9.2
test-related items
Participate in system training from the contractor

OT7 and T&E process training as needed, and Monitor 7.3.4
maintain training records as required
Coordinate early during the T&E Planning and

OT8 Monitor 7.3.5
Support phase with OT field participants
VVSPT-A2-PDD-013 v4.0 C-2 05/16/2017


Status/
Artifacts 5
Plan and perform accreditation of OT test 7.3.6,

OT9 Monitor
capabilities 9.1
Participate in DT TWG meetings with the

Contractor Test Manager and DT Test Director
OT10 Monitor 6.2
for test program planning and resolution of
issues that may affect OT
Prior to OT, generate DRs for all issues

discovered that impact operational requirements
OT11 Monitor 7.6.4
but are not being addressed and need to be
evaluated during OT
Develop the OT Test Plan (including the OT 7.4.1,

OT12 VRTM) and follow the review & approval cycle Review 7.4.2,
prior to final distribution Fig. D-10
Meet with the ITT and ensure that all OT 7.4.3,

OT13 entrance criteria have been met prior to entering Monitor
the OT phase 7.6
VVSPT-A2-PDD-013 v4.0 C-3 05/16/2017


Status/
Artifacts 5
Collect, analyze, and report (using the OT IAR)
at major milestones on system performance,
requirements, issues, and risks having an
operational impact during system development,
DT, and OT 5.2.3.10,
OT14 - Generate optional IARs (minimally at required Review 7.7.1,
milestones) and follow the review & approval
Fig. D-12
cycle prior to distribution
- Electronically distribute completed IARs to the
respective T&E Senior Manager, Program
Manager, T&E First-Line Supervisor, and TSB
Generate the OT Test Procedures, including:

- Develop the draft OT test procedures 7.5,
(including checkout and debugging) and follow
OT15 Review 7.5.1,
the review & approval cycle
Fig. D-11
- Dry run the OT Test Procedures (dry runs
should include OT field participants)
Maintain an OT Test Procedures Status Matrix

during OT test procedure development, and
OT16 generate a Test Status Report at the conclusion Monitor 7.5.3
of each test activity (checkout, debug, and dry
run) and provide it to the ITT
Develop OT user-evaluation questionnaires and

OT17 submit them to the appropriate union(s) for Monitor 7.5.2
approval prior to use during formal OT
VVSPT-A2-PDD-013 v4.0 C-4 05/16/2017


Status/
Artifacts 5
Coordinate all required logistical activities for
OT18 N/A 7.6.1
OT prior to the TRR
Conduct the OT TRR prior to the start of formal

OT19 Monitor 7.6.2
OT activities
Conduct an OT Pre-Test Briefing prior to the

OT20 N/A 7.6.3.1
start of each OT test
During all formal OT test executions:

- Maintain a test log
7.6.3.2,
OT21 - Collect completed questionnaires (if used) Monitor
immediately following the test execution 7.6.4
- Generate DRs for all issues/anomalies
encountered during the testing
Conduct an OT Post-Test Review at the

OT22 N/A 7.6.3.3
conclusion of each formal OT test
Generate a Test Status Report at the conclusion

OT23 Monitor 7.6.3.3
of each formal OT test
Conduct DR reviews on a regular basis and

OT24 N/A 7.6.4
maintain the DR database
Generate an OT Problem Traceability Matrix to

OT25 identify the COIs, MOEs, MOSs, and CPRs Monitor 7.6.5
impacted by the open DRs and PTRs
VVSPT-A2-PDD-013 v4.0 C-5 05/16/2017


Status/
Artifacts 5
Conduct an OT Caucus after the completion of
formal OT testing, including:
- Review the OT Problem Traceability Matrix to
assess the operational impact of open DRs and
OT26 PTRs Monitor 7.6.5
- Generate an OT Caucus Summary Report to
document the status of each COI, MOE, MOS,
and CPR, and to support the resolution of critical
PTRs
Generate an OT Quicklook Test Report,

including:
- Generate the draft report and follow the review 1.8(e),
& approval cycle prior to distribution
1.9,
OT27 - Electronically distribute the approved OT Review
Quicklook Test Report to the respective T&E 7.7.2,
Senior Manager, Program Manager, T&E First- Fig. D-13
Line Supervisor, Technical Center Director, and
TSB within 15 calendar days from test
completion
Conduct OT Regression Testing to verify the

OT28 Monitor 7.6.6
integrity of solutions to DRs/PTRs
Ensure that all OT exit criteria have been met

OT29 Monitor 7.4.4
prior to exiting OT
VVSPT-A2-PDD-013 v4.0 C-6 05/16/2017


Status/
Artifacts 5
Generate an OT Final Test Report, including:
- For large or high-risk programs, conduct an OT
Final Test Report Out-Brief to the ITT prior to
delivering the draft report for review
- Follow the OT Final Test Report review and
7.7.3,
OT30 approval cycle prior to final distribution, Review
including an OT Final Test Report Out-Brief to Fig. D-14
the Technical Center Director, if required.
- Deliver the approved OT Final Test Report
within 60 calendar days from test completion to
the TSB, and the Program Manager, to support
the ISD process
Provide an OT Final Test Report Out-Brief, as

required, to the ANG Assistant Administrator for
OT31 NextGen, ANG-1, prior to the ISD milestone Monitor 7.7.3
(Technical Center Director function)
If required, provide test support to the In-Service

OT32 Management (ISM) organization during the Monitor 7.8
transitional period from OT to ISM
If required, develop a Field Familiarization (FF) 7.9,

Support Plan and provide it to the TSB for
OT33 Review 7.9.1
review and comment prior to final delivery to the
Program Office and site representatives Fig. D-15
If required, develop and dry run FF test 7.9.1,

OT34 Monitor
procedures 7.9.2
VVSPT-A2-PDD-013 v4.0 C-7 05/16/2017


Status/
Artifacts 5
If required, participate in FF testing, including
OT35 Monitor 7.9.3
all activities associated with formal testing
If required, generate and distribute to the

Program Office, TSB, and site representatives a
OT36 final FF Report for each site to document the Review 7.9.4
activities and summarize any issues encountered
during FF testing
Note 6: Each signed and dated version (Initial, Revisions, and Checklist Complete) of this checklist will be maintained by the OT Test Director in either a hard copy or in an
electronic copy with a scanned signature page. Each “working copy” of this checklist can be maintained electronically on the OT Test Director’s PC. All signed and
VVSPT-A2-PDD-013 v4.0 C-8 05/16/2017

APPENDIX D. T&E WORK PRODUCTS AND REVIEW PROCESS CYCLES

The following table identifies the major work products associated with each phase of the T&E
process, along with a reference to their associated review and approval cycle flow diagrams
which are contained in this appendix.
T ABLE D-1. T&E W ORK P RODUCTS

TSB
TSB Final TSB Flow
Early
Test Phase Work Product Review Endorsement Diagram
Review
Required 3 Required 4 Number
Cycle 2
Program Requirements Document (PRD)
X X D-1
T&E Section
T&E Implementation Strategy and Planning
X X X D-2
Program Document (ISPD) T&E Section
Planning Test and Evaluation Master Plan (TEMP)
X X X D-3
(Requires WJHTC Director Approval)
T&E Inputs to: Specification; SIR, SOW D-4
Contractor Master Test Plan (CMTP) X D-5
DT Test Plan X D-6
Development TSB’s role is to monitor
DT Test Procedures D-7
Test (DT) the test team activity
DT Test Report X D-8
DT Accreditation Plan/
X D-9
DT Accreditation Report
OT Test Plan X X X D-10
OT Test Procedures TSB reviews as needed D-11
OT Interim Assessment Report (IAR) X D-12
Operational OT Quicklook Test Report X D-13
Test (OT)
OT Final Test Report (Requires WJHTC
X X X D-14
Director Approval)
OT Test Capability Accreditation X X X D-15
Field Familiarization Support Plan X X D-16
2
An early review is recommended for the purpose of identifying major work product deficiencies. This is typically
accomplished by the project’s TSB POC and one other TSB member. It is recommended that the TSB Early Review
start in parallel with the Peer Review but be complete prior to Comment Workoff and Tech Edit.
3
The completed review results in a “tabling” meeting in which the TSB determines the level of conformance to
established standards and the TSB’s position, if endorsable. Consult the TSB Work Product Review and
Endorsement Guide regarding the number of reviewers to include commensurate with the priority level of the
program.
4
Endorsement is defined as a recommendation for approval or disapproval of a work product with supporting
comments. Endorsement can be accomplished via email, endorsement letter, or a written signature on the signature
page of the document by the endorser(s).
VVSPT-A2-PDD-013 v4.0 D-1 05/16/2017

The following pages contain diagrams of the process flow cycles for reviewing and approving
the test work products identified in the Handbook. The processes in the diagrams may be
tailored to meet program needs. A process may be restarted if major revisions are required based
on reviews or major program changes.
The following terminology is presented below for use in understanding the flow diagrams:
o Signature: On specified documents, signature(s) are required by the responsible author(s)
of the document (normally the Test Director(s)) on the signature page.
o Review: A Review is defined as an assessment of a draft or final draft document to
provide comments and input. The outcome of a review results in the delivery of a revised
draft or a final document. A Peer Review is a structured type of review which involves a
methodical examination of a completed draft document. Peer Reviews are conducted in
accordance with the Peer Review PDD to ensure the quality of the work product. Peer
Reviews are conducted by unbiased subject matter experts that have independence from
the development and approval of the document. Peer reviewers utilize knowledge,
experience, established standards and known best practices to provide editorial and
technical comments.
o Endorsement: Endorsement is defined as a recommendation for approval or disapproval
of a work product with supporting comments. Endorsement can be accomplished via
email, endorsement letter, or a written signature on the signature page of the document by
the endorser(s). Note: The TSB endorses work products via TSB Endorsement Position
Papers.
o Approval: For DT work products, approval is defined by the provision of an approval
recommendation from the DT Test Director to the Contracting Officer after all
appropriate Government authorities have reviewed and endorsed the document. For OT
work products, approval is defined by the written signature on the signature page of the
document by the designated authority after his or her review and approval of the
document.
The following symbology definitions are presented below for use in understanding the connector
lines between the blocks in the flow diagrams:
o Solid line: Mandatory path for activity to follow
o Dashed line: Recommended path for activity to follow
VVSPT-A2-PDD-013 v4.0 D-2 05/16/2017

F IGURE D-1. PRD (T&E S ECTION ) R EVIEW & A PPROVAL C Y CLE
VVSPT-A2-PDD-013 v4.0 D-3 05/16/2017

F IGURE D-2. ISPD (T&E S ECTION ) R EVIEW & A PPROVAL C Y CLE

VVSPT-A2-PDD-013 v4.0 D-4 05/16/2017
F IGURE D-3. P TEMP, I TEMP & F TEMP R EVIEW & A PPROVAL C YCLE
VVSPT-A2-PDD-013 v4.0 D-5 05/16/2017

F IGURE D-4. T&E I NPUT R EVI EW C YCLE FOR FAA S YSTEM

S PECIFICATION , SIR P ROPOSAL R EQUIREMENTS AND S TATEMENTS OF
W ORK
VVSPT-A2-PDD-013 v4.0 D-6 05/16/2017

F IGURE D-5. CMTP R EVI EW & A PPROVAL C YCLE
VVSPT-A2-PDD-013 v4.0 D-7 05/16/2017

F IGURE D-6. DT T EST P LAN R EVIEW & A PPROVAL C YCLE
VVSPT-A2-PDD-013 v4.0 D-8 05/16/2017

F IGURE D-7. DT T EST P ROCEDURES R EVI EW & A PPROVAL C YCLE
VVSPT-A2-PDD-013 v4.0 D-9 05/16/2017

F IGURE D-8. DT T EST R EPORT R EVIEW & A PPROVAL C YCLE
VVSPT-A2-PDD-013 v4.0 D-10 05/16/2017

F IGURE D-9. DT A CCREDI TATION P LAN /R EPORT

R EVIEW & A PPROVAL C Y CLE
VVSPT-A2-PDD-013 v4.0 D-11 05/16/2017

F IGURE D-10. OT T EST P LAN R EVIEW & A PPROVAL C YCLE
VVSPT-A2-PDD-013 v4.0 D-12 05/16/2017

F IGURE D-11. OT T EST P ROCEDURES AND T EST C APABILITY

P ROCEDURES R EVIEW & A PPROVAL C YCLE
VVSPT-A2-PDD-013 v4.0 D-13 05/16/2017

F IGURE D-12. OT I NTERIM A SSESSMENT R EPORT

VVSPT-A2-PDD-013 v4.0 D-14 05/16/2017

F IGURE D-13. OT Q UICK LOOK T EST R EPORT

VVSPT-A2-PDD-013 v4.0 D-15 05/16/2017

F IGURE D-14. OT F INAL T EST R EPORT

R EVIEW &A PPROVAL C YCLE
VVSPT-A2-PDD-013 v4.0 D-16 05/16/2017

F IGURE D-15. F IELD FAMILI ARI ZATI ON S UPPORT P LAN

VVSPT-A2-PDD-013 v4.0 D-17 05/16/2017

F IGURE D-16. OT A CCREDI TATION P LAN /R EPORT

VVSPT-A2-PDD-013 v4.0 D-18 05/16/2017


FAA W J. H T C T E H: Illiam Ughes Echnical Enter Est and Valuation Andbook

Uploaded by

Copyright:

Available Formats

FAA W J. H T C T E H: Illiam Ughes Echnical Enter Est and Valuation Andbook

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

FAA W J. H T C T E H: Illiam Ughes Echnical Enter Est and Valuation Andbook

Uploaded by

Copyright:

Available Formats

FAA W IL L I AM J.

FAA WILLIAM J. HUGHES TECHNICAL CENTER

T HIS D OC UM E NT B E C OME S U NC ONTR O LLE D O NC E PR INTE D

VVSPT-A2-PDD-013 v4.0 ii 05/16/2017

DOCUMENT VERSION CONTROL

VERSION DESCRIPTION OF CHANGE DATE

Draft 1.0 Initial delivery to management 12/20/2006

Final 1.0 Initial delivery for T&E process implementation 08/21/2008

Incorporated comments against Draft V1.2 as detailed in

Final 3.0 Release Version 3.0 09/24/2013

3.1 Incorporate comments against Final V3.0 2/24/2016

Incorporated additional items from V&V Comment

3.3 Incorporated comments from TSB Review 6/10/2016

3.4 Incorporating comments from External Peer Review 7/25/2016

3.5 Proofed version for Formal Branch Review 2/8/2017

Incorporated Branch Review comments. Proofed version

4.0 Final for Division Signature 5/16/2017

VVSPT-A2-PDD-013 v4.0 iv 05/16/2017

VVSPT-A2-PDD-013 v4.0 v 05/16/2017

6.1.1DT Activities ..................................................................................................................................6-1

VVSPT-A2-PDD-013 v4.0 vi 05/16/2017

7.3.1 OT Test Team Organization...........................................................................................................7-3

VVSPT-A2-PDD-013 v4.0 vii 05/16/2017

9.2 IN-SERVICE REVIEW CHECKLIST AND DEPLOYMENT DECISION PLANNING ..................................................9-5

VVSPT-A2-PDD-013 v4.0 viii 05/16/2017

VVSPT-A2-PDD-013 v4.0 1-1 05/16/2017

F IGURE 1-1. T&E A PPLICATION T O T HE V-M O DEL

VVSPT-A2-PDD-013 v4.0 1-2 05/16/2017

1.4 R OLES AND R ESPONSIBILITI ES

1.5 F UNDAMENTAL P RACTICES FOR Q UALITY T&E

VVSPT-A2-PDD-013 v4.0 1-4 05/16/2017

VVSPT-A2-PDD-013 v4.0 1-5 05/16/2017

1.6.2 TAILORING GUIDELINES

1.7 T&E S TANDARDS C ONFORMANCE P ROCESS

1.8 T&E Q UALITY M ANAGEMENT S YSTEM P ROCESSES

VVSPT-A2-PDD-013 v4.0 1-7 05/16/2017

VVSPT-A2-PDD-013 v4.0 1-9 05/16/2017

VVSPT-A2-PDD-013 v4.0 1-10 05/16/2017

2 RELATED DOCUMENTS AND REFERENCES

VVSPT-A2-PDD-013 v4.0 2-1 05/16/2017

VVSPT-A2-PDD-013 v4.0 2-2 05/16/2017

3 T&E SUPPORT ACROSS THE AMS LIFECYCLE

VVSPT-A2-PDD-013 v4.0 3-2 05/16/2017

F IGURE 3-2. T&E P HASES AND R EQ UIREMENTS

VVSPT-A2-PDD-013 v4.0 3-4 05/16/2017

4 T&E SUPPORT TO SERVICE ANALYSIS & STRATEGIC

VVSPT-A2-PDD-013 v4.0 4-1 05/16/2017

VVSPT-A2-PDD-013 v4.0 4-2 05/16/2017

5 T&E SUPPORT TO INVESTMENT ANALYSIS

VVSPT-A2-PDD-013 v4.0 5-1 05/16/2017

5.2 T EST AND E VALUATION M ASTER P LAN D EVELOPMENT

VVSPT-A2-PDD-013 v4.0 5-2 05/16/2017

F IGURE 5-1. T EST C ATEGORY H IERARCHY AND D OCUMENTATION

VVSPT-A2-PDD-013 v4.0 5-3 05/16/2017

VVSPT-A2-PDD-013 v4.0 5-4 05/16/2017

j) Specify equipment and resource requirements to support testing

5.2.2 TEST MANAGEMENT USING THE TEMP

VVSPT-A2-PDD-013 v4.0 5-5 05/16/2017

F IGURE 5-2. TEMP R ELATIONAL D I AGRAM