Get Advanced Testing of Systems-of-Systems, Volume 2 - Practical Aspects 1st Edition Bernard Homes Free All Chapters

Full download test bank at ebookmeta.
com
Advanced Testing of Systems-of-Systems, Volume 2 -

Practical Aspects 1st Edition Bernard Homes
For dowload this book click LINK or Button below
https://ebookmeta.com/product/advanced-testing-of-
systems-of-systems-volume-2-practical-aspects-1st-
edition-bernard-homes/
OR CLICK BUTTON
DOWLOAD EBOOK
Download More ebooks from https://ebookmeta.com

More products digital (pdf, epub, mobi) instant
download maybe you interests ...
Advanced Testing of Systems-of-Systems, Volume 1 -

Theoretical Aspects 1st Edition Bernard Homes
https://ebookmeta.com/product/advanced-testing-of-systems-of-
systems-volume-1-theoretical-aspects-1st-edition-bernard-homes/
Advanced Computing and Systems for Security Volume 13

1st Edition Rituparna Chaki
https://ebookmeta.com/product/advanced-computing-and-systems-for-
security-volume-13-1st-edition-rituparna-chaki/
Practical Aspects of Vaccine Development The Practical

Aspects 1st Edition Parag Kolhe (Editor)
https://ebookmeta.com/product/practical-aspects-of-vaccine-
development-the-practical-aspects-1st-edition-parag-kolhe-editor/
Intelligent Sustainable Systems Selected Papers of

WorldS4 2021 Volume 2 1st Edition Atulya K. Nagar
https://ebookmeta.com/product/intelligent-sustainable-systems-
selected-papers-of-worlds4-2021-volume-2-1st-edition-atulya-k-
nagar/
Advanced Computing and Systems for Security Volume 14
Lecture Notes in Networks and Systems 242 1st Edition
Rituparna Chaki (Editor)
https://ebookmeta.com/product/advanced-computing-and-systems-for-
security-volume-14-lecture-notes-in-networks-and-systems-242-1st-
edition-rituparna-chaki-editor/
Legal Aspects of Autonomous Systems: A Comparative

Approach 1st Edition Dário Moura Vicente
https://ebookmeta.com/product/legal-aspects-of-autonomous-
systems-a-comparative-approach-1st-edition-dario-moura-vicente/
Radiation Detection Systems: Medical Imaging,

Industrial Testing and Security Applications (Devices,
Circuits, and Systems) 2nd Edition
https://ebookmeta.com/product/radiation-detection-systems-
medical-imaging-industrial-testing-and-security-applications-
devices-circuits-and-systems-2nd-edition/
Applications of Big Data and Artificial Intelligence in

Smart Energy Systems: Volume 2 1st Edition Neelu Nagpal
(Editor)
https://ebookmeta.com/product/applications-of-big-data-and-
artificial-intelligence-in-smart-energy-systems-volume-2-1st-
edition-neelu-nagpal-editor/
Advanced Dynamics Modeling, Duality and Control of

Robotic Systems 1st Edition Edward Y.L. Gu
https://ebookmeta.com/product/advanced-dynamics-modeling-duality-
and-control-of-robotic-systems-1st-edition-edward-y-l-gu/
Advanced Testing of Systems-of-Systems 2
Advanced Testing of
Systems-of-Systems 2
Practical Aspects
Bernard Homès
First published 2022 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc.
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as
permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced,
stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers,
or in the case of reprographic reproduction in accordance with the terms and licenses issued by the
CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the
undermentioned address:
ISTE Ltd John Wiley & Sons, Inc.

27-37 St George’s Road 111 River Street
London SW19 4EU Hoboken, NJ 07030
UK USA
www.iste.co.uk www.wiley.com
© ISTE Ltd 2022

The rights of Bernard Homès to be identified as the author of this work have been asserted by him in
accordance with the Copyright, Designs and Patents Act 1988.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the
author(s), contributor(s) or editor(s) and do not necessarily reflect the views of ISTE Group.
Library of Congress Control Number: 2022944148
British Library Cataloguing-in-Publication Data

A CIP record for this book is available from the British Library
ISBN 978-1-78630-750-7
Contents
Dedication and Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
Chapter 1. Test Project Management . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1. General principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1. Quality of requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2. Completeness of deliveries . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.3. Availability of test environments . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.4. Availability of test data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.5. Compliance of deliveries and schedules . . . . . . . . . . . . . . . . . . . 5
1.1.6. Coordinating and setting up environments . . . . . . . . . . . . . . . . . . 6
1.1.7. Validation of prerequisites – Test Readiness Review (TRR) . . . . . . . . 6
1.1.8. Delivery of datasets (TDS) . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.1.9. Go-NoGo decision – Test Review Board (TRB) . . . . . . . . . . . . . . . 7
1.1.10. Continuous delivery and deployment . . . . . . . . . . . . . . . . . . . . 8
1.2. Tracking test projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3. Risks and systems-of-systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.4. Particularities related to SoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.5. Particularities related to SoS methodologies . . . . . . . . . . . . . . . . . . . 11
1.5.1. Components definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.5.2. Testing and quality assurance activities . . . . . . . . . . . . . . . . . . . . 12
1.6. Particularities related to teams . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Chapter 2. Testing Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.1. Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2. Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.1. Project WBS and planning . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
vi Advanced Testing of Systems-of-Systems 2
2.3. Control of test activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.4. Analyze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.5. Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.6. Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.7. Test execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.8. Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.9. Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.10. Closure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.11. Infrastructure management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.12. Reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.13. Adapting processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.14. RACI matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.15. Automation of processes or tests . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.15.1. Automate or industrialize? . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.15.2. What to automate? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.15.3. Selecting what to automate . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Chapter 3. Continuous Process Improvement . . . . . . . . . . . . . . . . . . . 37

3.1. Modeling improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.1.1. PDCA and IDEAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.1.2. CTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.1.3. SMART . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.2. Why and how to improve? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.3. Improvement methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.3.1. External/internal referential . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.4. Process quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.4.1. Fault seeding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.4.2. Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.4.3. A posteriori . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.4.4. Avoiding introduction of defects . . . . . . . . . . . . . . . . . . . . . . . 47
3.5. Effectiveness of improvement activities . . . . . . . . . . . . . . . . . . . . . . 48
3.6. Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Chapter 4. Test, QA or IV&V Teams. . . . . . . . . . . . . . . . . . . . . . . . . . 51

4.1. Need for a test team . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.2. Characteristics of a good test team . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.3. Ideal test team profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.4. Team evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.4.1. Skills assessment table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.4.2. Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.4.3. Select, hire and retain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.5. Test manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Contents vii
4.5.1. Lead or direct? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4.5.2. Evaluate and measure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.5.3. Recurring questions for test managers . . . . . . . . . . . . . . . . . . . . 62
4.6. Test analyst . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.7. Technical test analyst . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.8. Test automator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.9. Test technician . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.10. Choose our testers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.11. Training, certification or experience?. . . . . . . . . . . . . . . . . . . . . . . 67
4.12. Hire or subcontract? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.12.1. Effective subcontracting . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.13. Organization of multi-level test teams . . . . . . . . . . . . . . . . . . . . . . 68
4.13.1. Compliance, strategy and organization . . . . . . . . . . . . . . . . . . . 69
4.13.2. Unit test teams (UT/CT) . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.13.3. Integration testing team (IT) . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.13.4. System test team (SYST) . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.13.5. Acceptance testing team (UAT) . . . . . . . . . . . . . . . . . . . . . . . 71
4.13.6. Technical test teams (TT) . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.14. Insourcing and outsourcing challenges . . . . . . . . . . . . . . . . . . . . . . 72
4.14.1. Internalization and collocation . . . . . . . . . . . . . . . . . . . . . . . . 72
4.14.2. Near outsourcing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.14.3. Geographically distant outsourcing . . . . . . . . . . . . . . . . . . . . . 74
Chapter 5. Test Workload Estimation . . . . . . . . . . . . . . . . . . . . . . . . 75

5.1. Difficulty to estimate workload. . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.2. Evaluation techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.2.1. Experience-based estimation . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.2.2. Based on function points or TPA . . . . . . . . . . . . . . . . . . . . . . . 77
5.2.3. Requirements scope creep . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.2.4. Estimations based on historical data. . . . . . . . . . . . . . . . . . . . . . 80
5.2.5. WBS or TBS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
5.2.6. Agility, estimation and velocity . . . . . . . . . . . . . . . . . . . . . . . . 81
5.2.7. Retroplanning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
5.2.8. Ratio between developers – testers . . . . . . . . . . . . . . . . . . . . . . 82
5.2.9. Elements influencing the estimate . . . . . . . . . . . . . . . . . . . . . . . 83
5.3. Test workload overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.3.1. Workload assessment verification and validation . . . . . . . . . . . . . . 86
5.3.2. Some values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.4. Understanding the test workload . . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.4.1. Component coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.4.2. Feature coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
5.4.3. Technical coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
viii Advanced Testing of Systems-of-Systems 2
5.4.4. Test campaign preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

5.4.5. Running test campaigns . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.4.6. Defects management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
5.5. Defending our test workload estimate . . . . . . . . . . . . . . . . . . . . . . . 91
5.6. Multi-tasking and crunch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.7. Adapting and tracking the test workload. . . . . . . . . . . . . . . . . . . . . . 92
Chapter 6. Metrics, KPI and Measurements . . . . . . . . . . . . . . . . . . . . 95

6.1. Selecting metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
6.2. Metrics precision. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
6.2.1. Special case of the cost of defaults . . . . . . . . . . . . . . . . . . . . . . 97
6.2.2. Special case of defects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
6.2.3. Accuracy or order of magnitude? . . . . . . . . . . . . . . . . . . . . . . . 98
6.2.4. Measurement frequency . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
6.2.5. Using metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
6.2.6. Continuous improvement of metrics . . . . . . . . . . . . . . . . . . . . . 100
6.3. Product metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
6.3.1. FTR: first time right . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
6.3.2. Coverage rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
6.3.3. Code churn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
6.4. Process metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
6.4.1. Effectiveness metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
6.4.2. Efficiency metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.5. Definition of metrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.5.1. Quality model metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
6.6. Validation of metrics and measures . . . . . . . . . . . . . . . . . . . . . . . . 110
6.6.1. Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
6.6.2. Historical data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6.6.3. Periodic improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
6.7. Measurement reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
6.7.1. Internal test reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.7.2. Reporting to the development team . . . . . . . . . . . . . . . . . . . . . . 114
6.7.3. Reporting to the management . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.7.4. Reporting to the clients or product owners . . . . . . . . . . . . . . . . . . 115
6.7.5. Reporting to the direction and upper management . . . . . . . . . . . . . . 116
Chapter 7. Requirements Management . . . . . . . . . . . . . . . . . . . . . . . 119

7.1. Requirements documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
7.2. Qualities of requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
7.3. Good practices in requirements management . . . . . . . . . . . . . . . . . . . 122
7.3.1. Elicitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
7.3.2. Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Contents ix
7.3.3. Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

7.3.4. Approval and validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
7.3.5. Requirements management . . . . . . . . . . . . . . . . . . . . . . . . . . 124
7.3.6. Requirements and business knowledge management . . . . . . . . . . . . 125
7.3.7. Requirements and project management . . . . . . . . . . . . . . . . . . . . 125
7.4. Levels of requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
7.5. Completeness of requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
7.5.1. Management of TBDs and TBCs . . . . . . . . . . . . . . . . . . . . . . . 126
7.5.2. Avoiding incompleteness. . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
7.6. Requirements and agility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
7.7. Requirements issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
Chapter 8. Defects Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

8.1. Defect management, MOA and MOE . . . . . . . . . . . . . . . . . . . . . . . 129
8.1.1. What is a defect? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
8.1.2. Defects and MOA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
8.1.3. Defects and MOE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
8.2. Defect management workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
8.2.1. Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
8.2.2. Simplify . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
8.3. Triage meetings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
8.3.1. Priority and severity of defects . . . . . . . . . . . . . . . . . . . . . . . . 133
8.3.2. Defect detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
8.3.3. Correction and urgency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
8.3.4. Compliance with processes . . . . . . . . . . . . . . . . . . . . . . . . . . 136
8.4. Specificities of TDDs, ATDDs and BDDs . . . . . . . . . . . . . . . . . . . . 136
8.4.1. TDD: test-driven development. . . . . . . . . . . . . . . . . . . . . . . . . 136
8.4.2. ATDD and BDD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
8.5. Defects reporting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
8.5.1. Defects backlog management . . . . . . . . . . . . . . . . . . . . . . . . . 139
8.6. Other useful reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
8.7. Don’t forget minor defects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
Chapter 9. Configuration Management . . . . . . . . . . . . . . . . . . . . . . . 143

9.1. Why manage configuration? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
9.2. Impact of configuration management . . . . . . . . . . . . . . . . . . . . . . . 144
9.3. Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
9.4. Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
9.5. Organization and standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
9.6. Baseline or stages, branches and merges . . . . . . . . . . . . . . . . . . . . . 147
9.6.1. Stages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
9.6.2. Branches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
x Advanced Testing of Systems-of-Systems 2
9.6.3. Merge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

9.7. Change control board (CCB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
9.8. Delivery frequencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
9.9. Modularity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
9.10. Version management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
9.11. Delivery management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
9.11.1. Preparing for delivery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
9.11.2. Delivery validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
9.12. Configuration management and deployments . . . . . . . . . . . . . . . . . . 155
Chapter 10. Test Tools and Test Automation . . . . . . . . . . . . . . . . . . . 157

10.1. Objectives of test automation . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
10.1.1. Find more defects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
10.1.2. Automating dynamic tests . . . . . . . . . . . . . . . . . . . . . . . . . . 159
10.1.3. Find all regressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
10.1.4. Run test campaigns faster . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
10.2. Test tool challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
10.2.1. Positioning test automation . . . . . . . . . . . . . . . . . . . . . . . . . . 162
10.2.2. Test process analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
10.2.3. Test tool integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
10.2.4. Qualification of tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
10.2.5. Synchronizing test cases . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
10.2.6. Managing test data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
10.2.7. Managing reporting (level of trust in test tools). . . . . . . . . . . . . . . 165
10.3. What to automate? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
10.4. Test tooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
10.4.1. Selecting tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
10.4.2. Computing the return on investment (ROI) . . . . . . . . . . . . . . . . . 169
10.4.3. Avoiding abandonment of tools and automation . . . . . . . . . . . . . . 169
10.5. Automated testing strategies. . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
10.6. Test automation challenge for SoS . . . . . . . . . . . . . . . . . . . . . . . . 171
10.6.1. Mastering test automation . . . . . . . . . . . . . . . . . . . . . . . . . . 171
10.6.2. Preparing test automation . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
10.6.3. Defect injection/fault seeding . . . . . . . . . . . . . . . . . . . . . . . . 173
10.7. Typology of test tools and their specific challenges . . . . . . . . . . . . . . . 174
10.7.1. Static test tools versus dynamic test tools . . . . . . . . . . . . . . . . . . 175
10.7.2. Data-driven testing (DDT) . . . . . . . . . . . . . . . . . . . . . . . . . . 176
10.7.3. Keyword-driven testing (KDT). . . . . . . . . . . . . . . . . . . . . . . . 176
10.7.4. Model-based testing (MBT) . . . . . . . . . . . . . . . . . . . . . . . . . 177
10.8. Automated regression testing . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
10.8.1. Regression tests in builds . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
10.8.2. Regression tests when environments change . . . . . . . . . . . . . . . . 179
Contents xi
10.8.3. Prevalidation regression tests, sanity checks and smoke tests . . . . . . . 179
10.8.4. What to automate? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
10.8.5. Test frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
10.8.6. E2E test cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
10.8.7. Automated test case maintenance or not? . . . . . . . . . . . . . . . . . . 184
10.9. Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
10.9.1. Automated reporting for the test manager . . . . . . . . . . . . . . . . . . 186
Chapter 11. Standards and Regulations . . . . . . . . . . . . . . . . . . . . . . . 187

11.1. Definition of standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
11.2. Usefulness and interest. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
11.3. Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
11.4. Demonstration of compliance – IADT . . . . . . . . . . . . . . . . . . . . . . 190
11.5. Pseudo-standards and good practices . . . . . . . . . . . . . . . . . . . . . . . 191
11.6. Adapting standards to needs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
11.7. Standards and procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
11.8. Internal and external coherence of standards. . . . . . . . . . . . . . . . . . . 192
Chapter 12. Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

12.1. Case study: improvement of an existing complex system . . . . . . . . . . . 195
12.1.1. Context and organization . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
12.1.2. Risks, characteristics and business domains . . . . . . . . . . . . . . . . 198
12.1.3. Approach and environment . . . . . . . . . . . . . . . . . . . . . . . . . . 200
12.1.4. Resources, tools and personnel . . . . . . . . . . . . . . . . . . . . . . . . 210
12.1.5. Deliverables, reporting and documentation . . . . . . . . . . . . . . . . . 212
12.1.6. Planning and progress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
12.1.7. Logistics and campaigns . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
12.1.8. Test techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
12.1.9. Conclusions and return on experience . . . . . . . . . . . . . . . . . . . . 218
Chapter 13. Future Testing Challenges . . . . . . . . . . . . . . . . . . . . . . . 223

13.1. Technical debt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
13.1.1. Origin of the technical debt . . . . . . . . . . . . . . . . . . . . . . . . . . 224
13.1.2. Technical debt elements . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
13.1.3. Measuring technical debt . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
13.1.4. Reducing technical debt. . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
13.2. Systems-of-systems specific challenges . . . . . . . . . . . . . . . . . . . . . 228
13.3. Correct project management. . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
13.4. DevOps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
13.4.1. DevOps ideals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
13.4.2. DevOps-specific challenges . . . . . . . . . . . . . . . . . . . . . . . . . 231
13.5. IoT (Internet of Things) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
xii Advanced Testing of Systems-of-Systems 2
13.6. Big Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233

13.7. Services and microservices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
13.8. Containers, Docker, Kubernetes, etc. . . . . . . . . . . . . . . . . . . . . . . . 235
13.9. Artificial intelligence and machine learning (AI/ML). . . . . . . . . . . . . . 235
13.10. Multi-platforms, mobility and availability . . . . . . . . . . . . . . . . . . . 237
13.11. Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
13.12. Unknown dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
13.13. Automation of tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
13.13.1. Unrealistic expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
13.13.2. Difficult to reach ROI . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
13.13.3. Implementation difficulties . . . . . . . . . . . . . . . . . . . . . . . . . 242
13.13.4. Think about maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . 243
13.13.5. Can you trust your tools and your results? . . . . . . . . . . . . . . . . . 244
13.14. Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
13.15. Blindness or cognitive dissonance . . . . . . . . . . . . . . . . . . . . . . . . 245
13.16. Four truths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
13.16.1. Importance of Individuals . . . . . . . . . . . . . . . . . . . . . . . . . . 247
13.16.2. Quality versus quantity . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
13.16.3. Training, experience and expertise . . . . . . . . . . . . . . . . . . . . . 248
13.16.4. Usefulness of certifications . . . . . . . . . . . . . . . . . . . . . . . . . 248
13.17. Need to anticipate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
13.18. Always reinvent yourself . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
13.19. Last but not least . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
Summary of Volume 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

Dedication and Acknowledgments
Inspired by a dedication from Boris Beizer1, I dedicate these two books

to many very bad projects on software and systems-of-systems development
where I had the opportunity to – for a short time – act as a consultant. These taught
me multiple lessons on difficulties that these books try and identify and led me to
realize the need for this book. Their failure could have been prevented; may they rest
in peace.
I would also like to thank the many managers and colleagues I had the privilege
of meeting during my career. Some, too few, understood that quality is really
everyone’s business. We will lay a modest shroud over the others.
Finally, paraphrasing Isaac Newton, If I was able to reach this level of

knowledge, it is thanks to all the giants that were before me and on the shoulders of
which I could position myself. Among these giants, I would like to mention (in
alphabetical order) James Bach, Boris Beizer, Rex Black, Frederic Brooks, Hans
Buwalda, Ross Collard, Elfriede Dustin, Avner Engel, Tom Gilb, Eliahu Goldratt,
Dorothy Graham, Capers Jones, Paul Jorgensen, Cem Kaner, Brian Marick, Edward
Miller, John Musa, Glenford Myers, Bret Pettichord, Johanna Rothman, Gerald
Weinberg, James Whittaker and Karl Wiegers.
After 15 years in software development, I had the opportunity to focus on

software testing for over 25 years. Specialized in testing process improvements, I
founded and participated in the creation of multiple associations focused on software
testing: AST (Association of Software Tester), ISTQB (International Software
1 Beizer, B. (1990). Software Testing Techniques, 2nd edition. ITP Media.

xiv Advanced Testing of Systems-of-Systems 2
Testing Qualification Board), CFTL (Comité Français des Tests Logiciels, the
French Software Testing committee) and GASQ (Global Association for Software
Quality). I also dedicate these books to you, the reader, so that you can improve your
testing competencies.
Preface
Implementation
In the first part of these two books on systems-of-systems testing, we identified

the impacts of software development cycles, testing strategies and methodologies,
and we saw the benefit of using a quality referential and the importance of test
documentation and reporting. We have identified the impact of test levels and test
techniques, whether we are talking about static techniques or dynamic techniques.
We ended with an approach to test project management that allowed us to identify
that human actor and how their interactions are essential elements that must be
considered.
In this second part of the book on systems-of-systems testing, we will focus on

more practical aspects such as managing test projects, testing processes and how to
improve them continuously. We will see the additional but necessary processes such
as the management of requirements, defects and configurations, and we will also see
a case study allowing us to ask ourselves several useful questions. We will end with
a perilous prediction exercise by listing the challenges that tests will have to face in
the years to come.
August 2022
1
Test Project Management
We do not claim to replace the many contributions of illustrious authors on good

practices in project management. Standards such as PMBOK (PMI 2017) or CMMI
and methodologies such as ITIL and PRINCE2 comprehensively describe the tasks,
best practices and other activities recommended to properly manage projects. We
focus on certain points associated with the testing of software, components, products
and systems within systems-of-systems projects.
At the risk of writing a tautology, the purpose of project management is to

manage projects, that is, to define the tasks and actions necessary to achieve the
objectives of these projects. The purpose, the ultimate objective of the project, takes
precedence over any other aspect, even if the budgetary and time constraints are
significant. To limit the risks associated with systems-of-systems, the quality of the
deliverables is very important and therefore tests (verifications and validations that
the object of the project has been achieved) are necessary.
Project management must ensure that development methodologies are correctly

implemented (see Chapter 2) to avoid inconsistencies. Similarly, project
management must provide all stakeholders with an image of the risks and the
progress of the system-of-systems, its dependencies and the actions to be taken in
the short and medium term, in order to anticipate the potential hazards.
1.1. General principles
Management of test projects, whether on components, products, systems or

systems-of-systems, has a particularity that other projects do not have: they depend
– for their deadlines, scope and level of quality – on other parts of the projects: the
development phases. Requirements are often unstable, information arrives late,
deadlines are shorter because they depend on evolving developments and longer
2 Advanced Testing of Systems-of-Systems 2
deadlines, the scope initially considered increases, the level of quality of input data –
requirements, components to be tested, interfaces – is often of lower quality than
expected and the number of faults or anomalies is greater than anticipated. All of
these are under tighter budgetary and calendar constraints because, even if the
developments take longer than expected, the production launch date is rarely
postponed.
The methodologies offered by ITIL, PRINCE2, CMMI, etc. bring together a set
of good practices that can be adapted – or not – to our system-of-systems project.
CMMI, for example, does not have test-specific elements (only IVV), and it may be
necessary to supplement CMMI with test-specific tasks and actions as offered by
TMM and TMMI.
Let us see the elements specific to software testing projects.
1.1.1. Quality of requirements
Any development translates requirements (needs or business objectives) into a

component, product or system that will implement them. In an Agile environment,
requirements are defined in the form of User Stories, Features or Epics. The
requirements can be described in so-called specification documents (e.g. General
Specifications Document or Detailed Specifications Document). Requirements are
primarily functional – they describe expected functionality – but can be technical or
non-functional. We can classify the requirements according to the quality
characteristics they cover as proposed in Chapter 5 of Volume 1 (Homès 2022a).
Requirements are provided to development teams as well as test teams.

Production teams – design, development, etc. – use these requirements to develop
components, products or systems and may propose or request adaptations of these
requirements. Test teams use requirements to define, analyze and implement, or
even automate, test cases and test scenarios to validate these requirements. These
test teams must absolutely be informed – as soon as possible – of any change in the
requirements to proceed with the modifications of the tests.
The requirements must be SMART, that is:

– Specific: the requirements must be clear, there must be no ambiguity and the
requirements must be simple, consistent and with an appropriate level of detail.
– Measurable: it must be possible, when the component, product or system is
designed, to verify that the requirement has been met. This is directly necessary for
the design of tests and metrics to verify the extent to which requirements are met.
Test Project Management 3
– Achievable: the requirements must be able to be physically demonstrated

under given conditions. If the requirements are not achievable (e.g. the system will
have 100% reliability and 100% availability), the result will be that the component,
product or system will never be accepted or will be cost-prohibitive. Achievable
includes that the requirement can be developed in a specific time frame.
– Realistic: in the context of software development – and testing – is it possible
to achieve the requirement for the component, product or system, taking into
account the constraints in which the project is developed? We add to this aspect the
notion of time: are the requirements achievable in a realistic time?
– Traceable: requirements traceability is the ability to follow a requirement from
its design to its specification, its realization and its implementation to its test, as well
as in the other direction (from the test to the specification). This helps to understand
why a requirement was specified and to ensure that each requirement has been
correctly implemented.
1.1.2. Completeness of deliveries
The completeness of the software, components, products, equipment and systems

delivered for the tests is obviously essential. If the elements delivered are
incomplete, it will be necessary to come back to them to modify and complete them,
which will increase the risk of introducing anomalies.
This aspect of completeness is ambiguous in incremental and iterative

methodologies. On the one hand, it is recommended to deliver small increments, and
on the other hand, losses should be eliminated. Small increments imply partial
releases of functionality, thus generation of “losses” both regarding releases and
testing (e.g. regression testing) – in fact, all the expectations related to these multiple
releases and multiple test runs – to be performed on these components. Any
evolution within the framework of an iteration will lead to a modification in the
functionalities and therefore an evolution compared to the results executed during
the previous iterations.
1.1.3. Availability of test environments
The execution of the tests is carried out in different test environments according
to the test levels envisaged. It will therefore be necessary to ensure the availability
of environments for each level.
The test environment is not limited to a machine on which the software

component is executed. It also includes the settings necessary for the proper
execution of the component, the test data and other applications – in the appropriate
versions – with which the component interacts.
Test environments, as well as their data and the applications they interface with
must be properly synchronized with each other. This implies an up-to-date definition
of the versions of each system making up the system-of-systems and of the
interfaces and messages exchanged between them.
Automating backups and restores of test environments allows testers to self-

manage their environments so that they are not a burden on production systems
management teams.
In DevOps environments, it is recommended to enable automatic creation of

environments to test builds as they are created by developers. As proposed by Kim
et al. (2016), it is necessary to allow to recreate – automatically – the test
environments rather than trying to repair them. This automatic creation solution
ensures an identical test environment to the previous version, which will facilitate
regression testing.
1.1.4. Availability of test data
It is obvious that the input test data of a test case and the expected data at the
output of a test case are necessary, and it is also important to have a set of other data
that will be used for testing:
– data related to the users who will run the tests (e.g. authorization level,
hierarchical level, organization to which they are attached, etc.);
– information related to the test data used (e.g. technical characteristics,
composition, functionalities present, etc.) and which are grouped in legacy systems
interfaced with the system-of-systems under test;
– historical information allowing us to make proposals based on this historical
information (e.g. purchase suggestions based on previous purchases);
– information based on geographical positioning (e.g. GPS position), supply
times and consumption volumes to anticipate stock replenishment needs (e.g. need
to fill the fuel tank according to the way to drive and consume fuel, making it
possible to offer – depending on the route and GPS information – one or more
service stations nearby);
– etc.
The creation and provision of quality test data is necessary before any test
campaign. Designing and updating this data, ensuring that it is consistent, is
extremely important because it must – as far as possible – simulate the reality of the
exchanges and information of each of the systems of the system-of-systems to be
tested. We will therefore need to generate data from monitoring systems (from
sensors, via IoT systems) and ensure that their production respects the expected
constraints (e.g. every n seconds, in order to identify connection losses or deviations
from nominal operating ranges).
Test data should be realistic and consistent over time. That is, they must either
simulate a reference period and each of the campaigns must ensure that the systems
have modified their reference date (e.g. use a fixed range of hours and reset systems
at the beginning of this range) or be consistent with the time of execution of the test
campaign. This last solution requires generating the test data during the execution of
the test campaign, in order to verify the consistency of the data with respect to the
expected (e.g. identification of duplicate messages, sequencing of messages, etc.)
and therefore the proper functioning of the system-of-systems as a whole.
1.1.5. Compliance of deliveries and schedules
Development and construction projects are associated with often strict delivery
dates and schedules. The impact of a late delivery of a component generates
cascading effects impacting the delivery of the system and the system-of-systems.
Timely delivery, with the expected features and the desired level of quality, is
therefore very important. In some systems-of-systems, the completeness of the
functionalities and their level of quality are often more important than the respect of
the delivery date. In others, respecting the schedule is crucial in order to meet
imperatives (e.g. launch window for a rocket aiming for another planet).
Test projects depend on the delivery of requirements and components to be

tested within a specific schedule. Indeed, testers can only design tests based on the
requirements, user stories and features delivered to them and can only run tests on
the components, products and systems delivered to them in the appropriate test
environments (i.e. including the necessary data and systems). The timely delivery of
deliverables (contracts, requirements documents, specifications, features, user
stories, etc.) and components, products and systems in a usable state – that is, with
information or expected and working functionality – is crucial, or testers will not be
able to perform their tasks properly.
This involves close collaboration between test manager and project managers in
charge of the design and production of components, products or systems to be
tested, as well as managers in charge of test environments and the supply of test
data.
In the context of Agile and Lean methods, any delay in deliveries and any
non-compliance with schedules is a “loss of value” and should be eliminated. It is
however important to note that the principles of agility propose that it is the
development teams that define the scope of the functionalities to be delivered at each
iteration.
1.1.6. Coordinating and setting up environments
Depending on the test levels, environments will include more and more
components, products and systems that will need to coordinate to represent test
environments representative of real life. Each environment includes one or more
systems, components, products, as well as interfaces, ETLs and communication
equipment (wired, wireless, satellite, optical networks, etc.) of increasing
complexity. The design of these various environments quickly becomes a full-time
job, especially since it is necessary to ensure that all the versions of all the software
are correctly synchronized and that all the data, files, contents of databases and
interfaces are synchronized and validated in order to allow the correct execution of
the tests on this environment.
The activity of coordinating and setting up environments interacts strongly with

all the other projects participating in the realization of the system-of-systems. Some
test environments will only be able to simulate part of the target environment (e.g.
simulation of space vacuum and sunlight with no ability to simulate zero gravity),
and therefore there may be, for the same test level, several test execution campaigns,
each on different technical or functional domains.
1.1.7. Validation of prerequisites – Test Readiness Review (TRR)
Testing activities can start effectively and efficiently as soon as all their
prerequisites are present. Otherwise, the activities will have to stop and then start
again when the missing prerequisite is provided, etc. This generates significant
waste of time, not to mention everyone’s frustration. Before starting any test task,
we must make sure that all the prerequisites are present, or at the very least that they
will arrive on time with the desired level of quality. Among the prerequisites, we
have among others the requirements, the environment, the datasets, the component
to be tested, the test cases with the expected data, as well as the testers, the tools and
procedures for managing tests and anomalies, the KPIs and metrics allowing the
reporting of the progress of the tests, etc.
One solution to ensure the presence of the prerequisites is to set up a TRR (Test
Readiness Review) milestone, a review of the start of the tests. The purpose of this
milestone is to verify – depending on the test level and the types of test – whether or
not the prerequisites are present. If prerequisites are missing, it is up to the project
managers to decide whether or not to launch the test activity, taking into account the
identified risks.
In Agile methods, such a review can be informal and only apply to one user story
at a time, with the acronym DOR for definition of ready.
1.1.8. Delivery of datasets (TDS)
The delivery of test datasets (TDS) is not limited to the provision of files or
databases with information usable by the component, product or system. This also
includes – for the applications, components, products or systems with which the
component, product or system under test interacts – a check of the consistency and
synchronization of the data with each other. It will be necessary to ensure that the
interfaces are correctly described, defined and implemented.
Backup of datasets or automation of dataset generation processes may be

necessary to allow testers to generate the data they need themselves.
The design of coherent and complete datasets is a difficult task requiring a good
knowledge of the entire information system and the interfaces between the
component, product or system under test on the one hand and all the other systems
of the test environment on the other hand. Some components, products or systems
may be missing and replaced by “stubs” that will simulate the missing elements. In
this case, it is necessary to manage these “stubs” with the same rigor as if they were
real components (e.g. evolution of versions, data, etc.).
1.1.9. Go-NoGo decision – Test Review Board (TRB)
A Go-NoGo meeting is used to analyze the risks associated with moving to the
next step in a process of designing and deploying a component, product, system or
system-of-systems, and to decide whether to proceed to the next step.
This meeting is sometimes split into two reviews in time:

– A TRB (Test Review Board) meeting analyzes the results of the tests carried
out in the level and determines the actions according to these results. This technical
meeting ensures that the planned objectives have been achieved for the level.
– A management review to obtain – from the hierarchy, the other stakeholders,

the MOA and the customers – a decision (the “Go” or the “NoGo” decision)
accepted by all, with consideration of business risks, marketing, etc.
The Go-NoGo meeting includes representatives from all business stakeholders,

such as operations managers, deployment teams, production teams and marketing
teams.
In an Agile environment, the concept of Go-NoGo and TRB is detailed under the
concept of DOD (definition of done) for each of the design actions.
1.1.10. Continuous delivery and deployment
The concept of continuous integration and continuous delivery (CI/CD) is

interesting and deserves to be considered in systems-of-systems with preponderant
software. However, such concepts have particular constraints that we must study,
beyond the use of an Agile design methodology.
1.1.10.1. Continuous delivery

The continuous delivery practices mentioned in Kim et al. (2016) focus primarily
on the aspects of continuous delivery and deployment of software that depend on
automated testing performed to ensure developers have quick (immediate) feedback
on the defects, performance, security and usability concerns of the components put
in configuration. In addition, the principle is to have a limited number of
configuration branches.
In the context of systems-of-systems, where hardware components and

subsystems including software must be physically installed – and tested on physical
test benches – the ability to deliver daily and ensure the absence of regressions
becomes more complex, if not impossible, to implement. This is all the more true
since the systems-of-systems are not produced in large quantities and the
interactions are complex.
1.1.10.2. Continuous testing

On-demand execution of tests as part of continuous delivery is possible for unit
testing and static testing of code. Software integration testing could be considered,
but anything involving end-to-end (E2E) testing becomes more problematic because
installing the software on the hardware component should generate a change in the
configuration reference of the hardware component.
Among the elements to consider, we have an ambiguity of terminology: the term

ATDD (Acceptance Test-Driven Development) relates to the acceptance of the
software component alone, not its integration, nor the acceptance of the system-of-
system nor of the subsystem or equipment.
Another aspect to consider is the need for test automation and (1) the continued
increase in the number of tests to be executed, which will mean increasing test
execution time as well as (2) the need to ensure that the test classes in the software
(case of TDD and BDD) are correctly removed from the versions used in integration
tests and in system tests.
One of the temptations associated with testing in a CI/CD or DevOps

environment is to pool the tests of the various software components into a single test
batch for the release, instead of processing the tests separately for each component.
This solution makes it possible to pool the regression tests of software components,
but is a difficult practical problem for the qualification of systems-of-systems as
mentioned in Sacquet and Rochefolle (2016).
1.1.10.3. Continuous deployment

Continuous deployment depends on continuous delivery and therefore automated
validation of tests, and the presence of complete documentation – for component
usage and administration – as well as the ability to run end-to-end on an
environment representative of production.
According to Kim et al. (2016), in companies like Amazon and Google, the
majority of teams practice continuous delivery and some practice continuous
deployment. There is wide variation in how to perform continuous deployment.
1.2. Tracking test projects
Monitoring test projects requires monitoring the progress of each of the test
activities for each of the systems of the system-of-systems, as well as on each of the
test environments of each of the test levels of each of these systems. It is therefore
important that the progress information of each test level is aggregated and
summarized for each system and that the test progress information of each system is
aggregated at the system-of-systems level. This involves defining the elements that
must be measured (the progress), against which benchmark they must be measured
(the reference) and identifying the impacts (dependencies) that this can generate.
Reporting of similar indicators from each of the systems will facilitate
understanding. Automated information feedback will facilitate information retrieval.
1.3. Risks and systems-of-systems
Systems-of-systems projects are subject to more risk than other systems in that
they may inherit upstream-level risks and a process’s tolerance for risk may vary by
organization and the delivered product. In Figure 1.1, we can identify that the more
we advance in the design and production of components by the various
organizations, the risks will be added and the impact for organizations with a low
risk tolerance will be more strongly impacted than others.
Figure 1.1. Different risk tolerance
In Figure 1.2, we can identify that an organization will be impacted by all the
risks it can inherit from upstream organizations and that it will impose risks on all
downstream organizations.
Figure 1.2. Inherited and imposed risks
We realize that risk management in systems-of-systems is significantly more

complex than in the case of complex systems and may need to be managed at
multiple levels (e.g. interactions between teams, between managers of the project or
between the managers – or leaders – of the organizations).
1.4. Particularities related to SoS
According to Firesmith (2014), several pitfalls should be avoided in the context

of systems-of-systems, including:
– inadequate system-of-systems test planning;
– unclear responsibilities, including liability limits;
– inadequate resources dedicated to system-of-systems testing;
– lack of clear systems-of-systems planning;
– insufficient or inadequate systems-of-systems requirements;
– inadequate support of individual systems and projects;
– inadequate cross-project defect management.
To this we can add:

– different quality requirements according to the participants/co-contractors,
including regarding the interpretation of regulatory obligations;
– the needs to take into account long-term evolutions;
– the multiplicity of level versions (groupings of software working and delivered
together), multiple versions and environments;
– the fact that systems-of-systems are often unique developments.
1.5. Particularities related to SoS methodologies
Development methodologies generate different constraints and opportunities.

Sequential developments have demonstrated their effectiveness, but involve
constraints of rigidity and lack of responsiveness, if the contexts change. Agility
offers better responsiveness at the expense of a more restricted analysis phase and an
organization that does not guarantee that all the requirements will be developed. The
choice of a development methodology will imply adaptations during the
management of the project and during the testing of the components of the system-
of-systems.
Iterative methodologies involve rapid delivery of components or parts of

components, followed by refinement phases if necessary. That implies that:
– The planned functionalities are not fully provided before the last delivery of
the component. Validation by the business may be delayed until the final delivery of
the component. This reduces the time for detecting and correcting anomalies and
can impact the final delivery of the component, product or system, or even the
system-of-systems.
– Side effects may appear on other components, so it will be necessary to retest
all components each time a component update is delivered. This solution can be
limited to the components interacting directly with the modified component(s) or
extend to the entire system-of-systems, and it is recommended to automate it.
– The interfaces between components may not be developed simultaneously and
therefore that the tests of these interfaces may be delayed.
Sequential methodologies (e.g. V-cycle, Spiral, etc.) focus on a single delivery,

so any evolution – or need for clarification – of the requirement will have an impact
on lead time and workload, both in terms of development (redevelopment or
adaptation of components, products or systems) and in terms of testing (design and
execution of tests).
1.5.1. Components definition
Within the framework of sequential methodologies, the principle is to define the

components and deliver them finished and validated at the end of their design phase.
This involves a complete definition of each product or system component and the
interactions it has with other components, products or systems. These exhaustive
definitions will be used both for the design of the component, product or system and
for the design of the tests that will validate them.
1.5.2. Testing and quality assurance activities
It is not possible to envisage retesting all the combinations of data and actions of
the components of a level of a system-of-systems; this would generate a workload
disproportionate to the expected benefits. One solution is to verify that the design
and test processes have been correctly carried out, that the proofs of execution are
available and that the test activities – static and dynamic – have correctly covered
the objectives. These verification activities are the responsibility of the quality
assurance teams and are mainly based on available evidence (paper documentation,
execution logs, anomaly dashboards, etc.).
1.6. Particularities related to teams
In a test project, whether it is software testing or systems-of-systems testing, one

element to take into account is the management of team members, and their
relationships with each other, others and to the outside. This information is grouped
into what NASA calls CRM (Crew Resource Management). Developed in the
1970s–1980s, CRM is a mature discipline that applies to complex projects and is
ideal for decision-making processes in project management.
It is essential to:
– recognize the existence of a problem;
– define what the problem is;
– identify probable solutions;
– take the appropriate actions to implement a solution.
If CRM is mainly used where human error can have devastating effects, it is
important to take into account the lessons that CRM can bring us in the
implementation of decision-making processes. Contrary to a usual vision, people
with responsibilities (managers and decision-makers) or with the most experience
are sometimes blinded by their vision of a solution and do not take into account
alternative solutions. Among the points to keep in mind is communication between
the different members of the team, mutual respect – which will entail listening to the
information provided – and then measuring the results of the solutions implemented
in order to ensure their effectiveness. Team members can all communicate important
information that will help the project succeed.
The specialization of the members of the project team, the confidence that we
have in their skills and the confidence that they have in their experience, the
management methods and the constraints – contractual or otherwise – mean that the
decision-making method and the decisions made can be negatively impacted in the
absence of this CRM technique. This CRM technique has been successfully
implemented in aeronautics and space, and its lessons should be used successfully in
complex projects.
2
Testing Process
Test processes are nested within the set of processes of a system-of-systems.

More specifically, they prepare and provide evidence to substantiate compliance
with requirements and provide feedback to project management on the progress of
test process activities. If CMMI is used, other process areas than VER and VAL will
be involved: PPQA (Process and Product Quality Assurance), PMC (Project
Monitoring and Control), REQM (Requirements Management), CM (Configuration
Management), TS (Technical Solution), MA (Measurement and Analysis), etc.
These processes will all be involved to some degree in the testing processes.
Indeed, the test processes will decline the requirements, whether or not they are
defined in documents describing the conformity needs, and the way in which these
requirements will be demonstrated (type of IADT proofs), will split the types of
demonstration according to the levels test and integration (system, subsystem,
sub-subsystem, component, etc.) and static types (Analysis and Inspection for static
checks during design) or dynamic (demonstration and tests during the levels of
integration and testing of subsystems, systems and systems-of-systems). Similarly,
the activities of the test process will report information and progress metrics to
project management (CMMI PMC for Project Monitoring and Control process) and
will be impacted by the decisions descending from this management.
The processes described in this chapter apply to a test level and should be
repeated on each of the test levels, for each piece of software or containing software.
Any modification in the interfaces and/or the performance of a component
interacting with the component(s) under test will involve an analysis of the impacts
and, if necessary, an adaptation of the test activities (including about the test) and
evidence to be provided to show the conformity of the component (or system,
subsystem, equipment or software) to its requirements. Each test level should
coordinate with the other levels to limit the execution of tests on the same
requirements.
The ISO/IEC/IEEE29119-1 standard describes the following processes, grouping

these in organizational processes (in yellow), management processes (pink) and
dynamic processes (green).
Figure 2.1. Test processes. For a color version of this

figure, see www.iste.co.uk/homes/systems2.zip
Defined test processes are repeatable at each test level of a system-of-systems:

– the general organization of the test level;
– planning of level testing activities;
– control of test activities;
– analysis of needs, requirements and user stories to be tested;
– design of the test cases applicable to the level;
– implementation of test cases with automation and provision of test data;
– execution of designed and implemented test cases, including the management
of anomalies;
– evaluation of test execution results and exit criteria;
– reporting;
– closure of test activities, including feedback and continuous improvement
actions;
– infrastructure and environment management.
Testing Process 17
An additional process can be defined: the review process, which can be carried
out several times on a test level, on the one hand, on the input deliverables, and on
the other hand, on the deliverables produced by each of the processes of the level.
Review activities can occur within each defined test process.
The proposed test processes are applicable regardless of the development mode
(Agile or sequential). In the case of an Agile development mode, the testing
processes must be repeated for each sprint and for each level of integration in a
system-of-systems.
The processes must complement each other and – even if they may partially
overlap – it must be ensured that the processes are completed successfully.
2.1. Organization
Objectives:
– develop and manage organizational needs, in accordance with the company’s
test policy and the test strategies of higher levels;
– define the players at the level, their responsibilities and organizations;
– define deliverables and milestones;
– define quality targets (SLA, KPi, maximum failure rate, etc.);
– ensure that the objectives of the test strategy are addressed;
– define a standard RACI matrix.
Actor(s):
– CPI (R+A), CPU/CPO (I), developers (C+I);
– experienced “test manager” having a pilot role of the test project (R).
Prerequisites/inputs:
– calendar and budgetary constraints defined for the level;
– actors and subcontractors envisaged or selected;
– repository of lessons learned from previous projects.
Deliverables/outputs:
– organization of level tests;
– high-level WBS with the main tasks to be carried out;
– initial definition of test environments.
Entry criteria:
– beginning of the organization phase.
Exit criteria:
– approved organizational document (ideally a reduced number of pages).
Indicators:
1) efficiency: writing effort;
2) coverage: traceability to the quality characteristics identified in the project test
strategy.
Points of attention:
– ensure that the actors and meeting points (milestones and level of reporting)
are well defined.
2.2. Planning
Objective:
– plan test activities for the project, level, iteration or sprint considering existing
issues, risk levels, constraints and objectives for testing;
– define the tasks (durations, objectives, incoming and outgoing, responsibilities,
etc.) and sequencing;
– define the exit criteria (desired quality level) for the level;
– identify the prerequisites, resources (environment, personnel, tools, etc.)
necessary;
– define measurement indicators and frequencies, as well as reporting.
Actor(s):
– experienced testers “test manager”, having a role of manager of the test
project (R);
– testers (C+I).
Testing Process 19
– information on the volume, workload and deadlines of the project;
– information on available environments and interfaces;
– objectives and scope of testing activities.
2.2.1. Project WBS and planning
Objective:
– plan test activities for the project or level, iteration or sprint considering
existing status, risk levels, constraints and objectives for testing;
– define the tasks (durations, objectives, incoming and outgoing, responsibilities,
etc.) and sequencing;
– define the exit criteria (desired quality level) for the level;
– identify prerequisites, resources (environment, personnel, tools, etc.)
necessary;
– define measurement indicators and frequencies, as well as reporting.
Actor(s):
– experienced testers “test manager”, having a role of manager of the test project
(R);
– testers (C+I).
– REAL and project WBS defined in the investigation phase;
– lessons learned from previous projects (repository of lessons learned).
– master test plan, level test plan(s);
– level WBS (or TBS for Test Breakdown Structure), detailing – for the
applicable test level(s) – the tasks to be performed;
Entry criteria:
– start of the investigation phase.
Exit criteria:
– test plan approved, all sections of the test plan template are completed.
Indicators:
1) efficiency: writing effort vs. completeness and size of the deliverables
provided;
2) coverage: coverage of the quality characteristics selected in the project Test
Strategy.
– ensure that test data (for interface tests, environment settings, etc.) will be well
defined and provided in a timely manner;
– collect lessons learned from previous projects.
– master test plan, level test plan(s);
– level WBS, detailing – for the test level(s) – the tasks to be performed;
– detailed Gantt of test projects – each level – with dependencies;
Entry criteria:
– start of the investigation phase.
Exit criteria:
– approved test plan, all sections of the applicable test plan template are
completed.
Indicators:
1) efficiency: writing effort;
2) coverage: coverage of the quality characteristics selected in the project’s test
strategy.
Testing Process 21
– ensure that test data (for interface testing, environment settings, etc.) will be
well defined and provided in a timely manner.
2.3. Control of test activities
Objective:
– throughout the project: adapt the test plan, processes and actions, based on the
hazards and indicators reported by the test activities, so as to enable the project to
achieve its objectives;
– identify changes in risks, implement mitigation actions;
– provide periodic reporting to the CoPil and the CoSuiv;
– escalate issues if needed.
Actor(s):
– CPI (A+I), CPU/CPO (I), developers (I);
– test manager with a test project manager role (R);
– testers (C+I) [provide indicators];
– CoPil CoNext (I).
– risk analysis, level WBS, project and level test plan.
– periodic indicators and reporting for the CoPil and CoSuiv;
– updated risk analysis;
– modification of the test plan and/or activities to allow the achievement of the
“project” objectives.
Entry criteria:
– project WBS, level WBS.
Exit criteria:
– end of the project, including end of the software warranty period.
Indicators:
– dependent on testing activities.
2.4. Analyze
Objective:
– analyze the repository of information (requirements, user stories, etc. usable
for testing) to identify the test conditions to be covered and the test techniques to be
used. A risk or requirement can be covered by more than one test condition. A test
condition is something – a behavior or a combination of conditions – that may be
interesting or useful to test.
Actor(s):
– testers, test analysts, technical test analysts.
– initial definition of test environments;
– requirements and user stories (depending on the development method);
– acceptance criteria for (if available);
– analysis of prioritized project risks;
– level test plan with the characteristics to be covered, the level test environment.
– detailed definition of the level test environment;
– test file;
– prioritized test conditions;
– requirements/risks traceability matrix – test conditions.
Entry criteria:
– validated and prioritized requirements;
– risk analysis.
Exit criteria:
– each requirement is covered by the required number of test conditions
(depending on the RPN of the requirement).
Another random document with
no related content on Scribd:
conceipt of Cynthia, (Phœbe and Cynthia being both names of
Diana.) So in the person of Prince Arthure I sette forth magnificence
in particular, which vertue for that (according to Aristotle and the rest)
it is the perfection of all the rest, and conteineth in it them all,
therefore in the whole course I mention the deedes of Arthure
applyable to that vertue, which I write of in that booke. But of the xii.
other vertues, I make xii. other knights the patrones, for the more
variety of the history: Of which these three bookes contayn three,
The first of the knight of the Redcrosse, in whome I expresse
Holynes: The seconde of Sir Guyon, in whome I sette forth
Temperaunce; The third of Britomartis a Lady knight, in whome I
picture Chastity. But because the beginning of the whole worke
seemeth abrupte and as depending vpon other antecedents, it needs
that ye know the occasion of these three knights seuerall
aduentures. For the Methode of a Poet historical is not such, as of
an Historiographer. For an Historiographer discourseth of offayres
orderly as they were donne, accounting as well the times as the
actions, but a Poet thrusteth into the middest, euen where it most
concerneth him, and there recoursing to the thinges forepaste, and
diuining of thinges to come, maketh a pleasing Analysis of all. The
beginning therefore of my history, if it were to be told by an
Historiographer should be the twelfth booke, which is the last, where
I deuise that the Faery Queene kept her Annuall feaste xii. dayes,
vppon which xii. seuerall dayes, the occasions of the xii. seuerall
aduentures hapned, which being vndertaken by xii. seuerall knights,
are in these xii books seuerally handled and discoursed. The first
was this. In the beginning of the feast, there presented him selfe a
tall clownishe younge man, who falling before the Queen of Faries
desired a boone (as the manner then was) which during that feast
she might not refuse: which was that hee might haue the
atchieuement of any adventure, which during that feaste should
happen, that being graunted, he rested him on the floore, vnfitte
through[638] his rusticity for a better place. Soone after entred a faire
Ladye in mourning weedes, riding on a white Asse, with a dwarfe
behind her leading a warlike steed, that bore the Armes of a knight,
and his speare in the dwarfes hand. Shee falling before the Queene
of Faeries, complayned that her father and mother an ancient King
and Queene, had bene by an huge dragon many years shut up in a
brasen Castle, who thence suffred them not to yssew: and therefore
besought the Faery Queene to assygne her some one of her knights
to take on him that exployt. Presently that clownish person
vpstarting, desired that aduenture: whereat the Queene much
wondering, and the Lady much gainesaying, yet he earnestly
importuned his desire. In the end the Lady told him that vnlesse that
armour which she brought, would serue him (that is the armour of a
Christian man specified by Saint Paul v. Ephes.) that he could not
succeed in that enterprise, which being forthwith put vpon him with
dewe furnitures thereunto, he seemed the goodliest man in al that
company, and was well liked of the Lady. And eftesoones taking on
him knighthood, and mounting on that straunge Courser, he went
forth with her on that aduenture: where beginneth the first booke, vz.
A gentle knight was pricking on the playne. &c.
The second day ther came in a Palmer bearing an Infant with bloody
hands, whose Parents he complained to haue bene slayn by an
Enchaunteresse called Acrasia: and therfore craued of the Faery
Queene, to appoint him some knight, to performe that aduenture,
which being assigned to Sir Guyon, he presently went forth with that
same Palmer: which is the beginning of the second booke and the
whole subiect thereof. The third day there came in, a Groome who
complained before the Faery Queene, that a vile Enchaunter called
Busirane had in hand a most faire Lady called Amoretta, whom he
kept in most grieuous torment, because she would not yield him the
pleasure of her body. Whereupon Sir Scudamour the louer of that
Lady presently tooke on him that aduenture. But being vnable to
performe it by reason of the hard Enchauntments, after long sorrow,
in the end met with Britomartis, who succoured him, and reskewed
his loue.
But by occasion hereof, many other aduentures are intermedled, but
rather as Accidents, then intendments. As the loue of Britomart, the
ouerthrow of Marinell, the misery of Florimell, the vertuousnes of
Belphœbe, the lasciuiousnes of Hellenora, and many the like.
Thus much Sir, I haue briefly ouerronne to direct your understanding
to the wel-head of the History, that from thence gathering the whole
intention of the conceit, ye may as in a handfull gripe al the
discourse, which otherwise may happily seeme tedious and
confused. So humbly crauing the continuaunce of your honorable
fauour towards me, and th’ eternall establishment of your happines, I
humbly take leaue.
23. Ianuary. 1589.
Yours most humbly affectionate.
Ed. Spenser.
FOOTNOTES:
[635] A Letter, &c.] Om. 1596 Bodl.
[636] l. 16 by accidents 1590
[637] l. 16 ezpresse 1590
[638] l. 43 through 1590
COMMENDATORY VERSES
¶ A Vision vpon this conceipt of the Faery
Queene.
Me thought I saw the graue, where Laura lay,
Within that Temple, where the vestall flame
Was wont to burne, and passing by that way,
To see that buried dust of liuing fame,
Whose tombe faire loue, and fairer vertue kept,
All suddenly I saw the Faery Queene:
At whose approch the soule of Petrarke wept,
And from thenceforth those graces were not seene.
For they this Queene attended, in whose steed
Obliuion laid him downe on Lauras herse:
Hereat the hardest stones were seene to bleed,
And grones of buried ghostes the heauens did perse.
Where Homers spright did tremble all for griefe,
And curst th’accesse of that celestiall theife.
Another of the same.

The prayse of meaner wits this worke like profit brings,
As doth the Cuckoes song delight when Philumena sings.
If thou hast formed right true vertues face herein:
Vertue her selfe can best discerne, to whom they written bin.
If thou hast beautie praysd, let her sole lookes diuine
Iudge if ought therein be amis, and mend it by her eine.
If Chastitie want ought, or Temperance her dew,
Behold her Princely mind aright, and write thy Queene anew.
Meane while she shall perceiue, how farre her vertues sore
Aboue the reach of all that liue, or such as wrote of yore:
And thereby will excuse and fauour thy good will:
Whose vertue can not be exprest, but by an Angels quill.
Of me no lines are lou’d, nor letters are of price,
Of all which speake our English tongue, but those of thy deuice.
W. R.
To the learned Shepheard.

Collyn I see by thy new taken taske,
some sacred fury bath enricht thy braynes,
That leades thy muse in haughtie verse to maske,
and loath the layes that longs to lowly swaynes.
That lifts thy notes from Shepheardes vnto kings,
So like the liuely Larke that mounting sings.
Thy louely Rosolinde seemes now forlorne,

and all thy gentle flockes forgotten quight,
Thy chaunged hart now holdes thy pypes in scorne,
those prety pypes that did thy mates delight.
Those trustie mates, that loued thee so well,
Whom thou gau’st mirth: as they gaue thee the bell.
Yet as thou earst with thy sweete roundelayes,

didst stirre to glee our laddes in homely bowers:
So moughtst thou now in these refyned layes,
delight the dainty eares of higher powers.
And so mought they in their deepe skanning skill
Alow and grace our Collyns flowing quill.
And fare befall that Faerie Queene of thine,

in whose faire eyes loue linckt with vertue sits.
Enfusing by those bewties fiers deuyne,
such high conceites into thy humble wits,
As raised hath poore pastors oaten reede,
From rusticke tunes, to chaunt heroique deedes.
So mought thy Redcrosse knight with happy hand

victorious be in that faire Ilands right:
Which thou doest vaile in Type of Faery land
Elyzas blessed field, that Albion hight.
That shieldes her friends, and warres her mightie foes,
Yet still with people, peace, and plentie flowes.
But (iolly Shepheard) though with pleasing style,

thou feast the humour of the Courtly traine;
Let not conceipt thy setled sence beguile,
ne daunted be through enuy or disdaine.
Subiect thy dome to her Empyring spright,
From whence thy Muse, and all the world takes light.
Hobynoll.
Fayre Thamis[639] streams, that from Ludds stately towns,

Runst paying tribute to the Ocean seas,
Let all thy Nymphes and Syrens of renowne
Be silent, whyle this Bryttane Orpheus playes:
Nere thy sweet bankes, there liues that sacred crowne,
Whose hand strowes Palme and neuer-dying bayes,
Let all at once, with thy soft murmuring sowne
Present her with this worthy Poets prayes.
For he hath taught hye drifts in shepeherdes weedes,
And deepe conceites now singes in Faeries deedes.
R. S.
Graue Muses march in triumph and with prayses.

Our Goddesse here hath giuen you leaue to land:
And biddes this rare dispenser of your graces
Bow downe his brow vnto her sacred hand.
Desertes findes dew in that most princely doome.
In whose sweete brest are all the Muses bredde:
So did that great Augustus erst in Roome
With leaues of fame adorne his Poets hedde.
Faire be the guerdon of your Faery Queene,
Euen of the fairest that the world hath seene.
H. B.
When stout Achilles heard of Helens rape

And what reuenge the States of Greece deuisd:
Thinking by sleight the fatall warres to scape,
In womans weedes him selfe he then disguisde:
But this deuise Vlysses soone did spy,
And brought him forth, the chaunce of warre to try.
When Spencer saw the fame was spredd so large,

Through Faery land of their renowned Queene:
Loth that his Muse should take so great a charge.
As in such haughty matter to be seene,
To seeme a shepeheard then he made his choice,
But Sydney heard him sing, and knew his voice.
And as Vlysses brought faire Thetis sonne

From his retyred life to menage armes:
So Spencer was by Sidneys speaches wonne,
To blaze her fame not fearing future harmes:
For well he knew, his Muse would soone be tyred
In her high praise, that all the world admired.
Yet as Achilles in those warlike frayes,

Did win the palme from all the Grecian Peeres:
So Spencer now to his immortall prayse,
Hath wonne the Laurell quite from all his feres.
What though his taske exceed a humaine witt,
He is excus’d, sith Sidney thought it fitt.
W. L.
To looke upon a worke of rare deuise

The which a workman setteth out to view,
And not to yield it the deserued prise,
That vnto such a workmanship is dew[640],
Doth either proue the iudgement to be naught
Or els doth shew a mind with enuy fraught.
To labour to commend a peece of worke,
Which no man goes about to discommend,
Would raise a iealous doubt that there did lurke,
Some secret doubt, whereto the prayse did tend.
For when men know the goodnes of the wyne,
T’is needlesse for the hoast to haue a sygne.
Thus then to shew my iudgement to be such

As can discerne of colours blacke, and white,
As alls to free my minde from enuies tuch,
That neuer giues to any man his right,
I here pronounce this[641] workmanship is such,
As that no pen can set it forth too much.
And thus I hang a garland at the dore,

Not for to shew the goodnes of the ware:
But such hath beene the custome heretofore,
And customes very hardly broken are.
And when your tast shall tell you this is trew,
Then looke you giue your hoast his vtmost dew.
Ignoto.
FOOTNOTES:
[639] Fayre Thamis. &c.] This poem and those that follow are
omitted in 1596 Bodl.
[640] l. 17 dew. 1590
[641] l. 30 this 1590
DEDICATORY SONNETS
To the right honourable Sir Christopher Hatton,
Lord high Chauncelor of England. &c.
Those prudent heads, that with theire counsels wise
Whylom the Pillours of th’earth did sustaine,
And taught ambitious Rome to tyrannise,
And in the neck of all the world to rayne,
Oft from those graue affaires were wont abstaine,
With the sweet Lady Muses for to play:
So Ennius the elder Africane,
So Maro oft did Cæsars cares allay.
So you great Lord, that with your counsell sway
The burdeine of this kingdom mightily,
With like delightes sometimes may eke delay,
The[642] rugged brow of carefull Policy:
And to these ydle rymes lend litle space,
Which for their titles sake may find more grace.
To the right honourable the Lo. Burleigh Lo. high

Threasurer of England.
To you right noble Lord, whose carefull brest
To menage of most graue affaires is bent,
And on whose mightie shoulders most doth rest
The burdein of this kingdomes gouernement,
As the wide compasse of the firmament,
On Atlas mighty shoulders is vpstayd;
Vnfitly I these ydle rimes present,
The labor of lost time, and wit vnstayd:
Yet if their deeper sence be inly wayd,
And the dim vele, with which from comune vew
Their fairer parts are hid, aside be layd.
Perhaps not vaine they may appeare to you.
Such as they be, vouchsafe them to receaue,
And wipe their faults out of your censure graue.
E. S.
To the right Honourable the Earle of Oxenford,

Lord high Chamberlayne of England. &c.
Receiue most Noble Lord in gentle gree,
The vnripe fruit of an vnready wit:
Which by thy countenaunce doth craue to bee
Defended from foule Enuies poisnous bit.
Which so to doe may thee right well besit,
Sith th’antique glory of thine auncestry
Vnder a shady vele is therein writ,
And eke thine owne long liuing memory,
Succeeding them in true nobility:
And also for the loue, which thou doest beare
To th’Heliconian ymps, and they to thee,
They vnto thee, and thou to them most deare:
Deare as thou art vnto thy selfe, so loue
That loues and honours thee, as doth behoue.
To the right honourable the Earle of

Northumberland.
The sacred Muses haue made alwaies clame
To be the Nourses of nobility.
And Registres of euerlasting fame,
To all that armes professe and cheualry.
Then by like right the noble Progeny,
Which them succeed in fame and worth, are tyde
T’embrace the seruice of sweete Poetry,
By whose endeuours they are glorifide,
And eke from all, of whom it is enuide,
To patronize the authour of their praise,
Which giues them life, that els would soone haue dide,
And crownes their ashes with immortall baies.
To thee therefore right noble Lord I send
This present of my paines, it to defend.
To the right honourable the Earle of Cumberland.

Redoubted Lord, in whose corageous mind
The flowre of cheualry now bloosming faire,
Doth promise fruite worthy the noble kind,
Which of their praises haue left you the haire;
To you this humble present I prepare,
For loue of vertue and of Martiall praise,
To which though nobly ye inclined are,
As goodlie well ye shew’d in late assaies,
Yet braue ensample of long passed daies,
In which trew honor yee may fashiond see,
To like desire of honor may ye raise,
And fill your mind with magnanimitee.
Receiue it Lord therefore as it was ment,
For honor of your name and high descent.
E. S.
To the most honourable and excellent Lo. the

Earle of Essex. Great Maister of the Horse to her
Highnesse, and knight of the Noble order of the
Garter. &c.
Magnificke Lord, whose vertues excellent
Doe merit a most famous Poets witt.
To be thy liuing praises instrument,
Yet doe not sdeigne, to let thy name be writt
In this base Poeme, for thee far vnfitt.
Nought is thy worth disparaged thereby,
But when my Muse, whose fethers nothing flitt
Doe yet but flagg, and lowly learne to fly
With bolder wing shall dare alofte to sty
To the last praises of this Faery Queene,
Then shall it make more famous memory
Of thine Heroicke parts, such as they beene:
Till then vouchsafe thy noble countenaunce,
To these first labours needed furtheraunce[643].
To the right Honourable the Earle of Ormond and

Ossory.
Receiue most noble Lord a simple taste
Of the wilde fruit, which saluage soyl hath bred,
Which being through long wars left almost waste,
With brutish barbarisme is ouerspredd:
And in so faire a land, as may be redd,
Not one Parnassus, nor one Helicone
Left for sweete Muses to be harboured,
But where thy selfe hast thy braue mansione;
There in deede dwel faire Graces many one.
And gentle Nymphes, delights of learned wits,
And in thy person without Paragone
All goodly bountie and true honour sits,
Such therefore, as that wasted soyl doth yield,
Receiue dear Lord in worth, the fruit of barren field.
To the right honourable the Lo. Ch. Howard, Lo.

high Admiral of England, knight of the noble
order of the Garter, and one of her Maiesties
priuie Counsel. &c.
And ye, braue Lord, whose goodly personage,
And noble deeds each other garnishing,
Make you ensample to the present age,
Of th’old Heroes, whose famous ofspring
The antique Poets wont so much to sing,
In this same Pageaunt haue a worthy place,
Sith those huge castles of Castilian king,
That vainly threatned kingdomes to displace,
Like flying doues ye did before you chace;
And that proud people woxen insolent
Through many victories, didst first deface:
Thy praises euerlasting monument
Is in this verse engrauen semblably,
That it may liue to all posterity.
To the right honourable the Lord of Hunsdon,

high Chamberlaine to her Maiesty.
Renowmed Lord, that for your worthinesse
And noble deeds haue your deserued place,
High in the fauour of that Emperesse[644],
The worlds sole glory and her sexes grace,
Here eke of right haue you a worthie place,
Both for your nearnes to that Faerie Queene,
And for your owne high merit in like cace,
Of which, apparaunt proofe was to be seene,
When that tumultuous rage and fearfull deene
Of Northerne rebels ye did pacify,
And their disloiall powre defaced clene,
The record of enduring memory.
Liue Lord for euer in this lasting verse,
That all posteritie thy honour may reherse.
E. S.
To the most renowmed and valiant Lord, the Lord
Grey of Wilton, knight of the Noble order of the
Garter, &c.
Most Noble Lord the pillor of my life,
And Patrone of my Muses pupillage,
Through whose large bountie poured on me rife,
In the first season of my feeble age,
I now doe liue, bound yours by vassalage:
Sith nothing euer may redeeme, nor reaue
Out of your endlesse debt so sure a gage,
Vouchsafe in worth this small guift to receaue,
Which in your noble hands for pledge I leaue,
Of all the rest, that I am tyde t’account:
Rude rymes, the which a rustick Muse did weaue
In sauadge soyle, far from Parnasso mount,
And roughly wrought in an vnlearned Loome:
The which vouchsafe dear Lord your fauorable doome.
To the right honourable the Lord of Buckhurst,

one of her Maiesties priuie Counsell.
In vain I thinke right honourable Lord,
By this rude rime to memorize thy name;
Whose learned Muse hath writ her owne record,
In golden verse, worthy immortal fame:
Thou much more fit (were leasure to the same)
Thy gracious Souerains[645] praises to compile.
And her imperiall Maiestie to frame,
In loftie numbers and heroicke stile.
But sith thou maist not so, giue leaue a while
To baser wit his power therein to spend,
Whose grosse defaults thy daintie pen may file,
And vnaduised ouersights amend.
But euermore vouchsafe it to maintaine
Against vile Zoilus backbitings vaine.
To the right honourable Sir Fr. Walsingham

knight, principall Secretary to her Maiesty, and of
her honourable priuy Counsell.
That Mantuane Poetes incompared spirit,
Whose girland now is set in highest place,
Had not Mecænas for his worthy merit,
It first aduaunst to great Augustus grace,
Might long perhaps haue lien in silence bace,
Ne bene so much admir’d of later age.
This lowly Muse, that learns like steps to trace,
Flies for like aide vnto your Patronage;
That are the great Mecenas of this age,
As wel to al that ciuil artes professe
As those that are inspird with Martial rage,
And craues protection of her feeblenesse:
Which if ye yield, perhaps ye may her rayse
In bigger tunes to sound your liuing prayse.
E. S.
To the right noble Lord and most valiaunt

Captaine, Sir Iohn Norris knight, Lord president
of Mounster.
Who euer gaue more honourable prize
To the sweet Muse, then did the Martiall crew;
That their braue deeds she might immortalize
In her shril tromp, and sound their praises dew?
Who then ought more to fauour her, then you
Moste noble Lord, the honor of this age,
And Precedent of all that armes ensue?
Whose warlike prowesse and manly courage,
Tempred with reason and aduizement sage
Hath fild sad Belgicke with victorious spoile,
In Fraunce and Ireland left a famous gage,
And lately shakt the Lusitanian soile.
Sith then each where thou hast dispredd thy fame,
Loue him, that hath eternized your name.
E. S.
To the right noble and valorous knight, Sir Walter

Raleigh, Lo. Wardein of the Stanneryes, and
lieftenaunt of Cornewaile.
To thee that art the summers Nightingale,
Thy soueraine Goddesses most deare delight,
Why doe I send this rusticke Madrigale,
That may thy tunefull eare vnseason quite?
Thou onely fit this Argument to write,
In whose high thoughts Pleasure hath built her bowre,
And dainty loue learnd sweetly to endite.
My rimes I know vnsauory and sowre,
To tast the streames, that like a golden showre
Flow from thy fruitfull head, of thy loues praise,
Fitter perhaps to thonder Martiall stowre,
When so thee list thy lofty Muse to raise:
Yet till that thou thy Poeme wilt make knowne,
Let thy faire Cinthias praises bee thus rudely showne.
E. S.
To the right honourable and most vertuous Lady,

the Countesse of Penbroke.
Remembraunce of that most Heroicke spirit,
The heuens pride, the glory of our daies,
Which now triumpheth through immortall merit
Of his braue vertues, crownd with lasting baies,
Of heuenlie blis and euerlasting praies;
Who first my Muse did lift out of the flore,
To sing his sweet delights in lowlie laies;
Bids me most noble Lady to adore
His goodly image liuing euermore,
In the diuine resemblaunce of your face;
Which with your vertues ye embellish more,
And natiue beauty deck with heuenlie grace:
For his, and for your owne especial sake,
Vouchsafe from him this token in good worth to take.
E. S.
To the most vertuous, and beautifull Lady, the

Lady Carew.
Ne may I, without blot of endlesse blame,
You fairest Lady leaue out of this place,
But with remembraunce of your gracious name,
Wherewith that courtly garlond most ye grace,
And deck the world, adorne these verses base:
Not that these few lines can in them comprise
Those glorious ornaments of heuenly grace,
Wherewith ye triumph ouer feeble eyes,
And in subdued harts do tyranyse:
For thereunto doth need a golden quill,
And siluer leaues, them rightly to deuise,
But to make humble present of good will:
Which whenas timely meanes it purchase may,
In ampler wise it selfe will forth display.
E. S.
To all the gratious and beautifull Ladies in the
Court.
The Chian Peincter, when he was requirde
To pourtraict Venus in her perfect hew,
To make his worke more absolute, desird
Of all the fairest Maides to haue the vew.
Much more me needs to draw the semblant trew,
Of beauties Queene, the worlds sole wonderment,
To sharpe my sence with sundry beauties vew,
And steale from each some part of ornament.
If all the world to seeke I ouerwent,
A fairer crew yet no where could I see,
Then that braue court doth to mine eie present,
That the worlds pride seemes gathered there to bee.
Of each a part I stole by cunning thefte:
Forgiue it me faire Dames, sith lesse ye haue not lefte.
E. S.
FOOTNOTES:
[642] l. 12 The] he 1590
[643] l. 33 furtheraunce, 1590
[644] l. 5 Emperesse, 1590
[645] l. 8 Souerain 1590
FINIS.

Get Advanced Testing of Systems-of-Systems, Volume 2 - Practical Aspects 1st Edition Bernard Homes Free All Chapters

Uploaded by

Copyright:

Available Formats

Get Advanced Testing of Systems-of-Systems, Volume 2 - Practical Aspects 1st Edition Bernard Homes Free All Chapters

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Get Advanced Testing of Systems-of-Systems, Volume 2 - Practical Aspects 1st Edition Bernard Homes Free All Chapters

Uploaded by

Copyright:

Available Formats

Full download test bank at ebookmeta.

Advanced Testing of Systems-of-Systems, Volume 2 -

For dowload this book click LINK or Button below

Download More ebooks from https://ebookmeta.com

Advanced Testing of Systems-of-Systems, Volume 1 -

Advanced Computing and Systems for Security Volume 13

Practical Aspects of Vaccine Development The Practical

Intelligent Sustainable Systems Selected Papers of

Legal Aspects of Autonomous Systems: A Comparative

Radiation Detection Systems: Medical Imaging,

Applications of Big Data and Artificial Intelligence in

Advanced Dynamics Modeling, Duality and Control of

ISTE Ltd John Wiley & Sons, Inc.

© ISTE Ltd 2022

Library of Congress Control Number: 2022944148

British Library Cataloguing-in-Publication Data

Dedication and Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

Chapter 1. Test Project Management . . . . . . . . . . . . . . . . . . . . . . . . . 1

Chapter 2. Testing Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.3. Control of test activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Chapter 3. Continuous Process Improvement . . . . . . . . . . . . . . . . . . . 37

Chapter 4. Test, QA or IV&V Teams. . . . . . . . . . . . . . . . . . . . . . . . . . 51

4.5.1. Lead or direct? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Chapter 5. Test Workload Estimation . . . . . . . . . . . . . . . . . . . . . . . . 75

5.4.4. Test campaign preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Chapter 6. Metrics, KPI and Measurements . . . . . . . . . . . . . . . . . . . . 95

Chapter 7. Requirements Management . . . . . . . . . . . . . . . . . . . . . . . 119

7.3.3. Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Chapter 8. Defects Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

Chapter 9. Configuration Management . . . . . . . . . . . . . . . . . . . . . . . 143

9.6.3. Merge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

Chapter 10. Test Tools and Test Automation . . . . . . . . . . . . . . . . . . . 157

Chapter 11. Standards and Regulations . . . . . . . . . . . . . . . . . . . . . . . 187

Chapter 12. Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

Chapter 13. Future Testing Challenges . . . . . . . . . . . . . . . . . . . . . . . 223

13.6. Big Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233

Summary of Volume 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

Inspired by a dedication from Boris Beizer1, I dedicate these two books

Finally, paraphrasing Isaac Newton, If I was able to reach this level of

After 15 years in software development, I had the opportunity to focus on

1 Beizer, B. (1990). Software Testing Techniques, 2nd edition. ITP Media.

In the first part of these two books on systems-of-systems testing, we identified

In this second part of the book on systems-of-systems testing, we will focus on

Test Project Management

We do not claim to replace the many contributions of illustrious authors on good

At the risk of writing a tautology, the purpose of project management is to

Project management must ensure that development methodologies are correctly

1.1. General principles

Management of test projects, whether on components, products, systems or

Let us see the elements specific to software testing projects.

1.1.1. Quality of requirements

Any development translates requirements (needs or business objectives) into a

Requirements are provided to development teams as well as test teams.

The requirements must be SMART, that is:

– Achievable: the requirements must be able to be physically demonstrated

1.1.2. Completeness of deliveries

The completeness of the software, components, products, equipment and systems

This aspect of completeness is ambiguous in incremental and iterative

1.1.3. Availability of test environments