Software Quality Assurance: by Mark J. Tseytlin Sr. SQE, Raytheon Company
Software Quality Assurance: by Mark J. Tseytlin Sr. SQE, Raytheon Company
Software Quality Assurance: by Mark J. Tseytlin Sr. SQE, Raytheon Company
by Mark J. Tseytlin
Sr. SQE, Raytheon Company
1
Let’s Get Acquainted…
Phone: 978-440-2098
E-mail: [email protected]
2
What is Software Quality
Assurance?
3
What is Quality?
4
What is Quality Management?
Problems:
• Intangible aspects of software quality can’t be standardized
(i.e elegance and readability)
5
What are SQA, SQP, SQC, and SQM?
SQA includes all 4 elements…
• Software Quality Assurance – establishment of network of
organizational procedures and standards leading to high-
quality software
2. Software Quality Planning – selection of appropriate
procedures and standards from this framework and adaptation
of these to specific software project
3. Software Quality Control – definition and enactment of
processes that ensure that project quality procedures and
standards are being followed by the software development
team
4. Software Quality Metrics – collecting and analyzing quality
data to predict and control quality of the software product
being developed
6
Software Development Process
7
Software Development Process Spiral
Top-Down Design Model IPDS
8
Software Development Cycle
Software Development Phase End product
• Software requirements • SRS, IRS
• Preliminary Design • SDD, PQT/FAT/SAT
Plans & Proc.’s, ICD, IDD
• Detailed Design • PDL, User Manuals
• Code • Code, UT Plan & Proc.’s
• Unit Test • UT Results
• Software integration •VDD
• Software Component Test • PQT Report
• Software System Test • FAT & SAT Reports
• Maintenance and Support • ECP’s leading to updates
9
Software Development Cycle (contd.)
Req.
Maint PD
DD
SAT
FAT CUT
PQT Int
10
Integrated Product Development System
11
Software Quality Assurance
Element I
12
Software Development Standards
13
Why are Standards Important?
• Standards provide encapsulation of best, or at least most
appropriate, practice
• Standards provide a framework around which the quality
assurance process may be implemented
• Standards assist in continuity of work when it’s carried out by
different people throughout the software product lifecycle
14
SDS a Simplistic approach
Organizational Organizational
IPDS
Quality Manual SD Process STD’s
16
Product Standards
Product Standards – standards that apply to software
product being developed
Organizational
Product STD’s
17
Quality Models
18
ISO - 9001 Elements
• Quality System Requirements • Software Quality Responsibilities
• Management Responsibility • Management Responsibility
• Quality system • Quality system
• Contract review • Contract review
• Design Control • Design Control
• Document control • Document control
• Purchasing • Purchasing
• Purchaser supplied product • -
• Product identification and traceability • Product identification and traceability
• Process control • Process control
• Inspection and testing • Inspection and testing
• Inspection, measuring and test equipment • -
• Inspection and test status • Inspection and test status
• Control of non-conforming product • -
• Corrective action • Corrective action
• Handling, storage, preservation, packaging • -
and shipping
• Quality records • Quality records
• Internal quality audits • Internal quality audits
• Training • Training
• Servicing • -
• Statistical techniques • Statistical techniques
19
Capability Maturity Model KPA’s
20
CMM Level’s 2-5
21
CMM Integration Model Architecture
22
CMM to CMMI Mapping
23
Documentation
24
Documentation Hierarchy
26
Process and Product Quality Creative Approach
• Quality Improvement – identifying good quality products,
examining the processes used to develop these products, and then
generalizing these processes so that they can be applied everywhere
27
Quality Improvement – The Wheel of 6Sigma
Six Sigma
28
Quality Improvement – Six Sigma Process
• Visualize – Understand how it works now and imagine how it
will work in the future
30
Software Quality Planning
Element II
31
Software Quality Plan
• Tailoring - SQP should select those organizational standards that
are appropriate to a particular product
• Standardization - SQP should use (call out) only approved
organizational process and product standards
• If new standards are required a quality improvement should be
initiated
• Elements - SQP elements are usually based on the ISO-9001
model elements
• SQP is not written for software developers. It’s written for SQE’s
as a guide for SQC and for the customer to monitor development
activities
• Things like software production, software product plans and risk
management should be defined in SDP, IP
• Quality Factor’s shouldn’t be sacrificed to achieve efficiency.
Don’t take the job if quality process can’t be upheld
32
Software Quality Control
Element III
33
Methods of Software Quality Control
SQC involves overseeing the software development process to ensure that the
procedures and STD’s are being followed
• Tests - end-result verifications of products. These verifications are conducted after the
software has been developed. Test procedures are followed during conduct of these
activities. SQE is responsible for keeping the logs and some times for writing the test
report.
35
Tests
• Engineering Dry-run - test conducted by engineering without SQE. These tests
include Unit Tests and engineering dry-runs of the formal tests. These engineering dry-
runs are used to verify correctness and completeness of the test procedures. Also, these
is the final engineering verification of the end-product before sell-off to SQE. All
issues found during these tests are documented on STR forms.
• SQE Dry-run - test conducted by SQE. These tests include PQT, FAT and SAT dry-
runs. These tests are used to verify the end-product before the formal test with the
customer. An SQE is sometimes responsible for writing the test report. However, if a
separate test group is available, then SQE is relived of this obligation. All issues found
during these tests are documented on STR forms.
• TFR - test conducted as “RFR - run-for-record” with the SQE and the customer.
These tests include FAT and SAT. These tests are conducted to sell the end-product off
to the customer. SQE is present at all such tests. All issues found during these tests are
documented on STR forms.
36
Quality Audits
• SQE Audits - audits conducted by SQE to verify that the process STD’s are
being followed. Examples of these audits are IPDS compliance, Configuration
Control, and Software Engineering Management. All findings for these audits are
documented on QER forms. The results of the audits are distributed to the next
level of management (above project level). If the issue(s) are not fixed then the
findings are elevated to upper management.
37
Defect Detection
Formal bug finding activities include Quality Reviews and Tests
i s
is ys
ys al
al n
e An ts
A
ur ts en
apt en m gn io
n
C re
m
ui
re si n rat l ific
e i q e ig eg
lin qu e D es t ua
se e R r y D In Q
Ba
R
ar
e in
a d st ar
e
ar
e
te
m
lim ai
le Te
o m s f tw e t de it ftw ftw
Fr Sy So Pr De Co Un So So
At Baseline Capture 0
System Requirements Analysis 0 79
D Software Requirements Analysis 0 0 1
E Preliminary Design 0 6 2 10
S
T Detailed Design 1 0 0 0 42
T
E Code 0 0 0 1 2 37
A
C Unit Test 0 0 0 0 0 0 0
G
T Software Integration 1 0 0 0 4 1 0 0
E
E Software Qualification Test 0 0 0 0 0 0 0 0 0
D System Integration 1 0 0 0 4 5 0 0 0
System Test 0 0 0 0 0 0 0 0
Post System Test 0 0 0 0 0 0 0 0 0
% De fe cts Originate d in This Phase That We re Containe d By This Phase 93% 33% 91% 81% 86%
% De fe cts Originate d In This Phase Out Of All De fe cts 44% 2% 6% 27% 22% 0% 0% 0%
38
A Bug’s Life
Postponed Rejected
P X
N A Engineer O R M T V
SCCB Engineer Software Lead Tester SCCB
Approves STR Accepts STR Resolves STR Verifies Fix
Plans Merge Agrees Closure
Integrator
Duplicate
Performs Merge
D
39
Software Configuration Management
SCM – activities assuring that software products are properly
identified and their transition is tracked. In many mature
organizations SCM is not part of SQA responsibilities.
Element IV
41
Metrics Collection
• Software measurement - the process of deriving a numeric value
for some attribute of a software product or a software process. Comparison of these
values to each other and to STD’s allows drawing conclusions about the quality of
software products or the process.
• Most of the “Ilities” can not be measured directly unless there’s historical data.
Instead tangible software product attributes are measured and the “Ility” factors are
derived using predefined relationships between measurable and synthetic attributes.
45
The Ilities
46
Examples of Software Metric – Chapter 24
47
Examples of OO Software Metric – Chapter 24
48
Defect Prevention
Defect Prevention – establishment of practices that lower the reliance
on defect detection techniques to find majority of the bugs
• Lessons learned – learning from other peoples experiences and sharing own
experiences with the other projects
• Managing With Metrics – collecting the metrics, understanding it, and making
changes to the product or process based on analysis. Metrics must be standardized to
be effective.
• Risk Analysis – identifying potential risks and opportunities early in the program
and tracking them to realization.
• Build freeze – no changes are made to the code during formal tests.
• Unit-level testing guidelines – test plans and procedures for each UT
• Baseline acceptance criteria – establishment of closure criteria in advance (i.e. no
P1 STRs at FAT TRR)
49
The end
50