Introduction To Software Quality Assurance & Testing
Introduction To Software Quality Assurance & Testing
Introduction To Software Quality Assurance & Testing
• Focus
– Completeness, Consistency, Feasibility, Testability
Software Quality Assurance
• The two primary SQA activities:
– Technical Reviews
– Software Testing
Technical Reviews
• A “review” is a meeting where a work product is reviewed by a small group of people
who are qualified to give feedback, find problems, suggest improvements, etc.
• Anything can be reviewed: requirements spec, functional spec, design, code, test plan,
user documentation
• Reviews are the most effective QA technique, both in terms of cost and number of defects
discovered
Review Meetings
• Review the product – not the producer
• Set an agenda and keep it
• Limit debate and rebuttal
• Enunciate problem areas but don’t try to fix anything
• Take written notes
• Limit the number of participants
• Insist upon advance preparation
• Develop a checklist
• Allocate resources and time schedules
• Conduct meaningful training
Software Testing
• Testing is the process of detecting errors by running the actual software and
verifying that it works as it should
– Test cases, Expected results, Actual results
• Testing is by far the most popular QA activity (but not the most effective)
• Formal technical reviews are cheaper and more effective than testing, but are
often ignored
• Research has shown that all forms of testing combined usually find less than
60% of the errors present
• We must do partial testing because we only have enough resources (time and money) to
run relatively few test cases
– If the system passes all your test cases, there could still be defects, you just need
more or better test cases to find them
Software Testing
• Effective testing lies in intelligently choosing the relatively few test cases that will
actually be executed
– Test all requirements and features defined in the requirements spec. and functional
spec.
– Test cases should not be redundant (i.e., each one should follow a different path
through the code)
– Analyze the program’s design and code to find potential weak areas
– Analyze all points at which data enters the system and look for ways to attack it
– Code coverage
Software Testing
• Approaches for test case design are generally divided into two broad
categories: Black Box Testing and White Box Testing
• Effective testing requires the assumption that you will find defects
• If you think you won't find defects, or you don't want to, you will have set up a
self-fulfilling prophecy
• Needs to be automated
Regression Testing
Build results
Test data for version x
Build tests
for version x
Run tests
for version x Compare Verdict
Automating Regression Testing
• Challenging
• What parts of program output should be checked?
• Simple but annoying issues
– Use of dates in output
– Changes in whitespace
– Format changes
– Lead text changes
• Answer
– Don’t use complete output
– Just extract the relevant information
Regression Testing
• Keep it updated!
– Bug fixes – tests to ensure they stay fixed
– Functionality additions
– Platform changes
– Etc.
• If you branch the code, you must branch the regression tests
Formal Verification
• In addition to Technical Reviews and Software Testing, Formal Verification is another
approach to QA
• “Check” the model by formally proving that it implements the desired behavior
– Automated theorem proving systems are often applied
– Or, prove that the model does not behave correctly, thus revealing a defect
• Historically, formal verification has been expensive and limited to relatively small
programs, but techniques are improving all the time. Challenges include:
– Complex systems are hard to formalize
– State space explosion: real systems have so many possible states that proving
things about them is hard
– Ensuring that the “model” accurately captures the system’s behavior
– Making it accessible to people who aren’t formal verification experts