Common Mistakes
Common Mistakes
Common Mistakes
Automating the execution of tests is there are four attributes that describe the proportion of the bugs, but they must also
becoming more and more popular as the quality of a test case. Perhaps the most ensure that the test cases are well designed
need to improve software quality amidst important of these is its effectiveness, in order to avoid excessive costs.
increasing system complexity becomes whether or not it finds bugs, or at least,
ever stronger. The appeal of having the whether or not it is likely to find bugs. Automating tests is another skill requiring a
computer run the tests in a fraction of the Another attribute reflects how much the test different approach and extensive effort. For
time it takes to perform them manually has case does. A good test case should be exem- most organizations, it is expensive to auto-
led many organizations to attempt test plary, testing more than one thing thereby mate a test compared with the cost of per-
automation without a clear understanding reducing the total number test cases forming it once manually. In order to
of all that is involved. required. The other two attributes are both achieve a respectable return on investment
cost considerations: how economical a test (ROI), they have to automate with the
Consequently, many attempts have failed to case is to perform, analyze and debug; and understanding that each test will need to be
achieve real or lasting benefits. This paper how evolvable it is, or how much mainte- performed many times throughout its use-
highlights a few of the more common mis- nance effort is required on the test case each ful life.
takes that have contributed to these failures time the software changes.
and offers some thoughts on how they may Whether a test is automated or performed
be avoided. Figure 1 depicts the four quality attributes manually affects neither its effectiveness
of a test case in a Kiviat diagram and com- nor how exemplary it is. It doesn’t matter
Confusing Automation pares the likely measures of each on the how clever you are at automating a test or
and Testing same test case when it is performed manu- how well you do it, if the test itself is unable
Testing is a skill. While this may come as a ally (shown as an interactive test in the fig- to detect nor confirm the absence of a bug.
surprise to some people, it is a simple fact. ure) and after it has been automated. In such an instance, automation only accel-
For any system there are an astronomical erates failure. Automating a test affects only
number of possible test cases although we These four attributes must often be bal- how economic and evolvable it is. Once
have time to run only a very small number anced one against another. For example, a implemented, an automated test is general-
of them. Yet this small number of test cases single test case that tests a lot of things is ly much more economic; the cost of run-
is expected to find most of the bugs in the likely to cost a lot to perform, analyze and ning it correlates to a mere fraction of the
software, so the job of selecting which test debug. It may also require a lot of mainte- effort to perform it manually. However,
cases to build and run is an important one. nance each time the software changes. automated tests generally cost more to cre-
Both experiment and experience has told us Consequently, a high measure on the exem- ate and maintain. The better the approach to
that selecting test cases at random is not an plary scale is likely to result in low meas- automating tests, the cheaper it will be to
effective approach to testing. A more ures on the economic and evolvable scales. implement new automated
thoughtful approach is required if good test As this demonstrates, testing is indeed a
cases are to be developed. skill. Not only must testers ensure that the
What exactly is a good test case? Well, test cases they use are going to find a high
Figure 1a The ‘goodness’ of a test case Figure 1b When the ‘good’ manual test Figure 1c Although an automated test is
can be illustrated by considering the four case is automated, its measures of good- more economic than the same test case
attributes in this Kiviat diagram. The ness change as shown by the broken line. performed manually, after it has been exe-
greater the measure of each attribute, the While the test case is equally as effective cuted for the first time it is much less eco-
greater the area enclosed by the joining – it has the potential to find the same nomic since it has cost more to automate
lines and the better the test case. This faults as the manual test case – it proves it (shown by the dotted line in this dia-
shows a ‘good’ manual test case. to be more economic but less evolvable. gram). It is important to understand than
the economic benefits of automation will
be achieved only after the automated tests
tests in the long term. Similarly, if no file called an automated test script. When
have been used a number of times.
thought is given to maintenance when it is replayed, the tool reads the script and
tests are automated, updating an entire passes the same inputs on to the software
automated test suite can cost as much, if under test (SUT) that usually has no idea it were first recorded. This implies that as
not more, than the cost of performing all is a tool controlling it rather than a real well as recording the inputs, the tool must
the tests manually. See FEWS99 for case person sitting at a computer. In addition, record at least some of the output from the
histories. the test tool generates a log file, recording SUT. But which particular outputs should
precise information on when the replay be recorded? How often should the out-
For an effective and efficient automated was performed and perhaps some details puts be recorded? Which characteristics of
set of tests (tests that have a low cost but a of the machine. Figure 2 depicts the replay the output should be recorded? These are
high probability of finding bugs) you have of a single test case. questions that have to be answered by the
to start with the raw ingredient of a good tester as the inputs are captured or possibly
test set, a set of tests skilfully designed by For many people this is all that is required (depending on the particular test tool in
a tester to exercise the most important to automate tests. After all, what else is use) during a replay.
things. You then have to apply automation there to testing but entering a whole series
skills to automate the tests in such a way of inputs? However, merely replaying the Alternatively, the testers may prefer to edit
that they can be created and maintained at captured input to the SUT does not the script, inserting the required instruc-
a reasonable cost. amount to performing a complete test. tions for the tool to perform comparisons
between the actual output from the SUT
Believe Capture/Replay There is no verification of the results. How and the expected output now determined
= Automation will we know if the software generated the by the tester. This pre-supposes that the
Capture / replay technology is indeed a same outputs? If the tester is required to sit tester will be able to understand the script
useful part of test automation but it is only and watch each test being replayed he or sufficiently well to make the right changes
a very small part of it. The ability to cap- she might as well have been typing them in the right places. It also assumes that the
ture all the keystrokes and mouse move- in since they are unlikely to be able to keep tester will know exactly what instructions
ments a tester makes is an enticing propo- up with the progress of the tool, particular- to edit in the script, their precise syntax,
sition, particularly when these exact key- ly if it is a long test. It is necessary for the and how to specify the expected output.
strokes and mouse movements can be tool to perform some checking of the out-
replayed by the tool time and time again. put from the application to determine that In either approach, the tests themselves
The test tool records the information in a its behavior is the same as when the inputs may not end up as particularly good tests.