2 IEEE TRANSACTIONS ON RELIABILITY, VOL. R-27, NO.

1, APRIL 1978

Taming the All-Equipment Reliability Test

Anthony Coppola, Senior Member IEEE 781C and since the 2-line version is now in common use,
we will consider only the 2-line version. The test plan
requires the testing of all equipments in a production run
Key Words-Reliability testing, Production reliability, All-equipment rea tatestinteral ofmts ime prountilures
for a stated short Interval of test time per unit. Failures
reliability test
are plotted against time (see Fig. 1). Two lines are drawn
Reader Aids- on the chart; the upper line (the y-axis represents failures)
Purpose: Widen state of the art. is a reject line and the lower a boundary line. If the plot
Special math needed: Elementary statistics. of failures vs. time penetrates the reject line, the equip-
Results useful to: Reliability engineers and program managers. ment is considered unacceptable and acceptance halted

Summary & Conclusions-As currently used, the all-equipment rel- until corrections are made (in principle anyway). If the
ability test provides unequal risks favoring the producer for short test plot reaches the boundary line, it continues along the
periods and favoring the consumer for long test periods. This paper de- boundary line until another failure raises it above the line.
scribes the empirical derivation of a method for modifying the test to The reason for this is to assure that any drop in production
provide equal producer and consumer risks over a range of test lengths. quality will quickly result in a reject decision, since the
With the formulas provided in the paper, the user can structure an all-
equipment test with equals-risks for discrimination ratios of 2.0: 1, 1.5:1,
or 3.0:1, as desired.
plot of failures vs. time is never firther from rejection
than the distance from the boundary line to the reject line.
Though the equations for the two lines could be ar-
bitrarily derived, the general method has been to use for
INTRODUCTION the reject line the same equation as the reject line in one
of the sequential test plans of MIL-STD-78 1. The bound-
A critical consideration in invoking reliability tests is ary line is drawn parallel to the reject line a given distance
the s-risks involved. Each test plan has a s-risk of rejecting below and could be (but is not necessarily) the same as
'good' equipment (the producer risk) and a s-risk of ac- the accept line in the sequential test plan.
cepting 'bad' equipment (the consumer risk). For a rea-
sonable application, these s-risks must be known and ac-
ceptable to both parties, for quantified values of 'good' DIFFICULTIES WITH THE TEST
and 'bad'. As used here, 'good' equipment is that having
a MTBF equal to the specified MTBF (0o) as defined in Since the test is never more than a given number of
MIL-STD-781B, and 'bad' equipment is that having failures away from rejection, a degradation in production
MTBF equal to the minimum acceptable value (01). In
MIL-STD-781C, these definitions are changed, but 01 is
still the lower test MTBF and 0o, the upper test MTBF.
Most of the reliability test plans in MIL-STD-781B have
well defined producer and consumer risks; in fact, they
are designed to establish desired s-risks based on the ratio C ACCEPT
of Go to 01. A notable exception is the all-equipment REJECT REGION ANDD
reliability test, Test Plan XXIX of MIL-STD-781B and
its improved (2-line) version reported in the literature 4 TESTIN
[1-3], used in current military contracts [4] and now in- ° R TEOTN
corporated into MIL-STD-781C. The risks provided by ,
the test are a function of its length and the test can be 4
extremely easy to pass (low producer risk, high consumer 2 / \3/
risk) or unreasonably stringent (high producer risk, low < PLOT NOT ALLOWED TO
consumer risk). This paper describes away to modify the ~ /% ENTER THIS REGiON
test to provide approximately equal producer and con-//
sumer risks over a range of test lengths.//
DESCRIPTION OF THE TESTI lII
Since the 1-line all-equipment test described in MIL- TEST TIME
STD-781lB is superseded by a 2-line version in MIL-STD- Fig. 1 All-equipment reliability test
U.S. Government work not protected by U.S. copyright.
COPPOLA: TAMING THE ALL-EQUIPMENT RELIABILITY TEST 3

quality can result in a quick rejection decision. However, Table I

if the test length is short, even poor equipment may not Test Time (Multiples of e1)
accrue enough failures to reach the reject line before the 5 10 20 40 80
test ends. On the other hand, there is always a finite - - - - 2
probability that satisfactory equipment will accrue suf- CrodsucmerRisk .55 .39 .12 .02 .26
ficient failures (by statistical chance) to cover the distance
between the boundary line and the reject line. This prob- Notes:
lme but as Producer
e Risk I-Probability of accepting an equipment with MTBF
althe test lnegth
length grows become buntithe
ability is negligible for any small amount of time,
the teSt grows, it becomes appreciab
appreciable until
=
the Consumer Risk Probability of accepting an equipment with MTBF
point is reached (for long tests) where even good equip- = e1.
ment has little probability of passing. This ought to con-
cern the consumer as well as the producer because re-
jecting good equipment is against the best interest of both
parties. Example: An all-equipment test, using the for- As Table I indicates, the shorter test periods favor the
mula T = -3.46 + 1 .386F to establish the reject line and producer and the longer the consumer (although, as stated
the formula T = 4.40 + 1.386F to establish the boundary above, the consumer ought not be pleased at high prob-
litne
The test rejecting
abilities is exceptionallyequipment).
of good
poor in the shorter lengths.
While short tests will always have greater s-risks than
T test length in multiples of 01 longer tests, the producer and consumer ought at least
F number of failures accept equal risks. Hence, a modification to the test plan
equalizing the s-risks is warranted.

These formulas are identical with those of the reject

and accept line of Test Plan III of MIL-STD-781B, a DERIVATION OF THE NEW TEST
sequential test providing 10%o producer and consumer
risks for a discrimination ratio of 2.0 (00/0, = 2.0). Equiv- The recommended new test plan is based on the hy-
alent forms are F = 0.72T + 2.50 for the reject line and pothesis that equal s-risks could be obtained by varying
F = 0.72T - 3.17 for the boundary line. the distance between the reject and boundary lines as a
Figure 2 presents the operating characteristics of the function of test length. This would be implemented by
test for total test lengths (number of equipments multiplied replacing the constant in each equation with a function
by time on each equipment) of 5, 10, 20, 40, 80 multiples of test length.
of O0, and Table I converts this information to the pro- The procedures for deriving the function were strictly
ducer and consumer risks. empirical.
1. The equations for a sequential test with nominal
10% risks and a 2:1 discrimination ratio were obtained
from the original Wald formulas. [7] This was done be-
1.0 cause the formulas used for Test Plan III of MIL-STD-
9 .-/ ,-s 781B were themselves modifications of the original for-
TOTAL TEST TIME mulas. (The reject line was moved to compensate for the
.8 (MULTIPLES OF 9) - /10 /20 j 80 effects of the truncation point on the risks). While not
7 strictly necessary, it was felt that the use of the original
.6 1 / /// formulas was preferable, if for no other reasons than that
the original formulas provided equations for the reject
> 5 / / DISCRIMiNATION RATIO= 2.0:1 and boundary line differing only in the sign of the constant
-m 41 / / / / / (IMPLIED BY USE OF TEST PLN m) term, which simplifies both the derivation and the use of
m / / I/ / the new test plan. The equations are T =4.394 + 1.386F
X .3 (or F = 3.170 + .7211).
2+// / / / ~~~~~~~~~2.
For test lengths of 5, 10, 20, 40, 80 multiples of 01,
/ / // ~~~~~~new values were found for the constant which would
T ,/, , ~~~~~~~~resultin approximately equal producer and consumer
L WW l , , I , l ~~~risks. This was done by trial and error, substituting con-
.5 1.0 1.5 2.0 2.5 3.0 stants into the equations and running a Monte Carlo sim-
TRUE MTBF ( MULTIPLES OF 61) ulation program to determine the producer and consumer
Fig. 2 Operating characteristics of the all-equipment reliability test risks resulting from the modified formulas. Results are
based on test plan III presented in Table II.
4 IEEE TRANSACTIONS ON RELIABILITY, VOL. R-27, NO. 1, APRIL 1978

Table II 5. The preceding steps were then repeated to derive

Empirically Derived Constants equations for equal risk all-equipment tests for discrim-
Test Length (Multiples of 01 ination ratios of 1.5:1 and 3.0:1. The Wald formulas for
5 10 20 40 80 sequential tests with 10% risks were the starting points,
Constant 1.6 2.3 3.5 5.4 8.3
and the final formulas were:
Producer Risk .27 .21 .13 .05 .01 T = 0.673(TL)0536 + 1.216F, for a discrimination ratio
Consumer Risk .25 .18 .12 .05 .01
of 1.5:1,
3. From the constants shown in Table II, a 'least T = +0.614(TL)0633 + 1.648F, for a discrimination ratio
squares' fit was made to the formula C = Cl (TL) C2: of 3.0:1
C function desired Equivalent forms are:
C1, C2 constants
TL test length in multiples of 01. F = +0.553(TL)0536 + 0.455T,
F = ±+0.373(TL)° 633 + 0.226T, respectively

The function is strictly empirical, but is tractable and Operating characteristics for these are given in Fig. 4&5.
represents the simplest function with a potential of sat-
isfying the need. C1 = 0.594 and C2 = 0.598. APPLYING THE NEW TEST
4. The Monte Carlo simulation was repeated for the
same range of test lengths using constants calculated from To apply the test, the user ought to select an all-equip-
the derived function. The results are shown in Table III ment test based on the same discrimination ratio as the
test he uses for reliability qualification. He then applies
Table III
Constants and Risks Using C = 0.594 (TL)0I598 the appropriate equation from the three provided above
to draw his reject and boundary lines. He must specify
5 T l 20 40 80 the test time required on each unit. From this, and the
number of units produced, he can determine his risks
Constant 1.56 2.35 3.56 5.39 8.16 using Figures 3-5 as appropriate. During the test, failures
Producer Risk .29 .21 .13 .05 .01 are plotted against time, where time is in multiples of 01.
Consumer Risk .27 .19 .11 .05 .01 Crossing the reject line indicates a production reliability
problem to be resolved. The plot is not allowed to cross
Comparing Table III with Table II shows the least the boundary line but follows the line until a failure occurs.
squares derived formula gives results comparable to the The only difference in this procedure and the existing 2-
constants derived by trial and error. A full set of operating line all-equipments test, is the use of the formulas derived
characteristics is provided in Fig. 3. above to establish the reject and boundary lines. This

1.0 1.0
.9 .9
.8O t TOTAL TEST TIME= 80 TOTAL TEST TIME=
83 T (MULTIPLES OF 81 ) ( MULTIPLES OF 86) //
.7 .7
LU LAJ
, .6 + ;6
/t ,r~~~~~~~~~~~~~~~TEST BOUNDRIES RAT.06.51
DISCRIMINATION
DETERMINED
.5 > .5 BY T=±.673(TL) *537 + 1.216F
DISCRIMINATION RATIO= 2.0:1 _//
< .4/ TEST BOUNDARIES DETERMINED 4
~ 3 /* BY Tt .594(TL) 9..386F ~ 3
= 03 < 1 - ,2.t, ,98 1 c

.5 1.0 1.5 2.0 2.5 3.0 .5 1.0 1.5 2.0 2.5 3.0
TRUE MTBF ( MULTIPLES OF 81 ) TRUE MTBF ( MULTIPLES OF 8, )
Fig. 3 Operating characteristics of modified all-equipment test plan Fig. 4 Operating characteristics of modified all-equipment test plan
 COPPOLA: TAMING THE ALL-EQUIPMENT RELIABILITY TEST 5

1.0 + Anthony Coppola; RADC/RBRT; Griffiss AFB; Rome, NY 13441 USA.

.9+ 4/20 10 Mr. Anthony Coppola (M'57, SM'77) was born in Brooklyn, New York
.8 TOTAL TEST TIME = 80 / / on 1935 Jul 14. He holds a Bachelor's Degree in Physics and a Master's
MULTIPLES OF 81 ) / / / - Degree in Engineering Administration, both from Syracuse University.
X 7 He is Chief of the Reliability and Maintainability Engineering Section,
o ; // Rome Air Development Center, Griffiss AFB and is responsible for the
c .6 development of reliabilitv and maintainability engineering techniques.
5 - tt DISCRIMINATION RATIO=3.0:1 He has published rnore than a dozen technical papers on the methods
_ X TEST BOUNDARIES DETERMINED and management of reliability and maintainability. He has been a mem-
m 4+ //z| BY T-+.6l4(TL)B33l 1.648F ber of the faculty of the Air Force Institute of Technology School of
m / Systems and Logistics and of the Staff of the George Washington
c 3 University School of Engineering and Applied Science. He is a past
.2 - //// /
2 chairman of the IEEE Mohawk Valley Section and the organizer and
first chairman of the Mohawk Valley Chapter of the IEEE Reliability
.1 / ,1 Group.
'//
-

.5 1.0 1.5 2.0 2.5 3.0 Manuscript received 1977 Jun 18; revised 1977 Oct 5. rn
TRUE MTBF ( MULTIPLES OF 81 )
Fig. 5 Operating characteristics of modified all-equipment test plan Reply to Referee Comments & Questions

difference, however, keeps the producer and consumer 1. Q. The reject line of Test XXIX is most important
risks approximately equal regardless of the test length. to both the producer and consumer, so that QC difficulties
This procedure will not, of course, solve all the diffi- can be detected and corrected early. It is not the intent
culties
cultieswith the all-equipment test. The
wihtealeupetts .
s-risks in
sik i the
h of MIL-STD-781B to demonstrate reliability in Test
XXIX. Rather it is to be used when inadequate QC or
XXX ratit
shorter test periods, though now equal, can still be too istbe usedw
high. For such cases, there is no recourse except to
lengthen test time, if equal s-risks are to be preserved.
On the other hand, the s-risks are very low for long tests
* one may wish to reduce the test time * forr economic*
A. Tet planuXXIXoto
~~~~~~~~test
btapouto reliabilit ation
isnot asterfresae.Hw
and
reasons, by testing every other unit, for example. The
resos btetnevroteuntfoexml.
,
bu a productereecstaes.rHow-
ever, like all statistical tests, producer and consumer risks
e

involved. II contend that they should be equal, pre-

are ioe
new procedure will, at least, provide that any test lengtn
selected will provide equal s-risks, preventing favoring
h are ont at theysh be eal, pre-
suming both are reasonable figures (i.e., the valueb of 0
the producer or consumer even if blindly applied by in- is about as high as one could expect with a good, but not
experienced personnel. Sic
exerene pesne. Since no
not evr
every prgami
program is extra-ordinary reliability effort and 01 is the lowest value
blessed with trained reliability specialists, this alone could which is operationdlly and economically acceptable).
be an important benefit.
2. Q. Par. 4.2.9.4 indicates that the MTBF can be eval-
uated at any time during the contract by totalling the
REFERENCES operating time and the failures. Corrective action will be
taken in accordance with par. 5.4.8.5 when the data justify
[1] R. Sesson, R. Neathammer, "An improved all-equipment reli- such action. The procuring agency will negotiate for the
ability test plan", Evaluation Engineering, 1969 Sept/Oct, pp 6-8, required corrective action. The producer takes no risk
50. except the disapproval of the customer, unless penalties
[2] R. Sesson, "Improved fixed length reliability qualification tests", are part of the contract. If penalties are part of the con-
Evaluation Engineering, 1972 May/Jun, pp 4, 6, 7, 40.
[3] R. Sesson, M. Anderson, G. Lindsay, E. Thomas, "Reliabil- tract, the producer wll make sure that his discrlmlnation
ity/maintainability demonstration verification", Evaluation En- ratio is large and that the test-time is reasonably long to
gineering, 1973 May/Jun, pp 10-13, 31. minimize his risk, before signing the contract.
[4] Aeronautical Systems Division Supplement I to Air Force Reg-
ulation 80-5, 25 Sept 1970, Wright-Patterson AFB, ASC/ENES, A. The producer cannot always "make sure his dis-
Ohio 45433 USA.
[5] MIL-STD-781B, "Reliability Tests: Exponential Distribution" crimination ratio is large". If he could, it would imply he
15 Nov 1967, Naval Electronics Systems Command, Elex 4702, can design to any value of 00 he desired, which is the-
Washington D.C. 20360 USA. oretically possible but can be a practical impossibility. In
[6] Draft of MIL-STD-781C, Naval Electronics Systems Command, a competitive environment, he will not willingly commit
Elex 4702, Washington D.C. 20360. USA. (Now published as a resources above normal for reliability effort. If the request
Standard and supercedes-781B.)
[7] RADC Reliability Notebook, 31 Dec 1961, Rome Air Develop- for proposal fixes the length of the test he may not wish
ment Center, RBRT, Griffiss AFB NY 13441 USA, Section 5 to take exception to it. (Indeed, he may not realize what
[8] ARINC Research Corp, Publ. No. 123-7-196. his s-risks are when writing his proposal.) Penalties are
6 IEEE TRANSACTIONS ON RELIABILITY, VOL. R-27, NO. 1, APRIL 1978

not the only consideration. We call the risk of rejecting 7. Q. Figure I, if used for a reliability test, can be de-
'good' equipment a producer risk, but it is also against veloped in accordance with [8] for equal risk and dis-
the best interest of the consumer. crimination ratios. However, this is a Sequential Test Plan
rather than a fixed-time-test.
3. Q. Test XXIX is a fixed-time as specified in the con- A. Again, the sequential formulas are the starting
tract and is not intended for use in the Reliability Qual- point for the test, but the test is a fixed length test (length
ification (Demonstration) Phase. determined by number of units produced times time on
A. I do not suggest using Test Plan XXIX as a qual- each unit). My source of the sequential test formulas was
ification test. I am only trying to get it to do its function [7], which are identical to the ARINC formulas [8].
of production control without skewing the s-risks either
in favor of the producer or consumer. 8. Q. A fixed-time-test can best be evaluated at any time
during the test by a point estimate from the chi-square
4. Q. Equal risks appear unnecessary for this test. Since distribution, knowing the total-test-time and observed
the test-time is part of the contract, the producer must number of failures.
design sufficiently above the specified MTBF to minimize A. This is irrelevant if the all-equipment test contin-
his risk of rejection if the test is contractually binding. ues until it hits a reject or all equipments are produced
A. Equal s-risks should always be necessary if the 0e and passed.
& 0, values are set realistically. If the customer has 9 Q. The longer the test, the less is the probability of
enough design margin to easily design above 0e, that value
has been made too low. In any event, if 00 describes r
the lod eq tent. test iX the 29l of
'good' equipment (in the customer's opinion) it should shows producer risk approaches 25%.
have a low chance of rejection. I feel we must make Oo A. My discussion is not on Test Plan XXIX in MIL-
& 01 realistic numbers, rather than rely on the contractor STD-781B but its 2-line replacement. The producer risk
having latitude to design above an arbitrary value of Oo, in Test Plan XXIX is lower than 25% for short tests and
if we are to be using an engineering discipline rather than increases as test-length increases. (See OC curves in MIL-
a numbers game. STD-781B.)

5. Q. The proposed test seems to be a variation of Se- 10. Q. The test time is specified in the contract and is
quential Test Plan III where total-test-time is in multiples undoubtedly limited only by the consumer's funds and
of specified MTBF (00). schedule.
A. The new test is derived from the sequential test A. Quite so. That is why the test should provide equal
formulas as are Test Plan XXIX and the 2-line version s-risks for all reasonable test periods.
in MIL-STD-781C. I am using multiples of 01 because
the new MIL-STD-781C will also be using multiples of 11. Q. Are you developing a new Sequential Test Plan
0, rather than of Oo as the current standard does. or a Fixed Time Test? The discrimination ratio appears
to be the variable here.
6. Q. What purpose does the boundary line of Figure A. We are developing a fixed time test with bound-
I serve? aries set by modifications to the sequential formulas. The
A. The boundary line keeps the curve within a given variable is test length.
number of failures from reject. This prevents the situation 12. Q. See Figure 2. The test period (par. 4.2.9.2, -78 iB)
where a good equipment strays far from the reject line .
b specified
can be as small as 1/4 of the MTBF (0o). If the
and then goes bad but cannot rapidly reach the reject line discrimination ratio were 2, then the test time would be
The boundary line permits more rapid detection of pro- 0.5 01. Is it true that a minimum of 10 equipments would
duction problems, but also can provide (in long tests) an be required here?
undesirable risk of a good equipment reaching a reject
point by chance. My modification to the test gives the A. I do not recommend time and sample combinations
best of both worlds by keeping a boundary line but chang- which provide high s-risks. One would increase test-time
ing its distance from the reject line sufficiently to reduce per unit to create reasonable risks but with a small number
the risks of rejects caused by chance rather than by pro- of samples, production control could perhaps be assured
duction degradation. I did not invent the boundary line, by a standard test (fixed time or sequential) applied to a
My references discuss its origin. It is used by the Air sample mid way in production rather than to all equip-
Force and is the plan in MIL-STD-781C in lieu of the ment. This is a tradeoff which the program office should
present Test Plan XXIX. make.