Applied R&M Manual for Defence Systems

Part D - Supporting Theory

CHAPTER 10

RELIABILITY DEMONSTRATION PLANS

CONTENTS

Page

1 Introduction 2

2 Concept and Terminology 3

3 Generation of Fixed Time/Failure Test Plans 6

4 Generation of Sequential Test Plans 8

Version 1.1 Page 1

Chapter 10
Reliability Demonstration Plans

1. INTRODUCTION
1.1 The pass / fail criteria for statistical tests are generally known as plans. Any given
plan is based on parameters agreed between the customer and the supplier.

1.2 Demonstrations are conducted on a sample of the population of a given system or type
of equipment. This may be a physical sample where a large number of items is to be
produced or a time sample where a few systems are tested for a sample period of their lives.
The term ‘population’ denotes those items of which the sample under test is considered to be
representative. The population, other than the sample, may not exist at the time of the test;
in general, the population has still to be produced at the time of the test. Indeed, production
of the current design standard may depend on the test outcome.

1.3 There are two styles of reliability demonstration:

• Where it is required to estimate the value of one or more reliability characteristics i.e.
MTBF, Reliability, etc. for one or more functions or logistical levels for comparison
with a specific value or for use in support planning; or alternatively;
• Where it is required to establish whether one or more reliability characteristic of the
equipment is better, or not, than a specified level, with some stated degree of
confidence.

1.4 The first case is addressed by operating the system and analysing the results as field
data. For planning purposes, it is useful to identify the operational time or number of
failures required. This identification can be achieved by calculating the confidence with
which a set values can be claimed for each characteristic for various combinations of test
time, numbers of failures experienced and actual values exhibited during the test period.
However this process is no longer generally included under the term 'Reliability
Demonstration Testing' and it is not covered further in this chapter.

1.5 In the last few years reliability demonstration testing has come to refer exclusively to
the second case which is, therefore, addressed by this chapter.

1.6 This chapter aims to introduce the reader to the basic concepts involved in choosing a
plan rather than providing a large list of plans. Although some example plans are provided.
more actual plans can be found in other handbooks and standards [1][2].

1.7 The computer application RDEM (sponsored by MOD) also provides assistance in the
choice of plans, see PtACh8.

1.8 Note that although MTBF is discussed, other characteristics can be demonstrated
using these plans. The limitation is that a number of events must be expected with a rate of
occurrence that can be considered to be constant. The plans can be applied to
maintainability demonstrations but only where each discrete test can be considered to be a
trial. The parameter under test must however be converted to a form addressing the mean
time between events; for example failure rate must be inverted to obtain MTBF as the
parameter to be demonstrated.

Page 2 Version 1.1

 Applied R&M Manual for Defence Systems
Part D - Supporting Theory

2. CONCEPTS AND TERMINOLOGY

2.1 General

2.1.1 Reliability demonstrations are hypothesis tests (see Pt4Ch7). The essential first step
is the definition of the hypothesis about the parameter to be tested.

2.1.2 The basic concept of a reliability demonstration is introduced by an example. This

can be found in Leaflet 1 to this chapter.

2.1.3 It is possible to generate several forms of demonstration plan. The two standard
forms are fixed trials/failure terminated and sequential. Sequential plans can be truncated or
non-truncated. Variations can be generated on the standard forms but are not recommended.
Only the two standard forms of plan will be addressed in this chapter. Both of these require
accept-reject criteria to be determined, as shown in Figure 1.

Fixed time / failure test

Sequential test
reject reject
accept
failures

reject
failures

continue accept
continue
accept

time / trials time / trials

Figure 1 - Styles of Test Plans

2.1.4 The two plan styles in Figure 1 are drawn to relate to the same test requirements. It
can be seen that the sequential test has a higher maximum duration, even when truncated,and
a higher maximum number of failures. However it is shown below that the expected time to
completion is shorter if the true value of the parameter under test is equal to or better than the
upper test value.

2.2 Risk

2.2.1 In any test of a parameter addressing random occurrences, there is a risk that a pass
result may be obtained with equipment that is actually outside the specification. Similarly a
fail result may occur with equipment that is actually inside the specification. Thus there is a
risk to both the supplier, of failing the test with ‘good’ equipment, and to the customer, of
passing the test with ‘bad’ equipment. Reducing these risks to zero would require an
infinitely long test and is not practical or financially viable.

Version 1.1 Page 3

Chapter 10
Reliability Demonstration Plans

2.2.2 The Producer’s Risk (α) is defined as the probability1 of rejecting equipment where
the parameter under test is inside the acceptable range. In MTBF terms this is the risk that
equipment is rejected with a true MTBF (that experienced over the whole life of the whole
population) greater than the upper test MTBF (θ0).

2.2.3 The Consumer’s Risk (β) is defined as the probability of accepting equipment where
the parameter under test is outside the acceptable range. In MTBF terms this is the risk that
equipment is accepted with a true MTBF, that experienced over the whole life of the whole
population, lower than the lower test MTBF (θ1).

2.3 Discrimination Ratio

2.3.1 In order to produce a practical test with acceptable timescales and termination criteria
it is necessary to apply the accept and reject criteria to different values of the parameter under
test.

2.3.2 The upper test point ( θ 0) is the value on which the reject criteria are based. This
relates to the minimum design target MTBF from the producer’s point of view.

2.3.3 The lower test point (θ1) is the value on which the acceptance criteria are based. This
relates to the minimum acceptable MTBF from the consumer’s point of view.

2.3.4 The ratio between these two test points is known as the Discrimination Ratio (d):

θ0
d=
θ1

2.3.4 The theoretical minimum value for d is 1. The closer d tends to 1, the more powerful
the test is at discriminating between 'good' and 'bad' items.

2.3.5 In general, larger discrimination ratios produce shorter test times. However the
resulting high MTBF against which the producer risk is based may be unacceptable.

2.3.6 The values of d in practicable plans normally range between 1.5 and 3. A commonly
used value is 2.

2.4 Operating Characteristic

2.4.1 Every set of accept-reject criteria has an operating characteristic (OC) curve associated
with it. This curve identifies the probability of accepting the hypothesis for all actual values
of the parameter. In the ideal case (α = β = 0.0) the curve would be a vertical line from 0.0
to 1.0 at the required value of the parameter. This however requires a 100% sample i.e. the
whole service life and is not practical.

1
The Producer’s risk and Customer’s risk are probabilities. The formulae presented later in this chapter require
a value between 0.0 and 1.0. However it is usual to quote the risk level as a percentage. Hence a fine line
exists in judging which to use in this chapter. Percentages are used when quoting levels of risk and
probabilities in calculations.

Page 4 Version 1.1

 Applied R&M Manual for Defence Systems
Part D - Supporting Theory

Probability of acceptance
1.2
1
0.8
0.6
0.4
0.2
0
0 1 2 3 4
MTBF as multiples of the low er test MTBF

Figure 2 - Operating Characteristic Curve for α = β = 13%, d = 2

2.4.2 Figure 2 shows an example of an OC curve. It can be seen that if the parameter is
really equal to the lower test point then the probability of a success result is α (13% in the
case shown). If the real value of the parameter is equal to the upper test point, then the
probability of a success result is 100% - β.

2.4.3 The actual values of α and β for any given set of acceptance-rejection criteria may
differ from the ideal values originally chosen. This is due to the approximations made to fit
to integer numbers of failures. Hence for the example in Figure 2 (Mil-Hdbk 781, Plan III-
D), the aim is α ≈ β ≈ 10% but in practice α = 12.82% and β = 12.79%.

2.4.4 Further deviation of α and β from the aim occurs when preparing a plan where
discrete success/failure trials are performed. In a trial related test only one failure can occur
in one trial. In a time-related test multiple failures can occur in one time period.
Quantisation effects are largest for small values of the parameter under test in terms of trials.

2.4.5 OC curves present this deviation for all forms of demonstration. However their
generation is resource intensive and knowledge of the general form is often sufficient.

2.5 Expected Test Time

2.5.1 Each plan contains a maximum time in its accept-reject criteria. However the reason
for using sequential plans is to enable an early conclusion to be reached. It is possible to
calculate the expected duration of the test for a range of real values of the parameter under
test. This is most readily done graphically. Taking the representations of accept-reject
criteria in Figure 1 and super-imposing a range of expected performances of the system under
test, as given by its MTBF, enables the expected time to be determined. Figure 3 shows a
line representing the idealised time/failure performance for an MTBF equal to the upper test
MTBF. Different MTBFs are represented by lines of different slopes, all passing through
the origin. The time at which this MTBF line intercepts the termination criteria is plotted in
Figure 4.

Version 1.1 Page 5

Chapter 10
Reliability Demonstration Plans

Fixed time / failure test Sequential test

failures

failures
time / trials time / trials

Figure 3 - Determination of the Expected Test Time

(multiples of lower test MTBF)

25
Expected test time

20 fixed
15

10
sequential
5

0
0 2 4 6 8
True MTBF (multiples of low er test MTBF)

Figure 4 - Expected Test Time Curves

2.5.2 A non-truncated sequential test plan is not shown in Figure 4. Its form can be
appreciated by considering the extension of the two curves for sequential tests without the
two truncation lines. Without this truncation, the two curves would extend to infinite time.

2.5.3 Figure 4 reinforces the earlier point (serial 2.1.4) that a sequential test plan is generally
shorter than a fixed time/failure plan but can be longer when the MTBF is midway between
the upper and lower test values.

3. GENERATION OF FIXED TIME/FAILURE TEST PLANS

3.1 A number of fixed time/failure test plans have been published in documents such as
Mil-Hdbk-781. The practitioner is recommended to select a published plan wherever
possible as this reduces the effort of both generation and review.

3.2 For a time or distance based demonstration it is possible for several failures to occur in
each period of time. The probability of a given number of failures in a given time period is
described by the Poisson distribution, assuming a constant failure rate.

3.3 A fixed time/failure test plan consists of:

Page 6 Version 1.1

 Applied R&M Manual for Defence Systems
Part D - Supporting Theory

• A time for acceptance; and

• A number of failures for rejection.

that are in accord with the required test parameters.

3.4 To address the rejection parameters ( β and θ 1), a set of values for the number of
failures at which to reject (F) and the corresponding times to accept (T) must be derived.
For each value of failure limit there is a time limit if the criteria are to be met such that:
i
F  T 1 − T
β = ∑  e µ
i=0  µ  i!

The resulting range of values F and T given β = 0.1 illustrated in Figure 5; upper line at lower
failures and times. θ1 is immaterial since the scale is in terms of θ1.

30.00
(multiples of the lower test MTBF)

25.00
time termination limit

20.00

15.00

10.00

5.00

0.00
0 5 10 15 20
failure termination limit

Figure 5 - Derivation of a Fixed Time/Failures Test Plan

3.5 Next the process is repeated for the acceptance parameters ( α and θ 0). The accept
criteria is now required but the cumulative Poisson distribution can only be used to derive the
reject probability. There is no simple distribution that is of use here. However the same
result is obtained by calculating the reject probability as before but with θ 0 as the MTBF and
1 - α as the required probability. The resulting range of values for F and T are illustrated by
the second line in Figure 5 (α = 0.1)2.

3.6 The point where the two sets of F and T intercept provides a set of accept-reject
criteria that satisfies both the customer’s and the producer’s criteria. Once this has been
determined, the exact values of α and β should be determined and reported in the test plan
document in order that the deviation from the original requirement can be appreciated.

3.7 The process is very similar for a trials based demonstration. Each trial can either
succeed or fail. The difference is that the resulting probability of a given number of failures
after a given number of trials is described by the binomial distribution:

2
Figure 5 relates to test plan 10-12 and 10-13 in Mil-Hdbk-781, α = β = 0.1 and the intercept is between F = 12
and F = 13 (T = 16.60 and 17.78 respectively).

Version 1.1 Page 7

Chapter 10
Reliability Demonstration Plans

P(m failures in n trials) = nCm . R(n-m) . (1 - R)m

where R is the probability of succeeding on each and every trial and nCm is the number of
combinations of m items in a total of n items. Substituting the cumulative binomial
distribution for the Poisson distribution allows the same process to be followed.

4. GENERATION OF SEQUENTIAL TEST PLANS

4.1 The fixed time/failures test plan has a disadvantage in that it requires the continuation
of a test when the result appears quite obvious well before the termination criteria are reached.
Examples of this include several failures early in the process and no failures before the half
way point. However, for the test to be rigorous, it has to be mathematically correct and
hence if early termination criteria are developed then the alteration to the existing criteria
must be taken into account.
Chargeable failures

h1
A
0
h0 Accumulated operating time / trials

Figure 6 - Generic Sequential Test

4.2 Sequential test methods have been developed which allow for early termination with
both accept and reject results as appropriate (see Mil-Hdbk-781 for references to the history).
This method creates two sets of criteria: one for acceptance and one for rejection. When
illustrated, as in Figure 6, these form parallel lines. Fixed failure and time truncation limits
are also applied to prevent the test extending indefinitely possible when an MTBF between
the upper and lower test points is exhibited.

4.3 A number of sequential test plans have been published in documents such as Mil-
Hdbk-781. The practitioner is recommended to select a published plan wherever possible as
this reduces the effort of both generation and review.

4.4 The parameters defining the accept and reject criteria are their slope(s) and their
intercepts on the failures axis h1 and h0, as illustrated in Figure 6 (s = tan A). The equations
for these lines are:

Reject: fr = s . t + h1

Accept: f a = s . t - h0

Page 8 Version 1.1

 Applied R&M Manual for Defence Systems
Part D - Supporting Theory

the test to continues if: s . t - h0 < f < s . t + h1

4.5 The values of s, h0 and h1 are calculated from α, β, θ0, θ1 using the formulae3:

 β 
ln 
1− α
h0 =
ln(θ 0 ) − ln(θ1 )

 1 − β d + 1
ln . 
 α 2d 
h1 = (d is the discrimination ratio (θ0 / θ1) )
ln(θ 0 ) − ln(θ1 )

1 − 1
θ1 θ0
s=
ln(θ 0 ) − ln(θ1 )

4.6 The failure truncation limit (F) is determined as the smallest integer that satisfies the
requirement that4:

χ 2(1−α ),2 F θ1
≥
χ β2,2 F θ0

If F is not an integer it must be raised to the next highest integer.

4.7 The time truncation limit is calculated using this value:

θ 0 . χ 2(1−α ),2 F
T=
2

4.8 The tables of rejection and acceptance times for each possible number of failures can
then be generated. Table 1, taken from Mil-Hdbk-781 plan III-D, provides an example of
such a table.

3
These formulae are taken from those in Mil-Hdbk-781 and include the correction factor ((1+d)/2d) to reduce
the differences between the consumer’s and producer’s risks arising from test truncation.
4
The chi-square distribution is addressed in PtDCh2 and tables provided.

Version 1.1 Page 9

Chapter 10
Reliability Demonstration Plans

Chargeable Standardised Termination Time

Failures Reject at tr ≤ Accept at ta ≥
0 do not reject 4.40
1 do not reject 5.79
2 do not reject 7.18
3 0.70 8.56
4 2/08 9.94
5 3.48 11.34
6 4.86 12.72
7 6.24 14.10
8 7.63 15.49
9 9.02 16.88
10 10.40 18.26
11 11.79 19.65
12 13.18 20.60
13 14.56 20.60
14 15.94 20.60
15 17.34 20.60
16 20.60 do not accept

Table 1 - Accept-Reject Criteria for α = β = 10%, d = 2

Page 10 Version 1.1

 Applied R&M Manual for Defence Systems
Part D - Supporting Theory

LEAFLET D10/0

REFERENCES

1 BSi Standards: BS5760 Pt 10. Methods of Equipment Reliability

Testing. Issued in several parts at various times.

2 US Department of Defense: Handbook for Reliability Test

Methods, Plans, and Environments for Engineering Development
Qualification and Production. MIL-HDBK-781A. dated 1 April
1996.

Version 1.1 Page 1

Chapter 10, Leaflet 0
Reliability Demonstration Plans

Page 2 Version 1.1

 Applied R&M Manual for Defence Systems
Part D - Supporting Theory

LEAFLET D10/1

RELIABILITY DEMONSTRATION OF A FUZE

1. INTRODUCTION
1.1 This example considers the demonstration of the Reliability of a fuze, as might be fitted
to an artillery shell. This is a system where a number of discrete trials will be performed.

1.2 The aim of the example is to illustrate the way in which the two types of demonstration
might progress. The probabilities of given numbers of failures after a given number of
firings are tabulated for various termination regimes. This provides a practical approach to
understanding the mathematics of the demonstration.

2. TEST PARAMETERS
2.1 First it is necessary to specify the characteristic and test parameters. Let us take the
customer’s point of view and define:

• The minimum acceptable Reliability as 0.75 on each firing; and

• The acceptable customer’s risk as not less than 10%.

2.2 This is sufficient for a procurement specification. The minimum design target for
Reliability and the producer’s risk are then for the producer to decide upon.

2.3 First the lower test point should be redefined in terms of the Mean Firings Between
Failures (MFBF).

Now R = e-λ for each firing (or trial)

∴ λ = -loge(R)

= 0.36 when R = 0.7

or MFBF (θ) = 2.8 firings

2.4 This value for θ 1 provides the basis for defining the upper test point (θ 0). There are
two conflicting drivers affecting the producer’s choice for θ0. A low value is desirable since
it also represents the minimum to be achieved by the design without increasing the risk of
failing the demonstration. However a high value will shorten the demonstration. To
illustrate this point several values will be considered in this example.

5
A low figure is used in order to simplify the example.

Version 1.1 Page 1

Chapter 10, Leaflet 1
Reliability Demonstration Plans

Discrimination Ratio (d) 1.5 2.0 2.5 3.0

Upper test point (θ0) 4.2 5.6 7.0 8.4

Table 1 - Potential Upper Test Points

2.5 The producer’s risk is an arbitrary choice. It is commonly chosen to be the same as
that for the consumer but there is no mathematical requirement for this. Therefore, for the
example, it will be set to 10%.

3. PRACTICAL DEMONSTRATION
3.1 No Termination Criteria

3.1.1 Consider conducting 20 firings when the real MFBF is 2.8. Table 2 shows the
probability of a given number of failures after each firing. Before any firings have taken
place, after 0 firings, the probability of 0 failures is 1. After 4 firings, the probability of 0
failures has reduced to 0.240 and that of exactly 2 failures risen to 0.265.

Table 2 - Probability of 'n' Failures After 'm' Firings When MFTF = 2.8

3.1.2 Now if the real Mean Firings Before Failure (MFBF) were not 2.8 firings but 5.6 then
the probabilities listed in Table 3 would apply. With this more reliable equipment, the
probability of 0 failures after 4 firings is raised to 0.490 and that for exactly two failures
reduced to 0.112. If the probabilities of higher numbers of failures are inspected, lower
probabilities are found for the higher MFBF.

Page 2 Version 1.1

 Applied R&M Manual for Defence Systems
Part D - Supporting Theory

Table 3 - Probability of 'n' Failures After 'm' Firings When MFTF = 5.6

3.1.3 This view of a number of firings needs to have pass/fail criteria superimposed upon it
in order to be a demonstration. This can be done for either a fixed trial/failure terminated
plan or a truncated sequential test plan.

3.2 Fixed Trial/Failure Terminated Plan

Table 4 - Probability of 'n' Failures After 'm' Firings (Fixed Time Test)

3.2.1 If the above tests are terminated then the probabilities are amended at and beyond the
termination criteria. Table 4 shows this for rejection on 4 failures and acceptance after 9
firings, subject to fewer than 4 failures, for an MFBF of 2.8 firings. This can be compared
with Table 2. It can be seen that with this MFBF there is a probability of 0.27 of failing the
test and 0.73 of passing it, the situation of the fourth failure occurring on the ninth firing
Version 1.1 Page 3
Chapter 10, Leaflet 1
Reliability Demonstration Plans

results in a rejection. If the table is recalculated for an MFBF of 5.6 firings, then the
probability of rejection, given the same criteria, reduces to 0.045. These criteria are
therefore likely to be acceptable to the supplier but not to the consumer.

3.2.2 Varying the criteria allows a match for the acceptable risk criteria to be chosen.
Figure 1 shows the way in which the risk reduces with the increase in the length of the test.
The data were generated by extending Tables 2, 3 and 4 and determining the acceptance
criterion, number of firings, appropriate to each potential rejection criterion, number of
failures, for similar consumer’s and producer’s risk. It should be noted that the risks are not
identical since only whole numbers of firings and failures can be selected. This is the cause
of the fluctuations of the lines from a smooth curve.

3.2.3 As can be seen from Figure 1, a short test, reject at 2 failures and accept at 7 firings,
has a high risk of producing an incorrect result, both risks are 32%. However increasing the
criteria to reject at 14 failures and accept at 60 firings brings the risk down to 10% as required
by the specification.

Figure 1: Risk Levels for Different Criteria

3.3 Truncated sequential test plan

Applying the specified criteria with the sequential test method and accept-reject criteria as
given in Table 5, produces Table 6 as a variant of Table 2. Here the truncation has been
chosen as 6 failures or 20 firings for example purposes, the producer’s and consumer’s risks
are too high for the method to be practicable. If the table was extended, it could be seen that
selecting truncation criteria of 19 failures and 85 firings still leaves a probability of 0.02 of
reaching the truncation criteria.

Page 4 Version 1.1

 Applied R&M Manual for Defence Systems
Part D - Supporting Theory

Table 5: Sequential Accept-Reject Criteria

Table 6: Probability of 'n' Failures After 'm' Firings (Sequential Test)

Version 1.1 Page 5

Chapter 10, Leaflet 1
Reliability Demonstration Plans

Page 6 Version 1.1