American Society for Quality

Factorial Sampling Plans for Preliminary Computational Experiments

Author(s): Max D. Morris
Source: Technometrics, Vol. 33, No. 2 (May, 1991), pp. 161-174
Published by: Taylor & Francis, Ltd. on behalf of American Statistical Association and
American Society for Quality
Stable URL: http://www.jstor.org/stable/1269043
Accessed: 09-06-2016 15:35 UTC

REFERENCES
Linked references are available on JSTOR for this article:
http://www.jstor.org/stable/1269043?seq=1&cid=pdf-reference#references_tab_contents
You may need to log in to JSTOR to access the linked references.

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
http://about.jstor.org/terms

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted
digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about
JSTOR, please contact [email protected].

American Society for Quality, American Statistical Association, Taylor & Francis, Ltd.
are collaborating with JSTOR to digitize, preserve and extend access to Technometrics

This content downloaded from 144.32.128.70 on Thu, 09 Jun 2016 15:35:35 UTC
All use subject to http://about.jstor.org/terms
? 1991 American Statistical Association and
TECHNOMETRICS, MAY 1991, VOL. 33, NO. 2
the American Society for Quality Control

Factorial Sampling Plans for Preliminary

Computational Experiments
Max D. Morris

Engineering Physics and Mathematics Division

Oak Ridge National Laboratory
Oak Ridge, TN 37831-6367

A computational model is a representation of some physical or other system of interest, first

expressed mathematically and then implemented in the form of a computer program; it may
be viewed as a function of inputs that, when evaluated, produces outputs. Motivation for this
article comes from computational models that are deterministic, complicated enough to make
classical mathematical analysis impractical and that have a moderate-to-large number of
inputs. The problem of designing computational experiments to determine which inputs have
important effects on an output is considered. The proposed experimental plans are composed
of individually randomized one-factor-at-a-time designs, and data analysis is based on the
resulting random sample of observed elementary effects, those changes in an output due solely
to changes in a particular input. Advantages of this approach include a lack of reliance on
assumptions of relative sparsity of important inputs, monotonicity of outputs with respect to
inputs, or adequacy of a low-order polynomial as an approximation to the computational
model.

KEY WORDS: Computational model; Factor screening; Latin hypercube sampling; Sen-
sitivity analysis.

In recent years the sciences and other disciplines putational experiment" emphasizes parallels with the
have come to rely on computational models as im- classical "physical experiment." The latter activity is
portant tools in many types of investigations. Here, an investigation of the "real world," whereas the
the phrase computational models refers to represen- former uses an invented substitute, but apart from
tations of physical or other systems of interest that this, experimental goals may be quite similar. For
are first expressed mathematically and then imple- example, Currin, Mitchell, Morris, and Ylvisaker (in
mented in the form of computer programs. Mete- press) and Sacks, Schiller, and Welch (1989) dis-
orological phenomena, heat transfer in engineered cussed statistically motivated procedures for pre-
structures, and global economic activity, as exam- dicting the output of a computational model for runs
ples, are modeled mathematically, often in the form not executed.
of large or complicated systems of differential equa- Often, computational models have as many as sev-
tions, and computer programs are written to evaluate eral hundred input variables. Additionally, for com-
the outputs of interest that result from specified in- plicated models, runs of the computer program may
puts. be time-consuming and hence expensive. It follows
For purposes of this article, a computational model that in the early stages of exploring a computational
is viewed as a representation of a function that pro- model a useful experimental activity is the discovery
duces unique values of outputs when executed for of which inputs are important-that is, which have
specific values of inputs. Although the function is a substantial influence on the outputs. "Factor
actually known-for example, a computer program screening" and "sensitivity analysis" are phrases
has been written that includes a complete implicit or commonly used to describe this kind of experiment.
explicit definition of it-it is complicated enough The subjects of this article are the kind of infor-
to defy classical mathematical analysis. In practice, mation that might be useful in a preliminary com-
the behavior of the model is investigated empirically putational experiment and a class of experimental
through a computational experiment, in which a num- plans for collecting that information. Experimental
ber of evaluations-that is, runs of the program at plans are proposed that are composed of individually
different values of inputs-are made and some anal- randomized one-factor-at-a-time designs in the input
ysis is carried out on the results. The phrase "com- variables. For these plans, data analysis can be based

161

This content downloaded from 144.32.128.70 on Thu, 09 Jun 2016 15:35:35 UTC
All use subject to http://about.jstor.org/terms
162 MAX D. MORRIS

on examination of changes in an output that are un- individual interactions cannot be detected using these
ambiguously attributed to changes in individual in- designs. The most obvious remedy to this problem
puts. Since model outputs do not contain random is to use a more extensive design, such as a central
error, the well-known inefficiencies of estimation composite design or a fractional factorial design of
based on linear-models analyses of noisy data from higher resolution, so that all coefficients that may be
one-factor-at-a-time plans are not an issue here. Fur- important in the approximating model can be indi-
thermore, this approach has the advantage of a lack vidually estimated. Baker and Bargman (1985) used
of reliance on tacit or explicit assumptions of rela- composite designs that allow fitting of cubic poly-
tively few inputs having important effects, monoto- nomials to computational models in a few inputs.
nicity of outputs with respect to inputs, or adequacy When k is large, however, even a two-level resolu-
of a low-order polynomial model as an approxima- tion V fractional factorial experiment may be too
tion to the computational model. expensive to perform in a preliminary study.
In Section 1, a brief review of some basic ap- The need to estimate a potentially large number
proaches that have been taken to designing prelim- of linear-model coefficients from a small physical ex-
inary computational experiments is presented. The periment led Satterthwaite (1959) to introduce ran-
basic idea on which the proposed approach is based dom balance designs. The basic strategy of these de-
is outlined in Section 2 and experimental plans, in- signs is to specify the values of each xi randomly and
cluding two examples, are presented in Sections 3 then to investigate the marginal effect of each input
and 5. In Section 4, comparisons between these de- individually, ignoring the values actually selected for
signs and Latin hypercube designs are discussed. the other inputs. He pointed out that under a first-
order model the least squares estimator of each coef-
1. NOTATION AND PREVIOUS APPROACHES
ficient, ignoring the rest, is unbiased; the bias that
Consider a computational model for which an out- would be present were the entire design considered
put y is a deterministic function of k inputs denoted to be fixed is absorbed into the variance of the es-
by xi, x2, x3. . , Xk of collectively by the k-element timator. Satterthwaite's arguments were not fully
row vector x. For given x, the model can be used to accepted for several reasons, however; see the dis-
evaluate y without error; that is, two evaluations of cussion of Youden, Kempthorne, Tukey, Box, and
y at the same x are identical. In this article, y will Hunter (1959) following Satterthwaite's article. A
be scalar valued, although most real computational major argument against their use was their lack
models actually produce several output variables of of efficiency in estimating individual coefficients.
interest. Alternatively, y may be thought of as a sca- A randomzied experimental plan often used for
lar-valued function of several model outputs, such as input screening with deterministic computational
the average of some quantity that changes over sim- models is the Latin hypercube design (Iman and Con-
ulated time. A computational experiment will consist over 1980). Latin hypercube designs were originally
of a collection of n runs, or evaluations, of the model. introduced in the context of a different problem, the
For a given experimental design, the ith row of the estimation of the distribution of an output that prop-
n-by-k design matrix X is the set of input values x agates as the result of a known joint distribution of
for the ith run.
inputs (McKay, Beckman, and Conover 1979). As
One common goal in a preliminary computational in the case of random balance designs, the values of
experiment is the determination of which inputs are each xi are independently generated using a restricted
important and so cannot reasonably be ignored in random procedure. Each xi takes on n unique values
future investigations. It is often necessary that this selected from n nonoverlapping intervals in its do-
be done using a limited number of runs of the model. main of interest.
In both physical and computational experiments, A primary appeal of these randomized plans is
fractional factorial designs of resolution III or IV and that, unlike central composite designs and fractional
Plackett and Burman (1946) designs have been used factorials of given resolution, k does not necessarily
for this purpose. An approximating first-order linear play a role in determining n. Satterthwaite appar-
model can be fit to the resulting data and the im- ently felt that random balance designs were not an
portance of each input assessed by the size of the attractive option for cases in which n > k; see Sat-
associated model coefficient, either in absolute terms terthwaite and Budne (1959). Latin hypercube de-
or in comparison to the size of the residuals. Since signs have been applied, however, in situations in
there are no physical random quantities in this set- which n is much greater than k; Iman and Conover
ting, it is generally difficult to justify such screening (1980) presented an example when k = 8 and n =
rules statistically. If the approximating model is ad- 200 in which the identification of influential inputs is
equate, however, this approach often works well in based on partial correlation coefficients computed on
practice anyway. Of course, nonlinear effects and the ranks of the data.

TECHNOMETRICS, MAY 1991, VOL. 33, NO. 2

This content downloaded from 144.32.128.70 on Thu, 09 Jun 2016 15:35:35 UTC
All use subject to http://about.jstor.org/terms
 PRELIMINARY COMPUTATIONAL EXPERIMENTS 163

In Section 2, a different approach to identifying The intent here is to discover useful information
the important inputs in a computational model about which inputs are important using a design in
is proposed, intended specifically for those cases in which the number of runs is proportional to k (rather
which n is to be of order k and no simplifying as- than k2, etc.). Therefore, it seems reasonable that
sumptions about the form of the model are to be the approach should be based on trying to discover
used.
a fixed number of properties about the influence of
each input on y (rather than individual interactions
2. ELEMENTARY EFFECTS
in an approximating polynomial model whose num-
For purposes of this article, each x, will be assumed ber grows with k). Here it is proposed that the finite
to be scaled to take on values in the interval [0, 1], distribution of pk-l[p - A(p - 1)] elementary ef-
and the region of interest, fI, will be assumed to fects associated with each input be estimated; for
be the k-dimensional unit hypercube. If y is at least input i, this distribution will be denoted by F,. A
once differentiable with respect to each input, 3a(x) large (absolute) measure of central tendency for F,
= ay/lxlx is a functional index of the influence of indicates an input with an important "overall" influ-
x, on y. For example, a,(x) may be exactly or ap- ence on the output. A large measure of spread in-
proximately (a) zero over all values of x, (b) a non- dicates an input whose influence is highly dependent
zero constant over all values of x, (c) a nonconstant on the values of the inputs-that is, one involved in
function of x,, or (d) a nonconstant function of one interactions or whose effect is nonlinear. In partic-
or more xj(j # i). These correspond, respectively, ular, estimates of the means and standard deviations
to situations in which the effect of x, on y is said to of these distributions will be used as indicators of
be (a) negligible, (b) linear and additive, (c) nonlin- which inputs should be considered important. The
ear, or (d) involved in interactions with other inputs. sampling plans to be discussed provide random sam-
The guiding philosophy in this work is that a major ples from each of these distributions on which such
role of a preliminary experiment is to determine, estimates may be based.
within reasonable uncertainty, which inputs may be 3. PLANS FOR INDEPENDENT
considered to have effects which are (a) negligible, RANDOM SAMPLING
(b) linear and additive, and (c or d) other. The reason
for combining (c) and (d) here is that, where im- In simplest form, randomly selecting a value from
portant nonlinearity or interaction exists, an exper- F, requires random selection of a value of each xj(j
iment of any design that is small relative to the num- = 1, 2, 3, . . ., k) and evaluation of y twice, once
ber of inputs will generally not produce enough at the selected values and again after increasing x,
information to resolve the nature of these effects; a by the quantity A; these two runs would then yield
subsequent detailed experiment in only those inputs one elementary effect. This could be repeated r times
whose effects are complicated is the appropriate stage to produce a random sample of r elementary effects
for this study. from F,. If the procedure were performed for each
Here, a "discretized" approach following the pre- input, the result would be a random sample of r
ceding idea is proposed. Attention will be restricted values from each F, at a total cost of n = 2rk runs.
to a region of experimentation, wo, which is a regular In this article, the economy of a design will be defined
k-dimensional p-level grid, where each xi may take to be the number of elementary effects it produces
on values from {0, 1/(p - 1), 2/(p - 1), . .. , 1}. divided by the number of experimental runs, so this
The following discussion will be based on what will sampling scheme has an economy of 2. (I have in-
be called the elementary effects attributable to each tentionally avoided use of the word "efficiency" here.
input. For a given value of x, define the elementary Although economy is closely related to statistical ef-
effect of the ith input as ficiency in the discussion of this section, the rela-
tionship is less direct for the plans of Sec. 5.) Note
di(x) = [y(xl, X2, . . , xi-l,Xi + A,
that under this plan all rk observed elementary effects
Xi+, . . . , Xk) - y(x)]/A, (1) are independently drawn.
More economical designs can be constructed if some
where x E o, except that xi 1 - A and A is a runs are used in computing more than one elemen-
predetermined multiple of 1/(p - 1). (The devel- tary effect. In this development, it will be convenient
opment to follow could easily be generalized to allow to restrict attention to the case in which p is even
assignment of potentially different values of p and A and A = p/[2(p - 1)]; the reason for this will be
to each input. Although some prior knowledge about clear shortly. To generate such designs, the first step
the function y might make this desirable, single val- will be the selection of an m-by-k sampling matrix,
ues of p and A have been used here to simplify the B, with elements that are O's and l's, which has
presentation.) the key property that for every column i = 1, 2, 3,

TECHNOMETRICS, MAY 1991, VOL. 33, NO. 2

This content downloaded from 144.32.128.70 on Thu, 09 Jun 2016 15:35:35 UTC
All use subject to http://about.jstor.org/terms
164 MAX D. MORRIS

. . ., k there are two rows of B that differ only in the entire experiment:
their ith entries; for example,
Bt
X= B*
0 0 0 0 (3)
1 0 0 O0
B*
1 1 0 0
0
(2)
1 1 1 The preceding randomization scheme defines the
. . . . . . .. . . ? o .
* *

probabilities with which elementary effects are se-

1 1 1 ? .' 1 lected with each orientation of B. To characterize
these probabilities, consider the two rows of B which
(In this section m will always equal k + 1; this is differ only in their ith elements,
generalized in Sec. 5.) AB could be used as a design
X1 X2 ' ' ' Xi- Xi Xi+l ' ' ' Xk
matrix (i.e., each row a value for x) for which the B(i) =
corresponding experiment would provide k elemen- X1 X2 ' ' ' Xi- Xi,2 Xi+ ' ' ' Xk
tary effects, one for each input, based on only k +
1 runs. These would not be random selections from and the result of only the first two stages of the ran-

the distributions Fl, F2, . . ., Fk, however. To obtain domization process on these rows; that is, J2.ix* +
a random selection from each distribution, a ran- (A/2)[(2B(i) - J2.k)D* + J2.k]. It is clear that in any
column except the ith, the two elements will be equal
domized version of the sampling matrix is employed
and have one of the values
as follows:

0 + O, l/(p - 1) + 0,
1. Let D* be a k-dimensional diagonal matrix in
which each diagonal element is either + 1 or -1 with 2/(p - 1) + 0, . ,(1 - A) + ,
equal probability. Letting Jm.k be the m-by-k matrix
0 + A, l/(p- 1) + A,
of l's, note that (1/2)[(2B - Jm.k)D* + Jm.k] is an
m-by-k matrix in which each column is either set 2/(p - 1) + A, .... (1 - A) + A,
equal to its corresponding column in B or is deter-
each with equal probability. If p is even and A has
mined by replacing l's for 0's and 0's for l's in the
been chosen to be p/[2(p - 1)], this list of values
corresponding column of B.
2. Let x* be a randomly chosen "base value" of reduces to 0, 1/(p - 1), 21(p - 1), . . . , (p - 2)/
(p - 1), and 1. Similarly, in column i the two un-
x for which each element is randomly assigned a
value from {0, 1/(p - 1), 2/(p - 1), ... 1 - A}, equal values will be 0 and A, l/(p - 1) and 1/(p -
each with equal probability.
1) + A, 2/(p - 1) and 2/(p - 2) + A,... or
- A and 1, each with equal probability. Since ran-
3. Let P* be a k-by-k random permutation matrix
domization is applied independently in each column,
in which each column contains one element equal to
1 and all others equal to 0 and no two columns have each of the pk-'[p - A(p - 1)] = pk/2 elementary
effects for input i has an eqtal probability of selection
l's in the same position, where each such matrix has
an equal probability of selection. following the first two randomization steps. It is easy
to see that the third step in the process does not
Attention is restricted to the case in which each change this, since it is simply an independent shuf-
decision in the randomization process-that is, se- fling of input labels.
lection of values for each element of D* and x* and Although the third step of this randomization pro-
selection of P*-is made independently of all others. cedure is not necessary to ensure equal-probability
Then B* = (Jm,lx* + (A/2)[(2B - Jm.k)D* + Jm,k])P* sampling from each F,, it results in a certain sym-
is called a random orientation of B. Like AB, B* also metric treatment of inputs which may be desirable.
provides one elementary effect per input, but one For example, using only the first two steps in the
which is randomly selected. [If it is desired to assign randomization process, inputs represented by the
values of p, and Ai = pil[2(pi - 1)] to each input, central columns in B from Equation (2) are likely to
step 2 of the randomization procedure would be have a more even distribution of values in X than
modified so that each element of x* takes a randomly those represented by the first and last columns. Add-
chosen value from the appropriate set and step 3 ing column permutation to the randomization pro-
would be modified to permute columns only within cedure averages this potential "lumping" of values
sets corresponding to inputs with equal values of pi across all columns and decreases the probability of
and Ai.] Finally, if a sample of r effects is required a severely imbalanced sample in any column.
from each Fi, r independent random orientations of Over the entire design X, r elementary effects are
B can be concatenated to form the design matrix for produced for each input at a total cost of n = rm

TECHNOMETRICS, MAY 1991, VOL. 33, NO. 2

This content downloaded from 144.32.128.70 on Thu, 09 Jun 2016 15:35:35 UTC
All use subject to http://about.jstor.org/terms
 PRELIMINARY COMPUTATIONAL EXPERIMENTS 165

runs, so the economy of the sampling plan is rk/rm bution with zero mean and unit standard deviation,
= k/m, or k/(k + 1) for the sampling matrix of and the remainder of third- and fourth-order coef-
Equation (2). The price of the increased economy is ficients were set to 0.
nonindependence among samples from different F,; The 21-run, 20-factor sampling matrix of Equation
within a single orientation of B, the elementary ef- (2) was used, and r = 4 random orientations were
fects observed from the different F, have a restricted generated using p = 4 (A = ) to produce an 84-
random relationship. Since only one elementary ef- run design. Based on the 84 computed values of y,
fect is observed from each F, with each randomization a random sample of four elementary effects was ob-
of B, however, the r observations within the sample served for each of the 20 inputs. In Figure 1, the
from each F, remain independent. mean and standard deviation of each sample of el-
Since X produces a simple random sample, with ementary effects are displayed; numbers on the graph
replacement, from each Fi, the sample mean (di) identify the inputs. Two lines are also graphed, cor-
and variance (S2) of the observed elementary effects responding to di = + 2 SEMi; if the coordinates for
for input i are unbiased estimators of the mean and input i lie outside of the wedge formed by these two
variance of F,, and the standard error of the mean lines, one might interpret this, approximately, as sig-
can be estimated as SEMi = Si /r. Since the sample nificant evidence that the expectation of F, is non-
is taken with replacement, two orientations of B can zero.

produce the same elementary effect for a given input, From a practical standpoint, judgment about the
but the probability of this happening is reduced rap- importance of particular values of d, and Si will usu-
idly as k increases relative to r; in the range of most ally be context dependent. For example, d, = 76
practical applications, it may occur very infrequently. would be declared significantly different from 0 by
When it does occur, it can be treated as in any other comparison to SEM,. But in the context of a real
sampling-with-replacement setting; for example, both problem, it is important to decide whether an av-
occurrences of the (same) elementary effect are in- erage change estimated to be 76 units in y due to a
cluded in summary statistics. As an aside, duplication change of . is scaled x, is physically meaningful and
of an elementary effect in a sample implies that at whether a standard deviation estimated to be S, =
least two pairs of rows in X are duplicates; in prac- 24 about this value, as one chooses d1 values ran-
tice, some computational expense would be spared domly from over co, constitutes physically meaningful
because one run of the deterministic model will be variation. In the absence of such knowledge and for
required for each distinct row in X. purposes of this example, we may examine the plot-
ted values of Figure 1 relative to each other to see
3.1 Example
which appear to be most important.
To demonstrate the basic idea of this article, an Inputs 1-10 are clearly separated from the cluster
artificial computational model with 20 inputs was of remaining outputs, which have means and stan-
constructed as follows: dard deviations close to 0. In particular, inputs 8, 9,
20 20 and 10 have mean elementary effects that are sub-
y = () + A /i + iw, wj stantially different from 0 while having small stan-
i <j dard deviations. Considering both means and stan-
20 20 dard deviations together, one might (correctly in this
+ A ij,lwiwW + E wi,jWiWi,, case) conclude that the first 10 inputs are important,
i<j<l i<j<l<s and that of these the first seven appear to have effects
that involve either curvature or interactions.
where wi = 2(x, - 2) except for i = 3, 5, and 7,
For comparison with what might be called a "stan-
where w, = 2(1.1x,/(x, + .1) - '). Coefficients of
dard method," an 84-run Latin hypercube design
relatively large value were assigned as
was also constructed and the corresponding experi-
Ai = +20, i= 1 . . ,10, ment performed using this model. (Actually, a pop-
ular variant of the original Latin hypercube design
A,i, = -15, i,j= 1, ..,6, was used; the randomized values included in each
,./,, = - 10, i,j,/ = 1, ... ,5, column of the design matrix were .5/84, 1.5/84, 2.5/
84, . . ., 83.5/84, rather than being random quan-
and
tities taken from 84 subintervals.) The ranks of the
observed values of y were then fit to a first-order
Afi./., = +5, i,j, l,s = 1,..., 4.
regression model in the inputs; the significance levels
The remainder of first- and second-order coefficients for the model coefficients are listed in Table 1. Six
were generated independently from a normal distri- inputs had effects significant at the .05 level. Not

TECHNOMETRICS, MAY 1991, VOL. 33, NO. 2

This content downloaded from 144.32.128.70 on Thu, 09 Jun 2016 15:35:35 UTC
All use subject to http://about.jstor.org/terms
166 MAX D. MORRIS

80 * I ' ' I* I I ,,1

2 //

60

-4

25
40 -

1 ' , i , i,~~~2~/I
-6 7
%% /,"~~~~~~~~~~~~~~~~~~ / 7
%, //'~~~~~~~~~~~~~~~~
1 1
'
15 1 4,l-c~~~~~~
'",~~~~~~~~~~~~~~~

20 ,o
L 'l't,"'"~~~~~~~~~~~
, I t I ~ I J I ,~~~~~

213 14^'

0
-20 0 20 40 60 80 100

d
Figure 1. Estimated Means (d,) and Standard Deviations (Si) of the Distributions of Elementary Effects in the Example of
Section 3. Lines correspond to d, = ?2 SEM,.

surprisingly, these were all among the truly active often call one-factor-at-a-time (o.a.t.) designs; Dan-
factors and in four cases, were inputs not involved iel (1973) offered an interesting discussion and sys-
in interactions. No coefficients had significance levels tem of classification of o.a.t. designs. They have
between .05 and .10. Input screening based on these historically received less attention than other main-
values alone would result in missing some active in- effects plans in the statistical literature because they
puts (4, 2, and perhaps 5 and 1), or including some lead to least squares estimators of model coefficients
that are inactive (perhaps 20, 13, and 15), depending that are relatively imprecise in settings in which the
on the significance value selected for screening. In response variable contains random error.
Section 4, some of the differences between Latin hy- Other (k + 1)-row sampling matrices exist that
percube designs and those described here are dis- allow calculation of one elementary effect per input.
cussed in more detail. Any such sampling matrix can, by reordering col-
umns and exchanging 0's and l's within some col-
3.2 Selection of a Sampling Matrix
umns, be put in a standard form:
The sampling matrix B of Equation (2) when viewed 0 0 *-0 0
as a design matrix is an example of what statisticians 1 0 0... 0
* 1 0 ... 0
* * 1 ... 0
(4)
Table 1. Significance Levels for Latin Hypercube Design of
Section 3
* * * ... 1
Input p value Input p value
where each asterisk can be either 0 or 1, but such
1 .12 11 .69
that
2 .31 12 .67
3 .01 13 .23
the ith row of B differs from some row
4 .41 14 .75
5 .20 15 .23
6 .01 16 .32
above it in only the (i - l)st element. (5)
7 .0002 17 .96 For a proof of this, see the Appendix. As an aside,
8 .0002 18 .87
Webb (1968) showed that under the usual first-order
9 .0001 19 .81
10 .0001 20 .15 linear model assumptions including random noise, in
the class of designs with design matrices of form (4),

TECHNOMETRICS, MAY 1991, VOL. 33, NO. 2

This content downloaded from 144.32.128.70 on Thu, 09 Jun 2016 15:35:35 UTC
All use subject to http://about.jstor.org/terms
 PRELIMINARY COMPUTATIONAL EXPERIMENTS 167

a necessary and sufficient condition for minimum Property LHS1 is certainly appealing in the sense
variance estimation of the model coefficients is (5). that an exploratory experiment should intuitively
All experimental plans generated by sampling cover the region of interest as completely as possible
matrices satisfying (4) and (5) have the same econ- in the event that some subregion turns out to be
omy, kl(k + 1). The following criterion is suggested particularly interesting. Moreover, if y is a relatively
as a possible aid for selecting a sampling matrix from simple function of the inputs, property LHS1 may
this class. Let the random variable Z denote the num- imply that a useful approximation to y may be con-
ber of runs in X for which a selected input appears structed from Latin hypercube data. The primary
at a selected value. It is easy to show that, with attraction of property LHS2 is that if it turns out that
respect to the three-stage randomization described only a few inputs have important effects, the design
previously, will likely be well distributed in the reduced input
space.
E(Z) = nlp (6)
The designs described in Section 3 do not have the
and
properties just listed, but they do have the following,
Vm - which are not properties of Latin hypercubes:
V(Z) = (2r/pk) E [j - (m/2)]2C(j) OAT1. Randomized sampling matrices allow di-
j=1

rect observation of elementary effects.

+ (m2r/p)[(1/2) - (llp)], (7) OAT2. The projection of each randomized sam-
where C(j) is the number of columns of B containing pling matrix into any subspace of n2 formed by ig-
j l's. Since Equation (6) is independent of the sam- noring some inputs is also a randomized sampling
matrix with one unique run lost for each dimension
pling matrix used, it describes a kind of expected
ignored.
balance, which is a property of each design generated
by the sampling matrices of form (4) and (5). On the Property OAT2 is much weaker than its counterpart
other hand, V(Z) depends on B through the function for Latin hypercubes; reduction to a lower dimension
C, and it seems desirable to minimize this quantity is costly in terms of the loss of unique design runs.
with respect to selection of B. For each value of k If it turns out that several of the inputs have impor-
up to 10, V(Z) was computed for each sampling tant effects, however, property OAT1 guarantees that
matrix in this class, and in each case the matrix which information can be extracted for each input, and that
minimizes V(Z) is the one described in Equation (2). this information (a) is meaningful regardless of the
[For a given value of k, (k - 1)! matrices can be complexity of the input's effect and (b) cannot pos-
constructed that satisfy Equations (4) and (5), but sibly be mistakenly attributed to that input through
this includes large groups of matrices that are equiv- partial aliasing or confounding with effects actually
alent in that one may be transformed to another by associated with other inputs.
reversing symbols in some columns and reordering The following artificial example illustrates a situ-
rows. In the calculation just described, all sampling ation in which the designs introduced here may have
matrices that minimize V(Z) are either of the form an advantage over Latin hypercube designs in the
displayed in Equation (2) or are equivalent to it in input screening context. Define y as follows to have
this sense.] I conjecture that this is also true for larger 100 inputs, where only the first 30 actually influence
k, but I have not proven it. the output:
30

y = E exp(5.5x, - 1.5x),
4. CONTRASTS WITH LATIN i=l

HYPERCUBE SAMPLING
30

As noted previously, Latin hypercube sampling where x = (1/30) E xi.

i=i
(LHS) has become a popular technique for gener-
The heavy curve in Figure 2 shows the cumulative
ating designs for computational experiments. These
designs have several appealing characteristics, in- distribution of Al(x), uniformly sampled over fl. (Since
cluding the following: the first 30 inputs play symmetric roles in y, deriv-
atives taken with respect to inputs 2-30 would be
LHS1. Latin hypercube designs tend to be spread distributed identically; of course, derivatives with re-
fairly uniformly throughout f. spect to inputs 31-100 are 0 throughout fl.) Although
LHS2. The projection of a Latin hypercube de- y is not monotonic in xl, al(x) is positive whenever
sign into any subspace of fI formed by ignoring some its absolute value is relatively large. Since y is a func-
inputs is also a Latin hypercube design with the same tion of only 30% of the inputs, some degree of "effect
number of unique runs. sparsity" (Box and Meyer 1986) is clearly present,

TECHNOMETRICS, MAY 1991, VOL. 33, NO. 2

This content downloaded from 144.32.128.70 on Thu, 09 Jun 2016 15:35:35 UTC
All use subject to http://about.jstor.org/terms
168 MAX D. MORRIS

1.0

0.8

0
_ 0.6
o.j
' 0.4

0.2

0.2

0.0
-100.0 0.0 100.0 200.0 300.0 400.0

d and 6
Figure 2. Cumulative Distribution Functions of Elementary Effects (d,) and Derivatives (a,) for Input 1 in the Example of Sec-
tion 4.

although perhaps not so much as may often be im- ure 3, contours of the joint cdf of d,il and Si are
plied by that phrase. The term involving x intro- graphed. Note that in less than 10% of such samples
duces some interaction among the inputs, and the would both statistics have values less than 20; 75%
exponential form results in strong curvature in the of the time either the absolute value of the mean
effects. would be greater than 25 or the standard deviation
One hundred Latin hypercube designs in n = 303 would be greater than 35. Of course, all samples of
runs were generated, and for each of these the rank- elementary effects associated with inputs 31-100 would
transformed y values were fit to all inputs by multiple contain only zeros.
linear least squares. (As in the example of Sec. 3, Admittedly, this example function is much simpler
each xi took on 303 equally spaced values, randomly than computational codes encountered in practice;
assigned to runs; regression on these input values is for example, all important inputs have the same ef-
therefore equivalent to regression on their ranks.) fect, and all others have absolutely no effect at all.
The importance of each input was assessed by the Yet it presents one situation in which an o.a.t. ap-
usual t statistic associated with the corresponding proach seems much more likely to succeed in screen-
regression coefficient; those with values of 2 or greater ing inputs than the use of rank analysis of data from
were declared to be important. Using this rule, the an LHS. It should be stressed, however, that the
overall proportion of tests that indicated that one of designs described here are not suggested as compet-
the first 30 inputs was important was .634, and the itors for Latin hypercube designs. Indeed, they are
corresponding proportion for inputs 31-100 was .048. unattractive alternatives in the cases for which Latin
Hence on average one might expect to correctly iden- hypercube designs may be best suited-for example,
tify 19 or 20 of the 30 important inputs and incorrectly relatively few important inputs with effects that are
identify 3 or 4 of the unimportant inputs with this not highly nonlinear or interactive. An important
procedure. distinction seems to be that, using a Latin hypercube
For comparison, consider any design of the form design, the effect of any one input can only be as-
discussed in Section 3 in p = 6 levels (A = 3/5) and sessed either by ignoring the other inputs or by ad-
n = 303 runs. For each of inputs 1-30, a simple justing for them using some assumed approximating
random sample of three elementary effects would be model, at least in the rank data. The designs dis-
drawn from a population with cumulative distribu- cussed here are intended for situations in which sev-
tion graphed as the lighter curve in Figure 2. In Fig- eral inputs may have complex effects, and the type

TECHNOMETRICS, MAY 1991, VOL. 33, NO. 2

This content downloaded from 144.32.128.70 on Thu, 09 Jun 2016 15:35:35 UTC
All use subject to http://about.jstor.org/terms
 PRELIMINARY COMPUTATIONAL EXPERIMENTS 169

100.0

80.0 -

60.0 -

40.0 -

20.0 -

0.0
0.0 20.0 40.0 60.0 80.0 100.0

Idl
Figure 3. Contours of the Joint Cumulative Distribution Function of the Absolute Value of the Mean (Id,I) and Standard
Deviation (Si) of a Sample of Three Elementary Effects for Input 1 in the Example of Section 4.

of analysis required by the Latin hypercube design cussed in Section 3. The result is not a simple random
seems risky. sample of size 2 from each Fi, however, because pairs
of elementary effects for the same input are selected
5. PLANS FOR CLUSTER SAMPLING together. As will be discussed, this may be thought
of as a cluster sample (e.g., Cochran 1977) of cluster
In the designs discussed in Section 3, the sampling size 2. So a sampling matrix that provides more than
matrix B was constructed to yield one elementary one elementary effect per input per orientation will
effect for each input. In this section, sampling matri- be referred to as a cluster sampling matrix and the
ces with m > k + 1 rows, which yield more than resulting design as a cluster design.
one elementary effect for each input, are considered. As usually defined, cluster sampling actually refers
These can be constructed for two different reasons, to situations in which the sampling unit is a subset
to increase the economy of the resulting design or of the population of interest and the subsets available
to ensure some degree of separation (in fl) of the for sampling constitute a partition of that population.
sampled elementary effects for each input. The collection of elementary effects for a given input,
sampled as a result of a randomized orientation of
5.1 Cluster Sampling for Increased Economy
B, does not constitute a cluster from the population
Consider first the sampling matrix of interest in this sense. Valid inferences about Fi
can, however, be obtained by treating this collection
0000
1000 as if it were a cluster sample. To see this, let the
0100
0 1 0 0 number of elementary effects observed for each input
B= 1 1 0 0 (8) from one orientation of B be c; in the preceding
1110
1 1 1 0 example c = 2. Moreover, suppose that these sam-
1 1 0 1
1101 pled elementary effects are arbitrarily labeled by the
order in which they can be calculated by examining
B* row by row from top to bottom as di, d], . . .
One random orientation of B yields two elementary dc for input i. Following the arguments of Section 3,
effects for each input at a cost of less than two runs each of the di can be any one of the pk/2 possible
per input, so a design constructed from this sampling elementary effects attributable to input i with equal
matrix has higher economy then the designs dis- probability. This means that, of the pkk! equally

TECHNOMETRICS, MAY 1991, VOL. 33, NO. 2

This content downloaded from 144.32.128.70 on Thu, 09 Jun 2016 15:35:35 UTC
All use subject to http://about.jstor.org/terms
 170 MAX D. MORRIS

probable random orientations of B, 2k! of them lead Table 2. Approximate (large k) Economies for Some
to any particular realization for each di. But no two Designs Generated by Equation (9)
of d!, d, .. ., df can yield the same realization on
Elementary effects Approximate
a given orientation, unless B contains duplicate rows. q Groups (m - 1)/k per input economy
Hence 2ck! of the possible orientations of B lead to
2 1,2 1.50 2 1.33
selection of any given elementary effect. Therefore,
3 1, 2,3 2.33 4 1.71
our sampling plan is equivalent to cluster sampling 5 1, 2, 5 3.20 5 1.56
from an artificial population containing 2ck! com- 5 3, 4, 5 3.20 5 1.56
plete copies of the actual population of interest. But 4 1,2, 3 3.50 6 1.71
since the distribution of this population is the same 4 1,2,3,4 3.75 8 2.13
as that of the population of interest-that is, Fi-
inferences about that distribution can be drawn using
the standard tools of cluster sampling.
5.2 Cluster Sampling for Separation of
One series of designs that yields more than one
Elementary Effects
elementary effect per run can be generated by sam-
pling matrices of form A second reason for using cluster sampling is to
0 0 0 ensure that the elementary effects sampled for each
0
C 0 0 ... 0 input will be separated, to some degree, in fi. For
J C 0 ... O example, each orientation of
B =
J J C (9) 0 0 0
. . .
0 0
1 0 0 0 0
J J J ... C 1 1 0 0 0
1 1 1 0 0
where o is a q-element row vector of 0's, O and J ? . o ? o .

are p x q matrices of 0's and l's, respectively, and 1 1 1 1 1 (10)

C is a p x q matrix of both 0's and l's. In this
0 1 1 1 1
investigation, selection of C has been limited as fol-
0 0 1 1 1
lows. Once a value of q has been determined, allow-
0
*
0 0 1 1
able rows for C are grouped by the number of l's 0
*

they contain; group 1 contains all q-element vectors

0 0 O 0 1
with a single 1 and the remainder 0's, group 2 con-
tains all vectors with two l's and the rest 0's, and so yields c = 2 elementary effects from each F, at a cost
forth through group q. (Group 0, the single vector of m = 2k runs. The economy of designs constructed
of q 0's is not included here because it is redundant.) with this sampling matrix is roughly that of those
A matrix C is then constructed of entire groups of discussed in Section 3, about one elementary effect
rows; for example, either all of group 1 appears or per run. Within each sampled pair, however, the two
no row with a single 1 appears. C is thus a special observed elementary effects represent response dif-
case of a balanced array of strength q. Selection of ferences along opposite edges of a hypercube of edge
these groups determines the value of p and hence length A [rather than, for example, adjacent edges
the row dimension of B.
as in the sampling matrix of Eq. (8)]. In some cases,
The number of elementary effects provided (per this may increase our ability to detect spread in the
input, per orientation) by the sampling matrix of distribution of each Fi. [An alternative justification
equation (9) is might be based on the fact that since B is a foldover
(Box and Wilson 1951) of a resolution III o.a.t. de-
l(1)I(q) + I(i - () sign matrix, each random orientation is a complete
i=2 1 1
resolution IV design.]
where l(i) is 1 if group i is included and 0 otherwise.
In Table 2, sampling matrices from this series are 5.3 Parameter Estimation
listed that yield the indicated number of elementary
The mean of Fi may be estimated from a cluster
effects per orientation with a minimum number of
sample using the same estimator as would be used
runs, where the number of rows in B is less than 4k.
The sampling matrix of Equation (9) has not been with an independent random sample-for example,
the sample average. As discussed by Cochran (1977),
shown to be optimal with respect to economy. This
an unbiased estimator of the variance of F, is
form is tried here primarily because of its similarity
to that of Equation (2). More economical plans may S= [(pkk! - 1)S,a + pkk!(c - 1)S2,]
well exist.
-(cpkk! - 1),
TECHNOMETRICS, MAY 1991, VOL. 33, NO. 2

This content downloaded from 144.32.128.70 on Thu, 09 Jun 2016 15:35:35 UTC
All use subject to http://about.jstor.org/terms
 PRELIMINARY COMPUTATIONAL EXPERIMENTS 171

where S2a and S2w are the among- and within-cluster Table 3. Inputs Used in the TWOLAYER Experiment of
Section 5
mean squares, respectively, from an analysis of vari-
ance of the observed elementary effects for input i. Input number Input description
This is nearly identical to the approximation S2
1 Liquid conductivity of left-layer material
[Sia + (c - 1)S2w]/c. The standard error of the 2 Solid conductivity of left-layer material
mean of elementary effects for input i can be esti- 3 Liquid conductivity of right-layer material
mated as 4 Solid conductivity of right-layer material
5 Liquid specific heat of left-layer material
SEMi = Si,a/rc. (11) 6 Solid specific heat of left-layer material
7 Liquid specific heat of right-layer material
5.4 Example 8 Solid specific heat of right-layer material
9 Liquid density of left-layer material
Next, an experiment is described that was based
10 Solid density of left-layer material
on a cluster design, performed to investigate prop- 11 Liquid density of right-layer material
erties of the computational model TWOLAYER, 12 Solid density of right-layer material
which was developed by Alan Solomon and his col- 13 Latent heat of left-layer material
leagues at the Oak Ridge National Laboratory. 14 Melt temperature of left-layer material
15 Latent heat of right-layer material
TWOLAYER models heat transfer into, out of, and
16 Melt temperature of right-layer material
through a wall containing two layers of possibly dif- 17 Width of left layer
ferent phase-change materials. Heat is applied to the 18 Width of right layer
wall during a 10-hour charge cycle, during which time 19 Charge film coefficient
some of the phase-change material melts. During the 20 Discharge film coefficient
following 14 hours (the discharge cycle), heat is re-
leased from the wall as the phase-change material a design with economy 1.11 and yields two elemen-
solidifies.
tary effects for each input, d, (xl) and d, (x2), for which
In this experiment, effects of k = 20 inputs were x' and x2 differ by an amount A in three elements.
investigated; they are material properties and phys- Note that this sampling matrix is not in the class
ical dimensions of the phase-change materials and discussed in Section 5.1; even though B is of the form
are listed in Table 3. Each input was scaled so that
described in Equation (9), B1 is not a balanced array.
0 and 1 represent 90% and 110%, respectively, of a This particular sampling matrix was chosen as a kind
nominal value. The output investigated was a "utility
of compromise between the two types of cluster plans
index," which takes on values in the interval [0, 1] described in Sections 5.1 and 5.2 in that it generates
and is a measure of the effectiveness of the wall as
a design with slightly better economy than the plans
a heat-storage device. The experiment was per- of Section 3 while ensuring some separation between
formed on a Cray X/MP computer, and each run of the elementary effects within each cluster for each
the model required approximately 20 seconds of ma-
input. r = 3 orientations of B were used, each spec-
chine time.
ifying m = 36 experimental runs, for a total of 108
The sampling matrix used to generate the design runs in the experiment.
is
Results of the experiment are displayed in Figure
0 0 0 0 0 4, where (di, Si) is plotted for each input. Points for
B1 0 0 0 0 which Id,l > 2 SEM, as computed using Equation
J B1 0 0 0 (11) are marked in boxes. For 11 of the 20 inputs,
J J B1 0 0 estimated means and standard deviations were both
J J J B1 0 exactly 0. This is probably due, at least in part, to
J J J J B1 the fact that TWOLAYER is a finite element code
in which the wall thickness is represented by (in this
where
case) 10 discrete nodes; at a particular point in time,
1 0 0 0 if three nodes represent the liquid state and seven
1 1 0 0 represent the solid state, the modeled wall is said to
1 1 1 0 be 30% liquid. As a result, there were only 23 distinct
1 1 1 1 values of y produced in the 108 runs of the model.
0 1 1 1 Although accuracy of the simulation would undoubt-
0 0 1 1 edly be improved by increasing the number of nodes
0 0 0 1 and consequently the amount of computer time needed
for each run, numerical curiosities of this type are
O and J are 7 x 4 matrices of 0's and l's respectively, not an uncommon result of computational experi-
and o = (0, 0, 0, 0). This sampling matrix generates ments.

TECHNOMETRICS, MAY 1991, VOL. 33, NO. 2

This content downloaded from 144.32.128.70 on Thu, 09 Jun 2016 15:35:35 UTC
All use subject to http://about.jstor.org/terms
172 MAX D. MORRIS

0.040 I I I

5 9

0.030 19

11
0.020

0.010 -

I 1 sIA
0.000 I I t 4BnB

-0.10 -0.05 0.00

Figure 4. Estimated Means (di) and Standard Deviations (Si) of the Distributions of Elementary Effects in the Example of
Section 5. Inputs for which Id,l > 2 SEMi are marked in boxes.

Of the remaining inputs, numbers 16 and 17 (melt ment were distinct. On the other hand, if only the
temperature of the right-layer material and width of first two significant digits of y are considered, only
the left layer) have consistently large negative ele- 10 values are distinct in this data set compared to 13
mentary effects. Inputs 1, 5, 9, 11, 12, 18, and 19 in the first experiment.
also have nonzero means, but each has a substantial
standard deviation relative to its mean, indicating 6. SUMMARY AND DISCUSSION
potentially extensive patterns of interaction or cur-
vature in the effects. This is at least partially caused In this article, experimental plans have been de-
by the discreteness of the code noted previously. scribed for the purpose of collecting random samples
For comparison, a second experiment was also run from the distributions of elementary effects associ-
using a Latin hypercube design in 108 runs. Regres- ated with each input. These plans require about one
sion of the rank-transformed response values on the model evaluation per elementary effect sampled
20 inputs yielded an R2 value of .91 and in this case for independent random sampling and can be con-
led to conclusions similar to those just described. The structed with fewer evaluations per elementary effect
inputs associated with model coefficients significant for cluster sampling. For a given fixed sample size
at the .05 level are nearly the same as the nine pre- to be obtained from each F,, these plans then require
viously indicated as important; the differences are a total number of runs, which is a linear function of
that input 18 is not significant at the .05 level and k, as opposed, for example, to resolution V fractional
input 15 is. It is interesting to note that input 18 was factorial plans for which the number of runs required
the single input for which both positive and negative is of order k2.
elementary effects were observed in the first analysis. There are certainly situations, however, in which
Inputs 4, 8, and 18 would also be selected as impor- even this number of evaluations is considered too
tant if a significance level of .10 were used, although large for an exploratory study. When model evalu-
if about 10 inputs really are unimportant, the less ations are extremely expensive or k is very large, it
stringent significance level could easily lead to at least may be important to learn something based on an
one "false positive." The ranges of response values experiment in which n may be considerably less than
observed in the two experiments were similar, about k, even though it must be recognized that this in-
.2 to .4. It is interesting to note, however, that all formation will not be as complete or conclusive as
108 response values in the Latin hypercube experi- what might be desired. In the framework of more

TECHNOMETRICS, MAY 1991, VOL. 33, NO. 2

This content downloaded from 144.32.128.70 on Thu, 09 Jun 2016 15:35:35 UTC
All use subject to http://about.jstor.org/terms
 PRELIMINARY COMPUTATIONAL EXPERIMENTS 173

conventional fractional factorial experiments, Wat- fixed but unknown physical constants, and the input
son (1961) and others developed the idea of group distribution may reflect subjective uncertainty about
screening for instances in which it is believed that their values. Our interest here is in the more fun-
most factors have little or no effect on the response. damental issue of which inputs influence the output
The basic idea is to intentionally confound factors in in an important way, leaving aside questions of how
groups for the purpose of screening. For example, inputs, or our assessments of them, might change
if k factors are completely confounded within groups with time or improved measurements. This philos-
of size 3, a screening design for the k/3 "group fac- ophy is consistent with applications in which codes
tors" may be used to isolate the relatively few im- like TWOLAYER are used; most inputs represent
portant factors, which are investigated more thor- physical characteristics that can be controlled more
oughly in subsequent experimentation. Typically, or less exactly by the experimenter, and the goal is
when the effect due to a group factor is negligible, to identify which of them can be used to adjust im-
it is concluded that the individual constituent factors portant characteristics (outputs of the code) of the
are negligible. Although this may be an incorrect modeled system.
conclusion in some cases, it often effectively isolates An important aspect of real computational models
most of the important factors quickly when inter- not discussed here is the typically large number of
actions are absent, grouping of factors is randomized, output variables produced by a single model run.
and the effect sparsity assumption is correct. A sim- These may represent different modeled characteris-
ilar approach to group screening could be based on tics of the system of interest or values of the same
the designs presented here. If inputs are confounded characteristic varying over simulated time. In fact
in groups of size 3, this leads to three-at-a-time sam- the utility index of the TWOLAYER example is a
pling matrices, which can be constructed in approx- scalar function of a multivariate output representing
imately ? of the original number of runs. Then with the phase of material at different points in space over
some chance of error the list of potentially important a length of time. Such summary functions can often
inputs may be shortened earlier. be used to reduce the cumbersome dimensionality of
It must be admitted that there is something intu- model output while preserving information about the
itively unappealing about an analysis that selectively most important aspects of the model. Alternatively,
"ignores" information. Once x has been selected and an approach of the type suggested here could be used
the resulting y observed, the modeler cannot be blamed to individually analyze several output variables re-
for wanting to use all of the x vector in the analysis. corded in the same computational experiment. More
The suggested practice of looking at sample statistics sophisticated approaches, which could take advan-
from each Fi is clearly not a fully efficient analysis in tage of any relationships existing among output vari-
the sense that it ignores the "location" of the sampled ables, would certainly be useful.
elementary effects in w. When one or more of the The underlying challenge in the input screening
samples from the F, contain unusual values, their problem is that of identifying as many of the inter-
location can be observed after sampling and some esting effects of x on y as possible using as few runs
conclusions may be drawn about the nature of in- as possible. When it is reasonable to assume some
teraction. For example, if the observed d4 values are simplicity in the structure of y-for example, effect
always unusually high when x8 = 1 and x,I = 0, this sparsity and monotonicity-screening methods based
hints at the existence of an important three-factor on Latin hypercube designs or fractional factorials
interaction. This line of thinking is more akin to the are effective and economical. The approach sug-
analysis proposed by Satterthwaite, however, and gested here leads to reliable information about how
shares its weakness of trying to make fine distinctions each element of x influences y, whether properties
about the specific nature of inputs given (relatively) of simplicity are present or not, and is particularly
very little data. useful when an assumption that these properties hold
It should be reiterated for clarity that the problem cannot be justified.
treated here is fundamentally different from the
problem addressed by McKay et al. (1979) in their
ACKNOWLEDGMENTS
introduction of the LHS, in which the primary ob-
jective was the efficient empirical determination of This research was sponsored by the Applied Math-
the distribution of y that follows from a known mul- ematical Sciences Research Program, Office of En-
tivariate distribution of x. This is often appropriate ergy Research, U.S. Department of Energy, through
when inputs represent uncontrolled quantities that Contract DE-AC05-840R21400 with Martin Mar-
vary over time or space, in which case the distribution ietta Energy Systems, Inc. I thank three referees
attached to the inputs may have a frequentist inter- whose comments on an earlier version of this article
pretation. Alternatively, the inputs may represent have led to major improvements in the presentation.

TECHNOMETRICS, MAY 1991, VOL. 33, NO. 2

This content downloaded from 144.32.128.70 on Thu, 09 Jun 2016 15:35:35 UTC
All use subject to http://about.jstor.org/terms
174 MAX D. MORRIS

I am also grateful to Dr. Alan Solomon for providing from either the first or second run by only the values
the TWOLAYER model used in the example of Sec- of a single input. Find such a run, reorder the col-
tion 5. umns 2-k of the design so that input which differs is
the second input, and call this the "third run." This
APPENDIX: A PROPERTY OF MATRICES
process is continued through identification of a "(k
DESCRIBED BY EQUATIONS (4) AND (5)
+ l)st run"; the resulting design will have the prop-
Theorem. A sampling matrix B of m = k + 1 erties described.
rows can provide one elementary effect for each input
[Received September 1989. Revised November 1990.]
iff it can be rewritten in the form of (4) and (5) by
reordering columns and exchanging 0's and l's in REFERENCES
some columns.
Baker, F. D., and Bargmann, R. E. (1985), "Orthogonal Central
Proof. The proof is based on the following ob- Composite Designs of the Third Order in the Evaluation of
servation: If any sampling matrix of m = k + 1 runs Sensitivity and Plant Growth Models," Journal of the American
is divided into two groups of runs, there must be at Statistical Association, 80, 574-579.
least one "split" pair of runs (i.e., one in each group) Box, G. E. P., and Meyer, R. D. (1986), "An Analysis for Un-
replicated Fractional Factorials," Technometrics, 28, 11-18.
that differ only in the value of a single input. To see
Box, G. E. P., and Wilson, K. B. (1951), "On the Experimental
this, suppose that it is not so and that such a sampling
Attainment of Optimum Conditions," Journal of the Royal Sta-
matrix has been partitioned into two such groups tistical Society, Ser. B, 13, 1-45.
without a split pair. Since there are no split pairs, it Cochran, W. G. (1977), Sampling Techniques (3rd ed.), New York:
is possible to exchange 0's and l's for some of the John Wiley.

inputs within one group without changing the num- Currin, C., Mitchell, T. J., Morris, M. D., and Ylvisaker, D. (in
press), "Bayesian Prediction of Deterministic Functions, With
ber of elementary effects which can be obtained from
Applications to the Design and Analysis of Computer Exper-
each Fi. Furthermore, this can clearly be done in iments," Journal of the American Statistical Association, 86.
such a way that some run in the first group becomes Daniel, C. (1973), "One-at-a-Time Plans," Journal of the Amer-
identical to some run in the second group. But then ican Statistical Association, 68, 353-360.
one of these two runs could be deleted without di- Iman, R. L., and Conover, W. J. (1980), "Small Sample Sensitivity
Analysis Techniques for Computer Models, With an Applica-
minishing the number of elementary effects that can
tion to Risk Assessment" (with discussion), Communications
be observed. This is impossible, however, because in Statistics, Part A-Theory and Methods, 9, 1749-1842.
the reduced sampling matrix could be used as a res- McKay, M. D., Beckman, R. J., and Conover, W. J. (1979), "A
olution III design matrix for k factors in fewer than Comparison of Three Methods for Selecting Values of Input
k + 1 runs. Variables in the Analysis of Output From a Computer Code,"
Technometrics, 21, 239-245.
Based on this observation, the theorem can be
Plackett, R. L., and Burman, J. P. (1946), "The Design of Optimal
proven easily by construction. Given any sampling Multifactorial Experiments," Biometrika, 33, 305-325.
matrix of m = k + 1 runs, pick any run and exchange Sacks, J., Schiller, S. B., and Welch, W. J. (1989), "Designs for
0's and l's for some inputs (throughout the design) Computer Experiments," Technometrics, 31, 41-47.
so as to transform this run to a vector of 0's; call this Satterthwaite, F E. (1959), "Random Balance Experimentation,"
Technometrics, 1, 111-137.
the "first run." The preceding observation implies
Satterthwaite, F. E., and Budne, T. A. (1959), "Author's Re-
that there must be at least one run in the remainder
sponse to Discussions," Technometrics, 1, 184-193.
of the design that differs from the first run by only Watson, G. S. (1961), "A Study of the Group Screening Method,"
the values of a single input. Find such a run, reorder Technometrics, 3, 371-388.

the columns of the design so that input that differs Webb, S. R. (1968), "Saturated Sequential Factorial Designs,"
Technometrics, 10, 535-550.
is the first input, and call this the "second run." Now
Youden, W. J., Kempthorne, O., Tukey, J. W., Box, G. E. P.,
the observation implies that there must be at least
and Hunter, J. S. (1959), "Discussion of the Papers of Messrs.
one run in the remainder of the design that differs Satterthwaite and Budne," Technometrics, 1, 157-184.

TECHNOMETRICS, MAY 1991, VOL. 33, NO. 2

This content downloaded from 144.32.128.70 on Thu, 09 Jun 2016 15:35:35 UTC
All use subject to http://about.jstor.org/terms