In Vitro Cell.Dev.Biol.

—Plant (2016) 52:547–562

DOI 10.1007/s11627-016-9786-1

INVITED REVIEW

Design of experiments (DOE)—history, concepts, and relevance

to in vitro culture
Randall P. Niedz 1 & Terence J. Evens 1

Received: 11 April 2016 / Accepted: 25 August 2016 / Published online: 12 September 2016 / Editor: John Finer
# The Society for In Vitro Biology 2016

Abstract Design of experiments (DOE) is a large and well- Introduction

developed field for understanding and improving the perfor-
mance of complex systems. Because in vitro culture systems It Started with Fisher BThe analysis of variance is not a
are complex and easily manipulated in controlled conditions, mathematical theorem, but rather a convenient method of ar-
they are particularly well-suited for the application of DOE ranging the arithmetic^ (Ronald A. Fisher restated in Gaither
principles and techniques. Successful use of in vitro technol- and Cavazos-Gaither 2012). Francis Bacon articulated the ba-
ogies in horticultural, plant breeding, or genetic applications sic form of the scientific method used today in his Novum
typically involves improving some aspect of a system’s Organum Scientiarum (Bacon 1650), New Method of
growth response—organogenesis, somatic embryogenesis, Science; the title indicated that his work was a replacement
metabolite biosynthesis, or responses required for crop im- for Aristotle’s Organon. Bacon did not use the term
provement such as ploidy manipulation, embryo rescue, cre- Bexperiment^ in quite the same sense as used today, i.e., as a
ation and manipulation of chimeras, somaclonal variation, and highly structured ordering and analysis of factors and
mutant isolation. How and why DOE is the appropriate re- variables. Schwarz (2012) explained that Bacon used the
search approach for developing and understanding in vitro terms Bexperientia^ and Bexperimentum^ to mean Bthe un-
systems research is explained. The presentation is a narrative forced observation which we might call experience^ and Bthe
of the historical context and the geometric basis of DOE to contrived experience which we might call an experiment^,
explain the underlying concepts. Examples illustrate the use of respectively. The principles and methods of how to actually
DOE in in vitro plant culture research. conduct Bacon’s Bcontrived experience^ would be formulated
260 years later by a young Cambridge-educated mathemati-
cian, Sir Ronald Aylmer Fisher, working at a private agricul-
Keywords Experimental design . Scientific method . Plant tural research station. Fisher would call these principles and
tissue culture . Mineral nutrition . Ion-specific effects the associated methods the Bdesign of experiments^ (DOE).
Fisher explored and developed the field of DOE single-
handedly and published the first books for researchers on ex-
perimental statistics (Statistical Methods for Research
Workers; Fisher 1925) and design of experiments (The
Electronic supplementary material The online version of this article
(doi:10.1007/s11627-016-9786-1) contains supplementary material,
Design of Experiments; Fisher 1937). How were experiments
which is available to authorized users. conducted prior to Fisher? Salsburg (2001) discusses a couple
of characteristics of science during the period between Bacon
* Randall P. Niedz and Fisher. First, because Bacon did not discuss how to actu-
[email protected] ally conduct an experiment, there was no established method.
Experiments were Bidiosyncratic^ to each scientist. BGood
1
U.S. Department of Agriculture, Agricultural Research Service, U.S.
scientists^ constructed Bexperiments that produced new
Horticultural Research Laboratory, 2001 South Rock Road, Fort knowledge. Lesser scientists … engaged in ‘experimentation’
Pierce, FL, USA that accumulated much data but was useless for increasing
548 NIEDZ AND EVENS

knowledge^ (Salsburg 2001). Second, experiments were not

published. What scientists published were their conclusions.
Papers included a sample of observations sufficient to
Bdemonstrate^ a conclusion, along with some descriptive sta-
tistics. For example, BGregor Mendel did not show the results
of all his experiments in breeding peas. He described the se-
quence of experiments and then wrote: ‘The first ten members
of both series of experiments may serve as an illustration....’^
(Salsburg 2001). Thus, the reader did not have access to the
complete results of an experiment.
In 1843 John Lawes, the owner of a factory that produced
artificial fertilizers, established a research farm at his
Rothamsted Estate in Harpenden, UK, for the purpose of de-
termining the effects on crop yields of inorganic and organic
fertilizers. To determine the effects of different fertilizers,
Lawes decided Bto make experiments at once more systematic
and on a larger scale^ (Johnston and Garner 1969). The meth-
od used to evaluate various fertilization treatments was to
divide a large piece of land into strips, with each strip contain-
ing a different treatment (Fig. 1). The same treatments were
applied to the same strip every year, and yield and rainfall data
were collected. Treatments were not randomized, replicated,
or blocked.
Fisher integrated the fundamental DOE concepts of ran-
domization, replication, and blocking into a single unified
approach that he called the Bdesign of experiments^. By
1919 Rothamsted had nearly 70 years of data, but could not
make sense out of it because of the large amount of variation
within the field and the differences in yearly rainfall.
Figure 1. Wheat crop in Broadbalk field at Rothamsted Research in
Variability obscures and hides effects, but is an inherent char- Harpenden, UK. Established in 1843, Broadbalk is one of the longest-
acteristic of agricultural and biological systems. Perhaps this running agricultural field experiments. It was data from Broadbalk, an
is why DOE developed out of agriculture and not the more unreplicated field experiment, that Fisher attempted to analyze and that
basic sciences of physics, astronomy, or chemistry. Ernest led him to develop the design of experiments (DOE). Photograph
provided by Graham Shephard of Rothamsted Research.
Rutherford, the Nobel Prize–winning physicist who first split
the atom, is quoted as saying, BIf your experiment needs sta-
tistics, you ought to have done a better experiment^ (as quoted collected. Statisticians of that period were not like modern-day
in Bailey, 1967). This statement appears derogatory of statis- statisticians, but were viewed something like data alchemists,
tics, but in the context of physical systems that exhibit little with an ability to extract information from data sets. As Fisher
variability, it makes sense. Physicists do not have to sort out would later say, Bno thought was given to the manner in which
confounding variability to determine effects—effects are just the data were obtained^ (Fisher 1947). Fisher showed that the
measured. Johannes Kepler simply recorded the orbits of the effects of the fertilization treatments were confounded, a con-
celestial bodies and showed that they were elliptical. Because cept he developed, with the effects of weather, and that it was
there is little random variation in the orbits, physicists can not possible to completely separate the effects of weather and
focus on understanding the underlying mechanisms of orbital fertilization. The result was nearly 70 years of planting, harvest-
mechanics. These basic physical systems contrast sharply to ing, and measuring for a pile of data that contained little infor-
the highly variable agricultural and biological systems in mation on the effects of different manures on crop yields. Fisher
which effects are obscured or even hidden, as occurred at realized from this experience that the amount of information
Rothamsted with the fertilizer experiments. The lesson of the extracted from data can never exceed what was inherently
Rothamsted experience is that approaching these systems contained in the data. He wrote that this experience caused
using the simple experimental framework of the basic sciences him to turn his attention away from the data and to focus on
will not work, or will not work well. "the process by which the data had come into existence" (Fisher
In frustration, the director, Edward John Russell, decided to 1947). From this point on, Fisher focused on understanding
hire a statistician to analyze the decades of data Rothamsted had what types of experimental structures yielded Bthe most
 DESIGN OF EXPERIMENTS (DOE) 549

information for a given expenditure in time, money, and labor^ Crop Variation. I." If it took about one minute to com-
(Fisher 1947). The results were the concepts and methods that plete a single large-digit multiplication, I estimate that
constitute the foundation of DOE. Fisher needed about 185 hours of work to generate that
table. There are fifteen tables of similar complexity and
After Fisher BStatistical thinking will one day be as necessary four large complicated graphs in the article. In terms of
a qualification for efficient citizenship as the ability to read physical labor alone, it must have taken at least eight
and write^ (H. G. Wells restated in Gaither and Cavazos- months of 12-hour days to prepare the tables for this
Gaither 2012). The initial impact of DOE principles and article. This does not include the hours needed to work
methods was a profound effect on agricultural science, but out the theoretical mathematics, to organize the data, to
more importantly, a new field was created. Additional con- plan the analysis, and to correct the inevitable mistakes.
cepts and methods were later developed that enhanced DOE (Salsburg 2001).
and extended the approach to fields and applications outside
of agriculture. Some key contributions after Fisher, driven Today, the ubiquitous spreadsheet makes such calculations
largely by the need to adapt DOE to industrial and trivial.
manufacturing applications, included response surface meth-
odology (Box and Wilson 1951), quality control (Barker, Why DOE? BInformation is not knowledge.^ (Zappa 1979)
1986; Deming 2000; Bayart 2001), robust process develop- There is a distinction between data and knowledge. The rea-
ment and six sigma (Andersson et al. 2006), and design opti- son an experiment is conducted is to learn something new
mality (Chernoff 1999; Nordström 1999). about a system, not to simply generate large quantities of data
Design optimality, in particular, is considered as a Bparadigm where more data equates to a better experiment. Placing an
shift^ in the further development of DOE (Montgomery 2013). emphasis on acquiring data leads to an Bindustrial model^
The reason is that design optimality algorithms free the re- concept of the research laboratory as a sort of Bdata factory^,
searcher from fixed design structures and allows for the con- where data are a commodity and the primary product of re-
struction of designs for virtually any situation, without limita- search; data acquisition is NOT the principal objective of a
tions in regards to factor type (e.g., process, discrete, mixture) research laboratory (DeLoach 2002). The product produced
or experimental constraints. Additionally, the cost of experi- by a laboratory is not data but knowledge (DeLoach 2000,
mentation is substantially reduced because optimal designs 2002). DeLoach (2002) says that this idea may at first appear
have fewer treatments than conventional factorial approaches. to be Ba distinction without a difference^, but that all notions
Optimal designs achieve these features because they are of quality and productivity change when we recognize that
nonorthogonal designs. Nonorthogonal designs have two unde- knowledge is our product, not data. Because DOE is
sirable statistical properties— factor effects are correlated and knowledge-based rather than data-based, DOE integrates true
variance estimates are inflated (Kuhfeld 2010). Fortunately, quality and productivity measures within the experimental
these drawbacks are relatively minor and do not interfere with process itself. The result is that knowledge is obtained with
the practicality and usefulness of optimal designs. the least expenditure of resources. DOE simultaneously min-
The modern era of design of experiments occurred in the imizes the quantity of data while maximizing data quality.
mid-1980s due to the availability of affordable, powerful DOE does this in two ways:
desktop computers and software capable of handling the nu- DOE identifies the vital few from among the trivial many.
merous and tedious calculations that DOE methods require. DOE minimizes data quantity by using two principles—
For perspective, Fisher’s first assignment at Rothamsted was hierarchical ordering and sparsity of effects (Box and
to analyze 70 years of field and weather data, but all Meyer 1986). The hierarchical ordering principle states
Rothamsted had was a Fuller’s Spiral Rule slide rule (Ross that lower-order effects are more likely than higher-order
2012). Fisher knew that analyzing this quantity of data could effects to dominate or Bdrive^ a system. In practice, di-
not be done by hand but required a calculator, so he convinced verse systems are consistently observed to be driven by
the Rothamsted director, John Russell, to purchase a main and 2-factor interaction effects; 3-factor interactions
Millionaire, a state-of-the-art mechanical calculator which he are rare. DOE reduces treatment combinations because
used for the next 14 years. The Millionaire was the first cal- only those treatments required to determine main and 2-
culator capable of direct multiplication. From Fisher’s paper factor interactions are required. For example, a 5-factor 2-
BStudies in Crop Variation. I.^ (Fisher 1921), David Salsburg level factorial is 32 treatment combinations and can de-
estimated the calculation effort required by Fisher using the termine the effects of the mean, the five 1-factor effects,
Millionaire calculator for a single table: and all of the interaction effects—ten 2-factor, ten 3-fac-
tor, five 4-factor, and one 5-factor. Applying the hierar-
To get some idea of the physical effort involved, con- chical ordering principle and including only 1- and 2-
sider Table VII that appears on page 123 of "Studies in factor effects reduces these 32 treatments to 16. The
550 NIEDZ AND EVENS

sparsity of effects principle states that the number of ef-

fects that drive a system is small. In the above example
this means that only a small number of the 1- and 2-factor
interactions are expected to actually drive the system. By
utilizing these two principles, the DOE approach is de-
signed to identify only the drivers of a system. This is
conceptually and fundamentally different from thinking
that all the effects must be determined to understand a
system. The approach of identifying drivers provides a
comprehensible framework that is extremely useful and
well-suited for studying and characterizing the complex Figure 2. How design of experiments (DOE) makes treatment points
work harder. (A) One-factor-at-a-time (OFAT) approach. An experimenter
systems representative of in vitro biology. wants to determine the best settings of 2 factors, A and B, where each
Data points work harder. A data point Bworks^ by provid- factor is varied one-at-a-time around its control level (0,0). For example,
ing information. The more information that a data point pro- the first experiment would vary Factor A around its control level at (0,0);
vides, the more work that it performs. DOE makes data points Factor B is therefore fixed at its control level at (0,0) during this first
experiment. The second experiment then varies Factor B around its con-
work harder by simultaneously varying multiple factors in trol level at (0,0); Factor A is fixed at its control level at (0,0) during this
very specific ways (DeLoach 2010). Compared to the con- second experiment. Imagine that a large difference is observed between
ventional single-factor experiment, or what is called in DOE the low (−1,0) and high (1,0) levels of Factor A in the first experiment, but
terminology the OFAT (one-factor-at-a-time) experiment, little difference between the levels of B tested in the second experiment.
The experimenter would conclude that the best results required setting
there are two results. One, the number of treatments is re- Factor A at its high level but leaving Factor B unchanged at its control
duced. Two, the quality (precision) of the information ob- level. The problem with this conclusion is that it was never determined if
tained is increased. How this is done is illustrated and ex- Factors A and B interact—is the response of A the same regardless of the
plained using a geometric perspective that compares the level of B? If A and B interact and the best result occurs when A and B are
at their highest settings (1,1), then the experimenter got the wrong answer.
OFAT to a DOE approach in understanding a 2-factor sys- (B) DOE approach. The DOE approach simply rotates the points to the
tem (Fig. 2). In practical terms this means less work and corners of the design space and removes the center point, although there
better or higher-quality information. In the OFAT approach, are good reasons to keep the center point in actual experimentation (to
not only were more treatments used (5 vs 4) to determine detect curvature), these are not relevant to this example. Shifting the
points to the corners results in essentially three experiments in one, and
the main effects of factors A and B, but the ability to detect each experiment uses all 4 points. This is a factorial experiment where
a difference, or statistical power, between the low and high both factors are varied together. The effect of Factor A is determined by
levels of A or B was substantially lower (e.g., 41% for comparing the response of the 2 points at the low level of A (−1,−1) and
OFAT vs 57% for DOE to detect a 2-SD [standard devia- (−1,1) to the 2 points at the high level of A (1,−1) and (1,1); the effect of
Factor B is determined by comparing the response of the 2 points at the
tion] difference with two replicates). Further, the OFAT ap- low level of B (−1,−1) and (1,−1) to the 2 points at the high level of B
proach cannot determine whether A and B interact. This is a (−1,1) and (1,1); the effect of the interaction of A and B is determined by
serious deficiency of the OFAT approach and is a result of comparing the diagonals—the 2 points at the low level of A and B
not simultaneously changing both factors away from their (−1,−1) and high level of A and B (1,1) compared to the 2 points at the
low level of A and high level of B (−1,1) and high level of A and low level
control levels. Simply rotating the outer points to the corners of B (1,−1).
of the design space, the DOE point positioning results in
each point being used three times—to estimate the low and
high levels of both factors and their interaction. This is, in can provide useful information, these situations are typically
effect, three experiments bundled into one. Each DOE point reserved for preliminary testing, such as factor range finding,
Bworks harder^ because each point contains more informa- prior to standard experimentation. Because interactions are
tion than each OFAT point. Having points with little infor- significant in any biological system, characterizing a system
mation is the basic problem of OFAT experiments, and why using OFAT experiments will often result in conclusions that
large quantities of data are required to characterize a system are incorrect and results that are often difficult or impossible
using the OFAT approach. OFAT experiments are, conse- to reproduce. Difficulty repeating published research results
quently, inefficient and wasteful, with results that are often is a well-recognized issue by the commercial plant tissue
misleading. Quality suffers because only a small fraction of culture community and is a major problem in biomedical
the total number of possible combinations can typically be (Collins and Tabak 2014; Begley and Ioannidis 2015;
run because of resource limitations. OFAT experiments are Freedman et al. 2015) and psychology research (Aarts
not well-suited for understanding complex systems, such as et al. 2015). There are multiple reasons for the repeatability
the in vitro environment, because interactions between fac- problem (e.g., insufficient method details, methods not
tors cannot be detected. Though there are some types of followed, unknown differences between the studies), but
problems and conditions where single-factor experiments OFAT-based approaches can contribute to the problem.
 DESIGN OF EXPERIMENTS (DOE) 551

Varying multiple factors simultaneously was an innovation Burman, Central Composite, Fractional Factorial, Simplex
by Fisher. He got the idea from his work in genetics on the Lattice, D-Optimal, Taguchi Orthogonal Array, and Irregular
Binheritance of Mendelian factors^ (Fisher 1952) and the fac- Resolution V, DOE can be initially confusing. However, there
torial combinations meiosis uses to generate genetic variabil- is an elegant, underlying simplicity to DOE. This simplicity is
ity. According to Fisher, multifactor experiments should be the inherent geometric nature of experimental design. The
the default approach when studying nature, with OFAT exper- sometimes difficult statistical concepts are brought together,
iments the exception. unified, and more easily grasped when viewed as a geometric
Bpicture^. Fisher was a very good geometrician and intuitively
No aphorism is more frequently repeated in connection understood the statistical concepts he developed in a geomet-
with field trials, than that we must ask Nature few ques- ric framework (Box 1978); he then reduced these concepts to
tions, or, ideally, one question, at a time. The writer is algebra for making the calculations. Geometry is the natural
convinced that this view is wholly mistaken. Nature, he setting for understanding DOE.
suggests, will best respond to a logical and carefully A geometric perspective to experimental design is enor-
thought out questionnaire; indeed, if we ask her a single mously useful for understanding and addressing the complex-
question, she will often refuse to answer until some oth- ity problem of in vitro systems. The in vitro environment is
er topic has been discussed. (Fisher 1926). enormously complex—16 essential mineral nutrients, plant
growth regulators, carbon sources, gelling agents, vitamins,
Varying multiple factors simultaneously requires a method environment factors (e.g., light type/quality/photoperiod, tem-
to assign the changes in the measured response to the factors perature, gases), and all of their interactions. OFAT experi-
that were varied; this is what the analysis of variance does. mentation, an approach not well-suited for complex multivar-
Fisher thought that approaching research using OFAT experi- iate systems, is often used to understand this complexity.
ments was inherently misleading: Further, because the goals of most in vitro experiments are
applied, finding the right combination of medium components
We are usually ignorant which, out of innumerable pos- and cultural conditions to grow the cells, tissues, or organs as
sible factors, may prove ultimately to be the most im- they need to be grown can be, and often is, a long and expen-
portant, though we may have strong presuppositions sive process. Geometry is useful for approaching this culture
that some few of them are particularly worthy of study. complexity problem, as it provides a context for understand-
We have usually no knowledge that any one factor will ing how to think about, design, and analyze multifactor in vitro
exert its effects independently of all others that can be experiments. Because the underlying basis of experimental
varied, or that its effects are particularly simply related design is geometric, experimentation is viewed from a differ-
to variations in these other factors…. If the investigator, ent perspective that is intuitive and especially helpful to the
in these circumstances, confines his attention to any sin- researcher in the actual planning, running, and analyzing
gle factor we may infer either that he is the unfortunate experiments.
victim of a doctrinaire theory as to how experimentation The geometric perspective is actually quite simple concep-
should proceed, or that the time, material or equipment tually and is based on two geometric shapes—the square and
at his disposal is too limited to allow him to give atten- the triangle. The geometry of the square results when factors
tion to more than one aspect of his problem.... (Fisher are varied independently which is the most common treatment
1937). of factors. The rule is that each factor is equivalent to a geo-
metric dimension. Thus, 1 factor = 1 dimension, and geomet-
Fisher is saying that the idea of OFAT experimentation is a rically, a 1-dimensional geometry is a line (Fig. 3A). The line
contradiction. The researcher is sufficiently ignorant of the spans from some low level of the factor to some high level.
system to recognize the need for research. But conversely, This range is called the Bdesign space^. The idea is to sample
the researcher is sufficiently knowledgeable of the system the design space, measure something (fresh weight, dry
both to select a very small subset of factors and factor combi- weight, color, multiplication ratio, shoot size, number of em-
nations to test and to know which very large subset to ignore. bryos, root number, etc.), and then determine how the mea-
This is Fisher’s Bunfortunate victim of a doctrinaire theory^ sured responses vary over the design space. Two factors
because the researcher is confident of obtaining sufficient in- equals 2 dimensions, or geometrically a plane (Fig. 3B), which
formation to understand the system under study. is defined by the ranges of each of the 2 factors. The plane is
the design space; it is sampled, the responses are measured,
Experimental Design is Inherently Geometric BA good and how the responses vary over the design space is deter-
sketch is better than a long speech.^ (Attributed to Napoleon mined. Three factors equals 3 dimensions or a cube (Fig. 3C).
I of France) Software is required for the design and analysis of Adding factors simply extends the design space into higher
DOE experiments. With design terms such as Plackett– and higher dimensions—4 factors equals 4 dimensions,
552 NIEDZ AND EVENS

Figure 3. Design space

geometry of independent factors.
(A) A 1-factor design is a 1-
dimensional geometry, or a line.
Treatment points (red dots) are
selected to sample the design
space. A response is measured
and how the response varies over
the design space is determined.
(B) A 2-factor design space is a 2-
dimensional geometry, or a plane.
Like the line, treatment points are
selected, responses measured, and
how the responses vary over the
design space determined. (C) A 3-
factor design space is a 3-
dimensional geometry, or a cube.
(D) A 4-factor design space is a
4D cube, termed a hypercube or a
tesseract. The 4D space shown is
an orthographic projection, and
the red dots are the corners of the
hypercube, not treatment points.
Public domain image reproduced
from Wikimedia Commons
(Piesk 2014). (E) Orthographic
projection of a 9D cube. Public
domain image reproduced from
Wikimedia Commons (Ruen
2011a). (F) Orthographic
projection of a 10D cube. Public
domain image reproduced from
Wikimedia Commons (Ruen
2011b).

represented by a hypercube or 4D cube. Because humans experiments of mixtures, the type of experiments gener-
cannot visualize dimensional spaces greater than 3D, var- ally most suitable, though rarely used, for studies of
ious ways to project these spaces into 2 or 3 dimensions in vitro culture medium formulation. Because mixture
are used (Fig. 3D). The point here is that the design space factors are not independent, the dimensionality of the
of a 9- or 10-factor experiment (Fig. 3E, F) is just as real design geometry equals the number of mixture compo-
and defined as that of the design space of a line, plane, or nents minus 1 (Fig. 4). Mixtures are defined as propor-
cube. Geometry reveals the beauty of order in experimen- tions, not amounts. The term Bcomponents^ is used
tal design. when referring to mixture Bfactors^ in recognition that
The geometry of factors that cannot be varied inde- they are not independent. Component proportions cannot
pendently is based on the triangle (Fig. 4); these are be varied independently because the sum of the
 DESIGN OF EXPERIMENTS (DOE) 553

Figure 4. Design space

geometry of factors that are not
independent (mixtures). Mixture
design spaces are a constrained
region within the larger
unconstrained region defined by
independent factors. The
constraint of a mixture is that all
the factor proportions must sum to
1. The figures show the
relationship of each mixture space
(constrained) to the unconstrained
region from which it arises (left),
and the resulting mixture space
(right). (A) Constraining 2 factors
within the 2D plane, A + B = 1,
results in the 1D line (blue) of the
2-component mixture. This
occurs because Factors A and B
are not independent. Defining the
proportion of either A or B
restricts the proportion of the
other factor because the
proportion of A + B must sum to
1. (B) Constraining 3 factors
within the 3D cube, A + B + C =
1, results in the 2D triangle (blue)
of the 3-component mixture. (C)
Constraining 4 factors within the
4D hypercube, A + B + C + D = 1,
results in the 3D tetrahedron of
the 4-component mixture. The
coordinates of points within a
mixture space are expressed as
proportions summing to 1, also
called barycentric coordinates.

proportions of the components of a mixture must equal accordingly to fill the pot. This 3-component mixture
1 or unity. This constraint means that varying the pro- therefore requires a 2-dimensional geometry suitable
portion of a component requires changing the proportion for 3 components—a triangle. A 2-component mixture
of one or more of the other components to maintain is simply a line that represents the ratio of the 2 com-
unity. Proportional effects are effects based on the pro- ponents. Mixture design geometries are based on the
portions of the components that make up the mixture triangle, tetrahedron, hypertetrahedron, and n-dimension-
rather than their amounts. For example, a potting sub- al tetrahedrons—all analogous to the planes, cubes, and
strate study that involves peat, sand, and perlite is a hypercubes used for independent factors. Proportional
mixture and requires a 3-component mixture design ge- and amount effects are always confounded when factors
ometry. Changing the proportion of peat means that the are treated as independent. Proportional effects can only
proportions of sand and/or perlite must be adjusted be determined with mixture design spaces.
554 NIEDZ AND EVENS

Why proportional and amount effects are

confounded when factors are treated as independent

A mixture effect only depends on the proportion of the com-

ponents; amount has no effect. Lemonade is a mixture of
lemon juice and sugar; only the proportion of lemon juice to
sugar affects taste. Because amount does not affect taste, lem-
onade tastes the same regardless of the size of the glass. To
illustrate how proportional and amount effects are confounded Figure 6. Design geometry for a 2-component mixture of sugar water
when factors are treated as independent, consider an experi- and lemon juice. The dimensionality of the design space is the number of
components minus 1.
ment to determine the best-tasting lemonade. Lemon juice and
sugar (sugar water) are varied independently using a 2×2 fac-
What type of design geometry is required
torial design (Fig. 5). The confounding between proportion
to determine the effects of proportion, amount,
and amount is evident when the proportions and amounts of
and proportion × amount?
the 4 treatment combinations are compared. Although the fac-
torial has 4 treatment combinations, there are really only 3
The design geometry to separate the effects of proportion,
unique treatment combinations for proportion and 3 unique
amount, and proportion × amount is a plane (Fig. 7). The 2
treatment combinations for amount. Two of the treatments
dimensions of the plane correspond to proportion and amount.
have the same proportion (1:1), but different amounts—they
This is a mixture-amount design.
will taste the same. Two of the treatments have the same
amount (110 mL), but different proportions (1:10 and 10:1).
The 4 treatment combinations resulting from treating the 2
Implications for In Vitro Culture BEquations are just the
factors as independent cannot separate these two effects. An
boring part of mathematics. I attempt to see things in terms
ANOVA will only detect an interaction, but cannot determine
of geometry.^ (Stephen Hawking as related in Larsen, 2005)
if the interaction is due to proportion, amount, or an interac-
What are the implications of a geometric perspective for
tion between proportion and amount.
in vitro culture experimentation? The primary implication is
that a culture medium is a coordinate in Euclidean space. To
illustrate this concept, consider a 3-component mixture design
space of NH4+, NO3−, and K+. The ionic composition of these
What type of design geometry is required
three ions in MS medium (Murashige and Skoog, 1962) is
to determine the effects of proportion?
20.6 mM NH4+, 39.4 mM NO3−, and 20 mM K+ for a total
ionic strength of 80 mM. Proportionally, this converts to 0.26
The design geometry to determine the effects of the proportion
NH4+, 0.49 NO3−, and 0.25 K+. Varying the proportion of an
of lemon juice and sugar water on taste is a line (Fig. 6).
ion requires changing the proportion of one or more of the

Figure 7. Design geometry for a 2-component mixture-amount design

of sugar water and lemon juice. The dimensionality of the design space is
Figure 5. The effect on proportion and amount when 2 factors, sugar one for the mixture and one for amount. The geometry of the design
water and lemon juice, are varied independently. Proportion and amount separates proportion and amount and allows their effects to be
are confounded. determined.
 DESIGN OF EXPERIMENTS (DOE) 555

other ions to maintain unity of the mixture. Because some space poorly. The problem of a poorly sampled design space
important factors in plant tissue culture, including NH4+, worsens as the dimensionality of the space increases, as would
NO3−, and K+, have proportional effects, it is necessary to occur by including additional mineral nutrient ions. What hap-
use a geometry suitable for capturing the design space that pens when 114 additional plant tissue culture media
represents the mixture. The design space of a mixture of (Supplement 1) are added? The space remains poorly sampled
NH4+, NO3−, and K+ is a triangle, and any point within the (Fig. 8B). DOE sampling of the design space requires only 6
triangle mixture design space is expressed as a proportion of correctly positioned points, and is much more useful and effi-
all the components that make up the mixture. cient then Brandomly^ trying prior formulations to locate re-
Fourteen Bclassic^ plant tissue culture media were selected gions where the in vitro cultures grow well. Twelve points
to illustrate this culture medium as a coordinate concept were selected by design optimality (DOE software required)
(Table 1). First, the proportions of NH4+, NO3−, and K+ were and plotted (Fig. 8C); these 12 points include the 6 points
calculated for each of the 14 classic medium. For illustration required for modeling plus an additional 6 points to test how
purposes the proportions were calculated in relation to each well the model predicts (referred to as lack-of-fit points).
other—e.g., the proportion of NH4+ = NH4+ mM/(NH4+ mM Another option is to sample the region generally defined by
+ NO3− mM + K+ mM). Calculating the proportions based on the 128 media (Fig. 8D). Either way, 12 points provide more
all the ions in the medium would only shift the constellation of information, with much less effort, than the 128 media or
points a small amount. Second, the media were plotted in the trying to determine/guess which of the 128 media to try.
3-component mixture design space using their proportions; For mineral nutrition, a culture medium as a coordinate
plotting proportional quantities uses the barycentric coordi- implies that a culture medium is not the same as the recipe
nate system (Fig. 8A). The triangle represents the entire uni- used to make the medium (Niedz and Evens 2006). One hun-
verse of possible proportional combinations of NH4+, NO3−, dred twenty-eight plant tissue culture media were converted to
and K+—i.e., there are no conceivable media formulations their ion equivalents, but the combinations of salts used were
that fall outside this space. For example, Knudson’s medium not unique. The same ion levels can be achieved with many
is shown—barycentric coordinate of 0.42 NH4+, 0.47 NO3−, different types and concentrations of salts, acids, and bases—
and 0.11 K+. Knudson’s medium was used for the first axenic multiple recipes. Though there are many recipes, all would
in vitro propagation of a plant, and was a major advance in result in exactly the same ion levels and are, therefore, indis-
plant tissue culture that culminated in biotechnology. tinguishable from each other once the salts, acids, and bases
Knudson’s medium is still used today for orchid seed germi- are dissolved into solution. Defining a medium as a coordinate
nation and orchid culture. The 14 classic media sample the is necessary for designing the types of experiments required to

Table 1. Fourteen classic plant tissue culture media proportions and and KI (0.83 mg L−1). When dissolved, these salts result in a solution that
amounts of NH4+, NO3−, and K+. Ion proportions and amounts were contains 20.6 mM NH4+, 39.4 mM NO3−, and 20 mM K+ or a total of
calculated from the reported recipes. For example, the NH4+, NO3−, and 80 mM of [NH4+] + [NO3−] + [K+]. Converting to proportions results in
K+ ions in MS medium (Murashige and Skoog 1962) are derived from 0.26 NH4+ (20.6/80), 0.49 NO3− (39.4/80), and 0.25 K+ (20/80)
NH4NO3 (1,650 mg L−1), KNO3 (1,900 mg L−1), KH2PO4 (170 mg L−1),

Classic medium NH4+ NO3− K+ [NH4+] + [NO3−] + Reference

[K+] (mM)

Knop’s (1865) 0 0.72 0.28 15 (George 1993)

White’s (1943) 0 0.66 0.34 5 (George 1993)
Knudson (1943) 0.42 0.47 0.11 18 (George 1993)
Vacin & Went (1949) 0.38 0.26 0.36 20 (Vacin and Went 1949)
Chu et al. (1975) N6 0.11 0.42 0.47 66 (George 1993)
Heller (1953) 0 0.41 0.59 17 (George 1993)
Murashige & Skoog (1962) MS 0.26 0.49 0.25 80 (Murashige and Skoog 1962)
Quoirin & Lepoivre (1977) 0.07 0.37 0.56 67 (Quoirin and Lepoivre 1977)
Morel & Muller (1964) 0.45 0.12 0.43 34 (George 1993)
Gamborg et al. (1968) B5 0.04 0.48 0.48 51 (Gamborg et al. 1968)
Nitsch & Nitsch (1969) 0.24 0.49 0.27 37 (Nitsch and Nitsch 1969)
Schenk & Hildebrandt (1972) SH 0.05 0.47 0.48 52 (Schenk and Hildebrandt 1972)
Lloyd & McCown (1980) WPM 0.18 0.36 0.46 27 (Lloyd and McCown 1980)
Driver & Kuniyuki (1984) DKW 0.24 0.48 0.28 68 (McGranahan et al. 1987)
556 NIEDZ AND EVENS

Figure 8. A culture medium as a coordinate in Euclidean space. Each plant tissue culture media added (Supplement 1). The space remains
diagram shows plant tissue culture media plotted in a 3-component poorly sampled. (C) 12 points (blue dots) to sample the entire space.
mixture space of NH4+, NO3−, and K+ using the proportions of these ions (D) 12 points (blue dots) to sample the subspace generally represented
as the coordinates (barycentric coordinates). (A) 14 classic plant tissue by the 129 plant tissue culture media.
culture media (Table 1). The space is poorly sampled. (B) 114 additional

determine the effects of the mineral nutrient ions directly and components, improved formulations and conditions are a
(Niedz and Evens 2006; Evens and Niedz 2008). natural result. This is an amazing benefit—less but more effi-
A culture medium as a coordinate provides a useful and cient experimentation that results in new knowledge in the
convenient framework to organize the in vitro culture litera- science of Bgrowing things^ and improved and better pro-
ture. From the previous example, the sampled and unsampled tocols (formulations and conditions). Treating a culture
areas are clear. Converting plant tissue culture media to coor- medium as a coordinate easily scales with the large num-
dinates and then associating species, plant part, type of re- bers of organic, inorganic, and environmental factors that
sponse, etc. provides a data structure suitable for data mining make up an in vitro system. The only difference between
techniques useful for bringing order to very complex systems. the illustrated 3-component NH4+, NO3−, and K+ design
Such analyses would be expected to generate new hypotheses space and spaces composed of additional factors such as
for testing. Though the efficient production of new knowledge PO43−, Ca2+, Mg2+, and the micronutrients is an increase
is the basis of DOE, more rapid identification of useful for- in the dimensionality of the space. Exploring these higher-
mulations is certainly implied. Because the use of geometry dimensional spaces represents a rich area of research for
provides an understanding of the effects of the various factors the in vitro community.
 DESIGN OF EXPERIMENTS (DOE) 557

In Vitro Examples of DOE slice through the 5D space (Fig. 9C), referred to as a subspace
slice. The region under the mesh is the region where fresh
BThe test of all knowledge is experiment^ (Feynman et al., weight accumulation was greatest. It turned out that this fast-
1963) Does designing experiments conceived geometrically growing region under the mesh was not suitable for general
and in dimensions higher than what can be visualized actually cell line maintenance—the tissue required more frequent sub-
work? Niedz and Evens (2007) tested this perspective by de- cultures because of its faster growth. However, if 10 kg of
termining the effect of various salts on fresh and dry weight tissue were required (a request once received from a biochem-
accumulation of a nonembryogenic sweet orange cell line ist), then ramping up production of these cells using a formu-
(Citrus sinensis (L.) Osbeck cv. ‘Valencia’) at the end of three lation from the region under the mesh would work well.
2-wk growth cycles. Multiple cycles were used to allow the The second example is a plant growth regulator study
tissue to adapt to the treatment formulations. In this experi- (Niedz and Evens 2011). Plant growth regulator studies are
ment MS salts were divided into 5 factors—KNO3, NH4NO3, widely used in plant tissue culture research. A common ex-
meso salts, trace metals, and Fe. A 5-factor experiment creates periment is to determine the best types and concentrations of
a 5-dimensional hypercube; DOE software was used to select cytokinins and auxins for a particular in vitro system. Plant
design-optimal points (coordinates) required to correctly sam- tissue culturists know that cytokinins and auxins interact, as
ple this space. Because the entire 5D space was being sam- was first demonstrated by the effects of kinetin and indole-3-
pled, there were some unusual factor combinations—combi- acetic acid (IAA) on shoot and root formation of tobacco pith
nations that probably would not have been selected if not callus (Skoog and Miller 1957). However, because Skoog and
using DOE. The data analysis revealed that fresh weight in- Miller treated kinetin and IAA as independent factors, using a
creased 690% at the end of the third growth cycle on the factorial arrangement, proportional and amount effects were
control MS medium, which was both a coordinate in the space confounded. The only design geometry that can separate these
and the medium used to maintain the cell line. A region was effects is a mixture-amount design. Mixture-amount designs
found where fresh weight growth increased by 928%, substan- not only separate the effects of proportion and amount, but
tially greater than the increase on MS (Fig. 9A vs 9B). The will reveal whether proportion and amount interact. All other
ANOVA revealed that NH4NO3 and Fe had the largest effects design geometries confound the effects of proportion and
on fresh weight growth. Their effects are easily viewed in a 3D amount. This confounding complicates understanding the

Figure 9. A 5D design space to

determine the effects of MS salts
on the growth of a
nonembryogenic cell line of
Valencia sweet orange. (A) MS
medium. Fresh weight increased
690% by the end of the 14 d
growth cycle. Springer-owned
imprint image reproduced from
Niedz and Evens (2007) with
permission from Springer. (B) A
coordinate (medium formulation)
where fresh weight increased
928% by the end of the 14 d
growth cycle. This specific
coordinate was located in the
region where the largest increases
in fresh weight occurred.
Springer-owned imprint image
reproduced from Niedz and
Evens (2007) with permission
from Springer. (C) 3D slice
through the 5D design space that
shows a region where the fresh
weight growth was equivalent to
MS (turquoise band), and the
region where fresh weight growth
was greatest (under the mesh).
NH4NO3 and Fe had the largest
effects on fresh weight growth.
558 NIEDZ AND EVENS

effects of the plant growth regulators, as it does in all systems dimension of 1 to 30 μM representing the total concentration
where proportion and amount effects occur (e.g., mineral nu- of the components being tested (Fig. 10A). The results were
trition). The objective of this study was to improve shoot quite amazing. A region was found where shoot regeneration
regeneration from grapefruit (Citrus paradisi Macd.) seedling was substantially greater than with our standard shoot regen-
epicotyl explants. Epicotyl explants are the most commonly eration medium (Fig. 10B vs 10C). Unfortunately, transforma-
used source tissue for Agrobacterium-mediated transforma- tion efficiency was not affected (unpublished data). This was
tion in citrus, but the transformation efficiency of grapefruit, disappointing, but it was helpful in understanding that shoot
along with sweet orange (Citrus sinensis (L.) Osbeck) and the regeneration was not the reason for low transformation effi-
mandarins (Citrus reticulate Blanco), is quite low. The idea ciencies in grapefruit.
was to improve transformation efficiency by improving shoot The third example is a mineral nutrient study to map the
regeneration. fundamental niches of two algae species, Chlorella vulgaris
A preliminary experiment with six cytokinins identified and Peridinium cinctum (Evens and Niedz 2011). The exper-
N6-benzyladenine (BA) and zeatin riboside (ZR) as the cyto- iment included 5 ions (K+, Cl−, NO3−, PO43+, and Na+). The
kinins that induced the greatest number of shoots. These two design geometry was a mixture-amount constructed as a 5-ion
cytokinins were then combined with two auxins commonly mixture (hypertetrahedron) projected through an amount di-
used in citrus in vitro culture, IAA and α-naphthalene acetic mension, where amount ranged from 1 to 30 mM (Fig. 11A).
acid (NAA). A 4-component mixture space, i.e., a tetrahedron, The design geometry is similar to the plant growth regulator
was constructed from these four plant growth regulators. This experiment described previously. Because the design space
space is the entire universe of all possible proportional com- was constructed from the ions directly, the experiment was
binations of these four plant growth regulators. To include free of ion confounding (Niedz and Evens 2006). Ion con-
amount (μM) requires adding an additional dimension to the founding is nearly universal in the experimental designs used
design geometry, which was done by projecting the 4- to determine ion-specific effects across all fields of research,
component mixture space (tetrahedron) through an amount including plant nutrition. This was a preliminary experiment

Figure 10. A mixture-amount

design geometry to determine the
effects of N6-benzyladenine
(BA), zeatin riboside (ZR),
indole-3-acetic acid (IAA), and
α-naphthalene acetic acid (NAA)
on shoot regeneration from
grapefruit epicotyl explants. (A)
Design geometry of 4-component
mixture projected through an
amount dimension of 1 to 30 μM.
This illustrates the separation of
the effects of proportion from the
effects of amount. For example,
two formulations can have the
same proportions of BA-ZR-
IAA-NAA but differ in amount,
or have the same amount but have
different proportions. (B) 10 μM
BA, the standard level used in our
laboratory. The coordinate is 1
BA and 10 μM. Springer-owned
imprint image reproduced from
Niedz and Evens (2011) with
permission from Springer. (C)
6.66 μM ZR + 13.34 μM IAA.
The coordinate is 0.33 ZR, 0.67
IAA, and 20 μM. Springer-owned
imprint image reproduced from
Niedz and Evens (2011) with
permission from Springer.
 DESIGN OF EXPERIMENTS (DOE) 559

Figure 11. The effects of 5 ions

on relative cell density of two
algae species, Chlorella vulgaris
and Peridinium cinctum. Images
are 3D multidimensional scaling
projects of the 5D design space.
(A) 5-ion design space geometry.
(B) Relative growth of
C. vulgaris. (C) Relative growth
of P. cinctum. Note the lack of
growth in the high-potassium
region compared to C. vulgaris.

in a larger project to map the growth of a number of algae directly (Evens and Niedz 2008). pH is simply another re-
species into higher-dimensional mineral nutrient ion spaces. sponse that can be mapped in the 5D ion design space. By
Because the design space included virtually any formulation using the determinants of pH, the ions themselves, to create
found in nature, a DOE approach was considered as potential- the design space, pH is not fixed but varies throughout the
ly useful for mapping a species’ fundamental niche. The idea space. This approach is quite different from how experiments
was to predict the types and concentrations of algae species are typically designed. Treating the ions using DOE removes
present in an aquatic ecosystem. By identifying the regions the ubiquitous problem of ion confounding and determines
within design spaces constructed from the nutritional and en- directly the effects of the ions. Further, any bias from Bbeliefs^
vironmental factors that drive algae growth, early-warning that the experimenter has as to the Bcorrect^ pH where the
monitoring systems of conditions favorable for toxic algae organism will/should grow are removed. An ion-defined
blooms (Landsberg 2002; Hallegraeff 2010; Glibert and space is created without preconceptions; pH is not controlled,
Burkholder 2011; Anderson et al. 2012; Davidson et al. but varies based on the ion composition. The result is that the
2012; Smith et al. 2015) might be possible. One result was organism is free to grow where it will. Approaching mineral
that only C. vulgaris grew well in a region with a high pro- nutrition from a DOE geometric perspective clearly reveals
portion of K+; this is clearly seen using a multidimensional the dependent nature of pH and provides a more comprehen-
scaling projection of the 5D design space into 3D space sive and useful framework for understanding pH effects. This
(Fig. 11B vs 11C). is true not only of pH but of all solution properties that result
Another result was that there was little correlation between from the ionic composition of a solution. The ions are the
growth and pH. Because ions determine pH, pH is inherently a independent factors. This perspective provides an experimen-
dependent variable and cannot therefore be manipulated tal approach that is quite different from the classical approach
560 NIEDZ AND EVENS

that treats pH as an independent variable. Treating pH as an It is the authors’ opinion that fully incorporating DOE into
independent variable results in studies that report BThe effects how the in vitro research community conducts research would
of pH on …^. Because pH is not an independent variable, pH have several beneficial outcomes. First, labs would directly
effects cannot be determined directly, but can only be under- benefit from the inherent efficiency and cost reduction im-
stood through correlation. provements. Second, the higher quality of in vitro papers
would result in higher recognition and respect in the overall
Getting Started and Conclusions BA journey of a thousand scientific community. In vitro papers that essentially report
miles begins with a single step.^ (Attributed to Lao Tzu) DOE better recipes or protocols typically have a low impact in the
requires software, and software is the researcher’s primary literature. This reflects the small amount of information ob-
tool to use DOE; thus, selecting a DOE software application tained from these sorts of studies. Because DOE is knowledge
is critical. The role of the researcher in planning an experiment based, the substantially greater amount of information obtain-
is to define the objective(s), factors to be varied, and responses ed on factor effects is more widely applicable. Third, the re-
to be measured, and to conceive of possible design geometries sults of DOE-based experimentation are highly publishable.
that might be useful for achieving the objective(s) and answer- OFAT-based research to find the right conditions and formu-
ing the question(s) of interest. The role of the software is to lations to grow something typically have many treatments
translate the researcher’s conceptualized design geometry to where growth is poor or nonexistent. Publishing such
include the points required to sample that geometry, data anal- Bnegative^ results is awkward and difficult. When such re-
ysis, data and model quality measures, graphical displays, and search is structured by DOE, all results become publishable.
response/multi-response optimization capabilities. The soft- There are no failures or negative results but a constant im-
ware should have an intuitive interface, a complete user man- provement, even if only incremental. This makes working
ual, and comprehensive software and statistical support in the on the more recalcitrant-type systems much less risky since
form of tutorials, webinars, workshops, online resources, and all work can be adequately documented by publication. Fourth
consultation. and lastly, DOE brings a level of sophistication to the science
Learning DOE, like any new skill, requires support. The of growing things that complements well the advances made
support is enormously helpful for learning new concepts and in other fields of biology such as molecular biology and the
ideas, identifying the range of options, and understanding how various big-data -omics fields. Mapping responses in struc-
and why to select the most appropriate option. A good foun- tured and defined design spaces provides coordinates to inter-
dation of support greatly assists in deepening one’s under- esting regions that can then be explored in depth by other
standing. Support will include books, online forums, work- research communities.
shops, and most importantly, regular interaction and discus-
sion with other DOE practitioners. Interacting with practi-
tioners in the same field is of particular value in learning
how to apply DOE to the particular types of problems and References
constraints of that field. There is almost always more than
one approach to designing an experiment, and having access Aarts AA, Anderson JE, Anderson CJ, Attridge PR, Attwood A, Axt J,
to people with experience can greatly accelerate this learning Babel M, Bahník Š, Baranski E, Barnett-Cowan M, Bartmess E,
process. With discussion and reflection in planning the exper- Beer J, Bell R, Bentley H, Beyan L, Binion G, Borsboom D,
iment, it is generally the case that important differences Bosch A, Bosco FA, Bowman SD, Brandt MJ, Braswell E,
Brohmer H, Brown BT, Brown K, Brüning J, Calhoun-Sauls A,
emerge, sometimes quite nuanced, between the various possi- Callahan SP, Chagnon E, Chandler J, Chartier CR, Cheung F,
ble designs. Because an experiment is just like a question, how Christopherson CD, Cillessen L, Clay R, Cleary H, Cloud MD,
the question is Bphrased^ will affect the type of answer re- Conn M, Cohoon J, Columbus S, Cordes A, Costantini G, Alvarez
ceived. These discussions with persons with knowledge of the LDC, Cremata E, Crusius J, DeCoster J, DeGaetano MA, Penna
ND, Den Bezemer B, Deserno MK, Devitt O, Dewitte L, Dobolyi
subject or related areas often results in useful information for DG, Dodson GT, Donnellan MB, Donohue R, Dore RA, Dorrough
building the final design. A, Dreber A, Dugas M, Dunn EW, Easey K, Eboigbe S, Eggleston
Start simple with design geometries that can be visual- C, Embley J, Epskamp S, Errington TM, Estel V, Farach FJ, Feather
ized—2 or 3 dimensions. That way all aspects of the DOE J, Fedor A, Fernández-Castilla B, Fiedler S, Field JG, Fitneva SA,
Flagan T, Forest AL, Forsell E, Foster JD, Frank MC, Frazier RS,
process, from point selection to analysis, can be visualized and Fuchs H, Gable P, Galak J, Galliani EM, Gampa A, Garcia S,
an intuition of how DOE works will begin to develop; there is Gazarian D, Gilbert E, Giner-Sorolla R, Glöckner A, Goellner L,
no better learning experience than one’s own experiment and Goh JX, Goldberg R, Goodbourn PT, Gordon-McKeon S, Gorges
data. BSeeing^ how DOE works with simpler designs makes B, Gorges J, Goss J, Graham J, Grange JA, Gray J, Hartgerink C,
Hartshorne J, Hasselman F, Hayes T, Heikensten E, Henninger F,
moving to more complex designs with more factors (higher- Hodsoll J, Holubar T, Hoogendoorn G, Humphries DJ, Hung COY,
dimensional geometries) much easier because the principles Immelman N, Irsik VC, Jahn G, Jäkel F, Jekel M, Johannesson M,
and concepts are the same. Johnson LG, Johnson DJ, Johnson KM, Johnston WJ, Jonas K, Joy-
 DESIGN OF EXPERIMENTS (DOE) 561

Gaba JA, Kappes HB, Kelso K, Kidwell MC, Kim SK, Kirkhart M, DeLoach R (2000) Improved quality in aerospace testing through the
Kleinberg B, Knežević G, Kolorz FM, Kossakowski JJ, Krause RW, modern design of experiments. 38th Aerospace Sciences Meeting
Krijnen J, Kuhlmann T, Kunkels YK, Kyc MM, Lai CK, Laique A, and Exhibit. American Institute of Aeronautics and Astronautics,
Lakens D, Lane KA, Lassetter B, Lazarević LB, Le Bel EP, Lee KJ, Reston, Virginia
Lee M, Lemm K, Levitan CA, Lewis M, Lin L, Lin S, Lippold M, DeLoach R (2002) The objective of aerospace ground testing: an alterna-
Loureiro D, Luteijn I, MacKinnon S, Mainard HN, Marigold DC, tive perspective. GTTC Newsletter 5:18–19
Martin DP, Martinez T, Masicampo EJ, Matacotta J, Mathur M, May DeLoach R (2010) Analysis of variance in the modern design of exper-
M, Mechin N, Mehta P, Meixner J, Melinger A, Miller JK, Miller M, iments. 48th AIAA Aerospace Sciences Meeting Including the New
Moore K, Möschl M, Motyl M, Müller SM, Munafo M, Neijenhuijs Horizons Forum and Aerospace Exposition. American Institute of
KI, Nervi T, Nicolas G, Nilsonne G, Nosek BA, Nuijten MB, Olsson Aeronautics and Astronautics, Orlando, Florida
C, Osborne C, Ostkamp L, Pavel M, Penton-Voak IS, Perna O, Deming WE (2000) The new economics for industry, government, edu-
Pernet C, Perugini M, Pipitone RN, Pitts M, Plessow F, Prenoveau cation, 2nd edn. MIT Press, Cambridge, Massachusetts
JM, Rahal RM, Ratliff KA, Reinhard D, Renkewitz F, Ricker AA, Driver JA, Kuniyuki AM (1984) In vitro propagation of Paradox walnut
Rigney A, Rivers AM, Roebke M, Rutchick AM, Ryan RS, Sahin rootstock. HortSci 19:507–509
O, Saide A, Sandstrom GM, Santos D, Saxe R, Schlegelmilch R, Evens TJ, Niedz RP (2008) Are Hofmeister series relevant to modern ion-
Schmidt K, Scholz S, Seibel L, Selterman DF, Shaki S, Simpson specific effects research? Scholarly Research Exchange.
WB, Sinclair HC, Skorinko JLM, Slowik A, Snyder JS, Soderberg doi:10.3814/2008/761829
C, Sonnleitner C, Spencer N, Spies JR, Steegen S, Stieger S,
Evens TJ, Niedz RP (2011) Mapping the fundamental niches of two
Strohminger N, Sullivan GB, Talhelm T, Tapia M, Te Dorsthorst
freshwater microalgae, Chlorella vulgaris (Trebouxiophyceae) and
A, Thomae M, Thomas SL, Tio P, Traets F, Tsang S, Tuerlinckx F,
Peridinium cinctum (Dinophyceae), in 5-dimensional ion space. Int
Turchan P, Valášek M, Van't Veer AE, Van Aert R, Van Assen M,
J Ecol 2011:1–12
Van Bork R, Van De Ven M, Van Den Bergh D, Van Der Hulst M,
Feynman RP, Leighton RB, Sands M (1963) The Feynman Lectures on
Van Dooren R, Van Doorn J, Van Renswoude DR, Van Rijn H,
Physics. Addison-Wesley Publishing Company, Reading, MA
Vanpaemel W, Echeverría AV, Vazquez M, Velez N, Vermue M,
Verschoor M, Vianello M, Voracek M, Vuu G, Wagenmakers EJ, Fisher RA (1921) Studies in crop variation. I. An examination of the yield
Weerdmeester J, Welsh A, Westgate EC, Wissink J, Wood M, of dressed grain from Broadbalk. J Agric Sci 11:107–135
Woods A, Wright E, Wu S, Zeelenberg M, Zuni K (2015) Fisher RA (1925) Statistical methods for research workers. Oliver and
Estimating the reproducibility of psychological science. Science Boyd, Edinburgh
349(6251). doi:10.1126/science.aac4716 Fisher RA (1926) The arrangement of field experiments. J Minist Agric
Anderson DM, Cembella AD, Hallegraeff GM (2012) Progress in under- (GB) 33:503–513
standing harmful algal blooms: paradigm shifts and new technolo- Fisher RA (1937) The design of experiments, 2nd edn. Oliver and Boyd,
gies for research, monitoring, and management. Ann Rev Mar Sci 4: Edinburgh
143–176 Fisher RA (1947) Development of the theory of experimental design.
Andersson R, Eriksson H, Torstensson H (2006) Similarities and differ- Proceedings of the International Statistical Conferences
ences between TQM, six sigma and lean. The TQM Magazine 18: (Washington) 3:434–439
282–296 Fisher RA (1952) Statistical methods in genetics. Heredity 6:1–12
Bacon F (1650) Novum organum scientiarum. Adriani Wyngaerden, Freedman LP, Cockburn IM, Simcoe TS (2015) The economics of repro-
London ducibility in preclinical research. PLoS Biol 13:e1002165
Bailey NTJ (1967) The mathematical approach to biology and medicine. Gaither CC, Cavazos-Gaither AE (2012) Gaither’s dictionary of scientific
John Wiley & Sons, London, New York, Sydney xiii, pg. 23 quotations, 2nd edn. Springer, New York
Barker TB (1986) Quality engineering by design: Taguchi's philosophy. Gamborg OL, Miller RA, Ojima K (1968) Nutrient requirements of sus-
Quality Progress 19:32–42 pension cultures of soybean root cells. Exp Cell Res 50:151–158
Bayart D (2001) Walter Andrew Shewhart. In: Heyde CC, Seneta E (eds) George EF (1993) Plant propagation by tissue culture. Part 1: The tech-
Statisticians of the centuries. Springer-Verlag, New York, pp. 398– nology, 2nd edn. Exegetics Limited, Edington, UK
401 Glibert PM, Burkholder JM (2011) Harmful algal blooms and eutrophi-
Begley CG, Ioannidis JP (2015) Reproducibility in science: improving cation: "strategies" for nutrient uptake and growth outside the
the standard for basic and preclinical research. Circ Res 116:116– Redfield comfort zone. Chin J Oceanol Limnol 29:724–738
126 Hallegraeff GM (2010) Ocean climate change, phytoplankton community
Box JF (1978) R.A. Fisher: the life of a scientist. Wiley, New York responses, and harmful algal blooms: a formidable predictive chal-
Box GEP, Meyer RD (1986) An analysis for unreplicated fractional fac- lenge. J Phycol 46:220–235
torials. Technometrics 28:11–18 Heller R (1953) Researches on the mineral nutrition of plant tissues. Ann
Box GEP, Wilson KB (1951) On the experimental attainment of optimum Sci Nat Bot Biol Vég, 11th Ser 14:1–223
conditions. Journal of the Royal Statistical Society Series B Johnston AE, Garner HV (1969) Broadbalk: historical introduction.
(Methodological) 13:1–45 Rothamsted Experimental Station Report for 1968 Part 2:12–25
Chernoff H (1999) Gustav Elfving's impact on experimental design. Stat Knop W (1865) Quantitative Utersuchungen Über den
Sci 14:201–205 Ernährungensprozeß der Pflanze. Landw Versuchssat 7:93–107
Chu CC, Wang CS, Sun CS, Hsu V, Yin KC, Chu CY, Bi FY (1975) Knudson L (1943) Nutrient solutions for orchid seed germination. Am
Establishment of an efficient medium for anther culture of rice Orchid Soc Bull 12:77–79
through experiments on the nitrogen source. Scient Sin 18:659– 668 Kuhfeld WF (2010) Experimental design: efficiency, coding, and choice de-
Collins FS, Tabak LA (2014) NIH plans to enhance reproducibility. signs. Available via https://support.sas.com/techsup/technote/mr2010c.
Nature 505:612–613 pdf
Davidson K, Gowen RJ, Tett P, Bresnan E, Harrison PJ, McKinney A, Landsberg JH (2002) The effects of harmful algal blooms on aquatic
Milligan S, Mills DK, Silke J, Crooks AM (2012) Harmful algal organisms. Rev Fish Sci 10:113–390
blooms: how strong is the evidence that nutrient ratios and forms Larsen K (2005) Stephen Hawking: a biography. Greenwood Press,
influence their occurrence? Estuar Coast Shelf Sci 115:399–413 Westport, Connecticut, p. 43
562 NIEDZ AND EVENS

Lloyd G, McCown B (1980) Commercially-feasible micropropagation of Quoirin M, Lepoivre P (1977) Improved media for in vitro culture of
mountain laurel, Kalmia latifolia, by use of shoot tip culture. Proc Prunus sp. Acta Hortic 78:437–442
Int Plant Propagators Soc 30:421–427 Ross G (2012) Fisher and the millionaire: the statistician and the calcu-
McGranahan GH, Driver JA, Tulecke W (1987) Tissue culture of lator. Significance 9:46–48
Juglans. In: Bonga JM, Durzan DJ (eds) Cell and tissue culture in Ruen T (2011a) 9-cube t0.svg. In: Wikimedia Commons.
forestry: case histories: gymnosperms, angiosperms and palms. https://commons.wikimedia.org/wiki/File:9-cube_t0.svg. Cited 27
Springer Netherlands, Dordrecht, pp. 261–271 June 2016
Montgomery DC (2013) Design and analysis of experiments, 8th edn. Ruen T (2011b) 10-cube t0.svg. In: Wikimedia Commons.
Wiley, Hoboken, NJ https://commons.wikimedia.org/wiki/File:10-cube_t0.svg. Cited 27
Morel G, Muller J F (1964) La culture in vitro du merstem apical de la June 2016
pommede terre. Compt Rend Acad Sci 258:5255–5250 Salsburg D (2001) The lady tasting tea. W. H. Freeman and Co., New
Murashige T, Skoog F (1962) A revised medium for rapid growth and bio York
assays with tobacco tissue cultures. Physiol Plant 15:473–497 Schenk RU, Hildebrandt AC (1972) Medium and techniques for induc-
Niedz RP, Evens TJ (2006) A solution to the problem of ion confounding tion and growth of monocotyledonous and dicotyledonous plant cell
in experimental biology. Nat Methods 3:417 cultures. Can J Bot 50:199–204
Niedz RP, Evens TJ (2007) Regulating plant tissue growth by mineral Schwarz A (2012) The becoming of the experimental mode. Scientiae
nutrition. In Vitro Cell Dev Biol Plant 43:370–381 Studia 10:65–83
Niedz RP, Evens TJ (2011) Mixture screening and mixture-amount de- Skoog F, Miller CO (1957) Chemical regulation of growth and organ
signs to determine plant growth regulator effects on shoot regener- formation in plant tissues cultured in vitro. Symp Soc Exp Biol 11:
ation from grapefruit (Citrus paradisi macf.) epicotyls. In Vitro Cell 118–131
Dev Biol Plant 47:682–694 Smith DR, King KW, Williams MR (2015) What is causing the harmful
Nitsch JP, Nitsch C (1969) Haploid plants from pollen grains. Science algal blooms in Lake Erie? J Soil Water Conserv 70:27A–29A
163:85–87 Vacin EF, Went FW (1949) Some pH changes in nutrient solutions. Bot
Nordström K (1999) The life and work of Gustav Elfving. Stat Sci 14: Gaz 110:605–613
174–193 White PR (1943) Nutrient deficiency studies and an improved inorganic
Piesk T (2014) 4-cube t0.svg. In: Wikimedia Commons. nutrient for cultivation of excised tomato roots. Growth 7:53–65
https://commons.wikimedia.org/wiki/File:4-cube_t0.svg. Cited 27 Zappa F (1979) Packard Goose (song). On: Joe's Garage. Zappa Records,
June 2016 Los Angeles