Lesson 16: Design of Experiments

Prof. Olivier de Weck

Dr. Jaroslaw Sobieski

16-1
 Design of Experiments (DoE)

• Collection of statistical techniques providing

systematic way to sample design space
• Useful when tackling new problem in which
you know very little about design space
• Study effects of multiple input variables on 1
or more output parameters

16-2
 Design of Experiments (DoE)
(cont.)
• Often used before setting up formal
optimization problem
– Identify key drivers among potential design
variables
– Identify appropriate design variable ranges
– Identify achievable objective function values
• Often, DoE used in context of robust design
• Today, discussion is about design space
exploration

16-3
 Design of Experiments (DoE)
(cont.)
• Factors & variables
– Design variables = factors
– Values of design variables = levels
– Noise factors = variables over which we have no control:
e.g., manufacturing variation in blade thickness
– Control factors = variables we can control
e.g., nominal blade thickness
– Outputs = observations (= objective functions)

Factors
& “Experiment” Observation
levels
(Often analysis code)
16-4
 Matrix Experiments

• Each row of matrix corresponds to 1 experiment

• Each column of matrix corresponds to 1 factor
• Each experiment corresponds to different combination of factor levels &
provides 1 observation

Expt No. Factor A Factor B Observation

1 A1 B1 1

2 A1 B2 2

3 A2 B1 3

4 A2 B2 4

Here, we have 2 factors, each of which can take 2 levels

16-5
 Full-Factorial Experiment
• Specify levels for each factor
• Evaluate outputs at every combination of values
– Complete, but expensive!
n factors
ln observations Factor
Expt
l levels No. A B

1 A1 B1
2 factors, 3 levels each: 2 A1 B2

ln = 32 = 9 expts 3
4
A1
A2
B3
B1
5 A2 B2
4 factors, 3 levels each: 6 A2 B3
7 A3 B1
ln = 34 = 81 expts 8 A3 B2
9 A3 B3
16-6
 Fractional Factorial Experiments

• Due to combinatorial explosion, you cannot

usually perform full factorial experiment
• Instead, consider just some possible
combinations
• Questions to be answered
– How many experiments do I need?
– Which combination of levels should I choose?
• Need to balance experiment cost with design
space coverage

16-7
 Fractional Factorial Design

• Initially, may be useful to look at large number of

factors superficially rather than small number of
factors in detail
f1 l11, l12
f2 l 21, l 22
f1 l11, l12 , l13 , l14 , vs.

f2 l21, l 22 ,, l 23 , l24 , 
f3 l31, l32 ,, l33 , l34 , fn ln1, ln 2
Many levels Many factors

16-8
 DoE Techniques Overview
TECHNIQUE COMMENT EXPENSE
(l=# levels, n=# factors)
Full factorial design Evaluates all possible ln grows exponentially
designs with number of factors

Orthogonal arrays Don’t always seem to Moderate—depends on

work - interactions? which array

1 at a time Order of factors? 1 + n(l – 1)—cheap

Latin hypercubes Not reproducible, l—cheap

poor coverage if
divisions are large
Parameter study Captures no 1 + n(l – 1)—cheap
interactions
16-9
 Parameter Study
• Specify levels for each factor
• Change 1 factor at a time, all others at base level
• Consider each factor at every level
Factor
Expt
n factors No. A B C D
1 + n(l – 1)
1 A1 B1 C1 D1
evaluations 2 A2 B1 C1 D1
l levels
3 A3 B1 C1 D1
4 A1 B2 C1 D1
4 factors, 3 levels each: 5 A1 B3 C1 D1
6 A1 B1 C2 D1
1 + n(l – 1) = 7 A1 B1 C3 D1
8 A1 B1 C1 D2
1 + 4(3 – 1) = 9 expts 9 A1 B1 C1 D3
Baseline : A1, B1, C1, D1
16-10
 Parameter Study (cont.)

• Select the best result for each factor

Factor
Expt 1. Compare 1, 2, 3
Observation
No. A B C D
 A*
1 A1 B1 C1 D1 1 2. Compare 1, 4, 5
2 A2 B1 C1 D1 2  B*
3 A3 B1 C1 D1 3 3. Compare 1, 6, 7
4 A1 B2 C1 D1 4  C*
5 A1 B3 C1 D1 5 4. Compare 1, 8, 9
6 A1 B1 C2 D1 6  D*
7
7 A1 B1 C3 D1
“Best design” is
8 A1 B1 C1 D2 8 A*, B*, C*, D*
• Does
9
not capture
A1 B1
interaction
C1 D3
between
9
variables 16-11
 One Factor At a Time (OFAT)
• Change 1st factor, all others at base value
• If output is improved, keep new level for that factor
• Move on to next factor & repeat
Factor
n factors Expt
No. A B C D
1+n(l – 1)
evaluations 1 A1 B1 C1 D1
l levels 2 A2 B1 C1 D1
3 A3 B1 C1 D1
4 factors, 3 levels each: 4 A* B2 C1 D1
5 A* B3 C1 D1
1 + n(l – 1) = 6 A* B* C2 D1
7 A* B* C3 D1
1 + 4(3 – 1) = 9 expts 8 A* B* C* D2
9 A* B* C* D3
• Result depends on order of factors 16-12
 Orthogonal Arrays

• Specify levels for each factor

• Use arrays to choose subset of full-factorial experiment
• Subset selected to maintain orthogonality between
factors
n factors
subset of ln evaluations
l levels

• Does not capture all interactions, but is efficient

• Experiment is balanced

16-13
 Orthogonal Arrays (cont.)
Factor
Expt
Factor No.
Expt A B C D
No. A B C 1 A1 B1 C1 D1

1 A1 B1 C1 2 A1 B2 C2 D2
3 A1 B3 C3 D3
2 A1 B2 C2
4 A2 B1 C2 D3
3 A2 B1 C2 5 A2 B2 C3 D1
4 A2 B2 C1 6 A2 B3 C1 D2
7 A3 B1 C3 D2
L4(2 )
3
8 A3 B2 C1 D3
9 A3 B3 C2 D1

4 expts 3 factors L9(34)

2 levels
9 expts 4 factors
3 levels
16-14
 Activity 16–1
Design Experiment

(Experiment reprinted courtesy of Prof. Steven D. Eppinger) 16-15

 Activity 16–1 (cont.)

• Objective
– Discover & understand which airplane
configuration flies maximum distance
• Desired outcome
– Understand value of designing experiments
when resources are limited

16-16
 Effects

• Once experiments have been performed,

results can be used to calculate effects
• Effect of factor is change in response as level
of factor is changed:
– Main effects: averaged individual measures of
effects of factors
– Interaction effects: effect of factor depends on level
of another factor
• Often, effect is determined for change from
minus level (–) to plus level (+) (2-level
experiments)
16-17
 Effects (cont.)

• Consider this experiment:

– We are studying effect of 3 factors on price of aircraft
– Factors are number of seats, range & aircraft
manufacturer
– Each factor can take 2 levels:
Factor 1: Seats 100 < S1 < 150 150 < S2 < 200

Factor 2: Range (nm) 2,000 < R1 < 2,800 2,800 < R2 <
3,500

Factor 3: Manufacturer M1 = Boeing M2 = Airbus

16-18
 Main Effects

• Main effect of factor is effect of that factor on output

averaged across levels of other factors
Expt Seats Range Mfr Price
No. (S) (R) (M) (observation)
1 S1 R1 M1 P1
L8(23) 2 S1 R1 M2 P2
(full factorial 3 S1 R2 M1 P3
design) P4
4 S1 R2 M2
5 S2 R1 M1 P5
6 S2 R1 M2 P6
7 S2 R2 M1 P7
8 S2 R2 M2 P8
16-19
 Main Effects (cont.)
• Question: What is main effect of manufacturer?
• That is, from experiments, can we predict how price is
affected by whether Boeing or Airbus makes aircraft?
Expt Seats Range Mfr Price
No. (S) (R) (M) (observation)
1 S1 R1 M1 P1 Expts 1 & 2 differ only
2 S1 R1 M2 P2 in manufacturer
3 S1 R2 M1 P3
4 S1 R2 M2 P4
5 S2 R1 M1 P5
6 S2 R1 M2 P6
7 S2 R2 M1 P7
8 S2 R2 M2 P8

(P2 -P1)+ (P4 -P3 )+ (P6 -P5 )+ (P8 -P7 ) Main effect of
4
= manufacturer 16-20
 Main Effects: Another Interpretation
Overall mean P1  P2 + P3  P4 + P5  P6 + P7  P8
m
response: 8
Avg over all expts P  P3  P5  P7
mM 1  1
when M = M1: 4

Effect of mfr Effect of mfr = mM 2  m

= mM 1  m
level M1 level M2
Effect of factor level can be defined for multiple levels

Main effect Main effect of factor is defined as

= mM 2  mM 1 difference between 2 levels
of mfr
NOTE: Main effect should be interpreted individually only if
variable does not appear to interact with other variables 16-21
 Main Effect Example
Expt Seats Range Mfr Price
Aircraft
No. (S) (R) (M) ($M)
1 717 S1 R1 M1 24.0
2 A318-100 S1 R1 M2 29.3
3 737-700 S1 R2 M1 33.0
4 A319-100 S1 R2 M2 35.0
5 737-900 S2 R1 M1 43.7
6 A321-200 S2 R1 M2 48.0
7 737-800 S2 R2 M1 39.1
8 A320-200 S2 R2 M2 38.0
Sources: 100 < S1 < 150 150 < S2 <
Seats/Range data: Boeing Quick Looks 200
Price data: Aircraft Value News 2,000 < R1 < 2,800 2,800 < R2 <
Airline Monitor, May 2001. Reprinted with 3,500
permission of Prof. Olivier de Weck M1 = Boeing M2 = Airbus
16-22
 Main Effect Example (cont.)

Overall mean price = 1/8*(24.0 + 29.3 + 33.0 + 35.0 + 43.7 + 48.0 + 39.1 + 38.0)
= 36.26

Mean of experiments with M1 = 1/4*(24.0 + 33.0 + 43.7 + 39.1)

= 34.95
Mean of experiments with M2 = 1/4*(29.3 + 35.0 + 48.0 + 38.0)
= 37.58

• Main effect of Boeing (M1) = 34.95 – 36.26 = -1.3

• Main effect of Airbus (M2) = 37.58 – 36.26 = 1.3
• Main effect of manufacturer = 37.58 – 34.95 = 2.6

Interpretation?
16-23
 Interaction Effects

• Can also measure interaction effects between

factors
• Answers question: Does effect of factor depend on
level of another factor?
– For example, does effect of manufacturer depend on
whether we consider shorter range or longer range
aircraft?

Mfr  range Avg mfr – Avg mfr effect

=
interaction effect with with range 1
range 2 2

16-24
 Interaction Effects (cont.)

• Interaction between manufacturer & range is

defined as half the difference between average
manufacturer effect with range 2 & average
manufacturer effect with range 1

Mfr  range Avg mfr effect

– Avg mfr effect
=
interaction with range 2 with range 1
2

16-25
 Interaction Effects (cont.)
Expt Seats Range Mfr Price
No. (S) (R) (M) ($M)
1 S1 R1 M1 24.0
2 S1 R1 M2 29.3
Range R1 : Expts 1, 2, 5, 6 3 S1 R2 M1 33.0
4 S1 R2 M2 35.0
Range R2 : Expts 3, 4, 7, 8 5 S2 R1 M1 43.7
6 S2 R1 M2 48.0
7 S2 R2 M1 39.1
8 S2 R2 M2 38.0

Avg mfr effect (P2 -P1)+ (P6 -P5 ) (29.3-24.0)+ (48.0-43.7)

with range 1
   4.8
2 2

Avg mfr effect (P4 -P3 )+ (P8 -P7 )  (35.0-33.0)+ (38.0-39.1)  0.45
with range 2  2 2

Mfr  range 0.45  4.8 Interpretation?

  2.2
interaction 2 16-26
 Interpretation of Effects
Range Main effects are Expt
No.
Seats
(S)
Range
(R)
Mfr
(M)
difference between 1 S1 R1 M1

2 averages 2 S1 R1 M2
3 S1 R2 M1
Seats
4 S1 R2 M2
Mfr 3 7 5 S2 R1 M1
3 7
6 S2 R1 M2
R2 7 S2 R2 M1
4 8
4 8 8 S2 R2 M2

3 7
S1 S2
M1
4 8
1 5
1 5
R1
2 6
2 6
Seats Range
M2
1 5

Main effect  (P8 +P6 +P4 +P2 )-(P7 +P5 +P3 +P1)
2 6
of mfr 4
Manufacturer
From Fig 10.2 Box, Hunter & Hunter. Reprinted with permission of Prof. Olivier de Weck 16-27
PM SG-56
 Interpretation of Effects (cont.)
Expt Seats Range Mfr
Range Interaction effects are No. (S) (R) (M)

also difference between 2 1 S1 R1 M1

averages, but planes are 2 S1 R1 M2

Seats no longer parallel 3 S1 R2 M1

4 S1 R2 M2
5 S2 R1 M1
Mfr 3 7 3 7 6 S2 R1 M2
7 S2 R2 M1

4 8 4 8 8 S2 R2 M2

3 7

4 8

1 5 1 5

2 6 2 6
Seats  range Mfr  seats 1 5

Mfr  range (P +P +P +P )-(P7 +P6 +P3 +P2 ) 2 6

 8 5 4 1
interaction 4 Mfr  range
From Fig 10.2 Box, Hunter & Hunter. Reprinted with permission of Prof. Olivier de Weck 16-28
PM SG-57
 Discussion

• How does design of experiments apply…

– To Orion design problem?
– To egg-drop exercise?

16-29
 Summary

• Role of designing experiments to explore design

space
• With simple models or prototypes:
– Conduct simple experiments to understand main
effects of model
– Understand interaction effects between factors
• Statistical techniques provide systematic way to
sample design space

16-30