Lecture 2:

Linear and Mixed Models

Bruce Walsh lecture notes
Introduction to Mixed Models
SISG, Seattle
18 – 20 July 2018

Quick Review of the Major Points

The general linear model can be written as

y = Xb + e
• y = vector of observed dependent values
• X = Design matrix: observations of the variables in the
assumed linear model

• b = vector of unknown parameters to estimate

• e = vector of residuals (deviation from model fit),

e = y-X b
2
 y = Xb + e
Solution to b depends on the covariance structure
(= covariance matrix) of the vector e of residuals

Ordinary least squares (OLS)

• OLS: e ~ MVN(0, s2 I)
• Residuals are homoscedastic and uncorrelated,
so that we can write the cov matrix of e as Cov(e) = s2I
• the OLS estimate, OLS(b) = (XTX)-1 XTy

Generalized least squares (GLS)

• GLS: e ~ MVN(0, V)
• Residuals are heteroscedastic and/or dependent,
• GLS(b) = (XT V-1 X)-1 XTV-1y
3

BLUE
• Both the OLS and GLS solutions are also
called the Best Linear Unbiased Estimator (or
BLUE for short)
• Whether the OLS or GLS form is used
depends on the assumed covariance
structure for the residuals
– Special case of Var(e) = se2 I -- OLS
– All others, i.e., Var(e) = R -- GLS

4
 Linear Models
One tries to explain a dependent variable y as a linear
function of a number of independent (or predictor)
variables.

A multiple regression is a typical linear model,

Here e is the residual, or deviation between the true

value observed and the value predicted by the linear
model.

The (partial) regression coefficients are interpreted

as follows: a unit change in xi while holding all
other variables constant results in a change of bi in y
5

Linear Models
As with a univariate regression (y = a + bx + e), the model
parameters are typically chosen by least squares,
wherein they are chosen to minimize the sum of
squared residuals, S ei2

This unweighted sum of squared residuals assumes

an OLS error structure, so all residuals are equally
weighted (homoscedastic) and uncorrelated

If the residuals differ in variances and/or some are

correlated (GLS conditions), then we need to minimize
the weighted sum eTV-1e, which removes correlations and
gives all residuals equal variance.
6
 Linear Models in Matrix Form
Suppose we have 3 variables in a multiple regression,
with four (y,x) vectors of observations.

The design matrix X. Details of both the experimental

design and the observed values of the predictor variables
all reside solely in X

Rank of the design matrix

• With n observations and p unknowns, X is an n x p
matrix, so that XTX is p x p
• Thus, at most X can provide unique estimates for up
to p < n parameters
• The rank of X is the number of independent rows of
X. If X is of full rank, then rank = p
• A parameter is said to be estimable if we can provide
a unique estimate of it. If the rank of X is k < p, then
exactly k parameters are estimable (some as linear
combinations, e.g. b1-3b3 = 4)
• if det(XTX) = 0, then X is not of full rank
• Number of nonzero eigenvalues of XTX gives the
rank of X.

8
 Experimental design and X
• The structure of X determines not only which
parameters are estimable, but also the expected
sample variances, as Var(b) = k (XTX)-1
• Experimental design determines the structure of X
before an experiment (of course, missing data almost
always means the final X is different form the
proposed X)
• Different criteria used for an optimal design. Let V =
(XTX)-1 . The idea is to chose a design for X given the
constraints of the experiment that:
– A-optimality: minimizes tr(V)
– D-optimality: minimizes det(V)
– E-optimality: minimizes leading eigenvalue of V

Ordinary Least Squares (OLS)

When the covariance structure of the residuals has a
certain form, we solve for the vector b using OLS
If residuals follow a MVN distribution, OLS = ML solution

If the residuals are homoscedastic and uncorrelated,

s2(ei) = se2, s(ei,ej) = 0. Hence, each residual is equally
weighted,

Sum of squared
residuals can
be written as

10
 Ordinary Least Squares (OLS)

Taking (matrix) derivatives shows this is minimized by

This is the OLS estimate of the vector b

The variance-covariance estimate for the sample estimates
is

The ij-th element gives the covariance between the

estimates of bi and bj. 11

Sample Variances/Covariances
The residual variance can be estimated as

The estimated residual variance can be substituted into

To give an approximation for the sampling variance and

covariances of our estimates.

Confidence intervals follow since the vector of estimates

~ MVN(b, Vb)

12
 Example: Regression Through the Origin
yi = bxi + ei

13

Polynomial Regressions
GLM can easily handle any function of the observed
predictor variables, provided the parameters to estimate
are still linear, e.g. Y = a + b1f(x) + b2g(x) + … + e

Quadratic regression:

14
 Interaction Effects
Interaction terms (e.g. sex x age) are handled similarly

With x1 held constant, a unit change in x2 changes y

by b2 + b3x1 (i.e., the slope in x2 depends on the current
value of x1 )

Likewise, a unit change in x1 changes y by b1 + b3x2

15

The GLM lets you build your

own model!
• Suppose you want a quadratic regression
forced through the origin where the slope of
the quadratic term can vary over the sexes
(pollen vs. seed parents)
• Yi = b1xi + b2xi2 + b3sixi2
• si is an indicator (0/1) variable for the sex (0 =
male, 1 = female).
– Male slope = b2,
– Female slope = b2 + b3

16
 Generalized Least Squares (GLS)
Suppose the residuals no longer have the same
variance (i.e., display heteroscedasticity). Clearly
we do not wish to minimize the unweighted sum
of squared residuals, because those residuals with
smaller variance should receive more weight.

Likewise in the event the residuals are correlated,

we also wish to take this into account (i.e., perform
a suitable transformation to remove the correlations)
before minimizing the sum of squares.

Either of the above settings leads to a GLS solution

in place of an OLS solution.
17

In the GLS setting, the covariance matrix for the

vector e of residuals is written as R where
Rij = s(ei,ej)

The linear model becomes y = Xb + e, cov(e) = R

The GLS solution for b is

The variance-covariance of the estimated model

parameters is given by

18
 Model diagnostics
• It’s all about the residuals
• Plot the residuals
– Quick and easy screen for outliers
– Plot y or yhat on e
• Test for normality among estimated residuals
– Q-Q plot
– Wilk-Shapiro test
– If non-normal, try transformations, such as log

19

OLS, GLS summary

20
 Fixed vs. Random Effects
In linear models are are trying to accomplish two goals:
estimation the values of model parameters and estimate
any appropriate variances.

For example, in the simplest regression model,

y = a + bx + e, we estimate the values for a and b and
also the variance of e. We, of course, can also
estimate the ei = yi - (a + bxi )

Note that a/b are fixed constants are we trying to

estimate (fixed factors or fixed effects), while the
ei values are drawn from some probability distribution
(typically Normal with mean 0, variance s2e). The
ei are random effects.
21

This distinction between fixed and random effects is

extremely important in terms of how we analyzed a model.
If a parameter is a fixed constant we wish to estimate,
it is a fixed effect. If a parameter is drawn from
some probability distribution and we are trying to make
inferences on either the distribution and/or specific
realizations from this distribution, it is a random effect.

We generally speak of estimating fixed factors (BLUE) and

predicting random effects (BLUP -- best linear unbiased
Predictor)

“Mixed” models (MM) contain both fixed and random factors

y = Xb + Zu + e, u ~MVN(0,R), e ~ MVN(0,s2eI)
Key: need to specify covariance structures for MM
22
 Random effects models
• It is often useful to treat certain effects as
random, as opposed to fixed
– Suppose we have k effects. If we treat these as
fixed, we lose k degrees of freedom
– If we assume each of the k realizations are drawn
from a normal with mean zero and unknown
variance, only one degree of freedom lost --- that
for estimating the variance
• We can then predict the values of the k realizations

23

Environmental effects
• Consider yield data measured over several years in a
series of plots.
• Standard to treat year-to-year variation at a specific
site as being random effects
• Often the plot effects (mean value over years) are
also treated as random.
• For example, consider plants group in growing
region i, location j within that region, and year
(season) k for that location-region effect
– E = Ri + Lij + eijk
– Typically R can be a fixed effect, while L and e are
random effects, Lik ~ N(0,s2L) and eikj ~ N(0,s2e)

24
 Random models
• With a random model, one is assuming that
all “levels” of a factor are not observed.
Rather, some subset of values are drawn from
some underlying distribution
– For example, year to year variation in rainfall at a
location. Each year is a random sample from the
long-term distribution of rainfall values
– Typically, assume a functional form for this
underlying distribution (e.g., normal with mean 0)
and then use observations to estimate the
distribution parameters (here, the variance)

25

Random models (cont)

• Key feature:
– Only one degree of freedom used (estimate of
the variance)
– Using the fixed effects and the estimated
underlying distribution parameters, one then
predicts the actual realizations of the individual
values (i.e., the year effects)
– Assumption: the covariance structure among the
individual realizations of the realized effects. If
only a variance is assume, this implies they are
independent. If they are assumed to be
correlated, this structure must be estimated.

26
 Random models
• Let’s go back to treating yearly effects as random
• If assume these are uncorrelated, only use one
degree of freedom, but makes assumptions about
covariance structure
– Standard: Uncorrelated
– Option: some sort of autocorrelation process, say with a
yearly decay of r (must also be estimated)
• Conversely, could all be treated as fixed, but would
use k degrees of freedom for k years, but no
assumptions on their relationships (covariance
structure)

27

Identifiability
• Recall that a fixed effect is said to be
estimable if we can obtain a unique estimate
for it (either because X is of full rank or when
using a generalized inverse it returns a
unique estimate)
– Lack of estimable arises because the experiment
design confounds effects
• The analogous term for random models is
identifiability
– The variance components have unique estimates

28
 The general mixed model

Vector of fixed effects (to be estimated),

e.g., year, sex and age effects
Vector of
observations Incidence matrix for random effects
(phenotypes)

y = Xb + Zu + e Vector of residual errors

(random effects)

Incidence Vector of random

matrix for effects, such as
fixed effects individual
Breeding values
(to be estimated)

29

The general mixed model

Vector of fixed effects

Vector of
observations Incidence matrix for random effects
(phenotypes)

y = Xb + Zu + e Vector of residual errors

Incidence Vector of random

matrix for effects
fixed effects

Observe y, X, Z.

Estimate fixed effects b

Estimate random effects u, e 30

 Means & Variances for y = Xb + Zu + e
Means: E(u) = E(e) = 0, E(y) = Xb
Variances:
Let R be the covariance matrix for the
residuals. We typically assume R = s2e*I

Let G be the covariance matrix for the vector

u of random effects
The covariance matrix for y becomes
V = ZGZT + R

Hence, y ~ MVN (Xb, V)

Mean Xb due to fixed effects
Variance V due to random effects
31

Estimating fixed Effects & Predicting

Random Effects

For a mixed model, we observe y, X, and Z

b, u, R, and G are generally unknown

Two complementary estimation issues

(i) Estimation of b and u
Estimation of fixed effects
BLUE = Best Linear Unbiased Estimator
Prediction of random effects
BLUP = Best Linear Unbiased Predictor
Recall V = ZGZT + R 32
 Different statistical models
• GLM = general linear model
– OLS ordinary least squares: e ~ MVN(0,cI)
– GLS generalized least squares: e ~ MVN(0,R)
• Mixed models
– Both fixed and random effects (beyond the residual)
• Mixture models
– A weighted mixture of distributions
• Generalized linear models
– Nonlinear functions, non-normality

33

Mixture models
• Under a mixture model, an observation potentially
comes from one of several different distributions, so that
the density function is p1f1 + p2f2 + p3f3
– The mixture proportions pi sum to one
– The fi represent different distribution, e.g., normal with mean µi
and variance s2
• Mixture models come up in QTL mapping -- an
individual could have QTL genotype QQ, Qq, or qq
– See Lynch & Walsh Chapter 13
• They also come up in codon models of evolution, were
a site may be neutral, deleterious, or advantageous,
each with a different distribution of selection coefficients
– See Walsh & Lynch (volume 2A website), Chapters 10,11

34
 Generalized linear models

Typically assume non-normal distribution for

residuals, e.g., Poisson, binomial, gamma, etc 35