1 Properties of the Least Squares Estimators

The statistical properties of the least squares estimators β̂0 and β̂1 may be
easily described. Recall that we have assumed that the error term ε in the
model y=β0 +β1 x +ε is a random variable with mean zero and variance σ 2 .
Because the values of x are fixed, y is a random variable with mean
µy|x = β0 + β1 x and variance σ 2 Therefore, the values of β̂0 and β̂1 depend
on the observed y’s; thus, the least squares estimators of the regression
coefficients may be viewed as random variables. We will investigate the bias
and variance properties of the least squares estimators β̂0 and β̂1 .

1.1 Unbiasedness
For the least squares estimators of the regression coefficients β̂0 and β̂1 , they
are considered unbiased if the following conditions are met:

E(β̂0 ) = β0 and E(β̂1 ) = β1

Proof for β̂0 :

The estimator β̂0 is:

β̂0 = ȳ − β̂1 x̄
1
P
Substituting ȳ = β0 + β1 x̄ + ϵ̄, where ϵ̄ = n
ϵi :

β̂0 = β0 + β1 x̄ + ϵ̄ − β̂1 x̄
Taking the expectation:

E[β̂0 ] = β0 + β1 x̄ − E[β̂1 ]x̄ = β0

Thus, β̂0 is an unbiased estimator of β0 .

Proof for β̂1 :

The estimator β̂1 is:

P
(xi − x̄)yi
β̂1 =
Sxx
Substituting yi = β0 + β1 xi + ϵi :

1
 P P
(xi − x̄)(β0 + β1 xi + ϵi ) (xi − x̄)ϵi
β̂1 = = β1 +
Sxx Sxx
Taking the expectation:
P
(xi − x̄)E[ϵi ]
E[β̂1 ] = β1 + = β1
Sxx
Thus, β̂1 is an unbiased estimator of β1 .

Then We have shown that E(β̂0 ) = β0 and E(β̂1 ) = β1 , meaning that the
least squares estimators β̂0 and β̂1 are unbiased.

1.2 Uncorrelatedness
The least squares estimator β̂1 and ȳ are uncorrelated. Under the normality
assumption of yi , for i = 1, 2, . . . , n, β̂1 and ȳ are normally distributed and
independent.

Proof:
 
Sxy 1
cov(β1 , ȳ) = cov , ȳ = cov(Sxy , ȳ)
Sxx Sxx
n
!
1 X
= cov (yi − ȳ)(xi − x̄), ȳ
nSxx i=1
n
!
1 X
= cov yi (xi − x̄), ȳ
nSxx i=1
n n
!
1 X X
= 2 cov yi (xi − x̄), yi
n Sxx i=1 i=1
n n
1 XX
= 2 (xi − x̄)cov [yi , yj ]
n Sxx i=1 j=1

Note that E(εi ) = 0 and εi are independent for i = 1, 2, . . . , n

cov[yi , yj ] = E [(yi − E(yi ))(yj − E(yj ))]

= E(εi εj ) = σ 2

2
 Then,
n
σ2 X
cov(β1 , ȳ) = (xi − x̄) = 0
n2 Sxx i=1

And so the least squares estimator β1 and ȳ are uncorrelated.

1.3 variance :
The variances of β̂0 and β̂0 are :

σ2 x̄2
Var(β̂0 ) = (1 + )
n Sxx

and

σ2
Var(β̂1 ) =
nSxx
Proof Var(β̂0 ):

We know that
β0 = ȳ − β1 x̄

Var(β̂0 ) = Var(ȳ − β1 x̄) = Var(ȳ) + x̄2 Var(β̂1 ) − 2Cov(ȳ, β̂1 )

σ2
Now, the variance of ȳ is just Var(ȳ) = n
, and the covariance between ȳ
and β̂1 can be shown to be zero. Thus,

σ 2 σ 2 x̄2
Var(β̂0 ) = Var(ȳ) + x̄2 Var(β̂1 ) = +
n nSxx
The final expression becomes:

σ2 x̄2
Var(β̂0 ) = (1 + )
n Sxx

Proof Var(β̂1 ):
We start by writing the general form of the variance of the slope estimator
in simple linear regression:

3
 σ2
Var(βˆ1 ) =
nSxx
 
Sxy 1
Var(β̂1 ) = Var = 2
Var(Sxy ),
Sxx Sxx
where: n
1X
Sxy = (xi − x)(yi − y)
n i=1
Since the variance of a sum of independent terms is the sum of the variances,
we get:
n
!
1 X
Var(Sxy ) = Var ( ) (xi − x)(yi − y)
n i=1
n
1 X
Var(Sxy ) = ( )2 (xi − x)2 Var(yi − y)
n i=1
n
1 X
Var(Sxy ) = ( )2 (xi − x)2 σ 2
n i=1
1
= (nSxx )σ 2
n2
Now, substituting back into the variance of β̂1 :

nSxx σ 2 σ2
Var(β̂1 ) = =
n2 Sxx
2 nSxx
The final expression becomes:

σ2
Var(β̂1 ) =
nSxx
Estimated Standard Errors :
In simple linear regression, the estimated standard error of the slope
and the estimated standard error of the intercept are

s s
σˆ2
 
1 x̄2
se(β̂1 ) = and se(β̂0 ) = σ2 +
Sx x n Sx x

SSE
respectively, where σ̂ 2 is computed from Equation σ 2 = n−2