Chapter Three

THE CLASSICAL REGRESSION ANALYSIS

[The Multiple Linear Regression Model ]

3.1 Introduction

In simple regression we study the relationship between a dependent variable and a single
explanatory (independent variable). But it is rarely the case that economic relationships
involve just two variables. Rather a dependent variable Y can depend on a whole series
of explanatory variables or regressors. For instance, in demand studies we study the
relationship between quantity demanded of a good and price of the good, price of
substitute goods and the consumer’s income. The model we assume is:
Y i= β0 + β 1 P 1 +β 2 P2 + β 3 X i + ui -------------------- (3.1)

Where Y i= quantity demanded, P1 is price of the good, P2 is price of substitute goods,

Xi is consumer’s income, and β' s are unknown parameters and ui is the disturbance.
Equation (3.1) is a multiple regression with three explanatory variables. In general for K-
explanatory variable we can write the model as follows:
Y i= β0 +β 1 X 1i + β2 X 2i +β 3 X 3i +. . .. .. . ..+β k X ki +ui ------- (3.2)
X k =( i=1,2,3 ,. . .. .. . , K )
Where i are explanatory variables, Yi is the dependent variable

and β j ( j=0,1,2,.... (k+1)) are unknown parameters and ui is the disturbance term.
The disturbance term is of similar nature to that in simple regression, reflecting:
- the basic random nature of human responses
- errors of aggregation
- errors of measurement
- errors in specification of the mathematical form of the model and any other

(minor) factors, other than x i that might influence Y.

In this chapter we will first start our discussion with the assumptions of the multiple
regressions and we will proceed our analysis with the case of two explanatory variables
and then we will generalize the multiple regression model in the case of k-explanatory
variables using matrix algebra.

3.2 Assumptions of Multiple Regression Model

In order to specify our multiple linear regression model and proceed our analysis with
regard to this model, some assumptions are compulsory. But these assumptions are the
same as in the single explanatory variable model developed earlier except the assumption
of no perfect multicollinearity. These assumptions are:
1. Randomness of the error term: The variable u is a real random variable.

2. Zero mean of the error term: E(ui )=0

3. Hemoscedasticity: The variance of each ui is the same for all the x i values.
E(u 2 )=σ 2
i.e. i u (constant)

4. Normality of u: The values of each ui are normally distributed.

2
i.e. U i ~ N (0 , σ )

5. No auto or serial correlation: The values of ui (corresponding to Xi ) are

independent from the values of any other ui (corresponding to X ) for i j.

j

i.e. E(ui u j )=0 for x i≠ j

6. Independence of ui and Xi : Every disturbance term ui is independent of the

explanatory variables. i.e. E(ui X 1 i )=E(ui X 2 i )=0

This condition is automatically fulfilled if we assume that the values of the X’s are
a set of fixed numbers in all (hypothetical) samples.
7. No perfect multicollinearity: The explanatory variables are not perfectly linearly
correlated.
We can’t exclusively list all the assumptions but the above assumptions are some of the
basic assumptions that enable us to proceed our analysis.

3.3. A Model with two Explanatory Variables

In order to understand the nature of multiple regression model easily, we start our
analysis with the case of two explanatory variables, then extend this to the case of k-
explanatory variables.

3.3.1 Estimation of parameters of two-explanatory variables model

The model: Y =β 0 + β1 X 1 +β 2 X 2 +U i ……………………………………(3.3)

is multiple regression with two explanatory variables. The expected value of the above
model is called population regression equation i.e.
E(Y )=β 0 +β 1 X 1 + β 2 X 2 , Since E(U i )=0 . …………………................(3.4)

where β i is the population parameters. β is referred to as the intercept and β

0 1

and β2 are also some times known as regression slopes of the regression. Note that,
β2 for example measures the effect on E(Y ) of a unit change in X2 when X 1 is
held constant.

Since the population regression equation is unknown to any investigator, it has to be

estimated from sample data. Let us suppose that the sample data has been used to
estimate the population regression equation. We leave the method of estimation
unspecified for the present and merely assume that equation (3.4) has been estimated by
sample regression equation, which we write as:
Y^ = β^ 0 + β^ 1 X 1 + β^ 2 X 2 ……………………………………………….(3.5)
^
Where β j are estimates of the β j and Y^ is known as the predicted value of Y.
Now it is time to state how (3.3) is estimated. Given sample observation on Y , X 1∧ X 2
, we estimate (3.3) using the method of least square (OLS).
Y^ = β^ 0 + β^ 1 X 1i + β^ 2 X 2i + ei ……………………………………….(3.6)

is sample relation between Y , X 1∧ X 2 .

e i=Y i −Y^ =Y i − β^ 0 − β^ 1 X 1 − β^ 2 X 2 …………………………………..(3.7)

To obtain expressions for the least square estimators, we partially differentiate ∑ e2i
^ ^ ^
with respect to β 0 , β1 and β 2 and set the partial derivatives equal to zero.
∂ [ ∑ e 2i ]
=−2 ∑ ( Y i − β^ 0− β^ 1 X 1i − β^ 2 X 2i )=0
∂ β^0 ………………………. (3.8)
∂ [ ∑ e 2i ]
=−2 ∑ X 1i ( Y i− β^ 0 − β^ 1 X 1 i− β^ 1 X 1 i )=0
∂ β^1 ……………………. (3.9)
∂ [ ∑ e 2i ]
=−2 ∑ X 2i ( Y i− β^ 0 − β^ 1 X 1 i− β^ 2 X 2i )=0
∂ β^2 ………… ………..(3.10)
Summing from 1 to n, the multiple regression equation produces three Normal
Equations:

∑ Y =n β^ 0+ β^ 1 ΣX1 i + β^ 2 ΣX 2i …………………………………….(3.11)

∑ X 2i Y i= β^ 0 ΣX 1i + β^ 1 ΣX 21i + β^ 2 ΣX1 i X 1i …………………………(3.12)

∑ X 2i Y i= β^ 0 ΣX 2i + β^ 1 ΣX 1i X 2 i + β^ 2 ΣX 22i ………………………...(3.13)
^
From (3.11) we obtain β 0
β^ 0 =Ȳ − β^ 1 X̄ 1 − β^ 2 X̄ 2 ------------------------------------------------- (3.14)

Substituting (3.14) in (3.12 and 3.13), we get:

 Σx1 y . Σx 2 −Σx 1 x 2 . Σx2 y
2
β^ 1=
Σx 2 . Σx 2 −Σ( x1 x 2 )2
1 2 …………………………..…………….. (3.15)
Σx2 y . Σx 2 −Σx1 x 2 . Σx1 y
1
β^ 2=
Σx 2 . Σx 2 −Σ( x1 x 2 )2
1 2 ………………….……………………… (3.16)

We can also express β^ 1 and β^ 2 in terms of covariance and variances of

Y , X 1 and X 2
Cov( X 1 , Y ) . Var ( X 1 )−Cov ( X 1 , X 2 ) . Cov( X 2 ,Y )
β^ 1= −−−−−−−−−(3 . 23)
Var ( X 1 ). Var ( X 2 )−[ cov ( X 1 , X 2 )]2
Cov ( X 2 , Y ) . Var ( X 1 )−Cov ( X 1 , X 2 ) . Cov( X 1 ,Y )
β^ 2= 2
−−−−−−−−−( 3 . 24 )
Var ( X 1 ). Var ( X 2 )−[Cov ( X 1 , X 2 )]

3.3.2 The coefficient of determination (R2): two explanatory variables case

In the simple regression model, we introduced R2 as a measure of the proportion of
variation in the dependent variable that is explained by variation in the explanatory
variable. In multiple regression model the same measure is relevant, and the same
formulas are valid but now we talk of the proportion of variation in the dependent
variable explained by all explanatory variables included in the model. The coefficient of
determination is:
Σe 2
ESS RSS
R2 = =1− =1− i
TSS TSS Σy 2
i ------------------------------------- (3.17)
In the present model of two explanatory variables:
i.e Σe2i =Σy 2 − β^ 1 Σx 1 i y i− β^ 2 Σx2 i y i

⇒Σy2underbracealignl T⏟
otalsumof ¿ variation) ¿¿=β^ 1 Σx1i yi+ β^ 2 Σx2i yiunderbracealignl ⏟
Explained sumof ¿ variation) ¿¿¿+Σe 2 underbracealignl R⏟
esidualsumof squares ¿ ¿¿¿¿
i
square (Total ¿ square (Explained ¿ (unexplained variation) ¿ ----------------- (3.18)
^ ^
ESS β1 Σx1 i y i + β 2 Σx 2i y i
∴ R 2= =
TSS Σy 2 ----------------------------------(3.27)
As in simple regression, R2 is also viewed as a measure of the prediction ability of the
model over the sample period, or as a measure of how well the estimated regression fits
the data. The value of R2 is also equal to the squared sample correlation coefficient
^
between Y ∧Y t . Since the sample correlation coefficient measures the linear
association between two variables, if R2 is high, that means there is a close association
^
between the values of Y t and the values of predicted by the model, Y t . In this case,
the model is said to “fit” the data well. If R 2 is low, there is no association between the
^
values of Y t and the values predicted by the model, Y t and the model does not fit the
data well.

2
3.3.3 Adjusted Coefficient of Determination ( R̄ )
2
One difficulty with R is that it can be made large by adding more and more variables,
even if the variables added have no economic justification. Algebraically, it is the fact
that as the variables are added the sum of squared errors (RSS) goes down (it can remain
2
unchanged, but this is rare) and thus R goes up. If the model contains n-1 variables
2 2
then R =1. The manipulation of model just to obtain a high R is not wise. An
2
alternative measure of goodness of fit, called the adjusted R and often symbolized as
2
R̄ , is usually reported by regression programs. It is computed as:
Σe2i /n−k
2
R̄ =1− 2
Σy /n−1
=1−( 1−R2 ) ( n−1
n−k ) --------------------------------(3.28)
This measure does not always goes up when a variable is added because of the degree of
freedom term n-k is the numerator. As the number of variables k increases, RSS goes
2 2
down, but so does n-k. The effect on R̄ depends on the amount by which R falls.
While solving one problem, this corrected measure of goodness of fit unfortunately
2
introduces another one. It losses its interpretation; R̄ is no longer the percent of
2
variation explained. This modified R̄ is sometimes used and misused as a device for
selecting the appropriate set of explanatory variables.
3.4. General Linear Regression Model and Matrix Approach

So far we have discussed the regression models containing one or two explanatory
variables. Let us now generalize the model assuming that it contains k variables. It will
be of the form:
Y =β 0 + β1 X 1 +β 2 X 2 +.. . .. .+β k X k +U
There are k parameters to be estimated. The system of normal equations consist of k+1

equations, in which the unknowns are the parameters β 0 , β1 ,β 2 .......β k and the known
terms will be the sums of squares and the sums of products of all variables in the
structural equations.
Least square estimators of the unknown parameters are obtained by minimizing the sum
of the squared residuals.
Σe2i =Σ( y i − β^ 0 − β^ 1 X 1− β^ 2 X 2 −.. .. . .− β^ k X k )2

With respect to β j ( j=0,1,2 ,....(k+1))

The partial derivations are equated to zero to obtain normal equations.
∂ Σe2i
=−2 Σ( Y i− β^ 0 − β^ 1 X 1 − β^ 2 X 2 −. .. .. .− β^ k X k )=0
∂β ^
0
2
∂ Σei
=−2 Σ( Y i− β^ 0 − β^ 1 X 1 − β^ 2 X 2 −. .. .. .− β^ k X k )( x i )=0
∂ β^ 1

……………………………………………………..
2
∂ Σei
=−2 Σ( Y i− β^ 0 − β^ 1 X 1 − β^ 2 X 2 −. .. .. .− β^ k X k )( x ki )=0
∂ β^ k

The general form of the above equations (except first ) may be written as:
2
∂ Σei
=−2 Σ(Y i− β^ 0 − β^ 1 X 1 i−−−−−− β^ k X ki )=0
∂ β^ j ; where ( j=1,2,....k)
The normal equations of the general linear regression model are
ΣY i =n β^ 0 + β^ 1 ΣX 1 i + β^ 2 ΣX 2i +.. .. . .. .. . .. .. .. . .... . .. .. . .. ..+ β^ k ΣX ki
ΣY i X 1 i= β^ 0 ΣX 1i + β^ 1 ΣX ^
2 +. . .. .. .. . .. .. . .. .. . .. .. . .. .. . .. ..+ β k ΣX 1i X ki
1i
 ΣY i X 2 i= β^ 0 ΣX 21 i + β^ 1 ΣX 1i X 2i + β^ 2 ΣX ^
2 +. .. . .. .. . .+ β k ΣX 2i X ki
2i

: : : : :
: : : : :
ΣY i X ki= β^ 0 ΣX ki+ β^ 1 ΣX 1 X ki + ∑ X 2i X ki . .. .. . .. .. . .. .. . ..+ β^ k ΣX 2
i ki

Solving the above normal equations will result in algebraic complexity. But we can solve
this easily using matrix. Hence in the next section we will discuss the matrix approach to
linear regression model.

3.4.1 Matrix Approach to Linear Regression Model

The general linear regression model with k explanatory variables is written in the form:
Y i= β0 +β 1 X 1i + β2 X 2i +. . .. .. . .. .. . .+ β k X ki +U
i

where (i=1,2,3,........n) and β 0 = the intercept, β 1 to β k = partial slope coefficients

U= stochastic disturbance term and i=i th observation, ‘n’ being the size of the observation.
Since i represents the ith observation, we shall have ‘n’ number of equations with ‘n’
number of observations on each variable.
Y 1 =β 0 +β 1 X 11 +β 2 X 21+β 3 X 31 .. . .. .. . .. .. .+β k X k 1 +U 1
Y 2 =β 0 +β 1 X 12 +β 2 X 22 +β 3 X 32 . .. . .. .. . .. ..+β k X k 2 +U 2
Y 3 =β 0 +β 1 X 13 +β 2 X 23 +β 3 X 33 .. . .. .. . .. .. .+β k X k 3 +U 3
…………………………………………………...
Y n =β 0 + β 1 X 1 n +β 2 X 2n + β3 X 3 n .. .. . .. .. . .. .+β k X kn+ U n

These equations are put in matrix form as:

 Y1 1 X 11 X 21 . .. .. . . Xk1 β0 U1

[] [Y2

Yn
Y =
1 X 12
Y 3 = 1 X 13
. . .
1 X1 n
X
X 22
X 23
.
X 2n
. β
. .. .. . .
. .. .. . .
. .. .. . .
. .. .. . .
+ U
Xk2
Xk3
.
X kn
][] [ ]
β1
β2
.
βn
+
U2
U3
.
Un

In short Y = Xβ+U ……………………………………………………(3.29)

The order of matrix and vectors involved are:
Y =(n×1 ), X={ (n×(k +1) } , β={( k+1)×1 } and U =( n×1)

To derive the OLS estimators of β , under the usual (classical) assumptions mentioned
^
earlier, we define two vectors β and ‘e’ as:
^
β e1

[] []
0
^
β e2
1
^
β= and e= .
.
. .
^
β k
en

^
Y = X β+e e=Y − X β^
Thus we can write: and
We have to minimize:
n
∑ e 2i =e 21+e 22+ e23+ .. .. . .. ..+ e2n
i =1

e1

=[e 1 , e 2 .. .. . . en ]

2
=∑ ei =e ' e
[]e2
.
.
en
= e' e

e' e=(Y − X β^ )' (Y −X β^ )

^ X ' Y −Y ' X β+
=YY '− β' ^ β^ ' X ' X β^
………………….…(3.30)
^
Since β' X' Y ' is scalar (1x1), it is equal to its transpose;
 ^ X' Y =Y ' X β^
β'
e' e=Y ' Y −2 β^ ' X ' Y + β'
^ X ' X β^
-------------------------------------(3.31)
^
Minimizing e’e with respect to the elements in β
2
∂ Σei ∂( e ' e )
= =−2 X ' Y +2 X ' X β^
∂ β^ ∂ β^
∂( X ' AX )
=2 AX
Since ∂ β^ and also too 2X’A
Equating the expression to null vector 0, we obtain:
^
−2 X ' Y +2 X ' X β=0 ^ 'Y
X ' X β=X

^ X ' X )−1 X ' Y
β=( ………………………………. ………. (3.32)
^ β^ , β^ , β^ ,........ β^
Hence β is the vector of required least square estimators, 0 1 2 k .

3.4.2. Statistical Properties of the Parameters (Matrix) Approach

We have seen, in simple linear regression that the OLS estimators ^ β^ ) satisfy the
( α∧
small sample property of an estimator i.e. BLUE property. In multiple regression, the
OLS estimators also satisfy the BLUE property.
1. Linearity
2. Unbiased ness
3. Minimum variance

3.4.3. Coefficient of Determination in Matrix Form

2
The coefficient of determination ( R ) can be derived as follows.

Explained sum of squares

R2 =
Since Total sum of squares
 1
β^ ' X ' Y − ( ΣY i )2 ^
n β ' X ' Y −n Ȳ
∴ R 2= =
1 Y ' Y −n Ȳ 2
Y ' Y − ( ΣY i )2
n ……………………(3.50)

Dear Students! We hope that from the discussion made so far on multiple regression
model, in general, you may make the following summary of results.

(i) Model: Y = Xβ+U

^ X ' X )−1 X ' Y
β=(
(ii) Estimators:
(iii) Statistical properties: BLUE

(iv) Variance-covariance: var( β^ )=σ 2u ( X ' X )−1

^ X'Y
e' e=Y ' Y − β'
(v) Estimation of (e’e):
1
β^ ' X ' Y − ( ΣY i )2
n
R2 =
1 β^ ' X ' Y −n Ȳ
Y ' Y − (ΣY i )2 =
(vi) Coeff. of determination: n Y ' Y −n Ȳ

3.5. Hypothesis Testing in Multiple Regression Model

In multiple regression models we will undertake two tests of significance. One is

significance of individual parameters of the model. This test of significance is the same
as the tests discussed in simple regression model. The second test is overall significance
of the model.
3.5.1. Tests of individual significance
 2
If we invoke the assumption that U i ~. N (0 , σ ) , then we can use either the t-test or
standard error test to test a hypothesis about any individual partial regression coefficient.
To illustrate consider the following example.

^ ^ ^
Let Y = β 0 + β1 X 1 + β 2 X 2 +e i ………………………………… (3.51)

A. H 0 : β 1=0

H 1 : β 1≠0

B. H 0 : β 2 =0

H 1 : β 2≠0

The null hypothesis (A) states that, holding X2 constant X1 has no (linear) influence on
Y. Similarly hypothesis (B) states that holding X1 constant, X2 has no influence on the
dependent variable Yi.To test these null hypothesis we will use the following tests:
i- Standard error test: under this and the following testing methods we test only
^ ^
for β 1 .The test for β 2 will be done in the same way.

σ^ 2 ∑ x 22 i
SE ( β^ 1 )= √ var( β^ 1 )=

^ 1 ^
√ ∑ x 21i ∑ x 22 i−(∑ x1 x 2 )2 ; where
σ^ 2 =
2
Σei
n−3

 If SE ( β1 )> 2 β 1 , we accept the null hypothesis that is, we can conclude

that the estimate β i is not statistically significant.
^ 1 ^
 If SE ( β1 < 2 β 1 , we reject the null hypothesis that is, we can conclude that

the estimate β i is statistically significant.

Note: The smaller the standard errors, the stronger the evidence that the estimates are
statistically reliable.
^
ii. The student’s t-test: We compute the t-ratio for each β i
 β^ i −β
t∗¿ ~ t n-k
SE( β^ )
i , where n is number of observation and k is number of
parameters. If we have 3 parameters, the degree of freedom will be n-3. So;
β^ 2−β 2
t∗¿
SE( β^ )
2 ; with n-3 degree of freedom

In our null hypothesis β 2=0 , the t* becomes:

β^ 2
t∗¿
SE( β^ 2 )

 If t*<t (tabulated), we accept the null hypothesis, i.e. we can conclude that

β^ 2 is not significant and hence the regressor does not appear to contribute to

the explanation of the variations in Y.

 If t*>t (tabulated), we reject the null hypothesis and we accept the alternative

one; β^ 2 is statistically significant. Thus, the greater the value of t* the

stronger the evidence that β i is statistically significant.

3.5.2 Test of Overall Significance

Through out the previous section we were concerned with testing the significance of the
estimated partial regression coefficients individually, i.e. under the separate hypothesis
that each of the true population partial regression coefficient was zero.

In this section we extend this idea to joint test of the relevance of all the included
explanatory variables. Now consider the following:
Y =β 0 + β1 X 1 + β 2 X 2 +.. . .. .. ..+ β k X k +U i
H 0 : β 1= β2 =β 3 =.. . .. .. . .. ..=β k =0
H 1 : at least one of the β k is non-zero
This null hypothesis is a joint hypothesis that β 1 ,β 2 ,........ β k are jointly or

simultaneously equal to zero. A test of such a hypothesis is called a test of overall

significance of the observed or estimated regression line, that is, whether Y is linearly

related to X 1 , X 2 ,...... .. X k .
Can the joint hypothesis be tested by testing the significance of individual significance of

β^ i ’s as the above? The answer is no, and the reasoning is as follows.

In testing the individual significance of an observed partial regression coefficient, we

assumed implicitly that each test of significance was based on different (i.e. independent)

sample. Thus, in testing the significance of β^ 2 under the hypothesis that β 2=0 , it

was assumed tacitly that the testing was based on different sample from the one used in

testing the significance of β^ 3 under the null hypothesis that β 3=0 . But to test the

joint hypothesis of the above, we shall be violating the assumption underlying the test
procedure.

“…..testing a series of single (individual) hypothesis is not equivalent to testing those

same hypothesis. The institutive reason for this is that in a joint test of several
hypotheses any single hypothesis is affected by the information in the other hypothesis.”1

If the null hypothesis is true, we expect that the data are compliable with the conditions
placed on the parameters. Thus, there would be little change in the sum of squared errors
when the null hypothesis is assumed to be true.

R2 / k −1
F=
1−R2 /n−k …………………………………………..(3.55)

1
This implies the computed value of F can be calculated either as a ratio of ESS & TSS or
R2 & 1-R2. If the null hypothesis is not true, then the difference between RRSS and
URSS (TSS & RSS) becomes large, implying that the constraints placed on the model by
the null hypothesis have large effect on the ability of the model to fit the data, and the
value of F tends to be large. Thus, we reject the null hypothesis if the F test static
becomes too large. This value is compared with the critical value of F which leaves the
probability of α in the upper tail of the F-distribution with k-1 and n-k degree of
freedom.

If the computed value of F is greater than the critical value of F (k-1, n-k), then the
parameters of the model are jointly significant or the dependent variable Y is linearly
related to the independent variables included in the model.