Learning Outcomes

• Based on the material in this Lecture, you will be able to:

• Explain what is econometrics and what it is used for

• Distinguish between types of data

• Estimate a simple linear regression model with the Ordinary Least

Squares (OLS) estimation method

• Discuss the interpretation of the slope and intercept of the estimated

equation

• Make prediction with the estimated equation

• Explain the Gauss-Markov assumptions and the implication of the Gauss-

Markov Theorem

@Elisabetta Pellini 2
Quantitative Methods and Econometrics

• In economics and business disciplines of accounting, finance, and

management quantitative research is the systematic empirical
investigation of economic variables and their relationships via
statistical and mathematical methods

• These methods are collectively known as quantitative methods

• Econometrics, which literally means measurement in economics,

refers to the application of statistical methods to analyze relationships
between economic/financial/accounting variables

@Elisabetta Pellini 3
Quantitative Methods and Econometrics

• Econometrics is used for:

1. Prediction or Forecasting: forecasting demand levels, forecasting asset
returns, forecasting volatility
2. Quantifying the impact of a change in a variable on another variable: if
inflation goes up by one p.p. by how much consumption will decrease? if a
company’s leverage goes up by one p.p. by how much its stock price will
change?
3. Testing hypothesis: testing whether a stock is as risky as the whole market
rather than riskier than the whole market; testing whether dividends
announcements have no effect on stock prices; testing if companies managed
by female CEOs perform as well as companies managed by male CEOs rather
than better

@Elisabetta Pellini 4
Economic and Econometric Models
• How can we structure an empirical analysis?

• The first step is that we formulate a theoretical model of relationship

between variables. This model could come from:

1. Economic or financial theory:

• Consumer Theory: demand of a given good depends of price, income,…
𝑄𝑄𝑑𝑑 = 𝑓𝑓 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝, 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠_𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝, 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐_𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝, 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖, 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡

• Finance Theory: Capital Asset Pricing Model (CAPM): Expected return on a

stock i is equal to the free-risk rate plus a risk premium

𝐸𝐸 𝑟𝑟𝑖𝑖 = 𝑟𝑟𝑓𝑓 + 𝛽𝛽 𝐸𝐸 𝑟𝑟𝑚𝑚 − 𝑟𝑟𝑓𝑓

2. The researcher’s intuition that it exists a relationship between

variables that can be used to explain how changes in a variable
determine changes in another variable

@Elisabetta Pellini 5
Economic and Econometric Models

• The second step consists in formulating an econometric model that

specifies the statistical relationship that is believed to hold between the
variables of interest

• An econometric model is a simplified representation of

economic/financial problems, thus:
• When formulating econometric models the are variables that we cannot
observe

• We must make an assumption about the form of the function that relates the
variables

@Elisabetta Pellini 6
Economic and Econometric Models

• Econometric Model of Demand: demand of a given good depends on price,

income and on many other variables that we cannot observe
𝑞𝑞 = 𝛽𝛽0 + 𝛽𝛽1 𝑝𝑝 + 𝛽𝛽2 𝑖𝑖𝑖𝑖𝑖𝑖 + 𝜀𝜀

• Econometric Model of the CAPM: excess returns for a stock depend on the
market’s excess return and on many other factors we cannot observe

𝑟𝑟𝑖𝑖 − 𝑟𝑟𝑓𝑓 = 𝛽𝛽0 + 𝛽𝛽1 𝑟𝑟𝑚𝑚 − 𝑟𝑟𝑓𝑓 + 𝜀𝜀

• 𝜀𝜀 is called the error term and it collects all unobserved factors affecting
the variable on the left-hand side of the equation and any approximation
error that arises because the linear functional form we have assumed may
be only an approximation to reality

@Elisabetta Pellini 7
Economic and Econometric Models

• An econometric model describes a relationship which is supposed to

hold in the population of interest

• Because it is impossible to examine every population member, then the

third step of an empirical analysis consists in collecting a sample
data from the population

• The fourth step consists in using the data collected to estimate the
parameters of the population model (this is also called model
estimation)

@Elisabetta Pellini 8
Economic and Econometric Models

• The fifth step is the statistical evaluation of the estimated model

• The sixth step is the evaluation of the estimated model in the light of the
theory: are the parameters estimates of the sign and size required by the
theory?

• Finally if a model is adequate, one can use it to conduct various types of

analyses

@Elisabetta Pellini 9
Economic and Econometric Models

Economic or Financial Theory or Intuition

Formulation of an Econometric Model

Collection of Data

Model Estimation

Is the Model Statistically Adequate?

No Yes

Reformulate Model Interpret Model

Use for Analysis

@Elisabetta Pellini 10
Data

• Econometrics requires data

• Different kinds of economic datasets:
• Cross-sectional dataset: it consists of a sample of individuals (consumers,
firms, banks, countries, cities….) of whom we observe the features
(income, wage, education,…) at a given point in time

• Time series dataset: observations of a variable or several variables over time

(daily stock prices, annual GDP, monthly inflation,...). A feature of time
series is the data frequency

• Panel dataset: also known as longitudinal dataset, it consists of time series

data collected for each cross-sectional member

@Elisabetta Pellini 11
Data

Cross-sectional dataset on wages and other characteristics of workers

Time series dataset on stock market

indeces returns

@Elisabetta Pellini 12
Data

Panel dataset on quarterly economic indicators for

European countries

Panel dataset on daily stock prices for

several company

@Elisabetta Pellini 13
The Simple Linear Regression Model

• In the previous slides we looked at some example of econometric models

• These are examples of Linear Regression Models, a class of econometric

models

• In the next slides we will examine in detail the Simple Linear Regression
Model

@Elisabetta Pellini 14
The Simple Linear Regression Model

• Suppose that we want to model the relationship between two variables

y and x representing some population, for example CEO salary and ROE
of CEO’s firm

• In other words, suppose we want to construct a model that will explain

how CEO salary (y) varies as ROE (x) varies

@Elisabetta Pellini 15
The Simple Linear Regression Model

• We face three issues:

• Many factors other than ROE may determine CEO’s salary variability
(CEO’s tenure, firm’s size, type of industry, gender, CEO’s skills …), how do
we take these factors into account?

• What type of functional relationship can we assume to exist between

ROE and CEO salary?

• How can we be sure of capturing the effect of a change in ROE on CEO

salary all other factors being equal?

@Elisabetta Pellini 16
The Simple Linear Regression Model

• The following model resolves these issues:

𝑦𝑦 = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥 + 𝜀𝜀
Epsilon

• This equation is called the simple linear regression model or bivariate

linear regression model because it relates the two variables x and y

• The variable 𝜺𝜺, called the error term or disturbance, captures all factors
affecting y other than x and any approximation error that may arise
because we approximate the relationship between x and y with a linear
form

@Elisabetta Pellini 17
The Simple Linear Regression Model

• The variables y and x have several different names used interchangeably:

y x
Dependent variable Independent variable
Regressand Regressor
Effect variable Causal variable
Explained variable Explanatory variable

@Elisabetta Pellini 18
The Simple Linear Regression Model

• If the all the factors in 𝜺𝜺 are held fixed, so that the change in 𝜀𝜀 is zero (in
formulae 𝚫𝚫𝜺𝜺 = 𝟎𝟎), then x has a linear effect on y

Δ𝑦𝑦 = 𝛽𝛽1 Δ𝑥𝑥 since Δ𝜀𝜀 = 0

and the change in y (namely Δ𝑦𝑦) is simply 𝜷𝜷𝟏𝟏 multiplied by the change in x
(that is Δ𝑥𝑥)

• 𝜷𝜷𝟏𝟏 measures the ceteris paribus (all other factors being equal) effect of x
on y. It is the slope parameter in the relationship between y and x

• 𝜷𝜷𝟎𝟎 is the intercept parameter also called the constant term

@Elisabetta Pellini 19
The Simple Linear Regression Model

• Can we really be sure that 𝜷𝜷𝟏𝟏 measures the ceteris paribus effect of x on
y? Not really, since we cannot hold all other factors constant, when in
fact we do not know them

• We can give 𝛽𝛽1 a ceteris paribus effect interpretation only by making

some assumptions about the unobservable 𝜺𝜺

@Elisabetta Pellini 20
The Simple Linear Regression Model

• A first fundamental assumption about the unobservable 𝜺𝜺 is that the

average value of 𝜀𝜀 in the population is zero:

𝐸𝐸 𝜀𝜀 = 0

• For example, with reference to the CEO salary equation, we can assume
that unobservable things like average skills are zero in the population of
all CEOs

@Elisabetta Pellini 21
The Simple Linear Regression Model

• A second fundamental assumption refers to how 𝜺𝜺 and x are related

• Since 𝜺𝜺 and x are random variables, we can define the conditional

distribution of 𝜺𝜺 given any value of x, and we can define the expected
value of 𝜺𝜺 given any value of x

• The crucial assumption we make is that the expected value of 𝜀𝜀 does not
depend on the value of x and it is the same for any value of x. This
corresponds to writing:
𝐸𝐸 𝜀𝜀|𝑥𝑥 = 𝐸𝐸 𝜀𝜀

• When this assumption holds, we say that 𝜀𝜀 is mean independent of x. By

combining the two assumptions, we get the zero conditional mean
assumption
𝐸𝐸 𝜀𝜀|𝑥𝑥 = 𝐸𝐸 𝜀𝜀 = 0
@Elisabetta Pellini 22
The Simple Linear Regression Model

• Assume that 𝜺𝜺 is the same as innate ability of CEOs. Then the zero
conditional mean assumption 𝐸𝐸 𝜀𝜀|𝑥𝑥 = 𝐸𝐸 𝜀𝜀 = 0

says that the average level of ability of CEOs is the same, regardless the
level of firm’s ROE

• This also means that knowledge of ROE does not help in predicting
CEO’s ability

• This assumption implies zero covariance cov 𝑥𝑥, 𝜀𝜀 = 0 and hence zero
correlation between 𝜀𝜀 and x

@Elisabetta Pellini 23
The Simple Linear Regression Model

• If the zero conditional mean assumption 𝐸𝐸 𝜀𝜀|𝑥𝑥 = 0 holds, then the

expected value of y given x is:
𝐸𝐸 𝑦𝑦|𝑥𝑥 = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥

• This is called the population regression function (PRF)

• β1 can be thought of as the effect of a change in x on expected value of y,

holding all else constant:

Δ𝐸𝐸(𝑦𝑦|𝑥𝑥)
= 𝛽𝛽1
Δ𝑥𝑥
• The PRF is able to tell us only how the average value of y changes with x

@Elisabetta Pellini 24
The Simple Linear Regression Model

• Using the example of CEO salary the PRF is:

𝐸𝐸 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠|𝑅𝑅𝑅𝑅𝑅𝑅 = 𝛽𝛽0 + 𝛽𝛽1 𝑅𝑅𝑅𝑅𝑅𝑅

• β1 can be thought of as the effect of a change in ROE on expected value

of salary, holding all else constant:

Δ𝐸𝐸(𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠|𝑅𝑅𝑅𝑅𝑅𝑅)
= 𝛽𝛽1
Δ𝑅𝑅𝑅𝑅𝑅𝑅
• The PRF is able to tell us that if ROE takes on a given value, than the
expected value of salary will be 𝛽𝛽0 + 𝛽𝛽1 𝑅𝑅𝑅𝑅𝑅𝑅. Whether a specific CEO
earns more or less the average salary will depend of the unobservable
factors

@Elisabetta Pellini 25
The Simple Linear Regression Model

• What if this assumption is not true?

• If 𝐸𝐸 𝜀𝜀|𝑥𝑥 ≠ 0 then as x changes so does 𝜀𝜀 and we cannot claim that 𝛽𝛽1

measures the effect of a change x on the expected value of y holding all
other factors constant

@Elisabetta Pellini 26
The Simple Linear Regression Model

• We can look at the simple linear regression model as breaking y in two

parts:
𝑦𝑦 = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥 + 𝜀𝜀 = 𝑬𝑬 𝒚𝒚|𝒙𝒙 + 𝜺𝜺

1. 𝐸𝐸 𝑦𝑦|𝑥𝑥 = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥 systematic part. This part of y varies systematically

with the variable x. It is the part of y explained by x

2. 𝜀𝜀 random or unsystematic part. This part of y that varies randomly

@Elisabetta Pellini 27
The Simple Linear Regression Model

The curves in grey represent fictitious conditional distributions of CEOs

salary given any level of ROE.
The blue line is the related PFR, which tells us the expected value of CEO
𝑓𝑓𝑌𝑌|𝑋𝑋 salary for any level of ROE. The slope of the blue line 𝛽𝛽1 represents the change in
the expected CEOs salary when ROE changes by one percentage point

Fictitious probability
distribution of CEOs salary
for firms with 20% ROE

𝜇𝜇𝑦𝑦|10
𝜇𝜇𝑦𝑦|20

@Elisabetta Pellini 28
Estimating the Regression Parameters: OLS

• With the simple linear regression model 𝑦𝑦 = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥 + 𝜀𝜀 we describe

a relationship that we assume to hold for all the individuals of the
population

• Since a population is a large entity, it is impossible (or impossibly

costly) to have the data for all its members

• However, with a sample of data from the population we can estimate 𝛽𝛽0
and 𝛽𝛽1

• Let 𝑥𝑥𝑖𝑖 , 𝑦𝑦𝑖𝑖 : 𝑖𝑖 = 1,2, … , 𝑛𝑛 denote a random sample of size n from the
population, we can write:
𝑦𝑦𝑖𝑖 = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥𝑖𝑖 + 𝜀𝜀𝑖𝑖

@Elisabetta Pellini 29
Estimating the Regression Parameters: OLS

• Suppose we have data on salary and ROE from a random sample of 15 CEOs

• The values of salary are in

thousands of $, ROE is in %

• Thus, for the first CEO, annual

salary is $1,095,000 and the firm
where she/he works has a 14.1%
ROE

@Elisabetta Pellini 30
Estimating the Regression Parameters: OLS

• As a first step we plot one variable against the other with a scatter plot:

• We observe a positive relationship between ROE and CEO salary

2000 1500
salary, thousands $
1000 500

0 20 40 60
ROE

@Elisabetta Pellini 31
Estimating the Regression Parameters: OLS

• How can we use this data to estimate the unknown parameters of the
PRF 𝐸𝐸 𝑦𝑦|𝑥𝑥 = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥?

2000 1500
salary, thousands $
1000 500

0 20 40 60
ROE

• We could draw a line through the middle of the points and use the intercept
and the slope of this line as estimates of the parameters of the PRF

@Elisabetta Pellini 32
Estimating the Regression Parameters: OLS

• We need a rule that tells us how to draw the line. Many rules are possible,
but the one that we will used is based on the ordinary least squares (OLS)
principle

OLS asserts that to fit a line to the y

data we should make the sum of the

squares of the vertical distances
from each point to the line as small
as possible. The distances are
squared to prevent positive distances
from being cancelled by negative
distances
x

@Elisabetta Pellini 33
Estimating the Regression Parameters: OLS

• The intercept and slope of this line are the

OLS estimates of 𝛽𝛽0 and 𝛽𝛽1 y
Fitted line
𝑦𝑦�𝑖𝑖 = 𝛽𝛽̂0 + 𝛽𝛽̂1 𝑥𝑥𝑖𝑖

• The line 𝑦𝑦�𝑖𝑖 = 𝛽𝛽̂0 + 𝛽𝛽̂1 𝑥𝑥𝑖𝑖 is called the fitted line 𝑦𝑦3
𝜀𝜀3̂
a hat (ˆ) over a variable or parameter denotes an 𝑦𝑦1
𝑦𝑦�2
estimated value 𝑦𝑦�3
𝜀𝜀1̂
𝜀𝜀2̂
𝑦𝑦2
𝑦𝑦�1

• 𝜀𝜀𝑖𝑖̂ is called the residual of the regression. 𝜀𝜀𝑖𝑖̂ is

x
the difference between the actual value of y
and the corresponding fitted or predicted Fitted line 𝑦𝑦�𝑖𝑖 = 𝛽𝛽�0 + 𝛽𝛽�1 𝑥𝑥𝑖𝑖 is also
called the sample regression
value for this data point: 𝜀𝜀𝑖𝑖̂ = 𝑦𝑦𝑖𝑖 − 𝑦𝑦�𝑖𝑖 function because it is the estimated
version of the population
regression function

@Elisabetta Pellini 34
Estimating the Regression Parameters: OLS

• The OLS principle asserts that to fit a line to the data we should make the
sum of the squares of the vertical distances from each point to the line
as small as possible. The vertical distances are called residuals, thus the
sum of squared residuals is:
𝑛𝑛
� 𝜀𝜀𝑖𝑖̂ 2
𝑖𝑖=1

Which is better known as the Residual Sum of Squares (RSS)

𝑛𝑛 𝑛𝑛 𝑛𝑛 2
𝑅𝑅𝑅𝑅𝑅𝑅 = � 𝜀𝜀𝑖𝑖̂2 =� 𝑦𝑦𝑖𝑖 − 𝑦𝑦�𝑖𝑖 2
=� 𝑦𝑦𝑖𝑖 − 𝛽𝛽̂0 − 𝛽𝛽̂1 𝑥𝑥𝑖𝑖
𝑖𝑖=1 𝑖𝑖=1 𝑖𝑖=1

By minimising the RSS we get the formulae that allow us to estimate the
unknown population parameters given a sample of data

@Elisabetta Pellini 35
Estimating the Regression Parameters: OLS

𝑛𝑛
min � (𝑦𝑦𝑖𝑖 − 𝛽𝛽̂0 − 𝛽𝛽̂1 𝑥𝑥𝑖𝑖 )2
𝛽𝛽 �1
�0 ,𝛽𝛽 𝑖𝑖=1

• Take the first order partial derivatives (FOC) using the chain rule and set
them =0.

𝜕𝜕𝑅𝑅𝑅𝑅𝑅𝑅
�0 = −2 ∑𝑛𝑛𝑖𝑖=1(𝑦𝑦𝑖𝑖 − 𝛽𝛽̂0 − 𝛽𝛽̂1 𝑥𝑥𝑖𝑖 ) = 0 (eq. 1)
𝜕𝜕𝛽𝛽

𝜕𝜕𝑅𝑅𝑅𝑅𝑅𝑅
�1 = −2 ∑𝑛𝑛𝑖𝑖=1 𝑥𝑥𝑖𝑖 (𝑦𝑦𝑖𝑖 − 𝛽𝛽̂0 − 𝛽𝛽̂1 𝑥𝑥𝑖𝑖 ) = 0 (eq. 2)
𝜕𝜕𝛽𝛽

@Elisabetta Pellini 36
Estimating the Regression Parameters: OLS

• From equation (1) we have:

𝑛𝑛 𝑛𝑛
� 𝑦𝑦𝑖𝑖 − 𝑛𝑛𝛽𝛽̂0 − 𝛽𝛽̂1 � 𝑥𝑥𝑖𝑖 = 0
𝑖𝑖=1 𝑖𝑖=1

• Since ∑𝑛𝑛𝑖𝑖=1 𝑦𝑦𝑖𝑖 = 𝑛𝑛𝑦𝑦� (where 𝑦𝑦� is the mean of y), then we re-write the above
as:
𝑛𝑛𝑦𝑦� − 𝑛𝑛𝛽𝛽̂0 − 𝛽𝛽̂1 𝑛𝑛𝑥𝑥̅ = 0

• From which we get that 𝛽𝛽̂0 (the estimated intercept) is:

𝛽𝛽̂0 = 𝑦𝑦� − 𝛽𝛽̂1 𝑥𝑥̅

@Elisabetta Pellini 37
Estimating the Regression Parameters: OLS

• Now we substitute 𝛽𝛽̂0 = 𝑦𝑦� − 𝛽𝛽̂1 𝑥𝑥̅ into equation (2)

𝑛𝑛
� 𝑥𝑥𝑖𝑖 (𝑦𝑦𝑖𝑖 − 𝑦𝑦� + 𝛽𝛽̂1 𝑥𝑥̅ − 𝛽𝛽̂1 𝑥𝑥𝑖𝑖 ) = 0
𝑖𝑖=1

• So as to get:
𝑛𝑛 𝑛𝑛 𝑛𝑛 𝑛𝑛
� 𝑥𝑥𝑖𝑖 𝑦𝑦𝑖𝑖 − 𝑦𝑦� � 𝑥𝑥𝑖𝑖 + 𝛽𝛽̂1 𝑥𝑥̅ � 𝑥𝑥𝑖𝑖 − 𝛽𝛽̂1 � 𝑥𝑥𝑖𝑖2 = 0
𝑖𝑖=1 𝑖𝑖=1 𝑖𝑖=1 𝑖𝑖=1

That we can re-write as:

𝑛𝑛 𝑛𝑛 𝑛𝑛 𝑛𝑛
𝛽𝛽̂1 � 𝑥𝑥𝑖𝑖2 − 𝛽𝛽̂1 𝑥𝑥̅ � 𝑥𝑥𝑖𝑖 = � 𝑥𝑥𝑖𝑖 𝑦𝑦𝑖𝑖 − 𝑦𝑦� � 𝑥𝑥𝑖𝑖
𝑖𝑖=1 𝑖𝑖=1 𝑖𝑖=1 𝑖𝑖=1

𝑛𝑛 𝑛𝑛
𝛽𝛽̂1 � 𝑥𝑥𝑖𝑖2 − 𝛽𝛽̂1 𝑛𝑛𝑥𝑥̅ 2 =� 𝑥𝑥𝑖𝑖 𝑦𝑦𝑖𝑖 − 𝑛𝑛𝑦𝑦� 𝑥𝑥̅
𝑖𝑖=1 𝑖𝑖=1

@Elisabetta Pellini 38
Estimating the Regression Parameters: OLS

∑ 𝑛𝑛
𝑥𝑥𝑖𝑖 𝑦𝑦𝑖𝑖 −𝑛𝑛𝑦𝑦� 𝑥𝑥̅
• 𝛽𝛽̂1 = ∑𝑖𝑖=1
𝑛𝑛 2 ̅2
which can be also written as
𝑖𝑖=1 𝑥𝑥𝑖𝑖 −𝑛𝑛𝑥𝑥

∑𝑛𝑛𝑖𝑖=1 𝑥𝑥𝑖𝑖 − 𝑥𝑥̅ 𝑦𝑦𝑖𝑖 − 𝑦𝑦�

𝛽𝛽̂1 =
∑𝑛𝑛𝑖𝑖=1 𝑥𝑥𝑖𝑖 − 𝑥𝑥̅ 2

• This is equivalent to the sample covariance between x and y divided by the

sample variance of x:

𝑠𝑠𝑥𝑥𝑥𝑥
𝛽𝛽̂1 =
𝑠𝑠𝑥𝑥2
𝑠𝑠𝑥𝑥𝑥𝑥 𝑠𝑠𝑥𝑥𝑥𝑥 𝑠𝑠𝑦𝑦 𝑠𝑠𝑦𝑦
• But also: 𝛽𝛽̂1 = 2 = � = 𝑟𝑟𝑥𝑥𝑥𝑥 � (sample correlation multiplied by the
𝑠𝑠𝑥𝑥 𝑠𝑠𝑥𝑥 𝑠𝑠𝑦𝑦 𝑠𝑠𝑥𝑥 𝑠𝑠𝑥𝑥

ratio of the sample standard deviations)

@Elisabetta Pellini 39
Estimating the Regression Parameters: OLS

• The formulae:

𝑛𝑛
∑𝑖𝑖=1 𝑥𝑥𝑖𝑖 −𝑥𝑥̅ 𝑦𝑦𝑖𝑖 −𝑦𝑦�
𝛽𝛽̂0 = 𝑦𝑦� − 𝛽𝛽̂1 𝑥𝑥̅ and ̂
𝛽𝛽1 = ∑𝑛𝑛 2
𝑖𝑖=1 𝑥𝑥𝑖𝑖 −𝑥𝑥̅

Are known as the OLS ESTIMATORS of the true values of 𝛽𝛽0 and 𝛽𝛽1 . OLS
estimators are random variables

• The numerical values that are obtained from applying the formula to the
sample of data are known as OLS ESTIMATES

@Elisabetta Pellini 40
Estimating the Regression Parameters: OLS

• Suppose we have data on salary and ROE from a random sample of 15 CEOs

• Let’s use this sample to estimate

the parameters 𝛽𝛽0 and 𝛽𝛽1 of the
model
𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 = 𝛽𝛽0 + 𝛽𝛽1 𝑅𝑅𝑅𝑅𝑅𝑅 + 𝜀𝜀

@Elisabetta Pellini 41
Estimating the Regression Parameters: OLS

• To find the OLS estimates we plug the data into the OLS estimator(s):

∑ 𝑛𝑛
𝑥𝑥𝑖𝑖 −𝑥𝑥̅ 𝑦𝑦𝑖𝑖 −𝑦𝑦�
𝛽𝛽̂0 = 𝑦𝑦� − 𝛽𝛽̂1 𝑥𝑥̅ and 𝛽𝛽̂1 = 𝑖𝑖=1
∑𝑛𝑛 𝑥𝑥 −𝑥𝑥̅ 2
𝑖𝑖=1 𝑖𝑖

𝑦𝑦=(1095+1001+1122+578+1368+1145+1078+1094+
�

+1237+833+567+933+1339+937+2011)/15= 1089.2

𝑥𝑥̅ =(14.1+10.9+23.5+5.9+13.8+20+16.4+16.3+10.5+

+26.3+25.9+26.8+14.8+22.3+56.3)/15=20.25

14.1 − 20.25 1095 − 1089 + 10.9 − 20.25 1001 − 1089 + ⋯

𝛽𝛽̂1 = = 15.885
14.1 − 20.25 2 + 10.9 − 20.25 2 + 23.5 − 20.25 2 + ⋯
𝛽𝛽̂0 = 1089.2 − 15.885 20.25 = 767.529

@Elisabetta Pellini 42
Interpreting the Estimates
� 𝑖𝑖 “salary hat” indicates that this is an
𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠
estimated equation. It is the equation of
the SRF or fitted line for this sample of 15
data
� 𝑖𝑖 = 767.529 + 15.885𝑅𝑅𝑅𝑅𝑅𝑅𝑖𝑖
𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠

𝛽𝛽̂0 = 767.526 is the intercept of the

Δ𝑦𝑦�
fitted line. It is the salary predicted 𝛽𝛽̂1 = Δ𝑥𝑥 = 15.885 is the slope of the
from the estimated equation when fitted line. If ROE increases by one
ROE=0. Since salary is measured in percentage point, then salary is
thousands, this figure becomes predicted to increase by $15,885. It is an
$767,529. It is an estimate of estimate of slope of the population
intercept of the population model. model. Thus, it is an estimate of the
Thus, it is an estimate of the amount by which expected salary
expected CEO salary for a firm with changes when ROE increases by one
ROE=0. percentage point.

@Elisabetta Pellini 43
Interpreting the Estimates

� 𝑖𝑖 = 767.529 + 15.885𝑅𝑅𝑅𝑅𝑅𝑅𝑖𝑖 n=15

The fitted line is: 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠

2000
1500
1000
500

0 20 40 60 80
ROE

Remark: In many models we must be careful when interpreting the estimated

intercept, since we often do not have data points near x =0. For this reason the
estimated intercept may not be a good approximation of the population intercept
@Elisabetta Pellini 44
Prediction

• The estimated equation can be used for prediction or forecasting

purposes. Suppose that we want to predict the average CEO salary for a
firm with ROE=20%

• This prediction is carried out by substituting ROE=20 into the estimated

equation so as to obtain:
� 𝑖𝑖 = 767.529 + 15.885 20 = 1085.229
𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠

• With our estimated model, we predict that a CEO working for a firm whose
ROE=20% will earn on average $1,085,229

@Elisabetta Pellini 45
Prediction

• We get predicted values (or fitted values) of

CEOs salary for each ROE in the dataset:
� 1 = 767.529 + 15.885 14.1 = 991.5075
𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠

� 2 = 767.529 + 15.885 10.9 = 940.6755

𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠

….

• We calculate the residuals:

� 1 = 1095 − 991.5075
𝜀𝜀1̂ = 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠1 − 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠
= 103.4925
� 2 = 1001 − 940.6755
𝜀𝜀2̂ = 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠2 − 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠
= 60.32452

….

@Elisabetta Pellini 46
Algebraic Properties of OLS Estimates

• Algebraic properties of OLS estimates:

∑𝑛𝑛𝑖𝑖=1 𝜀𝜀𝑖𝑖̂ = 0 ∑𝑛𝑛𝑖𝑖=1 𝑥𝑥𝑖𝑖 𝜀𝜀𝑖𝑖̂ = 0

Deviations from regression line sum up to Sample covariance between deviations and
zero. This comes from the first FOC of the regressor is zero. This comes from the second FOC:
𝑛𝑛
minimisation problem:
𝑛𝑛 � 𝑥𝑥𝑖𝑖 (𝑦𝑦𝑖𝑖 − 𝛽𝛽̂0 − 𝛽𝛽̂1 𝑥𝑥𝑖𝑖 ) = 0
� (𝑦𝑦𝑖𝑖 − 𝛽𝛽̂0 − 𝛽𝛽̂1 𝑥𝑥𝑖𝑖 ) = 0 𝑖𝑖=1
𝑖𝑖=1

@Elisabetta Pellini SMM150 Quantitative Methods for Finance 47

Assessing the OLS Estimators

• We have estimated the parameters of the population model

𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 = 𝛽𝛽0 + 𝛽𝛽1 𝑅𝑅𝑅𝑅𝑅𝑅 + 𝜀𝜀

using a sample of data and applying the OLS method

• Now we want to know how good these estimates are

• This question is not answerable. We will never know the true values of
the population parameters 𝛽𝛽0 𝑎𝑎𝑎𝑎𝑎𝑎 𝛽𝛽1 , so we cannot say how close 𝛽𝛽̂0 =
767.529 and 𝛽𝛽̂1 = 15.885 are to the true values

• However, we can evaluate the quality of OLS estimation method, which

means we can check whether the OLS estimator has two important
properties, namely Unbiasedness and Efficiency

@Elisabetta Pellini 48
Unbiasedness

• Definition of Unbiased Estimator: an estimator of an unknown population

̂ is unbiased if 𝑬𝑬 𝜽𝜽
parameter, that we generically call 𝜃𝜃, � = 𝜽𝜽. In other words an

estimator is unbiased if its probability distribution has an expected value equal to

the parameter it is supposed to be estimating

• The specific estimate that we get may be smaller or larger than the true parameter,
depending on the sample that is selected. However if sampling is done repeatedly
and estimates are computed each time, then the average of these estimates will be
equal to the parameter
�
𝑓𝑓(𝜽𝜽)

Sampling
distribution of an
unbiased estimator

𝜃𝜃 𝜃𝜃�
@Elisabetta Pellini 49
Efficiency
• The variance of an estimator measures the precision of the estimator in the sense
that it tells us how much the estimates can vary from sample to sample

• The smaller the variance of an estimator is, the greater the sampling precision of
that estimator

�
𝑓𝑓(𝜃𝜃), Sampling distribution of the
If we compare two unbiased �
𝑓𝑓(𝜃𝜃) estimator 𝜽𝜽 � , the sampling
variance its smaller than that
estimators, say 𝜃𝜃̂ and 𝜃𝜃,
� the of the other estimator as the
distribution is more tightly
one with the smallest centrend about 𝜽𝜽
𝑉𝑉𝑉𝑉𝑉𝑉 𝜃𝜃̂ < 𝑉𝑉𝑉𝑉𝑉𝑉 𝜃𝜃�
variance is said to be
efficient relative to the other Sampling
distribution of
the estimator 𝜃𝜃�

𝜃𝜃 � 𝜃𝜃�
𝜃𝜃,

@Elisabetta Pellini 50
Gauss-Markov Assumptions

• Is the OLS estimator unbiased and efficient?

• Yes, but providing that some assumptions hold

• These assumptions are known as the Gauss-Markov assumptions as

they serve to demonstrate the Gauss-Markov Theorem

• The next slides examine the Gauss-Markov assumptions for cross-

sectional data

@Elisabetta Pellini 51
Gauss-Markov Assumptions

• Assumption 1: The population model is linear in the parameters:

𝑦𝑦 = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥 + 𝜀𝜀

“linear in the parameters” means that the model can be seen as a linear combinations
of parameters or a weighted sums of parameters with the variables being the
weights

• A model like 𝑦𝑦 = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥 2 + 𝜀𝜀 is a linear combinations of parameters or a

weighted sums of parameters with the variables being the weights

1
• This y = + 𝜀𝜀 is not a linear model as it is not a linear combinations of
𝛽𝛽0 +𝛽𝛽1 𝑥𝑥

parameters or a weighted sums of parameters with the variables being the

weights

• Nonlinear models require estimations methods other than the OLS

@Elisabetta Pellini 52
Gauss-Markov Assumptions

• Assumption 2: We have a random sample of size n 𝑥𝑥𝑖𝑖 , 𝑦𝑦𝑖𝑖 : 𝑖𝑖 = 1, 2, … , 𝑛𝑛

following the population model. Selecting a random sample means
choosing individuals in a way such that the selection of any individual is
statistically independent of the selection of any other individual

• Assumption 3: there exists some sample variation in the explanatory

variable, that is the values of x are not all the same (otherwise it would be
impossible to study how variation in x leads to changes in y)

@Elisabetta Pellini 53
Gauss-Markov Assumptions

• Assumption 4: zero conditional mean. The error 𝜀𝜀 has an expected

value of zero given any value of the explanatory variable x. This means
that x and 𝜀𝜀 are mean independent and hence uncorrelated:

𝐸𝐸 𝜀𝜀|𝑥𝑥 = 0

• if 𝐸𝐸 𝜀𝜀|𝑥𝑥 = 0 then x is said to be an exogenous regressor, while if the

assumption fails x is said to be an endogenous regressor

@Elisabetta Pellini 54
Gauss-Markov Assumptions

• Using Assumptions 1-4 it is possible to demonstrate the unbiasedness of the

OLS estimators, 𝐸𝐸(𝛽𝛽̂0 ) = 𝛽𝛽0 and 𝐸𝐸(𝛽𝛽̂1 ) = 𝛽𝛽1

• Unbiasedness generally fails if any of four assumptions fail

• Econometricians are usually very concerned about failure of Assumption 4:
𝐸𝐸 𝜀𝜀|𝑥𝑥 = 0
• One of the leading case in which this assumption fails is that of omitted
variables. Recall that 𝜀𝜀 contains anything else affecting y other than x. If we have
omitted anything that is important and that is also correlated with x, then 𝐸𝐸 𝜀𝜀|𝑥𝑥
≠0
• This is called omitted variable bias

@Elisabetta Pellini 55
Gauss-Markov Assumptions

• Assumption 5: homoscedasticity. The error 𝜀𝜀 has the same variance given

any value of the explanatory variable:
𝑉𝑉𝑉𝑉𝑉𝑉 𝜀𝜀|𝑥𝑥 = 𝜎𝜎 2

• When this assumption does not hold, we say that the error term is
heteroscedastic
• Remark: the conditional variance of the dependent variable is:
𝑉𝑉𝑉𝑉𝑉𝑉 𝑦𝑦|𝑥𝑥 = 𝑉𝑉𝑉𝑉𝑉𝑉 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥 + 𝜀𝜀|𝑥𝑥 = 𝑉𝑉𝑉𝑉𝑉𝑉 𝜀𝜀|𝑥𝑥

(the element 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥 is not random when we condition on x, that is when we
treat x as if it were known)
• Because of this relationship, we can say that heteroscedasticity is present
whenever the conditional variance of 𝒚𝒚 is function of 𝒙𝒙

@Elisabetta Pellini 56
Gauss-Markov Assumptions

• Graphical illustration of homoscedasticity vs heteroscedasticity

The conditional variance of y does not

depend on the value of the explanatory
variable

The conditional variance

of y depends on the value
of the explanatory variable

The homoscedasticity assumption

implies that at each level of x, we
are equally uncertain about how
far values of y might fall from their
mean value, and the uncertainty
does not depend on x.
x

@Elisabetta Pellini 57
The Variance of OLS Estimators

• The OLS estimator is a random variable because the estimates vary

from sample to sample. The probability distribution of an estimator is a
sampling distribution

• The sampling variance of an estimator tells us how much the estimates

can vary from sample to sample, thus it is a measure of the precision of
the estimator

• The lower the sampling variance of an estimator the greater its

precision

@Elisabetta Pellini 58
The Variance of OLS Estimators

• The variances of the OLS estimators 𝛽𝛽̂0 and 𝛽𝛽̂1 are:

2
∑ 𝑥𝑥𝑖𝑖 𝜎𝜎 2
𝑣𝑣𝑣𝑣𝑣𝑣 𝛽𝛽̂0 = 𝜎𝜎 2 𝑣𝑣𝑣𝑣𝑣𝑣 𝛽𝛽̂1 =
𝑛𝑛 ∑(𝑥𝑥𝑖𝑖 − 𝑥𝑥)̅ 2 ∑(𝑥𝑥𝑖𝑖 − 𝑥𝑥)̅ 2

(See Wooldridge (2019),pp 47-48 for the derivation of the formulae)

• The sampling variance of the OLS estimators will be the higher, the
larger the variance of the unobserved factors (i.e. 𝜎𝜎 2 ), and the lower, the
higher the variation in the explanatory variable (∑(𝑥𝑥𝑖𝑖 − 𝑥𝑥)̅ 2 )

@Elisabetta Pellini 59
Estimating the Error Variance

• The previous formulae require knowledge of the variance of the error term
𝜎𝜎 2 . This quantity is unknown, given that the error term is unknown

• However we can estimate it using an unbiased estimator of the error

variance:
1 𝑛𝑛 𝑅𝑅𝑅𝑅𝑅𝑅
2
=𝑠𝑠 2 � 𝜀𝜀̂ =
𝑛𝑛 − 2 𝑖𝑖=1 𝑖𝑖 𝑛𝑛 − 2
We subtract 2 because there are two OLS first order conditions (two
parameters)
• The square root of this quantity is:
1 𝑛𝑛
𝑠𝑠 = � 𝜀𝜀𝑖𝑖̂2
𝑛𝑛 − 2 𝑖𝑖=1

is know as standard error of the regression

@Elisabetta Pellini 60
Estimating the Variance of the OLS estimators

• Having an unbiased estimator of the error variance, we can get estimators of the
variances of the OLS estimators 𝛽𝛽̂0 and 𝛽𝛽̂1 by replacing the unknown variance
𝜎𝜎 2 with 𝑠𝑠 2

2
∑ 𝑥𝑥𝑖𝑖 𝑠𝑠 2
� 𝛽𝛽̂0
𝑣𝑣𝑣𝑣𝑣𝑣 = 𝑠𝑠 2 � 𝛽𝛽̂1
𝑣𝑣𝑣𝑣𝑣𝑣 =
𝑛𝑛 ∑(𝑥𝑥𝑖𝑖 − 𝑥𝑥)̅ 2 ∑(𝑥𝑥𝑖𝑖 − 𝑥𝑥)̅ 2

• The square roots are called Standard Errors of 𝛽𝛽̂0 and 𝛽𝛽̂1 :

∑ 𝑥𝑥𝑖𝑖2 1
𝑆𝑆𝑆𝑆 𝛽𝛽̂0 = 𝑠𝑠 𝑆𝑆𝑆𝑆 𝛽𝛽̂1 = 𝑠𝑠
𝑛𝑛 ∑(𝑥𝑥𝑖𝑖 −𝑥𝑥)̅ 2 ∑(𝑥𝑥𝑖𝑖 −𝑥𝑥)̅ 2

• 𝑆𝑆𝑆𝑆 𝛽𝛽̂0 and 𝑆𝑆𝑆𝑆 𝛽𝛽̂1 are random variables when we think of OLS running over

different samples. For one given sample 𝑆𝑆𝑆𝑆 𝛽𝛽̂0 and 𝑆𝑆𝑆𝑆 𝛽𝛽̂1 are just numbers that
gives us an estimate of the precision of the OLS estimators (not of the accuracy of a
specific set of parameter estimates)

@Elisabetta Pellini 61
Estimating the Variance of the OLS estimators

• We calculate the standard error of the regression:

1 𝑛𝑛 1178055
𝑠𝑠 = � 𝜀𝜀𝑖𝑖̂ 2 = = 301.03
𝑛𝑛 − 2 𝑖𝑖=1 13

• We calculate Standard Errors of 𝛽𝛽̂0 and 𝛽𝛽̂1 :

∑ 𝑥𝑥𝑖𝑖2 8106.78
𝑆𝑆𝑆𝑆 𝛽𝛽̂0 = 𝑠𝑠 = 301.03 = 158.32
𝑛𝑛 ∑(𝑥𝑥𝑖𝑖 − 𝑥𝑥)̅ 2 15(1953.817)

1 1
𝑆𝑆𝑆𝑆 𝛽𝛽̂1 = 𝑠𝑠 = 301.03 = 6.81
∑(𝑥𝑥𝑖𝑖 − 𝑥𝑥)̅ 2 1953.817

@Elisabetta Pellini 62
Gauss-Markov Theorem

• GAUSS–MARKOV THEOREM: Under the Assumptions 1–5, the OLS

estimators 𝛽𝛽̂0 and 𝛽𝛽̂1 have the smallest variance of all linear and unbiased
estimators of 𝛽𝛽0 and 𝛽𝛽1 . They are the best linear unbiased estimators
(BLUE) of the population parameters 𝛽𝛽0 and 𝛽𝛽1

@Elisabetta Pellini 63
Gauss-Markov Theorem

• Best: the OLS estimators have the smallest variance when compared to
similar estimators that are linear unbiased estimators

• Linear: the formulae of the estimators can be written as weighted

averages of the y values, that is for 𝛽𝛽̂1 we can write:

𝛽𝛽̂1 = ∑𝑛𝑛𝑖𝑖=1 𝑤𝑤𝑖𝑖 𝑦𝑦𝑖𝑖

Where 𝑤𝑤𝑖𝑖 is a function of sample values of the independent variable (a

similar formulation can be derived for 𝛽𝛽̂0 )

• Unbiased: it means that 𝐸𝐸(𝛽𝛽̂0 ) = 𝛽𝛽0 and 𝐸𝐸(𝛽𝛽̂1 ) = 𝛽𝛽1 , that is on average,
the value of the 𝛽𝛽̂0 and 𝛽𝛽̂1 will be equal to the true values 𝛽𝛽0 and 𝛽𝛽1

@Elisabetta Pellini 64
Gauss-Markov Theorem

• The Gauss-Markov theorem justifies the use of the OLS method rather
than other competing methods (or estimators): when the assumptions
hold no estimator will be better than OLS

• However, if any of the assumptions fails, then the OLS is not BLUE

@Elisabetta Pellini 65
References

Chapter 2 in: Wooldridge, J.M., (2019). Introductory Econometrics: A

Modern Approach, 7th Edition. CENGAGE.

@Elisabetta Pellini 66