Edit 2-Why Randomize
Edit 2-Why Randomize
Edit 2-Why Randomize
Ben Olken
Harvard University and J-PAL
www.povertyactionlab.org
Agenda
• Examples:
– Does giving scholarships increase attendance at school?
– Does auditing road projects reduce corruption?
– Do bed-nets prevent malaria?
Basic setup of program evaluation
• Some definitions:
– Outcome (Y): What outcome treatment might affect
– Treated (T): Group affected by program
– Control (C): Group unaffected by program
– Schooling. People who finish high school earn more than people
who drop out before finishing high school. Is this because of the
schooling? Or because people who are smarter get more
school? Or some combination?
www.povertyactionlab.org
Example: Iron supplements in Java
• Background:
– Anemia (lack of iron) causes low-energy and reduces people’s
ability to work
– Problem can be severe in agricultural areas where people mostly
eat unfortified foods
• Program:
– Iron fortification pilot program in Purworejo, Central Java
– Health workers come to households and encourage them to take
iron pills once per week
9 (Observed)
7
(Observed)
time
2002 2003
PROGRAM
We need to identify what would have
happened in the absence of the program
Monthly earnings
(Rp 100,000)
9 (Observed)
time
2002 2003
We need to identify what would have
happened in the absence of the program
Monthly earnings
(Rp 100,000)
9 (Observed)
Impact 9 – X
Impact:
time
2002 2003
Idea: Use a control group to estimate X
Monthly earnings
(Rp 100,000)
time
2002 2003
What makes a bad/good control group?
Treatment Group
Control Group
time
2002 2003
Selection Bias
Monthly earnings
(Rp 100,000)
Treatment Group
Control Group
time
2002 2003
One solution… randomized evaluation
Treatment Group
True impact: 9 – 7.9 =1.1
Impact you would
estimate using ‘wrong’
X
control group: 9 – 7 = 2
Control Group
time
2002 2003
Example: Results from Purworejo Study
10 9.36
Monthly earnings Rp. (100,000)
7.91
8
6.6
6 5.5
Treatment
4 Control
0
Men Women
III – WHAT IS A RANDOMIZED
EVALUATION EXACTLY?
www.povertyactionlab.org
The basics
Start with simple case:
• Take a sample of program applicants
• Randomly assign them to either:
– Treatment Group - receives treatment
– Control Group - not allowed to receive treatment (during the
evaluation period)
• Random means that which of these people gets
treatment or does not get treatment is chosen by lottery:
– Can be computer lottery
– Can be a live public lottery
• Note: random assignment of treatment and control is not
the same as random sampling
Why does random assignment work?
• Why does random assignment of treatment and control
generate a good counterfactual?
• So…
– If there was no treatment, the two groups would be the same
– So the only difference between treatment and control is the
effect of the treatment!
Basic Setup of a Randomized Evaluation
Target Population
Potential Participants
Evaluation Sample
Random Assignment
Treatment Control
Group Group
• Important factors:
– What is the decision-making unit
– At what level can data be collected?
IV - ADVANTAGES AND
LIMITATIONS OF RANDOMIZED
EVALUATIONS
www.povertyactionlab.org
Validity
• In assessing any study, there are two types of issues to
think about:
– Internal validity: relates to ability to draw causal inference, i.e.
can we attribute our impact estimates to the program and not to
something else
– External Validity
(e.g. are the results generalizable to the population of interest?)
• Partial equilibrium
Other limitations of experiments
• Ethical issues
– Most programs are rationed due to lack of resources
– Randomization is a “fair” way to allocate resources
– May also make sense to remove discretion from allocation for
other reasons (e.g., prevent favoritism)
– Phase-ins or Pilots lend themselves naturally to randomization
– Exploit Pilots or phase-in due to budgetary constraints
V –HOW WRONG CAN YOU GO:
THE VOTE 2002 CAMPAIGN
www.povertyactionlab.org
Case 1 – “Vote 2002” campaign
• Intervention designed to increase voter turnout in 2002
U.S. election
• Phone calls to ~60,000 individuals
• Only ~35,000 individuals were reached
• Key Question: Did the campaign have a positive effect
(i.e. impact) on voter turnout?
– 5 methods were used to estimate impact
Methods 1-3
Estimated Impact
Method 1 10.8 pp *
Method 2 6.1 pp *
Method 3 4.5 pp *
pp=percentage points
Estimated Impact