W6 - FIN4715 Impact Measurement
W6 - FIN4715 Impact Measurement
W6 - FIN4715 Impact Measurement
Week 6
Impact Measurement
Learning objectives
To understand:
• Why we need to measure impact
• How to properly measure impact
5
Social impact creation cycle
What to
invest?
What
How to problem to
increase address?
impact?
How to
measure What steps
success? to take?
Source: Epstein, M. J., and Yuthas, K. (2014). Measuring and improving social impacts: A guide for nonprofits, 6
companies, and impact investors. San Francisco, ca: Berrett-Koehler.
Why measure?
Why?
Actions Accountability
Target audience: internal Target audience: external
& external Reasons:
Reasons: - Reports performance
- Guides future behavior - Strengthen willingness
- Makes adjustments to collaborate
- Gets buy-ins 7
Defining the success criteria
• Base it on the intended impact objectives
– Mission statements Intention
– Framework
• Baseline
After-intervention
Additionality
Baseline?
Baseline?
• Benchmark
9
Indicator of success
• Come up with the metric (indicator) to measure
success
• Based on the evidence
– Often linked to scientific testing
– Has the intervention (statistically) significantly changed
the behavior/condition?
• Need reliable data:
– Where do you get your data?
– How do you collect your data?
– How often do you collect your data?
– How do you analyze your data? 10
Data collection methods
1. Primary data collection:
– Data collected directly from main sources
– Examples: surveys, focus groups, interviews,
randomized control trials, natural experiments
11
Before-and-after analysis
Simple difference
If the after measurement is different than the before measurement, then the
intervention created a change.
12
Potential problems
1. Mixing of effects
2. Does the dataset represent the entire
population (of beneficiaries)
3. Is the result caused by the intervention?
13
Mixing of effects
Changes Changes
Observed
attributable to attributable to
changes
factors other the
after
than the intervention
intervention
intervention
14
Mixing of effects - example
Intervention Observed outcome
Higher score
Exposure to on
arts standardized
test
Interventions
Mentoring Observed outcome
Tutoring
Higher score
on
standardized
Exposure
to arts
test
15
Obtaining the effect of the intervention
16
Difference-in-difference (DID)
Treatment application
8
0
Pre-treatment Post-treatment
Time
Treated Control
17
Difference-in-difference (DID)
8 B
7
6 The effect of the intervention
Outcome
5
4
3 A D
2
1
0 C
Pre-treatment Post-treatment
Time
Treated Control
18
Table source: https://www.researchgate.net/figure/1-The-Difference-in-Differences-Method_tbl4_280098184
Statistical matching
• At least 2 groups:
– Treated group: receive treatment
– Control group: do not receive treatment
4 4
2 2
0 0
Pre-treatment Post-treatment Pre-treatment Post-treatment
Treated Control Treated Control
X 19
Random sampling
In most cases, measurement is done on a subset of the entire
population (of beneficiaries)
– Data should be representative of the entire population
– Random sampling: takes a small, random portion of the entire
population to be part of the dataset
• Simple random sampling
– Subjects are chosen randomly from the entire population
• Stratified random sampling
– Divides the population into smaller groups based on shared
characteristics, then subjects are chosen randomly from each
group
If the data set does not represent the population, we may run into
sampling/selection bias
20
Sampling bias example
Source: https://rtmagazine.com/disorders-diseases/critical-care/icu-ventilation/early-data-on-ventilated-covid-19-patients-reveals-
severe-mortality/ 21
Sampling bias (cont.)
Different baseline
survival rate
Patients Mechanical
6%
with severe ventilation
survive
symptoms used
25
RCT (cont.)
Advantages
– Show causal relationship
– Could potentially show a more reliable evidence
Disadvantages
– Costly
– Requires strict control over the entire experiment
– Small samples – impact may not materialize
– Generalizability
• People behave differently during experiment
• Specific setting, small samples
26
Summary
• We measure to monitor and to evaluate
• Impact measurement:
– Before and after analysis
– Potential biases
– Methods to reduce the biases
27
Arts and Economic Prosperity 5
2015
American for the Arts
Mission
29