Caribe - Michel Camilo
Caribe - Michel Camilo
Caribe - Michel Camilo
BEFORE-AND-AFTER STUDY
TECHNICAL BRIEF
MAY 2009
The Institute of Transportation Engineers (ITE) is an international educational and scientific
association of transportation and traffic engineers and other professionals who are responsible
for meeting mobility and safety needs. ITE facilitates the application of technology and
scientific principles to research, planning, functional design, implementation, operation, policy
development and management for any mode of transportation by promoting professional
development of members, supporting and encouraging education, stimulating research,
developing public awareness, exchanging professional information and maintaining a central
point of reference and action.
Founded in 1930, ITE serves as a gateway to knowledge and advancement through meetings,
seminars and publications, and through our network of nearly 17,000 members working in more
than 92 countries. ITE also has more than 90 local and regional chapters and more than 130
student chapters that provide additional opportunities for information exchange, participation
and networking.
ISBN-10: 1-933452-46-3
ISBN-13: 978-1-933452-46-3
© 2009 Institute of Transportation Engineers. All rights reserved.
1. INTRODUCTION ............................................................................................................................... 1
2. BACKGROUND................................................................................................................................. 2
4. SUMMARY ...................................................................................................................................... 11
5. REFERENCES ................................................................................................................................ 13
Figure 2. Number of Monthly Fatal Accidents from January 1993 to December 2004 in Ontario,
Canada ................................................................................................................................................... 3
Figure 3. Number of Monthly Fatal Accidents and Annual Average Monthly Fatal Accidents from
January 1993 to December 2004 in Ontario, Canada ........................................................................... 4
Table 1. Before Period Accidents for Treatment and Yoked Comparison Groups ....................................... 8
Table 2. After Period Accidents for Treatment and Yoked Comparison Groups .......................................... 8
The main objective of this technical brief is to provide practitioners with a quick reference on the key
considerations and components of a valid before-and-after observational study. This document also
serves as a tool to increase the level of understanding of before-and-after study techniques so that they
can be more effectively conducted in the future.
By providing better assessment tools and resources to allow practitioners to explicitly consider safety
impacts in their decision-making, there has been a positive shift in North American road safety. However,
there are cases where local studies and research have employed inferior analysis techniques in before-
and-after assessments due to lack of understanding of proper techniques, time or resource constraints
and/or budget limitations. As transportation safety practitioners, shortcomings must be identified in
before-and-after study methodologies so that good research results are identified and poorly conducted
work is not propagated.
This document is not intended to be a prescriptive reference on the methodologies and formulae for
completing different types of before-and-after studies. This has been left to more comprehensive
documents such as the updated Highway Safety Manual and research work available in the public realm
and provided in the final section of this brief.
Section 2 of this technical brief outlines the background fundamentals and definitions required to
understand the primary components of a before-and-after study, the techniques adopted to conduct such
a study and how each technique differs from the others. Section 3 provides an introduction and a brief
description of each technique, its requirements, strengths and weaknesses.
The Transportation Safety Council would like to thank the following authors for their contributions in the
preparation of this brief:
The council would also like to acknowledge the assistance of Russell Brownlee, transportation safety
engineer at Giffin Koerth Forensic Engineering and Accounting, and Brian Malone, president of Synectics
Transportation Consultants, for being co-editors of this document. In addition, the council’s gratitude goes
to Elizabeth Wemple and Calvin Mollett for being reviewers of this document.
Observational studies themselves can be categorized into two groups: before-and-after studies and
cross-section studies. A before-and-after study is used when it is desired to study the safety implications
of a certain improvement or operational change. In an observational before-and-after study, many of the
attributes of a facility remain unchanged. For instance, the study of safety implications associated with
installing traffic signal controls at an all-way stop-controlled intersection falls under the before-and-after
observational study category. In this example, the geometry and other site characteristics of the
intersection retain their original configurations.
However, in cross-section observational studies, the safety effects of one group of facilities are compared
with another group. These two groups of facilities have some common features, and the safety effect of
those features that are not in common must be evaluated. The comparison of the safety of a roundabout
and a stop-controlled intersection is an example of a cross-section observational study. This technical
brief is directed at the techniques that can be utilized for before-and-after observational studies in the
context of road safety.
Figure 1 shows the continuum of events in a traffic stream. In this figure, the volume of each situation
relates to the corresponding frequency. The number of dangerous situations within a transportation facility
is greater than number of accidents per unit of time. Given the fact that events that happen more
frequently can be measured more easily, some researchers have tried to define the safety of
transportation facilities based on surrogate measures of safety.1 Accidents are directly proportional to
dangerous situations, so safety can be defined in terms of the prevalence of accidents.
Normal Traffic
Incipient Danger
Dangerous Situations
Accidents
Unfortunately, randomness of accidents is also one of the primary characteristics of safety, which makes
the evaluation of a safety treatment more challenging. Figure 2 shows the number of total monthly fatal
accidents in Ontario, Canada, from the period between January 1993 and December 2004. If a random
year is selected from this figure, one is not able to determine a specific trend in the number of accidents
among different months. Similarly, if a random month is selected, as can be seen from 1993 to 1994, the
number of accidents decreased from 106 to 83. However, in 1995, it increased to 100. As with any
transportation facility, accident occurrences are random; this must be explicitly recognized in any effort to
measure safety performance.
100
80
Accidents
60
40
20
0
Jan- Jan- Jan- Jan- Jan- Jan- Jan- Jan- Jan- Jan- Jan- Jan- Jan-
93 94 95 96 97 98 99 00 01 02 03 04 05
Months of Years
Safety is an attribute of a facility that is believed to be the same over time if all influencing parameters
(such as environment, users, volumes, etc.) remain unchanged. Consequently, the safety of a facility can
be defined as an expected accident frequency or, more formally, “the number of accidents, or accident
consequences, by kind and severity, expected to occur on the facility during a specified period.”2
Figure 3. Number of Monthly Fatal Accidents and Annual Average Monthly Fatal Accidents from
January 1993 to December 2004 in Ontario, Canada.
100
Fatal Accidents Frequency
R2 = 0.9544
80
60
40
20
0
Jan- Jan- Jan- Jan- Jan- Jan- Jan- Jan- Jan- Jan- Jan- Jan- Jan-
93 94 95 96 97 98 99 00 01 02 03 04 05
Months of Years
The differentiation and determination of which accident types belong to either of the above categories is a
challenge. In general, this division will not be definitive. A firm understanding of the contributory factors in
specific accident types is required to reduce the likelihood that comparison accidents are analyzed as
target accidents and vice versa.
According to the two groups of causal factors, change in safety performance from the before period to the
after period can be disaggregated into four components: treatment effect, exposure effect, trend effect
and random effect.4 All of these effects are explained in the following sections.
The treatment effect is the change in safety performance of a transportation facility caused by
implementation of a specific treatment. In a before-and-after study, the treatment effect must be isolated
from the other causal factors to determine the net improvement/deterioration in terms of safety
performance. The net safety benefits/costs of the treatment are obtained by finding and comparing the
answers to the following two questions:
• What would have been the safety performance of the facility in the after period had the
treatment not been applied?
• What is the safety performance of the treated facility in the after period?
To answer the first question, the causal factors outlined in the following three subsections must be fully
quantified and isolated. The second question needs to recognize the importance of the target accident
discussion in the preceding section.
The exposure effect is caused by change in traffic volume and patterns on a facility. Traffic volume and
accident frequencies have a direct relationship. Therefore, it is conceivable that the accident frequency of
a facility increases as traffic volume increases and vice versa. This effect could be significant if the
remedial action applied to the facility significantly changes the operations or capacity of the facility, such
as placing an intersection under traffic signal control or providing a two-way left-turn lane on a specific
road section.
The trend effect is due to causal factors that are not recognized, measured and understood. For example,
traffic composition (such as a higher/lower percentage of trucks or pedestrians), driver composition (in
terms of behavior, age, etc.), enforcement level, weather conditions, etc. can be changed from the before
period to the after period.
Based on the definition of the above four effects, it can be concluded that even if no safety treatment had
been applied to the facility, it would have been likely to observe a change in accident frequency from the
before to the after periods. Consequently, analysts must recognize the impact of each of these effects on
their evaluation results and must employ techniques that seek to minimize or account for these
extraneous effects. Properly designed studies extract the treatment effect from the total change in safety
performance in order to assess if the safety or operational treatment has resulted in a safety improvement
or deterioration.
Source: Harwood, D.W. et al. “Safety Effectiveness of Intersection Left- and Right-
Turn Lanes.” Transportation Research Record, No. 1840 (2003): 131–139.
In this method, it is hoped that the unknown causal factors should affect the comparison group in the
same manner that they influence the treatment group. Therefore, the change in the number of accidents
from the before period to the after period, had the treatment sites been left unimproved, would have been
in the same proportion as the matching comparison site. Under this assumption, the accident frequency at
each treatment site in the before period is multiplied by the ratio of after-to-before accidents at the
comparison site to predict the expected number of accidents in the after period at the treated site without
• It makes use of only one comparison site, and it is conceivable to have different estimates when
other comparison sites are used. Consequently, the findings based on the evaluation of the
facility will be variable with relatively wide confidence limits.
• It is unable to address the issue known as regression-to-the-mean bias. If the treatment site is
chosen based on the fact that the agency has observed high accident counts in a short term, the
accident frequency will likely be lower in the after period even if no treatment is applied. However,
this method cannot identify whether the lower accident frequency is due to the treatment or the
intrinsic randomness of accidents.
• It is unable to deal with cases where the comparison site has no history of accident occurrences.8
A before-and-after study was performed with yoked comparison in the state of Illinois to evaluate the
effectiveness of continuous shoulder rumble strips (CSRS).9 The target accidents in this study were
single-vehicle run-off-the-road accidents. The author selected 55 treatment sites and 55 comparison
sites. The selected yoked comparison sites in this study were freeway sections adjacent and upstream to
the treatment sites.
Tables 1 and 2 illustrate the time series of the accident counts in the before and after periods. As shown,
the durations of the before and after periods are similar to each other in this study.
Table 1. Before Period Accidents for Treatment and Yoked Comparison Groups.
Before Period
Years 1987 1988 1989 1990 1991 1992–93 Total
Treatment 276 644 863 596 310 112 2801
Comparison 240 515 646 521 259 107 2288
Source: Griffith, M. “Safety Evaluation of Continuous Rolled-In Rumble Strips Installed on Freeways.”
Transportation Research Record, No. 1665 (1999): 28–34.
Table 2. After Period Accidents for Treatment and Yoked Comparison Groups.
After Period
Years 1990 1991 1992 1993 1994 1995 Total
Treatment 70 391 500 534 255 145 1895
Comparison 112 462 460 454 212 133 1833
Source: Safety Evaluation of Rollen-In Continuous Shoulder Rumble Strips Installed on Freeways.
Highway Safety Information System Summary Report, FHWA-RD-00-32. Washington, DC, USA: Federal
Highway Administration, U.S. Department of Transportation, 1999.
The researcher found that the CSRS treatments contributed to an 18.3-percent reduction in target
accidents.10
Source: Harwood, D.W. et al. “Safety Effectiveness of Intersection Left- and Right-
Turn Lanes.” Transportation Research Record, No. 1840 (2003): 131–139.
In this evaluation approach, the facilities in the comparison group do not have to be exactly similar to the
facilities in the treatment group. However, it is important that a close agreement exist with regard to
accident history at the treatment and comparison sites in the before period. This approach resolves the
first issue associated with the before-and-after study with the yoked comparison; however, it is unable to
address the phenomenon of regression-to-the-mean bias. Also, it should be noted that this technique is
similar to the yoked comparison approach in that it cannot determine the treatment effectiveness if
accident counts in either the before or the after period in the comparison group equal zero. This situation
is unlikely to occur due to having a group of comparison sites rather than only one single comparison site
for each specific treatment site.
The Insurance Corporation of British Columbia (ICBC) launched the STOP Sign In-Fill Program in 1998 to
reduce the frequency and severity of accidents in residential areas in the greater Vancouver, British
Columbia, Canada, regional district. The effectiveness of the program was evaluated using an
observational before-and-after study with comparison groups to determine whether the program had been
effective. This program promoted the idea that STOP signs should be installed at every second
intersection in residential neighborhoods.11
One of the unique features of this study was the fact that the intersections treated in this study had not
been selected based on their accident frequencies, but on the basis of a general rule that a STOP sign
should be installed at every second intersection. Consequently, the effect of the regression-to-the-mean
phenomenon was minimal in this project.
The project study area consisted of 22 zones including 380 intersections. The treatment of intersections
had been completed in 1999, 2000, 2001, or 2002 in order to provide the analysts with adequate time for
a thorough before-and-after study. The accident data associated with each intersection were extracted
from the ICBC database. The comparison group in this study was a collection of 133 existing two-way
stop-controlled intersections located in the 22 study zones.
The research found that the total accidents and severe accidents were reduced by 52.8 and 66.9 percent,
respectively. It should be noted that the researchers also conducted a before-and-after study with yoked
comparison. They found that the total and severe accidents were reduced by 44.8 and 61.1 percent,
respectively.12
Treatment Group
Accidents
Independent
Variables
Source: Harwood, D.W. et al. “Safety Effectiveness of Intersection Left- and Right-
Turn Lanes.” Transportation Research Record, No. 1840 (2003): 131–139.
In the Empirical Bayes approach, the accident frequency in the after period if the treatment had not been
applied is predicted using accident prediction models or SPFs developed for a reference group and the
observed accidents in the before period. The reference group is similar to the comparison group
introduced in other methods. Consequently, the reference group consists of similar facilities that have not
undergone any improvements from the before to the after periods.
The safety effects of roundabout conversions in the United States were investigated using the Empirical
Bayes approach. The researchers analyzed 23 intersections in seven states: Colorado, Florida, Kansas,
Maine, Maryland, South Carolina and Vermont. All of these intersections were converted to roundabouts
between 1992 and 1997. Among the 23 intersections, 19 had been stop-controlled, while the remainder
were signalized.13
The researchers employed accident prediction models prepared for signalized intersections and those
developed for rural stop-controlled intersections.14,15 However, new regression models were calibrated for
urban stop-controlled intersections based on data from Toronto, Canada, and the states of Maryland and
Florida. Using these models, the frequency of accidents during the before period was calculated. For 23
intersections converted to roundabouts, the total accidents and the injury accidents were reduced by 40
percent and 80 percent, respectively.16
This brief describes four common methodologies for conducting before-and-after studies for
transportation facilities as well as their benefits and shortcomings. The naïve before-and-after study is the
simplest technique for this kind of study. Within this methodology, the effect of passage of time on the
safety of a facility is ignored, which is an unreasonable assumption in terms of statistical validity. The
application of this technique in real-world projects is not recommended.
The most statistically rigorous method of the four reviewed is the Empirical Bayes technique. There is
general consensus among researchers and practitioners regarding the superiority of this technique, and it
is recommended for use in all circumstances where the data and required SPFs are available. The next
preference is to perform a before-and-after study with comparison groups. However, if the number of
facilities is limited in the comparison group, the yoked comparison is the next best analysis choice.
The decision to use one analysis methodology versus another ultimately is in the hands of the
transportation practitioner undertaking the before-and-after study. This brief provides an understanding of
the merits of each method to weigh into the decision process where data availability, resources, time
constraints and other decisive factors are a reality.
Table 3 is a summary of the abilities of each of the four before-and-after study methods to address the
primary causal factors attributed to a change in safety performance.
Table 4 shows the data requirements of each technique and briefly states the weaknesses and strengths
associated with each approach.
2. Ibid.
3. Ibid.
4. Mollett, C.J. The analysis of road traffic accident data in the implementation of road safety remedial
programmes. University of Stellenbosch, South Africa, 2001.
5. Griffin, L.I. and R.J. Flowers. A Discussion of Six Procedures for Evaluating Highway Safety Projects.
Washington, DC, USA: U.S. Department of Transportation (U.S. DOT), Federal Highway
Administration (FHWA), 1997.
6. Safety Effectiveness of Intersection Left- and Right-Turn Lanes. Report No. FHWA-RD-02-089.
Washington, DC: U.S. DOT, FHWA, 2002.
8. FHWA 2002.
9. Griffith 1999.
10. Ibid.
11. Sayed, T., K. El-Basyouny and J. Pump. “Safety Evaluation of Stop Sign In-Fill Program.”
Transportation Research Record, No. 1953 (2006): 201–210.
12. Ibid.
13. Persaud, B.N., R.A. Retting, P.E. Garder and D. Lord. “Safety Effect of Roundabout Conversions in
the United States: Empirical Bayes Observational Before-After Study.” Transportation Research
Record, No. 1751 (2001): 1–8.
14. Lord, D. The Prediction of Accidents on Digital Networks: Characteristics and Issues Related to the
Application of Accident Prediction Models. Toronto, Ontario, Canada: University of Toronto, 2000.
15. Bonneson, J.A. and P.T. McCoy. “Estimation of Safety at Two-Way Stop-Controlled Intersections on
Rural Highways.” Transportation Research Record, No. 1401 (1993): 83–89.