The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice

EDUCATION POLICY FOR ACTION SERIES
E D U C A T I O N C H A L L E N G E S F A C I N G N E W Y O R K C I T Y
Can Teachers be Evaluated

by their Students’ Test Scores?
Should They Be?
The Use of Value-Added
Measures of Teacher Effectiveness
in Policy and Practice
Sean P. Corcoran
in collaboration with
Annenberg Institute research staff
EDUCATION POLICY FOR ACTION SERIES
E D U C A T I O N C H A L L E N G E S F A C I N G N E W Y O R K C I T Y
Can Teachers be Evaluated

by their Students’ Test Scores?
Should They Be?
The Use of Value-Added
Measures of Teacher Effectiveness
in Policy and Practice
Sean P. Corcoran
in collaboration with
Annenberg Institute research staff
About the Annenberg Institute for School Reform
The Annenberg Institute for School Reform is a national
policy-research and reform-support organization, affiliated
with Brown University, that focuses on improving condi-
tions and outcomes for all students in urban public
schools, especially those serving disadvantaged children.
The Institute’s vision is the transformation of traditional
school systems into “smart education systems” that develop
and integrate high-quality learning opportunities in all
areas of students’ lives – at school, at home, and in the
community.
The Institute conducts research; works with a variety of
partners committed to educational improvement to build
capacity in school districts and communities; and shares
its work through print and Web publications. Rather than
providing a specific reform design or model to be imple-
mented, the Institute’s approach is to offer an array of
tools and strategies to help districts and communities
strengthen their local capacity to provide and sustain
high-quality education for all students.
Annenberg Institute for School Reform at Brown University

Box 1985
Providence, Rhode Island 02912
233 Broadway, Suite 720
New York, New York 10279
www.annenberginstitute.org
© 2010 Brown University
CONTENTS
Figures iv
About the Author v
About the Series vi
Acknowledgments vi
Introduction ............................................................................................................................ 1
1 Value-Added Measurement: Motivation and Context ................................................ 1
2 What Is a Teacher’s Value-Added? .................................................................................. 4
3 Value-Added in Practice: New York City and Houston .............................................. 6
The New York City Teacher Data Initiative 6
The Houston ASPIRE Program 12
4 Challenges to the Practical Implementation of Value-Added ................................. 14

What is being measured? 14
Is the measurement tool appropriate? 16
Can a teacher’s unique effect be isolated? 18
Who counts? 19
Are value-added scores precise enough to be useful? 21
Is value-added stable from year to year? 26
5 Discussion ............................................................................................................................ 28
References ........................................................................................................................... 29
Appendix A: Race to the Top Definitions of Teacher Effectiveness
and Student Achievement ...................................................................... 34
Appendix B: Sample New York City Teacher Data Report, 2010 ........................ 35
Appendix C: Sample New York City Teacher Data Report, 2009 ....................... 36
FIGURES
Figure 1 Factors affecting average achievement in two classrooms: hypothetical decomposition ............... 4
Figure 2 Factors affecting year-to-year test score gains in two classrooms:

hypothetical decomposition ...................................................................................................................... 5
Figure 3 New York City Teacher Data Initiative timeline ................................................................................... 6
Figure 4 New York City value-added model: predictors ................................................................................... 10
Figure 5 Teacher value-added on two reading tests: Houston fourth- and fifth-grade teachers ............... 17
Figure 6 Percent of students with a test score and percent contributing to value-added estimates,
grades four to six, Houston, 1998–2006 ............................................................................................. 20
Figure 7 Average confidence interval width, New York City Teacher Data Reports, 2008-2009 ............ 23
Figure 8 Percent of overlapping confidence intervals, ELA and math ........................................................... 24
Figure 9 Year-to-year stability in value-added rankings: HISD reading test, 2000–2006 .......................... 26
Figure 10 Year-to-year stability in ELA and math value-added rankings: New York City Teacher Data
Reports, 2007-2008 .................................................................................................................................. 27
iv Can Teachers be Evaluated by their Students’ Test Scores? Should They Be?
About the Author His recent publications can be found in the
Journal of Policy Analysis and Management, the
Sean P. Corcoran is an assistant professor of Journal of Urban Economics, Education Finance
educational economics at New York Univer- and Policy, and the American Economic Review.
sity’s Steinhardt School of Culture, Education,
and Human Development, an affiliated faculty
of the Robert F. Wagner Graduate School of
Public Service, and a research fellow at the
Institute for Education and Social Policy
(IESP). He has been a research associate of the
Economic Policy Institute in Washington,
D.C., since 2004 and was selected as a resident
visiting scholar at the Russell Sage Foundation
in 2005-2006. In addition to being a member
of the board of directors of the Association for
Education Finance and Policy (formerly the
American Education Finance Association), he
serves on the editorial board of the journal
Education Finance and Policy.
Corcoran’s research focuses on three areas:
human capital in the teaching profession, edu-
cation finance, and school choice. His recent
papers have examined long-run trends in the
quality of teachers, the impact of income
inequality and court-ordered school finance
reform on the level and equity of education
funding in the United States, and the political
economy of school choice reforms. In 2009,
he led the first evaluation of the Aspiring Prin-
cipals Program in New York City, and he is
currently working on a retrospective assessment
of the Bloomberg-Klein reforms to school
choice and competition in New York City for
the American Institutes for Research. He co-
edits a book series on alternative teacher com-
pensation systems for the Economic Policy
Institute, and in recent years he has been inter-
ested in value-added measures of evaluating
teacher effectiveness, both their statistical
properties and their obstacles to practical
implementation.
Annenberg Institute for School Reform v

About the Series Acknowledgments
Education Policy for Action: Education I thank the Annenberg Institute for School
Challenges Facing New York City is a series Reform for the invitation to conduct this
of research and policy analyses by scholars in research and write this report. Deinya Phenix
fields such as education, economics, public was an immense help from start to finish.
policy, and child welfare in collaboration with Norm Fruchter, Ivonne Garcia, Megan Hester,
staff from the Annenberg Institute for School Christina Mokhtar, and Eric Zachary offered
Reform and members of a broadly defined thoughtful and constructive feedback at multi-
education community. Papers in this series ple points during the process of writing and
are the product of research based on the Insti- preparing for my January 27, 2010, presenta-
tute’s large library of local and national public tion, which was part of the Education Policy
education databases; work with the Institute’s for Action conversation series. Many audience
data analysis team; and questions raised and members at this event offered insightful
conclusions drawn during a public presenta- thoughts and comments, and I would particu-
tion and conversation with university and larly like to express my appreciation to Leo
public school students, teachers, foundation Casey from the United Federation of Teachers
representatives, policy advocates, education for serving as a discussant.
reporters, news analysts, parents, youth, and
I would also like to thank Jennifer Jennings
community leaders.
and Andrew Beveridge for sharing data from
Among the issues that the series addresses are the Houston Independent School District,
several pressing topics that have emerged from Rhonda Rosenberg and Jackie Bennett of the
the Institute’s research and organizing efforts. United Federation of Teachers for providing
Some of the topics covered in the series are: assistance with the New York City Teacher
• Confronting the impending graduation crisis Data Report data, and Rachel Cole for her
• The small schools experiment in New York research assistance. Amy McIntosh and Joanna
City Cannon of the New York City Department of
• Positive behavior and student social and Education were instrumental in providing
emotional support background on the Teacher Data Initiative and
assisted me in correcting factual errors. All
• Modes of new teacher and principal induc-
remaining errors are my own.
tion and evaluation
Many thanks to the Robert Sterling Clark
Foundation for its support of the public con-
versations from which this report and the other
publications in the series grew.
For a downloadable version of this report and
more information about the series, please visit
<www.annenberginstitute.org/WeDo/NYC_
Conversations.php>.
vi Can Teachers be Evaluated by their Students’ Test Scores? Should They Be?
Introduction
Value-added measures of teacher effectiveness
1 Value-Added Measurement: Motivation
and Context
are the centerpiece of a national movement to Traditional measures of teacher quality have
evaluate, promote, compensate, and dismiss always been closely linked with those found in
teachers based in part on their students’ test teacher pay schedules: years of experience, pro-
results. Federal, state, and local policy-makers fessional certification, and degree attainment.
have adopted these methods en masse in recent As recently as the 2001 No Child Left Behind
years in an attempt to objectively quantify Act (NCLB), teacher quality was commonly
teaching effectiveness and promote and retain formalized as a set of minimum qualifications.
teachers with a demonstrated record of success. Under NCLB, “highly qualified” teachers of
Attention to the quality of the teaching force core subjects were defined as those with at least
makes a great deal of sense. No other school a bachelor’s degree, a state license, and demon-
resource is so directly and intensely focused on strated competency in the subject matter
student learning, and research has found that taught (e.g., through a relevant college major
teachers can and do vary widely in their effec- or master’s degree).
tiveness (e.g., Rivkin, Hanushek & Kain 2005; However, these minimum qualifications have
Nye, Konstantopoulos & Hedges 2004; Kane, not been found by researchers to be strongly
Rockoff & Staiger 2008).1 Furthermore, predictive of student outcomes on standardized
teacher quality has been found to vary across tests (e.g., Goldhaber 2008; Hanushek &
schools in a way that systematically disadvan- Rivkin 2006; Kane, Rockoff & Staiger 2008).
tages poor, low-achieving, and racially isolated Knowing that a teacher possesses a teaching
schools (e.g., Clotfelter, Ladd & Vigdor 2005; certificate, a master’s degree, or a relevant col-
Lankford, Loeb & Wyckoff 2002; Boyd et al. lege major often tells us little about that
2008). teacher’s likelihood of success in the classroom.
But questions remain as to whether value-added There are many reasons not to totally dismiss
measures are a valid and appropriate tool for
identifying and enhancing teacher effectiveness.
1
This literature is frequently misinterpreted as stating that teacher
In this report, I aim to provide an accessible quality is more important for student achievement than any other fac-
introduction to these new measures of teaching tor, including family background. Statements such as “Studies show
that teachers are the single most important factor determining stu-
quality and put them into the broader context dents’ success in school” have appeared in dozens of press
releases and publications in recent years. For an example, see
of concerns over school quality and achieve- the May 4, 2010 statement from the U.S. House Committee on
ment gaps. Using New York City’s Teacher Education and Labor at <http://edlabor.house.gov/newsroom/
2010/05/congress-needs-to-support-teac.shtml>. I know of no
Data Initiative and Houston’s ASPIRE (Acceler- study that demonstrates this.
ating Student Progress, Increasing Results &

Expectations) program as case studies, I assess
the potential for these measures to improve
outcomes in urban school systems. In doing so,
I outline some of the most important chal-
lenges facing value-added measures in practice.
these qualifications,2 but common sense sug- teaching effectiveness. Leaders of both political
gests this information alone can only be a parties have strongly endorsed linking teacher
crude indicator for differentiating teaching evaluation to student test scores, and founda-
effectiveness. tions such as the Bill & Melinda Gates Foun-
dation, the Milken Family Foundation, and the
Over the past fifteen years, research on teacher
Broad Foundation have provided significant
quality has adopted a new paradigm: measur-
financial resources to support these efforts. In
ing effectiveness on the basis of student out-
promoting Race to the Top – President Barack
comes, as opposed to teacher inputs (e.g.,
Obama’s $4 billion competitive grant program
Rivkin, Hanushek & Kain 2005; Rockoff
aimed at systemic education reform – President
2004; Sanders & Horn 1994). While out-
Obama (2009) stated, “Success should be
come-based measures of teaching effectiveness
measured by results. . . . That’s why any state
are not a new concept (Murnane & Cohen
that makes it unlawful to link student progress
1986; Odden & Kelley 2002), several forces
to teacher evaluation will have to change its
have converged to reorient the definition of
ways.” It is widely believed that Race to the
teacher quality around student achievement.
Top will serve as a template for the reauthoriza-
First, recent policies of high-stakes accountabil-
tion of the federal Elementary and Secondary
ity have increased pressure on schools to meas-
Education Act in coming years.
ure and demonstrate results. Given teachers’
close contact with students, the extension Race to the Top is quite specific in its concep-
of high-stakes accountability to individual tion of an “effective teacher” (see Appendix A).
teachers was perhaps inevitable. Second, new “Highly effective teachers” are those whose stu-
longitudinal data systems now exist that allow dents achieve high rates of growth, defined –
student achievement to be tracked over time narrowly – by the program as a change in test
and matched to classroom teachers.3 Arguably, scores between two or more points in time
no credible test-score–based system of teacher (U.S. Department of Education 2010). Supple-
evaluation could exist in the absence of mental measures are encouraged, as are alterna-
such systems. Third, advancements in data- tive metrics for teachers of non-tested grades
processing capacity and statistical modeling and subjects, but the primary emphasis rests
have yielded an array of value-added tech-
niques with the potential for isolating teachers’ 2
For example, these qualifications can be viewed as minimum
unique contribution to student outcomes. expectations for classroom teachers. Evidence that some practicing
teachers without these qualifications are as effective as those who
Perhaps most importantly, political and phil- possess them cannot tell us how the wholesale removal of these
minimum requirements would affect the quality of the teaching
anthropic preferences have aligned to bring force. Moreover, virtually all of the research on teacher qualifica-
tions has focused on a narrow measure of success: single-year
about seismic shifts in our conception of gains on standardized tests. We know little about how qualifica-
tions affect other outcomes, such as organizational stability, student
behavior and motivation, aspirations, engagement, persistence,
and the like.
3
The Data Quality Campaign, founded by the Bill & Melinda Gates
Foundation, tracks and supports state efforts to create these sys-
tems; see <www.dataqualitycampaign.org>.
2 Can Teachers be Evaluated by their Students’ Test Scores? Should They Be?
squarely on test-score growth. One of the pro- evaluation used in many public schools: infre-
gram’s major selection criteria, “Great Teachers quent classroom observations and a pro forma
and Leaders,” contributes at least 70 of the 500 tenure (Toch & Rothman 2008; Weisburg et
possible application points to the linking of al. 2009). But whether or not the shift to
teacher evaluation and student test perform- intensive use of value-added measures of effec-
ance. For example, in their applications, states tiveness will improve our nation’s system of
will be judged by the extent to which they or teaching and learning remains to be seen.
their districts (U.S. Department of Education Indeed, there are good reasons to believe these
2010): measures may be counterproductive.
• measure individual student growth;
• implement evaluation systems that use stu-
dent growth as a significant factor in evaluat-
ing teachers and principals;
• include student growth in annual evaluations;
• use these evaluations to inform professional
support, compensation, promotion, reten-
tion, tenure, and dismissal;
• link student growth to in-state teacher prepa-
ration and credentialing programs, for public
reporting purposes and the expansion of
effective programs;
• incorporate data on student growth into
professional development, coaching, and
planning.
Race to the Top and the “new view” of teacher
effectiveness have stimulated a largely produc-
tive and long-overdue discussion between
policy-makers, researchers, and the public over
how to assess teacher quality and address,
develop, and support under-performing teach-
ers. And it is fair to say that there is little
enthusiasm for the traditional model of teacher
Annenberg Institute for School Reform 3

2 What Is a Teacher’s Value-Added?
It is useful to distinguish between the theoreti-
forced a district to lay off one of its two
fourth-grade teachers, and the district prefers
to dismiss the least effective of the two, as evi-
cal conception of value-added and the methods denced by their state test results. Mrs. Apple-
used to calculate it for an actual set of ton’s students averaged a 42 on the most recent
teachers.4 In theory, a teacher’s value-added is math exam, while Mr. Johnson’s students aver-
the unique contribution she makes to her stu- aged a 75 (see Figure 1).
dents’ academic progress. That is, it is the por- Is it fair to say that Mr. Johnson is the more
tion of her students’ success (or lack thereof ) effective teacher? Not without more informa-
that cannot be attributed to any other current tion. Mr. Johnson’s higher scores could reflect a
or past student, school, family, or community host of factors that have nothing to do with his
influence. Any system of test-based teacher effectiveness in the classroom: greater family
accountability that does not have value-added resources and involvement, higher initial levels
as its ultimate object of interest risks crediting of student ability, superior third-grade instruc-
or faulting teachers for outcomes beyond their tion, greater out-of-school support, and so on.
control. The hypothetical contributions of these other
Despite its theoretical appeal, isolating a factors to average achievement are illustrated
teacher’s unique contribution is a very difficult by the colored bars in Figure 1. One could
exercise. A simple example will help illustrate look for a way to statistically “remove” the
this point. Suppose recent budget cuts have effects of these influences from the achieve-
ment measure, but many of the factors that
Figure 1
matter most – parental support and student
Factors affecting average achievement in ability, for example – are difficult, if not
two classrooms: hypothetical decomposition impossible, to quantify.
80 An alternative comparison is the extent of stu-
Teacher & other dent progress from year to year (see Figure 2).
70
Community
If we assume that many of the external factors
60 School
influencing a student’s fourth-grade achieve-
Family
Average student scores
ment are the same as those influencing her

50 Student
third-grade achievement, then the change in
40 the student’s score will cancel out these effects
and reveal only the impact of changes since the
30
third-grade test, with the year of fourth-grade
20 instruction being the most obvious.5 Impor-
tantly, the focus on gains takes into account
10
4
0 There are many excellent and readable introductions to value-
Mrs. Appleton Mr. Johnson added methods. In writing this section, I benefited greatly from
Braun (2005), Buddin et al. (2007), Koretz (2008), Rivkin (2007),
Harris (2009), and Hill (2009).
5
This statement assumes something about the scale on which
achievement is measured. I return to this point in sections 4 and 5.
that these teachers’ students began the year at achievement gains across teachers. In reality,
very different levels. The idea is illustrated in students are not randomly assigned to classes –
Figure 2: Mrs. Appleton’s students started out in many cases, quite purposefully so. Conse-
at a much different point than Mr. Johnson’s, quently, value-added methods use a statistical
but most of the factors determining these ini- model to answer the question: “How would
tial differences in achievement “net out” in the these students have fared if they had not had
average gain score. The remainder represents [Mrs. Appleton or Mr. Johnson] as a teacher?”
the impact of the fourth-grade teacher, as well
This is a difficult question that is taken up in
as other influences that may have produced
the next section. For now, it is useful to think
differential growth between the two tests.
of a teacher’s value-added as her students’ aver-
Figure 2 shows that Mrs. Appleton’s students age test-score gain, “properly adjusted” for
had an average gain of ten points, while Mr. other influences on achievement. The New
Johnson’s gained an average of four. Can we York City Teacher Data Initiative (TDI) and
now declare Mrs. Appleton the more effective the Houston ASPIRE programs are two promi-
math teacher? Do these gain scores represent nent value-added systems that have their own
these teachers’ value-added? Not necessarily. methods for “properly adjusting” student test
While we may have removed the effects of scores. These two programs and their methods
fixed differences between student populations, are described in the next section.
we need to be confident that we have
accounted for other factors that contributed to Figure 2
changes in test performance from third to Factors affecting year-to-year test score gains in two classrooms: hypothetical
decomposition
fourth grade. These factors are potentially
numerous: family events, school-level interven- 80
tions, the influence of past teachers on knowl- Teacher & other
70
edge of this year’s tested material, or a disrup- Community
Student scores, averages and points gained
tive or especially helpful student in the class, School

60
Family
among others. Many of these factors are ran-
50 Student
dom events, while others systematically affect
teachers from year to year.
40
If value-added measures are to be successfully
30
used in practice to recognize effective teachers,
one needs a high level of confidence in the 20
attribution of achievement gains to specific
teachers. Were students randomly assigned to 10
teachers, this would be straightforward: any
0
systematic differences between classroom 3rd grade 4th grade Gain 3rd grade 4th grade Gain
achievement gains would almost certainly be Mrs. Appleton’s students Mr. Johnson’s students
due to the teacher. All other factors influencing
year-to-year changes would effectively average
out, allowing us to detect real differences in

3 Value-Added in Practice: New York City
and Houston
D.C., have made extensive use of value-added
methods in rewarding and evaluating teacher
effectiveness. Although the New York and
The New York City Department of Education Houston programs have similar goals, they
(NYCDOE) and Houston Independent School have quite different histories and methodolo-
District (HISD) have been at the forefront of gies, as described in this section.
developing and implementing value-added
The New York City Teacher Data Initiative
measures of teacher effectiveness in their dis-
tricts. Other large school districts, including In January 2008, the New York Times revealed
Dallas, Denver, Minneapolis, and Washington, that the NYCDOE was conducting an experi-
ment with a team of academic researchers that
randomly distributed teacher value-added
Figure 3
New York City Teacher Data Initiative timeline
reports to 140 participating school principals
and collected subjective teacher evaluations
2007 from these and an additional 140 control prin-
SUMMER 2007
cipals (Medina 2008b; Rockoff et al. 2010).
The experiment was intended to reveal whether
TDI experiment begins
and how principals use teacher value-added
JANUARY 2008 2008 reports in practice. At the time, the NYCDOE
publicly supported the idea of using test-
Experiment reported in
New York Times MARCH 2008 score–based measures of teacher performance,
but had no official position on how these
Bill passes Assembly
banning use of scores measures should be used (Medina 2008b;
in teacher evaluations Keller 2008).6
OCTOBER 2008
Two months later – possibly in response to the
Klein and Weingarten
announce expansion New York Times revelation – the New York
of TDI State Assembly passed a controversial bill that
2009 JANUARY 2009
barred New York City and other districts in the
More than 12,000 state from tying teacher tenure decisions to stu-
Teacher Data Reports
released to schools dent test scores (Medina 2008a) (see Figure 3
JULY 2009
for a timeline of these events). The bill – sup-
New contractor ported by then-president of the United Federa-
announced
tion of Teachers (UFT) Randi Weingarten –
2010
was later signed into law by Governor David
FEBRUARY 2010
Patterson, intended to be in effect through
Year 2 of Teacher
June 2010.
Data Reports released
to schools
MAY 2010
State and teachers

union agree to tie
40% of job evaluation
to student achievement
Despite the official ban on using test scores for and teachers in an accessible and usable form.
tenure decisions, the NYCDOE pressed for- The reports are to be viewed as “one lens” on
ward with the TDI, with conditional support teacher quality that should be “triangulated”
from Weingarten and the UFT. In accordance with other information about classroom effec-
with the law, TDI information was explicitly tiveness to improve performance. The reports
not to be used for rewarding or dismissing are seen as an “evolving tool” that will con-
teachers. As stated in a joint Klein/Weingarten tinue to be refined over time based on princi-
letter to teachers in 2008, the TDI’s value- pal and teacher feedback. Second, it is hoped
added reports were to be used solely for profes- the reports will “stimulate conversation” about
sional development, to “help [teachers] pin- student achievement within schools and pro-
point [their] own strengths and weaknesses, mote better instructional practices through
and . . . devise strategies to improve” (Klein & professional development. Finally, the measures
Weingarten 2008). will help the district learn more about “what
works” in the classroom. Value-added measures
The TDI released its first complete set of
have already enabled a wide range of studies on
Teacher Data Reports to more than 12,000
teacher effectiveness in New York City (e.g.,
teachers in 2009. These reports consisted of
Boyd et al. 2009; Kane, Rockoff & Staiger
separate analyses of English language arts
2008; Rockoff 2008), and the introduction of
(ELA) and mathematics test results and were
these measures into schools will enable addi-
generated for teachers who had taught these
tional research. In 2009, the NYCDOE and
subjects in grades four to eight in the prior
UFT signed on to participate in a historic,
year. (A more detailed description of the report
large-scale Gates Foundation study – the Meas-
itself is provided later in this section.) A second
ures of Effective Teaching, or “MET” project –
year of reports was released in 2010, reflecting
that intends to benchmark value-added meas-
significant revisions made by the NYCDOE
ures against alternative measures of teaching
and its new contractor, the Wisconsin Value-
effectiveness and identify practices that are
Added Research Center.7
associated with high value-added (Medina
The NYCDOE has expressed several broad 2009).9
goals for the TDI program.8 First, its data
New York’s law banning the use of test scores
reports are intended to provide measures of
for teacher evaluation would later complicate
value-added that can be reported to principals
the state’s application for Race to the Top
funding. Guidelines for the federal grant pro-
6
In another article appearing that month, then–Deputy Chancellor gram explicitly penalized states with such laws,
Chris Cerf stated that he was “unapologetic that test scores must
be a central component of evaluation” (Keller 2008). and Mayor Bloomberg and members of the
7
See <http://varc.wceruw.org/>. state legislature pushed for a reversal. Speaking
8
This description is compiled from a phone interview with Amy in Washington in November 2009, Mayor
McIntosh, Joanna Cannon, and Ann Forte of the NYCDOE
(January 14, 2010), an October 2008 presentation by Deputy
Chancellor Chris Cerf (“NYC Teacher Data Initiative”), and a
September 2008 training presentation by Martha Madeira (“NYC
Value-Added Data for Teachers Initiative”).
9
For information on the MET project, see <http://metproject.
org/project>.

Bloomberg announced that he had “instructed Data Reports changed their evaluations of
City Schools Chancellor Joel I. Klein to begin teachers in response to the new information.
using student performance data immediately to Less effective teachers were more likely to leave
inform teacher tenure decisions.” 10 The mayor their schools or be assigned “unsatisfactory”
further encouraged the state legislature to man- ratings when their principal received a value-
date the use of student performance data in added report.
teacher evaluation systems statewide. The law
Early indications – such as the value-added
was not repealed in time for the state’s first bid
experiment described above – suggest that
for Race to the Top, and New York ultimately
principals can and will use value-added infor-
failed to receive an award in the first round.
mation in their assessment of teacher effective-
For its second Race to the Top bid in summer ness. More recent data released by the NYC-
2010, New York took dramatic steps toward DOE showed that a higher fraction of
tying teacher evaluations to student progress. low–value-added teachers were denied tenure
An agreement between the state department of in 2010 than were high–value-added teachers
education and the teachers’ unions linked 40 (Martinez 2010). If principals are to use value-
percent of teachers’ performance evaluations to added reports in making consequential person-
student achievement measures (Medina 2010), nel decisions, it is critical that their underlying
though these measures were not based on measures be valid, reliable, and precise indica-
value-added alone. The agreement did not go tors of teacher effectiveness (criteria that are
as far as some states – including Florida, Indi- described in Section 4). Just as importantly,
ana, Rhode Island, Tennessee, and Colorado – users must have a rich understanding of their
that chose to base more than half of teachers’ methods and limitations.
job evaluations on student performance, but it
Appendix B illustrates a sample New York City
was a major departure from the 2008 law ban-
Teacher Data Report from 2010 produced by
ning the practice altogether.11 In 2010, the
the NYCDOE for “Mark Jones,” a fictional
U.S. Department of Education announced
eighth-grade teacher at “I.S. 000.” This partic-
New York as one of the Race to the Top win-
ular report is for mathematics; if Mr. Jones also
ners, awarding the state $700 million in federal
taught ELA, he would receive a separate report
aid.
for that subject.
The results of the 2008 NYCDOE value-
The Teacher Data Report provides value-added
added experiment were finally released to the
measures summarized in a number of ways.
public in July 2010 (Rockoff et al. 2010).
(The calculation itself is explained later in this
Authors of the study found that value-added
section.) The most important thing to note is
measures on the Teacher Data Reports were
that the key metric on the report is the
positively related to principals’ subjective rat-
teacher’s value-added percentile in the citywide
ings of teachers collected prior to their knowl-
distribution of teachers teaching the same
edge of the test results. That is, teachers
grade and subject, with similar amounts of
deemed more effective based on value-added
were more likely to have been rated as effective
by their principals. More importantly, however,
principals randomly selected to receive Teacher
experience (in Mr. Jones’s case, ten years). That would appear to place him squarely in the
is, results are not reported in units of “achieve- “average” performance category.
ment,” but rather as percentile rankings.12
Another element of the Teacher Data Report
Thus, value-added, in practice, is a relative worth noting is the reported range of per-
concept. Teachers are, in effect, graded on a centiles associated with Mr. Jones’s value-added
curve – a feature that is not always obvious to ranking (the black line extending in two direc-
most observers. A district with uniformly tions from his score). In statistical terminology,
declining test scores will still have “high” and this range is referred to as a “confidence inter-
“low” value-added teachers; a district’s logical val.” It represents the level of uncertainty asso-
aspiration to have exclusively “high value- ciated with the value-added percentile measure.
added” teachers is a technical impossibility. As the report’s instructions describe these
The value-added percentile simply indicates ranges: “We can be 95 percent certain that this
where a teacher fell in the distribution of teacher’s result is somewhere on this line, most
(adjusted) student test-score gains. likely towards the center.” These ranges – or
confidence intervals – are discussed more in
Value-added is reported for both last year’s test
Section 4. For now, note that Mr. Jones’s range
results (in this case, 2008-2009) and on all
for the prior year’s test extends from (roughly)
prior year’s test results for that teacher (in this
the 15th percentile to the 71st. Based on his
example, the last four years). Mr. Jones’s value-
last four years, his range extends from the
added places him in the 43rd percentile among
32nd percentile to the 80th. His value-added
eighth-grade math teachers last year; that is, 43
percentiles – 43 and 56 – fall in the middle of
percent of teachers had lower value-added than
these ranges.
he did (and 57 percent had higher value-
added). His value-added based on the last four On the second page of the data report (see
years of results places him in the 56th per- <http:/schools.nyc.gov>), value-added measures
centile. The percentiles are then mapped to and percentiles are reported for several sub-
one of five performance categories: “high” groups of students: initially high-, middle-, and
(above the 95th percentile), “above average” low-achieving students (based on their prior
(75th to 95th), “average” (25th to 75th), year’s math achievement); boys and girls; Eng-
“below average” (5th to 25th), and “low” lish language learners; and special education
(below 5th). Mr. Jones’s percentile rankings students. Mr. Jones performed at the “above
average” level with his initially high-achieving
10
Press release PR-510-09, Office of the Mayor, November 25, students, but fell into the “average” category
2009.
11
for all other subgroups. Ranges, or confidence
On Colorado and Tennessee, see “Colorado Approves Teacher
Tenure Law,” Education Week, May 21, 2010, and “Tennessee intervals, are also reported for each of these
Lawmakers Approve Teacher Evaluation Plan,” Memphis Commer-
cial Appeal, January 15, 2010.
subgroups.
12
I address the reported “proficiency” scores (e.g., 3.27, 3.29) later How are these value-added percentiles calcu-
in the report.
13
A useful and concise explanation is provided at the top of the
lated, exactly? 13 Recall that a teacher’s value-
Teacher Data Report itself. We benefited from a more technical added can be thought of as her students’ aver-
explanation of the 2009 methodology in an internal 2009 techni-
cal report by the Battelle Memorial Institute. The model is similar to
that estimated by Gordon, Kane, and Staiger (2006) and Kane,
Rockoff, and Staiger (2008).

age test scores, “properly adjusted.” New York Mr. Johnson’s fourth-grade students, Melissa
City operationalizes this idea by comparing and Doug, earned a 42 and a 65 on the test,
students’ actual scores under a given teacher to respectively. As noted in Section 2, there are a
their predicted score. This predicted score can host of possible explanations for their perform-
be thought of as each student’s counterfactual ance on this test, among which Mr. Johnson’s
level of achievement – that is, their predicted instruction is but one. For example, Melissa
achievement had they been taught by a differ- was identified last year with a learning disabil-
ent teacher (say, the average teacher in the dis- ity, and her low third-grade math score – 35 on
trict). The prediction itself is based on a num- the same scale – reflected this. Doug scored rel-
ber of things, the most important of which is atively high on his third-grade test, earning a
the student’s prior achievement. How a student 72. Melissa comes from a single-parent family
actually performed under Mrs. Appleton rela- whose low income qualifies her for reduced-
tive to how he would have performed under a price lunches. Doug, on the other hand, has
different teacher represents Mrs. Appleton’s two upper-middle-class parents.
value-added for that student.
Because the school district has richly detailed
A stylized example will help specify this idea. data on thousands of students’ academic histo-
Suppose students are given a math test each ries, it can provide a statistical prediction of
year that is scored between 1 and 100. Two of how Melissa and Doug were likely to perform
on their fourth-grade math test. Effectively,
Figure 4 this prediction is the average district score for
New York City value-added model: predictors students with Melissa and Doug’s prior
achievement and other relevant characteristics.
Student Prior year achievement ELA
characteristics
(The full list of student, classroom, and school
Prior year achievement math
characteristics used in New York City’s model
Free or reduced-price lunch eligibility
is shown in Figure 4.) Suppose the model pre-
Special education status
dicts that Melissa – with her past achievement
English learner status
and other characteristics –would be expected to
Number of suspensions, absences
earn a 39 on her fourth-grade test, while Doug
Student retained in grade
Student attended summer school
would be expected to score a 76.
Student new to school The three-point difference between Melissa’s
Race actual score of 42 and her predicted score of
Gender 39 represents Mr. Johnson’s value-added to
Classroom Classroom averages of each of the above Melissa’s achievement. Similarly, the four-point
characteristics negative differential between Doug’s score of
Class size
72 and his predicted score of 76 represents Mr.
School Average classroom characteristics
characteristics Johnson’s value-added (or “value-subtracted’’?)
Average class size
to Doug’s achievement. If we repeat this exer-
Source: New York City Department of Education. Based on the 2009 cise for every student in Mr. Johnson’s class, we
Teacher Data Reports.
can average the results and call this his value-
added for that year.14 If Mr. Johnson has taught
for ten years, we could also average his stu- As a compromise, most value-added methods
dents’ value-added measures over all of those rescale test scores to have a mean of zero and a
years. standard deviation (SD) of one.15 This new
scale tells us where students are in the distribu-
A few key features of this approach are worth
tion of test scores in each grade. For example,
highlighting. First, students’ predicted scores
Melissa may have moved from a -0.25 to a
are based on how other students with similar
-0.20 on this scale, from 0.25 SDs below the
characteristics and past achievement performed
average third-grader to 0.20 below the average
– who were taught by other teachers in the dis-
fourth-grader, a gain of 0.05 SD. Under cer-
trict. Thus, value-added is inherently relative: it
tain assumptions, this scale is appropriate, but
tells us how teachers measure up when com-
it is important to keep in mind that students,
pared with other teachers in the district or state
like teachers, are being measured on a curve –
who are teaching similar students. Second, test
that is, relative to other tested students in the
scores are rarely of the vertical scale type sug-
same grade.
gested by the above example. That is, we can
rarely say that a student like Melissa moved Rather than reporting results on the admittedly
from a 35 to a 42 on the “math” scale as she less-than-transparent SD scale, the NYCDOE
progressed from third to fourth grade. converts value-added results to a scale with
which teachers are familiar: the state’s “perfor-
14
mance levels” (1 to 4). On Mark Jones’s report,
This concept is also known as the year-specific “teacher effect” or
“classroom effect” for that teacher and year. we see that his twenty-seven students were pre-
15
A standard deviation is a measure of variation in a distribution. dicted to perform at an average of 3.29 in
Loosely, it can be thought of as the “average difference from the
mean.” For example, the average score on a test might be a 70, math, somewhere between proficient (3) and
with a standard deviation of 8. We can think of this as saying that,
advanced (4).16 In practice, his class averaged
on average, students scored 8 points above or below the average
of 70. (This is not technically correct, but it is a useful way of think- 3.26 last year, for a value-added on this scale of
ing about the standard deviation.) The SD depends on the scale of
the original measure. Thus we often put measures on a common -0.03. Over Mr. Jones’s past four years, his
(“standardized”) scale with a mean of zero and SD of one. In this
value-added was +0.03. All of Mr. Jones’s sub-
example of a test with a mean of 70 and SD of 8, a student who
scored an 82 would receive a score of 1.5 on the alternate scale group results are presented in the same way,
(1.5 standard deviations above the mean).
16
with predicted and actual performance levels,
New York City’s “performance level” scale itself is somewhat puz-
zling. The performance levels of 1, 2, 3, and 4 are based on cut value-added, and a percentile based on this
scores determined by the state at certain points in the test score dis-
tribution. They are ordinal categories that represent increasing lev-
value-added. Of course, the number of stu-
els of tested skill (below basic, basic, proficient, and advanced). dents used to estimate each subgroup value-
They are not an interval measure, where the difference in skill
between, say, a 2 and a 3 is equivalent to that between a 3 and added measure is smaller; for example, Mr.
a 4, nor were they intended to be used in such a way. The NYC-
DOE further converts its scores to “fractional units” on this scale. For
Jones taught eighteen initially high-achieving
example, a student that receives a raw score that places them students and twenty-seven who initially scored
between the cut scores of 2 (“basic”) and a 3 (“proficient”) might
be assigned a performance level of 2.42. Because the proficiency in the middle one-third. He also taught more
categories are ordinal, it isn’t clear what a proficiency level of
2.42 means. It plausibly could be interpreted as being 42 percent
boys (forty-two) than girls (nineteen).
of the way between basic and proficient, but it remains that a
movement of 0.10 between each point on the scale (1 to 4) will
represent different gains in achievement. In practice, this unusual
system does not present a problem for the Teacher Data Reports.
There the value-added measures are calculated using standardized
scores and only converted to performance levels after the fact
(Source: internal Battelle Memorial Institute technical report, 2009).

Challenges associated with this method of eval- Houston has awarded bonuses for student per-
uating teachers are discussed in greater detail in formance since 2000, though its awards pro-
Section 4. Generally speaking, the value-added gram was initially tied to aggregate school-level
system in place in New York City is no worse test results (HISD 2009). A new system based
and no less valid than models used elsewhere. on teacher value-added – the Teacher Perform-
In fact, in many ways the city’s Teacher Data ance Pay Model – was announced in 2006,
Reports are superior to those found in other with its first set of awards totaling $17 million
districts. The reports themselves are easy to to be paid in 2007 (Blumenthal 2006). Hous-
read, and the results presented on the report ton’s most recent round of bonuses awarded
are explained in clear (and largely accurate) more than $40 million, up $8.5 million from
language. The 2010 report was substantially the prior year (ASPIRE 2010).
improved and simplified from the 2009 design
The Teacher Performance Pay Model gained
(pictured in Appendix C), though the underly-
substantial notoriety in its first year when
ing concepts and methods remain largely the
bonus payments to teachers and principals
same. On the back of the report, teachers are
were made public in the Houston Chronicle
encouraged to use their results for self-
and some highly recognized teachers failed to
improvement and are provided a template for
receive awards. Several months later, it was
thinking about them.
revealed that a computer glitch had overpaid
close to 100 teachers (Associated Press 2007).
The Houston ASPIRE Program
In September 2007, the HISD school board
Unlike the NYCDOE, the Houston Independ-
voted to overhaul the system, renaming the
ent School District (HISD) has made its
program ASPIRE and adopting the EVAAS
teacher value-added system an explicit part of
model (Olson 2007)17.
its performance pay plan since 2007. In recent
months, the district moved to allow dismissals ASPIRE consists of three tiers, or “strands.”
based on value-added. This section provides Strand I is a school-level award that rewards all
a brief overview of this nationally known staff in schools where students demonstrated
program. gains and whose progress ranked in the top
two quartiles for their grade level. The Strand I
ASPIRE, Houston’s teacher performance bonus
progress measure is a school-level value-added
program, is based on value-added calculated
measure conceptually similar to that described
using EVAAS (Education Value Added Assess-
for New York teachers. Strand II awards indi-
ment System), a value-added method pio-
vidual teachers whose students’ progress ranked
neered by William Sanders in Tennessee
in the top two quartiles for their grade and
(Sanders, Saxton & Horn 1997). EVAAS is
quite different from the model used by the
NYCDOE, but its objective is fundamentally
the same: isolating a teacher’s contribution to
student progress.
subject. The teacher performance measure is a superintendent Terry Grier stated that his
“value-added cumulative gain index” for a intended use for the value-added measures was
given teacher and subject. Finally, Strand III for them to be added to “the list of reasons that
offers a mix of additional bonus opportunities, can be used in teacher dismissal.” He expressed
including a bonus for attendance. a willingness to create “a screening process for
principals who propose that teachers gain term
Taken together, the three strands of ASPIRE
contract by requiring them to discuss the per-
amount to a maximum bonus that ranges from
formance/effectiveness of all probationary
$6,600 to $10,300 for classroom teachers.
teachers,” adding, “this discussion will include
Teachers of self-contained classes can receive
the review of value-added. . . . If principals
awards in Strand II in as many as five subjects:
want to grant term contracts to teachers with
reading, math, language arts, science, and
regressive value-added scores, . . . they should
social studies. A teacher scoring in the top
be able to provide a compelling reason for
quartile in all five subjects can receive a bonus
doing so” (Mellon 2010b).
as high as $7,000. According to the Houston
Chronicle, almost 90 percent of eligible school Although ASPIRE’s value-added model differs
employees received a bonus for 2008-2009, markedly from that used by the Teacher Data
with classroom teachers earning an average of Reports in New York City, the programs share
$3,606 and a maximum of $10,890 (Mellon the same core objective: differentiating teachers
2010a). based on their contribution to student achieve-
ment and recognizing and rewarding effective
The EVAAS model is considerably more com-
teachers. Houston’s decision to link its value-
plex and is much less transparent than the
added measures explicitly to pay and tenure
model adopted by the NYCDOE.18 The model
decisions were precursors to similar decisions in
combines results on multiple tests – the Texas
New York City in recent months. In the next
Assessment of Knowledge and Skills (TAKS)
section, I provide an overview of the most sig-
and the Stanford 10 Achievement Test (or the
nificant challenges facing value-added measure-
Aprenda, its Spanish language equivalent) and
ment in practice, drawing upon data from
“layers” multiple years of test results to calcu-
HISD and the New York City Department of
late teachers’ cumulative value-added (McCaf-
Education to illustrate these challenges.
frey et al. 2004). Like the New York City sys-
tem, expected scores in each year are estimated
17
for students in each subject and compared with ASPIRE currently receives funding from several sources: the Broad
and Bill & Melinda Gates foundations ($4.5 million), a U.S.
their actual scores. However, unlike the New Department of Education Teacher Incentive Fund (TIF) grant ($11.7
million), and a Texas District Assessment of Teacher Effectiveness
York City model, the predicted scores rely on a grant. The contract firm Battelle for Kids has provided professional
relatively sparse list of student background development for the ASPIRE program since 2007.
18
EVAAS has been sharply criticized for its lack of transparency and
characteristics. inadequate controls for student background. See, for example,
Amrein-Beardsley (2008).
In February 2010, the HISD board of educa-
tion voted to approve the use of value-added
measures in teacher tenure decisions (Sawchuk
2010). In a letter to the school board, HISD

4
Challenges to the Practical
Implementation of Value-Added
What is being measured?
Testable skills
Value-added measurement works best when
As exemplified by New York City’s Teacher
students receive a single objective numeric test
Data Reports and Houston’s ASPIRE program,
score on a continuous developmental scale –
value-added measures have been embraced by
that is, one that is not tied to grade-specific
school leaders nationwide as a means to objec-
content. The domain of skills that can be ade-
tively quantify teacher effectiveness and to
quately assessed in this way is, however,
reward and retain teachers with a demonstrated
remarkably small. For example, elementary
record of success. Few can deny these measures’
math skill may progress in a way that lends
intuitive appeal: if a statistical model can iso-
itself to annual standardized testing and a “ver-
late a teacher’s unique contribution to students’
tically equated” scale that spans multiple
educational progress, the possibilities for its
grades. Its basic computational and problem-
productive use appear endless. However, these
solving skills are relatively easy to assess on a
tools have limitations and shortcomings that
well-designed short-answer or multiple-choice
are not always immediately apparent. Before
test.
adopting these measures wholesale, policy-
makers should be fully aware of their limita- But history, civics, English literature, music,
tions and consider whether the benefits of their foreign language, critical thinking, writing, and
adoption outweigh the cost. research skills may not be so easy to assess in
this way, and it makes little educational sense
I categorize the conceptual and practical chal-
to force such skills to conform to such a struc-
lenges to value-added methods of evaluating
ture purely for value-added assessment. For this
teachers into six areas:
reason, skills readily assessed by standardized
• What is being measured? tests reflect only a small fraction of what stu-
• Is the measurement tool appropriate? dents are expected to know and do. Not all
• Can a teacher’s unique effect be isolated? subjects are or can be tested, and even within
• Who counts? tested subject areas, only certain skills readily
• Are value-added scores precise enough to be conform to standardized testing. These points
useful? are made so frequently that they have virtually
• Is value-added stable from year to year? lost all meaning; we simply shrug and
acknowledge that of course, tests don’t capture
everything. Yet value-added measures of teach-
ing effectiveness rest exclusively on skills assess-
able on very narrow standardized tests.
In a recent essay, economist Alan Blinder
(2009) persuasively argued that the skills vital
for success in the labor market in the near
future will be those least amenable to standard-
ized testing: “skills that a computer cannot Teachers or schools?
replicate and that an educated worker in a low- Even in cases where tests do adequately capture
wage country will have a hard time doing” (p. desired skills, it behooves us to ask whether
22). These skills, Blinder argues, include “cre- value-added – a teacher’s individual impact on
ativity, inventiveness, spontaneity, flexibility, students’ academic progress – is, in fact, what
[and] interpersonal relations . . . not rote is educationally relevant. Teachers certainly
memorization” (p. 22). Similarly, in their book vary in effectiveness, and school leaders should
calling for a broader conceptualization of be cognizant of their teachers’ contribution to
school accountability, Rothstein, Jacobsen, and student success. Yet to the extent schooling is a
Wilder (2008) highlight the broad scope of group or team effort involving principals,
skills that students develop in school, including teachers, and other school professionals (e.g.,
“the ability to reason and think critically, an instructional coaches, librarians, counselors),
appreciation of the arts and literature, . . . Herculean efforts to isolate, report, and reward
social skills and a good work ethic, good citi- individual value-added ignores critical, interre-
zenship, and habits leading to good physical lated parts of the educational process. At worst,
and emotional health.” narrow interest in individual results may
This is not to say that value-added measure- undermine this process, a point I return to
ment cannot aid in evaluating certain basic – later.
and even critically important – skills. Rather, This concern is hardly unique to education.
they are simply too narrow to be relied upon as Statistics on narrow metrics of individual pro-
a meaningful representation of the range of ductivity have their place in many organiza-
skills, knowledge, and habits we expect teach- tions, from business and government to profes-
ers and schools to cultivate in their students. sional sports. Yet in most cases business leaders
and athletic coaches recognize that the success
of their organization is much more than the
sum of their individual employee or player sta-
tistics (Rothstein 2009). HISD, and, to a lesser
extent, the NYCDOE, with its small school-
based performance bonus program (Springer &
Winters 2009), have recognized that organiza-
tional outcomes are as important to recognize
as individual successes. As value-added systems
begin to be implemented in school systems
nationwide, policy-makers should be aware of
the potential educational costs of a narrow
focus on individual metrics.

Is the measurement tool appropriate? Even for the standards that can be tested, many
assessments are poor representations of these
In assessing readily testable skills – fourth-
standards. Recent studies analyzing state test
grade arithmetic, for example – a measurement
content in New York, Massachusetts, and Texas
tool is needed that provides a valid and reliable
find that over many years of test administra-
assessment of students’ mastery of the skill
tion, some parts of the state curriculum are
domain. No test will cover all skills that stu-
never tested (Jennings & Bearak 2010; Hol-
dents are expected to master. By necessity, a
colmbe, Jennings & Koretz 2010). To take one
test instrument must sample items from a
extreme case – the 2009 New York State
much broader domain of skills. The resulting
eighth-grade math test – 50 percent of the pos-
test may consist of, say, thirty-five to fifty mul-
sible points were based on only seven of the
tiple-choice questions. A student’s performance
forty-eight state standards; what’s more, only
on this test then provides an inference, approx-
51 percent of the points were required to pass.
imation, or “educated guess” of his or her mas-
tery of the broader skill domain (Koretz 2008). Among skills that are consistently assessed,
some receive predictably greater emphasis than
A well-constructed test that draws evenly from
others. Teachers aware of systematic omissions
the broader domain of skills is more likely to
and repetitions can substantially inflate stu-
provide a valid inference about student learn-
dents’ scores by narrowly focusing on fre-
ing. In the case of state tests – such as those
quently tested standards (popularly known as
administered in New York and Texas – the rele-
“teaching to the test”).20 For many tests, it is
vant domain is the state curriculum, which
also possible to “teach to the format.” Studies
articulates what students should know and be
have illustrated how teachers focus their
able to do at each grade level. As noted above,
instruction on the format of the state test by
however, many of these skills are not amenable
presenting material in the same manner as it
to standardized testing and inevitably will be
appears on the test (Darling-Hammond &
under-represented on the state test.19
Wise 1985; Shepard & Dougherty 1991;
19
As two examples, the New York State standards for ELA include,
McNeil & Valenzuela 2000).21 To the extent
“as speakers and writers, students will use oral and written lan- “teaching to the test” and “teaching to the for-
guage for self-expression and artistic creation” and “students will
use oral and written language for effective communication with a mat” behaviors differ across teachers – which
wide variety of people.” Few would disagree that these are impor-
tant objectives; they are not, however, skills that are easily assessed
they almost certainly do – true “value-added”
on a standardized test. comparisons will be compromised.
20
For example, when a novice math teacher’s colleague informs her
that the state test “never asks students to calculate the volume of a The fact that test items are sampled from a
cylinder,” that teacher can reliably improve scores by devoting less
time on this concept. broader domain is relevant for more than just
21
A striking example is reported in Shepard (1988). In that study, “gaming” behavior. Teachers vary in the extent
when a set of questions involving the addition of decimals was pre-
sented in a vertical format – as was standard on the state test – 86
to which their time and efforts align with con-
percent of students answered these questions correctly. When the tent specifically emphasized on the state test,
same problems were presented in a horizontal format, only 46 per-
cent of students did.
22
These categories are quintiles based on value-added. Only stu-
dents who took both the TAKS and Stanford Achievement Test are
used to generate the value-added estimates.
for a variety of valid reasons. This variation were based on the same students, tested in the
may be due to the average ability level in their same subject, at approximately the same time
classroom, priorities of school leadership, of year, using two different tests.
parental demands, and so on. Given two teach-
We found that a teacher’s value-added can vary
ers of equal effectiveness, the teacher whose
considerably depending on which test is used.
classroom instruction happens to be most
This is illustrated in Figure 5, which shows
closely aligned with the test – for whatever rea-
how teachers ranked on the two reading tests.
son – will outperform the other in terms of
Teachers are grouped into five performance
value-added.
categories on each test (1 to 5), with the five
Evidence that the choice of test can make a TAKS categories on the horizontal axis.22 We
difference to value-added comes from recent see that teachers who had high value-added on
research comparing value-added measures on one test tended to have high value-added on
multiple tests of the same content area. Since the other, but there were many inconsistencies.
1998, Houston has administered two standard- For example, among those who ranked in the
ized tests every year: the state TAKS and the top category (5) on the TAKS reading test,
nationally normed Stanford Achievement Test. more than 17 percent ranked among the low-
Using HISD data, we calculated separate est two categories on the Stanford test. Simi-
value-added measures for fourth- and fifth- larly, more than 15 percent of the lowest value-
grade teachers for the two tests (Corcoran, added teachers on the TAKS were in the
Jennings & Beveridge 2010). These measures highest two categories on the Stanford.
Figure 5
Teacher value-added on two reading tests: Houston fourth- and fifth-grade teachers
50
Q1-Stanford
Q2-Stanford
Percentages of Stanford quintiles in TAKS
40
Q3-Stanford
Q4-Stanford
30 Q5-Stanford
20
10
0
Q1 Q2 Q3 Q4 Q5
Quintile of value-added: TAKS Reading
Source: Corcoran, Jennings & Beveridge (2010)

These findings are consistent with those from Can a teacher’s unique effect be isolated?
other districts, including several large districts
As described in Section 2, the successful use of
in Florida (Sass 2008) and a large anonymous
value-added in practice requires a high level of
district in the northeast (Papay 2010). To show
confidence in the attribution of achievement
what kind of an impact these inconsistencies
gains to specific teachers. One must be fairly
might have in practice, Papay calculated hypo-
confident that other explanations for test-score
thetical ASPIRE bonuses using his two differ-
gains have been accounted for before rewarding
ing sets of value-added estimates. He found
or punishing teachers based on these measures.
that “simply switching the outcome measure
We saw earlier that something as simple as test
would affect the performance bonus for nearly
timing can complicate the apportionment of
half of all teachers and the average teacher’s
gains between teachers. In practice, there are a
salary would change by more than $2,000” (p.
countless number of factors that hinder our
3). He concludes that the two value-added esti-
ability to isolate a teacher’s unique effect on
mates “are not sufficiently comparable to rank
achievement.
consistently the same individual teachers as
high- or low-performing” (p. 3). Given one year of test score gains, it is impossi-
ble to distinguish between teacher effects and
One of Papay’s explanations for variation in
classroom-level factors. In a given year, a class
value-added across tests is when the test was
of students may perform particularly well or
administered. That is, teachers’ value-added
particularly poorly for reasons that have noth-
looks different depending on whether the test
ing to do with instruction. The proverbial
was given in the early fall, mid-spring, or the
“barking dog” on test day is one such explana-
end of the school year. Differences in test tim-
tion, as is a classroom illness or particularly dis-
ing impact some teachers more than others,
ruptive student who affects the quality of
particularly those serving poor students who
instructional time. Over many years, this varia-
suffer a “summer learning loss” relative to their
tion averages out, but in a single year the
more advantaged peers (Alexander, Entwisle &
impact of the teacher cannot be separated from
Olsen 2001). When testing occurs in the mid-
these influences. More years of test results
dle of the year, value-added measures are made
helps, but this may be of little comfort to a
more difficult in that one has to apportion
teacher or school leader looking for actionable
learning gains between two teachers. Until
information today.
recently, New York State administered its grade
three to eight exams in January and March; Most value-added models used in practice –
consequently, test score gains between two tests including New York City’s – also fail to sepa-
were due to two teachers, not one. It is not rate teachers’ influence from the school’s effect
obvious how one appropriately isolates one on achievement.23 That is, they don’t account
teacher’s impact from the other. for the fact that performance differs systemi-
cally across schools due to differences in school
policy, leadership, discipline, staff quality, and
student mix. This omission is not simply an
oversight. Value-added experts have pointed
out, rightly, that teacher effectiveness varies Who counts?
across schools within a district and to focus
Another significant limitation of value-added
only on variation within schools would ignore
systems in practice is that they ignore a very
important variation in teacher quality across
large share of the educational enterprise. Not
schools (e.g., Gordon, Kane & Staiger 2006).
only do a minority of teachers teach tested
The cost of this view, however, is that teacher
subjects, but not all students are tested, and
effects end up confounded with school
not all tested students contribute to value-
influences.
added measures. In other words, from the
Recent research suggests that school-level fac- standpoint of value-added assessment of
tors can and do affect teachers’ value-added. teacher quality, these students do not count.25
Jackson and Bruegmann (2009), for example,
In most states, including New York and Texas,
found in a study of North Carolina teachers
students are tested in reading and mathematics
that students perform better, on average, when
annually in grades three to eight, and again in
their teachers have more effective colleagues.
high school. Other subjects, including science
That is, Mrs. Appleton might have higher
and social studies, are tested much less often.26
value-added when teaching next door to Mr.
Because value-added requires a recent, prior
Johnson, because she benefits from his example,
measure of achievement in the same subject
his mentoring, and his support. Other studies
(usually last year’s test score), only teachers of
have found effects of principal leadership on
reading and math in grades four to eight can
student outcomes (Clark, Martorell & Rockoff
be assessed using value-added. Without annual
2009). Consequently, teachers rewarded or
tests, teachers cannot be assessed in other sub-
punished for their value-added may, in part, be
rewarded or punished based on the teachers
23
Technically, the value-added model often does not include “school
with whom they work.24 This possibility cer- effects.”
tainly runs counter to the intended goal of 24
In another study, Rothstein (2010) finds that a student’s fifth-grade
value-added assessment. teacher has large effects on her fourth-grade achievement, a tech-
nical impossibility given that the student has not yet advanced to the
fifth grade. He suggests that this finding may be due to “dynamic
Finally, as argued earlier, in many contexts, tracking,” where a student’s assignment to a fifth-grade teacher
attempts to attribute achievement gains to indi- depends on their fourth-grade experience. When such assignment
occurs, it biases measures of value-added.
vidual teachers may not make sense in princi- 25
This is not a concern unique to teacher value-added measurement.
ple. This is most true in middle and high The same issue arises when considering the design and effects of
school accountability systems. When state testing systems (appropri-
school, when students receive instruction from ately) allow exclusions for certain categories of students, incentives
are created for schools to reclassify students such that they are
multiple teachers. To assume that none of these exempted from the tests (see Figlio & Getzler 2002; Jacob 2005;
teachers’ effects “spill over” into other course- and Jennings & Beveridge 2009).
26
work seems a strong – and unrealistic – In New York, students are tested in social studies in fifth and eighth
grade, and science in fourth and eighth grade. See <www.
assumption. Indeed, Koedel (2009) found that emsc.nysed.gov/osa/schedules/2011/3-8schedule1011-
021010.pdf>.
reading achievement in high school is influ-
enced by both English and math teachers.
Learning may simply not occur in the rigid way
assumed by current value-added models.

jects such as science and social studies. Thus, shows the percentage of students in grades four
elementary, middle, and high school teachers to six over eight years of testing in Houston
of subjects other than reading and math are who were tested in reading, and the percent of
necessarily excluded from value-added assess- students who also had a test score for the prior
ment. Moreover, in many districts, students in year (and thus could contribute to a value-
grades six to eight attend middle schools where added estimate).27 Due to disabilities, very low
they receive instruction from multiple teachers. competency in the English language, absen-
As discussed in the last section, attributing teeism, and a variety of other reasons, approxi-
achievement gains to individual teachers in mately 14 percent of students in HISD did not
these settings is especially problematic. have a test score in a typical year. This fraction
varies by subgroup, depending on the extent of
Some students are routinely exempted from
exemption in each group; for example, 15.6
testing, or for one reason or another are miss-
percent of Black students failed to have a test
ing a test score. Figure 6 illustrates how miss-
score, and 26 to 29 percent of recent immi-
ing data can affect “who counts” toward a
grants and ESL students were not tested.
teacher’s value-added assessment. This figure
Figure 6
Percent of students with a test score and percent contributing to value-added esti-
mates, grades four to six, Houston,1998–2006
100
All students
90 % with a test score
80 % with a test score
and a lag score
70
60
Percentage
50
40
30
20
10
0
All Economically Black Hispanic Recent ESL
students disadvantaged immigrants
Quintile of value-added: TAKS Reading
Source: Author’s calculations using data from the Houston Independent School District. The percent of
students with a test score and a lag score is only calculated for students in grades four to six (third
grade is the first year of testing).
Because of high rates of student mobility in Are value-added scores precise enough
this population (in addition to test exemption to be useful?
and absenteeism), the percentage of students As described in sections 2 and 3, value-added
who have both a current and prior year test is based on a statistical model that effectively
score – a prerequisite for value-added – is even compares actual with predicted achievement.
lower (see Figure 6). Among all grade four to The residual gains serve as an estimate of the
six students in HISD, only 66 percent had teacher’s value-added. Like all statistical esti-
both of these scores, a fraction that falls to mates, however, value-added has some level of
62 percent for Black students, 47 percent for uncertainty, or, a margin of error. In New York
ESL students, and 41 percent for recent City’s Teacher Data Reports, this uncertainty is
immigrants. expressed visually by a range of possible per-
The issue of missing data is more than a tech- centiles for each teacher’s performance (the
nical nuisance. To the extent that districts “confidence interval” for the value-added
reward or punish teachers on the basis of score). Some uncertainty is inevitable in value-
value-added, they risk ignoring teachers’ efforts added measurement, but for practical purposes
with a substantial share of their students. it is worth asking: Are value-added measures
Moreover, they provide little incentive for precise enough to be useful in high-stakes deci-
teachers to invest in students who will not sion-making or for professional development?
count toward their value-added. Unfortunately, Let’s return to the case of Mark Jones, whose
districts like New York City and Houston have data report is shown in Appendix B. Based on
very large numbers of highly mobile, routinely last year’s test results, we learned that Mr. Jones
exempted, and frequently absent students. ranked at the 43rd percentile among eighth-
Moreover, these students are unevenly distrib- grade teachers in math. Taking into account
uted across schools and classrooms. Teachers uncertainty in this estimate, however, his range
serving these students in disproportionate of plausible rankings range from the 15th to
numbers are most likely to be affected by a the 71st percentile. Although the 43rd per-
value-added system that – by necessity – centile is our best estimate of Mr. Jones’s per-
ignores many of their students. formance, we can’t formally rule out estimates
ranging from 15 to 71. Using the NYCDOE
27
The latter is calculated only for students in grades four to six.
performance categories, we can conclude that
Because third grade is the first year of testing, none of these stu- Mr. Jones is a “below average” teacher, an
dents have a prior year score.
“average” teacher, or perhaps a borderline
“above average” teacher.
What is the source of this uncertainty, exactly?
Recall that value-added measures are estimates
of a teacher’s contribution to student test-score
gains. The more certain we can be that gains
are attributable to a specific teacher, the more
precise our estimates will be (and the more

narrow the estimate’s confidence interval). Jones is most likely an average teacher, ranking
With one year of data, it is impossible to sepa- somewhere around the 43rd and 56th per-
rate teacher effects from other factors affecting centile citywide, who may be below average,
the performance of students within a classroom average, or above average. Accounting for
(such as a disruptive classmate). While a partic- uncertainty, we can’t rule out a twenty-point
ularly high set of test-score gains is suggestive margin in either direction.
of an effective teacher, one can only know if
It is unclear to this author what Mr. Jones or
these gains are systematic after additional years
his principal can do with this information to
of observations. These additional years are par-
improve instruction or raise student perform-
ticularly important when scores are “noisy,”
ance. More years of teaching experience allow
that is, when achievement gains are not well
the data to tell a slightly more precise story,
explained by variables accounted for in the sta-
but knowing that better data will be available
tistical model.28
in the future is of little use to Mr. Jones or his
Value-added estimates become more precise principal, who are looking for actionable infor-
with additional years of data, as Mr. Jones’s mation in real time. Value-added results for
report illustrates. Based on his last four years of student subgroups would appear to be more
results, Mr. Jones ranked in the 56th per- promising to the extent they highlight subject
centile, though his plausible range still extends areas or target populations in need of improve-
from the 32nd percentile to the 80th, which ment – students who are English language
overlaps the “average” and “above average” per- learners, for example. Yet in most cases, the
formance categories. Taking his two sets of number of students used to calculate these sub-
results together, we have learned that Mark group estimates is so small that the resulting
level of uncertainty renders them meaningless.
28
McCaffrey et al. (2009) show that much of the variation in teacher It is worth noting that – by design – 50 per-
effects is due to random noise.
29
cent of teachers will perennially fall in the
It is often argued that policies of differentiating teacher effectiveness
on the basis of test scores will lead to a long-run increase in the “average” performance category on the Teacher
number of outstanding graduates willing to enter the teaching pro-
fession. Because highly effective teachers are not currently
Data Report; another 40 percent will be con-
rewarded for their efforts – through higher salaries, for example, or sidered “below average” or “above average. The
promotion – candidates who are likely to be effective may be less
likely to pursue a teaching career (Hoxby & Leigh 2004). How- remaining 10 percent are either “exceptional”
ever, there is currently little available evidence on either side of this
question. It may turn out that the system for differentiating effective-
(in the top 5 percent) or “failing” (in the bot-
ness matters. In the case of the NYCDOE Teacher Data Reports, tom 5 percent). Thus, out of all teachers issued
only a modest fraction of teachers will be deemed “above aver-
age” or highly effective (top 5 percent). It could be that this has a a value-added report in each year, half will be
discouraging effect on excellent teachers. In the words of a teacher
who recently blogged about his data report results: “In fact, I’m
always be told little more than that they are
mostly wholly average as an educator when it comes to teaching “average.”29 At most, one in three will receive a
both math and reading. Not exactly the vote of confidence I was
looking for. . . . This is disappointing to say the least. I did not join signal to improve, though wide confidence
NYC Teaching Fellows to be an average teacher” (Brosbe 2010).
intervals may and should raise doubt in the
minds of some “below average” teachers. Of
course, teachers who persistently score in the
“high” category are probably doing something
right and should be recognized; teachers per- Figure 7
sistently in the bottom 5 percent deserve Average confidence interval width, New York City Teacher Data
Reports, 2008-2009
immediate scrutiny. Still, it seems a great deal
of effort has been expended to identify a small
fraction of teachers. In the end, a tool designed ELA
for differentiating teacher effectiveness has Citywide (2007-2008)
done very little of the sort.
Bronx (3 years)
To get a better sense of the average level of
Manhattan (3 years)
uncertainty in the Teacher Data Reports, I
examined the full set of value-added estimates Brooklyn (3 years)
reported to more than 12,700 teachers on the Queens (3 years)
NYCDOE 2008-2009 reports. As we saw for
Citywide (3 years)
Mark Jones in Appendix B, each value-added
Staten Island (3 years)
ranking is accompanied by a range of possible
Citywide
estimates. To begin, I simply calculated the (teachers w/3 yrs data)
width of this interval for every teacher in read- 0 10 20 30 40 50 60 70
ing and math. Average widths across teachers Percentage points
are reported in Figure 7.

As expected, the level of uncertainty is higher
M AT H
when only one year of test results are used (the Citywide (2007-2008)
2007-2008 bars) as against three years of data Bronx (3 years)
(all other bars). But in both cases, the average
Manhattan (3 years)
range of value-added estimates is very wide.
For example, for all teachers of math, and Brooklyn (3 years)
using all years of available data, which provides Queens (3 years)
the most precise measures possible, the average
Citywide (3 years)
confidence interval width is about 34 points
(i.e., from the 46th to 80th percentile). When Staten Island (3 years)
Citywide
looking at only one year of math results, the (teachers w/3 yrs data)
average width increases to 61 percentile points. 0 10 20 30 40 50 60 70
That is to say, the average teacher had a range Percentage points
of value-added estimates that might extend

from, for example, the 30th to the 91st per-
centile. The average level of uncertainty is
higher still in ELA. For all teachers and years,
the average confidence interval width is 44
points. With one year of data, this rises to 66
points.

Figure 8 As we saw earlier in this report, schools with
Percent of overlapping confidence intervals, ELA and math high levels of mobility, test exemption, and
absenteeism tend to have fewer students con-
ELA tributing to value-added estimates. And fewer
student observations introduce a greater level
.08
of uncertainty associated with these estimates.
Thus, a predictable pattern exists when com-
.06
paring average levels of uncertainty across sec-
tions of the city. For example, the widest confi-
Fraction
.04
dence intervals are found in the Bronx – whose
schools serve many disadvantaged students – at
.02
37 percentile points in math and 47 points in
ELA (both based on up to three years of data;
0
0 20 40 60 80 100 see Figure 7 on page 23). The most precise
Percentage of overlapping confidence intervals estimates, in contrast, are observed in relatively
more advantaged Staten Island.
Another way of understanding the effects of
M AT H uncertainty is to compare two teachers’ ranges
of value-added estimates and ask whether or
.08
not they overlap. For example, suppose that
based on her value-added, Mrs. Appleton ranks
.06
in the 41st percentile of ELA teachers, with a
confidence interval ranging from 24 to 58 (on
Fraction
.04
the low end of the widths presented in Figure
7). And suppose Mr. Johnson ranks in the 51st
.02
percentile of ELA teachers, with an equally
wide confidence interval from 34 to 68. Based
0
0 20 40 60 80 100 on their “most likely” rankings, Mr. Johnson
Percentage of overlapping confidence intervals appears to have out-performed Mrs. Appleton.
However, because we can’t statistically rule out
Source: Author’s calculations using data from the 2008-2009 Teacher Data Reports, estimates in their overlapping intervals, we
based on up to three years of test results. Bars show the fraction of teachers that overlap
with X percent of other teachers in the district teaching the same grade and subject. can’t say with confidence that this is the case.
Using the 2008-2009 Teacher Data Report
estimates, I compared all possible pairs of
teacher percentile ranges in the city, within the
same grade and subject, to see how many
teachers could be statistically distinguished
from one another.30 For example, if Mrs.
Appleton’s range of 24 to 58 overlaps with 56 simply be ignored; they represent the extent of
percent of all other fourth-grade teachers, we statistical precision with which the value-added
could not rule out the possibility that she was estimate was calculated. Confidence intervals
equally effective as these 56 percent of teach- such as these are reported in any academic
ers. The results are summarized in Figure 8, study that relies on inferential statistics, and
which shows the fraction of teachers who over- any academic study that attempted to ignore
lap with X percent of all other teachers. For these intervals in drawing between-group com-
example, the bar above 60 shows the fraction parisons would in most cases be rejected out-
of teachers who cannot be statistically distin- right.
guished from 60 percent of all other teachers
in the district.
Given the level of uncertainty reported in the
data reports, half of teachers in grades three to
eight who taught math have wide enough per-
formance ranges that they cannot be statisti-
cally distinguished from 60 percent or more of
all other teachers of math in the same grade.
One in four teachers cannot be distinguished
from 72 percent or more of all teachers. These
comparisons are even starker for ELA, as seen
in Figure 8. In this case, three out of four
teachers cannot be statistically distinguished
from 63 percent or more of all other teachers.
Only a tiny proportion of teachers – about 5
percent in math and less than 3 percent in
ELA – received precise enough percentile
ranges to be distinguished from 20 percent or
fewer other teachers.
As noted before, it is true that teachers’ per-
centile ranking is their “best” or “most likely”
estimate. But the ranges reported here cannot
30
In other words, I compared every teacher’s percentile range with
every other teacher’s percentile range. For example, if there are
1,000 teachers in the city, teacher 1’s percentile range is com-
pared to teachers 2 through 1,000; teacher 2’s percentile range is
compared with teachers 1 and 3 through 1,000, and so on, for
all possible pairs.

Is value-added stable from year to year? top quintile of performance last year, 38 per-
cent remain in the top quintile in the following
Given the wide range of uncertainty observed
year. Again, however, there are many inconsis-
in teacher value-added estimates, it would not
tencies. Twenty-three percent of last year’s low-
be surprising if these estimates fluctuated a
est performers are in the top two quintiles in
great deal from year to year. In fact, this is gen-
the following year. Twenty-three percent of last
erally what is observed in both the HISD data
year’s highest performers are in the bottom two
and the New York City Teacher Data Reports.
quintiles in the following year.
Figure 9 shows how Houston teachers’ value-
added in reading correlates from one year to This variability from year to year would be
the next, using fourth- and fifth-grade data expected of any statistical estimate that is esti-
from 2000 to 2006. The format of this figure mated with error, and variability is reduced as
is very similar to that in Figure 5: Each bar additional years of data are added to the analy-
shows the percent of teachers in each perform- sis. But here again, this may be of little com-
ance category in one year (“last year”) that are fort to a teacher who is using her annual esti-
in these same categories in the next year (“this mates to improve her own practice. A top-
year”). performing teacher may be rewarded (or pun-
ished) one year based on her latest round of
As in Figure 5, there is generally a positive cor-
test results, only to get the opposite feedback
relation in teachers’ value-added from one year
the following year. Wisely, districts that have
to the next. For example, among those in the
adopted value-added systems – including the
bottom quintile of performance last year, 36
NYCDOE – caution users of this data against
percent remain in the bottom quintile in the
making rash decisions based on one year
following year. Similarly, among those in the
Figure 9
Year-to-year stability in value-added rankings: HISD reading test, 2000–2006
40
Percentages of this year's quintiles in last year's quintiles
This year’s Q1
35 This year’s Q2
This year’s Q3
30
This year’s Q4
25 This year’s Q5
20
15
10
0
Q1 Q2 Q3 Q4 Q5
Last year’s quintiles
of estimates. But this estimate Figure 10
is one of only a few made Year-to-year stability in ELA and math value-added rankings, New York City Teacher Data
Reports, 2007-2008
available to teachers on their
annual report, and thus they
are hard to ignore. Inexperi- ELA

enced teachers – those
40
arguably most in need of This year’s Q1
35
immediate feedback – simply This year’s Q2
30 This year’s Q3
will not have the multiple
25 This year’s Q4
years of data on which to rely.
20 This year’s Q5
It seems unlikely that teachers
15
and their school leaders will
10
not pay close attention to these
5
noisy and imprecise estimates.
0
Q1 Q2 Q3 Q4 Q5
I repeated the analysis in Fig-
Last year’s quintiles
ure 9 for New York City’s
Teacher Data Reports, using
teachers who had at least three
M AT H
years of value-added estimates,

comparing value-added cate- 40
This year’s Q1
gories in 2007 and 2008 for 35
This year’s Q2
both ELA and math. The 30 This year’s Q3
results are presented in Figure 25 This year’s Q4
10. A pattern emerges that is 20 This year’s Q5
nearly identical to that found 15

for Houston, though the year- 10
to-year correlation appears 5
much weaker, especially in 0
Q1 Q2 Q3 Q4 Q5
ELA. In math, about 23 per- Last year’s quintiles
cent of bottom quintile teach-
ers in 2007 ranked in the top
two quintiles in 2008. Top
performers were much more consistent: 40
percent remained in the top quintile in 2008,
while only 12 percent fell to the bottom two
quintiles. ELA results were considerably more
mixed. Most low performers in 2007 were not
low performers in 2008, with some 31 percent
ranking in the top two categories.

5
Discussion
At least in the abstract, value-added assessment
not count toward teachers’ value-added esti-
mates. In many districts, including New York
City and Houston, these students constitute a
of teacher effectiveness has great potential to substantial share of many teachers’ classrooms.
improve instruction and, ultimately, student Often, state tests are predictable in both con-
achievement. The notion that a statistical tent and format, and value-added rankings will
model might be able to isolate each teacher’s tend to reward those who take the time to
unique contribution to their students’ educa- master the predictability of the test. Recent evi-
tional outcomes – and by extension, their life dence from Houston presented here showed
chances – is a powerful one. With such infor- that one’s perception of a teacher’s value-added
mation in hand, one could not only devise sys- can depend heavily on which test one looks at.
tems that reward teachers with demonstrated Annual value-added estimates are highly vari-
records of success in the classroom – and able from year to year, and, in practice, many
remove teachers who do not – but also create a teachers cannot be statistically distinguished
school climate in which teachers and principals from the majority of their peers. Persistently
work constructively with their test results to exceptional or failing teachers – say, those in
make positive instructional and organizational the top or bottom 5 percent – may be success-
changes. fully identified through value-added scores, but
However, the promise that value-added systems it seems unlikely that school leaders would not
can provide such a precise, meaningful, and already be aware of these teachers’ persistent
comprehensive picture is not supported by the successes or failures.
data. As the discussion in this report showed, Research on value-added remains in its infancy,
value-added assessments – like those reported and it is likely that these methods – and the
in the New York City Teacher Data Reports tests on which they are based – will continue
and used to pay out bonuses in Houston’s to improve over time. The simple fact that
ASPIRE program – are at best a crude indica- teachers and principals are receiving regular
tor of the contribution that teachers make to and timely feedback on their students’ achieve-
their students’ academic outcomes. Moreover, ment is an accomplishment in and of itself,
the set of skills that can be adequately assessed and it is hard to argue that stimulating conver-
in a manner appropriate for value-added assess- sation around improving student achievement
ment represents a small fraction of the goals is not a positive thing. But teachers, policy-
our nation has set for our students and schools. makers, and school leaders should not be
The implementation of value-added systems in seduced by the elegant simplicity of “value-
practice faces many challenges. Not all students added.”
are tested, and many, if not a majority of,
teachers do not teach tested subjects. Students
without a prior year test score – such as chron-
ically mobile students, exempted students, and
those absent on the day of the test – simply do
References Boyd, Donald J., Hamilton Lankford, Susanna
Loeb, Jonah Rockoff, and James Wyckoff.
Alexander, Karl L., Doris R. Entwisle, and 2008. “The Narrowing Gap in New York
Linda S. Olsen. 2001. “Schools, Achieve- City Teacher Qualifications and Its Implica-
ment, and Inequality: A Seasonal Perspec- tions for Student Achievement in High-
tive,” Educational Evaluation and Policy Poverty Schools,” Journal of Policy Analysis
Analysis 23:171–191. and Management 27:793–818.
Amrein-Beardsley, Audrey. 2008. “Method- Braun, Henry I. 2005. Using Student Progress
ological Concerns about the Education to Evaluate Teachers: A Primer on Value-Added
Value-Added Assessment System,” Educa- Models. Policy Information Perspective.
tional Researcher 37:65–75. Princeton, NJ: Educational Testing Service.
ASPIRE. 2010. “2008-2009 ASPIRE Award Brosbe, Ruben. 2010. “My Disappointing
Program Highlights,” <portal.battelleforkids. Data and What to Do With It,” Gotham
org/ASPIRE/Recognize/ASPIRE_Award/ Schools Classroom Tales Blog (March 10),
2009_aspire_highlights.html>. <gothamschools.org/2010/03/10/my-
Associated Press. 2007. “School District Asks disappointing-data-and-what-to-do-with-it/>.
Teachers to Return Pay,” New York Times Buddin, Richard, Daniel F. McCaffrey, Sheila
(March 11). Nataraj Kirby, and Nailing Xia. 2007. “Merit
Blinder, Alan S. 2009. “Education for the Pay for Florida Teachers: Design and Imple-
Third Industrial Revolution.” In Creating a mentation Issues.” RAND Education Work-
New Teaching Profession, edited by Dan ing Paper, WR-508-FEA. Santa Monica, CA:
Goldhaber and Jane Hannaway, pp. 3–14. RAND.
Washington, DC: Urban Institute Press. Center for Educator Compensation Reform.
Blumenthal, Ralph. 2006. “Houston Ties 2008. Performance Pay in Houston. Washing-
Teacher Pay to Student Test Scores,” New ton DC: U.S. Department of Education.
York Times (January 13). Downloadable PDF at <www.cecr.ed.gov/
guides/summaries/HoustonCaseSummary.
Boyd, Donald, Pamela Grossman, Hamilton
pdf>.
Lankford, Susanna Loeb, and James Wyck-
off. 2006. “How Changes in Entry Require- Clark, Damon, Paco Martorell, and Jonah
ments Alter the Teacher Workforce and Rockoff. 2009. “School Principals and School
Affect Student Achievement,” Education Performance,” CALDER Working Paper No.
Finance and Policy 1:176–216. 38. Washington, DC: Urban Institute.
Boyd, Donald J., Pamela L. Grossman, Hamil- Clotfelter, Charles T., Helen F. Ladd, and Jacob
ton Lankford, Susanna Loeb, and James Vigdor. 2005. “Who Teaches Whom? Race
Wyckoff. 2009. “Teacher Preparation and and the Distribution of Novice Teachers,”
Student Achievement,” Educational Evalua- Economics of Education Review 24:377–392.
tion and Policy Analysis 31:416–440.

Corcoran, Sean P., Jennifer L. Jennings, and Hill, Heather C. 2009. “Evaluating Value-
Andrew A. Beveridge. 2010. “Teacher Effec- Added Models: A Validity Argument
tiveness on High- and Low-Stakes Tests.” Approach,” Journal of Policy Analysis and
Paper presented at the Institute for Research Management 28:700–709.
on Poverty summer workshop, Madison, Holcombe, Rebecca, Jennifer L. Jennings, and
WI. Daniel Koretz. 2010. “Predictable Patterns
Darling-Hammond, Linda, and Alfred E. that Facilitate Score Inflation: A Comparison
Wise. 1985. “Beyond Standardization: State of New York and Massachusetts.” Working
Standards and School Improvement,” Ele- Paper. Cambridge, MA: Harvard University.
mentary School Journal 85:315–336. Hoxby, Caroline M., and Andrew Leigh. 2004.
Figlio, David N., and Lawrence S. Getzler. “Pulled Away or Pushed Out? Explaining the
2002. “Accountability, Ability and Disability: Decline of Teacher Aptitude in the United
Gaming the System.” Working Paper No. States,” American Economic Review 94:236–
9307. Cambridge, MA: National Bureau of 240.
Economic Research. Houston Independent School District, Depart-
Goldhaber, Dan. 2008. “Teachers Matter, but ment of Research and Accountability. 2009.
Effective Teacher Policies are Elusive.” In 2005-2006 Teacher Performance Pay and
Handbook of Research in Education Finance 2006-07 ASPIRE Award Program Evaluation.
and Policy, edited by Helen Ladd, and Houston, TX: HISD.
Edward B. Fiske. New York: Routledge. Jackson, C. Kirabo, and Elias Bruegmann.
Gordon, Robert, Thomas J. Kane, and Dou- 2009. “Teaching Students and Teaching
glas O. Staiger. 2006. Identifying Effective Each Other: The Importance of Peer Learn-
Teachers Using Performance on the Job. The ing for Teachers,” American Economic Jour-
Hamilton Project White Paper 2006-01. nal: Applied Economics 1:85–108.
Washington, DC: The Brookings Institution. Jacob, Brian A. 2005. “Accountability, Incen-
Hanushek, Eric A., and Steven G. Rivkin. tives and Behavior: The Impact of High-
2006. “Teacher Quality.” In Handbook of Stakes Testing in the Chicago Public
the Economics of Education, edited by E. A. Schools,” Journal of Public Economics
Hanushek and F. Welch. Amsterdam, The 89:761–796.
Netherlands: Elsevier. Jennings, Jennifer L., and Jonathan M. Bearak.
Harris, Douglas N. 2009. “Teacher Value- 2010. “Do Educators Teach to the Test?”
Added: Don’t End the Search Before it Paper to be presented at the Annual Meeting
Starts,” Journal of Policy Analysis and Man- of the American Sociological Association,
agement 28:693–699. Atlanta.
Jennings, Jennifer L., and Andrew A. Bev-
eridge. 2009. “How Does Test Exemption
Affect Schools’ and Students’ Academic Per-
formance?” Educational Evaluation and Pol-
icy Analysis 31:153–175.
Kane, Thomas J., Jonah E. Rockoff, and Dou- McCaffrey, Daniel F., Tim R. Sass, J. R. Lock-
glas O. Staiger. 2008. “What Does Certifica- wood, and Kata Mihaly. 2009. “The
tion Tell Us About Teacher Effectiveness? Intertemporal Variability of Teacher Effect
Evidence from New York City,” Economics of Estimates,” Education Finance and Policy
Education Review 27:615–631. 4:572–606.
Keller, Bess. 2008. “Drive On to Improve Eval- Medina, Jennifer. 2008a. “Bill Would Bar
uation Systems for Teachers,” Education Linking Class Test Scores to Tenure,” New
Week (January 15). York Times (March 18).
Klein, Joel, and Weingarten, Randi. 2008. Medina, Jennifer. 2008b. “New York Measur-
Joint letter. Principals’ Weekly (October 1). ing Teachers by Test Scores,” New York Times
Koedel, Cory. 2009. “An empirical analysis of (January 21).
teacher spillover effects in secondary school,” Medina, Jennifer. 2009. “A Two-Year Study to
Economics of Education Review 28:682–692. Learn What Makes Teachers Good,” New
Koretz, Daniel. 2008. “A Measured Approach,” York Times City Room Blog (September 1),
American Educator (Fall), 18–27, 39. <http://cityroom.blogs.nytimes.com/2009/
09/01/a-2-year-study-to-learn-what-makes-
Lankford, Hamilton, Susanna Loeb, and James
teachers-good/>.
Wyckoff. 2002. “Teacher Sorting and the
Plight of Urban Schools: A Descriptive Medina, Jennifer. 2010. “Agreement Will Alter
Analysis,” Educational Evaluation and Policy Teacher Evaluations,” New York Times (May
Analysis 24:37–62. 10).
Martinez, Barbara. 2010. “School Tenure Mellon, Ericka. 2010a. “HISD to Pay Out
Crackdown,” Wall Street Journal (July 30). More Than $40 Million in Bonuses,” Hous-
ton Chronicle (January 27).
McCaffrey, Daniel F., Daniel M. Koretz, J. R.
Lockwood, and Laura S. Hamilton. 2004. Mellon, Ericka, 2010b. “HISD Spells Out
Evaluating Value-Added Models for Teacher Teacher Dismissal Process, Part II,” Houston
Accountability. Santa Monica, CA: RAND. Chronicle School Zone Blog (February 8),
<http://blogs.chron.com/schoolzone/2010/
McNeil, Linda, and Angela Valenzuela. 2000.
02/hisd_spells_out_teacher_dismis.html>.
The Harmful Impact of the TAAS System of
Testing in Texas: Beneath the Accountability Murnane, Richard J., and David K. Cohen.
Rhetoric. Educational Resources Information 1986. “Merit Pay and the Evaluation Prob-
Center Report ED 443-872. Washington, lem: Why Most Merit Pay Plans Fail and a
DC: U.S. Department of Education. Few Survive,” Harvard Educational Review
56:1–17.
Nye, Barbara, Spyros Konstantopoulos, and
Larry V. Hedges. 2004. “How Large Are
Teacher Effects?” Educational Evaluation and
Policy Analysis 26:237–257.

Obama, Barack. 2009. “Remarks by the Presi- Rockoff, Jonah E. 2008. “Does Mentoring
dent on Education (July 24).” Press Release. Reduce Turnover and Improve Skills of New
Washington DC: Office of the Press Secre- Employees? Evidence from Teachers in New
tary. Available online at: <www.whitehouse. York City.” Working Paper No. 16240.
gov/the_press_office/remarks-by-the- Cambridge: National Bureau of Economic
president-at-the-department-of-education> Research.
Odden, Allan, and Carolyn Kelley. 2002. Pay- Rockoff, Jonah E., Douglas O. Staiger, Thomas
ing Teachers for What They Know and Do: J. Kane, and Eric S. Taylor. 2010. “Informa-
New and Smarter Compensation Strategies to tion and Employee Evaluation: Evidence
Improve Schools, 2nd ed. Thousand Oaks, from a Randomized Intervention in Public
CA: Corwin Press. Schools.” Working Paper. New York: Colum-
Olson, Lynn. 2007. “Houston Overhauls bia Business School.
Merit Pay System,” Education Week (Septem- Rothstein, Jesse. 2010. “Teacher Quality in
ber 14). Educational Production: Tracking, Decay,
Papay, John P. 2010. “Different Tests, Different and Student Achievement.” Quarterly Jour-
Answers: The Stability of Teacher Value- nal of Economics 125:175–214.
Added Estimates Across Outcome Meas- Rothstein, Richard. 2009. “The Influence of
ures,” American Education Research Journal Scholarship and Experience in Other Fields
published online (April 19); print version on Teacher Compensation Reform.” In Per-
forthcoming. formance Incentives: Their Growing Impact
Rivkin, Steven G. 2007. Value-Added Analysis on American K-12 Education, edited by
and Education Policy. CALDER Policy Brief Matthew G. Springer. Washington, DC: The
No. 1. Washington, DC: National Center Brookings Institution.
for Analysis of Longitudinal Data in Educa- Rothstein, Richard, Rebecca Jacobsen, and
tion Research. Tamara Wilder. 2008. Grading Education:
Rivkin, Steven G., Eric A. Hanushek, and Getting Accountability Right. Washington,
John F. Kain. 2005. “Teachers, Schools, and DC and New York: Economic Policy Insti-
Academic Achievement,” Econometrica tute and Teachers College Press.
73:417–458. Sanders, William L., and Sandra P. Horn.
Rockoff, Jonah E. 2004. “The Impact of Indi- 1994. “The Tennessee Value-Added Assess-
vidual Teachers on Student Achievement: ment System (TVAAS): Mixed-Model
Evidence from Panel Data,” American Eco- Methodology in Educational Assessment,”
nomic Review 94:247–252. Journal of Personnel Evaluation in Education
8:299–311.
Sanders, William L., Arnold M. Saxton, and
Sandra P. Horn. 1997. “The Tennessee
Value-Added Assessment System: A Quanti-
tative-Based Approach to Educational Assess-
ment.” In Grading Teachers, Grading Schools: Toch, Thomas, and Robert Rothman. 2008.
Is Student Achievement a Valid Evaluation Rush to Judgment: Teacher Evaluation in Pub-
Measure? edited by Jason Millman. Thousand lic Education. Washington, DC: Education
Oaks, CA: Corwin Press, Inc. Sector.
Sawchuk, Stephen. 2010. “Houston Approves U.S. Department of Education. 2010.
Use of Test Scores in Teacher Dismissals,” “Overview Information: Race to the Top
Education Week Teacher Beat Blog (February Fund; Notice Inviting Applications for New
12), <http://blogs.edweek.org/edweek/ Awards for Fiscal Year (FY) 2010,” Federal
teacherbeat/2010/02/houston_approves_ Register 75, no. 71 (April 14), Part III: U.S.
use_of_test_s.html>. Department of Education, pp. 19,499–
Sass, Tim R. 2008. The Stability of Value- 19,500. Washington DC: U.S. GPO.
Added Measures of Teacher Quality and Impli- Downloadable PDF at: <www2.ed.gov/
cations for Teacher Compensation Policy. legislation/FedRegister/announcements/
CALDER Policy Brief #4. Washington, DC: 2010-2/041410a.pdf>.
National Center for Analysis of Longitudinal Weisburg, Daniel, Susan Sexton, Susan Mulh-
Data in Education Research. ern, and David Keeling. 2009. The Widget
Shepard, Lorrie A. 1988. “The Harm of Meas- Effect: Our National Failure to Acknowledge
urement-Driven Instruction,” Paper pre- and Act on Differences in Teacher Effectiveness.
sented at the Annual Meeting of the Ameri- Brooklyn, NY: New Teacher Project.
can Educational Research Association,
Washington, DC.
Shepard, Lorrie A., and Katherine Dougherty.
1991. “Effects of High-Stakes Testing on
Instruction.” Paper presented at the annual
meetings of the American Educational
Research Association and the National
Council of Measurement in Education,
Chicago, IL.
Springer, Matthew G., and Marcus A. Winters.
2009. The NYC Teacher Pay-for-Performance
Program: Early Evidence from a Randomized
Trial. Civic Report No. 56. New York: Man-
hattan Institute.

APPENDIX A Race to the Top Definitions of Teacher learning, such as those described in paragraph
(b) of this definition, provided they are rigor-
Effectiveness and Student Achievement ous and comparable across classrooms. (b) For
Effective teacher means a teacher whose stu- non-tested grades and subjects: Alternative
dents achieve acceptable rates (e.g., at least one measures of student learning and performance
grade level in an academic year) of student such as student scores on pre-tests and end-of-
growth (as defined in this notice). States, course tests; student performance on English
LEAs, or schools must include multiple meas- language proficiency assessments; and other
ures, provided that teacher effectiveness is eval- measures of student achievement that are rigor-
uated, in significant part, by student growth ous and comparable across classrooms.
(as defined in this notice). Supplemental meas- Student growth means the change in student
ures may include, for example, multiple obser- achievement (as defined in this notice) for an
vation-based assessments of teacher perform- individual student between two or more points
ance. in time. A State may also include other meas-
Highly effective teacher means a teacher ures that are rigorous and comparable across
whose students achieve high rates (e.g., one classrooms.
and one-half grade levels in an academic year)
of student growth (as defined in this notice).
States, LEAs, or schools must include multiple
measures, provided that teacher effectiveness is
evaluated, in significant part, by student
growth (as defined in this notice). Supplemen-
tal measures may include, for example, multi-
ple observation-based assessments of teacher
performance or evidence of leadership roles
(which may include mentoring or leading pro-
fessional learning communities) that increase
the effectiveness of other teachers in the school
or LEA.
Student achievement means (a) For tested
grades and subjects: (1) A student’s score on
the State’s assessments under the ESEA; and, as
appropriate, (2) other measures of student
Source: U.S. Department of Education, “Overview Information:

Race to the Top Fund; Notice Inviting Applications for New
Awards for Fiscal Year (FY) 2010,” Federal Register 75, no. 71
(April 14, 2010), Part III: U.S. Department of Education, pp.
19,499–19,500, Washington DC: U.S. GPO. Downloadable
PDF at: <www2.ed.gov/legislation/FedRegister/announcements/
2010-2/041410a.pdf>.
Sample New York City Teacher Data Report, 2010 APPENDIX B
Source: New York City Department of Education

APPENDIX C Sample New York City Teacher Data Report, 2009
Source: New York City Department of Education
Providence
Brown University
Box 1985
Providence, RI 02912
T 401.863.7990
F 401.863.1290
New York
233 Broadway, Suite 720
New York, NY 10279
T 212.328.9290
F 212.964.1057
www.annenberginstitute.org

The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice

Uploaded by

Copyright:

Available Formats

The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice

Uploaded by

Copyright:

Available Formats

EDUCATION POLICY FOR ACTION SERIES

Can Teachers be Evaluated

Can Teachers be Evaluated

Annenberg Institute for School Reform at Brown University

4 Challenges to the Practical Implementation of Value-Added ................................. 14

Figure 2 Factors affecting year-to-year test score gains in two classrooms:

Figure 3 New York City Teacher Data Initiative timeline ................................................................................... 6

Figure 4 New York City value-added model: predictors ................................................................................... 10

Figure 8 Percent of overlapping confidence intervals, ELA and math ........................................................... 24

Annenberg Institute for School Reform v

ating Student Progress, Increasing Results &

Annenberg Institute for School Reform 3

ment are the same as those influencing her

tive or especially helpful student in the class, School

Annenberg Institute for School Reform 5

State and teachers

Annenberg Institute for School Reform 7

Annenberg Institute for School Reform 9

Annenberg Institute for School Reform 11

Annenberg Institute for School Reform 13

Annenberg Institute for School Reform 15

Source: Corcoran, Jennings & Beveridge (2010)

Annenberg Institute for School Reform 17

Annenberg Institute for School Reform 19

Annenberg Institute for School Reform 21

are reported in Figure 7.

of value-added estimates that might extend

Annenberg Institute for School Reform 23

Annenberg Institute for School Reform 25

Percentages of this year's quintiles in last year's quintiles

years of value-added estimates,

nearly identical to that found 15

Annenberg Institute for School Reform 27

Annenberg Institute for School Reform 29

Annenberg Institute for School Reform 31

Annenberg Institute for School Reform 33

Source: U.S. Department of Education, “Overview Information:

Source: New York City Department of Education

Annenberg Institute for School Reform 35

Source: New York City Department of Education

You might also like