Abstract Title Page

Not included in page count.

Title: Effects of Teacher Professional Development on Gains in Student Achievement:

How Meta Analysis Provides Scientific Evidence Useful to Education Leaders

Author(s): Rolf K. Blank and Nina de las Alas

2010 SREE Conference Abstract Template

 Abstract Body
Limit 5 pages single spaced.

Background/context:
Description of prior research, its intellectual context and its policy context.

In the present education policy environment a high priority has been placed on improving
teacher quality and teaching effectiveness in U.S. schools (Darling-Hammond et al., 2009;
Obama, 2009). Standards-based educational improvement requires teachers to have deep
knowledge of their subject and the pedagogy that is most effective for teaching the subject.
States and school districts are charged with establishing and leading professional development
programs, some with federal funding support, which will address major needs for improved
preparation of teachers. The whole issue of teacher quality, including teacher preparation and
ongoing professional development, and improving teacher effectiveness in classrooms, is at the
heart of efforts to improve the quality and performance of our public schools.
More recently, several major research synthesis projects have broadly analyzed evidence
on the effects of mathematics and science teacher preparation and development initiatives on
student achievement. One approach to reviewing evidence across studies is to apply a logic
model and to examine the relationship of teacher preparation on student achievement through
effects on intervening variables such as teacher knowledge and instructional practices (Clewell et
al., 2004; Ingvarson, Meiers & Beavis, 2005). This kind of full analytic model allows educators
and leaders to identify key decisions about the organization, delivery and support of teacher
development that are ingredients to positive outcomes.
State and local education agencies are responsible for directing and managing the use of
federal funds for teacher development and improvement as well as guiding programs supported
by states. Additionally, states are now required under NCLB to report on the qualifications of
teachers in core academic subjects and the proportion of teachers that receive high quality
professional development each year. Finally, states provide leadership for local systems on how
to design, select, and implement professional development for teachers. Strong, research-based
program designs, and evidence on their effects, are now in high demand across the U.S.
States and in turn local districts seek models for designing and implementing effective
professional development and particularly models supported by research evidence.
The intended audiences for the study’s findings are education leaders, decision-makers
and researchers. The study design builds on prior research and reporting on professional
development programs and evaluation findings (Blank, de las Alas & Smith, 2007, 2008). The
study was designed to measure and summarize consistent, systematic findings across multiple
studies that show significant effects of teacher professional development on student achievement
gains in K-12 mathematics or science.

Purpose / objective / research question / focus of study:

Description of what the research focused on and why.

The meta analysis study focused on identifying and analyzing research studies that
measured effects of teacher professional development with a content focus on math or science.
The meta analysis was carried out to address two primary questions:

2010 SREE Conference Abstract Template 1

 1) What are the effects of content-focused professional development for math and
science teachers on improving student achievement as demonstrated across a
range of studies?
2) What characteristics of professional development programs (e.g., content focus,
duration, coherence, active learning, and collective participation of teachers)
explain the degree of effectiveness, and are the findings consistent with prior
research on effective professional development?
Setting:
Description of where the research took place.

The study took place in the United States over a period of two years from 2006 to 2008,
with analysis extended to the first part of 2009.

Population / Participants / Subjects:

Description of participants in the study: who (or what) how many, key features (or characteristics).

Across all the studies reviewed, the focus was on teachers in public elementary and
secondary schools teaching math or science at one or more grades K-12 and teachers who
participated in a professional development program aimed to improve their teaching in math or
science.

Intervention / Program / Practice:

Description of the intervention, program or practice, including details of administration and duration.

The meta analysis identified 16 studies of programs that had significant effect sizes and
provided teachers with professional development in mathematics or science. The information
available on program interventions indicated they included combinations of learning activities
such as summer institutes, coursework, study group, classroom mentoring, and professional
networking. Eight of the programs also offered teachers opportunities to put into practice newly-
learned lessons from the professional development by leading classroom instruction, and seven
of the programs bring teachers to observe a classroom with either an exemplary teacher modeling
instruction or a peer teacher implementing lessons learned during the professional development.
More details about program characteristics are available in Table 2. <Insert Table 2 here.>

Research Design:
Description of research design (e.g., qualitative case study, quasi-experimental design, secondary analysis, analytic
essay, randomized field trial).

Meta analysis

Data Collection and Analysis:

Description of the methods for collecting and analyzing data.

The design for the meta analysis built on prior studies in education (Borman et al., 2002;
Yoon et al., 2007; Lipsey & Wilson, 2001) and applied it to findings about professional
development across states and districts.
The design had four steps:

2010 SREE Conference Abstract Template 2

 1) identification and collection of potential studies,
2) determination of study eligibility and coding process,
3) data analysis,
4) reporting and dissemination.

The design for the meta-analysis was also informed by a review of findings on teacher
development programs conducted by the American Institute for Research (Yoon, et al., 2007).
Figures 1 and 2 illustrates the process in more detail. <Insert Figures 1, 2>. In particular, the
meta-analysis study design centered on two areas: capturing the characteristics of the
professional development programs discussed in the studies and documenting the resulting
measurable student outcomes the studies attribute to the professional development programs.
The search process for potential studies included published and unpublished works as
well as evaluation reports from funded state and federal professional development projects. The
study authors conducted an intensive electronic search, using multiple and well-known databases
and meta-databases. In addition, searches were conducted targeting certain periodicals in which
evaluation studies of professional development programs would be featured. Publications and
databases of major education research centers were also examined. Moreover, the study authors
contacted principal investigators listed by program grants from the U.S. Department of
Education Title II-B project evaluations and the research studies funded by the Institute of
Education Sciences, the NSF Teacher Preparation Continuum and MSP project evaluations, and
studies of the Local Systemic Initiatives. Lastly, cross-checks were carried out with findings
from prior reviews and synthesis studies in teacher professional development. Four hundred
sixteen studies or reports were identified for pre-screening. A review of the corresponding
abstracts of those studies reduced the count to 74 studies. These remaining studies were
screened by a team of trained coders who utilized a coding form based on a coding and
reconciliation software program and form developed by AIR (Yoon, et al., 2007). Figure xx
outlines the document review process and the resulting studies included in the meta analysis.
<Insert Figure xx>. Meta regression analyses were conducted with the remaining sixteen
studies, with the focus on studies that featured professional development in mathematics, since
these studies produces the greater number of effect sizes than professional development in
science.

Findings / Results:
Description of main findings with specific details.

The meta analysis of studies of teacher professional development programs in

mathematics and science found that 16 studies reported significant effect sizes for teacher
development in relation to improving student achievement. The evidence for the findings in the
16 studies was based on scientific research designs. These studies reported effect sizes for
student achievement gains for a treatment group as compared to a control group and the studies
provided adequate data and documentation for the research team to compute or re-analyze effect
sizes. The large majority (12 of 16) studies were focused on analyzing mathematics teacher
professional development and effects on student achievement in mathematics. The mean effect
size for mathematics studies using a pre-post design is 0.21. These results show consistent
positive effect on gains in student achievement in mathematics from teacher professional
development in mathematics education. Table 1 details mean effect size findings for math

2010 SREE Conference Abstract Template 3

studies by research design and measure type. <Insert Table 1>. Effect sizes were larger when
measures of achievement were used that were specifically selected or developed to be aligned
with the content focus of the professional development. However, the review of research did
identify several studies with significant effects using large-scale statewide assessment programs.
Several common patterns were found across the sixteen studies on professional
development program designs. The program designs included strong emphasis on teachers
learning specific subject content as well as pedagogical content for how to teach the content to
students. The implementation of professional development included multiple activities to
provide follow-up reinforcement of learning, assistance with implementation, and support for
teachers from mentors and colleagues in their schools. In terms of duration of development
activities, 14 of 16 programs that were reported continued for six months or more. The mean
contact time with teachers in program activities was 91 hours.
The numbers of teachers that were involved in the programs that were analyzed and
found to be effective varied from less than ten to more than 90. The research and evaluation for
the 16 studies employed multiple measures of student achievement and outcomes. The studies’
analysis of effects on student achievement included scales to measure learning in specific content
areas (e.g., algebra, measurement). The use of multiple measures allowed use of different types
of test items. A majority of the studies analyzed professional development for elementary and
middle grades teachers. The analysis of effects showed a pattern of stronger effects for
elementary level professional development than for middle or high school teachers. Table xx
details the professional development program features found in the studies.
The results for the 16 studies with effect sizes demonstrates to the education research and
policy communities how meta analysis can and should be used in education to provide
comparisons and aggregations of research findings over time and across many different studies.
The process of review and analysis employed involved several thousand citations, initial pre-
screening of 400-plus documents, and intensive coding and review of 74 studies. The methods
of identifying, coding, and quantifying data used in the study can be employed for a variety of
objectives in education research.

Conclusions:
Description of conclusions and recommendations based on findings and overall study.
Based on the results of the meta analysis of findings from teacher professional
development studies, several recommendations can be made about use of meta analysis methods
and their use for researchers, evaluators and education leaders.

• The meta-analysis design and procedures employed in the study proved to be

effective in identifying a set of common findings regarding effects of teacher
professional development on student achievement, and the procedures proved
useful to determine which studies and their results met high standards for
scientific validity and reliability.
• A scientific research design can be efficiently employed to evaluate teacher
professional development, and a design to measure effects of teacher development
on subsequent student achievement should be strongly considered for each funded
program for teacher and teaching improvement.
• The use of research designs involving treatment and control groups should
become a regular practice and built into the plan and organization for professional
development and other initiatives.

2010 SREE Conference Abstract Template 4

 • Measures of implementation of professional development are critical to
evaluation design in order to document and measure activities to reinforce and
extend learning for teachers in their school setting.
• Multiple measures of student achievement should be included in the research
design if possible to provide for different types of assessments of learning and
analysis of subject content learned.
• State and local education leaders should ensure that data systems are structured so
that data on teacher development initiatives can be linked to student achievement
measures, and these data can be effective for evaluation even where individual
identifiers are removed.
• Procedures for meta analysis modeled in this study provide a consistent,
quantified methodology for application and use in other studies, including initial
identification, multiple coding and validation of reviews, comparison of research
design with established criteria, and consistent procedures for effect size analysis
and coding of treatment variables.

This meta analysis review did not include systematic identification or review of
intervening measures of the professional development treatment, such as measures of gains in
teacher knowledge, improvement in practices, or fidelity of implementation of what was learned.
Several of the studies identified did report analysis of differences on these kinds of measures
between teachers in the treatment and control groups. Further analysis across studies would
provide stronger evidence and useful information about the relationship between professional
learning of teachers from a specific initiative and subsequent improved learning by students.

2010 SREE Conference Abstract Template 5

 Appendices
Not included in page count.

Appendix A. References

References marked with an asterisk (*) indicate studies included in the meta analysis.

Ball, D. L., & Bass, H. (2000). Interweaving content and pedagogy in teaching and learning to
teach: Knowing and using mathematics. In J. Boaler (ed.), Multiple perspectives on the
teaching and learning of mathematics. (pp. 83–104). Westport, CT: Ablex.

Banilower, E. R., Boyd, S. E., Pasley, J. D., & Weiss, I. R. (2006, February). Lessons from a
decade of mathematics and science reform: A capstone report for the Local Systemic
Change through Teacher Enhancement Initiative. Retrieved from Horizon Research, Inc.
website: http://www.pdmathsci.net/reports/capstone.pdf

Birman, B. F., & Porter, A. C. (2002). Evaluating the effectiveness of education funding streams.
Peabody Journal of Education, 77(4), 59–85.

Birman, B. F., Le Floch, K. C., Klekotka, A., Ludwig, M., Taylor, J. Walters, J. et al. (2007).
State and local implementation of the No Child Left Behind Act: Volume II — teacher
quality under NCLB: Interim report. Washington DC: U.S. Department of Education,
Office of Planning, Evaluation and Development, Policy and Program Studies Service.

Blank, R. K., de las Alas, N., & Smith, C. (2007, February). Analysis of the quality of
professional development programs for mathematics and science teachers: Findings from
a cross-state study. Retrieved from Council of Chief State School Officers website:
http://www.ccsso.org/content/pdfs/year%202%20new%20final%20NSF%20Impde%20F
all%2006%20%20Report%20-032307.pdf

Blank, R. K., de las Alas, N., & Smith, C. (2008, February). Does teacher professional
development have effects on teaching and learning? Evaluation findings from programs
in 14 states. Retrieved from Council of Chief State School Officers website:
http://www.ccsso.org/content/pdfs/cross-state_study_rpt_final.pdf

Borko, H. (2004, November). Professional development and teacher learning: Mapping the
terrain. Educational Researcher, 33(8), 3-15.

Borman, G. D., Hewes, G. M., Overman, L. T., & Brown, S. (2002, November). Comprehensive
school reform and student achievement. A meta analysis. (Report. No. 59). Baltimore,
MD: Center for Research on the Education on Students Placed At Risk, Johns Hopkins
University.

Carey, K. (2004, Winter). The real value of teachers: Using new information about teacher
effectiveness to close the achievement gap. Thinking K-16, 8(1), 3–41.

2010 SREE Conference Abstract Template A-1

*Carpenter, T., Fennema, E., Peterson, P., Chiang, C., & Loef, M. (1989). Using knowledge of
children's mathematics thinking in classroom teaching: An experimental study. American
Educational Research Journal, 26(4), 499–531.

Chambers, J. G., Lam, I., & Mahitivanichcha, K. (2008, September). Examining context and
challenges in measuring investment in professional development: a case study of six
school districts in the southwest region. (Issues & Answers Report, REL2008-No. 037).
Retrieved from U.S. Department of Education, Institute of Education Sciences website:
http://ies.ed.gov/ncee/edlabs/regions/southwest/pdf/REL_2008037.pdf

Choy, S. P., Chen, X., & Bugarin, R. (2006, January). Teacher professional development in
1999-2000: What teachers, principals, and district staff report. (NCES 2006-305).
Washington, DC: National Center for Education Statistics.

Clewell, B. C., Cosentino de Cohen, C., Campbell, P. B., Perlman, L., Deterding, N., Manes, S.,
et al. (2004, December). Review of evaluation studies of mathematics and science
curricula and professional development models. Report submitted to the GE Foundation.
Unpublished manuscript.

Coalition for Evidence-Based Policy. (2003). Identifying and implementing educational

practices supported by rigorous evidence: A user friendly guide. Washington, DC: U.S.
Dept. of Education, Institute of Education Sciences, National Center for Education
Evaluation and Regional Assistance.

Cohen, D. K., & Hill, H. C. (1998). Instructional policy and classroom performance: The
mathematics reform in California (RR-39). Retrieved from Consortium for Policy
Research in Education website: http://www.cpre.org/Publications/rr39.pdf

Corcoran, T. B. (2007). Teaching matters: How state and local policymakers can improve the
quality of teachers and teaching. (CPRE Policy Briefs RB-48). Philadelphia, PA:
Consortium for Policy Research in Education, University of Pennsylvania.

Corcoran, T., & Foley, E. (2003). The promise and challenge of evaluating systemic reform in an
urban district. Research perspectives on school reform: Lessons from the Annenberg
Challenge. Providence, RI: Annenberg Institute at Brown University.

Correnti, R. (2007). An empirical investigation of professional development effects on literacy

instruction using daily logs. Educational Evaluation and Policy Analysis, 29(4), 262–295.

Council of Chief State School Officers. (2006). Improving evaluation of teacher professional
development in math and science, year 1 project report. Washington, DC: Author.

Darling-Hammond, L. (1999). Teacher quality and student achievement: A review of state policy
evidence. Retrieved from Center for the Study of Teaching and Policy website:
http://depts.washington.edu/ctpmail/PDFs/LDH_1999.pdf

2010 SREE Conference Abstract Template A-2

Darling-Hammond, L., Wei, R. C., Andree, A., Richardson, N., & Orphanos, S. (2009).
Professional learning in the learning profession: A status report on teacher development
in the United States and abroad. Washington, DC: National Staff Development Council.

Desimone, L. M., Porter, A. C., Garet, M. S., Yoon, K. S., & Birman, B. F. (2002). Effects of
professional development on teachers’ instruction: Results from a three-year longitudinal
study. Educational Evaluation and Policy Analysis, 24(2), 81–112.

*Dickson, T. K. (2002). Assessing the effect of inquiry-based professional development on

science achievement tests scores. (Doctoral dissertation). Available from ProQuest
Dissertations and Theses database. (UMI No. 3076239)

Frechtling, J (2001). What evaluation tells us about professional development programs in math
and science. In C. R. Nesbit, J. D, Wallace, D. K. Pugalee, A.-C. Miller, & W. J. DiBiase
(Eds.), Developing Teacher Leaders: Professional Development in Science and
Mathematics (pp. 17–42). Columbus, OH: ERIC Clearinghouse for Science Mathematics,
and Environmental Education.

Garet, M. S., Birman, B. F., Porter, A. C., Desimone, L., Herman, R. & Yoon, K. S. (1999).
Designing effective professional development: Lessons from the Eisenhower program
and technical appendices (Report No. ED/OUS99-3). Washington, DC: American
Institutes for Research.

Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes
professional development effective? Results from a national sample of teachers.
American Educational Research Journal, 38(4), 915–945.

!"#$%&'()'!))'*+%+",-$&'*)'.)&'/'(0-12"++$34-5$+"&'6)'7899:&'(;<<+"=)'.+#"$>$2'%-'%+#?0'
<#%0+<#%>?,'>$'%0+'?-$%+@%'-A',B,%+<>?'"+A-"<)'!"#$%&'()*+,&'-%.('/)0#1#'$&2)
3.,$('/4)557C=&'DE93F8)

Guskey, T. R. (2003, June). What makes professional development effective? Phi Delta Kappan,
84(10), 748–750.

Harris, D. N., & Sass, T. R. (2007, March). Teacher training, teacher quality and student
achievement. (Working Paper 3). Retrieved from National Center for Analysis of
Longitudinal Data in Education Research website:
http://www.caldercenter.org/PDF/1001059_Teacher_Training.pdf

Hedges, L. V. (2007, December 1). Effect sizes in cluster-randomized designs. Journal of

Educational and Behavioral Statistics, 32(4), 341–370.

*Heller, J. I., Curtis, D. A., Rabe-Hesketh, S., Clarke, C., & Verbencoeur, C. J. (2007, August
29). The effects of "Math Pathways and Pitfalls" on students' mathematics achievement:
National Science Foundation final report. Retrieved from ERIC database. (ED498258)

2010 SREE Conference Abstract Template A-3

Hezel Associates, LLC. (2007). PBS TeacherLine national survey of teacher professional
development 2005-2006. Syracuse, NY: Author.

Hiebert, J. (1999, January). Relationships between research and the NCTM standards. Journal
for Research in Mathematics Education, 30(1), 3–19.

Hill, H. C., Schilling, S. G., & Ball, D. L. (2004, September). Developing measures of teachers’
mathematics knowledge for teaching. Elementary School Journal, 105(1), 11.

Ingvarson, L., Meiers, M. & Beavis, A. (2005, January 29). Factors affecting the impact of
professional development programs on teachers’ knowledge, practice, student outcomes
& efficacy. Education Policy Analysis Archives, 13(10). Retrieved from
http://epaa.asu.edu/epaa/

*Jagielski, D. A. (1991). An analysis of student achievement in mathematics as a result of direct

and indirect staff development efforts focused on the problem-solving standard of the
National Council of Teachers of Mathematics. (Doctoral dissertation). Available from
ProQuest Dissertations and Theses database. (UMI No. 9119821)

Kennedy, M. (1998). Form and substance in inservice teacher education. (Research Monograph
No. 13). Madison, WI: University of Wisconsin, Staff National Institute for Science
Education.

Lipsey, M. W., & Wilson, D. B. (2001). Practical meta analysis. Applied Social Research
Methods Series (Vol. 49). Thousand Oaks, CA: Sage.

Loucks-Horsley, S., Hewson, P., Love, N., & Stiles, K. E. (1998). Designing professional
development for teachers of science and mathematics. Thousand Oaks, CA: Corwin
Press.

McMillan, J. H., & Schumacher, S. (1997). Research in education. A conceptual introduction.

(4th ed.). New York: Addison-Wesley Educational Publishers, Inc.

*META Associates. (2006, March). Northeast Front Range math/science partnership (MSP) to
increase teacher competence in content. Year 2 evaluation report: January 1, 2005–
December 31, 2005. Golden, CO: Author.

*META Associates. (2007, March). Northeast Front Range math/science partnership (MSP) to
increase teacher competence in content. Final evaluation report: January 1, 2004–
December 31, 2006. Golden, CO: Author.

*Meyer, S. J., & Sutton, J. T. (2006, October). Linking teacher characteristics to student
mathematics outcomes: Preliminary evidence of impact on teachers and students after
participation in the first year of the Math in the Middle Institute Partnership. Paper
presented at the MSP Evaluation Summit II, Minneapolis, MN.

2010 SREE Conference Abstract Template A-4

Miles, K. H., Odden, A., Fermanich, M., Archibald, S. (2004). Inside the black box of school
district spending on professional development: lessons from five Urban Districts. Journal
of Education Finance, 30(1), 1–26.

National Center on Education Statistics. (n.d.). Statewide longitudinal data systems grant
program: Grantee state [Website]. Retrieved from
http://nces.ed.gov/Programs/SLDS/stateinfo.asp

National Commission on Teaching & America’s Future (1996). What matters most: Teaching for
America’s future. New York: Author.

*Niess, M. L. (2005). Oregon ESEA Title IIB MSP: Central Oregon consortium. Report to the
U.S. Department of Education, Mathematics and Science Partnerships. Corvallis, OR:
Department of Science & Mathematics Education, Oregon State University.

Noyce, P. (2006, September 13). Professional development: How de we know if it works?

Education Week, 26(3), 36–37, 44.

Obama, B. (2009, March 10). Taking on education [Web log message]. Retrieved from
http://www.whitehouse.gov/blog/09/03/10/Taking-on-Education/

O’Reilly, F. E., & Weiss, C. H. (2006, April). Opening the black box: Using theory-based
evaluation to understand professional development for k-12 teachers of math and
science. Paper presented at the Annual Meeting of the American Educational Research
Association, San Francisco, CA.

*Palmer, E. A., & Nelson, R. W. (2006, September). Researchers in every classroom. Evaluation
report, 2005-06. Barnes, WI: ASPEN Associates.

*Rubin, R. L., & Norman, J. T. (1992). Systematic modeling versus the learning cycle:
Comparative effects of integrated science process skill achievement. Journal of Research
in Science Teaching, 29, 715–727.

*Saxe, G. B., Gearhart, M, & Nasir, N. S. (2001). Enhancing students’ understanding of

mathematics: A study of three contrasting approaches to professional support. Journal of
Mathematics Teacher Education, 4, 55–79.

Scher, L. S., & O’Reilly, F. E. (2007, March). Understanding professional development for k-12
teachers of math and science: A meta-analysis. Paper presented at the Annual Meeting of
the American Educational Research Association, Chicago, IL.

*Scott, L. M. (2005). The effects of science teacher professional development on achievement of

third-grade students in an urban school district (Doctoral dissertation). Available from
ProQuest Dissertations and Theses database. (UMI No. 3171980)

2010 SREE Conference Abstract Template A-5

Shavelson, R. J., & Towne, L. (2002). Scientific research in education. Retrieved from National
Academies of Sciences website: http://www.nap.edu/books/0309082919/html/index.html

Showers, B., Joyce, B. & Bennett, B. (1987). Synthesis of research on staff development: A
framework for future study and state-of-the-art analysis. Education Leadership, 45(3),
77-87.

*Siegle, D., & McCoach, D. (2007). Increasing student mathematics self-efficacy through
teacher training. The Journal of Secondary Gifted Education, 18(2), 278–331.

*Snippe, J. (1992, July). Effects of instructional supervision on pupils' achievement. Paper

presented at the annual meeting of the American Educational Research Association, San
Francisco, CA.

Supovitz, J. A. (2003). Evidence of the influence of the National Science Education Standards on
the professional development system. In K. S. Hollweg & D. Hill (Eds.), What is the
influence of the National Science Standards? (pp. 64–75). Washington, DC: National
Academy Press.

*Walsh-Cavazos, S. (1994). A study of the effects of a mathematics staff development module on

teachers' and students' achievement (Doctoral dissertation). Available from ProQuest
Dissertations and Theses database. (UMI No. 9517241)

Wayne, A. J., Yoon, K. S., Zhu, P., Cronen, S., & Garet, M. S. (2008, November).
Experimenting with teacher professional development: Motives and methods.
Educational Researcher, 37(8), 469–479.

Weiss, I. R., Banilower, E. R., McMahon, K. C., & Smith, P. S. (2001). Report of the 2000
national survey of science and mathematics education. Retrieved from Horizon Research,
Inc. website: http://2000survey.horizon-research.com/reports/status/complete.pdf

Wilson, S. M., & Berne, J. (1999). Teacher learning and the acquisition of professional
knowledge: An examination of research on contemporary professional development.
Review of Research in Education, 24, 173–209.

Yoon, K. S., Duncan, T., Lee, S., W.-Y., Scarloss, B., & Shapley, K. (2007). Reviewing the
evidence on how teacher professional development affects student achievement. (Issues &
Answers Report, REL 2007-No. 033). Retrieved from Institute of Educational Sciences
website: http://ies.ed.gov/ncee/edlabs/regions/southwest/pdf/REL_2007033.pdf

2010 SREE Conference Abstract Template A-6

Appendix B. Tables and Figures
Not included in page count.
Figure 1: Logic Model

High Quality PD
Teacher Instructional
• Content-focused
Knowledge & Practices
• Active Learning
Skills
• Coherence
• Duration/Frequency
• Collaborative
Participation
Effects on Students
• Measures of Achievement
• Cohorts over Time
• Student Unit Records
• Linked to Teachers
Figure 2 Steps in Study Design
 Identification and collection of potential studies
• Prior and ongoing literature reviews and meta-analyses
• Electronic search through databases, meta-databases & use of search terms
• Journal search
• Research centers
• NSF & US ED recently completed studies

Pre-screening

Study Eligibility, Screening & Coding Process

• Study Criteria
• Creation of Coding Form & Process (See Figure 3 for
more details)

Data Analysis
• Test of Homogeneity
• Descriptive Statistics
• Correlations between Professional Development
Programs and Student Outcome Effect Sizes

Reporting & Dissemination

• Formal report to NSF
• Presentations at national
conferences
• Journal article submissions
 †
Table 1: Identified Studies and Key Characteristics & Effect Sizes
Treatment
Teachers N Median
Study Grade/ School Level; Size (All Effect Number of
Study Design Content Area Teachers) Size Effects Student Outcome Measure
Carpenter, et RCT Elementary; Math 20 (40) .39 7 ITBS (Level 7)
al., 1989
.68 Oral test
.32 Study-specific tests (Scale 1,2,3)
Dickson, QED Gr. 8-10; Science 4 (8) .1 2 Texas Assessment of Academic Skills
2002
.43 EOC Biology Test
Heller et al., RCT Gr. 2, 4, 6; Math 48 .69 6 Math Pathways and Pitfalls pre-posttest
2007 gain
Jagielski, QED Gr. 3-8; Math 43 (70) .77 20 MCIP/89 using released NAEP pre-
1991 posttest gain
Lane, 2003 QED Elementary; Math 12 (22) .13 2 Constructed CSAP pre-posttest gain
META QED Gr. 6-8; Math 19 (34) .13 6 Colorado Student Assessment Program
Assoc., 2006 pre-posttest gain
META QED Gr. 6-8; Math 17 (40) -.19 2 Colorado Student Assessment Program
Assoc., 2007 pre-posttest gain
Meyer & QED Gr. 6-8; Math 31 (155) -.02 10 Metropolitan Achievement Test posttest
Sutton, 2006
.10 Criterion Referenced Test posttest
Niess, 2005 RCT Gr. 3-8; Math 24 (42) .11 4 Technology Enhanced State Assessment
pre-posttest gain

†
For Cohen's d an ES > 0.0 ! 0.3 is a "small" effect, >0.3 and <0.8 a "medium" effect and "0.8 a "large" effect.
Table 1 – continued
Treatment
Teachers N Median
Study Grade/ School Level; Size (All Effect Number of
Study Design Content Area Teachers) Size Effects Student Outcome Measure
Palmer & QED Gr. 5-10; Science 16 (43) .11 5 Northwest Evaluation Association
Nelson, assessments pre-posttest gain
2006*
Rubin & RCT Middle; Science 7 (16) .64 8 Middle Grades Integrated Process Skill
Norman, Test pre-posttest
1992
.12 Group Assessment of Logical Thinking
Test pre-posttest
Saxe, QED Elementary; Math 17 (6) xx Study-specific assessments
Gearhart, & (Computational Scale)
Nasir, 2001
1.63 6 Study-specific assessments (Conceptual
Scale) posttest
Scott, 2005 QED Gr. 3; Science 3 (6) .20 2 Iowa Test of Basic Skills pre-posttest gain
Siegle & RCT Gr. 5; Math 7 (15) .20 2 Math Achievement Test
McCoach,
2007
Snippe, 1992 RCT High; Math 87 (198) -.01 21 Terra Nova
.20 ACCUPLACER
.06 WorkKeys
Walsh- QED Gr. 5; Math 4 (6) .26 2 PSG Achievement Assessment pre-
Cavazos, posttest gain
1994
Table 2: Professional Development Features

Study Contact Duration PD

Authors, Year Professional Development Location Hrs. (in mo.) Components
Carpenter, et al., 1989 Cognitively Guided 24 schools in Madison, 80 4.5 Coursework
Instruction WI metropolitan area Mentoring, Network
Study group
Summer institute
Dickson, 2002 Inquiry Institute Science Suburban school 24 8 Internship
district north central Networking
Texas
Heller et al., 2007 Mathematics Pathways and Five diverse districts 10 8 Internship
Pitfalls across the U.S. Leading instruction
Summer institute
Jagielski, 1991 Mathematics Curriculum Chicago, IL 36 8 Conference
Improvement Project Leading instruction
Networking
Study group
Lane, 2003 Problem-solving and Five schools from one 17 8 Classroom observation
reasoning Math district in Colorado Developing assessment
Study group
META Assoc., 2006; Northeast Front Range Five school districts in 120 7.5 Coaching
2007 Math/Science Partnership Colorado front range Devel. assessment
Leading instruction
Mentoring, Network
Summer institute
Meyer & Sutton, 2006 Math in the Middle Institute Lincoln, NE 540 16 Coursework
Partnership Summer institute
Table 2 - continued

Study Professional PD Provider Contact PD

Authors, Year Development Teachers Location Agency Hrs. Duration Components
Niess, 2005 High Desert MSP Five school districts in Oregon State 304 8 Classroom observation
Math teaching central Oregon University Leading instruction
Networking
Summer institute
Palmer & REC Lesson Study Ten school districts in University, 60 8 mos. Classroom observation
Nelson, 2006 Science Minnesota Global Developing assessment
Resources Networking
Leading instruction
Study group
Summer institute
Rubin & Systematic Detroit, MI Wayne State 30 3 mos. Courses
Norman, 1992 Modeling Strategy University Mentoring
Science Teaching
Saxe, Integrating Los Angeles Researchers/ 41 8 mos. Classroom observation
Gearhart, & Mathematics metropolitan area Authors Developing assessment
Nasir, 2001 Assessment Internship, Study
groupLeading instruction
Mentoring, Network Summer
institute
Table 2 - continued

Study Professional PD Provider Contact PD

Authors, Year Development Teachers Location Agency Hrs. Duration Components
Scott, 2005 TEAMS Professional Suburban-Urban School District 168 8 mos. Classroom observation
Development Model district Texas Coaching
metropolitan area Conference
Leading discussion
Mentoring
Networking
Study group
Summer institute
Siegle & Self-Efficacy Ten districts varying University of 2 1 day Coaching
McCoach, 2007 Teaching Strategies urban, suburban, Connecticut Leading instruction
& Implementation rural in six states Networking
Math (MA, MD, MI, MT,
NC, NE)
Snippe, 1992 National Research Teachers from University of 14 3 days Classroom mentoring
Center for Career/ several states; Minnesota Networking
Tech Education providers traveled to Study group
each location
Walsh- Probability, Statistics, South Texas school Researcher/ 12 3 days In-service activity
Cavazos, 1994 and Graphing Module district Author
Mean 91 hrs. 6 mos. 3.3 activities
Range 2 - 540 hrs. 1 day–16 1 - 6 activities
months
Table 3: Mean Effect Sizes for Teacher Professional Development Effects On Student Achievement, Mathematics Studies [include only part a math]

Pre-Post Effect Size Post-Only

Categories (SE) N Effects Effect Size (SE) N Effects
Math Studies 0.21 (0.08) 21 0.13 (0.03) 68
Research Design
RCT 0.27 (0.13) 5 0.26 (0.05) 35
QED 0.17 (0.08) 16 0.04 (0.04) 33
Measure Type
PD Specific 0.32 (0.08) 15 0.28 (0.09) 25
State Criterion- Referenced 0.01 (0.08) 6 -0.07 (0.14) 7
National Norm-Referenced -- -- 0.17 (0.04) 25
Local Test -- -- 0.05 (0.02) 11
N Effects = number of effect sizes per category (across studies identified with at least one significant effect size); details on statistical tests in Appendix.

[
Table 4: Correlation Table of Math Post-Only Professional Development Design Elements

1 2 3 4 5 6 7 8 9 10 11 12 13
Time
1. Contact Hr. 1
2. Frequency .741** 1
3. Duration .834** .623** 1
PD Activities
4. Summer
.577** .399** .655** 1
Institutes
5. College
.744** -.171 .596** .618** 1
Courses
6.Conferences -.196 .094 .146 -.403** -.249* 1
7. Study Group -.694** -.253 -.602** -.524** -.369** .287* 1
Active Learning
8. Lead Discussion -.196 .094 .146 -.403** -.249* 1.000** .287* 1
9. Learning Network -.657** .048 -.601** -.351** -.471** .249* .796** .249* 1
10.Develop Assessments -.138 .398** .135 .345** -.249* -.172 .021 -.172 .155 1
11. Observe Teachers -.154 .562* .084 .418** -.360** -.249* -.298* -.249* -.093 .692** 1
12. Classroom Mentoring -.421** -.571** -.742** -.394** -.028 -.347** .579** -.347** .502** -.347** -.502** 1
Coherence
13. Link to curriculum, .043 -.161 .106 -.406** -.244* .221 .163 .221 -.158 -.080 -.324** -.059 1
Goals
Two-tail test: * significant at p<.05; ** significant at p<.01

2010 SREE Conference Abstract Template B-1

Appendix: Detailed significance tests for Table 3

Pre-Post Post Only

Categories 95 % CI Q statistic 95 % CI Q statistic
Math Studies (0.06, 0.36) QT = 153.72* (0.07, 0.20) QT = 328.78*
Research Design QB(1) = 46.12* QB(1) = 66.72*
RCT (0.01, 0.53) QW = 53.24* (0.16, 0.35) QW = 78.37*
QED (0.01, 0.34) QW = 54.35* (-0.04, 0.11) QW = 183.70*
Measure Type QB(1) = 84.46 QB(3) = 90.43*
PD Specific (0.16, 0.49) QW = 46.81 (0.10, 0.46) QW = 91.73*
State Criterion- Referenced (-0.15, 0.16) QW = 22.45 (-0.35, 0.21) QW = 111.25*
National Norm-Referenced -- (0.10, 0.24) QW = 16.33
Local Test -- (0.02, 0.09) QW = 19.05*
Notes: *p < .05; if QT is significant a random-effects model is applied. See further If QW is not significant a fixed-effects model is applied. If QW is significant a random-
effect model is used for that category. QB refers to differences between groups.

2010 SREE Conference Abstract Template B-2