ERO 15569 Effective School Evaluation June16 FULL WEB 002
ERO 15569 Effective School Evaluation June16 FULL WEB 002
ERO 15569 Effective School Evaluation June16 FULL WEB 002
G ULA N
RE
P
Learners
HI
DS
AR
EFFECTIVE
STEW
Strategic
MAHI TAHI
Evaluation EMER
SCHOOL
GE
NT
EVALUATION
EV
LU
A
AT
I ON
EV
AL
UA
TI
O
N
IN
How to do and use internal evaluation for improvement
,
Q
UI
RY
IC EVAL
T EG UA
A TI
R O
T N
S
Learners
E M E R G
REG U L AR
EN
T
E
V
E
A
V
L
A
U
U A
A
T
T
IO
IO N
N
Effective School Evaluation
Published 2016
© Crown copyright
ISBN 978-0-478-43842-0
Except for the Education Review Office and Ministry of Education logos, this copyright work is
licensed under Creative Commons Attribution 3.0 New Zealand licence. In essence, you are free to
copy, distribute and adapt the work, as long as you attribute the work to the Education Review Office
and the Ministry of Education and abide by the other licence terms. In your attribution, use the
wording ‘Education Review Office’ and ‘Ministry of Education’, not the Education Review Office and
Ministry of Education logos or the New Zealand Government logo.
Contents
Introduction 4
How this resource is structured 5
Context 6
4 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
How this resource is structured
Engaging in effective internal evaluation identifies three different but related purposes for and
approaches to internal evaluation in schools: strategic, regular and emergent. Examples highlight
particular features of the different approaches. This section will help you to engage in internal
evaluation and understand what is needed to do it well. It highlights the actions you can take
to develop the organisational conditions, capability and collective capacity to do and use internal
evaluation for improvement purposes. Pages 9 to 23
The third section, School improvement for equity and excellence, will help you to explore the
relationship between internal evaluation and school improvement. It describes the kinds of shifts
in practice that some of the case study schools were making as a result of internal evaluation.
This section includes examples of improvement in two of the case study schools showing how
they used evaluation and reasoning processes to inform their actions, with resulting shifts
in practice and positive impacts on student outcomes. Pages 24 to 38
The fourth section, Using the school evaluation indicators to support continuous improvement,
will help you to better understand the relationship between internal and external evaluation and how
aligning the two, with the evaluation indicators as a common framework, will support ongoing,
sustained improvement in achievement and equity. Pages 39 to 40
The final section, Integrating internal and external evaluation for improvement, reiterates the
value to schools of harmonising their internal evaluation processes with ERO’s external evaluations.
It supports schools to use their internal evaluation findings to engage with others in their school
community and beyond. Domain six of the indicator framework, Evaluation, inquiry and knowledge
building for improvement and innovation, highlights how organisational conditions and capability
influence engagement with external evaluation. Pages 41 to 43
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 5
Context
School evaluation can be either internal or external; in New Zealand it is both. ERO has mandated responsibility
for external reviews while National Administration Guideline 2 explicitly requires every school to:
>> develop a strategic plan which documents how it is giving effect to the National Education Guidelines
through its policies, plans and programmes, including those for curriculum, National Standards,
assessment, and staff professional development
>> maintain an ongoing programme of self review in relation to the above policies, plans and programmes,
including evaluation of information on student achievement.2
The intention is that all schools and their communities should be engaged in an ongoing, cyclical process
of internal evaluation and inquiry that is directed at improving educational outcomes for all their students.
Effective internal evaluation at a community/cluster/school level also contributes to improving outcomes
for learners in the wider community and at a system level.
As used in this guide, the term ‘internal evaluation’ includes all the activities and processes that schools and their
communities engage in to evaluate how effective they are in supporting success for all learners. These include
schools’ ongoing programme of self-review, planning and reporting processes, and other inquiry and evaluation
activities that boards, leaders and teachers engage in for the purposes of accountability and improvement.
The New Zealand Curriculum and Te Marautanga o Aotearoa both position teaching within an inquiry
framework and highlight the importance of teachers and leaders having the expertise to inquire into their
practice, evaluate the impact, and build knowledge about what works.3
Schools and their communities need to be continuously evaluating the impact of their endeavours on learner
outcomes. To do this, they need strong leadership and evaluation expertise. Their systems, processes and
resources should support purposeful data gathering, collaborative inquiry and decision making and align
closely with the school’s vision, values, strategic direction, goals, and equity and excellence priorities.
Using a range of quality achievement data, boards of trustees, leaders and teachers in highly effective
schools collaboratively monitor and evaluate the impact of improvement strategies. They are responsive
to findings, making changes as necessary, and successful strategies are embedded into school practice
so that all learners benefit.
Periodic external evaluation supports this cycle of continuous improvement by bringing an external lens
to bear on achievement, the extent to which conditions support improvement, and next steps.
The primary purpose of school evaluation is to achieve equity and excellence in outcomes for all learners.
Effective internal evaluation processes enable trustees, leaders, teachers, parents, families, whänau
and the wider school community to better understand:
>> how individual learners and groups of learners are performing in relation to valued outcomes
>> how improvement actions have impacted on learner outcomes and what difference is being made
>> what needs to be changed and what further action needs to be taken
>> the patterns and trends in outcomes over time
>> what kinds of practices are likely to make the most difference for diverse learners and in what contexts
>> the extent to which the improvements achieved are good enough in terms of the school’s vision,
values, strategic direction, goals, and equity and excellence priorities.
2 Ministry of Education. (2015). The National Administration Guidelines (NAGs). Retrieved from http://www.education.govt.nz/ministry-of-
education/legislation/nags/.
3 Timperley, H., & Parr, J. (2010). Evidence, inquiry and standards. In H. Timperley & J. Parr (Eds.), Weaving evidence, inquiry and standards
to build better schools (pp. 9–23). Wellington: NZCER Press.
6 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
1. What matters most in schooling
The major challenge for the New Zealand education system is achieving equity and excellence
in outcomes for an increasingly diverse student population. Although a significant proportion
of New Zealand students achieve at the highest levels in core areas such as reading, mathematics
and science, our performance in international assessment studies shows persistent achievement
disparities, in particular for Mäori and Pacific students. Achieving excellence must go hand in hand
with achieving equity of education outcomes.
Equity in education can be seen through two dimensions: fairness and inclusion. Equity as
fairness implies that personal or socio-economic circumstances such as gender, ethnic origin
or family background, are not obstacles to success in education. Equity as inclusion means
ensuring that all students reach at least a basic minimum level of skills. Equitable education
systems are fair and inclusive, and support their students in reaching their learning potential
without either formally or informally erecting barriers or lowering expectations.4
ERO has redeveloped its indicators framework (Figure 1) to highlight six domains that are known
to be key influences in improving outcomes for all learners. The aim has been to create a framework
that can be used not only by ERO’s evaluators, but also by schools to focus their internal evaluation
and improvement activities.
ANAUNGATANGA
WH
LE
AD
Responsive
ER
P
HI
curriculum,
SH
DS
effective teaching
IP
AR
and opportunity
STEW
to learn
MAN
Learners
MAHI TAHI
AAKITANGA
Educationally
powerful
connections and
relationships
EV
ON
AL
TI
VA
UA
AK O
NO
TI
O
IN
PR
N
IN O TY
,
AN F E S SIO N A L C A PA BILI
D
Q AN
UI DC
RY OLLEC CITY T
TIVE CAPA N
AN ME
D
KN VE
OW RO
LEDG IMP
E B UI LDIN G FO R
4 Schleicher, A. (2014). Equity, excellence and inclusiveness in education: policy lessons from around the world.
Paris: Organisation for Economic Co-operation and Development. www.oecd.org/publishing
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 7
Learners are at the centre of the indicators framework, with the goal being successful, ‘confident,
connected, actively involved, lifelong learners’ as envisioned by The New Zealand Curriculum
and Te Marautanga o Aotearoa5. This means that every young person should leave school:
>> confident in their identity, language and culture as a citizen of Aotearoa New Zealand
>> socially and emotionally competent, resilient and optimistic about the future6
>> a successful lifelong learner
>> participating and contributing confidently in a range of contexts (cultural, local, national and global)
to shape a sustainable world of the future.7
The six domains included in the framework have all been identified by education research studies
and analyses of studies as significant influences on school effectiveness and improvement. Two of
these domains, educationally powerful connections and relationships and responsive curriculum,
effective teaching and opportunity to learn, have the greatest influence, but all are mutually
interdependent.
Research evidence highlights how critical leadership is for achieving equity and excellence goals.
It falls to leaders to establish the necessary conditions and relational trust, and it is leaders who
ensure that teachers have opportunities to collaboratively inquire into, reflect on and evaluate their
practice, thereby building professional capability and collective capacity.
Entrusted with a stewardship role, boards of trustees represent and serve the school community.
They have a responsibility to scrutinise student achievement and evaluation information and to
maintain a relentless focus on learning, wellbeing, achievement and progress. They must also
evaluate their own performance and ensure that they are meeting their accountabilities.
The remaining domain, evaluation, inquiry and knowledge building, is the engine that drives
improvement and innovation.
All education activities take place within a cultural context. For this reason the indicator framework
singles out four concepts, manaakitanga, whanaungatanga, ako and mahi tahi, which have the
power to transform the learning environment for students. Together, these concepts challenge
teachers to abandon deficit theorising as a way of rationalising failure and to focus instead on
how they can change their approach and practices to realise the potential in their students.8
See School Evaluation Indicators – Effective Practice for Improvement and Learner Success
for an explanation of these concepts.9
A school’s effectiveness closely correlates with the quality of its practices in all six domains
and the extent to which those practices are integrated and coherent.
8 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
2. Engaging in effective internal evaluation
The whole point of internal evaluation is to assess what is and is not working, and for whom, and then
to determine what changes are needed, particularly to advance equity and excellence goals. Internal
evaluation involves asking good questions, gathering fit-for-purpose data and information, and then
making sense of that information. Much more than a technical process, evaluation is deeply
influenced by the school’s values and how it sees its role in the community. Effective internal
evaluation is always driven by the motivation to improve, to do better for the students.
When internal evaluation is done well, processes are coherent and align with schools’ visions and
strategic goals. Leaders and teachers work collaboratively across teams, syndicates, departments,
faculties and in some cases communities of learning, to ensure that the efforts that go into evaluation
lead to improvement. The urgency to improve is shared by all, and can be articulated by all.
Evidence from research and the case study schools shows that there are organisational conditions
that support development of the capacity to do and use evaluation for improvement and innovation.
These include:
>> evaluation leadership
>> a learning-oriented community of professionals that demonstrates agency in using evaluation
for improvement in practice and outcomes
>> opportunity to develop technical evaluation expertise (including access to external expertise)
>> access to, and use of, appropriate tools and methods
>> systems, processes and resources that support purposeful data gathering, knowledge building
and decision making.
See the Domain 6 indicators for practices that are characteristic of effective evaluation, inquiry and
knowledge building for improvement and innovation and Section 3 of this guide for examples of
how the conditions, capability and capacity to undertake effective internal evaluation in a school can
be developed. Section 6 of this guide emphasises the importance of using the same set of indicators
for both internal and external evaluation.
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 9
Learners
DS
AR
STEW
Strategic
EMER
Evaluation
Scope, depth and focus
GE
NT
EV
LU
A
A
Internal evaluations vary greatlyT IinO scope, depth and focus depending on the purpose and the
EV
N
context. An evaluation may be strategic, linked to vision, values, goals and targets; or it may be
AL
UA
a business-as-usual review of, for example, the curriculum or a learning area; or it may be a response
TI
O
to an unforeseen (emergent) event or issue. Figure 2 shows how these different purposes can
N
IN
,
all be viewed as part of a common improvement agenda. Q
IC EVAL
T EG UA
A TI
R O
T N
S
Learners
E M E R G
REG U L AR
EN
T
E
V
E
A
V
L
A
U
U A
A
T
T
IO
IO N
N
10 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Strategic evaluation
Strategic evaluations focus on activities related to the vision, values, goals and targets of the
school community. They aim to find out to what extent the vision is being realised, goals and targets
achieved, and progress made. For boards of trustees and leaders strategic evaluations are a means
of answering such key questions as: To what extent are all our learners experiencing success?
To what extent are improvement initiatives making a difference for all learners? How can we
do better? Because strategic evaluations delve into matters that affect the school as a whole,
and the wider community, they need to be in-depth and they take time.
A strategic evaluation
An intermediate school and a secondary school decided to adopt a joint approach to engaging
with their community. In 2007 and 2011 they carried out extensive community consultation
to determine what parents and community members considered important outcomes for their
children. The results led to the development of seven strategic goals that were reaffirmed
in 2011 and 2013, included in the schools’ strategic plans, and incorporated into annual
goals and targets.
In the intermediate school, a ‘story board’ approach is used to report on the annual action plan.
The principal’s reports to the board use a ‘traffic light’ system to indicate how the school is
tracking in relation to its different strategic goals. This reporting is to a schedule, with one
strategic goal as the focus each month. This approach keeps board members informed and
supports their decision making.
To monitor progress towards its goals the board seeks regular reports from school leaders,
asking How well are we doing? What evidence of progress do we have? What is working well?
What do we need to adjust and why?
Regular monitoring and reporting enables the board to make resourcing decisions that align
with its strategic priorities.
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 11
Regular evaluation
Regular (planned) evaluations are business-as-usual evaluations or inquiries, where boards of
trustees, leaders and teachers (and, where appropriate, students) gather data, monitor progress
towards goals, and assess the effectiveness of programmes or interventions. They ask: To what
extent do our policies and practices promote the learning and wellbeing of all students? How fully
have we implemented the policies we have put in place to improve outcomes for all learners?
How effective are our strategies for accelerating the progress of target learners? Business-as-usual
evaluations vary in scope and depth and feed back into the school’s strategic and annual plans.
A regular evaluation
The social sciences faculty came to evaluate its junior programme as part of its regular review
cycle. Feedback from some senior students had highlighted an apparent lack of alignment
between the junior and senior curriculums.
The first step was to find out what the students thought about the social sciences curriculum,
so faculty staff developed and carried out a survey. But the resulting data was compromised
(some students misunderstood questions and some completed the survey too quickly) and did
not give staff the necessary depth of information. They then realised that they lacked the
technical expertise to design a survey that would tell them what they needed to know.
Deciding that a different approach was needed, staff set up think tank groups that included
students from each class. Teachers presented the survey data to the students and, by
discussing it, were able to get in-depth answers to the survey questions.
Following the think tanks there was a lot of discussion within the faculty about the philosophy
underpinning the current curriculum, the types of assessments used, and what was and was
not working. Staff visited other schools to see their programmes and assess the possible
relevance for them of the different approaches. A Google Doc was set up to facilitate
collaborative redevelopment of the faculty’s programme. A framework for planning units
was agreed, and responsibilities for specific topics allocated. Much of this work took place
at the faculty’s regular professional learning meetings.
Because the level 5 social studies achievement objectives (AOs) are very broad, staff
developed explicit curriculum links to senior social sciences subjects – links that the students
could understand. A benefit of this process was that it required staff to focus closely on the
AOs and ask themselves whether they were really meeting those objectives and providing
a platform for student achievement in the senior years. Sometime later, when faculty staff
conducted a follow-up evaluation, again using a survey and think tank, the response to the
changes was overwhelming positive.
12 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Emergent evaluation
Emergent (or spontaneous) evaluations are a response to an unforeseen event or an issue
picked up by routine scanning or monitoring. Possible focus questions include: What is happening?
Who for? Is this okay? Should we be concerned? Why? Do we need to take a closer look?
Emergent evaluations arise out of high levels of awareness about what is happening for learners.
An emergent evaluation
Leaders and teachers in a primary school found that samples of student writing gathered at the
start of the school year revealed that the writing of 42 percent of year 4–6 students deteriorated
over the summer holiday break. This decline in performance was observed across ethnicity, year
level and teacher. An analysis of November-to-November data showed the students not making
the expected progress. Leaders recognised that they needed to do something differently, but that
the school had limited influence over what happened in the holidays.
Thinking about what action they might take to improve the situation the staff did some reading
on the ‘summer effect’ but found little of use. So they decided to experiment with an entirely
new strategy. This involved collecting a sample of each student’s writing at the end of the year,
pasting it into the front of their exercise book at the start of the new year, and then making
their final learning intention for the old year their first learning intention for the new year.
Expectations are now set on day one. Teachers refer back to the previous year’s writing sample
and discuss with their students the quality of work that is expected of them, emphasising that
this is what they are capable of. Leaders ensure that teachers have the student data from the
previous year and that they have the time to determine learning strategies for the start of the
year. In this way each teacher can start where the previous teacher left off, without the need
to reassess students.
In the first year of implementing this strategy the leadership team rigorously monitored writing
data and what teachers were doing in their classrooms, ensuring that the students were writing
every day. These strategies have worked well. Now, less than 10 percent of students go
backwards in their writing over summer. Explicit teaching strategies target this group with the
aim of accelerating their progress.
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 13
Processes and reasoning
Internal evaluation requires boards, leaders and teachers to engage in deliberate, systematic
processes and reasoning, with improved outcomes for all learners as the ultimate aim.
Those involved collaborate to:
>> investigate and scrutinise practice
>> analyse data and use it to identify priorities for improvement
>> monitor implementation of improvement actions and evaluate their impact
>> generate timely information about progress towards goals and the impact of actions taken.
Figure 3 identifies five interconnected, learner-focused processes that are integral to effective
evaluation for improvement.
Figures 4 to 8 unpack each of these processes in terms of the conditions that support their
effectiveness, the reasoning involved, and the activities or actions involved.
Monitoring
and evaluating
impact
Prioritising
to take Noticing
action
Learner-focused
evaluation processes
We can do better
Collaborative
sense Investigating
making
14 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Monitoring
and evaluating
impact
Prioritising
to take Noticing
action
Learner-focused
evaluation processes
We can do better
FIGURE 4. NOTICING
Collaborative
sense Investigating
making
Noticing
Noticing involves
Scanning
Being aware of hunches,
“If the results don’t look good
gut reactions and anecdotes
we need to be honest about them”
Knowing when to be deliberate
“Knowing what the problem
and intentional
is, is critical”
Recognising the context and focus for
evaluation – strategic, regular or emergent
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 15
Monitoring
and evaluating
impact
Prioritising
to take Noticing
action
Learner-focused
evaluation processes
We can do better
FIGURE 5. INVESTIGATING
Collaborative
sense Investigating
making
Investigating
Investigating involves
Taking stock
Bringing together what we already
“We try and use data from
know (data/information)
whatever source we can get it”
Using existing tools or developing
“We had to know how teachers
new ones to gather data
were teaching writing”
Identifying relevant sources
of data/evidence
Seeking different perspectives
16 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Monitoring
and evaluating
impact
Prioritising
to take Noticing
action
Learner-focused
evaluation processes
We can do better
Collaborative
sense making
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 17
Monitoring
and evaluating
impact
Prioritising
to take Noticing
action
Learner-focused
evaluation processes
We can do better
Prioritising
to take action
Identifying what external expertise we “Everything we’ve done has been decided
might need with data, both quantitative and qualitative”
Prioritising our resources to achieve “If we keep doing the same things we will
equitable outcomes keep getting the same results”
18 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Monitoring
and evaluating
impact
Prioritising
to take Noticing
action
Learner-focused
evaluation processes
We can do better
Monitoring
and evaluating
impact
Why not start a discussion about what each of the five evaluation processes might mean in your own
school community? This will clarify your thinking about evaluation and evaluation practices and help
identify areas where you need to develop greater capability or capacity.
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 19
To think about and discuss
>> What processes do we currently use for the purposes of evaluation and review?
>> How do our processes reflect those described above?
>> Which parts of these processes do we do well? How do we know?
>> How might we use the processes described above to improve the quality and effectiveness
of our evaluation practice?
Effective evaluation requires us to think deeply about the data and information we gather and what
it means in terms of priorities for action. By asking the right questions of ourselves, we will keep
the focus on our learners, particularly those for whom current practice is not working. The twin
imperatives of excellence and equitable outcomes should always be front and centre whatever
it is that we are evaluating.
20 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Organisational conditions and collective capacity
Internal evaluation is most effective when the organisational conditions are supportive and staff
members are encouraged to develop the capabilities to do it well. There is no one way of developing
these conditions and capabilities, but there are some actions and decisions that are likely to help.
The diagrams that follow (Figures 9 and 10) are based on the experiences of some of the case study
schools as they went about developing the conditions and capabilities they needed to engage in
effective internal evaluation. Consider using them in your school community to initiate discussion
about the extent to which you have practices in place that promote effective evaluation and identify
areas for improvement.
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 21
22
FIGURE 9: ORGANISATIONAL CONDITIONS FOR INTERNAL EVALUATION
How can we develop the organisational conditions to support evaluation and inquiry in our school community?
FROM TO
We have not yet developed coherent Examples of development/improvement actions We have coherent organisational
organisational conditions that support taken by case study schools conditions that support evaluation
evaluation and inquiry and inquiry
A collaborative approach to evaluation and reporting
Our vision, values, goals and priorities are is encouraged by involving trustees, leaders and teachers We embed evaluation into the way
developed, planned and implemented as in reviewing student achievement outcomes and we plan and take action to realise our
isolated and disconnected activities rather identifying areas for strategic focus for the next year. vision, values, goals and priorities.
than underpinned by systematic evaluation
We draw on an increasing range of information We use appropriate tools and methods
and inquiry processes.
as a starting point for our regular curriculum/syndicate to gather, store and retrieve a range
We have a limited range of tools and reviews: emerging issues; teacher reflection and feedback; of valid and fit-for-purpose data.
methods to gather, store and retrieve student achievement data; student feedback through
We recognise the importance of student
a range of valid and fit-for-purpose data. surveys and classroom observation; and research.
and community voice and draw on these
We gather feedback from students and Student feedback is actively sought to ascertain the perspectives to identify priorities for
our community on a regular basis but are success of initiatives and issues for further investigation. inquiry and improvement.
not using this well to identify priorities for Several methods for eliciting student feedback are
We have high levels of relational trust
inquiry and improvement. being trialled including focus groups or think tanks
that support collaboration, risk taking, and
and in-class discussions.
Our efforts to change and improve openness to change and improvement.
outcomes for all learners need to be better Leaders take time to introduce new processes to build
We allocate sufficient resources
aligned across the school, building on the trust necessary for engagement in meaningful inquiry.
(for example, time, expertise, staffing)
‘pockets’ of collaboration that already exist. They keep workloads and change processes manageable.
to support change and improvement.
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
We are not yet allocating sufficient Leaders have aligned our inquiry and evaluation activities,
We use systematic, coherent and
resources (for example, time, expertise, for example linking teacher appraisal and inquiry projects
‘smart’ evaluation and reasoning
staffing) to support change and to maximise the impact on what is happening for
processes at all levels of our school.
improvement. identified learners.
We share and disseminate new
Our processes for evaluation and inquiry Professional learning groups provide a forum to discuss
knowledge in ways that promote
tend to be ad hoc and disconnected. and inquire into particular issues. Teachers explore, test
improvement and innovation.
and monitor their responses to their identified issues. New
We have limited ways of sharing and
learnings are shared through whole staff feedback sessions.
disseminating new knowledge and this is
a barrier to improvement and innovation.
FIGURE 10: CAPABILITY AND COLLECTIVE CAPACITY
How can we build the capability and collective capacity to do and use evaluation for improvement?
FROM TO
We are yet to build the capability Examples of development/improvement We have the capability and collective
and collective capacity to do and actions taken by the case study school capacity to do and use evaluation
use evaluation for improvement for improvement
Staff meetings are professional forums. The focus
Our professional learning is focused on is on building professional capability by teachers We engage in professional learning, mentoring
teachers’ interests and/or the expertise sharing readings, research and practice. and coaching to develop the knowledge,
of available professional development skills and confidence needed for evaluation
Leaders are clear about the steps in the change
providers. We are yet to engage in and inquiry.
process, framing up questions for teachers to ask
professional learning that develops
of their practice and developing tools to support We carefully select and use relevant expertise
the knowledge, skills and confidence
the inquiry process. (internal and external) to build our capability
needed for evaluation and inquiry.
and collective capacity in evaluation and inquiry.
Leaders adopt a more collaborative approach to
We have limited or no access to expertise
working with data by increasing the use of data Our leaders and teachers are data literate.
(internal/external) to build our capability
as a focus for discussions in syndicate meetings; They ask good questions, use relevant data,
and collective capacity in evaluation
conversations between teachers; and within the clarify purposes, recognise sound and unsound
and inquiry.
senior leadership team. evidence, understand statistical concepts,
Our leaders and teachers are starting focus on interpretation, and engage in
Leaders engage with research evidence
to develop data literacy. We do rely on evidence-informed conversations.
and share relevant research with teachers
one or two people to manage/analyse
to make better use of evidence in improving When generating solutions and making
and interpret data.
teaching practice. decisions, we draw on current research
We are not aware of/well informed about evidence about what makes a bigger
Leaders have a clear focus on embedding
current research evidence about what difference for learners.
evaluation and inquiry processes by supporting
makes the biggest difference for learners.
team leaders to work with teachers to unpack Engaging in evaluation contributes to changes
We are not yet confident in using their data; deepening conversations about data in our thinking and behaviour and builds inquiry
evaluation and inquiry processes and practice; de-privatising classroom practice ‘habits of mind’.
for improvement. through use of video; and extending teacher
Participation in evidence-based decision
engagement in professional reading through
Our use of evidence in decision making builds our efficacy and agency.
regular discussions and debate.
making is limited.
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
23
3. School improvement
for equity and excellence
While every school community’s improvement is unique it can be described under these
four headings:
>> Context for improvement
>> Improvement actions taken
>> Shifts in practice
>> Outcomes for learners.
24 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Shifts in practice
By monitoring the implementation of improvement actions and evaluating their impact, boards,
leaders and teachers come to learn what works or does not work, for which learners, and why.
In the absence of systematic monitoring and evaluation, shifts in practice, and their impact, can
go unnoticed. Even a small shift, in conjunction with other shifts, can increase forward momentum
or contribute to the realisation of a big goal. The biggest shifts are those that penetrate to the core
of teaching practice. Shifts in the conditions that support effective evaluation will contribute to shifts
in teaching practice.
Monitoring allows for real-time adjustments to be made to improvement actions when they are seen
to be not having the intended impact; it also allows for new knowledge to be harvested and used
more widely for improvement purposes.
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 25
TABLE 1. Examples of shifts in practice that schools have made to improve outcomes for learners
Stewardship
FROM TO
Tabling reports on curriculum review and student Scrutinising and interrogating reports to understand
wellbeing and achievement their implications for decision making
Setting broad targets as a paper exercise Involving teachers in setting appropriate targets
for specific cohorts of students
Focusing on policies and planning Focusing on how well the board is enacting
its stewardship roles and responsibilities
Focusing narrowly on what is happening for students Situating students on a pathway of lifelong learning
while at school
Leadership
FROM TO
Senior leaders monitoring classroom practice Senior leaders mentoring teachers – engaging in
challenging conversations and providing structure
to support teacher reflection
Leaders having little knowledge of recent Leaders keeping up to date with educational research
educational research and using it to help prioritise actions within the school
Expectations of teachers and students not clearly Leaders setting clear expectations of teachers
articulated or consistently implemented and students
Relying on public meetings for parents and whänau Personalised communication to parents and whänau,
seeking feedback on the school’s performance
and direction
Accepting that parents and whänau seldom come Making engagement with parents and whänau an
to school events and interviews ongoing priority
Offering limited opportunities for parents and Providing opportunities for parents and whänau to
whänau to find out about their children’s learning communicate regularly about their children’s learning
15% attendance by parents and whänau 85% attendance after introducing a whänau tutor
at interviews system and giving parents and whänau a
20-minute interview with one teacher
26 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Responsive curriculum, effective teaching and opportunity to learn
FROM TO
Students not being aware of their achievement levels Students knowing where they are at in their learning
or next steps and what their next steps are
A focus on behaviour management – ‘putting out fires’ A focus on effective teaching strategies and providing
opportunities for all students to learn
Deficit thinking and blaming students for poor outcomes Teachers recognising the need to improve their teaching
Appraisal processes in which teachers ‘cherry pick’ Teacher goals linked to improvement actions and
their goals student learning
Data analysis happening at the leadership level Teachers actively engaged in analysing their data
for reporting to the board and using it in their planning and teaching
Evaluation happening as isolated activities A coherent and connected approach to school evaluation
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 27
To think about and discuss
>> Which of these shifts in practice resonate with us?
>> What impact might these shifts have for students?
>> What actions might these schools have taken to make these shifts in practice?
>> What can we learn from these examples?
28 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
School improvement journeys: two examples
The following examples outline how two of the case study schools engaged in internal evaluation
with an improvement agenda. In the first example, a school set out on a two-year journey to raise
achievement in a particular area of the curriculum. In the second example, a school embarked
on a much longer improvement journey that had a number of different but related foci.
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 29
An improvement journey with a curriculum focus
30 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
IMPROVEMENT ACTIONS
>> Deliberate use of senior leadership team capabilities
>> Collaborative development of the writing progressions led to teachers taking more responsibility
for analysing their own classroom data
>> Workloads and change processes kept manageable
>> Videoing of lessons introduced as a tool to support teachers to reflect on their practice
>> Professional discussions about assessment practices, making teacher judgements, and building
a shared understanding of student progress and achievement
>> Students are given access to the writing progressions to support their learning.
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 31
A longer-term improvement journey
This example is of a longer-term improvement journey (2006–14). This school began with an
investigation into the achievement of Mäori students, followed by an investigation into the
achievement of boys, and then a third investigation into student engagement and wellbeing.
Although the three evaluations each had a different focus they were obviously connected.
As the board of trustees, leaders and teachers inquired into their practice, their individual
capabilities and collective capacity to engage in internal evaluation developed over time.
The diagrams on the next three pages show how, for each of the three inquiries, the school
went through the five evaluation processes described on pages 15-20.
Developing a
values approach
to student
wellbeing
Raising the
achievement
of boys
Raising
Mäori students’
achievement
32 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Raising the achievement of Mäori students
Investigating
All the available data was closely re-examined.
Using the Effective Teacher Profile as a guide, all year 9 and 10 teachers were observed in the
classroom; observations were followed by feedback sessions.
Improvement actions
“If we were going to keep doing for our Mäori students what we had always done,
we were going to get what we’d always got – and it was way not good enough.”
“We are going to keep working to achieve the target of raising our
Mäori student achievement to mirror the achievement of our non-
Mäori students. We are going to keep going until there is no gap.”
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 33
Shifts in practice
From To
Having the ‘will’ to make a difference for Mäori students Finding the ‘way’ to make a difference for Mäori students
Accepting that whänau are seldom seen at school Whänau engagement is an ongoing focus
events and interviews for improvement
100
80
60
40 European/Päkeha
20 students
Mäori Students
0
2005 2014
2014 – The disparity between Mäori and Päkeha gaining NCEA level 2 is reducing
34 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Raising the achievement of boys
Investigating
Randomly selected boys from across the school’s seven year levels were involved in focus groups
to find out what worked well for them in terms of supporting their learning and achievement,
and what didn’t.
Improvement actions
A professional learning group was established. The teachers in this group committed to using
a teaching-as-inquiry approach with four or five boys in their classroom, trying different strategies
and reflecting on what worked and what didn’t. They met regularly to share their reflections with
others in the group.
An external facilitator worked with the staff on ‘schema’, which was about understanding the
different ways in which girls and boys engaged and behaved. This ‘opened the door for staff
to admire boys’ behaviour’.
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 35
Shifts in practice
From To
100
80
60
Level 1
40
20 Level 2
0
2005 2014
2014 – T
he percentages of boys gaining NCEA levels 1 and 2 have increased significantly
36 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Developing a values approach to student wellbeing
In 2011 the school’s approach to behaviour management consisted of a set of expected
behaviours (rules), consequences for breaching the rules, and disciplinary steps.
Investigating
Positive Behaviour for Learning (PB4L) facilitators gathered data to find out how well the school’s
behaviour management approach was known by staff and students.
Improvement actions
The school embarked on an 18-month process
of consultation to identify an agreed set of school
W We are learners – Whaia te mätauranga
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 37
Shifts in practice
From To
A discipline system based on poorly understood A values-based approach ‘lived’ in every area
rules and consequences of school life
2014 – T
he numbers of stand downs and suspensions have fallen progressively over the
last 4 years
Suspensions 42 25 22 15
38 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
4. Using the school evaluation indicators
to support continuous improvement
ANAUNGATANGA
WH
LE
AD
Responsive
ER
P
HI
curriculum,
SH
DS
effective teaching
IP
AR
and opportunity
STEW
to learn
MAN
Learners
MAHI TAHI
AAKITANGA
Educationally
powerful
connections and
relationships
EV
ON
AL
TI
VA
UA
AK O
NO
TI
O
IN
PR
N
IN O TY
,
AN F E S SIO N A L C A PA BILI
D
Q AN
UI DC
RY OLLEC CITY T
TIVE CAPA N
AN ME
D
KN VE
OW RO
LEDG IMP
E B UI LDIN G FO R
The school evaluation indicators identify practices that contribute to effective evaluation. These
practices should be viewed holistically, not used in isolation or as a checklist. Use them as a ‘sieve’
or trustworthy reference when trying to make sense of data and to answer the question, How good is
our practice? Use them too when investigating the relationship between different – and, sometimes,
apparently unconnected – aspects of your school’s performance.
Table 2 provides a broad framework for thinking about how you might use the indicators in your own
school. Possible uses are suggested in relation to each of the five evaluation processes identified
and discussed in this guide.
The indicators can be used in different ways during different phases of internal evaluation.
For example:
>> Use the outcome indicators as the starting point for evaluating the educational opportunities
that your school offers its students and investigating who is/is not achieving, and identifying
those students whose progress needs to be accelerated
>> Use the six domains of influence as a framework for identifying where you might concentrate
your inquiry and data gathering
>> Use the process indicators and associated effective practice statements as a framework when
trying to make sense of your data and identify and prioritise improvement actions
>> When monitoring and evaluating the impact of your improvement actions, the outcome indicators
can help you maintain focus on what is happening for all learners.
The indicators for Domain 6, Evaluation, inquiry and knowledge building for improvement
and innovation, and the associated effective practice statements can be used to evaluate the
effectiveness of your school’s approach to evaluation.
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 39
TABLE 2. USING THE SCHOOL EVALUATION INDICATORS
Outcome indicators focus What does our information tell us about the extent to
evaluation on the learner. which all learners are achieving high-level curriculum
They can assist in mapping outcomes: confidence in identity, language and culture;
Noticing
the context and identifying wellbeing; achievement and progress; and participation
what matters most in terms and contribution? Which learners are/are not achieving,
of student outcomes and in what contexts?
Outcome indicators How are we going? What is our data/information telling us?
Monitoring
provide the starting point Do we need to modify or change what we are doing?
and evaluating
for monitoring and evaluating What impact is our action having? Are we providing
impact
the impact of actions taken opportunities for all learners to learn, progress and achieve?
40 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
5. Integrating internal and external
evaluation for improvement
School evaluation has both internal and external components, and the two should be seen
as working together.
New Zealand has ... gone furtherest among countries internationally towards a collaborative
school evaluation model, incorporating at the same time a sequential process ... [the] approach
is collaborative in the sense that both parties attempt to work together to agree on a rounded
picture of the school in which there is mutual recognition of its strengths and consensus on
areas for development.10
Schools set in place their own evaluation processes and, as part of an annual reporting cycle,
provide regular accounts of student achievement in relation to goals and targets, along with planned
improvement actions.
Then periodically ERO conducts an external evaluation. The school’s student achievement information
and associated internal evaluation forms the starting point for ERO’s external evaluation and for
engagement with other education agencies and professionals.
School Evaluation Indicators: Effective Practice for Improvement and Learner Success expresses
the intent of ERO’s external evaluations in this way:
The indicators are designed to make it easier for internal and external evaluations to talk to each other:
The indicators provide a common language for the interaction and dialogue between a school
and ERO about the development since the last review, the current state and future direction.
(ERO, 2015, p6)
Domain 6 of the indicators framework, Evaluation, inquiry and knowledge building for
improvement and innovation is the engine that drives improvement. The indicators in this
domain highlight the relationship between organisational conditions and the capacity to do and
use evaluation for improvement, and how these influence engagement with external evaluation.
In effective schools internal evaluation processes are systematic, coherent and connected at all
levels of the school. This alignment ensures that leaders, teachers and boards of trustees are able
to purposefully engage with external evaluation, using it as an opportunity to review, validate and
support their own improvement actions.
10 Nusche, D., Laveault, D., MacBeath, J., & Santiago, P. ( 2012). OECD Reviews of Evaluation and Assessment in Education: New Zealand 2011,
OECD Publishing (p.105).
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 41
Getting the most out of external evaluation
In thinking about and preparing for your ERO external evaluation the questions below provide
a useful framework for developing an account of your school’s development since the last evaluation,
where that development is currently at, and areas for future focus.
What are the outcomes that are valued for all learners in this school community, as learners in Aotearoa
New Zealand, and as global citizens?
How well are all our learners achieving in relation to those outcomes?
To what extent is every student in our school a successful “confident, connected, actively involved,
lifelong learner”?
How well are we identifying and accelerating the achievement of those learners at risk of not achieving
equitable outcomes?
How do we know? What sources of evidence tell us about our performance and effectiveness?
What are our continuing and/or next big areas for development and improvement?
This will provide an overview of your internal evaluation findings to present to ERO. ERO uses your
school’s learner outcome information and internal evaluation account to work with you to design
the external evaluation for your school context.
During the evaluation process the ERO team will discuss the emerging evaluation findings with
you and involve you in the interpretation of the evidence gathered to develop a shared understanding
of the quality and effectiveness of education provision in your school and next steps.
The evaluation insights provided by ERO as an outcome of the external evaluation process should
support every child to succeed as a lifelong learner.
42 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
School evaluation for equity and excellence
N IN
IO T
E
T
R
A
N
U
AL
EVAL
Learners
EVAL
AL
N
U
R
A
E T
T IO
EX N
EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 43