R E VA L U AT I O

G ULA N
RE

P
Learners

HI
DS
AR
EFFECTIVE

STEW
Strategic

MAHI TAHI
Evaluation EMER

SCHOOL
GE
NT

EVALUATION
EV

LU
A

AT
I ON

EV
AL
UA
TI
O
N
IN
How to do and use internal evaluation for improvement

,
Q
UI
RY

IC EVAL
T EG UA
A TI
R O
T N
S

Learners
E M E R G

REG U L AR
EN
T

E
V
E

A
V

L
A

U
U A
A
T
T
IO
IO N
N
Effective School Evaluation
Published 2016
© Crown copyright
ISBN 978-0-478-43842-0

Except for the Education Review Office and Ministry of Education logos, this copyright work is
licensed under Creative Commons Attribution 3.0 New Zealand licence. In essence, you are free to
copy, distribute and adapt the work, as long as you attribute the work to the Education Review Office
and the Ministry of Education and abide by the other licence terms. In your attribution, use the
wording ‘Education Review Office’ and ‘Ministry of Education’, not the Education Review Office and
Ministry of Education logos or the New Zealand Government logo.
Contents
Introduction 4
How this resource is structured 5
Context 6

1. What matters most in schooling 7

2. Engaging in effective internal evaluation 9

Scope, depth and focus 10
Processes and reasoning 14
Organisational conditions and collective capacity 21

3. School improvement for equity and excellence 24

School improvement journeys: two examples 29

4. Using the school evaluation indicators to support continuous improvement 39

5. Integrating internal and external evaluation for improvement 41

 Introduction
This guide to using internal evaluation for improvement purposes is published jointly by ERO
and the Ministry of Education as a companion to School Evaluation Indicators: Effective Practice
for Improvement and Learner Success and is supported by Internal Evaluation: Good Practice.1
This resource describes what effective internal evaluation is, what it involves, and how to go about
it in ways that will enhance educational outcomes for students. It draws on current knowledge about
internal evaluation, and on case studies of New Zealand schools that have used internal evaluation to
inform the development of strategies that have been successful in raising achievement, particularly
that of Mäori and Pacific learners. The case study schools also exemplify the use of inquiry processes
and evaluative reasoning.
The intention is that this resource will assist English-medium schools and their communities
to understand and practise effective evaluation for improvement. It can be used in different
ways with different groups.

1 See www.ero.govt.nz for both publications

4 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
How this resource is structured

This resource has five sections.

The first section, What matters most in schooling, will help you to understand the framework
that underpins the school evaluation indicators. This framework integrates the key learner outcomes
highlighted in The New Zealand Curriculum and Te Marautanga o Aotearoa with six domains
identified by contemporary research and evaluation studies as being significant influences
on school effectiveness and improvement. Pages 7 to 8

Engaging in effective internal evaluation identifies three different but related purposes for and
approaches to internal evaluation in schools: strategic, regular and emergent. Examples highlight
particular features of the different approaches. This section will help you to engage in internal
evaluation and understand what is needed to do it well. It highlights the actions you can take
to develop the organisational conditions, capability and collective capacity to do and use internal
evaluation for improvement purposes. Pages 9 to 23

The third section, School improvement for equity and excellence, will help you to explore the
relationship between internal evaluation and school improvement. It describes the kinds of shifts
in practice that some of the case study schools were making as a result of internal evaluation.
This section includes examples of improvement in two of the case study schools showing how
they used evaluation and reasoning processes to inform their actions, with resulting shifts
in practice and positive impacts on student outcomes. Pages 24 to 38

The fourth section, Using the school evaluation indicators to support continuous improvement,
will help you to better understand the relationship between internal and external evaluation and how
aligning the two, with the evaluation indicators as a common framework, will support ongoing,
sustained improvement in achievement and equity. Pages 39 to 40

The final section, Integrating internal and external evaluation for improvement, reiterates the
value to schools of harmonising their internal evaluation processes with ERO’s external evaluations.
It supports schools to use their internal evaluation findings to engage with others in their school
community and beyond. Domain six of the indicator framework, Evaluation, inquiry and knowledge
building for improvement and innovation, highlights how organisational conditions and capability
influence engagement with external evaluation. Pages 41 to 43

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 5
 Context

School evaluation can be either internal or external; in New Zealand it is both. ERO has mandated responsibility
for external reviews while National Administration Guideline 2 explicitly requires every school to:
>> develop a strategic plan which documents how it is giving effect to the National Education Guidelines
through its policies, plans and programmes, including those for curriculum, National Standards,
assessment, and staff professional development
>> maintain an ongoing programme of self review in relation to the above policies, plans and programmes,
including evaluation of information on student achievement.2

The intention is that all schools and their communities should be engaged in an ongoing, cyclical process
of internal evaluation and inquiry that is directed at improving educational outcomes for all their students.
Effective internal evaluation at a community/cluster/school level also contributes to improving outcomes
for learners in the wider community and at a system level.
As used in this guide, the term ‘internal evaluation’ includes all the activities and processes that schools and their
communities engage in to evaluate how effective they are in supporting success for all learners. These include
schools’ ongoing programme of self-review, planning and reporting processes, and other inquiry and evaluation
activities that boards, leaders and teachers engage in for the purposes of accountability and improvement.
The New Zealand Curriculum and Te Marautanga o Aotearoa both position teaching within an inquiry
framework and highlight the importance of teachers and leaders having the expertise to inquire into their
practice, evaluate the impact, and build knowledge about what works.3
Schools and their communities need to be continuously evaluating the impact of their endeavours on learner
outcomes. To do this, they need strong leadership and evaluation expertise. Their systems, processes and
resources should support purposeful data gathering, collaborative inquiry and decision making and align
closely with the school’s vision, values, strategic direction, goals, and equity and excellence priorities.
Using a range of quality achievement data, boards of trustees, leaders and teachers in highly effective
schools collaboratively monitor and evaluate the impact of improvement strategies. They are responsive
to findings, making changes as necessary, and successful strategies are embedded into school practice
so that all learners benefit.
Periodic external evaluation supports this cycle of continuous improvement by bringing an external lens
to bear on achievement, the extent to which conditions support improvement, and next steps.
The primary purpose of school evaluation is to achieve equity and excellence in outcomes for all learners.
Effective internal evaluation processes enable trustees, leaders, teachers, parents, families, whänau
and the wider school community to better understand:
>> how individual learners and groups of learners are performing in relation to valued outcomes
>> how improvement actions have impacted on learner outcomes and what difference is being made
>> what needs to be changed and what further action needs to be taken
>> the patterns and trends in outcomes over time
>> what kinds of practices are likely to make the most difference for diverse learners and in what contexts
>> the extent to which the improvements achieved are good enough in terms of the school’s vision,
values, strategic direction, goals, and equity and excellence priorities.

2 Ministry of Education. (2015). The National Administration Guidelines (NAGs). Retrieved from http://www.education.govt.nz/ministry-of-
education/legislation/nags/.
3 Timperley, H., & Parr, J. (2010). Evidence, inquiry and standards. In H. Timperley & J. Parr (Eds.), Weaving evidence, inquiry and standards
to build better schools (pp. 9–23). Wellington: NZCER Press.

6 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
1. What matters most in schooling
The major challenge for the New Zealand education system is achieving equity and excellence
in outcomes for an increasingly diverse student population. Although a significant proportion
of New Zealand students achieve at the highest levels in core areas such as reading, mathematics
and science, our performance in international assessment studies shows persistent achievement
disparities, in particular for Mäori and Pacific students. Achieving excellence must go hand in hand
with achieving equity of education outcomes.

Equity in education can be seen through two dimensions: fairness and inclusion. Equity as
fairness implies that personal or socio-economic circumstances such as gender, ethnic origin
or family background, are not obstacles to success in education. Equity as inclusion means
ensuring that all students reach at least a basic minimum level of skills. Equitable education
systems are fair and inclusive, and support their students in reaching their learning potential
without either formally or informally erecting barriers or lowering expectations.4

ERO has redeveloped its indicators framework (Figure 1) to highlight six domains that are known
to be key influences in improving outcomes for all learners. The aim has been to create a framework
that can be used not only by ERO’s evaluators, but also by schools to focus their internal evaluation
and improvement activities.

FIGURE 1. SCHOOL EVALUATION INDICATORS FRAMEWORK

ANAUNGATANGA
WH
LE
AD

Responsive
ER
P
HI

curriculum,
SH
DS

effective teaching
IP
AR

and opportunity
STEW

to learn
MAN

Learners
MAHI TAHI

AAKITANGA

Educationally
powerful
connections and
relationships
EV

ON
AL

TI
VA
UA

AK O
NO
TI
O

IN

PR
N

IN O TY
,

AN F E S SIO N A L C A PA BILI
D

Q AN
UI DC
RY OLLEC CITY T
TIVE CAPA N
AN ME
D
KN VE
OW RO
LEDG IMP
E B UI LDIN G FO R

4 Schleicher, A. (2014). Equity, excellence and inclusiveness in education: policy lessons from around the world.
Paris: Organisation for Economic Co-operation and Development. www.oecd.org/publishing

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 7
 Learners are at the centre of the indicators framework, with the goal being successful, ‘confident,
connected, actively involved, lifelong learners’ as envisioned by The New Zealand Curriculum
and Te Marautanga o Aotearoa5. This means that every young person should leave school:
>> confident in their identity, language and culture as a citizen of Aotearoa New Zealand
>> socially and emotionally competent, resilient and optimistic about the future6
>> a successful lifelong learner
>> participating and contributing confidently in a range of contexts (cultural, local, national and global)
to shape a sustainable world of the future.7

The six domains included in the framework have all been identified by education research studies
and analyses of studies as significant influences on school effectiveness and improvement. Two of
these domains, educationally powerful connections and relationships and responsive curriculum,
effective teaching and opportunity to learn, have the greatest influence, but all are mutually
interdependent.
Research evidence highlights how critical leadership is for achieving equity and excellence goals.
It falls to leaders to establish the necessary conditions and relational trust, and it is leaders who
ensure that teachers have opportunities to collaboratively inquire into, reflect on and evaluate their
practice, thereby building professional capability and collective capacity.
Entrusted with a stewardship role, boards of trustees represent and serve the school community.
They have a responsibility to scrutinise student achievement and evaluation information and to
maintain a relentless focus on learning, wellbeing, achievement and progress. They must also
evaluate their own performance and ensure that they are meeting their accountabilities.
The remaining domain, evaluation, inquiry and knowledge building, is the engine that drives
improvement and innovation.
All education activities take place within a cultural context. For this reason the indicator framework
singles out four concepts, manaakitanga, whanaungatanga, ako and mahi tahi, which have the
power to transform the learning environment for students. Together, these concepts challenge
teachers to abandon deficit theorising as a way of rationalising failure and to focus instead on
how they can change their approach and practices to realise the potential in their students.8
See School Evaluation Indicators – Effective Practice for Improvement and Learner Success
for an explanation of these concepts.9
A school’s effectiveness closely correlates with the quality of its practices in all six domains
and the extent to which those practices are integrated and coherent.

5 Retrieved from Te Marautanga o Aotearoa Graduate Profile. Retrieved from http://www.tmoa.tki.org.nz/Te-Marautanga-o-Aotearoa/Te-Anga/

Te-Ahua-o-a-Tatou-Akonga
6 See Education Review Office (2013). Wellbeing For Success: Draft Evaluation Indicators for Student Wellbeing. See also The New Zealand
Curriculum (2007) Health and Physical Education.
7 See Bolstad, R., & Gilbert, J., with McDowall, S., Bull, A., Boyd, S., & Hipkins, R. (2012). Supporting future-oriented teaching and learning
– a New Zealand perspective. Retrieved from https://www.educationcounts.govt.nz/publications/schooling/109306
8 See Bishop, R., O’Sullivan, D, & Berryman, M. (2010). Scaling up education reform: The politics of disparity. Wellington: New Zealand
Council for Education Research.
9 Education Review Office (2015). School Evaluation Indicators: Effective Practice for Improvement and Learner Success. Wellington:
Education Review Office.

8 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
2. Engaging in effective internal evaluation
The whole point of internal evaluation is to assess what is and is not working, and for whom, and then
to determine what changes are needed, particularly to advance equity and excellence goals. Internal
evaluation involves asking good questions, gathering fit-for-purpose data and information, and then
making sense of that information. Much more than a technical process, evaluation is deeply
influenced by the school’s values and how it sees its role in the community. Effective internal
evaluation is always driven by the motivation to improve, to do better for the students.
When internal evaluation is done well, processes are coherent and align with schools’ visions and
strategic goals. Leaders and teachers work collaboratively across teams, syndicates, departments,
faculties and in some cases communities of learning, to ensure that the efforts that go into evaluation
lead to improvement. The urgency to improve is shared by all, and can be articulated by all.
Evidence from research and the case study schools shows that there are organisational conditions
that support development of the capacity to do and use evaluation for improvement and innovation.
These include:
>> evaluation leadership
>> a learning-oriented community of professionals that demonstrates agency in using evaluation
for improvement in practice and outcomes
>> opportunity to develop technical evaluation expertise (including access to external expertise)
>> access to, and use of, appropriate tools and methods
>> systems, processes and resources that support purposeful data gathering, knowledge building
and decision making.

See the Domain 6 indicators for practices that are characteristic of effective evaluation, inquiry and
knowledge building for improvement and innovation and Section 3 of this guide for examples of
how the conditions, capability and capacity to undertake effective internal evaluation in a school can
be developed. Section 6 of this guide emphasises the importance of using the same set of indicators
for both internal and external evaluation.

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 9
 Learners

DS
AR
STEW
Strategic

EMER
Evaluation
Scope, depth and focus

GE
NT
EV
LU

A
A
Internal evaluations vary greatlyT IinO scope, depth and focus depending on the purpose and the

EV
N
context. An evaluation may be strategic, linked to vision, values, goals and targets; or it may be

AL
UA
a business-as-usual review of, for example, the curriculum or a learning area; or it may be a response

TI
O
to an unforeseen (emergent) event or issue. Figure 2 shows how these different purposes can

N
IN

,
all be viewed as part of a common improvement agenda. Q

FIGURE 2. TYPES OF INTERNAL EVALUATION

IC EVAL
T EG UA
A TI
R O
T N
S

Learners
E M E R G

REG U L AR
EN
T

E
V
E

A
V

L
A

U
U A
A
T
T
IO
IO N
N

10 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Strategic evaluation
Strategic evaluations focus on activities related to the vision, values, goals and targets of the
school community. They aim to find out to what extent the vision is being realised, goals and targets
achieved, and progress made. For boards of trustees and leaders strategic evaluations are a means
of answering such key questions as: To what extent are all our learners experiencing success?
To what extent are improvement initiatives making a difference for all learners? How can we
do better? Because strategic evaluations delve into matters that affect the school as a whole,
and the wider community, they need to be in-depth and they take time.

A strategic evaluation

An intermediate school and a secondary school decided to adopt a joint approach to engaging
with their community. In 2007 and 2011 they carried out extensive community consultation
to determine what parents and community members considered important outcomes for their
children. The results led to the development of seven strategic goals that were reaffirmed
in 2011 and 2013, included in the schools’ strategic plans, and incorporated into annual
goals and targets.
In the intermediate school, a ‘story board’ approach is used to report on the annual action plan.
The principal’s reports to the board use a ‘traffic light’ system to indicate how the school is
tracking in relation to its different strategic goals. This reporting is to a schedule, with one
strategic goal as the focus each month. This approach keeps board members informed and
supports their decision making.
To monitor progress towards its goals the board seeks regular reports from school leaders,
asking How well are we doing? What evidence of progress do we have? What is working well?
What do we need to adjust and why?
Regular monitoring and reporting enables the board to make resourcing decisions that align
with its strategic priorities.

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 11
 Regular evaluation
Regular (planned) evaluations are business-as-usual evaluations or inquiries, where boards of
trustees, leaders and teachers (and, where appropriate, students) gather data, monitor progress
towards goals, and assess the effectiveness of programmes or interventions. They ask: To what
extent do our policies and practices promote the learning and wellbeing of all students? How fully
have we implemented the policies we have put in place to improve outcomes for all learners?
How effective are our strategies for accelerating the progress of target learners? Business-as-usual
evaluations vary in scope and depth and feed back into the school’s strategic and annual plans.

A regular evaluation

The social sciences faculty came to evaluate its junior programme as part of its regular review
cycle. Feedback from some senior students had highlighted an apparent lack of alignment
between the junior and senior curriculums.
The first step was to find out what the students thought about the social sciences curriculum,
so faculty staff developed and carried out a survey. But the resulting data was compromised
(some students misunderstood questions and some completed the survey too quickly) and did
not give staff the necessary depth of information. They then realised that they lacked the
technical expertise to design a survey that would tell them what they needed to know.
Deciding that a different approach was needed, staff set up think tank groups that included
students from each class. Teachers presented the survey data to the students and, by
discussing it, were able to get in-depth answers to the survey questions.
Following the think tanks there was a lot of discussion within the faculty about the philosophy
underpinning the current curriculum, the types of assessments used, and what was and was
not working. Staff visited other schools to see their programmes and assess the possible
relevance for them of the different approaches. A Google Doc was set up to facilitate
collaborative redevelopment of the faculty’s programme. A framework for planning units
was agreed, and responsibilities for specific topics allocated. Much of this work took place
at the faculty’s regular professional learning meetings.
Because the level 5 social studies achievement objectives (AOs) are very broad, staff
developed explicit curriculum links to senior social sciences subjects – links that the students
could understand. A benefit of this process was that it required staff to focus closely on the
AOs and ask themselves whether they were really meeting those objectives and providing
a platform for student achievement in the senior years. Sometime later, when faculty staff
conducted a follow-up evaluation, again using a survey and think tank, the response to the
changes was overwhelming positive.

12 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Emergent evaluation
Emergent (or spontaneous) evaluations are a response to an unforeseen event or an issue
picked up by routine scanning or monitoring. Possible focus questions include: What is happening?
Who for? Is this okay? Should we be concerned? Why? Do we need to take a closer look?
Emergent evaluations arise out of high levels of awareness about what is happening for learners.

An emergent evaluation

Leaders and teachers in a primary school found that samples of student writing gathered at the
start of the school year revealed that the writing of 42 percent of year 4–6 students deteriorated
over the summer holiday break. This decline in performance was observed across ethnicity, year
level and teacher. An analysis of November-to-November data showed the students not making
the expected progress. Leaders recognised that they needed to do something differently, but that
the school had limited influence over what happened in the holidays.
Thinking about what action they might take to improve the situation the staff did some reading
on the ‘summer effect’ but found little of use. So they decided to experiment with an entirely
new strategy. This involved collecting a sample of each student’s writing at the end of the year,
pasting it into the front of their exercise book at the start of the new year, and then making
their final learning intention for the old year their first learning intention for the new year.
Expectations are now set on day one. Teachers refer back to the previous year’s writing sample
and discuss with their students the quality of work that is expected of them, emphasising that
this is what they are capable of. Leaders ensure that teachers have the student data from the
previous year and that they have the time to determine learning strategies for the start of the
year. In this way each teacher can start where the previous teacher left off, without the need
to reassess students.
In the first year of implementing this strategy the leadership team rigorously monitored writing
data and what teachers were doing in their classrooms, ensuring that the students were writing
every day. These strategies have worked well. Now, less than 10 percent of students go
backwards in their writing over summer. Explicit teaching strategies target this group with the
aim of accelerating their progress.

To think about and discuss

>> What is the focus of evaluation in our school community?
>> Do we use all three types of evaluation (strategic, regular and emergent)?
>> In what areas could we improve our understanding and practice of evaluation?
>> How do we ensure that all evaluation supports our school’s vision, values and strategic goals?
>> How do we ensure that all evaluation promotes equitable outcomes for learners?

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 13
 Processes and reasoning

Internal evaluation requires boards, leaders and teachers to engage in deliberate, systematic
processes and reasoning, with improved outcomes for all learners as the ultimate aim.
Those involved collaborate to:
>> investigate and scrutinise practice
>> analyse data and use it to identify priorities for improvement
>> monitor implementation of improvement actions and evaluate their impact
>> generate timely information about progress towards goals and the impact of actions taken.

Figure 3 identifies five interconnected, learner-focused processes that are integral to effective
evaluation for improvement.
Figures 4 to 8 unpack each of these processes in terms of the conditions that support their
effectiveness, the reasoning involved, and the activities or actions involved.

FIGURE 3. LEARNER-FOCUSED EVALUATION PROCESSES AND REASONING

Monitoring
and evaluating
impact

Prioritising
to take Noticing
action
Learner-focused
evaluation processes

We can do better

Collaborative
sense Investigating
making

14 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
 Monitoring
and evaluating
impact

Prioritising
to take Noticing
action
Learner-focused
evaluation processes

We can do better

FIGURE 4. NOTICING
Collaborative
sense Investigating
making

When noticing We ask ourselves

We maintain a learner focus What’s going on here?
We are aware of what is happening For which learners?
for all learners Is this what we expected?
We have a deep understanding of Is this good?
The New Zealand Curriculum
Should we be concerned? Why?
We have an ‘inquiry habit of mind’
What is the problem or issue?
We are open to scrutinising our data
Do we need to take a closer look?
We see dissonance and discrepancy
as opportunties for deeper inquiry

Noticing

Noticing involves
Scanning
Being aware of hunches,
“If the results don’t look good
gut reactions and anecdotes
we need to be honest about them”
Knowing when to be deliberate
“Knowing what the problem
and intentional
is, is critical”
Recognising the context and focus for
evaluation – strategic, regular or emergent

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 15
 Monitoring
and evaluating
impact

Prioritising
to take Noticing
action
Learner-focused
evaluation processes

We can do better

FIGURE 5. INVESTIGATING
Collaborative
sense Investigating
making

When investigating We ask ourselves

We focus on what is happening for What do we already know about this?
all our learners How do we know this?
We ensure we have sufficent data What do we need to find out?
to help us respond to our questions
How might we do this?
We check that our data is fit-for-purpose
What ‘good questions’ should we ask?
We actively seek students’ and
How will we gather relevant and
parents’ perspectives
useful data’?
We check we know about all learners
in different situations

Investigating

Investigating involves
Taking stock
Bringing together what we already
“We try and use data from
know (data/information)
whatever source we can get it”
Using existing tools or developing
“We had to know how teachers
new ones to gather data
were teaching writing”
Identifying relevant sources
of data/evidence
Seeking different perspectives

16 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
 Monitoring
and evaluating
impact

Prioritising
to take Noticing
action
Learner-focused
evaluation processes

We can do better

FIGURE 6. COLLABORATIVE SENSE MAKING

Collaborative
sense Investigating
making

When making sense of our data We ask ourselves

and information
What is our data telling us/what insights
We ensure we have the necessary does it provide?
capability (data literacy) and capacity
Is this good enough?
(people/time)
How do we feel about what we have
We are open to new learning
found?
We know ‘what is so’ and have
Do we have different interpretations
determined ‘so what’
of the data? If so, why?
We know what ‘good’ looks like so that
What might we need to explore further?
we can recognise our strengths and areas
for improvement What can we learn from research
evidence about what ‘good’ looks like?
We have a robust evidence base to inform
our decision making and prioritising How close are we to that?

Collaborative
sense making

Collaborative sense making involves

Scrutinising our data with an open mind
Working with different kinds of data,
“We want to know what’s good
both quantitative and qualitative
– and what’s not good enough”
Drawing on research evidence and using
“To effect change, teachers needed
suitable frameworks or indicators when
to be on board – it was not going to
analysing and making sense of our data
be a two-meeting process”
“What are we doing well? What can we
improve on? How can we enrich and
accelerate the learning of our students?”

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 17
 Monitoring
and evaluating
impact

Prioritising
to take Noticing
action
Learner-focused
evaluation processes

We can do better

FIGURE 7. PRIORITISING TO TAKE ACTION

Collaborative
sense Investigating
making

When prioritising to take action We ask ourselves

We are clear what problem or issue What do we need to do and why?
we are trying to solve
What are our options?
We understand what we need to improve Have we faced this situation before?
We are determined to achieve equitable What can we do to ensure better progress
progress and outcomes for all students and outcomes for more of our learners?
We are clear about the actions we need How big is the change we have in mind?
to take and why
Can we get the outcomes we want within
We have the resources necessary the timeframe we have specified?
to take action
What strengths do we have to draw
We have a plan that sets out clear on/build on?
expectations for change
What support/resources might we need?

Prioritising
to take action

Prioritising to take action involves

Considering possible options in light
of the ‘what works’ evidence “Prioritising is based on having capacity
– you can’t stretch yourself too far”
Being clear about what needs to change
and what doesn’t “Everything we have done has been
based on the evidence”
Identifying where we have the capability
and capacity to improve “We are about wise owls not bandwagons”

Identifying what external expertise we “Everything we’ve done has been decided
might need with data, both quantitative and qualitative”

Prioritising our resources to achieve “If we keep doing the same things we will
equitable outcomes keep getting the same results”

18 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
 Monitoring
and evaluating
impact

Prioritising
to take Noticing
action
Learner-focused
evaluation processes

We can do better

FIGURE 8. MONITORING AND EVALUATING IMPACT

Collaborative
sense Investigating
making

When monitoring and evaluating We ask ourselves

We know what we are aiming to achieve What is happening as a result of our
We are clear about how we will improvement actions?
monitor progress What evidence do we have?
We know how we will recognise progress Which of our students are/are not
We are focused on ensuring all learners benefiting?
have equitable opportunities to learn How do we know?
We have the capability and capacity Is this good enough?
to evaluate the impact of our actions Do we need to adjust what we are doing?
What are we learning?
Can we use this learning in other areas?

Monitoring
and evaluating
impact

Monitoring and evaluating involves

Keeping an eye on the data for evidence
of what is/is not working for all learners
“Success is still fragile – if you have
Having systems, processes and tools
a group (that is not achieving to
in place to track progress and impact
expectations) you focus on them
Developing progress markers that will and keep focusing on them”
help us to know whether we are on the
“Results after one year showed a
right track
14 percent upward shift in students
Checking in with students and their achieving at or above the standard”
parents and whänau
Knowing when to adjust or change
actions or strategies

Why not start a discussion about what each of the five evaluation processes might mean in your own
school community? This will clarify your thinking about evaluation and evaluation practices and help
identify areas where you need to develop greater capability or capacity.

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 19
 To think about and discuss
>> What processes do we currently use for the purposes of evaluation and review?
>> How do our processes reflect those described above?
>> Which parts of these processes do we do well? How do we know?
>> How might we use the processes described above to improve the quality and effectiveness
of our evaluation practice?

Effective evaluation requires us to think deeply about the data and information we gather and what
it means in terms of priorities for action. By asking the right questions of ourselves, we will keep
the focus on our learners, particularly those for whom current practice is not working. The twin
imperatives of excellence and equitable outcomes should always be front and centre whatever
it is that we are evaluating.

20 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Organisational conditions and collective capacity

Internal evaluation is most effective when the organisational conditions are supportive and staff
members are encouraged to develop the capabilities to do it well. There is no one way of developing
these conditions and capabilities, but there are some actions and decisions that are likely to help.
The diagrams that follow (Figures 9 and 10) are based on the experiences of some of the case study
schools as they went about developing the conditions and capabilities they needed to engage in
effective internal evaluation. Consider using them in your school community to initiate discussion
about the extent to which you have practices in place that promote effective evaluation and identify
areas for improvement.

To think about and discuss

>> To what extent do the organisational conditions in our school community promote evaluation?
>> How well do our policies, systems and practices embed evaluation that contributes to ongoing
improvement for all students?
>> To what extent do we have the relational trust needed to support collaboration and risk taking
at all levels of the school community?
>> How deliberate are we in using our internal evaluation processes as opportunities for individual
and collective professional learning?

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 21
22
FIGURE 9: ORGANISATIONAL CONDITIONS FOR INTERNAL EVALUATION
How can we develop the organisational conditions to support evaluation and inquiry in our school community?

FROM TO

We have not yet developed coherent Examples of development/improvement actions We have coherent organisational
organisational conditions that support taken by case study schools conditions that support evaluation
evaluation and inquiry and inquiry
A collaborative approach to evaluation and reporting
Our vision, values, goals and priorities are is encouraged by involving trustees, leaders and teachers We embed evaluation into the way
developed, planned and implemented as in reviewing student achievement outcomes and we plan and take action to realise our
isolated and disconnected activities rather identifying areas for strategic focus for the next year. vision, values, goals and priorities.
than underpinned by systematic evaluation
We draw on an increasing range of information We use appropriate tools and methods
and inquiry processes.
as a starting point for our regular curriculum/syndicate to gather, store and retrieve a range
We have a limited range of tools and reviews: emerging issues; teacher reflection and feedback; of valid and fit-for-purpose data.
methods to gather, store and retrieve student achievement data; student feedback through
We recognise the importance of student
a range of valid and fit-for-purpose data. surveys and classroom observation; and research.
and community voice and draw on these
We gather feedback from students and Student feedback is actively sought to ascertain the perspectives to identify priorities for
our community on a regular basis but are success of initiatives and issues for further investigation. inquiry and improvement.
not using this well to identify priorities for Several methods for eliciting student feedback are
We have high levels of relational trust
inquiry and improvement. being trialled including focus groups or think tanks
that support collaboration, risk taking, and
and in-class discussions.
Our efforts to change and improve openness to change and improvement.
outcomes for all learners need to be better Leaders take time to introduce new processes to build
We allocate sufficient resources
aligned across the school, building on the trust necessary for engagement in meaningful inquiry.
(for example, time, expertise, staffing)
‘pockets’ of collaboration that already exist. They keep workloads and change processes manageable.
to support change and improvement.

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
We are not yet allocating sufficient Leaders have aligned our inquiry and evaluation activities,
We use systematic, coherent and
resources (for example, time, expertise, for example linking teacher appraisal and inquiry projects
‘smart’ evaluation and reasoning
staffing) to support change and to maximise the impact on what is happening for
processes at all levels of our school.
improvement. identified learners.
We share and disseminate new
Our processes for evaluation and inquiry Professional learning groups provide a forum to discuss
knowledge in ways that promote
tend to be ad hoc and disconnected. and inquire into particular issues. Teachers explore, test
improvement and innovation.
and monitor their responses to their identified issues. New
We have limited ways of sharing and
learnings are shared through whole staff feedback sessions.
disseminating new knowledge and this is
a barrier to improvement and innovation.
 FIGURE 10: CAPABILITY AND COLLECTIVE CAPACITY
How can we build the capability and collective capacity to do and use evaluation for improvement?

FROM TO

We are yet to build the capability Examples of development/improvement We have the capability and collective
and collective capacity to do and actions taken by the case study school capacity to do and use evaluation
use evaluation for improvement for improvement
Staff meetings are professional forums. The focus
Our professional learning is focused on is on building professional capability by teachers We engage in professional learning, mentoring
teachers’ interests and/or the expertise sharing readings, research and practice. and coaching to develop the knowledge,
of available professional development skills and confidence needed for evaluation
Leaders are clear about the steps in the change
providers. We are yet to engage in and inquiry.
process, framing up questions for teachers to ask
professional learning that develops
of their practice and developing tools to support We carefully select and use relevant expertise
the knowledge, skills and confidence
the inquiry process. (internal and external) to build our capability
needed for evaluation and inquiry.
and collective capacity in evaluation and inquiry.
Leaders adopt a more collaborative approach to
We have limited or no access to expertise
working with data by increasing the use of data Our leaders and teachers are data literate.
(internal/external) to build our capability
as a focus for discussions in syndicate meetings; They ask good questions, use relevant data,
and collective capacity in evaluation
conversations between teachers; and within the clarify purposes, recognise sound and unsound
and inquiry.
senior leadership team. evidence, understand statistical concepts,
Our leaders and teachers are starting focus on interpretation, and engage in
Leaders engage with research evidence
to develop data literacy. We do rely on evidence-informed conversations.
and share relevant research with teachers
one or two people to manage/analyse
to make better use of evidence in improving When generating solutions and making
and interpret data.
teaching practice. decisions, we draw on current research
We are not aware of/well informed about evidence about what makes a bigger
Leaders have a clear focus on embedding
current research evidence about what difference for learners.
evaluation and inquiry processes by supporting
makes the biggest difference for learners.
team leaders to work with teachers to unpack Engaging in evaluation contributes to changes
We are not yet confident in using their data; deepening conversations about data in our thinking and behaviour and builds inquiry
evaluation and inquiry processes and practice; de-privatising classroom practice ‘habits of mind’.
for improvement. through use of video; and extending teacher
Participation in evidence-based decision
engagement in professional reading through
Our use of evidence in decision making builds our efficacy and agency.
regular discussions and debate.
making is limited.

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
23
 3. School improvement
for equity and excellence
While every school community’s improvement is unique it can be described under these
four headings:
>> Context for improvement
>> Improvement actions taken
>> Shifts in practice
>> Outcomes for learners.

Context for improvement

Every context is different. It may be that the appointment of a new principal provides the catalyst
for change. It may be that a principal returning from sabbatical with fresh eyes and new thinking is
strongly motivated to tackle previously unaddressed challenges. It may be that external evaluators,
by posing ‘stop and think’ questions, motivate leaders to address low levels of progress and
achievement or review aspects of their curriculum. It may be that the board of trustees initiates
improvement efforts. But any teacher who is curious, open, and aware of what is happening
for students in their school will find opportunities to initiate inquiry and evaluation.
Whatever the context, schools that engage in evaluation for improvement are motivated
to make changes that will have a positive impact on the learning and wellbeing of all their
students, and they are sustained by the belief that they – leaders and teachers – can do better.

Improvement actions taken

Improvement actions are actions that emerge out of evaluation processes.
When deciding how to respond to evaluation findings, an early consideration is whether the school
has the internal capability to forge ahead with the necessary changes or whether it should seek to
engage the support of external expertise, perhaps from another school or a provider of professional
learning and development. Good decisions at this point rely heavily on leaders knowing what it takes
to bring about significant educational change.
Capability building is often high on the list when it comes to improvement actions, to ensure
that leaders and teachers have the skills and knowledge they need to make the desired changes.
Another priority is often to improve how leaders and teachers work together because professional
collaboration is such a crucial aspect of any school improvement endeavour that better outcomes
for students will depend on it.

24 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Shifts in practice
By monitoring the implementation of improvement actions and evaluating their impact, boards,
leaders and teachers come to learn what works or does not work, for which learners, and why.
In the absence of systematic monitoring and evaluation, shifts in practice, and their impact, can
go unnoticed. Even a small shift, in conjunction with other shifts, can increase forward momentum
or contribute to the realisation of a big goal. The biggest shifts are those that penetrate to the core
of teaching practice. Shifts in the conditions that support effective evaluation will contribute to shifts
in teaching practice.
Monitoring allows for real-time adjustments to be made to improvement actions when they are seen
to be not having the intended impact; it also allows for new knowledge to be harvested and used
more widely for improvement purposes.

Examples of shifts in practice

Table 1 provides examples of the kinds of shifts in practice that boards of trustees, leaders and
teachers have made with the express purpose of improving outcomes for their learners. They are
from ERO’s case studies of schools that have effective internal evaluation processes. The examples
are grouped under the six organisational influences on student outcomes (domains) identified in
School Evaluation Indicators – Effective Practice for Improvement and Learner Success.
You could use the examples in the table as a starting point for discussion about shifts that you
are currently making or to inform your thinking about actions you need to consider for a particular
evaluation context.

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 25
 TABLE 1. Examples of shifts in practice that schools have made to improve outcomes for learners

Stewardship
FROM TO

Tabling reports on curriculum review and student Scrutinising and interrogating reports to understand
wellbeing and achievement their implications for decision making

Setting broad targets as a paper exercise Involving teachers in setting appropriate targets
for specific cohorts of students

Focusing on policies and planning Focusing on how well the board is enacting
its stewardship roles and responsibilities

Focusing narrowly on what is happening for students Situating students on a pathway of lifelong learning
while at school

Leadership
FROM TO

Task and budget-focused leaders Lead learners and leaders of learning

Senior leaders monitoring classroom practice Senior leaders mentoring teachers – engaging in
challenging conversations and providing structure
to support teacher reflection

Leaders having little knowledge of recent Leaders keeping up to date with educational research
educational research and using it to help prioritise actions within the school

Expectations of teachers and students not clearly Leaders setting clear expectations of teachers
articulated or consistently implemented and students

Educationally powerful connections and relationships

FROM TO

Relying on public meetings for parents and whänau Personalised communication to parents and whänau,
seeking feedback on the school’s performance
and direction

Accepting that parents and whänau seldom come Making engagement with parents and whänau an
to school events and interviews ongoing priority

Offering limited opportunities for parents and Providing opportunities for parents and whänau to
whänau to find out about their children’s learning communicate regularly about their children’s learning

15% attendance by parents and whänau 85% attendance after introducing a whänau tutor
at interviews system and giving parents and whänau a
20-minute interview with one teacher

26 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
 Responsive curriculum, effective teaching and opportunity to learn
FROM TO

Teacher-directed learning Deliberate acts of teaching in linguistically and

cognitively rich classrooms

Students not being aware of their achievement levels Students knowing where they are at in their learning
or next steps and what their next steps are

Teaching writing Teaching writers

A focus on behaviour management – ‘putting out fires’ A focus on effective teaching strategies and providing
opportunities for all students to learn

Deficit thinking and blaming students for poor outcomes Teachers recognising the need to improve their teaching

Professional capability and collective capacity

FROM TO

Reliance on external support to build capability Building self-sustaining internal capability

Staff meetings focused on administrative and Well-structured professional learning conversations

organisational matters

Appraisal processes in which teachers ‘cherry pick’ Teacher goals linked to improvement actions and
their goals student learning

Evaluation, inquiry and knowledge building and innovation

FROM TO

Data analysis happening at the leadership level Teachers actively engaged in analysing their data
for reporting to the board and using it in their planning and teaching

Informal approaches to evaluation Systematic and manageable evaluation aligned to

school priorities

Evaluation happening as isolated activities A coherent and connected approach to school evaluation

‘We reckon’ ‘We know’ – and have the evidence of improvement.

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 27
 To think about and discuss
>> Which of these shifts in practice resonate with us?
>> What impact might these shifts have for students?
>> What actions might these schools have taken to make these shifts in practice?
>> What can we learn from these examples?

Outcomes for learners

Effective internal evaluation processes enable trustees, leaders, teachers, parents, families,
whänau and the wider school community to better understand:
>> how individual learners and groups of learners are performing in relation to valued outcomes
>> how improvement actions taken have impacted on learner outcomes and what difference
is being made
>> what needs to be changed and what further action needs to be taken
>> the patterns and trends in outcomes over time
>> what kinds of practices are likely to make the most difference for diverse learners and
in what contexts
>> the extent to which the improvements achieved are good enough in terms of the school’s
vision, values, strategic direction, goals and priorities for equity and excellence.

28 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
School improvement journeys: two examples

The following examples outline how two of the case study schools engaged in internal evaluation
with an improvement agenda. In the first example, a school set out on a two-year journey to raise
achievement in a particular area of the curriculum. In the second example, a school embarked
on a much longer improvement journey that had a number of different but related foci.

An improvement journey with a curriculum focus

This first example is of an ‘emergent’ evaluation (see page 13) triggered when their analysis of data
at the mid-year checkpoint in 2012 revealed poor achievement in writing. Key points are described
in the diagram on pages 30-31. Note the context for this evaluation, the links between the evaluation
processes and reasoning, and the actions taken and shifts in practice made. By comparing the data
for 2012 and 2014, the school was able to measure the progress of different cohorts of learners.

To think about and discuss

>> What were some of the organisational conditions that enabled this evaluation?
>> What evaluation capabilities did the leaders and teachers have going into this evaluation?
What capabilities might they have developed by engaging in this evaluation?
>> What relationship can you see between the improvement actions and the shifts in practice?
>> What might have been some of the ‘reasoning’ that contributed to this evaluation process?

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 29
 An improvement journey with a curriculum focus

Context for improvement NOTICING

Analysis of data in mid-2012 revealed that
>> the most recent ERO review (2010)
despite recent professional development
had focused on self review
achievement in writing was poor. Senior
>> The senior leadership team had increased
leaders decided to take a closer look at
capabilities in the areas of self review,
what was happening.
pedagogical strategies, inquiry/reflection
and coaching
>> Student achievement data revealed
issues in writing
>> The quality of planning varied greatly
from teacher to teacher
>> Achievement targets needed to be
realigned following the introduction
of National Standards. INVESTIGATING
Phase 1. Visits to classrooms, looking
at teacher planning and students’ writing,
and talking with students convinced leaders
that an in-depth, school-wide evaluation of the
teaching of writing was needed. “What we
were doing was not getting the results.”
Phase 2. Leaders read up on what the research
evidence had to say about teaching writing.
They listened to the teachers’ theories about
teaching writing. They collected samples of
student writing from across the school and
from the contributing intermediate school.

30 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
IMPROVEMENT ACTIONS
>> Deliberate use of senior leadership team capabilities
>> Collaborative development of the writing progressions led to teachers taking more responsibility
for analysing their own classroom data
>> Workloads and change processes kept manageable
>> Videoing of lessons introduced as a tool to support teachers to reflect on their practice
>> Professional discussions about assessment practices, making teacher judgements, and building
a shared understanding of student progress and achievement
>> Students are given access to the writing progressions to support their learning.

PRIORITISING TO TAKE ACTION SHIFTS IN PRACTICE

The leadership team decided that the priority From To
was to increase teacher pedagogical knowledge.
Teaching writing Teaching writers
They knew that this would involve challenging
existing practice through discussion, coaching Limited use of data Using data to
and guided critical reflection. Expectations for inform teaching
change were clear, and external expertise was A focus on managing A focus on
engaged to facilitate the process. behaviour effective teaching

Blaming external factors Recognising the

need to improve

COLLABORATIVE OUTCOMES FOR LEARNERS

SENSE MAKING A comparison of the numbers of students (2012 and 2014) at or
The leadership team’s above National Standards for writing reveals these percentage shifts:
analysis of the information Whole-school: Up 7% to 52% (131 students)
they had gathered revealed
Students in years 4, 5 and 6: Year 4 up 15% to 56% (24
low levels of teacher
students); year 5 up 13% to 46% (19 students); year 6 up 27%
pedagogical knowledge,
to 62% (30 students)
hugely variable teaching
Mäori students in years 4, 5 and 6: Year 4 up 16% to 52%
practice, and that some
(19 students); year 5 up 20% to 47% (13 students); year 6
teachers attributed poor
up 27% to 62% (30 students)
achievement to factors
outside of the school. Pacific students in years 4, 5 and 6: Year 4 up 25% to 75%
(3 students); year 5 up 10% to 43% (3 students); year 6 up 39%
to 69% (9 students).

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 31
 A longer-term improvement journey

This example is of a longer-term improvement journey (2006–14). This school began with an
investigation into the achievement of Mäori students, followed by an investigation into the
achievement of boys, and then a third investigation into student engagement and wellbeing.
Although the three evaluations each had a different focus they were obviously connected.
As the board of trustees, leaders and teachers inquired into their practice, their individual
capabilities and collective capacity to engage in internal evaluation developed over time.
The diagrams on the next three pages show how, for each of the three inquiries, the school
went through the five evaluation processes described on pages 15-20.

To think about and discuss

>> What were some of the organisational conditions that enabled these evaluations?
>> What evaluation capabilities did the school have going into these evaluations?
What capabilities might they have developed by engaging in this evaluation?
>> What was the motivation for improvement in these examples?
>> How are the three evaluations connected? What do they have in common?
What is different in each?

Developing a
values approach
to student
wellbeing

Raising the
achievement
of boys

Raising
Mäori students’
achievement

32 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Raising the achievement of Mäori students

Overview of evaluation and reasoning processes

Noticing
NCEA results (levels 1 and 2) masked the disparity between Mäori and Päkeha students.
None of the initiatives implemented by the school had changed outcomes for Mäori students.

Investigating
All the available data was closely re-examined.
Using the Effective Teacher Profile as a guide, all year 9 and 10 teachers were observed in the
classroom; observations were followed by feedback sessions.

Collaborative sense making

Co-construction meetings were held once a term with all teachers of year 9 and 10 students.

Prioritising to take action

A commitment was made to sustaining the principles of Te Kotähitanga.

Monitoring and evaluating impact

Ongoing gathering, analysis and use of data at all levels of the school.

Improvement actions

“If we were going to keep doing for our Mäori students what we had always done,
we were going to get what we’d always got – and it was way not good enough.”

Ongoing classroom observations, feedback sessions, co-construction meetings and regular

professional development for teachers.
Modelling discursive teaching strategies: “If you are using discursive teaching strategies
and co-constructing or power sharing the kids are going to enjoy learning, learn, and have fun.”

“We are going to keep working to achieve the target of raising our
Mäori student achievement to mirror the achievement of our non-
Mäori students. We are going to keep going until there is no gap.”

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 33
 Shifts in practice

From To

Having the ‘will’ to make a difference for Mäori students Finding the ‘way’ to make a difference for Mäori students

Traditional approaches to teaching Discursive, relational approaches to teaching and learning

Accepting that whänau are seldom seen at school Whänau engagement is an ongoing focus
events and interviews for improvement

2005 – Major disparity between Mäori and Päkeha NCEA results

Processes and improvement actions

Trial school for Te Kotähitanga
Discursive teaching practices
Focus on relationships
Classroom observations of all year 9 and 10 teachers
Co-construction meetings every term

Students gaining NCEA level 2

100

80

60

40 European/Päkeha
20 students
Mäori Students
0
2005 2014

2014 – The disparity between Mäori and Päkeha gaining NCEA level 2 is reducing

34 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Raising the achievement of boys

Overview of evaluation and reasoning processes

Noticing
At the 2005 prize giving, the new board chair, noticing that the procession of students coming up
for awards consisted mostly of girls, wrote ‘boys’ on a piece of paper and passed it to the principal.

Investigating
Randomly selected boys from across the school’s seven year levels were involved in focus groups
to find out what worked well for them in terms of supporting their learning and achievement,
and what didn’t.

Collaborative sense making

Data from the focus groups was analysed and narratives written. These narratives, together with data
relating to boys’ achievement and boys’ discipline, were analysed and shared with staff. A key theme
in the narratives was ‘boys just want to have fun, too!’

Prioritising to take action

It was decided to make boys’ achievement a school-wide focus.

“What is good for Mäori is good for all, especially boys.”

Monitoring and evaluating impact

Ongoing scrutiny of data through a ‘what is happening for boys?’ lens was put in place. Coupled
with the ongoing implementation of Te Kotähitanga, this initiative has contributed to significant
shifts in boys’ achievement.

Improvement actions
A professional learning group was established. The teachers in this group committed to using
a teaching-as-inquiry approach with four or five boys in their classroom, trying different strategies
and reflecting on what worked and what didn’t. They met regularly to share their reflections with
others in the group.
An external facilitator worked with the staff on ‘schema’, which was about understanding the
different ways in which girls and boys engaged and behaved. This ‘opened the door for staff
to admire boys’ behaviour’.

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 35
 Shifts in practice

From To

A pedagogy focused on positive relationships

Traditional pedagogical practices and discursive teaching practices (‘rainbow
effect’ – Te Kotähitanga)

2005 – Boys’ achievement is behind that of girls

Processes and improvement actions

Professional learning group established to use teaching-as-inquiry methodology
Focus groups set up by English department
Focus groups with boys at each year level
Boys’ narratives and data collected

Boys gaining NCEA levels 1 and 2

100

80

60
Level 1
40

20 Level 2

0
2005 2014

2014 – T
 he percentages of boys gaining NCEA levels 1 and 2 have increased significantly

36 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
Developing a values approach to student wellbeing
In 2011 the school’s approach to behaviour management consisted of a set of expected
behaviours (rules), consequences for breaching the rules, and disciplinary steps.

Evaluation and reasoning processes

Noticing
Leaders found the data on stand downs, suspensions and referrals to the deans’ centre
to be unacceptable.

Investigating
Positive Behaviour for Learning (PB4L) facilitators gathered data to find out how well the school’s
behaviour management approach was known by staff and students.

Collaborative sense making

Analysis of data showed that staff could name only three or four of the expected behaviours
and that these behaviours were not well known by the students.

Prioritising to take action

The ongoing priority is to create a climate where student behaviour is appropriate for the context,
and focused on learning.

Monitoring and evaluating impact

Regular gathering and analysis of data about stand downs, suspensions and referrals to the student
centre enables the school to monitor and evaluate the impact of actions taken.

Improvement actions
The school embarked on an 18-month process
of consultation to identify an agreed set of school
W We are learners – Whaia te mätauranga

values. The views of staff, students, the Parent-

A Act with respect – Manaaki te tangata
Teacher Association (PTA), board of trustees and
wider community were sought. This was a slow K Keep ourselves safe – Tiaki tangata
and deliberate process, ‘sowing seeds’ – it wasn’t a
revolution. Leaders recognised the importance of A Always proud – Kia manawanui
ownership. The outcome of this process was WAKA.
Improvement actions included changing the
name of the deans’ centre to ‘student centre’;
using a ‘stop the bus’ mechanism, every teacher
taught the same WAKA lessons on how to recognise
good behaviour; and the school’s reward system “Every developmental step was collaborative and
was redeveloped with the introduction of WAKA reflective – a slow but effective progress owned
cards to acknowledge WAKA behaviour. collectively by leaders, staff and students.”

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 37
 Shifts in practice

From To

A discipline system based on poorly understood A values-based approach ‘lived’ in every area
rules and consequences of school life

2011 – Student wellbeing, stand down and exclusion data unsatisfactory

Processes and improvement actions

Involvement in PB4L
Data gathered about how well rules and consequences were understood by staff and students
Deliberative development of values

2014 – T
 he numbers of stand downs and suspensions have fallen progressively over the
last 4 years

2011 2012 2013 2014

Stand downs 111 75 67 47

Suspensions 42 25 22 15

38 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
4. Using the school evaluation indicators
to support continuous improvement

ANAUNGATANGA
WH

LE
AD
Responsive

ER
P
HI

curriculum,

SH
DS

effective teaching

IP
AR

and opportunity
STEW

to learn

MAN
Learners
MAHI TAHI

AAKITANGA
Educationally
powerful
connections and
relationships
EV

ON
AL

TI
VA
UA

AK O
NO
TI
O

IN
PR
N

IN O TY
,

AN F E S SIO N A L C A PA BILI
D

Q AN
UI DC
RY OLLEC CITY T
TIVE CAPA N
AN ME
D
KN VE
OW RO
LEDG IMP
E B UI LDIN G FO R

The school evaluation indicators identify practices that contribute to effective evaluation. These
practices should be viewed holistically, not used in isolation or as a checklist. Use them as a ‘sieve’
or trustworthy reference when trying to make sense of data and to answer the question, How good is
our practice? Use them too when investigating the relationship between different – and, sometimes,
apparently unconnected – aspects of your school’s performance.
Table 2 provides a broad framework for thinking about how you might use the indicators in your own
school. Possible uses are suggested in relation to each of the five evaluation processes identified
and discussed in this guide.
The indicators can be used in different ways during different phases of internal evaluation.
For example:
>> Use the outcome indicators as the starting point for evaluating the educational opportunities
that your school offers its students and investigating who is/is not achieving, and identifying
those students whose progress needs to be accelerated
>> Use the six domains of influence as a framework for identifying where you might concentrate
your inquiry and data gathering
>> Use the process indicators and associated effective practice statements as a framework when
trying to make sense of your data and identify and prioritise improvement actions
>> When monitoring and evaluating the impact of your improvement actions, the outcome indicators
can help you maintain focus on what is happening for all learners.

The indicators for Domain 6, Evaluation, inquiry and knowledge building for improvement
and innovation, and the associated effective practice statements can be used to evaluate the
effectiveness of your school’s approach to evaluation.

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 39
 TABLE 2. USING THE SCHOOL EVALUATION INDICATORS

Process Possible use Evaluation reasoning:

of the indicators questions we might ask

Outcome indicators focus What does our information tell us about the extent to
evaluation on the learner. which all learners are achieving high-level curriculum
They can assist in mapping outcomes: confidence in identity, language and culture;
Noticing
the context and identifying wellbeing; achievement and progress; and participation
what matters most in terms and contribution? Which learners are/are not achieving,
of student outcomes and in what contexts?

Domains/process indicators Considering the six domains of influence, where do we

can assist in focusing inquiry need to focus our evaluation? What do we know and how?
Investigating
and data gathering What further information do we need? How might we find
out? What questions do we need to answer/focus on?

Domains/process indicators What is our evidence telling us? Do we know enough

and the associated effective about the effectiveness of our practice to determine
Collaborative practice statements can what we need to improve and where we need to go next?
sense making assist with sense making What else can we use/draw on that will assist us to plan
and identifying areas for next steps?
an improvement focus

Domains/process indicators What is the most important action we need to take

Prioritising to can assist in determining and why?
take action priorities and taking action To take action, what kind of support do we need
to draw on or put in place?

Outcome indicators How are we going? What is our data/information telling us?
Monitoring
provide the starting point Do we need to modify or change what we are doing?
and evaluating
for monitoring and evaluating What impact is our action having? Are we providing
impact
the impact of actions taken opportunities for all learners to learn, progress and achieve?

40 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
5. Integrating internal and external
evaluation for improvement
School evaluation has both internal and external components, and the two should be seen
as working together.

New Zealand has ... gone furtherest among countries internationally towards a collaborative
school evaluation model, incorporating at the same time a sequential process ... [the] approach
is collaborative in the sense that both parties attempt to work together to agree on a rounded
picture of the school in which there is mutual recognition of its strengths and consensus on
areas for development.10

Schools set in place their own evaluation processes and, as part of an annual reporting cycle,
provide regular accounts of student achievement in relation to goals and targets, along with planned
improvement actions.
Then periodically ERO conducts an external evaluation. The school’s student achievement information
and associated internal evaluation forms the starting point for ERO’s external evaluation and for
engagement with other education agencies and professionals.
School Evaluation Indicators: Effective Practice for Improvement and Learner Success expresses
the intent of ERO’s external evaluations in this way:

Participation in periodic external evaluation supports the school’s cycle of continuous

improvement through providing an external lens on the school’s improvement journey: its
performance in relation to excellence and equity of outcomes for every student, the extent
to which school conditions support ongoing improvement; and next steps for development.
(ERO, 2015, p6)

The indicators are designed to make it easier for internal and external evaluations to talk to each other:

The indicators provide a common language for the interaction and dialogue between a school
and ERO about the development since the last review, the current state and future direction.
(ERO, 2015, p6)

Domain 6 of the indicators framework, Evaluation, inquiry and knowledge building for
improvement and innovation is the engine that drives improvement. The indicators in this
domain highlight the relationship between organisational conditions and the capacity to do and
use evaluation for improvement, and how these influence engagement with external evaluation.
In effective schools internal evaluation processes are systematic, coherent and connected at all
levels of the school. This alignment ensures that leaders, teachers and boards of trustees are able
to purposefully engage with external evaluation, using it as an opportunity to review, validate and
support their own improvement actions.

10 Nusche, D., Laveault, D., MacBeath, J., & Santiago, P. ( 2012). OECD Reviews of Evaluation and Assessment in Education: New Zealand 2011,
OECD Publishing (p.105).

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 41
 Getting the most out of external evaluation
In thinking about and preparing for your ERO external evaluation the questions below provide
a useful framework for developing an account of your school’s development since the last evaluation,
where that development is currently at, and areas for future focus.

Outcomes for learners

What are the outcomes that are valued for all learners in this school community, as learners in Aotearoa
New Zealand, and as global citizens?
How well are all our learners achieving in relation to those outcomes?
To what extent is every student in our school a successful “confident, connected, actively involved,
lifelong learner”?
How well are we identifying and accelerating the achievement of those learners at risk of not achieving
equitable outcomes?
How do we know? What sources of evidence tell us about our performance and effectiveness?

School conditions supporting ongoing improvement in learner outcomes

Since our last ERO evaluation:

What areas of development have we focused on to improve learner outcomes?
How have we enabled the school community to participate in/contribute to the development focus?
What actions have we taken? How effective have those actions been in promoting the improvements
needed?
What have been the successes and challenges? Which domains of school activity have been most
significant in supporting our improvement journey (stewardship; leadership; responsive curriculum,
effective teaching and opportunity to learn; educationally powerful connections and relationships;
professional capability and collective capacity; evaluation, inquiry and knowledge building for improvement)
What has been the impact?

What are our continuing and/or next big areas for development and improvement?

This will provide an overview of your internal evaluation findings to present to ERO. ERO uses your
school’s learner outcome information and internal evaluation account to work with you to design
the external evaluation for your school context.
During the evaluation process the ERO team will discuss the emerging evaluation findings with
you and involve you in the interpretation of the evidence gathered to develop a shared understanding
of the quality and effectiveness of education provision in your school and next steps.
The evaluation insights provided by ERO as an outcome of the external evaluation process should
support every child to succeed as a lifelong learner.

42 EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT
 School evaluation for equity and excellence

N IN
IO T
E
T

R
A

N
U

AL
EVAL

Learners

EVAL
AL
N

U
R

A
E T
T IO
EX N

EFFECTIVE SCHOOL EVALUATION: HOW TO DO AND USE INTERNAL EVALUATION FOR IMPROVEMENT 43