Australian Council for Educational Research

ACEReSearch
2005 - Using data to support learning Research Conferences

1-1-2005

Data-driven school improvement through the VCE

Peter Congdon

Recommended Citation
Rowley, Glenn and Congdon, Peter, "Data-driven school improvement through the VCE data
service" (2005).
http://research.acer.edu.au/research_conference_2005/12

This Conference Paper is brought to you for free and open access by the Research Conferences at ACEReSearch. It has been accepted for inclusion in
2005 - Using data to support learning by an authorized administrator of ACEReSearch. For more information, please contact [email protected].
Data-driven school improvement through
the VCE Data Service
Abstract
As the holder of student achievement
data spanning three sectors and four
levels, the Victorian Curriculum and
Assessment Authority (VCAA) has a
responsibility to provide these data to
schools in ways that enable school staff
to use them effectively and easily.
Glenn Rowley Peter Congdon
With the discontinuation of the
Victorian Curriculum and Victorian Curriculum and
publication of school achievement
Assessment Authority Assessment Authority
indices, the VCAA was forced to
confront a range of issues surrounding
Glenn Rowley has been General Manager for Peter joined the Victorian Curriculum and
Policy, Measurement and Research at the Victorian Assessment Authority in 2002 as the Manger of
the question of which data belonged to
Curriculum and Assessment Authority since Educational Measurement. Prior to this the student, which was the property of
February 2002. He began his career as a appointment he was the Head of Assessment the school, and which belonged to the
secondary teacher in Victorian schools before Services Psychometric Support Team at the general public. In 2002, a new balance
undertaking graduate studies in educational Australian Council of Educational Research, where was struck. A key component in this
measurement and evaluation at the University of he worked for 11 years. Peter’s main focus of
balance was the introduction of the
Toronto, where he completed a Masters degree work has been in large-scale testing, involving data
in 1972 and a PhD in 1975. analysis, equating and reporting of educational VCE Data Service.
outcomes at individual and group levels. The VCE Data Service is an online
Following lecturing appointments at the University
of Toronto and La Trobe University, he joined the Peter is a member of the National Measurement service that connects schools to the
staff of the Faculty of Education at Monash Advisory Group of the Australian Government. entire VCE data set going back to 1998,
University in 1983, lecturing in research He has participated in educational standard
and provides them with the capacity to
methodology and school assessment. He was setting exercises for the Malaysian Government.
appointed Associate Professor in 1990, Associate He has studied psychometrics at the University of
generate a range of analyses related to
Dean (Research) in 1991 and Associate Dean Chicago along with various statistical analysis and their own school, and how its results
(Staff) in 2001. He has been a member of the educational measurement courses at The compare to those of other schools in
Scaling Committee of the Victorian Tertiary University of Melbourne. Peter specialises in data the State, schools in the same sector
Admissions Centre since 1994, and a member of analysis techniques including Rasch measurement, (government, Catholic and
the ACER Council and Board of Directors. multi-faceted analyses, test equating and
Independent), and to schools in its Like
differential item functioning.
Since joining the VCAA, Glenn has had major School Group.
involvement with the delivery of the AIM testing Peter has produced a number of book chapters
program, the changed arrangements for the and journal articles in educational measurement Through VCEDS, senior management
publication of VCE data and the development and along with numerous presentations at national teams in each school can easily
implementation of the VCE Data Service. He is and international conferences. generate charts and tables addressing a
currently working on the development of a new
range of questions including the
‘Like Schools’ measure.
following:
• How were our school’s overall
results this year? Were they better
or worse than in previous years?
How do they compare with schools
like ours?
• How did our students perform,
study by study in terms of
completions, Study Scores,
examinations and school assessment
grades?

Research Conference 2005

46
• How did our school’s results privacy and the public’s right to know. achievers than when placed in a class of
compare to reasonable The issue became particularly acute in lower achievers. In schools in which
expectations? Did our students 2002, in relation to the publication of students are generally of high ‘ability,’
perform as well as students of data on school performance in the end- students do perform better than would
comparable ability in other schools? of-schooling Victorian Certificate of have been predicted from their ability
• Is our student cohort changing Education (VCE). alone. In schools in which students are
over time, in ability and/or generally of lesser ‘ability,’ students do
achievement? Public reporting of the perform less well than would have been
predicted from their ability alone.
• How can we develop better VCE, 1996–2001 Hence we should have expected to find
understandings of the patterns of
From 1996 until 2001, the Victorian higher achievement indices in schools
group performance by identifying
Board of Studies and its successor, the with higher ability students. And, year
how individuals contribute to those after year, that is what was found.
VCAA, had provided information to
patterns?
newspapers to facilitate the publication The six-year adventure with
This paper provides an account of the of tables documenting VCE performance achievement indices came to an end in
first two and a half years of the VCE by school. Conscious of differences in 2002.The VCAA took a view similar to
Data Service, outlining the user school intake, it was decided in 1996 that of Goldstein, who had observed, in
feedback and the VCAA response to it, not to publish raw achievement data, relation to the value-added measures
and the growth in usage over the but instead to publish an index, in which that he had pioneered, that:
period. It concludes with lessons raw achievement is adjusted according
learned and plans for the future. to students’ performance on another set their use as public accountability
of measures, the General Achievement measures, e.g. in the form of
When educators study their performance tables or ‘value
Test (GAT).
schools and classes, they seek an added league tables’ is
answer to the ageless question: Is The GAT is administered in June of the inappropriate and would destroy
it good because we’ve been doing same year, and its key purpose is to their credibility and usefulness. If
it for a long time, or is it good enhance the quality control measures they were ever to become ‘high
because we have tangible evidence on VCE assessment. It is presented to stakes’ pieces of information like
of its worth? In many instances students as a measure of their ‘general the current DfEE league tables of
one must conclude the former examination results, then they
knowledge and skills in the areas of
because no evidence exists to would inevitably become distorted
written communication; mathematics,
support the latter (Johnson, 1997). and no longer reflect any
science and technology; and humanities, underlying reality of school
Because of its unique role in Victorian arts and the social sciences.’ performance (Goldstein, 1999).
education, the Victorian Curriculum and
The achievement indices were created In arriving at this position, the VCAA
Assessment Authority (VCAA) has an
using a multiplicative model in which was forced to confront a range of
important role to play in assisting
the three components of the GAT issues surrounding the question of
schools in their efforts to become
were included at the level of the which data belonged to the student,
more effective.The VCAA is the holder
student, but school GAT means were which was the property of the school,
of data on student achievement that
not. It had very different properties to a and which belonged to the general
span three sectors (government,
measure of growth that could have public. In 2002, a new balance was
Catholic, Independent) and four levels been developed had the full multilevel
(Years 3, 5, 7 and 11–12), and at each struck. A key component in this balance
modelling described by Goldstein was the VCE Data Service.
level it has a responsibility to provide (1995, 1999) and Goldstein, Huiqi, Rath,
these data to schools in ways that & Hill (2000) been used.There are
enable school staff to use them Public reporting of the
many reasons for this.
effectively and easily. VCE since 2002
It is well known that a student of a
Like other jurisdictions,Victoria has had given ability is likely to perform better From 2002, the achievement indices
to deal with issues of accountability, when placed in a class of higher were discontinued. Instead, a wide

Using Data to Support Learning

47
range of information is published, Since 2004, the professional • How were our school’s overall results
including information about: development program has been this year? Were they better or worse
followed by a series of consultations in than in previous years? How do they
• school programs (enrolments in
which VCAA staff have met with school compare with schools like ours?
VCE and VCAL, range of VCE and
leadership teams for confidential advice • How did our students perform,
VET studies offered)
on the analyses they can generate from study by study? Are there important
• student achievements (satisfactory their own school data.To provide this differences among groups within the
completions in VCE,VCAL and VET assistance,VCAA staff have travelled to school (e.g., gender, class groupings)
units, median study scores, all corners of the State, meeting with and between exams and
percentage of high achievers) staff from more than 100 schools in coursework assessments?
• student pathways (percentage applying each of the past two years.
• How did our school’s results
for and achieving tertiary selection and
compare to reasonable
various employment options). What does the VCE expectations? Did our students
The information that is made public is Data Service provide to perform as well as students of
the information that the public can use schools? comparable ability in other schools?
to make decisions about schools. • Is our student cohort changing over
The VCE Data Service does not provide
Further, more detailed information is, of time? Can we detect trends in
schools with a predefined set of reports.
course, available from school web sites. student ability from year to year, and
Instead, it puts schools in touch with the
entire VCE data set going back to 1998, if so, are these changes matched by
The introduction of the and invites users to generate reports in changes in student performance
VCE Data Service response to the issues and research over the same period?
questions that address their concerns. • Can we develop better
The detailed information needed by
They can, of course, only draw out understandings of the patterns of
schools in their planning for school
analyses related to their own school, group performance by identifying
improvement is now made available
and how its results compare to those of how individuals contribute to those
through a new online service known as
other schools in the state, schools in the patterns? Have the results of
the VCE Data Service (VCEDS).This
same sector (government, Catholic and particular students or groups of
service was launched in November
independent), and to schools in its Like students distorted overall patterns
2002, following several years in which a School Group. of achievement?
limited form of ‘value-added’VCE
reporting had been successfully trialled As in the trial that preceded it, VCEDS School staff are also able to design their
in a sample of schools.The reports provides ability-adjusted estimates in own analyses by entering data fields
available to schools through VCEDS which adjustment is made in each unique to the school and so introducing
include, but are not limited to, the study for: comparisons tailored to their specific
ability adjusted measures pioneered in • each of the three component GAT needs. For example, one school might
the VCE Data Project. scores (Written Communication, like to see results broken down by
Mathematics/Science/Technology campus, another by language
The information provided by VCEDS is background, and another by residential
confidential to the school, and access to and Arts/Humanities)
location. An inventive use by one school
it is strictly controlled to protect its • the mean GAT score of all students
has been to monitor the long-term
privacy. Professional development in the school taking the study, and
effectiveness of an innovative middle
programs for teachers are provided in gender.
school program by comparing VCE
February/March each year, focusing on But this is only a small part of the performance several years later.
the range of reports available to schools information provided.Through VCEDS,
and how they can be obtained and Comparisons between the performance
senior management teams in every
interpreted. Each year, these sessions school can easily generate charts and of males and females are, of course,
have been attended by teachers from tables addressing questions like routinely provided, along with those
between 200 and 300 schools. the following: between class groupings and student

Research Conference 2005

48
year levels. Schools can use VCEDS to • more intuitive reporting of adjusted In summary, users have drawn most
monitor the performance of Year 11 estimates, and heavily on the following report types,
students taking accelerated VCE • capacity to study the performance listed in the logical sequence that users
programs, and use their findings to of individuals and their contribution might have approached them, rather
assess the wisdom of the advice that to reports on overall performance. than their frequency of usage, which is
students are receiving as they approach documented in Table 1 (over the page):
their VCE studies. These and a number of other
enhancements were included in a major • All VCE Studies (overall Study Score
For each report, VCEDS provides users enhancement of the system in time for distribution)
with a choice of a graph, a table, or the 2004 data upload. • Study score distributions, showing
both.These reports may be sent directly multiple studies
to the printer, or pasted into another User response to these improvements
has been monitored in the course of • Study score and assessment grade
application, such as a Word document.
27 group professional development distributions (breakdowns within a
In this way, schools can use VCEDS to
report publicly to parents, or sessions attended by 617 senior school single study)
confidentially to department heads or staff and 102 individual school • Study score distributions, plotted
individual teachers. With the information consultations in 2005, and has been over time
provided through VCEDS, schools are overwhelmingly positive.
• Adjusted score distributions,
able to examine all aspects of their Total usage has grown steadily since the showing multiple studies
students’ achievements in a more release of the 2004 data in the first • Adjusted score distributions
detailed way than ever before, and to week of February 2005. Figure 1 shows (breakdowns within a single study)
use the knowledge they gain to identify the cumulative hits recorded on the
• Adjusted score distributions, plotted
problems and build upon their strengths. first days of February and each month
over time
until June.The most heavily-accessed
Schools’ experience report to this date has been Report 17 • Scatterplots of achieved versus
(Student Results by Study), which expected study scores
with the VCE Data
shows a scatterplot of achieved versus • Results for individual students,
Service predicted Study Scores for each study, across studies.
The VCE Data Service in now in its third with 1692 hits in the four-month
year of operation. After the first full period. Another seven reports have
year, evaluation largely focused on the exceeded 1000 hits in this period.
question ‘How can we make this
better?’ During 2004, feedback was
18000
systematically sought, and the key
16000
improvements sought included:
14000
• the capacity to generate detailed
Cumulative hits

12000
reports on assessment grades and
scaled (VTAC) ENTER Subject 10000
Scores as well as Study Scores in 8000
each study; 6000
• access to student results by Home 4000
School and/or by Assessing School
2000
(for schools that share teaching
0
programs); 1 February 1 March 1 April 1 May 1 June
• capacity to trace GAT scores over Date
time and their relationship with
achievement scores Figure 1 Total VCEDS usage Feb–June 2005

Using Data to Support Learning

49
 Table 1 Cumulative VCEDS usage by report type: Feb-June 2005

Report type Mar 1 April 1 May 1 June 1

1. All VCE Studies (Study Score distributions) 296 740 876 1080
2. Assessment Grades (frequencies of each) 188 482 584 722
3. Study Score frequencies 104 265 330 418
4. Completions (Units 3 & 4 & qualifying for a Study Score) 65 160 200 268
5. Study Scores of 40 and above, over time if required 149 374 522 660
6. Location of school within Like Schools Grouping 70 186 233 325
7. Multiple VCE Studies (Study Scores) 284 804 949 1167
8. Single VCE Study (Study Scores and Graded Assessments) 242 769 922 1198
9. Single VCE Study (Study Scores over time) 246 728 895 1152
10. Multiple VCE Studies (adjusted scores) 250 678 802 1016
11. Single VCE Study (adjusted scores) 170 514 601 752
12. Single VCE Study (adjusted scores by time) 233 717 866 1087
13. Single Study adjusted scores (subgroup comparisons)) 179 521 611 796
14. Single Study (achieved versus predicted scores) 218 602 723 949
15. GAT and Study Score distributions (single studies) 178 511 613 795
16. GAT and Study Score distributions over time 155 395 494 699
17. Student results by Study (scatterplots of achieved versus predicted) 338 1101 1307 1692
18. Individual student results across Studies 298 781 895 1116

The ‘hit count’ records each time a user constantly finding new ways of using the part of their limited professional
chooses that item from the next higher- information that they now have the development funds on buying in the
level menu. For each hit, many reports power to access. expertise that will enable them to
can be produced.The usage reports do become self-sufficient, and in our view
But empowerment can be resource-
not enable us to record the number of this would be money well spent,
intensive. Over 2004 and 2005,VCAA
actual reports accessed, printed or particularly if it leads to self-reliance
staff have conducted 50 Professional
pasted into other documents, nor can and not to continuing dependence on
Development seminars in all areas of
we determine the number of schools assistance from outside the school.
Victoria, with attendance totalling 1600
accessing the service. Changes to the
senior school staff. In addition, they have There are a number of issues that the
system currently in train will provide
held 236 individual consultations with VCAA needs to address. Some of these
the latter information.
staff from 161 different schools. Over are relatively minor, such as how many
time, the need for this level of support years of data the system should retain
Lessons learned from may decline, but with an influx of new for everyday access. User feedback is
the VCE Data Service users each year, it is unlikely to go away. telling us that the current seven years is
more than sufficient, and that when the
The VCE Data Service is built on a Beyond the existing support program,
2005 results are uploaded, one or more
philosophy of empowerment. It the capacity of VCAA staff to support
years could be dropped.
challenges its users, and those who rise schools to provide individual support is
to the challenge are excited by the limited. Many schools are comfortably A vital issue for the VCAA to resolve is
capacity that they gain by using it. self-sufficient already, and others can the uneasy compromise between the
Feedback from users has been quickly become so with minimal desire for ease of access and the need
consistently positive, and schools are support. Schools may wish to spend to maintain the confidentiality of the

Research Conference 2005

50
data.To protect confidentiality, access APPENDIX
has been restricted to the Principal of
each school, the VASS coordinator, and Sample reports from the VCE Data Service
other users as nominated by the
Principal.This may be over-restrictive,
particularly in schools where the
Principal is not aware of the service or
nervous about security issues.The
VCAA is planning consultations with
user-Principals about the possibility of
setting up different categories of users
with access to different reports within
the service. For example, we could have
the category ‘Super User,’ with access to
everything; a ‘Department Head’
category, with access to reports for a
defined set of Studies (such as a Key
Learning Area), and a ‘Subject Teacher’
category, with access to one Study only.
These and a number of other issues will
be resolved by consultation with users
and implemented in time for the upload
of the 2005 results in January 2006.

References
Goldstein, H. (1995). Multilevel
statistical models (2nd ed.) London:
Edward Arnold.
Goldstein, H. (1999). Using value-added
data for school improvement
purposes. Oxford Review of Education,
25, 469-483.
Goldstein, H., Huiqi, P., Rath,T., & Hill, N.
(2000). The use of value-added
information in judging school
performance. London: Institute of
Education. Available at:
http://www.ioe.ac.uk/hgpersonal/
Using-value-added-information.pdf
Johnson, J. 1997. Data-driven school
improvement, ERIC Digest, No. 109.
Available at: http://www.ericdigests.org/
1997-3/data.html

Using Data to Support Learning

51
 Research Conference 2005

52
Using Data to Support Learning

53
 Research Conference 2005

54
Using Data to Support Learning

55
 Research Conference 2005

56
Using Data to Support Learning

57