Moderate Assessment PDF
Moderate Assessment PDF
Moderate Assessment PDF
assessment
Department of Education
Moderate assessment
A module of the Advanced Certificate: Education (School Management and Leadership)
For any reuse or distribution, you must make clear to others the license terms of this work.
Any of these conditions can be waived if you get permission from the Department of Education.
Department of Education
Sol Plaatje House
123 Schoeman Street
Tshwane
South Africa
http://www.education.gov.za
1
How does the Department of Education define commercial use? A commercial use would involve the assessment of a direct or indirect fee by a
for-profit entity for use of the Department of Education Creative Commons (CC) materials, or any derivation or modification of the Department of
Education CC material, or any other commercial exploitation of the Department of Education CC materials.
Moderate
assessment
Acknowledgements ............................................................................................................................. 6
Acronyms and abbreviations used in the programme ............................................................................. 7
Word of welcome .................................................................................................................................. 9
Quality education ................................................................................................................................... 9
Module focus......................................................................................................................................... 9
Module outcomes.................................................................................................................................10
The structure of the module..................................................................................................................11
Learning time........................................................................................................................................11
List of tables
Table 1: Life Orientation (Grades 7 to 9) ................................................................................................19
Table 2: Moderators RPL Checklist.......................................................................................................31
Table 3: Evidence checklist ...................................................................................................................35
Table 4: Life Orientation Exemplar (Grades 7 to 9).................................................................................39
Table 5: Analysis of assessment instrument..........................................................................................41
Table 6: Rubric: Values and Human Rights Research.............................................................................49
Table 7: Quality Management Models...................................................................................................59
Table 8: Criteria for Quality Systems .....................................................................................................60
Table 9 : SWOT Analysis.......................................................................................................................64
Table 10: Outcomes-based assessment................................................................................................89
Table 11: Moderators RPL Checklist...................................................................................................107
Table 12: Evidence checklist ...............................................................................................................108
Table 13: SWOT Analysis....................................................................................................................109
List of figures
Figure 1: Normal distribution curve indicating that the level of assessment was appropriate..................21
Figure 2: Right skewed bell curve indicating assessment was perhaps too easy ...................................22
Figure 3: Left-skewed bell curve indicating assessment was perhaps too hard (Unisa 1981:71-72)........22
Figure 4: Social Sciences Memorandum................................................................................................48
Figure 5: Values and Human Rights Rubric............................................................................................49
Figure 6: Moderators rubric..................................................................................................................50
Figure 7: Rubric for open-ended math problems....................................................................................97
Figure 8: From the ACTFL Spanish Proficiency Guidelines.....................................................................99
Figure 9: From A Canadian Social Studies Essay Exam........................................................................100
Figure 10: Writing Rubric.....................................................................................................................101
Acknowledgements
Cooper King
Department of Education Education Management and Governance Development
Directorate
GMSA Foundation
Management of Schools Training Programme (MSTP)
Matthew Goniwe School of Leadership and Governance (MGSLG)
National Association of Professional Teachers of South Africa (NAPTOSA)
National Association of Teacher Unions (NATU)
Regenesys
South African Democratic Teachers Union (SADTU)
South African Institute for Distance Education (SAIDE)
South African Principals Association (SAPA)
Shuttleworth Foundation
Ukukhula Projects
University of Fort Hare
University of South Africa (UNISA)
University of Stellenbosch
University of the Free State
University of the Witwatersrand
Zenex Foundation
as well as a number of individual consultants, and reflects feedback from personnel from all
the Higher Education Institutions involved in developing this course.
Foreword
Acronyms and abbreviations used in the programme
AC Assessment Criteria
ACE Advanced Certificate: Education
AGM Annual General Meeting
CCFO Critical cross-field outcome
CHE Council on Higher Education
DoE Department of Education
DSG Development Support Group
EMD Education Management Development
ETQA Education and Training Quality Assurance body
HEQC Higher Education Quality Committee
INSET In-service Education and Training
IQMS Integrated Quality Management System
NCS National Curriculum Statement
NLRD National Learners Records Database
NQF National Qualifications Framework
OBA Outcomes-Based Assessment
OBE Outcomes-Based Education
PGP Personal Growth Plan
PoE Portfolio of Evidence
RPL Recognition of Prior Learning
SACE South African Council of Educators
SAQA South African Qualifications Authority
SAUVCA South African University Vice-Chancellors Association
SDT Staff Development Team
SGB School Governing Body
SGB Standards Generating Body
SMT School Management Team
SO Specific Outcome
US Unit Standard
Overview
Word of welcome
Welcome to every student who has registered for this module on Moderation of
Assessment. We trust that this module will provide you with the knowledge and
skills you need to moderate assessments and to manage moderation processes
at your school. We wish you every success in completing the module.
Quality education
Gone are the days when educators were respected and even admired just
because they were educated. Nowadays educators have to earn peoples
respect and they are held accountable for the way they teach, assess, and
manage institutions and people. The spirit of the time is one in which every
teacher has to prove her/his worth and has to submit her/himself and the
institution where s/he works to various quality control measures and processes.
It is this checking process that is the focus of this module and, because all of
you have already had some experience of these processes, much of the work
done in this module might be familiar to you. The purpose of this module is not
simply to repeat what you have already learnt experientially. Rather, it is aimed at
stimulating you to think critically about your own and others checking/moderation
practices and to apply whatever new insights you gain from such reflections, to
your own practice and/or context.
Module focus
The focus of this module is not only on moderation but also on the management
of moderation processes at subject, department and school level. The current
emphasis on moderation is the result of an increased emphasis on accountability
| 10 | ACE SML | MODERATE ASSESSMENT
As you know, South Africa has opted for an outcomes-based education and
training system. In such a system, where the emphasis is on the demonstration
of competence rather than on discrete knowledge or skills, reliable assessments
are of crucial importance. In the past, the various departments of education
accepted responsibility for ensuring that assessments were reliable, valid and
fair, with the results of the external Grade 12 examination being the prime
indicators of quality teaching and learning. Just watch the newspapers each year
when the Grade 12 results are published. Every one of them has something to
say about the drop in subject averages or the percentage of learners who fail
and/or get distinctions. Although this type of verification is still important in an
outcomes-based system, it is not the only form of verification any more. Every
teacher, every subject head, every head of department and every school principal
also has to accept responsibility for ensuring that school assessments are
rigorous i.e. of a sufficiently high standard yet fair, and that the teaching and
learning that precedes assessments provide learners with the necessary
knowledge, skills and attitudes that will enable them to demonstrate the
competence required in culminating assessments. Because of this it is important
that all educators, but especially educational managers, acquire the knowledge
and/or expertise required to establish and maintain systems and processes that
reflect best practice. Moderation is one such process, and this module is aimed
at assisting current and aspiring educational leaders and managers to develop the
requisite expertise in this area.
Based on the assumption that everyone who is registered for this module is an
educator and that all of you are aspiring to be educational leaders and managers
some day, the learning content has been carefully selected to reflect educational
situations where a critical understanding of quality assurance, standards and
moderation is crucial.
Consequently, the activities and assignments in this module will require you to
apply everything you learn in establishing and maintaining moderation systems
and processes to your own school.
Module outcomes
By the end of this module you should be able to demonstrate:
An understanding of moderation within the context of an outcomes-based
assessment system, particularly as it relates to schools
The ability to plan and prepare for moderation
The ability to conduct moderation and to manage moderation processes and
procedures
The ability to record and report moderation results in ways that support and
develop educators ability to plan and conduct quality assessments
The ability to manage moderation at various institutional levels in ways that are
aligned with the policies of the department of education
The ability to conduct reviews of own and others moderation systems and
processes.
OVERVIEW | | 11 |
Learning time
This module carries 10 credits. It should, therefore, take the average student
approximately 100 hours to successfully complete the module. The 100 hours
includes contact time, reading time, research time, time required to discuss
insights with colleagues and fellow students, time to plan and conduct
assessments and time to write assignments. A more specific indication of time to
be spent on each of these activities will be provided in each of the units that
make up this module.
Moderation
Unit 1:and
Moderation and
Outcomes-Based Education
Outcomes-Based
Education
Introduction
Unit 1 outcomes and assessment criteria 1.1
What is moderation? 1.2
Criterion referencing 1.3
External and internal moderation 1.4
Conclusion 1.5
Moderation and Outcomes-
Based Education
1.1 Introduction
The quality of education is continuously being assessed all over the world. Just
open a newspaper and you will probably find some or other person offering
his/her view about the quality or standard of education in our own country.
Some people judge quality in terms of Grade 12 results, some by safety and
security in schools, some by learner or educator behaviour, some by all of these.
Debates about quality and standards are conducted every day, everywhere and
about everything: the quality of products, the quality of service, the quality of
educators, the standard of living, standards for political behaviour, the standard of
education, etc. Bureaucrats have written documents detailing quality assurance
processes and procedures; academics have researched notions of quality, and
practitioners have devised various systems and/or strategies to promote and
maintain quality.
One of the ways in which education authorities attempt to ensure the quality of
education is by the development of policies that guide teaching, learning and
assessment practice. The development of curriculum frameworks, the
formulation of standards for the development of resources and the regular re-
training or upgrading of educators all contribute to the maintenance or
improvement of quality teaching, learning and assessment.
There are various ways in which the quality of assessments could be monitored.
This module deals with one of these strategies, namely moderation. In this unit,
which forms the basis for the rest of the module, the term, moderation, and
other key concepts used in moderation discourse are clarified. Moreover, some
of the frames of reference that inform moderation processes are also briefly
discussed and the relationship between quality assurance and moderation in an
outcomes-based context is introduced with a view to further discussions in
subsequent units.
Have you ever been an external marker for end-of-year Grade 12 examination
1a papers? If you have not, find a colleague who has and ask him/her to tell you how
a typical marking session proceeds. If you have, reflect for a moment on the
processes or procedures followed during a typical marking session. What happens
first, second, third, etc.? Do you think these steps are necessary? Why/why not?
Bearing in mind your experiences or the result of your enquiries, complete the
15 min paragraph below.
How do classroom and school assessments differ from the process you have just
described? Which of the two do you think is most likely to result in assessment
that is fair and of a high quality? Why is this the case, do you think? Do you think
that there is a place for moderation in primary schools? Why/why not?
We know from experience that not all teachers/markers apply the same standards
when marking papers and/or judging performance: some mark very strictly, others
very leniently; some, especially in essay marking, are more subjective, others
more objective. Because of this variation in standards, the person who is being
assessed can never be absolutely sure what s/he is really worth in terms of her/his
demonstration of competence.
In the text box that follows you will find a summary of a literature review on
sexual harassment. Pretend that you are a university lecturer and that you have
been asked to mark this review as an assignment.
Having marked it, give it a mark out of 100 but do not write the mark on the
assignment itself.
UNIT ONE | MODERATION AND OUTCOMES-BASED EDUCATION | 17 |
Now ask two of your colleagues to mark it and to give it a mark out of 100.
Afterwards, have a meeting with these two colleagues. Compare your mark with
theirs and discuss the reasons for differences in your marks.
What questions do these differences raise concerning consistency in the
standard of marking? How could differences like these best be avoided or
eliminated?
Sexual Harassment
Sexual harassment has many consequences. Adams, Kottke, and Padgitt
(1983) found that some women students said they avoided taking a class
or working with certain professors because of the risk of harassment.
They also found that men and women students reacted differently. Their
research was a survey of 1,000 men and women, graduate and
undergraduate students. Benson and Thomsons study in Social Problems
(1982) lists many problems created by sexual harassment. In their
excellent book, The Lecherous Professor, Dziech and Weiner (1990) give a
long list of difficulties that victims have suffered.
Researchers study the topic in different ways. Hunter and McClelland
(1991) conducted a study of undergraduates at a small liberal arts college.
They had a sample of 300 students and students were given multiple
vignettes that varied by the reaction of the victim and the situation.
Jaschnik and Fretz (1991) showed 90 women students at a mid-eastern
university a video recording depicting a classic example of sexual
harassment by a teaching assistant. Before it was labeled as sexual
harassment, few women called it that. When asked whether it was sexual
harassment, 98 percent agreed.
Weber-Burdin and Rossi (1982) replicated a previous study on sexual
harassment, only they used students at the University of Massachusetts.
They asked 59 students to rate 40 hypothetical situations.
Reilley, Carpenter, Dull, and Bartlett (1982) conducted a study of 250
female and 150 male undergraduates at the University of California at
Santa Barbara. They also had a sample of 52 faculty members. Both
samples completed a questionnaire in which respondents were presented
vignettes of sexually harassing situations that they were to rate.
Popovich et al. (1986) created a nine-ten scale of sexual harassment. They
studied 209 undergraduates at a medium-sized university. They divided
these students in groups of 15 to 25 and found disagreement and
confusion among the students regarding the issue being studied.
(Adapted from Neumann, 2000: 461)
These expectations are typically used as criteria against which students literature
reviews are assessed. Based on these criteria we are of the opinion that this is a
relatively poor literature review and we would give it no more than 40 to 45%.
The writer made all the mistakes commonly made in the writing of reviews. S/he
simply summarized a number of individual sources/research reports without in any
way trying to link them to each other. Because her/his review reads like a set of
notes strung together, it fails to communicate a sense of purpose. The right way
to write a review is to organize common findings or arguments together and then
to list (in brackets) all the researchers who share the same view. The only
redeeming factor, as far as we are concerned, is that the reviewer has obviously
used a relatively wide range of sources and has acknowledged these in the
review.
The difference between the way we marked and the way you and your colleagues
probably marked is that we made our judgements in terms of a specific set of
criteria whereas you, acting according to our instructions, awarded a global mark.
Because we used specific criteria and you did not, our marking was probably more
objective than yours and our marks are likely to correlate better and/or show a
greater correspondence than yours would.
The use of criteria, like those in the bulleted list above, known as criterion-
referenced assessment, is one of the means that assessors and moderators
could use in their attempts to establish and/or maintain a common standard.
UNIT ONE | MODERATION AND OUTCOMES-BASED EDUCATION | 19 |
Study Table 1, which contains an extract from the RNCS for the Learning Area
Life Orientation, to see what we mean.
The learner will be able to demonstrate an understanding of and commitment to constitutional rights and
responsibilities and to show an understanding of diverse cultures and religions
Assessment Standards: We know this when the learner -
GRADE 7 GRADE 8 GRADE 9
Discusses the application of Discusses violations of human Debates issues with regard to
human rights as stated in the SA rights and plans counter- citizens rights and personal
Constitution strategies choices
Explains how to counter gender Explains how to use democratic Reports on participation in or
stereotyping processes to address a local planning of the local celebration
problem of a national day
Discusses the significance of Discusses how the learner would Critically discusses social
volunteer organizations promote nation-building in relationships in a variety of
different contexts situations
Explains how recognition of Critically evaluates changes in Critically investigates issues of
diverse cultures can enrich SA cultural norms and values in diversity in SA and finds ways in
society relation to personal and which to promote understanding
community issues of diverse cultures
Explains the role of oral traditions Discusses the contributions of Reflects on and discusses the
and scriptures in a range of the various religions to social contributions of various religions
worlds religions development in promoting peace
You will notice that the outcome for the 3 grades is exactly the same. Because of
this the outcome cannot be used to monitor progress (from Grade 7 to Grade 9)
or to judge competence. In order to do this, the assessor will have to use the
| 20 | ACE SML | MODERATE ASSESSMENT
You have now gone through the motions of unpacking the assessment
standards of one outcome for a specific Learning Area and Phase. Lets see if you
can now apply this to the grade and learning area/subject you teach/ have taught.
When you asked your colleague to re-assess or remark your work you were in
effect asking him/her to moderate your assessments. By providing him/her with
the criteria you used you were asking him/her to operate in a criterion-referenced
UNIT ONE | MODERATION AND OUTCOMES-BASED EDUCATION | 21 |
What though, if the class average is very low or if all the learners in a class or
school fail, as was the case in a school in North West in 2006? Does it mean that
all the learners are poor or could there be other reasons? Perhaps the test was
too difficult; perhaps the teacher does not know how to teach; perhaps the
person who marked the examination papers did not do so properly!
In cases like these, statisticians fall back on norm-referencing to ensure that the
standards against which learners have been assessed in this particular
examination/test are the same as the standards used the previous year. The
procedure they use to ensure consistency and fairness would look something like
this.
Firstly, they look at the averages for a particular subject over the past few years
to identify a trend. If, for example, the average mark for this subject for the past
5 years was somewhere between 48% and 52%, they would reason that this is a
normal average the norm and would adjust the marks of the entire group
upwards or downwards to get the average as close as possible to the norm, that
is, between 48% and 52%.
Norm referencing, and its associated statistical applications, ensure that the
distribution of marks is normal. In this case the primary concern is not with
individual marks but with the normal spread/distribution of all the marks which,
when plotted on a graph, would form an ideal bell curve (see Figures 1 through
3). In the diagrams that follow, the mean is the statistical average; the median is
the mid-point mark and the mode refers to the mark obtained by the greatest
number of learners.
Figure 1: Normal distribution curve indicating that the level of assessment was appropriate
| 22 | ACE SML | MODERATE ASSESSMENT
Figure 2: Right skewed bell curve indicating assessment was perhaps too easy
Figure 3: Left-skewed bell curve indicating assessment was perhaps too hard (Unisa 1981:71-72)
To correct a graph that leans too much to the left (assessment too difficult), the
marks of those learners who performed below the norm would be raised; if the
graph leant too much to the right (assessment too easy), the marks of those who
performed above the norm would be lowered. The only ones whose marks would
not be adjusted would be those whose marks were average or normal.
Re-read the last two paragraphs very carefully. Do you think it is fair to make use
of norm-referencing to adjust marks? Why/why not? Perhaps you and your
colleagues could debate this issue. Perhaps you could simply jot down your
thoughts in the Reflection Section of your Learning File/Folder for later reference.
UNIT ONE | MODERATION AND OUTCOMES-BASED EDUCATION | 23 |
Why dont you use norm-referencing as a form of action research in your own
1c classes? Use any of the following methods, depending on what it is that you want
to find out.
Compare the averages of the different classes for the same assessment
task if you want to see which class performs the best.
Add the marks obtained by the girls and boys in the same class separately
3 hours and work out separate averages for them. This would indicate which of
the two genders performed best in a single assessment task.
If you are interested in seeing whether either of these occurrences is
indicative of a trend e.g. that the same class always does better than the
other one or that a particular gender group always outperforms the other
you will have to repeat this exercise at least three times. If you get the
same results each time, you could infer that this is a trend and, should you
be interested in the reason for the trend you would have to investigate it
further.
| 24 | ACE SML | MODERATE ASSESSMENT
This was one example of external moderation. Another example can be found in
the way decisions are made about the awarding or not of a masters or doctoral
degree. In this case the study leader of the HEI where the student is enrolled for
his/her degree would typically appoint three external moderators i.e. assessors
who are not associated with the HEI where the student submitted his
dissertation/thesis. The external moderator/s will each assess the students
dissertation or thesis separately, using the criteria provided by the study leader.
The moderators marks, qualitative comments and/or recommendations will be
written down in a formal report that s/he will submit to the study leader. The
study leader will then compare the three reports submitted by the respective
external moderators with the marks s/he had initially allocated to the student and,
if applicable, the students mark will be adjusted. Usually the student will also
have to effect corrections to his/her dissertation or thesis as recommended by
the moderators before s/he can graduate. If the external moderators marks are
radically different from the mark allocated by the internal examiner, the study
leader will simply take the average of their marks as the final mark.
Moderation could, and should, however, also take place internally, at different
institutional levels. At a school this would mean that moderation could take place:
In the classroom, for example with the teacher moderating the results of self-
or peer- assessments
UNIT ONE | MODERATION AND OUTCOMES-BASED EDUCATION | 25 |
Per grade, where the Head of Department (HOD) for that grade, would
moderate samples of evidence assessed by those who teach learners in that
grade
Per learning area/subject, with the HOD for the learning area/subject sampling
and reassessing evidence collected and judged by teachers teaching those
learning areas/subjects.
1.5 Conclusion
In this unit we tried to introduce you to moderation as a concept. We also
described some of the strategies currently used to moderate assessments and
the frames of reference within which assessment and moderation judgements
might be made. From a systems perspective, however, the evidence collected
during assessments may not necessarily be sufficient to make judgements about
the quality of teaching, learning and/or assessment. It may well be that the
results were contaminated by other factors by the way in which evidence was
collected, interpreted or recorded, by assessor inefficiency or lack of expertise, et
cetera.
In the next unit we shall be looking at the more practical aspects of moderation,
at who the moderators should be, at when moderation should take place and at
the procedures and processes that should be followed in conducting moderation
at various institutional levels.
Introduction
Unit 2 learning outcomes and assessment criteria 2.1
Assessment and moderation 2.2
Who should moderate? 2.3
Collecting evidence 2.4
Moderating assessment instruments
2.5.1 Analyzing the standards 2.5
2.5.2 Analyzing the assessment instrument
2.1 Introduction
This unit continues the discussion of moderation that started in Unit 1. However,
whereas the focus of Unit 1 was on moderation as a concept, the focus in Unit 2
is on the practical aspects of moderation, i.e. what should be moderated, when
and how often moderation should take place; what processes and procedures
should be followed before, during and after moderation; who should accept
responsibility for what in the moderation process.
This unit is, therefore aimed at empowering you, as an educator, with the
knowledge, skills and attitudes required to become a moderator, in your
institution and elsewhere. Closely associated with this relatively specialized
knowledge is an understanding of the roles that different members of the school
community have to play in the moderation process. This, too, will be discussed in
this unit.
With regard to moderation at school level we indicated that it could take place at
classroom level, with the educator acting as moderator; at grade, subject or
learning area level, with the Head of Department assuming the role of moderator,
or at provincial/national level, with duly appointed chief markers moderating the
quality of external markers.
At classroom level the educator/s teaching a particular subject/learning area
could check/moderate whether or not learners judged their own or their
peers performance fairly and objectively during self- or peer assessments. In
this case the teacher would moderate the results of self- and peer
assessments at the end of the period or day on which these took place and, if
required, make the necessary adjustments in consultation with the learners
involved.
At subject, grade or learning area level, the subject head or Head of
Department would typically moderate end-of-term and end-of-year
assessment instruments, processes and results with a view to ensuring that
learners in the same grade are assessed fairly and with equal rigour.
What role do you think the school principal should play in the moderation process?
Should s/he act as a moderator? Should s/he decide who the moderators are?
Should s/he conduct or organise workshops to train teachers as moderators?
Think critically about these questions. Discuss them with your colleagues and/or
with your principal. Surf the Internet (see what you can find by entering the key
words moderation and school and principal) or visit a library to find
objective answers to this question. Jot down your views as well as the insights
gained from your colleagues, your literature review and your Internet survey in
your Learning File/Folder.
UNIT TWO | HOW DO WE MODERATE? | 31 |
In terms of the unit standard on which this module is based (see Texts 2 and 3),
moderators should have a clear understanding of quality assurance, should be
acquainted with moderation processes and procedures and should be able to
conduct moderation in a fair, consistent and unbiased manner. More specifically,
moderators should have the ability to plan for and prepare others for moderation;
conduct moderation; advise and support assessors on the basis of moderation
results; record and report moderation results, and review moderation systems
and processes.
Could you honestly say that those staff members at your school who act as
moderators satisfy these criteria? If not, do you think that the principal should
create opportunities for staff members to acquire knowledge of and expertise in
moderation? Why/why not?
Do you think you have the knowledge and/or experience to act as a moderator?
Do the following activity to find out.
Complete the checklist that follows to determine whether or not you would be
2a eligible to act as a moderator of other peoples assessments.
15 min
Do you know and/or understand? Yes No Explain or give examples to justify your claim
Do you know and/or understand? Yes No Explain or give examples to justify your claim
The NQF and how it contributes to
accountability and quality assurance?
The terminology/concepts used in
OBE and quality assurance discourse?
What assessment is and what it
requires the educator to do?
What moderation is and what it
requires the educator to do?
What the difference is between
assessment and moderation?
The principles that govern assessment
and moderation?
If you answered yes to all these questions, could satisfactorily complete the
extreme right hand column of the table, and if you are convinced that you satisfy
the criteria of the unit standard (see Reader, Texts 2 and 3), you should be able to
perform the role of moderator in your school. Then you need not do this module.
Instead, you should apply to an ETQA (Education and Training Quality Assurance
Body) for recognition of prior learning (RPL). Should you convince this body that
you are, in fact competent, you will be registered as a moderator with the ETQA
concerned and you will be called upon from time to time to act as a moderator.
If you skipped or were unsure about some of the questions, this module is
definitely for you. We would like to believe that, on completion of this module, you
would have acquired the requisite knowledge and skills you need to perform the
role of moderator in your own subject group, learning area, department and/or
school. We would suggest, however, that each time you do act as a moderator,
you file the evidence of your having done so in a Portfolio of Learning, which you
can then submit to the ETQA of your choice for RPL and registration purposes.
File your rough selection criteria in the appropriate section of your Learning
File/Folder because you will be asked to refer to them in a subsequent unit when
we talk about the selection and appointment of moderators.
| 34 | ACE SML | MODERATE ASSESSMENT
[Note that this kind of evidence should be filed in learners and teachers
portfolios.]
It is clear, therefore, that moderators are expected to review and/or evaluate not
only the quality of the end results but also the quality of the processes and
instruments that led to these. In order to comment on the fairness of
assessments, moderators will have to determine whether or not there are
mechanisms in place to ensure that learners do not have access to assessment
instruments beforehand, that those who invigilate during written and practical
assessments do not unfairly assist learners, and that nobody learners, teachers,
parents or anybody acting on their behalf - will be able to tamper with the
assessment results in any way.
What do you think the moderators responsibility should be with respect to the
schools assessment policy: checking that it is there in writing and/or whether or
not the policy is actually implemented?
UNIT TWO | HOW DO WE MODERATE? | 35 |
Pretend that you are the school principal or a member of the School Management
2b Team and you have been tasked to manage the moderation process at your
school. Bearing in mind the kind of information moderators should have access to
in order to make fair judgements prepare a checklist that assessors and
moderators could use to ensure that the latter has all the information s/he needs.
Use a tabular format for your checklist and include at least the information
60 min indicated in Table 3 below. To help you we have included a brief description of the
headings in the table and have filled in one line as an example.
Aspect to be Documents
When required Provider Comments
moderated required
Indicate here what Indicate here Indicate here by Indicate who Use this column to
the focus of the exactly what when the (person or capacity) monitor the
moderation process written information moderator will need has to provide the collection and
is, e.g. assessment the moderator will the documentation. moderator with the return of
process, require in terms of required documentation.
assessment the aspect listed in documentation
instruments, column 1. Also note here any
appeals procedures, problems or
etc. If, for example, the successes in this
moderator is regard.
focusing on
assessment
instruments, s/he
will need copies of
the test/exam
papers, assignment
instructions, etc.
Sufficiency of Grade 4 At the end of the The Grade 4 Head Assessment plans
assessment Assessment Plan term/year when (Add the name if were provided
evidence decisions are made possible) timeously.
regarding learner
progression Problem: The
repeating a grade or number of
moving to the next assessments in this
one grade varied with
(Add the day and some learners
date if possible) being assessed
every week and
others only once a
term.
Test papers
Exam scripts
Marked
assignments
Assessment plans
Assessment
records
Etc.
| 36 | ACE SML | MODERATE ASSESSMENT
Having compiled the list, file it in the Activities Section of your Learning
File/Folder for later use.
We wonder what you have added (or deleted) from the list we started in column 1
classroom level assessment plans; the homework timetable perhaps. The list
you make will reflect the particular context in which you work, highlighting its
strengths, weaknesses, challenges, etc. There is, therefore, no right or wrong list,
simply one that is context appropriate.
Carefully study our example in Table 3. Did you notice that no opinions or
judgements are expressed in it: it contains information only. Specifically look at the
Comments column. The moderator has inserted a comment on the number of
assessment opportunities learners had in this Grade but s/he did not indicate
whether the variation in numbers was negative or positive. This value judgement,
with appropriate recommendations, will be included in the moderators report only.
By inserting it in the checklist, however, s/he ensures that s/he remembers it
when s/he is writing the moderation report.
What would you write in your report if, as a moderator, you discovered that not all
the Grade 4 learners were assessed the same number of times? Could this have a
negative impact on learners? What does this suggest about communication and
cohesion amongst the Grade teachers concerned? What recommendations would
you make to address the variability of assessment opportunities?
UNIT TWO | HOW DO WE MODERATE? | 37 |
have studied the right work, might have had a fight with his/her parents the
night before and this might well affect his/her performance. Each learner
should, therefore, be given multiple opportunities of demonstrating what they
do know and can do.
Valid, if it assesses what it is supposed to assess. An assessment instrument
that supposedly assesses a persons critical ability would not be valid, for
example if all the questions or activities simply require the learner to
regurgitate memorised information. Similarly, learners should not be penalised
for spelling mistakes in a comprehension test the test is aimed at assessing
comprehension, not spelling. Spelling could, however, be penalised in an
essay test: essays are written to demonstrate ones writing ability and
spelling is one of the writing skills. At a more complex level, consider for
example giving someone a certificate for first aid, but only assessing his/her
written answers to questions if his/her practical ability to handle the scene
of an accident is not assessed, then peoples lives could be in danger. A
person who can answer well on paper may be scared of blood and completely
unable to intervene in a medical crisis.
Reliable, if what it assessed and/or if the kind of evidence collected will be
the same irrespective of how many times it is used for this purpose. For
example, a test aimed at determining whether or not learners can count in
multiples of three should always provide evidence of this ability no matter
how many times it is used. Another example would be a test aimed at testing
the speed at which learners can type. The same test, applied in different
contexts and to different learners should always provide evidence of learners
typing speed, irrespective of the context, learner and number of times the
test is used.
Imagine that you have been appointed as a moderator in the Learning Area Life
2c Orientation.
The teacher whose work you are moderating gave you a copy of the group
assignment in the text box that follows.
60 min
Life Orientation
Grade 8
Group Assignment
Although South Africa is called The Rainbow Nation, suggesting that we are
all living happily together, newspapers often carry reports of cultural conflict.
Instructions
As a group, cut out five newspaper reports from different newspapers
describing conflicts of culture in SA.
UNIT TWO | HOW DO WE MODERATE? | 39 |
Paste the reports onto a clean sheet of paper and then write one
paragraph on each, summarizing the conflict it describes.
Underline the cause/s of each conflict in each of the newspaper reports.
Using your summaries and the causes you underlined as a basis, prepare
a speech on cultural conflict in SA.
Your speech should:
- clarify the concept cultural conflict
- give examples of cultural conflict in SA
- suggest possible causes for such conflict
- suggest ways in which conflict could be minimised/eliminated in future
The learner will be able to demonstrate an understanding of and commitment to constitutional rights
and responsibilities and to show an understanding of diverse cultures and religions
As the appointed moderator you have to decide whether or not the assessment
instrument that the teacher is using is acceptable, i.e. the assignment, is:
Valid i.e. does it assess what it is supposed to?
Fair i.e. does it accommodate learner differences and/or barriers to learning?
Appropriate i.e. are the tasks required of the learners in line with the
assessment standards for Grade 8s as set out in Table 4?
Outcomes-based i.e. it assesses integrated competence/ability rather than
specific skills/knowledge.
Having made your decision, you are required to provide the teacher/assessor in
question with a brief report in which you:
Indicate, with reasons, why you regard the instrument as valid, fair and
appropriate or not
| 40 | ACE SML | MODERATE ASSESSMENT
File your report in your Learning File/Folder for later reference and then read our
comments.
Understanding something is not the same as knowing it. When you understand
something it means that the knowledge you have gained has become part of you,
it has been internalized. Applied to our example, this would mean that Grade 7
9 learners should, therefore, convince the assessor/s that their knowledge of
constitutional rights and responsibilities as well as their knowledge of diverse
cultures and religions has become so much a part of who they are that they will
never forget it.
The second requirement of the learning outcome was commitment. When one
is committed to something, it means that one will stick with it regardless of the
consequences and/or difficulties associated with doing it. A person can, for
example, show his/her commitment to truth by never telling a lie, even when the
truth will get him into trouble. Commitment is not something that happens in
ones head; it is a heart thing. Commitment is an attitude: it reflects where one
stands in connection with a specific thing.
Are you committed to exercising your constitutional rights and accepting your
constitutional responsibilities? If you are, this means that you know exactly what
they are, why they are important, how you should exercise them, and what will
happen if you dont. Can you see that this kind of knowledge is much more than
mere knowing? How would you go about assessing it and/or determining
whether or not an assessor has done so?
VALUES AND/OR
KNOWLEDGE UNDERSTANDING SKILLS
ATTITUDES
Learners must Learners must explain, Learners must be able Learners must show
recognize in their own words to that they
A newspaper report What cultural conflict is Can follow written Are willing and work
when they come across instructions (as with their peers
it provided in the
assignment)
Incidents of cultural
conflict as reported in Write summaries of Respect deadlines (as
newspapers what they have read stipulated in the
assignment)
Analyze written texts
with a view to
identifying causes of
and solutions to cultural
conflict
Synthesize information
into the form of a
speech
Do an oral presentation
| 42 | ACE SML | MODERATE ASSESSMENT
- on cultural conflict - to
a group of peers
(Did you notice how many of these life skills are in fact literacy skills? Why do you
think this is so?)
CRITERION 1: VALIDITY
As regards its validity, we believe that the tasks set will give learners the
opportunity to demonstrate their investigation skills (they have to search
for applicable newspaper articles and analyze them) and their
understanding of cultural diversity (identifying the causes of cultural
conflict), but we are not so sure about whether it adequately assesses
their commitment to the creation and maintenance of harmonious cultural
relations (recommending possible solutions). Merely recommending
possible solutions is not really evidence of commitment. We can say that
it is necessary but not sufficient. Commitment would involve
demonstrating practically in a real situation of conflict not an easy thing
to do in a classroom. In the area of life skills, which involves assessment
of ethical attitudes and values, validity is often difficult to achieve. We
often have to read between the lines in assessing how learners react to
and talk about case studies, role plays etc.
We should also note that a single assessment, even one that uses a
variety of methods (oral presentation, written) is usually not sufficient for
the demonstration of the outcomes, particularly those that relate to ethical
values and behaviour.
Based on this evidence we would say that this assessment instrument
was only partially valid.
inappropriate and unfair - the task set is too difficult for Grade 8 learners; it
would be more appropriate for Grade 9 learners. It therefore fails to satisfy
criteria 2 and 3. It could be argued, however, that suggesting ways in
which conflict could be minimized is part of critical analysis. It does not
require the active promotion of understanding of diverse cultures. On this
basis, the assessment could be considered fair for some Grade 8 learners.
What would make it unfair/inappropriate would be if the learners havent
had practice in summarising newspaper reports, and couldnt do the task
for that reason, rather than because they didnt have an understanding of
cultural conflict; or if they didnt have access to different newspapers, or
any newspapers at all, because they live in a rural area where newspapers
dont reach.
CRITERION 4: OUTCOMES-BASED
As regards its outcomes-based nature (criterion 4), a careful reading of the
assignment reveals that it is aimed at assessing not only knowledge or
skills; rather it focuses on competence resulting from an integration of
these (see Table 4). In the sense that the assessment instrument is aimed
at assessing competence rather than discrete skills, and integrates life and
language skills or knowledge it is outcomes-based.
You would have noticed from our analysis that the assessment instrument
satisfied only two of the four criteria. If you had been the moderator, what advice
would you have given the teacher/assessor in terms of aligning the instrument to
the assessment standards for Grade 8?
Note also that in order to judge the fairness or appropriateness of the assessment
the moderator has to understand the context of the learners for whom the
assessment is intended.
Having done so, workshop it with colleagues at your school and/or a neighbouring
school. Ask colleagues for critical comments and/or suggestions regarding ways in
which they could use this information in designing their own assessment
4 hours instruments and/or moderating one anothers assessment instruments prior to
their being used to collect evidence.
Some of the questions moderators might ask in this regard could be:
Role What role does assessment play in monitoring learner progress? How
is assessment evidence used to improve teaching and learning in your
school? Do you use assessment for diagnostic purposes? If so, do you involve
parents and/or other stakeholders in addressing identified problems?
Sufficiency - How often do you collect evidence of learner competence?
Fairness - Do learners know in advance when and how they are going to be
assessed? How long in advance do they know? Does your school have an
assessment timetable/schedule? Do learners have copies of these?
Integrity - How do you prepare learners for assessment? Do you tell them
how they will be assessed (orally, in writing, other) and what criteria you are
going to use in interpreting their work? Does your school provide for learners
to lodge appeals against assessment procedures and/or results? If so, how
does it work?
Having read these questions you would realize that the moderation of
assessment processes is trickier than moderating assessment instruments.
Because the processes are not always available in writing people can lie about
them, deliberately deceiving the moderator to protect the image of the institution.
Of course an internal moderators familiarity with the evidence collection and
interpretation processes at his/her institution will enable him/her to describe the
processes in detail but his/her ability to evaluate them objectively cannot be
guaranteed. An external moderator, on the other hand, would be better able to
evaluate the processes objectively, because s/he is an outsider, but his/her
unfamiliarity with the processes might impact on the validity and reliability of
his/her evaluation. The only way s/he can gather information is to ask assessors
for a description or breakdown of the processes followed and/or to ask for the
institutions assessment policy and assessment plan. Even then, there is no
guarantee that the assessors are not covering up for themselves or their school
or that what is written in the policies/plans are actually taking place.
This unit focuses on internal moderation, i.e. moderation by people who are part
of the institution.
However, you also need to think about the kinds of guidelines that already exist
for moderation of continuous assessment, for example. If we work in school
clusters, we can set up moderation arrangements between schools in a cluster to
ensure that internal assessments are verified and benchmarked against other
schools in the cluster which operate in a similar context.
UNIT TWO | HOW DO WE MODERATE? | 45 |
Complete the checklist that follows in respect of the school/institution where you
2e work.
30 min
YES NO
Do you monitor learner progress by means of continuous
assessment?
Does assessment evidence result in changes to in teaching
methods?
Do you use special diagnostic assessment instruments?
Do you make special provisions for those with special needs or
barriers to learning and development?
How many formal examinations do you have per year?
Are formal assessments supplemented with class tests?
Do you give learners assignments/projects to do?
Does your school have an assessment policy?
Does your school have an assessment plan?
Are teachers provided with an annual assessment schedule?
Are learners provided with an annual assessment timetable?
Are parents provided with copies of assessment timetables?
Do learners know the assessment processes and procedures?
Do you provide learners with assessment criteria beforehand?
Do you regularly inform parents of assessment results?
Do you address weaknesses identified during assessments?
Are learners allowed to appeal against assessment results?
Do you give parents feedback on diagnostic assessment?
Do you have a safe place to keep assessment instruments?
Do you enter marks on mark sheets or something similar to
these?
Do you issue reports every term?
Based on the checklist you just completed, do you think that the assessment
processes and procedures in you school have integrity? If not, in what sense are
they inadequate and to what extent could these inadequacies affect learner
performance? What do you think you could do to improve said processes and
procedures?
PART A
2f Having thought about the integrity of the assessment procedures at your
school, organise a meeting with teachers, learners and/or parents to
determine their perspectives on assessment procedures and processes at
your school.
6 hours
Ask someone to keep minutes of the meeting.
PART B
Organize a workshop aimed at the development of an assessment policy
and/or plan that clearly spells out the processes and procedures that should
be followed to ensure the integrity of assessment events. Use the minutes of
the meeting as a basis for the workshop. If you already have a policy/plan,
review it in the light of the meeting.
PART C
Call another meeting with teachers, parents and/or learners where you
present them with your policy/plan for comment. Take at least 30 minutes for
questions, comments and points of clarification.
If necessary, adjust the policy/plan in line with decisions taken at the meeting
and then start implementing it.
File all the documents generated at the meeting and the workshop as well as
the assessment/plan/policy resulting from these in your Learning File/Folder.
UNIT TWO | HOW DO WE MODERATE? | 47 |
refrained, for example, from penalising a learner for language usage in a non-
language subject.
Question 1
a.Compass
b.Quadrant
c.Cross-staff
d.Astrolabe
Question 2
a. Compass
b. Quadrant
c. Astrolabe
d. Cross-staff
Question 3
Question 4
Question 5
Accept any answer that shows the learner has correctly identified the problem and has
considered at least two relevant solutions.
Refer to Text 6: Rubrics in your reader and then design a rubric that could be
2g used to decide whether or not someone is competent to act as a moderator.
The aspects to be assessed have already been filled in for you. All you have to
do is to distinguish between someone who is competent, someone who is not
yet competent but showing progress, and someone who demonstrates no
moderation knowledge and/or abilities at all. This means you will need to
2-3 hours complete column 2, Statement of competence first, i.e. describe what it would
mean to be able to be evaluated as having a competent knowledge of
moderation.
Use your rubric to moderate your own and colleagues competence as
moderators.
File your rubric and the results of your evaluations in your Learning File/Folder.
| 50 | ACE SML | MODERATE ASSESSMENT
Moderation of
assessment
instruments
Moderation of
assessment
processes and
procedures
Moderation of
assessment
results
Overall
impression of
moderation
abilities
Implied in the whole process of moderation are the notions of transparency and
accountability. To ensure that the moderation process is both accountable and
transparent, the moderator has to record his/her findings and recommendations in
a written moderators report. The moderators report should, ideally, be presented
to assessors at a meeting between the moderator and all the assessors
concerned, one-on-one meetings between the moderator and assessors
individually and, if moderation also serves a formative purpose, the meeting should
be followed by means of post-moderation training.
During feedback sessions, whatever form they may take, assessors should be
presented with the moderation findings, their implications for subsequent
assessment processes, and the processes that were followed during evaluation.
Assessors should be given the opportunity to discuss the findings and/or the
processes with the moderator and, if applicable, to appeal against the outcome/s.
Feedback sessions should be aimed at the establishment of a common
understanding between assessors and moderators as regards outcomes,
standards and processes to be followed in order to ensure quality assessment
which would, in turn, promote adherence to sound assessment and moderation
principles, enhance the development and maintenance of institutional quality
management systems, contribute towards the further development of assessors
as needed and the promotion of collaborative approaches to assessment.
UNIT TWO | HOW DO WE MODERATE? | 51 |
2.8 Conclusion
In this unit we not only discussed the relationship between assessment,
moderation and quality education but also indicated what knowledge, skills and
attitudes a person should have if s/he wanted to become a moderator. We
indicated that, while the school principal need not necessarily be involved in the
moderation process as a moderator, s/he had to accept responsibility for the
quality of the moderation process by ensuring that the structures and processes
required for moderation are in place and that teachers in his/her school are given
the opportunity to be trained as moderators.
Assignment 1
Approach one or two of your colleagues who teach the same grade, subject or
learning area that you do and tell them that you are studying a module on
moderation. Ask them whether you could moderate one of their formal
assessment events for an assignment that you have to do.
If they agree, tell them that they need to provide you with a copy of the
instruments they used to collect and interpret evidence of learner competence as
well as a 10% representative sample of marked assessment tasks.
Having received these from them, formulate criteria that you will use to moderate
the assessment instruments and assessment results respectively your
evaluation instrument.
Having designed your instrument, use it to evaluate the instrument used by your
colleague/s to collect evidence of learner competence.
Now remark the sample of assessment tasks provided by your colleague in terms
of the assessment instruments s/he used when doing this.
Having evaluated both the assessment instrument and the assessment results,
write a report on the quality of the assessment that you moderated and submit it
to your lecturer for assessment purposes.
Introduction
Unit 3 learning outcome and assessment criteria 3.1
What is quality? 3.2
Quality Management Systems
Assignment 2
3.3
Moderation management 3.4
Conclusion
Final Assignment
3.5
Managing internal moderation
3.1 Introduction
In Unit 1 we discussed moderation as a concept. In Unit 2 we looked at the
practicalities of moderation how, when and by whom it is conducted. In this
unit the focus is on the creation of systems and procedures that will ensure the
quality and integrity of assessment practices and events and, by implication, will
enhance the quality of teaching and learning by default.
In discussing quality assurance and the role moderation plays in this regard, we
shall first compare different opinions of and/or perspectives on quality,
standards and quality assurance. Following this comparison, we shall consider
different quality management models, relating them to the South African quality
assurance system for education and training. Finally, we shall consider different
ways in which school principals and/or school communities could create and
manage their own quality assurance systems, with particular reference to the role
moderation plays in this regard.
Of course cost is not the only indicator of quality but it definitely is an important
one and most of us, if we can afford it, would make a multitude of small
sacrifices in order to wear something with a designer label.
The same thing often happens when we choose a school or a university for our
children. We know many people who send their children to private schools, even
if they can ill afford to do so, because they equate the high school fees with
quality education and/or high standards. Are you perhaps one of them?
Before we discuss this concept, we would like you first to turn to your Reader
and critically read Text 7: Standards and Quality, which is an adaptation of an
academic paper on this topic.
Having read the preceding extract, which notions of quality and standards do you
think are most prevalent in your institution? Talk to your colleagues, to learners
and to the parent community about these notions, jot down their responses and
then formulate your own definitions - for quality and standards as they apply to
school education.
File your definitions in the Reflection Section of your Learning File/Folder and keep
on adjusting them as you work through this unit.
UNIT THREE | MANAGING INTERNAL MODERATION | 57 |
SAQA, in distinguishing between the different roles that various members of the
NQF family had to perform, had the following to say:
Since schools are technically part of the SAQA family and the principal is the
head of the school family, s/he is accountable, either explicitly or implicitly, for
everything that happens in his/her school, including the creation and maintenance
of the requisite quality assurance systems.
| 58 | ACE SML | MODERATE ASSESSMENT
The ETQA for schooling is called Umalusi (derived from the Nguni word uMalusi
3a which means shepherd) and it is tasked with ensuring the quality of assessments
in school contexts (GET and FET). Go to their website (www.umalusi.org.za) to
see what it is that they are doing and what their criteria are for quality
assessments. Either download information that you think you could use in
improving assessment in your school or jot down and file relevant points in the
45 min Activities or Resource section of your Learning File/Folder.
(for initial
survey but this
is an ongoing
In practice, most of the work of Umalusi centres on the FET band (it took over all
the functions of the former SAFCERT) and more recently the General Education
and Training Certificate.
MODERATION
37. The Assessment Policy for Grade R 9 (Gazette 19640 of 1998) states
that moderation will be carried out to ensure that appropriate standards
are maintained in the assessment process. This will be done on a sample
basis at the different levels of the system. Moderation mechanisms will
be put in place at school, provincial and national level.
38. The Assessment Policy for Grade R 9 (Gazette 19640 of 1998) further
states that continuous assessment should be moderated externally by
professional support services within the guidelines set by the provincial
education departments.
39. Provincial Departments of Education will be responsible to ensure that
appropriate moderation mechanisms at school / school cluster / district
levels are in place to moderate School-Based Assessment.
40. The Common Tasks for Assessment (CTA) will be geared to function as a
moderating tool for school-based assessment.
41. It is the function of the Provinces to moderate learner performance on the
Common Tasks for Assessment (CTA).
UNIT THREE | MANAGING INTERNAL MODERATION | 59 |
42. SAFCERT [role subsequently taken over by Umalusi] would moderate all
the different forms of assessment (i.e. Continuous Assessment and
External Assessment) in grade 9.
This means that schools need also to look to the national and provincial
departmental websites for the most recent legislation, policy and curriculum
guidelines. The key websites are listed below:
National: www.education.gov.za and www.thutong.org.za
Eastern Cape: www.ecprov.gov.za/education
Free State: www.fs.gov.za
Gauteng: www.education.gpg.gov.za
KwaZulu-Natal: www.kzneducation.gov.za
Limpopo: www.limpopo.gov.za
Mpumalanga: www.mpumalanga.gov.za
North West: www.nwpg.gov.za/education
Northern Cape: www.ncedu.ncape.gov.za
Western Cape: www.wced.gov.za
However, it is possible that one or all three of these documents will have been
superceded by the time you work on this module and that it why it is important
for you to visit the relevant websites and maintain contact with your circuit and
district offices on a regular basis.
In a policy document dealing with quality management systems for education and
training providers (SAQA 2001), SAQA indicates that debates on the most
effective quality assurance model are ongoing, with some factions favouring a
Total Quality Management (TQM) model and others a Conformance to
Specifications (CTS) model (see Table 6).
NOTE:
If you have not studied either of these models during the course of this
programme, you are advised to read up on these issues use the Internet and/or
your local library for information. Also see the Recommended Reading List in your
reader. You would also have studied a number of other models in the rest of this
programme and should, as a matter of course, consider them in the creation,
maintenance and review of your school system/s.
Informed by these models, SAQA formulated a set of criteria that they expect
providers to satisfy (see Table 8) in setting up or reviewing their own quality
management systems. Should SAQA, or any other Education and Training
Quality Assurance Body, decide to conduct a review or an audit of a school or of
any other educational institution it would by implication use these criteria to
assess the systemic effectiveness and efficiency of the organisation in question.
Quality management systems Outline procedure for Clear description of nature &
implementing quality operations within organisation,
management in the organisation. indicating how quality is assured
at all levels, e.g.
How does it create & sustain
a quality culture?
How are relevance,
comprehensiveness & clarity
UNIT THREE | MANAGING INTERNAL MODERATION | 61 |
Review mechanisms Outline ways in which the Should indicate how monitoring
implementation of policies would activities contribute to
be monitored. development, i.e.:
What monitoring mechanisms
are in place?
How do they work?
How often are they used?
By whom?
How are findings reported to
the organisation?
How do findings inform
organisational improvement
efforts?
Programme delivery Outline ways in which learning Should provide a clear description
programmes would be of the programme/s offered and
developed, delivered and the ways in which they relate to
| 62 | ACE SML | MODERATE ASSESSMENT
Management systems and Indicate the financial, Indicate its capacity to manage
polices administrative and physical the organisation in an effective,
structures and resources of the efficient and accountable way,
organisation as well as the e.g.:
accountability procedures What does its management
followed within the organisation. and administrative structure
look like?
How are decisions taken, by
whom & in relation to what?
What is its financial resource
base and how is it managed?
Does it have adequate human
& material resources to carry
out its functions?
UNIT THREE | MANAGING INTERNAL MODERATION | 63 |
Carefully study Table 8, which contains SAQAs criteria for quality management
3b systems as well as explanations of what these mean.
Then, using the SAQA criteria, determine whether or not the quality management
system in your school is in place and whether or not it is as effective and/or
efficient as it should be.
5 days
We suggest that the easiest way to conduct the evaluation is to organize staff
and/or learner workshops aimed at determining whether or not existing systems in
your school/institution contribute to the planning and conducting of quality
assessments and/or whether all the parts/elements of the system contribute to
the achievement of this purpose. Since workshops like these are in effect critical
reflection exercises, you should encourage discussion and/or debate on the
manageability, validity and effectiveness of current assessment standards,
processes, procedures and decisions at the school.
We suggest moreover, that you reformulate the SAQA criteria as questions that
you could use for group discussions at the workshop.
Having analysed the existing system, complete the table that follows with specific
reference to assessment and/or moderation in your school at this particular
moment.
| 64 | ACE SML | MODERATE ASSESSMENT
Now, using the information in the table as a basis, draw one or more conclusions
about the adequacy of the system discussed and end with one or more
resolutions about the way forward.
Assignment 2
Using the self-evaluation activity you have just completed as a basis, write an
evaluation report in which you:
Detail the assessment and moderation systems in your school
Explain the purpose of the evaluation
Describe the procedures you followed in conducting the evaluation
List the criteria (those in the SAQA Table) you used for the evaluation
Summarize the results of the evaluation in the form of findings/outcomes
Draw a conclusion about the effectiveness and/or efficiency of the schools
system
Suggest/recommend strategies for improving the existing system so that it
will satisfy the SAQA and relevant DoE criteria.
Submit this report, with relevant attachments (evidence that the workshop and/or
the evaluation actually took place) to your lecturer for assessment purposes.
UNIT THREE | MANAGING INTERNAL MODERATION | 65 |
Think about this for a minute. What do you think someone other than the
moderator him/herself who has to manage the moderation process would have to
do? In a school context, who would that person most likely be? What structures
would s/he have to create to ensure that the moderation process could take place?
What processes and procedures should s/he put in place to ensure that the
moderation process is effective and efficient? What should s/he consider and/or
acquire in terms of resources human, physical, financial to ensure that
moderators can do their work properly? How should s/he respond to the results of
moderation processes and/or what should s/he do with the information provided
by moderators?
One of the things that the moderation manager usually the school principal or
3c the Head of a Subject or Department will have to do is to select suitable people
to act as moderators.
Remember the rough notes you jotted down in Unit 1 when we asked you to think
of what qualities you would like to see in a moderator?
60 min
Find those notes in your Learning File/Folder and, pretending that you are the
moderation manager, use them as basis for compiling a set of criteria that could
be used in the selection of moderators in your school.
One way of doing so is to organize a general staff meeting where the manager
shares the moderators report with the staff members. Another strategy could be
to create opportunities for the moderator/s to meet individually or in groups with
the teachers whose work was moderated.
Whichever strategy the manager chooses, s/he must ensure that moderators
share with assessors the moderation findings, their implications for subsequent
assessment processes, and the processes that were followed during the
moderation process. Assessors should be given the opportunity to discuss the
findings and/or the processes with the moderator and, if applicable, to appeal
against the outcome/s.
Which of the two feedback approaches would you prefer if you were an assessor
the general staff meeting or the individual, one-on-one sessions? What are the
advantages and disadvantages of each? Which, do you think, would contribute
most to staff development? Why?
Agreement trials can be held at all levels of the system. In schools, they could,
for example, be held between Heads of Departments, same grade teachers
and/or teachers who teach the same subject or learning area. Trials could be an
internal affair that is, involving teachers from a single school or a collaborative
affair that is, involving teachers from different schools as a networking exercise.
Agreement trials are a means whereby assessors can establish a basis for reliable
and valid judgements of learner achievement in general. Moderation, on the other
hand, is primarily concerned with determining whether or not the judgements
made by assessors were, in fact, valid and reliable. The primary purpose of
agreement trails is to develop assessor expertise. The primary purpose of
moderation is to review assessors judgements assessor development is purely
incidental.
Agreement trials will typically be institutionally based and can take place at any
time, that is, whenever assessors feel a need for them. Moderation might be an
internal or an external event, or a combination of the two takes place after a
specific assessment occasion/event. Judgements and/or comments are,
therefore, applicable to the results of that particular event only.
Can you think of other reasons why agreement trialing might be necessary in your
school? What other processes/techniques could be used alongside agreement
trialing, within a training program for assessors? Are there better ways of
achieving the same results/outcomes? What timescale would be needed to run
such a programme?
| 68 | ACE SML | MODERATE ASSESSMENT
Read the section on agreement trialling in your Reader (Text 8) and then draw a
3d table consisting of two columns, one titled Agreement Trials and the other
Moderation. Indicate, in the table, the similarities and differences between
moderation and agreement trialling.
Now call a meeting with teachers who teach the same grade, subject or learning
90 min area that you do. Introduce the notion of agreement trials to them (use your table
as a visual aid or a handout) and then facilitate a discussion on the advantages
and/or disadvantages of using agreement trials at your school. Consider, for
example:
Principles of agreement
Approach to be adopted in conducting agreement trials
Resource and time implications
Alignment of agreement principles to existing management structures
Getting staff buy-in for agreement trials.
It is unlikely that initial training in assessment will do much more than provide an
understanding of the issues involved, some frames of reference within which to
operate, and a limited amount of experience of working as an assessor.
Experience suggests that the investment in training programmes is generally
wasted unless some type of continuous support is available. Actual monitoring of
assessment activity is not, by itself, an adequate support mechanism if assessors
expertise is to develop.
2 hours
UNIT THREE | MANAGING INTERNAL MODERATION | 69 |
3.5 Conclusion
This was the final unit in the module on Moderation. As such it focused on
notions of quality, the creation, maintenance and evaluation of a quality
management system in general and in the context of educational institutions in
particular.
Together the units in this module created the opportunity for you to learn more
about quality assurance, with specific reference to the roles played by
moderators in this regard. You were shown how moderators go to work in
judging the quality of assessment instruments, processes, procedures and
results and you were given the opportunity of acting as moderators in your own
work context. Hopefully the module also stimulated your interest in the many
strategies that could be used for staff development purposes, with particular
reference to development in assessment and moderation.
We trust that you have enjoyed this module, that it has broadened your
understanding of accountability and quality assurance in general and of
moderation in particular and that it has given you the confidence not only to act
as a moderator but also to use moderation as a means of quality assurance in
your own classroom or school.
Final Assignment
Revisit the outcomes and assessment standards of this module and compile a
portfolio of evidence that will convince your lecturer of your competence as a
moderator and submit your portfolio for assessment.
NOTE:
You should refer to all the activities you filed in the Activities and Reflection
Sections of your Learning File/Folder in doing this assignment, using these as
sources of evidence and indicating in your report how and when such evidence
was collected or generated.
OR
Compile a training manual for use in the training of moderators (duties, functions,
ethics, processes/procedures, criteria, accommodation of special needs,
timelines), and ask some of your colleagues (own or neighbouring school) to
assess it in terms of its purpose and target audience prior to the training session.
Hand in both the manual and their report.
| 70 | ACE SML | MODERATE ASSESSMENT
OR
Design a set of guidelines that moderators could use in evaluating the integrity
and standard of assessment procedures at your school & ask some of your
colleagues (own or neighbouring school) to assess it in terms of its purpose and
target audience prior to the training session. Hand in both the guidelines and your
colleagues report.
Texts & Templates
Readers & Templates
The reader contains a variety of different texts which were referred to in the
Learning Guide and which provide the basis of activities or extensions of the
discussion in the main text. Some templates have also been repeated for
easier duplication.
READER
Rubrics TEXT 6
Templates
Bibliography
READERS AND TEMPLATES | | 73 |
Reader
Text 1: Blooms taxonomy: a tabular illustration
Knowledge Refers to the ability to recall Learners should show that they
what was learnt, e.g. facts, recognise things for what they
processes, procedures, are by identifying, defining,
formulae, etc. listing, labelling, matching, and
naming these
Understanding Refers to the ability to make Learners should be able to
sense of something, i.e. to explain things in their own
process information and/or to words by, for example,
read between the lines describing, summarizing,
paraphrasing, generalizing, etc.
Application Refers to the ability to use what Learners should prove that they
one knows and understands at can do something with what
an abstract level as basis for they know and understand by,
doing practical things. for example, solving problems,
creating texts, doing research,
managing a project, constructing
models, using some kind of
apparatus (e.g. a computer,
stove, camera), etc.
Analysis Refers to the ability to identify Learners should show that they
relationships between different can identify relationships
elements, and/or to break down between different things by
materials into their composite comparing/contrasting and/or
parts to see how they are put classifying/categorizing these;
together and/or how they work. justifying their responses;
providing examples/illustrations
of things, drawing
inferences/conclusions, etc.
Synthesis Refers to the ability to create Learners should show that they
something whole by can combine, compile,
rearranging individual parts into construct, organize, write, draw
something new/original. As is and build things. What they
the case in analysis, this need to do to prove this ability is
assumes an awareness of the the direct opposite of what they
relationships between parts. did in analysis.
Evaluation Refers to the ability to judge the Learners should show that they
value or quality of something. can make judgements on the
basis of evidence rather than on
their own biases or perceptions,
i.e. to distinguish between what
they prefer/like and what is
good/right, e.g. a poem, a
performance, a piece of writing,
a lesson.
| 74 | ACE SML | MODERATE ASSESSMENT
N ote:
While these categories are given here as discrete categories they are
interdependent, with the higher cognitive skills dependent on the lower ones. A
person would not, for example, be able to judge whether a sonnet is good or bad
unless s/he has analysed it to see whether it satisfies the criteria for a good
sonnet, and s/he will not be able to analyse it unless s/he knows what a good
sonnet should look like. Knowledge and understanding are, therefore, essential to
the development of all the other cognitive abilities.
READERS AND TEMPLATES | | 75 |
In particular, people credited with this unit standard are able to:
Plan and prepare for moderation
Conduct moderation
Advise and support assessors and assessment agencies
Report, record and administer moderation
Review moderation systems and processes
SPECIFIC OUTCOME 1
Plan and prepare for moderation
OUTCOME RANGE
The planning and preparation is to take place within the context of an
existing moderation system.
ASSESSMENT CRITERIA
ASSESSMENT CRITERION 1
1. A moderation system is confirmed to be in place and planning and
preparation activities are aligned with moderation system requirements.
ASSESSMENT CRITERION 2
2. The scope of the moderation is confirmed with relevant parties.
ASSESSMENT CRITERION RANGE
Parties include the assessors or assessment agencies and moderating
bodies.
ASSESSMENT CRITERION 3
3. Planning of the scope and nature of the moderation activities ensure
the manageability of moderation and enable a fair judgement to be
passed on the assessments under review.
ASSESSMENT CRITERION 4
4. The contexts of the assessments under review are clarified with the
assessors or assessment agency, and special needs are taken into
consideration in the planning.
ASSESSMENT CRITERION 5
5. Moderation processes are sufficient to deal with all common forms of
evidence including evidence gathered for recognition of prior learning.
ASSESSMENT CRITERION 6
6. The documentation is prepared in line with the moderation system
requirements and in such a way as to ensure moderation decisions are
READERS AND TEMPLATES | | 77 |
clearly documented.
ASSESSMENT CRITERION 7
7. Required physical and human resources are ensured to be ready and
available for use. Logistical arrangements are confirmed with relevant
role-players prior to the moderation.
ASSESSMENT CRITERION 8
8. A variety of moderation techniques are described and compared in
terms of strengths, weaknesses and applications. The descriptions
address the need to uphold the principles of assessment and the need
for manageable, credible and reliable moderation.
ASSESSMENT CRITERION RANGE
See "Supplementary information" for a definition of assessment
principles.
SPECIFIC OUTCOME 2
Conduct m oderation
OUTCOME RANGE
Moderation to address the design of the assessment, activities
before, during and after assessment, and assessment
documentation.
Moderation to include assessments of candidates with special needs
and for RPL cases.
Evidence must be gathered for on-site and off-site moderation.
Evidence must be gathered for moderation in situations where
- the moderation process confirms the assessment results, and
where
- the moderation process finds it cannot uphold the assessment
results.
ASSESSMENT CRITERIA
ASSESSMENT CRITERION 1
1. The moderation is conducted in accordance with the moderation plan.
Unforeseen events are handled without compromising the validity of the
moderation.
ASSESSMENT CRITERION 2
2. The assessment instruments and process are checked and judged in
terms of their appropriateness, fairness, validity and sufficiency for
assessment. The moderation decision enables the quality assurance
body`s requirements for fairness, appropriateness, validity and
sufficiency to be achieved.
ASSESSMENT CRITERION RANGE
Requirements include the interpretation of assessment criteria and
correct application of assessment procedures.
| 78 | ACE SML | MODERATE ASSESSMENT
ASSESSMENT CRITERION 3
3. Confirmation of assessment decisions enables the quality assurance
body`s requirements for consistency to be achieved.
ASSESSMENT CRITERION 4
4. Moderation confirms that special needs of candidates have been
provided for but without compromising the required standards.
ASSESSMENT CRITERION 5
5. The proportion of assessment decisions selected for verification meets
the quality assurance body`s requirements for consistency and reliability
and the use of time and resources.
ASSESSMENT CRITERION 6
6. Appeals against assessment decisions are handled in accordance with
the appeal procedures.
ASSESSMENT CRITERION 7
7. Key principles of assessment are described in terms of their
importance and effect on the assessment and the application of the
assessment results.
ASSESSMENT CRITERION 8
8. A variety of assessment methods are described and compared in
terms of strengths, weaknesses and applications.
ASSESSMENT CRITERION RANGE
The description of methods should cover situations for gathering
evidence of abilities in problem solving, knowledge, understanding,
practical and technical skills, personal and attitudinal skills and values.
SPECIFIC OUTCOME 3
Advise and support assessors and assessment agencies.
ASSESSMENT CRITERIA
ASSESSMENT CRITERION 1
1. The nature and quality of advice facilitates a common understanding of
the relevant standards and issues related to their assessment by
assessors.
ASSESSMENT CRITERION 2
2. The nature and quality of advice promotes assessment in accordance
with good assessment principles and enhances the development and
maintenance of quality management systems in line with ETQA
requirements.
READERS AND TEMPLATES | | 79 |
ASSESSMENT CRITERION 3
3. All communications are conducted in accordance with relevant
confidentiality requirements.
SPECIFIC OUTCOME 4
Report, record and administer moderation.
ASSESSMENT CRITERIA
ASSESSMENT CRITERION 1
1. Moderation findings are reported to designated role-players within
agreed time-frames and according to the quality assurance body`s
requirements for format and content.
ASSESSMENT CRITERION RANGE
Role-players could include ETQA or Moderating Body personnel, internal
or external moderators and assessors.
ASSESSMENT CRITERION 2
ASSESSMENT CRITERION 3
3. Confidentiality of information relating to candidates, assessors and
assessing agencies is preserved in accordance with the requirements of
the assessing agency and ETQA requirements
SPECIFIC OUTCOME 5
Review moderation systems and processes.
ASSESSMENT CRITERIA
ASSESSMENT CRITERION 1
1. Strengths and weaknesses of moderation systems and processes are
identified in terms of their manageability and potential to make
judgements on the quality and validity of assessment decisions.
ASSESSMENT CRITERION 2
| 80 | ACE SML | MODERATE ASSESSMENT
ASSESSMENT CRITERION 3
3. The review enhances the credibility and integrity of the recognition
system.
Definition of Terms:
Assessment - a process in which evidence of performance is
gathered and evaluated against agreed criteria.
Performance - includes skills, knowledge, understanding and
attitudes, and the ability to transfer these to new situations.
Assessment criteria - state the type and quality of performance
against which the candidate is assessed.
Candidate - person whose performance is being assessed by the
assessor.
Moderation - a process for making judgements on the quality and
result of assessments, with the purpose of confirming assessment
judgements.
| 82 | ACE SML | MODERATE ASSESSMENT
Principles of assessment:
Appropriateness: The method of assessment is suited to the
performance being assessed.
Fairness: The method of assessment does not present any barriers to
achievements, which are not related to the evidence.
Manageability: The methods used make for easily arranged, cost
effective assessments which do not unduly interfere with learning.
Integration into work or learning: Evidence collection is integrated into
the work or learning process where this is appropriate and feasible.
Validity: The assessment focuses on the requirements laid down in
the Standard; i.e. the assessment is fit for purpose.
Direct: The activities in the assessment mirror the conditions of actual
performance as closely as possible
Authenticity: The assessor is satisfied that the work being assessed
is attributable to the person being assessed.
Sufficient: The evidence collected establishes that all criteria have
been met and that performance to the required Standard can be
repeated consistently.
Systematic: Planning and recording is sufficiently rigorous to ensure
that assessment is fair.
Open: Learners can contribute to the planning and accumulation of
evidence. Assessment candidates understand the assessment
process and the criteria that apply.
Consistent: The same assessor would make the same judgement
again in similar circumstances.
Plan and prepare for Role and function of moderation as an Planning and preparation activities are
moderation. integral quality assurance process. aligned with moderation system
requirements.
Framework of policy, planning,
communication, methodology, The scope of the moderation is
timeframes and logistical elements confirmed with relevant parties.
governing moderation.
Planning of the extent of moderation
and methods of moderation ensures
manageability of the process.
Planning makes provision for sufficient
moderation evidence to enable a
reliable judgement to be passed on
the assessments under review.
Conduct moderation. Does the instrument assess the The moderation is conducted in
outcomes? accordance with the moderation plan.
Advise and support Reflective practices being able to The nature and quality of advice
assessors analyse the results of the moderation facilitates a common understanding of
process, the feedback thereof and the relevant outcomes and criteria,
and issues related to their assessment
develop constructive and timely
by assessors.
remedial response.
The nature and quality of advice
promotes assessment in accordance
with good assessment principles and
enhances the development and
maintenance of quality management
systems in line with ETQA
requirements.
Report, record and Knowledge of participative strategies of Moderation findings are reported to
administer moderation. management and leadership to promote designated role-players within agreed
a collaborative approach to assessment. time frames and according to the
quality assurance bodys requirements
Knowledge of appropriate administrative for format and content.
procedures, e.g. recording, reporting and
confidentiality. Records are maintained in accordance
with organisational quality assurance
Knowledge of assessment and and ETQA requirements.
moderation of assessment instruments.
Confidentiality of information relating
to candidates and assessors is
preserved in accordance with
organisational quality assurance and
ETQA requirements.
Review moderation systems Developing a school wide moderation Strengths and weaknesses of Reflect on assessment moderation
and processes. policy and the systems required for its moderation systems and processes practices in our school. Aspects that
implementation. are identified in terms of their must be addressed are:
manageability and effectiveness in
Standard
facilitating judgements on the quality
Areas for improvement
and validity of assessment decisions.
Benchmarks
Manageability
Overall effectiveness of
implementation.
Recommendations contribute towards
the improvement of moderation
systems and processes in line with
ETQA requirements and overall
manageability.
SPECIFIC OUTCOMES CURRICULUM OUTLINE ASSESSMENT CRITERIA ASSESSMENT ACTIVITIES
The review enhances the credibility
and integrity of the recognition
system.
READERS AND TEMPLATES | | 89 |
1) Multiple-choice tests are perceived as objective tests that are best used to
assess learners knowledge and understanding but also their ability to analyse
and to apply knowledge and understanding in solving problems and/or arriving
at the correct answer.
EXAMPLE
The content, tone and style of a piece of writing usually reflect the source
from which it comes. Study the extract that follows and then answer the
questions that follow.
2) True and False tests are primarily used to assess learners knowledge and
understanding. True and false items could, however, also be used to assess
learners thinking ability and/or their attitudes. In these instances they would
consist of a short text in which someone argues for or against something and
the learners has to indicate whether or not the argument is sound and/or
where someones behaviour is described and learners have to indicate
whether or not the behaviour is acceptable/appropriate. To eliminate the
| 92 | ACE SML | MODERATE ASSESSMENT
danger of learners simply guessing the answer in such tests is to ask them to
give a reason for their choice in each case.
EXAMPLE
THE SOUTH AFRICAN BILL OF HUMAN RIGHTS
1) Forms part of the Constitution TRUE FALSE
This is an example of a straightforward true or false test but these tests could
also be used to assess learners thinking ability and/or their attitudes. In these
instances they would consist of a short text in which:
Someone argues for or against something and the learners has to indicate
whether or not the argument is sound
Someones behaviour is described and learners have to indicate whether or
not the behaviour is acceptable/appropriate
Read the extract below and then indicate whether or not Sams
behaviour is acceptable or unacceptable.
Sam Isaacs is a 17-year old boy. About a month ago he started feeling
very tired. He also contracted a cold and, eventually, a cough.
Notwithstanding the fact that he used the medicine prescribed to him by
the doctor he could not shake off the cold. One of his friends suggested
that he should go for an Aids test. To Sams dismay he tested positive.
He remembered that his uncle told him that HIV/Aids could be cured if
one had intercourse with a virgin. Desperate to be cured, Sam went to
the station, waylaid a girl on her way back from school and raped her.
Happy that he would now be cured, Sam went back home without
feeling any guilt about his act.
ACCEPTABLE UNACCEPTABLE
The biggest weakness of True or False Tests is that they lend themselves
to guessing. Guessing could, however, be eliminated if learners are
required, at the end of each item, to give a reason for their answer.
3) Mix and match tests assess learners knowledge and understanding only. The
tests consist of two columns of words, phrases, or statements. The
statements in the right hand column are usually muddled or mixed up.
Learners are required to find a word, phrase or statement in the right hand
column that matches each item in the left hand column.
READERS AND TEMPLATES | | 93 |
EXAMPLE
Find the word in the right-hand column that is closest to the meaning of
each of the words in the left hand column. Do not rewrite the words;
simply write the number of the word in the left hand column, followed by
the appropriate symbol in the right-hand column.
4) Cloze procedure tests are texts from which certain words, phrases or clauses
have been left out and learners are required to fill in the blanks. In order to
construct a cloze test, teachers/assessors take a text (from a textbook or a
set of notes) and rewrite this, leaving blanks where learners have to fill in the
answers. In content subjects such tests do little more than assess learners
knowledge, i.e. what they have learnt or memorized and can remember. For
each gap there is usually only one correct answer. In languages they are used
to assess learners knowledge and understanding of language usage. In this
case there is seldom only one correct answer. The only criterion is that the
learners answers must demonstrate that they understand how language
works and the text must make sense when all the words have been filled in.
You will note that, in this example, every fifth word has been underlined.
These are the words that will be left out and which, ideally, learners will
have to fill in to complete the blanks. You will also note that they represent
a wide range of language structures such as nouns, verbs, prepositions,
conjunctions, pronouns, etc. Many of the correct answers could be
replaced with others that have more or less the same meaning and these
would have to be accepted as alternate answers, e.g.
Builds = m akes
Returns = com es back
Som e = m ealie
Sustenance = food, energy, strength
| 94 | ACE SML | MODERATE ASSESSMENT
You will note that closed questions are aimed at assessing learners
knowledge while open questions are aimed at assessing their
understanding and their ability to apply analyze and to solve problems.
Open questions are, therefore, more demanding both for learners and for
teachers. Teachers must be very careful when marking open questions
that they do not mark them subjectively but assess their worth/quality
against objective criteria and/or standards.
6) Essay type questions are probably the most demanding types of paper and
pencil tests because they assess learners ability across the whole of Blooms
Taxonomy. Essays are aimed at assessing learners ability to synthesize
(create their own texts), but, in order to do so, learners have to know,
understand and be able to apply what they have learnt in order to create this
text. In argumentative essays they also have to demonstrate their ability to
think critically and logically.
QUESTION 2
A renowned Shakespearean critic and political analyst once made
the claim that Macbeth is the epitome of the power-hungry ruler.
Informed by your knowledge and understanding of the play,
Macbeth, construct an argument in which you either support or
refute this statement. Your essay should show evidence of:
Your knowledge of the play
Your understanding of the main character, Macbeth
Your ability to construct a sound argument
Your ability to express yourself in clear and appropriate
language
Your ability to construct a coherent and cohesive text.
Text 6: Rubrics
The word, rubric, derives from the Latin word for red - a rubric was the set of
instructions for a law or liturgical service, typically written in red. Thus, a rubric
instructs people on how to lawfully judge a performance. A good rubric allows
valid and reliablecriterion-referenceddiscrimination of performances.
Consider a common problem in teacher grading. The student gets only one grade
for a subject. We do not typically see the criteria or dimensions of performance
that are involved in the grading. Johnny and Suzy both get a B for the year, but
for vastly different reasons. In one case it relates to inconsistent test scores, in
another, consistent scores but poor homework. No one except the teacher
| 98 | ACE SML | MODERATE ASSESSMENT
knows the reasons for the gradecertainly not the reader of the transcript. The
same teacher might give the students different grades at different times for
varying reasons.
What happens if we use a vague, holistic rubric and each judge is then free to
highlight different criteria? One math assessment judge is totally turned off by a
students computational errors; another sees them as minor, given the overall
quality of the proof in response to a problem. Each judge is unknowingly applying
different criteria. Unless the criteria specifically state a degree of computational
accuracy and provide rubrics for how to weigh this factor in judging the overall
quality of the problem, a judge can easily be distracted by obvious errors or
achievements that may not be central to what is being assessed.
The more explicit the rubric is, the more consistent and reliable the scoring will
be. Good rubrics define criteria and specify dimensions of performance, i.e. the
category or aspect of a performance that is being assessed. One dimension to
score in assessing writing, for example, is organization. The criteria for that
dimension typically involve phrases like there is a clear structure; the thesis is
apparent and developed in a logical way; there is a beginning, a middle and an
end with a conclusion. Sometimes an assessment that scores different
dimensions separately is called an analytic-trait scoring process. For example, the
dimensions we use to assess writing might be ideas, organization, voice, word
choice, sentence fluency and conventions. In this case there would be a rubric for
each dimension.
Assessors have to take great care to ensure that their rubrics are not only reliable
but also valid. Suppose a rubric for judging story writing places exclusive em-
phasis on spelling and grammatical accuracy. The scores will likely be highly
reliablebecause it is easy to count the errorsbut yield invalid inferences about
students ability to write stories, It is unlikely that spelling accuracy correlates
with the ability to write an engaging, vivid, and coherent story. Many fine spellers
cant construct engaging narratives, and no doubt many wonderful storytellers did
poorly in school grammar and spelling tests.
A rubric should always detail better and worse in tangible, qualitative terms. What
specifically makes this argument or proof better than another one? If you use
comparative language to differentiate quality, make sure that what is being
compared is relative quality, not relative, arbitrary quantity.
Compare the following excerpts from the ACTFL guidelines (Figure 8) and a social
studies rubric (Figure 9) to see the point. The ACTFL rubric is rich in descriptive
language and provides insight into each level and its uniqueness. The social
studies rubric never gets beyond comparative language for the dimensions being
READERS AND TEMPLATES | | 99 |
assessed. In the social studies rubric note that the only difference between each
score point is a change in one adjective.
NOVICE-HIGH
Able to satisfy immediate needs using learned utterances
Ask questions or make statements with reasonable accuracy only
where this involves short memorized utterances or formulae.
Most utterances are telegraphic, arid errors often occur when word
endings and verbs are omitted or confused
Speech is characterized by enumeration, rather than by sentences.
There is some concept of the present tense forms of regular verbs
and some common irregular verbs.
There is some use of articles, indicating a concept of gender, although
mistakes are constant and numerous.
INTERMEDIATE-HIGH
Able to satisfy most survival needs and limited social demands.
Developing flexibility it, language production although fluency is still
uneven.
Can initiate and sustain a general conversation on factual topics
beyond basic survival needs.
Can give autobiographical information.
Can sporadically, although not consistently, give simple directions and
narration of present past and future events.
Limited vocabulary range and insufficient control of grammar lead to
much hesitation and inaccuracy
Can control the present tense of most regular and irregular verbs
Comprehensible to native speakers used to dealing with foreigners,
but still has to repeat utterances frequently to be understood by
general public.
3. Basic Story. Paper describes a series of events, giving details (in at least
two or three sentences) about some aspect of the story (the events, the
characters goals. or problems to be solved). But the story lacks cohesion
because of problems with syntax, sequencing, events missing, or an
undeveloped ending.
resolved: the beginning does not match the rest of the story: the internal
logic or plausibility of characters actions is not maintained).
In terms of the
Exceptional view, quality is seen as something really special, distinctive, and,
except in extraordinary circumstances, unattainable
Perfectionist view, quality is seen as something that is flawless, without any
faults, something that everybody strives for and sometimes achieves
Fitness for purpose view, quality is seen as the extent to which actions are
aligned to purpose, or the extent to which they are just right for something
or somebody
Value for money view, quality is seen in terms of a return on an investment,
i.e. whether or not the result/outcomes of a particular action or activity
warranted/was worth the effort put into the action/activity
Transformation view, quality is equated with flexibility, i.e. the ability to adjust
to circumstances and/or to adjust the circumstances themselves, should this
be necessary
In education, standards are typically associated with three areas of activity, viz.
academic performance, competence, and service.
Academic standards specify the expected levels of academic achievement
and/or performance, i.e. learners ability to do what is required of them in
specific programs of study or, in schooling, in specific learning areas or
subjects and grades. Such standards typically specify the requisite cognitive
abilities, i.e. what a person should know, understand, analyze and evaluate
and to what extent s/he should be able to apply such cognitive skills to the
solution of theoretical problems or imaginary situations.
Competence standards specify what a person should be able to do, i.e. what
skills or competencies s/he should demonstrate. Competencies might include
skills required for the job market, skills required for induction into a
profession, problem-solving skills, communication skills, as well as cognitive
skills - like the ability to analyze, synthesize, evaluate and extrapolate. Often,
there is some overlap between academic and competence standards since
the one may be implicit in the other.
Service standards are usually associated with the service industries
restaurants, hotels, public service, etc. and specify what constitutes good
service. We often hear, for example, that service in South Africa is
horrendous, that nobody cares about quality, only about money. One service
READERS AND TEMPLATES | | 103 |
standard aimed at addressing poor service in South Africa is the Batho Pele
principle for civil servants. If you were to apply this standard to the way public
servants treat some people, for example, would you rate their service to be of
a high quality or not?
All these standards could be applied to education and training contexts: academic
standards to what learners/educators should know and understand; competence
standards to what they should be able to do with what they know, and service
standards to how they should conduct/behave themselves. Service standards
could also be applied to the way in which teaching and learning processes are
managed, i.e. to turnaround times in assessments, to class sizes, punctuality,
availability of resources, complaints procedures, etc.
Irrespective of whether the trial occurs pre- or post assessment, it will usually
start with the leader/chairperson providing participants with a brief overview of
the task to be undertaken, followed by some or other procedure aimed at
determining what materials are available for trialling. The facilitator/chairperson
would then invite an assessor briefly to describe the material/activity that has to
be scrutinized, explain the context in which it was or will be used, the purpose it
served/would serve, and the criteria against which evidence was or would be
judged. This is an important part of the trialling because it ensures that the other
assessor participants will consider the evidence presented in an informed way.
Once the assessor has made his/her presentation, the assessment instruments
and/or assessment evidence, whichever is on trial, would be circulated to all
present and they would be given the opportunity of studying and discussing it in
terms of agreed upon criteria/element, for example:
The aspects of competence/achievement that were or will be assessed
The kinds of evidence that were or will be regarded as proof of
achievement/competence
Whether or not it served or would serve its intended purpose
Whether or not it catered for individual learner needs (learning style, learning
tempo, barriers to learning, etc.)
Judgements on the quality of the instrument/interpretation of evidence and/or
recommendations regarding adjustments and/or changes
Templates
This section of the module contains blank forms of the templates that were
discussed in this module. You may feel free to copy or adapt these templates to
suit your particular needs and circumstances.
Aspect to be Documents
When required Provider Comments
moderated required
Indicate here Indicate here Indicate here by Indicate who Use this column
what the focus of exactly what when the (person or to monitor the
the moderation written moderator will capacity) has to collection and
process is, e.g. information the need the provide the return of
assessment moderator will documentation. moderator with documentation.
process, require in terms of the required
assessment the aspect listed documentation Also note here
instruments, in column 1. any problems or
appeals successes in this
procedures, etc. If, for example, regard.
the moderator is
focusing on
assessment
instruments, s/he
will need copies
of the test/exam
papers,
assignment
instructions, etc.
READERS AND TEMPLATES | | 109 |
TA BLE13:SW O T A N A LYSIS
Bibliography