Assessing

Productive Skills:
Writing and
Speaking
Session 7
Assessing Writing
General Approach

Teachers frequently ask their

Students to write on a variety of
topics
They generally assess them on:
 the message contained in the writing,
 the clarity and organization of the
message
 the mechanics (spelling, capitalization
and punctuation) utilized
 IMITATIVE Examples
• Handwriting letters,
words, punctuation
• Keyboarding
• Copying
• Listening cloze
• Picture-cued writing
exercises
• Completing
forms/questionnaires
• Converting numbers and
abbreviations to words
• Spelling tasks
• One-word dictation
It is simply the ability to parrot
back a letter, word, phrase or
a sentence.
 INTENSIVE
Examples
• Dictation of phrases/simple
sentences
• Dicto-comp
• Grammatical transformation
exercises
• Picture descriptions
• Use vocabulary in a sentence
• Ordering task
• Short –answer tasks
• Sentence completion tasks
It is the production of
short and simple of
written text.
RESPONSIVE Examples
• Paraphrasing
• Guided writing
(question/answer)
• Responding to
charts, graphs,
diagrams
• Responding to a
reading or lecture

The tasks require a response to a

prompt within a structured framework.
 EXTENSIVE Examples
•Essay writing tasks
•Different types of
writing (narrative,
descriptive,
argumentative,
persuasive, etc.)
•Tasks in genres of
writing (lab reports,
opinion essays,
technical reports,
research papers,
The tasks include different purposes etc.)
and various genre.
Approaches to Writing
Assessment
• Two major approaches to writing assessment:
• direct
• indirect
Indirect Writing Assessment

• very commonly used

• measures correct usage
(grammar/vocabulary)
• sentence level
• focuses on spelling and punctuation via
objective formats like MCQs and cloze
tests.
• construct is that writing ability is made
up of discrete writing subskills
• largely concerned with accuracy rather
than communication.
Direct Writing Assessment
• ability to communicate through written texts in
which students
• come up with the content,
• find a way to organize the ideas and
• employ the use of appropriate vocabulary, grammatical
conventions and syntax.
• Construct is that writing skills are integrative and
communicative
Your writing experience

What did your teachers ask you to write?

How did they assess your writing?
Designing Writing Assessment
Tests/Tasks
• four basic elements:
• Rubric
• the instructions
• Prompt
• the task
• Expected response
• what the teacher intends Ss to do with the task
• Post task evaluation
• assessing the effectiveness of the writing task.
1: Rubric
The basic element of assessment…
the instructions for carrying out the writing task
as clear and as comprehensive as possible
include information like…
 the procedures for responding
 resources can use (dictionaries, spell/grammar
check, etc.)
 method of delivery (i.e. paper, printout, online…)
 the task format
 time allotted
 how the test/task will be evaluated
the information in the rubric comes from the
test specification.
 Test specifications for rubric

• What should test specifications include?

• details on the topic
• the language use/patterns/rhetoric to be tested
• the intended audience
• how much information should be included
• the number of words the student is expected to
produce
• overall weighting
2: Prompt

• What the student must respond to

• three main prompt formats…
• base = task is given in direct and very simple
terms
• framed = a situation is presented, which the
students use as framework for the task
• text-based = a text is given, which students
have to respond to or utilize in their writing
• Which types of prompts were you asked most
often?
 Sample Base Prompts

• Do you favor or oppose a complete ban on smoking?

Why? Why not?
• Why do many people say that women are better drivers
than men?
• Many say that “Money is the root of all evil.” Do you agree
or disagree with this statement?
Sample Framed Prompts

• On a recent flight back home to Turkey, Pegasus lost

your baggage. Write a complaint letter to Mr. Altıok, the
General Manager, telling him about your problem. Be
sure to include the following:
• Your flight details
• A description of the baggage lost and its contents
• What you would like Mr. Altıok to do for you
Sample Text-based Prompts

You have been put in charge of

selecting an appropriate
restaurant for your graduation class
party. Use the restaurant reviews
below to select an appropriate
venue and then write an invitation
letter to your fellow classmates
persuading them to join you there.
 Writing Good Prompts
• A writing prompt defines the task…
• a question or a statement to address
• the conditions to complete the task.
• What are the criteria of a good prompt?
• will generate the desired type of writing, genre or rhetorical
pattern;
• involves students in thinking and problem-solving;
• accessible, interesting and challenging to students;
• topics are meaningful, relevant, and motivating;
• does not require specialist background knowledge;
• uses appropriate signpost verbs;
• fair and equal opportunities for all students to respond
• clear, authentic, focused and unambiguous;
• specify an audience, a purpose, and a context.
 Signpost Verbs
• Choose ‘signpost’ terms to match rhetorical pattern.
• What are some common verbs and the associated patterns?
• Some common verbs..
• Describe: give a detailed account
• Discuss: argue a thesis, identifying pros and cons
• Explain: state and interpret
• Compare: show similarities between two things
• Contrast: show the differences between two things
• Analyze: identify main points and evaluate them
• Define: provide the definition and exemplify
• Summarize: produce a concise account, no details or examples
• Outline: provide a summary of main points and sub
headings
• Evaluate: appraise the worth or value of something
3. Expected Response

• What the teacher intends students to do with the

writing task.
• What is necessary?
• You need to have a clear picture of what type of
response you want the assessment task to generate
• Should be clear in the test specification
4. Post Task Evaluation

 Reflect on the effectiveness of your writing

tasks/tests.
 What questions would you ask yourself?
 Did the prompt discriminate well among my Ss?
 Were the essays easy to read and evaluate?
 Were Ss able to write to their potential and show what
they knew?
Keep your responses in mind when designing the
next writing task.
Issues in Writing Assessment

• Timing
• Process versus Product
• Conditions of administration
• Topic limitation
• Classroom teacher as rater
• Multiple raters
 Timing
• How much time to complete writing tasks?
• Depends on whether it’s process or product oriented
• Product-oriented--a good rule of thumb?
• allowing 30 minutes is probably enough time for most
students to produce an adequate sample of writing.
• Process oriented writing or portfolios?
• Flexible, but generally a lot more time should be
allocated for assessment tasks.
Process vs. Product
 What’s the difference?
 In the process approach…
 Students submit their work in a portfolio that
includes all draft material.
 Assessment involves looking at all steps in the
process
The product approach is more traditional…
 accomplished through timed essay which
usually occurs at the mid and end point of the
semester
 Which to use?
 Attempt to use a combination of the two
approaches
 Administration Conditions
 What is the impact and role of technology in writing?
 Should students be allowed to use computer, spell and
grammar check, thesaurus and online dictionaries?
these tools would be available to them in real-life contexts.

 What are the issues in using technology?

The skills divide: could put those who write by hand at a
disadvantage.
electronic writing assessment is also a test of keyboarding and
computer skills.

 You decide what to use for your writing assessments, but…

 be consistent.
 Topic Limits
• Should all students write on the same topic or be given alternatives?
• Arguments for same topic?
• More reliability in scores
• More consistency in following assessment criteria
• difficulty to provide alternatives at the same level of difficulty.
• General consensus to write on same topic(s) because students
don’t waste time selecting a topic instead of spending that time
writing.
• Arguments for alternative topics?
• Can select topics that interests them and ones they have
background knowledge on.
• Reduces student anxiety.
• Make sure your alternative topics are the same genre and
rhetorical pattern to achieve inter-rater reliability
Classroom Teacher as Rater

• Should classroom teachers mark their own

students’ papers?
• Divided opinions on this issue.
• Against: Ts have bias for or against students
• For: Ts know their own students best
• Ideal assessment procedure?
• Double blind marking is the recommended ideal
where no student identifying information appears
on scripts.
Multiple Raters
 Do we really need more than one marker for
student writing samples?
 YES!
Why?
 Multiple judgments lead to a “true” score than
any single judgment
 All reputable writing assessment programs use
more than one rater to judge essays
 How many do we need?
 Two is recommended, with a third in case of
extreme disagreement or discrepancy.
 Writing Marking Procedures
• How can we make writing assessment reliable?
• carefully thought-out set of procedures
• calibration of the rating process.
• Steps in the writing calibration process:
• experienced raters elect a number of sample benchmark scripts
from completed exam papers.
• Benchmark scripts need to be representative of the levels:
• Clear pass : Good piece of writing that is solidly in the A/B
range)
• Borderline pass: a paper that is on the borderline between
pass and fail but shows enough to be a pass)
• Borderline fail : a paper that is on the borderline between pass
and fail but does not have enough to pass)
• Clear fail : a below average paper that is clearly in the D/F
range
 Writing Marking Procedures
• First, experienced raters rate the scripts using the scoring
criteria and agree on a score.
• note down a few reasons why the script was rated in such a
way
• Next, lead arbitrator conducts a standardization or
norming session
• the entire pool of raters rate the sample scripts and try to agree
on the scores that each script should receive.
• In these calibration sessions Ts should evaluate and discuss
benchmark scripts until they arrive at a consensus score.
• These sessions are time consuming and not very popular
• Ts want to get started on the writing marking right away.
• Typically there are heated difference of opinion, especially when
raters of different experience and background.
• Despite these disadvantages, they are an essential
component to standardizing writing scores.
 Holistic Marking Scales
 What is holistic marking?
 based on the marker's total impression of the essay as a whole.
 variously termed impressionistic, global or integrative marking
 quick and reliable with multiple raters (3 to 4 mark each
script)
 mark for two hours and then rest -- grading no more than 20
scripts per hour
 most successful when scales are limited (i.e. ranging from 0-6).
 Advantages?
 reliable if no time constraints and standardization done.
 generally quicker than other types of writing assessment
 a large number of scripts scored in a short period of time.
 since overall writing ability is assessed, Ss are not
disadvantaged by one lower component such as poor
grammar bringing down a score.
 the scores tend to emphasize the writer’s strengths
 Holistic Marking Scales
• Disadvantages
• can be unreliable if marking is done…
• under short time constraints
• with inexperienced, untrained teachers
• longer essays…
• often tend to receive higher marks.
• reducing a score to one figure..
• tends to reduce the reliability
• difficult to interpret a composite score from a holistic mark
so…
• low or negative washback.
• Tendency to overlook the various sub-skills that make up
writing.
• seen to be limited in that it does not provide a profile of the
student's writing ability.
Analytic Marking Scales
• What is analytic?
• Separate assessments for each of a number
of aspects of performance (discrete point)
• raters mark selected aspects of a piece of writing and
assign point values to quantifiable criteria.
• Advantages?
• generally more effective with..
• inexperienced teachers.
• more reliable
• scales have a larger point range.
• with a population of inexperienced Ts who have had little
training and grade under short time constraints
• Training raters is easier because the scales are
more explicit and
• provides teachers with a "profile" of their students’
strengths and weaknesses in writing for better washback
Analytic Marking Scales

• Disadvantages?
• appears more time consuming because..
• each individual aspect is assessed separately
• needs a set of very specific criteria and frequent…
• calibration sessions.
• marks are often lower when essay is not viewed…
• as a whole, but rather according to content,
organization, grammar, mechanics and vocabulary,
• the typical analytic writing scale
• five component skills, each focusing on an important aspect
of composition and weighted according to its approximate
importance:
• content (30 points), organization (20 points), vocabulary
(20 points), language use (25 points) and mechanics (5
points).
• The total weight for each component is further broken
down into numerical ranges that correspond to four
levels from “very poor” to “excellent”
 10 Things to Remember and why...
1. Give students multiple writing assessment opportunities.
• Provide plenty of opportunities for Ss at all levels to practice the type
of writing that you expect them to do on the writing test.
2. Test a variety of writing skills and create tasks of varying lengths.
• Take more than one sample of a students’ writing. This reduces the
variation in performance that might occur from task to task.
3. Develop prompts that are appropriate for the Ss.
• Make sure that the prompts you select or develop invite the desired
type of writing. They should be realistic, sensitive to the cultural
background of students, and within their realm of experience.
4. Evaluate all answers to one question before going on to the next.
• This practice prevents a shifting of standards from one question to
the next and helps the rater mark more consistently.
5. Mark only what the student has written.
• Don’t be influenced by other factors in addition to the quality of the
work like the quality or legibility of the handwriting.
 10 Things to Remember and why...
6. Have a systematic approach for dealing with marking discrepancies.
• take the average of the two raters for a small discrepancy and to
utilize a third rater if there is a big discrepancy.
7. Get students involved.
• in both the development and marking of their writing tests.
8. Provide students with diagnostic feedback.
• Use writing assessment results to identify what Ss can and cannot
do well and make sure to provide this information to Ss. With
analytic marking you will have access to a profile to give Ss
feedback. With holistic marking scales, be sure to take notes on
Ss’ strengths and areas for improvement.
9. Practice blind or double blind marking.
• Mark essays without looking at Ss’ names as the general impression
we have of our Ss is a potential form of bias.
10.Calibrate and recalibrate.
• The best way to achieve inter-rater reliability is to practice. Start
early in the academic year by employing the marking criteria and
scale in non-test situations. Make Ss aware from the outset of the
criteria and expectations for their work.
 Assessing Speaking
Objectives:
a. Identify basic types of speaking
b. Provide examples of testing intruments that fit
the types
c. Develop scoring instruments
 Basic types of speaking

• Imitative: performing prosodic features

• Intensive: short stretches of oral language, eg. Directed
response tasks, reading aloud, dialogue completion,
limited picture-cued tasks and translation
• Responsive: including interaction at somewhat limited level
of a very short conversation
• Interactive: length and complexity—transactional and
interactional
• Extensive: speeches, oral presentation, story telling
Micro and Macro skills (Brown &
Abeywickrama, p. 143)
Designing assessment tasks
 Imitative
• Word repetition
 Intensive speaking
• Directed response
 Intensive speaking
• Read aloud
 Intensive
• Dialogue completion
 Intensive
• Directed response task – form filling/ oral questionaire
 intensive
• Picture-cued
Responsive speaking (QA)
Responsive (Giving direction)
Responsive: paraphrasing
Interactive speaking: interview
Extensive: oral presentation
Picture-cued story telling
Groupwork: Poster
presentation
Design some other item types, which are different from the
one(s) provided. The items should assess the same types of
speaking performance.