Exam Development Guideline
Exam Development Guideline
Exam Development Guideline
As compared to the WHO standard of health professionals to population ratio for developing countries,
Ethiopia has wide gap yet and the government has been working to increase the number of health
professionals by increasing the intake of health science colleges and by opening new institutions.
While this helped to reduce the gap we have compared to the standard, the quality of the teaching and
learning process hence, the service provided by the professionals became an issue for different
stakeholders and the government as well. For this reason and since licensure exams are widely practiced
internationally in different countries to insure the quality of health service, the federal ministry of health
took the initiative to launch licensure exam initiative within human resource directorate since July 2015.
Since then two round pilot tests given and reports discussed with different stakeholders.
The initiative currently led by a directorate established for this purpose, which is composed of three case
teams and has been doing different activities to strengthen the system and capacitate the directorate to
implement the exam in wider range with a much better organization and quality. As part of this, different
documents were prepared to help facilitate the exam process and create transparency and sustainability of
the program.
In the preparation of this document and other manuals, different examining institutions and countries
experiences were reviewed, highly valued experts from different higher teaching institutions and Jhpiego-
Ethiopia were involved for whom my deep appreciation goes for and for the high commitment and
hardworking of the staffs at the federal ministry of health.
This exam development manual for competency assessment and licensure is a contribution from several
educators and concerned individuals with a genuine interest to propel Ethiopian health professions’
education forward. The manual serves as a springboard for introducing the principles of defining and
maintaining standards in health sciences education. This document was prepared in different workshops
held in Debrebirhan, Adama, and Bishoftu. FMoH would like to acknowledge the following participants
for their immense contribution.
Contributors Institution
HTN Hypertension
Figure 1: Flow chart depicting the overall exam development process of NHPCAL
The Government of Ethiopia recognizes that healthcare is one of the crucial components of
basic social services with direct linkage to growth and development of the country as well as to
the welfare of the society. The quality of health care delivered in a country has an immediate
and long-term impact on the quality of life, livelihood, morbidity and mortality of its citizens
and on the nation’s economy and its national security. Accordingly, the Ministry of Health
(MOH) is committed to reduce morbidity, mortality and disability and to improve the health
status of Ethiopian people by providing and regulating health services. Health professionals’
competency assessment is among the approaches of regulating health service; hence MOH
designed health professional Competency assessment.
The sole purpose of competency assessment is to identify health professionals who possess the
minimum basic knowledge and experience necessary to perform tasks on the job safely and
competently, but not to select the “top" candidates. It is also to validate safe and effective
medical practice, job-related skills and knowledge to provide an independent assessment and
documentation of competency. Therefore, competency assessment is very different from
academic or employment examination.
The purpose of exam development is to assure reliability and validity of exams by following
important steps in the exam development process by applying an expertise made blueprint.
This manual is organized to describe the objective, development process, review, assembly
system, standard-setting methods, and security-related issues to ensure the development of
quality multiple choice questions and objective structured clinical examination (OSCE)
stations.
1
Exam Development Manual
Section 2: Objective
➢ The objective of this manual is to effectively guide the exam development process for
standardized competency assessment of health professionals.
2
Exam Development Manual
Section 3: Definition of Terms and Operational Definitions:
Blueprint: is a document which is used to represent the knowledge, skill, and attitude obtained
from higher health institutions in a way that enables to measure the content, method of
assessment and its weight.
Exam assembling: is a process of selecting and compiling items and stations according to the
blueprint
Exam bank: a platform or system where exam item and stations are sorted, filtered and stored
for future use
Exam developer: is a professional who prepares exam according to the exam blueprint.
Exam reviewer: is a professional who validates, identifies items and makes ready for final
exam printing after the development of exam
International candidates: Are those candidates who have undertaken their educational
training abroad and want to practice in Ethiopia
Multiple Choice Questions (MCQ): is one of a written assessment tool which enables the
examinee to choose the single best answer from the list of options.
Objective Structured Clinical Examination (OSCE): is a tool for assessment of skills and
attitude obtained from higher health institutions
Standardized patients (SPs): is someone who has been trained to portray, in a consistent,
standardized manner, a patient in a medical situation.
3
Exam Development Manual
Section 4: Roles and Responsibilities
4
Exam Development Manual
4.4 Exam Reviewer
• Validate the developed exam items for content relevance, technical accuracy; clarity and
sensitivity
• The editorial reviewer checks grammar, punctuation and spelling errors
• The reviewer will also verify exam fairness to all examinees and absence of culturally
and religiously sensitive material to any particular group
• Maintain a high level of ethical principles during developing the exam
• Secure all information appropriately
5
Exam Development Manual
Section 5: Preparation Phase of Exam Development
Preparation for Exam Development process is targeted to select and train individuals who will
be involved in the exam development and review activities. Based on the intended activity,
selection criteria and the components of training provided may be different.
The goal for selecting exam developers is to form a team of subject matter experts i.e. health
professions education experts and psychometric experts (from different educational institutions
and health facilities across the country) to develop high-quality MCQ items and OSCE
stations.
• Subject matter expert from different educational institutions and health facilities across the
country
• Currently working in higher institutions as an instructor and having a Master's degree and
above in the relevant field of study with a minimum of four years of experience as an
instructor. For those disciplines instructed by BSc degree instructors; exam developers
who hold first degree with instructing experience of four years can be selected
• Specialization in medical fields with a minimum of four years of experience
• Trained as an exam developer is preferable
• Can bring a support letter from his/her institution
6
Exam Development Manual
Training for MCQ item developers
• General information, specific details, and components specific characteristics about MCQ
exam development
• Principles of evidence-based exam development techniques
• Should have a minimum of four years’ experience after completion of a Master degree or
specialty certificate in a specific field of study
• Should currently be delivering a course in the subject area (closely-related Course)
• Must bring a support letter from his/her working organization
NB: Before conducting an exam review, orientation shall be given for exam reviewers
7
Exam Development Manual
Section 6: Exam Development Process
Conducting
Task Analysis
Blueprint
Preparation
Constructing Items
& Stations
Exam Revision
Psychometric
Analysis
Not
Selected
Exam Review
Selected
Exam Banking
Exam Assembling
Figure1: Flow chart depicting the overall exam development process of HPCALE
8
Exam Development Manual
6.1 Conducting Task Analysis
The purpose of task analysis in this manual is to systematically document the tasks that health
care workers perform on-the-job; rather than relying on obsolete curricula or out-of-date
national documents.
• HPCALD in collaboration with respective professional associations shall conduct the task
analysis every 5 years but may do it more often to keep pace within change in health care
services
• Task analysis should target those health care cadres who currently practice in their specific
field of study that reflect current curricula and training
Task analysis involves two phases:
Components of Blueprint:
9
Exam Development Manual
Key steps in constructing a blueprint
The blueprint is prepared by plotting the program contents against the domains of the clinical
competencies desired to be assessed. Blueprints are often written as a grid, usually content-by-
process matrix (Annex-1).
The health professionals’ competency assessment and licensure exam includes the following
components:
1. Stick to the manual and blueprint for constructing quality MCQ items and stations
2. Check for technically flawed items and tricky questions
3. Consider reviewer’s feedback as needed
4. Make sure that an item has fulfilled all the technical and structural requirements before
submission
10
Exam Development Manual
6.3.1. Developing Multiple-Choice Questions:
➢ A single response (one best answer) type of MCQ item with four options should be
used.
Components:
• Stem
• Lead-in
• Options/Alternatives, suggested solutions/:
• Key: This is the correct choice for the item.
• Distracters: These are the incorrect choices for the item
Sample Items
11
Exam Development Manual
• Avoid “tricky” and overly complex items
• Write options that are grammatically consistent and logically compatible with the stem
• List them in logical or alphabetical order
• Write distracters that are plausible and the same relative length as the answer
• Avoid using absolutes such as always, never, and all in the options; also avoid using vague
terms such as usually and frequently
• Avoid negatively phrased items (e.g., those with except or not in the lead-in). If you must
use a negative stem, use only short (preferably single word) options
• Components are assessed in a planned or structured way with attention being paid to the
objectivity of the examination
• The station should be designed and developed by expertise which is assigned by OSCE
team
• The development process of OSCE should be revised by exam reviewers
• Pilot testing of the station should be done before the stations are stored in the station bank.
• Each station should include a brief of the station profile like:
➢ The objective of the station, station types, station date and duration, instructions,
station resources, and opening statement etc.
• Each station is designed to assess a range of skills (practical skills, communication skill,
data interpretation, and clinical decision-making skills) drawn from the elements of
competence
• If necessary, an oral question may be applied and it should be well designed, more
objective, scored in the separate sheet, and weight calculated accordingly
• A station may test one or a combination of different elements of competence
• The number of stations per exam should be 8 to 12 each lasting for 5 to 10 minutes
• The sequence of stations should be maintained in a way that stations will not affect each
other
• All candidates pass through the same stations to maintain standardization
12
Exam Development Manual
• When standardized/ simulated patients are used, detailed scripts are provided to ensure
that the information that they give is the same to all candidates, including the emotions
that the patient should use during the consultation
• There should be an instruction/ orientation package for examinee, examiners and
standardized/ simulated patients
• There should be an organized team for designing and implementation of OSCE stations
13
Exam Development Manual
6. Objective of the station
• It is a statement that shows the task expected from the candidate and should demonstrate
clearly
7. Opening statement (Scenario)
• The OSCE Scenario should be prepared based on common or critical patient presentations
which the candidates will encounter as a practitioner/entry level health professional
✓ It should provide relevant patient information to guide the examinee to perform the
required task
✓ The scenario should be written along with clear instruction
✓ It should be written clearly (better to use patient language). (See the following
example)
Example
A 54 years old man presented to ------ hospital OPD, complaining of chest pain of 1-month
duration.
8. Station resources
• The resources needed for each task should be clearly indicated
9. Instructions
• Instructions for examiners, examinees, and SPs should be clearly written and communicated
timely
• It should be complete, stating clearly and precisely the setting, time allotted and the task that
the examinee should perform
• Should be printed and laminated separately and given to the examinee or posted in a wall that
can be easily read and re-read at a glance
14
Exam Development Manual
Instructions for Examiners
• Should be briefed about the station profile before engaging them into the exam
• Should be instructed about the instructions given to the examinee and remind students to read
the instructions if they appear to have forgotten what they should do
• Should be familiar with the checklists
• Greet students in a similar way and don’t coach students through facial expression and other
non-verbal communication
• Intervene only for safety reasons
• Stay at the assessment area during the entire duration of the assessment activity and observe
and record the students’ performance on the checklist
• Should complete the mark sheet and prepare to receive the next candidate,
• Record and report the result timely and provide feedback on the whole process of the OSCE
to the Directorate using a score summary sheet (Annex-3).
• Should be brief, complete and clear; so that the standardized patients (SPs) portray the
scenario consistently for all examinees.
15
Exam Development Manual
Items regarding generic skills which are expected to be in most or all stations should not be
given a high or less mark as this will affect the overall mark of the exam.
16
Exam Development Manual
Section 7: The Exam Review Process
➢ The main purpose of the exam review process is for evaluating content relevance,
technical accuracy, clarity and sensitivity related to culture and religion(Annex-8 and
Annex-9)
MCQ field testing and OSCE piloting will be conducted in randomly selected higher education
institutions among graduating students.
➢ Item Difficulty Index and distracter analysis shall be employed to improve the items
17
Exam Development Manual
8.3 Post-pilot testing Exam Review
The analyses performed at the time of pilot testing are used to inform item reviewers in the
following ways:
• The percentage of candidates responding correctly to an item (or the mean item score)
informs test developers as to whether the item is more or less difficult for the students than
intended.
• The percentage of candidates selecting each multiple-choice item response option informs
test developers as to whether each potential distracter is sufficiently attractive to students.
• The relative number of candidates in each of the score categories of constructed-response
items helps test developers to evaluate the scoring rubrics and definitions of the
categories.
• It will help to adjust whether the duration of each OSCE station or the MCQ is sufficient
or not
18
Exam Development Manual
Section 9: Exam bank
9.1. Item Banking
This step is done after test items undergo a rigorous review and editorial process as well as
field testing. Exams put in a bank are believed to accurately measure the knowledge, skill, and
attitude of the candidate.
➢ Selection of items can be made through an automated item bank by randomly selecting
items to meet pre-specified parameters.
➢ There are software programs that manage to file, sorting, storing, retrieval, statistical
analysis and updating of items.
➢ As new items are added, others will have completed their service and will be ready to
retire.
➢ The shelf-life of each item shall be ten years with slight modifications if necessary,
after which it will be removed from the item bank.
➢ Items can be sorted and filtered to enable easy review by content experts and
psychometric staff.
➢ Categorize items according to the content outline or blueprint
19
Exam Development Manual
• Author of item
• Item status-coded whether it is new or old
• Reference for answer verification
• Cognitive level of items across Bloom’s Taxonomy
• Equivalent items (i.e. similar items that should not appear on the same form)
• Graphic link, if a graphic is part of an item
• Comments-of the psychometrician
• The frequency of items uses within five or ten years
➢ Discrimination: Questions that fall into the Good category (greater than 0.3)
➢ Difficulty: Questions that fall into the Medium category (between 30% and 80%)
➢ Questions in the Poor category of a discriminatory level and questions in the
Easy or Hard categories are recommended for further review.
➢ Follows similar basic principles of MCQ banking mentioned in this guideline above.
The following information is to label each of the OSCE stations for banking
• Unique station identifier
• Objective/learning outcome number from test blueprint
• Station type (OSCE, Interpretation)
• Checklists / Rubrics
• Elaboration parameters
• Opening statement
• Examination dates that station was used and when it is scheduled for next use.
• The sequence number of station
• Number of candidates who attempted the OSCE station
• Classical statistics (Cronbach’s alpha and Discrimination index)
20
Exam Development Manual
• Average time to perform the station
• Author of station
• Station status-coded whether it is “new” or “old”
• Station resources (candidate, examiner’s, simulated patient and non-standardized patients
instructors, equipment list, special requirements e.g. restocking technician and also a
scoring form
• Reference for verification
• Equivalent items (i.e. similar items that should not appear on the same form)
• Graphic link, if the graphic is part of an item
• Comments of the psychometrician
• A Cronbach's alpha of 0.7 to 0.8 should be taken as an acceptable level of reliability for the
health professionals’ competency assessment and licensure exam. The examination board
should apply Cronbach's alpha that allows the detection of the OSCE stations which are
main sources of error, by removing one station at a time from the analysis and looking at the
reliability of the remainder.
• Multi-facet ANOVA analysis should be conducted to obtain the estimates of the
components of variance for each of the following facets: stations, sites, test versions, and all
their interactions. G-coefficient of 0.7 to 0.9 should be taken as an acceptable level of
reliability for the health professionals’ competency assessment and licensure exam.
• A good correlation (R2>0.5) will indicate a reasonable relationship between checklist scores
and global grades. The existence of low R2 values at certain stations and/or widespread
marks for a given grade should prompt a review of the item checklist and station design.
• When low R2 reported, quality improvement should be undertaken byre-writing of the
station and checklist with plans for re-use of this station and subsequent analysis of
performance within a future OSCE.
21
Exam Development Manual
Section 10: Exam Assembly
10.1. Exam assembling/formatting criteria
Items for inclusion on a test can be selected manually, randomly drawn by the computer from
all existing items in the bank, or drawn by the computer from pre-specified parameters.
➢ Both MCQ and OSCE exams should also be assembled according to the exam blueprint
and get ready for exam administration.
➢ When an item is selected or assembled for test inclusion, any graphics, tables, or cases
associated with it should automatically be included.
➢ After completion of assembling the items, they shall be coded/ arranged in different four
exam booklets in order to reduce the chance of cheating before administration of each
exam.
Exam versions are prepared by the exam assemblers using the computerized system and
selection of items from exam bank. The exam format can also be structured by the computer
software.
Item modeling and equating are the methods used for the preparation of equivalent exam
version for a similar set of the examinee.
Item modeling is more successful with MCQs that have longer stems, especially clinical
vignettes.
1. Select a source item- a well-written MCQ on a topic for which you want additional items
2. Highlight the specific terms in the stem that are important clinical content, (e.g., clinical
setting, medical history, presenting complaint(s) etc.
3. Identify the correct (keyed) response and the content category to which it belongs
4. Review the available wrong options (distracters), and discard any that are inconsistent or
flawed
5. For each clinically important term in the stem, list several significant alternatives
22
Exam Development Manual
6. Prepare complete specifications for each new item. Identify the content of the new stem by
labeling one clinically reasonable combination of the alternatives. Then, for each new
stem, identify or provide a keyed response and stored to the bank
23
Exam Development Manual
Section 11: Standard Setting
Standard setting is used to categorize candidates who score at or above the cut score and those
who fail to score that. It should involve policymakers, test developers and content specialists.
Modified Angoff (1971) model is one of the most widely used standard settings that employ a
test-centered approach for both written and skill licensure exams. Modified Angoff method is a
judgmental approach in which a group of expert judges makes estimates about how borderline
candidates would perform in the examination, i.e. the proportion of borderline examinees who
will answer an item correctly.
• Select at least (7-15) Subject Matter Expert (SME) raters using the criteria for exam
developers
• Assemble a diverse group of SMEs based on (e.g., gender, age, and educational
background etc)
24
Exam Development Manual
Table 1: Sample test item rating form
5 10
Rating Process
• The raters will be given the test items along with the test item rating format above
• Raters are not provided with the answer key
• Separate the raters and have them provide estimates for each test item. Allow
approximately two (2) hours for a 100-item test
• Reconvene raters and proceed to the next step
• Collect the raters test item rating forms and enter the results Expert Ratings Spreadsheet
(Table 2)
• Tabulate the average percentage correct for each test item by adding the raters’ percentages
and dividing by the number of experts
• Determine the standard deviation
• For any test item whose standard deviation exceeds 10, raters should discuss the reasons for
variations in the estimates
• Finally, the cut score should get approval from the health professionals’ competency
assessment and Licensure Examination Committee
25
Exam Development Manual
Test Expert 1 Expert 2 Expert 3 Expert 4 Expert 5 Average Standard
Items Name Name Name Name Name Percentage Deviation
Correct
1
2
3
4
Average Cut score
Table 2: Sample Expert Ratings Spreadsheet
• Scoring methods and any other information from which the panelists may benefit
• Where the faculty is not familiar with the OSCE stations, a mini OSCE is set as part of the
standard setting orientation procedure
o Faculty may play the role of examinees, or examiners, by observing each other
o This is done to avoid overestimation or underestimation of the station difficulty.
Their own level of performance as experts may serve as a `ceiling’ effect for the
standard-setting ratings.
26
Exam Development Manual
Step 3: Characteristics of borderline candidates
• Panelists should write independently the characteristics of the borderline candidate per skill
component
• The panelists’ statements are then posted and the facilitator discusses with the panelists
each statement
• Arguments and disagreements are clarified and the group reaches a consensus as to what
would be an appropriate list of borderline characteristics per skill component
• Panelists are asked to make judgments as to how many items should be answered correctly
by the borderline candidate in order to pass the stations
o Note: in this manual, it is recommended that the number of items will constitute the
ratings and not percentages of items.
• The facilitator should present all panelists’ ratings on the board by assigning a number to
each panelist.
• The panelists discuss their ratings
• The facilitator encourages panelists with the highest and lowest ratings to react to their
judgments.
• The facilitator will average the ratings of the panelists to produce a cut-off raw score for
the station.
Provision of actual performance data
• The distribution outlines the cumulative numbers and percentages of students who got one
item correct, two items correct, three items correct etc.
• The facilitator should indicate the percentages of students who might fail the stations if the
panelist average ratings are applied to the distribution as a cut-off score.
• A discussion should revolve around the issue of the `consequential data', i.e. percentage
failure. Are panelists surprised? Should they expect a lower or a higher number of failures?
Does the percentage failure seem to coincide with their experience? This provides another
source of information, which helps panelists adjust their ratings when they are asked to
attempt a second rating on the form
• Panelists provide their second ratings, which are posted on the board by the facilitator.
27
Exam Development Manual
• A final cut-off score is calculated by averaging all the ratings
General consideration
• A large group of panelists (i.e. 18) into three groups of six each.
• The groups will set standards on different stations but one or two stations will be rated by
all. This is done to examine the consistency of ratings among panelist groups
Sta Rater 1 Rater 2 Rater 3 Rater 4 Rater 5 Rater 6 Rater 7 Rater Rat an
tio 8 er avera
n 1st 2nd 1st 2nd 1st 2nd 1st 2nd 1st 2nd 1st 2nd 1st 2nd 1st 2nd Av ge of
Rat Rat Rat Rat Rat Rat Rati Rati Rating Rat Rat Rati Rati Rati Rat Ra era the
ing ing ing ing ing ing ng ng ing ing ng ng ng ing tin ge statio
g n
scores
(stude
nt)
1
2
3
4
5
6
7
8
9
10
11
12
Passing score
Table 3: Sample standard setting spreadsheet for skill assessment
28
Exam Development Manual
Section 12: Exam Security
• The exam bank will be kept in a secure location at the HPCALD and only the authorized
personnel shall decide on who shall have access to the exam bank and when.
• Subject Matter Experts and other technical experts shall have restricted access as deemed
necessary by the Directorate for the purpose of accomplishing their own respective tasks only.
• The HPCALD shall exercise Maximum Security in collaboration with National Educational
Assessment and Examination Agency (NEAEA), Information Network Security Agency
(INSA) and Addis Ababa University/Institute of Educational Research (AAU/IER) and other
concerned authorities.
• During exam development, there needs to be a security camera in the exam development
rooms.
• The exam will also be stored in a secure computer where the passwords are handled only
by authorized staff and disabled from any of internet connection.
• To that end, regular spot check visits and review processes will be conducted at all steps.
• All individuals who are involved in any of the above processes are expected to sign an
Exam Security Agreement (Annexe-10) and abide by the rules of the HPCALD with
regards to overall exam security. Items shall be reviewed and edited by exam reviewers
from different institutions
• The HPCALD will prepare an ethical charter/security agreement form and members and
other individuals involved should sign and commit to the cause and shall be held
accountable for any security breach.
29
Exam Development Manual
Section 13. References
• Scott A. Cunningham, Sarah M. Callahan, Jason K. Feld, and John Richard Bergan, Item
Development and Assessment Construction Guidelines for Common Core State Standards,
2013 (SQA 2018)
• Bluestone J, Johnson P, Fullerton J, Carr C, Alderman J, and Bon Tempo J. 2013. Effective
in-service training design and delivery: evidence from an integrative literature review.
Human Resources for Health 11:51. doi:10.1186/1478-4491-11-51.
• Leah Hart, PheleloMarole, Galina Stolarsky, (2014) Peter Johnson and Elizabeth Wagstaffe
• Linking Licensure Examinations to Nursing Practice through Task Analysis. A Case for
BotswanaJhpiego Brown’s Wharf 1615 Thames Street Baltimore, Maryland 21231-3492,
USA www.jhpiego.org
• Oshio S, Johnson P, and Fullerton J. 2002. The 1999–2000 task analysis of American
nurse-midwifery/midwifery practice. Journal of Midwifery & Women’s Health 47(1): 35–
41. doi:101.1016/S1526-9523(01)00218-5.
• AMCB. 2012 Task Analysis Report: A Report of Midwifery Practice. Linthicum, MD:
American Midwifery Certification Board; 2012.
• The Federal Democratic Republic of Ethiopia, Ministry of Health, 2015. Draft
Implementation Guideline for health professionals’ competency assessment and Licensure
Examination of Health Professionals. FDRE MoH.
• Airasian, P. (1994) Classroom Assessment, Second Edition, NY: McGraw-Hill.
• Brame, C., (2013) Writing good multiple choice test questions.
• Brown, F. G. (1983). Principles of educational and psychological testings (3rd Ed.). New
York: Holt, Rinehart, and Winston
• Cangelosi, J. (1990) Designing Tests for Evaluating Student Achievement. NY: Addison
Wellesley.
• Cassin, W. E. (1987). Improving essay tests. Idea Paper, No. 17. Manhattan, KS: Center for
Faculty Evaluation and Stevens,
• Greenland, N (1993) How to make achievement tests and assessments, 5th edition, NY:
Allen and Bacon.
• Implementation Guideline for health professionals’ competency assessment and Licensure
Examination of Health Professionals,2015
30
Exam Development Manual
• Wickens, T. D., Multiday contingency tables for the social sciences. Hillsdale, NJ:
Lawrence Erlbaum Associates (1989) p. 238.
• Williams, J. R., Tiller, M. G., Herring, H. C. III, & Steiner, J. H. (1988). A framework for
the development of accounting education research. Sarasota, Florida: American Accounting
Association.
• Burton, S. et al., 1991. How to prepare better multiple-choice test items: Guidelines for
university faculty. Brigham Young University Testing Services and The Department of
Instructional Science, p.36. Available at: http://testing.byu.edu/info/handbooks/better
items.pdf\nhttp://www2.swccd.edu/~mseteachingresources/assets/multiple
choice/multiplechoiceguidelines.pdf.
• Callahan, S.M. & Ph, D., 2013. Item Development and Assessment Construction
Guidelines for Common Core State Standards by.
• Cohen, A.S., Handbook on Test Development : Helpful Tips for Creating Reliable and
Valid Classroom Tests and James A. Wollack Testing & Evaluation Services University of
Wisconsin-Madison 1. Terminology Used in Testing. Test.
• Medical Council of Canada, 2013. Guidelines for the development of objective structured
clinical examination (OSCE) cases, (November), pp.2–36.
• Mullis, I.V.S. & Martin, M.O., 2011. TIMMS 2011 Item-Writing Guidelines. , pp.1–28.
• Prince Georges Community College, Best Practices for Writing Multiple-Choice Questions
Best Practices for Writing Multiple-Choice Questions. In pp. 1–3.
• Saudi Commission for Health Specialties, 2014. Objective Structured Clinical Examination
OSCE,
• Shirwaikar, A., 2015. Objective structured clinical examination (OSCE) in pharmacy
education- A trend. Pharmacy Practice, 13(4), pp.1–5.
• Isaac I. Bejar., Standard Setting: What Is It? Why Is It Important? Copyright © 2008 by
Educational Testing Service. http://www.ets.org/research/contact.html
• Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and
evaluating performance standards on tests. Thousand Oaks, CA: Sage.
• AMEE Guide No. 18. 2000. Miriam Friedman Ben-David: Standard setting in student
assessment Medical Teacher, Vol. 22, No. 2
• Angoff, W.H. (1971). Scales, norms and equivalent scores. Educational Measurements.
31
Exam Development Manual
Washington, DC: American Council on Education
• Alan Wheaton and Jim Parry, (2012). Using the Angoff Method to Set Cut Scores. Test
Defensibility in the US Coast Guard - Using the Angoff Method to Set Cut Scores
• Kane MT. 2006. Validation. In: Brennan RL, editor. Educational Measurement. Westport,
CT: American Council on Education and Praeger Publishers. pp 17–64
• Ben-David MF. 2000. AMEE Guide No. 18: Standard setting in student assessment. Med
Teach 22:120–130.
• Epstein RM. 2007. Assessment in Medical Education. N Eng J Med 356:387–396.
• The Federal Democratic Republic of Ethiopia, Ministry of Health, November 2015.
Implementation Guideline for health professionals’ competency assessment and Licensure
Examination of Health Professionals.
• Federal Ministry of health professionals’ competency assessment and licensure
examination for health professional directorate. OSCE Implementation guideline, 2017
• Hays R. 2008. Assessment in medical education: Roles for clinical teachers. Clin Teach
5:23–27.
• Hossam Hamdy 2006. Blueprinting for the assessment of health care professionals,
Practical
• Assessment. Blackwell Publishing Ltd 2006. THE CLINICAL TEACHER 2006; 3: 175–
179
• Kamran z. Khan. The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81.
Part I: An historical and theoretical perspective
• Mary E. McDonald, 2007. The Nurse Educator’s Guide for Assessing Learning
nd
Outcomes.2 edition.Jones and Bartlett Publishers.
• Saudi Objective Structured Clinical Examination (OSCE) MANUAL. Saudi Commission for
Health Specialties, Riyadh – 2014, ISBN: 978-9960-9832-8-8
• Steven M Downing, 2003. Validity: on the meaningful interpretation of assessment data. Medical
Education 37:830–837
• Sood R. 2001. Long Case Examination - Can it be improved. Indian AcadClin Med 2:251–255
32
Exam Development Manual
Section 14 Annexes
14.1. Annex-1 Blueprint Example for Midwifery Profession (Obstetric II Module)
34
Exam Development Manual
respectful manner
35
Exam Development Manual
labor and delivery 7.3
36
Exam Development Manual
• Provide appropriate care OSCE 1.83 4.83
for patients with
obstetrics emergency in a
respectful manner
• Provide HE for a pregnant OSCE 1.5
woman about obstetrics
emergency
Evaluation • Follow the patient Direct 0.67
condition after Observation
intervention
• Decide on alternative MCQ/OSCE 0.4 2.17
measures when primary
intervention has failed
• Evaluate the current OSCE 0.4
condition of a patient after
intervention
• Link with other services if MCQ/ OSCE 0.3
necessary
• Counsel when to return OSCE 0.4
about the patient follow up
Manage Basic Science • Describe the MCQ/Essay 0.75
abnormal pathophysiology of
puerperium abnormal puerperium 2.5
• Explain types of abnormal Essay 0.5
puerperium
• Analyze the consequence Essay 0.75
of abnormal puerperium
• Identify sign symptoms of MCQ 0.5
abnormal puerperium
Assessment • Take full Hx OSCE 0.9
• Perform P/E OSCE 0.87
• Approach a patient in a OSCE 0.7 3.67
friendly manner during Hx
and P/E
• Order appropriate lab OSCE 0.7
investigation
• Order U/S OSCE 0.5
Diagnosis • Identify abnormal MCQ/Short 1.5
puerperium Answer
• Identify the diagnostic MCQ/Short 1.5 4
methods for abnormal Answer
puerperium
• Interpret lab results Short Answer 1
Management • Describe the management MCQ/Essay 1.4
of abnormal puerperium
• Provide appropriate care OSCE 1.49
for patients with 4.3
abnormality during
puerperium obstetrics
emergency in a respectful
37
Exam Development Manual
manner
• Provide HE for a pregnant OSCE 1.4
woman about abnormal
puerperium
Evaluation • Follow the patient Direct 0.57
condition after Observation
intervention
• Decide on alternative MCQ/ OSCE 0.4 2.5
measures when primary
intervention has failed
• Evaluate the current OSCE 0.4
condition of a patient after
intervention
• Link with other services if MCQ/OSCE 0.3
necessary
• Counsel when to return OSCE 0.5
about the patient follow up
38
Exam Development Manual
14.2. Annex-2 Sample OSCE Station profile template
Scenario:
Resources needed (list all the necessary resources for this particular station: equipment,
standardized patients)
39
Exam Development Manual
14.3. Annex-3 Score Summary Sheet
40
Exam Development Manual
14.4. Annex-4 Elaboration parameter for each activity in the checklist
41
Exam Development Manual
14.5. Annex-5 Scoring and Marking Tool (Checklist)
Rate/Score
Task/Activities/Items Not attempted at all Attempted but not Satisfactory
satisfactorily
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
OSCE Total Score (100%)
42
Exam Development Manual
14.6. Annex-6 Oral Questions for OSCE
2.
3.
4.
____________________________________________________________________________
____________________________________________________________________________
________________________________________________________________
43
Exam Development Manual
14.7. Annex 7: OSCE station Quality assurance checklist
Station Author/s:
Station Title:
44
Exam Development Manual
14.8. Annex 8: Checklist for reviewing Multiple-Choice Items
Item Type
Date Written
Item Developer
Reviewers
Review Date
Comments: _______________________________________________________________
45
Exam Development Manual
14.9. Annex-9 Checklist for reviewing OSCE
Station Type
Date developed
station Developer
Reviewers
Review Date
46
Exam Development Manual
14.10. Annex-10 Test Security Agreement
As a member of the team for the exam development, standard setting, exam review or assembling
of the Health Professionals’ Competency Assessment and licensure exam in Ethiopia, I, the
undersigned, accept the responsibility for maintaining strict confidentiality of items, stations,
materials and information related to the examination as detailed below:
I declare that no one from my family members, including my spouse, children, brothers,
and sisters, is a candidate for the licensure examination in the year of this exam
development.
I will not share any information about the examination with anybody including my
families and friends except duly authorized persons (i.e., team members from HPCALD
and AAU/IER), though any means or media including telephone, text message, pictures,
internet, print materials, face to face discussion, etc.
I am aware that examination items under my control including, but not limited to scratch
papers, draft copies, are to be kept in a secure location.
I am aware that I may be sued in accordance with the country's law if I violate these
provisions.
I have read and understood the provisions of this security agreement. My signature below
signifies that I do accept the terms and conditions of this security agreement.
Signature: _________________________
Date: _____/_____/_________
47
Exam Development Manual