NBA Accreditation and Teaching-Learning in Engineering (NATE)

N J Rao and K Rajanikanth

Module 2: Course Design and Instruction

Week 6: Design Phase and Development Phase (Key Words: Design Phase, Targets,
Assessment Pattern and Assessment Instruments, Item Banks, and Development Phase)

20
M2 U5: Design Phase
Recap

 Understood the need for creating sample assessment items for all the COs, locating the course
outcomes in the taxonomy table, mapping COs to POs and PSOs, and elaborating CO into

20
competencies where required.

M2 U5 Outcomes

M2 U5-1: Understand the nature of assessment.

M2 U5-2: Identify the sub-processes of Design Phase.

Design Phase

It is concerned with

EL
Identifying what the learners should be able to perform to demonstrate the attainment of the
PT
learning objectives (course outcomes in the context of OBE and NBA).

Assessment

 Assessment is a measure of performance

 Evaluation is an interpretation of assessment
-N

 Assessment drives student learning

 Our assessment tools tell students what we consider to be important
 Teachers guide students to learn through their assessments
 It is a glue that links the components of a course – its content, instructional methods, and skills
development
E

Test Items/Items and Questions

 Questions + Additional related information = Test Items/Items
AT

Additional Information/Tags
 Time expected to be taken to solve by an average student
 Sample answer
 Hints

N

Tags including, Course Outcome code, competency code, cognitive level, knowledge category,
difficulty level etc.

Assessment Items
 Items that require short/long written responses
 Quizzes
 Assignment Problems

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 1

 
 Simulations
 Laboratory experiments to be conducted
 Projects to be done
 Field work
 Reports to be written
 Presentations to be made

20
Assessment Instruments
 Are collections of assessment items
 Have a purpose and context
 Contain different types of items

20
Examples
 Quizzes
 Mid-term tests
 Final examinations




Group Projects

Types of Assessment
EL
Formative Assessment (Assessment for Learning or Educative Assessment)
Summative Assessment (Assessment of Learning)
PT
Types of Test Items

 Written Test Items

o Selection type: Response consists of making choices from given alternatives

-N

o Supply type: Response consists of giving a short or long answer

 Performance Test Items

Selection Type Items

 True/False  Rearrangement
 Multiple Choice  Checklists
E

 Multiple Selection  Rating Scales

 Matching Blocks
AT

Supply Type Items

 Completion  Structured Response
 Fill in the Blanks  Viva Voce
 Sketch  Numerical Questions
 Label  Detailed Answers
N

 Short Answers
Performance Type Test Items
 Experiments  Prototypes to be built
 Simulations  Things/objects/equipment to be tested
 Projects  Discussions

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 2

 
 Seminars/Presentations  Viva-voce
 Field studies  etc.
Quality of Assessment
Characterized by

 Validity
 Reliability

20
Validity: Degree to which assessment measures what it purports to measure.
Reliability: Degree to which assessment scores are consistent.

Summative Assessment Instrument

20
EL
PT
-N
E

 
Is Reliability a Relevant Issue?
AT

 A teacher may not teach the same course second time in the same college.
 There is considerable attrition of faculty.
 External paper setter may not follow all criteria of validity.
 In an affiliated university system if all the concerned persons (new teachers and external paper
N

setters) follow all the steps to achieve validity, it is possible to meet the requirements of
reliability.

Evaluation

 Scoring (1, 2, 5, 10 or more marks)

 Rubrics

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 3

 
Rubrics

 A rubric is a scoring tool for subjective assessments.

 Rubric is a set of criteria and standards, linked to competencies, used to assess a student’s
response on performance test items.
 Rubrics allow for standardized evaluation according to specified criteria, making grading simpler
and more transparent

20
 http://rubistar.4teachers.org/

Sample: Presentation

 Nonverbal Skills

20
o Eye contact, Facial Expressions and Posture

 Vocal Skills

o Enthusiasm, and Vocalized Pauses (uh, well uh, um)

EL
Content

o Topics announced, Time Frame, Visual Aids, Content Compliance, and Professionalism of
Content and Presentation

Difficulty
PT
 Difficulty Level refers to increased cognitive load, time to solve, larger numbers of facts to
recalled, number of concepts involved, more elaborate procedures to be used etc.
 Difficulty Level (learner independent and subject independent) is characterized by Content, Task
and Stimulus
-N

 Content difficulty is related to elements of knowledge: facts, concepts, procedures and

metacognition.
 Task difficulty refers to the difficulty that the students face when they generate their responses.
 Stimulus difficulty is related to the manner the item is presented to the students which includes
words, phrases and information which is packed along with the item.
E

Sample Items of different Difficulty Levels

 Determine the time period of a simple pendulum of length 1m on the surface of earth (DL1)
AT

 Determine the time period of a simple pendulum of length 1m when it is placed in a lift which is
moving upwards with an acceleration 2ms-2 (DL2)
 Determine the time period of a simple pendulum of length 1m with its bob dipped in a non-viscous
medium of density one tenth of bob, and is placed in a lift which is moving upwards with an
N

acceleration 2ms-2 (DL3)

Complexity and Difficulty

 Complexity refers to higher cognitive levels.

 Difficulty is not necessarily associated with higher cognitive levels.
 One can have lower level difficulty test items at higher cognitive levels.

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 4

 
 It will be a disservice if test items from relevant cognitive levels are not included in the name of
difficulty.

Sub-processes of Design Phase

In the context of an engineering course the proposed sub-processes and their sequence are
 Selecting the technology for assessment and evaluation
 Setting targets for CO attainment

20
 Designing the Assessment Pattern and Assessment Instruments
 Creating the Item Bank

Exercise

20
 List aspects of assessment not addressed in this Unit, but still considered relevant to your course.
 Describe any different sub-processes you consider necessary to be included in the Design Phase
of ADDIE with respect to designing your course.
Thank you for sharing the results of the exercise at [email protected]

M2 U6


EL
Understand the sub-processes of selecting technology for assessment and setting targets for CO
attainment.
PT
-N
E
AT
N

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 5

 
M2 U6: Technology for Assessment; Setting Targets
Recap

 Understood the Nature of Assessment

 Identified the sub-processes of Design Phase

M2 U6 Outcomes

M2 U6-1: Understand the nature and role of technology for assessment and evaluation.

20
M2 U6-2: Set targets for attainment of course outcomes.

ICT

can be used at several stages of assessment and evaluation in engineering programs

20
 Managing item banks (question banks)
 Creating assessment instruments
 Conducting examinations
 Aggregating the responses of students


 EL
Evaluation of student responses
Computing attainment of outcomes and grades
Learning analytics

Item (Question) Banks

PT
 There are different opinions on the role of Item Banks.
 A right sized and well managed Item Bank can greatly contribute to the quality learning.
 Well managed item banks at University level can have impact on very large number of students.
The key is well managed item banks.
-N

 An individual instructor in an autonomous institute will find it hard to create a good-sized item
bank. One can be built over time with collaboration and internet.
 Instructor’s item bank should be constantly upgraded to meet the needs of changing instructional
situation and the curriculum.

Good Item Banks

E

 Processes exist for designing and managing good item banks.

 Standards exist (QTI Standard from IMSGlobal) for item banks.
AT

 Technologies are available for designing and managing item banks.

 LMSs, such as MOODLE, can facilitate creation and management of item banks.

Creation of Assessment Instruments


N

Requires designing an assessment plan depending on the COs under consideration, weightages
selected for cognitive levels, marks allotted, and difficulty level.
 Tools are available to create multiple structures of assessment instruments as per the chosen
assessment plan.

Conducting Examinations
 Legally acceptable records related to conducting the examinations are to be created.

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 6

 
 Systems exist for conducting examinations in multiple places as required for affiliated and
non-autonomous colleges.
 Technologies exist transferring assessment instruments from a centralized place to multiple
examination centers in a secure fashion.
 Many autonomous institutions created their own systems for conducting examinations.

Aggregating the Responses of Students

20
 Systems and technologies exist to aggregate the responses of students from multiple
examination centers through scanning and encoding and distributing them to centralized
evaluation centers.
 Simple procedures are adequate for collecting the responses of students and making them

20
available to the evaluators in autonomous institutions.

Evaluation of Student Responses

 Technologies exist for evaluating the responses of students to multiple choice, multiple selection
and filling the blanks in a variety of contexts.




EL
Tools exist to flag the test items if large percentage of students did not correctly answer them.
Some tools have features to flag the answers that differ from correct spelling to items that require
filling the blanks.
An LMS like MOODLE and commercially available Academic Management Systems provide tools
PT
for evaluating responses to MCQ/MSQ items.
 If the SEE paper contains sections on objective items, and the colleges have computer-based
examination facility, the examination and evaluation of student performances can be done using
technology.
 Even if the college does not use such tools, the individual teacher can incorporate such features
-N

in their courses.

Computing attainment of outcomes and grades

 Commercial Academic Management Systems exist for computing the final grades and marks,
computing outcomes attainments.
 Simple tools based on spreadsheet can be easily created at the level of individual courses for
E

computing grades, attainment of outcomes, distribution of students under teacher defined

categories.
AT

Learning Analytics
 Academic Management Systems provide tools for learning analytics that can provide information
required by management, facilitate better placements, identify trends and generate data as
required by the accreditation and regulatory agencies.
N

Quizzes

 Teachers increasingly started using quizzes as summative assessment instruments in Continuous

Internal Evaluation (CIE) using open source tools like Google Forms.
 Manual methods associated with quizzes can be time consuming to teachers.
 Designing, conducting, evaluating and giving feedback to students on quizzes can be done very

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 7

 
 effectively using any LMS and Internet Devices.

Assignments
 Several colleges are using WhatsApp and LMS for interaction between teacher and students in
relation to assignments.
 Date and time of submission are communicated through WhatsApp.
 The assignment is communicated through group mails or through WhatsApp.

20
 Students submit their responses to assignments using camera images from smart phones. The
evaluation is manual.
 Tools offered by LMS can be effectively used to communicate with the students and give
personalized feedback.

20
Technology for Assessment and Evaluation
 The choice of technologies depends on the nature of subject, access, comfort levels of faculty and
students with the technology, instructional methods used, and system under which the courses
are offered.



EL
Some institutions, very small in number, started using assessment and evaluation tools offered
by open source Learning Management Systems (LMS).
Teachers can personalize the technology for courses using an LMS with some technical support.

Setting Targets
PT
 Targets need to be set for attainment of COs
 The performances of students need to be compared with the set targets.
 This comparison enables the teachers to plan for continuous improvement in learning.
 The targets influence the design of assessment instruments
-N

 There are several ways of setting targets presented in Module 1.

Setting CO Attainment Targets: A sample
 Targets are set for each CO of a course separately
 It does not directly indicate the distribution of performance among the students. However, it has
the advantage of finding out the difficulty of specific COs

CO Target (Class Average)

E

CO1 70%

CO2 80%
AT

CO3 75%

CO4 65%

CO5 70%

CO6 80%
N

Exercise
 List the specific technologies you used in connection with assessment in courses you taught.
Thank you for sharing the results of the exercises at [email protected]
M2U7

 Understand the process of designing the assessment plan for a course.

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 8

 
M2 U7: Assessment Plan and Assessment Instruments
Recap
 Understood the nature and role of technology for assessment and the process of setting targets
for attainment of COs.

M2 U7 Outcomes
 Understand the process of designing an assessment plan and assessment instruments for a

20
course.

Assessment
 The method of assessment varies from institution to institution.
 Assessment can be broadly divided into:

20
o Continuous Internal Evaluation (CIE) and
o Semester End Examination (SEE).

 The relative weightages given to CIE and SEE also vary considerably from Institution to
Institution. They may range from 20:80 to 60:40!


EL
Much of the assessment requires written responses.

Continuous Internal Evaluation (CIE)

 The number of assessment instruments that can be used and the variety of assessment
PT
instruments in CIE allowed depend on the guidelines provided by the Institute / Affiliating
University.
 In all cases, the Internal Assessment taken as a whole must address all the COs adequately.
 Ensuring this requires planning upfront.
-N

CIE Plan

 Step 1: Based on the available guidelines, the instructor must finalize the details of the
assessment instruments to be used and the schedule for administering these instruments for CIE.
 Example: Total CIE Marks: 50

Instrument Marks Schedule

E

Test 1 (T1) 15 Week 7

AT

Test 2 (T2) 15 Week 13

Quiz 1 (Q1) 5 Week 5

Quiz 2 (Q2) 5 Week 14

N

Assignment 1 (A1) 5 To be submitted in week 9

Assignment 2 (A2) 5 To be submitted in week 15

Note:
 The assessment plan may indicate 5 marks for Assignment 1. However, the actual assignment

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 9

 
 given to the students need not be for 5 marks! In such cases, the performance of students is
scaled appropriately afterwards.
 The same is true for any assessment instrument!
For each CO, we must decide:
 Marks to be allocated to this CO out of the total marks for CIE
 The distribution of these marks over the relevant assessment instruments
 The cognitive levels of the assessment items related to this CO

20
CIE Marks Allocation to COs

 Marks allocated to a specific CO out of total CIE marks is the choice of the Instructor with the only
constraint being that every CO must have non-zero marks allocated to it.

20
 However, Instructor may wish to consider the following two parameters in deciding on the
marks to be allocated.

o Proportion of classroom hours devoted to the CO

o Relative “importance” of this CO in later courses.

Example:



CO5

EL
Total number of class sessions = 56
Number of class sessions devoted to CO5 = 8 = 14%
Total CIE Marks = 50
PT
 14% of 50 = 7
 CO5 is perceived by Instructor to be quite important.
 CIE Marks allocated by Instructor to CO5 = 10.
 Completed allocation: (Total marks for CIE = 50)
-N

CO1: 8; CO2: 8; CO3: 8; CO4: 8; CO5: 10; CO6: 8.

Assessment Instruments for a CO

 CIE marks for each CO have been determined.

 Now, Instructor must decide on the distribution of these marks over the relevant assessment
instruments.
E

 Again this is the choice of the Instructor.

 Constraint: If a CO is to be addressed by an Assessment Instrument, the instructional material
AT

related to that CO must already have been completed before the scheduled time of that
assessment instrument.
 Example: T1 is scheduled for week 7. Lesson plan indicates that CO6 is planned for weeks 12
and13. Evidently CO6 can not be addressed by T1!
N

 Example of a valid plan:

Total Marks Quiz Marks T1/ Assessment Marks

CO1 8 Q1=3 TI=5

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 10

 
 CO6 8 Q2=3 A2=5

Distribution Over Relevant Cognitive Levels

 Cognitive Level of an Assessment Item is generally expected to be at the same cognitive Level as
that of the CO.
 However, we may also have assessment items from lower cognitive levels, for a variety of

20
reasons!
 The weightage given to assessment items belonging to the lower cognitive levels is decided by
the instructor.
 A useful rule of thumb: The weightage given to assessment items belonging to the lower

20
cognitive levels < ~40%.
Example:
 A total of 10 marks are planned for CO5.
 Assume that CO5 is at ‘Apply’ Cognitive Level
 One possible allocation of marks over cognitive levels could be:

o
o
Apply Level:

EL 6 Marks (60% of 10)

Lower Levels: 4 Marks (40% of 10)
(Understand Level: 2 Marks;
Remember Level: 2 Marks)
PT
Final Plan for addressing a CO in CIE

 Next we need to decide how these marks are allocated to the assessment instruments already
planned for a given CO.
-N

Example (continued): For CO5:

T2: Apply Level - 6 Marks
Understand Level - 2 Marks
Q2: Remember Level - 2 Marks

Summary of CIE Plan Process

E

 Determine the CIE instruments, marks for each instrument and the schedule for these
instruments
AT

 For each CO:

o Allocate marks for CIE.

o Distribute these marks over selected CIE instruments.
o Determine the marks for relevant cognitive levels.
N

CIE Plan – Example

CO CL T1 T2 Q1 Q2 A1 A2

CO1 U 5 - 3 - - -
(U: 5) (R: 3)

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 11

 
CO2 U 5 - 2 - 1 -
(U: 5) (R: 2) (R: 1)

CO3 Ap 5 - - - 3 -
(Ap: 5) (U: 3)

CO4 Ap - 7 - - 1 -
(U: 2; Ap: 5) (R: 1)

20
CO5 Ap - 8 - 2 - -
(U: 2; Ap: 6) (R: 2)

CO6 Ap 3 5

20
(U: 3) (Ap: 5)

SEE Plan Process

 There is only one Semester End Examination and thus there is only one assessment instrument
that needs to be planned.


 EL
The structure of SEE Instrument varies considerably from Institute to Institute.
In any case, this instrument must address all the COs!
Thus the plan includes the following steps:

o For each CO:

PT
– Allocate marks for SEE
– Determine the marks for relevant cognitive levels
The rationale for these two steps is same as the one used in CIE

o Combine the assessment items suitably based on the structure of the SEE instrument. (The
-N

structure depends on the guidelines in use.)

Example (continued):
 Total Marks: 100
 Allocation for COs:
CO1: 10; CO2: 10; CO3: 20; CO4: 20; CO5: 20; CO6: 20.
E

For CO5 (CO5 is assumed to be at Apply Cognitive Level):

 Marks allocated: 20
AT

 Apply Cognitive Level: 12 Marks

 Understand Cognitive Level: 6 Marks
 Remember Cognitive Level: 2 Marks

Choice in an Assessment Instrument

N

 “Choice” in an assessment instrument in CIE / SEE depends on the guidelines in force.

 It is important that the choice does not lead to any gaps in addressing the COs!
Example:
 Test 1 in CIE is designed to address CO1, CO2, and CO3.
 The question paper has 3 questions, one for each CO.

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 12

 
 Students are required to answer any 2 questions!
 Poor way of providing “choice”!
Example (Continued):
 Test 1 in CIE is designed to address CO1, CO2, and CO3.
 The question paper has 3 questions, one for each CO.
 Students are required to answer all the 3 questions!

20
 However, each question has 2 parts and student can answer either part. (“Internal choice”)
 Both parts in a question address the same
CO at same cognitive level(s) and are at the same level of difficulty.
 Much better way of providing “choice” in the Question paper!

20
Exercise

 Design T1 from CIE of your course as per the structure given.

 Design an SEE instrument as per the structure given.
Thank you for sharing the results of the exercise at [email protected]

M2 U8


EL
Understand the structure and use of item banks; and the process of designing item banks.
PT
-N
E
AT
N

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 13

 
M2 U8: Item Banks
Recap

 Understood the process of designing assessment plan and assessment instruments for an
engineering course.

M2 U8 Outcomes

 Understand the process of designing Item Banks for a course.

20
What is an Item Bank?

 Item Bank is also known as Question Bank in popular usage.

 It is a collection of assessment items organized according to the course outcomes and cognitive

20
levels.
 The purpose of an item bank is to meet the needs of designing assessment instruments for
quizzes, assignments, tests of CIE; and SEE.
 It may be convenient to design item banks separately for quizzes, assignments and CIE tests /




SEE.

EL
Items included in an Item Bank need to go through a formal or informal review process.
Items are tagged with several parameters.
Item Bank needs to be organized as a database if assessment instruments are to be generated
PT
using software tools.

Why are Item Banks needed?

 A survey of many assessment instruments from several Universities and Colleges across the
country shows the following issues with many assessment instruments:
-N

o Language ambiguities and technical inaccuracies

o Incompatibilities between assumed time required to respond and the scope of the question
o Uneven distribution of questions across COs (Units/topics) and cognitive levels
o Uneven difficulty levels
 The cognitive levels of assessment items are reduced to avoid some of these issues. This results
E

in reducing the quality of learning of all students.

 Item Banks help us overcome theses limitations!
AT

Advantages in having Item Banks

While it takes considerable effort to create good item banks for first time, there are several
advantages in having a good item bank.
 Faculty can save considerable time while designing good quality quizzes and class tests.
N

 Paper setters at the University level can greatly benefit from an item bank when setting
assessment instruments of good and uniform quality.
 Some or all of the processes associated with creating and administering assessment and
evaluation can be computerised.

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 14

 
Types of Item Banks

 India has Tier 1 and Tier 2 colleges with Tier 2 colleges being dominant
Item Banks for Tier 2 Colleges:
 Item banks for quizzes, assignments, and class tests are created and managed by a teacher or a
group of teachers.
 Item banks for SEE are created and managed at the University level.

20
Item Banks for Tier 1 Colleges:
 Item banks for all summative assessment instruments are created and managed by a teacher or
a group of teachers.

Assessment Items

20
 A wide variety of assessment items is possible as noted in an earlier unit.
 Quizzes mostly use objective (Select / Supply) questions.
 Some times, internal tests also may have some objective type questions.
 Some universities require SEE Question Paper to have a part that is composed of objective type


questions.

EL
Such objective questions are typically 1-mark questions. Occasionally, these may be 2-mark
questions.
Mostly, “Remember” and “Understand” are the cognitive levels associated with objective
PT
questions. Occasionally, we may have objective questions at “Apply” level also.
 Marks allocated to assessment items used in Assignments, Internal Tests and SEE depend on
specific Assessment Plans as discussed in an earlier unit. They may range from 1 mark to 20
marks! (with or without scaling of responses at the end).
-N

 Assessment items in Assignments tend to be mostly at “Apply” or higher cognitive levels

including “Analyze”, “Evaluate” and “Create”. Depending on the context some assignments may
have items belonging to Understand or even Remember cognitive levels.
 Assessment items used in Internal Tests and SEE tend to be at the levels of “Remember”,
“Understand”, or “Apply” only.

Creation of Item Banks

E

 No unique way of creating the Item Bank.

AT

 Here, we discuss general principles and useful guidelines.

 The examples presented are for the purpose of illustration only.
 Each institution must develop its own process for creating the item banks.
 The size of the Item Bank and the variety of assessment items in the Item Bank grow with time.
 However, initially, an Item bank must have some minimum number of items before it can be used
N

effectively for composing assessment instruments.

 Items addressing the same CO at the same cognitive level and having the same number of marks
are considered to be in one category.
 Suggested minimum number of items in each category = 5 * Required number in that category.
 Required number of items depends on the Assessment Plan as discussed in an earlier unit.

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 15

 
 It is suggested that the item bank be made more general than required strictly by the assessment
plan. This would make the item bank more useful in the long run.

Item Banks for Quizzes

 Quizzes are used both for formative and summative assessment.

 Quizzes normally consist 5 to 10 one-mark questions
 All quiz questions belong to Remember or Understand cognitive level.

20
 If all quiz items belong to categories that can be readily evaluated automatically by a tool like an
LMS, the quizzes can be very conveniently administered and evaluated. The results can be
discussed in the classroom without any delay, after the quiz is administered.
 Assume 5 quizzes and 5 questions per quiz.

20
 Total questions required = 25.
 If at least five times the required number of questions are to be included in the item bank, then
we need to have about 125 items in the item bank for the quizzes.
 The items in the Quiz Item bank are to be distributed over all the COs approximately in proportion


EL
to the number class sessions associated with each CO.
They can be distributed suitably over the relevant cognitive levels.

Item Banks for Assignments

Assignments dominantly will have items belonging to the cognitive level “Apply”.
PT
 In some cases, ill-defined problems and items belonging to Analyze, Evaluate and Create
categories can be given as assignments.
 Marks allocated to the items for assignment vary considerably.
 We must decide on the number of items, their cognitive levels, and the marks to be allocated to
-N

them based on the assessment plans.

 If on the average there are three items in an assignment, two assignments in a semester, and the
number items in the item bank needs to be at least five times the required number initially, then
the number of items in Assignment Item Bank should be 30.
 The items should be designed to address all the COs of the course at the concerned cognitive
E

levels.
 Items must be designed taking into account the cognitive levels as well as the marks to be
allocated to them.
AT

Item Banks for Tests

 All tests are designed, conducted and evaluated by the teacher of a course both in the Tier 1 and
Tier 2 institutions.
N

 The assessment plan, as discussed earlier, includes the COs addressed in a test and the marks
allocated to the COs at different cognitive levels.
 Teacher has to further decide on the composition of items for given marks.
 Any “choice” to be provided in the tests also needs to be taken in to account.
Example:
 Consider the assessment plan presented in the earlier unit.

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 16

 
 As per that plan, CO5 is addressed in Test 2; marks allocated are 8; and their break up is: 6
marks at “Apply” level and 2 marks at “Understand” level.
 Teacher decides to have 2 questions at “Apply” level, each for 3 marks, to get a total of 6 marks
for that level and one question of 2 marks at “Understand” level.
 There is to be internal choice. Both the questions involved in the choice are to be similar.
Example (continued):
Items required CO5, for Test 2, taking choice in to account:

20
 4 questions, each for 3 marks, at “Apply” level.
 2 questions, each of 2 marks, at “Understand” level.
Item Bank requires:

20
 5 * 4 = 20 questions, each for 3 marks, at “Apply” level.
 5 * 2 = 10 questions, each for 2 marks, at “Understand” level.
Total requirements for Item Banks for tests can be determined by considering all the COs and all the
tests as per the assessment plan.

Features of Item Banks for Tests


 EL
Teacher makes the choices at every stage.
The choices are based on her perception of the subject, her instruction, cognitive abilities of
students, and other contextual factors.
PT
 One structure is not applicable to all courses.

Item Banks for SEE

 SEEs are more structured than tests.

 Normally one structure is used for all courses of all programs.
-N

 Paper setter (University) and Teacher (Tier 1 Institution) has the choice of distributing the marks
allocated to a question over three or four sub-questions.
 Initial Item Bank can be designed with five times the required number items after taking “choice”
into account.

Sample Structure of SEE Instrument

E

CO CL Item A Item B

CO1 U 1A - 6U+4U 1B - 6U+4U

AT

CO2 U 1A - 6U+4U 1B - 6U+4U

CO3 Ap 2A - 4U+8Ap+8Ap 2B - 6U+6Ap+8Ap

CO4 Ap 3A - 6U+7Ap+7Ap 3B - 5U+8Ap+7Ap

CO5 Ap 4A - 4U+8Ap+8Ap 4B - 4U+8Ap+8Ap

N

CO6 Ap 5A - 4U+8Ap+8Ap 5B - 4U+8Ap+8Ap

Items Required for the Item Banks for SEE

Example: Consider CO5 and take “choice” in to account.

Required questions:
 4 questions, each for 8 marks, at “Apply” level

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 17

 
 2 questions, each for 4 marks, at “Understand” level
Item Bank must have:
 5 * 4 = 20 questions, each for 8 marks, at “Apply” level
 5 * 2 = 10 questions, each for 4 marks, at “Understand” level
Similar considerations apply to the other COs also.

Managing Item Banks

20
 All items need to get reviewed for the language, technical correctness and time required by an
average student.
 All items should be tagged with COs, CLs, marks, preferably difficulty levels, and sample
answers.

20
 The item bank should be suitable for manual usage also.
 The item bank should be kept dynamic by archiving about 10% of the items and adding about
10% new items every year.
 When the item bank is large enough, a representative segment of the item bank can be made


EL
visible to students.

Changing Views about Item Banks

Teachers and education administrators have started seeing the usefulness of well managed item
banks, though they were not acceptable a few years ago.
PT
 Item bank creation should always be group effort.
 Creating a good and large Item Bank at the University level requires considerable coordination.
The resulting benefits can be substantial.
 Use of software tools for management of item banks and also for generating good quality
-N

assessment instruments will reduce the load on the faculty.

Exercise

 Design the structures for items banks for quizzes, assignments, tests and SEE for your course
making your own assumptions.
Thank you for sharing the results of the exercise at [email protected]
E

M2 U9
AT

 Understand the processes of Development Phase.

N

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 18

 
M2 U9: Development Phase
Recap

 Understood the process of designing and managing Item Banks for quizzes, assignments, tests
and SEE for an engineering course.

M2 U9 Outcomes

 Understand the sub-processes of Development Phase for an engineering course.

20
Development Phase

Development Phase consists of

 Identifying the delivery technologies

20
 Choosing Instruction Type
 Development of Instructional Material
 Identification/selection of Learning Material

Delivery Technologies




EL
Some delivery technologies for consideration
Classroom with Blackboard/White Board/Smart Board
Classroom with LCD Projector
Electronic Classroom with LCD Projector, students with laptops/smart phones
PT
 Electronic Classrooms with Learning Management System (LMS)
 The choice of and access to a delivery technology will have a great influence on the instruction.
 ICT tools can open many possibilities.
 ICT enables the teacher to make the curated learning material available to the students.
-N

 All technologies have their advantages provided the instruction is adjusted to the technology.

Blackboard/Whiteboard

 This technology is the oldest and most prevalent.

 The pace and the sequence followed generally syncs with delivery of new information giving
enough time to the students to understand.
E

 It suits the courses that are dominantly mathematical.

 Complex figures can not be easily drawn on the boards.
AT

 Descriptive courses permit only key phrases to be written on the board.

Smart Board

 It enables the teacher to capture the material presented on the board as an image.
N

 It allows the teacher to interact with the material projected as slides.

 Teacher can highlight and write comments on the slides projected, and capture them together as
images.
 While it is adopted in some places, its use has not become that popular.

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 19

 
Classroom with LCD Projector

 It permits teacher to organize the information to be presented in a better manner and allows
her/him to show complex pictures readily.
 Pictures, animations and simulations can be included.
 The slides presented can be converted into doc/pdf format and shared with the students.
 The pace at which information is delivered can make some students mentally drop out.

20
 Crowded slides, and instructors merely reading the slides can completely demotivate students.

Electronic Classroom

 Student engagement can be significantly enhanced.

20
 Slides material can be shared in advance with students. Students can write their own notes and
tag them with slides.
 Teacher can ask questions and make all students respond, and immediately discuss the
responses.
 Make students simulate a problem and explore the behavior of a system with different


parameters.

EL
Requires considerable planning and work on the part of teachers.

Electronic Classroom with LMS

PT
 LMS, such as MOODLE which is open source, enables the teacher to create many activities that
engage students.
 LMS also enables the teacher to readily create and conduct quizzes and give feedback
immediately.
 LMS also enables the teacher to keep track of students’ performance without excessive effort.
-N

 LMS also enables the teacher to create documentation as needed by NBA/ NAAC.

Instruction Types

 Development of instruction material greatly depends on the choice of instruction type.

 Instruction Type
E

o Face-to-Face/Direct instruction in classroom sessions

o Blended Learning
AT

o Flipped Classroom
o On-line Courses/MOOCs

Face-to-Face/Direct Instruction

 This is the most popular type.

N

 It can use any of the delivery technologies.

 Teacher-student interaction plays a very important role in all learning activities in the classroom.
 Learning communities get created more easily.
 Both teachers and students associate instruction only with this type.
 Any deviation from this is not easily acceptable.

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 20

 
Blended Learning

 It is an approach to education that combines traditional place-based classroom methods with

online educational materials and opportunities for interaction online.
 Students still attend "brick-and-mortar" schools with a teacher present, and are involved in
face-to-face classroom practices at least some of the time.
 It was observed that student achievement was higher in blended learning experiences when

20
compared to either fully online or fully face-to-face learning experiences.

Flipped Classroom

 It is an instructional strategy that reverses the traditional learning environment.

20
 The content is often delivered online and the students are expected to study the material outside
the classroom.
 The activities, normally considered as homework, are moved into the classroom.
 In a flipped classroom, students watch online lectures, collaborate in online discussions, or carry
out research at home while engaging in concepts and problem solving in the classroom with the

EL
guidance of the teacher.

Online Courses/MOOCs

 All teaching and learning activities and assessments are done online without any face-to-face
PT
interaction between teachers and students.
 Instruction is through video sessions.
 Students can learn at their own pace.
 Sometimes it is confined to registered students of one institution.
 They can be offered to interested students across the country or across the world like in NPTEL.
-N

 When the numbers are large some additional support systems need to be created.

Instruction Material

 Instruction material is what the instructor uses for facilitating the students to achieve the stated
COs/Competencies.
E

 Instructional material is organized as per Instructional Units associated with COs/Competencies.

 The components of instruction material will depend on instruction method followed. Several
instructional methods will be explored in the later Units of NATE.
AT

 Instruction material will also depend on the time allocated to each element.
Instruction material consists of
 Presentations (oral/ppt) the instructor wants to make
 Problems he/she wants to solve in the classroom
N

 Quizzes the instructor wishes to conduct

 Discussions proposed to be conducted
 Group activities proposed to be conducted
 Anecdotes to be presented

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 21

 
Delivery Technology and Instruction Type

 The choice of delivery technology and the instruction type chosen will influence the instruction
design.
 Instruction design, dealt in greater detail in later Units, will determine the structure and contents
of instruction material.

Instructor and Instruction Design

20
 Every instructor prepares instructional material according to which she conducts classroom
activities.
 Instructional material has a plan of the sequence of activities (can be called a script) and how

20
those activities are going to be executed (can be called dialogues).
 Instructor has freedom to write the script (the sequence and nature of activities) in her own way
to facilitate the students to attain the stated competency.
 The framework within which a script is written is called Instruction Design
 There are several instruction design frameworks.

Learning Material



EL
Learning material is what learners use.
It is selected from books and internet sources, and if necessary supplemented by material
PT
prepared by the Instructor.
 It is selected/prepared for each Instructional Unit.
 Some Academic Management Systems permit curating material from textbooks and related video
material from online resources.
-N

M2 U10

 Understand the nature and sub-processes of Implement Phase

E
AT
N

NATE-Module 2-Week 6 N J Rao & K Rajanikanth 22