Training Evaluation: Module 6

DR A HANS
 Introduction (1 of 2)

Training effectiveness refers to the benefits that

the company and the trainees receive from training

Training outcomes or criteria refer to measures

that the trainer and the company use to evaluate
training programs
 Introduction (2 of 2)

Training evaluation refers to the process of

collecting the outcomes needed to determine if
training is effective

Evaluation design refers to from whom, what,

when, and how information needed for determining
the effectiveness of the training program will be
collected
 Reasons for Evaluating Training (1 of 2)

1. Companies are investing millions of dollars in

training programs to help gain a competitive
advantage

2. Training investment is increasing because learning

creates knowledge which differentiates between
those companies and employees who are successful
and those who are not
Reasons for Evaluating Training (2 of 2)

1. Companies have made large dollar investments in

training and development and view training as a
strategy to be successful
2. They expect the outcomes or benefits related to
training to be measurable.
Training evaluation provides the data
needed to demonstrate that training
does provide benefits to the company.
 Formative Evaluation

1. Formative evaluation – evaluation conducted

to improve the training process
 Helps to ensure that:
a) the training program is well organized and runs smoothly
b) trainees learn and are satisfied with the program
 Provides information about how to make the
program better
 Summative Evaluation

1. Summative evaluation – evaluation conducted

to determine the extent to which trainees have
changed as a result of participating in the training
program

 May also measure the return on investment (ROI)

that the company receives from the training
program
 Self –Learning

Q1. Compare the importance of summative &

formative evaluation.
Why Should A Training Program Be
Evaluated? (1 of 2)

1. To identify the program’s strengths and

weaknesses
2. To assess whether content, organization, and
administration of the program contribute to
learning and the use of training content on the job
3. To identify which trainees benefited most or least
from the program
Why Should A Training Program Be
Evaluated? (2 of 2)

4. To gather data to assist in marketing training

programs
5. To determine the financial benefits and costs of the
programs
6. To compare the costs and benefits of training
versus non-training investments
7. To compare the costs and benefits of different
training programs to choose the best program
 Self-Learning

Q1.Why training should be evaluated? Justify

The Evaluation Process

Conduct a Needs Analysis

Develop Measurable Learning Outcomes and

Analyze Transfer of Training

Develop Outcome Measures

Choose an Evaluation Strategy

Plan and Execute the Evaluation

 Training Outcomes: Kirkpatrick’s Four-
Level Framework of Evaluation Criteria
Level Criteria Focus
1 Reactions Trainee satisfaction

2 Learning Acquisition of knowledge, skills, attitudes, behavior

3 Behavior Improvement of behavior on the job

4 Results Business results achieved by trainees

Outcomes Used in Evaluating Training
Programs: (1 of 4)

Cognitive Skill-Based
Outcomes Outcomes

Return on
Affective Results Investment
Outcomes
Outcomes Used in Evaluating Training
Programs: (2 of 4)

Reaction Outcomes: Trainees perception of the

program including facilities(admin), trainers &
content. (questionnaire )

Cognitive Outcomes(Learning)
 Determine the degree to which trainees are familiar with the
principles, facts, techniques, procedures, or processes
emphasized in the training program
 Measure what knowledge trainees learned in the program
(Paper –pencil tests, Quizzes)
Skill-Based Outcomes (Level 2-3)
1. Assess the level of technical or motor skills
2. Include acquisition or learning of skills and use of skills on
the job
3. Can be evaluated by observing the performance on the job
by immediate supervisors/peers/subordinate/self
Outcomes Used in Evaluating Training
Programs: (3 of 4)

Affective Outcomes
1. Include attitudes and motivation to learn & safety attitude,
engagement
2. Trainees’ perceptions of the program including the facilities,
trainers, and content
3. Survey methods can be used (Attitude towards career goals,
plans & interests)
Results
 Determine the training program’s payoff for the
company( Productivity , Quality & Cost reduction,
satisfaction)
Outcomes Used in Evaluating Training
Programs: (4 of 4)

Return on Investment (ROI)

 Comparing the training’s monetary benefits with the cost of
the training
1. Direct costs( Salaries& Benefits)
2. Indirect costs (office supplies , facilities,
equipment, contextual expenses(travel)
3. Benefits(tangible + Intangible value gained
from the program)
 Self- Learning

Q1. What are the outcomes used to evaluate the

effectiveness of training programme?
Q2. According to you which out is the best outcomes
to measure the actual effectiveness of T&D
Programmes?
How do you know if your outcomes are good?

Good training outcomes need to be:

1. Relevant
2. Reliable
3. Discriminative
4. Practical
 Good Outcomes: Relevance

1. Criteria relevance – The extent to which

training programs are related to learned
capabilities emphasized in the training
program(outcomes related to training
objective)
2. Criterion contamination – Extent that
training outcomes measure inappropriate
capabilities or are affected by extraneous
conditions (external factors)
3. Criterion deficiency – Failure to measure
training outcomes that were emphasized in
the training objectives
 Criterion deficiency, relevance, and
contamination:

Outcomes Identified
Outcomes Related by Needs
Outcomes Assessment and
Measured in to Training
Objectives Included in Training
Evaluation Objectives

Contamination Relevance Deficiency

 Good Outcomes (continued)

4. Reliability – degree to which outcomes can

be measured consistently over time
5. Discrimination – degree to which trainee’s
performances on the outcome actually reflect
true differences in performance
6. Practicality – refers to the ease with which
the outcomes measures can be collected
Percentage of Courses Using Outcome Training Evaluation Practices

80% 79%
70%
60%
50%
40% 38%
30%
20% 15% 9%
10%
0%
Reaction Cognitive Behavior Results

Outcomes
 Training Program Objectives and Their
Implications for Evaluation:
Objective

Learning Transfer

Outcomes
Reactions: Did trainees like the program? Skill-Based: Ratings by peers or managers
Did the environment help learning? based on observation of behavior
Was material meaningful?

Cognitive: Pencil-and-paper tests Affective: Trainees’ motivation or job attitudes

Skill-Based: Performance on a work sample Results: Did company benefit through sales,
quality, productivity, reduced
accidents, and complaints?
Performance on work equipment
 Evaluation Designs: Threats to Validity

Threats to validity refer to a factor that will lead

one to question either:
1. The believability/credibility of the study results (internal
validity)
or
1. The extent to which the evaluation results are generalizable to
other groups of trainees and situations (external validity)
 Threats to Validity

Threats To Internal Threats To External

Validity Validity
1. Company 1. Reaction to pretest
2. Persons 2. Reaction to evaluation
3. Outcome Measures 3. Interaction of selection
and training
4. Interaction of methods
Methods to Control for Threats to Validity

Pre- and Posttests

Use of Comparison Groups

Random Assignment
 Types of Evaluation Designs

Posttest – only Time Series

Pretest / Posttest Time Series with

Comparison Group
Posttest – only with and Reversal
Comparison Group
Solomon Four–
Pretest / Posttest Group
with Comparison
Group
 Comparison of Evaluation Designs
(1 of 2)
Measures
Design Groups Pre-training Post-training Cost Time Strength

Posttest Only Trainees No Yes Low Low Low

Pretest / Posttest Trainees Yes Yes Low Low Medium

Posttest Only with Trainees and No Yes Medium Medium Medium

Comparison Group Comparison

Pretest / Posttest with Trainees and Yes Yes Medium Medium High
Comparison Group Comparison
 Comparison of Evaluation Designs
(2 of 2)
Measures
Design Groups Pre-training Post-training Cost Time Strength

Time Series Trainees Yes Yes, several Medium Medium Medium

Time Series with Trainees and Yes Yes, several High Medium High
Comparison Group Comparison
and Reversal

Solomon Four-Group Trainees A Yes Yes High High High

Trainees B No Yes
Comparison A Yes Yes
Comparison B No Yes
 Example of a Pretest / Posttest Comparison
Group Design:
Pre-training Training Post-training Post-training
Time 1 Time 2
Lecture Yes Yes Yes Yes

Self-Paced Yes Yes Yes Yes

Behavior Yes Yes Yes Yes

Modeling

No Training Yes No Yes Yes

(Comparison)
 Example of a Solomon Four-Group
Design:
Pretest Training Posttest
Group 1 Yes IL-based Yes

Group 2 Yes Traditional Yes

Group 3 No IL-based Yes

Group 4 No Traditional Yes

 Factors That Influence the Type of
Evaluation Design
Factor How Factor Influences Type of Evaluation Design
Change potential Can program be modified?
Importance Does ineffective training affect customer service, product development, or
relationships between employees?
Scale How many trainees are involved?
Purpose of training Is training conducted for learning, results, or both?
Organization culture Is demonstrating results part of company norms and expectations?
Expertise Can a complex study be analyzed?
Cost Is evaluation too expensive?
Time frame When do we need the information?
Conditions for choosing a rigorous evaluation
design: (1 of 2)

1. The evaluation results can be used to change the

program
2. The training program is ongoing and has the
potential to affect many employees (and
customers)
3. The training program involves multiple classes
and a large number of trainees
4. Cost justification for training is based on
numerical indicators
Conditions for choosing a rigorous evaluation
design: (2 of 2)

5. You or others have the expertise to design and

evaluate the data collected from the evaluation
study
6. The cost of training creates a need to show that it
works
7. There is sufficient time for conducting an
evaluation
8. There is interest in measuring change from pre-
training levels or in comparing two or more
different programs
Importance of Training Cost Information

1. To understand total expenditures for

training, including direct and indirect costs
2. To compare costs of alternative training
programs
3. To evaluate the proportion of money spent
on training development, administration,
and evaluation as well as to compare monies
spent on training for different groups of
employees
4. To control costs
To calculate return on investment (ROI), follow
these steps: (1 of 2)

1. Identify outcome(s) (e.g., quality, accidents)

2. Place a value on the outcome(s)
3. Determine the change in performance after
eliminating other potential influences on
training results.
4. Obtain an annual amount of benefits
(operational results) from training by
comparing results after training to results
before training (in dollars)
To calculate return on investment (ROI), follow
these steps: (2 of 2)

5. Determine training costs (direct costs +

indirect costs + development costs +
overhead costs + compensation for trainees)
6. Calculate the total savings by subtracting the
training costs from benefits (operational
results)
7. Calculate the ROI by dividing benefits
(operational results) by costs
 The ROI gives you an estimate of the dollar return
expected from each dollar invested in training.
Determining Costs for a Cost-Benefit Analysis:

Direct Costs Indirect Costs

Compensation
Development Overhead for
Costs Costs Trainees
 Example of Return on Investment
Industry Training Program ROI
Bottling company Workshops on managers’ roles 15:1

Large commercial bank Sales training 21:1

Electric & gas utility Behavior modification 5:1

Oil company Customer service 4.8:1

Health maintenance Team training 13.7:1

organization
 Important:

1. PPTs are mainly highlighting the conceptual

dimensions & provide a broader framework of
the subject contents.
2. Consulting/Reading Books related to the subjects
and other similar resources(online and hardcopy)
are mandatory for better and in-depth
understanding of the subject contents.