CTR Level 3 Presentation Evaluation Kirkpatrick

Capture
Elusive Level 3 Data:

The Secrets of
Survey Design
Presented by:
Ken Phillips
Phillips Associates
March 22, 2018
Phillips Associates 1
Agenda
1. Examine Level 3 evaluation facts

2. Analyze survey creation errors in a
sample Level 3 evaluation
3. Discover 12 tips for creating valid,

scientifically sound Level 3 evaluations
Kirkpatrick / Phillips Evaluation Model
Degree to which participants find the training
Level 1: Reaction
favorable, engaging, and relevant to their jobs
Degree to which participants acquire the intended

knowledge, skills, attitude, confidence
Level 2: Learning
and commitment based on their participation in the
training
Degree to which participants apply what they learned

Level 3: Behavior
during training when they are back on the job
Degree to which targeted outcomes improve as a

Level 4: Results result of the training, and the support and
accountability package
Degree to which monetary program benefits exceed

Level 5: ROI
program costs
Level 3 Evaluation Facts
Organizationsview
Organizations
of live
of classroom
18%
tech based
60%
75%
33% evaluate
programs
asprograms
some
data collected
programsbeing
evaluated
at Level
or very
being
having high
evaluated
high3value
Source: ATD Research Study, “Evaluating Learning Getting to Measurements That Matter,” 2015
Data Collection Methods
Source: Donald & James Kirkpatrick, “Evaluating Training Programs: The Four Levels,” 2006.
5
Possible Survey Respondents
Direct
Peers/ External
reports
Learners Colleagues customers
Managers
of learners
How to Decide
Who has first-hand knowledge of

learners’ behavior?
How credible do results need to be?
Sample
Level 3
Participant Survey
Instructions
1. Form a group of 3, 4 or 5 persons

2. Review sample Level 3 participant survey in
handout and see how many different survey
creation errors you can find (Hint: 9 different
errors are built into the survey)
3. Be prepared to discuss your findings with the

whole group
Note: Survey respondents are the direct reports

of managers/ supervisors who attended an
interpersonal feedback learning program.
Format
Content
Measure
ment
Scientifically Sound
Survey Design
Content
Phillips Associates
What’s Wrong With These?
8. Before providing employees with feedback

about their job performance, my manager
considers whether or not he or she is
knowledgeable about their job.
25. When giving feedback to an employee my

manager considers whether it should be
done privately or in the presence of others.
Tip 1: Content
Focus on observable behavior

not thoughts or motives.
Source: Palmer Morrel-Samuels, “Getting the Truth into Workplace Surveys”,

Harvard Business Review, 2002.
14. My manager gives his or her employees

feedback just as soon as possible after an
event has happened and avoids getting
emotional or evaluative.
18. My manager provides employees with

regular ongoing feedback about their
job performance and speaks in a normal
conversational tone or manner when
delivering the feedback.
Tip 2: Content
Limit each item to a

single description of behavior.
Source: Palmer Morrel-Samuels, 2002
Example
My manager gives his or her employees

feedback just as soon as possible after
an event has happened.
My manager avoids getting emotional or

evaluative when giving feedback to his or
her employees.
2. My manager doesn’t get to know his or

her employees as individuals before
providing them with feedback about their
job performance.
7. When giving employees feedback about

their job performance, my manager
doesn’t distinguish between patterns of
behavior and random one-time events.
Tip 3: Content
Word about 1/3 of the survey items so that

the desired answer is negative.
Format
Building Trust Feedback Timing
Credibility Feedback Frequency
Feedback Sign Message Characteristics
Tip 4: Format
Keep sections of the survey unlabeled.
Tip 5: Format
Design sections to contain

a similar number of items
and questions to contain
a similar number of words.
Tip 6: Format
Place questions regarding

respondent demographics
(e.g. name, title, department, etc.)
at end of survey, make completion optional
and keep questions to a minimum.
Measurement
Tip 7: Measurement
Collect data from multiple observers

or a single observer multiple times.
Source: Ken Phillips, “Capturing Elusive Level 3 Data: The Secrets of Survey Design”,
Unpublished Article, 2013.
What’s Wrong With This?
Strongly Strongly
Agree Agree Disagree N/A
Disagree
4 3 2 1
Tip 8: Measurement
Create a response scale

with numbers at regularly spaced intervals
and words only at each end.
*Palmer Morrel-Samuels, 2002.

Examples
This:
Not at all Completely
True True
1 2 3 4 5 6 7
Not This:
Not at all Rarely Occasionally Somewhat Mostly Frequently Completely
True True True True True True True
1 2 3 4 5 6 7
Or This:
Not at all Rarely Occasionally Somewhat Mostly Frequently Completely
True True True True True True True
Tip 9: Measurement
Use only one response scale

with an odd number of points
(7, 9 & 11 point scales are best)

Odd vs. Even Scale
This:
Not This:
Tip 10: Measurement
Use a response scale

that measures frequency
not agreement or effectiveness.
Examples
This:
Never Always
1 2 3 4 5 6 7
Or this:
True True
1 2 3 4 5 6 7
Tip 11: Measurement
Place small numbers

at left or low end of scale
and large numbers at right
or high end of scale.
Source: Ken Phillips, 2013
Examples
This:
True True
1 2 3 4 5 6 7
Not This:
Completely Not at all
True True
7 6 5 4 3 2 1
Tip 12: Measurement
Include a
“Did Not Observe” response choice
and make it different.

Example
Not at all Completely Did Not

True True Observe
1 2 3 4 5 6 7
Summary: Content
Focus on observable behavior
Limit ideas to a single description of behavior
Word 1/3 of items as reverse score
Summary: Format
Keep survey sections unlabeled
Design sections to contain similar number of

items & questions similar number of words
Place questions regarding respondent

demographics at end of survey, make
completion optional and keep questions
to a minimum
Summary: Measurement
Collect data from multiple observers or

multiple times
Create a response scale that:

 Has words only at each end
 Has an odd number of points
 Measures frequency
 Has small numbers at left and large numbers
at right
 Includes a “Did Not Observe” that is different
The difference between a good survey
and a bad one… quite simply, is careful
and informed design.
Free Articles
Phillips, Ken, “Eight Tips on Developing Valid Level 1
Evaluation Forms”, Training Today, Fall 2007, pps. 8 & 14.
Phillips, Ken, “Developing Valid Level 2 Evaluations”, Training

Today, Fall 2009, pps. 6-8.
Phillips, Ken, “Capturing Elusive Level 3 Data: The Secrets of

Survey Design”, Unpublished article, 2013.
Phillips, Ken, “Level 1 Evaluations: Do They Have a Role in

Organizational Learning Strategy?”, Unpublished article, 2013.
Phillips, Ken, “Business Results Made Visible: Designing Proof

Positive Level 4 Evaluations”, Unpublished article, 2013.
Ken Phillips
Phillips Associates
[email protected]
(847) 231-6068
www.phillipsassociates.com
34137 N. Wooded Glen Drive

Grayslake, Illinois 60030

CTR Level 3 Presentation Evaluation Kirkpatrick

Uploaded by

Copyright:

Available Formats

CTR Level 3 Presentation Evaluation Kirkpatrick

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CTR Level 3 Presentation Evaluation Kirkpatrick

Uploaded by

Copyright:

Available Formats

Capture

Elusive Level 3 Data:

1. Examine Level 3 evaluation facts

3. Discover 12 tips for creating valid,

Degree to which participants acquire the intended

Degree to which participants apply what they learned

Degree to which targeted outcomes improve as a

Degree to which monetary program benefits exceed

Who has first-hand knowledge of

How credible do results need to be?

1. Form a group of 3, 4 or 5 persons

3. Be prepared to discuss your findings with the

Note: Survey respondents are the direct reports

8. Before providing employees with feedback

25. When giving feedback to an employee my

Focus on observable behavior

Source: Palmer Morrel-Samuels, “Getting the Truth into Workplace Surveys”,

14. My manager gives his or her employees

18. My manager provides employees with

Limit each item to a

Source: Palmer Morrel-Samuels, 2002

My manager gives his or her employees

My manager avoids getting emotional or

2. My manager doesn’t get to know his or

7. When giving employees feedback about

Word about 1/3 of the survey items so that

Source: Palmer Morrel-Samuels, 2002

Building Trust Feedback Timing

Credibility Feedback Frequency

Feedback Sign Message Characteristics

Keep sections of the survey unlabeled.

Source: Palmer Morrel-Samuels, 2002

Design sections to contain

Source: Palmer Morrel-Samuels, 2002

Place questions regarding

Source: Palmer Morrel-Samuels, 2002

Collect data from multiple observers

Create a response scale

*Palmer Morrel-Samuels, 2002.

Use only one response scale

*Palmer Morrel-Samuels, 2002.

Use a response scale

Source: Palmer Morrel-Samuels, 2002

Place small numbers

Source: Ken Phillips, 2013

*Palmer Morrel-Samuels, 2002.

Not at all Completely Did Not

Focus on observable behavior

Limit ideas to a single description of behavior

Word 1/3 of items as reverse score

Keep survey sections unlabeled

Design sections to contain similar number of

Place questions regarding respondent

Collect data from multiple observers or

Create a response scale that:

Source: Palmer Morrel-Samuels, 2002

Phillips, Ken, “Developing Valid Level 2 Evaluations”, Training

Phillips, Ken, “Capturing Elusive Level 3 Data: The Secrets of

Phillips, Ken, “Level 1 Evaluations: Do They Have a Role in

Phillips, Ken, “Business Results Made Visible: Designing Proof