APPENDIX A: FROM START TO FINISH -- SAMPLE

SCENARIOS OF USING AND/OR CONSTRUCTING

SURVEY INSTRUMENTS

Appendix A includes two examples of how providers may use the survey instruments
and/or their subscales included in this Guide. One example shows how a nursing home
decided to measure one topic of interest (Job Design) among CNAs. The other
illustrates how a continuing care retirement community (CCRC) constructed a multi-
topic survey instrument from among the scales/subscales in the Guide. Both scenarios
follow the steps laid out in Appendix C on data collection planning and implementation
issues.

Measuring a Single Topic of Interest

Step #1: Purpose of data collection effort

A nursing home is experiencing high turnover among its CNAs. The Administrator
wants to identify the parts of their jobs that CNAs are most concerned about and those
that are least rewarding. Using this information, she would like to decide what actions
management can take to try to address some of these problems.

Step #2: Specify the target population for data collection

CNAs in nursing home

Step #3: Determine project team, budget and schedule

The Administrator has started to call the nursing departments at local universities in an
attempt to identify potential researchers with whom she can collaborate. She has also
asked her Director of Human Resources to obtain price quotes from data collection
vendors for conducting an employee survey of all 40 CNAs. Lastly, she has asked her
Director of Finance to assess what kind of budget the organization has for staff
development as she realizes she will have to act on the survey findings in order to
maintain credibility among her CNAs. The survey fielding/data collection period will last
for three weeks.

Step #4: Decide whether to include all members of the population or a sample

All 40 CNAs will be surveyed (a census).

A-1
Step #5: Decide the topics, subscales, and/or formulas on which to collect data

The nursing home Administrator has heard rumors about tension between CNAs and
certain charge nurses. While she knows for certain that this topic area is one that her
CNAs will be asked about, she really wants to keep the employee survey broad so she
can really get at what may be causing the high CNA turnover she is experiencing.

Step #6: Decide how the questionnaire will be administered and set the response
rate goal

The Administrator was able to form a relationship with a local researcher who will
oversee the data collection process. Each CNA will receive an advance letter informing
them of the survey and its goals. After the letters are distributed, a CNA staff meeting
will be held to allow a question-and-answer period focused on the survey. The project
team has determined that the survey will be administered by the researcher at pre-
appointed times for each employee on each shift in a common area. Free food will be
available in this common area during these times. A lock box will be placed in this room
so that employees will feel comfortable responding. The goal is to have a 100 percent
response rate among CNAs.

Step #7: Design and pretest the questionnaire

Together, the Administrator and researcher examined the instruments in the Guide
around the topic areas of job satisfaction, job design and worker-supervisor
relationships. Using the advice of her research collaborator, the Administrator has
decided to keep the survey broad and use subscales of the Job Role Quality
Questionnaire. She is particularly interested in assessing the degree to which her
CNAs are concerned about the workplace environment and the degree to which they
find certain aspects of their jobs rewarding in order to inform an appropriate
organizational response. The JRQ includes five items to measure “job concern factors”
and six items to measure “job reward factors.”

The Director of Nursing has recruited CNAs on each shift with whom the research
collaborator, an objective outside source, will hold focus groups to get feedback on the
length and content of the questionnaire.

Step #8: Monitor data collection

The researcher has agreed to provide frequent reports on response rates among the
CNAs via email and phone calls with the Administrator, Director of Nursing, and Director
of Human Resources. The survey will be conducted over a three-week period.

A-2
Step #9: Analyze data and present findings

Scores for each item in the eleven subscales were added across the 40 CNAs and then
averaged. The results for each item are as follows:

Subscale averages for job concern factors (N = 40 CNAs)

– Overload 2.8
– Dead-end job 2.5
– Hazard exposure 1.1
– Poor supervision 3.4
– Discrimination 1.4

*For job concern factors, a lower score reflects better job design.

Subscale averages for job reward factors (N = 40 CNAs)

– Helping others 3.8
– Decision authority 3.0
– Challenge 2.9
– Supervisor support 1.8
– Recognition 1.9
– Satisfaction with salary 2.2

* For job reward factors, a lower score represents poorer job design.

The results of the survey show that the Administrator’s suspicion that the employee-
supervisor relationships may need to be strengthened has been reinforced. Based on
survey results, it appears that CNAs seem most concerned about the poor supervision
they receive. They report the least rewarding parts of their jobs to be supervisor
support and recognition they receive.

The Administrator and Director of Nursing are putting together a presentation for the
next CNA staff meeting to report the survey results. At that time, they will solicit CNAs
who would like to work on a team to develop a strategic plan for improving employee
supervisor relationships and the overall work environment of the nursing home.

Constructing a Multi-Topic Survey Instrument

Step #1: Purpose of data collection effort

A CCRC wants to see how committed its employees are, how empowered they feel,
and whether those who feel more empowered are more likely to be committed to their
employer. The Administrator would like to see how employees’ perceptions in these
areas differ across department so that an informed organizational response can be
developed.

A-3
Step #2: Specify the target population for data collection

A random sample of employees in all departments of a CCRC.

Step #3: Determine project team, budget and schedule

This CCRC has a research unit on campus, so the Administrator will work with the
Director of Research on campus to develop a reasonable schedule. The Administrator
will also coordinate with the Director of Finance so the appropriate distribution of
resources across the CCRC and its research unit is clearly spelled out among all
parties.

The team determined ahead of time that the survey will be administered in-person,
since many of the employees are Spanish-speaking. Added expenses the team has
already considered include the hiring of outside interpreters, time staff spends on
completing surveys (and the overhead costs associated with a lengthy survey process
as a result), and efforts to increase response rate. The survey fielding/data collection
period will last six weeks, including survey administration and follow-up to improve
response rate.

Step #4: Decide whether to include all members of the population or a sample

Given the cost of doing in-person interviews and the number of staff members at the
CCRC, the Administrator and Director of Research decided to draw a random sample of
the 1,100 employees at the CCRC. Employees from all departments, on all shifts will
be included in the random sample. The research unit will ensure that enough
employees are drawn from each department to make appropriate comparisons and to
ensure confidentiality.

Step #5: Decide the topics, subscales, and/or formulas on which to collect data

The Administrator and department heads determined that organizational commitment

and empowerment among employees were the topic areas most appropriate to focus on
for this first employee survey. After looking at these topic areas in the Guide, the team
narrowed their choices to the following scales and subscales: Intent to Turnover
measure (behavioral intent to leave job) from the Michigan Organizational Assessment
Questionnaire and three items from the opportunity subscale of the Conditions for Work
Effectiveness Questionnaire (CWEQ I) and (CWEQ II Short Form). A total of 6 items
were chosen to keep the in-person survey short.

A-4
The specific items selected, which include entire subscales,1 are:

Items from the Michigan Organizational Assessment Questionnaire -- Intent to Turnover

measure (3 items):

Here are some statements about you and your job. How much do you agree or
disagree with each? (Likert Scale ranging from 1-7, where 1 = strongly disagree and 7
= strongly agree.)

Item #1. I will probably look for a new job in the next year.

Item #2. I often think about quitting.

Please answer the following question.

Item #3. How likely is it that you could find a job with another employer with about the
same pay and benefits you now have? (Likert Scale ranging from 1-7, where 1
= not likely at all and 7 = extremely likely)

Items from the Conditions for Work Effectiveness Questionnaire (CWEQ I) and (CWEQ
II Short Form) -- Opportunity subscale (3 items):

Items #1-#3. How much of each kind of opportunity do you have in your present job?
(Likert Scale ranging from 1-5, where 1 = None and 5 = A Lot)
• Challenging work.
• The chance to gain new skills and knowledge on the job.
• Tasks that use all of your own skills and knowledge.

Step #6: Decide how the questionnaire will be administered and set the response
rate goal

The team determined that it would like a 70-percent response rate across the CCRC.
The questionnaire will be administered in-person. The Director of Human Resources
will work with each department head to schedule times for employee surveys. At least 2
interviewers will be available per shift. Survey interviews will be conducted on campus,
in areas away from the departments in which interviewed employees work. Advance
letters and reminder postcards will be mailed to selected staff at their residence, flyers
will be posted throughout the campus and a “Share your Voice!” kick-off meeting will be
held the week survey administration begins where refreshments will be served and door
prizes given out.

1
It is important to include all items in a subscale because our review and the findings on the properties of the
instruments reported in this Guide are based on the entire subscales (not individual items within each subscale). If
you choose to take only some items from a subscale, the properties we reported (e.g., reading level, reliability,
validity) do not apply to the individual items.

A-5
Step #7: Design and pretest the questionnaire

Because this CCRC has many Spanish-speaking employees, cognitive testing will be
done on the survey item translation. Pretesting will also be conducted with 10 English-
speaking employees.

Step #8: Monitor data collection

Survey fielding will be conducted over a six-week period. The research unit on the
CCRC campus and the department heads will meet weekly to discuss the progress of
data collection. That way, the team is updated on the survey response rate and can,
subsequently, strategize on what types of efforts are needed to increase the response
rate (if any).

Step #9: Analyze data and present findings

Below is an illustrative example of the scores that correspond to answers three workers
gave to the six questionnaire items. The CCRC will tabulate the results of all
responding employees (of the random sample) using this same scoring process.

Worker ID Intent to Turnover Items Conditions for Work Effectiveness

(response scale ranges from 1 to 7) Questionnaire II Items
(response scale ranges from 1 to 5)
Item #1 Item #2 Item #3 Item #1 Item #2 Item #3
Worker #1 2 3 3 4 3 3
Worker #2 6 4 2 4 2 2
Worker #3 3 3 4 5 4 3

To calculate the score for each employee for the behavioral intent to leave job subscale,
sum the scores given for all three items. In this example, below are the scores for each
worker on this organizational commitment measure.

Worker #1: 8 = (2 + 3 + 3)
Worker #2: 12 = (6 + 4 + 2)
Worker #3: 10 = (3 + 3 + 4)

Lower scores on this measure indicate greater organizational commitment, with

possible scores on this 3-item measure ranging from 3 to 21. At the individual worker
level, worker #1 shows the highest commitment (score of 8) followed by worker #3
(score of 10), with worker #2 (score of 12) showing the least commitment.

A-6
To calculate the score for each employee for the opportunity subscale of the “Conditions
for Work Effectiveness Questionnaire II,” average the scores given for all three items.
In this example, below are the scores for each worker on this empowerment measure.

Worker #1: 3.3 = [(4 + 3 + 3)/3] = 10/3 items

Worker #2: 2.7 = [(4 + 2 + 2)/3] = 8/3 items
Worker #3: 4.0 = [(5 + 4 + 3)/3] = 12/3 items

Higher scores on this measure indicate greater empowerment in the form of more
perceived opportunity, with possible scores on this 3-item measure ranging from 1 to 5.
At the individual worker level, worker #3 shows the greatest level of empowerment
(score of 4.0) followed by worker #1 (score of 3.3), with worker #2 (score of 2.7)
showing the least empowerment.

The average is usually the statistic used to indicate the summary score on a measure
across all respondents when using Likert-type response scales. Using the
empowerment measure above as an example, here is how to calculate the average
empowerment score for all respondents.

Worker #1 total score + Worker #2 total score + Worker #3 total score

3 (number of respondents)

Working through this formula we get these figures below, for an average of 3.3 among
all three workers:

3.3 + 2.7 + 4.0 = 10/3 = 3.3

So, on average, this sample of workers at this CCRC tend to feel that they have “some”
opportunities at work. However, based on the score of 3.3, there is room for
improvement toward a score of 4 or 5.

Management believes the needs of each department may differ and has decided to put
together employee focus groups for each department. The goal of these focus groups
is to get a sense of the types of things needed to make employees feel more
empowered in their jobs. All employees will be given the opportunity to be part of the
focus groups. After examining each department’s focus group findings and comparing
across departments, management will work with teams of staff members (across all
departments and titles) to determine how to allocate resources across all staff in the
best manner. The ultimate goal is to increase satisfaction with the working environment
and to improve retention of staff. Results of each stage in the process will be shared at
all-staff meetings.

A-7
 MEASURING LONG-TERM CARE WORK:
A Guide to Selected Instruments to Examine Direct
Care Worker Experiences and Outcome

PDF Files Available for This Report

Cover, Table of Contents, Acknowledgments and Executive Summary

http://aspe.hhs.gov/daltcp/reports/dcwguide.pdf

CHAPTER 1: Introduction and Purpose of Guide

http://aspe.hhs.gov/daltcp/reports/dcwguide1.pdf

CHAPTER 2: How This Guide Can Help Organizations Use Information to Address the
Challenges of Job Retention and Performance Among DCWs
http://aspe.hhs.gov/daltcp/reports/dcwguide2.pdf

CHAPTER 3: Ready to Use Instruments http://aspe.hhs.gov/daltcp/reports/dcwguide3.pdf

References http://aspe.hhs.gov/daltcp/reports/dcwguide4.pdf

APPENDIX A: From Start to Finish -- Sample Scenarios of Using and/or Constructing Survey
Instruments http://aspe.hhs.gov/daltcp/reports/dcwguideA.pdf

APPENDIX B: Overview Charts of Chapter 3 Measures, By Topic

http://aspe.hhs.gov/daltcp/reports/dcwguideB.pdf

APPENDIX C: Data Collection Planning and Implementation Issues

http://aspe.hhs.gov/daltcp/reports/dcwguideC.pdf

APPENDIX D: Resources for Providers Considering Use of Employee Surveys

http://aspe.hhs.gov/daltcp/reports/dcwguideD.pdf

APPENDIX E: Individual Measures from Chapter 3 that Use Survey Instruments to Collect
Data, By Topic http://aspe.hhs.gov/daltcp/reports/dcwguideE.pdf

APPENDIX F: Ready Made Multi-Topic Survey Instruments

http://aspe.hhs.gov/daltcp/reports/dcwguideF.pdf

APPENDIX G: Instruments Needing Work http://aspe.hhs.gov/daltcp/reports/dcwguideG.pdf

APPENDIX H: Guide Reviewers http://aspe.hhs.gov/daltcp/reports/dcwguideH.pdf