Questionnaire Design

1
 Contents
1. Terms
2. Methods of data collection
3. Why questionnaire
4. Steps in designing a questionnaire
5. Types of items/questions
6. Source of measurement errors & their remedies
(pre-testing, training, translation, etc)

2
 1) Definition of Terms

any tool used to collect data

 Instrument (Research instruments):
the general process of collecting, synthesizing &
 Assessment:
interpreting information (about people)

the process of quantifying or scoring a subject’s

 Measurement:performance.

A tool with list of items used to audit characteristics

(focusing on availability, status, functionality, etc)

 Checklist:
3
 Question:

List different “data collection methods”

4
 2) Methods of data collection

i. Observation: ii. Questionnaire:

 Lab. diagnosis  Face to face interviews
 Self-administered
 Physical exam.
questionnaire
 Measurements
 Postal, telephone,
 Classification electronic

5
 Methods of data collection Cont…..

iii. Data abstraction (checklist)

iv. Interview- guides or Interview-schedule

(loosely structured tools)

v. FGDs - guide

6
 In general, the various data collection techniques-

 Using available information (2ry data)

 Observing (open or concealed)
 Interviewing (face-to-face)
 Administering written questionnaires (self-administered
questionnaire)
 Focus group discussions [FGD] (8-12 informants per group)
 Projective techniques (if I were……)

7
 Cont…

 Mapping is a valuable technique for visually displaying

relationships and resources.
 Scaling is a technique that allows researchers through
their respondents to categorize certain variables that
they would not be able to rank themselves.

*Mapping & scaling may be used as ‘participatory

techniques’ in rapid appraisals or situation analyses

8
Summary: Data collection Technique Vs tools

9
 3) Why use a questionnaire?

 Cheap & effective way of collecting data

 It can be simple & very focused
 Can collect vast quantities of data from variety of
respondents
 Relatively little training to develop
 Produce rich data that can easily & quickly be analysed

10
  When to use a questionnaire?

 There is no ‘all-encompassing rule’ for when to use a

questionnaire

 The choice will be made based on a variety of factors

including
 the type of information to be gathered &
 the available resources for the experiment

11
 When to use a questionnaire? Cont……

 A questionnaire should be considered in the following

circumstances-
1. When resources & money are limited
2. When it is necessary to protect the privacy of the
participants
3. When corroborating other findings

12
 When not to use questionnaires

 Little knowledge
 Type of the variables is difficult to quantify

 Lack of adequate time:

– Questionnaires compete for respondents time
– Lack of adequate time to complete the instrument may result in the
return of superficial data

 Low response rate due to lack of personal contact:

– Lack of personal contact (if questionnaire is mailed) may mean that
response rates suffer, requiring the expense of follow up letters,
telephone calls & other means of chasing the respondent

13
4. General steps in designing a questionnaire

Research questions

Literature review

Conceptual framework

Specific research questions

14
Data level questions
 Class Exercise

1) Draft a conceptual framework for diarrheal diseases &

associated factors among U5 children
2) Identify important categories of risk factors
3) Give your suggestion how the conceptual framework can
be used to structure a questionnaire

15
 4) Steps ----------cont’d

 Decide on potential sources of questionnaire:

 Use available standard questionnaires (adopt as is)

 Modify available questionnaire & use for own
purpose (adapt)
 Develop new questionnaire…… (2-methods: cont….)

16
 4) Steps ----------cont’d

Method I: General steps to develop new questionnaire:

1. Decide on broad categories/areas of information needed

2. Search for existing related resources/literature
3. Conduct focus groups to generate information (optional)
4. Draft new questions/revise existing ones
5. Sequence the questions
6. Get peer evaluation and revise-&-test on self/co-workers
Cont…..

17
 4) Steps ----------cont’d

Method I: Cont….

7. Revise/eliminate questions based on the feedback

obtained in ‘step-6’
8. Translate into local language (if necessary), back translate &
check the consistency
9. Train interviewers/enumerators
10. Pilot test the questionnaire
11. Revise/eliminate questions based on feedback from
respondents & interviewers’ comments in ‘step-10’
12. Finally, use the tool !!!
18
 4) Steps ----------cont’d

Method II: Identify each specific objective

Review similar literature & obtain key variables

1. Draft questions for each objective using key terms

2. Formulate the key terms into full questions
3. Sequencing: put the questions into logical order
4. Formatting the questionnaire (arrange/organize)
5. Translating into local language of respondents
6. Pretesting and implementation

19
 5) Types of questions/Items

 Closed questions
– Dichotomous responses (yes/no), and
– Multiple-choice questions:
• All possible answers covered (and include other)
• Ordinal responses (Likert-scale)

 Open-ended questions
– No restrictions of answers
• But, can make analysis more difficult
• Must be carefully coded

20
 Closed Vs Open ended
 Closed questions are usually better:
– Easier for the respondent
– Less coding is required later
– Categories help define the question
 Disadvantage of close ended:
• Categories may be ‘leading’ to respondents
• May make it too easy to answer without thinking
• Not best when:
– asking for frequency of sensitive behaviors
– there are numerous possible responses

21
Exercises: which is open/closed question?

A. What is the most important issue facing today's youth?

B. Which of these is the most important problem facing
today's youth?
1. Unemployment
2. National unity
3. Environment
4. Youth violence
5. Rising tuition fees
6. Drugs in schools
7. Need for more computers in schools
8. Career counseling

22
Principles for response options of a question

 Response categories should be consistent with the

question
 Categories must be exhaustive, including every possible
answer
 Categories must be mutually exclusive (no overlap)
 If appropriate, include a “don’t know” category

23
 Ordering response categories

 Usually better to list responses from the lower level to

the higher level
 Associate greater response levels with greater numbers
 Start with end of a scale that is least socially desirable

24
  User friendly/good questions

 Three Rules for writing good questions

1) Think through your research questions & objectives

before you write individual questions
2) Prepare an analysis plan before you write questions
3) Ask yourself, in relation to points #1 & #2 above, if each
question on your list is necessary? Even if the data would
be ‘interesting’ it has to ultimately be used in analysis to
make the cut!
25
  What constitute a good question

 The one that:

 yields a truthful, accurate answer
 asks for one answer on one dimension
 accommodates all possible contingencies of response
 uses specific, simple language
 has mutually exclusive response options
 minimizes social desirability; and it is pretested

26
  Art of asking questions

 Ask the right question

 Respondents must understand your question
 Respondents must know the answer
 Respondents must be willing & able to tell you
the answer

27
 Art Cont…..

 Ask one question at a time

Question A ……(√) Question B: ……..(X)

In the past 6 months, Now I’m going to read a list of
what major food items food items. As I read each one,
have you purchased? please tell me whether or not
you have purchased this type
of food items during the past 6
months. How about…
- eggs? Yes/no
- meat? Yes/no
- fruits? Yes/no

28
 Ask one question at a time Cont….

Question A: Question B:

 Compared to one year ago,

 Compared to one year ago,
are your expenses more, less,
are your expenses more, or about the same for:-
less, or about the same for a. Eggs? Less/more/the same
your food items? b. Meat? Less/more/the same
c. Fruits? Less/more/the same

29
 Art Cont…

 Specify the questions

 Specify… ‘who, what, when, where & how’.

 For e.g., whose income? What’s included? & Over what
period of time?

 E.g. “In 2002, what was your total household income, before taxes?
Please count income from all members of your household, including
wages from employment, social security, public aid, & if any other
sources.”

30
 Specify through cues (reminder)

Example
People drink beer in many places – for e.g., at home, at
restaurants, at bars, sporting events, at friends’ homes, etc.
During the past 30 days, did you drink any beer?
1) yes
2) No
3) I do not remember
####

31
6) Source of errors during administration of questionnaire

• Inexperienced data collectors (poor training)

• Interrupting during administration sessions
• Time- the questionnaire is written (like heat, light & ventilation)
• Mood, fatigue, attitude accustomed to completed questionnaires
• Reactions of participants to items (social desirability bias)

32
  How to Minimize ‘Social desirability bias’?

 For socially desirable behavior:

 ask whether respondents have ever engaged in the
behavior, before asking whether they currently engage
in the behavior. (e.g. physical exercise)

 For socially undesirable behavior:

 ask about current behavior first, rather than asking
about their usual or typical behavior. (e.g. smoking)

33
 Minimizing Social desirability bias Cont…

 Train interviewers to maintain a professional attitude

 Self-administered procedures can reduce question

threat & improve reporting on sensitive questions

 Longer questions reduce sensitivity when obtaining

information on frequency of socially undesirable behavior

34
 Assessing measurement quality

 The two key components of quality

include:

 Validity

 Reliability

35
 Types of validity

Content

Face Validity Criterion

Construct

36
 Face validity:
 Subjective assessment of the relevance of
the questionnaire as confirmed by FGD
 Relevant questions
 Unambiguous
 Reasonable, & Clear

37
 Content validity:
 Judgment made by a panel (team/committee)
 Comprehensiveness
 Balance information
 Address the research scope

 Criterion validity(its standard):

 Agreement b/n the measures & another
measurement

 Construct validity(the concept/theory/hypothesis):

– extent of testing the hypothesis

38
 Reliability

Chance

Bias Reliability Real change

(instrument)

Bias
(observer)
39
Summary: Rules/Requirements for questionnaire development

1. Start with easy questions that all respondents can answer

with little effort

2. Avoid leading questions

3. Avoid sensitive or touchy questions

4. Avoid negatively phrased items

5. Should be simple & specific (stick to one idea)

6. Questions should be arranged in logical sequences

7. Must be in the language respondents understand well

40
 Length/size of the questionnaire:

 Better not exceed 20-30 minutes (Interviewer administered)

 Should not exceed 10-12 pages (for self-completed)

*But, it can be governed by the feedback from the pretesting

41
 Preparing a Questionnaire for Administration

 Write a descriptive title for the questionnaire

 Write an introduction to the questionnaire
 Group the items by content & provide a subtitle for each group
 Within each group of items, place items with the same
format together
 At the end of the questionnaire, indicate what respondents
should do next

42
 Preparing a Questionnaire Cont…..

 Prepare an informed consent form, if needed.

 If the questionnaire will be mailed to respondents, avoid
having your correspondence look like junk mail
 If the questionnaire will be mailed, consider including a
token reward (reminder/thanks in advance)
 If the questionnaire will be mailed, write a follow-up letter.
 If the questionnaire will be administered in person, consider
preparing written instructions for the administrator.

43
 Physical format of checklist or questionnaire

• Cover of the questionnaire

– Include a title, graphic, name/address of sponsor
• Introduction should indicate:
– who is conducting the survey
– the topics to be covered in the survey
– how long it will take- depends on mode, topic, population
– Explain what/who is being asked and why
– Explain the purpose, importance and significant
– State a commitment to share the results!
– State about anonymity and confidentiality

44
 Organization of the questions:
 Number all questions sequentially
 Use large, clear type & do not crowd
 ‘White space:’ Place more blank space b/n questions
than b/n subcomponents of questions
 List answer categories vertically instead of horizontally
 Be consistent with direction of response categories
 Be consistent with placement of response categories
 Don’t split questions across pages
 Put special instructions on questionnaire as needed, next
to question. (Cont.…next page)

45
 Cont…

 Distinguish directions from questions

 Pre-code the questionnaire (vs. check boxes)
 Use a booklet format:
 Easier to turn pages
 Prevents lost pages
 Permits double-page formats
 That looks more professional

46
  Pre- testing questionnaire

 Purpose of pretesting:
– To identify limitations of the tool & take corrective
measures before the main study starts

 Important inputs - include:

– Availability of the study population & how respondents’
daily work schedules can best be respected
– Acceptability of the questions asked
– Willingness of the respondents to answer the questions

47
 Cont…

 Pretesting is useful for obtaining further inputs including:

 The sequence of questions is logical
 The wording of the questions is clear
 Translations are accurate
 Space for answers is sufficient

On whom
 Test you plan to test the
the questionnaires questionnaires?
with people
similar to source population for the study
but………….?
48
 Summary: In general-Questionnaire testing

 is a fundamental step in developing a questionnaire

 Testing helps to:
 discover poor wording or ordering of questions
 identify errors in the questionnaire layout & instructions
 determine problems caused by the respondent's inability
or unwillingness to answer the questions

49
 Questionnaire testing Cont….

suggest additional response categories that can be pre-

coded on the questionnaire
provide a preliminary indication of the length of the
interview and any refusal problems

 Testing can include the entire questionnaire or only a

particular portion of it …….(on about 5% to 10% of ….???).
*However, it will at some point in time have to be fully tested.

50
 Limitation of questionnaires:
– Mostly result of poor design:
 Leading questions
 Complicated questions
 Irritating questions
 Ambiguous or unclear questions
 Too many open-ended questions

51
Danger in Designing a "Bad" Questionnaire:

 In designing a questionnaire the following points should

be observed in its design:
1. questions should be simple
2. questions should be unambiguous
3. the best kinds of question are those which allow a pre-
printed answer to be ticked
4. the questionnaire should be as short as possible; …..

52
 Danger in Cont……

5. questions should be neither irrelevant nor too personal

6. Leading questions should not be asked. A "leading question" is

one that suggests the answer, e.g. the question “Don’t you agree that
all sensible people use XZ-Soap?” suggests the answer "yes"

7. The questionnaire should be designed so that the

questions fall into a logical sequence. This will enable the
respondent to understand its purpose, and as a result the quality of his
answers may be improved.

53
THANK YOU FOR GOOD
ATTENTION!!!

54