Tutorial Letter 101/3/2018: Basic Questionnaire Design and Assessment
Tutorial Letter 101/3/2018: Basic Questionnaire Design and Assessment
Tutorial Letter 101/3/2018: Basic Questionnaire Design and Assessment
Semesters 1 and 2
Department of Psychology
BARCODE
CONTENTS
1 INTRODUCTION ............................................................................................................................................. 3
1.1 Tutorial matter .................................................................................................................................................. 3
2 PURPOSE OF AND OUTCOMES FOR THE MODULE ................................................................................ 3
2.1 Purpose............................................................................................................................................................ 3
2.2 Outcomes......................................................................................................................................................... 4
3 LECTURERS AND CONTACT DETAILS ....................................................................................................... 4
3.1 Lecturers .......................................................................................................................................................... 4
3.2 Department ...................................................................................................................................................... 4
3.3 University ......................................................................................................................................................... 4
4 MODULE-RELATED RESOURCES ............................................................................................................... 6
4.1 Prescribed books ............................................................................................................................................. 6
4.2 Recommended books ...................................................................................................................................... 7
4.3 The PYC2606 study guide ............................................................................................................................... 7
4.4 Electronic Reserves (e-Reserves) ................................................................................................................... 8
5 STUDENT SUPPORT SERVICES FOR THE MODULE ................................................................................ 8
5.1 Contact with fellow students ............................................................................................................................ 8
5.1.1 Study groups .................................................................................................................................................... 8
5.1.2 myUnisa ........................................................................................................................................................... 8
5.2 Tutorial classes ................................................................................................................................................ 8
5.3 Discussion classes .......................................................................................................................................... 8
5.4 Repeat students ............................................................................................................................................... 8
6 MODULE-SPECIFIC STUDY PLAN ............................................................................................................... 9
7 MODULE PRACTICAL WORK AND WORK-INTEGRATED LEARNING ..................................................... 9
8 ASSESSMENT ................................................................................................................................................ 9
8.1 Assessment plan ............................................................................................................................................. 9
8.2 General assignment numbers.......................................................................................................................... 9
8.2.1 Unique assignment numbers ......................................................................................................................... 10
8.2.2 Due dates for assignments ............................................................................................................................ 10
8.3 Submission of assignments ........................................................................................................................... 10
8.4 Assignments .................................................................................................................................................. 11
9 SUMMATIVE ASSESSMENT METHODS .................................................................................................... 11
10 EXAMINATION .............................................................................................................................................. 12
11 FREQUENTLY ASKED QUESTIONS .......................................................................................................... 13
12 APPENDIX SEMESTER 1............................................................................................................................. 14
13 APPENDIX SEMESTER 2............................................................................................................................. 43
2
PYC2606/101
1 INTRODUCTION
Dear Student
This paper is called Basic Measurement and Questionnaire Design in Psychology. The code of
this paper is PYC2606. Paper names and paper codes are part of Unisa-speak. So if you want
to ask anybody at Unisa something about this paper remember the name Basic Measurement
and Questionnaire Design in Psychology and the code PYC2606.
PYC2606 (Basic Measurement and Questionnaire Design) is one of the modules included in the
Alternative Assessment Project. We will use an alternative assessment measure for this
module. Students will sit for a timed online assessment, which means they will be required to
complete the assessment online on a particular date and time. This means that there is no
venue based written examination for this module. You will sit for an online assessment. The
online assessment means you will be required to log-on online. If you cannot do so from home
or from an internet café you have to reserve a computer at a Unisa regional office.
Bear in mind that in distance education the written word is the main means of study and
communication between lecturers and students. We will also use myUnisa to bring important
aspects to your attention and therefore they must all be read carefully, acted upon where
necessary, and kept for future reference.
At the time of registration, you will receive an inventory letter that will tell you what you have
received in your study package and also show items that are still outstanding. Some of this
tutorial matter may not be available when you register. Tutorial matter that is not available
when you register will be posted to you as soon as possible, but is also available on myUnisa.
The Department Despatch should supply you with the following study material for this module:
• Study Guide
The study material is also available under the link official study material on myUnisa.
2.1 Purpose
This module aims to:
3
2.2 Outcomes
The module reacts to a practical need, namely to compile and validate a basic questionnaire in
psychology. Through this practical task learners are introduced to basic concepts in the field of
psychometrics.
The tuition imitates real-life learning situations in the sense that learning takes place in a
purposeful and just-in-time manner. In other words, in real-life situations one does not first study
an area of knowledge and then go out to apply it. In real-life situations one is forced to use
knowledge and to gain experience as events unfold.
General enquiries with regard to this course may be directed to Ms Patracia Sekhaulela,
(Assistant Administrator) at 012 429-8088 of [email protected] (both 1st and 2ndsemester).
3.2 Department
Departmental telephone number: (012) 429-8088
Departmental physical address: Theo van Wijk Building, 5th floor, Room 5-49.
3.3 University
Please note that there are different ways to communicate with the university, by telephone, e-
mail, fax, ordinary mail, and via myUnisa.
If you need to contact Unisa about matters not related to the content of this module, please
consult the brochure my Studies @ Unisa, which you should have received with your study
material. In this document you will find the contact details of various administrative departments.
If you did not receive this document and have access to the Internet, you can find it on myUnisa
under the ‘Resources’ option. Remember to always have your student number ready when
making an enquiry.
4
PYC2606/101
Please ensure that you contact the correct section or department for each enquiry as this
will result in a speedy resolution of the enquiry.
E-mails Addresses:
Examinations: [email protected]
Aegrotat and special examinations: [email protected]
myUnisa
To make use of myUnisa, you will need a computer with a modem and an Internet
connection, as well as a browser such as Netscape or Internet Explorer.
Using your browser, go to the Unisa web page, which is located at the Internet address
http://www.unisa.ac.za.
5
On this web page, select the option “myUnisa”. If you are a first time user, you must now click
on the option “Register as myUnisa user” which will enable you to register online (this does not
cost anything). Type in your name, student number and a password (the password must be at
least 6 characters long).
Please note that you are not allowed to submit assignments by fax or e-mail.
For detailed information and requirements as far as assignments are concerned, see my
Studies @ Unisa, which you received with your tutorial matter.
4 MODULE-RELATED RESOURCES
An important note
The Department of Despatch will send you the following study material for this module: Tutorial
Letter 101/2018 and Study Guide. Apart from these you will also receive other tutorial letters (for
instance, for feedback on assignments). These tutorial letters will be despatched to you as soon
as they are available or needed.
Additional material
The inventory letter contains information about your tutorial matter. The inventory lists only
those items that are available at the Department of Despatch in Pretoria or at the University's
regional centres on the date of your registration. Students who register at a regional centre will
receive the rest of the available tutorial matter by registered mail from Pretoria.
Please check the tutorial matter you have received against the inventory letter. Unless indicated
otherwise (e.g., out of stock), you should have received all the items listed in the inventory
letter. Should any of the items be missing, please proceed immediately according to the
instructions on the reverse side of the inventory letter.
6
PYC2606/101
Use the telephone number on the reverse side of the inventory letter only when enquiring from
the Department of Despatch about tutorial matter which you have not received
More details are contained in the publication my Studies @ Unisa brochure that you received
with your tutorial matter.
An important note
Please do not contact the Psychology Department if you have not received your tutorial matter.
For any outstanding tutorial material, please send an e-mail to the Department of Despatch at
[email protected].
Remember to have the following ready when you call: (a) Your student number, (b) the module
code (PYC2606 in this case), and (c) the numbers of the tutorial letters you have not received.
(Please see information above.)
The study guide contains the required resource material and learning opportunities. The module
gives you the opportunity to acquire the basic practical competencies required to design and
develop a questionnaire and to compile a short manual for the questionnaire. We will guide you
through a number of activities (see Study Guide) to obtain these objectives. For each activity
you are required to study the relevant resource material (see Study Guide), complete the
necessary actions, and produce the specified outcomes. Note that the activities follow on each
other. If you do not complete all the activities you cannot expect to obtain the desired outcomes.
Content
In particular you will define a psychological construct or concept to measure, and based on this,
describe the content domain of your questionnaire. Designing a questionnaire involves
decisions about the item format, the scaling method, the number of questions and the
questionnaire layout. In the resource material you will find guidelines for writing items. Once you
have completed your draft questionnaire, you will select a group of people and try out the
questionnaire. When you get the results you will evaluate how well the questionnaire works.
This means you will have to evaluate how well the items work. You will then evaluate if the
questionnaire measures consistently and if it actually measures what it is supposed to measure.
You will report this process in a manual, together with procedures for using the questionnaire.
These procedures deal with how to administer the questionnaire and how to score and interpret
the results. You will then apply the competencies you acquired in developing your own
questionnaire and writing a manual for your questionnaire to evaluate other questionnaires and
manuals.
7
4.4 Electronic Reserves (e-Reserves)
It is advisable to have contact with fellow students. One way to do this is to form study groups.
The addresses of students in your area may be obtained from my Studies @ Unisa.
5.1.2 myUnisa
If you have access to a computer that is linked to the internet, you can quickly access resources
and information at the University. The myUnisa learning management system is Unisa's online
campus that will help you to communicate with your lecturers, with other students and with the
administrative departments of Unisa – all through the computer and the internet.
Please contact our Regional Centres for more information on tutorial classes.
It is expected from a repeating student to submit all assignments again. Students will not obtain
examination admission without the submission of assignments.
8
PYC2606/101
8 ASSESSMENT
8.1 Assessment plan
Examination admission
You have to submit your first assignment before the closing date for the semester you are
registered for. If you do not fulfil this requirement you will not be allowed to sit for the
examination.
Year mark
The module has two assignments and the marks that you obtain for both assignments will count
towards a year mark. The year mark is added to the examination mark to give a final mark. The
year mark contributes 20% towards the final mark and the online examination mark contributes
80%.
Assignment 01
9
Assignment 02
The computer requires a unique assignment number when marking an assignment. The unique
numbers for each assignment for each semester are provided in this tutorial letter.
SEMESTER 1 SEMESTER 2
(Also see Appendix Semester 1) (Also see Appendix Semester 2)
01 721404 01 747855
02 682897 02 571446
Electronic assignment submission: Visit the Unisa home page at http://www.unisa.ac.za and go
to myUnisa for a detailed explanation.
Submission of a multiple choice assignment is simple fast and reliable via myUnisa. Just follow
the step-by-step instructions on the screen. You may be wondering about the reliability of this
service. Submission through myUnisa is much faster and more reliable than through the postal
services. There are two ways of checking that your online assignment has been successfully
submitted:
Upon submission online, you will receive a clear status message. If the message reads
“Assignment received in good order”, you can be sure that your assignment is safely on
our system.
10
PYC2606/101
Your list of assignments for the course under myUnisa will be updated immediately after
successful submission of an assignment.
If you use a dial-up connection, have your multiple choice answers ready before dialling
up. It will cost you money if you have to work out the answers while you are online.
Please check for mistakes before you submit! No requests to cancel assignments
submitted via myUnisa will be accepted by the Assignment Section.
Please do not submit the same assignment both through myUnisa and via mail on a mark
reading sheet.
Do not wait until the closing date of the assignment. Submit it at least two working days
before the closing date. This is to make provision for unforeseen problems, e.g. your
internet connection could be down/not available on the closing date.
Feedback on assignments
You will receive the correct answers automatically. Commentaries on the assignments will be
sent in follow-up tutorial letters (Tutorial Letters 201/2018 and 202/2018). Detailed feedback
about assignments will be provided. It will become available under Additional Resources on the
PYC2606 site on myUnisa.
8.4 Assignments
Important information
Please make sure that you provide the correct assignment unique number and student number.
9 SUMMATIVE ASSESSMENT
Please Note: PYC2606 is not a portfolio. This is a timed online exam and the delivery mode is a
non-venue exam. All non-venue examinations are indicated as Departmental requirement /
Portfolio. If you completed the online exam there is no other assignment or portfolio you must
complete.
Assignment 3 is the online exam.
A timed online examination for this module is scheduled.
Please Note: There is no allocated exam venue. You can complete the online exam in a venue
of your own choice (home, library, internet café etc.).
To complete the online exam you need to login to myUnisa. The logon procedures will be
posted on myUnisa. Questions are scenario embedded because they are related and
formulated in terms of a scenario that requires practical application of knowledge. The scenario
will be uploaded on the date of the examination.
11
10 ONLINE EXAMINATION
How do I answer the online questions? — The duration of the online assessment is 2½ hours. It
consists of 50 multiple-choice questions. The paper total is 50 marks. Your mark out of 50 is
automatically converted to a percentage - that is, a mark out of 100. The assessment covers the
entire syllabus. Assignment 01 together with Assignment 02 provides a good indication of the
nature and the format of the examination paper (assignment 3) you will complete in online
assessment.
When is the examination? — There are only two examination sessions per year in the semester
system.
Students in other countries will have to adhere to the scheduled South African time. We require
you to write at the scheduled time regardless of where in the world (and what time) you find
yourself.
VERY IMPORTANT
We recommend that all students run a "mock" exam to test your access on myUnisa. It will also
provide a good indication of the nature and the format of the examination paper you will
complete in online assessment. If you experience any problems running this mock exam, you
should email Exam technical support at [email protected] to resolve them before
your exam date.
The mock trial is not compulsory. It is a fake exam that doesn't count for a mark. It's an exam
given to the same standards and under the same conditions as the real online exam. The
purpose is to prepare a student for the online exam conditions.
To complete the mock exam you need to login to myUnisa. The mock exam date and logon
procedures will be posted on myUnisa.
Also note that detailed feedback about the mock exam will be provided on myUnisa under
additional resources.
When do I pass? — You pass if you obtain an online assessment mark of at least 40% and a
final mark (online assessment mark plus year mark) of at least 50%. The year mark contributes
20% towards the final mark and the online assessment mark contributes 80%.
What happens if I fail? — If you obtain a final mark of at least 40 you may sit for a
supplementary online assessment. If you fail with a mark below 40% you have to re-register,
and redo the module.
12
PYC2606/101
When is the supplementary examination? — You have to write the supplementary online
assessment at the next examination period.
13
11 APPENDIX SEMESTER 1
Assignment instructions
What to do:
For Assignment 01:
• Read the scenario context (A), the questionnaire (B) and the mini manual (C). These
sections contain information required for answering the discussion questions of the first
meeting. Do not complete the questionnaire (B) itself
• Answer the 25 questions of the first discussion meeting (D) and submit your answers as
Assignment 01
• Note that this assignment is a multiple choice question assignment. It is better to submit
the assignment online using myUnisa. Only use the printed mark reading sheet and the
postal service if you are unable to submit the assignment online.
• Answer the 25 questions of the second discussion meeting (E) and submit your answers
as Assignment 02
• Note that this assignment is a multiple choice question assignment. It is better to submit
the assignment online using myUnisa. Only use the printed mark reading sheet and the
postal service if you are unable to submit the assignment online.
14
PYC2606/101
Scenario context A
You have recently been appointed as the psychologist at a school. The school board and the parent
association are becoming increasingly concerned about illegal drug use amongst learners at the school.
Your predecessor began a preventative program, but the programme was aimed mainly at disseminating
information. You realise that you also have to identify and target learners who are at risk for illegal drug
use. You decide that the easiest and most economical way of doing this is to use a questionnaire to
identify learners at risk. This will allow you to tailor prevention programmes to accommodate the needs
and circumstances of these learners.
You consult academic literature on the issue of drug abuse and the use of illegal drugs. In the literature
you come across an interesting fact, namely that there is a correlation between alcohol abuse and the
tendency to abuse other drugs or to use illegal drugs. Thus information about learners’ use of alcohol
could indicate whether they should be considered at risk for drug abuse in general. You also notice that
many of the publications refer to a particular theory that explains how behaviour can be predicted. You
think the theory could be important because it could help to predict whether somebody would actually
begin to abuse drugs or use illegal drugs. You search the literature for papers on this theory, and come
across two articles explaining what the authors refer to as ‘the theory of planned behaviour’.
According to the theory of planned behaviour there are psychological factors that predict whether or not
a person will execute a particular form of behaviour. Behaviour is preceded by the intention to execute
the behaviour. The stronger the intention to execute the behaviour the more likely a person is to actually
execute the behaviour in question. Three factors underlie the intention to execute behaviour. These
factors are the person’s attitude towards the behaviour, his/her subjective norm concerning the
behaviour and his/her perceived control with regard to the behaviour. Thus a person will form a strong
intention to act (execute a particular form of behaviour) if he/she feels positive about the behaviour
(positive attitude), thinks the behaviour is socially acceptable (positive subjective norm) and that nothing
stops him/her from executing the behaviour (positive control).
You contact several test publishers and finally come across a questionnaire called the Alcohol use
Intention Questionnaire (AIQ). You order the questionnaire from the test publisher. When the AIQ arrives
you decide to conduct a thorough analysis to determine whether the questionnaire is a scientifically valid
and reliable instrument. There are four psychology students undergoing practical training at the school
under your guidance. You ask them to study the questionnaire (B) and its mini manual (C) and you
schedule two meetings (D and E) to discuss the instrument.
15
Questionnaire B
Alcohol use Intention Questionnaire (AIQ)
Instructions
Please read the following questions carefully and indicate your response to each question by ticking the
answer that best describes how you feel about the issues raised by these questions. Please answer all
questions. Do not spend too much time thinking about your answers. The questionnaire should not take
long to complete.
Agree Disagree
Agree Unsure Disagree
strongly Strongly
2.8 I think drinking is a waste of time [ ... ] [ ... ] [ ... ] [ ... ] [ ... ]
3.3 You have to not drink to not earn the respect of your friends
(a) true [ ... ]
(b) false [ ... ]
3.4 Many people drink too much at social events. In your opinion, is excessive use of alcohol a
acceptable practice? Yes/No
3.5 Rate the importance of the following effects of alcohol consumption on a scale from 1 (least
serious) to 5 (most serious)
liver failure […]
heart disease […]
cognitive impairment […]
memory loss […]
social isolation […]
personality change […]
17
3.6 An excessive drinker can be defined as somebody who drinks
(a) at night
(b) at least 6 drinks a day
(c) 6 to 10 drinks a day
(d) at least 20 drinks a week
3.7 There are stereotypes associated with the kinds of alcoholic drink somebody consumes. A person
who consumes ……… may be seen as a ‘heavy’ drinker, whereas a person who consumes
………. may be seen as a ‘light’ drinker.
4.5 Participating in extramural activities keeps one’s mind focussed True / False
.........................................................................................................................................
.........................................................................................................................................
End of questionnaire
18
PYC2606/101
Mini manual C
Mini manual for the Alcohol use Intention Questionnaire (AIQ)
1. Aim and design
The Alcohol use Intention Questionnaire (AIQ) can be used to determine the level of a person’s intention
to use alcohol. The questionnaire is meant to be used in a preventive context – in other words, it should
be used with individuals who have not been submitted to a treatment programme. The aim is to identify
individuals who may be at risk of becoming heavy drinkers.
The AIQ is based on the theory of planned behaviour. The theory of planned behaviour links the
intention to execute behaviour with the actual execution of the behaviour. Therefore the AIQ can be used
to predict whether or not a person is likely to abuse alcohol. According to the theory of planned
behaviour there are psychological factors that can be used to predict whether or not a person will
execute a particular form of behaviour. Three factors underlie the intention to execute behaviour. These
factors are the person’s attitude towards the behaviour, his/her subjective norm concerning the
behaviour and his/her perceived control with regard to the behaviour. Thus a person will form a strong
intention to act (execute a particular form of behaviour) if he/she feels positive about the behaviour
(positive attitude), thinks the behaviour is socially acceptable (positive subjective norm) and that nothing
stops him/her from executing the behaviour (positive control).
The AIQ was pilot tested in a drug abuse treatment clinic in Germany. Five patients took part in the initial
pilot study. They were tested when they arrived at the clinic. The AIQ was divided into two sets of
scores. The first set consisted of the even item numbers and the second set consisted of the odd item
numbers. The scores of the five patients are indicated in Table 1. The patients were tested again three
days after their arrival. This information is shown in Table 2.
The researchers were also interested in how each of the items correlated with the participants’ total
scores. Table 3 shows this information for one of the items. The researchers compiled similar tables for
each item in the questionnaire.
19
Table 3: The relationship between participants’ Item 2.1 scores and
their total scores
Item 2.1 scale values 1 2 3 4 5 Total
Score for participant 1 x 44
Score for participant 2 x 53
Score for participant 3 X 68
Score for participant 4 x 57
Score for participant 5 x 76
Y-axis
75-79 [5]
70-74
65-69 [3]
60-64
55-59 [4]
50-54 [2]
45-49
40-44 [1]
1 2 3 4 5 X-axis
Figure 1:
Soon after the initial pilot study the clinic added another 5 participants to the pilot. These individuals were
tested on arrival at the clinic. The results obtained from the expanded pilot study enabled the
researchers to calculate the correlations between each of the items and the total scores. These
correlations were used to discard weak items. Table 4 shows the scores obtained by each of the
research participants for four of the items (questions 2.9, 2.11, 2.13 and 2.17) together with their total
test scores.
Note: A total score is the sum of a person’s items scores for all items in the questionnaire. Only four of the items
scores are shown in this table.
20
PYC2606/101
The scatter plots for the items listed in Table 4 are shown in Figures 2, 3 and 4.
100
90
80 ■ ■
70 ■
60 ■ ■
Total scores 50 ■ ■
40 ■
30 ■ ■
20
10
1 2 3 4 5
Item scores
Figure 2: Scatter plot for Item 2.9 — each ■ represents a research
participant
100
90
80 ■ ■
70 ■
60 ■ ■
Total scores 50 ■ ■
40 ■
30 ■ ■
20
10
1 2 3 4 5
Item scores
Figure 3: Scatter plot for Item 2.11 — each ■ represents a research
participant
100
90
80 ■ ■
70 ■
60 ■ ■
Total scores 50 ■ ■
40 ■
30 ■ ■
20
10
1 2 3 4 5
Item scores
Figure 4: Scatter plot for Item 2.13 — each ■ represents a research
participant
21
100
90
80 ■ ■
70 ■
60 ■
Total scores 50 ■ ■
40 ■ ■
30 ■
20 ■
10
1 2
3 4 5
Item scores
Figure 5: Scatter plot for Item 2.17 — each ■ represents a research
participant
Over a period of six months the clinic was able to add more participants to the pilot study. The final pilot
study included 30 participants between 25 and 45 years of age, with 18 males and 12 females. During
their stay at the clinic these participants were asked to complete two additional questionnaires, the Drug
Dependency Personality Questionnaire (DDPQ) and the Art Appreciation Questionnaire (AAQ). The
DDPQ was used to assess whether a person had personality characteristics associated with drug
dependency. The AAQ was used to determine a person’s level of art appreciation. The occupational
therapists were interested in the participants’ art appreciation because they attended painting and
sketching sessions as part of their treatment. The clinic staff also rated the participants in terms of
suitability for the treatment programme. These were indicated as Treatment Suitability Ratings (TSR).
The following tables contain further information that was gathered during the pilot study. Table 5 shows
the inter-correlations among the factors addressed by the AIQ, obtained for the pilot group of 30
participants. Table 6 contains the inter-correlations among the questionnaires and the ratings provided
by the clinic staff.
Administration
Instructions for completing the questionnaire are given with the items of the questionnaire in the AIQ.
Individuals respond on the questionnaire and need a pencil to do so. The questionnaire is a
psychological test and this affects who is allowed to administer, score and interpret the questionnaire.
22
PYC2606/101
Respondents should be instructed to answer the questionnaire honestly. Words may be explained but
respondents should not be guided towards a specific answer.
Scoring
Note that some items require reversed scoring. Scores should be assigned in a manner that reflects
higher scores for a positive attitude towards alcohol use, and higher scores for a subjective norm in
favour of alcohol use. For example, agreement with the statement “I think drinking is relaxing” (Item 2.1)
reflects a positive attitude towards drinking (‘agree strongly’ receives 5 marks, and ‘disagree strongly’
receives 1 mark). However, agreement with the statement “Getting drunk is undesirable” (Item 2.2)
indicates a negative attitude. Therefore Item 2.2 requires reversed scoring in which ‘agree strongly’ gets
1 mark and ‘disagree strongly’ gets 5 marks.
Interpretation
Norm tables are provided for determining a person’s alcohol abuse risk index. The norm tables associate
the scores that people obtain on the AIQ with a five point risk scale, ranging from very high risk to very
low risk. To determine a person’s alcohol abuse risk index: Add his/her attitude and norm scores, and
subtract his/her recreational score from this sum. Use the person’s age and gender information to select
the relevant norm table, and compare his/her obtained AIQ score with the scores listed in the norm table.
The risk index is the index associated with the matching score in the norm table. Individuals with high
and very high risk indices should be targeted by preventative and treatment programmes.
End of manual
23
Discussion questions (Assignment 01) D
Question 1
Bets: I presume a good place to begin this discussion is the intention of the AIQ.
Temba: Well, the intention of the questionnaire clearly is to determine the level of an
individual’s intention to use alcohol
Ben: No, the questionnaire’s intention is to identify and target learners who are at risk for
illegal drug use
Fanie: Actually the intention of the questionnaire is to assess all learners in the school
You agree most with, second most with and least with:
Question 2
You: Do you think the questionnaire is fit for purpose given our context? In other words, do
you think we can use the questionnaire for what we need it for?
Ben: Yes, because we need the AIQ to identify learners who are at risk for illegal drug use,
which fits the intention of the AIQ
Bets: Yes, because we need the AIQ to identify learners who intend to use alcohol, which
fits the intention of the AIQ
Temba: Yes, because the intention of the AIQ is to assess all learners and we want to use
the AIQ to tailor prevention programmes to accommodate the needs of these
learners
You agree most with, second most with and least with:
Question 3
Fanie: Given the intention of the AIQ, what is its content domain?
24
PYC2606/101
You agree most with, second most with and least with:
Question 4
Bets: I am not convinced that the scope of the AIQ is sufficient in light of its intention and in
light of what we need it for.
Temba: I share your concern, Bets. The items included in the AIQ do not fully represent the
content domain
Fanie: I am not concerned about the scope of the AIQ. I think it has sufficient scope to
assess learners at any school
Ben: If one takes into account that the AIQ consists of different sections with different
kinds of items it is clear that the AIQ has sufficient scope to realise its purpose
You agree most with, second most with and least with:
Question 5
You: Do you think the AIQ is the correct type of questionnaire for our purposes?
Fanie: The AIQ is a survey type questionnaire and because we want to survey intention to
use alcohol I think the AIQ is the correct type of questionnaire for our purposes
Bets: I agree that the AIQ is a survey type questionnaire. But we want to determine the
learner’s ability to become an illegal drug user and therefore I think the AIQ may not
be the correct type of questionnaire for our purposes
Ben: Actually the AIQ is an ability type questionnaire, but it is not the correct type of
questionnaire for our purposes because we want to determine the personality of the
learner who uses illegal drugs
You agree most with, second most with and least with:
25
Question 6
Ben: Do you think the instructions that the AIQ provides to respondents are correct and
complete?
Temba: The instructions provide the correct information but are incomplete
Bets: I think the instructions provide the correct information and they are complete
Fanie: No, I think the information provided by the instructions are incorrect and the
instructions are also incomplete
You agree most with, second most with and least with:
Question 7
You: Do you think confidentiality is an issue considering the way we plan to use the AIQ?
Ben: I do not think confidentiality is an issue because the AIQ does not require
respondents to provide their names
Temba: I think confidentiality is an issue because our ultimate goal is to identify learners who
are at risk for illegal drug use
Bets: To my mind confidentiality is not an issue because these learners are under-aged
You agree most with, second most with and least with:
Question 8
Bets: What do you think of the sequence in which items are presented in the AIQ?
Temba: The items of the AIQ are sequenced well. They are grouped into sections which allow
for better focussed responses and they progress from questions about personal
experiences to questions about personal opinions to questions requiring factual
information, ending with personal experiences again
Ben: The sequencing of the items is not really an issue in the AIQ. All the items except the
last one are equally easy to respond to, requiring tick responses only
Fanie: The items in Section 1 should be moved to the end of the questionnaire due to the
emotional impact of these questions and the possibility of them putting the
respondent in a particular frame of mind
26
PYC2606/101
You agree most with, second most with and least with:
Question 9
You: Another aspect we should consider is whether the AIQ incorporates different item
response formats. Can you identify items that demonstrate different response
formats?
Ben: Items 1.2 and 3.1 are questions with limited choice of answers whereas others, such
as 4.2, provide a checklist. I also noted that some items such as 2.12 and 2.15
require inversed responses
Fanie: Items 2.2 and 2.7 are further examples of questions that require inversed responses,
and the semantic differential type of items contained in Section 2 clearly differ from
checklist type items, such as 3.5
Bets: I identified two more items in Section 2 that require inversed responses, namely 2.1
and 2.7, and the open question type such as 4.6 differs from the multiple choice type
items provided in Section 2.
You agree most with, second most with and least with:
Question 10
Temba: Item 3.6 is an example of a filter question, but unfortunately it does not operate
correctly as a filter
Ben: Item 4.1 is an example of a filter question, and I think it operates correctly as a filter
Bets: I agree that item 4.1 is an example of a filter question, but I do not think it operates
correctly as a filter
You agree most with, second most with and least with:
27
Question 11
You: I identified some items that I think may be problematic, and I want to year what you
think about them. The first item I identified is 2.16
Fanie: The item is relevant in terms of content, but I think there is a problem regarding its
construct relevance and of the clarity of its formulation
Temba: I do not see a problem with the clarity of the item, but I think there is a problem with
its construct as well as its content relevance
Bets: I think the item is flawed in terms of content and construct relevance as well as
clarity.
You agree most with, second most with and least with:
Question 12
Ben: Although the item is pitched at the correct language level, I think there is a problem
with its clarity and I am also concerned that the item may require prerequisite
knowledge
Bets: The item seems clear to me but I think there is a problem with language level and the
fact that the item requires prerequisite knowledge
Temba: I think clarity is the main problem. I do not see language level as a problem and do
not think the item requires prerequisite knowledge.
You agree most with, second most with and least with:
Question 13
Temba: The item is configured incorrectly, it requires prerequisite knowledge, and it may
have a social desirability bias
Fanie: I do not see a problem with the item’s configuration, but it may elicit a socially
desirable response and it does require prerequisite knowledge
Ben: I do not think the item has a social desirability response bias, but it does require
prerequisite knowledge and there is a problem with the way in which it is configured
28
PYC2606/101
You agree most with, second most with and least with:
Question 14
You agree most with, second most with and least with:
Question 15
Temba: I see two problems with this question, namely that it requires prerequisite knowledge
and that it is formulated as a closed question, whereas an open question is required
Bets: I agree that the question requires prerequisite knowledge, but I do not agree that this
is a closed question.
Fanie: I do not see any problem with the question. It does not require prerequisite
knowledge and it is not a closed question
You agree most with, second most with and least with:
29
Question 16
Fanie: I agree that this item is problematic – it is not configured correctly in terms of the
required response
Temba: I agree that the item is not configured correctly, the basic problem being that this
question should be formulated as a semantic differential type question
Ben: No, I disagree. I think the item is configured correctly and I think it should be kept in
its current Likert scale form
You agree most with, second most with and least with:
Question 17
You agree most with, second most with and least with:
Question 18
Bets: Yes, I see the problem. The question is not properly formulated as a true-false type
item
Temba: I think the main problem is that the question may have a socially desirable response
bias
Fanie: The real problem is that the item does not record respondents’ responses in a
consistent manner
You agree most with, second most with and least with:
30
PYC2606/101
Question 19
You: I have another item I would like to add to our list of problematic questions, namely 4.3
Ben: The item has contextual relevance, but it does not contribute to the purpose of the
questionnaire
Fanie: I agree, Ben, and I think the item would have served the purpose of the questionnaire
better if it was an open and not a closed question
Bets: I disagree. I think the question has high construct relevance and fits the purpose of
the questionnaire well
You agree most with, second most with and least with:
Question 20
You: Let us have a look at the empirical information that we have for the AIQ. This is how
the 30 participants in the pilot study responded to Item 2.5:
Strongly Agree Unsure Disagree Strongly
Agree Disagree
2 6 5 5 12
What do these responses tell us about the attitude of the pilot group towards drinking
and health?
Bets: The distribution of the responses is skewed to the right, but this does not tell us about
the pilot group’s attitude towards drinking and health
Fanie: Generally speaking the pilot group was indecisive about the health effects of drinking
Ben: The majority of the pilot group had a positive outlook on drinking and health
You agree most with, second most with and least with:
Question 21
Bets: Table 1 in the manual shows the participants’ test scores for two halves of the AIQ.
But I do not understand what it means.
Fanie: Table 1 gives the total scores for two halves of the AIQ. For example participant 1
obtained 26 on one half of the AIQ and 24 on the other half of the AIQ
Temba: I agree, and it means the table shows the correlation between the two halves of the
AIQ, and in this case the numbers reflect a high correlation
Ben: I disagree. The information in the table shows a low correlation between the odd and
even item items of the AIQ
31
You agree most with, second most with and least with:
Question 22
Bets: But does the information in Table 1 tell us anything about the reliability of the AIQ?
Temba: Yes, the information in Table 1 shows that the AIQ has a high level of internal
consistency
Ben: I disagree. The information shows a low level of internal consistency
Fanie: You are both wrong. The information in Table 1 indicates a high degree of concurrent
validity
You agree most with, second most with and least with:
Question 23
Bets: I am still unsure about correlations. For example, in Table 2, how do I see whether
scores are correlated?
Fanie: Draw a graph consisting of a horizontal X-axis and a vertical Y-axis and plot the
relationship between the ‘tested at arrival totals’ and the ‘tested after 3 days totals’
Ben: Scale the X-axis from 1 to 5, and the Y-axis from 44 to 76, and plot the ‘tested after 3
days totals’
Temba: Scale the X-axis from 44 to 76 and the Y-axis from 33 to 62 and plot the
corresponding values
You agree most with, second most with and least with:
32
PYC2606/101
Question 24
You: So, now that you have plotted the information provided in Table 2, what do you think
the information tells us?
Bets: A weak correlation was obtained in the pilot study, which meant the AIQ did not
demonstrate test-retest reliability
Temba: The weak correlation obtained in the pilot study referred to criterion-related validity
and not to the reliability of the AIQ
Ben: Actually the information in Table 2 does not reflect a weak correlation, which means
the AIQ shows good criterion-related validity
You agree most with, second most with and least with:
Question 25
You: So, if you compare the information from Table 1 with the information from Table 2,
what do you think?
Ben: The AIQ has high test-retest reliability but its criterion related validity is questionable
Berhta: The items of the AIQ seem to measure the same thing, but the questionnaire is
vulnerable to external influences
Temba: The information obtained from these two tables supports the reliability of the AIQ
You agree most with, second most with and least with:
33
Discussion questions (Assignment 02) E
Question 1
Temba: I think there is a printing error in the manual. Figure 1 is incomplete with no proper
explanation provided. Do you know what this figure is about?
Bets: Looking at the numbers I think the figure is a graphical representation of the
information provided for Item 2.1 in Table 3
Fanie: If your assumption is correct, Bets, the figure expresses the item-total correlation for
Item 2.1
Ben: No, I think the figure expresses the fact that five people completed each item in the
questionnaire
You agree most with, second most with and least with:
Question 2
Fanie: I think the numbers on the X-axis represent the possible responses to Item 2.1
Bets: If the numbers on the X-axis represent the possible responses to Item 2.1 it means
five people completed the item
Ben: If the X-axis represents Item 2.1 I would say it means the numbers 1 to 5 are
associated with the scale categories that range from disagree strongly to agree
strongly
You agree most with, second most with and least with:
Question 3
Temba: And those numbers in the square brackets, [1], [2], etc. What do they stand for?
Fanie: The numbers in the square brackets are discrimination values. They reflect the
proportion of individuals who completed the item correctly
Bets: No, each point indicates the item score and the total score of a particular individual
Ben: For example, [1] means that participant 1 obtained a total score of 44 and also
disagreed strongly with the statement that drinking is relaxing
34
PYC2606/101
You agree most with, second most with and least with:
Question 4
You: Compare Item 2.9 as it is formulated in the questionnaire with the item’s item-total
correlation shown in Figure 2. Do you find this result surprising?
Temba: The item-total correlation is weak, which is not surprising because it is not clear
whether the item reflects a positive or negative attitude towards alcohol consumption
Ben: The item-total correlation is positive and negative, which is not surprising because
the item reflects positive and negative attitudes towards alcohol consumption
Fanie: The item-total correlation is negative, which is not surprising because it is not clear
how the item relates to attitude towards alcohol
You agree most with, second most with and least with:
Question 5
You: Compare Item 2.11 as it is formulated in the questionnaire with the item’s item-total
correlation shown in Figure 3. Do you find this result surprising?
Temba: The item-total correlation is positive, which is not surprising because the item reflects
a clear stance in relation to alcohol consumption
Ben: The item-total correlation is negative, which is not surprising because the item
requires reversed scoring
Fanie: The item-total correlation is negative, which is surprising because the item reflects a
positive attitude against alcohol consumption
You agree most with, second most with and least with:
35
Question 6
You: Compare Item 2.13 as it is formulated in the questionnaire with the item’s item-total
correlation shown in Figure 4. Do you find this result surprising?
Fanie: The item-total correlation is positive which is not surprising because individuals with a
high intention to use alcohol should get a high score on the item
Bets: The item-total correlation is positive which is surprising because the item expresses
an attitude that promotes alcohol consumption
Temba: The item-total correlation is weak which is not surprising because the item requires
reversed scoring
You agree most with, second most with and least with:
Question 7
You: Compare Item 2.17 as it is formulated in the questionnaire with the item’s item-total
correlation shown in Figure 5. Do you find this result surprising?
Bets: The correlation is negative, which is not surprising because the item required
reversed scoring
Ben: The correlation is weak, which is not surprising because the item has a very strong
social response bias with low discrimination between individuals
Temba: The correlation is positive, which is surprising because the item is expresses a
negative value against alcohol consumption
You agree most with, second most with and least with:
Question 8
You: Given your previous considerations, which of the four items would you discard and
why?
Fanie: Item 2.9 because of low item discrimination and Item 2.11 due to negative item
discrimination
Temba: Item 2.11 because of its negative item discrimination and 2.13 because it has low
item variance
Ben: Item 2.9 because of low item discrimination and Item 2.17 due to low item variance
36
PYC2606/101
You agree most with, second most with and least with:
Question 9
You: Given the way in which we want to use the AIQ, how would you approach the
reliability of the questionnaire?
Ben: It is important to make sure that the AIQ has high internal consistency because we
need a reliable measure of the ‘intention to use alcohol’ construct
Bets: I think test-retest reliability is more important than internal consistency, given the way
in which we want to use the AIQ
Temba: Because we want to identify learners at risk of illegal drug use it is important to
consider criterion-related reliability as the most important component of the AIQ’s
reliability
You agree most with, second most with and least with:
Question 10
Bets: I presume the same goes for validity. We have to consider our approach concerning
the validity of the AIQ
Temba: Because our goal is to identify learners who could benefit from drug use prevention
programmes I think it is important to consider the concurrent validity of the AIQ
Ben: I agree, Fanie, but we have to make sure the AIQ has construct validity otherwise the
AIQ’s concurrent validity may be suspect
Fanie: Face validity is the most important aspect of validity to consider. If the learners
cannot see that the AIQ is about intention to use alcohol they may think the
questionnaire has no value
You agree most with, second most with and least with:
37
Question 11
You: Considering the information provided in Table 5, what do you think of the construct
validity of the AIQ?
Bets: The high correlation between the Attitude and Norm factors supports the construct
validity of the AIQ
Ben: The low correlation between the Recreation factor and the Attitude and Norm factors
shows low construct validity of the AIQ
Fanie: The information in Table 5 supports the construct validity of the AIQ, but the AIQ
does not fully represent the content domain of the construct it purports to assess
You agree most with, second most with and least with:
Question 12
Temba: I think Table 6 provides further information about the construct validity of the AIQ?
Ben: Yes, the correlations among the questionnaires show convergent and discriminent
validities that support construct validity for the AIQ
Fanie: In addition to supporting construct validity for the AIQ the correlation between the
AIQ and the treatment suitability ratings also indicates support for concurrent validity
for the AIQ
Bets: The table shows convergent validity but it does not support discriminant validity for
the AIQ
You agree most with, second most with and least with:
You: I think at this stage of the discussion it is a good idea to sit down and each on our own
rate the AIQ and its manual. The rating questions are listed below. Each question has a
list of criteria (a, b, c, etc) that have to be applied. The criteria are followed by a rating
scale in which ratings (1, 2, 3, etc) are linked with criteria combinations. Because we
apply the same criteria we should come up with similar ratings. Simply consider each
criterion and decide whether or not the criterion is fulfilled. If it is fulfilled it counts,
otherwise it does not count. For example, suppose a question has three criteria, ‘a’, ‘b’
and ‘c’, and you decide that criterion ‘a’ was fulfilled, but criteria ‘b’ and ‘c’ were not
achieved. Find the rating that corresponds to ‘a’, (e.g. rating 1) and indicate this rating
as your answer.
38
PYC2606/101
Question 13
Question 14
Question 15
39
Question 16
Question 17
Question 18
40
PYC2606/101
Question 19
Question 20
a the information gathered by the AIQ concerns the intended content domain
b the AIQ is suitable for the intended target population
c the AIQ produces information in a format that facilitates data processing
Question 21
Question 22
Question 23
Rate the manual description of the procedures for utilisation of the AIQ
Question 24
Question 25
42
PYC2606/101
13 APPENDIX SEMESTER 2
Assignment instructions
What to do:
For Assignment 01:
• Read the scenario context (A), the questionnaire (B) and the mini manual (C). These
sections contain information required for answering the discussion questions of the first
meeting. Do not complete the questionnaire (B) itself
• Answer the 25 questions of the first discussion meeting (D) and submit your answers as
Assignment 01
• Note that this assignment is a multiple choice question assignment. It is better to submit
the assignment online using myUnisa. Only use the printed mark reading sheet and the
postal service if you are unable to submit the assignment online.
43
Scenario context A
You have recently been appointed as the psychologist at a school. The school board and the parent
association are becoming increasingly concerned about the apathy, disinterest and poor performance
displayed by learners at the school. Your predecessor began a motivational and academic support
program, but the programme consisted mainly of additional classes offered for poor performers. You
realise that you have to identify and target learners who are at risk of losing interest (i.e. showing
emotional burnout) and becoming underperformers. You decide that the easiest and most economical
way of doing this is to use a questionnaire to identify learners who are at risk. This will allow you to tailor
motivational and support programmes to accommodate the needs and circumstances of these learners.
You consult academic literature on emotional burnout in and motivational support for young people. In
the literature you come across an interesting fact, namely that suppressed leadership qualities correlate
strongly with emotional burnout resulting in underperformance. Thus information about learners’ latent
leadership qualities could indicate whether they should be considered at risk for emotional burnout and
underperformance.
You also notice that many of the publications refer to a transformational model of leadership. The model
involves two components, namely an interpersonal and a visionary component. At an interpersonal level
the personal traits, abilities, etcetera of the leader are important as these determine the extent to which
others identify with and are prepared to follow the leader. The visionary level on the other hand implies
the leader’s ability to inspire others to identify with and strive towards common goals. Four leadership
styles have been identified that represent the interpersonal and inspirational levels of leadership, namely
consideration (the extent to which the leader helps followers in their personal development), influence
(the extent to which the leader serves as a role model and to which others identify with him or her),
inspiration (the extent to which the leader is able to motivate others to strive towards a vision), and
intellectual stimulation (the extent to which the leader encourages the creativity of others).
You think the model is important because not only could the suppression of these behavioural styles
lead to emotional burnout and underperformance but knowing more about learners’ preferred styles may
help the interaction between learners and teachers, and also among learners themselves. You contact
several test publishers to see whether you can obtain a questionnaire based on this model. You finally
come across a questionnaire called the Leadership Questionnaire (LQ). You order the questionnaire
from the test publisher. When the LQ arrives you decide to conduct a thorough analysis to determine
whether the questionnaire is a scientifically valid and reliable instrument. There are four psychology
students undergoing practical training at the school under your guidance. You ask them to study the
questionnaire (B) and its mini manual (C) and you schedule two meetings (D and E) to discuss the
instrument.
44
PYC2606/101
Questionnaire B
Instructions
Please read the following questions carefully and indicate your response to each question by ticking the
answer that best describes how you feel about the issues raised by these questions. Please answer all
questions. Do not spend too much time thinking about your answers. The questionnaire should not take
long to complete.
1.1 I have been told by a person I look up to (e.g. parents/teachers) that I am worthless and that I will
not get far in life.
(a) Yes
(b) No
If ‘No’ go to Question 2.1
Section 2: Influence
45
2.9 I have a strong sense of purpose [ _ ] [ _ ] [ _ ] [ _ ] [ _ ]
2.10 I try to convince others in subtle ways [ _ ] [ _ ] [ _ ] [ _ ] [ _ ]
of the importance of trust
2.11 I often show people at the office how [ _ ] [ _ ] [ _ ] [ _ ] [ _ ]
to fix machines because I have
mechanical instinct
2.12 My style can be described as leading [ _ ] [ _ ] [ _ ] [ _ ] [ _ ]
or following
2.13 I influence people to become better
persons
Section 3: Inspiration
3.2 When working as a team, I am confident that we will achieve our goals.
(a) Always
(b) Most of the time
(c) Never
3.6 Rate the future by placing a cross closest to the word that you associate with it.
3.7 Inspirational leaders inspire people even when they do not lead when not required to do so
(a) True
(b) False
3.8 There are stereotypes associated with inspirational people. A person who ……… is classified as
highly inspirational, whereas a person who ……. is not seen as inspirational.
46
PYC2606/101
4.7 I seek input from various people when dealing with problems.
(a) True
(b) False
4.8 Bright people like intellectual stimulation, and society needs bright people. I prefer tasks that
require stimulating ideas.
(a) True
(b) False
47
5.2 Mark the answers that apply to you
People who participate in extramural activities:
- are bored [ ... ]
- are socially deprived [ ... ]
- should rather spend more time with their families [ ... ]
5.5 Participating in extramural activities keeps one’s mind focused True / False
5.6 On a scale of 1 (least serious) to 5 (most serious), rate the importance of the following effects of
physical passivity and obesity
liver failure […]
heart disease […]
cognitive impairment […]
memory loss […]
social isolation […]
personality change […]
.........................................................................................................................................
.........................................................................................................................................
End of questionnaire
48
PYC2606/101
Mini manual C
Mini manual for the Leadership Questionnaire (LQ)
1. Aim and design
The LQ is based on the transformational model of leadership. This model distinguishes between an
interpersonal and a visionary component of leadership. Personal identification serves as motivator in the
case of the interpersonal component. Leaders are seen to have exceptional skills or talents and
followers strongly identify with them and show a great deal of commitment to them. Leadership styles
that represent this component are influence and consideration. Questions in the Influence Scale deal
with the leader as a role model, confidence in the leader, trust in the leader, ethical behaviour by the
leader, etcetera. Questions in the Consideration Scale deal with coaching of followers, the development
of followers, attention to the followers’ needs, etcetera. In the case of the visionary component, goal
identification is the motivator. Followers identify with common goals, change their values accordingly,
and are empowered to achieve these goals. Leadership styles that represent this component are
inspiration and intellectual stimulation. Questions in the Inspiration Scale deal with optimism about the
future, with enthusiasm, with formulating a vision, etc. Questions in the Intellectual stimulation Scale deal
encouraging creativity, following new approaches to problems, new ideas, etc. Leadership implies
followership. Therefore assessment of leadership could include ratings by the leader and/or ratings by
followers. The LQ is based only on the former. It is a self-report measure. The aim of the LQ is to
measure a person’s preferred behaviour in terms of the four leadership styles, influence, consideration,
inspiration and intellectual stimulation.
2. Properties of the LQ
The LQ was pilot tested in a banking environment. For the initial pilot study five participants were
selected randomly form a larger group attending a trainee programme offered by the bank. They were
tested when they arrived for the training programme. The LQ was divided into two sets of scores. The
first set consisted of the even item numbers and the second set consisted of the odd item numbers. The
scores of the five trainees are indicated in Table 1. The trainees were tested again four days into the
training programme. This information is shown in Table 2.
The researchers were also interested in how each of the items correlated with the participants’ total
scores. Table 3 shows this information for one of the items. The researchers compiled similar tables for
each item in the questionnaire.
49
Table 3: The relationship between participants’ Item 2.1 scores and
their total scores
Item 2.1 scale values 1 2 3 4 5 Total
Score for participant 1 x 44
Score for participant 2 x 53
Score for participant 3 x 68
Score for participant 4 x 57
Score for participant 5 x 76
Y-axis
75-79 [5]
70-74
65-69 [3]
60-64
55-59 [4]
50-54 [2]
45-49
40-44 [1]
1 2 3 4 5 X-axis
Figure 1:
Soon after the initial pilot study the bank added another 5 participants to the pilot. These individuals were
tested when they arrived for training. The results obtained from the expanded pilot study enabled the
researchers to calculate the correlations between each of the items and the total scores. These
correlations were used to discard weak items. Table 4 shows the scores obtained by each of the
research participants for four of the items (questions 2.10, 2.6, 2.9 and 2.13) together with their total test
scores.
Note: A total score is the sum of a person’s items scores for all items in the questionnaire. Only four of the items
scores are shown in this table.
50
PYC2606/101
The scatter plots for the items listed in Table 1 are shown in Figures 1, 2, 3 and 4.
100
90
80 ■ ■
70 ■
60 ■ ■
Total scores 50 ■ ■
40 ■
30 ■ ■
20
10
1 2 3 4 5
Item scores
Figure 2: Scatter plot for Item 2.10 — each ■ represents a research
participant
100
90
80 ■ ■
70 ■
60 ■ ■
Total scores 50 ■ ■
40 ■
30 ■ ■
20
10
1 2 3 4 5
Item scores
Figure 3: Scatter plot for Item 2.6 — each ■ represents a research
participant
100
90
80 ■ ■
70 ■
60 ■ ■
Total scores 50 ■ ■
40 ■
30 ■ ■
20
10
1 2 3 4 5
Item scores
Figure 4: Scatter plot for Item 2.9 — each ■ represents a research
participant
51
100
90
80 ■ ■
70 ■
60 ■
Total scores 50 ■ ■
40 ■ ■
30 ■
20 ■
10
1 2 3 4 5
Item scores
Figure 5: Scatter plot for Item 2.13 — each ■ represents a research
participant
Over a period of six months the bank was able to add more participants to the pilot study. The final pilot
study included 30 participants between 27 and 40 years of age, with 16 males and 14 females. During
their attendance of the training programme these participants were asked to complete two additional
questionnaires, the General Management Style Questionnaire (GMSQ) and the Knowledge of Financial
Systems Questionnaire (KFSQ). The GMSQ was used to asses behaviour characteristics associated
with effective and efficient management. The KFSQ was used to determine a person’s knowledge of
financial processes and procedures typically encountered in a banking environment. The presenters of
the trainee programme as well the immediate managers of the trainees also rated the participants in
terms of suitability to take up a leadership role in the bank. These were indicated as Presenter Manager
Ratings (PMR).
The following tables contain further information that was gathered during the pilot study. Table 5 shows
the inter-correlations among the factors addressed by the LQ, obtained for the pilot group of 30
participants. Table 6 contains the inter-correlations among the questionnaires and the ratings provided
by the trainee programme presenters and the trainees’ immediate managers.
52
PYC2606/101
Administration
The LQ contains the questionnaire items and instructions for completing the questionnaire. Individuals’
responses are recorded on the questionnaire. Test takers need a pencil to indicate their responses. The
questionnaire is a psychological tests and this affects who is allowed to administer, score and interpret
the questionnaire. It can be administered to individuals or to groups and should not take more than 30
minutes to complete. Respondents should be instructed to answer the questionnaire honestly. Items may
be explained if necessary but respondents should not be guided towards a specific answer.
Scoring
Score the items on all scales (Influence, Inspiration, Intellectual stimulation by giving a mark of 1 for low
and a mark of 5 for high.
Interpretation
The results of the questionnaire provide information on overall leadership potential as well as of the
extent to which a person displays each of the four leadership styles. The manner in which this
information is used depends on the purpose of testing. If a group is assessed to select those with
leadership potential, the total score is used. If an individual is evaluated to determine if there are specific
areas that need attention, scores on the subscales are interpreted in the following manner. A high score
on Influence implies that followers have trust in the leader, they identify with him or her, and they are
committed to the leader’s values and goals. A high score on Inspiration implies that the leader is
enthusiastic and optimistic about the future and by providing meaningful goals he or she also inspires
others to work towards these goals. A high score on Intellectual Stimulation implies that the leader
questions assumptions and approach problems in a new manner. He or she also encourages followers
to do the same. A high score on Consideration implies that the leader understands the developmental
needs of each individual follower and efforts are made to meet these needs.
End of manual
53
Discussion questions (Assignment 01) D
Question 1
Hes: I presume a good place to begin this discussion is the intention of the LQ.
You agree most with, second most with and least with:
Question 2
You: Do you think the questionnaire is fit for purpose given our context? In other words, do
you think we can use the questionnaire for what we need it for?
Werner: Yes, because we need the LQ to identify learners who are at risk emotional burnout
and underperformance, which fits the intention of the LQ
Hes: Yes, because we need the LQ to identify learners’ preferred behaviour styles, which
fits the intention of the LQ
Vera: Yes, because the intention of the LQ is to assess all learners and we want to use the
LQ to tailor motivational and support programmes to accommodate the needs of
these learners
You agree most with, second most with and least with:
Question 3
Pule: Given the intention of the LQ, what is its content domain?
54
PYC2606/101
You agree most with, second most with and least with:
Question 4
Hes: I am not convinced that the scope of the LQ is sufficient in light of its intention and in
light of what we need it for.
Vera: I share your concern, Hes. The items included in the LQ do not fully represent the
content domain
Pule: I am not concerned about the scope of the LQ. I think it has sufficient scope to
assess learners at any school
Werner: If one takes into account that the LQ consists of different sections with different kinds
of items it is clear that the LQ has sufficient scope to realise its purpose
You agree most with, second most with and least with:
Question 5
You: Do you think the LQ is the correct type of questionnaire for our purposes?
Pule: The LQ is a survey type questionnaire and because we want to survey preferred
behaviour style I think the LQ is the correct type of questionnaire for our purposes
Hes: I agree that the LQ is a survey type questionnaire. But we want to determine the
learner’s ability to cope with emotional burnout and underperformance and therefore I
think the LQ may not be the correct type of questionnaire for our purposes
Werner: Actually the LQ is an ability type questionnaire, but it is not the correct type of
questionnaire for our purposes because we want to determine the personality of the
learner who suffers from emotional burnout and underperformance
You agree most with, second most with and least with:
55
Question 6
Werner: Do you think the instructions that the LQ provides to respondents are correct and
complete?
Vera: The instructions provide the correct information but are incomplete
Hes: I think the instructions provide the correct information and they are complete
Pule: No, I think the information provided by the instructions are incorrect and the
instructions are also incomplete
You agree most with, second most with and least with:
Question 7
You: Do you think confidentiality is an issue considering the way we plan to use the LQ?
Werner: I do not think confidentiality is an issue because the LQ does not require respondents
to provide their names
Vera: I think confidentiality is an issue because our ultimate goal is to identify learners who
are at risk for emotional burnout and underperformance
Hes: To my mind confidentiality is not an issue because these learners are under-aged
You agree most with, second most with and least with:
Question 8
Hes: What do you think of the sequence in which items are presented in the LQ?
Vera: The items of the LQ are sequenced well. They are grouped into sections which allow
for better focussed responses and they progress from questions about personal
experiences to questions about personal opinions to questions requiring factual
information, ending with personal experiences again
Werner: The sequencing of the items is not really an issue in the LQ. All the items except the
last one are equally easy to respond to, requiring tick response only
Pule: The items in Section 1 should be moved to the end of the questionnaire due to the
emotional impact of these questions and the possibility of them putting the
respondent in a particular frame of mind
56
PYC2606/101
You agree most with, second most with and least with:
Question 9
You: Another aspect we should consider is whether the LQ incorporates different item
response formats. Can you identify items that demonstrate different response
formats?
Werner: Items 1.2 and 3.3 are questions with limited choice of answers whereas others, such
as 5.2, provide a checklist. I also noted that some items such as 2.4 and 2.6 require
inversed responses
Pule: Items 2.2 and 2.7 are further examples of questions that require inversed responses,
and the semantic differential type of items contained in Section 2 clearly differ from
checklist type items, such as 3.3
Hes: I identified another item in Section 2 that requires inversed responses, namely 2.10.
Also keep in mind that the open question type such as 5.7 differs from the multiple
choice type items provided in Section 2.
You agree most with, second most with and least with:
Question 10
Vera: Item 3.4 is an example of a filter question, but unfortunately it does not operate
correctly as a filter
Werner: Item 5.1 is an example of a filter question, and I think it operates correctly as a filter
Hes: I agree that item 5.1 is an example of a filter question, but I do not think it operates
correctly as a filter
You agree most with, second most with and least with:
57
Question 11
You: I identified some items that I think may be problematic, and I want to hear what you
think about them. The first item I identified is 2.11
Pule: The item is relevant in terms of content, but I think there is a problem regarding its
construct relevance and of the clarity of its formulation
Vera: I do not see a problem with the clarity of the item, but I think there is a problem with
its construct as well as its content relevance
Hes: I think the item is flawed in terms of content and construct relevance as well as
clarity.
You agree most with, second most with and least with:
Question 12
Werner: Although the item is pitched at the correct language level, I think there is a problem
with its clarity and I am also concerned that the item may require prerequisite
knowledge
Hes: The item seems clear to me but I think there is a problem with language level and the
fact that the item requires prerequisite knowledge
Vera: I think clarity is the main problem. I do not see language level as a problem and do
not think the item requires prerequisite knowledge.
You agree most with, second most with and least with:
Question 13
Vera: The item is configured incorrectly, it requires prerequisite knowledge, and it may
have a social desirability bias
Pule: I do not see a problem with the item’s configuration, but it may elicit a socially
desirable response and it does require prerequisite knowledge
Werner: I do not think the item has a social desirability response bias, but it does require
prerequisite knowledge and there is a problem with the way in which it is configured
58
PYC2606/101
You agree most with, second most with and least with:
Question 14
Hes: Although the item has content and construct relevance it is leading and likely to elicit
a socially desirable response
Pule: I agree that the item has content and construct relevance and that it may elicit a
social desirable response, but it is not leading
Werner: I do not think the item is leading and I do not think it has a social desirability bias, but
I do think it does not have content and construct relevance
You agree most with, second most with and least with:
Question 15
Vera: I see two problems with this question, namely that it requires prerequisite knowledge
and that it is formulated as a closed question, whereas an open question is required
Hes: I agree that the question requires prerequisite knowledge, but I do not agree that this
is a closed question.
Pule: I do not see any problem with the question. It does not require prerequisite
knowledge and it is not a closed question
You agree most with, second most with and least with:
59
Question 16
Pule: I agree that this item is problematic – it is not configured correctly in terms of the
required response
Vera: I agree that the item is not configured correctly, the basic problem being that this
question should be formulated as a semantic differential type question
Werner: No, I disagree. I think the item is configured correctly and I think it should be kept in its
current Likert scale form
You agree most with, second most with and least with:
Question 17
You agree most with, second most with and least with:
Question 18
Hes: Yes, I see the problem. The question is not properly formulated as a true-false type
item
Vera: I think the main problem is that the question may have a socially desirable response
bias
Pule: The real problem is that the item does not record respondents’ responses in a
consistent manner
You agree most with, second most with and least with:
60
PYC2606/101
Question 19
You: I have another item I would like to add to our list of problematic questions, namely 5.3
Werner: The cannot see the contextual relevance of this question, and it does not contribute
to the purpose of the questionnaire
Pule: I agree, Werner, and I think the item would have served the purpose of the
questionnaire better if it was an open and not a closed question
Hes: I disagree. I think the question has high construct relevance and fits the purpose of
the questionnaire well
You agree most with, second most with and least with:
Question 20
You: Let us have a look at the empirical information that we have for the LQ. This is how
the 30 participants in the pilot study responded to Item 2.3:
Strongly Agree Unsure Disagree Strongly
Agree Disagree
1 6 5 5 13
What do these responses tell us about the nature of the pilot group’s influencing
behaviour?
Hes: The distribution of the responses is skewed to the right, but this does not tell us about
the nature of the pilot group’s influencing behaviour
Pule: Generally speaking the pilot group was indecisive about the nature of their
influencing behaviour
Werner: The majority of the pilot group denied the nature of their influencing behaviour
You agree most with, second most with and least with:
61
Question 21
Hes: Table 1 in the manual shows the participants’ test scores for two halves of the LQ.
But I do not understand what it means.
Pule: Table 1 gives the total scores for two halves of the LQ. For example participant 1
obtained 27 on one half of the LQ and 23 on the other half of the LQ
Vera: I agree, and it means the table shows the correlation between the two halves of the
LQ, and in this case the numbers reflect a high correlation
Werner: I disagree. The information in the table shows a low correlation between the odd and
even item items of the LQ
You agree most with, second most with and least with:
Question 22
Hes: But does the information in Table 1 tell us anything about the reliability of the LQ?
Vera: Yes, the information in Table 1 shows that the LQ has a high level of internal
consistency
Werner: I disagree. The information shows a low level of internal consistency
Pule: You are both wrong. The information in Table 1 indicates a high degree of concurrent
validity
You agree most with, second most with and least with:
Question 23
Hes: I am still unsure about correlations. For example, in Table 2, how do I see whether
scores are correlated?
Pule: Draw a graph consisting of a horizontal X-axis and a vertical Y-axis and plot the
relationship between the ‘tested at arrival totals’ and the ‘tested after 3 days totals’
Werner: Scale the X-axis from 1 to 5, and the Y-axis from 43 to 77, and plot the ‘tested after 4
days totals’
Vera: Scale the X-axis from 43 to 77 and the Y-axis from 32 to 63 and plot the
corresponding values
62
PYC2606/101
You agree most with, second most with and least with:
Question 24
You: So, now that you have plotted the information provided in Table 2, what do you think
the information tells us?
Hes: A weak correlation was obtained in the pilot study, which meant the LQ did not
demonstrate test-retest reliability
Vera: The weak correlation obtained in the pilot study referred to criterion-related validity
and not to the reliability of the LQ
Werner: Actually the information in Table 2 does not reflect a weak correlation, which means
the LQ shows good criterion-related validity
You agree most with, second most with and least with:
Question 25
You: So, if you compare the information from Table 1 with the information from Table 2,
what do you think?
Werner: The LQ has high test-retest reliability but its criterion related validity is questionable
Bets: The items of the LQ seem to measure the same thing, but the questionnaire is
vulnerable to external influences
Vera: The information obtained from these two tables supports the reliability of the LQ
You agree most with, second most with and least with:
63
Discussion questions (Assignment 02) E
Question 1
Vera: I think there is a printing error in the manual. Figure 1 is incomplete with no proper
explanation provided. Do you know what this figure is about?
Hes: Looking at the numbers I think the figure is a graphical representation of the
information provided for Item 2.1 in Table 3
Pule: If your assumption is correct, Hes, the figure expresses the item-total correlation for
Item 2.1
Werner: No, I think the figure expresses the fact that five people completed each item in the
questionnaire
You agree most with, second most with and least with:
Question 2
Pule: I think the numbers on the X-axis represent the possible responses to Item 2.1
Hes: If the numbers on the X-axis represent the possible responses to Item 2.1 it means
five people completed the item
Werner: If the X-axis represents Item 2.1 I would say it means the numbers 1 to 5 are
associated with the scale categories that range from disagree strongly to agree
strongly
You agree most with, second most with and least with:
Question 3
Vera: And those numbers in the square brackets, [1], [2], etc. What do they stand for?
Pule: The numbers in the square brackets are discrimination values. They reflect the
proportion of individuals who completed the item correctly
Hes: No, each point indicates the item score and the total score of a particular individual
Werner: For example, [1] means that participant 1 obtained a total score of 44 and also
disagreed strongly with the statement that 'others are proud to be associated with
me"
64
PYC2606/101
You agree most with, second most with and least with:
Question 4
You: Compare Item 2.10 as it is formulated in the questionnaire with the item’s item-total
correlation shown in Figure 2. Do you find this result surprising?
Vera: The item-total correlation is weak, which is not surprising because it is not clear
whether the item reflects inspirational or non-inspirational behaviour
Werner: The item-total correlation is positive and negative, which is not surprising because
the item reflects inspirational and non-inspirational
Pule: The item-total correlation is negative, which is not surprising because it is not clear
how the item relates to inspirational behaviour
You agree most with, second most with and least with:
Question 5
You: Compare Item 2.6 as it is formulated in the questionnaire with the item’s item-total
correlation shown in Figure 3. Do you find this result surprising?
Vera: The item-total correlation is positive, which is not surprising because the item reflects
a clear stance to inspirational behaviour
Werner: The item-total correlation is negative, which is not surprising because the item
requires reversed scoring
Pule: The item-total correlation is negative, which is not surprising because the item
reflects non-inspirational behaviour
You agree most with, second most with and least with:
65
Question 6
You: Compare Item 2.9 as it is formulated in the questionnaire with the item’s item-total
correlation shown in Figure 4. Do you find this result surprising?
Pule: The item-total correlation is positive which is not surprising because individuals who
measure high on inspirational behaviour should get a high score on this item
Hes: The item-total correlation is negative which is surprising because the item expresses
influential behaviour
Vera: The item-total correlation is weak which is not surprising because the item requires
reversed scoring
You agree most with, second most with and least with:
Question 7
You: Compare Item 2.13 as it is formulated in the questionnaire with the item’s item-total
correlation shown in Figure 5. Do you find this result surprising?
Hes: The correlation is negative, which is not surprising because the item required
reversed scoring
Werner: The correlation is weak, which is not surprising because the item has a very strong
social response bias with low discrimination between individuals
Vera: The correlation is positive, which is surprising because the item portrays influencing
behaviour in a negative light
You agree most with, second most with and least with:
Question 8
You: Given your previous considerations, which of the four items would you discard and
why?
Pule: Item 2.10 because of low item discrimination and Item 2.6 due to negative item
discrimination
Vera: Item 2.6 because of its negative item discrimination and 2.9 because it has low item
variance
Werner: Item 2.10 because of low item discrimination and Item 2.13 due to low item variance
66
PYC2606/101
You agree most with, second most with and least with:
Question 9
You: Given the way in which we want to use the LQ, how would you approach the
reliability of the questionnaire?
Werner: It is important to make sure that the LQ has high internal consistency because we
need a reliable measure of the ‘preferred behaviour style’ construct
Hes: I think test-retest reliability is more important than internal consistency, given the way
in which we want to use the LQ
Vera: Because we want to identify learners at risk of emotional burnout and
underperformance it is important to consider criterion-related reliability as the most
important component of the LQ’s reliability
You agree most with, second most with and least with:
Question 10
Hes: I presume the same goes for validity. We have to consider our approach concerning
the validity of the LQ
Vera: Because our goal is to identify learners who could benefit from motivational and
support programmes I think it is important to consider the concurrent validity of the
LQ
Werner: I agree, Pule, but we have to make sure the LQ has construct validity otherwise the
LQ’s concurrent validity may be suspect
Pule: Face validity is the most important aspect of validity to consider. If the learners
cannot see that the LQ is about preferred behaviour style they may think the
questionnaire has no value
You agree most with, second most with and least with:
67
Question 11
You: Considering the information provided in Table 5, what do you think of the construct
validity of the LQ?
Hes: The high correlations between the Influence, Inspiration and Intellectual stimulation
factors support the construct validity of the LQ
Werner: The low correlation between the Recreation factor and the Influence, Inspiration and
Intellectual stimulation factors show low construct validity of the LQ
Pule: The information in Table 5 supports the construct validity of the LQ, but the LQ does
not fully represent the content domain of the construct it purports to assess
You agree most with, second most with and least with:
Question 12
Vera: I think Table 6 provides further information about the construct validity of the LQ?
Werner: Yes, the correlations among the questionnaires show convergent and discriminent
validities that support construct validity for the LQ
Pule: In addition to supporting construct validity for the LQ the correlation between the LQ
and the presenter-manager ratings also indicates support for concurrent validity for
the LQ
Hes: The table shows convergent validity but it does not support discriminant validity for
the LQ
You agree most with, second most with and least with:
You: I think at this stage of the discussion it is a good idea to sit down and each on our own
rate the LQ and its manual. The rating questions are listed below. Each question has a
list of criteria (a, b, c, etc) that have to be applied. The criteria are followed by a rating
scale in which ratings (1, 2, 3, etc) are linked with criteria combinations. Because we
apply the same criteria we should come up with similar ratings. Simply consider each
criterion and decide whether or not the criterion is fulfilled. If it is fulfilled it counts,
otherwise it does not count. For example, suppose a question has three criteria, ‘a’, ‘b’
and ‘c’, and you decide that criterion ‘a’ was fulfilled, but criteria ‘b’ and ‘c’ were not
achieved. Find the rating that corresponds to ‘a’, (e.g. rating 1) and indicate this rating
as your answer.
68
PYC2606/101
Question 13
Question 14
Question 15
69
Question 16
Question 17
Question 18
70
PYC2606/101
Question 19
Question 20
Question 21
Question 22
71
Rate 1 if: none of a, b, c, d
Rate 2 if: a
Rate 3 if: a+b
Rate 4 if: a + b + c or a + b + d
Rate 5 if: a+b+c+d
Question 23
Question 24
Question 25
72