Feedback Assignment 01 Semester 2 2016

Study the scenario carefully before The school community

you attempt to answer the
assignment questions. Always
unpack an assessment scenario in
terms of the following: The
assessment context

The concern raised in the The apathy, disinterest and poor

assessment context performance displayed by learners
at the school

The owner of the concern The school board and parent

association

The consultant The psychologist who is

responsible to address the concern

The intention of the consultant To identified information about

learners’ behaviour styles (latent
leadership qualities) to prevent
emotional burnout resulting in
underperformance

The test user The psychologist who uses the

questionnaire (In this scenario the
consultant and the test user is the
same person, namely the
psychologist)

The intention of the test user To identify and target learners who
are at risk of loosing interest (i.e.
showing emotional burnout) and
becoming underperformers.

The group targeted for assessment The learners at the school

Open Rubric
The questionnaire Leadership Questionnaire
Questionnaire (LQ)

The intention of the questionnaire To determine individual learners’

preferred behavioural styles that
lead to emotional burnout and
under performance

Question 1

Joly: I presume a good place to begin this discussion is the purpose of the LQ.
Sanna: Well, the purpose of the questionnaire clearly is to determine an individual’s
preferred behaviour style
Paul: No, the questionnaire’s purpose is to identify and target learners who are at risk for
emotional burnout and underperformance
Lesiba: Actually the purpose of the questionnaire is to assess all learners in the school

You agree most with, second most with and least with:
1. Lesiba, Paul, Sanna
2. Paul, Sanna, Lesiba
3. Sanna, Lesiba, Paul
4. Sanna, Paul, Lesiba
5. Paul, Lesiba, Sanna

Answer: Option 4

When Joly refers to the purpose of the questionnaire she actually refers to the
intention of the questionnaire and not to ‘purpose’ in its technical sense and as it
should be described in the questionnaire’s manual. The intention of the
questionnaire is to determine the individual learners’ preferred behavioural styles
that lead to emotional burnout and under performance. Therefore Sanna is correct.
Paul considers the purpose of the questionnaire as the intention of the test user.
The purpose of the test user (in this case the psychologist) is to identify learners
who are at risk for showing emotional burnout and becoming underperformers.
Although Paul’s answer is incorrect it is a better answer than Lesiba’s. The intention
of the test user is a purpose for using the text (although not the questionnaire’s
purpose in itself). Lesiba’s response is incorrect because his interpretation of
purpose does not refer to a reason for using the test. In its technical sense the term
purpose includes the target group of the questionnaire. But Lesiba refers to the
target group of the assessment exercise, which should be but in this case is not the
same as the questionnaire’s target population.
Question 2

You: Do you think the questionnaire is fit for purpose given our context? In other words,
to you think we can use the questionnaire for what we need it for?
Paul: Yes, because we need the LQ to identify learners who are at risk emotional
burnout and underperformance, which fits the purpose of the LQ
Joly: Yes, because we need the LQ to identify learners’ preferred behaviour styles,
which fits the purpose of the LQ
Sanna: Yes, because the purpose of the LQ is to assess all learners and we want to use
the LQ to tailor motivational and support programmes to accommodate the needs
of these learners

You agree most with, second most with and least with:
1. Paul, Joly, Sanna
2. Joly, Paul, Sanna
3. Sanna, Paul, Joly
4. Joly, Sanna, Paul
5. Paul, Sanna, Joly

Answer: Option 2

To answer the question, one first has to determine the purpose of the assessment.
The ultimate goal is to identify information about learners’ latent leadership
qualities to prevent emotional burnout resulting in underperformance. In order to
achieve this goal the psychologist has to identify individual learners’ preferred
behavioural styles that lead to emotional burnout and under performance. The
result obtained from the LQ is based on the transformational model of leadership.
This model distinguishes between an interpersonal and a visionary component of
leadership. In light of this information it is clear that the LQ can contribute useful
information in the context in which it is considered. The three discussants are
correct in saying ‘yes’. Joly provides the best response because she indicates the
correct intention of the LQ. Paul is correct in suggesting that the LQ is required to
identify learners who are at risk emotional burnout and underperformance, but it is
not correct to indicate this as the purpose of the LQ itself. The identification of
learners who at risk of emotional burnout and underperformance is what the
psychologist would like to do with the result of the LQ. Sanna’s response is the
weakest of the three. To assess all learners is the intention of the psychologist is,
not the intention of the LQ.
Question 3

Lesiba: Given the purpose of the LQ, what is its content domain?
Paul: All behaviour associated with emotional burnout and underperformance
Sanna: The content domain is behaviour associated with influence, inspiration, intellectual
stimulation and physical behaviour
Joly: I think the content domain is all forms of behaviour associated with consideration,
influence, inspiration and intellectual stimulation

You agree most with, second most with and least with:

1. Sanna, Paul, Joly

2. Joly, Paul, Sanna
3. Sanna, Joly, Paul
4. Paul, Sanna, Joly
5. Joly, Sanna, Paul

Answer: Option 5

The aim of the LQ is to measure a person’s preferred behaviour in terms of the four
leadership styles, influence, consideration, inspiration and intellectual stimulation.
The LQ is based on the theory of planned behaviour. The theory of planned
behaviour has three factors or constructs, namely the person’s attitude towards the
behaviour, his/her subjective norm concerning the behaviour and his/her perceived
control with regard to the behaviour. These constructs constitute the content
domain of the LQ. Joly’s response is the best answer because she refers to all three
of the required constructs. Sanna’s answer is second best. She refers to two of the
three factors, but adds physical behaviour, which is a section included in the LQ but
not part of its underlying content domain. Paul has the weakest response. It is not
correct to indicate all behaviour associated with emotional burnout and
underperformance.

Question 4

Joly: I am not convinced that the scope of the LQ is sufficient in light of its purpose and
in light of what we need it for.
Sanna: I share your concern, Joly. The items included in the LQ do not fully represent the
content domain
Lesiba: I am not concerned about the scope of the LQ. I think it has sufficient scope to
assess learners at any school
Paul: If one takes into account that the LQ consists of different sections with different
kinds of items it is clear that the LQ has sufficient scope to realise its purpose

You agree most with, second most with and least with:
1. Sanna, Paul, Lesiba
2. Lesiba, Paul, Sanna
3. Sanna, Lesiba, Paul
4. Paul, Sanna, Lesiba
5. Paul, Lesiba, Sanna
Answer: Option 1

Sanna’s response is correct. The LQ does not cover its underlying content domain
fully. Its intention is to determine a person’s preferred behaviour in terms of the four
leadership styles, influence, consideration, inspiration and intellectual stimulation,
but it does not cover all aspects that suppression of these behavioural styles lead
to emotional burnout and underperformance. Therefore it cannot give the full
picture of an individual’s risks, and using it to identify learners who are at risk of
showing emotional burnout and becoming underperformers will be limited if not
invalid. Paul understands scope not in terms of the representation of the underlying
content domain but in terms of the fact that the questionnaire has different sections
(i.e. covers different aspects of the phenomenon being assessed) and uses different
types of items/questions. Although this is a concrete and not really correct
interpretation of scope his answer is better than Lesiba’s. Lesiba is wrong on two
accounts. He should be concerned about the scope of the LQ, but he should also
improve his understanding of the meaning of scope. Scope does not refer to the
individuals in the target population. It refers to the questions included in the
questionnaire.

Question 5

You: Do you think the LQ is the correct type of questionnaire for our purposes?
Lesiba: The LQ is a survey type questionnaire and because we want to survey preferred
behaviour style I think the LQ is the correct type of questionnaire for our purposes
Joly: I agree that the LQ is a survey type questionnaire. But we want to determine the
learner’s ability to cope with emotional burnout and underperformance and
therefore I think the LQ may not be the correct type of questionnaire for our
purposes
Paul: Actually the LQ is an ability type questionnaire, but it is not the correct type of
questionnaire for our purposes because we want to determine the personality of
the learner who suffers from emotional burnout and underperformance

You agree most with, second most with and least with:
1. Lesiba, Joly, Paul
2. Paul, Joly, Lesiba
3. Lesiba, Paul, Joly
4. Joly, Lesiba, Paul
5. Joly, Paul, Lesiba

Answer: Option 1

In this question respondents consider ‘type of questionnaire’ and the ‘purpose for
which the questionnaire is used”. The questionnaire is a survey type questionnaire.
The intention of the test user is to identify information about learners’ preferred
behaviour styles. Lesiba’s response is correct. He identifies the LQ as a survey type
questionnaire and he indicates that the intention is to survey learners’ preferred
behaviour styles. His concludes that the LQ is the correct type of questionnaire for
their purposes. Joly’s response is partially correct. She correctly identifies the type
of questionnaire that the LQ is an example of. However she is incorrect in stating
that the purpose is to determine the learner’s ability to cope with emotional burnout
and underperformance. Although her conclusion is correct in light of her own
reasoning it is incorrect because her misunderstanding of the purpose of the
questionnaire. Paul is incorrect in stating that that LQ is an ability type
questionnaire and he is also incorrect in stating that the purpose is to determine the
personality of the learner.

Question 6

Paul: Do you think the instructions that the LQ provides to respondents are correct and
complete?
Sanna: The instructions that are provided are correct but they are incomplete
Joly: I think the instructions provided are correct and complete
Lesiba: No, I think those instructions are incorrect

You agree most with, second most with and least with:
1. Sanna, Lesiba, Joly
2. Lesiba, Joly, Sanna
3. Sanna, Joly, Lesiba
4. Joly, Sanna, Lesiba
5. Joly, Lesiba, Sanna

Answer: Option 3

The answer should be: Sanna, Joly, Lesiba — the instructions are incomplete
because they do not refer to the intention of the questionnaire and the matter of
confidentiality. There is also a problem with the information that is provided. It asks
respondents to respond to each question by ticking an answer. Some questions
cannot be ticked.

Please note that questionnaire instructions refer to how the questionnaire should be
completed. It does not refer to the instructions provided with each question.

Question 7

You: Do you think confidentiality is an issue considering the way we plan to use the LQ?
Paul: I do not think confidentiality is an issue because the LQ does not require
respondents to provide their names
Sanna: I think confidentiality is an issue because our ultimate goal is to identify learners
who are at risk for emotional burnout and underperformance
Joly: To my mind confidentiality is not an issue because these learners are under-aged

You agree most with, second most with and least with:
1. Sanna, Paul, Joly
2. Joly, Paul, Sanna
3. Sanna, Joly, Paul
4. Paul, Sanna, Joly
5. Paul, Joly, Sanna

Answer: Option 1

It is important to understand what confidentiality and anonymity means in the

context of the assessment scenario. In this particular scenario the learners’ LQ
scores are used to identify learners who are at risk of showing emotional burnout
and becoming underperformers. This clearly means that the information provided
by each learner cannot be anonymous and that it has to be treated confidentially.
Sanna clearly understands the implications for confidentiality in the context of the
assessment. Paul does not think confidentiality is an issue, and he does so on the
grounds of anonymity. In doing this he forgets that although the LQ does not ask
respondents to fill in their names the information cannot be treated anonymously in
the context of the assessment scenario. He provides a reason for his answer. The
reason Paul provides for his response has some value (he compares confidentiality
and anonymity, but he fails to see the implications of the assessment context).
However, Joly’s response is downright incorrect. It is not true that the information
of under-aged learners does not have to be treated confidentially.

Question 8

Joly: What do you think of the sequence in which items are presented in the LQ?
Sanna: The items of the LQ are sequenced well. They are grouped into sections which
allow for better focussed responses and they progress from questions about
personal experiences to questions about personal opinions to questions requiring
factual information, ending with personal experiences again
Paul: The sequencing of the items is not really an issue in the LQ. All the items except
the last one are equally easy to respond to, requiring tick response only
Lesiba: The items in Section 1 should be moved to the end of the questionnaire due to the
emotional impact of these questions and the possibility of them putting the
respondent in particular frame of mind

You agree most with, second most with and least with:
1. Lesiba, Paul, Sanna
2. Sanna, Lesiba, Paul
3. Lesiba, Sanna, Paul
4. Paul, Sanna, Lesiba
5. Sanna, Paul, Lesiba

Answers: 3

Lesiba considers the emotional impact of questions. He understands that one

should not include questions with potential high emotional impact at the beginning
of the questionnaire. Sanna focuses on the fact that it is good to group items into
sections, dealing with particular topics. Paul understands the matter of sequencing
of items in terms of how much effort it takes to respond to the items. Paul’s
response is incorrect. The sequencing of questionnaire items is not done in terms
of the effort it takes to respond to the items. Although Sanna refers to a principle for
sequencing that is correct, she fails to see that there is a problem with the
sequencing of the LQ’s items. Lesiba offers the best response because he identifies
that fact that there is a problem with the sequencing of the LQ’s items, and he
indicates the problem correctly.

Question 9

You: Another aspect we should consider is whether the LQ incorporates different item
response formats. Can you identify items that demonstrate different response
formats?
Paul: Items 1.2 and 3.3 are questions with limited choice of answers whereas others,
such as 5.2, provide a checklist. I also noted that some items such as 2.4 and 2.6
require inversed responses
Lesiba: Items 2.2 and 2.7 are further examples of questions that require inversed
responses, and the semantic differential type of items contained in Section 2
clearly differ from checklist type items, such as 3.3
Joly: I identified another item in Section 2 that requires inversed responses, namely
2.10. Also keep in mind that the open question type such as 5.7 differs from the
multiple choice type items provided in Section 2.

You agree most with, second most with and least with:
1. Joly, Lesiba, Paul
2. Joly, Paul, Lesiba
3. Paul, Lesiba, Joly
4. Lesiba, Joly, Paul
5. Paul, Joly, Lesiba

Answer: Option 5

The item types indicated by Paul are correct. However, note that in the case of items
2.4 and 2.6 Paul does not refer the inversed scoring. He refers to items that require
inversed responses relative to each other. Joly’s response is partially correct but
contains at least one inaccuracy, namely the reference to the items in section 2 as
multiple choice questions. Lesiba offers the weakest response. It is not correct to
suggest that items 2.2 and 2.7 are inversed response items. It is also incorrect to
suggest that the items in section 2 are semantic differential items, and item 3.3 is
not a checklist.

Question 10

Lesiba: Does the LQ include any filter questions?

Sanna: Item 3.4 is an example of a filter question, but unfortunately it does not operate
correctly as a filter
Paul: Item 5.1 is an example of a filter question, and I think it operates correctly as a
filter
Joly: I agree that item 5.1 is an example of a filter question, but I do not think it operates
correctly as a filter
You agree most with, second most with and least with:
1. Paul, Sanna, Joly
2. Joly, Paul, Sanna
3. Joly, Sanna, Paul
4. Sanna, Paul, Joly
5. Paul, Joly, Sanna

Answer: Option 2

Question 5.1 is a filter question, but it does not function correctly because it
requires people who should answer the next question to skip the question.
Respondents who participate in extramural activities (those who say ‘yes’) are
required to skip question 5.2, which is a question that can best be answered by
people who participate in extramural activities. Therefore Joly’s answer is the best
response. Although Paul’s correctly identifies the question as a filter question, he is
wrong about the question functioning correctly. Sanna’s response is the weakest.
She does not understand what a filter question is.

Question 11

You: I identified some items that I think may be problematic, and I want to year what
you think about them. The first item I identified is 2.11
Lesiba: The item is relevant in terms of content, but I think there is a problem regarding its
construct relevance and of the clarity of its formulation
Sanna: I do not see a problem with the clarity of the item, but I think there is a problem
with its construct as well as its content relevance
Joly: I think the item is flawed in terms of content and construct relevance as well as
clarity.

You agree most with, second most with and least with:
1. Sanna, Joly, Lesiba
2. Joly, Lesiba, Sanna
3. Sanna, Lesiba, Joly
4. Lesiba, Joly, Sanna
5. Joly, Sanna, Lesiba

Answer: Option 1

Item 2.11 is about mechanical instinct. The item appears in a questionnaire that
intends to identify learners’ preferred behaviour styles that lead to emotional
burnout and under performance, and more particularly it is included in a section
that deals with behaviour that Influence people. There is no link between the item
and the content of the questionnaire. The item does not refer to preferred behaviour
styles. It merely states a basic fact namely that a person who have mechanical
instinct can fix a machine. Responding to the question requires knowledge about
the relationship between fixing a machine and mechanical instinct. However
knowledge of mechanical skills is not part of the content domain of the LQ. It also
does not contribute to determining somebody’s preferred behaviour style. Therefore
although the item is clearly stated it does not demonstrate content and construct
relevance.

In light of this information it is clear that Sanna provides the best response. Joly’s
response is second best because she identifies the problem of content and
construct relevance. However, she also considers the item as unclear, which is not
correct. Lesiba’s response is worst because his response is incorrect on two
accounts, namely content relevance and clarity.

Question 12

You: The second item I identified is 3.7

Paul: Although the item is pitched at the correct language level, I think there is a
problem with its clarity and I am also concerned that the item may require
prerequisite knowledge
Joly: The item seems clear to me but I think there is a problem with language level and
the fact that the item requires prerequisite knowledge
Sanna: I think clarity as the main problem. I do not see language level as a problem and
do not think the item requires prerequisite knowledge.

You agree most with, second most with and least with:

1. Joly, Paul, Sanna

2. Paul, Sanna, Joly
3. Sanna, Joly, Paul
4. Sanna, Paul, Joly
5. Paul, Joly, Sanna

Answer: Option 4

The language level of the item does not pose a problem, and it does not require
prerequisite knowledge. However it is clarity is a problem due the way in which is
formulate. Paul’s response is correct on two accounts. Joly’s response is incorrect
on all accounts.

Question 13

You: The third item I identified is 3.4

Sanna: The item is configured incorrectly, it requires prerequisite knowledge, and it may
have a social desirability bias
Lesiba: I do not see a problem with the item’s configuration, but it may elicit a socially
desirable response and it does require prerequisite knowledge
Paul: I do not think the item has a social desirability response bias, but it does require
prerequisite knowledge and there is a problem with the way in which it is
configured

You agree most with, second most with and least with:
1. Sanna, Lesiba, Paul
2. Sanna, Paul, Lesiba
3. Lesiba, Sanna, Paul
4. Paul, Lesiba, Sanna
5. Paul, Sanna, Lesiba

Answer: Option 5

Incorrect configuration is the biggest problem of Item 3.4. The obvious

configuration problem is the overlap caused by including 6 in alternative (c) and 6 in
alternative (d). A less obvious configuration problem is that the question should be
about inspiration (the name of the section in which the question is included), but
instead is phrased as a factual knowledge question, asking what the optimal size of
a good design team is. Thus the question requires pre-requisite knowledge.
However, the question does not have a social desirability bias. In other words,
respondents’ responses to the question do not make them look good or bad.

In light of the fact that Paul is correct on all accounts, Sanna is correct on two
accounts and Lesiba is correct on one account the correct sequence of responses
from best to worst is Paul, Sanna, Lesiba.

Question 14

You: The fourth item I identified is 4.8

Joly: Although the item has content and construct relevancy it is leading and likely to
elicit a socially desirable response
Lesiba: I agree that the item has content and construct relevancy and that it may elicit a
social desirable response, but it is not leading
Paul: I do not think the item is leading and I do not think it has a social desirability bias,
but I do think it does not have content and construct relevance

You agree most with, second most with and least with:
1. Paul, Lesiba, Joly
2. Lesiba, Joly, Paul
3. Joly, Paul, Lesiba
4. Joly, Lesiba, Paul
5. Lesiba, Paul, Joly

Answer: Option 4

Joly’s response is correct on all accounts. The item has content and construct
validity (concerns subjective norm for bright people like intellectual stimulation),
but is leading (prefer tasks that require stimulating ideas) and likely to elicit a
socially desirable response (society needs bright people). Lesiba’s response is
incorrect on one account, whereas Paul’s response is incorrect on all accounts.
Question 15

You: The fifth item I identified is 3.8

Sanna: I see two problems with this question, namely that it requires prerequisite
knowledge and that it is formulated as a closed question, whereas an open
question is required
Joly: I agree that the question requires prerequisite knowledge, but I do not agree that
this is a closed question.
Lesiba: I do not see any problem with the question. It does not require prerequisite
knowledge and it is not a closed question

You agree most with, second most with and least with:
1. Lesiba, Sanna, Joly
2. Joly, Sanna, Lesiba
3. Joly, Lesiba, Sanna
4. Sanna, Joly, Lesiba
5. Lesiba, Joly, Sanna

Answer: Option 5

Item 3.8 is an open question. It does not contain optional responses. It requires
respondents to provide their own information. The question does not require
prerequisite knowledge because it taps into stereotypes that people have about
inspirational people. In light of this Lesiba offers the best response. Joly and Sanna
agree that the question requires prerequisite knowledge, but they are wrong in
thinking this. They disagree about whether the question should be seen as open or
closed. Joly correctly indicate that it is not a closed question, whereas Sanna
incorrectly indicates that it is a closed question. Therefore Joly provides a better
response than Sanna.

Question 16

Joly: I would like to add 2.12 to the list of problematic questions.

Lesiba: I agree that this item is problematic – it is not configured correctly in terms of the
required response
Sanna: I agree that the item is not configured correctly, the basic problem being that this
question should be formulated as a semantic differential type question
Paul: No, I disagree. I think the item is configured correctly and I think it should be kept
in its current Likert scale form

You agree most with, second most with and least with:
1. Sanna, Lesiba, Paul
2. Paul, Lesiba, Sanna
3. Sanna, Paul, Lesiba
4. Lesiba, Sanna, Paul
5. Lesiba, Paul, Sanna

Answer: Option 1
The item is not configured correctly. It cannot be answered in the way it is
presented. It would be possible to respond to the item if it is was constructed in a
semantic differential form with ‘positive’ at the one end of the scale, and ‘negative’
at the other. Thus Sanna’s response is correct. Lesiba’s response is also correct,
but he fails to offer a suggestion as to how it could be fixed. Although Paul is
correct in indicating the item as a Likert type item he is incorrect in suggesting that
the item is configured correctly.

Question 17

Lesiba: I would like to add 5.6 to the list of problematic items.

Sanna: Why? I do not see anything wrong with this item
Paul: If you want to change this into a semantic differential type item I would advise
against it. I think it should be kept in its current Likert scale form
Joly: To my mind the problem with this item is that ‘importance of effects’ cannot be
judged without proper criterion reference

You agree most with, second most with and least with:

1. Joly, Paul, Sanna

2. Paul, Joly, Sanna
3. Joly, Sanna, Paul
4. Sanna, Paul, Joly
5. Sanna, Joly, Paul

Answer: Option 1

Item 5.6 requires prerequisite knowledge, specifically the criteria in terms of which
the importance of the effects needs to be rated. The question is in the form of a
rating scale. It is not formulated as a Likert type question, and it cannot be
expressed in a semantic differential form.
Joly offers the best response. Paul’s response contains correct and incorrect
information. The question cannot be rephrased as a semantic differential type item,
but Paul is incorrect in suggesting that the question is in the form of a Likert scale.
Sanna’s response is worst because it is incorrect. Also note that Sanna’s response
refers to the correctness/incorrectness of the item, but that it provides no additional
information in terms of which she justifies her conclusion.
(Note that the conversation is not initiated with a particular question. Lesiba simply
indicates that he finds Item 5.6 problematic, but he does not provide further
information. This leads to diverse responses. Joly focusses on the fact of
prerequisite knowledge and Paul focusses on the form in which the item is
presented as possible problems).
Question 18

Paul: The question I would like to add as a problematic item is 3.9

Joly: Yes, I see the problem. The question is not properly formulated as a true-false
type item
Sanna: I think the main problem is that the question may have a socially desirable
response bias
Lesiba: The real problem is that the item does not record respondents’ responses in a
consistent manner

You agree most with, second most with and least with:
1. Joly, Lesiba, Sanna
2. Sanna, Joly, Lesiba
3. Lesiba, Sanna, Joly
4. Lesiba, Joly, Sanna
5. Joly, Sanna, Lesiba

Answer: Option 4

The question does not contain a social desirability response bias. Sanna’s response
is incorrect. The matter to decide is whether Lesiba or Joly is more correct. Joly’s
response presupposes that the question should be a true/false question (this is an
assumption on her side. In other words, she assumes something that may or may
not have been the intention of the person who compiled the item). Lesiba does not
make any such assumption. He simply states that the options provided to the
respondent does not allow for consistency. In fact, it is not clear how ‘almost’ fits in
In fact, it is not clear how ‘almost’ fits in between ‘true’ and ‘false’. For example,
‘true’, ‘perhaps’ and ‘false’ would have been more consistent.

Question 19

You: I have another item I would like to add to our list of problematic questions, namely
5.3
Paul: The cannot see the contextual relevance of this question, and it does not
contribute to the purpose of the questionnaire
Lesiba: I agree, Paul, and I think the item would have served the purpose of the
questionnaire better if it was an open and not a closed question
Joly: I disagree. I think the question has high construct relevance and fits the purpose of
the questionnaire well

You agree most with, second most with and least with:
1. Lesiba, Paul, Joly
2. Paul, Lesiba, Joly
3. Lesiba, Joly, Paul
4. Joly, Lesiba, Paul
5. Paul, Joly, Lesiba

Answer: Option 2
Paul’s response is correct. The item has contextual relevance (it is the kind of
question that could be asked in the context of the learner’s world, but it does not
contribute to the purpose of the questionnaire because it does not belong the
content domain of the questionnaire. Lesiba is correct in so far as he agrees with
Joly, but he is incorrect in suggesting that a particular form of question would have
served the purpose of the questionnaire better. There is no reason why an open
question would have been better than a closed question for the purpose of the
current use of the LQ. Joly’s answer is incorrect on both accounts she offers.

Question 20

You: Let us have a look at the empirical information that we have for the LQ. This is
how the 30 participants in the pilot study responded to Item 2.3:

Strongly Agree Unsure Disagree Strongly

Agree Disagree
1 6 5 5 13

What do these responses tell us about the nature of the pilot group’s influencing
behaviour?
Joly: The distribution of the responses is skewed to the right, but this does not tell us
about the nature of the pilot group’s influencing behaviour
Lesiba: Generally speaking the pilot group was indecisive about the nature of their
influencing behaviour
Paul: The majority of the pilot group denied the nature of their influencing behaviour

You agree most with, second most with and least with:
1. Paul, Joly, Lesiba
2. Paul, Lesiba, Joly
3. Joly, Paul, Lesiba
4. Lesiba, Joly, Paul
5. Joly, Lesiba, Paul

Answer: Option 1

The table provides the response frequencies for item 2.3. In total 30 individuals
responded to the item. Of these 1 strongly agreed and 13 strongly disagreed.
Furthermore we can see that 5 individuals were unsure which leaves 7 individuals
on the agreement side and 18 on the disagreement side.
Based on this information we can see that Lesiba’s statement is incorrect. The
group clearly is not indecisive. Joly provides a better response. She is correct in
stating that the responses are skewed to the right (that is, towards strongly
disagree), but she is incorrect in stating that this does not provide information
about the group’s influencing behaviour. Paul offers the best opinion. More than
half of the group (18) disagreed and denied the nature of their influencing
behaviour.
Question 21

Joly: Table 1 in the manual shows the participants’ test scores for two halves of the LQ.
But I do not understand what it means.
Lesiba: Table 1 gives the total scores for two halves of the LQ. For example participant 1
obtained 27 on one half of the LQ and 23 on the other half of the LQ
Sanna: I agree, and it means the table shows the correlation between the two halves of
the LQ, and in this case the numbers reflect a high correlation
Paul: I disagree. The information in the table shows a low correlation between the odd
and even item items of the LQ

You agree most with, second most with and least with:
1. Sanna, Lesiba, Paul
2. Lesiba, Joly, Paul
3. Paul, Lesiba, Joly
4. Sanna, Paul, Lesiba
5. Lesiba, Paul, Joly

Answer: Option 1

Joly indicates that she does not understand what the information portrayed in Table
1 means, which leads to the information provided by Lesiba, Sana and Paul. The
information provided by Lesiba is correct. Sanna agrees with Lesiba, and then adds
further information, namely that the numbers show a high correlation between the
two halves of the LQ (because higher totals on one half tend to go with higher totals
on the other half, and lower totals tend to go with lower totals). The information
Sanna adds is also correct. Thus her response provides more information than
Lesiba’s, and because the information she provides is correct it is a better response
than Lesiba’s. Paul is incorrect when he suggest that the information provided in
Table 1 shows a low correlation between the two halves of the questionnaire.

Question 22

Joly: But does the information in Table 1 tell us anything about the reliability of the LQ?
Sanna: Yes, the information in Table 1 shows that the LQ has a high level of internal
consistency
Paul: I disagree. The information shows a low level of internal consistency
Lesiba: You are both wrong. The information in Table 1 indicates a high degree of
concurrent validity

You agree most with, second most with and least with:
1. Sanna, Lesiba, Paul
2. Paul, Sanna, Lesiba
3. Sanna, Paul, Lesiba
4. Lesiba, Paul, Sanna
5. Paul, Lesiba, Sanna

Answer: Option 3
The information provided in Table 1 concerns the totals of two halves of the
questionnaire. This is a matter of reliability, in particular internal consistency. Sanna
and Paul are correct in understanding this information as referring to internal
consistency. Lesiba is incorrect when he suggests this information is about validity.
What remains to be resolved is whether the information means a high or a low level
of internal consistency. This is a matter of correlation. Note that higher ‘even’ totals
tend to go with ‘higher ‘odd’ totals. This means a positive correlation between the
two halves of the test, which means high internal consistency. Therefore Sanna’s
response if more correct than Paul’s.

Question 23

Joly: I am still unsure about correlations. For example, in Table 2, how do I see whether
scores are correlated?
Lesiba: Draw a graph consisting of a horizontal X-axis and a vertical Y-axis and plot the
relationship between the ‘tested at arrival totals’ and the ‘tested after 3 days totals’
Paul: Scale the X-axis from 1 to 5, and the Y-axis from 43 to 77, and plot the ‘tested
after 4 days totals’
Sanna: Scale the X-axis from 43 to 77 and the Y-axis from 32 to 63 and plot the
corresponding values

You agree most with, second most with and least with:

1. Lesiba, Sanna, Paul

2. Sanna, Lesiba, Paul
3. Sanna, Paul, Lesiba
4. Paul, Lesiba, Sanna
5. Lesiba, Paul, Sanna

Answer: Option 2

Table 2 provides the information that allows one to depict the correlation between
‘tested at arrival’ and ‘tested after 4 days’. The discussants explain how depicts the
correlation in a graph. The information provided by Lesiba is correct on how depicts
the correlation in a graph, but it is tested after 4 days totals. Sanna’s answer is
better because she indicates (correctly) what numbers should be associated with
the X-axis and the Y-axis. In other words, she indicates how these axes should be
scaled. Although Paul also expands on Lesiba’s response, the information he
provides is incorrect. The numbers 1 to 5 refer the participants and not to their
scores. The correlation is between the scores of the participants and not between
the participants and their scores.

Question 24

You: So, now that you have plotted the information provided in Table 2, what do you
think the information tells us?
Joly: A weak correlation was obtained in the pilot study, which meant the LQ did not
demonstrate test-retest reliability
Sanna: The weak correlation obtained in the pilot study referred to criterion-related validity
and not to the reliability of the LQ
Paul: Actually the information in Table 2 does not reflect a weak correlation, which
means the LQ shows good criterion-related validity

You agree most with, second most with and least with:
1. Sanna, Joly, Paul
2. Sanna, Paul, Joly
3. Joly, Paul, Sanna
4. Paul, Sanna, Joly
5. Joly, Sanna, Paul

Answer: Option 5

To answer this question one must understand correlation and also know the
difference between test-retest reliability and criterion-related validity. The
information provided in Table 2 reflects test-retest reliability and not criterion-
related validity. Therefore Joly’s response is correct, whereas Sanna’s and Paul’s
are incorrect. Next one has to look at the correlation. Plotting each pair of numbers
[(43,63), (52,51), etcetera] shows a slight tendency of a downward sloping graph.
This may be difficult to see because there are only 5 graph points. But what is clear
is that there is no definite tendency of an upward sloping graph, which would have
shown a positive correlation. Therefore the correlation is weak and slightly
negative. This means Paul is incorrect. Apart from thinking that the information
refers to criterion-related validity (which is incorrect) Paul also thinks the numbers
reflect a good correlation (which is incorrect). Therefore his answer is worse than
the one provided by Sanna.

Question 25

You: So, if you compare the information from Table 1 with the information from Table 2,
what do you think?
Paul: The LQ has high test-retest reliability but its criterion related validity is
questionable
Bertha: The items of the LQ seem to measure the same thing, but the questionnaire is
vulnerable to external influences
Sanna: The information obtained from these two tables supports the reliability of the LQ

You agree most with, second most with and least with:
1. Paul, Sanna, Bertha
2. Sanna, Bertha, Paul
3. Bertha, Paul, Sanna
4. Bertha, Sanna, Paul
5. Sanna, Paul, Bertha

Answer: Option 4

Note that the information in Table 2 shows there is a tendency of higher values on
the first administration of the LQ to go with lower values on the second
administration. This means a low correlation or perhaps a negative correlation
between the two administrations of the questionnaire. Paul is incorrect because the
information does not support high test-retest reliability, and also these tables do not
provide any criterion related information. Sanna offers a better answer. Table 1
supports the reliability (internal consistency) of the questionnaire, but Table 2 does
not. Bertha’s gives the best response. The items measures the same thing (high
internal consistency), but the questionnaire is vulnerable to external influences
because the test-retest correlation is low. High test-retest correlation means
respondents’ responses are similar on the two administrations. But Table 2 shows
this not to be the case. Therefore their responses are influenced by something
(external to the questionnaire).