Journal of Interprofessional Care

ISSN: 1356-1820 (Print) 1469-9567 (Online) Journal homepage: http://www.tandfonline.com/loi/ijic20

Development and initial validation of the

interprofessional team learning profiling
questionnaire

Gillian Nisbet, Stewart Dunn, Michelle Lincoln & Joanne Shaw

To cite this article: Gillian Nisbet, Stewart Dunn, Michelle Lincoln & Joanne Shaw
(2016) Development and initial validation of the interprofessional team learning
profiling questionnaire, Journal of Interprofessional Care, 30:3, 278-287, DOI:
10.3109/13561820.2016.1141188

To link to this article: http://dx.doi.org/10.3109/13561820.2016.1141188

Published online: 06 May 2016.

Submit your article to this journal

Article views: 65

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=ijic20

Download by: [University of Nebraska, Lincoln] Date: 31 May 2016, At: 00:05
 JOURNAL OF INTERPROFESSIONAL CARE
2016, VOL. 30, NO. 3, 278–287
http://dx.doi.org/10.3109/13561820.2016.1141188

ORIGINAL ARTICLE

Development and initial validation of the interprofessional team learning profiling

questionnaire
Gillian Nisbeta, Stewart Dunnb, Michelle Lincolna, and Joanne Shawc
a
Faculty of Health Sciences, The University of Sydney, Lidcombe, New South Wales, Australia; bSydney Medical School, Royal North Shore Hospital,
The University of Sydney, Sydney, New South Wales, Australia; cPsycho-oncology Co-operative Research Group, School of Psychology, The
University of Sydney, Sydney, New South Wales, Australia

ABSTRACT ARTICLE HISTORY

Informal workplace interprofessional learning occurs as health professionals interact with each other as Received 13 October 2014
part of everyday work practice. Participation in interprofessional team meetings is a practical way to Revised 29 October 2015
foster learning. However, a gap exists in the availability of a reliable and valid instrument that Accepted 8 January 2016
adequately captures the nuances of informal workplace interprofessional learning in this setting. The
Downloaded by [University of Nebraska, Lincoln] at 00:05 31 May 2016

KEYWORDS
purpose of this study was to develop a questionnaire to measure the different components of inter- Collective learning;
professional learning that contribute to the quality of interprofessional learning within the interprofes- interprofessional learning;
sional team meeting. Questionnaire items were developed from a review of the literature and interviews questionnaire designs;
with health professionals. Exploratory factor analysis was used to determine the underlying factor statistics; work-based
structure. Two hundred and eighty-five health professionals completed a 98-item questionnaire. After learning
elimination of unreliable items, the remaining items (n = 41) loaded onto four factors named personal
and professional capacity; turning words into action—“walk the talk”; the rhetoric of interprofessional
learning—“talk the talk”; and inclusiveness. Internal consistency was high for all sub-scales (Cronbach’s
alpha 0.91, 0.87, 0.83, and 0.83, respectively). Content, construct, and concurrent validity were assessed.
The instrument developed in this study indicated consistency and robust psychometric properties.
Future studies that further test the psychometric properties of the questionnaire will help to establish
the usefulness of this measure in establishing evidence for the perceived effectiveness of interprofes-
sional learning in a healthcare setting.

Introduction workplace learning occurring between health professionals

within the workplace.
In today’s complex healthcare environment, considerable
A recent review by Nisbet et al. (2013) highlighted a
emphasis is placed on health professionals from different
number of potential benefits to the health system associated
professions working together to deliver high-quality care.
with informal workplace learning, including (i) innovation,
This often occurs in teams (Deneckere et al., 2013; Körner,
(ii) practice improvement, (iii) performance improvement,
Wirtz, Bengel, & Göritz, 2015). Less evident is the focus on
(iv) patient safety, (v) working together, and (vi) better patient
learning in this team context, particularly interprofessional
outcomes. However, the challenge for healthcare is how to
learning (IPL). Yet from the broader workplace and organiza-
shift from a task-orientated service provision focus to one that
tional learning literature, we know that learning is associated
also recognizes, values, and incorporates a learning frame of
with culture change (Hager, 2001), staff satisfaction (Jeong,
reference.
Lee, Kim, Lee, & Kim, 2007), and improved performance
Within the hospital setting, the interprofessional team
(Ellinger, Ellinger, Yang, & Howton, 2002; Kontoghiorghes,
(IPT) meeting is one context that provides opportunities for
Awbrey, & Feurig, 2005). This learning is not necessarily
informal workplace learning, particularly interprofessional
formal structured learning but rather the unstructured, infor-
learning (IPL). We use the term interprofessional team in
mal learning that occurs between staff through everyday work
this study. However, we recognize that in some countries
practice (Billett, 2006; Marsick & Volpe, 1999; Regehr &
the term multidisciplinary team is still used interchangeably
Mylopoulos, 2008), for example, problem solving during tea
regardless of the degree of interprofessionalism (e.g., Health
room interactions, corridor conversations to update a collea-
Workforce Australia, 2013; The National Academies of
gue, or informal debriefs about an unanticipated event. This
Sciences Engineers and Medicine, 2015). We also acknowl-
learning can occur within individuals, in groups, teams, and
edge that within health, the terms team and teamwork are
across an organization and forms an important part of the
commonly used but may not reflect literature definitions (e.g.,
professional development of workers (Marsick & Watkins,
Alexanian, Kitto, Rak, & Reeves, 2015; Lewin & Reeves, 2011).
2001). Hence this study focuses on capturing the informal

CONTACT Gillian Nisbet [email protected] The University of Sydney, Faculty of Health Sciences, Building J, Cumberland Campus, 75 East Street,
Lidcombe, NSW 2141, Australia.
Colour versions of one or more figures in the article can be found online at http://www.tandfonline.com/ijic.
© 2016 Taylor & Francis
 JOURNAL OF INTERPROFESSIONAL CARE 279

Salas et al. (2005) define a team as “. . . two or more indivi- concurrent validity to identify the perceived IPL strengths
duals with specified roles interacting adaptively, interdepen- and weaknesses within interprofessional teams.
dently, and dynamically toward a common and valued goal”
(pp. 559, 562). They emphasise the importance of team mem-
bers being able to “. . . coordinate and cooperatively interact Methods
with each other” as they work towards their task objectives
through a shared understanding of each other’s skills, knowl- Overview of team IPL profiling questionnaire
edge, and experiences. While it is recognised that some We developed the Team IPL Profiling questionnaire to cap-
healthcare teams might operate as per this definition, it is ture attitudes, perceptions, and behaviours relating to IPL
also recognised that others do not, despite being called within the IPT meeting. The questionnaire content was devel-
“teams”. This reflects what others have referred to as “pseudo oped from the IPL and health literature and from themes
teams”: teams that are in name only and do not meet the established in our previous qualitative research (Nisbet et al.,
necessary requirements of a team (Dawson, Yan, & West, 2015). Health professionals completed a 98-item online ques-
2007; West & Lyubovnikova, 2012). This article uses the tionnaire. Participants were asked to indicate their level of
terms team and teamwork to reflect current workplace termi- agreement with each statement on a 5-point Likert scale
nology, and suggests that the above definition by Salas et al. (strongly agree to strongly disagree). A “don’t know/not
(2005) represents where healthcare teams aspire to be. applicable” choice was also offered. Demographic character-
Downloaded by [University of Nebraska, Lincoln] at 00:05 31 May 2016

Although the overall purpose of the IPT meeting is to istics (profession, years of experience; years working within
develop a comprehensive, coordinated patient care plan the IPT; frequency of attendance at the IPT meeting; number
(Lanceley, Savage, Menon, & Jacobs, 2008; Sidhom & of new members joining the IPT in last 6 months; and main
Poulsen, 2006), the regularity of team meetings and the inter- contribution to the IPT meeting) were also collected.
actions that occur provide potential opportunities for knowl-
edge, ideas, and perspectives to be shared between team
members (Nisbet, Dunn, & Lincoln, 2015). However, teams Participants
require practical strategies to capitalize on this learning
opportunity. One strategy is to profile a team or organiza- Recruitment
tion’s learning characteristics, then implement specific We followed a purposeful sampling approach (Cohen,
approaches to address learning deficits. There are many gen- Manion, & Morrison, 2005) to recruit healthcare professionals
eric instruments available in the literature. However, the cur- from seven metropolitan hospitals in Sydney, Australia.
rent instruments are unsuitable for use in the Recruitment occurred during team meetings. One researcher
interprofessional health setting as they are either geared (GN) attended each IPT that met study inclusion criteria
towards the individual rather than the team (Tannenbaum, (Table 1) to explain the study. Participation was voluntary.
1997), are too general (Moilanen, 2005), and/or fail to capture Interested participants provided written consent and were
the interprofessional influences on learning (Marsick & emailed a link to the online survey (SurveyMonkeyTM
Watkins, 2003) such as hierarchy, power imbalances, status, [Computer software]).
and professional confidence (Nisbet, 2013).
Several instruments have been proposed for the healthcare Sample
context (for examples see Bess, Perkins, & McCown, 2011; Two hundred and eighty-five health professionals from 62
Henderson, Creedy, Boorman, Cooke, & Walker, 2010; IPTs participated in the survey (59% response rate). Table 2
Rushmer et al., 2006). However, these instruments do not presents characteristics of the sample population. Figure 1
adequately address IPL issues (Nisbet, 2013). Additionally, presents the percentage of participants from each clinical
two commonly used instruments within the IPL literature— specialty included in the study.
the Readiness for Interprofessional Learning Scale (RIPLS) The following section describes the approaches used to
(Parsell & Bligh, 1999; Reid, Bruce, Allstaff, & McLernon, develop and validate the Team IPL Profiling questionnaire.
2006) and the Interdisciplinary Education Perception Scale
(IEPS) (Luecht, Madsen, Taugher, & Petterson, 1990;
McFadyen et al., 2005)—also have limited applicability in
Item development and content validity
informal workplace learning. The RIPLS focuses on “shared
learning” rather than the “learning from” aspect of IPL and Questionnaire items were developed based on thematic ana-
factors influencing this and the IEPS focuses on individuals’ lysis of qualitative interviews with health professionals (Nisbet
professional competency, co-operation, and degree to which et al., 2015). We identified conceptions of learning, IPT meet-
this occurs. Such instruments do not address complexities of ing as a source of knowledge, learning through participation,
workplace informal IPL. and medical influence on learning as important concepts
The growing recognition within health of the importance relevant to IPL. A draft version of the items identified was
of workplace and organizational learning necessitates a psy- reviewed by the qualitative study participants to confirm their
chometrically reliable measure of IPL. The current instru- relevance and clarity and also by a further five health profes-
ments do not adequately capture the nuances of informal sionals to gain independent interpretation of each item.
workplace IPL. Therefore, the purpose of this study was to Finally, we compared items developed with those in existing
develop an instrument with content, construct, and questionnaires to identify any gaps in content.
 280 G. NISBET ET AL.

Table 1. Criteria for inclusion of interprofessional team meeting in study. Table 2. Characteristics of sample population.
Criteria for inclusion of interprofessional team meeting in study Number of participants 285
Gender
● The IPT meeting focuses on primarily one clinical specialty (e.g., Female 216 (75.8%)
cardiothoracic) Male 69 (24.2%)
● The specialty identifies itself as an interprofessional healthcare team and Number of hospitals 7
includes health professionals from medicine, nursing, and at least one other Number of IPT meetings 62
profession from either dietetics, occupational therapy, pharmacy, phy- Average number of team members per IPT 12 (range: 5–28)
siotherapy, psychology, speech pathology, or social work meeting at time of recruitment
● A formal timetabled IPT meeting is held at least on a monthly basis—patient Average number of consents per IPT meeting 7 (range: 1–20)
list or agenda; meeting chaired Number of clinical specialties represented by IPT meetings 23
● The purpose of IPT meeting is to provide a forum to discuss patient Percentage breakdown of team focus:
Medical 56.5%
diagnosis, assessment, and/or management
Surgical 26.1%
● Attendance at the IPT meeting is mandatory (not optional) Rehabilitation 4.3%
● The IPT meeting is attended by representatives from medicine, nursing, and Acute and rehabilitation combined 13%
at least one other profession from either dietetics, occupational therapy, Participant breakdown by team focus
pharmacy, physiotherapy, psychology, speech pathology, or social work Medical 171 (60.2%)
● The meeting is held at the hospital site Surgical 58 (20.4%)
● The patient does not attend the meeting Rehabilitation 39 (13.7%)
● The meeting may be either an inpatient-based IPT meeting or ambulatory Medical/rehabilitation combined 16 (5.6%)
care/outpatient-based IPT meeting Number of professions/occupations represented 16
Distribution by professional grouping
Downloaded by [University of Nebraska, Lincoln] at 00:05 31 May 2016

Allied health 45.0%

Nursing 27.7%
Medicine 25.3%
Construct validity and factor analysis Other 1.8%
Ratio of allied health:nursing:medicine 1:0.6:0.6
Time working in current profession
Construct validity was established through exploratory factor Over 10 years 50.5%
analysis as outlined below. 5–10 years 19.5%
2–5 years 15.5%
6 months–2 years 10.8%
1–6 months 3.2%
Item reduction Less than 1 month .4%
We excluded participants who failed to complete the ques- Length of time a member of this IPT
Over 10 years 15.0%
tionnaire as well as those with 5% or more “don’t know/not 5–10 years 21.6%
applicable” responses and/or missing data. Items were also 2–5 years 23.8%
excluded if they had poor discrimination (defined as >90% 6 months–2 years 19.0%
1–6 months 17.9%
responses falling on 1–2 points of the 5-point Likert scale) Less than 1 month 2.6%
or >5% “don’t know/not applicable” responses and/ or Frequency of attendance at IPT meeting in past 6 months
80–100% of time 87.0%
missing data (suggesting poorly worded or irrelevant 60–79% of time 5.3%
items) or poor between item correlation (>10% Spearman 40–59% of time 2.8%
rho less than 0.3). 20–39% of time 2.5%
Less than 20% of time 2.5%
Principal component analysis (PCA) with varimax rotation New team members in past 6 months
was used to extract factors. We used a hybrid approach to None 2.8%
1–3 people 42.1%
determine the number of factors to extract. For interpretation 4–6 people 26.3%
of the rotated structure, an item was said to load onto a factor 7–9 people 9.1%
More than 9 people 10.9%
if the factor loading was greater than 0.40 for a factor, and was Don’t know 8.8%
less than 0.40 for the other factors. Main contribution to IPT meeting
Provide profession-specific expertise 82.5%
Advocate for patient 70.9%
Raise alternative points of view 49.8%
Facilitate discussion 45.3%
Factor interpretation and reliability Provide leadership 26.7%
Teach others 22.8%
Factors were used to develop sub-scales within the ques- Not sure .4%
Other 7.7%
tionnaire. Items that loaded onto each factor were
summed to provide sub-scale scores. Negatively worded
items were reverse scored. Reliability and internal consis-
tency was confirmed using Cronbach’s alpha for both the Global item correlations
total score and sub-scale scores. Cronbach’s alpha was
calculated with each item removed in turn. If reliability The questionnaire included three global items to further estab-
was improved by exclusion of an item, this item was lish construct validity: two related to overall perceptions of the
deleted from the scale. extent to which the IPT meeting had the necessary elements to
Mean sub-scale scores and an overall score for the Team encourage learning between health professionals and were the
IPL Profiling questionnaire were calculated and compared reverse of each other (our IPT meeting has all the necessary
with the maximum possible score. We also calculated the elements to encourage learning between health professionals;
proportion of respondents with the highest (ceiling) and low- our IPT meeting lacks the necessary elements to encourage
est (floor) scores for each factor and overall score. learning between health professionals) and one item on overall
 JOURNAL OF INTERPROFESSIONAL CARE 281
Downloaded by [University of Nebraska, Lincoln] at 00:05 31 May 2016

Figure 1. Percentage of participants from each clinical specialty.

perceptions of learning from others within the IPT meeting singular meaning. No further changes to items were made.
(overall, I learn important information from other health profes- This resulted in a total of 98 items being included in the Team
sions by attending this IPT meeting). Respondents were asked to IPL Profiling questionnaire distributed to study participants.
indicate their level of agreement with each of the statements,
using a 5-point Likert scale. Spearman’s rank correlation coeffi-
Construct validity: Factor analysis
cients were calculated for the global items and overall Team IPL
Profiling questionnaire and sub-scale scores. Initial inspection of items for item reduction
Initial inspection of data resulted in the removal of 36 items—
13 with poor discrimination, 16 with poor correlations with
Concurrent validity other items, and 7 with greater than 5% missing data/“don’t
We used two domains of the Safety Attitudes Questionnaire know” responses.
(SAQ) (Sexton et al., 2006)—Teamwork Climate and Safety
Climate—to assess concurrent validity of the Team IPL Establishing the factor solution
Profiling questionnaire. The SAQ was derived from work within The underlying constructs represented by the retained items
the aviation industry and has been validated extensively within were identified using exploratory factor analysis. Varimax
health as a measure of attitudes towards patient safety issues rotation confirmed a four-factor solution which accounted
(e.g., Makary et al., 2006; Nordén-Hägg, Sexton, Kälvemark- for 47% of the variance. Twenty-one items failed to load
Sporrong, Ring, & Kettis-Lindblad, 2010; Relihan, Glynn, Daly, above 0.4 on any factor or cross-loaded. Table 3 provides
Silke, & Ryder, 2009; Watts, Percarpio, West, & Mills, 2010). The the factor loadings.
Teamwork Climate domain has six items, and the Safety Climate
domain has seven items. Items were rated on a 5-point Likert Factor interpretation and reliability
scale (disagree strongly to agree strongly). Scores were computed Factor 1 consisted of 15 items related to perceptions of capa-
for each domain and correlation with the Team IPL Profiling city within the team meeting to contribute to others’ learning.
questionnaire overall score determined. All statistical analyses This factor included both: (i) personal capacity to contribute,
were undertaken using SPSS statistical software. for example, level of self confidence; and (ii) professional
capacity to contribute, for example, the perceived added
value to the meeting of one’s professional knowledge. Based
Ethical considerations on the factor structure, this sub-scale was labelled Personal
The study was approved by Northern Sydney Central Coast and Professional Identity.
Human Ethics Committee and individual hospital human Factor 2 consisted of 11 items related to attitudes and
research ethics committees. perceptions of behaviours associated with implementing prin-
ciples of IPL. This factor focuses on the “how” of IPL, the
rules of engagement, highlighting that it is not enough to
Results agree in principle to the value of IPL—teams must actually
turn words into action, “walk the talk”. This sub-scale was
Content validity: Item development
labelled Turning words into action—“Walk the talk”.
After an initial review for clarity and relevance, 11 items were Factor 3 consisted of nine items related to perceptions of a
reworded and 5 items were divided into two items to ensure team’s support for the philosophical underpinnings and
 282 G. NISBET ET AL.

Table 3. Factor structure and loadings.

Factor loading
Factor Factor Factor Factor
Item 1 2 3 4
In this IPT meeting I often am reluctant to speak up for fear of sounding dumb .798 .048 .048 .124
Sometimes I am not sure what other information or knowledge is relevant to share in this IPT meeting .789 .108 −.061 .048
Sometimes I am not sure what patient information is relevant to share in this IPT meeting .742 .128 −.034 .070
I am not sure what information other staff really want to hear from me at this IPT meeting .661 .197 .110 .086
I don’t feel I contribute new patient information or knowledge to this IPT meeting .654 .032 .193 .240
I don’t feel I contribute new professional or academic knowledge to this IPT meeting .644 .181 .194 .210
It is stressful to speak up in this IPT meeting .616 .299 .120 .227
In this IPT meeting I have a say in patient care decisions .570 .302 .120 .117
I feel my presence at this IPT meeting adds value to the meeting .551 .130 .356 .200
In this IPT meeting I would not feel comfortable raising an issue that I had only a ‘gut feel’ about .535 .208 −.050 .372
In this IPT meeting I would not have any new professional or academic knowledge worth sharing with others .514 .022 .272 .070
In this IPT meeting I feel comfortable to bring up patient care information or issues outside my immediate area of professional .481 .357 .216 .215
expertise
In this IPT meeting I am uncomfortable asking questions when there is something I don’t understand .465 .158 .008 .211
In this IPT meeting it is acceptable to question the doctor about why particular decisions were made .425 .332 .226 .196
In this IPT meeting my unique skills and talents are utilised .422 .355 .229 .183
In this IPT meeting differing views are actively encouraged and sought during discussion .218 .753 .206 −.015
In this IPT meeting it is acceptable to question other professionals on their patient care plans and actions .192 .669 .117 .037
Downloaded by [University of Nebraska, Lincoln] at 00:05 31 May 2016

In this IPT meeting disagreements and differing views on patient care are ignored or avoided .210 .656 .191 .249
The culture in this IPT meeting makes it easy to learn from the errors of others .250 .644 .204 −.070
In this IPT meeting team members give open and honest feedback to each other on the care delivered to patients .108 .627 .138 .245
Members of this IPT meeting are able to bring up tough team issues and problems .060 .603 .150 −.035
In this IPT meeting, team members are encouraged to bring the patient’s views into the decision making process .131 .583 .236 .274
In this IPT meeting learning is valued more than just getting through the patient list .105 .521 .224 .063
In this IPT meeting we make time to regularly reflect on how the team meeting is functioning .107 .505 .138 .079
Our IPT meeting has all the necessary elements to encourage learning between health professionals .203 .499 .357 .165
In this IPT meeting psychosocial aspects of care are adequately discussed .285 .450 .059 .388
This IPT meeting is essential for learning from other health professionals .029 .269 .728 .020
This IPT meeting is essential for team building .152 .180 .698 .060
The IPT meeting is a venue to learn about other health professionals’ clinical practice and wider aspects of their professional .095 .245 .679 .027
and academic knowledge
This IPT meeting is essential for bringing the patient’s views into the decision making process .051 .142 .629 .117
Learning from other health professions is not relevant to this IPT meeting .228 .255 .598 .108
Hearing from other health professionals at this IPT meeting increases my understanding of their scope of practice −.082 .332 .580 .170
This IPT meeting is essential to reach consensus on the patient’s management plan .047 .052 .463 .390
Learning about my patients is all that I learn from attending a IPT meeting .394 .049 .435 −.043
I can gain information more efficiently and effectively through other means .263 .286 .429 .054
In this IPT meeting conversations are mainly between the doctors .274 .137 .071 .769
In this IPT meeting most of the talking is done by the doctors .244 .038 .028 .734
In making patient care decisions I mainly consult with colleagues from my own profession .142 −.113 .127 .602
Compared with treating disease, discussion about the psychosocial issues are not allocated enough time in this IPT meeting .244 .313 .077 .574
In this IPT meeting you have to ‘push’ your way in to the discussion .364 .371 .040 .528
This IPT meeting achieves whole team discussion and whole team input .273 .396 .259 .412
Cronbach’s alpha .91 .87 .83 .83
% variance explained 30.6 7.8 4.6 4.2

application of IPL to the team meeting. Items described learn- IPL (Factor 3) and weakest in perceptions of applying these
ing from other health professionals; the centrality of the principles within their teams (Factor 2). There was no evi-
patient to IPL; the scope of IPL potential; and the focus on dence of ceiling or floor effects (Table 4).
team relationships. This sub-scale was labelled The rhetoric of
interprofessional learning—“Talk the talk”.
Factor 4 consisted of six items related to perceptions of Global item correlations
inclusiveness, including the influence of doctors on com- There was a moderate positive correlation between the overall
munication flow and interactions within the team meeting Team IPL Profiling questionnaire score and global item 1
as well as the content discussed. Based on the factor (Our IPT meeting lacks the necessary elements to encourage
structure, this sub-scale has been named Inclusiveness. learning between health professionals) (r = 0.63) and between
Cronbach’s alpha reliability coefficients for each sub-scale overall Team IPL Profiling questionnaire score and global
indicated high internal consistency (Table 3). Exclusion of item 2 (Our IPT meeting has all the necessary elements to
individual items did not increase internal consistency. The encourage learning between health professionals) (r = 0.57).
final Team IPL Profiling questionnaire contained 41 items There was a significant but weaker positive correlation
across four sub-scales (Appendix 1). between the overall Team IPL Profiling questionnaire score
The maximum possible score for sub-scales ranged from 30 and global item 3 (Overall, I learn important information
to 75. Mean sub-scale scores for this sample, expressed as a from other health professions by attending this IPT meeting)
percentage, were between 70% and 79% of maximum possible (r = 0.46). Correlations between individual sub-scale scores
score (Table 4). Participants in this study were strongest in and each of the global item scores were fair to moderate
their level of agreement with the importance and principles of (range: r = 0.33 to r = 0.64).
 JOURNAL OF INTERPROFESSIONAL CARE 283

Table 4. Average factor scores as a percentage of maximum score possible. IPL. This domain encompasses key learning processes recog-
Factor 1: Factor 2: nized within the broader workplace learning literature, for
Personal Turning example, the role of questioning, giving feedback, and reflec-
and words Factor 3:
professional into Rhetoric Factor 4: Overall tion. However, this questionnaire provides a more nuanced
capacity action of IPL Inclusiveness IPL score* assessment of behaviours specific to the real-world interpro-
Mean score (SD) 58 (8.6) 39 (6.4) 36 (4.8) 23 (4.5) 155 (19.6) fessional healthcare environment, such as challenging the
% maximum 78 70 79 75 74
score
practice of a team member from another profession, valuing
possible all contributions regardless of profession, and fostering parti-
“Floor”—% 0 0 0 0 0 cipation to enhance IPL. Teams who score high on this
with lowest
possible
domain exhibit these behaviours.
score The third sub-scale, the rhetoric of interprofessional learn-
“Ceiling”—% 1.8 .7 1 4 0 ing—“talk the talk”, assesses team members’ beliefs about and
with highest
possible support for IPL and its relevance to the team meeting context.
score Teams who score high on this domain hold the belief that IPL
*Overall Team IPL Profiling questionnaire score. should be included as a fundamental philosophy of IPT
meetings.
The final sub-scale, inclusiveness, assesses the extent to
Downloaded by [University of Nebraska, Lincoln] at 00:05 31 May 2016

One of the global items (Our IPT meeting has all the which the team meeting captures the unique skill sets and
necessary elements to encourage learning between health pro- knowledge within the team as a means of facilitating IPL.
fessionals) loaded onto factor 2 (Table 2) so was considered High scores are indicative of greater sense of inclusiveness.
relevant to sub-scale 2. Teams with high inclusiveness scores are more likely to follow
a bio-psychosocial model of patient care. This domain also
assesses the implicit tensions between medicine and other
Concurrent validity
professions—the “elephant in the room”. Such tensions, iden-
There was a strong positive correlation between overall IPL tified previously (Nisbet et al., 2015), have a strong influence
Profiling questionnaire score and teamwork climate score (r = on IPL in this IPT meeting context.
0.69; n = 239). There was also a moderate positive correlation This questionnaire provides a first step in the development
between overall IPL Profiling questionnaire score and safety of IPL with healthcare teams. Further validation of the ques-
climate score (r = 0.61; n = 238). tionnaire and refinement of individual items in a second
sample is necessary to ensure a psychometrically robust mea-
sure. Despite this ongoing work, the moderate correlations
Discussion
between the questionnaire and the global aspects of IPL sug-
Within the IPL literature there is a pressing need for reliable gest the questionnaire is consistent with theoretical
and valid methods for evaluating intervention-based studies. expectations.
The Team IPL Profiling questionnaire was developed to fill a We have included the global item “Our IPT meeting has all
gap in the availability of suitable instruments to profile health- the necessary elements to encourage learning between health
care teams’ IPL capacity. The questionnaire addresses both professionals” in the final questionnaire as this item loaded
individual and team learning aspects of IPL, highlighting the onto sub-scale 2 and therefore appears relevant to this sub-
interdependency between the two and focuses on a significant scale. Because the other two global items did not load onto the
component of health professional practice—the IPT meeting. factor analysis and were not considered to add any additional
The results of this study provide evidence for the initial information, we decided to delete these items from the final
validation and reliability of the questionnaire. The four sub- questionnaire. Subsequent versions of the questionnaire may
scales identified confirm the complexity of IPL in the context further refine items included.
of IPTs. The questionnaire will provide a starting point for The Team IPL Profiling questionnaire was developed for
assisting IPTs become successful interprofessional learning teams to identify their IPL strengths and weaknesses in the
teams. context of IPT meetings. Initial completion can serve as a
The personal and professional capacity domain within the diagnostic process for teams, thus laying the foundations for
questionnaire highlights the interconnection between personal creating and fostering an IPL culture—one where learning
ability to contribute and professional role. Teams that score from others is part of everyday practice and part of the culture
high on this domain have team members with a clear under- of the team. Subsequent regular completion of the question-
standing of their own professional role in contributing to IPL naire, for example annually, could provide a means of mon-
within the team and have the personal confidence and ability itoring progress in achieving this change in culture. The team
to fulfil that role. They know when, what, and how to con- discussion that arises from results is also an important step in
tribute within the meeting to enhance IPL. Identification of creating an IPL culture. By identifying a team’s strengths and
teams with low scores is crucial given the potential legal weaknesses, interventions specific to that team’s needs can be
implications associated with accountability in teams and fail- developed and implemented. For example, if scores are low
ure to act when required (Sidhom & Poulsen, 2006). for sub-scale 1, establishing “ground rules” for contributing
The second sub-scale, Turning words into action—“walk may be necessary. If scores are low for sub-scale 2, including a
the talk” relates to mechanisms that support and promote process for regular team reflection may be useful. If scores are
 284 G. NISBET ET AL.

low for sub-scale 3, emphasizing to all new team members Conclusion

that IPL is part of the IPT meeting ethos is critical. If scores
This study reported on the development and initial validation
are low for sub-scale 4, establishing leadership training and
of the team IPL profiling questionnaire. The psychometric
meeting facilitation training for all IPT meeting chairs may
properties of the questionnaire suggest a robust instrument
address the issues.
for teams to efficiently and reliably monitor their IPL
Within the IPL literature there is a need for reliable and
strengths and weaknesses. The four domains of the question-
valid methods for evaluating intervention-based studies. The
naire capture key constructs required for IPTs to become
Team IPL Profiling questionnaire provides an ideal opportu-
successful interprofessional learning teams.
nity to collect longitudinal empirical data to track team pro-
While this study has provided initial validation of the
gress. Ultimately however, researchers are interested in the
Team IPL Profiling questionnaire, further testing of its psy-
impact of IPL. Investigation of correlations between scores
chometric properties needs to be performed when applying
from the Team IPL Profiling questionnaire and relevant out-
the instrument in further research. Future testing and analysis
come measures might provide additional evidence for the
at a team level will further strengthen our understanding of
effectiveness or otherwise of IPL. It is feasible that improve-
informal workplace IPL. Future analysis at a team level will
ments in IPL within the IPT meeting context can influence
enable comparisons to be made across teams (i) within the
workplace IPL more broadly, for example, in continuing pro-
same specialty, (ii) at the same institution, and (iii) from
fessional development in-service or journal club forums. This
Downloaded by [University of Nebraska, Lincoln] at 00:05 31 May 2016

different clinical specialties, potentially enabling more tar-

requires further investigation.
geted improvement strategies.
Although this research was conducted with qualified health
As healthcare continues to increase in complexity, the
practitioners, findings have implications for student health
centrality of learning in this process is becoming ever more
professional training. First, by creating an IPL culture within
apparent. The Team IPL Profiling questionnaire provides a
the IPT meeting, it role models the attitudes and behaviours
practical means for healthcare teams to embrace informal
we wish to develop within our next generation of health
workplace learning, particularly in relation to health profes-
professionals. Second, an environment that encourages IPL
sionals learning with, from, and about each other, as part of
between health professionals is likely to also actively include
their everyday work practice. We encourage researchers that
students in the meeting process. This warrants further inves-
further testing of psychometrics need to be performed when
tigation. Third, the Team IPL Profiling questionnaire, while
applying the instrument in further research.
developed for health professionals, may also be suitable for
student interprofessional teams. However, further validation
of the questionnaire in this setting would be required.
In terms of study limitations, as this work is the first Declaration of interest
attempt to develop a reliable and valid instrument that ade- The authors report no conflicts of interest. The authors alone are
quately captures the nuances of informal workplace IPL we responsible for the content and writing of this article.
intentionally approached questionnaire development with a
large number (98) of items (Streiner & Norman, 2008).
Despite the number of items, questionnaire length did not References
appear to negatively affect completion rates (96% completion Alexanian, J. A., Kitto, S., Rak, K. J., & Reeves, S. (2015). Beyond the
rate), although may account for the 59% response rate. The team: Understanding interprofessional work in two North American
reduction in number to 41 items is better suited to the busy ICUs. Critical Care Medicine, 43(9), 1880–1886.
clinical environment. Although we had a large sample size (n Bess, K. D., Perkins, D. D., & McCown, D. L. (2011). Testing a measure
= 285), the large number of items meant the subject to vari- of organizational learning capacity and readiness for transformational
change in human services. Journal of Prevention & Intervention in the
able ratio (4.6:1) was slightly less than generally recommended Community, 39(1), 35–49.
(5–10:1) (Field, 2009). This might have impacted the stability Billett, S. (2006). Constituting the workplace curriculum. Journal of
of the factor solution. Future studies are therefore required to Curriculum Studies, 38(1), 31–48.
confirm the robustness of the factor structure across Cohen, L., Manion, L., & Morrison, K. (2005). Research methods in
populations. education (5th ed.). London, UK: Routledge Falmer.
Dawson, J., Yan, X., & West, M. (2007). Positive and negative effects of
The questionnaire developed for this study elicited percep- team working in healthcare: Real and pseudo-teams and their impact
tions of IPL within meetings. As highlighted by Alexanian on safety: Working paper. Birmingham, UK: Aston University.
et al. (2015), what is reported by participants can differ from Deneckere, S., Euwema, M., Lodewijckx, C., Panella, M., Mutsvari, T.,
what is observed in practice. This needs to be taken into Sermeus, W., & Vanhaecht, K. (2013). Better interprofessional team-
account when using the Team IPL Profiling questionnaire. work, higher level of organized care, and lower risk of burnout in
acute health care teams using care pathways: A cluster randomized
However, the self-completion of the questionnaire by team controlled trial. Medical Care, 51(1), 99–107.
members is likely to encourage involvement and ownership Ellinger, A. D., Ellinger, A. E., Yang, B., & Howton, S. W. (2002). The
by the team in the learning process. Furthermore, the ques- relationship between the learning organization concept and firms’
tionnaire is designed as a starting point for further discussion financial performance: An empirical assessment. Human Resource
of changes and actions required to improve IPL capacity Development Quarterly, 13(1), 5–21.
Field, A. (2009). Discovering statistics using SPSS (3rd ed.). London, UK:
within the team. SAGE Publications.
Hager, P. (2001). Construction industry case study In G. Meagher, S.
Ryan, J. Buchanan, G. Considine, P. Hager, & J. Kitay (Eds.),
 JOURNAL OF INTERPROFESSIONAL CARE 285

Changing Nature of Work: Industry Case Studies (pp. 27–39). Sydney, Nisbet, G., Dunn, S., & Lincoln, M. (2015). Interprofessional team meet-
Australia: NSW Board of Vocational Education and Training. ings: Opportunities for informal interprofessional learning. Journal of
Health Workforce Australia. (2013). National Clinical Supervision Interprofessional Care, 29, 426–432.
Competency Resource Validation Edition Retrieved from https://hwa. Nisbet, G., Lincoln, M., & Dunn, S. (2013). Informal interprofessional
gov.au/sites/uploads/HWA-National-Clinical-Supervision- learning: An untapped opportunity for learning and change within the
Competency-Resource-VE-201305.pdf workplace. Journal of Interprofessional Care 27, 469–475.
Henderson, A., Creedy, D., Boorman, R., Cooke, M., & Walker, R. Nordén-Hägg, A., Sexton, J. B., Kälvemark-Sporrong, S., Ring, L., &
(2010). Development and psychometric testing of the Clinical Kettis-Lindblad, Å. (2010). Assessing safety culture in pharmacies:
Learning Organisational Culture Survey (CLOCS). Nurse Education The psychometric validation of the Safety Attitudes Questionnaire
Today, 30(7), 598–602. (SAQ) in a national sample of community pharmacies in Sweden.
Jeong, S. H., Lee, T., Kim, I. S., Lee, M. H., & Kim, M. J. (2007). The BMC Pharmacology and Toxicology, 10(1), 8.
effect of nurses’ use of the principles of learning organization on Parsell, P., & Bligh, J. (1999). The development of a questionnaire to
organizational effectiveness. Journal of Advanced Nursing, 58(1), assess the readiness of health care students for interprofessional learn-
53–62. ing (RIPLS). Medical Education, 33, 95–100.
Kontoghiorghes, C., Awbrey, S., & Feurig, P. L. (2005). Examining the Regehr, G., & Mylopoulos, M. (2008). Maintaining competence in the
relationship between learning organization characteristics and change field: Learning about practice, through practice, in practice. Journal of
adaptation, innovation, and organizational performance. Human Continuing Education in the Health Professions, 28(S1), S19–S23.
Resource Development Quarterly, 16(2), 185–211. Reid, R., Bruce, D., Allstaff, K., & McLernon, D. (2006). Validating the
Körner, M., Wirtz, M. A., Bengel, J., & Göritz, A. S. (2015). Relationship Readiness for Interprofessional Learning Scale (RIPLS) in the post-
of organizational culture, teamwork and job satisfaction in interpro- graduate context: Are health care professionals ready for IPL? Medical
Downloaded by [University of Nebraska, Lincoln] at 00:05 31 May 2016

fessional teams. BMC Health Services Research, 15(1), 243. Education, 40(5), 415–422.
Lanceley, A., Savage, J., Menon, U., & Jacobs, I. (2008). Influences on Relihan, E., Glynn, S., Daly, D., Silke, B., & Ryder, S. (2009). Measuring
multidisciplinary team decision-making. International Journal of and benchmarking safety culture: Application of the safety attitudes
Gynecological Cancer, 18(2), 215–222. questionnaire to an acute medical admissions unit. Irish journal of
Lewin, S., & Reeves, S. (2011). Enacting ‘team’and ‘teamwork’: Using medical science, 178(4), 433–439.
Goffman’s theory of impression management to illuminate interpro- Rushmer, R., Kelly, D., Lough, M., Wilkinson, J. E., Greig, G., & Davies,
fessional practice on hospital wards. Social Science & Medicine, 72(10), H. T. O. (2006). The Learning Practice Inventory: Diagnosing and
1595–1602. developing learning practices in the UK. Journal of Evaluation in
Luecht, R. M., Madsen, M. K., Taugher, M. p., & Petterson, B. J. (1990). Clinical Practice, 13, 206–211.
Assessing professional perceptions: Design and validation of an inter- Salas, E., Sims, D. E., & Burke, C. S. (2005). Is there a “big five” in
disciplinary education perception scale. Journal of Allied Health, 19(2), teamwork? Small Group Research, 36(5), 555–599.
181–191. Sexton, J. B., Helmreich, R. L., Neilands, T. B., Rowan, K., Vella, K., Boyden,
Makary, M. A., Sexton, J. B., Freischlag, J. A., Holzmueller, C. G., J.,. . . Thomas, E. J. (2006). The Safety Attitudes Questionnaire:
Millman, E. A., Rowen, L., & Pronovost, P. J. (2006). Operating Psychometric properties, benchmarking data, and emerging research.
room teamwork among physicians and nurses: Teamwork in the eye BMC Health Services Research, 6(44). doi:10.1186/1472-6963-6-44
of the beholder. Journal of the American College of Surgeons, 202(5), Sidhom, M. A., & Poulsen, M. G. (2006). Multidisciplinary care in
746–752. oncology: Medicolegal implications of group decisions. Lancet
Marsick, V. J., & Volpe, M. (1999). The nature and need for informal Oncology, 7(11), 951–954.
learning. Advances in Developing Human Resources, 1(1), 1–9. Streiner, D. L., & Norman, G. R. (2008). Health measurement scales. A
Marsick, V. J., & Watkins, K. E. (2001). Informal and incidental practical guide to their development (4th ed.). Oxford, UK: Oxford
learning. New Directions for Adult and Continuing Education, University Press.
2001(89), 25–34. SurveyMonkeyTM [Computer software]. Retrieved from http://www.sur
Marsick, V. J., & Watkins, K. E. (2003). Demonstrating the value of an veymonkey.com/
organization’s learning culture: The dimensions of the Learning Tannenbaum, S. I. (1997). Enhancing continuous learning: Diagnostic
Organization Questionnaire. Advances in Developing Human findings from multiple companies. Human Resource Management, 36
Resources, 5(2), 132–151. (4), 437–452.
McFadyen, A. K., Webster, V., Strachan, K., Figgins, E., Brown, H., & Watts, B. V., Percarpio, K., West, P., & Mills, P. D. (2010). Use of the
McKechnie, J. (2005). The Readiness for Interprofessional Learning Safety Attitudes Questionnaire as a measure in patient safety improve-
Scale: A possible more stable sub-scale model for the original version ment. Journal of Patient Safety, 6(4), 206–209.
of RIPLS. Journal of Interprofessional Care, 19(6), 595–603. West, M. A., & Lyubovnikova, J. (2012). Real teams or pseudo teams?
Moilanen, R. (2005). Diagnosing and measuring learning organizations. The changing landscape needs a better map. Industrial and
The Learning Organization, 12(1), 71–89. Organizational Psychology, 5(1), 25–28.
The National Academies of Sciences Engineers and Medicine. (2015).
Improving diagnosis in health care. Washington, DC: The National
Academies Press. Appendix: Team Interprofessional Learning Profiling
Nisbet, G. (2013). Meaning and measurement of informal workplace Questionnaire
interprofessional learning. (Doctor of Philosophy), The University of
Sydney, Sydney, Australia. The questionnaire is designed for online administration.
 286 G. NISBET ET AL.

Please read each statement carefully and mark which response best that you answer each statement as honestly as possible. All information
describes YOUR Interprofessional Team (IPT) Meeting. It is important collected will remain confidential.

Strongly Neither agree Strongly Don’t know/

In this IPT meeting I often am reluctant to speak up for fear of sounding dumb Agree Agree nor disagree Disagree Disagree not applicable
Sometimes I am not sure what other information or knowledge is relevant to share in Strongly Agree Neither agree Disagree Strongly Don’t know/
this IPT meeting Agree nor disagree Disagree not applicable
Sometimes I am not sure what patient information is relevant to share in this IPT Strongly Agree Neither agree Disagree Strongly Don’t know/
meeting Agree nor disagree Disagree not applicable
I am not sure what information other staff really want to hear from me at this IPT Strongly Agree Neither agree Disagree Strongly Don’t know/
meeting Agree nor disagree Disagree not applicable
I don’t feel I contribute new patient information or knowledge to this IPT meeting Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
I don’t feel I contribute new professional or academic knowledge to this IPT meeting Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
It is stressful to speak up in this IPT meeting Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
In this IPT meeting I have a say in patient care decisions Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
I feel my presence at this IPT meeting adds value to the meeting Strongly Agree Neither agree Disagree Strongly Don’t know/
Downloaded by [University of Nebraska, Lincoln] at 00:05 31 May 2016

Agree nor disagree Disagree not applicable

In this IPT meeting I would not feel comfortable raising an issue that I had only a ‘gut Strongly Agree Neither agree Disagree Strongly Don’t know/
feel’ about Agree nor disagree Disagree not applicable
In this IPT meeting I would not have any new professional or academic knowledge Strongly Agree Neither agree Disagree Strongly Don’t know/
worth sharing with others Agree nor disagree Disagree not applicable
In this IPT meeting I feel comfortable to bring up patient care information or issues Strongly Agree Neither agree Disagree Strongly Don’t know/
outside my immediate area of professional expertise Agree nor disagree Disagree not applicable
In this IPT meeting I am uncomfortable asking questions when there is something I Strongly Agree Neither agree Disagree Strongly Don’t know/
don’t understand Agree nor disagree Disagree not applicable
In this IPT meeting it is acceptable to question the doctor about why particular Strongly Agree Neither agree Disagree Strongly Don’t know/
decisions were made Agree nor disagree Disagree not applicable
In this IPT meeting my unique skills and talents are utilized Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
In this IPT meeting differing views are actively encouraged and sought during Strongly Agree Neither agree Disagree Strongly Don’t know/
discussion Agree nor disagree Disagree not applicable
In this IPT meeting is it acceptable to question other professionals on their patient care Strongly Agree Neither agree Disagree Strongly Don’t know/
plans and actions Agree nor disagree Disagree not applicable
In this IPT meeting disagreements and differing views on patient care are ignored or Strongly Agree Neither agree Disagree Strongly Don’t know/
avoided Agree nor disagree Disagree not applicable
The culture in this IPT meeting makes it easy to learn from the errors of others Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
In this IPT meeting team members give open and honest feedback to each other on Strongly Agree Neither agree Disagree Strongly Don’t know/
the care delivered to patients Agree nor disagree Disagree not applicable
Members of this IPT meeting are able to bring up tough team issues and problems Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
In this IPT meeting, team members are encouraged to bring the patient’s views into Strongly Agree Neither agree Disagree Strongly Don’t know/
the decision making process Agree nor disagree Disagree not applicable
In this IPT meeting learning is valued more than just getting through the patient list Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
In this IPT meeting we make time to regularly reflect on how the team meeting is Strongly Agree Neither agree Disagree Strongly Don’t know/
functioning Agree nor disagree Disagree not applicable
Our IPT meeting has all the necessary elements to encourage learning between health Strongly Agree Neither agree Disagree Strongly Don’t know/
professionals Agree nor disagree Disagree not applicable
In this IPT meeting psychosocial aspects of care are adequately discussed Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
This IPT meeting is essential for learning from other health professionals Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
This IPT meeting is essential for team building Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
The IPT meeting is a venue to learn about other health professionals’ clinical practice Strongly Agree Neither agree Disagree Strongly Don’t know/
and wider aspects of their professional and academic knowledge Agree nor disagree Disagree not applicable
This IPT meeting is essential for bringing the patient’s views into the decision making Strongly Agree Neither agree Disagree Strongly Don’t know/
process Agree nor disagree Disagree not applicable
Learning from other health professions is not relevant to this IPT meeting Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
Hearing from other health professionals at this IPT meeting increases my Strongly Agree Neither agree Disagree Strongly Don’t know/
understanding of their scope of practice Agree nor disagree Disagree not applicable
This IPT meeting is essential to reach consensus on the patient’s management plan Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
Learning about my patients is all that I learn from attending a IPT meeting Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
I can gain information more efficiently and effectively through other means Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
In this IPT meeting conversations are mainly between the doctors Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
(Continued )
 JOURNAL OF INTERPROFESSIONAL CARE 287

(Continued).
Strongly Neither agree Strongly Don’t know/
In this IPT meeting I often am reluctant to speak up for fear of sounding dumb Agree Agree nor disagree Disagree Disagree not applicable
In this IPT meeting most of the talking is done by the doctors Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
In making patient care decisions I mainly consult with colleagues from my own Strongly Agree Neither agree Disagree Strongly Don’t know/
profession Agree nor disagree Disagree not applicable
Compared with treating disease, discussion about the psychosocial issues are not Strongly Agree Neither agree Disagree Strongly Don’t know/
allocated enough time in this IPT meeting Agree nor disagree Disagree not applicable
In this IPT meeting you have to ‘push’ your way in to the discussion Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
This IPT meeting achieves whole team discussion and whole team input Strongly Agree Neither agree Disagree Strongly Don’t know/
Agree nor disagree Disagree not applicable
Downloaded by [University of Nebraska, Lincoln] at 00:05 31 May 2016