RM 22 text (Final) 26/7/16 07:02 Page 17

This is a single article from Research Matters: A Cambridge Assessment publication. http://www.cambridgeassessment.org.uk/research-matters/
© UCLES 2016

Collaboration in the 21st century: Implications for

assessment
Simon Child Research Division and Stuart Shaw Cambridge International Examinations

Background The status of collaboration in the 21st

century
In recent years, there has been an increasing focus on conceptualising and
defining so-called 21st century skills. The literature on 21st century skills The focus of this article is the skill of collaboration. Collaboration has
includes a number of frameworks for categorising the skills and knowledge recently been identified as an important educational outcome in its
required for participation in the workplace and in society (Lai & Viering, own right, rather than just a means to develop or assess knowledge,
2012). These frameworks have been motivated by observed changes in which is learned through engagement and practice (Kuhn, 2015; Lai,
how students (and others) have to apply and demonstrate their acquired 2011). Collaboration has been described as a skill that encourages
knowledge; using advanced technologies within multicultural societies in an learning mechanisms (such as induction, deduction and associative
age of increasing economic competition (Suto, 2013). Examples include the learning) to be enacted (Dillenbourg, 1999; Hunter, 2006).
Partnership for 21st Century Learning (P21®), Assessment and Teaching of The NRC (2011) outlined several justifications for collaboration’s
21st Century Skills (ATC21S) and the National Research Council (NRC). status as a key 21st century skill. First, there is a growing emphasis on
Whilst definitions of 21st century skills differ in terms of the placement project and enquiry-based learning. This is motivated by research that
of individual skills within their frameworks (Silva, 2009), there is a degree shows that collaboration has influential effects on student learning and
of consensus established with regards to skill identification. Skills include knowledge retention (Fall, Webb, & Chudowsky, 1997; Rojas-Drummond
creativity and innovation, critical thinking, problem solving, metacognition, & Mercer, 2003; Saner, McCaffrey, Stecher, & Bell, 1994; Webb, 1993).
information and ICT literacy, citizenship, communication, and collaboration It is claimed that collaboration has distinct advantages over individual
(see Suto, 2013, for an overview). Recently, these skills have been linked to problem solving because it allows for: an effective division of labour;
future economic prosperity for individuals and nations, as they provide key the incorporation of information from multiple sources of knowledge,
qualities required to succeed in the global skills race (see Development perspectives, and experiences; and enhanced creativity and quality of
Economics, 2015; P21, 2008). solutions stimulated by ideas of other group members (Organisation for
Given the current status of 21st century skills, there is an increased Economic Co-operation and Development [OECD], 2013). Similarly,
motivation to develop modes of assessment that allow students to collaboration has also been found to increase students’ social
demonstrate their abilities in these domains. As Shute and Becker (2010) competency (e.g., conflict resolution skills and use of helping
note: behaviours) and academic self-concept (Ginsburg-Block, Rohrbeck,

We need to re-think assessment, identify new skills and standards & Fantuzzo, 2006).

relevant for the twenty-first century, and then determine how to best Secondly, there is an increasing need for students to be able to

assess students’ acquisition of the new competencies… Moreover, the apply their knowledge and problem-solving skills in social settings

envisioned new competencies should include not only cognitive (OECD, 2013). Organisations, faced with the need to innovate, use

variables (e.g., critical thinking, reasoning skills) but also non-cognitive collaboration to combine the potential and expertise of their

variables (e.g., teamwork, tolerance, tenacity) as the basis for new employees (Knoll, Plumbaum, Hoffmann, & De Luca, 2010). This is

assessments to support learning. (p.3) linked to recent advancements in technology, which have opened up
new opportunities for how collaboration can be enacted (Salas, Cooke,
The appropriate assessment of 21st century skills is also important as it & Rosen, 2008). The application of social technologies by individuals
provides value and motivation to students, and can help structure and across organisations has become a legitimate mode of enquiry
pedagogical approaches (e.g., Swan, Shen, & Hiltz, 2006). However, any (Blaskovich, 2008), and this ability has been regarded as important for
assessment has to resolve tensions related to its validity, reliability, the workforce of the future (OECD, 2013).
comparability and delivery. Satisfactory construct definition for the The stated importance of collaboration means that appropriately
purposes of assessment has always been considered an essential principle defining its construct remains an important aim. The main issue here is
in testing. If these constructs are not well defined, then it is difficult to that the notion of collaboration, although almost universally accepted
support the claims awarding bodies make about the usefulness of their as being useful for application in the classroom and beyond, is
assessments. Awarding bodies are challenged with the task of articulating conceptually vague (Brna, 1998). Different frameworks of 21st century
how their assessments represent the target construct, how potential skills place collaboration as either a learning skill (P21, 2015), an
contaminating factors related to the assessment are controlled, and how interpersonal skill (NRC, 2011) or a way of working (ATC21S, 2015).
the assessment achieves a desired level of reliability. This is challenging for These frameworks have different conceptualisations of collaboration as
21st century skills due to the potential for subjectivity in the assessment a construct, and in terms of its interaction with other skills (Lai &
process (Suto, 2013). Viering, 2012).

R E S E A R C H M AT T E R S : I S S U E 2 2 / S U M M E R 2 0 1 6 | 17
RM 22 text (Final) 26/7/16 07:02 Page 18

Aims of the article the goal requires more than one person to pool resources. This view is
shared by Roschelle and Teasley (1995), who broadly define collaboration
When assessing collaboration, there is a need for a clear understanding of as a “coordinated, synchronous activity that is the result of a continued
what is being tested, based on a theoretically-sound and agreed upon attempt to construct and maintain a shared conception of a problem”
definition. In light of this important issue, this article has two main (p.70), and Dillenbourg (1999) who defines collaboration as “a situation
intentions. First we aim to provide an overview of how collaboration is in which two or more people learn or attempt to learn something
conceptualised, and how it is distinguished from other related group together” (p.1). The sharing of roles and responsibilities during
activities (e.g., cooperation). Integral to this aim is the ambition to collaboration closely relates to the concept of the ‘joint problem space’
develop a coherent understanding of the abilities underlying the targeted (or JPS, Roschelle & Teasley, 1995). The JPS implies that group members
construct. enter into a social contract with the joint aim of achieving a desirable
The second aim is to discuss how the conceptualisations of outcome. In this sense, group members enter into a collaborative ‘state’
collaboration underpin the development of appropriate methods of (Brna, 1998) that has to be effectively maintained until the problem is
assessment. Specifically, we explore how the task given to students can solved, or the outcome is reached.
potentially optimise the opportunities for collaboration to occur amongst There is an important distinction to be made here between
group members. We also consider how different conceptualisations of collaboration as it has been defined above, and cooperation. These terms
collaboration are currently assessed, and the issues raised in the have often been used synonymously when referring to group-related
development of large-scale assessment. activities (Lai & Viering, 2012) yet have important conceptual differences.
Cooperation is typically a division of labour among group members,
but can also be part of a process of allowing information to be accessed
Defining the construct of collaboration amongst group members. It occurs when a task is divided up into

The basic facets of what constitutes a collaborative activity are reasonably individually manageable subparts, which are subsequently constructed

well rehearsed in the literature. Academics who have attempted to into a final outcome. To cooperate in this way, group members do not

delineate collaboration from other related activities have articulated three need to maintain a mutual understanding of the task goals, as individuals

fundamental aspects to collaborative learning. These three aspects are simply focus on their subtasks. It can also encourage asymmetric

expressed in the definition provided by the OECD (2013): individual contributions towards the task goal. Collaboration, on the other
hand, contains inherent flexibility of roles and responsibilities with
Collaborative problem solving competency is the capacity of an
regards to the various subtasks in achieving a goal (Lai, 2011).
individual to effectively engage in a process whereby two or more
Another key distinction is between collaboration as process and
agents attempt to solve a problem by sharing the understanding and
collaboration as outcomes. The collaborative ‘state’ is related to process.
effort required to come to a solution and pooling their knowledge,
This broadly relates to how well the collaborative state is maintained
skills and efforts to reach that solution. (p.6) [emphasis added]
and progressed. We have identified six fundamental facets of the
Each of the three emphasised aspects are important factors in the collaborative process in Figure 1.
maintenance of collaborative activity. For a collaborative ‘state’ to be On the other hand collaboration as outcome implies that the final
constructed (Brna, 1998) there has to be a task where the achievement of product takes precedence over the means to achieve the goal.

Social interdependence Conflict resolution

When the outcome of individuals is affected by their own and others’actions. Peer interaction promotes cognitive conflict by exposing discrepancies
Positive interdependence is when individuals believe that they can achieve their between peers’ own and others’ knowledge. The negotiation of conflicts of
goals if other individuals achieve their goals as well. Negative interdependence viewpoint is an important aspect of effective collaborative task design
(or competition) is when individuals believe they can only achieve their goals if (Fawcett & Garton, 2005; Rosen, 2014).
others fail. Implies a degree of synchronicity between group members, in that
they are compelled to work together, and are thus motivated to do so
(Johnson & Smith, 2007).

Introduction of new ideas Sharing of resources

Related to conflict resolution, team members should be effective in offering Part of the maintenance of the collaborative state. An effectively designed
solutions for the task at hand, which can then be negotiated collaborative task should not be able to be solved by individual effort.
(OECD, 2013). Subsequently, resources should need to be pooled amongst team members
(Brna, 1998).

Cooperation/task division Communication

Cooperation is a division of labour betweengroup members. It occurs when a Communication in a collaborative task comprises rich interactive features, of which
task is divided up into individually manageable subparts, which are subsequently only one is the speech (or text) produced by group members. During the collaborative
constructed into a final outcome. Although this is conceptually different to task, communication acts to bring implicit thought to explicit explanation
collaboration, at a fine-grained level, all collaborative tasks have a degree of cooperation (Webb, 1991).
(Lai & Viering, 2012).

Figure 1: The six facets of the collaborative process

18 | R E S E A R C H M AT T E R S : I S S U E 2 2 / S U M M E R 2 0 1 6
RM 22 text (Final) 26/7/16 07:02 Page 19

This approach assumes that the task itself has encouraged collaborative effective assessment of collaboration. The first implication for assessors
processes to be enacted, and that the task is sufficiently complex that is that, before the group task is set, practitioners need to engender a
collaboration is required for its successful completion. The separation of sense of group identity and rapport amongst the group members.
process (i.e., how well the collaborative state is maintained and Similarly, high levels of trust and shared understanding, and depth of
progressed) from outcome (i.e., the final product following a period of relationships have been identified as pre-conditions to collaboration
collaboration) is a key distinction that emerges from the literature, and (Monteiro & Morrison, 2014; Peters & Manz, 2007). Crucial to this is the
has implications for how collaboration is optimally assessed. This is role of the task setter, as they can encourage group members to build
discussed in the next section. trust and mutual understanding before the assessment task (Mercer,
1996; Laurillard, 2012). To encourage true collaboration (which can then
be observed and measured) assessors need to manipulate group
Implications for the assessment of members’ experiences with one another so that channels of

collaboration communication and mutual understanding are optimised before

assessment commences.
The complexity of collaboration as a construct leads to two main
challenges for assessors. First, they must create the working conditions
necessary for collaboration to be engendered and encouraged. Secondly,
Task setting
they must be able to pinpoint aspects of an individual’s behaviours
within a group task, so that a judgement can be made about that A fundamental element of a successful assessment of collaboration is
individual’s general capacities for collaboration. These issues are that the task itself should provoke all members of the group to share
intimately related, with aspects related to collaboration formulation their views and ideas on potential courses of action (Dillenbourg, 1999).
constraining (or optimising) the possibilities for assessment. The As mentioned in the previous section, this can be encouraged by setting
approach to assessment (e.g., the distribution of individual or group up effective pre-task relationships among group members. However,
marks) can also influence the potential for collaboration to be this alone is unlikely to be sufficient for collaborative strategies to be
engendered. utilised. We have identified five criteria that assessors should meet
Here we first explore how the task given to candidates can be when devising a collaborative problem-solving task. Some of these
optimised so that collaborative processes can be observed. We then criteria relate specifically to the task itself, whilst others relate to
consider the modes of observation available to assessors interested in aspects of group composition.
either the collaborative process or outcomes.
1. Task is sufficiently complex: The common factor in all
assessments of collaboration is that group members are set a
problem. Ideally the problem engenders alternative suggestions
Pre-task from within the group about the best course of action, or requires
group members to research potential solutions to the problem.
It is important to note that collaboration among group members is not
Overly simplistic or trivial tasks do not encourage group members
an automatic outcome of setting a task with a shared goal (Kreijens,
to collaborate because there is little need to share cognitive load.
Kirschner, & Jochems, 2003). Indeed, there are significant barriers to
High-quality collaborative tasks are thus likely to include elements
collaboration taking place at all. For example, in some cases, group
of constructive argumentation (Brna & Burton, 1997).
members may value achieving a quick consensus above the potential
difficulties that can be encountered when introducing new ideas or 2. Task is ill-structured: A good collaborative task is one that cannot
negotiating contrasting positions. This ‘rapid’ consensus may be of be solved by one capable member of the group. Task complexity is,
detriment to the eventual outcome (e.g., Janis, 1982; Rimor, Rosen, & at least in part, determined by the structure of the task. Tasks
Naser, 2010). should be sufficiently open, with more than one plausible (or
Collaborative success is therefore dependent on establishing optimal defensible) solution (Webb, Nemer, Chizhik, & Sugrue, 1998).
group dynamics. Key aspects include the development of a sense of Furthermore, individual roles should not be designated by the task
community among individual group members, setting up relationships setter (at least initially) as this encourages unnecessary processing
among group members so that they all have the opportunity to constraints. Strictly defined roles can also create the illusion of
perform the same range of actions, and an equality of status of collaboration. This also introduces the problem of the group being
individuals. Dillenbourg (1999) refers to symmetry on three planes, all led by the expectations of the task setter, which may restrict novel
of which are required for collaboration to occur: or innovative solutions.

1. Symmetry of action: The same range of actions is allowed to each 3. Task should utilise technologies that facilitate the
group member. collaborative process: There are a number of ways in which
technology can be introduced into a collaborative task: as a
2. Symmetry of knowledge: The group members have a similar level
resource in information gathering; as a focus of the interaction; or
of expertise (but different viewpoints on the task).
as a collaborative partner. Tasks that involve group members
3. Symmetry of status: Individual group members have a similar
collaborating using computers as a means of communication
status with respect to other group members.
typically use email, instant messaging applications, discussion
Whilst these points of symmetry refer to collaboration in numerous forums or videoconferencing. The advantage of these modes of
(although not all) contexts, it has some important implications for the communication is that they can enhance the reach of

R E S E A R C H M AT T E R S : I S S U E 2 2 / S U M M E R 2 0 1 6 | 19
RM 22 text (Final) 26/7/16 07:02 Page 20

communication, and increase the potential for different perspectives ‘best-fit’ decision being made. In this process, the assessor implicitly
to be expressed (e.g., Thorpe, 1998). Despite these perceived creates an evidence base from which to ground their decision-making.
advantages, it remains to be seen whether computer-supported The use of technology has been identified as a potential means from
means of communication within a collaborative task can overcome which observation of the collaborative process can be enhanced
challenges created by the initial distance of participants from each (e.g., Austin, Smyth, Rickard, Quirk-Bolt, & Metcalfe, 2010; MacDonald,
other, both physically and in terms of the creation of a JPS (Kreijns, 2003). For example, the use of wikis can provide a full record of
Kirschner, & Vermeulen, 2013). individuals’ contributions to a task, in addition to the responses from
4. Group member dynamics engender negotiation: Negotiation is other group members (Judd, Kennedy, & Cropper, 2010). Taken together,
unlikely if all group members agree on a solution to a problem, or if assessors can analyse and reflect on these interactions off-line, potentially
one group member forces their will or assumed knowledge onto improving the evidence base from which judgements are made. However,
another (e.g., in a tutoring scenario). When assessing collaboration, different methods of analysis of these data are possible, and so careful
it is therefore important to place students in groups where there consideration of how this evidence is used alongside more typical
may be differences in opinion (Brna & Burton, 1997). However, the observational approaches is required.
evidence on creating effective heterogeneous groups is mixed (see Interestingly, there have been recent attempts to standardise the
Webb et al., 1998). Some research has found that groups manifesting process of collaboration through the use of computer partners (see Rosen
a range of abilities collaborate more productively compared to more & Tager, 2013; OECD, 2013). These assessment procedures have the
homogeneous groups. This effect is observed more clearly in ill- advantage of controlling the task scenario, so that the student can be
structured tasks. Where the task is clearly specified, low-ability group encouraged to negotiate and offer different courses of action. It is
members are more likely to display negative behaviours such as debatable as to whether the level of control possible using this assessment
‘social loafing’ (Salomon & Globerson, 1989). Social loafing appears method outweighs issues of ecological validity.
to also be a function of group size. In general terms, the larger the A third issue relates to the distribution of marks among individuals and
group the more likely that some group members will not contribute the group. When marks are given to individuals, there is the potential for
to the task due to asymmetric interactions among group members. collaboration to become competition, and for individuals to feel aggrieved
if their contributions are not noted. However, when marks are given at the
5. Group is motivated to work together: In setting the task, the
group-level, this potentially obscures individual contributions. Further
assessor needs to motivate group members to work together. If the
issues are raised when we consider that the usual aim of assessment
criteria outlined above are met, then the group dynamic and the task
(and qualifications more broadly) is for a judgement to be made on
itself are likely to be highly motivating. This is closely related to the
individuals. For any assessment of collaboration, then, it is imperative that
concept of social interdependence, which is based on mutual
group members are given individual marks. The focus of this individual
encouragement and accountability (Johnson & Johnson, 1989, 2002 –
mark, however, should centre on positive contributions to the
see Figure 1). How group members are assessed during the
collaborative process. The balance between group-level and individual-
collaborative task may contribute to developing social
level marks for a collaborative task is an important consideration in the
interdependence among group members. Research has found that
future development of models of assessment of collaboration.
productivity is improved when members are rewarded as a group,
Related to this issue is the origin of the marks: can a case be made for
within a context of individual accountability (Bossert, 1988; Slavin,
the assessors to be located within the group, via either self- or peer-
1983).
assessment? These models of assessment have been identified as
improving group processes, motivation and engagement, and achieving a

Towards an assessment model for good level of reliability (Mills & Glover, 2006; Race, 2001). However,
concerns remain about their appropriateness as part of an assessment
collaboration
strategy for large-scale qualifications.
The appropriate assessment of collaboration as a process or as an
outcome reflects the distinct characteristics of these two Assessment of collaborative outcomes
conceptualisations. If the focus of assessment is on the learning achieved during collaboration,
then the assessment itself should specifically relate to the quality of the
Assessment of the collaborative process final product. This is typically assessed by a terminal demonstration of
The first aspect to consider regards the desirable characteristics of an learning either by a group presentation or the creation of a portfolio,
individual who is effectively collaborating with their peers. We have where learning could be showcased (MacDonald, 2003). The use of
identified six elements that comprise effective maintenance and progress portfolios, which are held centrally within a shared network, allows a
of the collaborative state, as depicted in Figure 1. This framework may be longitudinal record of learning to be held by the assessor over time
a useful starting point in directing assessors to consider the fundamental (Hauge & Wittek, 2002). This can encourage the assessor to understand
skills within the collaborative process. each student’s development of understanding of a topic area.
The next issue relates to how the process of collaboration can be In assessing the outcome of individual learning within a collaborative
optimally observed, from which judgements on performance can be context, two main considerations need to be made. First, the assessor
made. Assessors have the challenging task of relating individuals’ needs to have measured each student’s understanding of the topic of
behaviours to both the context of the task and to the dynamics of the interest prior to the task, so that the ‘before’ and ‘after’ of learning can be
group. Appropriate adjustments of these judgements are required as established. Secondly, the assessor needs to set a task where learning
group members negotiate and progress towards a solution, with a final relies to an extent on the collaborative process.

20 | R E S E A R C H M AT T E R S : I S S U E 2 2 / S U M M E R 2 0 1 6
RM 22 text (Final) 26/7/16 07:02 Page 21

Conclusions and future questions changes the aspects of the interaction that makes collaboration more
likely. Technology will need to overcome significant challenges for it to be
This article first aimed to briefly outline different conceptualisations of a suitable mode from which collaboration can be derived and observed.
collaboration, and made the important distinction between the
Acknowledgements
collaborative process (which is demonstrated within the collaborative
activity) and the outcome (which is demonstrated by the quality of the We are grateful to Irenka Suto, Research Division, for her helpful
knowledge or understanding of the group members). The article has also suggestions on an earlier draft of this article. We would also like to thank
explored the implications for how the different constructs of Sylvia Green, Research Division, Paul Bullen-Smith, Cambridge
collaboration can be assessed, focusing mainly on task conditions that International Examinations, and Helen Eccles, formerly of Cambridge
need to be met for collaboration to be encouraged. International Examinations, for their insightful discussion over the course
There remain several questions for future research. Specifically, the of the research.
future development of effective assessments of collaboration relies on
several decisions being made by developers regarding the desired References

direction of the assessment. These include the following: ATC21S (2015). Assessment and Teaching of 21st Century Skills. Official website.
Available online at: www.atc21s.org
What should be the focus of the assessment of Austin, R., Smyth, J., Rickard, A., Quirk-Bolt, N., & Metcalfe, N. (2010):
collaboration? Collaborative digital learning in schools: Teacher perceptions of purpose and
The main distinction made here is between collaboration as process effectiveness. Technology, Pedagogy and Education, 19(3), 327–343.

and collaboration as outcome. This decision routes the possible options Blaskovich, J. L. (2008). Exploring the effect of distance: An experimental
for assessment. If the purpose of the assessment is to target the investigation of virtual collaboration, social loafing, and group decisions.
Journal of Information Systems, 22(1), 27–46.
collaborative process, then assessment must focus on targeting individual
contributions to the collaborative effort. However, some focus on project Bossert, S. T. (1988). Cooperative activities in the classroom. Review of Research in
Education, 15, 225–250.
outcomes may be required for the purpose of student motivation, and
to gather a more holistic view of a student’s collaborative skill. If the Brna, P. (1998). Models of collaboration. Proceedings of the Workshop on
Informatics in Education, XVIII Congresso Nacional da Sociedade Brasileira de
aim of assessment is to measure student learning via collaboration
Computação, Belo Horizonte, Brazil.
(a specific form of collaborative outcome), then group-based assessment,
Brna, P. & Burton, M. (1997). The computer modelling of students collaborating
for example, is not appropriate. Assessment of individual learning would
in learning about energy. Journal of Computer Assisted Learning, 13, 193–204.
likely rely on multi-stage assessment procedures.
Development Economics. (2015). The value of soft skills to the UK economy.
Furthermore, the focus of assessment will be closely related to the
Retrieved from http://www.backingsoftskills.co.uk/The%20Value%20of%
other objectives of the target qualification. The relative importance of 20Soft%20Skills%20to%20the%20UK%20Economy.pdf
collaboration within the entire structure of the target qualification
Dillenbourg, P. (1999). What do you mean by ‘collaborative learning?’
framework will have implications for its assessment. In P. Dillenbourg (Ed.), Collaborative-learning: Cognitive and Computational
Approaches (pp.1–19). Oxford: Elsevier.
If the focus of assessment is the collaborative process, how
Fall, R., Webb, N., & Chudowsky, N. (1997). Group Discussion and Large-Scale
should the identified subskills be weighted? Language Arts Assessment: Effects on Students’ Comprehension. CSE Technical
We have identified six subskills that contribute to the collaborative Report 445. Los Angeles: CRESST.
process. However, the status of these skills, and how they can best be Fawcett, L. M., & Garton, A. F. (2005). The effect of peer collaboration on
observed, is a source for future investigation. children’s problem-solving ability. The British Journal of Educational
Psychology, 75(2), 157–169.
What is the desired division of individual/group marks for Ginsburg-Block, M. D., Rohrbeck, C. A., & Fantuzzo, J. W. (2006). A meta-analytic
students? review of social, self-concept, and behavioral outcomes of peer-assisted
learning. Journal of Educational Psychology, 98(4), 732–749.
Giving an individual score to candidates meets the imperative for them
to be rewarded for their contributions, and to prevent negative Hauge, T. E. & Wittek, L. (2002). Portfolios as mediators for collaborative learning
and professional development in a distributed environment of teacher
collaborative behaviours. The inclusion of a group score encourages a
education. Paper presented at the European Association for Research on
degree of mutual accountability which is essential in encouraging Learning and Instruction (EARLI) Assessment Conference: Learning
students to display the desired construct. To encourage full participation, Communities and Assessment Cultures, Newcastle, England.
both individual and group effort therefore need to be assessed. However, Hunter, D. (2006). Assessing collaborative learning. British Journal of Music
the weighting of this scoring approach remains an open question. For Education, 23(1), 75–89.
example, the idea of providing a single mark for an entire group related to Janis, I. L. (1982). Counseling on personal decisions: Theory and research on
the final outcome has been criticised on the basis of fairness. short-term helping relationships. New Haven: Yale University Press.

Johnson, D. W., & Johnson, R. T. (1989). Cooperation and competition: Theory and
How can technology be utilised to optimal effect? research. Edina, MN: Interaction Book Company.
Maintaining a consistent and reliable record of interaction is problematic, Johnson, D. W., & Johnson, R. T. (2002). Learning together and alone: Overview
particularly when assessing large groups. For example, the use of online- and meta-analysis, Asia Pacific Journal of Education, 22(1), 95–105.
based forums and wikis can provide a useful record of interactions Johnson, D. W., & Johnson, R. T., & Smith, K. (2007). The state of cooperative
between participants which can then be utilised for assessment purposes. learning in postsecondary settings. Educational Psychology Review, 19(1),
Interestingly, the very process of introducing technology fundamentally 15–29.

R E S E A R C H M AT T E R S : I S S U E 2 2 / S U M M E R 2 0 1 6 | 21
RM 22 text (Final) 26/7/16 07:02 Page 22

Judd, T., Kennedy, G. & Cropper, S. (2010). Using wikis for collaborative learning: Rimor, R., Rosen, Y., & Naser, K. (2010). Complexity of social interactions in
Assessing collaboration through contribution. Australasian Journal of collaborative learning: The case of online database environment.
Educational Technology, 26(3), 341–354. Interdisciplinary Journal of E-Learning and Learning Objects, 6, 355–365.
Knoll, S. W., Plumbaum, T., Hoffmann, J. L., & De Luca, E. W. (2010). Collaboration Rojas-Drummond, S. & Mercer, N. (2003). Scaffolding the development of
ontology: Applying collaboration knowledge to a generic group support effective collaboration and learning. International Journal of Educational
system. In G-J. De Vreede (Ed). Proceedings of the Group Decision and Research, 39, 99–111.
Negotiation Conference 2010, Delft, The Netherlands (p.37).
Roschelle, J. & Teasley, S. D. (1995). The construction of shared knowledge in
Kreijns, K., Kirschner, P. A., & Jochems, W. (2003). Identifying the pitfalls for social collaborative problem-solving. In C.E. O'Malley (Ed.), Computer-supported
interaction in computer-supported collaborative learning environments: a collaborative learning (pp.69–97). Berlin: Springer-Verlag.
review of the research. Computers in Human Behavior, 19, 335–353.
Rosen, Y. (2014). Comparability of Conflict Opportunities in Human-to-Human
Kreijns, K., Kirschner, P. A., & Vermeulen, M. (2013). Social aspects of CSCL and Human-to-Agent Online Collaborative Problem Solving, Tech Know Learn,
environments: A research framework. Educational Psychologist, 48(4), 19, 147–164.
229–242.
Rosen, Y., & Tager, M. (2013). Computer-based assessment of collaborative
Kuhn, D. (2015). Thinking together and alone. Educational Researcher, 44(1), problem solving skills: Human-to-agent versus human-to-human approach.
46–53. Research & Innovation Network: Pearson Education.
Lai, E. R. (2011). Collaboration: A Literature Review. Retrieved from Salas, E., Cooke, N. J., & Rosen, M. A. (2008). On teams, teamwork, and team
http://images.pearsonassessments.com/images/tmrs/Collaboration- performance: Discoveries and developments. Human Factors, 50, 540–548.
Review.pdf
Salomon, G. & Globerson, T. (1989). When teams do not function the way they
Lai, E. R., & Viering, M. (2012). Assessing 21st century skills: Integrating research ought to. International Journal of Educational Research, 13(1), 89–100.
findings. Paper presented at the annual meeting of the National Council on
Saner, H., McCaffrey, D., Stecher, B., Klein, S., & Bell, R. (1994). The effects of
Measurement in Education, Vancouver, B.C., Canada.
working in pairs in science performance assessments. Educational Assessment,
Laurillard, D. (2012). Teaching as a design science: Building pedagogical patterns 2(4), 325–338.
for learning and technology. Abingdon, UK: Routledge.
Shute, V. J., & Becker, B. J. (2010). Innovative Assessment for the 21st Century.
MacDonald, J. (2003). Assessing online collaborative learning: process and Supporting Educational Needs. New York: Springer.
product. Computers and Education, 40, 377–391.
Silva, E. (2009). Measuring skills for 21st-century learning. Phi Delta Kappa, 90(9),
Mercer, N. (1996). The quality of talk in children’s collaborative activity in the 630–634.
classroom. Learning and Instruction, 6(4), 359–377.
Slavin, R.E. (1983). When does cooperative learning increase achievement?
Mills, J., & Glover, C. (2006) Using assessment within course structure to drive Psychological Bulletin, 94, 429–445.
student engagement with the learning process. Retrieved from
Suto, I. (2013). 21st Century skills: Ancient, ubiquitous, enigmatic? Research
http://www.open.ac.uk/fast/pdfs/John%20Mills.pdf
Matters: A Cambridge Assessment publication, 15, 2–8.
Monteiro, E. & Morrison, K. (2014). Challenges for collaborative blended
Swan, K., Shen, J., & Hiltz, S. R. (2006). Assessment and collaboration in online
learning in undergraduate studies. Educational Research and Evaluation:
learning. Journal of Asynchronous Learning Networks, 10(1), 45–62.
An International Journal on Theory and Practice, 20(7–8), 564–591.
Thorpe, M. (1998). Assessment and “third generation” distance education.
National Research Council (2011). Assessing 21st century skills. Washington, DC:
Distance Education, 19(2), 265–286.
National Academies Press.
Webb, N. M. (1991). Task-related verbal interaction and mathematical learning in
OECD (2013). Programme for International Student Assessment (PISA) 2015:
small groups. Research in Mathematics Education, 22(5), 366–389.
Draft Collaborative Problem Solving Framework. Retrieved from
http://www.oecd.org/pisa/pisaproducts/Draft%20PISA%202015% Webb, N. M. (1993). Collaborative group versus individual assessment in
20Collaborative%20Problem%20Solving%20Framework%20.pdf mathematics: Processes and outcomes. Educational Assessment, 1(2),
131–152.
Partnership for 21st Century Learning (2015). Official website. Available online
at: http://www.p21.org/ Webb, N. M., Nemer, K. M., Chizhik, A. W., & Sugrue, B. (1998). Equity issues in
collaborative group assessment: Group composition and performance.
Peters, L. M., & Manz, C. C. (2007). Identifying antecedents of virtual team
American Educational Research Journal, 35(4), 607–651.
collaboration. Team Performance Management, 13, 117–129.

Race, P. (2001). A briefing on self, peer, and group assessment. Assessment Series
Number 9. York, UK: Learning and Teaching Support Network.

22 | R E S E A R C H M AT T E R S : I S S U E 2 2 / S U M M E R 2 0 1 6