Introduction

Widespread agreement around the importance of the body in learning, and increasing access to embodied technologies, have led the field to explore new ways to leverage embodied learning in technology designs. Research in embodied cognition generally explores how cognition is tied to sensory and perceptual systems grounded in the body (Horn et al., 2012). Building on this, researchers have demonstrated the utility of embodied activity within mixed-reality (MR) environments for young students’ science learning (Enyedy et al., 2012; Lindgren & Johnson-Glenberg, 2013). Mixed-reality learning environments offer opportunities to explore concepts with the body as the technology tracks and displays movements on a virtual display. As participants learn through movement in the space, the system provides real-time feedback on how movements connect with the disciplinary concepts to be learned. Over time, learners can see the metaphorical connection between their embodied actions and the disciplinary concepts made visible by the MR simulation. Drawing these connections relies heavily on discussion, and these discussions are about embodied activity and use the body and gestures as resources for communication.

Embodiment studies in computer-supported collaborative learning (CSCL) contexts have also emphasized the importance of motion and gesture in students’ communication and knowledge construction (Broaders et al., 2007; Lindgren et al., 2016a, b). To analyze how embodied communication helps young children collectively explore complex science concepts, we build on the Learning in Embodied Activity Framework (LEAF; Danish et al., 2020), to unpack the role of embodiment in an MR environment at both individual and collective levels. We view these levels as connected and mutually constituting; individual experiences shape and are shaped by collective experiences. By exploring students’ embodied communication within an MR environment, we can support designers, instructors, and researchers in reflecting on how these are connected to learning through embodiment and assessing embodied learning outcomes. In particular, we analyze data from the Science through Technology Enhanced Play (STEP) project (Danish et al., 2015, 2020) and focus on embodied communication within the classroom and how students use the embodiment to represent their understanding afterward.

Within the context of one elementary first and second grade classroom in which young children used the STEP MR environment to explore a particle model of the states of matter, we explore the following questions:

  1. (1)

    How do students communicate differently when asked about states of matter in multiple-choice written tests and when they demonstrate their understanding by moving their bodies and gesturing in an interview?

  2. (2)

    How do students use embodiment to help represent the characteristics of particles within different states of matter in an interview context?

  3. (3)

    How do students’ embodied interactions in the classroom activities utilizing the STEP MR environment build on and lead to the development of shared ways of communicating their ideas about states of matter across contexts?

Literature review

Theoretical framework: Learning in embodied activity framework

Due to the rapid development of technology, a rising number of research studies have focused on the way students collaboratively learn within a computer-supported collaborative learning (CSCL) environment (Birchfield & Megowan-Romanowicz, 2009; Danish et al., 2015, 2020). Mixed-reality (MR) environments offer students an interactive environment that integrates real-world and digital resources in new ways to support learning. For example, when students interact with the real world, such as running in the classroom, their movement could be transferred to digital characters in the computer simulation. Thus, students’ learning is mediated by their interaction with digital resources, physical resources, peers, and teachers. Therefore, it is important to better understand how these different resources mediate student interactions and learning within these kinds of computer-supported learning environments. Numerous studies have noted the importance of communication among participants for understanding how they learn collaboratively within computer-supported learning environments (Enyedy & Hoadley, 2006; Herrmann & Kienle, 2008; Kershner et al., 2010). However, much of this research into communication in CSCL studies has focused on improving students’ communication to construct their shared understanding (Ludvigsen & Mørch, 2003; Herrmann & Kienle, 2008) rather than unpacking the role of communication in supporting learning.

There is also a gap in exploring the role of embodiment in communication within MR environments. Thus, in this paper, we focus on the way students’ embodied interaction in a mixed-reality environment supports their communication of key ideas. Embodied cognition is one of the foundational pillars of the learning sciences. Drawing from a range of disciplines, including cognitive science, cognitive linguistics, neuroscience, artificial intelligence, and phenomenology, proponents of this theory broadly view cognition, learning, and problem solving as rooted in the way the human body interacts with the social and material world (Alibali & Nathan, 2018; Barsalou, 2008; Church & Goldin-Meadow, 2017; Núñez et al., 1999). Naturally, this has immense implications for education (Glenberg, 2008) as it means that the construction of the learner’s environment, the tools made available to learners, and the representational/semiotic and communicative actions of learners are all richly involved in the processes of learning. Indeed, a growing and diverse literature depicts embodied practices in various settings, such as the different ways hand gestures are used in classrooms (Alibali & Nathan, 2012; Crowder, 1996; Roth, 2001; Singer, 2017), the role of movement in learning (Marin, 2020), the relational and temporal dimensions of learning (Vossoughi et al., 2020), interactions with technologies (Davidsen & Ryberg, 2017; Hall et al., 2020), and interactions with MR technologies (Johnson-Glenberg et al., 2014). However, as there is no unified theory of embodied cognition (Wilson, 2002; Ziemke & Frank, 2007), education researchers design for embodied learning in very different ways (Steier et al., 2019).

As the focus of this paper is on the collective embodied learning experiences of young children, we describe two prominent perspectives and corresponding instructional designs found in CSCL and the learning sciences. One perspective leverages the embodied intuitions of the individual learner to foster learning. In this perspective, the human conceptual system is understood to be grounded in our experiences of living and being in the world (Barsalou, 2008; Johnson-Glenberg et al., 2009; Lakoff, 1993), which, in pedagogy, translates to guiding learners to grasp/access embodied groundings of the disciplinary content in different ways. The second perspective on embodied learning underscores interactions across learners and how their communication and actions are layered to foster embodied learning (Goodwin, 2011; Roth, 2007). Communication is an important part of students building shared understanding (Herrmann & Kienle, 2008). Researchers from this perspective carefully examine the interactions between learners’ bodily, social, and material resources to understand how learners make meaning and develop intersubjectivity (Goodwin, 2013) and design learning contexts that foster learners’ opportunities to use tools and develop practices while immersed in culturally rich, semiotic inquiry (Horn, 2018).

To explore the importance of embodied communication within an MR environment, this study draws on the Learning in Embodied Activity Framework (LEAF; Danish et al, 2020). LEAF is a framework that aims to synthesize Cultural Historical Activity Theory (CHAT; Engstrom, 1987) and cognitive theories of embodiment (e.g., Lindgren & Johnson-Glenberg, 2013) in a way that highlights the importance of examining embodied cognition and learning at both the individual and collective levels, as well as interactions across these levels. Colloquially, this framework notes that when a child in one of our studies is running across the room pretending to be a gas particle, they experience that moment at both individual and collective levels simultaneously. Individually, they may feel the speed and recognize how this both requires energy and brings them closer to or further from their peers. At the collective level, they may also notice how macro states (solid, liquid, gas) change if they move in different directions from their peers or chase each other around. The former leads to increasing and decreasing distances, and the latter to more consistent distances. Individually, they may be considering how far they can get from peers. Collectively, the group may be debating whether they want to produce a solid or a liquid and how that might lead them to change their movements. In later discussions, students may recount the feeling of moving fast and how it helped them achieve the group goal, and others may comment on what this helped them notice. LEAF argues that these are all interrelated experiences, shaping each other, and thus a robust approach to understanding embodied cognition needs to attend to them all simultaneously.

Like other CHAT-based theories, LEAF recognizes that learning occurs in activities oriented toward a shared collective object or set of goals (Engestrom, 1987). In many MR environments like STEP, the participants have a shared goal or object, often provided by the teacher, and yet negotiated in interaction. In the STEP Particles project, this means they might be focused on producing a specific state of matter or finding out what the state of matter is. While this set of goals may emerge and shift over time (Danish et al., 2020; Enyedy et al., 2015), it is also valuable for understanding when and how participants interact and move. From a CHAT perspective, it is also important to recognize that individual actions are all mediated by elements of the activity system (Wertsch, 2017). That is, each action is not only shaped by the individual’s goals and shared object but also by the presence of artifacts such as the projected simulation by members of the community, including peers and the teachers, by rules or norms, and by the division of labor that shapes how peers interact with each other.

LEAF aims to connect activity to grounded experiences by recognizing that individuals feel their bodies and experience these moments in rather unique ways that are grounded in their own prior physical experiences (Barsalou, 2008; Johnson‐Glenberg et al., 2009; Lakoff, 1993). The experience of running across the room as a particle is tied to other experiences of running on the playground, driving in a car, or chasing a friend, to name a few. Thus, the experience of being a particle in STEP builds in some important ways on each child’s other physical experiences. When these experiences are analogous, they can provide powerful insights for learners (Lindgren & Johnson-Glenberg, 2013). Importantly, LEAF notes that these two levels of experience are not parallel but intertwined. The same physical experience might be tied to different meanings or past experiences. For example, a pushing gesture with two hands might imply a force being imparted to an object in an embodied physics model (Enyedy et al., 2015) or the transfer of energy to a particle in a model of gas (Danish et al., 2020).

In this paper, we focus on how students build their understanding through embodied communicative acts. That is, learners, negotiate and recognize the meaning of an embodiment such as a push through a host of social and communicative acts, which require them to also learn the meaning that others ascribe to these actions even as they contribute to that shared understanding (Vygotsky & Cole, 1978). Furthermore, embodiment, including gesture, is often recognized as central to this communicative, meaning–making process (Goodwin, 2020). Returning to our example from above, a child who wants to encourage their peers to move as quickly as them might use a mix of words like “speed up!” and gestures, such as waving their hands. Similarly, as these learners explain what they understood as happening in the MR environment, they may again use gestures to indicate where they moved and how. While gesture has proven powerful for helping learners to express their emergent ideas before they can express them in talk (Goldin-Meadows, 2004), it is also socially constructed. It takes on a shared meaning within each activity and the larger society. Thus, there is a need to understand how such embodied communication develops and mediates young children’s learning within an MR environment.

Young children’s embodied learning within MR environments

Numerous studies demonstrate the potential of mixed-reality environments to support children’s learning (Danish et al., 2015; Lindgren & Moshell, 2011; Malinverni & Burguès, 2015). Moreover, studies show that learning gains are higher when children use MR environments than when they use traditional curricula (Johnson-Glenberg et al., 2011a) or comparable desktop computer simulations (Lindgren & Moshell, 2011). Johnson-Glenberg et al. (2011b) highlight three reasons for these higher learning gains: (1) embodied learning in these mixed-reality environments activates multiple sensorimotor systems, which create more stable memory traces and knowledge representations; (2) these learning environments support collaborative learning; and (3) language and gameplay act as mediators that contribute to higher levels of motivation and on-task learning. Embodied resources become “objects to think with” (Malinverni et al., 2016) as children use embodied resources to communicate and facilitate intersubjective meaning–making (Davidsen & Ryberg, 2017; Steier et al., 2019), to scaffold knowledge building (Davidsen & Ryberg, 2017) by “grounding” abstract concepts (Lindgren et al., 2016a), and to “shepherd” and instruct each other (Davidsen & Ryberg, 2017).

Despite a growing understanding of how students use embodiment as a resource for learning, there is still a gap in capturing students’ embodied interactive learning outcomes within the computer-supported environment. Given the importance of embodiment in these environments, it may be crucial to develop assessments that are similarly immersive and embodied to mirror the kinds of interactions that learners engage in while learning (Lindgren & Johnson-Glenberg, 2013). Without these kinds of assessments, we may fail to recognize the richness of children’s learning in embodied environments. For example, a prior analysis of learning within the STEP environment suggested that multimodal interviews offer students alternative ways to express their understanding that align well with the embodied activities that are core to the learning designs and help them to express deeper understanding than may be visible in more traditional interview questions (Saleh et al., 2015; Tu et al., 2022). This suggests that embodiment can offer a broader picture of what concepts students understand and how deeply they understand them. Indeed, Lindgren et al. (2016b) found that prompting students to use gestures during science content interviews elicited more mechanistic reasoning. That said, questions remain about reliably drawing out students’ embodied understandings in their learning outcomes.

Thus, when we analyze students’ embodied learning within the MR environment, we focus on their collective and individual communicative acts. We aim to explore how body motion helps students to represent their understanding of target science concepts better and how they collectively developed this understanding along with the ways of communicating it through their embodied learning within an MR environment. We, therefore, aim to advance both theoretical and pragmatic ideas about making sense of students’ collaborative embodied learning within an MR environment. This analysis is part of a larger design-based research (DBR) study (DBR Collective, 2003) in which we have tested and refined the design of the STEP MR environment for young learners over the course of the past 10 years (Danish et al., 2020). Throughout the lifecycle of the overall project, we have adapted our designs for collaborative embodied learning for different classrooms, different science topics, and in response to the needs of different teachers and students. Each subcycle within the project has offered theoretical insights regarding how to better support reflection, collaboration, scientific inquiry, and modeling practices within embodied classroom activities. The present analysis reports on the sixth meso-cycle (McKenney & Reeves, 2012) within the larger project as a part of the evaluation and reflection phase of this cycle. While previous findings have offered significant evidence for the benefits of an embodiment for young science learners (Danish et al., 2015, 2020; Tu et al., 2019), the current paper narrows in on the mechanisms that are hypothesized to effectively facilitate students’ learning in these spaces: the unique communicative opportunities offered by both individual and collective embodiment.

Method

Design of STEP

The STEP technology setup

The study was part of the Science through Technology Enhanced Play (STEP) project and builds upon the STEP software platform (Danish et al., 2020, funding no. 1628918). The STEP platform was designed and developed in collaboration with the Inquirium software development team. Each iteration of the design included iterative refinement of the design based on implementations with learners to see how they engaged with the embodied interface. From a technical perspective, the STEP platform utilizes the Open PTrack person tracking system to monitor student movement via Kinect cameras (Open PTrack, http://openptrack.org/). This system identifies participants’ center of mass and reports their location to the STEP environment, which is a web-based application developed in JavasScript and uses node.js for the server. This combination of technologies made it easy to deploy the environment on a networked laptop within the classroom, which was then projected onto a screen in the front of the room where all the students could view it.

Particle simulation and curriculum

For the STEP Particle curriculum, when students move in the tracking space, they are shown as colored circles that represent particles in the projected simulation. These particles are linked to their nearest neighbor via color-coded lines representing some of the bonds between particles. When the particles are behaving like a solid, they are purple, as a liquid they are blue, and as a gas they are orange. The proportion of particles in each state is also displayed on the side of the screen as a series of linked bar graphs (see Fig. 1). The target learning content involves understanding and demonstrating particle behaviors in the three states of matter as well as the connections between microscopic and macroscopic characteristics of matter. For particle behaviors, we focused on four characteristics: speed, distance, attraction, and trajectory. To form a certain state of matter, students need to collectively adjust their relative speed and distance so that they can be recognized by the simulation as the target state. For example, to produce a solid, the students had to be relatively still, and yet slightly apart from each other. Therefore, STEP is designed to help young children explore states of matter through collective whole-body movement. While the facilitators often encouraged students to explain how they were moving or planning to move, the gestures that emerged in the activity were student generated and were not prescribed by the simulation or the facilitators.

Fig. 1
figure 1

STEP technology: as students act as particles in the STEP tracking space (left), the STEP interface, which was projected on a shared screen (right), represents their interactions as particles and displays a dynamic state meter

Full size image

Data sources

Participants and procedure

We invited all members of a collaborating first- and second-grade mixed-age classroom in a Mid-Western elementary school in the USA to participate in this study. Twenty-two students in the classroom assented to participate, and their parents also signed the consent form for this study. Students participated in a seven session STEP activity, with each session consisting of 30 min of activity in the STEP environment (see Table 1). Students participated in a multiple-choice pre-test and a one-on-one interview the week before the project started. An individual post-interview and the multiple-choice post-test followed after the class engaged with all STEP activities.

Table 1 Description of STEP activities
Full size table

Instruments

To assess students’ learning, we designed two instruments. The instrument and coding scheme demonstrate different ways of understanding the particulate nature of matter and build on the questions and coding schemes used in the prior STEP projects (Davis et al., 2019). In Davis and their colleagues’ work, they have developed a coding scheme to measure students’ understanding of particle behavior with early iteration of the STEP environment. The original coding scheme was built on work from Paik et al. (2004) to depict different ways of understanding states of matter at the micro and macro levels and understanding what leads to the changes in states of matter. We developed two categories: (1) the MT codes, which focused on the characteristics of particle behavior in a certain state and (2) the MMC codes, which focused on the microscopic particle behavior and macro state. The coding scheme is summarized below and in Table 2.

Table 2 Codes applied to interviews
Full size table

Multiple choice questionnaire

We designed a 14-item multiple-choice questionnaire based on our curriculum that covers (1) macro-state changes and (2) the microscopic behavior of particles in response to macro-state changes. We have listed a few sample questions from our multiple-choice questionnaire in Table 3.

Table 3 Example questions from multiple choice questionnaire
Full size table

Interview protocol

In addition to the multiple-choice test, we designed an interview with 26 questions. The interview questions aligned with the target content measured in the multiple-choice questionnaire. In the interview, we also included interactive props such as ice cubes, containers of water, and pretend “hot plates” (i.e., pieces of colored felt) to help students visualize the content of the questions being asked. For example, in the multiple-choice questionnaire, we asked students: “When water evaporates into steam, what happens to the particles?” We measured the same concept in the interview by putting water in a container and then asking students how the water particles would move if we added heat to the container. In the interview, we encouraged students to use body movements and gestures to demonstrate particle behaviors. Thus, both questions assessed students’ understanding of how particles move when the macro state changes, but when contrasted the interviews can help us understand how the inclusion of materials and embodiment may help learners communicate additional nuance. We video-recorded all classroom activities and interviews for analysis.

Data analysis

To answer our first two research questions, we analyzed the pre- and post-multiple-choice questionnaires and interview data. To examine whether students’ overall understanding of particle behavior improved after they engaged in STEP activities, we conducted a paired t-test to compare the aggregate of students’ scores on pre-and post-multiple-choice questionnaires. Specifically, we assigned one point to each correctly answered item of the 14 question multiple choice test, then added up students’ total scores on the pre-test and post-test and conducted the paired t-test with these aggregate scores. Next, we identified the two most difficult multiple-choice questions as those with the lowest numbers of correct responses. These were chosen to help highlight whether and how the interview context helped students to communicate an understanding of the same concepts despite their difficulty in answering on the multiple-choice test.

To analyze students’ performance in the one-on-one interviews, we coded both pre- and post-interviews using the coding scheme listed in Table 2. We segmented each question into one quotation in Atlas.ti 8 and applied the MT or MMC codes to each quotation. For MT codes, since these represent one characteristic of particle behavior, coders were allowed to assign up to four codes to the same question. For MMC codes, coders only assign one code to each question. To calculate the interrater reliability (IRR), the lead author randomly picked four videos from pre- and post-interviews, then three research assistants coded all four videos individually. The agreement among the three research assistants was 81% of all codes applied across the four interviews. Then we conducted interaction analysis to examine how students use a combination of verbal responses and non-verbal motions to answer challenging questions during target interview questions—those that covered the same content that were identified as the most challenging in the multiple-choice instrument. In the target interview questions, students were shown manipulatives and asked (1) how water sitting on a hot plate would change as it heats up and (2) how these changes would impact particle behaviors. We coded students’ responses for evidence of understanding the target concepts and counted the codes applied to each student’s answer to each question. Then we compared the percentages of each code for the same question between the pre-interview and post-interview. Then three researchers did another round of analysis on students who answered those questions correctly (as evidenced by at least one code being assigned) with a particular focus on how embodiment during the interview supported their verbal explanations.

We next worked to answer the RQ3: How do students’ embodied interactions in the classroom activities utilizing the STEP MR environment build on and lead to the development of shared ways of communicating their ideas about states of matter across contexts? We conducted an interaction analysis (IA; Jordan & Henderson, 1995) to explore the role of body motion in helping students to refine and communicate their emergent understanding during the lessons. We first content-logged all videos of classroom activities. Next, the lead author selected several video clips for further analysis that focused on students exploring how microscopic changes of characteristics of particle behavior such as distance between particles may change the macro states of matter. We selected these moments because these topics were identified as challenging concepts when we analyzed the pre- and post-assessment data to answer the prior research questions. Then four researchers rewatched the selected video data and focused on communicative acts, including whole-body movements and gestures. For each data session, the authors watched one clip. We collaboratively analyzed the functions of these communicative acts at both collective and individual levels and explored how each communicative act helps students collectively develop their understanding of the target science concepts. Through multiple rounds of collaborative video analysis, we finalized several representative video clips to illustrate the collective and individual functions of communicative acts within STEP.

Findings

RQ1: How do students communicate differently when asked about states of matter in multiple-choice written tests and when they are asked to demonstrate their understanding in an interview?

To answer our first research question, we first ran a paired t-test on the pre-and post- multiple-choice tests. We found there was a significant improvement from the pre-test score (M = 5.5, SD = 2.24) to the post-test score (M = 11.2, SD = 2.32), t (21) =  − 8.72, p < 0.01, which showed that students’ overall understanding of the target content improved after they engaged with the STEP environment. Given these learning gains on multiple-choice questions, we also wanted to know whether more embodied measures reveal added nuance about their learning and/or additional learning that occurred within the embodied learning environment. Therefore, we further analyzed students’ interview data.

Our descriptive statistics showed two multiple-choice questions (Q11 and Q13, listed in Table 4) that students had difficulty answering. These questions asked how particle behaviors would change to reflect the changes at the macroscopic level. For each question, only two students were able to answer correctly in the post-test. However, in the post-interview, 12 students demonstrated their understanding of the same concepts. Therefore, we analyzed interview questions 13 and 15 (listed in Table 4), which assessed similar concepts to the difficult multiple-choice questions regarding how changes in energy level impact particle behaviors. We found that 18 students were able to accurately answer the questions in post-interviews and describe at least one characteristic of the particle behavior compared with pre-interviews. For question 13, 18 students were able to describe how changes in energy level affect macro states of matter and characteristics of particle behavior in terms of speed, distance, attraction, or trajectory. For question 15, we found that despite most students’ inability to answer the question correctly on the multiple-choice assessment, 13 of the 21 students were able to elaborate on the mechanisms of particle behavior. Table 4 presents the number of students who answered the question with mechanism codes. Students’ greater ability to answer interview questions correctly led us to investigate research question 2, to analyze video records of the interviews to tease out the factors that help students express their understanding in this context. We hypothesized that the freedom and encouragement to gesture/embody ideas would provide new opportunities for students to articulate their understanding with added nuance that was not present in their verbal answers.

Table 4 Numbers of students who correctly answered questions about macro–micro changes in post-multiple choice questions and post-interviews
Full size table

RQ2: How do students use embodiment to help represent the characteristics of particles within different states of matter in an interview context?

To examine how gestures and full-body movements help young children to demonstrate their understanding, we completed two analytic passes of the interview data. In the first pass, we coded each question to indicate simply whether the learner used their body in answering the question or not. We then conducted a preliminary analysis and found that for the interview questions, 15 of the 18 students who answered the questions correctly used gestures to represent the particles’ behavior. This suggested there was value in unpacking how they used these gestures, therefore we completed a second analytic pass. In the second pass, we used interaction analysis to explore how those students used embodiment to represent their ideas. We found that they used their bodies primarily to add nuance when communicating particle movement, speed, and distance. For example, in Fig. 2, we show two students using gestures to represent state change at the microscopic level, they were not able to answer the question correctly in the multiple-choice questions. In Fig. 2a, when the student was asked how particles behave when the plate is really hot, she quickly moved her fingers in the water and explained that the particles “move faster and faster.” In Fig. 2b, when another student was asked the same question, she moved her hands up to represent how liquid water evaporates to gas when particles get energy. These examples suggest that the inclusion of gestures is valuable for many of the participants to communicate aspects of their understanding.

Fig. 2
figure 2

Students’ gestures for microscopic changes during interviews. Note. A student moves her fingers in the water container to represent particles’ speed (a), and another student moves her hands up to represent how liquid evaporates when getting energy (b)

Full size image

We next present a segment from one post-interview to demonstrate how students used gestures to represent particle behaviors. Lily is one of the students who was not able to answer the multiple-choice questions correctly and yet was able to accurately describe the changes in particle behavior during the interview. In her interview, the researcher asked Lily about the changes in macro states of matter and “what will happen to the particles in the water” as the water on the hot plate evaporated and turned to gas. To answer the questions, Lily described changes in the distance between particles and the speed of particles with gestures (Table 5, line 4). She explained, “They will go further apart, and go faster and have more energy,” while pulling her hands far apart horizontally to show the increasing distance between particles in liquid as they get more energy (Fig. a in Table 5) and then demonstrating the speed of particles by making her hands into fists and moving them quickly in the air (Fig. b in Table 5). When asked, “Why do you think the particles would do that?” she used the same gesture to describe the distance between particles as farther apart (Fig. a). In her responses, Lily’s gestures demonstrated certain characteristics of the particles’ behavior (distance and speed). Notably, she could also verbally describe the changes in particles’ speed, which is the concept she failed to answer in multiple-choice tests. This pattern of gestures and embodiment supporting more elaborate descriptions is consistent across the interviews analyzed.

Table 5 An example excerpt from a student’s interview
Full size table

Summary

Taken together, these findings suggest that embodiment helped students to explain their understanding in an assessment context. When students were asked challenging questions, gestures played an important role in students demonstrating the particle behaviors and giving richer and more detailed verbal descriptions of changes in states. Our analysis reveals that the gestures students used in the post-interview aligned with their embodiment developed within the STEP environment. To illustrate how students develop their understanding through embodied learning, we now present our analysis of students’ communicative acts that help them collectively construct their understanding of particles during classroom activities.

RQ3: How do students’ embodied interactions in the classroom activities utilizing the STEP MR environment build on and lead to the development of shared ways of communicating their ideas about states of matter across contexts?

To answer our third research question, we conducted embodied interaction analysis (Davidsen & Ryberg, 2017; Streeck et al., 2011) to explore how students’ embodiment helped them to form their conceptual understanding in a classroom activity. We iteratively identified specific communicative acts that involved embodiment as a way of clarifying the broader patterns in how the learners used their embodied resources and as a way of grouping superficially different embodied or gestural moves together based on the role they appeared to play in collaborative interactions. We have listed the three types of communicative acts that we identified in Table 6, along with an explanation of the collective and individual functions played by each. After the initial list was developed, the research team met to refine the acts and the communicative functions they appeared to play individually and collectively through an iterative review of the video data. The team also reviewed the data to look for the presence of these communicative functions across different interactional contexts: during the STEP embodied activities, during debriefing in class, and during students’ post-interviews.

Table 6 Description of communicative acts
Full size table

Gestures to show particles’ behaviors

Across the seven activities and interviews, we found the first type of communicative act was one in which students used gestures to demonstrate what they knew about particle behavior. We found that these gestures occurred across different interactional contexts. First, when students engaged in the STEP environment, the facilitators often asked them to explain what they were doing or how they were moving to create a particular state in the STEP. The students used gestures to supplement their answers to these questions, such as in Fig. 3a, when a student gestured to demonstrate how they were collectively moving and the differences in speed that they found to be necessary to make either solid, liquid, or gas. In this instance, the student highlighted in the circle held his palm out and waved it back and forth, slowly and then faster, to demonstrate how the particles needed to speed up in order to create gas. These sorts of explanatory gestures seemed to appear when verbal explanations were not initially understood by the facilitators; in this example, the students noted they were moving at “normal speed” and then followed up with gestures when the facilitator questioned what “normal” meant.

Fig. 3
figure 3

Students’ use of gestures to show particle behavior across activities. Note. Students used gestures to show particle behavior while modeling within STEP (a), during debriefing (b), and in post-interviews (c)

Full size image

Second, we found that students showed gestures during the planning and debriefing phases of the activity in order to represent the particles’ characteristics, such as attraction, distance, and speed, to the facilitators and their peers. For example, in Fig. 3b, students were reviewing particle attraction when they were asked what attraction looks like if the particles are far away. The girl in the black-sleeved shirt gestured the distance between particles, then talked about how the attraction between particles would be weak if particles were far away. These gestures were used by students during debrief discussions multiple times across the 7 day curriculum.

Third, as presented in our interview analysis, students use gestures to demonstrate particle movements in the post-interview (Fig. 3c). In the interview setting, students also used gestures with their verbal explanations to represent the characteristics of particles. Such movements helped students to better demonstrate their understanding in an assessment context.

Across these three contexts, the gestures about particle behaviors seemed to serve a shared purpose: to communicate with others what a student understood and to add depth to that communication (or repair it) to help others more fully understand the ideas. In addition to these collective functions, we also saw these gestures serve an important individual function for the students to represent their own ideas in addition to the verbal description.

Guiding and directing peers’ movement in collective activities

In addition, as STEP was designed to support students’ collective embodied learning, we found that another important role of communicative acts is guiding or directing peers’ movements. These sorts of acts appear when students interact with each other within the STEP environment. The guiding gestures were used by (1) students who were tracked as particles within STEP and (2) nontracked peers who pretended to change the energy level. First, students must collectively adjust their relative speed and distance when they embody particles in STEP. Thus, guiding gestures suggesting students adjust their relative movement are found on most days of the STEP curriculum. For example, in Fig. 4a, students explored how liquid particles move. The boy in the striped shirt pointed to his peers and also gave out suggestions, such as “you stay” and “you move slower.” Such guiding gestures help the embodied group to adjust their collective movement.

Fig. 4
figure 4

Students’ use of gestures to guide peers. Note. A student in the embodied group uses a guiding gesture to help his peers adjust their collective movement (a), students direct their peers in the embodied group to change their speed and distance with an arm scooping gesture that indicates taking energy away (b), and a student pulls his peers toward him to make the relative distance between particles in the simulation closer to collectively form the target state (c)

Full size image

Another type of directing movement was taken up by the nontracked students pretending to change the energy level of their tracked peers. During the day 4 activity, students were split into a particle group and an energy group. As the particle group embodied a certain state of matter within the STEP environment, the energy students pretended to give or take away energy from the particle group. Figure 4b shows a moment in which the energy group used an arm-scooping gesture to take away energy from the students embodying particles, which directed them to change their speed and distance. In this way, the energy group also experienced how changes in energy levels lead to changes in particle behavior.

We found using gestures to direct each other later in the curriculum when students embody particles and guide each other to form a certain state of matter within the STEP environment. On day 6, students had already engaged in several collective embodied activities and had some understanding of particle behavior. When they were acting as particles in different states of matter, they used gestures to direct others to adjust their movement to collectively form the target state. In Fig. 4c, the embodied particle team was trying to form a solid state. To accomplish this, the boy in the middle tried to pull his peers towards himself to make the relative distance between particles in the simulation closer.

The examples above show that guiding gestures appeared throughout the curriculum, from beginning to end. However, the underlying collective functions of the guiding gestures differed slightly at different points in the curriculum. At the beginning of the curriculum, students had little knowledge of how the particles move within the STEP environment; therefore, the guiding gestures, along with students’ verbal suggestions, helped them to explore how the particles move. Later, as students co-constructed more understanding, the guiding gesture to adjust their relative movement demonstrated their understanding of particle movements in target states.

Respond to suggestions from others

As communication is not only about transmitting information but also about how others receive the information, it is important to examine how students respond to other students in STEP. Another sort of communicative act we found helpful for students was one in which students responded to suggestions from others, including other students embodying particles, observers, and facilitators. This sort of communicative act showed how students received the information from others and then gradually constructed their own understanding. Within the STEP environment, we found two types of interactions students used most: (1) students responding to the guidance and prompts from their peers and adjusting their movement accordingly, and (2) students mimicing other students’ movements. Regarding the former type of interaction, in the example discussed above where the boy pulled his peers together to form a solid (Fig. 3c), his peer in the foreground responded by stepping back to adjust his relative distance. There were also other moments showing that students respond to the prompts and guiding movements of others to help them collectively form the target state and become recognized by STEP simulation.

We also found instances of students responding to peers by mimicking their movements. For example, on day 6, students collectively embodied particles in a liquid. At the beginning, the entire embodied particle group was running around and were not recognized as liquid by the STEP simulation (Fig. 5a). Then a girl developed a “jellyfish” movement, in which she moved her arms up and down slowly and also walked slowly in the tracking space (Fig. 5b, girl with blonde hair and black shirt). Then this movement was quickly picked up by the rest of the embodied particle group (Fig. 5c). With this “jellyfish” motion, students represented the medium speed and moved closer to form a liquid.

Fig. 5
figure 5

Students mimic peers’ movements. Note. As students are running in the STEP space (a), a girl in the middle starts the “jellyfish” movement (b), and then all other students pick up this motion to represent “middle speed” (c)

Full size image

Across the seven activities, we found such communicative acts helped students explore the particles’ behavior collectively by (1) adjusting their motion to respond to others and (2) reacting to guidance and suggestions. Then for the individual function, it also supports students to experience relative speed and distance between their peers and themselves and reconstruct their own representation.

Communicative acts in the MR environment

We have identified the most used communicative acts in STEP and discussed their collective and individual functions. However, these acts did not occur independently; rather, there were many moments when they overlapped within STEP activities. Here, we present an example of how these communicative acts help students understand liquid particle movements. On day 3, students engaged in a STEP activity to explore particle behaviors, particularly speed and distance, in three states of matter as the simulation displayed a blank tank with particles. The physical positions of students in the embodied group were projected as particles on the shared screen, and they needed to move collectively to form different states of matter. Meanwhile, observers sat outside of the tracking area, watched how the embodied particle group moved and provided suggestions to the embodied group. Students rotated groups so that each student had a chance to experience both roles.

We present a segment in which students collectively attempt to embody a liquid (see Fig. 6). The students entered the tracking space and started moving freely and trying different motions (e.g., running, standing still) to form a liquid, as indicated by the state meter in the simulation. However, students’ movements did not form a liquid. Then a girl, Lily (Fig. 6a, the girl in pink and purple shirt), looked at the screen and suggested to the researcher, “We need to…” with a gesture pulling her hands far apart, indicating her group should be farther apart. This movement represents her understanding of the distance between liquid particles and her suggestion to change their relative distance. The researcher encouraged her to show the gesture to her peers. Lily went back to the tracking area, pointed to the floor, and told her neighbor, “Stay away back” (Fig. 6b). Here, Lily gestured to guide the group to work collectively to adjust the relative distance between particles. The rest of the group adjusted their relative distance in response to Lily’s guidance. Even though the embodied particle group moved away from each other, they were not recognized as liquid particles in the simulation. At this moment, an observer outside of the tracking space pointed out that the embodied group was not moving. The embodied group started to move around but did not maintain their distance from one another, leading to the state meter showing multiple different states of matter. Then Lily suggested, “Why don’t we move around and still keep some space?” (Fig. 6c). At this moment, Lily suggested that her team collectively adjust their distance and speed. In response, the embodied group walked in one direction to keep relatively the same speed and amount of space. Finally, the group was recognized as a liquid by the state meter (Fig. 6d).

Fig. 6
figure 6

Students’ collective embodiment of liquid particles in STEP

Full size image

In summary, we present this episode as an example of how students collectively develop their understanding of particle behavior through embodiment within the MR environment. In this episode, we found that embodied action helps students to (1) demonstrate their ideas to others, (2) guide students to adjust their relative distance, and (3) respond to the directions and prompts of others. As students failed to represent liquid particles at the beginning, some students used gestures to demonstrate their ideas and try to plan the collective embodiment. The students also used guiding movements, such as pointing to the floor and directing their peers’ movements. As other students adjusted their relative speed and distance based on the guiding gestures and suggestions from the embodied particle group and observers’ group, they finally formed liquid. Their understanding of particle behavior in a liquid is formed through multiple communicative acts.

Overall, we analyzed the communicative acts that students made in the STEP environment and how they used their experience embodying particles in the STEP environment to demonstrate the particles in a post-interview context. We found that the embodied experience within MR environment offers students ways to collectively explore complex science concepts and helps them develop and demonstrate their understanding within a collaborative technology-enhanced learning environment.

Discussion

In this paper, we identified the collective and individual functions of students’ communicative acts as they embodied concepts when they were learning. Building on the LEAF framework, we further explored how these communicative acts mediate students’ understanding of the target science concepts within STEP and an interview context. We found that such communicative acts play an important role in supporting young children in collectively exploring the characteristics of particles and representing their ideas. Furthermore, these communicative acts often leverage embodiment as a resource for sensemaking.

Embodiment and communication: Supporting collaborative learning within the MR environment

Former studies on embodiment suggested that embodiment can help students learn science concepts within a mixed-reality environment (Mathayas et al., 2019). CHAT theory (Engstrom, 1987) argues that learning occurs in activities and is oriented toward a set of goals. The LEAF framework further explores learning from a sociocultural perspective (LEAF, Danish et al., 2020). It offers a theoretical perspective of embodied learning, that is, to understand embodied activity simultaneously at both individual and collective levels. In our study, we covered different stages of the activities and explored how the goal of communicative acts shifts across these stages. Our work focused on the unit of students’ communication within the STEP environment and showed concrete examples of how communicative acts shift from classroom activities to the interview context. We found that the communicative acts supported students’ collective modeling in different ways: (1) at the beginning, students use gestures or whole-body movement to collectively explore the target content, (2) when students develop more nuanced conceptions of particles, they use gestures to demonstrate their ideas, and (3) in the post-interview, gestures serve as a useful tool to represent particles’ behavior.

As the LEAF framework argued that the collective and individual levels of embodied experience are not parallel but intertwined, we furthered our analysis of communicative acts. We provided a detailed analysis of how the individual and collective functions of communication help students collectively construct their understanding of states of matter. In studies of communication in CSCL contexts, there used to be communicators who shared the information and recipients who received and proceeded with the information (Herrmann & Kienle, 2008). We found that embodied communicative acts help students act as both communicators and recipients. As students use their bodies to represent ideas and guide others, they also respond to the prompts of others and reconstruct their representations. In addition, we view STEP as a dynamic collective activity system, and we found that students simultaneously enact their roles of communicator and recipient. For example, when students respond to others by changing their speed or mimicking others’ movements, they receive the information and pick up the suggestion from other students as the recipient. Then their embodiment is also sharing their ideas with other students in the classroom simultaneously.

These findings suggest that it is valuable for teachers, designers, and researchers to explicitly plan to (1) design, (2) encourage, and (3) discuss the communicative role of gestures and embodiment within their embodied and MR environments. We elaborate on these briefly below.

Design

When we refer to design here, we are suggesting that it is helpful to identify (a) functions of embodiment, (b) specific embodiments, and/or (c) opportunities to create embodiments that align with the target content. Designing functions of embodiments is the process of identifying the role that we hope embodiment will play in sensemaking in our designs. This suggests we begin the design process by asking ourselves what we hope the learners will use their bodies to communicate or accomplish in interaction? Is our hope to show energy, for example, or movement? If our hope is to show energy, we might think differently about how we encourage students to gesture than if our hope is that they will show speed; for example, we might identify prompts such as “How will we show that particles get energy from the heat?” or “How will they know how fast to move?” respectively. From a CHAT perspective, this also does not involve simply providing prompts, but rather reorganizing the activity to make these specific embodiments relevant and useful. For example, if learners need to coordinate giving energy to each other at a distance (i.e., one is the heat source and one is the water particle) then they are more likely to spontaneously develop and use a gesture for that role, especially if they are discouraged from using talk.

Once we have identified the function of learners’ embodiments, we can then decide whether we want to suggest a specific embodiment (b) or provide an opportunity to create one (c). In some cases, we have found that we have specific embodiments in mind to convey particular aspects of the information to peers or the software environment. For example, in earlier work, we explored how pointing is useful for conveying the direction of a force, and then because it does not convey magnitude effectively it can prompt a productive discussion of how to accomplish that (Enyedy et al., 2015). This leads us to option (c), which is that there are times when it is valuable to have learners discuss and debate shared gestures or embodiments so that they get on the same page. The added benefit is that this makes their reasoning public, and thus open to discussion and debate. This also provides a space for more clearly developing and articulating one’s ideas than some of the spontaneous gesture creations did above. For example, we have asked students at times how to show they are part of the ice. Early in our studies they often try and clump up and hug each other to convey the nonmoving rigid nature of ice. However, as they come to appreciate the fact that ice particles sit at a greater distance from each other than liquid, they revise their embodiment to include holding hands to position themselves appropriately far away.

Encourage

We find that because embodied learning environments are often both unfamiliar and chaotic, teachers tend to focus on high-level conceptual issues (e.g., what state are we in?) or low-level classroom management issues (e.g., do not run too fast!). This means there is value in helping them see the value of embodiment in communicating ideas here so that they can support and encourage students in using embodied productively. Thus, they might not only point out a student’s idea (e.g., “gas particles have the energy”) but also how they conveyed it (e.g., the girl in Fig. 3b used her hands to show lots of energy). Furthermore, if facilitators can promote discussions of how and why specific embodiments are productive for learners, this will validate these efforts and encourage learners to discuss their embodiments in these ways.

Discuss

Using one’s body to make sense of the world is quite intuitive. Nonetheless, learners spend a lot of time in school being told not to use their bodies and to sit still. Therefore, we find it helpful to encourage students to not only use their bodies but to be intentional and reflective about how using their bodies can help them explore the concepts they are studying. As our results indicate, being given this freedom is crucial for the articulation of their ideas. We suspect that the more students recognize this, the more likely they are to try this or to ask for opportunities to embody their ideas. Students in studies such as this one are routinely asked to show ideas that are complicated, and this is something we want to encourage. Future work can further explore how students understand these ideas, and how their understanding can support their sensemaking efforts.

While we believe these steps are valuable to designers, researchers, and teachers, we also recommend that researchers aim to articulate these in advance as a conjecture (Danish et al., 2020; Sandoval, 2014), so that they can vet their success over the course of implementation.

Embodied interview: A way to capture students’ learning within a collaborative MR environment

Previous studies (Johnson-Glenberg et al., 2011a; Saleh et al., 2015; Tu et al., 2022) have shown how including opportunities for embodiment in an assessment context can better capture students’ embodied learning in a computer-supported learning environment. In our paper, we analyzed both interviews and classroom activities. Across this analysis, we found that body motion is important in supporting young children in representing their understanding of the target science concepts. As shown in the findings, students also used a combination of talk and gesture to present their understanding to the interviewer. In addition to the interview analysis, we also found that these communicative acts were consistent from an early stage of the activities to their individual assessment or the post-interview in our study. The interaction analysis provides solid evidence that such communicative acts within an MR environment are helpful for students in constructing their understanding of target science concepts.

Future work might also explore whether including opportunities for embodiment in assessments, even when embodied learning is not a core instruction mechanism, may still provide students with valuable opportunities to articulate their emergent thinking. Another direction for future research would be to explore instructional support that helps students re-represent embodied ideas without the body for traditional modes of assessment, such as multiple-choice tests. This may be especially important given the widespread and persistent use of traditional approaches to assessment. Relevant to either direction, we suggest assessment instrument designers encourage students to use embodiment to demonstrate their understanding of target concepts and consider adding physical props as conceptual tools to help students represent complex ideas.

For teachers, there are two ways they might build on these recommendations for formative and summative assessment. For formative assessment, it may prove helpful to remember to encourage learners to answer questions and demonstrate ideas with their bodies or gesture to help the teacher understand what they are attempting to communicate. This will also serve to further help students recognize the value of embodiment and gesture. For summative assessment, the same is true. However, it may be helpful to think about how to prepare students to articulate their embodied ideas in talk or writing before the assessment is given, which might aid students in making these connections. While our results show deep understanding in the interviews, the multiple-choice answers suggest that students have not yet found ways to articulate these ideas fully without the body. Future work can help them explore this challenging transition as well.

Using debrief discussions to scaffold students’ understanding of and ability to communicate target science concepts

The LEAF framework (Danish et al., 2020) highlights the importance of mediators such as the community of learners participating in the activity, whose collective interactions can influence how the group achieves its goals. Thus, one design principle of collective embodied activities is to have discussions and let students reach a consensus to understand the key concepts in addition to their initial collective embodied exploration of target science concepts. In this paper, we found that the debriefing time during STEP is where students often used gestures to demonstrate their understanding. These gestures were similar to what they used to explain certain ideas in the post-assessment context. Thus, in addition to constructing their understanding, they are learning how to communicate that understanding during the debriefing time. Moreover, Gómez et al. (2013) discussed the importance of mediating students’ experiences using CSCL and learning to articulate their understanding. We found that facilitators’ prompts in response to students’ embodiment plays an important role in helping students verbalize their embodied learning. The mediation was shown both in the STEP activities and interview. When students use body movement to demonstrate ideas or plan how to move collectively, the teacher verbalizes their movement and then uses the scientific term to recapture students’ thinking. In this way, the teachers help notice and highlight students’ gestures, which also helps students to recognize the key features of particle behavior. For future studies, it will be important to consider how to support the development of these gestures more intentionally, either during the embodied activity or during the debriefing.

Limitations

The design of this study leverages a design-based research approach (DBR, Anderson & Shattuck, 2012). Therefore, we focused on how the collective modeling practice happened within the MR environment rather than experimentally contrasting students’ performance with those who did not participate in such an environment. However, the interactional complexity above suggests that focusing so narrowly on the presence or absence of embodiment is likely problematic given the ways that embodiment might or might not be taken up and supported by participants. In addition, due to the advanced nature of the technology, we only have a limited number of participants in this study. We hope future studies can expand the design of collaborative learning within the MR environment and support students on a larger scale. The current qualitative analysis focuses on the communicative acts of a group of students exploring together. Still, much can also be explored in terms of the individual functions of embodiment and gesture for students’ understanding and sensemaking. It is likely that the gestures students used in our study to share their ideas also had important influences on their development of ideas over the course of the unit, though this would require reorganizing the study somewhat to disentangle students’ embodiments from the interaction. However, such a move would likely make the findings less relevant in classroom contexts. Fortunately, numerous other studies focus on the individual impacts of embodiment on learner understanding (Lindgren & Johnson-Glenberg, 2013; Lindgren, 2014), and thus we chose to focus on collective and communicative layers of embodied activity to complement that work. While we have attempted to weave these layers together in our analysis, and we can certainly incorporate more individual measures in future studies to make this more salient.

Conclusions

This paper demonstrates the fundamental importance of attending to the communicative role of embodiment in embodied and MR learning environments. Prior work has clearly demonstrated the value of having learners use their bodies either individually or collectively. It is often implicit that learners need to communicate with and about this process. This work aims to advance the conversation by explicitly centering this communicative role for embodiment and thus highlight the many ways we can and should support, encourage, and attend to these communicative functions. Furthermore, as embodied learning increasingly comes to contexts such as classrooms that are inherently social, interactional spaces, in contrast with laboratory experiments that may be less so, we need to focus as a field on how we can attend to this communicative aspect while also continuing to explore how individual learners can leverage their bodies to support cognition and learning.