Mixed Reality based Collaboration for Design Processes

2.1 Collaboration Using AR and VR

Figure 3

Mixed-presence localization by Billinghurst et al. [1] within the space-time taxonomy of Johansen et al. [18].

In remote collaboration, a distinction must be made regarding actor roles, which should not only be divided into those of the presenter and the spectator, but must allow dynamic role changes [1]. Isenberg et al. [15] propose to characterize the user’s current behavior by three different levels of engagement:

1. Viewing: passively consuming content presented by others

2. Interacting/Exploring: actively changing views, move within the virtual space, explore the content

3. Sharing/Creating: manipulate (create, retrieve, update, delete) data and share with other users

1. work allocation and division

2. awareness & common ground

3. (spatial) references, and deictic expressions

4. incentives and engagement

5. identity, trust and reputation

6. group dynamics

7. consensus and decision making

2.2 Interaction in Distributed Scenarios

Distributed collaboration offers the potential of interacting with different modalities. A commonly described use case is the gesture-based projection of a remote expert to work on a physical task with local workers [30], [12], [9]. Thus, there is no equivalent collaboration between these two actors as information transmission and communication is mainly directed. For this purpose, the adaptation of perspective change to objects for traceability and mutual understanding is necessary [8]. Possible input methods for this are gaze control, gesture input by means of controllers or the use of tangibles [27]. Technology influences the way of interaction [10], thus, the way in which communication between actors takes place in a synchronous collaboration.

To understand the process of collaborative work, Isenberg et al. [16] emphasized that individual and group-related efforts must interleave and thus, identified different collaboration styles. Meaning that besides group activities, individual activities should also be supported, and this transition is best carried out smoothly. Whereas Okada et al. [24] have identified several hierarchical layers in terms of virtual collaboration, in which the importance of communication is highlighted, as this influences or moreover, alone achieves collaboration.

To stress the importance of communication, the level of collaboration ranges from the coordination of individual tasks to group-relevant work in collaboration. As a basis, McGrath et al. [20] organized all group-related tasks into four main goals, called the circumplex model, and divided them into further categories (see Figure 4). It differs between classes of categories that reflect basic tasks (generate, choose, negotiate, execute). Thus, the tasks are related to one another in with two-dimensional attributes reflected by the horizontal and vertical axes, for instance cooperational task as well as tasks that often lead to conflict between actors.

Figure 4

Group task typology by McGrath et al. [20] with four quadrants and the corresponding task types.

The main tasks presented by [20] can also be found in the design-thinking model by [26], which was then modified by [19], originally emerging from the software development life cycle (see Figure 4). It describes the development within a four-square model, which should be run in an optimal way, always with the person at the center of the development work. These four steps are important categories of product development. In the end, finding suitable solutions collaboratively will always be based on decision-making.

2.3 Work Spaces & Shared Visualizations

The last aspect targets different features of work spaces. In addition to technological challenges, such as synchronizing work spaces with different properties in terms of size and shape of physical spaces [5], [28], the visual context of the actors must match these individual physical environments [22]. The use of heterogeneous display technologies for Mixed Reality collaboration poses further challenges in linking different forms of virtual spaces [21] and performing spatial tasks in a common scope [27]. Usually actors have different roles that can perform different tasks. These different roles may be integrated by different immersive technologies as well as mixed spaces.

Since our use case consists of an actor who discusses his or her findings with stakeholders, the role ratio can be described as a one-to-many format. On the one hand, this role relationship describes how cooperation is performed together, but is not intended to exhaust it [16]. As a basis for our proceeding considerations, Jankovic et al. [17] identified three important roles in the automotive context that contribute to decision-making in collaborative collaboration: the collaborative decision-making pilot who leads the decision-making process; the decision makers who are members of the project team; and the contributors, who bring important aspects into the decision-making process, but have no decision-making power themselves. Communication between the individual roles is important for each task. These roles must be embedded in the system to support the development process according to the design-thinking model. Moreover, all stakeholders must be identified and involved before assigning them to specific roles.

4.1 Design Process & Methods

Based on focus group interviews regarding the existing tablet-based AR-application, which focused on a co-located scenario, in which the supplier has to travel to the manufacturer to discuss issues about specific components (see Section 3), we started analyzing the problem space (see Figure 5). After the requirement analysis we developed some concepts to investigate the different possibilities for the use of virtual and augmented reality techniques to facilitate the collaboration between distributed teams.

Figure 5

Iterative design process: based on focus group interviews, we analyzed the problem space and developed prototypes for evaluating possible approaches in two iterations. The methodology is inspired by the double-diamond design process [6], [23], extended with rapid prototyping in the first design phases (Discover/Define) for early feedback and evaluation.

We chose an iterative design approach, loosely adapted from the double-diamond design process [6], [23], employing rapid prototyping approaches to showcase first ideas and evaluate them with domain experts. The resulting two prototypes will be discussed in more detail in the following section. We performed a qualitative evaluation by conducting unstructured expert interviews for each prototype.

We observed the users while interacting with the prototypes and asked questions afterwards and collected comments while they were using the system. We chose this evaluation approach, as we wanted to gather a wide variety of ideas, criticism and statements about possible extensions or refinements as well as feedback about the overall impression of the different concepts. By not constraining the interview to specific questions, we didn’t push the users into specific directions, but gave them the opportunity to talk freely about the presented ideas and concepts. This reflects the early stage of the prototypes, which demonstrate only key concepts. The user group for the first prototype consisted exclusively of experts from the automotive QA department of the manufacturer. The feedback gathered was analyzed and served as a starting point for additional concepts and extensions which were included in the second prototype.

This second evaluation was opened to domain experts from different disciplines, but still related to the automotive industry. This approach gave us the opportunity to investigate which concepts can be adapted to other, familiar use cases and processes. Both evaluations took place in conjunction with a technology congress, the participants were other exhibitors and visitors – engineers, designers and managers. Example questions for the second evaluation included:

1. What is your opinion about the concept of using AR for manufacturer’s perspective and VR on the supplier’s side?

2. Which advantages/disadvantages do you see in this use of the technologies?

3. Which recommendation do you have for improving/extending the scenario?

4. What are concerns that would prevent you from utilizing this system in your QA process?

5. Do you have remarks about specific differences regarding your own branch?

In the following section, we will briefly describe some of our key concepts of these showcases and their link to the underlying theories in the field of Collaborative Immersive Analytics. We focus on the mixed adoption of Augmented and Virtual Reality technology to investigate the benefits of each technology and to find reasonable configurations and identify use cases in which each technology fits best and to support each task regarding the circumplex model [20].

4.2 Iteration I (AR/AR): Combine HMD-based AR and Tablet-based AR

Our first iteration of the prototype focuses on a scenario, in which AR techniques are used for both the supplier and the manufacturer, meaning that both share a similar representation of the physical object. The supplier has provided a sample, which has been reviewed by the manufacturer in terms of build quality, materials, size accuracy and further aspects. The manufacturer now discusses issues with supplier, who has a physical duplicate on his remote working place. The reviewed part is located in front of the manufacturer. The supplier is wearing an AR HMD, which presents additional information, and the manufacturer presents his points on a tablet computer, which also shows an augmented view on the object (see Figure 1). In the following section we will describe the prototypes by first determining the collaborative role of the user and then describe core features, types of interaction and visualization modalities, as well as possible extensions discussed with the users.

This setup corresponds to the synchronous & distributed case in the space-time taxonomy of [15] (see Section 2), but could also be extended to be used asynchronously, where the supplier listens to a previously recorded session and independently browses through the associated data set. However, for this paper we concentrate on synchronous collaboration.

4.2.1 Prototype Description

When designing the showcase, we considered the aspect of asymmetric expertise and authority [32] for distributed collaboration, there is no guided tour – instead, the manufacturer and the supplier can independently access the information they need to understand the discussed issue. Another aspect is the concept of different views for each participant [1]: Each participant has the information needed to analyze and a personal view, which may be different, based on the role the participant incorporates in the given scenario. For example, the manufacturer needs to view measurement values to locate an issue and give an estimate about the severity of the issue, whereas the supplier views additional technical documents to the affected part to identify the cause for the issue and derive actions or adjustment of the manufacturing process to solve the issue.

Features (Tablet AR). The identified role model [15] for the manufacturer using the tablet is a mixture of interacting & exploring (browse issues and associated documents) and sharing & creating (communicate information about the issue with the supplier and document next steps). The core functionality for the manufacturer on the tablet therefore consist of the following:

1. browse issues and related details (rating, detailed photo, additional information)

2. display measurements/visualization overlays

3. view additional documents

4. take notes/protocol next steps

Interaction (Tablet AR). The interaction is based on touch input and a digital pencil for taking notes and protocol arrangements. Augmented reality is used to locate the information on the object and provide appropriate visualization using 3d overlays, spatially located measurements and labels as well as the option to display documents or photos in-place (see Figure 6, left). Using a tablet, it is also possible to use traditional forms for documenting actions in a standardized way (see Figure 6, right).

Figure 6

Augmented Reality is used to display 3d overlays and spatially located measurements. Left: Head-coupled AR provides a more intuitive way of showing the data. Right: Tablet-based AR uses traditional forms for documenting actions.

Besides the implemented functions, we also presented some possible conceptual extensions to the users, to get an impression, which of these options are feasible in the given use case. We chose this approach, as we intended the given showcase as an early mock-up, in which we wanted to investigate which features are working and which needs to be extended based on the provided feedback. The observations then influenced the implementation of the second showcase which will be described afterwards.

Extensions (Tablet AR). We proposed to provide additional input methods – pencil input to draw on the 3d model and speech recognition for documentation – and supplementary visualization options – the integration of intersections or technical drawings to augment the real object. As the collaboration functions in this showcase were quite limited, we also created concepts for enhanced collaborative functionality, such as the visualization of the viewport of the partner and highlighting functions to point the attention to specific parts of the object.

Features (HMD AR). The supplier wearing an HMD focuses on the role of interacting & exploring, by viewing different data sets and visualizations to understand the cause of the discussed issues. Therefore, the option to display visualization overlays, browse issues and access related details is available.

Interaction (HMD AR). Interaction with the HMD is, due to technical conditions, limited to gesture input and the visualization is focused on the augmentation of the real model with analytical 3d visualizations and spatially arranged labels.

Extensions (HMD AR). The conceptual extensions also included the use of speech recognition and the option to use a spatially tracked physical pen (or the finger) as input device to draw (virtually) on the object. Enhanced visualization techniques could include the display of measurement values and surface drawings. We proposed a voice record function for protocol findings and extended collaboration functions for viewing the position and viewport of the partner and specific pointing methods to highlight areas on the object surface.

4.2.2 User Feedback

The feedback gathered from this showcase contained interesting aspects. Whereas the concepts for better collaboration features were obviously mentioned as necessary optimizations to make the scenario work, several statements suggested to consider the use of Virtual Reality. In the given scenario, we conducted some general thoughts about defining a design space for distributed collaboration, one of the open challenges in research for CIA [1]. To decide which technology is suited for specific aspects of collaboration in manufacturing processes, we tried analyze the inherent characteristics of AR and VR to identify the most suitable areas of application and how to reasonably combine these technologies.

Figure 7

Demonstration of showcase prototype: Left: Collaboration between HoloLens (AR) and HTC Vive Pro (VR) in a distributed synchronous scenario. Right: Working on a real object using AR-based mode.

AR versus VR. According to our observations, the key aspect of AR in design and manufacturing processes is the physicality. Information is placed exactly where it is needed and the mixed sensation of real object and virtual additions, allows for real-world visual, auditory or even tactile assessment of the object, which is crucial especially when discussing materiality or very sophisticated aspect of the look and feel of an object, which still cannot be reproduced by virtual representations. This also applies to very subtle complaints, that can only be reproduced under certain lighting conditions (see Section 3). Another very encouraging aspect of AR is that the agenda of a review or discussion can be directly attached to the real object – all agenda points can be placed on the object, the users moves physically through the presentation, one can literally “move back and forth” through the specific agenda points, which serves as an intuitive interaction technique. On the other hand, AR incorporates some of the benefits of virtual techniques: displaying or overlaying different layers of information, manipulate digital information such as annotations, labels or surface drawings and the option to highlight specific aspect by changing the visual appearance of the object. Especially surface drawings represent a AR-exclusive feature in terms of haptic sensation and intuitive use. This makes AR suitable, especially for reviewing situations and discussion of different aspects of a real object. The drawback is, that there always have to be representing real objects, which may be difficult in distributed scenarios, and also that reality cannot be hidden, meaning, that the limitations of real objects, such as occlusion, size and physical immutability remain. The last aspect is the key strength of using VR for collaboration: the users can arbitrarily modify the object, especially in terms of duplicating, deforming, changing materiality, hiding parts or changing the perspective, for instance large magnification or intersections. This makes it feasible to be used for problem solving, exploring, and manipulating different object aspects. It can be reasoned, that the use of VR is especially beneficial in earlier project phases, in which the shape and object properties are not finally defined or at least optimizations or changes for adapting to manufacturing processes are made.

Figure 8

Collaborative Session with different Mixed Reality modalities creating mixed interacting & exploring and sharing & creating environment. Left: A third collaborator uses a head-mounted display to participate. Right: The role model for the head-coupled AR subtly changes and adds additional features to enhance communication as well as collaboration.

Figure 9

Screen captures from the prototypes using a wooden cube for presentation purposes. Left: Line drawing in AR and viewing position and direction of VR user as shown in HoloLens, Middle: view in VR (HTC VIVE) with synchronized line drawing and menu for controllers, Right: interactive labels in AR depicting issue details on the object.

4.3 Iteration II (AR/VR): Remote Collaboration

Based on these observations, we adapted the scenario by a third collaborator using VR. Therefore, the scenario now consists of the manufacturer with a tablet-based AR visualization, the supplier wearing an AR HMD and the designer who participates via VR, in our case also using an HMD (see Figure 7). The use case still concerns the assessment of a manufacturing sample, but in an early pre-production phase, respectively design phase, in which production and design parameters are not yet defined completely. The idea is, to use VR to facilitate the decision-making process in this phase by examining different design versions or the impact of changing different object parameters (see Figure 8, left). The setup remains synchronous and distributed, but again could be adapted to asynchronous cases.

5.1 Visualization of Collaborating Actors

The showcases demonstrate the highly interactive character of these types of collaborative scenarios, the roles between active presenter and more or less passive spectator changes on-the-fly. This corresponds with the findings of Billinghurst et al. [1] for the general nature of collaborative immersive analytics. It is therefore necessary to offer mechanisms to communicate the location, current view and executed actions of collaborating actors. Humans can indeed communicate directly via speech, although many messages are transported between the lines. This makes non-verbal communication not just an extension of auditory communication. The showcases uses a simple visualization of positions and viewing directions to raise awareness of the current state of the participants – Where are they? What are they doing? What are they looking at? – which is crucial for communication.

However, in a complex scenario, techniques that are more sophisticated need to be explored. This ranges from traditional approaches such as sharing the current view as video stream embedded in the view of the collaborating users. Yet, augmented and virtual reality offer superior concepts, which can be chosen, based on the needs of the current use case. Fully animated avatars or even abstract ones could serve the purpose to use gestures and communicate non-verbal aspects, such as gestures [29] and facial expressions [25], more effectively.

Especially in VR, the option to “posses” another user by overtaking or controlling his view could be possible, but needs to be treated with caution regarding motion sickness and orientation factors. In AR, blending viewports or visualizing a target position from where the current issue is viewed best, represent other options for sharing views. In the opposite, especially for group interaction or larger environments some kind of overview could be useful to keep track of the position and actions of the collaborators. Moreover, [14] found complementary non-verbal cues in form of a computer character more engaging.

5.2 Speech Input and Voice Communication

One of the most demanded features was the use of speech recognition for text input, as well as using voice commands for specific system functions. Opposed to natural human communication, embedding voice commands in a scenario in which several users communicate with each other represents a severe issue. As stated above – the environment is highly in flux, with rapidly changing roles, and communication was rather chaotic, maybe also due to the fact, that there were only abstract visualizations of the partners available, so that it was difficult to judge whether a user listens or wants to speak.

Therefore, communication patterns need to be investigated further and the influence of virtual avatars or the user representation in general as non-verbal features has the potential to enhance verbal communication [7]. Thus, it may provide better real-time feedback on basis of conversational metrics. Mixing this with voice commands and speech recognition may work if there is a dedicated signal for the system for activating and stopping the speech recognition. Activating keywords are an alternative but rarely fit naturally into the normal interaction flow. A positive aspect is, however, making it transparent to all users, which current dialogue is being recorded or analyzed, as it is crucial regarding security and privacy aspects.

5.3 Transferred Data, Security and Privacy Issues

Several aspects fit into this category. As we were showing the prototypes to industry partners, one of the more prominent remarks were made regarding the security of transferred data. As data is highly sensitive, encryption is crucial. One benefit of using virtual presentations could be, that only data about positions, actions or similar data needs to be transferred, circumventing issues regarding low bandwidth and sending of sensitive data in a stream around the world.

As the manufacturer and the supplier do not necessarily send the 3d data or technical documents over the network (because they have their respective data sets, which could have been exchanged via secure traditional channels), the security question can be minimized, at least in some use cases. If high detail overlays or surface drawings need to be transferred, the contained 3d coordinates again could be used to extract at least part of the object shape.

Privacy is another issue, as the participating actors are necessarily tracked more or less completely. Therefore, it is crucial to store only a minimum of information needed and clearly communicate, when data is recorded (and what this data includes), as mentioned in the section about speech recognition. This becomes even more complex when asynchronous communication takes place, for instance if parts of the presentation are recorded for later replay or review.

Bandwidth was another concern, especially when visualizations that are more sophisticated are presented. However, this issue could be solved by exchanging a predefined data set beforehand as mentioned above. In this case only control and status messages need to be exchanged between the clients, making the live video streaming of views far more demanding than the actual 3d part. Modern 3d game engines offer fairly easy to use mechanisms to control the network replication of data, states or control signals, so this aspect represents an opportunity even for limited bandwidth scenarios.

5.4 Use of VR in Early Design Phases

The integration of VR in the given scenario demonstrated how different this technology is used and perceived compared to AR. Whereas AR is fueled by the sensation of immaterial virtual elements embedded into an physical, tactile, tangible environment, VR offers the opportunities for manipulating the environment without physical constraints. This includes deforming, deconstructing, duplicating or diving deep into the object.

Rapidly changing perspective, without being restricted to size, weight or material is a unique feature of VR. To our impression, VR therefore is suited best in early design phases, especially when it comes to rapid prototyping, and immediate evaluation of design decisions by analyzing and comparing different variants of an object. Our prototype was very limited in terms of interaction, but there are several easy to use approaches for rapid prototyping or sketching using VR such as Google Tilt Brush,^[1] Google Blocks^[2] or Unreal Engine VR Mode,^[3] just to name a few. Integrating these tools into the engineering process or enable the transition from rapid prototypes to more complex 3d models represents an issue.

When it comes to critical assessment of detailed design studies, the focus is less on manipulating the object itself but on the authentic representation of materiality, look and feel. VR still can be useful for these tasks. However, this is the point, where in manufacturing processes the transition to AR-visualizations begins, as the real world look and feel can’t be completely simulated by current technology (see Figure 10). Based on these observations we propose a structure for the design space of AR and VR technologies in manufacturing processes.

Figure 10

Classification of preformed roles described in Section 4 and the respective technology used, as well as allocation to the corresponding phase of the development process. In addition, the prototypes based on the described showcases are located. As the phase progress, the use of VR shifts to AR, since the existence of physical objects is then more prevailed.

5.5 The Design Space for Collaborative Mixed-Presence in Manufacturing

Starting from early design phases, in which rapid generation of ideas and evaluation is crucial, we see the advantages of VR as a preferred technology to use. Distributed teams could meet in virtual playground to exchange ideas, for sketching and prototyping and for assessment of virtual mock-ups (see Figure 10, phase design).

As the design phase continues and the objects become more detailed and complex, manipulation and modeling become less important. However, the opportunity to disassemble a virtual object, to hide parts and to rapidly adjust magnification, viewing angle and position makes VR still a very viable technology to use, but slowly the first physical prototypes emerge and with them AR technologies rise in importance. In specific phases, a combination of AR and VR is useful – for judging issues under real-world conditions in an unobstructed view and to analyze technical details or access design variants. In the pre-production phase, AR becomes more and more significant, as issues can be re-evaluated on the physical model, whereas data can be directly mapped onto the real surface (see Figure 10, phase pre-production).

Reaching the end of the production cycle – especially in collaboration with the customer, collaborative tools may focus on traditional tools which may be combined or extended with AR – techniques (see Figure 10, phase production). However, the different roles of the participants must be distinguished [17] and functionalities provided. In our presented use case, we have the supplier in form of the contributor, the designer as part of the decision makers and the manufacturer as the decision pilot who leads the discussion. At the same time, [20] model must be embraced when developing functionalities supporting these roles, since it takes into account the different characteristics of task-related group work.