Measuring the Impact of Augmented Prototyping Systems in Co-Design Activities

Abstract

In recent years, research reached a very high level of development and validation of augmented prototyping systems in support of collaborative design activities. However, there is still great scepticism in companies when it comes to integrating these new technologies within a consolidated working model. Among others, the main barrier to overcome concerns the lack of understanding of the impact of AR systems on the key objectives of a business, such as improving its efficiency and revenue. For this reason, this paper aims to quantify these indicators by observing the technological impact not on a single design session but on an entire product development process, during which the aspects related to its integration are also considered. Thanks to the collaboration with a design agency, it was possible to compare parameters such as the lead time, number of iterations, person-hours and costs between two similar and realistic projects in which only one was supported by projection-based AR technology.

1. Introduction

The growing demand for shorter product life cycles pushes companies to improve their design process to become more competitive in generating and developing design alternatives. The adoption of new tools and technologies that support design activities by digitalising most of the product features and behaviours enabled a significant step forward [1]. Nevertheless, the efficient integration of these applications in the design process strongly depends on the users’ capability to interpret the information simulated by the computer. In the field of creative industries, for instance, these results are presented in the form of hyper-realistic images or 3D models that digitalise the visible features of the physical world to make digital twins indistinguishable from real products. On the other hand, working with virtual models only results in several limitations when inexperienced users participate in design activities due to (i) the difficult interpretation of the output’s physical properties, (ii) the complexity of performing real-time modifications and (iii) the low support in facilitating the communication and the idea sharing. The current practice is not adequate for supporting collaborative sessions where the involved actors (i.e., designers, clients, end-users and other stakeholders) have different levels or types of expertise, but they still need to communicate with each other for the continuation of the activity [2]. In this context, the interpretation of a physical feature provided by a digital model is not uniform among the participants, and their knowledge gap, if not appropriately mitigated, is the main cause of misunderstandings [3]. In turn, lengthy, inefficient and often undesired iterations are inevitable before reaching the definition of the final solution. By increasing the lead time and the related development costs, unnecessary design cycles are responsible for economic inefficiencies in the industry [4].

With the aim of reducing language barriers with clients, designers are encouraged to manufacture physical prototypes from the first stages of their activities. These acquire different forms (e.g., soft models, hard models, presentation models and prototypes) according to the design sector, stage, manufacturing process and purpose [5]. Prototypes are efficient means for sharing and evaluating the current state of the development by offering multiple users the advantage of tangible interactions, spatial perceptions, and hand–eye coordination for the exploration of the product’s operating principles [6]. The number of physical prototypes, high for the detailed design stage, is minimal in the conceptual phase of the development process due to the necessity of applying rapid and continuous changes to the product’s working features [7]. The static nature of the prototypes, in fact, implies the manufacture of a new variant every time a modification is applied to its design, and their high prototyping cost and time prevent having a real-time visualisation of the results.

One way to afford a more effective and rapid prototyping method for product design activities is to merge physical and virtual prototypes in a single environment so as to limit their intrinsic disadvantages: the former being static, expensive and time-consuming and the latter requiring a high level of expertise to comprehend as well as a lack of physical properties and tangible interactions. Interactive Augmented Prototyping systems (IAPs) [8] provide the technical means for this goal by integrating sensing hardware components to perceive the physical environment surrounding the users, display devices to add spatial coherence to computer-generated information (usually in the form of visual feedback but sometimes also involving the senses of touch and smell) and human–computer interfaces to modify the output of the digital renderings. This research aims to measure the impact of a collaborative design session supported by a Projection-based Spatial Augmented Reality (P-SAR) platform on the performance of an entire design process and its workflow. P-SAR is a technological variation of IAP systems where coloured light beams generated by projector displays change the external appearance of physical and tracked objects. It creates a shared environment that, by mixing physical and digital contents, is suitable for supporting those design sectors that mainly work on the definition of superficial details of an object (e.g., packaging, user interface, fashion accessories, product look, ergonomics, etc.). As highlighted in Figure 1, P-SAR technology has only recently begun to develop within scientific research. Still, it has benefited from the widespread adoption of more traditional solutions based on Head-Mounted Displays (HMDs) or Hand-Held Devices (HHDs). In just 20 years, significant functionalities have been developed, implemented, and validated in specific usage contexts, leading to initial explorations in commercial applications. This trend can be justified by the relative affordability of the hardware components compared to other spatial or portable systems. However, its potential impact is constrained by the need to adopt complex calibration procedures to define the hardware configuration.

Paper Organisation

The paper is organised as outlined below. Section 2 describes the research gaps that led to the execution of this activity; these resulted from the analysis of the state of the art, which was conducted from both the perspective of technology development and its validation. Section 3 outlines the P-SAR technology developed in partnership with a design agency to implement all the functionalities necessary to perform effective collaborative design activities. This section also provides all the details regarding the platform’s installation, preparation and usage. Section 4 concerns the description of the experiments carried out to measure the influence of the technology with details on their organisations, set-ups, and metrics. Finally, Section 5 and Section 6 present and discuss the results collected during the development of the projects under investigation in terms of industrial-relevant performances (i.e., lead time, iterations, man-hours and costs).

2. Previous Works

For the past twenty years, the scientific community has exhibited a growing interest in AR and co-design topics. Figure 2, generated using the Scopus database and considering all articles published in journals, conferences, and books containing the words AUGMENTED REALITY and COLLABORATIVE DESIGN in the title, abstract, or keywords, highlight this significant, consistent, and steady growth. However, searching for publications combining both themes makes it apparent that there are still few and highly fluctuating instances. These are examples of AR applications developed for supporting co-design activities where the augmentation of the real world is performed by various display technologies that, placed in between the users’ eyes and the real objects, alter the perception of external visual inputs. Among all the available variants, this paper deals with systems that use spatial devices like projectors to project single or multiple-coloured light beams onto the external surfaces of three-dimensional objects. The adaptability to different use case scenarios is the main strength of P-SAR technology, which implies no limitations for the dimension, quantity and position of three-dimensional physical objects and no restrictions for the number of users that can simultaneously collaborate. A projector, in fact, generates a shared space for collaboration where two distant worlds, physical and digital, are merged into a seamless experience for the users. These preserve their own point of view without being forced to stay in uncomfortable positions or wear any external device [9].

The following analysis of the state of the art is conducted across various databases, such as Scopus and Google Scholar, using augmented reality as the main search keyword. The collected papers, filtered to include only peer-reviewed research published after 2001 and focused on projection-based technology, are finally classified into two categories: those primarily addressing technological advancements and those validating the technology in specific application domains. This classification is aligned with the two subsections outlined below.

2.1. P-SAR Technology

Raskar et al. [10] presented one of the first systems that explored the use of multiple projectors to alter the aesthetic aspect of an object. This application allowed the authors to investigate (i) the parameters linked to the visualisation of the augmented images that have an influence on the projected quality (i.e., alignment, light intensity, occlusion, and overlap), (ii) the modalities for interacting with the digital contents with high intuitiveness despite the three-dimensional space and (iii) the manipulation of those prototypes that act as projection targets. These, being the three key functionalities that constitute an IAP, have been developed over the years in multiple variants according to the needs of the design session. The detection of the physical environment, assigned to the tracking module, is mainly realised by computer vision-based approaches that use optical sensors to recognise some visible features of the targets (e.g., shape, markers, colours). Yu Sheng et al. [11], for instance, implemented an algorithm that combined the detection of colours and edges to enable the manipulation of different types of walls during the simulation of daylight conditions in architectural projects. Gervais et al. [12] used infrared sensors to detect, by means of triangulation algorithms, the spatial placement of reflective markers directly attached to the object surfaces. On the other hand, the interpretation of the modifications performed by the users onto the projected outputs, realised by the interaction module, can use more unconventional paradigms like interactive/tangible surfaces [13], remote controllers [14] or gesture interfaces [15]. These are often preferred over standard input systems for not inhibiting the participation of multiple people and facilitating the comprehension of the interaction modalities for those users with lower expertise in the field. The generation of the SAR output is finally made by the visualisation module, which uses the data from the tracking and the interface to align the physical prototype with the current configuration of the virtual model. The limitations of the hardware and the technology involved have been addressed to increase the reliability and the accuracy of the rendering, which is extremely relevant in design applications: colour fidelity and image resolution [16], multi-projection treatment [17] and shadows caused by occlusions [18].

2.2. P-SAR Assessment

Most of the works presenting projection-based IAP systems highlighted the following benefits when applied in design activities: easy design process integration [19], facilitation of the in situ [20] and remote [21] collaboration, simplified manipulation of the projected contents [22], availability of low-expensive and interactive mockups [23], direct manipulation of mixed prototypes [24] and availability of real-sized prototypes [25]. These results were mainly obtained by carrying out experimental campaigns in laboratory set-ups with the involvement of either students or real experts of the investigated design discipline. The tests aimed to reproduce typical design scenarios of the current industrial practice in order to measure, with objective and/or subjective metrics, how much the technology can influence the execution of the activity. One of the fastest methods to validate the demand of P-SAR in design has been presented by Verlinden et al. [26,27], during which the functionality of the technology was described by using a demonstration video and a presentation. Their outcomes, related to the potential impact on the design work, were derived from the analysis of semi-structured interviews with 13 senior managers coming from actual product design and engineering companies. With a panel of reviewers with a similar level of experience, demo sessions were performed by Irlitti and Itzstein [28] to investigate the potential use of their system in the industrial design process, whilst open-ended sessions were organised by Hartmann et al. [20] to understand how the user’s behaviour can change when using P-SAR technology for both individual and collaborative scenarios. More realistic tests, aiming to simulate the use of technology during a design task, were performed by Akaoka et al. [29] and Park et al. [30]. For both studies, a larger number of subjects were recruited among university students in order to collect reliable usability data from objective metrics and subjective assessments.

More recent studies presented alternative methods to assess the technology while maintaining a controlled environment. Cascini et al. [31], for instance, considered as case studies actual projects being developed by the designers involved in the test campaign to improve the realism of the scenarios, the active participation of all the actors and the interest of the professional for a successful result. Thanks to that, the testing activity aimed to measure the performances of a collaborative design session supported by P-SAR technology and compare them against more conventional co-design setups. In this context, it is worth mentioning the work of Thomas et al. [32] in which a laboratory system of 40 projectors and tracking technologies was used to support interior design tasks. The paper presented a detailed description of a design scenario characterised by collaborative activities between designers and stakeholders to point out the potential benefits of the technology.

In summary, the literature review reveals that different types of P-SAR systems have high maturity and the potential to be implemented as supporting tools in design activities. On the other hand, their impacts on daily industrial practices, which are not conceived to be a one-time-only session, are still unknown.

2.3. Aim and Significance

The leading purpose of the current study is to fill the knowledge gap identified within the previous literature review and provide preliminary results revealing the effects of a collaborative design session supported by P-SAR systems throughout an entire design process. For this reason, a design agency collaborating on the development of specific features and functionalities of the P-SAR platform was involved in benchmarking—by means of industrial-related indicators like lead time, man-hours and costs—the workflow of two design projects: one supported by a P-SAR platform and another performed with traditional methods. This company mainly works on the packaging style for FMCG, which is a design sector that was proven to exploit most of the strengths of IAP systems [33], and the engaged experts already had significant experience in using the technology. Moreover, the objective of the current study is to provide realistic measurements related to the impact of this technology on the business, which was accomplished thanks to the adoption of a set-up that prevents participants from being able to distinguish whether they are attending a simulated session rather than a real one. These experimental choices can be summarised as follows:

• Adoption of a full-working P-SAR platform installed at the premises of the design agency;
• Selection of two actual packaging design projects of the agency that had several similarities with each other even though they are still at the beginning of their development;
• Involvement of agency employees only and, if possible, of their customers during all the design iterations that occurred for the development of the packaging;
• Organisation of a single P-SAR-supported collaborative session between designers and clients at the beginning of one project only to also account for the resources spent on the installation procedures and the creation of the contents (Section 3).

According to the intent of this paper, to perform a full technological assessment, the following section details all the steps required by the designers to install and run a collaborative design session supported by a P-SAR platform.

3. P-SAR Platform

SPARK (SPatial Augmented Reality as a Key for co-creativity (website: www.spark-project.net (accessed on 27 October 2023)) is an interactive platform based on P-SAR technology aiming at generating augmented objects, called Mixed Prototypes (MPs), that support collaborative design activities. The possibility of having co-creative sessions where multiple design practitioners, together with their stakeholders, can work or simply observe the realistic product rendering in the form of a MP is essential to collect more effective feedback regarding the project advancements [31,34].

The SPARK platform architecture has been designed, developed, and integrated to meet the requirements driven by professional use of the system and to facilitate the collaboration of the participants (Figure 3). Like most the IAP systems with the same technology, our SAR system has been subdivided into three modules to enable three different MP functionalities: spatial projection on tri-dimensional surfaces (visualisation), direct manipulation of physical objects (tracking) and real-time modification of the external appearance (interaction). Moreover, a back-end application called the Information System (IS) has been used as a repository of all the data used and created during the session as well as for managing multiple projects. Once the session is running, a simplified interface based on touch and tangible interactions allows the manipulation of images or textures that compose the appearance of the virtual product. This digital information is finally aligned and deformed according to the placement of the white mockup to generate a real-time and realistic rendering of the MP.

The following paragraphs describe the functionalities implemented to enable the use of the SPARK platform during collaborative design sessions. To do so, the hardware calibration and the session preparation procedures have been considered together with the session execution and the data elaboration phases to highlight the novel requirements and activities introduced by the platform (

Table 1. SPARK workflow description for the execution of a design session. The table defines, for each stage of the workflow, the activities, the modules of influence (G = General; V = Visualisation; T = Tracking; I = Interaction) and the actors involved (T = Technician; D = Designer; C = Client).

	Activity	Module	Actor	Definition
Platform preparation	Room set-up	G	T	- Configuration of the structure on which the hardware is fixed; - Number, position and type of projectors (augmented space); - Type and position of tracking system (interaction volume); - Type of interaction device (session participants).
	Projector calibration	V	T	- Transformation matrix camera to projector reference systems; - Validation and refinement of the results accuracy; - Creation of a new projector colour profile.
	Tracking calibration	T	T	- Definition of a ground plane as tracking origin; - Transformation matrix camera to ground reference system; - Transformation matrix projector to ground reference system.
Session preparation	Physical prototype	V, T	D	- Prototyping of the physical model; - Preparation, if needed, of the physical model external surface; - Arrangement of the IR markers for the tracking system.
	Virtual prototype	V, T	D	- 3D modeling and mesh generation of the prototype; - Definition of the mesh’s UV maps for each surface/body.
	Prototype calibration	T	D	- Transformation matrix between the reference systems located   on the pivot points of the virtual prototype and the markers; - Selection of the appropriate multi-image blending algorithm;
	Assets collection and upload	G	D	- Collection of the bi-dimensional assets; - Session management based on clients, products and users; - Upload of the bi and tri-dimensional assents on the IS.
Session	Virtual interaction	I, V	D, C	- Manipulation of all the bi-dimensional assets instantiated; - Tag of each preferred version of the designed product.
	Physical interaction	V, T	D, C	- Manipulation of all the physical prototypes available inside   the working volume;
Post	Data export	G	D	- Generation and elaboration of the sessions’ LOG files and the   configuration of the preferred versions of the products;

). Whilst the first stage is performed only once during the installation phase of the platform, the second one needs to be re-executed before the beginning of each new session with a considerable impact on the traditional design workflow. Taking into account all the phases is essential to make a fair assessment of the actual impact of IAP in an industrial context.

3.1. Stage 1: Platform Installation

The “platform installation” stage implies the physical arrangement of the hardware components embedded in the SPARK platform and their subsequent digital reconstruction by means of calibration procedures. It starts with the definition of the hardware features (i.e., projector, tracking system and interface), which allow the three respective modules to work properly and in accordance with the session’s needs (Figure 4). Once decided, the equipment, projectors and cameras dispositions are defined on a ceiling frame structure to not excessively obstruct the movements and sights of the session’s participants. The system is developed to work with a dual-projection system that defines an augmentable area of approximately 1 square meter. The session participants interact with the physical prototype thanks to an array of IR cameras and emitters. Three different devices have been integrated for the manipulation of digital content: a wide touch screen, a tablet device, and a standard monitor mouse. This choice has a relevant impact not only on the way the users interact with the system but also on the disposition and engagement of the participants of the collaborative session, which are more focused on the display in the first scenario and on the mixed prototypes in the second and third ones [35].

Once the devices are correctly steadied, the hardware calibration phase can be executed. This is the only part of the entire workflow that must be performed by a skilled operator with specific expertise in the P-SAR technology. On the other hand, such competence is required only when the arrangement of either the projectors or the tracking system is modified. This means that if the platform is kept fixed in the same initial position, there is no need to revise the calibration results before a new design session. The objective of this phase is to detect the relative locations and orientations (i.e., extrinsic parameters) of the projectors and IR cameras with respect to an absolute reference system as well as some properties of their integrated sensors (i.e., intrinsic parameters) like the focal length, the optical centre, the skew, and the distortion coefficients. These data are required to build the virtual environment where the digital contents are correctly rendered to overlap the prototype’s surface. For this reason, the higher the accuracy of the calibration results, the better the projected images’ quality, alignment and robustness.

Within this stage, an optional calibration procedure can be performed to compensate for the technological limitation of the projectors to render a defined input digital colour accurately. The task is accomplished thanks to commercial devices capable of measuring the difference of the output colour of a display with the input data and computing a new, more reliable colour profile based on the actual use conditions [36].

3.2. Stage 2: Session Preparation

This second stage includes the generation of the assets (i.e., images composing the product’s surface layout, 3D virtual models and physical prototypes) required for the execution of a design session supported by SPARK. It is worth noting that even in this stage, some of the following described tasks are executed only in preparing the first activity with the platform since their outputs can be reused for the following sessions as long as the product under development remains the same. In addition, the demand for high-level technical skills for the previous calibration procedures is avoidable; designers can use their knowledge, together with a short initial training period, to make the P-SAR system work properly.

The stage starts with the creation of the physical model to be used as a projection target during the session. It can be appositely manufactured by means of rapid prototyping techniques, like 3D printing and cardboards, or it can be a product already available on the market with an identical shape and dimension. Extremely relevant is the preparation of the prototype’s exterior, which needs to have a uniform and matte colour as well as a smooth and rigid surface. White or grey spry paintings, if the projected images are too bright, are used to achieve the required quality of the external finishing. Once the physical prototype is ready for the projection, four or more IR reflective markers are attached on its top sides to facilitate their detection with the tracking system. Examples of the configuration of the physical prototypes are illustrated in Figure 5, where seven products, prepared to perform design activities in different sectors, are equipped with various markers according to the needs of the P-SAR session. This figure also reveals the high adaptability of the system to work with different prototype types, shapes, dimensions, and materials.

The session preparation phase also requires the execution of 3D modelling activities to digitalise the shape and the dimension of the physical prototype previously manufactured. This task is performed with CAD software, based on either solid or surfaces, that supports the .obj format or with reverse engineering approaches. Not only is the accuracy of the mesh relevant to achieve high-quality projection but also: (i) the subdivision of the 3D model into separated parts that replicate the product composition; (ii) the simplification of the superficial features that can be simulated by textures; and (iii) the elimination of the hidden components that are not involved by the projection. This allows for decreasing the mesh complexity by obtaining only the product’s primitive shape and, therefore, reducing the computational cost of the P-SAR real-time rendering. At last, the procedure of UV mapping is performed on the 3D model to wrap an image on its external surface correctly. Since complex prototype shapes can excessively distort the tri-dimensional wrap of an image, the selection of appropriate cutting edges for the mesh opening is the key factor to avoid defects in the final rendering. These lines are usually defined according to the needs of the design session and can be used to simulate the shape of real junctures of the product, variations of the material or the presence of labels.

At this point, the physical prototype and the virtual model are ready to be calibrated by deriving the transformation matrix that links the constellation of the IR markers with the 3D model and the projectors. With the need to have a homogeneous rendering in a multi-projection configuration and independently from the shape of the target prototype, two blending treatment algorithms have been developed (Figure 6): standard blending and sharp edge. While the first performs better with circular or cylindrical artefacts where the projection overlap is limited, the second algorithm works perfectly with squared artefacts.

The images composing the layout of the mixed prototype are the second type of asset required to work with the SPARK platform. These are any brand logos, photos, illustrations, material textures or animated images created with common graphic editor software and converted in .png format. The recommendations for the correct generation of these 2D assets are related to the transparent background of the images, the seamless borders of the textures and the separation of the frames composing the animations. Moreover, this is the only step of the session preparation phase whose execution can be partially postponed even during the collaborative session. Any update of the asset’s library, in fact, is immediately notified to the SPARK platform in order to have the new images available during the use of the P-SAR.

Once all the session’s assets are ready, they are uploaded to the IS, which is a cloud database that is in charge of storing the assets in libraries distinguished by type. This is a back-end component of the SPARK platform that supports the execution and the management of the design activities. It creates the sessions, subdivided into products and clients, defines which users can access or modify the design and, if needed, it allows the generation of the first version of the prototypes. The IS, which constitutes the fourth module of the SPARK technology, includes other useful functionality capable of helping design agencies manage their projects and activities.

3.3. Stage 3: Collaborative Design Session Execution

The third stage consists of the actual collaborative design session supported by SPARK, in which the participants interact with the physical and digital contents that compose the mixed prototype. It is worth noticing that once the set-up of the platform and the session are both completed (as described in Section 3.1 and Section 3.2), the subsequent uses of the platform based on the same prototype’s model can start with this stage. The P-SAR application, in fact, can store the calibration data and let the user choose the correct one to use. Due to the different use conditions of the platform (i.e., profile and number of participants, type of design session, activity, stage, etc.), four different types of interfaces have been developed. These, illustrated in Figure 7, are based on a mouse and keyboard as input devices, on touch screens of multiple sizes or on a tabletop interactive surface with a combination of touch and tangible interactions. The choice to increase the display dimension is a consequence of the collaborative activities enabled by the P-SAR technology, where more persons are interested in giving their contributions to the development of the project. Moreover, means of interaction based on more natural manipulation of the projected images and/or direct touch on the mixed prototype have been excluded in favour of a higher input accuracy and a lower risk of occlusion problems. Among these variations of the interactive devices, the system’s functionality is preserved. This has been studied to provide the users with the minimum number of functions sufficient to perform the design tasks without any complex overload, which is unsuitable for collaborative activities. The software architecture and functioning allows the user to select, move, rotate, scale, delete and swap any 2D assets, change the background colour or the texture of each part of the model, play/pause, speed up/down and change the loop type of the animated images, and tag the current version.

The Graphical User Interface (GUI) is not only considered as a means for interacting with the prototype’s external finishing but also as an additional display where the virtual model is rendered, and fundamental information is provided to the users. Three GUI visualisation styles are integrated with the platform to enable a more agile switch of the system functionality according to the needs of the designers. Among them, three ways of presenting the data and three modalities of manipulating the assets are provided. While the 3D model and the canvas views are based on the layout of common 3D modelling and image editor software, the touch area relies on the fact that the mixed prototype alone is sufficient to perform a design session without focusing the users’ attention on the display itself. Moreover, a SPARK-supported design session provides the participants with an additional means of interaction: the direct and tangible manipulation of the tracked physical prototype. This functionality is enabled thanks to the simultaneous cooperation of the tracking and the visualisation modules. The artefact’s placement data, measured in real time by the first, are used by the second to compute the deformation of the projected images with the transformation matrices. The physical manipulation, thanks to which each person can look at the same surface from their own perspective, facilitates collaboration in a P-SAR environment and extends the number of participants allowed during the session. It also enables the user to perform ergonomic analysis with the mixed prototype, which is a relevant resource in product and interface design activities. On the other hand, two strong occlusion limitations are still present due to the technology used for both modules: (i) the occlusion of the projection and (ii) the occlusion of the IR markers.

3.4. Stage 4: Post-Session Elaboration

The final stage is focused on data collection and elaboration. The SPARK application, in fact, can record in a .csv file all the actions performed on the GUI or the physical object and send the current state of the mixed prototype to the IS. The latter functionality allows the creation of a realistic timeline of the product development and the revision of all the steps performed at the end of the collaborative design session. Once a product version is tagged from the interface, a local file containing the current asset’s configuration is saved, and a notification is sent to the IS to facilitate the identification of the preferred layout from the list of available variants.

4. Effects of IAP in Creative Industries

The analysis of the industrial application of the SAR platform has been conducted with the involvement of designers, accountants, and clients of a design agency located in Milan (Italy). In particular, the development process of two similar projects concerning the renewal of two food product packaging types has been observed starting from the initial briefs with the clients until the delivery dates of each final output. The two products selected were a frozen pizza and a fresh pizza (P1 and P2, respectively) that, according to the involved experts, had analogous development steps and similar input instructions, although they were commissioned by two different companies at two distant moments. The analogy between the two design strategies prescribed with the initial briefs is also related to the profile of the clients that had similar requests and expectations: two product variants, meaning two pizza flavours, with a consistent packaging layout except for the graphical elements and texts showing the different ingredients.

Both projects have been carried out without any constraint by the involved actors for what concerns the adoption of the supporting tools (software, mockups) and the selection of the design procedures. The everyday practice of the design agency included, among others, face-to-face meetings and distant collaborations (with synchronous and asynchronous exchange of information) where the proposals were shown by means of printed mockups or digital renderings and the written or spoken observations were collected. The only variation between the two product development activities was the organisation of a single review meeting with the support of the SAR platform, where the designers showed and discussed the current state of the packaging together with the customers of the design agency. This SAR-supported session (Figure 8) happened at the end of the first iteration of the fresh pizza product to present the outputs of the first development round after the delivery of the brief and to eventually collaborate with the clients to generate a new version. In the context of creative industries, the term iteration is used to indicate the single period of the entire development process that starts with the collection of feedback from the clients, ends with the release (or presentation) of the new design version and usually happens cyclically until the definition of the final product version ready for the production.

4.1. Apparatus

The SAR tool adopted for the execution of the collaborative design activity of P2 is a customised version of the SPARK platform previously described. The hardware installed at the premises of the design agency is selected to fulfil better the need for a shared space for visualisation and interaction that is typical of collaborative activities (Figure 9). Two EPSON EB-5530U (website www.epson.co.uk/products/projectors/installation/eb-5530u?productfinder=EB-5530U (accessed on 27 October 2023)) projectors are installed, thanks to a metallic frame structure, close to the ceiling of the room and at the two opposite sides with respect to the meeting table. This multi-projection choice enables the augmentation of a higher area of the prototype surface and simultaneously reduces the risk of occlusion due to the presence of the participants. The tracking of the physical prototype is realised by means of the Optitrack V120:Trio (website www.optitrack.com/cameras/v120-trio/ (accessed on 27 October 2023)) bar fixed above the table to have better visibility of the IR markers glued on the top sides of the target object. Finally, a Philips 43BDL4051/T (website www.philips.it/p-p/43BDL4051T_00/signage-solutions-display-multi-touch (accessed on 27 October 2023)) touch screen is horizontally placed in front of the meeting table to encourage all the participants to apply the desired changes to the product layout and to prevent the occlusion of the light beams caused by standing persons. The physical prototype of the fresh pizza packaging is realised starting from the actual product available on the market, as no changes in shape were expected from the new version. This prototype is firstly made rigid to avoid any deformations concerning the 3D virtual model and to simulate the weight of the food contained within it. At last, its surface is prepared for projection by using a white matte spray painting and by attaching, on the top side, to be clearly visible and distinguishable by the tracking system, the constellation of five IR reflective markers.

4.2. Participants

As the two case studies previously described two actual projects for the company, the number and the type of participants have been defined by the design agency itself based on the complexity of the works and the deadline of the activities. A total of 11 persons were involved during the development of the two packaging design projects (six for P1 and five for P2) with the roles of designer (two for P1 and two for P2), accountant (two for P1 and one for P2) and client (two for P1 and two for P2). The designers, one of whom was appointed as creative director, were responsible for developing the packaging of the assigned product according to the client’s expectations. They used their traditional tools and approaches for the execution of the design work without being influenced by the experimental activity. Moreover, the same designers of P2 were also involved in preparing the assets required to execute the collaborative session supported by the P-SAR platform. Due to the novelty of the tool and its impact on everyday design practice, they were supported by two experts in the technology for the set-up phase of the session and during its execution to avoid mistakes and setbacks while presenting the results. The accountants were instead responsible for managing the activities of the designers and keeping in contact with the clients, the ones that have commissioned the work to the design agency, for sharing the advancements and collecting feedback.

4.3. Metrics

The impact of the P-SAR platform in design practices is measured by means of five industrial-related metrics with the capability of comparing the development activities of P1 and P2 (

Table 2. Metrics selected for the comparison of the development activities of P1 and P2.

Metric	Definition
Iterations	Number of versions (i.e., iterations) released after the initial brief until the end of the creative phase (i.e., when the packaging is ready for the production)
Lead time	Number of working days passed between the start of the project (when the initial brief is received so that the designers have sufficient information to initiate the work) and the launch of the product (i.e., when the packaging is ready for production)
Man-hours	Sum of all the hours spent during the development of the product by designers and accountants involved in the project (including unbilled hours from the design agency)
Development cost	Sum of all the direct costs incurred by the design agency during the development of the project (only up to the definition of the final layout since the post-production costs are not considered)
Prototyping cost	Sum of all the costs incurred by the design agency for the preparation of the the design representations used during the collaborative sessions or sent to the client (these include materials and labour)

). The decision to analyse these specific metrics was based on prior research that already evaluated the impact of the platform based on P-SAR technology while supporting collaborative design sessions. Previous studies, such as [31], were focused on the measurement of the performance metrics (including the duration, quantity, variety, and novelty of generated ideas, usability, and effectiveness of filtering) in a single co-design session supported by P-SAR technology, comparing them to other scenarios based on traditional AR technologies and conventional design approaches. These evaluations also captured the subjective opinions of the participants involved. Our study aimed to expand on this research by including an assessment of the resources involved, which is a crucial aspect often overlooked in most technology assessment papers.

The metric iteration corresponds to the number of times a new version(s) of the packaging layout is presented to the clients. A single iteration includes the following: (i) the debriefing phase, where the feedback of the clients related to the previous variant(s) is collected and interpreted, (ii) the design phase, where the layout is modified according to the perceived evaluations of the clients, (iii) the prototyping phase, where the new packaging with the new layout(s) is physically printed and/or digitally rendered, and (iv) the presentation phase, where the output(s) of the work is(are) disclosed to the clients to gather their new feedback. According to the design activity and the objectives of each iteration, it is possible to classify them according to an early design stage or a design refinement. In the first case, since the preferred proposal has not been selected yet, the iterative work is completed simultaneously on different variants of the same product. In the second phase, instead, a single design layout is chosen and used to reach the optimal arrangement of its graphical assets and textual elements.

The lead time parameter estimates the total time span required by the design agency to conclude the creative work and thus to release the final design solution ready for production. It is measured in terms of working days for the completion of the entire project as well as for the accomplishment of every single iteration. Since the computation of the lead time includes off moments where none of the involved persons are working on the design or the management (e.g., when waiting for the clients’ reply), this analysis also considers the actual hours of labour spent in all the activities related to the project. This metric, called man-hours, sums up all the effective hours (including those not billed) allocated by the employees of the design agency for the execution of the work. These are subdivided according to the moment (i.e., iteration) and the worker profile (i.e., designer or accountant) to better understand any variation between the two parallel product development phases.

The last two parameters evaluated in this experimental activity are related to analysing the costs incurred by the design agency for the project development and prototype manufacturing. Both are computed without accounting for the expenses associated with the installation of the P-SAR platform since using the technology across different projects can be considered an indirect cost. There are two relevant features of these metrics:

• A considerable part of the development costs is composed of the labour fare that, in the case of the involved design agency, is equal to 75 euros per hour (no matter the role of the employee);
• Apart from the session performed with the P-SAR platform where the actual set-up time is accounted for, it is common practice for the design agency to consider the prototyping expenses of one iteration equal to 15% of the creative labour costs incurred in the same period.

In addition to the previous objective parameters, the qualitative feedback related to the set-up of the session and the use of the platform are collected after the first iteration of P2. These are obtained by means of semi-structured interviews or informal discussions with the designers, accountants, and clients at the end of the collaborative activity. The objective is to explore how the potential end-users of the system have perceived the session workflow in terms of gathering feedback from their customers and how much the platform has affected their standard way of performing the design work.

5. Results

This section presents the analysis of the data collected during the development activities of P1 and P2 together with the results of each metric previously illustrated. The full project timelines for P1 and P2 are shown in

Table 3. Timeline of the P1 project (frozen pizza) with the man-hours’ distribution reported for each iteration.

Iteration	Brief	I1	I2	I3	I4	I5	I6	I7	I8	Release	Total
Date	09/03	15/03	23/03	28/03	30/03	03/04	05/04	13/04	24/05	23/06	77 Days
Proposals	—	4	3	2	2	1	1	1	1	—	17 Versions
Designer 1	—	17.5 h	7.25 h	2.5 h	4.0 h	4.0 h	1.0 h	3.0 h	—	—	39.25 h
Designer 2	—	—	—	—	—	—	—	3.5 h	5.0 h	—	8.5 h
Accountant 1	—	1.0 h	—	—	—	—	—	—	—	—	1.0 h
Accountant 2	—	3.5 h	2.0 h	1.0 h	—	—	—	1.0 h	2.0 h	—	9.5 h
Total creative	0.0 h	17.5 h	7.25 h	2.5 h	4.0 h	4.0 h	1.0 h	6.5 h	5.0 h	0.0 h	47.75 h
Total account	0.0 h	4.5 h	2.0 h	1.0 h	0.0 h	0.0 h	0.0 h	1.0 h	2.0 h	0.0 h	10.5 h
Total	0.0 h	22.0 h	9.25 h	3.5 h	4.0 h	4.0 h	1.0 h	7.5 h	7.0 h	0.0 h	58.25 h

and

Table 4. Timeline of the P2 project (fresh pizza) with the man-hours’ distribution reported for each iteration.

Iteration	Brief	I1 (P-SAR)	I2	I3	I4	Release	Total
Date	10/05	17/05	22/05	28/05	31/05	15/06/2018	27 Days
Proposals	—	3	2	1	1	—	7 Versions
Designer 1	—	10.5 h	2.0 h	—	—	—	12.5 h
Designer 2	—	5.0 h	4.0 h	7.0 h	4.5 h	—	20.5 h
Accountant	—	3.0 h	1.0 h	0.5 h	1.5 h	—	6.0 h
Total creative	0.0 h	15.5 h	6.0 h	7.0 h	4.5 h	0.0 h	33.0 h
Total account	0.0 h	3.0 h	1.0 h	0.5 h	1.5 h	0.0 h	6.0 h
Total	0.0 h	18.5 h	7.0 h	7.5 h	6.0 h	0.0 h	39.0 h

, respectively, starting from the date when the initial brief was received from the clients until the date of the release of the packaging ready for production. Each column indicates the number of iterations that were necessary to complete the creative design, together with the dates when a new version of the packaging layout is presented to the clients and the number of proposals is simultaneously developed. In addition, the man-hours required to accomplish each iteration are shown according to the person in charge and the role inside the agency.

The most noticeable difference between the two tables is the reduction in the number of iterations performed for P2 that are halved with respect to P1 (from eight to four). A similar trend is also evident by considering the early synthesis phase of the two project developments, where designers worked simultaneously on multiple variants of the same product. P2, in fact, required only two iterations and nine working days from the initial brief before identifying the preferred layout of the packaging. In comparison, four iterations and 15 working days were necessary for P1 to reach the same phase. The latter, called the refinement stage, is focused on the detailed development of the features of the single product that the clients selected among the available proposals. Thanks to this distinction, the role played by the P-SAR collaborative session in facilitating the identification of a solution that better meets the requirements of the design project is evident. Right after the first iteration of P2, in fact, the designers were able to design a completely different layout of the packaging with respect to the three initially proposed thanks to the suggestions of the clients participating in the session that saw the immediate rendering of the product by means of the mixed prototype. This layout, still in its raw state, was the one selected to perform the final refinements during the subsequent iterations. In P1, instead, three iterations were performed before collecting enough feedback from the clients to design the packaging they approved. It is worth noticing that the designers involved in both projects kept working on two different proposals even after the identification of the layout that better fulfils the client’s needs (I4 for P1 and I2 for P2); this choice was dictated by the necessity of presenting an alternative to the clients, which extended by one iteration the transition into the refinement stage.

The acceleration in the number of iterations also influenced the total lead time of the projects. While in P1, 77 working days had passed to conclude the design activities, only 27 working days were spent in P2 on the fresh pizza packaging development. Despite this difference, the man-hours are still equally distributed between the two design phases of both the projects: 68.2% in P1 and 65.4% in P2 of the total workforce was dedicated to the early design stages. At the same time, the remaining time was used to refine the proposals. The time spent in P2 to prepare the collaborative session supported by the P-SAR was also considered in this count. These operations, detailed in

Table 5. Definition of the operations required for the preparation of the collaborative session supported by the P-SAR with reference to the time spent and cost incurred by the agency.

Operation	Quantity/Type	Time	Cost
3D modeling	580 KB	17 min	21.25 €
UV mapping	580 KB	6 min	7.5 €
Physical prototype manufacturing	Real product	0 min	0 €
Surface treatment	Spray painting	20 min	25 €
Marker arrangement	1 prototype	15 min	18.75 €
Prototype calibration	1 prototype	30 min	37.5 €
Session set-up	1 session	5 min	6.25 €
Assets preparation	143 assets	382 min	477 €
Assets arrangement	12 versions	160 min	200 €
Total session set-up procedure	—	10.5 h	794 €

, included the manufacturing of the physical prototype as the target for the projection, the 3D modelling of its virtual version, the calibration of the prototype for the tracking system, and the generation and the arrangement of the assets that composed the layout of each version of the packaging. It is apparent that the 10.5 h spent on the preparatory process had a considerable impact on the total creative time of the project (37.9%), which meant the design agency had to involve an additional designer from the one initially estimated. Here, the two items related to the elaboration of the assets required more than 9 h by themselves, being the most time-consuming activities of the process.

The last two metrics considered in this analysis are the total development and prototyping costs. As explained in Section 4.3, the first one is computed by adding the labour expenses with other direct voices incurred by the design agency for the specific project (i.e., photo-shooting) and ignoring those required for the post-production phase. A total of 7069 € were necessary for P1 (composed of 4369 € for labour, 2000 € for photo-shoot and 700 € for printing preparation) while only 4585 € have been spent for the completion of P2 (comprising 2925 € for labour, 1200 € for photo-shoot and 460 € for printing preparation). The second voice, instead measuring the costs only related to the production of the prototypes, is computed differently from the two projects. In P1, in fact, the 537 € are estimated as 15% of the creative time costs (47.75 h 75 €/h = 3581 €), while the 1047 € of P2 are calculated as the sum of the set-up costs indicated in Table 5 (794 €) and 15% of the remaining creative time costs (22.5 h 75 €/h = 253 €).

The results obtained for each metric of this analysis are summarised in

Table 6. Comparison of the results for each metric applied on the longitudinal projects considering the whole development activity first and only the early stage after. Variations with a positive impact on company metrics are highlighted in green, while the negative ones are highlighted in red.

Metric	Unit	Whole Project			Early Stage
		P1	P2	Variation	P1	P2	Variation
Iterations	[number]	8	4	−50.0 %	2	4	−50.0 %
Lead time	[working days]	77	27	−64.9 %	15	9	−40.0 %
Man-hours	[hours]	58.25	39.0	−33.1 %	39.75	25.5	−35.9 %
Development cost	[euro]	7069	4585	−35.1 %	4331	2742	−36.7 %
Prototyping cost	[euro]	537	1047	+94.9 %	363	918	+152.8 %

together with their variation in percentage between the two projects (teal colour is used to indicate a better performance of the project developed with the support of the P-SAR session, while red colour is used for the opposite condition). By observing the whole project development, P2 had higher performances in all the metrics (50% reduction of the iterations, 65% reduction of the lead time, 33% reduction of the working hours and 35% reduction of the development costs) apart from the items related to the prototyping costs, where an increase of the expenses of almost 95% was recorded. On the other hand, if all the cost-related items are considered together, it is possible to claim an overall saving rate of the P-SAR technology across the whole project of 25.9% (7606 € for P1 and 5632 € for P2). Similar differences can be obtained by considering only those iterations that occurred during the early design phase, where the collaborative design session was executed with the support of the P-SAR platform. In this case, the two differences from the previous data are the prototyping costs and the lead time: the increase of 153% of the expenses related to the preparation of the prototypes between the two projects is a consequence of the higher impact of the session set-up time; the reduction of the working days variation (only 40%), instead, is due to the elimination of the last two iterations of P1 that drastically slowed down the process.

Limitations

It is worth noticing that by following the developments of two projects only, the performances obtained with the current investigation do not have statistically significant relevance and, for this reason, further case studies should be identified before extending the outcomes to other applications. The reasons for this choice are the difficult identification of coherent projects for comparing the five metrics and the high duration of the data collection phase for each development activity. The previously presented results can only be considered preliminary indicators of how P-SAR tools influence a design activity from the industrial point of view.

6. Discussion

The above-presented results illustrate to what extent the adoption of the SPARK platform improves the efficiency of a design project across several metrics related to its development. By halving the number of iterations from the initial briefing to the final definition of the packaging layout, in fact, it has been possible to simultaneously reduce the project lead time, the person-hours and the total costs with the introduction of a P-SAR-supported collaborative activity. This innovative modality of reviewing design proposals allowed the definition of a completely new packaging layout that, according to the participants, would not be possible to obtain so fast with the traditional process. Moreover, the layout completed during the co-creation session turned out to be the preferred one by the clients and, therefore, the one further developed during the refinement phase and released at the end of the project. This effectiveness is also demonstrated with an analysis of the variation of the average man-hours required for the completion of each release of the two projects. The slight increase from 7.3 h in P1 to 9.75 h in P2 reveals that it was possible with the fresh pizza product to condensate a higher amount of work in a more limited time span. The only parameter that was negatively affected by the P-SAR session was the cost of the prototype production. Its value almost doubled the one obtained in the benchmark development process due to the hours required to prepare the digital content and their initial arrangement on the mixed prototypes. The insights from the interviews with the designers help to understand better the reasons why this preliminary phase took such a long time. These can be summarised with the necessity to achieve the following:

• Decompose the design proposals, already completed with the professional software, into single assets;
• Export every single asset as a separated .png image without any background or clipping mask;
• Use the exported images to re-compose the design proposals onto the virtual prototypes of SPARK;
• Think and create extra variants of the same assets to enrich the library and the alternatives of modifications during the collaborative session;
• Generate an appropriate UV map of the virtual model according to the needs of the collaborative session.

The difficulties met by the designers during this phase can be drastically reduced by gaining more experience in using the system and by exporting the assets as soon as they are used. Another solution is to improve the integration of the platform with the graphic editor software commonly adopted by the experts so that the layout of the product is automatically updated when a new modification is applied. Moreover, the negative impact of the prototyping costs can be mitigated by organising multiple sessions within the same project or by considering the application of the technology in other design fields (i.e., product looks, interfaces, accessories, ergonomics). The latter is a consequence of the limited range of assets necessary for the execution of the related activities as well as the higher knowledge of the designer in performing 3D modelling tasks.

Figure 10 summarises the results obtained from the two projects investigated during the testing activity (in the form of packaging renderings) together with the timelines and the development iterations. It can be noticed that the output produced during the P-SAR session, only re-elaborated using professional software, is very close to the final one in terms of assets and configuration. This means that when used in a collaborative activity, the P-SAR platform allows the user to quickly reach the optimal solution (not true in the other project where multiple layouts were developed simultaneously during the first four iterations). From the top graph of Figure 10, it is also easy to see that a longer time elapsed between the last two iterations of the project with respect to the previous ones. This, as a consequence of missing information regarding the product and not related to the design process itself, is why the five metrics have also been referred to as the only early design phase.

The consideration of the early stage as separated from the entire development process of the products allows reducing the number of variables that can influence the results of the benchmark. In addition, being likewise positive, the metrics validate the use of the current P-SAR platform during the initial design activities where there are no requests for high-precision asset manipulations and high-quality rendering of the outputs but only to rapidly create and switch between different alternatives.

The feedback collected from the participants of the first collaborative session of P2 revealed other significant insights into the perceived functionalities of the SAR platform. Firstly, the availability of a design tool that potentially allows the exploration of an infinite number of variants of a single product worried the designers about the actual improvement of the development process. If not adequately supervised, in fact, the workflow of the single session can lead to product variants that do not comply with the project requirements initially detailed. A possible mitigation of this risk is limiting the number of assets uploaded in the session library to those strictly necessary to accomplish the design task. Following this idea, the preparatory work becomes even more complex and time consuming due to the necessity of thinking in advance about all the possible scenarios and needs of the future design session. Similarly, the devices adopted for displaying the platform interface were not always perceived as a benefit. Although the designers noted higher and more active participation of the clients during the collaborative session, they claimed that a shared interaction space could hinder the differences between the roles of the participants. This means that only the designers have the right expertise and experience to manipulate the digital contents that compose the external layout projected, while the clients must pay more attention to the result rendered by means of the mixed prototype.

7. Conclusions

The current research explored the adoption of IAP systems based on P-SAR technology as design tools for creative industries in the field of packaging design. The system implemented to perform this activity, called SPARK, is first presented from the point of view of its installation procedures to highlight the novel activities a designer should learn before its adoption in everyday practice. The analysis, despite its preliminary nature and the low statistical evidence of the results, was necessary for validating the initial assumptions about the platform’s benefits in the field. In this case, the test campaign was focused on the complete study of two complementary packaging developments where, as the main difference, only one had the possibility of organising a collaborative session supported by the P-SAR technology. Industrial-related parameters, like the lead time, number of iterations, man-hours and costs, have been collected and compared between the two activities to quantify the improvements concerning the use of the P-SAR. This positive benefit becomes even more evident by considering the early design phase only, where multiple design strategies were already ongoing.