Computer Science and Information Systems 2022 Volume 19, Issue 2, Pages: 1047-1073
https://doi.org/10.2298/CSIS211123013L
Full text (
1615 KB)
Cited by
A low-cost AR training system for manual assembly operations
Lavric Traian (IP Paris - Telecom SudParis, Evry, France + ELM LEBLANC SAS, Drancy, France), [email protected]; traian.lavric@fr
Bricard Emmanuel (IP Paris - Telecom SudParis, Evry, France), [email protected]
Preda Marius (ELM LEBLANC SAS, Drancy, France), [email protected]
Zaharia Titus (ELM LEBLANC SAS, Drancy, France), [email protected]
This research work proposes an AR training system adapted to industry, designed by considering key challenges identified during a long-term case study conducted in a boiler-manufacturing factory. The proposed system relies on lowcost visual assets (i.e., text, image, video, and predefined auxiliary content) and requires solely a head-mounted display (HMD) device (i.e., Hololens 2) for both authoring and training. We evaluate our proposal in a real-world use case by conducting a field study and two field experiments, involving 5 assembly workstations and 30 participants divided into 2 groups: (i) low-cost group (G-LA) and (ii) computeraided design (CAD)-based group (G-CAD). The most significant findings are as follows. The error rate of 2.2% reported by G-LA during the first assembly cycle (WEC) suggests that low-cost visual assets are sufficient for effectively delivering manual assembly expertise via AR to novice workers. Our comparative evaluation shows that CAD-based AR instructions lead to faster assembly (-7%, -18% and -24% over 3 assembly cycles) but persuade lower user attentiveness, eventually leading to higher error rates (+38% during the WEC). The overall decrease of the instructions reading time by 47% and by 35% in the 2nd and 3rd assembly cycles, respectively, suggest that participants become less dependent on the AR work instructions rapidly. By considering these findings, we question the worthiness of authoring CAD-based AR work instructions in similar industrial use cases.
Keywords: augmented reality, training, content authoring, work instructions, assembly, user study, industry 4.0
Show references
Bellalouna, F.: Industrial use cases for augmented reality application. In: 2020 11th IEEE International Conference on Cognitive Infocommunications (CogInfoCom). pp. 10-18 (2020), https://doi.org/10.1109/CogInfoCom50765.2020.9237882
Bosch, T., Könemann, R., Cock, H., Rhijn, G.: The effects of projected versus display instructions on productivity, quality and workload in a simulated assembly task. In: Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA ’17. p. 412-415. Association for Computing Machinery, New York, NY, USA (2017), https://doi.org/10.1145/3056540.3076189, dOI:
Brooke, J.: System Usability Scale (SUS). https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html (1986), online; accessed 2021-07-06
Caudell, T., Mizell, D.: Augmented reality: an application of heads-up display technology to manual manufacturing processes (2003), https://doi.org/10.1109/hicss.1992.183317
Dey, A., Billinghurst, M., Lindeman, R., Swan, J.: A systematic review of 10 years of augmented reality usability studies: 2005 to 2014. Front. Robot. AI 5 (2018), https://doi.org/10.3389/frobt.2018.00037
Egger, J., Masood, T.: Augmented reality in support of intelligent manufacturing - a systematic literature review. Comput. Ind. Eng 140, 106195, (2020-02), https://doi.org/10.1016/j.cie.2019.106195
Funk, M., Kosch, T., Schmidt, A.: Interactive worker assistance: Comparing the effects of insitu projection, head-mounted displays, tablet, and paper instructions. UbiComp 2016 - Proc. 2016 ACM Int. Jt. Conf. Pervasive Ubiquitous Comput pp. 934-939, (2018-07), https://doi.org/10.1145/2971648.2971706
Gabbard, J., Fitch, G., Kim, H.: Behind the glass: Driver challenges and opportunities for ar automotive applications. Proc. IEEE 102(2), 124-136, (2014), https://doi.org/10.1109/JPROC.2013.2294642
Gattullo, M., Evangelista, A., Uva, A., Fiorentino, M., Gabbard, J.: What, how, and why are visual assets used in industrial augmented reality? a systematic review and classification in maintenance, assembly, and training (from 1997 to 2019. IEEE Trans. Vis. Comput. Graph 2626(c), 1-1, (2020), https://doi.org/10.1109/tvcg.2020.3014614
Hahn, J., Ludwig, B.,Wolff, C.: Augmented reality-based training of the pcb assembly process. In: Proceedings of the 14th International Conference on Mobile and Ubiquitous Multimedia (MUM ’15. p. 395-399. Association for Computing Machinery, New York, NY, USA (2015), https://doi.org/10.1145/2836041.2841215, dOI:
Hanson, R., Falkenström, W., Miettinen, M.: Augmented reality as a means of conveying picking information in kit preparation for mixed-model assembly. Comput. Ind. Eng 113(August), 570-575, (2017-11), https://doi.org/10.1016/j.cie.2017.09.048
Hart, S.G.: Nasa-Task Load Index (NASA-TLX); 20 Years Later. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting. p. 904-908 (2016), https://doi.org/10.1177/154193120605000909.K
Irawati, S., Green, S., Billinghurst, M., Duenser, A., Ko, H.: An evaluation of an augmented reality multimodal interface using speech and paddle gestures. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics (January), 272-283, (2006), https://doi.org/10.1007/11941354_28
Kim, K., Billinghurst, M., Bruder, G., Duh, H.,Welch, G.: Revisiting trends in augmented reality research: A review of the 2nd decade of ismar (2008-2017. IEEE Trans. Vis. Comput. Graph 24(11), 2947-2962, (2018), https://doi.org/10.1109/TVCG.2018.2868591
Lavric, T., Bricard, E., Preda, M., Zaharia, T.: An AR Work Instructions Authoring Tool for Human-Operated Industrial Assembly Lines. In: 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR. pp. 174-183, (2020), https://doi.org/10.1109/AIVR50618.2020.00037
Lavric, T., Bricard, E., Preda, M., Zaharia, T.: Exploring low-cost visual assets for conveying assembly instructions in ar. In: 2021 International Conference on INnovations in Intelligent SysTems and Applications (INISTA. pp. 1-6, (2021), https://doi.org/10.1109/INISTA52262.2021.9548570
Lavric, T., Bricard, E., Preda, M., Zaharia, T.: An industry-adapted ar training method for manual assembly operations. In: HCI International 2021 - Late Breaking Papers: Multimodality, eXtended Reality, and Artificial Intelligence. pp. 282-304. Springer International Publishing, Cham (2021), https://doi.org/10.1007/978-3-030-90963-5_22
Lee, G., Nelles, C., Billinghurst, M., Kim, G.: Immersive Authoring of Tangible Augmented Reality Applications Introduction Application Domain Analysis Immersive Authoring Design. Ismar (2004), https://doi.org/10.1109/ISMAR.2004.34
Lee, G.A., Ahn, S., Hoff, W., Billinghurst, M.: Enhancing first-person view task instruction videos with augmented reality cues. In: 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). pp. 498-508 (2020), https://doi.org/10.1109/ISMAR50242.2020.00078
Li, W., Wang, J., Jiao, S., Wang, M., Li, S.: Research on the visual elements of augmented reality assembly processes. Virtual Real. Intell. Hardw 1(6), 622-634, (2019-12), https://doi.org/10.1016/j.vrih.2019.09.006
Lopik, K., Sinclair, M., Sharpe, R., Conway, P., West, A.: Developing augmented reality capabilities for industry 4.0 small enterprises: Lessons learnt from a content authoring case study. Comput. Ind 117, 103208, (2020-05), https://doi.org/10.1016/j.compind.2020.103208
Lorenz, M., Knopp, S., Klimant, P.: Industrial augmented reality: Requirements for an augmented reality maintenance worker support system. In: Adjun. Proc. - 2018 IEEE Int. Symp. Mix. Augment. Reality, ISMAR-Adjunct 2018. pp. 151-153, (2018), https://doi.org/10.1109/ISMAR-Adjunct.2018.00055
Lu, F., Davari, S., Lisle, L., Li, Y., Bowman, D.: Glanceable ar: Evaluating information access methods for head-worn augmented reality. In: 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Mar. pp. 930-939, (2020), https://doi.org/10.1109/VR46266.2020.00113
Marquardt, A., Trepkowski, C., Eibich, T., Maiero, J., Kruijff, E., Schoning, J.: Comparing non-visual and visual guidance methods for narrow field of view augmented reality displays. IEEE Trans. Vis. Comput. Graph pp. 1-1, (2020), https://doi.org/10.1109/tvcg.2020.3023605
Martinetti, A., Marques, H., Singh, S., Dongen, L.: Reflections on the limited pervasiveness of augmented reality in industrial sectors. Applied Sciences 9. 3382 (2019), https://doi.org/10.3390/app9163382
Masood, T., Egger, J.: Augmented reality in support of industry 4.0-implementation challenges and success factors. Robotics and Computer-Integrated Manufacturing 58, 181-195 (2019), https://www.sciencedirect.com/science/article/pii/S0736584518304101
Masood, T., Egger, J.: Adopting augmented reality in the age of industrial digitalisation. Comput. Ind. 115(C) (feb 2020), https://doi.org/10.1016/j.compind.2019.07.002
Mengoni, M., Ceccacci, S., Generosi, A., Leopardi, A.: Spatial augmented reality: An application for human work in smart manufacturing environment. Procedia Manuf 17, 476-483, (2018), https://doi.org/j.promfg.2018.10.072
Merino, L., Schwarzl, M., Kraus, M., Sedlmair, M., Schmalstieg, D., Weiskopf, D.: Evaluating mixed and augmented reality: A systematic literature review (2009-2019 (2020-10), https://doi.org/10.1109/ISMAR50242.2020.00069.
Microsoft: HoloLens 2 - AR Headset. https://www.microsoft.com/en-us/hololens/hardware (2021), online; accessed 2021-07-06
Microsoft: Introducing instinctual interactions. https://docs.microsoft.com/en-us/windows/mixed-reality/design/interaction-fundamentals (2021), online; accessed 2021-07-06
Microsoft: Microsoft Mixed Reality Toolkit v2.4.0. https://github.com/microsoft/MixedRealityToolkit-Unity/releases/tag/v2.4.0 (2021), online; accessed 2021-07-06
Microsoft: Mixed Reality Dynamics 365 Guides (2021), https://dynamics.microsoft.com/en-us/mixed-reality/guides/,, accessed: 27 April 2021.
Microsoft: Overview of authoring a guide in Dynamics 365 Guides. https: //docs.microsoft.com/en-us/dynamics365/mixed-reality/guides/ authoring-overview (2021), online; accessed 2021-07-06
Nicolai, T., Sindt, T., Kenn, H., Witt, H.: Case study of wearable computing for aircraft maintenance. 3rd Int. Forum Appl. Wearable Comput (June), 1-12, (2006)
Nizam, S., Abidin, R., Hashim, N., Lam, M., Arshad, H., Majid, N.: A review of multimodal interaction technique in augmented reality environment. Int. J. Adv. Sci. Eng. Inf. Technol 8(4-2), 1460, (2018-09), https://doi.org/10.18517/ijaseit.8.4-2.6824
Palmarini, R., Erkoyuncu, J.A., Roy, R., Torabmostaedi, H.: A systematic review of augmented reality applications in maintenance. In: Robotics and Computer-Integrated Manufacturing, vol. 49 (2018), https://doi.org/10.1016/j.rcim.2017.06.002
P.T.C.: Vuforia expert capture (2021), https://www.ptc.com/en/products/augmented-reality/vuforia-expert-capture, accessed:
Quandt, M., Knoke, B., Gorldt, C., Freitag, M., Thoben, K.: General requirements for industrial augmented reality applications. Procedia CIRP 72, 1130-1135, (2018), https://doi.org/10.1016/j.procir.2018.03.061
Sanna, A., Manuri, F., Lamberti, F., Paravati, G., Pezzolla, P.: Using handheld devices to support augmented reality-based maintenance and assembly tasks. In: 2015 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas. pp. 178-179,. NV (2015), https://doi.org/10.1109/ICCE.2015.7066370
Sethi, A., Sethi, S.: Flexibility in manufacturing: A survey. Int. J. Flex. Manuf. Syst 2(4), 289-328, (1990), https://doi.org/10.1007/BF00186471
Souza Cardoso, L., Mariano, F., Zorzal, E.: A survey of industrial augmented reality. Comput. Ind. Eng 139, 106159, (2019-11), https://doi.org/10.1016/j.cie.2019.106159
Tainaka, K., Fujimoto, Y., Kanbara, M., Kato, H., Moteki, A., Kuraki, K., Osamura, K., Yoshitake, T., Fukuoka, T.: Guideline and tool for designing an assembly task support system using augmented reality. In: 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). pp. 486-497. IEEE Computer Society, Los Alamitos, CA, USA (nov 2020), https://doi.ieeecomputersociety.org/10.1109/ISMAR50242.2020.00077
Tang, A., Owen, C., Biocca, F., Mou, W.: Comparative effectiveness of augmented reality in object assembly. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. p. 73-80. CHI ’03, Association for Computing Machinery, New York, NY, USA (2003), https://doi.org/10.1145/642611.642626
UnityTechnologies: Unity3d - 2019.4. https://unity3d.com/get-unity/download/archive (2021), online; accessed 2021-07-06
Uva, A., Gattullo, M., Manghisi, V., Spagnulo, D., Cascella, G., Fiorentino, M.: Evaluating the effectiveness of spatial augmented reality in smart manufacturing: a solution for manual working stations. Int. J. Adv. Manuf. Technol 94(1-4), 509-521, (2018), https://doi.org/10.1007/s00170-017-0846-4
Wang, X., Ong, S.K., Nee, A.: A comprehensive survey of augmented reality assembly research. Adv. Manuf 4(1), 1-22, (2016), https://doi.org/10.1007/s40436-015-0131-4