The Infinite Retina: Spatial Computing, Augmented Reality, and how a collision of new technologies are bringing about the next tech revolution

The Infinite Retina

Spatial Computing, Augmented Reality, and how a collision of new technologies are bringing about the next tech revolution

Irena Cronin

Robert Scoble

BIRMINGHAM - MUMBAI

The Infinite Retina

Copyright © 2020 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Producer: Ben Renow-Clarke

Acquisition Editor – Peer Reviews: Suresh Jain

Content Development Editor: Dr. Ian Hough

Technical Editor: Gaurav Gavas

Project Editor: Radhika Atitkar

Proofreader: Safis Editing

Indexer: Dominic Shakeshaft

Presentation Designer: Sandip Tadge

First published: May 2020

Production reference: 1060520

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-83882-404-4

www.packt.com

packt.com

Subscribe to our online digital library for full access to over 7,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe?

Spend less time learning and more time coding with practical eBooks and Videos from more than 4,000 industry professionals

Learn better with Skill Plans built especially for you

Get a free eBook or video every month

Fully searchable for easy access to vital information

Copy and paste, print, and bookmark content

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.Packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.

At www.Packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

Foreword

When I was young, I told my dad that I'd rather own a computer than a home. Of course, that was back when a 4K Data General Nova cost more than a house in Silicon Valley.

Today, just a few decades later, we all carry a computer that is thousands of times more powerful than the computers of my childhood. Computers used to be huge, but today's tech is getting so small, we can wear them. The $250 Apple AirPods Pros have more compute power inside them than an iPhone 4 and certainly a lot more than that Data General computer that cost tens of thousands of dollars back in the 1970s.

People often ask me what I would engineer if I were starting out today. When I worked at HP we built calculators and huge laser printers. Currently, there's tons of research being done in Spatial Computing, which includes the hardware and software surrounding Robotics, Autonomous Vehicles, Augmented Reality, and Virtual Reality. All of these bring interesting engineering problems that would keep me interested for sure. I see the potential in all of them to bring computing to people in a way that is ubiquitous, more powerful, and far more personal.

Spatial Computing will move computing from something that sits on a desk, or that you hold in your hands, to something you move around with. There, you'll be joined by robots, virtual beings, and assistants. I'm optimistic that Spatial Computing will be used to serve us with a deeper understanding of ourselves, and give us wild new ways to experience the world and new ways to become more productive in businesses and classrooms.

Children soon will have imagination machines on their faces. That will be in contrast to the toys I grew up with. One of my favorites was a transistor radio, which had six transistors (a modern phone has billions of transistors, by comparison). That radio brought the world to me. Spatial Computing will open up new worlds to today's kids that simply didn't exist when I was growing up.

To me I'm excited by what the children of today will do with these technologies and hope they remember that life is about happiness. Smiling and laughing is good. Hearing music in new ways is good. Telling new jokes to each other is good. Spatial Computing can do all that and can bring new ways to learn, work together, and travel the world to boot.

What Irena Cronin and Robert Scoble have done in their book The Infinite Retina is provide readers with an accessible, useful and entertaining guide to all of the latest advances in Spatial Computing and how this new technology is influencing and priming the way forward for a host of new innovations that will affect how we live our lives, how we go to work, and how we communicate with each other for years to come.

– Steve Wozniak

Contributors

About the authors

Irena Cronin serves as CEO of Infinite Retina, which provides research and business strategy to help companies succeed in Spatial Computing. She was previously CEO of Transformation Group, which advised decision makers on business strategies related to Artificial Intelligence, Augmented and Virtual Reality, Machine Learning, Facial Recognition, Robotics, Autonomous Vehicles, and related disruptive technologies. Previous to this, she worked for several years as an equity research analyst and gained extensive experience of evaluating both public and private companies.

Cronin has a joint MBA/MA from the University of Southern California and an MS in Management and Systems from New York University. She graduated with a BA from the University of Pennsylvania with a major in Economics (summa cum laude). She has near-fluent proficiency in Mandarin, intermediate in Japanese, and beginner in Korean.

Robert Scoble is Chief Strategy Officer at Infinite Retina and works with companies that are implementing Spatial Computing technologies. He is a futurist and technology strategist and the author of three books about technology trends, being the first to report on technologies from autonomous vehicles to Siri. Previous positions include being a strategist at Microsoft, a futurist at Rackspace, and the producer and host of a video show about technology at Fast Company.

About the reviewer

Robert Crasco, futurist, thought leader, influencer, social media manager, and occasional developer and designer. His focus is on Spatial Computing (Virtual Reality, Augmented Reality, Mixed Reality, 360 Video, VR180, Volumetric Capture, and virtual worlds), Artificial Intelligence, Machine Learning, Robotics, 5G and technology's impact on society. He has a background in computer science and marketing and has worked for various tech companies, including AT&T, Ziplink, NewsCorp, and iBasis. He was a contributing author on Convergence: How The World Will Be Painted With Data. He has over a decade experience working with virtual worlds. He currently works with technology-focused individuals and brands on social media management, social amplification, social promotion, content curation, and content creation.

Contents

Introduction

Spatial Computing – The New Paradigm

Exploring Technological Change

Transformation Is Coming

New Vision

Pervasive Change – Shopping, Healthcare, and Finance

New Ways to Learn

Meeting the Pioneers

Thinking Ahead

Starting the Journey

Part I: Why Spatial Computing and Why Now?

Prime Directive

What Makes Us Human?

Drivers and Benefits

Potential Dangers

Understanding the Natural World

Faster Wireless Leads to New Affordances

Data, Data, Data Everywhere

The Impulse to Capture, Understand, and Share

Dancing Into Different Worlds

The Frustrated Pioneer

Breakthroughs in Sensors

Driven to Improve

Hands-on Use

Beginning

Four Paradigms and Six Technologies

Introduction

The Four Paradigms

Paradigm One – The Arrival of the Personal Computer

Paradigm Two – Graphical Interfaces and Thinking

Paradigm Three – Mobile

Paradigm Four – Spatial Computing

Six Technologies

Optics and Displays

Wireless and Communications

Control Mechanisms (Voice, Eyes, and Hands)

Sensors and Mapping

Computing Architectures

Artificial Intelligence (Decision Systems)

Evolution

Part II: The Seven Visions

Vision One – Transportation Automates

A Future Vision

The Paths to Autonomy

Falling Car Transportation Costs

Autonomous Vehicle Round-up

Zoox

Ford

General Motors' Cruise

Uber

Waymo

Tesla

Lyft, Magna, and Aptiv

Hyundai, Pony.ai, Aptiv, and Aurora

Aurora

Driven and Oxbotica

Autonomous Trucks

Flying Cars (eVTOLs)

The Autonomous and Electric Culture Shock

The Future City

The Changing City

Auto Dealers

Smart Roads

Interstates

Factories

Real Estate

Remote Working

Travel Times and Ranges

Navigation

Green Thinking

AI Assistants

Entertainment

Fewer Cars

Data Bubbles Will Abound

Vision Two ‒ Virtual Worlds Appear

From 2D to 3D

Increasing the Bandwidth

Change is Coming

Emergence as a Human Hook

The Brain and VR

The Right Tool for the Job

The Glassholes Show the Way

Defeated Expectations

Where Did Things Go Wrong?

What It Means to Be Human

Embodiment, Immersion, Presence, Empathy, and Emergence

Embodiment and Immersion

Evolution

Facebook has the Edge

Market Evolution

2014's New Wave

Inside-out

Mistakes Were Made

The Spectrum ‒ Augmented Reality from Mobile to Mixed Reality

Closer Computers

The Origins and Limitations of Augmented Reality

Wearable AR

Too Much and Too Little

Getting Past the Setbacks

Vergence and Accommodation

Fast, Cheap, or Good

Differences To Be Aware Of

From Kinect to the HoloLens

Technology Takes a Leap

Patents and Problems

Volumetric and Light Field: Capturing Our World in a New Way

Shrinking Concepts, Expanding Horizons

Shooting Some Volumetric

Camera Magic

AR Clouds (3D Mapping) and Internet Infrastructure Including 5G

A New Kind of Map

Building a Map

Ready for 5G

New Audio Capabilities Arrive

Audio Augmentation

Spatial Audio

Gaming and Social VR Experiences

A World Without Rules

The Ripples of the VR Phenomenon

Building a VR Community

Great Expectations

Location-based VR

The Void

Dreamscape Immersive

Sandbox VR

Cinematic and Interactive Experiences

Festivals That Include VR

Marketing the Unmarketable

Engineering Experiences

Content Production and Previsualization

The Merging of the Virtual and the Real

Vision Three – Augmented Manufacturing

Robots and Other Augmentation

Augmented Reality Visions and Informed Robots

The Mirror World Comes to Factory Floors

The Human Robot

Vision Four ‒ Robot Consumers

What Drives Consumers?

Getting Emotional

Now in 3D

Retail Destruction? No, But Radical Change Underway

Trying to Keep Up

Looking Ahead

Variations on a Retail Theme

Amazon Go is Watching You

Spatial Computing for Logistics

Spatial Computing Delivery Methods – Robots, Drones, Future

In Your Living Room and Elsewhere ‒ Spatial Computing Marketing Experiences

Starting in the Living Room

A Personalized Experience

The VR World

Making an Impact

A No Install Needed AR World

The Past and The Future

Will Anything Remain the Same?

Vision Five ‒ Virtual Healthcare

The Virtualist

Fixing Humans – Surgery Preparation and Surgery

Brain Tricks/Virtual Medicine

The Virtual Coach and the Check Engine Light

Data, Data, Everywhere

Precision and Personalization

Vision Six – Virtual Trading and Banking

3D Data Visualization

Virtual Trading

ATM Security and Facial Payment Machines

Virtual Branch Functionality and Customer Service

Future Adoption

Vision Seven ‒ Real-Time Learning

Experiential Digital Learning

No Teachers, Books, or Movies? New Approaches in the Offing

Brainier Learning

The Future of Education

The Grabbable Classroom

Workplace Training and Simulation

A Virtual Arm for Human Resources

Our Learning Futures

Part III: The Spatial Business

The Always Predicted World

The Predictive Breakthrough

Glasses That Understand You

(Human) Assistance Not Required

Pervasive

Proactive

Connected

Data Dance With Seven Verticals

Transportation

Technology, Media, and Telecommunications (TMT)

Manufacturing

Retail

Healthcare

Finance

Education

The Bubbleland

The Bubble Network

Unified Learning, Shared Experiences

Pervasive and Proactive Future Assistance

Spatial Computing World-Makers

The Market Illuminator

The Pain Reducer

The Investing Visionary

The Immersive Genius

The Synthetics Craftsman

The Chip Innovator

The Future Flyer

Spatial Computing Paths

How Human?

Transformative Experience

Privacy and Security

On Facebook and Cambridge Analytica

OJ Simpson and Location Data

Apple and the FBI

Identity and Ownership

Blurred Faces on Google Street View

China and Facial Recognition

Issues with Facial Recognition in the US and Europe

Human Social Good

Epilogue

Index

Introduction

We live in a world that is today undergoing, or perhaps by the time you read this, has undergone, a shift due to the novel coronavirus disease, officially called COVID-19. COVID-19 has caused a change in our way of communicating and the way in which we conduct business, albeit, most everyone is still using two-dimensional visual communication apps, such as Zoom, versus three-dimensional visuals at this point and will be for the next few years. Following the outbreak in 2020, schools and universities were shut down for weeks, in some cases months (depending on the location), with classwork expected to be done online. Many venues closed and events were postponed or cancelled, from Disneyland, to conferences and music festivals, to baseball and all other live sports. Governments around the world ordered people to stay home. The world was not prepared for a pandemic like COVID-19.

As we stayed home for weeks, many of us discovered new ways to work, play, educate, entertain, and shop. We also saw new technology become more important, whether it be Artificial Intelligence being used to look at CAT scans, to look for the virus, or to create a vaccine, or new wearable computers used in China and elsewhere to diagnose people with the disease. Plus, having autonomous vehicles or robots preparing our food or delivering it seemed a lot smarter to us all of a sudden.

At home, music artists released performances for Virtual Reality (VR) headsets to keep fans entertained during the weeks when people had to stay home. New virtual events, some of which were being trialled in Virtual Reality, were announced to replace the physical conferences lost for the year. These are early signs of a set of new technologies that promise to deeply transform all of computing. We call this set of new technologies Spatial Computing.

Spatial Computing – The New Paradigm

Spatial Computing comprises all software and hardware technologies that enable humans, virtual beings, or robots to move through real or virtual worlds, and includes Artificial Intelligence, Computer Vision, Augmented Reality (AR), VR, Sensor Technology, and Automated Vehicles.

Seven industry verticals will see transformational change due to Spatial Computing: Transportation; Technology, Media, and Telecommunications (TMT); Manufacturing; Retail; Healthcare; Finance; and Education.

These changes are what is driving strategy at many tech companies and the spending of billions of dollars of R&D investment. Already, products such as Microsoft'sHoloLens AR headset have seen adoption in places from surgery rooms to military battlefields. Devices like this show this new computing paradigm, albeit in a package that's currently a little too bulky and expensive for more than a few of the hardiest early adopters. However, these early devices are what got us to be most excited by a future that will be here soon.

Our first experiences with HoloLens, and other devices like it, showed us such a fantastic world that we can predict that when this new world arrives in full, it will be far more important to human beings than the iPhone was.

We were shown virtual giant monsters crawling on skyscrapers by Metaio years ago near its Munich headquarters. As we stood in the snow, aiming a webcam tethered to a laptop at the building next door, the real building came alive thanks to radical new technology. What we saw at Metaio had a similar effect on Apple's CEO, Tim Cook. Soon, Apple had acquired Metaio and started down a path of developing Augmented Reality and including new sensors in its products, and the capabilities of this are just starting to be explored. Today's phones have cameras, processors, small 3D sensors, and connectivity far better than that early prototype had, and tomorrow's phones and, soon, the glasses we wear will make today's phones seem similarly quaint.

In Israel, we saw new autonomous drones flying over the headquarters of Airobotics. These drones were designed to have no human hands touching them. Robots even changed memory cards and batteries. New Spatial Computing technology enabled both to see each other and sense the world around us in new ways. The drones were designed to fly along oil pipelines looking for problems, and others could fly around facilities that needed to be watched. Flying along fences and around parking lots, their Artificial Intelligence could identify things that would present security or other risks. These drones fly day or night and never complain or call in sick.

Focusing on just the technology, though, would have us miss what really is going to happen to the world because of these technologies. Our cities and countryside will reconfigure due to automation in transportation and supply chains as robot tech drives our cars, trucks, and robots rolling down sidewalks delivering products. We'll spend more time in virtual worlds and metaverses. More of our interfaces, whether they are the knobs on our watches, cars, doors, and other devices, will increasingly be virtualized. In fact, many things that used to be physical may be virtualized, including stores and educational lessons from chemistry experiments to dissection labs.

Computing will be everywhere, always listening, always ready to talk back, and once we start wearing Spatial Computing glasses, visual computing will always be there ready to show us visualizations of everything from your new designs to human patterns in stores, on streets, and in factories and offices. Some call this invisible computing or ambient computing, but to us, these systems that use your eyes, voice, hands, and even your body as a controller are part of Spatial Computing.

At the same time, all this new computing is joined by radically fast new wireless technology in the form of 5G. The promises of 5G are threefold. First, we'll have more than a gigabit of data if we have the highest bitrates, and even the lowest rates promise to give us more bandwidth than current LTE phones. Second, wireless will soon add almost no latency, which means that as soon as you do something, like throw a football in a virtual game, it'll happen, even if you have many players viewing the football in real time. Third, 5G supports many more devices per tower, which means you will be able to live stream even a Taylor Swift concert with tens of thousands of fans filling a stadium.

When you combine 5G with all the new things under the Spatial Computing umbrella, you get a big bang. All of a sudden, cars can work together, one sending detailed 3D imaging of streets instantly to others behind it. New kinds of virtual games will be possible in the streets, where hundreds of people can play virtual football games in parks and other places. Crazy new virtual shopping malls appear where virtual celebrities show you around, and you can imagine new products directly in a virtual scan of your home and in other places as well.

A range of new capabilities will appear over the next few years in devices you wear over your eyes. There will be very light ones that are optimized for utility, showing you navigation, notifications, reminding you where you have left things, or nagging you to do some exercise or meditation to keep on top of your physical and mental health. There also will be heavier devices that will be more optimized for everything from detailed design or architecture work to entertainment and video game work. We can even imagine owning several different kinds of Spatial Computing devices, along with some smart contact lenses, that will let us go out on a date night without looking like we have any computing devices on at all.

As the 2020s dawn, we have VR devices that cost a few hundred dollars that are great for games and a few other things, like corporate training. On the more expensive side of the scale, we have devices that can be used by car designers, or even as flight simulators to train airline pilots. The expensive ones, though, will soon look as out of date as one of the first cell phones does today. By 2025, the computing inside will shrink to a fraction of the size of today's devices and the screens inside will be much sharper and capable of presenting virtual and augmented worlds to us that far exceed what we can experience today.

It is this next wave of devices that will usher in the paradigm shift in computing and in human living that we are discussing here. Already these changes are benefiting many enterprises, raising productivity. Inside many warehouses, hundreds of thousands of robots scurry about, moving products from trucks to packages. These new warehouses have evolved over the past decade and enable retailers to keep up with floods of new online orders that, back in 2000, were only dreamt of in futuristic books like this one.

The productivity gains will spread to many jobs. At Cleveland Clinic, surgeons are already using similar technology that shows them digital views from ultrasound, CAT scans, and other sensors. Like the warehouse worker who sees a blue line on the floor telling her how to find the product she's looking for, in this case, when a surgeon navigates to the right place to cut out a cancerous tumor, it lights up like a missile guidance system and tells the surgeon they are in the right place.

Other systems help workers phone a friend with new remote assistance features. This can help companies that have expensive machinery, or other work forces, including surgeons, architects, and engineers, save money. At some plants, the savings will be substantial. It took us 30 minutes to simply walk across the Boeing floor where it builds airliners. Asking someone for advice virtually might save someone an hour of walking just to come over and see your problem in a plant like that.

New devices let these remote helpers see what you are dealing with, and they can often show you visually what to do. Imagine trying to remove an engine while holding a phone or tablet in your hand. These systems, because they use wearable glasses, can let workers use both of their hands while talking and showing the remote assistant what is happening. The savings in downtime can be extreme. Imagine a problem that is causing a shutdown in a line at Ford. Every minute it is down, Ford loses about $50,000.

Even for salespeople and managers, the cost savings add up. A flight from San Francisco to Los Angeles usually costs about $700, including airfare, hotel, a decent meal, and an Uber/taxi ride or two. Increasingly, these meetings will be replaced by ones held in Virtual Reality. Two headsets are $800. In just a couple of virtual meetings, the headsets pay for themselves. Social VR from Facebook, Spatial, Microsoft, and others are rapidly improving to make these virtual meetings almost as good as physical ones. The time and cost saved will add up to big numbers, and workers will be happier. Workforces will be less likely to pick up viruses from travelers, too, which will also add up to big savings for corporations and reduced risks.

More lives could be saved, too. Mercedes-Benz had an Augmented Reality system built to show first responders how to cut apart a wrecked car. The app showed where fuel and electrical lines were so that firefighters working to free an accident victim wouldn't start a fire or electrocute themselves. This isn't the only example we have of this technology helping first responders, either through better training or by giving them various assistance on scene. One such system helps police gather evidence and then be able to recreate crime scenes for juries that they can virtually walk around.

Here, we've given you a taste of just how much the world is about to get reconfigured because of Spatial Computing technologies. Let's dig into the details of the chapters ahead.

Exploring Technological Change

Spatial Computing's technological change is laid out in Chapter 1, Prime Directive. Mobile phones soon will give way to headsets and glasses that bring computing to every surface. What is driving all of this new technology? We have a need for complex technologies to keep us around on this planet longer and in a more satisfied and productive state. What will drive us to build or buy new headsets, sensors, and vehicles, along with the connected systems controlled by Artificial Intelligence? Augmentation is coming, and that can mean a lot of different things, which we will explore.

We look back in Chapter 2, Four Paradigms and Six Technologies, at the previous three foundations of personal computing and include the new Spatial Computing paradigm. The six technologies discussed are those that enable Spatial Computing to work: Optics and Displays, Wireless and Communications, Control Mechanisms (Voice and Hands), Sensors and Mapping, Compute Architectures (new kinds of Cloud Computing, for instance), and Artificial Intelligence (Decision Systems).

It all started with the personal computer of the late 1970s. That paradigm shift was followed by graphical user interfaces and networking in the 1980s and the mobile phone and other devices that started arriving in the 1990s, culminating with the iPhone in 2007. Then, we look forward to the next paradigm and why it will be so different and why so many more people will be able to get more out of Spatial Computing than the laptops, desktops, and smartphones that came before.

Human/machine interfaces are radically changing, and we visit the labs that brought us the mouse to understand the differences between how humans interfaced with computers with keyboard and mice to how we'll interface with cloud computing that is hyper-connected by using voice, eyes, hands, and other methods, including even wearing suits with sensors all along our bodies. It's amazing to see how far we've come from the Apple II days, where there were very few graphics, to Spatial Computing where cameras see the real world, decipher it, and decide how to drive a car around in it.

Transformation Is Coming

Because of this new technology, cities and even the countryside will change, autonomous vehicle pioneers tell us in Chapter 3, Vision One – Transportation Automates. Soon you will tell your glasses, Hey, I need a ride, and you'll see your ride arrive. Sounds like Uber or Lyft, right? Look closer, there isn't a driver inside. Now think of the cost and the other advantages of that. Economists see that such a system could be a fraction of the cost, and could do many other things as well: Hey car, can you go pick up my laundry and then dinner for our family? The problem with such a world is that it is probable, many tell us, that we'll see much more traffic near cities as we use transportation to do new things, like pick up our laundry. This is why Elon Musk came up with the Boring Company to build tunnels under cities. We show some other solutions pioneers have come up with, including special roads for these vehicles and new kinds of flying vehicles that will whisk commuters into city centers, passing above all that new traffic.

Transportation soon will include more than just cars and trucks, too. Already, lots of companies are experimenting with new kinds of robots that will deliver products much more efficiently and, in a world where viruses are a new threat, without human hands touching them either. We talk with a company that is already rolling out such robots on college campuses and elsewhere.

Autonomous cars might look like they are rolling around the real world, but often they are developed by rolling them around inside a simulation. Simulations are how engineers are able to test out AI systems and come up with new ways to train the AIs. After all, you don't want to wreck 500 cars just to come up with how to handle someone running a red light, do you? If you walk around the simulations built by Nvidia and others, they look like real streets, with real-looking and acting traffic, pedestrians, and even rainwater and puddles after rain. This technology has many new uses other than training robots and autonomous vehicles, though. The technology inside is a radically different form of computing than was used to make Microsoft Windows for the past few decades, too.

Here, new AI systems fuse dozens of sensor and camera readings together and then look for patterns inside. Some of the cars rolling around Silicon Valley and other cities, like Phoenix, Arizona, have more than 20 cameras, along with half a dozen spinning laser sensors that see the world in very high-definition 3D.

Is that a stop sign or a yield sign? Humans are good at that kind of pattern recognition, but computers needed to evolve to do it, and we dig into how these systems work for autonomous cars and what else this kind of technology could be used for―maybe for playing new kinds of games or visiting new kinds of virtual amusement parks where virtual actors interact with you? How will such things be possible? Well, let's start with the huge amount of bandwidth that will soon appear as 5G rolls out and new devices show up on our faces and in our pockets to connect us to these new Spatial Computing systems. Yes, 5G can support these new kinds of games, but it also can tell cars behind you that there's a new hazard in the road that needs to be avoided.

New Vision

Games aren't the only things that better devices will bring. Chapter 4, Vision Two – Virtual Worlds Appear, provides details on Technology, Media, and Telecommunications, another of our seven industry verticals to be disrupted. We start out by detailing the different kinds of devices that are available to bring a spectrum of Spatial Computing capabilities to your face, from Virtual and Augmented Reality headsets to lightweight smart information glasses, and even contact lenses with displays so small that it will be very hard to tell that your friend is wearing one.

There are pretty profound trade-offs made as manufacturers bring devices to the market. VR headsets emphasize immersion, or the feeling you get when you see something beautiful wrapped all around you. Augmented Reality headsets focus on the virtual layer that they reveal on top of the real world. Often it's amazing and magical, albeit usually with less of that I'm in a dark movie theater with a huge screen feeling. Then there are a few other devices that focus mostly on being lightweight, bringing navigation and notification-style functionality. Our guide isn't designed to be comprehensive, but rather to you to understand the market choices that both businesses and consumers will have to soon make.

While cataloging the device categories, we also show some of the new entertainment capabilities that soon will come, which will be captured with new arrays of volumetric and light-field cameras. We visited several such studios and delivered you into a new entertainment world, one where you can walk around in, and interact with, objects and the virtual beings inside.

These new media and entertainment experiences are arriving with a bundle of novel technologies, from AR Clouds, which contain both 3D scans of the real world and tons of virtual things that could be placed on top, to complete metaverses where users can do everything from build new fun cities to play new kinds of games with their friends. In enterprises, they are already building a form of an AR Cloud, called a Digital Twin, which is changing a lot about how employees are trained, work together, and manage new kinds of factories.

We have visited the world's top manufacturing plants, and in many of them, we see new kinds of work being done with lots of robots that didn't exist just a few years ago, with workers walking around wearing new devices on their faces helping them learn or perform various jobs. In Chapter 5, Vision Three – Augmented Manufacturing, you'll learn about how Spatial Computing is changing how factories are even designed. Increasingly, these factory floors are using robots. The robots are different than they used to be, too. The older ones used to be kept in cages designed to keep humans away. Those can still be found welding, or like in Ford's Detroit factory, putting windshields into trucks. Newer robots work outside cages and sometimes, can even touch humans. These types of robots are called cobots because they cohabit with humans and can greatly assist workers.

In the upcoming years, humans will both be trained to work with these new robots using new VR and AR technologies as well as train the robots themselves in new headsets with new user interfaces that let humans virtually control factory floors. As these new Spatial Computing technologies are increasingly used on factory floors, they bring new capabilities, from virtual interfaces to physical machines and new productivity enhancers. For instance, in many of these systems, workers can leave videos, 3D drawings, and other scans, and other notes for workers on the next shift to see. Hey Joe, the cutting machine is starting to misbehave. I ordered a new motor for it so you can fix it when the line is down at 2 p.m.

Pervasive Change – Shopping, Healthcare, and Finance

The products that Joe makes will eventually be found in retail stores, many of which will be quite different than the ones that Sears ran decades ago. Walk into an Amazon Go store and look at the ceiling. You'll see hundreds of cameras and sensors aimed back at you, watching and categorizing every move you make. In the Go store, they charge you for the products you take without you having to talk to a checkout clerk or pull out a credit card, or really do anything other than walk out of the store. In Chapter 6, Vision Four – Robot Consumers, we detail these changes, along with others that make retail stores, even traditional ones, more efficient and better for both consumers and sales for producers, and useful new Augmented Reality technologies that make shopping at home much easier.

As we were finishing up this book, Apple announced a new iPad with a 3D sensor, and it demonstrated why people should buy one of these new iPads by showing off one of these futuristic shop-at-home experiences where the person holding the tablet could drop chairs and other items into their real home to see how they fit. The changes, though, don't stop at the shopping experience. Behind the scenes, these technologies are making things more efficient, helping logistics companies pick and pack products faster and getting them to stores faster and with fewer losses, and we detail how Spatial Computing is making a difference there.

Similar changes are underway in healthcare. They are so profound that Dr. Brennan Spiegel, Director of Health Research at Cedars-Sinai, says that we should expect a new kind of healthcare worker: The Virtualist. This new kind of practitioner will perform several roles: help patients prevent disease, help doctors deliver a new form of healthcare, and help the nurses, doctors, and other staff perform their jobs more efficiently using new Spatial Computing technology. For instance, let's say you need surgery. Well, a new surgery team at Cleveland Clinic is already using MicrosoftHoloLens 2 headsets to see inside you, thanks to images from scanners being visualized inside the surgeon's headgear. We discuss, in Chapter 7, Vision Five – Virtual Healthcare, how its system guides the surgeon to the right spot to cut out a patient's cancerous tumor.

That's just a tiny piece of what's happening in healthcare due to Spatial Computing technology. On another wing of the hospital, doctors are using VR to address mental illnesses and ailments from PTSD to dementia, with more applications on the way. At the University of Washington, they discovered it often is much better at treating pain than using opiates, which are much more dangerous, killing tens of thousands of Americans every year due to addiction. In other places, nurses noticed that patients going through tough procedures preceding childbirth felt a lot less pain, too, if they were watching a 360-video experience during the process. Other doctors even found ways to enhance athletes' perception. These brain tricks and virtual remedies have the capacity to significantly change healthcare. Pfizer's head of innovation told us that she views Augmented and Virtual Reality as the future of medicine.

How does all this work? Largely on data. New predictive systems will watch your health by having sensors look into your eyes, watch your vascular system, or blood streams, and sense other things, too, maybe to the point where they see that you are eating too much sugar or smoking too many cigarettes. One could even warn your doctor that you aren't taking your medicine or performing the exercises that she prescribed. Are we ready for new tough conversations with our doctors? Remember, in this future, maybe your doctor is a virtual being from whom you don't mind hearing the harsh truth. One of the studies we found showed that patients actually are much more honest about their mental health problems when talking with a virtual being, or in a chat that's run by Artificial Intelligence.

These AIs won't just be helping us keep our health on track, either. Similar systems might let us know about market changes that we need to pay attention to or, like banks already do when they notice that buying behavior is out of bounds, warn you about other things. The financial industry is generally a relatively conservative one, so adoption of Spatial Computing technologies there is contingent upon demonstrated and clear utility, and they must be additive to the bottom line. Currently, there is very little Spatial Computing that is being actively used there; however, the possibilities are very promising. In Chapter 8, Vision Six – Virtual Trading and Banking, we cover the future uses for Spatial Computing in the financial industry.

In this chapter, we review the functional areas where we think Spatial Computing will have its greatest impact, including 3D data visualization, virtual trading, ATM security and facial payment machines, and virtual branch functionality and customer service.

Someday soon, we may never go into a physical bank again due to these changes, but could the same happen with our schools? Already, teachers are using Augmented and Virtual Reality to teach all sorts of lessons, from chemistry experiments to math visualizations, to even virtual dissections of real-looking animals.

New Ways to Learn

COVID-19, though, showed us that sometimes we might need to rely completely on teaching virtually, and in Chapter 9, Vision Seven – Real-Time Learning, we talk with educators and others who are using technology aggressively to make learning more virtual. It isn't only for kids, either. Soon, because of automation, we'll need to retrain millions of adults around the world, and schools and universities are responding with new curricula, new learning programs for Virtual and Augmented Reality, and new support systems to enable even truck drivers to change careers. Speaking of careers, already at companies like Caterpillar, it is using Augmented Reality glasses to train workers to fix their expensive tractors in real time. Many new VR-based training systems are being developed, from simulators to help police learn how to deal with terrorist situations to ones that show quarterbacks how to perform better, to training at Walmart that shows retail workers how to manage stores better. Verizon even trained its retail store workers on what to do if they are being robbed using VR-based training. What if, though, the system could do even more, we asked, and predict what we might do next and assist us with that?

How can computers predict our next move? Well, truth be told, we are somewhat predictable. We buy groceries at the same store every week, visit the same gas stations, go to the same churches, schools, offices, movie theaters, laundries, and head home at pretty much the same time every evening.

Watching our friends, we can usually predict what they will order from menus or how they will complain when we try to get them off pattern. Ever try to take someone who prefers steak and potatoes to a sushi restaurant ? Can't we predict that they will have the same preferences tomorrow night? Yes, and so can computer algorithms, but Spatial Computing systems could soon know a lot more about us than even our best friends do, since they could watch every interaction, every product touched, every music song picked, and every movie watched. In Chapter 10, The Always Predicted World, we show how that data will be used in each of our seven disruptable industries to serve users in radically new ways.

Meeting the Pioneers

We predict that you will enjoy learning more about the seven people who are pushing Spatial Computing further in Chapter 11, Spatial Computing World-Makers. Instead of ratcheting through a list, we'd like to tell you why we picked the people here. One spends time in retail stores the world over and uses technology to help them become not only more profitable, but more customer-centric. Another developed Google's autonomous vehicle technology and has gone on to further build out a huge vision of the future of transportation. You'll meet one leader who, from their perch at Qualcomm, sees