F O R U M I N T E R A C T I O N T EC H N O L O G I E S

Envisioning, designing, and implementing the user interface require a comprehensive understanding
of interaction technologies. In this forum we scout trends and discuss new technologies with the potential
to influence interaction design. — Albrecht Schmidt, Editor

The Future of Tangible

User Interfaces

T
Lars Erik Holmquist, Northumbria University

he notion of tangible user adoption is expected in “more than 10 of HP Labs, Kimiko Ryokai of UC
interfaces (TUIs) was years” [2]. TUI has not appeared on Berkeley, Haiyan Zhang of Microsoft
introduced by Hiroshi the chart since, whereas many other Research, and Oren Zuckerman of
Ishii and Brygg Ullmer technologies listed as emerging at the miLAB [3]. Representing a variety
of the MIT Media same time in 2010, such as autonomous of viewpoints from academia and
Lab at the ACM CHI vehicles, social analytics, 3D printing, industry, we wanted to engage with
conference in 1997 [1]. and speech-to-speech translation, the proven success as well as the
The fundamental idea is that users have already become a commercial untapped opportunities of TUI, and
should be able to “grasp and manipulate reality. And indeed, compared with also understand what the potential
bits in the center of users’ attention by other forms of interaction that have impediments to its adoption might be.
coupling the bits with everyday physical been commercialized in the past two A number of themes emerged
objects and architectural surfaces” [1]. decades, such as touchscreens and voice during the panel that we argued
TUI has been highly influential in the assistants, the amount of successful would have unique implications for
HCI research community, and to date tangible interfaces is comparatively tangible interaction. It would have
the 1997 paper alone has been cited small. There could be multiple reasons, been impossible even for the great
almost 5,000 times. Special issues of including the relatively high cost of minds assembled (both on stage and
prominent journals have been devoted producing bespoke tangible devices, in the audience) to give any definitive
to tangibles, and there is an annual the lack of general-purpose interaction answers to these questions, but having
ACM conference on the topic: Tangible, paradigms, or even simply not enough an open dialogue helped crystalize some
Embedded, and Embodied Interaction marketing to the consumer market. of the most important issues for TUI
(TEI). At CHI 2019, Hiroshi Ishii going forward. The following reflects a
received the prestigious ACM CHI TUI THEMES synthesis of the some of the main panel
Lifetime Research Award for his work At the CHI 2019 conference in themes and my own thoughts on the
on tangibles, firmly establishing it as Glasgow, I co-organized a panel to subject:
one of the cornerstones of modern HCI discuss the past, present, and future Users. Who is the recipient or
research. And, speaking personally, of tangible user interfaces. In addition customer of future tangible user
that original tangible bits talk at CHI to Hiroshi Ishii himself, the panelists interfaces, and what settings will they
’97 (also my first CHI conference!) blew included Rafael “Tico” Ballagas be in? When TUI was introduced,
me away. To this day, it fundamentally personal computers were still limited
influences how I think about user mostly to desktops. Even laptops were
interfaces and interaction design. Insights comparatively rare, and WiFi in public
That said, the real-world impact →→ Tangible user interfaces (TUIs) places was basically unheard of. With
of TUI is still subject to some debate. have a rich history in academia the explosion of the mobile consumer
Several products and interfaces have and are one of the cornerstones Internet, and aided by portable devices
been launched that incorporate tangible of modern HCI research. such as smartphones and tablets,
interaction, but industry adoption →→ TUIs have many benefits and computer use has thoroughly dissipated
is not as prominent as the research opportunities, but there are out into the world—everything from
volume would indicate. In 2010, also impediments to their social media to reading, watching
Gartner’s Emerging Technologies mainstream adoption. movies, wayfinding, and shopping. To
Hype Cycle listed tangible user →→ One way forward could be find a place for tangibles in this wider
interfaces at the Technology Trigger bespoke, specialized tangible world of computer use—that is already
stage, which denotes promising new interfaces that perform per definition beyond the desktop—it
technologies where mainstream a specific task really well. will be important to create tangible

82 INTER ACTIONS S E P T E M B E R – O C T O B E R 2 0 19 @INTER ACTIONSMAG

 experiences that engage people in their and squishy versus dry and brittle. In launching yet another interactive
personal, social, and emotional lives. other words, TUI offers a whole new button on the Internet; consequently,
Embodiment. How do we then world of materials to build interactive neither design principles nor usability
engage all of the six human senses— experiences, truly beyond pixels. But standards for TUIs have had the
including proprioception, the sense realizing this potential will require same chance to emerge. One way
of one’s own body? The senses of equally new forms of input, output, forward would be a move toward more
touch, smell, and other modalities are and sensor technology to truly take standardized tangible UI components
still undervalued in commercial user advantage of such non-digital materials. from which physical interfaces can be
interface design, which has largely Process. How do we invent, built quickly, just like GUI builders
focused on the visual, more recently construct, and, perhaps most have made it easier to quickly produce
including voice. Even paradigms such important, evaluate tangible graphical interfaces.
as augmented and virtual reality experiences? While Internet companies Scalability. How can we make
still put the emphasis on pixels before can quickly produce new user tangible user interfaces—or embodied
the real world. By its nature, TUI thus experiences in apps or on the Web, and interaction in general—scale
has a unique opportunity to engage perform A-B testing with thousands of economically and realistically? After
the senses and provide a more embodied users literally at the flick of a switch, all, pixels are cheap, but atoms are
experience of digital systems, which producing and studying tangible devices comparatively expensive. This means
will help interactive systems become is much more laborious. Constructing that the cost for a tangible user interface
more naturally integrated into human a brand-new TUI requires much more may not equal the perceived benefit, and
activities. in the way of engineering than just a screen-based one may win out even if
I M A G E B Y M I T M E D I A L A B / TA N G I B L E M E D I A G R O U P

Materiality. Following on from the tangible user experience is arguably

embodiment, TUI potentially offers better. However, users are prepared to
a much richer materiality than that of pay large sums to own the latest phone
graphical interfaces. TUIs can be made TUI has a unique or the newest smartwatch, indicating
from substances like plastic, wood,
glass, metal, or even dynamic materials
opportunity to engage that if the value proposition is right,
people will pay.
such as liquids or living organisms like the senses and provide Sustainability. From an
plants—each offering a completely a more embodied environmental perspective, the
different tactile and sensory experience.
They can also be hot or cold to the
experience of digital tech industry is rapidly becoming
a worldwide problem, with greatly
touch, light or heavy, and even moist systems. increasing amounts of electronic waste
INTER ACTIONS. ACM.ORG S E P T E M B E R - O C T O B E R 2 0 19 INTER ACTIONS 83
 FORUM INTER ACTION TECHNOLOGIE S

ACM Transactions on
Internet of Things

ACM TIOT publishes

novel research
contributions and
experience reports
in several research
domains whose synergy
and interrelations
enable the IoT vision.
TIOT focuses on system
designs, end-to-end Figure 1. A visitor is greeted by the virtual Figure 2. A bespoke tangible interface
receptionist. controls the remote screen.
architectures, and
being produced every year. To take just are finally seeing mainstream adoption
enabling technologies, one example, the battery life in wireless after being well known and explored in
and on publishing earbuds such as Apple’s AirPods limits the research community for decades.
them to only a few years’ practical Given its undeniable influence in the
results and insights use, and replacing a worn-out battery academic world, TUI should be well
corroborated by a is not cost effective. Thus, the whole positioned to have a similar impact. On
item most likely gets thrown away, the other hand, some would argue that
strong experimental dangerous lithium-ion cells and all. The research also has value for its own sake
component. same goes for mobile phones and other and does not have to have any direct
battery-powered devices. This will commercial impact to be valuable.
obviously have a drastic impact on the Which takes us to the last point…
environment. Yet TUI by its very nature Inspiration. There is one aspect
implies the production and distribution of TUI research that is perhaps more
of more physical devices. Would a important than all else: the steady
world where TUI is widely popular see creation of new ideas and new impulses
an even greater increase in electronic in human-computer interaction. Ishii
junk, or can we create interfaces using himself once explained his creative
sustainable and recyclable components approach to research as putting on
and materials? new glasses that let you see the world
Impact. What creates the need and in a different way. If the research
push for tangible user interfaces? In community can inspire young students
many cases, the research community to see the world differently and try new
will come up with a novel vision or things, this in itself has a great value.
technology paradigm, then work on it
for decades to experiment, validate,
and come up with solutions, before
industry eventually picks it up. This If the research
For further information has been the case, for instance, with
community can inspire
the notion of ubiquitous computing,
or to submit your which arguably was a major influence students to see the
manuscript, both on the Internet of Things and world differently and
visit tiot.acm.org
the consumer smartphone and tablet
market [4]. Another example is virtual
try new things, this in
and augmented reality, both of which itself has a great value.
84 INTER ACTIONS S E P T E M B E R – O C T O B E R 2 0 19 @INTER ACTIONSMAG
 Northumbria University. He has been This bespoke tangible interface
working on the task of constructing solution has become a substantial
a tangible controller for an embodied success. The staff at the two buildings
telepresence system—essentially, a where the system is installed are using
two-way video communication system it literally every workday to cover for
to be used in a building reception. The each other, to greet and help visitors
concept is that a receptionist should be remotely, and even to just keep an eye
able to virtually greet and help visitors on the other reception, letting the
at a reception in a different building, screen act as a remote window in the
for instance covering for regular staff background. We believe that a large
when they need to step away from the factor in the success of the system is
reception to solve some urgent issue, or the naturalness of the interaction,
simply go on break (Figure 1). which does not force users to learn an
A novel feature of Nemeth’s system unfamiliar generic controller, but rather
is that the operator can physically move allows them to use one that is exactly
the screen of the remote system. This right for the task.
means that the screen—located in Thus, while there are many questions
another building—can be turned to see still to be answered on the future of
different parts of the lobby or to better tangible interfaces—a few of which
interact with a visitor. However, while I have indicated above—there are
a standard controller for such a system also clear paths to success. By solving
might be a joystick or a keypad, in early well-defined problems in a way that has
studies Nemeth found that this was not a clear benefit to users, tangible user
Figure 3. An old-fashioned hotel bell is used suitable for the staff that would be using interfaces can show their value and fully
to call attention. the system. enter the mainstream.
Instead, he constructed a highly
WHAT’S NEXT? specialized interface, where all functions Endnotes
1. Ishii, H. and Ullmer, B. Tangible bits:
All things said and done, where will TUI were combined and immediately
Towards seamless interfaces between
go in the immediate future? One way available (Figure 2). A single box people, bits and atoms. Proc. of the ACM
forward that was suggested during the includes an on/off switch for the live SIGCHI Conference on Human Factors
panel is the creation of bespoke tangible video, a microphone for communicating in Computing Systems. ACM, New
interfaces for very specific situations. with visitors, and, finally, a dedicated York, 1997, 234–241; http://dx.doi.
These would be interfaces to computer controller for the motion of the remote org/10.1145/258549.258715
2. Gartner Research. Hype Cycle for
systems that do only one thing but do screen. This takes the form of a half-
Emerging Technologies, 2010. Aug. 2,
them very well. One striking example sphere, made of polished wood. The 2010; https://www.gartner.com/en/
in recent years is Microsoft’s Surface controller is limited so that it can only documents/1414917/hype-cycle-for-
Dial, introduced in 2017. Billed as a turn the screen left or right, avoiding emerging-technologies-2010
“completely new way to interact with the risk of losing control of the remote 3. Holmquist, L.E., Zuckerman, O.,
technology and create in the most device. It also provides a one-to-one Ballagas, R., Ishii, H., Ryokai, K., and
natural, immersive way,” the Dial is a correspondence between the controller Zhang, H. The future of tangible user
interfaces. Extended Abstracts of the
repurposable tangible interface that can in the operator’s hand and the position
2019 CHI Conference on Human Factors
be adapted for a variety of functions. of the remote screen. Finally, the natural in Computing Systems. ACM, New
It can be configured as a “physical material gives a pleasant analog feel York, 2019, Paper panel02; https://doi.
shortcut” to perform a single specific that makes the system more naturally org/10.1145/3290607.3311741
function, for instance scrolling, which incorporated in the everyday task. 4. Weiser, M. The computer for the 21st
is performed very often yet requires Another bespoke tangible interface century. SIGMOBILE Mob. Comput.
cumbersome precision maneuvers can be found at the visitors’ end: an Commun. Rev. 3, 3 (July 1999), 3–11;
http://dx.doi.org/10.1145/329124.329126
when using a GUI (and in fact, scrolling old-fashioned hotel bell that is used to
has already been incorporated in some call the attention of the remote operator Lars Erik Holmquist is Professor of
mouse models as a dedicated separate (Figure 3). This has also proven to be an Innovation at Northumbria University. He is an
control). However, there are perhaps intuitive and simple way of performing internationally leading researcher in human-
other generic functions that could a single, dedicated function. At its core, computer interaction, interaction design, and
potentially be “tangiblized,” so to this is simply a switch connected with ubiquitous computing. His book Grounded
Innovation: Strategies for Creating Digital
speak, to unload specific burdens when USB and hidden inside the bell, but
Products (Morgan Kaufmann, 2012) provides a
operating a complex computer system. visitors immediately grasp the function, practical guide to the design-driven innovation
An example of this bespoke since it exactly corresponds to the usage process, with many examples drawn from his
approach was recently demonstrated you would expect—while seamlessly own research and elsewhere.
by Sam Nemeth, a Ph.D. student at interfacing with a digital system. →→ [email protected]

DOI: 10.1145/3352157  COPYRIGHT HELD BY AUTHOR. PUBLICATION RIGHTS LICENSED TO ACM. $15.00

INTER ACTIONS. ACM.ORG S E P T E M B E R - O C T O B E R 2 0 19 INTER ACTIONS 85