Design and Development of A Virtual Dolphinarium For Children With Autism
Design and Development of A Virtual Dolphinarium For Children With Autism
Design and Development of A Virtual Dolphinarium For Children With Autism
IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 21, NO. 2, MARCH 2013
AbstractThe recent proliferation of virtual reality (VR) technology applications in the autism therapy to promote learning and
positive behavior among such children has produced optimistic results in developing a variety of skills and abilities in them. Dolphin-assisted therapy has also become a topic of public and research interest for autism intervention and treatment. This paper
will present an innovative design and development of a Virtual
Dolphinarium for potential autism intervention. Instead of emulating the swimming with dolphins, our virtual dolphin interaction program will allow children with autism to act as dolphin
trainers at the poolside and to learn (nonverbal) communication
through hand gestures with the virtual dolphins. Immersive visualization and gesture-based interaction are implemented to engage
children with autism within an immersive room equipped with a
curved screen spanning a 320 and a high-end five-panel projection system. This paper will also report a pilot study to establish
trial protocol of autism screening to explore the participants readiness for the virtual dolphin interaction. This research will have two
potential benefits in the sense of helping children with autism and
protecting the endangered species.
Index TermsAutism, dolphins, immersive and interactive
learning, nonverbal communication, virtual reality (VR).
I. INTRODUCTION
A. Autism Spectrum Disorder
Manuscript received March 09, 2012; revised July 04, 2012, September 14,
2012; accepted November 11, 2012. Date of publication January 24, 2013; date
of current version March 07, 2013. This work was supported by the Institute for
Media Innovation under the Seed Grant Scheme. Y. Cai and N. K. H. Chia are
first authors of the paper with no order difference.
Y. Cai, D. Thalmann, and N. M. Thalmann are with the Institute for
Media Innovation, Nanyang Technological University, 637553 Singapore (e-mail: [email protected]; [email protected]; [email protected]).
N. K. H. Chia and N. K. N. Kee are with the National Institute of Education,
637616 Singapore (e-mail: [email protected]; [email protected]).
Y. Cai is with the School of Mechanical and Aerospace Engineering,
Nanyang Technological University, 639798 Singapore (e-mail: myycai@ntu.
edu.sg).
J. Zheng is with the School of Computer Engineering, Nanyang Technological University, 639798 Singapore (e-mail: [email protected]).
Digital Object Identifier 10.1109/TNSRE.2013.2240700
than purposely broadened diagnostic criteria, coupled with deliberately greater public awareness and intentionally improved
case finding [5]. We became interested to do research for possible help of these children with the state-of-the-art of intervention and treatment for autism.
Although Bleuler [6] first coined the term autism, our understanding of the disorder with its various subtypes comes from
the pioneering works of Kanner [7] and Asperger [8]. Today,
the DSM-IV-TR [9] provides the most widely used definition
of autism. It is based on twelve diagnostic criteria categorized
under three areas describing autism as a disorder of neural
development characterized by impaired social interaction and
communication, and by restricted and repetitive behavior.
Overt symptoms from the age of six months are manifested by
age two or three. These symptoms tend to continue through
adulthood, though often in a muted form. As said, autism
is distinguished not by a single symptom, but by a characteristic triad of symptoms. Autism is also a highly variable
neurodevelopmental disorder first appearing during infancy
or childhood. Generally, it follows a steady course without
remission. The DSM-IV-TR has classified autism as one of the
five pervasive developmental disorders. The current diagnosis
of autism is based on the manifested behavior as described in
the DSM-IV-TR, not the cause or the mechanism.
Currently, there are several major theories and perspectives
on the causes of autism. For instance, the empathizing-systemizing theory explains that an individual with autism has affective or empathizing deficits. According to Myers et al. [10], empathizing involves two major steps: 1) the ability to attribute
mental states to other people as a natural way of understanding
them, and 2) having an automatic appropriate emotional response to other peoples mental states that makes one care about
their feelings [10, p.57]. However, such an individual may have
intact or even superior cognitive ability, known as systemizing,
to analyze and build systems so as to understand and predict the
functional behavior of impersonal events or inanimate or abstract entities. Executive dysfunction is another theory depicting
autism as a form of perseveration or result of an inability to shift
attention presumably to arise from some form of frontal lobe
brain damage. It explains that people with autism lack the control over conation (or conscious effort to carry out self-determined acts) and thus, they cannot plan [11]. The theory of weak
central coherence, however, claims the preference of a person
with autism for local detail over global processing. A fourth example is the theory of sensory integration deficits. It is based on
the premise that adequate processing and integration of sensory
information by the central nervous system plays a significant
role in an individuals ability to participate in daily activities
[12], [13]. Sensory processing difficulties have been reported to
CAI et al.: DESIGN AND DEVELOPMENT OF A VIRTUAL DOLPHINARIUM FOR CHILDREN WITH AUTISM
209
TABLE II
CLASSIFICATION OF I&T METHODS IN AUTISM THERAPY
TABLE I
FUNCTIONAL DEVELOPMENTAL LEARNING AND BEHAVIOR
210
IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 21, NO. 2, MARCH 2013
urge caution for the sake of the endangered species with dolphin-assisted therapy, there exist risks [27] in putting weak
children in water for therapy and in contact with the strong
animals even though they are well trained. Further, critics claim
that the therapy has potential for causing harm to participants
since there have been reports of injuries during dolphin-swim
programs [28].
D. IDM-Enabled Autism Therapy
There is proliferation of VR technology applications in
autism therapy specially designed to promote learning and
positive behavior among such children. Studies [29][32], [48]
have suggested that VR technology application has shown
optimistic results in developing a variety of skills and abilities
(e.g., social and life skills, joint attention and communication)
in children with autism. Weiss et al. [33] presented an interesting game Join-In Suite for children with autism using a
multi-user tabletop shared active surface. According to Alers
and Barakova [34], children with autism like technological
gadgets or contraptions and logical thinking since they are stimulated and challenged to communicate using them [16], [35],
[36]. One explanation for their technophilic behavior is these
children with autism may possess strong visual modality that
accounts for their fascination with and propensity for learning
from videos and computer games [37][39] delivered through
electronic screen display [44], [40]. Another explanation can
be found in the high systemizing ability of these children with
autism [10], perhaps with particular preferences for mechanical
(e.g., machines and tools) and abstract (e.g., mathematics or
computer programs) systems.
E. Research Objectives
VR research has been promoted positively in the recent years
for the purpose to assist learning of children with autism. Dolphin-assisted therapy has also become a topic of public and research interest for autism intervention and treatment. To our best
knowledge, there is no research reported on the use of virtual
dolphins to help children with autism to learn. In particular, instead of emulating the swimming with dolphins, we are interested in the design and development of Virtual Dolphinarium to
assist children with autism to learn nonverbally communication
through gesturing. As an initial effort, a virtual dolphin interaction (VDI) will be designed allowing children with autism to
act as dolphin trainers at the poolside. Immersive visualization
and gesture-based interaction will be developed to engage children with autism to establish joint attention and to learn (nonverbal) communication. The trial protocol of autism screening
from a pilot study will be reported with a small number of children with autism to explore their readiness for this VDI. The potential benefits of the research include 1) the elimination of the
risks of children with autism in the physical dolphin encounters;
2) protection of the endangered species by introducing virtual
dolphins in alternative dolphin-assisted therapy. Eventually, we
hope to have the VDI in the Virtual Dolphinarium as a new and
effective method to aid children with autism in their functional
development learning and behavior.
Fig. 2. (a) Dolphin show at UWS Dolphin Lagoon and (b) UWS Pink Dolphin
Encounter Program.
CAI et al.: DESIGN AND DEVELOPMENT OF A VIRTUAL DOLPHINARIUM FOR CHILDREN WITH AUTISM
211
Fig. 3. (a) The immersive room, (b) Virtual Underwater World visualized on
the 3-D screen, and (c) Virtual Dolphinarium visualized on the screen.
212
IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 21, NO. 2, MARCH 2013
Fig. 4. (a) A professional instructor training a real dolphin in the UWS Dolphin
Lagoon and (b) a child with autism acting as a dolphin trainer in the Virtual
Dolphinarium.
Fig. 5. Engaging a child with autism through 3-D immersive visualization and
hand gesturing.
gage children with autism. They gain high quality immersive visual and interactive experience through the VDI program with
the Virtual Dolphinarium. In Fig. 5, a child with autism is interacting with the virtual pink dolphins in the Virtual Dolphinarium
to learn to communicate with appropriate hand gestures.
C. Increasing Nonverbal Communication Through Hand
Gesturing
Selected hand gestures used by the dolphin trainers with the
UWS Dolphin Lagoon are implemented to enable hand gesture
control with the virtual pink dolphins. The gesture is fairly easy
to learn and its design is in consistent with many gesture controlled games supported by Microsoft Kinect.
To act as dolphin trainers, children with autism can stand in an
area close to the screen of the immersive room and the graphics
will visualize on the 3-D screen the Virtual Dolphinarium immersively and allow them doing gesturing at poolside. Each
time, when they successfully command a dolphin for an action;
several effects will be activated including dolphin sounds, water
wave and splash, etc. Typically, a specific gesture activated dolphin show will last about 3060 s. After the completion of an
interaction cycle, they can try other gesture options for different
dolphin interaction. The options of the gestures can be single or
mixed input based. These children can combine two hand gestures to initiate mixed dolphins shows.
Together with the Microsoft Kinect hardware device, the
latest Natural Interface Programming is applied in this project
to facilitate hand gesture detection and recognition for children
with autism to act as dolphin trainers in the Virtual Dolphinarium. Fig. 6(a) shows a hand gesture of greeting and Fig. 6(b)
shows a child with autism using the greeting gesture to communicate with a virtual pink dolphin. Fig. 7 shows another child
CAI et al.: DESIGN AND DEVELOPMENT OF A VIRTUAL DOLPHINARIUM FOR CHILDREN WITH AUTISM
213
speech and the remaining five are verbal and could hold a decent conversation with others though rather egocentric. Parents
or guardians of these selected children were informed and their
written consents were obtained.
B. Two Tests
Fig. 6. (a) Greeting with left hand waving and (b) a virtual pink dolphin responding by spraying water to the greeting of a child with autism.
Fig. 7. A child with autism gesturing to control a virtual dolphin in the Virtual
Dolphinarium. (a) Gesture for dolphin to spin. (b) Gesture for dolphin to leap.
(c) Gesture for dolphin to swim.
214
IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 21, NO. 2, MARCH 2013
RESULTS
TABLE III
NVIQS AND AQS
OF
TABLE IV
SCREENING RESULTS FOR FUNCTIONAL DEVELOPMENTAL LEARNING
AND BEHAVIORAL SUITABILITY OF THE VDI
70 and 129. Higher AQ of a participant likely indicates more severe form of autism with the participant displaying symptoms
pertaining to the triad of impairments in autism.
Table IV shows the results based on six screening tasks to
determine the functional developmental learning and behavioral
suitability of the 15 participants in preparation for the VDI.
In this study, the 15 participants demonstrated different levels
of interests to interact with the virtual pink dolphins. Eight
male participants had no problem with wearing 3-D glasses and
they showed strong interest with enthusiasm and volunteering
throughout the whole session of VDI. While one of the twin
girls (Participant 4F) was totally not interested and refused to
enter the immersive room even with her parents mediation,
the other twin girl (Participant 3F) showed initial interest after
her parents mediation and interacts with the virtual dolphins.
She was amenable to wear the 3-D glasses for about 5 min to
interact with the dolphins. In terms of performance, Participants
1M, 2M, and 5M were able to actively learn and function with
minimal supervision; Participant 3F, 6M, and 12M were able to
learn and function but required prompting or some mediation
from time to time. Participants 7M, 8M, 10M, 11M, and 13M
were overwhelmed by the VR experience and high parental
mediation was needed in order for them to be able to learn and
function throughout the session. Participants 4F, 9M, 14M, and
15M were overwhelmed by the VR experience such that even
with parental mediation, they refused to cooperate and were
unable to learn and function (see also Table V).
V. DISCUSSION AND CONCLUSION
This article starts with an introduction on autism spectrum
disorder followed by a review on intervention, treatment and
therapy for children with autism. The CCAS conceptual framework is used in this research. VR has been studied as a new
CAI et al.: DESIGN AND DEVELOPMENT OF A VIRTUAL DOLPHINARIUM FOR CHILDREN WITH AUTISM
TABLE V
OBSERVATION SCORING FOR BEHAVIOR IN LEARNING
FUNCTIONING DURING THE VDI
AND
215
into simulated dolphin lagoon. Children with autism could interact in real-time with Pink Dolphins within the Virtual Dolphinarium. Emphasis was placed on teaching these children hand
gesture-based communication in a fun way by enabling them to
play the role of a dolphin trainer. This research also explores
the use of stereographic visualization and hand gesturing as a
way to engage children with autism in poolside activities. The
Virtual Dolphinarium has advantages of controllability, replicability, and being programmable for such children to play with
the virtual dolphins. In other words, the VDI program allows instruction to be delivered with consistency and can be repeated an
infinite number of times without degradation of fidelity which
is difficult for physical dolphin encounters to achieve. Besides,
VR technology is capable of providing a wide range of situations and scenarios that can be used to teach or impart certain
basic hand gestures that form functional sign language to children with autism. Also VR technology can simulate scenarios
that are too risky or impractical especially for children with
autism to experience in real life. Hence, it provides ample opportunities for them to practice without having to worry of being
hurt or injured.
TABLE VI
PHYSICAL DOLPHIN ENCOUNTER VERSUS VDI
media to administer autism intervention and treatment. An innovative design of Virtual Ddolphinarium was presented to engage
children with autism for nonverbal communication through gesturing. Multiple senses such as vision and hearing, and hand
gesture etc. built in the Virtual Dolphinarium were discussed.
A pilot study whose tests require the use of skills of Cognition,
Conation, Affect, and Sensation was also discussed. The relation between the pilot results and the CCAS conceptual framework can be easily inferred. Table VI shows a comparative view
between physical dolphin encounter and VDI. As can be seen
from the table, the VDI and the physical dolphin encounter are
somehow complementary.
A. Novelty
We have developed a Virtual Dolphinarium providing a participating child with autism the sense of being totally immersed
ACKNOWLEDGMENT
The authors would like to thank M. X. Wu, D. J. Lu, B. Y.
Yang, and many others for their helps in this project. The project
team would also like to thank the support from the Underwater
World Singapore and A. G. T. Ng, the principal therapist from
the LDcentre, who has kindly arranged for parents of those children from the Pink Dolphin Encounter for Special Children program to participate in the study.
216
IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 21, NO. 2, MARCH 2013
REFERENCES
[26] T. L. Humphries, Effectiveness of dolphin-assisted therapy as a behavioral intervention for young children with disabilities, Bridges,
vol. 1, no. 6, pp. 19, 2003.
[27] C. Williamson, Can swimming with dolphins be a suitable treatment?, Develop. Med. Child Neurol., vol. 50, no. 6, p. 477, 2008.
[28] Humane Soc. U.S., Swim-with-dolphins attractions Sep. 25, 2009
[Online].
Available:
http://www.humanesociety.org/issues/captive_marine/facts/swim_dolphins.html
[29] D. W. Austin, J. M. Abbott, and C. Carbis, The use of virtual reality hypnosis with two cases of autism spectrum disorder: A feasibility
study, Contemp. Hypnosis, vol. 25, no. 2, pp. 102109, 2008.
[30] U. Lahiri, Z. Warren, and N. Sarkar, Design of a gaze-sensitive virtual
social interactive system for children with autism, IEEE Trans. Neural
Syst. Rehabil. Eng., vol. 19, no. 4, pp. 443452, 2011.
[31] D. Moore, Y. Cheng, P. McGrath, and N. J. Powell, Collaborative
virtual environment technology for people with autism, Focus Autism
Other Develop. Disabil., vol. 20, no. 4, pp. 231243, 2005.
[32] B. A. Mineo, W. Ziegler, S. Gill, and D. Salkin, Engagement with
electronic screen media among students with autism spectrum disorders, J. Autism Develop. Disorders, vol. 39, pp. 172187, 2009.
[33] P. L. Weiss, S. Cobb, E. Gal, L. Millen, T. Hawkins, T. Glover,
D. Sanassy, S. Eden, L. Giusti, and M. Zancanaro, Usability of
technology supported social competence training for children on the
autism spectrum, in Int. Conf. Virtual Rehabil., Zurich, Switzerland,
Jun. 2729, 2011, pp. 18, 2011.
[34] S. H. M. Alers and E. I. Barakova, Multi-agent platform for development of educational games for children with autism, in Proc. Int.
IEEE Consum. Electron. Soc. Games Innovat. Conf., U.K., Aug. 2528,
2009, pp. 4753.
[35] J. W. Jacobson, R. M. Foxx, and J. A. Mulick, Controversial Therapies
for Developmental Disabilities: Fad, Fashion and Science in Professional Practice. Mahwah, NJ, USA: Lawrence Erlbaum, 2005.
[36] Committee on Educational Interventions for Children with Autism.
Division of Behavioral and Social Sciences and Education, in Educating Children With Autism. Washington, DC, USA: Nat. Acad.
Press, 2001.
[37] N. K. N. Kee and N. K. H. Chia, , Y. Cai, Ed., Mediated and engaged
learning using COTS video games in ASD special education, in IDM
and Virtual Reality for Education in Virtual Learning Environment.
New York, NY, USA: Nova Sciences, 2010, pp. 140159.
[38] M. Heiman, K. E. Nelson, T. Tjus, and C. Gilberg, Increasing reading
and communication skills in children with autism through an interactive multimedia program, J. Autism Develop. Disorders, vol. 25, no.
4, pp. 459480, 1995.
[39] B. Nally, B. Houlton, and S. Ralph, The management of television and
video by parents of children with autism, Autism, vol. 4, pp. 331337,
2000.
[40] T. Buggey, Video self-modeling applications with students with
autism spectrum disorder in a small private school setting, Focus
Autism Other Develop. Disabil., vol. 20, pp. 5263, 2005.
[41] H. C. Shane and P. D. Albert, Electronic screen media for persons with
autism spectrum disorders: Results of a survey, J. Autism Develop.
Disorders, vol. 38, no. 8, pp. 14991508, 2008.
[42] K. Watanabe and J. Lee, Dolphin encounter for special children
(DESC): Proposal for standard procedures: Special dolphin program
for children with autism, down syndrome, and physically disability,
Underwater World Singapore, Singapore, 2004.
[43] N. K. H. Chia, N. K. N. Kee, K. Watanabe, and P. T. C. Poh, An investigation on the effectiveness of dolphin encounter for special children
(DESC) program for children with autism spectrum disorder, J. Am.
Acad. Special Ed. Professionals, no. 3, pp. 5787, Fall 2009.
[44] L. Brown, R. J. Sherbenou, and S. K. Johnsen, Test of Nonverbal Intelligence, 3rd ed. San Antonio, TX, USA: Pearson, 1997, (TONI-3).
[45] B. Siegel, The World of the Autistic Child: Understanding and Treating
Autistic Spectrum Disorders. New York, NY, USA: Oxford Univ.
Press, 1996.
[46] J. Gilliam, Gilliam Autism Rating Scale Examiners Manual. Austin,
TX, USA: Pro-Ed, 1995.
[47] S. Cobb, S. Parsons, L. Millen, R. Eastgate, and T. Glover, Design
and development of collaborative technology for children with autism:
COSPATIAL, presented at the Int. Technol., Edu. Develop. Conf., Valencia, Spain, Mar. 810, 2010.
[48] S. Parsons, P. Mitchell, and A. Leonard, The use and understanding of
virtual environments by adolescents with autistic spectrum disorders,
J. Autism Develop. Disorder, vol. 34, no. 4, pp. 449466, 2004.
CAI et al.: DESIGN AND DEVELOPMENT OF A VIRTUAL DOLPHINARIUM FOR CHILDREN WITH AUTISM
217
Nadia M. Thalmann is currently Professor and Director of the Institute for Media Innovation, Nanyang
Technological University, Singapore. She is also the
Founder and Director of the MIRALab, an interdisciplinary lab in human computer animation, University
of Geneva, Geneva, Switzerland. She is Editor-inChief of the journal The Visual Computer (SpringerVerlag). Her research area is the modelling of Virtual
Humans and Social Robots including mixed realities.
Prof. Thalmann has received many artistic and scientific awards, among them, the 2012 Humboldt Research Award, and two Doctor honoris Causa (from University of Hanover in
Germany and from the University of Ottawa in Canada).