Immersive Virtual Reality in Improving Communication Skills in Children with Autism

Individuals in the Autism Spectrum often encounter situations where they have to respond to questions and situations that they do not know how to respond to, such as, questions asked by strangers including ones related to daily-life activities. A variety of research has been done to improve social and communication impairments in children with autism using technology. Immersive virtual reality is a relatively recent technology with a potential to bring an effective solution and used as a therapeutic tool to develop different skills. This paper presents a virtual reality solution to reduce the gap experienced by autistic children due to their inability to establish a communication. An interactive scenario-based system that uses role-play and turn-taking technique was implemented to evaluate and verify the effectiveness of immersive environment on the social performance of an autistic child. Preliminary testing of the system demonstrated the feasibility of VR-based system as a tool for improving the communication skill in Autism Spectrum Disorder (ASD) children. The results of the comparative usability study show the effectiveness of immersive VR in motivating and satisfying the autistic. Keywords—Autism spectrum disorder, communication skill, immersion, social performance, virtual reality. 146 http://www.i-jim.org Paper—Immersive Virtual Reality in Improving Communication Skills in Children with Autism

with the use of explanatory environments, the VE replicates as a real-life scenario in the form of human to avatar interaction [8]. Therefore, VR based systems can prove to be effective as a medium of educating children with ASD in controlled interactive environments [3] [8].
Using techniques of Role-play and Turn-taking are approaches that are efficient in establishing conversations that are similar to the virtual scenarios in reality. In a controlled environment, VEs provide the advantage of designing specific social situations where users can mimic avatar(s) by receiving instructions from them.
The study presented in this paper is an initial prototype of a VR-based system intended to explore the effectiveness of VR solution targeting communication skills in high-functioning children with autism. Therefore, the objective of this study has twofold: (1) the development and design of immersive VR-based system with verbal/nonverbal input interaction that is based on the time required for the user to respond in the VE task, (2) study the impact of immersion level on the performance of highfunctioning autistic users. The virtual reality experience presented to the subject in study using three different installations, four walls projection room CAVE (Computer Augmented Virtual Environment) that provides high-level of immersion and interactivity, Head Mounted Display (HMD), and the third is non-VR normal desktop screen. Through our system was not designed as an intervention platform, the preliminary feasibility study was designed as a proof-of-concept application.

System Design
The VR-based system is comprised of two main modules: 1) system architecture framework; 2) social communication task module.

System architecture
The system architecture comprises many modules integrated together towards VRbased interactive simulation three subsystems: display and trackers, auto-navigation, speech recognition, and gesture recognition Figure 1 shows an overview of the system architecture with all modules integrated.
Display and trackers: An immersive surround-screen display system comprising four rear-projected stereoscopic screens, three as front, left and right wall, and the fourth one as floor. Polarized glasses were worn on the user's head with two markers on the rim of the glasses to be tracked by PPT-E tracking cameras from Worldviz [10] to calculate the head position. The second display used is the Oculus Rift head mounted display. The third type is normal 20 inches LCD display. These are used in the automatic navigation module to guide the user though the virtual environment based on the user's view. There are also parts of the display module to present different level of immersion of the virtual environment.
Speech recognition: Since this project is centered on improving primary social communication skills in children with autism, the system uses this module to efficiently recognize the speech coming from the user as response to the virtual charac-ter(s) in the VE scenario. A wearable microphone is used to allow the system to receive user's voice. Microsoft speech SDK was used to detect the recommended word within the scenario. The only problem we faced is that the engine can only deal with English language. We need to have support for Arabic language, as most of the patients in the gulf region may not be able to communicate in English Gesture recognition: We included this module in the system to be able to detect hand gestures, such as waving hello etc. from the user. This contributes toward the non-verbal communication that user exhibits instead of verbal, to be able to track any kind of (verbal/non-verbal) response from the user. LEAP Motion device is used to obtain the position of the hand and its skeleton information for mapping the user's hand into the system.
Virtual environment and avatar: In this work, we developed the VR scenario in order to achieve verbal and/or non-verbal response from user with ASD using Vizard from WorldViz [10]. The Software comes with limited resources such as avatars, virtual objects and scenes. So we had to design customed avatars and virtual scenes to create specific social communication environment for our study. The virtual world and the character were designed and rendered in 3Ds Max [10]. Each of the avatars was built with a purpose that contributes to the scenario as can be seen in; and were generated using Autodesk's Character Generator while they were rigged and animated using Autodesk's 3Ds Max and Motion Builder software.
The environment selected to be a class room at school which is common place and should be familiar with most of the subjects. Figure 2 shows the school virtual environment and the avatars created for the simulation Auto-navigation module: This module uses the Tracking Sensors as input and starts the automatic navigation through the virtual environment based on the user's view.
All the above modules manipulate the Virtual Environment and Characters of the system which affects the view and the sound output as per the input change. User experiences changes based on these modules and the input provided on the output devices.

Social Communication Task Module
To be able to improve communication skills in high-functioning children with autism, it is essential to carefully make sure the environment we develop is efficient in triggering expected response from the user. We developed a scenario of 'greeting' within a classroom (school) setting which involves communication between a virtual character and the user.
Our communication task module is comprised of (1) the 'greeting' scenario, and (2) the 3D environment and characters to implement the scenario.
Design the scenario: We developed the 'greeting' scenario in consultation with experts and practitioners in the local center of education for the autistic who handle autistic children on a daily basis. The scenario was also inspired from the novel techniques of 'Social Stories' being used by a number of autism-related VR solutions that further justify our approach [11]. The scenario of 'greeting' is a general situation that an average individual faces multiple times every day; which can help us track the user's level of social interaction with multiple usages of the VR system. In the scenario, we start by virtually auto-navigating from outside a school environment going into the school and reaching the classroom through the corridor. The classroom environment is the main setting where the conversation development task takes place. A virtual teacher avatar within the classroom is standing with two other virtual student avatars in front of the chalkboard. The teacher initiates conversation and instructs all the students including the subject to perform the greeting task, see Figure 3 to Figure 8.
The virtual avatar of the teacher was designed to look similar to what we found the instructors/therapists wear within the school's premises during their sessions with autistic students. The virtual avatar of the students within the virtual environment as well as the virtual environment (school and classroom environment) itself try to follow a similar conduct of presentation to provide high amount of relatability to the participant. Keeping in mind the constraints related to the curriculum being followed at these traditionally-practicing institutions, and the inherent limitations of highfunctioning autistic children in general, the scenario has been developed to cope with the barrier of initiating or responding to a conversation with a stranger.
Role-play: Since our scenario should be shaped around getting a response from the child, all the tasks and conversations within the scenario build to rise the potentiality of the user responding to the task given to him/her by the virtual teacher (i.e. to greet her back) by showing the user (through the other virtual students playing their character in the scenario) how to reply to a greeting. User response is triggered/instigated (but not forced) by using the technique of role-playing the student avatars in VE.
Turn-taking:In order to ensure structured form within the task, each virtual student character is asked to perform the assigned task of 'greeting' turn-by-turn. After the teacher has instructed the students of the task, each student is greeted and asked to reply to the greeting. The user is assigned the same task in the end of the scenario. This teaches the user to respond when spoken to and wait for his/her turn when in groups.
The Scenario in the virtual environment: We designed the virtual world to display and integrate the above mentioned scenario in the following order:

Experimental setup
The investigation experiment was developed using the VR environment described in previous section. The participants' view orientation was tracked by Positional Tracker in Oculus Rift HMD and/or PPT Tracking Camera System for CAVE display experiments.
The scenario is presented on three interactive displays with different levels of immersion, namely: (1) Desktop computer, with no immersion factor, (2) Head-Mounted Display -Oculus Rift, and (3) CAVE immersive display.
Start time and the participant's response time is recorded and displayed at the end of successful completion of the assigned task. The response time is logged when the system detects user's speech recognized by the voice input device. The task progress can be observed during the sessions by therapists and for tracking non-verbal response. The system also has the ability to receive input from a monitor to record response time in cases of non-verbal response, such as waving etc.
Tasks and procedures: We designed a usability study as proof-of-concept to investigate the implications of the designed system with the three different interactive displays. The participants were required to commit to a total of two sessions for the study (Sessions 1 and 2) on separate days for a duration of approximately 20 minutes per session plus their feedback. We first conducted a learnability session where the team briefed the participants and their caregivers about the session and had the participants try the immersive display setup(s) (CAVE and HMD) to get used to the technology for genuine results. The participants were also told that they could choose to withdraw at any point during the session for reasons such as feeling uncomfortable with the system, or dizziness due to wearing HMD, etc. Participants were introduced to each system one by one, i.e. the session started with the scenario being presented to the participant on the desktop computer first, where the participant was equipped with only a microphone (for voice input) which was used for all three interaction sessions.
Second, the participant was presented the same scenario and assigned the same task on the Head-Mounted Display (Oculus Rift) and the response time was recorded for it. Finally, the scenario was replayed on the CAVE with the trackers and 3D glasses and the response time for each session was recorded similarly. The next day of the tests, Session 2 was performed on the participants where they went through the same procedures as Session 1 but without any guidance or pre-testing. Each participant was asked to provide feedback for the questionnaire after every session.
The questionnaire included multiple-choice questions and statements of the following themes: (1) Usability Satisfaction of the system on each interactive display, and (2) Impact of the level of immersion on the response of the participant. We extracted information about the satisfaction and impact of immersion level on the user (participant) based on their feedback.

Results and Comparative Study
According to our goals and objectives, the system was to be tested on highfunctioning autistic children of ages 9-12. Although, due to the unavailability of targeted participants, due to the lack of administrative and logistical support at local autism-related institutions, we had to perform our tests on typically developing children (not in the autism spectrum) with equivalent ages. We found that, typically developing children of ages between 3-6 are closest in mental and cognitive development to that of high-functioning autistic children aged 9-12 [12].
System usability and acceptability: In our usability study, we investigated the effectiveness of our VR-based interactive system by understanding the 'Usability Satisfaction' feedback from the participants on every interactive device per session. Using our participants' exit survey feedback at the end of the experiment, it was revealed that it was a general trend among our participants to favor the CAVE environment while being satisfied with all the different interactive displays and virtual environments. They had a certain (acceptable) level of understanding of the scenario presented to them in the virtual environment. They were also able to recognize the system's virtual classroom, school, classmates and teacher. They faced very minimal problems trying to feel at-ease within the virtual environment, according to our observation. Moreover, they kept focus on the VR-system, in an attentive manner.
We found the learning curve to stabilize around an average response time of 20-30 seconds for the first training session the participants had to get acquainted with our VR-based interactive system in the CAVE. As a conclusion, the system proved to be an easy-to-use platform that our participants would benefit interacting with. In the present study the questions in the exit survey were subjective. Thus, from our finding from exit survey, we can infer that our system has a potential to be accepted by the target population.

Quantitative analysis of participant's' performance in multi-session interactions
User satisfaction: Our findings indicated that users were, in general, satisfied with the virtual environment in our VR-based system as per their feedback, as well as their decreased response time. A decreased response time shows the users' understanding of the scenario and how many sessions they take to understand a virtual scenario, to be able to respond promptly and correctly to the system.
In our comparative study, the three aforementioned immersive interfaces displayed the same scenario to every participant per session. The user satisfaction for each level of immersion for each session is shown in Figure 9.
Impact of Immersiveness on Participants' Performance: The scenario was tested on each user, with the three different displays per session to determine the impact of the level of immersion of the VR system on the users' performance. From Figure  10 we find that according to the response time of the participants in every individual interactive display device, the general impact of the CAVE is much higher and effective in obtaining the desired response from the user. Thus, we find an improvement in performance of our participants when exposed to immersive VR interface. Since CAVE provides the highest level of immersion in VR, this proves the general effectiveness of an immersive interface in improving communication skills in children with autism using a realistic scenario.

Conclusion
In the present work, we developed a VR-based (verbal/non-verbal) interactive system for developing communication skills in children with autism based on a predefined and planned greeting scenario.
We have presented the system's development with results and comparisons from the usability study to test the efficacy of the developed system as an effective tool for the development of communication skills for autistic children.
The results of the comparative usability study discussed above shows the effectiveness of the system contributing to improvement of the autistic user's communication skills. Although, the study has shown positive impact in almost all the participants, there are certain limitations to considering these results in general for all highfunctioning children with ASD. This is due to the fact that the participants could be real autistic users to provide us with more accurate results in order to understand how our findings will relate to real autistic participants, instead of substitutes. This is also essential so as to verify how the impact of our VR-system can be generalized to all high-functioning population of individuals with the disorder.
In this research, the system is tested to help overcome communication impairment challenges that can result in development of targeted skills in targeted autistic children thereby helping them proceed toward a more independent life.