Development of an Auditory Rehabilitation Tool for children with Cochlear Implants through a Mobile-Based VR and AR serious game

It is known that the development of auditory skills in children with hearing loss, who use assistive listening devices, requires training and practice. The aims of this research were a) to describe an auditory training software developed in order to help children with cochlear implants and/or hearing aids improve their auditory skills and monitor their performance, and b) to demonstrate the usability of the auditory training tool. The software is mobile-based and uses VR (Virtual Reality) and Immersive technology. In order to use it, the user must wear a VR headset. This technology was adopted because these devices are considered to be innovative, and are especially popular among children. The software was tested on fourteen hearing-impaired children. Eleven of these children use a cochlear implant and three use hearing aids. The results of this research show that the children with hearing loss were able to play the game successfully. This positive outcome supports the use of VR and Immersive technology as Auditory Training Tools. Keywords—Mobile-based, auditory rehabilitation, children with cochlear implants, virtual reality, augmented reality.


Introduction
Both serious games and virtual reality are used in health care. In VR technology, the users wear special devices such as Head-Mounted Displays (HMD) or Head-Coupled Displays (HCD) or use devices like Google Cardboard. In virtual environments, the artificial world mimics the real world and the user feels that he/she is part of it [4]. This feeling is created for the user with the use of Immersive technology can enhance and speed up the recovery process [6].
VR technology can offer solutions in many different fields. These include the education and training of people in general, but also specifically of people with special needs, as well as the rehabilitation of patients, simulations, therapeutic treatment, entertainment, video games, interactive storytelling, military, and medicine [8].
In many cases, children with hearing loss have used serious games as a means of training and rehabilitation [7]. The purpose of the software we developed was to use Immersive technology and the gyro sensor as a means of observing the virtual world but also as a means of interacting with various 3D objects. In order to use the application, the user must wear a VR headset [3].

1.1
The aim of the software The aim of the software is to provide children with an interventional tool suitable for everyday practice to help them improve their auditory skills. Additionally, data about the user's performance are recorded so that the speech therapist can evaluate the user's performance "see Figure 6".

2
The Game

Description of stages and key features
The mobile-based rehabilitation serious game was designed to help cochlear -implanted children improve their auditory skills [2]. Two familiarization stages were added prior to the rehabilitation training part: a) learn sounds b) learn sounds and words. The serious game was based on Erber's model of development auditory skills [1], i.e. speech detection, speech discrimination, speech recognition, and speech comprehension. The completed stages for rehabilitation training of the user were: a) sound detection, sound discrimination, word discrimination, word recognition, and sentence comprehension. The stages used by the speech therapist have also been completed. These stages are Setup and Data View. All objects contained in the application are three-dimensional.

Stages Learn Sounds and Learn Words
In the stages Learn Sounds and Learn Words "see Figure 1", the user gets to know the environment and becomes acquainted with the use of the device and the gyro sensor that operates as "eye tracking".

Stage Detection
The stage Detection "see Figure 2" is the only scene where the user uses their fingers to interact with the application. In this stage, a sound corresponding to an animal that appears on the screen is produced at a random time and the user must touch the screen within a reasonable time following the onset of the animal sound. The animals are taken at random from an array of selected farm animals.

Discrimination stage
In the Discrimination stage "see Figure 3", "eye tracking" is used to interact with the application in order to select an animal. In the beginning, the user listens to the sounds of two different animals that are sequentially introduced and then must match a third sound they hear with the corresponding animal. If the user does not select the correct animal, the application shows the correct choice using a hand-shape and an avatar respectively. The same procedure is repeated for word discrimination task.

Comprehension stage
In the Comprehension stage "see Figure 4", unlike in the other stages, the user listens to verbal phrases. In this stage, the user watches different animals walking in pairs. When they hear an imperative phrase, consisting of a carrier phrase and a target word, "X, eat the apples!", where X is the animal name, they have to choose the correct animal. If the user does not select the correct animal, the application shows the right choice by playing the phrase again and using a hand-shape and an avatar respectively to show the user the correct animal.

Setup and Data View stages
In addition, as mentioned above, there are two stages that are specially designed to enable the speech therapist to monitor and adjust certain parameters of the game and/or receive feedback to regarding the patient's performance with different types of auditory stimuli. The two stages are Setup and Data View. In Setup "see Figure 5", the speech therapist has the ability to adjust the sound and noise level (using the sound of a waterfall), adjust the output valid delay time and also specify whether animal sounds or animal names will be reproduced. In the Data View stage "see Figure 6", the speech therapist can see the data produced from the application use. The registered data are the date and time of use of the various stages, the volume of the animal sounds and their respective names. The user's response is also registered, as well as the reaction time that is the time elapsing between the presentation of an auditory stimulus and the user's response. When the user plays with the software, the data entries appear on the screen of the speech therapist's mobile phone itself or are sent via e-mail or Bluetooth.

Familiarization
Before the test began, the VR headset lenses were properly adjusted to each child's needs. We moved the lenses together or apart to fit them to the distance between each child's eyes and also moved the lenses towards or away from the eyes to adjust the image sharpness. The volume of the device was adjusted to 100% as we wanted to make sure that all children would be able to hear the sounds and verbal phrases. Each child was given the opportunity to become familiar with the application and with the "eye tracking". Familiarization was conducted by playing with the stage Learn Sounds "see Figure 1".

Testing and usability observations
Then the children played the next stages. Before each stage, the researcher, with the help of the speech therapist, told the children what they would see and how to react. During the research, it was observed that all children were able to play the game successfully. In younger children, there was a need for additional instructions to be able to play with the application. During the use of the application, some children encountered difficulties in the detection stage. At this stage, they needed to use their fingers to interact with the application; when they heard the sound of the animal, they had to touch the screen immediately. The problem was that, while they were able to hear the sound, they did not touch the screen within the time frame that was set up for the task. The reason for the delay was that they did not have their fingers inside the headset. After additional instructions were given, the children were able to successfully play the Detection stage. In the following stages, "eye tracking" was used as the means of interaction and input for the selection of the animal. This method of animal selection worked well.

Evaluation
A usability study was conducted in order to test the experience, ease of use and satisfaction of the target group for which the auditory (re)habilitation VR 3D game was developed for, i.e. children with hearing loss that utilize hearing technology.

Method
Participants were 14 children, aged from 2.7 to 17.6 years, with hearing loss, who are fitted with auditory devices and follow speech/language therapy for improving their auditory and oral language skills. The sample was drawn from two sites that serve as habilitation units for the pediatric population with hearing loss, a CI Center in Macedonia region of Greece and the Parent's Association of children with hearing loss in Central Macedonia. The user profiles are shown in Table 1.
A questionnaire was constructed to enable users to rate several aspects of usability of the auditory rehabilitation virtual reality 3D game (AVREG). It consisted of 9 ques-  The question format was designed to fit the communication skills of children with hearing loss, where topic familiarity, redundancy, and repetition of verbal information are important elements for achieving good comprehension by young children with hearing loss [6]. Hence, each question was composed of • A focus question (yes/no), aimed to familiarize participants with the topic and the specifics of the query • A target question aimed to obtain responses from the participants via a ranking process (highly positive, somewhat positive and negative). The ranking process consisted of a three-interval forced choice task where children head to choose among three emoticon types, i.e. a happy face, a somewhat neutral face, and an unhappy face.
Prior to testing, the experimenter obtained parental written consents and parents filled out a background history information sheet. Then, the experimenter interacted with the children for a few minutes in order to establish rapport, described to them the purpose of the study and obtained the child's verbal consent before handing them the AVREG. Subsequently, he gave instructions for placing the mask and playing the game. Children were allowed to handle the apparatus and explore the game on their own. Instructions and prompts on how to play the game at each stage were given when necessary. After completion of the game, children were asked to fill in the questionnaire by listening to each question and responding by touching one emoticon in the threeinterval forced-choice task.

Results
A percentage of group responses (N=14) corresponding to each level of the threelevel Likert-type scale was calculated for every question, for a total of 9 questions. Subsequently, a weighted average was computed in order to derive a single rating score from responses to each question.
As shown in Table 2, the children users' ability to control the VR was rated positively, ranging from 1.29-1.36, thereby indicating that children with hearing impairment who are fitted with auditory-assistive devices, i.e. hearing aids or cochlear implants, corresponded closer to the rating 'very much' as compared to 'little'. Regarding game enjoyment, the weighted scores ranged from 1.36-1.71, suggesting that users in this study enjoyed playing the VR game above the level of 'little', and for 2 out of 3 questions in this category, the rating was closer to 'very much'. A qualitative analysis revealed that younger children gave more positive responses than older children as the topic was more suitable for the former. The less favorable response was obtained on the question 'how long do you like to play with this game' because the set of stimuli and number of scenes were relatively small. Finally, children with hearing loss rated their experience with the VR technology as highly positive, as weighted ratings ranged from 1.21 to 1.29 for the set of three questions allocated in this questionnaire category.

Qualitative finding after children tested the AVREG
The children commented that they preferred this method of chosing over using their fingers. Yet, one problem that children encountered in selecting animals with "eye tracking" was that they could not easily choose small-sized animals. The younger children liked the content of the game and the virtual environment. The older children liked the virtual environment but not the content as compared to the video games that they play at home.

Conclusion
In this article, we have presented the structure of a new auditory rehabilitation tool along with results of the usability of software based on a study involving 14 children with hearing loss. The software is mobile-based and uses VR and Immersive technology. The purpose of the software is to help children with cochlear implants to improve their auditory skills. The results of the research show that all the children were able to play the game successfully. All participants liked the virtual environment and preferred selecting animals with the use of "eye tracking". However, the content of the game was better liked by the younger children. Although almost none of the children had ever used a headset in the past, all children participants who wore auditory assistive devices were enthusiastic about using it. Based on the results of the research it could be concluded that the use of VR, immersive technology and VR headset devices are suitable product designs for creating an Auditory Training Tool. Future work should be geared into designing materials and game scenarios that suit the interests of older children as well.