Assisting Individuals with Autism and Cognitive Disorders: An Augmented Reality based Framework

— Individuals with autism require systematic assistance while dealing with the surrounding environment and its objects. The project aim is to develop a framework that could be of substantial help to people with autism and cognitive disorders. The framework is based on common mobile devices and freely available Augmented Reality (AR) applications. The Augmented Reality used in our approach is marker-based AR that employs a camera and a visual marker to trigger and present media content on the mobile device’s screen. The developed framework allows parents and teachers to easily create educational augmented environments for children with autism and cognitive disorders by populating a real-world space with visual markers of favorite cartoonish helpers that can evoke helpful AR content and embed it in the real-world environment. The paper analyzes and discusses the use of the proposed framework from conceptual and technological points of view.


Introduction
Persons with autism spectrum disorder require systematic assistance while dealing with the surrounding environment, its objects and particular boundaries.
Impairments in communication remains as one of the most prominent limitations of children with autism [11]. Children in the autism spectrum commonly demonstrate difficulties in initiating as well as responding to social interactions. These difficulties include processing the situation and dialogue surrounding them, along with the inability of understanding non-verbal aspects of communication, lack of human engagement, and the inability to generalize between environments [12], [13] Traditional ways to cope with these challenges that autistic children face include one-on-one repetitive interactions, sustained attention activities, and engagement in reinforcement strategies.
A systematic review of the literature on advantages and challenges associated with Augmented Reality for education identified an increase in the number of AR studies during the last four years and has revealed that the most reported advantage of Augmented Reality is that it promotes enhanced learning achievement [14]. In particular, Augmented Reality technology allows promising results to be achieved in non-traditional learning environments specially constructed for people with autism and cognitive disabilities. See, for example, [15], [16], [17] as well as the latest publications in this area, such as [18], [19], [20].
Augmented reality was used to make markers represents toys that can be reflected through a screen that has an effect as a mirror [21]. In the research done by Lizbeth and Monica [22] a mobile augmented reality was used to provide additional visual support. The application developed in [23] used applied behavior analysis technique to train children with autism in AR system. The result shows that children could recognize new images and objects as well as ability to use appropriate words.
Social interaction difficulties in ASD child are due to abnormal visual processing that is taking place when recognizing human face; however, they have similar strategy with normal children when recognizing cartoon faces. Using cartoon characters has shown promising research results [24] as well as using video chat systems [25]. In [26] a cartoon character was used in AR system to perform chat with their teacher where teach face was masked with cartoon character. These approaches were beneficial in promoting the communication skills and study abilities to children with ASD. Based on these result, the proposed framework is proposing the visual appearance of the assistant to be a cartoon character that the ASD child can select from his favorite characters in cartoon film, comic book, or computer game.
The principal aim of this project is to develop a general framework that could be of substantial help to people with autism and cognitive disorders. The framework is based on the use of common mobile devices (such as smartphones or tablet computers) and freely available Augmented Reality applications (such as Layar, Aurasma or others). The Augmented Reality used in our approach is marker-based AR that employs a camera and a visual marker to trigger and present media content (3D animation, video, audio, image, text or their multimedia combination) on the mobile device's screen as if the content is embedded in the real-world environment. For the description and analysis of different types of Augmented Reality, see [27]. Basically, there are the following two types of Augmented Reality: marker-based and location-based [28].
AR supplements reality, rather than completely replacing it. Ideally, it would appear to the user that the virtual and real objects coexisted in the same space. Fig. 1 shows marker-based AR where the marker is replaced by the graphics objects. The graphics and real objects are coexisting together. AR has been utilized in the proposed system in such a way that it recognizes the markers and play related synthesized video. These markers can be used for providing different type of information and graphics on mobile devices as it becomes easy to create AR application for mobile phones recently [29] and [30].

Fig. 1. AR sample shows how graphics is integrated with real world
With the development of technology, these markers can be any image replacing the old black and white unique barcode-like markers. These enable us to use the image of any character to trigger the AR contents, see Fig. 2. We have chosen the marker-based AR, because in this phase of the project, our research objectives are limited to cognitive environments within a building. If, in the future, we decide to explore larger scale environments (such as a park or a city), then location-based AR will be the technology of choice. In this case, the images of a virtual helper should probably be presented on big AR billboards placed in a variety of locations across the larger environment.

2
A General AR Framework for assisting cognitively challenged persons

Architecture overview
The input is the real world image of surrounding objects using the mobile's camera to capture the image. Then, marker identification stage will detect the marker and extract the image of the marker. The next phase is to compare the recognized character image with the media database to select the corresponding media to be rendered. Before, playback the media, one last process step is carried out to make the media transparent and delete the background to keep only the head of the character 'talking head'. The last step is to render the media to the mobile by replacing the image with the media.

2.2
The selection of the approach and its components The search for the most appropriate and effective types of visual AR markers comprised the initial and highly critical part of the project. It resulted in the following findings.
Firstly, to get the visual appearance of the assistant, the disabled persons should choose a favorite character by themselves; it can be one from a cartoon film, comic book or computer game.
In the prototype, shown and discussed in this paper, the chosen favorite character is Mario -a short and pudgy Italian plumber that has appeared as a fictional character in the Super Mario platform game series (in over 200 video games since his creation), owned by Nintendo [31].
Mario is just an example character used in our prototype. The number of cartoon heroes is huge (see, for instance, various top 25, top 50 and top 100 lists of most popular cartoon characters at [32], [31] and [34]). This quantity and variety can be considered as one of the merits of the proposed framework, because it allows any disabled person not only to choose an assistant, but also to make available hundreds of the character's postures, which can serve as attractive AR markers.
This kind of AR markers has several other advantages, which are relevant to their use for individuals with autism and cognitive disorders: the beloved character is being perceived by the cognitively challenged individual in a positive way as a genuine assistant that is ready to help with advice in a difficult situation; the stickers/markers placed on real-world objects are most recognizable and identifiable by the disabled person; a large variety of the character's body postures (available as animation frames -see, for example, Fig. 4) facilitates the production of any required number of unique AR markers and therefore to cover/trigger any amount of AR multimedia content. After the successful selection, the chosen character is to be printed on sticky labels that can then be attached to any object in the surrounding real-world environment in order to serve as visual markers, as shown in Fig. 5. The choice of an effective type of AR multimedia content was another challenging task of the research. It was found that the use of particular media types (video, audio, image or text) in general depends on specific tasks, situations, and the problems to be solved. However, from the use of Augmented Reality as a cognitive tool, the most powerful effects seem to be produced by videos, 'embedded' in the real world. The idea of using video is well supported by some previous research publications. See, for example: [36], [37], [38] and [39].
This type of 'talking head' videos with a transparent background can be produced relatively easily. Their advantage lies in the illusion that they exist in the surrounding real world, rather than on the screen of the mobile device. See Fig. 6. Another effective media type could be provided by audio augmentation of the realworld objects and environments by 'attaching' sound-only files to the markers/objects.
Working prototype and implementation of the framework In the current first phase of the project, the problem was thoroughly analyzed and possible solutions and requirements were proposed. This has led to the production of a working prototype of the anticipated framework and its testing it 'in vitro' for its technical and some special experimental functionalities.
The prototype used the Mario game character as an assistant presented in the form of printed AR markers on a variety of real-world objects within a particular learning environment. See Fig. 4, 5 and 6 for more details.
Our research on the market of freely available and easy to use AR software platforms and tools concluded that at the moment the best one is the Aurasma platform by Hewlett-Packard Development Company [40]. It is available as a free app for iOS-and Android-based mobile devices. Also, it includes the Aurasma Studio that is easy to use and that can successfully be handled by teachers with any level of computer skills. Technologically, the Aurasma Studio also allows us to implement all the AR-related features of the proposed framework. However, we keep searching the AR software market for new arrivals with novel approaches and functionalities.

Experiments
The main objectives of these experiments is to test the limits of distance and angle of the Markers to decide the active workspace of the system.
Marker distance test: This test is conducted to decide the maximum distance between the camera and the marker with ability to recognize the marker. Eight distances were tested and in each test the detection is observed. For each distance test, the experiment repeated five times to be able to calculate percentage of detection accuracy for each distance. The results are shown in Table 1. The result shows that the maximum distance for correct marker detection is 70 cm. This could be related to the type of the camera and the focus, so these results could differ for another type of camera, nevertheless, the figures gives a good idea about the range of possible distances. Marker angel test: Angle test is necessary to be tested as the user may not always pointing the camera directly to the marker rather with different angles. To determine the range of legible angles, eight angles were tested and each test the accuracy of detection observed. For each angle test, the experiment repeated five times to be able to calculate percentage of accuracy for each angle. The results are shown in Table 2. The result shows that the maximum angle for correct marker detection is 70 degrees. As expected, the prototype has shown good and stable technical characteristics conforming to the main working hypothesis of the project. It can be populated with any other assistant/marker characters and can be used within any small-scale environments. These are relevant technical results for the general AR framework researched and proposed in the project.
Usability test: To explore the acceptance and efficiency of the framework for autism, three autistic patients were invited with their instructor to perform testes. A questionnaire is used to collect the response of the instructor regarding the system. The answers were on 5 points Likert scale (5 strongly agree to 1 disagree). The questionnaire tested many factors, in this paper we present few related results. One important factor is the acceptance of the system in subjects, another factor is to what extent are the subject attracted and their focus time. The result is presented in Fig 7. The approach could assist instructors in improving communication (86%) and increase the focus in autistic child (79%). The system provides easy interaction (82%) and can attract the child's attention (83%).

Conclusion and Future Work
Our working hypothesis is that the developed framework allows parents and teachers to easily create educational augmented environments for children with autism and cognitive disorders by populating a real-world space with visual markers of favorite cartoonish helpers that are able to evoke assisting AR content embedded in the real-world environment.
The next phase of the project will include a short field test of the prototype in order to enable its fine-turning and debugging on several AR platforms and applications. After this, the final phase of full-scale implementation of the proposed general AR framework is planned. It will involve a group of individuals with autism and/or other cognitive disorders. The results of the experimental study of the framework will be carefully analyzed and published.