A Pictorial Mobile Application for Improving Communication Skills in Non-Verbal Autism

It is estimated that as many as 25 percent of individuals living with autism spectrum disorders are nonverbal. That is, they cannot functionally communicate with others using their voice. Despite that substantial fraction, we still know very little about these individuals, their abilities, and their needs. "We still know very little about the cognitive capabilities of nonverbal people with autism, and how best to help them learn to communicate," said Geri Dawson, Ph.D., Autism Speaks chief science officer. Non-verbal people with autism are usually unable to communicate normally using natural languages. They can, however, learn to communicate through specific symbols and images. Special education instructors have adopted this method of communication to teach non-verbal people with autism. They introduce the symbols and images to them through different methodologies. This learning process appeared to be effective but it is very long. The process is carried out manually and requires a lots of times, dedication, and resources. The instructors should find the materials in different formats and circumstances. They should repeat the lessons several times and normally in a faceto-face framework. We propose in this paper a mobilebased application that allows nonverbal people with autism to learn and communicate with their surroundings using a smart device. They can then be taught to use specific symbols and images through the smart mobile phones. They can form simple words and sentences to express their feelings and needs. The application is flexible and allows the addition of new contents very easily. To assess the progress of the users, different exercises and puzzles are proposed. These allow the users to improve their skills and to continue learning outside the classrooms.


INTRODUCTION
Around 10% of the world's populations have been identified with some kind of disability. This percentage is estimated to rise to 32% by the year 2026. People with impaired mobility, the visually impaired, those with speech difficulties, or who have a hearing impairment, can become totally isolated from the digital-based society as obtaining information from their surroundings becomes more and more difficult and problematic. These people have, however, at least one way to communicate with others (i.e., it could be head movements, eye blinks, voice intonations, and mimics) that could be enhanced with assistive technology and mobile-based applications. Experts recommend that people with disability should be given access to special computer applications and devices to avoid a digital divide between those who have access to IT and those who do not [5].
One of the strangest aspects of nonverbal autism is the fact that no one really knows why some people with autism can't -or don't -use spoken language. True, some people with autism also have childhood apraxia of speech, a neurological disorder that makes spoken language extremely difficult. But many nonverbal individuals on the autism spectrum don't have apraxia --they just don't speak. Clearly, there are differences in brain function that inhibit spoken language, but at this point there is no agreement on just what those differences are or how they impact any given individual.
There are many techniques for encouraging and improving spoken language for children with autism, though there is no guarantee that any particular approach will be effective for any given child. Research suggests that speech therapy, behavioral interventions, and even play therapy can improve verbal communication. Some early research also suggests that music therapy and related techniques can make a positive impact on speech.
Research about people with disabilities has been done primarily in western countries. Governments and private sector companies supported these research projects through specific funds, taxes and grants. Hundreds of researchers were allocated to conduct deep research to build assistive tools for people with disabilities and help them to integrate the society Some of the important projects for people with disability are: "AmlVital" [4] launched at the University of Granada (2007) which creates devices for elderly and other people with disability; Archimedes [10] which promises accessible technology for the disabled; Chelin et al. [5] proposed a system that uses natural language to assist visually handicapped people in writing compositions. Alper and Raharinirina [2] reviewed literature on assistive technology for people with disabilities and offered guidelines for future researchers in this area. First, each person should be given an individual assessment before the assistive technology is given to him/her and the person's needs should be taken into consideration. Second, family involvement in the learning process is critical if the person is to gain maximum benefit from the proposed technology. Jemni and Elghoul [6] presented an untested web based tool called WebSign that translates text to sign language. ALFakheer from King Saud University has announced the development of an educative webbased tool named FAHEEM for non-verb persons with Autism. However, applications are in the testing phase. In [1] a system has been proposed to allow children with disability to learn about new objects/entities by tapping on physical objects through a specially designed RFID-Bluetooth based Tangible User Interface (TUI) tool. In [7] an assistive computerized system has been proposed using the Arabic language for children with intellectual and learning disabilities that are resident at Shafallah Center in Doha, Qatar. The system is flexible and can be used by parents, children and teachers, where they can employ the materials according to specific needs. In [8] a picturebased communication system has been developed to teach Qatari children with autism and limited capability of expressions by means of digital images and pictures. This system was desktop-based and specific to the Shafalah centre. We mention also Mechling [9] has conducted a literature survey (1990)(1991)(1992)(1993)(1994)(1995)(1996)(1997)(1998)(1999)(2000)(2001)(2002)(2003)(2004)(2005) on research on assistive technologies as self-management tools for prompting students with intellectual disability to initiate and complete daily tasks. She mentioned that "although Kimball, Kinney, Taylor and Stromer (2003,2004)" outline instructions on how to create computer-based activity schedules with photographs and video models using Power Point, to date no research base exists to support this new and creative use of high tech systems in providing students with visual, auditory, and animated cues for following and transitioning between activities or use in other forms of self-management.
Teaching people with autism is a challenging task. Instructors spend many days to explain to non-verbal people with autism the meaning of images, symbols and photos. They select different shapes and models of the same image and show them daily and repeatedly to the autistic people. These images are often taken from the Broadmaker software and the instructors use specific tools to materialize these images (i.e., generation of glossy cards, power point slides, etc.). Figure 1 shows some images taken from the English-based Broadmaker software.
Few research projects are carried out in the Arab world for people with disability and the obtained results are moderate [3]. In addition, most of the Arabic software engineering companies in the MEAN region do not develop educational applications for special users since they do not see great financial benefits from this market.
In this paper, we propose a new mobile-based application that allows the non-verbal people with autism to learn and communicate using a smart device (i.e., iPhone, iPad). We can then improve their understanding and communication skills and break their isolation. Whenever the non-verbal persons understand very well the meaning of symbols and images, they will be asked to form simple sentences and select the correct images to initiate and complete a specific activity (i.e., go to school, select items for lunch, etc.). The proposed application supports inclusion, data exchange, communication and creativity by making communication more accessible and more costeffective.
The application provides ways to choose images and symbols, and their corresponding words to express moods, needs, and thoughts. It is oriented in particular to satisfy the needs of non-verbal persons in countries using Arabic language as medium of communication. It allows the users to from words using images and sym- bols. The application can help persons who struggle to speak properly due to several conditions such as nonverbal, autism, brain injuries, cerebral palsy, intellectual disability, and stroke to communicate. The application has an educational section which gives the non-verbal the ability to learn new words through reading simple Arabic sentences by getting the corresponding images using a multimedia database and online search engines (i.e., google, yahoo). The application allows the nonverbal people to become more independent and to be better integrated within the sociality, with the ability to communicate whenever they need and in different circumstances. Figure 2 below shows the percentage of the population in Egypt who are Arabic/English speaking, illiterates, disturbed or having a problem in speech communication, and those who are with intellectual and hearing impairments.

II. OBJECTIVES
The following questions have motivated us to propose this research paper: • How can we use technology to help non-verbal?
• What are the effects of images, sounds and animation on people with developmental or intellectual disabilities? • Can we increase their thinking abilities and skills? • Can they find suitable jobs? • Can they be productive?
• Can we decrease their isolation from the digital age?
• Which programs and tools should be offered to them? • How can we help them to express their feelings? • How we can assist them to communicate more effectively with their surroundings?
Digital inclusion is a concept much talked about these days, but what software and services do people with disabilities really need if they are to take part in our information rich society?
In Egypt, there exist many children who suffer learning disability and attention deficit hyperactivity disorder. Thus, it is time to recognize the problem and to propose solutions.
The long term research intention at the BUE to use technology to develop augmented communication systems for people with intellectual and learning disabilities is under consideration. Such systems should support collaboration, data exploration, communication and creativity. The broad objectives of such research could be summarized as follows: 1. To develop an Arabic tutorial-based system for people with disabilities other than non-verbal. We can then unveil the hidden potential within these people and improve their skills. 2. To enhance the thinking and memorization skills of these people. We could propose puzzles of different levels associated with the tutorials. People with disabilities can try to solve them and improve their thinking, writing and memorization skills. 3. To assist the disabled to improve their understanding of electronic text and dialogue. For example, we could try to develop new algorithms and techniques to extract keywords from electronic text (i.e., web sites). We propose to use concepts analysis and Natural Language Processing. We envisage studying other techniques, such as Latent Semantic [Ibrahim et. al, 2007]. 4. To explore the e-learning/M-learning experiences of disabled learners. There are many complex issues and interactions introduced by disabled learners' requirements for accessible e-learning, compatible assistive technologies and effective learning support. Assistive technologies should be considered. 5. To make BUE a center of excellence for applied research in the field of special education for people with special needs in Egypt and the Arab world. Thus, research can be sustained in Egypt. In addition, our work can benefit others who have the same conditions in other countries.
Research about people with disabilities has been done primarily in western countries. Governments and private sector companies supported these research projects through specific funds, taxes and grants. Hundreds of researchers were allocated to conduct deep research to build assistive tools for people with disabilities and help them to integrate the society. Very few research projects are carried out in the Arab world for the disabled people and the obtained results are moderate (Al Gain & Al Abdulwahab, 2006). In addition, most of the Arabic software engineering companies do not develop programs for users with special needs because they do not see financial benefits from this market.
It should be kept in mind that Technology can provide motivation, perhaps by allowing success where past experiences have only resulted in failure, or by increasing the user's self-esteem. On the other hand, technology is critical because of the rates of poverty among the disabled family. Many families of children with special needs cannot afford adaptive technology; even though it might help their children obtain a better education. In addition, many schools do not allocate the necessary funds to purchase equipment to help children with disabilities.
The objective of this paper is restricted to enhance the communication skills of Arabic speaking people with autism. We have gathered the related information about these people from existing documents and from several interviews with stakeholders (i.e., families and teachers).
The technical objective is to develop, maintain, and enhance an easy-to-use augmentative communication mobile-based application. The target people are those with autism. In addition, it can also be used by non-verbal Arabic speaking people.

III. SERVICES DELIVERED BY THE APPLICATION
The application allows non-verbal people better inclusion within the community using the state-of-the-art technology. This can certainly unveil the hidden potential within these individuals. It uses augmentative communication technique through a smart mobile device, which can help non-verbal people to communicate with their surroundings effectively.
The application can also contribute to the education and employment of non-verbal people in the increase of livelihood opportunities. It is developed in an open architecture where code and contents are totally separated. Thus, teachers can update the contents whenever they need. They may also create specific folders for every person according to her/his needs.

IV. PROCEDURES
In order to fulfill the aforementioned tasks, we built first the general artwork of the application, set the graphical user interface and design the structure of the database that will store the links to the information (i.e., symbols, sounds, and images). We tested this application with children with autism to explain to them simple Arabic stories through images, photos and short clips. We have gathered 20 simple stories for children and annotated manually their text. This annotation process link words in the story with their corresponding images. These images were collected from different sources including specific CDs and the Internet. The application is developed based on the techniques of software engineering as depicted in Figure  3a and Figure 3b.
We have also developed a component that can annotate simple Arabic sentences automatically. The objective is to get as input an Arabic sentence and convert it automatically to its corresponding images and photos. Thus, the nonverbal people can understand these sentences in an easy way. In addition, the component allow the potential users to form Arabic sentences by selecting images and symbols and the component will generate the corresponding sentence.  A new module of reading loudly the sentence is being developed. This module will allow direct communication between verbal and non-verbal people. They can exchange their ideas through texts and images by using the smart phones.
The developed tutorials have been made attractive by using multimedia technology (i.e., Flash, Dreamwaver, Studio). The advantages of a multimedia tutorial to develop the skills of non-verbal people with autism are as follows: 1. It is an audiovisual and interactive medium. 2. The treatment or situation can be reproduced, i.e., the same condition can be repeated several times. 3. It can be adapted to the individual needs. 4. It has an effect on more than one sense, and can be more effective. 5. It can help creativity. In addition, the user feels the success. 6. It can include "puzzles and games" to add to a children's motivation to use the software.
7. It can be used both in individual and in small-group lessons. 8. Parents can use it with success and participate actively in the learning process. 9. Most important thing is that the children should get interested and their interests are sustained for long periods of time. This is not an easy task, but multimedia presentations are very effective in this aspect.
The feedback from the users will be taken into consideration to improve the general interface of the system and the tutorials. The content of the tutorials and its corresponding puzzles will be tested at the end of each tutorial/puzzles. In addition, we will make a comparison between the non-verbal students who used the system versus those who don't. This comparison will help us to evaluate the effectiveness of the system.

A. Modules
The developed application is composed of four components as shown in Figure 4 below. The DataStore which is responsible about getting stored information from the database. The Parser which is taking charge of loading the annotated stories, process them, and get relevant images. The SearchEngine which pertains online search to get additional images. The MultimediaLoader which is responsible about loading multimedia items (i.e., symbols, sounds, and images) and display them.

B. Whole System Architecture
The new system is composed of two parts: an Arabic language processing server and the mobile application as shown in Figure 5. The server is responsible for providing educational services for the mobile application (i.e., processing Arabic story, loading online images, etc.). The mobile application is a web based application that does not require any connection with the server. The mobile application is created using the HTML 5 technology and connects the server only if the user intended to use the educational section.

C. Educational Model Architecture
We have created a database that stores simple Arabic stories. We have annotated diverse useful information (i.e., location, actors, tools, etc.) that are helpful for nonverbal people and can be highlighted automatically while reading. We have collected information about words in contrastive domains (i.e., foods, animals, tools). Figure 6 illustrates the structure of animals information. The system provides a coherent and flexible user interface that can be used easily with all users and allows them to navigate easily. Below are 4 different snapshots of the system: The user can first choose the first icon on the right for "possible" (Arabic: Momken), thereafter click the second icon "drink" (Arabic: Ashrab). Lastly, the user would choose one of the other icons to indicate the type of juice to drink.

B. Snapshot-2
After the non-verbal participant finishes the experiment, a general feedback will be discussed.

C. Snapshot-3
We get this layout by clicking on the icon on the upper row in snapshot-2, namely the boy playing the football. As see in this snapshot, this icon is the middle icon. To get snapshot-2, click the icon on the upper row, namely the boy eating. This icon is the icon on the first row down at the most left corner.

D. Snapshot-4
Here the user first chooses the first icon "I" (Arabic: Ana).
The user then chooses the second icon "want" (Arabic: Aayes).
Thereafter, the user chooses one of the three icons listed below to indicate the action to be taken, namely "eat" (located at the right), "drink" (located at the middle), "bath or WC" (located at the left).

E. Snaphost-5
Here the user can read stories and click on the highlighted word "Ant" (Arabic: Namla) inside the educational section.
Then he will get couple information about it with images.

VIII. CONCLUSION
This paper focuses on innovative practices directly related to non-verbal persons with Autism. The developed system will allow the non-verbal person to learn and communicate using a smart device. The suggested mobile application will provide ways to choose images and symbols and their corresponding words to express moods, needs, and thoughts. The developed system will provide !!"# ! !"#$ !"#$ ! ! !" !"" !"#