Augmented Reality based Educational Design for Children

— Human-computer interaction (HCI) has developed rapidly in re-cent years, and more and more researchers are interested in applying HCI tech-niques into education. Compared with traditional approaches in the real world, gesture recognition is considered as a reasonable alternative since it is vivid and flexible. However, most of educational equipment nowadays achieves the function of augmented reality, without any interaction. This paper implemented a prototype, not only based on augmented reality system, but also especially we think about the interactive design. Accessibility is achieved by mobile devices and the dynamic switch of gesture recognition. By this interactive method, children are able to interact with the virtual objects easily and naturally. Conse-quently, children can have a profound and deep understanding of what they learn, and the quality of education will be improved.


Introduction
Rapid advancements of computer technology have created a great deal of research attention in HCI which has provided great convenience for people's daily life in both studying and working fields. As interactive information increases fast, more and more people are not merely satisfied with the traditional interactive device like mouse and keyboard [1]. In order to make the communicating and interaction more flexible and convenient, researchers are devoted to design various better approaches. Subsequently, voice recognition and gesture recognition have been established as an alternative to physical touch. After decades of researches, voice recognition has made great progress in some areas where the accuracy of automatic speech recognition is quite high, therefore, a speech recognition system has already been used in several commercial applications [2,3]. Differ from voice recognition, the recognition and operation of gesture are more intricacy, but there is enormous implication to complete and improve gesture recognition as this kind of gesture interactive crafts without traditional handheld devices are still needed in lots of fields, from astronauts' space flight training to simple display of merchandise and medical models [4].
Based on prompt improvement of virtual reality technology and three-dimensional modeling, gesture interaction has undergone a tremendous change nowadays. Data gloves like "Sayre Glove" and "Power Glove" have been utilized in computer games [5]. However, this architecture comes with some problems like only the average curvature of all fingers are able to calculate and no solution exists for complex motions. Then Kinect has become an attractive estimating method of gesture recognition in vision, which uses general camera to acquire human hand images [6]. Although by this way, people can obtain a stable image sequence and even the depth of information, the extraction of manual information must go through a complex image and identification process, which is difficult to ensure the accuracy and efficiency of rapid estimation on manpower posture at the same time. In order to enhance the efficiency of the capture of gestures, a company in America produce a special motion controller called Leap Motion, which can specifically recognize fingers and interact with gestures [7]. Its advantage lies in the ability to obtain both of the static and dynamic information.
Now Leap motion has been applied in many fields, such as the display of 3D commodity, motion sensing entertainment like Kinect, teaching assistant of driving and so forth [8]. Simultaneously, based on the virtual reality (VR) technology, augmented reality (AR) system which are also called hybrid system provides a novel development for Leap Motion. By using the information provided by the computer system, AR system can enhance the sense of experience about the scene. The implementation is dependent on appropriate objects and scenes generated by the computer virtual technology, which forms indiscriminate integration of real scene. Ultimately, the perception of reality is enhanced. As there has been no attempt to incorporate both technology, it appears a novel field where combines Leap Motion with augmented reality system. Considering about the situation in the present that children can only attain knowledge from two-dimensional form like books and electric documents, we aim at facilitating the perception of information and improving the capacity of understanding by providing help about virtual situation for children. In this study, a novel interactive scheme about gesture recognition based on augmented reality tailored specifically to education of young children is established.
In the following sections, we provide preliminary information in Section 2. Section 3 provide a literature review of some related work like previous researches about children's education and HCI. In Section 4, we propose our design of application for helping young children to learn based on leap motion and ARToolKit in detail. Section 5 discuss an analysis of the accuracy and reflecting time. Conclusions are given in Section 6.

Preliminaries
Leap Motion is a potentially revolutionize device which is designed to track the movements of hands and fingers. People are allowed to utilize its property of advanced HCI to create several predefined gestures. Unlike the depth camera like Kinect, Leap Motion does not provide a cloud of points about complete depth map. In-stead, it just captures a few key points of data and it is targeted to the orientation of hands while the gesture recognition is mainly about positions of fingers. Therefore, its accuracy of extreme finger detection can easily up to 0.01 mm. As it is intended to be an interface for visual gesture interaction, there is no doubt that it is optimized for recognizing human hands and pointy objects.
ARToolKit is a secondary developing software tool which can enhance the reality system based on computer vision and it includes four major sections: the graphics processing section, the VRML rendering section, the AR calculation section, and the image capture section [10]. The image capture module is used for image rendering and hypothetical spatial view settings, at the same time, the VRML rendering module is the VRC form of 3DCG representation which relies on OpenVRML [11]. The role of AR calculation module is identifying the marked processing picture and marking the recognition algorithm. Consequently, user can interact with the tri-dimensional user viewpoint which is converted from virtual image in real-time and accurately consistent with the practical objects in the real world.

Related Works
Rapid development of HCI and the deep concern of the quality of education have attracted researchers within various fields to design valuable applications of interaction which can help children obtain thorough comprehension of worlds. This section provides a review of several relevant existing strategies established by previous researchers. These strategies encompass some potential convenience for making Internet products more accessible to the education of children but remain several limitations.

Two-dimensional design for children
Most of educational equipment nowadays are merely in two-dimensional plane space where children can only experience and cognize new things from the visual aspect. These kinds of products usually consist of screens, texts, buttons and some simple structures. Majority of papery materials like books or electric devices like computers are typical form of two-dimensional appliances [12]. However, all these existing products lacks reality and cogent sensing, so that it is difficult to come up with a proper approach, catering to demands of children who long for recognize the world in more vivid way.
A good case in this point is that children with less experience are not able to real feel the outside objects, ranging from diversified animals to magnificent landscape, through reading books. Instead, children just obtain limited knowledge and realize the world through characters and pictures on the paper which no more than providing a rough and blurry description. As a result, children will only know that bee is a kind of flying insect, without understanding the softness of its fuzz or the frequency of flapping wings while flying. Moreover, as children are inaccessible to actually touch and perceive, it is obscure for them to maintain long-term cognition and their memories are fragile.

3.2
Augmented reality design for children Augmented reality system is involved into the manufacture of electric productions so as to improve children 's sensation of new things and unknown objects. Combined with the development of three-dimensional technology which makes the previous two-dimensional space more active and vivid, a variety of augmented reality tools come into our sight, such as Unity 3D, ARToolKit, 3D Studio Max and so forth [13]. Virtual objects which do not exist in the real environment can be generated and accurately integrated with the reality as-assisted by computer graphics technology and visualization technology [14]. Therefore, children can actually have a lively experience of the real world through the sensor and augmented reality technology.
Also taking the example of reading books, there is no deny that with the threedimensional technique which can enhance the reality, children can be personally on the scene and have a better understanding of the content in the book. Unlike former two-dimensional space, augmented reality system makes it possible for children to know the exact look of outside objects. For example, when they learn about bees, a dynamic figure will be shown on the picture, so that it is easy for children to have a profound impression of how bees fly.
However, there still exist some deficiency. Although children are able to see the dummy pictures which is similar to what in the real world, they have no solution to interact with these virtual objects. Thus, there is a need to add some interactive section on the teaching products. This paper designs interactive behaviors and operations in terms of assisting children fondle the bees and control their motion by hand gestures, providing a vivid animation for children which leads to an unforgettable longterm memory.

4
Our Design

Design principles
To better understand how young children interact with educational products, we interview 30 children and their parents about the features and functions of the products they are willing to use and our exploratory interviews shed light on the nature of the children. Considering all their opinions and suggestions, the proposed interactive scheme focuses on three key characteristics: • More vivid approach for children to study and acquire knowledge. As our scheme is chiefly designed for young children to help them have a profound understanding of the outside world, we take the need of children as our central task. Considering about their weak cognition and perception, it is necessary to make the design more easy and convenient to use.
• Real-time interactions between young children and our educational strategy. Most artifacts nowadays are just about virtual 3D image while they do not take HCI into consideration. However, interactive behavior will make a positive effect on children's learning and make young people more passionate about attractive interacting technological products. Therefore, apart from conventional visual methods which hardly incorporate human-computer into augmented reality devices, our scheme makes a rational combination of both two technologies. Furthermore, the efficiency of the design is achieved by real-time interacting mechanism.
• Better precision of the hand gesture recognition. There is no doubt that the accuracy and speed of recognition by computers play a significant role on the sense of utilization, so that we make our effort to improve these two factors at the aim of making children have a better experience during this kind of interactive education.

Prototype design and implementation
Based on the HCI and augmented reality system, the goal of our interactive design is to display the objects in three-dimensional form on the real environment and ensure people can control and move these virtual images, so that young children can enhance their capacity of recognition and have a better memory of what they learn.
Our concrete implements are divided into three phases: gesture control, dynamic switch and virtual display. Firstly, the gesture recognition system recognizes the children's finger gesture with Leap Motion and the gesture is sent to Android device through WiFi. Then, the program running on Android device changes the picture according to the action of hands. At last, ARToolKit program recognizes particular picture and upload different three-dimensional animation depending on different circumstances. In following paragraphs, we illustrate each phase in detail.
Phase of gesture control: In order to achieve the core function of interaction, the Leap Motion is utilized to recognize the gesture of children, which accuracy can be trained and learned through a diversity of their hands' motions. The available interaction with this kind of educational equipment is established by programing a computer procedure to recognize each gesture and complete corresponding reaction. Ten gestures are defined as an example and are divided into two sections: static gestures and dynamic gestures. Four static gestures are consisting of pointing up, down, right, left, while six dynamic gestures respectively represent turning over, pushing, shrinking, blowing up, clicking and circling. The key points models of some typical gestures are shown in Figure 1 and we explain it in the following paragraph.
For the sake of better estimating the hand pose in a shorter time, the positions of key points rather than all are calculated, such as palm, wrist, fingertips, far knuckles, proximal joint, finger joint and other fifteen points. Also, to obtain the threedimensional coordinate and constitute the vector matrix which is suitable for computation, several notations relevant to figure gestures of each hand and finger are required: palm position -coordinates of the center of the palm; palm normal -the vertical vector pointing to the inside of the palm; hand direction -the direction of the finger from the center of the finger vector; hand length -the length of the finger; tip position -the position of the fingertip. As the recognition of static gesture is irrelevant to previously shown gestures, each position and orientation can be treated as a new data point. The algorithm of Support Vector Machines (SVMs) has contributed to the accomplishment of rapid classification and recognition. Diversely, the dynamic gesture depends a lot on time, which is difficult to acquire consecutive positions of fingers. Fortunately, Hidden Markov Model (HMM) comes into our sight as an effective solution with which implicit gesture factors could be affirmed and the efficiency of recognition could be improved. With these effective recognizing method, children can easily use their gestures to control the interactive devices.
Phase of dynamic switch: Figure gestures are successful transmitted, and a further control of the terminal devices is completed by sending a number of the gestures to the terminal program like Android program which can make a real-time response. With the assist of WiFi communication thread which is responsible for receiving data, the gesture from children can be sent to the main thread through Handler messaging mechanism and be dealt with in the terminal devices. In the light of motions with respect to different gestures, the programs running on the educational equipment can react quickly and switch to corresponding images.
Phase of virtual display: When the gesture has already been recognized by leap sensor and be received by the terminal device, the augmented reality system like AR-ToolKit converts the two-dimensional picture to three-dimensional dynamic animation, which complete the accordance with real-world objects by covering the reliable interface with real-time virtual images. With the detection of camera and the tag tracking technology, virtual objects are overlapped with relative positions and orientations of tag markers. Consequently, it is realizable to integrate the virtual image with the real world and make them in an accurate aligning. As a result, children are capable to interact with the three-dimensional dynamic display by controlling their gestures. Figure 2 describes the process of how augmented reality system work in our design, showing the virtual-real combination and the real-time interaction.

Experiment and Analysis
This section describes how young children think about our scheme with regard to five primary terms. We also contrast its performance efficiency in comparison against previous non-interactive products. In addition, we analyze several important features of our strategy by designing a credible experiment and doing a great deal of tests.

Evaluation analysis
For the sake of making our design cater to the need of children at the maximum extent, a research, letting children evaluate our strategy and give a grade for both our scheme and other previous schemes like Alaamri et.al's less-interactive schemes or Kerdvibulvech et.al's augmented reality applications, is conducted [15,16]. Considering about the current situation of educational devices, we list five evaluate standards. Firstly, what children concern is about whether the gesture recognition is accurate and real-time. Secondly, the virtual display of the scene plays an important role, that means, it is significant to know whether our interactive augmented reality module is more attractive and easy to use. The third standard is that compared with the traditional approach, whether our system is more consistent with the way of people's communication. Moreover, the stability of the system also needs to be taken into the consideration. Finally, we let the whole group of children make a comprehensive evaluation of our scheme. In order to better illustrate the results, we apply seven-point systems, and each child grades our scheme and previous strategies according to evaluation in many aspects. Results can be summarized that compared with traditional singular products, majority of children are attracted by our interactive design, which affiliate augmented reality design with the interactive applications.

Experimental analysis
The feasibility of our interactive design and operational results is verified by an experiment which tests the accuracy of gesture recognition and calculates the average corresponding time. Ten pre-defined gestures are involved in the examination in sequence: pointing up, down, right, left, turning over, pushing, shrinking, blowing up, clicking and circling. Their concrete description and features are shown in Figure 3.
Then in the following experiments, we utilize these finger gestures to test our scheme.

Fig. 3. Features of ten predefined gestures
As our experiment aims at providing more persuasive evidences on better accuracy and corresponding time of our interactive design, each category of gesture is controlled to appear 100 times in the tests, which is considered as the total tested number. In the other word, it means that we asked the subject to do 1000 gestures and each gesture were performed randomly and 100 times. The computation formula of average corresponding time is listed below: Average corresponding time = ∑ (correct recognition time) / (Total Tested number * Correct recognition rate).
The final experimental statistical results are shown in Table 1, showing that the correct recognition rate of our design is sufficient for children to have several interactions with our design. And although the corresponding time of dynamic gestures is a little slow, our scheme can respond to the static gestures quite quickly. Therefore, our interactive design is effective and attractive for children to improve their cognitive capacities. In the future, we will improve our recognition algorithm and corresponding time and will involve more interactive gestures.

Conclusion
This study developed an educational application based on hand gesture recognition which we designed with enhanced the accuracy and efficiency. With the assist of augmented reality technology on children`s education, which provides a virtual-reality combination, representative three-dimensional animations and augmented circumstance are overlaid. Furthermore, as we specially integrate HCI with that augmented reality system, children are likely to accept more comprehensive and vivid information, so that they can receive a better education and have a profound understanding of the real world. And also, we will improve the correct recognition rate in the future.