InerTouchHand System – iTH Demonstration of a Glove Device with Distributed Inertial Sensors and Vibro-tactile Feedback

The InerTouchHand System shows the usage of an instrumented glove-like device for HMI-Human Machine Interaction applications. We explored the use of distributed inertial sensors and vibro-tactile stimulators on the hand. Distributed Inertial Measurement Units (IMUs) can be used to infer structure and reconstruct 6D pose of nodes, as well as relative motion. Our system uses MEMS IMUs on each fingertip for providing relative angular pose by using gravity as a vertical reference for acceleration measurements. Although not always fully observable it can be complemented by the IMUs’ gyro rotation and magnetometer measurements. When combined with vibrotactile stimulators a hand worn device or glove can provide spatial aware feedback. At exp.at’13 (2nd Experiment@ International Conference) an interactive demo presented InerTouchHand (iTH), the hand device for gesture recognition and HMI with touch feedback.


I. INTRODUCTION
In this work we explored the use of distributed inertial sensors and vibro-tactile stimulators on the hand, supported by a glove like device specifically designed for that purpose, to be used for enhanced virtual reality interaction and HMI (Human Machine Interaction) applications.Distributed MEMS accelerometers can provide rich information about the orientation relative to the vertical gravity reference, as well as dynamic information about motion.These sensors are so minute that they can be distributed over the hand, approaching the still illusive smartdust concept.However for sensing hand pose and motion distributed accelerometers are an interesting solution.The hand can be seen as a piece-wise rigid body with joint restricted movements, as well as some compliant parts.Minute sensors can be linked in a local bus and provide rich data on the pose and motion.The accelerometers complemented with gyro rotation sensing and magnetic orientation information provide a more robust full 6D pose recovery.Future implementations might even tap into the concept of energy harvesting, taking advantage of the hand kinetic energy to power the sensing or even the complete system.
Our target application for demonstration is the control of a human-like robotic hand.The pose of the fingers is mapped to the robotic hand, that is linked and controlled using ROS (Robotic Operating System). Figure 1.shows the sensors on the fingers and corresponding rendered view of the robotic hand simulated under ROS.

II. RELATED WORK
In order to have gesture recognition and humanmachine interfaces (HMI), many ways of attaching sensors to the human hand have been pursued.An acceleration sensing glove was, to one's best knowledge, first presented in [1] as an input device for static gestures and as pointing device for HMI.AcceleGlove has been presented as a whole-hand input device for virtual reality [2], although the focus is on HMI for mouse control and American Sign Language alphabet recognition.A more extensive survey of glove-based systems and their applications is presented by Dipietro [3].In Wang [4] a color based hand tracking scheme is presented, also references therein provide a good overview of hand gesture capture systems, namely based in color markers, reflective markers or active LEDs, as well as some based on placing sensors on the hand.Our approach is the latter, avoiding external cameras or projectors, and aims at having a self-contained system, using minute sensors to have a minimally intrusive system.

III. IMPLEMENTED INTERTOUCHHAND SYSTEM
The current prototype of the InerTouchHand is glove based, but further miniaturization can enable a lighter system in the future.It is a low cost solution that uses small MEMS sensors that retrieve orientation data from its magnetometers and accelerometers.A FPGA is used so that we can ensure the parallel synchronous data acquisition of all the sensors.Using reconfigurable logic also facilitates future updates, such as incorporating more onboard processing.The developed board also has power drivers for the vibro-tactile pancake motors, a wireless link, and uses fast charging novel batteries.This prototype can also be used in other configuration since the main system has universal connections to allow the final user to connect easily connect/disconnect sensors/actuators.The InerTouchHand prototype had been already integrated with Robot Operating System (ROS) to allow a generic use of the prototype.
The onboard firmware forwards the bundled data to the computer, where the developed software driver computes relative angles and controls the virtual robotic hand under the ROS framework.
For details of the underlying data processing see [5].In our previous work [5] hand distributed accelerometers were used to identify static gestures, including the Portuguese Sign Language alphabet.The feature space consisted in the relative angular pose between each fingertip and the palm, and the roll and pitch of the palm.Figure 3. provides an overview how to identify the relative pose between each sensor in the hand and a world reference that can be set in the back of the palm.The pose was determined by using gravity as a vertical reference.A nearest neighbor method identified the performed gesture against a library of gestures.This followed from the work of Lobo [6] on using gravity as a vertical reference for camera-IMU cross calibration and in robot inertial aided vision [7].With the new InerTouchHand prototype we enhanced the sensing and added vibro-tactile feedback.
The IMU sensor also provides magnetic and gyro rotation information which is used to complement the pose extracted by measuring acceleration.Magnetic direction helps to overcome the lack of observability when using gravity as a reference.The drift in the gyro rotation does not pose a relevant obstacle to accuracy given the fact that its use is proposed as a redundancy and for a more accuracy pose recovery.

IV. INTERACTIVE DEMONSTRATION
At exp.at'13 the system was available for the interactive demo, where users wore the instrumented glove and control the virtual robot hand, or move the wooden model of the hand, as shown in Figure 1

SHORT PAPER INERTOUCHHAND SYSTEM -ITH V. CONCLUSION
In this paper we briefly presented a glove like device that incorporates distributed inertial sensing and vibrotactile feedback.The demo at exp.at'13 showed the system in action.While this system is somewhat complex, the components are not expensive, and subparts can be assembled for simpler interaction purposes to be more easily available to users of remote and virtual labs.In Figure 5. shows the stand at exp.at'13 where users were able to interact with the system and understand the potential of the technology and science involved for further applications.

Figure 1 .
Figure 1.Sensors on the hand controlling robotic hand.

Figure 2 .
Figure 2. shows the iTH hardware.InerTouchHand is composed of one FPGA small board (Terasic DE0-nano); 11 inertial and magnetic sensors (CMPS10); 14 vibration motors; one wireless module (WiFly RN-XV); and two batteries (A123 LiFePO4 cells).We are currently assembling an improved version with final boards and better sensors.

Figure 3 .
Figure 3. Frames of reference overview when each sensor is placed in tip of the fingers.

Figure 4 .
Figure 4. InerTouchHand (iTH) glove mounted or on single fingers (on wooden hand in the picture).