A Gamified Approach for Hand Rehabilitation Device

— This work details developments made in a system for hand rehabilitation, that aims to improve recovery of fine motor control, mostly for those recovering from stroke. The system consists of an instrumented device that is used to interact with a variety of games designed to improve fine motor control, enhancing rehabilitation practices. These games were tested with actual disa-bled individuals and therapists, having received overall positive feedback.


Introduction
Stroke remains one of the leading causes of disability throughout the world. In 2013, it was the third greatest cause of disability, having been responsible for 113 million disability-adjusted life-years [1]. Most stroke patients suffer from some kind of upper limb impairment [2], which can often be overcome through motor rehabilitation. This rehabilitation is of critical importance, as good control over the hands is fundamental for insuring one's independence and quality of life. Specifically, grip force control plays a major role on daily tasks, mostly on manipulating everyday objects. Training of grip force control is also important in other scenarios, such as the case of individuals with newly implanted myoelectric prosthetics [3]. The use of augmented feedback has been shown to enhance rehabilitation for the two aforementioned cases [4,5]. Furthermore, the use of devices that enable automatic recording and objective measurement of the patients' capabilities has been of growing interest, as most current rehabilitation practices are based on subjective progress evaluation [6,7].
The use of an instrumented device for training and accessing hand dexterity and grip force control of a patient enables both the use of augmented feedback and progress recording, besides allowing the incorporation of games in the rehabilitation sessions. The use of games in rehabilitation is also increasing, as they have been shown to increase patient engagement and therapy effectiveness [8][9][10]. There are a number of devices under development that provide some of these features [11][12][13], but to the best of the authors knowledge there are no commercially available devices that provide grip force control training. This paper further details developments made in a hand rehabilitation system previously developed by the authors [14], with a focus on the development of a set of therapeutic games.

Hand Rehabilitation System
With the aim of creating a system that enables remote or autonomous hand rehabilitation, a device with different transducers and an accompanying virtual environment were developed.
The instrumented device (Figure 1 a)) was conceived in a way that allows easy replacement of the outward shell, in order to replicate different objects. It houses the sensors that enable measurement of the applied grip force, spatial orientation and detection of whether or not it is placed on a surface. This information is transmitted wirelessly, either through MiWi or Bluetooth Low Energy to a computer or mobile device. Furthermore, it also contains a battery and charging circuit for true wireless operation. For the grip force measurement, a custom-made load cell is used. Lift detection is achieved with a light dependent resistor (LDR) [14].
Orientation is measured with a commercially available inertial measurement unit (IMU). Earlier prototypes used an IMU that was able to implement sensor fusion algorithms to combine raw values from its accelerometer and gyroscope into a quaternion, that was output to the microcontroller. However, there was drift in the rotation around the vertical (gravity) axis. This led to a constant increase in the discrepancy between the real position of the device and the virtual model that was noticeable on longer usage sessions. This problem was initially mitigated through frequent calibration of the device. However, as some of the developed applications eventually required longer continuous sessions, this approach proved not to be adequate.
In order to improve the orientation measurement accuracy and compensate drift over time, in the latest prototype the IMU was replaced with one that includes a 3-axis magnetometer. The magnetometer serves as an absolute heading reference that can be used to continuously correct the estimated values from the gyroscope. This IMU, however, is incapable of internally combining the magnetometer data with that of the other two sensors, which meant that the sensor fusion algorithms had to be run on the main microcontroller. Thus, the raw data acquired from both accelerometer and gyroscope axis in the IMU is fused with the magnetometer raw data using a Madgwick filter implemented on the microcontroller [15]. We found this approach was able to effectively compensate the heading drift to within values compatible with our applications.
As for software, the first developed application was a virtual environment that displays a 3D model of the device. This model moves in accordance to the device: replicating its orientation and deforming proportionally to the applied force, as can be seen in Figure 1 b). This environment also allows the recording of sessions for later playback and analysis by either the patient or the therapist. It also allows session streaming for real-time monitoring by the therapist.

Game integration
From the beginning, this hand rehabilitation system was developed taking into account its potential expandability, mainly through the integration of new software features. By developing the virtual environment using the Unity game engine, it was ensured the development of games that fully use the hardware for different rehabilitation tasks, allowing the portability of the games for different hardware/software platforms (Android/ IOS/ Windows). The first games developed for the system were "Force sequence" and "Flying force". These games are currently under test with patients in a medical institute. A third game "Cup follower" is still under development.
Force sequence This game takes advantage of the orientation and force sensing in the device in order to provide a kind of occupational therapy. The game has a set of levels, where the patients are instructed to either squeeze the device and maintain a certain force value or tilt the device, mimicking the action of pouring water from a glass. For both modes there is a scoring system, that allows patients to keep track of their performances.
On the force control levels (Figure 2 a)), a target force and duration are presented on-screen. The patients must then sustain a force within a certain tolerance of the target value for the specified duration. If the force remains within the specified limits for the duration of the timer, it is possible to advance to the following level. Otherwise, the user can try again or skip the level. Throughout the exercise, augmented feedback of the applied force is provided as deformation of the virtual model.
On the water pouring levels (Figure 2 b)) a written command requests that the patients pour water either forwards, backwards, left or right for a given time period.
There is a 3D model of the cup on-screen that rotates in the same way as the physical device and from which water will pour if tilted past a certain level in any direction. This provides audiovisual feedback as well as improved immersion to the patients. If the patients are able to pour virtual water in the right direction for the specified time they go on to the next level. As with the previous game, they can attempt for as many times as needed or skip to the following level if desired. Flying force This game only uses the force sensing capabilities of the device to enable a simple form of interaction with the potential to increase engagement in rehabilitation exercises, in particular with children with special needs.
It includes a dynamic scene, where an airplane is constantly moving right and falling (moving down) at the same time. Pressing the cup with a certain force causes the airplane to rapidly move upwards. The goal is to avoid a random set of mountains, appearing on the top and bottom of the screen. A gameplay example image is shown in Figure 3. The game is scored in accordance to the time that the patient is able to keep avoiding the obstacles.
Even though this game is very simple, it can be beneficial for training of grip force control, as the user must constantly control the force it is applying on the device: they must be able to quickly increase the force to dodge an obstacle and relieve it again without dropping the device to prepare for the next, all as a coordinated movement. Cup follower The objective of the game is to place the blue cup (controlled by the user) in the same place and orientation as the reference red cup. The application has two difficulty levels, "Easy" and "Hard". The main difference between the two modes is how the blue cup is moved: on "Easy" mode the user only needs to turn the cup in the direction he wants it to move, while on "Hard" mode, the user must also apply a force greater than 1 kgf.
The positioning and orientation process is addressed in two phases. While positioning, tilting the cup moves it in the horizontal plane in a certain direction. When it is in place, the cup movement locks and tilting the cup controls its orientation. When the orientation is the same as the reference red cup, the cup will turn green, and the user must hold that orientation until the cup disappears.
Once the cup disappears, the user is awarded 100 points, 30 seconds are added to the timer and a new reference cup will spawn in a random position and orientation. The user should then repeat the process of positioning and reorienting the blue cup, until the time runs out.
To help with user perception of the cups placement, there's a top down view of the scene in the top right corner of the application, with two circles representing each of the cups in the scene. If the centers of the circles are coincident, then the placement is correct.

Discussion
The first game, "Force sequence", was intended as a comprehensive hand training suite, as it allows both grip control training as well as simple coordination and range of motion exercises.
"Flying force" enables a higher level of abstraction for the therapy, as the patient can play it as an actual game, without concentrating on the rehabilitation aspects, which has previously been shown to be desirable [8].
"Cup follower" was developed with the aim of helping users improve their hand coordination. As the game demands controlled hand movement in order to interact with the visual model, it enables the development of fine motor skills. In terms of fine motion, this game is more demanding than "Force sequence", which means it could be helpful in the latest stages of therapy.
In all of these games, there is a scoring system that can both keep the patients motivated and provide useful metrics for patient progress monitoring.
A pre-evaluation of games acceptance in clinical environment has been made, in a different Portuguese Institution, involved in rehabilitation tasks and occupational therapy support for people with cognitive problems.
A specific questionnaire was designed for the pre-evaluation procedure. A total of 27 patients were inquired, where 13 were female and 14 were male, with an average age of 54 years old (ranging between 25 and 80 years old). As the patients were from two different institutions, they were inquired in two groups: the first group includes patients with pathologies associated with the central nervous system (1/3) and others with several physical and muscular pathologies (2/3), while the second group is comprised of patients mostly with mental disorders and low cognitive problems.
In the first group of 21 patients, simple evaluations were performed based on answer average and standard deviation. The questionnaire comprises 22 brief questions aiming to inquire about three main topics: patient satisfaction, difficulty level and game features, and benefits and results of use. Each question was answered using the scale: agree -3, neutral -2 and disagree -1.
The results showed good acceptance (2.71±0.63), motivation (2.90±0.29) and ease of use, whose lower level (2.57±0.58) was in line with the system objective -to help the rehabilitation tasks. All patients demonstrated interest in using the system at home, if available. This feedback is a very good indicator of game acceptance considering that this group of patients under rehabilitation is naturally characterized by a high level of discouragement, due to the typical slow recovery process.
For the second group (of 6 patients), the evaluation was based on the therapists' observation concerning the games' patient-friendly characteristics, since this group of patients has mental disorders and low cognitive problems. The therapists consider that the patients' feedback on playing the games was very good, because these patients did not present motor limitations and evidenced generous acceptance.
In both rehabilitation units, a longer treatment period without the need of specialized therapist attendance was achieved. Patients also referred that after several sessions using the system, the perception of the proper grip force for handling daily objects improved.
Since this form of rehabilitation is a slow process that needs an extended period of evaluation in order to quantify an improvement in the state of health, it was not yet possible to obtain feedback regarding the influence of the games in the patient improvement over time. Furthermore, the developed games are still being used by patients in a clinical environment ( Figure 5).

Conclusion
This paper reports the integration of an instrumented hand rehabilitation device with a variety of games intended to enhance rehabilitation practices.
This system was used in the rehabilitation program of a physiatrist unit of a private Portuguese Hospital and the results will be clinically evaluated in near future.
All patients also declared to be interested in using the system. In both cases, the therapists considered the system as very interesting for use in rehabilitation and in support of long distance training activities.