A Multi-Sensor Fusion Approach for Music Rhythm-Based Rehabilitation System with Mobile Interaction

Abstract

Motor dysfunctions caused by diseases such as stroke and Parkinson’s disease present significant challenges for traditional home-based rehabilitation, including low training adherence and a lack of quantitative assessment. The proliferation of mobile devices and the advancement of multi-sensor technologies offer new pathways for personalized rehabilitation. Music rhythm can promote neural plasticity through the auditory-motor neural pathways, but its application is hindered by issues such as interference from single-sensor data, insufficient robustness of traditional fusion algorithms, lack of personalized rhythm adaptation, and imbalances in accuracy and real-time performance on mobile platforms. To address these challenges, this paper proposes a multi-sensor fusion mobile interaction music rhythm rehabilitation system. The core contributions are: 1) Developing a multi-sensor architecture comprising “Inertial Measurement Unit (IMU) + portable Electromyography (EMG) + built-in microphone” to achieve three-dimensional data collection of “motion trajectory-muscle activation-rhythm synchronization,” balancing portability and completeness; 2) Designing a lightweight algorithm chain that processes data through Kalman filtering and wavelet denoising, using convolutional neural network – long short-term memory (CNN-LSTM) for end-to-end fusion of multi-modal temporal features, combined with proximal policy optimization – generative adversarial network (PPO-GAN) for generating adaptive rhythms, overcoming mobile device resource constraints; 3) Conducting case-control experiments to establish a quantitative rehabilitation assessment system. This system provides an integrated solution for home rehabilitation, offering “precise perception-intelligent adaptation-quantitative evaluation,” and presents a new paradigm for interdisciplinary research in mobile healthcare and neurorehabilitation.