VR/AR-Based Mobile Interaction for Virtual Simulation Training in Higher Education

Abstract

In the context of practical instruction in higher education, traditional simulation training is constrained by factors such as spatial limitations, equipment availability, safety concerns, and high costs, making it difficult to meet the demands for hands-on experience. Mobile interaction technology based on virtual reality (VR) and augmented reality (AR), characterized by immersion, strong interactivity, and portability, provides an innovative pathway for virtual simulation training, emerging as a promising solution to the limitations of conventional models. Although the application of VR/AR in virtual simulation has been preliminarily explored, current approaches exhibit significant limitations: interaction modalities are predominantly restricted to unimodal gestures or voice commands, lacking coordinated multimodal integration; system feedback is often reliant on predefined rules, limiting the ability to deliver adaptive and precise guidance; and model optimization fails to adequately leverage prior feedback, resulting in suboptimal learning efficiency and adaptability. To address these challenges, a reinforcement learning framework for VR/AR-based mobile interaction in virtual simulation training was developed in this study. This framework includes the design of a multimodal feature extraction method combining voice and gesture inputs, the construction of an auxiliary decision-making and feedback mechanism, the formulation of principles for reward function design, and the proposal of a model optimization strategy informed by prior feedback data. The core innovations of this study are threefold. First, multimodal interaction feature extraction—integrating both speech and gesture inputs—was implemented to overcome the limitations of unimodal interaction and to enhance interaction naturalness. Second, a dynamic feedback mechanism based on real-time operational data was established, replacing traditional rule-based feedback systems to improve instructional precision. Third, prior feedback information was embedded within the model optimization loop to accelerate model iteration and enhance adaptability across diverse training scenarios. This study provides technological support for improving the quality of virtual simulation training in higher education and offers novel insights into the integration of VR/AR technology with educational practice.