Integrated UAV Swarm and Human-Machine Interaction Platform for Real-Time Mobile Training and Evaluation

Abstract

Unmanned aerial vehicle (UAV) swarm systems have become a frontier in the field of intelligent unmanned systems. However, the high costs and risks of physical experiments pose significant challenges for professional talent development. Building an efficient training and evaluation simulation platform is key to overcoming this challenge. Existing UAV simulation platforms often rely on traditional input devices such as keyboards and mice, which have limitations in terms of intuitive interaction, immersion, and real-time evaluation capabilities. To address these issues, this paper focuses on the mobile device environment and aims to design and construct a multi-agent system educational platform integrating UAV swarm simulation and human-machine interaction. First, a dual-modal mobile interaction framework based on gestures and voice commands is proposed. Next, a high-precision command recognition system is developed by creating a gesture database, performing data preprocessing, feature fusion, and convolutional neural network (CNN) model training, and integrating a self-trained cloud-based speech recognition model. Finally, a complete platform prototype is developed, enabling users to train in real-time UAV swarm formation control, obstacle avoidance, and other tasks through natural interaction on mobile devices, while simultaneously providing a quantitative evaluation of the operation process. This paper integrates the simulation system, multimodal interaction, and real-time evaluation mechanisms, establishing a unified “trainingevaluation” educational platform, offering an innovative solution for multi-agent system teaching and skill assessment.