Design and Evaluation of an Augmented Reality and Generative AI-Driven English Speaking Training System

Abstract

This paper addresses the demand for English speaking training that integrates augmented reality (AR) with generative artificial intelligence (AI). We design and implement a system that supports highly real-time and robust speech interaction. At its core, the system employs mobile-oriented streaming speech recognition, enhanced with a Conformer encoder using causal convolution, a hybrid Connectionist Temporal Classification (CTC)/Attention decoding mechanism, dynamic block-wise streaming training, dynamic signal-to-noise ratio (SNR) data augmentation, and a multi-speaker speech filtering model. These techniques effectively address challenges in AR environments, such as complex noise, heterogeneous devices, and multi-speaker interference, thereby significantly improving recognition accuracy and response speed. Building on this foundation, we develop an English speaking training application that integrates AR scenarios with a generative AI dialogue engine, offering an immersive and adaptive practice environment. Experimental results demonstrate that the system substantially reduces word error rates in noisy and multi-speaker conditions, achieves latency suitable for real-time interaction, and provides a positive user experience, validating both its technical effectiveness and application feasibility.