A Multimodal Interaction Mechanism for Mobile English Learning in Higher Education

Abstract

Sensor-enabled mobile devices have provided new opportunities for interaction analysis and cognitive assessment in mobile English learning within higher education. However, existing approaches are constrained by insufficient data synchronization accuracy, limited flexibility in multimodal feature fusion, high computational costs associated with sequence modeling, and the lack of effective intervention mechanisms. To address these limitations, an edgeside multimodal interaction analysis and adaptive intervention framework was developed. Temporal alignment of heterogeneous sensor data was achieved, while lightweight crossmodal feature fusion was implemented using channel-attention-based representations. A bidirectional Mamba architecture was introduced for processing long-sequence data, and multitask learning was employed to simultaneously evaluate interaction quality and cognitive load. Behavioral evolution patterns were further explored through sequential pattern mining and cognitive network analysis. On this basis, a hierarchical feedback strategy was designed to enable intelligent intervention. The proposed framework provides both theoretical foundations and technical support for the development of intelligent mobile educational applications.