Digital Twin and Mobile Touch Interaction for High-Rise Construction Visualization and Collaboration

Abstract

Mobile digital twin systems for construction progress management are constrained by low interaction efficiency, as touch gestures are typically limited to viewpoint manipulation and are not semantically linked to progress-management operations. To address this limitation, a semantic mapping method from touch gestures to construction progress states was proposed. A bidirectional mapping model between gesture kinematic features and construction progress semantic vectors was established, whereby touch interaction was extended from conventional viewpoint manipulation to semantic querying and collaborative command execution. A seven-dimensional gesture feature vector and a five-dimensional progress semantic vector were constructed, and a mapping matrix was trained through supervised learning. An inverse mapping mechanism was further designed to drive visual feedback in the digital twin model, thereby forming a closed interaction loop. The mapping rules were designed to support key interaction scenarios: two-finger pinch gestures were mapped to switching between detailed and overview representations of construction progress; single-finger long presses were used to trigger progress information cards and initiate collaboration; two-finger rotation gestures were mapped to navigation along the four-dimensional timeline; and three-finger downward swipes were used to activate comparisons between planned and actual progress. Incremental learning was additionally incorporated to enable personalized adaptation. A prototype system was developed using the web graphics library, while Draco compression and level-of-detail rendering were employed to address the performance constraints associated with Building Information Modeling (BIM) models on mobile devices. Experimental results demonstrated that the proposed method effectively improved interaction efficiency and facilitated on-site collaboration in mobile digital twin systems.