Applying Deep Learning and Computer Vision Techniques for an e-Sport and Smart Coaching System Using a Multiview Dataset: Case of Shotokan Karate

Abstract

Smart coaching and e-sport platforms have shown a great interest in the recent research studies. Through this study, we aim to globalize the practice of sport, especially Shotokan Karate, by connecting participants and coaches on an international scale through the integration of Artificial Intelligence techniques such as computer vision and deep learning, to give the possibility of carrying out national and international virtual training courses without logistical constraints. The proposed work aims to apply the latest action detection, action recognition, and action classification methods for different Karate movements using the LSTM and the ST-GCN algorithms and proposes these movements in 3D using Video Inference for Human Body Pose and Shape Estimation (VIBE). Our proper Multiview Dataset contains pose estimations of a set of basic movements captured by a karate Shotokan expert (6th DAN Black Belt) from three views (Front view, Left view, and Right view) using OpenPose and FastPose for human body keypoint detection. The current study sets out to detect, recognize, classify and score different participants' movements. We achieved greater than 96% recognition accuracy of this dataset using the LSTM algorithm, and 91.01% using the ST-GCN algorithm.