An Explainable Early Warning System for Student Academic Risk Prediction Based on Time-Windowing and SHAP

Yanyan Zhu; Niu Shi; Yan Chen; Peiqiang Wang

doi:10.3991/ijet.v21i04.62480

Authors

Yanyan Zhu Pingdingshan Polytechnic College, Pingdingshan, Henan, China
Niu Shi Pingdingshan Polytechnic College, Pingdingshan, Henan, China
Yan Chen Pingdingshan Tianan Coal Mining Co., Ltd., Pingdingshan, Henan, China
Peiqiang Wang Pingdingshan Polytechnic College, Pingdingshan, Henan, China

DOI:

https://doi.org/10.3991/ijet.v21i04.62480

Keywords:

Educational Data Mining; Early Warning System; Explainable Artificial Intelligence; SHAP; XGBoost; Online Learning

Abstract

High dropout rates in online learning remain a persistent challenge, while many early warning systems (EWS) provide risk scores without explaining what instructors should do next. This study proposes an explainable EWS that integrates XGBoost with SHAP under a strict 30-day time-windowing protocol on the OULAD dataset, enhanced with five behavior-pattern features. The 30-day window is validated through a 14/30/60-day ablation, with PR-AUC reported to address class imbalance and robustness tested on the KDD Cup 2015 MOOC dataset. To preserve prospective prediction, OULAD records from students who had already withdrawn before the prediction-window endpoint were excluded. On OULAD Day-30, XGBoost achieved the strongest ranking and F1 performance under student-grouped cross-validation, with AUC 0.7865, PR-AUC 0.7652, and F1 0.6745. Day-14 reduced AUC to 0.7353 due to sparse early assignment submissions, whereas Day-60 improved AUC to 0.8154 but reduced intervention time. On KDD Cup 2015, XGBoost achieved AUC 0.8639 and PR-AUC 0.9498 under username-grouped cross-validation. SHAP identified first-assignment score, assignment lead time, active days, and selected academic-background features as key risk drivers, supporting targeted interventions rather than generic risk alerts.

References

[1] T. R. Liyanagunawardena, A. A. Adams, and S. A. Williams, “MOOCs: A systematic study of the published literature 2008-2012,” International Review of Research in Open and Distance Learning, vol. 14, no. 3, 2013, doi: 10.19173/irrodl.v14i3.1455.

[2] K. F. Hew and W. S. Cheung, “Students’ and instructors’ use of massive open online courses (MOOCs): Motivations and challenges,” 2014. doi: 10.1016/j.edurev.2014.05.001.

[3] Y. Wang and R. Baker, “Content or platform : Why do students complete MOOCs ?,” J. Online Learn. Teach., vol. 11, no. 1, 2015.

[4] C. Romero and S. Ventura, “Educational data mining and learning analytics: An updated survey,” Wiley Interdiscip. Rev. Data Min. Knowl. Discov., vol. 10, no. 3, 2020, doi: 10.1002/widm.1355.

[5] D. Gašević and A. Merceron, The Handbook of Learning Analytics. 2022. doi: 10.18608/hla22.

[6] J. Gardner and C. Brooks, “Student success prediction in MOOCs,” User Model. User-adapt. Interact., vol. 28, no. 2, 2018, doi: 10.1007/s11257-018-9203-z.

[7] Z. Swiecki et al., “Assessment in the age of artificial intelligence,” Computers and Education: Artificial Intelligence, vol. 3, 2022, doi: 10.1016/j.caeai.2022.100075.

[8] C. Brooks and C. Thompson, “Predictive Modelling in Teaching and Learning,” in The Handbook of Learning Analytics, 2022. doi: 10.18608/hla22.003.

[9] E. Howard, M. Meehan, and A. Parnell, “Contrasting prediction methods for early warning systems at undergraduate level,” Internet and Higher Education, vol. 37, 2018, doi: 10.1016/j.iheduc.2018.02.001.

[10] A. Villar and C. R. V. de Andrade, “Supervised machine learning algorithms for predicting student dropout and academic success: a comparative study,” Discover Artificial Intelligence, vol. 4, no. 1, 2024, doi: 10.1007/s44163-023-00079-z.

[11] P. Cortez and A. Silva, “Using data mining to predict secondary school student performance,” in 15th European Concurrent Engineering Conference 2008, ECEC 2008 - 5th Future Business Technology Conference, FUBUTEC 2008, 2008.

[12] A. A. Mubarak, H. Cao, and S. A. M. Ahmed, “Predictive learning analytics using deep learning model in MOOCs’ courses videos,” Educ. Inf. Technol. (Dordr)., vol. 26, no. 1, 2021, doi: 10.1007/s10639-020-10273-6.

[13] Singh, “Enhancing Student Retention and Performance Prediction in E-Learning Environments Using Supervised Machine Learning Models,” Journal of Informatics Education and Research, 2026, doi: 10.52783/jier.v5i4.4055.

[14] T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016. doi: 10.1145/2939672.2939785.

[15] A. Bettahi, F. Z. Belouadha, and H. Harroud, “A Modular and Explainable Machine Learning Pipeline for Student Dropout Prediction in Higher Education,” Algorithms, vol. 18, no. 10, 2025, doi: 10.3390/a18100662.

[16] G. Ke et al., “LightGBM: A highly efficient gradient boosting decision tree,” in Advances in Neural Information Processing Systems, 2017.

[17] L. Grinsztajn, E. Oyallon, and G. Varoquaux, “Why do tree-based models still outperform deep learning on typical tabular data?,” in Advances in Neural Information Processing Systems, 2022.

[18] M. Hlosta, Z. Zdrahal, and J. Zendulka, “Ouroboros: Early identification of at-risk students without models based on legacy data,” in ACM International Conference Proceeding Series, 2017. doi: 10.1145/3027385.3027449.

[19] S. M. Lundberg and S. I. Lee, “A unified approach to interpreting model predictions,” in Advances in Neural Information Processing Systems, 2017.

[20] M. T. Ribeiro, S. Singh, and C. Guestrin, “‘why should i trust you?’ explaining the predictions of any classifier,” in NAACL-HLT 2016 - 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Demonstrations Session, 2016. doi: 10.18653/v1/n16-3020.

[21] H. Khosravi et al., “Explainable Artificial Intelligence in education,” Computers and Education: Artificial Intelligence, vol. 3, 2022, doi: 10.1016/j.caeai.2022.100074.

[22] Z. Liu, X. Zhou, and Y. Liu, “Student Dropout Prediction Using Ensemble Learning with SHAP-Based Explainable AI Analysis,” Journal of Social Systems and Policy Analysis, vol. 2, no. 3, 2025, doi: 10.62762/jsspa.2025.321501.

[23] T. Saito and M. Rehmsmeier, “The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets,” PLoS One, vol. 10, no. 3, 2015, doi: 10.1371/journal.pone.0118432.

[24] J. Kuzilek, M. Hlosta, and Z. Zdrahal, “Data Descriptor: Open University Learning Analytics dataset,” Sci. Data, vol. 4, 2017, doi: 10.1038/sdata.2017.171.

[25] W. Feng, J. Tang, and T. X. Liu, “Understanding dropouts in MOOCs,” in 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, 31st Innovative Applications of Artificial Intelligence Conference, IAAI 2019 and the 9th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, 2019. doi: 10.1609/aaai.v33i01.3301517.

[26] M. R. Khatun, M. A. Mim, M. M. Tasin, and M. M. Hossain, “A hybrid framework of statistical, machine learning, and explainable AI methods for school dropout prediction,” PLoS One, vol. 20, no. 9 September, 2025, doi: 10.1371/journal.pone.0331917.

[27] S. M. Lundberg et al., “From local explanations to global understanding with explainable AI for trees,” Nat. Mach. Intell., vol. 2, no. 1, 2020, doi: 10.1038/s42256-019-0138-9.

[28] M. Adnan et al., “Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention Using Machine Learning Models,” IEEE Access, vol. 9, 2021, doi: 10.1109/ACCESS.2021.3049446.

[29] M. Nagy and R. Molontay, “Interpretable Dropout Prediction: Towards XAI-Based Personalized Intervention,” Int. J. Artif. Intell. Educ., vol. 34, no. 2, 2024, doi: 10.1007/s40593-023-00331-8.

[30] G. Haixiang, L. Yijing, J. Shang, G. Mingyun, H. Yuanyue, and G. Bing, “Learning from class-imbalanced data: Review of methods and applications,” 2017. doi: 10.1016/j.eswa.2016.12.035.

[31] C. Herodotou, G. Naydenova, A. Boroowa, A. Gilmour, and B. Rienties, “How can predictive learning analytics and motivational interventions increase student retention and enhance administrative support in distance education?,” Journal of Learning Analytics, vol. 7, no. 2, 2020, doi: 10.18608/JLA.2020.72.4.

[32] I. Borrella and E. Ponce-Cueto, “Design Principles and Impact of a Learning Analytics Dashboard: Evidence from a Randomized MOOC Experiment,” Applied Sciences (Switzerland), vol. 15, no. 21, 2025, doi: 10.3390/app152111493.