A Lightweight LINE-Based Knowledge Chatbot for Mobile Technical Certification Learning: An Empirical Study with ChatGPT as a Baseline

Cheng-Hsiu Li

doi:10.3991/ijim.v20i18.62472

Authors

Cheng-Hsiu Li National Taitung Junior College, Taitung, Taiwan, R.O.C. https://orcid.org/0000-0002-7821-2578

DOI:

https://doi.org/10.3991/ijim.v20i18.62472

Keywords:

knowledge-based chatbot, LINE-based chatbot, mobile learning, technical certification learning, ChatGPT, learning effectiveness, equivalence analysis

Abstract

This study developed a lightweight LINE-based mobile knowledge chatbot for the Level B Computer Hardware Repair certification subject and examined its instructional effectiveness in comparison with ChatGPT. The system was deployed through the LINE mobile messaging platform, allowing learners to access certification-related explanations on their own smartphones through natural-language interaction. Unlike large language model (LLM)-based systems that rely on retrieval-augmented generation (RAG), vector embedding, and index construction, the proposed chatbot was built around a Knowledge Base Recommendation Algorithm (KBRA), which generates instructional responses through structured keyword matching and fuzzy search over a domain-specific knowledge base. This mobile deployment was designed to provide a low-cost, easily accessible, and interactive learning support tool for bounded technical certification domains. A nonequivalent pretest–posttest quasi-experimental design was adopted. The experimental group used the LINE-based KBRA chatbot for mobile self-directed learning, whereas the control group used ChatGPT. Both groups participated in an eight-week intervention and completed pre-test and post-test assessments consisting of 10 open-ended questions. The results showed that both groups achieved substantial learning gains. After controlling for pre-test scores, analysis of covariance (ANCOVA) revealed no significant difference in post-test performance between the two groups, and the effect size associated with group membership was negligible. In addition, the two one-sided tests (TOST) equivalence analyses indicated that the learning gains of the two groups were practically equivalent within a ±10-point margin. These findings suggest that, in this bounded certification-learning context, the LINE-based KBRA chatbot may provide learning support comparable to ChatGPT under the conditions examined. However, this comparability should be interpreted cautiously because the study was conducted with a small sample, a single certification domain, and a specific eight-week self-directed learning setting. The proposed chatbot should therefore be regarded as a feasible and deployment-friendly learning support option for structured technical certification learning, rather than as a general alternative to ChatGPT.

Author Biography

Cheng-Hsiu Li, National Taitung Junior College, Taitung, Taiwan, R.O.C.

Cheng-Hsiu Li is an Associate Professor in the Department of Information Management at National Taitung Junior College, Taiwan (R.O.C.). He received his Ph.D. degree in 2014. His research interests include e-learning, the Internet of Things (IoT), virtual reality (VR), mixed reality (MR), and computer vision.

References

[1] R. L. Burke, CAI Sourcebook. Englewood Cliffs, NJ: Prentice-Hall, 1982.

[2] B. F. Skinner, The Technology of Teaching. New York: Appleton-Century-Crofts, 1968.

[3] W. C. Hsu and C. H. Li, “A competency-based guided-learning algorithm applied on adaptively guiding e-learning,” Interactive Learning Environments, vol. 23, no. 1, pp. 106–125, 2015. https://doi.org/10.1080/10494820.2012.745432

[4] C. H. Li, “A proficiency-based learning algorithm applied to e-learning system for knowledge-oriented course,” MATEC Web of Conferences, vol. 169, article no. 01002, 2018. https://doi.org/10.1051/matecconf/201816901002

[5] C. H. Li, “Integrating gamification elements into a personalized cognitive mobile-learning LINE Bot,” Emerging Science Innovation, vol. 3, pp. 27–42, 2024. https://doi.org/10.46604/emsi.2024.12980

[6] C. H. Li, “Enhancing vocational certification learning through a gamified chatbot: Evidence from a quasi-experimental study,” Advances in Mobile Learning Educational Research, vol. 6, no. 1, pp. 1653–1667, 2026. https://doi.org/10.25082/AMLER.2026.01.001

[7] H. Essel, D. Vlachopoulos, A. Tachie-Menson, E. Johnson, and P. Baah, “The impact of a virtual teaching assistant (chatbot) on students’ learning in Ghanaian higher education,” International Journal of Educational Technology in Higher Education, vol. 19, article no. 57, 2022. https://doi.org/10.1186/s41239-022-00362-6

[8] L. Chambers and W. Owen, “The efficacy of GenAI tools in postsecondary education,” Brock Education Journal, vol. 33, no. 3, pp. 57–74, 2024. https://doi.org/10.26522/brocked.v33i3.1178

[9] I. Pesovski, R. Santos, R. Henriques, and V. Trajkovik, “Generative AI for customizable learning experiences,” Sustainability, vol. 16, no. 7, article no. 3034, 2024. https://doi.org/10.3390/su16073034

[10] A. O’Cain, B. Fedoruk, Z. Masri, R. Frost, and A. Alahmar, “A system for the improvement of educational assessment using intelligent conversational agents,” SSRN Electronic Journal, 2023. https://doi.org/10.2139/ssrn.4393234

[11] D. Chang, M. Lin, S. Hajian, and Q. Wang, “Educational design principles of using AI chatbot that supports self-regulated learning in education: Goal setting, feedback, and personalization,” Sustainability, vol. 15, no. 17, article no. 12921, 2023. https://doi.org/10.3390/su151712921

[12] F. Ali, D. Choy, S. Divaharan, H. Tay, and W. Chen, “Supporting self-directed learning and self-assessment using TeacherGAIA, a generative AI chatbot application: Learning approaches and prompt engineering,” Learning: Research and Practice, vol. 9, no. 2, pp. 135–147, 2023. https://doi.org/10.1080/23735082.2023.2258886

[13] F. Kieser, P. Wulff, J. Kuhn, and S. Küchemann, “Educational data augmentation in physics education research using ChatGPT,” Physical Review Physics Education Research, vol. 19, article no. 020150, 2023. https://doi.org/10.1103/PhysRevPhysEducRes.19.020150

[14] Q. Xia, X. Weng, O. Fan, T. Lin, and T. Chiu, “A scoping review on how generative artificial intelligence transforms assessment in higher education,” International Journal of Educational Technology in Higher Education, vol. 21, article no. 40, pp. 1–22, 2024. https://doi.org/10.1186/s41239-024-00468-z

[15] D. Baidoo-Anu and L. Ansah, “Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning,” SSRN Electronic Journal, 2023. https://doi.org/10.2139/ssrn.4337484

[16] M. Giannakos, R. Azevedo, P. Brusilovsky, M. Cukurova, Y. Dimitriadis, D. Leo, S. Järvelä, M. Mavrikis, and B. Rienties, “The promise and challenges of generative AI in education,” Behaviour & Information Technology, vol. 44, no. 11, pp. 2518–2544, 2024. https://doi.org/10.1080/0144929X.2024.2394886

[17] U. Mittal, S. Sai, V. Chamola, and D. Sangwan, “A comprehensive review on generative AI for education,” IEEE Access, vol. 12, pp. 142733–142759, 2024. https://doi.org/10.1109/ACCESS.2024.3468368

[18] M. Allen, U. Naeem, and S. Gill, “Q-Module-Bot: A generative AI-based question and answer bot for module teaching support,” IEEE Transactions on Education, vol. 67, pp. 793–802, 2024. https://doi.org/10.1109/TE.2024.3435427

[19] W. Ma, W. Ma, Y. Hu, and X. Bi, “The who, why, and how of AI-based chatbots for learning and teaching in higher education: A systematic review,” Education and Information Technologies, vol. 30, pp. 7781–7805, 2024. https://doi.org/10.1007/s10639-024-13128-6

[20] Y. A. Malkov and D. A. Yashunin, “Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 4, pp. 824–836, 2020. https://doi.org/10.1109/TPAMI.2018.2889473

[21] J. Johnson, M. Douze, and H. Jégou, “Billion-scale similarity search with GPUs,” IEEE Transactions on Big Data, vol. 7, no. 3, pp. 535–547, 2021. https://doi.org/10.1109/TBDATA.2019.2921572

[22] J. Wang, X. Yi, R. Guo, H. Jin, P. Xu, S. Li, X. Wang, X. Guo, C. Li, X. Xu, K. Yu, Y. Yuan, Y. Zou, J. Long, Y. Cai, Z. Li, Z. Zhang, Y. Mo, J. Gu, R. Jiang, Y. Wei, and C. Xie, “Milvus: A purpose-built vector data management system,” in Proceedings of the 2021 International Conference on Management of Data (SIGMOD ’21), 2021, pp. 2614–2627. https://doi.org/10.1145/3448016.3457550

[23] M. Aumüller, E. Bernhardsson, and A. Faithfull, “ANN-Benchmarks: A benchmarking tool for approximate nearest neighbor algorithms,” Information Systems, vol. 87, article no. 101374, 2020. https://doi.org/10.1016/j.is.2019.02.006

[24] M. A. Kuhail, N. Alturki, S. Alramlawi, and K. Alhejori, “Interacting with educational chatbots: A systematic review,” Education and Information Technologies, vol. 28, pp. 973–1018, 2023. https://doi.org/10.1007/s10639-022-11177-3

[25] I. El mourabit, S. Jai Andaloussi, O. Ouchetto, and M. Miyara, “AI chatbots in higher education: Opportunities and challenges for personalized and mobile learning,” International Journal of Interactive Mobile Technologies, vol. 19, no. 12, pp. 19–37, 2025. https://doi.org/10.3991/ijim.v19i12.54163

[26] C. Papakostas, C. Troussas, A. Krouska, and C. Sgouropoulou, “A rule-based chatbot offering personalized guidance in computer programming education,” in Generative Intelligence and Intelligent Tutoring Systems, 2024, pp. 253–264. https://doi.org/10.1007/978-3-031-63031-6_22

[27] E. Kasneci, K. Seßler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Günnemann, E. Hüllermeier, S. Krusche, G. Kutyniok, T. Michaeli, C. Nerdel, J. Pfeffer, O. Poquet, M. Sailer, A. Schmidt, T. Seidel, M. Stadler, J. Weller, J. Kuhn, and G. Kasneci, “ChatGPT for good? On opportunities and challenges of large language models for education,” Learning and Individual Differences, vol. 103, article no. 102274, 2023. https://doi.org/10.1016/j.lindif.2023.102274

[28] A. D. Samala, S. Papadakis, and S. Rawas, “Global insights into mobile learning in higher education: A PRISMA-guided bibliometric analysis from 2007 to 2023,” International Journal of Educational Reform, 2025. https://doi.org/10.1177/10567879251341869

[29] H. Uğraş, M. Uğraş, S. Papadakis, and M. Kalogiannakis, “ChatGPT-supported education in primary schools: The potential of ChatGPT for sustainable practices,” Sustainability, vol. 16, no. 22, article no. 9855, 2024. https://doi.org/10.3390/su16229855

[30] N. A. Dahri, W. M. Al-Rahmi, K. A. Alhashmi, and F. Bashir, “Enhancing mobile learning with AI-powered chatbots: Investigating ChatGPT’s impact on student engagement and academic performance,” International Journal of Interactive Mobile Technologies, vol. 19, no. 11, pp. 17–38, 2025. https://doi.org/10.3991/ijim.v19i11.54643

[31] K. Baba, N.-E. El Faddouli, and N. Cheimanoff, “Mobile-optimized AI-driven personalized learning: A case study at Mohammed VI Polytechnic University,” International Journal of Interactive Mobile Technologies, vol. 18, no. 4, pp. 81–96, 2024. https://doi.org/10.3991/ijim.v18i04.46547