Designing for Self-Regulated Learning in AI-Assisted Quizzing: A Classroom Pilot Study

Authors

  • Michael Pin-Chuan Lin Mount Saint Vincent University, Halifax, NS, Canada; Simon Fraser University, Burnaby, BC, Canada https://orcid.org/0000-0002-7646-7024
  • Daniel H. Chang Simon Fraser University, Burnaby, BC, Canada; Athabasca University, Athabasca, AB, Canada
  • Vasudevan Janarthanan Fairleigh Dickinson University, Vancouver, BC, Canada
  • Michael S. Hsiao Virginia Tech, Blacksburg, VA, USA https://orcid.org/0000-0003-1562-4409
  • Marco Ho British Columbia Institute of Technology, Burnaby, BC, Canada https://orcid.org/0000-0002-1120-475X
  • Jeeho Ryoo Fairleigh Dickinson University, Vancouver, BC, Canada https://orcid.org/0009-0003-0401-3685

DOI:

https://doi.org/10.3991/ijet.v21i03.61437

Keywords:

self-regulated learning, AI in education, learning analytics, computer science education, assessment

Abstract


AI support is increasingly embedded in online quizzes, yet instructors often lack clear, actionable signals about where students struggle during those assessments. We present a classroom-deployed quiz system that combines integrity-preserving hinting (TA-AI), trace summarization (Analytics-AI), and feedback-driven refinement (AI-Improver) to generate instructor diagnostics from routine interaction logs. The system was used in a graduate assembly programming course over five quiz weeks (N = 18). We report deployment evidence focused on reliability and instructional usefulness for monitoring: promptintent coding reached substantial agreement (Cohen’s kappa [κ] = 0.81); fixed-effects models (with student and item controls) showed a negative association for one-hint interactions (odds ratio [OR] = 0.231, indicating approximately 77% lower odds of a correct response for single-hint interactions relative to 0-hint interactions); and item-level demand spikes were operationalized via a demand × success prioritization process for weekly review. Rather than producing automated judgments or claims of causal learning gains, the analytics are designed as practical prioritization cues that direct instructor attention toward high-need items during AI-assisted quizzes.

References

[1] S. Lau and P. J. Guo, “From “ban it till we understand it” to “resistance is fu-tile”: How university programming instructors plan to adapt as more students use AI code generation and explanation tools such as chatgpt and github copilot,” in Proceedings of the 2023 ACM Conference on International Computing Education Research V.1. New York, NY, USA: ACM, 2023, pp. 106–121. [Online]. Availa-ble: https://doi.org/10.1145/3568813.3600138

[2] M. Kazemitabaar, L. Hou, A. Z. Henley, B. J. Ericson, P. Denny, and T. Gross-man, “Studying the effect of AI code generators on supporting novice learners in in-troductory programming,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. New York, NY, USA:

ACM, 2023, pp. 2877–2893. [Online]. Available: https://doi.org/10.1145/3544548.3580919

[3] R. Azevedo and D. Gasevic, “Analyzing multimodal multichannel data about self-regulated learning with advanced learning technologies: Issues and challenges,” Computers in Human Behavior, vol. 96, pp. 207–210, 2019. [Online]. Available: https://doi.org/10.1016/j.chb.2019.03.025

[4] M. Siadaty, D. Gasevic, and M. Hatala, “Trace-based micro-analytic measurement of self-regulated learning processes,” Journal of Learning Analytics, vol. 3, no. 1, pp. 183–214, 2016. [Online]. Available: https://doi.org/10.18608/jla.2016.31.11

[5] K. R. Koedinger and V. Aleven, “Exploring the assistance dilemma in experiments with cognitive tutors,” Educational Psychology Review, vol. 19, no. 3,

pp. 239–264, 2007.

[6] R. S. Newman, “Adaptive help seeking: a strategy of self-regulated learning,” in Self-Regulation of Learning and Performance: Issues and Educational Applica-tions, D. H. Schunk and B. J. Zimmerman, Eds. Hillsdale, NJ: Lawrence Erl-baum, 1994, pp. 283–301.

[7] B. J. Zimmerman, “Becoming a self-regulated learner: an overview,” Theory Into Practice, vol. 41, no. 2, pp. 64–70, 2002.

[8] J. Sweller, P. Ayres, and S. Kalyuga, Cognitive Load Theory. New York, NY: Springer, 2011.

[9] A. Collins, J. S. Brown, and S. E. Newman, “Cognitive apprenticeship: teaching the craft of reading, writing and mathematics,” in Knowing, Learning, and Instruc-tion: Essays in Honor of Robert Glaser, L. B. Resnick, Ed. Hillsdale, NJ: Lawrence Erlbaum, 1989, pp. 453–494.

[10] D. Wood, J. S. Bruner, and G. Ross, “The role of tutoring in problem solving,” Journal of Child Psychology and Psychiatry, vol. 17, no. 2, pp. 89–100, 1976.

[11] E. L. Deci, R. J. Vallerand, L. G. Pelletier, and R. M. Ryan, “Motivation and edu-cation: the self-determination perspective,” Educational Psychologist, vol. 26, no. 3-4, pp. 325–346, 1991.

[12] R. M. Ryan and E. L. Deci, “Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being,” American Psychologist, vol. 55, no. 1, pp. 68–78, 2000.

[13] C. P. Niemiec and R. M. Ryan, “Autonomy, competence, and relatedness in the classroom: applying self-determination theory to educational practice,” Theory and Research in Education, vol. 7, no. 2, pp. 133–144, 2009.

[14] J. R. Landis and G. G. Koch, "The measurement of observer agreement for categori-cal data," Biometrics, vol. 33, no. 1, pp. 159–174, 1977.

[15] M. L. McHugh, "Interrater reliability: the kappa statistic," Biochemia Medica, vol. 22, no. 3, pp. 276–282, 2012. [Online]. Available: https://doi.org/10.11613/BM.2012.031

[16] T. K. Koo and M. Y. Li, "A guideline of selecting and reporting intraclass corre-lation coefficients for reliability research," Journal of Chiropractic Medicine, vol. 15, no. 2, pp. 155–163, Jun. 2016.

[17] J. Ryoo, M. P.-C. Lin, S. Rai, W. He, S. M. Park, and M. Ho, “WIP: Multi-Agent Artificial Intelligence Model to Enhance Self-Regulated Learning and Con-ceptual Understanding in Computer Science Education,” in 2025 IEEE Frontiers in Education Conference (FIE), Nashville, TN, USA, 2025, pp. 1–5. [Online]. Avail-able: https://doi.org/10.1109/FIE63693.2025.11328198

[18] M. P.-C. Lin and D. Chang, “CHAT-ACTS: A pedagogical framework for person-alized chatbot to enhance active learning and self-regulated learning,” Comput. Educ. Artif. Intell., vol. 5, p. 100167, 2023. [Online]. Available: https://doi.org/10.1016/j.caeai.2023.100167

[19] D. H. Chang, M. P.-C. Lin, S. Hajian, and Q. Q. Wang, “Educational Design Principles of Using AI Chatbot That Supports Self-Regulated Learning in Educa-tion: Goal Setting, Feedback, and Personalization,” Sustainability, vol. 15, no. 17, p. 12921, 2023. [Online]. Available: https://doi.org/10.3390/su151712921

[20] S. M. Park, M. Ho, M. P. -C. Lin and J. Ryoo, "Evaluating the Impact of Assis-tive AI Tools on Learning Outcomes and Ethical Considerations in Programming Education," 2025 IEEE Global Engineering Education Conference (EDUCON), London, United Kingdom, 2025, pp. 1-10, doi: 10.1109/EDUCON62633.2025.11016517.

[21] K. Keahey, J. Anderson, Z. Zhen, P. Riteau, P. Ruth, D. Stanzione, M. Cevik, J. Colleran, H. S. Gunawi, C. Hammock, J. Mambretti, A. Barnes, F. Halbach, A. Rocha, and J. Stubbs, "Lessons learned from the Chameleon testbed," in Proceed-ings of the 2020 USENIX Annual Technical Conference (USENIX ATC '20), Jul. 2020.

Cover iJET – Vol 21, No 03 (2026)

Downloads

Published

2026-07-22

How to Cite

Lin, M. P.-C., Chang, D. H., Janarthanan, V., Hsiao, M. S., Ho, M., & Ryoo, J. (2026). Designing for Self-Regulated Learning in AI-Assisted Quizzing: A Classroom Pilot Study. International Journal of Emerging Technologies in Learning (iJET), 21(03), pp. 30–43. https://doi.org/10.3991/ijet.v21i03.61437

Issue

Vol. 21 No. 03 (2026)

Section

Papers

License

Copyright (c) 2026 Michael Pin-Chuan Lin, Daniel H. Chang, Vasudevan Janarthanan, Michael S. Hsiao, Marco Ho, Jeeho Ryoo

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.