Using a Random Controlled Trial to Explore the Impact of AI-Enabled Learning

Abstract

Much is being made of AI’s role in learning. However, there have been few studies that evaluate AI’s ability to make learning more efficient or more effective and specifically the learner’s willingness to embrace and use AI-enabled learning tools over conventional learning methodologies.

Working with a manufacturing company, we used a randomized control trial approach (matching pairs) to present two groups of employees with a course on design thinking, one presented via a traditional learning management system (LMS Group) and other through an AI enabled tool (AI Group), with the content being identical in order to compare completion rates, time to completion, learning outcomes (subject matter knowledge retained) and the learner’s perceptions of value and experience. We also were able to compare the AI enabled version of the course to the same version with the addition of a face-to-face instructional component.

While we found that those participants in the AI Group completed the course in a shorter period of time, had better learning outcomes and expressed higher perceptions of course value, the AI Group significantly underperformed the LMS Group in terms of course commencement and completion rates. Our finding suggest that the adoption of AI enabled learning tools may follow similar patterns of adoption rates of new technologies -- and require educating learners regarding the use, benefits, value and limitations of new AI enabled tools before widespread acceptance and usage.