openSAP: Learner Behavior and Activity in Self-Paced Enterprise MOOCs

Massive Open Online Courses (MOOCs) have been a subject of research since 2012, especially in the context of professional development and workplace learning due to their flexible schedule and format, which is a prerequisite for on the job learning. But MOOCs often do not fulfill the promise of flexible learning as it is only possible to achieve a certificate during the course runtime. An unpredictable workload and thus a lack of time often results in not showing up to a course or dropping out during the course runtime. Therefore, some platform content remains accessible even after the course runtime in self-paced mode. These courses differ from live courses as participants still can access all of the content and the discussion forum in read-only mode, but are not able to take the graded assignments and exams which are a prerequisite to achieving a certificate at the end of a course. Even though it is only possible by paying a fee to earn a graded certificate for these self-paced courses, we identified a high share of additional enrollments after the course end that suggests there is still interest from participants. Nevertheless, learning behavior in self-paced courses has not been a major subject of research, yet. This work contributes to closing this research gap by exploring the learner behavior in self-paced courses. The results show tendencies of more time-efficiency and engagement of self-paced learners under certain conditions and pave the way for further research and practical applications.


Introduction
Massive Open Online Courses (MOOCs) have been around for almost a decade. Over time, the original premises of democratizing education through this scalable and open e-learning format [1] changed towards a cost-effective measure for professional learning and development in and beyond organizations [2]. It was shown early on that the format mainly attracted professionals [3] and proved to be most effective when providing job or career-relevant knowledge and skills [4,5]. The format and its disadvantages, i.e., participants not showing up during the course runtime, dropouts resulting in low completion rates, deviating quality of the overall course content and the instructional design [6,7], as well as its advantages, i.e., providing the flexibility of consumption in addition to a social and interactive environment [8], have already been broadly discussed in recent publications and over the last years. This research, however, was mainly limited to different aspects tied to the runtime of a MOOC. In fact, the catalog of archived or self-paced courses has been continuously growing in the past few years, with ClassCentral, one of the biggest MOOC aggregator platforms, listing more than 13.5 thousand courses in 2019 [9]. A certain portion of this still relevant and stable knowledge remains available, yet untapped. As a result, new business models from major online course providers offering subscription-based course flat rates to their users, e.g., CourseraPlus, were recently developed [10].
The same applies to the context of MOOCs for professional development and workplace learning. Business MOOC platforms have to deal with the same phenomenon: a growing course catalog with to this day relevant knowledge and with professionals as a target audience. Professionals are interested in the content itself, even after a course has ended, as it is relevant to their job performance and therefore not necessarily interested in receiving a certificate. As this target audience has to deal with a tight business schedule and a high workload, which oftentimes conflicts with the weekly schedule of a MOOC, the consumption of content in self-paced mode becomes even more relevant. This is confirmed by a share of over 30% of enrollments in self-paced courses, at least on Enterprise MOOC platforms like openSAP [11]. Even though some argue that MOOCs offered in self-paced mode are less effective, as students will only consume parts of the content, but not complete a course as a whole [12], there is little known about the behavior and activity of students in selfpaced courses compared to those taking the course during its official runtime.
Due to the lack of existing research in this field and the relevance of the topic for professional learning, this paper will examine the following research question by analyzing courses from the Enterprise and Corporate MOOC platform openSAP, which is described in more detail in the section below: How does the learning behavior differ between students in MOOCs who participate during the course and afterward in self-paced mode? Therefore, the paper is structured as follows: Section 2 introduces the background and scope of the investigated MOOC platform before Section 3 describes the study design, data set and the methods used for the analysis. Afterward, Section 4 presents the study results. At last, Section 5 concludes the paper.

2
The openSAP University The openSAP learning platform (available at open.sap.com) was established in 2013, in the course of SAP's strategy to digitize parts of its education business and the increasing demand to provide partners, customers and prospects knowledge about new business innovations and product releases in a contemporary and timely manner [13]. The main delivery format is online courses based on the xMOOC principles, scalable to serve a larger audience, open to everyone, providing videos, quizzes, and tools for interaction in a digital classroom over a fixed period. The main topics covered in these courses are either technology and software, business, design or corporate social responsibility-related. The technical infrastructure is based on the HPI MOOC platform developed at the Hasso Plattner Institute in Potsdam, Germany. At the beginning of 2020, the award-winning platform counts more than 900 thousand registrations, with a share of about 85% having a professional background, from over 200 countries with more than four million enrollments in around 180 different courses. In 2019, openSAP Podcasts and Microlearning have been added as an additional format to complement the existing MOOCs on openSAP.

Enterprise and Corporate MOOCs
The term Enterprise and Corporate MOOC originated with the founder of openSAP, Clemens Link, in 2014 and describes the two main delivery formats. Corporate MOOCs are mostly limited to employees and are only available within an organization with custom-built in-house content, whereas Enterprise MOOCs are an extension of this concept, opening relevant corporate knowledge to external stakeholders, e.g., customers and partners [14].
Regarding their design, the courses follow the classical xMOOC structure. Every course has a fixed start and end date with a registration period of several weeks in advance. After a course has started, new content is released on a weekly basis, containing several video elements of approximately 15 minutes in length. Each video segment is followed by a short ungraded self-test with multiple choice and multiple answer questions to reflect on the content. In some cases, this content is complemented by additional hands-on exercises. Moreover, each course has a dedicated discussion forum available, to allow participants to interact with their fellow students and to ask the content experts, who are also active in the forum during the course runtime, questions. A set of collaborative tools is provided in so-called Collab Spaces, which allow dedicated breakout sessions or collaboration in smaller groups. At the end of each week, a graded assignment containing ten questions about the respective content enables participants to collect points for a certificate of achievement to document their learning performance. The average learning effort per week is four to six hours. A course concludes with a final exam containing additional questions, covering all the course contents and thereby counting 50% of the highest attainable score. Participants can finish the course with two kinds of certificates. They will receive a so-called Confirmation of Participation (CoP) by accessing at least 50% of the overall course content. In addition, the participants will obtain a Record of Achievement (RoA) by collecting at least 50% of the overall points available in the weekly assignments and the final exam.
After a respective course has ended, it switches to self-paced mode. The course content remains available; however, graded assignments are not accessible anymore and the forum is closed. This means that learners are still able to read threads in the forum, but the forum is not moderated anymore, and it is therefore not possible to start a new discussion. Most courses offer the possibility to reactivate graded quizzes for eight weeks by redeeming an openSAP Reactivation Code Option to receive a Record of Achievement even after the official runtime of the course has ended.

Usage Statistics for Self-Paced Courses
In early 2020, the overall catalog on openSAP listed more than 180 courses, which are all available in self-paced mode. In the first quarter of 2020, the share of new enrollments for self-paced courses on the platform was 38.6%. This indicates that even though courses ended and switched to self-paced mode, the content remains relevant for a notable number of users. Therefore, we have selected six representative courses from the openSAP platform, which differ in their scope of content, to allow a more detailed analysis. The study design, i.e., the evaluated courses, the utilized platform data, and the methodology used to analyze the data is described in the next section.

Study Design
We conducted a case study to investigate the presented research question in an authentic real-world learning environment. Therefore, we observed and analyzed the learning behavior of students, which highly depends on their goals and intention [15]. To reduce the intention-behavior bias we assigned students based on their learning outcome to different cohorts as this is the only indication about their motivation for enrollment (if it was not captured through self-reported questionnaires or technically supported features like personalized learning objectives [16]). Our assumption here is that students who achieved the same certificate had a similar motivation to work on the course content, whether for private or professional purposes. Certainly, this is a limitation of the study design and excludes students who had a similar initial motivation but then did not achieve their learning objective.
We selected six courses for the analysis: two 2-weeks nutshell courses, two 4weeks courses, and two 6-weeks courses to represent and include the different course formats on openSAP. All courses were held in 2017 or 2018 and were therefore already in the self-paced mode for 2-3 years at the time of the analysis which ensured the availability of sufficient data. They were free of charge during their runtime including both certificates and all material has been freely available since then. A charge is only required for the reactivation of the Record of Achievement after the courses had ended. We split the students into two mutually exclusive categories: (1) students who received a Confirmation of Participation, and (2) students who received a Record of Achievement. For both, we assigned the students into cohorts depending on whether they gained the corresponding certificate during the course or afterward in self-paced mode. We then examined the two cohorts in both categories for statistically and practically significant differences in their learning behavior based on various metrics.

Sample Courses
The six courses "SAP Cloud Platform Essentials" (cp1-2), "Getting Started with Data Science" (ds1), "Find Your Path to SAP S/4HANA" (s4h5), "Introduction to SAP HANA Administration" (hsha1), "SAP Solution Manager for SAP S/4HANA Implementation in a Nutshell" (solman1), and "Understanding SAP Fiori Launchpad" (fiops1) were evaluated. All courses were held in English on openSAP and targeted technical and business professionals. The two 2-weeks courses were graded by a single final exam. The two 4-weeks courses and the two 6-weeks courses were graded with weekly assignments and a final exam. Between 15,076 and 24,228 users enrolled for the courses until their end with a total number of 121,146 enrollments. These numbers increased to 25,970 up until 38,404 enrollments per course in self-paced mode, resulting in a total number of 206,387 enrollments. The data snapshots for this study were taken on February 14, 2020. More detailed numbers can be seen in Table  1.

Data and Analysis
To access the learning behavior metrics and more properties to assign each enrollment of the courses into the cohorts we utilized the platform's course reports which additionally pseudonymize the data beforehand. Afterward, we analyzed the exported data with external tools. The two categories with two cohorts each were determined as follows: (1.a) students who received a Confirmation of Participation but not a Record of Achievement with an enrollment date before the end of course, (1.b) students who received a Confirmation of Participation but did not book a reactivation with an enrollment date after the end of course, (2.a) students who received a Record of Achievement with an enrollment date before the end of course, and (2.b) students who booked a reactivation and received a Record of Achievement with an enrollment date after the end of course. The last cohort is much smaller compared to the others since the reactivation of a Record of Achievement includes a charge. The specific numbers of each cohort can be seen in Table 2. To assess statistically significant differences, we used the nonparametric Mann-Whitney U test for two independent samples. Additionally, we calculated Cohen's d to measure the effect sizes. The practically significant differences were evaluated on the basis of the descriptive statistics, based on the authors' long-term experience with the operation of several MOOC platforms and courses.

Results and Discussion
The comparison of the learning behavior of guided and self-paced learners who received a Confirmation of Participation can be seen in Table 3 (Appendix). A highly statistically significant difference in visited items can be found in four of six courses with small effect sizes in two courses. However, the mean values show no practical relevance which contradicts the initial premises of fewer item discovery from participants in self-paced courses. Nevertheless, the overall item discovery confirms a lack of consumption, either selective or linear incomplete. This has to be the subject of further research. The average and total session durations have highly statistically significant differences in almost all courses with small to large effect sizes. Self-paced learners have longer sessions on average compared to guided learners, but a shorter total learning time. This indicates that self-paced students who received a CoP learned more time-efficiently and with fewer interruptions. As all of the course content is already available in self-paced mode, there is no need to wait for new content to be released on a weekly basis as during the course runtime, which makes fewer and therefore more extended learning sessions possible. A plausible reason for the longer total session durations during the course runtime is the availability of additional tools, e.g., interactions in the course forum and the collab spaces. Also, learning on a weekly basis requires a regular recap of the past content which takes some additional time. The self-test quiz performances reveal statistically significant differences in all courses with small effect sizes in four courses. Guided learners perform slightly better, however, we see no practical relevance here. We have the assumption that learners are more engaged by focusing on the weekly content available during the runtime rather than rushing through the course in self-paced mode and thus performing better in quizzes, and spending more time in total on the platform which is confirmed by the total session duration. The played videos, downloaded videos, downloaded slides, and passive forum activity metrics do not show any clear practical relevant differences, even if a lot of statistical significance and effect sizes are found, reasonably because such large sample sizes lead quickly to significant results and affect the probability of Type II errors.
The comparison of the learning behavior of guided and self-paced reactivated learners who received a Record of Achievement can be seen in Table 4 (Appendix). On average, self-paced learners visit more items than guided learners with statistically significant differences in three courses with small effect sizes in two of them, however, without any practical relevance. A possible reason for that is paying users are more thoroughly engaging with the content compared to some participants during the free of charge runtime, rushing only through the course elements that are necessary to receive a certificate at the end. Self-paced students also have much higher average session durations with highly statistical differences in all six courses with high effect sizes. This confirms our results from the CoP cohorts comparison and may lead to the same interpretation. The availability of content enables longer sessions, browsing through more content at once. They also stay much longer on the platform in total (on average) and we consider both metrics as practical relevant, even if the total session duration has only highly statistical differences in two courses with small effect sizes. Interestingly, this metric contradicts our findings from the CoP comparison where the total session duration of self-paced learners decreased. This can also be attributed to the perceived higher value of the for-fee RoA which leads to an increase in learning time. Self-paced students also perform slightly better in quizzes on average with statistically significant differences in three courses and small effect sizes in two of them. However, the differences are not of practical relevance. Regarding played videos, we discovered practical significant differences in five of the six courses, whereby self-paced students watched more video lectures, but only two courses showed statistically significant differences. In all courses, self-paced learners downloaded notably more videos with statistically significant differences in three courses. They also downloaded more slides than guided students in three courses with statistically significant differences in two of them, and about the same amount in the other three courses. These three categories support the premise that learners who reactivated courses for a fee show higher rates of engagement for all aspects of the content provided. Additionally, guided students passively used the forum more often than self-paced students in all courses, but only two courses showed statistically significant differences. This is probably caused by the fact that the course forum attracts more attention during course runtime due to a growing number of discussions.
All in all, we consider the following findings to be most interesting: Self-paced learners with a CoP have longer sessions on average compared to guided learners with a CoP, but a shorter total learning time, which indicates that they learned more timeefficiently and with fewer interruptions. Self-paced learners with an RoA tend to be more engaged in general, regarding the total and average session duration. They also perform slightly better in quizzes, and played and downloaded more videos and slides. This higher engagement is probably related to the fee charged for reactivating a course to obtain a RoA. The only exception here is the passive forum use, which is higher for guided students with a RoA.

Conclusion
This paper conducted a case study to examine the research question of how the learning behaviors differ between students in MOOCs who participate during the course and afterward in self-paced mode. Therefore, six courses from the Enterprise and Corporate MOOC platform openSAP were analyzed with currently more than 200 thousand enrollments and assessed regarding several learning behavior metrics. The students were split into different cohorts based on their course outcome, if they gained a Confirmation of Participation or a Record of Achievement. In general, self-paced learners tend to learn more time-efficiently and perform slightly better in quizzes. When a fee was charged for a graded certificate in self-paced mode, the overall time spend and platform engagement increases. Nevertheless, the practical impact needs to be studied in further research. One of the future topics to be examined will be item discovery to better serve this target audience learning in self-paced courses. Furthermore, we will also consider additional metrics and characteristics, e.g., differences in the instructional design, the structure of the content and domainspecific differences. Thereby we aim to by better understanding our user's needs be able to improve the overall learning experience on business MOOC platforms.