openSAP: Evaluating xMOOC Usage and Challenges for Scalable and Open Enterprise Education

—The openSAP University is a co-innovative initiative founded in 2013 by SAP SE in partnership with the Hasso-Plattner-Institute (HPI) located in Potsdam, Germany. With its new course offering, SAP responds to a rising demand for scalable knowledge transfer due to the digital transformation by making use of the Massive Open Online Courses (MOOC) format. This paper provides a brief introduction to the first Enterprise MOOC platform openSAP, including details about the learning environment and the underlying platform, different team roles and usage statistics. In addition, the topic of dropouts in enterprise MOOCs will be addressed and discussed. A standardized calculation model for enterprise MOOCs to measure completion and consumption rates is proposed. The paper closes with an outlook about the future work on enterprise MOOCs.


INTRODUCTION
The New York Times claimed 2012 to be the "The Year of the MOOC" [1]. Only a few months later a certain amount of frustration raised. MOOC-pioneer Thrun seemed to be quite pessimistic: "We were on the front pages of newspapers and magazines, and at the same time, I was realizing, we don't educate people as others wished, or as I wished. We have a lousy product." [2]. While some players on the market failed to reach their self-put targets, the enterprise sector discovered the MOOC as a possibility to implement efficient and scalable knowledge transfer. This paper at hand will discuss the MOOC offer of the software company SAP SE, called openSAP.
Some of the critics on the first MOOCs stressed out that MOOCs failed to reach a diverse audience. Instead the typical MOOC user was located in the USA or Europe and used MOOCs in the context of lifelong learning [3]. So if users used MOOCs to learn in this context anyway, wouldn't it be a natural fit if companies used MOOCs?
SAP is a fast changing company. Technology changes like the introduction of new products like SAP HANA (a In Memory based Database) require not only employees of SAP itself, but also developers, users and other stakeholders around the world to become familiar with new concepts and products within short timeframes.
While a vibrant course community (and therefore a social learning experience) can be achieved with small enrollment numbers, the predominant motivation to increase course sizes, is to increase cost efficiency. Given the fact that MOOCs are good at scaling, companies can reach a large amount of users while the investment nearly peaks at a certain threshold [4]. It doesn't matter if a course is offered to 10.000 or 20.000 users, as it will result in a similar financial effort for the course provider (as long as now additional resources like mentoring or virtual labs are provided as part of the course). This means that the cost per learner decreases with the increasing size of the course.
MOOCs openness can be understood in different ways. Either the free access to the courses itself. Secondly the openness of the learning material. While in academic MOOCs the content itself is often published as Open Educational Resources (OER), in Enterprise MOOCs the content right will often remain at the course authors.

II. XMOOC USAGE FOR ENTERPRISES
xMOOCs follow a well-defined structure. openSAP is following the xMOOC paradigm very closely. While in some courses the learners can select from alternative sections, most courses offer a very linear learning path and a course journey consisting of mandatory content only.
The duration of the courses is typically in between four to six weeks. Every week new content is published and can be accessed by the enrolled course participants. The content of the single weeks follows the concept of Mastery Learning as defined by Bloom. This learning cycle consists of an iteration of videos, and self-tests, accompanied by a forum and collaboration spaces (see figure 1). Each week a homework assessment is offered. At the end of the course a final assessment is offered. While ungraded self-tests provide an instant feedback about the submission, homework assessment results are not displayed until the submission deadline for the assessment is reached. This should avoid discussions about assessment details before all learners submitted. Furthermore, homework assessment usually only allows one submission per user which must be completed in a given timeframe (60 minutes).
Some courses offer peer assessments, which allow a more creative type of user contributions like coding mobile apps. While every user that accesses more than 50% of the learning material receives a Confirmation of Participation (CoP), only users that achieve more than 50% of the available course points receive a Record of Achievement (RoA).
Based on a report from Towards Maturity 8% of companies are using MOOCs in an enterprise context, while another 7% are planning to do so in the future [5]. The motivation for using or providing MOOCs can be based on one or more of the following items: A checklist guide for companies that consider using MOOCs in an enterprise context was developed by Sillak [6] as part of her thesis and can be found online.
III. OPENSAP UNIVERSITY openSAP (available at open.sap.com) claims to be the first dedicated Enterprise MOOC platform [7]. Since 2013 the openSAP University offers free to access courses about product and innovation topics, addressing partners, customers and other stakeholders.
Hasso Plattner, chairman of SAP held his first academic MOOC in 2012 on openHPI (available at open.hpi.de), the MOOC platform of the Hasso Plattner Institute (HPI). This specific MOOC dealt with the academic basics of In Memory Databases, the underlying technology powering SAP HANA. Over 15.000 students enrolled. Inspired by the success of this MOOC and the positive user feedback, he triggered a collaboration with the HPI to build up a dedicated enterprise MOOC offer to empower the fast and agile distribution of product innovations.
In May 2013 the first course "Introduction to Software Development on SAP HANA" started on openSAP and over 40.000 learners enrolled [8].
While the HPI is responsible for the MOOC platform and the accompanying academic research, SAP is in charge of course production, course management and support during the courses. Since the launch of openSAP, the team has grown from five initial members to more than 25 members. As openSAP resides within SAP SE, an agile and rapid content production and knowledge transfer can be achieved. That´s because respective content experts can align closely with project managers as well as the internal video production team and other required resources.

A. Mission and Objectives
The primary objective of openSAP is to enlarge the SAP ecosystem. The target audience of the course offerings are in particular stakeholders who want to update their knowledge or improve their practical skills, and students with a personal interest in IT and business innovation topics. Through its scalable, time flexible and free of charge learning solution, SAP-specific knowledge can be transferred to a broader audience, independent of their geographic region, social background or financial status. This enables innovation adoption at scale, as new SAP IT and business innovations can be shared widely within a short period of time at low costs per contact. openSAP also serves as an entry point for additional SAP Education offerings by only providing basic content and referring to these offerings for more in-depth learning materials [9].
Besides the xMOOC driven company-to-user knowledge transfer, the used demo systems, implemented feedback features and learning analytics create a user-tocompany feedback channel. These data can be used to provide product owners direct user feedback or to derive product enhancements for future releases. Furthermore, the implemented collaboration features lead to an increasing interaction among users and thereby fosters knowledge exchange of a large user base with various different backgrounds and prior knowledge which can boost creative and innovative problem solving within a social group. First successes could be achieved by making use of Peer Assessments complementing the traditional xMOOC format [10].
With openSAP, SAP is targeting a new emerging market which fulfills the rising customer needs for scalable and instructed knowledge transfer by virtue of shortened product lifecycles, progressive digitization and various other innovation topics in a fast changing environment. Partnerships with universities focused on instructional design or computer science for empirical research strengthens SAP´s position to shape the future of this uprising market segment.

B. Learning Environment
openSAP runs on a proprietary software provided by the HPI, internally called the Xikolo Learning Management System, its UI is shown in figure 2. While this software is also used for other MOOC platforms like open.hpi.de, some specific functionalities have been introduced exclusively for openSAP. This includes a Single-Sign-On option, so users can sign in easily with one click if they are already owner of an SAP account. The User Interface is also adapted to match SAP Corporate design.
Technically the software consists of several microservices that are used in conjunction with an API (that is used for the native apps) and a responsive web frontend. Nearly all services are implemented using the Ruby On Rails framework. The technical concept for this new service oriented architecture is outlined in the referenced paper [11]. The learning environment is fully web based and requires no installation of plugins like Flash or Java. A responsive design allows the simple usage from mobile devices. Furthermore, native apps are available for Android tablets, smartphones and the iPad. As around 30% of the users are downloading content and learn offline (and therefore outside of the learning environment) the apps are an offer to provide offline learning while still working within the given course structure.

C. Internal Organisation of openSAP
The openSAP University is a subordinate of the Scale, Enablement and Transformation business unit within the Service and Support business area of the SAP SE. The different team roles are illustrated in figure 3 and described below. Michaela Lämmler serves as the dean of openSAP and is thereby in charge of the team management. Clemens Link is responsible for reporting and course portfolio management. Seven different team units are subordinated by the leadership team.
The Helpdesk employees are distributed across Germany, the US and the Philippines to cover most global time zones. This is crucial to provide contemporary user support with regards to course specific or technical issues. The Communications unit is responsible for global and course-specific user communication, administration of social media channels and public relation activities targeting other stakeholders. A team of instructional designers enable steady improvement of the media, content and platform features according to businessspecific requirements. The Operations unit is in charge of the platform administration and maintenance. This includes the publishing process of delivered courses, content management and quality assurance. Project managers are responsible for the end-to-end production process of an openSAP course together with the respective content owners and experts. The area of Graphics and Design develops design improvements for the openSAP User Interface and is responsible for content branding of used materials. The Infrastructure unit serves as an interface to the technical team of the HPI hosting the Xikolo Learning Management System. As openSAP is one entity of this system, both teams jointly develop further technical platform improvements and thereby provide operational support.
In addition, external services are required to carry out a set of processes more efficiently or with higher quality standards. Examples for these kinds of services are translation and quality check, video production and video hosting. These services are provided either by different departments within SAP or by external service providers.

D. Course Production
The openSAP course production cycle is depicted in figure  4. The production is triggered by a course request from another business unit (External Source) within SAP. These requests are included into the portfolio planning with an evaluation of the strategic fit and a rough estimation of required resources. In addition, a responsible project manager is assigned to the course and gets in contact with the respective content experts. During the initial kick off meeting, a project timetable, and a first structure of the included contents are created. Subsequently, the detailed course content per week and thus the course design is defined. After that, the content production takes place. This includes: • recording and post-production of the course units by the SAP video team in coordination with the respective content expert speakers.
• production of lecture slides by the content experts including a copy-edit check, a graphics check and optional translation of the material into different languages. • development of a question pool for self-tests, weekly assignments and the final exam in coordination with the openSAP instructional design expert. • creation of course announcements for the openSAP platform by the internal communications team. • Building of the course and its contents on the platform by the operations team. In a timeframe of four to six weeks prior to the start, the course is announced officially to all openSAP users and is thenceforth visible on the platform, ready for enrollment. During the course runtime, the respective project manager and the content experts manage the forum to answer user questions and to ensure a smooth conduct of the assignments. At the end of a course, the user feedback in the forum as well as other KPIs are evaluated and summarized. If there is valuable feedback for the respective content owners, this will be handed over by the respective project manager. Today, there a three ways of further utilization of a course. Firstly, a course can be repeated in the future with its same content (repeat). Secondly, the course can be repeat with up-to-date knowledge, replacing obsolete content (Update). Thirdly, advanced courses to the same topic can be produced (Next Steps). For the second and third option, a re-run of the whole production cycle is unavoidable.  Learners from more than 180 countries enrolled to courses on openSAP. Only a small amount of this users are SAP internal users (approx. 15%). Other user groups include external developers, partners, consultants and students. More than 50% of all users are located in the USA, Germany and India. A dedicated branch of the platform (open.sap.cn) is offered in china to provide good internet connectivity and a chinese speaking course community to users located behind the "chinese firewall".

V. ADRESSING DROP OUTS
One major drawback which is always addressed when it comes to the evaluation of the MOOC format is the insufficient completion rate and thus the high dropout rate of most courses [12]. As broadly discussed in the academic context, these rates may vary highly with an average of 15% [13]. The completion rate is generally defined as the total number of participants who achieve a kind of certificate in the end or simply pass the course [13]. But the calculation of this rate is far more complex.
To approach drop outs in an enterprise context, it has to be understood beforehand what kind of student patterns are emerging in MOOCs in general. Hill proposed five different patterns of MOOC students which are illustrated in figure 5 and described in more detail below [14].
The group of No-Shows includes all students who initially enroll for a course, but then never show up. This category of users corresponds to approximately 50% of the total course enrollments [14].
Lurker, also called observers, are described as a group of participants who enroll for a course, but then consume less than 50% of all learning materials without taking any assignments [14]. Observers only consume content considered to be relevant for their individual needs.
Students labeled as active participants are enrolled for a course to access most or all of the available learning materials and pass the assignments and the final exam to earn a certificate. In addition, these kinds of users like to get involved in the discussion forum, create, share and evaluate content and thus are interested to use other collaboration features as well.
Passive participants also access most or all of the available learning materials and pass the assignments and the final exam to earn a certificate. However, this group neither interacts with other students during the run-time of a course nor do they make use of other collaborative features, like learning groups or the discussion forum.
Other participants enroll for the course not until its start but to some other point during its run-time. These are called drop-ins. This group reflects the continuous growth of enrollment numbers during a course. New participants in turn join one of the four groups described above. Due to the scheduled weekly assignments, with ongoing progress of the course it gets increasingly difficult to collect the minimum points required for the issue of a final certificate.
Based on the differentiated consideration of the different user patterns, MOOC drop outs can be defined as participants with the initial intention to complete a course, but then change their attitude by virtue of internal or external stimuli over time and drop out early. Thus drop outs can only be found in the clusters of observers, active or passive course participants. Moreover, only drop-ins who join the course at a time it is still possible to achieve a final certificate should be included in further calculations. In the academic context the widespread model to calculate completion rates is the number of certificates issued divided by the number of total enrollments to course start. Other MOOC platforms subtract the number of No-Shows from their total enrollments before performing the calculations [15].
The completion rates for enterprise MOOCs delivered by the openSAP University reach from 15% to 30% depending on the course format and number of past repetitions on the platform. The applied calculation model comprises of the number of issued RoAs divided by the total number of enrollments half-way of the course. This chosen population is based on the assumption that late drop-ins only have a fair chance to earn a RoA till mid of the course term.

VI. RETHINKING PERFORMANCE MEASUREMENT & LEARNING OBJECTIVES
The currently applied definition of completion rates and learner success discloses two major weaknesses. Firstly, with regards to the calculation models and the derived key performance indicators (KPI) for companies. The respective models may differ based on the user groups taken into account at different points in time. If No-Shows are excluded from further calculation, completion rates are likely to be twice as high, but at the same time decreasing the total population. In addition, also drop-ins are increasing the total course population over time which then in turn may affect the calculation results. Enterprises have to decide which KPI is more important for internal evaluation purposes.
Furthermore, as two main objectives for MOOCs in an enterprise context are to enlarge the ecosystem and to enable innovation adoption at scale, not only completion but also consumption of the shared knowledge should be taken into account as another important performance measure. Since only learners who accessed more than 50% of the course contents receive a Confirmation of Participation, this additional KPI enables companies to measure the partial consumption of provided content. This is important, because in some cases, participants may only browse for the knowledge they need, and this may not include all of the course contents, but is nevertheless estimated to be a relevant measure for successful knowledge transfer. Due to these arguments, a standardized calculation model for enterprise MOOCs to measure completion and consumption rates is proposed by the authors: Lurker cannot be included in the calculation model above right now, which leads to the second weakness of the current definition. MOOCs have been developed with a single course objective -completion. But the group of observers indicates that course participants have diverged subjective learning objectives, which can be contradictory to the single goal of completing a course [16]. If participants only grasp a small part of the course materials, they are nevertheless in a position to reach their subjective learning objective. This issue has already been addressed by Liyanagunawardena, Parslow und Williams [12] proposing another definition of drop outs: Drop outs are participants, who could not achieve their self-set learning objective in a course and by virtue drop out early. Although it is already possible to track the users initial learning objective with a survey at the beginning of a course, the actual behavior cannot be tracked sufficiently by the system right now. Advanced learning analytics are needed to gain a better understanding of the different user groups and to measure performance of the learning format more precisely. This action would also result in a refinement of the two formulas given above, e.g. including observers in these calculations. A first step to track the subjective learning objectives of course participants will be rolled out at openSAP in the near future. Therefore, a feature will be implemented to offer users the chance to set a flag within a course, if they have reached their own learning objective, even if they did not complete the course by accessing all of the course contents or taking all of the assignments.

VII. CONCLUSION & FUTURE WORK
In case of enterprises with a certain size MOOCs are an excellent solution, to provide scalable digital transfer of knowledge. The xMOOC learning format with its small chunks of learning material and its unique mixture of freedom (in terms of where and how to learn) and its given structure (course start and end, deadline and sequential content publishing) seems to fit the needs of learners in the context of lifelong learning. However, while there is an high rate of satisfaction, several topics may be worth further investigation and research to optimize both the learning experience and the learning outcome of users in enterprise MOOCs.
Decrease Drop Outs Real drop outs (see chapter VI) could be addressed in different ways. Assuming that the drop out was based on external factors (high workload in the day job, private situation, sickness) and not based on non fitting content or a loss of the intrinsic motivation the platform should be able to lower the amount of restrictions and allow the finishing of the course based on new individual personal deadlines.
However, this should be combined with some kind of effort for the learner to take care that this function would not be misused by the learners. One possibility could be to make this function a payment function.
Decrease No-Shows Beside lowering the drop-out rate, no shows are a critical phenomenon. If someone enrolls for a course but never shows up it can be assumed that there is a basic interest for the given course topic but the user was not willing to invest a certain amount of his time to at least consume some of the course material. At first it should be taken care that the user is aware of the course start. Especially if the users enrolled a long time before the course start regular email communication should take care and make the learner aware of the upcoming course start. Beyond that, several actions could be considered and tested: A simple action could be to send out catch up mails after course start ("Hey <username>, we miss you"). Another option could be to introduce soft deadlines in case the user didn't started because he was afraid he wouldn't be able to complete the course during its runtime. Finally, a highly condensed version of the course ("ToGo") could be an option and delivered in a manner that would be consumable while commuting or traveling.

Determine factors for course completion in enterprise
MOOCs In 2015 an empirical study has been conducted at openSAP in scope of a master thesis. Using an entry and an exit survey, several characteristic values of enterprise MOOC participants could be acquired (>8000 participants for the entry survey, >700 for the exit survey). These values include the diverse learning objectives, user demographics, prior experiences with the MOOC format, motivation aspects, characteristics of the instructional design, the institutional framework and additional individual characteristics. In further studies these results will be supplemented by additional process information, like the number of accessed course items and assignment scores. The aim of this extended research is to determine significant factors for a successful course completion to derive recommendations for an improvement of the current learning environment and the instructional design.
Better Cross Device Support While there is support for mobile learning in terms of an available Android app and the given responsiveness of the website, a more intense support for seamless learning could be helpful. Especially for those users that change devices and places for learning more often, e.g. who learn at work with a computer and at home with a mobile device. Currently two ongoing master theses focus on this topic. While one of the theses will focus on mobile analytics, handover features and multi device usage scenarios, the other thesis focus on the usage of Smart TVs as part of an multi device learning experience.