Paper—Using Learning Analytics to Improve MOOC Instructional Design Using Learning Analytics to Improve MOOC Instructional Design

Massive Open Online Courses (MOOC) allows teaching and learning for everyone. This means that people from any learning background can join any of the courses offered through MOOC platforms. Although learning materials are offered for free, learning retention and learning engagement were found to be consistently low although some MOOC are offered by wellknown instructors. Many recent studies tried to understand the suitable instructional design in MOOC to improve learning engagement and retention. This study is an exploratory study to evaluate the potential of using learning analytics to improve instructional design in MOOC. Data were collected from a MOOC offered for two consequent years in a public university in Malaysia. The impact of learning analytics on MOOC instructional design was also discussed. Keywords—MOOC, learning analytics, learning engagement, instructional de-


Introduction
Students drop out rate in MOOC is often discussed in many studies yet the real solution to this problem remains scarce.High dropout rate among MOOC students is partly due to MOOC availability as a free and flexible course that allows students to enroll and drop the course at their own convenience.Among the reasons that the students drop out in MOOCs are, no real intention to complete the course, lack of time, course difficulty and lack of support, lack of digital skills and learning skills, bad experiences in previous MOOC participation, low expectation towards what they have to do in MOOC, starting late and the availability of peer review that is less favored by the students in MOOC (Onah, Sinclair & Boyatt, 2014).
To address this issue many is concern about MOOC instructional design that can accommodate students' diversity yet allow personalized learning.Guàrdia, Maina and Sangrà (2013) identified three perspectives to study instructional design in MOOC which is by reverse engineering approach of addressing MOOC as artefacts, an evolutionary approach linking the cumulated body of knowledge about online course design to MOOC or by looking through a learner's perspective reporting -from the field‖.In this study we attempted to study instructional design for MOOC using learn-ing analytics hence looking at MOOC as artefacts and collecting data from field serve as an important foundation of this study on using learning analytics.

2
Background of Problem

Learning retention and learning engagement in MOOC
Many studies reported the increasing dropout rate in MOOCs particularly towards the end of the course (Parr, 2013;Jordan, 2013).In Jordan (2013), only 6.5% students were found to complete the course and that the completion rate is across time, university rank, and total enrolment, but negatively correlated with course length.Similarly, Kloft, Stiehler, Zheng and Pinkwart (2014) found that the dropout probability is particularly high in the first two weeks on MOOC, and similar trend was observed at the end of the course starting around week 11 and 12. Exploring this issue in depth provides a better ground for Malaysia MOOC to move forward and to have a clear expectation about learning in MOOC.For example, Onah, Sinclair and Boyatt (2013) found that some ‗dropout' students simply did not complete the task on time because they would like to advance in the course at their own pace.However, Stracke (2017) highlights the importance of measuring the completion of individual goals and intentions by the MOOC learner as MOOC quality indicator rather than measuring drop-out rate.Based on learner's engagement pattern while learning in MOOC, Khalil and Ebner (2017) found that extrinsic factors is insufficient to make students' to be continuously engaged in the course so that learners can complete their learning goals.By ‗Gaming the System', students were able to be highly committed to the learning tasks although they did not necessarily read and watch all the learning materials.The ‗Perfect Students' who completed all the exercises and watch all the lessons in this study were students who were satisfied extrinsically and intrinsically (Khalil & Ebner, 2017).Additionally, students in MOOC are sometimes active video viewers, passive video viewers, display active interaction in forum activity and are passive towards forum activity (Sinha, Li, Jermann & Dillenbourg, 2014).
This learning pattern calls for the need for studying students' learning analytics in MOOC to assist instructional design in MOOC for improved MOOC quality.This investigation suggests redesigning MOOC is necessary for better presentation that can accommodate students' learning goals and learning intentions.Hence, finding the appropriate instructional design in MOOC is one of the ways how educators can improve students' learning retention and engagement.
Instructional Design for MOOC: There have been many on-going debates on the instructional design in MOOC that could support the diversity of massive learners in a course at one time.A study by Margaryan, Bianco, and Littlejohn (2015) on 76 random MOOCs found that MOOCs might have good organization and presentation of materials but lack the quality of instructional design.These courses lack the implementation of problem-centered activities, collaborative learning, did not engages students to co-construct knowledge, lack of support to students' learning needs, lack of iJET -Vol.14, No. 24, 2019 feedback to learners and some courses lack of authentic MOOC learning resources (Margaryan et al., 2015).
Before deciding on the kind of instructional design suitable for MOOC, one should understand the course audience.This is explained in detailed in many instructional design models such as Dick and Carey model (Dick, Carey and Carey, 2014) as well as ADDIE model.Learners in MOOC came from diverse learning background including culture, gender, age, and they have different motivation for learning (DeBoer, Stump, Seaton, and Breslow, 2013), yet the only common character that they have is the interest towards the course.This explains the need for MOOC instructors to study their massive learners' characteristics so that they can improve and choose the suitable instructional design that accommodate students learning needs and tackle students' diversity (Chatti et al., 2016).Until now studies on examining and determining the best pedagogical approaches that MOOCs should be based on remain very scarce.

Learning analytics parameters in MOOC
Given the abundance of data that can be collected in MOOC, learning analytics serves as a useful guide to identify instructional design parameters in MOOC that can improve students' learning experiences.In many studies related to MOOC, parameters such as number of views, time spent on viewing materials or number of comments, are used to measure students' retention and engagement in MOOC (Jiang, Williams, Schenke, Warschauer and O'dowd, 2014).These parameters are also used to predict students' completion rate at the end of the course.However, little is known on how these parameters can be useful to assist instructional designers and instructors on designing and redesigning their MOOC instructional design.This way, the instructional design would be better fit for their learners because this design is based on learners' behavior during learning in the course which Guàrdia et al., (2013) termed as researching from learner's perspective reporting ‗from the field'.
Students' interaction with course material can be measured by collecting analytics on number of views.In a study by Murray, Pérez, Geist, and Hedrick (2012), students' number of viewing online learning materials in a course were collected and they found a strong relationship between viewing course materials and study success.In fact, the more resources a student interacts with, the greater chance they have of achieving a higher level of success in the course (Murray et al., 2012).
Additionally, identifying the highest average time spent on a learning material among the learners in the course is an important indicator of the importance of the learning material to the learners.Wong (2013) stated that students who spend more time on a specific material give us idea that the learning material is highly useful.
A good online course should also allow for students-students interaction and student-teacher interaction.Hence, features such as online discussion board or chatting are becoming compulsory in any online course.Learning analytics on number of comments can inform online course instructors on a general idea about students' interaction in the course.de Lange, Suardy and Mavondo (2003) found that online discussion features significantly influence students' satisfaction towards the course.
MOOC is like many other online courses but diversity of the students in course left very small room for personalized learning.Using learning analytics, instructors can learn more about their MOOC students hence an instructional design to response to their needs can be developed.Hence, this study would like to answer the following research question:  How learning analytics can be used to improve instructional design in MOOC?

Research Methodology
This study is an exploratory study to evaluate the potential of using learning analytics to improve instructional design in MOOC.In this way, this research is proposing understanding the instructional design from learner's perspective reporting ‗from the field' (Guàrdia et al., 2013).

Sampling
Using convenient sampling method, we chose two MOOCs from a learning institution as samples namely MOOC A and MOOC B. Gašević, Dawson, Rogers, and Gasevic (2016) highlighted about the one size fit all effects of instructional design in MOOC.In using learning analytics to understand students' learning, there such should be precaution in interpreting the results where consideration about the coursespecific features and instructions should be taken account (Gašević et al., 2016).Hence, this study only focused on two specific MOOC where the nature of the course is both social science but they differ in several aspects as shown in Table 1.

Instrumentation
We used the MOOC Open Learning (www.openlearning.com)platform to provide us with the data related to learning analytics parameters.The 2 courses were hosted on MOOC Open Learning platform.The MOOC platform promotes social learning where feature such as commenting can be made available on every learning pages.‗Page' is created in MOOC to include lecture notes, video lectures, learning activities, or assessment.Data on students' interaction with pages in MOOC are collected for learning analytics.

Data collection
Data was retrieved from MOOC platform databases for 2 different courses.These courses are new courses being offered on the same platform.

Data analysis
Data was analyzed by conducting descriptive statistical analysis in comparing MOOC A and MOOC B.

Results and Findings
To explore how learning analytics can be used to improve MOOC instructional design, descriptive analysis on the pages available in the samples was carried out.Table 2 shows the comparison between the number of pages for learning activities, assessments, lecture notes and lecture videos.‗Other' includes pages that are created for the purpose of giving update about the course, inviting students to introduce themselves, or any pages that did not belong in all the other categories.Generally, results shows that MOOC A has more pages compared to MOOC B. Although both courses are social science courses, MOOC A provide more lecture notes and lecture video pages.MOOC A also provides tutorial pages to assist students to practice more of the learning content.In this study, the tutorial pages in MOOC A were categorized as Lecture Notes and Lecture Video pages.For assessment, both MOOCs used online quizzes but some of the learning activities in MOOC A include crossword puzzles.MOOC A requires students to develop project in groups but MOOC B assess students based on design document that students have to upload in the system.Due to the complexity of the course in MOOC A, more learning activities (number of pages = 10) were provided and students were given points whenever they participated in the learning activities.Generally, students in both MOOCs did not actively comment on pages.In MOOC A, there are pages that did not receive any feedback at all from the students.In MOOC B, Assessment pages did not receive any comment from the students.
In both MOOCs, most students viewed ‗Other' pages more frequent compared to other pages (MOOC A =85.85, MOOC B=88.14).In a detailed analysis, these pages are usually the ‗HomePage' where students frequently visit to get updates about the course.
Because MOOC A is a university course, more students visited video lectures and notes compared to MOOC B. But more students visited the learning activity pages in MOOC B (% of students viewed = 35.59)and has higher percentage of students completion (% of students completed=10.17)as compared to MOOC A.
Next, it is important to note that the Learning Activity Pages for MOOC B has more views than MOOC A. However, this is probably due to some of the learning activity pages in MOOC B also embedded lecture videos and lecture notes, hence students have to view the pages more frequently to learn.
In both MOOCs, social interaction among the students and instructors is rather low but it is MOOC B has higher number of comments in Learning Activity pages compared to MOOC A indicating interaction among students and instructors during learning activities.MOOC A has more interaction in -Other‖ pages and this is mostly analytics from -Home Page‖ page.
Average time spent is another important indicator of students learning preference.Interestingly, although MOOC A has less interaction among students and instructors, students spent more time in Learning Activity pages in MOOC A. They also spent more time in viewing the lecture videos and notes as compared to students in MOOC B. However, students in MOOC B spent more time during assessment as compared to MOOC A which informed us that assessment in MOOC B might be more interesting than MOOC A.
iJET -Vol.14, No. 24,2019 Based on these findings, Table 4 shows the instructional design description in pages with maximum values across 5 learning analytics parameters in both MOOCs.For every page that has the maximum parameter value, the details of the page were analyzed qualitatively to identify the instructional design used in the pages.Because Home Page was frequently viewed in both courses but did not serve as a learning page, Home Page analytics was excluded from the analysis.The similar colour of shaded areas in the table column indicates similar pages.On one hand, the first lecture video page in MOOC A has the highest % of students viewed, % of completion, number of views and number of comments.However, this page did not record the most average retention time; another lecture video later in the course has embedded step-by-step tutorial about the lesson hence students spent more time on this lecture video.For learning activity, although more students completed the crossword puzzle activity, students spent more time to do problem solving activity pages.The page presents a learning mission in an interesting way (relating to love story) that students have to solve accompanied by a protected password that can only be retrieved by using a password.This page recorded the highest average time spent for MOOC A (4 hours).They also completed the prepared online quizzes but spent more time on quizzes with increased complexity.
On the other hand, some of the video lecture pages in MOOC B combined video lecture, lecture notes and learning activity in the same page hence student spent more time on these pages to either learn from video, study notes or complete learning activities.MOOC B provided a page describing the conclusion of the course content and also asked the students to give feedback about the course.This page recorded the most average time spent on page for MOOC A (60 minutes).

The importance of 'first impression' in MOOC
Based on learning analytics, this study found that Home Pages in both MOOCs were frequently viewed.Home Page is the page where students get any course update and the first page that they visited once they enrolled in the course.This page serves as the induction set for the course hence ensuring the page to be interesting and attractive is very important.Becker-Lindenthal (2015) found that impression management is important in MOOC to help students achieve their personal goal.This is where students learning how to ‗fit-in' in MOOC hence MOOC Home Page should allow students experiment themselves with different personalities to fit-in the course.When students have a perception that the MOOC content is beneficial for them, it is more likely that the students would stay in the course (Hone & El Said, 2016).

Problem based learning activities in MOOC
In this study, the longest average time spent on a course material is 240 minutes (4 hours) as reported in MOOC A. The page contains video that presents students with problem solving activities which the course instructor relates to ‗Love Story' problem.The course also provides the element of ‗fun' in the page where to unlock the provided .pdfdocument, students have to watch the video until the end to get the password.This results in better learning retention which in line with the importance of learner centered approach that change the learners as active participants to promote student empowerment and engagement (Guàrdia, Maina and Sangrà, 2013).This is also in iJET -Vol.14, No. 24,2019 Paper-Using Learning Analytics to Improve MOOC Instructional Design line with findings by Hew (2016) on factors affecting MOOC to be highly rated and a popular course.
However, to encourage and sustain students' motivation in MOOC it is also important to create simple exercises and later proceed with complex exercises.Task complexity has effect on self-efficacy that is students' belief about their own ability to trigger motivation, cognitive resources, and courses of action needed to successfully execute a specific task within a given context (Bandura, 1997).Self-efficacy plays the role to mediate students' conscientiousness which involves the extent to which a person is efficient, hardworking, and dedicated.According to Chen, Casper, and Cortina (2001), highly conscientious individuals are more likely to be willing to engage and work hard on tasks then those low on conscientiousness, they are more likely to expect to succeed on tasks.In this study, students who had to work on complex exercises spent more time on a MOOC page as compared to simple exercises.

Course evaluation in MOOC
Based on learning analytics, an important finding in this study for instructional design improvement is the role of reflection, lesson conclusion as well as evaluation of the course.Although the page did not recorded the highest number of views, the percentage of students who completed this page is the highest in MOOC B. Apart from that, the students spent more time to complete the assessment in Course Evaluation page.Baird, Fensham, Gunstone and White (1991) indicated that reflection can improve both teachers' and students' knowledge, awareness, and control of themselves and their classroom practice.It allows them to recall back the learning experiences and hence thinking of ways to improving them.

Embrace lurking in MOOC
Learning analytics in this study also informs us that some of the pages did not necessarily demands students to be socially interactive among them.Instead, less studentstudent interaction could lead to longer time spent with the materials.The more important aspect that has to be considered is the availability of the course instructor when students seek for assistance (Hew and Cheung, 2014).In essence, social interaction in MOOC is necessary but embracing lurking activities in MOOC as a way in which students learn is also important.The freedom and flexibility to interact at their own desired time provide personalization for students learning in MOOC.

Conclusion
Learning retention and engagement is a common problem in MOOC.Lack of learning retention and engagement with the course caused students to eventually drop out the course.In many studies, designing instruction that can meet the demand of diverse learners in MOOC is a challenge.However, this study attempts to redesign instructional design using learning analytics where systematic data collection from students learning experience in MOOC was carried out.The collected data were analyzed quantitative and qualitatively to explore the learning experiences that have better percentage of completion, highest number of views, as well as highest average time spent.Learning these data informs us about the kind of activities and instructions that the students favour in MOOC.It was found that attracting the students' attention on the very first page that they visited is very important to encourage them to stay in the course.Other than that, active learning activities such as problem based learning helps to promote students empowerment and engagement.It is also important to design activities of different complexity particularly arranging them from simple to complex tasks.Instructional designers should also allow space for students to reflect on their learning and allow them to give feedback about the course understudy.Finally, not interacting in MOOC is also an indicator for students learning where lurking activities should be embraced as the way it provides flexibility for students to learn in MOOC.

References
In your text, number citations consecutively in square brackets [1].You may refer to them like -as stated in [3]‖ or -as stated in Ref. [3].A list of all cited references is placed at the end of your document, that is, in a list that is formatted and numbered automatically by applying the referenceitem style.

Table 2 .
Descriptive analysis of Pages in MOOCs across 2 courses

Table 3 .
Descriptive statistics for Pages in MOOC A and MOOC B

Table 3
provides a more detailed learning analytics of pages in MOOC A and MOOC B. Minimum and maximum values are used to describe students' preferences in engaging with materials provided in different type of pages.

Table 4 .
Instructional design description based on maximum values of learning analytics parameters.