The Online Course as Initial Stage of a Course in Higher Education: Implementation and Evaluation of the Pre-MOOC Concept in a Technical Degree Course

clarissa.braun@tugraz.at Abstract —In university teaching, a massive open online course (MOOC) is often regarded as an isolated online course, typically addressing a broad public of lifelong learners or future students. Nevertheless, some universities use MOOCs as a lecture for their students or use MOOCs as integrated part of their own lectures. In this study, we focus on a lecture concept called “Pre-MOOC”, where a MOOC is an obligative first part of a lecture in higher education, followed by lecture-part only open for own students and typically in presence. Therefore, we explore the current state of practices and research on Pre-MOOCs and describe our design-based research approach, the development of the MOOC and evaluation of its implementation in 2019. The article describes the instructional design process of the lecture in a master program of electrical engineering in which we used the instructional design model ADDIE. Finally, we will analyze the results of the evaluation with regard to the concept of a Pre-MOOC. Based on feedback from students and teachers, recommendations for designing a course with the Pre-MOOC design will be provided and implica-tions as well limitations of the study will be


Introduction
For several years, so-called "Massive Open Online Courses" (MOOCs) have been a part in research in technology enhanced learning. MOOCs are online courses with many participants, at least 150. They have to be open for everyone, cost-free, accessible online, and must have the general framework of a course with a defined start and end time of the course, weekly new content etc. [1]. MOOCs are not necessarily a substitute for a conventional lecture, but are used in a variety of ways at universities.
The initiative of our research activities bases on a concrete wish of one of our university's lecturer, to use this design for a course for electrical engineers at Graz University of Technology (TU Graz) in Austria: The course consists of two partsone part is a regular lecture and the other part is a laboratory tutorial, latter usually needs to start with detailed instruction into the laboratory procedures and a test on the theo-retical basis for the experimental setup. Since the introductions and entrance tests always take place in the same way and also take up valuable working time in the laboratory, the idea was to develop a preparatory course with learning videos and quizzes. Since there was no reason not to share this information with others and third parties could benefit from it, the idea was born to develop this preparatory phase as a MOOC. The MOOC should then be the first part of the course and at the same time the prerequisite for the work in the laboratory, so a "Pre-MOOC" as we will describe later.
So, we decided to explore the expedient implementation of the Pre-MOOC design in a lecture in higher education according to the design-based research approach. Our design-based research and results on the preparatory literature research and formative as well summative evaluation processes will be described in this contribution, starting with a description of our questions.

Research Questions
In this article, we will document and describe our design-based research following these questions: a) What can we learn from existing implementations of the Pre-MOOC design in literature? b) How is the redesign of a conventional lecture into a Pre-MOOC-concept carried out? c) How does the redesign affect the lecture from the students' point of view? d) What recommendations can we make for implementing the concept to other practitioners?

Research Approach
The described research uses a "design-based" or "design research" approach as described by Reeves et al. [2] and others, which has the main idea of: "Theory informing practice is at the heart of the approach, and the creation of design principles and guidelines enables research outcomes to be transformed into educational practices" [2, p. 107].
As sources we use our documentation and accompanying research on the development and implementation of a novel teaching concept, which we developed with the help of the ADDIE model. This model provides for a detailed evaluation of the teaching concept. We used a mixed methods approach. First, formatively, we used the implemented learning analytics tools and services of our learning analytics experts to get current data and insights into learners' activities in the MOOC system. Additionally, we will present excerpts of the summative evaluation results in which we interviewed students of the course with online questionnaires (with simple descriptive statistics) and problem-based interviews (following a short interview guideline). Furthermore, the main lecturer was also asked to provide recommendations for the pre-MOOC setting in the overall lecture evaluation. Fig. 1 gives an overview of the development of the course and parallel research activities.

Fig. 1. Course development and parallel research activities
Finally, we extract preliminary recommendations developed in a group discussion of the authors, including the lecturer and formulated preliminary recommendations for practitioners.
First, we will give a brief insight into existing literature.

Literature Review: Experiences with MOOCS as Initial Phase in Higher Education
Recently, we presented seven teaching scenarios with MOOCs as we have observed or accompanied them in recent years on the Austrian MOOC platform iMooX [3,4]. With a so-called "Pre-MOOC"-scenario, the MOOC is used as a preparation for a face-to-face event. The MOOC is already completed before the start of the face-toface classroom meeting (see Fig. 2): "The Pre-MOOC is used when learners require previous knowledge on a specific topic, which will then facilitate understanding and communication between teachers and learners in the following classroom meetings. In some cases, examined in the study, there was an (additional) inquiry into the level of knowledge before the personal interaction between teachers and learners, and, usually in a university context, a subsequent examination. Both examinations are to be seen as optional." [3]. First, we have to state that the lecture concept of a "Pre-MOOC" is not a common term in literature. Systematically, Pre-MOOCs can be assigned to "blended learning", i.e., an enrichment of face-to-face with online learning [5], and also partly to the idea of the "flipped classroom". However, the concept of the Flipped or Inverted Classroom does not require learning videos in the shape of MOOCs. The Pre-MOOC does not require including methodological-didactic features, such as a more interactive presence phase with a powerful practice character like we see it in flipped-classroomsettings [6]. Others have also tried to describe the distinct forms of "blended" MOOCs: [7] describes implementations of "hybrid MOOCs" based on the dimensions "curriculum fit" and "institutional support".
Although we have identified such a pattern of MOOC integration in a lecture in higher education, there are only some research articles or other shared experiences about this lecture concept. Basically, all documentation of MOOCs that we could identiy as "Pre-MOOC" are described by colleagues who did a MOOC at our iMooX.at platform: For example, the "Mathe-Fit MOOC" is a bridging course designed to help school graduates to close the knowledge gap between school and university in the subject 'math': The six-week MOOC was a prerequisite for classroom exercises to prepare for the start of the course, both of which were scheduled before the start of the first semester [8]. The "Informatik-FIT MOOC" described in the same text was conducted similarly [9]. Furthermore, a MOOC was offered to prepare for the EMOOCs conference in 2016. All these descriptions are on a more or less superficial level so we cannot extract recommendation from it, beside our involvement as platform host.
As the iMooX.at platform is used intensively by lecturer who want to enrich their lectures with a MOOC and/or integrate it into their lectures, this might not as common for others: Only a small percentage of MOOCs are developed in a blended learning design from the outset: In a survey of 143 MOOC developers, only 5 of them (4%) have already implemented a "blended" MOOC [10]. Existing MOOCs are also difficult to integrate into the further course of the study [11].
To sum up: We could not find a detailed case description or study of a Pre-MOOC in the German and English literature where we could start with as from a raw script of recommendations or assumptions about this special type of integrating a MOOC into a lecture.

Lecture Re-Design Measures
Basing on our experiences, the re-design of a lecture concerning elements of technology-enhanced learning or more special, a MOOC are activities which are not only focused on didactical issues.

a. Re-design of the complete course "Protection and Security of Supply of Electrical Energy Systems"
The re-design of the course "Protection and Security of Supply of Electrical Energy Systems" at Graz University of Technology (TU Graz) is based on the Instructional Design Model ADDIE. ADDIE is an acronym for "analysis, design, development, implementation and evaluation". In literature, the model is a generic term that refers to a family of procedural instructional design models that share a common basic structure in instructional design [12,13]. According to Seel and Schott (2012), instructional design comprises the following tasks [13]: Conducting a needs analysis, identifying (teaching) goals, breaking down goals into sub-goals, identifying the knowledge required to achieve the sub-goals, defining learning content, developing criteria-based tests and/or other assessment measures, developing teaching strategies, developing materials and using media, conducting a formative and/or summative assessment, and conducting a course evaluation for quality assurance and student satisfaction.
So, first, the needs analysis was carried out together with the head person responsible for the course. This included the exact definition of the target group and the level of expected prior knowledge. We defined the target group as a homogeneous group of master students in the 7th semester. In the curriculum, they directly link the lecture to a laboratory which takes place in the same semester after the lecture has finished. Furthermore, concerning the design, the aim was to identify lecture content which can be understood without further explanations by the lecturer, can further serve the preparation of the laboratory and can also be tested well with a multiple-choice quiz.
Based on Ebner, Schön and Braun's different (didactic) scenarios with MOOCs [3,4] distinct possibilities of realizing the blended learning environment with the support of a MOOC were discussed. We made the decision to develop a Pre-MOOC named "Electrical Network and System Protection".
The contents within the MOOC should also serve as a theoretical basis for the subsequent laboratory. For the laboratory, it was determined that Master students from other semesters should also have access, provided that the MOOC was completed with a certificate and the lecture was already attended in previous years.

b. (Check of) Needed resources and possibilities: Learning videos and MOOC platform
Besides these potential possibilities of a re-design, reality has restrictions concerning needed resources. For a MOOC development, online course development or use of technology-enhanced learning in general, this typically is the calculation of the needed (extra) effort for development of the new resources and the potential future expectations and effects concerning their positive effects on learning as well on future teaching efforts.
In our case, the teacher had already recorded several learning videos for the previous lecture. These videos were later used in the MOOC but still had to be divided into smaller parts. Based on these, then 19 videos were produced (each 4-to 6-minute learning videos), another 8 videos were to complete the MOOC. The videos were professionally produced by the department "Educational Technology" of TU Graz. The main objective was to create a sustainable lecture (as the videos can be used multiple years by different lecturers) with enhanced student interactivity during the lecture. The revision took place during regular working hours. The teacher expects that the materials and the concept can also be used in the next few years and that the additional expenditure will pay for itself.
The support of the development and hosting of the MOOC is a service of the "Educational Technology" team of the TU Graz. Since, 2014 the team provides the Austrian MOOC platform iMooX. iMooX offers courses at university level in different disciplines. Every course on iMooX uses Open Educational Resources (OER), so that every single learning object is openly licensed (Creative Commons) [14]. This feature can be very useful for universities, as the courses or individual videos can be freely used and adapted by other teachers in the respective institutes. Also relevant for the decision was the fact that the course has typical xMOOC features, including videobased implementation, weekly new units with multiple selection tests and a discussion forum for student interaction. Participants will receive a certificate of participation if they have passed over 75 percent of the tasks in the individual tests, whereby the individual tests can be taken several times [3,4].
c. Verifying the conditions for online or "virtual learning" at the university Finally, the legal aspects also had to be taken into account: From the analysis of the university framework conditions for virtual teaching it can be concluded that "virtual teaching as a didactic tool in lectures […] can be freely implemented by the teacher up to a threshold value of 20% of the SSt [weekly hours per semester]" [15, p. 3, own translation]. It was therefore necessary to include this threshold value in the design process and to design the online phase accordingly.

6
The Detailed Lecture, Implementation and Participation It was important for the teacher to gain the support and understanding of the students for this change and also to enable them to get to know each other for the first time before a purely online phase. So, the course begins with a face-to-face meeting. Then the developed MOOC took place with the final MOOC certificate. It is followed the face-to-face lecture and finally a laboratory (see Fig. 3). The schedule thus corresponds to the design of the Pre-MOOC.

Fig. 3. Complete lecture "Protection and Security of Supply of Electrical Energy Systems"at a glance
At the beginning of the semester, the students were informed in advance about the new concept and procedure, and the exact time-and deadlines, the meaningfulness of the teaching scenario (outsourcing of face-to-face parts, learning when and where you want, better combination with supervision duties). The simple login on the MOOC platform iMooX through the registration data of TU Graz enables a low-threshold course entry for the students and is also relevant for the issuing of the certificate.
The MOOC was offered as a five-week online course on the MOOC platform iMooX.at. During this time, it enrolled 202 people in the course. Of these, 44 participants completed the course successfully. 43 persons requested the certificate download 36 of which were students of the course, among them no women. The prerequisite for attending the lecture and the laboratory was the positive completion of the MOOC with a certificate. The certificate can be obtained by correctly answering 75 percent of the 51 multiple-choice questions on the specific topics of the MOOC.
During face-to-face sessions the lecturer tried to deepen (consolidate) the knowledge which the students gained in the videos and quizzes. The lecturer was primarily coaching the students while they were working on real-life tasks provided by him. For those students attending the lecture the main benefit was gaining a lot of future-work-related project knowledge. The lecture was held in blocks of four 60minute sessions spread over 4 weeks and concludes with a written and oral examination (in one session) at the end of the semester (see "Examination" in Fig. 3). The students know questions of the examination as a sample collection of examination questions provided by the lecturer. 28 students attended the lecture (only male). The face-to-face phase of the lecture was adapted to the concept of the Pre-MOOC: Since this concept requires comparable considerations to the flipped classroom concept, we used here the literature on interactive flipped to prepare adequate classroom activities. For example, the lecturer prepared lecture hall games e. g. "Ring the Bell" [16] and examples of use for electrical engineers. Students were asked to work in groups to accomplish the tasks given by the lecturer rather than just listening to presentations.
The lecture-phase was followed by the laboratory which was carried out over 6 weeks on a two-hourly basis. The regulation of the laboratory-instructor, which was valid in t he years before the new course design, was to attend the lecture previous to the laboratory to get some basic knowledge. In the study semester, when the introduction of the Pre-MOOC-design took place, students could also attend the laboratory only by showing the MOOC-certificate to the instructor. However, it was still obliga-tory to pass the oral entrance test for each of the 6 laboratory units. This oral test are oral questions to check whether the student has enough basic knowledge (all passed). The number of participants in the laboratory was 14 students. 8 out of these had already attended the lecture in previous semesters (without MOOC), so we did not include them or their answers in the following evaluation.

Formative and Summative Evaluation of the Course and the Pre-MOOC Design
Formative during the MOOC as well as summative evaluation was implemented. Formatively, the lecturer was especially monitoring the discussion of the learners in the MOOC forum as well monitoring the progress of the learners with the support of the Learning Analytics team and tools implemented in the MOOC platform. In this way, the activities of the students in the MOOC, for example their success in the quizzes or "difficult" and "easy" questions, or gaps in knowledge of the students could be identified in a timely manner (see Fig. 4). This was also a good support for the teacher in preparing the lecture, as he had a clear idea of tricky questions or wrong concepts of the students.

Fig. 4. Exemplary learning analytics visualization used within formative evaluation through monitoring students' course activities and (preliminary) results
The entire course and its components were evaluated, i.e., surveys were conducted in the Learning Management System as well as interviews with participants and the teacher. In the following, only those evaluation results will be presented that specifically refer to the concept of the Pre-MOOC-design. For evaluation, questionnaires were sent to the 36 participants of the entire course via the TU Graz internal learning management system TeachCenter (response rate 31%), and we conducted interviews with relevant laboratory participants.
The results showed that most students are satisfied with the new design of the course: over 90 percent agree with the statement that the course should be held in the same way again; we satisfy the students with the embedding and reference of videos and laboratory work, and the majority also state that they were stimulating (see Fig.  5). Two-thirds also prefer the upstream version of the MOOC to a parallel implementation (Fig. 5, 2nd question). The interviews with the laboratory students showed a consistently positive consensus towards the course concept "Pre-MOOC". The majority of the respondents had not yet had any experience with online courses, but felt that universities should integrate more of such into teaching. Concerning the interviews, the contents of the MOOC have prepared them well for the laboratory requirements and weekly tests. The interviewees highlighted the better flexibility for them in the MOOC. A student said: "As long as he [the teacher] demands to watch the course not only in one day, we students have the opportunity to prepare even more flexibly". Flexibility means as well, that some students used the MOOC (only) as a "crash course" just before the laboratory started. Interviewees also mentioned that it is great to watch the videos again to get easily prepared for the exam. Many of them took notes while watching the videos, just like they were sitting in the face-to-face classroom.
From the perspective of the lecturer, there was a top level of satisfaction with the insights gained by the Learning Analytics team affiliated with iMooX.at. In the analysis phase of the ADDIE model, however, he had to weigh up the costs and benefits of the concept, but consciously opted for the sustainable Pre-MOOC concept for his course. Another additional aspect for him was the redesign of the face-to-face phase into a more interactive setting; here, too, he was positively surprised by the cooperation of the students.

Preliminary Recommendations for Designing a Pre-MOOC
Design-based research in higher education needs to "reflect on the process to reveal design principles that can inform other instructors and researchers, and future development projects" [2, p. 109].

a) Collection of recommendations
Based on the results of the accompanying research, also against the background of existing experiences (see above Section 3), we collected recommendations for designing courses in the Pre-MOOC-manner. These recommendations were shared within our submission and first version of this article. Building upon the reviewers' feedback of our submission, we decided to develop a more hands-on visualization.

b) A decision tree for decision on the Pre-MOOC development
To make the preliminary recommendations clearer, we have designed them in the form of a decision tree as questions, that are of relevance in the decision process of a Pre-MOOC development (see Fig. 6). • In principle, the teachers have initial experience and materials for online teaching and are interested in producing and implementing a (Pre-) MOOC • You are looking for the possibility to make the actual classroom event more interactive or practical and want to teach theory parts in advance during a blockheld face-to-face phase as preliminary preparation (as an alternative to the flipped classroom method) • Learners might have different prior knowledge • The content is not changing each semester or year so that your parts of the contents needs to be revised in short-time • Learners should get or are interested in more flexibility in learning • The legal basis is given (e.g., allowed percentage of online teaching).

b) A MOOC as base of the online part can be of interest, if
• It is of interest that students (in the MOOC phase) also get in contact with other learners outside the university • Students from different universities should take part • The lecturers are interested to share their teaching offers with students from other universities or life-long learners as well • Tests and certification implemented in a MOOC platform are helpful and/or needed.

c) A Pre-MOOC design is fitting, if
• You are looking for a possibility to only let students who already have previous knowledge into your course (the Pre-MOOC as access authorization) • The needed content is, according to your analysis of target group and curriculum understandable with limited possibilities to discuss and train in parallel, e.g., in exercises • The expenditure of a MOOC development can be afforded, also because small expenditures may be expected in the future • The MOOC's course materials do not need to be revised and adapted in the foreseeable future • The framework conditions of the platform are suitable or flexible enough (e.g. start and end date, access, data protection, licenses) Additionally, the lecturer in our cases implemented a face-to-face-meeting as starting event of the course. We see this as very helpful to share the idea of the concept and our expectations on it. So, the students are better informed and might appreciate it more also.

Summary, Limitations of the Study and Outlook
Finally, we want to come back to our research question and summarize the found answers as described in this article. First, we asked what we can learn from existing implementations of the Pre-MOOC design in literature. According to our literature research our paper might be one of the first contribution were details and recommendation are shared. Then we gave an answer on how the re-design of a conventional lecture into a Pre-MOOC-concept was carried out: It based on some questions on motivation, efforts, existing material and the rules for online lectures, and a development of the re-design following the ADDIE principles. Our evaluation gave answers on the question how does the redesign affect the lecture from the students' point of view: The students appreciated the course-design and highlighted their possibility of getting more flexible. Our final research question on recommendations can we make for implementing the concept to other practitioners was given with a decision tree on if a course in higher education could be re-designed as Pre-MOOC.
The accompanying research presented here was oriented towards the ADDIE model, i.e., the evaluation phase of ADDIE refers to the success of the students in the course and the success of the course in general. In our case, this means that the accompanying research did not only focus on designing the university course as a Pre-MOOC.
As another limitation, we should note that because of missing data only the lecturers can draw a direct comparison with the usual lecture procedure, the students however are in most cases not aware of any alternative setting, e.g., MOOC integration in courses or the flipped classroom method.
Another limitation is that design-based research base builds upon "long-term engagement involving continual refinement of protocols and questions" [2, p. 103]. As we only designed and researched a single development and implementation of a Pre-MOOC, we might tend to overgeneralize our experience within our special setting and requirements. Therefore, we want to share our experiences as recommendations to others with similar design or research ideas, but highlight this limitation with framing the following recommendations as "preliminary".
For our future research, this is as well a recommendation for other researchers, we will more focus on design considerations and an even more detailed documentation on design-issues, because we see the opportunity for better re-capitulation of lessons learned and recommendations for others.
We hope that with the help of this preliminary list of recommendations, it will be easier in future to carry out consultations and decisions for courses in the Pre-MOOC design. When developing an own MOOC, which is part of the core mission of universities in our educational system, it may make additional teaching materials of a university available to the public. And of course, we would be happy to support similar projects of other researchers with our preliminary insights [17,18,19].