Monitoring Indicators of the Flipped Classroom Learning Process based on Data Mining – Taking the Course of “Virtual Reality Technology” as an Example

In recent years, due to the superiority of its teaching effect, the flipped classroom has attracted more and more attention from colleges and universities. In the information age, MOOC and micro-lectures have gradually entered the college classroom and become new ways of learning and teaching. Considering the further requirement on the quality of teaching, it is not enough to monitor merely the learning results. Instead, the curriculum reform requires more attention to the learning process. Data mining was combined with MOOC and micro-lectures in this study for the case study of the course of “Virtual Reality Technology”. Academic performance, learning input and learning behavior were selected as the indicators to monitor the process of learning “Virtual Reality Technology’. Relevant data were collected and analyzed. The analysis results were used as the basis to provide ideas and approaches for the implementation of reliable and effective teaching intervention methods in the process of teaching “Virtual Reality Technology”, so as improve the quality of teaching and provide reference for other related courses. Keywords—MOOC, learning process monitoring, micro-lectures, data mining, flipped classroom technology


Introduction
Since the emergence of the three major MOOC platforms, namely Coursera, Udacity, and edx, online education has been developing rapidly. In 2013, Tsinghua University established the MOOC platform xuetangx.com, promoting the innovation of Chinese educational technology and offering a new learning mode and education model for Chinese education [1]. Based on the Internet, MOOC platform has the basic characteristics of the Internet. To be specific, they have a large amount of unstructured data, and can store the learning behaviors of students on the MOOC platform in the form of data records. These seemingly scattered and irregular data, in fact, can reflect students' learning situation and learning characteristics. Teachers can use the data saved by MOOC to track, improve and evaluate students' learning situation [2]. In traditional courses, it is quite difficult for teachers to provide personalized guidance for thousands of learners: it is hard for teachers to identify learners' behaviors and learning effect or improve curriculum design by analyzing learning behavior data. Personalized guidance cannot be provided at the appropriate time to improve learners' learning effect, thereby reducing learners' learning efficiency and further affecting the teaching effect. Data mining refers to the use of data analysis to obtain valuable information that researchers need from a large amount of random and fuzzy data collected. When applied to classroom education, it means that teachers analyze students' learning behavior data to understand their degree of learning, learning effect, learning problems, etc., so as to carry out targeted adjustment and curriculum design. The combination of data mining and MOOC courses can provide teaching evidence and support for MOOC education [3]. In recent years, since the flipped classroom teaching model was confirmed to be more effective than the traditional teaching model, the flipped classroom has also been valued in college education and applied to teaching practice. The core of the flipped classroom is to shift the classroom center from teachers to students and combine the classroom with information technology. In the flipped classroom, teachers use more scenarios, while students conduct cooperative inquiry and learning through pre-class study, inclass practice and after-class thinking. The emergence of the flipped classroom provides a new direction for college classroom teaching [4]. Virtual Reality Technology is not only an important part of college art major courses [5], but also indispensable knowledge of computer major courses [6]. Whether it is basic public art, vocational combat training or art design, it is related to Virtual Reality Technology. Therefore, it is quite necessary to explore a new teaching method for Virtual Reality Technology to improve students' learning effect in this course. With the focus on the course of Virtual Reality Technology, the flipped classroom model was used in this study, on the basis of combining data mining and MOOC platform, to analyze the monitoring indicators of the flipped classroom learning process based on data mining and explore effective teaching intervention methods for the flipped classroom learning process based on data mining in the course of Virtual Reality Technology, for the purpose of improving the quality of teaching.

Research questions
The teaching based on MOOC and micro-lectures was designed in this study for the course of Virtual Reality Technology and was applied in teaching practice. At the same time, the method of data mining was combined with some learning process monitoring indicators to monitor students' learning situation in the process of learning Virtual Reality Technology, thereby carrying out teaching intervention in students' learning process. Focusing on the teaching and learning process of Virtual Reality Technology based on MOOC and micro-lectures, this study aims to discover the underlying characteristics of students' learning behavior in the course of Virtual Reality Technology based on MOOC and micro-lectures through data mining, so as to adjust students' learning behavior in time, help them better master MOOC and microlectures, and provide a basis for improving the teaching effect of the flipped classroom.

Research methods
The method of combining indicator monitoring with statistical analysis was mainly adopted for the analysis of the process of learning Virtual Reality Technology. Indicator monitoring refers to the selection of the indicators suitable for monitoring in the learning process and the use of the indicators to collect data and monitor students' situation in the process of learning Virtual Reality Technology. There are three major indicators, including the indicator of academic performance [7], which is the evaluation indicator used to reflect students' learning effect, the indicator of learning input [8], which is the measurement indicator because the learning input degree affects academic performance, and the indicator of learning behavior [9], which mainly refers to the response of students after encountering difficulties or setbacks in the learning process. Their choice to give up or discuss further and study videos repeatedly, etc. also influences their academic performance. In terms of statistical analysis methods, statistical software was used to analyze the data collected according to the selected indicators, and to find the law. In general, data mining has four statistical analysis methods, namely cluster analysis, classification analysis, numerical analysis and correlation analysis [10]. In this study, academic performance was selected as the dependent variable. Because it is a continuous variable, numerical analysis, specifically the regression analysis method, was adopted to analyze the data obtained, thus to provide guidance and support for the teaching process.

Research objects and research process
A grade of University of Science and Technology Liaoning was selected in this study to conduct a teaching practice study in 15-week experiment teaching, covering a semester. Firstly, the teaching model for the course of Virtual Reality Technology was designed based on the teaching concept of the flipped classroom. Secondly, under the flipped classroom teaching model, MOOC platform and micro-lecture teaching were combined to carry out online video teaching and design the teaching content. After that, the selected class engaged in the course learning and practice of Virtual Reality Technology on the MOOC platform. During this learning process, the MOOC platform automatically recorded and saved the students' learning behavior data. After the end of the semester, data were mined from all the data records, to explore students' learning characteristics in the process of learning Virtual Reality Technology, and provide effective teaching intervention experience to improve the flipped classroom based on MOOC and micro-lectures.

2
Flipped Classroom Teaching Design based on MOOC and Micro-Lectures

Design of pre-class learning of the flipped classroom
Pre-class learning of the flipped classroom is also a very important link. It is the basis and premise for the implementation of flipped classroom teaching activities. Therefore, the teaching design of the pre-class learning link should be firstly carried out for the flipped classroom. The core of the flipped classroom pre-class teaching design is the task. MOOC is used as the learning platform, while micro-lectures are used as the teaching resources, to integrate self-study and cooperative inquiry, evaluate students and form feedbacks. First of all, the teaching objectives are identified and a series of tasks are then set up according to the knowledge points and difficulties of teaching determined in the syllabus. Secondly, micro-lecture resources and the network platform should be prepared well, so that students can learn, practice and experiment independently before class. Thirdly, the evaluation feedback mechanism should be designed carefully, so that students can monitor their own preclass learning, record and store their problems, and provide a basis for the teaching design of classroom activities in the flipped classroom in the next step. The teaching design of the flapped classroom pre-class learning link is shown in Figure 1.

Teaching design of classroom activities in the flipped classroom
Classroom activities are the key to the success of the flipped classroom [11]. In particular, classroom activities determine whether students can internalize what they have learned into their own knowledge structure and improve their learning ability. The teaching design of classroom activities in the flipped classroom should be centered on the internalization of knowledge and the improvement of ability, follow the principle of problem-oriented learning, emphasize learning in practice, combine various learning methods, and simultaneously conduct real-time evaluation and feedback on teaching activities. First of all, the objectives and tasks of teaching students' learning problems over the years, and their pre-class and in-class learning situation should be combined to raise the questions to be discussed in the classroom. Secondly, learning in practice and interactive communication are carried out around the raised questions. On the one hand, students are the subjects, for whom discussion and collaboration should be strengthened. Meanwhile, teachers provide guidance. On the other hand, after discussion and inquiry are completed, students need to interact and exchange their discussion results and ideas. Thirdly, in the process above, teachers should give evaluation and feedback to students in a timely manner, so that problems can be resolved promptly, mobilizing students' attention and enthusiasm. The teaching design of classroom activities in the flipped classroom is shown in Figure 2.

Teaching design of the flipped classroom for Virtual Reality Technology
The flipped classroom model consists of pre-class, in-class and after-class part. Before class, students independently watch videos of micro-lectures and practice. After the teacher completes the teaching in class, students explore, discuss, evaluate and solve the questions. After class, students consolidate and reflect on the knowledge they have learned. The design of the flipped classroom teaching model for Virtual Reality Technology should combine the afore-mentioned flipped classroom teaching model, as well as the teaching content and teaching objectives of Virtual Reality Technology, and take into account the operability and practicality of Virtual Reality Technology. First, before class, the teacher should use the MOOC platform to produce courseware, audios, and videos containing knowledge, skills, demonstrations, exercises, and evaluations related to Virtual Reality Technology according to the teaching objectives and content, so that students can watch them and learn before class. Students learn and evaluate on the MOOC platform, and communicate with the teacher or classmates on difficult issues. In class, the teacher starts from the tasks, sets the scenarios, carries out specific practical teaching, provides counseling to students, allows students to experiment independently, and evaluates the results. After class, students complete after-class exercises and consolidate what they have learned. The flipped classroom teaching model for Virtual Reality Technology is shown in Fig. 3.

Selection and analysis of indicators for measuring academic performance in Virtual Reality Technology
Due to the particularity of MOOC, it is different from traditional scientific measurement in which data are measured according to the indicators after the indicators are determined. MOOC usually means that data appear before indicators. Therefore, it is only possible to mine the data provided by MOOC according to the expected targets. This process must be both scientific and practical. Hence, data mining was firstly used in this study to find the indicators that reflect student's learning effect in the course of Virtual Reality Technology. Virtual Reality Technology is divided into 18 weeks, consisting of 3 steps, namely video, exercise and homework each week. The video is mainly for students to watch and the teacher to explain, presenting the knowledge points of Virtual Reality Technology; the exercise is for students to practice and discuss the content presented in the video in class; and the homework is to measure students' mastery of the knowledge they have learned in one week. At the end of the study period, a final exam is conducted to reflect students' overall learning of Virtual Reality Technology. Therefore, the indicators for measuring academic performance mainly include the exercise score, homework score, and final score. Among them, the final score is obtained only at the end of the semester, so that it has almost no intervention in the whole learning process but can offer intervention guidance in the next learning process; once the homework exceeds the deadline, the score cannot be recorded, leading to incomplete information. Therefore, the exercise score was chosen in this study as the main indicator for measuring academic performance in Virtual Reality Technology. Because of its class immediacy, it not only provides a large amount of data, but also makes it possible to intervene in the classroom according to the results at any time.

Selection and analysis of indicators for measuring learning input in Virtual Reality Technology
Generally speaking, the length of learning time is the most accurate measurement indicator to reflect learning input. In the flipped classroom, each student learns for the same length of time. Therefore, the difference lies mainly in the length of time students spend on watching videos of micro-lectures on the MOOC platform. However, the behavior of students is very complicated when they are watching videos. For example, some students are used to finishing watching videos at one time. Some students are used to recording while watching, so that they would play and pause continuously, which undoubtedly increases the length of time. Some students really like to skip some part, resulting in the reduction of the length of time. Thus, it is not credible to use the length of play time as a measurement indicator. In this study, the play frequency was selected as an indicator for measuring learning input. This indicator is also biased, but it can be adjusted to reduce the bias. The first step was to measure the ratio of the frequency of playing micro-lecture videos before and after the exercise in the week; the second step was to measure the ratio of the frequency of playing micro-lecture videos before and after the exercise in the previous week; the ratio of the week and the ratio of the previous week are compared. If they are both greater than 1, the ratio of the week to the ratio of the previous week is used as the ratio of the adjusted frequency of the week. If either of them is less than 1, then no adjustment is made. After that, regression analysis was used to analyze the relationship between the frequency of playing videos in the week and the exercise score (a continuous variable between 0-1), that is, the relationship between learning input and academic performance. At the same time, it was investigated whether there is difference in R2 of the regression models with different learning characteristics, to explore whether the explanatory power is different. The regression model for the ratio of the play frequency before adjustment and the ratio of the play frequency after adjustment is shown in Table 1 below. It can be seen from Table 1 that the play frequency after adjustment is slightly more powerful than the play frequency before adjustment in interpreting the regression model. This indicates that the adjustment is effective and that the play frequency indeed has a positive effect on the exercise score. However, the increase in explanatory power is not large, demonstrating that this positive effect is also limited. Further analysis found that only 20% to 30% of students were affected by the adjustment of the play frequency in the process of learning Virtual Reality Technology, suggesting that the play behavior may be only one aspect of students' difference in learning behavior, and other data which have not been found in this study might have an impact on academic performance.

Selection and analysis of indicators for measuring learning behavior in Virtual Reality Technology
In the research process above, it was found in this study that there is a strong positive correlation between students' learning input and academic performance, and there is also a positive correlation between academic performance in the previous week and academic performance in the week, because of the correlation and progressive nature of the weekly knowledge of Virtual Reality Technology. Besides, this relationship between learning input and academic performance provides the foundation for the following study of students' behavior when they encounter difficulties or frustrations in the learning process. From the data recorded by the MOOC platform, it can be found that after giving wrong answers to exercise questions, students usually chose to review videos, in order to avoiding giving wrong answers in this way. The exercise score and whether the problem is solved after reviewing videos were taken as the independent variables, while the weekly homework score (a continuous variable between 0 and 1) was used as the dependent variable to examine the correlation between them. The results are shown in Table 2. Table 2 shows that the learning behavior of reviewing videos cannot increase students' academic performance score, since it fails to solve the problem of giving wrong answers and to improve the learning effect. This suggests that reviewing videos to solve difficult questions after students give wrong answers is not an effective intervention in the flipped classroom. At this time, external help is needed, such as the guidance from the teacher and the discussion between students, to form teaching intervention and solve the problem. Although reviewing videos does not help students solve the problem, it allows the teacher to know which students need teaching intervention. In addition to the ratio of reviewing videos, the discussion area on the MOOC platform was also monitored in this study. The ratio of the students participating in discussion and communication was adopted as a monitoring indicator, leading to the finding that the ratio of the students participating in discussion and communication is much lower than that of the students reviewing videos. On the whole, only 8.7% of the students discussed the knowledge of Virtual Reality Technology; and only 3% to 14% of the students discussed the exercise questions after giving wrong answers; the ratio of the students reviewed videos is between 40% and 80%. These ratios indicate that students are more willing to divorce themselves from the masses when learning Virtual Reality Technology, and are less willing to discuss. Monitoring the indicator of discussion and communication can help teachers understand students' learning situation and learning style, judge students' enthusiasm for learning, and position their roles. Teachers can thereby determine whether they need to offer simple guidance or aggressively guide students to cooperate and discuss, providing important teaching intervention methods.

Conclusion
For a long time, researchers only use the final score as the indicator to adjust and intervene in classroom teaching, but ignore the more important teaching process. In this process, timely understanding of students' learning situation and teachers' teaching situation and teaching intervention carried out according to data feedback results are helpful for solving students' problems immediately, offer assistance to students and enhance students' learning effects. In contrast, the use of the final comprehensive examination as a basis for teaching intervention allows only understanding the final result, and it is difficult to judge where the problem occurs and impossible to provide assistance to current students. Therefore, it is very necessary to monitor the teaching process. In order to explore students' learning process in the course of Virtual Reality Technology under the flipped classroom teaching model based on MOOC and micro-lectures, data mining was combined with the flipped classroom teaching design concept in this study to conduct micro-lecture design on the MOOC platform, and monitor students' learning process in the course of Virtual Reality Technology. The indicators for monitoring the learning process were selected and analyzed, including academic performance, learning input and learning behavior. Meanwhile, effective intervention measures were provided for the teaching process according to the data mining results. Research shows that the monitoring of the learning process and the use of data mining methods enable teachers to acquire timely understanding of the problems students encounter in the learning process, and identify the specific problems of specific students, so that teachers can give targeted teaching guidance according to the different situation of different students; on the other hand, teachers can understand students' learning habits and learning behavior characteristics, combine these characteristics with the nature of the flipped classroom, adopt appropriate methods to guide students to interact, and stimulate students' learning initiative. At the same time, teachers can thereby strengthen the management of students and prevent students from giving up halfway. Although this study is focused on the course of Virtual Reality Technology, the research methods and ideas used can also provide reference for other related courses. It should be noted that data mining requires enough data samples. Generally speaking, more data result in more reliable results. Thus, when conducting research, the acquisition of the minimum data sample should be guaranteed according to the specific situation. This requires subsequent research and the continuous accumulation of data, to provide teachers with sufficient and reliable data analysis results as early as possible, so that teachers can rely on the data mining results to take teaching intervention in the flipped classroom and improve the flipped classroom teaching.

5
Provincial Education Department) and Research and Practice of "Three Main Lines and Six Platforms" Cross-integration Talents Training Model (JG18DB276).