Integrating MOOC and Flipped Classroom Practice in a Traditional Undergraduate Course : Students ’ Experience and Perceptions

The purpose of this experimental study was to redesign a traditional undergraduate course by integrating MOOC content and flipped classroom practice and to see its effectiveness through students’ experience and perceptions. The course named “Internet and Distance Education” was taught in Winter Semester, 2013 at the Zhejiang University in Hangzhou, China to 15 undergraduate students majoring in education. E-learner satisfaction surveys found that students were generally satisfied with many aspects of the redesigned course, including instructor response timeliness, instructor attitude toward the technology, e-learning course flexibility, technology quality, Internet quality, perceived usefulness, perceived ease of use, and perceived e-learner satisfaction, while learner perceived interaction with others was not so satisfactory. Based on the findings, several suggestions to improve the course design are offered.


I. INTRODUCTION
It has been advocated for many years that the use of learner-centered instructional environments would facilitate the learning process and improve the overall learning outcomes [1].Nevertheless, researchers have tended to find that instructors have difficulties designing their courses based on principles of learner-centered learning because of time constrains and physical boundaries [2].With the emergence of massive open online courses (MOOCs) and flipped classroom practices, researchers and educational practitioners are increasingly arguing that integrating MOOCs and flipped classroom practices based on shared online videos can facilitate the creation of learner-centered learning environments.Such environments help maximize individual learning outcomes and democratize higher education [3].The present study documented an experimental study related to the integration of MOOCs and flipped classroom practices in a traditional undergraduate course.In particular, it explored students' experiences and perceptions about such emerging forms of learning delivery.Before introducing the study, key literature concerning MOOCs and flipped classroom practices is summarized.

A. MOOC
A massive open online course (MOOC) is an online course aimed at unlimited participation and open access via the Web [4].The term MOOC was first introduced by George Siemens and Stephen Downes in 2008 when they described an experimental online course named "Connectivism and Connective Knowledge" that had 25 paid enrollments (for credit) with around 2,200 non-credit, nonfee paying students.This course quickly gained great attention around the world and, after that, MOOCs were increasingly offered by some world-class universities in partnership with different MOOC providers and services [5].At present, Coursera, edX, and Udacity are three major MOOCs platforms; among them, Coursera is the most well known MOOC-related venture with 116 higher education institutions (HEIs) as partners.As of March 26, 2015, it offered more than 992 courses from 117 institutions to 12,088,380 registered users from 190 countries [6].
MOOCs provide free access to premier courses taught by elite professors in an asynchronous format available anywhere and on any device with Internet access.Some suggest that this type of course delivery format can help to democratize education [7].Furthermore, MOOCs generally have no prerequisites, fees, formal accreditation, or predefined required level of participation [8].As such, registering for a MOOC is completely voluntary and is dependent on the interest of the individual.Researchers have found that the major reasons for individuals to sign up for MOOCs included the desire to learn about a new topic or to extend current knowledge, being curious about MOOCs, the need for a personal challenge, and the desire to collect as many completion certificates as possible [9].
As growing numbers of colleges and universities experimented with MOOC integration, diverse approaches regarding MOOCs adjustment, integration, and research were employed [10].Researcher advocated using flipped classrooms as a strategy for hybrid learning concerning MOOCs [11].Researchers and teachers have attempted to investigate MOOCs users' behaviors and satisfactions in multiple ways [12] [13].Several studies have applied quantitative methods with big data that learners left behind as digital traces [14].Although meaningful and significant insights were obtained related to users' online learning behaviors, this set of analytic methodologies focuses basically on the macro level of the entire learning community of a course, which is less suitable for understanding smaller clusters of users.
In addition, some researchers borrowed various scales and assessment schemes to better understand the impact and outcomes of MOOCs [15].These measures included the Computer Emotion Scale (CES) [16], Intrinsic Motivations Measure (IMM) [17] and System Usability Scale PAPER INTEGRATING MOOC AND FLIPPED CLASSROOM PRACTICE IN A TRADITIONAL UNDERGRADUATE COURSE: … (SUS) [18].Such measures provide a paradigm for exploring participation in MOOCs as part of a community at the institutional level.Nevertheless, research using more open-ended measures was noticeably lacking [19].
Since 2008, the numbers of MOOC providers, MOOC courses and articles, discussion groups, and blogs discussing MOOCs have, not too surprisingly, increased in a rapid pace.Most MOOC initiatives, however, are based on economic perspectives (such as scalability, productivity and being free) and technology perspectives (such as platforms, forums, peer-to-peer learning support, automatic grading systems).In fact, extremely few researchers have analyzed MOOCs from a learning science perspective [20].
A critical analysis of MOOC discourse throughout 2012-2013 highlighted broader societal struggles over education and digital technology.As such, those discussions capture a significant moment in time about the role of technology and openness in education before these debates subsides with the anticipated normalization and assimilation of MOOCs into educational practice [21].Some scholars acclaimed that researchers from the learning sciences should not only collect data about existing practices, they should also develop a vision about the future of education.Such a vision might emerge from explorations into open issues related to MOOCs and more open forms of education (e.g., assessment, accreditation, retention, instructional approaches, etc.) [22].

B. Flipped Classroom
The use of a flipped, or inverted, classroom model, is a pedagogical approach characterized by reversing the traditional role or expectation of teachers giving lectures in classroom and students doing homework at home and allowing students to watch online video lectures before class and to participate in interactive activities such as problem solving, discussions, and debates during the inclass sessions [23].In 2007, two high school chemistry teachers, Jonathan Bergmann and Aaron Sams, developed the flipped classroom as a means of providing athletes who missed class due to attending athletic competitions a chance to learn the unit or lesson.They reported that, after they flipped their classroom, students began interacting more in class and those students who were behind received more individual attention while advanced students continued to progress [24].
Advocators of flipped classroom approaches have pointed out that traditional lectures could be an effective way to help students acquire new knowledge, while the pace of instructors' lectures in the classroom sometimes causes problems [25].For some students, lectures' information comes too slowly or has no new information or content.At the same time, however, other students may feel that that same material is quite difficult to understand because they lack sufficient prior knowledge about the topic.At the same time, teachers often assign homework and ask students to complete it after school.Many students perceive it to be a quite onerous, frustrating, and confusing task.
The flipped classroom provides students opportunities to identify knowledge gaps needing clarification in an advanced time.During the in-class time, faculty interact with students by clarifying points of confusion, giving more individual guidance, challenging students to thinking deeply about complex processes, and monitoring peer-to-peer, team-based learning activities [26].Researchers argue that thoughts behind flipped classroom practice are consistent with both behavioral and constructivist learning theories [27].The pre-prepared lectures, tutorials, drills, demonstrations, and other forms of instructions are typically based on behavioral learning theory and they are mainly teacher-centered contents.In contrast, during the in-class sessions, students are exposed to many viewpoints and perspectives.In such settings, they are learning in a more socially-and constructivist-based environment with their classmates.Teachers are not the "sage on the stage," but instead collaborate with students to ensure mastery of essential course concepts.The flipped classroom model utilizes both theories by adhering to behaviorist principles outside the classroom and constructivist tenets within it [28].In-class learning activities that emphasizing problem solving and cooperative learning are the key components of flipped classroom practice.Some researchers further contend that when students work together to solve realworld problems, engagement, attention, and knowledge retention increase dramatically [29].
Some studies found that students perceived flipped classroom-based courses as having a positive impact on their learning [30].According to the Flipped Learning Network, membership on its social media site rose from 2,500 teachers in 2011 to 9,000 teachers in 2012.Some preliminary nonscientific data suggest that the flipped classroom may have benefits.In one survey of 453 teachers who flipped their classrooms, 67% reported increased test scores, with particular benefits for students in advanced placement classes and students with special needs [31].In addition, in this particular study, 80% percent reported improved students' attitudes and 99% indicated that they would flip their classroom again next year.However, some other studies indicated that there was no significant difference in student performance between flipped classrooms and traditional classrooms [32].To date, there's no scientific research base to indicate exactly how well flipped classroom models or approaches work.
To improve flipped classroom practice, Brame once proposed four design principles, which included: (1) providing an opportunity for students to gain first exposure prior to class; (2) providing an incentive for students to prepare for class; (3) providing a mechanism to assess student understanding; and (4) providing in-class activities that focus on higher-level cognitive activities [33].Lately, Kim et al. revised Brame's principles by changing the fourth principle's content into "providing clear connections between in-class and out-of-class activities" and by adding five more principles: (1) providing clearly defined and well-structured guidance; (2) providing enough time for students to carry out the assignments; (3) providing facilitation for building a learning community; (4) providing prompt/adaptive feedback on individual or group works; and (5) providing technologies familiar and easy to access [34].

II. PURPOSE AND OBJECTIVES
The purpose of this study was to redesign a traditional undergraduate course (named "Internet and Distance Education") by integrating MOOC content and flipped classroom practices and then exploring the effectiveness of this instructional approach through students' experiences and perceptions.Given that the course was Web-based, there was an opportunity for students to extensively practice The course aimed to introduce distance education theory and practice under an international perspective.Its major contents included the history and current status of distance education; distance learners and educators; instructional design and interaction in distance education; technologies and multimedia design in distance education; support and assessment and evaluation in distance education; and globalization and distance education.Since open educational resources (OER) and MOOCs are important trends in distance education arena, the course spent significant time introducing OER and MOOCs.

A. Course Format and Organization
Traditionally, the course was taught chapter by chapter by the instructor in a face-to-face classroom format.In the MOOC with integrated flipped classroom practices, the course was designed as a Web-based and blended course, which mainly included two parts.In the first part of the course (first five weeks), students were invited to take a MOOC named "Emerging Trends & Technologies in the Virtual K-12 Classroom."In the latter part of the course (last six weeks), students were invited to learn from chapter contents built into course using Sakai as well as complete an assignment in a proposal format.
For the first part of the course, during course design stage, the instructor and two teaching assistants searched the most popular MOOC search engines and portals to find a suitable MOOC for the study.Considering content relevance, time arrangement, and students' language proficiency, the instructor finally decided to select the course "Emerging Trends and Technologies in the Virtual K-12 Classroom," which was carried out during November.During the course, a flipped classroom practice was utilized.Brame's design principles were adopted as the guidance to design the flipped classroom practice.In effect, the students carried out Web-based self-study (WBSS) in the first lesson of each class.In addition, the instructor organized FTF discussion and guided the completion of student assignments during the second and third lessons in each class.
In the first lesson, students took Coursera course or reviewing online OER resources at their own pace using their laptops.It is important to also note that the instructor and two teaching assistants stayed in the classroom for consultation and observational purposes.In the second lesson, the instructor and two teaching assistants typically arranged discussions.For some weeks, the discussion was topic-specific, such as "sharing and discussing MOOC's peer assessment assignment."For a few other weeks, the instructor and two teaching assistants simply answered every student's specific questions.In the third lesson, students mainly focused on doing course assignments, which include MOOC course assignments, course quizzes, and writing a learning diary using the Sakai platform.Figure 1 recaps the course timeline and major activities.

B. Course Evaluation Design
The instructor employed a continuous evaluation method to assess students' overall performance in the course.Multiple assignments were designed to allow students to express their thoughts and understandings about what they learned from the course.As Figure 2 makes evident, to successfully complete the course at a high level of performance, students needed to complete the following assignments: (1) obtain MOOC experience (e.g., finish Coursera course, obtain a course certificate, and write an online diary); (2) complete a comparative study and proposal; (3) finish the course quizzes; and (4) actively participate in the course, which meant engaging in the course discussions, group activities, and the online diary.The second component entailed using a comparative study about one selected national and international OER project, students had to generate a proposal for the Chinese MOE or ZJU about Chinese OER development and application.As part of this effort, they had to make a presentation about the proposal.At the beginning of the course, the table describing the course assignments and criteria was presented to the students.

A. Participants
Participants in this study include one instructor, two teaching assistants, and 15 undergraduate students taking the "Internet and Distance Education" at Zhejiang University, China, during the winter semester of 2013.During the first week, 21 students participated in the class and all of them took the pre-test.In the second week, when students needed to decide whether or not they actually choose the course, 16 students continued with the course.During the course time, one student experienced a longtime illness, and, therefore, was unable to attend most of course activities, including the post-test.

B. Data Collection Instrument
The study employed multiple ways to collect data, including a pre-test instrument, post-test instrument, students' online diary, and classroom observation.
The pre-test mainly included 24 statements, which were part of the e-learning satisfaction survey instrument [35], were related to students' attitudes toward computers and the Internet.They were categorized into three dimensions: (1) learner attitude toward computers, measured by seven items and sample statement was "I believe that working with computers is very difficult;" (2) learner computer anxiety, measured by four items and sample statement was "Working with a computer would make me very nervous;" and (3) learner Internet self-efficacy, measured by thirteen items and sample statement was "I feel confident starting the Internet program."A 7-point Likert scale was utilized to indicate student attitudes toward these statements.
The post-test was about e-learning satisfaction.Sun et al.'s e-learning satisfaction survey instrument was adopted and modified (deleting reverse worded statements) as a major instrument to measure students' perceptions about MOOC-integrated flipped classroom practice in the course.Table 1 shows the initial source of the instruments, number of related statements, and reliability of each item involved in the survey instrument.General perceived e-learner satisfaction 6 0.910 [41] Students' online diary was designed as part of student's performance evaluation content.On the Sakai platform, blogs were created for every student.At the end of each course section (three lessons), students were required to write online diary in their individual blog space.Importantly, they could type their diary in Chinese or in English.The instructor did not give a word limit for each student's diary post.Once logged in the platform, the instructor, teaching assistants, and peers could review and make comments over every student's diary contents.Content analyses were utilized to analyze participating students' diaries.
The instructor and two teaching assistants employed classroom observation skill to record students' overall performance and possible problems in the students' daily classroom activities.The major contents on which classroom observations focused included: (1) the degree to which students' participation in the flipped classroom practice; (2) students' willingness to and frequencies of interaction with their classmates; and (3) students' willingness to and frequencies of interaction with instructor or teaching assistants.

C. Data Collection and Analysis
Data collection was conducted at the beginning and at the end of the course.A pre-test was carried out on November 14, 2013.Twenty-one ZJU students who attended the first class of the course took the pre-tests.A post-test was carried out with 15 students in January 2014.In the first class, the instructor introduced the nature and purpose of the experimental instructional activity.
The collected data were compiled and analyzed using the SPSS software to answer the research questions with an alpha value set at .05.Descriptive statistics were used to describe each variable.

V. FINDINGS
As to Objective 1 (describing participating students' perceptions about computers and Internet before and after course), in the pre-and post-class surveys, the teacher asked students to indicate their perceptions about computers and Internet in three aspects: (1) learner attitude toward computers, (2) learner computer anxiety, and (3) learner Internet self-efficacy.Table 2 shows the results of a T-test between pre-and post-tests' outcomes.It indicates that learner attitudes toward computers and learner Internet self-efficacy improved statistically significantly after taking the course (p 0.5 : (1) learner attitude toward computers, t (34)=2.76 and (2) learner Internet selfefficacy, t (34)=3.10,respectively.through an e-learning satisfaction after the course), as detailed in Table 3, participating students' e-learning satisfaction was generally at a high level (M=5.33,SD=0.88).Students self-rated themselves high in most of ten sub-dimensions of e-learning satisfaction, especially in such dimensions as diversity in assessment (M=6.00,SD=0.76),Internet quality (M=5.53,SD=0.64), technology quality (M=5.47,SD=0.59), instructor attitude toward the technology (M=5.33,SD=1.18), perceived ease of use (M=5.28,SD=0.76), perceived usefulness (M=5.08,SD=0.89), and instructor response timeliness (M=5.07,SD=1.34).Comparatively, students rated a little lower in such dimensions as e-learning course quality (M=4.80,SD=1.15), e-learning course flexibility (M=4.91,SD=0.93), and learner perceived interaction with others (M=4.97,SD=0.75).In terms of Objective 3 (describing participating instructor and two teaching assistants' observation about students' experience in MOOC-integrated flipped classroom), the instructor and two teaching assistants' classroom observations found that students' overall participation in the flipped classroom practice was quite active.In the first lesson of each class, most of students devoted themselves into browsing course websites (on Coursera or Sakai platform), watching course videos, or reviewing other kinds of online instructional resources.They did these activities mainly by themselves and with earphones on.In the second class, which was designed for discussing questions proposed by students, the instructor and two teaching assistants found that students tended to be too shy to express their questions.Most students seemed to wait for the instructor to propose questions for them to answer.Such a result might due to the fact that they were accustomed to being passive receivers of information in traditional courses and, therefore, they were not used to being active question-raisers.
As a result, instructors deployed two strategies to potentially enhance the situation.One strategy was to ask randomly selected students to share with their classmates the most important/interesting/difficult points they had learned in the first lesson.Next, based on such students' shared responses, the instructor added additional explanations regarding these points and asked other students to provide comments and feedback on them.The second strategy was to ask students to share with their peers their concerns or obscured points when overviewing course materials or doing their assignments.
Through such guided questions and answers (Q&A), students expressed their questions and concerns.In the third lesson, which was mainly designed for students to complete their assignments, students indicated a strong willingness to interact with the instructor or her teaching assistants.If they had difficulties, they would raise their hand and the instructor or one of the teaching assistants would offer assistance.At the same time, students would interact with the instructor and her teaching assistant through Q&A or a blog post on the Sakai platform.Among the 283 postings, 52 entailed feedback and comments from the instructor and the teaching assistants.
In regards to student-to-student interaction, the observers found that, during the classes, students would interact with the classmate who sat right besides them.They normally communicated in very low voices.It was rare for students to communicate with peers sitting far from them, except when raising questions during various student presentations.

VI. CONCLUSIONS AND SUGGESTIONS
This study was carried out an experimental MOOCintegrated flipped classroom practice in a Chinese undergraduate course.Although the main objectives of the study were reached, there, nonetheless, were several key limitations to mention.Due to time and other contextual limitations, the study only explored a small group of Chinese college students majoring in the field of education in a comprehensive university.Given the fact that China currently has about 3,000 registered higher education institutions with wide variations in goals, students, majors, and levels and types of courses, the results do not provide a complete picture of Chinese college students' reactions to MOOC-integrated flipped classroom practices.Further research should be carried out to explore college students' perceptions and usages of MOOC and flipped classroom in larger and more diverse populations or among some specific groups or types of institutions, such as students from research-oriented universities, normal universities, community colleges, and vocational colleges.Such research should also look at students majoring in different disciplines (liberal and arts, social science, science, technology, engineering, etc.).
As to the measurement tools employed in the study, there were internal limitations to point out.For example, other factors such as personality traits, cognitive styles, and motivation might provide additional insights into the benefits of the flipped classroom and participant changes over time.As to the e-learning satisfaction survey instrument, although it was tested in other studies, it was prepared for a typical e-learning course and it did not take into account the characteristics of a flipped classroom.Therefore, an instrument specifically considering flipped classroom practice might be needed to reflect more authentically and effectively students' actual perceptions about their experiences.
Before the class, participating students' attitudes toward computers were at a moderate level.Results of a t-test between pre-and post-course surveys indicated that learner attitudes toward computers and learner Internet selfefficacy significantly improved after taking the MOOC-PAPER INTEGRATING MOOC AND FLIPPED CLASSROOM PRACTICE IN A TRADITIONAL UNDERGRADUATE COURSE: … integrated flipped course.Compared with traditional faceto-face classroom experience, this redesigned course was emphasized students' usage of online technologies and activities, which forced students to practice their computer skills.Therefore, it is understandable that students' attitudes toward computer technology became more favorable and their self-efficacies related to the Internet in general were increased.
Participating students' perceived level of e-learning satisfaction was generally at a high level.Students rated high in such dimensions as diversity in assessment, Internet quality, technology quality, instructor attitude toward the technology, perceived ease of use, perceived usefulness, and instructor response timeliness.Comparatively, students rated a little lower other dimensions such as elearning course quality, e-learning course flexibility, and learner perceived interactions with others.Such findings are partly supported by observations by the instructor and two assistants' in the class.Students felt happy that they had freedom and autonomy to lead their own learning process.At the same time, timely person-to-person guidance and Q&A from the instructor and two teaching assistants helped students feel that their personal questions and concerns were important.Instructor-student interaction is vital to students' flipped classroom experiences and attitudes.Comparatively, student-student interactions were lower than what were expected.It seems that students were not used to proposing their questions and concerns and communicating with each other publicly.With facilitation from the instructor during Q&A sessions, students indicated greater willingness to participate in peer-to-peer interactions.
Based on the findings, we propose some suggestions for improving MOOC-integrated flipped classroom practices in institutions of higher education.Firstly, course design should be based on comprehensive learner analysis and be more learner-centered overall.The more the target learners are understood, the better the instructional design of the course can meet their needs.For instance, instructors may perform more competence analyses through comprehensive pre-study surveys and formal as well as informal interviews with the target learners.Secondly, in a flipped classroom--where students have more autonomy in their own pace of learning--the instructor should pay more attention to interaction design, including the types and forms of face-to-face and online interaction.Thirdly, the instructors should prepare optional plans and strategies when unexpected situations emerge during flipped classroom practice.If, as expected, students lack skills for selfdirected and independent learning or they are not active in discussions or Q&A sessions, the instructors may need to adjust their predesigned activities with alternative strategies that could engage more passive students.Finally, more specific measurement tools are needed to measure the effectiveness of such experimental instructional activities.
PAPER INTEGRATING MOOC AND FLIPPED CLASSROOM PRACTICE IN A TRADITIONAL UNDERGRADUATE COURSE: … their computer and Internet skills during the course.As a result, this study explores the possible changes in students' perceptions about computers and the Internet after the use of a MOOC-integrated flipped classroom course.Specifically, the study had four primary objectives: (1) better understand participating students' perceptions about computers and the Internet before and after course; (2) examine students' perceptions about the MOOCintegrated flipped classroom practice through an elearning satisfaction survey after the course; and (3) document participating instructor and two teaching assistants' observations about students' experience in a MOOCintegrated flipped classroom.III.INSTRUCTIONAL DESIGN The course "Internet and Distance Education" was a 2credit course and it was taught in Winter Semester, 2013, at the Zhejiang University (ZJU) in Hangzhou, China.It was taught twice per week with three lessons each time (Tuesday and Thursday, from 9:50 am to 12:15 pm).
11, 2013 to December 13, 2013 and was taught by Melissa Joell Loble from the University of California, Irvine.For the latter part of the course, the instructor set up a Sakaibased online learning platform (http://ocw.zju.edu.cn/portal).Each chapter's content was videotaped in advance and put on the platform.Another key step involved the collection of OER-related resources and academic publications in the learning platform.

Figure 1 .
Figure 1.Course timeline and major activity

Figure 2 .
Figure 2. Course assignments and criteria IV.METHODS This research employed both quantitative and qualitative methodologies to collect data.Pre-test, post-tests, and content analysis of students' diaries were utilized to measure the effectiveness of the course.Descriptive statistics were used to describe the results.

TABLE I .
RELIABILITIES FOR ITEMS IN THE E-LEARNING SATISFACTION SURVEY INSTRUMENT

TABLE II .
PARTICIPATING STUDENTS' PERCEPTIONS ABOUT COMPUTERS AND INTERNET IN PRE-AND POST-TEST Volume 10, Issue 6, 2015 PAPER INTEGRATING MOOC AND FLIPPED CLASSROOM PRACTICE IN A TRADITIONAL UNDERGRADUATE COURSE: …