What are the Latent Topics Associated with Experience in Flipped Learning? An Empirical Investigation of University Students' Interview Responses

With the advancement of ICT in the 21st century, ICT adopted education environment and teaching approaches are introduced in higher education setting globally. In the year of 2020 when global education sector encountered unprecedent difficult phenomena in Covid-19 Pandemic, teachers, students, and education administrators had to familiar with the terminology of online platform, hybrid learning and Flipped Learning (FL). With this strong intervention of ICT into the Education setting, this study aims to explore the students’ experience and perception toward the FL by analyzing and interpreting interview data from the qualitative studies on FL between 2014 and 2020. This study conducted text mining analysis, and topic modeling method from the selected 102 SSCI and Scopus level research articles on FL. The result of the study categorized the findings from the pure text analysis into three latent topics for the FL which were ‘teacher and classroom’, ‘motivation and students’ growth’, ‘educational needs and difficulties. From the result, the study confirmed the other two topics which are related to student’ experience and characteristics such as growth of motivation, practical challenges, and difficulties from the FL experience. The result indicates that teachers need to give attentive attention and observation to the challenges that students are encountering during the classroom activities in FL setting. Keywords—flipped learning, university student, teacher, classroom, educational need


Introduction
The Flipped Learning (FL) is getting more attention in higher education setting these days [1,2]. The FL converted old-fashioned pedagogy, which means that deliver direct classroom activity out of the class and devote more time to solve the problems in class or project-based learning [3]. In this environment, the teacher's role becomes a learning assistant rather than an instructor [3]. Thus, the FL offers more student centered environment with practical activities during the class. The benefits of the FL include increased motivation, self-initiative learning, master learning, collaborative learning and prompt response through formative assessment [4,5]. More and more higher education institutions have adopted the FL in order to offer more chances of peer learning and material engagement [6]. It is necessary to explore perception and needs of the students who have experienced FL whether they have experienced these positive functions and features of FL. This is because there may be a difference in perception between the educational subject and the learner, or the difference may be large [7]. Online classes were greatly enforced due to the Covid-19 pandemic in 2020 and cases of college students having online classes including Fl are increased.
Understanding university students' responses, including students' perceptions and attitudes towards the quality of instruction, have been employed to enhance a quality of the instructor's lectures [8,9]. These students' interview responses through a qualitative research also encourage the instructors to create an expertise of instruction and facilitate administrative evaluations for promotion and tenure positions [10,11]. In addition, university learners' interviews can help future learners select courses and assist university administration in making investment decisions about the future learning environment [12,13]. Even though students' experience on flipped learning is getting more importance, there are not enough studies conducted to analyze students' interview data through qualitative research methods. An empirical investigation of university learner's interviews will help better understanding in the aspect of university student's learning by flipped learning. The text mining has been adopted to analyze 102 research articles. Six meaningful and latent topics have derived which is helpful for the successful implementation of FL in higher education.

Theoretical framework
Flipped learning is defined as "a pedagogical approach in which direct instruction moves from the group learning space to the individual learning space, and the resulting group space is transformed into a dynamic, interactive learning environment where the educator guides students as they apply concepts and engage creatively in the subject matter" [14] It is clearly stated that Flipped Learning (FL) is different from the Flipped Classroom (FC). Many educators may already flip their classes by having students read text outside of class, watch additional videos, or solve extra problems, but to critically engage in FL, educators must incorporate four pillars into their FL practices: 1) Flexible Environment, 2) Learning Culture, 3) Intentional Content, and 4) Professional Educator [14].

Flexible environment
FL is a type of blended learning that has the potential to foster deep learning [15]. There has been increasing literature to seek a best FL examples in the university setting. Moving the passive learning activities of students out of the classroom allows the inclass time to be used for further collaborative exploration of the instructional material and shifts the focus of attention from the teacher to the student [16]. When combined with blended learning technologies [17], FL has been shown to support increased learner independence and acceptance of responsibility for learning [18]. Engaging students in active learning tasks in the FL is understood to lead to learning and is known to be effective [19].
In this regard, educators need to create a flexible learning environment in terms of time and location. Also, educators who flip their classes are flexible in regards of timelines of study and their assessment. Many studies have pointed out that the flexible learning environment is the key to success for effective FL implementation [20,16,21].

Learning culture
Many studies have shown that a positive learning culture can contribute to successful FL implementation [22,23,24]. It is reported that students can engage more in constructive criticism and problem-solving oriented activities as well as instructors can consider more on students' perspective [25,26]. These meaningful learning activities could be found in non-traditional classrooms [25,26].
A learning culture in FL allowed educators to move content delivery out of the class by offering more time and chances to students to cultivate 21st century skills such as critical thinking, creativity, communication and collaboration [22,23,24]. The FL can also be useful for the large size class [20,16,21]. Also, the FL encourages problembased learning in which students developed different ways of handling classroom problems [22,23,24]. Usually, less formal engagement in the class helps students understanding to develop better relationship between the students and the instructors [25,27].
In this regard, in-class time is dedicated to exploring topics in greater depth and creating rich learning opportunities in an FL. As a result, students are actively involved in knowledge construction as they participate in and evaluate their learning in a manner that is personally meaningful.

Intentional content
Many FL advocates emphasize the student's preparation, for instance, in the form of a quiz before or at the beginning of class [28]. Such mechanisms can serve as motivators for student's learning process [28]. It is documented that the international content in the FL can be useful for more interaction in the classroom with student-centered activities for the students' understanding of the class. In result, students will get better grades than traditional classroom environments [1,2].
According to studies comparing the impact of intentional content in the FL and traditional class, study found that FL students have better semester projects and final grades by demonstrating the knowledge, conceptual understanding. Most FL students satisfied with the new approach of FL [4,5]. Additionally, the students' attitude towards the FL became increasingly positive over the course of the semesters, which helped them to be better prepared for future classes [29]. The FL has also been shown to enhance the students' task value and peer learning, enabling students to make gains at twice the rate of their peers in non-flipped classes [26]. Thus, educators need to determine what they need to teach and what materials students should handle independently.
In other words, educators maximize intentional content in the FL to maximize classroom time in order to implement methods of student-centered, active learning strategies, depending on grade level and subject matter.

Professional educator
The role of a professional educator is more important and often more demanding in an FL than in a traditional course. During in-class time, the professional educators need to observe students, providing them with instant feedback and an assessment of students' assignments and projects. While professional educators might take on less prominent roles in the classroom, they remain the essential part that enables FL to occur successfully [3]. Tomas et al. [36] argued that faculty favored the FL in a higher education setting because it increased student engagement and reinforced essential skills for real-world application [30,31]. Thus, the professional educators in the FL can positively impact students' acceptance and favorable perceptions of flipped classrooms. In this regard, it is believed that training students to transition from passive learners to active learners can promote success in the flipped classrooms [6]. To achieve this goal, the role of a professional educator who understands the FL plays a major role in the classroom [6].

3
Research method

Data collection
In order to ensure that relevant studies were located, a wide variety of databases (ERIC, ProQuest, EBSCO, and Google Scholar) were searched. The keywords searched in all the databases included "university students" learning experiences in the flipped learning". To narrow down topics that were reverent to our research goal, "university students' interviews" or "qualitative research method in the FL" were searched. Since the terminology, the FL was first introduced to the previous literature in the field after 2013 by Bergmann and Sam [32]; thus, we searched the database from 2014 to 2020. The following inclusion criteria were applied: (a) empirical research on the flipped learning approach in a university setting; (b) description of the flipped course design; and (c) publication in SSCI and Scopus level journals. As a result, a total of 543 interview entries from university students were collected 102 SSCI and Scopus level articles on FL. Similar words and signs were not included from the list of entries. Meaningful entries which met predesigned criteria were retained from the list. The results are shown in Table 1 and Figure 1.
The database has been shown to have grown largely through 2018. All articles of flipped learning have increased significantly until recently, but the students' perception dealt in this study has declined slightly since 2019. This is interpreted as diversifying research on flipped learning.

Processing model
The processing model for this study has been categorized into four stages to analyze the data (Figure 2). Step 1 was to select optimized articles to collect university students' experiences and opinions on FL. Steps for a processing model.
Step 2 was the text cleaning and data generalization, so that core concepts and topics appropriate for the study purpose can be derived from 102 articles. For this, a keyword frequency analysis was performed to primarily explore the university students' opinions mentioned in FL articles, and Term Frequency (TF) was calculated using the R program for the main word frequency analysis. In order to generate more useful and related results, data generalization has been adjusted by word processing manuals and conceptual root setting. This study implemented units of coding as sentence text blocks. Also, low semantic information was not included (e.g., of, and) and specific words (e.g., HyLearn) from the data to ensure general concept. Those words which does not influence the root selection such as highly-frequent and weak-semantic words have been set as for the configuration.
After the data generalization, as step 3, 'topic modeling' was conducted to extract the main issues of university students' experiences in FL. Topic modeling is an analysis method that statistically analyzes the frequency of words in text data and automatically extracts potential topics that penetrate the entire data [33]. This study utilized Latent Dirichlet Allocation (LDA), which complemented the previous Probabilistic Latent Semantic Analysis (PLSA) to solve the overfitting problem [34]. As for the number of topics, the decreasing complexity trend was confirmed through cross-validation, and the number of 6 topics that were meaningfully appeared was set. The study used reiterative inspecting and modifying approaches until relative and meaningful topic is generated.
In step 4, based on the 25 words extracted for each topic, the topic was characterized in the context of university FL identified in previous studies. In this process, prior studies and coding rules were reflected and checked, and the opinions of the authors and external experts were applied to improve the validity of data generalization. Basically, statistics program can guide to an initial point of analyzing large volume of data without replacing researcher's subjective idea, understanding and explanation. [35]. In addition, for words with a probability in a high frequency within each topic, and a high rate of words being classified as a specific topic, 25 words were selected for each topic. Next, the words used in the analysis were corrected to the original words (e.g., don't), and the topics were rearranged in the order of the highest significance (%).

Results
The 66 concepts have been generated through the text analysis. Figure 3 provides latent network structure into conceptual connectivity and represent number of co-occurring events between one concept to another. the concepts are grouped into six topics: '1: Educators', '5: Environment', '3: Students' Participation', '4: Educational Needs', '6: Questions and Feedback', and '2: Students Growth'. Each topic has a group of texts based on commonality, which is derived from the proximity on the topic distance map. In the rest of this section, the study examined each topic in greater detail, as shown in Table 2.

Fig. 3. Distance map between topics
Topic 'Educator (1)' was identified as the most significant topic, affirming that FL in university is designed to enable and facilitate student understanding beyond traditional methods. This topic was the most related concepts which explains the key features of FL in university (e.g., 'understand', 'video', 'information'). The topic indicates that individuals have understood what is being learned through FL and learned information from video materials. Below are illustrative examples for this topic: "We need to think about how to explain the content in our own words… (because of group discussion) the next time I watch videos, I will try harder to make sure that I understand the video." "

I prepared the topic being discussed in the classroom through the multimedia resources in the form of the video which is shared by the teachers in LMS. I could also watch the video when I wanted, when I slept on my bed or when I woke up in the morning, and also watch the video many times when I did not understand."
Also, there are important conceptual texts within the topic that merit attention: 'traditional' and 'different'. These texts indicate that FL afforded new and different lecture that went beyond traditional teaching methods: "It's not traditional and it kind of breaks up the boredom. You can actually learn and any questions they will go right into I mean they will explain it. I mean they will actually take the time to explain what you do not understand...and I do like that." "Most satisfying was the 'before class' section. I believe learning is something different than just information given in class. This style provides students with an opportunity to learn as much as they want before they come to class -if they are willing." The second important topic is 'Environment (5)'. The topic contains two layers of meaning: (a) FL creates a more enjoyable and interesting environment (e.g. 'enjoy', 'interest', 'better'), and (b) students tried to comprehend the lecture through discussion opportunities and answers from the instructor (e.g. 'discussion', 'answer', 'opportunity'). Below are illustrative examples for this topic: "This type of teaching made the course lively, interesting, and fun. We thoroughly enjoyed this FL, which enabled us to learn while having fun." "During the online learning process, I feel relaxed, and I am not afraid of not being able to answer the questions, as we (teachers and students) are not face to face, I will not be afraid of being laughed at for my mistakes. " "I greatly enjoyed the quizzes we took as groups. We determined the correct answers by discussing amongst ourselves. The other groups did the same also. Being a part of a team made me feel good. It was very fun." In addition, university students positively reacted to teaching approaches and methods chosen by teacher to make the content more interested. The following excerpts illustrate how the use of question and the answer made flipped learning interesting:

"Sometimes, I get confused and not understand the video on the LMS, but with the question I post there, my friends and teacher can answer and explain me clearly."
"For example, the questions and feedback in the video prepared for the Communication week were very useful. When I had difficulty understanding some points, I was able to look them up on the Internet. If there were questions that were still unanswered, I freely asked them to the teacher in class." The third important topic was 'Student participation (3)'. This topic was associated with the involvement of students being active and motivating towards FL (e.g., 'activity', 'motivate') and the sources of these responses. The proximity of '3' to topic of 'activity' recommends that students were easy to participate and be active in learning more actively if students thought the FL to be interesting. Below are illustrative examples for this topic: "The supportive atmosphere of flipped approach motivated most students to confidently ask questions, debate and support other points of view inside and outside the class." "The teacher role in flipped classroom motivates the students that they can learn on their own and also it helps them to be autonomous learners." Regarding to the topic, there were students' experiences and opinions that pre-learning online content induces better participation.
"Using the video lessons outside the class enabled me to better understand the subject." "I think online learning is a great way to learn. This is especially true when you are able to research and further find out theories and information that you are interested or unsure of. Through online platform learning, it allows me to gain a better insight of this module." "I think this application is more than excellent because it gives us the chance to watch the recorded lecture at home more than once. This definitely helps us to understand it better and also, it makes it easy for us to have an idea about the next lecture so that we can be prepared for it." Topic 'Educational need (4)' was identified as a fourth important topic. This topic was related to the problems and difficulties experienced by students in flipped learning. (e.g. 'didn't', 'difficult'). It was their first experience with flip learning, and it has to do with adaptation. Below are illustrative examples for this topic: "The way the scientific research methods course (FL) was delivered was the sort that we had never experienced before, so we had difficulty adapting to it." "I do not have enough technological equipment now; the undergraduate education did not provide it to me. I may face difficulties. These are the things that need experience, and some unplanned issues can occur at the moment. We also need to receive in-service training." "At the beginning, it was difficult to adapt to the system. When I did not watch the video before coming to class, it was at first difficult to solve problems. I needed help. But it was impossible for the lecturer to conclude the entire topic just for me. However, sometimes I could ask my classmate and get feedback from them. I did not like watching at home. I think that lecture should be done in class. The lecturer explains, and I take notes. If there is a problem, one can ask at any time. No need for video …" "Some of the students, including myself came to the class without reading the book (i.e. the pdf uploaded to the learning management system). This caused a problem for covering that week's subject, and our instructor also indicated our indifference to the tasks." Within the topic 'Questions and Feedback (6)' was related to the experiences of solving difficulties in FL through friends. Problems were solved by sharing important information with friends. "I think the problem is that some parts already have answers, but others are lacking them. For example, we recorded some audio there and wrote descriptive answers that were not corrected yet, and it's been a while... The activity we recorded, we are waiting for their feedback and there is none, and in the test the activities and pronunciation are not corrected either. I have not accessed the website very much since I realized that the speech recordings are not corrected or that it takes longer than expected." The sixth topic was labeled as 'Student growth (2)'. This topic incorporates concepts that describe the quality of FL and lecture components relevant to practice. The proximity of the topic to 'communicate', 'discussion', and 'example' recommends FL is recommendable in helping students solve problems and increasing learning outcomes. "It allowed me to identify what I need help with and knowing that I can get that specific help in class has been a positive thing." "Furthermore, they stated that the role they played as teachers is another reason for their positive perceptions towards the use of FL. Students indicated that the role played, as well as the role of the teachers being facilitators and mentors, helped them to be more confident and active rather than being passive learners."

Discussions and implications
This study explored university students' learning experience in flipped learning with the chosen method of text mining approach to the collected 102 research articles. And we found six latent topics which were 'educators', environment', 'students' participation', 'educational needs', 'questions and feedback' and 'student growth'. While this study confirmed four pillars proposed in Flip Learning Network in 2014 [14]: flexible environment, learning culture, intentional contents and professional educators, the study added more specific classroom activity 'questions and feedback' for better FL as well as 'student growth' as FL's promising educational outcome.
In summary, Table 3 displays the six latent topics expressed by university students with flipped learning in their various ratings and reviews. The six topics could be structured into three theme sections. The sections also highlight the characteristics of each topic, including the topic scope and connections between topics. The topic scope of 'Teacher and Classroom' is about FL as an external factor around the student and alternative teaching approach to the existing traditional method. This study confirmed the emphasis on the fun and exciting elements to classroom activities in addition to a systematic and flexible environment. In general, the learning culture required in FL setting has been perceived as a positive culture, but students' perception of positive culture is more focusing on the elements of 'fun and excitement'. These findings reinforced that teacher might overlook the students' actual needs in FL setting and whether the teachers are prepared enough to create a fun and exciting culture over the FL teaching approach.
From the result, the study confirmed the other two topics which are related to student' experience and characteristics such as 'Motivation', 'Growth', 'Practical needs', and 'Difficulties' from the FL experience. The result indicates that teachers need to give special attention to the challenges that students are encountering during the classroom activities in FL setting. It is more difficult for teachers to monitor students who needs more attention in the process of solving tasks with their peers during the FL class. Teacher needs not only to provide an individual feedback to the student but also to oversee the entire classroom culture and environment in the FL setting. The findings of this study, which show that college students participating in FL class around the world have similar difficulties, suggests that there is a need to develop a practical alternative approach to respond to the actual needs from the students.
Also, the study's findings above are closely linked with the current global pandemic as well as four pillars of FL [14]. Especially, the year of 2020, the worldwide Covid-19 pandemic brought highly disruptive innovative approach to higher education around the globe, forcing institutions, teachers and administrators to make rapid choses about how to continue to educate students while remaining physically separated. It enhanced concept of four pillars of FL in 'flexible environment' and 'learning culture'. This unusual situation enabled an experiment to take place in which many faculty members implemented a FL which is also very closely linked with the concept of 'professional educators' from the four pillars of FL. This pandemic also expedites the adoption of FL in various ways such as taking advantage of internet/video resources such as Massive Open Online Courses (MOOCs). To certain extent, using the open online contents is closely related to the concept of intentional contents. Again, the pandemic created an unintended but natural environment for adopting a FL not only for higher education but also K-12 setting.
While this study confirms and expanded the current existing knowledge and research body of FL, it has several limitations. The study found the three themes and six latent topics that described above from by pure analysis of the text from the university students' interview data of the selected articles in FL, the findings can only suggest other researchers to adopt this theme to analyze the further in-depth study in FL. However, the study also found out that FL does not only create the positive outcomes but also difficulties to both instructors and students since they implement this new pedagogy for the first time. The FL approach also suggest the instructor's role in future education is changing from the agent of knowledge transfer to facilitator or coordinator of students' learning. It suggests to teacher training institutes and relevant ministries for government that they need to develop and design an in-service training program for the instructors to be more professional and competent to adopt FL into their teaching as well as counseling and facilitating skill set.