Paper—Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning Use of a Mobile Anonymous Question-Raising System to Assist Flipped-Classroom Learning

Flipped classroom emphasizes the following. First, the instructor uses a digital platform to store course contents, including video lectures. Then, students retrieve these contents from the platform and learn by themselves before class. Finally, the instructor helps the students acquire their knowledge by asking questions in class, thereby helping the students to attain their learning goals. In previous studies, it was found that students might be affected by their peers and their own personalities, causing them not to express their opinions and views; this in turn might cause the students not to learn well; in some extreme cases, some students might even give up entirely. To solve the problem that students may hesitate to ask questions in class, this research developed a mobile anonymous question-raising system (MAQ) and used it to help students raise questions. By conducting an educational experiment, it was found that the use of MAQ in class could indeed help students to learn better in a flipped classroom setting. Keywords—Anonymous question-raising, blended learning, flipped classroom, problem-based learning.


Introduction
Researchers have pointed out that combining classroom learning with online learning can bring a positive effect on the learning motivation of students [3] [9] [26]. Flipped classroom is a more recent teaching strategy. Compared to traditional didactic teaching, flipped classroom is versatile in that a variety of teaching strategies can be used in conjunction with digital learning. Students first use a digital platform to study course materials online and gain an understanding of both course contents and reasons why they need to know such contents. In the process, they also identify what they don't understand well. The instructor then uses questions in class to guide students in their learning. This forces the students to reflect on their understanding of the topics and make the necessary adjustments. As a result, students are able to develop independent ways of thinking and improve their problem-solving abilities. By studying course materials before class and finding acceptable answers to questions (raised by the instructor and other students) in class, students are able to gain a deeper understanding of course materials [2] [14] [15] [24] [25].
Even though flipped classroom can help students understand more about course contents and improve the interactions between the instructor and students, not all students can benefit from the use of this teaching strategy. As some researchers suggest, if students do not interact well with the instructor in class, it is likely that they do not learn well. The use of flipped classroom is based on the premise that peers are met face to face [29]. This means that everyone knows what everyone else is asking. Therefore, students may be influenced by peer pressure and their own personalities, causing them to compare with one another and also have the fear of losing face. Consequently, they may not dare to ask questions owing to cognitive pressures such as fear, disgust and worrisome [8] [17].
Therefore, researchers have developed mobile learning systems for students to interact with their instructors in class; this includes the use of texting and voting to ask or answer questions [23]. It was found that the use of mobile devices to communicate with instructors could not only improve learning motivation but also lower learning pressure [20]. Therefore, it is not hard to see that with a good teaching strategy, mobile devices can be a very good assisted learning tool to use [19]. However, the effect of just answering questions from the instructor or using texting to send questions to the instructor may be limited. If there is a way to help students conduct peer discussions in an anonymous way, it can help to lower the fear of peers [16]. In addition, anonymous discussions can help students to find answers to questions and also improve their learning efficiency [1].
Based on the above discussions, this research investigates the effects of using a mobile, anonymous question-raising system (MAQ) for in-class learning assistance. This system was developed to help students asking questions in class, but do so in an anonymous way. The goal was to find out whether such a system is able to solve the problem of students not asking questions in class due to shyness or peer pressure. In other words, the goal was to investigate whether the use of a mobile system to ask questions in an anonymous way can effectively encourage students to raise their questions without having to care too much about peer pressure and personalities.
To further enhance the students' impression of course contents, questions raised, and answers to these questions, the system used in this research also includes functionalities for reviewing question-answering records. Students can review course contents, questions raised, and answers to these questions after attending classes, and they can also do so before quizzes and exams. Not only can each student review the questions he/she himself/herself has asked in class and the answers to those questions, the student can also review what others have asked in class and what the answers to those other questions were. The hope was that by going through these question-answering records, a better learning effect is achieved.
This research aimed at finding answers to the following three research questions:  Is there a significant difference in learning achievement between the experimental group, which uses MAQ in class, and the control group, which does not use MAQ in class?
 For those who do not belong to the experimental group but nevertheless attend the same classes as members of the experimental group, will they be affected in their learning achievement?  Will those who do not belong to the experimental group but nevertheless attend the same classes as members of the experimental group feel somewhat being discriminated upon or isolated? 2 Related Works

Problem-based learning
In 1965, McMaster University Medical School, newly established in Canada, adopted a new curriculum called problem-based learning for teaching and evaluation [7]. There are several characteristics with problem-based learning [7].
Learning is student-centered: Each student has to be responsible for his/her own learning, and he/she must also understand and manage the knowledge he/she needs and the questions he/she has.
Learning occurs in small student groups: In the early days, problem-based learning was such that students were randomly assigned into groups of five to eight; upon the completion of each course unit, the students were regrouped. This was intended to help students to work in a serious and more effective way when they work with different people.
Teachers are facilitators or guides: When the instructor discusses with students, he/she asks various kinds of questions without directly giving them answers. In other words, the instructor serves only as a guide on the side and helps students to reflect on their own thoughts and manage their own problems.
Problems for the organizing focus and stimulus for learning: To put group discussions in focus and to stimulate thinking, the instructor can give appropriate questions for a group to find answers to.
Problems are a vehicle for the development of clinical problem-solving skills: To develop clinical problem-solving abilities, one has to keep on solving problems. New information is acquired through self-directed learning: Students should actively acquire knowledge and professional ability by themselves. They need to work with their peers; in so doing, they should also discuss about and review what they have learned.
Problem-based learning not only helps to establish students' learning motivation but also help them to improve in their active learning ability and their ability to comprehend knowledge. Through the use of problem-based learning, students get improvements in inter-personal relations and their ability to work together; they also get improvements in intellectual development and integrated learning [4] [5]. Having participated in problem-based learning, students considered it more interesting than the traditional classroom learning, and they became more used to sharing with other group members, causing the collective intellectual ability of their respective groups to increase [10]. Spaulding [27], for example, observed that interns of a hospital became more eager to work with patients to solve problems together.

Flipped classroom
Flipped classroom is a form of blended learning. It combines online learning and face-to-face teaching [3] [9]. The instructor first digitizes course materials and lectures and asks students to learn it online before they attend classes. Then, during class time, the instructor discusses questions with the students and guides them in problem solving. By helping students to solve problems and do practices, students learn by working on the four upper levels of (revised) Bloom's taxonomy [14]: application, analysis, evaluation, and creation. Bristol [26], for example, let students stay home and learn medical knowledge online; then, he lectured about practical clinic experiences during class time. In retrospect, Bristol [26] considers flipped classroom helpful in helping students to learn more. On the one hand, it (flipped classroom) saved time for learning basic medical knowledge in class. On the other hand, the instructor and students could use class time for discussing clinical practices.

A Mobile Anonymous Question-Raising System for Assisting Flipped-Classroom Learning
To encourage those who are shy in asking questions to ask questions in class and also to help students understand course contents, this research utilizes the convenience and immediateness of mobile devices and developed a mobile, anonymous question-raising system (MAQ) that assists flipped-classroom learning. By using MAQ, students can freely raise questions in class, and the instructor gets to know what the students do not understand when the class is in process. Students can use MAQ to do three things:

Raise questions in class
As Figure 2 shows, when a student does not fully understand what the instructor talks about in class, he/she may want to ask questions. However, if the student did not do an adequate preview before class, he/she may not know what to ask. Even if the student knows what he/she does not understand, he/she may also have problems to express himself/herself clearly. Therefore, MAQ provides its user with two alternatives for asking questions. The first alternative is to choose from a list of likely questions, a list that was prepared beforehand by the teaching assistants; these questions are the ones that were raised in the past by those taking the same course. The second alternative is to directly enter one's own questions. The ideal scenario then is the following. When a student is puzzled, he/she first checks the list of available questions and sees if there is any item in the list that matches with his/her puzzlement. If there is such a question in the list, he/she just selects it and clicks "send." If there are no questions in the list that correspond to his/her puzzlement, he/she then directly enters his/her question in the space provided and clicks "send."

View questions raised by others in class
As Figure 3 shows, it is also possible that when a student is puzzled, he/she cannot find his/her question(s) in the given list, nor can he/she directly specify his/her question. When this is the case, it may be helpful for this student to see what others are asking. For some students, it may also be of some comfort to know that "at least there are others who have the same question(s) as mine." By allowing each student to look at what others are asking, it helps the student to identify what he/she does not understand.

Review all questions raised in the course
As Figure 4 shows, students can review, after class, what they have asked in class. This helps to strengthen their impression of what they have learned in class. By reviewing their own questions (along with the answers given in class), students can better prepare for the exams. It is also a way for students to make better use of their time when they study. For example, they can try to find better or more convincing answers to their own questions. The Experiment

Course used in the experiment and participants of the experiment
The course used in this experiment was Systems Programming, offered by Department of Information and Computer Engineering of Chung Yuan Christian University, Taiwan. This course introduces three subjects to students enrolled in the department, and the three subjects were, respectively, database systems, software engineering, and operating systems. Teaching assistants of this course prepared a list of questions that the instructor of this course had asked over the years, and they put these questions in MAQ so that students could quickly go over relevant questions whenever the instructor talked about some specific topic in class. Participants of this experiment were 99 students taking that course. As Table 1 show, there were two classes, an "MAQ class" and a "normal class," all taught by the same instructor. The normal class had 54 students and was designated as Control Group A. The MAQ class had 45 students and was divided into two groups. 18 students of the MAQ-class had Android based smart phones and were designated as the experimental group. 27 students of the MAQ-class, whose cell phones were not Android based, were designated as Control Group B. Members of the experimental group could use MAQ to raise questions in class and review questions after class, whereas Control Group A and Control Group B did not have MAQ to use.

Procedure
The experiment lasted for four weeks. The instructor used the strategy of flipped classroom of blended learning to teach. When teaching any topic in class, he used problem-based learning to guide the students in their learning. The students were also encouraged to ask their own questions in class. Prior to the experiment, all participants took a pretest to evaluate their learning achievement; the teaching assistants also taught members of the experimental group how to install MAQ in their smart phones. During these four weeks, members of the experimental group could freely raise questions (in an anonymous way and also in real time) in class. The instructor used (the instructor part of) MAQ to see what the students' puzzlements were and also guide them in finding appropriate answers to their questions (along with other questions that the instructor had prepared for the class beforehand). The two Systems Programming classes were arranged in a way so that the experimental group (and Control Group B) always took their classes before Control Group A did; when the instructor later taught Control Group A, he always included in his question-answering session questions raised by the experimental group earlier in a separate class session. After the experiment, all participants took a posttest to evaluate their learning achievements; they also took part in a questionnaire survey to give feedback.

5
Result and Discussion

5.1
Learning achievement analysis Table 2 shows ANOVA analyses of pretest and posttest results of the three groups. By making pair-wise comparisons, it can be seen that for pretests, the F-value was less than the critical value for any two groups that are being compared. This means that all three groups had the same learning achievement before the experiment began. Table 2 also shows that there is a significant difference between the experimental group and Group A in posttest results. This means that the use of MAQ helps students to improve in their learning achievements. By using MAQ, students can raise questions in class in an anonymous way; this helps them to better understand what was taught in class. Even though members of Control Group A could still raise their hands and ask questions in class, they were nevertheless influenced by peer pressure and personalities in asking questions, causing them to understand less about what was taught in class.
Interestingly, there is no significant difference in posttest results between the experimental group and Control Group B. This means that students with no MAQ to use were influenced by students who used MAQ when the two groups took classes together. Even though members of Control Group B were not able to use MAQ to raise questions, they still benefited from questions raised by the experimental group. To some extent, knowing a question asked by the instructor actually comes from a peer (and not from the instructor himself) provides a stimulus for students to learn better in finding an appropriate answer to the question. This seems to be the case for all students in the class, even if some students of this class do not have MAQ to use. To find out more about what had happened to members of Control Group B, ten samples of these students were randomly selected for individual interviews. Section 5.3 analyses the results of these interviews.
The average score of Control Group B is higher than the average score of Control Group A. However, there were no significant differences. This suggests that even though members of Control Group B benefited from knowing some questions were raised by the experimental group, they nevertheless have some questions of their own that remained unanswered (since they did not ask their own questions due to shyness or peer pressure), causing them not to understand fully about what was taught in class.

Analysis of questionnaire results about system usage
To investigate how members of the experimental group felt about MAQ and how satisfied they were about its use, a questionnaire was designed. There are 13 items in the questionnaire. With the exception of the first question, which is a yes-no question, and the last question, which requires a simple answer, the rest are all questions that are answered using a four-point Likert scale, ranging from "strongly agree" to "strongly disagree." After the experiment was over, a questionnaire survey was conducted for the experimental group. 18 responses were collected. Cronbach's Alpha is 0.945. Therefore, the feedback provided by users of MAQ is trustworthy. Table 3 shows the questionnaire results. More than half of the subjects considered themselves to be of a shy type (Question 1). Therefore, by using an anonymous mechanism to raise questions, students that were shy had a way of asking the instructor questions. It also helped to improve the instructor-student interactions (Questions 2, 3, and 4). This is in agreement with the findings of earlier research, which suggested that the use of interaction tools could raise the level of interaction among the instructor and students, thereby helping to improve instructor-student relations [6] [11] [12] [18] [21] [22].
In general, students found MAQ useful in finding what they wanted to ask in the pre-arranged list of questions (Question 5). However, there were also some that preferred to phrase the questions by themselves (Questions 6 and 7). Most students agreed that the "questions raised by others" functionality of MAQ was useful (Question 8). This means that even though some might not know what their questions were, they could nevertheless identify their own puzzlements from questions raised by others in class. In addition, the students generally agreed that by reviewing questions that were raised in class, it helped them to review what was taught in class (Questions 9 and 10). Finally, from Questions 11 and 12, it can be seen that students generally gave approval to the use of MAQ in class and they also wished to use MAQ to raise questions in future classes.
To further improve the working of MAQ and also to know how the subjects felt about MAQ, Question 13 allowed the subjects to write down their opinions about system use. Basically, the subjects thought that it might be better to have more platforms that support MAQ (since the current research only provides MAQ for Android-based smart phones). They also gave various suggestions regarding system use. For example, when the system is about to send a question to the instructor, there ought to be a confirmation dialog so that a student does not mistakenly send a question to the instructor. As another example, it may be desirable for each student to edit questions (along with the instructors' answers to these questions) that were asked in class so as to rephrase the questions and answers in the student's own words. This could help students to better prepare for quizzes and exams when they later review what were asked and answered in class. The "questions raised by others" functionality of MAQ helps me to see whether my questions are also others' questions.
11.11 83.33 5.56 0 9 The "review of all questions raised" functionality of MAQ helps me to learn more about questions I did not quite have answers to in class.

Interview results for control group B
To understand how members of Control Group B felt about taking the same class with the experimental group but without MAQ to use, ten students of this group were randomly selected for interviews. The purpose was to see whether there were any negative effects on members of Control Group B and whether they felt being somewhat "left out." Members of Control Group B took their classes with members of the experimental group. Did this help them to learn better?
The actual question asked in the interview was: "Some of your classmates could use Android based smart phones to ask questions in an anonymous way. Then, they got answers from the instructor. Among the questions they raised, there might be questions that you also wanted to ask. Did you find it helpful when the instructor answered their questions in class?" Student 01: Yes. Student 02: Yes, indeed.
Student 03: Yes (it helped). Even if I knew the answer to that question, the instructor's explanation might help me to understand the whole thing better.
Student 04: It was very helpful. Since I might not have thought of those questions, the fact that they asked those questions helped me to learn better.
Student 05: Yes. Student 06: Yes, because sometimes I did not know how to ask. When that was the case, it helped when my classmates asked what I wanted to ask (but did not know how to ask).
Student 07: There is a prerequisite for getting help from what they asked, and the prerequisite is that the one who asks questions (using MAQ) must have seen the (instructor's) film first; otherwise, he/she would not know the right question to ask. If what the person asked was what I did not understand, it could be very helpful.
Student 08: I think this is very helpful, because I could not use my iPhone (with iOS) to ask.
Student 09: It is very helpful for catching up with the progress of the course. Sometimes, the questions asked happened to be what I did not understand, and I got to know the answers. But if I did not understand the questions in the first place, it would not be helpful. Anyway, when the instructor explained, I more or less got to learn some extra-curricular knowledge that was helpful. Student 10: I think MAQ is a nice system, because it makes it possible for those who usually do not ask questions to raise their questions. It also helped us to notice questions that we might not pay attention to.
From the students' answers to this question, it can be seen that most students who were interviewed felt that even if they themselves did not have MAQ to use, they nevertheless benefited from the fact that members of the experimental group used MAQ to raise questions. This may be the reason why Group B also had a higher learning achievement when compared to their counterparts in Group A.
Did students feel somewhat isolated, since they did not have MAQ to use (while some of their classmates had MAQ to use)?
The actual question asked in the interview was: "Some classmates of yours had MAQ to use, and they did use MAQ to raise questions to the instructor. Do you feel that this is unfair to other classmates like you who did not have MAQ to use (to raise questions)?" Student 01: No! Because we are always free to ask questions anytime. For example, if I know what stack overflow is and others do not know it, then I have an edge over others. The resource (i.e., the instructor who can answer questions) is there. It is just a matter of whether you know that it is there and whether you know how to use this resource.
Student 02: Yes! Because those who had a question, did not have MAQ to use, and also did not dare to ask could not ask their questions (and get the answers they needed) in time.
Student 03: No. If anyone has a question, he/she can always go to the instructor and ask.
Student 04: I think it is OK. There are many ways to raise one's questions to the instructor (and MAQ is only just one of the ways).
Student 05: Actually, it is OK. If one is serious in asking questions, he/she can do it without the use of MAQ.
Student 06: I don't feel that it is unfair. If you cannot use smart phones or pad computers to raise your questions, you can also raise your hand (and ask questions) in class; or you can also go to the instructor during class breaks and ask your questions. I believe the instructor is always happy to answer your questions. In other words, one can still ask questions without using smart phones or pad computers.
Student 07: Even though I did not use MAQ to raise questions, I do not feel that it is unfair.
Student 08: No. Because we can also raise our questions on i-learning. Student 09: Actually not. Because when they got answers from the instructor for the questions they raised, we also benefited from it. Besides, I don't think there is so much difference between directly asking questions (to the instructor) and using MAQ to raise questions.
Student 10: No. It is just that those without Android-based smart phones will have to borrow Android-based smart phones from their classmates in order to raise their questions using MAQ. This is a little bit troubleso me. Corbeil & Valdes-Corbeil [13] pointed out that students without mobile devices had a feeling of being isolated. However, this is not the case with Control Group B in this experiment, because they also benefited from their classmates who used MAQ to raise questions. From the results of the interview, it can be seen that members of Control Group B did not feel that it was unfair, because they had other alternatives of asking questions. The use of MAQ in this experiment not only helped those in the experimental group to learn better by asking questions, it also helped those in Control Group B to learn better, because members of Control Group B were also affected when their MAQ-counterparts raised questions in class.
Therefore, it may not be necessary for everyone in a class to use MAQ to raise questions. When those with Android based smart phones use MAQ to raise questions, those without Android based smart phones (including those who had no smart phones) could be stimulated, thereby causing all students of the same class to learn better.

Conclusion
This research associates MAQ, an anonymous question-raising system, with flipped classroom, a form blended learning. Central to this approach is problem-based learning. The instructor uploads video lectures and course-related materials on the Web, making it possible for students to learn course content online by themselves before they come to class. Then, the instructor helps the students understand course topics by discussing questions in class. In so doing, the instructor provides guidance to students and interacts with them so that the students learn by solving problems [4] [5] [7] [10] [27]. One problem with this approach is that some students may be shy in raising questions to the instructor, causing them not to learn well [28]. This is what MAQ was developed for. With the use of MAQ, students that are shy or in fear of peer pressure can freely raise questions in class, without having to reveal their true identities.
A related educational experiment demonstrated that not only could MAQ help students that are shy to raise questions in class, the use of MAQ could also help to alleviate learning achievement. It was also found that, so far as learning achievement is concerned, there is no need for all students to have MAQ to use. When students with MAQ raised questions in class, those without MAQ could benefit from their understanding that their classmates had raised such questions, causing these latter students to also learn well.