Classroom Response Systems in the Wild: Technical and Non-Technical Observations

—Classroom Response Systems (CRS) provide lecturers a communication channel to get feedback from their students. In lessons with large audiences, CRS allow students to ask questions or state issues as the lesson continues. During the development and usage of our new CRS "Tweedback", we observed several technical and non-technical problems, which are likely to be general CRS issues. Observed problems are caused by necessary devices, their connectivity and lecturers’ and students’ different ways to use CRS. In this paper, we describe our observations of technical and nontechnical problems and suggest solutions, which may be applied generically to interactive feedback systems.


INTRODUCTION
Smartphones, Tablets and Notebooks have become students' everyday companions. They are used for communication, entertainment, and assistance in various scenarios. Over the last years, researchers started to use these devices to create an additional communication channel between lecturers and students in lessons with large audiences. So called "Classroom Response Systems" (CRS) allow lecturers to get feedback from their students as well as it allows students to get feedback from the lecturer while the lessons proceeds [3].
Whereas there are several types of feedback that can be given during a lecture, we use the following three types [18]. First, students may submit questions to both the audience and the lecturer. To indicate important questions an up voting mechanism may be implemented. We call this a Chatwall. Second, lecturers can challenge their students with quizzes, i.e. multiple choice questions. Third, students can provide feedback to lecturers by rating specific parameters of their speech, for example talking speed or volume.
This publication provides an overview on approaches, existing projects and current solutions for live feedback in lessons with large audiences developed at German universities. Whereas a live feedback application has to deal with many technical problems, nontechnical problems are often invisible -until they grow up and become an obstacle for users. Thus, we documented technical and nontechnical problems we observed in CRS and suggest solutions to them. We hope that future and existing projects can use these results to improve their work.
In general, we observed three main technical difficulties. First, the wireless communication infrastructure has to be available permanently and must carry the huge amount of data, which may occur in large lessons with more than 100 students. Second, lecturer's screen stand and screen angle can become important issues to lecturers. Insufficient stands were very frustrating experiences for lecturers. Third, a CRS should be aware that all user input has to be reviewed prior to processing, as users may otherwise misuse the communication channel offered by the CRS.
In addition to technical problems, we observed several nontechnical problems with CRS: Students want to reply to other students questions and they want to access a lesson's content even after the actual lesson ended. Furthermore, young students seem to have a higher motivation to write comments, which are unrelated to the content of teaching. This behavior abates with the time of use, but lecturers should be aware of it.
This paper provides solutions regarding these observations and resulting issues. A fine-grained publication model provides students a communication channel which enables them to address only relevant recipients. Moreover, a CRS integrated into the existing presentation application reduces the lecturer's distraction.
The rest of the paper is organized as follows. Section II presents and compares existing projects. Section III describes the technical observations we made regarding our CRS' use, whereas section IV covers nontechnical observations. In section V we present our solutions to technical and non-technical problems. Section VI concludes the article with a summary and an outlook on future work.

II. RELATED WORK
The following sections give an overview on the most advanced projects and approaches, which allow at least one of the three live feedback types as previously defined. We compare several attributes of these projects (Table I).
To categorize CRS we start by distinguishing between native and web applications, as devices not supported by native applications reduce the number of people, who can give feedback. Whereas native applications benefit from their underlying operating system and are able to use device sensors or can be integrated into the system (and are often easier to maintain), web applications can be used on nearly every mobile device [7].
Furthermore, we take a look at different usage barriers. A question which reveals the asking person's identity may build up a barrier, as students may hesitate to ask questions perceived as embarrassing. Anonymity or pseudonymity may decrease students' inhibition to provide feedback [4], because anonymity and pseudonymity prevent students from identification by their lecturers [10].  Another barrier of use is integration into the university's software system. While an application can benefit from integration into existing university software system, it is often a problem to use this application without this software system [7]. Deeply integrated applications may exclude users, who are not members of this system, e.g. visiting lecturers.Finally we inspected projects for their implementation of the live feedback types Chatwall, Quiz and Speech Parameter.Kundisch et al. implemented Pingo, which aims to support the peer instruction concept for large classes [8] [14] [17]. By using a public accessible web application, it can be viewed on every modern browser, regardless of the used device. At the moment, lecturers have to create an account in order to use it, whereas students can access lessons without creating an account. Pingo features only quizzes and is not integrated into any existing presentation software. A great advantage of Pingo is its capability to create questions in advance. Moreover it provides an overview on all created and used lessons. According to their statistics 7 , Pingo is well accepted (more than 1700 created lessons between October 2012 and November 2013).
Feiten et al. developed Smile [1] [2], an application that consists of two different parts. First, Smile provides a Windows application for lecturers. Second, students can access Smile using a web application, an app on iOS, or an app on Android. The lecturer's application is used to create and control lessons, so lecturers are able to create their Quiz questions in advance. Furthermore, Smile allows students to ask and answer questions, which may be used as replacement for a Chatwall. Moreover, students can state their general understanding: They can send a signal whenever they do not comprehend the lecturer. Smile has a highly adjustable user interface, featuring many settings. The developers found an elegant way to combine the Quiz feature with the lecturer's slides presented in a third party application: By overlaying the slides with a small transparent window containing the current quiz' results, the lecturer is able to integrate the quiz into his presentation easily. Whereas this user interface allows lecturers to configure the application for their needs, Weber et al. state that the usability and responsibility of Smile does not yet fit students' needs [19].
Gommlich et al. extended their university's personal learning environment myTU with a functionality, which allows students to rate the speed of the lecturer's presentation [6]. It also features a generic "Stop" button, which may be used whenever students have an issue regarding the lecture or the lecturer's presentation. Moreover, students may ask short questions, which are immediately shown on the lecturer's device. The myTU-App can be considered more an assistant for student's life than a pure CRS: It is used to manage students class schedules, their lending from the library, the cafeteria plan and provides many other tools supporting students' everyday life. Because this application has been developed for the TU Freiberg, it is deeply integrated with the university's software system and every user needs a university account to use the app on Android or iOS.
ARSNova is a CRS implemented as a web application. It may be used anonymously or after authentication with a university, Facebook, or Google account. It is possible to integrate ARSNova into different learning management systems such as Moodle, Ilias, or StudIP. It provides functionality to create real-time quizzes and to rate a lecturers Speech Parameters. Furthermore, a presentation view similar to Smile allows lecturers to overlay the presentation slides with the current quiz' results [12].
Tweedback implements all three previously described features, and is provided as a public accessible web application. Therefore, no accounts are needed to create or participate in a lesson. Tweedback provides pseudonymity for students and lecturers and is not integrated in the universities software system [5]. Currently it is subject to a refactoring process. Subsequently to this process, Tweedback will be released under an open source license inVote is a project developed by Netzmanufaktur GmbH for TU Dresden. It serves real-time quizzes and is implemented as a web application. Lectures have to create an account prior to starting a quiz, whereas students can answer anonymously and without an account to these quizzes [16].  Table I compares the discussed projects. It provides all properties and features considered important. It seems remarkable, that all presented projects provide anonymity or pseudonymity. Only Tweedback and Smile provide all three kinds of live feedback, whereas ARSNova and Smile stand out with their overlay functionality, which provides information directly on a lecturers presenting screen.

III. TECHNICAL OBSERVATIONS
Due to the wide spread presence of smartphones, it seemed plausible to use a purely web-based architecture. Most of our students own a smartphone, with which they can use the Universities Wi-Fi network to connect to the Internet. Moreover, GSM and UMTS connections can also be used. In contrast to CRS hardware solutions, e.g. "Qwizdom" [13] and "Powervote" [11], which build up their own communication infrastructure, web-based architectures use existing Wi-Fi infrastructure. Therefore, they also share problems, such as connectivity problems caused e.g. by massive peer-to-peer file sharing. Our observations regarding communication infrastructure, input devices, and software requirements are described in the following section.

A. Communication Infrastructure Availability
Prior to using a web-based CRS, Wi-Fi has to be available in lecture halls. Not only the Wi-Fi availability but also the capacity has to be checked. It is important to know the exact location of installed Access Points (AP) and which area they cover. It is difficult to make estimations about upper bounds of the number of users in a Wi-Fi network. This number depends highly on the users' behavior, installed hardware, and the web application. Consulting the corresponding IT administration and using their experience is one way to estimate the maximum number of users which an AP can handle in practice.

B. Input Device
The process of introducing a web-based CRS to lecturers consists of the following steps: • Verification of communication infrastructure • Learn how to use the CRS • If necessary: Introduce additional display/input device Lowering the initial barrier is important for a wide acceptance. Therefore, offering support for lecturers new to a CRS can help accomplishing the tasks mentioned above. Depending on the CRS' capabilities, an additional display/input device is needed. Only a small number of CRS, namely ARSNova and Smile, have some sort of overlay window for presentation slides. Anyhow, this overlay shows only a small amount of information. For example, it is possible to show a smiling face if users are satisfied with the current talk or a sad smiley if it is not understandable. For features showing more information content, such as a Chatwall, a small overlay is not feasible. To keep an overview on new content, it is important to avoid interrupting the teaching process. Therefore switching between presentation and CRS window on one device cannot be recommended. One solution is using additional display devices, e.g. tablet computers.
When mounting the tablet, horizontal and vertical angle are important to assure an optimal view on the screen. Many stands for tablets are designed to place the tablet in a fixed place for watching films or reading a book. Consequently, many stands are insufficient for a CRS scenario, in which lecturers interact with devices. Lecturers scroll a webpage, activate features and highlight questions. Many stands are not designed for pushing devices at the upper corners. This physical problem has -regardless of how trivial it may seem -to be respected when choosing a stand, as this detail has significant influence on the lecturers' experience with a CRS.

C. Platform
During the design of Tweedback only very few assumptions regarding the user's platform were made. The only requirement was a browser application and integrated Wi-Fi. This requirement is fulfilled by almost all modern tablets. Therefore, building a web application with user interface frameworks, such as the Bootstrap framework, and the use of standard technologies (HTML, JS, and CSS), minimizes restrictions on end-user devices.
On the server side, we were able to verify the experience with users the Pingo project [17] made. As we are working with pseudonyms, every user has a nickname such as "user123" or "user211". In early versions of Tweedback this information was saved in a cookie. First tests with Computer Science students lead to massive manipulation of usernames. By changing the cookie's content, students changed their nicknames. Therefore it is highly recommended to treat all transmitted data send by users as possible manipulation attempts.

IV. NONTECHNICAL OBSERVATIONS
Further observations deal with user behavior and new usage scenarios for lecturers.
A Chatwall feature can have a restricted functionality by design. One possibility is to allow only new questions and to deny responses. This way, users can only ask questions and lecturers have to answer them. The idea of providing a restricted usage is also supported by Cognitive Theory findings [9] [15] which states that working memory of human cognitive system consists of channels which can be overloaded. Too many pictures or too much text overloads these channels. Too much unstructured information has the same effect.
As a consequence, we decided not to allow users to chat with each other or to comment on other posts.
Contrary to our intention, users began to use our system for communication with each other anyway. They replied to other posts by prefixing new posts with terms such as "@user123". This behavior shows that there is a need for a reply function which contradicts with recommendations of didactic researchers.
In lessons with high activity, the Chatwall has to serve several needs: On the one hand, users want to see the latest posts and vote on them. On the other hand, users want to see the "hot topics", i.e. those questions with most votes. Providing different sorting of questions turned out to be important for lecturers. In general, lecturers use a "most voted" view of the Chatwall, as this view emphasizes questions and problems addressed by most students and thus probably important. If new questions are the lecturer's focus, the "latest posts" view can be used. Any-PAPER CLASSROOM RESPONSE SYSTEMS IN THE WILD: TECHNICAL AND NON-TECHNICAL OBSERVATIONS how, in case of a large audience with many students who ask many questions, it can be hard to overview all new messages in short time.
We were able to distinguish between usage patterns, which seem related to the duration of CRS usage. We identified an initial phase concerning students who used Tweedback for the very first time. They began to "play" with it, creating lots of small messages without lecturerelated content. The initial phase is followed by a productive phase, in which the amount of such messages drastically reduces. The productive phase lasted for the rest of the term. From the lecturers' view, especially in the initial phase many messages can be classified as spam, as they do not address a problem related to the current lecture. This behavior was expected and after some lectures, we observed a normal behavior where students asked only serious questions and used the system as intended.
We also experienced a demand to access all results after the lesson ended. Lecturers wanted to evaluate and analyze the data that has been generated by the students to learn and understand their students' problems. Students on the other hand wanted to share these emerged problems in order to get a deeper knowledge in specific topics, slides, formulas, definitions, or simply to solve issues that are not directly related to the actual lesson's subject. For this reason, durable access to the lessons results turned out to be very important.
In addition to the ability to have durable access to a lesson's data, some lectures demand a functionality to ask questions before the lesson starts. Lecturers may use such a feature to check their students' knowledge of the previous lessons or to ask questions which may give an overview on the students' motivation ("Which semester are you?", "Which study path are you associated?", …). Because students are not entering the lesson simultaneously, it may be useful to ask such questions in a loop before the lesson starts.

V. SOLUTIONS
Our observations showed that users want to comment on questions and find themselves a way to override the system's restrictions regarding replies on other comments. Keeping this experience in mind, we will implement a reply-feature for the Chatwall. In the first place, this seems to contradict with the didactic researchers' recommendation to not create another communication channel between students. But the problem is that there already exist other communication channels once the smartphone is up and running within a lesson. All apps on smartphones which allow communicating with others present a form of parallel communication channel. Therefore, our aim is to fulfill students' need and implement a reply function as it may keep them in the CRS. Because there are already other communication channels available, we believe that uses will not be more distracted than before. In case of many replies to a question,, the Chatwall page will become very long. Additionally, this will make it more difficult to keep track of current questions in short time. Therefore, we suggest different views on the Chatwall. In the following, we will differentiate between three motivations of a reply and explain their resulting views.
First, private replies (which are exchanged between two users) will reduce messages perceived as "noise" by other users. For a better overview, these messages are not shown to other users than the regarding recipient since others were not addressed explicitly.
Second, semi-private (audience only) replies reduce messages perceived as "spam" by lecturers. We observed a relevant amount of questions which are directed to the audience, e.g. organization of collective lunch. Lecturers perceive these messages as spam. Allowing for semiprivate messages lets the audience communicate without disturbing teachers.
A possible third kind of replies are public comments. Such comments are visible to all users who have joined a lecture. Separating these three reply motivations and creating a unique view for each of them enables users to communicate without disturbing those not interested in the communication. Figure 1 summarizes all three motivations and their covering area in an auditorium. Different needs for sorting, as stated above, can be implemented in a simple way. Depending on their needs, users can switch between two views of Question Wall. The view "voted most" will be primarily used by teachers. This way they keep an overview of questions many people voted for and they are probably important. The second view "latest questions" aims for students, as they can recognize new questions without wasting time to scroll down.
Up to now, we recommend using an additional tablet device when using a CRS in a lesson. This measure eases interaction with the Chatwall. This contradicts with the obvious interest of lecturers who want to carry as few devices as possible. To solve this problem, we plan to incorporate features into presentation software such as Microsoft PowerPoint. Once implemented, lecturers will not need to switch between devices or applications in case of Quiz and Speech Parameters.
There exist other solutions, e.g. to incorporate the Chatwall into presentation slides on audience devices. Our vision is that students can write their comments live directly into the slides. This would be a great advantage, as many questions refer to specific parts of a presentation. If students could leave comments directly in the slides, e.g. as annotation, the problem would be directly related to its origin. We imagine a web-based solution for this scenario, where presentation slides are available online. Users can view the slides simultaneously on the presenter's projection as well as on their own device. Furthermore, students PAPER CLASSROOM RESPONSE SYSTEMS IN THE WILD: TECHNICAL AND NON-TECHNICAL OBSERVATIONS may highlight or comment interesting parts visible to other students.
Concerning the feature of durable access to a lesson's user generated content, we implemented an interface showing all this information. Therefore we store and display all quizzes and all Chatwall posts. The Speech Parameter will be shown as a line graph, where each line represents the ratings of a specific parameter's development during the lesson.

VI. SUMMARY
Using students' smartphones to access a CRS has many advantages. To use a web based CRS, a communication infrastructure, such as Wi-Fi network, is necessary.
Getting an overview of the audience's feedback in short time is very important to lecturers. A Chatwall can contain many questions. Therefore, we recommend an external device (e.g. a tablet computer) to present this additional information to lecturers.
We observed that students use the communication channel in lessons to reply on questions -in groups or personally. Thus, we suggest providing private, semiprivate (audience only) and public conversations. This allows users to comment on other users' posts and preserves a low spam level. It has to be evaluated whether this new possibility of commenting is distracting users significantly more than the existence of other communication channels on the smartphone.
Another observation were repeated requests for different sortings regarding the Chatwall. We identified two different sorting methods: 'by date' and 'by upvotes'.
Additionally, we observed the need for durable access to a lesson's results. This feature should be implemented in a read-only view, where all Chatwall posts and quizzes are listed. For Speech Parameters, a line graph showing ratings of each parameter over a lesson's duration seems a promising visualization.
Future research should focus on a wider application for this real-time communication channel between audience and lecturer. This may cover different types of live feedback, such as comments and annotations directly in the lecturer's slides. Furthermore, there is need for long-term didactical evaluations in CRS with large audiences.