Students Prefer Audience Response System for Lecture Evaluation

Objectives: Student evaluation of courses is an important component of overall course evaluation. The extent of student participation in the evaluation may be related to the ease of the evaluation process. The standard evaluation format is a paper form. This study examines medical students preference of utilizing Audience Response System compared to a paper method. Methods: Following several medical school lectures, students were queried if they preferred Audience Response System versus a paper method, and if they would prefer using Audience Response System more for future course evaluations. Results: 391 students were queried. Overall response rate was 94%. Using a five point Likert scale, 299 out of 361 (82%) responded they agreed, or strongly agreed with the statement “We should use ARS more...” When asked which format they preferred to use for evaluation, 299/367 (81%) responded Audience Response System, 31 (8%) preferred paper, and 37 (10%) were not sure, or had no opinion (chi squared = 378.936, df2, p<0.0001). Conclusion: The medical students surveyed showed a strong preference for utilizing Audience Response System as a course evaluation modality, and desired its continued use in medical school. Audience Response System should be pursued as a lecture evaluation modality, and its use in medical school education should be encouraged.


INTRODUCTION
Student feedback is an important aspect of evaluation of course effectiveness. Instructors use feedback to implement changes, adjust curriculum, and assess utility [1]. Student evaluations are reliable, valid, have high utility, and are reasonably free from bias [2]. Mechanisms that encourage feedback, or that students find convenient, would conceivably increase feedback participation. Maximizing the rate of student feedback would improve the accuracy of the response, representing a truer representation of the entire class rather than the opinions of a motivated segment.
Student feedback regarding lectures, although encouraged, remains mostly voluntary. Mechanisms which increase student comfort and ease providing feedback are more likely to increase participation and enhance honest evaluations. While there have been great technological advances in education delivery recently [3], the use of hand written lecture evaluation forms is still common.
Audience Response Systems (ARS) have been used in education for many years in undergraduate [3,4,5,6,7] and medical school [8,9,10,11] and post graduate training [12,13,14,15]. A device, commonly referred to as a clicker, typically has a numeric keypad, which allows participants to respond anonymously in real time to a multiple choice slide, often as part of a PowerPoint presentation. The answers can be shared with the class, and are easily entered into an electronic database for further analysis. This mechanism enables instructors to keep students engaged, evaluate effectiveness of delivery, and to assess comprehension of lecture material. ARS has been shown to encourage active participation, enhance student learning and retention, and increase attendance [6,7,13,14]. Students like using ARS as they feel it enhances learning, improve retention, allows active participation, is anonymous, and they feel more comfortable using this modality to express unpopular views [13,14,15,16]. Instructors receive better evaluation when using ARS as compared to similar lectures given without using ARS [17].
Despite its expanding role in higher education, there is a paucity of information regarding utilizing ARS as a lecture evaluation mechanism. It has been previously shown utilizing ARS as a course evaluation tool increased feedback participation [9], but no published paper was found describing students preference using ARS as a lecture evaluation modality.
The purpose of this study was to determine if students preferred using ARS instead of standard evaluation forms to evaluate lectures.

II. METHODS
An ARS method was used to evaluate different lectures given to seven different medical school classes in various years of study during the 2006-2010 academic years. The evaluation questions were included as multiple choice slides at the end of the lectures. The results of the questions were not shared with the class. The three questions posed, utilizing the ARS multiple choice format, were: 1. Have you used ARS before? Key: SA = strongly agree, A = agree, N = neutral, D = disagree, SD = strongly disagree, ARS = audience response system The answers were captured on the computer program Turning Point ® (Turning Technologies, LLC, Youngstown, OH, USA). The Institutional Review Board was consulted, and determined this study was exempt.
Data were analyzed to assess if previous use of ARS was associated with a preference for using ARS in the future, and for using ARS versus paper form for lecture evaluation. For the sake of the analysis, those who answered in favor were compared with all other answers.
Categorical data were analyzed with chi squared or Fisher's exact. For the choice of modality, the assumption was for an equal distribution. The responses of those with previous ARS experience were compared with those without previous experience using an independentsamples t-test. Effect sizes (Cohen's d) were also calculated to determine the practical significance of the differences (where .2 is small, .5 is medium, and .8 is large).

I. RESULTS
The seven lectures were attended by 391 students. Three questions queried the students on their past use, and preference of using ARS, for a possible total of 1173 responses. The overall response rate for all questions was 94%. Additionally, 232 out of 370 who answered had used ARS previously (63%).
Using a five point Likert scale, 299 out of 361 (82%) responded they agreed, or strongly agreed with the statement "We should use ARS more..." When asked which format they preferred to use for evaluation, 299/367 (81%) responded ARS, 31 (8%) preferred paper, and 37 (10%) were not sure, or had no opinion. A chi-square test indicated a statistically significant preference for using ARS (chi squared = 378.936, df2, p<0.0001)(See Table 1).
Regarding preference for future use, students who were using ARS for the first time averaged 4.44 on a 5 point Likert scale (1=strongly disagree-5=strongly agree, "We should use ARS more") compared to 4.18 for those who had used it before. An independent-samples t-test found this difference to be statistically significant, t(347) = -2.38, p = 0.018, d = 0.26 (See Table 2).
When comparing the groups who had used ARS previously with those using it for the first time, a Fisher's exact test revealed there was no statistical significance between the groups in their preference for evaluation modality ( Table 2). Likert score is the average of total responses (Strongly Agree =5, Agree = 4, not sure = 3, Disagree = 2, Strongly Disagree = 1).

II. DISCUSSION
These results show students indicated a strong preference for using ARS as a lecture evaluation format compared to the standard pre-printed paper tool. Students also stated a preference for more ARS use in lecture evaluation. This is congruent with other published results, where 88% of students stated they enjoyed using clickers in class [16]. Students have also consistently scored greater than 4 on a 5 point Likert scale when queried about their approval for using ARS [16].
The results also showed those who were using ARS for the first time were more in favor of more ARS use in the future. However, the effect difference was small, making it of little practical significance, as both groups showed strong preference for ARS use. Regarding preference of the evaluation modality, there was no difference between the groups; both groups strongly favored ARS.
The small, but significant difference demonstrated by the group using ARS for the first time expressing a preference for more ARS use in future evaluation may represent a novelty effect that may erode over increase exposure. Such a phenomenon has been observed previously, where upper level students, while still in favor of using ARS in classes, scored ARS slightly lower that lower level stu-PAPER STUDENTS PREFER AUDIENCE RESPONSE SYSTEM FOR LECTURE EVALUATION dents [7]. This conjecture is purely speculative; this study was not designed to ascertain the reasons for this difference. Such an inquiry can be the subject of future investigations. A study looking quantitatively at previous use (number of times used in the past) and desire for further use may determine if there is a linear relationship, or a plateau effect.
Potential weaknesses of this study include the use of non validated questions. The questions are straight forward, and there is not much room for interpretation. In analyzing the Likert answers, an arbitrary cut off was made comparing the number of "strongly agree" and "agree" versus "neutral" and the negative answers: this is felt to be a fair and rational dividing point. In analyzing the main data point, which modality the students preferred, ARS was compared versus all other answers combined and the difference was found to be statisticallysignificant, making the results more profound compared to analyzing each answer separately. No assessment of reliability or accuracy was attempted; the anonymous nature of using ARS was thought to help increase honesty of responses. ARS allows the instructors to share responses with the class in "real time", however, for this study, the responses were not shown to the class to eliminate a potential bias of being influenced by peer responses, and subsequent affect to future answers.
ARS use has increased substantially in medical education. While there are many studies outlining the virtues of ARS in delivery and implementation of course material, goals, and objectives [3-8, 10-15, 17, 18], no studies were identified specifically looking at student preferences utilizing ARS for course or lecture evaluation.
Torbeck [19] described utilizing ARS as a residency program evaluation tool. She stated "The residents always seem to enjoy using. . . ARS. . .". However, she offers no data, and the conclusions appear to be totally subjective.
Student evaluations of courses are mandates by the Liaison Committee of Medical Education (LCME) [20]. These evaluations are used to give feedback to teachers, utilized in promotion consideration, and as outcomes research [1]. Data collected with ARS is easily transferred into an electronic database, which can then be utilized for research and other considerations. Even if student evaluations were not mandated, motivated instructors would encourage feedback in order to provide the best learning experience. As medical school curricula become more and more crowded, lecture time should be utilized for maximum effectiveness. Part of the assessment of this effectiveness would be student perception of the utility of a course or lecture.
In order to maximize participation, a modality should be selected that is easy to use, promotes participation, is anonymous and where results can be readily accessed and analyzed. Conceivably, students will be more likely to respond in a format they prefer using, which would lead to both an increase in, and more accurate feedback. ARS has been previously shown to improve student participation [9] and also meets the other criteria. Utilizing ARS as an evaluation modality would help facilitate a more effective evaluation process.

III. CONCLUSION
This study demonstrated the medical students surveyed showed a strong preference for utilizing ARD as a course evaluation modality, and desired its continued use in medical school. ARS should be pursued as a lecture evaluation modality, and its use in medical school education should be encouraged.

ACKNOWLEDGMENT
The author would like to acknowledge the assistance of Marcel D'Eon, PhD, and Krista Trinder, MA, in the preparation of this manuscript.