Can Web Thin Clients be used to Create Flexible Assessment Spaces in a Higher Education Setting?

For many HE institutions, a major factor inhibiting the wider deployment of computer-based objective assessment is the lack of suitable IT suites. The advent of pervasive wifi within HE campus environments, combined with the development of portable and low-cost browserbased devices, motivates the question of whether existing general-purpose teaching spaces can be used as venues for computer-based assessment. This work describes the results of a pilot study aimed at investigating some of the technical and logistical issues surrounding this question and evaluating student perceptions of this particular implementation of an educational technology.


I. INTRODUCTION
Objective testing, which requires students to provide a response to a question whose correct answer is predetermined, is particularly well-suited to forms of computerbased assessment that incorporate automated marking. If the test has been designed correctly then the marking of responses is free from subjective biases. The drawing of questions from large question banks offers an opportunity for students to practice and develop mastery in addition to adding a layer of security to an online test. Furthermore, with online tests students can receive immediate feedback on their performance, regardless of class size. For these and other reasons, in recent years many teachers in higher education have increasingly experimented with delivering computer-based objective assessments [1]. Such testing plays an obvious role in summative assessment [2], but online tests as a type of e-learning can be used in purely formative contexts [3] or as a combination of formative with summative [4]. Online tests are also used extensively for diagnostic testing (see [5] for a discussion of how basic mathematical concepts can be tested via mobile devices).
Within many institutions, a limiting factor to increased take-up of computer-based objective assessment is the availability of suitable IT suites: purchasing large numbers of desktop computers is expensive, constructing venues in which to situate those computers is more expensive still. However, the advent of pervasive wifi within HE campus environments, combined with the development of portable, low-cost, browser-based devices, suggests the possibility of transforming an existing teaching space into a venue that is appropriate for the delivery of computer-based assessment activities. In this paper we describe the results of a small-scale project that aimed to evaluate some of the technical, logistical and usability issues surrounding this possibility.

II. THE PROJECT
The massification of higher education within the UK and the increasingly diverse student populations entering university has meant that diagnostic testing is increasingly seen to be a useful tool in a lecturer's repertoire. When employed appropriately, and in conjunction with suitable support measures, diagnostic testing can assist in student progression and aid retention [6]. However, diagnostic testing typically needs to take place early in a student's first year of study, ideally during the induction period, and this is a time when an institution's fixed IT suites see particularly heavy demand. This particular constraint motivated us to ask whether portable web thin clients might provide a solution.
Web thin clients are relatively recent devices that provide a web browser and rely upon web applications to provide general-purpose computation. The Chromebook is perhaps the most common example of a web thin client: it runs ChromeOS as its operating system and, although it has some local storage and can be used in an offline mode, it is primarily designed to be used whilst connected to the internet. The Chromebook can also be used as an ultrathin client for desktop virtualization, giving access to PC or Mac applications.
Factors such as affordability, simple device management and the presence of a physical keyboard -factors that are often missing in tablet devices -have led to the increasing use of Chromebooks in the education sector. Indeed, in the third quarter of 2014, Chromebooks for the first time outsold iPads in US schools [7]. A further element in their adoption by US schools is that the devices are verified by both Smarter Balanced [8] and PARCC consortia [9] for online student assessments. Take-up in UK schools has been slower than in the US, although there is some evidence of increasing usage in primary and secondary institutions in the UK [10].
Although Chromebooks have been used extensively as secure platforms for online assessment within secondary education and tertiary education, little work has been published on their use within the higher education sector. An internet-based search of the IT and technology enhanced learning websites of UK universities revealed that several institutions have purchased large numbers of SHORT PAPER CAN WEB THIN CLIENTS BE USED TO CREATE FLEXIBLE ASSESSMENT SPACES IN A HIGHER EDUCATION SETTING?
Chromebooks and made them available for staff and students to borrow, but we found no evidence for systematic trials having taken place. This is somewhat puzzling, given the likely expectations of students progressing to higher education from an environment in which Chromebooks are increasingly used. In order to research the potential of Chromebooks in our particular question of interest -whether web-thin clients make it possible to create, in effect, peripatetic examination and testing spaces -we therefore decided obtained a small number of different models of Chromebook and work with academics and students to determine whether the devices met basic usability and disability requirements. Building on feedback from that initial evaluation we then obtained a larger number (n = 30) of Chromebooks to evaluate their potential for assessment-related tasks. In particular, we investigated: their suitability as a secure platform for testing students; their compatibility with delivering Moodle-based tests (our institution deploys the open-source Moodle as its VLE, but the results should independent of the particular quiz engine being used); the administrative overheads involved in switching between an "exam mode" and a "classroom mode" (such a mode being necessary because, as discussed in Section 3.4 below, the economic case for deploying these devices is boosted if they can be used for interactive in-class use); wireless infrastructure requirements; and logistical issues.

III. RESULTS
The following subsections present the results of the various different elements of the project, including the outcomes of a pre-pilot phase and a larger pilot in which Chromebooks were used to deliver 15 exam sessions and a large number of in-class activities.

A. Usability
A small pre-pilot involving level 5 and level 7 engineering students (n = 10) was conducted with three different Chromebook models in order to ascertain whether there were any significant usability issues surrounding their use in computer-based testing. The students were asked to take a sample test, which consisted of questions unrelated to the subject area being taught. The intention was not to test student knowledge, rather their experience of interacting with the full range of question types available in Moodle. The pilot relied completely on the Chromebook devices: no external pointing devices or keyboards were used. After the session, students were asked to complete a short survey relating to their experiences.
The results of the survey showed that all students supported the idea of using Chromebooks for CBT. All students registered a preference for a particular make of device (the one with the largest screen; this device was also deemed to possess a better touchpad and keypad). However, students were encouraged to take advantage of a freetext field in the survey in order to make additional comments, and 40% of them (n = 4) did so. Analysis of these comments showed that some students would prefer access to a traditional mouse, since they were unfamiliar with Chromebook technology and found the experience of scrolling and selecting using a touchpad to be somewhat time consuming. Another comment was that students should be given the opportunity to practice on the devices before taking a test, in order to become familiar with them. This would be possible, for example, if the devices were used during teaching periods for formative in-class activities. Such use would also improve the economic case for purchasing Chromebooks (though note that, by definition, it would be difficult to give students the opportunity to practice taking diagnostic tests if those tests were scheduled early in their transition to university). Some models of Chromebook replace the caps-lock key with a search key; this caused problems when students wanted to use capital letters or entered a password containing capital letters. The provision of an external keyboard overcame this issue.
We also commissioned a report on device accessibility by a specialist disability support officer who noted that, at the current time, the main suppliers of assistive and supportive software do not support Chromebooks but that the variety of available apps, extensions and built-in system tools can be used to give a reasonable level of support. It was noted, however, that the built-in magnification accessibility option in no way replaces specialized software for the visually impaired. Furthermore, one make of device could not easily be opened with one hand since there was no lip; other makes were better on this score. Following this report, and reaction from students, the recommendation was that the accessibility options should be set as the default.
Given the encouraging student feedback from this small-scale pilot we decided to source 30 Chromebooks plus flight cases and a smart-charging secure cabinet in order to investigate other issues.

B. A secure platform for high-stakes assessment?
The educational version of the Chromebook has a blacklist function and a blacklist_exception function; taken together, these functions allow an administrator to block access to all URLs except those specified. We used this as the basis for developing an examination-based Google administration "container". In exam mode the device was locked down: upon logging in, the student was taken directly to an exam landing page and could access only those websites and applications specified in the blacklist_exception. The exam mode also disabled the devices' USB ports (although USB mice and keyboards could still be used) and SD card slot.
The project team asked security specialists to attempt to access unauthorized sites while the device was in exam mode. The consensus was that the container provided sufficiently strong security to be employed in summative assessment tasks.
The more established online exam platform within the authors' institution consists of a combination of PCs running the open source SafeExamBrowser and an exam theme. The level of security provided by Chromebooks would appear to be at least as high as that given by SafeExamBrowser; students expressed a clear preference for Chromebooks over SafeExamBrowser.

C. Compatibility with VLE-based tests
The Moodle Quiz engine constitutes the main delivery platform for computer-based assessment within the authors' institution. This particular deployment of the Quiz provides test-setters with access to 20 different question types: the standard complement of Moodle question types, several of the question types developed by the Open University and subsequently released to the community, plus a question type developed in-house. The project team confirmed that all question types performed correctly on the Chromebook devices. Existing online examinations therefore required no modification to allow for Chromebook-based exams. It should be noted, however, that Chromebooks do not support Java applications; this would affect some question types published by the Moodle community.

D. Device management
Economic considerations dictate that thin client devices are unlikely to be purchased purely for assessment purposes. Devices that can be employed for in-class purposes when not being used to deliver assessments constitute a more efficient utilisation of resources. Furthermore, using them in a classroom setting enables both students and academics to gain familiarity with the devices before being faced with them in the high-pressure setting of a high-stakes examination. To this end we developed two Google administration containers in order to support two different modes of operation. The exam mode, which locked the device down, was described in section 3.2. In classroom mode the device could be used freely and without restriction; students had access to their usual apps for use in class-based interactive sessions. (We shall publish our experiences of using the device as an interactive mobile technology in a classroom setting in a forthcoming paper.) Both staff and students raised privacy concerns regarding their inability to remove personal wifi account details from the devices while in classroom mode. This issue was resolved by adding a setting to the classroom mode profile so that, upon signing out and powering off, all user information and settings are erased.
One academic involved in the pilot suggested the development of a third container to support a restricted mode of operation. The restricted mode would blacklist all websites except for those that a teacher planned to use during a session. The intention behind the restricted mode is that students would be unable to use the devices to visit inappropriate sites during the sessions. This container is currently under development.
Switching a device from the normal mode to the exam mode was quick (typically 30 seconds per device) because an automatic login process for wifi was applied. However, although switching a device from the exam mode to the normal mode was found to be relatively straightforward, it was potentially time-consuming. The activity requires an administrator to "move" the devices into the appropriate container so that the appropriate Google policy could be applied. We found that it could take up to three hours for the devices to detect the changed policy. After this time, the devices had to be switched on individually and left for 15 seconds before rebooting. It then required a manual login to the wifi network on each device. It follows that if Chromebooks are to be used in summative assessment situations then those responsible for the management of the devices must be given sufficient notice of requirements in order to ensure that the Chromebooks are in the correct container at the correct time.

E. Wireless infrastructure
As is the case with the vast majority of UK universities, the authors' institution deploys eduroam; this service provides users with authenticated network logon and access to the Internet via a single wifi profile and set of credentials. An eduroam outage that coincided with a scheduled exam session highlighted the need for an active wifi connection when Chromebooks are involved: the exam session had to be rearranged. Following this episode we investigated whether plug-in wifi routers or a dedicated exam wifi network would mitigate against such conflicts. However, due to restrictions imposed by JANET, the UK supplier of Eduroam, plug-in wifi routers are not an option; and a dedicated network would still be dependent on the institution's infrastructure. We therefore concluded that the adaptation of business processes would be a better approach to minimizing problems caused by maintenance of the wireless infrastructure; in particular, eduroam maintenance could be performed in defined at-risk periods (during which time no exams should be scheduled) while an exams calendar could be published automatically to relevant members of technical support staff to assist with their planning.
Network performance was monitored during various Chromebook-based assessment sessions and no problems were visible. The project team also undertook a wifi load test by using all 30 devices in exam mode and working through a specimen Moodle Quiz containing embedded video and audio. The session ran smoothly, with no observable lag when playing media. The wifi load was monitored during this session; the network dealt easily with the load. We attempted to profile the session in terms of a local load benchmark. The university library has an above-average client connection count but bandwidth here is low since students are not browsing heavily; in halls of residence students stream HD media services, engage in online gaming, use person-to-person sharing as well as general surfing and so there is high-bandwidth use with fewer connections. Load during the Chromebook test session fell neatly between those extremes.
The load test took place in a room with one wireless access point with the capacity to handle 60 simultaneously connected devices. The infrastructure worked as designed and therefore it is perhaps unsurprising that the test was successful. However, in a real exam situation it is possible that students might bring smart devices that are already connected to eduroam. This would represent an extra load on access points. It would therefore be prudent to ensure that the number of Chromebooks to be used in an exam session should be no more than half the capacity of a room's access point. With this caveat in mind, an eduroam would seem to provide an Internet connection that is as stable as a wired connection. If an institution possesses an enterprise-level wireless infrastructure then the institution's rules, policies and procedures governing online examinations should not require modification to allow for Chromebook-based exams.

F. Logistical requirements
The project team purchased a lockable charging cabinet that offered a secure location to house the Chromebooks in addition to a power management system that enabled all devices to be charged at the same time. In real-world trials it was found that the Chromebooks could be used actively for ten hours without needing to be recharged (the standby time is of course much longer); this level of battery performance meant that academics could have confidence that the devices would be available for use by students throughout a typical working day.
The charging cabinet was situated in a central location, from which the Chromebooks had to be transported to a SHORT PAPER CAN WEB THIN CLIENTS BE USED TO CREATE FLEXIBLE ASSESSMENT SPACES IN A HIGHER EDUCATION SETTING? number of different locations across the campus. Each Chromebook had a mass of 2.1kg. Although the cabinet was capable of being transported, the combined mass of charging cabinet and a full device load was 131 kg. It was thus deemed to be impractical to transport the cabinet to the examination or classroom venues. Two foam-lined flight cases were therefore purchased to transport the devices from the charging location to the venues where they were to be used.
For many institutions, the logistical issues involved in transporting devices from a central location to spatially distributed sites might prove to be the biggest challenge in offering a centrally-supported examination service based on Chromebooks. For those institutions a more feasible and sustainable solution might be to devolve the service to departments and base the charging cabinets and devices closer to the point of use.

IV. FUTURE DEVELOPMENT
This work has looked at the potential of portable web thin clients to deliver objective test style examinations, which are the most widely implemented types of computer assisted assessment. On the surface, the same devices would appear to be suitable for permitting students to word process their traditional essay-type examinations. The use of word processors in essay-type examinations is well established in law schools in the US, but there are relatively few examples of this practice within UK universities [11]. Given that examinations are stressful for those taking them, and given students' increasing familiarity with keyboards, it seems reasonable to offer students the choice of typing or handwriting their essays. This is planned to be the next stage in our investigation of the use of web thin clients in an examination setting. The current study has shown that the technology itself is unlikely to be an issue in such an investigation; of more concern will be issues of fairness. Will students opting to use a word processor be at an advantage, or perhaps even at a disadvantage, over students opting for handwriting? It is known that in a given period of time students will typewrite more words than they will handwrite, and the amount of writing correlates with the essay mark [12]. On the other hand, students are unlikely to have taken a written examination using a computer, and it is possible that their approach to constructing an essay (planning, editing and revising it) will be different and lead to differences that might be picked up by the marker [11]. It is also important to understand, and if necessary account for, the reactions of markers to word processed versus handwritten scripts [13,14]. These will all be topics of future research.
The portability and simplicity of these thin client devices opens up a new space in which innovation can occur, outside of the examination setting described above. By putting the devices into classroom mode, and allowing teachers and students the freedom to explore the possibilities afforded by the devices, we noted that a variety of technology enhanced active learning (TEAL) techniques were trialled. The devices were used to facilitate online polling, collaborative activities such as wikis, interactive mobile presentations and more. Our observations about this innovation will be published elsewhere.

V. CONCLUSION
We have shown that, within a suitably equipped wireless infrastructure, web thin clients can be used to create flexible assessment spaces. The security and management features of such client devices are sufficient to provide an appropriate platform for high-stakes summative assessment, and an institution's existing policies and regulations surrounding online assessments should require no amendments to take account of the fact that wireless as opposed to wired connections are being used. Usability and accessibility options are adequate, although students need to be given the option of using external pointing devices. With the exception of a specific subset of questions, existing question banks can be used without amendment.
However, in order to deliver an assessment service based upon web thin clients, particularly if those devices are also to be used for in-class use, care must be taken to understand the implications of using multiple device "containers". In particular, the duration required to swap devices from one container to another must be factored in when setting up an online examination. Furthermore, the logistical issues outlined in section 3.6 must be discussed and addressed.