Automatic Assessment of Progress Using Remote Laboratories

— In this paper we present an automatic assessment model for the development of competencies in a physics course using VISIR remote experiment, based on a rubric, and using learning analytics techniques to process data automatically collected from students’ activity using Weblab-Deusto platform.


I. INTRODUCTION AND BACKGROUND
The University of Deusto (UD) introduced in 2001 a competence-based Learning Model (UD-LM), with the student as the center of the learning process and the teacher as a facilitator [1].In this context we define competence as "the integration of a series of elements (knowledge, techniques, attitudes, procedures, values) people brings into play in a specific problematic situation, demonstrating they are able to solve it" [2].
The UD-LM includes five steps to achieve the objective of an autonomous and meaningful learning (see Fig. 1).The main innovation of the proposed model, compared to traditional teaching and learning, focuses in the last two phases: active experimentation and assessment and evaluation.
This learning model relies on active experimentation for the development of competencies and to consolidate knowledge and skills: that is why in technical careers the fourth phase, active experimentation, is even more relevant [3].In this sense, one of the most innovative technologies is Remote Laboratories, which are real experiments accessed through a web interface [4].Assessment is the other phase of the model we want to highlight due to its importance.If competences are not only based on the acquisition of content, assessment cannot be the result of a unique theoretical exam.A competency-based model requires evaluation methods adapted to their nature, this is, the assessment activities and the learning activities must be of the same type [2].Assessment has to be continuous, developed over time, and taking into account not only the results but the process itself.If the concept of continuous assessment demands teacher higher involvement and dedication than other traditional methods, when we talk about instrumental competencies, predominant in technical

II. THE CHALLENGE
Nowadays, an exercise done with WebLab-Deusto VISIR is corrected manually.Teachers access students' exercises and evaluate them according to the competences students are supposed to have acquired when completing the activity.If no additional tools were available, the evaluation with VISIR would not differ from evaluating activities in a traditional laboratory: the teacher should check each student's post and analyze the circuits built with their corresponding measurement instruments.This way of evaluating is not efficient with an average number of students; moreover if we want to do it in a continuous way, for example two exercises per student and session.It is very time consuming and involves the problems derived from a "visual" evaluation.As an example we showed the assemblies in figure 5: even if they are equivalent and work properly, they are difficult to put on the same level (even they have only one resistance as a component).This problem increases as the difficulty of circuits do.In addition, only the result of the activity is assessed, the final deliverable, and teachers do not know the process carried out to obtain it.

III. PROPOSED SOLUTION
Here is where Learning Analytics (LA) techniques [11][12][13][14] apply.In our case LA is possible thanks to a software layer over VISIR, the WebLab-Deusto platform, which gathers data about the circuit and the configuration of measurement instruments in the laboratory.WebLab-Deusto has been developed to register a great amount of data from students' interaction with experiments.Available data are rich and very useful because we can analyze not only the final result -the circuit students have to design with VISIR -but we have the trace, and thus the whole process to get it.
Each time "perform experiment" button is clicked (see fig 3, right-low button), the system records that information in a data-base.That is what we call "system traces", and Learning Analytics techniques enable their use to analyze students' behavior.
Data collected from user-system interaction, even does not records all single activity, provides a great amount of valuable data for analysis.Each "perform experiment" button click involves a row in a table as shown in figure 6.
Learning Analytics techniques enable their exploitation for prediction.Collected traces enable the analysis of students: built circuits, session information (date, time and duration), errors, etc.These data can be processed for an individual student, for a group, a session, a period of time, or other type of mixed analysis.We can obtain usage statistics, predictions and deeper knowledge about students' learning [15][16].However, since remote laboratories are an innovative learning resource, no tools are available for automatic evaluation.
In this paper we propose a model to assess students' competences that relies in two elements: 1) data provided automatically by the VISIR experiment running over WebLab-Deusto platform, and 2) a rubric designed by the teacher with information about expected results of the activity.These data, processed using Learning Analytics techniques, allow teachers to obtain valuable information about students' behavior and their progress; it permits to personalize learning and predict results.

IV. REQUIREMENTS AND DESIGN GOALS
The proposed model follows all stages of Learning Analytics process [14] (Fig. 7): it starts with automatic data collection from teachers' and students' interaction with WebLab-Deusto.Following, integrated datasets are pre-processed to remove useless data and separate circuit design from additional information.Further process contrasts teacher's rubric with students' activity, resulting in the comparison of circuits and additional reports about student's progress.
The outcome of the model feeds back the teaching and learning process for improvement.
The final goal of the proposed model is to provide teachers valuable information to assess students' knowledge and progress, as well as to redesign their teaching process.On the other hand, students benefit from a tool to self-assess their knowledge and skills.
The model has to fulfill the following requirements: • Data collection from Weblab-Deusto is done transparently to the user.
• Students' assessment has to be done with the same tool they have been practicing.
• The interface of the rubric must be simple, so it eases teachers' work.
• The results of the automatic assessment have to be clear and understandable for the final user.
Different tools are implemented to match the requirements: Rubric, Trace analyzer, and Comparator and analyzer.

A. The rubric
In education a rubric is a set of criteria and standards used to carry out subjective assessment about the performance level or a learning activity, allowing more simple and clear score.We can see experiences where rubrics are used [17], as well as tools to generate them [18].
In our case the rubric includes two parts: 1) the exercise solved by the teacher using VISIR, and 2) additional data to interpret the result, because acquired competences should be different depending on the type of exercise (if it is test, analysis or evaluation activity).
In Figure 8 we show part of the rubric designed to assess students' knowledge using VISIR in a physics course.It specifies the competence to be developed (specific competence and level of mastery), indicators and their weight in the overall score.Each indicator has five descriptors.The teacher has to select the elements that define each descriptor and balance to reach the 100% of the indicator.Descriptors within each indicator can be described in terms of things students have to do (correct answer) or in actions students should not have done (errors).When teachers create the rubric they have to select, among all possible options, those that assess they competence in the task or errors that are serious enough to invalidate the exercise.
When students complete an exercise using the VISIR remote experiment through Weblab-Deusto, it records data about the interaction with the system, such as the number and position of components in the circuit, connections, power supply, measurement tool, etc.
The rubric and the student's exercise's trace is the input for the evaluation process.

B. Trace Analyzer
This module analyzes traces provided by Weblab-Deusto.It has to be able to distinguish the elements not involved in the circuit such as time, IP address, system failures, etc.The execution provides as an output a "clean" dataset that is the input to the last part of the process.However, initially separated data are useful for analysis later in the process.
From this information you can extract the configuration of developed circuit.In Table 1 you can see an example: the user has built a circuit with a resistance of 1K, connected in series with another resistance of 1K, and both connected in parallel with a third resistance of 1K; power source of +25 V has been connected; the multimeter is ready to measure resistance values (even it is not connected).
We merge information about the developed exercise and information about the objectives of the learning activity, the rubric from the previous step.The teacher will have to express what means a correct activity, what a permitted error is, and which characteristics are not taking into account at that moment of the learning process; and in each case he/she will have to assign grades to each item.At this stage we have all data needed to analyze and assess student's learning.

C. Comparator and Analyzer
Finally, the model compares the rubric designed by the teacher (circuit and additional data) with the dataset obtained from students' traces.
Once data have been analyzed we have to show them in a friendly and understandable way to the user (teacher and student).With that purpose in mind we have developed a software (see figure 9) that shows clearly the differences and similarities between the teacher's and student's exercise, to obtain not only a summative evaluation, but formative.
This software could be used by students for their own evaluation and feed-back.
The support to assessment provided with this tool not only saves time dedicated to this task, but makes it more objective.

V. MODEL STATUS AND FUTURE WORK
The model has been reviewed by several physics teachers that use VISIR at de UD.
In addition, in a workshop held within the XVI International Symposium of Educational Computing (SIIE 2014) [19], we have demonstrate in practice our work to support the assessment of competencies using Remote Laboratories, Visir in particular, combined with Learning Analytics techniques.During this workshop participants were able to test themselves its utility.
At this moment we are testing developed tool with data registered from 2012-2013 and 2013-2014 courses.
The initial implementation of the model focuses on direct current circuits, because they represent a more varied type of exercise and can take advantage of automatic evaluation.However, tools being developed take into account later usage for the assessment of alternating current circuits.
In the line with other authors [15][16], [20], and based on the results of assessment, in the future we will provide recommendations both to students, about their learning, and to teachers, about the design of the course.