Evaluation Plan and Preliminary Evaluation of a Network of Remote Labs in the Maghrebian Countries

The need for adopting virtual and remote labs in engineering education is high in Algeria, Morocco and Tunisia. This paper proposes the evaluation plan of a network of remote labs in the Maghrebian countries. This plan is based on the significant research that has been done until now concerning the evaluation of remote labs. More specifically the proposed evaluation strategy of the project will focus on five different but interrelated directions (a) usability of remote labs; (b) learners’ attitude towards remote labs; (c) technical evaluation of remote labs operation; (d) evaluation of the e-learning content namely the teaching units previously described; and (e) learning outcome. Furthermore, this paper presents preliminary evaluation results based on this evaluation plan and resulted from a pilot case study.


INTRODUCTION
This paper proposes the evaluation plan of a network of remote labs in the Maghrebian countries Morocco, Tunisia and Algeria. This work is carried out within the framework of the European Tempus project called "eSience".
The consortium of this project involves many partners from different countries of different cultures and educational systems. More specifically, the consortium of the project consists of sixteen different partners from seven different countries (France, Austria, Greece, Romania, Morocco, Tunisia and Algeria).
The main challenge of this paper is a. (a) to review the state of the countries where the eSience project is implementing the network of remote labs art solution; b. (b) to set off the characteristics of such a network that should be evaluated and measured, in order to prove that this network is successful or not; c. (c) to present a suitable evaluation plan of such a distributed network of remote labs in the Maghrebian countries.
This paper is structured as follows: in the next section it describes the overall and specific objectives of the eSience project; afterwards, it presents the related work in the area of evaluation of remote labs. Next, it presents the situation in the Maghrebian countries that are participating in the project (i.e. Morocco, Tunisia and Algeria), in order to present the difficulties of the application of the evaluation plan. Afterwards, it presents the evaluation strategy along with the instruments that will be used in order to support this evaluation plan. The next two sections present the case study along with the results.

II. ESCIENCE PROJECT DESCRIPTION
eSience Tempus project "Maghreb Network of remote labs (2012-2015)" launched in January 2013. The objective of the project is to create an efficient remote labs network in the Maghreb region for the modernization of higher education in technological sciences. Morocco, Tunisia and Algeria are involved in this TEMPUS project to create a network of remote labs whose goal is efficiency and the sharing of high performance instrumentation. In addition, the development of teaching units related to distance will train a larger number of technicians and engineers.
This approach enables the implementation of a training system at the international level and the creation of a new generation of versatile graduates able to adapt to a changing global context. This is fully inline with the development strategy of university education in the Maghreb. Among the specific goals of the project are the following: (a) Adaptation of content to the development in science and technology, taking into account the expectations of the professional word; (b) Creation of 3 remote laboratory platforms in the Maghreb and their networking; (c) Implementation of practical courses for on-distance education; (d) Creating teaching units in the e-learning format; (e) Setting up and using an LMS (Learning Management System); (f) Evaluation of the newly created educational resources; (g) Use of the pedagogical resources.
A main objective concerning the development is the interconnection of a well-known Learning Management System (LMS) (e.g. moodle) to specific remote lab architectures. Examples of such type of architecture are the following: iLab (http://www.ilab-europe.net/) and Lab2go (http://www.lab2go.net/fr/). More information about the technical solution of eScience project is available at [9]. General information about eSience Tempus project is available at http://www.esience.org/.

III. RELATED WORK
There are several studies about the impact of remote laboratories in the learning outcomes. Gomes [13] refer that "most existing remote laboratories…offer stand-alone solutions, with limited or no capability to cooperate with other platforms". From this point of view, eSience objective to interconnect LMS with a remote lab platform seems to be a significant advantage. To this direction, various issues should be resolved either form the technical point of view or from the educational point of view. According to [13], the issues that should be investigated are interoperability, accessibility, availability, configurability, and interconnection with collaborative applications. Similar findings are presented in [13]. In this research, the authors refer that there are many significant "logistical" factors that should be considered, such as flexibility, cost, and resource sharing. However, as it is referred in [13], there are many factors that may have an impact on learner's satisfaction and achievement [15], and, thus, they should be investigated. This research showed that "giving learners authority over the physical learning environment and offering different formats for collaboration, either online or in person, contributed to the learners' overall satisfaction with the course structure". Other factors include the relationships between the user and the technology, between the instructor and students, and among the students [13], [16].
A significant work has been presented in [17], concerning a model for evaluating the effectiveness of remote engineering laboratories and simulations in education. This model investigates key aspects about the effectiveness such as (a) Adoption of the lab experiment; (b) Lab procedure/guide; (c) Teamwork; (d) Lab supervision; (e) Data acquisition; (f) Reliability of experiment set-up; (g) Lab report writing; and (h) Experience gained for future career. According to this model, Balakrishnan and Woods [18] have elaborated comparative study on real lab and simulation lab in communications engineering from students' perspective. They found that real and simulation labs are complementary and not competitors.

IV. RATIONALE OF EVALUATION STRATEGY
The need for adopting virtual and remote labs in engineering education is high in Algeria, Morocco and Tunisia nowadays according to [9]. Furthermore, the urban population in Algeria, Morocco and Tunisia represents a very high percentage of the total population. More specifically, the urban population in Algeria is 67.7%, in Morocco is 59.4%, and in Tunisia is 68% of the total population. Therefore, by adopting and evaluating remote labs in urban places we can cover a rather high percentage of the population in all three countries. According to OECD Global Competitiveness Index 2013, these countries have a rather low rank in the Technology readiness rankings among 148 countries. The percentage of individuals using Internet is 15.2% in Algeria [10], 41.4% in Tunisia [12] and 55.0% in Morocco [11]. According to this data we can notice that there is diversity concerning the percentage of individuals using the Internet. However, the technology readiness index in the Maghrebian countries is rather low in general.
The participants of the evaluation will be students and teachers of Universities from Maghrebian countries that are participating in the project (i.e. Morocco, Tunisia and Algeria). The main test bed universities are the following ( Figure 1): Due to the large geographical scale and the number of participating organizations, the limited familiarization of the target groups with remote labs, as well as, the low technology readiness, the evaluation will be conducted in two sequential phases: • Phase I -pilot evaluation • Phase II -large scale evaluation The first phase will be the pilot evaluation of all remote labs in small scale usage. During this phase we plan to assess the remote labs' usability and proper functioning as well as learners' attitude towards remote labs.
The remote labs operation and the user interface will be improved according to the results of the first evaluation phase, and they will be deployed in a large scale usage. After that period of usage, the large scale evaluation will be accomplished. This evaluation will focus on: the usability of the remote labs; learners' attitude towards remote labs; the evaluation of the e-learning content; and the assessment of the learning outcome.
The evaluation strategy of the project will focus on five different, but interrelated directions (a) usability of remote labs; (b) learners' attitude towards remote labs; (c) technical evaluation of remote labs operation; (d) evaluation of the e-learning content namely the teaching units described previously; and (e) learning outcome. Table I presents the eSience evaluation strategy along with the instruments proposed to be used during each phase, in order to achieve each goal.

V. INSTRUMENTS
The instruments that were proposed to be exploited in each phase are presented in Table 3 as well. As referred in [13], the nature of the learning outcomes arising from laboratory experiences has a complex relationship with the characteristics of the interaction modality. Therefore, a research on the impact of remote labs in education should SPECIAL FOCUS PAPER EVALUATION PLAN AND PRELIMINARY EVALUATION OF A NETWORK OF REMOTE LABS IN THE MAGHREBIAN COUNTRIES consider, among other things, the way in which the technologies that are used affect the nature of the interaction. More specifically, in order to measure usability we plan to exploit the "USE Questionnaire" presented by Lund [1] in both evaluation phases I and II. USE stands for Usefulness, Satisfaction, and Ease of use. For many applications, Usability appears to consist of Usefulness and Ease of Use, and Usefulness and Ease of Use are correlated. Concerning learners' attitude towards remote labs we plan to use an instrument adapted from Douka [6] in order for the students to rate the remote labs regarding the following characteristics: Comprehensive, Sensible, Educational, Easy, Enjoyable, Interesting, Satisfactory, Well done, Scientific, Serious, Well prepared, Important, Innovative, Pedagogic, Modern, and Different.
Regarding the technical evaluation of remote labs, the main focus will be given on functional suitability which according to ISO/IEC [5] (SQuaRE) is the "degree to which a product or system provides functions that meet stated and implied needs when used underspecified conditions". It has the following sub-characteristics: functional completeness; functional correctness; and functional appropriateness The e-learning content evaluation is proposed to be done by experts in every field in order to make a formative evaluation. The proposed instrument in this field is the "Checklist for a Didactically Sound Design of eLearning Content" [2]. This checklist covers the content, segmenting, sequencing and navigation, adaptation to target audience, design of text and graphics, learning tasks and feedback, and motivation.
The evaluation of the learning outcome will be conducted by exploiting a knowledge test in two different groups of students: control group and experimental group. The knowledge test will be created by the tutors of each course. The experimental group will use remote labs and e-learning content in order to learn the concepts that will be assessed by the knowledge test, whereas the control group will learn the same concepts by following the traditional educational process. To ensure that the subgroups are equally distributed according to the factors affecting the results, the Felder-Soloman Individual Learning Style (ILS) questionnaire [3] will be performed. This questionnaire finds the preferred learning style of participants. This questionnaire is conducted to isolate the effects that preferred learning styles could have on learning outcomes for the different lab access modes [7].

VI. CASE STUDY
The evaluation of the learning outcome will be conducted in parts. In particular, every educator will undertake to use the knowledge test in a control and a treatment group as mentioned in the previous subsection. Therefore, a pilot case study was conducted at the Hassan I. University -Khouribga -Morocco in order to examine differences between the performance of students who used the traditional method of learning and those that used the remote labs. Moreover, possible correlations between the results from the USE questionnaire were also examined. The desired outcome of this case study was an equal, at least, performance between the two groups and not a better learning outcome from the experimental group. Thus, the null hypothesis of this study is the following: H0: There is a statistically significant difference between the performance of the control and the experimental group.
The methodology that was followed in this case study is described in the following subsections.

A. Research design
The research design that was followed in this study was based on the "Posttest Control Group" [19] model, including one control and one experimental group of randomly assigned subjects. This design offers high levels of internal validity and is often used in educational studies.

B. Measures
The collection of the data was conducted using a knowledge test and the USE questionnaire [1], evaluating the learning outcome and the usability of the remote labs, respectively. The knowledge test was paper-based, whereas the USE questionnaire was answered online. Both questionnaires were administered to the students after the treatment period. Apparently, the USE questionnaire was answered only from the experimental group.

C. Participants
The participants of this study were 72 undergraduate students. There were 39 male and 33 female students and their ages ranged from 18 to 20. The control group consisted of 43 students whereas the participants of the experimental group were 29.

D. Procedure
Khouribga OnlineLab is being used by more than 200 students from different Engineering degrees: Physical Sciences, Industrial Engineering, and Automatic & Telecommunications during the 2013 academic year in the polydisciplinary faculty in Khouribga, and this was their first experience in working with a remote laboratory. This study profiles the Khouribga OnlineLab experience with about 29 students (Treatment Group) from the Electrical Engineering and Industrial Informatics curricula, enrolled in electrical machine and power electronics Modules. These 29 students used the OnlineLab to handle and measure the machine laboratory. This course was distributed over 80 hours. The students could remotely conduct To perform an experiment, students browse to the service broker domiciled at http://www.onlinelabuhp.ma/ilabservicebroker to register and request membership to the group associated with the laboratory. To do this, students must access the service broker, login page and ask to supply their user names and passwords. In addition, The Learning Management System (LMS) Moodle provides students with all the theoretical documentation and other complementary resources.
Another group of 33 students (Control Group) followed the traditional method. Fig. 2 depicts the procedure in detail.

E. Data analysis
The analysis of the data concerning the learning outcome was conducted using an independent samples t-test. This test is a parametric test and it is used for comparing the means between two independent groups on the same continuous, dependent variable. In this study the dependent variable for the two groups was the achievement in the knowledge test. Preliminary checks were conducted to ensure that there was no violation of the assumptions for the use of parametric tests. The statistical analysis was performed using the SPSS 19 statistical package and the level of significance was set to 0.05.
Furthermore, measures of central tendency were computed for all USE questionnaire items as well as the four questionnaire dimensions (combined variables). The latter dimensions, which reflected an overall satisfactory reliability, included 8 'usefulness' items (Cronbach's alpha = 0.72), 11 'ease of use' items (Cronbach's alpha = 0.80), 4 'ease of learning' items (Cronbach's alpha = 0.83) and 7 'satisfaction' items (Cronbach's alpha = 0.63). Moreover, a Pearson product-moment correlation coefficient test was conducted measuring the relationships among the questionnaire dimensions.

VII. RESULTS AND DISCUSSION
The results from the independent samples t-test showed that there was no statistically significant difference between the learning achievement of the control group (M = 13.41, SD = 2.91) and the experimental group (M = 12.52, SD = 2.91) having the following results: t (70) = 1.29, p = 0.200. Therefore, the null hypothesis (H0) of the study is rejected.
These results meet the desired outcome that was set for this study. In particular, the control group did not achieve better learning outcome than the experimental group. Thus, it is safe to conclude that a remote lab can efficiently replace the traditional method of teaching. This replacement will produce the same learning outcome for students and it will also provide several advantages such as the opportunity to access special equipment and tools from a distance, the opportunity to use expensive tools with no cost, and the safety to participate remotely, in an experiment that may otherwise expose humans to danger. However, these results are preliminary and a more thorough investigation, by implementing more case studies, is required. As aforementioned, the effects on learning outcome will be examined in parts until we have data from all the educators. Then, it will be possible to have a more complete understanding and therefore to generalize the results. Table II depicts the results emerging from the analysis of all the post-task USE questionnaire variables.
Likewise, Tables III present the descriptive statistics calculated for the four post-task questionnaire dimensions. The study findings reveal a positive students' opinion towards the usefulness, ease of use, ease of learning and satisfaction of the remote labs session. Table IV illustrates the statistically significant results of the Pearson product-moment correlation tests that were conducted to assess the relationship among the USE questionnaire combined variables. It was indicated that the students who were satisfied or perceived the remote lab interface as easy-to-use believed that the remote lab session was useful. Additionally, two positive correlations were reported between the 'ease of learning' and both the 'usefulness' and 'ease of use' variables. This result shows that the students that are able to readily understand how remote labs are operated tend to rate the remote lab session as considerably more usable and useful.
A Pearson product-moment correlation coefficient was also computed to investigate the relationship between the learning outcomes and the USE questionnaire variables. The results of the correlation analysis did not yield any statistically significant correlations among the compared variables. This paper had as goal to present an evaluation plan of a network of remote labs in the Maghrebian countries. Following a short description of the eScience European project along with its remote lab network, a brief overview of relative evaluation strategies has also been presented. It seems that a considerable amount of research work concerning the evaluation of remote labs has been done until now. Based on this work this paper has presented an evaluation plan that tries to meet the special characteristics of Maghrebian countries concerning their technology readiness. More specifically, the authors have outlined an evaluation model for testing the relative effectiveness of remote laboratories in these countries. We expect this model to form the basis for more robustly designed future case studies, which could be valuable for educators seeking to understand how the choice of laboratory technology can affect the educational outcomes.
Moreover, we have presented a preliminary case study exploring the impact of both the remote and hands-on labs in the context of a university course offered by the Hassan I. University -Khouribga -Morocco. Though limited in scope, this case study was based on the evaluation plan proposed in this paper. The study findings showed that there was no statistical difference between the learning outcomes of the control (students using traditional labs) and treatment groups (students using remote labs). In addition, the results revealed that the students who participated in the remote labs session were satisfied and had a positive opinion about the ease of use, ease of learning, and usefulness of the remote labs used.

ACKNOWLEDGMENT
This project has been funded with support from the European Commission. This publication reflects the views only of the author, and the Commission cannot be held responsible for any use which may be made of the information contained therein.