An open learning environment for the diagnosis, assistance and evaluation of students based on artificial intelligence

—The personalized diagnosis, assistance and evaluation of students in open learning environments can be a challenging task, especially in cases that the processes need to be taking place in real-time, classroom conditions. This paper describes the design of an open learning environment under development, designed to monitor the comprehension of students, assess their prior knowledge, build individual learner profiles, provide personalized assistance and, finally, evaluate their performance by using artificial intelligence. A trial test has been performed, with the participation of 20 students, which displayed promising results


INTRODUCTION
Interest in student-centered learning has grown dramatically during the past few decades.Technological developments such as the adoption of computers and the World Wide Web (WWW) brought about teaching and learning approaches which were infeasible or even inconceivable a few decades ago.
Open Learning Environments (OLEs) have been in the center of educational research during the past two decades [1], with research booming during the last few years [2][3][4][5][6][7], greatly aided by the global adoption rates of the WWW.Numerous different OLEs have been developed, each following a different approach and for various educational purposes [8][9][10][11].Higher education institutions are increasingly moving towards the WWW for the delivery of material and or courses [12], with particular interest in OLEs [13].
In OLEs, as well as in computer-assisted education in general, the system has to adapt to the needs of the students if the delivery of personalized education is to be effective [11,14,15].Diagnosing the cognitive capability of the student is crucial for the development of adaptive systems, making the monitoring and evaluation of the students a critical research subject about OLEs [16][17][18].
Monitoring is the process of measuring progress and continuous assessment within a process with the purpose of teaching [19].Perhaps the most challenging tasks in education are the monitoring and evaluation of students, whether during theoretical educations or over practical experiments.The difficulty of such tasks is increased in real classroom conditions, as the monitoring, profiling and evaluation of students needs to take place almost simultaneously.Monitoring the sequence and duration of the interactions a student has with the educational materials of a course in real time allows for the development of a learning behavioral profile for each individual.A common problem in education is that evaluation usually takes place by taking into account the end result alone.Individualities, such as the total time a student took to solve the problem, the number of commands executed and the route the student has followed are usually ignored or, in the best case, only qualitatively considered [20,21].The use of artificial intelligence and the monitoring/logging of a student's actions also allows for the monitoring and evaluation of the students through every step of the educational process [22].This may be used in order to diagnose the qualities and weaknesses of a student, allowing the provision of personalized support, taking into account the entire problem-solving process rather than just the end result [23].
This paper describes the five basic subsystems of a dialogue-based open learning tool under development, designed to monitor the comprehension of students, assess their prior knowledge, build individual learner profiles, provide personalized assistance and, finally, evaluate their performance by using artificial intelligence [24][25][26].The Student Diagnosis, Assistance, Evaluation System based on Artificial Intelligence (StuDiAsE) under development is an open interactive learning system based on the text comprehension theory by Denhière & Baudet [27] and dialogue theory [28].The five basic subsystems of StuDiAsE are: 1.
The monitoring subsystem 2.
The logging subsystem 3.
The modeling subsystem 5.
The evaluation subsystem Figure 1 displays how these subsystems are linked to the main database and between each other.The operation of these subsystems is imperceptible by the students, as StuDiAsE provides personalized educational material and support based on the profile and performance of the student.The profiling, modelling and evaluation of the students is being performed by the use of artificial intelligence and, specifically, fuzzy logic [29,30].Through the use of artificial intelligence and by exploiting the data logged during the educational process, StuDiAsE is capable of deriving personalized student profiles.These profiles can then be used to assess the capabilities and weaknesses of a student, as well as for their evaluation [31,32].
The five subsystems which have been developed for a research trial are being presented in the following paragraphs in detail, followed by the results of the trial which took place for an engineering course module.

A. Aim
The monitoring subsystem, as the name suggests, monitors and logs the actions of students, in order to collect all those selected actions which take place from when the student begin using the system and until the final exit from it.Data collected during the navigation of the student may be correlated with the prior knowledge (theoretical background), skills to use new technologies, attitudes and patterns during his use of the learning environment, as well as the times and points where assistance has been sought.The objective of the subsystem is the logging of sufficient data, in terms of both number and quality, which can be then used to build a profile for the student and provide personalized material and assistance.

B. Types of Information
StuDiAsE is capable of monitoring and logging several different types of information, depending on the required application.For the means of this trial study, the types of information recorded have been selected from a series of research studies, references and based on the experience of the research team members.Specifically, the following basic types of information are being collected: 1. Static information, such as the name and registration number of the student.
2. Dynamic information, such as the total time spent in the educational environment, the time devoted to each of the questions, the number and order of the questions which have been answered, as well as the number of questions answered correctly.
3. Other information which are relevant to the test, such as the frequency and type of assistance which the student requested, the type and frequency of errors committed by the student and specific information derived from the navigation of student within the learning environment and which can be used during his evaluation.

C. Subsystem Functions -Monitoring and Logging
Process When the student begins using the educational environment, the monitoring subsystem automatically triggers and records the actions taken by the user.The student is required to enter all the vital static information used to identify him and the logging subsystem initiates, standing by to receive the dynamic information.At the same time, it initiates the system timer, which records the time required for every action of the student.Any response or action (help, skipping a question, etc.) by the student is then being recorded.Upon the exit of the student from the learning environment, the monitoring subsystem completes a report and records it to the database, filed under the static information provided by the student.

III. LOGGING SUBSYSTEM
The logging subsystem operates in parallel with the monitoring subsystem, recording the answers given by the student alongside the recording of the actions taken during the process.By recording responses to questions on a specific theme it is possible to reveal the educational needs of the student: the inert knowledge, misconceptions, contradictions, gaps, etc.Similarly, the recording of responses facilitates the analysis, processing and coding of the arguments of the student, in order to form the initial cognitive profile.
The output of the logging subsystem includes information: 1.
for the particular student , 2.
for the selected module / sub-module 3.
for the selected test, a specific number of questions and alternative answers per question 4.
for the kind of questions / answers: multiple choice, justification, matching, fill the gap. 5.
for the type of questions: a. Questions R -type (relational) b.Questions M -type (transformational) c.Questions T -type (teleological) 6.
for the total number of questions and the individual number of questions per question type and per subject The above information is used by other subsystems of the system, which are being discussed in the following paragraphs, for the diagnostic, modeling and evaluation processes.

A. Aim
The profiling subsystem is designed to extract the original cognitive profile of a student, which represents the prior knowledge on the selected topic based on the options which have been selected by the student.The current status of the student is then represented by particular characteristics, such as level of prior knowledge, knowledge gaps, contradictions, learning PAPER AN OPEN LEARNING ENVIRONMENT FOR THE DIAGNOSIS, ASSISTANCE AND EVALUATION OF STUDENTS BASED ON… style, attitude during the study and his willingness to participate.The aim is to study the characteristics of the learner student which are important for personalization of the environment and those that are expected to arise through interactive diagnostic feedback process.
The aim of the diagnostic process is utilizing the diversity of learning needs and abilities of students, who will be identified based on their responses to questions (cognitive profiles) to set the main educational objective and design interactive feedback.
Objective of the profiling subsystem also is the investigation and evaluation of possible ways to engage students in the diagnostic process, which aims for the proper generation of a cognitive profile.

B. Educational strategies
The research team has developed a series of educational strategies for the extraction process of the cognitive profile.These strategies are based on research studies and references, but also in the teaching experience of the team members.
Logged results of the diagnostic tests, including appropriate questions with alternative answers related to the theme, must be interpreted by the profiling system.
There are three types of questions, equally numbered: a. Questions R -type (relational) b.
Questions T -type (teleological) It is educationally useable to record the number of successful responses to questions separately, R-type, Mtype and T-type, but of all the questions as well.This focuses on identifying the skill of the student to respond successfully to questions of R-type, M-type or T-type, as well as any of their combinations.
1st example: Such is the case of identifying the skill of the student able to only answer R -type questions successfully.This success is interpreted as the skill of the student to do: A description of the units that compose a technical system A description of the part -whole relationships that connect units in the system between them A description of situations which the parts of the system can be at Description of events and complex events consisting of sequences of events and performed by the parts of the system A description of the part -whole relationships between the events hierarchy Therefore, we consider that the student with 76% -100% successful responses to R -type questions is able to make the above descriptions.This is an indication that the student has adequate knowledge of relational texts and the system should indicate the student to avoid texts of relational structure.It is being suggested that student should pursue educational material only of transformational structure.

Initial cognitive profile description
Profile abbreviation R-High -Low -High -High PAPER AN OPEN LEARNING ENVIRONMENT FOR THE DIAGNOSIS, ASSISTANCE AND EVALUATION OF STUDENTS BASED ON… 2 nd example: Such is the case of identifying the skill of the student able to only answer M -type questions successfully.This success is interpreted as the skill of the student to perform a description of causal and temporal relationships between events and identify changes / transformations which are being caused on the situations of the technical system.
So we consider that the student with 76% -100 % successful responses to M -type questions is able to make the above descriptions.This is an indication that the student has adequate knowledge of transformational texts and the system should indicate the student to avoid texts of transformational structure.It is being suggested that student should pursue relational texts based on teleological texts.
3 rd example: In the case of identifying the skill of the student to answer questions only of T -type, this success is interpreted as a skill the student to perform a description of the system through a teleological tree of targets and sub-targets for each transition of the system from one state to another.
So we consider that the student with 76% -100 % successful responses to T -type questions is able to make the above descriptions.This is an indication that the student has adequate knowledge of teleological texts and the system should indicate the student to avoid texts of teleological structure.It is being suggested that student should pursue educational material based on relational texts at first, then based on transformational texts.The student with 76%-100% successful answers in questions of R-type and -type and -type has a cognitive profile of type: R/ / -High. 2 The student with 76%-100% successful answers in questions of R-type and -type has a cognitive profile of type: M/ -High. 3 The student with 76%-100% successful answers in questions of R-type and -type has a cognitive profile of type: R/ -High.

4
The student with 76%-100% successful answers in questions of -type and -type has a cognitive profile of type: / -High.

5
The student with 76%-100% successful answers in questions of R-type and -type has a cognitive profile of type:R/ -High.6 The student with 76%-100% successful answers in questions of R-type has a cognitive profile of type: R-High.7 The student with 76%-100% successful answers in questions of -type has a cognitive profile of type: -High.8 The student 76%-100% successful answers in questions of -type has a cognitive profile of type: -High.9 The student with 26%-75% successful answers in questions of R-type and -type and -type has a cognitive profile of type: R/ / -Medium. 10 The student with 26%-75% successful answers in questions of R-type and -type has a cognitive profile of type: R/ -Medium.

11
The student with 26%-75% successful answers in questions of R-type and -type has a cognitive profile of type: R/ -Medium. 12 The student with 26%-75% successful answers in questions of -type and -type has a cognitive profile of type: / -Medium.

13
The student with 26%-75% successful answers in questions of R-type and -type has a cognitive profile of type: R/ -Medium.14 The student with 26%-75% successful answers in questions of R-type has a cognitive profile of type: R-Medium.15 The student with 26%-75% successful answers in questions of -type has a cognitive profile of type: -Medium.16 The student with 26%-75% successful answers in questions of -type has a cognitive profile of type: -Medium.17 The student with 0%-25% successful answers in questions of R-type and -type and -type has a cognitive profile of type: R/ / -Low.

18
The student with 0%-25% successful answerss in questions of R-type and -type and has a cognitive profile of type: R/ -Low.

19
The student with 0%-25% successful answers in questions of R-type and -type has a cognitive profile of type: R/ -Low.

20
The student with 0%-25% successful answers in questions of -type and -type has a cognitive profile of type: / -Low.

21
The student with 0%-25% successful answers in questions of R-type and -type has a cognitive profile of type: R/ -Low.22 The student with 0%-25% successful answers in questions of R-type has a cognitive profile of type: R-Low.23 The student with 0%-25% successful answers in questions of -type has a cognitive profile of type: -Low.24 The student with 0%-25% successful answers in questions of -type has a cognitive profile of type: -Low.25 The student with 76%-100% successful answers in questions of R-type has a cognitive profile of type: R-High.26 The student with 76%-100% successful answers in questions of -type has a cognitive profile of type: -High.27 The student with 76%-100% successful answers in questions of -type has a cognitive profile of type: -High.PAPER AN OPEN LEARNING ENVIRONMENT FOR THE DIAGNOSIS, ASSISTANCE AND EVALUATION OF STUDENTS BASED ON… 4th example: Where success rates are the same in all types of questions and low (0% -25% successful responses to questions of R-type or M-type or T-type), it can be assumed that the student has no particular skills or strengths.In such cases the system should guide the student to begin his study with educational material of relational structure.

C. Cognitive profile structuring
The educational strategies set by the team members led to the determination of the structure of the cognitive profile and the classification of its potential cases.Thus, the possible structures of profiles presented in this study are i) descriptive notation per answer for each question type (R-type, M-type or T-type), with the possible classifications being high, medium or low profile and 2) numerical notation in percentages: 0%-25%, 26%-75% and 76% 100% respectively.Table 1 shows in detail the 27 possible cases cognitive profile and the corresponding abbreviations for the characterization of student profiles.

D. Profile generation rules
According to the data of table 1 and the educational strategies set by the team members, a set of rules has been formulated in order for the artificial intelligence of the profiling subsystem to function.These are being summarized in table 2.

E. Subsystem operation -process of the cognitive profile generation
The profiling subsystem takes as input the output of the logging subsystem, which recorded the answers which the student selected over a specific set of questions.Considering the rules displayed in table 2 and in conjunction with the data recorded by the logging system, the initial cognitive profiles are being generated.
Thus, if a student partakes a diagnostic test in order to determine his initial cognitive profile consisting of, for example, 15 questions total (5 of each type) and the logging subsystem records 4 or 5 correct answers out of the 5 questions for a specific type, then the student's skill over texts of this type is considered high.Accordingly, if the student has 2 or 3 correct answers, then his skill is considered medium, while 1 or 0 correct answers would rate his skill as low.
For example, if the student answers correctly 5 R-type based questions, 4 M-type based questions and 1 T-type based question, then the initial profile of the student will be set as R/M -High and the system will provide personalized text and assistance over teleological texts.Accordingly, if the student answers correctly 1 R-type based questions, 5 M-type based questions and 0 T-type based questions, then the system would diagnose the student as of M-high type, offering personalized assistance primarily over texts and examples of relational type, then of teleological type.According to the initial cognitive profile of the student, which has been generated by the profiling subsystem, the modeling subsystem begins the assembly of a student model.This includes: The initial cognitive profile.Personalized feedback, according to the results of the initial cognitive profile.
The final cognitive profile obtained after any additional activities have been partaken.
The feedback is given to the student after the initial diagnosis as an personalized activity that includes text and questions with alternative answers.Additional feedback can be given in the form of assistance, in the form of suggestions-advice, in the form of didactic instruction, in the form of examples, or any combination of the above, in order to achieve the best possible learning and diagnostic result.There are 26 possible initial cognitive profile cases which activities in table 3.As it can be seen, the artificial intelligence is programmed to provide personalized feedback to students in order to first maximize their relational text comprehension, then their transformational text comprehension and finally their teleological text comprehension.
The final cognitive profile includes any changes which may have occurred on the initial cognitive profile of the student after the personalized feedback procedure.The student is provided with the same exact test which has been used to generate the initial cognitive profile, assuming that the answers to the initial questions did not became known to him and or were not included in any of the personalized tests provided during the feedback process.Otherwise, a test with similar questions and of the exact same difficulty level may be used.Table 4 displays an example of a student with an initial cognitive profile designated as R/M-Medium, which student improved his cognitive skills through personalized feedback based on a relational educational text.

Final cognitive profile R-High -Medium -Low
As it can be seen, the profile of the student has changed from R/M-Medium to R-High.The next step for the system would be to provide personalized assistance based on M-type educational material, in order to improve the comprehension of the student on texts of transformational type.If the comprehension of the student improves until the final cognitive profile becomes R/M-High, then the system will proceed to provide assistance with teleological educational material.

VI. EVALUATION SUBSYSTEM
The involvement of students throughout the entire process depends on individual decisions, from answers and movements, the willingness to participate, from the compliance with instructions and encouragement offered by the system in various phases.
If the student persists on or wrong the artificial intelligence system should be designed to seek the minimization of conflicts and focus on trying to change the reasoning of the student.The minimization will be possible when the learner alone removes the contradiction and thus becomes able to construct a more coherent argument (reflection) [27,33].
The evaluation subsystem has as its core the student model.By using artificial intelligence techniques, it is possible to evaluate the details of the initial and final cognitive profile.In this subsystem fuzzy logic techniques have been applied, which were using as inputs the following items, while the output of the system is the cognitive profile of the student.
A) Recorded data of student involvement during modeling Elements indicating the engagement of the student in the diagnostic process: informing the student for initial cognitive profile Elements indicating the engagement of the student in the process of creating a cognitive profile and model: the number of times that the cognitive profile characterization has changed, the student's decision to reconsider contradictory answers to questions / errors, etc. Elements indicating the engagement of the student in the improvement of his cognitive model: steps leading to a change in thinking and changes in the model.B) Recording of system navigation elements Recording data on getting help Recording data of moving between previous and later stages of the activity Recording time intervals corresponding to engagement with each activity Any other information that may be associated with this activity.
The utilization of information not related to answering questions depends largely on the type of exercise and is customizable.The rules should be based on the comparison with the corresponding figures of an expert who solved the same exercise.

VII. TRIAL TEST
A trial test has been performed in order to assess the functionality of the subsystems, gauge the response of the students, iron out any software bugs and create proper fuzzy logic rules.The trial is based on educational material of the "Foundations of Energy" module, currently taught in the MSc of Energy, a course of Heriot-Watt University, Scotland, UK.An expert has taken the first test, setting the standards for time-related functions.Then, 20 volunteers has participated in this study, taking a diagnostic test which has been used to create their initial cognitive profile.The system provided personalized feedback to each one of them and, after the students completed going through the extra educational material provided by the system, they undertook the first test again, with the system generating the final cognitive profile of each student.The whole process is also being PAPER AN OPEN LEARNING ENVIRONMENT FOR THE DIAGNOSIS, ASSISTANCE AND EVALUATION OF STUDENTS BASED ON… performed by an expert in parallel, in order to identify any flaws in the diagnostic and or evaluation processes.Two qualitative rules have also been set, one for the time that the student required to take the test and one for the number of times the student sought help through additional material.Table 5 summarizes the two qualitative rules which act as coefficients for the final verdict of the evaluation subsystem.The diagnostic test used in this study includes 15 questions, 5 of each question type (Relational, Transformational or Teleological).Supplementary educational material of each type has been added into the system, which is being provided by the system as personalized feedback after the initial cognitive test.After providing additional educational material to the student once, the student is called to take the first test again, in order to create a final cognitive profile.The results of the trial study are being summarized in table VI.
From table 6, it can be seen that the proposed learning system can improve the comprehension of students on particular types of test over a single trial run.That is particularly true for weaker students, such as students 9 and 10 of our study, which displayed significantly increased performance after going through the supplementary material.There has been a case (#2) which, despite the number of correct answers increased, the system did not improve the final cognitive profile of the student.In this case, analysis of the student's profile indicated that a penalty coefficient of 0.8 has been applied, for taking over twice the time required by an expert to complete the test.Such information can be used by the educator or by a complete system to provide personalized assistance in order to improve the student's time management skills.

VIII. CONCLUSIONS
The implementation of each subsystem has been tested by members of the research team, repeatedly, to improve and to find any major problems associated with the structure, design and compatibility of the subsystems.Further tests were carried out by groups of volunteer students in the Heriot-Watt University, in the form of a trial test based on the «Foundations of Energy» course, through which important information became known regarding the application of the system on the particular subject.Findings include observations regarding the proper modification and delivery of educational texts, the formulation of questions, the format of the rules and the severity and number of parameters to be taken into account in any subsystem.The trial test results were very positive, especially considering the improvement that a simple test which has been created for the purposes of a single trial run had on the students.The students responded very well, with high participation levels and positive feedback.Future works will include detailed observations on the number and type of questions, texts and educational material appropriate for different subjects, recommendations on evaluation recommendations questions, as possible improvements on the monitoring and logging subsystems.

Figure 1 .
Figure 1.The structure of StuDiAsE

TABLE VI .
SUMMARY OF THE TRIAL TEST OF THE SYSTEM WITH 20 STUDENTS ENVIRONMENT FOR THE DIAGNOSIS, ASSISTANCE AND EVALUATION OF STUDENTS BASED ON…