Educational Recommender System Based on Learner’s Annotative Activity

—In recent years, Educational Recommender Systems (ERSs) have attracted great attention as a solution towards addressing the problem of information overload in e-learning environments and providing relevant recommendations to online learners. These systems play a key role in helping learners to find educational resources relevant and pertinent to their profiles and context. So, it is necessary to identify information that helps learner’s profile definition and in identifying requests and interests. In this context, we suggest to take advantage of the annotation activity used usually in the learning context for different purposes and which may reflect certain learner’s characteristics useful as input data for the recommendation process. Therefore, we propose an educational recommender system of web services based on learner’s annotative activity to assist him in his learning activity. This process of recommendation is founded on two preparatory phases: the phase of modelling learner’s personality profile through analysis of annotation digital traces in learning environment realized through a profile constructor module and the phase of discovery of web services which can meet the goals of annotations made by learner via the web service discovery module. The evaluation of the developed annotation-based recommendation system through empirical studies realized on groups of learners based on the Student’s t-test showed significant results.


Introduction
Recently, the recommender systems are presented as a new technology in educational context to deliver a learning support to learners. These advanced technologies are increasingly being used in educational practice and as convenient platforms for rigorous educational research. These systems play a key role in assisting learners to find educational resources relevant and pertinent to their profiles and context [16]. So, it is necessary to identify information that helps learner's profile definition and in identifying requests and interests [33].
In order to generate personalized recommendations, it is important to identify information through different learner's activities that will help to define user profile and find useful information and relevant resources to carry out their training [2]. Among these activities, we focus in our research on the annotative activity of learner because annotation practice is very common and omnipresent [13]. While reading, the learner usually uses comments, highlights and posts it to annotate the consulted resources [3; 7].
In our work, we suggest to take advantage of the annotation activity used usually in the learning context for different purposes and which may reflect certain learners' characteristics useful as input data for the recommendation process. Therefore, we present the architecture of an educational recommender system that bases their recommendation services on learner's annotation traces yielded during the learning process. The educational recommender system is composed essentially of four basic modules: annotation module; web service discovery module; profile constructor module and recommendation module. The evaluation of the two principal modules (web service discovery module & profile constructor module) through empirical studies realized on groups of learners based on the Student's t-test showed significant results.
The rest of this paper is structured as follow. In Section 2, we give a brief overview on the literature of recommender systems in the educational context. Thereafter, Section 3 presents the architecture of our system and details its principal components. Section 4 evaluates the two main modules constituting the architecture of the proposed educational recommender system. Finally, we draw some conclusions and we cite certain possible directions for future works.

Related Work
Recommender systems are presented as a new technology in educational context to deliver a learning support to learners. These advanced technologies are increasingly being used in educational practice and as convenient platforms for rigorous educational research [16]. In this context, many educational recommender systems are designed with different functionalities and recommendation services [33].
There are mainly three approaches in educational recommender systems, the content based filtering, which is based on the description of the item and works with data provided by learner, either implicit or explicit in order to give recommendations, the second approach is the collaborative filtering its principal object is to offer to the learner the items that are relevant to the users who are close to him in a certain way, and finally the hybrid filtering which combine different approaches to give a better recommendation and increase customer satisfaction [5; 9].
In order to recommend items for the target student, similarities between him and other learners are computed based on their common taste. This approach is called user-based approach. A different way to recommend items is by computing the similarities between items in the matrix. This approach is called item-based approach [12].
Certain works transfer the technology of recommender systems from commercial to educational contexts on a one-to-one basis regarding the datasets and methods used to deliver recommendations without taking account to the particularities of learning environment [15]. For instance, certain learning portals integrate recommender engines to assist their users during their learning experiences [30]. In order to allow the recommender engines to produce an efficient recommendation, the system collects datasets which include such usage related data source recommendation (ratings, votes, tags, reads or downloads, bookmarks, etc.) and apply data analysis techniques (collaborative filtering, content-based filtering and hybrid filtering technologies) to help users find items that are likely of relevance [25; 32]. Based on these analysis techniques and data sources recommendation, we present in Table 1 a comparison between some education recommender systems. From this study, we deduce that recommendation in the learning context is more challengeable than in other contexts. In fact, the educational recommender systems deal with information about students and learning activities which means that it should be personalized with consideration to learner's characteristics (level of knowledge, learning activities, learning achievement, learning goals, learning style, personality traits, etc.). Thus, the posed issue concerns the data to be gathered from the user side, how to be acquired and how to be analyzed to extract the needed knowledge for recommendation purposes [33].
In our work, we suggest to take advantage of the annotation activity used usually in the learning context for different purposes and which may reflect certain learners' characteristics useful as input data for the recommendation process.

Contribution: Annotation-Based Recommender System
We suggest to take advantage of the annotation activity used usually in the learning context for different purposes and which may reflect certain learners' characteristics useful as input data for the recommendation process [18; 20]. Therefore, we present the architecture of an educational recommender system that bases their recommendation services on learner's annotation traces yielded during the learning process. The architecture of the annotation-based recommender system under present discussion is shown in Figure 1 which illustrates the interaction between the various modules of the system along with the flow of information data. We follow, in our framework, a novel approach of recommendation based on learners' characteristics extracted from their annotation traces yielded during learning experience [21].
We choose to follow learners through their annotation activities for many reasons. For instance, annotation is a practice which bridges between reading and writing [3] and constitutes the most prominent habits of active reading activity [8]. The architecture of our proposed recommender system consists of four principal modules: The Annotation Module, the Web Service Discovery Module, the Profile Constructor Module and the Recommendation Module. In what follow, we describe the works carried out for each module of our system, and we give an overview about their basic functionalities.

Annotation module
Through the annotation module, learner makes a set of annotative acts during his reading activity that will be stored in an annotation ontology [10] which is composed of three aspects of the annotation: • Physical aspect: This aspect represents the physical dimension of the annotation.
• Contextual aspect: The contextual aspect consists of the set of attributes that describe the context of reading activity during the creation of the annotation. • Semantic aspect: This aspect allows us to clarify the semantic of the annotation. It is used to interpret the meaning of the annotation by the annotator himself or by the software agents.
The relationships between the ontology's concepts are described as follows: An annotation is created by the annotator with a device and has a space-temporal context (place and date). It is presented by a particular shape, is pointed to an anchor and related to an annotated content which is a part of the read document. The annotation contains an annotating content. The annotator is identified by name, has an age, speaks a native language and has some personality traits (conscientiousness and neuroticism) which can be accurately predicted form his annotations.
The annotator reads a document in a specific domain that contains a set of documents and a set of reading goals. The annotation is realized through an annotative act which may mean one or more annotation goals. A service community satisfies one or several annotation goals. This service community includes one or more web services that realize one or several effects. The annotation undergoes one or more effects.

Web service discovery module
To take advantage of the success of web services on the one hand and to better exploit the semantics of the annotative activity in other, we propose in this article the idea of a recommender system of web services through the learner's annotative activity. Based on a new approach presenting the student's annotative activity as a means to invoke web services implicitly, the proposed recommender system tries to assist the reader via web services during his learning activities [14].
Therefore, we consider the annotation not only as a means of memorization of the learner's reactions in the reading process but also as a potential source of web services invocation that can assist the annotator and help him to satisfy his annotation goal. So, from a user's annotation, our system is able to interpret a semantics implicitly expressed which presents a need for a web service to meet annotation goals. Based on this extracted semantics, the annotation system discovers and invokes the requested web service. These new features are based on an ontology of annotation which presents the different properties of the learner's annotative activity available to computer processing.
The relations among the elements composing the semantic aspect of the annotation ontology can be described as the follows (see Figure 2): The reader chooses to read a document in a specific domain. Each domain of reading contains a set of reading goals which are strongly related to the type of the read document. For example, in the e-learning domain, the likely goals of reading a poem are learning and understanding it, while the likely goal of reading a comic story is enjoyment. Motivated by this reading goal, the reader begins to read and annotate the consulted document.
The annotative activity is the result of an active reading, so this activity certainly helps the annotator to satisfy his reading goal. For that purpose, we present for each reading goal the list of annotative acts realized by the annotator in the reading process. For each annotative act, the corresponding one or several annotation goals are presented. This objective represents semantics implicitly expressed by the annotator through the annotative act for a need for means which answer the goal of this annotation. Services community represents these needed means. Thus, for each annotation goal, there is corresponding a service community whose goal coincides with the goal of the annotation [11].

Profile constructor module
Personality traits are one of the learner's individual characteristics which extensively interest educational experts. Several research works study the impact of student's personality on academic achievement and the learning process in general [2]. Further studies shed the light on the relationship between the learners' personality and certain factors relative to the learning construct like: approaches to learning, learner's autonomy, motivation towards achievements and academic achievement. Such works conduct empirical studies that demonstrate the need to establish guidelines for incorporating learners' personality traits in designing computer-based learning systems.
The big five personality model is the best accepted and most commonly used scientific measure of personality and have been extensively researched [24]. That personality is well described as five traits was discovered through the study of the adjectives from natural language that people used to describe themselves and then analyzing the data with a statistical procedure known as factor analysis that is used to reduce lots of information down to its most important parts. The five traits representing the main personality dimensions are: openness to experience; conscientiousness; extraversion; agreeableness; neuroticism (see Figure 3).

Fig. 3. Big five personality traits
In our prior work [23; 26], we seek the connection between learners' annotation practices and their personality traits in the "pen-and-paper" context. We conducted an empirical study to show the implicit relation between the annotator activity and his personality traits [27]. Indeed, we consider group of 120 volunteers.
The subjects selected were recruited with respect to certain criteria. In fact, the age of the volunteers is equal or superior to 18 and they have different occupations and interests. In our sample we have the two sexes (44 women and 76 men). Furthermore, all the selected volunteers have frequently the habit of reading and annotation of their documents [19]. Each subject was instructed to answer a standard Five Factor Model questionnaire called the NEOIPIP Inventory which is a computer-based Personality Inventory, able to measure people Personality Traits. Then, he obtained a feedback regarding his personality based on his responses. This step gives us the personality scores based on the Big Five Model for each volunteer [28].
To associate personality scores to subjects' annotative activities, we gathered annotation practices for each people and we collected a simple set of statistics about their annotative activity. These included the following: This set of statistics tends to characterize quantitatively the reader's annotation practices. We studied the Pearson correlation between subjects' personality scores and each of the features obtained from analyzing their annotative activities.
Our findings show significant correlation of annotation practices to certain personality traits (Consciousness and Neuroticism). The constructed personality profile will be used as input data to the recommendation module to filter and adapt the list of web services compiled with regard to the objectives of learners' annotations [29].

Recommendation module
The recommendation module receives a data flow from both the profile constructor module and the web service Discovery module. The received data is composed of a compiled list of the found web services with regard to learner's objectives deduced via his annotation practices and his personality profile which is built by reference, also, to his annotation traces [17].
We hope refining and filtering the list of service regarding the learner's personality characteristics [18]. For instance, the student with high level of Neuroticism prefers a web service which reacts instantaneously to display the required result. Thus, the system recommends the services with low response time to satisfy the learner's personality characteristics. The filtered web services list will be sent to learner to select the desired recommendation for execution. The selected web service will be invoked and the system stores the user choice to be used later in refining process of system's recommendation. So, to automate the process of deduction of the appropriate web service according to the learner's personality traits, we propose to use a pattern of annotation.
The proposed pattern allows the deduction of the annotation goal from its shape, and then based on this objective, the system interprets the web services community which assists the user to achieve the goal of the annotation. Finally, according to the learner's personality traits (conscientiousness and neuroticism), the recommendation module selects the appropriate web service to be invoked by the annotator.
The patterns are represented by an ontology that refers to the elements of the annotation ontology. Indeed, an annotation pattern represents a conceptual solution to a problem related to the annotative activity. The annotation pattern proposes a solution (the semantic of the annotation) to a problem (find this semantic for a given annota-tion shape in a given context according to learner's personality traits). Our annotation pattern is composed of five elements presented in Table 2.

Evaluation of the web service discovery module
To evaluate the effectiveness of the web service discovery module, we integrate this module in an annotation system called "New-WebAnnot" developed in the work of Kalboussi et al. [13; 14]. The choice of this tool is justified by the fact that it represents an annotation system offered to the learner to annotate his learning activities [6].
We tested the use of this system by a contribution to another classic annotation tool that does not provide assistance with web services for two different learners' samples. The experiment lasted four reading sessions of a set of English courses. In two different samples, 60 students (A with N1 = 30 and B with N2 = 30) had spent two sessions per week that lasted an hour of reading in university room equipped with computers.
• Computers in sample A are equipped with the annotation plug-in "New-WebAnnot" in their Mozilla Firefox web browser. • Computers in sample B are equipped with a classic annotation plug-in "Annozila" in their Mozilla web browser.
We presented to learners of sample A six categories of web services which can be invoked during the annotation session to meet the goals of some annotations made by the learner. These services are: • Dictionary service: helps explain terms • Translator service: helps translate text • Memory service: helps memorize information • Agenda service: helps plan tasks in the future • Summarizer service: helps summarize texts • Social network service: helps share information with others via social networking We asked to the learners to annotate English courses consulted in a specialized website which offers a free training in English [4]. The objective of this experiment is to test the motivation of each learner sample towards the presented annotation system. We try to prove that the assistance offered by our annotation system through web services motivates learners to annotate more the consulted documents. To measure this motivation, we define two metrics: • X: The number of annotations in a session for each learner • Y: The duration of the annotation session for each learner. The maximum duration of an annotation session is 60 min, which represents the time reserved for each reading session The choice of these two factors is justified by the fact that a learner motivated to annotate realizes logically a high number of annotations, and his session of annotation lasts more than that of an unmotivated learner. The objective of this evaluation is to answer the following question: Is there a difference in the number of annotations (X) and the duration of the annotation session (Y) between the two student samples?
This means that we should test the following hypotheses: • Null hypothesis (H0): There is no significant difference in the means of X and Y between both samples; H0: μXA = μXB and μYA = μYB. • Alternative hypothesis (H1): There is a significant difference in the means of X and Y between both samples; H1: μXA ≠ μXB and μYA ≠ μYB.
Through this evaluation, we try to verify the existence or non-existence of a significant difference between two samples for two variables of measure (X, Y). The statistical method Student's t-test appears as the most appropriate test the validity or nonvalidity of this hypothesis. The Student's t-test is defined as a parametric test based on a comparison of means for quantitative data between two independent samples. So, we use the Student's t-test statistical approach to calculate the value texp. Thus, if texp (experimental) < tth (theoretical), there is no significant difference (H0) or If texp (experimental) ≥ tth (theoretical), there is a significant difference (H1).
The results of the Student's t-test are obtained through the tool STATISTICA in Table 3.
• For the variable X: texp = 19.586 & tth = 3.496 texp > tth⇒ rejection of the null hypothesis (H0) and acceptance of the alternative hypothesis (H1). So, we can conclude for the variable X (number of annotations of the students) that a difference between means in samples A and B at the 99.99% level is very highly significant. P = 0.00021 indicates that there is one in 5000 chance of being wrong for this result.
• For the variable Y: texp = 13.553 & tth = 3.496 texp > tth⇒ rejection of the null hypothesis (H0) and acceptance of the alternative hypothesis (H1). So, we can conclude for the variable Y (duration of annotation session) that a difference between the means in samples A and B at the 99.99% level is very highly significant. P = 0.00012 indicates that there is one in 10,000 chance of being wrong for this result.  Figure 4 clearly shows the highly significant difference between the means of samples A and B for both quantitative variables X and Y. Thus, there is no intersection between the confidence intervals of the mean of X in samples A and B (top plot), and similarly for the confidence intervals of the mean of Y (bottom plot).

Fig. 4. Means plots of X and Y grouped by sample
These results proved that there is a highly significant difference between samples A and B for the mean of X (P = 0.00021) and the mean of Y (P = 0.00012). μXA > μXB and μYA> μYB. Thus, it is clear that our annotation system motivates more the learners in sample A to annotate than the "Annozila" tool proposed in sample B.

Evaluation of the profile constructor module
To assess whether the profile constructor module measures accurately the user's traits, we integrate this module in the annotation system "New-WebAnnot" used in the previous evaluation, and we invited students to annotate consulted resources on the web via this annotation tool to achieve their reading and annotation activities. Next, learners were instructed to answer a standard Five Factor Model questionnaire (the NEO-IPIP Inventory) to obtain a feedback regarding their personality based on their responses.
To show the system's efficiency to measure accurately the scores of reader's conscientiousness and neuroticism traits compared to the values determined using the NEO-IPIP Inventory, we applied the paired t-test to compare the scores of certain user's personality traits obtained through the two different methods of measurement.
We look to determine whether there is a significant difference between the paired values of scores. Both measurements are made on each subject in the selected sample, and the test is based on the paired differences between these two values.
Tables 4 shows descriptive statistics of t-test measure of the difference significance between the paired values of learner's conscientiousness and neuroticism traits scores measured with two different methods: the "New-WebAnnot" system and the Neo-IPIP inventory. Analytical results indicate that the scores of learner's conscientiousness and neuroticism characteristics obtained through the annotation system "New-WebAnnot" did not differ significantly (Sig1 = 0,72 > 0.05; Sig2 = 0,53 > 0.05) versus the scores measured using the Neo-IPIP inventory. We show that the neuroticism trait is negatively correlated to the different annotation features considered in our study. This result clearly indicates that learners with high level of emotional stability are more productive of annotation traces which reflect their deep reading of textual material. Thus, those who have low score of neuroticism are more stable and they have the ability to focus more on their current activities and they can deal with reading materials qualified with high level of complexity.
Finally, we are interested to evaluate the system's performance to compute accurately the learner's scores of conscientiousness and neuroticism traits compared to the values determined using the NEO-IPIP Inventory. So, we applied the Pearson Product Moment Correlation to measure the linear correlation between the traits scores obtained through the two different systems. We also measure the root-mean-square error (RMSE), which are the root mean squared differences between predicted values (measured with New-WebAnnot system) and observed values (measured with Neo-IPIP).
Given that correlations are significant for both consciousness and neuroticism traits and the lower values of RMSE, we show that the regression models for the considered personality traits is well-fitting and the predicted values close to the observed data values which means that there is no significant difference between the scores of user's personality traits computed using the "New-WebAnnot" system and those measured using the Neo-IPIP inventory (see Figure 5). Thus, the experimental results show the efficiency of the "New-WebAnnot" system to measure conscientiousness and neuroticism with reasonable accuracy using digital annotation activity.

Conclusion and Future Work
In this paper we presented a new approach of an educational recommender system which refers to learners' annotations activities to implement personalized recommendations. We explained the architecture of our recommendation system consisted essentially of four basic modules: annotation module; web service discovery module; profile constructor module and recommendation module.
The evaluation of the two main modules (web service discovery module & profile constructor module) through empirical studies realized on groups of learners based on the Student's t-test showed significant results. Our prior works show plainly the opportunity to consider annotations to extract certain learners' characteristics (personality traits and learning goals) with regard to learning context (reading materials).
Actually, the current system constitutes a proof of concept tested with a set of 30 learners of computer science. This could entail some bias in the learners' profile. As additional work, we will try to extend the evaluation tests to a large number of learners with more heterogeneous profiles. Finally, as future work, we expect taking advantage of the combination of learner's personality and his annotation goals which may guide our recommender system to derive other learning parameters.