Embedding Emotion Awareness into E-Learning Environments

In this work we investigate the importance of emotion awareness in e-learning environments, with emphasis to Computer Supported Collaborative Learning (CSCL) activities. Our presented solution involves a new conceptual model of emotions of interest in e-learning context. In the basis of this model, a computational model has been implemented employing self-report of emotions, affective feedback and effective emotion visualisations. Both models have been tested in real education settings, contributing to the research agenda.


INTRODUCTION
Despite the progress that has been achieved the last two decades in education technology, the e-learning community is still talking about the promise of this technology whereas is questioning its realistic classroom use [1,2,3].One major criticism is that these innovative technologies "tend to focus exclusively on cognitive factors […] and are often unable to adapt to real-world situations in which emotions play a significant role" [4].
Emotions strongly influence human's behaviour in social situations and must be seriously considered when forming collaborations.Just placing students together, it does not mean that they will indeed collaborate [5].The embodiment of emotional awareness features into learning environments could offer a more authentic and challenging learning experience, either individual or social [6].
In literature, emotion awareness is defined by the awareness of self (self-awareness) and others' (empathy) emotions [7].In e-learning context, emotion awareness can be identified by (a) the process of receiving emotion input, implicitly (motor-behavioural actions or biophysiological signals) or explicitly (self-reporting of emotions) and (b) the respective affective response that can be provided manually (by human) or automatically (by machine).
Towards that direction, Affective Computing is offering remarkable system implementations that detect and recognise students' emotional states with high accuracy by employing machine learning algorithms [2,8,9].In response to this emotion recognition process, systems are able to provide effective feedback aiming at both students' cognitive performance and emotional regulation [10,11].However, research still lacks studies to address the presence of emotions especially in CSCL activities within Virtual Learning Environments (VLE) [11].Furthermore, we are still far from adequate empirically proven strategies to respond affectively to individual or group detected emotions [12,13].
The current paper investigates the importance of emotion awareness in e-learning environments.To this end, a new conceptual, and a computational model have been developed to embed emotion awareness into elearning environments.The integrated solution has been validated in real education settings.
We begin our analysis in Section 2 by presenting previous research on the field together with the necessary theoretical background on emotions.Section 3 presents a new conceptual model of affective states that usually appear in e-learning situations.Section 4 describes the implemented computational model.Section 5 presents four cases studies validating the proposed solution.In Section 6, we present main findings followed by, limitations and future steps in Section 7.

A. Emotion Definitions
Despite the few attempts to understand and define emotion, scientists are still refraining from a widely acceptable definition and there is one significant reason for that: everyone knows what emotion is until they are asked to define it [14].How can you define something verbally that it has been established before verbal logic?
In literature, the words emotion, affect, feeling, mood are often treated as synonyms.An interesting analysis and review of emotion definitions is found in [15].Emotion contains the organism's reaction to any disturbance of the perceptual environment.When this disturbance is appraised as "suspicious" in first time (milliseconds), it triggers somatic changes (rapid heartbeat to produce more blood, ephidrosis to cool the body, eye blinking to protect the eyes, etc.) in an attempt to prepare the organism, as quickly as possible to face a potential threat against the organism's survival.This motion/activity/alertness is felt back or sensed by the organism, which also tries first to appraise it cognitively according to the organism's subjective experiences, and then, in second time, to control the somatic changes.All the effects (cognitive processes, feelings) aroused by the specific emotion (i.e.fear) constitute the affect, (i.e.anxiety) the effect of the PAPER EMBEDDING EMOTION AWARENESS INTO E-LEARNING ENVIRONMENTS emotion in the organism, which the organism is aware of.Mood (i.e.sadness) identifies the feeling left in the organism, after the cause of the emotion is ceased, and is more cognitive and less emotive.
In this work, the perception of affective state is deployed to ascribe both emotions (for instance excitement, confusion, anger, stress), and other, less emotive, and more cognitive states (for instance inspiration, interest, boredom, fatigue), in the context of e-learning.The perception of mood (e.g.very bad, good, very good) has been also used to give the respondent's feelings a tendency in positive or negative direction.

B. Classification of Emotion Theories
Emotion researchers are struggling to define and classify emotions, trying to decode something that has been imprinted in the human brain, before the verbal system developed enough to support communication.Scherer (2005) has distinguished three major schools in emotion research: • Basic emotion: Patterns of physiological reactions that can be easily recognised universally, e.g., fear, anger, happiness [16,17,18,19,20].• Emotion dimension: In this case, various measureable parameters are adopted i.e. arousal, valence, intensity, in an attempt to quantify emotion [21,22].• Eclectic approach: Use of labels that seem appropriate to the aims of a particular study, e.g., academic emotions [23,24].In a different approach, Boehner et al. [25] posit an interactional approach that sees emotion constructed through interaction and expression.They focus on emotion as a social and cultural product, not as a measurable, biological fact.In fact, they skip the emotion recognition process, that is, the encoding/decoding of input and output physiological signals to specific emotion information.

C. Emotions in learning
Despite their wide acceptance, much of the research on basic emotions is of little relevance to the developers of computer learning environments [11], which focus more on learning-centred emotions such as confusion, frustration, boredom, flow, curiosity, and anxiety.
For almost two decades, Pekrun and his team [24,26] examined the impact of the so-called academic emotions.According to their findings, positive mood fosters holistic, creative ways of thinking.Harmful effects can only be expected in situations, where students are in a good mood and the learning topics are of less importance to them.In this case, the positive emotion might detach them from learning [13].Negative emotions, on the other hand, direct students' attention to themselves, in most of the cases.However, negative mood proved to enhance an analytical, detail-focused way of processing information.
Curiosity and puzzlement may lead to investigate problems and even frustration may lead to action, despite its negative valence [27].The state of confusion is considered more positive than frustration because students are motivated to overcome the source of their misunderstanding, whereas in frustration they are more likely to disengage from learning [28].
In literature, uncertainty is encountered as an opportunity to learn [29].Op't Eynde et al. have reported that during a mathematical problem, students experienced negative emotions such as worry, frustration and anger, which helped them to reorient their efforts to solve the problems, following different paths, for instance, starting from the last part and then go to the first part [30].
With regard to emotion transitions, D'Mello et al. have shown that, students are more likely to remain in the same affective state if no intervention is provided [31].This tendency appears to be particularly strong for students in negative affective states.Robison et al. [28] have shown that frustrated learners were more likely to transit to confusion or fear, and expected less to enter a positive state such as flow or excitement.Students experiencing the positive state of flow were likely to transit to confusion, which is still considered positive for learning and were unlikely to move to the more negative state of frustration.Interestingly, confused learners were equally likely to transit to flow and frustration.These findings suggest that the states of confusion and its antecedents and consequences are worth additional study to determine which factors contribute to a positive transition to flow or a negative transition into frustration.
In general, there are no adequate empirically proven strategies to address the presence of emotions in learning, especially the negative ones [13].Theoretical background has been built upon theoretical foundations of pedagogy/affect or recommendations made by pedagogical experts [31].And despite the evidence of the positive effects of positive mood and emotions, there are no clear rules such as: positive emotions foster learning, and negative emotions are detrimental [13].

D. Collection of Emotions
In face-to-face situations, students communicate and exchange information, enriched with emotion cues like facial expressions, voice intonations, gestures, body positions, etc., in an attempt to transmit what the respondents really want (need, desire, love, etc.) or do not want (afraid, dislike, hate, detest etc.).On the contrary, the exchange of emotion data in online learning environments is quite limited.Developers and designers are striving to empower learning environments with usable interfaces that trace student emotions in an unobtrusive and noninvasive way, without extra cost, equipment or expertise, and without language barriers.
In the majority of these research studies, three main input channels are used to collect emotional information: • Physiological signals (electromyogram-EMG, electrodermal activity-EDA, electrocardiogram-EKG or ECG, blood volume pulse-BVP, etc).• Motor-behavioural activity (facial expressions, voice intonation, body posture, etc).• Self-report (verbal or pictorial scales, questionnaires, etc).A separate category refers to emotion recognition from the user's text input through sentiment analysis or opinion mining, deploying special Natural Language Processing (NLP) algorithms.
There have been various updated endeavours in modelling the management of emotions and affectivity in learning systems, showing promising results [2, 8, 9, 10, PAPER EMBEDDING EMOTION AWARENESS INTO E-LEARNING ENVIRONMENTS 28,31].Nevertheless, emotion recognition is susceptible to the same vulnerabilities of speech recognition; despite the advancements that have been attained in experimental settings, their utilization to cope with everyday needs is restricted by the lack of resources (cost of equipment, complexity of systems, etc.).Sensors are more precise but costly and time consuming.Self-reporting on the other hand is less costly but often considered subjective and out of context.In the majority of the studies, multimodal integration is preferred (combination of the input channels).
A fundamental criterion to select an appropriate detection method is dictated by the availability of resources.Most computers on a lab or portable devices are equipped with a camera that can be used for facial expression recognition.Students' emotions can be inferred by analyzing mouse and keyboard movements that are stored in log files.Text is also an important modality for sensing emotion since the majority of computer user interfaces today are textually-based.

E. Affective feedback
Once the learner's affective state is recognised, the obvious next question is what to do with this valuable information.The user needs to see some reaction from the system; an adaptation to his/her feelings.Affective feedback can be either parallel-empathetic (exhibit an emotion similar to that of the target), reactive-empathetic (focus on the target's affective state, in addition to his/her situation) or task-based (change task sequencesupplementary to empathetic strategies) [28].Common tools include dialogue moves (hints, prompts, assertions, and summaries), immersive simulations or serious games, facial expressions and speech modulations, images, imagery, cartoon avatars, caricatures or short video-audio clips [31].
Unfortunately, there are few studies that exploit computer mediated affective feedback strategies and their impact on users' task performance or affective state.Furthermore, the number of tools and strategies to design expressive avatars in response to learner's emotion detection is quite limited.

III. THE CONCEPTUAL MODEL
!he design of a system that shows respect to students' emotions is a complicated process because of the nature of emotions.In search of emotions of interest, we have built a new conceptual model evaluating tentative affective states of interest (inspiration, excitement, interest, curiosity, confusion, stress, anger, despair, disinterest, embarrassment, boredom, fatigue, relief/relaxation, neutral), usually found on e-learning situations, especially in CSCL activities (Figure 1).
Our emotion model has been built upon the theories of the Learning Cycle [23], Flow [32], Circumplex of Affect [21] and Academic emotions [24] mainly following the information processing approach [33].It has been structured according to five gradual layers in the appraisal of an emotion information (survival!fear/cease of fear!emotions!affective states!moods) highlighting the significance of fear, the root of all emotions, and its strong connection with the survival instinct.
We have also projected our conceptual model in a twodimensional diagram to quantify the respective affective states, deploying the emotion dimensions of: (i) valence, for the polarity in a positive-negative x-axe and (ii) activation, for the arousal in an activating-deactivating yaxe [40].Mood is measured in 5-likert scale.
The place of the each emotion label is not standard but only indicative with respect to the valence and activation dimensions, and to the rest of the affective states, deriving from Russell's model [21], as well as other relevant studies that validate this model [34,35].For instance, curiosity is considered the same activating emotion as anxiety, but of the opposite valence.Hence, curiosity is more possible to transit to excitement while anxiety to anger.
A main objective is to evaluate the proposed conceptual model in e-learning context, especially within CSCL activities and mine for possible affective sequences and patterns that lead to enhanced task performance and fruitful collaborations over time (i.e.time intervals like morning, afternoon, night or task phases).Certainly, it constitutes a first attempt to encompass human emotions with respect to learning.Nevertheless, it is difficult to set the border lines among different emotion labels that pervade real life.It is an ambition to test and inform this model with findings from experimentation in real education settings.

IV. THE COMPUTATIONAL MODEL
Based on the conceptual emotion model, we have developed a web tool, named emotcontrol, consisting of three main components: (a) the basic system that is a selfreporting interface for students to report their affective states, (b) the affective virtual agent component that provides affective and task-based feedback to learners, and (c) the visualisation monitors that depict the individual and group affective states together with the respective feedback from the virtual agent.

A. Emotion report
Emotcontrol is a cross-platform, open-source web tool that has been programmed in JavaScript for the front-end, and PHP and My SQL for the back-end system (Figure 3).It has been designed to open as a remote-control window, utilizing rich multimedia (fancy colours, playful images and text labels) to project specific emotions.It has adopted the shape of a spiked circle, in accordance to the Learning Cycle [23] and the Geneva Emotion Wheel [36], involving a range of emotions that usually appear in a learning experience.The colours were also adapted from the Geneva Emotion Wheel [36], as well as the Wheel of Emotions [20].Additional text boxes have been provided for respondents to report other emotions than the default ones.

B. Affective Agent
The affective feedback mechanism consists of animated, virtual agents (avatars) that provide a variety of scaffolds (hints, prompts, jokes, games, etc.) of various types (empathetic, task-based, entertaining, relaxing, etc.) [28], with respect to the user's both cognitive and affective states.These avatars are displayed embedded in the learning environment (Figure 3) or in the emotcontrol tool (Figure 5) using a JavaScript player to avoid frequent browser problems with embedded media.The affective agents were activated automatically, based on fuzzy rules, or manually, in order to experiment on affective feedback strategies and possible affective sequences.In the experiments that were conducted, two avatars were used: (a) Alice (Figure 3) an implemented virtual face that employs facial expressions and voice intonations to provide empathetic dialogue moves, and (b) Voki, a FREE online application1 to create customized avatars providing a plethora of comic characters, synthesized voices and scene backgrounds.

C. Emotion Visualisation
Emotcontrol provides both group members and instructors with several visualisation monitors of the individual and group affective states.The participants can be aware of their group-members' affective states, while tutor can supervise all the respective information per group, student, date and time, emotion or mood.An emotional cartography is also provided (Figure 4), visualising students' affective states together with the respective feedback from the virtual agent, as well as students' text responses, in the form of a storyboard per group and date/time.

Group emotions button
Alice affective virtual agent

A. Educa experiments
In the Educa experiments, the emotcontrol system was integrated into the Moodle 2 learning environment, in the context of semester courses.By simply clicking on an image button at the top-right corner of each course page, the respondents could report their affective state, any time they wanted (Figure 3).Their selection was displayed in the initial emotcontrol block, inside the button.Additionally, they could report an emotion other than the default ones or to expound their emotion selection by writing in a small text box (see Error! Reference source not found.for more details).
Based on the emotion information submitted, the affective agent provided respondents with affective feedback, stimulating them sometimes to ask for additional help by typing their message in the respective text box.Their message was stored in a helpdesk system that informed both the tutor and the support team by email.The agent provided the respective feedback (tips, additional material, web links, etc) automatically (Educa I) or manually (Educa II).
Group discussions (forums or chats) were also directly supported by emotcontrol, since the application was integrated into the chat/forum module and students could easily deploy it to express their emotions during a debate.In addition, the emotcontrol system provided the functionality of group emotion awareness so that each 2 https://moodle.orggroup member could be aware of the emotions of their peers at any time.
The main objectives of the Educa experiments were: • To evaluate the emotcontrol interface usage, usability, expressiveness and effectiveness (Q1 and Q3 research questions).• To mine for patterns of tentative affective sequences (Q2 research question).• To examine the impact of emotion awareness on individual and group performance, as well as, on self emotion regulation (Q3, Q4, Q5 research questions).The Educa I experiment was conducted for a control group in the sense that there was no provision for personalized feedback to the students.The virtual agent was automatically activated for providing instant feedback and triggering the reasons behind a student's affective state through specific intentional questions (Q4 research question).On the contrary, Educa II experiment differed significantly from the previous one because there was a variety of feedback characters and responses produced adhoc and not automatically by fuzzy rules.In this way, a variety of both cognitive and affective responses (empathetic, task-based, amusing, etc.) was examined, to discover feedback strategies with considerable impact.(Q4 and Q5 research questions).The main target was to mine for effective dialogue moves and off-task activities that improve the learner's cognitive performance and affective state.It must be mentioned that although the feedback was provided manually, most of the students never got sure about who was responding: the system or a human behind the system.
The experiments results were collected by analysing the user and group activity log files, by deploying adhoc iJET -Volume 9, Issue 7: "ALICE 2013", April 2014 PAPER EMBEDDING EMOTION AWARENESS INTO E-LEARNING ENVIRONMENTS evaluations and user behaviour metrics (response time, number of reports over time) and finally by conducting evaluation questionnaires.The impact of the affective response was evaluated in four ways: a.By observing the student's affective sequence and looking for affect transitions after the feedback was provided.b.From the student's text response, whenever the student provided one.c.From the emotcontrol interface in the form of a 5likert scale: (Was this helpful?1-Yes, very much, 2-A little, 3-So and so, 4-No, 5-You'd better not helped me) d.From the experiment's final questionnaire.

B. VCS Experiments
The emotcontrol system has been also validated in the context of Virtualized Collaborative Sessions (VCS).Two experiments took place in real context of learning of the Open University of Catalonia (UOC), incorporating emotional awareness into real virtual learning scenarios aimed to enhance motivation and engagement during the learning process (see Error! Reference source not found.and Figure 5 for more details).A Virtualized VCS system is made to be compatible and interoperable with broad collaborative sessions, such as chats and forums, in order to create general types of Learning Objects named "Collaborative Complex Learning Objects" (CC-LO) [37].

Open University of Catalonia (UOC) Department of Computer Science, Multimedia and Telecommunications
Participants: Students of the department (age: 32 on average)

Methodology:
The participants worked individually on an activity consisted in filling a test with questions, after watching an interactive CC-LR material embedded in the VCS prototype from their virtual classroom of the UOC.

Affective Feedback:
The virtual affective agent Alice was automatically activated according to fuzzy rules.In VCS I, the SLO Editor was enhanced with the incorporation of emotional scenes in which the system was able to inspect the emotional behaviour of the learner during the consumption of the CC-LR.The respondent's affective state was inferred with the assistance of combinational questions set to the participant.In response to the user's affective state, affective feedback was provided by the virtual affective agent Alice.

Learning
Unlike VCS I, in VCS II, the emotcontrol was embedded in the same VCS window, experimenting on emotion reporting in parallel with the user's task Similar to Educa experiments, VCS experiments evaluated the emotcontrol interface usage, usability, expressiveness and effectiveness, in VCS context this time (Q1 and Q3 research questions).The effectiveness of the affective feedback was also evaluated by examining the adequacy of providing additional material (i.e.documents, links) relevant to the task, as well as recreational activities (i.e.ambient calm music and video, online entertaining games) irrelevant to the task (Q4 research question).
The evaluation process was implemented through a questionnaire in the scale 0-10.For the quantitative analysis basic descriptive statistics were employed, such as Mean (M), Standard Deviation (SD) and median (Md) to analyze the scores obtained in the questionnaire.The quantitative analysis was complemented by employing accepted statistical procedures, such as Chi-square (X 2 ), so as to compare the observed scores to the expected ones.

VI. RESULTS & FINDINGS
From our experimentation it appears that both the conceptual and the computational models achieved satisfactory results, revealing high expressiveness (only embarrassment appeared to have infrequent reports).Users found emotion reporting through emotcontrol, useful (~72%), usable (~89%), expressive (~75%) and effective (~84%), supporting that emotion self-report can be provided in parallel with the task taking into consideration the availability of resources in real education settings, without requiring extra costs or expertise and without introducing much invasiveness in the system, However, our experimentation raised also a speculation considering the sufficiency of verbal or pictorial labels, to express the respondent's exact feelings, validating in a sense that emotion is better represented by continuous values [38].
The states of interest and fatigue were the most frequently reported, followed by relief/relaxation, excitement, inspiration, confusion and stress.Fatigue was often reported before starting doing anything revealing that fatigue frequently refers to the psychological, affective tiredness that is often reported in the place of stress [39].In VCS, anger was reported infrequently revealing the connection of this emotion to the social context.
The experiment analysis also supports the existence of patterns of affective sequences that can be exploited by computer intelligence to provide effective emotional scaffolds and mediation.For instance, students usually begin with frequent overlapping of negative and positive emotions until a point of time (~40% of the project's total time, in our cases).After that point, positive emotions seem to predominate over the negative ones.Students also begin with high confusion rates, which however, gradually PAPER EMBEDDING EMOTION AWARENESS INTO E-LEARNING ENVIRONMENTS reduce, while special caution must be given in the case of sharp increase of anger reports in order to prevent undesired situations among peers.Finally, the experiments results show that inspiration trend line is located close to the middle, representing the state of flow: the balance in learning visualising that learning takes place somewhere between boredom and excitement or stress.
Affective feedback can improve student's cognitive performance and mood, at least when it is conducted by human decision (manual affective feedback).A good example is parallel-empathetic affective feedback enriched with task-oriented scaffolds, that has a considerable impact on student's performance, as long as it is provided instantly or after a very shot time.Group emotion awareness also appears to improve social interaction.
On the other hand, an automated affective agent should be carefully designed and developed, employing complex machine intelligence in order to model the least of human emotional life but with some considerable impact.Simple affective responses based on simple fuzzy rules may lead to the opposite result.The age factor must be also taken into consideration: emotion intelligent systems in e-learning seem to have more impact on younger ages.The provision of other than the task options (i.e. a relaxing song, a nice video, an amusing game) had no impact on learners' task performance.Special caution be given when providing affective scaffolds in response to a negative affective state.In such cases, inappropriate feedback might activate anger or aggressiveness.
From the Educa II activity logs, it was inferred that participants often reported their affective states -especially anxiety, boredom, fatigue or confusion-whenever they spent some time in the environment, visiting different pages for few seconds, without posting, editing or doing anything.Emotcontrol seemed to offer students an alternative to notify their presence, especially in cases of low task performance.Such cases offer the supervisors good opportunities to intervene and provide affective and task-based scaffolds, supporting the student meaningfully.It seems that emotion reporting can empower students' engagement.
Finally, group emotion awareness was a useful functionality since many members had the chance and the initiative to intervene in their groups when they noticed bad feelings of their peers.

VII. CONCLUSSION AND FUTURE STEPS
In the current paper we have presented a solution towards the provision of e-learning environments with emotion awareness features, which is based on selfreporting of emotions and affective response from the system.We have briefly presented our implemented conceptual and computational models together with findings derived from experiments that were conducted with university students, validating our solution in real education context.
According to our results and findings, students are willing to participate and express their affective state, once a tool provides an easy and usable way for them to do it, empowering in that way their engagement.However, students need to see a reaction from the system in response to their emotions' sharing, immediately or after a very short period of time.Affective feedback, enriched with task-oriented scaffolds, can improve student cognitive performance and emotion regulation, at least when it is conducted by human decision.Group emotion awareness also appears to improve social interaction.
Our findings uncover an initial limitation of emotions labels to adequately express the respondent's exact feelings.Future plans involve emotion collection based more on emotion dimensions (valence, activation or dominance).
Moreover, this works emphasized the need for also an implicit way to evaluate user's emotions.Sentiment analysis and opinion mining methodologies can undertake that role, involving lexical analysis of text in order to identify words that are predictive of the affective states of writers.These two channels can be coupled to unfold interesting patterns of individual or group emotional behaviour.
With respect to affective feedback, future plans include the extension with alternative paths in the learning sequence, based not only on the task performance, but also on the emotion reporting, in an attempt to provide more personalized and adaptive learning.
All findings could be further validated by different research studies, in different contexts and e-learning environments.The role of affect in e-learning is at best in its infancy and there is still much to be discovered.There is an open call to develop and design methodologies that are pedagogically guided and which would lead to affective systems of e-learning.

Figure 1 :Figure 3 :Figure 4 :
Figure 1: The conceptual emotion models Figure 2: The two-dimensional projection of the conceptual model

Figure 5 :
Figure 5: Emotcontrol embedded in the SLO Player

University of the Aegean (UoA) Department of Cultural Technology and Communication Participants:
Students of the department (ages: 20~22) Note 1: The emotion label embarrassed was replaced from fatigue in Educa II, due to infrequent