Emotions in Serious Games : From Experience to Assessment

Serious Games represent a global revolution that promises to develop intuitive, affordable, accessible and familiar learning environments for a wide range of educational and training applications. Serious Games are computer or video games in which education is the primary goal, rather than entertainment. Serious Games offer different benefits due to the integration of simulation aspects, game aspects and pedagogical elements at the same time. Among the different assets that Serious Games provide for learning, we will focus in this paper on their ability to leverage the synergy between emotional and learning appropriation processes. We will first outline the specific features of learning opportunities offered by Serious Games considering the involvement of emotions in this learning process. The following part of the paper will be dedicated to the description of the methods for the analysis of the emotional experience of SG users. Finally, we will explore a new promising research direction concerning the use of Affective Computing in Serious Games.


I. INTRODUCTION
Serious Games (SG) represent a global revolution that promises to develop intuitive, affordable, accessible and familiar training environments for a wide range of educational and training applications, from medicine and healthcare to defence, from national heritage to multicultural interaction.They can be broadly defined as computer or video games in which education, in its various forms, is the primary goal, rather than entertainment [1].
Serious Games offer different benefits for learning, among which: allowing situated "learning by doing"; making users feel responsible for their actions within a protected simulated environment; combining high-quality contents to high levels of engagement; turning mistakes into learning elements, preventing the idea that errors are irredeemable.These benefits are due to the unique match of simulation aspects, game components and pedagogical elements at the same time [2].Among the different assets offered by SG, we will focus here on their ability to leverage the synergy between emotional and learning processes.In the first part of the paper we will briefly describe the main advantages that SG (as compared to more traditional e-learning forms) provide in terms of appropriation and emotional involvement, highly functional to learning outcomes.Emotions, in fact, play a primary role inside the structure and experience of Serious Games.Emotions are strongly related to action [3][4] since they prepare humans for actions (action readiness), as for example approach-ing/avoiding something in function of available resources.The linkage between emotions and actions is extremely relevant in learning processes: according to widely recognized contributions [5][6], learning is best viewed as a process of participatory appropriation, where the learner acts and actively participates in doing within specific routinized contexts.Learning, therefore, is not only the outcome of the transmission of knowledge and concepts from an active expert to a passive novice, but it necessarily involves the learner's active participation.
Within this process, the emotional component is necessarily entangled to learning, as long as he emotional experience triggers and maximizes the opportunity that learners take up a real active role within the situated and experiential learning process.
Serious Games comprise specific affective and motivational features that can enhance learning outcomes by exploiting the interdependence between emotions and participatory appropriation.
In the following paragraphs we will focus on two specific aspects related to emotions in Serious Games: the pleasure dimension they support and their ability Games to modify the cognitive appraisal of errors (and the consequent emotional experiences) in learning processes.
Given the relevance of systematically investigating emotions in Serious Games, we will then describe how it is possible to empirically analyse users' emotional experience according to the multimodal perspective outlined by the current methodological trends in emotion psychology.In the last part of the paper we will explore a new promising research direction that concerns the integration of Affective Computing features (i.e., the ability to automatically recognize emotions) in Serious Games.

II. SERIOUS GAMES AND THEIR OPPORTUNITY FOR PARTICIPATORY APPROPRIATION IN E-LEARNING
The study of the potential advantages of Serious Games has increased gradually within the research community during the last few years.Currently it is widely accepted that educational games can easily increase the attractiveness of learning, giving an additional boost in the struggle against low-motivation and dropouts.Moreover, Serious Games may train specific competencies using a friendly environment, where the learner is able to play, explore, make mistakes, and learn from her actions and choices [7][8].
The latest eLearning environments, with their emphasis on simulations and interactive experiences, represent powerful devices in technology-enhanced learning.However, only as recently as in the mid 90's, eLearning systems were generally no more than huge digital-boxes of iJET -Volume 5, Special Issue 3: "Creative Learning with Serious Games", November 2010 EMOTIONS IN SERIOUS GAMES: FROM EXPERIENCE TO ASSESSMENT knowledge and contents, principally provided in amodal linguistic format, where learning was purely conceived as transmission and assimilation of formal and explicit knowledge from experts to novices.
At the end of the 20th century, the development of Web applications triggered a profound renovation of the eLearning field [9] Traditional environments for eLearning have been replaced by the Learning Management System [10].These systems allow the active participation of instructors in the educational process, giving them the opportunity to track students' activity and leading towards a more cooperative learning process.In addition, the representational format of learning contents has evolved into multimedia and interactive elements [11].
Within this trend, Serious Games encompass multiple further advantages, among which the most relevant are the situated, multimodal and experiential representational format of the contents provided and the related appropriation opportunity within e-learning platforms.In other words, as knowledge, being encyclopaedic, procedural, tacit or formal, is translated into sensorial multimodal simulated experiential formats (not only symbolic or linguistic), players are involved in wholly experiencing the situations which are simulated in the games.
According to current trends in pedagogical sciences [6] learning is best conceived as an interactive process where expert and novice reciprocally influence each other while novice actively participates in concrete activities within specific routinized context supported by experts.
Novices and experts need therefore to be involved in routinized contexts of practices: by actively participating to these contexts and acting within them, novices have the opportunity to co-construct their knowledge with experts, in a process of participatory appropriation.Appropriation is a transformative and constructive process in which both the source of knowledge and learner are reciprocally transformed through individuals' engagement in goal-directed activity.
Serious Games represent a possibility to promote situated and simulative learning as participatory appropriation, and not as mere knowledge transmission, in several learning domains, from history to medical or communication competencies.In other words, using Serious Games learners are unavoidably required to simulate, take decisions and concretely act within virtual scenarios adequately re-created to mirror specific experiential routinized formats.
Besides situated and simulative participatory activities, Serious Games imply, as crucial portion of their distinctive nature, the aspects of gaming, entertainment and fun.As grounded in gaming and participatory activities, SG trigger high emotional involvement and interest in users.This emotional component contributes, in turn, in significantly activating and maintaining attention resources, in improving memory and learning processes.
In this way gaming, via positive emotional experiences, supports the upholding of active participation in experiential learning.Game-related positive emotions therefore facilitate the unremitting approach to learning experiences even in front of mistakes, errors or failures.Likewise, the fun profile embedded in SG reduce negative emotions and the avoiding of possibly highly-challenging situations, where criticism and judgmental attitudes usually follow learners' unsuccessful actions.
Thus, on the one hand, the gaming aspects of SG, via emotional involvement, far from being sources of distraction, support the active participation of learners, expedite the appropriation of contents and competencies, as well as the grounding of learning in users.On the other hand, the users' involvement required by Serious Games participatory activities represents a protective factor to prevent boredom, frustration and negative-mood typically related to passiveness and inhibition.
In the following paragraph, some specific emotional aspects involved in SG will be introduced, focusing on positive emotions, pleasure dimensions as well as to emotions related to errors' cognitive appraisal.

A. The Complex Range of Emotions in Serious Games
Experience In literature it is widely accepted that emotions play an important role in any learning experiences [12][13], be it traditional frontal lesson [14] or e-learning [15][16].Therefore, also in Serious Games, as global revolution in terms of education and behavioural changes, the tight bond between emotions and learning is undeniable.Therefore, as students in a classroom can be bored or positively involved during lessons, similarly SG users may experience high levels of engagement or frustration as a function of the quality of their own learning-gaming experience.
However, Serious Games encompass features and elements that importantly raise up the likelihood of experiencing positive affects and effectively regulating emotions during learning, compared to more traditional e-learning settings.
What are the distinctive Serious Games features boosting up positive emotions in learners?Well-designed SG have intrinsically pleasant and gratifying traits, and are therefore desired, appreciated, nicely perceived, and repeated from players showing a high degree of identification with the proposed activity.Accordingly, players feel competent, with growing self-confidence and perception of own-ability expansion [17].
The dimension of pleasure in Serious Games may refer to different levels.The sensory pleasure, for example, is given by the wealth of the visual stimuli (colours, forms, dimensions, etc.) and by their rhythmic succession.Second, gaming aspects involve the pleasure of the potential or actual success and victory.The attainment of the winning gives great satisfaction, since it represents the defeat of the fear of the loss [18] This level of pleasure also refers to the awareness of succeeding in doing a good job and achieving good performances, as well as to the perception of control of the situation [19].In this case the pleasure comes from the attainment of a goal that is not totally taken for granted, the so-called "functional pleasure" [20].
Moreover, in Serious Games there is also the pleasure for discovery connected with the curiosity and the desire of novelty and exploration.Tied to this form of pleasure, the emotional demand to close an assignment and completely accomplish a task becomes relevant [21].
On the other hand, as in every learning process, Serious Games have to face many issues related to negative emotions.Players have to manage negatively valenced emotions related to their possible sense of failure, such as dis-appointment and frustration for their mistakes, or even anger in front of obstacles.
Within a traditional concept of learning, mistakes and errors are potentially evaluated as impeding factors for the learning process.Etymologically, the concept of error is connected to "to err", that is to wander about without destination.The concept of error implies therefore that one is wandering without a clear clue of how to reach a goal.Erring without a precise destination is not always and absolutely negative, since it represents a creative possibility of knowing and exploring other innovative and unknown paths.Consistently with this perspective, Serious Games are able to enhance the emotional and cognitive benefits of incorporating errors into training while minimizing their negative effects [22].This is feasible given their ability to re-frame the traditional concept of mistake from a negative and frustrating experience into a positive and stimulating one.As stated by Lewin [23] we should not be too worried about the objective because it is only wandering that an individual gets to know.On the one hand, error is an objective difference between those who succeed better and worse, on the other hand error becomes an opportunity.Serious Games are featured by the balance between the possibility of exploring diverse paths proceeding via trial-and-error's strategies, and adequate scaffolding structures [24].Therefore, Serious Games embrace Lewin's vision of error and its transformation from an obstacle into a chance to know and explore.
Specifically, the three core and integrated components which characterize SG, i.e. simulation, gaming and pedagogical elements [2], contribute at different levels to a positive appraisal and evaluation of mistakes by players.
First, as far as regards simulation aspects, in Serious Games playing within simulated environments can be considered as a mode of human action serving as a bridge between reality and fantasy [25].Playful actions are limited to a simulated situational frame which blocks out any real consequences of action results.In this simulated environment, players can explore different paths and solutions without being excessively worried about possible consequences of actions as in real situations [26].
Second, gaming aspects allow to manage errors in a positive way, helping learners in reducing anxiety related to failures [27].In fact, Serious Games provide opportunities for learners to experiment and to freely make mistakes in enjoyable and protected environments without any risks for their professional career, self-esteem and self-efficacy.Also the game frame protects learners' self-efficacy and self-esteem, allows to free-up cognitive resources, to maintain focused attention and to enhance performance during learning-gaming experience.
Finally, the presence of specific pedagogical feedbacks is a third SG feature which facilitate players in considering their errors as important educational factors.Locke, Shaw, Saari, and Latham [28] highlight that goals regulate performance more reliably when feedbacks are present than when they are not.Feedbacks are provided to guide, motivate and reinforce functional behaviours.Positive feedbacks are thought to encourage individuals to set or accept higher goals [29-30-31], while negative ones are thought to create awareness of discrepancies between actual performance and goals, and to motivate individuals to change their strategies to reduce these discrepancies [32][33].Interestingly, from a meta-analysis by Kluger and DeNisi [34] it has been shown that the benefits of feedbacks are stronger when they are focused on learning tasks and contents, rather than on the player's individual identity.Serious Games generally provide feedbacks related to the specific gaming experience, choices and acts rather to the player's identity.
On the whole, given these elements, errors and negative feedback are cognitively appraised as a natural part of the learning process within the game frame [35][36].Therefore, Serious Games, promoting a non-judgemental cognitive appraisal of errors and failures as natural and foreseen steps within an educational process, have the potential to foster a learning experience characterized by more positive emotions.Multiple advantages are derived from the occurrence of positively-valenced emotions within this learning setting.Positive emotions uphold and don't disrupt active involvement in participatory appropriation, and trigger action tendencies aimed at approaching even difficult experiences.Again, positive emotions affect attention, as they capture attentional resources, and they make it difficult to distract attention towards non-emotional stimuli [37].Similarly, positive affects significantly promote memory, as it has been highlighted that tasks triggering medium emotional activation arouse memory empowerment, with more detailed and persistent recall [38].

IV. A MULTIMODAL APPROACH TO ASSESS EMOTIONS
Given these premises, an accurate assessment of affective user gaming experience and its moderation effects on learning outcomes should become a scientific priority within game-based learning field.Moreover, the measurement of emotions during gaming can provide essential information for the testing of Serious Games' and their subsequent tuning in order to achieve an optimal learning experience.
The most recent theoretical perspectives in affective sciences highlight how emotions are multidimensional and multicomponential processes, pervading our lives and having the function to monitor events and situations and to orient individual's resources according to his/her goals and interests.In particular, it was pointed out that emotional experience is the result of the complex interaction between the cognitive process of appraisal of eliciting situations, the activity of central and autonomous nervous system, the vocal and kinesic expression (facial expression, gestures, posture), as well as the aspects related to the definition of individuals' behavioral goals and the triggering of specific action tendencies [39][40].
Given this complexity, the assessment of emotional experience, requires highly sophisticated experimental methodologies and apparatus, able to measure all the different systems involved (neuro-physiological, cognitive, expressive, motor, etc.; [39]).
Currently, the research community working on emotions can rely on a consistent number of assessment methods (see [41], for a review); they range from widely used paper-and-pencil self-report scales to the coding of facial expressions, from the analysis of vocal acoustic features to the analysis of neuro-physiological correlates of emotional experience.Moreover, there is growing acknowledgment of the limitations of running studies on emotions based on single measurement techniques (be it self-report scales, facial behaviors, physiological correlates, and so on) and EMOTIONS IN SERIOUS GAMES: FROM EXPERIENCE TO ASSESSMENT researchers are moving systematically towards multidimensional and multi-modal assessment studies [42].
In the following sections we will briefly describe, in addition to self-report measures, channels considered to be more emotion-sensitive, and thus potentially used for the detection of emotions in humans: physiological correlates, facial expressions, vocal-nonverbal features, and gestures.

A. Self-Report Measures of Emotions
Self-report measures have been proposed to detect emotions, in form either of questionnaires to be administered to participant, [43][44] or of pictorial methods (e.g.Self Assessment Manikin Scale; [45]).Countless measures have been developed, tested, and used throughout the history of research on emotions.
Five comprehensive emotion self-report measures have been particularly prominent in the literature [46]: the Mood Adjective Checklist (MACL; [47]), the Profile of Mood States (POMS; [48]) the R-Multiple Affect Adjective Checklist (MAACL-R; [49]), the Expanded Form of the Positive and Negative Affect Schedule (PANAS-X, [50]) and the Differential Emotions Scale (DES, [51]).Each scale has been used extensively in studies of self-rated emotions.For example, Profile Of Mood States (POMS) is popular among sport psychologists.The scale encompasses six mood states: tension, depression, anger, vigour, fatigue, confusion.Subjects are given a score for each trait according to their responses to certain statements which include key words such as unhappy, tense, careless, and cheerful.For each statement, subjects state how they feel at that moment, or how they felt over the previous day, few days, or week, by choosing one of the following responses: not at all; a little; moderately; quite a lot; extremely.
Also the expanded form of the Positive and Negative Affect Schedule (PANAS-X) provides quick, simple and easy assessment of positive and negative emotions.It is the most widely used measurement tool in affect research [46].The PANAS consists of 55 items divided into different scales.Four scales assess specific negative emotion: fear, sadness, guilt, hostility.Three scales assess positive emotions: joviality, self-assurance, attentiveness.Respondents rate the extent to which they have experienced each emotion on a 5-point scale (1: very slightly or not at all, 5:extremely).
It has to be noticed that on the one hand, all these cited measures are psychometrically validated, non-intrusive, fast and cheap, but, on the other hand, they can be affected by distortions due to social desirability factors, as well as to the possible individual's inability to properly express her own emotional experience.

B. Physiological correlates of emotions
The autonomic nervous system (ANS) is the physiological system responsible for modulating peripheral functions [52].This system consists of sympathetic and parasympathetic branches, which are generally associated with activation and relaxation, respectively.ANS activity is not exclusively a function of emotional responding, but rather encompasses a wide variety of other functions related to digestion, homeostasis, effort, attention, and so forth [53].
In emotion research, the range of psycho-physiological measures available is vast, including: skin conductance level (SCL; [54]); cardiovascular correlates of emotion as heart rate (HR), heart beat amplitude (HBA), heart rate variability (HRV), finger pulse amplitude (FPA), pulse transmission times to the finger (PTF), pulse transmission time to the ear (PTE), pre-ejection period (PEP), diastolic blood pressure (DBP) and systolic blood pressure (SBP); electromyography; respiration rate and amplitude; level of blood glucose; temperature etc.Each of these measures varies in terms of whether it primarily reflects sympathetic activity, parasympathetic activity, or both.For example, SCL and PEP predominantly reflect sympathetic activity, HR and DBP or SBP reflect a combination of sympathetic and parasympathetic activity, and HRV has been closely linked to parasympathetic activity [55].
Ekman, Levenson and Friesen [56] suggested that the identification of distinct emotions on the basis of Autonomic Nervous System (ANS) activity requires taking into account different indices simultaneously.Researchers are continuously putting efforts in order to identify specific patterns of physiological changes corresponding to different emotions (e.g., anger, fear, etc.) considering also the extreme variability of these signals across individuals.For example, the physiological pattern of "fear" is characterized by an increase of the systolic blood pressure and heart rate combined with an increase of the values of respiratory rate and skin conductance.
Although some evidence for autonomic specificity has been reported [57-56-58], an extensive meta-analysis has highlighted such effects as inconsistent [55].In this metaanalysis, only a small set of the 37 measures reviewed reliably differentiated discrete emotions and replicable findings were specific to specific comparisons (e.g., finger temperature decreases less in anger than in fear, but finger temperature does not differentiate other discrete emotions).
Given these considerations, ANS may be viewed as responding in terms of broader dimensions, such as arousal [55-59-60].Also Lang and colleagues have shown, in a number of studies [61][62], that SCL increases systematically and linearly according to the self-rated arousal of emotional stimuli.Such findings are consistent with theories stating that ANS activity would indicates the arousal level of an experienced emotion rather than a specific discrete emotion [63-64-59].
In order to obtain an accurate detection of psychophysiological correlates, detecting instruments should interfere at minimum with individual's behaviors in its ecological setting.Various instruments for the collection of physiological signals are already used, while researchers devote continuous efforts to achieve a less degree of detection devices' intrusiveness (e.g.incorporation into more natural devices, such as mouse or wearable jackets).
Physiological data can be collected and analysed through specific devices such as for example the Pro-comp+ Thought Technology and Biograph Infinity software.Procomp+ is able to detect real-time measures of different physiological indices: skin conductance level (SCL), heart rate (HR), temperature, etc. Biograph Infinity software enables the extraction of physiological features and calculates statistics exporting values on a spreadsheet for statistical analysis.

C. Facial expressions
Facial expression has been a focus of emotion research for over a hundred years [65-66-67-68].Ekman [69-70-EMOTIONS IN SERIOUS GAMES: FROM EXPERIENCE TO ASSESSMENT 71] considered each facial expression a single and closed Gestalt, universal across cultures and essentially fixed, discrete unit, corresponding to a specific emotion and controlled by innate neuro-motor programs.According to Ekman's theoretical accounts [70], each basic emotion is univocally associated to a specific pattern of facial actions.For example increased activity over corrugator supercilii, which draws the brow down and together into a frown would be associated with negative emotions [72][73].Among several limits of this theoretical approach, it has to be highlighted that emotions' prototypic expressions occur infrequently in everyday life, since people tend more to express emotions using subtle facial actions.Moreover, this theory underestimates the role of the context in the explanation of facial expressions' occurrence [74]: out-of-context facial expressions are difficult to infer and recognize as the context is routinely encoded during emotion perception [75].
Recently, an alternative perspective has been developed specifically focusing on the connections between emotions' facial expressions and the context of use [76-77-78].According to this perspective, facial expressions adopt a precise emotional value referring in reference to a specific situation.Emotional facial expressions are not abstractly planned following fixed and universal standards, but rather in relation to the context and to the local conventional cultural standards.
As far as regards the facial expressions coding, a number of observer-based systems of facial expression measurement have been developed [79-71-80-81-82].Among these systems, the Facial Action Coding System (FACS) developed by Paul Ekman and colleagues [79][80][81][82][83] is the most comprehensive and widely used [84][85] in the research community, even by those researchers who don't share Ekman's theoretical positions.
FACS is a coding system of all visible facial movements, grounded in the detection of their anatomicalphysiological correlates.Using a componential method, FACS allow to analytically and precisely measures different motor components wholly constituting a specific facial expression.An indecomposable visible motor component (or facial movement) is called Action Unit.FACS consists of 44 Action Units, describing upper and lower face, head's and eyes' positions and movements; lots of combinations of these AUs may originate all in all more than 7.000 facial expressions.
Therefore, using FACS on video-recorded facial behaviors at frame rate and slow motion, coders are able to code nearly all possible facial expressions, decomposed into action units (AUs).
As for specific software tools, Observer XT 10 (http://www.noldus.com/)and Theme 5.0 (http://www.patternvision.com/)are examples of useful instruments for the coding and analysis of facial expressions using FACS as categorization system.

D. Vocal-nonverbal features
Among other nonverbal channels, the voice can be considered another fundamental system for the signalling of emotions, due to variations in pitch (fundamental frequency of voice, or F0), volume (intensity), rhythm, pauses, and vocal qualities [86-87-88].These changes in vocal features correspond to changes occurring in respiration, phonation, and articulation during emotional experi-ences.Over the past decades, several studies have examined the vocal cue profiles associated with different emotions.According to Juslin and Laukka [86] there is evidence for the existence of emotion-specific patterns of acoustic cues.For instance, anger would be characterized by "fast speech rate, high voice intensity, wide voice intensity variability, much high-frequency energy, high F0, wide pitch variability, rising pitch contour, fast voice onsets, and microstructural irregularity" (p.802).In turn, sadness is associated with "slow speech rate, low voice intensity, little voice intensity variability, little highfrequency energy, low F0, little pitch variability, falling pitch contour, slow voice onsets, and microstructural irregularity" (p.802).
The acoustic analysis of voice can be carried out using, for example, the Computerized Speech Lab (CSL).The CSL developed by KayPENTAX is a speech and signal processing computer workstation (software and hardware) that contains a broad range of analysis tools for efficient, accurate, and repeatable recording and measurement of speech signals.
The PRAAT software (www.fon.hum.uva.nl/praat/),free and open-source, also represents a useful tool for emotion-related acoustic profile analysis.

E. Gestures
Gestures are dynamic and characterized by a large spatial and temporal variability.These gestures' features refer to what Damasio [89] has called the "theater of emotions", which is the tangible and public representation of emotions through body movements.Culturally-grounded, several emotional gestures may have a specific symbolic value.For example, in the Italian culture anger can be manifested by showing a fist, or joy can be expressed by raising up arms while jumping.Anger is marked also by high-level of motor activity, vigorous and extended in space gestures, while sadness is connected to low-level of motor activity and feeble gestures.In anger and joy shoulders are raised up, while in disgust, despair and fear they are moved forward.Similarly, the movements of left hand are more often associated with relaxed emotional condition, while the movements of right hand are correlated with hostile and aggressive attitudes [90].So, even if specific configurations of gestures were not distinctly observed for each emotion, specific movements and distinctive motor activity features have been detected [91].Moreover, besides gestures that express emotions or feelings, through which people manifest a state of tension or relax, directly expressing the emotion "showing" its somatic response, there are gestures which can be used to describe (or symbolize) an inner state or emotion.Frequently they are symbolic gestures (e.g., hands on frown for concentration) or iconic gestures that represent visual characteristics (e.g.imitate the heart beat).Finally there are intentional and not intentional self-manipulation gestures, also called "self-adaptors".These signals seem to be directly related to the present emotion or mental state (like anxiety, stress, shyness, etc) and not necessarily related to the verbal contents while the individual is talking.
In literature, some categorization systems for gestures have been proposed [92][93].In terms of specific analysis' instruments, Observer XT 10 (http://www.noldus.com/)or Theme 5.0 (http://www.patternvision.com/),already mentioned in relation to facial expressions, may be conven-iJET -Volume 5, Special Issue 3: "Creative Learning with Serious Games", November 2010 EMOTIONS IN SERIOUS GAMES: FROM EXPERIENCE TO ASSESSMENT iently employed for the analysis of gestures and body movements.
As limitations of emotional assessment based on single measurement techniques are now widely recognized in literature, multi-dimensional and multi-modal emotions' assessment is highly recommended in whatever situations in which it would be relevant to recognize human emotions.
Indeed, for example, while self-report measures suffer social desirability effects, non-verbal signalling systems are to a minor scale subjected to intentional manipulation [94][95] or may express emotions not easy to communicate using words.Moreover, observations and measurements of facial expressions, gestures and acoustic cues avoid to interrupt the unfolding of the emotional experience, while asking individuals to verbally self-report "How do you feel now?" necessarily disrupt and change the emotional process.
Therefore, as in face-to-face classroom interactions [96] individuals (teachers and/or peers) combine different signals deriving from several signalling systems (e.g., facial expression, voice, posture, etc.) in order to infer the emotional state of a student during a task, similarly in digitalbased-game learning the integration of multiple emotional-sensitive channels represents the premise for an effective recognition of emotions.
Furthermore, as we will see in the next section, thanks to advanced digital devices, research is now strenuously moving towards forms of automatic computer emotional detection and recognition [97][98].Automated emotion recognition necessarily needs to face the additional challenges of rapid real-time detection, processing and inferences of emotions as dynamic processes, constituted by the combinations of continuously changing multiple components; also, contextual information should be properly and precisely involved in automatic emotion recognition process.
V. NEW DIRECTIONS: AFFECTIVE SERIOUS GAMING Using a multimodal approach to measure users' emotional experiences may be valuable in Serious Games' world facing aspects related to the scientific enquiry on the linkage between emotions and learning, as well as to the design and prototyping of effective Serious Games.In the testing phase of Serious Games' prototypes, in fact, the assessment of emotions elicited in players during gaming may provide precious indicators on users' experience.Emotional indicators give the opportunity, on the one hand to verify the balance between learning and fun aspects, and on the other hand to monitor the occurrence of specific moderating effects of emotions on learning outcomes.Moeover, they could contribute to identify problematic game designs before the game is released.
Therefore, combining multimodal emotional assessment to traditional post-gaming in depth interviews would be noteworthy to investigate relationships between specific games' aspects or events and related cognitive evaluations potentially triggering distinctive emotional experiences in gamers.
Some preliminary examples of this approach can be found in the domain of entertainment videogames.Drachen and colleagues [99] reported a study on subjective emotional experiences correlations with Heart Rate and Electro-Dermal Activity responses, as emotional indexes, during first-person shooter games, testing the feasibility of the use of these measures in commercial game development contexts.
When discussing the potential future applications of multimodal emotional assessment (especially in its realtime, automatic forms), a promising direction would be the development of affective adaptive Serious Games.
In the last decade, the world of affective adaptive interfaces, being purely entertaining videogames, e-learning systems/platforms, robots or virtual environments has attracted profound research interest mainly endorsed by the interdisciplinary research field called Affective Computing (AC).AC is a branch of the Human-Computer-Interaction domain focusing on the study and development of systems and devices that can recognize, interpret, process, and simulate human emotions [100].
The specific task of automatic emotional inference is actually the result of two distinct processes: the detection of affect-related signals in the user, and the recognition of her emotional state based on the processing of that information.In a few words, detecting emotional information implies first of all sensors which capture data about the user's physical state or behavior: for example, bio-sensors to measure physiological data, a video camera to capture facial expressions, and a microphone to record voice and speech.Computerized emotions' recognition instead consists in the extraction of significant patterns from the detected emotional signals and in the following inference of emotions.Based on these premises, one of the highest AC ambition is to create digital interfaces able to finely and precisely detect, recognize users' emotions and adapt their behaviours to them, giving an appropriate response for those emotions [97][98] Following the path of this research trend, in recent years the so-called Affective Gaming (AG) perspective has been developed bearing increasing attention [101-102-103-104].So far, Affective Gaming has been mainly investigated and implemented in videogames or digital games in which the only goal is players' pure entertainment and fun.In this area of application, the assessment of users' emotions answers to the requirement of maintaining high levels of engagement and playfulness.Current focus in affective video gaming is primarily on the sensing and recognition of the players' emotions, and on tailoring the game responses to these emotions [101].
For example, Gilleade and Allanson [105] developed a toolkit called the Intelligent Gaming System (IGS) in order to facilitate the creation of affective games.Using this toolkit, a videogame was developed that used the player's heartbeat rate to influence gameplay, creating an affective feedback loop.Physiological data was used to change the nature of the challenge the game presented, aimed at keeping high-level engagement.From this study, it is evident how psychophysiological user experience testing is emerging as a research area concentrating on the collection, analysis, and cross-correlation of physiological data.However, so far, few studies in digital gaming context aim at investigating and employing other emotional expressive channels towards a more complete multimodal assessment of emotions.Van den Hoogen, and colleagues [106] describe a first exploration of behavioral expressions that could serve as real-time indicators of experiences related to playing digital games, like postural responses and pressure patterns exerted on a physical control device.
The issue of affective adaptive interfaces is important in a digital gaming context where a pleasant user experience is vital to the success of the game [107].Nevertheless, affects and emotions play a key role in the user experience, not only in entertainment but, even more decisively, in Serious Games where learning process and behaviour changes necessarily entail emotional aspects and emotion-related action tendencies [108].Therefore, Affective erious Games may represent a cutting-edge research trend to be developed.Currently, some interesting contributions can be found as auspicial initial steps towards emotional adaptive Serious Gaming.
One of them concerns, the potential integration of expressive and empathic virtual agents in SG environments.Becker and colleagues [109], for example, applied in a computer card game setting the merging of two research strands of affective computing: one on an emotion simulation system for an expressive 3D humanoid agent called Max, the other one on a real-time system for empathic agent feedback that is based on human emotional This physiologic information is used to enable empathic feedback to players through non-verbal behaviors of the humanoid agent Max.Even though the integration of both systems is motivated in the setting of a cards game that is played by a human game partner and Max, it would be possible to adapt the humanoid agent to the role of "affective tutor" within a Serious Games platform.Affective virtual agents such as Max may be integrated in Serious Games, not only as games' characters, but specifically as devices able to recognize and adaptively respond to players' emotions, boosting high-quality emotional and learning experiences.
Another interesting research and development experience in terms of Affective Serious Gaming is the Play-Mancer project (www.playmancer.eu/).PlayMancer represents a current challenge to effectively implement multimodal emotion recognition and affective adaptive responses in games having serious purposes like behavioural changes and rehabilitation [110].The aim of Play-Mancer is to implement a platform for Serious Games, with a special focus on therapeutic psycho-educational support games for addictive disorders, eating disorders, or gait rehabilitation.PlayMancer platform integrates a game engine and a spoken dialog management system, and involves emotion recognition from audio data, facial expression and physiological correlates.The Emotion Recognition component of the platform detects the multimodal signals and infers the emotion of the player among a group of emotional categories, such as neutral, angry, happy, panic, fear, boredom, etc.The information about the emotional condition of the player is used as a supplementary parameter for estimating the cognitive load of the player.Depending on the purpose of the game, the emotional state and the cognitive load of the player, the Game Engine can implement different strategies, change the interaction style, or the level of difficulty at the current scenario, or providing biofeedback indications [111].VI.CONCLUSIONS Serious Games are computer or video games in which education is the primary goal, rather than entertainment.In this paper we discussed the different benefits offered by the integration of simulation, gaming and pedagogical elements at the same time.Specifically, we addressed their ability to leverage the synergy between emotional and learning appropriation processes.After a brief outline of the specific features of learning opportunities offered by Serious Games, we analyzed the involvement of emotions in this learning process.We then described the range of assessment methods currently used within the scientific community of emotion research, in order to understand how to introduce them in SG assessment and evaluation.In the final section we explored some promising research directions in the domain of affective Serious Games.
On the whole, the systematic integration of multimodal emotional assessment in the domain of Serious Games might play an important role, both for the scientific understanding of the complex interplay between emotions and learning in SGs, and for the development of SGs that fully exploit this synergy.Moreover, although still at a preliminary stage, the experiences on affective gaming show that the possibility of providing Serious Games with the ability to multimodally recognize users' emotions and to adaptively respond to these emotions (changing game's path, scenarios or characters' expressions, providing feedback, etc.) may be an important step for the enhancement of game-based behavioral changes and learning.