Extending the Activity Theory Based Model for Serious Games Design in Engineering to Integrate Analytics

Serious Games (SG) have been shown to have instructional potential and a number of formal models, frameworks and methodologies have emerged to support their design and analysis. The Activity Theory-based Model of Serious Games (ATMSG) facilitates a systematic and detailed representation of educational SG describing how game elements are connected together to contribute to pedagogical goals. This paper proposes and presents an extension to the ATMSG framework to facilitate the identification, selection and integration of analytics into serious games. A practical example of the approach in use in the analysis and design phase of a SG for engineering is demonstrated. Keywords— Activity theory, electrical engineering education, game based learning, analytics, serious games


Introduction
Serious Games have been shown to have instructional potential in an educational context and are capable of presenting realistic simulations of real-life situations [1], [2]. However careful planning during the design process of an educational game is essential to ensure the correct balance between gameplay and meeting learning objectives through the effective use of proven educational and game design principles where individual game elements connect together in a coherent way and contribute to meeting the desired pedagogical goals [3], [4]. To assess the effectiveness of the game implementation in helping the learner/player meet the requisite learning outcomes using analytics, the learning domain, application concept and the learner should be modelled in such a way that facilitates relevant data extraction and analysis [5], [6]. Determining and identifying which information is relevant and needs to be extracted for subsequent processing and analysis is an essential step for using analytics effectively and depends not only on the learning goals, setting and tasks to be completed but also on the game genre, mechanic(s) and platform(s) and as a result designing a general methodology to do this has proved challenging given the variety of games types available [7], [8], [9]. One possible approach to overcoming these hurdles is the use of formal frameworks, models and methodologies for serious games analysis and design which describe how game elements interact and connect together and contribute to achieving desired pedagogical goals [10]. This paper proposes and presents an extension to one formal methodology, the Activity Theory-based Model of Serious Games (ATMSG) conceptual framework, to help facilitate the integration of analytics into SG games by aiding in the identification, classification, selection and extraction of different types of learners' interactions with the learning content for subsequent aggregating and analysis. A practical example of using this extended framework for the analysis and design stage of a SG is presented and through this process demonstrates how this approach can be used to rapidly prototype simulations to teach advanced electrical and electronic circuit theory where students must practically apply their knowledge and understanding to bias a series of electronic circuits successfully to complete the game. Section II of this paper discusses challenges related to the integration of analytics into SG. Section III provides a summary of the ATMSG framework and rationale for its development and how extending the ATMSG framework could address some of the issues raised earlier. Section IV looks at the practicalities of developing SG for engineering education within this framework. Section V examines issues related to validation in SG and the use of analytics in this context. Section VI presents the conclusion and possible future work in this area.

Serious games and analytics
For Serious Games to achieve widespread mainstream acceptance as effective tools in a learning context their efficacy must be proven [11], [12]. Analytics can be used as part of an evaluation approach to assess whether games are meeting learning outcomes and can provide real-time insights into possible shortcomings of a game while providing a wealth of data about player interactions [13], [14]. Determining which information needs to be extracted from a game for subsequent analysis is an essential step in this process but the development of a generalized methodology to do this has proved challenging given the variety of educational games available [15], [16, [17]. In this context, the GLEANER (Games and LEarning ANalytics for Educational Research) system was developed to support user tracking and to analyze learners' in-game activities [18]. The system has two main parts, a Learning Analytics Model (LAM) which defines at a higher level the information required for each step in the analysis process and a Learning Analytics System (LAS) which determines how this information is processed. Through practical experience of using GLEANER a set of universal high-level traces were identified comprised of game, generic and genre traces which are common across a variety of educational video games and have the potential to be used to track learner progression and for assessment [19], [20]. A game start trace is generated at the beginning of the game and includes relevant additional information e.g. a timestamp or tracking id for a particular game session. A game end trace is generated at the completion of the game session and includes data about the length of the session and the overall time spent playing. A game quit trace is generated when the player leaves the game and usually includes some context e.g. at what game stage did they leave and how much of a stage did they complete. Phase change traces are related to the internal narrative structure of the game e.g. number of levels or chapters completed and are subdivided into phase starts and phase ends. Meaningful game variables relate to in-game events and contain data related to the game logic e.g. time taken to complete the current phase or the score achieved. Input traces are user generated and offer the ability to record every single click, tap or interaction with the game. However, not all the traces/information extracted from a game are equally meaningful and these complex systems can produce large amounts of partially irrelevant data which must then be filtered before further analysis [5]. In order to efficiently explore, analyze and understand how the learner is interacting with a game, the entire experience should be designed holistically using formal models where learning outcomes/mechanics are clearly mapped over to game mechanics which can subsequently be mapped over to the identification and selection of events (or traces) to log which are unambiguous and sufficient for understanding the context in which they happened. In addition, the approach used should ensure that no key information is lost or significant data not evaluated [21,22].

ATMSG framework and approach
Carvalho et al reviewed models, frameworks and methodologies to investigate and analyze serious games and concluded that there are some notable deficiencies in existing approaches i.e. overly focused on the high-level aspects of a game, no consideration of educational elements and not offering sufficient explanation of how the game mechanics contribute to the realization of the higher-level game objectives [23]. To address these issues a modified version of the LM-GM (Learning Mechanic-Game Mechanic) model was proposed which used the conceptual framework of activity theory [24], [25], [26] and existing taxonomies of games, learning and instructional design theory to understand the structure of educational serious games and the relationship between games mechanics and the educational goals of a game [18]. Using the Activity Theory-based Model perspective, educational SGs are viewed as complex, dynamic systems comprised of three main activities: the gaming activity, the learning activity and the instructional activity with two main subjects, the learner (player/student) and the instructor (whose roles include game design and development), each with different motives e.g. have fun, fulfil a course requirement or engage a student. Games can either be self-contained (intrinsic instruction) or require additional/external effort or resources from the instructor (extrinsic instruction). The hierarchical structure of this approach allows a change of focus in the game analysis on to different levels of detail, providing a flexible tool to analyze and design interaction and gameplay where activities can be divided into actions and the game into smaller pieces i.e. each activity is broken down into a sequence of actions mediated by tools with specific goals and can be classified as gaming, learning or instructional according to the activity supported. Actions can also be broken down into their constituent operations to an extent where the SG is seen as a combination of its low-level elements (buttons, menus, etc.) which mediate operations performed by the subject (reading text, clicking a button, etc.). The ATMSG approach involves 4 progressive stages of analysis which guides the instructor from a high-level overview of the activities taking place to an understanding of how the core components contribute to implementing these activities aided by existing taxonomies of games, learning and instructional design theory (Fig 1). In the first step, the instructor describes the main activities involved in the activity system and identifies their subjects and corresponding motives. Each subsequent description increases the understanding of the game and the main aspects of each activity, encouraging the instructor to observe the game from a number of different but complementary aspects (Fig.1).

Fig. 1. Four-step approach for applying ATMSG analysis
In the second step the game sequence is created as a diagram to aid in the identification of the main elements of the game establishing a starting reference point to uncover how aspects of the activity system are connected (Fig.2). The flexible approach can be used across game genres and visually describes the overall structure of the game showing key elements and points of interest. The sequence is created using Unified Modelling Language (UML) activity diagram notation [27]. In the third step the instructor identifies elements related to each node of the game sequence where each event in the game is decomposed into its actions, tools and goals i.e. "what is the learner doing, how, and why?". The instructor chooses the relevant element directly from an existing taxonomy of serious game mechanics [10] which are then represented in a layered table, vertically matching the node of the game sequence to which they are related. In the fourth step the instructor groups each set of actions, tools and goals that are from the same type of activity and that are related to the same node of the game sequence. For each of those blocks, the instructor provides a more complete description of their implementation, explaining what is being done at that point in the game, using which tools, and for what purpose and how these elements are being used to support the pedagogical goals of the game. This is done for each type of activity. When these steps are completed the instructor has a comprehensive view of the structure of the game, the learning and instructional elements and its implementation.
Who is the player? Why is the subject playing? What are the general objectives of the game? Learning Who is the learner? Why is the subject engaging with the game? What are the games learning objectives? Intrinsic instruction Who designed and produced the game?
Why was the game produced? How is the game conveying its learning contents? Extrinsic instruction Who is using the game to teach?
Why is the subject using the game? How is the game used to teach? Is there external supplementary material.

Phase 2 -Analyze actions (intermediate/low level)
Step 2 -Represent game sequence Step 3 -Identify actions, tools and objectives Step 4 -Extend description of implementations

Phase 1 -Analyze activities (high level)
Step 1 -Identify/describe activities in the activity network

Extending the ATMSG approach
The hierarchical nature and structure of the ATMSG approach allows the instructor to focus on different levels of detail in a game i.e. actions can be broken right down into their constituent operations and low-level elements. This paper proposes adding another step (map actions to the appropriate game traces) to the ATMSG analysis process (Fig. 3) and adding an additional game trace layer (Start/End/Quit game, phase changes, meaningful variables and input traces) to extend the game sequence diagram. This allows the instructor to map actions and tools to the appropriate category of identifiable game traces [20]. The sequential/layered nature of the diagram layout, which identifies the game elements and major/noteworthy game events/variables and interactions along with additional detail related to game, learning and intrinsic instruction help the instructor in this process by identifying events/traces in the game that are indicators of learner progress and show that notable/actionable milestones have or have not been reached. The next section demonstrates the use of this extension in a practical example.

Phase 2 -Analyze actions (intermediate/low level)
Step 2 -Represent game sequence Step 3 -Identify actions, tools and objectives Step 4 -Map actions to appropriate game traces Step 5 -Extend description of implementations 4 Game based learning for engineering The Circuit Warz project was conceived to investigate if creating compelling, engaging, immersive and competitive games to teach electronic circuit theory and principles would increase student engagement [28]. An initial game prototype was developed based on the principles of positive feedback in an operational amplifier oscillator. Oscillators are astable devices that produce an alternating or pulsing output voltage which is primarily dependent on the values of resistor/capacitor combinations chosen. The game core loop centered around presenting the learner with randomly generated output values/responses from the circuit i.e. peak to peak voltage (Vpp) and period of the waveform and the formulas to calculate these values (Fig 4). The learner had to calculate/select the correct value(s) of individual circuit components i.e. resistors/capacitors, to generate the given circuit output/response based on a known value of in-put/stimulus and the formulas provided. A score was awarded based on how close the value of actual output of the circuit (frequency/Vpp) was to the target output and provided feedback to the student on their level of under-standing of circuit theory [16]. The game logic, component values and outputs were modelled in Excel to fine tune core gameplay (Fig. 5).  After the completion and validation of the approach taken for the development of the game prototype the design of the full game commenced. A cross platform, single player game was envisaged, built using the Unity3D engine [29], which would be used as part of a blended learning approach and as a supplementary resource to complement/augment existing teaching resources on first year of an undergraduate electronic and electrical engineering course [28]. The extended version of the framework was used in the game analysis process where the first step was to identify the main activities involved in the activity system, their subjects and corresponding motives (Table 1).

Table 1. Initial analysis to identify activities description
To engage and immerse the learner in the game, a simple and easy to understand back story was added using the heuristic framework for the design of educational games, where the game is considered as a narrative [30], [31]. The heuristic approach provides a systematic and structured approach to the integration of the story elements i.e. game backstory, design of the physical environment and settings, player challenges, character design, appropriate puzzles, player feedback and game resolution with the learning outcomes (Fig.6). The game is set in the near future when the Earth is under imminent threat of alien invasion. As the alien ship approaches Earth, it passes our last line of defense, a giant laser facility on the Moon. However, the generator/laser was sabotaged by the aliens and malfunctions. The learner/engineer has to solve a series of increasingly difficult puzzles through the practical application of circuit theory under severe time constraints, while under attack by the compromised moon base security system (sentinels), to fix the generator and fire the laser and save the planet from destruction. The game scope was extended to include seven increasingly difficult levels for the learner to complete, based around fundamental electronic and electrical circuits typically found on first year undergraduate engineering courses. Individual levels in the game provide landmarks to support orientation and integrate the game elements and learning objectives in pedagogically meaningful ways by embodying units of knowledge with concrete, focused activities involving a sequence of small tasks to develop skills, each of which has a specific instructional target or learning out-come [32], [33], [34]. To complete the game successfully the learner is required to having a high level of understanding circuit theory. Table 2 provides a description of each game level, circuit type, learner objectives, related circuit theory, learning outcomes and game mechanics. Each individual level encompasses all the game mechanics described. Table 3 extends the ATMSG analysis, mapping game mechanics to learning mechanics selected from an existing taxonomy of serious game components [10]. The mapping of the learning objectives and outcomes to game mechanics involved establishing the game context using a backstory (cut-scenes) which defined the learner's objectives and rationale for subsequent actions (Fig.7). A tutorial introduced the learner to the core mechanics where they became familiar with the controls and user interface. The core loop of the game (across each of the seven levels) involving solving increasingly difficult circuit problems in stages, where the learner explores each level, tries to understand its purpose (cascading information) and how to efficiently solve the problem (strategy) using a simulate/response approach to observe, experiment and analyze circuit behavior under time constraints. This was facilitated by dynamically changing the value(s) of the target output responses of the circuit and ensures that the learner has to fully understand and practically apply the underlying circuit theory to successfully complete each level at each attempt. The end of each level provides feedback to the player on their progress (score achieved). Table 2. Game stages and levels with player objectives Table 3. Circuit Warz extended analysis The final level/generator room provides an overall score (assessment) and completes the story arc. To ensure the game has replay value and offers new (educational) challenges each time the game is played, elements of the problems to solve are different each time a level is attempted. The physical layout of the levels, the time pressure element and the design of the game puzzles compel the learner to make strategic decisions about how to complete the game as the overall score obtained is based on a combination of accuracy and time taken to complete each level. This allows the learner to decide on whether to take more time to accurately calculate the values of the individual components to obtain the required target output or save time by using a "rule of thumb" or intuitive approach to circuit biasing e.g. estimating current flow relative to electrical resistance.  This element of the game, based on behavioral momentum and repetition (of the core game loop) helps to further develop the learners understanding of circuit analysis by exploring different strategies and approaches to circuit biasing. The game leaderboard and achievements allow the learner to compare their score, performance and achievements earned against other players adding a competitive element and replayability. The final game sequence and structure (Fig. 7) shows each notable event in the game decomposed into its actions, tools and goals. From this process events/traces/variables in the game, which are effectively indicators of a learner's progress or actionable milestones were identified and extracted for later analysis (Fig 8).

Game implementation
The game uses a first-person shooter/perspective. This viewpoint and control system was chosen as the overall experience was intended to be a fast-paced action game which is appropriate for the sequential, level based layout of Circuit Warz (Fig. 9). This perspective allows the learner to experience the action through the eyes of the protagonist and provides greater immersion into the game and meant that game implementation did not require the overhead of designing a full third person character negating the need for complex camera control systems. The inclusion of the sentinels (base guards) and gun were added to increase the intensity and pressure of the game, reinforcing the idea of an imminent alien invasion and possible base infiltration, adding a sense of urgency to complement the backstory and overarching narrative (Fig.10). The physical layout of individual circuits on each level was accurately recreated. This is required as the circuit layout and physical operation have to accurately reflect the constraints of their realworld counterparts (Fig. 10). Orientation and overall cognitive load are important considerations in the game design. Timely and meaningful feedback in a game, particularly related to progress and rewards, is essential in educational products as it motivates students [28]. There are two main feedback/progress mechanisms to provide orientation and status information in Circuit Warz, the head-up display (HUD) and the generator status board. The HUD (Fig. 11) is accessible to the student at any time and gives context/location related information on current location, objectives and task(s) to complete, component values and current status, target value(s) to achieve or achieved and score on level. The generator status board (Fig.11) provides progress and status information about the current state of the game i.e. remaining tasks to complete.  This board is available outside of each game level with a final board with the total score and time taken to complete the game on the final reactor/laser level. The game uses a summative and formative approach to assessment based on data collected from the game to demonstrate that the stated learning objectives are being met by a learner [35]. Summative assessment is carried out at the end of the game with a total accumulated score based on overall time taken and score on each level. Formative assessment is stealth based/implicit and carried out throughout the game, continuously monitoring student progress and providing feedback through the HUD and Generator status boards. Elements of formative assessment include the time taken to complete level(s) and score achieved per level. This approach has a number of advantages as it can be carried out in real time without interrupting the learner's flow [36].

Game validation
Validation of a game is essential to ensure it achieves its intended purpose and outcomes [37]. The validation approach taken here followed a four step procedure: (1) analysis of the learning outcomes for each level and performance indicators ( Table 2 and Fig. 8), (2) development of learning/game mechanics and detailed game scenarios (Tables 2+3), (3) careful design of the scoring mechanism), gameplay challenges (i.e. time constraints to complete each level and possible components permutations) (Figs. 7 + 8) to ensure that the performance indicators relate directly to the learner meeting the learning outcomes i.e. high scores are only attainable through a deep understanding of the theoretical content and its practical application and (4) performance indicators are cross checked with in-game measure traces in the analytics (Fig.8) i.e. the recording of a high number of component selections/connections by a learner on a level would indicate that a trial and error approach was used. Advances in game and analytics allow instructors to gain new insights into how learners interact with their games by simplifying the collection of large amounts of data. Serrano-Laguna et al., proposed a two-step generic approach to using analytics in educational games [38] where in-game measures/generic traces are gathered from gameplay (Fig. 8) and then queried using specific assessment rules aligned with the games educational objectives. The Game analytics package [39] offers similar functionality in the form of core and custom metrics. Core analytics measure standard metrics e.g. general game usage, daily and monthly active users, time of day and length of sessions. Custom metrics can record game specific actions or traces of interest which are decided on by the instructor during the game design process e.g. level completion or score which can be cross referenced and analyzed further using cohorts and funnels [39]. Circuit Warz uses a combination of core and custom analytics to track learner activities. Figure 12 maps in-game measures discovered earlier (Fig.8) to game analytics and their subsequent use in the game. Core analytics allow the measurement of learner engagement/retention levels (i.e. daily/monthly active users and session times). Custom analytics track phase changes (e.g. level completions), meaningful variables (e.g. levels scores, HUD usage and component connections/disconnections) and design events which can be used to track multiple types of data in a game (e.g. tutorials completed). These metrics can be cross referenced using cohorts and funnels to carry out further analysis of learner activity. Funnels allow the visualization and measurement of learner movements through a series of predefined events/stages and determine if particular game stages, elements or levels are causing difficulty e.g. how many learners completed level 6 with a score of 100% or how many times have learners started but not successfully completed level 7. Cohorts are groups of learners that completed specific actions within specific time periods. From an instructor's perspective, the use of custom measurements, along with funnels and cohorts, inside the game analytics platform allows the exploration of the usage data to determine learner retention i.e. continuous and ongoing engagement with the game. Custom measurements can be used to check how many levels the learner completes which can then be cross referenced using funnels and cohorts e.g. how many learners who completed level 1 subsequently completed level 7 within a defined time period and returned regularly to the game over a period of months. As the game design evolves in later iterations or is enhanced, the use of analytics would allow the instructor to check the impact on usage and retention these changes caused e.g. changing the relative difficulty of a level and adding or removing features. Using a combination of assessment and analytics can provide instructors with the tools to quantify the effectiveness of the learning activities and can serve as a starting point for validation to evaluate if the game achieves its purpose and learning outcomes.

Conclusion, future work and discussion
This paper proposed an extension to Activity Theory-based Model of Serious Games (ATMSG) conceptual frame-work to add an additional step into the analysis process and to extend the layers of the game sequence diagram to add a game traces section based on the two-tiered approach to using analytics in educational games [17]. These additional steps and functionality subsequently aid the instructor in mapping actions and tools to the appropriate category of identifiable game traces and formalize the process of determining which information needs to be extracted from a game for subsequent analysis. The Circuit Warz game was introduced and its implementation using a heuristic approach presented. The practical use of the ATMSG in the game design process to map the pedagogical elements/learning outcomes to game elements while maintaining the balance between entertainment and learning was demonstrated. An approach to the design and integration of analytics to assess student retention and engagement levels was presented. The game design and implementation phase of the Circuit Warz project is now complete and the approach taken potentially offers a new, engaging and highly interactive way to teach engineering related material. The game has been released and the next stage in the project is the evaluation of the game with the target user demographic to prove the efficacy of the approaches taken.