Experiential Learning at Scale with Computer-Based Roleplay Simulations

Experiential learning is an effective method for changing behavior and teaching new material at work. Unfortunately, experiential learning is difficult to deliver at scale as it requires hands-on participation and interactivity. EmpowerTheUser (ETU)’s computer-based roleplay simulations enable experiential learning at scale. In a simulation, learners take on a role and have to react to various situations and make choices that have downstream consequences. ETU Simulations engage the learner in cycles of assessment and practice, with summative scoring in the assessment mode and real time coaching and feedback in the practice mode. ETU partnered with one of the oldest custodian banks headquartered in New York to deliver a series of three simulations on risk management to nearly 6,000 managers on five continents. User feedback shows that learners are highly satisfied with the program. Performance improved from simulation I to simulation II, indicating transfer of learning. Underperforming learners were remediated using the assess/practice/assess learning loop, resulting in a 227% improvement in performance. EmpowerTheUser Simulations provided fast, consistent training in risk management. Feedback was overwhelmingly favorable and decision tracking within the simulations demonstrated effective learning.


I. INTRODUCTION
Learning and development are essential to employee success.Experiential learning, i.e. learning by doing, promotes behavioral change and retention of training content [1], making it ideally suited to the workplace.Unfortunately, experiential learning has historically been difficult to deliver at scale, leading to the use of lesseffective passive learning methods such as on training manuals, linear didactic web based training, or videos.
New technological developments have enabled experiential learning at scale through computer-based roleplay simulations, where learners take on a role and have to react to various situations and make choices with downstream consequences in the simulation.Simulations can be delivered to hundreds of thousands of employees simultaneously.Research has shown that simulations result in content retention and behavioral change because they are a) emotionally engaging, b) require active participation and decision making, and c) give learners control and responsibility for their learning [2].

II. EXPERIENTIAL LEARNING WITH EMPOWERTHEUSER SIMULATIONS
Figure 1.EmpowerTheUser Simulations at Scale EmpowerTheUser (ETU) uses rapid collaborative authoring to enable companies to quickly develop situational simulations that incorporate personalized learning and behavioral analytics.ETU Simulations engage the learner in cycles of assessment and practice.In Assess Mode, learners complete the Simulation without assistance and receive their benchmarked scores at the end.In Practice Mode, learners receive real time coaching and feedback for each decision and can evaluate different decision-making strategies to arrive at optimal outcomes.

A. Challenge
ETU was contacted by one of the world's oldest and largest global custodian banks (GCB), who offer a variety of investment management and investment services.The importance of understanding and managing Risk Culture (i.e. the system of values and behaviors present in an organization that shapes risk decisions of management and employees) is paramount to their continued success.As a global organization, training needs to be administered consistently and effectively to a broad range of locations around the world.

B. Solution
In All three simulations focused on the same risk-related skills, here referred to as skills A-G, but in three different roles.In Simulation I, the learner takes the role of a Relationship Manager meeting with a prospective client.Their job is to negotiate the contract and onboard the client.In Simulation II, the learner takes the role of an Operations Manager addressing a problem presented by a new client.Their job is to take ownership of the problem and make a plan for next steps to resolve it.In Simulation III, the learner takes the role of a team member doing a post-mortem on the problem incident described in Simulation II.Their job is to devise an appropriate institutional and procedural response to understand why the incident occurred and how similar incidents could be prevented in the future.In all three simulations, the learner has first-person point-of-view "conversations" with relevant stakeholders: Clients, fellow team-members, and members of other teams.Learners choose their questions, and respond to stakeholders' answers, by selecting one of a menu of possible actions or statements.The options represent optimal (the best choice), suboptimal, or critical (a serious error) decisions.Stakeholders' questions and comments are shown via pre-recorded videos that play in response to the learner's choices.The simulation changes as the situation progresses to reflect the learner's behaviour.Figure 2 shows an example of one decision-point in a simulation.
The script for the simulation, both the learner's choices and the various responses of the actors, was written in collaboration between GCB L&D and the ETU Simulation development team, and reviewed by relevant Subject Matter Experts at GCB.In addition, coaching videos provided additional support and scaffolding in the "Practice" mode.
It was expected that Simulation III would be the most difficult as it involves the broadest scope, while Simulations I and II would be of similar difficulty.

A. User Satisfaction
Learners reported high levels of satisfaction with the simulations, with 80% saying they agreed or strongly agreed with 9 multiple-choice questions asked by GCB, including "The way the program was delivered [...] was an effective way for me to learn the content."

B. Quantifying Learning
Each ETU Simulation incorporates automated personlevel tracking of all decisions.Decisions can be categorized by the specific risk-related skills they involve and scored by whether the decision is optimal, a suboptimal mistake, or a critical mistake.This tracking yields rich and granular data, as shown in Figures 2 and  3. GCB established a minimum performance threshold and over 99% of learners passed on their first attempt.As shown in Figure 3, scores improved from Simulation I to Simulation II (all t-values > 9.4, p < .0001).The effect sizes of the improvement ranged from d=.13 to d=.60 with a mean of d=.40, suggesting that meaningful transfer of learning occurred from one context (Relationship Manager) to another context (Operations Manager).Scores for Simulation III dropped by an average of 4% from Simulation I, supporting the expectation that the risk manager context was the most challenging.These insights can be used to fine-tune further training.

C. Remediation through Practice
Learners whose scores fall below the passing benchmark set by GCB in Task Mode are prompted to complete the simulation in Practice Mode and then to retake the Task Mode.Forty of the 62 learners in the repeat group retook at least one simulation, and 39 of 40 scored above the passing threshold on the second try.As shown in Figure 4, on average performance improved by 227% on the second attempt.This is the equivalent of going from a score of 30 to a score of 98.1.Under-performing learners were prompted to practice and re-assess, resulting in substantial improvements in performance.Statistics on performance across simulations identified opportunities for further training.

IELA AWARD WINNER EXPERIENTIAL LEARNING AT SCALE WITH COMPUTER-BASED ROLEPLAY SIMULATIONS
2017 GCB successfully deployed a program of three ETU Simulations to a target global population of 5,728 managers in the United States, Europe and the Middle East, and Asia Pacific.The set of three 17-minute (approximate) simulations covered material equivalent to three 60-minute vILT (Virtual Instructor-Led Training) programs.This program delivered an approximate IELA AWARD WINNER EXPERIENTIAL LEARNING AT SCALE WITH COMPUTER-BASED ROLEPLAY SIMULATIONS savings of over $700,000 relative to the cost of typical vILT training.

Figure 2 .
Figure 2. Screenshot of learner point-of-view in a simulation.The left topmost dialogue box contains the learner's statement from the previous decision-point, with the stakeholders' responses below.The right-side column of dialogue boxes contain the learner's menu of response options.Once an option is chosen, the simulation will progress with the stakeholder's reply, delivered both in text and as a video with professional actors.

Figure 3 .
Figure 3. Change in performance from learners' first attempt at Simulation I (mean-centered so that the Simulation I average is 0) to their first attempt at Simulation II (with the mean for Simulation I subtracted out) for seven different skills.Vertical error-bars represent 95% confidence intervals for the average scores.

Figure 4 .
Figure 4. Change in performance between first and second attempts for learners who fell below the passing benchmark on their first attempt.Error bars represent 95% confidence intervals around the average percent change represented by the columns.