Cogent: a case study of meaningful gamification in education with virtual currency

This paper analyzes students’ experience with Cogent, a virtual economy system used throughout the 4 years of a B.S. degree in a Technology major. The case study explains the rules of the Cogent system and investigates its effectiveness to motivate students to learn. Using focus groups and interviews, we collected qualitative data from students about their experience and perceptions of Cogent. The results indicate that Cogent played an encouraging and motivational role for these students and suggest potential for the successful design and implementation of meaningful gamification systems to promote student motivation and engagement within an educational context.


Introduction
Increasing student motivation is a major challenge for the American educational system (Bringeland, Dilulio, 2006). In addition to motivation, evidence suggests a student's sense of personal engagement, or flow (Csikszentmihalyi, 1991), is also positively correlated to learning outcomes (Carini et al., 2006). Traditional educational approaches are not always credited with providing opportunities for such motivation and engagement (Rathunde, Csikszentmihalyi, 2005). Meanwhile, video game players invest countless hours developing problem solving skills for leveling up and reaching their final goals (Gee, 2008). It is no surprise then, that game thinking has been introduced as an educational strategy. Gamification, defined as using game elements in non-game contexts (Deterding et al., 2011) has been adopted in many fields, including education, and has gained popularity since 2010 (Hamari et al., 2014). However, gamification's potential negative impact on students' intrinsic motivation raises an academic discussion of students' dependency on consistent extrinsic motivation, as well as other possible negative effects (Zuckerman, Gal-Oz, 2014). It is therefore important to study various implementations of gamification in order to design a successful system that minimizes negative side effects. The goal of this paper is exactly that -to conduct a case study of a specific gamification system, in order to identify transferrable design elements (Stake, Trumbull, 1982) for implementing gamification systems in an educational setting. The next section establishes the paper's theoretical foundations in educational literature regarding motivation and gamification. Then, the Cogent system is described and evaluated. The paper concludes with a discussion of the results and design implications for similar systems.

Theoretical framework
This section discusses ideas that provide the theoretical foundation for our study: the issue of motivation and engagement in education; the introduction of game thinking and gamification in education as a way to increase student motivation and engagement; practices of gamification in education; and the theory of meaningful gamification.

A. Motivation and engagement in education
Motivation and engagement are recognized as major components in education, but also as major challenges for the U.S. educational system (Bringeland, Dilulio, 2006). Self-determination theory (Deci et al., 2001) provides a general explanation of motivation and how it applies to education. Selfdetermination theory distinguishes between two types of motivation: intrinsic and extrinsic. These two types work differently but not distinctly. Intrinsic motivation refers to motivation that is driven by an interest or enjoyment in the task itself and exists within individuals. As a natural motivational tendency, it is a critical element in cognitive, social, and physical development, without reliance on external pressures or a desire for reward (Ryan, Deci, 2000). Extrinsic motivation comes from outside the individual and refers to the performance of an activity in order to attain an outcome, whether or not that activity is also intrinsically motivated. Rewards for showing the desired behavior, and the threat of punishment following misbehavior are common extrinsic motivations (Ryan, Deci, 2000). Motivation is particularly important for creative work, mundane tasks, and behavioral change (Werbach, Hunter, 2012). It can be argued that education spans the range of creative and mundane activities and also requires behavioral change. Therefore, motivation is a crucial component in education. Also of critical importance is the experience of engagement, or flow. According to flow theory (Csikszentmihalyi, 1997), a flow experience is both an overall assessment and a kind of mental state that fully absorbs an individual in an activity. It is characterized by a feeling of energized focus, full involvement, and success in the process of an activity (Shin, 2006). Flow leads to increased exploratory behavior, communication, training, and satisfaction and acceptance of learning (Csikszentmihalyi, LeFevre, 1989;Hoffman, Novak, 1996). The findings of a study by Shin (2006) suggest that students experiencing high flow states were more likely to be satisfied with a virtual course than students who were not experiencing flow, or were in low flow states. With the awareness that motivation and engagement are important in education, optimizing learners' motivation and improving their engagement calls for educators' and researchers' consistent attention. One promising way to increase these factors is to borrow from activities that already benefit from Numero 1-2/2015 them -namely, playing computer games. The next section reviews studies that apply game thinking to education to increase motivation and engagement.

B. Game thinking in education
Most formal educational systems use scores as a kind of token economy. Students' learning outcomes are evaluated using a grading system and graduation is possible after meeting specific criteria and accumulating a required number of points. These features could make formal education the ultimate gamified experience where students are engaged and enjoy learning. However, something about this environment fails to do so (Rathunde, Csikszentmihalyi, 2005). In contrast, video games and virtual worlds excel at engagement (McGonigal, 2011). Within the context of games, players voluntarily invest countless hours in developing their problem-solving skills for leveling up and reaching their final goals (Gee, 2008). Students recognize the value of extended practice and develop personal qualities such as persistence, creativity, and resilience through extended play (McGonigal, 2011). Scholars argue that the problem of disengagement from school is exacerbated by its formal rules. The rules of school, as they stand, must be understood, not only in terms of their formal effects, but also in terms of their emotional and social impact on students (Rock, 2004). Applying game thinking to traditional education may offer students the opportunity to experiment with different rules, emotions, and social roles. By playing by game rules in school, students develop new frameworks for understanding school-based activities, which can affect emotional experiences, sense of identity and social positioning (Lee, Hammer, 2011). As suggested by Leblanc (2004) this can motivate students to participate more deeply and even to change their self-concept as learners. The use of game design elements in non-game contexts is defined as gamification (Deterding et al., 2011). With its wide implementation in marketing, politics, health and fitness, gamification is predicted to become a multi-billion dollar industry by 2015 (MacMillan, 2011). To provide more insight into gamification theory as applied in the educational field, the next section discusses literature on educational gamification practices.

C. Gamification practices in education
Within the field of education, gamification has been integrated in many areas, such as professional training in the area of physical knowledge (Ludvigsen, Wallervand, 2012), driver behavioral change (McCall, Koenig, 2012), and early childhood learning (Nolan, McBride, 2013). Hamari, Koivisto and Sarsa (2014) conducted a systematic literature review of empirical studies on the efficacy of gamification. Among the 24 studies they reviewed, all cases within an educational/learning context considered the learning outcomes of gamification to be mostly positive in terms of increased motivation and engagement. However, a meta-analysis of 128 studies that examined motivation in educational settings found that almost all forms of reward (except for non-controlling verbal rewards) reduced intrinsic motivation (Deterding et al., 2011). The implication is that a gamification approach may decrease intrinsic motivation (Nicholson, 2012). If a gamification program based upon external rewards is introduced, students may become dependent on it for rewards and once the rewards stop, motivation may drop to levels even lower than before the program was introduced (Deci et al., 2001). Zichermann and Cunningham (2014) proposed that gamification could be used to control behavior by replacing internal motivations with extrinsic rewards, but that "once you start giving someone a reward, you have to keep them in that reward loop forever". Other negative outcomes have also been discovered, such as the effects of increased competition between learners (Hakulinen, 2013), and difficulties of designing and evaluating tasks for educators (Dong et al., 2012;Domínguez et al., 2013). According to the research discussed in this section, gamification can be a way of increasing student engagement and motivation, but with potential negative side effects. The concept of meaningful gamification -discussed next -aims to reduce this potential negative impact.

D. Meaningful gamification
Nicholson (2012) introduced the concept of meaningful gamification as a possible solution to the problem of decreased intrinsic motivation. In meaningful gamification, design elements are meaningful to the user in ways that maintain and increase intrinsic motivation, with less emphasis on external rewards. Meaningful gamification is based on user-centered design theory and aims to replace external rewards by making connections between the nongame activity and the user's goals and needs. The theory of user-centered design emphasizes users' needs and goals as the primary consideration at every stage of the design process (Norman, 2002). In turn, this enables users to have a positive internal experience and eventually, a deeper engagement with other participants, even in non-game activities, and supporting organizations (Nicholson, 2012). Informed by the lens of meaningful gamification theory, this paper introduces a system called Cogent at [removed to anonymize]. The study is organized around two major goals (1) to introduce the Cogent system and describe how it works, in order to enable other programs to implement similar systems and (2) to evaluate the experience from a user-centered perspective. By addressing these two goals, this case study aims to inform the implementation of meaningful gamification designs in education. Case studies rely on naturalistic generalization (Stake, Trumbull, 1982) rather than statistical generalization to create transferrable knowledge to other reasonably similar situations. This study's results can inform the implementation and customization of meaningful gamification programs in education by articulating lessons learned from the use and analysis of Cogent.

Cogent and the cogent management system
It has been pointed out that the methodological limitation in most existing studies of gamification in education is the short experiment timeframe (Laster, 2010;Corbett, 2010). The novelty might have skewed the test subjects' experiences in a significant way (Hamari et al., 2014). Cogent offers a case study with a long history of usage, as it has been used in the Department of [removed to anonymize] at [removed to anonymize] University for about 14 years, and was created before the theoretical concept of gamification emerged. Although this program was designed without knowledge of subsequent gamification principles, it fits the criteria of gamification and presents several gamification features. Cogent, short for COlleGe ENTerprise, is used by [removed to anonymize] University's Department of [removed to anonymize] as an incentive and virtual currency to encourage undergraduate engagement in activities inside and outside the classroom throughout their collegiate career. The idea of Cogent was created by a professor and developed by students through several iterations. A virtual monetary-based learning environment was built, allowing students to gain experience with common business transactions such as hiring a contractor or paying taxes. Cogent is different from a simple "pointsification" program (Robertson, 2010). It operates as a meaningful game in which players/ students can create a successful business and accumulate individual monetary value. All undergraduates in the major are introduced to Cogent and given an account during the introductory freshman year course. Once signed up, accounts are managed through the Cogent Management System (CMS) website 1 . This system records students' Cogent amounts, displays an RSS news feed for students with information about real-time changes of Cogent market values and allows users to access and edit personal and group profiles. Students can also submit Cogent requests for specific activities from an administrator (usually a professor) and manage stocks on CMS. Activities are worth varying amounts of Cogent and range from grades earned in a semester, to volunteer work. Students are also rewarded with Cogent for academic and co-curricular activities such as being a part of the All-American Marching Band or having a research assistantship with a professor. Students use the Cogent they have earned to "purchase" a passing grade in the capstone experience course. During this course, students work in teams to form student companies that create realworld products. By playing with the virtual currency, students are supposed to learn how to work as technology makers as well as project managers. Each group must have an outside sponsor, either a member of the faculty or an entity from outside the school. The companies are issued stock which can be sold to raise funds for the company. Multiple factors influence the stock market, including the groups' class performance on milestone assignments and business performance on projects. The Cogent administrator, who is the capstone course instructor, determines the ultimate value of the stock. Students can purchase stocks and cash them out later, but must run the company to receive payment for its services from sponsors. They pay themselves a wage, as well as pay taxes and weekly cost-of-living expenses. Additionally, they purchase the services of students outside the company, usually while fulfilling the research aspect of the class, which also allows younger students to engage in the capstone course and earn Cogent. At the end of the semester, the administrator pays out stock dividends and students use the cogent earned to purchase their final course grade. Access to the Cogent system does not end when a student graduates. Cogent is a mechanism for maintaining engagement with alumni, as well. Alumni can continue to purchase stock each semester to buy services from students and even become company sponsors in the capstone course. Cogent aims to encourage development of students' potential by engaging them in a real business with virtual money. It connects students within the education community by employment, gives students autonomy with capital, and teaches students useful lessons about finances. In addition, students accomplish personal education goals using the Cogent platform. They have the freedom to choose their own business, projects, employees, and customize the learning experience in ways that are meaningful to them, which manifests the overarching theme of meaningful gamification. Having described Cogent, the remainder of this paper addresses and analyzes students' experience with the system.

Methods
As indicated in the review of gamification studies by Hamari and colleagues (2014), most of their samples of 24 papers were quantitative studies. Fully qualitative studies were in the minority (Montola et al., 2009;Dong et al., 2012). These reviews indicated that current gamification research focuses on usage data and inferring user behavior, without paying enough attention to the actual users and their experience. Our study used qualitative methods to understand Cogent users' perceptions. Because Cogent has been available for a long time, we were able to include both current students and alumni in this research. Focus groups and individual interviews were used as triangulation methods for collecting data from them respectively, as explained below.

A. Focus groups
As a method that can collect relatively quick and rich data on user experience (Wilson, 2009), focus groups were used to study current students' experience with Cogent. Criterion sampling was used to select participants. According to Patton (2002), criterion sampling involves "selecting cases that meet some predetermined criterion of importance". In this study, the participants had to be current full-time undergraduate students at [removed to anonymize] campus. We recruited students from three classes of different levels (freshman, sophomore, and senior) in order to have access to students from all four years. The specific courses we recruited from were selected based on their instructors' willingness to allow us to visit their class and talk about the research project. During the focus groups, the moderator asked structured questions about user experience including understanding, likes/dislikes and comments about Cogent and CMS. After the focus group discussion, a short post survey with demographic information and Cogent usage was given to each participant. Each focus group was conducted by two researchers: a moderator who asked questions and an assistant who recorded the data on audio and written notes. All research procedures were approved by the appropriate institutional review board.

B. Individual interviews
Individual interviews were conducted to collect data from alumni. We chose interviews in order to get more indepth insights, since alumni might have more complete and comprehensive opinions on Cogent, informed by perspective gained since graduation. Considering the feasibility of interviewing alumni, we identified graduate students in the Master's program who had completed their undergraduate in the same department as the criterion sample. Then, convenient sampling was used to recruit participants by inviting graduate students who fit the criteria to participate. A semi-structured interview schedule was used, containing questions that paralleled those asked in the focus groups. All interviews were audio recorded and transcribed.

C. Data analysis
The data collected from focus groups and interviews was analyzed following the fundamental tenets of the interpretive paradigm. The interpretive paradigm supports the belief that reality is constructed inter-subjectively and that the goal of research is to achieve in-depth understanding of the realities people construct around their experiences (O'brien, 2001). The procedure we used to analyze data was thematic analysis (Braun, Clarke, 2006), which enables the researcher to identify patterns and major themes in bodies of unstructured qualitative data. We followed four steps in our data analysis process, as recommended by Braun and Clarke (2006). First, the qualitative data from audio recordings were transcribed into text. Second, we identified initial codes in the data by highlighting recurring ideas that were relevant to our research goals. Third, we grouped different codes into potential themes. And fourth, we reviewed and refined themes to determine if any should be discarded or combined. Based on the criteria that data within themes should be coherent together meaningfully, and that differences between themes should be identifiable and distinguishable (Braun, Clarke, 2006), the final themes were generated according to the consensus of all researchers. The themes systematically identify user experience issues of Cogent and CMS and suggest improvements. During data analysis, it became apparent that there was significant consistency between current students' perceptions and those of alumni. Therefore, we grouped their insights into the same main themes. The themes are presented in the next section, along with participants' demographics.

A. Focus group participants
We conducted five focus groups with 32 participants, over a period of two days. Each session lasted about 30 minutes. In the fourth and fifth sessions, it became apparent that we were reaching data saturation (Patton, 2002), as the answers reflected information previously collected. Of the 32 participants, 18 were female and 14 were male, with ages ranging between 18 and 26. The participants were split about equally among the four years of study. When asked about the last time they accessed Cogent, 68% had never accessed it, 13% had visited Cogent last week and the percentages of those accessing it within last month, last semester and last year were 7%, 6% and 6%, respectively. Among all the participants, only 3% had accessed it frequently, 14% did occasionally, and the remainder had rarely or never visited the Cogent website. This pattern is explained in the Discussion section.

B. Interview participants
Four graduate students, alumni of the undergraduate program, participated in the interviews. One of them was a first year graduate student and the other three were in their second year of graduate study. They are all males with an average age of 22. The interviews lasted from 15 to 35 minutes and were conducted on different days.

C. Themes
Five major themes about the Cogent user experience emerged from our thematic analysis. First, we present themes that discuss the idea of Cogent in general (themes 1 and 2), then, specific usage of Cogent (themes 3 and 4), and finally, the Cogent Management System (theme 5). Most of the time, perspectives of current undergraduate students and alumni converged. We specify when the perspectives diverged.

Theme 1: Participation is encouraged inside and outside of class
A major theme that emerged from both focus groups and interviews was a sense of encouragement to participate in activities in and outside of class, such as internships, research studies, student associations and paid work experience. Students reported feeling that Cogent motivated them to strengthen their resume and to be mindful of future career goals beyond graduation. Graduate students emphasized that Cogent enabled greater research activity as their efforts were facilitated by the system, which compensated undergraduate participation as test subjects or undergraduate assistants. One interview participant expressed appreciation for program benefits that extended beyond the academic credit earned for participation, saying it was nice to be rewarded for building his resume -something he needed to do anyway.
2. Theme 2: Interested in the idea, but confused Cogent was seen as an interesting idea according to the focus group discussions and interviews.
It not only helped with getting an "A" grade in their capstone classes, but also provided a platform for doing business and dealing with economic matters, which students perceived as having very "real world" correlates. Participants expressed the belief that Cogent had a lot of potential. Undergraduate students, however, also expressed confusion about its origins and purpose, such as "why did this idea come up in the first place," and "what did they use before the system?" whereas graduate students generally did not pose these kinds of existential questions. One focus group participant described Cogent as the "elephant in the room" in that it everyone acknowledged it, but no one consistently explained or understood it.

Theme 3: Confusion regarding Cogent's complexity
Some students, including seniors with four years' experience, felt the initial explanation of Cogent in the introductory course was confusing and that the system itself was fairly complicated. They were not clear about what activities could be submitted, nor when and how much an activity was worth. Graduate students reported less confusion regarding operations, but also felt some components of Cogent, such as virtual living expense and imaginary taxes, were complex, tedious, and perhaps not especially useful. Students reported having learned how to earn Cogent as early as freshman year, but not actually earning much until near the end of senior year. By and large, students viewed the taxes and living expenses components as chores, rather than useful tools to aid in professional development. The general consensus was that it was peripheral and unnecessary. One participant likened its usefulness to "a racing program that taught racers to park and leave their cars on the side of the street".

Theme 4: Limited application range and utilization
Students reported a complete absence of Cogent use between their freshman and senior years; it seemed completely forgotten in their sophomore and junior years. Despite being designed for usage throughout a student's entire college career, the system was used only in the freshman introduction course and then in the capstone courses, all taught by the same professor who championed the system. A typical complaint among students was that only one in six of their professors during a given semester even seemed aware the program existed. While all freshmen were supposed to be assigned a Cogent account in the freshman introductory class, some students reported that due to certain technical and logistical issues, they weren't assigned one until they were seniors. Others mentioned they had simply forgotten they had an account altogether because it was not consistently emphasized throughout their studies. Many expressed a desire to be reminded, via regular reminder emails, or Facebook notifications. Additionally, students suggested setting deadlines for the submission of Cogent requirements.

Theme 5: Usability issues of the content management system (CMS) user interface
Users reported dissatisfaction with the current website stating that it was not intuitive or user-friendly. Each year, students had worked on improving the CMS website, but the results remained at a student/amateur level of sophistication. It was suggested that an outside web development company be hired to make it more professional and easy to use. Other requested improvements included adding e-banking functions such as online stock trading, regular bank statements, account history, a progress bar of cogent goals and suggestions for meeting semester/yearly requirements. Students also expressed the desire for the site to be mobile friendly, similar to a mobile banking application. In summary, the five major themes that emerged from the focus groups and interviews present a complex picture of the user experience of Cogent and CMS. Currently, Cogent and CMS have not fully developed their motivation and engagement potential. While Cogent is well regarded on a theoretical level, there remain many technical and logistical implementation issues that diminish its potential.

Discussion
Although there were many reports of negative user experience with Cogent, the participants recognized its potential and its direct impact on motivation and engagement. Furthermore, even given all its implementation and usability issues, Cogent nonetheless succeeded in motivating and engaging students to participate more in various activities. It also helped students learn financial skills. We can therefore conclude that the fundamental idea of a virtual economy is very promising for meaningful gamification. Student feedback can inform future designs by suggesting the following improvements: First, a good educational gamification system needs to be explained well and consistently. Its history, educational intention and planned learning outcomes should be repeatedly shared with students, and perhaps systematically integrated, so the design itself communicates its relevance to users. It is important to provide a thorough introduction to the system, but also to make frequently asked questions and documentation about it available online. Communicating to users the educational intention behind the system and its features is what makes the program meaningful, after all. Second, complexity needs to be carefully controlled. The system could be more meaningful by highlighting a few major functions related to clear learning and motivation outcomes. Alternatively, complexity can be introduced gradually. As students earn more Cogent, they can be charged with more responsibility such as paying taxes, etc. Incremental complexity based on amount of Cogent and length of experience with the system can provide a more manageable learning curve to students. Help and documentation should be available, easy to access and understand. Third, it is important to communicate information about the system internally and ensure facultywide adoption and buy-in. As students participating in this study recognized, Cogent could have much more impact if consistently used throughout their educational career, as intended. It was beyond the purpose of this study to collect data about faculty motivations and barriers related to Cogent adoption, but this is an important factor that needs to be addressed before such a programscale system is publicly introduced. Fourth, the user interface is critical to a gamification program. The program's success depends on it, to a large extent, since it communicates with users most directly and frequently. As attention to user experience and user interface is apparent in a wide range of commercial products, students expect a similarly usable and pleasant experience from educational systems. The four design implications discussed here emerge directly from this study's results. They are relevant not only for Cogent designers and developers, but also for other educational designers with an interest in meaningful gamification. The design implications, however, should be considered with an eye toward the study's limitations, as discussed next.

A. Limitations
The major limitations of this case study are related to sampling. Although our participants represent a wide variety of students from different years of the program, a systematic stratified sample of Cogent users was not feasible at the time of the research. Moreover, the alumni we were able to include in the research were all graduate students and relatively recent graduates. It is possible that alumni working in the industry, who have faced the financial realities of adult life outside of school would have a different perspective on the aspects of Cogent that this group perceived as cumbersome.

B. Validity and Credibility
Both qualitative and quantitative researchers need to demonstrate that their studies are credible.
In the field of quantitative research, the idea of reliability indicates the repeatability of results or observations and validity determines whether the research truly measures that which it was intended to measure and how truthful the research results are. Validity and reliability are also two factors that any qualitative researcher should be concerned about while designing a study, analyzing results and judging the quality of the study (Patton, 2002). The credibility in quantitative research depends on instrument construction, while in qualitative research, "the researcher is the instrument" (Patton, 2002). It seems the credibility of a qualitative research relies on the ability and effort of the researcher. Although reliability and validity are treated separately in quantitative studies, these terms are not viewed separately in qualitative research (Golafshani, 2003). The validity in quantitative research is very specific to the test, whereas triangulation methods are used in qualitative research (Patton, 2002). Triangulation is typically a strategy for improving the validity and reliability of research or evaluation of findings (Golafshani, 2003). Patton (2002) advocates the use of triangulation by stating "triangulation strengthens a study by combining methods. This can mean using several kinds of methods or data, including using both quantitative and qualitative approaches". In this study, to address validity concerns, we triangulated the viewpoints of participants from different school years, and also of both current students and alumni. We also made sure to collect sufficient data to reach data saturation (Patton, 2002). To increase reliability of data analysis, all researchers immersed themselves deeply in the data and worked together to analyze the data and reach a consensus on the resulting themes. Also, each theme was supported by citations from the participants' responses. Moreover, the interview questions were framed based upon discussions with an expert in qualitative research in order to avoid leading questions and to minimize the researcher's personal interpretations of participant responses.

Conclusion
This case study introduced a virtual economy educational system and analyzed it from the perspective of user experience, and impact on student engagement and motivation. Details about Cogent implementation are presented in order to provide ideas other educators interested in meaningful gamification can use. An evaluation of the student and alumni user experience with Cogent suggests how this idea can be implemented in ways that maximize its impact on student engagement and motivation. Based on naturalistic generalization (Stake, Trumbull, 1982), we suggest design implications for other meaningful gamification systems. Future research is needed to more closely explore the relationship and integration between meaningful gamification design and specific learning outcomes beyond student motivation and engagement.