An Overview of Competency Management for Learning and Performance Support: A Canadian Perspective

Despite the turbulent economy, recent expenditures on workplace learning in North America have increased. Technology-based methods including tools that enable social learning are making significant gains and account for 39% of all training hours in 2012. A majority of companies are moving from static classroom training to workplace learning that is more interactive and driven by technology. Companies actively experiment with new methods such as personalized learning, performance support, and gamification to encourage employees’ motivation to learn and promote continuous workplace learning, practice and application. However, the divide between the training and competencies people have and the training and competencies companies need still remains. The National Research Council Canada (NRC)’s Learning and Performance Support Systems (LPSS) program, by implementing adaptive and personalization strategies, develops software components for learning, training, performance support and enterprise workforce optimization. These technologies have the potential to facilitate lifelong learning, reduce learning and training costs, and reduce demands on physical infrastructure. Software components being developed for learning, training and performance support also enable streamlined and rapid skill development, as well as reduce time to competency, support informal, personal and personalized learning, increase learner engagement, address workforce optimization and sustainability, and increase operational performance and productivity. An overview of the LPSS system and capabilities is presented along with the results of our review of the current state of competency management in Canada and some challenges in this area, followed by recommendations for further work on competency functionality in the context of the LPSS program. Keywords—Competency management, learning and performance support, personal learning environment, workplace learning.


Introduction
In this paper we provide an overview of the competency management frameworks and workplace learning in Canada based on our literature and market review that aimed to provide recommendations for competency functionality development within the Learning and Performance Support (LPSS) program at the National Research Council of Canada (NRC). The LPSS program is a combination of research and de-velopment work conducted by researchers in learning technologies and AI at the NRC. Starting in 2008, connectivist-type Massive Open Online Courses (cMOOCs) were launched as part of the research and development work on Personal Learning Environments (PLEs) at NRC. An overview of our previous and current work, including the LPSS program and capabilities, will be presented in the paper followed by recommendations for development of competency management functionality in the context of the LPSS program.

Personal Learning Environments (PLEs)
Since 2008, the National Research Council of Canada (NRC) has been engaged in research and development of software technologies to support networked learning environments outside formal education, including learning in the workplace. Such environments are referred to as personal learning environments (PLEs). Some of the features of these environments include intelligent information streams with editing and publishing tools, scaffolds for learning and communication and support structures for learners [1]. The important distinction of PLEs is that the learners typically choose the content, the learning path and work at their own pace, while in traditional elearning platforms the content is decided by the provider and the Learning Management Systems are mostly course centric [2].
Learning personalization addresses the need to adapt materials for individual leaners in order to increase their learning performance. This is done by taking into account individual learning styles and social aspects of learning [3][4]. Individual user characteristics are crucial to consider in designing features that are relevant for advancing quality learning and enabling positive learning experiences with proper performance support mechanisms in place, beyond a one-size-fits-all approach [5]. Personalization has been shown to increase learner motivation and improve learning efficiency and effectiveness [6][7][8].
The design and development of next generation learning environments, including PLE ecosystems include a common set of standards to enable personalization while utilizing available tools and components, with learning resources and interaction derived from learner profiles, and features designed specifically to meet individual needs and preferences. Our market searches revealed current technology design considerations and strategic areas of focus related to the personalization of learning [8]. Table I provides a summary of our findings related to personalization trends. Critics of personalization argue that the software system is still very much in control, not the users; however, the literature on PLEs points to interesting opportunities for creating serendipitous moments that are valuable for learning through content aggregation [9]. For example, PLE mashups could create an environment for serendipitous discoveries. Mashup environments within PLEs include content from different sources, tools, and applications defined by the users and based on users' needs and personal preferences, and displayed in a single interface that give learners control over the content. The common building blocks for a mashup PLE ecosystem include: access features, user engagement and relevance features, knowledge based services, social media services, and content management services [10].
Learning openness is also important within PLEs, not only for enabling the diversity of content from different sources that are connected to groups and networks, but also for creating extended opportunities for sharing ideas in an environment that fosters social or crowd-sourced serendipity [9]. In today's digital world, a web of learning resources surrounds every individual. It is an environment where each resource connects to others, creating an overall structure in which all learning takes place as a new learning ecosystem [11].
Learning personalization research is currently in its infancy. Researchers are experimenting with different ways in which information can be personalized for users, with judgement calls at work in deciding what to include and what not to include. Information could be personalized via mobile device personalization (technical specifications/issues); student level (academic levels); student preferences (graphics, video, audio, text); network speed; subject specific and location specific (e.g., location of a student, sending materials specific to that place, such as the library vs. coffee shop) [4]. The evaluation of experts, such as personalized academic researcher recommendations [12] and personalized web searches [4] could also play a role in the personalization of PLE ecosystems. Another level of personalization involves delivering personalized learning pathways and learning resources based on users' existing and desired competencies [8].

Massive Open Online Courses (MOOCs)
Massive Open Online Courses (MOOCs) have become a part of the personal learning ecosystem and informal learning landscape, offering a range of open and accessible learning opportunities to learners worldwide [13]. These technologies make it possible to connect with other people, exchange information and create knowledge on an unprecedented scale. The term MOOC (Massive Open Online Course) was coined in 2008 by George Siemens (University of Texas Arlington) and Dave Cormier (UPEI) who facilitated their first online course with hundreds of participants distributed geographically. The content, communication and collaboration within this first MOOC, a connectivist-type MOOC (or cMOOC), were hosted by a large variety of social media platforms.
Distributed platforms, autonomy, diversity, openness, and connectivity have been identified by Downes and Siemens to be key principles for learning in networks [14][15]. This is reflected in four key activities in MOOCs: 1) aggregation (filtering, se-lecting, and gathering personally meaningful information); 2) remixing (interpreting the aggregated information and bringing to it personal perspectives and insights); 3) repurposing (refashioning the information to suit personal purposes); and 4) feeding forward (sharing the newly fashioned information with and learning from other participants) [16][17][18].
The connected aspect of learning is brought to the fore in a cMOOC -it's a chaotic experience that is inherently personal and subjective, as participants create their meaning, build, and navigate their own web of connections. A cMOOC framework was proposed for conducting research in the area of MOOCs as a particular instance of PLE [19]. The ultimate aim of this work was to help online learners in the context of PLEs to work more effectively and to contribute to a higher level of engagement and learning. Efforts to develop support systems within MOOCs included components such as a profiler, an aggregator, an editor, scaffolds, and services.
Research and evaluation efforts around MOOCs as an instance of PLE [19][20][21] have provided important baseline data about user experiences with emerging technologies in open educational environments. Considerations for developing personalized learning systems include providing recommender systems that list new courses and options for the user, with the capability to get further details about the recommendations, and suggestions for appropriate learning activities based on learners' competence levels and learning styles [8]. MOOC providers can now introduce the functionality of signing up with LinkedIn (via OAuth) to exploit different fields of user profiles to provide personalized MOOC recommendations. Research indicates that a skill-based user modeling strategy performs best in providing good personalized MOOC recommendations [21]. Other research suggests that learning styles affect the quality of performance in e-learning environments; students who use different learning styles did better in the course and those who were collaborative learners scored the highest [7].
Findings from NRC case studies of PLEs and MOOCs have suggested that the design of quality learning experiences in MOOCs is a complex undertaking that is impacted by much more than tools and technologies [21]. In particular, the importance of human factors in creating motivation, incentives, and providing support (organizational, social networks, either online or in the community) for creating high-quality learning experiences has been highlighted.

PLE Evaluations
There are several ways to evaluate the impact of learning systems, including measuring variety (level of emergence from individual learning paths to successful learning routes); centrality (number of ties to other learners, indicating connectivity); closeness (degree that they are close to other learners -direct or indirect connections); cohesion (how strongly they are connected, peer groups); usability, efficiency; effectiveness; and dropout rates [8]. The literature on evaluation of personal learning environments and recommender systems highlights user satisfaction as one of the important impact factors [4], [7], [8], [12]. User satisfaction is close to the motivation of a learner and therefore is an important measure for learning [8]. User satisfaction (also called User Information Satisfaction (UIS) in the literature) is defined as the extent to which users believe that the information provided to them meets their information requirements [12]. There are various ways in which user satisfaction could be measured, either directly (asking the user directly through surveys or usability evaluations) or indirectly (examining click through data and logs from search engines).
Best practices for evaluating personal learning environments emphasize the need to run studies with as many users as possible due to the high dropout rate and the varying levels of participation amongst users [8]. There also is a need for stronger statistical analyses in evaluating personal learning environments, with studies conducted with regular measures and over fixed periods of time as determined by the researchers. In addition, there needs to be a common virtual learning environment (e.g., Drupal, Moodle or in house solution) for all users [8]. When evaluating the learning system, researchers should consider not only the technical measures but the actual needs and characteristics of learners and how these learners interact with the system [8], including learners' requirements for competency development and their satisfaction with suggested learning resources.

Competency Management Definitions
Many definitions of competency exist, for example Gartner [22] defines competency as a set of characteristics of an individual that are observable, measurable and predictive of superior performance in a given role. These characteristics define how people get their job done. Others have suggested that competencies include a combination of observable and measurable knowledge, skills, abilities and personal attributes that contribute to enhanced employee performance and ultimately result in organizational success [23]. The Canadian government definition of competencies is [24]: "…the knowledge, skills, abilities and behaviors that an employee applies in performing his/her work and that are the key employee-related levers for achieving results that are relevant to the organization's business strategies." The National Research Council Canada (NRC) definition of competencies is [25]: "An observable and measurable knowledge, skill, ability or personal characteristic required by NRC employees to achieve the superior performance output/outcome needs of the organization of excellence."

Competency Management Frameworks in Canada
In our literature review we looked at competency management research and frameworks worldwide. In addition to the leading role of the United States [22], [23], the area of competency management, frameworks, competency-based education and competency matching is a very active and extensively funded research area in the EU. There has been several EU funded projects that developed technologies for competency management and competency based education such as TENCompetence (Building the European Network for Lifelong Competence Development), PROLIX (Processoriented Learning and Information eXchange), TRACE (TRAnsparent Competence in Europe) and WATCHME (Workplace-based e-assessment technology for competency-based higher multi-professional education). EU projects on competency management and competency based education develop technologies focused on lifelong competence development, personal competency management, authoring tools to help users organize and coordinate learning environments, activities, competencies and learning paths, as well as ePortfolio to help lifelong learners to reflect on the competences and competence profiles they have acquired [26], [27], [28], [29], [30], [31].
Regrettably, the area of competency management and competency-based education does not get similar attention and research funding from the Canadian government and industry, with the exception of the healthcare industry. Our literature review on competency management frameworks revealed that the bulk of the Canadian literature on competencies deals with medical education and professional development, within the scope of the Canadian Medical Education Directives (CanMEDS) Framework for Canada's medical postgraduate training programs [32][33][34]. The CanMEDS framework was developed in 1996 and since then has been modified for use in other countries [32]. Only few other professions and sectors in Canada employ competency-based education and competency frameworks. For example, the competency framework has been developed for public service sectors in Canada based on the existing job classification system in place within the Canadian Public Service.
Reference [35] outlines the implementation of a competency framework in the Canadian Federal Public Service. The framework is based on the existing job classification system in place within the Canadian Public Service. The framework contains core competencies for all public sector employees making up more than 140 competency profiles. This competency framework is available through the CBM (Competency Based Management) Web Suite which is accessible to all employees and provides information on CBM and on various tools such as the national competency dictionary, competency profiles, competency self-assessment questionnaires for employees, assistance on developing learning plans and the "National Learning Inventory" which links all departmental learning and development activities to competencies. The site also provides online tools and information for managers to assist them in applying competencies to staffing and other HR processes.
While some Canadian Public Service departments have embraced competency management frameworks, some others have not, or they adopted the competency framework only partially. For example, based on the core competencies in the Canadian Public Service Framework, NRC has developed behavioral competencies for nine roles in the organization including NRC-wide competencies [25]. They include: However, while NRC's behavioral competencies are widely used for the leadership training within the organization and in hiring, the technical (functional) competencies framework is not developed yet; this demonstrates the challenge in implementing comprehensive competency frameworks in Canadian government organizations.

Implementation Challenges
Our literature and market review revealed that there are significant challenges with the implementation of competency-based training and management in Canada, including lack of a national competency classification system, non-uniform implementation of accepted medical competency frameworks, such as CanMEDS, and the fact that competency assessment methods vary widely and could be unreliable. Another challenge in broad implementation of competency frameworks is that there are no universal standards applicable across countries, provinces and/or professions; besides, frameworks require constant revisions and updates. The particular challenge in using competency frameworks for small and medium size businesses is that since the process of developing a competency framework internally is costly, the companies have to rely on purchasing competency frameworks from other businesses or acquiring ones developed by industry associations, if they exist.
Significant difficulties also exist in capturing and especially assessing informal learning (learning that takes place outside of a lecture-based curriculum) [31] in the workplace and beyond, and incorporating knowledge acquired via informal learning (including prior-learning assessment) into the competency framework structure. In addition, researchers observed some user resistance in accepting competency frameworks in industry and in medical training [32], [33], [37], [38].

Competency-Based Education and Prior Learning Assessment
Competency-based education (CBE) and prior learning assessment (PLA) are both important strategies and companion tools for post-secondary education programs and workplace learning. CBE focuses on what students should learn rather that where or when the learning occurs, while PLA presents strategies for evaluating informal learning [39] Canadians was fully recognized, between $4.1 and $5.9 billion in additional income could be generated. The same study found that Canadian post-secondary education has established significant barriers that prevent the recognition of credentials other than those gained through formal education.

Workplace Learning and Competencies
Traditionally, workplace learning and training varies for different skill levels of employees [42]. Text-based training and formal on the job training are considered more effective with lower skilled workers, and e-learning, both synchronous and asynchronous, is considered more effective for higher skilled employees. However, the asynchronous learning modality is becoming more common for all learners, with training delivery moving to a more asynchronous environment and trainers serving as moderators of social learning [41]. In the long term Frost & Sullivan [42] estimate that organizations may move from formal learning management systems (LMSs) to informal learning environments (such as PLEs) where people combine websites, ebooks, blogs and other resources in order to support learning.
According to reference [43], online competency-based learning is also on the rise. This type of learning credits learners for what they already know and targets specific skills or competencies required for the work they are currently doing [44]. Industry competency management practices are closely intertwined with learning management. The functionality of both competency and learning management systems are frequently included in a single software application, or two software systems might be integrated. The software tools can be used to schedule training sessions, keep track of attendees, and create reports on who completed each session [45]. New LMSs incorporate social media, training, certification and mentoring systems. Similarly, highly evolved learning and performance ecosystems are made up of a combination of talent management, performance management and knowledge management; they also provide access to experts, social networking and collaboration, and structured learning [46], [10].
About 10 to 30% of workplace learning occurs through formal development yet 80 to 90% of the learning and development budgets are spent on this [47]. According to [48], while formal processes always will have their place, informal processes and interactions are becoming more important to facilitate career growth and learning. In spite of the recent and continued emphasis on social media to support learning, the most significant types of informal learning are not currently technology based: namely, on-the-job experiences, mentoring, and discussions and networking with other professionals [49]. The major shortcoming of the competency management and competency-based training in the workplace is the absence of a direct mechanism to translate informal learning in to the competency management framework infrastructure. This creates a problem since a major portion of learning at work is informal and 87% of Americans say Internet helps them learn new things [37], [50]. Currently there is a significant level of activity within the industry related to personalization of workplace learning, including the following top rapidly growing elearning sectors and training trends such as: individualization; virtual reality; mobile learning; gamification; social learning; measuring results and big data learning management systems; real-time learning; and geo location [51], [52]. With the proliferation of social learning in the workplace, one of the most serious concerns for companies in using social media for learning is that employees might disclose company's intellectual property (IP) via social media. It could be product information or internal work processes, workflows, company-specific training, know-how, trade secrets, etc. To address this issue, when designing a personal learning environment to be used in the workplace by company/organization employees, technology developers should focus on providing appropriate tools for the employers that can help in mitigating disclosure risks for sensitive business information. Such tools could include software modules that would detect and flag risky email messages, or social media interactions containing company IP [44].
Learning industry insiders advise technology developers to focus their technology solutions in the following strategic areas [53]: • Better measuring improvement in learning outcomes • Improving learner engagement • Delivering personalized, competency-based learning pathways • Having a mobile strategy • Supporting badges & micro-credentials 3 Learning and performance support program

Program Description
The National Research Council of Canada (NRC)'s Learning and Performance Support (LPSS) program implements adaptive and personalization strategies and develops software components for learning, training, performance support and enterprise workforce optimization. These technologies are designed to benefit NRC clients and their users by: facilitating lifelong learning, reducing learning and training costs, reducing demands on physical infrastructure, enabling streamlined and rapid skill development, reducing time to competency, supporting informal, personal and personalized learning, increasing learner engagement, optimizing sustainable workforces, and increasing operational performance and productivity.
The LPSS program is developing a learning and performance support suite of tools that will maximize a users' potential by enabling them to manage and earn competencies and achievements by matching their skills and expertise to stated customer or employer needs. The tools will help to understand training needs by automatically collecting and analyzing learning and performance reports to show gaps between existing competencies and learner or employer needs. The goal is to improve efficiency of training by using learning records and performance analytics to recommend the most useful learning services and resources specific to workplace environments and competency profiles.
The LPSS technologies have the potential to aid in lowering the cost of learning by enabling access to a wide range of learning services and resources from multiple providers within the context of relevant multiple workplace environments and productivity tools. LPSS tools originated as a web-based prototype open to the public at lpss.me that offered personal learning. The prototype lpss.me was active from Fall 2014 to Fall 2016. Currently it is being redesigned as a set of tools to address the needs of NRC clients.

Competency Handling within LPSS
The initial competency management functionality developed within the lpss.me prototype was based on a set of preloaded competencies and competency profiles. The users, through "Browse competency" functionality, could choose the competency from the preloaded sets, or could define their own competencies and skills they want to achieve such as "carpentry" or "creative writing". After choosing or defining a competency within the "My competency profile" functionality, the user would receive recommendations for learning resources for a chosen competency (or a set of competencies). The user also had the option to self-assess the level of skills acquired.
The evaluation of LPSS functionality, including competency management, was conducted in 2016. Next section of the paper describes results of user surveys and the usability study conducted during the period of 2014-2016.

4
User feedback on LPSS competency functionality

Online Survey
An invitation to participate in the online survey was sent to 299 users who were registered with the lpss.me personal learning platform. Between November 2014 and October 2016, 57 users responded to the survey, a response rate of 19%. The majority of respondents were male, from 40 to 69 years old, involved in online learning and highly educated. As a result, the findings from the survey cannot be seen as representative of the general population, but are sample viewpoints from power users within the online learning community.
A total of 24 questions were asked, ranging from demographics questions (e.g. age, gender, education, and familiarity with online learning, etc.) to more specific questions about the LPSS system, including "Which of the LPSS pages or features have you looked at so far?" [54], [55]. LPSS questions that elicited responses concerning competency functionality within the system included the open ended text questions: "What would you like a learning and performance management system (like LPSS) to do for you? How would you like to use it?"

User Feedback on Competency Functionality
User feedback on LPSS functionality and on competency management and competency and learning resource matching within LPSS was elicited via an online survey of lpss.me users and through users' responses to questions regarding LPSS functionality in the course of the remote usability testing of lpss.me in 2016. Overall user feedback on LPSS functionality was positive [56] and included valuable user suggestions on improving competency functionality and social networking.
The competency management functionality within LPSS attracted significant interest, with roughly 80% of the respondents indicating familiarity with the "My competency profile" functionality and 96% indicating familiarity with the "Browse competency" functionality. When asked to give feedback on LPSS platform features, the participants provided the following feedback and suggestions for improving competency features, including: • Making competency profiles less general and more specific, tied to a particular task. • Making a better connection between competencies and resource recommendations.
• Adding social connections for people with similar competencies and providing a way to network around competencies and learn from what others interested in similar competencies are studying. • Providing the ability to browse competencies with the user's profile associated, to optimize the social network effect. • Adding social connections for people with similar competencies to provide a better way to network and learn.
Examples of lpss.me user responses related to the LPSS competency functionality are listed below: "…If I am correct, LPSS is about lifelong learning through the perspective of your competences. The idea and its features are interesting, but you can see the realization is too much inspired by research and less through how people actually want to experience stuff online. For example, I would suggest that at the start of the LPSS, LPSS already would know what kind of competences I might be looking for based on LinkedIn or something. It should give suggestions. Also, learning -for me at leastshould be much more visual. Probably this is on the roadmap, but I would like to stress the importance of visual stuff, also to make competences more readable…" "…Being linked into a group of other users who share common interests / competencies would help…" "I like the idea of having competencies listed that I want to achieve as part of a personal learning and development plan. Then to point to resources that will help to achieve competencies. I would like more detail in the competencies section. I could see naming a competency area like Professionalism ... and then have several more detailed competency statements ... and then have some way to measure progress toward achieving each competency statement".
Updates to the LPSS competency management functionality have since been implemented based on the user feedback received so far.

Way Forward
Based on the results of the literature review and the user feedback on desirable LPSS functionality, future LPSS development will focus on enhanced competency functionality within LPSS, greater personalization for the user and a better user interface created through the use of multimedia. The development of future competency functionality will focus on providing the ability to showcase the competencies, enabling social networking for people with similar competencies and including the assessment of both formal and informal learning.
Current redesign of the lpss.me has resulted in the development of the LPSS showcase platform. The platform provides a site where the benefits of individual LPSS research tools can be demonstrated to clients and potential users; it also allows for tool integration to create broader service offerings. The competency functionality will be further developed within the LPSS showcase platform.
Competency assessment in most cases is a complex process that involves many actors and software; sometimes it might be too complex to automate. LPSS can provide functionality to access the learning records and activities related to a particular competency for actors such as experts, instructors, and for learning and performance analytics. We also plan to focus on tools for assessing both formal and informal learning. LPSS technology components currently in development for competency management and competency-based training include competency key phrase extraction, matching courses to competencies, characterizing informal learning and improved recommender technologies to recommend learning resources based on existing and desired competencies.

Conclusions
This paper reports on the findings of the literature review on the current trends within competency-based education, training and management, as well as the challenges with competency management framework implementation in Canada. Results from user surveys and usability study for the lpss.me learning and performance management platform, related to the competency management functionality, are presented. We found that in Canada competencies are most commonly used within the medical profession, including medical education, professional training and accreditation and in public service.
One major concern with competency framework implementation is that frequently competencies are applied in a varied and un-uniform manner, and they can be assessed in sometimes unreliable ways. In addition, competencies might be too rigid or inflexible for some organizations, too focused on the cultural contexts of a specific country and have limited international transferability. Contextual factors also limit competencies transferability -for example, in the area referred to as public health human resources, the needs of each community are dependent upon the context, the place, location and its particular needs.
Feedback from LPSS users showed that the users want better resource recommendations, improved social connectivity with others with similar competencies and the ability to showcase their competencies. Future LPSS development of competency functionality will focus on providing capability to capture both formal and informal learning and will target competency functionalities that address LPSS users' concerns such as the lack of ability to showcase competencies, and strengthening the connection between competencies, job opportunities, and social networking opportunities.