Data Driven Education in Personal Learning Environments – What About Learning beyond the Institution?

Learning happens not only in the traditional educational institutions but beyond them, and it is not limited to an established academic time period. We study anywhere, anytime; many different and tools and services that facilitate learning are not confined to a specific environment or platform. Such set of tools and services used for educational proposes set up what can be understood as Personal Learning Environments. While knowing what happens in Personal Learning Environments is highly interesting, it is not an easy task due to the heterogeneity of the different tools, the difficulty to access learning evidence, ethical aspects or interoperability problems, among others. This study explores the main issues that need to be addressed to analyze what is occurring in a Personal Learning Environments and how to represent the information obtained from such environments.


Introduction
Educational processes have a central role for the development of society. Education allows individuals to grow, progress and increase their performance in any area. Educational institutions (school, high school, university, etc.) are the first to come to mind when talking about teaching and learning, but… do we learn only in these specific contexts? The answer for this question is a categorical 'No, we do not'. People learn along their lives anywhere and anytime: when talking to friends, playing sports, browsing the Internet, working, etc. In other words, education is not restricted to a specific context or environment [1][2][3]. Therefore, in order to have a clear vision of what a person is learning, or has learnt, it is necessary to consider what happens beyond the institution, also known as informal or non-formal learning [4,5].
So, why is it relevant to know about informal learning? First, if educational institutions knew more about their students it would possible to define personalized learning pathways, adapted to the student's real educational needs; second, knowledge about employees' informal and non-formal learning activities would help companies define optimized learning programs and hire or promote people based on their actual skills; and third, improving the visibility of individual's competences and skills could lead to new professional and personal opportunities [6,7].
Thus, making visible what happens in informal and non-formal learning contexts is a critical albeit very complex task [2,8,9]. It requires to address several problems: • The heterogeneity of the tools that individuals use to learn outside the educational institution • The awareness problem, which refers to an individual's lack of awareness about the extent to which he or she has developed a competence in an informal environment • To achieve interoperability between the traditional educational contexts and the external tools and platforms a person could use to learn • Ethical and privacy issues, such as a person deciding not to make public a competence or a learning evidence about something that happens in his or her personal context • How to adequately deal with the exchange, management and representation of learning evidence.
In order to address the first issue, it is necessary to explore the environment where the non-formal and informal learning activities take place. That is, the learning tools and services that the individual uses to learn beyond the educational institution, known as Personal Learning Environment (PLE). PLEs facilitate users' learning processes by allowing them to use the tools they want, and when they prefer, without feeling bound to a specific institutional context or learning period [10], such as in traditional Learning Management Systems (LMS). With PLEs, self-regulation comes into place: learners become responsible of their own learning because they can decide what tools to use and solve their specific problems, and thus they are not only consumers of learning but also providers [10,11]. However, the introduction of a PLE does not necessarily involve the demise of LMS [10] or other institutional tools. The likely coexistence of LMS and PLEs entails the need for the interoperability between both [1]. The problem then becomes how to connect these environments, and how to deal with the heterogeneity of tools. These problems are addressed in the literature by applying e-learning specifications that will be described in the next sections [3,[12][13][14].
Addressing the second problem is more difficult. An individual may learn in different situations, and not always in a conscious way. In order to gain awareness, the individual should first reflect about what he or she has done, or has not, and then find a way to classify and publish the evidence about the knowledge gained. In these cases, it is necessary to teach users how to discover what they have learnt, and to give them tools to manage and share such knowledge. Several projects have dealt with this problem, such as TRAILER [9,15], IBAK [16] or Open Badges [17].
The problem of interoperability, is closely related to the first issue. There is a wide range of tools a person can use to learn in their PLE, and it is necessary to allow the exchange of learning evidence between such tools and other platforms (LMS, Enterprise Resource Planning Systems or ERPs, ePortfolios, etc.). By facilitating that exchange, what happens in the informal environment may be taken into consideration in other contexts, but an effective implementation of this exchange requires a thoughtful consideration of the data to be exchanged, the exchange format, the security and integrity of the exchange process, etc.
The fourth issue is related with ethics and privacy concerns; individuals need to grant access to companies and educational institutions about their personal learning evidence in order to make visible what they learn, and some people might not want to give their consent. Therefore, it is necessary to develop tools and methodologies that ensure learners that the institutions are only going to have access to the data they decide to publish or share.
The last issue relates to how to deal with non-formal learning evidence. While institutions would collect a great amount of data, the question then is how to extract useful and actionable information from them, which requires the use of learning analytics tools and methods. Learning analytics refers to "the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs" [18]. With the information obtained by using learning analytics, institutions and instructors may plan interventions and make the necessary changes to help students, redesign courses, adapt learning content and methods, etc. [19].
We acknowledge the difficulty in addressing the above-mentioned issues. The heterogeneity of tools is tightly linked to the nature of the PLE, and learning awareness requires not only the application of technological solutions, but also to implement effective teaching initiatives. Therefore, this study proposes an open and flexible approach based on the use of service framework and interoperability specifications to facilitate the integration of institutional and personal contexts, as well as the application of learning analytics solutions to provide useful representations of the information about the learning processes. The proposal is then laid out as a draft methodology about how to assess the learning happening in PLEs.
The rest of the study is structured as follows. Section 2 describes the research context and presents the different elements involved in the proposal; Section 3 discusses how a solution could be implemented; Section 4 presents a methodology to assess informal learning activities by applying learning analytics in PLEs; finally, Section 5 summarizes the main ideas and conclusions of the study.

Research Context
In order to better understand our approach, it is necessary to explore the different components involved. That is, the institutional systems (mainly LMS and ERPs), the PLEs where the informal and non-formal learning occurs, the interoperability specifications (as a way to facilitate the interaction of the different components), Service Oriented Architectures (a flexible solution to structure the interaction between these different concepts) and learning analytics tools (as way to analyze the information collected from the PLEs).

Institutional environments
We mentioned before that making the activities carried out in Personal Learning Environments visible may be useful both for individuals and institutions. Regarding the latter, it is possible to differentiate between educational institutions and companies. Educational institutions usually have an LMS that supports online, blended and face-to-face classes, and then other tools such as ERPs used for administrative and academic management purposes. Companies mainly rely on the use of ERPs for their day-to-day operation and employee management, and they may also use some kind of LMS for training purposes. The main difference between both is the relative importance of both systems in both types of organizations, with LMS having a more relevant role in educational institutions and ERPs having a capital relevance in companies.
LMS are systems that [20]: • Fulfil institutional learning management requirements • Provide teachers and academic staff with tools for the management of courses, students, resources, activities, etc. • Create specific areas for students in which they may perform their academic activities, complement their lectures and (to a greater or lesser extent) collaborate with other students and teachers. These systems focus on the course and provide users with tools that support and extend the traditional concept of classroom. Some of the most common LMS are Moodle, Sakai, Blackboard, Desire2Learn or Canvas.
ERPs are integrated software solutions that cover a company-wide range of business processes and enable managers to have a holistic view of the company. The idea of ERP could be summarized in: one database, one application, and a unified interface across the entire enterprise [21]. ERPs may optionally include an LMS or other tools to collect information about learning evidence. The best-known ERPs in the market include the systems developed by SAP, SAGE, Microsoft and ORACLE.

Personal learning environments
PLEs are not just a set of tools, but also incorporate contexts and interactions. There are different possible implementations of a PLE. The first one contemplates the use of any technology may use to learn outside the institution, such as mobile phones, tablets, computers, smartwatches, video-game consoles, etc. In these cases, the device itself-and the tools used in the device-act as a PLE. The main problem of this implementation becomes how to collect the information about what the user has done, and which part of it is related to learning and can be useful for the organization. The solution implies filtering and recording all the interactions, and then send them to the organization for processing, for example using an access point that launches all the applications [22][23][24].
Other possible PLE implementation is to employ a web launcher that facilitates access to other applications in the form of apps or widgets [2,3,25]. This can be done in many different ways: through web portals such as iGoogle (http://www.google.com/ig), MyYahoo (http://my.yahoo.com), or Elgg (http://www.elgg.com); using widgets, portlets and web containers, such as Apache Wookie (Incubator), Netvibes (http://www.netvibes.com) or Liferay (http://www.liferay.com); as a personal part of the learning platform in an LMS; as social apps in social media networking sites, such as Facebook (http://facebook.com); or integrated as part of an ePortfolio like Mahara (http://www.mahara.org) or MyStuff (http://mystuff.anniesland.ac.uk/). The problem of these possible implementations is that not all the individuals use the same tools to learn, and each tool should be adapted to the web container. Its main advantage is that it is easy to collect the activity of users.
A final option is to have users recognize the informal activities carried out outside the institution, making them responsible of their storage and management them through a digital portfolio, as shown in the TRAILER project [9,15].

Interoperability specifications
It is critical for people in charge of the organization (teachers, managers) to have access to what individuals are doing in their PLE. This requires the adaptation of PLEs and institutional environments by applying interoperability techniques, as suggested in prior literature [11,[26][27][28] and implemented only in a few instances of PLE [2,3]. There are different ways to achieve interoperability between learning contexts, mostly based in the use of web services and interoperability specifications [14]. The most relevant interoperability specifications include the following: • IMS Learning Tools Interoperability (LTIhttps://www.imsglobal.org/activity/ learning-tools-interoperability) and Basic Learning Tools Interoperability (BLTI). IMS LTI is one of the most popular specifications because it facilitates a real and full integration between tools and learning platforms. However, it does not count with wide support among LMS or other tools due to its difficult implementation. A light version of the specification, Basic LTI, aims to overcome this problem. BLTI is supported by the most relevant LMS and also by some ERPs [29], facilitating the creation of an external tool instance inside the learning platform, launching it and providing a single-sign-on access. A major problem of BLTI is that it does not provide real integration, but only authentication, and therefore there is no exchange of information about the activity performed on the tool with the LMS or ERP (i.e. the grade achieved in an activity, or users' activity logs). This is mostly solved in versions 1.2 and 1.3 of LTI. • xAPI (eXperience API, formerly known as Tin Can APIhttps://xapi.com/over view/) is a new e-learning specification designed to support the learning community in standardizing and collecting both formal and informal distributed learning activities. The xAPI specification describes packaging and transmission of learner ac-tions called 'Activity Statements' between any tool and a learning record store (LRS), the database model that validates and stores activity statements [30]. It is used for the recognition of informal learning activities [31,32] and it is supported by most LMS. • Caliper Analytics (http://www.imsglobal.org/activity/caliper). This specification enables institutions to collect learning data from digital resources to better understand and visualize learning activity, product usage data, and present this information to students, instructors, and advisors. Each learning activity has one or several associated metric profiles. A metric profile defines the information model that shapes the types of events emitted by the learning activity. It also provides a semantic for later analysis [33]. It is specially oriented to the connection of different learning components [34].
These specifications facilitate the integration of activities carried out in the PLE, but the application requires an adaptation both in the LMS and in the tool, which involves extra development and customization work.

Service oriented architectures
As we have shown in the previous sections, it is evident that the effective exchange of information between the PLE and the LMS/ERP requires an open solution, not tied to a specific technology or tool, easy to implement and as transparent as possible for the user. Service-based solutions comply with all these requirements. Service Oriented Architectures (SOAs) are "a software engineering approach that provides a separation between the interface of a service, and its underlying implementation. For consumer applications of services, it does not matter how services are implemented, how contents are stored and how they are structured. In the SOA approach consumer applications can interoperate across the widest set of service providers (implementations), and providers can easily be swapped on-the-fly without modification to application code" [35].
In eLearning research field, SOAs have been used to adapt LMS to emerging technologies or frameworks, and to integrate different specifications; in other words, to enable opening of the learning platforms. Several LMS, such as Moodle [36], Blackboard [13] or Sakai [37] include services layers for different uses, such as the adaptation of LMS to mobile devices [38], the use of information from the LMS in external platforms [39] or the application of visual analytics tools to facilitate making decisions based on learning evidence gathered from the LMS [40]. Approaches to define PLEs based on service approaches include MUPPLE [28], PLEF [41], MeMeTeKa [42], Peret, Leroy y Leprêtre's work [43], cloud services [44], and other related to the communication between PLEs and LMS [9,14,22,25,45].

Learning analytics tools and methods
When students are completing an activity in the LMS or another tool, it is possible to record what they have done for later analysis. However, this information tends to be stored as raw data, which is difficult to analyze and manage for decision-making. In order to explore the information and extract actionable knowledge out of it, disciplines such as educational data mining [46], academic analytics [47] or learning analytics have emerged in the last decade. These disciplines offer different but convergent perspectives, methodologies, techniques and tools aiming to facilitate this transformation process [48]. There is currently a wide choice of tools that facilitate educational data extraction and analysis for learning analytics purposes. A first broad categorization of these tools would include [49]: • Cross-platform and platform-specific general-purpose dashboards: Dashboards provide information about platform activity of the different learning agents-mainly, students and teachers-, generally in a visual and condensed form. Dashboards can be applied to different platforms [50,51], such as Google Analytics, or to a specific one [52] (e.g. Moodle Dashboard). • Ad hoc tools: The design and implementation of ad hoc tools seek to perform tracking and analysis of very specific types of information adapted to very specific contexts. For instance, a specific tool to assess the acquisition of the teamwork competence from Moodle logs [53]. • Learning analytics tools for analysis of specific issues: These tools aim to provide information, and usually have very specific types of representation. It is also very common that they offer cross-platform capabilities. For example, tools for social network analysis such as GraphFES [54]. • Learning analytics frameworks and tools: The design of learning analytics frameworks is oriented toward standardization of learning ontologies and their implementation in different systems. They also pursue the exploration of student behaviors in different educational contexts and offer the user customizable visual representations of the information. Prominent examples include VeLA (Visual eLearning Analytics) [55] or GISMO [56].
The tool mentioned above may be included into an LMS (e.g. Moodle Dashboard, GISMO) or ERP, or may be used as external tools (e.g. Google Analytics, GraphFES). The main problem in the case of informal learning contexts is that it is necessary to first define what type of learning activity requires tracking, what information the tool is going to analyze and return, and how will the information be presented to the user. Because users may freely choose the tools of their PLE, the immediate consequence is a great variety of tools and information. The next sections present a possible implementation approach and a methodology to facilitate the assessment.

Implementation Approach
Considering the discussion in the previous section, our implementation to collect and present learning evidence from the PLE requires the design of a Service Oriented Architecture, implemented through interoperability specifications, and able to connect the PLE and the organizational system, as shown in Figure 1. Such an implementation requires the following: • Definition of a PLE model: Given the different possible implementations of a PLE, it is necessary to make a decision about the most adequate way to solve the problems and the devices and tools that need to be included. It is mandatory that the PLE is flexible enough to extend its functionality by using services, and that it facilitates the implementation of interoperability specifications. study the evidence generated in the PLE, how they are, or can be, stored and how the data will be exchanged between the different components. Probably an adaptation will be necessary to store, exchange and analyze the data, due to the heterogeneity of the tools involved in the learning and assessing process. • Design of the Software Oriented Architecture: Once the PLE, the institutional environment and the data model are defined, it is necessary to study the different extension possibilities of the different environments, and to define an architecture based on a service choreography and a set of service contracts. • Implementation of the SOA using interoperability specifications: Once the design of the SOA and development of the services are completed, it is time to make a decision about the specification or specifications necessary to implement the architecture. Most likely, it will be necessary to include connectors both in the institutional and the personal ends. The connectors will act as consumer and providers of services.
• Select and adapt learning analytics tools: In the organization end, it is necessary to analyze and carefully select the most appropriate tools to be used, develop new ones if needed, and/or adapt the existing ones to analyze the learning evidence. • Testing and refining: Once a first version of the approach is implemented, effective deployment requires testing the system, adapting the services and tools, and solving the different problems that may arise.
Beyond the definition of the environment, another important issue is how to assess what happens in the personal end-e.g. the PLE-. The next section proposes a methodology to assess the effective implementation of the proposed approach.

Methodology for assessment of PLEs
From the above, this section lays out a methodology, or a set of steps, that are necessary in order to facilitate assessment of learning happening in PLEs. The layout of the proposed assessment has six stages: • Stage 1 -Definition of the informal learning activity. The initial stage involves thinking about the data of interest about the individuals' behavior and interactions with their PLE. This step requires instructor's planning and reflection about the learning process because, if the activity is defined formally, the learning in the PLE might be constraint. However, knowledge about how the individual is interacting helps making the student's learning evidence visible. For example, if the student uses YouTube to watch educational videos, it may be of interest to record the type of videos he or she is watching, or the time the student spends watching them. • Stage 2 -Environmental set-up. This stage requires to explore the tools that the user is going to use to carry out the activity, while considering the type of institutional platform he or she is using and the tools or platforms employed as PLE. The best case scenario would be a central tool, a PLE platform and the organization's platform with support for interoperability specifications. However, most commonly the fact is that a there is a system that is difficult to modify. In this case, the organization's platform and the tools used in the PLE should be adapted to support interoperability specifications. The choice of the interoperability specification will depend on whether they are supported or not. For example, it would be possible to use Moodle with an xAPI plug-in, and then integrate users' activity in YouTube and in their personal blogs. • Stage 3 -Definition of the exchanged information. This stage occurs concurrently with Stage 4, in a feedback loop. During this stage, it is necessary to define exactly what information must be exchanged between the organization's system and the PLE. The information must comprise the configuration settings for the activity or users' outcomes and all the information defined by the learning analytics policy. For example, it would be necessary to exchange user's identification information, and also information about the outcomes or publication date, such as the number of times that a video is played in YouTube.
• Stage 4 -Define the learning analytics policy. After defining what information is being exchanged, it is necessary to identify which indicators, beyond students' outcomes, can provide evidence of competence acquisition after completing a learning activity or session. This information needs to be included as part of the data exchange. In this stage, it is also necessary to consider how to explore the information and how to present results. In addition, the learning analytics being applied should be incorporated to the organization's system. For example, it might be interesting to record the number of times a user has watched a video, but also with whom they have shared it. This information might be used as feeding source for a general dashboard in the LMS, but also as a source of information for a tool like GraphFES to display interaction between students as a social graph. This strategy may would give visibility to who is working more intensively, and also who is sharing the knowledge with their peers. • Stage 5 -Piloting. The next stage requires testing with a sample of students, and the results of the test may leads to adjustments of the previous stages. As an example, if testing the activity with twenty students shows that five of them are using tools other than YouTube to watch videos, it might be of interest exploring whether they are achieving better results or not. The result of this assessment might involve an adaption of the previous four stages. • Stage 6 -Activity development. At this point, the activity is carried out by a larger number of students, in a real context. At his point error detection comes into place and changes in the strategy might be required. Once this stage finishes, the cycle begins again with stage 1, including the feedback with information about lessons learnt.

Conclusion
This study explores the main issues involved in the assessment of learning activities that happen beyond the institution; that is, non-formal or informal learning that occurs in individuals' personal environments. This assessment is a complex problem that requires to analyze learning activity happening in a non-standard and noncontrolled environment, involving people with different interests and motivations, educational backgrounds and technology preferences. We posit that assessing what happens in the PLE is possible, but also acknowledge that it may be a daunting task that requires changes both in the organization's system and the personal environments, and also that this is not always possible. Such an approach requires not only changes in the technologies being used, but also to deal with learner's awareness and selection of the best analytical tools by the organization. The complexity of the problem requires a flexible and generic approach, as well as a a methodology to implement solutions able to solve a wide range of different contexts, such as the one presented in this study. We also recommend the implementation of varying versions of the solution proposed in this study and a rigorous assessment of results to give further directions for future practice and to reveal further potential barriers.