Learning Management Systems: A Shift Toward Learning and Academic Analytics

This paper presents the major findings from a study conducted with six different universities in the U.S. regarding their use of the Learning Analytics (LA) capabilities available within their learning management systems (LMS). Data was collected from an online survey instrument, in-depth interviews with IT directors and academic administrators, and a case study in Embry-Riddle Aeronautical University. One observation is that universities are attempting to make better use of new analytics functions and the data stored in the university LMS in order to make more informed decisions regarding short-term and longterm goals and objectives. The new functions include analytics performed at the institutional level, college level, degreeprogram level, course level, and even course section level. Courses and degree programs as well as learning performance and objectives can be measured and analyzed using different goals, criteria, and accreditation requirements.


INTRODUCTION
Terms such as "information society" have been used to identify this era in human development, and describe how societies today create, share, and use knowledge to improve their well-being [2]. Knowledge management is now required in all industries, and education is no exception [1].
The environment of higher education institutions today is more complex and competitive than ever before. Universities are facing increasing pressures to respond in a timely and efficient manner to political and social changes at the national and global levels, albeit with fewer enrollments, declining support from the government and the enterprise society, and with growing regulatory demands for transparency and accountability [12], [29]. Universities today are more accountable; they must provide students, faculty members, accrediting bodies, governments and administrators with evidence of students' performance and learning achievements. To remain viable and competitive, Universities must also make better decisions in the course of academic administration regarding issues such as course, program and faculty performance while monitoring resource allocation and return on investment [9]. The decisions required to deal with the challenges and opportunities of the higher education environment require a constant flow of valid, timely, and relevant information that institutions can use in an efficient and effective manner. In many cases this data is already available in one form or shape in one or more of the different information systems used by the institution, but often that data is either overlooked, underused, or otherwise not properly utilized [9], [22], and [20].
This study sheds some light on the potential utilization of one of those underused key higher education systems, namely the Learning Management System or LMS, to support universities' decision-making processes. The significance of the LMS is that it is the primary system that contains information related to students' performance against specific rubrics, outcomes, and other metrics, and as such, it contains a wealth of information that can help universities make more-informed decisions regarding their performance [28].
Over the past decade, many universities have purchased or developed LMSs for managing curriculum, training materials, and for use as evaluation tools. According to research from Bersin by Delloite, global spending on LMSs has increased by a 52 percent since 2012, with a 21 percent increase in 2014 alone, totaling more than $2.5 billion annually [27]. Nine in ten institutions in the US use one of the top five LMS vendors. Blackboard has the largest market share with 42% [16]. A key reason for this growth is the evolution of LMSs from learning environment software suites into tools that universities use to develop intelligent electronic coursework and to deliver that coursework with wide-reach and flexibility [22], [20]. According to a study by [11], 15% of U.S. institutions are planning to replace their LMSs within the next three years, and they are looking for enhanced features that include analytics, especially to support outcomes assessment and course and program reviews [10].
The use of analytics in higher education is a relatively new area of practice and research. Learning Analytics (LA) apply the model of analytics to "the specific goal of improving learning outcomes. LAs are used to collect and examine the records of students, interactions with various computer systems and to look for correlations between those activities and learning outcomes" [11]. The type of data gathered varies by institution and by application, but in general it includes information about the results of assessments from student exercises and activities. The types of analyses performed vary, but one approach involves the evaluation of historical student data to create predictive models of successful and at-risk students.
Reports can take various forms, but most feature data visualizations designed to facilitate quick understanding of which students are likely to succeed [11]. The effectiveness of LAs depends heavily on the frequency and nature of faculty and student use. In addition, objective interpretation of data is critical, as patterns revealed by the PAPER LEARNING MANAGEMENT SYSTEMS: A SHIFT TOWARD LEARNING AND ACADEMIC ANALYTICS data for some students may not be applicable to other students [11].
Given the importance of higher education in today's information society and knowledge economy, and the role LMSs can play in higher education decision making, the aim of this study is to provide an overview of the current status of LAs available in LMSs, paying particular attention to how they can be used to provide decision-makers with information to evaluate university performance versus institutional needs and requirements. This study is significant to higher education institutions, as well as to other stakeholders involved in the hierarchy of higher education, including students, educators, researchers, institutions, and government agencies [23].

II. RELATED WORK
Business Intelligence (BI) describes the "technologies, systems, practices, methodologies, and applications used to analyze large amounts of diverse business data to help organizations convert large amounts of raw data into meaningful information to support sound and timely decision-making" [6]; [33]. Analytics is a component of business intelligence that provides techniques to recognize trends from patterns in data and to make decisions based on those trends for the overall advantage of the organization [34]. Learning Analytics (LA) is the "measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs" [17]. And while LA is more concerned with the learning process, Academic analytics (AA), in contrast, is "the application of education analytics for better decision making at institutional, regional, and international levels'' [17] (see Table 1). More broadly, LA and AA tools are used to improve universities' processes and workflows, measure academic and institutional data, and improve organizational effectiveness [15].  [17] The Horizon Reports of 2012 [13] predict Learning Analytics to be in "mainstream use" in two to three years. Research in the area of LA has seen significant growth since 2010 in three distinct areas; the first is the development of LA, its concepts, implications and impact on higher education; the second is the technical aspects of LA; and the third is research on the use of LA in social learning [26]. The acceleration of LA began as applications emerged in the form of learning performance solutions, like SunGard and Desire2Learn (see Table 2), and learning management systems where data specific to the school or university could be collected [3]. A key element of LA called visualization, is used in a way in which analysis results are displayed so they are easily understood by decision makers [4]. With LA, stakeholders will have easy, visualized access to massive amounts of digital data left behind from learners about learning experiences in various systems in the same way that the business intelligence market analyzes consumer data today [14]. Organizations from a wide range of industries have reported improvement to business processes and decision-making by implementing analytics [6]; [30].  [11] The Value of LA is that it can transform all aspects of the institution, including administration, research, teaching and learning, and support resources. With LA, universities can improve decision making and resource allocation, they can identify at-risk learners and areas of concern, they can get a better insight into their strengths and weaknesses, they can drill down on causes of complex challenges, and they can create and try different academic models. LA can help to discover and reveal information and make connections at a course or program level that can in turn be used to make predictive models that can be used at an institutional, regional, and national/international level [18]. Optimization of learning requires not only retrieval of useful information and knowledge about learning processes and relationships between learning agents, but also the transformation of data gathered into actionable information. The ultimate objective of LA must be to enable data-driven educational decision making at all levels [31].
This becomes especially important in course settings where numbers of enrollments are high and instructors need help in monitoring activities and student performance [25]. LA can help higher education institutions by gathering data from various sources to make decisions about academic progress, predictions about future performance, and to recognize potential issues [13]. While there are numerous datasets of learner information available for the field of education, there is still a need for improvement in the process of measuring, collecting, analyzing, reporting, and sharing data across institutions themselves [32]. One of the most significant challenges facing higher education today is the lack of knowledge about the ways that PAPER LEARNING MANAGEMENT SYSTEMS: A SHIFT TOWARD LEARNING AND ACADEMIC ANALYTICS students interact with learning materials. To that end, the study of [25] was important in noting the most important information requirements from instructors. Among them were, the students' overall success rate, the mastery level of concepts, skills, methods and competencies as well as the most frequently diagnosed mistakes [25].
LMSs can be instrumental in helping with LA. LMSs have been adopted as LA tools because the data captured is structured and reflects the learners' interaction within a system [17]. "An LMS is defined as software that provides an integrated suite of online resources and communication capabilities in support of traditional course delivery and can also serve as a platform for fully online courses. A typical LMS provides a range of learner activity options, such as forums, databases, and wikis; facilitates student assignments and quizzes; and enables monitoring of student engagement and reporting of grades. Many LMS implementations are integrated with student information systems" [16]. LMSs manage, track and report on the interaction of the learner, the content, and the instructor. LMSs track learner progress, record test scores, and indicate course completions, and allow instructor trainers to assess the performance of their learners [8]. The systems centralize course preparation; educational content and resources; the delivery and tracking of student activities, such as discussion and collaboration; the administration of assessment activities; and the accumulation and presentation of grades and assessments. New functions and features also provide information that can be analyzed to detect patterns that might suggest how students can be better supported [35]. Most LMSs are Web-based, built using a variety of development platforms, such as Java/J2EE, Microsoft .NET or PHP. The common idea behind an LMS is that learning is organized and managed within an integrated system [8].
The first LMS was developed and used in 1924 to administer multiple choice questions. LMSs started taking an entirely new look with the introduction of personal computers and the birth of internet. LMSs were originally developed as back office applications used to schedule and manage formal training, mainly using Mainframes. In the late 80s a new generation of LMSs was introduced to manage not only formal training, but also e-learning [5]. Some of the well-known LMSs include Blackboard, De-sire2Learn, Canvas, Moodle, Pearson LearningStudio, and Sakai. LMSs today have basic built-in analytics capabilities, such as early alerts, content aggregation and analysis and progress tracking. [21].
A number of research works analyzing data stored by LMS have been published (e.g. [24], [36]). In general, they investigate five main areas: the learning experience, users' interaction, clustering students, and understanding and predicting performance. These research efforts are important as they provide clues into which routine analyses are useful, and which techniques are mature enough for use in academic decision making.
Many universities don't take full advantage of LMS capabilities because of the complexities of the data and systems integration process [7]. Also, higher education data has its own unique characteristics which must be considered. For example, educational data is text heavy; many educational goals are difficult to quantify or measure (e.g. improving the learning process); and, the data analysis process involves multiple dimensions such as students, instructors, courses, course sections, grades, degreeprograms, and the like [28].

III. METHOLDOOGY
While the importance of Learning Analytics is recognized, there is very little data and research available about its effectiveness within the higher education and academic administration settings. The objective of this study is to offer an overview of key LA initiatives available in LMSs, and how they could be transformed into actionable educational policy [19]. This study was conducted over a two year period using three research methodologies: survey, in-depth semi-structured interviews, and a case study. The survey mode of inquiry was employed to obtain data beyond the immediate environment of the researcher to provide insight into how other universities utilized their LAs. Five in-depth, semi-structured interviews were held with IT directors, and another five interviews were held with academic administrators to gain deeper understanding of the usefulness of the LA functions selected and a case study was conducted at Embry-Riddle Aeronautical University to validate and triangulate the results of the survey and the interviews.
The online survey was sent to more than thirty six IT Directors and CIOs in universities in the US. Twenty seven responses were received. The survey was comprised of twenty five questions aiming to understand how LMSs were being utilized to generate LAs within those different higher education environments. The questions were concerned with LMS functions utilized, extent of use, range of services, structure and setup, etc. Ten semi-structured interviews were held with IT directors and academic chairs and deans, five interviews for each group. The interviews were designed to explore links between the analytics functions in the LMSs used and the type, relevance, usefulness, and timeliness of the functions relative to the academic decision making process. The Embry-Riddle Aeronautical University case study examined the use of LAs and AAs and their impact on decision-making on four undergraduate programs and three graduate programs.
The results of the extended study provide a number of key observations. Educational institutions can utilize the findings of this study to guide data collection, and analysis and measurement of courses and degree program metrics.
The study confirmed students' performance can be measured, analyzed and benchmarked against a set of goals and objectives. Additionally, curriculum coverage and scope can be easily examined and analyzed. Most importantly, higher education institutions can automate the collection of student learning evidence (artifacts) and significantly improve their outcome assessment planning and review process in a more efficient and cost-effective manner. From a cost-effectiveness and resource perspective, the automation of data collection is significantly less expensive than the cost of collecting evidence manually, and the results of automated analytics can be of much greater value in terms of quantity and quality. Lastly, from a strategic perspective, educational institutions need the assist of data analytics to evaluate faculty performance, course performance, students' performance, and academic process efficiency. The demands of increased accountability combined with hyper competition between educational institutions for enrollments as well as more comprehen-PAPER LEARNING MANAGEMENT SYSTEMS: A SHIFT TOWARD LEARNING AND ACADEMIC ANALYTICS sive requirements for certifications and accreditations all lead to the conclusion that it is time to start collecting and analyzing data about institutional performance with the goal of improving current decision making processes.

IV. DISCUSSION
In this section, we introduce the most common LMS-LA functions used by the examined institutions, and their possible use in improving the teaching and learning, and academic administration processes

A. Curriculum Coverage and Mapping
The course coverage and curriculum mapping functions allow users to create tables, maps, and show associations and relationships. Users can associate curriculum goals and other metrics in order to measure, collect, report, and analyze course and program coverage against those goals and metrics. Goals and metrics can be classified at different hierarchical or non-hierarchical levels (e.g. course, program, institutional, accrediting body, industry standards, etc.).
Once those associations are established, users are able to collect, report, measure, and analyze the appropriateness of curriculum coverage against any goal or metric. Those interactive mapping functions eliminate the need to manually maintain such information outside the digital systems and allow users to easily validate curriculum changes and understand the impact of different curriculum models in a more efficient and real time manner, even before they commit to those changes The data supported findings of this study reveal that the use of LMSs' LAs have significantly helped academic chairs and administrators track and improve their course and program coverage, associations, and metrics. Academic administrators that maintained those processes manually were often faced with inconsistencies, redundancies, data errors, labor-intensive process, and higher maintenance cost. In several cases the use of LMS LAs revealed an over allocation, under allocation, or misalignment of outcomes, activities, or competencies within a course, program, college, or even a university. The use of LMS-LAs functions were extremely useful in realigning the curriculum to reflect the intended emphasis of the course and the program. Respondents also testified to the usefulness of LMS-LAs functions in helping prepare for accreditation self-studies, annual assessment planning, and program reviews. LMS-LAs enabled users to analyze a variety of curriculum maps used to illustrate relationships between courses and program outcomes or courses and skill acquisition and/or explore other useful relationships that can help make more sense of the degree program structure and assist with its assessment and review. Finally, academic administrators and instructors were able to create better relationships and alignments between academic content and industry accreditation and certification standards.

B. Goal Performance
This LMS-LA function enables academic administrators and instructors to create associations between gradable assignments and specific learning outcomes or goals, and then measure, collect, report, and analyze students' performance data against those specific outcomes and goals. Outcomes can be created at different levels (e.g. course, program, institution, universal, etc.). Evaluations can be reported against the original score, and can also be sent for a secondary evaluation. Outcome evaluation can be reported and analyzed at the course section level, course level, program level, or even higher. This function allows an institution to examine how its students are performing in areas such as critical thinking, case study analysis, research activities, etc.
As one of the interviewees noted "The ability to look at students' performance at any point of time using a multidimensional lens created confidence in the integrity and validity of the data examined for course and program assessment". Another interviewee stated "our ability to automatically gather a large volume of data on actual students work and performance, not extracurricular activities is unprecedented". Academic administrators are able to easily identify and isolate the weakest link.
By analyzing performance using the multi-dimensions of students, instructors, assignments, course sections, outcomes, terms, etc., patterns and consistencies become more apparent. In a sense these techniques can be used to fault isolate issues related to academic learning and course administration. For example, respondents reported that they frequently analyze the same goal/outcome against several course sections to quickly spot inconsistencies and weak performers, and they use the drill down function to look for possible triggers across those sections. And while several respondents reported the use of this function at the course-level, many reported the use of this function at the program level, especially for outcomes and goals shared by program courses. Users are also able to target/harvest course gradable assignments related to an outcome or set of outcomes and send it for secondary evaluation using an assigned rubric.

C. Interactive Rubrics
Another attractive LMS-LA function is interactive rubrics. Interactive rubrics allow instructors and academic administrators to create custom and reusable rubrics that can be assigned at the course-section level, course level, program-level, or universally. Rubrics can be created using different types or forms (point, range, percentage, etc.), and they can reflect multiple criteria. Rubric results can be displayed in a matrix of rows and columns. The rows correspond to the various criteria of an assignment, and the columns correspond to the level of achievement expressed for each criterion. A description and point value for each cell in the rubric defines the evaluation and score of an assignment. Interactive rubrics can be associated with any type of gradable assignment. As instructors grade students' assignments, they must enter a grade for each criterion within the rubric. This is not only useful because instructors' feedback is more detailed and specific, but also because academic administrators can aggregate and analyze students' performance against each rubric and each criterion within a rubric at the course, program, or institutional level. Interviewees reported that while students and instructors appreciated the mutual understanding of assignments expectations and grading criteria, instructors tended to spend more time on grading. Academic administrators at the other end, reported that the use of interactive rubrics helped students gain better understanding of how the assignment is evaluated and learn from their mistakes. It also forced instructors to provide more specific feedback and relate it to the grade. Academic PAPER LEARNING MANAGEMENT SYSTEMS: A SHIFT TOWARD LEARNING AND ACADEMIC ANALYTICS administrators were able to examine students' performance against rubrics and criteria at the different academic levels, and analyze that data accordingly to detect patterns and behaviors.

D. Alerts and Early Warning Systems
This function provides real-time data that can be used in academic advising and early warning systems. Further, this data can be utilized by students to learn about their performance compared to others in the course. Students can use different views allowing them to see how much time they spend on each activity compared to others in the class. Instructors can also compare and identify at-risk students not only using course grades, but also, by activities, modules, learning outcomes, rubrics, etc. Administrators can examine course design and faculty performance. At the institution level, universities may use the data from the LMS with the data of the Student Information Systems SIS to understand the big picture on issues such as students' retention, grades, etc. Early warning system rules use grades to create rules and create alerts to student performance based on those rules, as well as assignments or assessments that are not completed by the deadline. The course dashboard includes information about the course alerts and the number of warnings and the number of total rules that may trigger a warning. This function allowed instructor a quick look into each student's performance and standing as the course progresses. It also allowed academic administrators enough time to identify students at risk and provide them with the needed guidance/support.

V. CONCLUSION
Universities can begin to make more sense of the data contained within LMSs than ever before. The data gathered in this study illustrates not only the availability of new LA and AA features within LMSs today, but also the value that could be gained in teaching and learning and more-informed decision making with the use of those function. The results of this study show that academic institutions have begun to collect data, analyze and measure courses and degree programs metrics, specifically in areas such as curriculum coverage, students' performance, and alerts and early warning systems. The data collected is not only significant in terms of volume, but is also highly relevant, timely, and multi-dimensional. These advantages allow academic administrators to make better academic, financial, and strategic decisions regarding the strengths and weaknesses of courses and programs offered. It also allows them to easily analyze cause-effect scenarios.
Indeed all stakeholders in the higher education hierarchy should be able to take more complete ownership of educational processes by utilizing information about student success factors, the allocation of resources and effectiveness of teaching and institutional programs. These improvements, in turn, allow for real accountability and efficiency, more accurate measurement of the quality of learning and the raising of completion and retention rates, Performance prediction, attrition risk detection, data visualization, intelligent feedback, course recommendation, student skill estimation, behavior detection, and planning and scheduling are all resultant capabilities.