Redesigning a First Year Physiology Course using Learning Analytics to Improve Student Performance

Learning analytics (LA), a fast emerging concept in higher education, is used to understand and optimize the student learning process and the environment in which it occurs. Knowledge obtained from the LA paradigm is often utilized to construct statistical models aimed at identifying students who are at risk of failing the unit/course, and to subsequently design interventions that are targeted towards improving the course outcomes for these students. In previous studies, models were constructed using a wide variety of variables, but emerging evidence suggests that the models constructed using course-specific variables are more accurate, and provide a better understanding of the learning context. For our current study, student performance in the various course assessment tasks was used as a basis for the predictive models and future intervention design, as they are conventionally used to evaluate student learning outcomes and the degree to which the various course learning objectives are met. Further, students in our course are primarily first-year university students, who are still unfamiliar with the learning and assessment context of higher education, and this prevents them from adequately preparing for the tasks, and consequently reduces their course performance and outcome. We first constructed statistical models that would be used to identify students who are at risk of failing the course and to identify assessment tasks that students in our course find challenging, as a guide for the design of future interventional activities. Every constructed predictive model had an excellent capacity to discriminate between students who passed the course and those who failed. Analysis revealed that not only at-risk students, but the whole cohort, would benefit from interventions improving their conceptual understanding and ability to construct high-scoring answers to Short Answer Questions. Keywords—interventional learning activity, curriculum design, course outcomes, assessment, prediction 4 http://www.i-jai.org Paper—Redesigning a First Year Physiology Course using Learning Analytics to Improve Student Perf...


Introduction
Research in education sciences has demonstrated that student academic performance is influenced by a myriad of complex and interconnected factors, including those that are specific to a course taught within a tertiary institution [1]. Outcomes from a number of studies have demonstrated that student behaviors, their engagement with course content and performance are strongly influenced by how courses are designed [2,3]. The analytics paradigm used in education, termed Learning Analytics (LA), can provide an effective way to identify and subsequently address course-specific factors that correlate with students failing a course. This paradigm has been shown by some studies to be effective in improving student outcomes through interventions or changes in course design (e.g. [4,5]). Results from LA can also be used to construct predictive models that assist in the identification of students who are at risk of failing the course. Inviting these students to targeted interventions further refines the effectiveness of these activities.
Previous studies have constructed predictive models using student data, often data pertaining to factors that can be generalized across different contexts, in an attempt to identify at-risk students for interventional action (e.g. [6][7][8]). However, growing evidence suggests models constructed using contextual or course-specific factors, such as instructional design and assessments, would have a higher accuracy as student learning and associated outcomes are influenced by the context in which learning occurs [8,9]. By extension, results from these contextual models would also inform effective interventional design as they would allow researchers to develop an accurate understanding of contextual factors contributing to deficiencies in student learning and outcomes [8][9][10]. In our study, we used student performance in the various course assessment tasks to construct statistical models that identify at-risk students and design interventions for implementation in future course iterations. Student performance in assessment tasks has been shown to serve as an effective predictor of student course trajectory [10][11][12] and thus as effective targets for interventions. This is unsurprising considering that these tasks are conventionally used to evaluate student understanding of course content [13] and achievement relative to the various learning objectives of the course. However, some tasks, such as the short answer questions (SAQs) in an examination, are more difficult than others due to the challenging task requirements that students will have to meet in order to perform well [14][15][16]. This issue is exacerbated by the fact that students in our course are primarily in their first year of university study and do not have a clear understanding of these requirements and how to achieve them [17], thus hindering their capacity to prepare adequately for these tasks [18,19]. In our course, where students are exposed to a variety of assessment types, it is imperative that we identify assessment(s) that they find challenging to inform the design of interventions aimed at helping students better prepare for such tasks in the future.
A student's prior knowledge has been shown to be a good predictor of their course outcome (e.g. [12,20]). This is unsurprising as prior knowledge provides a student with the necessary foundation on which new knowledge can be built. Briefly, individuals are thought to gain knowledge on a particular topic by gradually accumulating concepts pertaining to the topic, and then making links between these concepts to develop a deeper understanding of the subject matter [21]. This is highly pertinent in physiology, in which an integrated understanding of different physiological knowledge is necessary to develop a complex understanding of the various, highly coordinated and integrated body systems that constitute this discipline [16]. It is also important to note that prior knowledge from other domains, particularly chemistry, has also been argued to be critical in understanding the various chemical processes needed to sustain life in physiological systems [22].
The current study aimed to construct predictive models at two different time points within the 13-week academic semester (the typical course length in the tertiary institution where this study was conducted), which would be used to identify those students who are at risk of failing the course in future course iterations. Furthermore, the study attempted to identify assessment task(s) that students find challenging to inform the design of future interventions, ultimately aimed at improving student course performance and outcomes, especially for those at risk of failing the course.

Predictive model construction
All data from students involved in this study (N = 876) was de-identified prior to any analysis. Statistical analysis was conducted using R v3.1.1 software [23]. Analysis was carried out as described in previous studies [24], [25]. Logistic regression was used in our study as student course outcome is binary (i.e. pass or fail) and has been shown to be capable of predicting student course outcomes with high accuracy [26]. A model that included every course assessment task and pre-university variable, with some exceptions detailed below, was constructed ("Model Full") in order to identify assessment tasks and/or any broader underlying learning issues that play a crucial role in negatively affecting student performance in the course. Two additional models were constructed to predict student course outcomes at week 5 ("Model A") and week 12 ("Model B") of the teaching semester in future course iterations. Every variable that was available at each time point was used to construct the models, with some exceptions detailed in the section below. Model parameters were estimated using maximum likelihood and ordinal variables were coded as continuous. Threshold for significance was p<0.05 for all statistical analysis conducted. The estimated coefficient for each variable in all models was tested for significance using the Wald test.
Section 1: Defining variables. The assessment tasks in this particular course are divided into four main components: Practical Core Competencies, Practical, Communication and Knowledge. The Practical Core Competencies component evaluates student mastery of four laboratory techniques. Students are given multiple attempts to demonstrate that they can perform these techniques satisfactorily and unassisted. Assessment tasks within each of the remaining components, and variables sourced from student pre-university experiences, are shown in Table 1. Broadly, the Practical component consists of four tasks (three reports, one worksheet) that are based on results collected from student-run experiments in laboratory practical sessions. The last task required students to generate a concept map that integrates concepts across different physiological systems taught in the course. Tutors assist students in all tasks and are responsible for marking and providing students with constructive feedback. The Communication component focuses on improving student skills in writing about science in a variety of genres (see [27,28] for more details). Knowledge is the most important component as the course is designed to ensure that students prioritize learning physiological concepts, so that they are adequately equipped for the higher level courses that they will study in the future. The final course grade is determined using a grading matrix approach that values student performance in all of the assessment tasks of the course to determine the final outcome [29]. Specifically, students are only assigned a specific course grade if they have achieved the minimum standard for every assessment component assigned to the grade. Assessment tasks Comm 2 to 5 were not included in the subsequent analysis, as they do not fulfil the independence of observations assumption. Briefly, for tasks Comm 2 and 3, students within each group would be assigned the same grade based on their group performance in these tasks. Comm 4 is a task that students are encouraged to complete individually, but this rule is not strictly enforced. It is also difficult to determine if students were objectively evaluating their groupmates in the Comm 5 task as student performance in the task is dependent on the grades they receive from their groupmates and these assigned grades are not moderated by the instructors (e.g. students could decide as a group to assign the highest grade to every group member, regardless of their actual performance). Thus, the assumption is violated because it is difficult to isolate the contribution of each student to the grade they have obtained for these tasks. Student performance in the core competencies was not included in the model, because it perfectly discriminates between the two outcomes, i.e. students who do not demonstrate competence in any of the four techniques would fail the course by default. In the construction and implementation of the predictive models, students who were exempted from a particular assessment task were assigned the average grade, or score for quizzes, obtained by the cohort (4% of students in the cohort received at least 1 exemption). EOS, MCQs and SAQs were not included into any multivariable analysis due to reasons described below (Section 3.2).
Section 2: Construction of the three logistic regression models. First, logistic regression models where each variable was individually regressed against the outcome was constructed and the estimated coefficients tested for significance. Multivariable models -Model Full, A and B were then constructed and were tested to determine if they fulfilled the following regression assumptions: i. a minimum of 10 cases of the rarer outcome exist per variable and there are no missing values ii. absence of damaging multicollinearity iii. linearity between the logit outcome and continuous variables (including ordinal variables that were coded as such) Model predictive performance was summarized using the Area Under the Receiver Operating Curve (AUCROC) as well as the Brier score. Both were estimated using Efron's enhanced bootstrap [30] that was repeated 200 times (see [24]). Briefly, AUCROC (ranging from 0 to 1) estimates the capacity of a model to discriminate between the two outcomes (pass and fail) and an accurate model would have a high AUCROC score. In contrast, the Brier score (ranging from 0 to 1) measures the difference between the predicted and actual probability of a student failing the course and an accurate model would have a relatively low Brier score.

Descriptive statistics and univariable regression outcomes
Descriptive statistics of Pass and Fail students are presented in Table 2. Pass students performed better than Fail students in every assessment task evaluated in this current study. However, it was noted that not all pre-university experiences contribute to student performance in this first year university course. Table 2. Descriptive statistics of a) categorical and b) continuous/ordinal data of students who passed and failed the course for each variable of interest. a) The percentage of students who studied at least one subject of interest (1) and those who did not (0) among students that passed (% Pass) and failed (% Fail) the course is presented for each categorical variable. Knowledge scores in b) were also scaled to 5% and are presented in parentheses. p<0.05 was set as threshold required for significance. N = 876 a) Univariable regression models were constructed for each variable to provide a preliminary indication of their relationship with each student's probability of failing the course. There is evidence to suggest that every variable (Table 3), except BioHS (p>0.05) has a significant inverse relationship with the outcome, where the odds of failing the course decreases when student performance in the assessment tasks increases (p<0.001 each variable) or when students studied at least one chemistry subject in high school (p<0.01).

Multivariable analysis
Every variable from the univariable analysis was used to construct multivariable models to determine how each variable, together with other variables, influences the outcomes as student course outcomes. This is important as, in reality, each variable cannot influence students' course outcomes independently of the other variables. Others have also argued that the variables in the multivariable model should not be derived solely from the results of the univariable regression analysis (i.e. only retaining significant variables) as the contribution of certain variables to the outcome can vary depending on whether they are placed in a univariable or multivariable regression model [24,31].
Student performance in the EOS exams was removed from Model Full as there is evidence to suggest that student performance in the EOS exam alone can be used to accurately predict student course trajectory. Bootstrap model performance for the EOS univariable model had a higher AUCROC (0.992) and Brier score (0.018) compared to the model with every predictor except Comm 2 to 5 and EOS (AUCROC: 0.906, Brier: 0.084, Table 4), Model 1 (AUCROC: 0.883, Brier: 0.095, Table 5) and Model 2 (AUCROC: 0.901, Brier: 0.086, Table 6). This result is also unsurprising given that the course curriculum was designed with a greater focus towards developing and assessing student understanding of biological concepts to better prepare them for more advanced courses in the future.

Discussion
Factors influencing student academic performance in tertiary education are multifaceted and are often dependent on the context in which learning occurs [1][2][3]. The LA paradigm is often employed by researchers to develop an understanding of the student learning process, which in turn allows them to identify and address ineffective learning practices through targeted interventions (e.g. [4,5]). Assessment tasks would serve as suitable targets for interventional action and also as reliable variables to construct highly accurate predictive models, given that they have been known to drive the student learning process and to predominantly influence student course outcomes [13,18,32]. This is particularly apt for first-year university students, and even more so for at-risk students, as they have a poorer understanding of how to perform competently in assessment tasks [17][18][19]33]. The current study attempted to construct predictive models, which would be used to identify those in the student cohort that are at risk of failing the course in future course iterations. Furthermore, the study attempted to identify assessment task(s) that students find challenging, as this would inform the design of future interventions.
Similar to the findings of other studies [8,9], the current study highlights the importance of course-specific factors in predicting student course performance. However, unlike other studies (e.g. [8,12]), our models did not include data on student interactions with the course Learning Management Systems (LMS), but instead included preuniversity variables that could influence student assessment task performance. This was because our primary goal was to identify assessment tasks or variables relating to assessment task performance that our students found challenging to serve as the basis of future interventions. Additionally, models constructed using such variables were highly accurate, in that both of our predictive models (Tables 5 and 6) have excellent capacity to discriminate between the two outcomes, as shown by an AUCROC value between 0.8 and 0.9 [34], and thus allow us to better identify those students at-risk of failing the course in future course iterations. Evidently, as others have also demonstrated [11,12], assessment tasks are highly capable of predicting student course outcome due to the intrinsic nature of assessments, specifically to expose student understanding of course content [13] and to indicate the extent to which students achieve course learning objectives. Our results from this study, in tandem with education literature and instructor insights, allowed us to identify deficiencies in certain aspects of the student learning process and issues underlying poor assessment performance.
Firstly, this analysis showed that students were found to perform the worst in assessment tasks that evaluated their understanding of physiological concepts taught in our course (Tables 2 and 4). Observed poor performance across every knowledge-based task suggests issues with the learning process, and specifically this could be attributed to students using inappropriate learning goals to direct their learning, resulting in poor learning outcomes [37]. First-year university students have been shown to learn didactically, where students passively learn content in the manner in which it is delivered [35]. As traditional university courses are modular in nature, students would learn concepts without integrating them across modules and physiological systems [36], and this in turn hinders the development of complex conceptual understanding in our course [16,21]. The lack of such integration by our students is further exemplified by the fact that they scored the lowest in the SAQ task (Tables 2 and 4), where conceptual integration is an important feature of high-scoring answers to these questions [14,16]. This finding also validates the necessity of the inclusion of this learning activity which was designed using insights from course coordinators as well as education literature, and implemented in our course prior to this study [37]. As conceptual integration primarily occurs during learning outside of lectures, the learning activity was designed to optimize the student learning process by highlighting the importance of such integration in developing complex understanding of physiological systems [37]. Results were favorable, as analysis revealed that students who participated in the learning activity developed a better understanding of course content by the end of the course than those who did not participate [37].
Secondly, the current study findings also show that the SAQ task was the most challenging assessment task for our students (Tables 3 and 4). Unlike other assessment tasks, students are only exposed to the SAQs in the EOS exam and, therefore, are not given feedback on their performance until after they have completed the course. The lack of feedback to develop the already limited understanding of assessment requirements by first-year students, results in poor self-evaluation of future task performance, and this in turn prevents students from adequately preparing for the task [17,38]. This is especially true for poor performing students for whom self-evaluation is most inaccurate and often results in them overestimating themselves -a well-documented phenomenon that is termed the Dunning-Kruger effect [39][40][41]. SAQs also have complex, often subjective task requirements, which further exacerbate the problem [14][15][16]. These requirements include coherence of presented ideas, inclusion of concepts that are relevant to the question and the fact that it is difficult for naïve assessors to determine the level of detail required in their answers to the SAQs [14][15][16]. Even though the preexisting activity [37] improved student SAQ performance via improvements in student conceptual understanding, the feedback given in the activity was too complex to serve as goals or standards that students can use to prepare for the SAQ task. Therefore, feedback in the pre-existing activity [37] was refined to include key SAQ requirements and advice about how to adequately demonstrate them in answers, and this was articulated to students via a second learning activity [42]. Results suggests that the second activity improved student SAQ performance and that students thought the activity was useful for learning and that it helped with their preparation for the EOS exam [42].
It can be concluded that student performance in assessment tasks is a strong predictor of student course trajectory. However, implementing models at earlier time points (earlier than week 5 in this instance) would require the inclusion of additional variables in order to ensure that the relatively high predictive power of the models is maintained. These variables can include student performance in formative learning activities, such as questions administered at set intervals during lectures [11], in class tasks designed to improve student learning and course outcome (e.g. the two interventional learning activities [37,42]), or other proxy measures of performance such as student learning behaviors and dispositions [43]. These variables could also provide additional insights into the student learning process and issues that can be addressed in future interventions. For instance, future interventions in our course could be designed to enrich students' fundamental physiological knowledge, as poor performing students in our course lacked the prior knowledge necessary (as evidenced by a low ATAR rank) to learn content taught in our course effectively [21]. A 'recommender' system (e.g. [44]) can also be utilized to help students identify gaps in their prior knowledge and facilitate student engagement with said content prior to the start of the teaching semester to ensure that they are better prepared for the course.
Lastly, findings demonstrate that the combined use of data-driven methods of LA, education science principles (e.g. the Structure of Observed Learning Outcomes (SOLO)) and course coordinator experience is valuable in the design and redesign of learning activities, as was with the case with the first and second interventions described in this study [37,42]. Furthermore, outcomes from each intervention will then feed into the LA data in subsequent course iterations, leading to even better, improved course design in the future (See Figure 1). Fig. 1. Learning analytics cycle, adapted for the current study. Chart depicts how LA can be used in tandem with course coordinator experience and learning theories to enhance effectiveness of learning activities in improving student course outcome.
Learning activities to address factors that negatively affect student course outcome