Establishing a Sustainable Process to Measure Learner Performance

The quality of student learning and academic rigor is central to higher education. Nonetheless, colleges often prioritize metrics such as enrollment and graduation rates or use assessment data to solely fulfill accreditation requirements. The Academic Quality Assurance team at a university ventured to expand the academic quality data landscape to learn more about student success. This paper shares the team’s journey to collect and report on student performance data for continuous improvement of academic programs. Specifically, this paper includes the methods to promote a culture of assessment by incorporating new concepts into the academic quality assurance process: Data visualization and storytelling with data. This paper includes the methodology to collect and report on data, samples of the systems and visualizations used, and the challenges faced in the context of people, process, and tools. Keywords—Outcomes, student, learner, performance, Tableau, academic quality, rubrics, grades, assessment, faculty, dashboards, assignments, competencies,


Introduction and Historical Context
How do higher education institutions define student success? Institutions define success based on competing priorities -while funding models in one institution may deem graduation and retention rates the hallmark of student success, student demographic in another institution may prescribe enrollment and completion rates as critical data points to track. Critics argue that the data from such variables tells us about what happened in the past and fails to inform us of how we can fix the present with practical approach or research. When we consider student experience as another vital piece of the puzzle, can the aforementioned variables truly gauge student success holistically? According to Pelletier, "if we are measuring only retention and graduation, we lose the opportunity to learn if what we are doing to support student success is working and to make any needed course corrections" [1, p. 60]. How can we move beyond typical reporting on institutional success and focus on the individual learner?
For faculty, who focus on class or individual learner performance, one of the most critical aspects of student success is achievement of the intended learning outcomes within a degree program or course. According to the Inside Higher Ed survey of facul-ty attitudes on technology, "faculty members are most likely to rely on student grades on exams and other assessments, and on student demonstration of competency in specified learning outcomes, to gauge whether their course is meeting its intended aims" [2, p.7]. Student assessment metrics help define the learner's success in terms of quality learning, that is, competencies and attitudes gained in order to lead a successful professional and personal life. As Suskie states, "Because we're not telling the stories of our successful outcomes in simple, understandable terms, the public continues to define quality using the outdated concept of inputs like faculty credentials, student aptitude, and institutional wealth -things that by themselves don't say a whole lot about student learning" [3].
This challenge afforded the opportunity for a college to refocus the attention on assessment data to obtain a more holistic picture of student success in the context of learning. To better assess learning and academic quality, the college formalized its curriculum assessment practices in 2012 and created a dedicated Academic Quality Assurance team. The mission of the team was to implement a systemic approach to achieving program and course quality using research, best practices, and analytics. The primary goals of the team were to:  Provide authentic evidence of student learning  Productively use assessment data to inform and strengthen the teaching and learning practice  Support faculty in designing authentic assessment in their programs Per recommendations from accrediting bodies and professional organizations such as the National Institute for Learning Outcomes Assessment, the team established Assessment Plans for academic programs. These Assessment Plans included the typical Student Learning Outcomes (SLOs) and faculty-selected direct measures, or authentic demonstrations of learner performance such as capstone projects, case studies, projects with employers, or field exams [4]. These signature assignments are programs' direct measures to assess if the learners achieved the expected outcomes.
Faculty are essential to the successful adoption and implementation of any academic assessment initiative or framework. One of the major findings of the national survey on the current state of student learning outcomes conducted by the National Institute of Learning Outcomes Assessment was that "faculty are the key to moving assessment forward" [4, p. 11]. The implications of this and other findings call for urgent responsiveness to more faculty participation in the assessment and evaluation of learning at institutions. Consequently, while the Assessment Plans were created and approved, the team was unable to collect data initially due to faculty skepticism, and lack of alignment and implementation across the programs. This resulted in the inability to 'close the loop' for analysis and improvement. When the data finally was collected, it was inconsistent and difficult to interpret.
As a result, the team crafted new goals to overcome the challenge of faculty buy-in and to build trust. The first task was to streamline the assessment cycle by redefining the processes and repurposing the reporting tools to make it easier for faculty to understand assessment and partner with the team. Next, the college reevaluated the original framework used to define the student learning outcomes and collect assessment data. One-on-one consultations with faculty leads helped redefine the outcomes and measures applicable to their individual program and students. The team hosted several assessment workshops to inform and identify faculty champions, who later fully embraced the assessment process and shared best practices from their own programs.
Outputs are only as good as the inputs. This holds true for assessment data, which includes well-defined learning outcomes, measures, and grading criteria. The collaborative effort of the team and faculty led to the successful completion of the planning cycle where each academic program had a well-defined Assessment Plan with concrete learning outcomes, well-aligned direct and indirect measures, and analytic rubrics for grading. The team finally established a process to report on this recently defined data and made it meaningful for faculty by drawing from methods of data visualization and effective storytelling. This resulted in a new appreciation for the assessment data and the connection to program goals and curriculum among faculty. Finally, the results were used, as intended, to inform teaching and learning.

The Assessment Process
The team supports the assessment of student learning through an Annual Program Evaluation process. This is a systematic and standardized process to assist in the mapping, tracking, analyzing, and reporting of program student learning outcomes, based on the program competencies and LEAP VALUE Rubrics frameworks [5]. The plan is implemented in collaboration with the lead faculty for each academic program. The academic programs collaborate with the team on the annual program reporting process (see Figure 1). Program faculty complete Annual Academic Program Reports in consultation with other relevant faculty. These reports provide faculty and academic programs with current data for decision-making and planning, reinforcement of best practices, and assisting with systematic data collection for program evaluation and accreditation processes. The data collection phase (Phase II) of the assessment cycle includes the program measures. The team compiles both direct and indirect data on student learning outcomes, as well as educational effectiveness data for programs to help faculty analyze the data. The types of data in this program reporting consists of:  Direct measures: The signature assignments, rubric scores or grades to measure student learning outcomes  Indirect measures: Surveys, or self-reported data on student learning outcomes  Educational effectiveness data: Retention, graduation rates, course completions, licensure and/or job placement rates In Phases III and IV, faculty review the data, draw conclusions, and craft a narrative in relation to defined goals, previous reporting cycles, and then create new goals for the subsequent year. The Annual Academic Program Reports in the Assessment Cycle include a critical element to connect assessment data with action items for continuous improvement for the subsequent year: Progressive Refinement Goals. This allows academic programs to reflect upon the findings and create tangible goals to address gaps or enhance the curriculum. Enhancements typically consist of a myriad of curriculum changes such as course design, teaching practice, program branding or research-related projects, including survey research.

Assessment for Improvement
Assessment reporting is typically categorized as 'assessment for accountability' for accreditation requirements, and 'assessment for improvement,' for internal analysis and program enhancement [6]. Assessment for accountability is essential, and the accrediting body, The New England Commission of Higher Education, provides many examples of metrics such as graduation and retention rates for student achievement [7]. However, the curriculum and improvement components are often overlooked, and faculty often associate the assessment work with accountability. Jeremy Penn articulates the difference between assessment for improvement and assessment for accountability in Assessment for 'Us' and Assessment for 'Them': when we perform assessment for "us" we are not afraid to discover bad news. In fact, when we assess for "us," it is more stimulating to discover bad news about our students' performance because it provides clear direction for our improvement efforts. In contrast, when we perform assessment for "them," we try our best to hide bad news and often put a positive face on the bad news that we can't hide [8, p. 1].
As the primary goal of the team is to inform teaching and learning at the program and college level, it prioritizes work around assessment for improvement. However, the team reporting was perceived as an obligation for accountability as faculty were more interested in current data on teaching and learning as opposed to traditional reporting that reflects upon the past academic year to identify gaps and successes. Breaking the conceptual tensions between assessment for accountability and assessment for improvement was a difficult task. Ewell cautions institutions "adopting either one of these perspectives will decisively influence institutional choices about what and how to assess, how to organize assessment, and how to communicate assessment results" [6, p. 5]. The team recognized that the data from the Annual Academic Program Reports is made available too late to influence decision-making for faculty and students, who have moved on to a new academic year. This made it difficult for faculty to view assessment as a means for improvement, and not merely a required reporting function for regional or national accreditation agencies. The team adopted more 'real-time' assessment approaches to track and report measurable impact on student learning [9]. The team focused on building data dashboards and processes that would update regularly to give insights into student performance, realtime. This helped the team establish the foundation for a robust process for reporting actionable data on academic quality indicators of student learning outcomes.
Furthermore, to prioritize assessment for improvement and inquiry, the team engaged in special projects based on faculty interests by asking faculty what competencies, data points or general questions they are curious to explore within their academic programs. Independent projects and ad-hoc requests were identified where the team provided consulting support, such as with survey research, prior learning assessment, or conducting custom grade and rubric analysis. To illustrate this, the figure below displays a custom rubric analysis for several sections of a communication class as an independent project outside of the yearly reporting. In interpreting the results, learners performed the best in Speaking as 86% of learners scored Meets or Above Standards on the rubric (summing top two levels of achievement). Similarly, learners performed well on Critical Thinking as 80% of learners scored Meets or Above Standards on the rubric. Learners did not perform as well on Academic Integrity, as 47% of learners scored Meets or Above Standards on the rubric. Faculty were able to use this data to address foundational skills and knowledge gaps for international students in this introductory course so they were better prepared to take the core courses in the program.
Overall Distribution of Rubric Categories Communication Class

Fig. 2. Assessment within a Single Class
This type of work gained interest in assessment in general, away from the typical enrollment and graduation rates reporting or Annual Academic Program Reports. Rachel Ebner identified the need for this type of work for the purposes of Rebranding Student Learning Assessment to come to a common understanding [10]. This kind of specialized reporting not only informed our assessment work but also helped build organizational capacity and data literacy for the college.

Methodology
Data collection for direct measures on student learning outcomes finally occurred after the team worked with faculty to develop consistent assessments within programs and used rubrics in the Learning Management System (LMS). Each program has defined competencies that dictates the student learning outcomes and signature assignments/direct measures in their Assessment Plan. The team facilitates the collection and analysis of learner performance data in the defined measures through grades and rubric scores. The definitions and interrelationship among the competencies, outcomes, assessments, and measures explain the approach to building content and assessment for learners based on the student learning outcomes (see Figure 3). The defined student learning outcomes, as derived from the competencies, are mapped to the learning objectives and assessed through granular, rubric-based assessments to measure student learning. The rubrics itemize the evidence that students need to show on any given assessment to demonstrate competency.

Fig. 3. Direct Measures Framework
The team compiles the grades and rubric scores after each semester. To protect learner data, course sections and student identification are removed, and data is stored in a secure site, and then results are compiled. The results are analyzed against set benchmarks and visualized using Tableau Server and shared with faculty to interpret and make decisions regarding their curriculum. The team is involved in each stage of the process. Faculty identify the courses and measures, and the team consults with the faculty who teach the classes and provide guidance on best practices. The team works with faculty to design their rubrics and provide trainings on how to build and grade Competencies Learning Outcomes

Learning Objectives
Measures the rubrics in the learning management system. To ensure reliability, the following assumptions are made:  Alignment: Rubrics are aligned to the learning outcome and assignment  Consistency: The same assignments are used across class sections  Accuracy: Rubrics are calibrated so the data is the same (reliability) To ensure alignment, the team regularly review the rubrics with faculty. Oftentimes measures are changed, thus the team suggests keeping the measures and rubrics the same for at least a year. For consistency and accuracy, the team relies heavily on lead faculty to communicate the requirements to other faculty teaching so the direct measures are the same. Any missing data is reported to the lead faculty. For accuracy, rubric results might demonstrate grading differences among sections rather than solely student performance. Therefore, rubric calibration sessions are suggested for faculty to improve inter-rater reliability. Figure 4 below shows sample data on learning outcomes for a program, where the results are displayed as the percent of learners who met the goal set by the program: Results: Performance on Student Learning Outcomes (SLOs)

Fig. 4. Sample Data: Performance on SLOs
This data becomes part of the Assessment Cycle, where faculty review the performance data along with graduation and retention rates, survey data, and participation in experiential activities in their Annual Academic Program Reports. The visualizations and storytelling with data element began in 2018, as the team provided a list of data points based on the student lifecycle. Figure 5 includes a sample 'insight sheet' for a program that contains direct and indirect measures, and educational effectiveness data. The metrics include the results on students learning outcomes, but also retention and graduation rates, survey results, and participation in experiential activities. The requests for additional data points and more real-time data helped prioritize what to include (e.g. the 'inputs'). This data is compiled by the team and shared with faculty as part of their reporting.

Fig. 5. Sample Program 'Insight Sheet' Storytelling with Data
Effective storytelling is imperative when interpreting assessment data. This type of data and reporting requires context and a narrative, as faculty have the details within the curriculum. Rather than just sharing the link with Tableau, results are discussed in the faculty meetings. Cole Nussbaumer Knaflic identifies the need to know your audience, the who in the importance of context [11, p 19.]. By having a relationship with the audience, the team views the data results with faculty who have an initial understanding. Faculty can look at their results and then make informed decisions such as updating curriculum, providing resources to learners, or sharing success stories.
To enhance the storytelling, each faculty writes a yearly narrative based on the results. The Annual Academic Program Reports include a narrative section where the faculty tell a story of the academic program based on the data, along with gaps and success stories, as well as goals for continuous improvement for the subsequent year. The team asks faculty to:  Reflect on direct measures related to student learning outcomes: Other than GPA, what data/evidence is used to determine that graduates have achieved the stated learning outcomes for the degree (e.g., capstone course, portfolio review, licensure examination)?  Reflect on indirect measures that provide self-reported information based on student, teacher, or outsider's perceptions of what students have learned. Examples may include student surveys, licensure and/or job/graduate school placement rates, or employer surveys. Other indirect measures include educational effectiveness data such as retention, graduation rates, course completions, and market analysis.  Discuss the program evolution based on university, college, and/or industry or market needs  Discuss program changes based on the gaps or wins revealed by the data or other means As Daniel Kahneman states, "No one ever made a decision based on a number. They need a story" [12]. In addition, one source data can never tell the whole story and triangulation is imperative. For example, poor performance on an assignment may be due to lack of clarity on the assignment instructions, an increase in rigor or tough grading, or change in demographics of students in each cohort. Lead faculty have this context.
The first several years of reporting resulted in simply listing out data points, but not connecting to the exciting work that the programs are currently doing. To garner engagement, the team workshopped with faculty on a session solely on storytelling with data. Elissa Fink and Susan Moore in Presenting Data Does Not Always Equate with a Good Story recall that "the story is simple but the analytical findings are not." [13, p. 4]. As a result, the narratives became much more robust, two samples are below.
Sample I: While meeting our benchmark for the research manuscript requirement, more work is needed in supporting students who have difficulty with the discussion and data analysis sections of research manuscripts. Our plan for next year includes moving the IRB application assignment from XXXX to XXXY in the fall, thereby giving students an earlier start on collection of research data during the winter quarter in XXXX.
Sample II: The Alumni snapshot profiles 248 alumni, the majority are employed in public relations and fundraising, followed by program managers and staff. Anecdotal evidence shared with faculty reveals that students requested more education in nonprofit development. This is consistent with the majority of alumni job titles and reports by Nonprofit HR, an industry association, indicating fund development is a top 3 skill sought after by nonprofit employers. In 2018, we instituted a PLA (prior learning assessment) for XXXX for students who had earned the CFRE (certified fundraising executive) credential. To address the need, we proposed, and were approved for the addition, of a fundraising concentration.
At the conclusion of the reporting cycle, faculty share the report with their constituents. A summary report on trends is shared with the Deans, and awards are given for the best narrative in the report. These reports are internal but helped shift the notion of assessment for accountability to assessment for improvement.

Challenges
The means to operationalize and automate a systematic reporting of student achievement was a challenge. As frameworks and goals change, the notion of using data to inform teaching and learning remains the same. For the process to work and be sustainable, several challenges were overcome.

People
The first challenge involved building capacity and obtaining executive support. The team requested monthly check-ins with Academic Deans and secured time to speak about assessment and garner faculty input during monthly full faculty meetings. The team organized information sessions when consensus was required on assessment standards. A similar challenge relates to the prioritization of key performance indicators for the college around metrics such as enrollment reporting versus reporting on student performance. The team worked closely with the Academic Deans and faculty leads to define benchmarks for each program to analyze performance data on student learning outcomes and share findings. The data helped measure progress and craft progressive refinement goals for the subsequent year.
Another key challenge involving people is the need to map actual learning in the individual courses within a program to the overall program mission and student learning outcomes stated in the Assessment Plan. This requires faculty buy-in, which largely involves collaborating with faculty on course redesign. The IMPACT program at Purdue illustrates that any number of evidence-based assessment and teachinglearning models can be successful, but the true adoption will occur only if we enable faculty to create student-centered and engaging learning environments [14]. In other words, "course transformation rather than course redesign" driven by faculty [15, p. 16]. In order to maintain momentum, the team schedules reoccurring monthly meetings with faculty and have done so since 2012. The one-on-one consultations and meetings cover a myriad of topics, such as curriculum mapping, assessment planning, research, and faculty training.
Another significant challenge is data literacy among faculty and staff. As Williams accurately claims, "data literacy can help colleges and universities create cultures that value assessment and continuous quality improvement" [16, p. 1]. However, often the institutional metrics shared with leaders and faculty do no yield concrete conclusions as the metrics are presented in their raw from or lack context. It is critical for assessment teams to help faculty make sense of the data in a meaningful way and make direct connections with their teaching and learning. The team hosted information sessions to educate faculty leads on how to interpret and report from data on dashboards and a workshop on storytelling with data. "A data-informed approach is one step up the evolutionary ladder from data-driven." [17, p. 1]. Data literacy enables leaders and faculty to interpret data to shape their understanding of the strengths and weaknesses within the context of the students and the learning ecosystem.
Lastly, assessment professionals assume different roles depending upon the organization's state of maturity in assessment. These roles change based on the situation from being an assessment expert to a translator, a facilitator, a political navigator, a change agent, or a project manager. In the varied levels of roles played by assessment professionals, the support from administration and leadership is identified as one of the most critical factors as well as a challenging one that contributes to the most successful completion of assessment work [18].

Process
There were several factors that led to the new process and approach. In 2012, the team selected the Degree Qualifications Profile (DQP) framework and the AACU Value rubrics to help drive the direction of the process [19]. This framework, developed by the Lumina foundation, is "a learning-centered framework for what college graduates should know and be able to do to earn the associate, bachelor's or master's degree" [19]. The framework offered a shared terminology for student learning outcomes across the programs. However, the college is committed to continually exploring and adjusting frameworks based on the industry needs. Thus, the college decided on a different framework based on industry domains and demands for new workforce skills such as technology and data literacy. This 'domain approach' enables faculty and course designers to discover and map connections across multiple areas of global and societal needs, academic expertise, and industry practices. Domain work is a collaborative process undertaken by faculty from multiple academic disciplines, as well as learning designers, industry practitioners, and others. In addition, the shift towards competencies helps address employer and industry needs by working on next generation offerings such as modularized and competency-based learning.
This posed yet another challenge of aligning assessments with a competency-based model unique for each program. One of the benefits of any framework as reported is "At the institutional level, the Degree Qualifications Profile provides reference points that allow faculty members to articulate and better align institutional student learning outcomes with departmental objectives. Instructors and students can then refer to the Degree Qualifications Profile as a common source of understanding and point of departure for agreement on more detailed and specific expectations about programs, courses, assignments and assessments." [19]. The team hopes that the domain competency framework will do the same for faculty and students while being closely aligned with relevant professional industry or employers. The team began working with faculty leads to help them align the programs with the university vision and new frameworks. The primary challenge was the transition away from the Degree Qualifications Profile framework to a new framework to be embraced by the college to help define the entire assessment cycle from student learning outcomes to curriculum mapping, data collection and reporting.
Another process challenge was to obtain buy-in for shared assessment terminology and standards around rubrics for consistent data collection and reporting. This is to improve the data reliability. Access to context based and targeted feedback on assignments via rubrics can empower and motivate students to take ownership of their own learning while access to evidence-based data on student performance and learning via rubrics can empower faculty to make changes to instructional and assessment design, real-time. This kind of data can help faculty make informed decision about their teaching and student learning, real-time.

Tools
One common challenge was that the metrics to measure success and learning live in distinct systems. While there are many points of direct connection with Operational Databases such as the Student Information System, not everything is connected, such as survey data. The team still compiles some data from different data sources into a single view. Also, the absence of a university-wide assessment or curriculum management platform calls for some manual collection and analysis, which is not an ideal solution for reporting, both from an efficiency and visual appeal standpoint.
Another challenge was faculty's interest in real-time assessment (per semester) as opposed to the after-the-fact reporting (annually) which is associated with accountability, not improvement. Given the faculty interest in real-time assessment data and learning analytics, one aspirational goal of the team is to explore the guiding process for the development of institutional polices and strategic planning for learning analytics at the college and university. One such model that has been helpful in formulating a framework for identifying the steps and challenges is the SHEILA framework that outlines six dimensions of ROMA-rapid outcome mapping approach [20]. The team hopes to explore this model to lay down the foundation for learning analytics that would help in informed decision making related to student success and experience. http://www.i-jai.org

Next Steps
A recent review of the Annual Academic Program Reports and program goals surfaced several issues. First, for data reliability there is a need to establish consistency across the academic programs, including course outcomes, signature assignments, and rubrics. Similarly, there is an urgency for revisiting the curriculum mapping on a regular basis to align with the new competency framework. There is also a goal to increase the number of authentic experiential assignments and invest in innovative projects to improve the learner experience, such as implementing virtual technical labs, and utilizing more predictive models in our analytics. The themes led to the following priorities for the college related to academic quality, including:  Rubrics: Facilitate consistent implementation of rubrics in the learning management system to report results by competencies.  Curriculum Mapping: Investigate an assessment platform and curriculum management system to assist in efficient competency course mapping and documentation for all programs.  Part-Time Faculty Support and Training: Improvements have been made in terms of training in assessment, analytics, and storytelling with data. The team continues to collaborate with faculty to share best practices guides and training.
 Data: Build or enhance learning analytics and visualizations to expand faculty access to real-time assessment of key program quality metrics: faculty engagement, student engagement, and competencies via rubrics.
In general, a solid Academic Assessment Plan for an academic program is a framework for aligning a university mission that aims to "empower humans to be agile learners, thinkers, and creators, beyond the capacity of any machine." [21, p. 1].
Only an effective and intentional curriculum design can ensure the alignment of the defined measures for learning outcomes, which will lead to learner-centered, high quality learning experiences. Furthermore, only the consistent implementation of the curriculum will lead to strong evidence of learning (e.g. the "data") that speaks to the academic quality or health of an academic program. The Academic Quality Assurance team will continue to work with faculty and staff to achieve its goals of not only assessment for accountability, but more importantly, assessment for improvement.

Authors
Dr. Mamta Saxena is the Director of Assessment at the College of Professional Studies at Northeastern University in Boston, Massachusetts in the Academic Quality Assurance unit. In this position, she is responsible for providing leadership to execute a long-term strategy for data collection and measurement of student learning outcomes and learning analytics for the purpose of program improvement and instructional innovation. She works closely with the faculty, as well as the academic leader-ship in the College of Professional Studies to implement a culture of inquiry and continual improvement of academic programs. Prior to this, she has worked in corporate and academic settings to implement innovative and effective instructional design and assessment approaches, both in India and in the U.S.
Melanie Kasparian is the Associate Director of Assessment at the College of Professional Studies at Northeastern University in Boston, Massachusetts. She is both a professional and lifelong academic with expertise in e-learning, faculty development, curriculum design, and assessment. She has a strong enthusiasm for collaborating with faculty in the creation of high quality and intuitive learning experiences. With a background in Mathematics, she is highly experienced in interpreting data and applying it within needs analysis scenarios to implement targeted improvements to curricula based on experiential and evidence-based learning principles. She supports the Director of Assessment in finding ways to collect and report on data related to learning analytics and the overall student experience.