Improving Academic Decision-Making through Course Evaluation Technology

The objective of this study is to offer a broad understanding of how end of course evaluations can be used to improve the academic outcomes of a higher education institute. This paper presents the key findings from a study conducted using twenty-three academic degree-programs, regarding their use of end of course evaluation technology. Data was collected from an online survey instrument, in-depth interviews with academic administrators, and two case studies, one in the US and another in the UAE. The study reveals that while historically end of course evaluations were primarily used to gauge the performance of instructors in the classroom, there are several new trends in the use of end of course evaluations that can help higher education institutions improve academic assessment, teaching and learning, and academic administration decision making. Those trends include sectioning and categorization; questions standardization and benchmarking; alignment with key performance indicators and key learning outcomes; and grouping by course, program outcome, program, college, etc. in addition to those vertical structures, higher education institutions are vertically examining a specific question(s) across. End of course evaluations are now poised as an integral tool and a key health indicators of academic programs. Keywords—Higher Education, End of Course Evaluation, Academic Programs, Academic Assessment


Introduction
In today's "information society" Higher Education Institutions (HEI) must create, share, and use knowledge to improve their well-being [2]. The use of Knowledge management to operate efficiently and effectively is now required in all industries, and education is no exception [1]. With increasing global competition, HEI are facing increasing pressures to respond in a timely and efficient manner to political and social changes at both the national and global levels, albeit with fewer enrollments, declining support from the government and the enterprise society; and with growing regulatory demands for transparency and accountability [6]. HEI must provide students, faculty members, accrediting bodies, governments and administrators with evidence of students' performance and learning achievements [18].
To address all the above, and to remain viable and competitive, HEI must be able to make better decisions regarding academic issues such as course, program and faculty performance while monitoring resource allocation and return on investment [5], and [12]. Those decisions required dealing with the challenges and opportunities of the higher education environment require a constant flow of valid, timely, and relevant information that institutions can use in an efficient and effective manner. In many cases, this data is already available in one form or shape in an information systems used by the institution, but often that data is either overlooked, underused, or otherwise not properly utilized [5], [15] and [14].
This study sheds some light on the potential utilization of one of those underused key higher education technologies, namely the End of Course Evaluation (EOCE) technology. In general, EOCE are provided by students and faculty members at the end of each term. The significance of the EOCE system is that it is the primary system that contains quantitative and qualitative information coming directly from university students and faculty and as such, it contains a wealth of information that can help HEI make more-informed decisions regarding their performance [17].
Technological advances have enabled institutions of higher education to administer course evaluations online, forgoing the traditional paper-and-pencil [27]. While there are wide differences of opinion as to the validity of End of course evaluations (surveys) as a measure of instructor's performance, many administrators feel that end of course evaluations is a primary measure that fills this gap [25]. Today, course evaluation technology is an integral part of a university's overall evaluation process [30]. This study however takes end of course evaluations to a whole new dimension of analytics by illustrating how end of course evaluations can be structured more objectively to measure more than instructor's performance.
Given the importance of higher education in today's information society and knowledge economy, and the role EOCE technology can play in higher education decision making, the aim of this study is to provide an overview of the emerging trends in the use of EOCE technology and how those capabilities could be utilized to improve academic assessment, teaching and learning, and academic administration decision making. This study is significant to higher education institutions, as well as to other stakeholders involved in the hierarchy of higher education, including students, educators, researchers, institutions, and government agencies [16].

Related Work
Business Intelligence (BI) describes the "technologies, systems, practices, methodologies, and applications used to analyze large amounts of diverse business data to help organizations convert large amounts of raw data into meaningful information to support sound and timely decision-making [4] and [21]. Analytics is a component of business intelligence that provides techniques to recognize trends from patterns in data and to make decisions based on those trends for the overall advantage of the organization [22]. Learning Analytics (LA) is a sub category of BI concerned with the "measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.
While LA is more concerned with the learning process, Academic Analytics (AA), in contrast, is the application of education analytics for better decision making at institutional, regional, and international levels'' [11] (see Table 1). More broadly, LA and AA tools are used to improve HEI' processes and workflows, measure academic and institutional data, and improve organizational effectiveness [9] and [23]. Table 1. Learning and Academic Analytics. Source [10] A key element of LA called visualization is used in a way in which analysis results are displayed so they are easily understood by decision makers [3]. With LA, stakeholders will have easy, visualized access to massive amounts of digital data left behind from learners about learning experiences in various systems in the same way that the business intelligence market analyzes consumer data today [8]. LA can help higher education institutions by gathering data from various sources to make decisions about academic progress, predictions about future performance, and to recognize potential issues [7]. While there are numerous datasets of learner information available for the field of education, there is still a need for improvement in the process of measuring, collecting, analyzing, reporting, and sharing data across institutions themselves [20]. One of the most significant challenges facing higher education today is the lack of knowledge about the ways that students interact with the learning process.
Optimization of learning requires not only retrieval of useful information and knowledge about learning processes and relationships between learning agents, but also the transformation of data gathered into actionable information. The ultimate objective of LA must be to enable data-driven educational decision making at all levels [19].
EOCE serve several potential goals. According to a study conducted by [26], the goals of course evaluation in order of importance and popularity and as perceived by both faculty and academic administration are shown in table 2. Remarkably, both faculty and administration ranked "teaching improvement" and "course improve-ment" as the most relevant and popular goals of course evaluation. In comparison, "evaluation of faculty performance" appears low on the priority list. Table 2. Goals of EOCE. Source: [26] Electronic course evaluation systems have several benefits and drawbacks relative to traditional paper-and-pencil technology that need to be carefully analyzed and compared before selecting the most appropriate evaluation technology. In addition, while student response rates are an important factor in evaluating the success of a course evaluation system, it is only one of many performance parameters [24]. Table  3 shows the pros and cons of electronic course evaluation technology Table 3. Pros and Cons of EOCE. Source: [24] Today, many highly capable electronic course evaluation systems are available whose capabilities are ever improving. Available technology varies widely in aspects including (1) hosted vs. host-yourself solutions, (2) online-only vs. hybrid (paper plus online), (3) University-focused vs. generic survey-focused, and (4) flexible question set vs. fixed survey format. Also, the amount of applied data analysis varies widely [24]. Some of the key vendors in the EOCE higher education market include Evaluation-Kit, eXplorance-Blue: and Gap Technologies.
In general, when evaluating acquiring EOCE technology, most institutions will consider key criteria such as ease of use, integration with learning management systems (LMS) and student information systems (SIS), who manages assessment deployment, collection, and analysis, can faculty add questions, etc.
Today, many higher education institutions equally evaluate EOCE technology based on how the produced course evaluation data could be analyzed, as not all evaluation systems support all potential evaluation methods equally well. For example, HEI ask if the EOCE software allows for: • Comparison of performance measures over time • Benchmark comparison against peer groups • Comparisons among units • Comparisons among faculty • Benchmark comparisons against national norms • Benchmark comparisons against institutions using the same evaluation system [28].

Methodology
While the importance of higher education technology is recognized, there is little data and research available about its effectiveness within the higher education and academic administration settings. The objective of this study is to offer a broad understanding of how one of the key higher education technologies, end of course evaluations, can be used to improve the academic outcomes of a university [13]. Researchers generally consider student evaluations of instructors to be highly reliable and at least moderately valid [29]. Moreover, since the data collection mode of semester-end of course evaluations changed from using paper surveys to employing web-based applications, little research has been conducted on further uses of this technology [24]. HEI have a need for statistical adjustments to compare course evaluation data across units [28].
Nowadays, a wide range of course evaluation systems (both paper and electronic) are available from several vendors. While these systems have common traits, their evaluation philosophies and the applied evaluation technologies vary widely. Evaluation technologies range from pure paper-based systems to hybrid systems that combine paper with web-based services, to online-only technologies. Evaluation philosophies differ depending, for instance, on whether the system is focused on supporting faculty development or on assessing faculty performance. Hence, selecting or designing an optimal course evaluation concept for any university is a non-trivial task that requires the careful consideration of many parameters [24].
This study was conducted over a four-year period between 2012 and 2016 using three-research methodologies: survey, in-depth semi-structured interviews, and two case studies. The survey mode of inquiry was employed to obtain data beyond the immediate environment of the researcher to provide insight into how other HEI utilized their EOCE. Ten in-depth, semi-structured interviews were held with academic administrators to gain deeper understanding of the usefulness of EOCE and two case studies were conducted, one in Embry-Riddle University and a second in Zayed University to validate and triangulate the results of the survey and the interviews.
The online survey was sent to more than 4500 students and 80 faculty members. Response rate was 88 per cent. The survey was comprised of twenty questions aiming to understand how EOCE were understood and utilized. The questions were concerned with ease of use, subjectivity, and the effectiveness of technology. The results of this study provide several key observations. Educational institutions can utilize the findings of this study to guide data collection, and analysis and measurement of courses and degree program metrics.

Discussion
In this section, we introduce the emerging analytics trends in the use of EOCE, and their possible use in improving academic assessment, teaching and learning, and academic administration decision making processes. As Figure 1 shows, all academic programs examined utilized sectioning. 90 per cent of programs examined utilized grouping. 45 per cent utilized query building, while 70 per cent utilized some form of instructor-level customization.

Sectioning and Categorization
This function allows HEI to turn EOCE from a single survey with a set of questions mainly about instructor's performance into several sections within the surveytargeting key desirable-measurable of the learning experience. For example, one section is devoted to questions about the instructor, a second section is devoted to the course (e.g. textbook, learning outcomes, etc.), a third section about the classroom experience, etc.
The data findings of this study reveal all programs examined utilized sectioning and categorization. Key advantages included helping program chairs in understanding and isolating key triggers of satisfaction or dissatisfaction within the learning experience. Academic administrators that attempted to maintain those processes manually were often faced with inconsistencies, redundancies, data errors, labor-intensive process, and higher maintenance cost.
In several cases, it was reported that because EOCE allows for large data sets, academic administrators had a higher confidence level in using EOCE input in course restructure, preparation for accreditation self-studies, annual assessment planning, and program reviews. EOCE technology enabled users to analyze a variety of curriculum maps used to illustrate relationships between courses and program outcomes or courses and skill acquisition and/or explore other useful relationships that can help make more sense of the degree program structure and assist with its assessment and review. Finally, academic administrators and instructors were also able to create better alignments between academic content and industry accreditation and certification standards.

Questions Standardization and Benchmarking
The data findings of this study also reveal that EOCE technology allows HEIs to standardize and benchmark questions and performance against other HEIs. For instance, The University of Toronto offers a question bank to users of eXplorance-Blue survey software. Questions are assigned to categories (e.g. questions concerned with critical thinking, global awareness, information literacy, etc.). Using EOCE technology, HEIS can benchmark their survey results to measure and understand areas of strength and areas of improvements of their students compared to other HEIs that participate in the same pool.
As one of the interviewees noted "The ability for me to see how my students are rating their understanding in areas like information literacy against how other students from other HEIs are rating their understanding information literacy is a huge plus that we could not do before. I reach out and collaborate with curriculum coordinator all over the world to improve key areas of students'' performance". EOCE technology can also align questions with key performance indicators and/or key learning outcomes. This alignment provides a key indirect measure that can be regularly and easily used to measure performance. The strength of these types of indirect evidence comes from the ability to collect and analyze a large number of responses.
For example, if 90 per cent of 4000 students don't believe that the university is providing proper resources in the library, then it is important for the university to appreciate that perception, even if it was not founded.
This function was reported as one of the most critical when choosing a new EOCE technology. HEI that are no longer satisfied with their survey technology or wishing to introduce new features have reported grouping as a critical functions. Survey technologies such as The Evaluation-Kit and eXplorance-Blue have worked with client to develop different grouping schemas to satisfy clients' needs. Using student's degree programs or course-program relationships, EOCE technologies can help HEI group and aggregate students'' response to the course, program, college, and institutionallevels.
This data is extremely helpful in revealing patterns, consistencies, and inconsistencies across courses, programs, and institutions, and across semesters. Examples of those patterns may include a consistent level of dissatisfaction with a specific course, or specific instructor, or a specific number of courses across programs, or a specific question across the institution. Those findings can be easily triangulated with findings from other data sources to provide validation as well. Academic administrators reported that the grouping function helped them see consistent feedback from EOCE that allowed them to make timely decisions in areas such as changes of textbooks assigned changes of assessment instruments, syllabi changes, instructors' reassignments, etc.

Horizontal Questions -Query Builders
EOCE technology also allows HEIs users to slice and dice data by looking at a single question or several questions across all surveys or across a specific population. For example, if a college dean is interested in understanding students' evaluations of course learning outcomes, s/he can isolate that question from the survey and group it across all surveys at the college level. In fact, some EOEC technology is so powerful and user friendly, they come with query builder ability that any user can easily utilize. Query builder ability allows users to build almost any kind of report out of the stored survey data. Users can choose specific courses, course sections, terms, delivery modes, instructors, etc. to build any report, and conduct their analysis.
Program chairs find the query builder tool useful when building reports to support outcomes assessment. Program chairs can choose courses that map to specific outcomes and group specific questions accordingly.

Customized Templates -Add on Questions
In addition to the new functions listed above, EOCE technology also allows HEI to build specific templates, and then allow each instructor to add custom questions when the survey is deployed. This approach affords the university standardization, and yet provides instructors with the flexibility to add specific questions that they wish to measure in a separate section. Those questions can be specific to an outcome, or specific to the instructor, or specific to an assignment. Some instructors still prefer to have some level of flexibility and control over the survey questions, even if it was partial. This option satisfies the standardization objective of many HEIs, while still allow the instructor to ask questions that could be more relevant to the specific course section or course.

Conclusion
HEI are starting to reap the recent advances in EOCE technology. Although HEI have always had access to EOCE results, they were mostly maintained individually at the instructor or the course section level. Now that technology allows HEI to group, drill down, slice and dice data, EOCE have taken a whole new dimension. Not only because of the large data set they represent, but also because they offer quantitative and qualitative data from two key data sources, students, and faculty. The data gathered in this study illustrates not only the availability of new EOCE technology, but also the value that could be gained in teaching and learning and academic decision making with the proper utilization of the new technology. This study reveals a number of new trends in the use of end of course evaluations that can help higher education institutions improve academic assessment, teaching and learning, and academic administration decision making. Those functions include sectioning and categorization; questions standardization and benchmarking; alignment with key performance indicators and key learning outcomes; and grouping by course, program outcome, program, college, and question-customization. End of course evaluations are now poised as an integral tool and a key health indicators of academic programs.
Indeed, all stakeholders in the higher education hierarchy should be able to take more complete ownership of educational processes by utilizing information about student success factors, the allocation of resources and effectiveness of teaching and institutional programs. These improvements, in turn, allow for real accountability and efficiency, more accurate measurement of the quality of learning and the raising of completion and retention rates, Performance prediction, attrition risk detection, data visualization, intelligent feedback, course recommendation, student skill estimation, behavior detection, and planning and scheduling are all resultant capabilities.