Evaluation of an Academic Program: The Case of Computing Accreditation Commission Framework in Higher Education

This study aims to present the ABET-CAC accreditation framework and discusses how to manage program’s constituencies of an academic program in higher education. It describes the program’s associated entities and the methods of creating required evidence for the program accreditation in computer science education. It follows a mixed method to evaluate and measure the program performance. It discusses the program’s performance measurement methods and its analysis. It emphasizes the program’s entities such as students learning outcomes, course assessment and performance evaluation. It shows a scientific approach that measures the program’s performance. It suggests the evidence should be considered as important indicators for both when planning the quality improvement for the program delivery and program’s benchmarking. For any program accreditation in computing education, the framework needs to be followed at least for two years to ease the program’s review process. This will help prepare competently for the accreditation, ahead of program’s review visit by the commission. Keywords—Accreditation framework, computing education, information systems, assessment and evaluation, academic program, higher education


Introduction
Since 1985, the Computing Science and Accreditation Board has been accrediting many programs [1] on the compliance with accrediting criteria. Accrediting bodies have been helping in defining the professional fields and recognizing them as academic programs for career profession. Also, accrediting bodies are working with organizations such as UNESCO which promotes international collaboration in higher education [2].
Since 2003, the Arab states have been emphasizing quality assurance in higher education [3]. In Saudi Arabia, the Ministry of Education, which governs all universities (" Higher Education," n.d.), is currently supporting accreditation for the academic programs in higher education. The ministry also promotes accredited programs as quality standards for national and international rankings. Generally, ABET accredited programs attract the best students in the region.
Since long in this region, sincere efforts have been delivered for quality improvement, attracting international recognitions, and professional accreditations of academic programs. Undoubtedly, quality education has taken major importance and more demanding in labor market. This motivates universities emphasize on quality education and get recognition internationally by accrediting their academic programs. For computing education, Accreditation Board for Engineering and Technology (ABET)-Computing Accreditation Commission (CAC) is a reputable accreditor and the accreditation by this agency is highly regarded, globally [5]. Program accreditation recognizes the potentiality of the program such as the graduates' ability in their professional life [6].
Since the inception of the program, the department's goal is to provide an efficient learning environment that will enable its graduates a professional competence. Program accreditation is one of the ways that recognizes the quality of the program. Accreditation by the agencies is based on commitments to certain core values [2]; the most important one is to assure threshold quality in higher education [2]. Basically, accreditation is a process of the program review and acknowledges the quality of the program by an external agency. It is also considered as the program has been labeled quality and attains threshold-quality potentials (University of Arkansas, 2014). An accredited program has significant identity among the students and overwhelming response for the program enrollment.
The aim of this study is to provide a systematic approach for the preparation of ABET-CAC accreditation. It describes how to manage program's constituencies, academic activities, documents and evidence, which are required during the program review by the accreditation commission. Definitely, the suggested approach eases the preparation process for the program accreditation. It facilitates both the methods of developing required evidence and arranging of infrastructure for the program review. Significantly, this study fills the literature lacuna for managing the program's constituencies and the methods for developing required evidence that hasn't been focused, yet. The approach has been applied successfully for the program accreditation. It should be adopted for a successful accreditation of any academic program in computing.

Contributions and outline
The study introduces a novel approach that explains how an academic program's constituencies have to be managed for a successful program accreditation. The original contributions of this paper are: • The methods of managing program's constituencies and the framework (Section 4). • Sample of program tree i.e. logical relationships among program's entities.
• Documents and evidence in course file (Table 14).
• SLOs measurements in program's skills.
• An effective approach to closed assessment loop (Section 6).
• Emphasis has given to both SLOs measurement benchmarking and the required infrastructure for the program's delivery and review (Section 7).
The paper is completed by Section 1, indicates the importance of computing accreditation in higher education and the purpose of study. Section 2, presents the related work in literature review. Section 3, presents the study environment and the adopted methods. Section 5, discusses course assessment and evaluation method. Finally, we present the study's challenges in Section 8, and conclude.

Literature Review
ABET accredits programs for both bachelor and master's degree, in 40 disciplines of computing, engineering, engineering technology and applied science [8]. The accredited programs meet the quality standard and capable of producing skills graduates [9]. An accredited program is a significant attraction for the students' enrolment. [2]. The search for the methods of developing documents and evidence related to program accreditation is laborious; since, the work on required documents and evidence for the accreditation aren't available with several publications. Thus, this was very difficult to find such approach across several studies.
An open federated search of multiple publishers including Elsevier/ScienceDirect, SAGE, Taylor and Francis, Emerald, ACM, and IEEE transactions was conducted. We could find the work for accreditation [6], accreditation criteria (Iqbal Khan, Zahid, 2016), program education objectives (Fitzpatrick, & Kennedy, 2009), learning outcomes [11], assessment [12], evaluation and similar work [5]. However, to the best of our efforts, none of the studies have covered the management of program's entities and the methods of generating evidence for the program accreditation. These entities are crucial for the program review for the accreditation. This motivates us to develop a framework and includes essential evidence that facilitates the accreditation aspirants [5]. Our study provides an approach to manage required activities, documents and evidence which are essential for the program review [1]. It also provides a systematic approach to present the case, competently to the accreditation commission.

Study Environment
In this section, we discuss the study's environment. The study has been evolved in a real time approach at the college of computer science in a university environment. All the methods and activities discussed in this study have adopted for the successful accreditation of the academic program in computing education.
The authors are the faculty members at the college and have additional responsibilities of managing the program's constituencies. It includes both documents generation and evidence management for all the activities which are required for the program accreditation. In the process of preparation, three committees of faculty members were formed and the required work was assigned to them. Many coordinated efforts were required during the preparation to achieve a successful program accreditation.
The authors were active participants and heading the committees exclusively formed for the program accreditation. We have been involved in every activity such as developing Program Educational Objectives (PEOs), deciding Students Learning Outcomes (SLOs), adjustment of program associated entities, integrating assessments and evaluation data, developing Key Performance Indicators (KPIs) for measuring the program performance, developing summary reports [5], documenting associated evidence, arranging and organizing the relevant materials in the preparation room [13]. Moreover, the adopted methods include analysis of evidence, interpretation of course assessment result, and action plan for the accreditation and program quality improvement, too.

Process to be followed
The institutions and the programs seeking accreditation must follow the guidelines which are easily available on the ABET's official website [13]. The review process is typically twenty months long with eight steps to be followed. The study doesn't emphasize the general guidelines such as program criteria and self-study report (SSR) [8].

Accreditation Framework
The framework discusses how the program's constituencies and their associated entities should be managed for a successful program accreditation. ABET is an organization that is recognized by the Council for Higher Education Accreditation (CHEA). It is a non-governmental organization in the United States; solely accrediting academic programs in computing [9], [5]. ABET-CAC accreditation means the process of continuous improvement of an academic program in computing education [1].

Program associated
An academic program is the study defined by any combination of courses or sets of academic requirements that leads to a degree, which the university is authorized to offer (Academic program, 2014). A program can be realized by its hierarchy of associated entities or through its graphical abstract i.e. program tree shown if Fig 1. PEOs: According to ABET criterion 2, PEOs need to map with the university mission, correlate with the SLOs, and have associations with other program's constituencies [8]. ABET describes PEOs are as broad statements which describe what graduates are expected to attain within a few years of graduation [5]. PEOs need to be reviewed periodically to meet the professional requirements of the time [6]. Table 1 lists PEOs, Table 2 shows program's mission key-words map to PEOs, Table 3 shows mapping between SLOs and PEOs, and Table 4 lists SLOs.   Table 3. Sample of mapping PEOs to SLOs

ABET code Students Learning Outcomes for Information Systems program
a An ability to apply knowledge of computing and mathematics appropriate to the information systems discipline. b An ability to analyze a problem, and identify and define the computing requirements appropriate to its solution. c An ability to design, implements, and evaluate a computer-based system, process, component, or program to meet desired needs. d An ability to function effectively on teams to accomplish a common goal. e An understanding of professional, ethical, legal, security and social issues and responsibilities. f An ability to communicate effectively with a range of audiences.
g An ability to analyze the local and global impact of computing on individuals, organizations, and society. h Recognition of the need for and an ability to engage in continuing professional development. i An ability to use current techniques, skills, and tools necessary for computing practice. j An understanding of and an ability to support the use, delivery, and management of information systems within an Information Systems environment.
Program tree: A program tree, shown in Fig 1, is a graphical representation of the relationships among the program's associated entities. It represents a hierarchical mapping of key words from the university mission and the key words from the college and the program mission. It also shows the mapping between PEOs and SLOs. At the end it represents PEOs, SLOs, domain codes, and their key words [8].
Program tree has to be displayed clearly in a poster size at a significant location in the department's premises. This will help the stakeholders to understand the program's hierarchy and the logical relationship among its entities [3].
SLOs: ABET-CAC defines a set of learning outcomes for computing programs [8] [14]. Beside, one or two learning outcomes exclusively defined for each program types such as information systems i.e. SLO-j, computer science, and information technology [8]. The SLOs (a-i, shown in Table 4) defined by the ABET are common for all computing programs. Similarly, some institutions seek accreditations from other national and international organizations, where SLOs need to be categorized into learning domains [15], shown in Fig 1. Most importantly, SLOs must be realistic, attainable, measurable, and periodically (4-5 years of time) have to review for the continuous improvement of the program. It is advisable to take expert feedback on SLOs from academia and industry. Ultimately, SLOs are the reflections of the skills learned from the program content [6].
Curriculum: ABET defines a curriculum is the fundamental requirement under program criteria. Program criteria are discipline specific and implemented by the program title [8]. Curriculum is the most essential discipline of a program constituencies, this should be designed [16], inline to the both PEOs and SLOs. The curriculum should be specified in subject areas according to the accrediting agency [8].
Mapping curriculum vs. SLOs: Generally, a program's curriculum has to be categorized into three levels, introductory-I, proficient-P, and advanced-A [15]. With respect to ABET guidelines the courses are required to be labeled as I, R, and E [17].
Introduced-I: This category represents basic courses of the curriculum. Students need to be familiar with the technical knowledge of these courses.
Reinforced-R: this category indicates advancement in the first category and with increased learning objectives. These courses enhance students' learning skills, strengthen the knowledge, and minimize learning complexities.
Emphasized-E: courses under this category represent the program's learning activities and correspond to three learning skills cognitive, interpersonal and communication.  Table 5 represents the potential courses correspond to each SLO. Students' performances in these courses are measured applying the associated entities which correspond to each SLO. It also describes the associated entities which are used for SLOs performance measurement.

Assessment and Evaluation
For the performance measurement, assessment and evaluation and their evidence are inseparable process for the ABET accreditation. The evidence of assessment and evaluation (Hussain, & Mathew, 2017), SLOs measurement and program evaluation reports are crucial for the program review. These are the stepping stones for the initial accreditation of the program [19]. Furthermore, the outcomes of these activities must be documented and should be utilized [20] for the program improvement, skills development, decision making and logistic planning [13]. Fig 2 describes a cyclic process of program performance evaluation. From the top, it shows the courses are categorized into three program's learning skills. Then, it shows the process of course assessment and evaluation. And the outcomes of course assessment are documented in a folder say, course file. Next, it shows the grouping of courses measurement in program's learning skills. Later, these measurements are integrated to learn the overall program performance. Finally, the performance report is considered when making the program's action plan for the quality improvement [21]. This assures improved performance in the next cycle of program delivery.

Assessment planning
At the beginning of each semester the assessment schedule is planned. The details of course assessment methods are described in the course specification of each course. Course coordinators have been responsible for conducting the assessments and submitting the performance reports to the head of accreditation committee. These reports have to be integrated for SLOs measurement, discussed in Section 5.2.

SLOs measurements scheme
Generally, ten to twelve courses have been selected from the curriculum, necessarily of higher level and should be the core courses. By, following the ABET guidelines, the SLOs have to be measured into three sets of learning outcomes [5] shown in table 6. The table shows SLOs is distinguished into three sets of learning domains: first, technical knowledge [11], second includes cognitive, interpersonal, communication and lifelong learning. The third set is related to behavioral skills, such as ethical, social and professional responsibilities.

Measurement process
In the following subs-sections, it is explained how a SLO should be measured. In this case, we measured students' performance in one of the courses (493ISM, shown in table 5) selected for SLO-a. The performance is measured using KPIs. These KPIs are predefined for each SLO and described as a set of well-defined rules called rubrics [22], shown in table 9 and the measurement in table 11.
Course assessment approach: The course assessment is based on the mapping between course learning outcomes (CLOs) and the SLOs, shown in table 7. Necessarily, each of the CLOs (first column of table VII) has to be mapped with any of the SLO, which is described in the course specification of the selected course. The students' performance in this course is measured using KPIs evaluation [22] shown in table 9. Latter, the measured outcomes have to be integrated with other course(s) measured to obtain the overall performance measured for SLO-a. Similarly, the whole process repeats for every course selected for the SLOs (a-j).  Tables 8 & 9 should be considered together, which show the course assessment details. Table 8 describes the CLOs (1.1-1.3) and the mapping with SLO-a. The course performance is measured applying KPIs shown in Table 9. It shows the KPIs (a1-a3) evaluation description, assessment method and assessment type.    Table 10 represents the sample of single student's performance in the assessment, applying KPIs evaluation, shown in table 9. Table 10 shows the performance of single student in all the KPIs (a1-a3). It also shows the performance level, 'S-satisfactory i.e. >=4<=5', 'D-developing i.e.>=3 ', and 'U-unsatisfactory i.e. <3', and the student's obtained grades (ü). Similarly, the assessment performance is recorded for all the 14 students for KPIa1, shown in table 11. Similarly, the process repeats for KPIa2 and KPIa3 & recorded in table 12.
Scientific approach: For measuring the students' performance in the course, both tables 10 & 12 should be used and listed the performance. This can be achieved with additional calculation on the marks obtained in the assessment. Table 11 shows the arbitrary values derived from table 10 and table 12. The overall performance in the assessment can be measured using both the equations 1 and 2.   Table 12 shows the numerical performances of all the 14 students in the three KPIs a1, a2 & a3. These values (performance scale 5) have taken from the course assessment that students obtained against each KPI. At the bottom, it shows the KPIs overall performance measured in the course for SLO-a. from table XI 1 = 3.09 , is the overall performance measured for (KPIa1) of 14 students Similarly, students' performance is measured for 2 & KPIa3 2 = 3.21, similarly obtain and also shown in table 12 3 = 3.50, similarly obtain and also shown in table 12 ′ is the number of KPIs = (3.09 + 3.21 + 3.50)/3. = 3.26, is the overall performance measured in single course, for SLO-a.  Table 13 shows the sample course report based on the assessment outcomes. It shows the numerical values from table 12 have interpreted into meaningful information. Similar report is made for every course selected for SLOs measurement. Further, these reports will be utilized for the overall program performance report. The assessment's result and the samples of students' work should be arranged in a folder, say course file, shown in table 14.  1. Few students need to understand the concept, models, services, and the essential characteristics of cloud technology. 2. Some students must understand the organizational needs of IT resources & various aspects. 3. Some students should learn the importance of cloud adoption in an existing environment.
1. Students should study following the guidance. Must spare more time on given assignments. 2. Students should visit an IT center to understand the resources and its efficient consumption. 3. Students should take an assignment of analyzing the existing resources in terms of cost-benefit while suggesting cloud adoption.
Course file: Course file is the set of processed documents of a particular course, developed throughout the course delivery in a semester. At the end of the semester and on course completion, the responsible course coordinator submits the course file to the preparation committee. The course files should be placed in the preparation room. Course files should be available in both soft and hard copies. The whole process repeats for each semester. Table 14.
A sample of course file index

Measurements in program's skills
The SLOs have to be measured for the program's skills and the scheme is shown in table 6. The measured outcomes of all the SLOs have to be grouped into program's skills, which is described in tables 15, 16, and 17 [8] . A summary report for SLOs measurement has to be prepared, same as of the course shown in table 13. Based on the summary report, a program action plan should be prepared to overcome the weakness, when delivering the program for next cycle [10]. Necessarily, all the SLOs have to be measured at least once throughout the program evaluation process [15]. The measured outcomes should be benchmarked and described in table 18.     The assessment of students' learning and the performance evaluation are the major aspects of the program's review for the accreditation. This facilitates to understand how much skills learned of the programs' content by the students [5] [12]. It is also equally important to attain intended learning outcomes (SLOs) while seeking accreditation [6]. To achieve this, we developed a cycle performance analysis, called, closed assessment loop (CAL) shown in fig 6. It involves the program's associated entities: CLOs, KPIs, assessment, measurement, evaluation, recommendations, and suggested actions. The CAL effective implementation assures continuous improvement and facilitates to attain the SLOs [24]. The cycle should apply to all the selected courses (shown in table V) for SLOs measurement. On the basis of CAL analysis, modifications should be made in the program's entities; if they are required. Necessarily, the

SLOs Measurement for Behavioral Skills
CAL process needs to be documented at least for two years [8] for the program's initial accreditation.

Program evaluation
The program's performance should be evaluated on the outcomes of SLOs' measurement (Chaiyaphumthanachok, T, & Sujiva, 2016). Besides, some exclusive KPIs should be identified to evaluate other features of the program, such as facilities. These KPIs have to be measured at least once in a cycle-time (i.e. four years) [15].

Surveys and feedback
For the initial ABET accreditation, a lot of students' surveys about the program performance have to be conducted at the various levels of program delivery. The surveys' results have to be analyzed and documented for the commission's visit. Besides, ABET requires faculty's feedback who are involved in program delivery [8]. It is also required feedback of program's representatives from the industry and academia about the program's constituencies. These feedbacks provide what the participants felt about the program and facilitate the administration to make the program an embodiment of quality standard [17].

SLOs Measurement Benchmarking
One of the committees is solely responsible for integrating the SLOs measurements data into a single report. The committee analyzes the data and interprets into meaningful information. Finally, the committee sets the benchmarks, shown in table 18.

Presenting the reports
Presenting efficiently the final reports to the visiting commission during the site visit is a crucial aspect of the program review. Generally, the reports are represented in both tabular and graphical. We have adopted the interactive data visualization approach that updates automatically any data occurrences. Many software tools and applications freely are available [26] for academia that can be used to present the case. Even, Google provides similar open source tools (Google Fusion Tables, 2017) that allows computing operations on data tables, facilitated with interactive charts, and accessible in mobile environment, too [28].

Infrastructure
The essential infrastructure includes both static and operational for the program delivery. The department has to provide sufficient number of class-rooms, commonroom, equipped computer-labs, library with necessary resources, wash-rooms and similar facilities. Necessarily, all the documents and potential evidence must be arranged in the preparation room. Besides, course files for previous two years, required text-books, data show, computers with internet access, printer, and other facilities should be arranged.

Challenges
The program's accreditation review process must be supported by the substantial amount of documents and the evidence which should describe the continuous improvement in the program delivery. Managing the necessary documents, including minutes of the meetings of all working committees is very challenging (ACM Digital Library., & Pittarese, 2002). The summary reports for the activities have to be simple and concise as the reviewers willing to check succinct evidence. The other challenging task is to adjust the program entities and update the curriculum without majorly affecting the existing one. The curriculum has to be updated inline to the ABET program criteria. The significant challenge we have faced the coordination among different individuals such as head of different committees, course coordinators, teachers, administrative staff, and other stakeholders. The accreditation process requires coordinated efforts. Truly, all the authors of this study participated as head of designated committees and actively involved in every activity from the start to the successful program accreditation.

Conclusion
Our study contributes for the initial accreditation of an academic program in computing education. It provides the process of managing program's constituencies and framework for developing the required documents and evidence. It describes the potential methods for course assessment and the samples of evidence which are vital for the program's evaluation process. The most significant part of the study is the approach to SLOs measurement and the benchmarking of program performance. It also facilitates the faculty members and the administrations understand the process of continuous improvement of an academic program in higher education. The study presents a systematic approach that optimizes the accreditation provision and places the efforts in appropriate direction. The adopted approach has been well appreciated by the commission. This study should be an encouragement to the ambitious institutions that intends to seek accreditation for their bachelor program in information systems. Undoubtedly, the approach can be adopted for any academic program in computing education with an increased chance of getting accredited by the ABET-CAC commission.