Evaluation of Engineering Students’ Learning Outcomes: Creating a Culture of Continuous… Evaluation of Engineering Students’ Learning Outcomes: Creating a Culture of Continuous Quality Improvement

Paper— Abstract— Assessment and evaluation of outcomes and continuous quality improvement often lead to uneasiness among engineering educators, which is a barrier to implementing an effective system of results-assessment. The sophisti-cated evaluation models and analytical instruments adopted by higher learning institutions are believed to worsen this uneasiness, impeding the successful implementation of programme quality improvement. In this paper, the challenges in assessing and evaluating learning outcomes, types of programme outcome assessment model and an analytical tool known as Engineering Outcome Analytics developed by Tunku Abdul Rahman University College for the assessment were presented. Documents related to quality improvement from two higher learning institutions in Malaysia were reviewed and discussed based on the key elements of continuous quality improvement. The results can be used to avoid the shortcomings and adopt the best practices in continuous quality improvement. With proper understanding on the key elements of continuous quality improvement, and adoption of culminating assessment model and a highly integrated analytical tool, engineering educators and programme owners are expected to benefit from this research in three ways. First, it cultivates a culture of continuous improvement of quality by increasing the educators' willingness to evaluate results. Second, it enhances their existing analytical tools to reflect students' actual develop-ments, resulting in significant continuous actions to improve quality. Third, it satisfies the accreditation requirements on outcomes-based assessment. Paper— Evaluation of Engineering Students’ Learning Outcomes: Creating a Culture of Continuous… Paper— Evaluation of Engineering Students’ Outcomes: Creating a of Continuous…


Introduction
Programme Outcomes (POs) are probably the most critical criterion for Outcomesbased Education (OBE) that emphasises on improving the intellectual skills and capabilities of graduates [1,2,3,4,5,6]. The Engineering Accreditation Council Malaysia (EAC) Programme Accreditation Standards [6] has included the assessment and evaluation of POs in the Engineering Curriculum to ensure that Continuous Quality Improvement (CQI) is carried out on a regular basis. Similar to the other engineering accreditation bodies in the Washington Accord [7,8,9], the EAC Programme Accreditation Standards [6] stated that engineering programmes seeking accreditation are expected to continually improve.
The EAC Programme Accreditation Standards [6] has listed three requirements on outcomes-based assessment. Firstly, the curriculum, teaching-learning activities, and assessment tools shall support the attainment of the POs. Secondly, attainment of the POs must be adequately assessed and used for improvements at the course and programme levels and finally, the engineering programmes must also show a high degree of stakeholders' involvement in the mentioned processes. The first requirement has strong relationship to constructivist philosophy which aligns curriculum, pedagogy and assessment [10], the second requirement on CQI is directly resulted from the first requirement on constructive alignment, and the final requirement is related to the existent of a faculty culture that supports the outcomes-based assessment system.
The key elements of CQI have been discussed in some of the accreditation standards such as ABET [8] and the Canadian Engineering Accreditation Board (CEAB) [9]. An engineering programme ought to evaluate the extent to which its POs are being attained [9] and utilise the evaluation results as input for programme improvement [8]. This input may include specific curriculum improvement or improvement in the achievement of outcomes or the assessment process. It is expected that such action(s) for improvement is clearly explained and supported by a clear justification [9]. The timelines and implementation plans are also expected to be established and monitored [9]. According to CEAB [9], activity-specific assessment results are often provided in the form of achievement levels. These indicate the levels of student achievement with respect to the assessment tool used, and typically expressed on point scale such as "Fails to meet expectations" and "Exceeds expectations". It is important that the CQI process is clearly documented as well [8]. These key elements could be used to determine the best practices or shortcomings in CQI.
The literature reports on the challenges of institutional-level assessment and evaluation of POs as early as the 2000s [1,11,12,13,14,15]. The challenges are categorised into two. The first category is more technical, focusing on constructive alignment and CQI in curriculum, and teaching and learning activities. The second category is rather subtle, involving all stakeholders' participation in the first category's processes. The challenges in these two categories, which set the objectives of the present study are presented in the next section.

The Challenges in Outcomes Assessment and Continuous Quality
The former President of the Accreditation Board for Engineering and Technology, Inc. (ABET), John Prados has discussed the impact and influence of ABET's accreditation standards on the Higher Learning Institutions (HLIs) in the United States [16]. He highlighted that outcomes assessment and CQI are foreign to academic experience and culture, leading to a high uneasiness level as it requires a significant investment of effort. However, he added that if such a system has been established, less effort is needed and limited to maintaining and continuing the system's operation. Researchers [17,18,19] reckoned that the greatest challenge to implement an effective outcomes-assessment system is the institutional culture of a faculty. It is widely recognised that managing resistance from educators is among the challenges faced by the HLIs [1,15]. Accordingly, for an assessment system to work effectively, educators ought to be willing to communicate their expectations on course contents, students' performance, and the resulting outcomes, provide and accept feedback, and improve their teaching and course contents to the input. Shared ownership is the first step towards evolving to the desired culture.
Cultivating institutional culture is the key to developing an effective outcomes-assessment system. When designing the outcomes-assessment system, other technical factors should also be considered. Prados et al. [16] reported that some HLIs misinterpreted POs' assessment and evaluation requirements and often failed to perform a meaningful analysis of the results. Consequently, those HLIs presented ambiguous plans on the utilisation of data for CQI on their programmes. Liew et al. [20] detailed the issues in assessing and evaluating POs faced by the HLIs in Malaysia. Despite the deployment of a comprehensive computing system by some HLIs in computing the attainments of PO, the authors reported that the results did not reflect the students' real outcomes. Furthermore, the level of details, namely the breakdown of marks for each assessment, is often too overwhelming for the educators to record. The situation is further worsened by the lack of understanding among the educators and HLIs on constructive alignment principles [20]. Mohammad and Zaharim [21] also reported that choice of assessment model and CQI of engineering programmes remained pressing issues whereby outcome attainments were not reflective of the students' real outcomes and resulted in vague CQI actions. The seriousness of the abovementioned challenges has also prompted some researchers [22,23,24,25] to explore the use of technology and assessment rubrics to overcome the challenges.
This study aims to identify the shortcomings and best practices in the CQI processes at the course and programme levels based on the culminating assessment model [20] and facilitated by the analytical tool developed by Tunku Abdul Rahman University College, Malaysia (TARUC). The shortcomings and best practices were identified through the key elements in CQI described in Section 1. The outcomes are expected to benefit engineering faculties through the readiness of educators to expand efforts on assessment and evaluation of outcomes, and create a culture of CQI among them. The following sections will introduce the culminating assessment model and the analytical tool before dwelling into the measurement and evaluation of outcomes.

3
The Engineering Programme Outcomes Assessment Models In general, there are two types of assessment model employed by HLIs in Malaysia, i.e., culminating and accumulating. The culminating assessment model identifies a list of essential courses that can demonstrate the attainment of POs [1,20,26]. The model requires a more straightforward computing system for the tabulation of PO attainments of its students, unlike the accumulating assessment model which uses all courses within a programme for tabulation. In the accumulating model, students' individual performance of the assessment components (such as continuous assessment, final examination, etc.) of each course corresponding to each outcome is entered into a computing system. This is repeated for all courses within a programme to compute students' attainments of POs at the point of graduation or completion of the engineering programme [20].
In this paper, the culminating assessment model of the Bachelor of Engineering in Communication Engineering offered by University A is discussed to further illustrate students' PO attainment. Figure 1 presents the list of courses offered in addressing some of the POs. There are two types of engineering course in this assessment model, i.e., enabling and culminating. A culminating course is a specialised engineering course that defines a specific engineering discipline and typically contains the pre-requisite knowledge of a few enabling courses. These enabling courses, on the other hand, are engineering fundamentals that are typically taught in the first and/or the second year of an engineering programme. Students' POs attainments are generally demonstrated through these culminating courses with underachievement in any of the POs could be easily traced and improved through the respective group of enabling and culminating courses associated to the PO in question. The culminating assessment model requires fewer data entries from the faculty members on their students' performance than the accumulating one because culminating courses are smaller in number [20]. In essence, the culminating assessment model uses a smaller number of courses, and the PO attainments are demonstrated towards the end of a programme. The advantages of this model include reducing engineering educators' workload and a more accurate representation of students' outcomes.

OBE Analytical Tool
An OBE analytical tool known as Engineering Outcome Analytics (EOA) was developed by the Faculty of Engineering and Technology of TARUC in 2015 with the aim of easing the administrative work among its faculty members. It also designed to fulfil two of the key elements in CQI that are to: facilitate the evaluation of the attainments of learning outcomes; and represent the attainments in the form of achievement levels.
There are two main highlights of the tool that differentiate it from the tools practiced by other HLIs. The first highlight of the tool is its integration of OBE measurement with the institutional administrative reports and documents rather than an independent system. To illustrate this, Figure 2 shows the list of institutional administrative reports and documents (labelled A to L) which was integrated into the EOA. These reports and documents, such as assessment plan, assessment moderation, students' performance reports, etc., are typically needed in most educational institutions. The faculty believed that with the integration of these documents and information into the tool, educators are more readily persuaded to expand OBE measurement efforts, as highlighted by Anagnos et al. [19]. The EOA is divided into two implementation levels based on the MS Excel VBA platform, namely the Course-Design-and-Analysis (CDnA) and Programme-Design-and-Analysis (PDnA). The choice of MS Excel as the platform for system development is motivated by its popularity, accessibility, affordability, and familiarity with the software.
The CDnA tool has evolved significantly. It was previously a user-friendly interface for outcomes measurement and automatic report generation for course outcomes analysis, integrated with the institutional moderation process. Currently, it is a highly integrated tool that eases institutional administrative mechanisms in facilitating the assessment moderation process and automatic generation of a range of course reports for quality improvement and accreditation records. The PDnA automatically populates and compiles data from various courses to automatically compute students' PO attainments at the programme level. It provides analytical charts and diagrams for students' PO attainments to identify CQI actions unique to each cohort of students. Besides, the PO attainments computed for each student are utilised by educators to guide the individual student for self-improvement. A function to batch email is also integrated to track their progress and informed them of their PO attainments through an official notification which is the second main highlight of the tool that reduces the administrative duties of programme owners. As a whole, the CDnA and PDnA formed the EOA to ease educators' efforts in both OBE measurements and administrative duties.

Conceptual Framework
In conceptualising this study's framework, the authors applied Bigg's Model of Constructive Alignment [10] as the requirements of outcomes-based assessment by EAC [6]. Aligned curriculum is one of the keys to the successful PO attainments by the students and reflective CQI initiatives that would improve a programme. The other requirement is on the importance of an institutional culture that supports assessment as an integral part of the curriculum described by Shaeiwitz [17] and Tener [18].
This study aims to identify the shortcomings and best practices in the CQI processes at the course and programme levels based on the culminating assessment model and facilitated by the analytical tool developed by TARUC. The key elements of CQI from the literatures [8,9] described in Section 1 were used to classify the practices as shortcoming or good. This is expected to benefit the engineering faculties in three areas. First, cultivating the culture of CQI among educators by increasing their willingness to expand efforts on assessment and evaluation of outcomes. Second, improving their existing analytical tools by reflecting on students' actual results, resulting in significant continuous actions to improve quality. Third, it satisfies the accreditation requirements on CQI. The conceptual framework for this study is as shown in Figure 3.

Methodology
The methodology deployed in this study includes document and literature review. Review of documents related to quality improvement of two HLIs (i.e., University X and University Z) that offer 4-year engineering degree programmes in Malaysia was performed to identify the shortcomings and best practices in CQI. In order to classify the practices as shortcoming or good, literature review was carried out to establish the key elements of CQI for comparison. The CQI practices were evaluated by the authors based on these established key elements. The process was complemented with TA-RUC's OBE analytical tool to highlight the shortcomings and best practices of CQI specific to outcomes-based assessments.

Results and Discussions
There are two levels of CQI that educators and programme owners can accomplish, i.e., course and programme [27]. The course level is generally performed by educators responsible for their respective courses. The programme level, on the contrary, constitutes several or more courses, is more complicated and executed by both educators and programme owners. The following sections provide the examples of CQI at the course and programme levels by highlighting the tips to overcome common shortcomings made by HLIs. Identification of appropriate CQI actions and the case studies from two HLIs are also presented. Finally, in each section, the example(s) are evaluated against the key elements of CQI.

CQI at Course Level
A common practice by many HLIs is to average the attainment of outcomes of all students for CQI, analogous to the traditional way of computing the average of marks of a course taken by them. However, averaging will potentially mask the better and poorly performed students, giving the average between the two groups. One method is to present the attainment of outcomes the same way as the normal distribution i.e., by categorising students' population according to their performance in each outcome described by CEAB [9]. Figure 4 illustrates the students' attainment of CLOs for the Engineer and Society course. 170 students from various engineering disciplines of University Z in the October 2018 semester were plotted using EOA. The bar chart indicates the percentage of the population of students at different ranges of CLO attainment. For example, for CLO1, 14.7 percent (25 students) attained below 50 percent. However, there is a need to understand the significance of those numbers before statistically analysing them. Some HLIs compared the attainments of CLO within a course (e.g., CLO1 versus CLO2 or CLO2 versus CLO3) which is entirely unnecessary. A quicker way is to look at the CLOs that need immediate attention while providing the maximum impact with the educators' minimum effort. For example, CLO3 and CLO1 are in the bar chart's red zones in Figure 4. It is more sustainable for the educators to identify the CQI action(s) that focus on improving these critical CLOs rather than having a long list of CQIs to improve all possible CLOs. These CQI actions may result in improvement in the CLOs in question which could be evaluated in the subsequent semester.

Fig. 4. Percentage of student population at different range of CLO attainment
Another common shortcoming is the educators' early provision of solutions without understanding the actual problem. Educators must identify the root cause of the situation through the principles of constructive alignment during the evaluation process. The following section illustrates the example of an acceptable CQI practice. The educator recorded an observation on CLO3 that reads, "CLO3 on energy issues is of main concern with 39.4 percent of the students did not achieve the outcome though it is an improvement from the previous semester which recorded 64.1 percent of underachievement. CLO3 was found to be aligned and solely relied on an examination question. Some issues faced by the students in answering that examination question include: weak on the "whys" CO2 emission is still rising despite the efforts by developed countries; and understanding of global issues which required further reading." The educator demonstrated an understanding of the root cause of students' poor performance of CLO3. He further identified that a formative assessment in the form of quiz to be introduced as CQI action to address the common issues faced by students for the following semester.
The above example from University Z fulfilled most of the key elements of CQI described in Section 1 [8,9]. The findings in this section are summarised against these key elements as shown in Table 1. Specific action(s) for improvement was mentioned (e.g., curriculum, outcomes, assessment process, etc.) Yesintroduction of formative assessment without significant increase in student workload

Action(s) is clearly explained and supported by a clear justification
Yes -the root cause of students' poor performance was identified 4 The timeline(s) and implementation plan(s) are established and carried through Yeshowever, follow-up action is yet to be observed The CQI process is clearly documented Yes

CQI at Programme Level
Two case studies were highlighted in this section to illustrate the poor and best practices of CQI. The list, number, and order of POs adopted by Universities X and Z are identical to the EAC Programme Accreditation Standards [6]. There are twelve POs, namely, PO1 Engineering Knowledge, PO2 Problem Analysis, PO3 Design or Development of Solutions, PO4 Investigation, PO5 Modern Tool Usage, PO6 Engineer and Society, PO7 Environment and Sustainability, PO8 Ethics, PO9 Teamwork, PO10 Communication, PO11 Project Management and Finance, and PO12 Lifelong Learning.
The first case study is University X's model of accumulating evaluation. Students' performances in Year 1, Year 2, Year 3, and Final Year are averaged for each academic year to contribute to the achievement of POs at the end of the programme. Figure 5 shows the achievement of graduating students' POs for two academic years, 2013 and 2014 with 33 and 34 students respectively in the Mechatronic Engineering degree programme. The performance target set for the programme is 75 percent. It is clear that the programme demonstrated improved performance for all programmme outcomes in 2014 (shaded in grey) compared to 2013 (colored in black) and well above the target (75 percent) in the two academic years ( Figure 5).

Fig. 5. Programme outcome attainments of graduating students in 2013 and 2014 -University
X's assessment model The observation made by University X on the attainment of POs reads as: "The contribution of most courses to the attainments of programme outcomes is very good with a small number of courses contributing less than 75 percent to most programme outcomes. To improve the performance of the programme outcomes, the CQI of these small number of courses need to be carefully monitored." The statement suggests that the programme will continue to monitor "the small number of courses" as one of its CQI actions. However, the second CQI action highlighted the need to "Organise leadership talk series and leadership programme to improve students' soft skills" indicating a need to improve students' leadership and communication skills. The CQI on the leadership programme was non-reflective of the attainment of POs (see Figure 5), demonstrating University X's failure in identifying the relevant CQI action based on the collected data. For the record, Figure 5 shows that most of the students performed very well in communication and teamwork skills (PO9 and PO10) but scored poorly in applying engineering knowledge and analysing problem (PO1 and PO2).
The second case study is based on a culminating assessment model practiced by University Z. The culminating assessment model adopted by University Z identifies a list of selected courses to determine students' attainment of POs at the end of a programme. Figure 6 shows the achievement of the POs for 47 students in the final year of the Electrical and Electronic Engineering degree programme for the October 2019 semester. The percentages represent the number of students at the four classifications: within 0 to 29 percentnot attained; between 30 and 49 percentweakly attained; between 50 and 74 percent -attained; above 74 percentstrongly attained for each PO. It can be observed that the majority of the students (over 74 percent or above, coded in the green zone) strongly attained the POs on PO8 Ethics and PO10 Communication. However, some students (below 50 percent, coded in the orange zone) weakly attained the POs on PO4 Investigation and PO7 Environment and Sustainability. Coincidently, the two POs are among those which recorded the lowest number of students coded in the green zone. Subsequently, a meeting among the educators and programme owners was held to discuss the poor performance of these outcomes and to identify the appropriate CQI actions. In an open-ended approach, the problem may have multiple solutions, and there is no obvious solution. The evaluative criteria in these areas need to be established in the assessment rubrics." In this regard, literature [28] reported that open-ended laboratory focuses on student's ability to design experiments, identify the variables or results or information to be collected and determine the appropriate instruments for the assigned problem. These observable outcomes can be incorporated in the assessment rubrics.
The second CQI action on the weakly attained programme outcome for Environment and Sustainability read as follows: "Students' performance on environment and sustainability correlates with their performance in the Engineer and Society (shown in Figure 4) and Renewable Energy courses despite their reasonable performance in the integrated design project. The committee identified that the two courses' corresponding assessments have to be redesigned for the following semester." These two courses emphasise the engineers' responsibilities, namely, knowledge in protecting the environment and sustainable engineering solutions as described in literature [29].
The above examples demonstrated that the culminating assessment model would result in CQI actions that reflect students' PO attainments. Besides, the achievement of outcomes that is well-classified by identifying the number of students under each classification (i.e., not attained or attained) allows for meaningful CQI actions.
The findings of the examples from University X and Z are summarised against the key elements of CQI described in Section 1 [8,9] in Table 2. The CQI process is clearly documented Yes Yesdiscussion for improvement was carried out at the department level

Conclusion
The examples discussed in this study have resulted in two important observations against the key elements of CQI. Firstly, programme owners and educators must ensure that the recommended CQI actions are reflective of the attainment of outcomes and supported by clear justifications otherwise, the exercise of data collection will be wasted. Secondly, the attainment of outcomes is recommended to be in the form of achievement levels to determine the extent to which the outcomes are being attained. By presenting the attainment of outcomes the same way as a normal distribution, categorising students' population according to their performance in each outcome, can identify the results that required immediate attention.
The study has also highlighted that educators should focus on a few CQI actions that are more meaningful and less exhaustive compared to a long list of efforts to improve all possible outcomes. In addition, educators should avoid jumping into a solution too soon. Instead, they should recognise the students' problems during the evaluation process to find the most effective CQI actions.
Lastly, this study has demonstrated that the culminating assessment model effectively drives CQI on engineering programmes with the educators' sustainable effort. An effective outcomes-assessment system should focus on selected courses or assessments that promote constructive alignment where educators are more readily to expand efforts on assessing and evaluating outcomes.
CQI is a method that strives to ensure that the engineering graduates meets the desired outcomes, optimised with a sustainable effort from the educators. CQI is an operational practice, hence it must embrace everyone in the system. Creating a CQI culture among educators is a priority that should begin by identifying the key elements of CQI followed by adopting the appropriate assessment model assisted by an integrated analytical tool. These would result in evaluation practices that promote CQI culture, benefitting internal and external stakeholders, from HLIs and educators to students and their future employers, and fulfil accreditation requirements. Dr. Marlia Puteh is an Associate Professor at Faculty of Social Science and Humanities, Universiti Teknologi Malaysia. Her areas of expertise transcend into multidisciplinary research work including higher education policy research, engineering education and web-based teaching and learning. She was a visiting professor at the UNESCO Chair for Problem Based Learning in Aalborg University, Denmark where she expanded her research inquiry by benchmarking the European accreditation system with the Malaysian practice. She has published widely and her publications include policy documents, refereed journals, conference papers, book chapters and academic books in the domains of higher education policies, engineering education research and technology in language learning.
Dr. Lim Li Li received her Advanced Diploma in Electronics Engineering from Tunku Abdul Rahman University College, MSc in Electronics from Queen's University of Belfast, UK and PhD in Electrical and Electronic Engineering from University of Nottingham Malaysia. She is currently the Dean of the Faculty of Engineering and Technology in Tunku Abdul Rahman University College. She is a member of the Engineering Technology Accreditation Council, Board of Engineers Malaysia Programme Standards Committee.
Dr. Yu Lih Jiun is an Assistant Professor at UCSI University and an award recipient in teaching excellence in the university. She is an active member in the Institute of