An Approach to Evaluate Program Outcomes and Program Educational Objectives through Direct and Indirect Assessment Tools

The model of outcome-based education is based on achieving attainments at the end of each course by the students in any undergraduate or postgraduate program. It was implemented in all technical institutions of India as per the guidelines of All India Council for Technical Education, India. The attainments are calculated by deploying some direct and indirect tools. This include courses results, placements, projects and various surveys like alumni, employer etc. The paper discusses the attainment of Program Educational Objectives and Program Outcomes for any undergraduate or postgraduate program. In outcome-based education, certain targets are to be set on the basis of previous year performance of students and these targets are achieved in the form of attainments. In this research, a comparative study of last three batches of a postgraduate course is done in the form of attainments. Keywords—outcome-based education, course outcome, program outcome, program educational objectives


Introduction
The outcome-based education is based on the assessment of students through attainments on the basis of certain defined targets.The model can be implemented on any undergraduate or postgraduate course.The faculty members are acting as a facilitator and mentor to the students for achieving the targets.The assessments are evaluated in form of attainments of Program Educational Objectives (PEO) and Program Outcomes (PO).
The model begins with formulating Vision and Mission of the department in aligns with Vision and Mission of Institute.The PEO and PO of a program are defined in accordance to Vision and Mission of the department and as per twelve National Board of Accreditation (NBA) graduate attributes.The PEO are borderer statements defining the objectives of any undergraduate or postgraduate program.Now as per the university curriculum we have to define different course outcomes for each course or subjects.The courses include theory subjects, practical subjects and projects.Similarly, we have to define course outcomes for various surveys like alumni survey, curriculum survey, exit survey etc.The course outcomes are written in the form of statements describing the outcome of a particular course.A mapping of Course Outcome and Program Outcome is established with strong, medium and weak correlations.The target for each course can be set, which has to be attained.The target can be set on the basis of result of previous years.i.e. the performance of various batches of graduated students.
The Vision, Mission, PEO and PO are then approved by stakeholders like management, alumni, industry persons and Board of Studies (BoS) before implementation.It can be discussed and communicated to the students or learners.
A systematic process is followed for finding the attainments of each course including surveys.The Attained value of Program Educational Objectives and Program Outcomes are calculated.A gap analysis is performed to check the attainment against targets, if it is not achieved then gaps are identified.The gaps are fill by redefining targets, revising Course Outcome, conducting remedial classes for the weak students or by giving extra assignments to the students in the same academic year or consecutive year.

Background
In the review study the authors stated that outcome-based education is widely practiced by educational institutions from primary schooling to degree level.The outcome-based education is helpful to both educators and learners.It can be used in the designing of curriculum [1].Vivek [2] discussed how the poor practices of current educational system can be overcome by outcome-based education.The Course Outcome is designed for a subject and minimum target is to be set.If a student attains that target then only that course is completed.A gap analysis is to be performed at the end of each course to check whether targets are achieved or not.Jayashree [3] expressed the fact that the curriculum must be measured in terms of outcomes and at the end of each course students has to achieve the outcomes.The curriculum may be designed in way to meet the challenges of industry to make the students more market ready.Subbraman et al. [4] distributed the contribution of course study as 80% and graduate exit survey as 20% for the attainment of Program Outcomes.In the attainment of Program Outcomes an additional weight factor is associated as per AICTE curriculum model.Aziza et al. [5] evaluated the effectiveness of learning outcomes from student's perspectives.The results show that learning outcomes is to be reviewed.As a result of this curriculum is redesigned by introducing new courses and changing teaching approach from surface to deep learning.Oriahand et al. [6] studied the performance of students in some course and find that there is significant difference in outcome-based education and non-outcome-based education learning.The outcome-based education mean grade point average is higher than non-outcome-based education.Rajak et al. [7] discussed the process for the attainment of Program Educational Objectives for post graduate courses.A comparative study of attainments was made for different batches.The outcome-based education starts from defining Vision and Mission to the department to the attainment of PO and PEO [8].The assessment process is used to calculate the attainments of course and Program Outcomes.These attainments can be used to find how much the assessment process fulfills the criteria of accreditation body [9].Marks et al. [10] discussed how the course evaluation can be used to improve academic programs.They conducted a study based on 23 academic programs.Uziak et al. [11] highlighted about problem-based learning, which is an effective tool in primary and secondary education.It can also be used in degree level courses.They implemented it on mechanics course of mechanical engineering.

Direct and Indirect Assessment Tools
The attainments are calculated by deploying direct and indirect assessment tools.The direct tool comprises of class tests, assignments, tutorials, projects, placements etc.The indirect tools include various surveys like exit survey, alumni survey, employer survey, curriculum feedback etc.In the process of calculating the attainments the outcomes of these tools are designed by writing the outcomes in few statements.These are known as Course Outcomes.The department may run different programs but program outcome for each program has to be separately designed.The PO are based on twelve NBA graduate attributes.There can be separate Program Outcomes for engineering and management programs.
The mapping of Course Outcome and Program Outcome is established in matrix form having 3 (strong), 2 (moderate) and 1 (weak) correlations as shown in Table 1.The empty cells in CO-PO mapping shows no correlation.The Program Outcomes are attained through direct and indirect tools.The different assessment tools are shown in Figure 1.

Direct tools
The direct tools are the assessment techniques related to the performance of students in different examinations.These examinations include class tests, assignments, practical, projects and performance in end semester examination.The direct tools also include placement data.A weightage to results, project and placement is to be given for calculating attainments.In our research we gave 50 percent to result and 25 percent each to project and placement.
Internal and External Exam assessments of a course is carried using class tests and end semester examinations.The students are also given regular assignments and they have to submit these assignments from time to time, which are properly evaluated by the subject teacher.The assessment of a course is based on the attainment of Course Outcomes and external average in university exam.The Course Outcomes of DSA-RCS406 and its mapping with Program Outcomes is given in Table 1 and 2 respectively.The Course Outcome may be defined unit wise or on the basis of delivery of contents set by the teacher.Here the Course Outcome of DSA-RCS406 is defined as unit wise and the course has five units.

CO1
The fundamental concept of data structures, algorithms and will be familiar with recursive functions.

CO2
The concept of linked list data structure and implementation of stack, queue etc.

CO3
The non-linear data structures like tree and graph.CO4 The complexities and implementation of sorting and searching algorithms.

CO5
The various graph algorithms such as shortest path and minimum spanning tree.Projects Evaluation are the assessment of projects which the students have developed during the tenure of their degree.The projects are evaluated and the project assessment is included in calculation of attainments.The evaluation is based on quality of project, technology used, project presentations and its documentation.
Placement Records are also included in calculating attainments.The placement assessment is based on percentage of placement of a particular batch and average package of students.

Indirect tools
The other tools used in the attainment process are indirect tools.The indirect tools include surveys from different stakeholders like graduated students, industry, parents etc.The survey can be carried out online or offline by filling feedback forms.A good sample size can be taken from surveys and can be used in attainment process.
Alumni Survey is conducted through alumni members of a particular batch.All the alumni who have graduated from the institute are requested to give feedback.This feedback can be used by the Institute for reviewing the academic process so as to enhance the quality of imparted education and as well as in the attainment process.
Exit Survey is conducted through final year students to access their understanding, knowledge gained and applicability by learning the program.The questionnaire can be based on student's analysis towards analysis of complex engineering problems, designing of solutions for complex engineering problems, able to apply reasoning within the contextual knowledge, can able to apply ethical principles and commitment to professional ethics and responsibilities.
Curriculum Feedback is also conducted through final year students to get feedback on university designed curriculum.The questionnaire can be related to design of syllabus, coverage of advance topics, contents of syllabus are industry oriented, learning value (in terms of skills, concepts, knowledge, analytical abilities, or broadening perspectives), its applicability in real life etc.The Course Outcome of curriculum feedback and its mapping with Program Outcome is shown in Table 3 and 4 respectively.

CO1
The syllabus is suitable to the course.

CO2
The aims and objectives of the syllabi are well defined and clear to teachers and students.

CO3
The course content is followed by corresponding reference materials.

CO4
The syllabus has good balance between theory and application.

CO5
The syllabus has made me interested in the subject area.

CO6
It covers modern and advanced topics.

CO7
The syllabus is industry oriented.

CO8
The syllabus has learning values in terms of skills, concepts, knowledge, analytical abilities etc.

CO9
It has applicability in real life.CO10 It helps for going to higher studies.Employer Feedback is collected from the industry where the graduated student is working.The feedback is accessed on different parameters.The parameters can be its level of adherence to personal and professional ethics whilst working on engineering problems, its ability to provide solutions in engineering problems individually and as a member of the team, has ability to communicate effectively both verbal as well as written etc.

Tools, Criteria and Frequency of Assessment Tools
The assessment process for calculating attainments utilizes direct and indirect tools.The Table 5 demonstrates the direct and indirect tools used in the assessment process for the achievement of PO and PEO.The assessment tools have certain criteria for evaluation and each tool has certain period when the evaluation is done.In final attainments equal weightage is given to direct and indirect tools.The procedure for calculating the attainments is discussed in next section.

Attainment Process
The attainment process begins from formulation of Vision and Mission of the department along with defining PEO and PO of a postgraduate or undergraduate program.The Course Outcomes are defined for different subjects including surveys and are mapped with Program Outcomes.The direct attainments of various courses are calculated based on the performance of students in internal and external examinations.Similarly, surveys from alumni, employer etc. are conducted and the indirect attainments are calculated.The Expected and Attained value is calculated from direct and indirect tools.
Further, the Expected and Attained Program Outcomes are calculated from Course Outcomes.In PO calculation equal weightage is given to direct and indirect tools.The PEO are mapped with PO and average of mapped PO gives Expected and Attained PEO.
The PO and PEO attainments of last three passed out batches are calculated along with finding gap in case of deviation for very low or very high attainments [12].
The different formulas used in the calculation of attainments are as follows: The Attained PO is based on the assessment of internal and external exam of each course by calculating Net_CO attainment.In Net_CO calculation 30% weightage is given to Internal Exam and 70% weightage is given to External Exams.
In final attainments of PEO and PO equal weightage is given to direct and indirect tools. (4) The PEO are calculated by taking average of mapped Program Outcomes.For example, in case of PEO1 it is calculated as 6

Results and Discussion
The data of 2018 graduated students are given in this section.The results of different students for all semester examinations are recorded and the Expected and Attained value is calculated for each subject including projects, placements and surveys.From Equation 1, the average of CO-PO mapping gives Expected PO.
iJET -Vol.14, No. 23, 2019 In Table 5 the Expected PO attainment for different subjects is given.In Table 6 the Attained PO is given and it depends on student's performance in internal and external exams including assignments and quizzes.The Attained PO is calculated from equation 2 and 3.In case of surveys the Attained PO depends on the feedback of stake holders.The feedbacks can be collected online or offline.The Expected PO and Attained PO are calculated for different surveys.The curriculum feedback is shown in Table 8.The Expected and Attained PO for curriculum feedback is given in Table 9.The Final Expected and Attained PO is calculated from direct and indirect tools through equations 4, 5 and 6.The final PO attainment for last three batches is given in Table 9.The Expected PO value for different batches varies due to elective subjects.The PEO attainment is calculated from equation 7. The PEO attainment for the last three batches of postgraduate course is given in Table 11.

Table 11.
The PEO attainment for last three batches of postgraduate course

Fig. 2. PO attainments for different batches of postgraduate program
The Figure 2 gives percentage of final PO attainment for last three batches.The attainment level of all Program Outcomes is good.In case if there is very high attainment, very low attainment or poor attainment from last year's then there is a need for gap analysis.The gap analysis will tell the reason for the non-attainments or very high attainments.This gap analysis will be discussed in department BoS meeting along with the stakeholders.A corrective action plan is to be taken to reduce the gap.
The Table 11 shows a mapping between PEO and PO.The PEO are calculated from equation 7. The PEO attainment is given in Table 12 and the percentage of PEO attainment is given in Table 13.The comparative study of PEO attainment for the last three batches is given in Figure 3.The PEO value is gradually increasing from its previous value.In case of low attainments gap analysis is to be performed and is discussed in department BoS along with various stakeholders of program.

Conclusion
The outcome-based education is an innovative teaching learning methodology based on achieving targets.In this paper, we discussed about the direct and indirect tools used in the process of calculating attainments.We represented the complete attainment process starting from defining Vision, Mission, PEO and PO to final attainment of PO and PEO.The results from last three years of postgraduate course is presented.We also discussed how to find gap analysis in case of non-attainment of targets.The outcome-based learning is very helpful in validating the performance of students and helps in accreditation and assessment of program form various government bodies of India.

Fig. 1 .
Fig. 1.Direct and Indirect assessment tools used in the attainment process

Fig. 3 .
Fig. 3. PEO attainment for different batches of a postgraduate program

Table 2 .
CO-PO mapping of DSA-RCS406 course

Table 4 .
Mapping of curriculum feedback with Program OutcomesPaper-An Approach to Evaluate Program Outcomes and Program Educational Objectives through...

Table 6 .
Expected PO attainment for different subjects

Table 7 .
Attained PO for different subjects

Table 8 .
Curriculum Feedback of n graduated students

Table 9 .
Expected and Attained PO of Curriculum Feedback

Table 10 .
The final PO attainments for last three batches of postgraduate course

Table 12 .
The mapping of PEO with PO

Table 13 .
The PEO Attainment for different batches

Table 14 .
The PEO attainment percentage for different batches