An Evaluation of Instructional Process of Expert System Course Program by Using Mobile Technology-based CSE-UCLA Model

One of various types of application program which could facilitate human activities is known as expert system application program. The concept of the expert system of the application program so important that it requires to be shared to other people by developing a course program called expert system conducted in particular by an institution which specialized in the field of information technology. This course program of expert system which is offered to the students need to be designed in terms of good instructional process to improve its effectiveness. In fact, not all existing course programs conducted based on expert system could run more effectively than it was expected in order to achieve higher quality of learning outcomes. This could be explained as a numerous problems occurred during the process of implementation. In order to discover the extend of program effectiveness in the expert system of the course, a through program evaluation is required to be conducted. One model called CSE-UCLA will be recommended in making deeper instructional program evaluation in relation to expert system of course program. Through this model the instructional affectivity of application program of expert system could be measured by involving system assessment, program planning, program implementation, program improvement, and program certification. Key Words—Evaluation, Instruction Process, CSE-UCLA, Mobile Technology


Introduction
In the era of high speed development of technology a lot of application programs had been invented which could easily facilitate every human in his daily life either in the social activities, economic aspect, government activities, in the health sector, as well as education, and others [1]. One of the various types of the famous application program developed nowadays which could be able to support every single human activity is known as an expert system. The concept of this program is considerably important that a broad dissemination is required by developing and designing a course program called expert system, in particular around an institution where a field of information technology is greatly being concerned. A course of expert system presented for the students' needs to be designed into a more interesting instructional process that it could improve its' effectiveness.
An instructional process is called effective when the outcomes of the process itself bring an effect to the students in understanding the goal of instruction, the platforms of the instruction, the concept being presented, and they could manage the results, preparing all the related soft and hardware required in learning an expert system, they could also manage to set up either the software as well as hardware, they could be able to make rules based on the ability to manage the basis of knowledge, and finally they are able to present information as accurately as possible.
However, in reality not all processes of expert system courses developed in this institution could be able to run as effectively as expected. This indication is the results of the immersion of unexpected problems around the process activities. To find out the extend of how effective the process of instruction of an expert system course an evaluation is indeed required. By conducting an evaluation to a process of an instruction to a related course the problems could be identified then solutions and recommendations could be provided in order to improve the running process of instruction as well as to develop further the well effective running program of instruction.
One model proposed in this study in order to evaluate the process effectiveness of an expert system of course instruction is known as a mobile technology-based CSE-UCLA model. By using this model the extend of process effectiveness of the expert system course program could be assessed based on the five different dimensions, such as assessment system, program planning, program implementation, program improvement and program certification. In support with a mobile technology, the evaluation process by involving five dimensions of CSE-UCLA to the process of expert system course of instruction by evaluators could be managed quickly and easily, whenever and wherever possible. In term of system assessment, several aspects of expert system course of instruction to be evaluated include the students' ability to understand: the goal of instruction, the platform of instruction, and the basic concept the knowledge. In term of program planning, several aspects of expert system course of instruction to be evaluated include the students' ability to prepare all the required soft-wares and hard-wares in learning an expert system subject matter. In terms of program implementation, several aspects of expert system course of instruction to be evaluated include the students' ability to set up the required soft-wares and hard-wares in learning an expert system subject matter. In terms of program improvement, several aspects of expert system course of instruction to be evaluated include the students' ability to manage the basic knowledge and rules in relation to the course being concerned. In terms of program certification, the aspects of expert system course of instruction to be evaluated include the students' ability to present as accurately as possible the information based on the basic knowledge and rules being concerned.
The problems of this research, among others: 1) How is the effectiveness of the standard implementation of instructional process of expert system course program by using CSE-UCLA model based on mobile technology?, 2) How is the result of Evaluation of Expert System Learning Process using Model CSE-UCLA with manual calculation and the calculation based on mobile technology?, 3) What are the constraints found in the implementation of evaluation?
Some of the previous researches that underlies the implementation of this research include: research conducted in 2015 by Divayana [1] is about the use of CSE-UCLA model in evaluating the quality of expert system application program, where the research was conducted with the aim to know the good or bad quality of expert system application program. The result of this research is through the use of CSE-UCLA model in evaluating expert system application program, so it can be seen the percentage of program quality as a whole and also can be seen the percentage of program quality from five dimension that is: system assessment, program planning, program implementation, program improvement, and program certification. The constraints in the research is the absence of technology used to evaluate the quality of expert system application program is still in the form of application program, so it has not been able to conduct the evaluation process quickly and easily.
Another research in 2016 which is also done by Divayana [2] about evaluation of computer certification program at Universitas Teknologi Indonesia using CSE-UCLA model, where that research is conducted with the aim to know effectiveness of computer certification program at Universitas Teknologi Indonesia and obtained evaluation result is the implementation of the computer certification program at the University of Technology Indonesia as a whole at the level of effectiveness that is good categorized. The constraints found in the research that is not yet found a technology capable of performing evaluation quickly and easily using CSE-UCLA model without being limited by a certain time and place. From some constraints that are still found in previous research, then in this research is a breakthrough in the utilization of mobile technology to evaluate the learning process of expert system course, so that evaluation can be done easily and quickly, whenever and wherever.
There are several studies related to this research, namely research conducted by Ariawan, Sanjaya, and Divayana in 2016 about an evaluation of the implementation of practice teaching program for prospective teachers at Ganesha University of Education Based on CIPP-Forward Chaining [3], where the study has similarities with this research in the object of research is related to the learning process. The difference lies in the utilization of the evaluation model used to perform the evaluation, where Ariawan, Sanjaya, and Divayana use CIPP evaluation model, while in this research use CSE-UCLA model. Research conducted by Divayana in 2017 about evaluation of blended learning implementation at SMK TI Udayana using CSE-UCLA model [4], where the research have similarities with this research in terms of utilization of CSE-UCLA model to conduct an evaluation. The difference lies in the object under study, where the object of the study is blended learning, while the object of this study is the learning process of expert system course.
Based on research objectives, research problems, previous researches base, and related researches, the researchers are interested in conducting research about evaluation of instructional process of expert system course program by using mobile technology based on CSE-UCLA model.

2
Literature Review

Evaluation
An Evaluation is an activity to collect information whether something could manage to work or not, and further the information will be used to decide an accurate alternative to make decision [5]. Evaluation is a mean for understanding how things going. Based on the evaluation goal, criteria should be identified before evaluation could be conducted [6].
Evaluation is a research in order to collect, analyse, and present worthwhile information in relation to an object of evaluation, judging by making comparison with the indicators of evaluation and the results would be utilized when a decision should be made related to the object of evaluation [7].
Evaluation is an activity to collect, understand, and report the results of an analysis about a particular program/object so that the results can be used for consideration in taking a decision whether the program is continued or stopped [2]. Evaluation is an activity that consists of the process of gathering, describing, and explaining various pieces of information about the effectiveness of something that can be used later as the consideration for making a decision and a recommendation [3].
Evaluation is an activity for data collecting, data analyzing and data presenting into information about a particular object under study so that the results can be used to take a decision [8]. Evaluation is a process of understanding, giving meaning, obtaining and communicating an information for the purpose of taking a decision [9].
Evaluation is a process of judging something based on a predetermined criteria or purpose, and then followed by making decision to the object being evaluated [10]. We can define two key criteria to evaluate the quality of educational resources: veracity of learning activities, and content of the educational resource [11].
Mardapi states that the evaluation can be interpreted as a determination of the conformity between the results achieved and goals to be achieved [12].
Based on the above definitions a conclusion could be drawn that in general an evaluation is an activity to collect, analyse, and present an information about a particular object being studied and the result of an evaluation could be used as an aspect of consideration to make a decision.

CSE-UCLA (Center for the Study of Evaluation-University of California in Los Angeles) Evaluation Model
Evaluation framework was developed by Alkin in (1969,1991) at University of California-Los Angeles. This framework was called the UCLA Evaluation Model and was extensively used by the Center for the Study of Evaluation at UCLA. The Alkin framework has five types of evaluations within it (see Table 1) [13]. Assist in the selection of particular programs likely to be effective in meeting specific educational needs Program Implementation Provide information about whether a program was introduced to an appropriate group in the manner intended Program Improvement Provide information about how a program is functioning, whether interim objectives are being achieved, and whether unanticipated outcomes are appearing

Program Certification
Provide information about the value of the program and its potential for use elsewhere An evaluation CSE-UCLA is a model developed by Alkin used to evaluate a program in five different stages of evaluation which include: system assessment, program planning, program implementation, program improvement, and program certification [14]. CSE-UCLA model evaluation was accomplished in several phases, namely: system assessment, program planning, program implementation, program improvement, and program certification [15].
Tayibnapis stated that the evaluation model of CSE-UCLA developed by Alkin has five evaluation stages: system assessment, program planning, program implementation, program improvement, and program certification [16]. System assessment is evaluation component that provides information about the state or position of the system. Program planning is evaluation component that helps the selection of certain programs that may successfully meet the needs of the program. Program implementation is evaluation component that prepares information whether the program has been introduced to the appropriate group as planned. Program improvement is evaluation component that provides information about how the program works, works or runs, whether to a certain achievement. A Program certification is evaluation component which provides information on the value or use of the program.
A CSE-UCLA is a model of evaluation having five different evaluations (system assessment, program planning, program implementation, program improvement, program certification) and is appropriately reliable to evaluate service program and instructional program [17]. CSE-UCLA model is evaluation model that has five dimension of evaluation, such as: 1) system assessment, which provides information about the state of the system, 2) program planning, which selection of specific programs to meet program needs, 3) program implementation, which provides information to introduce programs, 4) improvement program, which provides information about program functions/ performance, 5) certification program, which provides information about benefits or uses of program [2].
Based on the above statements it could be concluded that in general an evaluation of CSE-UCLA a form of evaluation focusing on five aspects including,: providing information about the condition of the program being evaluated (system assessment), selection of effective programs relevant to meet the needs in planning the program (program planning), giving information/ introducing the program to the groups being appointed about the program implementation according to the re-determined planning (program implementation), providing information about the program performances (program improvement), and providing information about the results and significance of the program (program certification).

3
Research Methodology

Object of the Study
The object of the study include expert system of course program.

Subject of the Study
The subjects of the study include: teaching staff members and students. The selection of the subjects was conducted based on a technique called Purposive Sampling, such as all the related parties with the instructional process of expert system of course program.

Location of the Study
The study was conducted at the Department of Information Technology, Universitas Teknologi Indonesia, Denpasar, Bali.

Methods of Collecting Data
The data of the study were collected by distributing questionnaires, making observation, and documentation.

Design of the Study
The study utilized a model designed called as CSE-UCLA which could be presented as follows:

Technique Data Analysis
The data analysis of evaluation instructional process of expert system course program was conducted based on five different components, such as system assessment, program planning, program implementation, program improvement, and program certification by using descriptive quantitative technique.

4
Results and Discussion

Results
1. Standard effectiveness of instructional process of expert system course program according the CSE-UCLA model on the basis of mobile technology. A program implementation is called successful when at least the pre-determine effectiveness standard was met. The criteria of effectiveness standard used in the process of expert system of instruction based on CSE-UCLA model based on mobile technology is presented in the following

The Results of Evaluation of Process of Expert System Course Instruction
Using CSE-UCLA Model based on Mobile Technology Based on the process of evaluation of expert system course program by using CSE-UCLA model, the results based on manual calculation is presented in the following table 3 and the results based on the mobile technology supported by Android application program could be seen in the figure 2.

Discussion
Based on the results of the study described above, some points are required to be discussed, like: 1. The results of calculation about the instructional process effectiveness of the expert system course program both manually as well as automatically using mobile technology-based CSE-UCLA model indicated the same and accurate results such as: a) On the system assessment dimension, the extend of effectiveness was about 85.33%, this indicated a high category; b) On the program planning dimension, the extend of effectiveness was about 86.00%, this indicated a high category; c) On the program implementation aspect, the extend of effectiveness was about 86.00%, this indicated a high category; d) On the program improvement aspect, the extend of effectiveness was about 90.00%, this indicated a high category; e) On the program certification dimension, the extend of effectiveness was about 92.00%, this indicated a high category; f) The effectiveness of the instructional process of expert system course program as a whole totally about 87.87%, this indicated a high category.

Some problems encountered during the implementation of Evaluation
In general the extend of effectiveness of instructional process of expert course program belongs to a high category, however, according to the results of analysis made based on effectiveness standard of expert system course program implementation, presented on the Table 2, compared with the results of evaluation of the program implementation, as presented on Table 3, the results of quantitative analysis indicated that a problem was still found, in particular on the system assessment dimension, on the aspect of understanding the goal of learning, since the effectiveness score was still 84.00 % lower than the pre-determined effectiveness standard which was about >=85.00%. It was also proved from the results of observation that there were found many students did not understand deeply about the goal of learning. In addition, on the aspect of "understanding the learning platform" a problem also found, since the effectiveness score was about 84.00 % lower than the level of the pre-determined effectiveness standard which was about >=85.00%. It was also proved from the results of observation that some students still did not understand about the base of instruction. On the dimension of program planning, a problem was also found in particular on the aspect of "Understanding about hardware setup" since the score effectiveness was about 84.00 % lower than the pre-determined effectiveness standard which was about >=85.00%. It was also proved from the results of observation that some students still did not understand about how to prepare the hard-wares required to learn expert course. On the dimension of program implementation, a problem was also found in particular on the aspect of "Understanding about hardware set up" since the effectiveness score was about 84.00 % lower than the pre-determined effectiveness standard which was about >=85.00%. It was also proved from the results of observation that some students still did not understand how to install the hard-wares required to learn expert course. Other problems encountered from the application of CSE-UCLA model based on mobile technology when using Android facility that the students were unable to indicate the information about the extend of coherent effectiveness scores from the lowest to the highest scores.