Paper—A Web-Based Adaptive E-learning Application for Engineering Students: An Expert-Based ... A Web-Based Adaptive E-learning Application for Engineering Students: An Expert-Based Evaluation

The primary purpose of this study is to evaluate the web-based adaptive e-learning application based on the expert-based assessment. There are two aspects of assessment considered in this study, the first one will evaluate the e-learning system in terms of the learning content and its structure, and the second one will focus on the media aspect. The process of evaluation was started by developing the instruments of evaluation by taking into account some related literature. Then, the content validity of the instruments was checked by scientific experts. After that, the assessment was conducted by two groups of experts in a paper-and-pencil format by marking one out of 4 points Likert scale. The result was then analyzed through some justification approaches and indicated that the adaptive e-learning application was categorized acceptable to use in the learning process. Keywords—Web-based e-learning, adaptive e-learning, personalized elearning, engineering students, expert-based evaluation


Introduction
In the learning process, it is essential to recognize the characteristics of the students. There is a general perception that each student has different characteristics compared to the other. Vincent & Ross mentioned that every single person takes in and processes information and knowledge in specific ways based on their individual preferences [1]. This situation may stimulate the learning strategy that one student cannot be treated as same as the others. Dunn and Dunn learning style model has determined that every individual has his/her own specific learning style and unique personal strengths [2]. The model also suggested that it will be much more effective to teach individuals by looking at their own preferences. By adopting that approach, the educators could easily find an accurate strategy in the learning and teaching process. Consequently, it may lead to a positive learning outcome.
It is inevitable that every sector is significantly influenced by the rapid growth of information and communication technology, including in the educational field [3]- [5]. It is often found that the traditional learning process partly mediated by the existence of technology or totally transformed into the digital learning concept. The one, that is becoming more and more popular now, is the adaptive e-learning application.
The adaptive e-learning is one type of e-learning that generally runs on the network platform which has the capability to suit learner's preferences [6]. The network platform may provide broader access to any students in any place and any time. The strategy used in the adaptivity mechanism of e-learning changes significantly. It is started with one aspect of personalization as an input of adaptivity or another one that prefers to consider multiple aspects. One widely used as a trigger for adaptivity is learning style. Other possible aspects are knowledge state, cognitive style, or student's behavior [7]. All of those strategies in the personalization input aim to provide a suitable and convenient learning platform for learners. This may drive to learning satisfaction and lead to higher learning achievement.
In order to assure the adaptive e-learning system meets the primary objective to provide a personalized learning environment to learners, it needs some evaluations. In the context of software application, the evaluation can be conducted through debugging [8], functional-based testing [9], and structural-based testing [10]. Software testing is a vital stage in software development to ensure that there are no bugs in the software and the software can work as designed [11]. Other essential evaluations in the domain of Human-Computer Interaction (HCI) are expert-based assessment [12], [13] and end-user evaluation [14]. The basis of the web-based adaptive e-learning application in this study is the system that we have been designed and developed in previous research [7]. The system has also been tested based on functional-based software testing and reported in a good result. Therefore, the main objective of this study will focus on the expert-based assessment to evaluate the adaptive e-learning application. This choice comes from the reason that the expert-based assessment proposed many benefits i.e., cheap, fast, and easy to use [13]. This kind of assessment is also one of the most common inspection methods used by HCI practitioners for decades [15]. The expert-based assessment will be conducted in two different elements, the first will evaluate the learning content, and the second will focus on the media aspect.

A Web-Based Adaptive E-learning Application
In this study, the web-based adaptive e-learning application that we have been previously proposed [16] and developed [7] is used as an object of this expert-based evaluation. The subject installed in the e-learning is a Digital Simulation. The digital simulation is one of the mandatory subjects in the department of Computer Network Techniques in Vocational High Schools in Indonesia. This subject discusses the utilization of information and communication technology for the development of learning material.
Concerning the adaptation mechanism, there are two aspects considered in this personalized e-learning application. The first one is the student's learning style adopted from the Felder and Silverman Learning Style Model (FSLSM). This learning style model is chosen because this model is specially designed for the engineering context [17], [18] and very often used in research related to technology-enhanced learning [19]. This model describes the learning style in more detail, distinguishing the learner preferences into four dimensions (active-reflective, sensing-intuitive, visual-verbal, and sequential-global) [20]. Theoretically, these four dimensions possibly generate 16 (or 2 4 ) different learning styles. In order to collect the learning style information with regard to this model, the Index of Learning Styles (ILS) questionnaire, which consists of 44 multiple-choice items [21], is implemented in this e-learning system. The second aspect of adaptation is information concerning the student's initial knowledge. The pre-knowledge of each student is gathered through a pre-test. This pre-test is constructed in a multiple-choice form that corresponds to a specific subject.  Figure 1 presents the screenshot of the main window of the web-based adaptive elearning application. The system interface of this e-learning is primarily composed of three main areas. The first, which is located on the left side, is the navigation area. This area has the adaptive ability to automatically fit the student's initial knowledge and the sequential-global dimension of Felder and Silverman model. To this concern, the adaptation in navigation support proposed by Brusilovsky [22] is implemented. With regard to the student's pre-knowledge, the link representing the course units will be shown or hidden depending on the pre-test score. Regarding the sequential-global aspect, the links in this navigation area will be acted in two conditions. For the sequential learners, the sub-units links will automatically disappear. They only show the main-units links. As a consequent, the sequential type of user may only explore the course's materials by hitting the next and previous button. They cannot jump directly into other pages as the global learners do. The different links format will appear for the global learner. The student in this type will be provided the links of main-units and including sub-units. These all links may present a brief overview related to the course. The student may explore the material offered by the course directly to each independent page by utilizing the more completed links available.
The second area, which occupies the middle part, is called the fundamental content area. This area is created as a static content area; it means that this area has no capability for adaptation to change its content fit to student's preferences. This fundamental content area may present the learning material in both visual-and verbal-based forms. The third area is the additional content area, which is located on the right side. As the navigation area has, this area also has adaptation ability. The adaptation in presentation technique from Brusilovsky [22] is considered to overcome the situation. Therefore, the learning material presented in this area is depending on the student's learning style, particularly to accommodate three dimensions of FSLSM (activereflective, sensing-intuitive, and visual-verbal). For the visual learner type, the information will provide mostly in visual media formats such as image, video, animation. Otherwise, for the verbal learner type, it will present the material in verbal media formats such as text, audio. There are some buttons attached to the top part of this area. The function of those buttons depends on the active-reflective and sensingintuitive dimensions of the student's learning style. When a particular button is clicked, the additional floating window will present the supplementary learning material related to a particular button. The learning material presented is following the set of rules mentioned in our previous research [7]. The set of rules is made as guidance for the system to automatically show the learning objects related to the activereflective and sensing-intuitive dimensions.
The application of the web-based adaptive e-learning system has been developed as well as tested in a functional-based test approach. The functional testing, Luo [10] mentioned as black-box testing, is an essential element in the software development in order to assure the system free from bugs and act as designed [11]. This test ignores the internal mechanism of a system or component and focuses only on the outputs generated by the system with certain inputs [23]. Williams [9] mentioned that this functional-based test is one of the basic essential tests for software testing. The functional-based test is performed by administering some different test inputs to the system and then looking at the behavior of the system. The test results reported that the adaptive e-learning system could react as designed by automatically changing the learning environment and learning path based on the student's learning style and preknowledge [7].

Research Method
The main objective of this study is evaluating the web-based adaptive e-learning application in order to assure the development of the instructional system meets what it is supposed to. The evaluation is conducted based on the expert-based assessment in two different perspectives. The first assessment is concerning the learning content aspects, and the second is focusing on the media aspect. The first assessment deals with the structure and its content on an intended topic whether or not the learning content matches the curriculum. This evaluation will be conducted by a group of people who have expertise on an intended subject. Meanwhile, the second assessment refers to the evaluation of the appearance of the e-learning application with regard to the multimedia components (color, text, picture, sound, etc.). The process starts by creating the questionnaires based on several criteria of assessment. Then, the validity of questionnaires is checked thoroughly. The following step is administering the questionnaire to the group of experts. Finally, the collected data are discussed comprehensively.

Developing the instruments for evaluation
The initial step in developing the instruments of assessment always takes into account the primary purpose of evaluation [24]. After getting a clear insight on that respect, the next step is identifying the criteria of assessment by reviewing some related literature. Concerning the evaluation with regard to the learning content aspect, the very compatible one is adopted from the established learning content questionnaire made by the Ministry of National Education of Indonesia [25].
The instrument of learning content evaluation comprises of two aspects, namely: material substance and learning design. The material substance aspect has four indicators, with nine items of questions. Meanwhile, the learning design aspect consists of eight indicators, with ten items of questions. All of the questions are in a positive wording format and structured on a 4-point Likert scale. The detailed outline of the questionnaire can be seen in Table 1.  [27], and Vaughan [28] has been analyzed. As a result, there are two aspects considered in the questionnaire. The first aspect is related to the visual interface and the second aspect concerns software utilization. The visual interface aspect consists of five indicators with 16 items, while three indicators represent the software utilization aspect with four questions. The positive wording format is also used to develop all of the questions. The most widely used 4-point Likert scale ranging from 'strongly agree' to 'strongly disagree' is offered to measure the respondent's opinion. Table 2 shows the outline of the media aspect questionnaire.

The validity of the instruments
Validity is the degree to which the assessment tool accurately measures what it is supposed to measure. It is essential to check the validity of the instruments before handing it out to the respondents. To address this, the first draft of those questionnaires is reviewed in terms of content validity. Many definitions of content validity have been published [29]- [34]. Haynes et al. [35] encapsulated the definition of the content validity from those researchers. The content validity, sometimes called a logical or rational validity, refers to the degree to which the items on a test are reasonably representative of the construct of the test which will be used to measure. This validity can be managed through a rational analysis from the competent panel or experts' judgment. Two scientific experts with an academic background in evaluation and the intended topic involved in this content validity check. According to the comments and suggestions from experts, some of the statements on the questionnaire need modification on the basis of clarity and readability. After refinement of the questionnaire, those are then made as a final version.

The evaluation procedure
The following step is administering the questionnaires to the experts. It must be noted that there are two questionnaires, the first one is related to the learning content aspect, and the second is concerned with the media aspect. There were two groups of experts involved in this study. The first group consisted of three subject-related teachers who evaluated the learning content aspect. The selection of teachers was based on their expertise in the intended subject. Another group that examined the media aspect comprised of two media-based experts. The experts involved in the media aspect evaluation were chosen based on their competences in multimedia design. The questionnaires were then distributed in a paper-and-pencil format to the respective groups. In the beginning, a brief explanation about the web-based adaptive e-learning soft-ware and the operation procedure was explained to both groups. Then, they were given a chance to access and explore the e-learning software individually. To express what they thought, they had to mark one out of 4 points Likert scale on each item of the questionnaire. The collected data were then tabulated and analyzed.

Research Results and Discussion
This section will talk about the results of this study, including its discussions. As mentioned previously that there are two evaluations based on the expert-based judgment. The first one was to measure the adaptive e-learning application based on the learning content aspect. The second was to measure the same application in terms of the media aspect.

4.1
The research results Table 3 shows the results of the learning content evaluation. There are two aspects of evaluation; material substance and learning design. In both aspects, the scores collected are presented in two different scales; the 1-4 Likert scale and 0-100 score transformation. It can be seen that the total average Likert scores are 3.59 and 3.67 for the aspect of material substance and learning design, respectively. Moreover, the total average Likert scale is 3.63. Looking at the "0-100 score" column, the material substance aspect reaches score 86.42, and the learning design aspect achieves 88.89. Meanwhile, the total average score is 87.72.  Table 4 exhibits the results of the evaluation based on media-based experts. The evaluation results are divided into two aspects, namely, visual interface and software utilization. For the assessment element of the visual interface, it scores 3.31 (Likert scale) or 77.08 (0-100 score). Meanwhile, for the software utilization part, it records 3.50 (Likert scale) or 83.33 (0-100 score). The total average for the media-concerned evaluation is 3.35 (Likert scale) or 78.33 (0-100 score).

Discussion
From the results of evaluation based on two groups of experts (learning content and media-based), it can be noted some interesting points to analyze and discuss. The results only show the number whether on Likert scale or 0-100 score format. There is no further information what the meaning of those numbers is. Therefore, it is important to define whether or not those scores achieved represent the acceptance criteria. To this concern, there is no specific way to describe those scores. However, since the questionnaires constructed in the Likert scale format, it can be taken into account the score justification based on the Likert scale characteristic.
Babbitt and Nystrom [36] suggested the "dichotomously justification" approach based on the direction of response. The way to categorize the Likert response is by simply classifying the rating scale as accepted or unaccepted based on the agreement or disagreement response for each item. The agreement response from respondents will be categorized as accepted, while the disagreement response will be classified as unaccepted. Marreez et al. [37] proposed a similar method. The method is done by converting the Likert rating into "binomial data" (accept/reject). For example, when there is a 4-point Likert scale spanning from 1 (strongly agree), 2 (agree), 3 (disagree), until 4 (strongly disagree). It then puts score 1 and 2 as the "Accept" category and rating 3 and 4 as the "Reject" category.
From both approaches, it can be drawn that their strategies are obviously divided the response into two opposite categories (accept or reject). As a consequence, there is a threshold to separate those two categories. At this point, the threshold could be simply a middle score of the intended Likert scale. Since the Likert scale used is ranging from 1 to 4, hence the threshold score equal to 2.5. Therefore, it can be summarized that the Likert score which is the same or exceeding the threshold can be classified as accepted. Otherwise, it is included in the category of unaccepted (or rejected).
Accordingly, based on the Likert score collected and the approaches for justification described above, it can be concluded that the web-based adaptive e-learning application can be accepted to feasibly use for the students in terms of the learning content aspect and media-based aspect. It is proved by the total average score from both aspects (3.63 and 3.35 respectively) which are exceeding the threshold. Looking into detail for each element in the learning content evaluation, the Likert score in the material substance (3.59) and learning design (3.67) elements are also exceeding the positive limit (2.5). Therefore, they are categorized in the acceptable criteria. The same situation reveals in the media-based evaluation. Since the Likert score for the visual interface (3.31) and software utilization (3.50) components are higher than the threshold, both are also classified in the acceptable criteria.
It is also important to investigate how high the acceptance level of the experts on the adaptive e-learning application. To address this, the typical traditional school score suggested by Debevc & Bele [38] is used to describe the level of acceptance. It can be seen from Table 3 particularly at the column 0-100 score that the total average score of learning content assessment is 87.72. This score means that the level of acceptance of the experts in terms of learning content is 87.72%. This assessment is significantly high; it almost reaches 90%. Meanwhile, the total average score (column 0-100 score of Table 4) of media-based assessment is 78.33. It means that the level of acceptance of the media-based experts is 78.33%. This score is relatively lower compared to the assessment of the learning content aspect, but it is still good.

Conclusion
It is noteworthy that each student differs. They have their own preferences to learn a particular subject. Consequently, they cannot be treated as a homogenous student. In order to overcome this demand, our previous study suggested the adaptive e-learning system that had the capability to automatically provide a different learning environment and learning path to a specific student [7], [16]. The proposed system has already proven well in terms of functional-based evaluation [7]. However, it still needs a comprehensive evaluation to assure the personalized e-learning system to meet the students' personalization.
The evaluation based on the experts' assessment in this study has been conducted in two categories. The first category assessed the adaptive e-learning system concerning the learning content while the second focused on the media aspect. The experienced experts have been chosen to provide the assessment on the respective expertise. After the data gathered, the approaches from Babbitt and Nystrom [36], Marreez et al. [37], and Debevc & Bele [38] were used to justify those data. As a result, it can be concluded that the adaptive e-learning application was categorized acceptable to use in the learning process. The results also indicated that the experts rated relatively high (87.72%) for the learning content assessment and relatively good (78.33%) for the media aspect assessment. These findings implicate to the feasibility of the adaptive elearning application to use in the learning process by the students. The more feasible the instructional application offered, the more comfortable the students could utilize it. This may lead to higher learning-outcome achievement.
Though the result seems promising, there are still some limitations interesting to discuss. The first one concerns the number of experts involved in this study. There were five experts in total participated in this evaluation. Those were divided into two and three for each specific task. Although the number of experts involved in this work was relatively small, it is compatible with other studies conducted by Daramola et al. [39] and Habsy et al. [40]. Nevertheless, for future studies, it is necessary to consider a rather bigger number of evaluators in order to get more comprehensively results.
Another limitation of this study is concerning the set of evaluations. The ideal assessment should be taken into account a broad range of perspectives, namely, software evaluation [8], [11], expert-based review [12], [13], and end-user perception [14]. In terms of software evaluation, the functional-based test has been conducted in our previous study [7]. This functional-based test or black-box testing has been applied also by Song & Zhang [41] to evaluate the college teaching system. In this current study, it merely assesses the web-based adaptive e-learning system based on the experts' assessment. Lallemand et al. [42] recommended combining with the userbased evaluation in order to increase the validity and reliability of the assessment. This shortcoming could be considered for further improvement in future work.