Paper— Development and Evaluation of a Mobile-Learning Application Based on the Felder-Silverman… Development and Evaluation of a Mobile-Learning Application Based on the Felder-Silverman Learning Styles Model

— Every student has a different learning style. Not all of these learning styles can be facilitated through classroom learning. Mobile learning (m-learning) is the use of mobile technology to support learning activities and facilitate a variety of student learning styles. However, no previous studies of the development of m-learning products have tested the possibility of facilitating different learning styles within a single M-learning application. This study describes the development and evaluation of an adaptive m-learning application that can facilitate the needs of different students. The Felder-Silverman Learning Style Model was used as the reference for learning styles. The prototype application was developed using the user-centered design method, which includes user needs analysis, interaction design, application development, and usability testing. The designers followed the best practices laid out in Shnei-derman's Eight Golden Rules of Interface Design and Gagné’s Nine Events of Instruction. A usability test of the prototype involving ten participants. Usability testing is carried out by conducting usability testing and filling out the System Usability Scale (SUS) questionnaire by all participants. The result found that it had good usability.


Introduction
The development of technology that is growing very rapidly eliminates the limitations of something related to space and time. The world of education is one area that utilizes the use of technology to maximize learning or what is commonly referred to as e-Learning [1]. The emergence of e-Learning has a good impact on the world of education because with its emergence, now students can learn anywhere and anytime [2]. However, the emergence of e-Learning does not have a purpose to replace learning methods in the classroom, but to support learning activities in the classroom [3].
Previously, e-Learning (electronic learning) started with d-Learning (distance learning) which currently has m-Learning as a new form of e-Learning that utilizes mobile technology to do learning [1]. The existence of m-Learning as a new model of e-Learning is considered to have interesting things because it can have an effect on classroom teaching and can connect formal and informal learning [4]. In addition, the emergence of m-Learning also strengthens the original purpose of e-Learning, which is to make it easier for students to learn anytime and anywhere.
This study aims to create m-learning applications that can meet the needs of users with a variety of learning styles. The interaction design of the application was developed by considering user needs and following Shneiderman's Eight Golden Rules of Interface Design [5] and Gagné's Nine Events of Instruction [6]. We used the definition of user needs developed by Hasani because it is considered capable of producing interaction designs with good usability [7]. Additionally, input from users was employed during the evaluation phase of the first iteration of this study to create an mlearning application with good usability. Through this process, we developed an mlearning product and evaluated it through usability testing and System Usability Scale (SUS) assessment.
Before beginning our study, we established two research questions:

M-learning
Classroom learning is usually delivered through the teacher-centered model. In this learning method, the teacher is the primary driver of the learning process. But with the emergence of the internet and e-learning, the teaching and learning process can happen anywhere and anytime [2].
Aparicio, Bacao, and Oliveira [9] have described and defined several e-learning concepts (Table 1). Mobile learning, known as m-learning, is a category of e-learning. It refers to learning processes that employ mobile devices; m-learning will have an important role in teaching and learning in the twenty-first century [10].
M-learning is intended to support the implementation of e-learning. The existence of e-Learning indeed allows the learning process to be carried out anytime and anywhere. However, with m-Learning, students or teachers can use the cell phone feature to get a better user experience [11]. With m-learning, students can complete learning activities more quickly than with desktop or laptop computers. Both students and instructors can get feedback easily because cell phones allow users to receive notifications. Things like this make it more likely for students to access learning in places that are difficult to implement when using a desktop [12]. Thus, m-Learning allows students to continuously learn and remain active during the learning process.

Learning Styles
Each student has their own learning style, and this style is influential in determining their learning strategy. Students have different levels of motivation, different opinions, and different responses to the learning process in a particular classroom environment [13]. Learning processes that do not accord with the ways that a student learns will be ineffective and unpleasant for the student [14]. This learning-style problem has encouraged researchers in the field of education to examine and formulate models of learning styles. Among the theories of learning styles that have been proposed are Kolb's Model, the Myers-Briggs Type Indicator (MBTI) Model, and the Felder-Silverman Learning Styles Model (FSLSM).
This study uses the FSLSM as the reference for learning styles. According to Deborah, Baskaran, and Kannan in 2014, the FSLSM is the most suitable learning-style model for evaluating the learning styles of e-learning users [15]. The FSLSM also can be used to analyze learning object preferences and interaction designs of e-learning systems [7]. There are four dimensions of learning styles in this model, including (1) perception (sensory / intuitive), (2) input (visual / verbal), (3) processing (active/reflective), and (4) understanding (sequential / global) [16]. Perception is a dimension related to how students absorb information, input is a dimension related to how students remember information, processing is a dimension related to how students convert information, and understanding is a dimension related to how students understand information [14]. These four dimensions can be detected by using an instrument called the Index of Learning Styles, which contains 44 questions that have values from 0 to 11 for each dimension. Each dimension receives a score that indicates the strength of the test-taker's preference for it (strong, moderate, or mild).

User-Centered Design as one of Design Methods
Three methods are used in interaction design, namely, activity-centered design (ACD), user-centered design (UCD), and goal-directed design (GDD). There has been little research into ACD [17]. The main focus of ACD is on the activities the system must provide for its users. GDD has six phases, which include research, modeling, requirements, framework, refinement, and support [18]. UCD and GDD is a similar design method because users have a crucial role in design phases to meet their requirements. The difference between the two is that GDD has more specific process than UCD based on the phases [19].
As we developed our prototype using UCD, we evaluated user understanding, feedback from users, and the overall user experience, using our findings to guide improvements. Using UCD lead to user involvement throughout the design and development process. The design process involved four phases: specifying the context of use, specifying requirements, creating design solutions, and evaluating designs.

Usability evaluation
UCD is an iterative method that contains an evaluation process so that there is a close relationship between UCD and the evaluation of the design. This evaluation process is designed to ensure that a product fulfills all aspects of usability, including being (1) useful for users, (2) easy to use, (3) effective and efficient in use, and (4) satisfying when used [20]. There are two approaches to evaluating an interaction design, namely, qualitative and quantitative. The evaluation will be more complete if both types of evaluations are performed. Thus, this study utilizes usability testing for a qualitative approach and the SUS for a quantitative approach. Usability testing was chosen because it provides positive and constructive input to improve the usability of an application. SUS was chosen because it is a well-known and widely used instrument.

Gagné's nine events of instruction
Gagné's Nine Events of Instruction is a theory that describes how instructional material is conveyed from instructor to student. The theory identifies nine steps in this process: gaining attention, informing the learner of the objective, stimulating recall of the prerequisites, presenting the stimulus material, providing learning guidance, eliciting performance, providing feedback about performance correctness, assessing the performance, and enhancing retention and transfer. Gagné's theory was used to map the instructional material in this study because it is considered a best practice for this purpose [7].

Usability goals and best practice in interaction design
An interaction design is considered to be easy to use if it meets six usability goals: effectiveness, efficiency, safety, utility, being easy to learn, and memorability (being easy to remember) [20]. To meet these goals, we used Shneiderman's Eight Golden Rules of Interface Design, which are widely used and considered best practice. They can be applied to almost all types of interactive systems [5]. Shneiderman, in his rules, advises designers to (1) strive for consistency, (2) enable frequent users to use shortcuts, (3) offer informative feedback, (4) design dialogue to yield closure, (5) offer simple error handling, (6) permit easy reversal of actions, (7) support internal locus of control, and (8) reduce short-term memory load.

Previous research
From the literature review conducted, three previous studies were found related to the development of m-learning and adaptive e-learning based on the Felder-Silverman learning style model. Previous studies were analyzed for comparison with the research conducted. From the literature review conducted, three previous studies were found related to the development of m-learning and adaptive e-learning based on the Felder-Silverman learning style model. Previous studies were analyzed for comparison with the research conducted. The following is a comparison of the contribution and relevance of this research to existing research ( Table 2).

Researcher Contributions and findings Issues and limitations
Fahri and Samsudin [21] Comparison of Likert-type scales between mobile use on e-Learning and existing e-Learning.
Mobile learning cannot be personalized according to individual characteristics and usability testing is not carried out.
Ilona-Elefteryja and Stavros [22] The research carried out resulted in an m-Learning in accordance with SCORM standards.
Mobile learning cannot be personalized according to individual characteristics and usability testing is not carried out.
Hasani [7] The research produces an alternative interaction design that has a better evaluation value than Ovariyanti's [23] evaluation value.
The research is limited to the prototype.
Based on Table 2, this research focuses on developing m-learning with interaction designs adapted from research conducted by Hasani [7] because its alternative interaction design is an interaction design with good usability values and also becomes an improvement to the interaction design of research conducted by Ovariyanti [23]. The final goal of this research is the creation of a mobile-based e-Learning application that can be personalized according to the learning characteristics of each individual with an interaction design adapted from research conducted by Hasani [7] to maintain good usability values.

Research Methodology
In this section, we describe our research approach, participants, and instruments.

Research Approach
An interaction design development process will be considered successful if the resulting application can be used easily and pleasantly by users. Thus, we used a method that focuses on the user. There are two user-focused methods in interaction design: GDD and UCD. This study uses UCD because in the UCD phase, there is an evaluate phase so that it can help to identify design problems from the start. Other than that, UCD is based on the needs and characteristics of users and is believed to produce products with good usability.
There are five stages in UCD, including (1) planning and studying literature, (2) analyzing user research, (3) designing interaction design and software development, (4) evaluation, and (5) formulating recommendations and conclusions. We went through one iteration of this process. Figure 1 visualizes the five stages of our research. The first stage was used to formulate the research questions. The second and third stages were used to answer research question 1 (PP1), namely, how to design an m-learning application using the FSLSM. The fourth and fifth stages were used to answer research question 2 (PP2), namely, whether the application we designed had good usability. Each of the five stages of our UCD research (the study of the literature, analysis of user research, interaction and software design, evaluation, and formulation of conclusions) produced an outcome.

Fig. 1. Research phases
Problem formulation and literature study is a stage that has three activities, namely (1) designing a research idea, (2) formulating a literature study, and (3) determining the research methodology. In the first activity, the research begins by formulating the background of this research and the expectations of the research. After formulating the background, a research gap was formulated between this study and several previous studies. There are two categories of research referred to for this research, namely research in the development of interaction designs for adaptive e-learning and the development of m-Learning applications. Several previous studies have also been collected, for the category of interaction design development for adaptive e-learning, research was conducted by Hasani [7], then for the category of m-Learning applica-tion development, research conducted by Fahri and Samsudin [21], Ilona-Elefteryja and Stavros [22]. In the second activity, a literature study was formulated. The literature study that has been formulated becomes the theoretical basis for how this research was made. After the second activity was completed, the next activity was determining the research methodology. The research methodology used for this research is User Centered Design (UCD) which is considered as a further iteration of the research conducted by Hasani [7]. After all activities are carried out, this research is continued to the user research analysis stage.
This study is a continuation of the UCD iteration of the research conducted by Hasani [7], so that an analysis of the user research in the study was carried out. In this study, the requirement gathering stage was carried out for four different learning style groups, including (1) Figure 2 is the result of a requirement gathering analysis conducted in Hasani's research [7].

Fig. 2. User research results [7]
Interaction design and software design are the stages where interaction design and m-Learning software are designed and created. There are three main activities carried out at this stage, including (1) analyzing the learning object mapping, (2) designing a low-fidelity prototype, and (3) creating applications. In the first activity, the research begins by analyzing the learning object mapping. In this activity, the mapping of learning objects for interaction design has three design bases, namely (1) design strategies (Hasani, 2019), (2) best practices for interaction design development, and (3) Gagne Nine Events of Instructions (Gagne, Briggs, & Wager, 1992).
The evaluation stage has two main activities, namely usability testing and filling out the System Usability Scale (SUS) questionnaire. In addition to the main activities, at this stage there are also several supporting activities, including (1) designing tasks and scenarios for usability testing, (2) making evaluation instruments, (3) evaluating designs with participants, (4) analyzing evaluation results, (5) make points of improvement. After all activities have been carried out, this stage produces several final results, including (1) the SUS scale, (2) participant responses to the developed application, and (3) several points of improvement of the analysis results from the evaluation.
After all the stages have been passed, this research is in the final stage of the research, namely the formulation of conclusions. The formulation of conclusions is carried out to find points of improvement and user experience issues that arise in the application that has been made. The study ends with a conclusion that answers the research questions asked.

Research participants
The research participants were defined as those individuals who participated in the usability testing activity. They were students at the Faculty of Computer Science at Universitas Indonesia who were enrolled in a human-computer interaction course. They were considered to have an understanding of how to design a product that can meet user needs and the ability to provide constructive feedback about unanticipated issues. According to Jakob Nielsen [24], the minimum number of participants for usability testing is 5, but more is better. However, the greater the number of participants, the lower the benefit-cost ratio of user testing. Therefore, this study chose only 10 participants.

Research instruments
We used two research instruments in evaluating our prototype application, a usability testing sheet and the SUS questionnaire. A usability testing sheet is a tool used to record things that must be observed. It supports the research participants in their role as facilitators of usability testing. We used a Google form for our testing sheet. Things that were recorded and entered into the form during the usability testing process were (1) the participant's identity, (2) the participant's level of success in carrying out a task, (3) the difficulties encountered by the participant, (4) things that the participant liked, and (5) feedback related to the task being done. SUS is one of widely used questionnaire-based instrument for evaluating the usability of a system. Its reliability is well-documented [25].

Interaction design and implementation
The m-learning application was produced in this stage of the study. The user needs, identified using the method proposed by Hasani [7], were analyzed and mapped against Gagné's Nine Events of Instruction. The results of this mapping were used as a basis for designing low-fidelity prototypes.
A low-fidelity prototype is a wireframe with a low level of precision that is used to determine the layout of an application. A pencil sketches on paper are used to design the low-fidelity prototype. The sketches made at this stage are also used for reference when a high-fidelity prototype is designed.
A high-fidelity prototype is designed with precision. At this point, the color, size, and shape of the elements have been fully determined. The high-fidelity prototype in this study was designed during the implementation phase. In designing it, we took into account Shneiderman's Eight Golden Rules of Interface Design and Gagné's Nine Events of Instructions. The high-fidelity prototype was used during the next steps in the study.

Course page interfaces and subject chapters
Gagné's Nine Events of Instructions were used to map the learning objectives. The first, gaining attention, occurs on the start page, where students are presented with a chapter. Students with a global learning style get a presentation in the form of a grid view for all chapters, whereas those with sequential learning styles get a presentation in the form of a list that displays the chapters in order. Figure 3 shows the chapterappearance design for global learning styles (right) and sequential learning styles (left).

Fig. 3. Course Page
Trigger questions relating to the chapter to be studied can also be used to gain students' attention. Different trigger questions are used depending on whether a student has a sensing or intuitive learning style. Trigger questions for students in the sensing group involved daily life, whereas questions for those in the intuitive group were abstract. Figure 4 shows the design of the trigger questions targeted at sensing and intuitive learning styles. The second of Gagné's Nine Events is informing the learner of the learning objectives, that is, what competencies are expected to be achieved. The presentation of the learning objectives was the same regardless of the student's learning style. Figure 5 shows the appearance of the learning objectives. Stimulating recall of prerequisites is the third event required before students begin to learn the material. To meet this need, the application was designed to provide prequiz features. Figure 6 shows these features. The presentation of the stimulus material is the most important event in the learning process. We used slides designed by considering visualization principles to deliver the material as Hasani [7] found that this mode of presentation was popular regardless of learning type. The way the slides were presented varied depending on the student's learning style. Students with a global learning style were able to access all of the sections within a chapter at will, as shown in Figure 7.
Those with a sequential learning style were required to complete each section before being able to access the next section (so that they would learn linearly). Figure 8 shows a section designed for sequential learning styles.
The events which Gagné refers to as eliciting performance and assessing performance are addressed by a final test after the student has undertaken learning activities. These events are facilitated for every learning style by the same learning object, which is the post-quiz, shown in Figure 9.   Gagné's ninth event, enhancing retention and transfer, is addressed by the presentation of a summary at the end of the chapter. The summary, shown in Figure 10, is designed to facilitate the recall of the material. The last event addressed in our application (although it is Gagné's seventh rather than last event) is the provision of feedback. The user is given their pre-quiz and postquiz scores as feedback, as shown in Figure 11.

Evaluation
We conducted usability testing to evaluate the interaction design of the application. There were 10 participants, who could have any learning style. The learning styles of the participants are presented in Table 3. Table 3. Usability testing participants encountered, the things they liked about the application, and their suggestions for each task Figure12 shows the success rates for each task.

Fig. 12. Usability testing result
After the participants completed the eight tasks, they were given the SUS. SUS is a questionnaire that has 10 questions on the questionnaire. In its use, research participants were given an SUS questionnaire containing 10 questions and each question was given a choice of a scale of one to five based on how much they agreed with each statement on the questionnaire regarding the product being tested. Figure 13 represents the 10 SUS questionnaire questions that were asked to study participants. From the results of usability testing and the SUS score obtained by the application conducted on 10 study participants, it can be concluded that the resulting application has good usability. The eight tasks performed by participants during usability testing resulted in a 100% success rate. In addition, the SUS score obtained in this study also strengthens because the SUS score obtained was 85.75 which gets grade A+ [8].

Conclusion
We created an m-learning application based on the FSLSM and found that it had good usability. The SUS overall score obtained from the research participants was 85.75 (an A+ grade). However, we received some criticisms and suggestions at the evaluation stage that merit further research. First, the application developed in this study was designed to be used offline. Therefore, the data collected was stored only on each user's device. As a result, students could not collaborate on activities. That also meant that the processing dimension of the FSLSM could not be included because it requires interactive activities such as forums and threads.
Second, the only courses available on this application relate to interaction systems, so the application will need further development to be used to deliver other courses. Third, the slides available in this application are not specific to a learning style. Thus, the application has not facilitated the input dimension of the FSLSM (visual/verbal). This application will need to be developed further to facilitate the processing and input dimensions of the FSLSM and allow the delivery of courses other than those relating to interaction systems.