The Evaluation of User Experience of Learning Management Systems Using UEQ

— The rapid development and improvement of learning management system (LMS) is driven by the rise of ubiquitous computing. As LMS design shifts from technology-centric to user-centric, developers need to prioritize user experience (UX) to improve their LMS. LMS as part of an e-learning system can benefit from UX research to measure ease of use and user satisfaction. Many academic institutions around the world prefer to use their own custom-ized LMS, such as: B. Moodle is an open source LMS platform built and main-tained by most Jordanian universities. Therefore, UX evaluation and measurement is very important for LMS. Several studies have been conducted to ana-lyse and measure LMS products for user feedback. On the other hand, these studies only partially consider LMS products. Instead of observing the quality of LMS, they conducted a study focused on comparing LMS and gaining awareness. The purpose of this study is to collect comments and feedback and evaluate the UX when dealing with LMS. Using Moodle as a case study, this study evaluates the user experience of LMS. Data were collected from 867 participants from various Jordanian institutions using LMS. An online survey was prepared and distributed through various social media groups. This study pro-poses to use the UX Survey Tool (UEQ) to evaluate and quantify the user experience of LMS. This study aims to get feedback and evaluate and identify the UX while interacting with the LMS. In general, implementing a user experience questionnaire to evaluate and evaluate an LMS provides not only complete user awareness, but also the relative quality level of the LMS. The results provide feedback and support assistance in the implementation of the LMS.


Introduction
In 2019, COVID-19 first appeared in Wuhan, China, and was quickly proclaimed a pandemic as it spread over the world due to high infectious rate. Over 130 nations and territories had confirmed the presence of the Coronavirus when cases emerged in mid-March 2020, according to the [1]report. Due to the COVID-19 crisis, governments all over the globe enacted policies and regulations such as social distance, isolation, and self-isolate [2], leading to unprecedented economic and psychological consequences. While millions of individuals worldwide remain at home to forestall the spread of the Coronavirus, their livelihoods and, in the case of students and learners, their access to educational resources has been affected. During the Coronavirus pandemic, it is essential to investigate the enormous changes brought about by the advancement of information technology in all aspects of life, particularly its involvement in higher education. Even the most fundamental errands, such as conventional learning, have always benefited from technological advancement. A technology that falls under the category of e-learning has made it possible to continue learning during the lockdown [3].
Using a learning management system (LMS) in the learning process promotes elearning by delivering teaching and learning resources regardless of location or time [4] [5], allowing students and instructors to interact via the internet, and facilitating the sharable course content and resources [6] [7].
This implies that employing this technology throughout the COVID-19 pandemic is important to maintaining the learning process. LMSs used at educational institutions include Moodle, WebCT, Blackboard, and Desire2Learn [8] [9].Moodle is largely viewed as the most extensively used learning management system in academic and business organizations worldwide. It has a total of 291 million users in 241 countries [10]. Although reasonable improvements to Moodle UX have been released as early as the end of 2019, [10]. [11]discovered UX problems in Moodle's UI that prohibited users from engaging with it properly. Design, the internal search function, content, linkages, ease of use, and speed were the four areas for UX issues. As indicated by [12], based on a systematic literature review, no study in usability and UX of LMSs offered solutions to the identified issues in usability and UX of the investigated LMS. Subsequently, the present research will address this gap by performing a pragmatic study that incorporates users' feedback and suggestions on an LMS's UX to give feasible solutions to recognized difficulties in an LMS's UX and usability.

Moodle learning management systems
Martin Dougiamas designed the initial Moodle prototypes in 1999, and Moodle 1.0 was released in August 2002 [10]. A LMS is a sort of e-learning system [13]. LMS is a integrated modules software that is used to improve learning process [14]. A learning management system (LMS) automates course material distribution and student learning progress tracking [15]. LMSs are classified into two types: open-source and closed-source. Open-source LMSs are frequently free and customizable at a low cost depending on user preferences [16]. Furthermore, lecturers may use Moodle-based elearning systems to improve their students' learning experiences. Moodle is a wellknown LMS in higher education [17] [18]. Moodle is now largely regarded as the most extensively utilized learning management system in academic and business organizations across the world. It has a total of 291 million subscribers in 241 countries In 2019, COVID-19 first appeared in Wuhan, China, and was quickly proclaimed a pandemic as it spread over the world due to high infectious rate. Over 130 nations and territories had confirmed the presence of the Coronavirus when cases emerged in mid-March 2020, according to the [1]report. Due to the COVID-19 crisis, governments all over the globe enacted policies and regulations such as social distance, isolation, and self-isolate [2], leading to unprecedented economic and psychological consequences. While millions of individuals worldwide remain at home to forestall the spread of the Coronavirus, their livelihoods and, in the case of students and learners, their access to educational resources has been affected. During the Coronavirus pandemic, it is essential to investigate the enormous changes brought about by the advancement of information technology in all aspects of life, particularly its involvement in higher education. Even the most fundamental errands, such as conventional learning, have always benefited from technological advancement. A technology that falls under the category of e-learning has made it possible to continue learning during the lockdown [3].
Using LMS in the learning process promotes e-learning by delivering teaching and learning resources regardless of location or time [4][19] [20],allowing students and instructors to interact via the internet, and facilitating the sharable course content and resources [6] [7].
This implies that employing this technology throughout the COVID-19 pandemic is important to maintaining the learning process. LMSs used at educational institutions include Moodle, WebCT, Blackboard, and Desire2Learn [8] [9].Moodle is largely viewed as the most extensively used learning management system in academic and business organizations worldwide. It has a total of 291 million users in 241 countries [10]. Although reasonable improvements to Moodle UX have been released as early as the end of 2019, [10]. [11]discovered UX problems in Moodle's UI that prohibited users from engaging with it properly. Design, the internal search function, content, linkages, ease of use, and speed were the four areas for UX issues. As indicated by [12], based on a systematic literature review, no study in usability and UX of LMSs offered solutions to the identified issues in usability and UX of the investigated LMS. Subsequently, the present research will address this gap by performing a pragmatic study that incorporates users' feedback and suggestions on an LMS's UX to give feasible solutions to recognized difficulties in an LMS's UX and usability.

User experience (UX) and usability
The attractiveness, usability, and satisfaction with a product or system are all factors in user experience assessment. The evaluation of user experience (UX) in mobile applications is seen as a promising topic of research in the human-computer interaction (HCI) field [21]. The primary goals of UX evaluation are to identify the effectiveness, efficiency, and satisfaction of the user as well as issues that may lead to human errors and user frustration [22] [23]. The UX evaluation of mobile device applications is slightly different from desktop computer applications referring to unique features of mobile devices, such as limited bandwidth, small screen interface, and changing mobile context (i.e. location and limited memory). Thus, the methods and guidelines proposed for the evaluation of desktop applications may not be directly applicable to mobile applications [24] [25].Users' feedback on the UX of products or systems, such as websites, software, devices, or applications, can be collected using a variety of approaches such as usability test and questionnaires. Different UX quantification methods have been published. Usability test by [26]is the most prevalent, where the UX quality quantitative indicators for a product are measured by many detected issues and the time needed by participants to complete certain tasks. Nonetheless, this method holds some drawbacks since it is time and budget-consuming: participants' recruitment, preparing test plan, materials, logistics, and booking test facilities. Thus, this testing method usually is conducted on a small scale (10-15 users). Standardized UX questionnaires, which are carefully constructed and not arbitrary, ensure that the intended UX qualities are measured accurately [27]. This method is an efficient and popular tool that can assess all sorts of quality measurements and gather feedback from a large sample size over a short period. There are several similar questionnaires available. Questionnaires about pure usability, for example, are specified in [28] [29]. In their research [30][31] [32], they describe questionnaires that cover a broad range of UX topics. Each questionnaire has several scales that cover and measure various UX aspect categories. As a consequence, the best-fit questionnaire is determined by the objective of the study as well as the UX dimensions that must be evaluated and assessed. For larger assessments, it may be necessary to use more than one questionnaire. We used UEQ [33]and its expansions [34]to quantify UX measurement for LMS in this study.

Methodology
The Moodle UX assessment test included the distribution of a post-test questionnaire to participants, which was available in both Arabic and English editions. A representative sample of users was enlisted to provide feedback on their experiences. The study respondents divided into two groups to include users (students and lectures) and UX experts. The first group invited 867 users (401 males and 466 females) who came from diverse backgrounds. The participants are students and lecturers in public and private universities. The second group invited 7 expert were experienced in the LMS (Moodle). The study was facilitated because the questionnaire is prompted for participants while they are logging into LMS. In general, the first group of participants cooperated since they received encouragement to participate in the study because of its effect in contributing to the improvement of their usage experience. Thus, this eases the task of collecting data from the participants.

Experiment tool
Questionnaires are a popular tool instrument used by users to evaluate the consistency and user experience of software quality. They enable quantitatively measuring the features of software systems efficiently. Allowing end-users to immediately assess what bothers them the most is a highly effective approach to obtain meaningful feedback. For example, how did the product engagement seem, and how was the en-tire usage experience? The UEQ (User Experience Questionnaire) was utilized in his research [32] [35][34]expanded the UEQ for user experience assessment. The UEQ questionnaire is accessible in a variety of languages, including English, Arabic, and others [27]. UEQ combines the Attractiveness, Perspicuity, Dependability, Efficiency, Novelty, and Stimulation measures, as well as a survey of 26 contradictory adjective pairs. The 26 left items besides their polarity (i.e. positive or negative order of term per item). The UEQ has six components and 26 measures, such as attractiveness, which represents the overall perception of the product. Do users like or dislike the product? Perspicuity: Is it simple to get familiar with the product? Is it simple to figure out how to utilize the product? Efficiency: can users do their activities without exerting undue effort? Dependability: Does the user perceive that he or she has control over the interaction? Stimulation: Is the product fun to use and motivates you to do so? Novelty: Is the product novel? Is it unique and innovative? Is the product in the users' best interests? All participants were familiar with their responsibilities during their interaction with LMS since they were demanded to carry out specific tasks according to their roles. Lectures are demanded to create new files, material folders, quizzes, assignments, create sessions, and other related actions. On the other hand, students were demanded to download materials, join sessions, submit assignments, take quizzes and exams, and do other related actions.
The UEQ's elements are represented by two paradoxical expressions in the form of a semantic differential. The terms are ordered in a random order per item, with half of the constructs beginning with an affirmative expression and the other half with a negative expression. To reduce the central tendency bias for such items, UEQ employs a seven-stage construct scale. The scales range from -3 to +3. As a result, as alluded to in UEQ Data Transformation, The most negative answer is -3, the most neutral is 0, and the most affirmative is +3 [27].

Participants and context of the study
The Google form was distributed to several departments of undergraduate and postgraduate programs at different institutions that represent the researcher's universities. As a result, data was gathered from student and lecturer groups who had prior expertise and practice using Moodle. Data was collected during a four-week period in June 2021, with a total of 883 respondents completing the questionnaire. Sixteen (16) respondents were identified as not valid and were omitted remaining with 867 respondents. The expert group was selected from academic staff in usability and user experience research interest and from software development industry that specialized in UX design and evaluation from different software companies in Jordan. Experts contacted by email to ask them to participate in the assessment of Moodle UX. Seven experts accepted to participate in the study.

Results
The findings of assessing the data acquired from participants on their experiences with Moodle's features and usability on desktop/laptop devices and mobile/tablet interfaces are presented in this section.

The Cronbach's alpha consistency CAC
The findings of the Alpha coefficient values of the scales from the responder group are shown in Table 1. The responses from all respondents, as shown by the Alpha values above, are all greater than 0.7, showing that the items of all six UEQ scales are consistent with the responses. The following is the outcome of the UX evaluation: Tables 2 and 3 indicate the demographics of the two groups.  The vast majority of the students were between the ages of 18 and 23. Males made up 64.3 percent of the participants, while females composed 53.7 percent of the students. The majority of lecturers were between the ages of 36 and 41. Males made up the majority of the lecturers (13.2%), whereas females represented (7.3%). The majority of students participated (89 percent) (692 out of 778) had more than four years of experience using the Internet, and all participants utilized the Internet on a daily basis, whereas the majority of lecturers had more than six years of experience using the Internet. According to students' Moodle LMS experience, the majority (82%) 638 out of 778 had more than two semesters of experience using Moodle LMS, and the majority (91%) utilized Moodle regularly.

Moodle-LMS UX assessment-user group
The results of Moodle LMS user experience assessment among 26 items distributed among six constructs as mentioned earlier are shown in Figure 1. The construct items are listed alongside each structure in the table, along with the mean, standard deviation, and variance for each item. The representation of Figure 1 is illustrated in Table 4 for more clarification as the legend beside. It is clear that there is no item misinterpretation during the Moodle UX assessment since the entire measured item >0. It is obvious that none of the items on this scale have a negative mean and that they all have a high positive mean. This shows that there is no problem with this item in this context. According to the results shown in Table 4, 60.5% of participants believed that Moodle system was easy to learn, easy, understandable, and clear as corresponding to perspicuity (Mean=1.815 out of 3). The efficiency with mean (1.703 out of 3), the respondents were agreed that Moodle was efficient. According to attractiveness (pleasant, attractive, good, etc) 55.8% scored mean (1.676) (55.8%). Dependability and novelty of Moodle have issues from the user perspective, it is clear from the table above shows the results of UX scale percentage (48%;40.6%) with scored means (1.5;1.22) respectively. When the data is analyzed, it becomes evident that users rated Moodle as medium (attractiveness, perspicuity, stimulation, and efficiency) on all three scales, while ranking the program low on the other two (dependability and novelty). The UEQ scale was divided into three pragmatic quality dimensions: perspicuity, efficiency, and reliability. The stimulation and novelty dimensions of the hedonic quality component are grouped together. Since attractiveness is defined as the overall impression of the software application it is considered as individual parity dimension. The mean for the three groups of dimensions is shown in Table 5. The UEQ scales have been categorized into three main categories; Attractiveness, pragmatic quality (Perspicuity, Efficiency, Dependability), and hedonic quality (Stimulation, Novelty). The UEQ questionnaire considered that hedonic quality and pragmatic quality separate constructs of the user experience. The task-oriented goals of software UX design are at the heart of the pragmatic quality factor.
Users can do their tasks with greater efficiency and effectiveness when the pragmatic quality is higher. The non-task-oriented quality factors of a software UX design are at the core of the hedonic quality factor. Figures 2 and 3 of UX scales and quality factors show the mean of the six UX scales of quality aspects are measured as shown in Figure 3. In comparison to other scales, the perspicuity and efficiency scales have relatively high means (see Figure 2). The findings revealed that Moodle users have a positive experience with it and use it frequently. The novelty category, on the other hand, has the lowest Moodle mean score.

Fig. 2. Moodle UX scales
Attractiveness represents general attitude as a self-contained quality factor that has relative a high mean score relative to other categories. Dependability is a part of pragmatic quality also having a relatively good mean score while the novelty scored with the lowest mean. These results are rational since the goal of the user is to access Moodle is to learn and/or provide learning in a virtual environment as a matter of distance learning.  Table 6 shows the scale means' 5 % confidence intervals. The confidence interval reflects the precision of the scale mean is that estimated and indicating a high score for relevant UX scales. The smaller the confidence interval, the greater the precision of the estimation and the greater the likelihood that the results can be trusted.

Moodle UX comparison to benchmark
The UEQ benchmark data collection currently comprises data on 452 software product assessments utilizing the UEQ, according to [27]. Table 7 depicts the scale measurement based on the UEQ. Comparison to benchmark indicates for the UX quality assessment of Moodle on distributed over six scales as shown in Table 7. We can notice that the UX quality scales range between good and above the average of benchmark data set for efficiency, attractiveness, perspicuity, stimulation, and novelty. Dependability offers some modifications that should be considered in the forthcoming updates of Moodle. Figure  4 shows the benchmark graph for Moodle system. With a benchmark in place, it is reasonably easy to determine whether the Moodle system has adequate UX to be successful as a learning management system and to get a greater rate of acceptance and usage. The strengths and shortcomings of the Moodle system may be determined by comparing the UX results for the given scales with the results of the other software and applications in the benchmark. Since the available benchmark is derived from data of developed software and applications, Moodle systems should score at least a decent on all measures. The comparison to the benchmark is a first measure to determine whether a Moodle has adequate UX to gain higher acceptance and usage rates.

Expert group
The results of Moodle LMS expert experience assessment among 26 items distributed among six constructs as mentioned earlier are shown in Table 8. According to the results shown in Table 8, the experts confirmed that Moodle system was efficient as corresponding to efficiency (Mean=2.107 out of 3). The experts rated the system low regarding to novelty, this result confirmed with user assessment as the lowest rating for both of groups.

Comparison between mean of users and experts
The following Table 10 and Table 11 respectively show the differences between users and expert toward Moodle system.  Based on the tables (10 and 11) there is no different significance between users and experts, because p-value is more than 0.05. Based on the output above, Mean value of users score is 1.56. Mean value of expert score is 1.57. Moreover, the correlation values of user score with expert score is 0.746. The researchers can conclude that there is a strong relationship between user and expert score. The difference in the mean value between users score and expert score is -.018. The researchers conclude that the users mean scores were equal the mean expert scores. P-value of the test is 0.898. Thus, there is no difference significant mean between user and expert scores.

Discussion
Various UX scales results are compared to the applications in the benchmark, assumptions about the assessed application's strengths and weaknesses may be made. In this study, it is fairly obvious that attractiveness, perspicuity, efficiency, stimulation, and novelty are measured with Good indicators in comparison to the benchmark, whereas dependability is measured with Above-average, which is interpreted by 25% of the results in the benchmark becoming better than the evaluated product, as confirmed by [35]. The system's natural goal is to score in the good category on UX scales, or at least above average when compared to other UEQ scales [34]. A benchmark, along with a clear understanding of the value of UX quality elements as measured by UEQ scales, can assist in the formation of clear and understood quality objectives for software product development [34]. Later on, the UEQ questionnaire might be used to verify these objectives.
The UX assessment of Moodle shows good means values concerning perspicuity and efficiency with overall means (1.82; 1.70) respectively. This finding is compatible with [36]who also stated that the informative LMS is determinant and important factor of student experience of LMS. According to conducted UX assessment, Moodle system is easy to understand and can be used efficiently [11]. In other words, it shows good means values considering all pragmatic quality scales except the dependability [37] with a scored mean (1.44) above the average compared to other UEQ benchmarks as confirmed by [38]. Despite the low means score of stimulation (M=1.51), and novelty (M=1.22) as identified by [38], it is clear with respect to the UEQ benchmark that Moodle showed stimulation and novelty are scales with good when are compared to the other evaluated application in the available benchmark dataset.
The assessment results of hedonic quality which are represented by novelty and stimulation are not encouraging with a scored mean (M=1.36), so it needs an extensive effort from the developer and UI/UX designer to enhance creativity and sufficient capturing of user attention. Novelty as a part of hedonic quality express the user perspective in term of creative, inventive, leading-edge and innovative is identified by [27]scored a lowest mean score. This indicates further consideration to the UX designer to give more attention to UX guidelines for novelty scale items. This also applies to stimulation which is represented by if the Moodle is valuable, exciting, interesting, and motivating.
The score of stimulation (exciting, interesting and motivating) is not promising, so infusing Moodle's user experience with some fun will yield fruitful results in improving the stimulation score [39]. The UEQ benchmark comparison is an important measure for determining if Moodle provides a satisfying and successful user experience. Because the benchmark was built by collecting data from all accessible UEQ assessments, comparing the Moodle system to the available benchmark data set was deemed adequate for gauging UX by a broad representative sample of users. Fast revision for the scales dimension (mean and percentage) will guide to some areas of improvement that should be taken into consideration by Moodle UX design teams. The topics for development include Moodle's dependability, Moodle's UI in terms of appeal, Moodle's attractiveness as represented by innovativeness, and Moodle's creativity. The latter two are seen as more vital than the former. UX designers of LMSs must give more attention to the LMS user interface design and enhance attractiveness of Moodle with a comparison with other learning platforms and the motivation of Moodle by users. As a conclusion for the above discussion, users prefers Moodle because the user interface (UI) is simple, the Moodle content and features is appears in the same page with simple interlinking between pages and sub-modules [40]. As [41]stated that Moodle is the most a user-friendly e-learning system among some of open source LMSs.

Conclusion
This paper discusses user experience-based evaluation and measurement of a learning management system. The paper explained what a user experience questionnaire is and how it can be used to evaluate a product. The tools were chosen as techniques because they are simple, efficient, and give broad information that may be used to fully define a product. The information generated from the data is sufficiently valid because the correlation value validates consistency. The tools do create a map of a product's relative quality in comparison to other products. It is useful for more indepth evaluation analysis, particularly when undertaking competitor analysis.
According to the results, the majority of participants were pleased with Moodle's efficiency, attractiveness, and perspicuity. On the other hand, the UX assessment of Moodle shows good means values concerning perspicuity and efficiency with overall means. By comparing an application's present level of user experience to a wide number of other apps, the previously stated benchmark gives an additional chance to as-sess if an application's current level of user experience is acceptable. The benchmark only offers a high-level assessment of the state of an accepted application by health authorities and organizations, and it should be supplemented by comparisons to direct competitors to gain a more complete perspective. According to conducted UX assessment, Moodle system is easy to understand and can be used efficiently. In other words, it shows good means values considering all pragmatic quality scales except the dependability compared to other UEQ benchmarks. Despite the low means score of stimulation and novelty but it is clear with comparison to UEQ benchmark that Moodle showed stimulation and novelty are scales with good when are compared to the other evaluated application in the available benchmark dataset.The results show that the UX design relies on how educational institute design Moodle. Therefore, Moodle must be by designed by user interface design experts and UX professionals, and then the UX might be assessed positively by the Moodle users.