Modeling User Psychological Experience and Case Study in Online E-Learning

— In the post WWW era, the research of e-learning focuses on facilitating intelligent and proactive services for learners. The quality of user experience determines whether e-learning services would be accepted by learners. However, many researchers traditionally focus on the effectiveness of computer systems or the accuracy of algorithms themselves rather than on user-centric psychological experience. How to model and evaluate user experience taking into account user psychological and cognitive properties are challenging research topics. There are some traditional methods typically proposed to evaluate users’ psychological experience, such as interview, questionnaire etc. They are qualitative and easy to conduct but need more time and resource. And they are liable to subjective views. Based on user web log data, the current paper presents a quantitative approach of modeling user psychological experience in the context of intelligent e-learning. The properties and elements, which affect user experience, are analyzed and quantified. The holistic user experience is quantified through the fusion of analytic hierarchy process (AHP) and Delphi methods. A case study, at a university in China, is conducted for diagnosing whether the result of the proposed approach can be uniform with user subjective experience, and indicates that the proposed approach is effective and complements existing user experience research in intelligent e-learning.


INTRODUCTION
Nowadays information and computer technologies provide various services which facilitate human's work, learning and life, and experience economy is emerging. Whether users are satisfied with the services, have good experience and are willing to take the services perpetually is an important problem which attracts much attention in diverse fields [1][2][3][4][5]. The current paper presents a quantitative approach of evaluating user psychological experience in the context of smart education, such as intelligent elearning.
User experience, which is a holistic and subjective psychological perception, is built by a user during Web systems provides services for the user, such as search information, video and audio requisition, e-learning etc. It is an ill-defined concept, and lacks well-developed assessment methods and metrics [2,6]. Hosts of research have been conducted in diverse domains and from different views respectively [7,8].
There is a range of properties that are commonly considered when assessing Web systems' user experience. How to model and evaluate user experience taking into account user psychological and cognitive properties is a problem [1,3,7].
Some promising approaches have recently attempted tackling the problem. Hu et al. provide a broad and indepth review of existing work and 20 useful guidelines from the initial preference elicitation process, the preference refinement process and the presentation of the system's recommendation results [1]. Paramythis et al.'s work shows many evaluation methods available in the field of Human-Computer Interaction (HCI) for the evaluation of interactive adaptive system [4]. In the paper, we try to apply data mining technique for evaluating user psychological experience in the context of online elearning.
Based on the theory that users' objective behavior reflect their psychological and cognitive properties [9,10], a quantitative approach is proposed to evaluate user psychological experience based on users' web visit log data in intelligent e-learning systems.
There are three critical issues as follows: (1) Identify properties and elements that may affect user experience. There are a variety of properties and elements that may influence user experience, such as content construction form [1,12], vision affection [10], system delay [1,13], ease of use, and so forth. As different applications have different needs, we analyze which properties and elements are relevant for intelligent e-learning applications. (2) How to measure the properties and elements above according to user web log data. In the paper, some quantitative methods are proposed to measure them. (3) Construct holistic user psychological experience model. The analytical hierarchy process approach is applied to construct the overall quantitative user experience model and the weights of the elements are obtained.
Additionally, because of the differences among the individuals, different users have different user experience to the same system. Lindgaard and Chattratichart's study found that the representativeness of users impact the results of study directly [11]. Moreover, due to the limit of time, manpower and material recourses, it is impossible to measure user experience of every user, so how to determine the number of representative users to study is another key problem which user experience research faces.
The rest of the paper is structured as follows. First, the paper surveys the state of the art user experience research in diverse fields and provides an extensive overview of properties and elements that may be relevant for user experience. Second, the paper discusses how to choose representative users and how to determine the number of PAPER MODELING USER PSYCHOLOGICAL EXPERIENCE AND CASE STUDY IN ONLINE E-LEARNING them (i.e. sampling size). Following that, the paper presents the proposed quantitative approach to evaluating user psychological experience in smart education, such as intelligent e-learning. Finally, the experiments and suggest guidelines are discussed for improving user experience.

II. RELATED WORK
The part first surveys the state of the art of user experience research in diverse fields and then discusses how to choose the representative user and how to determine the number of them.

A. User experience research in diverse fields
User experience is a holistic psychological perception and includes both pragmatic characteristics(such as ease of use) and hedonic characteristics(such as appeal) [2]. There is a vast amount of user experience research in diverse fields and from different views respectively, such as computer science, psychometrics and social informatics fields, which have studied the quantitative and qualitative properties and elements of user experience [11][12][13][14].
To improve the design and evaluation of computer systems, James Lewis develops standardized subjective usability satisfaction measures (IBM Computer Usability Satisfaction Questionnaires) which has the components of user satisfaction, such as system usefulness, information quality, interface quality, to assess users satisfaction with system usability [15]. Jetter and Gerken propose new perceptions which go far beyond the pragmatic concepts of pure functionality or ease of use etc., such as hedonic quality like "attractive", "exciting" and "joy of use" etc [11]. Rodden etc. describe large-scale user-centered behavioral metrics framework, HEART: Happiness, Engagement, Adoption, Retention, and Task success [4].
Properties that affect user experience include as follows: (1) accessibility or functionality, which means users feel ease to use a system. The property comprises other concrete, detailed elements, for instance response speed (which addresses that users can access to the service rapidly and easily), easy of navigation (which means that users can obtain the service through the shortest route) and simple manipulation etc., (2) usefulness of information quality, which means that users can obtain valuable materials from a system. The property comprises other concrete, detailed elements, such as satisfying users' needs, improving work efficiency etc., (3) other properties, such as users' characteristics, emotion states etc. Users who have different characteristics or emotion states may feel different experiences for a system. In the paper, we focus mainly on the properties in (1) and (2) above, which refer to pragmatic characteristics.
In human computer interaction and interface field, the methods are used commonly, such as observation, interview, questionnaire and role action etc., to analyze users' needs and habits [4,16]. The methods need more time and resources, and are usually less generalizable and cannot be statistically validated [2].
There are some quantitative methods in the literature, such as GOMS CPM-GOMS, NGOMSL [9], which emphasize operate time. GOMS method constructs Goals, Objects, Method and Selection rules model, and disassembles users' behavior into behavior unit. It can evaluate the time cost in the specific scenario that an experienced user conducts. The model is simplified and CPM-GOMS model improves it, which is suitable for evaluating the complicated, overlapping and time-depended interface. GOMS model is suitable for describing and forecasting an experienced user's usual, skilled conduct and is not for a fresh user, but also focuses on the time efficiency of users' mechanical conduct in specific scenario, which does not consider the impact of human's endurance and emotion on user experience.
Tanenbaum etc. present a mixed method to study user experience of adaptivity which includes a wide variety of data, such as pre-and post-interaction surveys, postinteraction interview, video of the participants using a system, and log data generated by a system itself [16]. It is a valuable way of recovering the information of user experience from system logs. Our paper tries to go further on the way and proposes a systemic and quantitative approach to evaluating user experience from log data.

B. User sampling review
User experience research focuses on users, but studying thousands of users will cost a great deal of human resources etc., and a deviation will also be easily caused without good management [18]. So how to use scientific and systematic methods to choose representative users with the appropriate size is a problem. This section firstly analyzes and discusses the problem, called user sampling, which focuses mainly on how to choose representative users and how to determine the size of the representative users to represent the real situation of the whole user population.
Sampling methods allow researchers to use a subset of the studied objects (i.e. population) to explore the whole population [19]. This technique can solve the above problem. Of course, if the total amount of the population is no more than 50 and the researchers are able to support the cost of resources and control the deviation well, investigating all the members of the population is much more suitable. But if users' data are massive, using sampling methods would provide a very effective solution. In this way, processing massive data can be more efficient and cost less.
How to sample users to reduce the differences between the conclusions of the samples and the facts of the population to the greatest extent is the issue of sample designing. Sampling theories in statistics science provide solid mathematical and theoretical foundation for solving the issue. Sample designing focuses on two questions: (1) how to choose the samples used to represent the population, (2) how to judge whether the samples represent the population well or not [17].
Social survey fields have in-depth research on user sampling and provide many methods, such as simple random sampling, systematic sampling, stratified sampling etc. These methods have different advantages and disadvantages [19]. If the sampling method is selected improperly, it will cause the errors [18].
Statistically, since sampling design should solve different problems, the methods to calculate the sample size (i.e. the number of samples) are also different [20]. If using the estimation of the proportion method to single population and random sampling with replacement, the calculation of the sample size is shown in the following formula.

PAPER MODELING USER PSYCHOLOGICAL EXPERIENCE AND CASE STUDY IN ONLINE E-LEARNING
represents the number of expected samples, Z is the fractile of standard normal distribution, is the proportion of population, represents absolute error. If using random sampling without replacement method, the finite population correction factor (FPC) must be taken into consideration [19], the calculation method is shown in the following formula.
represents sample size, represents population size.
In addition, if great accuracy is not required, the approximate range of sample number can be determined by experience according to the scale of population [20].

III. A QUANTITATIVE APPROACH TO EVALUATING USER PSYCHOLOGICAL EXPERIENCE IN INTELLIGENT E-LEARNING
The paper proposes a quantitative modeling approach to evaluate user psychological experience in smart education, such as intelligent e-learning. The approach involves in user experience properties and elements selection, user sampling and web log data fusion etc. This section first overviews the approach and then discusses the key issues.

A. The proposed modeling approach overview
The quantitative approach to evaluating user psychological experience in the paper first analyzes and builds properties and elements that may affect user experience, and designs quantitative methods to measure the properties and elements through web log data fusion, and proposes some user sampling methods to calculate user psychological experience. The architecture of the proposed modeling approach is depicted in Figure 1.
The architecture of the proposed modeling approach focuses on solving three key issues of user experience respectively in the context of e-learning. The issues are as follows: (1)Analyzing properties and elements that may affect user experience; (2)Measuring the properties and elements according to user web log data; (3)How to choose the representative user and how to determine the number of them.
As discussed in sections above, there are a large body of work on the properties that may affect user experience, but they may be too generic to apply to a particular research question [5], such as intelligent e-learning. So, based on the generic properties, we first sought to propose and analyze some properties which are suitable for intelligent e-learning context, such as usefulness and ease of use. The properties, usefulness and ease of use, can be decomposed, respectively, into more detail elements which can be quantified based on user web log data fusion. The usefulness property is decomposed into three elements which are coverage of the resources, recommend hit rate and user loyalty. The property, ease of use, is also decomposed into three elements which are response speed, ease of navigation and efficiency of task. Then the analytic hierarchy process is applied to the properties and elements for identifying the weight of importance among the index- Meanwhile, user sampling strategies and methods are provided for selecting the representative users and the appropriate volume of users to ensure the quality of user data and to eliminate the noise data.
Finally, the holistic user psychological experience is modelled based on the hierarchical structure and the quantitative methods of properties and elements along with user sampling strategies and methods.

B. The analysis and quantitative methods of properties and elements
As discussed in sections above, there are many properties and elements that affect user experience, but usefulness and ease of use are the most important properties for intelligent e-learning. So the paper mainly focuses on usefulness and ease of use. The property, usefulness, refers to whether the services, provided by an e-learning system, are useful for users and can facilitate learning. The property, ease of use, refers to a user can get a services, provided by an e-learning system, quickly and easily, and the system is simple to operate, easy to learn, ease of navigation and meets the user's habits, etc.
But the properties can not be described and quantified directly through user web log data in e-learning systems. We need refine the properties to draw the elements which can be quantified, directly, based on user web log data. We suppose the usefulness property of an e-learning system is coined by the three elements: coverage of resources, recommend hit rate and user loyalty. The property, ease of use, is coined by the three elements: response speed, ease of navigation, efficiency of task. The properties and elements are organized in hierarchical structure shown as Figure 2.
To clarify the paper, we defined some terms first.
Knowledge object . An e-learning system provides knowledge objects for users to learn. A target knowledge PAPER MODELING USER PSYCHOLOGICAL EXPERIENCE AND CASE STUDY IN ONLINE E-LEARNING object is the one that a user must learn for completing a learning task. The set of target knowledge objects is defined as .
Learning time refers to the interval from the moment that a user clicking on a knowledge object to the moment that the user clicking on the next knowledge object . We have defined the interval as the duration that the user has learned the knowledge object .

Minimal time
is the shortest time that a user spends on learning the target knowledge object . And are the minimal times of the target knowledge objects , respectively, which can be defined according to experts' experience or can be calculated from system web log data by statistical learning methods.
If learning time that a user actually spends on learning an knowledge object is longer than the minimal time , we suppose the user learns the knowledge object successfully and define shown as formula 1.

(1) Task success
. Suppose is the minimal number of knowledge objects that a user must learn for completing a learning task.
can be defined according to experts' experience and will be different values in different learning tasks. If knowledge objects, which a user has learned, belong to the set of target knowledge object , we think the user has completed the learning task successfully. Task success ( ) is defined as formula 2. (2) Task complete time . It is the duration from loading a system to completing a learning task and is the sum of time that a user has spent on learning all knowledge objects during he/she completed the task. It is defined as formula 3.

(3)
The login time of a user refers to the time that a user logins an e-learning system and it can be obtained from an e-learning system logs. The start time of a learning task, , refers to the time when a system has finished loading and has begun to provide services for a user. can be also obtained from a system web log data.

Response time
refers to the duration from the moment of user requesting a service in an e-learning system to the moment that the e-learning system provides the service.
is defined as formula 4.
We have defined as the benchmark of the response time.
can be measured according to user web click log data.
The three elements about the usefulness property are discussed in more detail below.
(1) Coverage of the resources refers to the extent of resources provided by an e-learning system and whether which is enough for the needs of users.
(2) Recommend hit rate. If a user has clicked on a knowledge object recommended by a system for the user, we think that the recommend is hit. The recommend hit rate is defined as the ratio of recommend hit times and recommend times. Assume the times of recommending knowledge objects for a user in an e-learning is and the times of recommend hit is , then recommend hit rate, , equals the ratio of and , shown as formula 5.
(5) (3) User loyalty. If a system is useful for a user, the user should visit the system again. User loyalty is defined as how strong willingness a user has to visit a system. It can be measured by calculating the frequency of users' visit from user web logs data. It is defined as formula 6. Define the benchmark times as and if a user used a system times in a week, user loyalty equals the ratio of visit times and benchmark times. If is more than 100%, then take user loyalty as 100%. (6) In an e-learning system, we use the following elements to reflect ease of use property.
(1) Response speed. It can be calculated according to the response time . The response speed ( ) is calculated as formula 7.
(7) (2) Ease of navigation. The is the learning contents which a user has to learn in an e-learning system. If a user had learned knowledge objects when he/she completed the task, smaller is and clearer the navigation of the system is. Ease of navigation ( ) is the ratio of and shown as formula 8.
(8) (3) Efficiency of task. It is defined as the ratio of the time that a user has spent on the target knowledge objects and the time spent on completing the overall learning task. Efficiency of task ( ) is shown as formula 9.

C. Building the holistic user psychological experience quality
User psychological experience is the holistic subjective psychological feeling of a user who visits an e-learning system. According to the analysis in section 3.1, the holistic user psychological experience can be decomposed into usefulness and ease of use, and both of them can also be decomposed into more detail elements. The details are shown as Fig.2.
The hierarchical user psychological experience has three levels. The first level is the holistic index of user psychological experience. The second level includes two indexes, which is usefulness and ease of use. The third level includes six sub-indexes. The usefulness index is decomposed into three sub-indexes which are coverage of the resources, recommend hit rate and user loyalty. The ease of use index is also decomposed into three indexes which are response speed, ease of navigation and efficiency of task.
The weights of the above indexes have identified based on the fusion of AHP (analytic hierarchy process) [22] and Delphi methods [23,24]. The comparative weights among the indexes have been acquired through AHP method. To deminish experts' subjective influence to a minimum, we introduce Delphi method for complementing AHP shortcoming. Refer to [23,24] for the details.
If there are sub-indexes, such a matrix A can be obtained.
Within the index of usefulness, the following matrix (see Table 1) is obtained to compare the importance of each pair of the sub-indexes, which are coverage of the resources, recommend hit rate and user loyalty. The Delphi method has been applied to the acquirement of the weight values in the comparing matrix [23,24]. The item values in the comparing matrix denote the relative weights among the properties and elements. The value, 1/3 in the second row in Table 1, denotes that the weight of the element, recommend hit rate, is one third of the weight of another element, coverage of the resources. The value, 3, in the third row in Table 1, denotes that the weight of the element, user loyalty, is three times of the weight of another element, coverage of the resources. Other values in comparing matrix have the same meaning like the above. Within the index of ease of use, such a matrix (see Table 2) is obtained to compare the importance of each pair of the sub-indexes, which are response speed, ease of navigation and efficiency of task. The Delphi method has also been applied to the acquirement of the weight values in the comparing matrix. The item values in comparing matrix have the same meaning like Table I.  The comparing matrices are obtained from different experts' suggestions through Delphi method, so the weight values from different experts are probably inconsistent. The consistency of the matrices should be checked.

Supposing
is the maximal eigenvalue of the comparing matrix and will be greater than if the matrix is inconsistent. The greater is, the more inconsistent the matrix has. The consistency index ( ) is computed according to the following formula [25].
(10) As greater can result in a smaller , the random consistency index(RI) is used to eliminate the impact of on . Its value is shown as Table III [25]. Finally, the consistency ratio(CR) is used to judge whether the comparing matrix is consistent. (11) When is less than 0.1, we can treat the matrix as consistent [22,[25][26][27][28]; otherwise, we should revise the matrix and check its consistency again until is less than 0.1. After this, the eigenvector, corresponding to the maximal eigenvalue , is obtained. The eigenvector is normalized and the weight value of each sub-indexes is obtained.
The comparing matrix of usefulness is shown as Table   1 and the maximal eigenvalue is . According to formula 10, we can obtain . As obtained from Table 3, . Because , the matrix is consistent. The eigenvector corresponding to is . After normalization, we have defined as the weights of sub-indexes respectively, which are coverage of the resources, recommend hit rate and user loyalty.
As for the comparing matrix of ease of use, the maximal eigenvalue . According to formulae 10 and 11, and the matrix is consistent. , which are the weights of sub-indexes, which are response speed, ease of navigation and efficiency of task.
The weights of the indexes are obtained, which are in the third level of the hierarchical structure of user psychological experience. Next we discuss how to obtain the weights of the indexes in the second level of the hierarchical structure of user psychological experience. The comparing matrix of the indexes in the second level is shown as Table IV. Then the maximal eigenvalue is . According to formulae 10 and 11, , the comparing matrix is consistent. And the normalized eigenvector is .
The matrix denotes the weights between second lever and third level in the hierarchical structure of properties and elements shown as Figure II .
The matrix denotes the weights between first lever and second level in the hierarchical structure of properties and elements shown as Figure II .
The final weight vector is . Then the composed consistency check can be done as the following steps [28].
The random consistency index is computed as following.
The composed consistency ratio of the third level is computed as follows.
The composed consistency ratio of the third level to the first level is computed as follows.

As
, the result shows that the consistency of the matrix is acceptable.

IV. CASE STUDY AND ANALYSIS
We applied the quantitative approach to an e-learning system, in a university of China, for evaluating user experience quality and diagnosed whether the results of quantitative approach can be uniform with user subjective experiences.

A. User experience case study
In this section, we first analysed the size and representative of user samples, and introduced multi-source web log collection and fusion methods to obtain the data for studying the quality of user experience.

1) Sampling users
As describled in sections above, we reviewed the methods of user sampling. Sample designing focuses on two questions: (1) how to choose the samples used to represent the population, (2) how to judge whether the samples represent the population well or not.
Users of an e-learning system are those who have online learning conditions and learning needs, so the paper chose undergraduates in a university of China who are the direct users of an e-learning system and have universal representation. The user experience research of the paper involves in the subjective feelings of users and it is important whether the users are pleased to use the system, so recruiting users in the voluntary way is appropriate.
According to the characteristics of an e-learning system, the number of user sampling is calculated by using the random sampling without replacement method. The coverage rate of user sampling is 95% and Z=1.96 and is unknown and taking it to 0.5 generally [18,19]. In this way the population variance is maximum and the value of the sample size will also be maximum. Taking absolute error, , to 5%. On the basis of the above parameters, if using the random sampling without replacement method, the population , , and users sample size is 90. It means we can explore the whole user population by researching no fewer than 90 users.
We have recruited 108 participators and have reconducted the experiments again. The size, 108, is greater than 90 and is the appropriate size of the sample.
The following algorithm is applied to extracting the learning behavior sequence.

2) Quantifying user experience quality
In the experiment, participants were asked to complete a task: Using an intelligent e-learning system individually to complement the knowledge about circuit switching and packet switching in the course of computer network in the Spring semester 2012.
The set of target knowledge objects is defined as =(packet switching, message switching, packet, line switching, data exchange technology, packet switched, message switching mode characteristics, advantages of packet switching, message switching property, method of packet switching, packet switching process, contrast circuit switching with message switching and packet switching, packet switching property, circuit switching, switching technology).
In the experiment, suppose the minimal time is half of the average time spent in a knowledge object, shown as Table V.
According to sections above, the hierarchical structure of user psychological experience quality includes six elements: coverage of the resources, recommend hit rate, user loyalty, response speed, ease of navigation and efficiency of task. The weights of them are calculated and the weight vector is shown in section 3.3.
At last the holistic quality of user psychological experience is quantified.
The holistic quality of user psychological experience for the e-learning system is shown in Figure 3. The value of 100% denotes the best quality of user psychological experience. In the case, the highest value is 62%, the lowest is 27%, and the average is 38%. The result uncovers that the quality of user psychological experience of the system is not high.

B. Comparison with traditional methods
As users' objective behavior reflect their psychological and cognitive properties [11,12], the paper proposes an  Figure 3. Curve of quality of the holistic user psychological experience automatic quantitative approach to evaluating user experience according to users' objective behavioral logs. However, the traditional methods of user experience research field, such as questionnaire, interview etc., are mainly applied to evaluate user experience, so it should be taken into account that whether the evaluation result obtained by the approach proposed in the current paper is coincident with that obtained by the traditional methods. The rest of the paper discusses the problem above. The approach proposed in the paper and the traditional methods are applied to evaluate user experience simultaneously, then the correlation and consistency between both are analyzed.
There are 110 participants, in a university of China, recruited to visit an intelligent e-learning system and to fill out user questionnaires after they finish visiting. A total of 110 questionnaires were retrieved and two of them were invalid because the results provided by users were selfcontradictory. Firstly, according to the 108 valid question-, PAPER MODELING USER PSYCHOLOGICAL EXPERIENCE AND CASE STUDY IN ONLINE E-LEARNING naires, the traditional method was adopted to obtain user experience of the intelligent e-learning system and is called as "subjective-survey quality of user experience". Secondly, according to the visit logs data of 108 users, the approach proposed in the paper was adopted to obtain user experience of the same system, and is called as "objectiveevaluation quality of user experience".
Following, the comparisons were done between "subjective-survey quality of user experience" and "objectiveevaluation quality of user experience". In Fig. 4, ordinate represents the quality of user experience and abscissa stands for user identification while blue diamond represents the quality of user experience obtained according to the approach proposed in the paper and the red round stands for the quality of user experience obtained according to the traditional method, such as questionnaire. The five-level error is ±10% presented in the red rectangle which shows to most users that the quality of user experience, obtained according to the approach proposed in the paper, is close to that obtained according to the traditional method. Then the correlation was analyzed between objectiveevaluation quality of user experience and subjectivesurvey quality of user experience by SPSS. The part of the results is displayed in Table VI, where Pearson correlation coefficient is 0.702, Kendall correlation coefficient is 0.624 and Spearman correlation is 0.743, which indicate that the results of both is significantly correlated. Last, the consistency of both was analyzed and the results are showed in Table VII. The observed consistent rate of both is 81.5% and the corresponding Kappa coefficient is 0.668, indicating that both have moderate-intensity consistency. And the average absolute error is 8.7%, less than 10%, showing that the result, obtained according to the approach proposed in the paper, is close to that obtained according to the traditional method such as questionnaire. Also the former one can improve the timeconsuming problem of the latter one. V. CONCLUSION The paper analyzed the properties and elements which affect the quality of e-learning user psychological experience, designed the methods to quantify them and built the whole quantitative approach based on the analytic hierarchy process. These can inform the design and development of intelligent e-learning and complement existing user experience research methods. Now, usefulness and ease of use were mainly analyzed in the paper and the future work will focus on other properties (such as emotion etc.) and will construct a better quantitative approach to analyze the quality of user psychological experience in intelligent e-learning context.