Impact of Prompting Agents on Task Completion in the Virtual World

Based on a realized 3D virtual laboratory and virtual agents, this study was to investigate whether the virtual agents influence task completion. A total of 90 students participated in the experiment. They were evenly divided into three groups one for the control group with no intervention and the other two for experiment groups with different interventional levels. The results show students who used the virtual agents worked faster and more efficient than those who did not. The satisfaction of the experiment groups was superior to that of the control group. The correlation analysis showed a weak negative correlation between time in total and satisfaction (r = -.365), a strong positive correlation between speed and satisfaction (r = .626), and a strong negative correlation between time per task and satisfaction (r = -.685). However, correlation degree wasn't in concord with the interventional levels. This study suggests that virtual agents have promising effects on task completion and perceived satisfaction. Keywords—3D Virtual World, Prompting Agent, Task Completion, Satisfaction


Introduction
In the last decades, an increasing number of works used 3D virtual world(3DVW) to conduct simulated scientific experiment [1][2][3] with an intention that students not only get access to laboratory resources conveniently [4] but also they obtain real-like hands-on experience [5].The 3DVW provides strong immersion and fidelity, and has a promising outlook for learning [6].The current research projects have reported that such environments can effectively improve learning motivation, engagement, learning achievement and so on [1,7].
However, most students enter the 3DVW at the different time and in the different place.Usually, they are completely independent with less instructor's guidance and fewer learning companions [8].Due to the lack of presence, they are more likely to feel lonely, remain little interest and ultimately get unsatisfactory learning achieve-iJOE -Vol.13, No. 6, 2017 ments.To address this problem, researchers adopted prompting agents [9,10] to provide one-on-one tutoring.They have rich expression, vivid action and life-like image which could promote meaningful learning in lessons [11].Much literature has noted that the virtual agents are capable of enhancing interaction and emotion and improving learning interest and self-efficacy [12][13][14].
This study investigates how virtual agents influence the task completion of the novice users.This refers to measurements of individual's skill to accomplish tasks and subjectively perceived satisfaction under the virtual agents' intervention.The skill means the effort degree of task completion while satisfaction implies the active evaluation that influences the future usage.By exploring these two aspects, we expected to gain more insights on the effect of virtual agents and their pedagogical implications, thereby enlighten the design of more complex experiment activity henceforth.To do this, we develop virtual prompting agents to adapt to different challenges of tasks and measure students' responses.Our research questions are: 1. To what extent do students afford to effectively complete tasks in the 3DVW? 2. In what growth of perceived satisfaction do students exhibit in the 3DVW? 3. What is the relationship between task completion and perceived satisfaction?This is to identify if the higher intervention result in better task completion and more satisfaction.

Agents in 3DVW
3DVW uses computers to simulate the real situation and interaction.Not only do users construct information actively but also they become actors who play particular roles.Many research projects reported that the 3DVW enhances learning primarily due to the following characteristics.
1. Immerse-real experiences.Objects and their mode exercises are close to or even accord with the real situations.The students experience in real-like conditions that they feel immersed in the visual world as if being in the physical one [15].2. Flexible-interactive approaches.Students control tempo with immediate feedbacks, and their computer-animated gestures and expressions approximate to the so-called "face-to-face" communication [16,17].3. Hands-on learning experiences.Students create visual objects, actively engage in activities, and gradually achieve the goals [18].It is particularly used when it's impossible to conduct a real experiment or the conditions are happening frequently [19].
Many research projects target to bring real experience to the virtual world.Thus, they go beyond the border of physical environment and enhance the understanding of experimental process.One merit refers to the sense of being there which comprises not only the presence in the environment but also the experience of interacting with real and visual objects as well as the immediate feedback of the results [20].This environment integrates modeling and simulation to create a highly interactive envi-ronment [5] where a user can select, use, assemble virtual devices and modify their parameters through flexible manipulations.
The virtual prompting agent is also called animated pedagogical agent (APA) in some literature [10-13, 21, 22].The APA was described as a human-like or cartoon animated character which is displayed within a computer-based learning environment to provide learners with pedagogical assistance.The virtual agents can enhance teaching interaction, elicit learning interest and improve the learning achievement [6,7].From the constructivist point of view, teaching and learning are a learning context that provides students with various cognitive tools and learning materials, upon which they build individual meaning through interacting with the surroundings.In the meaning construction, virtual agents play a positive role which directs attention to relevant information, provides feedbacks or delivers methodological instruction [23].It keeps students focusing on essential information and provides context-specific learning strategies [24].

Research context
We prepared a 3DVW to conduct scientific experiments orienting to novice users.Three aspects are included.
Architectures and interactions.The realization was based on an open-source 3DVW called Open Simulator.The architectures included a hall, a meeting center, a showroom of experiment apparatuses, a reference room, a testing section, an electrical hall of fame, and an operating room.Each of them had scripted interactions such that one get immediate responses as he/she triggered the object of interest.
A course subject."The current in series and parallel circuits" in the part of Electrical Science of high school was incorporated.The experimental apparatuses including virtual powers, voltage meters, current meters, light bulbs and sliding rheostats were displayed in the showroom.All the items can be selected and recombined depending on specific tasks.
Virtual prompting agents.We currently developed several prompting agents to provide students with one-on-one tutoring.To this end, we divided the virtual laboratory into six scenes to accommodate different tasks progressively ranging from easy to difficult.The virtual agent in each scene provides information assistance such that users complete the task in the right direction.A sample screenshot is shown in Figure 1.To distinguish the difficulties, the first four tasks are about the technology literacy and the last two are about the course unit.These tasks were respectively the movement and interface control, communication, changing appearance, seeking information, location navigation, and creating.Correspondingly, the virtual agents were categorized into 6 classes namely control guidance, troubleshooting, character guidance, information guidance, navigation guidance, and creating guidance.The control guidance was adaptively positioned at where a user registered in the laboratory.It prompted messages at short intervals, and carried out the corresponding actions and demonstrated the visual effects.The troubleshooting agent monitored a user's proposed question, abstracted the possible keywords, and matched them with a preinstalled repository.If matched, the virtual agents give answers to the user.The character guidance was to coach students to change the character's appearance, such as height and weight, facial features and dress.The information guidance provided assistance associated with the experiment such as seeking experiment apparatuses and specific materials.The navigation guidance solved the problem of users not finding the way to a destination or not knowing whether or not they were on the right destination.This agent popped up a menu and prompted key information about how to travel the destination according to the user's selection.The creating agent supplied methodological help of hands-on operation, such as connecting experiment apparatuses and taking notes to record the experiment process and results.To complete these tasks it is important to establish a conversation between users and agents so that users can follow the instructions of the agents step by step The conversation between agents and users has two forms.One is the passive prompt.Whenever a user asks a question, he could enter a description of the question via a private media channel.The channel is specialized in the conversation between the agents and users and could be established by typing a simple "/0".Then, the agent analyzes the text, retrieves it in the knowledge base and displays the results in the interface.The analytics is based on a language processing script which obtains stems of the text.Sometimes, there may be a variety of uncertain answers to the question.The agent pops up a menu where the user selects a proper option to get a fine-grained presentation.The other form is the proactive prompt.Whenever a user registers in a new scene or does not act for a long time, the agent asks the user's states and show what to do at present through friendly prompts.Whether the information is prompted is determined by the user 's behavior rather than the text he has inputted.When students respond to the agents by doing something, the agent decides whether it is right or wrong according pre-established logics, and then feeds back to the user.
Students communicate with the virtual tutor when they feel it necessary.They are told to select the proper instruments in the showroom, bring them to the laboratory, and connect them in the right way.The task is completed if the light bulb is lightened; otherwise, they should repeat the process following the instructions prompted by the virtual tutor.The students will be able to 1) know the names of instruments in order to select the instruments that are on-target and task-relevant; 2) tell the functions and roles in order to further understand the concepts of electronics; 3) analyze the principles that make the circuits work; 4) find the correct methods to connect the instruments; and 5) use a note card to record the readings.In addition, this study concerns the skill level and attitude level of students.As to the skill level, we compute the cumulative number of tasks, cumulative time of task completion, the time consumed by per task and the completion speed.As to the attitude level, we develop a scale to assess the students' satisfaction.

Procedure
We conducted a quasi-experimental design.This study was performed with the approval of the local ethical committee.A total of ninety high school students voluntarily participated in this experiment (mean = 16.4,SD = 2.58).They were evenly assigned to three classes, one for the control group, one for the experiment group A (EG-A), and the other for the experiment group B (EG-B).Each group had 30 students.The independent variable of this study was the treatment of virtual agents.The control group did not use virtual agents.All the members in the group experienced independently.Students of EG-A and EG-B were both provided with virtual agents.EG-A was provided with low intervention, while EG-B was provided with high intervention.In the low intervention, virtual agents prompted information about what a student was doing, whether he/she was at the right position, and how to direct success.In the high intervention, the virtual agents prompted how to precisely achieve the goals with detailed suggestions.The dependent variables were the number of tasks completed, the time consumed when tasks were completed, and users' perceived satisfaction after using virtual agents.
Furthermore, we also minimize the influence of confusing variables.Firstly, we carefully took participants to adequately exhibit in the environment, then explained the media interface operations and virtual objects including experiment instruments, learning materials and virtual agents.The participants first took one week in their daily 45-min training classes before the experiment.Then they were asked to read a guide material to acquaint the experiment settings and task objectives.The students were asked to get the basic details about the 3D virtual laboratory and be informed the specified tasks.And secondly, the participants were selected randomly followed by an appropriate adjustment of personnel structure.This makes the boys and girls have nearly the equal gender proportion, specialty (Engineering vs non-Engineering) and experience (less than one year vs more than one year) across groups.As a result, the students within a group were closely balanced between male and female, the ratio of Engineering to non-Engineering was approximate to 2:1, and students who had prior experience are 7 in control group, 6 in EG-A, and 7 in EG-B.
The tasks in both control group and experiment groups were the same.The students in the control group were asked to experience independently without any help, neither by artificial cue nor by virtual agents.The experiment groups were required to complete the tasks only with the help of virtual agents.The virtual agents were used to provide students with information assistance, either by adapting the virtual tools or by giving explicit guidance and technical support.The process was time-limited and they had to try their best to complete the tasks.To conduct one-one tutoring, we exclude the influence of one character/avatar on other ones.Therefore, six server copies were created in a virtual machine that installed Open Simulator.Then, six participants could get access to the 3D virtual laboratory individually.The data was gathered using logs, so scriptlets are developed and hidden in the scenes.After each of the tasks was completed, they recorded the completion time and the serial number of tasks and then prompted a message to guide students to fill out an inventory to measure the perceptions.Finally, we computed the cumulative number of tasks and cumulative time of task completion via retrieving and processing logs.Besides, we considered the time consumed by per task and the completion speed.The time per task was presented as the cumulative time divided by cumulative number and the completion speed is the reciprocal of the time per task.

Instruments
We designed six tasks, and each contained several subtasks shown in Table 1.To address the research question one, the quantitative indexes of the inventory were the number of operated components and time consumed in total.
We then developed a scale to assess the students' satisfaction.The scale contained 10 items with a five-point Likert rating scheme with "1" being not at all true and "5" being very true, which examined students' subjective perceptions of the process, such as "I often felt disoriented", "I often felt lonely", "I can effectively manage my avatar", "I can easily find science lab and apparatuses" and so on.The estimated Cronbach's alpha value of the questionnaire was 0.724, showing good reliability in internal consistency.

Analysis
As to the question one and question two, we first answer whether virtual agents have effects on task completion followed by the extent to which the effects are.Therefore, we ran independent sample t-test on the control group and EG-A to observe whether the effect is significant in terms of the time in total, time per task, completion speed, and satisfaction with the use and non-use of virtual agents.Then we ran descriptive statistics and analysis of deviation on the control group, EG-A, and EG-B followed by a post hoc test to observe the statistical difference among groups.As to the question three, we ran Pearson correlation analysis between the satisfaction, time in total, completion speed, time per task.We also plotted mean charts to present the target relations.We assume the less time the participants invested represented the lower operational challenge.The higher satisfaction scores the participants got represented the better experiences.The analysis software was SPSS V.18.We used an alpha level .05 to establish significance unless otherwise noted.

To what extent do students afford to effectively complete tasks?
Normal distribution tests on total time, time per task, and speed between the control group and EG-A were conducted using P-P plot and showed they were approximate normal distribution.Therefore, the independent sample t-test was carried out.There was no significant difference in variance between the two populations (F = .485,p = .489> .05).The mean of time populations between the control group and EG-A had significant difference (t = 4.177, p = .00< .05).The population variance of time per task had significant difference (F = 12.651, p = .001< .05).The mean of time per task populations between the control group and experiment group A had significant difference (t = 6.304, p = .00< .05).The population variance of speed had significant difference (F = 7.966, p = .007< .05).The mean of speed populations between the control group and experiment group A had significant difference (t = -6.913,p = .00< .05).These results showed that using virtual agents had significant impacts on the students' task completion.
The descriptive statistics among control group and experiment groups were presented in Table 2.We learned that the mean, standard deviation, mean square error, minimum, and maximum were bigger than the experiment groups in terms of the time in total and time per task.While with respect to speed, they were smaller than those of experiment groups.The results of one-way ANOVA were presented in Table 3, we saw there were significant difference in time in total (p = .00< .05),time per task (p = .00< .05),and speed (p = .00< .05).

4.2
In what growth of perceived satisfaction do students exhibit in it?
Normal distribution test on satisfaction between the control group and experiment group A was conducted using P-P plot.It was showing normal distribution.Therefore, the independent sample t-test was carried out.The population variance of satisfaction had significant difference (F = 4.027, p = .049< .05).The mean of satisfaction between control group and experiment group A had significant difference (t = -4.570,p = .00< .05).The descriptive statistics and one-way ANOVA on satisfaction were respectively showed in Table 2 and Table 3.The experiment groups got higher satisfaction than the control group.And the standard deviation of experiment groups reduced compared to the control group, which indicated that the gap on satisfaction within experiment groups got smaller.Statistically, the satisfaction among control group and experiment groups had significant difference (p = 0.00 < .05).

4.3
What is the relationship between task completion and perceived satisfaction?
The correlation analysis was presented in Table 4.As can be learned that the Pearson correlation coefficient of time investment and satisfaction was -.365, indicating weak negative correlation.The two populations were non-zero correlation (p = .000< .01).This implied that the less time investment would result in higher satisfaction.On the other hand, the satisfaction was a strong positive correlation with completing speed (r = .626).If they completed the tasks faster, they ran into fewer roadblocks in progress.In other words, they might receive more useful information from the virtual agents.It also supplied that satisfaction was strongly negatively correlated with time per task (r = -.685),implying that the less time per task consumed would result in much higher satisfaction.Correlation is significant at the 0.01 level (2-tailed).
Figure 2, Figure 3 and Figure 4 display the answer of the second question further.In Figure 2, a higher intervention resulted in more satisfaction overall.However, this relation was not very obvious.For some students, the reason might be that a higher intervention interrupted their task completion.They didn't need too much information indeed.Therefore, they felt unsatisfactory.For the other students, on the contrary, they benefited from the enriched information and felt less loneliness, thus perceived more satisfaction.According to the Figure 3, it sacrificed completion speed, which confirmed that they were interrupted by the excess prompts.They spent more time in filtering information.In Figure 4, however, the time per task kept roughly constant between the two experiment groups.iJOE -Vol.13, No. 6, 2017

Discussion and implications
The agents provided immediate assistance through life-like characters and played a role of learning companions.This process enhanced emotional experience and selfpleasure.For example, members with virtual agents in both experiment groups scored much higher than those in the control group in terms of satisfaction.The control group got 28.17 points on average, compared to 35.27 of the experiment group A. Three students scored between 0 and 20 in the control group but no one did in the experiment group A. Fifteen students scored between 20 and 30 in the control group compared to 6 students in the experiment group A.
Moreover, they promote task completion.Students could find the destinations quickly with the help of virtual agents.In the task of "find virtual electrical meters", twelve students in the control group failed to reach the destination, while all the students in the experiment groups did it.With respect to the completion time, the EG-A got a faster speed by 0.4 minutes on average.In the three simple tasks of "control", "communicate", "change appearance", there were 22 students in the control group achieved it.The sum of mean time was 7.2 minutes.In comparison, all the members in the experiment group A achieved the goal, and the sum of mean time was 4.0 minutes, faster by 3.2 minutes than the control group.In the medium difficult task of "seek information", 18 students in the control group completed the task and spent 3.6 minutes averagely, compared to all the members in the experiment group A. The mean time was 2.8 minutes, faster by 0.8 minutes.And, in the more difficult task of "navigate", only 7 students in the control group succeeded, compared to all the students did in the experiment group A, who spent less time than the control group by 1.2 minutes.Five students in the control group completed the advanced task of "create".However, all the members in the experiment groups did it.These data indicated that the virtual agents improved the efficiency of task completion.
In conclusion, this paper considers two learning outcomes when users explore the virtual world with the virtual agents.The results show the virtual agents improve individual's skill and promote task completion by prompting related information.Also, the virtual agents improve users' satisfaction by providing immediate assistance when users are trapped.The limitation of the paper is that we don't present correlations between interventional level of the agent and the perceived satisfaction although we present correlations between satisfaction and tasks such as time in total and time per task.To address this, we need to develop a new scale.It is very interesting and we will include it in the new version of the paper.

Fig. 4 .
Fig.4.Mean of time per task across groups.

Table 1 .
Tasks and sub-tasks