Feasibility Study of Developing a University Course Selection Support System Which Uses Informal Peer Suggestions

The authors proposed a support system for university students to create their own course schedules with informal information using semantic web technology. The system provides course information, such as syllabus, students' assessment scores and informal peer reviews, while participants create their own course schedules. The informal peer reviews are students’ impression and comments. A prototype system was developed for course selections of two departments, and its effectiveness was determined. As a result, the number of courses selected increased significantly, and participants' subjective responses were encouraged when they consulted the system. Frequent access and system usability enhanced the number of courses selected using the system. These provide evidence that the semantic web function of a support system plays a significant role in university course selection.


INTRODUCTION
Universities provide their curriculum to students according to each academic discipline.The students have to have core courses, but they can also choose some optional courses which are basic or advanced.Most departmental curricula are independent of each other.
Students seek to learn several disciplines in response to the needs of society, as they perceive them.Therefore, students seek a variety of information about courses, including informal peer reviews, the value of joining, and the usefulness of the content, etc.Also, student's satisfaction with a course is an important factor for the assurance of quality instruction in universities [1], since student's feedback about teachers is used for the improvement of teaching methods [2].Recently, many universities have begun to provide formal information such as a syllabus of courses via web sites and LMS (Learning Management Systems) as these services are an important responsibility for university management.Most syllabus include course names, information about teaching staff, academic credit, course aims, and course content, etc.
Students can easily obtain the official information about courses, and maximize their level of satisfaction.There are some restrictions however, such as the number of optional courses which can be chosen, the limit to the number of courses which can be taken per week, or that some courses are conducted at the same time during the day.As a result, the course selection problem can be de-fined as an optimization task where the effort taken can benefit each student at the beginning of every academic term.Beyond the academic value of a course, students also consider the trade-off they have to make when making selections, such as being afraid of losing course credits through failure or the workload requirement.As courses outside of their department are an unknown area for them, they seek to exchange opinions in advance about their experiences, impressions of courses, the lecture style and the lecturer's characteristics as informal information.Recently some informal information has been gathered using anonymous web sites [3], but they are merely presented formally at official university sites and even at other sites.This suggests that students have to browse a number of web sites which include various sources of informal information, only one of which contains the official syllabus during the course selection task.
For these operations, semantic web technology [4][5] can be a powerful tool for gathering related documents and reproducing referable data.Semantic web technology has been applied to numerous educational tools which are based on specific ontology [6] [7].
In this paper, we propose a support system for university students to create their own course schedules each term with informal peer reviews using semantic web technology, where the effectiveness of the system to promote course selection is examined while participants review their own behavior.For this purpose, we have developed a prototype of the system and conducted an evaluation experiment.

A. Course selection support system
The proposed system for course selection support is illustrated in Figure 1.The procedure for course selection is displayed as a flow chart on the right side of the figure.In the first step, students set their own planning policy for course registration, and they survey course information.During this step, they need precise information and systematic support.Students review their own course schedules and repeatedly revise them using the system.To show the appropriate information on a web browser, a semantic web system has been developed.The system consists of the ontology for university courses and some databases.Here, the ontology defines the logical relationship between required data formats and databases [8].The databases consist of following data: This is provided by university offices, and includes course names, information about teaching staff, academic credit, course content, the requirements for students, and an assessment guideline.Students can browse these on the university's web site.
• Informal course assessments Assessments for each course are from anonymous evaluations by students.These evaluations using 9 point scale are averaged to allow comparisons of usefulness between courses.
• Informal reviews by students Student's comments about courses: Impressions, recommendations and criticisms are noted.
The first type of data can be obtained from the university office, and some universities provide these via their web sites formally using a unified format.As the rest of the databases are distributed across the Internet as informal information [3], a procedure for referring to other web sites is required.Also, since their data formats are not unified, they are transformed into resources for the semantic web using RDF (Resource Description Framework) schema [9].].In this case, RDF simply contains assessment data which should be described as a set of "is-a" rules.The target information is extracted from the RDF files using a program with SPARQL (Simple Protocol And RDF Query Language) [9], and is converted to files with XML tags to facilitate browsing.For this prototype system, both formal and informal information were gathered experimentally, and stored in a prototypical database using a format specified in advance.The procedure for gathering the information is explained in Experiment Section IIIa.In this paper, the semantic web function mainly plays the role of showing both informal information about a course.

B. Web interface of the system
A screen shot of the web interface for a course selection support system with informal information using semantic web technology is displayed in Figure 2. The left vertical sub-window shows both formal and informal course information such as the syllabus and students' comments.The horizontal column on the top-right side of the panel shows class periods (1-8 per day), the mid-right side shows the names of course available on Monday, and the bottom-right side of the panel shows names of course available on Tuesday.Here, this image shows courses for two departments: Control Engineering and Industrial Management.This information is commonly displayed on systems both with and without semantic web technology.When the system uses the semantic web technology, in- formal review information is also displayed in the left side vertical window.The mean value of students' assessments are indicated next to course names, and the color of the background of the course name is illustrated using a color to indicate the level of usefulness, according to students' evaluations using a 9 point scale.This mean value is represented by three colors: red = less than 3.0, yellow = from 3.0 to 6.0, and blue = higher than 6.0.These color designations, which are based on peers evaluations, may attract participant's attention during the selection of courses.When users click the right mouse button of a course name, the interface sends a query message to the semantic web system and displays information about the course.
The system was developed in an integrated development environment (eclipse 3.3), using JAVA.
The semantic web function was developed using a Jena framework.This can work on Windows and Linux platforms.Therefore, the system can be accessed using any type of browser.

A. Experimental design
To determine the effectiveness of the semantic web technology system, the performance of student's course schedule creation process is tested during an experiment. .The experiment was conducted based on quasi-experiment design [13].One-group design was applied to this assessment because it was not easy to create two identical groups of experimental students.Participants in this experiment were asked to compare courses of two departments and then create their own course schedules using the system.The participants were 13 undergraduate level students who were studying in the Control Engineering department.As seniors, they knew most of the content of courses in their department.Because course selection is not an easy task, experienced seniors were assigned to subjects to assess this system function, though they had little information about courses in other department before participation.In this experiment, the support system displayed all courses in the spring term for the Control Engineering, and Industrial Management Departments.If the courses were recommended for sophomores or junior students, these recommendations were clearly displayed as spring term courses (3rd and 5th terms) in Figure 2.
Our hypothesis is that students select some Industrial Management courses in addition to courses in their own department (Control Engineering), since most students would like to extend the range of their education to various disciplines when the system can provide sufficient course information, in particular informal peer assessments, for them to make informed decisions.
It was not easy to gather enough informal information about all of the courses which were displayed on the system in this experiment.Therefore, the review comments for some courses surveyed for this experiment used a reviewing system developed in advance.For the courses in the Industrial Management Department, 8 disparate senior students who were students in that department rated all of the courses and specified which lecturers are desirable.Their scores and review comments were stored in an experimental database.

B. Procedure
All 13 participants were asked to create their own course schedule twice, once using the system and once using a system without a semantic web function.The procedure was as follows: • Instruction of the experiment's aim to participants.
• Questionnaires about learning attitudes (1st) • Creation of a course schedule without a semantic web function system • Break • Creation of course schedule with a semantic web function system • Usability assessment • Questionnaire about learning attitudes (2nd) To evaluate system performance, the following metrics were gathered and analyzed.
• The number of courses selected The number of courses of each department which were selected (steps 3 and 5).
• Questionnaire about learning attitudes The participants were asked to rate their own learning attitudes twice using 6 questionnaires with 4 point scales, before and after the experiment (steps 2 and 7).
• Usability assessment Usability plays a significant role in the success of software applications [11].Several studies suggest various methods of assessment; user-testing is one of these [12].This assessment experiment is based on a user-testing procedure.The questionnaire assessment is also an assessment procedure for usability.The system usability was evaluated using 10 questionnaires with 4 point Likert scales.These were created using Brook's system usability

FEASIBILITY STUDY OF DEVELOPING A UNIVERSITY COURSE SELECTION SUPPORT SYSTEM WHICH USES INFORMAL PEER SUGGESTIONS
score as a reference [10], with half of the 10 questionnaires describing positive situations about the system and the other half describing negative situations.The responses for the negative questionnaires were converted into scores with reversed score values.The sum of those scores was adjusted into a range from 0 to 100 (step 6).
The number of accesses The number of accesses was extracted from access log data of each session (steps 3 and 5).

A. Effect on course selection
The number of courses chosen during each experimental session is summarized in Figure 3.For the first session (without semantic web technology), most courses chosen were related to Control Engineering, the participants' own department.The number of courses for Industrial Management was less than 30% of total courses chosen.
When participants could refer to course information using a semantic web technology, the number of courses chosen for both departments were comparable.This suggests that the number of courses chosen for Control Engineering decreased significantly (t(12)=3.4,p<0.01), while the number for Industrial Management increased significantly (t(12)=3.1,p<0.01).The total number of courses chosen were comparable because there was a restriction for choosing courses.
This suggests that course selection shifts to an other department's courses when the semantic web technology shows detailed course information including informal assessments.

B. Attitude change of participants
The response for learning attitude questionnaires is summarized in Figure 4.At the beginning of the experiment, the means of all responses were distributed in the neutral range, between 2 and 3 on a 4 points scale.At the end of the experiment, the means for responses of 4 out of 6 questionnaires responses were higher than 3.0.The scores for 4 questionnaires were significantly higher than the previous scores.Therefore, this system may encourage participants' involvement, such as is shown in "Interest in the other department" (t(12)=3.4,p<0.01), "Voluntary course selection" (t(12)=3.3,p<0.01).For "Interest in Industrial Management department courses" (t(12)=2.6,p<0.05) and "Easiness of planning course schedules" (t(12)=2.5,p<0.05), the means for the responses after the experiment using semantic web technology were also significantly higher than were the means for the 1st responses.
This suggests that the system support may affect student's attitudes and actions regarding course selection.The participants had some interest in courses of the other departments, however, further study of this will be required.

C. Usability assessment
The system usability was assessed with a Brook's type questionnaire [10].To examine the influence of system usability on course selection, the relationship between these is illustrated in Figure 5.The horizontal axis shows the usability score, and the vertical axis shows the difference in the number of course selections for the Industrial Management department between the two sessions.
According to the distribution of the usability scores, most participants gave scores which were higher than the neutral assessment (62.5).This system dealt with only two departments, so the assessment may be simple.
The scatter-gram shows the relationship between the usability score and the amount of the increase of the number of courses selected.The correlation coefficient is r=0.54 (p<0.05).A regression line overlaid onto Figure 5 suggests that participants choose more courses in the Industrial Management department when they assessed the system without apprehension.Therefore they may choose more of these courses once they have recognized the benefit of the system.There were participants who rated the usability high, but the number of courses they chose decreased, so careful consideration of this is required.

D. Relationship with access frequency
The frequency of referring to course information by participants who rated usability high was calculated.The correlation coefficients between these two components is r=0.59 (p<0.05).This suggests that participants who rated the usability high use this system frequently.
The relationship between the frequency of referring to the course information and the amount of the increase of the number of courses selected is summarized in Figure 6.The horizontal axis shows the number of accesses for the  FEASIBILITY STUDY OF DEVELOPING A UNIVERSITY COURSE SELECTION SUPPORT SYSTEM WHICH USES INFORMAL PEER SUGGESTIONS semantic web system, the vertical axis shows the difference in the number of course selections for the Industrial Management department between the two sessions.A regression line was inserted into the figure.There is a positive correlation (r=0.65,p<0.01).Also, the figure indicates that a few participants took a look at the course information frequently.This frequent access sometimes suggests the difficulties of user operation of the system.The usability results show the number of accesses correlate with the usability score, however.Therefore, participants carefully consult the course information using this system, and they choose courses according to their own interests.
V. CONCLUSION This paper proposed a support system to help university students create their own course schedules with informal information using semantic web technology, including RDF based ontology, for courses in two departments.The system can provide course information, such as syllabus, and informal students' assessment scores, reviews while participants create their own course schedules for an academic term.To determine the effectiveness of the system, a prototype system was developed for course selection support for two departments: Control Engineering and Industrial Management.
In the results, the number of courses in Industrial Management which participants in the Control Engineering department chose increased significantly, and subjective responses about their attitudes and interests were encouraged when they consulted a system which had semantic web functions for obtaining informal peer reviews.The system usability evaluation and the frequent access to the site also promoted an increase in the number of course selections using the system.
These results provide evidence that the semantic web function of a support system using informal peer reviews plays a significant role in the selection of university courses.The enhancement of assessment items and improvement of the system for use by multiple departments will be a subject of our further study.

FEASIBILITY
STUDY OF DEVELOPING A UNIVERSITY COURSE SELECTION SUPPORT SYSTEM WHICH USES INFORMAL PEER SUGGESTIONS • Formal course information (syllabus)

Figure 1 .Figure 2 .
Figure 1.A diagram of the system

Figure 4 .
Figure 4. Mean number of courses chosen.

Figure 5 .
Figure 5. Mean score for learning attitude questionnaires.

Figure 6 .
Figure 6.Relationship between the usability scores and the amount of the increase of the number of courses selected.

Figure 7 .
Figure 7. Relationship between the number of accesses and the amount of the increase of the number of courses selected.