Integrating Regular, On-Line Evaluation by Students into the Curriculum Review Process in an Australian Medical Program

The School of Medicine Resources Management System (SOMRMS) supports the medical program at Griffith University via on-line delivery of various types of learning resources. One of the key functionalities of this system is the capacity for regular collection of on-line evaluations. PBL tutorial groups evaluate their perceptions of the group’s learning against the stated learning objectives and these results are immediately available for review by staff. This paper presents data demonstrating the important role that this evaluation tool plays. It promotes student engagement by linking their evaluations to consequential course and curriculum improvements, highlights staff development needs and assists in the identification of students at risk.


INTRODUCTION
The Griffith MBBS program began in 2005 and was accredited with the Australian Medical Council to serve the needs of outer metropolitan communities and to have a strong community focus. The program is graduate entry, requiring all students to possess at least a university Bachelor's Degree. By removing all other prerequisites, the School aims to ensure equity and diversity in student enrolments in accordance with the Griffith University Equity and Diversity Plan 2007-2010 [1]. The first cohort of 80 students graduated in December 2008 and current student enrolments average 150 per year.
In each of the four years of this graduate entry program, students enroll in one course where teaching, learning and assessment are integrated. During the first two years the medical program involves problem-based learning (PBL) through facilitated, small group tutorials supported by lectures, practical classes, workshops and other formats including on-line activities. Clinical skills are developed through a program of tutorials, workshops and hospital-based practice. Students also engage in a series of placements in community organisations and general practice clinics. The final two years consist of rotations through various hospital departments. Students access online resources to support teaching in the hospitals.
The medical curriculum was designed to integrate teaching, learning and assessment with the support of a totally new, electronic delivery, communication and evaluation system. Griffith University devoted considerable effort to this to strengthen its initiatives towards human-centered e-learning and increased student engagement in curriculum design and course management. The School of Medicine Resources Management System (SOMRMS) was developed by Robert Loudon and other members of the Flexible Learning and Access Services (FLAS) team in consultation with the academics responsible for the development and delivery of the MBBS program [2].

A. Problem-Based Learning (PBL)
During the past 40 years, problem-based learning has had a marked impact on higher education, especially in medical schools [3]. This method of interactive learning focuses on patient problems as a context for students to acquire problem-solving skills and knowledge about the basic and clinical sciences [4,5]. The PBL process involves students working in small groups with a tutor who has the role of facilitator rather than provider of content. The students address a clinical problem through discussing possible hypotheses. They develop strategies to test the hypotheses; collect and analyze new information; refine their hypotheses and in so doing determine the gaps in their knowledge and understanding; and develop learning goals to direct their independent learning outside the PBL tutorials [6].
From the inception of the new medical curriculum at Griffith University, it was recognized that the design of any on-line delivery environment should incorporate and support the theoretical foundations that underpin the constructivist approach to learning and cognition underpins the PBL process [7,8]. During the delivery process, both tutors and learners need the support of comprehensive learning resources, designed explicitly according to a constructivist approach. Students and tutors need opportunities to reflect on the learning process and avenues for feedback, both directly and via the on-line delivery system. The on-line delivery system should emphasize teambased collaborative reflection on the learning process, i.e. responses recorded per team not per individual [7]. These requirements have been built into the SOMRMS on-line management system.

B. Curriculum Review and Adjustment
Many universities world wide now carry out course and teacher evaluation programs for staff performance evaluation and for monitoring the quality of their courses. The connection of student evaluation to staff development and curriculum review was also noted by Everingham and Feletti [9] who recommended that the focus of evaluation should be on course content and process while also addressing staff development issues that need to be addressed. Fish and Coles [10]  There are some distinctive characteristics of medical education that further impact on course evaluation. Medical knowledge is continually changing and a medical curriculum needs to adapt to new topics and emerging fields of study [11]. Frequent review and updating of PBL cases is of particular importance as the educational rationale for the PBL process is predicated on students encountering novel situations and adapting their reasoning processes as they grapple with the complexities of a case. PBL cases that are continually recycled without modification will become predictable and will not achieve this goal. A recent review of the University of Queensland medical program emphasized the importance of updating and rewriting PBL cases to ensure that they fulfill the aim of developing clinical reasoning skills [12]. Student engagement in evaluation is therefore a crucial component of the ongoing development of the medical curriculum at Griffith University School of Medicine.
The SOMRMS plays a role not just in the electronic delivery of the Griffith constructivist curriculum but as a communication and evaluation system integrating learning, evaluation and assessment, thus facilitating ongoing curriculum review and adjustment. This function is highlighted in the model shown in Fig. 1.

C. School of Medicine Resource Management System
The School of Medicine Resource Management System (SOMRMS) is an on-line database application that supports the Griffith MBBS program by integrating various elements of the curriculum. The prime role of the system is to deliver to the students' various learning resources. These include PBL case study notes, resources and learning objectives; tutor support material; lecture, tutorial and workshop resources; on-line lectures for years 1 and 2 and other resources for the clinical topics for years 3 and 4 of the program. These resources are delivered seamlessly within the university's enterprise Blackboard Learning Management System, which provides additional support to the program via tools such as discussion forums, file exchange areas and on-line tests and surveys.
In addition, the SOMRMS allows for the management of the above mentioned resources including the timed and staged release of PBL case study notes, the automatic generation of on-line evaluation and feedback reports relating to the defined learning objectives as well as collection of usage statistics. It also includes various administrative functions including management of tutorial groups; staff, contact and student data; on-line timetables for each student; management of student placements and rotations through clinical placements; on-line case logbook and a student case presentation system. Development of SOMRMS commenced six months before the first intake of students. It was undertaken by Robert Loudon and other members of the FLAS team in consultation with the academics responsible for the development and delivery of the program. The application consists of an Oracle database and PHP web pages that are located within the Blackboard frameset. The administration backend application is built using Adobe Flex. The system has been developed with careful regard to the robustness of coding, ability to extend the system even to program (modified from Tedman, Alexander and Loudon [7] functionalities not yet identified, integration with the wider LMS and security issues.
To date, the system has proven to be very reliable and has provided significant benefits including reduction of printing costs; fast updating of content; solid organization of large amounts of data; the ability to create meaningful relationships between various resources and parts of the curriculum; on-line entry, storage and review of student identified learning issues; on-line collection of student evaluations and feedback.

A. Objectives of the Evaluation Tool
While evaluation is now a mandated component of all university courses, the Griffith University School of Medicine approach was to implement a comprehensive system of on-going evaluation that would guide continuous curriculum development. Other medical programs have improved evaluation beyond end of course surveys. The Stanford University medical program, through the Division of Evaluation, conducts on-line surveys for each course but adds focus groups midway through courses to collect data that allow for mid-course corrections and improvements [13]. Given that the length of time between a lecture and an evaluation or the placement of an evaluation before or after a course assessment are both believed to impact validity and reliability [14], other medical schools have introduced more frequent evaluation opportunities. Medical Schools at the Australian National University [15] and University of Sydney [16] recognize the importance of regular feedback from the students and use weekly evaluations as part of their monitoring process. However, there appear to be no published reports of comprehensive evaluation systems that provide a complete package of data collection, analysis, curriculum monitoring, change and feedback to students in an electronic format on a sustainable, regular basis in Australian medical schools.
Prior to the commencement of the first intake of the Griffith MBBS program, there was a strong desire to put in place measures to evaluate the program to confirm that the initial curriculum design was sound, and to provide a good benchmark for ongoing refinement of the curriculum. During the initial development phase of SOMRMS, it was realized that the proposed on-line delivery system would allow for the efficient collection of evaluation data from students. A third goal of the comprehensive evaluation design was to ensure student engagement and satisfaction with the medical program. It has long been recognized that successful incorporation of e-learning into problem based learning requires good communication between students and staff [17]. Promoting students' confidence, motivation and subsequent participation in course learning opportunities [18] plus the ease of access to learning resources [19] are important in improving student learning and satisfaction. It is also important to engage students in quality improvement initiatives [20]. Griffith medical school staff recognised the importance of creating a teaching and learning environment that combined modern pedagogy with initiatives that promote student and staff satisfaction and involvement. It is this engagement with students that has enabled the evaluation system to collect on-going reliable data in a consistent manner, in a timely fashion and with minimal effort on the part of students and staff.
Griffith University is committed to the integration of assessment, learning and evaluation. Assessment in the MBBS program is criterion-referenced, that is, students' performance is compared to pre-determined standards rather than to the performance of other students. Assessment formats are selected to align with the required knowledge, skills and attributes and the contexts in which they are being learned. Challenges posed to students and judgments made of their performance are designed to encompass the range of specified learning outcomes and to align with formative learning opportunities. Student learning issues identified for each PBL case and the School Learning Objectives (LOs) released after each case are the best guides to what is considered important for purposes of assessment in Years 1 and 2. Students are expected to interpret the LOs broadly so that they can apply their understanding of mechanisms to a variety of contexts. Hence, since the LOs are of critical importance in the assessment process, the School has incorporated a system for evaluating the students' perceptions of the LOs on a weekly basis and developed a sustainable mechanism for display and storage of the evaluation data.

B. Description of the Evaluation Tool
As described earlier, the SOMRMS database contains all the PBL case studies and their learning objectives, and the release dates for the cases, as well as data about the PBL tutorial groups and the students in them. It has therefore been possible to automatically create a customised evaluation form that can be made available to each PBL tutorial group at the end of the PBL case. Responses to the evaluations are collected via an on-line form and stored in the database. The weekly student evaluations comprise of two parts. First, each of the learning objectives for the week is listed, and tutorial groups are asked to rate their opinion on a five-point scale of how well they feel they have achieved the learning objective. For each learning objective, the various delivery modes of relevant learning resources are listed, to remind the stu- Secondly, there are number of general questions soliciting feedback from students. Responses are entered into free text entry fields and are also saved to the database. Although it is possible to have different questions for each PBL case, in practice the same questions are repeated each week. In the early days of the MBBS program, there were 2 standard questions. This has increased over time to the current 5 questions (see Table I).
The evaluations are collected per tutorial group rather than per student. Although the latter would have been technically equally feasible, it was felt that the there was a danger of survey fatigue which may have led to unreliable responses. Currently, the evaluations are done by consensus among the group while they are gathered together in their PBL tutorial room and this provides an opportunity for further reflection and discussion, enhancing the learning process.
Access to the evaluation form automatically becomes available as a link on the PBL Case Study web page only after the release date and time of the last tutorial of the case. Once the evaluation form has been submitted, the tutorial group cannot access or edit that evaluation again. Once the evaluation form has been submitted, a new link becomes available which allows the members of the tutorial group to view a list of the learning objectives for the case. As assessment content in the program is based on the learning objectives, the students have a strong incentive to complete the evaluation so they have access to the lists of learning objectives.
The benefits of an on-line system extend beyond the management and collection of on-line survey data. As the results from the evaluations are immediately stored in the database, it is therefore possible to make these data immediately available for review and simple statistical analysis. The SOMRMS administration site includes a special section that allows staff to view collated data for any of the PBL cases, including the learning objective ratings, the feedback questions and the student-identified learning issues. As will be discussed later, these data, and the timeliness of its availability is of great value to both the immediate and longer term curriculum review process.

C. Survey of Student Use of the Evaluation Tool
A survey of the current (2009) Year 1 and Year 2 cohorts was carried out in July. This survey contained questions specifically aimed at providing information about how students perceive the weekly evaluation process. (A portion of the survey instrument is shown in Table II   Further, 81.7% and 73.9% of year 1 and 2 students, respectively, indicated that more than half the group members contributed to the scoring of the LOs. The LO evaluation is either mostly or always taken seriously by 65.8% of year 1 students and 60.9% of Year 2 students. These data suggest that the LO evaluation scores are representative, and supports the decision to collect these on a group basis. The high value of the comments part of the evaluation is confirmed by the student responses whereby 97.5% of year 1 and 93.5% of year 2 students agreed or strongly agreed that the comments section is an important avenue for feedback. Individual students regularly or always contribute to the discussion about text comments (62.2% of year 1 and 65.2% of year 2 students). These data suggest that the feedback is representative of the group. The process allows individuals to contribute but group discussion and sharing of knowledge probably resolves some issues and may filter out extreme views.

B. Analysis of Comments Data collected using SOMRMS Evaluation Tool
The SOMRMS tool has provided data that demonstrate consistent engagement (about 30 comments per case from the 17 PBL groups) with the SOMRMS evaluation system, with some increase in year 2 (see Table III). Between 77 and 79% of PBL groups contribute at least one comment per case. Data also indicate the consistent use of the SOMRMS evaluation system by students to provide unsolicited positive comments about the program in years 1 and 2 (23% to 26% of all comments). The comments also provided consistent feedback regarding staff development implications, with a mean of 4 per case in year 1 and 5.5 in year 2. The increase in year 2 coincides with implementation of new PBL processes involving online discussion boards and changes in the report back process.

C. Curriculum Improvements resulting from SOMRMS Evaluation Tool
Analysis of evaluation data from the Year 1 and 2 PBL evaluations has resulted in changes to the curriculum at many levels. As can be seen in Fig. 3, the student evaluation data (available immediately on-line after collection) are disseminated to various key actors for analysis and appraisal, and these considerations often result in curriculum review and/or modification.
Short term responses: The on-line evaluation allows for a prompt response to student concerns and some immediate changes have been made to the current curriculum for the cohort of students providing the feedback. The most obvious indication of a problem is the allocation of a low score to a Learning Objective (LO), and students' comments generally shed light on the reasons for this. Comments may indicate confusion over the meaning of the LO, a lack of relevant, high quality references relating to the LO, dissatisfaction with a lecture purportedly covering the LO, or the need for additional lectures to cover concepts relating to the LO. If the issue has been raised by several groups, indicating a widespread concern within the cohort, then the feedback is acted on quickly by the School. Many of the student evaluation comments do not refer directly to LOs but raise other issues that have also been addressed swiftly. Responses have varied from making minor improvements to PBL cases, such as correcting Figure 3. Dissemination pathways for student evaluation data and corresponding curriculum review and modification processes and actors typographical errors in case materials and renewing inactive electronic links, to scheduling extra revision practical sessions and overview lectures, or providing practice exam questions. Individual lecturers, clinicians, General Practitioners, and anatomy tutors are often acknowledged as outstanding in student feedback comments. These comments are forwarded to the relevant individuals, to ensure that they are aware of the student's appreciation of their efforts, and to encourage their continued involvement in the medical curriculum. Sometimes an immediate response is required to a more significant area of concern, for example, student reports of rare, unprofessional behavior exhibited by an actor in clinical skills sessions were also addressed without delay. As communicating the outcomes of evaluation to the students is essential to maintain their support for the process, the "feedback loop" was closed by informing students of the actions implemented by the school. Initially, responses to student evaluations were fed back to students via announcements in the learning management system. Feedback action was then recorded in a document as a more permanent record for each cohort. Currently, some feedback also tends to occur more interactively via the student cohort discussion board.
Long term changes: As the on-line evaluations are completed at the end of each PBL case, their most wideranging application has been in the planning of the curriculum for forthcoming cohorts. In this context, more significant changes have been made to LOs in response to the evaluation scores they attract. Many changes have also been incorporated into the PBL cases to enhance their ability to draw out the required LOs. Tutor notes have been continuously updated to assist tutors in facilitating learning within their groups. Many details within the cases have been updated. Because it is a post-graduate course with many students possessing specialist knowledge in diverse health related fields such as pharmacology, radiology, immunology and microbiology, these students have provided constructive and pertinent feedback that has been incorporated into the updating of the cases.
Student evaluation has also been used to determine the most suitable order of cases to ensure a sensible progression of knowledge acquisition. With subjects such as anatomy, pathology, immunology and pharmacology entirely integrated in a PBL curriculum, it is important to evaluate larger issues such as the overarching organization of the curriculum, whether the order of topics is logical, and whether there is redundancy or gaps in the curriculum [11]. Student feedback has also identified subtle and gradual indications that students may be disengaging from the PBL process. Student-identified learning issues were studied and found to be "cut and pasted" questions taken from the PBL case screens. It was felt that using these questions as a template for the generation of learning issues was not conducive to the PBL process and the questions were removed from the Year 2 PBL cases. Decreased engagement with the PBL process was also partly due to the availability of previous PBL cases and 'super notes'. As a result, changes to the PBL process have been implemented in the areas of posting learning issues and reporting back. Additional tutor training sessions have been introduced to reinforce the PBL process and encourage the development of student clinical reasoning skills.
Collation of student feedback relating to lecturers has resulted in the valuable review of lecture content and delivery. This type of evaluation has a significant advantage in a course such as a medical degree. Many of these lecturers may only present one lecture within the complexity of the course and would be unlikely to feature in an "endof-course" evaluation. However, weekly evaluation of lectures has allowed students to give timely feedback on lecturers which has been used to direct the planning of the following year. Complaints relating to lectures have included poor delivery and lack of interaction with students, poor correlation of lectures to LOs, repetition of material covered by another lecture, or too much irrelevant detail. Changes have consequently been made to the content and scheduling of lectures. This feedback has also been passed on to lecturers in a constructive way.
These ongoing adjustments to PBL cases and other aspects of the curriculum appear to have produced a significant improvement in scores given to learning objectives as collected by the student evaluations. Table IV illustrates the changes in the weekly student evaluations of the LOs in the SOMRMS evaluation system for the first 10 week block ("Human Homeostasis") of teaching for Year 1 students.  The data show that there has been an increase in mean rating for individual cases each year from 2005 to 2009 and an increase in mean rating for the whole block each year except one from 2005 to 2009. This trend may be due to improvements instigated by the school in response to feedback from students in SOMRMS and possibly due to improvements in PBL tutor training.

D. Staff Development
Data from the SOMRMS evaluation tool have allowed the Academic Manager for Years 1 and 2 to monitor staff development needs and instigate appropriate development activities (see Fig. 3). Some of these activities involve discussions in tutor briefings and meetings with individual tutors and lecturers while others entail half day training sessions. Typically, important issues arising from the SOMRMS tool data that require attention relate to (i) the "dominant tutor", where the tutor takes over the discussions , (ii) the tutor who does not provide the right balance of guidance for the group such that the group has difficulties understanding the appropriate depth of knowledge required and (iii) the tutor who does not fully accept the guidance from the Academic Manager regarding the PBL process such that the consistency across all 17 PBL tutors in a particular year cohort is at risk. As Hmelo-Silver and Barrows [21] acknowledged, staff development in the form of workshops is important in helping PBL tutors (facilitators) learn and practice appropriate strategies for being good facilitators.

E. Students at Risk
The on-line evaluation system includes a report page which lists the score data for each of the LOs of a PBL Case. The mean score for each LO is also calculated. Data can be viewed in numerical and graphical format. In addition, it is possible to highlight the scores from a particular tutorial group, thus allowing a comparison with the rest of the groups (see Fig. 4). It should be noted that the composition of PBL groups is not via self-selection by the students but is carefully determined by staff with regard to balancing the types of people in each group (e.g. gender, academic performance, previous studies). If a group is consistently scoring LOs lower than the other groups, further investigation is warranted to see if there is any explanation. One possibility is that the tutor responsible for the group may be having difficulties facilitating the PBL process properly and the group may be suffering as a result. This situation has occurred on a number of occasions since the program began, and in these cases, targeted remedial action has been taken to help tutors improve. Continued low ratings in conjunction with other assessments of tutor performance have led to some tutors ceasing to be involved in the PBL process.
Another possibility is that the group itself has a problem, for example unresolved personality conflicts between students. Once again, this indicates the need for further investigation and remedial action.
In this way the regular collection and analysis of student evaluations can be seen to provide an "early warning" system that can help avoid a group of students from being disadvantaged. Of course, since the evaluations are collected on a group basis not an individual basis, it is not possible to pinpoint particular students at risk of poor performance.

IV. CONCLUSION
The Griffith MBBS program has been regularly evaluated using the university's standard course evaluation survey instrument. Such standardized evaluations are cost-effective and necessary, but as noted by Everingham and Feletti [9], it is also important to gather and interpret objective data by knowing and considering context: " On the other hand curriculum managers and teachers are better placed to 'read' the implications of data and particularly of feedback from open-ended questions. Also significant is the contextually informed teacher who has the capacity to 'hear' the significance of anecdotal feed-back and share this with the teaching team in a timely, mutually supportive way." The regular collection of student evaluation data has enabled the School to achieve this benefit, because feedback is immediately shared with and discussed by the PBL tutors via weekly tutor briefing sessions. Data from university standard course evaluation surveys are, of course, also considered carefully, but tend to relate to 'big picture' issues and because they are collected at the end of each year, do not provide opportunity for immediate response to urgent issues. The medicine course is also delivered by multiple lecturers, clinicians and presenters, and it is unlikely that a standard end of course evaluation would incorporate feedback specific to them.
Web-based courses need to be designed to incorporate mechanisms to enable response to change, whether administrative, technical or pedagogical and failure to plan for change can have severe consequences [22]. The evaluation system that has been built into the SOMRMS in conjunction with the organizational structure (see Fig.  3) that has been put in place to manage the MBBS program has enabled staff to manage change. The automatic and immediate collation of results, accessed via the website with numeric and graphical representations, is a very important feature in terms of sustainability. Staff can access the website and view the results as soon as the students have submitted their evaluations. It is the management of the evaluation data through the Learning Resources Officer, under guidance from the Academic Manager for Years 1 and 2 that is the key to completing the evaluation cycle. Part of this process is through discussions in weekly briefings with PBL tutors, enabling timely monitoring of learning activities both for the whole cohort and individual tutorial groups as well as reflection and updating of content and learning objectives.
As has been reported elsewhere, communicating the outcomes from the evaluation with the students is vital to maintain the student engagement with the system and therefore ensure the reliability of the data and the high response rate. Sarah Watson from the Centre for Research into Quality, University of Central England, acknowledged the importance of student feedback for informing improvements in higher education institutions and emphasized the importance of using the data to identify issues, delegating responsibility for action and of informing students of outcomes from the evaluation process [23]. Surveys of higher education institutions in the UK by the Quality Assurance Agency reveal that student evaluation, whether of courses, teaching quality or the overall student experience, is extremely important and has a significant role to play in the quality assurance process [24]. This emphasis on student experience and pastoral care has helped build an atmosphere of team work, confidence, care, respect and understanding within the student community as well as collegiality plus student ownership and satisfaction with the medical program.
Today, professional accreditation bodies are very alert to the ability or otherwise of tertiary programs to 'complete the loop' whenever student feedback is collected. For the Griffith MBBS program, this concern was evidenced in a recent report by the Medical School Accreditation Committee to the Australian Medical Council (AMC). The report addressed current and proposed mechanisms for sharing evaluation results and conse- quences with key stakeholders, and mechanisms to enhance commitment of key stakeholders to the medical course. The regular collection of student evaluation data and its integration in the curriculum review process as discussed in this paper has proven to be of great value in meeting the standards required by such accreditation reviews. The regular, planned collection of student evaluations has given the Griffith School of Medicine a sound basis on which to review and modify its curriculum on an ongoing, incremental basis. Early results suggest that this has had a significant and positive effect on enhancing the curriculum. Student engagement and staff development has also improved. The integration of the student evaluation within the program's on-line system is proving efficient and cost-effective. It provides timely data and enables quick responses in line with the expectations of accreditation bodies.