Using Peer Assessment for Promoting the Learning Process in a Doctoral Blended Learning Program

This paper presents an innovative longitudinal study developed during three years, using a design-based approach consisting of four phases. Taking into account the analysis of students’ behavior and opinions, teachers redesigned the peer assessment (PA) strategies used in the context of a Distance Education blended learning module, of a Doctoral Program, in order to improve the effectiveness of students’ learning. The designed cycle allowed, for instance, the identification of problems, such as the lack of students’ familiarity with PA, the design/adaptation of assessment frameworks and therefore practical solutions were implemented and evaluated. The results show that the applied solutions were valued by the PhD students. Moreover the PA tasks fostered constructive criticism, collaboration and active involvement of students in their own and their peers’ learning process. Although the developed PA strategies were efficient, in the described context, their applicability should be analyzed in similar environments.


INTRODUCTION
Recently, in Higher Education, "assessing of learning" is being replaced by "assessing for learning" [1], [2], [3]. Assessment for learning focuses on learning tasks and includes learning in assessment [3], [4]. According to these authors, in this perspective, three main elements should be considered in the design of the assessment process: i) assessment tasks which focus on learning; ii) participation of students in the assessment process; and iii) providing feedback to increase learning. Moreover, assessment for learning focuses on the usage of assessment strategies which can increase students' learning as it intends to make students plan their own learning, define their weaknesses and strengths and improve their transferable skills [5]. Thus, the primary beneficiary of assessment should be the student. To achieve this, students and tutors must engage in a fruitful process of dialogue and feedback [2].
According to Ref. [6], summative assessment focuses only on the cognitive aspect of learning, generally includes only one performance grade and is designed and applied by the teacher. On the other hand, formative assessment is an integral part of the learning process and should take place not only at the end of a programme but also throughout the entire programme [7]. Formative assessment can focus on cognitive, social and affective aspects. It generally includes a multi-method approach and creates a complete profile rather than simply a final grade [4].
In the context of formative assessment, or assessment for learning, peer assessment (PA) plays an important role [1], [8]. Ref. [6] defines PA as an educational mechanism where students judge qualitatively and quantitatively works and performances of one of their peers and where students are encouraged to engage in reflection, discussion and cooperation. Ref. [9] defines PA as a mechanism for learners aimed at determining and examining the level, value or quality of a product or of the performances of other learners at the same level. The author states that PA activities can be applied in different programme domains and subjects. Various products and outputs, including a writing essay, a portfolio, an oral presentation, a test performance and other skills, can be assessed by peers.
Several authors [9], [10], [11] argue that students can benefit from PA, since it implies the active involvement of learners in shaping their own learning processes, while it promotes an authentic assessment and increases the autonomy and collaboration among students. However, recent literature, investigating students' perceptions about online PA and the nature of students' resistance to PA [12], [13], [14] indicates that students sometimes consider PA as unfair and often believe that peers are unqualified to review and assess their work. On the other hand, students' perceptions about the fairness of PA increases significantly after an' experience in doing PA; and, students' fairness conception is most significantly associated with their perceptions about the extent to which peers' feedback is useful and positive.
Although PA has been extensively used to support students' learning in f2f classrooms [15], little is known about its effectiveness in online contexts [16], in particular in collaborative learning contexts using Web 2.0 technologies. Furthermore, ref. [17] mentions that there is a gap in the literature in what regards "good assessment practices" when students are asked to create and publish content, or participate in networking activities, using social web technologies. This research fills in the above mentioned gaps describing a case that will be presented chronologically, as showing how the explored PA strategies have been developed, considering the learners' voices to be crucial to PAPER USING PEER ASSESSMENT FOR PROMOTING THE LEARNING PROCESS IN A DOCTORAL BLENDED LEARNING PROGRAM evaluate the development of online learning, and assuming that PA is a powerful strategy to improve students' learning. An innovative longitudinal study, in a doctoral module in a blended learning context, is presented in order to understand the process along three years of experience using PA, by refining methodologies so that students can most benefit from it. As far as the methodological approach is concerned, the study discusses how a design-based research approach is being used to plan and implement an online course based on assessment for learning principles [7] to create a more interactive and engaging online learning experience for higher education students.
After this brief introduction, the methodological options are described and discussed. Then, the context of the study is presented as well as the students' profile, in what concerns their age, job and academic qualifications. The section "case description" provides a detailed outline of the experience in chronological terms, as the main results and main decisions will emerge. In the last section, final considerations and current challenges are put forward in order to take full advantage of online assessment for learning.

II. METHODOLOGY
In this section the methodological approach used in the study is described. Given the lack of longitudinal studies regarding PA linked to "good assessment practices" in blended learning contexts, this study has a qualitative, exploratory and descriptive nature and the methodology is based in a design-based research approach, (Fig. 1). With the aim of exploring how to design and develop authentic online PA within the Higher Education sector, taking into account the previous know-how and results, assessment strategies were re-designed, so that students can most benefit from it. The design-based research approach is being used more and more in education and the key elements of this approach include: addressing complex problems in collaboration with practitioners, integrating design principles with new technologies to develop practical solutions to the problem and conducting effective evaluations to improve the proposed solution and identify new design principles. According to ref. [18], designed-based methodologies comprise four phases: phase 1 -Analyze the problem; phase 2 -Design and develop potential solutions; phase 3 -Implement and evaluate; and phase 4 -Reflect and report (see Fig. 1).
The phase 1 addresses three key areas: the problem, the literature review and the practitioners' experiences. During this phase, the researcher clearly articulates the problem and investigates what work has already been done in the same or related fields. By the end of Phase 1, the researcher should be able to create preliminary research questions to guide the study [19].
The phase 2 of the design-based research approach focuses on designing and developing solutions to the problem. This phase corresponds to the design of the module, where the tasks and the goals are thought-out and conceived.
In phase 3, the module designed and developed in phase 2, as a potential solution to the problem, is implemented and evaluated to determine the effectiveness of the framework. In this phase, data are gathered and analyzed.
The phase 4 is where the researcher reflects on the entire project and disseminates information to the broader educational community. The result of this work will be the final learning design principles comprising guidelines for the design of effective e-learning environments in higher education. This last stage might lead to a first stage of another cycle, as the reflection may provide other issues to be analyzed.
In the context of the present research, the authors went through three cycles of design-based research, where each cycle corresponds to an academic year, as described in the following sections. Data were collected using an online questionnaire, applied in each academic year, in three editions of the DE module. The questionnaire was anonymously answered in the final f2f session of the modules, and the majority of the students enrolled in the modules submitted their answers -22 (out of 24) from the 2008/09 edition, 18 (out of 18) from the 2010/11 edition, and 13 (out of 14) from the 2011/12 edition. A total of 53 answers were obtained and analyzed using descriptive statistics. The questionnaire designed to evaluate the above-mentioned modules included five sections. The first section was used to characterize the students' profile in terms of age, job and academic qualifications. The second section aimed at collecting the students' opinions on the module in general, the teachers and the teaching and learning strategies. The third section intended to gather the students' opinions on the tasks and the fourth on the eassessment process.

III. THE CONTEXT OF THE STUDY
This section provides a background of the doctoral programme organization, which is the basis of the context of the study.
The post-graduation module under analysis here is the "Distance Education" (DE) module, which is part of the curricular year of the doctoral programme on Multimedia in Education, offered at the University of Aveiro (Portugal). The PhD programme is organized in a four weeks blended learning context, with two f2f sessions (one at the beginning and the other at the end of the module for presentation and discussion of group works) and the rest is done at a distance in group works, with

Analyse Conceive
Evaluate Implement Design-based approach PAPER USING PEER ASSESSMENT FOR PROMOTING THE LEARNING PROCESS IN A DOCTORAL BLENDED LEARNING PROGRAM extensive online elements, including the teachers' feedback.
The research skills that students are supposed to develop in the doctoral programme, and that underlie its creation, are those required for independent research, such as: seeking and organizing information, data gathering and data analysis, communication, collaborative work, and assessment skills (self and peer assessment). Bearing this in mind, in the DE module, the main activity that students are proposed to develop is to collaboratively produce a literature review paper (in groups) about a topic of their choice related to Distance Education.
The expected learning outcomes of the DE module are: i) to retrieve, select and analyze relevant information (papers, books, dissertations, reports…) about the selected topic; ii) to share, discuss, negotiate meanings and points of view expressed in the selected information; iii) to contribute towards the creation of a friendly and participatory atmosphere; iv) to organize and synthesize information regarding the production of an academic publication; v) to use ICT properly and critically in the research process; vi) to communicate, orally and in writing, and contribute to the development of that skill among peers; vii) to assess the progress of the work that was produced collaboratively, as well as individual contributions (self and PA) and provide constructive suggestions, based on the literature; and, finally, viii) to reflect upon the competencies that were developed by each one and by the colleagues [7]. Project work, problem solving, collaborative learning and assessment for learning are valued as strategies that promote effective online learning.
Concerning the elements involved in the three editions of the Distance Education post-graduation module (2008/09, 2010/11 and 2011/12), the authors of this paper were directly involved having a double role (observers/researchers and participants/teachers). In the 2009/10 year, the teachers involved in the module were not the same and, even though the methodology of the module was similar, the evaluation questionnaire was not applied.
The students' profile, in what concerns their age, job and academic qualifications, in the different editions, was similar and can be summarized, as follows: • the participants in the three editions were more than 26 and less than 57 years old; • 19 students enrolled in the first edition, 12 in the second and 5 in the third edition of the DE module were teachers in different education levels. The others had other jobs as freelancers, researchers, and so on; • the majority of these students (16 in the first edition, 18 in the second edition, and 10 in the latest edition) had already finished a MsD (pre-Bologna Masters Degree in Sciences or Languages) and thus had some research experience. The others had a graduation degree in several different areas.

IV. RESULTS
In this section, a detailed outline of the experience will be presented, while describing how the explored PA strategies have been developed considering the learners' voices, crucial to evaluate the development of online learning and assuming that PA is a powerful strategy to improve students' learning.
Following the results of the first edition's evaluation [7] some changes in the design of the DE module were introduced in the second edition [20], and the same occurred in the third edition, namely concerning the eassessment activities, as well as the communication technologies used (see table 1).

A. First Edition (2008/09)
From a previous experience, in a similar context (blended learning module), where the students were asked to evaluate educational software and blogs were extensively used, the authors found a lack of collaboration among groups of students and realized that the students' assessment framework should be negotiated and transparent [21].
In the first edition of the Distance Education module, a wiki (http://ead0809.wetpaint.com) was used "for the whole class in order to increase the collaboration among groups" [7, p.220] and several assessment strategies were tested, including assessment for learning such as self and PA. In the following paragraph these strategies and the tools used to implement them are described.
To contribute towards the reflection about the ongoing work and the learning outcomes, students were asked to do a closed individual reflection (self assessment) during the 1st week (table 1). GoogleDocs was used for that purpose, in order to facilitate individual diagnostic of the students' previous ideas about the module's topic.
Another assessment strategy, implemented in this edition, to improve the collaboration between groups, was the involvement of students in the assessment of the work done by at least two other groups, during the 3rd week (see table 1). For each group, an assessment page was created for this purpose where the peers and the teachers wrote their comments following the page structure. This way, students would gain a deeper knowledge of each other's work and could provide constructive criticism and suggestions to their mates' work. Thus, the assessment instrument used by both students and teachers to assess the ongoing work (literature review paper) was available at the module website (http://ead0809.wetpaint.com/). It includes criteria and indicators to assess literature review papers, such as: i) clear definition of the research questions or objectives; ii) writing adequacy (for instance, use of academic writing guidelines); iii) relevance of the structure (including an introduction, the methodology used to retrieve and analyze the information, its organization, and a conclusion); iv) originality of contribution (addedvalue, new information…); and v) adequacy of the information seeking and evaluation strategies (that should indicate the databases, the keyword, criteria for the inclusion and exclusion of the gathered information…).
In order to examine the effectiveness of the module, specifically in what concerns PA strategies, students' views were collected. The results (see Fig. 2) show that the assessment activities were considered difficult by some students, and the majority of the students didn't value the intra and inter-group self and PA activities [7]. This indicates that the evaluation culture is not common, even among teachers (as reported above, the majority of the students enrolled in the doctoral module were teachers). Besides that, the literature shows that many students ignore extensive written formative feedback and pay attention only to their marks [22], and thus that assessment is still more a way to get marks than a learning strategy.
The above reported results could also be related to the fact that students: i) felt uneasy and confused, because they are not used to assess other colleagues; ii) may have some prejudices about f2f evaluation; and iii) seemed to be reticent to expose their own reflections to the world, openly, as this task was accomplished using a Wiki-based environment. Although this last option could strengthen the lack of confidence feelings, as well as the fear of exposing errors [9], it could also help students to fulfill the task by learning from the feedback posted by their peers.
From the results of this edition, it was decided to encourage students to discuss their reflections openly, so that they can share and discuss possible common problems, and most of all, to encourage students to face assessment as a learning strategy and not only as a way to mark the student's real individual contributions within the workgroups.
The assessment tasks included a formative component to which special emphasis was ascribed. The final marks and the corresponding weights were negotiated within the class in the first f2f session. However, from the teachers' reflection about the assessment tasks the module activities did not preview the negotiation of the assessment framework, in what concerns the production of the literature review paper, which was included in the second edition.

B. Second Edition (2010/11)
In short, the problems analyzed above encompass the students' perceptions about assessment (usual practices value summative assessment) and their lack of familiarity with assessment for learning strategies (self and peer assessment).
Considering the first of the above-mentioned problems (students of the first edition didn't value the assessment strategies) and to promote assessment for learning, the main changes in the second edition were: i) individual reflections, which occurred during the 1st week, were opened to the whole class (a social networking was used for that purpose); ii) the assessment framework (e.g. the weight of the different dimensions, criteria and indicators, students' involvement…) was negotiated within the class in the first f2f session (see table 1), consequently, the transparency of the assessment criteria and indicators was higher; iii) it was decided to follow the students' work more closely and provide them regular feedback on the development of the main task (based on the peers' and teachers' appreciations).
Furthermore, in what concerns the first version of the literature review paper, which took place during the 3rd week (see table 1), the formative PA (each group assessed other group) and the formative assessment made by teachers were, in this edition, closed (using the GoogleDocs form). This occurred because in the previous year, the students felt uncomfortable to assess their colleagues openly, even though it was a formative assessment.
As the success of the online PA seemed to be dependent on the students' perceptions about the assessment strategies, a qualitative approach was used again to analyze them. The results showed that the introduced changes led to an increase in the number of students valuing the e-assessment tasks. When comparing the difficulty and the relevance of the proposed e-assessment tasks (see figure 2), in 2008/09, the percentage of students that considered the tasks relevant was between 20% and 40% (depending on the task), while in the 2010/11 edition, at least 89% of the respondents considered the eassessment tasks relevant to achieve the learning objectives [20]. Hence, the results seemed to provide evidence that the students were more conscious of the relevance of the eassessment tasks mostly because they were more aware that they could benefit from PA, since this topic was discussed in the first f2f session by presenting assessment principles underlying the module syllabus as well as the results of the previous experience.
In this edition the teachers decided to analyze the quality of the PA provided by the students. Examples of quality criteria for PA used for this analysis were adapted from Ref. [23] and include the: i) use of the negotiated criteria, adequacy of the chosen vocabulary; iii) provision of constructive feedback (a-criticism, b-questions, cimprovement suggestions); iv) fairness of the score and v) adequacy of the final considerations. The analysis of the quality of the formative PA provided by the students in the 2010/11 edition showed that the overall quality of students' PA could be better (figure 3), since the majority of the groups didn't provide enough constructive feedback. Moreover, although PA included criticism (both positive and negative) and improvement suggestions, the groups didn't question their colleagues [24]. Besides that, the considerations didn't summarize the main positive and negative aspects of the literature review paper. This happened probably because the structure of the instrument didn't have a specific place to it. In the following edition of the module, the PA instrument was revised taking that into account.

C. Third Edition (2011/12)
From the results of the previous edition, and considering the quality principles mentioned by Ref. [25], the usefulness of PA could be better and was beyond the expected.
In the third edition, once more, and accordingly to the previous results, the assessment strategies and process were improved. For example, a content management system (CMS) was used to develop the activities of the module. The CMS provided a common place to organize, plan and work on the learning tasks (literature review and assessment) to the elements involved in the module (students and teachers). Therefore this collaborative environment allowed the students to work openly (all the interactions and documents are available at http://cms.ua.pt/RedeSIDEdu/?q=node/171) in the different tasks. Moreover, the CMS allowed the development of a collaborative bibliographic database.
The assessment framework, used to assess the quality of PA (Fig. 3,) was shared and discussed with the students, as well as the results of the previous edition, in the first f2f session (table 2). With this strategy it was expected to deepen the familiarity and engagement of the students with that assessment tasks and to show them the importance of fruitful dialogue and 'good' feedback [2], [26], as well as that the primary beneficiary of assessment is the student.
Another main change was to provide students with extra opportunities to use the assessment framework of the group work, e.g. the PA was performed twice: in a preliminary version of the paper produced by the group work (in the 2nd week) and in the first version of the paper (3rd week). This helped students to fulfill the task by learning from several rounds of feedback. This decision aimed at the improvement of the students' attitudes and perceptions about assessment for learning, as suggested by Ref. [20] or Ref. [27]. As suggested by Ref [26], students' training could raise the assessment competences of both assessed and assessors. .
On the other hand, the assignment of the assessment tasks to the different groups was made by using CMS and email. In summary the PA was not confidential, compulsory, supplementary (teachers assessed the ongoing group work, after PA), which made the students feel more confident on the task, than in the previous years, as reported below.
An online questionnaire was applied at the end of the module. The results of the closed questions related to PA are triangulated with the students' opinions gathered either during the semester or in the open questions of the questionnaire. Figure 4 summarizes the students' opinions about PA, during the 2011/12 academic year. The figure points out that only one student felt uncomfortable upon knowing the assessment made by his/her colleagues and that four students felt uncomfortable by assessing the work in progress of the other groups (formative assessment).
On the other hand, the majority of the students (9 or more students out of 13) were satisfied with the adopted PA strategy. The results are illustrated with the students' own words. They were satisfied with the: • assessment criteria and indicators -"It was important for the PA to have the criteria and evaluation indicators already pre-defined, so that we can be focused on the most relevant aspects when thinking and reflecting upon the work of our colleagues"; • time spent for PA, nevertheless some students mentioned that "PA was difficult due to the lack of time and knowledge of each other's work"; • commitment with the PA -"This was one of the tasks that I liked the most because it was a way to 'get out' of our group work and be part of the other groups, through reflection"; • PA carried out by colleagues, although it was considered that "… the assessment made by peers should be regarded with some relativism, since they are also training and building their own knowledge, so, in my opinion, the PA must always come with the teacher assessment in order to reduce possible inconsistencies, uncertainties and injustices".
Furthermore, the majority of the students mentioned that PA was helpful as it promoted self-reflection (12 students) and that it was very useful for the development PAPER USING PEER ASSESSMENT FOR PROMOTING THE LEARNING PROCESS IN A DOCTORAL BLENDED LEARNING PROGRAM of their own work and the final results (10 students). Thus, PA "… proved to be a good strategy, both when I assessed or when I've been assessed, since it contributed to selfreflection about the developed work". Finally, five students felt PA difficult and 9 students agreed or completely agreed that formative assessment made by peers was intellectually challenging. For instance, one student pointed out that "PA was initially difficult because it was the first time that we tried it; however, it was very helpful as it promoted self-reflection". One student also referred that "…another relevant point is the focus on PA, because this is something that we are not 'trained' at; therefore this module helped us to create new knowledge and new skills, contributing effectively to our personal and professional development". The literature (for example, [28]) mentions that students may question the fairness of an assessment or disagree with their peers' assessment. In fact, students may regard PA as unfair and often believe that peers are unqualified to review and assess other students' work. Ref [12] also refers that these effects may lead to changes in the collective emotional state of the learning community and affect the organizational climate and pattern of interactions. Considering the results reported above, in the last edition, despite the fact that some students still had some concerns about the new assessment strategies (although less than in the other editions), they clearly perceived the benefits of the assessment for learning. Some findings also showed that students had increasing (comparing the different editions) positive attitudes towards reciprocal PA and some reported that they intended to implement it in their own practices.

V. CONCLUSION
In line with Ref. [23], the findings of the present study suggest that training students in providing constructive feedback can raise the performance of assessment and avoid inaccurate comments. Since the quality of PA can affect its impact [26], the study results provide evidence that this quality can be enhanced by guiding students when performing PA, negotiating the assessment framework, so that students can be familiar with it and produce 'good' feedback. This experience also shows that negative perceptions about PA, namely students' resistance, decreases significantly when students' experience on PA increases. Also, students' fairness perceptions are most significantly associated with their consciousness about the extent to which PA can be useful and positive for their learning. Furthermore, instructional interventions and frequent dialogue between students and teachers may raise an attentive reception of the feedback, since if it is left unattended or not acted upon, it cannot be effective [21], [29]. Figure 5 represents not only the methodological approach of the study, but also the main changes that have been introduced in the Distance Education module, taking into account the results of the evaluation of the different editions (described in detail previously). Design-based approach and summary of the changes done during the academic years as a result of the process' refinement.
In summary, from the results of this study, different challenges should be considered when using e-assessment for learning strategies: • Transparency of the assessment process, as students have to be aware that the primary beneficiary of PA is the student [2]. This transparency can be achieved by analysing and discussing the assessment framework and engaging students and teachers in a process of dialogue and feedback, as mentioned by Ref. [2] or Ref. [29]; • PA may support active and autonomous learning, as it seems to promote a higher order of thinking and lifelong learning and develop students' critical thinking, communication, problem solving, meta-cognitive awareness, deep and creative learning [24]; • PA needs training, the students should be given the opportunity to practice and get more familiar with it [12], enhancing their sense of ownership, responsibility and motivation, so they can find it useful, attractive and enjoyable. As the study encompasses some limitations, related to its qualitative nature, further research is needed. Nevertheless, the study provides several contributions to the field of distance education, namely in blended learning contexts, such as the above reported challenges and the methodological approach that can be very useful since it can be adopted in similar contexts.

ACKNOWLEDGMENT
The authors would like to thank the collaboration of the students enrolled in the study, as well as all the support provided by the Research Centre "Didactics and Technology in Education of Trainers".