Mapping Learning Outcomes and Assignment Tasks for SPIDER Activities

Modern engineering programs have to address rapidly changing technical content and have to enable students to develop transferable skills such as critical evaluation, communication skills and lifelong learning. This paper introduces a combined learning and assessment activity that provides students with opportunities to develop and practice their soft skills, but also extends their theoretical knowledge base. Key tasks included self directed inquiry, oral and written communication as well as peer assessment. To facilitate the SPIDER activities (Select, Prepare and Investigate, Discuss, Evaluate, Reflect), a software tool has been implemented in the learning management system Moodle. Evidence shows increased student engagement and better learning outcomes for both transferable as well as technical skills. The study focuses on generalising the relationship between learning outcomes and assignment tasks as well as activities that drive these tasks. Trail results inform the approach. Staff evaluations and their views of assignments and intended learning outcomes also supported this analysis.


INTRODUCTION
Engineering cohorts are becoming more diverse and include, for example, mature age students and students from diverse social and educational backgrounds.This effect is elevated at the University of Southern Queensland (USQ), a small regional university, where approximately 76% of the cohort are distance education students.These students are located off campus and access learning materials in a variety of formats including printed material and electronic content.To address transferable and technical skills in an equitable manner between on-campus and offcampus students is difficult as distance students have limited opportunities to engage with lecturers and their peers.The cohort is also very diverse.On one extreme are young digital natives [1], on the other extreme are mature edge students with a wealth of professional experience [2].Computer literacy does not follow this classification [3]; however, practical skills and motivation generally follow the age boundary.
This paper reports on an assignment structure that is supported by a custom implementation in the Learning Management System (LMS) Moodle [4] and focuses on the issues outlined above.The strategy has been named SPIDER: Select, Prepare and Investigate, Discuss, Evaluate, Reflect as it centres around these activities.The overarching goal is twofold, to engage students in assessment tasks and to encourage teachers to use a formal process to linking learning outcomes to learning activities and assessment.
The assessment tool incorporates a number of features including, topic selection, content preparation, investigating and research, discussions and peer evaluation and reflection.These key tasks have been widely acknowledged as activities with positive learning outcomes.For example, benefits of self and peer assessment include improved motivation and ownership, critiquing, evaluation skills and lifelong learning [5].For graduates to develop lifelong learning skills, they must also become adept at objectively assessing their own learning [6].The initial assessment structure [7] has evolved into a comprehensive learning model.To support this assessment, a prototype has been developed and trialled in a number of third and fourth year courses.Perception and learning outcomes were evaluated by student surveys, academic results and interviews with students.The evidence shows that this approach not only improves off-campus student engagement, but also learning outcomes.This includes both, soft skills and technical competencies.
Comments by staff have suggested that this integrated activity and assessment structure would be useful in other courses ranging from first to fourth year and across discipline majors.The overall approach remains consistent but individual requirements vary between courses.The aim of this work is to evaluate the transferability of this learning approach, extend the technical results and place the initial results on a firm scholarly and research footing.A new technical implementation and further development work was necessary to seamlessly integrate the activity with the learning management system.A formal evaluation of learning outcomes and their relationship to tasks, supported by these tools were undertaken to make the system more accessible for other academics.This serves two purposes; it helps with rigorous requirements analysis of the system, but also increases staff awareness of alternative learning and assessment strategies and of implementing a more scholarly approach to assessment in general.This evaluation results in a matrix that correlates intended learning outcomes with assignment task.In turn, this information is used to identify specific assignment flow charts and structures.There is clear evidence that SPIDER assignments enhance learning outcomes and the approach is transferable to a multitude of settings.It leads to more efficient assessment of student learning and most importantly it helps to drive student learning.
The remainder of the paper is organised as follows.Section II discusses the theoretical framework that forms the bases for this project.Section III highlights the motivation and the methodology that was employed.The SPIDER tool and its relationship to learning outcomes are discussed in Section IV.Section V presents flow charts of SPECIAL FOCUS PAPER MAPPING LEARNING OUTCOMES AND ASSIGNMENT TASKS FOR SPIDER ACTIVITIES actual and potential assessments.General observations and future directions are outlined in Section VI.

II. THEORETICAL FRAMEWORK
To make sure that students gain the intended graduate attributes and achieve the prescribed learning outcomes it is important that the learning activities and assessment tasks are linked.To close the learning cycle and ensure that competencies are met, constructive alignment [8] has been a strong focus of educationalists but also engineering bodies.[9] introduces a generalised pedagogical model in the context of electronic tools that highlights the relationships between outcomes and tasks.
The model centres around five nodes: graduate attribute, learning objectives, learning activities, assessment tasks, assessment criteria and standards.The authors suggest that course development can start in any of the five nodes and then more iteratively through the process and they acknowledge that the model is general and omits a number of linkages which are not included.This paper uses the model as a starting point to engage lecturer in a process of developing assessments that follow a formal process.
In this particular case the focus is on a predefined assessment tool that includes a number of components that naturally lead to a specific set of learning outcomes that are constructively aligned with the subject/course objectives.The aim is to integrate learning and assessment tasks as it is difficult to engage student with learning activities if they are not directly rewarded with marks.Literature suggests e.g.[10,11] that this lack of engagement and focus on grades impacts significantly on student learning with only surface learning occurring rather than the deep learning.
Therefore this work suggests a simplified four node model that combines learning activity and assessment task.Figure 1 depicts a graphical representation.Graduate attributes are expanded to specify learning objectives.These objectives are achieved though learning activities that also form part of the assessment.The measure is based on criteria and standards that demonstrate the attainment of graduate attributes.The authors believe that this presents a more suitable approach to capture the purpose of the SPIDER tool.
The benefits of the individual components that are employed by SPIDER have been widely acknowledged in the literature.Key references are summarised below.
Rubrics are systematic scoring methods that use predetermined criteria to measure student learning for grading assessment.[12] suggests that this enables a more objective and consistent assessment of student work.Marking rubrics also clarify grading criteria and demystify the marking process.Students' self-judgement is fostered.Practical suggestions, examples, guidelines and principles of effective rubrics use are provided by [13].
Potential benefits of self and peer assessment are acknowledged by the research community, e.g.[14,15].Concrete benefits include improved motivation and ownership, critiquing and evaluation skills.[14] highlights that graduates that are expected to develop as lifelong learners, must be able to objectively asses their own learning.[12] enforces this view by that suggesting that greater emphasis should be placed on technology-supported tools and techniques to assess context based learning; rather than disempowering learners with strict summative assessments.[5] adds that peer assessment also improved 'soft skills' and [15] proposes that student assessment of other students' work has potential learning benefits by encouraging student autonomy and higher order thinking skills.
Students need to become 'reflective practitioners' able to critically reflect on their own professional practice [16][17][18] which in turn contributes to their life-long learning skills.The need for these skills will only increase in the future as the acquisition of knowledge and skill cannot be restricted to any particular phase of education and much continue throughout professional life [19].
Independent of the benefits of individual activities, diversified learning opportunities will be beneficial to all students as variation theory suggests e.g.[20].The following section describes the initial motivation for this project and the methodology that has been used to undertake it.

III. MOTIVATION AND METHODOLOGY
The initial focus of this approach was a third year introductory course into computer systems and communication protocols.The course is offered to undergraduates in their final years and postgraduate master by coursework students.The goal of the course is to provide a basic introduction into computer systems and the Internet.A number of reasons have motivated the introduction of this new assessment structure that led to the development of the SPIDER tool.
As suggested above, there is an increasing focus on generic skills, graduate attributes and constructive alignment.This need for new skills is also motivated by changing and diverse student cohorts.For an increasing number of students the Internet is becoming a key source, often the main source of information.Therefore is essential for students to acquire and practise critical thinking skills to assess and evaluate this information.The course content in many areas of engineering is rapidly changing and including alternative learning opportunities can assist with the currency of the teaching material and foster independent inquiry.This also helps to provide up-to-date study materials to distance students.
Another aspect relates to flexibility, in particular for distance students.Traditional assignments offer little choice to mature age students with a range of prior professional and educational experience and students with other SPECIAL FOCUS PAPER MAPPING LEARNING OUTCOMES AND ASSIGNMENT TASKS FOR SPIDER ACTIVITIES commitments as to when the assignment has to be done.The SPIDER approach can be used to give more choices in assessment topics as well as assessment due dates.This effort also encourages students to work outside their comfort zone and to try new things.Last, but not least, the engagement of distance students with the course and subject material is an important driver for the new assignment and learning approach.The overall aim is to foster student engagement, provide tools for lifelong and reflective learning and naturally teach the relevant technical content.
To address these issues a comprehensive new assignment structure was devised [7] that include individual student reports and presentations as well as peer assessment.As there were no electronic tools available to support the new assessment strategy, a prototype was developed.This was a web-based solution using a php/mysql environment.This was necessary as the number of peer assessments surpassed 1,400 in a course with about 70 active students.This volume would have been impossible to handle manually.The assignment and the tool were tested in four classes in 2009 and 2010.The impact has been assessed by a number of surveys, student interviews and analysis of examination results.
These trials informed the design and the specification for the SPIDER tool.To expand the scope, a workshop with academic staff at the Faculty of Engineering and Surveying and follow up interviews was held.The key aims were to identify additional requirements of the tool and to explore linkage between activities and learning outcomes.The workshop sought to encourage academics to think about, and clearly define, the assessment strategies in a variety of undergraduate courses.It highlighted the fact that many academics do not use assessment as a learning tool and have not rigorously made all the links between graduate attributes, learning objective, learning activities and assessment criteria.For example many academics, whilst having carefully planned assessment activities and assessment criteria have not clearly link them back to graduate attributes.Others have clear and precise learning objectives, but again have not sort to rigorously align these with graduate attributes.The workshop highlighted these weaknesses and the SPIDER tool assisted in replanning assessment activities and learning objectives.

IV. THE SPIDER TOOL
The SPIDER tool has been implemented as an assignment type in the Learning Management System (LMS) Moodle [4].It is based on the workshop module and it is able to support assignments that can address a number of learning outcomes.It consists of multiple assessment components that are based on marking rubrics, including self assessment, peer assessment, marker assessment and moderation.The tool supports multiple submissions per assignment, e.g.reports and presentations.Results are presented with a comprehensive statistical analysis.The tool also includes a reflection component that allows students to comment on their final results.This was included to force students to revisit assignment feedback and reflect on the assessor's comments in an attempted to close the learning cycle.
One of the key differences to similar other tools is the possibility to have individual deadlines for each student submission.This allows this tool to be used for seminar style classes where students present different topics in different weeks.It can also be used to assess non elec-tronic work such as art pieces or performances.For particular assignments, the various components of the tool can be individually activated or deactivated.This leads to a number of potential assignment workflows.Example flow charts of assignment strategies are discussed in Section V.
To decide which components are suitable for what learning outcomes, a decision matrix has been devised.Table 1 shows matches between components of the SPIDER tool and individual learning outcomes these activities foster.Topic section offers student the opportunity to select an area and topics can have different deadlines.Background research is a self directed inquiry task and undertaken offline.Presentations and reports are also prepared offline and summarise the findings of the research activity.On campus students give their presentations in class, external students' record their presentation and upload it to the system.Student submissions are self assessed.The system supports group as well as individual submissions; however, self assessments are always activities that are undertaken by an individual.A key component is the peer assessment that requires students to mark the submissions of other students.The final reflection task gives students the opportunity to comment on their results.
Table 1 also includes numbers that refer to Engineers Australia graduate attributes and ABET Criteria that are listed in Table 2. Most attributes are addressed by the individual learning activity tasks.However, to a large extend more specific reference to learning outcomes and graduate attributes will depend on the actual assignment, i.e. specifications that include the subject specific content and deliverables.
V. ASSESSMENT WORKFLOWS This section introduces four sample assessment workflows that highlight the capability of the tool.Rounded boxes (yellow) indicate tasks that are undertaken by lecturers, square boxes (blue) show tasks that are done by students.Boxes with dashed borders indicate offline activities.The flow charts include actual as well as future workflows: Chart C has been used numerous times.It is envisaged that Chart A and C will be used in Semester 1 2011.Chart D describes an application that is currently not used.

A. Simple Assignment
The first example, depicted in Figure 2, shows a simple assignment workflow that largely replicates a traditional assessment strategy.Students submit assignments and these are subsequently marked by lecturers.The only difference is that it uses an online marking rubric for the assessment.The simple workflow can be extended by student self assessment, peer assessment and a reflection phase at the end to address further learning outcomes and students engage.The following examples show more complex structures.

B. SPIDER
The original application of the assessment structure was an introductory computer systems and communication networks course as discussed above.The main assessment item consists of a report, a presentation and peer assessment of 20 peer submissions.The tasks account for 10%, 10% and 8% of the final marks, respectively.One of the key differences between a traditional workflow and this structure are individual topics and deadlines for students.Figure 3 depicts the complete workflow for this assignment.The two bars on top show the phases of the online tool and how they match the SPIDER approach of Select, Prepare and Investigate, Discuss, Evaluate, Reflect.
The lecturer configures the tool during the setup phase.Submissions are added; in this case, reports and presentations.Topics and their respective deadlines are configured.Marking rubrics are defined, for both, reports and presentations.The tool is changed into the selection phase.During this phase students select one of the topics.The next phase of the tool covers the major part of the activities.Students undertake the background research and prepare presentations and reports.External students record their presentation, on campus student present their findings in class.Both, report and presentation are uploaded.
Students self assess their submissions and submissions are also peer assessed.After the topic selection is completed, a given number of topics (e.g.20) are randomly assigned to each participation student.Throughout the semester, the student has to assess those assignments as they become available.Once all topics have been presented and marked, the tool is moved into the moderation phase.The lecturer gets an extensive overview of all the results that relate to the assessment item.At this point the marks can be adjusted and bonus or penal marks can be added or deducted.In the final reflection phase, students have the opportunity to comment on their results and the lecturer can adapt the results in response to comments.

C. Problem Solving
The Faculty of Engineering and Surveying at USQ has four courses which utilise a Problem Based Learning (PBL) paradigm [22].All students are allocated to teams [23] and via electronic communication media undertake, as a team, finding solutions to a number of open ended contextual problems.The assessment in each of the courses vary in accordance with the specific course learning objectives but the underlining approach of the assessment structure is that students undertake an assignment task, receive feedback and then submit an updated version of the assignment.Details of this assessment method are described in [24].However due to increasing student enrolments, undertaking peer assessment (individual and team) is administratively unmanageable.In addition there is significant evidence to support the advantages to student learning of self and peer assessment and it is believed that this should be expanded to other assessment items in the course.
It is envisaged that the SPIDER tool will be used in S1 2011. Figure 4 depicts a flowchart for problem solving courses.The initial setup phase is similar to the previous two examples.Submissions in this case are group assignments.All team members self assess the common submission.Students from other groups, peer assess the initial submission and the peer assessment is moderated.Groups prepare and submit a second assignment that has been modified based on the feedback.The second submission is again self assessed and assessed by markers.During the reflection phase, students have the opportunity to comment on their final results.

D. Professional Practice
Many engineering programs include a final year capstone project.In the Faculty of Engineering and Surveying at USQ students undertake a two semester research project.As part of the formal project milestones, students have to complete a number of deliverables including a project specification, project appreciation, initial project seminar, project conference presentation and the final project dissertation.Figure 5 depicts the proposed flow chat which shows a process that gives students multiple additional learning opportunities.Students have to submit a number of deliverables; hover, they have also the opportunity to comment on the work of others.
The initial project seminar and final project conference presentation offer ideal opportunities to engage the whole student cohort in the assessment process.Currently students must attend presentations and student presentations are assessed by an academic staff member.However the student audience is not attentive and an ideal learning opportunity is being missed.Providing as well as receiving feedback can be a powerful learning tool as well as a necessary professional skill.A similar structure could also be used for postgraduate research seminars.

VI. OBSERVATIONS AND FURTHER DIRECTIONS
The tool offers great flexibility to design engaging learning activities that are assessable at the same time.One issue is the complexity of the tool.Initially academics will require support to use the SPIDER tool to its full extent.Current activities focus on ways how the tool and the methodology could be further simplified.This paper is an initial attempt to address this issue.
SPIDER is fully integrated in Moodle 2, the current USQ Learning Management System.There are no direct technological barriers specific to the tool.Especially from a student perspective, required actions are listed on the screen and have to be completed to proceed.
There has been extensive evaluation of the trial implementation; however, the trials were limited to two courses only.It will be interesting to investigate if the outcomes are directly transferable to the new tool and what the student's perception will be.
Academic staff must also be willing to engage in the process of critically reviewing and changing assessment practices.[25] notes that rarely do academic staff act on issues covered in professional development workshops and seminars and for successful outcomes a "one-to-one collaborative, mentoring relationship" between developer and academic should be encouraged.The SPIDER project offers an ideal opportunity for the development of scholarly knowledge of assessment and changing assessment practices in a practical and constructive environment.
Similarly student perceptions of assessment must be understood.[26] discussed shifting practices from testing or evaluation to assessment where this integrates teaching and learning and involves students and active and informed participants.The assessment tasks must be contextual, meaningful, engaging and authentic and contain elements of self and peer review [27] peer review.SPECIAL FOCUS PAPER MAPPING LEARNING OUTCOMES AND ASSIGNMENT TASKS FOR SPIDER ACTIVITIES VII.CONCLUSIONS Previous work has shown that this assignment structure helps with student engagement and student learning; the SPIDER tool is a implementation in Moodle 2.0 that can be readily used by others for a variety of assessment strategies.The tool provides a number of important stepping stones to encourage a scholarly approach to assessment.Firstly it explicitly links graduate attributes, learning outcomes, learning activities and assessment criteria.This may require the academic to revisit both the learning objectives of the course and current assessment practices placing these in an appropriate scholarly and theoretical framework.It is envisaged that the completed tool will be made available to the wider Moodle community.
Refocusing from testing or evaluating to a constructive assessment culture has well defined and evidenced links to improved student learning.Further development and rigorous evaluation of the SPIDER tool is required but initial data suggests that it offers many advantages in supporting changing assessment strategies and enhanced student engagement and learning.SPECIAL FOCUS PAPER MAPPING LEARNING OUTCOMES AND ASSIGNMENT TASKS FOR SPIDER ACTIVITIES entific articles and one patent.He is a member of the Telecommunication Society of Australia and Engineers Australia; and he is a senior member of the Institute of Electrical and Electronics Engineers (IEEE).
Lyn Brodie is an Associate Professor in the Faculty of Engineering and Surveying.Her research interests include engineering education, Problem Based Learning, assessment and the first year experience.She is a board and founding member of the USQ Teaching Academy and Director of the Faculty Engineering Education Research Group.Lyn was the academic team leader for the teaching team which successfully designed a strand of PBL courses for the faculty.Her work has been recognised through several awards including a University Award for Design and Delivery of Teaching Materials, Carrick Institute Citation and Australian University Teaching Award for Innovation in Curricula Learning and Teaching, USQ Associate Learning and Teaching Fellowships for curriculum and assessment development and recognition from the Australian Association of Engineering Educators for innovation in curricula.

TABLE 1 MAPPING
SPIDER COMPONENTS AND LEARNING OUTCOMES.