SELF-EVALUATION OF DISTANCE LEARNING STUDY PROGRAM AS A PART OF INTERNAL QUALITY ASSURANCE Self-Evaluation of Distance Learning Study Program as a Part of Internal Quality Assurance

This paper features quality assurance of specific distance learning master study program through selfevaluation. This unique program involving e-learning as the program content, as well as delivery method, is presented in the paper from the aspects of its quality assurance. Student evaluation of this study program as a part of the internal quality assurance is performed at the end of every school year in the aim of its quality assurance. Results and conclusions of self-evaluation conducted in this school year by known SEVAQ+ evaluation tool are presented here.


INTRODUCTION
Quality assurance is an ongoing, continuous process of evaluating the quality of a higher education system, institutions, or programs.Many systems make a distinction between internal quality assurance (i.e.inter-institutional practices in view of monitoring and improving the quality of higher education) and external quality assurance (i.e.inter or supra-institutional schemes assuring the quality of higher education institutions and programs).[1] The original Bologna Declaration of 1999 identified the "promotion of European cooperation in quality assurance, with a view to developing comparable criteria and methodologies" as one of the core areas [2].In the 2003 Berlin communiqué the Ministers of Education committed themselves to supporting further development of quality assurance at institutional, national and European level [3].
The Berlin Communiqué requested the ENQA 1 , a European forum for exchange of practice in quality assurance, EUA 2 , EURASHE 3 and ESIB 4 , to agree on a set of standards, procedures and guidelines on internal and external quality assurance and a peer review system for quality assurance bodies.In response, ENQA's published a set of "Standards and Guidelines for Quality Assurance in the European Higher Education Area" in 2005 [4].
One of the recommendations for further development of the Bologna Process was "to extend quality assurance, accreditation and qualifications frameworks to e-learning and other non-classical modes of delivery in an integrated approach encompassing the full range of higher education" [5].
Development of the Distance Learning (DL) Quality Assurance (QA) system required firstly a firm set of responsibilities and activities performed by the higher educational institution.QA of DL has to integrate into the overall institutional QA system.
There is no unified approach to QA of DL adopted from relevant European institutions.Still, there have been several initiatives in the past to address quality in ICT-based learning or e-learning.
For example, The Quality Assurance Agency for Higher Education (QAA) from UK produced a Code of Practice for Quality Assurance in Higher Education.QAA Code of Practice covers the following four areas in terms of QA in DL [1]: 1. System design -the development of an integrated approach 2. The establishment of academic standards and quality in program design, approval and review procedures 3. The assurance of quality and standards in the management of program delivery 4. Student development and support Also, several agencies and organizations from Europe, North America and Australia have published their frameworks and/or models of QA in DL.
Norwegian Association for Distance Education (NADE) is an organization for institutions involved in DL.Its framework/model criteria are divided into four distinct phases: prerequisites, implementation, results and followup [6].These phases are monitored in the context of four broad aspects: information and counselling, course development, education, and organization.They all form a 2dimensional framework/model known as NADE quality matrix.
The SEEQUEL core quality Framework is a result from the SEEQUEL project5.It is an integrated set of quality criteria categorized along the following 2-level set of aspects:  learning sources  learning context The framework comes as a long table, fully available online [7].The ELQ model developed by the Swedish National Agency for Higher Education [8] includes ten aspects of quality assessment in DL, and for each quality aspect several quality criteria have been developed.These criteria are recommendations for concrete measures for dealing with the problems and issues identified at an institutional level.
The Distance Education and Training Council (DETC) is a non-profit educational association located in Washington, D.C.It has gained the approval of the U.S. Department of Education as the "nationally recognized accrediting agency" under terms of Public Law.The Council for Higher Education Accreditation (CHEA) also recognizes the DETC Accrediting Commission, which defines, maintains and promotes educational excellence in DL institutions.The DETC model includes 12 aspects [1].Each of these aspects is covered by criteria similar to those of ELQ.
One of QA procedures in assuring quality of DL systems, processes and institutions listed by [9] and [10] is: devising and implementing continuous review processes within the arrangements for course management and QA for these awards.The process of self-evaluation is a part of continuous review processes within study programme QA.
SEVAQ+ tool 7 for self-evaluation used in framework of the survey concerning student evaluation of the DL master study program in e-learning at the Technical faculty Čačak, University of Kragujevac.The student evaluation of study programs, courses, teachers, services, etc., is the required part of the Faculty internal quality assurance process.

ČAČAK
The approach to internal QA at Technical faculty Čačak is very systematic and in general is the same for all study programs regardless on delivery method.Self-evaluation process is in the function of quality assurance of study programs, teaching processes and working conditions at the Faculty as well as their promotion and development.Internal QA at Technical faculty is defined by following documents: The Quality Assurance Policy and The regulations on self-evaluation of study programs, teaching, teachers' work, services and working conditions.According to those regulations, Faculty established Selfevaluation Commission, which coordinates all activities within the process of self-evaluation.
The student evaluation is the required part of integral self-evaluation process.The process of student evaluation is carried out by The Faculty Self-evaluation Commission, trough surveys at the end of each academic semester, in getting certificates/diplomas and after a certain period after getting a diploma and graduates are experienced in working place.The surveys were anonymous except for the surveys in getting certificates/diplomas and after graduation.The objective of the student evaluation, graduates and undergraduates, is to determine students' opinions on:  the pedagogical work of teachers and associates;  the quality of study programs;  the quality of teaching and working conditions and quality of non-teaching support;  their contribution to the success of the teaching process The Commission creates report on student evaluation, which is based on statistical analysis.If any poor evaluations/scores are determined, steps are taken to improve the specific aspect/issue.Afterwards, Faculty's departments analyze results and prepare proposals for improvement of teaching process.The statistical results are published at Faculty web site and discussed by The Faculty Academic Council and The Faculty Student Parliament.The Faculty uses discussion results and conclusions for continuous improvement of study programs offer and delivery.
Student evaluation of the quality of the teaching process for single course is based on the evaluation of the following:  the course content (volume, difficulty and adaptability), objectives, purpose and structure;  teachers and teaching assistants;  the course organization and delivery (the use of different methods, teaching materials and literature, the organization of exercises, course specificity, etc);  the methods and effects of learning and teaching during the lectures and the exercises and their integration;  the exams procedures for the assessment of skills and knowledge and procedures compliance with prescribed forms;  8 .In this program, QA aspects are taken great care of, and the program as a whole is fully compliant to the principles of the Bologna declaration.
At the end of each academic semester, a poll is conducted with the DL students with the objective to evaluate study program, all study courses, regarding content, the teachers, the service, and the like.The Faculty use results of this survey for improvement of DL study program offer and its delivery in the following study years.This approach is corresponding to the following one from the QAA Code of Practice [1]: Learning, although at a distance, should be treated as an activity that involves all participants in the system, and as an activity in which monitoring, review and feedback are regularly used to enhance all components of teaching, learning and the system of delivery.
Apart from the standard QA procedures for all study programs at the Faculty, two separate evaluations were conducted for distance learning study program.One of them developed by faculty staff, deals with evaluation of the effects, process and qualitative dimensions of the master study program in e-learning.We started it in (2008/2009) and target group consisted of online students of e-learning master study program at Technical faculty in Čačak.The extensive analyses and results are given in [11].The following eight evaluation domains are merged: content and structure, goals and outcomes, teaching organization, evaluation, monitoring and grading, e-content organization and technology demands, LMS and technical support, evaluation of teachers, composite curriculum grade.
The second evaluation of DL master study program in e-learning conducted in 2010/2011 by applying well known self-evaluation tool of Quality in e-learning in VET and HE, SEVAQ+ v2.0.We have applied SEVAQ tool because the existing surveys for student selfevaluation, defined by Faculty regulations on selfevaluation of study programs, teaching, teachers' work, services and working conditions, don't cover all aspects of quality assurance in DL.
SEVAQ+ represents combination of tool and methodology for the self-evaluation of quality in Technology-Enhanced Learning.It merges two widely recognized evaluation and quality approaches -the Kirkpatrick and EFQM (European Foundation for Quality Management) models.SEVAQ+ enables three domains of the evaluation from the EFQM model:  The resources used by the learner during his learning experience;  The processes (activities) proposed to the learner during the delivery of the course;  The results: learning objectives achieved, effects of the experience on the learner, some measure of the transfer in the workplace Each domain contains more criteria, which consist of more sub criteria.Questions are linked with those sub criteria (Fig. 1).
We have chosen 40 questions from the questionnaire.These questions refer to all three domains with the following criteria and sub criteria: Within selected sub-criteria, each statement is defined by three questions (Fig. 2).For the first question students have to assess, on a scale from 1 -4 (from strongly disagree to strongly agree), the extent to which agreed with the statement.The second question is about importance of the topic in the declaration for a given course, also the scale of 1 -4 (from not important to very important).The last question, consults the student about the need to improve the topic (yes or no).

IV. DL STUDY PROGRAM IN E-LEARNING -DISCUSSION OF SELF-EVALUATION RESULTS
There are many conclusions that can be derived based on the survey results.Some conclusions are easier to notice than others.Certain questions that were marked as important by students, received different ratings, depending on the courses.Figure 3 shows the histograms that illustrate the answers to two representative questions: "The student had the possibility to complete practice exercises and self assessments, as required" and "Assessment processes were relevant to the objectives and content of the course".It is notable that for the course "Tools and technologies for e-learning" the rate for those questions has the same number of responses 3 and 4, for the course "Infrastructure for e-learning" prevailing grade is 4, while on the course " Teaching and Learning in e-Education" all grades are present.
The evaluation results are available in the form of radar diagram which one can identify which sub-criteria are the weakest and the strongest for the evaluation.Radar diagram shows in red the overall mean estimate for entire survey and in the blue mean estimation for all the subcriteria.Based on the radar chart (Figures 4-6) the highest overall average rating for courses are:  Infrastructure for e-learning 3.76  Tools and technologies for e-learning 3.49  Teaching and Learning in e-Education 3.42 Sub-criteria that are needed improvement for all three courses:

Course prospectus (off the shelf opportunities),
 There was detailed prospectus, listing all the learning programs available   To improve student access to course presentations and processes. To improve education efficiency by: providing increased opportunities for collaborative and problembased learning; encouraging eLearning practices that can be used to 'free up' class contact time for more productive pedagogical approaches than didactic lecturing; reducing the necessity of excess time teaching areas that can be more clearly illustrated using eLearning tools and storing class resources in a Webbased repository for all hour access. To improve education effectiveness by: enhancing delivery in areas that students typically find conceptually difficult; enabling and encouraging student interaction and structured discussion; facilitating increased levels of tutor involvement with students as a group and as individuals; providing opportunity for preview / review of resources online; providing an overall education context that ensures the sound application of eLearning tools within a course; working with subject matter experts to ensure that technology is applied in a way that identifies their unique needs, and that sets innovative approaches in ways relevant to the subject matter.
V. CONCLUSION Since DL study programmes have to be part of integral institutional study programmes offer, QA of DL study programmes should be integrated into the overall institutional QA system by extensions of current regulations for QA of standard (F2F) study programs.Taking into account the three mentioned evaluation systems, we can derive the cross-matching table showing the coverage of specific evaluations towards DL study programme, Table I.
The table shows quite coverage for several evaluation criteria across the three evaluation tools.However, there are additional DL QA aspects that are not covered or partially covered.Thus, table clearly demonstrates that Faculty official QA self-evaluation tool (developed for F2F study programs), shows the lack in evaluation criteria for LMS and technical support, services offered to the learner, eLearning activities concerning student-student and tutorstudent interactions, training approach, personalization, etc.It is obvious that this should be included as important QA issues for DL studies.Besides, when comparing SEVAQ tool and Faculty's own DL QA self-evaluation tool it can be noted that SEVAQ covers a much wider range of QA issues and thus significantly better collect student feedback and enables correction in each evaluated aspect.Student evaluation polls of face-to-face courses could be adjusted so that applies to DL courses by including, for example, eLearning activities criteria as defined in SEVAQ+ v2.0 self-evaluation tool.
The QA regulations of High education institution which delivers DL study programs also should be include:  the standards for authorship, procedure for review, frequency of review, the criteria for program approval, integration of iterative feedback mechanisms into the course curriculum,  list of requirements (pedagogical and technical) one needs to become an e-learning tutor/DL coursedesigner, quality standards for online communication with students (means of communication and frequency, policies on content formats and types etc.)  procedures for ensuring off quality in technical matters -technical support for teachers and students, the continuity of service procedures, the criteria used in selecting, purchasing and customizing a specific Learning Management System (LMS) Quality culture should be promoted and trough DL program offer.Elements of this might include evidence of iterative feedback procedures, continuing training for tutors, publication policies, research into DL, etc.

8Figure 1 .
Figure 1.A questionnaire based on a three level structure Figure 2. A questionnaire based on a three level structure

Figure 3 .
Figure 3. Histogram of selected questions for all three courses Assessment process management  The assessment process took into account your personal behaviour during the online course (participation, respect of the schedule, etc…)  The mark obtained in the end of course assessment was a fair reflection of the knowledge/competencies developed during the course.Sub-criteria that requires some or none improvement for all three courses:  Awareness of learning preferences  Learner's perspective  Training approach  Navigation and resource options Some of these feedbacks, especially those marked as Improvement needed, helped as to bring conclusions on how to improve the quality of single courses and consequently the whole study program.

Figure 4 .Figure 5 .Figure 6 .
Figure 4. SEVAQ+ survey results for the course "Teaching and Learning in e-Education" SPECIAL FOCUS PAPER SELF-EVALUATION OF DISTANCE LEARNING STUDY PROGRAM AS A PART OF INTERNAL QUALITY ASSURANCEUNIQUe6is the first Europe wide quality certification supporting universities to achieve excellence in using ICT for innovation in learning.The UNIQUe quality criteria break down as follows: SPECIAL FOCUS PAPER SELF-EVALUATION OF DISTANCE LEARNING STUDY PROGRAM AS A PART OF INTERNAL QUALITY ASSURANCE III.SELF-EVALUATION OF DL MASTER STUDY PROGRAM IN E-LEARNING Technical Faculty Čačak, established a new DL master study program M.Sc. in e-Learning, in 2008, as a result of TEMPUS project JEP-41016-2006, M.Sc.Curriculum in E-Learning the student's expectations fulfilment in relation to the scope, level and usefulness of the acquired knowledge and skills;  general assessment of the course iJET -Volume 7, Special Issue 1: "eLearning Belgrade 2011", February 2012

TABLE I .
CROSS-MATCHING OF EVALUATION CRITERIA AND QA SELF-EVALUATION TOOLS iJET -Volume 7, Special Issue 1: "eLearning Belgrade 2011", February 2012