Evaluation of a Virtual Learning Environment for the Professional Training in Public Administration

—In this paper, we describe a framework for the evaluation of a virtual learning environment in the Public Administration context. We explain the question of e-learning education of the Italian Public Administration employees and we analyze the characteristics of these learners. These adults do not belong to the Internet Generation but they have to use the new educational tools, with the problems that this implies


A. E-learning in the Italian Administration contest
Citizens expect public administration services to be of high quality, efficient and transparent; they also expect services to be personalized or at least personalizable to their particular needs. The Italian Public Administration began a process of modernization in order to meet these expectations of citizens and to improve administrative services.
This process involves a re-examination of the role and logic of the work in order to find the key changes required in the Public Administration (PA) work [1]. An important element of this modernization process and for the reorientation of public administration services is the education of employees within the public administration sector [2] [3].
In 2000, the European Council 1 invited the member States of the European Union to adopt e-learning methodologies for the education of the PA employees [4] [5]. Since then, a large number of employees have received training via e-learning.
The European Unions Public Administrations have adopted e-learning for the education of their employees on a large scale because this format for in service-training removes barriers of space and time. Moreover, it allows the use of numerous new educational techniques.
The learning policies adopted by the governments highlight the importance of education in adulthood to be competitive in the knowledge society [4]. This type of 1 The European Council is the highest political body of the European Union. It comprises the heads of state or government of the Union's member states along with the President of the European Commission. Its meeting is chaired by the member from the member state currently holding Presidency of the Council of the European Union. education is oriented to provide and to develop the ability that can be utilised in a professional field, not to make up for knowledge opportunities missed at the time of compulsory school education.
Why e-learning in the PA? The PA is addressed in elearning for different reasons: for e-government, eprocurement and e-commerce. The top priority of egovernment is the improvement of the service. This can be done by improving the efficiency and transparency of the administration task. E-procurement is a set of technologies and procedures that allow a person to acquire services online. E-commerce refers to the on-line commercial transactions.
To carry out these services it is indispensable that the stakeholders have the capacity and the knowledge to produce, manage and archive digital documents.
Therefore the Action Plan for e-government by the Public Function Ministry (29/06/2000) has these goals: 1) To allow citizens and companies to access the services and the information offered by the Public Administration (PA) through the Internet. 2) To find documents held by other Departments by means of the Internet. 3) To facilitate the integration of various services offered by different Departments. The European Public Administration has defined different levels of interaction between customers and the PA: informative, form download, instances forwarding, economic transactions and on-line services. The interaction between virtual PA and citizens could differ from a minimum level of simplicity (for instance, the presence in the Internet of information about administrative procedures and papers) to a maximum complex (to carry out the procedures by Internet) [6].
These goals could be achieved through the redefinition of the roles and the competencies of the human capital. Lifelong learning is indispensable in order to train and update the employees' competencies. Minimum common levels of knowledge make better relationships between citizens-customers and the services offered. The training courses in the P.A. are essentially in these areas [7] [8]: • Training courses for new employees.
• Basic knowledge for all employees.
• Teaching of new procedures.
• Soft skills for managers.
E-learning can reduce expenditure for training and is a methodology that potentially can improve the efficiency and efficacy of the employees [9]. The Central and the local P.A. have both spent money in e-learning, (more money has been spent by the former) [10]. In 2006 AITECH-ASSINFORM (Italian e-learning observatory) led a research about the trends and the characteristics of the use of e-learning in Italy. It had sent a questionnaire to 251 government agencies at all levels of the public administrative body: ministry, province and municipality. The questionnaire had been completed by 45%. From this it emerged that in Italy in 2005, 79% of the Central Public Administration, 27% of the districts and 25% of the municipalities had provided e-learning courses. All together 5,100,000 euros had been invested in e-learning projects. These projects had involved 22,000 civil servants. The majority of the courses concerned informatics and foreign languages.
It is important to know what the employees think, especially about the problems that they had encountered during the course. All the interviewed employees agreed that the lack of tutors, supporting staff and the content material, all make the experience disappointing. Moreover, the interviewed declared that the presence of face-to-face encounters and collaborative work are an important factor that determines their happiness [10].
Considering these results, we have identified the primary needs for civil servants education: • Need to have a helpdesk to solve navigational problems and problems concerning the download of didactical material. Without such support people loose motivation and have a negative attitude toward the course/organization. • Need to communicate and share information with peers/staff. • Need to have the tools to communicate and manage the information. Adults in a context of online education need to learn a new Virtual Learning Environment (VLE) every time they start a new course, with an expenditure of energies and time that are subtracted from study. If the VLE is well framed, the student employs little time to learn how to surf the platform and the functionality of the tools.

B. Characteristic of the adult learners
An important element when adults learn is the diagnosis of the learning needs. The adult learner is motivated to learn if he thinks that a matter could be useful for him. In andragogy there is emphasis on the involvement of the adult learner in the self-diagnosis of the needs that will orientate the learning. Knowles considers this process in 4 phases [11] [12]: 1) The first phase is characterized by the construction of a competency model and the important characteristics for the performance achievement. In this case the expectations of the learner, the teacher, the institution and society are important. 2) The second phase is characterized by the evaluation of the actual level of the competencies. This evaluation is done directly by the learner. 3) In the third phase the teacher can help the learner to measure the gap between possessed competencies and required competencies. This situation of inadequacy could motivate the adult learner to start the new training process in order to make up for his/her shortcomings. 4) In the last phase, the planning phase, it is important that the adults are involved in the planning of the formative path because the adult learner decides intentionally what to learn. If there are too many people in the class, we can have a delegation (or task force) and decide with them the learning plan. The learning plan is the transfer of the needs into specific learning goals, the experience and evaluation plan. For course planning, we have the need to individuate the formative necessities of the learners and all the stakeholders. The stakeholders are interested in having specific expertise, knowledge and transversal competencies that they can apply during the work. The individual interview permits to make a complete and effective requirement analysis. Unfortunately, the individual interview is rarely used, perhaps because it has a high cost and attention. The quality of the analysis depends on different factors [13]: • On the professional awareness of the people that conduct the analysis (namely on the ability to analyse the working situation and on the capacity to have a dialog with the managers). • On the global cultural level of the Administration.
• On the analysis organization process (research of a balance between quality, in depth examination and commitment of few human resources). We have different variables to consider for the analysis of the formative needs [13]: a) The formative actions: they could be the work placement of the new civil servant, the refresher training, the skills update associated with the task efficiency, the professional development of knowledge linked to a specific professional figure, the managerial development, etc.
b) The recipients of the training: this variable is connected with the formative lines. The aims of the formative intervention are related to the clear identification of the recipients.
c) The formative goals: defined as the outcome that must be achieved with the formative action. The goals of the formative action can be direct and indirect goals. Direct goals can be the improvement of the competencies and the creation of new professional figures. Indirect goals can be the support for the changing in the administration and the solution of organizational problems. Another important factor is that the goals can be defined at the global level of organization or at the level of single organizational units. The financial resources are given with respect to the priority of each goal.

C. The pedagogical model and the virtual learning environment
There are many learning theories but no one of them could be considered universally valid. For each theoretical model we have a specific didactic learning methodology setting: for example programmed instruction (behaviourism), problem solving (structuralism), and non directional instruction (psychodynamics). Different paradigms give origins to different Virtual Learning Environments. For instance: if we consider learning as a product, we shall adopt a model of knowledge transmission. It will be a sort of learning centred on contents and their capability to transmit knowledge, while the role of the student group will not be of the active type. On the contrary, if we consider learning as a process, we shall choose a co-operative environment, centred on the student. This way of considering learning sees interaction as the focus of learning and the exchange of interactions among teachers, students and tutors as the process of acquiring knowledge. In order to make this happen, it is necessary to apply those tools which may favour on-line communication among stakeholders, which is the people directly or indirectly involved in the VLE.
On the basis of what has been described so far, we can infer that great attention has to be put on the choice of tutors (who are supposed to promote co-operation) and on the development/choice of materials and environment (which must satisfy all requirements of accessibility in order to allow all users to live at the best their on-line learning experience) The pedagogical approach of the course must supply the main guidelines for the development/choice of the platform.
Keeping in mind the profile of the target (adults in working environment), their different cognitive styles, the type of contents, the objectives to be achieved and the kind of environment in which it has to be realised, the most suitable pedagogical model seems to be a mixed training model, based on constructivism. Its main features are: interaction among pupils, communication many to many, possibility to follow an individualized training process and the presence of a tutor.
Trying to simplify a complex paradigm such as the constructivist paradigm, we would like to consider a few fundamental elements which can define its mission, then we shall try to define the most effective Virtual Learning Environment and finally to reach the identification of the criteria to evaluate an e-learning environment.
Starting from Merrill's famous synthesis [14], we shall list five key elements in the constructivist paradigm: knowledge as personal construction, active learning, cooperative learning, the role of context, and intrinsic evaluation. Now let's try and see which e-learning environment can be built in order to satisfy the requirements of this pedagogical model. a) Knowledge as personal construction. How can elearning satisfy the need to personalize the processes, itineraries and results of learning? The differences in learning styles require the use of different strategies which may be personalized according to the student's characteristics. Therefore it is necessary to prepare flexible learning itineraries, which may offer such a composed didactic proposal to stimulate different types of learning: global-analytical, audio-visual, autonomous-cooperative and systematic-intuitive learning. From this point of view e-learning environment can satisfy a great variety of learning styles. Let's list some of them: Those who have an analytical kind of learning use the inductive or discovery method, namely they first analyse details and then obtain a global vision, therefore they learn information in small units, tackle problems by dividing them and solving sub-problems, tend to follow instructions and have a mnemonic type of study [15]. For this kind of students, it is necessary to organize a structured environment, with tasks arranged in strict progression, and the methodology of problem solving can be useful. Furthermore, the tutor plays an important part, accompanying the learning process, giving orientation and helping focus on key elements.
• The main feature of global learning is the deductive method, which starts first from the global vision to arrive then to detail analysis. A student who has this kind of approach uses the context to understand; he is attracted by fantastic aspects, does not give great importance to mnemonic study, and mainly identifies guiding ideas. E-learning can offer him several tools, such as the use of concept maps, to get a global view, forums, to compare and share his theories with others, simulations, to acquire knowledge through multiple variable control thanks to the possibility of acting in a complex context. • Audio-visual learning can be stimulated through the organization of lessons in videos, patterns and images. • Autonomous-co-operative learning can find in the discussion forum an important tool to discuss and share knowledge both with fellow pupils and tutors and teachers. • Systematic-intuitive learning. Systematic learning can be helped by an appropriate composition of modular learning itineraries, while intuitive learning can be supported by a reticular structure of contents, which may allow the student to study the topics in depth following his own logical itinerary. For instance, the hyper text, both in the practice of navigation and the experience of construction, allows the student to structure and organize his knowledge in a personal manner. Besides, the intuitive style can be stimulated by a non-structured discovery approach, such as exercises, simulations and problem solving. b) Active Learning. To involve the student, the learning by doing methodology might be useful. With this methodology, we can have a learning typology that is born from direct experience and from the reflection about this experience. Experimenting methods, rather than assimilating contents, enables to learn in a meaningful and lasting, as well as simple, way. This modality is fruitful especially in technical and practical matters, for instance to learn how to use a software or surf the Internet, the best modality is by having practical experience. Instead, for vehicular abstract contents we can use the anchored instruction: the student solves the problems anchored on a realistic situation (simulation, storytelling, case study). Moreover, active learning could be stimulated structuring the learning object in reticular mode, with no rigid path but paths that are flexible and adaptable to the curiosity of the learner. c) Collaborative learning. E-learning has different tools to stimulate the interaction between the students and between the students and the teacher/tutor. The tools that we can use for collaboration are: communication systems (synchronous and asynchronous), systems for resource sharing, systems to support group processes (calendars), tools for collaborative writing.
d) The role of the context. In order to reproduce situations which may happen in real life, it may be useful to adopt managerial games which allow the simulation of enterprise management in protected conditions, without any real risk. In this way it is possible to learn the basic concepts of economics and the emotional involvement deriving from game dynamics can be useful to fix what has been learnt in the learner's mind. The birth of professional communities of practice fosters growth and dialogue.
e) Intrinsic evaluation. The student's learning is an important indicator to verify the efficacy of the system/course. The metacognitive awareness of the student is fundamental in the constructivist model. It is important that the students know the indicators that are at the basis of the evaluation. Thus, they can operate selfevaluation process. In a work contest, it is better to make not an evaluation of the notions but to analyze the capacity of the student to reprocess and apply the knowledge. Therefore, the goal of the evaluation is double: it is useful for the teacher who is then able to have information to reorganize the formative path and adapt it to the students' needs. It is useful for the students who can then understand the strength and weakness points of their method of study. E-learning can join together quantitative and qualitative evaluation procedures. We can use qualitative techniques such as narration, participant observation, and other techniques. Diary's note, interviews, spoken thoughts could be useful instruments.
Moreover, quantitative techniques can be utilized, i.e. comparing a pre-test with a final test for measuring knowledge.

A. Criteria for evaluate a Virtual Learning Environment
It is essential to provide the employees-learners with a good quality learning object and Virtual Learning Environment [8] [9] [16]. For these reasons, we had worked out a framework to evaluate the quality and the pedagogical efficacy of the Virtual Learning Environment (VLE) from the point of view of lifelong learning. The evaluation offers feedback to developers, tutors, teachers and the other people involved, and it is an important part for the quality guaranty [17]. We are interested in finding general principles that are useful for the planning of the Public Administration employees' learning environment.
We describe the pedagogical implications of the adoption of e-learning in adult education, and the reasons for its broad use in Public Administration.
The virtual environment is a social space, and in this system a set of people (students, tutors, teachers, administration assistant, etc) interacts, having different roles and needs. For this reason, the evaluation of an elearning environment embraces the evaluation of the process of delivery, the maintenance, and the people involved [18]. The evaluation instruments could be created in order to estimate the quality expected, looked for and reached. For these reasons it is necessary to detect indicators able to find the minimum criteria of quality [19]. In a Virtual Learning Environment, there are different elements that can be considered in order to evaluate the quality of the learning system. We have considered: • The quality of the communication system.
• The quality of the support services and the didactic staff.
• The quality of the administrative services.
• The quality of the learning objects.
We also analyzed the learners' needs regarding on-line education. The aim of this study was to find a learning environment suitable for Public Administration employs.

B. The quality of the communication system
The choice of the interaction and communication tools depends on the following different factors: the learning environment management, the communication between students and the educational staff, and the tools that are available in the platform.
The communication tools could be: • Synchronous (i.e. chat, videoconference and shared blackboard). • Asynchronous (i.e. electronic bulletin board, messaging, calendar, document repository, and Wiki). The presence of many communication systems is not a guaranty of the platform quality. Generally, their quality can be verified in itinere and ex post depending on the frequency of use. However, we think that in a wellstructured environment the presence of basic tools, such as telephone, chat, forum and e-mail, are essential to guarantee the student's assistance and the guidance. In a perspective of cooperative learning [20] [21] [22] it is important to plan a space for sharing the material that is produced by the students.

C. The quality of the technical and the didactical services
The support services change depending on the chosen learning paradigm. For the training of the Public Administration employees we have chosen a paradigm based on cooperative and collaborative learning. In this case, colleagues are not in competition, but cooperating with each other. This situation can valorise the different competencies and experiences. For these reasons, the support services could include the orientation and pedagogical support and the technical support.  Those services can be offered either using illustrated brochures, or face-to-face meetings, or interacting on line with a tutor. The use of practical demonstrations of the platform tools, i.e. using audiovisual formats, is another important element to guarantee the quality of the service. In this way, everybody could easily perform various tasks following the instructions given for example by a guiding voice or watching a video.
The presence of qualified staff that can readily help students solve navigation and platform use problems. It might be useful either the creation of a thematic forum, or a FAQ, or the possibility to send an e-mail to the technical support staff, or the activation of a telephone help line (if the students cannot connect to the platform via internet), or, even better, the possibility of making an internal search in the forum using a search engine.
In the case of collaborative learning, the tutors have to be able to explain to the learners [Cerfad, 2002] what "to learn" means using a collaborative method; to emphasize what the students might acquire participating to the workgroup and in which way they can contribute to the group activities [23].
The tutors can function as pedagogic support animating, moderating and facilitating the discussion between the students. They have to guarantee the involvement and participation of all students to the activities and tasks, to facilitate the achievement of the shared goals, and to help the students giving them useful information. Tutors and the didactical staff involved in the student assistance are required to use an informal communication style to make the students feel more comfortable. It plans an initial blended training.
It scheduled an initial on-line training, with benefit. It involves the teaching staff before the course starts, inviting the teachers to communicate utilizing the platform with the tools. (in this manner the teacher could acquire experience about the on-line communication.
During the course the teacher have joined a person expert in e-learning The teacher has already taught in an elearning course. The teacher has experience as a student in an e-learning course. The teacher has had experience in tutoring activities. The teacher has a positive conception about e-learning. The teacher has experience of collaborative writing. The teacher has experience in participation a forum.
The main functions of the tutor are: [24] facilitation, support, guidance, motivation and assistance during the study and the evaluation. The tutor has the duty to collect the reports of the individual and the group activities and to analyse the essays.
The online services of learning support like tutoring and mentoring (the mentor gives a more informal support than a tutor) give the student the possibility of organizing his own learning activities utilizing the most appropriate methods in the e-learning modality. Ohiou University proposed a useful and complete analysis sheet for the evaluation of tutors [http://www.ohiou.edu/aac/tuto ring/tutor_eval.html] and we used it, together with the MECA guide [http://www.adeit.uv.es/mecaodl/], to develop our evaluation table.
The support services could be evaluated at the start, during and at the end of the course. The didactical staff could be evaluated based on the strength of the participation in the synchronous and asynchronous activities.
The support services could be evaluated at the start, during and the end of the course. The didactical staff could be evaluated on the strength of the participation in the synchronous and asynchronous activities.

D. The quality of the administration system
In the evaluation of the virtual learning environment, it is important to know how the learners perceive the administrative services such as the enrolment (easy or complicate), payments, quality of the virtual library, etc. This evaluation is necessary during the following phases of the course: the design (for example, with focus group), the supply and at the end. The students' opinion on the administrative services of their course is important to improve the quality of the course itself. The basic data to consider are the level of satisfaction of the students. Khan [18] presents a broad collection of indicators used by various institutions to evaluate the quality of administrative services.

E. The quality of the learning objects (LO)
Accessibility, usability and didactic are important factors that influence the quality of e-learning material. We have identified the characteristics a Learning Object (LO) must have to guarantee the students a complete and effective learning experience so that it may be a didactic item which facilitates learning and supplies competencies and/or theoretical contents [25]. We have created an evaluation table to estimate the LO in the pipeline. We have defined 3 main macro-categories for the evaluation: organization, presentation and didactics, discussed in Sannia & al., 2007 [26].
For each macro-category, we have defined some relevant categories. Thus in the LOs organization we have inserted the meta-cognitive framework and content structure. Presentation mainly refers to the media used in the LO and its usability. In the didactic material section, we analyse the quality of didactic strategies and evaluation tests.
At the end of the table it is also possible to have an overall evaluation of the LO. During LO preparation the table has been filled by the author himself as well as by third parties, in order to verify the LO quality.
III. CONCLUSION Many PA are actually using Virtual Learning Environment to supply administrative services and for the training of their employees. The employees-learners have an ambivalent reaction to e-learning: on one side they consider it adequate to their status of working people; on the other side they are still linked and used to a traditional face to face education in which the teacher has a role of "knowledge transmitter".
In conclusion, we supplied a variety of important elements for the design of an e-learning course. An efficient course must be consistent to the proposed course, the organization strategies and the roles performed by the employees. Therefore, it is important to offer not a generic course but training finalized to provide a professional improvement. The criteria that we have presented would be a useful tool to carry out evaluation by e-learning designer in the PA. The indicators that we had proposed here are the general criteria of quality. We have thought that these criteria can be useful to reflect about the things that we consider important in the VLE design.
We think that the efficacy of the learning experience can be estimated through impact evaluation that is the impact that the knowledge had in a specific Public Administration. Therefore, the achievement of a VLE depends on many factors: easy access to the information, staff expressly trained to help the students, an effective administrative service, and didactical material prepared on purpose.