Case Study —Quality of Continuing Engineering Education: DAETE and EFQM Quality of Continuing Engineering Education: DAETE and EFQM

— DAETE was a project created to develop accreditation procedures to ensure quality of continuing engineering education (CEE) courses. Financed by the European Commission and by the USA government it was adapted to evaluate and to improve quality of the centres on both sides of the Atlantic to foster recognition and cooperation. Among the several quality management models available in the academic and industrial contexts the partnership chose EFQM. The model was developed in two years with four partners from Europe and one from the USA. The development consisted in the adaptation of the EFQM descriptors to the specific contexts of CEE centres. Issues like dimension of centres, relevant results, processes relevant for the goals and appropriate indicators were thoroughly discussed and defined. During the years that followed the implementation and validation occurred with similar financing and with the participation of about 100 centres around the world. The analysis of the implementation of the modified EFQM variant in several contexts is made in this paper and conclusions are presented. The conclusions also include recommendations for the adoption of the EFQM model to organizations involved in education and training.


1
The Beginning During the World Football Cup in 2006, a group of practitioners and experts in continuing engineering education (CEE) had a meeting at the Dipoli centre of Helsinki University of Technology (HUT) to discuss quality issues of CEE. One of the issues was about finding a method to recognize and accredit CEE programs across the Atlantic. The goal was to ensure quality of the CEE programs that could be shared by centres in Europe and in the USA. Based on the opportunity that the European Commission and the USA Department of Education had an open call of the recently launched Atlantis program it was decided to work on a proposal called DAETE.
DAETE was the acronym for Development of Accreditation in Engineering Training and Education. The proposal had as partners, in Europe, the Imperial College (IC), Helsinki University of Technology (HUT) [1], Universidad Politecnica de Valencia (UPV) and Universidade do Porto (UP) and, in the USA, the University of Wisconsin (UW). The amount requested on each side of the Atlantic was about €50 thousand for a period of two years. The proposal was financed by the European Commission [2] and by FIPSE, Fund for the Improvement of Postsecondary Education [3].
The project started November 1, 2006, and the coordinators were UW (USA) and UP (EU). First steps consisted in defining the place and date of the first meeting. The meeting was fundamental to choose the project methodology, the action plan and the task distribution. The proposal was flexible enough to allow the partnership to decide most of the activities. The major decision was about which quality management tool, or tools, should be adopted to have acceptance for CEE centres on both sides of the Atlantic.
The objectives of the project were to present among the partners the experience and practices of the participating institutions in quality control and accreditation for engineering education and vocational training. Partnership then planned to analyse and to debate the models that can be used taking into account the different experiences on both sides of the Atlantic, to produce guidelines advisable for quality assurance and accreditation, to test the recommendations and to adapt the guidelines to the feedback results.
The first meeting occurred in Finland, Lapland, Jarisjervi, January 16-17, 2007. The place was in a TEK engineers and architects organization in Finland. Lodging and the logistics were chosen graciously and effectively by Dr. Markku Markkula (HUT). Place and dates were relevant for a proper workshop of the five partners to debate and to interact looking at devising a proper methodology and action plan [4].
The activities of the first meeting were crucial to define the following years of the project. During the four-day debates and conversations in indoor and outdoor settings, the main existing tools and protocols for quality management were presented and analysed. Centres of CEE have, generally, a structure and operational contexts that are different from the universities or higher education institutions to which they belong. Some have large degrees of autonomy, others operate as private foundations or organizations and others are departments or units. The use and adoption of quality management tools in education and training are difficult and follow specific rules that are not usual and proper. CEE centres therefore need proper quality management tools that allow diversity of nature and of operations in an already difficult context of education and training. A major decision was that the project would focus on quality assurance of CEE centres and would not address the individual CEE courses' quality.
Therefore, the group analysed, in terms of essence and of examples in education and training, the programs from International Organization for Standardization norms, Baldrige Performance Excellence Program, European Foundation for Quality Management, Six Sigma, Total Quality Management and Continuous Quality Improvement (PDCA). Two partners, UPV and UP, had participated in a project between the European Union and South America that worked on the application of EFQM in universities. Based on the analysis of the existing options and on the existing example, the partnership decided to adopt the EFQM to be tested and validated during the two-year project. Figure 1 is a photo of the group that conceived the original strategy and steps to be used in DAETE. The innovative strategies of DAETE consisted in addressing the different approaches in the European Union (EU) and USA about quality assurance of CEE allowing exchange of experiences and results of case studies. This tactic allowed a reciprocal understanding of the benefits of the methods that can be used by the engineering educational and training community in the EU and USA. The added value of the proposed transatlantic cooperation can be measured under two perspectives. The first perspective is related to sharing the experiences of the partners of the EU and USA concerning the quality assessment of education and training in CEE. In effect, the environments that framed the progress of this area have been different in these two regions. In Europe the research and development about quality evaluation of CEE has been based on joint projects supported by the EU funding. In the USA the quality assessment of CEE centres has been developed using market analysis indexes.
EFQM was considered by the partnership as a practical tool to help organizations achieve high levels of quality by measuring where they are in order to improve to achieve excellence. The model is based on nine criteria that serve as a basis to guarantee an organization's search for excellence. The criteria include five chapters on processes and four chapters on results. The first group deals with what the organization does and the last group of four analyses the results that an organization achieves.
The model emphasizes that leadership and strategy success are achieved through proper combinations of people and of resources. The EFQM criteria determine the level of satisfaction of the organization's users, the satisfaction of the people working in the organization and the impact on the organization's society. The EFQM model proposes self-assessment as a method of complete, systematic and regular analysis of the organization's activities and proposes that these results be compared with results of similar organizations within the EFQM model of excellence.
The fundamental concepts of excellence of the EFQM model are the underlying principles that constitute the essential sustainable basis for any organization. These can be used as a basis to describe the attributes of an excellent organizational culture. The eight fundamental concepts of excellence that were considered essential for the CEE centres were: user focus, results-oriented, leadership and constant guidance, process and fact management, involvement and development of people, continuing training, innovation and improvement, development of partnerships, and social responsibility.
The EFQM model divides organizational processes into nine criteria, each with sub-criteria. In the evaluation of the CEE centres the DAETE project created several sub-criteria in each of the criterion. It was necessary to adapt the sub-criteria to the processes and procedures to the needs of the CEE centres. In this way, the processes of a CEE centre may be evaluated, developed and improved in the different functional areas. Regular monitoring, internally and externally, of these processes may improve the quality and effectiveness of each activity. The evaluation of relevant results may be used to determine success against reference values and allow space to implement improvements. Figure 2 exemplifies the major concepts and phases used in the DAETE approach. In the DAETE project the five levels of each sub-criterion were defined after long debates among the partnership. The agreed definitions of these levels were obtained from the project UNIQM [5]: Level 1: Quality depends exclusively on the individual (there is no process); Activities depend on individual initiatives and activities are not programmed globally.
Level 2: Quality is based on basic processes; responsibility for each activity is no longer individual and there is a tendency to share responsibilities across the CEE centre, with some short-term planning; there is some degree of process definition, however there is no documentation; performance is assessed occasionally.
Level 3: There is vision through processes and some quality assurance (intermediate processes); there are established standards, procedures and guidelines known throughout the CEE centre; activities are carried out in accordance with these procedures; activities are planned with medium-term objectives and indicators are defined for evaluation.
Level 4: There is systematic evaluation and process improvement (sophisticated processes); the established procedures are systematically evaluated to create possible improvements; there is a clearly visible orientation for the CEE centre's user; the activities are planned with well-defined objectives, in the medium and long term.
Level 5: There is the objective of having recognized external excellence (processes of excellence); there is an exchange of knowledge and experience throughout the organization, within the organization and with entities outside the organization (including competitors); the formulation and improvement of the CEE centre's procedures are in accordance with internal and external standards; the experiences and best practices are shared with other entities; there are partnerships and exchanges of information with users, trainers and other centres, etc.
In terms of agreement among the partners about the points for each sub-criteria, several discussions and debates occurred due to the different opinions about the relative value of each sub-criterion within the respective criterion that has a fixed value of points. The agreed procedure was to take the average value from the opinion of each partner. Differences were significant due to the diversity of the CEE centres and to the managerial experience of each partner. It is interesting to note that when tasked to define proposals for the sub-criteria, the European partners chose the first five and the North-American chose the last four. The debate and final agreement occurred during the several meetings. It is also relevant that just in the first year there were six meetings, formal and informal, and in the second year there were four meetings. That reflected the interest and importance that DAETE created in the partners that took every chance to discuss the progress and the agreement about subcriteria and respective relevance for the points distribution. The partnership defined the CEE centre's nine criteria of EFQM model as: 1. Leadership: CEE centre's leaders develop and facilitate the fulfilment of the mission and vision of the continuing education centre. They develop organizational values and systems necessary for sustainable success and implement these through actions and behaviours. During periods of change, they maintain a constancy of purpose but whenever necessary, leaders are able to change the direction of the organization and inspire other members of the organization. 2. Policy and strategy: CEE centres implement the mission and vision, developing a strategy focused on stakeholders and that takes into account the external needs and those of the sector in which it operates. Policies, plans, objectives and processes are developed. 3. People: CEE centres manage, develop and unleash the potential of the organization's people at the individual level, based on teamwork and the organization. They promote equity and equality, involve and empower people in the organization. CEE centres reward and recognize the people to motivate them to use the skills and knowledge to the benefit of the organization. 4. Partnerships and resources: CEE centres intend to manage external partnerships and with internal resources to support the policy and strategy in order to have an efficient functioning of the processes. During planning the management of resources and partnerships is done to balance the current and future needs of the organization. 5. Processes: CEE centres design, manage and improve processes in order to fully satisfy and generate more and more value for users and other interested parties. 6. User results: CEE centres thoroughly measure learning results in order to have good results. 7. Results of people in the organization: CEE centres thoroughly measure employee results. 8. Results related to society: CEE centres thoroughly measure the results that concern society. 9. Performance results: Excellent organizations thoroughly measure the key results of the policy and strategy adopted.

Testing and Validation
The DAETE project partnership created a matrix that comprised the information for each criterion and sub-criterion in the columns and each of the respective five levels in the rows. Each cell was adapted to the agreed CEE activities considered relevant and specific. That matrix allowed the CEE centres to register the selfevaluation and obtain a total number from 0 to 1000 points according to the performance perceived by the CEE centre. This self-assessment matrix for CEE centres addressed the enablers and the outputs of its activities.
The matrix is available for downloading in English and in Chinese at the DAETE website [7]. The matrix can be used as a diagnostic tool for self-diagnosis and as a tool for continuing improvement. The matrix needs to be used by staff of the CEE centre used by the leaders of the institution, centre or department. In some cases, it may be appropriate for the institution to extend participation in self-assessment to other elements in order to address all aspects of self-assessment. When using the matrix as a self-assessment tool it has to be considered that there are different types of CEE centres with a wide variety of characteristics such as diversity of work areas, size, organization, financing, legal status, private or public, non-profit, commercial, etc.
Involving the whole CEE centre in the self-assessment exercise can bring up insights that one might not get without. However, the self-assessment itself can be a very heavy process, when involving everybody. Often it is very hard to find time to get everyone to participate in discussion. A disadvantage of self-assessment may happen if the work to assess and consequently to improve had been done 'alone' inside the CEE centre, with only occasional attempts to benchmark and to learn from best practices. Also, the effort to define a common framework with similar CEE centres may be a good opportunity for the CEE centre to move towards more systematic improvement of its activities. The active participation in the development of the model and selfassessment matrix gave a good background to really assess CEE centre's activities and discuss about the reasons and evidences on the scored levels. The self-assessment may be carried out by the CEE centre management group with an advantage that the group may be aware of the enablers and can find most of the evidence to the results. Then the discussion on differing opinions on the levels for some criteria may be fruitful and, in most cases, the final result may derive from a consensus instead of the average of different opinions.
DAETE proposed a path to excellence of CEE centres based on continuous improvement, self-assessment, good management practices and a planning discipline. The following principles and phases were considered and proposed to the CEE centres that participated in the testing and validation of EFQM model: 1. Assess where the CEE centre is now: One way to do this is to self-assess the CEE centre. The self-assessment process can help the centre to understand the current status in terms of quality. 2. Define the priorities of the activities of the centre: To align the activities and CEE centre strategy, it is necessary to understand the existing trends and areas for improvement. The fundamental concepts of excellence can be used to compare the centre strategy. 3. Identify what needs to be improved: Self-assessment can help provide a detailed map for people in the CEE centre. It can help to answer: "Where do we need to improve?" 4. Identify how to improve: CEE centre staff and management can learn from other centres through comparison ("benchmarking"). It can help to identify good practices from other centres. These comparisons can include processes, organizations or indicators to develop a performance evaluation strategy that may direct improvement efforts.
As a result of the dissemination effort, 122 CEE centres, academic and commercial, around the world used the DAETE model after the dissemination and training events. The CEE centers of universities and of companies that participated in the several workshops were from the USA (38), China (35), France (16), United Kingdom (15), Italy (8) and the rest were from other European countries (10). Results in terms of EFQM points were gathered and conclusions were drawn. The results were of a quantitative and qualitative nature. The calculation of the points was done during the workshops by the representatives or obtained on a later stage after internal debate locally at the CEE centre. During the seminars results were presented and discussed especially about validity for comparison among the different centres. In terms of quantitative analysis the EFQM average score for the centers was 443 out of 1000 with a standard deviation of 104. This pair showed the diversity of management modes and the uncertainty of the interpretation of the DAETE approach by each center. Examples for Enablers of three universities obtained from workshops and working sessions are presented in Table 1 of records obtained. The main consequence is that training is needed for the CEE centers about quality management and the convenience of external auditing to benchmark results and the self-assessment. Qualitative feedback was mainly obtained through two documents. One document obtained from the conclusions was a description of the CEE centre that provided information about the dimension, type of operations, management structure, budgets, number of participants, etc. This document allowed proper comparison of results among CEE centres that were similar. The second qualitative document was denominated as good examples. The information provided about the examples was filled by each CEE centre with the intent of explaining why level 4 or 5 had been awarded to a sub-criterion. This group of examples permitted that other CEE centres had access to good practices, to understand the procedures that led to the high score and learn from other centres' activities. These documents and related data were compiled and were distributed among the participants. The data related with these efforts to apply EFQM in CEE centres is available from IACEE where other developments have taken place for registered users after the end of the project DAETE [7].

Conclusions and Recommendations
The main conclusion is that the DAETE project created a self-evaluation matrix based on the EFQM model that can be used to improve the quality of CEE centres. The validation phase provided sufficient evidence of the robustness, flexibility and effectiveness of the tool. Posterior experiences also provided the conclusion that the approach is also applicable in education and training organizations dealing with other knowledge areas and levels of qualification. Sectors like vocational training, elementary and secondary schools, professional organizations and government agencies have adopted the model as a tool to enhance the quality provision. This is particularly relevant for the sector of education and training where quality evaluations are based mostly on processes. EFQM model, based on the DAETE project, is a good combination of quality assessment mixing procedures and results.
Examples of the application of the DAETE approach comprehend several applications. One example is the adoption of a professional engineering association [8] to externally accredit CEE centres that pay for each accreditation. Another example is the use of a derived model of the DAETE matrix by a CEE world organization as service to paying members [7]. The third example is the use by CACEE to train CEE managers from the centres to improve the quality of results [9]. It is relevant that CACEE deals with about 68 million professional and technical personnel nationwide in terms of CEE. Of course, these examples illustrate different uses of the DAETE matrix but show the relevance and impact of the approach. Many CEE centres have also expressed the appreciation of having a tool that allows a selfevaluation tool that fosters reflection, diagnosis, action plan and improvement.
From the several years of following the use of the model several recommendations emerge. The first one deals with the fact that the sub-criterion are context dependent and may need adaptation. For instance, the annual income may be irrelevant for CEE centres where the funding is public and strategy relies on society interest and not on the market rule. Another recommendation is that users of the DAETE model should be trained on the EFQM principles. The danger of using it as an accreditation tool by unprepared auditors may lead to a false evaluation. Another recommendation is that providing a service to CEE centres without proper mentoring and interaction may lead to lack of interest by potential users. A final recommendation is that creating a community of CEE centres sharing the results of the application of the tool depends on anonymous sharing and proper communication. Finally, the DAETE matrix, based on EFQM model, maybe be a proper path to excellence in education and training but needs to be properly used and interpreted.

Acknowledgments
Appreciation goes to all that participated that were truly involved and interested in the cooperation across the Atlantic, and consequently the world, in improving quality of CEE and in promoting better education and training. The special recognition is dedicated to the late Prof. John Klus that was a reference in CEE, in general, but an effective leader and expert in the DAETE project. He was a person who did not appreciate those who did not contribute and, at the same time, did not allow others to contribute. For him barriers in the project were to be surpassed and his training as a civil engineer probably played an important role in that behavior. That leadership and wisdom were particularly relevant while dealing with two different academic and professional cultures and with the idiosyncrasies of the partners linked with the respective CEE contexts. He was fundamental to the conclusion of the project and to the successive steps that led to the dissemination and experimentation of the model [10]. The European Commission and FIPSE also deserve appreciation for a project that was innovative and useful for quality of CEE and of education and training.