Paper—Evaluation Results of an Online Teacher Training Course Specialized in Engineering Education Evaluation Results of an Online Teacher Training Course Specialized in Engineering Education

The purpose of this article is to present the results of the evaluation of the implementation of a teaching training course in Engineering Pedagogy (EP) at a Chilean university of applied sciences. The research questions that guided the research process were: (i) How do the participants evaluate the course in general? (ii) How do the participants evaluate the didactic design of the course? (iii) How do they evaluate the teaching competencies? (iv) How do they value the usefulness of the learning outcomes for their teaching practice? (v) How they self-evaluate their participation and their learning process? Based on different authors a questionnaire with closed and open-ended questions was developed and implemented online. For the statistical analysis was applied an exploratory-descriptive analysis. The training course consists of two online modules with 90 working hours in LMS, and was designed by the Center of Engineering Education (CIEI) at the University of Talca, according to the IGIP Curricula of the IGIP center at the Technische Universität Dresden (TU Dresden, Germany). From the first results of the pilot project, it can be noted that there is a high level of motivation and interest to participate in a teaching training course based on EP, which has been specially oriented and designed to meet the specific requirements of the academic staff of engineering schools. Keywords—university teacher training, engineering pedagogy, online teaching program, online learning in engineering


Engineering pedagogy and education in Chilean context
Although Engineering Pedagogy (EP) has a long tradition in German-speaking are as (particularly in Dresden, with a tradition of more than 60 years) [1,2,3], the concept of "EP" is relatively recent in Spanish-speaking countries. In the specific case of Chile, the application of this concept has been made possible thanks to the projects: "Engineering Pedagogy in Chilean Universities 2014-2018" (PEDING Project) and "Strengthening engineering training at Chilean universities through practice partner-ships" (STING Project) [4,5]. Several authors recognize the different factors and aspects that condition the engineering pedagogy and education and the training of the teaching staff in engineering careers [6,7,8,9,19,20,21], but the societies of the 21st century evidences diverse demands derived for example from different actors of the society, the economy and the scientific engineering fields, among others. Gormaz-Lobos et al. [10] shows multiple factors that condition and influence the EP: (i) the production and service structures of each country (needs determined by the labour market), (ii) the society and culture (the needs for the development of technology and technique is also determined by cultural and social aspects), (iii) the engineering sciences (needs determined by matters of research, methods and technologies at the engineering subjects), (iv) the academic staff or university teachers (with specific competencies of the engineering fields but also of with pedagogical competencies), and (v) the learners (students) or individuals studying engineering (with age-specific psychological characteristics, individual needs, characteristics of the personality, values, attitudes among others). Specifically, Melezinek defines EP as a discipline oriented to the engineering education process which connect the engineering sciences and techniques with pedagogy and the educational system [6]. The author describes various component elements for EP such as: (i) engineering fields (sciences) themselves, (ii) natural sciences, (iii) psychological and sociological approaches, (iv) educational sciences, (v) communication sciences, (vi) information and communication technologies (ICT), and (vii) ethics, among others [6]. Based on different authors, Figure 1 presents the interaction of some fields that should be considered for the development of a conception of engineering pedagogy and education [1,2,3,4,6,7,22,23].

Proposal for a training course based on Engineering Pedagogy
In 2019, the International Center for Engineering Education CIEI (University of Talca, Chile) developed a proposal for a teacher training program in EP based on the IGIP Curriculum offered at the IGIP center at the TU Dresden and the results of the PEDING and STING projects (see Figure 2) Figure 2 shows a general overview with the structure of the training program (curricula). According to IGIP, the training program contemplates a total of 600 hours for all learning activities, including lectures, presentations, workshops, short reports, and group activities, among others. The program will be carried out through an Online Learning and also Blended Learning modality: including classroom sessions (face to face or videoconferences) and remote learning through a digital learning platform (LMS). The teaching-learning program seeks to expand the field of Engineering Pedagogy and Education in Chilean context and also to improve the competencies profile for university and non-university teaching staff (teachers and trainers) in engineering faculties and vocational education institutions.
The main educational objectives of the training program in Engineering Pedagogy and Education are [24]: 1. To development and improvement of pedagogical competencies of the academic staff (teachers and trainers) in engineering education and similar fields, 2. To improvement of teaching and learning (planning, resource development, among others) and evaluation methods in scientifically and technical subjects, 3. To development of practice-oriented curricula that correspond to the needs of students and employers, and strengthening the university-industry collaboration, 4. To development and strengthening of innovation and research competencies, 5. To facilitate the acquisition of theoretical-practical knowledge about curriculum design and development, 6. To promote the use of media in university and technical teaching, and 7. To development of communication competencies and strategies for online (synchronous and asynchronous learning) and face to face learning.

The first teacher training in EP at INACAP: a pilot project
INACAP (Universidad Tecnológica de Chile/ Instituto Profesional/ Centro de Formación Técnica) is a Chilean vocational school (post-secondary education) and university of applied sciences, with campuses (more than 27) in all regions of Chile. The main programs for undergraduate students are related to health and engineering fields. The teacher training programs for the academic staff are frequently not strongly linked to their pedagogical practice in the scientific and technological fields [11,23]. For this reason, the authorities from INACAP decided to specialize the teacher training, and generate a cooperation strategy with the CIEI at the University of Talca (Chile), under the pedagogical support of a group of academics of the TU Dresden (Germany), Faculty of Education. The activities started with a pilot project to strengthen specifically engineering teaching and technical and professional education, through an online program for teachers in different fields of EP. But before starting the course, the research group decided, as the first step, to recognize and analyze the training needs of the academic staff with a pilot group: a group of academics in engineering fields at INACAP at the Talca campus.
Based on the results obtained in the surveys about multiple needs on Engineering Education and Pedagogy were identified [11]: a) "Evaluation and assessment of learning achievements", b) "Knowledge about theoretical and practical approaches of the didactics for the teaching and learning in engineering", c) "Structuring of teaching and learning processes for engineering education", d) "Use of learning resources and information and communication technologies" (ICT), e) "Use, development and evaluation of new didactic means in the training of engineers", and f) "Knowledge about how to design effective measurements of the learning accomplishments" among others.
From these results, was developed a training course that consists of two modules of the training course in Engineering Education (training program) designed at CIEI (see Figure 2), with specific contents [11,22,23,24,25,26,27], according to INACAP teacher training needs. An overview of the modules, goals, and units is presented in Table 1. Table 1. Description of the teaching training course modules for INACAP-Pilot Project.

M1 Teaching and Learning Process Design in the Engineering Education (2 CP) Goal Units
Participants will be able to structure teaching and learning processes to train engineers, taking into consideration the specific scientific knowledge of the discipline they teach, focusing on the learning process of the students.
Unit 1 -Trends, future perspectives, and approaches for engineering education. Some new national and international trends, perspectives, and approaches that support the context of Engineering Education and the formation of Engineers of the 21st Century. Unit 2 -Basic principles of Engineering Teaching. Fundamental principles, theoretical foundations and terminology associated with Engineering teaching and learning. Unit 3 -Structuring of the teaching and learning processes in Engineering Sciences.
Components of the teaching and learning process to meet the requirements of the professional profile of students graduated from Engineering. Unit 4 -Fundamental principles for the elaboration of didactic Media. Fundamental principles, approaches and concepts for the elaboration and design of didactic Media.

Learning Assessment in Engineering Education (2 CP) Goal Units
Participants will be able to design control and evaluation processes of the learning results based on scientific and theoretical foundations.
Unit 1 -Theoretical fundamentals and approaches about the learning assessment at the university level.
Theoretical fundamentals and approaches that support the functions of the assessment of learning outcomes at the university level, applying these fundamentals and approaches by the structuring of evaluation processes. Unit 2 -Operationalization of the learning objectives and outcomes. Components that characterize a quality evaluative process and the different areas that need to be evaluated, considering these to operationalize the learning objectives and the expected learning outcomes of the learning processes, to make them evaluable according to the learning objectives. Unit 3 -Evaluation methodologies and strategies, and register of learning outcomes. Different methodologies, strategies, and procedures to design evaluation processes and registers of the learning outcomes, analysing their limitations and potentialities, for developing and selecting an appropriate method according to the object or item to evaluate. Unit 4 -Evaluation of learning outcomes and key competencies of engineers.
To know and identify different forms, strategies, and evaluation systems and register of the learning outcomes, developing and selecting an appropriate procedure, according to the professional profile for the training of engineers, based on its expected knowledge, key competencies, and skills.
The course was offered in online learning form (with (synchronous and asynchronous activities) between January and May 2020 with the participation of a group of 24 academics of INACAP Talca campus. Most of the academics are part of four Engineering Schools of INACAP at the campus Talca: Electrical Engineering, Industrial Engineering, Information and Computer Engineering and Mechanical Engineering. Three participants had no teaching responsibilities: two are supervisors and one is department director of this campus. The course contemplated a work of 4 Credit Points (CP) (each Module 2 CP), also 120 hours of training according to the SCT-Chile system. The course design was based on a mix of synchronous activities carried out on the Zoom platform (30 hours), together with work on a Moodle-based LMS platform for asynchronous independently and collaborative work (90 hours). The course evaluation was formative and based on the independent work materials. At the end of the course, was requested a final product (with planning, development of learning and evaluation material plus rubrics, among others), that allowed to evaluate the competencies by each participant along the course.
As an evaluation form of the project, the research group implemented a survey to obtain feedback about the course from the participants, its design, topics, and effectiveness among others. Although different authors [9,21,23] show the need for training in technical and engineering-related disciplines, research on the effects and evaluation of university teaching training programs has not been widely reported in the specialized literature for the Chilean context. For this reason, the authors of this paper would like to know the valuation and opinions of the teaching staff who participated in this course.

2
Evaluation results of the teacher training course in EP

Methodology
The main goal of the study was to identify the perception of the participants of the training course (academic staff) about their experiences and valuation of the competencies development during the course. The research questions that guided the research process were: 1. How do the participants evaluate the course in general? 2. How do the participants evaluate the didactic design of the course? 3. How do they evaluate the teaching competencies? 4. How do they value the usefulness of the acquired knowledge for their teaching practice? 5. How they self-evaluate their participation and their learning process?
Based on [12,13,14,15,21] and their previous experience in engineering pedagogy research projects in Germany and Chile [5,16], the authors developed a categories system with indicators for the instrument design. The instrument consists in a questionnaire with closed and open-ended questions organized in five main categories with their respective items. The five categories or dimensions to be evaluated are (see Table 2): 1. The development of the course (in general), with 4 items 2. The didactic design of the course, with 9 items 3. The teaching competencies, with 5 items 4. The perceived utility of the acquired knowledge for their own teaching practice, with 12 items 5. Self-assessment of the own learning process, with 4 items Due to the location of the participants (Chile), the questionnaire was developed in Spanish and consisted of 34 items on a Likert-type scale (5 levels) grouped into five categories (see Table 2). Table 2. Dimensions and conceptual items for the evaluation instrument [5,16] Dimensions and conceptual items (Summarized version for this publication)

Categories / Dimensions
Conceptual Items

Population, available sample and procedure.
The sample of the research was composed of 24 participants at the training course for INACAP at the Talca campus. Only 21 questionnaires were considered for the analysis because they were fully completed (three questionnaires were incomplete). The instrument was applied online, ensuring the anonymity of the participants. The first part collected information of the participants about gender, subject matter, fields and years of teaching experience, previous teacher training, among others. The second part consists in the information collection of the closed questions. For the statistical analysis was applied an exploratory-descriptive analyze [17,18].

2.3
Characterization of the sample Table 3 presents the characterization of the sample from INACAP. In total, 21 academics answered fully the questionnaire: 29% are women (6) and 71% men (15). Regarding the age ranges of the respondents, 90% (19) of the participants are between 30-39 years old and 10% (2) are between 40-49 years old. There no participants older than 50 years. Related to the participant´s distribution by engineering school, most of the participants work in mechanical engineering (38%) and industrial engineering (24%). The same number of participants works in the fields of computer sciences (18%) and electrical engineering (18%). Concerning the years of teaching practice experience, 81% have between 0-5 years (17) and 14% have between 6-10 years (3). Of the total number of respondents, approximately 86% (18) have already participated in some teaching training course.
Closed questions. The results about the perception of the respondents regarding the evaluation of the training course are presented in this section. Table 4 exposes the results for the 34 considered items (see Table 2).

Low
Medium High

[1][2] [3] [4][5]
1 In general, all aspects were as high (4 points) or very high (5 points) valuated (more than 75% of the preferences). The most relevant aspects are related to the dimensions "Evaluation of the didactic design of the course", "Evaluation of teaching competencies" and "Evaluation of the perceived utility of the acquired knowledge for the own teaching practice" (see Table 2) with more than 85% of the preferences. Specifically, 22 indicators of these categories were valuated with 100% of the preferences as high (4 points) or very high (5 points). An example for this are the items: "Satisfaction with synchronous learning activities", "Usefulness of workshops for learning processes and competencies development", "Clarity of teachers to guide the teaching-learning process", "Satisfaction level with the learning resources and tools used for competencies development" and "Utility of the acquired learning for professional development". The item with the lowest relevance is related to the category "Self-assessment of the own learning process" (under 80%), and was: "Commitment and motivation with the course". The item "Dissatisfaction with the participation in the course" was valuated as very low or low with 100% of the preferences.
Another important aspect is the perception of the participants about the relevance of the different items /questions by gender. Figure 3 shows the differences between the participants related to the relevance of the indicators. For the female participants of the teacher training course, most of the items were low valued, except the items "Satisfaction with the development of the course" and "Satisfaction with the duration of the course". By grouping the participants by engineering fields (Figure 4), the participants from Computer sciences gave the high average of preferences in the valuation of all items (96,76%) and the participants from Mechanical engineering the lowest average with 89,49%. (Industrial engineering with 91,06% and Electrical engineering with 90,59%). Figure 5 shows the differences between the relevance of the items for the participants by years of teaching experience. For the participants with up to 5 years of teaching experience, the average value of preferences in all items was 92,11% and for participants with up to 10 years was 92,35%. For participants with more than 10 years of teaching experience, the most of the "indicators" were low valued (average of 77,65% in all items). Computer Sciences Industrial Eng.
Electrical Eng.

Discussion and conclusion
The main goal of the study was to identify the perception of the participants of the training course (academic staff) about their experiences and valuation of the competencies developed during an online course. Concerning the categories "Evaluation of the course development (in general)", "Evaluation of the didactic design of the course", "Evaluation of the teaching competencies" and "Evaluation of the teaching competencies" -in line with other studies [15,16,28] -all aspects related were as high valuated. The item "Dissatisfaction with the participation in the course" was valuated as low with 100% of the preferences. These results coincide with the comments of the participants during the teaching-learning activities. The findings confirm the relationship between the average of evaluations with the years of teaching experience of the participants [16,28]: participants with up to 5 years of experience evaluated the course better, and participants with over 10 years of teaching experience tend to evaluate the course lower; something that had already been demonstrated in the literature [16]. The item with the lowest relevance ("Commitment and motivation with the course") is related to the category "Self-assessment of the own learning process". That can be explained because most of the participants had a high amount of teaching time and said that they wanted to dedicate more hours to the program but could not because of their other responsibilities.
This pilot research project and training program was aimed to show the different training needs in Engineering Education and Pedagogy, and to implement and evaluate an online training course for the teaching staff of engineering schools of INACAP at the Talca campus. The project directors recognize the challenge and the hard work involved in specializing the development of competencies of academic staff in the field of EP because this is relatively "new" in Spanish-speaking countries and also in Chile. For this reason, an international university working group has been formed to promote the EP and the specialization of the teaching profile of academics of engineering fields. This group is formed by academics from the Technische Universität Dresden (Germany), the University of Talca, and INACAP (Chile). From the first results of the pilot project, it can be noted that there is a high level of motivation and interest to participate in a teaching training program in EP, which has been specially oriented and designed to meet the requirements of the specific academic staff of engineering schools. The evaluation results of the training course and its online implementation showed a high valuation of the course by the participants. However, due to the "pilot nature" of this project and the small sample of academics who participated (24 people), some questions remain open about the impact and valuation that this type of training course can have on other INACAP campuses throughout Chile. Due to the size of the sample, the authors consider that the results of the research cannot be generalized. However, they serve to analyze (case study) the teachers' experience about a training course specifically oriented towards EP. For this reason and as future work, the authorities of the institution and the project directors want to design a complete training plan for 2021 that includes other modules of the CIEI program that may be relevant for the teaching staff from other campuses. This program will be evaluated by implementing quantitative and qualitative methods, to incorporate consistency and data triangulation into the research findings. To contribute to the development of the concept and the effects of EP in engineering schools, the authors of this paper hope to continue expanding the teaching training in Engineering Education in universities and vocational institutions in different regions of Chile.