A Human Resource Development Program for Information Technology Engineers using Project-Based Learning

—This paper reports the case of an educational practice and an evaluation of team learning using project-based learning, in virtual learning environment to develop the competences, such as their communication skills, problem solving within teams, and others, of Information Technology engineers. As a result of learning evaluation for participants, the feasibility of the program has been determined. All functions designed for the learning management system were used during the practical experiment


I. INTRODUCTION
The development of engineers in various disciplines requires human resources training in vocational and higher education systems, and the improvement of these systems [1]. The development of educational programs and evaluation methodologies have been regularly discussed and disseminated [2], resulting in many revisions to these programs.
Also in the area of information technology (IT), human resource development issues such as the number of engineers needed and the quality of engineering performance [3] have been discussed. In particular, the IT industrial sector claims that information technology engineers should have systems development experience as members of a team which has practiced resolving problems while they were learning fundamental engineering theory. Additionally, it has often been suggested that IT engineering graduates from departments of computer science have insufficient communication skills, leadership qualities and project management experience [3].
Since most IT engineers are trained at computer colleges, these colleges have to develop educational program to meet the needs of students and industry. Most college students in Japan are around 20 years old, and have little experience as engineers. Also, IT engineers have to learn business manners in order to better communicate with customers and business partners. One approach is to employ team learning as a form of project based learning (PBL).
Team learning means that participants work together as a project team, and resolve problems collectively [4], [5]. These teams require collaboration, and members have to play individual roles, which are assigned in advance [6]. As team learning is based on group work in a team, these activities are similar to group learning or collaborative learning [7], [8]. Team learning may emphasize discussions, learning through experience, and teaching each other. Studies have reported that team learning can provide training, which improves academic achievement and human performance, two measures of work related skills [9]. Therefore, it can also be applied to various areas of training for engineers [10].
Currently, the online learning environment is being used to support engineers in the work place, and it can provide a kind of virtual learning environment (VLE). These two can play a role in supporting learning, even for PBL [11]. However, enhanced computer supported collaborative learning or work (CSCL/W) environments cannot always promote team learning as a form of PBL, because some of the human factors of the teams are so important. Therefore, careful design and support will be important if team learning is to be effective.
In this paper, the feasibility of developing a team learning style for practical learning by IT engineers [12] is examined, and also the interaction between team learning activities and functions of learning management system (LMS) have been discussed. Team learning sometimes influences individual learning [13], so a key point is whether it is feasible to measure the effectiveness of positive learning. Another hypothesis is that some functions of learning management systems may contribute to learning, such as the team-learning requirement for discussion between participating members and the recording of their discussions.
In considering the merits of PBL and VLE, this paper will address the following topics:  The design and develop of an educational program to provide practical job experience in IT systems development using face to face and online learning, including communication skills training in response to customer's expectations. These are taught as a subject called the "Upper Process". PAPER A HUMAN RESOURCE DEVELOPMENT PROGRAM FOR INFORMATION TECHNOLOGY ENGINEERS USING PROJECT-BASED LEARNING  The evaluation of learning performance during the course, by developing and using evaluation items to assess participant's performance.  The enhancement of team learning, by developing a learning management system. An evaluation of this system by participants was conducted, and the relationship between learning performance and system evaluation was analyzed.
II. RELATED WORK PBL is the preferred method of introducing stimulating learning activities at schools and institutes of higher education. The main purposes of learning using PBL are not limited to the acquisition of knowledge and problem solving methodologies. Flexible thinking and investigative skills may be acquired during PBL. Some courses introduce PBL in order to target these experiences. Even in VLE, PBL activity is expected to bring additional learning performance, compared to ordinary course lessons.
The reason why these types of learning activities are favored may be that they enable students to attain the required level of performance, which leads to employability at an actual company.
Here, "employability" is different from the usual meaning of the term. Instead, it is defined as "a set of achievements, understanding and personal attributes that make individuals more likely to gain employment and be successful in their chosen occupations" [14]. IT engineers currently in demand must have the flexibility to perform their jobs beyond the specific level of knowledge mentioned in the introduction. These abilities, sometimes called employability, are not only acquired knowledge and skills. Employability education programs have been introduced at some higher education institutes to teach students these skills [14].
They are also defined by the OECD-DeSeCo project as follows [15]. There are three broad categories of abilities necessary: interactive tools use, interaction in heterogeneous groups, and autonomous action. The first two categories are concerned with both learning and training.  Competency category 1: Interactive tool use 1) Use of language, symbols and texts interactively 2) Use of knowledge and information interactively 3) Use of technology interactively  Competency category 2: Interaction in Heterogeneous Groups 1) Relate to others well 2) Co-operate, work in teams 3) Manage and resolve conflicts The first category consists of basic knowledge and skills, including specific technical issues such as computer programming and system development. Most of these are taught at school.
The current IT industrial sector claims, which are mentioned in the introduction, may suggest an insufficiency in competency category 2. The demand for these abilities coincides with the needs for employability education at institutes of higher education. To achieve these competencies, PBL is frequently employed in engineering education [16], [17]. Achieving competency  The uploading of report documents Team products Database of the team's final products Self evaluation Online questionnaire for self evaluation beyond knowledge and skills is often an implicit goal of education and training. Currently, virtual learning environment (VLE) is being used throughout engineering education, and PBL is also being used with the existing information technology platform [11]. In particular, as IT engineers are required to operate within a virtual environment, PBL using VLE should be promoted vigorously.
This paper describes a feasibility study for developing competency category 2 of IT engineers using VLE throughout PBL.

A. Learning content
The "Upper Process" of IT systems development and consulting was selected as a subject to determine the feasibility of team learning using project-based learning. It was studied in team using project-based learning. The project is the detailed analysis of "requests for proposals" and the proposing of "requirement definitions" while PAPER A HUMAN RESOURCE DEVELOPMENT PROGRAM FOR INFORMATION TECHNOLOGY ENGINEERS USING PROJECT-BASED LEARNING participants learned communication with customer skills, business manners, and problem solving methods involving teamwork. This course was originally designed as on-thejob training. To determine its feasibility, a 15 session course with modified content and evaluation criteria was introduced at a computer college where students develop information technology skills. One lecturer and one mentor managed the 40 participants, which were divided into 5 teams of 8 members each.
The problem assigned as a project involved the following two tasks.
1) The first task Analysis of "requests for proposals": The team members pointed out which questions should be asked to better understand the needs and recording detailed explanations while they talked with someone in the role of customer. 6 sessions × 90 minutes each.
2) The second task Preparation of a "requirement definition": The team members summarized the proposed system requirements after interviewing someone in the role of customer and then analyzing and understanding the request as a team. 6 sessions × 90 minutes.
These two tasks are independent of each other. Every team's activity was evaluated as follows:  Analysis of documentation and interviews with customers  Analysis of the session minutes, which recorded team activity  Individual session reports  Individual self-assessments of team learning  Assessments of proposals resulting from team collaboration Additionally, the first session was a course orientation, the 8th session was an intermediate discussion and the 15th session was used for overall reflection and assessment. Again, the tasks consisted of project based learning, and teams were asked to participate in the learning mention above. This design was intended to enhance team working toward PBL.

B. Learning environment
All participants in this course used their notebook PCs, which were connected together using a learning management system. A diagram of the system is illustrated in Figure 1. The system, which was developed using the Moodle system, provided learning materials and recorded the learning process, the session minutes and individual reports. The main functions of the system are summarized in Table I.
The role of each member was self-assigned and the team working sessions were also conducted at each team's own pace in a normal classroom. The role of the customer was assigned to a lecturer, so that the lecturer could introduce various business skills, such as conventional communication formats during interviews. Participants summarized the their work and reported their meeting minutes. They could discuss unresolved points using online forums after the face-to-face team working sessions. As a VLE, this environment promotes communications and the sharing of resources, such as  documents and other materials, between members, and also between teams and teaching staff.

C. Evaluation methodology
The evaluation criteria of the two tasks were different because the objectives were different. The evaluations standard for participant activities were designed in advance as a rubric [2]. The following two types of team activities were rated using a 4-point scale (1=the worst, 4=the best). 1) Evaluation of the team's products The number of evaluation items was 9 for the first task and 11 for the second task. Three of these were common for both evaluations: (1) Quality of behavioral and lang-  This function was easy to use to review products of other teams Schedule management 7 The schedule management function was useful 8 This function was useful to manage the team schedule Overall evaluation 9 I would like to use this system frequently 10 I found this system unnecessary complex (reverse scoring) 11 This system was easy to use 12 This system provided many functions 13 Most participant would learn to use this system very quickly uage presentation (2) Performance and understanding of presentations (3) Progress of work as a team, individual session reports and the meeting minutes.
2) Participant self-evaluation Individual perception of learning performance is a very important aspect of the evaluation of the program. The summative evaluation methodology for the competency, which is targeted in this program, is not easy to define, so that participant's responses are key pieces of information.
Participants were asked to evaluate their learning activities themselves twice, using questionnaires. The first questionnaire contained 10 questions and the second questionnaire contained 12.

D. Evaluation of learning environment
Participants have to use the functions of the LMS, which are mentioned in Table 1 above, since all functions are required to perform the two tasks. The usability of the learning environment may affect the effectiveness of team learning and self-evaluation; so five aspects of the system are evaluated in 13 of the questions.
The five system aspects are: 1) Team discussion forums 2) Individual session reports 3) Team product uploads 4) Schedule management 5) Overall evaluation Though participants assessed their own grades, the responses were scored using a 4-point scale (1=the worst, 4=the best). The questionnaires were given to 40 participants after the completion of the course.

A. Learning Evaluation
The lecturer rated the products of the 5 teams using evaluation items, which were designed in advance as a rubric. First, the means of three common items across the two tasks are summarized in Figure 2. All means for the first task are comparable with the median 2.5, but the ones for the second task have increased slightly. Though practical education may provide a few improvements, there are no significant differences between means for the two tasks.
Second, the means for the other items are summarized separately in Tables II and III, because their evaluation points are different. To confirm the differences across the evaluation items, interval estimation is conducted. In the results, all means are not significantly different from the median of 2.5.
Therefore, as all evaluations are located around the median rate, this suggests that all team performances are acceptable to the lecturer. Additionally, these means suggest that the lecturer does not reject the team products.
The means of self-evaluation across 10 question items were calculated for both the first and second tasks, and for the two additional questions in the second task. The results are summarized in Table IV. Means for some questions are higher than the median, such as those regarding team discussions, session reports and minute reports. Again, interval estimation was conducted for all means. Symbols are used for mean values significantly higher or lower than the median. "*" represents mean PAPER A HUMAN RESOURCE DEVELOPMENT PROGRAM FOR INFORMATION TECHNOLOGY ENGINEERS USING PROJECT-BASED LEARNING values that are significantly higher, and "#" represents mean values that are significantly lower. As the results show, some negative results from the first task improved significantly in the second task. In particular, means for questions from the second task (Nos. [10][11][12] are significantly smaller than the median, so that participants may have recognized that they have not performed these tasks sufficiently. To extract factors of self-evaluation, exploratory factor analysis was conducted for responses to common and identical questions about the two tasks. As a result, one factor model consisting of 6 items (Nos. 2-7) was extracted. The internal consistency is evaluated using a Chronbach α coefficient, where α is 0.73. Therefore, the sum of these rates can be defined as an index of selfevaluation. The mean scores for each task are calculated as 2.59 for the first task and 2.64 for the second task. There is no significant difference between scores of the two tasks, and they are also comparable with the median. Though the tasks are independent of each other, the scores are comparable, and the sums of the two scores are calculated as the self-evaluation score.

B. Effectiveness of Learning Support Systems
As there was no serious problem with the LMS during practice sessions, all functions of the LMS successfully supported the learning. Five major functions of the learning environment were evaluated using 4-point scale questionnaires. The 13 question items are listed in Table  V. Mean scores for the 5 major functions were calculated. The mean for the team discussion forum is the highest, and the means for individual session reports; team product database and overall evaluations are also high. According to the results of interval estimation, the four means for the above functions are significantly higher than the median (p < 0.05). However, the mean for the schedule management function is significantly lower than the median. The team sessions were conducted periodically, so that additional scheduling might not be required. As most means for functions are relatively high, participants have positively evaluated this system.
According to the results of the system evaluation, participants agreed that the functions of the LMS as a learning environment help their team learning activities. It is hypothesized that there are some correlational relationships between team learning performance and system evaluation. The correlation coefficients between these were calculated. The results are summarized in Figure 3. First, the coefficient for evaluation of team products and the function of the team product database is the highest, at 0.44. This means that members of teams whose presented products, which scored highly, evaluated the function of team product database positively. The system may contribute to the results of team work activities. However, as the coefficients for other functions are relatively small, their effectiveness may be small. To confirm the relationship between self evaluation and system evaluation, correlation coefficients for each function were calculated. The variation in these coefficients is illustrated as a bar graph in Figure 4. The coefficients for team discussion forums, individual session reports and overall evaluations of the system are higher than 0.4. The system performance may affect individual self evaluation.  V. DISCUSSION According to the objectives of this study, the effectiveness of PBL and team learning, and the benefits of using LMS as a VLE should be discussed.
First, a program to develop the human resources of IT engineers has been designed, and was conducted as a course at a computer college during the regular class schedule. The task of team learning and the evaluation methodology included assessing rubrics and criterion which were developed for the assessment of the program. According to the criterion, inventories such as question items for participant self evaluation were created. It was necessary for both the lecturer and the mentor to provide detailed instructions and support to promote team learning activities. Also, they were able to evaluate participant's learning activities using the rubric. As a result of the team learning approach used in the study, team product proposals reached an acceptable level of competence, and participants reflected appropriately on their learning activity. Since the program was originally developed as on-the-job training for IT engineers, students of computer college could not do the task well. Additionally, as the participants had little experience with team projects and customer relations, some hesitation amongst team members was observed. Participants learned job variation and about the job requirements for IT engineers, and they also recognized the importance and benefits of PBL and team learning. In particular, participants studied the relationship between job requirements and their level of knowledge, business manners and team management. These confirmed the possibility of using a team learning approach to IT system development education.
Also, a learning management system (LMS) was introduced to promote team learning. The system was used frequently and most functions of the system were evaluated positively. In addition, the scores of system evaluations correlated with both the evaluations of team products and with participant's self-assessments. These results provide evidence that since a learning management system can assist participants with their education, a more appropriate system may bring even better performance. Moodle, a commonly used system and a piece of public domain LMS software, was used for this experiment. However, the functions LMS provided to the learning environment should be carefully considered vis a vis the actual learning activity. The appropriate functions of the learning environment may be preferred in order to conduct team learning. This suggests that consideration of the design of the system is quite important. Of course, other types of learning platforms using information technology may contribute to this kind of learning as well. Further development of the learning performance merits consideration.
As these results are from a case study, team-learning programs for IT engineers can now be conducted at computer colleges. However, it is not clear whether these programs will contribute to achieving the level of competency for IT companies that is required. In particular, the evaluation methodology for measuring these competencies may not yet have been developed. The effectiveness of this program should be continuously evaluated. At the very least, the same practical experiment should be conducted elsewhere, to confirm the feasibility in another topic in addition to "Upper process". Also, these kinds of programs are required to develop engineers in various areas of engineering, and not only for IT. An effective procedure for the development of the program should be created. The key design factors should be extracted and analyzed for use with other courses. Incidentally, a LMS used as a VLE did not sufficiently contribute to this learning. The advantages of VLEs should be conducted again, using a distributed learning course, for example. The interaction between PBLs in teams and VLEs for developing competencies, as mentioned in the related work section should be carefully studied. These processes will be subjects of our further study.

VI. CONCLUSION
This study examined a course, which used team learning to introduce project-based learning, while participants worked with a LMS as a VLE, in order to develop IT engineers who could achieve the required competencies needed to adapt to the current requirements of the industry.
The feasibility of using the program with students of a computer college was determined. Evaluation methodologies were also developed, and they were employed throughout the program. The relationship between learning performance, including participants' evaluations of their work and their evaluations of the LMS, were measured. This experiment confirmed the utility of some contributions of the LMS.
The overall effectiveness of the program was discussed, by evaluating the results of the case study, and future research work is also presented.