Automatic Profiling of Open-Ended Survey Data on Medical Workplace Teaching

On-the-job medical training is known to be challenging due to the fast-paced environment and strong vocational profile. It relies on on-site supervisors, mainly doctors and nurses with long practical experience, who coach and teach their less experienced colleagues, such as residents and healthcare students. These supervisors receive pedagogical training to ensure that their guidance and teaching skills are constantly improved. The aim of such training is to develop participants’ patient, collegiate and student guidance skills in a multi-professional environment, and to expand their understanding of guidance as part of their work as supervisors of healthcare professionals. In this paper, we investigate open-ended answers on guidance experience of 281 healthcare supervisors that participated in these training courses. To automate the analysis of the contents of the answers, we apply clustering to the natural language processed textual data. The results summarize the most common guidance experiences and allow an automatic grouping of the healthcare supervisors’ reflection. Such an evidence-based knowledge can be used to further improve the organization of the training courses. Keywords—Healthcare vocational training, guidance interaction, Finnish natural language processing


Introduction
The emphasis on evidence-based medicine calls medical professionals to stay current in a rapidly changing work-place, for example, due to the increased role of new technology. Both the trainers and the trainees in the healthcare work environments have to learn leveraging new technologies [1,2]. Moreover, the healthcare supervisors (for example, doctors and nurses with long vocational experience) have to teach their less experienced colleagues on-the-job. This on-the-job teaching can be challenging because the trainers not only have to be experts in their job as healthcare professional but also need didactic skills. For example, when teaching a surgical trainee it is not enough to physically show the trainee how to perform a surgery. It is as important to communicate how to do accomplish the different surgical motions and small technical moves [3, 4,5]. Hence, the trainers need training to stay current in their core mission to support utilization of efficient medical practices and in their role as healthcare supervisors.
To support the healthcare supervisors in their job the healthcare district in Central Finland offers training courses to its workplace trainers. These so-called train-thetrainer courses are arranged by a multi-professional and multidisciplinary simulation training center that is the Center of Medical Education (CME). 1 the aim of these courses is to train healthcare supervisors in mentoring and counselling. More precisely, the courses aim at developing the health-care supervisors' patient, collegiate and student guidance skills. Furthermore, they intent to expand the understanding of guidance as part of their daily work. As a whole, in the train-the-trainers courses, theory and the healthcare supervisors' experiences are combined, and guidance methods are reflected and practiced together.
To systematically develop the educational arrangements and to adjust the upcoming train-the-trainers courses, all participants of these courses are asked about their former guidance experiences and guidance definitions before a course starts. Usually this is done by structured questionnaires using Likert scale. Open questions are rarely used in large scale for this purpose, because they are laborious and slow to analyze. However, regarding the course's aim to improve each individual trainer's pedagogical skills, the participants' theoretical understanding and former practical experiences should be known when the course begins. In order to gain this information open questions are better tools in addressing personal experiences and understanding than structured questionnaires [6]. Thus, the CME now utilizes open-ended questions in their pre-course surveys.
Our aim in this paper is to analyze the open-ended survey data collected by the CME in order to produce meaningful and useful insights to the participants' thoughts on guidance. However, because of the large number of participants it is too laborious to assess all feedback from the courses simply by reading, especially considering the brief time frame as the data is collected only shortly before a course starts. Moreover, the collected data is in Finish language, what makes the analysis more complicated than natural language processing of English language. Hence, with the goal to automate parts of this assessment our research questions in this study are: • Can commercially available software tools (IBM Watson, the Finnish word parser, and Matlab) be used to analyze the train-the-trainers courses open answer data in Finnish language (i.e., a rare language with unique grammar) in a more automatic manner?
• Can we find different guidance profiles or styles through clustering the answers given by the healthcare supervisors? • How many clusters can be found and what are the most common words used by the healthcare supervisors in the same cluster? Or-phrased differently-what word cloud characterize the different clusters?
To automatically analyze the guidance experiences and to increase the evidencebased understanding of workplace teaching experiences, we use cognitive computing [7] and text-based clustering. Our purpose is to study what kind of profiles can be found through automatic analysis and what information the profiles in general-and their most typical answers in particular-can provide for the arrangements of the train-the-trainers courses.

Context, Material and Methods
In this section, we first summarize related work of guidance interaction in healthcare training and the context in which the medical train-the-trainers courses are organized (Section 2.1). After that, we describe the collected and analyzed data (Section 2.2), and the applied methodology (Section 2.3).

Background
In medical education, learning from experts, learning by example and learning-bydoing continue to be common teaching practices [8,9]. These occur in contexts that offer experts opportunities to observe learners performances and to support actively during tasks [4]. Apprenticeship learning is a matter of observation, imitation, demonstration and assessing personal skills during authentic work processes [10]. In learning processes, guidance occurs via social interaction with peers, co-workers, trainees or students and even with patients. Interactive talk during guidance plays an important role in the apprentice's learning [4,5]. Typically, the trainers make their own reasoning transparent, to think aloud and to listen while trainees share their thoughts.
The guidance includes direct guidance-oriented interactions (whether verbal or non-verbal) and physical guidance, so that practice-based skills may be acquired in learning-by-doing settings [3]. In addition, guidance plays an important role in organizing learning and in helping the trainees to progress from being a novice toward becoming an expert. The teaching and guiding methods in workplace practices include: (i) focusing the attention of learners on the salient features of the tasks; (ii) monitoring the degree of improvement; and (iii) using different types of demonstrations [11,12]. Acts of guidance of this kind, which take place within guidanceoriented interactions between the trainer and the trainee, are not taught in the degree education. Thus, working together with the competent professional is an effective way to learn vocational skills and to become a competent professional [13,14].

Data
The data for this study comes from an online survey that all participants were asked to answer, before attending a particular train-the-trainers course. The participants answered the following two open-ended questions • Describe a successful instruction/guiding situation related to your normal work.
• What does instruction/guiding means in your opinion?
Besides the open-ended questions, we also surveyed demographic data of the participants, such as the age, gender, highest education, current work assignment, workplace (organization), and the specialty of the medical professionals.
Altogether, data from 16 train-the-trainers courses organized between 2012 and 2018 were collected. This resulted in data from 313 participants. Next, all participants who did not agree that their information could be used for scientific purposes were filtered out. This left us with 281 observations. IBM Watson [7] and the Finnish word parser [15] were then employed to obtain the Finnish words stems and the information on how many times the different stems were used by each participant (the so-called bag-of-words or vector space model; see [16], Chapter 20).
We used only nouns, verbs, and adjectives-altogether 2342 word stems-for further processing, discarding other parts of speech like pronouns and connectives, as they are unlikely to make a meaningful contribution (see, for example, [17]). With stems or root word forms we refer to the most basic form of a word (for example, 'learning', 'learned', and 'learner' would all be mapped to the stem 'learn').
The resulting stem frequency matrix was then processed further with Matlab. First, the number of features was reduced by deleting the rarest stems used only once. Second, the remaining set of stems was manually assessed by two domain experts (the director of CME who is a medical doctor and the principal administrator of CME who is an educational expert with a Master's degree in adult education). These domain experts evaluated all stems based on their meaningfulness concerning the medical training. After non-meaningful words were deleted by the domain experts, Matlab was employed again to remove null observations i.e., those 10 participants that did not use any of the remaining meaningful words. This led to our final participant/used stem matrix of 271 participant and 347 stems.

Method
Let be our set of documents, that is, each document corresponds to the answers given by a participant of a train-the-trainers course. Moreover, let denote the set of distinct stems occurring in after removal of nonmeaningful and rare words as described in Section 2.2. Each document is then represented as a -dimensional vector . Let denote the frequency of stem in document . Then the vector representation of a document is To construct different profiles of the documents, we applied unsupervised clustering. As clustering algorithm, we used the well-known prototype-based partitional clustering framework as depicted in Algorithm 1 in [18]. However, instead of using the Minkowski metric as distance measure and the corresponding statistical estimate like mean or median to represent the centroid of a cluster, we used the cosine similarity. The cosine similarity is one of the most popular similarity measures applied to text documents [19]. It is defined as , where and are two documents in (i.e., -dimensional vectors over the word stem set ). Simply, it measures the angle between the two document vectors. To determine the number of clusters in the data, we used the silhouette cluster validation index [20] with cosine similarity as distance measure.

Experimental Results
To simplify the discussion in the following, we refer to the word stems simply as words. The silhouette index suggested that there are five clusters in our data (see Fig.  1). Fig. 2-6 show the word clouds of the different clusters. As can be seen from the figures, different clusters emphasize different words and we used the most characteristic word of each clusters to name the corresponding cluster.  The name of the cluster as well as the background information of all healthcare supervisors belonging to this cluster are summarized in Table 1. As can be seen from this table, most of the train-the-trainers course participants are females in their forties with a higher education degree. Table 2 depicts the percentagewise distribution of the current work assignment of the healthcare supervisors in the different clusters. As can be seen from the table, most of the healthcare supervisors that participated in the train-trainers courses belong to the nursing staff. Cluster 2 and cluster 4 are the only clusters with doctors. The remaining clusters constitute mainly nurses with different specialties. Below we describe the five clusters and qualitatively summarize the most typical answers of the healthcare supervisors closest to the centroid of the corresponding cluster. This qualitative analysis of actual feedback further profiles the content of the five clusters and through analysis of metadata we aim to explain from which healthcare supervisors such answers were obtained.

Cluster 1: Conversation
Cluster 1 is the conversation cluster. The healthcare supervisor closest to the centroid of this cluster describes a successful guidance situation as "a good connection with trust and shared feelings. A trouble-free situation with no hurry. The patient or the student is willing to start the conversation. Interaction is customer-oriented. The conversation is respectful of others and all participants will be heard. After the guidance situation the participants feel that they have gained something they did not have before." As a whole, conversation and feelings are stressed in the conversation cluster. Only female healthcare supervisors belong to this cluster.

Cluster 2: Learn
Cluster 2, the learn cluster, includes similarly to the conversation cluster exclusively female healthcare supervisors and the second most nurses (i.e., 78%, see Table 2). The healthcare supervisor closest to the centroid of this cluster describes"guidance is supporting other people in learning a new thing, whether the person is a student or a patient." When we analyzed the original answers of the five closest observations to the centroid, we observed that the healthcare supervisors belonging to this cluster mainly reported witnessing the learning and, more generally, the positive effects of sharing information to trainees. The ideas of transferring knowledge and learning are heavily present but also interaction/interplay is mentioned.

Cluster 3: Diagnostic
The healthcare supervisors in cluster 7 emphasize diagnostic in their answers. Here it should be noted that the Finnish word 'tutkimus' could-depending on the context-also be translated as 'patient treatment' or 'disease diagnostic.' This cluster has percentagewise the least healthcare supervisors with a higher education degree (42%, see Table 1). Most of the healthcare supervisors belonging to this cluster have a basic or professional degree. No doctor belongs to this cluster. It consists solely of nurses (92%) and x-ray nurses (8%). All respondents belonging to the diagnostic cluster describe some kind of concrete patient treatment situation when asked to describe a successful instruction/guiding situation. In other clusters, answers to that question are more general in their nature. Our qualitative characterization of the closest observations to the cluster centroid highlighted the whole process of patient treatments (such as that the treatment room is properly equipped and proper clothes are given).

Cluster 4: Skill
Cluster 4 is the skill cluster. This skill cluster consists percentagewise of the least nurses and the second most doctors. The healthcare supervisor closest to the centroid reports the most important guidance competences as the "transfer of professional skills and professionalism to future health care workers." In line with transferring professional skills, the healthcare supervisors in this cluster also speak a lot about supporting the individual goals of the trainees.

Cluster 5: Feedback
The healthcare supervisors in cluster 5 emphasize feedback in their answers. Most of the trainers in this cluster are nurses specialized in psychology. There are no doctors in this cluster. The healthcare supervisors in the feedback cluster are all female and the youngest (mean age 42). The healthcare supervisor closest to the centroid of this cluster describes a successful guidance" the patient understands the guidance and can shortly repeat the given instructions with his/her own words. It is also important to get and give (good) feedback." This connects to Gordon Pask's theory [21]: Having a goal and taking actions to achieve that goal and getting feedback to know if the goal was achieved or if one gets closer to it. Similarly, to the conversation cluster this cluster does not include any male healthcare supervisors. Our qualitative characterization of the answers closest to the centroid of this cluster emphasized trainer-trainee interaction and joint problem solving, to ensure a common understanding and that trainees are capable of managing relevant concerns independently.

Conclusion
CME organizes regularly training sessions for its healthcare supervisors and trainers. These train-the-trainer courses are constantly developed and refined by a team of medical and educational professionals. The courses begin with an open-ended questionnaire and evaluation of the participants' former experiences and aims related to the course topic (i.e. guidance). This wording is then utilized in tailoring the course to meet the aims and needs of the trainees. However, assessing and processing the openended answers is time consuming and usually infeasible to perform if a course is already about to begin. That means that there is an obvious need for an automatic text analyzing method.
In this study, we utilized natural Finnish language processing in order to speed up the text analyzing process. The method included a somewhat time consuming preclassification phase by the two domain experts, but after this activity-that needs to be done only once-the automatic text analysis was fast. We clustered the guidance experience of the participants based on the cosine similarity of the used word stem frequencies of the open-ended answers.
The obtained results were very promising, since the proposed educational knowledge mining process with clustering as the data mining method was able to differentiate content and even varying styles of the participants' answers. The answers were grouped into meaningful clusters that were composed of different guidance profiles. Word clouds and qualitative analysis of the five most characteristic raw feedback texts of the unique clusters summarized the different profiles, providing implications to develop the training arrangements in the future.
As a whole, our paper presented a proof-of-concept for an automatic processing of open-ended answers. The proposed technique can be used for quicker feedback assessment to refine the workplace training courses. Moreover, as far as we are aware of, this is one of the first papers utilizing Finnish natural language processing and workplace learning analytics to realize a seamless chain from collected data to automatic assessment of healthcare education, continuing professional development, and decision making to improve medical practice.