Effectiveness of Instruction for Summarising Handouts and Academic Writings

Academic writing skills are expected to acquire logical thinking and to lead to effective learning. The summarised handout is an academic writing activity. To promote the creation of suitable handouts for learners, a guideline of the preparation procedure was developed in a previous study. In this study. The validity of the procedure is evaluated experimentally. First, the factors useful for assessing handouts were confirmed through a handout generation and evaluation experiment. When the guideline instructions were given before handout creation session, factor scores increased in comparison to the condition when instruction was not given. The effectiveness of instruction was investigated by assessing the handouts generated using the guideline items. In the results, two factors were extracted: "Format" and "Content". The instructions promoted the contribution of the "Content" factor regarding the degree of usefulness of the handout. The contributions of handout content instructions were then evaluated lexically using morphological analysis. The instructions encouraged taking concise notes from handouts, and the total number of morphemes and nouns subjects wrote down decreased, though the number of adjectives increased slightly. Accordingly, the effectiveness of instructions for generating handouts was evaluated. Keywords—handout, writing skills, academic literacy, partial notes


Introduction
Handouts are brief summaries, used to present ideas or topics, which have been produced by a presenter in advance, and these help the audience to understand the contents of the presentation [1]. During educational activity, such as conventional lectures, handouts play a major role as teaching materials, as they are hard copies of the slides presented, and promotes student's note taking activities [2,3]. Therefore, the creation and usage of handouts is an important skill for enriching the transfer of information, and this skill is often developed as a part of academic writing skill development, for writing reports, taking notes, etc. Specific procedures and guidelines are required, as the technique may be different in regards to its function. Adachi et al. [4] focused on handout writing, such as the production and assessment of handouts as tools for communication. One result of their experiment has been the proposal of guidelines for the generation of better handouts. This procedure contributes positively to creating a subjective impression, though it is based on the results of the experiment. As even note-taking skills can be improved following a short period of instruction [5], instruction such as providing guidelines for handout writing may improve student's ability to summarise handouts. The effectiveness of providing guidelines should be examined experimentally.
This paper examines the effectiveness of providing a set of 9 principles as a guideline to be used while generating handouts from documents. The expected effectiveness of this was confirmed.
The following topics are also addressed in this paper: • The improvement of the assessment of subjective impressions of handouts with instruction given are measured using responses from questionnaires answered by reviewers. • The factors are extracted from instructions, and the effectiveness of instructions on the improvement of handouts is measured. • The effectiveness of contents in handouts is evaluated using lexical analysis.
• To evaluate the performance of handout generation and assessment of handouts, a series of summarising experiments was conducted.
To determine these points, handout generation and assessment experiments were conducted.

Related works
Student's writing skills are often discussed as a part of a basic learning skill set for college and university students. In particular, most Japanese universities emphasise the learning of writing skills, because this ability promotes student's active learning abilities through the use of critical thinking skills. The assurance of quality in higher education has become a frequently discussed topic in many countries, and robust academic skills are commonly required and evaluated to confirm the level of ability [6]. In actual practice, many universities develop educational programs with common interdisciplinary learning activities [7,8,9].
However, academic writing skills are considered to be a part of student's academic literacy [10,11]. Some skills are promoted, such as text abstraction [12] and note taking [13,14,15,16]. The impact of note-taking activity has been discussed as being empirical evidence of learning effectiveness [17,18,5]. Partial notes provide some information toward the creation of complete notes, and the effectiveness of partial note taking is also discussed [2,3].
Handouts are presented materials, and most students record additional information on them. In a sense, the function of handouts is similar to partial notes. Therefore, the presentation of handouts is an important factor for learning. Sadoski et al. proposed a list of factors [19]. In addition, annotation factors and strategies were also proposed [20]. In regards to these studies, there are two phases to handout usage, namely handout preparation and handout assessment. Results from the experiments by Adachi et al. [4] show the relationships between factor scores of handouts assessment and the degree of their usefulness. In addition, the relationships between scores of usefulness and qualitative data, which consisted of reviewer's comments and lexical analysis of annotations added to the handouts, was analysed.
The following principles were summarised as training points for the generation of handouts [4].
1. Make headings 2. Choose information, be concise, and produce a well spaced layout 3. Use arrow lines and symbols 4. Consider the logic of contents 5. Make a conclusion 6. Use itemising 7. Highlight important points 8. Write comprehensively 9. Give a title for the contents The effectiveness and the features of instructions were not examined, however. These questions are the motivation for this study.

Experimental procedure
Participants were 10 university students, who were well experienced with summarising texts and learning things using handouts such as textual summaries.
Ten documents were provided to participants as target materials, with each text containing over 2,000 Japanese, Kanji or Hiragana characters (2033 -2385 characters). The contents were two categories with 5 topics on health, science, earth issues, the economy or psychology, for a total of 10 topics. These documents were provided randomly to participants. The experimental task was to produce a handout as a 2 page A4 paper summary of the document in 20 minutes. The participants were encouraged to do their best making handouts as comprehensive as possible and easy to understand.
For the first 5 documents, the task was performed without any instruction given. After that, the 9 principles of handout preparation were given to participants by the experimenter. Thereafter, handouts which were produced followed two conditions.
• Condition 2 (instruction given): Participants were given a procedure for creating 50 handouts using the 9 principles.

Impression and content assessment
Three subjects joined as handout assessors to independently evaluate all handouts produced using the following criteria [4].
1. Impressions of the handouts. Every handout is evaluated with question items which are indicated in Table 1 and use a 5-point semantic differential scale. 2. Impressions of contents of the handouts. Question items are indicated in Table 2, and use a 5-point Likert scale. 3. The degree of usefulness of the handouts is evaluated using a 10-point scale. The usefulness rating is to promote understanding or the necessity of listening to the presentation. 5 points are assigned if the evaluation is neutral.

Requirements assessment
Three subjects independently evaluated the degree of reflection upon instruction which followed the 9 principles using a 5-point scale, 5: Presented sufficiently or 1: Not presented at all.
The overall impression of all handouts was also evaluated. In total, 300 responses were gathered.

Assessment of handouts
Impressions of handouts were assessed using factor scores which were extracted from the responses to question items [4].
Two sets of responses from the questionnaires were analysed using Factor analysis with surveyed data (N = 300). The factor loading matrices with Promax rotation are summarised in Table 1 for impressions of assessments, and in Table 2 for impressions of contents. The two factors were extracted, and labelled as "Well-organised" and "Conciseness" for impressions in Table 1, and "Easy to understand" and "Structured" for impressions of contents in Table 2. The factor structures and question items were almost identical to the previous study [4]. After this, the factor scores were summarised in regards to the factor patterns, as were the usefulness scores.

Effectiveness of instructions
To statistically determine the effectiveness of instruction using the 9 principles for creating a handout, the following analysis was conducted.
As the scores for usefulness may be an indication of overall handout performance, the scores for usefulness are summarised for two conditions (instruction not given and instruction given), in Figure 1. The score for instruction given is higher than the score for instruction not given. To measure the significance of the instruction given factor, one-way ANOVA was applied to the scores. The factor is significant (F (1, 146) The contribution of the factor scores was tested using the same procedure. The factor scores between the two experimental conditions are shown in Figure 2. The horizontal axis indicates the experimental condition, and the vertical axis indicates the respective factor score. Except for "Conciseness", all factor scores for the Instruction given condition increased. As the contents of the source documents used during handout creation might have influenced scores, two-way ANOVA was introduced to test experimental conditions and the contents of each of the categories. The experimental factor for instruction given is significant for "Well-organised" (F (1, 290) = 7.8, p < 0.01), "Easy to understand" (F (1, 290) = 4.4, p < 0.05), and "Structured" (F (1, 290) = 7.2, p < 0.01). For "Structured" factor scores, the interaction between the two factors, instruction (not given/given) and content category (5 topics), is also significant (F (4, 290) = 3.6, p < 0.01). For "Conciseness" factor scores, any source of variation did not affect the scores.   Since the participants who generated the handouts were always mindful of "Conciseness", any contribution this made was not detected. Therefore, the participant's original readiness might have influenced handout creation performance. To confirm this influence, the scores for usefulness of the 3 levels of factor scores and the two experimental conditions were compared. Figure 3 shows the scores for the three levels (Low, Medium, High) of "Easy to understand" in the instruction not given condition, as the original ability levels. When a two-way ANOVA test was conducted, the factors for both original ability level and the experimental condition were significant (p < 0.01). For "Structured" factor scores, both factors were significant (p < 0.01). Therefore, the factor scores for impression of contents were influenced significantly by the readiness of handout creators. Table 3. Factor loading matrix for handouts generated using the 9 principles.

Assessment of the instruction
In the above results, instruction produced an improvement in the contents of handouts. To better understand the effectiveness of giving instructions, key instruction factors, such as the 9 principles, were extracted. First, a conventional exploratory factor analysis was conducted using Promax rotation.
In regards to the Eigenvalues and the contributing factors, two factors were extracted. The factor loading matrix is summarised in Table 3. The overall contribution ratio is 62.3%, and the internal correlation coefficient is 0.74. Here, factor labels are termed "Format" for the first factor, and "Content" for the second factor.
A confirmatory factor analysis was also conducted, and the results are illustrated in Figure 4. All variable labels are as indicated in Table 3.
The factor scores for the two experimental conditions were compared. Figure 5 shows the changes in scores. Both factor scores for the instruction given condition increased, "Format" (F (1, 290) = 62.5, p < 0.01) and "Content" (F (1, 290) = 7.3, p < 0.01). Also, there was a significant difference between the two factor scores in the instruction not given condition, but no significant difference when instruction was given. The results suggest that for most handouts "Format" is not considered a factor in the instruction not given condition. Both factor scores were sufficient once instruction was given.   iJET -Vol. 13, No. 8, 2018 These factor scores may contribute other measurements, such as the degree of usefulness, and this relationship was confirmed using a structural equation modelling technique for multiple groups, such as the two experimental conditions being studied [21]. The relationship is illustrated in Figure 6. All path coefficients are estimated using AMOS software [22]. The model is evaluated using a fitting index, namely the goodness-of-fit index (GFI). In regards to the results, the model is only marginally significant (GF I = 0.91, AGF I = 0.84, RM SEA = 0.08). All path coefficients from the two factors to the variables are significant for the two groups (instruction not given/instruction given). In comparing the path coefficients between the two factors and the two conditions, the factor scores for "Format" affected "Usefulness" when instruction was not given. Also, the factor scores for "Content" positively affected the influence of "Format" when instruction was given.
The handout path coefficients from the two factors for assessment targets to the four factors for impression are summarised in Table 4. All coefficients are standardised. The values of GFI are independent and comparable when the targeted variables are changed in the path model, as shown in Figure 6. Here, responses to questions about the 9 principles produced a major deviation, as the index of fitness of the model is controlled by these contributions. Path coefficients from "Format" increased once instructions were given, except for the degree of "Usefulness" and the scores for "Conciseness". The remaining two factors were influenced by the other factor, "Content", when instruction was provided.
Therefore, the effectiveness of instruction which follows the 9 principles produces different results, as shown by the factors extracted for "Format" and "Content".

Lexical analysis
In the above sections, handout impressions and contributing factors are discussed. The improvement in handouts which occurred when instructions were given was confirmed using metrices of quantitative impressions. These changes are based on the descriptions of the contents of the handouts. In this section, mainly annotated lexical descriptions of handouts were analysed.
First, all annotated text descriptions were converted into machine readable texts. The spatial allocation factor and the layout of handouts are ignored in this analysis.
The texts included all paragraphs and lists of words. All 100 handouts were surveyed. All texts were processed using a Japanese language morphological analysing tool [23]. This tool can extract terms from paragraphs and identify the grammar category of each word. The handouts were composed mainly of nouns and adjectives, and the number of nouns and adjectives was summarised according to experimental condition, and compared to source documents used during handout creation. The mean numbers of morphemes, nouns and adjectives for all handouts are summarised in Table 5. As mentioned above, the original text materials consisted of around 2200 characters, and participants summarised these into around 125 morphemes. The majority (around 60%) of the morphemes were nouns, and some adjectives were also used. As the number of morphemes and nouns decreased after instruction was given, the participant's additional care resulted in slimmer handouts. To confirm the effectiveness of giving instruction, the frequencies of terms, nouns and adjectives for each of the two conditions were compared using two-way ANOVA, to evaluate the effect of the category of content of the source document (5 categories).
The main effect of instruction given is confirmed in the three metrics: morpheme (F (1, 90) = 6.57, p < 0.05), noun (F (1, 90) = 6.39, p < 0.05), and adjective (F (1, 90) = 9.22, p < 0.01). In addition, the factors for category of content of the source document and the interaction between instruction condition and category of content of the source document are significant for only adjectives (F (4, 90) = 3.83, p < 0.01) and their interaction (F (4, 90) = 7.25, p < 0.01). Though frequency of adjectives may depend on the source content category, the frequency is too low and thus the reliability of statistical comparison of the number of adjectives should be considered.
The participant may choose certain more appropriate words, as the degree of usefulness and other factor scores increase once instruction has been given. As a result, from the point of view of assessment factors, better handouts are produced with less numbers of morphemes due to instruction having been given. Though the instructions were simple, and were displayed to participants only briefly, the effectiveness is significant and is confirmed. Therefore, any type of instruction may be useful when making handouts. Various approaches to enhancing learner's use of appropriate instructions will be a subject of our further study. In particular, the contributions of both factors to the overall assessment should studied, so that they may be enhanced. A more detailed set of instructions should also be developed.

Conclusion
This paper confirms the effectiveness of giving instructions to improve handout generation, and the impact of doing so is evaluated using responses from reviewer's assessments, and from lexical analysis.
In the results, the following points were confirmed.
1. Regarding the assessment factors for handouts, four factors were extracted, namely "Well-organised", "Conciseness", "Easy to understand" and "Structured". After instruction was given, all factor scores except for "Conciseness" increased significantly. 2. Factors for the instructions used to generate handouts were extracted and termed "Format" or "Content". The overall assessment of handout "usefulness" depend on the "Format" when instruction was not given. Giving instructions raised the contribution of the "Content" factor. 3. The results of lexical analysis of handouts suggests that instruction, once given, decreases the overall number of morphemes and nouns written while increasing the number of adjectives. The handouts were summarised concisely using specific key words, according to the instructions given.
In order to provide greater effectiveness, more detailed instructions for generating handouts should be developed. A detailed examination of this will be a subject of our further study.