An Evaluation of Online and Automated English Writing Assistants: Collocations and Idioms Checkers

In this paper, I attempt to investigate whether online writing checkers, including tools looking at grammar, punctuation, spelling and composition, do in fact help potential ESL writers and, if so, to what extent, in what areas, to what effect and at what cost. The small-scale study involved twentythree university level ESL students who composed various texts as part of their training. Data were collected using pre and post writing samples, questionnaires and semi-structured interviews. The analysis used the Likert-scale and correlations as descriptive statistical methods and qualitative analysis for the interviews. The results are inconclusive as students reported different opinions. It was found that students who are frequent users of features such as MS Word Thesaurus and Online Corpora are more likely to incorporate online writing checkers and therefore found them more helpful. It was also found that, regardless of their effect, students always welcomed available tools that provide instant feedback to their compositions. The writing software used, AccurIT, contributed to improvements in the students’ ability to write correct collocations. Keywords—ESL Writing, Technology in Education, Collocations, Idioms


1
Literature Review

Technology and ESL Writing
The last few years have witnessed an increasing interest in developing automated assistance software in ESL writing. These programs benefit from the availability of free and easily accessible databases and concordances online as well as texts from various sources from all over the English speaking world. Comparisons and inferences can be drawn easily using algorithms that can in theory benefit ESL students by directing them to potential errors in their writing and what to do about them.
[1], for example, noted that the capabilities and availability of technology have resulted in their implementation in L2 writing instruction. The studies in the field according to [1] are both quantitative and qualitative in nature, as well as being collaborative and individual in target. They believe that the overall majority of research in L2 writing technology has a positive effect on both achievement and perception. Howev-er they draw attention to the fact that L2 writing performance and the impact of technology is not conclusive as other factors should also be investigated. [2], on the other hand, looked more exclusively at students' perceptions of L2 writing programs. She found that students expressed positive attitudes towards feedback provided in the forms of screencast, video and text feedback for their efficiency, clarity, ease of use and heightened understanding. Her observations further corroborated students' perceptions since those who employed technology in L2 writing took less time to revise their texts. Furthermore, students who employed technology in their writing asked less clarification questions. Finally, she found that video feedback offered 33% time saving compared to conventional approaches of feedback in writing. [3] looked at the role of social media such as Google+ towards improving ESL writing skills at secondary school level in Malaysia. Participants in the study had positive perceptions about using social media in writing classes. Students believed they had enjoyable and attentive lessons. They also believed they had a sense of purpose which meant that they became more engaged in writing tasks. Students also reported more interaction in social media writing classes than conventional classes.
In the same manner, [4] also investigated the role of automated feedback on the progress of ESL writing and their study reveals similar findings as to students' perceptions and the positive impact on their results. [5] developed an analyser for casual discourse in ESL writing using automated writing evaluation (AWE) to provide corrective feedback. The study statistically proved that students who used AWE were considerably more accurate than their counterparts who did not use any writing software.
On the other hand, [6] and [7] mentioned a few drawbacks with these tools, most specifically the automated writing evaluation (AWE) programs. AWE's impact on writing instruction and performance is inconclusive. [6] however conceded that AWE corrective feedback has a positive impact on writing accuracy and led to increased revisions.
Despite all the positive impact of modern technologies in ESL writing, [8] argued that they are not formally included in writing classes, not to the extent that they would like to see anyway. They also mentioned the benefits of online blogs on ESL writing and the mutual benefit on students' writing.
As we can see from the previous literature, the general perception of online and electronic tools in L2 writing is positive, both from the students and in terms of saving time and effort providing feedback. It is therefore worth investing time and effort introducing these tools to ESL students according to their needs and expectations. The experiment should be carefully documented and investigated in order to highlight any potential challenges and come up with possible solutions.

Online concordances and corpora
These are generally the most professional of online ESL writing tools, used almost exclusively by experts and researchers but can be of great benefit to any potential ESL writers. They rely on mathematical algorithms to identify and highlight instances of word associations that cannot be clearly explained by grammatical rules and gen-eral instructions. The most prevalent manifestation are those of collocations and, to a similar degree, known phrases and idioms. Collocations are defined by the Oxford Dictionary as the "the habitual juxtaposition of a particular word with another word or words with a frequency greater than chance".
The theory is that certain words appear together with more frequency than just by chance, yet there are no clear rules governing their composition and structure. ESL writers may easily fall into the trap of literal translation because there may be more than one word with similar or even identical meanings which do not usually combine with others. For example, the phrases 'taking a shower' and 'having a breakfast' cannot usually substitute each other's verbs despite having identical meanings in the ESL students' mother language, resulting in confusion and inevitable mistakes. Other examples are 'strong tea' and 'heavy drinker' where the adjectives cannot usually be replaced.
The problem with teaching collocations to students is their sheer number and wide ranging differences according to English variation or even context [9], [10] and [11]. To solve this problem, technology now provides feedback as to which word combines with which other and in what context. This is particularly helpful to students who are not familiar with the idiomatic use of language and who are not familiar with the concept of collocation. Examples of word concordances are plentiful ranging from free to paid services including AccurIT, Conc, Alceste, Hamlet, Intext, Oxford Concordance Program, MicroConcord.

AccurIT collocation and idioms checker
AccurIT is an online programme jointly developed by King Abdulaziz University and Umm Al Qura University to help students write correct phrases and collocations in English. [12] It aims to eliminate incorrect combinations of words mainly due to literal translation which negatively affects students' ESL writing by drawing examples from online concordances and corpora as well as its built-in databases. In this study, English major students were trained to incorporate recommendations as to which words combine in sentences using AccurIT.

Methodology
The study was carried out in King Abdulaziz University's English department, which offers Bachelor and Master Degrees to students in English literature, linguistics and translation.
Ten students from each year of the Bachelor degree were contacted, selected according to their GPA results, to create a representative sample of the research population. In total, twenty-three students responded and agreed to take part in the study. All students are registered in either Writing 1 or Writing 2 in the department, both of which are compulsory courses for all English majors.
Students' writing samples were accumulated throughout the semester and they included pre and post exposure to AccurIT ESL writing software.

Written samples
Students' writing was assessed at the beginning of the course, prior to taking part in the AccurIT training session and afterwards. All participants were asked to write a one-page passage in sixty minutes and they were free to consult dictionaries. The compositions were assessed by two professional writing instructors using a predetermined checklist. In total, forty-six writing samples were collected, assessed and analysed. The errors were categoried but the focus was on collocational, idiomatic and phrasal as well as incorrect word choice since AccurIT deals with these types of errors.

Questionnaires
Students had also taken part in a questionnaire that followed a Likert-scale approach. The questionnaire looked at various aspects of the students and their familiarity with technology in writing as well as their competence and expectations. The questionnaire asked general questions about students' preferences and expectations from ESL writing tools, how easy or otherwise they are and how often they want to see them in their writing classes. SPSS was used to accumulate the data and generate statistical results.

Interviews
Two students from each academic year were involved in a post-experiment interview. They were selected according to their performances in the exit writing test which means a high performer and a poor performer were selected from each year. Students A, B and C are from the group that relied exclusively on conventional feedback (Group A), while students D, E and F are from the group that was taught to use feedback from AccurIT as well (Group B).
The interviews followed a semi-structured approach to allow for some freedom as to the direction of the discussions. The themes which emerged from the interviews were highlighted and the interesting points raised by these students were accounted for.

Written tests
Students' writing did improve throughout the semester regardless of which training they received, AccurIT plus conventional or conventional only. However, accuracy and correct word choice were considerably better in the test that employed AccurIT to provide feedback. The following table shows the number of errors in terms of word choice, phrases and collocations between the two groups. Group A is the one that relied exclusively on conventional means of feedback and their total number is eleven iJET -Vol. 15, No. 4, 2020 students, five second years, five third years and one in his final year. The other group is the one that used AccurIT in the exit test. Their total number is twelve divided equally between the second and third years.  Some interesting results can be witnessed here. The first is that the idiomatic use of language was less evident in both groups thus there was a lower number of errors. Secondly, the relatively high standard deviation in both groups shows the individual differences between students, although Group B showed more consistent results among its members.
Group A's results show that the most frequent type of error is collocations. As noted in the literature above, this type of error can be very common and students' first language and literal translation do play a role in that. The problem is further compounded by the absence of any corrective feedback during the test.
As with Group A, the other group also committed the most number of errors in collocations. However, the average number of errors was far lower than in the other group.
Statistically speaking, both groups made errors in all three categories investigated. However, the group that employed AccurIT did considerably better in all aspects, lending support to the argument that L2 writing tools do help students write more accurately and therefore conform to the previous research as indicated in the literature review above.

The surveys
The questionnaire was distributed to all students after the writing tests. (See Appendix A) The results were variable and interesting. As for students' familiarity with different technologies that can be used in ESL writing, all twenty-three students were familiar with Microsoft Word and online dictionaries. They, on the other hand, knew almost nothing about online corpora, concordances and very few knew about online writing blogs (4 out of the 23 surveyed).
The results are interesting, yet not entirely surprising since they are required to submit assignments using word processors and certain subjects in the department require consulting dictionaries online.
As for students' familiarity with already available tools such as corpora and concordances, it can be said that the result was a disappointment. None of the twentythree students even knew what these were and only four knew about online ESL blogs.
The three remaining questions asked students about their opinions regarding how helpful they think these online tools are, whether to make them part of the writing curriculum and how much time teachers should devote training students to use them. The results were all in favour of technologies in ESL writing.  Table (3) shows that, on a scale from 1 to 10, where 1 means strongly disagree and 10 means strongly agree, none expressed any disagreement as to whether technology is helpful in writing classes. In fact, the majority of students had very positive attitudes towards technology in ESL writing classes.
As to whether to make these tools compulsory in writing classes, the students were all strongly in favour of implementing them and all had very similar attitudes. There was a slight surprise here given the fact that none of them had even heard of common tools such as corpora and concordances, but the fact that online and automated programs in writing give more chances of interaction and corrective feedback, students approved of them nonetheless.
Finally, as to whether teachers should devote more time training students on these programs, students were extremely positive and they expected that from their ESL writing teachers. (See Figure 1) It must be noted, however, that in comparison to the two previous questions, a larger number of participants did not actually have an opinion about the issue which may reflect their concern about spending more time learning technologies that they are not familiar with and/or sure about. This question will be raised in the following section.

The interviews
The interviews with six students followed both the written tests and the subsequent questionnaire. The semi-structured approach used aimed to answer specific questions yet allow for more input from the participants when required. The interviews with members from group A asked general questions about ESL writing, what challenges students faced, what can be done to solve them and the role of technology in ESL writing. Members of the other group were also asked about their experiences with using AccurIT in their writing test, the value of feedback they received from the tool and whether they think it should be integrated in future writing classes in the department.
Regardless of which group the interviewees came from, all seemed to have very positive views about technology in writing. Interviewees pointed out that such tools can provide more feedback to their writing and, in the case of group B, provide detailed information about their selection of vocabulary in a way other conventional means usually cannot.
Students from both groups raised concerns about the amount of corrective feedback they received from their instructors. Interviewee B, for instance, believed that the amount of feedback he received from his writing instructor was negligible.
All interviewees believed that suitable online and automated tools can balance the need for more corrective feedback compared to the little information they get from their instructors.
Interviewees D, E and F from Group B, who used AccurIT in their writing test, praised the detailed and accurate feedback they received as to how certain words combine. They were aware of the troublesome idioms and collocations in English and they believed AccurIT could genuinely improve the accuracy of their ESL writing.

Recommendations and Conclusion
As a general rule, any attempt to help students compose better ESL texts should be appreciated. The challenges that face ESL students in all aspects of language, especially writing, are enormous. Online and electronic writing assistance tools, including concordances, grammar checkers, thesauruses and others, are a great manifestation of these efforts that combine new technologies to solve an age old question. That said, it must be noted that these tools should never run counterproductive to their intentions, i.e. they should not add to students' workload in terms of learning how to use them and what results to consider.
The study, despite being small-scale, lends support to previous research which suggested that students have positive attitudes towards automated and online tools in ESL writing. Teachers should build on this and encourage students to be more interactive and employ any means necessary to improve their compositions. Online tools do, in fact, provide valuable corrective feedback tailored to students' needs in a way teachers may struggle to achieve.