Evaluating the Impact of Blended Learning on Performance and Engagement of Second Language Learners

Despite initial participant enthusiasm for elearning products, severe abandonment has plagued several commercially available e-learning materials intended for autonomous usage [1]. To mitigate attrition rates and provide maximally effective instruction fostering user interest and engagement, Voxy has designed an online learning program within the Task-Based Language Teaching (TBLT) framework, which provides English language learners authentic, relevant content in conjunction with online tutoring sessions. Early research by Voxy on how blended learning impacts performance and engagement has revealed that learners who participate in synchronous, one-on-one tutoring sessions are more engaged than learners who do not; not surprisingly, the learners who are more engaged also show greater proficiency improvements as well.


INTRODUCTION
Much research has been undertaken to evaluate the impact of blended (sometimes called hybrid) learning on outcomes in schools and corporate settings, with evidence suggesting that blended learning results in higher learner performance than distance learning or traditional face-toface instruction does.According to the Department of Education meta-analysis conducted in 2010 [2], a combination of face-to-face instruction and autonomous learning with relevant content is more effective than either alone because blended learning increases the time learners spend studying, thereby improving their learning outcomes.However, this research has typically centered on subjects like math [3,4], health [5,6], and science [7,8], and there has been little research on learning outcomes with second language learners.The research presented here is intended to address the paucity of languagelearning research by considering the impact of blended learning on the performance and engagement of English language learners.

II. PREVIOUS RESEARCH
Before blended learning, e-learning (also called online learning, or distance learning) was initially developed in response to a variety of learner and institutional needs, including convenience and course customization.Through e-learning, learners in remote locations are better able to access high-quality educational resources previously unavailable to them [9].Similarly, learners who do have access to physical learning environments, but do not have the time to attend in-person classes, are able to study at their convenience with online courses [10].Elearning also helps fulfill the need for customized and dynamic instruction for an increasingly fast-paced and digital generation [11], giving adult learners the opportunity to move at their own pace through content of their own choice, allowing for more control over their learning.
Over the past fifteen years, e-learning has remained a popular learning method for adults, as people have increasingly acknowledged the value and benefits of using technology to learn [10,12].Entire universities are online, and there are numerous e-learning platforms for nontraditional students to obtain job-related skills, from giving effective presentations to mastering software programs.Although e-learning has made it possible for busy adults to access learning materials at their own convenience, increased accessibility-and the attendant need for self-discipline-has frequently resulted in high learner attrition rates [1,13,14].
Blended learning has emerged as a way to address this issue by combining the advantages of autonomous elearning with the benefits of in-person instruction in order to keep learners motivated and engaged throughout the entirety of a course [13,15].While the concept of blended learning broadly refers to the combination of two or more learning methods, the current paper defines it as the blending of "online instruction with access to teacher [14]." Although there is research on the benefits of blended learning [16,17,18], few studies focus on the effects of blended language learning [19,20,21].This may, in part, be due to frequent noncooperation-by not taking both required pre-and post-tests-of distance learning participants [22], making it difficult to measure the effects of and draw conclusions about blended learning.This paper will report on a study that was conducted on users of a blended learning program who took two benchmark proficiency assessments.

III. THE CURRENT STUDY
The current study was designed to consider thirty-eight highly motivated adult learners who used self-study language-learning software.Learners were selected on the basis of their having used the software for at least three months, and having taken two hour-long proficiency assessments.The only difference among learners, who were matched according to starting proficiency level, was whether or not they participated in tutoring sessions in addition to the guided self-study activities.While there has been some research on the effectiveness of self-study language learning [1] as well as blended language learning PAPER EVALUATING THE IMPACT OF BLENDED LEARNING ON PERFORMANCE AND ENGAGEMENT OF SECOND LANGUAGE… [19,20,21], there is very little that considers the two in the same study.Because including an instructor-mediated, one-on-one component has frequently been advocated as a method to improve engagement and learning outcomes in distance learning [23], there is a need for empirical research to support this assertion.
The research conducted here was intended to investigate the effect, if any, of the tutoring sessions on learners' engagement with the self-study, as well as their performance.The learners used Voxy, an educational technology product offering synchronous and autonomous online English language instruction via an integrated, multiplatform system.Working within a Task-Based Language Teaching (TBLT) pedagogic framework [24,25], Voxy was developed using established principles of instructed Second Language Acquisition (SLA) [26,27] as well as effective online language learning [15,28].
The study set out to investigate the following research questions, in which performance is measured by score difference between proficiency tests and engagement is measured by days of autonomous learning activity between test completions: 1. To what extent does blended learning impact the performance of language learners?2. To what extent does blended learning impact the engagement of language learners?
The hypothesis was that synchronous language instruction, in addition to offering more practice in and of itself, would lead to more engagement with the software, and, therefore, better proficiency improvements.

A. Learners
Learners in the study included thirty-eight active Voxy learners who had taken at least two versions of the Voxy Proficiency Assessment (VPA), the in-product tool used to measure global English proficiency, discussed in more detail below.Nineteen learners participated in synchronous tutoring sessions while the remaining learners used Voxy's product autonomously and without participating in tutoring sessions.They were all adult learners not attending primary or secondary schools.Table 1 summarizes learners' gender, native language (L1), location, and English-learning goals (which are not mutually exclusive and are selected at the start of each learner's Voxy course).

B. Materials 1) Voxy Proficiency Assessment (VPA)
The VPA, a multiple-choice test, was designed to measure a learner's general level of English knowledge.It assesses the test-taker's grammatical competence, reading comprehension, and listening comprehension.Before learners begin using Voxy, they are first asked to self-assess their proficiency level on a scale of 1 (Beginner) to 7 (Advanced).Once in the product, learners are strongly encouraged to take the VPA in order to obtain a more accurate measure of their English-language abilities and, consequently, a course better suited to their levels.Subsequent versions of the VPAs are offered after three months of instruction, the results of which are used to measure improvements in learner proficiency.All versions of the VPA are parallel so that the content and difficulty level of each of the items is the same across all versions.This way, test-takers' scores do not change over time based on the difficulty level of the test item.
In order to ensure that each VPA is reliable and valid [29], early versions of the test are piloted and revised after performing a Rasch analysis on item responses and testtakers, and administered again with anchor items until a Cronbach's alpha of at least 0.70 is achieved.
Because Voxy levels are divided into seven sub-levels of proficiency, instead of having test-takers at one end of the spectrum answer test items that are either too easy or too difficult, a relatively easier or relatively more difficult version of the test is given.A learner who self-assesses as a beginner, high beginner, or low intermediate will receive a test that includes only the easy, medium, and hard items.A learner who self-assesses as an intermediate, high intermediate, low advanced, or advanced will receive a test that includes the medium, hard, and very hard items.
2) Tutoring Voxy's synchronous tutoring sessions are conducted online, in a one-on-one setting, and are designed to offer personalized instruction not as readily available in a traditional classroom setting or with autonomous learning.Tutors receive extensive training in how to troubleshoot and facilitate successful one-on-one language lessons [30], and learners can choose one of five different foci for their lessons: (1) conversation, (2) oral fluency, (3) grammar, (4) writing, or (5) role play.Upon booking, the learner receives a focus-specific Voxy resource, with attendant activities, to complete before the session.After the session, the learner receives a form from the tutor providing quantitative and qualitative feedback with recommended follow-up activities and resources.
All sessions are conducted via video chat, with the tutor and learner using interactive tools like a whiteboard, chat box, and shared screen to facilitate the lesson.Sessions iJAC -Volume 7, Issue 3, 2014 PAPER EVALUATING THE IMPACT OF BLENDED LEARNING ON PERFORMANCE AND ENGAGEMENT OF SECOND LANGUAGE… involve targeted speaking practice and frequent interactional feedback [31,32], which help learners notice mistakes they have been making and produce modified output thereafter [33,34].The sessions also incorporate goal-and interest-specific instruction based on the learner's Voxy profile and tutoring history, which is provided to the tutor before the session.Tutors are also provided with ideas for how to incorporate Voxy's or other real-world materials into the sessions so learners are exposed to as much genuine language as possible [24,25,26,35].

C. Procedures
The learners in this study used Voxy for at least three months between taking their first VPA (VPA 1) and their second VPA (VPA 2).They completed level-appropriate goal-and interest-based lessons that included news articles and other relevant texts, songs, and real conversations in audio and video form.Learners were also offered activities in each lesson that targeted crucial skills such as reading and listening comprehension, and were provided with the opportunity to produce the language by interacting with native speakers [27,36] through synchronous tutoring sessions.
Because this study was conducted with existing data, learner performance records were searched to identify learners who fit a pre-determined profile.Specifically, one group of Voxy learners was identified who met the following criteria: (1) had taken at least two VPAs and (2) had participated in at least one tutoring session.In order to compare the performance of learners who completed tutoring sessions with learners who did not, a group of learners who had taken two versions of the VPA but no tutoring sessions was also identified.However, this subset was much larger than nineteen (N=123).To compare groups with approximately equal proficiency levels, each of the nineteen learners who had taken at least two VPAs and completed at least one tutoring session (Group A) was matched to a Voxy user with a similar VPA starting proficiency who had completed no tutoring sessions (Group B).
A paired samples t-test indicated that there was no significant difference in the starting proficiency scores for learners in Group A (M=831, SD=279) and the learners in Group B (M=831, SD=272); t(18)=0.117,p =0.908.After the two groups of learners were identified, data on their VPA performance and engagement with the software were analyzed using a one-way analysis of variance (ANOVA).

A. Performance
Thirty-eight learners took two versions of the VPA, with a mean between-test period of 103 days (SD=18).While learners in both groups had virtually the same VPA 1 score, the descriptive statistics in Table 2 indicate clear differences in their VPA 2 performance.

B. Engagement
In addition to their performance on the VPAs, learners' engagement with the Voxy product was also considered in terms of the number of days spent using the software and the total number of activities completed between taking the VPAs (see Table 3).
The total number of activities completed was highly correlated with the number of days active (r=0.847,p=0.000), so only the mean days active was considered in the inferential statistical analysis.The standard deviations indicate that learner engagement was not uniform and that there were large differences in time spent using the Voxy product between VPAs for both learners who did and did not participate in tutoring sessions.That said, the mean number of days active for Group A was more than double that of Group B, and an ANOVA revealed that participating in tutoring sessions had a significant effect on the number of days learners used the software; F(1, 36)=5.103,p=0.030, !p 2 =0.124.

C. Number of Tutoring Sessions
As the number of learners was very small in each group and the range of tutoring sessions within the tutoring group was so large (M=7, SD=8), Group A was split into two subsets-Low Tutoring, and High Tutoring-to consider the effect of number of sessions on learner performance and engagement.Low Tutoring (n=10) comprised learners who had attended one to four tutoring sessions, and High Tutoring (n=9) comprised those who had attended five to thirty-three sessions.Given the very small nsize of the two tutoring subsets, these data were not considered with inferential statistical analyses; nevertheless, they reveal trends that are worth considering.
Because the VPA 1 scores of the two tutoring subsets were not equal (Low Tutoring M=731; High Tutoring M=943), the score difference between VPA 1 and VPA 2 was calculated for these subsets (see Table 4).
The score changes indicate that the High Tutoring group slightly outperformed the Low Tutoring group.To measure engagement between the groups, the descriptive statistics for total days active (see Table 5) were calculated.
As evidenced by the descriptive statistics, the Low Tutoring group was vastly more engaged with the software than the High Tutoring group though each group had high standard deviations.

VI. DISCUSSION AND CONCLUSION
The purpose of this study was two-fold: to provide more empirical research on the effects of synchronous tutoring sessions on otherwise autonomous language learning; and to measure the effectiveness of this blended method.This paper focused on investigating the extent to which blended learning impacts performance and engagement language learners.
The research conducted in this study indicates that those who use synchronous language instruction in conjunction with self-study do, on average, increase their proficiency scores to a greater extent than those who only use the latter.This proficiency increase is likely caused by the augmented time spent studying by the group that participated in tutoring sessions.Along these lines, the findings presented here also indicate that Group A was more engaged-or used the product more frequentlythan Group B. In terms of both number of days engaged with the product and number of activities completed, Group A was approximately twice as active as Group B. In terms of performance, despite having the same mean VPA 1 score of 831 points, Group A had a mean score increase of 71 points, while Group B had a mean score decrease of 40 points.
Because Group A varied so widely in terms of the number of sessions attended by each learner, the data was further examined in two separate groups: Low Tutoring and High Tutoring.While these groups had relatively similar proficiency gain scores (Low Tutoring M=63; High Tutoring M=79), the group that had fewer tutoring sessions was far more engaged with the software autonomously.It is possible that learners who were more engaged with tutors had less time to spend studying on their own, or that they felt they needed fewer hours of selfstudy because of the larger amount of one-on-one instruction.Future research should consider both the amount of tutoring and the number of activities completed by learners in groups with the same number of tutoring sessions in order to tease out the impact of specific numbers of tutoring sessions on both engagement and performance.
These findings could be interpreted as indicating that Group A's mean score difference was higher than that of Group B simply because of the tutoring sessions.However, Group A, on average, engaged far more with the product than Group B in addition to the one-on-one tutoring sessions.In fact, it could also be argued that the combination of one-on-one tutoring, and self-study is what drove more overall engagement with the product, in part because learners are encouraged to engage with Voxy resources and activities before, after, and during a tutoring session.Learners in Group A also received targeted feedback in-lesson and post-lesson, to which the better performance on their second VPA may also be attributed.This is underscored by the observation that though the Low Tutoring group was, on average, much more engaged than the High Tutoring group, the latter had a slightly higher score increase on VPA 2. It is important to consider that because no additional information was gathered about the learners' exposure to English outside of Voxy, there possibly were external factors that contributed to the higher engagement and performance of Group A in comparison with Group B.
The small sample sizes of learners in this study was in part due to the relatively recent addition of the tutoring and VPA components into the Voxy product, just eight months prior to analyzing the data.However, while the data is sparse, the findings are also encouraging.Additional research should be conducted in the future, when there is a larger number of tutoring users who have taken two versions of the VPA, so as to draw more meaningful conclusions about the extent to which blended learning impacts proficiency gains and fosters engagement.

TABLE I .
DEMOGRAPHIC INFORMATION ABOUT LEARNERS (N=38)

TABLE II
IMPACT OF BLENDED LEARNING ON PERFORMANCE AND ENGAGEMENT OF SECOND LANGUAGE… Figure 1.Total score difference for VPA 1 and VPA 2 for learners with and without tutoring THE

TABLE IV .
VPA PERFORMANCE BY TUTORING GROUP