Short Paper—Comparing the Impact of Programming Assessment Type: In-Class Vs Take-Home Comparing the Impact of Programming Assessment Type: In-Class Vs Take-Home

In engineering education, educational technologies and teaching interventions are often used to maintain the student retention high. Most of them are strongly related to either in-class or take-home assessment. This paper compares the impact of both assessments on student performance for one academic semester. The findings are expected to re-align the focus of educational technology and teaching intervention research. Fifty-five laboratory sessions were analyzed, involving five Introductory Programming classes with a total of 87 Information Technology and Information System undergraduates. The study shows that, by considering only significant changes, in-class assessment leads to higher assessment mark as the students can easily seek help and focus on the tasks. Technical problems are also unlikely to occur as the classroom (or the laboratory in our case) is well-facilitated. Keywords—Assessment type, programming, educational technology, teaching intervention, engineering education


Introduction
In the academic environment, a faculty or a department is often perceived as successful if they are able to maintain the student retention high [1]. A number of techniques are then developed in favor to such requirement [2], namely: Student Success Course [3], [4], the use of persuasive social media [5], [6], the integration of educational technologies [7], [8], and the introduction of new teaching intervention [9], [10]. For engineering majors, the last two are commonly applied.
A number of educational technologies have been utilized in engineering related major. They are ranging from hardware devices (e.g., electronic voting clicker [11], LEGO MindStorms [12], Arduino [13], or Raspberry Pi [14]) to educational software (e.g., Algorithm Visualization [15], Program Visualization [16], or Visual Programming tool [17]). In general, these technologies share the same objective: to assist students in learning.
There are several teaching interventions that are often used in engineering. Flipped classroom [18], [19] encourages students to learn the instructional material outside the class session so that the session can be used to do the activity or assessment. Team-based learning [20], [21] requires the students to learn as a group. Game bootstrapping [22], [23] encapsulates the teaching material as games to motivate students. Pair programming [24], [25] lets the students to complete a programming task as pairs. Peer code review [26], [27] encourages each student reviews another' code and provides some feedback.
In applying those educational technologies and teaching interventions, assessments are commonly involved to help students reflecting their understanding toward given materials. These assessments can be classified to two types: in-class and take-home.
To the best of our knowledge, no studies compare the impact of those assessment types on student performance, despite many comparative studies measuring the impact of the educational technologies and teaching interventions [28]- [30]. We believe comparing those two assessment types, finding the most suitable one for students, can realign the focus of educational technology and teaching intervention research. If in-class assessment is preferred, educational technologies should facilitate some in-class assessments [31] or it should be used for completing such assessments [16]. Further, teaching interventions like pair programming and code peer review should be performed during the class sessions. Teaching interventions focusing on in-class assessment like flipped classroom should also gain more attention. If take-home assessment is preferred, educational technologies can focus more on assisting students in understanding the materials. The allocated time for completing take-home assessment is not as tight as the inclass one. Hence, the technologies can be used by the students without time pressure. In addition, teaching interventions in classroom can focus more in understanding the course material (e.g., team-based learning and game bootstrapping). Whereas, teaching interventions outside the classroom can focus more on completing assessments (e.g., pair programming).
This study compares the impact of in-class and take-home assessments on student performance for one academic semester. The recorded data covers undergraduate students' assessment marks, involving three Information Technology classes and two Information System classes. All of them were taking Introductory Programming on their corresponding major, resulting programming as the assessment context.

Method
The study was conducted on Introductory Programming, a course where the students first learn how to code. In our faculty, this course was offered in two majors for undergraduates: Information Technology (IT) and Information System (IS). To compare the impact of in-class and take-home assessments, students enrolled in this course were asked to complete both assessments weekly, and their marks for each week would be further analyzed.
For IT students, the in-class assessment should be completed within two hours in a laboratory session. The take-home assessment was given at the end of that laboratory session and it was expected to be completed prior the next week's session. Both assessments were made to be comparable; each of them has three different Python programming tasks: easy, medium, and challenging. We are aware that while completing the take-home one, students probably have understood given course material further. Hence, the difficulty of the take-home assessment was slightly increased.
For IS students, the scenario is somewhat similar except that the course materials were given in a slightly different order. Further, the programming tasks should be solved with both Python and Java at once (i.e., two solutions per programming task). The lecturers believe that completing the tasks in two programming languages can encourage the students not to rely heavily on one programming language. Regardless, as our focus is not on that issue, no further discussion will be given.
The statistics of the recorded data can be seen in Table 1. IT-A, IT-B, IT-C are IT classes while the rest are IS classes. Both IT and IS classes actually had the same number of sessions (14). However, some of the sessions were used for tests, quizzes or review. IS-B has fewer laboratory sessions than another IS class due to human error. In total, there are 55 analyzed laboratory sessions with a total of 87 undergraduate students on board. The analysis was performed by pairing each in-class assessment with its corresponding take-home assessment from the same week. Their average marks would be calculated, and the significance was measured with two-tailed paired t-test with 95% confidence rate.
For each pair, students who only completed one of the assessments was excluded from the comparison. Any students who were indicated to plagiarize or collude would also be removed as their marks do not reflect their real performance. Plagiarism and collusion suspicion were raised manually by the lecturer and/or the tutors, mainly based on the shared similarities from given source code files.
After that, the changes of significant pairs would be observed. If most of them show higher mark for in-class assessment, it can be stated that in-class assessment is preferred. Otherwise, take-home assessment can be more beneficial. A discussion about the result would be given at the end of the analysis.

Result and Discussion
Among 55 analyzed laboratory sessions, eighteen of them have statistically significant difference between student marks for in-class and take-home assessments. The details can be seen in Table 2. Each session ID has three components (major, class, and session number) and they are separated with hyphen. In-class and take-home marks are the average value of all eligible corresponding marks for that week (i.e., those from students who completed both types of assessment with no suspicion of plagiarism and collusion). The highest value among those two marks is written in bold and the p-value can be seen in the next right column, followed by the covered material. Only a third of the significant changes favors take-home assessment, even though we expected that kind of assignment would outperform the counterpart as the allocated time is longer. Some students might need a considerable amount of time in understanding the task and then solving it. However, it seems that these students are more in need of guidance rather than longer allocated time. They need some hints in order to solve the tasks. Despite the hints can be gained by asking smart students or tutors outside the class sessions, only few of the students and the tutors are available to help and can be met in person.
Longer allocated time in take-home assessments can also lead more constructive discussion and more access to helpful resources (e.g., textbook and internet). But again, it seems that a little guidance is required to initiate those discussion and resource access.
Majority of the significant changes favor in-class assessment. During the completion of the assessment, it is easier to seek help, as the tutors have allocated that time for helping the students and they can be met in person. It is also easier to discuss how to solve the tasks with friends as all students are focused on solving the same tasks.
Another benefit of having in-class assessment is that the allocated time should be mainly used for completing the assessment. They cannot be distracted by other activities as they are being monitored by the lecturer and/or the tutors. Outside the classroom, the students can jump into other activities as no one monitors them.
In-class assessment also enforces the students to complete the tasks in the same place (e.g., the classroom or the laboratory), and that place is commonly well-facilitated. Many technical problems like hardware or software errors are unlikely to occur and delay the completion of the assessment. This can work in reverse for take-home assessment as student-owned computers are not guaranteed to facilitate that completion due to a wide range of computer specifications and variants. Further, not all students have their own computer. Some of them need to share a computer with their family, which can also delay the completion of take-home assessment.
No strong findings about the covered materials can be gained from the significant changes, even though some materials seem to be frequently occurred. However, we can say that in-class assessment is likely to be beneficial on looping (but not the nested one) and array-related materials (array, matrix, and function + array).

Limitations of the Study
Our study has three limitations. First, the study was performed only on introductory programming students in which the assessments are purely programming. The findings can be different if the students enrolled in other courses are taken into consideration, especially when the assessments are not programming-related. Second, the suspicion of plagiarism and collusion was raised manually, and it is possible that some of the perpetrators were not caught. The use of automated similarity detection tool like JPlag [32] is recommended to solve this limitation in the future. Third, our study ignores laboratory sessions with insignificant changes as our intention is to know which assessment type leads to higher assessment marks, not whether the change of assessment type affects assessment marks. Further study is needed to confirm the second statement.

Conclusion and Future Work
This study compares the impact of in-class and take-home assessments based on student performance for one academic semester. With only significant differences on board, it shows that in-class assessment can be more beneficial as the students can easily seek help and they can focus on completing the tasks with minimum distraction. Further, this kind of assessment mitigates the occurrence of technical problems as the classroom or the laboratory is commonly well-facilitated.
Based on the findings, it is suggested that in-class educational technologies should be able to assess the students' performance, while pair programming and code peer review should be conducted during the class or the laboratory session. We also believe that flipped classroom can be beneficial as the assessment is commonly completed during the class session.
For future work, we plan to conduct a larger study involving students enrolled in more advanced courses like Advanced Object-Oriented Programming or Artificial Intelligence. We also plan to do similar study on courses which assessments are not purely programming like Project Management or Calculus.