A Experience of Conducting Online Test During COVID-19 Lockdown: A Case Study of NMIMS University

prathamesh.churi@gmail.com Abstract —The extent of COVID 19 pandemic results in closing down the universities and colleges across the world. The most vulnerable of tuition-de-pendent institutions, particularly the ones already facing demographically driven declines in demand, will be the hardest hit by the pandemic. The online teaching learning and assessment at this time becomes a crucial part of education. Taking the online test was crucial as lots of malpractices would happen. The developing countries like India, where the advancement of e-learning, online education, MOOC has not reached to its peak level, the assessment and evaluation of students becomes very difficult. Due to unforeseen situation like COVID-19 lockdown, the above paper shares a fruitful experience of conducting online case study-based paper and its online evaluation. The case study was conducted in the classrooms of the undergraduate class of NMIMS University for Multimedia Systems Course. This case study-based paper leaves with the message of acceptability of the online teaching-learning and its effect near the


Introduction
In this day and age, where the number of people graduating every year is increasing, maintaining the standard of education has become critical. According to an article published by The Hindu Business Line, on May 11, 2018, "Annually, about 15 lakh engineers graduate, according to an AICTE official". When such a huge number is considered, professors and universities must make sure that the standard of education doesn't drop and that students are learning and understanding all the concepts and have the skill set necessary to be a befitting engineer. [1] In such a scenario, assessing how much a student has understood is necessary and possible only through continuous assessment. [2] Students will actively try to understand each concept if it changes their grade at the end of the semester. The professor's job here is to set a paper to cover significant concepts and make use of the student's knowledge; and after the student has taken the test -to realize what they've grasped and what they haven't. The traditional method of assessment has been to set a paper, which is directly based on the topics taught in class or described in reference books. Direct questions are asked which hardly require a person to think out of the box. Although indirect questions are asked, it is seldom and usually for very few marks to make sure the student doesn't lose a lot of marks. In such a scenario, where application-based questions are rarely asked, the student can mug up the answers instead of learning a concept. This hinders in their understanding of subjects and to become qualified engineers.
The test talked about, in this paper was conducted in the Engineering College of NMIMS University. NMIMS University follows a pattern where 30 marks are for two exams set by professors, 15 marks each. Additionally, 20 marks are for their performance in practical's, vivas, quizzes, etc. This makes sure that the students are continuously evaluated based on their knowledge, ability to perform experiments and ability to do well on tests. Multimedia Systems is an elective course taken by semester eight students. The test was supposed to be a 15mark test.
On 31st December 2019, 27 cases of pneumonia of unknown aetiology were identified in Wuhan City, Hubei province in China. These patients most notably presented with clinical symptoms of dry cough, dyspnea, fever, and bilateral lung infiltrates on imaging. On 30th January 2020, the WHO declared the Chinese outbreak of COVID-19 to be a Public Health Emergency of International Concern posing a high risk to countries with vulnerable health systems. [3] In India, the SARS-CoV2 COVID-19 epidemic has grown to 1,251 cases and 32 deaths as on 30 Mar 2020. [4] On 24th March, a nationwide lockdown was announced in India. Due to this, universities were shut down and lectures were shifted to online lectures.in fact, a worldwide rise in active users' number had been observed by Prioridata and Statista in figure 1. [5]. Professors were finding it challenging, due to their lack of touch with technology, inadequate resources and not much external aide. Since the interaction between the teacher and student decreased, it got tougher for professors to explain and students to follow. It also became difficult for professors to keep students engaged. Internet speed, resistance to the new system of learning, managing time along with all the household chores were further issues faced. Online lectures not only were a concern for teachers but students also. Willingness to study or do any sort of work was lost due to increased levels of anxiety, which was a result of being restricted for a long time. [6] Students were not used to such a lifestyle and it was affecting their emotions and mental health.
NMIMS University is using Zoom software to conduct lectures. With a not-so-great internet speed it became difficult to retain a student's interest, and so did teaching. In such a situation, taking a test was of utmost importance. The test in consideration was divided into two partspart 1 was theory-based, for 7 marks and part 2 was a complete case study based, for 8 marks. Studies have shown that case studies are an effective way to help students learn in an online environment. [7][8] [9] [28] This ensured that the student had a grasp on the theoretical part of the subject and could apply it too.
The case study was befitting the current global scenario. The question asked the students what information would they want to display on a dashboard providing news about the COVID-19 situation. Further, it also questioned the student's thinking ability, in the next question when they were asked why would they want to provide this information. The last question, made the student think about the humanitarian aspect, as it asked, what would the students do for the senior citizens who might be one of the worst affected aspects of the society. The three different questions made the student think in different areas, be it the statistical side, the reasoning side or the humanitarian side. The result was that students not only scored better but also had a deeper understanding of the subject, its need and value in today's circumstances. [10] [29][30].

Related Work
There have been several studies comparing Computer-Based Tests (CBT) and Paper-Based Tests (PBT), with varying results. Students' attitude towards online assessments is not primarily driven by examination results. According to Shannon, James, et al [11] anxiety due to lack of technology and system knowledge along with writing on scratch sheets, instead of directly on exam sheets, was a major issue affecting the preference of students. The inability to mark questions, annotating their rationale when eliminating an answer choice and making shorthand notes near the questions itself was another hindrance to the experience. Despite all the issues, the students' grades showed improvement in CBT as compared to traditional assessment patterns.
When considering the impact of the medium of conducting assessments on student grades, Sarah, Rashi, et al [12] observed contradictory results (i.e., a fall in student grades on CBT). A major limitation of conducting CBT was the lack of knowledge of both the students and educators in devices and technologies that are required to conduct the assessments. Students required substantial time in understanding and getting accustomed to the software and the teachers lacked the confidence to be original and often restricted themselves to MCQ's in formulating the assessments. There were contradicting views while considering essay/subjective type questions in online assessments; where some believed that typing provided more flexibility with editing and highlighting in answers others stated that they were unable to type fast and were unable to concentrate while typing.
Another research from Nardi et al [13] conducted studies where they compared students' performance and satisfaction between PBT and CBT, wherein CBT, the students were allowed to use their own devices. The majority of students preferred CBT using their own devices and demonstrated a high level of self-efficacy in CBT. The students had some concerns regarding the CBT, mainly fear of missing out important bits of information while scrolling, or missing unsaved data due to network issues. It is discussed that a high number of students preferred PBT because of familiarity with using paper, and a low level of self-efficacy with digital devices. In such cases, having the option to use your own device showed an increase in the confidence of students towards CBT. According to the participants of this paper, the vast majority of them liked the immediate feedback that was given. More than half of them liked the ability to modify answers till the end, while more than a third felt that the ability to monitor time till the end was beneficial.
To have an effective transition from a paper-based test to online assessments Geoff et al [14] suggest a stepwise learning progression. The quality of assessment questions must not depend on what is technologically possible but what is educationally desirable. Computers can change the difficulty of the next question based on current student performance, which is not possible in paper-based tests.
Gu, Chen, Liu et al [15] explored the benefits the students felt from various CBT systems in-depth, and developed a web-based education platform named Online Teaching Assistant System (OTAS). Some key features of OTAS were (1) Only 5 questions in a single quiz. (2) The questions are memory-based, and are dependent on only one key piece of definition or fact. (3) All questions are single choice, to reduce confusing situations. (4) The answer to each question is released as soon as it is attempted. (5) The questions are extracted randomly from a large dataset of pooled questions. According to their tests conducted on 100 students, for the subject Systems and Security, they have analyzed that there is a significant improvement in scores when using OTAS, as compared to not using it. Additionally, knowledge of basic concepts is highly correlated to an improved score when using OTAS.

About course -multimedia systems
Multimedia Systems (MS) is a technical elective course of School of Technology Management and Engineering for Bachelor of Technology (BTech) program. Multimedia Systems (Subject code-BTCO08009) course consists of 11 modules with 5 precise course outcomes. To learn Multimedia Systems as a course as a course, students need to learn Operating Systems, Computer Networks, and Digital Communications as prerequisite courses. Since it is an elective course, 55 students have opted for this course during the cohort of the year 2019-2020. The Course Outcomes (CO) of Multimedia Systems are given below in Table 1  Use multimedia applications and user interface for effective animations. CO4 Understand hypermedia messaging and its internet applications. CO5 Recognize the applications of distributed multimedia systems.
The bold rows in Table 1 above are the course outcomes covered in Online term test of MS course. The selective curriculum kept for the test is tabulated in table given in the next section.

Curriculum and assessment
The curriculum of Multimedia Systems covers study of basics of Multimedia Systems, file formats, Compressions and Decompressions schemes, Input-Output Devices. The advance topic of Multimedia System Includes Distributed Application of multimedia, Securing Multimedia Applications, Multimedia Authoring Systems etc. The Table  2 given below is the selective topic covered during the test. The assessment of the course consists of Internal Continues Assessment (ICA) and Term End Examination (TEE). The ICA consists of three tests which covers entire syllabus of MS. Another assessment tools of ICA like Assignments Lab work, Case Study and Viva Voce constitutes 50 marks of Internal assessments. Due to COVID 19 outbreak, the Term End Examination (TEE) was cancelled. The entire weightage was given to ICA.

Term test paper
The above case study-based paper is tested over an assessment of term test paper which is taken during COVID 19 pandemic. The paper consists of 15 marks in which 7 marks were allotted to Multiple Choice Based Questions. Total 7 questions were asked carrying 1 mark each. Another question was cased study subjective question which carries 8 marks. The question was purely opinion based. Students were allowed to use internet for possible answers and give the right feedback to the problem given in the case study. The time duration given for Multiple Choice Based questions was 10 minutes and for subjective Questions was 40 minutes. The total marks allotted for test was 15.
The snapshot of the question is given in following tables.

Experimental setup
The part I of the paper (Multiple choice-based question) was taken using Google forms (Quiz Mode). The setting of google forms is made in such a way that no single student can appear the test twice. Every registration of the test is authenticated by unique email id of the student. Part II of the paper (Case Study based question) was conducted through google drive in which the drive link is created and every student was asked to upload their answers in the drive link.
The conduction of the test was the biggest challenge as it was conducted on online mode. Students were appearing for the test from their respective home. Before conduction of the test following advisory was distributed among the students through WhatsApp group which was created for the purpose of the test.
1. The MCQ test duration will be for 10 minutes. The timestamp will be noted by the instructor through google forms. The submission of the test will not be acceptable if student submit the answers after the allotted time. 2. Continues monitoring will be there on the students through WhatsApp group. No discussion shall happen on the group during the test. 3. For case study-based question, students are allowed to use internet and other websites. 4. After complete conduction of the test, the drive folder will be locked so that no students will upload their answers.

Results
The online test paper was evaluated against a sample of 45 students. Evaluation of the student was done in two types of assessment as per [20]

Direct assessment
As stated in the previous sections, the test consists of two parts viz. part A and B. Part A consist of MCQs and part B consist of Case study-based question. The test marks are also compared with the classroom-based test which was conducted earlier with the same course. The results show that the online computer-based test outperforms with good results and the overall results of the students are increased. The figure 2 gives visualization of the evaluation of each student and its marks for Computer Based Online Test and Classroom-Based Test.
In the graphs shown below (Figure 2,3), we have discarded the students' record who were absent (AB) for the classroom-based test. The figure shows the difference between the Computer-based test and Classroom-based test. The graph shows that the number of students who scored more than 60% score in the course MS in both the mode (Computer-based test and Classroom-based test).

Fig. 2.
Comparison of no. of students who scored more than 60%

Indirect assessment
As stated in the previous session, indirect assessment consists of an online questionnaire about the feedback CBT which was forwarded to the students at the end of the test. Total 39 students participated in the feedback, with results in Table 3. The table shown above (Table 3) is the percentage response of samples on the scale from strongly agree to strongly disagree. It is clearly observed that on the scale of interestingness (question 1), 20.51% samples strongly agree to MCQs wherein 53.84% samples are strongly agreeing to Case Study based question. On the scale of interactiveness (question 2), 25.64% samples strongly agree to MCQs wherein 50% samples strongly agree that case study-based question is the best method. On the scale of acceptability (question 4) 33.33% samples strongly agrees that MCQs are best method wherein 46.15% samples are strongly agreeing for case study-based questions.
The following information is plotted on graph below (Figure 3). Check this fig.   Fig. 3. Bar Graph of students' feedback through indirect assessment

Discussion
The paper doesn't touch any new pedagogical approach, neither it assesses the students through innovative style of assessment. Due to COVID 19 pandemic lockdown, classroom teaching and classroom-based tests were prohibited as per the government directives which were applicable to all the government and private universities across India. Since the situation has put some restrictions and impacted on teaching-learning, we have experimented on test-taking strategy and presented our experience on the same. The paper does not come out on specific conclusion rather it shares an experience on the CBT. The authors of this paper consist of students and instructors and hence we would like to present discussion as a different perspective and we would like to keep the discussion open for the researchers.

5.1
Perspective of instructor i. Experimental: The test conducted in the pandemic may not be an innovative attempt but surely it was more experimental. In a regular commencement of a program, teacher used to online MCQ based test for 15 marks. Most of the student found it boring as for all the course of engineering, MCQ based test is not a good option [24]. Secondly MCQ based test is not covering all the aspect of engineering course. Engineering study is more about solving a complex problem [25], MCQ based test therefore doesn't cover the complex methodology of solving problems. ii. Feedback from students: In the feedback forwarded to the students, most of the students commented about the joy of giving exams rather than greed to get good marks. One the student who wrote-"I feel after solving a case study, doesn't matter how many marks I get, I am always curious to find what is the right way to solve this/ what should be the approach. This is why I feel an ideal answer to that case study should be released for everyone. People who solved it wrongly/partially can therefore know what the right approach is. Now I know there are various ways to solve a thing out of which I feel one approach should be given out as an ideal answer." The principle of outcome-based education is not achieving marks, rather it is about skills. It is also about getting knowledge and applying the same into real-life problems [26]. iii. Less probability of malpractice: Online and home-based test often have a chance of malpractice. The test which is conducted as a purely real-time problem based wherein usage of internet to search relevant solution was permitted. Rather for good answers, students are desired to search possible solutions from internet. iv. Time consuming for assessment: It is found that the above case study-based CBT takes consumes lot of time to evaluate the answers written by students. Evaluation must be done on empirical manner wherein all the answers need to read and assess on different scale and viewpoint. Such type of question may not have right or wrong answer. On the other hand, MCQ based test is very easy and online software makes the evaluation task very easy. v. Assessment viewpoint: it is also found that instructor gave more marks to the students whose viewpoints are matches with the answers written by students. Therefore, students scoring fewer marks in the case study-based question may not have wrong answers or fewer quality answers, It may be because of lower attainment of the viewpoint that student has with the viewpoint of the evaluator. Peer assessment [27] can be one potential to this problem.

Perspective of students
i. Doubt solving online -It was very convenient for students to be able to ask doubts to the faculty directly online. Moreover, it ensured that the other students are not disturbed by these doubts. ii. Changing applications midway -A downside of the above approach was that students had to constantly change applications midway if any new instructions were given by the faculty during the test. This broke the mental flow that the students were in while giving the test and in older, slower phones, this could result in significant lag. iii. Understanding based -One aspect of case study based online test that most people liked was that it was completely based on the understanding of the topic, rather than rote learning. This approach ensured that students paid more attention to the practicality of what was taught in class and learned how to apply it than just memorizing the definitions.
iv. Unclear Expectations -One major downside of a case study-based examination is that it isn't clear what is expected in an answer. There are always many ways and opinions to approach a particular problem, and students who conform to the opinions of the professor tend to get better marks. Even when that isn't the case, it becomes tedious for a professor to draw the line between a practical approach to a problem and something that is impractical/controversial. This issue, however, can be solved by the way of having well-defined rubrics of assessment and incorporating assessment methods like peer review (as mentioned in the above section). v. Freedom to access the internet -This might seem counter-intuitive, as allowing access to the internet might mean that the students can just copy answers. However, a carefully designed test, like in a case study-based test would ensure that the students form an educated opinion by referring to various online sources. It also closely mimics the practical scenario where people have a case to solve and have access to the internet to research in-depth. vi. Relevance to the current scenario -Because the questions were related to the current pandemic of COVID -19, students already had an understanding of the situation, and could personally put themselves in the case study that they were assessing and as users of the solution that they seek to provide. This ability to put yourself in the shoes of the end-user is an important aspect when it comes to designing solutions. As an added bonus, people who didn't have a complete understanding of the current situation were able to fill in their knowledge gap because of this case study.

Conclusion and Future Work
The paper does not conclude the decisive fact from the results, it rather gives and overview and experience of conducting online. The authors of this paper neither claims that good marks guarantee the method of conducting online test was correct. The paper shares experience through direct and indirect assessment and discusses how student enjoyed giving test. Most of the time, the traditional pattern of exam does not test the thinking and analytical ability of the students. Students most of time mug up the things as a preparation of the test. The testing of memory instead of analytical skill and knowledge becomes important in today's world. Industry 4.0 requires skilled engineers rather that knowledge engineers. From the available results it can be seen that the students are more interestingly solving the case study as it engages them during the test. More than 50% students agreed that the case study was interesting and it was good experience to solve. The paper also depicts the fact of emphasizing open book exams/ resources/study material is the most important factor to test the skill of the student rather than checking their memory. Another fact that this paper concludes is the acceptability of online tests in India. Universities, management needs to be more focused now on online classes, evaluation rather than traditional classroom teaching. Private universities need to be more focused on giving training to the students in online mode. The unforeseen situations like COVID Pandemic was a trigger that in future more strategy on teaching learning need to be improvised to avoid interference. In future, the teachers need to be taught and trained in online teaching and assessment.