Analysis of STACK Answer Data Using Pen-Stroke Data from a Calculation Notebook and Item Response Theory

Abstract

One of the most significant features of STACK, an automatic assessment system for mathematics, is the automatic classification of correct, semi-correct and incorrect answers using the Potential Response Tree, and the presentation of appropriate feedback for each. However, the answers provided by students are only the results of their calculations, and if they are semi-correct or wrong, it is necessary to check the notes describing the calculations to determine where mistakes have been made. We developed a function that allows students to submit their notes at the same time as their STACK answers, thereby creating an environment in which teachers can check where students have made mistakes and where they have stumbled. The note submission can be in the form of saving the results of calculations made with a digital pen on a tablet. This study examines a method of analysing answer data by visualising and analysing pen-stroke data from notes submitted using the latter method, and linking this to correct/incorrect data. From pen-stroke data, two types of visualisation were achieved on the submitted notes. One was to colour-code the strokes according to the writing speed of the digital pen, and the other was to display the strokes as thicker if the stagnant time in which nothing was written was above a threshold value. The difficulty of the problem was estimated from those data. Then, the item response theory is used to verify whether the difficulty is correctly estimated.