Data Visualisation Literacy in Higher Education: An Exploratory Study of Understanding of a Learning Dashboard Tool

The visualisation of data has become ubiquitous. Visualisations are used to represent data in a way that is easy to understand and useful in our lives. Each data visualisation needs to be suitable to extract the correct information to complete a task and make an informed decision while minimising the impact of biases. To achieve this, the ability to create and read visualisations has become as important as the ability to read and write. Therefore, the Information Visualisation community is applying more attention to literacy and decision making in data visualisations. Until recently, researchers lacked valid and reliable test instruments to measure the literacy of users or the taxonomy to detect biased judgement in data visualisations. A literature review showed there is relatively little research on data visualisations for different user data literacy levels in authentic settings and a lack of studies that provide evidence for the presence of cognitive biases in data visualisations. This exploratory research study was undertaken to develop a method to assess perceived usefulness and confidence in reporting dashboards within higher education by adapting existing research instruments. A survey was designed to test perceived usefulness, perceived skill and 24 multiple-choice test items covering six data visualisations based on eight tasks. The study was sent to 157 potential participants, with a response rate of 20.38%. The results showed data visualisations are useful, but the purpose of some data visualisations is not always understood. Also, we showed there is a consensus that respondents perceive their data visualisation literacy is higher than they believe their peers to be. However, the higher their overconfidence, the lower their actual data visualisation literacy score. Finally, we discuss the benefits, limitations and possible future research areas. Keywords—Data, Perceived Usefulness, Visualisation Literacy, Overconfidence, Reporting Dashboards. iJET ‒ Vol. 15, No. 17, 2020 115 Paper—Data Visualisation Literacy in Higher Education: An Exploratory Study of Understanding of ...


Introduction
One of the important pursuits of Information Systems (IS) has been using data to support decision making. This is often known as Business Intelligence (BI) which includes the infrastructure, tools, applications and best practices to enable access and analysis of information to improve decisions and execution [1], [2]. With advances in computing power Business Intelligence & Analytics (BI&A) has evolved from the active accumulation, storage, and processing of data in data warehouses into the vital ability to consume and interpret data. As a result, reporting dashboards have grown in popularity, in order to help to provide the right information to the right people at the right time [3].
Data visualisations communicate an increasing amount of complex data. Choosing the correct type of data visualisations to reveal the right information through data visualisation technologies is an important skill [4]. Our data visualisation literacy skill must be measured to understand our capability, realise our limitations and develop our ability to improve our information gathering and decision making.
Data visualisation is complex, and there are many types. How a data visualisation is perceived is affected by various factors, including education, politics, personal experience, usefulness, skill and cognitive biases [5]- [9]. As a result, due to time and resource restrictions, this study is limited to an assessment of data visualisations relative to perceived usefulness, literacy and confidence in authentic settings.
Subjective experience and research related to the user acceptance of information technology are used to measure the perceived usefulness of data visualisations. There are many examples of how Information Technology (IT) improves performance [10]- [12]. However, there are limitations to improvements in performance by the user's reluctance to accept and use the Information Systems (IS) available [13]. Several theoretical models have been unified to account for up to 70% of user acceptance of IT and include Davis's [5] Technology Acceptance Model (TAM), which contains the validated perceived usefulness measurement scale [14].
Research on data visualisation literacy is relatively new. Advances in how to measure data visualisation literacy included grammars, graph comprehension, frameworks, models, question items, visualisation tasks and test item difficulty and discrimination indexes [15]- [20], [7]. These developments enabled the creation of a visualisation literacy assessment test for regular users of data visualisations that are validated and reliable.
Researchers have identified biases in data visualisations related to representativeness, availability, adjustment, anchoring, uncertainty, framing and hyper-confidence [21]- [24]. The role of the person reading the data visualisation has two parts (1) Skill, which is the capability of the person to effectively complete essential tasks, and (2) Behaviour, which is the way the person chooses to read data visualisations. One of the biases that impact data visualisation is overconfidence.
The research field in learning dashboard research is relatively young, while the use of data visualisations is comprehensive [25]. Moreover, little research has been undertaken to study how learning dashboards are being used. In our case we wished to study a specific custom developed learning dashboard as part of a reporting tool used by academic staff in a higher education institution in Ireland.
The study had four research questions: 1) What is the perceived usefulness of the academic performance dashboard visualisations? 2) What is the perceived data visualisation literacy of users? 3) What is the actual data visualisation literacy of the users?
The third question is to measure the actual data visualisation literacy of the users by using an adapted version of Lee, Kim, and Kwon [7] Visualisation Literacy Assessment Test (VLAT) modified to the data visualisations in the reporting dashboard. (4) What is the perceived data visualisation skill versus the actual skill of users?

Methodology
The current study uses both quantitative and qualitative approaches, a review of Saunders et al.'s [26] choices selected the mixed method option as the correct match for the research design. A research instrument is created is the form of a survey by adapting the following measurement scales and assessment tests. The survey consists of two perceived usefulness questions, two perceived skill questions and 24 multiplechoice test items covering six data visualisations and eight tasks.
There is a large body of IS research related to the "How" and "Why" people accept new information technology [27], [28]. The perceived usefulness and perceived ease of use objective psychometric measurement scales developed by Davis [5] are reliable and valid research instruments for user acceptance. The perceived skill questionnaire consists of two questions to assess participants perceived skill compared to their actual skill to determine the level of over-confidence [29]. The test has been altered to assess data visualisation literacy. It contains two questions. The first question determines the participant's perception of their peer's data visualisation literacy skill. The second question assesses their perceived skill level. iJET -Vol. 15, No. 17, 2020 The Visual Literacy Assessment Test (VLAT) consists of 12 data visualisations and 53 multiple-choice test items covering eight data visualisation tasks (Fig. 1). Each of the 12 data visualisations has associated cognitive tasks based on a review of task taxonomies and dataset types. The tasks are appropriate for data visualisations and dataset types [7].
We adapted a subset of the VLAT to map to the visualisations of the learning dashboard we planned to study. adapted test has a total of six visualisations and 24 multiple-choice test items covering eight data visualisation tasks (Table 1).

Fig. 2. Comparison of Data Visualisation in the VLAT and Learning Dashboard
The questionnaire posed questions on perceived use questions, one open-ended for thematic analysis and one matrix question containing Davis et al. [30] modified perceived usefulness measurement scale. Two adapted perceived skill questions establish participants own perception of their visualisation literacy compared to their peers. Twenty four questions covering Lee, Kim, and Kwon [7] customised visualisation literacy assessment test measured participants data visualisation literacy skill by testing them on set tasks. Participants who self-reported as colour blind were excluded as the data visualisations do not use colour blind safe colours [31], [32].
The quantitative elements of the survey were selected from research instruments that are previously tested for content and construct validity and reliability [5], [33], [7]. Responses to the questionnaire were collected during April 2019 from 157 potential participants. There were 32 responses to the survey (20.38%).

Results
Question one attempted to uncover the perceived usefulness of the visualisations in the dashboard through thematic coding analysis of unstructured user responses [34], [35] and data obtained from the psychometric measurement scale by Davis [5]. Qualitative software tool QDA miner lite was used to analyse the text from the open-ended question and the FreeMind tool to chart and sort codes into overarching themes. To support the analysis of the text in the open-ended question, the frequency and the generality of the codes was assessed by identifying the number of codes and the number of respondents mentioning a code. The analysis identified two prominent codes: "useful" and "unhelpful". There was a general perception the data visualisations are interesting and very useful to programme directors. The charts participants found most useful were "box plot", "results banding", "pass/fail ratio", and "course comparative difficulty".
The frequency and generality of the word "unhelpful" and its synonyms was most common (n = 7). Therefore, 37% of respondents referred to how unhelpful the data visualisations are in the dashboard when referencing unhelpful in the context of the graphics, data, use, and users. There is a general perception that some data visualisations are less relevant than others. Also, some newer staff find the data visualisations challenging to use.
The psychometric measurement scale by Davis [5] accounts for approximately 35% of the variance of predicting intention and consists of six independent variables [14]. Initial analysis of the results found that most respondents found data visualisations useful in 72.22% (N18, 16.67%+55.56%) strongly agreeing or agreeing. Just over 61% (N18, 27.78%+33.33%) of respondents also feel visualisations enabled them to complete tasks faster, while 50% (N18, 27.78%+33.33%) believed it makes their job easier. Further analysis provides additional results with 27.78% (N18, 16.67%+11.11%) of respondents thinking data visualisations do not increase their job performance with the majority of 55.56% undecided. While 16.67% of respondents believed visualisations did not increase their productivity or effectiveness with 50% undecided.
Overall, opinion seemed to be divided about the perceived usefulness of the dashboard data visualisations with most respondents (N18, 13.89%+30.56%=44.44%) strongly agreeing or agreeing. The biggest single group were undecided about the perceived usefulness of the dashboard (N18, 38.89%), (Mdn=8) or undecided. The Inter-Quartile Range (IQR)=1, a relatively small IQR is an indication there is a strong consensus they are unsure about the dashboard data visualisations use. Approximately one-sixth (N18, 6.48%+10.19%=16.67%) strongly disagreed or disagreed with the use of the data visualisations in the dashboard.
Question two attempted to uncover the perceived data visualisation of users. Participants assess their own perceived data visualisation literacy related to their peers by using a method adapted from Moharrer's [29] study to position their peer's data visualisation literacy skill on a five-point scale and to position their own data visualisation literacy skills on the same scale.
The perceived peer skill score has a possible range from one "Well Below Average" to five "Well Above Average". The median peer perceived skill was "Average" (Mdn=3). The self-perceived skill score has the same range from one to five. The median self-perceived skill is above average (Mdn=3.5), indicating that typically, respondents consider their perceived expertise higher than their peers.
Further evidence to support respondent's perceived skill as higher than their peers is provided by calculating perceived confidence (PConfidence). PConfidence is meas-ured by deducting the peers perceived skill from self-perceived skill. Using the following method to differentiate the magnitude of the perceived confidence score defines the level of confidence. A score of zero indicates confidence is at par with their peers, minus one is a confidence level below their peers, and one is one confidence level above their peers. The descriptive statistical analysis calculates the median as zero (Mdn=0) indicating participants confidence is the same as their peers. The range is three, meaning a swing of between three confidence levels from their peers. An IQR of one (IQR=1) shows a consensus that perceived confidence is between zero to one. Therefore, respondents perceived their confidence is above their peers.
Questions three attempted to uncover the actual data visualisation literacy of users via a test. The adapted VLAT was created to measure the actual data visualisation literacy of users and contains six data visualisations with 24 multiple-choice items covering eight data visualisation tasks. To address the issues of guessing and to understand the performance of the test items the question items were corrected for guessing (CfG) and analysed related to classical test theory (CTT): basic statistics, item difficulty index and item discrimination index [7]. Participant's actual skill was assessed with a Visualisation Literacy Assessment Test (VLAT). The VLAT raw scores were corrected for guessing (CfG), [7]. For this study, six of the eight visualisations (75%) in the dashboard are tested using related question items in an adapted version of the VLAT. The maximum score available is 24. The raw scores of the respondents ranged from 12 to 24. The median score is 19.05, and the Standard Deviation (SD) is 2.81 indicating most scores are close to the mean and follow a normal distribution. The CfG scores range from 11.5 to 24. The mean is 18.20, and the SD is 3.08. After adjusting the raw score to the CfG score, the scores dropped an average of 0.85 of a point. As a result, the spread of the CfG results is corrected from a skewness of -0.543 to -0.121 with most observed values less than the median indicating reasonably symmetrical data to a normal distribution.

Line Chart
Line The results of the question items are corrected for guessing (QCfG). The item difficulty index (p) was calculated based on the number of test-takers who answered the item correctly [7]. The questions are ordered as easy (< 0.85), moderate, or hard (> 0.5). Among the 24 questions, there are 15 easy, six moderate and three hard. The average item difficulty index is 0.79.
The QCfG results of the adapted VLAT assessment test were measured using Pearson's correlation coefficient to assess the strength of the relationship for question difficulty, shows (r = 0.999, n = 24, p < 0.01) confirming a strong positive association (r > 0.7). The higher the item difficulty index score, the less difficult the question, and the higher the score.
The item discrimination index (D) differentiates between high and low scored testtakers [7]. The items are ordered as high (>0.3), medium (>0.1and <0.3) and low (<0.1). Among the 24 items, there is one high item, ten medium items and ten low items. Three questions have a negative discrimination index indicating low performing respondents are likely to get this item correct. Therefore, the question should be carefully analysed and changed or deleted. The average item discrimination index is 0.09, indicating low item discrimination.
The item difficulty index (p) results of the adapted VLAT assessment test are measured using Pearson's correlation coefficient to assess the strength of the relationship for item discrimination (D), shows (r = -.750, n = 24, p < 0.01 ) confirming a strong negative association (r > -0.7). Therefore, the higher the item difficulty index or easier the question, the lower the item discrimination index, meaning low performing respondents are likely to get this item correct.
When assessing the reliability of Lee, Kim, and Kwon [7] original VLAT, coefficient omega shows good reliability (W=.76), which indicates the tests are consistent and controlled for random error [36], [33]. The adapted VLAT test for this study needs to be reliably validated.
Calculating the perceived skill versus actual skill was achieved by assigning the VLAT CfG score to an actual point relative to the respondent's peer group. Then deducting the self-perceived score from the actual point, which results in a measure of over or under-confidence.
The 18 VLAT CfG test scores are used to calculate the actual performance [37], [38]. The lowest score is rated one, and the highest score is rated five. Based on this assumption, each participant receives a point for their performance, between one to five, comparatively. The results for the VLAT actual skill are shown in (Table 2).
After compiling the results, the final step was to deduct the actual skill point with the self-perceived skill score. Participants with higher marks for perceived skill compared to their real ability were overconfident (positive value). Participants with a lower value of perceived skill compared to their actual skill were underconfident (negative value). A value of zero indicates a neutral score signifying the respondent is neither under nor overconfident.
Differentiating the magnitude of the actual confidence (AConfidence) score defines the level of confidence. By assigning the word low, to values between -0.99 to zero and 0.99; Moderate, to values between -1.00 to -1.99 and 1.00 to 1.99; High, to values over -2.00 and 2.00. Different levels of over and under confidence can be determined.
The descriptive statistical analysis calculates the median as underconfident (Mdn= -0.1) indicating on average participants are cautious about their visualisation literacy skill, or their perceived skill is less than their actual skill. The range is 3.48 with an IQR of 1.7 showing a broader consensus the actual confidence in data visualisation literacy skills is between "Low Under Confidence" to "Moderate Overconfidence". Signifying that when reading data visualisations, some respondents are unsure of their abilities while others are overconfident. The remaining scores lie within the first and fourth quartile varying between "Moderately Under Confidence" to "High Overconfidence", representing some participants show moderate levels of caution and others have higher levels of overconfidence when reading data visualisations.
Further analysis of results between the AConfidence score and the VLAT CfG scores measured using Pearson's correlation coefficient to assess the strength of the relationship shows a strong negative association (r = -0.595, n = 18, p < 0.01). The more under confident a respondent, the higher the VLAT CfG score or, the more overconfident the respondent, the lower the VLAT score. Therefore, overconfidence results in poorer performance. These results support previous research into the effects of confidence on performance [39]- [42].

Discussion and Conclusion
The main contribution of this research to the existing literature is that it adds to research on data visualisations for different user data literacy levels in authentic settings (in this case through academic analytics dashboards), an area in which there is negligible research. The findings may influence further research in this field and provide data visualisation stakeholders with insight into how their data visualisations may be developed to account for perceived use, literacy levels and overconfidence [30], [43], [44], [14], [45], [7], [8]. This may have practical applications such as implementing periodic reviews of reporting dashboards to ensure stakeholders understand the purpose of the data visualisations, to have the ability to read the data visualisations and limit the impact of subconscious biases when making decisions.
Future research could address the limitations in the design of the questionnaire and update the tests [14], [46]. While the research is exploratory, a larger research project could introduce additional data visualisation types and cognitive tasks, to test perceived versus actual data visualisation literacy and assess the statistical significance of the results in authentic settings. Other scenarios can be developed to examine how the limits of human reasoning affect visual data analysis [8]. The development of assessment tests is an important step [7], [45]. These tests can be integrated into more extensive research projects to support assessing the data visualisation literacy capability by using diverse research instruments to identify different cognitive biases and build a body of research on this topic. There are many areas where data visualization has fruitful application in education such as for teaching subjects that may range from topics as diverse as mobile application development [47] to landscape design [48] and provide insights to how people learn in sophisticated digital environments [49]. Ultimately improved data visualisation literacy could help improve trust of users in the systems they interact with including educational ones [50] and help people make more informed and impactful decisions.