Improving Questionnaire Reliability using Construct Reliability for Researches in Educational Technology

—This paper is exploring on maneuver to improve research instrument reliability in scientific research related to Educational Technology by analyzing the traditional Cronbach’s alpha reliability using SPSS software and the newer statistical tool, AMOS using Construct Reliability (CR) approach. Two sets of data were used as sample to perform the comparison. The first set of data is from a research involving Technology Enhanced Learning Environment. The second data are sampled from research in digital competency. Find-ing from this paper concluded that, conventional approach of using Cronbach’s alpha have lower reliability than the newer approach of using CR. Using Cronbach’s alpha show tendency toward measuring consistency instead of reliability. CR offer better definition of reliability and give a robust measurement of reliability in research. This paper had shed light into offering alternative approach to the commonly and widely uses of research reliability especially when it involves questionnaire as instrument.


Introduction
Researches in educational technology are various in natures. The researches might be experimental, involving series of interventions to understand the effect of intervention. To understand the natural character of the samples without any intervention, questionnaire-based research is being widely applied. Reliability is a big issue for a questionnaire, without a solid reliability value the instrument deems invalid. Cronbach's alpha, also known as alpha coefficient, is widely used in educational technology, education as well as social sciences. As technology evolves, construct reliability which also known as composite reliability is taking place as a new reliability coefficient alternative. Yet, both coefficients generally returned a different value.

Problem Statement
The main concern for questionnaire-based research is, its instrument reliability. The popular application of reliability test is Cronbach's alpha (e.g. [1][2][3][4]). Calculating alpha is also simpler when compared to other estimates, as only one test is needed [5] and it is easier to be interpreted [6]. The formula for Cronbach's alpha is with  is the Cronbach's alpha coefficient, n referring to the number of questions in the instrument. Meanwhile, Vi is the variance of scores and Vtest is the total variance of overall scores. The recent development in educational technology research had gave researchers with the flexibility of adopting a new reliability test using CFA (e.g. [7,8]). The purpose of CFA is to understand the goodness of the researcher's factor model from the aspect of ensuring all items in the questionnaire are representing their respective latent variable as in the measurement model. By manipulating CFA, researcher can gain the value for its item construct reliability (CR), which is comparable to Cronbach's alpha. The formula for CR is CR is the construct reliability,  is factor loading and  representing the measurement error. Both tests are different statistically, yet somehow have the same function. Therefore, which reliability test is better and could give researcher a much solid needed instrument? Thus, two sets of data will undergo Cronbach's Alpha and CFA via CR value reliability test to find out which test give a better result.

Methodology
This research involves two sets of data labelled as Data Set I and Data Set II. The methodological approach for both data sets are:

Methodology for data set i
The same data was undergoing two different reliability tests. The data were originated from the authors' research on technology enhanced learning environment. Twelve respondents were sampled using random sampling technique for Cronbach's alpha test. Two hundred respondents were sampled using random sampling technique for measuring CR value using CFA. The sample size is based on the minimum size of sample for CFA research.
The internal consistency technique was applied for this purpose of measuring Cronbach's alpha. The reliability was found to be Cronbach's alpha = .878. The reliability details are as in Table 1. Relying on the Cronbach's alpha value, all the item reliability is excellent, and consistency was recorded to be very high. To make a comparison, the data then underwent CFA via Structural Equation Modeling (SEM) and measurement model as in Figure 1 was constructed. The measurement model had returned standardized regression weights as in Table  2. Data in Table 2 was later transformed into a square of factor loading, 2 and measurement error,  as in Table 3 and CR value for each construct is as in Table 4.

Methodology for data set ii
A set of data from the authors' research on digital competency was used as data set ii. The data also underwent two different reliability tests. Ten respondents were sampled via random sampling technique. Through internal consistency technique, the Cronbach's alpha was found at .868. The reliability values for each construct are as in Table 5. Items in Table 5 are highly reliable. Then, the data underwent the CFA test. For the first analysis, the model returned χ²df = 2.151, RMR = .026, CFI = .846 and RMSEA = .199 which not fulfilling the requirement of a fit model. Second analysis was conducted derived the value of χ²df = 1.318, RMR = .018, CFI = .960 and RMSEA = .109 that meet the minimum parameter of a fit model. Yet, item E7 indicating a factor loading .165, which is, less than the minimum factor loading of .5. To reinforce the finding, the value of Average Variance Extracted (AVE) and CR was calculated as in Table 6. The comparison of reliability test between Cronbach's alpha and CR from CFA shows a deviation. Both tests were conducted for the purpose of measuring reliability, yet it was found that using CR from CFA, the researcher will have much better reliability value. An instrument that was qualified as highly reliable using Cronbach's alpha has been dignified as unreliable using CFA. By using CFA, the reliability value returned is smaller in which gives more precise than using Cronbach's alpha.

Discussion
There are two ways of applying CFA commonly found in literature. In a number of researches, CFA was used for the purpose of assessing construct reliability and Cronbach's alpha for the purpose of measuring instrument internal consistency. While, some other research used CFA to validate their model and Cronbach's alpha as reliability test. Both approaches are well accepted by the scientific community.
Using CFA as a reliability test gives researcher with a much precision and smaller reliability value as reported in this study. Despite the fact that the coefficient Cronbach's alpha is the most widely used estimator for the purpose of reliability, it has been well criticized for being a lower bound that render the true reliability to be underestimated [9].
Cronbach's alpha requires the compliance toward classical item-score assumption, tau equivalency assumption and uncorrelated-errors assumption where, when a violation occurs, leading to negatively biased, relatively unbiased and positive biased coefficient alpha [7,10]. Cronbach's alpha is easy to be misinterpreted and is appropriate to interpret as an estimate of reliability according to the internal consistency between items [10]. Cronbach's alpha is also prone to the effect of test length [11] and has been subject to so much misunderstanding and confusion [12,13].
Helms et. al. [14] suggest researchers to calculate composite reliability rather than total-score reliability for the purpose of good practices in analyzing, interpreting and using reliability data. As Cronbach's alpha is based on total-score reliability approach, it is highly suggested for the researcher to use the construct reliability as did by [15]. It is suitable for future research improvement in educational technology such as technology enhanced learning environment and HOTS as did by [16].

Conclusion
Cronbach's alpha has been well accepted as a reliability test among researchers in educational technology as well as in other social sciences researchers. However, as verified by this study, Cronbach's alpha normally returning a higher reliability value when compared with the reliability value returned by CR. Despite the low value returned by CR, it is believed to be demonstrating a higher precision due to its composite nature. Literature had shown that Cronbach's alpha is subjected toward a number of assumptions that give negative effects when violated where in research, these assumptions is delicate to be confirmed. In addition, as Cronbach's alpha is being a lower bound, it is highly suggested for researchers to use CR as their reliability coefficient as a maneuver of intensifying questionnaire-based instrument reliability.