Measurement Sensitivity in Heterogeneous Treatment Effect Analysis: Evidence from the Project STAR Experiment

Mika Takechi; Shigeyuki Hamori

doi:10.3991/itdaf.v4i3.62410

Authors

Mika Takechi Yamato University, Suita, Japan
Shigeyuki Hamori Yamato University, Suita, Japan; Kobe University, Kobe, Japan https://orcid.org/0000-0003-1498-0188

DOI:

https://doi.org/10.3991/itdaf.v4i3.62410

Keywords:

Project STAR, Treatment heterogeneity, Outcome construction, Educational assessment, Class-size effects

Abstract

This paper examines how the construction and timing of outcome and heterogeneity measures influence the estimation of treatment heterogeneity in randomized educational experiments. Using data from the Project STAR experiment as a laboratory setting, we compare treatment-effect estimates across alternative definitions of student achievement and student heterogeneity. Baseline specifications using contemporaneous Grade 3 achievement measures suggest positive interaction effects between small-class assignment and measured student heterogeneity. However, these interaction effects become substantially weaker and generally lose statistical significance when heterogeneity is measured using pre-treatment achievement scores from kindergarten, grade 1, and grade 2. Similar patterns emerge across reading and mathematics outcomes as well as across alternative cross-domain specifications. The results indicate that conclusions regarding treatment heterogeneity are highly sensitive to the timing, domain, and construction of heterogeneity measures. More broadly, the findings demonstrate that measurement sensitivity represents an important issue in empirical data analysis of heterogeneous treatment effects. The results highlight the importance of robustness assessment under alternative measurement constructions before drawing substantive conclusions in experimental and policy evaluation studies.

References

[1] Imbens, G. W. and Angrist, J. D. (1994). Identification and estimation of local average treatment effects. Econometrica, 62(2):467–475.

[2] Krueger, A. B. (1999). Experimental estimates of education production functions. Quarterly Journal of Economics, 114(2):497–532.

[3] Heckman, J. J., Smith, J. A., and Clements, N. (1997). Making the most out of programme evaluations and social experiments. Review of Economic Studies, 64(4):487–535.

[4] Duflo, E., Hanna, R., and Ryan, S. P. (2012). Incentives work: Getting teachers to come to school. American Economic Review, 102(4):1241–1278.

[5] Cunha, F. and Heckman, J. J. (2007). The technology of skill formation.

American Economic Review, 97(1):31–47.

[6] Bond, T. N. and Lang, K. (2019). The sad truth about happiness scales.

Journal of Political Economy, 127:1629–1640.

[7] Angrist, J. D. and Lavy, V. (1999). Using maimonides’ rule to estimate the effect of class size on scholastic achievement. Quarterly Journal of Economics, 114(2):533–575.

[8] Athey, S. and Imbens, G. W. (2016). Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences, 113(27):7353–7360.

[9] Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. Econometrics Journal, 21(1):C1–C68.

[10] Wager, S. and Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):1228–1242.

[11] Knaus, M. C., Lechner, M., and Strittmatter, A. (2021). Machine learning estimation of heterogeneous causal effects: Empirical monte carlo evidence. The Econometrics Journal, 24(1):134–161.