Educational phenomena occur in multilevel contexts, such as students nested within classrooms and classrooms nested within schools. This multilevel structure is also reflected in the multi-stage sampling design and randomized experimental design by clusters in educational data collection and research design. The consequential challenge of dependent observations within clusters of each nesting layer is prevalently dealt with by Hierarchical Linear Modeling (HLM) in education studies. However, in many cases, the observed data's multilevel structure can be unidentified or misspecified that the complex multilevel data structure is partially presented. Thus, even with the advanced statistical tools, the estimated models with omitted clustering levels will still produce erroneous standard error estimates and result in either Type I or Type II errors that compromise and even undistort interpretations of educational mechanisms. Practical guidance is urgently needed for empirical research confronting this issue to judge and detect whether the estimated models are adequate in taking account of the clustering dependency. This paper contributes to investigate when a cluster level should be explicitly modeled but omitted and how much the standard error estimates would be biased. This paper examines these questions in settings of a true three-level clustered data structure, while a cluster level, either at the highest, middle, or the lowest level, is omitted in the estimated two-level models. The theoretical discussion of essential clustering levels in modeling due to multi-stage sampling design and randomized experiments by clusters is drawn on insights from Abadie et al. (2017) and Hedges and Rhoads (2011). The current study then derives corresponding mathematical formulas to quantify the standard error estimation bias for each level's predictors' estimated effect. These derived formulas are functions of the intraclass correlation coefficients and cluster sizes of the estimated and omitted cluster levels. Further, build on Frank, Maroulis, Duong, and Kelcey (2013), the current paper develops a sensitivity analysis framework with a scientific language to quantify the degree of statistical inferences robustness based on the clustering characteristics of the omitted levels of clusters. In each omitted clustering scenario at the lowest, middle, and highest level, empirical studies are provided as implication examples of the sensitivity analysis to demonstrate the potential inference robustness risks due to omitted clustering.
Read
