In randomized control trials (RCT), the recent focus has shifted to how an intervention yields positive results on its intended outcome. This aligns with the recent push of implementation science in healthcare (Bauer et al., 2015) but goes beyond this. RCTs have moved to evaluating the theoretical framing of the intervention as well as differing implementation effects. One example of a typical mediation design in education research is the 3-2-1 mediation design (Pituch et al., 2009) with random assignment at the school level, the mediator at the teacher level, and the outcome at the student level. In such situations, it is not uncommon for the mediator to be measured at multiple time points across the intervention period. However, the current mediation models are not equipped for a longitudinal mediator that does not measure growth and where the outcome measure is only measured once in a 3-2-1 model. This dissertation has three primary goals. The first is to provide a framework to answer research questions, such as the mediating effect of teacher practices on an intervention’s impact on student achievement. In this situation, the mediator is measured at multiple time points in a 3-2-1 design, which current methods are not equipped to answer. The second goal is to provide potential estimation methods for the framework and to evaluate their bias and power. The final goal is to understand how these estimation methods perform in an actual study.Again, this study combines multilevel mediation with a latent state-trait (LST) framework to provide a model that can answer mediation questions when the mediator is at level 2 (teacher level) in a 3-level design and is measured at multiple time points. It also provides four different estimation methods (averages of the summed mediator, averages of factor scores, factor scores from the LST model, and the fully specified model) and the assumptions required for the four methods. These assumptions include assumptions on restrictions of the latent structure, measurement error, and the presence of state vs trait variances. It then uses a simulation study to evaluate the four different methods under varying design conditions: sample size, factor loadings, and effect sizes. Finally, this study investigates these methods in a project-based learning (PBL) science intervention study (Crafting Engaging Science Interventions [CESE]; Schneider et al., 2022). The results of the simulation study show that the choice of measure for the mediator is critical in reducing bias and increasing power in the estimation of the multilevel LST mediation model. Mediators with low construct validity will lead to bias across estimation methods. These might be mediating measures that are not truly measuring the mediator, have small factor loadings, or otherwise, the variance in the mediator is not explained by the proposed underlying factors. Additionally, mediators with more time-specific variance than trait-specific variance also lead to more bias across the estimation methods. These are situations where the time point explains more variance than the level 2 (teachers) general trait. If the time points are teacher practices in a given class period, this would be the situation where the teacher’s practices vary widely from day to day and not from teacher to teacher. The simulation also indicates that the sample sizes required for such research questions are large (>200). Following the simulation study, the methods were evaluated in the CESE study, a cluster randomized control trial with 61 schools, 102 teachers, and 4238 students. The CESE intervention included professional learning for the teachers, 3 NGSS-aligned units (in either chemistry or physics) with driving questions and hands-on experiences for the students, and NGSS-aligned end-of-unit assessments. During the intervention, as part of data collection, a random sample of teachers was observed 1 to 5 times, and their PBL practices in the classroom were scored. Mediation in this study aimed to understand how the intervention affected teacher PBL practices in the classroom and how those practices directly affected student science achievement at the end of the study. The results of the estimations of these mediation effects indicate that the models can converge and provide results; however, this empirical study reiterates the findings from the simulation study of issues with small sample sizes and low trait-specific variances. Investigating these mediation effects for the CESE intervention also raises several additional design considerations for mediation research questions, such as the effect of using raters, confounders, and the design of the mediating measure (in this case, the observation protocol).
Read
