This study uses a multi-level multivariate propensity score matching approach to examine the synthetic cohort design (SCD) in estimating the schooling effect on mathematics proficiency of the focal cohort 2 (8th graders). Collecting 7th and 8th graders at the same time point, the SCD is sufficient in estimating the schooling effect under the historical equivalency of groups (HEoG) assumption.A structural equation modeling (SEM) framework is used to define the HEoG assumption. It has shown that HEoG assures that the use of SCD results in an unbiased estimate of schooling effect without randomized data. The post-hoc group matching is used to achieve the HEoG assumption in order to produce an estimate of schooling effect that is unbiased in SCD. Three matching approaches, level-1 matching, level-2 matching, and dual matching, are evaluated using simulated data generated based on USA participants of the Second International Mathematics Study (SIMS-USA, IEA, 1977).Two-level latent variable models based on situations that violated the HEoG assumption are created in order to examine the ability of matching to reduce the simulated selection biases to improve the accuracy of the schooling effect estimate in SCD. The three simulated situations involve hierarchically structured data, surrogate covariates with measurement errors, and omitted covariates.Results suggest the following: 1) To reduce initial bias and assure the HEoG assumption, three different matching approaches should be conducted on the covariates according to where the initial bias occurs: on level-1 covariates, on level-2 covariates, and on both level-1 and level-2 covariates; 2) When reliability is low (e.g, .25), latent variable matching does not help improve group comparability, but using observed surrogate variables to match can reduce bias by more than 50 percent. When reliability is high (e.g, greater than .75), latent variable matching reduces bias as much as matching on observed surrogate variables does; 3) When level-2 initial bias is large, increasing level-2 R2 does help to improve level-2 matching. The bias reduction of either individual or dual propensity score matching is not sensitive to the increase of R2 . The dual propensity score matching is more robust to the magnitude of initial selection bias, achieving a large bias reduction rate when the initial bias is small. Either level-1 matching or level-2 matching achieves lower bias reduction rate when the initial bias is small. This dissertation study provides a theoretical basis for future research to examine the effectiveness of propensity score matching in reducing the selection bias of SCD for casual inference and program evaluation. Practical considerations and suggestions for future research on hierarchically structured data in program evaluation are discussed.
Read
