Once research questions are posed, researchers must answer many aPriori questions regarding research design before analysis can be performed and any conclusions can be made, including sample selection criteria, data collection method, model specification, analysis and estimation technique. The choices made by researchers along this forking path of possible specifications, such as model specification and estimation technique, can lead to varying results. On one hand, researchers that are not able to identify the best answers to these questions are faced with a list of plausible specifications, inherent uncertainty in resulting model estimates, and are tasked with how to best balance alternative specifications. On the other hand, researchers suffer from an “embarrassment of riches” in computational capacity, in that they have more computational power than what is reflected in most journal articles and that the amount of alternative analyses researchers could perform have expanded dramatically (Young, 2018). Although it is common to choose one set of specifications and report the resulting estimates in the absence of other specifications, this research proposes a framework called extended ensemble estimation that utilizes the alternative specifications to quantify and visualize the sensitivity of an estimated treatment effect. This paper also proposes a method to combine the estimated treatment effects across specifications into a single estimated treatment effect, weighted by precision. Along with the proposed methodology, this work contains a best practice guide for users in order to best understand the sensitivity of an estimated treatment effect within the extended ensemble estimation framework, a proposed method to update an estimated treatment effect by utilizing alternative specifications, simulated performance within common covariance structures, and a case study application regarding the effects of kindergarten retention on math and reading performance.
Read
