How Much Bias Results if a Quasi-Experimental Design Combines Local Comparison Groups, a Pretest Outcome Measure and Other Covariates?: A Within Study Comparison of Preschool Effects

This study examines when nonexperiments might substitute for experiments that are done in real-world settings in order to learn what works to affect some socially valued outcome. The study probes whether a similar result is achieved in an experiment and in a nonexperiment that lacks the randomly formed control group of the experiment but that has instead a nonequivalent control group. However, this nonequivalent group is locally chosen to the treatment group in hopes of reducing the size of initial group differences and to match on whichever policies and practices are set locally. Moreover, the nonexperiment in question has a pretest measure of the study outcome and a “rich” set of other multidimensional measures with which to model whatever differences exist after the nonequivalent treatment and control groups have been selected from the same local pool. The study shows that the experimental and nonexperimental estimates are very close if the nonexperiment is defined in terms of local comparison group choice, a pretest measure of the study outcome and a rich set of other preintervention measures. This result now needs replicating.