How Much Bias Results if a Quasi-Experimental Design Combines Local Comparison Groups, a Pretest Outcome Measure and Other Covariates?: A Within Study Comparison of Preschool Effects

How Much Bias Results if a Quasi-Experimental Design Combines Local Comparison Groups, a Pretest Outcome Measure and Other Covariates?: A Within Study Comparison of Preschool Effects

Published: Mar 12, 2020
Publisher: Psychological Methods (online ahead of print)
Download
Authors

Thomas D. Cook

Naixin Zhu

Alice Klein

Prentice Starkey

Jaime Thomas

This study examines when nonexperiments might substitute for experiments that are done in real-world settings in order to learn what works to affect some socially valued outcome. The study probes whether a similar result is achieved in an experiment and in a nonexperiment that lacks the randomly formed control group of the experiment but that has instead a nonequivalent control group. However, this nonequivalent group is locally chosen to the treatment group in hopes of reducing the size of initial group differences and to match on whichever policies and practices are set locally. Moreover, the nonexperiment in question has a pretest measure of the study outcome and a “rich” set of other multidimensional measures with which to model whatever differences exist after the nonequivalent treatment and control groups have been selected from the same local pool. The study shows that the experimental and nonexperimental estimates are very close if the nonexperiment is defined in terms of local comparison group choice, a pretest measure of the study outcome and a rich set of other preintervention measures. This result now needs replicating.

Follow the Evidence

Interested in the most current findings from Mathematica? Subscribe to our bi-weekly newsletter, Evidence & Insights, to stay up to date with the issues that matter to you.

Sign Me Up