Comparative Regression Discontinuity and Regression Discontinuity as Alternatives to Randomized Controlled Trials for Estimating Average Treatment Effects
Evidence from the Benefit Offset National Demonstration, WP#2022-7
Download
Associated Project
Key Findings
The paper found that:
- Average bias from CRD and RD is generally below 0.02 standard deviations in absolute size for the groups of bias estimates we analyzed.
- Given the precision that may be needed to evaluate interventions like BOND, the standard deviation of bias (after accounting for sampling error) is nontrivial, generally between 0.02 and 0.07 standard deviations for the groups of bias estimates we analyzed.
The policy implications of the findings are:
- When designing and interpreting results from CRD and RD evaluations, it is important to note that both produce biased estimates suggesting that their results be interpreted with more caution than those from an RCT with similar standard errors.
- This bias appears to be larger in the presence of major non-linearities in the relationship between the running variable and the lagged outcome for CRD.
In this paper we use data from an evaluation of the Benefit Offset National Demonstration (BOND) to evaluate the efficacy of using comparative regression discontinuity (CRD) and regression discontinuity (RD) relative to a randomized controlled trial (RCT). BOND is a large demonstration intended to promote return to work among people with disabilities who receive Social Security Disability Insurance (DI). RD is known as a relatively rigorous non-experimental method but produces imprecise results that apply to small populations. CRD is a promising enhancement that addresses these issues. The CRD and RD methods are potentially attractive because they can be used in contexts in which RCTs are challenging or infeasible. However, the bias of findings from CRD and RD studies is unknown in the context of DI. In this paper, we estimate CRD and RD models using simulated assignment to the BOND treatment group based on the duration of DI receipt at the start of BOND. We compare the CRD and RD estimates to RCT estimates. While the findings are not intended to revise the well-established evidence evaluating BOND, they can be used to help interpret the results from CRD and RD studies on other income support interventions for people with disabilities and to inform future study designs.
More like this from Mathematica
How do you apply evidence?
Take our quick four-question survey to help us curate evidence and insights that serve you.
Take our survey