Moving Beyond Statistical Significance: The BASIE (BAyeSian Interpretation of Estimates) Framework for Interpreting Findings from Impact Evaluations

This brief describes the core tenants of an evaluation framework, known as BASIE, or BAyeSian Interpretation of Estimates. BASIE helps researchers interpret evaluation findings without misinterpreting statistical significance or sacrificing scientific rigor. Specifically, evaluators can calculate the probability that an intervention has meaningful effects by placing their impact estimate in the broader context of prior evidence. With BASIE, evaluators will continue to provide answers to important policy questions based on evidence, but now in a way that is more intuitive, better aligned to questions of interest to decision makers, and less susceptible to misinterpretation. The BASIE Framework has five components, which are summarized below:

1. Probability: In this framework, probability is a relative frequency, not the intensity of one’s personal beliefs.
2. Priors: Evaluators should draw upon earlier evidence, not beliefs, to inform the probability that an intervention has a meaningful effect.
3. Point estimates: Evaluators should report both the impact estimated using only data from the intervention AND the impact estimated using both data from the intervention and prior evidence (the ‘shrunken’ estimate).
4. Interpretation: Instead of misinterpreting p-values, evaluators should use prior evidence to calculate the probability an intervention had a meaningful effect.
5. Sensitivity analysis: Evaluators should assess the extent to which using different prior evidence affects the conclusions they draw about the impact of an intervention. This analysis is an important way of addressing the challenges associated with choosing an appropriate prior.