Bayesian Methods

Making Research, Data, and Evidence More Accessible

Bayesian research methods empower decision makers to discover what most likely works by putting new research findings in context of an existing evidence base. This approach can also be used to strengthen transparency, objectivity, and equity.

Using Bayesian Methods to Understand What Most Likely Works

For decades, researchers have relied on statistical significance tests as the basis for answering questions about the effectiveness of various policy programs and interventions. Because researchers believed that statistically significant findings are unlikely to be due to random chance, statistically significant findings were much more likely to be published in journals, reported in the media, and used to make important real-world decisions.

But statistical significance does not actually mean what people believed it meant. In 2016, the American Statistical Association (ASA) released an unprecedented statement about the misuse of statistical significance, explaining that scientific conclusions and policy decisions should not be based only on whether a p-value—or a measure of statistical significance—passes an arbitrary threshold. For example, negative side effects of a drug could be ignored because their p-value is a little larger than 0.05, which just misses the statistical significance threshold. An article in the scientific journal Nature, with over 800 signatories, argued for retiring statistical significance altogether. Statistical significance is too often easily misinterpreted to mean either “the intervention worked” or “the intervention did not work.” This artificially binary approach to interpreting study findings can mean that potential positive impacts that don't meet the threshold for being statistically significant are ignored.

A Paradigm Shift to Bayesian Methods Could Improve Evidence-Based Decision Making

On this episode of  On the Evidence, Mariel Finucane, John Deke, and Tim Day, of the Center for Medicare & Medicaid Innovation, discuss why there’s growing demand for alternative ways of assessing the impact of policies, and how a modernized, evidence-informed update to Bayes’s Rule can help decision makers assess whether a policy or program works and whether it would work for certain groups of interest.

Listen to podcast

On this episode of On the Evidence, Mathematica’s Mariel Finucane and John Deke join Tim Day of the Center for Medicare & Medicaid Innovation to discuss the application of evidence-informed Bayesian methods that not only confirm whether a policy or program works, but for whom.

Introducing BASIE: A Replacement for Statistical Significance Testing

BASIE (BAyeSian Interpretation of Estimates) is an innovative framework for using an evidence-based Bayesian approach to interpret traditional (non-Bayesian) impact estimates. BASIE, first introduced in 2019 by Mathematica experts John Deke and Mariel Finucane with support from OPRE, represents a substantial improvement over statistical significance. BASIE draws on prior evidence regarding the effectiveness of previously evaluated interventions to assess the probability that an intervention worked. With BASIE, researchers can avoid the pitfalls of misinterpreting statistical significance, as described in this JREE commentary.

With support from IES, Deke, Finucane, and Mathematica colleague Dan Thal developed a guide to help researchers and decision makers use BASIE to interpret findings from evaluations in the field of education. The guide provides step-by-step instructions, methodological details, computer code, and easy-to-use tools. Deke and Finucane also hosted a webinar on the BASIE Framework for Interpreting Findings from Impact Evaluations for the Society for Research on Educational Effectiveness (SREE).

The Next Step for BASIE: Bayesian Interpretation of Cluster-Robust Subgroup Estimates

In this journal article, Erin Lipman, John Deke, and Mariel Finucane demonstrate the value of enhancing BASIE to better understand which subgroups most likely benefit from efforts to reduce health care costs. By using Bayesian meta-regression analysis to interpret cluster-robust subgroup impact estimates, the authors show that their ‘best of both worlds’ approach produces more accurate and precise estimates, which can lead to healthcare cost savings in a plausible policy scenario.

Read More
 

Getting Accessible and Actionable Information to Those Who Need It

Imagine you’re a leader in your state’s human services agency, or a legislator in charge of investing resources in social services. With a limited budget, you’re eager to focus on programs that have a track record of working for the people you’re trying to reach. There’s an existing body of research, but it’s hard to interpret because studies vary both in quality and conclusions. Applying a Bayesian approach, we’re able to distill the existing evidence into a bottom-line assessment of the probability that something works.

  • We used a Bayesian model to assess how a health initiative known as Comprehensive Primary Care (CPC) impacted Medicare expenditures and health outcomes. The CPC initiative was a unique collaboration between the Centers for Medicare and Medicaid Services (CMS) and other private and public payers, including commercial insurers and Medicaid managed care plans, to help overcome barriers to better primary care. The research team compared the change in expenditures using both Bayesian and traditional models. Bayesian methods offer several benefits not found within traditional research methods, including more precise findings that incorporate prior evidence, adjusting for multiple comparisons, and framing results in plain language about the probability that the initiative would actually reduce Medicare expenditures.
  • Our Employment Strategies for Low-Income Adults Evidence Review (ESER) used a Bayesian framework to help HHS understand interventions that work best to improve employment outcomes and determine which are most effective under specific conditions and for which types of workers. Under more traditional research methods, the ESER database included few instances of potentially informative combinations of strategies and populations, so researchers were less able to determine precise and compelling conclusions using this approach. The Bayesian approach strengthens analysis of combinations of study features—outcomes, strategies, and populations—that appear only rarely in the data. Using Bayesian methods, we found that several interventions were effective at improving low-income adults’ labor market outcomes, including 10 that were highly likely to improve labor market outcomes by at least 5 percent.
  • Our research team used Bayesian estimates based on existing studies about the influence of education on civic engagement to measure the impact of a charter school program known as Democracy Prep on helping students become more active citizens (for example, visiting legislators, attending public meetings, and discussing literature on citizen engagement). We found a 98 percent probability that enrolling in Democracy Prep produced a positive impact on voter registration, and a 98 percent probability that enrolling produced a positive impact on voting in the 2016 election.

Bayesian Methods Better Support Equitable Evaluation

Cover image for Centering Equity in the Development of Federal Policies and Evaluations webinar

Play Video

There is an urgent need to better understand the effect programs and policies have on equity outcomes. Incorporating more equitable evaluation principles into social policy research helps ensure that all community stakeholders—including those who operate and participate in programs—have the chance to contribute to and benefit from the evaluation. Bayesian methods make equitable evaluation more likely by examining a wider range of outcomes for a greater diversity of communities. They can solve the problem of multiple comparisons (that is, examining multiple outcomes in the same analysis) without imposing artificial limits on what we can learn. We can also support community stakeholders’ suggestions to look at a broader set of outcomes or subgroups than originally specified in a study’s design.

These methods can also strengthen subgroup analyses by using information from related subgroups to increase the precision of each subgroup impact estimate. They challenge traditional research methods that are heavily reliant on larger subgroup sample sizes, which can fail to represent certain groups.