The following blog post by staff at the Walton Family Foundation describes a partnership between the foundation and Mathematica. The post originally published on the foundation's website and is reprinted on Mathematica’s blog with the foundation’s permission.
For anyone who gives to help others, one big question inevitably follows: Have I made a difference?
This is true for individuals donating to a personal cause close to their heart or philanthropies seeking to solve big systemic challenges in society.
Sometimes the results are clear and intuitive. A local food bank expands to serve 500 people instead of 250. A community nature trail extends another mile of access – and more people use it.
But how do you gauge impact when the outcomes are more complex and difficult to measure? Like whether increased funding for school wellness programs produces higher graduation rates for underserved students. Or whether investments to make the arts more inclusive and housing more affordable help improve a community’s quality of life. Answers to these kinds of questions are elusive, and usually not straightforward.
As a learning organization, the Walton Family Foundation is committed to creating greater understanding of the impact our giving has on the people and communities we serve.
Our mission is to create access to opportunity for people and communities. So we are always looking for better ways to determine the successes (and failures) of the individual grants we make. We also seek new insights into the bigger picture. Are our philanthropic strategies helping move the needle on the issues we focus on and care most about?
One of the ways we’re exploring how to better assess philanthropic impact is through a new project with Mathematica. The foundation is working with the Princeton, New Jersey-based research organization to explore the potential benefits of using Bayesian approaches to sensemaking. We want to expand the value of the data we collect to draw more accurate and insightful conclusions about our investments to create social and environmental impact.
Bayesian analysis is named after Thomas Bayes. He was an 18th century Presbyterian minister, philosopher and statistician and a pioneer in developing probability theory. One use of Bayesian analysis is to draw on prior evidence to assess the probability that a specific action led to a specific outcome.
Bayesian methods can be used as a complement to traditional measures of statistical significance to draw conclusions about the likely cause of an event or outcome.
Most of us are familiar with statistical significance through exposure to media reports on political polls or studies on lifestyle habits. (For example, does drinking red wine improve health? Here’s a study that says it does, or doesn’t.) If a research study produces an outcome that is statistically significant, it’s considered likely that a particular action or intervention caused that outcome, rather than it being due to random chance.
Studies using statistical significance have long been used by decision makers to guide real-world actions about legislation, policy and regulations. They can impact our most personal everyday decisions.
But statistical significance is imperfect and often misinterpreted. It creates sharp, binary distinctions – a sort of a pass/fail line – in the interpretation of data. If something is considered statistically significant, it matters. If it’s not, it doesn’t. Real life is rarely so clear cut.
“Data is not a perfect representation of the world. It's kind of an imperfect, noisy representation of the world. It can be misinterpreted,” says John Deke, a senior researcher at Mathematica who specializes in Bayesian methods. “Describing data is not the same thing as figuring out the implications of data for things that you care about.”
For Deke, Bayesian approaches represent a more complete way to interpret data and evidence because they evaluate it from a broader, related array of sources.
As an example, Deke cites research looking at traffic studies in several communities evaluating the impact of allowing right turns on red lights. Studies done on individual communities found no statistically significant impact of allowing red-light turns. Taken by themselves, these small studies might alleviate concerns about whether red-light turns lead to more fatal car crashes.
But when all studies were taken together and reinterpreted in aggregate, it was found that red-light turns were less safe than originally believed.
“That’s a story of Bayesian statistics,” says Deke. “By combining new evidence with prior evidence, we achieve a more complete understanding of what’s going on.”
Another example? National data showed a statistically significant drop in test scores over the course of the COVID-19 pandemic. But test score declines in individual school districts that were similar in magnitude to the national decline were considered statistically insignificant due to the smaller sample sizes available in localized results.
This led some media outlets to mischaracterize these school districts as “bright spots” even though the data showed a decline in test scores no less than the national average. But researchers using Bayesian methods were able to determine that the observed declines in individual school districts were most likely real, given the broader national context.
An over-reliance on statistical significance can lead decision makers to ignore or dismiss data that point to a probable cause for an outcome. In the case of student test scores, it could obscure important performance trends for smaller subgroups of students in certain schools, raising questions of equity in how student programming is handled as a result.
What are the implications for philanthropy, which often relies on data from pilot programs or relatively small grant portfolios to evaluate impact? This question is especially important when we’re looking to scale evidence-based solutions to have the greatest benefit possible.
Examining evidence through a different lens – one that is less binary and encompasses more sources of evidence both past and present – could reveal hidden benefits (or drawbacks) of philanthropic funding for specific programs or interventions. Bayesian approaches to interpreting evidence can provide more context and inform better decisions in the future.
In our work with Mathematica, we are exploring the ways in which Bayesian approaches could be a useful complement to current methods when trying to understand evidence, particularly data gathered from traditional experimental methods and newer approaches to illuminate causal pathways.
To be a stronger learning organization, the foundation needs to build a better evaluation toolkit that creates a clearer picture of the world as it is. One that increases our ability to draw new lessons and actionable insights from the evidence we’re gathering about impact.
By creating space for a more nuanced understanding of the world we live in, we can make knowledge-based decisions, not purely data-driven ones. If we do that, we have a better chance of solving big problems over the long term.
