As a measurement and research specialist in education, I have often had colleagues refer to me as a “data wizard” or ask me to do my “magic” in developing a theory of change or measurement plan or set of analyses. As flattering as this can be, it points to an important hurdle to racial and economic equity in educational opportunity for our nation’s students. Developing hypotheses for how to create impact, testing them, and analyzing the results to inform improvements is not magic; it is foundational to the work we do together and, when it is done hand in hand with strategy and decision making, it’s called strategic learning.
We need to rethink how organizations learn and improve if we want to shift from isolated measurement and research experts working in parallel to and disconnected from strategy and decision making, to an integrated approach to building hypotheses, testing, and improving. Ultimately, this will enable us to do more with limited resources to support educational equity.
The clearest path to impact is through strategic learning.
In a complex landscape of educational systems, we have to make decisions and bet on approaches that will work to improve educational equity. These decisions and bets can be smarter and more likely to result in real change for students if we rely on a strategic learning approach that enables iterative learning and improvement over time. A strategic learning approach is characterized by a few key activities: clearly defining an organization’s efforts to set goals; establishing hypotheses on how to reach those goals; building strong organization culture, practices, and systems to test hypotheses over time; and using that information to improve hypotheses on an ongoing basis.
“Yes! Of course these things are all true.” So what’s stopping us? When it comes to implementing strategic learning, the first hurdle is a mindset that does not give space for failure. Establishing hypotheses, testing them, and improving requires fully embracing that we might be (and likely are somewhat) wrong about the best ways to achieve impact. It feels bad to know we’ll be wrong when educators’ and students’ lives are at stake. However, that doesn’t make being wrong any less of a certainty. Admitting that up front and creating processes to improve is a better approach than pretending we have it all figured out. It would also likely make us more open to and ready for evidence contrary to our approaches rather than only seeking out confirmatory evidence. For funders, this means we have to consider how we empower grantees to feel comfortable embracing that uncertainty and how we give time and space for grantees to improve. The power imbalance between funder and grantee and the onerous processes associated with grantmaking can make it difficult to allow success to mean anything other than an on-time delivery on promised impact. But empowering grantees also means funders need to hold themselves accountable to learnings derived from a strategic learning approach—that the results of this approach inform not just grantee decisions but also the funder’s. What more could we do if we shifted to a model of success predicated on collective learning and improvement that is iterative? What kind of change is required to shift this mindset at all levels of an organization?
Another barrier to implementing strategic learning is tradition rooted in dominant norms about how to generate knowledge. The solution for many funders and organizations to the question of how to test hypotheses is evaluation—typically, an impact evaluation. These studies are characterized by long timelines, big budgets, and limited generalizability that often make interpreting results challenging. I don’t mean to say that evaluation, broadly, and impact evaluation, specifically, do not have their place. What I challenge us to do differently is to see it as one of many tools we can use to test hypotheses, and that it has weaknesses we can compensate for using other sources of information—monitoring data, qualitative insights from educators, and operational data from running programs. A shift to a view of strategic learning as the framework into which many sources of evidence can fit (including evaluation) opens us up to an opportunity for synthesis and sense-making—one that we can share within and across organizations and the beneficiaries of that work. How much more could we learn if we bring diverse voices to the table to interpret strategy-relevant evidence from multiple sources? For further reading, I recommend this piece on alternatives to impact evaluation from Mary Kay Gugerty and Dearn Karlan, and The Great Nonprofit Evaluation Reboot by Elena Harman.
Achieving equity goals requires us to center strategic learning approaches on equity.
If we want to achieve racial and economic equity in education, it is important that we name it and adapt our approach. That means shifting how we develop goals and hypotheses, what evidence we collect and how we collect it, and how we focus on learning and improvement. With regard to goals and hypotheses, this is a question of who informs and gets to decide what our goals should be and which hypotheses are worth pursuing. If we do not include people proximate to the problems we want to solve, we will never be able to establish hypotheses that are most likely to succeed. We will be at a disadvantage from the start. What would we do differently if our hypotheses were informed more fully by those we seek to serve?
This connects to the question of evidence. What counts as evidence? We tend to prioritize, whether explicitly or implicitly, evidence that comes from traditional sources: quantitative data, comparably collected, or quantitative causal research designs, implemented by researchers with certain pedigrees. These methods are valuable, but they are limited and have inherent weaknesses, namely related to validity and generalizability across real-world contexts. If we acknowledge those weaknesses and strive to balance them with other sources of information (with their own weaknesses), we could incorporate all kinds of evidence (quantitative and qualitative, comparable and contextual, causal and correlational) when testing hypotheses. This also means that everyone with experience with the problem and potential solutions (educators, families, students, and community members) has evidence that must be considered to test our hypotheses about how to achieve equity. If we saw all evidence as imperfect but meaningful, how would that change our practice?
Centering on equity also means challenging our approaches to interpreting evidence and how we apply evidence to learning and improvement. Evidence (quantitative or otherwise) is not useful without interpretation—someone has to assert “This is what this means” to apply it to decisions. Those with equity goals need to ask, who gets to interpret the evidence? Whose lens are we using to filter what the evidence means and how we use it to improve? Centering on equity means we have to challenge ourselves to ensure we see the evidence through the perspectives that are most proximate to the problem and solutions. Would we come to the same conclusions if we used lenses other than our own?
Though there is much to say about equity-centered strategic learning, the last thing I’ll say is everything I’ve described here is synonymous with good, rigorous practice. The reason I call it out separately (equity-centered strategic learning) is that we often assume that what is traditional is good instead of assessing the context in which we’re operating and the goals we seek to achieve to reevaluate what good practice should be. For example, if you were to ask any researcher whether hypotheses should be informed by observations of experience and behavior relevant to those hypotheses, I’m sure they would say yes. If we apply that logic to equity goals, it is obvious that our hypotheses should be informed by the communities with proximate experience. Equity-centered approaches are not a “nice to have” for strategic learning for organizations with equity goals—they are a nonnegotiable best practice. For further reading, I recommend the Why Am I Always Being Researched guidebook from Chicago Beyond; the Equitable Evaluation Initiative framework; the Data Equity Framework from We All Count; Expanding the Bench’s resources on culturally responsive and equitable evaluation and initiatives; and the Center for Evaluation Innovation’s resources on equity, power, and evaluation.
It takes resources—financial, political, and people—to do equity-centered strategic learning well.
I have worked in strategic learning as part of organizations that seek funding and give funding. On both sides, we underestimate the resources it takes to do equity-centered strategic learning well. In my work in a nonprofit organization, those of us working on strategic learning activities were siloed and did not often have a surplus of funds to build sustainable systems and processes. In many ways, this is understandable. If you have the choice between spending $5 to support a student who needs the help or to collect some data, I get why we would choose to fund the more immediate need. However, it’s more than just money. If equity-centered strategic learning is not treated as the core of organization improvement and impact, the elements of the process are often used after the fact to justify decisions already made or to fulfill requirements imposed by funders. For equity-centered strategic learning to achieve its goal—to support ongoing improvement of our work to achieve equity in education—it requires organizations to focus fully with space and time for reflection and learning and the ability to take risks and incubate ideas. This requires resources and a commitment to the process.
This brings me to funders. One of the reasons I originally wanted to join a foundation to work in strategic learning is that I think funders have a great responsibility and opportunity to change what funded organizations are incentivized and enabled to do in this area. Although many funders probably think of themselves as goal-driven or data-driven, what I have seen is that this often takes the shape of a set of requirements and accountability structures for grantees. These structures are not conducive to learning, and they are infrequently accompanied by sufficient resources. We expect grantees to set goals and hypotheses and test them and report, but we don’t consider the resources it would take to do this well and do so in a way that lends itself to real learning and improvement. What if we treated capacity in equity-centered strategic learning as a core organization competency? One we would not expect an organization to be able to function without?
What we tried: Capacity Building for Strategic Learning Initiative
The Capacity Building for Strategic Learning Initiative (CBSL) was a pilot to see whether giving resources—financial, coaching, learning, and time—could help our grantees advance their equity-centered strategic learning practice. The participating grantees received an assessment of their capacities in equity-centered strategic learning, coaching and support to plan a scope of work to improve those capacities, funds to support that scope of work, and ongoing coaching and learning support in a cohort. We learned so much from this pilot about what grantees need, what is required to benefit from this capacity-building work, and how funders can better support this work in the future. In the comprehensive resources and toolkits on the CBSL web page, you can review how we approached the pilot and how you might support future efforts (or, if you’re an organization seeking funding, request support from your funders). My hope—for you, reader, and for my own team—is that this pilot and its materials don’t just serve as a model that could be replicated, but also offer provocations for how we can build equity-centered strategic learning practices throughout the sector in ways that improve upon the work we did in CBSL.
In general, we have a lot of work to do. I am certainly not an expert and struggle with many of the challenges I laid out here. However, our K–12 team at the Bill & Melinda Gates Foundation is committed to this effort internally and, as funders, to supporting our grantees in this work. In the years I have been at the Bill & Melinda Gates Foundation, I have seen us make improvements in equity-centered strategic learning, and I believe it has made us better partners to our grantees and more likely to achieve our goals of educational equity for Black and Latino students and students experiencing poverty. It also means an ongoing commitment to being clear and transparent about our hypotheses, where we’ve been wrong, what we’ve learned, and how we seek to improve. I hope that this piece and the CBSL materials offer a starting point for conversation, learning, and action. Please share with us your thoughts and questions so that we can learn from each other and improve together.
