Over the past two decades, government and philanthropic investment in rigorous evaluations in education has grown substantially. These evaluations have provided valuable information about “what works.” The Every Student Succeeds Act (ESSA) of 2015 promoted the use of evidence-based interventions, defining tiers of evidence based largely on the rigor of the methods used to estimate the impacts of an intervention. Policymakers’ focus on clarifying the rigor of evidence on impacts and any causal inferences is very helpful, but it has sometimes come at the expense of considering other important characteristics of useful and timely evidence. There are at least four ways the research community can increase the utility of evaluations, helping educators determine whether and how to implement a particular intervention in a specific context:
- Clearly document interventions and how they were implemented. To replicate an intervention, practitioners must understand what was implemented, including the services delivered, the resources employed, and other contextual conditions. If there is a comparison group, studies must document the differences between the services they received and those received by participants in the intervention.
- Assess whether the findings can be generalized. To be most useful, studies should examine which groups of students benefited and gauge the extent to which findings may be generalizable to broader populations.
- Explore what it would take to expand or replicate an intervention. Researchers should try to identify factors that helped or hindered implementation. In addition, they should examine whether the resources used to implement the intervention are available in some places where replication might occur. For example, if the effectiveness of a program depends on recruiting teachers with special skills, scaling it up may be more challenging; by contrast, programs can be more easily scaled if many schools (or other organizations) can easily adopt them with their current staff (List, 2021). Studies should also provide educators and policymakers with an estimate of the financial cost of adopting the intervention.
- Make evidence timely. Funders and researchers should generate and disseminate timely evidence that can inform important decisions before the evidence becomes out of date and potentially less relevant. This sometimes means balancing the value of rigor and the three objectives above against the need to obtain timely evidence. Even if it is not possible to generate the most rigorous impact estimates, studies can sometimes provide useful evidence about an intervention’s approximate effects. For example, if a district has decided that they need to implement an academic summer program (for example to address the learning loss resulting from the pandemic) they should use whatever evidence is currently available to select or design the program for the current summer, while collecting additional evidence to inform what programs are used in future summers.
Growing Support for Developing More Useful Evidence
Some organizations and researchers are building useful education evidence that includes several of the features above:
- The Institute of Education Science (IES) has developed Standards for Excellence in Education Research (SEER) principles that focus on generating evidence to help decision makers determine what interventions are effective, for whom, and under what conditions. IES has also provided guidelines to help implement many of these principles, including ways to gauge which populations are likely to benefit from an intervention, how to document implementation in ways that can facilitate replication, and ways to assess the incremental costs of an intervention. IES has also provided new guidance to the Regional Educational Laboratories on ways to generate useful evidence on promising practices, even when the most rigorous impact methods are not feasible. This includes guidance on the types of data and methods that can account for differences in the backgrounds of students participating in an intervention and their peers in a comparison group.
- AmeriCorps and others have developed tools for assessing scalability of interventions. AmeriCorps’ tools can help studies determine whether and how to replicate an intervention. Economist John List has identified other factors that can help organizations scale effective programs.
- Foundations have supported grantees and researchers in building evidence. For the Bill & Melinda Gates Foundation, Mathematica has been helping grantees use evidence to pilot test and enhance their education programs. For this purpose, our staff developed a set of tools and are helping grantees use them. The William T. Grant Foundation has also sponsored many studies exploring ways to ensure evidence is useful and used by practitioners.
Moving Forward
Policymakers, foundations, and researchers should continue to push forward in making evidence more useful and helping practitioners use relevant research. Those involved in supporting and disseminating research can contribute in a variety of ways:
- Refining criteria for reviewing education evaluation proposals. In evaluating study proposals, funders should not focus exclusively on the rigor of impact estimates. They should also ask those proposing education evaluations to document an intervention in ways that can foster decision making and replication. Funding criteria should encourage research teams to assess the costs of an intervention, the generalizability of findings, and factors that could affect the challenges of replicating the intervention in particular contexts.
- Conducting methodological research and developing relevant tools. More methodological research and tools are needed to improve evaluations. In particular, more tools are needed to assess the generalizability of findings and the replicability of interventions. These efforts could build on the guides sponsored by IES, the tools the Gates Foundation has been developing, the work of John List, and the scalability tools developed by AmeriCorps.
- Facilitating access and use of relevant, timely evidence. Research organizations and funders can make it easier for decision makers to identify and assess potentially relevant evidence. This could mean refining existing clearinghouses to provide more information on how interventions were implemented and whether and how they might be replicated. There also should be some way for decision makers to access evidence that may not meet the most rigorous standards but does use acceptable statistical methods for gauging whether an intervention is promising.
Thinking broadly and deeply about evidence is making education research more useful. The ideas offered here can further increase the utility and timeliness of evidence, thereby enhancing educational opportunities.
