Developing Education Programs to Prevent Violence and Crime in Latin America and the Caribbean

[MELISSA CHIAPPETTA]

We wanted to make sure that the evidence was going to be used. So oftentimes this is what happens. As a researcher myself, I've written so many studies that have just gone and sat on shelves and gathered dust, which is so frustrating, right, because we work so hard on these, and we don't always have them in a timely manner to inform new interventions.

We don't always have the time to actually spend working with the key stakeholders who are going to use the evidence to implement it, and to help them think about how to put it into action in their activities and interventions. And so that's really what we had hoped that the RETO activity would do, and that we are working to do now, right?

[J.B. WOGAN]

I'm J.B. Wogan from Mathematica, and welcome back to On the Evidence. For this episode, we're going to talk about how research ultimately informs action, and we're going to zero in on specific research looking at whether education programs prevent or mitigate violence and crime. Mathematica conducted an evidence review on this topic for USAID, with the idea in mind that past evidence might help develop effective strategies for leveraging the education sector to address violence and crime in Latin America and the Caribbean. Since the evidence review published in 2021, USAID and the International Rescue Committee have partnered to launch an initiative called the Research in Education for Transformative Opportunities, or RETO for short, that seeks to prevent violence that affects youth in northern Central America by creating demand for evidence-based educational policies and programs. My guests for this episode are Juan Carlos Rodriguez, Katie Appel, Melissa Chiappetta, and Emilie Bagby. Juan Carlos and Katie are the director and deputy director, respectively, for USAID's RETO activity implemented by the International Rescue Committee and 12 national partners. Welcome, Juan Carlos and Katie.

[KATIE APPEL]

Thanks so much, J.B.

[JUAN CARLOS RODRIGUEZ]

Thank you, J.B.

[J.B. WOGAN]

Glad to have you here. Melissa is a senior education advisor with the Office of Regional Sustainable Development within USAID's Bureau for Latin America and the Caribbean. Welcome, Melissa.

[MELISSA CHIAPPETTA]

Thank you so much for having us today. Nice to see you.

[J.B. WOGAN]

Emilie is a director at Mathematica where she oversees our international education research. Welcome, Emilie.

[EMILIE BAGBY]

Thanks, J.B. I’m glad to be here.

[J.B. WOGAN]

Okay. So I'm excited about this episode's topic because it answers a question I have in the back of my mind often when I see some piece of research on Mathematica's site, and that is what happens after the research publishes? Who uses it and how? How does it inform change? And how might it inform change? And so, I want to start with this global evidence review that Mathematica conducted on behalf of USAID. Melissa, what was the motivation behind commissioning this review? I mentioned it briefly in my intro, but would you just mind fleshing out some of the details on what USAID hoped to learn and how the agency hoped to use the findings?

[MELISSA CHIAPPETTA]

Yeah, great question. So as you might imagine, USAID's focus in the education sector is really on learning outcomes and on access to education. So ensuring what we call access to quality education, really. And so those are the types of outcomes that we're primarily trying to achieve. But we also have a goal in the Latin American Caribbean region to reduce violence and irregular migration. And so we wanted to be able to look at the connection between education and learning outcomes and access and violence and irregular migration. We knew that there was evidence out there. We had read several reports, but yet we couldn't come up with a synthesis ourselves to really understand what are the complex dynamics between each of these outcomes. And we knew that there were things in migration, for example, we see oftentimes that that students will migrate because they don't have access to a quality education in their home country. And so they go looking for quality education elsewhere. But similarly, they may have access to quality education, which then if there are not jobs for them to apply for after they've graduated, may result in brain drain. And so figuring out, you know, how can we focus on some of the workforce development outcomes to be able to connect people to jobs in their communities as opposed to them having to move to other countries?

But you can see that these dynamics are varied and they work in different directions, right? And so we wanted to really narrow in on what those nuances are and what they are for specific populations, too, because we know that they're different between men and women. We know that they're different between children and youth with disabilities. We know they're different for Indigenous youth. And so how can we really support these targeted populations to try to both increase their access to quality education, but also reduce violence and irregular migration?

[J.B. WOGAN]

Okay. Excellent. Yeah. That point about brain drain, it's not something I'd ever thought about before, that that was another way. I'm guessing some of our listeners are probably familiar with the conversation around early childhood education, how that might help with better outcomes later in life, but I hadn't thought about ways in which education could also be sort of a potential negative if it's pulling people out of a country. Emilie, the evidence review that Mathematica conducted for USAID covers a lot of ground. It is, by design, very comprehensive. But if you

[EMILIE BAGBY]

Thanks for that excellent question. So first, I’d like to say that it’s pretty clear from this evidence review that the education system has an important role to play in preventing and mitigating violence and crime. Education systems can both directly keep kids safe by ensuring schools and transport to schools are safe, and can indirectly affect violent and criminal behavior through its effect on other outcomes that are malleable throughout an individual's development, including from ages 3 to 29. Such outcomes are standard education outcomes like completion and skills formation. But there’s also these other important factors such as social emotional skills or engagement in risky or protective behaviors or the environment in which children and youth live, including at school, home, and the community. Secondly, our review maps out a conceptual framework based on the social science literature that documents the many mechanisms through which these outcomes can affect violent crime, nonviolent crime, and school violence. For example, the education system can support the development of social emotional skills, which can then strengthen an individual's ability to control their behaviors under difficult circumstances, to enable youth to plan ahead, to avoid criminal activity, or to help them think through problems in ways that lead to solutions that can avoid violence.

Social emotional skills can also affect one's likelihood of engaging in risky or protective behaviors which are also highly correlated with violence and crime, or they can facilitate improved educational attainment and learning, which can in turn affect employment and earnings and the likelihood of direct engagement in violence and crime. So there's lots of pathways through which these outcomes can affect our outcomes of interest and the report really does a nice job of summarizing that and synthesizing that.

I think the third key point from it is that, is around the state of the evidence, and I think there’s a few things here. So first, we are confident that there are interventions that can work to mitigate violence and crime. The majority of the strong evidence is from high-income countries, with some from the LAC region or from low- and middle-income countries. But there’s a general dearth of high-quality evidence, so it's really hard to show that specific education programs affect violence and crime outcomes directly, due to the nature of the education sector and the outcomes that it is affecting. Also, the fact that it can take a long time for impacts to appear, sometimes 10 to 20 years, and long-term research takes time, funding, and commitment. And also, measurement of violence and of criminal outcomes is really difficult to do.

[J.B. WOGAN]

Emilie, let me stick with you for one more question. You mentioned that we are confident that interventions can work to prevent or mitigate violence and crime. I'm sure our listeners are wondering this as well. What does work, or where is the promising evidence of things that can work?

[EMILIE BAGBY]

Yeah, that’s a great question. So across the 43 different intervention types that we looked at, I think there’s a few that have particular promise, based on the evidence, and I’d like to talk about, just mention three of them right now. So first, by providing young children a safe environment where they can begin learning and developing social-emotional skills at an early age, early childhood, sometimes referred to as pre-primary education sets children up for success in the long run. There’s a strong evidence base for early childhood education for improving social-emotional skills in childhood and violence and crime outcomes in adulthood. Most of this is generated by well-known longitudinal studies in high-income countries. We also identified moderate evidence or early childhood education improvements in social-emotional skills in the LAC region. So this is a promising program to reduce violence and crime, and it is likely more cost-effective than many others, particularly since it focuses on prevention, not mitigation, right? But the benefits would not be realized until years after the program.

The next category of interventions that I think are important to think about are those that can foster a positive school environment and work with children and youth to facilitate engagement in school and provide agency in life. These programs such as classroom-based social-emotional learning, school-wide positive behavioral programs and supports, or restorative practices, classroom management, preventing school-based bullying and school-related gender-based violence, and preventing dropout from school, can improve engagement in school, improve social-emotional skills, and thus encourage behavior change to reduce risky or violent behaviors. Also, just by keeping kids in school longer, they also reduce the time available for young people to engage in risky, criminal, or violent behaviors.

Third is a set of interventions that help out-of-school children and youth to continue their learning, either by facilitating access to traditional academic schooling or to alternative schooling, workforce development, adult basic education programs. These can facilitate learning and social-emotional skill development, which can have direct effects on behaviors, as well as indirect effects on violence and crime through improved employment and earnings.

[J.B. WOGAN]

Okay, so, Melissa, we just heard about some of the key insights from that evidence review, but as I mentioned at the top, that published in 2021, and we're talking now in June of 2023. So I want to talk about what's happening in the present moment. How are USAID and the International Rescue Committee building upon that evidence review to support education policies and programs that research suggests will prevent crime and violence?

[MELISSA CHIAPPETTA]

Thanks, J.B. Great question, and thanks to Emilie. I think you've done such a great job of highlighting those key findings from the study, and I'm so glad that we hired you and Mathematica to do that because you said it so much better than I did in the beginning. But just to go into how we've expanded on some of this work, so we used the findings from the study to help inform some activities in the region, so some bilateral activities, but also specifically our RETO activity, which you highlighted at the beginning, J.B., the Research and Education for Transformative Opportunities activity that's implemented by IRC, so by Juan Carlos and Katie.

And what we thought is that, you know, as Emilie highlighted, there were so many gaps in the evidence, and we wanted to see if there were any Spanish language studies that might help to fill some of those gaps, and also to sort of expand on the gray literature in the local context, and to try to really narrow in on some key contexts in Guatemala, Honduras, and El Salvador, knowing that there is a significant amount of violence in these countries that we wanted to try to address. So that's why we commissioned the study through RETO and that activity, and then really the next step is we wanted to make sure that the evidence was going to be used. So oftentimes this is what happens. As a researcher myself, I've written so many studies that have just gone and sat on shelves and gathered dust, which is so frustrating, right, because we work so hard on these, and we don't always have them in a timely manner to inform new interventions.

We don't always have the time to actually spend working with the key stakeholders who are going to use the evidence to implement it, and to help them think about how to put it into action in their activities and interventions. And so that's really what we had hoped that the RETO activity would do, and that we are working to do now, right? So synthesizing that evidence, really tailoring in to those key contexts, and then finding the stakeholders in those contexts that might be able to actually implement the evidence and use it in their activities, to adapt their activities, to tailor curriculum, for instance, to include more socioemotional learning, as Emilie highlighted, or to think about maybe we need to expand into preschool education in this region, and we're not doing enough in that area. So really being able to work with those stakeholders to define how best to do that moving forward.

[J.B. WOGAN]

And Katie, let me turn to you. Is there anything that you would add in terms of how USAID and the International Rescue Committee are building on this evidence review?

[KATIE APPEL]

Yeah, thank you so much, J.B., and thank you, Emilie and Melissa. So definitely, as Melissa said it, we are all about implementation and putting that evidence into use. And so maybe to begin, I'll start to clarify a little bit how we perceive Mathematica's methodologies complementing what we're doing with RETO. So Mathematica and IRC have engaged in different methodologies to identify, organize, and present relevant evidence in the education space for violence prevention. So Mathematica's evidence review was really a systematic review, which tells us what works, what works in education for violence prevention. What RETO is doing through our Evidence Gap Maps, or EGMs, is really trying to summarize what evidence is available in northern Central America, in Guatemala, in El Salvador, and in Honduras.

And Emilie, please correct me if I get any of this wrong, but Mathematica's evidence review included a global review of over 48,000 studies in English that resulted in the identification of about 500, 476 rigorous and eligible studies, and a synthesis of key trends that Emilie just shared with us to prevent crime and violence through education programming. What we're doing is we're building on that, that amazing momentum, and we've developed these Evidence Gap Maps. So we're working with key education stakeholders in these three countries then to take that evidence and actually put it into practice. So taking their strategic priorities and really taking that evidence and finding strategic opportunities, really concrete documents, policies, programs, and infusing it with that evidence that works.

So the key objectives of these Evidence Gap Maps is to map, organize, and then assess the quality of that existing evidence, thereby facilitating that use of evidence by those key education decision makers, driving the policy and programming, and then identifying evidence gaps. So these Evidence Gap Maps not only tell us what evidence exists in the region, but where there might be gaps for further research and investigation in the future. And so for us in the IRC, these Evidence Gap Maps are really a critical first step for our in-country evidence-based decision making. And what they are is actually an online platform. It's a visual tool that presents to key education stakeholders through this online platform, education and community interventions, and then also violence prevention outcomes. And visually, if you can kind of visualize a matrix, it shows them through sort of bubbles where these evidence gaps exist and where there's a lot of evidence present.

So these ministry officials or NGO decision makers can see where there is evidence and where there isn't evidence, and that's where we really begin our conversation with them. So to date, RETO's three Evidence Gap Maps include a review of over 44,000 studies in both English and Spanish, specifically from those three countries, resulting in the identification of 77 rigorous and eligible studies. So we chose those 77 studies out of the 44,000 based on our inclusion and exclusion criteria after a really participatory, long process. So with its focus on implementation, our project is directly collaborating with these decision makers, and our hope, our ultimate hope, the main goal of our project, as Melissa said, is to take what works and support these decision makers over the next year to directly apply this evidence in education policies and programs across northern Central America. So just to again sum up, these Evidence Gap Maps don't tell us what the evidence says, but rather where it exists, and now we're in the process of developing evidence syntheses that will then help us share with these key education stakeholders what is effective.

[J.B. WOGAN]

Are those Evidence Gap Maps publicly available if people want to take a look at them online?

[KATIE APPEL]

We are in the process of publishing them, so yes.

[J.B. WOGAN]

Okay. All right, well, whenever they are available, we'll make sure to add them to the episode show notes.

[KATIE APPEL]

Great.

[J.B. WOGAN]

Emilie, some of the research in your evidence review, or the Mathematica Evidence Review, came from outside of Latin America and the Caribbean, and this sort of dynamic is common in policy research, where a community has to do the work of translating and adapting research from another context. So what are the challenges with trying to import evidence-based policies and programs based on evidence from other places or other contexts?

[EMILIE BAGBY]

Yeah, I think this is a really important point, because as you note, a lot of the research that exists is not from, for example, the LAC region, right? Or from low- and middle-income countries, so it’s really important to adapt interventions to the local context. And oftentimes, what you’re doing is you’re starting from an intervention that may have been shown to work in a really different context. From a researcher’s perspective, it’s really hard to say what’s going to work in what country in what context because we don’t have the existing evidence, and so it’s really important for actors that are on the ground, working with education stakeholders, in ministries, in schools, in communities, to figure out what is the right intervention and what can work here, and how it should work. And I think that our colleagues on the call here, Juan Carlos and Katie, can talk a lot about this.

[J.B. WOGAN]

Yeah, maybe I could put this question to you, Juan Carlos. I'm interested in your perspective on this question as well. What challenges do you see in importing policies and programs based on evidence from somewhere else?

[JUAN CARLOS RODRIGUEZ]

Thank you to all of you. Yes, due to the challenges that has mentioned by Emilie, and in consultation with our national partners, to date, RETO began to focus its evidence searches in sources exclusively from North and Central America. To drive this change from the key education stakeholders, the evidence should reflect the national context and its population. While context in Latin America may appear similar, specifically in North and Central America, the three countries have important differences, necessitating local evidence to guide local adaptations. Yet across the sub-region, RETO has achieved a common understanding on how to define evidence. RETO considers evidence as information systematically obtained and analyzed to better understand the nature of the problem and to establish whether a given intervention addresses the problem, how, why, and what cost.

Research evidence is generated using rigorous methods that answer the most critical questions for action, including a range of types, such as impact operational and design, and also a range of methods such as impact design, quantitative, qualitative, non-experimental, and others. RETO seeks to understand the key needs and priorities of the key education stakeholders in each program country to facilitate targeted use of evidence. For example, RETO collected survey data on 132 key education stakeholders' access to and use of evidence for decision making and design and implementation of violence prevention programs and policies. When asked about the evidence as a priority for decision making, 59% found evidence to be essential, and 31% found it to be a high priority. Conversely, when asked about existing knowledge of evidence, only 6% of key education stakeholders responded that they had a lot of knowledge on violence prevention, with an additional 38% reporting moderate knowledge and 37% reporting some knowledge.

In addition, only 22 of the responses, less than 1 in 4, indicated that they currently request evidence to design, implement, or evaluate programs and policies related to violence prevention with their institution, due to the lack of updated evidence and understanding of how evidence links to education policy and programming. Based on these baseline results, key education stakeholders seem to value evidence and express need for it, but we have a current gap in their knowledge about violence prevention evidence and how to use this for decision making, which RETO hopes to address. RETO also consulted the key education stakeholders to better understand how they are currently using the evidence within their institution.

In general, the results indicate that their level of knowledge on the use of evidence varies according to the key education stakeholders, type of institution, their role in the organization, their time working in the institution, their academic background, their age, and their self-perception of the use of evidence. To address these gaps in evidence use, RETO will implement a personalized technical assistance to foster collaboration among researchers, policymakers, and other stakeholders to have different levels of knowledge and experience to work together under a common goal of increasing the use of evidence in education policy and programming.

[J.B. WOGAN]

Okay. Thank you for sharing those survey findings. That's fascinating that on one hand you've got key education stakeholders who clearly understand, or many who understand the value of evidence for decision making, but who may not be aware of what that evidence is and who are currently designing, implementing, and evaluating programs to contribute to the evidence base. That's an interesting combination of -- it's an interesting picture, so I'll be interested to see how you're able to take advantage of that demand for more knowledge and be a catalyst for change there. Melissa, how are USAID, the International Rescue Committee, and their implementation partners approaching the challenges that Emilie and Juan Carlos just described? In other words, what is the plan for translating evidence-based policies and programs to be effective in Northern Central America?

[MELISSA CHIAPPETTA]

Yeah, it's such an important question. I think it's something that we haven't figured out how to master yet, right? It's a challenge of getting the use of the evidence and not having it sit on the shelf. But I think one of the keys that we know is really participation from the beginning by those key education stakeholders. So identifying who are the people that want this information, what is the information that they want, how can they use it, and then to be able to work really closely with them throughout the process. So one of the really great things about RETO, I think as you highlighted at the beginning, is that we do have 12 local partners.

And those local partners are the ones that really have led this Gap Map process and the synthesis of the evidence. They're the ones that will be working with the key education stakeholders who -- this sounds like some sort of amorphous group, but just to contextualize it, we're talking about ministries of education, we're talking about USAID working in the country, we're talking about partners like Mathematica or IRC or other organizations that are actually working to support interventions in countries. And so working with those stakeholders directly then to be able to implement these recommendations in their interventions and to find out how do they best receive evidence. So I think oftentimes we make the mistake as researchers, we get so in the weeds, and I can say this about myself, I'm such a nerd. I want to show you how I did the whole thing from step one to step 150, and I want to have every little nuance in there. And so then I produce a 150-page report that literally nobody has time to read, right?

So if we're talking about ministers of education, they have so much to do, so they really don't have time to spend reading these long reports. So figuring out what's the best way for them to receive the information. Do they need a PowerPoint presentation? Do they need a policy memo? Do they need a policy memo that is then followed up by some working actually together looking at the policy and saying, here's the way that this policy should be adjusted, the exact wording that we would recommend, or here's the way that you might add this component into your curriculum framework. And so that's really the hard work that they're doing now. And by having those stakeholders involved throughout the process, that really helps. And then the last thing I would just highlight is that we do have this challenge of a lot of this evidence is global, and so we really need to pilot it in these contexts in Northern Central America and find out, does it actually work there?

So there is value in evidence that comes from outside of the context, especially when we don't have evidence of what works in the context. So we see promising interventions, and as Emilie said, we can highlight what was the context in which those worked? Does that context seem like some of those same conditions exist in this other context? Then great, let's pilot it and see what happens, and then learn along the way. And so that's a key goal moving forward, and I think one of the great values of the Gap Maps is that we're going to be able to take those questions and those key areas that we've highlighted where there's gaps and to fill some of those gaps. In fact, we just produced a new learning agenda that I think will be published online in the coming couple of weeks. And there are two key questions around the link between education and violence and the link between education and irregular migration. So we will drill down under those questions based on these GAP maps moving forward to get that more contextualized information.

[J.B. WOGAN]

May I ask a question? I want to pick up on something that you just mentioned, Melissa, about education, key education stakeholders that can sound a little amorphous. I'm more familiar with education and domestic policy context of the US where I would think there'd be the Secretary of Education at the federal level, the state analogous position of a state Secretary of Education. You've got your superintendents at the school district level, and then you've got your school boards, and then maybe principals too. And then maybe you'd even go down to the instructor level, like a teacher. Is that the same kind of system we would expect to see? Are those the people that you would hope to inform and empower in Northern Central America? Or does it vary by country who exactly you're hoping to reach with this evidence? And maybe that's as much a question for you as it is for Juan Carlos and Katie, but I'm just curious. I think it would help make it a little more concrete for the listeners to know who it is you're hoping to reach with this information.

[MELISSA CHIAPPETTA]

Yeah. That's a great highlight. And there are a lot of those same similarities. The terms may be slightly different. So instead of states, we might have departments or divisions. We might have more local zones. So it's just the different levels of decentralization that we consider, but certainly those actors exist in these contexts. There's also civil society organizations, which we have here as well, or unions. So teachers' unions, thinking about that as a potential group. Thinking about civil society organizations that might be supporting marginalized groups to ensure that they have good quality access to education. Thinking about indigenous children, indigenous youth as an example, and we see some of those types of civil society organizations here in the US as well. So these do vary quite a bit across contexts, and that's one of the great parts about RETO is that one of the key activities that they had early on was this stakeholder mapping, and this was conducted in each country as well. So I don't know, Katie or Juan Carlos, if you want to say anything more about that.

[KATIE APPEL]

I can jump in. Thank you. Thanks for teeing me up, Melissa. Yeah. So we, in our first year, we had, as Melissa mentioned, stakeholder mappings in the three countries, and what that allowed us to do was really dive in and understand who are the actors in each country that demonstrate the most power and influence in the violence prevention space. And so when we say key education stakeholder, the reason it sounds so amorphous and vague is because it's a very diverse group. And as Melissa mentioned, our national partners are a diverse group and are really leading and driving our technical implementation. And so our 12 national partners come from the private sector. We have researchers. We have think tanks. We have youth organizations. We have education organizations. And so with their diverse knowledge and expertise, they were the ones that have a lot of these relationships in countries and helped to drive this process and this mapping. And so from that mapping process, we've identified 28 institutions across the region that now are our key education stakeholders that we're working with now hand-in-hand.

As Juan Carlos mentioned, we're launching this process of technical assistance. And we're super excited about this technical assistance that we're now launching because it's going to enable us to really get to the heart of our project, which is the use of evidence. So we're understanding what are the priorities that drive these institutions, where do they see value in using evidence. And so our technical assistance is not a one-size-fit-all training. It's really working with each NGO, each ministry of education, each private sector association, each youth organization, and understanding what their strategic priorities are and how that matches with what the evidence says. And how can we find really concrete, maybe small, maybe large ways to tweak their programs or their policies to put that evidence into practice and see impact, see results in the education sector to reduce violence.

And so we're launching that process now. We have evidence-to-action teams. IRC has this amazing evidence-to-action team that's run out of the Airbell Impact Lab. And so we have evidence-to-action specialists that are part of our IRC team who are working hand-in-hand with some of our national partners in each country who are really driving this change. And they're rolling out this customized technical assistance in the three countries. And so really it's demand-driven based on the needs and priorities of these institutions. And so we're hoping by the end of next year, we're going to see concrete change. And we're working with over 125 different individuals that will hopefully see real value and benefit from participating in this work with us.

[J.B. WOGAN]

Okay. So I have one final question, and I would put this to the group. I'm interested in all of your perspectives on it. But as you know, this podcast is called On the Evidence. And we've spent most of today talking about how to take action based on existing evidence. But I'm curious about what evidence we still need. What kinds of research would be useful after RETO? Do we have any takers for starting us off? I see Katie, you're nodding your head. Maybe you want to jump in.

[KATIE APPEL]

I can jump in.

[J.B. WOGAN]

All right.

[KATIE APPEL]

So what we saw from the Evidence Gap Map, what we've seen from, like I mentioned before, the Evidence Gap Map is this matrix. And when there's nothing in one of the boxes in that matrix, it means that there's no evidence or we haven't identified yet any evidence. And we have seen that there are gaps, for example, in evidence about migration. And so that is an area where there might be potential for further research in the region. So that's one place to start.

[J.B. WOGAN]

Okay. All right. Emilie, anything? Yeah, I see a raised hand there. That’s very polite.

[EMILIE BAGBY]

Thank you. So as I’ve said before, there’s a lot more research that we need to conduct to expand our understanding of what can work in different specific contexts. For each of the 43 different intervention types that we looked at in the review, we’ve made specific recommendations for improving the evidence base for each.

There’s some key cross-cutting recommendations that I’d like to mention here. First, most research on these promising—we need more, sorry. We need more research on these promising programs in our review, specifically in the LAC region, in low- and middle-income countries where the context is different from high-income countries. We’d like to have a better understanding of longer-term impacts of these programs, and improving measurements of violence and crime outcomes.

We also think it’s worthwhile for researchers to look at correlated outcomes that we lay out in our conceptual framework, even when interventions are not specifically designed to affect those outcomes, because we know the education system, when it is doing its job, it’s not only affecting learning outcomes, it’s affecting other things, right? And we know that there’s a complex interplay of the different outcomes. If researchers can focus on disentangling the effects of specific components, or different combinations of components in programs, we can then figure out what is most—what works best and what is the most cost-effective approach to improving the outcomes we seek to improve.

It’s also important to understand the heterogeneity of impacts of programs on different subgroups, because we know that different individuals can respond differently to interventions. And finally, I think documenting the costs and the cost-effectiveness of programs is so important, so that way policymakers can make more informed choices and figure out what’s going to work best in their context.

[J.B. WOGAN]

Melissa, I see a nodding head. Anything you would add in terms of research that is still needed? Any evidence that you'd love to have in the future?

[MELISSA CHIAPPETTA]

Yeah, I think everyone's done a wonderful job of highlighting it. Really, what is the specific component of the intervention? Is it the social-emotional learning? Or does that have more of an impact or does it make more impact if we just focus on improving reading outcomes and math outcomes that then ultimately lead to future jobs and violence reduction? So what is the package of interventions? And if we don't have all of the money and we can't do everything, where's the most cost-effective way for us to intervene and to provide support? And then I think also those long-term studies are so critical because we don't know -- I mean, we have a lot of evidence on preschool, for instance, and pre-primary education in the US and in other contexts, but we haven't had studies on that in LAC.

And we don't have studies on a lot of these longer-term outcomes that will help us to understand what happens 10 years down the road, 20 years down the road after you've had this quality of education. And so really following, especially if we can follow individuals through the process, so much learning could come from that. So I get excited about all of these potentials. And I think we narrow in on some of the key types of interventions in both the Gap Map and Mathematica synthesis. So I really encourage people to look at those documents too because we would welcome evidence and research that anybody wants to conduct on some of those gaps and partnering with anybody on those opportunities.

[J.B. WOGAN]

That sounds great. So if people look at the learning agenda that you referenced earlier in the conversation, that could actually be a place to start for addressing some of these gaps in evidence. Okay. Well, Carlos, I don't want to neglect any insights from you. Is there any kind of research or evidence that you would like to see in the future? Where should the research community go from here?

[JUAN CARLOS RODRIGUEZ]

Well, thank you, J.B., for your question. And yes, I also want to mention that we are also preparing implementing a comprehensive evidence synthesis report with all the evidence from the three Evidence Gap Maps to indicate which evidence is the least effective and why, and which appear promising. The comprehensive evidence synthesis report will leverage rigorous techniques of evidence synthesis such as meta-analysis or narrative synthesis. But finally, RETO will work with its 12 national partners to ensure sustainability of key activities so that key education stakeholders in Central America will continue to be supported to use evidence in education programs and policymaking to support violence prevention across El Salvador, Guatemala, and Honduras well into the future.

[J.B. WOGAN]

Okay. Excellent. Well, let me just get back to my notes here. All right. So I think that's a great note to end on. Juan Carlos, Katie, Melissa, and Emilie, thank you for a great conversation. To our audience who will be listening or watching this episode in the future, thank you for listening and watching another episode of On the Evidence, the Mathematica podcast. This episode was produced by the inimitable Rick Stoddard. Any research or other resources mentioned in our conversation will appear in the show notes section of the episode. You can catch future episodes of the podcast by following us on YouTube, Apple Podcasts, Spotify, Stitcher, or wherever else you listen to podcasts. You can also visit us at Mathematica.org/OnTheEvidence.