Reflecting on Evidence and Insights in 2021 and the Year Ahead

[PREVIEW CLIP FROM JILL CONSTANTINE]

There's a question, there's going to be a lot of work to do this year to see what kind of interrupted learning occurred and, and what is very likely the inequities in how much ground some students lost. That'll be something we're all taking on in 2022, but we've learned a lot about delivering all kinds of service to children and families, including under these difficult circumstances.

[J.B. WOGAN]

I'm J.B. Wogan from Mathematica, and welcome back to On the Evidence, a show that examines what we know about today's most urgent challenges and how we can make progress in addressing them.

For the final episode of 2021, I invited back three guests who helped us make sense of 2020 and make predictions about the year ahead: Adam Coyne, Jill Constantine, and Chris Trenholm. Together, they oversee Mathematica’s International, Human Services, and Health business units, respectively. Just like last time, I asked them to reflect on themes from the past 12 months and to give us a sneak peek at some of the most exciting things that listeners can expect to hear more about in 2022.

I interviewed them separately. You’ll hear from Adam first, then Jill, then Chris. Broadly speaking, we discuss the most pressing challenges in 2021 that Mathematica and its partners encountered; how we surfaced data-driven insights to meet those challenges; what we expect to be the big, thorny problems of 2022; and where we find optimism in the movement to improve public well-being by using evidence-based programs, policies, and practices.

We’re releasing this episode in conjunction with our annual year-in-review web feature I helped edit, which features a curated list of videos, podcasts, blogs, interactive data visualizations, and other products that capture some of our most impactful work of 2021. I’ll include a link to the page in the show notes and I highly recommend you check it out.

As I said, Adam Coyne is our first guest. Through the International Unit that Adam oversees, Mathematica works in more than 50 countries across Africa, Asia, and Latin America on projects related to child literacy, clean energy, agricultural development, maternal and sexual reproductive health, water and sanitation services, and more. Historically, it’s the unit that has been most active in contributing evidence related to climate change, and as you’ll notice in our conversation, it’s going to continue to play a big part in Mathematica’s efforts to support evidence-based climate action. With that, here’s my interview with Adam.

[J.B. WOGAN]

You know, at, at Mathematica, we're obviously big proponents of generating high quality evidence, but it's not just evidence for evidence sake. We want it to be used to make an impact. So I was just curious from your standpoint and the work that you've been involved in and overseeing in the international unit, through Mathematica's work in the past year, where has evidence made a difference in improving public well-being?

[ADAM COYNE]

Sure. In thinking about 2021 J.B. And really focusing on where evidence has made a difference in improving well-being, it would be difficult not to start with COVID. You know, and there's a few examples of our work around COVID that I want to highlight that really did make a difference this year. The first was our ongoing work around contact tracing, in Washington state. And that actually is a partnership between the international unit and the health unit, where we have been working hand in glove with local officials to really understand how the pandemic's spreading in the state and working very closely, to contact and notify people who are positive or who may have been exposed to do everything we can to mitigate the spread. So that's been a nice example of, you know, good old fashioned field work, where we're trying on prevent the spread of a pandemic, but moving on more internationally, we really did study this year, the impact of COVID on learning.

You know, there were school closures in more than 190 countries that certainly interrupted conventional in-person schooling, and students really experienced a marked reduction in learning levels, or slower progress than you'd expect in a typical year. Now, these conditions disproportionately affected disadvantaged children who lack the access and opportunities to remote learning. So Mathematica, we worked to adapt a recent project that could shed light on how to identify what works to support vulnerable student populations, to continue their learning in this new context, as part of our rapid feedback monitoring and evaluation research and learning consortium, or we call it RF MERL, because we love acronyms. We worked with some partners, Chemonics, and the United States Agency for International Development or US AID to implement and evaluate something called Lecture Pour Tous, which is an early grade reading program in Senegal. After schools reopened for in-person instruction during the 2021 school year lecture, Pour Tous piloted three models of teacher coaching to determine which one had the highest impact and was best suited for future use.

Now these models provided useful information about how to best support teachers, both in person and virtually and the pandemic highlighted the importance of supporting teachers to continue professional development when in person trainings are limited. Now the pandemic's full impact on responsive and equitable education certainly remains uncertain, but policymakers and researchers should work together now to identify how to help children in low resource settings, continue to develop their literacy skills. More than 90 percent of the world's education ministries adopted remote learning policies during the pandemic. But this is amazing. 1.3 billion school age girls and boys age three to 17, don't have internet access at home. So while some places offer radio or TV or you know, digital and app-based learning programs and other mass media alternatives, many children in the poorest and hardest to reach areas have no access to these options and many of their caregivers and teachers aren't prepared to support remote learning.

So the evidence that we've learned on what to do to really coach teachers in the community was very helpful here. There was another COVID study that we did that was also on the impact of economic shock on households in developing countries. And this was really to understand how COVID 19 government restrictions and global economic disruptions from the pandemic have affected household wellbeing. So Mathematica worked with FinMark Trust and GeoPoll to field high frequency cross-sectional surveys in several Sub-Saharan African countries. Specifically Kenya, Nigeria, and Uganda. We used an innovative approach to weigh that telephone based surveys and were able to create a nationally representative estimate to understand the impact of COVID 19 on households, specifically, to understand how it's affected households economically, what supports have been available and how they're coping with economic shocks. What we've learned is that households have experienced substantial financial insecurity since the pandemic began, a large majority of them reported reduced income compared to the same time a year earlier, and most households who have businesses report challenges with those businesses; we saw that rural areas were harder hit financially than urban areas with rural households being more likely to report reduced income in all countries.

What we also learned is that food insecurity has been high in these countries with a substantial number of households reporting reduced food availability and the need to skip meals. Skipping meals is increased over time in Kenyan, Nigeria, again, suggesting increased hardship as the pandemic's gone on. There's less government subsidy and support for these populations. So many are relying on loans to survive. There's a terrific blog entry actually, in the, in the spirit of cross promotion that really goes in depth on this study that I certainly would encourage, some of your listeners to check out.

[J.B. WOGAN]

Okay. And I'm hearing a lot of I'm hearing the pandemic is a major thru line in terms of pressing problems in the past year that we've gotten engaged in, that we're helping partners try to solve. Surely that is one of the pressing problems of 2021. Are there other themes that you would point to things where there was really a demand for evidence to try and problem solve in the past year?

[ADAM COYNE]

You know, aside from COVID and, and many of the ongoing issues, we focus our international work on education, maternal and childhood health, energy, sustainable agriculture, for example, I really would have to say that the most pressing is climate change. That's something that we really saw cross our desk in a very different way this year. We had some very interesting climate change projects, you know, including a clean energy job transition program for the World Bank. This is an example where the energy Sector Management Assistant Program or SMAP of the World Bank, they've invested, another acronym we love them here, has invested in a range of clean energy investments that target access, generation, efficiency, and policy reforms. So Mathematica, we've assessed the direct, indirect, and induced jobs that result from these kind of work.

So using both qualitative research survey and secondary data, Mathematica developed a case study that profile seven projects in six countries, Rwanda, Peru, Nigeria, Kosovo, Pakistan, and India. And, these case studies are helping the World Bank consider the impact of the clean energy transition on job creation. And it's also going to help inform future indicators and target for World Bank investments. Another interesting climate project that we worked on this year is called the Climate Judiciary Project of the Environmental Law Institute and the Environmental Law Institute really developed this project to provide objective, non-political climate science education to judges, law clerks, and law students in the United States. Recognizing that an increase in the number of climate change cases in the courts and based on input from judges and others in the legal profession, the Environmental Law Institute identified a need for these judges to receive information on basic climate science. Through this project, they're developing a curriculum that covers topics on climate science facts.

The intersection between science and the law and the social impacts of climate change. So in the coming years, the Environmental Law Institute is going to deliver this curriculum to judges, legal clerks and law students through both in-person and virtual seminars. The curriculum's also going to be available online for individual study. You know, the aim here is really to increase judges' knowledge of climate science facts and the understanding of the impacts of climate change with the ultimate goal of helping judges to make better informed decisions. We're working collaboratively with the climate judiciary project team to elaborate the theory of change, and to define the priority research questions and activities. And then a final climate project that, I would be remiss if I didn't mention is we created a decision support tool that's pretty cool. Agricultural and environmental outcomes are, are certainly inextricably linked, soil health, water availability, air quality, and climate change, all influence crop productivity.

Yet agriculture at the same time can drive deforestation, biodiversity loss, soil erosion, and soil and water pollution. And is responsible for for over one quarter of the total greenhouse gas emissions. So promising agriculture technologies that increase both productivity and positive environmental outcomes can greatly improve the sustainability of agriculture. Therefore having a fuller accounting of the benefits and costs of agricultural innovations can help optimize investments. So under a US AID-supported consortium and, and the Feed, the Future Sustainable Intensification Innovation lab, we're developing this sustainable agriculture decision support tool that fills this information gap and can inform the scale up of innovation technologies to help achieve those sort win-win agricultural and environmental outcomes. In lay terms, you basically put all the inputs into this tool, and then it actually tells you what the projected outcomes will be and the impact, so that as folks are making decisions about what to do, they can basically pre-game their thinking actually roll it out to the field. So it's something that's pretty exciting for us.

[J.B. WOGAN]

So if you could look into your crystal ball, looking at 2022, what do you think are going to be the pressing issues next year? I have to imagine the pandemic is still going to be a big part of our work. Climate change, sadly, I think is probably going to be a big part of our work for decades to come. But anything that you would tease out either from those issues or other issues we haven't talked about yet?

[ADAM COYNE]

What I love about this time of year is it's fun to take out the crystal ball and really predict. And if I, if I had to focus on what is the most pressing, if not one of the more pressing issues, I think again, in 2022, it's going to be climate change. And the reason I say this is in looking at two of our biggest clients and partners. For the first, the Millennial Challenge Corporation or MCC they've already shared that 50% of their future strategy is going to be climate focused. There's a similar story at USAID, in April USAID announced their commitment to developing a new agency climate strategy, to advance government objectives, to adapt to and lessen the effects of global climate change. Now they made their draft strategy available for public comment in November. And it highlighted a major increase in both the commitment and scope of what they want to accomplish.

So it's really exciting to see the major players in the field, stepping up in a meaningful way to take on what may be a defining issue of our time. Another area that I would probably have to focus on if I was thinking about the coming year is really highlighting the critical importance of equity. Equity underpins everything we do on almost every issue we work on: COVID education, labor, agriculture, where small changes can affect the most vulnerable populations in the largest way. So infusing equity in how we approach projects from inception to completion is critical, as is working together with people in the countries we're serving. It's not just important, but essential to have local perspectives and involvement when you're working internationally, which is, is why I'm so excited that we're working more and more in the field with local partners. And I'm really excited to see a focus on equity coming out in more of the request for proposals that we see from our clients. They're starting to understand, not just gender equity, rural versus urban, you know, sort of different countries, all sorts of different aspects equity, which I think is really, really exciting.

[J.B. WOGAN]

That's pulling out your crystal ball to identify challenges or pressing issues for the next year, but let's end on a positive note, considering projects and other initiatives in your unit and the company at large, where do you find signs for optimism in the year ahead?

[ADAM COYNE]

So maybe the biggest sign for optimism is really more and better COVID treatments. And I say this as, you know, vaccines are truly becoming more and more available everywhere, and you know what I'll call vaccine plus as we're seeing more and more treatments in development, which is going to be critically important, particularly in the developing world, you know, so I'm really excited at the possibility of, you know, renewed travel of getting back in the field of really seeing these signs of hope. I'm also really excited, you know, about a new early childhood education project with the Hilton Foundation that we were just awarded as it represents a hugely important movement in the development space with foundation and bilateral and multi-later donors finally investing more in the early years, which is age zero to three as well is beyond.

So they're starting to understand the critical importance of investing at the very beginning in childhood education and development, which is something that we've seen in terms of our education work here in the U.S. So that's actually very promising and exciting, but, you know, based on your question, I guess I'd say I'm incredibly optimistic and excited, about the work and effort we're actually doing to get closer to the field. In 2022, we've hired a new managing director of EDI, which is our international subsidiary. And, you know, next year we are opening new offices in Nairobi, Kenya, and Kampala, Uganda, addition to our two offices in Tanzania and our office in London. So having this type of on the ground presence really matters because you need to be part of the communities you're working with, to truly understand them and to truly make a difference.

And then I think the last thing I would say, you know, when it comes to, you know, sort of the, my, my as half full I'm always optimistic and inspired, just by my tremendous Mathematica colleagues, I'm, I'm hopeful and optimistic about getting into the office more and having the chance to spontaneously collaborate, you know, it never ceases to amaze me, you know, just how passionate, committed, and creative these 1500 plus employees really are in terms of making a difference. And just the resilience they've shown over the past two years gives me just tremendous hope as we start returning to some semblance of normalcy in 2022 of just how much, you know, is, is remains to be done and how much can accomplish and how much impact we can have.

[J.B. WOGAN]

Fantastic.

[J.B. WOGAN]

Thanks to Adam Coyne for his reflections. Next, you're going to hear from Jill Constantine, who oversees our Human Services work. Human Services at Mathematica encapsulates a wide range of domestic social policy, including nutrition, employment, criminal justice, child care, child welfare, child support, K-12 education, higher education, and more. Regular listeners may recall Jill’s appearance on a previous episode to talk about why we need greater diversity among teachers in K-12 education. I hope you enjoy our conversation.

[J.B. WOGAN]

All right. The first question I'm asking you and Chris and Adam is about the impact of evidence. And you know, at Mathematica, obviously we're big proponents of generating high quality evidence, but it's not just evidence for evidence sake. We want to see it improve people's lives, improve public wellbeing. And so I was, I wanted to put the question to you at the start, where have you seen in human services? Where have we seen evidence making a difference in improving public well-being?

[JILL CONSTANTINE]

Yeah, so the that's great question as I reflected on 2021, and I'd say when 2021 began, it was a lot like how 20, 20 ended. We were still doing a lot of work around helping human services agencies, programs, school districts manage through COVID. And so we had, we, we continued lots of work around modeling for school districts, spread of COVID with different mitigation policies, figuring out how to get children and families fed at school and outside of school and figuring out how to deliver services. So that all continued during the year and bringing what research and evidence we had to that was hard sometimes. You know, we did some work for the New Jersey Department of Education, where they said, what do we know about remote instruction and how you keep kids engaged? And we thought, and we said, well, there's a little bit of research on remote instruction.

None of it under this circumstance that everybody's home because of a pandemic. So we're always applying a little bit of the evidence to this new situation. So we continue that sort of work, but what's been great are kind of two things. One of them is as we began to, well, I, I, we can't say come out of the pandemic anymore, but as the recession specifically lessened and people started thinking harder about getting people back to work and our work shifted a little bit. And then the other big change for us in human services where we were able to bring evidence regularly was around equity. So that was our other big shift in January 2021 with the new administration, that was an executive order signed that basically said in all of the human services programs we support at the federal level, you have to consider how structural barriers, systemic racism has impacted service delivery and equitable outcomes.

So this was very exciting for human services. Foundations had been pushing us on this, but now the goals of the philanthropy sector, just very much aligned with the federal sector. So we have a lot of examples of then right there in real time in 2021 looking at programs we have been working with and, and bringing evidence to all along, making sure they embedded equity. So one of my favorite examples that people can hear about in the podcast that is now out is looking at our work in programs designed to promote responsible fatherhood, engaging fathers more with their families and their children that we were asked to now look at those programs and really examine how institutional bears or structural racism might cause disengagement by fathers, might make the programs not feel inclusive to all fathers and just having.

And it's just such a great lens to be able to now try to bring evidence to that there's all kinds of reasons that might cause people not to engage with a program. And certainly for some of these programs, how families, how clients, how men have been treated, contribute to disengagement. So really digging in to see that and ideas for correcting that, improving that, is a very rewarding aspect that we get to now dive into those issues on the work. Another great example, putting equity at the center of programs is work we did with the Gates Foundation and our partner Catalyst:Ed to help build capacity for strategic learning. And the goal there is to provide grantees of the foundation.

And that's all kinds of organizations that serve students at all different level. pre-K all the way to college, but it's to provide supports and tools to embed equity into their organizational goals, into their learning practices, all with the idea of enhancing their strategy around delivering services. So we have a page and a resource people can go to, to look at that. There's a great video that really one of the grantees encapsulates why this was important to them. As she put it, she said, so everybody can agree all of the grantees working in the education space, yes, we want to embed equity. That's so important. but how, like how do we do that? We can all talk about it. It's an important goal. And she talks about how the tools made things very concrete for her. So that's another good example of all sorts of organizations that provide human services, that support children and families trying to be very mindful, right from the beginning of their processes, about how they're going to embed equity and understand what they're doing and how that's affecting service delivery and how that's affecting outcome for kids.

The second area that I'd say evidence really played into and had immediate impacts was everything about work . So as we move from so many people being out of work, out of school, out of training programs to the recovery, it was important to try to understand, okay, now we're moving, jobs are returning. How do we make sure we're getting folks into jobs, especially the more vulnerable groups who had been displaced by the pandemic. So again, a project we had had some time, which is our Pathways Clearinghouse, which is a systematic evidence review for an agency within Health and Human Services, to try to understand best practices and programs to get people to work. They asked us, they said, okay, we're going from recession to recovery. We need to understand specifically the kinds of programs and the kinds of services that are most effective in that recession environment compared to the recovery environment.

So we went and did that type of research and found that what, what is most effective is actually what programs tended to lead to. And during the recession, when jobs are lacking or people are displaced, there was a lot of pivoting towards just basic services, general supports if just for, you know, connecting them with other agencies, just to make sure they're that clients are, have places to live and are food to eat, but then also the preparing for the return to work. So it's a much more focus on general services, but when we got to the recovery, it really shifts towards the employment and training services, when jobs return. So making sure, again, folks that have been displaced, been out of school, if they've lost ground on their skills, getting them back into training programs and getting them into placement programs.

And then the other area of focus, where we could bring evidence. And, and this is going to go on to 2022 is our work on the registered apprenticeship programs, again, programs that are really focused on people who were most displaced by the pandemic, which tended to be younger people and is a big priority of the Biden administration for training youth and getting people back to work. And the Biden administration is proposing lots of resources to go to registered apprenticeship program. And we in 2021 went out there and said, okay, how ready are states to take funding and make good use of it? And we found not so ready.

There were lots of is there were lots of states reported everything from challenges coordinating across organizations to deliver apprenticeship programs, to not even exactly understanding what they needed to do to access resources and implement apprenticeship programs. So that was very important too, for us being able to quickly respond to the administration about a program they want to support and expand and showing programmatic and federal leaders where the programs are in terms of their readiness for these resources. So those were like two main areas where evidence had an impact and were new and very responsive to both the federal priority and equity and the desire to get everybody back to work.

[J.B. WOGAN]

In your answer about some of the areas where evidence made an impact, you forecasted a little bit what the issues maybe were. I heard equity, you talked about the pandemic and not just from a public health standpoint, but in human services having to do with education, employment, families, family support. Is there anything in addition to the items I just mentioned that you would say were really kind of the driving issues in 2021, that evidence was sort of called upon to help address?

[JILL CONSTANTINE]

Yeah. So the other thing we're also beginning to see is it really is very much in the spirit also of work, but there was a emphasis on getting women back to work, because women were disproportionately impacted by the pandemic, left the labor force in bigger proportion than men, largely because they were dealing with children at home because there was no daycare or because there was only remote learning. So the Biden administration is also very focused on getting women back to work and very clearly sees the link to making sure child care is available, is high quality, and that women can access it, so they can get back to work. So we also began some work and again, one of our programs around getting parents back to work looked at lack of access to child care as a barrier to getting particularly women back to work. And this is going to continue to be a focus for the Biden administration going forward. Obviously there's a focus on pre-K for, we'll see if it's universal or just, or for families most unable to pay for other kind of pre-K and a continued focus on making sure early child care and pre-K is high quality. So we did some work on that this year, kind of bringing evidence on what are the barriers to accessing good child care and what improves the quality and the accessibility. And I think that'll continue to be a focus into next year.

[J.B. WOGAN]

So 2021 was sort of a funny year in the sense that in some ways it was another, just another year of the pandemic. But I think, we moved into a recovery phase, even though there were occasional setbacks with Delta and you know, the vaccination rate stalling and other things like that. What does 2022 look like? What do you think are going to be some of the big, hairy problems that we'll encounter or have to continue facing in this next stage of the recovery?

[JILL CONSTANTINE]

Yeah, so I think that's a great question. We have already kind of seen the stage set for some of that in late 2021. When we see what our foundation partners are interested in, what federal and state and local agencies are interested in, there's two big emphases ongoing on equity on work. So I'll talk a little bit about things we're developing already around that. And then I'll talk about the ongoing importance of evidence and how lots of entities I gave the example for the foundation in building capacity for helping their grantees think about how to embed equity, but it's very much an evidence-based approach for how do you make sure the goals you're trying to achieve the outcomes you're trying to achieve? What evidence are you bringing to bake it right into your processes? I'll talk a little bit about the federal Evidence Act, which has been around for a number of years, but we continue at Mathematica to partner with more and more agencies to implement the aspects of the Evidence Act that kind of meets the agencies wherever they are.

So let me talk about equity first, because that's going to continue to be a major focus of all of our work. So we're starting a project just this year where we're looking at child welfare agencies and their partners to advance equity and provide supports to underserved population and the child welfare system is another perfect example where institutional barriers, lack of coordination, and structural racism can really disproportionately affect certain families, affect children. So, so the work we'll really be doing there is trying to understand those barriers, try to provide assistance to agencies understanding those barriers, but that, that includes actually helping agencies access and understand data better and use it better. That's everything from just coordinating data across agencies, which is a huge problem in child welfare. That's where you see things get missed and trying to use data to kind of predict where you might have issues, but use that in an ethical way, in a way that doesn't perpetuate ongoing systemic racism or biases.

So it's a really important project, similar to how I talked about engaging fathers, to take a critical system for protecting children in the child welfare system and just really dig in on how much inequity, racism, those kind of barriers got in the way of providing good services. So we're very excited for that, because it's obviously one of the most vulnerable populations, children in the child welfare system that we focus on that we work on. So we're very, we're very much looking forward to that. Another project again, it's kind of, it's bringing the evidence and the equity together, is within health and human services. The Assistant Secretary for Planning and Evaluation as part of broader work we have with them have asked us to establish the equity TA center, pretty straightforward title for very ambitious goals, around training and technical assistance to agencies all throughout health and human services to make policy, research, and grants more equitable.

So that's going to very much bring equity and evidence together because it encompasses everything those agencies do, right? Whether it's policy implementation, whether it's grant making or just research. So we're very much looking forward to that and to the broad purview of being able to look across all health and human services agencies. I think for work again, we have a lot of examples. We think there'll be ongoing focus on training and especially looking at people who were more displaced by the pandemic. There will be an ongoing focus on the registered apprenticeship program. Another center we're propping up is one of the, there were four regional registered apprenticeship TA centers of excellence issued across the country and Mathematica is part of a team on one of those. And the idea there is again, using data and performance measure and best practices, we'll be in there to provide TA to help agencies who have articulated.

They, they have some struggles implementing a register apprenticeship program, helping them with their administrative data, helping them coordinate data systems and help improve the, the reporting on how they're implementing their registered apprenticeship program. So that'll continue to be a focus. And again, that's that's very much a focus in the Build Back Better plan by the Biden administration. So we're very much looking forward to that work. And the, the last focus, which again has been a focus for the last few years, and I talked about kind of how it's intersecting with equity in all of our work, but the Evidence Act, which was a federal decree for all federal agencies to make sure that they were building in for all the programs they implement, that they were building in a learning agenda for what is their intention for how their programs are delivering the services they're delivering and then how would, you know, if that happened?

So everything from just thinking through, alright, what's your learning plan? How are you going to think about what everything from what outcomes are you trying to affect in your program? How would you measure those and then how would, you know, if it was achieved? So everything from, you know, in the old days, the learning or the measurement was all what we called performance measurement, but it was very much like, okay, you're supposed to serve, this program's supposed to serve this many clients and you know, they're supposed to do these things, but it was less focused on really thinking through for all of your programs, well, what's your logic model about how this is to work at all, right? What kind of change are you trying to affect and how would you know that you're doing that?

And federal agencies are in all very different places for how they think about that. And we have engaged with more agencies around, I'd say there's kind of two sets of activities that have really increased for us in human services. One is just working directly with agencies on those learning agendas for all their different programs. So we have work right now within ACF for all of their offices on this exercise of, okay, what's your learning agenda? What programs are you trying to implement, what's the change you're trying to affect? And, and how would you measure that? So we've started with a few programs within ACF and we're expanding with Administration for Children and Families, and we're expanding that work. We've also been doing this work for a few years with the, with CISA, the Cybersecurity and Infrastructure Security Agency and they protect everything from our elections to all sorts of other ways that our cyber security can be threatened. So we've worked with them on some of their learning agendas. So for some agencies, it's really that very upfront. What are your programs? What are you trying to accomplish? And how would you know it, but for other agencies, it's moving to that next phase of, okay, we know we're trying to affect these kind of changes. This is, this is the outcomes we would look like, oh, we don't have have data on those outcomes here. We think it's out there, but it's kind of all over the place with our hundreds or thousands of grantees and agencies.

So increasingly we are doing work with agencies, we're doing it within the Office of Population Assistance within Health and Human Services. We are doing it at SAMHSA, the Substance Abuse and Mental Health Services Administration. We're doing it at NSF where we're building them the system, we're building them a system where their grantees and their agencies can put their data in. Right. So that way does two things. It helps the federal agency say, okay, here's, here are these programs we're trying to implement and here are their data. So we can start, you know, looking and seeing based on our learning agenda, these are the changes we are trying to affect. Is that happening? It also lets the grantees have a place to put their data. Again with the hope and the goal that not only do they is improve learning for them from their own data, but you can look at the data across several grantees or several agencies.

So that for us within human services is such a, and I've, there've been themes throughout, right? Child, welfare agencies, other agencies, the importance of even having, kind of consistent and integrated data to understand if human services agencies are delivering services, the way they intended are having the impact, they and really starts with that. Right? So all these diffused data streams out there, making sure or figuring out ways or providing support to bring them together so that the learning that needs to happen can happen. So, fueled by lots of things, including the Evidence Act. We're seeing much more of that with our federal partners and working with our state and local partners to make that happen. And so that's very exciting to us because that's, what's really going to let us all be advocates for evidence, right? We can all value evidence, but if nobody has their data to figure out what happened, it's going to be hard to be evidence based. So that's really exciting work for us, with our federal partners, with our state and local partners, even with our foundation partners. So we're really looking forward to that next year, too.

[J.B. WOGAN]

Okay. Well that, I mean, that, that already is giving me some, some reason to be optimistic about the year ahead, but I just wanted to ask if there was anything in particular, anything specific where you find optimism in public policy, in public policy research in 2022.

[JILL CONSTANTINE]

Yep. I find a ton of optimism in that focus on equity. Just the acknowledgement that there have been barriers, institutional, structural, that are built on racism and other sorts of disenfranchisement, the fact that that's being so acknowledged now that, you know, when you're looking to a program, what worked, what didn't work, you know, you're supposed to look at that, right? That we got to take that on head on again, in the federal space, before 2021 that just wasn't even allowed. Right? So it's a sea change. And you know, we weren't going to improve services. We weren't going to improve engagement if you were kind of turning a blind eye to any aspect like that. So we're very excited about that. We are very excited as difficult as the pandemic has been, we have learned so much about how to manage through a disruption like that.

So, you know, I started with schools, schools have been a relative success stories. They have not been in largely the sources of community spread in this country. Now some of that's just the public health, the younger students appeared to be less susceptible, of course, but you know, the fact that we could figure out mitigation strategies, the fact that we could move learning remote and get any learning done at all. There's a question there's going to be a lot of work to do this year to see what kind of interrupted learning occurred and, and what is very likely the inequities in how much ground some students lost. That'll be something we're all taking on in 2022, but we've learned a lot about delivering all kinds of service to children and families, including under these difficult circumstances, like, you know, remotely.

And we've learned, because we had to, we also learned a lot about how to keep wherever these services are targeted, children, families, youth, engaged in the service. Right. And that's very exciting, the amount of technology we all used to go to school, to deliver services, to observe what was going on in schools and at agencies. I don't think that's going to change. I think people learned, you know, the example I always like to give when we did our work with the New Jersey Department of Education. Now, when teachers wanted to observe another teacher, you weren't restricted to somebody in your building, right. You just got on Zoom and dropped in their classroom that way. Right. So that's, that's great to be in that and we can always do that. Right. So, those kind of examples and the ways of what we learned this year about how it could obviously help us if we're still managing through aspects of the pandemic, but can help how we deliver and how we use technology to improve services in the future. So I'm optimistic about that too.

[J.B. WOGAN]

Okay. Right. Well, that's, that's a, a positive end, a note to end on. So Jill, thank you so much for talking with me today. And I look forward to doing this again in 2022.

[JILL CONSTANTINE]

Thank you, JB. It's my favorite year end weekend that I do.

[J.B. WOGAN]

Thanks to Jill Constantine for her reflections and forecasting about human services. Our final interview for this episode is with Chris Trenholm, who oversees the Health Unit at Mathematica. As one might imagine, the pandemic has kept his unit quite busy: we’ve worked on the vaccine rollout, COVID-19 testing, contact tracing, and a host of other health issues related in one way or another to the pandemic.

In general, our health unit works with public and private sector partners on improving programs, such as Medicare and Medicaid, and employing health information education technology to address the quality, efficiency, delivery, affordability, equity, and financing of health care. Chris appeared on the podcast last year to discuss the intersection of the pandemic, growing concern about health inequity, and the increasing importance of digital technology in health. We touch on some of those same themes in this conversation. Here’s my interview with Chris.

[J.B. WOGAN]

I did want to start by talking about some of the ways in which evidence is improving people's lives. And, you know, I think it's, it's easy to talk about interesting research findings, but I don't want to get, you know, trapped in sort of talking about evidence for evidence sake. Like, can you think of some examples in the past year where you've seen research and evidence or data that was generated by Mathematica that actually improved public wellbeing and drove impact?

[CHRIS TRENHOLM]

I like your preface about evidence and how we think about evidence. This, this organization was instituted and has continued for decades in work in creating evidence and research and information that helps decision makers improve the decisions that they're making. And that continues. The work that we also engage in increasingly is taking that evidence and working alongside decision makers to strengthen those implementations. And if I think about this year, we've engaged in both in really, really important ways. I think the nature of this past year has been so much about implementation, trying to respond to issues that confront us right now and how can the Mathematica as an organization and the people in it work alongside that decision maker as an evidence based evidence driven organization to help strengthen those decisions. And there's lots of places we could go. I think the one that I would return to is the Medicaid program. The reason why Medicaid is so important to talk about is because it is the insurance program that is literally designed to provide health care coverage to the most vulnerable among us, to frail elderly, to disabled populations, to people whose health care crises have led them to have large loss of income, to children and families that lack the means to insure themselves through other means.

So it it's, it's an incredibly powerful safety net and in the pandemic, I can't think of a more important safety net that exists in the United States. And so the work we've done alongside organizations, states and the federal Medicaid program and a number of agencies that are connected to those populations, I think has been really important. And a couple that I would call out. One is that we continue to do work alongside states and the Centers for Medicare and Medicaid Services to implement a program affectionately called MOM that is this designed to fight the opioid epidemic by supporting the health and wellbeing at the community level of pregnant women in postpartum women that are suffering from opioid suffering from the opioid epidemic. And as we know that epidemic, it continues through today. And in many ways, it's now silent because so much of what's going on is happening virtually in our lives.

So it's as frightening as it's ever been. And I'm not sure we'll understand the magnitude of what's going on today and frankly until the pandemic eases. And so it's a scary time and we've worked very hard to see that that program is implemented as well as it can be, building the learning communities and acquiring the data and information that we can reflect back in those communities and those decision makers to help them understand where the need's most acute, how can those needs be improved, what's working, and how can you strengthen that MOM model on behalf of those vulnerable women and their families. So very proud of that work. And it's been very important in gratifying. A second example of that is we're working directly for a state right now, putting together monitoring and quality assurance program to support state's efforts, to help individuals who are suffering mental health disorder and are leaving institutional care settings and moving out into the community.

It's an incredibly vulnerable experience again, during a profoundly vulnerable time. And so again, having the evidence, having the data and the information available that stakeholders, states, decision makers, local communities can use to help support that experience and try to see that those people leaving those institutions continue to be supported is very gratifying and very important. I could keep going J.B., you know, maybe the third that I'll just state I'll, I'll kind of wraps into the same narrative of, of working inside the pandemic. One 1, 1, 1 recognized policy issue that that is going to arrive, that's very important is, early in the pandemic, there was a public health emergency declared and in the public health emergency individuals who were insured by the Medicaid program were assured their continued coverage.

So regardless of whether there was an established eligibility, they, the, the focus was let's ensure continuous coverage. Let's ensure continuous access to healthcare through that insurance program. And so that's been going on for some time now, and it's been incredibly important. It's assured that millions of those most vulnerable can make and 10 continuous access to providers and the, and the healthcare system when the public health emergency ends, which hopefully will end as soon as possible. There's a tremendous risk that those millions of Americans will find themselves uninsured. They'll find themselves with gaps in healthcare and gaps in insurance, cuz those the, the public health unwinding will end and they, in many cases is may no longer be able to acquire eligibility quickly enough. And so we're working now with the Medicaid program federally and, and engaging in early days with states to prepare them as well as we can to identify and support smooth and successful transitions into maintaining the health insurance coverage. For many of those likely millions of people, that would be affected by the, the end of the emergency. So I'm just taking one piece of kind of a tour of the overall health unit. But I, I, again, I think the Medicaid program is such an important piece of the safety net right now.

[J.B. WOGAN]

One of my favorite projects from this year from the Health Unit was about messaging around vaccine confidence and making sure you have the right person with the right message for the audience you're trying to reach. And it came out of New Jersey. It was specifically focused on communities of color, but it had relevance across the country and had some practical insights that would help with solutions if you are a public health official trying to figure out how to persuade your residents to get vaccinated.

[CHRIS TRENHOLM]

You're, you're exactly right. And again, I just, we, we, I think we had the opportunity to talk about this last year. You, you know, the, the pandemic has just laid bare the truth of health care equity and, and inequity generally in the United States in ways that are profound. And so the kind of lesson that comes out of work like that I, I think is is just so important. And, you know, it does tee up another narrative that I think has been with us throughout the last year and will continue to emerge next year, which is, health equity and, and work in equity and social justice.

[J.B. WOGAN]

I did want to ask, and we've talked about the pandemic. I think you're touching on what is clearly another theme in terms of issues and problems that we try to address that partners try to address? I mean, I think one, I think is also interesting about where we work is that you, you kind of have this measure of what our problems of significance by the RFPs that come out and the request for proposals, like you kind of know what federal agencies and foundations think are the most pressing problems through that process. So what kind of rose to the top for you and for health this year?

[CHRIS TRENHOLM]

Yeah, I I'll just take 30 seconds on that point, because it's so it's such a delightful aspect of working here because I remember leaving graduate school and thinking about whether I would pursue a career in academics or pursue a career like this. And the thing I concluded for myself is I'm able to think about questions that I think are important and pursue a path to answering them. But one of the remarkable things that happens here is you have hundreds and hundreds of questions, if not thousands of questions that people out in the world are experiencing and looking to find the best answers that they can . So for someone who's looking to answer complicated questions in places that can improve public wellbeing, it's a blessing I can't even describe just to have a chance to respond. So, this year has been more of that than ever.

And in addition to our federal work and state and foundations states is very much the case as well, obviously. And then we've had some very important clients in the private sector space that recognize likewise, the importance of evidence base in improving the mission of public wellbeing and servicing, you know, the, the cause of Mathematica and reaching its destination of, you know, equity and social justice for all of us. And for example, we've actually worked with pharma on their concerns around and vaccine hesitancy and how to, kind of what what's possible in addressing concerns on vaccine hesitancy and kind of where, where do those issues emerge from. And it's been a wonderful partnership and those really came out of questions. Again, that were being asked of organizations that are coming from that industry, not a state or foundation.

So anyways, I just wanted to totally affirm that that's an incredible experience working here and being able to see all those come through, I would say thematically, one of the things that shows up in every one of those spaces. And, and I think we know this is, is I'll say the word equity and for us it's health equity. And you know, it, it's a word that has a very whose meaning is important. And I actually think if you went up on Robert Wood Johnson Foundation's website and just Google health equity, I think they offer a really marvelous definition. There's definitions out there. I, I think people can acquire their own understanding of how they think about the issue, but, but I think for us at Mathematica, you know, we've been on our own journey in understanding, we've always worked with questions around health disparities. We've always recognized and been asked to bring evidence to bear that address issues of inequity.

But I, I think we've come to understand our own needs to grow as an organization in learning more about what it is to address the cause of health equity. And I think for an organization that has centered itself in evidence, I think we've, we've recognized that have to look at ourselves and see where our own lived experiences have come up and where our own biases or lack of understanding may emerge because the fabric of what it means to move inequity in the United States is very complex and Mathematica does not reflect nor is representative of the United States So Mathematica's ability to engage in a conversation about health equity must work in service of others whose expertise and lived experiences closer to populations, Black, indigenous, people of color for which we bring inside the organization, certainly bring lived experience, but as a collective don't represent those experiences as well as other organizations can and do.

And so we have, I think we've moved in the last year or two with a humility in recognizing the learning that we need to do. We've invested in DEI inside the organization. I think in ways that have unfolded very important discussions about what it is to be a diverse, equitable and inclusive Mathematica. And that's a journey that continues, but it's, it's also led us, I think to understand that one of our responsibilities is to serve other leaders and other organizations who are deeply expert in health equity and come from that lived experience and their causes about that lived experience. And that's, that's a different relationship with, with organizations than maybe we we've thought about before, where there's a question that's important. We, we, we build a relationship with a, with a partner who can help us and we work on that question. It's a different relationship to that question to say, no mathematics responsibility is to serve.

That might what might be a much smaller organization to strengthen their ability to empower them in answer question. I think empowerment is something that we've come to understand is so important that we need to really be allying in servicing other individuals and organizations who are close to these issues. And I'd like to say, we've always done that, but I don't. I think we've looked at it in a way that has been more about the science of how we answer question and not about what it means to bring a health equity lens to these questions. So, so that's been a very important experience that will continue into next year. I imagine it'll experience it continue throughout my career. I will say within that we've had opportunities to do some really remarkable work. And one of the things that has been really gratifying right now is we've had, we have an opportunity to build a equitable training program or partner with our department of health and human services and others on a training program that will, I believe in ultimately be something that will be shared with 70,000 plus employees of the department of health human services.

And I'll, you know, that's not a, that's, that's a, that's a small part of what again, is a journey that DS is making, but it's been, it's been something again that I'm very proud that we've had a chance to be a part of. And, and, and we, we contribute to these efforts and we learn from these efforts. So that's another important space. I think that we're, we're learning and engaging and hopeful progressing.

[J.B. WOGAN]

All right. So the pandemic and equity are obviously issues of 20, really 2020 and 2021 that we can expect to continue to be part of our experience and society's experience in 2022. Are there other emerging issues that you're watching for that you're looking out for that we'll likely be trying to help address through evidence?

[CHRIS TRENHOLM]

I, I think where my mind goes to isn't so much a program or policy or even an issue, it's probably the transformation that's, that's happening right in front of us, which is the world becoming increasingly digital. I don't even know if we've come to terms with it yet. Mathematica, again, is in its own progression in recognizing the importance and contribution and, frankly, the risks presented by digital transformation.

And so we're very engaged in that. And I think that engagement is only going to accelerate and I, I could pick up examples across all three of the risks have been spoken of in different settings, but you know, a, a machine is taught by a human. As we talked about earlier, you know, humans are flawed. They, they bring a range of biases and risks in training that machine, that machine now automates itself and begins to provide service. Well, it, it could, it could harm and there's yeah. There's evidence in use of artificial intelligence in the justice setting. That's been spoken of extensively, where the, where the risks are that you essentially apply the inequities of the justice system into the machine. And then at what comes is further injustices that now are a little bit lost because you're part of a machinery. That's one of what will be, I think an emerging string of examples.

That's the risky side. The promising side, I think, is the power of technology to change lives for good and, you know, J.B. That this gets distilled down to data science. So we've talked a little bit about this. I think in the past, from, from my perspective, data science is often talked about as if kind of a Einstein figure. Who's the data scientist who can, you know, who, who in and of themselves can, you know, form. And I'm not saying we have, we have brilliant data scientists at Mathematica who marry the capability of a data scientist. But for me, when I think about evidence-based work in the 2020s and the 2030s and forward, I think of it, I would reverse it and say, it's about the science of data. What's emerging is the science of data and the ability to marry technology and computer science with advanced methodologies and statistics, which is about, you marry those two and you bring artificial intelligence and machine learning forward, the critical marriage then to bring data science to life for the science of data is you have to marry with expertise about the world programs, people, issues.

And for us particularly issues that confront again, the most vulnerable among us, that's the expertise that enriches the data and the expertise of technology and, and method. And that's what brings that, that's what data science is the marriage of those three things. What's exciting for me and for us, I think at Mathematica is we are so relentlessly collaborative. We look at a problem and we recognize no one person can answer that question as well as it could be answered by a team, by a collective. And that team is strengthened by inclusion and diversity. So when we think about data science, we don't really think about a person who's a data scientist. We think about the science of data, how to bring that team together, those technologists, those data experts, those clinicians, those Medicaid experts, those people who, who bring the lived experience of health equity together to solve that problem, to create the necessary data, to translate that data into information and to put that information in front of decision makers in ways that they can understand and can help.

And I just think the power of that because of what the machine is being to offer is unbelievable. And we're only being to learn how to harness that. So that's that, that continues. And we've got just to give you a small example, working with large cloud platform providers is, is important because they have so much data and they have so many powerful machine tools to extract and work from those data. So we've begun to put ourselves up onto cloud. So for example, on Google cloud, in their Google health cloud environment, and to essentially open a marketplace that says, we really understand, say a health care provider system, how to take the data that are sitting up there in electronic health records, in other settings, you know, safe and de-identified and appropriately managed, and translate those in real-time into helpful decisions. And to do that in an ongoing way.

And that's a very early example of where I think we can work, not in, in kind of a transaction way in a project, but, but in a very real-time technology based-setting to work very, very rapidly. So, you know, I think the pandemic is an overarching issue. I think equity is an overarching issue. I pinned Medicaid in this discussion because it's such a centerpiece of the safety net and where we work as an organization. And then I would pin technology and digital as a force of change and one that I'm hoping we can contribute to use it as the most powerful force it could be for the good that it could provide, addressing the risk that it presents for all of us.

[J.B. WOGAN]

Well, you anticipated where I was going to go with the last question. And I think you just naturally are always thinking about both problems and solutions, but I did want to ask, you know, if there were any areas for optimism in 2022, something where you, you see that our, our ability to craft solutions and address problems is going to be bolstered because of evidence, because of data, because we know what works now.

[CHRIS TRENHOLM]

Yeah. And it probably, it probably relates to that last topic. One of the places where I can see a lot of optimism is health care data. Right now, health care data are siloed in ways that are remarkable for how apart they are. Some of that is because different, you know, different private sector entities hold those data and house those data and, and apply those data for private interest. I mean, they may serve a public wellbeing, but they're, they're not just simply going to give up those data. Other cases there's, you know, there's government entities and other organizations that just historically they've been sitting in, in vertical silos and they're hard to connect up because you might have a legacy system, one and a legacy system two and a three and they don't talk to each other.

And then of course there's all sorts of other actors out there sitting on really important kinds of health care data, individual providers, other organizations, and overarching that is a deep and valid concern about privacy and misuse. And so that, you know, that that is beyond appropriate, but it also encourages, you know, not bringing those data together. And I think that there are emerging opportunities because of data interoperability, because of the questions that we now recognize can only be answered with with marrying up very complex data. I think there's opportunities in the public health sector in particular, to address better the questions that have always vexed us. So you take health equity and COVID for example that dashboard that we're able to produce looking at how different parts of the country counties and cities have invested their resources and service of the public health emergency and vulnerable populations.

And I don't think the information we brought together to produce that was as readily available or as easy to bring together even a handful of years ago as it was for us now. In fact, it was relatively easy for us to bring that together now. So my point is it didn't require Mathematica did it. I don't think it required a Mathematica to do that. Others could do that as well. So this democratizing of data, the ability for data to communicate with one another, our ability to leverage those data for information, I think, I think we're in a next leap of that. And what, what is evidence after all? Evidence only comes from information and data. So the, a leapfrogging in the accessibility of data and quality of data leads to a leapfrogging of evidence and a leapfrogging of evidence, I believe leads to a potential leapfrogging in, in public wellbeing.

That's why we, that's why Mathematica's committed to that relationship. So I think the story is there J.B., I'm very, very, very optimistic about constant improvements and the quality, accessibility, and interconnectivity of data, and therefore incredibly optimistic about the power of us to produce better and, frankly, more equitable evidence. And then, in turn, provide information and findings and partnerships to decision makers that's more informed and evidence-based than it's ever been. So I can't imagine a more, you know, I can't imagine nothing's more exciting to me at Mathematica than that. So I'm very optimistic about what the power of evidence can do going forward.

[J.B. WOGAN]

Okay. I think that's a good place to end the, the interview. Thank you so much for talking with me today, Chris.

[CHRIS TRENHOLM]

J.B., thanks.

[J.B. WOGAN]

I want to thank all three of my guests for coming on the show, Adam Coyne, Jill Constantine, and Chris Trenholm. We'll post this podcast on the Mathematica blog, along with lots of links to the research and tools that Adam, Jill, and Chris referenced in our interviews. A full transcript of our conversation is also available on the blog. As I said before, we just published our annual year-in-review feature, which we call Inquiry to Insight in 2021. I’ll include a link to the page in the show notes and I’d greatly appreciate you’re giving it a read.

If you're a fan of this podcast, please share it widely. We want to make sure we're reaching everyone who cares as much about objective, data-driven policy as we do. You can also help us by giving us a rating and review. We're on Apple podcasts, Spotify, Stitcher, Google podcasts, and pretty much everywhere else you find podcasts.

We'll keep covering interesting work from Mathematica and our partners on the show, but another way you can stay up to date on everything Mathematica is doing is by following us on Twitter. I'm at J.B. Wogan. Mathematica is at MathematicaNow.