Inside Arizona’s Investment in Evidence-Based Foster Care Prevention

[KATHERINE GUFFEY]

Ultimately this evidence is important to us so that we're giving families the assistance they need to keep their children safe and well at home, and hopefully prevent any involvement with the Child Protection Agency.

[J.B. WOGAN]

All right, so I'm J.B. Wogan from Mathematica and welcome back to On the Evidence. Before we start the show, I want to remind listeners that we'll be recording our first live podcast on May 2nd at 2 p.m. Eastern time on LinkedIn Live. I'll be interviewing Tina Rosenberg about the role of data and other evidence in supporting solutions journalism. Tina's reporting has earned her both a Pulitzer Prize and National Book Award. About ten years ago, she co-founded the Solutions Journalism Network, an independent nonprofit that advocates for an evidence-based mode of reporting on the responses to social problems. The Network challenges journalists to look for data or qualitative results that show whether a solution they're covering is actually effective. We'll explore what solutions journalism is and how researchers who evaluate policies and programs can contribute to evidence-based reporting about solutions. And you can help. In the show notes I'll include an email address that you can use to submit questions in advance. You can also just join the live recording and type questions into the chat. Again, that's 2 p.m. Eastern time on LinkedIn Live. Okay. Now for this episode I'm talking with Katherine Guffey, Meg Dygert, and Allon Kalisher. Katherine is the Executive Consultant to the Director of the Arizona Department of Child and -- excuse me -- to the Director of the Arizona Department of Child Safety. Meg is the Senior Policy Associate for Child and Family Wellbeing at the American Public Human Services Association. And Allon is a Senior Researcher at Mathematica.

We're talking about a promising child welfare program in Arizona that a recent Mathematica evaluation showed can reduce the need for investigations for maltreatment and reduce the likelihood of children being removed from their homes. There's some nuance about when the program is effective and when it isn't, which we'll talk about. This podcast is being recorded about five years after the enactment of the Family First Prevention Services Act, which is important context for the conversation. We'll talk about how that law sought to encourage states to invest in programs like this one that focus on preventing children from being removed from the home, and how the law made evidence of effectiveness a requirement for states to receive federal reimbursement for child welfare programs. I hope you find this conversation useful.

So, Meg, I want to start with you. We'll be talking a lot about the implications of the Family First Prevention Services Act, both as it relates to this specific program in Arizona, but also in the context of other state programs. Could you give us some background on the law? Like, what does it do and why did Congress enact it?

[MEG DYGERT]

Yeah. Thank you. And Family First was enacted in 2018 and going forward the Family First Prevention Services Act, I’m just going to call Family First because it is a mouthful. It is one of the most significant federal child welfare legislation really since the establishment of the Federal Child Welfare Program in the early 60s through the Fleming Rule and the Aid to Families with Dependent Children Program. Family First changes the federal financing structure for child welfare programming by allowing states, territories, and tribes the option of using federal funding through Title IV-E for evidence-based prevention programs, offering the opportunity really for the first time for agencies to use this funding and target prevention services for families to prevent the entry intro foster care in formal system involvement. Previously, IV-E funding, which is the largest source of federal funding for child welfare, could only be used to help for the cost of foster care maintenance payments for eligible children. Program expenses associated with that, training for staff, and other assistance costs associated with children who are already in the foster care system with formal system involvement. And Congress passed Family First to open up this additional federal funding options for states to really align with investments that states, territories, and tribes were already targeting to really narrow the door of child welfare and instead provide services for families that support them, so that children can stay within their homes and avoid the trauma that we know is associated with foster care and formal involvement, which we also know disproportionately affect BIPOC families. This legislation was the first to set the tone federally for reimagining the role of child welfare systems and how it interacts with families.

[J.B. WOGAN]

Allon, Meg just sort of gave an overview of Family First. Is there anything else that you would like to flag for our listeners about the law that they should know before we get more into the nitty-gritty of -- of the conversation and this specific program we'll be talking about today?

[ALLON KALISHER]

Sure. I'll just add that -- that Family First, I think, provided extra motivation for states to -- to advance their goals on prevention. Most states are really thinking about this. There's been a movement in the field for a while to think about how to reduce foster care placements for children and think about services that may help avoid that. And so I think states cap -- look to capitalize on this law to advance those goals. Some have certainly found it challenging to -- to specifically build a full array of primary prevention to early intervention prevention services directly reimbursable through the act, but I think that there's been some movement in the field and from the federal government to try to help along those lines. And, certainly, you know, this is a -- an evolving -- you know, an evolving practice for the field to try to leverage this Act well.

[J.B. WOGAN]

Katherine, I want to turn to you. How has Family First affected Arizona's work around prevention services and evidence-based policy?

[KATHERINE GUFFEY]

So, first I'd say that we -- we had a foundation of evidence-based programs in 2018 when Family First was passed. Those were primarily early childhood programs that were -- were well established and serving families throughout our state. And it turned out to be some of the first programs that were rated as well supported through the IV-E clearinghouse. So that was fortunate for us and it was an easy decision to include those in our plan so that we could get that funding to support making those accessible to more families. But, you know, those -- those services primarily serve families who don't have a case opened with us, may not have had a report to -- to Department of Child Safety, and we serve a lot of families who have school-aged children and adolescents, so -- so we knew we needed to find some additional programs beyond those. So we -- we put our attention to the in-home parent skill building category of services under Family First and we -- we took what might be a -- a different approach.

We didn't start by looking at the clearinghouse and finding programs that were already well supported and building a plan around that. Instead, we started by looking at our administrative data and saying, you know, who are we serving today? Who -- who is coming to our attention? Who is getting in-home services today? Who is most likely to have their children removed? We wanted to know the demographics of those families. But, more importantly, why are we involved with them? And that would point us to the programs that will be most effective. We also surveyed our provider community to ask them, you know, what programs were they providing now? What did they see the families' needs as? What -- what -- what did they see as most effective? We talked to community members, particularly African-American and American Indian community members to get their input. And then we reviewed hundreds of programs to just find those programs that seem to have the best fit, that were feasible to implement, and looked at some of the studies that were there.

They might have not met the rigor that was required by the clearinghouse, but it still gave us some information to help us make some decisions about what would be the right programs. So, you know, we ended up picking programs that we believed would be the best in terms of the intention of FFPSA and our desire in Arizona to keep children home with their parents, and then that would secondarily meet our goal of obtaining the evidence base that we'd get that rating and federal funding.

[J.B. WOGAN]

Okay. Just -- just a flag. So you mentioned "well supported" which, I believe, one of four different potential ratings for, sort of, demonstrating that something is evidence based. And -- and I think that would be the -- the -- the best -- the best rating you could get. So you already had some programs that were there, and there's -- is there -- "supported" might be the tier below that. And then what are the other two tiers?

[ALLON KALISHER]

"Promising" and "not supported."

[J.B. WOGAN]

Okay. Thank you. Promising and not supported -- okay. I think on a planning call, Meg, did you mention to me something like only four -- there's a very small number of programs that actually met the criteria for "well supported" -- is that right?

[MEG DYGERT]

Oh, yeah. And I actually have an updated [inaudible]. I think it's fifteen now. The Imprint [inaudible].

[J.B. WOGAN]

Fifteen?

[MEG DYGERT]

Yeah. I -- I need to double check, but I think -- actually, I still have the article up. [inaudible]. There's a total of 65 programs rated through the clearinghouse that are technically eligible for IV-E reimbursement through Family First. Only fifteen of those are well supported. And for some additional context, 61 programs have been found to not meet the criteria. So it's -- it's a pretty stark contrast when you think, and -- and there's a requirement within Family First that states have to spend -- and I believe they -- they can scale it up in one of the transition acts that was passed, but they will have to spend 50% of all their Family First funding on well supported programs. There's really only fifteen and when we're talking about humans, which -- we're delightfully and wonderfully different and incredibly on the spectrum, those fifteen programs are not diverse enough, nor do they meet the needs of all the programs -- and the families [inaudible].

[J.B. WOGAN]

So that gets back to the point you were making earlier about this -- we're now moving to a phase where states are going to need to expand the pool of programs that meet the -- the evidence requirements because, at the moment, there aren't enough programs that meet the evidence requirements to meet the needs of the residents of -- of states.

[MEG DYGERT]

Yeah.

[J.B. WOGAN]

Is that right?

[MEG DYGERT]

To really be, like, diverse and really targeted in -- in population, and really it -- because these standards at this kind of rigorous have never been applied to programs like this. And it's also -- it's incredibly hard to measure what outcomes look like when you're talking about human beings. And especially when they don't have, really, a say in what does success look like.

[J.B. WOGAN]

And so, Meg, as you were listening to Katherine talk about the Arizona experience, how close -- how closely does it resemble the experience of other states that APHSA serves? Your other -- the rest of your membership.

[MEG DYGERT]

Yeah. Like Katherine shared, states have been working on evidence-based programs for different populations, and particularly zero-to-three and home-based and for populations of young families. So what Family First really did was increase the federal investment and try to align with states and really unite, and hopefully be able to pull in more families and affect more families with being able to add additional federal funds there. And then states could also prioritize and use their sequent funding for more innovative uses that might be covered under Family First yet. So -- really similar to what Katherine shared in Arizona, states had been thinking about and for a long time using state and local funding to think about how they can narrow the door to child welfare, and what programs really could be effective. And how the evidence and data plays into that in their decision making, to which programs they deploy and which populations they target which is so specific to each state and their populations -- and even down to county to county can vary. So, really, an exciting opportunity to align federal funding with some great work that [inaudible] localities were already doing.

[J.B. WOGAN]

And, Katherine, how is it -- how important is it for a state like Arizona to gather new evidence of innovative prevention programs? For listeners who may not be aware of the Family First evidence criteria and how it might affect a state's funding. What are the stakes for a state like yours?

[KATHERINE GUFFEY]

So I think, you know, ultimately this evidence is important to us so that we're giving families the assistance they need to keep their children safe and well at home, and -- and hopefully prevent any involvement with the Child Protection Agency. We -- we took an approach of -- of -- with the home visiting programs having -- seeking to get a plan approved that would allow us to provide services to families who don't ever have a call to us. And that -- that's ultimately what we'd like for everybody. You know, Arizona is also home to 22 Indian Tribes. We have a growing immigrant and refugee population. We have an African-American population that is disproportionately affected by contact by Child Protective Services. And so we have a responsibility to meet the needs of these diverse families with programs that are really meaningful to them and effective. So I think the word "innovative" in your question is -- is most important because, you know, sometimes it's hard to find programs that are meeting those -- the -- the needs of -- of diverse populations and require some innovation and some support to build those programs.

So that's what's most important but, you know, then the -- the challenge, then, is to shift systems from group care to family care which is the goal of Family First, and from parent-child separation to children at home with their parents. But in order to obtain that federal funding to support that shift, we need programs that meet Family First's evidence criteria, which is rigorous and doesn't exist for many of these innovative programs that may, in fact, be the best fit for those populations. So -- and keep in mind that, at the same time, Family First reduced financial participation for the care of children living in group care settings which then reduced our flexibility to fund in-home and kinship support services. So, again, what this led us to do was to say we're going to identify programs that we thought would give us the best success in achieving those goals, avoiding the cost of foster care and especially group care, and then we could reinvest those savings in developing the evidence base that was needed to get the federal funding and then -- and then create a virtuous cycle where we could then build on those programs that we had implemented.

[J.B. WOGAN]

So we're going to be talking about one of these parent skill building programs today, specifically a program that Mathematica evaluated in Arizona called the "Nurturing Parenting Program: Nurturing Skills for Families." So, Katherine, would you mind talking a little bit about what that program does, how it works, and why you sought an independent evaluation of it?

[KATHERINE GUFFEY]

Sure. So, yeah, Nurturing Parenting Program is a parent education and coaching program. It can be provided in a group setting, but -- but primarily in Arizona we provide it in-home with individual families. Parents receive curriculum-based lessons and coaching during planned nurturing activities that occur with their child. And then, after that nurturing activity, there is a coaching session to reinforce those nurturing skills. All of that is focused on core competencies for parents, of expectations of parents for children, parent empathy towards their children's needs, nonviolent discipline, parent/child family roles, and children's power and independence. And so all areas that sometimes bring families to the attention of -- of child protection agencies. So that service starts with an assessment of the current parenting attitudes and behaviors, and then lessons are chosen, individualized for that family, and could -- could even be specialized lessons for African-American families, Native American families, parents who identify as LGBTQ+ or other groups. Those sessions occur typically every week for 17 weeks. And -- and we were using this program in Arizona well before FFPSA which is a part of the reason that we were able to -- to do this evaluation. We had data that we could use from that historical provision of services. But we had identified that it was a good fit for Arizona, especially given its ability to be flexible to adjust to the family that's being served. We knew it was the best choice, but there weren't evaluations of the -- of the rigor to be reviewed by the clearinghouse. So there were evaluations, but they didn't meet those standards of rigor. So that caused us to need to -- to get an evaluation. Again, we -- we chose the service based on what we thought was best for families and -- and with the -- with the belief that it would meet the -- it would -- it would get its evidence standard in the end. So the fiscal support was a motivation, but really I think, more than that, we were seeking confirmation that our choice had been correct, that NPP would have measurable benefits to families in Arizona and that when we offer that service that, you know, we feel like we have the responsibility to know that it is likely to help a family. You know, asking them to trust us that we're providing a helpful solution and so, you know, I think we have an obligation to be worthy of that trust by offering services that have evidence of effectiveness.

[J.B. WOGAN]

You -- you mentioned that you did have some data, just not of the type of rigorous research kind. I'm curious -- what was it in the data that led you to -- that gave you this intuition that this is a program that works and that would meet those evidence standards? Is there anything in particular that stood out -- stands out in your memory?

[KATHERINE GUFFEY]

You know, I wouldn't say it was the -- the data that we had. We had read some other evaluations from other people's data that gave us some -- some hope in that direction. I think that it was also hearing from our provider communities about the experiences that they had had providing that service to families here.

[J.B. WOGAN]

Okay. All right. Perfect. So, Meg and Allon, I want to turn to you now and hopefully you can put this specific parenting program in Arizona into context. Like, do similar programs exist in other parts of the country? And for listeners who aren't enmeshed in child welfare policy, what aspects of this program in Arizona strike you as unusual or even innovative? And, Allon, perhaps we can start with you.

[ALLON KALISHER]

Sure. Yeah. I mean, first off, I -- there are lots of parenting skills programs and home parenting skills programs across the country, many that are already rated on the clearinghouse and so I -- so I don't think that it's at all uncommon to have a model like this. But what's unique about it is two-fold. First of all, as Katherine was describing, there's a lot of flexibility within this model that I think is not necessarily evident in some of the other parenting skills models. What's also really unique is that when you have so many models that are out there and many which are already rated on the clearinghouse that would guarantee you some level of federal reimbursement is -- and maybe I shouldn't say it's unique, but I think it's certainly a [inaudible] that -- that Arizona chose to pursue the program that they felt fit their -- their children and families best, despite the fact that it wasn't rated. And have obviously pursued an evaluation, as Katherine said, both to -- in hope that it would get the rating that it gets for the sake of a federal reimbursement, but more importantly for the sake of knowing that what's -- what they're doing is working. And the fact of the matter is this context matters, so despite the fact that, you know, there are lots of programs and many have been rated well, there are -- it isn't to say that every program works the same in every place, so that they knew this program. They've been familiar with it and saw that it fit. I think that's the -- the uniqueness that can only be true specifically in the way that it was for Arizona and why it made sense for them to pursue this model.

[J.B. WOGAN]

Okay. Perfect. And am I correct in -- in understanding that the model itself is a national model that -- that it -- it operates in some way in other places? That this program in Arizona is not -- it's not as if it only operates in Arizona. Is that correct?

[ALLON KALISHER]

Yeah. And there's another piece about that, J.B., just to say Nurturing Parents for -- Nurturing -- NPP -- Nurturing Skills for Families is one version of NPP. So there's also the fact that -- that NPP exists in many different varieties, and this is one of many models of that.

[J.B. WOGAN]

Okay. Okay. Excellent. And then, Meg, is there anything you would add in terms of just putting this program into -- into context.

[MEG DYGERT]

Yeah. And actually adding on to what both Katherine said and what Allon said on applauding them for choosing and listening to both their provider community and the outcomes and what was working for their families and with their families and children for wanting to engage with, and what they really see is working for their populations. And I think that calls attention to the tension that exists between the high and rigorous standards of the clearinghouse and the realities of where some of the programs that exist and are operated in Health and Human Services and very commonly with Medicaid as well, too. What evidence looks like and how that's measured. And how that's being applied in kind of an academic setting when every family is so unique. And what success for every family looks like is so different. And especially making sure that the programs that we are offering are as unique and as flexible as our families and as we're seeing the characteristics of them. And also the direction that they would like to take their lives in. And so I think there's a lot of opportunity in really -- looking at what is evidence and how are we measuring it and applying data, and what does that look like that is inclusive of family voice.

[J.B. WOGAN]

Gotcha, okay. Now -- you know, Katherine, this -- this -- this question is going to tie back to something I asked earlier about stakes, but I -- I -- and I think it connects to what Meg is saying about what is evidence in the sense that, you know, I -- I guess I have -- I wonder about what happens if the evidence comes back in a way that you're not hoping for. So with -- with the Nurturing Parenting Program: Nurturing Skills for Families, what was the plan in Arizona if the impact evaluation from Mathematica turned out results that were less encouraging? Like, did you -- did you have a plan in place if what Mathematica came back to you and said was it -- it's not -- it's not resulting in fewer removals from the home, or it's not resulting fewer investigations, or, you know, God -- God forbid, it actually is -- for some unknown reason -- increasing those child welfare outcomes that you don't want to see. Yeah. What -- what would -- what was sort of your mindset about the results of this evaluation before you got them back?

[KATHERINE GUFFEY]

Yeah. Well, that was certainly a risk to our approach. Right? Like, that could have happened. I -- I -- it never occurred to me that the program would be harmful to anybody. That never crossed my mind, but -- but, certainly, that's a risk that we were putting ourselves in a place that maybe we wouldn't get that rating and wouldn't get the financial support from -- from Family First. But -- but I think that even -- even evaluations that -- that don't show a positive effect or to the degree that we would like, are still informative. And so we could provide that information back to our -- our contracted providers, to our program development staff, back to the program developers of -- of NPP, and -- and to just think about what we would learn from that is a -- is an adjustment that's needed. Is it something about contacts, like Allon was mentioning? Is it something about the fidelity and delivery of the program? Or who we're referring?

So there are many questions that we could then ask if that -- to improve and try to get to a point where we were having a more positive effect. And one of the things that was interesting that happened for us is that the evaluation actually showed that you needed to complete the program in order to have the benefits of it, which makes sense. You know, we have -- we have, you know, a -- an amount of time that services are provided for a reason. And so -- but that's really important for us because then we can say to families, hey, stick with it. If you stay in this program, it's really likely that you're going to have some benefit from that. And we can say to our providers, we really need to think about how we break down barriers to participation and make sure that families stay in the program because that's when they're going to have the benefit. We -- the -- the evaluation found, and Allon could speak more to this, is that people who enrolled and started but didn't finish didn't have that same impact.

[J.B. WOGAN]

Thank you. That's -- that's great. And, actually, Allon, I would love it. Would you mind elaborating on the findings because I realize up to this point we've been sort of talking around them, but we haven't actually talked about what those key findings are in the evaluation. It probably would be good to put a finer point on exactly what Mathematica found.

[ALLON KALISHER]

Absolutely. And I -- and I give credit to -- to the team of researchers that specifically worked on this who found that, basically, simply referring to the program didn't really move the needle in terms of outcomes. Take up and drop out issues were -- were -- were obviously a challenge for actually benefiting in the program. But those families who did stick it out for at least three to four months and completed that program found very favorable impacts, actually, on reducing investigations and preventing removals. So that really -- that does set Katherine up and Arizona up well to think about why are families not completing the program and what supports might they need. And that's really not an all or nothing, you know, kind of situation when you're looking at an evaluation of -- of did it work? Did it not work? But actually what -- what do we need to do to make it work for more or to get it working? In this case, the good news is that it does work if they complete, so they could be very laser-focused on what they need to do to improve efficacy.

[J.B. WOGAN]

What role, Allon, does a research organization like Mathematica or just an evaluator in general -- what role or responsibility do -- do -- does an evaluator have to go beyond just assessing this binary question of is the program effective or not? And actually advise on a path forward for making a program effective.

[ALLON KALISHER]

Well, I mean, I think that the answer is in part, you know, what -- what your question really lays out the answer well because in -- in fact, it's not just about does this work or didn't work, but what works for whom under what conditions. Right? So that -- that's an evaluator's burden just to -- to do a good job is to think about how -- how that local context matters for what works for whom and under what circumstances. And -- and then also to inform, really, how to scale a program or tailor it to work better. You know? And so I think that, you know, one of the things that -- that we -- we do, not just at Mathematica, not just with respect to formal evaluations, is also help agencies like Katherine's think about the -- the continuous quality-improvement efforts. And that -- that is actually part of the burden of Family First Act -- the Family First Act is -- is to have robust CQI in place, thinking about fidelity to implementation, but also how do you overcome those barriers so that you're -- you're implementing well and that you can develop strategies to overcome barriers. And so I think the evaluate -- the evaluators ought to be informing that direction, not just simply answering the -- the primary research question.

[J.B. WOGAN]

I would imagine child -- child welfare agencies are -- are pretty budget strapped. And I know in general in government, setting aside money for evaluation is -- it's always going to be competing with other needs that might be -- might seem more important for serving families. Is that another aspect? There's maybe a tension? Like, how do you actually justify setting aside money for evaluating child welfare programs when there's such a high demand on just serving families in -- in the child welfare space?

[MEG DYGERT]

I would say yes, absolutely. And to the -- the credit of Congress and our federal partners, they did pass legislation in the Family First Transition Act that -- which year it was passed is escaping me right now -- but that did set aside specific funding for states to be able to use towards evaluation and really to use that funding to get them ready to get going with Family First and -- and to implement it. But, historically, the federal funding of child welfare -- and this varies across states and jurisdictions -- has been pretty low. The IV-E eligibility rate is still at the 1996 level when [inaudible] was passed. And it kept it at that level. So only children who meet that level are eligible for federal reimbursement. So each year, as inflation rises and that stays at that point, states are picking up more and more of the share of foster care maintenance payments as well as when they're looking to use programs that are not rated or cannot meet the criteria on the clearinghouse because, you know, we're -- we're struggling with what that evidence means, especially when we're applying to humans. They end up using more and more state and local investments to use those programs like concrete and economic support to be able to really stabilize and get families on a solid foundation.

[ALLON KALISHER]

J.B., I just wanted to add -- first off, the Family First Transition Support Act was in 2019 and -- and was an important -- yeah, it was an important boost to try to help states get ready. But it's also time limited. So if you think about this long term, the hope, then, would be that there's a robust network, a robust library, I guess, of -- of evidence-based services that states can choose from. And to Meg's point, with fifteen that are well supported, the burden that you were speaking of, J.B., about states to think about investing money in evaluation as opposed to services is real because if you don't have an evidence-based service, not only do you have to have 50% of spending on evidence-based, but you -- you -- you cannot waive the evaluation burden unless it's on an evidence-based service. So -- so if you're looking at the supported and promising, there's still a requirement to do evaluation. And so, yeah, this is -- this is the tension that exists with this Act, and it's growing pains, I think, for the field. But, as I say, I think Congress has been listening and -- and has implemented some, I think, efforts to try to address some of those points. And maybe there's still some more to go. So I hope, certainly, states are going to continue to push for what's necessary to build the right evidence base for the families that they're serving.

[MEG DYGERT]

I want to -- I want to make one more point, too, regarding Tribes, because I think it's a really -- it underscores this point perfectly, too, is that I believe that there's only -- Family Spirit is the only program that is currently -- there might be one more -- that is rated specifically for tribal families, Native children, in Indigenous populations. And Family First specifically calls for programs to be culturally inclusive and diverse. And when there's only really one or two programs in the clearinghouse that -- that states can use for their tribal populations that they partnered with, it really limits the effectiveness. And it's kind of that perfect example of -- these are pretty -- these are small populations but, you know, they're pretty widespread. And many states have these populations and administer services for them so, you know, how are we applying evidence to those diverse populations, and how is it not working, is really a question that we need to ask moving forward.

[J.B. WOGAN]

So -- Katherine, I want to turn it back to you. What is the state's planned approach for building the evidence base around preventative services in the future? I mean, you've had the success with this specific evaluation and this specific program, but can you kind of zoom out and give us a little bit of a taste of what -- what's to come for Arizona?

[KATHERINE GUFFEY]

Sure. So I think it's twofold. One is we have another program that we need to build an evidence base for that is already part of our service array. But I think we also want to go beyond that to support programs to get an evidence base that -- that aren't yet in our service array. You know, I think that some of the most innovative and interesting programs are -- are built, sort of, grassroots from the communities that they're designed to serve. And we're looking in that direction to see where we can find those that we think will be effective and -- and address some of the issues that -- that we have here in Arizona that are important to us, like reducing disproportionality for African-American families. So -- so that's one direction. You know, because I think sometimes those programs are small and don't have this -- don't have sufficient funding to do the evaluation on their own. And -- and we can help with that.

Secondly, I think, you know, for those programs that already have a rating of "supported" or "well supported," we want to periodically gather evidence about whether they are still working, whether they're being delivered as designed, whether they are still effective. You know, the context can change. The people that we're serving can change. So we need to be mindful of that, I think, and continue to be looking for evidence that we have the right programs. And sometimes we need to change. The other thing that we've been thinking about in relation to those existing programs is rapid-cycle evaluation and -- and using evaluation tools to give feedback to our providers at a regular interval so that we're improving the quality of services that -- that we're delivering.

[J.B. WOGAN]

Let me ask a question. This is sort of a similar question in terms of, you know, perspective, future looking. Meg and Allon, what do you expect to see from other states outside of Arizona as they continue to move forward with implementing evidence-based prevention programs in response to the Family First law?

[MEG DYGERT]

Yeah. Well, with any legislation that really makes a significant shift, especially on the federal level, there's going to be bumps in the road. And the -- we're still working through the growing pains. And implementation happened for many states mid public health emergency, or they were just getting started as COVID-19. So -- and then the resulting workforce crisis, too. States really had -- had operated for a long time in an emergency management where, you know, innovative and planning for the future was put a little bit on the back burner, but was still very important. So states now, as -- I think we can say -- that the public health emergency winds down, really looking to ramp that up and reinvest, and they're going -- really leaning into Family First and evidence-based prevention programs and what that means. So I think they'll -- we'll continue to see them push innovation and think about what does evidence look like and what, you know, through -- through the standards of what's allowable through Family First, what can they really maximize?

And I think we'll also continue to see states align cross-sector with substance misuse agencies, public health and Medicaid, to serve joint populations, and really move downstream. Like Katherine said, the goal is to get to families with preventative services so they can be stable and build wellbeing and generally -- generationally build that wellbeing before Child Welfare ever needs to get involved. So I think states are going to continue to work through that whole and really reimagining the role of what child welfare looks like and how it can be a bridge to other services.

[J.B. WOGAN]

Okay. That's -- yeah, that's great. I -- and, you know, you're right. I remember well before burnout was a common talking point for other workforces and sort of the -- the American workforce in general, I remember reading about burnout with child welfare and I think it was something like post-secondary traumatic --

[MEG DYGERT]

Yeah.

[J.B. WOGAN]

-- yeah, the trauma that -- that child welfare workers were experiencing, especially as they had, say, cases where, you know, a parent would -- would overdose from opioids and things like that, and just the huge workloads that case workers had. So I can imagine it probably was just even worse during the pandemic when burnout became such a nationwide issue. Allon, do you have any prognostications? What -- what should -- what should we be looking for in the future in terms of state trends in response to Family First?

[ALLON KALISHER]

Well, I have to tell you that I actually think that Katherine is part of a leading trend and a number of things that she said, I think, are going to be -- we're going to see more and more states thinking about. One of them is -- is how do we develop more culturally responsive services? And -- and fill those gaps? How do we learn lessons from each other as -- as each state is trying to understand what works, what doesn't, not just on a -- on a model basis but understanding how that context matters. Investing specifically in internal capacity to do evaluations, and to do continuous quality improvement work so that you can make their services more effective. You know, certain states certainly have jurisdictions and particularly think about county-run states that have small counties and want to tailor services to them, but they don't have large populations. They're not necessarily poised to do big evaluations and provide evidence. They don't have the population size to generate that kind of evidence. So there's going to need to be a lot more collaboration, I think, across states. And there's opportunities for economies of scale and states to think about how they can do this jointly where they have mutual interests to develop evidence in certain areas and build, you know, their capacity to do evaluations or to create, you know, again, evidence on the clearinghouse that support their -- their jurisdiction's goals.

So I think there's going to be a lot of that type of focus. Certainly, the trends in the field are towards more and more early first primary prevention efforts, and I think we -- we've certainly seen. I remember when Washington, D.C. was the first state to come out with their prevention plan, and when it was approved I think one of the first takeaways they had was it didn't quite provide the return on investment they were hoping for. And that had to do with the maturity of the law or the interpretation of the law in terms of how you can fit the law to some of those primary prevention efforts. With more recent clarification of that guidance, I feel like states are going to find ways to be able to do more primary prevention within the scope of FFPSA., which really means not necessarily having to serve children and families through the doors of the Child Protection Agency. So I see a lot of investment in figuring out what works at that level and trying to shrink, you know, the protective services, you know, door a little bit. But, definitely, this Act has pushed states to care more about evidence and that -- that, for me, is a trend in the right direction.

[J.B. WOGAN]

Okay. I think that's a great note to -- to end on. Katherine, Meg, Allon, thank you so much for speaking with me today and talking about this specific program and the larger context of the Family First law. Thanks for listening to another episode of On the Evidence, the Mathematica podcast. This episode was produced by the inimitable Rick Stoddard. Our guests were Katherine Guffey, Meg Dygert, and Allon Kalisher. In our show notes, we'll include resources discussed on today's episode, including the Mathematica evaluation of the Nurturing Parenting Program in Arizona. As a reminder, we have a live show on LinkedIn Live on May 2nd at 2 p.m. Eastern time. Follow Mathematica on LinkedIn for more updates about the show. You can catch future episodes of a podcast by following us on YouTube, Apple Podcasts, Spotify, or wherever you listen to podcasts. You can also visit us at mathematica.org/ontheevidence.