How and Why Foundations Push to Advance Equity through Evidence

[MINDELYN ANDERSON]

Especially now, organizations are very conversant in the language of equity work or culturally-responsive work, but when it comes down to the practice of it and codifying it in policies and processes, that’s where things get a little tight.

[J.B. WOGAN]

I’m J.B. Wogan from Mathematica and welcome back to On the Evidence, a show that examines what we know about today’s most urgent challenges and how we can make progress in addressing them. On this episode, we’re going to return to a topic that we have been exploring on the show for the past few years: how to advance equity, and racial equity in particular, through the methods and frameworks we use in research.

Previous episodes have explored ways to boost diversity in fields like economics and data science, how to use address structural racism through public policy research, and how to use evidence to incorporate culturally responsive teaching practices in K-12 education. The first time we produced an episode about diversity, equity, and inclusion in the field of public policy research, we didn’t set out to launch a formal series, but it has been a recurring topic, and one that we’ll no doubt continue to investigate with guests as Mathematica’s work evolves in this area. For example, Mathematica recently developed a guide to help federal evaluation staff navigate the complex challenges of conducting culturally responsive and equitable evaluations, and we hosted a follow-up webinar to discuss lessons from that guide as well as insights from other experts in this space.

Speaking of which, while researchers are important players in making progress on equitable research, so is the broader ecosystem of people and organizations who make research possible and who use research findings to change policies, programs, and practices to improve people’s lives. So, today, we’re zeroing in on one part of that ecosystem—funders—and specifically, the role that the philanthropic sector, as an early proponent and adopter of culturally responsive and equitable research in social programs, has played in advancing the field of public policy research. I have two wonderful guests to guide this discussion.

Mindelyn Anderson is a sociologist who has studied social inequality and stratification, race and migration, education and social mobility, and health. She is the founder and principal of Mirror Group, a consulting firm that brings collaborative, participatory, utilization-focused evaluation and capacity-building to communities and learning organizations. Mindelyn, by the way, is an alumna of Mathematica’s summer fellowship program.

Kimberlin Butler is the senior director of foundation engagement at Mathematica. Since joining Mathematica in 2019, Kimberlin has been a leader at the company on incorporating equity in all aspects of our work. As you’ll hear in the conversation, she helped organize internal working groups to develop internal tools and resources that would embed equity in Mathematica’s projects, proposals, and other activities. I invited Kimberlin onto the show because she brings a valuable perspective as someone working closely with many of the foundations that are setting the agenda today around embedding equity in research in areas like food security, early care and education, and economic mobility.

On this episode, Mindelyn and Kimberlin talk about their career paths, how they became interested in culturally responsive and equitable research, the role that philanthropy is playing in centering equity within research, and how research organizations can avoid common pitfalls as they seek to incorporate equity into their work.

I hope you enjoy the conversation.

[J.B. WOGAN]

I was hoping we could start kind of at the beginning. How did each of you become personally interested in culturally responsive and equitable evaluation, and is there an origin story that you’re able to share? Let’s go maybe Mindelyn first and then Kimberlin.

[MINDELYN ANDERSON]

Sure. I love that question. You know, this term of “culturally responsive and racially-equitable evaluation,” it’s pretty new but it’s part of a long tradition of other work that is aware of systems, you know, accountable to and with communities and in service of larger goals, systems change. And so I was actually trained in this tradition since 2001.

So, I first became aware of the ways that you can use research to really be able to advance transformational change when I was an undergraduate at UCLA and involved in sociological and ethnographic research then. And then later on, after I had my PhD in Sociology and back in my first career as a tenure-track professor, had a wonderful fellowship with the Annie E. Casey Foundation called LEEAD, Leaders in Equitable Evaluation and Diversity. And that’s where I first became formally introduced to and trained in culturally responsive and equitable evaluation.

[J.B. WOGAN]

Okay. I have some follow-up questions, but let’s, before we go into those, I’d love to hear, Kimberlin, what your answer is. What is your personal connection? How did you become interested? What’s the origin story behind your interest in this area?

[KIMBERLIN BUTLER]

Yeah, the origin story about how I became personally interested in culturally-responsive and equitable evaluation is really personal. I think about growing up as a Head Start kid in an underserved neighborhood in South Louisiana, Baton Rouge specifically, and really seeing the stark contrast on my side of the tracks and the clear investments on the other side of the tracks, signaling that we didn’t matter as much. So, at some point, I noticed that education was really the great equalizer. And there was some promise in community investments anchored in culturally-responsive grant-making before I even knew what that was. I saw clearly a menu of supports anchored in evidence that worked for hard-working families like mine. I saw family members really seeking to run for office and amplify the challenges that we saw in our community. And I also saw philanthropy making the right investments in youth like me that demonstrated the promise of equitable evaluation of the intersection of philanthropy. So, even before it was something wildly known or understood, I saw it very personal.

I also want to lift up one other story which was a touchpoint in graduate school at the Maxwell School of Syracuse. When I was pursuing and MBA, I’ll never forget this social policy class that was open to students from the School of Social Work. So, you had aspiring policy wonks, researchers, and social workers all grappling with the same issues under the same roof in this class. And there was a clear lack of understanding of those most proximate to the work or the change that we were all discussing and hoping for in the world or in the communities we were discussing.

And it’s the first time I actually said, data has a heartbeat, guys. Like, we’re not talking about just a dataset. It’s real people in real communities and we really do need to understand the context where this work sits. So, at that time, we were talking about education and so I was inspired to go into the classroom and become a teacher, and engage with communities and I was talking about, in school, the power of listening to the students, families and teachers who best know the challenges that we seek to learn about. So, those two stories really sort of anchor my origin story, both as a child and also as I sort of grew in my career to focus on this work.

[J.B. WOGAN]

Mindelyn, Kimberlin was just referencing personal experiences in her childhood and some of her aha moments, both in childhood and later in graduate school. Was there an aha moment for you where you realized this specific tradition of research, that it made you gravitate towards it, made you realize this is an area you’d like to pursue as a professional career?

[MINDELYN ANDERSON]

That’s a great question. I will share, every time I speak with Kimberlin, it’s always refreshing to hear more about her experiences and it also feels very resonant because, although we grew up in different places, I grew up in the Bay Area, right, in California, also a young Black girl like Kimberlin was at the time that she had her revelation, and I had similar revelation. I was completely struck, growing up in the Bay Area, how in my neighborhood, in my church, at my school, we had families with two parents and kids and both working and access to resources, and we had families that didn’t have one parent or any parent, didn’t have access to resources, all of this within walking distance of one another. So, as a child, I was extremely curious about the great inequities that we have in our world.

And it wasn’t until I went to UCLA – I actually came in as a Business Economics major, believe it or not, and I was doing really well in my classes, but I was bored because I kept thinking about these economic theories that seemed to really sort of take out the personal experience of what I knew that people were living with and experiencing. And I had an advisor who said, “Knowing you and your interests, I think you would love sociology.” And, you know, I was a 17-year-old who said, “What is that?” So, I knew what psychology and anthropology was. I had never heard of sociology at the time. And she said, “Well, it’s the study of society and people and how they interact.” And I said, “Okay, sure, I’ll check it out.” And I just completely fell in love.

I had no idea that there were all of these theories, all of these approaches, all of these studies that were exploring the questions I always had as a young child, and learning about and being mentored by some greats in the field of sociology, of education and other spaces, you know, really hearing that call to action, that, yes, we have these curiosities and questions because our lived experience tells us that there’s more to it than may be codified in policies and laws and taught in schools and universities, in practice and organizations. And it’s part of a great responsibility and a great commitment to be able to broaden that base. And so those things just really felt full circle for me and made, back then and continues to make today, the little girl in me just very excited about the work that we’re able to do to shed light on that and to change systems.

[J.B. WOGAN]

Okay. Great. Well, that was a great answer and I’m glad that I asked a little bit more. I’d like to do some table-setting for listeners who may not be completely familiar with these terms. As you mentioned, Mindelyn, I think they’re somewhat new, even though the tradition or the roots are old. So, what is equitable evaluation? What do we mean by that term? What do we mean by culturally-responsive research? And give listeners a sense of the history. How has the field of research and evaluation evolving to be more equitable and culturally responsive?

[MINDELYN ANDERSON]

That’s a big question. So, I’ll give it a try. And Kimberlin, looking forward to you involving yourself in this as well. So, equitable evaluation, at least the way that it was introduced to me, came from the Equitable Evaluation Initiative and the lead of that initiative, Jara Dean-Coffey. And so, again, it was just another one of those spaces where thoughts that I had, posture that we hold in this work at Mirror Group just really was reflected in the writings and understandings of someone else. And I thought, oh, how amazing.

And so for the Equitable Evaluation Initiative and their framework, it’s really about reimaging the purpose and practice of evaluation. So, it asks us to call into question, to become very curious about equity, notions of validity, objectivity, rigor, complexity, all of the things that make up the core of research and evaluation, and just really pointing out the traditions and orthodoxies that we may take for granted that are limiting the ways that we can utilize research and evaluation to really imagine how can we use this practice to create a better world. So, that was my introduction to equitable evaluation.

When it comes to culturally-responsive research and work, I always point to CREA, which is the Culturally Responsive Evaluation and Assessment organization out of Illinois, at Chicago, and just awesome colleagues there, you know, Dr. Rodney Hopson, and Melvin Hall is involved in establishing CREA, Stafford Hood, and just that amazing space that they hold for scholars and practitioners and activists and everyone else in between. So, those convenings are the space where it’s really lifting up what does it mean to craft culturally-responsive research and evaluation. How do we take into consideration lived experience, the ways that we traverse through the world and how that’s codified in institutions and systems, and how that’s reflected not only in the research that we produce but in how we produce it, our methodologies, our approaches to the work, again, at Mirror Group, what we call our posture in the work? So, those are the two spaces that always loom large for me when thinking about grounding in culturally-responsive and equitable evaluation.

[J.B. WOGAN]

Kimberlin, have you noticed a shift or an evolution in terms of culturally-responsive, equitable evaluation and maybe do you have any examples that you’ve encountered, either at Mathematica or in other positions, of how this is going right or, you know, bright spots in this area?

[KIMBERLIN BUTLER]

Yeah. I think I’d like to build on what Mindelyn shared, though, and really honor the founder, Jara Dean-Coffey here because Jara is the reason that I know Mindelyn. So, we were at the American Evaluation Association Conference and, in true Jara form, being a connector and a convener who really is passionate about equitable evaluation, she saw something in me and said you two should connect. And so here we are.

And so when I think about the Equitable Evaluation Initiative, I want to also anchor us in understanding that there is a definition through change elemental that deep equity really means working toward outcomes in ways that model dignity, justice, and love without recreating harm in our structures, our strategies, and our working relationships. So, understanding what that actually means and actually understanding that equitable evaluation is not just a doing but it is a way of being, and every researcher has a responsibility to come to the table in service of equity so that they’re not perpetuating any harm done in communities and partnering alongside communities is, to me, a central principle in valuing community and the context in generating evidence to drive transformational change. So, I just wanted to highlight that this work is really leaning into the research role as a change agent and trying to really get at advancing equitable outcomes.

In terms of bright spots, I can speak about philanthropy and specifically looking at how we are partnering with the sector and engaging in the ecosystem. There are some models and concepts, such as the Trust-Based Philanthropy Project which really addresses power and building equity in philanthropy. There are some bright spots in that we’ve done a lot of work with foundations such as the W.K. Kellogg Foundation, the Bill and Melinda Gates Foundation, the Kresge Foundation. There are a range of foundations that really center communities and understand systemic inequities. And they want to learn through partners like Mathematica around how can we engender trust and really partner with grantees in a way that we’re building healthier and more equitable nonprofit sector and communities in which we’re serving.

So, they’re looking at, you know, multi-year unrestricted funding models. They’re asking us to design strategy. They’re helping us think about the context of place and allowing our researchers to really get under the hood around challenges in the community to express that commitment to help them reach their goals and an impact they want to see in the community. So, I just wanted to speak more broadly about what that might look like and share that there are several examples where we partner with the sector and specific foundations around doing that.

[J.B. WOGAN]

Well, first, I really love that phrase “It’s not an act of doing, it’s an act of being.” I think that’s an interesting concept to sit with. I want to stick with the point you just made about the partnerships that Mathematica already has with foundations and how equity is manifesting there. Perhaps you could speak for a second about how equity is manifesting in Mathematica’s work currently and why.

[KIMBERLIN BUTLER]

Absolutely. So, when I think about Mathematica’s equity journey, it really is a deep commitment to diversity, equity, and inclusion. And we’ve done a really great job of being honest about where we sit, the opportunity to learn, and to learn together with our partners, and even grappling internally around the ways in which we can show up differently based on client needs but also what the sector actually demands, and what communities deserve I should say. So, our research is more robust because it is informed by a variety of perspectives and the mission to improve the social wellbeing has always been at the core of our work.

I think where I sit is sort of coming into the organization and thinking about my positionality as a woman of color, and a Black woman in particular, seeing the opportunity to be a catalyst for change and helping us articulate what it means to practice equitable evaluation and engaging our colleagues collectively in a process for doing that. So, launching and partnering with staff to launch our Equity Community of Practice, we had three goals, which the first of which is to internalize an equity results process and identifying the culture change required to partner with foundations, grantees, and government agencies.

And the second would be building our muscle or our capacity to continuously incorporate a set of equity principles into our project work, staffing, and leadership more broadly. And then thirdly, we had a goal to identify partnerships and collaborations across internal teams and external teams for achieving the vision and commitment to equity. And in many ways, this conversation we’re having today is an example of doing that. I mean, Mindelyn is such an incredible partner with us, and the Mirror Group and her colleagues. And it’s all about partnering and making progress together, and thinking about how we can inspect what we expect and, like, interrogate our methods and methodology that we use to do this work well.

[J.B. WOGAN]

Mindelyn, when organizations engage Mirror Group, what are the typical asks with regards to culturally-responsive research and equitable evaluation? Like, where do they need help and has the nature of those requests changed over Mirror Group’s lifespan?

[MINDELYN ANDERSON]

Oh, they certainly have. And so, in general, you know, we love to serve as evaluation, learning, and strategy partner with organizations. And they come to us because they appreciate our -- I like to describe it firm yet loving approach to all that it takes to engage in this type of work. We find often that, especially now, organizations are very conversant in the language of equity work or culturally-responsive work, but when it comes down to the practice of it and codifying it in policies and processes, that’s where things get a little tight. So, we love to be along for the journey just to reflect. I mean, that’s the core essential nature of Mirror Group. We’re here to be a reflection of the wisdom, exactly what’s happening on context and communities and organizations, and reflect that back to you, and invite you into the opportunity to do things a bit differently.

So, when they come, sometimes they are going through organizational transformation, and they want and need an external partner there with them to help to assess what are we doing and how are we doing it and why are we doing it and how can we make that more equitable. And we’re happy to be there to support. Sometimes they have a specific initiative of theirs that they would like evaluated with a lens that has attention to root cause analysis, systemic inequities, and what can be learned and understood from that work to inform not only the current project but ongoing initiatives and work in the future. So, we love to do that as well.

So, over time, our suite of services have expanded. So, at the core of it, we love to say we are the data plus equity geeks. So, we are a collective of researchers, evaluators, subject matter experts, changemakers who all really do believe that data and information are the currency of the 21st century. So, our posture in the work, the thing that keeps us going every day, is that we seek to make this data and information meaningful and accessible and actionable. That’s with learning organizations as well as the communities that we love. And so when there are aligned colleagues and aligned organizations who appreciate those desires and seek to deepen in that way, we love to serve as their partner.

[KIMBERLIN BUTLER]

Mindelyn, I have a follow-up question to you, if you don’t mind. I’m curious about, in your work or Mirror Group’s work, how do you respond to this myth or fallacy of a belief that this work is not rigorous enough in the sector, and I just wonder if you can touch upon that?

[MINDELYN ANDERSON]

Yeah, we were having a short chat about that earlier. So, I’m happy to pick back up on that. You know, I mentioned some of those spaces that are just really important, you know, the community that we’re in, the shoulders that we stand on. And I think about our partners at MPHI and the Center for Culturally Responsive Engagement. And, you know, we train. We train across the nation. We were headed to Europe. So, we would have been international, but then COVID hit. We train on utilizing a racial equity lens in strategic engagement and evaluation.

And so that question of rigor, objectivity, validity, right, like, “Oh, okay, we know that we’re supposed to be engaging in culturally-responsive ways” or “We get it, right, there’s this request or desire to be more equitable in our work.” And, you know, at the same time, if we’re engaging with researchers and evaluators, you know, we were trained in this way and this discipline and, you know, what does it mean to kind of uphold that training and those practices and do this work.

And what Paul Elam, who is Chief Strategy Officer at the MPHI and really helped to helm this work since the ‘90s always says is that it’s not a tradeoff. It’s really the both, and, and the work. And we oftentimes highlight that and say, when you’re doing this work in service of equity, when you come up against a situation where the research design that you put together and, you know, is absolutely perfect until it’s not perfect because you hit the real world, you hit the community and you see that those things that you were seeking to do with that design can’t quite be captured that way, you have a decision; right? You have a decision to make. And you can decide, I am going to remain in service of this fabulous research design that I’ve put together and I shall not move, or you can say, what adjustments and iterations can we make here to ensure that we’re still aligned with the purpose and intent of this work, and we’re able to reflect those realities through the work.

And so we say this is the rigorous work. If you are designing your research in an academic bubble or vacuum and it’s something that’s supposed to be engaging with communities and implemented in institutions but you’re doing it without those partners at the table, then your work is not rigorous. When it comes to equitable evaluation, if you are evaluating the outcomes of a program or an initiative and you’re not grounding it in the social and historical context of what has led to those inequities today, then we would say that work is not rigorous.

[KIMBERLIN BUTLER]

Wow. Powerful. Thank you, Mindelyn.

[MINDELYN ANDERSON]

You’re welcome. And I look forward to the continued conversations because, I will tell you, Kimberlin, that’s where, as I mentioned earlier, things get really tight, because the idea of doing this work, it’s so common now. It’s so sexy, if I can say that on this Mathematica podcast.

[KIMBERLIN BUTLER]

Or in vogue, if I’d say. Equity is in vogue; right?

[MINDELYN ANDERSON]

Right. Everyone is, you know, engaging in this and wants to do it. And so it’s the thought of it. The practice of it is so unsettling for everyone involved because the reality is that we’ve all been socialized in, trained in, we are all working in systems that are inequitable by design. So, when you’re doing this work, there are always uncomfortable points and times. There are always things that you took for granted that you now have to question. There are always those areas that feel like tradition, it’s sacred and it can’t be moved. And it’s like, well, if you’re engaging in this work for transformative and liberatory reasons, those things have to be moved. So, those are always very intriguing situations, especially when we’re engaging with partners and their boards, senior executives and leadership who have the power and say and modeling and money and resources and time to say, we will change and here’s how we’re going to support that change.

[J.B. WOGAN]

Now, Mindelyn, am I right in thinking that many of the organizations are mostly organizations that you interact with are in the philanthropic sector? Is that – does that make up the majority of the clients that you have?

[MINDELYN ANDERSON]

Yes. So, the majority of our work is in philanthropy, so working with foundations and national nonprofits and the communities that they partner with and we love. We do have some small work in government contracting. We have a number of cross-sector initiatives. We have government, for-profit, nonprofit, community all at the table. But the bulk of our work, yes, is in philanthropy.

[J.B. WOGAN]

Okay. Great. Because one of the things I was hoping we could talk about today is the role that philanthropy is playing in supporting culturally-responsive and equitable evaluation. And I know that they’re not the only sector in town. We just had an event earlier this summer that was looking at the role that the federal government can play, but it does seem like foundations were early adopters of this work, or at least some of them were. So, I’m wondering, if that’s right, why is that the case?

[MINDELYN ANDERSON]

I certainly believe that philanthropy as a sector has been a catalyst for this work. They can take risks and make what I call big investments, those that are bold, those that are innovative, and absolutely grounded in evidence and community context in ways that, you know, government, it may have been slower to do. You know, there are recent events that sort of pushed government in that way, and the pendulum can swing depending on who’s in the Oval Office of in Congress. But the sector also is leading the charge in demonstrating that it’s about return on relationships and not just return on investments. That’s the way I think about philanthropy.

They’ve historically played a convening role in bringing a diverse array of stakeholders together as drivers of change, but what I see now is this moment around equitable evaluation, culturally-responsive and equitable evaluation, and participatory grant-making is an opportunity to punch above their financial weight and really lean into those practices around trust-based philanthropy that I mentioned earlier, and actually listening to their grantees in a way that invites culturally-responsive and equitable evaluation into their strategies.

So, some of the communities, including the structural inequities that they bear, are really a demand for transformation, honestly, from my vantage point. And so philanthropy has been inspired to partner differently. And I see this in leaders like Darren Walker, who’s President of the Ford Foundation, who wrote “From Generosity to Justice: A New Gospel of Wealth.” That’s a book that’s currently out. And then also regularly quoted is Edgar Villanueva, who wrote “Decolonizing Wealth.” So, those are two resources that kind of signal where the field is and how they are playing a supporting role and being a catalyst in driving culturally-responsive and equitable evaluation.

[J.B. WOGAN]

In culturally-responsive and equitable evaluation, how would you say the relationships between researchers and community members is different, and what does it mean – this is something Kimberlin and I have talked about before offline, but what does it mean to partner authentically? And in terms of – to throw one more question at you – in terms of influencing public policy and now researcher audiences, why does the quality of those relationships, of those partnerships matter?

[MINDELYN ANDERSON]

I’ll start with quality. It matters immensely. You know, if you think about it, there have been these town-gown relationships between universities, colleges, research institutes, and the surrounding communities for over a century; right? Go to any major city, any major university and you’ll find that. And so the quality of that partnership matters a great deal for the vast majority of that time communities were the research subjects. They were not equal partners at the table. They were not setting the research agenda. They were the ones whose samples and insights were kind of taken from; right? They’ve experienced those relationships to be very extractive. And so when you’re thinking about influencing public policy with audiences, you know, equally at the table, the quality of that partnership matters a great deal.

Community is over being an advisory group sounding board for something that was already codified by researchers and other decision-makers and just being there to rubberstamp it. They are over it. They are savvy. They know exactly what is going on and they’re not having it. And so when you think about the relationships between researcher and community members, it’s important to come as your whole self in that work. Yes, I’m a trained researcher or evaluator, and I am also – when I come to the work, I’ll speak for myself, I’m also a wife and a mother of four young children; right? I’m also active in my community and in my church. I have all of these lived experiences that connect me to you and others, and also things that may be very different, and we’re here to learn about and from and with one another. And I just really appreciate the trust that we’re building together and that you all have in helping us to align community with these other partners, and how we can do that together.

So, I think that relationship is everything. Transparency is definitely very, very key, being frank, and providing and sharing information and context. I’m a curious one, so I always have all the questions in all of our community-engaged work when I’m getting together with Mr. So-and-so or Ms. Such-and-such or Brother So-and-so or Sister Such-and-such. Like, I have all of those questions and they’re just like, “Oh, that’s her, she’s such a curious bee.” I am.

And still, you know, how can we come together to help show this alignment? How can I help to serve as a translator of sorts, to be able to share some of the context and framing that’s coming from the funder or coming from the research organization partner, and equally share the context and experience that you have, and help to find that common alignment? And so it’s that sort of posture that we hold in the work at Mirror Group and we see as paramount in the relationship-building process, which is why when partners come to us and say that they want to engage in this work, we have so many questions about relationship that’s already existing between those at the table and how we’ll be able to come in and enter into those spaces.

[J.B. WOGAN]

Kimberlin, I think another way of phrasing partnering authentically is something I’ve heard you say in the past about relationships moving at the speed of trust. Why does that matter? Why is it so important in your work and in Mathematica’s work to have that kind of trusting, authentic relationship?

[KIMBERLIN BUTLER]

Yeah, J.B., you are reading my mind. And I was like, yes, Mindelyn, you nailed it. I really appreciate all of your comments because, J.B., honestly, you know, if evidence really matters, then we must care how it gets generated. And so the reason I say all the time that relationships move at the speed of trust is because evaluation and our research approach is all about building trust and being aware of the nature of the partnerships that we’re building in service of equity. And it is all about power. Researchers carry a great amount of power. And so as Jara likes to say, it’s not about losing power, it’s more about wielding power for good.

And so I think about, you know, researchers stepping into a space and sharing and understanding the space in which they’re stepping in to conduct research. You have to understand the context and biases that you carry. And so the systemic awareness comes to mind. That’s a signal of trust. What assumptions are built into the research approaches that you use most frequently? You know, how do your approaches reinforce privilege of those already powerful? You know, who’s competent? Who’s the person that you value more or the group that you value more? Is it the researcher or is it the community? And your answer to that question matters.

I think about the institutional awareness. You know, what is the history of research in this community, and particularly history for the community organization and these participants that we’re partnering with? What role has, you know, research played historically in the community, and actually caring about that? But then there’s also a very personal part of this work, which is where I started in this podcast. It’s like everyone comes to the work with some personal knowledge and assumption. So, what are the assumptions you’re making about the people and the context of your work? What is considered valid?

And then, finally, I always think about relationships matter because there is so much at stake. When I think about that definition that I mentioned earlier, it is really about making sure that we’re not perpetuating harm. And so from the community organization’s perspective, you know, what will actually benefit them in this process and how do you know it will benefit them? Are they engaged regularly in the process? Is there a bank of advisory members in the community, and elders when you think about indigenous research, that you can actually tap before you even design the study? So, all of that plays into trust-building and it signals to the world that we actually care enough to dedicate the time, the investment, and the thought partnership in doing that with community.

[J.B. WOGAN]

So, my understanding is that equitable evaluation and culturally-responsive research has – there are different benefits, different purposes for practicing that, and one of them is related to organizational strategy and learning. So, to stick with the philanthropic sector for a second, within philanthropy and within philanthropic organizations, how does the practice of equitable evaluation and culturally-responsive research inform strategy and learning?

[KIMBERLIN BUTLER]

Yeah, I’ll start and I’m happy to turn it over to Mindelyn because she has another lens as well. But I really do believe that this practice is philanthropy’s opportunity to hold up a mirror in new ways, to inspect what they expect and really partner with communities to co-create equitable outcomes. So, it invites listening differently, learning differently, and supporting grantees differently. So, equitable evaluation can also help a funder, as I mentioned, kind of look under the hood to really understand and make sense of their investments, and also see blind spots around the impact they actually committed to making in partnership with their trustees and their boards.

And it’s also – it’s good data when triangulated with grantee and community voices. That can help leaders in foundations, programs officers as well who are very close to those investments, you know, set better strategies. And it also helps funders take the long view in tackling real community-anchored solutions. And so all of that matters because you want to be constantly learning and centering the voices of community, and understanding the structural inequities and opportunities to address those regularly in the practice of making grants and setting strategy for improving the communities where you are making your investments.

[J.B. WOGAN]

And Mindelyn, what would you say in terms of the way this is informing strategy and learning?

[MINDELYN ANDERSON]

I think Kimberlin did a great job. You know, what came to mind as she was speaking is really radical accountability. And so what does that look like? It’s a mutual accountability because you do have some situations where it’s “Oh, we want to fund this culturally-responsive or equitable evaluation work of the portfolio. We want to know how grantees are doing. We want to know…we want to know…” you see where I’m going with this.

So, we always like to lift up that, on the other hand, grantees and community want to know what are your intentions long term with this investment strategy. You know, how will this sustain over time and not just be a flavor of the month or the year, depending on the funder, because these are their lives. These are their neighborhoods; right? These are the systems that they touch and structure their lives every single day. And so I think the real promise there is really in that radical mutual accountability. So, as Kimberlin said, not only to reflect on the portfolio or the initiative or the activities but also on the innerworkings of the foundation itself, how it’s structured and what that means for what’s possible through their philanthropic strategy.

[J.B. WOGAN]

All right. So, final question for today, and I really appreciate your time. Kimberlin, let’s start with you. What are the most common pitfalls that research organizations encounter when trying to adopt culturally-responsive and equitable evaluation practices? And if you can provide any tips to listeners for avoiding those pitfalls, or at least course-correcting when they fall into those pits?

[KIMBERLIN BUTLER]

Yeah, I think that a major pitfall is not taking the time to build in reflective practice. And, you know, thankfully, at Mathematica, we have what’s called an equity guide to support our researchers to make sure that they are actually building space and time to reflect about how to incorporate equity principles into their work and how to practice equity evaluation and culturally-responsive research. We’ve hosted a few brown bags through the equity community of practice and there is a community growing where we are leaning into supporting each other more and more around this. And, of course, in partnership with those stakeholders who are closest to the work that we want to achieve together.

I think another one is thinking about those most proximate to the work. You know, even the most prestigious group of researchers from the best organizations are unlikely to develop accurate research questions and design if they don’t share any proximity to the communities being served or being analyzed in the research, as the focal point. And so with that comes a need to really recognize value and understand those experiences, those lived experienced, those relationships and that knowledge, and place it front and center as really being information that should be valued for us to use in our work.

I think that how we can avoid the pitfall of only looking at researcher-generated quantitative data is by looking to do or generate data and research from the communities we seek to serve, and making sure it counts as equally valid evidence, as we mentioned earlier. So, that means, do the proposed methods and approaches provide the level of depth and data needed for this evaluation’s purpose? And so making sure that we redefine what is rigorous and really valuing the voices of communities as cocreators in the design and generators of the evidence to improve the communities in which they sit and live and work. So, to me, those are some very common pitfalls and we would all do better if we understood how to tackle that together and make progress together in partnership with communities.

[J.B. WOGAN]

So, Mindelyn, take us home. What would you say in terms of common pitfalls and advice for avoiding those pitfalls?

[MINDELYN ANDERSON]

That’s an excellent question. I mean, you know, we all want to act on these things. We all want to follow through and, still, there is no magic bullet; right? There’s no manner of magic bullet, special checklist, you know, that if I do all of these things, then everything will turn out perfectly. So, I would say that’s one of the most common pitfalls is just that urgency to understand what it is, get it right now, and that perfectionism, right, kind of perfect this.

And so what we really strive to do – and, again, this is in partnership with colleagues at MPHI, is really develop that individual and team and organizational self-reflective practice. Have those moments built into your team work processes and your workflow where you pause and reflect on how are we engaging in this work; right? What does it mean for the current work that we have at hand and what is it that we can do to deepen our practice to be more responsive to community, to really be able to catalyze that sort of change?

And so it’s at every step of the process. If you’re talking about your evaluators or your researchers, really pause and assess your awareness of cultural differences among populations; right? What is the diversity of your team and the teams and communities and contexts that you will engage with? Where are there similarities and differences and how will that personal awareness better equip you all to engage in this work? If you’re talking about the evaluation process, then it’s how am I/are we intentionally centering priority populations in the work? And that’s in everything from instrument development and protocol to identifying how and when we engage in that data collection, training others in the use of it, questions that we pose. You have to pause and really ask deep questions about that, not just use the broadly-used validated tools that are available because they may not be appropriate.

You know, if you’re talking about grantees and their attributes, you know, really being aware of governing bodies and the structure of their organizations and how they operate. All of those things matter for what you bring into the space. And when getting into inclusion and equity, really focusing on engagement and healing, there’s a lot of harm, either named or not named, that exists. And so to proceed as if it doesn’t can do greater harm. And so really having that frame of healing and coming together.

And, most importantly, advancing equity. Again, our nation, our organizations, our systems, we’re built by design to be inequitable. And when we all recognize that as the truth, it opens up so many possibilities as to how you can really transform those things and how it’s not going to happen overnight because this has been centuries in the making. And so I would share those are some of the pause points, I would call them, rather than pitfalls, just to consider while engaging in the work.

[J.B. WOGAN]

Thanks to my guests, Mindelyn Anderson and Kimberlin Butler. In the show notes, we’ll include a few resources for further reading and listening about culturally responsive and equitable research. As always, thank you for listening to another episode of On the Evidence, the Mathematica podcast. There are a few ways you can keep up with the show. You can subscribe wherever you listen to podcasts or you can follow us on Twitter. I’m at JBWogan. Mathematica is at MathematicaNow.