A new episode of Mathematica’s On the Evidence podcast explores recent research that experimented with a new approach to equitable evaluation that provides data collection while equipping individuals and organizations with new skills and resources. The research also provides early evidence that a recent philanthropic investment in local grassroots organizations focused on affordable housing issues has helped those grantees become more effective in achieving their goals.
About four years ago, the Robert Wood Johnson Foundation awarded about $90 million in grants to support community power building. Community power, according to the foundation’s website, is “the ability of communities most impacted by inequity to act together to voice their needs and hopes for the future and to collectively drive structural change, hold decision makers accountable, and advance health equity.”
The rationale for investing in community power building is that the people most directly affected by systemic barriers and inequities are also the ones best positioned to drive change in their communities. And yet, they may not feel empowered to do so. One of the persistent obstacles to achieving health equity has been a sense of powerlessness among residents from households with low income and communities of color who have been historically excluded from decision making on the policies and practices that affect their health and prosperity.
The foundation’s investment occurred in 2020 at a time of intensified national dialogue about racial discrimination, social justice, and deep inequities within American systems and structures. Media attention on and protests of the police killings of George Floyd, Breonna Taylor, and other Black Americans not only cast a spotlight on law enforcement, but increased public scrutiny of biases in systems that lead to disparate outcomes in other areas of life, such as education, health care—and housing. One way the foundation sought to respond to those concerns was by dedicating some of its $90 million in grant funds for community power building to housing justice, a movement to ensure everyone has access to safe and dignified housing as a human right.
The Robert Wood Johnson Foundation contracted with Mathematica and UBUNTU Research and Evaluation to study how the grant money was used and to help the foundation learn from the grantmaking experience for its future investments in community power building. The podcast episode discusses findings on the impact of the foundation’s investment and the Evaluative Power Fellowship Model, a novel approach to equitable data collection developed by Mathematica and UBUNTU. In this model, researchers train staff from grassroots organizations in evaluation methods, increasing the capacity of those organizations to learn from and improve their work in areas such as housing justice. During the fellowship program, the researchers learned more about the grassroots organizations and their use of the grants fund.
“It's a win-win situation,” says dr. monique liston, UBUNTU’s founder. The fellowship program “made the data collection process something that they got something from, something that they were able to take away immediately and implement into their work.” At the same time, she adds, “we felt like we were learning.”
The fellowship program established a transparent and trusting relationship between the researchers and grassroots organizers that helped reveal better insights, says Drew Koleros, a principal researcher at Mathematica who worked on the housing justice project. “The information we were getting about what it takes to build and wield power was so much richer than we ever could have got in the best structured interview guide or survey we could have done with people.”
Grassroots organizations such as those that participated in the fellowship are accustomed to filling out surveys or being interviewed, but it’s not always clear how the information the organizations provide benefits them or their communities. “Data collection can feel extractive for a community,” dr. liston says. The Mathematica and UBUNTU teams sought to show the fellows how evaluation could strengthen an organization’s internal capacity to learn from its campaigns and make evidence-informed changes.
"Through the fellowship, they were able to see evaluation as a tool that they could use in addition to something that happens to them," says Kelsey Cowen, a member of Mathematica’s research team on the housing justice project.
Evidence gathered through testimonials by the fellows suggests the investments did help the grassroots organizations be more effective in advocating for housing justice issues. The funding gave them access to new resources and expertise. Ultimately, it increased their capacity to execute their work faster and at greater scale, allowing them to expand their reach, mobilize more people, and take bolder actions more efficiently.
The grassroots organizations ran successful campaigns that changed local public policies, such as establishing rent control and codifying a tenant’s bill of rights. They also got landlords to improve the living conditions of tenants by installing window screens, removing black mold, and ensuring that security alarms were working.
Although the evaluation was not designed to establish causality or estimate the size of impacts, Koleros says, the research team could show a connection between the policy changes affecting tenants’ lives and changes in the grassroots organizations’ "infrastructure and capacity, which came as a result of these power building efforts.”
View transcript
[MONIQUE LISTON]
A lot of times data collection can feel extractive for a community org, can feel like a waste of their own time, something that has nothing to do with them, that doesn't better the work that they do, but betters other people. And the work that we did through the fellowship intentionally disrupted that, and it made the data collection process something that they got something from, something that they were able to take away immediately and implement into their work. And for us, that was really what the joy of the whole thing was, is that you felt like you were learning. We felt like we were learning. That means it's a win-win situation.
[J.B. WOGAN]
I’m J.B. Wogan from Mathematica and welcome back to On the Evidence.
On this episode, we’re going to talk about the relationship between policy evaluation and grassroots organizing for social justice and community change. At Mathematica, we’re big believers in the power of evaluation to help organizations assess progress and make data-informed changes to achieve their goals. We envision a world where everybody is using evidence to make decisions, but the reality is, too often, the capacity to collect, analyze, and learn from data is a kind of privilege. And if you’re a small nonprofit or community-level grassroots organization, you probably don’t spend your limited resources on evaluation. What you might encounter is outside parties, like academic researchers and their funders, asking for your data to make their own assessments of your performance. Not surprisingly, that dynamic doesn’t engender trust and can create the mindset that evaluation is extractive and not in the best interests of a grassroots organization and its mission.
A recent project at Mathematica explored ways to change that dynamic by experimenting with a novel way to approach equitable evaluation that simultaneously equipped individuals and organizations with new skills and resources.
OK. Let’s get more specific. About four years ago, the Robert Wood Johnson Foundation awarded about $90 million in grants to support community power building. Community power, according to the foundation’s website, is “the ability of communities most impacted by inequity to act together to voice their needs and hopes for the future and to collectively drive structural change, hold decisionmakers accountable, and advance health equity.”
The rationale for investing in community power building is that the people most directly affected by systemic barriers and inequities are also the ones best positioned to drive change in their communities. And yet, they may not feel empowered to drive the change. One of the persistent obstacles to achieving health equity has been a sense of powerlessness among residents from low-income households and communities of color who have historically been excluded from decision-making on the policies and practices that impact their health and prosperity.
The foundation’s investment occurred in 2020 at a time when the national dialogue about racial discrimination, social justice, and deep inequities within our systems and structures had intensified. Media attention and protests about police killing George Floyd, Breonna Taylor, and other Black Americans not only cast a spotlight on law enforcement, but increased public scrutiny of biased systems leading to disparate outcomes in other areas of life, such as education, health care, and housing. So one way the foundation sought to respond was by dedicating some of its $90 million in grant funds for community power building related to housing justice. The foundation then hired Mathematica and UBUNTU Research and Evaluation as partners, to both study how the money was used and help the foundation learn from the experience for its future grantmaking in community power building.
When I first learned about this project, part of what stood out to me was the challenge of evaluating something as broad as community power building. I thought back to one of my first experiences at Mathematica, where I helped write press releases and pitch reporters about an evaluation of taxes of sweetened beverages in Philadelphia and Oakland. Incidentally, that evaluation was also funded by the Robert Wood Johnson Foundation. In that case, the researchers had a much narrower set of questions to investigate about whether these taxes were changing residents’ behaviors. In other words, did people buy and drink fewer sodas, juices, and other beverages? Did they leave the city to keep buying the drinks? Did the taxes reduce consumption similarly across all people, or were there meaningful differences say, between adults and children?
[DREW KOLEROS]
It's such a much more straightforward thing. It's like, there's a policy change happening. Help us measure the impact of that policy change.
[J.B. WOGAN]
That’s Drew Koleros. He’s a principal researcher at Mathematica who worked on this Robert Wood Johnson Foundation project about housing justice and community power building. He’s also an expert on theories of change and recently published a book on the topic, which includes a discussion about how evaluation can be used to understand broader societal change processes, exactly the sort of thing the foundation was asking Mathematica to evaluate here.
Take the beverage tax example. Yes, evaluators can look at the impact of a policy change …
[DREW KOLEROS]
But how do we get to that policy developed, right? Not only like that initial policy but how does it get brought into the policy arena? Who has a seat of the table? Is it just decision makers, is just the elected officials there? Or what's the role that communities playing? Are decision makers even listening to community members about thinking bottom up some of the issues that are important to them? So what is the community's ability to even get like a soda tax at the table for discussion?
[J.B. WOGAN]
So, in a sense, an evaluation of community power building is about the ability of local individuals and grassroots organizations to effect change. The community’s ground-up capacity is what’s being evaluated, not a discrete policy change being issued by a legislative body or regulatory agency. And Drew pointed out the same kind of analysis can be applied to a community’s ability to influence local housing policies, such as eviction moratoriums or caps on rent.
[DREW KOLEROS]
So who holds power, who doesn't hold power in the situation? Who has something to gain and who has something to lose? Landlords have something to lose. If there's a cap on rent, and landlords have access to policy makers to influence them through lobbying and other advocacy activities that they can do. So they hold a lot of power in this space. Renters have a lot to gain from a renting cap, but individually, one individual renter doesn't have power to take on a landlord agency, a realtor agency, a big corporate landlord, doesn't have access to policymakers at a local municipal or even state level who are making any of these changes.
[J.B. WOGAN]
So the question that the Robert Wood Johnson Foundation was asking was, what can we, the foundation, do, through our investments, to correct the power imbalance and give individual renters more of a say, collectively, in local housing policy? And while the foundation was specifically interested in advancing housing justice, it was also asking the Mathematica-UBUNTU team for insights about the broader topic of investing in community power building. To get answers, the team needed to talk to the grassroots organizations who received money through pass-through grants, not only to learn how they used the grants to build community power for advancing housing justice, but also to relay back to the foundation more general insights about effective power building.
But in the context of an initiative to correct for power imbalances, the evaluation team soon realized that the easiest, most straightforward approaches to data collection were going to fall short. Here’s Drew Koleros again.
[DREW KOLEROS]
From the first, kind of, the beginnings of the evaluation, we were thinking about site visits, right? We would select maybe 5 to 7 communities and go out and spend a week kind of talking to people interviewing lots of different folks, observing what's happening, maybe even a few like participatory workshops where we got people together. And we kept just thinking about what it would take to get the right people in the room, what it would take. And we just started realizing that there was no way that we could do that that wouldn't be extractive, that wouldn't be one directional with us just taking and taking and taking and giving nothing back.
[J.B. WOGAN]
Long-time listeners of our podcast know that the policy research field has been exploring methods and approaches to evaluation that prioritize diversity, equity, and inclusion, not only because it’s less extractive, but also because it can be more rigorous. It can elicit better, more reliable information. For this project, Mathematica and UBUNTU were already planning to use culturally responsive and equitable evaluation methods, but even some of more traditional ideas in that space didn’t seem appropriate in this case, Drew said.
[DREW KOLEROS]
When we're thinking about more equitable evaluation approaches or culturally responsive evaluation, we're thinking about building relationships and building trust with people. And we didn't see a way that we could do that in a limited timeframe that we would just be asking people to do things, and even paying them, you know, even giving them huge honorarium to compensate them for their time in a way that felt authentic. It still felt very extractive in one way.
And we thought that that would also affect the quality of the data that we were getting. We're asking people about really complex, difficult, and often very individual experiences. How could we possibly expect some person to fly into your community in Milwaukee, you know, having just met you 30 minutes ago, ask you to divulge these stories about your past and how these things are working, without any relationship before, any trust between us. We just didn't think that we would get really high-quality information from that approach.
[J.B. WOGAN]
As Drew mentioned, researchers sometimes pay people for sharing their time and expertise. But the Mathematica and UBUNTU team thought of something else of value that they give in addition to compensating people for their time: resources and knowledge in evaluation methodology. They could help equip these individuals and their organizations with new capacities and strategies to collect data and leverage it to achieve their mission-driven goals. Here’s dr. monique liston, founder of UBUNTU Research & Evaluation.
[MONIQUE LISTON]
We were being told by the participants in this study that they wanted to do more with data and research, but really just didn't have the capacity.
[J.B. WOGAN]
And here’s Drew again.
[DREW KOLEROS]
It’s not something that people had those skills to do. They were learning it on the fly. They weren't quite sure what to measure, how to measure.
[J.B. WOGAN]
So these grassroots organizations didn’t have the internal capacity to evaluate their work, but because of their small size and limited financial resources, they also weren’t about to hire external evaluators. That put them in a bind, dr. liston said.
[MONIQUE LISTON]
Like we can't afford an outside consultant to do this work, but we know we need this kind of information in order to do our work.
[J.B. WOGAN]
Thus, the idea of an evaluation fellowship was born. Ten individuals from eight grassroots organizations working on housing justice issues participated in more than 50 hours of expert-led training over nine months provided by Mathematica and UBUNTU, where fellows learned basic concepts in evaluation and how to apply them. Some of the fellowship program involved in-person sessions in Milwaukee, Wisconsin, where UBUNTU is located. Fellows received a stipend for participating and were reimbursed for their travel and lodging expenses during the program.
Kelsey Cowen, part of the Mathematica research team that led the fellowship program, said the hope was that the training would double as an equity-informed data collection method. The evaluation team would learn from the fellows as the fellows learned from them.
[KELSEY COWEN]
So I think with the fellowship in particular, for everything that we were extracting from it, we were giving something to the fellows. And so it was this belief, I guess, or trust, faith that we would be able to learn about their work and like how they're seeing this ecosystem from their perspective, and how they're seeing the funding from the foundation from their perspective. We were going to be able to learn all of that as we were building their skills and offering them, you know, our knowledge and expertise in how to evaluate their campaigns and how to evaluate their work, and how to, you know, you know build an evaluation protocol, all of those things.
[J.B. WOGAN]
To a regular listener of On the Evidence, free training in evaluation might seem like an easy sell, but the research team had to overcome some initial skepticism among the fellows. Even though the fellows had expressed a need to build up data and research capacities at their organizations, they still felt ambivalent about the field of evaluation as a whole. Here’s dr. liston again.
[MONIQUE LISTON]
I think they were intrigued by the concept, but there was some hesitancy because the word evaluation, we are sort of coached in this culture to feel like, at the end of that is something bad. At the end of evaluation, a job is lost, a program is cut, we're not working in the area anymore. And so, it doesn't feel affirming to participate in any evaluative activities because it is a negative, a minus, a deduction to the work that we're doing. And so, we were really holding that intentionally, hoping to shift that evaluation isn't, doesn't mean that. It means learning and if we're focused on learning, whatever outcomes of it are, because that's what we learned, not because we're trying to punish, malign, or do something else negative to your organization.
[J.B. WOGAN]
During and after the fellowship program, there were signs that the mindset shift happened. Here’s Mathematica’s Kelsey Cowen again.
[KELSEY COWEN]
Over the course of the fellowship, they were saying more and more frequently that they felt like evaluators. And I think it, too, it also reoriented them toward what evaluation could do. I think a lot of these organizations are very used to evaluation being like, here's a survey and then you never hear from me again or like here's an interview and you never hear from me again. And so, I think now through the fellowship they were able to see evaluation as a tool that they could use in addition to something that happens to them.
[J.B. WOGAN]
For example, in an interview after the fellowship, Lupe Arreola, the executive director of Tenants Together in Berkley, California, said she had come to see evaluative thinking as a critical capacity for grassroots organizations like hers to build and wield power. Here’s Lupe:
[LUPE ARREOLA]
I personally believe, like, one of the biggest issues that we have in our movements is that we don't learn from our mistakes, and that we don't learn, that we're not digging enough into past history to be like, oh, this has happened before, because history is circular.
[J.B. WOGAN]
In a broad sense, having an evaluator’s mindset can help grassroots organizations like hers to consider past moments in history that are instructive about what works to address present challenges. For example, during the Great Depression, when so many people lost their jobs and income, what did housing advocates do to help people stay in their homes?
[LUPE ARREOLA]
Yeah, I just feel like we don't look back enough to be like, how, where has this happened before? How did it go? How did it play out? Can we learn anything from that? Or can we do anything different? Do we want to try to do anything different?
[J.B. WOGAN]
In another interview, Rebecca Song of United Renters for Justice, a nonprofit in Minneapolis, Minnesota, said the fellowship had given her new ideas of how to use evaluation in her organization’s operations. Here’s Rebecca.
[REBECCA SONG]
One thing I'm pushing us to do is actually more evaluation, you know, and more interim evaluation instead of like, now campaign's over, that's anywhere from three to five years. Now let's evaluate. You know, mostly everything that's happened. It's like well, yes, but also, like, to get funding you need, you can't wait that long. Like you need to evaluate constantly because things change, you know, there are times I have to write an entirely different proposal within three months because it's shifted so much.
[J.B. WOGAN]
Rebecca said she has already talked with her team about a few different focus areas for future evaluation.
[REBECCA SONG]
The thing we landed on is evaluating campaign wins. Because right now, we don't really have a structure for doing that and it feels like a lot of it is like between oral history and [inaudible] everywhere. You know, there really isn't one place where all that's captured and can be easily seen. So hopefully, that's something concrete that'll come out of this.
[J.B. WOGAN]
Lupe and Rebecca’s feelings about evaluation represented a more general warming among the fellows, dr. liston said.
[MONIQUE LISTON]
There was a definite shift in people's attitudes towards evaluation and some new curiosity on how to hold evaluative thinking, which I think is probably the evaluator’s dream. You know, there's the one thing to recruit more evaluators. Sure, the field could use definitely more people from all walks of life. But I think there's a thing to hold when you walk into more organizations and more people are practicing evaluative thinking. That's really exciting.
[J.B. WOGAN]
By working with Mathematica and UBUNTU, dr. liston said, the fellows could better articulate what their evaluation needs were and how to get resources to meet those needs. They had more precise language they could use in grant applications to funders, and could engage more confidently in conversations with program officers when seeking funding for evaluation. These are just some of the concrete ways fellows and their organizations were gaining something, rather than only giving something, by participating in the evaluation.
[MONIQUE LISTON]
A lot of times data collection can feel extractive for a community org, can feel like a waste of their own time, something that has nothing to do with them, that doesn't better the work that they do, but betters other people. And the work that we did through the fellowship intentionally disrupted that, and it made the data collection process something that they got something from, something that they were able to take away immediately and implement into their work. And for us, that was really what the joy of the whole thing was, is that you felt like you were learning, we felt like we were learning. That means it's a win-win situation.
[J.B. WOGAN]
The idea of offering resources in evaluation in order to also gather data for an evaluation was an experiment. The Mathematica and UBUNTU team didn’t know if it would work. In the end, the researchers believe they’ve developed a useful research practice that could be applied to other research questions and in other contexts. Here’s Drew again.
[DREW KOLEROS]
By doing the fellowship, not only were we getting the information that we needed for our evaluation, we were transferring evaluation capacity to the fellows. We were getting information and they were getting skills. And because we were clear about that and open and honest and transparent about that from the beginning, we were able to build that trust and relationship with the fellows over time that as we started getting into the fellowship, the information we were getting about what it takes to build and wield power was so much more rich than we ever could have got in the best structured interview guide or survey that we could have done with people.
[J.B. WOGAN]
So what did the team learn through the fellowship? Part of what the foundation wanted to know was how the funding had been used and whether it had made a difference in advancing housing justice. Although the evaluation was not designed to establish causality or estimate the size of impacts, evidence gathered through testimonials by the fellows suggests the investments did help the grassroots organizations be more effective in advocating for housing justice issues. The funding gave them access to new resources and expertise. Ultimately, it increased their capacity to execute their work faster and at greater scale, allowing them to expand their reach, mobilize more people, and take bolder actions more efficiently. Here’s Drew Koleros again.
[DREW KOLEROS]
I think what we learned from the grantees as part of this work is that a lot of the money that they received through the housing justice, the community power building initiative from Robert Wood Johnson helped these organizations build infrastructure and capacity of their organizations and their local grantees.
So some of the stories we heard, for instance, is, groups that had been organizing across a lot of different issues, whether that's, you know, the fight for 15 or sick leave and housing issues, and maybe some climate justice issues may, maybe they didn't have an organizer dedicated to the housing justice movement. With some of this funding, they were able to hire a dedicated organizer, so that organizer could start going to low-income housing or other tenant complexes and organizing tenants into tenant unions, finding leaders within those unions, developing those leaders to go on and then started building these collective tenant unions who are then being able to negotiate with landlords.
[J.B. WOGAN]
As groups used the foundation grants to build their capacity, they also reported important wins that made meaningful improvements in people’s lives. Some wins related to policy changes, but some were smaller in scale, like getting screens installed in homes. Here’s Drew again.
[DREW KOLEROS]
These are, you know, communities in the summer who didn't have screens on their windows and mosquitoes coming in all night and being able to collectively organize and advocate with landlords to make sure there were alarms on the doors, make sure that there were screens, removing black mold from people's apartments that had been there for years and years.
[J.B. WOGAN]
The researchers also heard that as the grassroots organizations brought individual renters together as tenant unions, they saw the renters successfully negotiate policy changes, too. For example, organizers at Miami Workers Center and Community Justice Project successfully campaigned the Miami-Dade Board of Commissioners to pass a tenant’s bill of rights, which grants tenants the ability to make repairs to their units and get the cost of repairs covered by the landlord if the landlord doesn’t respond to repair requests within seven days. The ordinance also bars the landlord from asking about or requiring a tenant to disclose their eviction history on an initial rental application.
Here's another example. In New Orleans, organizers with the Jane Place Neighborhood Sustainability Initiative helped get a Healthy Homes ordinance passed which requires owners of rental properties to register for a citywide database and sign a statement that the property meets basic health and safety standards. The law also prohibits landlords from retaliating against renters who report health and safety issues at the property.
And in Oxnard, California, a seaside city west of Los Angeles, organizers with the Coast Alliance United for a Sustainable Economy, or CAUSE, successfully campaigned the local city council to establish rent control for rental units constructed before 2007.
Drew is careful to say, this study provides some initial descriptive information about the community power building ecosystem in the housing justice movement and how grassroots organizations put the funds to use. In a traditional evaluation, researchers would also be trying to establish how much of the positive changes were attributable to the investment from the Robert Wood Johnson Foundation. But the evaluation wasn’t designed to answer those questions. What the research does, Drew said, is help evaluators like Mathematica and UBUNTU and funders like the Robert Wood Johnson Foundation …
[DREW KOLEROS]
… to see these trends and these big changes and lodging some of these instances of changes and changes in individual's lives, linking that back to changes in infrastructure and capacity, which came as a result of these power building efforts. So, those are, these are the kind of stories and narratives that we were trying to tell through the evaluation.
[J.B. WOGAN]
Mathematica is publishing a framework co-developed by the fellows, Mathematica, and UBUNTU that explains how to build community power in the housing justice movement. It’s too early to know how lessons from the project might inform the Robert Wood Johnson Foundation’s investments in community power building, or how they might inform philanthropy as a whole, but it’s clear that the sector continues to explore community power building and will need new ways of thinking about the efficacy of those investments.
What’s also certain is that Mathematica and UBUNTU believe the fellowship model, where evaluators learn while they train, can become a tool for the equitable evaluation field more broadly. They have plans to present findings about their experience with the fellowship model at an annual conference hosted by the American Evaluation Association in October.
That brings us to the end of our episode. In the show notes, I’ll include a link where listeners can learn more about Mathematica and UBUNTU’s evaluation of the housing justice investments by the Robert Wood Johnson Foundation. You can also check out the framework the researchers and fellows created, which is now an animated and annotated graphic that is free and publicly available for others to use.
Thanks to dr. monique liston, Drew Koleros, Kelsey Cowen, and Sabrina Fay for speaking with me for this episode. I also want to thank Rebecca Song and Lupe Arreola, who gave us permission to use clips originally recorded as part of qualitative research for the evaluation.
And thank you for listening to On the Evidence, the Mathematica podcast. This episode was made possible with financial support from the Robert Wood Johnson Foundation. If you liked this episode, please consider leaving us a rating and review wherever you listen to podcasts. To catch future episodes of the show, subscribe at mathematica.org/ontheevidence.
New to Mathematica’s On the Evidence podcast? Subscribe for future episodes. If you like this episode, consider leaving us a rating and review on Apple Podcasts to help others discover our show.
Show notes
Learn more about the housing justice and community power building evaluation conducted by Mathematica and UBUNTU on behalf of the Robert Wood Johnson Foundation.
Explore the interactive graphic that illustrates a new framework for community power building in the housing justice movement.
Learn more about the recent (open source) book Drew Koleros edited on updating theories of change for the field of evaluation, which includes sections that are relevant for evaluating community power building.
