Former Biden-Harris Administration Official Christina Ciocca Eller on Turning Evidence into Action within the Federal Government

[CHRISTINA CIOCCA ELLER]

I couldn't say, "No." I really have always been a scholar who's thought quite a lot about how research and policy might coexist – and not just coexist but even comingle -- in real and tangible ways. And I really couldn't think of a better way to try on that hypothesis for size than actually joining the Administration.

[J.B. WOGAN]

Hi, I am J.B. Wogan from Mathematica and welcome back to On the Evidence. This is the second installment on a new occasional series on the show that we're calling "Evidence in Government," where we talk about new developments in the halls of government and the role that evidence can or should play in decisions that could improve people's lives.

To lead these episodes, we've tapped my colleague Mike Burns, who recently became our Senior Director of Communications and Public Affairs at Mathematica. Mike previously worked in public affairs for the U.S. Department of Housing and Urban Development, the U.S. House Appropriations Committee, and for several members of Congress.

For this episode, Mike speaks with Christina Ciocca Eller on the one-year anniversary of the White House Year of Evidence for Action. Christina is an Assistant Professor of Sociology and Social Studies at Harvard University, but she took a public service leave between June of 2021 and December of 2022 to serve as the Assistant Director for Evidence and Policy at the White House Office of Science and Technology Policy. During that time, her office teamed up with the Office of Management and Budget to launch the White House Year of Evidence for Action which sought to accelerate evidence to make evidence, data, and science central at the highest levels of federal decision-making while pioneering new initiatives to drive evidence-based outcomes for the American people.

Mike talked with Christina about what motivated the year of focus activities on data, evidence, and science in the Federal Government, what the effort achieved, and what's next in the movement to embed evidence in decision-making. I hope you enjoy the episode.

[MIKE BURNS]

Okay, well, thank you for joining us. Today we'll be talking primarily about the work you did while serving as the Assistant Director for Evidence and Policy at the White House Office of Science and Technology Policy. But first, could you talk a little bit about your work as a researcher? What topics interest you?

[CHRISTINA CIOCCA ELLER]

Thanks so much for asking and thanks so much for having me.

I like to say I'm a scholar of multilevel inequalities. What I mean by that is that I'm really interested in how big-picture institutional forces – things like the law or our culture or kind of the way that we've always done things over time – impact organizations and their support of people and then how the organization therefore funnels down to individuals and really impacts what individuals can and can't do and how they do it and the inequalities that exist as a result. I tend to like to think about this in the context of U.S. higher education, especially public higher education, which is a space that is full of these multilevel relationships where things that are happening kind of way above the fray of individual people have really disproportionate impacts on what students can and can't do, which students have opportunities, which don't, which end up kind of ahead of the game, and which behind. So that's what tends to occupy my intellectual space.

I guess the other part of it is that I characterize myself as a mixed methods researcher. That means I use lots of different kinds of methodologic tools – whether that be big data on statistical analyses or interviews and ethnographies. I really kind of try to get a 360 picture of what's going on to make sense of how people have opportunities and how they don't.

[MIKE BURNS]

So I can see what would be appealing to you about serving as the Assistant Director for Evidence and Policy at the White House, but how did you come to that role?

[CHRISTINA CIOCCA ELLER]

Oh, geez, I mean it was a real big surprise to be honest. I was not expecting to get a call to serve by any means; in fact, I had just started in my Assistant Professor role in the fall of 2019 right before the pandemic. Then I got kind of a call out of the blue in the spring of 2021 to ask if I would take up this role. Since I'm a sociologist, I think, in relationships, it won't be a surprise that part of why that call came, I think, is because of preexisting relationships. I had known about sociologists were kind of starting to percolate up into the infrastructure of the Office of Science and Technology Policy; and in particular, there was some interest in having a kind of science and society new portion of the Office of Science and Technology Policy.

So it was kind of through the grapevine of sociologists that we all became aware that Dr. Alondra Nelson would be serving as the Deputy Director for that new portion, Science and Society, a renown scholar of race, technology, and equality. So It was Alondra who gave me a call and said, "Hey, I'm going to try to do this thing. Would you join me?"

Really, I couldn't say, "No." I really have always been a scholar who's thought quite a lot about how research and policy might coexist – and not just coexist but even comingle -- in real and tangible ways. And I really couldn't think of a better way to try on that hypothesis for size than actually joining the Administration.

[MIKE BURNS]

And your time at the Administration, of course, coincided with the White House's launch of the Year of Evidence for Action. We're recording this podcast in late April 2023, which is about a year after the launch of the campaign, which was of course of interest to Mathematica as an organization committed to evidence-based policymaking. So could you help us just set the scene a little bit? What led up to the creation of the campaign? Why was it important to the White House to make 2022 and into 2023 a year for evidence in action? What's really the origin story?

[CHRISTINA CIOCCA ELLER]

So the story, like many others, has people; it has ideas, and it has action. So let's go through each of those.

First, there are people. The Policy Council, the Federal Administration's Executive Office of the President, are so fascinating because they turn over with every administration; but long before the Biden/Harris Administration came up into the role of leadership, there were years of work going on inside the Federal Government, mostly across career civil servants, to try to think about the role of evidence and policymaking. This came up and through the Evidence-Based Policy Commission, which issued a report to Congress. Then there's the passage of the Evidence-Based Policymaking Act in 2019. So there was a lot going on for a long period of time on evidence, building evidence use, before Biden/Harris even began to think about what would it mean to make a federal framework and kind of put the full weight of the Administration behind this.

So I worked really closely with folks at the Office of Management and Budget, particularly in what's called the Evidence Team. This is just an incredible team of dedicated career civil servants who are trying to kind of create/generate/foster a culture of evidence inside the Federal Government -- not just them, there are certainly many people who are part of this infrastructure, but they certainly play a leadership role. Through working with them, it came to my attention – it came to all of our attention at OSTP – that the spring of 2022 would be the first time that federal agencies would release what they call their "learning agendas."

These are basically large-scale research plans that capture the kinds of questions that every federal agency was really interested in answering -- so again, career civil servants working really hard to create research plans that they would put out publicly into the world for the first time. There's always a case that there were some kind of research evidence generation/evaluation/capacity going on, but this is the first time that the plans would be introduced to the public.

So after all of this process of generation, people working really hard for a number of years, it seemed to my colleagues and the Evidence Team and me that this was a moment to kind of pause the planning and then jump into an action phase for evidence generation and evidence use inside the Federal Government. This was also really an important thing for the Biden/Harris Administration, which made the timing just perfect in that the Biden/Harris Administration within the first week of President Biden taking office, he issued his presidential memorandum on restoring trust in government through scientific integrity and evidence-based policymaking, which OSTP and OMB together had the responsibility to fulfill and implement.

So it was sort of like all the stars aligned on this idea – moved on to the idea – that we should move from idea generation to action. It was at that point that we said, "Hey, this is time for kind of a targeted campaign to see what we actually need to do," because it's one thing to have the idea that it's time for action; it's a completely other thing to put all of the details together, the brass tacks of strategy together, to actually do the action. So that's what we wanted to start thinking about with the Year of Evidence for Action.

[MIKE BURNS]

So what did that look like in terms of implementation of the strategy? What was it like on a day to day, or what was going into it?

[CHRISTINA CIOCCA ELLER]

A lot was going into it, but I think it's important to talk about the fact that we were grounded in three key goals that allowed us to actually structure pretty much all of our action according to those three goals. So I'll just say what they were, and then I'll talk about kind of how we acted into those goals.

The first big goal was to share leading practices for the use of evidence that were coming from Federal Government to use research facts, data evidence facts, in the generation of new policy and the evaluation of existing policy. So there was already a lot of really good work going on; but the fact of the matter is there was not a lot of uplift or narrative around what exactly everyone was doing, and what was working and what wasn't. So the first big thing was to share these leading practices across the Federal Government but also from outside the Federal Government, the non-government sector, back into the Federal Government. That was a great goal because it allowed us to have such important conversations with folks at different levels of government, not in government at all, just about what it really looked like to use evidence-informed policy well. So that was the first big one.

The second one was to basically strengthen and develop new strategies for more consistent evidence use across federal agencies. This really gets at the point that federal agencies are incredibly varied in terms of the extent to which they had built up an infrastructure for evidence generation and use over time. Not only are they very across from one another, they're super varied within one another. So part of this was about what does it look like to basically have those agencies with more familiarity to kind of share back to the agencies with less familiarity, or the units within agencies that are ahead of the curve to kind of help those that needed more experience and practice get higher up on the curve. So we had that big chunk of work.

Then the final one, which was my favorite just because of where I'm coming from, was to increase connection and collaboration among researchers, knowledge producers, and decision-makers both inside and outside of the Federal Government.

Getting back to that issue of how to do the work, it's just striking. Now we have so many research questions that have been make public by the Federal Government across all Federal agencies; but the fact of the matter is, within those agencies there is simply not enough capacity to answer all of the questions. There's just not.

So now that I'm back on the other side, part of my goal setter mission has been to explain to my colleagues in universities and non-government positions that just because the Federal Government has now named the research questions, it doesn't mean that they have totally figured out how to answer them. They need our help. So this third pillar was really about how to we create a broader conversation about this coordination gap that you might see between federal agencies who have lots of questions and researchers outside the Federal Government who have – I mean, it's our whole job to be producing research evidence. So what would it look like to harness that energy in service of these big questions that the Federal Government is asking?

So those are the three big goals, and then we did a bunch of things. I'm happy to talk about that too.

[MIKE BURNS]

Yes, please do. What I was – the point you made about kind of collaborating internally with your team throughout the Administration and then with other external stakeholders, were there any insights that came up that were kind of surprising to you that maybe did have an impact on the work that you were carrying out?

[CHRISTINA CIOCCA ELLER]

Yeah, a lot, which is fantastic. One of the things that I wanted to say to frame an answer to that though is that different people come to this work in different ways. So certain things that are surprising to some people might be completely run-of-the-mill for others. This is why communication across is so important.

But for example, some people who have been kind of based in what I'd call "translational organizations," which tend to be nonprofits or think tanks or these kind of units that sit between the hard rigor of academic research and the government, they are doing this kind of translational research work all the time. I'll speak for myself; I just didn't fully understand this layer of infrastructure that exists in the evidence space that can indeed connect with people who are kind of straight-up academic researchers; take big ideas; help to explain them in ways that might be more relevant/legible to particular policy issues; and kind of formulate them so they might be able to cross the transom. So that was a really big piece that was an important kind of learning – was how to tap the expertise and excellence of those kind of organizations.

Another was that oftentimes the most successful evidence use is just deeply cocreational, meaning that you're working with people in community – the communities that are receiving or taking in the policy actions – or the practitioners what are serving those people in order to generate the questions and the answers in the first place. This is part of a broader move to have a more democratic approach to research, which is quite kind of up-and-coming in the social science in particular – but just seeing how that might work to actually enable implementation of policy, especially at a local level – had big lessons for how the Federal Government might think about more effective policy generation using research too.

[MIKE BURNS]

Then just looking back over the campaign's efforts through that Year of Evidence for Action, what would you say are some of the biggest accomplishments?

[CHRISTINA CIOCCA ELLER]

Oh, man, if there is anything I feel is among the greatest accomplishments it is just knitting together a network of people who didn't know that the others existing but who all could be mutually supportive of this larger endeavor of connecting – I like to call it the "evidence ecosystem." So we had, for example, these 12 evidence forums, which were these events cohosted between my office, the Office of Science and Technology's Policy; OMB, Office of Management and Budget; and then a group of leading nonprofit orgs and academic institutions where we basically took up various tranches of these questions around what it looks like to do evidence-informed policy.

I think the coolest part of that for me – and certainly, the events were all very powerful – but was that then a broader network of kind of co-interested people started to form across these different nonprofits and academic institutions which then started to trickle out into their networks, and so on and so forth. So the ripple effect of creating this kind of community of practice, I think, will continue to be very powerful; and I'm also really, really pleased about that.

We also had a couple of other kind of more concrete wins, I guess you'd say. One of which I have been really excited about was a new program hosted by the National Science Foundation called the Analytics for Equity initiative, which basically provides NSF – and in this case, contracts instead of grants – but contract funds to research teams to directly answer federal agencies' research agenda questions, learning agenda questions. So now we're actually purposing federal dollars governed by NSF's use-inspired research mission, which sits alongside the basic research mission, in order to actually incentivize connecting up the academic research sector with Federal Government researchers and policy folks with the hopes of being able to kind of create a positive cycle of creating new knowledge that then could circle back into policy, and so on and so forth.

And then the final thing is that we were able to kind of remount some structures inside the government that we'll be able to attend to these issues and hopefully help with that culture of evidence and even flourish. The one that I was most involved with was called the Social and Behavioral Sciences Subcommittee of the National Science and Technology Council. This is a really longstanding body that, unfortunately, had its charter revoked under the Trump Administration for reasons that no one quite knows. But under Biden and Harris, we were able to remount it; and now that body is slated this spring to produce a new report on the role of social and behavioral sciences in evidence-based policy, basically.

So it was a real time of momentum, energy, and I think some concrete wins which if everything goes as we hope it will, part of the big goal is to create some proof of concept so that when federal agencies are going back to the Hill and saying, "Hey, this is a big deal. We're trying to do evidence-based policy here," folks on the Hill have their ears open and say, "Wow, you've done some incredible things; that's very effective. Perhaps we can budget for some more staff."

[MIKE BURNS]

Did you ever find a – because of course we talk about evidence-based policymaking, but the reality is at times there is disputes over evidence and what is true/what is not.

[CHRISTINA CIOCCA ELLER]

Oh, yeah.

[MIKE BURNS]

I imagine that would be a barrier to the work here.

[CHRISTINA CIOCCA ELLER]

Absolutely.

[MIKE BURNS]

I kind of thought of that when you mentioned going to the Hill. Obviously, that's a very political place. So how did you navigate that context throughout the campaign and throughout your work in the Administration?

[CHRISTINA CIOCCA ELLER]

It's a really great question. I think we have to situate this kind of push for evidence-based or evidence-informed as the new state-of-art policy alongside other conversations that had to do with things like scientific integrity, ethical conduct of research, and so on and so forth. Because there can be a fine line between evidence-based policy and policy-based evidence in terms of folks wanting to take research evidence to make points that they've already decided are true, regardless of what the body of work says, which is certainly not what the Biden/Harris Administration and not what any of us who are researchers would ever want to happen.

So the thing that we sort of held in order to navigate some of that was that many fields of knowledge are always in transition; they're always in flux; they're super dynamic. But many fields have arrived through many, many studies at some cores of consensus that are – nothing is indisputable, but that are quite clear. There are enough studies using enough different methods over a long enough period of time to show that the science is, for lack of better, relatively settled. It's in those areas where I think we felt most strongly that it would be viable to make arguments that this is an area that we absolutely need to allocate more resources towards.

And really, there are areas of settled science in most fields. It's just a matter of doing the work to see what's settled and what's not, which is hard for a government that's quite understaffed in terms of the human talent and capacity around these issues. So I feel like what I'm talking myself into is we're sort of even in the preemptive point of – we're like in the pre-political because the fact of the matter is the staffing is so lean on evidence and data issues that to even do the work of understanding what the settled science is would require more people than the Government currently has at its disposal.

[MIKE BURNS]

And then just to switch gears just a little bit, so I imagine as you're doing this work you're dealing with a lot of large datasets, multiple agencies accessing the same evidence. Were there considerations to protecting and stewarding that data? Can you speak to the overlap of data access, collection, sharing, protection?

[CHRISTINA CIOCCA ELLER]

Yeah, absolutely, this is a huge issue that we got at from a number of different angles. One of the big things that happened with the passage of the Evidence Act, which happened in 2019, was basically it was put into law that there had to be a single portal through which any user would be able to access federal data resources. So part of the time of me being in Government was the process of this single secure portal being piloted and now made public.

So questions of data usability, data sharing, data privacy – huge, huge issue, data privacy. This is at the same time that the Census Bureau was thinking about, for example, how to create datasets with enough privacy to allow you to both get a deep look at holistic processes but also to protect individual identity, especially in underrepresented small subgroups. Like "synthetic data" is the term that is used as one of the possible tools in that space. So all of these things are happening side by side, and they certainly are not disconnected.

I guess the other thing I would mention in that space, as many of you probably know, now there's a call out for Request for Information on Statistical Directive 15 of how we collect race and ethnicity data. That came out through the work of the Equitable Data Working Group, which I worked closely with, and was led by Dr. Alondra Nelson and Dr. Margo Schwab, who was the Acting Chief Statistician during the point at which all of that work was happening.

That was another big question about what should we have as minimum standards for data if we indeed want to pursue equity analyses and understand what policies and practices are doing for people from different backgrounds and different ways – so lots of action on data, not even to mention all of the stuff about AI, and so on and so forth. But these are extremely related, and I'm glad you brought it up.

[MIKE BURNS]

Then, you had highlighted a lot of the momentum around evidence in this Year for Action. What are some of the opportunities for improvements in terms of making evidence most useful and impactful for public policy?

[CHRISTINA CIOCCA ELLER]

I think that a piece of this is there are some kind of usual suspects that always come up in conversation. There is always if you're an academic researcher, you can't just give over your full paper; it doesn't mean anything. You need the one-pager that has all the snippets of exactly what the findings are and why that matters.

There's the "make relationships with people" because it's relationships rather than your writings that are going to matter. Like be the person who gets called up on the phone and asked about certain issues, and so on and so forth.

As I've thought about it, I actually think there's a piece of this equation that is much more foundational but potentially much more important, which is that I don't think that folks in the Academy on average know a whole lot about how the Federal Government works and vice versa. Because on average, there are not a whole lot of kind of academic-y people, especially sitting in the policy roles; and there are not a whole lot of policy-y people who are hanging out in academia for long periods of time and building a career there. It’s a little different in the statistical agencies where folks probably came up, did a Ph.D., and so on; but they, in fact, kind of have regulatory limitations on the extent to which they can cross over into the policy space at all.

So I think my key message here is that from the side of academia, there are some pretty concrete things that we could learn about and participate in that wouldn't be a huge burden in order to put our two cents in – things like RFIs that we've talked about, but also some basic things like now there are now dashboards on Evaluation.gov and Performance.gov, basically showing you what kind of research is going on across federal agencies and giving e-mail access to people who are creating those agendas.

You know, we're all people at the end of the day. I can tell you based on my experience, if I e-mail someone in Government and I say, "Hey, came upon this thing," it might take a week or two given capacity constraints, but I'm going to get an answer. So I think there's like this knowledge thing. I think this is like there is not – we don't have to see it like a black box because there's plenty of information about how government works.

Then from the side of government, a lot of structures in government do not make it easy for people to feel connected. So I think further, they don't typically – just again based on capacity – invest a whole lot in figuring out what motivates and incentives people in the Academy. So I think we just have a lot more cross-fertilized learning to do, and I think that that's really at the root of how to get better at serving each other.

[MIKE BURNS]

Well, you transitioned perfectly to my next question because I can tell that you have a background in communications, if I didn't already know that. But you worked in Communications for several years between college and going back to earn your Ph.D.

[CHRISTINA CIOCCA ELLER]

I did.

[MIKE BURNS]

What role can communications play in making data and other evidence useful for action?

[CHRISTINA CIOCCA ELLER]

I love the question. I'm glad you asked it. I think it's just to realize that people are attracted to stories; and what evidence and data are, are ways of telling stories about people doing things in the world. So it is not perhaps the most powerful to say one statistic or to cite one finding from a piece of research, but it is extremely powerful to tell the story of how a number of statistics alongside a number of empirical findings are giving us insight into how a given person might find themselves in spaces of opportunity or constraint.

So when I talk about my research and higher education, it's one thing for me to say, "Black and brown students on average graduate at lower rates than white and Asian students." It's another thing for me to tell you about Janelle C. and the way that her parents came to this country from the Dominican Republic in order for her to have a higher education, and how she worked two jobs and did all the things that she could and was afforded the resources of this, that, and the other thing and found a way to negotiate all of the challenges of her time in college in order to make her way to a degree, right?

And to recount quite clearly the policies that enabled her trajectory and the practices that were central to her ability to navigate, whether that be support, staff at the university, a kind of professor, the fact that a small nonprofit gave her a grant, and so on and so forth. So I think what we all could learn, those of us who see ourselves as evidence wonks, is that if we really want to say something about data and evidence, we probably also want to say something real about people in the world.

Knowing how to communicate that and knowing how to shape our message, depending on the audience that we're talking to – because, who knows, many my various academic college would be very happy to hear the statistic or the single empirical report, but folks on the Hill would not and vice versa. And having that approach to how we're communicating our knowledge and the evidence that we have could really powerfully change its impact for things that really matter to us and to society.

[MIKE BURNS]

Before we started recording, you'd mentioned that you're finishing up the semester here. So again, I want to thank you for taking time because I know it's busy. You are in a kind of a unique role in that you were in the Administration; now you're back in academia. At Mathematica, we're always interested in developing the advocates for evidence, right? And you have this kind of front-row seat with the next generation of potential advocates for evidence.

[CHRISTINA CIOCCA ELLER]

Right.

[MIKE BURNS]

How do we reach younger people on a topic that could be a bit wonky but I think is very important to having meaningful impact and driving data-driven, evidence-based policy?

[CHRISTINA CIOCCA ELLER]

That's a great question. I try on some of my communication strategies with many of my students. I think it might be quite dry for them to hear – to read a piece of research that doesn't have some sort of connection or impact on their day-to-day lives but actually quite profound to read a piece of research that they feel they connect to and they relate to in ways that are quite personal.

So from the perspective of pedagogy, this is a great example of when knowing your audience is really important because stimulating initial interest in these topics comes from personal connection and probably not necessarily from a super top-down approach. So I think that's one piece of it.

Then the other thing I would be absolutely remiss if I didn't say is part of this conversation is that as much as we'd like to think that evidence – we're using this term "evidence" very unilaterally – it's also like very vague, very complex, very nuanced. We still have a lot of work to do to figure out the extent to which evidence-based interventions have better and more positive impacts than "non-evidence-based" interventions. It's hard to know the extent to which evidence-based interventions sit alongside other things that are just simply human – like personal relationships or the concerns of people we care about, and so on and so forth.

So I actually really strongly feel that part of educating my students about evidence-based approaches is also to share with them the idea that we really owe it to ourselves to think quite critically about what evidence is, what it does, for whom, under which conditions, and to keep in mind that it really could run the risk of having quite a kind of top-down, technocratic slant to it if we're not embracing the idea that this is still a fluid space that needs to be questioned and interrogated, just like any good critical thinker that we're trying to educate in higher education would be able to do.

So I think by not taking the kind of like hammer and the nail approach and actually helping to raise the question sets around this, that also can be quite provocative to students and make them more interested in trying to understand and ascribe to various pieces of this overall issue that we're talking about.

[MIKE BURNS]

Christina, I could keep asking you questions all day. So I'm going to cut myself off after this one, but I have to ask. For folks who are "evidence wonks," as you called us, what should we keep an eye out for kind of in the year ahead coming out of the government in the evidence space?

[CHRISTINA CIOCCA ELLER]

So first of all, I would say that to me the evidence chapter of the president's budget is like a road map because it basically spells out what the things are that we more or less know are going to happen in the year ahead. Even if they're underfunded, there's energy towards doing those things.

So the first thing is just a basic teacher-y point, which is to go read the evidence chapter in some depth for hints about where we should kind of be looking next.

The other document that I think is really important, which usually comes out in August – so we're past the last one – is the R&D Memo cosponsored by OSTP and OMB. That is a huge road map also.

The other thing I would say is that Evaluation.gov has become kind of the hub of all things evidence in the Federal Government. When I was there, we did a bunch of content on the Year of Evidence for Action. Now they have this great dashboard, where you can basically subject sort all of the learning agendas into areas of interest. I think that there's going to be something even more exciting coming in that space – I hope – in the few months ahead. So do subscribe to their newsletter. You can just click a button on their website, and you'll be kept completely up-to-date with all of the kind of heavy-hitting things that are coming in Evidence Land from the Federal Government.

[MIKE BURNS]

We'll be sure to keep our eyes out.

[CHRISTINA CIOCCA ELLER]

Please do.

[MIKE BURNS]

Well, thank you so much, Christina. I really appreciate it.

[CHRISTINA CIOCCA ELLER]

It's my pleasure. Thanks so much for all the good questions.

[J.B. WOGAN]

Thanks for listening to another episode of On the Evidence, the Mathematica podcast. This episode was hosted by Mike Burns and produced by the inimitable Rick Stoddard. Our guest was Christina Ciocca Eller. Our show notes will include resources from the White House Year of Evidence for Action.

You can catch future episodes of the show by following us on YouTube, Apple podcasts, Spotify, or wherever you listen to podcasts. You can also visit us at www.Mathematica.org/OnTheEvidence.