Building on the Evidence Act to Increase the Positive Impact of Federal Policymaking

[REP. DEREK KILMER]

Part of the value of the Evidence Act is that you're really trying to transition the federal government into agencies that are making data-driven decisions. And that's -- that doesn't sound like rocket science, but it often doesn't happen in this place.

[J.B. WOGAN]

I’m J.B. Wogan from Mathematica and welcome back to On the Evidence.

On this episode, we’re going to talk about ongoing efforts to embed the use of data and evidence in federal decision making. In late May of 2023, Mathematica coordinated with Congressman Derek Kilmer’s office to hold a two-part discussion on Capitol Hill about the Foundations for Evidence-Based Policymaking Act, or Evidence Act for short, which was signed into law about four and a half years ago. Kilmer was one of the original co-sponsors of the Evidence Act and he’s the sponsor of a new bipartisan resolution to establish a new commission on evidence-based policymaking that would advise Congress on how to make greater strides in embedding the use of data and evidence in federal agencies but also in Congress itself. That resolution was introduced on the day of event in May.

We’re going to play an edited recording from the convening, which was moderated by Jill Constantine, a senior vice president at Mathematica. The first part features Congressman Kilmer along with Congressman William Timmons, who is a co-sponsor of the resolution to establish a new evidence commission.

The second part features two people who were part of a previous commission on evidence-based policymaking, whose recommendations informed the Evidence Act: Robert Groves, a provost of Georgetown University, and Nick Hart, who is the president and CEO of Data Foundation.

Robert and Nick were joined by Ruth Neild and Susan Jenkins. Ruth is an executive director of education and employment research at Mathematica and she previously served as the Delegated Director of the Institute of Education Sciences within the US Department of Education.

Susan directs the Division of Evidence, Evaluation and Data Policy at the U.S. Department of Health and Human Services, where she also serves as an evaluation officer.

The conversation begins with remarks from Congressman Kilmer who represents Washington state’s 6th congressional district.

[REP. DEREK KILMER]

Thanks. And it's great to be with all of you, I mentioned to the organizers that it felt a little bit early to be talking about evidence and data, and they said not for this room. So nerds of the world unite, I guess. But I think it's a really important conversation, you know, particularly as this institution often gets wrapped around the axle, not on the things that we know but the things that we posit with no evidence to back it up. And that I think underscores the value of these conversations. We are -- William and I led the charge on the Select Committee on the Modernization of Congress. And one of the things that I think we really came away with was an appreciation for the value of evidence-based policymaking, that when the institution makes decisions grounded in evidence, it makes better decisions on behalf of the American people. When it makes decisions around budgeting, it's more efficient in its budgeting when it uses evidence. When it drives public policy that's grounded in evidence, it's public policy that's more impactful and more effective when it does. And so one of the recommendations coming out of the Select Committee was to establish a commission that's focused on evidence-based policymaking within the legislative branch. And the timing of this couldn't be better because we're introducing that bill today. So great, great job Mathematica and the Data Foundation for really nailing the day.

[ Applause ]

You know, and I guess the other thing I would mention and, frankly, most of the folks that worked on this bill are no longer in Congress. But the -- you know, it was four years ago that the President signed the Evidence Act into law. And I think it's -- you know, it's a process. But the fact is that it is pushing agencies to better use data, to better aggregate data, to make more available data, to make sure that it's machine readable, that it -- to create more of a culture that's based on the use of evidence and -- in policymaking. And I think it's really valuable that you have speakers today that are going to share some insights on how that is used. I guess I'll just end there. I know we have limited time. So rather than talking at you, I'd rather have a little bit of dialogue with you.

[J.B. WOGAN]

Now we’ll hear from Congressman Timmons, who represents South Carolina’s 4th congressional district.

[REP. WILLIAM TIMMONS]

Thank you. Good morning. It really has been the honor of a lifetime to work with Derek on modernizing Congress the last four years. We made a difference. And while it's not going to fix this place overnight, it will fix this place in the years to come. And I really appreciate his leadership and everyone's support of that endeavor over time. Our country's in a difficult spot right now because we don't really have truth anymore. Everybody has their own truth. And if you tell me what position you want to take, I'll go find you a think tank that'll agree with your position. And that's a big problem. It's a big problem when we can't even agree on the variables of the challenge that is before us. And we're watching it play out right now with the debt ceiling. I mean, we had the Republican conference yesterday, and everybody's arguing over when or whether we will actually default if we do not lift the debt ceiling. And we're talking all these variables about how Treasury is prioritizing things and all this stuff. And I'm just sitting here, I'm like, this should not be a question about what is going to happen. It is how you resolve the problem. But when you can't agree on the problems that you're facing, how in the world are you ever going to figure out a path forward? And so, I mean, if you think about it, I blame -- I blame this. Technology is the problem. Technology has really killed media. It's killed journalism. It has allowed anybody that wants to take a position on anything to scream it from the mountaintop, and the fringe has been empowered to have a far greater impact on our decisions than they're entitled to. But we got to figure it out. And I would argue that, while technology is the cause of our current situation, it is also our salvation. And we have to figure out how to adapt as a society to technology to drive our decisions in a more thoughtful and evidence-based way. And for the last four years on the Modernization Committee, I time and time again said that Congress is supposed to engage in evidence-based policymaking in a collaborative manner from a position of mutual respect. We don't do any of that. We literally don't do any of that. Each -- we fail at each -- each threshold there. So I'm hopeful that the work -- I'm hopeful and I'm optimistic that the work that we did the last four years, the 202 recommendations that are still being implemented, will help set us on a better course and help tackle the challenges that we continue to fail to overcome and that the best days are ahead. So thank you again for all of your work. And I just can't say how proud I am to have worked with you over the last few years. And we will continue to fight the good fight. So thank you.

[J.B. WOGAN]

Now we’ll hear from Jill Constantine, a senior vice president at Mathematica.

[JILL CONSTANTINE]

Thank you both so much. I just know I'm going to speak for everybody in the room, a little presumptuous, to just say thank goodness. So many of us are here doing what we do hoping we have partners in Congress that want to do all the things that you're saying. So we so appreciate that, and we are so with you. So I think we'll all go off in our days with a new level of inspiration. So mindful of your time, I'm going to just ask you guys the same question. You can take whichever parts of it or another question if you'd like if you think there's a more important one. So it's kind of a two-part question. If you take that perspective or view -- so maybe this is more for you, Congressman Kilmer -- from that first commission that led to the Evidence Act, what do you think has been the most substantial impact so far? And then looking forward, so you guys can tag-team on this -- given the bill you're cosponsoring that's coming out today to create the next commission, what are you hoping for in terms of a future impact?

[REP. DEREK KILMER]

Yeah. Well, I think part of the value of the Evidence Act is that you're really trying to transition the federal government into agencies that are making data-driven decisions. And that's -- that doesn't sound like rocket science, but it often doesn't happen in this place; that, you know, that law was one that really highlighted the necessity of data collection, of evidence informing policy decisions. And, again, I think most of the American people would say, like, was that not already happening? The other value of it is, when you aggregate and make available that data, there are things that will come of it that we haven't even thought of yet, right. You know, I represent a coastal district in the state of Washington. Anybody who comes and visits my district, you know, if they check the waves, you know, for -- if they're surfing, right, the aggregation of NOAA data is driving the -- you know, the most popular surfing app on my iPhone, right? That's how public data was used in a way that, frankly, I'm sure when NOAA was like, let's make this available, they hadn't presupposed that it was going to drive the surfers off the coast of Washington State. But it has, right? So I think, similarly, if we're able to make smart decisions regarding the use and availability of data within the legislative branch, there's a whole lot of stuff that may happen that we haven't thought of yet, and I think that's probably a good thing, too.

[REP. WILLIAM TIMMONS]

I'll give you one example that I’m particularly excited about, and it's related to modernization. I spent really four years complaining about the calendar and the schedule. We don't spend enough time here. We spent too much time in airports. And, when we're here, it's chaos. And every member that experiences it knows that we have a problem, but there's all these different views on how to fix it. And without looking at the actual data and creating a way to analyze the problem set, we've been paralyzed. We did not change it for this Congress. We've done better in some areas but worse than others. So one of the recommendations we made was a common committee calendar, which is already being used. It just came out a few months ago. And it's still in the pilot stages. And in six or eight months, after we're able to look at all the conflicts that each committee has with the members, ’'m going to have literally the dataset before me. And ’'m going to say, all right. If you do this calendar, this schedule, if you change this this way, and you can literally show that this is the least efficient way to do it. If you change it this way, you'll get this percent increase in committee attendance. You'll get this percent increase in opportunity to actually do our job. And once we can show all of that with evidence, and I–- start with the question. I’'s like, what do you think about this? What do you think about this? This is what the problem is; these are the different ways to fix it. How do you want to fix it. And without actually being able to put all of that on paper and track the conflicts each member has when they're in committee and how many conflicts they have with votes or with whatever, we're not going to be able to have that conversation. So just putting that on paper, and putting that in a way that you can then have a legitimate conversation about how to solve the problem. That's just one example. We have challenges all over the federal government where we are wasting enormous amount of time and money. And the better you can analyze a problem with actual evidence, with actual data, the more efficient we will be. And we will solve these problems as we are trying to do more with less dollars going forward. So I just think that this is the approach that we should be taking to all of the challenges before us and not politicizing every single thing that we do.

[REP. DEREK KILMER]

Let me give you another example in the legislative branch, and it stems from one of the recommendations that we made. You have 435 members of Congress that are doing casework where our constituents call regarding a problem with Social Security or with the Veterans Administration or immigration or what have you. Currently, there is no means through which that data gets aggregated in an anonymized way so that we can identify systemic problems within agencies, right. So you've got everybody -- everybody's solving this as a one-off problem. But maybe there's instances where you could actually collect that data and say, Hey. We are seeing the same problem in like 400 offices. This is a systemic problem that requires a policy response. One of our recommendations was fix that, right? Aggregate and -- anonymize and aggregate that data in a way that we can identify trends, problems, and solutions, right? I don't think anybody in this room would say, like, that's a terrible idea. Right? Like, that's cool. That's a no brainer, right? This is -- this is, you know, it's -- it may not be big game hunting. But it's, you know, small game hunting that could actually yield benefits for our constituents.

[J.B. WOGAN]

At this point in the event, an audience member asked what role the Congressional Research Service can play in the future of evidence-based policymaking. Congressman Kilmer had to leave to attend another appointment, but Congressman Timmons offered his perspective.

[REP. WILLIAM TIMMONS]

So, one, they're fantastic. And I've had an incredible experience with every issue. I have -- I mean, honestly, it might be one of the best parts of this job, that you're able to get experts from any field to spend time with you discussing issues because they care about what they do. And I have relied on them heavily. I would argue that very few members do. So the problem isn't CRS. The problem is members not using CRS. And I just think it's very easy to over -- oversimplify a very complex problem and take a political position on it. It's hard to dig in the weeds, understand the issue, and try to figure out a way to solve whatever the challenge before you is. So CRS has done a very good job of maintaining a nonpartisan perspective. And they might even go as far as to -- I wish they would kind of make some policy recommendations, but they can't do that. But, you know, I'm always like, What do you think? Because, like, we don't think. We present. We present the evidence to you, and you think. And I'm like, that's helpful. Thank you. Counsel's the same way. I'm like, What's the best way to do this? Like, that's not what we do. But, you know, I just -- I just think, again, it comes back to time. I don't think that members have the time to invest in policymaking because there's so many things that we have to do. And some of them are not fun. Some of them are more fun. And I also think that Congress, it's really hard to serve in Congress. I mean, the travel schedule is terrible. You have to do all of this fundraising, which is ridiculous. And it really is difficult. You either have to be independently wealthy, or this is the best job you'll ever get. And neither one is necessarily helpful to addressing our challenges. So we made some recommendations -- some recommendations to try to make it less difficult to serve in Congress, and I think that those will make a difference. But we've just got to spend time on the issues. We've got to spend time together. I mean, the biggest challenge right now in Congress, I think, is a lack of trust, a lack of trust. And when -- Derek and I got so much done because we built a relationship, and we had trust. And I worked on things that he could have easily blown up in my face; he worked on things that I could have easily blown up in his face. But we didn't because we don't do that. But when you are engaging in the legislative process, you inevitably have to push/pull, and our society does not want any compromise; and they want people to dig in and fight. And that's not the way you solve problems. Thank you all.

[JILL CONSTANTINE]

Thank you so much.

[ Applause]

[J.B. WOGAN]

After a short break, the event continued with a discussion between Robert Groves, Nick Hart, Susan Jenkins, and Ruth Neild. Mathematica’s Jill Constantine continued in her role as moderator.

[JILL CONSTANTINE]

So I will jump right into questions for our panelists. And I'll start with you, Dr. Jenkins. What success do you feel your division at Health and Human Services or Health and Human Services broadly has seen in the implementation of the Evidence Act? Kind of what have been the successes? But, also, what have been some of the challenges, and how have you, your team thought to address those?

[SUSAN JENKINS]

Absolutely. And I'm so glad to talk about this. Those that know me know that I can talk about evidence all day. But what I would say for HHS and my role at HHS is for many, many years Health and Human Services has had a strong evidence building and evaluation culture. We had an Evaluation Council that became the Evidence and Evaluation Council with the passage of the Act. And we had these conversations. And we have parts of HHS that have always had very strong evaluation work. The Centers for Disease Control developed an evaluation framework 25 years ago that almost everyone uses, just as an example. But with the passage of the Evidence Act, I think we really were able to elevate the role of evaluation more fully across HHS. Everyone has to think about evidence. All of the operating divisions and many of the staff divisions have to think about evidence. We've tied it much more closely to the budget process as we develop our budget justifications. And that has really been changing year over year over year, and each year, we see more evidence being put into those budget justifications, more evaluations being done. We still have challenges with some unevenness with our ability to generate evidence across HHS. We are working on that. We did our capacity assessment as required under Title I of the Evidence Act. And we have a series of trainings. We have operating divisions such as CDC, NIH, and others that are developing their own trainings, and we highlight those at ASPI in the Office of the Secretary. But we also have trainings that we conduct to fill some of those capacity gaps. So we're very excited about what the Evidence Act has allowed us to do. It's open doors. It's created new conversations that we weren't necessarily having wholesale before.

[JILL CONSTANTINE ]

Thank you. I'm going to do an add-on questions since you mentioned all the different operating divisions. And, as you alluded to, they started in very different places in their capacity for using evidence. They have different histories of developing evidence. So how does that different starting point for that implementation differ across this range of experience? Like, what does that mean for them, how they've been able to implement moving forward?

[SUSAN JENKINS]

We meet everybody where they are. And so we do have with the Evidence Council, the Evidence and Evaluation Council, we have representatives from all of the operating divisions that participate in that council. We have specific liaisons, but we also allow anyone to come. Anyone who wants to hear about, talk about, or think about evidence is welcome to come. And so they learn from each other. We have presentations. We pair people together when they're working on similar types of things. But the expectations for what people will supply to us for our required reporting under the Evidence Act is different. We have different types of evaluations that are happening in different of the operating divisions based on what they focus on, based on the populations they serve and the types of services that they provide. A research-focused organization such as the National Institutes of Health is going to have different types of evaluations that look at the value, the quality of the research that they're doing and how that research may be then turned into specific programs, turned into, translated into other things. With programs with the Centers for Medicare and Medicaid Services, there's a lot of direct service that's funded related to medical care, clinical care. We also have human services agencies that focus on other things that are not as healthcare focus but are the human services side of things. And so we meet people where they are, and we do embrace the scope of evaluations. And so the number of evaluations is going to be different. And we ask, then, rather than having the operating divisions provide all of their different evaluations because that would overwhelm -- some would have a lot more, based on the size, based on the scope -- we ask for the five most significant or up to five most significant evaluations so that then, when we report that out at the agency level, there is a little more parity there. But every division is supposed to publish under our evaluation policy their evaluation work, so that is still available to the public. So we work with the different divisions where they are.

[JILL CONSTANTINE]

Thank you. I'm going to keep moving down the line and stay with the agency theme for a little bit. Moving on to Dr. Neild, you've worked both in partnerships with agencies, leading agencies on responding to the Evidence Act, and that includes agencies with a great deal of experience and some with less. So have you observed how the Evidence Act supports both types of agencies, those that are more nascent, those that are more sophisticated in their use of evidence and data? And, if so, how?

[RUTH NEILD]

Yes. And this is a really great question to follow on from Dr. Jenkins' question and response to. And I would have to agree with so many of the things that you have already said, and I'll just sort of maybe amplify them a little bit from a different perspective. So, you know, it's -- it is true that, when the Evidence Act was passed, that agencies -- by which I mean sort of agencies, subagencies, divisions, departments were in really different places with regard to data and evidence. So some had data, good data infrastructure. Some had a long history of evaluation. Some had less of that. But from partnering with several agencies through work at Mathematica, I would say -- I would make just two observations. One is that all of the agencies had something to work with to begin to develop their data and evidence capacity. They might have had a performance culture, for example. They had something to work with. And I would also say that all agencies had some places where they could deepen evidence and strengthen their data. And so I just have a couple of examples to give. And one of them might sound sort of similar to your example. So one agency that we worked with already had a really strong performance culture. I think this is really something to sort of keep in mind is that agencies do have a history through Gepra [phonetic], for example, of using data and metrics and assessing their progress against those. But they use the Evidence Act to what I would say is add tools to the toolbox. So, for example, this involved building some staff capacity to understand how evaluation could complement performance management and performance indicators. How can those things work together? And once the staff really began to understand that, they begin to understand that evaluation and additional tools in the toolbox could help them answer questions about why and how, in addition to the questions about what that they had been more used to answering through performance metrics. And once they began to sort of come to understand that, it was really interesting to see the questions begin to flow. So there were questions like, you know, I'd like to understand whether the communication vehicles and formats that we're using are the best ones to reach the audiences that we are trying to reach. These are really, like, practical kinds of questions like the representatives were talking to us about. I'd like to know whether and how people in the field use the training we've given them. And, if not, why not? I'd like to know what the challenges are for agencies to respond to our directives and how we could help them comply faster and more completely. And so what I would say that we observed in some of these agencies that were just getting started with evaluation and beginning to see the possibilities of it is that they have this real hunger for better and more systematic information about whether they're really providing the services and the resources to the public that the public needs. The Evidence Act really offers them, I think, the prospect of getting better answers about that aspect of their work. Another thing I would say I've observed is that senior leaders in these agencies have been really remarkable in helping their staff to understand how to face honestly data, whatever that data says. It takes courage and it takes resolve. And it takes real leadership to be able to say, we're going to look at these data, and we're not going to be afraid of them. And then we're going to change our practice if we need to. So that's just an example of a more fledgling agency. I -- the second example would be really like ditto. So, you know, triangulating data, I would say that I've seen the same kind of thing that you're describing where other agencies that have stronger histories of using data and evaluation have really used the Evidence Act to deepen and to really spread. There's always going to be some offices in an agency where there's just not as much data and evidence going on. And they really use the Evidence Act to try to meet them where they are and push it a little deeper.

[JILL CONSTANTINE]

Thank you, Ruth, very much picking up on the theme of Congressman Timmons about people facing their data and having common data to work from. All right. So now moving outside of the agencies and to the rest of the community focused in evidence database policymaking and infrastructure, for you, Dr. Groves, the Evidence Act addressed a range of issues related to federal data and evaluation. In your work now with the National Academies of Sciences, you're looking at the role of data infrastructure and mobilizing information for the common good. Yes, there's a book you can take on your way out showing that -- that summary in that discussion. But how do you see the work of the next commission that will hopefully come about from the bill that's being announced today, the work of the next commission and ongoing work of the Evidence Act supports or catalyzes that broader effort to use data infrastructure and mobilizing for the common good?

[ROBERT GROVES]

Sure, sure. Thank you very much. Happy to be here. When the Commission on Evidence-Based Policymaking took place, there were certain scope decisions that were made pretty early. Some things were out of scope, and some were in scope. The out of scope issues included state and local government data and private sector data. Right now, in the United States, what we need to do is to have a new bridge between the kind of program-level data that agencies create through their processes and population data that allow us to compare the program participants to the larger population for a variety reasons. So what the Committee on National Statistics, a board of the academies, observed was there's just tons of little experiments going on in agencies that are just wonderful things. They're little ad hoc, one-off blending of data from different sources. But what was missing was a vision of a national-level data infrastructure that -- of which all the Evidence Act data would be a part of. So we took that on as a bold task. And if there were just a sentence to describe this, it's a reconceptualization of data from the private sector, from the nonprofit sector, from state and local government and federal government, as those data as a national resource and the assertion that the country that uses those data in a coordinated fashion for the common good will build a society that's much more robust than another country. So we have this vision that we're promoting. It currently could not be done given the legal infrastructure that exists. Blending data across different agencies actually has all sorts of statutory and regulatory obstacles, so it's not going to happen with our current structure. But we felt the need to give that vision and describe how the common good could actually be served by uniting data in various ways so that the future -- I think this is a 10-year future, not a 1-year, not a 5-year. But there are pieces that are happening, and Nick could probably describe some of the pieces that are happening. There are a lot of different ways to get there, I think. But getting there in our belief is critical for the future of the country. Thank you.

[JILL CONSTANTINE]

Thank you. All right. Let's move it along to Dr. Hart. So let's go back for you to your -- the Data Foundation's September 2022 Report, also highly recommended reading if you haven't read that. It highlights -- that's on my laptop, though. I can't show that. That highlights some clear impacts and -- of the Evidence Act and clear areas for growth. So I'd like for you to talk a little bit about what you think has been the single most important impact and what the single most important area for growth is. And it's okay if it's changed since you wrote that so what you think now of either of those things.

[NICK HART]

Yeah. So the report's available on our website, datafoundation.org. And it was actually a synopsis of the Evidence Commission's 22 recommendations. And a lot of those recommendations, about half, were covered by the Evidence Act. And I think there's an important takeaway, which is not all of the recommendations out of the evidence commission were actually covered by the Evidence Act. So there's an incredible amount of work still to be done by Congress and by the Executive Branch, not all of which actually require Congress to take action. Some can be done administratively. I would say one of the biggest impacts of the Evidence Act was actually a little bit of what you've already heard, which is culture change, setting the expectation that we are now -- and Congressman Kilmer said this at the top, which I think a lot of the American public would be surprised to hear that we're not using data in all of our programs. In reality, we are. But now it's an expectation. We have the mechanisms and the capacity that are now expected to be present across government agencies. The evaluation officers, the chief data officers, the statistical officials, the mechanisms are to be present in government agencies. But are they? And to me, that is the biggest question or the takeaway out of what we said in September of last year. And the answer is, sort of. And that is the next step, which is there is an incredible amount of work to be done to implement the Evidence Act. Well, the Executive Branch agencies have, with a great deal of fidelity, tried their best to implement the directives of Congress, Title I in saying, Let's have learning agendas. Let's establish evaluation officers. Title II in creating -- and this was Congressman Kilmer's really important part and contribution of the Evidence Act, the Open Government Data Act, let's have Chief Data Officers. We have Chief Data Officers, but many of them are single person offices with no resources. Well, it's really hard to establish a data governance function in a large department if you have no money, no people, no authority. How do you do that? Well, you can't establish an open data plan that gets your senior executive officials buy-in, changing the culture across the department if you don't have the culture change that comes with that. So we had the intent, but did we have the real culture change? Then let's talk about Title III. Title III, and this is much of what I think the National Academies is really interested in as a very important part of the Evidence Act, ready for it? Confidential Information Protection and Statistical Efficiency Act or CIPSE. Very powerful part of the Evidence Act, which encourages and promotes data sharing across the statistical agencies. Very important new part of the statistical laws that were modified with very strong privacy protections. As of today, not a single one of the regulations that is directed under CIPSE has yet been promulgated by the White House. Let that sink in. So the Evidence Act, now almost five years old, changed the context of how we think about some of these really important big, wicked questions. Congress almost unanimously passed that authority, changed the direction of the culture. Agencies are now doing lots of work to implement. But some of the really important mechanisms that we needed the next steps on haven't fully been implemented. So this is an important moment. This is a really important conversation to be having on Capitol Hill, because it's also the moment that Congress gets to step in and say, let's do some oversight. Let's ask these questions about what the next step is. How do we lean in? How do we encourage the Executive Branch productively to help us? We, Congress, have -- we're talking about the debt ceiling. We have these big questions that we're trying to answer. We know we have data collection needs, data sharing needs. We need help answering these big questions. We gave you this authority. Let's make the next step. And today the congressmen are filing this resolution about the next evidence commission. Actually, that's a really important step too. And we didn't talk about this in the report, but this wasn't technically part of your question. But I actually think it is part of the answer, too, because the evidence commission from 2017 didn't explicitly say, What does Congress need? But that's what this resolution does is it has Congress, this institution take a look at itself. And this is a really low cost way of Congress answering that question. This isn't intended to be a billion dollar exercise of getting expertise and feedback. This is a very efficient mechanism for us to quickly answer. What kind of data needs this Congress have? So, I mean, I think this is a really important moment for Congress to do that and build public trust in itself.

[JILL CONSTANTINE]

Thank you. I'm going to do one more question for the panel and then open it up or including from the audience or you all for each other. I'm going to pick up on that importance that Nick raised about the chief data officer. And hopefully I'm not going to put you on the spot too much, Susan, but I'm going to talk about that. Can you talk about the collaboration between you and your chief data officer for your agency. If the Evidence Act helped develop and expand that collaboration, how important is it? I feel like Nick has given you the cover to go, Yeah, sorry. That's not really happening. But it'll be great to hear if it is.

[SUSAN JENKINS]

And it is. I'm not going to disagree with Nick about the staffing and the resourcing. But, in terms of having the three officials, the statistical official, the chief data officer, and the evaluation officer, we do talk a lot. We have regular meetings. We have different councils. And so, in terms of the data governance board that was structured based on the Evidence Act, Title II, I sit on the data governance board as a voting member. And the statistical official is there also. Under my office is a HHS Data Council. And we have the statistical official and the chief data officer that sit there. We have a number of other agency commissions, a number of cross government commissions where we sit there together, we hear the same things, we work together. We work on training together because there are needs in terms of evaluation training and capacity building but also other evidence building, not just evaluation. But we can't build that evidence if we don't have strong data, if we can't link our data, if we can't harmonize our data, if we don't have data scientists that know how to use those data and in some ways, more importantly, managers of the data scientists that know how to use their skills as best as possible. So we work on training together. We work on joint budget requests together. And so there is a long way to go. I don't disagree with anything that Nick said about this. But I would say, at HHS, we've been really fortunate. I've been really fortunate. The other officials are a joy to work with and open to working. We work together a lot. We talk together a lot. And that's really exciting. And I think the Evidence Act does deserve some credit for that.

[JILL CONSTANTINE]

Thank you. So we have a few minutes left. I just want to make sure if there's any questions from the audience we give an opportunity for that.

[MATTHEW STAGNER]

I always have questions.

[JILL CONSTANTINE]

Matt again.

[MATTHEW STAGNER]

I think it's probably for Dr. Groves mostly. But thinking about the generation of business data and the use of government data buy-ins, how what's happening in the next ten years there where government might have better access to business data and we protect the government generated data that businesses rely on but I think don't quite appreciate as much as they might. But once it's gone --

[ROBERT GROVES]

Yeah. So I think there's a fundamental change going on. The data now are so large that you can't move them, right. So the government or the private sector won't be downloading datasets. So the compute world that has to be formed has to have privacy protecting software inside the firewall of the data holder that would emit statistical or aggregated data usually of some character useful for the government statistical agency. And then I think I would assert there has to be a quid pro quo. There has to be an answer to the question, Why would a company ever allow that to happen? And I think the unique value of the federal government agencies is they have the universe frames. They know everybody in a particular population. The private sector firm knows their customers, and then they purchase data that is of much lower quality on population through private sector data sharing. But they know it's fallible. So, under the commission, one thing that the commission did that I thought was wonderful was to say you have to build trustworthy institutions in this new world. That requires transparency, really radical transparency -- that was a term that didn't make it to the report; I was voting for radical -- so that you, as a private citizen, could know whether a data record that has your ID on it is being used in any way, right. This is feasible, technically. And then you can see the reports that, you know, you were one little data record in a report describing things as a trust-building exercise. So if you just think about this, there's a lot to be done here. There are experiments going on right now as we speak with this sort of work going on behind the firewall. But we need legislative changes to really make it hum, I think.

[JILL CONSTANTINE]

All right. In our last few minutes. I'll either -- if there's another question. Yes, please.

[Audience member] I'm happy to ask a question. So you all briefly mentioned the state and local data issue as being out of scope. I have had many frustrations with the fact that, at the federal level, you have limited information about what's happening at the state and local level and run into some real problems myself in trying to answer questions that would involve ideally having like a federal aggregation of basic information from the states, at least. I realize this is a big question that speaks to federalism and may be insurmountable to address. But I'm wondering if you have any thoughts on how this could possibly be improved?

[ROBERT GROVES]

You want to do this one?

[NICK HART]

Well, that's an easy question.

[Robert Groves]

Hard to answer, though.

[NICK HART]

So, I mean, I will say the Evidence Commission did have recommendations about state and local data, to give a little bit of credit to the evidence commission, Bob. But they were not very detailed and essentially suggested that it was an issue we needed to spend more time on. Last year, there was a group that came out of the Evidence Act called the Advisory Committee on data for evidence building --

[ROBERT GROVES]

Right. ACDEB.

[NICK HART]

Did I get that right? ACDEB -- that actually had representatives from state and local governments that were participating and also issued some very specific recommendations about the need for more cooperative relationships between the federal and state and local governments. And, not surprisingly, some of those recommendations were very specific about the need for better resourcing. A lot of federal data is actually state and local data. I think it's a really key point for those of you in the room who are maybe working on authorizations that, as you're thinking about the design of programs, that a lot of that data is actually flowing up to government, not the federal data flowing down to state and local. So as we're thinking about how we're answering these big, wicked questions at the federal lens, it's actually a systems design problem. And this need for better data sharing covers the entire system. So we are not going to change the entire ship in a day. But this is something that we hear a lot from the state and local providers as well, that they're trying to answer questions at the local level, that they're actually trying to look at beneficiaries that might be crossing state lines. And if they can't access -- I mean, I'm from Missouri. If they can't access the information of the people that are in Kansas but they're dealing with workers that are crossing the state line, that's actually a problem. And so they have to negotiate data sharing agreements. Well, data sharing agreements take time. So this is an area where the federal government can actually help facilitate, and we have lots of examples like that. I mean, Ruth, you worked on this in education context. You all deal with this in the health context. Census helps navigate this from the statistical perspective quite a lot. So I think this is very much an issue where there are solutions that are in progress. I don't know if others want to jump in here.

[ROBERT GROVES]

I want to say one thing. I'm actually optimistic on the states getting together because they're very interested in comparing. And there are kind of volunteer coalition's happening. You know, this is all wonderful. The state to fed is complicated by federalism. But, again, I think it's sort of like the private sector. What is the state going to get out of this? And I think there are really good answers that we could invent that requires the feds to think differently about their services to the states. And so I'm optimistic on this one.

[NICK HART]

I like that.

[JILL CONSTANTINE]

It was very good question as we get to the state levels, at all levels. All right. Well, thank everyone for coming. You can -- we have a few minutes. We'll be around for a little bit after if you want to grab some food or thank any of our panelists. Thank you all for your participation.

[J.B. WOGAN]

Thanks for listening to another episode of On the Evidence, the Mathematica podcast. This episode was produced by the inimitable Rick Stoddard. All of the resources mentioned on the episode are available in the episode’s show notes. If you liked this episode, consider subscribing. We’re available on YouTube, Apple Podcasts, Spotify, Stitcher, or wherever you listen to podcasts. To learn more about the show, visit us at mathematica.org/ontheevidence.