On the Evidence Marks 100th Episode | Robert Shea on the Federal Government’s Progress in Using Evidence to Improve Programs and Policies

[ROBERT SHEA]

We have seen a sea change in the ability of agencies to understand that they need to articulate what they're trying to accomplish in terms of outcomes, report that transparently. Dig deeper – find out whether it's what we're doing that's contributing to the ultimate outcome rather than some other factor.

[J.B. WOGAN]

I’m J.B. Wogan from Mathematica and welcome back to On the Evidence.

This episode is part of an occasional series on the show that we call Evidence in Government, where we talk about new developments in the halls of government and the role that evidence can or at least should play in decisions that could improve people's lives.

To lead these episodes, we've tapped my colleague Mike Burns, who is the Senior Director of Communications and Public Affairs at Mathematica.

Our guest for this episode is Robert Shea, a member of the U.S. Commission on Evidence-Based Policymaking, a group whose recommendations helped shape the Foundations for Evidence-Based Policymaking Act. Robert is now the chief executive officer for a government management consulting firm called GovNavigators where he hosts a podcast called The GovNavigators Show.

Mike talked with Robert about embedding evidence in the federal government, progress that has been made since the Evidence Act was enacted, and where there is still more work to do.

We hope you enjoy this conversation.

[MIKE BURNS]

Thank you for joining us, Robert. I want to first congratulate you on your new firm, GovNavigators. Can you tell us a little bit about GovNavigators? Then we'll jump into government performance and evidence-based policy.

[ROBERT SHEA]

Sure, sure.

When I was at my previous firm, Grant Thornton, I had a team that helped Grant Thornton understand the evolving policy landscape as it related to management improvement efforts in the government. So we thought we would try our hand at offering those same services broadly to folks in and outside of government in the government management space. I hope that makes sense.

[MIKE BURNS]

It does...to me at least. First, because we're going to talk a lot about this today, could you define what evidence-based policy is?

[ROBERT SHEA]

That's a great question. So evidenced policy is those policies that are derived from the conclusions and recommendations resulting from rigorous evaluations. That's the way I see it. The more and more actionable intelligence we can get from the evaluations we conduct can lead to better policies and program implementations.

[MIKE BURNS]

So you've got extensive experience in this space and in the government management space. Staff member for the House Committee on Government Reform and Oversight, legislative director for Congressman Pete Sessions, counsel for the Senate Committee on Governmental Affairs, associate director at OMB, head of Grant Thornton's Public Policy Team, and now GovNavigators. I mention all of these roles because you've been involved with some important benchmarks in government performance and evidence-based policy, including the Government Performance and Results Act and the Foundations for Evidence-Based Policymaking Act, also known as the Evidence Act.

So let's start first with GPRA. During your time with the Senate Committee on Governmental Affairs, you advised committee leadership on implementation of the law which was enacted in 1993. Can you tell us a little bit about the law and what progress or major improvements you've seen at the federal level in terms of government performance management and prioritizing outcomes in federal programs from the time the law was enacted up until the Evidence Act?

[ROBERT SHEA]

Sure, first I feel like I need to adjust my walker after that long dissertation on my background.

So it's funny, a colleague sent me a piece he wrote on the 30th anniversary of that law noting that it set the foundation for long-term strategic planning, the setting of outcome goals, and the transparent reporting on success or failure to achieve them by agencies. I think one of the biggest contributions it made was providing a vocabulary of outcomes that departments and agencies can use to translate the activities they're performing into important outcomes – outcomes that are important to the American people, policymakers who establish the laws/programs that they are implementing.

Where we were when we first started implementing the law – it was far less mature of course. So though I'm not sure the law accomplished all it promised. We have seen a sea change in the ability of agencies to understand that they need to articulate what they're trying to accomplish in terms of outcomes, report that transparently. A lot of what we're talking about today is sort of dig deeper – find out whether it's what we're doing that's contributing to the ultimate outcome rather than some other factor.

[MIKE BURNS]

Right, so then moving to today – or really a few years ago back in 2016, the Evidence-Based Policymaking Commission Act becomes law. That creates the bipartisan Commission on Evidence-Based Policy Making, which you were appointed to. That Commission in 2017 releases a report and recommendations; and then in 2019, the Evidence Act becomes law addressing many of those recommendations. There's a lot to unpack here of course, but I'm curious. What was your experience on the Commission? Were there any surprising takeaways? And for those who might not be familiar with it, what exactly does the Evidence Act do?

[ROBERT SHEA]

So I was delighted to be appointed to the Commission. I was one of Senator McConnell's appointees and, frankly, advising them on other potential appointees. I think they ran out of time and chose me. I was a little bit of an anomaly on the Commission in that I was not a Ph.D. economist/evaluator/statistician/privacy expert or advocate. My experience really was in assessing and working to improve operations and implementation of programs. That really provided, I think my fellow commissioners would agree, a unique perspective on the practical implementation of a lot of the things they were considering and the recommendations they ultimately made.

The staff and the commissioners had an unbelievable breadth of experience in this space, but their unifying quality was a passion for bringing to bear rigorous evidence in policymaking and implementation in government. So we made recommendations; and because of the passion of the congressional authors or creators of this Commission – Patty Murray of Washington and Paul Ryan, who was then Speaker – that really gave us a leg up in converting those recommendations into statute.

You mentioned the Foundations for Evidence-Based Policy Making Act. There are three titles to that Act – one dealing with the data side, one dealing with the statistic side, and one dealing with the evidence building or governance over evidence-based policymaking side. It created evaluation officers and chief data officers in agencies. It required agencies to produce at the same time they were doing their strategic planning a learning agenda; that is, what are the big questions that the agencies need answering in order to better achieve their missions. Then, the annual evaluation plans – what actual evaluations are they going to conduct to help answer those questions embodied in the learning agendas.

The hope is that not only strengthening the governance over evidence-based policymaking but really transform agency cultures into evidence cultures, evidence-building cultures, ones with curiosity over whether or not they're achieving their goals most effectively; whether or not what they're doing is actually not having an impact. And as your organization knows, a lot of times we find that what we're doing is not necessarily moving the needle; sometimes it can actually produce perverse effects. But really, we want to tap into curiosity among agency leaders/program managers, so that we can accelerate the pace of learning from a lot of the evaluations that we're conducting.

[MIKE BURNS]

So on the subject of implementation and impact, it would be great to hear your reflections on implementation of the Evidence Act and how you think agencies could be helped to fulfill its promise. Has it worked as you'd hoped? What would you highlight as its biggest success and maybe what hasn't worked as well as you would have hoped?

[ROBERT SHEA]

Well first of all, I'm delighted that we've seen progress across administrations – Bush to Obama, Obama to Trump, Trump to Biden. It was President Trump who signed the Foundations for Evidence-Based Policy Making Act and really set the foundation for implementation that we've seen the last several years. But of course the first learning agendas/evaluation plans were produced only in the last couple years during the Biden Administration.

The appointment of evidence officers, chief data officers, the councils in which those organizations sit and collaborate across agencies, the learning agendas themselves, the evaluation plans – Evaluation.gov is a forum on which that community can share a lot of its activities. I think those are all fantastic developments. Last year, as I'm sure you know, was the Year of Evidence for Action. A lot of the Administration's resources were devoted to evaluating the massive investments we're making in our recovery from COVID and the economic slowdown that resulted from that.

The results of a lot of this evidence building have really yet to have an impact on operations. That's not true across the board because we've been doing evidence building for a long time. But I really do think we've set the foundation for an incredible growth in at least the amount of evidence we have to drive policymaking/program implementation. I'm excited to see what that means going forward – the actual application of that evidence on operations.

[MIKE BURNS]

So along those lines, lots of agencies are familiar with implementing GPRA; but they're relatively new to the evaluation side of things, which is an important piece of the Evidence Act as you've mentioned. What advice do you have for these agencies to add new tools – like evaluation of policies, programs, and practices – to their toolbox?

[ROBERT SHEA]

I have a lot of empathy for agencies that are saddled with overlaying new management improvement requirements because of course we're talking about two major statutory ones. There are a lot of administrative ones, activities embedded in the President's management agenda. Of course this is part of that but other ones as well. It really does strain resources. I was delighted to see in the President's budget an enumeration of all of the investments that have been made in evidence building.

Agencies ought to really tailor implementation of these and other laws to the priorities of their leadership. Make sure that the evidence you're gathering and injecting into policymaking, your agencies, really speak to the interests of leaders because that's – the worst thing that can happen is that we're developing an enormous body of evidence that goes unheard or unused in the policymaking process.

Collaboration across these silos – you mentioned the teams implementing GPRA. They're often different from the teams implementing the Evidence Act and different still from those implementing requirements of the chief data officer. Those are critical collaborations. We shouldn't be setting up infrastructures to implement these things separately. This really should be contributing to the cycle of evidence building and use that helps transform cultures into learning ones.

I know that all sounds – well, I don't know how elegant it sounds; but I try to make it sound elegant. It's really hard in practice because a lot of these folks would have to share resources. They would have to do things differently than they've done them since the implementation of GPRA in 1993. You get that plug-and-play working on implementation of GPRA; it's not that hard. People don't really reflect on the utility of it all. This is really an opportunity to step back and say, "What of all we're doing is most useful?"

One of the buckets of cold water thrown on the whole effort was during the Trump Administration when they repealed Section 200 of OMB Circular A-11. I don't mean to nerd out on OMB circulars, but there was a real risk to a lot of the performance management activities as a result of that gesture. It was ultimately reinstituted; and I think the controversy over it all sort of diverted us from the opportunity we had to really reflect on what of all of that really deserves to persist, and what could we use to simplify/clean out a lot of those legacy requirements.

[MIKE BURNS]

We're just switching gears real quick to Congress. You were at a recent event that Mathematica held on the Hill related to the Evidence Act and evidence building and use in Congress. At that event we heard from Congressmen Derek Kilmer and William Timmons, who have introduced the Congressional Evidence-Based Policymaking Resolution. This bipartisan resolution would establish a bipartisan commission on evidence-based policymaking to facilitate the integration of robust data into their legislative process. I'd love to hear your thoughts on that and the move to make Congress more evidence-based as well.

[ROBERT SHEA]

Well, I'm still bitter. I got up early in the morning, put on a tie, found parking, and showed up; and there were like 20 people in the room. I could not believe – this was such a fantastic – and maybe I'm just a total dweeb that this attracted me. But I left my COVID lair to head to the Hill, and you all put on a fantastic event. I wish more people were there. Hopefully we can link to the video because you all posted the video online.

The promise of what the members of Congress introduced is that we will take sort of the experience in the Executive Branch and move it to the Legislative Branch. They want to really study ways that we can bring more rigorous evidence to bear on the activities of the Congress. They used some great examples. Every member of Congress, Senate, and House has teams that field calls and e-mails from constituents about problems they're having navigating the bureaucracy. Gathering all of that data to see where the real sludge is in the process of serving citizens – that could really be an enormous way to monitor the way Government is responding to citizens. Because you know when people are motivated to call their congressman, that's when you know it's really hit a block.

But there are thousands of other ways that Congress could be made aware at the time they're making a policy decision of evidence that has been gathered that can help them make a decision one way or the other.

Then of course you hosted a panel at which various experts from Mathematica, Georgetown, Health and Human Services if I recall correctly, and the Data Foundation – really good perspectives on celebrating where we have come from but also what the promise of this could be going forward. I really do think bridging the gap between what the Executive Branch is doing and the way Congress operates could be really important.

[MIKE BURNS]

Just one more item related to the Evidence Act. OMB just released its 2023 Spring Regulatory Agenda. This agenda details actions that federal agencies are considering over the coming months. Included in this agenda, and it took a little digging, but there is a scheduled publication of the Evidence Act public trust role. This is related to producing and disseminating relevant and timely information, conducting credible activities, conducting objective activities, and protecting the trust of this information by ensuring confidentiality and exclusive statistical use of responses. What's the significance of this role?

[ROBERT SHEA]

Well, I think there was a lot of balancing going on in the Commission in producing its recommendations. We wanted to ensure that our recommendations were founded on the need to ensure the privacy of personal information that Americans or others share with the Federal Government; and that to the extent it was going to be used to accelerate the building of rigorous evidence, we want to make sure that we didn't unnecessarily – not unnecessarily but that we were protecting zealously the privacy of that information.

I think if we can report on the extent to which that was done successfully, we can increase trust in data sharing across government, which is one of the main barriers to really accelerating the building of evidence across government.

This transparency question brings to mind who is tracking the publication of planned evaluations. One of the quickest ways to erode trust in all of this is to launch an evaluation, have a result that folks don't necessarily think is consistent with their policymaking goals, and therefore bury it – not release it. I think we ought to do a better job of ensuring – holding agencies accountable for publishing the evaluations that they have invested the American taxpayers' money in.

[MIKE BURNS]

So, Robert, you've given us some great nuggets. I'm going to try to pull some more from you while we have you. Where do we go from here? I'm curious to know what you think needs to happen next whether it's from a legislation standpoint, an administration standpoint, to make federal policymaking even more data-driven than it is today. Do you think new legislation is required to implement unfulfilled recommendations of the Commission, and is there room for Congress itself to become more evidence-based and data-driven?

[ROBERT SHEA]

I think this law under consideration in the Congress – it's not a law, it's a resolution – to create a commission on evidence-based policymaking for Congress is a good idea. I don't think new laws are necessary, but Congress is always hungry to act in some way. So I'm sure they'll find other ways to make their voice heard in this realm. So we ought to be open to that. We ought not simply say, "Stay out of it; we've got this handled." That's what we often said at OMB when I was in the Executive Branch. But we ought to really work together with them to find out what would be helpful and help them avoid what would be distracting or counterproductive.

I think the more and more we can invest in evidence building, the better off we're going to be. You can see in the debate around IRS, we know where financial investment would produce greater returns. This evidence building ought to give us a better road map of where those kinds of investments can go to get a bigger return. I'm not suggesting we put budgeting on autopilot; but we do want to see more and more money invested not in things that aren't proven, more and more that are in what the evidence says is going to produce a quantifiable outcome.

I think those are a couple of things that would show us that all this work is not in vain.

[MIKE BURNS]

Excellent, and then just one last thought. You've worked for a Republican member of Congress. You worked at OMB under a Republican President. As you mentioned earlier, you were appointed to the Commission on Evidence-Based Policy Making by Senate Republican Leader Mitch McConnell. Is this topic of data-driven evidence-based policy a rare area of bipartisan consensus in Washington? Are there any certain aspects of data-driven and evidence-based policy that you've found are especially appealing to Republicans or to Democrats? Because I know that Commission that you worked on was bipartisan in nature.

[ROBERT SHEA]

Right, you know it's funny in this weird era of questioning the truth that the subject of evidence-based policy might be considered bipartisan. I guess that's surprised me – that it hasn't come under fire from those who habitually argue what appear to be known facts.

The event you hosted on the Hill was really inspiring to me in that you had very smart, passionate Democrats and Republicans sitting side by side advocating for greater evidence-based policymaking across the board but especially on Capitol Hill. So I think to date it has, again, sustained across bipartisan administrations, has bipartisan support on Capitol Hill. So it does seem to me that this is one of those – at least to date – rare islands/oases of bipartisanship that hopefully will persist.

[MIKE BURNS]

Well, thank you very much, Robert. I really appreciate it.

[ROBERT SHEA]

My pleasure, great to be with you.

[J.B. WOGAN]

Thanks to our guest, Robert Shea, and thank you for listening to another episode of On the Evidence, the Mathematica podcast. This episode was produced by the inimitable Rick Stoddard. If you liked this episode, please consider subscribing to catch future episodes of the podcast. We’re on YouTube, Apple Podcasts, Spotify, and other podcasting platforms. You can also learn more about the show by visiting us at mathematica.org/ontheevidence.