Fraud, abuse, and errors divert resources from government programs and undermine public confidence in these programs and the agencies that administer them. Earlier this year, the U.S. Government Accountability Office (GAO) reported that federal agencies estimated they spent $236 billion dollars in improper payments in federal fiscal year 2023. Furthermore, GAO noted that actual improper payments are likely higher, as estimates for some large and at-risk programs were not included in these figures. GAO also stated that federal agencies have reported $2.7 trillion dollars in improper payments since they began collecting this information in 1997.
To address this problem, government agencies have implemented various initiatives to protect their programs’ integrity. While they differ in important ways, a common feature of these initiatives is that they are complex, as reducing improper payments can require as much program knowledge, operational sophistication, and coordination as is needed to run the programs.
Evaluating and improving the effectiveness of these program integrity initiatives is critical, not only for ensuring prudent financial and program management, but for programs’ viability. Successful integrity initiatives help reduce the risk that programs will be eliminated or be mandated to reduce the benefits and services that they can offer. That said, evaluating these initiatives is not easy. It can be challenging for program administrators to establish which impacts they can attribute to the integrity initiative and to calculate its return on investment. Determining whether and how these initiatives can be improved can be even more challenging.
We have helped various organizations review their program integrity initiatives. (Because this work is typically nonpublic, and to ensure confidentiality, we are not providing specific examples that identify our clients unless that work is publicly available.) The complexity and scale of these reviews have ranged from multimillion-dollar, multiyear, rigorous mixed-methods evaluations to three-month advisory assessments that compare an initiative’s existing practices and results to benchmarks. The reviews differed in their methodological rigor and data sources. In some reviews, we used formal, rigorous research designs and multivariate models to establish what would have happened in the absence of the initiative; in others, we used structured interviews and document reviews to develop a rich understanding of the initiative and identify improvements.
Regardless of these and other important differences, the most successful reviews start with clients answering four basic questions that require them to step back from day-to-day mission-critical operations.
What are the goals of the program integrity initiative?
Key stakeholders often differ on the goals of an initiative. Some stakeholders might believe the key goal is to recover money lost to fraud and abuse. Others might consider the most important goals to be preventing fraud or improving program operations that result in either overpayments or underpayments, including instances in which eligible individuals are denied benefits to which they are entitled. Some might believe the initiative exists primarily to support teams tasked with generating savings and recoveries but do not see it as the vehicle for realizing these or other tangible outcomes. Clearly and explicitly defining goals is a critical first step for assessing program integrity efforts. In our program integrity work, we often ask key decision makers questions to clarify their expectations and vision for the initiative’s goals. We use that understanding to guide the evaluation.
How does the program integrity initiative operate and what factors hinder or enable its success?
Program integrity initiatives are typically complex and multifaceted. An array of factors, such as the extent of staff analytical skill and program knowledge, staffing levels, available budget, the extent to which processes are automated, and the results efficiency of the initiative’s operational reviews—essentially its ability to achieve substantial results at the lowest marginal cost possible—influence whether, how, and how well the agencies perform certain activities. For example, the use of self-audits to recover improper payments will ideally only require a modest level of the initiative’s reviewer staff time. In addition, the willingness of parties to act on review findings and the extent to which the initiative’s stakeholders work together and establish a common set of priorities can also influence success.
When helping clients, we often start by developing logic models to identify how the initiative should work and to show the anticipated impact of enabling and impeding factors. Logic models provide a visual road map of how a program operates and identify the key factors that contribute to success. We then overlay an operational process flow to show how the initiative works in practice. Comparing the logic model and process flow helps us recognize implementation issues, potential gaps, and areas for improvement. For one client, we developed a series of high-level and, detailed flow charts to explain how the program works and identify each process used to prevent improper payments. This helped the client focus on their most critical gaps and urgent areas for improvement.
What measurable results does the program integrity initiative attempt to achieve or avoid?
Many program integrity initiatives measure activities and outputs (for example, number of cases reviewed, number of referrals generated), which suggests these are the focus of the initiatives. Too often, initiatives do not measure intended outcomes (for example, savings, recoveries), unintended outcomes (for example, inappropriate denials, delays in payments or services), or a combination of both types of outcomes. Understanding and defining the initiative’s outcomes and the results it seeks to achieve—not just its activities and outputs—is another critical early step in any assessment.
We help our clients define appropriate, measurable outcomes by drawing on our expertise in identifying, designing, and using practical, efficient measures that capture the initiative’s performance on various dimensions. In our work evaluating prior authorization demonstration programs for the Centers for Medicare & Medicaid Services (CMS), available here and here, we considered how well these programs realized potential intended effects: cost savings and reduced utilization. We also developed several measures to capture potential unintended effects, including adverse events and reduced quality of care and access to care.
What constitutes success for the program integrity initiative?
After clearly defining the initiative’s goals, operations, and measurable outcomes, the next step is to define success. What specific benchmarks or targets determine whether an initiative has been successful? For many organizations, achieving savings that are greater than the initiative’s costs, or a multiple of its cost, is the most fundamental benchmark. However, this approach does not weigh the impact of unintended outcomes. Organizations such as CMS define success for its demonstration projects, including its program integrity demonstrations, as the ability to realize program savings without experiencing meaningful unintended beneficiary outcomes, such as impeding access to care. Whether these or other options are most appealing for a given initiative, the key point is that decision makers should explicitly and carefully define success.
Evaluations and assessments of program integrity initiatives are critical for determining whether the initiatives are effective and how to improve them. They can also help administrators identify and understand the specific components of the initiative that are or are not working. This information provides guidance on whether the initiative should continue, and if so, can help answer questions such as: Should it be expanded or instead refocused on a narrower range of problems? How could the initiative’s focus or specific components be changed to achieve greater results at the same or lower cost? To obtain these and other insights and make these evaluations and assessments most beneficial, however, it is necessary to first answer important questions about the initiative’s goals, operations, outcomes, and definition of success.
