At a time when private companies can mine online user data for new, sophisticated insights about their customers, public-sector agencies—particularly those charged with serving clients with low incomes and some of the most urgent needs—are struggling to keep up with their own data practices. Although public agencies collect reams of valuable information that could be used to improve residents’ health and well-being, they rarely have the ability to study, interpret, and use the data the same way many companies can.
To help state agencies in health and human services share, integrate, and use their data, the federal government funded a needs assessment in 2013 to gather information on the challenges state agencies faced in using their data to inform their programs.
Beth Weigensberg, a senior researcher at Mathematica, coauthored an article describing some of the findings from the needs assessment, which recently won an award from Public Administration Review, a peer-reviewed academic journal about government. In this week’s episode of On the Evidence, Weigensberg talks about the article, changes in the field since she and her coauthors conducted their research, and new ways state agencies might approach handling and using administrative data.
Click here to listen to the full interview. You can also read edited excerpts of the interview in the following transcript.
Big picture: Why do state administrative data matter, and how do these data affect people?
Overall, I would say that administrative data are a rich resource to learn about program implementation, to understand how participants are engaged in programs and what services they receive, and to learn ultimately if the programs are achieving their goals. The data are valuable for program administrators who are accountable to federal, state, and sometimes local authorities for the use of public funding to provide programs and services that help individuals in need. Program performance measures are often estimated based on the administrative data, so the data are important not only for program managers but also for government officials who make decisions about where to allocate funding and for taxpayers who provide the public dollars to support public programs.
In terms of the article, what questions were you trying to answer, and how did you go about answering them?
The findings in the article were based on a needs assessment that I helped lead as part of an effort to develop the Family Self-Sufficiency Data Center, which was funded in 2013 by the Office of Planning, Research, and Evaluation, the Administration for Children and Families, and the U.S. Department of Health and Human Services. I worked on this project while I was at Chapin Hall at the University of Chicago. The needs assessment was a first step in understanding how the data center could enhance the ability of state health and human services programs to use administrative data. The goal was to understand the current use of administrative data among officials in state health and human services agencies. We had the opportunity to talk with almost 100 people from public agencies or other users of administrative data, such as researchers or advocacy organizations.
The study included a range of agencies, from those who are leaders in the field to those who are struggling with using data. Could you talk about the decision not to focus solely on the superstar agencies?
Oftentimes you see studies that look at high performers and promising practices. When we wanted to develop the data center, it was important to understand the needs of all types of states and all types of agencies that have a range of experiences working with administrative data. If you’re building a resource to support all different types of agencies, you want to include those agencies that are at the forefront of the field, but you also want to include those that are the middle-of-the-road, average users and those that are doing the bare minimum of what’s required.
What were the challenges states faced in using administrative data?
Some common challenges included limitations in the capacity of agencies to use data and conduct analyses. There was really a wide range of staff and capacity within state agencies. Some had small data or analysis teams. Others might not have had any research or analysis staff. We heard from some state officials that it was a challenge to recruit and keep staff with the right analytic skills because they’re often competing with the private sector.
Another challenge was data quality. Because the data are collected for management purposes, not research or analytic purposes, there can be issues with accuracy. Sometimes a number can be transposed, or the digits are off on a birth date, things like that. The quality of the data affects the ability to analyze the data. Having a commitment to focus on good data quality supports good analysis of that data on the back end. Oftentimes, data quality is unfortunately an afterthought, and so states are working with data that may not be as reliable as they would like when doing an analysis.
Two other challenges the study mentioned are gaining access to data and being able to analyze the data once you get them. Are either of these challenges improving?
In some ways, yes. In some ways, it’s still a challenge. Getting access to data really depends on the type of data as well as what federal or state laws and regulations might be restricting the uses for which those data can be shared. There are still opportunities, but those protections make it hard to link data that contain sensitive information.
At the same time, there’s increasing interest in how individuals touch different programs across systems and agencies, which requires linking or integrating data from across different programs and agencies. For example, some federal investments have supported bringing data together in the education and employment areas because understanding those trajectories over time is useful.
To some extent, I would say the analysis is getting easier. There are more powerful and free analytic tools available. More people might be trained and familiar with how to use them, but you still have to make sure that you are using the data properly by understanding the data, understanding the limitations of the data, and understanding why the data are collected.
This study was conducted in your pre-Mathematica days. What have you continued to do in this area since coming to Mathematica?
I recently worked with a foundation, two states (Florida and Tennessee), and their agency partners to take a look at high levels of service utilization among kids in foster care. Many of the health services foster kids use are funded through Medicaid and are going to be captured in a Medicaid data set, not in a child welfare data set. If you want to look holistically at the service needs of kids in foster care, you need to bring together multiple data sets. For the project, we were able to access detailed data from the states’ child welfare and Medicaid agencies to look at patterns of service use and identify new measures of what it means to be a high-level service user.
At Mathematica, we’re working with state and local child welfare agencies to make better use of administrative data. We help agencies assess their data quality and identify ways to improve it. We’re now looking at ways child welfare agencies can apply some advanced analytics to their administrative data to inform their policy and practice decisions.
Looking ahead, are there any new efforts either at the state or federal level to improve the quality of data and agencies’ ability to analyze those data?
There’s a new federal rule and funding opportunity for child welfare agencies to improve their data systems through a Comprehensive Child Welfare Information System. The rule promotes integrating data, linking data, and leveraging data from outside of child welfare to inform decisions and to understand the needs of kids and families more holistically. There’s also a nice focus on assessing data quality and developing a plan to improve data quality. The rule provides an opportunity for states to take a hard look at the data they’re collecting for management purposes. If they want to use the data for analytic purposes and decision making, they’ll need to ask: Is the quality there? If not, what is it going to take to improve that quality?
Want to hear more episodes of On the Evidence? Visit our podcast landing page or subscribe for future episodes on Apple Podcasts or SoundCloud.
