AcademyHealth’s Aaron Carroll Discusses Impact of Generative AI, Climate Change on Health Services Research

[AARON CARROLL]

I think we need to lean heavily into science communication. I think health services research and health policy research haven’t done a great job of advocating for themselves or even explaining what they are. Too few people know what they are or why they're important or why we might need more about it. I also think that, as a field, we focus too often on problems and not solutions. We are very good at publishing data and evidence on how terrible the health care system is. We do it again and again and again and again and again. We don't spend nearly as much time talking about what we need to do about it, or what might make it better, or, you know, focusing on the need for this research, because it's going to make the world a better place, which is probably part of the communications problem.

[J.B. WOGAN]

I’m J.B. Wogan from Mathematica and welcome back to On the Evidence.

Our guest for this episode is Dr. Aaron Carroll, a pediatrician, health researcher, and science communicator who recently assumed the post of president and chief executive officer at AcademyHealth, the leading national organization for convening and sharing information across health services researchers, policymakers, and health care practitioners.

I spoke with Dr. Carroll ahead of the organization’s Health Datapalooza conference in mid-September. This year, the event is focused on a topic near and dear to our hearts at Mathematica: data-driven solutions that are addressing critical public health challenges.

During the interview, Dr. Carroll shares his insights about effective science communications based on his experience as a published researcher, blogger, podcaster, YouTuber, book author, and regular contributor to the New York Times. We also talk about climate change, public health data modernization, artificial intelligence, electronic health records, and social determinants of health.

I hope you enjoy the conversation. If you’re new to the show, please considering subscribing. More information on how to subscribe on your podcasting app of choice is available at mathematica.org/ontheevidence.

[J.B. WOGAN]

I want to focus specifically, initially on just the topic of science communications. And so how did you first become interested in science communications and is there an origin story? Like, was there a moment that convinced you, okay, this is important, this is something I need to focus on for the remainder of my career?

[AARON CARROLL]

There is, and it sort of has two parts. One was that was more practical, and was the sense that I had this epiphany in the late 2000s that, you know, my job was to a large extent being a writer and I wasn't very good at it. I needed to write grants. I had to write papers. I had to write emails. I had to write talks. I had to write white papers. And I just didn't have the skill set. English had always been sort of my worst subject in high school. And, even in college, I specifically looked for classes that were tests, not papers, because I didn't want to write. I had read a couple books at the time where you're making the very common-sense argument that you get good at something by practicing. You want to be a good basketball player, you practice basketball. You want to be a great piano player, you practice basketball. Practice the piano a lot. And if you want to be a good writer, you got to practice writing.

So I decided I needed to find a way to force myself to write. And I was like, you know what? I'll start a blog. I'll write 800 words a day until I get good at writing 800 words. And I really did commit to that for like a solid two years. And then of course it continued.

The other side was that this was also about the same time that the Affordable Care Act and health care reform arguments were taking off. And I was really moved and struck by the fact that on so many issues about policy, it seems as if people who disagreed would eventually go, Ah, you know what? Well, you know, we'll just have to agree to disagree and move on. We just don't know. There really is data, and evidence can answer a lot of these questions. They are answerable. They're not unanswerable questions. Given that I was a health services researcher, and I had an interest in health policy in this respect, like I knew a lot of the literature and data and evidence. So I thought as I'm starting a blog, what am I going to write about?

Well, I know what the data and evidence are on cost sharing. I know what the data and evidence are on, you know, if we expand with Medicaid versus private insurance, I know about other countries, health care systems and what they might look like and, what the pros and cons are. And so that all sort of brought me to, Hey, let's, know, this is a good way for me to develop the skills that I need to and also bring important information to light that help, could help at least bring data and evidence to the health policy decisions around the Affordable Care Act.

And from then it grew in a million different directions, but that that's the origin story.

[J.B. WOGAN]

Okay. And, do you have a sense of, I mean, you say you were not that good at writing. Do you think that it, that you were average for someone in, in, you know, a pediatrician, a health services researcher, or do you think, this may be impossible to know how you were relative to your peers? But, I guess I'm curious if you have any hacks or any principles that guide the way you approach communications now that you think might be helpful for the community that is a part of AcademyHealth and the kind of people who work at Mathematica.

[AARON CARROLL]

Hmm.

Yeah, I mean, I think the thing that it's hard for people to grasp is like many years I was grinding it out. I mean, you know, at the beginning, the blog was read by the tens of people. And most of them probably got there by mistake. And I had like, we had an open, I did, and then we did, had an open comment section.

So people would tell us, you know, what worked or did not work. And I still, I have gone back many times and looked at earlier blog posts and I read them and I'm like horrified. And now I edit them. It was a learned skill over time. I read lots of books. I was lucky to have editors at a variety of different stages, who helped show me what was wrong with my writing and how to improve it. You know, that would be whenever I wrote for a bunch of mainstream outlets. But a lot of it is, it was the practice. It's, you know, people say, how do you get to do what you're doing now? And I'm like, a million words in a blog, and you will eventually get better at doing that. And I'm not making up a number. I am sure that I've got more than a million words written at the Incidental Economist. That doesn't include everything I've written for other outlets or, or everything else that I've done.

But, you know, I also would say like, I learned a very specific skill. I think I'm pretty good these days at writing 800 to 1,200 words on a, you know, health policy related topic. I have no skills with a novel. Like, I wouldn't even know where to begin. I don't even know that I would be great at long-form, you know, magazine writing or, or even long-form reporting. Really what I have developed most of my skill for is like, you know, columns, and that took a long time and it was just that.

So the short advice is, if you want to get good at it, you got to practice. And I run into too many people who want to do one-offs or, you know, they want to, Oh, I want to write a piece and have it published here or there. And I'm like, you can try. But you probably won't be great at it the first time and it'll take time.

And, you know, if you're good and you get help with editing and everything else, maybe you'll bring it up to speed. But I didn't start where you see me now. It took years and years of really grinding practice to get better at this.

[J.B. WOGAN]

Okay. And then the other piece of advice I'm hearing implicit in that is being open to feedback, soliciting feedback and incorporating feedback. You know, I know writers who are not always so good about listening to what their editors say. You have to be an attentive listener and open to revising.

[AARON CARROLL]

I think that's such a great point. I think in some ways I benefited from coming up through journal publishing where you really have no choice. You get, you know, two, maybe three reviews and you have to do everything they say, or the journal is likely not going to publish it. And so, you very quickly develop a thick skin.

And if you want to be good in academia, you learn to take the constructive criticism, even if you don't agree with it or figure out ways to at least take most of it and bring it in. I've never been wedded to my words. I take no offense to people editing my writing. In fact, I almost always just accept all when things come back with critiques.

And that probably made working with me a little easier to my more mainstream media editors. I mean, I don't think the New York Times would be able to. I've worked with multiple editors who are like, you are easy to work with because they give me constructive criticism, editing suggestions, and I take them all because, who wouldn't?

Why? I want to learn. And so that's one area where I think it helped to come into more mainstream science communication already with an attitude of, it's perfectly acceptable for others to tell me what changes to be made and I'm going to take them and go.

[J.B. WOGAN]

I found that one of the, when I was in my previous job, I was a staff writer for a magazine, and I think one of the things that was hard for me to figure out, early on, was striking that balance of knowing when to just say yes to edits and where are the important parts to push back, where are those red lines, where are those boundaries? And I think I, I learned actually part of that was, each editor has their own preferences. You sort of have to develop a dialogue with each editor, but that, I think, that is a kind of its own skill, too.

[AARON CARROLL]

Yeah. I mean, even in just the last one I wrote, I mean, there are times I've pushed back, but they're more often than not, they're about content and meaning as opposed to words or structure. So, you know, this one was about, I think the fact that, you know, it had to do with physicians or pediatricians being forced to accept Medicaid.

And they took out a chunk that said Medicaid pays far less than Medicare. This is a big problem. And like, that's an important thing people need to know. It's not about, did you change my wording or this, I feel like we lost an important point. And they take it and you move on. Because again, I think it is building up that trust. And so I don't fight about wordsmithing, or even, you know, cuts that often need to be made because things are long more often than not, they're good. But, but I also know when something's been taken out or moved it, that shouldn't. I've also, I think I've also been pretty lucky to work with great editors where they make changes, they're like, did I screw anything up here? Or does this still make sense? And, you know, 99 times out of a hundred, they're right. But you know, once in a while you gotta be on it.

[J.B. WOGAN]

So the only, the other, only the thing I thought about asking, you know, sometimes we struggle with figuring out how to be vocal about what the evidence says without becoming advocates

[AARON CARROLL]

Yep.

[J.B. WOGAN]

on an issue. In your experience, have you found any good guidelines there about how to be vocal without being an advocate? How do you avoid activism?

[AARON CARROLL]

I think that, you know, the biggest problem we have. First of all, I think there's a real problem in that too often we want to win as opposed to solve a problem. and winning means that I have to get something, and you have to lose something. Trying to solve problem is maybe we both get some. And so when I am asked questions about say an issue, I can build, well, there's a number of ways to tackle this and let me discuss the pros and cons of each. And then if you'd like to, I'll tell you what I would choose based upon those criteria, there we go. And as opposed to, oh, no, this is the solution, let me tell you why. Because if you do that grounding, then it may be that they will choose something else, which still will have benefits and harms. But too often it's just like, okay, there's only two opposing viewpoints. Someone has to get it, someone has to have it. Whenever we talk about health, true health care reform in the United States, the only, it's baffling to me, it feels like the only two things we can do is status quo or single payer Canada style. Like there are only two options. Like single payer systems like Canada are rare in the world. Why don't we talk about, all of the different options, their benefits, and then you can choose. They're all fine. I don't care which one you pick.

You know, pick Switzerland, pick Singapore, you know, anything would be better than this mishmash we've got right here. Let's, you know, choose amongst the options. But I think instead what happens is that too often we make judgments about what we think is the best choice. And then we advocate for that choice as opposed to, let's bring all the data and evidence in, let's, you know, position it, let's describe it, let's explain it.

I'm happy to answer any questions. I'm happy to tell you what I think and why, but, you know, and also know that there are other decisions and compromise is often fine and that, you know, it isn't, there's very, very, very rarely a choice, but that's how it gets presented. One of the hardest parts of leadership in general, and this includes health policy and everything else is often you have to make difficult decisions with imperfect data where no decision is obviously correct. But we don't act like that. We act as if, I know the correct answer and I should, I need to convince you of it as opposed to saying like these are all imperfect. We should, you know, choose the one that we think gets the most benefit for the least amount of harm.

[J.B. WOGAN]

Okay, that's a very useful tip. Yeah. All right. Glad I asked that question. All right, so I want to return to science communications in a bit, but I was, I wanted to hear a little bit about your new role at AcademyHealth and why you made the switch, what was, almost like, what was their pitch, like why switch jobs and what was it about being a CEO at AcademyHealth that you found exciting and energizing?

[AARON CARROLL]

So, I think a lot of it was timing, as well as what the job is. I've been very happy in academia for a long, long time. I'd been at IU for 21 years before I moved. Although I will say in the last few years, it was much more of a service and leadership role, starting with the pandemic. I helped to run IU's pandemic response.

And then that later developed into my becoming chief health officer. And I actually was energized by the new challenges that those positions gave me, learning how to build teams, do broader communication with more people, and sort of have a leadership mandate and I found that I really enjoyed that. At the same time, I've been a member of AcademyHealth for since I've been a fellow, I've known Lisa forever, Lisa Simpson, the previous president.

She's been a mentor. You know, I would pick up the phone and talk to her about career changes. She's one of a, you know, a small list of people that I always wanted to talk to because I really valued her mentorship and her wisdom. And so we had had conversations. I asked her point blank, even at one point, like, do you think that this is a job I should apply for?

And, she thought absolutely, which I appreciate. And then to be honest, I was a little resistant at first because I was really happy in my other job, and doing that. But as I thought more and more about it, there's just a few dream jobs that I've always sort of said, you know, if that ever came open, I'd be really interested in that. And they always felt like they were mega high and out of reach. But I think like, you know, I'd reached a point where I'd done enough where I was like, this one, I feel like I could be good at this. And it really did speak to what I like to consider sort of my, you know, my raison d'etre, like what I really want to do.

All the work I've done as a researcher, as a mentor, as a science communicator, as a leader has been how do we take data and evidence and get it to where it needs to be to make better decisions for health care providers, for policy, for researchers, for the public. And AcademyHealth is all about the production and dissemination of data and evidence to improve health and health care for all. So it was, you know, a perfect fit for what I really love to do. I think a bigger platform and the ability to help move the entire field forward, and if I'm being totally honest as well, it just was a perfect time in my life.

My youngest just graduated from high school. You know, we probably have a bit more freedom, you know, in terms of ability to travel and everything else. And so it all sort of aligned at the right place, right time, right job that this, you know, I'm just grateful it all worked out so well.

[J.B. WOGAN]

And am I correct that you started at AcademyHealth earlier this year?

[AARON CARROLL]

March 18th of this year.

[J.B. WOGAN]

OK, in March, so it may be a little unfair to expect that you would already have all of your plans fleshed out in terms of what you’ll be focusing during your time as president and CEO of AcademyHealth, but I did want to know if there were any focus areas or priorities that we can expect during your tenure at AcademyHealth?

[AARON CARROLL]

I'm always very careful to say that, you know, we are about to enter strategic planning, thinking about where we want to go for the next five years. I really want this to be an opportunity for everyone to contribute, that includes staff, the board members, other partners, and so I don't want to be too prescriptive about where we think we should go.

But, you know, I did interview, and offer some ideas and I've been pretty consistent about them since. So, first and foremost, I think we need to lean heavily into science communication. I think health services research and health policy research haven’t done a great job of advocating for themselves or even explaining what they are. Too few people know what they are or why they're important or why we might need more about it. I also think that, as a field, we focus too often on problems and not solutions. We are very good at publishing data and evidence on how terrible the health care system is. We do it again and again and again and again and again. We don't spend nearly as much time talking about what we need to do about it, or what might make it better, or, you know, focusing on the need for this research, because it's going to make the world a better place, which is probably part of the communications problem. I think we need to, you know, continue to focus on all our important DEI work, but again, in those, in the same vein, lean into the evidence and lean into solutions, not problems.

Again, I think there's so much literature on the disparities exist, not nearly enough on what do we do about it? You know, what can, what changes can we make and how? I have also been very impressed with AcademyHealth's local and state-based efforts. And I want us to continue, if not grow those, because, you know, so much attention and energy gets paid to the federal government and policy changes that can be made at a federal level.

But it's pretty hard these days to get anything done at a federal level, but lots of stuff's happening at a state level or even a city level, a local level.

[J.B. WOGAN]

Yeah.

[AARON CARROLL]

And there's real opportunity to bring data and evidence to the, you know, to those decisions and try to push policy in a better direction. So I want to continue that. And then finally, we have to lean into artificial intelligence and big data sets and everything else. I mean, this is going to be changing in so many different ways for so many different sectors, but we need to make sure that we're using AI in our work to make it better. And we also have to make sure that we bring our methodologies to bear to study AI as it is being made and deployed so that we make sure that the, you know, both the benefits and the harms are quantified and make sure that this outweighs that, because too often when it comes to information technology, even in the past, electronic medical records, computerized decision support, we sometimes just assume it's an unequivocal good when it can do a lot of harm, and actually can make a number of things worse, and we need to study AI as we keep moving forward as well.

[J.B. WOGAN]

Hmm. We had a guest, I think it was last year. We had a guest on our podcast, Tina Rosenberg, who is a co-founder of the Solutions Journalism Network. I wonder if there may be some synergies down the road for the network and you guys to, because, they, one of their pillars is around assessing the effectiveness of solutions.

I could imagine at least the, you know, the members of AcademyHealth being a really critical resource for solutions journalists, and you, you are already a practitioner of solutions journalism. I was reading one of your stories recently about looking at other health care systems in other countries and what we could learn about them and how we could improve our own based on lessons from those countries. I can add that link, I'll link to that story in our show notes.

[AARON CARROLL]

Please, that'd be great.

[J.B. WOGAN]

It sounds like science communications will continue to be a part of what you're doing at AcademyHealth. Will you continue your involvement with the Incidental Economist, Healthcare Triage Podcast. What, what about some of these, you know, pre-existing platforms people know you from?

[AARON CARROLL]

Well, we've been thinking about, how do we sort of bring it all into the fold because it makes total sense to align it. And I really look forward to figuring out the details of that as we move into strategic planning, but, absolutely. I want to continue doing it, but I would love to completely align it with the work that we're doing at AcademyHealth.

That would be both podcasts and video. Incidental Economist, these days. Blogs have sort of gone out of fashion a bit. You know, people have switched to newsletters or doing other things. We still update it, but more my contributions to it more often than not, are to highlight things I'm doing in other areas.

There's not as much done novo for the incidental economist. Austin, on the other hand, has some junior faculty or fellows who still contribute to it. So it still does exist. I don't spend as much time with that. And I'm also hoping that that the position allows me more time than I had in the last few years to do some larger writing.

I just had a guest essay at the New York Times this previous week, and I'd like to continue to work on those kinds of areas as well.

[J.B. WOGAN]

So I know we’ve already talked a little bit about lessons learned in science communications but we haven’t talked too much about blogging and I did want to ask about any insights you have based on your experience with contributing to the Incidental Economist blog are there specific tips you have for our listeners in the blogosphere and any surprising takeaways based on writing for a blog?

[AARON CARROLL]

Well, one is that you have to learn to write simply. I have found that, you know, too much, certainly in medical literature, peer reviewed literature, there is, you know, a strange belief that, that you need to, like, make things as fancy as possible, you know, like almost so that no one else can understand and that doesn't work.

I mean, you need to make things as easy and transparent and you're writing as simple as possible so that anyone can understand it. And it turns out if you do that well, so that anyone can understand it, grant reviewers also understand it better. So I found that it, you know, it improved how I function with my writing across the entire spectrum from all the way to, you know, the very lay end all the way up to, you know, the very professional and, I also, you know, there's a need for those in science, for people in science who can do good communication.

There just aren't that many people who want to do it. And it's a lot of work. Everybody wants to find that magic tweet or that magic TikTok, which, you know, suddenly convinces everyone that they're correct. You know, that's apocryphal. It doesn't exist. But, you know, good science communication means repetition and, you know, patience and building trust and talking about why is, you know, to not just what, but also explaining the why and that there's a huge need for it. We used to, you know, beg people come write for the blog and they would come and they would, again, throw in a piece or two and then sort of disappear.

But I wish more and more people would really lean into it because it's a skill set that's desperately needed. But it is, you know, it's a fair amount of work and I think that, you know, again, it requires that level of practice.

[J.B. WOGAN]

Well, let me ask you, let me ask you on writing simply, I sometimes encounter this when we're advising our researchers on press releases or editing blogs is how to know when you've lost important nuance and you've sacrificed something, some level of precision that's important. I'm thinking about some papers early on in my time here where you're deciding how much going to say, what the null hyp—you know, what, no, no impact findings mean or, statistical significance or there's like, if you had any, if you found any useful guides in terms of, how simple is too simple or, yeah.

[AARON CARROLL]

I don't know if there's ever too simple. I got to be honest. You know, or just, you know, easy to read some, some things are complex, but you just need to break them down as low as you can go. You know, if I was thinking of sort of what tips do I think people, especially those who want to get into science communication need to take away. One is, peer review is not the be all end all, like peer review is the best of all worst, you know, of all the bad options that are available to us. And there's plenty of ways that the literature is still skewed, even if it is peer reviewed. And that's because of publication bias. That's because of, you know, selective outcome choosing. That's because of, you know, word choices that people make in their papers. And so you can, just because something is there, you know, peer review doesn't mean it's perfect. We have to be able to critique research no matter what.

Another thing is that we got to stop focusing on every new study as if it exists in a vacuum and it clears the field from what's come before. They're all a part of our fund of knowledge. And when there's a huge fund of knowledge already, a new tiny study is not going to be able to move it that much. But we still act as if, you know, if there's this enormous body of evidence that vaccines don't cause autism, and there's a tiny case control study, which has a P of less than 0.05, the news will cover it as if it's like, Oh my God, we just disproved all of this. And it's like, no, no, no. Sometimes you're going to get spurious results. We’ve got to take it in context. The thing is, we really do need to make a better effort to stick to absolute differences as opposed to relative differences.

You know, scary news stories all the time are this will raise something by 50 percent or double it, whatever it is. But really often you're moving from 0.0001 to 0.0002, which doesn't matter. And so, you know, talking about the absolute differences more often than not matters. I'd say that you can't cherry pick a benefit or a harm. You need to talk about both together, but too often, you know, we'll only talk about the bad side or only talk about the good side because we're pushing an agenda and that leaves people confused with a lot of whiplash. And then the last is that, you know, you can't cherry pick studies and people do this all the time.

You know, they find the one study that supports them and they, they push it, push it, push it, push it. You gotta look at all of them. And, you know, one of my favorite studies with this was a, it was a nutrition review in like 2012. I think the researchers took the Joy of Cooking, picked 50 ingredients at random, from it, and then went to the literature and studied, did it prevent or cure cancer, and they published a wonderful diagram where, you know, it, you know, it was right down the middle was a line where it's like it would make no difference. And to one side was a study that said it protected against cancer and how far out was how much, and the other, the other way was it caused cancer. And when you look at it, almost every single ingredient had tons of studies, which both argued to prevent it and cause cancer, which would argue that like, nothing like just

[J.B. WOGAN]

Right.

[AARON CARROLL]

But instead what people do is they cherry pick one or the other, and then, you know, they make all these dietary choices and it's just madness.

[J.B. WOGAN]

Yeah. I feel like I've seen, in my lifetime, probably a dozen or more stories about red wine and whether that's good or bad for my heart health. I want to pivot to a different topic, climate change, which is something that is of growing importance here in our organization at Mathematica. And we have this thing called ClimaWATCH, which is looking at heat and health and looking at health care claims data and sort of overlaying that with heat data.

But I think that's part of sort of a growing understanding more broadly about the connections between climate change and human health. And so I was curious what that kind of growing understanding means for AcademyHealth. Like, what do you see as AcademyHealth's role in understanding those connections, and to go back to a previous theme from one of your answers, providing solutions?

[AARON CARROLL]

Exactly. So, I mean, you know, we always, I always want to make sure that whenever we're talking about something that it fits with AcademyHealth's mission, but this one is so clearly obvious. I mean, again, it's about bringing data, production, and dissemination of data and evidence to improve health and health care for all. With climate change, you know, our wheelhouse is not, how do we solve climate change or how do we, you know, point out more problems with climate change? It is, but it is, there is, there are significant health effects and they range from the very obvious of it's, you know, changing weather issues, which can bring flooding or droughts or all kinds of changes, which can, you know, lead to famine or to new diseases or to infections or to a host of other problems, obviously heat alone can lead to significant health problems, especially those who are vulnerable. There are issues of mental health, in, you know, always hearing concerns and it, this is like been documented many times that this is, people are very worried about it and feel that they can't control it. And so, that can lead obviously to anxiety and other issues. There are new diseases and things sort of that because of shifting problems, you know, Zika coming up where it didn't exist before. I think we're going to see, you know, other pandemics. And we need to work on fixing this. They're not fixed. Obviously, we want to fix climate change, but on adapting our actions so that we can help combat the new health issues that are arising. We just did a 12-part Healthcare Triage series. I think it was 12 parts, maybe it was 10 parts, on climate change and health, and the last two episodes are about things we can do. And we wove through it. Like we don't want this to be all doom and gloom. We have to lay out the problems, but in each episode, we even tried to talk about potential solutions or things that we could do to mitigate those problems because we can. It is not like we can't do anything about, you know, adapting or trying to improve these things. There are things that we can do. It's also a good example where disparities are going to be an issue again, because this will disproportionately affect those with the least resources and means who also happen to be those who are most affected because of where they happen to live. But there are things we can do about that, too.

And I think growing that evidence base, promoting that evidence base, talking about the pros and cons, the tradeoffs of variety of different solutions is absolutely something that AcademyHealth could get behind.

[J.B. WOGAN]

Earlier this year, KFF Health News had a story about Oregon allowing for Medicaid funds to pay for air conditioners and air purifiers and power banks because they knew that it could help with patients or beneficiaries’ asthma and other issues related to extreme heat. That seems to me like a ripe opportunity for a researcher in Oregon to study the impacts of those, you know, the reimbursement for those services and see, see what happens to like hospitalizations.

[AARON CARROLL]

Yeah. Too often, I think also we, focus only on the cost argument. Like they'll be like, Oh, well, this is actually cost savings for us because if we invest the money here, we'll get ROI because they won't get sick later. I would move past it and be like, sometimes good things cause money. And I would even focus on the outcomes good of like, you will make people healthier and happier by spending money. Why do I need to wait for them to be sick before you will spend money to make them healthier and happier? What if we can also prevent disease and prevent illness? And not only that, make people happier. Besides being healthier. So those kinds of investments in housing, in food security, like we too often, we don't include those in our discussions of health, they're massively important, and other countries are often better about recognizing that.

[J.B. WOGAN]

So what are the gaps in knowledge or gaps in data that would help treat health problems created by, or exacerbated by climate change? Are there kinds of health data sets or just, you know, a type of data that you would like us, that you would like to see more of?

[AARON CARROLL]

I believe the WHO has recently, I mean, like maybe the last year released some enormous data sets, which actually correlate health outcomes with climate. And I know this partially because I've seen more prospective papers coming in on this, than I have never seen before, and they're all using that data set.

So it's like, it's clear that it exists. Now I will say, a lot of the research is happening in other countries. The data set is available to all, but this is not something I've seen yet from a ton of American scientists, but, we're not good at this. Part of that is like, we're not great in this country about collecting data. You know, everything is so siloed, you know, even electronic medical records are just kept, you know, within the institution. It's like, not even that a patient can easily get them from one place to the other, health information exchanges try to fix that. But they're still, if anything, state based and there's no collective way for like most of that to be sort of brought up nationally.

And we don't have the big, broad national health databases that would really help link those kinds of outcomes to, you know, what's occurring with weather, or climate. It'd be great if we did, but we're just not great. And so, yeah, again, promoting the production of evidence as well as the dissemination, I think, you know, better data collection, figuring out how to link up data sets into more useful ways, to tackle real world problems is absolutely a necessity.

[J.B. WOGAN]

So we're talking about health data and we are speaking today ahead of the 2024 edition of Health Datapalooza. And I understand that public health data modernization is going to be a central focus of this year's event. So I want to ask why that focus this year and what else can you share about the upcoming Health Datapalooza?

[AARON CARROLL]

So that is, that is our next sort of big, you know, meeting that we put on each year. Health Datapalooza usually is in the early fall. And it's a fantastic mix of, industry and government and, you know, public sector, as well as, you know, academics, you know, all sort of getting together to talk about how do we use data in better and novel ways you know, to make things better this year?

For the first year, we're collaborating with the CDC to look at data driven collaborations between health systems and public health, you know, which can lead to better informed policymaking. We can find new ways for those sectors to work together to improve communities. So, really focusing on public health, which I think that the pandemic and other things in the past years have shown us is definitely necessary. We're really exciting—we're excited about it. We're putting together our speakers as we speak. The website, however, is open this year. The meeting is September 16th to 17th in Washington, D.C. There'll be constant updates to the website announcing speakers as they fill in, but it's already time to register. I think it's going to be a phenomenal meeting.

[J.B. WOGAN]

So I imagine one thing that also might be new this year, while there are certain themes that I imagine are sort of consistent across every Health Datapalooza, you know, the importance of data, probably data science has been a component in the past, but generative AI and ChatGPT, you know, we're witnessing some important leaps forward in generative AI, and I'm seeing, you know, lots of different conversations about what that means for, you know, all aspects of life, you know, what it means for PR, what it means for the publishing industry, what it means for online search engines—can you tell I work in communications? I've heard some of some applications for health care, like ambient listening devices that transcribe doctor's visits. But what do you think it means for health services and for policy research?

[AARON CARROLL]

Well, it's going to have a major impact, as it does everywhere else. I think in health care right now, like all the, you know, all the research that I keep seeing is, is very much of a flavor. It's, you know, we're taking ChatGPT and can it answer questions as well as a doctor? Can it take a test and do well?

And it's like, that's very interesting, but that's just scratching the surface. People have been trying to use, you know, machine learning to help, you know, improve diagnosis, improve prognosis, improve treatment plans for years and given how quickly things are developing, I think there's going to be much more of that in the future. There'll be ways for AI to help with radiology and, you know, look at scans, to, you know, investigate all different kinds of factors that we might not otherwise be able to see to determine who might be at risk for a variety of issues, and then and then how we might get ahead of it. And of course, there's some of the things you brought up before.

I mean, I can imagine qualitative research methodology is going to change immensely because if AI starts helping to analyze qualitative data, that's going to that's going to be a game changer in that area. But even for quantitative data, I mean, pharmaceutical companies are already using it to try to like, you know, you know, look at tons and tons of potential drugs and what might work, in ways that it would take us as human beings much longer to do. And there's things I can't even think of. But it's important for us always, like I said, I think earlier, to worry about the potential downsides to this, as well as the upsides to this, what blind spots might the AI have? What biases might it bring? What mistakes might it make that we won't pick up? I mean, we see already that it can hallucinate. So we'll need to have checks in place to decide, you know, what is good enough, in order to sort of release it to the domain of patient care. But there's just a ton of work that's probably really already happening now that we're just only going to start to hear about in the next few months to years.

And to pretend that we can put the brakes on this or stop it, is ignoring how much potential this has. So I think what we really need to do is make sure we're a part of it and make sure that we are bringing our skills to bear on studying it as well as helping to deploy and use it.

[J.B. WOGAN]

So generative AI definitely has the buzz this year. But our chief information officer, Akira Bell, was at a conference recently, and I noticed that in a lot of her talking points with reporters, she was talking about the untapped value of other more established forms of AI, like supervised machine learning, which you referenced, and automation.

And so I was just curious, are you seeing promising advancements more generally in AI that have implications for health and health research?

[AARON CARROLL]

So I haven't seen anything like published in the literature, which would convince me that like, Oh, this has been licked, you know, that we've solved it. But I hear of things ongoing. I do think a huge amount of attention is being paid to the generative AI, right now. And that's certainly what most people who use AI are talking about, but there are people who have been, you know, knee deep in AI for a long time, and they are much more familiar with the things you described, you know, neural nets or, you know, supervised machine learning or, you know, a number of other domains, which probably are going to be what's the kind of stuff behind the scenes that you might not hear about, but are probably going to be implemented earlier.

You know, I think that that's where you get into how does it do a radiology scan? Can it actually read it? Or how can it help with prognosis or diagnosis? That's not going to be generative so much as it's going to be probably more supervised machine learning, but all of it is important. And I think it's also that one of the things where it's like those applications may have targeted large impacts, but the generative AI is going to have just the broadest impact at the moment. Maybe not as deep of an impact, but broad? Breadth? It's enormous. I mean, so many people are using it for so many different things in so many different ways, different ways that it's, I think very soon it's going to have to start changing.

But, you know, we're talking even internally, we need to get some policies in place, decide ethically where it can be used or, you know, how it is because we don't want to just replace human beings working in an unsupervised manner. That can be very dangerous.

[J.B. WOGAN]

You mentioned electronic health records earlier and the siloing, the, you know, frustratingly, you know, how much they are still siloed. And I did want to ask about that because, you know, we've seen the proliferation of electronic health records or EHRs since the passage of the Affordable Care Act. I was curious to hear what your take is on how are we doing at extracting useful information from those systems, and where do you think we've made progress, and what still needs to improve? Maybe integration or interoperability as part of the, as part of your answer there, but just, yeah, what, how are we doing and where do we need to go?

[AARON CARROLL]

So look, I was a pessimist when, you know, the law was passed in 2008 or 2009 when it was going to be a, we were going to push everyone to get into EHR. At the time, something like 25% of health care providers had what would be considered like a really good or operational EHR, but that worked.

I mean, truly, like the vast, vast, vast majority of health care providers these days have some form of a pretty good electronic health record. What's not great is they still don't talk to each other. The interoperability is not great. I mean, you know, if you're if, I used to work in a hospital system where, like, inpatient was Cerner, outpatient was like Epic.

And it's like you literally couldn't they couldn't, like, talk to each other. I mean, it was crazy. And, you know, still, if your records are here, you want them over there, it can be a huge, difficult move. But more recent policy that's been made is really trying to push in the direction of forcing that kind of interoperability, that there has to be a way to get your records from here to there, and I'm optimistic that those kinds of changes will, you know, come in the future and certainly people have their eye on.

[J.B. WOGAN]

And I want to flag, this is an area near and dear to your heart, right? I was reading this is kind of the focus of your scholarly research from pediatric care.

[AARON CARROLL]

A lot of my work was on computerized decision support, clinical decision support, and how we could, you know, try to bring data and evidence to clinicians so that they would practice more guidelines and evidence-based care. I've also done some work in medical decision making and whatnot. But, you know, my earliest study, when I was a fellow, was I, and this will date me, but I really thought we could get residents to use palm pilots, to do more of their note taking, and this would reduce documentation discrepancies because, you know, we asked them to rewrite the same notes day after day after day and after a while errors get inscribed. And when I did the research, I found that it did reduce some documentation exam.

But it increased others because it actually wound up propagating errors that would just get copied over and over again. And it was a good learning lesson for me that technology can also do harm. And so, you know, I think I was pushing what palm pilots could do a bit too much. If you've ever used one, they weren't that great compared to what we have right now. But we did a lot of work with, clinical decision support. And we found more often than not, it really did make a difference, you know, finding ways to bring to collect data from patients in better ways, you know, at the time with Scantron and then tablets, and then get that information and prioritize it.

So, you know, we might have time in the waiting room to ask, you know, to ask patients 20 questions that we thought were important. And we'd only alert the doctor to the top six. And that way they could they could really prioritize. And, and it felt like we were making great [inaudible]. And I can totally see how I could get involved and help streamline that, decide which questions we need to ask and which, and what information to give to the clinician and really do recognize the potential that that could be there. But we need to study it like we need to, we need to actually make sure that it's doing what we want it to do.

[J.B. WOGAN]

So let me ask about one other topic, which is social determinants of health. It's certainly an area of increasing importance at Mathematica. I imagine that it's on the radar of AcademyHealth as well. Yeah. Steady nodding there.

[AARON CARROLL]

Absolutely. Oh, yeah. I mean, yeah, you know, we like we like to think that, you know, health is determined by either, you know, things in our genetic factor or things in our control or, you know, some lifestyle choices. But so much of what contributes to health are, are things that, that we still can control, but we don't necessarily think about. These can include housing. They can include food security, you know, they can include racism. They can include, issues of, you know, access and ability to, get to where you need to be. It can be transportation, it can be climate. I mean, there are just so many other factors, which truly influence health.

I've been to Singapore twice to learn about the health care system for, on two, on two different trips and, both of them, we spent half a day learning about housing in Singapore because they are if you ask them to talk about their health care system, they will also talk about housing and about how, you know, something like 90 percent of people in Singapore live in public housing, and most of them own their housing, and it's public housing that's subsidized because they recognize that, you know, having a place to live that is stable and affordable and safe is a major component of health. We don't do that. Like we think health is what you get in a hospital or a doctor's office. And we’re willing to spend an ungodly amount of money on that, but we don't nearly invest as well in so much of the public health or preventive sides, which would affect the social determinants of health. Other countries are much better at that.

[J.B. WOGAN]

Yeah. And I've seen a statistic. I'm sure you've come across this too, that something like 80 percent of the, of a person's health is determined by non-clinical factors, which who knows exactly how precise that number is, but that's an astonishing number.

[AARON CARROLL]

The thing is, it's like and this is what I'm like, people that quibble about whether it's 80 or 70 or 60, I'm like, it's a lot like it's a lot. And we are willing to spend an enormous amount on drugs or devices, to treat the clinical aspects and next to nothing, you know, relatively on, on all the social determinants, which likely have a larger factor.

And it's really part of what makes me most angsty about our health care system. It's like we are so willing to spend so much money. If we could just figure out a way to spend it a little bit better or, you know, be willing to thinking about broadening the definition of what constitutes health or spending for health. It's not that we don't have the money, it's just that we keep putting it into the wrong part.

[J.B. WOGAN]

So similar to some of the other questions I had, what do you think is the role of health services research in making progress on, you know, on the social determinants of health, on kind of reaping the rewards of housing stability or food security or other factors that harm people's health?

[AARON CARROLL]

I mean, a lot of it is getting back to sort of first principles about research. It's like what, you know, let's sit down and look at and say, like, what's the problem we're trying to solve? Where are the gaps in our knowledge on how to solve that? What kind of research could we set up to answer that question to provide, you know, data and evidence towards a solution?

And then let's go do those studies. That's not often how science happens these days, or certainly not in a health services and health policy research, where instead, too often it's, we want to do research to support or oppose a certain policy. Or it's, you know, oh, let's figure out what's associated with that issue without ever taking the next step of what do we do about it?

Or do we do we really know if it's causal, or do we know what lever to push to fix it? We need to really think about what would help what would contribute most towards, you know, taking steps to fix whatever issue we are focusing on and then going out and doing that work and then disseminating in our way that people will grasp and understand, that may require trying to figure out new ways of funding, new ways of thinking about research, new ways about disseminating and publishing it probably need to focus more on implementation science as well, because, you know, the research that's done in this very tightly controlled bubble doesn't often work in the real world. And what, how do we take it from there to there, maybe get everybody involved from the beginning? Some of it may mean also doing a better job of organizing of, you know, involving patients and community organizations and lots of others who do the work and can do all of that dissemination and make it happen at a local level.

If they're not brought in early enough, then you can do all of this work over here and it never gets implemented over there. And then why did you do it at all?

[J.B. WOGAN]

The points you were making earlier about interoperability. It seems like they'd be even more important and also harder to crack if you're trying to combine the, you know, human services data sets with the health care data sets.

[AARON CARROLL]

Yeah. And I mean, most of the health care data sets we have, of course, are like hospitalizations and, maybe, maybe some outpatient visits. But, you know, often what we need or like also all of the social determinants and all of those other factors which are out there and not part of EHRs, or, you know, we're what we think of as health care data, but as we discussed, probably have more of an impact than a lot of the stuff we're gathering in the EHRs.

[J.B. WOGAN]

So, rather than end on, down note about the challenges of data, you know, what's giving you life right now? Is there something that you're really excited about in AcademyHealth’s work or just see you working on in general?

[AARON CARROLL]

I think, you know, the fact that I have, you know, a whole staff and an organization that really wants to work on making this better and finding their solutions totally energizes me. We just had, you know, our Annual Research Meeting and we had, you know, we far exceeded expectations in terms of registration. There's a whole workforce out there, which I think is hungry for, you know, renewed optimism.

I think that there's plenty of new people in industry and, you know, data companies that that might not be considered health services researchers, but they've got tons of data which can be used for health, and are doing analysis to make decisions that all affect health. We should bring them into the fold as well. I think there's lots of new areas.

You know, I think AI does have real potential to make a lot of things easier and better, and, you know, I I'm energized by the new position and the chance to really help try to steer the field in a new and more, productive manner. And that gives me a ton of energy. I'm super excited about it.

[J.B. WOGAN]

All right. Excellent. Well, I've exhausted my battery of questions. Aaron, thank you so much for talking with me today.

[AARON CARROLL]

Happy to. Thanks for having me.

[J.B. WOGAN]

Thanks again to my guest, Dr. Aaron Carroll of AcademyHealth. This episode was produced by my Mathematica colleague, the inimitable Rick Stoddard. As always, thank you for listening to On the Evidence, the Mathematica podcast. If you liked this episode, please consider leaving us a rating and review wherever you listen to podcasts. To catch future episodes of the show, subscribe at mathematica.org/ontheevidence.