Embedding Data and Innovation Across California State Government

[JEFFERY MARINO]

It's our mission to partner with state departments to improve service delivery for Californians to make more efficient, effective, and equitable services for Californians. And then we do that with human-centered design, technology, and of course data.

[J.B. WOGAN]

I’m J.B. Wogan from Mathematica and welcome back to On the Evidence.

Our guest for this episode is Jeff Marino, the director of California’s Office of Data and Innovation, otherwise known as ODI.

Governor Gavin Newsom created the office in 2022 and appointed Jeff as its director in January of 2024.

 The office’s mission is to partner with state departments to improve service delivery for Californians through data and human-centered design.

Mathematica’s President and Chief Executive Officer, Paul Decker, is back in the host chair for this episode. They talk about how Jeff’s background in literature and data journalism led him to his current role; they also talk about using data to tell the story of government’s impact, the implications of generative AI in government, how the office is upskilling the state workforce, and how the office facilitates data sharing across agencies.

If you’re new the podcast, please take a moment to subscribe. We’re on YouTube, Apple Podcasts, Spotify, and elsewhere. And if you like this episode, consider leaving us a rating and review. It helps others find our show.

With that, I’ll turn it over to Paul and Jim. I hope you enjoy the conversation.

[PAUL DECKER]

Jeffrey, welcome to Mathematica and the "On The Evidence" podcast. It's great to have you here.

[JEFFERY MARINO]

Thank you. It's great to be here.

[PAUL DECKER]

This year, you were confirmed as the Director of the California Office of Data and Innovation. And I want to ask about ODI and your work there. But first, I wanted to get a little bit about your background leading up to your current role. And in looking at some of your background, I saw in your LinkedIn site that you referred to yourself as a data journalist whose expertise lies at the intersection of advanced analytics and storytelling. So could you tell me a little bit about the path by how you got there to make those statements? And what do you mean by data journalist?

[JEFFERY MARINO]

I was always just an avid reader when I was a kid, and I was really into literature and books. And my path late in high school, really, I imagined was going to be as an academic. And so I really wanted to be a college professor. I wanted to be a literature professor. And so as an undergrad, I went and I studied English Lit and had a great time doing it, wanted to go to grad school, ended up going to grad school at UC Santa Cruz. And while I was there, I got introduced to a lot more than just literature.

UCSC is a really collaborative environment. A lot of cross-functional studies, not just literature, but I began to become really interested in economics as well. And it was sort of in the air because I was in grad school around the time of the Great Recession. And so I really started becoming obsessed. This is where I really started to make the leap of being interested not just in literacy, but really getting interested in financial literacy. And it was also touching me personally.

I come from Southern California, the Inland Empire, from a working-class background, and many of my friends and family were losing their homes, and that was the case because of predatory lending practices. And quite frankly, it made me angry and really curious. I wanted to learn everything about credit default swaps and everything else, right? At that time, it sort of also converged with the fact that I was starting to get a little older and thought maybe I might, you know -- it's a long way to becoming a tenured professor at a university and then earning enough to be able to support a family.

So I started looking around at other things and I found a really interesting job at a real estate publishing company that was doing continuing education materials for real estate professionals. And I found a mentor there named Fred Crane. He was a real estate attorney in the '80s and won big cases against the banks during the savings and loan crisis. And this is when I really developed this sort of view of a broken system. At that time, it was really sort of the mortgage lending and real estate market. And then empowering myself with as much information as I could have, knowing that I was also a good communicator in order to help others and help others understand what was at stake, what was at risk, and help them make more informed choices around these big investments in their lives.

And so that led me to work my first job in tech at Redfin because the CEO there, Glenn Kellman, is a great iconoclast. Right? I think he was on "60 Minutes" or something like that and got in a lot of trouble with the real estate industry because he said it's the most screwed-up industry in the United States of America. And I was like, I like this guy. I want to work for him.

And so what I did there, I joined -- it was in the communications department, communications and marketing, but it also was a team of data scientists and analysts. And because Redfin had this huge platform, millions of users every day clicking on homes, searching homes, not only did we have the real estate market data that I was used to analyzing in my previous job, home sales, prices, and everything else, supply and demand, but we also had access to big data. And that big data was all driven through this engine of digital engagement with consumers, with home buyers and home sellers. So we were able to find out who was clicking on what, when, and why, and what those patterns were. And there were brilliant teams at Redfin. It just blew my mind what they were able to do with this.

And so I started to become educated, taught myself SQL, started to learn how to do those kinds of analytics. And that way, I could do it coming from that place of curiosity and a desire to tell a story about what I was analyzing. And really, my job, from a mission-focused perspective, was to empower home buyers and home sellers with data in order to make better decisions.

And so I repeated that process again at a couple of other companies. I consulted in this space as a data journalist. But to answer your question specifically, defining a data journalist, it's really that. It's telling stories through numbers and recognizing the fact that the numbers that we're talking about all represent people in some way. Their behavior, their needs, their blockers. And so being able to analyze that data and then reflect back to a home buyer or to a job seeker, what other strategies they might be able to take to get a higher-paying job or to find a more affordable mortgage. And so I'll wrap up here with saying that through that experience, I made some friends who also had a background in civic tech and invited me to volunteer during the pandemic because the governor was data-rich and insights poor at the time. We still are in the state of California. But at that time, the governor was really looking for insights on how the COVID-19 policies were impacting Californians -- restaurant closures, stay-at-home policies. And so we had a really big data challenge in front of us. And I can share more details about that if you're interested, but that's really how my journey in public service began. And I really -- while I was working for mission-driven companies in tech, it was still about a bottom line at the end of the day. And in public service, it's a much purer mission and objective. And I fell in love with that, and I saw the value in what I was doing, and I could feel the impact. And so I'm hooked.

[PAUL DECKER]

As the CEO of Mathematica, I appreciate your prioritization of the mission because we're in the same business, basically, of using evidence to drive better decision-making and using that process to improve public well-being.

[JEFFERY MARINO]

Yeah.

[PAUL DECKER]

So I really appreciate your perspective on that. I also appreciate your perspective on research or analytics and storytelling. And I think sometimes people take away from that statement that, oh, well, I can tell whatever story I want from the same data. And my attitude is it's a little bit the opposite. To what extent can I let the data lead me to the story?

[JEFFERY MARINO]

Sure.

[PAUL DECKER]

Because the story is still critical. No matter how clear-cut the data are, you still got to tell the story in order to be able to motivate people to build that story into their decision-making and for that work to have influence.

[JEFFERY MARINO]

Yeah, absolutely. And it's really an important thing to keep in mind as it relates to, well, especially in the social and political landscape that we are now, and truth in data and storytelling is more important than ever. And it is true that you can twist and turn numbers to make them be what you want them to be, right? And so there's an ethical component involved as well and a transparency component involved as well that's really important. Where did I get these data? And how did I do this analysis? This has become a really important topic for us in California as we're experimenting with Gen AI and artificial intelligence in general. And sort of the black box algorithms where you can't tell what's happening in there, right? And so that's where it's important that we have the skills in state government to be able to have folks who can explain what's happening in the algorithms. And that's a new frontier for hiring and recruitment in California because traditionally we've not needed that talent.

[PAUL DECKER]

So I mentioned previously that you're now the director of the California Office of Data Innovation. First of all, I love the title of the agency. And you spent several years with the agency before being appointed as director. When was ODI created and what is ODI's mission?

[JEFFERY MARINO]

So ODI was established in 2019 in Governor Newsom's first budget. And the basic concept behind ODI as an independent organization within state government is that it would need to be just that, right? We have established in California state government, this is in all governments really, these subject matter specific silos, right? And they're very effective and useful for what they do.

We have the California Department of Technology. We have CalHR who handles recruitment and hiring. We have DGS that handles procurement and statewide processes. And I think the governor had the wisdom to understand that innovation is not any one of those things but is a combination or a collaboration across all of those areas.

In addition to a partnership with a service delivery organization like the Department of Social Services, because where they may need to innovate -- well, I was going to say maybe, but it's almost always about people first and then what technology those people need and what skills those people need, what training, what tools in order to realize better outcomes for Californians. So the office was created independently in that regard within the Government Operations Agency to be that sort of center of innovation, that center of collaboration and draw those connections, bringing the skills that are needed to address whatever problem it is we're trying to solve.

And so in that, it's our mission to partner with state departments to improve service delivery for Californians to make more efficient, effective, and equitable services for Californians. And then we do that with human-centered design, technology, and of course data. And so within ODI, we also have the Office of the Chief Data Officer, the statewide Chief Data Officer, Jason Lally, and also recognizing the importance of data as that fundamental component of being able to deliver better services because we have to have the right tools in place, the right skills in place to measure our success. We can't improve what we can't measure, which is a huge challenge in government, and it's something that we're driving towards every day, which is really creating -- we want to create a data-driven culture. We want to create a customer service culture. And those two things aren't necessarily natural for government operations.

So that's why we're here. That's what we're here to do.

And when I had the opportunity to join after volunteering, it was just a no-brainer. It was like, an organization like this even exists? Oh my gosh. And then of course at that time, I was so new to public service, I was not aware at all of the civic tech movement. I had not heard the words Code for America before. And so I got introduced to a whole new community. And it's just been, I mean, it's the privilege of my lifetime so far to be in this position of leadership at this really incredible organization.

[PAUL DECKER]

The ODI is really supporting agencies in pursuing their mission, basically. So what you're doing is, I would say, building excellence and customer experience, but doing it by empowering other California agencies to deliver their services more effectively.

[JEFFERY MARINO]

That's right. I would say, or I do say, that we're a G2G organization in that way. We're government to government. Another story that I tell sometimes that I think my team is probably sick of hearing, which is that I'm from Los Angeles and my wife and I were stuck on the 405, as we often were, in gridlock traffic one day. And I was complaining endlessly about the traffic and she looked over at me because she's much smarter and wiser than I am. And she said, we are the traffic. And that really stuck with me and it's something that I really embrace in this role now, that we are the government. And that's something that's interesting about our department. We love our partners in the vendor community, in the civic tech community. We couldn't do what we do without their support and without their partnership. And then hopefully have some influence on making some bigger structural changes that need to happen.

[PAUL DECKER]

Yeah. Well, this is a process that's going on all around.

[JEFFERY MARINO]

Yeah.

[PAUL DECKER]

All around the nation, all around the world right now. And you're right in the middle of it.

You and I have talked about Jennifer Pahlka's book, "Recoding America". And in that book, Jennifer discusses the efforts that took place in California aimed at modernizing and improving the state's digital infrastructure to achieve the goals that we've been talking about, to bolster delivery of government services and create better outcomes for residents. One of the themes of the book that I found intriguing, because I operate in both of these worlds, is that in government, there's a huge focus of attention on policymaking, but too little attention on the nuts and bolts of implementation. Oftentimes, the implementation is treated as an afterthought by leaders who get the policy set, hand off implementation to lower-level leaders, and they stop thinking about it. So I just wondered kind of how you see that from your perch, and how is ODI in a position to help solve that?

[JEFFERY MARINO]

Yeah. Yeah, that's a great book. Everybody in this office has it on their desk, and she's absolutely right in that observation. A couple of thoughts. I think what I would lead with, and I know she saw this when she was working with the state at that time, there are opportunities -- there's a ton of opportunity to improve service delivery without touching a single policy, without speaking to the legislature, without working with high-level stakeholders. A lot of the big structural things that are in place, we absolutely need that. But in terms of bringing a product mindset to an existing digital service and just having the right team and the right skills and the right support from that departmental leadership in place, not just pressure from higher-ups, but genuine support, there's a lot that can be done in terms of creating a better user experience. We just need those skills, and so we're here to provide that for sure, and we're doing that now. I think that it's certainly true that, and it's a great disconnect, and this goes back to the collaborative mission of ODI. One of our key stakeholders is the legislature, and we're working with the legislature all of the time right now on our efforts around Gen AI. And when I meet with folks in the legislature, the first thing I say is, you're closest to our end users, and I want to know what you're learning. Right? And so when you hear from -- and I think that oftentimes is the policy disconnect, is that lawmakers will hear directly from their constituents about a problem that they're experiencing and then put in place a potential solution, and so, having those conversations early in that process about what's possible is really, really important, and then also having a feedback loop with the legislature to say, you know, we're working on this now, and we're running into these roadblocks, because this is the other thing that I've really gotten excited about once I've joined the leadership at ODI, which is that we make all of these rules, and we can change all of these rules. Right?

And it's not just legislative policy. It's not just policy big P. It's also policy small p, because we're creating administrative policies all the time that guide how we do project approval and oversight, that guide how we do procurement, and we get in our own way all the time. And that's always done with the best of intentions for security, privacy, protection, good stewardship of the tax dollar. But we have to find a way to -- and this is where I really preach the gospel of human-centered design and an agile approach as a risk management tool, because a lot of these policies, the accretion, I think, as Jen says, of these policies over so many years, often occur because of a low risk tolerance.

But if we can do things incrementally and deliver value early and often, then we're managing that risk, and we're managing it in a way where value gets delivered. It might not be the best right away, but we also have the opportunity then to continuously improve. [JW1] 

[PAUL DECKER]

So your discussion brings up a question about, well, how do you find out what's happening on the front lines? What's your approach to gathering the information you need to figure out which pressure points to address, how to go about addressing those pressure points?

[JEFFERY MARINO]

We have a lot of different ways, a lot of different business development, I guess, I would say, techniques. We do solicitations, so we'll do a call, right, for problem statements. And I think what's interesting, we did that our first time with our Data and Innovation Fund, which was something that was established at ODI with the recognition that you can't really be -- you can. It's very difficult to be agile and innovative on an annual budget cycle. And so within ODI, we had a relatively small bucket of money that we were able to use to fund on, you know, off-cycle.

And so we got a lot of requests. We got requests for dashboards, and we got a lot of requests for apps. And so it was really interesting how that opened the conversation, and they were really uncomfortable conversations at first.

And this was, I'm thinking several years ago, where we were working with the Department of Toxic Substance Control, and they wanted to increase their public comment. But they came to us and they said, and we know that we will do that with an app. And we said, well, why will it be an app? They said, because we need to reach underserved communities. These communities are primarily mobile first. And so we think that this is how we're going to reach them best. And then we were like, okay, so it's not just about an app. It's about being able to reach a particular community that has a particular set of constraints, and we want to get to them through mobile.

And then we went through this problem definition exercise, right, to determine whether or not that solution would actually solve the particular problem that they were after. We ended up instead of building an app, because that creates a lot of barriers to entry -- you have to download, you have to find the app, you have to be motivated to download it, you have to keep it on your phone, all this kind of stuff. So we experimented with them with digital ad buys and in social media, and it was really, really effective. It was relatively low-cost. And the best thing about it, because previously -- I think I'm going on a tangent, so you should stop me. The best thing about this was previously in order to solicit public comment -- well, this is still the case, but it was all done through the mail. They would send a letter.

[PAUL DECKER]

So low-tech.

[JEFFERY MARINO]

Low-tech, and also a good thing to keep in place in terms of making sure we have, you know, an equitable approach, maybe mail is the best option for some users. So we're not saying take that away necessarily. But a lot of folks don't like to open mail from the government and it goes straight in the trash can. But the worst thing about it is that you will never know. You'll never know if they received it. You'll never know if they opened it. You'll never know if they read it. You'll never know.

And so that's the incredible thing about moving to a digital space as it relates to service delivery, is that not only are we able to reach people more quickly and efficiently oftentimes and meet them where they are, but we're also able to measure it and find out, did they open the email? Did they click through to the webpage that we created so they could put in a public comment? So that's one example of folks that come to us and ask for help.

And then of course, you know, the administration has priorities. And so we're often asked to be on specific statewide initiatives. Right now, Gen AI is the biggest one that we're working on. And I'd be happy to talk more about that, but that's a whole other story.

[PAUL DECKER]

Yeah. Let's talk a little bit about Gen AI. So how is ODI involved in that?

[JEFFERY MARINO]

Sure. So the governor issued an executive order on Gen AI. It's been a year. Actually, it was a year in September, I believe, which is crazy to think of. And there are a number of different deliverables that were requested in the executive order. But the spirit of the order was really to do two things.

Well, primarily the spirit of the executive order was to say, California is going to lead on this. We can't be left behind. This is a transformative technology, and we need to figure out as soon as possible how we can harness its power to improve services and do so safely and securely and ensure that we're not having adverse effects on our communities. And so, you know, we worked on a report on the best use and highest risk of the technology as we saw it at that time. We have subject matter experts at ODI who, in the AI space, data scientists who understand both the technology and the policy, the existing policy around it. So we were able to be partnered with CDT and CalHR, DGS on a lot of these things because another key thing that had to be addressed is procurement. And it's like, if we're going to do this, we need to figure out how to buy it. And it's a totally different model of technology, right? It's not a piece of software per se, right?

And so really, and this is in the spirit of doing this sort of agile iterative development, what we're using as a procurement method with five departments right now, and we're looking at doing it with several more, is the RFI squared, which is a request for innovative ideas. And it's a procurement model where rather than putting together a list of requirements, a huge request for proposal, throwing that over the fence to the vendor community, waiting to get responses, you put out a problem statement and you say, too many people are being injured on our roadways, right? Vulnerable roadway users is one of the ones that we're working on right now with the Department of Transportation. And how might Gen AI help us with this problem? And that's it, right?

And then you get to hear back from the best of the best, especially here in California, on all of the great ideas that they have for potentially solving that problem. And then you choose two or you select a few to build a proof of concept. And that in itself is a contract, but it's a very small contract that's limited in time and scope, and it's not billions of dollars. You know, we're using publicly available data. It's not necessarily using real-time data at this point. We're just proving out a concept.

And the way that ODI is participating in this is that we're really taking this as an opportunity to start to inject a lot of this product mindset and product thinking. Because if you have a challenge between two innovators who are developing a potential solution, you have to understand, how do I evaluate the business value here? How do I evaluate the outcomes that I can get with this tool? And who do I need to talk to in order to determine whether or not it's producing value? I have to do some user research. Right? In the case of the POCs in the state of California, that user research has been primarily with state staff to say, does this work?

We're doing another proof of concept with the California Department of Tax and Fee Administration, and they're doing it with their call center because they have a huge knowledge base at CDTFA and how do we plug their talent into this huge knowledge base and help them get that information at the touch of a button?

First and foremost, once we get something viable, something that kind of sort of works, we put it in their hands and say, how is this working for you? Which is just absolutely a total paradigm shift in thinking about product development and procurement in government. So it's pretty exciting. It's really exciting in how it has become a catalyst for change in many different ways, not just the emerging technology, but in how we think about how we buy technology, how we think about how we include the workforce, how we think about how we need to upskill and train the workforce in order to make the best use of the tools. And so it's something that we're deeply involved in at ODI and really excited about.

[PAUL DECKER]

As someone who's written a lot of multi-hundred page proposals, I appreciate this approach.

[JEFFERY MARINO]

Yeah.

[PAUL DECKER]

And one of the things that really interests me about it is it just threads learning, continuous learning, into the process. You know, it doesn't assume everybody's going to have the best idea in the beginning and will stick with whatever the best ideas in the beginning are and kind of freeze those for the life of a five-year contract. It says, look, let's build the learning process and how we approach this.

[JEFFERY MARINO]

And that includes data, right?

[PAUL DECKER]

Yeah.

[JEFFERY MARINO]

Because what happens when you build a proof of concept, when you do something small, you produce a data point. You produce hopefully multiple data points that help you drive to the next decision, whereas in the older way of doing things, you don't get your first data point in terms of performance or outcomes until, as you say, you're five years into a project. You're past the point of no return. You've already got the money from the legislature and you've got to get it done because success is getting to done, right?

[PAUL DECKER]

Yeah.

[JEFFERY MARINO]

And that's so important and that's very real, and I don't blame anyone for that because if you were entrusted by the legislature vis-a-vis the people of the state of California with a billion dollars, you better finish. You better finish on time, but more than anything else, it better work well, right? And by the time we get to the end of that five-year long process, who knows? That's the other thing about this approach with Gen AI that's such a watershed moment is it's starting to get this recognition of the fact that technology is moving so quickly. If you take five years to complete a project, you're obsolete by the time you're done.

And hopefully without alienating too many of my colleagues, but I think I'll say this in any forum, today's modernization effort many times can be tomorrow's legacy system. And the way that you avoid that is through that continuous improvement model, through delivering value small. And the other thing that I find, there's so many positive things about this. When you deliver the small value, you get to talk about it and take credit for it as well and you get to build more trust.

[PAUL DECKER]

Build more energy, build more trust. Yeah.

[JEFFERY MARINO]

Yeah.

[PAUL DECKER]

That's great. So one of the programs that ODI supports is the CalAcademy, which I think is an interesting program. So could you share with us what the objectives of the program are as well as how it fits into the mission of ODI?

[JEFFERY MARINO]

Totally. I mean, CalAcademy is like, in many ways, the engine of change, right? Because it's our training organization at ODI. But it's also the center for where the collaboration happens because this is where we're connecting with other state staff. This is where we're hearing about their challenges. This is where we're helping to address some of those skill gaps. And we're doing in-person trainings. We're leading workshops. We're partnering with CalHR to put asynchronous trainings onto their learning management system. But it's really about upskilling the state workforce and the skills that they need to do those things I was talking about earlier, especially in terms of developing services that deliver a better customer experience.

And so, I'll give you an example of the strategic way that we're rolling out CalAcademy right now, which is at ODI, we have the statutory ability to create statewide standards. One of the standards that we created that we're really proud of is the statewide plain language standard. And plain language is a thing that government has talked about for a long time. The Obama administration passed the Plain Writing Act. I think that was in 2010. Was he still in office in 2010? The Plain Writing Act. And then there's even a California state statute around writing in plain language.

The problem with that is that in these statutes, there's no benchmark. There's nothing to measure it by. And you can imagine it would be very difficult to get into statute that the reading level must be at an eighth grade reading level, because you would have lots of people saying that's too high, lots of people saying that's too low, and you can't be as flexible when it's in statute.

This is the beauty of, I think, releasing a standard. And so, we release a standard that all public-facing government documents need to be at an eighth grade reading level or below. And we rolled out that standard. And this is a principle of ours at ODI, a principle of mine, is that we're never going to release guidelines without guidance or standards without enablement and training. Right?

And so, with this standard comes our coursework on writing in plain language. And it's just been wildly successful. And people are so into it because it's such low-hanging fruit, first of all, right? It's not a big technical leap. It doesn't take a ton of -- I mean, we use technology in order to measure the reading level, but it doesn't take a ton of technology. And it's pretty cheap to do. You just need the skills and the will.

And so, we've trained a couple thousand state staff in plain language and I'm hearing it across the administration, right? I hear cabinet secretaries saying, and that needs to be in plain language. And we're like, yes. So that's really how we see CalAcademy working, is where ODI can, in practice, right -- and before we created that standard, by the way, we worked with departments on implementing a redesign of some of their documentation in plain language. We measured whether or not it was successful. And people were more responsive to a document. People were more likely to complete a form, right, all of these things, because it was in plain language. And so, we can establish that best practice as it relates to good human-centered design, set a standard around that, and then create that training and capacity building within CalAcademy so it can be scaled out statewide. That is the sort of strategy for the training arm.

[PAUL DECKER]

In some of your material, you had a statement that you're bringing a culture of data and innovation to the -- or a culture of innovation to the state government, California. And I think, you know, this is kind of a tangible example of that. That, you know, you don't want to be the only person driving innovation in the government. You want to get other people sprinkled across the other agencies, the capability of applying some of those same innovative principles.

[JEFFERY MARINO]

Absolutely. When, you know, we are 65 headcount when we're at full capacity at ODI, we're a small team. And while, don't get me wrong, I would love to and have plans to grow the organization, you couldn't grow it big enough to be the innovation for the state of California, right? Like, we have to partner and co-create and co-develop these new ways of doing things together in order for this to be successful. And so, really that's why the training division was put into ODI from the very beginning in the recognition of the fact that we'll never be able to just be the service delivery arm. We have to change the culture. And to do that, we have to bring people along.

[PAUL DECKER]

Mathematica, our organization, spends a lot of time working with data to address issues, to help people make better decisions, and generally in the social policy realm. A key challenge that we face, at least doing this work in the U.S., is the siloing of data. And it creates a barrier to flexibility in addressing some of the issues that we face that are reflective of the way in which they actually arise.

So a good example of that is the social determinants of health. We know things like housing, food security, that does have an impact on people's health. Yet, to be able to study that, we have to bring health data together with data on social programs to relate to housing or food security. And that's not particularly easy in the U.S. And sometimes, it's almost impossible.

So I know part of what ODI has been doing is trying to facilitate greater data sharing across state agencies. And based on your experience with that so far, I'm just curious about what you've learned about what's needed to support and promote data sharing that can be used in analysis and generating insights.

[JEFFERY MARINO]

Yeah. Yeah, data silos are very real and something that I was very spoiled with in working in larger private enterprises. I think they're still, the larger you grow as an enterprise, you probably experience similar issues.

But where I was previously, it's like if we have the data, it's at your fingertips, right? Go wild. And I've seen it happen as it relates to us trying to get things done and it not happening or not happening for a very long time because we couldn't unlock the data.

And so, a couple of things that we've done, and this was spearheaded by the Chief Data Officer prior to them joining ODI because this happened. The Chief Data Officer became a part of ODI in 2022, a few years after we were established. But we are now the stewards of it and are carrying it forward, the Interdepartmental Data Exchange Agreement. And so, what we did there was try to create a shared boilerplate language that took a long time to get statewide buy-in signatories of at different departments that say, we back this up, this is a workable agreement. And then the only thing that changes within that template is a business use case. And so, you have to establish your business, why you need the data. But everything else is already done. The lawyers have already agreed to it and your department leadership has already agreed to it.

And we've had a lot of success in being able to facilitate data sharing in that way much more quickly than you would otherwise in a government where we have 150-plus different state entities and 150-plus different ways of doing things. And so, that's been successful. I think we also have, to your point about social determinants of public health, we have really, really great partners at Public Health. And their Chief Data Officer over there, John Ohanian, is a great partner of ours. And we learned so much throughout the pandemic of how critical it was to share data and share it quickly. And they're doing really great things over there as well. And John says all the time, out of some frustration, but also kind of joking, that the other P in HIPAA stands for portability. And the point of these regulations is sharing, right? It's sharing while keeping it private.

And so, I think the third thing that gets in the way of that is a real infrastructure thing. And we're trying to help with that as well, which is, it may be one thing from a policy perspective for us to share our data, but it's a completely other thing that there is no standard approach to data governance. Everything is in a different format, on a different machine, in a different system. And so, even the mechanics of being able to share your data are really challenging. And that's a key component of our data strategy and something that our Chief Data Officer is working on.

And again, there's a tooling issue there, but mostly it's really about the governance, which is a people issue. And it's about keeping your data in shape in such a way that when it is shared, it's of value and it's useful. And really, with the value that the data has, we should be investing as a state. We should be investing heavily in training folks in better data governance practices, in getting the right data stack, the modern data tools in their shops, because any private entity would just kill for the amount of data that California has at our fingertips. Not to mention the public imperative of the fact that this is the public's data. And so, we should be very focused on making it as open and as accessible as possible.

[PAUL DECKER]

I can imagine getting agreement, commitment across all those agencies, that wasn't easy. And one, is that a true statement? And two, is there any advice to people in similar position to yours and other states about how you make that happen?

[JEFFERY MARINO]

Well, having a statewide Chief Data Officer is a really, really valuable role. And that's really where I would start, because data is not really IT, right? Data is a business asset in many ways.

Do we use technical solutions to store it and move it? Absolutely. But as I'm referring to the governance of the data and the ownership of the data, the use of the data is really -- it's not an IT problem space.

And so, having an expert that has a statewide view, and that has statewide authority, and that will be backed up by administration leadership, is really key. Because otherwise, you have no leader in place to address breaking down the silos and building the connections. And so, I think that that's absolutely key.

And our Chief Data Officer right now, he's on a listening tour. His job is to go listen, really, and hear. Because if you're not sharing your data, it's not because you don't want to necessarily, it's because you feel like you can't, right, or that you shouldn't, or that it would be too risky. And so, it's really understanding and honoring, taking the time to understand and honor the particular issues that organizations have, and then co-creating ways to address those together, where you can actually get to a solution to actually start sharing this stuff. So having that person in place, I think, is really important.

And then, above that, I think, in terms of an interagency data exchange or something like that, boiling it down to its simplest components, right? What are the things that we can all agree upon? Rather than rehashing what we all agree upon every single time we need to share the data, it's just a waste of resources, a waste of time. And so, I think, again, it's the people, it's the collaboration, and it's bringing folks along.

[PAUL DECKER]

Jeffrey, I just wanted to bring this back full circle to close us out. So I want to step back a second to storytelling. Government, state government, federal government, local government, there are amazing things like this that get done in government. So how can government do a better job of telling the story about its services and their impact?

[JEFFERY MARINO]

I really think that it starts with that ability to deliver value in increments, incremental value delivery. It's really hard to tell. And it's very ironic, right? Because oftentimes, when we think of a Big Bang Waterfall project, we understand that as one day, we're going to be able to tell the story of the great success, you know? And there have been those. And it is not impossible. And in some cases, it's necessary.

I mean, you don't have incremental improvements in building the Golden Gate Bridge, right? You have to start and then you have to finish and then it can be a bridge and we can all celebrate it. But as it relates to improving service delivery, and especially in the digital space, if you can get on the track of being able to deliver value quickly, you can start telling that story one piece at a time. And that's actually a more interesting way of telling a story as well, I think, or it can be.

I think if I go back, I'll go back to my literature graduate school days, the epistolary novel, which is like a series of letters, right? And so you can tell a story in small chunks as well, that becomes compelling, because you want to understand what happens next.

I think the other thing, too, is all about impact. And so to do that, I mean, that's the why, that's what makes these stories interesting. And so to hear that you were able to close an eligibility gap, that you were able to get more pandemic aid into hands faster. And in order to tell that story most effectively, you need to include the people who were impacted, the people who had the positive outcome as well, which is another thing we're really bad at, is talking to Californians, both before and after we're trying to help deliver a service, right? And so if you can find a way to include the voices of those that you're helping to improve their lives, it's a fantastic way of telling your story.

[PAUL DECKER]

Jeffrey, this has been a great conversation. I really enjoyed it. I appreciate your insights on data and on innovation. And thank you for the conversation.

[JEFFERY MARINO]

Thank you so much. It was my pleasure. I think that ODI is going to have to start a podcast so we can interview you because I have a bunch of questions for you that I wasn't able to ask. And so I'm looking forward to the next conversation. Thank you so much for having me.

[J.B. WOGAN]

Thanks again to our guest, Jeffery Marino. And thanks to Paul Decker for stepping in as the guest host for this episode. And thank you for listening to this episode of On the Evidence, the Mathematica podcast. This episode was produced by the inimitable Rick Stoddard. If you liked the show, please consider leaving us a rating and review wherever you listen to podcasts. To catch future episodes of the show, subscribe at mathematica.org/ontheevidence.