How Virtual and Group Tutoring Could Address Learning Loss

[GREG CHOJNACKI]

I think there's so much interest in tutoring generally right now because we're at a moment where there's an incredibly urgent need to boost math learning, and there's a strong body of evidence showing that tutoring can move the needle and really help students accelerate their learning.

[J.B. WOGAN]

I’m J.B. Wogan from Mathematica and welcome back to On the Evidence.

The COVID-19 pandemic is over, but its harmful effects on the U.S. education system still linger. Standardized test scores continue to show that students have fallen behind in learning. To take one example, recent research found that, from 2019 to 2022, math achievement for eighth graders fell by roughly three-quarters of a grade level, on average, on the National Assessment of Educational Progress, a test that is mandated by Congress and administered every year.

Past research points to at least one way we can accelerate student learning in math: high dosage tutoring. That’s generally defined as one-on-one tutoring or tutoring in very small groups of students, as few as one student and as many as four, for at least three times a week. And the strongest evidence to date shows the effectiveness of in-person tutoring.

The problem is, that form of high-dosage tutoring is too expensive to be a national solution to math learning loss. During the pandemic, Congress provided school districts with temporary stimulus funds through the American Rescue Plan Act to address the negative impacts of COVID-19 on student learning, but the money is running out. Districts have to spend down those funds no later than January of 2025.

A few years ago, the Bill & Melinda Gates Foundation recognized that the education field needed evidence on effective tutoring models that would cost less and help overcome educator staffing shortages, with the ultimate goal of serving a greater number of students.

On this episode, we’re going to discuss promising evidence on alternative tutoring approaches that could still help students catch up in the classroom, but would be more feasible for school districts to adopt at scale—even after the federal funds from the American Rescue Plan have expired.

The evidence stems from evaluations Mathematica conducted for the Gates Foundation of eight tutoring programs that piloted a variety of tutoring approaches, such as virtual and group tutoring, among diverse groups of students in grades 4 through 10 during the 2021–2022 school year.

Our guests for this episode are Brandi Garza, Hasan Ali, and Gregory Chojnacki.

Brandi is with the Corpus Christi Independent School District, which participated in one of the evaluations conducted by Mathematica. Brandi also oversees the districts reporting and use of the federal education stimulus funds provided through the American Rescue Plan.

Hasan is the founder and CEO of Air Tutors, an online tutoring company that participated in the evaluation with the Corpus Christi Independent School District. A Bayesian analysis based on Mathematica’s impact evaluation indicated that “Air Tutors had a 93 percent chance of boosting math knowledge,” though the size of the boost varied based on program details like whether the tutoring happened during or after the school day.

And Greg is a senior researcher at Mathematica who co-authored reports on the tutoring evaluations we’ll be discussing.

I hope you find the conversation useful.

[J.B. WOGAN]

Greg, I want to start with you. You co-authored a research brief about eight alternative tutoring programs intended to improve math learning among a range of students from the fourth grade all the way up to the tenth grade. And there were some promising results, which we'll get to later in the conversation, but for now, can you just help us set the stage? Why is there so much interest right now in alternative forms of tutoring? Like alternative to what? And why would tutoring be especially important in K-12 education at this moment?

[GREG CHOJNACKI]

Sure. Well, first, I think there's so much interest in tutoring generally right now because we're at a moment where there's an incredibly urgent need to boost math learning, and there's a strong body of evidence showing that tutoring can move the needle and really help students accelerate their learning. So it'll come as no surprise to anyone on this podcast when I say there's never been a more pressing need for more support for students' math learning than there is right now. And that's both as a result of the COVID-19 pandemic and in response to long-running inequity in students' access to high-quality math instruction. So to just put some numbers to that statement, between 2019 and 2022, we saw the largest decline in math scores nationwide among eighth graders that's ever been recorded between testing periods. And those declines were concentrated in districts that serve large numbers of students who are racial minorities or whose families have low incomes. And so far nationwide, we've only made up about a third of that learning loss. And the inequality between wealthier and poorer districts that grew worse during the pandemic hasn't returned to pre-pandemic levels in most states, according to recent research that was released by Harvard and Stanford.

And at the same time, there's a robust body of evidence showing that in-person, one-on-one tutoring at a high dosage of multiple sessions per week can provide a big boost to students' learning. So tutoring is seen as an important part of the solution to this huge challenge we're facing as a country. And then as to why there's so much interest in alternative forms of tutoring right now, although as I mentioned, there's strong evidence that in-person, one-on-one tutoring can provide a big boost to students' learning, states and districts often can't implement that form of tutoring widely because it would be too expensive and would probably require more tutors than are out there right now.

So there's an interest in alternatives to both the in-person and the one-to-one aspect of the sort of standard tutoring model that I was just talking about a moment ago, with the hope that there might be effective models that are either a lower cost for students and/or models that could help spread each tutor's support to more students. And the hope is ultimately that providers can reach more students by tutoring students in groups, not one-on-one, and that using online models can make it easier to find tutors and connect them with students. And so with all that context in mind, the goal of the studies that we conducted with partners like Air Tutors and Corpus Christi was to understand whether these alternative tutoring models could be effective in boosting students' learning.

[J.B. WOGAN]

Hasan, I want to turn to you. Air Tutors was one of the programs evaluated by Mathematica. So first, for listeners who aren't familiar yet with Air Tutors, could you summarize the Air Tutors model?

[HASAN ALI]

Yeah, for sure. And thank you, J.B., for having us on. Really, really appreciate it. Love sharing our story with you all and the awesome collaboration we have with Corpus Christi ISD. Shout out, Brandi, you're awesome. So Air Tutors ,we're an online tutoring program that started in 2016, and we have created our own platform which allows us to innovatively deliver our tutoring services that spans every grade level in K-12 and in any academic subject using expert tutors that have master's and PhD's that have awesome personalities that shine through the web.

[J.B. WOGAN]

All right. That sounds lovely, awesome personalities that shine through the web. I love that phrase. Okay. So one thing I did want to ask about in terms of the model, it's online tutoring, but I think when I was reading the evaluation of Air Tutors, the students are sometimes receiving the instruction on-site, on-campus, is that right? Where do students receive this online tutoring?

[HASAN ALI]

Right. So it depends on the implementation strategy that best aligns with the school district and the school site, but the majority of sessions either happen in three different ways. So during school hours, after school at school, or after school at home. So the during school hours approach is usually like an interventional period, push-in, pull-out type program, which we -- yeah.

[J.B. WOGAN]

Okay. All right, that's helpful. Before I read that, I assumed that it would be online at home, so it was interesting to hear about these other options where it could be on-campus. So you founded Air Tutors well before the pandemic. Initially, what did you see as the service gap that your organization could fill?

[HASAN ALI]

Yeah, so thanks for pointing that out, because it's interesting being in the space for nearly a decade now, because when we started off, it was like, there wasn't any questions about what differentiates you from other organizations. It was mostly like, who are you and why are you doing tutoring? So it's been an awesome journey growing this thing. Basically, when we started off, we already knew that nationwide data could use a lot of work. So it was basically, how do we get math and literacy scores up to speed? How do we support English learning students with bilingual educators? So it was basically the same needs were there prior to the pandemic, but after the pandemic, those needs were really, really exacerbated, really, really much more elevated and a higher sense of urgency to help fill in the gap.

[J.B. WOGAN]

So I remember doing a podcast back in March of 2020, when we were talking about what the evidence was on remote learning, and one of the challenges was there really wasn't a lot of evidence of -- just remote learning was relatively rare, and there wasn't great evidence on exactly the context that suddenly everyone found themselves in during in-person shutdowns. But you were ahead of the game. So how did needs of school districts change during the pandemic?

[HASAN ALI]

Yes, it wasn't necessarily that they changed. It was more like there was just the urgency to catch students up was really heightened. So thankfully, like with partnerships with Brandi and Corpus Christi, we're able to really, like you're mentioning, fill in the research studies out there, highlighting the efficacy with online tutoring, because we were seeing it prior to the pandemic, the wonderful results that we had with school districts across the country prior to the pandemic, we were seeing that. But those weren't like third-party, scientifically evaluated in a very rigorous manner. So being able to have this partnership was really amazing, especially since like the need was so high and the team collaboration to build out the program was really awesome.

[J.B. WOGAN]

Brandi, I want to turn to you now. Your school district participated in one of the tutoring pilot programs that Mathematica evaluated. What appealed to you about this specific form of tutoring? Like, why did it seem like a good solution for the students in your school district and their current or at least at the time, their learning needs?

[BRANDI GARZA]

Well, like you mentioned, we were behind in math, but that with the pandemic hitting us and then House Bill 45-45 passing in Texas, which required our students to get 30 hours of tutoring beyond their regular instructional day for any of their STAR, their high stakes testing exams that they did not pass. And so we had students who if they didn't pass multiple tests, they had to make up all these hours. So we had to find a way that we could fit that in. So we were, I would say, a little bit of a state of a panic, like, how are we going to fit this in? We've got a regular instructional day. We don't have the manpower to do it. And so I actually saw, or our team saw a webinar that TEA put on where Ector County ISD had actually discussed how they had utilized Air Tutors and some other tutoring companies to meet this need. So we were very intrigued by that and so reached out to Hasan, met with him. And I have to say, 24-7 customer service, we worked on it, you know, throughout the night, you know, emailing, responding, looking at what it would cost us. And then he had presented us with the idea of the Mathematica study, which was great because we were looking at how can we fund this?

You know, we were very interested in it. We thought that it would really solve the problem that we had in meeting those needs of our students. And we had a very high number of students that we needed to service for those hours. So we put it all into play and we were very impressed with it. The students built the relationships with their tutors. It helped us to meet those hours and get a jump on the game. So we were very impressed with it. We had an elementary school and a middle school that we participated in the study. And so we varied a little bit with who we picked for the different campuses. So we did overall have good results.

One of the campuses I felt like if we would have picked a little bit different students to participate, we would have had even better results. But we were very impressed with the outcome. And from then, after the study, we then went ahead and contracted with their tutors to continue that relationship to get our students to get in those hours. So it really helped us fill that need. And then, the teacher shortage, that's something we still continue. And I think it's a nationwide shortage and we're still dealing with that as well. And it really helped us to meet that need as well as the -- I think it was a 3-to-1 ratio that we had to work with, which was another very huge challenge for us to deal with. And so Air Tutors was instrumental in us being able to get House Bill 45-45 off the ground in our district.

[J.B. WOGAN]

TEA is a national association?

[BRANDI GARZA]

Oh, sorry. It's the Texas Education Agency.

[J.B. WOGAN]

Great. And then did you say Hector County? So this was an independent school district?

[BRANDI GARZA]

Ector County ISD.

[J.B. WOGAN]

And that's another school district?

[BRANDI GARZA]

Yes, and they were featured on that Texas Education Agency webinar. What they do is they feature different districts and then they came together to say how they were addressing the House Bill 45-45 issue in their district. And so from there, we got some ideas of how we could work with it as well.

[J.B. WOGAN]

And then you said if you had selected different students, maybe the outcomes would have been even better. Different how?

[BRANDI GARZA]

So one of the campuses selected those students who were right on the cusp, they just needed that extra push to be successful. And they had very good results. The other one, the other campus, we picked more of a diverse group of students, some of them who were very, very low and they did show gains. But, you know, our goal was to push them over that edge and get them to pass their state assessment. So I just feel if we would have had like a little bit different grouping there, we would have had even better results, although they did make gains and we did show growth. So overall, that was satisfactory with us because that was the goal. And for them to build that math, you know, understanding the math concepts and building that foundation. So that did help them long term as well.

[J.B. WOGAN]

And then you mentioned teacher shortages. Is that specifically in math or is that across the board?

[BRANDI GARZA]

It is across the board, but math is a difficult subject area to find teachers in as well.

[J.B. WOGAN]

Okay, and I was curious about what kinds of tutoring services were already available to students before the pilot. What was the tutoring landscape like?

[BRANDI GARZA]

So what we did in our district is we did after school tutoring. We did offer Saturday school and we do have summer school. But the issue that we were faced with and continue to be faced with is a lot of time the teachers are tired. They're burned out. You know, they're giving their all all day long and they're tired. They don't want to stay after school and tutor more. Although they do, because we have amazing teachers are dedicated and they're willing to do it. But you do have that teacher burnout. And so partnering with Air Tutors and Mathematica and doing the study and having them participate in the tutoring program, we had different tutors come in. There were consistent tutors, which was great because the students built those relationships with them. And it gave our teachers that little bit of a break. And it also gave a different flavor to our students. So they're with this one teacher all the time. And maybe they're not grasping a concept -- when they went to Air Tutors and met with their consistent tutor, they got a different flavor and they got more of a one-on-one approach or a small group approach. So a lot of times they were getting or grasping those concepts that they were having a difficult time grasping in the classroom.

[J.B. WOGAN]

I wanted to ask what was different and potentially better about the kind of tutoring students your students would get through the pilot. You've already kind of implied some of that answer in terms of teachers get a break, students get a different flavor. But is there anything else you want to say in terms of what was potentially better about this kind of tutoring?

[BRANDI GARZA]

Well, the thing that really stuck out to us was initially the relationship building. The tutors genuinely took an interest in the students. They got to know them. They built those relationships and built that trust. So once they had that, the students were willing to work hard and do whatever they needed to do for them. So monitoring our part was a huge thing. We had to make sure that we had effective monitors that were ensuring our students were engaged. But for the most part, the students liked the program. They enjoyed that time with their tutors because they had that relationship. And then we had students, we asked them what they enjoyed about the program. And a lot of them liked that they got more of that one-on-one attention. If they didn't understand something, they were not in a classroom of 22 to 27 students or what have you, or 25 students. They could ask the tutor and they maybe had a group of four or five, some of them maybe six, seven students, but they were still able to get more of that one-on-one assistance. And so they felt that it really helped them to understand the concepts where they may have struggled a little bit more with it in the classroom. So that was great to see. In fact, funny story, once the tutoring had ended, we had a student that actually emailed and reached out because they missed their tutor.

[HASAN ALI]

Oh. That's so cute. That's so cute.

[BRANDI GARZA]

So, yeah, it's very different from what we were providing and just a different setting. I think the students really enjoy using technology, and I've never liked the word virtual tutoring because it really, truly, in my opinion, isn't virtual tutoring because they are live tutors and they're providing that live assistance. And then another thing that was great on our part, and this may be veering off the question, but Air Tutors also provided the notes for us, anecdotal notes on how the students perform, what they struggled with, their attendance, if they were engaged or not. And so we were able to look at those reports and immediately address any issues with the students that there may have been issues or maybe they needed some extra help. So it really was a tool that worked hand-in-hand with the district and our teachers to help our students make as much progress as possible.

[J.B. WOGAN]

Greg, I want to get back to the evaluation that Mathematica conducted. I know it was an evaluation of eight alternative tutoring programs. Give us sort of the Cliff Notes. What should state and local education leaders know about your findings?

[GREG CHOJNACKI]

Sure, so first of all, Brandi, it was really great to hear about how the tutoring showed up from your perspective, and a lot of really interesting sort of details in what you shared. So thank you. So, yeah, as far as what district and state education leaders should take away from these findings, I would point out three take-home messages that I think they should take from these eight studies that we conducted. First, as a group, these studies show that multiple approaches to tutoring, including group and online tutoring, can boost math knowledge and other student outcomes. All of the eight programs that we studied were either online or they delivered tutoring to groups of students or they did both. So online group tutoring, and Air Tutors is a great example of that online group tutoring approach. And overall, the group tutoring models that we studied, as Brandi kind of alluded to, tended to have around three or four students per group, maybe a little bit more, although programs did vary across the eight different programs in their sort of intended group size. And some programs had groups as small as two students, where others had groups as large as 10 students across the different programs that we studied. So out of the eight programs that we studied in these evaluations, five of them meaningfully boosted students' learning.

And it's important to remember that these studies took place during the 2021-22 school year, when many schools were still facing lots of disruptions to their schedules and staffing. And the tutoring providers also, in many cases, were working through high staff turnover and other challenges related to all the many disruptions that came up during the pandemic. So if we kind of zoom in on the Air Tutors study as an example, we found that students who were randomly selected to receive tutoring from Air Tutors really scored substantially higher on a standardized math test than students who didn't receive that tutoring. The test score gain for participating students, which was equivalent to about four percentile points, was large enough that even though the study had a relatively small sample of about 275 students, we still were able to conclude with a high degree of confidence, 93 percent, as you mentioned earlier, J.B., that this tutoring did have a positive impact on students' math learning.

And again, as Brandi kind of mentioned earlier, Air Tutors students also reported strong relationships with their tutors and they also reported meaningful boosts in their confidence in their math ability. And we saw that with several other tutoring providers that we studied as well. And this matters because some research has shown that students with higher math confidence are more likely to have other positive outcomes, including pursuing a major in science, technology, engineering or math once they get to college. So that is maybe not the Cliff Notes version of the first big takeaway. That takeaway is these studies show that multiple approaches can really boost students' math knowledge and other outcomes. The second key takeaway has to do with what we learned about the programs that were effective compared to the ones that weren't. So when we look across the range of different programs that we studied, what we found was the quality of implementation of these programs is likely to be as important for students' learning as any specific design features like the exact number of students that were in each tutoring group.

So looking across these programs, the ones that had meaningful effects on student learning had high student attendance, consistent staffing and logistics like scheduling all worked out. Those with smaller effects tended to either have low attendance or they encountered challenges when they were implementing core components of the program, or they might have encountered both of those categories of challenges. And then the third big takeaway, I would say, is these study results suggest that strong student-tutor relationships may be a really important part of successful tutoring. And Brandi kind of spoke to this earlier as well. So we saw across programs that programs with high attendance and positive effects on learning also tended to have a large share of students that were reporting a really strong relationship with their tutor and a sense of belonging in their tutoring sessions.

[J.B. WOGAN]

Okay, let me let me ask a follow up question. Thank you for that summary. I think this is specific to the evaluation of Air Tutors, but I think it's also important for the larger takeaway across the evaluation of all eight programs. But with the Air Tutors study, I saw that Mathematica found that the schools where students received online tutoring during the school day had higher attendance. Why might the timing during or after school matter? And I know, Greg, that your team investigated this and generated some answers, although I would be interested in hearing what Brandi and Hasan have to say as well.

[GREG CHOJNACKI]

Yeah, I completely agree. I'll just note that other times that we've had a chance to talk about what we observed with Hasan, you've shared some really helpful insights about what might be behind some of that. But what we saw was, as you noted, when tutoring was happening during the school day, attendance rates were higher than when tutoring was happening outside school. And one of the things that I think is really important to underscore, and this applies, I believe, both with Air Tutors and as well as with students who attended other tutoring programs. These are middle schoolers, and sometimes these are middle schoolers who have responsibilities outside of school. They might have responsibilities to take care of their siblings.

They may have extracurricular activities or other enrichment activities that are happening outside of school as well. And so once we're looking at scheduling tutoring that's outside of those confines of the school day, other obligations can kind of create a tension between attending tutoring and being fully present for that tutoring versus other responsibilities that students have. So that can be one factor that is behind why attendance is highest when the tutoring is offered during the school day. It kind of provides the tutoring at a time when students have that ability to focus fully on the tutoring with as few potential other draws on their time and attention as possible. But yeah, absolutely, welcome for others to jump in. And sorry, J.B., you were saying something?

[J.B. WOGAN]

No, yeah, Hasan, anything you'd like to add, any insights you have from the Air Tutors perspective about during versus after the school day?

[HASAN ALI]

Yeah, for sure. And one thing that I loved about this study is that we're able to really assess and understand the different efficacy for the different types of models of implementation, meaning like this randomized control trial we did was geared towards understanding why and how and by how much after school, at school, after school at home and during school hours tutoring is most effective. So it was really cool to really dig deep into that because, you know, we always had the understanding that during school hours would yield the best results, but we didn't know by how much and why. So it was cool to see that it was like significantly better results during school hours. It wasn't like just a little bit. It was drastically different, very, very impactful results during the school hours. It was also interesting to note that during after school at home had better attendance rates, which resulted in better efficacy for the program instead of after school at school, mostly because, you know, kids would forget to stay after school. They'd have to get picked up early. Something would come up where a student would not stay for the required dosage and the whole duration of the program.

So that was really interesting to note that the after school at home model is a better approach than the after school at school. I just think that in the learning environment created in a during school approach is much more conducive to yielding much better results. For instance, the teacher in the room is our lifeline. I mean, teachers would help with the classroom management. They'd help make sure that the students can sign in. And it's fun with it for the teacher, too, because they have a more laid back session, meaning a class period. So it was becoming an integrated community member during school hours we found to be like really, really effective.

[J.B. WOGAN]

And so, okay, that was something I wanted to clarify, too. So there are people in person who are supporting the sessions, even if the tutor is on the computer, is doing an online session. So, Brandi, could you speak to what that looked like for the middle school that was studied? And then I want to understand it, because I'm assuming there must be parents or guardians or there's someone in the home that might be acting in a similar role to help.

[BRANDI GARZA]

Yes, so absolutely. That was a huge piece of the puzzle, although Air Tutors did set everything up for us. Everything was done, the scheduling, everything. So our part was to have either a parent or a teacher in the lab setting that could monitor that class. And so it was a learning experience for us as well, because we would do instructional visits to the campuses that were implementing this program. And it varied, you know, you may have someone who's really active, involved, they're up, they're walking around, they're monitoring, they're seeing what the students are doing. They're assisting with technical issues, which, by the way, when we had any, we were able to call in and it was taken care of immediately.

So that was great. But you need that facilitator in the room that can make sure that that's taken care of. So, I mean, having that person there is huge. Where it helped us is that we were able to have that lab full of students, although they're tutoring with their breakout groups in the sessions. So we don't have to have as much manpower. We didn't need that many teachers to monitor all the students. We just needed a person in that lab that could go and facilitate and make sure that everybody was engaged. They were on task. If they needed assistance, they could assist. And again, it was a learning experience for us because we learned we ended up doing a checklist for our people and what the expectations were for them. And so as we went through, we learned that, okay, they need to be doing this. They need to be up. They need to be monitoring.

And so it is more of a, I guess you would say, as Hasan said, more of an easier or laid back session for them. But we still wanted to ensure that they were engaged, that they were active and that they were making sure that all the students were on task so that we could get the most bang out of it. We wanted to make sure our students were truly engaged, not goofing off, not being a behavior problem, which middle school, they're a little bit different animals. So you do have to make sure we have that adult present and monitoring. Although I will say that they did enjoy it and we had other programs going on and we did actually have students that ask, can I go in there? I want to be in that lab and participate in the tutoring. So that was good to see. But yeah, we have to have the adult in the room. But it can be a paraprofessional, which, you know, is easier for us to staff and certified teachers as well.

[J.B. WOGAN]

Got it. Okay, so there's some demand on in-person staffing, but it's way less. And you have some flexibility in terms of who. Yeah. So we've kind of talked, we've done the prologue, we've done the sort of the main movie. I want to hear the epilogue. I want to hear what's happened since the pilot ended. So in Corpus Christi, Brandi, what has been the impact of these forms of alternative math tutoring in your district?

[BRANDI GARZA]

Well, it definitely is. Greg had mentioned earlier, our students have built that math confidence. You know, they did build that foundation. They grasped those concepts that they were struggling with. Math continues to be an area of focus for us, just as it is nationwide. So we're still building on that. ESSER funding has come to an end or is coming to an end this year. So funding with districts everywhere is always an issue. So we are looking at alternative ways to fund different programs and all. But for the most part, it was very effective. Our students did gain the knowledge, the confidence to move forward. They made gains. We saw that growth. And so we're hoping to continue to see that trickle over to our state assessments. And House Bill 45-45 has now changed a little bit. It's now House Bill 14-16. So we still do have to meet some requirements in that area. So, I mean, the tutoring has helped us get there. It helped us build that foundation. And so it's something that definitely we feel is effective. But like all other districts with our federal funding coming to an end, we're looking for ways to continue to fund programs.

[J.B. WOGAN]

And the ESSER funding you mentioned, that is -- I'm forgetting what the acronym stands for.

[BRANDI GARZA]

It's the Federal COVID Relief Funds.

[J.B. WOGAN]

Thank you.

[BRANDI GARZA]

So funding that we use to accelerate learning for our students to make up for the learning loss.

[J.B. WOGAN]

All right, perfect. And so that money is coming to an end or has come to an end. And are you continuing to use virtual and group tutoring at the moment?

[BRANDI GARZA]

We are not at this time because of the funding issue. And with the ESSER funding, we had to earmark our funds for the three years that we had it. And so the money was earmarked for different things. So we did utilize the money that we had earmarked for the tutoring. And so there is a requirement that 20 percent of the funding that districts receive have to go towards accelerating that learning loss. And so most definitely utilizing the services of Air Tutors and the virtual tutoring platform did meet that requirement. So that was part of our funding that we used it for.

[J.B. WOGAN]

Okay, so at this point, you're looking for new funding sources to perhaps revive that in some form in the future.

[BRANDI GARZA]

Yes, for programs across the board.

[J.B. WOGAN]

Okay, so if there are any funders listening.

[BRANDI GARZA]

Yes.

[J.B. WOGAN]

Hasan, I'm curious about the evaluation's impact on Air Tutors as well. What did you learn about how the model works and which parts would you say were validating, it was good news, but kind of what you already expected, and which parts maybe challenged you to tweak the program you have.

[HASAN ALI]

So I think the part that was tweak the program, to answer that part, was mostly understanding that after school at school has more attendance issues than after school at home. That was pretty surprising for us. We didn't really think that was going to happen. We knew that during school hours were going to yield the best results. We didn't know that it was going to yield that much greater results. So that was really cool.

So what we're doing now is really trying to create a sustained federal funding model for high dosage tutoring. We launched the country's first online high dosage tutoring program back in 2017. So seeing these types of studies, seeing larger, like getting larger population sizes to really hone in on a randomized controlled trial, hone in on the efficacy is really what we're trying to do now to build that case for sustaining federal funding. Also, we're being really innovative in that approach. So we're partnering with the Southern Education Foundation to bring outcomes-based contracting to the space so that the district is paying for results, not services.

So meaning like we're hoping to -- because no tutoring provider is going to go out there and say, oh, our tutors are boring, our tutors are lame, our results are kind of cool. So, you know, we're really trying our best to hold ourselves accountable so that the district pays for the outcomes that we promise using outcomes-based contracting. Sarah Glover, Brittany Miller over there at Southern Education Foundation have been awesome pioneers in bringing OBCs, outcomes-based contracting, to the space. So trying to be innovative and really helping come up with the funding model while we're pushing ahead for the federal funding case.

[J.B. WOGAN]

Fascinating. Greg, I know this research was by design very collaborative with the tutoring providers like Air Tutors, and I was hoping you could explain how the design was different from a traditional large-scale, multi-year, randomized control trial or impact evaluation and how that was intended to actually help providers refine their programs.

[GREG CHOJNACKI]

Sure, absolutely. And I want to start by just echoing something you said a moment ago, which is the collaborative nature of this work. Hasan and Brandi were both absolutely the best collaborators you could ever hope for in this type of work. And on behalf of myself, as well as my colleague here at Mathematica, or I should say, our colleague here at Mathematica, Lily Fesler, who really led this evaluation in partnership on Mathematica's side in partnership with Hasan and Brandi, on her behalf, my own, would just say we were very fortunate to have this pair of collaborators who we worked with on this. So thank you to both of you, Hasan and Brandi. And then digging in a little bit to what that looked like in practice and how it differed from a large kind of multi-year, randomized control trial, I would start by saying, so usually the focus of a large multi-year RCT, or randomized control trial, is to develop the most precise and accurate estimates possible of a program's impact. But that level of precision and accuracy takes a lot of time and a lot of resources to achieve. In those large studies, findings are often not available for as long as five years after the study begins. And the cost of the study might be 10 times higher than the work that we completed with Air Tutors and Corpus Christi.

So in contrast to that type of study, which certainly has its value, the evaluation that we did with Air Tutors was designed to achieve two key goals. First, we wanted to develop initial rigorous evidence on Air Tutors and to do that fast enough that as soon as possible, those findings could inform decisions about virtual and group tutoring by districts, states, funders and other tutoring providers who are really racing to try to, you know, boost student learning after the pandemic. And then we also wanted the evaluation to provide useful information along the way to Air Tutors and to other providers to help them continue innovating and refining the tutoring, as we just heard Hasan talking about some of those lessons learned or takeaways.

So one of the ways that we achieved that second goal was to work in steps with Air Tutors and other providers. First, we completed an initial pilot evaluation with them in spring of 2021. And then Air Tutors used those results to both learn in sequence or in steps and to make some refinements to their tutoring, even as we planned the second step, which was the rapid randomized trial that took place in two districts during the next year. And one of those districts was Corpus Christi. So this allowed the collaboration, the partnership that we were working with to both build evidence on the program's effectiveness that we could be increasingly confident about while also providing timely information to Air Tutors as they continue to refine the tutoring model. And I'll just note, Hasan, you mentioned the sort of after school versus in school and then the contrast between after school at home versus after school at school.

And those are really interesting to hear about. And then I remember one of the outcomes of that early pilot was also, and I'm curious if this resonates with your recollection, was a question about group size and deciding, what's the group size that we want to test moving forward in this second year of study in the kind of rapid randomized control trial? And my recollection was that some of the findings from that first pilot semester helped to kind of firm up and ground the decision at Air Tutors about the group size that you all wanted to work with in the second year. But, you know, keep me honest on that. That was one area where I thought the learnings from the pilot phase kind of helped to inform the second phase. But you tell me if that sounds right.

[HASAN ALI]

Yeah, it's definitely right. You know, what we thought was common sense is that one-on-one or two-to-one was the best approach. So when we did that back in like 2019, 2020, in our collaborations, we realized that student participation actually increases way more when they're in a small group. So that was a really cool learning lesson. So thank you, Greg, for bringing that one up. You know, student achievement data in those small groups where group sizes range from three to five were really, really large and impactful. Students also spoke math more during the sessions as opposed to one-on-one or two-to-one sessions. So that was really, really cool, learned lesson. We tended to prefer and see the best results in groups of three-to-one or four-to-one, meaning three students to one tutor or four students to one tutor.

[J.B. WOGAN]

So Mathematica just hosted an event in January that was about the release of a new book from Oxford University Press that is about evaluation and how impact evaluations need to incorporate lessons from these other kinds of evaluations, like from program evaluations, process evaluations, implementation evaluations. And when I was reading the study from Mathematica, it seemed to me like it actually is incorporating some of this more pioneering viewpoint or approach where there were insights about the program design and implementation that then were informing not just -- it wasn't just did it improve math scores or not, but I was getting some insights from the study about why. Were there actual kind of process -- like, yeah, were there those other forms of evaluation taking place as part of this research, Greg?

[GREG CHOJNACKI]

Yes, absolutely. So one of the things that the team at Mathematica and very much in partnership with our partners in tutoring programs and in the districts that we were partnering with, one of the areas of emphasis was to learn in stages and to tailor the learning goals of each stage to answer the questions that would be most useful for kind of all of the different groups involved. Because we knew that we were going into partnerships and we wanted the partnerships to be valuable for everyone.

So an example would be with one of the tutoring providers, we completed an extended period of rapid cycle design and testing sprints, which were really focused on nuts and bolts of the tutoring model, including things like if tutoring is going to be provided in a group setting, what are some of the moves that tutors can use to encourage as much peer-to-peer learning as possible? And Hasan was speaking just a moment ago to some of those benefits of peers learning from each other and students being more engaged when they're in a group of other students who are all kind of sparking each other's interest in the subject matter.

So that's an example of a very specific aspect of a tutoring approach that we partnered with the tutoring provider. In this case, it was a different provider, although we know that Air Tutors gave careful thought to this as well. To try to dig into, are there things that tutors can do to really support that peer-to-peer learning? And that's just one example of several kind of stages of small-scope, rapid design testing that we did with our partners.

And then we also went all the way from those very fine-grained learning moments to learning about the implementation and any barriers that a tutoring provider might have as they really kind of dialed in their supports for implementing tutoring the way that they envisioned it being implemented, and then all the way to measuring the effects on student outcomes. So we did try to work across a fairly broad spectrum of different types of learning in the course of this project.

[J.B. WOGAN]

One of the speakers at the event in January, it was somebody from USAID, was saying that a common problem she sees in a lot of reports is a very thorough approach to everything but the policy implications and recommendation section. And, you know, like that would be the shortest section, and it seems to her like the part that takes the least investment in time and effort. And I'll just say that that did not seem to be the case here. There's actually an entire brief on recommendations to school districts that is trying to offer action-oriented insights based on this research, which I really appreciated. I want to wrap up by asking each of you to weigh in on where we go from here. And I'll put a finer point on that. If alternative forms of tutoring ought to be on the menu for school districts that are wrestling with post-pandemic learning loss, do you really have any advice about how to do it well? And where might future research play a role in improving the efficacy of these alternative tutoring approaches? So, Brandi, let's start with you. Do you have advice for school districts on how to do it well? And what do you still need from the research community?

[BRANDI GARZA]

How to do it well. Well, I would say professional development on our end. A lot of things are beneficial, but fortunately for us working with Air Tutors, a lot of those things were covered by them and taken care of already for us. So on our end, I would just say having that professional development for our lab managers to ensure that it's as effective as it can possibly be on our end. Because that really plays such a key part in the effectiveness. Because the tutors that are online are doing their part. They're doing everything right and they need to do. But we need to make sure that on our end that we're controlling that environment and that we're optimizing the learning for the students. Because everything else was in play. And it was, like I said, it was a learning experience for us. And so I would just say to districts that are considering it, it is a great option. Just make sure that you are really providing some good quality training for your lab managers to ensure that they're maximizing the instructional environment for their students.

[J.B. WOGAN]

Okay. And then what can research organizations like Mathematica or universities, how can they help you? Where might more research be helpful?

[BRANDI GARZA]

Gosh, what they did was amazing. It was very eye-opening. And we really appreciate, shout out to Mathematica, Greg, your team, and Hasan for including Corpus Christi in that study. It was eye-opening to us. We learned a lot about how to group our students, how to get them to grasp those concepts. And it was just totally thinking outside of the box from what we were used to. And so it took some effort to try and push that and to get it to fly. And once it did, I think it really opened up that box for us where we see that we do have some other options.

And so I would just say keep doing what you're doing. I can't think of anything else that they could have covered. I think the group sizing, just the implementation, the way they did it, the results that were shared with us, I mean, we really did get a lot out of it. It did help us. And then moving forward, it's something that I think is very beneficial for districts. It's just with us, it's a funding issue. Where do we go from here? So maybe in your research, Greg, y'all can add how to get some additional funding for districts. Maybe some grant opportunities so that we can continue to implement it. I mean, that's really all I can think of, because I really feel like it was a very, very well-organized and very informational program. I mean, it taught us a lot. And so we're very fortunate to have been selected to participate.

[J.B. WOGAN]

What works to secure funding for education services that work?

[BRANDI GARZA]

Yes, absolutely.

[J.B. WOGAN]

Hasan, for school districts looking to implement something other than in-person, high-dose tutoring, what's your advice for doing it well? And in terms of making this form of tutoring more effective, where do you think further research might help?

[HASAN ALI]

So, I think to dig into the demographics of the students, like, okay, what works best for English learning students? What works best for students in rural settings, remote settings, meaning like really not in an urban area? What works for suburban students? Really digging into that in a quasi-experimental design, which is what we're doing now, trying to increase the population size to hone in on that during school hours and meticulously understand that data since we saw really strong growth over there. We want to replicate it in a larger population.

What I think would be most helpful is like what Brandi was saying, is just having an interventional period and making sure that the teachers and the paraprofessionals are, you know, bought into the process as well, so that they don't think that it's just a free period. It's more like, hey, it's more relaxed than you're used to, but it also has some classroom management portion to it. But also, I think it would be really interesting to see how many of the paraprofessional teachers monitoring the class end up turning into certified teachers, because the idea is to get as many tutors and paraprofessionals to help with the teacher shortage that we're seeing across the country. So, that could be one really interesting study is to see how many of our tutors and paras in the room end up pursuing a career in teaching.

[J.B. WOGAN]

So, Greg, I'll let you have the last word. And you've actually co-authored a brief of four recommendations, as I mentioned earlier, recommendations for school districts looking to select and support tutoring programs. So, in addition to a burning question you would love to see further research investigate, do you want to highlight one or two of those recommendations here?

[GREG CHOJNACKI]

Yeah, absolutely. And I'll just highlight two of the recommendations that were in the brief you referred to. One of them, I think, is very aligned with what Hasan and Brandi were just saying. One takeaway we had in terms of supporting really strong implementation was just the value for both the provider and the district partner to proactively monitor implementation and attendance regularly. And I think that is something that the monitor, that paraprofessional role, can really help out with. What we saw was that all the tutoring programs that generated moderate to large effects had the sort of planned staffing, logistics, and support in place, and students were showing up and participating at high rates. And we also saw that, in some cases, when attendance challenges showed up, there was kind of a red flag that there might be other complications going on in the program implementation.

So, this is not something we saw with Air tutors, but with some of the other programs that we were learning about, it proved really helpful when there were attendance challenges showing up for the district and the program partner to kind of try to dig in a little bit to what might be behind those. And oftentimes there was something really informative about either a piece of technology or some additional support for staff that could be useful to make sure that the other core components of the tutoring were happening the way that they really were intended to.

So that sort of attendance and monitoring piece is one. And then the other piece of advice for districts would just be to choose providers who offer their tutors professional supports that are tailored to the tutor's level of experience. And I say that because one of the ways that these programs differed from one to another was they used a lot of different staffing models. And actually, a lot of different staffing models were successful. But what we saw was that programs that used either volunteers or AmeriCorps staff who might be less experienced as educators tended to provide scripted lessons and substantial training to build up those staff members' capacity as tutors.

In contrast, some programs hired very, very experienced tutors and offered higher wages to those tutors, but they didn't invest as much and kind of didn't need to invest as much in heavy, heavy, substantial tutor training and scripting of lessons and things like that. So different staffing approaches can work, but the support for tutors kind of needs to be tailored to the expertise that tutors are bringing in was one of the takeaways that we saw. So those are a couple maybe tips for districts thinking about how to support implementation and how to select providers. And then as far as the burning questions that we would really love to pursue or for the field to pursue going forward, you know, in light of the findings suggesting that there's a link between student-tutor relationships and improvements in math knowledge, we think it would be really useful to further explore what might be behind that link.

And particularly just to understand better whether and how investing in strong student-tutor relationships might be driving stronger outcomes like knowledge and attendance. Certainly, we saw that pattern showing up in the research that we did in these eight studies, but we'd love to dig in further there. And then maybe the last piece I would mention is just that because the studies that were conducted in this round were relatively small in scale, as Hasan was mentioning, we think there's a lot of interest and a lot of value in exploring whether providers who are delivering virtual and group tutoring at a larger scale as they expand their programs are still able to maintain kind of those positive effects as they work with a broader range of students and work at a larger scale.

[J.B. WOGAN]

All right. I think that's a great note to end on. Hasan, Greg, Brandi, thank you so much.

[GREG CHOJNACKI]

Well, thank you so much, J.B., and a huge thanks to Hasan and Brandi, both for making time for this conversation and also just for all of the incredible way that you all were partners along the way. Thank you.

[BRANDI GARZA]

Thank you to you all as well. Appreciate it.

[HASAN ALI]

Thank you.

[J.B. WOGAN]

Thanks again to my guests, Brandi Garza, Hasan Ali, and Greg Chojnacki. I also want to thank the inimitable Rick Stoddard, who produced this episode. This episode is dropping in February of 2024, which happens to be the five-year anniversary of Mathematica’s On the Evidence podcast. Whether this is the first time you’re hearing us, or you’ve been with us since 2019, thanks for listening. If you’re fan of the show, please leave us a rating and review wherever you listen to podcasts. It helps others discover our show. To catch future episodes, subscribe by visiting us at mathematica.org/ontheevidence.