
By Greg Toppo, The 74
This story first appeared at The 74, a nonprofit news site covering education. Sign up for free newsletters from The 74 to get more like this in your inbox.
Last December, a small British startup made a big claim: its AI-powered math tutor, with a human checking its work, could outperform a human tutor working alone.
Now the company behind that study is back with a trial seven times larger — and a more complicated narrative.
In a randomized controlled trial of 1,179 students in 10 U.K. public schools, the London-based ed-tech startup Eedi, along with Google DeepMind, found that their AI tutor matched expert human tutors at helping students fix mistakes in the moment. But after 10 weeks, neither kind of tutoring moved scores on a broad standardized math test. And when researchers dug into the data, they found that both kinds of tutoring seemed to help girls while possibly setting boys back.
Eedi is releasing the data on Thursday, including subgroup results, conversation-level analysis and “results we didn’t expect,” said Bibi Groot, Eedi’s chief impact officer.
Here are five things to know about the study:
1. During sessions, an AI tutor was as good as an expert human
In the trial, which ran from April through July, secondary school students spent one class period a week, about 30 to 35 minutes, practicing math on Eedi’s platform with a teacher in the room.
Whenever they got a question wrong, students were randomly assigned to one of four kinds of help: standard remediation in the app; a live, text-based chat with an expert human tutor; a chat with an AI tutor backed up by a human approving each interaction, or an AI tutor with “personalized context,” i.e., a summary of each student’s past misconceptions and work habits, such as whether they’d been rushing through problems.
The four conditions stayed the same for each student for the entire study.
Students who got tutoring of any kind — AI or human — answered similar follow-up questions correctly 56% to 58% of the time, compared with 43% after standard remediation. “The AI and human tutor are basically on par,” Groot said in an interview.
The new results also show a shift from last year’s smaller study, in which the supervised AI tutor slightly outperformed humans, 66.2% to 60.7% on new problems in later topics.
While the results for AI tutors might seem disappointing at first glance, Groot said, they’re actually promising, partly because “human tutors are expensive” and most schools can’t afford them. Technology that can perform just as well could be more workable.
For the record, she added, keeping expert human tutors in the loop to monitor AI is also expensive and “not a scalable solution.” She noted that Eedi in September began another trial with more than 15,000 students, in which the AI interacts directly with students, relying on humans only if conversations go off the rails.
2. Tutoring gains didn’t show up on a broad math test
Researchers designed the new trial partly to answer a common critique of AI tutoring studies: that they measure learning too soon after the fact, with samples too small to trust.
In the new study, students took Star Maths, a standardized test, before and afterwards.
The results were flat. “We don’t see that the interactive tutoring improves Star Maths scores, over and above the standard remediation,” Groot said.
Among the 916 students who sat for both tests, those tutored by AI scored about the same as those who got “static practice” without tutoring. Students with human tutors scored slightly lower than both groups, likely because the tutored students covered less material in more depth.
Another surprise in the new findings, Groot said, was that giving the AI personal context about students didn’t really improve outcomes. That’s likely because the AI already knew the student misconceptions behind each wrong answer — the extra information was probably superfluous.
3. Boys didn’t benefit as much from tutoring of any kind
On Star Maths, girls tutored by either AI or a human tended to do slightly better than girls who got static practice. Tutored boys actually did worse — especially those who worked with live human tutors.
While the findings are tentative, researchers said some boys may be discouraged by the way the system works, since they’re pulled into a conversation with a tutor after every mistake — that may translate as a “signal of failure.” Likewise, relying on text-based chat, as the tutoring does, may favor more confident readers and writers.
The study found no clear differences in either gender by family income or age and students’ prior achievement mattered in just one case: lower-achieving students who worked with human tutors scored lower than similar students on standard practice, possibly because they covered the least material.
4. The AI was safe and — mostly — accurate
Human tutors approved 94.7% of the AI’s draft messages to students without changes, rewriting about 1 in 20. That’s an improvement from the 2025 study, in which humans tweaked about one in four messages to make them appropriate for students.
While no harmful content reached any student — tutors blocked two drafts containing a winking emoji that could be read as overly familiar — the AI wasn’t error-free. Researchers uncovered 11 AI messages with false math statements, seven of which reached students. For context, human tutors’ messages also had seven math errors, as well as “many more typing slips.”
In many cases, Groot said, the AI almost seemed too eager to move students forward, asking follow-up mastery questions more than 90% of the time, even after a student solved a problem, compared to about 60% for human tutors. “The AI sometimes kept asking questions, right?” she said. “It’s trying to be Socratic, and sometimes maybe excessively so.”
5. Most students engaged — but 1 in 5 never replied
Among the biggest worries about ed tech is the notion that it mostly helps students who need it least. In a widely read 2024 Education Next essay, OKO Labs co-founder Laurence Holt called it “the 5 percent problem,” noting that impressive results for popular online math programs, for instance, typically apply only to students who use them as recommended — in one study, that was just 4.7% of students. These students, he wrote, would likely thrive with any practice.
Groot said Eedi’s results show something slightly different and more promising: in thousands of tutoring conversations they analyzed, more than 80% of students engaged with the math at hand. “It’s definitely not a 5% problem,” she said, “because the majority of students do engage.”
That may be because the Eedi sessions were built into weekly class periods with a teacher present — for his part, Holt also found that tech usage often hinged on how committed teachers and schools were to a program.
Still, in the Eedi study, about one in five students never replied when a tutor popped up. That’s an ongoing problem, Groot said. AI tutoring is “great to support students who are already motivated, but we need to figure out how to motivate students who aren’t motivated in the first place.”
—
This story was produced by The 74, a non-profit, independent news organization focused on education in America.
***
Subscribe to our Email Newsletter:
Why Subscribe? Because this conversation matters.
When you subscribe, you’re directly supporting independent, mission-driven journalism about masculinity, relationships, mental health, fatherhood, and social change. Your inbox becomes part of a movement that’s been challenging stereotypes and expanding what it means to be a good man since 2010. Plus, our newsletter is curated by humans, not algorithms — thoughtful context in an AI-flooded world.
The Good Men Project is a mission-driven men’s media platform. A major platform for stories about men, identity, fatherhood, and emotional intelligence—and a trusted home for the national conversation about masculinity.
The Good Men Project accepts paid guest posts and provides bulk guest post packages for SEO agencies and resellers. Email [email protected] for more information
If you believe in the work we are doing here at The Good Men Project, please join us as a Premium Member today.
All Premium Members help support our mission and get to view The Good Men Project with NO ADS.
Need more info? A complete list of benefits is here.
Photo credit: iStock.com

