Twice the learning

In September 2024 the Harvard Gazette ran a story about a physics course whose AI tutor, its headline said, had doubled engagement. Inside, it put a question to readers about learning: "Might you learn more? Maybe even twice as much?"¹ It also said the final results were "pending publication," so the number reached the public before the peer-reviewed paper that would justify it.¹ The paper has since appeared, and so have other randomized studies of AI tutoring. The short answer from laying them side by side: the best trial supports a narrower claim than "twice the learning". My confidence is moderate for the narrow claim and low for the headline version.

01The claim, stated precisely

The claim I am testing is a composite: the Gazette's framing reduced to a plain sentence, "AI tutors double learning", which is my paraphrase and not a quotation from any one source. Stated as plainly as I can, it says that AI tutoring doubles learning compared with a classroom. It has four parts hiding in it. It says AI tutoring in general, when the evidence covers particular tutors. It says learning, which could mean a score right after a lesson or something a student can still do in March. It says doubles, which is a ratio and needs a baseline. And it says compared with a classroom, which could mean a lecture, a good active-learning class or no instruction at all. The studies below each speak to a different combination of those four parts, and none of them speaks to all four.

02What the Harvard trial did

The trial behind the Gazette story is Kestin and colleagues' paper in Scientific Reports.² It ran in Harvard's largest physics class, a course for life-sciences majors with 233 enrolled students, of whom 194 were eligible for the analysis. Eligibility required consent, taking part in both the in-class and the AI-tutored lesson, and completing every pre-test and post-test.²

The design is a randomized crossover. Students were randomly assigned to two groups, with the constraint that students who normally worked together in peer instruction landed in the same group. In the first week group one did an AI-supported lesson at home on surface tension while group two did an active-learning lesson in class. The next week the conditions swapped, on fluid flow.² Because every student experienced both conditions, differences between students largely wash out. The analysis also controlled for topic, test version, prior physics knowledge and prior ChatGPT use.²

The tutor design matters to the result. The authors report that a system prompt alone could not reliably keep a multi-part problem in order, so the platform walked students through each part of each problem in sequence, mirroring how the instructor taught the in-class version. To limit hallucinated answers, the tutor was not left to solve problems from scratch; the design relied on detailed step-by-step solutions.² This is a tutor built around lessons written by experienced instructors, and the authors say as much: they tie the result to a carefully structured framework and to content that suits that format.²

The headline result, in the authors' words: "students learn significantly more in less time when using the AI tutor, compared with the in-class active learning."²

03What "twice" is measuring

Here is the arithmetic behind the word. The median post-test score was 4.5 in the AI condition (N = 142) and 3.5 in the in-class condition (N = 174), against a combined pre-test baseline of 2.75.² The paper gives those Ns as 142 and 174, with 316 at baseline, which is their sum. I read them as counts of scored tests and not of students, and I did not find where the paper explains why 316 is below the 388 a full set of two lessons from 194 students would give.² Gains over baseline were therefore about 1.75 points for the AI tutor and 0.75 for the classroom, and the paper describes the AI group's median gain as over double the classroom's.² So "twice" is a ratio of gains, and the ratio is sensitive to the baseline you subtract. A one-point difference in medians, on a test whose length I did not find reported, becomes a ratio above two once a low baseline of 2.75 is subtracted.

The paper also gives effect sizes, which are easier to compare across studies. A linear regression puts the effect at 0.63 standard deviations. The authors say that figure underestimates the effect because of a ceiling in the post-test scores, and a quantile regression gives a range of 0.73 to 1.3 standard deviations.² That is a large effect on the conventional reading of standard deviations. It is also a wide range, and the width is itself information: when scores bunch against the top of a test, the test cannot tell you how much better the best-served students did.

There is a benchmark in the background that explains why the number landed so easily. The introduction to the Nigerian working paper, which I come to below, recalls Bloom's 1984 finding that one-on-one tutoring outperformed traditional classroom teaching by an average of two standard deviations.⁴ I have not read Bloom's paper and cite it only as that report describes it. The comparisons also differ: Bloom's was against conventional teaching, Harvard's is against active learning, and the Nigerian trial's is against no program, so I do not line the effect sizes up. What matters here is expectation. Two standard deviations is the folk figure for what a personal tutor can do, so a headline saying a chatbot doubles learning sounds like the long-promised tutor arriving.

Two other numbers matter. The AI group's median time on task was 49 minutes.² For the in-class group, the authors assume 60 minutes, which is the 75-minute period minus 15 minutes of testing.² So the "less time" claim compares a measured number with an assumed one. It is a reasonable assumption, and a reader should know it is one. And engagement was measured on a five-point agreement scale, where the AI group averaged 4.1 and the class group 3.6.² That records how students said they felt. It is a different thing from what they learned. The Gazette's headline said engagement doubled. The paper's own engagement scores, 4.1 against 3.6, are not a doubling, and I did not trace where the headline's figure comes from.

04What the Harvard trial did not test

I want to be fair to the authors here, because the limits below come from their own paper. They tested one tutor on two lessons in one course. The topics were chosen so that students would meet them for the first time: over 90% of students reported they had not studied these topics in depth before the course.² That is a good design for isolating the effect of the teaching method, and it also means the result is about first exposure to new material. It says less about practice, review or revision.

The post-tests followed each lesson.² I found no delayed test in the paper, so nothing here tells us whether the advantage was still there a week or a semester later. The authors list retention strategies such as spacing among directions for future work.² They are also direct about scope: "we do not presume that structured AI tutoring will always outperform in-class active learning in all contexts," and they name tasks that need complex synthesis of several concepts as one place it might not.²

More limits come from the design and the setting. The comparison bundles the AI with studying alone at home, at one's own pace, on scripted material. There was no arm with the same sequenced problems and worked solutions and no AI, so the trial cannot say how much of the gain came from the model. Eligibility required finishing every test, which leaves about 17% of the class (39 of 233) out of the analysis,² and the AI lesson was done at home, so the result describes students who did the work unsupervised.

The students were Harvard undergraduates, and the authors report that the in-class instructors' teaching evaluations exceeded departmental averages.² Student evaluations are a weak proxy for how much students learn, so I read that as the authors' reassurance that the classroom was a good one, and no more than that. We still do not know what happens against a weaker one. The authors declare no competing interests, and they publish their data.² I have no reason to doubt the trial. My point is that twice as much, in one hour, on a first look at new physics, measured right afterward is the whole of what was shown.

05Three other sources, three different questions

Among the sources I read, the Harvard trial is the only one that compares an AI tutor with an active-learning classroom. The others ask something else, and the differences are the useful part.

Unguarded access can lower later performance. Bastani and colleagues ran a field experiment in a high school in Turkey, with nearly 1,000 students in grades 9 to 11, across four 90-minute sessions.³ Classrooms were assigned to one of three arms: no AI, a plain GPT-4 chat interface, or a tutor version designed to give hints and not answers. While AI was available, students did much better on the practice problems, 48% better with the plain interface and 127% better with the tutor version.³ That 127% is itself more than a doubling and is another place a "twice" can come from, but it measures practice with help, a different thing from what students could do afterwards.³ Then the AI was taken away and students sat an exam alone. The plain-interface group scored 17% lower than the group that never had AI, a statistically significant drop.³

The paper puts the guarded tutor this way: "this negative effect is essentially eradicated in the GPT Tutor arm, though we still do not observe a positive effect."³ The authors report that students often used the plain interface as a crutch, asking for and copying solutions, and that students did not appear to perceive any loss in their own learning.³ The study is short-term by the authors' own account, and they say other tutor designs and settings need testing.³

Put next to the Harvard result, this is the most important finding in the pile. The same underlying kind of model produced a learning loss in one design and no loss in another. The Harvard tutor beat a classroom, and the Turkish tutor's exam scores were statistically indistinguishable from those of a control group that had course books and notes.³ The two results are compatible. They are different comparisons with different controls, and the word "tutor" hides that.

A six-week after-school program in Nigeria. De Simone and colleagues at the World Bank ran a randomized trial in nine public schools in Benin City.⁴ Students who expressed interest, about 52% of those eligible, were randomly assigned either to attend twelve 90-minute sessions over six weeks in computer labs, using GPT-4-based tutoring on curriculum-aligned English activities guided by teachers, or to continue their regular classroom learning.⁴ The treatment group scored 0.23 standard deviations higher on English, the main outcome, after six weeks.⁴ The overall assessment gave 0.31, but it also covered AI knowledge and digital skills, which the program taught and the comparison group was not taught, so I use the English figure as the learning estimate.⁴ The comparison is with no program, which is a much lower bar than an active-learning class, and the setting is a low-resource school system with a large learning gap, which may be where a structured tutor has the most to add.

Three details matter here. The paper says 657 students were assigned to treatment and 671 to control, but only 422 and 337 completed the final assessment.⁴ The authors note that the difference in dropout between the two groups is significant, so they report Lee bounds, a method that estimates the range of the effect under that kind of attrition, and say the effects stay positive and significant.⁴ That is the right thing to do, and the result survives it. Two further points follow. Fewer control students stayed to the end (about 50%, against about 64% of the treatment group).⁴ And only students who expressed interest were in the trial at all, so a class where nobody opted in is a different population. The paper is also a working paper and not yet peer reviewed, and it names the tool as ChatGPT-4 in the abstract and as Microsoft Copilot powered by ChatGPT-4 in the body.⁴ The two descriptions may refer to the same model, and I would not quote the product name from memory.

A vendor report. Macmillan Learning's page on its AI Tutor, last updated April 2026, reports on more than 1,200 students across 36 institutions and says outcomes improved most when the tutor was used on 25 to 75 percent of homework questions, with exam score increases of up to 10 percent for students who wrote to it in their own words.⁵ I could read the summary page but not the full report, and the page does not state a design. Without one, those comparisons could simply reflect which students chose to use the tool in which way. That is a limit of what I could verify: this page gives a number and leaves the method one click further away. I list it because people will hand it to you, and I treat it as the company's account of its own product.

06The evidence at a glance

07What the evidence supports and what it does not

08How sure I am

My confidence depends on which sentence you want to rely on. For the narrow one, that a carefully built and instructor-scripted AI tutor can produce higher immediate scores than a good active-learning class on first exposure to new material, I would say moderate. It rests on one well-designed randomized trial with a large effect, no declared conflicts and published data, and the Nigerian trial points the same way, though it answers a different question and I do not count it as support for this comparison. I stop short of high because there is one trial of this comparison, in one course, with two lessons and no delayed test.

For the broad sentence, that AI tutoring doubles learning, I would say low. The word "twice" comes from a ratio of median gains, and the effect-size range around it is wide. Nothing here shows what happens to the advantage over weeks, against a weaker classroom, for a subject that is not introductory physics, or with a tutor nobody scripted. The Turkish experiment adds a warning: in the one trial that tested an unguarded chat interface, exam performance fell once the tool was removed.

What would change my mind: a replication of the Harvard comparison in a second course and institution; a delayed post-test showing the advantage persists; a trial that pits a scripted tutor against a classroom taught by an average teacher; and a trial that measures what happens when students use an unguarded chatbot alongside a structured one.

09What I would do with this

If you sit on a procurement committee, or you are the person who forwards a vendor's demo link to the people who do, there are three questions this evidence makes fair. What was the comparison: no instruction, a lecture or an active-learning class? When was learning measured, and was there a test after the tool was taken away? Who wrote the lessons the tutor follows? The first two decide whether a vendor's number can be compared with the Harvard one at all. The third decides who pays for the quality, which is the next point.

If you teach, the Harvard tutor ran on instructor-written, step-by-step solutions for every problem, and the paper's authors tie the result to that structure.² I did not find the paper reporting how many hours that took. For a part-timer with four sections or a seven-week term, that cost may settle the question before any tool does. It is also possible that the structure carries much of the effect, since the trial had no scripted non-AI control. The Turkish study adds a caution: its hint-giving tutor removed the harm but showed no gain, and the students did not appear to notice their own loss.³ A satisfaction survey would not have caught it. A short unaided quiz after the tool is gone would.

If you write policy for a program or an institution, treat any sentence that begins "studies show" and ends in a multiplier as a prompt to ask what was compared with what, and when.

10Method

An AI research assistant working under my direction ran the search and read the sources for this piece, and I review and publish it; the site's about page describes that arrangement. On November 1, 2026 it ran a general web search for AI tutor trials and edtech claims, then followed the claim back to the Harvard Gazette story and the Scientific Reports paper. It read the full text of the Harvard, Turkish and Nigerian papers through their open-access versions, and the Gazette story and the Macmillan summary page directly. It did not open Macmillan's full report. The DOI of each journal article was looked up on Crossref and its retraction status checked on OpenAlex, and a script checked every passage in quotation marks against its source. Those quotations come from the Gazette, the Harvard paper and the Turkish paper; the Nigerian paper and the Macmillan page carry none, so their figures were checked by reading and not by script. The results are printed below the source list.

This is a review of one claim. The search was a web search followed by tracing the claim to its origin, so it is not reproducible the way a database search would be, and I did not use ERIC or Google Scholar. The largest gap is that I did not read the meta-analyses of earlier intelligent tutoring systems, which are the proper benchmark for any tutor-versus-classroom claim. On what is in and out: the Nigerian working paper is in because it is a randomized trial that reports its own attrition problem, and the Macmillan page is in because readers will be handed it, flagged as the company's account. I left out a 2026 master's thesis that reported no effect in observational data, because I did not read it in full and cannot judge it, so I cannot say whether it would change anything. A preprint of the Harvard study also appears to have been posted before peer review. I did not read it or confirm its date, so I treat the Gazette as the public origin of the number only. Studies I did not find may change the picture, and if you know of one, write to me and I will add it with a note.