The number that flatters the tool is the one to distrust

August 16, 2026

A while back I nearly repeated a statistic I had not checked. It was one of those figures that travels well: AI tutoring produces 54 percent better results. Tidy, quotable, the kind of number you forward to a colleague without reading past the headline. It flattered the tool, and it flattered me for noticing it, so I almost let it stand.

Then I went looking for where it came from. I could not put my hands on the source. A different 54 percent was within reach, and it pointed close to the opposite way. In a poll run by NPR and Ipsos, 54 percent of K-12 teachers said AI is making it harder for students to learn to think critically. Same number, aimed in the other direction. I had been about to lead with the hopeful version for one reason: it was the one that made the technology look good.

Two claims about AI in education, both reporting 54 percent, set side by side. On the left, the claim that AI tutoring produces 54 percent better results, with the provenance line reading that no source could be found for it. On the right, a poll by NPR and Ipsos in which 54 percent of K-12 teachers said AI is making it harder for students to learn to think critically, with its source named. The two figures are identical and point in opposite directions, and only the second one can be traced back to anything. The panel beneath states the point: the flattering version is the one that had the least checking done to it. — The two 54 percents side by side. The one that flattered the tool traces back to nothing.

That small moment holds the whole argument I want to make about learning and working with AI. The version that flatters the tool can be the version that has done the least checking. And in learning, the checking is the real work.

Why the flattering AI statistic is the one to check

Think about the scale in your bathroom. If you own two of them and one reads five pounds lighter, you know exactly which one you will trust on a Monday morning. The kind one, never the accurate one. Wanting a number to be true is a powerful thing, and it has nothing to do with whether the number is any good.

When a claim about a tool makes the tool look wonderful and makes you feel clever for repeating it, that is precisely the claim to slow down on.

That is what makes the flattering statistic dangerous. It arrives pre-approved by your own hopes. You are in no hurry to test it, because testing it might take the good feeling away. The 54 percent that says AI tutoring works better slides straight past your guard. The 54 percent that says teachers are worried gets a raised eyebrow and a demand for a source, if it gets shared at all.

So here is a rule of thumb that has held up for me: when a claim about a tool makes the tool look wonderful and makes you feel clever for repeating it, that is precisely the claim to slow down on. It may well be right. The trouble is that your incentive to check it just quietly dropped to zero, and a claim nobody checks is a claim nobody should be leaning on.

What the sourced figure measures

The verified figure, the one I can source, is a signal worth keeping rather than a headline about a single bad week in education. When more than half the teachers in a serious poll tell you the tool is making it harder for students to think, the useful response is to ask what those teachers are seeing up close that the glossy adoption numbers were never built to show.

The poll, reported by NPR, surveyed 545 K-12 teachers between April 27 and May 5, 2026. It tells you what teachers believe they are seeing in their classrooms, and it measures nothing about what students learned.

The tutoring side has a real study behind it too. Kestin and colleagues ran a randomized controlled experiment in a large Harvard physics course with 194 students. The median learning gains in the AI-tutored group were over double those of the in-class active learning group. The authors say their approach was applied where students were meeting the material for the first time, so read it as a result about one carefully built tutor in one course.

Why the AI tool was never the danger

I have spent thirty years building with new tools, and a few of those years studying how people actually learn. Keep using AI. It is genuinely useful, and pretending otherwise is its own kind of dishonesty.

The cost shows up when the tool gets used with nobody's judgment left in the loop.

You have seen this outside of school, on the road. Drivers follow a mapping app straight into a field or down the shallow end of a boat ramp, because the screen sounded certain and they stopped looking through the windshield. The National Park Service warns about it on its Death Valley page. "Numerous travelers have been directed to the wrong location and dead-end or closed roads by following incorrect GPS information." The problem was treating a confident output as the end of thinking instead of the start of it. The drivers who stay out of the field are the ones who kept one eye on the road the whole time, whatever app they were running.

The same trap in a classroom

AI in learning works the same way. A model will hand a student a finished paragraph, a solved equation, a clean summary, and it will do it with total confidence and no idea whether it is right. Kalai and colleagues describe the mechanism. Language models sometimes guess when uncertain, "producing plausible yet incorrect statements instead of admitting uncertainty." If the student takes that output as settled fact, the tool has done their thinking for them and left them a little less able to do it next time. That habit is quiet and easy, and it can be expensive. Take a field experiment with nearly a thousand high school math students. Those who practised with an unguarded GPT-4 tutor scored 17 percent lower on the exam once access was taken away than students who never had access, while a version built with safeguards largely removed the harm.

The learners who come out ahead are the ones who still do the checking. They take the model's answer as a first draft to be tested. That is a skill, and it can be taught. I wrote more about the two ways this goes wrong, and how to keep judgment in the loop, in the two ways teaching with AI goes wrong.

Why checking the numbers is the real lesson

Here is the shift that matters most, and it is aimed at anyone who teaches or trains: the checking is not the boring part you do before the learning. The checking is the learning.

When a student catches the model in a confident mistake, that is the lesson, right there. They have just practised the exact skill the whole thing is supposed to build, which is the ability to look at a plausible answer and ask whether it is actually right. A teacher who makes that catch the centre of the class, rather than something to prevent, is teaching the skill employers already rank first.

You can see the market already pricing this in. The World Economic Forum's Future of Jobs Report 2025 draws on over 1,000 leading global employers. In it, analytical thinking "remains the most sought-after core skill among employers, with seven out of 10 companies considering it as essential in 2025." That is judgment under another name. It is the ability to look at an output and know whether to trust it or throw it out.

What a policy should require

If you are shaping policy, watch for one thing above all the others. Ask whether a rule requires a human to check the work, or whether it only names who is allowed to use the machine. Only one of those protects the learning. A policy that says "students may use AI" and stops there has answered the easy question and skipped the hard one. In the NPR and Ipsos poll, just one in three teachers (33 percent) said their school has a formal policy on student use of AI.

Keep the part that was yours to keep

The thing you cannot hand over is your own judgment, because judgment is the muscle the whole exercise is meant to build. Outsource it entirely and you have skipped the learning rather than saved time on it.

So use the tools. Use them for the parts they are good at, which is more parts than the skeptics admit. But keep doing the checking, and if you teach, make the checking the thing you are actually teaching. When a number or a paragraph shows up looking a little too good, that is your cue to lean in. The habit of leaning in is the part that was always yours to keep.

What is the last number about a tool that you passed along before you knew where it came from?

If you want help building that habit into how your team or your classroom works with AI, that is a lot of what I do in my learning and training work. A shorter, week-by-week version of this thinking runs in my newsletter if you would rather take it in small doses. Either way, the line is a simple one: keep the human in it.

References

Checked against the Ipsos poll page, the NPR story, the Kestin et al. paper and the Bastani et al. paper, on 2026-10-11.