A while back I nearly repeated a statistic I had not checked. It was one of those figures that travels well: Artificial intelligence (AI) tutoring produces 54 percent better results. Tidy, quotable, the kind of number you forward to a colleague without reading past the headline. It flattered the tool, and it flattered me for noticing it, so I almost let it stand.
Then I went looking for where it came from. I could not put my hands on the source. What I could put my hands on was a different 54 percent, and it pointed close to the opposite way. In a poll run by NPR and Ipsos, 54 percent of K-12 teachers said AI is making it harder for students to learn to think critically. Same number, aimed in the other direction. I had been about to lead with the hopeful version for one reason: it was the one that made the technology look good.
That small moment is worth sitting with, because it holds the whole argument I want to make about learning and working with AI. The version that flatters the tool is almost always the version that has done the least checking. And the checking is not a chore you do before the real work. In learning, the checking is the real work.
Why the flattering number is the one to check
Think about the scale in your bathroom. If you own two of them and one reads five pounds lighter, you know exactly which one you will trust on a Monday morning. Not the accurate one. The kind one. Wanting a number to be true is a powerful thing, and it has nothing to do with whether the number is any good.
That is what makes the flattering statistic dangerous. It arrives pre-approved by your own hopes. You are not inclined to test it, because testing it might take the good feeling away. The 54 percent that says AI tutoring works better slides straight past your guard. The 54 percent that says teachers are worried gets a raised eyebrow and a demand for a source, if it gets shared at all.
So here is a rule of thumb that has held up for me: when a claim about a tool makes the tool look wonderful and makes you feel clever for repeating it, that is precisely the claim to slow down on. Not because it is always wrong. Because your incentive to check it just quietly dropped to zero, and a claim nobody checks is a claim nobody should be leaning on.
The verified figure, the one I can actually source, is not a headline about a single bad week in education. It is a signal worth keeping. When more than half the teachers in a serious poll tell you the tool is making it harder for students to think, the useful response is not to argue with the teachers or to defend the software. It is to ask what those teachers are seeing up close that the glossy adoption numbers are not built to show.
The tool was never the danger
I have spent thirty years building with new tools, and a few of those years studying how people actually learn. I am not here to tell you to put AI down. It is genuinely useful, and pretending otherwise is its own kind of dishonesty.
The danger was never the tool. The cost shows up when the tool gets used with nobody’s judgment left in the loop.
You have seen this outside of school, on the road. Every year a few drivers follow a mapping app straight into a field, a closed road, or the shallow end of a boat ramp, because the screen sounded certain and they stopped looking through the windshield. The app was not the problem. The problem was treating a confident output as the end of thinking instead of the start of it. The drivers who never end up in the field are not the ones with a better app. They are the ones who kept one eye on the road the whole time.
AI in learning works the same way. A model will hand a student a finished paragraph, a solved equation, a clean summary, and it will do it with total confidence and no idea whether it is right. If the student takes that output as settled fact, the tool has not helped them learn. It has done their thinking for them and left them a little less able to do it next time. There is a real concern in the field that learners are starting to treat whatever the machine says as true by default, and that habit is quiet, easy, and expensive.
The learners who come out ahead are the ones who still do the checking. They take the model’s answer as a first draft to be tested, not a verdict to be copied. That is a skill, and it can be taught. I wrote more about the two ways this goes wrong, and how to keep judgment in the loop, in the two ways teaching with AI goes wrong.
Make the checking the lesson
Here is the shift that matters most, and it is aimed at anyone who teaches or trains: the checking is not the boring part you do before the learning. The checking is the learning.
When a student catches the model in a confident mistake, that is not a delay in the lesson. That is the lesson. They have just practised the exact skill the whole thing is supposed to build, which is the ability to look at a plausible answer and ask whether it is actually right. A teacher who makes that catch the centre of the class, rather than something to prevent, is teaching the one skill that will still be scarce in ten years.
You can see the market already pricing this in. The pattern in the workforce reports is consistent: senior leaders expect AI to redraw roles rather than erase them, and the premium is moving toward people who can use the tool and still bring something it cannot. That something is judgment. It is the ability to look at an output and know whether to trust it, ship it, or throw it out. Schools are adopting these tools quickly, and the demand climbing right alongside adoption is not for more software. It is for help using it without the learning draining out.
If you are shaping policy, watch for one thing above all the others. Ask whether a rule requires a human to check the work, or whether it only names who is allowed to use the machine. Only one of those protects the learning. A policy that says “students may use AI” and stops there has answered the easy question and skipped the hard one.
Keep the part that was yours to keep
The thing you cannot hand over is your own judgment. Not because the tool is too weak to be trusted with it, but because judgment is the muscle the whole exercise is meant to build. Outsource it entirely and you have not saved time on the learning. You have skipped it.
So use the tools. Use them for the parts they are good at, which is more parts than the skeptics admit. But keep doing the checking, and if you teach, make the checking the thing you are actually teaching. When a number, or a paragraph, or a solved problem shows up looking a little too good, that is your cue to lean in, not to relax. The habit of leaning in is the part that was always yours to keep.
If you want help building that habit into how your team or your classroom works with AI, that is a lot of what I do in my learning and training work. A shorter, week-by-week version of this thinking runs in my newsletter if you would rather take it in small doses. Either way, the line I keep coming back to is a simple one: keep the human in it.
Leave a reply