Look down the list you made last week. If you ran the exercise from chapter one, you learned something new with an artificial intelligence (AI), closed the laptop, and wrote down what fell out of your head once the machine was gone. Arriving cold, that is orientation enough, and the exercise keeps: the list is a record of the distance between feeling like you understood a thing and being able to reproduce it without help.
Here is my bet about that list. The broad shape of the thing probably stayed with you. What slipped is the specific stuff: the exact parameter, the particular exception, the one step that only matters in your corner of the field. Hold onto that pattern, because by the end of this chapter you will know why the machine made those details so easy to feel sure about, and why they are the first thing to fail you when it counts. To get there we need an honest picture of what the machine is, since you cannot teach with a tool you have only reacted to.
Trust and fear are the same reflex
Almost every strong opinion I hear about these tools turns out to be one of two reflexes, and the part that took me a while to see is that they are the same reflex in different clothes. One person treats the machine as an authority. It answers instantly, in whole sentences, on any topic you name, so it earns the trust you would hand a reference book. I understand the pull: anything that fluent and that widely read has plainly absorbed more than any of us could read in ten lifetimes, and it would be strange not to respect that.
Another person does the opposite and treats the machine as a black box, unknowable and faintly dangerous, best kept at arm’s length until someone smarter has vetted it. That has real sense in it too. You are responsible for other people’s understanding, you did not build this thing, nobody has shown you its workings, and caution in front of a tool you cannot see inside is the same good caution that stops a careful teacher from repeating a claim they cannot source.
What the two reflexes share is worth noticing. Both are postures you reach for when you have no working model of the machine, and a posture is just what fills the space where understanding should be. Hand it your full trust or keep it at arm’s length, and either way you are let off the harder hook: knowing the tool well enough to say where it earns its keep and where it will quietly hurt you. This chapter builds that model, because once you have it the authority and the black box both dissolve into something more useful: a tool with a shape you can reason about.
A working model you can actually use
Start with what the machine is reaching for when you ask it something, because everything else follows from this. It is not retrieving a fact from a store, and it is not reasoning its way toward an answer the way you would. It is predicting what an answer to your question tends to look like. It has read an enormous amount of human writing and learned, in fine statistical detail, the texture of how we lay words next to each other. Ask it a question and it assembles the shape of a plausible answer, one likely piece at a time, drawn from that texture.
Sit with what that means, because it is the whole thing. The machine carries a model of what true-sounding language looks like, which is a different thing wearing the same clothes as a model of truth. Most of the time the two line up, because accurate statements are well represented in what people have written, so the plausible-sounding answer is very often the correct one. That overlap is why the tool is useful at all. When they come apart, though, nothing in the answer flags the difference, because plausibility is all it was ever reaching for. Fluency is the only dial it has, and the dial reads the same whether the machine is on solid ground or inventing.
This is why it can be wrong with the exact same confidence it is right. A person who is unsure usually sounds unsure: their voice drops, they hedge, they tell you they would have to check. That signal does real work for you, and I will come back to it. The machine has no equivalent. Its most solid sentence and its most invented sentence come out of the same process, in the same steady, articulate voice. Calling it a liar misses what is happening, because lying takes knowing the truth and choosing against it, and the machine knows neither. It is doing the only thing it does, with no way to tell a real answer from a fabricated one.
Thick in the middle, thin at the edges
Now the part that matters most for teaching, and the reason I had you look at your list. The machine’s reliability comes from how well-represented an answer is in everything it read, so that reliability is uneven. It runs thick where human writing is thick and thin where it is thin. Ask about something common and well-trodden, written about ten thousand times over, and it has an enormous, mostly-agreeing pile of text to draw from, and it is genuinely reliable. Ask about something niche, recent, or specific to your corner of a field, and the pile thins out. The evidence thins. The machine’s confidence does not. It hands you an answer of the same confident shape, now assembled from far less, with much more of it guessed.
I think of a tour guide I once had in a city I did not know. On the main square he was flawless, every fact worn smooth by a thousand tellings. Three streets past where the tours go, he did not slow down, did not say he was unsure, did not break character for a second. He kept narrating in the same warm, certain voice, and I learned later that a good part of what he told me out there was invented. The tourists could not tell, because the delivery never changed. The machine is that guide. Downtown it is superb. Out at the edges it is exactly as confident and no longer correct, and it never lowers its voice to tell you which street you are on.
Teachers lean on something we rarely name out loud: calibration, the match between how sure a source sounds and how likely it is to be right. A good textbook hedges where the field is unsettled. A good mentor says I am not certain, let me check. That correspondence between confidence and correctness is information, and learners use it constantly to know when to relax and when to verify. Researchers have studied it for decades, and the short version is that well-calibrated confidence is one of the quiet things that makes a source worth trusting. The machine is radically uncalibrated. Its confidence is flat. It sounds equally sure across the whole map, so the one signal your learners have relied on their whole lives to gauge trust has been disconnected without their knowing it.
Here is the inversion, and it is the thing to carry out of this chapter. The machine is most reliable exactly where you need it least, on the common ground you could have covered from any decent source, and least reliable exactly where you are reaching, out at the specific and the novel edge of what you know. Reaching is where learning happens. Real learning tends to sit just past what a person can already do, an idea a later chapter will name properly, and that edge is the machine’s thin territory. So the tool is most confident and least correct in precisely the zone where your learner is least equipped to catch the error, because if they could catch it, they would not be learning it. Reliable where you least need it, unreliable exactly where you are stretching: that is the trust boundary the rest of this book keeps circling back to.
Ask it for a hook that does not exist
I want to be honest about where this bites me, in the one domain where I can catch it in seconds and most people never could. I have built software for about thirty years and shipped a good number of WordPress plugins, so that platform’s internals are home ground. Ask one of these tools for a specific function or hook, and a fair share of the time it hands you one that is perfectly named, perfectly plausible, wrapped in a confident code sample, and completely fictional. The function does not exist. Or it existed a few versions back and was removed. Or it is real but the arguments are subtly wrong in a way that only fails in the exact case you cared about.
The tell, when there is one, is faint enough that only the ground under my own feet gives it away. I catch these fast, but only because this corner is where my knowledge runs thickest and the machine’s runs thinnest, and the two meet right at the spot where I can feel the difference. I keep a public page on how I use and disclose these tools, at my stances, and this is part of why.
Now move that one field over, into a subject where you are the expert and your learner is not. The machine invents with the same fluency there. The difference is that your learner is standing where you were, out at the thin edge, without your thirty years to feel the floor give way. They receive a confident, well-formed, fictional answer and no way to know it is fiction, because the tell they would lean on, a drop in the machine’s certainty where the ground goes thin, is a signal it does not reliably give. And when it does hedge, the hedge is not hooked to whether it is right. That is not an argument for banning the tool. It is an argument for knowing precisely where its map goes blank, so you can teach into that gap on purpose instead of being ambushed by it.
Try this before next week
Do not take the inversion on my word. Put it in your own hands this week, because a trust boundary you have only read about will not change how you teach. This one takes about fifteen minutes, and it works best inside your own field, where you are the one who can judge the answers.
Open an AI tool and ask it two questions from your subject. Make the first one dead centre: the kind of thing a good textbook has covered a thousand times, the question you could answer in your sleep, the settled common ground of your field. Make the second one genuinely specific: a niche case, a recent development, a local particular, an edge that only surfaces when you actually do the work. Choose a second question whose answer you can check for yourself rather than only feel, because the whole exercise rests on your being able to tell right from wrong out there. Before you read either reply, jot one line for each: how confident you expect it to sound, and how accurate you expect it to be. Then ask them in the same session, one straight after the other.
Then read the two answers side by side, but hold off on grading them for a moment and watch the confidence first, against the prediction you wrote. My near-certain guess is that the tone does not move at all. The second answer arrives with the same fluency and the same steady authority as the first, while the accuracy quietly falls off a cliff. Now grade them, and mark where the specific answer went soft, invented a detail, or stated something plainly false in a clean, confident sentence. If it happens to come back correct, that does not undo the lesson. Notice that nothing in its tone told you it was safe, that you know only because you could check, then push the question further out, somewhere more recent or more local, until the ground goes thin enough to feel the floor give way. That marked-up pair, and the gap between what you predicted and what you found, is the whole chapter made concrete. Keep it near you: you have just watched, in your own subject, the tool be most convincing exactly when it became least trustworthy, which is the thing your learners cannot yet see for themselves.
What comes next
So now you have a working model. The machine predicts the shape of a plausible answer with no notion of whether it is true, it is reliable across the thick common ground and unreliable out at the specific edge, and its confidence never tells you which of the two you are on. That is enough to reason honestly about where the tool belongs in a lesson and where it does not.
There is a harder problem waiting, and it has nothing to do with the machine’s wiring. It shifts to the person reading the answer. When the machine hands a learner a smooth explanation, they get more than an answer. They get the feeling of having understood, and that feeling turns up whether or not any learning happened underneath it. Next week, in chapter three, I want to look straight at that: why the feeling of learning and the fact of learning come apart so easily, why AI widens the gap, and what it means that a learner can feel most sure exactly when they have learned the least. Bring the marked-up pair you make this week. The place where the machine stayed confident and stopped being right is the same place your learners will feel they understood and will not have. We are going to need it.
Leave a Reply