Where AI actually goes too far, from someone who defends it

Somewhere today a claim is being denied, and somewhere a loan is being refused. On the other end of each of those is a person who will, sooner or later, ask the only question that matters to them, which is who decided this and why. More and more often the honest answer is that no one did. The claim was scored. The number came in under a line that somebody drew a long time ago and then walked away from. When the person asks to speak to whoever made the call, they are told, in so many words, that “the system decided.”

The chair where that decider should be sitting is empty, and nobody is coming to fill it.

I have spent this series defending the tool that is quietly emptying that chair. I have argued that most fear of Artificial intelligence (AI) is misplaced, that a student using it is usually working rather than cheating, and that an institution banning it owes you the name of the thing it claims to protect. I still believe all of that, which is exactly why I owe you this one. An advocate who cannot name his own lines is just a salesman with a better vocabulary. So here are mine, drawn in good faith, precisely because I defend the tool nearly everywhere else.

The capacity test, and a different kind of value

The earlier pieces turned on one word: capacity. The test I set out in the cornerstone was about what a task is for. If the point of the task is the output, the tool augments you and you should reach for it. If the point is a capacity being built or measured inside a person, the tool can hollow that person out, and you should move with care. Cheating and bans both sit on that axis.

The lines I am drawing here sit on a different one. This is a kind of value that exists only because a specific person supplied it, and it collapses the moment you subtract the person. Two things live in this territory. The first is accountability: someone answerable by name for a decision that landed on you. The second is presence: a person choosing to spend a little of their limited attention on you. Both are worth something for one reason, which is that a human did them.

It is the same test the cornerstone set, aimed at a new target. Ask what the act was for. If the act was only ever worth anything because a named human stood behind it or showed up for it, then a machine standing in for that human hollows the act out while leaving its shape perfectly intact. You get the note, the decision, the sentiment, the sign-off. You just no longer get the person, and the person was the point.

Accountability laundering

Go back to the empty chair. Some decisions carry their whole weight for a single reason, which is that a named person is answerable for them. Who denied the claim. Who rejected the application. Who made the diagnosis, who signed the termination, who handed down the sentence. In cases like these the decision and the accountability are one object. You cannot keep the decision and quietly delete the name and still expect to have the thing you started with.

AI crosses my line the moment it becomes the thing that decided and there is no longer anyone who can be asked why and made to stand behind the answer. The worse version is quieter. A human is still sitting right there but hides behind the tool, using “the system decided” to keep the authority and shed the ownership. That is laundering. You run a human judgment through a machine so it comes out the far side with no fingerprints on it.

Using AI as an input to that decision is a different matter, and a genuinely good one. Let it read the whole file and flag the detail a tired adjuster missed at the end of a long day. A claims examiner who reads the machine’s summary and then owns the call is doing better work than one who never had the summary in front of her. The test states easily and dodges hard. Can a named person be required to answer for this, and stand behind it, in front of the person it landed on? If yes, use every tool you can get your hands on. If the tool has become the answer to “who decided,” you have gone too far.

This is, as it happens, most of what people actually hire me to look at. When I audit an organization’s systems, the question I keep coming back to is who answers for the outcome when it goes wrong. If that reads like a pitch, set it aside; you can ask the question of your own systems this afternoon without me in the room. Walk one real decision backward and see whether it ends at a person or at an empty chair.

Forged presence: synthetic likeness and deepfakes

The condolence note. The apology that cost something to write. The recommendation, the vow, the thank-you a person actually meant. What makes any of these worth having is a single fact sitting behind the words: another person stopped, thought about you in particular, and spent a piece of their short supply of attention setting it down.

Picture the friend who drives two hours to sit with you in a hospital waiting room, next to the friend who sends a gift card from an app on the way past. Both are real kindnesses. Only one of them handed you the thing that was actually valuable, which was the presence itself, the hours they will not get back. A gift card is honest about being a gift card. The trouble starts only when the manufactured thing is dressed up as the handmade one.

So when a machine writes the condolence note and it goes out under a human name, as though that human had sat with the loss and reached for the words, the tool does not augment the sentiment. It manufactures the appearance of one that was never actually spent.

I want to be careful here, because the lazy version of this argument swings at exactly the wrong people. The non-native speaker reaching, in a second language, for words to carry a feeling that is completely real is not forging anything. The exhausted nurse using the tool to shape three sentences she means but cannot find at the end of a double shift has forged nothing either. Using the tool to reach the words for something you actually feel is human and fine. The hard case is subtler than the non-native speaker: it is the passing sympathy the machine swells into a paragraph that reads like a vigil, a feeling that was real but thin, dressed up as attention that was never spent. What the test turns on is a single thing, whether you actually spent the attention the note claims you spent. A person carrying a real feeling to the page has forged nothing. A person letting the machine counterfeit the depth of an attention they never gave has forged the only thing that counts.

Let me name the one that costs me, because it lives in my own trade. The outreach that mentions a detail it scraped about you, generated a thousand at a time and sent as though I had stopped and thought of you in particular, is forgery too, and saying so out loud loses me work. The machine spent the attention. I did not. The note pretends I did. I do not get to exempt my own industry from a line just because the line is inconvenient where I earn.

Deskilling: the point where you can no longer catch the error

This one I will keep short, because I have already made the case underneath it. The cornerstone argued that some skills have to be built by hand first, so that later, when the machine does the work for you, you can still feel it in your gut when the machine is wrong. That felt sense is the whole safeguard.

My own red line runs straight out of it. Any use of the tool that erodes your ability to notice when the tool is wrong has crossed it. Up to that point, the tool augments a person who remains in charge. Past it, the person in the chair is only performing the part of being in charge, nodding along to outputs they can no longer check against anything. The augmentation has quietly carried off the one thing that made it augmentation, which was a human who could look at the answer and say no.

Hallucination: a lie with no one behind it

The tool will sometimes invent a fact: a citation to a case that was never decided, a source that does not exist, a confident and well-formed sentence that is simply false. Every practitioner has watched it happen. On its own that is survivable, because every tool can be wrong. A kitchen scale drifts out of true. A reference book carries its typos.

The line is not that the tool can be wrong. The line is that its wrongness can travel all the way into a court filing, a medical chart, a hiring decision, a published record, with no human who ever checked it and no one who owns the falsehood once it lands. A typo in a book still has an author, an editor, a publisher, a whole chain of people you can point at. The machine’s invented citation, shipped straight through untouched, has none of that. It is a falsehood that no one actually told and no one is answerable for, which is a genuinely new object in the world. The fix for it is old and unglamorous. A person reads the thing before it goes out, and that person owns whatever they passed along.

Where I still want the tool everywhere

I want to be clear about how narrow these four lines are, because I would hate to see them read as a retreat. Away from them, I want the tool in as many hands as I can put it. Draft the report with it, debug the code with it, let it summarize the thousand-page filing, translate the manual, rough out the lesson plan, check your arithmetic, and argue against your thesis until the thesis is harder to knock down. Almost everything is output, and output is exactly where the tool earns its keep.

The four places I keep it out share one feature. In every one of them, a person was the point. Someone had to be answerable by name, or someone had to actually spend their attention, and the machine’s entire trick is to hand you the result while skipping the person. Everywhere else, skipping the toil is the gift. In these four, the toil was the person, and skipping it skips them.

One disclosure, because the first of those lines demands it of me. I drafted this the way I draft most things now, with the tool it discusses open beside me, and I argued with it until every claim here was mine and not its. The attention was mine too, the hours, the lines I moved and put back, the sentences I sat with until they were true. That is the whole essay in one gesture: the machine can hand you the words, but it cannot spend the attention or answer for the claim, and here both are mine. Then I signed it. A piece about accountability had better have a name attached, and a piece about presence had better have had some paid. This one has both.

What you would have to believe

If you think I have put one of these lines in the wrong place, I can tell you exactly what you would have to believe, and I would honestly like to hear you make the case. You would have to believe that accountability can survive with no one able to answer for the decision. That a condolence carries the same weight whether or not a person spent anything to send it. That someone can be de-skilled past the point of catching the error and still be the one in charge. That a filing needs no human who owns what is inside it. I might be wrong about where any single line belongs. What I will not do, as the tool’s advocate, is pretend an advocate has no lines at all. Tell me which one you would move, and why, and bring your reasoning with you. I will answer for mine.