Episode 5: the test that was lying to me
In this episode
Christopher deleted his own bug fix on purpose, put a live bug back into a shopping cart, and ran the test to watch it catch him. The test said everything was fine. Frances would have shipped it, and the five paranoid minutes Christopher almost skipped are the only reason real customers never paid for that green check-mark.
Takeaway: A test you can’t fail is a story you’re telling yourself.
The artifact: The Delete-the-Fix Test Check. You can grab it from the blog.
Full transcript: episode 5
CHRISTOPHER: This week I deleted my own bug fix on purpose, put a live bug back into a shopping cart, and ran the test to watch it catch me red-handed. The test said everything was fine. Frances would have shipped it without blinking. Green means go, she’s built her whole business on green means go. The five paranoid minutes I almost skipped are the only reason real customers never paid for that “fine.”
FRANCES: This is Sites I’ve Never Seen, the show where Christopher and I open up the work behind the work: the tests and the green check-marks, and the five paranoid minutes that quietly save somebody’s Saturday. I’m Frances. I run my own shop, and when a test comes back green, I ship it and I sleep. Which is exactly why what Christopher did this week makes no sense to me, and he is going to sit here and explain it.
FRANCES: Green means go. I have built an entire business on green means go. You see the green and you ship. The invoice clears and you sleep. So when you tell me you saw green and then you went back into your own work and deleted the thing that made it green, on purpose, I need you to walk me very slowly through why a grown man does that.
CHRISTOPHER: Because the first four seconds felt amazing, Frances. That’s the honest answer. Four seconds of feeling like a genius.
FRANCES: And then?
CHRISTOPHER: And then I reached back in and pulled the fix out. The actual fix. Just took it out of the code. So now the bug’s back, fully live, doing exactly the thing I’d written the fix to stop. And I ran the test again to watch it catch the bug.
FRANCES: To watch it fail.
CHRISTOPHER: To watch it fail. That was the whole point. The test fails, I learn the test works, I put the fix back, everybody goes home.
FRANCES: So it failed.
CHRISTOPHER: It went green.
FRANCES: Hold on. Green with the fix?
CHRISTOPHER: Green with the fix. Green with the fix gone. It was going to be green no matter what I did to that code.
FRANCES: Okay, but you’re skipping the part I’d have lived in. With the fix gone, the bug is real again, yes? Like, a customer-touches-it real?
CHRISTOPHER: A customer-touches-it real. This is a piece of WooCommerce, the shopping-cart software running a huge slice of small online stores. Real carts. Real money.
FRANCES: And the test looked at all of that and said, this is fine.
CHRISTOPHER: It looked at all of that and said, this is fine, carry on. With a little badge on it that said verified. And I want to be careful here, because the test looked perfectly healthy. It ran and it passed, and by every surface sign it was doing its job.
FRANCES: It just wasn’t doing it.
CHRISTOPHER: It just wasn’t doing it. And the only reason I know that is the paranoid five minutes I almost skipped.
FRANCES: See, this is where you and I are going to fight, Chris. Because I’d have skipped it.
CHRISTOPHER: I figured.
FRANCES: No, I want to actually defend that, because I don’t think it’s the dumb position. Walk it through with me. I’ve got a developer. I’m paying him. He hands me a fix, and he hands me a test, and the test is green. What exactly am I buying when I pay him to then spend an afternoon trying to break his own green light?
CHRISTOPHER: You’re buying the thing the green light is supposed to mean and might not.
FRANCES: That’s a sentence. Give me the number. Because from where I sit, you’ve got a working fix, you’ve got a passing test, and you’re telling me there’s a third invoice on top of those two, for a feeling.
CHRISTOPHER: It’s not a feeling, it’s… okay. Let me put it the way it actually went. The fix was real. The fix worked. If I’d shipped the fix without the test, the bug would’ve been gone and nobody would’ve known the difference.
FRANCES: So ship it.
CHRISTOPHER: But I didn’t ship just the fix. I shipped the fix and a test, and the test is a promise. The test says, to the next person who touches this code, six months from now, when I’m not in the room, it says this is checked, you’re safe here, don’t worry about this part. And that promise was a lie.
FRANCES: Whose lie, though? You didn’t mean it.
CHRISTOPHER: Nobody means it. That’s what makes it dangerous. The test passes because I knew the answer when I wrote it. I had the whole thing in my head, how it all fits, and that knowledge sort of leaked into the test. So it checks something I already knew would hold, instead of the thing that could actually go wrong.
FRANCES: Okay, that I’ll grant you as a craft point. It’s a pretty one, even. But I run a shop, so let me drag it back onto my desk. Pretend that lying test ships. What does it actually cost me? Walk it to a real invoice or I’m not buying.
CHRISTOPHER: All right. There’s a shop owner in this story. I’ll call her what she is, she’s the person who runs the store this cart lives on. Let’s say she’s selling something on a Saturday, busiest day, discount code running. Call the code SUMMER20, twenty percent off, she’s been pushing it in her newsletter all week. And there’s the bug, the one the fix was for, sitting live, because the fix got refactored six months later and the test that was supposed to guard it waved it through.
FRANCES: Because the test never actually checked it.
CHRISTOPHER: Right. So here’s the specific failure. The bug is in how the cart recalculates after the coupon’s applied. Customer puts two things in the cart, applies SUMMER20, removes one item, and… right, and the discount doesn’t recalculate. It keeps the twenty percent off the original two-item total. So she’s now selling one item at a discount that’s bigger than the item. Some carts it under-charges. A few it double-charges, because the totals desync and the second pass tries to true it up against a number that was already wrong.
FRANCES: So she’s losing money on some orders and over-billing on others, and from the customer’s chair it just looks like the store is broken.
CHRISTOPHER: She doesn’t see code. She sees angry email. Sunday morning she’s got eleven of them and no idea what they have in common.
FRANCES: Eleven. Okay, let me price the ones she can actually see, because that part I know in my bones. The double-charges, those come back as chargebacks, not refunds, because the angry ones don’t ask, they just call the bank. A chargeback is the order amount plus a fee on top. My processor charges fifteen dollars a chargeback whether I win the dispute or not. Fifteen, sometimes twenty-five, depending on which tier they’ve decided I am that month, which is a whole other rant I’ll spare you.
CHRISTOPHER: So the order that double-charged…
FRANCES: costs her the forty dollars she has to give back, plus the fifteen-dollar fee, plus the hour she spends pulling the order to fight it, plus, if enough of them land in one month, the processor flags her account as high-risk and her rate goes up on every order after that. That’s the part nobody tells you. One bad Saturday raises the price of every good one for the next quarter.
CHRISTOPHER: And that’s still only the customers who fought.
FRANCES: and the customer who just quietly buys from someone else next time and never tells her why. That one’s the expensive one. She never even gets the receipt for that.
CHRISTOPHER: Right. That’s the one that never even leaves a mark she can point at.
FRANCES: I had a client like this. Not a software thing, a fulfillment thing, but it’s the same shape. She had a checkbox on her order form, “yes, ship to the billing address,” and it defaulted checked, and for about five weeks it silently ignored whatever they typed in the shipping field. So a chunk of her orders went to the wrong city. The ones who phoned, she made whole, reship, eat the postage, throw in a gift card, fine, that’s a known cost, I can add that up. Came to maybe eight hundred dollars over the five weeks.
CHRISTOPHER: And the ones who didn’t phone.
FRANCES: The ones who didn’t phone just decided she was the company that sends your order to the wrong house. And they told their sister. I will never see that number. But I watched her repeat-rate that quarter and it was down, and there is no invoice anywhere that says “the checkbox did this.” The eight hundred I could see was the cheap part.
CHRISTOPHER: That’s exactly it. And here’s the thing that connects your checkbox to my dead test, somewhere upstream of both there was probably a test, or a QA pass, or somebody’s afternoon, that was supposed to catch it and signed off instead.
FRANCES: Okay. Hang on. That last one. The silent one.
CHRISTOPHER: That’s the one that compounds.
FRANCES: No, you don’t get to say compounds, that’s my word, you stay in your lane. Heh. But… yeah. That’s a real number and it doesn’t show up on any statement. That’s the kind of cost I actually lose sleep over, because you can’t put it on a line item, so nobody upstream ever pays for it.
CHRISTOPHER: And here’s the part that I think you’ll hate. The test that was supposed to stop all of that, it cost her twice. Once when she trusted it. And again when it turns out it was never checking the thing it claimed to check.
FRANCES: A bill you pay twice. Okay, that’s annoying, because it’s good. Say more about why it passes when there’s no fix. Because I still half-think your developer just wrote it wrong and you’re dressing it up as philosophy.
CHRISTOPHER: That’s fair, and sometimes that’s exactly what it is. But the honest version is subtler. A test is a trap you set for one specific problem. The fix is the thing that lets you walk past the trap safely. So the only way to know the trap actually works…
FRANCES: is to take the fix out and see if it snaps.
CHRISTOPHER: Is to take the fix out and see if it snaps. And if you take the fix out and nothing happens, all you built was a picture of a trap. You hung it on the wall and called it done.
FRANCES: A picture of a trap. Hm. So the green light’s just paint.
CHRISTOPHER: The green light’s just paint. And it feels exactly like proof. That’s the whole problem. It feels identical to the real thing right up until the Saturday it doesn’t.
FRANCES: And you only catch it by deliberately breaking your own work.
CHRISTOPHER: By deliberately breaking my own work while there’s nothing riding on it. Which, I’ll admit, looks completely insane from the outside. It looks like I don’t trust myself.
FRANCES: I mean… do you?
CHRISTOPHER: Not on a green light I haven’t tried to kill. No.
FRANCES: All right. I’m going to be straight with you, because I think you can take it. Right up until the shop owner, I had this filed as over-engineering. The paranoid afternoon. Senior-guy theatre. The thing you do so the invoice feels earned.
CHRISTOPHER: I know you did. I could hear it.
FRANCES: But it was the same work either way. That’s what I keep snagging on. Same hours, same effort, and then you turned around and tested whether the test was even awake. And it wasn’t.
CHRISTOPHER: That’s the part I’d actually defend to the wall. The fix was fine. The risk was all in the test, because the test is the part everyone downstream trusts without ever looking at it again. I’d already watched the code work with my own eyes. So the thing I was really checking was my proof.
FRANCES: Checking your proof.
CHRISTOPHER: Right. And it looks exactly like proof, in a font that looks official. That’s what makes it so easy to trust.
FRANCES: And nobody downstream ever opens it back up to look.
CHRISTOPHER: Nobody ever does.
FRANCES: And that story ships as a recall. Because that’s where it lands for me, Chris, that’s the cost I’d been pricing wrong. That dead test doesn’t just sit there being harmless. It’s the thing that lets the bug out wearing a badge that says somebody checked. The recall, the refunds, the Saturday, your green light signed off on all of it.
CHRISTOPHER: That’s exactly the translation I couldn’t make. I had the craft half. You just put the comma in the number.
FRANCES: So let me revise on the record, because I started this episode wrong and I’d rather say so than let it sit. That afternoon you spend deleting your own fix, I’d have called it wasted hours. I’d have priced it at zero, maybe negative, because it’s slowing down a ship that’s ready to go.
CHRISTOPHER: And now?
FRANCES: Now I think I had it backwards. You’re spending one afternoon against a number with a comma in it. I had it backwards.
CHRISTOPHER: I want to land on something, though, before that turns into a tidy little rule, because… let me back up, that came out like I’ve got it all sorted and I don’t. The habit itself is easy. That’s the thing I keep telling people. You write a test, you delete the fix, you run it again. If it still passes, the test guards nothing. One sentence. You could read it to your developer over the phone.
FRANCES: So why doesn’t everybody do it?
CHRISTOPHER: Because the part that matters is the hard part: the instinct that makes you suspicious of a green light in the first place. The checklist tells you to delete the fix. It does not tell you which green light to distrust. Out of forty passing tests, which one is the one quietly lying to you.
FRANCES: That’s the part you can’t hand somebody on a page.
CHRISTOPHER: That’s the part that took me nineteen years and a real number of burns to grow. The page is free. The wariness next to it, that’s the thing you’re actually hiring. And I’d be lying if I pretended the worksheet was the whole answer. It’s the part you can do today. The suspicion is the part you pay for.
FRANCES: This is the same fight we had in the very first one, you realize. The forty findings.
CHRISTOPHER: Go on.
FRANCES: Episode one. You came in proud of a forty-item audit, and I was the one going, who on earth is going to fix forty things, give me three. I was pricing your thoroughness as theatre, you backed down to three, and I felt very clever about it.
CHRISTOPHER: You did look pleased with yourself.
FRANCES: I was insufferable. But look where we ended up. Today it’s forty passing tests and I’m the one who’d have trusted all forty and shipped. Two weeks ago I didn’t want your forty findings. Today I’d have bought forty green lights without checking one of them. That’s me on the wrong side of the exact same wall, and you standing in the same spot you were standing in then.
CHRISTOPHER: Huh. Hang on, let me actually sit in that, because I want to make sure it’s true and not just a tidy rhyme. In episode one the forty things were findings, problems, named out loud, and you said nobody fixes forty problems, give me the three that matter. And you were right, by the way. That was a good call.
FRANCES: It was a good call.
CHRISTOPHER: But a finding is a thing that’s announcing itself. It’s on the page, in red. The work there is choosing. Today the forty things are tests, and a test announces the absence of a problem. Forty little green lights all saying nothing’s wrong here.
FRANCES: Which is so much worse, when you say it that way.
CHRISTOPHER: It’s the opposite failure mode. In episode one the danger was drowning in things that were all visibly true. Today the danger is one of forty things that’s quietly false and looks identical to the thirty-nine that are fine. You can’t pick the three that matter, because the one that matters is hiding by looking exactly like a non-problem.
FRANCES: So you didn’t just flip sides. The thing you’re distrusting flipped, too.
CHRISTOPHER: That’s the part I hadn’t lined up. You’re right that it’s the same green light both times. You just trusted it from the other direction this time, and the other direction is the one that bites, because nobody audits the things that look fine.
FRANCES: Which is humbling, so I’m going to need you to not enjoy it.
CHRISTOPHER: I would never. No, here’s what I actually want to say, and it’s the thing I didn’t understand walking in. Walking in, I thought the discipline was about the code being right. And the code was right the whole time. The real discipline is holding off on believing my own proof until I’ve tried to make it fail. That’s the part I didn’t have words for until just now.
FRANCES: And here’s mine, because I owe you a clean one. The afternoon is the premium on the insurance, and the premium is a rounding error against the claim. I have never once in my life found insurance where the premium was this much smaller than the thing it covers. If a broker offered me that ratio I’d assume it was a scam.
CHRISTOPHER: Neither one of us could’ve said the other’s sentence.
FRANCES: Good. Then we both did our jobs. So the thing for anybody listening is small, and you don’t need a single line of code to use it. After somebody builds you a test, ask them to delete the fix and run it again. If it still passes, it’s guarding nothing.
CHRISTOPHER: And you’ll want to make somebody try to break it on a quiet afternoon. Not the loud one. The loud one tests it for you whether you’re ready or not.
FRANCES: The loud one being the Saturday.
CHRISTOPHER: The Saturday. And there’s a one-page version of that habit. We called it The Delete-the-Fix Test Check, and it’s short on purpose. You really could read it to your developer over the phone.
FRANCES: Make him do it while you’re on the line. Don’t trust him to do it after you hang up.
CHRISTOPHER: She’s not wrong. This is Sites I’ve Never Seen. Glad you spent the half-hour at the bench with us.