Newsletter AI Learning, Keeping It Human Call 647-641-0643
Christopher Ross
,

Give them the problem, not the method

Christopher Ross

9 min read
A single, worn hand tool – a small, blunt chisel – lies angled against a larger, unworked block of the same material.
In this post

Turkish high school students were split three ways for their maths practice: no Artificial intelligence (AI) at all, an ordinary GPT-4 chat interface, or a version their teachers had configured to withhold answers and hand out hints instead.

The restricted version won the practice sessions by a wide margin. Students with no AI averaged 28 percent on those problems. The ordinary chat interface lifted that by 48 percent in relative terms. The hint-only version lifted it by 127 percent, taking those students to roughly 64 percent correct.

Then the tools were taken away for an exam.

Students who had used the ordinary interface scored about 17 percent below classmates who never had access at all. Students who had used the hint version scored the same as the ones who never had access. Not better. The same.

Two-panel chart of the Bastani 2025 study. Practice: no AI about 28 percent correct, ordinary GPT-4 about 41 percent (+48 percent), hint-only tutor about 64 percent (+127 percent). Exam with tools removed: ordinary GPT-4 users about 17 percent below students who never had AI; hint-only users the same as them.

Some of you have met this study before. It is Bastani and colleagues in PNAS, and it has come up here twice already (Bastani et al., PNAS, 2025). In the last edition I told you the hint version kept the gains. Going back to the paper this time rather than the summary I linked the first time, that is not what it says. Those students finished level with the ones who never had the tool at all.

So the careful design bought them no learning. What it bought was the absence of harm, which covers the whole distance between a tool that quietly costs your class something and one that does not.

So what is doing the teaching?

Key takeaways

  • Tanmay Sinha and Manu Kapur pooled 53 studies and found people learn more when they attempt a problem before being taught the method.
  • The biggest gains came when instruction built on what learners actually produced, when they attempted in groups, and when the teaching ran as dialogue.
  • Setting an AI tutor to give hints avoided harm. The learning gain came from the order of the lesson.

What the restriction is protecting

The answer is more specific than “let people struggle.”

Tanmay Sinha and Manu Kapur pooled 53 studies and 166 comparisons of two sequences: the problem before the instruction, or the instruction before the problem (Sinha and Kapur, Review of Educational Research, 2021). Problem first came out ahead, at a Hedges’s g of 0.36 on conceptual knowledge and transfer.

Worth knowing what that was measured on. Transfer, meaning whether people can use an idea somewhere other than where they first met it, which is the hardest thing to detect.

Daniel Schwartz and Taylor Martin found the mechanism twenty years ago and gave it an unusually honest name, the hidden efficiency of encouraging original student production (Schwartz and Martin, Cognition and Instruction, 2004). Their ninth-graders invented their own statistical methods first. Most did not arrive at the canonical answer, which looks like waste on any dashboard you could build. They then outperformed the students who had been handed the clean version up front.

The failed attempt builds the structure that the explanation then attaches to.

The three decisions that move the number

The 0.36 headline hides a wide spread, and the part of the paper that rarely gets quoted is the part reporting what drives it. Three implementation choices each roughly double the effect or better. They are reported separately, so do not assume that doing all three multiplies out.

  • Learners attempt in groups: 0.49, against 0.19 working alone.
  • The instruction that follows is built on what the learners actually produced: 0.56, against 0.20 for instruction delivered as prepared.
  • That instruction runs as dialogue: 0.55, against 0.24 for a monologue.

The middle one is the one I would put in front of a team, because it is the only one that changes what the teacher does. Collect what people tried, teach against it, and their attempt has altered your lesson. Deliver the lesson you prepared anyway and the attempt was a warm-up, and the numbers say it performs like one.

It is tempting to read the Turkish result as this same finding implemented in a chatbot, on the grounds that a hint is instruction built on where the student actually is. I do not think that holds, and the exam result is why. The moderator describes instruction arriving after a completed attempt, built from what the learner produced during it. A hint arrives mid-attempt and stops it failing cleanly. Different interventions, and they gave different outcomes.

Configuring the AI is how you avoid a cost. Getting a gain is a question about the sequence of your lesson, and the tool has very little to do with it.

The objection you will hit

Learners rate this lower. They finish an attempt frustrated, and on a survey afterwards they will tell you the lesson where you explained things cleanly first was the better one.

They are not being difficult. Louis Deslauriers and colleagues, whose study I have quoted here before, randomly assigned physics students to active or passive classes with identical content and the same instructor. The active students learned measurably more and rated their own learning lower (Deslauriers et al., PNAS, 2019). Their felt sense of how the class had gone pointed away from what the tests found.

So if your quality signal is end-of-module satisfaction, this design will look like a mistake throughout the whole period it is succeeding. Say that to your stakeholders before you start, and agree something else to measure.

Where it stops working

Attempt-first is not a universal setting, and pretending otherwise would be the same overreach I keep complaining about.

The expertise reversal effect, which I spent most of the last edition on, says supports that help novices turn counterproductive as knowledge grows (Kalyuga, Ayres, Chandler and Sweller, Educational Psychologist, 2003). The same logic runs the other way for struggle. A genuine novice with no foothold does not fail productively. They just fail.

Concretely, in case that reads as a formality. Teaching somebody their first spreadsheet formula, there is nothing for them to attempt with. There is no wrong idea about how a cell reference behaves sitting in their head for the attempt to surface, so ten minutes of trying produces frustration where you wanted a question. Attempt-first needs the learner to have something to be wrong with. When they have nothing, teach first and save the attempt for the second thing you teach them.

Fading is the term for managing that, and I used it in the last edition without saying where it comes from. It is the 1976 paper that gave us the word scaffolding (Wood, Bruner and Ross, 1976). You build the support in and plan its removal at the same time.

For the next module you touch

Give them the problem before the method. Keep the attempt short enough that failing at it is not demoralising, and I should say plainly that the ten minutes I use is a working rule from practice with no research behind it. Have them attempt in pairs where you can. Then collect what they produced and teach against that, out loud.

If there is an AI in the workflow, set it to hint rather than answer during the attempt window, and let it do as it likes afterwards. On the evidence, that buys protection and nothing more, which is still worth having.

Almost none of this is mine, and the scaffolding paper is fifty years old. What has changed is that we now have a tool that is very good at completing people’s attempts for them, which makes an old sequencing question urgent. The answer has been sitting in the literature the whole time.

Next edition: the satisfaction-score problem. How do you agree a learning measure with somebody who has only ever been shown happy sheets? It is the question this edition leaves open, and the answers to the one below will shape it.

These editions run alongside Learning to Learn With AI, a longer series working the same ground in order.

How this was made. Full disclosure, and I mean it as a courtesy, not a confession. AI tools help me research, check my claims, tidy my grammar, and stay on top of far more reading than one person can. The thinking and the judgment are mine, I edit every word into my own voice by hand, and then I put my name on it. Telling you that is the whole point of keeping it human.

One question I would like answered, because it is the part I handle worst. If you have run a sequence like this in your own module, what did you watch instead of the satisfaction scores to tell whether it was working? I have never found a signal I trusted inside a single run, and I suspect somebody reading this has.

Common questions

What is productive failure? It’s the finding that learners understand a concept better when they try a problem before they are taught the method. The failed attempt builds the structure the later explanation attaches to. Tanmay Sinha and Manu Kapur pooled 53 studies on it.

Does trying a problem first work for complete beginners? Only if they have something to be wrong with. A genuine novice with no foothold doesn’t fail productively, and an expert made to reinvent what they already know is having their time wasted.

References

Working through something on your own site? Get in touch →