No. 12 Essay
Measure it after you take it away.
An AI tutor made practice easier and the exam no better. Performance with a tool and capability without it are different numbers.
Every metric on your dashboard measures performance while the product is still in the room.
Completion, time on task, questions answered, streaks, satisfaction. All of it is performance with the tool. Capability once the tool is gone is a different variable. The two can point in opposite directions, in the same students, in the same week. Almost nobody measures the second one.
One model, two prompts, a thousand students
Predict before you look
About a thousand high-school students in Turkey had four maths lessons. Some had GPT-4 as it comes. Some had a version that had been given worked solutions and common misconceptions, and told to hint, never answer. Some had no AI at all.
A Plain GPT-4
Students ask, it answers
- Practice performance up 48%
B Scaffolded GPT-4
Hints, never answers
- Practice performance up 127%
Then the AI was taken away, and everyone sat a closed-book exam.
Which group beat the students who never had AI?
The plain version did worse than students who never had AI. The scaffolded version showed no measurable difference from them.
Be precise about that second result, because most write-ups aren’t. The careful design didn’t win. It broke even. It bought back a deficit that careless design had created. Same model, same students, same lessons. Only the prompt differed, and one version quietly damaged learning while making everyone feel more productive.
Generative AI has taken the cost of producing learning material to roughly zero. When production is free, design is what separates one product from another. It also makes bad design cheaper to ship than ever.
Ease and learning come apart
The previous series covered the fluency trap: learners who learned more felt they had learned less. Here is the opposite mistake, made by people who took that lesson too far.
Do hard-to-read fonts improve learning, by making people work harder? Across 25 studies and 3,135 people:
+0.52
Longer study time
−0.43
Lower confidence
−0.01
Recall gain. None.
They worked longer, felt worse and learned exactly the same. Difficulty pays only when it is the difficulty of retrieving, choosing or explaining. Everything else is a tax.
Remove interface friction. Preserve cognitive effort.
Stating it is easy. Applying it is the hardest judgement in this work, because the two feel identical from the inside. A student stuck on your navigation and a student stuck on your question both look stuck, sound frustrated and file the same support ticket.
The test that works is removal. Take the friction out and watch what is lost.
- If nothing is lost, it was waste.
- If the task suddenly feels effortless, you have probably just removed the learning.
Both failures are expensive, and only one is visible. Add difficulty for its own sake and at least people complain. Strip everything and you ship a product that feels wonderful, tests beautifully, teaches nothing and reports success on every metric you own.
Interface friction is waste. Cognitive effort is the whole product.
Two tests for the thing you designed
The delete test
Remove the product and keep the activity. If something valuable remains, you designed a learning experience. If nothing remains, you built a delivery mechanism with nothing to deliver. This failure is almost invisible from inside a product team, because every artefact a product team produces is a screen.
The subtraction test
Can you cut 40% without losing the argument? The best-evidenced principles in the learning-materials literature are subtractive. Take out the interesting extras. Stop narrating the screen. Put the label on the thing. Every release plan is additive. Ask instead: what is the smallest version of this that still carries the idea?
Take this with you
- Performance with the product and capability without it are different numbers. Measure the second.
- Difficulty only pays when it is the thinking. Everything else is a tax.
- Delete the product and keep the activity. If nothing is left, nothing was designed.
Build one measure that happens when your product isn’t in the room. It will be the least flattering number you own, and the only honest one.
Try this week
Design one delayed, unaided check.
- Pick one thing your product claims to teach.
- Write three questions a learner should be able to answer a week later, without the product, ideally on paper.
- Find five learners who used it last week. Ask them. Compare with what your dashboard says.
To mull over
- Which of your metrics would look better if your product did more of the thinking for learners?
- Where have you smoothed something because smoothing is what you were trained to do?
- If an AI feature makes practice twice as easy, what would convince you it wasn’t also making the exam harder?
Sources
- Bastani, H., et al. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. PNAS, 122(26).
- Xie, H., Zhou, Z., & Liu, Q. (2018). Null effects of perceptual disfluency on learning outcomes in a text-based educational context: A meta-analysis. Educational Psychology Review, 30.
- Bjork, R. A., & Bjork, E. L. (2020). Desirable difficulties in theory and practice. Journal of Applied Research in Memory and Cognition, 9(4).
- Mayer, R. E., & Fiorella, L. (2014). Principles for reducing extraneous processing. Cambridge Handbook of Multimedia Learning. Sundararajan, N., & Adesope, O. (2020). Educational Psychology Review, 32.