No. 13 Essay
Aim at the floor.
The average user doesn’t exist. The people who need a product most are the furthest from it, and the biggest gains are found there.
The average user doesn’t exist. Designing for them fails three different groups in three different ways.
- Treatment varies. The same design can help one learner and harm another.
- Exposure varies. A design can’t help people who never encounter it.
- Gains land unevenly. An average gain can hide who got it.
The same scaffolding has opposite effects
Across 60 experiments and 176 effect sizes, more instructional guidance helped learners with little prior knowledge by about half a standard deviation. It harmed learners with a lot of prior knowledge by nearly as much.
Support that carries a novice becomes clutter for someone who already has a working mental model. So there are two honest options:
- Adapt support to what the learner already knows. Expensive, and usually done badly.
- Name the learner out loud, and accept that you are wrong for everyone else.
Most products quietly pick the novice, then act surprised when their strongest students stop showing up.
Assume many people will never open it
Predict before you look
Two randomised trials offered primary-school children an AI reading tutor, with a recommended 30 minutes a week.
What share of the children never logged in once?
~50%
never logged in once
2–5 min
weekly use among those who did
30 min
the recommended dose
The lead researcher’s verdict: they never got close enough to the dose to find out whether it worked. The trials don’t tell us why use was low. They do tell us the question was never answered.
The delivery routine is part of the intervention. A product that depends on regular use needs a credible answer to who schedules it, who supervises it and where it fits in the week. Another engagement feature is rarely that answer. A person, a routine or a slot on the timetable usually is.
And teacher time isn’t free. Dutch secondary teachers get roughly 15 minutes of paid preparation per lesson. A design that assumes an hour of setup has already failed, whatever it did in the pilot.
Take-up is often the binding constraint, not quality.
The biggest gains are at the bottom
Tutor CoPilot gave human tutors live suggestions they could accept, edit or ignore. It cost about $20 per tutor per year.
+4
percentage points of topic mastery, overall
+9
for students of the lowest-rated tutors
$20
per tutor, per year
The transcripts showed why. Tutors using the tool asked students to explain their reasoning more often, and gave generic encouragement less often. The Kenyan programme from No. 11 had the same shape: gains everywhere, largest for the lowest achievers.
Consumer product design optimises the delight case. That instinct travels badly. Education is a variance problem, and the wins come from lifting the floor. That demos poorly and matters enormously. Nobody has ever been promoted for it either.
Take this with you
- Name your learner, or adapt. “Our users” is too broad when learners need opposite amounts of help.
- Design for exposure. Who schedules it, who supervises it, where does it fit?
- Aim at the least confident teacher, the oldest device and the worst connection. That is where most of the available effect is.
If your pilot ran with your three most enthusiastic teachers, it predicts almost nothing about the fourth school.
Try this week
Spend an hour at the bottom of the distribution.
- Find one person your product is failing: the teacher with 15 minutes, the student who never logged in, the school with the oldest laptops.
- Watch or ask how the week actually goes for them. Don’t demo anything.
- Write one sentence on what would have to be true for your product to reach them. Take it to your next planning meeting.
To mull over
- Which learner does your product quietly assume, and who did that choice leave out?
- What would your roadmap look like if success meant raising your lowest decile?
- Where is a person, not a feature, the real answer to your adoption problem?
Sources
- Tetzlaff, L., Simonsmeier, B., Peters, T., & Brod, G. (2025). A cornerstone of adaptivity: A meta-analysis of the expertise reversal effect. Learning and Instruction, 98.
- Robinson, C. D., Gormley, D., Ribeiro, A. T., & Loeb, S. (2026). Access is not enough: Human support improves engagement with AI tutoring. EdWorkingPapers.
- Janssen, F. J. J. M., Könings, K. D., & van Merriënboer, J. J. G. (2017). Participatory educational design. European Journal of Education, 52(3).
- Wang, R. E., Ribeiro, A. T., Robinson, C. D., Loeb, S., & Demszky, D. (2025). Tutor CoPilot: A human-AI approach for scaling real-time expertise. EdWorkingPaper 24-1054.