No. 11 Essay
Zero is an average.
Forty years of edtech research pool to one modest number. The spread underneath it is where design decides the outcome.
Two well-funded programmes, both meant to improve learning. One of them did nothing at all.
Which one worked?
Predict before you look
A Peru
A laptop for every child
- 319 rural schools
- Laptops arrived, teachers were trained, children used them
- Reading and maths measured after 15 months
B Kenya
Tablets for the teachers
- Scripted lessons on the teacher’s tablet
- Teachers deliver; heads check delivery
- Places allocated by lottery; measured over two years
Which programme improved learning?
0.003
Peru: effect on maths and reading, in standard deviations. Statistically nothing.
0.81
Kenya: above the 90th percentile of measured effects in international education.
Peru’s delivery wasn’t the problem. Kenya’s gains appeared across schools and teacher profiles, and were largest for the lowest-achieving children.
A fair warning: these are two different programmes, not a controlled pair. The contrast doesn’t prove what caused the gap. It makes the question concrete. Was the value in distributing hardware, or in designing what happened in the lesson?
The average describes nobody
Pool forty years of research on technology in education, 25 meta-analyses covering 1,055 studies, and you get a mean effect of about 0.35 standard deviations. That number is quoted constantly.
It is also nearly useless for predicting what any one implementation will do, because the spread around it is enormous. The spread isn’t noise to apologise for. It is the finding.
You can’t get a useful prediction out of a distribution this wide by quoting its mean. The useful question is what decides where an implementation lands.
Compare devices and you find nothing. Compare designs and you find something.
This is the cleanest pair in the literature. One review team, using the same methods, asked two different questions.
2004 Compare media
Online versus classroom
- 232 studies
- Average difference: about zero
- Plenty did much better, plenty much worse
2009 Compare designs
How interaction was designed
- 74 studies, sorted by student–student, student–teacher and student–content interaction
- Average effect: 0.38
That is evidence that designed learning interactions matter more than the delivery medium. It isn’t proof that every kind of design causes learning.
Richard Clark made the underlying point in 1983. Media are vehicles. Whether anything useful is inside the vehicle is a separate question. The field has spent forty years not absorbing it, which is why procurement exercises still compare devices.
It is not the technology per se that makes the difference; rather it is the pedagogy and the context.Education Endowment Foundation evidence review, 2019
So ask a better question
Stop asking whether a category of technology works on average. Ask: which design works, for whom, under what conditions? The gap between the top of the distribution and the bottom is enormous, and it isn’t random.
Design isn’t a single cause. It is the discipline of making the teaching method, the rollout and the setting work together for a named learner. That narrower claim is the one the evidence can carry.
Take this with you
- A pooled average hides a spread wide enough to contain both total failure and large success.
- Comparing devices finds little. Comparing how the learning is designed finds much more.
- The question is never “does edtech work?” It is “which design, for whom, under what conditions?”
Good intentions didn’t separate Peru from Kenya. Both programmes had plenty.
Try this week
Find the vehicle and the cargo in one decision.
- Pick the last edtech decision you were part of: a purchase, a rollout, a feature.
- Write down what was specified about the vehicle (device, platform, feature) and what was specified about the learning (what learners would do, with whom, how often).
- If the second list is shorter, you have found the design work that is still missing.
To mull over
- Where in your work is an average being quoted as if it were a prediction?
- What would you need to know to guess where your product sits in the spread?
- Kenya’s programme put scripted lessons in front of teachers. What does that make you think about teacher autonomy, and is that a question of evidence or of values?
Sources
- Cristia, J., et al. (2017). Technology and child development: Evidence from the One Laptop per Child program. AEJ: Applied Economics, 9(3).
- Gray-Lobe, G., Keats, A., Kremer, M., Mbiti, I., & Ozier, O. (2022). Can education be standardized? Evidence from Kenya. Working paper.
- Tamim, R. M., et al. (2011). What forty years of research says about the impact of technology on learning. Review of Educational Research, 81(1).
- Bernard, R. M., et al. (2004). Review of Educational Research, 74(3). Bernard, R. M., et al. (2009). Review of Educational Research, 79(3).
- Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4). Lewin, C., et al. (2019). Using Digital Technology to Improve Learning. Education Endowment Foundation.