No. 14 Essay
Feed the pig.
Two school-technology failures with more money than most well-designed things. The honest case for design is insurance, and our own statistics need checking first.
Ask what it costs someone in your organisation to say the new thing is worse than the old one.
If the answer is anything above zero, bad news will arrive late, or not at all.
A quarter every time you said it
Los Angeles Unified built a new student-records system for a district serving hundreds of thousands of students. Engineers stress-tested it for 1,200 users at once.
Staff kept saying the old system had worked better. Leadership responded with a scheme called Feed the pig. Every time someone made the comparison, they put a quarter in the pig.
1,200
concurrent users it was tested for
25¢
fine for saying the old system was better
197 of 201
schools later reported it couldn’t reliably identify students with disabilities
The lesson isn’t the slogan. It is the sequence. Weak testing produced warnings. Leadership penalised the warnings. Serious failures survived into use. Nobody involved intended any of it.
A team that charges people for reporting a fault has built a machine for destroying the only signal that could save the project.
Same district, a bigger cheque
The same district wrote a specification for classroom iPads that a federal review later found heavily favoured Apple. It committed about $1.3 billion, including 43,000 devices at $768 each, with a maths and English curriculum chosen from samples rather than evidence. Two of 69 schools used the curriculum. Around 5% of students could reach the software consistently.
Neither project failed for lack of money. They had more money than almost any well-designed thing you can name. What they lacked was a way to catch a problem while it was still cheap to fix.
The honest business case is insurance
The case for design is usually pitched as growth. These cases support something duller and far more defensible: early risk detection. Catching one expensive failure early can pay for years of design work, even when the average gains are hard to attribute.
Less exciting. It also survives scrutiny, which is more than can be said for the usual pitch.
Burn our own sales material first
Predict before you look
Nearly every design and engineering consultancy has quoted this: a fault costs 100 times more to fix after release than in design, from a study by the IBM Systems Sciences Institute.
Is the study real?
Someone went looking. The trail ends at a 1987 textbook citing internal IBM course notes. The “Institute” was a staff training programme, and no data was ever published. A version of the idea does appear in a real paper by Boehm and Basili, so the fair line is that the slogan is a rumour, not that the idea is false.
It isn’t alone.
- The design index that beat the market by 211% was built from 16 companies chosen after they had succeeded. Apple dominates it. Remove one company and the result largely collapses.
- The consultancy study linking design to 32 points more revenue growth is a correlation, from an unpublished dataset, by a firm that sells design consulting.
- The review closest to home screened 190 papers on usability in learning products. Not one linked a usability gain to a learning outcome. That claim isn’t false. It is untested.
Bring one of those numbers into a room with a competent analyst and you have handed the sceptics their case permanently. In an organisation that already half suspects design of being decoration, you get to do that once.
Take this with you
- If it costs anything to say the new thing is worse, you will hear it too late.
- The defensible case for design is catching expensive failure early, not promising growth.
- Check our own profession’s numbers before someone else does. Retiring a weak statistic buys more credibility than another case study.
Bad design loses more than good design gains. Make that argument. It is the one that holds.
Try this week
Audit the numbers in your last deck.
- List every statistic in the last presentation you gave about the value of your work.
- Trace each to its original source. Not the blog that quoted it: the study.
- Cut any you can’t trace. Replace one with a failure your work caught early.
To mull over
- What is the pig in your organisation: the quiet cost of saying something isn’t working?
- Which number have you repeated without ever checking it?
- What disaster did your team prevent last year, and who knows about it?
Sources
- LAUSD Inspector General reports, via LA School Report. Newcombe, T. (2015). A cautionary tale for any government IT project: L.A.’s failed iPad program. Governing.
- Anderson, T. (2021, July 22). Everyone cites that “bugs are 100x more expensive to fix in production” research, but the study might not even exist. The Register.
- Boehm, B. W., & Basili, V. R. (2001). Software defect reduction top 10 list. Computer, 34(1).
- Sheppard, B., et al. (2018). The business value of design. McKinsey & Company. Mauro, C. L., & Thurman, P. W. (2019). Critical analysis of McKinsey’s report. Rae, J. (2016). Design Value Index exemplars outperform the S&P 500. Design Management Review, 27(4). Rumsey, R. (2023), on the index without Apple.
- Nakamura, W. T., Oliveira, E. H. T., & Conte, T. (2017). Usability and user experience evaluation of learning management systems: A systematic mapping study. ICEIS 2017.