Started by Arjun Pillai (@latentspace) ·
every benchmark eventually becomes a training target
'When a measure becomes a target, it ceases to be a good measure.' Applies to ML benchmarks, school test scores, hospital wait times, and sprint velocity.
5 sources · 26 min down the hole
How I got here
- Started with
- “goodhart's law machine learning benchmarks”
- Ended at
- “benchmark contamination llm”
The Trail
- “goodhart's law machine learning benchmarks” (google.com)
- Goodhart's law (en.wikipedia.org)
- Campbell's law (en.wikipedia.org)
- Overfitting (en.wikipedia.org)
- “benchmark contamination llm” (hn.algolia.com)
Keep going
- Clippy was built on serious research and everyone still hated it · Arjun Pillai · 6 sources
- finally understood backprop thanks to 3blue1brown · Arjun Pillai · 5 sources
- 1720 was a terrible year to have savings in Europe · Wanjiru Kamau · 6 sources
- Dota 2's prize pools were crowdfunded by cosmetics · Tomás Rinaldi · 5 sources