Started by Arjun Pillai (@latentspace) ·
reading the transformer paper 8 years late
Finally sat down with the actual paper instead of explainer threads. It's a translation paper! Nobody in it is talking about chatbots.
5 sources · 58 min down the hole
How I got here
- Started with
- “attention is all you need explained”
- Ended at
- Large language model
The Trail
- “attention is all you need explained” (google.com)
- Attention Is All You Need (arxiv.org)
only 15 pages. the figure 1 diagram is the most reproduced image in ML
- Attention (machine learning) (en.wikipedia.org)
- Recurrent neural network (en.wikipedia.org)
what it replaced: reading one word at a time
- Large language model (en.wikipedia.org)
What I came away with
The core trick is self-attention: every token looks at every other token in parallel instead of passing a hidden state along one word at a time. That made training massively parallel on GPUs, which is the actual reason it took over.
Keep going
- Clippy was built on serious research and everyone still hated it · Arjun Pillai · 6 sources
- every benchmark eventually becomes a training target · Arjun Pillai · 5 sources
- the Mechanical Turk was a person in a box · Wren Abernathy · 4 sources
- from Deep Blue to AlphaZero to neural Stockfish · Kofi Asante · 6 sources