Distributed systems
Technology
16 rabbit holes · 7 people exploring
Common starting questions
- Was Three Mile Island caused by bad design?
- Why do fighting game players care so much about rollback?
Rabbit holes
- Facebook took itself off the internet in 2021
A config change withdrew the BGP routes to Facebook's DNS servers, so nothing could find facebook.com, and the internal tools needed to fix it were also down. Reportedly engineers had trouble getting into the buildings.
Ben Adler · 4 sources
- Three Mile Island was partly an interface problem
A relief valve stuck open, but the light on the control panel only showed that the valve had been told to close, not whether it actually had. The operators trusted the light while alarms went off by the dozen. Afterwards the industry got a lot more serious about control room design and human factors.
Pete Lindgren · 6 sources
- why time is so hard in distributed systems
Alistair Grant · 5 sources
- F1 teams run a mission control back at the factory
The car streams data from hundreds of sensors to the garage, and from there to a room full of engineers at the factory watching tyre temps, brakes and engine data live. Two-way telemetry, where the pit wall changed settings on the car, was allowed briefly around 2001–2002 and then banned, which is why drivers are still the ones turning dials on the steering wheel.
Arjun Pillai · 6 sources
- the fallacies of distributed computing
'The network is reliable. Latency is zero. Bandwidth is infinite.' Written at Sun in the 90s and still the best checklist for design reviews.
Arjun Pillai · 3 sources
- how one bad config file crashed 8.5 million Windows PCs
Samuel Okafor · 4 sources
- CAP theorem, and then why everyone says PACELC instead
Arjun Pillai · 5 sources
- SQLite is probably the most deployed software in the world
In every phone, every browser, every Mac, planes, cars. Three main developers. Public domain. They have a code of conduct based on the Rule of Saint Benedict, which was its own rabbit hole.
Samuel Okafor · 4 sources
- the internet is mostly cables on the sea floor
Something like 99% of intercontinental data goes through cables about as thick as a garden hose. Sharks used to bite them. Ship anchors still cut them.
Nils Eriksson · 5 sources
- git is a content-addressed database pretending to be version control
Samuel Okafor · 4 sources
- local-first software: you own your data and the cloud is a backup
The Ink & Switch essay lays out seven ideals: fast, multi-device, offline, collaborative, long-lived, private, and user-owned. CRDTs are the trick that makes merging without a server possible.
Ben Adler · 5 sources
- the year 2038 problem
At 03:14:07 UTC on 19 January 2038, a signed 32-bit Unix timestamp overflows and wraps to 1901. Most modern systems moved to 64-bit time, but embedded stuff lives a long time.
Callum Reid · 4 sources
- the 11 lines of code that broke the internet (left-pad)
2016: one developer unpublished a tiny package after a name dispute and broke builds for React, Babel, and thousands of projects. npm changed its unpublish policy after.
Arjun Pillai · 4 sources
- the xz backdoor was caught because SSH logins were 500ms slower
A years-long social engineering campaign to become a trusted maintainer of a compression library, then a backdoor hidden in test files. Found by a Microsoft engineer benchmarking Postgres who noticed sshd using too much CPU.
Samuel Okafor · 5 sources
- rollback netcode, explained to myself
Arjun Pillai · 6 sources
- Byzantine generals, Paxos, and Raft
Samuel Okafor · 5 sources