GitHub outage post-mortem exposes retry loop failure pattern
GitHub published its August 17 outage post-mortem, and the mechanism that turned a bad situation into a full outage was a client-side retry loop that amplified traffic exactly when the system was trying to recover. Monthly commits on GitHub had grown from 1.4 billion to 2.9 billion since April, which means the blast radius of any failure has roughly doubled in a few months.
The HN thread reaction was immediate recognition: this pattern shows up in almost every serious outage. A service degrades, clients retry aggressively, the recovering system gets hammered again, and recovery stalls. It is one of the most well-understood failure modes in distributed systems and one of the most reliably rediscovered in production.
The criticism in the thread toward GitHub's response was sharp. One commenter summarized the subtext: GitHub is committed to fixing problems as long as it doesn't involve buying non-AI hardware, hiring humans, or moving off Microsoft. That framing reflects a real concern that infrastructure investment is being deprioritized in favor of AI product development.
So what?
Any system you build that has clients retrying on failure needs exponential backoff with jitter baked in by default, not as an afterthought. The GitHub case is a reminder that growth itself creates new failure modes: what worked at 1.4 billion monthly operations can collapse at 2.9 billion. Model your retry behavior under degraded conditions before you need to.