News Feed
Fan-out on write versus on read, why accounts with millions of followers break both, and the hybrid that real systems use
A feed looks like a database query and is not. The whole problem is that one write can require a million updates — or one read can require a million lookups.
1. Clarify and Scope
| Question | Assumption taken here |
|---|---|
| Feed ordering? | Reverse chronological, with room for ranking later |
| How far back? | The most recent few hundred posts |
| Real time? | Seconds of delay is fine |
| Media? | Posts carry text plus image URLs |
| Follow model? | One-directional, like Twitter |
In scope: post, follow, read your feed. Out of scope: direct messages, notifications, comment threads.
2. Put Numbers On It
300 million daily active users
WRITES
300M × 2 posts/day = 600M posts/day
÷ 100,000 sec ≈ 6,000 posts/sec
× 3 peak ≈ 18,000 posts/sec
READS
300M × 10 feed loads/day = 3B reads/day
÷ 100,000 ≈ 30,000 reads/sec
× 3 peak ≈ 90,000 reads/sec
Read : write ≈ 5 : 1
Average followers per user ≈ 200
Largest accounts ≈ 100,000,000 followers ← the whole problemThat last line is the design
6,000 posts/sec × 200 average followers = 1.2 million feed writes per second. That is already large.
But an account with 100 million followers generates 100 million feed writes from a single post. If a few such accounts post at once, the system must absorb hundreds of millions of writes in seconds.
Averages are useless here. The design is decided entirely by the tail.
3. Define the Interface
POST /posts { "text": "...", "media": [...] }
GET /feed?after=cursor the caller's feed, newest first
POST /follow/{user_id}Cursor pagination, not offset — new posts arrive constantly, so offset pagination would duplicate and skip items.
4. The Two Approaches
Fan-out on read — assemble at read time
Store each post once. When a user opens their feed, fetch the people they follow, query recent posts from each, merge and sort.
Write: 1 insert ← cheap
Read: fetch 200 followees
query recent posts from each
merge and sort ← expensive, on every readWrites are trivial. Reads are heavy — and reads outnumber writes 5:1, so the cost lands in the worst place.
Fan-out on write — precompute the feed
When someone posts, immediately push the post id into a stored feed list for every follower.
Write: 1 insert + N feed pushes ← expensive, N = follower count
Read: read one precomputed list ← a single lookupReads become trivial. Writes explode for popular accounts.
The core trade-off, stated plainly
| Fan-out on read | Fan-out on write | |
|---|---|---|
| Write cost | O(1) | O(followers) |
| Read cost | O(followees) | O(1) |
| Feed latency | Slow — assembled live | Fast — already built |
| Storage | Minimal | One entry per follower per post |
| Breaks on | Users following thousands | Accounts with millions of followers |
Neither is correct on its own. Fan-out on write is right for the 99.9% of users with normal follower counts; fan-out on read is right for the handful with millions.
5. The Hybrid
Use fan-out on write for ordinary accounts, and fan-out on read for the small number of very large ones.
Why the merge stays cheap
A user might follow 500 accounts, but only a handful are large enough to be excluded from fan-out.
So the read is: one lookup for the precomputed list, plus perhaps three to five lookups for large accounts. That is six lookups, not five hundred — the hybrid keeps reads nearly as fast as pure fan-out on write.
Storage
posts
post_id BIGINT PRIMARY KEY
author_id BIGINT
text TEXT
media_urls JSON
created_at TIMESTAMP
INDEX (author_id, created_at DESC) ← for the celebrity read path
feeds (Redis, one sorted set per user)
key: feed:{user_id}
score: post timestamp
member: post_id
capped at ~800 entriesCap the feed lists
Without a cap, feed lists grow forever and Redis runs out of memory.
Nobody scrolls past a few hundred posts. Trim each list to around 800 entries on write. Users who scroll deeper fall through to a slower database query, which is a fine trade for a rare action.
Storing only post ids in the list, and hydrating the post bodies from a separate cache, keeps memory small and means an edited post is corrected everywhere at once.
6. What Was Traded Away
| Decision | Gained | Cost |
|---|---|---|
| Fan-out on write for most users | Feed reads are one lookup | Huge write amplification |
| Skip fan-out for large accounts | Write load stays bounded | Reads must merge two sources |
| Async fan-out via queue | Posting is fast | Followers see the post seconds later |
| Capped feed lists | Bounded memory | Deep scrolling is slow |
| Store ids, not post bodies | Small memory, edits propagate | Extra hydration lookup |
Follow-on Questions
Related Reading
- Queues and Async Work — the fan-out pipeline
- Caching — feed lists are a cache with a cap
- Databases — sharding the post store
- Heaps — merging k sorted post streams is Merge k Sorted Lists