Chat System
Long-lived connections instead of polling, how messages find an online user, and getting delivery and ordering right
Chat inverts the usual model. Most systems answer requests; a chat system must push to a user who is not asking for anything.
1. Clarify and Scope
| Question | Assumption taken here |
|---|---|
| One-to-one, or groups? | Both, groups capped at 500 |
| Delivery receipts? | Sent, delivered, read |
| History? | Stored permanently, searchable |
| Media? | Uploaded separately, messages carry URLs |
| End-to-end encryption? | Out of scope — it changes the storage design entirely |
2. Put Numbers On It
50 million daily active users
Assume 10 million connected at peak
MESSAGES
50M × 40 messages/day = 2B messages/day
÷ 100,000 sec ≈ 20,000 messages/sec
× 3 peak ≈ 60,000 messages/sec
CONNECTIONS
10M concurrent connections
~10 KB memory per connection
10M × 10 KB = 100 GB across the fleet
At 100k connections/server → 100 servers
STORAGE
message ≈ 300 bytes (text, ids, timestamps)
2B/day × 300 bytes = 600 GB/day
× 365 ≈ 220 TB/yearConnections are the resource, not CPU
A chat server is mostly idle. It holds sockets open and occasionally forwards a small message.
That makes memory and file descriptors the limit, not processing power. It also means the servers are stateful in a way most services are not — a specific user is on a specific machine — which is the central complication of the whole design.
3. Define the Interface
CALLED BY THE CLIENT
send(conversation_id, client_msg_id, text) → { message_id, server_ts }
history(conversation_id, before=cursor) → messages, newest first
mark(message_id, state) → delivered | read
PUSHED TO THE CLIENT over the live connection
message a new message in a conversation you belong to
receipt someone received or read one of yours
presence someone came online or went awayTwo halves, and the second one is why this is hard
Most systems only have the top half — the client asks, the server answers. Chat has a bottom half: the server must reach a client that asked for nothing.
That single requirement is what forces long-lived connections, a way to find which server holds a given user, and a plan for messages that arrive while someone is offline. Sections 4 and 5 are entirely about the bottom half.
`client_msg_id` is generated by the sender, not the server
The client invents an id before sending. If the acknowledgement is lost and the client retries, the server sees the same id and does not store the message twice.
Without it, "my message sent twice" happens on every flaky connection — and a chat app is used mostly on flaky connections. The idempotency key belongs in the interface, not bolted on later.
4. Choosing the Transport
| Option | How it works | Verdict |
|---|---|---|
| Short polling | Ask every few seconds | Wasteful and still laggy |
| Long polling | Request held open until a message arrives | Workable, awkward reconnects |
| Server-sent events | Server pushes over one HTTP stream | One-directional only |
| WebSocket | Full duplex, persistent | The right choice |
Why polling fails at this scale
10 million clients polling every 3 seconds is 3.3 million requests/sec — for a message rate of 60,000/sec.
That is 98% wasted work, and it still delivers messages up to 3 seconds late. Polling trades away both efficiency and latency at once.
WebSocket gives a single connection carrying messages in both directions with no per-message HTTP overhead. Keep a fallback to long polling for networks that block WebSocket.
5. Finding the Recipient
Here is the real problem. Sara is connected to chat server 42. Ali, connected to server 7, sends her a message. Server 7 has no socket to Sara.
Persist before you deliver, always
It is tempting to push the message to the recipient first and save it afterwards, since that feels faster.
Then the server crashes between the two, and the message is gone — but the recipient's client already displayed it. Now the two devices disagree about history forever.
Store first. A slightly slower send is far better than a chat log that does not match itself.
The presence registry
Redis: online:{user_id} → server_id TTL 30 secondsServers refresh the TTL with a heartbeat. If a server dies, its users' entries expire naturally rather than pointing at a dead machine.
Presence is the most write-heavy part of a chat system
Every connect, disconnect and heartbeat is a write. With 10 million connections and a 30-second heartbeat, that is over 300,000 writes/sec just to say "still here" — more than five times the message rate.
Reduce it by lengthening the heartbeat interval and accepting a slower "user went offline" signal, and by batching heartbeats per server rather than per user. Precise presence is genuinely expensive, and "last seen recently" is usually enough.
6. Storage and Ordering
messages
channel_id BIGINT ← partition key
message_id BIGINT ← sortable, time-ordered
sender_id BIGINT
body TEXT
created_at TIMESTAMP
PRIMARY KEY (channel_id, message_id DESC)Partitioning by channel_id means one conversation lives on one partition, so reading recent history is a single sequential scan. This is a natural fit for a wide-column store.
Wall-clock timestamps are not an ordering
Two servers' clocks differ by milliseconds. Two messages sent "at the same time" can be stored in an order neither user saw.
Use ids that sort correctly — Snowflake-style ids combining a timestamp, a machine id and a sequence number. They are unique, roughly time-ordered, and generated without coordination.
Within a single conversation, a per-channel sequence number is even stronger, since it makes gaps detectable.
Delivery guarantees
| Receipt | Meaning |
|---|---|
| Sent | The server stored it |
| Delivered | It reached the recipient's device |
| Read | The recipient opened it |
Each is a separate acknowledgement travelling back. Because the network can drop acknowledgements, delivery is at-least-once — so clients must deduplicate by message id.
7. What Was Traded Away
| Decision | Gained | Cost |
|---|---|---|
| WebSocket over polling | Instant delivery, far less traffic | Stateful servers, complex reconnection |
| Persist before deliver | No message is ever lost | Slightly slower sends |
| Presence in Redis with TTL | Dead servers clean themselves up | Offline status lags by up to the TTL |
| Partition by channel | Fast history reads | A very large group is a hot partition |
| At-least-once delivery | Nothing is dropped | Clients must deduplicate |
Follow-on Questions
Related Reading
- Queues and Async Work — the message bus between servers
- Reliability — reconnection, backoff and heartbeats
- Databases — partition keys and hot partitions
- News Feed — the fan-out problem, in its other form