File Storage and Sync
Keeping files identical across devices — chunking, deduplication, delta sync, and resolving edits made in two places at once
The naive version — upload the file, download the file — works and is enormously wasteful. Everything interesting here comes from not transferring what the other side already has.
1. Clarify and Scope
| Question | Assumption taken here |
|---|---|
| Core feature? | Files sync across a user's devices |
| File size? | Up to 10 GB |
| Sharing? | Out of scope |
| Version history? | Yes, 30 days |
| Offline editing? | Yes — conflicts must be handled |
2. Put Numbers On It
50 million users, 10 GB average
50M × 10 GB = 500 PB raw
But files are heavily duplicated — the same PDFs, installers, photos
Typical deduplication saving ≈ 50-70%
After dedup ≈ 150-250 PB
WRITES
50M users × 20 file changes/day = 1B changes/day
÷ 100,000 sec ≈ 10,000 changes/sec
BANDWIDTH (naive: whole file each time)
average file 2 MB × 10,000/sec = 20 GB/sec
BANDWIDTH (delta sync: changed blocks only)
~5% of blocks change ≈ 1 GB/secThe 20× bandwidth difference is the whole design
Uploading a whole 50 MB presentation because one slide changed is the obvious approach and it is twenty times more expensive than necessary.
Splitting files into blocks and transferring only the changed ones is what makes this affordable — and it is also what makes deduplication possible, since identical blocks from different users can be stored once.
One decision, two large wins.
3. Define the Interface
list_changes(device_id, since_cursor) → files changed since this device
last synced, + a new cursor
get_manifest(file_id) → ordered list of chunk hashes
has_chunks(hashes[]) → which ones the server already holds
put_chunk(hash, bytes) → stored
commit(file_id, hashes[], parent_ver) → { version } or { conflict }
get_chunk(hash) → bytesThere is no `upload_file`, and that is the entire design
Read the list again. Not one operation takes a file. They take chunks, addressed by the hash of their contents.
Everything this system is known for falls out of that one decision:
| Because the unit is a chunk… | You get |
|---|---|
has_chunks can be asked first | Deduplication — never send bytes the server holds |
| Only changed chunks are sent | Delta sync — a 1 GB file with a small edit costs kilobytes |
Each put_chunk stands alone | Resumable uploads — a dropped connection loses one chunk |
commit is separate from upload | A version appears all at once, never half-written |
Had the interface been upload_file(bytes), none of those would be possible without changing it. Sections 4 to 6 only explain machinery this step already committed to.
`parent_ver` on commit is how conflicts get noticed
The client says which version it edited. If the server has moved on, it returns conflict instead of silently overwriting.
A system that supports offline editing must be able to detect this. Detection has to be in the interface, because only the client knows what it started from.
4. Chunking
Split each file into blocks of roughly 4 MB. Hash each block. The file becomes a list of block hashes — its manifest.
report.pdf → [ hash_a, hash_b, hash_c, hash_d ]
Edit page 3:
→ [ hash_a, hash_b, hash_X, hash_d ]
↑ only this block is uploadedFixed-size chunking breaks on insertion
Cut every 4 MB from the start, then insert one byte at the beginning: every subsequent block boundary shifts, every hash changes, and the entire file is re-uploaded.
Content-defined chunking fixes this. Boundaries are chosen by a rolling hash of the content — cut where the hash matches a pattern — so an insertion shifts only the block containing it. Neighbouring boundaries realign naturally.
This matters enormously for documents and source files, where insertion in the middle is the normal edit.
5. The Architecture
Separating metadata from blocks is the key structural decision
Metadata — manifests, versions, ownership — is small, relational and frequently updated. It belongs in a database.
Blocks — the actual bytes — are large, immutable and content-addressed. They belong in object storage.
Because a block's name is its hash, blocks are immutable and can be cached, replicated and shared freely. Two users with the same file automatically share storage, with no coordination.
6. Deduplication
Content addressing gives deduplication for free: if a block's hash already exists, there is nothing to upload.
| Level | Saving | Note |
|---|---|---|
| Per user | Moderate | The same file in two folders |
| Global | Large — 50%+ | The same installer across a million accounts |
Global deduplication leaks information
If uploading a file completes instantly, the uploader learns that someone else already has that exact file.
That is a real privacy leak: an attacker can test whether a specific document exists in the system by trying to upload it and timing the response.
Mitigations: deduplicate only within a user's own account, always require the upload and discard it server-side, or add randomised delays. Global dedup saves a lot of storage — say so, and say what it costs.
7. Conflicts
Two devices edit the same file while offline. Both come back online. Both have a valid new version derived from the same parent.
Do not silently pick a winner
Last-write-wins discards someone's work with no notification. For a photo that may be tolerable; for a document someone spent an hour on it is unacceptable.
The standard behaviour is to keep both:
report.pdf ← one version
report (conflicted copy, Sara's Mac).pdf ← the otherUgly, and correct. The user decides, because only they know which edit matters. Automatic merging is only safe for structured formats where merge semantics are defined.
Detecting a conflict needs version history, not timestamps. Each version records its parent, so the server can tell "B is a descendant of A" (fine, fast-forward) from "A and B share a parent" (a genuine conflict) — the same causality question that vector clocks answer.
8. What Was Traded Away
| Decision | Gained | Cost |
|---|---|---|
| Content-defined chunking | Insertions do not re-upload the file | More complex than fixed chunks |
| Global deduplication | 50%+ storage saved | Existence of a file is observable |
| Metadata / blocks split | Each scales independently | Two systems to keep consistent |
| Conflicted copies | No work is ever lost | Users must resolve manually |
| Push notification of changes | Sync within seconds | Long-lived connections per device |
Follow-on Questions
Related Reading
- Storage Engines — object storage and immutability
- Distributed Key-Value Store — causality and conflict resolution
- Chat System — sequence-number catch-up after being offline
- Security — what deduplication reveals