Payment System
Where correctness matters more than latency — idempotency, the double-entry ledger, reconciliation, and why money is never deleted
Most designs optimise for speed. This one optimises for never being wrong. A feed showing a stale count is a nuisance; a payment taken twice is a serious incident.
1. Clarify and Scope
| Question | Assumption taken here |
|---|---|
| What are we building? | Charging a card and recording the result |
| Do we handle card details? | No — a third-party provider does |
| Refunds? | Yes, full and partial |
| Currencies? | Multiple, no conversion |
| Consistency requirement | Strong. Money must never be double-charged or lost |
Not storing card numbers is a design decision, not a shortcut
Handling raw card data pulls the entire system into strict compliance scope, with heavy audit obligations on every service that touches it.
Instead, the client sends card details directly to the payment provider, which returns a token. Your servers only ever see the token. This shrinks the sensitive surface to almost nothing, and it is what nearly every real system does.
2. Put Numbers On It
10 million payments/day
÷ 100,000 sec ≈ 100 payments/sec
× 5 peak (sales, paydays) ≈ 500 payments/sec ← small
STORAGE
a payment record ≈ 1 KB, and ledger entries are ~200 bytes each
10M/day × ~2 KB = 20 GB/day
× 365 × 7 years retention ≈ 50 TB
Latency budget: users tolerate 2-3 seconds — this is not a latency problemThe numbers are small, which changes the priorities
500 payments per second fits comfortably on one database. There is no scaling problem here.
Every design decision below is therefore about correctness, not throughput — which is the opposite of most systems. If you find yourself sharding a payment database for performance, check the arithmetic first.
3. Define the Interface
charge(idempotency_key, amount, currency, payment_method, order_id)
→ { payment_id, status: pending | succeeded | failed, reason? }
get_payment(payment_id) → current status
refund(idempotency_key, payment_id, amount) → { refund_id, status }
Delivered to the caller, not returned by the call:
payment.succeeded / payment.failed / refund.completed (webhook)`idempotency_key` is the first argument, and it is required
Not a header. Not optional. The caller generates it once for one intended charge and sends the same key on every retry.
Consider what happens without it. A request times out. The caller does not know whether the card was charged — a timeout is silence, not a "no". It retries. If the first attempt did go through, the customer has now paid twice.
There is no way to fix that after the fact from inside this system, because two identical charges are indistinguishable from a customer legitimately buying the same thing twice. The only place it can be solved is the interface, by making the caller say "this is the same attempt, not a new one".
Section 5 covers how the key is enforced. This step is about it being required at all.
`charge` returns `pending`, not a final answer
It would be nicer to return succeeded or failed and be done. You cannot. A card network takes seconds, sometimes asks the cardholder for confirmation, and sometimes stops responding mid-request.
So the honest interface hands back a payment_id and a status that may still change, and the real outcome arrives later by webhook.
An interface that promised a final answer would have to lie whenever the network was slow — and a lie about money becomes a support ticket and a refund.
4. The Flow
The state machine
┌──────────┐
│ PENDING │
└────┬─────┘
┌──────┼──────┐
▼ ▼ ▼
SUCCEEDED FAILED UNKNOWN ──► requires investigation
│
▼
REFUNDED (partial or full)UNKNOWN is a real state and must be modelled
When the provider call times out, you genuinely do not know whether the money moved. Guessing either way is wrong: assume success and you may ship goods for free; assume failure and you may charge twice.
Record UNKNOWN, then resolve it by querying the provider using your idempotency key. If it cannot be resolved automatically, reconciliation catches it.
Designs that model only success and failure will, at scale, silently take money without recording it.
5. Idempotency
The single most important mechanism in the system.
POST /payments
Idempotency-Key: 8f2c-4a91-b3e7
first request → process, store the outcome under the key
retry → key found → return the SAME outcome, charge nothingIdempotency must be end to end
Deduplicating at your API is not enough. Your call to the provider can also be retried, so the key must be passed through to them as well.
Two layers of deduplication, one key. If either layer is missing, a retry at that layer double-charges.
The key must also be stored durably and written in the same transaction as the payment record. Keeping it only in a cache means a cache eviction reopens the double-charge window.
6. The Ledger
Payment state says what happened. The ledger says where the money is, and it uses double-entry bookkeeping — every movement is recorded twice, as a debit and a matching credit.
Customer pays £100 for an order:
account debit credit
──────────────────────────────────────
customer_receivable 100
revenue 100
Sum of debits must always equal sum of credits.Why double entry, in a computer, in this century
Because it makes errors detectable. If debits and credits do not balance, something is wrong — and you find out from the data itself rather than from a customer complaint.
A single balance column can be wrong silently forever. A ledger that must balance cannot.
The ledger is also append-only. Correcting a mistake means adding a reversing entry, never editing or deleting. That preserves the full history, which is both an audit requirement and the only way to answer "what did we think was true last Tuesday?"
| Rule | Reason |
|---|---|
| Append only | History must be reconstructable |
| Every entry balanced | Errors become detectable |
| Corrections are new entries | Never rewrite the past |
| Amounts in minor units as integers | Never floating point |
Never store money in a floating-point number
0.1 + 0.2 = 0.30000000000000004Floating point cannot represent most decimal fractions exactly. Errors accumulate across millions of transactions, and the resulting discrepancies are nearly impossible to trace.
Store integer minor units — 1050 for £10.50 — or use a decimal type. Always store the currency alongside the amount; an integer with no currency is meaningless.
7. Reconciliation
Even with everything above, your records and the provider's will occasionally disagree — a lost webhook, a timeout, a provider-side adjustment.
Nightly, fetch the provider's settlement report and compare it line by line with your ledger.
| Mismatch | Likely cause |
|---|---|
| They charged, we have no record | A lost response we never resolved |
| We recorded, they have nothing | A write that should have failed |
| Amounts differ | Currency handling or a partial refund |
Reconciliation is the safety net that makes the rest trustworthy
Every other mechanism reduces the chance of error. Reconciliation is what detects the errors that get through anyway.
A payment system without it will drift out of agreement with reality, and nobody will know until an audit or an angry customer.
Mismatches should alert a human. This is one of the few places where the correct response to an anomaly is a person looking at it, not an automatic retry.
8. What Was Traded Away
| Decision | Gained | Cost |
|---|---|---|
| Tokenised cards | Sensitive data never touches us | Dependent on the provider's availability |
| Write PENDING before charging | No untraceable money movement | An extra write on the critical path |
| Idempotency keys everywhere | Retries are safe | Keys must be stored durably |
| Append-only ledger | Full auditability, detectable errors | More storage; corrections are indirect |
| Strong consistency | Money is never lost or doubled | Cannot use eventual-consistency shortcuts |
| Nightly reconciliation | Errors are caught | Up to 24 hours to detect one |
Follow-on Questions
Related Reading
- Consistency and CAP — idempotency and why this path is CP
- Security — why card data is kept out of scope
- Reliability — timeouts, retries and circuit breakers
- Databases — transactions and isolation levels