Design a Payment System: System Design Interview Guide

Core features:
- Process card payments (charge, authorize, capture)
- Handle refunds (full and partial)
- Manage payment methods (store cards securely)
- Webhook notifications for payment events
- Transaction history and reporting

Out of scope:
- Multi-currency / forex
- Subscription billing (higher-level concern)
- Marketplace payouts (adds complexity)
- Physical card issuance

Scale:
- 10M transactions per day
- Peak: 1,000 transactions/second (Black Friday)
- 99.999% durability (never lose a transaction)

Performance:
- Payment processing < 3 seconds end-to-end
- 99.99% availability
- Zero double-charges (idempotency is non-negotiable)

Compliance:
- PCI-DSS Level 1 (card data handling)
- SOX compliance (financial reporting)
- AML (anti-money-laundering) screening

Key insight: Correctness > speed. A slow payment is acceptable.
A wrong payment (double charge, lost money) is catastrophic.

┌─────────────────────────────────────────────────────────────┐
│                     MERCHANT API                             │
│            (REST API with idempotency keys)                  │
└───────────────────────┬─────────────────────────────────────┘
                        │
                        ▼
┌─────────────────────────────────────────────────────────────┐
│                   API GATEWAY                                │
│        Auth, rate limiting, idempotency check                │
└───────────────────────┬─────────────────────────────────────┘
                        │
         ┌──────────────┼──────────────┐
         │              │              │
         ▼              ▼              ▼
   ┌──────────┐  ┌──────────┐  ┌──────────┐
   │ Payment  │  │  Fraud   │  │  Ledger  │
   │ Service  │  │  Engine  │  │ Service  │
   └────┬─────┘  └──────────┘  └────┬─────┘
        │                           │
        ▼                           ▼
   ┌──────────┐              ┌──────────┐
   │ Payment  │              │  Ledger  │
   │ Gateway  │              │ Database │
   │ (Visa,   │              │ (append- │
   │  MC)     │              │  only)   │
   └──────────┘              └──────────┘

Problem:
1. Client sends payment request
2. Server processes payment, charges card
3. Network fails BEFORE response reaches client
4. Client retries (same payment)
5. Without idempotency: customer charged TWICE

Solution: Idempotency keys
- Client sends a unique idempotency_key with each request
- Server stores: idempotency_key → {status, response}
- On retry: if key exists, return cached response
- Never process the same key twice

Implementation:
POST /v1/payments
Headers:
  Idempotency-Key: "order_123_payment_attempt_1"
  Authorization: Bearer sk_...

Server logic:
1. Check idempotency store for this key
2. If found AND completed → return cached response
3. If found AND in-progress → return 409 Conflict
4. If not found → create entry with status "processing"
5. Process payment
6. Update entry with response
7. Return response

Storage:
- Redis for hot lookups (TTL: 24-48 hours)
- PostgreSQL for permanent record
- Key must be unique per merchant + operation

The ledger is the source of truth for all money movement.

Double-entry bookkeeping principle:
Every transaction creates TWO entries that sum to zero.
Money is never created or destroyed, only moved.

Example: Customer pays $100 to merchant
Entry 1: Customer account  -$100 (debit)
Entry 2: Merchant account   +$100 (credit)
Sum = $0 ✓

Example: Merchant refunds $30
Entry 1: Merchant account   -$30 (debit)
Entry 2: Customer account   +$30 (credit)
Sum = $0 ✓

Database schema:
CREATE TABLE ledger_entries (
  id              BIGINT PRIMARY KEY,
  transaction_id  UUID NOT NULL,        -- groups related entries
  account_id      BIGINT NOT NULL,
  amount          DECIMAL(20,2) NOT NULL, -- positive=credit, negative=debit
  currency        CHAR(3) NOT NULL,
  entry_type      VARCHAR(20) NOT NULL, -- 'payment', 'refund', 'fee'
  created_at      TIMESTAMP NOT NULL,
  -- APPEND-ONLY: no updates or deletes allowed
);

-- Account balance = SUM of all ledger entries for that account
-- Never store balance as a mutable field (derive from ledger)

Critical constraints:
- Ledger is APPEND-ONLY (never update or delete entries)
- Every transaction must have entries that sum to zero
- Use database transactions for atomicity
- Reconciliation job runs hourly to verify sums

Full payment flow (authorize + capture pattern):

1. AUTHORIZE
   Merchant → Our API → Fraud Check → Payment Gateway → Card Network → Bank
   Bank reserves $100 on customer's card
   Returns authorization code
   No money moves yet

2. CAPTURE (within 7 days)
   Merchant → Our API → Payment Gateway → Card Network
   Bank transfers reserved $100
   Money moves to settlement account
   Ledger entries created

3. SETTLEMENT (next business day)
   Batch process: aggregate captured payments
   Transfer funds to merchant's bank account
   Create ledger entries for settlement

Why separate authorize and capture?
- E-commerce: authorize at checkout, capture at shipment
- Hotels: authorize at booking, capture at checkout
- Reduces refund volume (cancel authorization instead)

Payment states:
CREATED → AUTHORIZED → CAPTURED → SETTLED
                ↓
           VOIDED (cancel before capture)
                          ↓
                     REFUNDED (after capture)

Fraud checks run before authorization:

Rule-based checks (fast, synchronous):
- Velocity: > 5 transactions in 1 minute → flag
- Amount: single transaction > $10,000 → manual review
- Geography: transaction from new country → step-up auth
- Card testing: many small amounts → block

ML-based scoring (parallel):
- Feature vector: amount, merchant category, time of day,
  device fingerprint, IP geolocation, transaction history
- Model outputs risk score 0-100
- Threshold: > 80 → decline, 50-80 → 3DS verification, < 50 → approve

Response time budget:
- Total fraud check must complete in < 200ms
- Rule engine: < 10ms
- ML scoring: < 100ms (pre-loaded model)
- Parallel execution, take max of both

Payment systems use specific reliability patterns:

1. Saga Pattern (for multi-step transactions)
   - Each step has a compensating action (undo)
   - If step 3 fails: undo step 2, undo step 1
   - Example: authorize → charge → settle. If charge fails, void authorization

2. Outbox Pattern (for reliable event publishing)
   - Write to database AND outbox table in same transaction
   - Separate process reads outbox and publishes events
   - Guarantees: if payment saved, webhook WILL be sent

3. Dead Letter Queue
   - Failed webhook deliveries go to DLQ
   - Retry with exponential backoff (1min, 5min, 30min, 2hr, 24hr)
   - Alert after all retries exhausted

4. Circuit Breaker
   - If payment gateway is down, fail fast
   - Don't queue payments (money is time-sensitive)
   - Route to backup payment processor if available