Design an Object Storage Service

S3 System Design

Overview

An object storage service stores opaque blobs — an image, a video, a database backup, a log archive — under a name, and hands them back on demand, at effectively unlimited scale and with a durability guarantee so strong that losing an object is a once-in-a-hundred-billion-years event per object. This is Amazon S3, Google Cloud Storage, Azure Blob. You PUT an object under a bucket and a key, and later GET it back. That is the entire surface a user sees. Everything hard is underneath.

It is labeled "Hard" because the two headline requirements pull in different directions and the naive answers to each are wrong. Durability — the industry target is eleven nines (99.999999999% annual object survival) — cannot come from a single disk or even a single machine; it demands redundancy across independent failure domains, and the choice of redundancy scheme (3× replication vs erasure coding) is a storage-cost decision worth many millions of dollars at exabyte scale. Scale — trillions of objects under a flat namespace — means the thing that maps key → bytes is itself a massive distributed system, and it must be kept separate from the bytes.

The single most important structural idea: split a metadata service (which maps bucket/key → object id → the list of shard/chunk locations, plus size, ETag, and version) from the data storage nodes that hold the actual bytes. The metadata is small, hot, and needs strong consistency; the bytes are enormous, cold-ish, and need cheap massive durability. Confuse the two — store bytes in the metadata database, or model keys as a filesystem tree — and the design collapses.

In this walkthrough you'll scope the service, size it, model the data, design the API (including multipart upload), draw the read and write paths, then go deep on durability math, the metadata service at scale, and multipart upload with strong read-after-write consistency.

Out of scope (say so): the CDN edge caching layer in front of the store (that's a separate content-delivery problem), fine-grained IAM policy evaluation, lifecycle/tiering to cold archival storage beyond a mention, cross-region replication policies, and the SQL-like query layer (S3 Select / Athena). We keep the core: PUT/GET/DELETE an object, buckets and keys, multipart upload, durable placement, and consistent metadata.

Functional requirements

• PUT an object. Upload bytes under a bucket + key. Overwrites any existing object at that key, creating a new version.
• GET an object. Retrieve the full bytes — or a byte range — for a bucket + key. Reads the current version by default.
• DELETE an object. Remove the object at a key (logically; the bytes are garbage-collected later once no version references them).
• Multipart upload for large objects. Initiate an upload, PUT many parts in parallel, then commit — so a 5 TB object isn't one fragile HTTP request.
• List objects in a bucket. Paginated listing by key prefix (this is how the flat namespace fakes "folders").
• Bucket operations. Create and delete a bucket; a bucket owns a region, an access policy, and its namespace.

Out of scope (state it): CDN edge caching, detailed IAM policy language, archival tiering mechanics, cross-region replication rules, and any query-over-objects layer.

Non-functional requirements

• Durability is the headline — target ~11 nines. An object the user trusted us with must survive disk failures, machine failures, rack failures, and correlated failures. This is achieved by redundancy across independent failure domains, plus continuous background repair.
• Strong read-after-write consistency for new objects. After a successful PUT returns, any subsequent GET of that key must return the new object — no "you might read a stale version for a while." Modern S3 is strongly consistent, and we'll match that.
• High availability. The service should keep serving reads and writes through the loss of individual nodes, racks, or even a datacenter within a region. Availability and durability are related but distinct — an object can be safe (durable) yet temporarily unreachable (unavailable).
• Massive scale, low per-byte cost. Exabytes of data, trillions of objects. The storage overhead of the redundancy scheme is a first-class cost lever: 3× replication is 200% overhead; erasure coding can hit similar durability at ~40% overhead. That difference is enormous at exabyte scale.
• Bit-rot protection. Disks silently corrupt bits over time. Every stored shard needs a checksum, and a background scrubber must continuously verify and repair.
• Elastic throughput. A single hot bucket or hot key shouldn't cap the whole system; the design must spread load.

Estimations

State assumptions first; the goal is to justify the metadata/data split and the erasure-coding choice — not to be exact.

Rounded assumptions

• Objects stored: ~1 trillion objects total.
• Average object size: ~1 MB (heavily bimodal — billions of tiny thumbnails and a long tail of multi-GB videos and backups; the average is small, which matters for metadata sizing).
• Total logical data: 1e12 objects × 1 MB ≈ ~1 exabyte (EB) logical, before redundancy overhead.
• Physical footprint: with EC(10,4) at ~1.4× storage (40% overhead) that's ~1.4 EB physical; with 3× replication it would be ~3 EB — the ~1.6 EB difference is the entire reason erasure coding exists.
• Request throughput: assume ~10M requests/sec at peak across the fleet, read-heavy (say ~10:1 read:write), so ~9M GET/sec and ~1M PUT/sec.
• Metadata rows: one row per object (+ per version) ⇒ ~1 trillion metadata rows, each ~a few hundred bytes ⇒ hundreds of TB of metadata — small next to 1 EB of bytes, but far too big for one database. It must be sharded.

How the numbers affect the design

Core entities / data model

The model keeps the map (metadata) separate from the bytes (data). Here's what each side holds.

The heart of the model is the object metadata row. It maps a (bucket, key, version) to an object_id and, crucially, to a placement — the list of where the shards live (which data nodes / disks hold which shard of this object). The metadata row is small (a few hundred bytes) and never contains the object bytes — only the pointers to them and the bookkeeping (size, ETag, content type).

The key is a flat string, not a path. photos/2026/cat.jpg is a single opaque key in a giant per-bucket map; the slashes mean nothing to the store. There are no directory nodes, no inodes, no "move folder" operation. "Folders" in a UI are a fiction produced by prefix listing (list keys starting with photos/2026/). This is the single biggest structural difference from a filesystem, and getting it right is what interviewers watch for.

The Shard/Chunk rows record the physical layout of one object's redundant pieces: for an erasure-coded object, the k data shards + m parity shards, each with its data node, disk, offset, and per-shard checksum. This is what the durability and repair logic reads.

API design

Six operations carry the system: PUT, GET, DELETE, and the three-step multipart flow (initiate, upload part, complete). Label request Body: and response Returns:.

PUT an object

PUT /{bucket}/{key}
Body: <raw object bytes>
Headers: Content-Length, Content-Type, optional Content-MD5
200 OK
Returns: { "etag": "d41d8cd9...", "version_id": "v-01H..." }

The gateway streams the bytes, computes shards (replicas or erasure-coded), writes them to data nodes across failure domains, and — only after the shards are durably stored — writes the metadata row. The etag is a content hash the client can verify against. The PUT is the atomic point at which the new version "becomes" current (deep dive 3).

GET an object

GET /{bucket}/{key}
Headers: optional Range: bytes=0-1048575
200 OK (or 206 Partial Content for a range)
Returns: <raw object bytes> + ETag, Content-Length, Last-Modified headers

The gateway reads the metadata row for the current version, gets the placement, reads enough shards to reconstruct (all replicas-worth for replication, or any k of k+m for EC), verifies checksums, and streams bytes back. Range reads fetch only the needed shard offsets.

DELETE an object

DELETE /{bucket}/{key}
204 No Content

A logical delete: with versioning enabled, writes a delete marker version (so the key now reads as "not found") rather than immediately erasing bytes. The physical shard bytes are reclaimed later by a garbage-collection sweep once no live version references the object's shards. This keeps DELETE fast and makes it safe to retry.

Multipart upload — initiate

POST /{bucket}/{key}?uploads
200 OK
Returns: { "upload_id": "mpu-01H..." }

Creates a MultipartUpload record. No object is visible yet.

Multipart upload — upload a part

PUT /{bucket}/{key}?partNumber=3&uploadId=mpu-01H...
Body: <bytes for part 3>   (each part ≥ 5 MB except the last, can run in parallel)
200 OK
Returns: { "etag": "<part-3-etag>" }

Each part is stored as its own set of durable shards and recorded under the upload_id. Parts can be uploaded in parallel, out of order, and retried independently. Nothing is visible under the key yet.

Multipart upload — complete

POST /{bucket}/{key}?uploadId=mpu-01H...
Body: { "parts": [ {"part_number":1,"etag":"..."}, {"part_number":2,"etag":"..."}, ... ] }
200 OK
Returns: { "etag": "<final-multipart-etag>", "version_id": "v-01H..." }

The server verifies every listed part exists, then writes a single metadata row whose placement stitches the parts' shards together in order. This one metadata write is the atomic flip that makes the whole object visible at once (deep dive 3). An Abort variant deletes the parts and the upload record.

List objects by prefix

GET /{bucket}?prefix=photos/2026/&delimiter=/&continuation-token=...&max-keys=1000
200 OK
Returns: { "keys": [...], "common_prefixes": [...], "next_continuation_token": "..." }

Cursor-based pagination over the sorted key range for the prefix — the flat-namespace way to render "folders." delimiter=/ rolls up everything under a common prefix into a single common_prefixes entry, faking one directory level.

High-level architecture

Two flows matter: a write path (bytes → shards → data nodes → metadata commit) and a read path (metadata lookup → shard reads → reconstruct → stream). A stateless request router / gateway fronts both. We'll walk each, then combine.

1. Write path — PUT

The gateway shards the bytes and writes them durably before it records the metadata, so the metadata never points at bytes that aren't there.

flowchart LR
    Client["Client"] -->|"PUT bytes"| GW["Request Router / Gateway"]
    GW -->|"encode: k data + m parity shards"| Enc["Erasure Coder"]
    Enc -->|"write shard 1..k+m across failure domains"| DN[("Data Nodes")]
    GW -->|"only after shards durable:\nwrite object row + placement"| Meta["Metadata Service"]
    Meta -->|"sharded by bucket/key hash"| MetaDB[("Metadata Store")]
    GW -->|"200 OK + etag + version_id"| Client

1. The gateway receives the bytes and passes them to the encoder, which produces the redundant pieces — either N replicas or k data + m parity shards.
2. It writes those shards to data nodes chosen so that no single failure domain (rack, power domain, datacenter) holds enough shards to matter — a rack loss must never drop you below k surviving shards.
3. Only after enough shards are durably acknowledged does the gateway write the object metadata row (with the placement) to the metadata service.
4. That metadata write is the atomic commit; the PUT returns after it succeeds.

2. Read path — GET

flowchart LR
    Client["Client"] -->|"GET bucket/key"| GW["Request Router / Gateway"]
    GW -->|"look up current version\n+ placement"| Meta["Metadata Service"]
    Meta -->|"shard locations"| GW
    GW -->|"read k shards (any k of k+m)"| DN[("Data Nodes")]
    DN -->|"shards + per-shard checksums"| GW
    GW -->|"verify checksums, reconstruct, stream"| Client

1. The gateway resolves (bucket, key) to the current version's metadata row and its placement.
2. For an erasure-coded object it reads any k of the k+m shards (often the k fastest-responding, ignoring slow/failed nodes); for replication it reads any one healthy replica.
3. It verifies each shard's checksum (catching bit-rot), reconstructs the original bytes if needed, and streams them back. A range read touches only the shards covering the requested byte range.

3. Combined architecture

flowchart LR
    Client["Client"] --> GW["Stateless Gateways\n(request router)"]
    GW -->|"metadata read/write"| Meta["Metadata Service\n(sharded, strongly consistent)"]
    Meta --> MetaDB[("Metadata Shards")]
    GW -->|"shard read/write"| DN[("Data Nodes\n(across failure domains)")]
    Scrub["Scrubber / Repair"] -.->|"re-read + checksum"| DN
    Scrub -.->|"lost shard? rebuild from surviving k"| DN
    Placement["Placement Service"] -->|"pick nodes across domains"| GW
    GC["Garbage Collector"] -.->|"reclaim shards of deleted/dead versions"| DN

The gateways are stateless and horizontally scaled. The metadata service is the sharded, strongly-consistent hub that owns the key→placement map. The data nodes are dumb byte stores. Three background services keep the bytes healthy: a placement service that spreads shards across failure domains, a scrubber/repair loop that continuously re-verifies checksums and rebuilds lost shards, and a garbage collector that reclaims shards from deleted versions and orphaned uploads. These background loops are where durability actually lives — redundancy alone decays without active repair.

How placement picks nodes. The placement service keeps a live data-node registry — each node's failure domain, free capacity, and health, gossiped or heartbeated in. For each write it selects k+m target nodes by capacity-weighted random choice (nodes with more free space are likelier picks, so writes drain away from hot or nearly-full nodes) while enforcing the failure-domain spread. Adding a node just registers fresh capacity that naturally attracts new writes; removing or draining one marks it ineligible for new placements and hands its existing shards to the repair loop to rebuild elsewhere. Existing objects aren't reshuffled — only new writes and repairs respond to the changed topology.

Deep dives

1. Durability — erasure coding vs replication

This is the mathematical heart of an object store, and the place candidates most often hand-wave. The requirement is ~11 nines of durability at the lowest storage cost. Two schemes deliver redundancy; the choice is a cost-vs-repair tradeoff.

Replication. Store N identical copies on N different nodes across N failure domains. To lose the object you must lose all N copies before any is repaired.

Erasure coding (Reed-Solomon). Split the object into k data shards, compute m parity shards via Reed-Solomon, and store all k+m shards on different nodes across failure domains. Any k of the k+m shards reconstruct the original. To lose the object you must lose more than m shards before repair.

store 1 MB object → 3 MB physical
storage overhead  → 200% (you pay for 3× the logical bytes)
survives          → loss of any 2 of 3 copies
repair            → copy one surviving replica to a new node (cheap: 1 read, 1 write, no compute)

store 1 MB object → split into 10 data shards (0.1 MB each) + 4 parity shards
physical stored   → 14 × 0.1 MB = 1.4 MB
storage overhead  → 40% (vs 200% for 3× replication)
survives          → loss of any 4 of 14 shards simultaneously
durability        → TUNABLE to meet or exceed 3× replication — raise m and repair
                    speed to buy more nines. Tolerating 4 losses vs 2 is necessary
                    but not sufficient; repair speed is the real lever.

Shard placement across failure domains. Durability nines assume failures are independent. They aren't if all 14 shards sit in one rack — a single top-of-rack switch or power domain failure takes them all, and your 11 nines evaporate. So the placement service spreads the k+m shards across distinct failure domains (different racks, power domains, ideally datacenters within the region) such that no single domain holds more than m shards — and prudent placement caps well below m, since losing exactly m shards leaves zero margin until repair completes. Losing an entire rack then costs you a handful of shards, and the object still reconstructs with room to spare.

Repair and scrubbing — durability is active, not static. Redundancy decays: disks die, bits rot. Two background loops keep the guarantee true:

• Scrubbing. Continuously re-read every shard and re-verify its stored checksum. A checksum mismatch means silent bit-rot; the scrubber treats that shard as lost and triggers repair. Without scrubbing, corruption accumulates undetected until a read fails.
• Repair. When a shard is lost or corrupt, rebuild it: read any k surviving shards, decode the missing shard, write it to a fresh node in a proper failure domain. The repair speed is what actually sets your durability — if you can repair a lost shard in minutes, the probability of losing m+1 shards within that window is what gives you the 11 nines. Slow repair, not the code parameters, is usually the real durability risk.

2. The metadata service at scale

The metadata service maps bucket/key → object_id → placement (+ size, etag, version). It is small per row but there are ~1 trillion rows, it takes ~10M lookups/sec, and it must be strongly consistent (deep dive 3 depends on it). No single database holds this; it must be sharded — and sharding a flat namespace with arbitrary keys has sharp edges.

Sharding a flat namespace. The map is (bucket, key) → row. How you partition it decides your hot-spot behavior.

Hot buckets and hot keys. Even with hashing, two skew problems remain:

• Hot bucket. One bucket receives a disproportionate share of traffic. Because hashing includes the key, a hot bucket's keys still spread across shards — the bucket isn't pinned to one shard. But if a single bucket needs more throughput than the shards it touches can supply, you split its keyspace across more shards, or dynamically re-partition (range-split a hot shard into two).
• Hot key. A single object read millions of times/sec (e.g. a viral image's metadata row). One row on one shard can't absorb that. Mitigate by caching the metadata row in a read-through cache in front of the metadata service — the placement rarely changes, so it caches beautifully — and by serving the bytes through replicas/CDN. Strong consistency still holds because a PUT invalidates the cached row.

Why the metadata store is strongly consistent and replicated. Losing a metadata shard orphans every object it maps — the bytes survive on data nodes but are unfindable. So the metadata store is itself replicated (e.g. a consensus-replicated store like a Paxos/Raft group per shard) for both durability and strong consistency. This is exactly the kind of small-but-critical data where you pay for consensus; the bytes, being content-verifiable and erasure-coded, don't need consensus, only durable placement.

3. Multipart upload and strong read-after-write consistency

Two requirements converge here: large objects must upload in resumable parallel parts, and a new object must be strongly consistent — the instant a PUT (or multipart complete) returns success, every GET sees the new object. The mechanism that delivers both is the same: a single atomic metadata flip.

The multipart flow. A 5 TB object as one HTTP PUT is absurd — one dropped packet near the end wastes hours. Multipart splits it:

sequenceDiagram
    participant C as Client
    participant GW as Gateway
    participant DN as Data Nodes
    participant M as Metadata Service

    C->>GW: POST ?uploads (initiate)
    GW->>M: create MultipartUpload(upload_id)
    GW-->>C: upload_id
    par upload parts in parallel
        C->>GW: PUT part 1 (bytes)
        GW->>DN: write part-1 shards (durable)
        GW-->>C: etag part 1
        C->>GW: PUT part 2 (bytes)
        GW->>DN: write part-2 shards (durable)
        GW-->>C: etag part 2
    end
    C->>GW: POST complete { parts: [...] }
    GW->>M: verify parts, write ONE object row (placement = parts in order)
    Note over M: this single write is the atomic flip — object now visible
    GW-->>C: 200 + etag + version_id

1. Initiate creates a MultipartUpload record and returns an upload_id. Nothing is visible under the key.
2. Upload part stores each part as its own durable set of shards, keyed under upload_id + part_number. Parts go in parallel, out of order, and each retries independently — a failed part re-uploads just that part.
3. Complete verifies every listed part is present, then writes one object metadata row whose placement references the parts' shards in order.

Why the metadata flip makes it atomic and strongly consistent. Throughout the upload, the parts' shard bytes exist on data nodes, but there is no object row for the key — a GET returns 404. The complete step's single metadata write is the only thing that makes the object visible, and it either commits or it doesn't. There is no intermediate state where half the object is readable. Because the metadata service is strongly consistent (deep dive 2 — consensus-replicated per shard), the moment that write commits, the very next GET routed to that shard reads the new row. That's read-after-write consistency: the object becomes visible at exactly the instant its metadata row commits, not before, not gradually.

The same principle governs a plain (non-multipart) PUT: write shards durably, then commit one metadata row; the object exists the instant that row commits.

The overwrite atomic flip. Each key carries a per-(bucket, key) current-version pointer in the metadata shard. Overwriting writes a brand-new version row (new shards, new version_id) and then flips that single pointer to it in one strongly-consistent commit. Because a GET resolves the current version by reading that pointer, a concurrent reader sees either the entirely-old version or the entirely-new one — never a half-written mix, and never a torn pointer. The old version's row and shards stay intact behind the pointer until GC.

Versioning lifecycle. When versioning is enabled, each PUT/overwrite mints a monotonic version_id (e.g. a timestamp-ordered ULID) and appends a row rather than replacing one; the current-version pointer advances to the newest. GET ?versionId= reads a specific version; a versioned LIST walks all version rows for a key, newest first. A DELETE writes a delete marker version — the current pointer moves to it, so an unqualified GET returns 404, but prior versions remain retrievable by id (and a delete marker can itself be deleted to "undelete"). Shard bytes are reclaimable only when no version row references them: GC deletes a version's shards after that version is explicitly deleted or aged out by lifecycle policy, never merely because a newer version exists.

Cleaning up abandoned uploads. Parts uploaded for an upload that is never completed (client crashes, forgets to abort) are orphaned shards consuming space with no object row referencing them. A lifecycle rule + garbage collector reclaims parts of multipart uploads older than N days — the same GC that reclaims shards of deleted versions.

Tradeoffs & bottlenecks

• Replication vs erasure coding. Replication is simple, gives fast single-copy reads, and cheap copy-only repair — but 200% overhead. Erasure coding cuts overhead to ~40% and tolerates more concurrent failures — but repair reads k shards and burns CPU, and small objects fragment. The production answer is both: EC for the large/cold bulk tier, replication for small/hot objects and the metadata store.
• Small-object overhead. Erasure-coding a 4 KB object into 10 sub-KB data shards plus 4 parity is pathological — tiny shards dominated by fixed per-shard and per-object-metadata costs. Mitigate by replicating small objects, or by packing many small objects into one larger EC-coded container blob and storing each object's byte-offset in its metadata row; a GET then does a range read into the shared blob. Packing amortizes the EC fixed cost across many objects at the price of more complex GC (a slot frees only when its object is deleted).
• Metadata sharding: hash vs range. Hashing gives uniform writes but scatter-gather listing; range gives cheap listing but ordered-key write hotspots. Hash for the dominant point-operation path, add a secondary sorted index for listing.
• Durability vs repair traffic. More parity (larger m) buys more nines but more parity to compute and store; faster repair buys more nines but consumes read bandwidth and CPU that competes with user traffic. Repair speed, more than code parameters, usually sets real-world durability.
• Consistency cost on metadata. Strong read-after-write consistency requires the metadata commit to be consensus-replicated and strongly consistent, which caps per-shard write throughput and adds commit latency — paid on the small metadata, never on the bulk bytes, which is why the split is worth it.
• Hot key/hot bucket. A single viral object's metadata row or a single high-traffic bucket can exceed one shard's capacity; cache metadata rows, split hot buckets across shards, and serve bytes through replicas/CDN.

Extensions if asked

If the interviewer wants to go beyond the core store, the natural extensions:

• Storage tiering / lifecycle. Move objects not accessed in N days to a cheaper, colder tier (wider erasure codes, slower media, or offline archival) via lifecycle policies. Colder tiers trade retrieval latency for lower cost per byte.
• Versioning and object lock. Keep every version of a key (the model already writes version rows); add immutable "object lock" / WORM retention for compliance so a version can't be deleted before a retention date.
• Cross-region replication. Asynchronously copy objects to a second region for disaster recovery or lower read latency elsewhere — eventually consistent across regions by design; name that it weakens the strong-consistency guarantee to same-region only.
• Server-side encryption. Encrypt shards at rest with per-object data keys wrapped by a KMS master key; the metadata row stores the wrapped key. Largely orthogonal to the storage design.
• Access control / presigned URLs. Bucket policies and IAM for authorization, plus time-limited presigned URLs that let clients PUT/GET directly without proxying credentials — a whole subsystem worth flagging rather than designing.

What interviewers look for & common mistakes

What interviewers usually reward:

• The metadata/data split — a separate, sharded, strongly-consistent metadata service mapping key → placement, over dumb data nodes holding the bytes. This is the structural spine of the whole answer.
• The flat namespace — knowing keys are one flat string map per bucket, that "folders" are prefix listing, and that there is no directory tree or folder-rename operation.
• The erasure-coding math — being able to say EC(10,4) is ~40% overhead vs 3× replication's 200%, that any k of k+m shards reconstruct, and that EC's cost is higher repair traffic (read k shards + decode) — not just naming "erasure coding" as a buzzword.
• Durability as active — placement across independent failure domains, per-shard checksums for bit-rot, and a background scrubber + fast repair, with the insight that repair speed sets the real durability.
• Strong read-after-write via a single atomic metadata commit — object visibility gated by one strongly-consistent metadata write, distinct from byte durability, distinct from byte replication convergence.
• Multipart upload — parallel resumable parts, nothing visible until a single complete step stitches them into one object row.

Before you finish, do a quick mistake check:

• Did you split a strongly-consistent metadata service from the data nodes, and keep bytes out of the metadata DB?
• Did you model keys as a flat namespace (prefix listing), not a filesystem directory tree?
• Did you present the erasure-coding vs replication tradeoff with real overhead numbers (40% vs 200%) and explain EC's higher repair cost?
• Did you spread shards across failure domains and include checksums + background scrubbing/repair — and connect repair speed to durability nines?
• Did you achieve strong read-after-write consistency through a single atomic metadata commit, and explain why byte durability is answered before that commit?
• Did you design multipart upload so nothing is visible until the complete step, and handle abandoned-part cleanup?