AWS S3 Architecture: Petabyte-Speed Performance on HDD
AWS S3 handles over 400 trillion objects, 150 million requests per second, and peak traffic exceeding 1 petabyte per second. At its core are hard disk drives—despite their limitations in IOPS (around 120) and latency up to 16 ms for 0.5 MB random reads. The economics of HDD storage enable exabytes of data to be stored at the lowest possible cost: capacity has increased 7.2 million times, while cost per byte has dropped by a staggering 6 billion times over decades.
HDD physics dictate performance: track seek time averages 8–9 ms, rotational latency is 4 ms, and transfer time is 2.5 ms. Sequential access is efficient (up to 300 MB/s), but random access is capped at 32 MB/s per drive due to mechanical head movement.
Parallelism via Erasure Coding
S3 overcomes HDD constraints through massive parallelism. Data is split into shards using an erasure coding (EC) scheme of 5 of 9: 5 data shards plus 4 parity shards. This delivers 1.8x overhead instead of 3x with replication, provides resilience against 4 failures, and offers 5 read sources for improved throughput.
Benefits of EC:
- Load distribution across 9 shards minimizes hot spots.
- Parallel reading from 5 shards accelerates access.
- Sharding enables hedged queries to bypass latency delays.
A 1 TB file on a single HDD achieves 300 MB/s; on 20,000 drives, it scales to terabytes per second by aggregating bandwidth.
Parallelism Across All Layers
Parallelism is implemented across three planes:
- Client frontend: HTTP connection pools distribute requests across multiple endpoints, avoiding proxy and cache overload.
- Frontend backend: EC shards objects across thousands of disks in ShardStore (an LSM-tree built on log-structured storage).
- PUT/GET operations: Multipart uploads for writes, byte-range GETs for reads—split into 100+ streams of 10 MB/s instead of one gigabit stream.
Writes are sequential (log-like), while reads require sharding randomization to avoid queue buildup.
Fighting Hot Spots
With tens of millions of disks and hundreds of millions of shards per second, maintaining even load distribution is critical. A single disk limited to 32 MB/s becomes a bottleneck, compounded by rebalancing overhead.
S3’s strategy:
- Randomization: sharding hashing ensures uniform resource consumption.
- Rebalancing: continuous data migration without downtime.
- Horizontal scaling: adding disks without impacting availability.
The LSM-based ShardStore structure is optimized for append-only logs—where PUTs are fast and GETs use parallel index lookups.
Key Takeaways
- S3 achieves 1+ PB/s on HDDs using 5/9 EC and parallelism across 100+ streams.
- Average random I/O latency: 16 ms per 0.5 MB, mitigated by 5 read sources.
- Economics: 1.8x EC overhead vs. 3x replication saves exabytes in storage costs.
- Randomization + rebalancing prevent cascading failures from hot spots.
- Backend LSM-tree maximizes HDD sequential performance.
— Editorial Team
No comments yet.