Re-reading "Designing Data-Intensive Applications" Five Years Later
Most technical books age like milk, especially anything with a chapter on distributed systems. Pulling my dog-eared copy of Martin Kleppmann’s Designing Data-Intensive Applications off the shelf after five years, I expected half the tool names to feel dated. A few specific Apache ecosystem references certainly show their 2017 vintage, yet the underlying mental models have aged remarkably well.
Chapter 3 still pays rent every month
The comparison between B-trees and LSM-trees (SSTables plus memtables and compaction) remains the clearest twenty-five pages written on storage engines. The first time I read the book early in my career, I raced straight toward the consensus and replication chapters because Raft and Paxos sounded exciting. Re-reading it with more production scars, the quiet parts of Part I stood out much more:
- Write amplification during background LSM compaction causing tail-latency spikes on cheap NVMe drives.
- Why an append-only write-ahead log (
fsyncdiscipline, checksummed frames, crash recovery) shows up everywhere once you notice it—inside SQLite, PostgreSQL, filesystem journals, and message brokers alike. - Schema evolution rules (forward vs. backward compatibility) that prevent rolling deploys from painting a team into a corner.
The case for staying on one machine longer
One takeaway that sharpened on a second read is how many engineering headaches disappear when you resist splitting state across network boundaries prematurely. Modern single-socket servers have enough RAM and NVMe throughput to run workloads that needed a small cluster a decade ago, and local ACID transactions eliminate entire categories of partial-failure bugs that no retry loop quite heals.
Distributed state isn’t an upgrade badge; it’s a tax you pay only when geography, availability targets, or sheer dataset size leave you no single-node choice.
Whenever I catch myself sketching a message queue and two separate datastores for a modest side project, opening Chapter 7 on weak isolation levels and write skew is still the fastest cure.