Build Distributed Systems from Scratch

· Build Distributed Systems

3 min read Original article ↗

34 Tracks, 445 Challenges, 6 Projects

Build Distributed Systemsfrom Scratch

Learn distributed systems the hard way, by implementing real algorithms. Progress from basic message passing to consensus protocols, storage engines, and event-driven architectures through hands-on, Maelstrom-compatible challenges.

Track 1 and Mini-Dynamo are free. No card needed. See pricing

Build Real Systems

Implement actual distributed algorithms, from message passing to consensus protocols.

Progressive Learning

Start simple and build complexity. Each track builds on the previous one.

8 Languages

Solve challenges in Python, Go, Rust, C++, TypeScript, JavaScript, C, or Java.

Concept Explanations

Each task includes detailed explanations of the distributed systems concepts involved.

Learning Path

8 categories with 34 tracks and 445 tasks. Each track builds on concepts from previous ones, covering foundations through advanced distributed systems topics.

Foundations

Message passing, unique IDs, gossip protocols, logical and physical clocks, and TCP/gRPC.

Consensus

CRDTs, leader election, Raft log replication, a linearizable key-value store, and replication strategies.

Advanced

Sharding, distributed transactions, Byzantine fault tolerance and DHTs, workflow orchestration, event sourcing, and MapReduce.

Scalability

Caching strategies, proxies and API gateways, indexing, load balancing, message queues, and performance under load: admission control, backpressure, pooling and batching.

Storage

Write-ahead logs, LSM trees and B-trees, distributed file systems, and a Redis-style in-memory store.

Real Systems

Rebuild production systems: a Kafka-style streaming log, a ZooKeeper-style coordination service, and distributed search.

Data

Indexing at scale: B-tree and inverted indexes, and Bloom filters.

Operations

Job scheduling, distributed tracing and alerting, authentication and encryption, schema migrations, and resilience: deadlines, circuit breakers, bulkheads, retries and fallbacks.

Capstone Projects

Rebuild 6 real systems end to end, each with its own tracks, deterministic tests and diagrams.

6 tracks, 43 tasks

Consistent hashing, quorums with hinted handoff, vector clocks, Merkle anti-entropy, gossip membership and a storage engine.

Explore

Mini-Kafka

6 tracks, 40 tasks

Segments and log compaction, the ISR and leader epochs, consumer groups, exactly-once transactions and KRaft metadata.

Explore

Mini-Spanner

6 tracks, 33 tasks

Multi-Paxos with leader leases, TrueTime and commit wait, wound-wait transactions and an external consistency checker.

Explore

Mini-etcd

9 tracks, 48 tasks

Raft with Pre-Vote and joint consensus, a WAL with snapshots, MVCC, watches, leases and a linearizability checker.

Explore

Mini-Temporal

9 tracks, 40 tasks

Event histories and deterministic replay, activity retries and timeouts, durable timers, signals and updates, sagas, sharding and simulation testing.

Explore

Mini-VectorDB

9 tracks, 35 tasks

HNSW, IVF and product quantization, segments with deletes and compaction, filtered and hybrid search, sharded search paths, consistency levels and replicas.

Explore

Learning by Building

Distributed systems are notoriously difficult to understand through passive learning. The only way to truly grasp concepts like consensus, eventual consistency, and partition tolerance is to implement them yourself.

This platform takes inspiration from the Fly.io Gossip Glomers challenges and the Maelstrom testing framework. You will write code that runs against simulated distributed environments, complete with network partitions, message loss, and timing failures.

Each challenge is designed to teach a specific concept through implementation. By the time you complete all tracks, you will have built everything from simple echo servers to fully functional Raft consensus implementations, distributed storage engines, and event-driven architectures.