Distributed Key-Value Store

Overview

Architecture

Engineering Challenges

• Even key distribution as nodes join or leave — solved with virtual nodes (5 per physical node) on the hash ring instead of one point per node, which avoids hot spots.
• Durability across crashes without a full embedded database — every PUT/DELETE is written and flushed to a write-ahead log before being applied in memory, with periodic snapshots (every 30s) to bound WAL growth and speed up recovery.
• Being honest about primary-node failure — the primary is a statically configured coordinator, not a Raft-elected leader, so writes fail fast with a 503 when it's down instead of silently succeeding without real consensus.

Design Decisions

• Configured primary over full Raft/Multi-Paxos leader election: enough to demonstrate replication and failure handling without the log-compaction and split-brain complexity a production system would need.
• Write-ahead log plus periodic snapshots over a full embedded database, trading some read performance for simple, auditable recovery semantics.

Reliability & Scalability

• Replication factor of 3: each write propagates from the primary to the next two nodes on the ring.
• Reads keep succeeding from surviving replicas when other nodes are down; writes fail explicitly (503) if the primary is unreachable rather than silently dropping data.
• A built-in failure-simulation endpoint lets you take a node down on demand and verify recovery behavior end-to-end.

Technology