Roadmap

Sequencing principle

Every phase ends at a finished, demoable, benchmarked, chaos-tested system. If work stops after any phase, the result is a complete smaller system, never a broken half of a larger one. The source spec groups the LSM engine and Raft consensus as one delivery cycle ("Phase 1"); this page splits them into their own rows because they're independently large and at different points of completion right now.

Status

Phase 1 — Custom LSM storage engine (shipped)

crates/storage (cairn-storage): append-only WAL with checksummed, crash-tolerant replay; an ordered memtable; immutable SSTables with a footer index; per-table bloom filters; full compaction that drops shadowed versions and tombstones. 32 tests passing, including a property test against a BTreeMap reference model and explicit crash-recovery integration tests. See LSM Storage Engine for internals and Benchmarks for real numbers.

Phase 2 — Raft consensus (core shipped)

crates/raft (cairn-raft) shipped the consensus core across three cycles, built strictly bottom-up and each merged in turn:

• Log store — durable, index-addressed log entries, CRC'd hard state, and crash-tolerant replay with truncate-on-open. See Log Store.
• Transport — a Transport trait with a deterministic, seeded in-memory implementation (fault injection: partition, drop, delay) and a length-prefixed framed TCP implementation. See Transport.
• RaftCore — a pure, synchronous, I/O-free step function: pre-vote leader election, log replication with the consistency check and conflict back-up, commit-index advancement under Raft's current-term rule, and read-index linearizable reads. Proven by a deterministic multi-node simulation against four safety invariants (election safety, log matching, state-machine safety, leader completeness) under partition, drop, reorder, and crash-restart fault injection — 104 library tests plus 9 simulation tests, green. See Safety Invariants & Simulation.

Building a dedicated log store rather than reusing the LSM engine, and real TCP behind a Transport trait with a deterministic in-memory substrate for tests, were both resolved early — see Raft-over-Paxos for why Raft was chosen, LSM-over-B-tree for why the log isn't just another LSM consumer, and Pre-vote and read-index for a real cross-cutting safety bug the pre-vote design caught before merge.

Next, in order: InstallSnapshot handling and log compaction, plus joint-consensus membership changes (config is currently fixed at construction); then a node driver — the async event loop wiring RaftCore to the real transport, a real clock, and a disk-backed log-store adapter, proven with a real-socket TCP integration test; then the chaos/Jepsen harness, which drives a live cluster through the same fault classes the simulation already covers and checks the resulting history with a per-key linearizability checker (lincheck, already built alongside RaftCore) instead of hand-asserted invariants.

Phase 3 — MVCC transactions

Multi-key transactions at snapshot isolation, layered on Raft's commit ordering for a timestamp/version source: begin(), txn.get/put, txn.commit() → ok | conflict. Garbage collection of obsolete versions folds into the existing compaction path. Cross-shard transactions are explicitly deferred past this phase — the initial guarantee set is single-group. Blocked on the rest of Phase 2 (node driver + real integration), not on RaftCore itself — the commit-ordering primitive this phase needs is already proven, just not yet wired to a running node.

Phase 4 — Multi-Raft

Many independent Raft groups on the same node set, each owning a contiguous key range. Deliberately sequenced after Phases 1–3 are fully finished and chaos-tested, so sharding can't destabilize a proven single-group base.

Phase 5 — Shard router + control plane

The one language boundary in the system: a TypeScript/Bun control plane that decides shard placement, routes client keys to the right Raft group, triggers splits/rebalances, and renders a live dashboard of shard map and per-group leadership.

Testing centerpiece: the chaos suite

Not a phase of its own — it's the thread that runs through Phases 2–4. Deterministic, seeded multi-node simulation catches most consensus bugs before real sockets are involved, and that simulation is already running against RaftCore — see Safety Invariants & Simulation. A Jepsen-style chaos harness will extend the same fault classes (partition, drop, delay, reorder, crash) to a real cluster over real sockets and check the resulting history for linearizability and snapshot isolation violations. The per-key linearizability checker (lincheck) that harness needs already exists; it isn't wired into a live-cluster harness yet. That checker output, committed as evidence, is the actual deliverable — proving the guarantees, not asserting them.