Building detsim

this is my first time writing a writeup, so please excuse my language.

so detsim, its a library for deterministic simulation testing like the FoundationDB/TigerBeetle style of testing, mocks all I/O, runs on virtual time, seeds everything and replays any failure exactly. a couple of attempts at this exist (gosim, simtest-go) but nothings established. this is where detsim comes in. detsim on github if you want to look at it directly.

detsim, a standalone go module

three layers.

the deterministic kernel (sim.go, plus the heap itself split out into eventqueue.go once it got big enough to earn its own file). a discrete-event simulator, basically a min-heap of events ordered by (virtual_time, insertion_sequence). no real sleeps anywhere in it. Sim.After(duration, callback) schedules work on virtual time instead. same seed always produces the identical event order, so a scenario simulating an hour of network traffic runs in milliseconds and replays byte for byte identical every time. added RunUntil(t) too, for stuff like leader heartbeats that reschedule themselves forever and would otherwise never let the queue drain on its own.

network.go, fault-injectable networking. seeded message drops, random delays, network partitions where you can block specific node pairs and heal them later, all running off the same seeded rng as the kernel so its all reproducible together.

storage.go, fault-injectable storage. FaultyStorage stands in for a real disk, does torn writes, byte corruption, syncs that silently do nothing (looks committed to the caller, isnt actually), reordered syncs, and a Crash() that just discards whatever wasnt durable yet. had to rebuild this one already, more on that below.

dogfooding it on two real things

didnt want to just build the library and call it done, a tool that only proves it works against its own test suite doesnt really prove anything. so i built two real systems on top of it.

examples/raft: from scratch, single threaded, event driven raft. leader election, log replication, the fast backtrack optimization from the paper (§5.3 if you want to look it up). written straight against detsim's callback model instead of goroutines and real timers. funny side effect of that: since everything runs single threaded during a sim run, theres zero mutexes anywhere in the implementation. no locks, no races. the whole thing never needed to be threadsafe in the first place, so i never had to write a single mutex to make it one. ended up splitting it into separate files as it grew (messages, node, election, replication, handlers) instead of one giant file, easier to navigate once the state machine logic and the message handling stopped being the same thought.

the test that actually matters here is TestThousandsOfSeedsNoSplitBrain. spins up 5000 independently seeded 5 node clusters, each one goes through normal operation, gets partitioned, then healed, and the whole time it checks the one thing raft exists to guarantee: no two leaders in the same term, ever. all 5000 of those trials run in about 7 seconds. a version of this same test using real timers and real goroutines could never afford anywhere close to that many runs.

examples/kv: small write ahead log key value store sitting on FaultyStorage, with a checksum toggle so i could actually prove checksums matter instead of just saying they do like everyone does.

the bugs, this is the part worth reading

bug 1. the raft cluster test setup started all the nodes with a for loop over a go map. go deliberately randomizes map iteration order, i knew that going in and still tripped on it. that randomized start order changed which node's timer landed first whenever two election timeouts happened to fire at the exact same virtual time, which fed into the tie break the kernel uses to order simultaneous events. so i ran TestSeedIsExactlyReproducible twice with the literal same seed and got two different outcomes, run one had a leader elected in term 3, run two had no leader at all yet. that broke the one promise this whole project exists to make, and it got caught by a test whose entire job was to catch exactly that. fixed it by iterating the ordered id slice instead of the map.

bug 2. FaultyStorage itself had a byte addressing bug. the first version stored committed data as map[offset][]byte, treating every write as one opaque block keyed by its exact starting offset. thats not how a real disk works. the kv store's recovery code reads a fixed size header at some offset, then separately reads the variable length body starting right after it, and since that second offset was never itself a write target, the read just silently came back empty. first time i ran TestBasicPutSyncRecover it failed immediately, read back nothing at all. traced it down to the storage layer, not my kv code. rebuilt FaultyStorage around an actual growable byte buffer with real offset:offset+len semantics, like disks actually behave.

bug 3, smaller but worth mentioning. my first torn write test kept writing the exact same string to the exact same offset a hundred times in a row. once i fixed the byte addressing bug above, a torn (truncated) write would just leave the old identical bytes sitting untouched in the tail, so the corruption became literally invisible, the "torn" write reassembled back into the correct value purely by coincidence. fixed it by varying the payload every iteration. good reminder that a fault injection harness can pass for the completely wrong reason if the workload underneath it isnt varied enough to actually make corruption visible.

the proof

TestNoChecksumStoreCanServeCorruptData runs the no-checksum version of the kv store through up to 5000 seeded fault scenarios, 30% torn write rate, 40% corruption rate. found a seed, seed 3, where it serves back straight up corrupted data with no error whatsoever. TestChecksummedStoreNeverServesCorruptData runs the identical fault injection across 2000 seeds on the checksummed version. never once serves corrupted data. seed 3 is the actual proof, sitting right there in the test output.

numbers if you care

5000 raft cluster trials, partition plus heal each, run in about 7 seconds. 2000 kv checksum integrity trials run in about 0.2 seconds. the 5000 no-checksum corruption hunting trials found the bad seed at seed 3. zero mutexes anywhere in the raft implementation. two real bugs found and fixed through the harness itself, one in its own determinism guarantee, one in the storage primitive, plus one test design bug alongside those.

what this isnt

doesnt intercept the real go scheduler for true goroutine level determinism the way gosim does, that needs patching the actual go runtime which is a whole different project. anything you want to test against this has to be written event driven, callback style, against the kernel. you cant just drop in some existing goroutine heavy code unmodified and expect it to work.

whats left

raft snapshotting and log compaction, linearizable reads, cluster membership changes without a full restart, a wal replay idempotency test specifically for crashing during recovery itself, a disk full fault type, actual benchmarks, ci, no fuzzing wired up yet either.

code's public, detsim on github if you want to poke at it or find the next bug before i do.