OverviewFilesCommitsIssuesDocsReleasesPeers

keet-daemon commits / 9849224

docs: TESTING-DESIGN.md — keetd testing redesign (approved 2026-10-05)
Generated op sequences diffed JS vs keetd with a ddmin shrinker and a regression corpus; deterministic simulation (sim-timers + a lockstep controller) for the networked layers; coverage on every change, mutation as an occasional audit. Decisions: own generator, 60 s per layer in ctest, tools in keetd/tools. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
984922457f8457184bcc155e6bb5a22afec58fe6
by plankatron <fleeky@gmail.com> · 2026-10-05T16:19:45.000Z

Changed files (1) +231 -0

docs/TESTING-DESIGN.md +231 -0
……@@ -0,0 +1,231 @@
1# keetd testing — redesign (2026-10-05)
2
3Status: DESIGN, for review. Approved in principle by the user on 2026-10-05: generated tests
4diffed JS vs C++, a deterministic fake clock, and mutation demoted to an occasional audit.
5Companion to `CPP-PORT-DESIGN.md` §4 (verification). Measurements are from the 2026-10-05
6audit (scratchpad `audit/`); estimates are marked as such.
7
8## 1. Why
9
10What the audit measured (keetd 4cf2283 + R3b step 1):
11
12- **ctest** 69/69 in 141 s wall (821 s of test time on 16 cores).
13- **Drills:** 712 hand-written mutants in 36 sets, ~12.5 h serial. The estimate matched the
14 measured m5g4e2 (45 min).
15 - The core gate is 9 h of that: every mutant reruns all 20 scripts, 86–99 s per mutant.
16 - Inside the core gate, keetd itself is **5 %**. 62 % is 272 separate `bare dump.js`
17 launches; 17.5 % is the JS reference run, recomputed identically for every mutant.
18- **Coverage:** 151 of 288 core-gate mutants are executed by exactly ONE of the 20 scripts.
19 Selecting only the covering scripts cuts 6.89 h to 0.90 h (7.6×).
20 - The 5 mutants no script executes are the known permanent survivors.
21- **Pipe gates:** they wait on wall-clock time (a fixed 250 ms per step in wakeup / autobee;
22 s18 runs 91 s on real recovery timers). They also recompute their JS↔JS reference on every
23 run.
24- **Drills edit the working tree in place**, which blocks all editing. Their anchors are
25 checked only when a drill starts, and three sets were silently unusable.
26
27**The structural problem:** hand-written scripts test what we thought of, and planted bugs
28were the only way to learn what the scripts miss. Industry practice for reimplementations is
29different: Cedar, Turso/Limbo vs SQLite, the Ethereum clients (goevmlab), Wasmtime, and
30LevelDB under QuickCheck all **generate operation sequences, run them on the reference and
31the port, diff the results, and shrink every divergence to a minimal regression case.**
32Timing-heavy distributed code is tested under **deterministic simulation** (FoundationDB,
33TigerBeetle VOPR, S2). Mutation testing at scale (Google) is coverage-gated, diff-based and
34occasional, never a whole-catalogue gate.
35
36## 2. Principles
37
381. **The JS stack is the oracle.** We write no model: every generated case runs on JS and on
39 keetd, and any difference is a finding.
402. **Generated, valid by construction.** The generator tracks state (cores, sessions,
41 lengths, keys, atoms) and emits only ops the JS API accepts, plus a weighted share of
42 hostile ops (bad proofs, out-of-range, wrong keys) whose JS errors are compared too.
433. **Every divergence becomes a short permanent test.** A shrinker reduces it, and the
44 reduced script joins the regression corpus that ctest runs.
454. **Deterministic or it doesn't count.** Seeded randomness, a pinned or virtual clock, and
46 no wall-clock waits in a comparison. Each run is checked by running the same seed twice and
47 requiring identical traces (S2's meta-test).
485. **Measure adequacy with coverage first, mutation second.** Coverage is cheap and runs on
49 every change. Mutation runs occasionally, only on covered lines, and only to find
50 *covered-but-unobserved* effects.
516. **Never touch the working tree.** Instrumented builds and mutant builds live out of tree.
52
53## 3. Architecture
54
55```
56 seed ─▶ generator (per layer, stateful) ─▶ op script (JSON, existing vocabularies)
57 │
58 ┌──────────────────────────────────────────────┴──────────────┐
59 ▼ ▼
60 JS runner (persistent Bare process) keetd runner (persistent process)
61 runs scripts back to back; JS output cached by same op vocabulary (core_ops, store_ops,
62 (script hash + JS fingerprint) bee_read, blobs_ops, l2_ops, …)
63 │ │
64 └────────────▶ comparator (lines + batched store dumps) ◀─────┘
65 │ divergence
66 ▼
67 shrinker (ddmin over ops, then values) ─▶ corpus/<layer>/<id>.json
68 (+ the classified reason, if any)
69```
70
71Networked layers use the same pipeline, but the "script" is a **simulation scenario** and
72both runners execute it under the simulator (§6).
73
74## 4. Phase P0 — infrastructure (also speeds up today's ctest)
75
761. **Persistent runners.** `core-ops.js`, `store-ops.js`, `bee-read.js`, `blobs-ops.js` and
77 `storage-ops.js` gain a `--many` mode that reads script paths from stdin and runs each in a
78 fresh directory in one process. This removes a Bare start (~0.1 s plus module load, about
79 0.76 s per core script measured) per script. Same for the C++ tools.
80 - Risk: module-global state carried between scripts (hypercore's `enable(SMALL_WANTS)`,
81 the `Date.now` pin, the uncaught handlers). Each runner resets it, and a gate keeps one
82 run per script in a fresh process as the control.
832. **Batched dumps.** `tools/store-dump/dump.js --many` dumps every store of a run in one
84 process. It stays the JS reader, the independent referee; only the launch count changes.
853. **JS reference cache, for every gate.** This generalises `M3C_CONTROL_CACHE`: the JS
86 output lines and the JS store dumps are keyed by script content plus a fingerprint (the
87 runner sources, the JS tools and the node_modules manifest). Cross reads and round trips
88 depend on keetd's stores and are never cached.
894. **Gate units.** Each gate gets `--only <names>` and a JSON summary; a unit is one script or
90 scenario, which is what drills and coverage select.
915. **Parallel units** within a gate (bounded by cores and memory; bee_read peaks ~430 MB).
92
93P0 exit gate: every existing gate gives the same PASS/FAIL per unit as before. On a
94calibration set of mutants (m5g4e2 complete, plus 10 core mutants that are red today) the
95verdicts are identical. All timings are reported before and after.
96*Estimate:* the core gate drops from ~87 s to ~10–15 s.
97
98## 5. Phase P1 — generated differential testing (non-networked layers)
99
100Layers, in order of value: **core** (core_ops) → **corestore** (store_ops) → **hyperbee2**
101(bee_read: writes, reads, reindex) → **hyperblobs** → **L2 storage** (l2_ops) → **codecs**
102(L0 / schemas: already vector-generated).
103
104**Generator** (`tools/fuzz/gen-<layer>.js`, runs under Bare, no new npm dependencies):
105- A seeded PRNG (splitmix64, the same in JS and Python so cases reproduce anywhere).
106- A state model: the open cores and sessions with their kinds (default / snapshot / named /
107 atomic / exclusive / encrypted), lengths, forks, writability, atoms and stores, and which
108 are closed.
109- An op table with weights and preconditions, taken from the runner's existing vocabulary
110 (core-ops.js has ~70 ops). Unit weights start uniform and are tuned from coverage (§7).
111- Value domains biased to the edges: lengths around page sizes (32768 bits), block sizes
112 0 / 1 / 4096 / boundary, forks 0 / 1 / large, out-of-range indices, `maxLength`, and the
113 64x+56-style length classes.
114- Hostile ops at a fixed share (~10 %): tampered proofs, wrong keys, bad signatures, ops on
115 closed sessions. Compared as JS error text.
116- Determinism inputs pinned exactly as today (key pairs from BLAKE2b seeds, the `time` pin).
117
118**Comparator:** the existing core-gate rules (lines, store dumps, OPTIONS, cross reopen,
119round trip), as a library. The known nondeterministic JS cases are **classified, never
120masked**: Promise.all rejection order, streamx read-ahead, cross-core atom commit order. The
121classifications live in one table with a reason and the evidence (today they are in
122`compare.py` files). The generator avoids generating them unless a mode asks for them.
123
124**Shrinker** (`tools/fuzz/shrink.js`): ddmin over the op list, then per-op value shrinking
125(fewer blocks, smaller blocks, smaller indices). The predicate is "still diverges with the
126same first differing line kind", so it can't shrink onto a different bug. It is bounded in
127time and logs every step.
128
129**Corpus and regression:** each reduced case is saved as `corpus/<layer>/<id>.json` with its
130seed, generator version and finding, and becomes a gate unit run by ctest. A fixed-seed short
131campaign runs in ctest (budget ~60 s per layer); a long campaign runs nightly / on demand
132(hours, all cores) with its results ledger.
133
134**P1 exit gate (the ruler):** a generated campaign must kill at least every core-gate mutant
135the hand-written scripts kill today (calibration from the drill ledger), and must reproduce 3
136historical keetd bugs from git history when they are reverted. Then the hand-written scripts
137become seed corpus, not the only tests.
138
139## 6. Phase P2 — deterministic simulation for the networked layers
140
141Scope: replication (m3c / m4c pipe), wakeup, autobee-wakeup, the blobs monitor, the runtime
142(rt) and secret-stream. Later autobee and keet-core.
143
144**Virtual time on both sides:**
145- **keetd:** `hc::rt` virtual mode (exists, gated against Bare in R1/R2).
146- **JS:** a `sim-timers` module installed on `globalThis` before any module loads. It
147 implements bare-timers 3.2.0's API and handle semantics exactly (`setTimeout` /
148 `setInterval` / `setImmediate` and their clears, plus `refresh` / `ref` / `unref` on
149 handles), with batching as bare-timers does it, over a virtual `now`; `Date.now` reads the
150 same clock. It must match bare-timers' handle shape because `timeout-refresh` picks its
151 implementation at load time by probing `setTimeout(...).refresh`.
152 - Measured 2026-10-05: hypercore, corestore, protomux-wakeup, autobee, autobase,
153 hyperswarm, hyperdht, dht-rpc and timeout-refresh all use the global timers; protomux uses
154 `queue-tick` (a microtask). So the global install reaches the whole stack.
155 - Gate: the existing rt trace scripts run on `sim-timers` and give Bare's traces.
156- **Lockstep controller** (C++, in the harness): owns virtual time and the links between
157 peers (JS processes and keetd instances). Each tick it delivers the frames due now (seeded
158 delay, reordering, drops and duplication only when a scenario enables them), tells every
159 peer to run until quiescent at `now`, collects each peer's next timer due, then advances
160 `now` to the earliest due event. JS and keetd peers can share one simulated link.
161- **Quiescence (the main unknown):** a JS peer's RocksDB I/O completes on real threads. A peer
162 is quiescent when its microtask queue is drained and its harness-counted outstanding storage
163 operations are zero. **Spike first**, with an acceptance test: the same seed twice gives
164 byte-identical traces on 50 seeds, for the existing m3c scenarios.
165- **Traces:** one canonical event line per wire frame, emitted event, timer firing and
166 storage write, with fixed key order and hex buffers (the EIP-3155 lesson: agree the format
167 first).
168
169**Generated network scenarios** extend P1's generator: ops on peers plus link ops (connect,
170disconnect, delay, reorder, partition, restart a peer) at virtual times.
171
172**P2 exit gate:** every pipe scenario has a simulated twin that gives the same verdicts on
173the calibration mutants (m5g1e, m5g2a–d, m5g3e3, wakeup, autobee, monitor). The wall-clock
174pipe gates are then retired, except a small **real smoke suite**: the JS interop over real
175sockets, M3d on two machines, and one real-timer s18.
176
177## 7. Phase P3 — adequacy: coverage always, mutation occasionally
178
179- **Coverage on every change:** an instrumented build out of tree (cached by source hash)
180 runs the regression corpus plus the short campaign, and reports uncovered lines in changed
181 files. Coverage also steers the generator's op weights toward cold code.
182- **Mutation as an audit** (weekly or on demand, and before a milestone closes):
183 - Mutants on covered lines only. An uncovered mutant is reported as **UNCOVERED** and not
184 run.
185 - Each mutant runs only the units that execute it, stopping at the first FAIL.
186 - Workers are rsync'd copies of the working tree in `build/drill/wN/` with their own build
187 directories and ccache. The working tree is never touched.
188 - Trivial compiler equivalence: a mutant whose object file is byte-identical to the original
189 is marked EQUIVALENT without running.
190 - A committed `drill/ledger.json` holds each verdict, the source hash it was tested at, and
191 the reason for each classified equivalent.
192 - The existing 712 hand-written mutants stay as the calibration set.
193 - Later, optionally: mutant schemata (all mutants compiled into one binary, switched by an
194 environment variable, e.g. Dredd / Mull) to remove the rebuild per mutant.
195- **Drill health in ctest:** `drill_lint` checks every set's anchors against HEAD in
196 milliseconds.
197- What mutation is still for: effects that don't reach the compared output (internal state,
198 caches, timer and resource cleanup, swallowed errors). Survivors there point at a missing
199 trace field or op, not at a missing hand-written script.
200
201## 8. Order and how it meets the port
202
2031. **P0** (infrastructure). Then commit R3b step 1 once its drills are green on the P0 drill.
2042. **P1 core** (generator, comparator, shrinker, corpus), then corestore / bee / blobs / L2.
2053. **R3b step 2 → R3c → R4** continue on P1's tests; each change runs the short campaign plus
206 the corpus.
2074. **P2** (sim-timers + controller spike, then the pipe twins) before 4e-4 (`replicate()`),
208 because 4e-4 is the first fully networked composition.
2095. **P3** runs alongside from P0 onwards (coverage first; the mutation audit once P0's
210 isolation exists).
2116. Autobee (M6) and later layers get a generator from day one instead of hand-written
212 scripts.
213
214## 9. Decisions (the user's, 2026-10-05)
215
2161. **Generator and shrinker: our own** (seeded PRNG + ddmin, no new dependency, runs on Bare).
2172. **ctest budget: 60 s per layer** for the fixed-seed campaign; the long campaign runs on
218 demand / nightly.
2193. **Tools live in `keetd/tools`** for now (`tools/fuzz`, `tools/sim`). They may later move to
220 hyperswarm-peer with the shared libraries.
221
222## 10. Risks
223
224- **Generator blind spots:** it only produces what its op table can express. Mitigations:
225 coverage steering, the mutation audit, and hand-written scripts kept as seeds.
226- **JS nondeterminism we haven't catalogued** shows up as flaky divergences. The double-run
227 meta-test catches it, and each case goes into the classification table with evidence.
228- **Quiescence in the simulator** (§6) may need hooks into the JS storage layer. That's why
229 the spike comes first.
230- **False confidence from the corpus:** a green corpus says nothing about ops the generator
231 never emits. The campaign ledger reports op and edge coverage per run.