| … | … | @@ -3054,6 +3054,101 @@ Options: |
| 3054 | 3054 | `{ block, blinding }`, a JS bug. It is unreachable while corestore opens cores with |
| 3055 | 3055 | `compat: false`. |
| 3056 | 3056 | |
| 3057 | ### 9.4 The runtime — one microtask queue + libuv (2026-10-04) |
| 3058 | |
| 3059 | **Why.** keetd grew per-layer stand-ins for JS's runtime: |
| 3060 | - `repl::Clock` and `Core::onsettimeout`; |
| 3061 | - `mux::Host::defer`, one queue per host; |
| 3062 | - the replicator's `resumeVia_` (resume through a peer's mux); |
| 3063 | - `ss::Loop` and `ss::Timers`. |
| 3064 | |
| 3065 | Each gate drove one layer on its own queue, so each layer matches JS ALONE. Composed, they |
| 3066 | interleave differently from JS's single queue. The M5f monitor bug (a `get` resumed before |
| 3067 | 'download') was this class, patched locally. In JS there is no clock: every library calls the |
| 3068 | runtime's globals, and Bare has ONE microtask queue on ONE libuv loop. |
| 3069 | |
| 3070 | **Timer audit** (2026-10-04). Every scheduling call site, tests excluded, in the libraries keetd |
| 3071 | ports or will port, at keet-daemon's versions, plus their direct dependencies: |
| 3072 | |
| 3073 | | Library | Sites | What each needs | |
| 3074 | |---|---|---| |
| 3075 | | hypercore 11.37 | request timeouts (`req.context.setTimeout`, cleared on settle); the not-downloading linger (`unref`); the debounced `updateAll` bump (100–3000 ms); `backoff()` sleeps (200 ms–5 s); session-state recovery polls (5 s, 250 ms) | "fire after ≥ ms", cancellable | |
| 3076 | | hypercore 11.37 | `yieldToLoop()`: `await setImmediate` between batches of the range-processing loop | **the one phase use**: let pending I/O run before continuing (fairness, not correctness: I/O arrival is nondeterministic in JS too) | |
| 3077 | | @hyperswarm/secret-stream 6.9.2 | timeout-refresh `Timer.once` (timeout), `Timer.on` (keepalive), `refresh` / `destroy` / `unref` | one-shot + interval with refresh | |
| 3078 | | corestore 7.12.6, protomux-wakeup 2.9.0, xache 1.3.0, speedometer 1.1.0, hypercore-blob-server 1.15.0 | gc / measurement / update intervals | "every ≥ ms" | |
| 3079 | | signal-promise 1.0.3, autobee 2.11.8 (`coreLength`), autobase 7.28.2 (`fast-forward`) | race timeouts | "fire after ≥ ms" | |
| 3080 | | autobase 7.28.2 `lib/timer.js` (the ack timer) | RANDOM intervals (`random2over1`, `Math.random`), a `Date.now()` limit, `unref` | "after ≥ ms"; no order is defined at all | |
| 3081 | | bare-http1 4.5.6 (the blob server) | socket timeouts | a net-layer timeout | |
| 3082 | | streamx (`qmt`); protomux, mutexify (via queue-tick) | microtasks | `queue-tick` uses `process.nextTick` only if a global `process` exists. Measured: Bare has none, and nothing in keet-daemon or hyper-tps installs one. So it is `queueMicrotask`: ONE queue | |
| 3083 | | safety-catch, bare-events (unheard 'error'), autobee / autobase `crashSoon` | `queueMicrotask(() => { throw err })` | the uncaught path | |
| 3084 | | everything | Promise / `await` | microtasks on the same queue | |
| 3085 | |
| 3086 | **Conclusions:** |
| 3087 | 1. **No timer order needs preserving.** No call site depends on the relative order of two timers |
| 3088 | with the same deadline, or on a timer's order against I/O. Every callback is self-contained |
| 3089 | (reject / destroy / gc / measure / a full rescan), and autobase randomizes its own. Bare's |
| 3090 | scheduler (`scheduler._timeout` / `_immediate`) is therefore NOT ported: its internal |
| 3091 | ordering is unobservable to these libraries. |
| 3092 | 2. **One phase dependency:** `yieldToLoop`, which yields to I/O. |
| 3093 | 3. **One microtask queue,** shared by every library: Promise / await, queueMicrotask, |
| 3094 | queue-tick, streamx. |
| 3095 | 4. **`unref` matters only for process liveness.** |
| 3096 | 5. **`Math.random` / `Date.now` are runtime globals too.** The libraries read them directly |
| 3097 | (autobase's timer, the replicator's random picks, timestamps), and JS gates pin them (the |
| 3098 | harnesses already set `Math.random = () => 0` and pin `Date.now`). |
| 3099 | |
| 3100 | **Design: `hc::rt`, the runtime every ported library calls.** It uses JS's names, and nothing is |
| 3101 | injected per layer: |
| 3102 | - **The loop is libuv itself** (`uv_loop_t`), the same library Bare runs on. It is linked, not |
| 3103 | ported or modelled. hyperswarm-peer's UDX transport is already libuv. |
| 3104 | - **`rt::queueMicrotask(fn)`:** ONE process-wide FIFO, drained to EMPTY after every libuv |
| 3105 | callback (timer, I/O, check), as V8's checkpoint does. A ported `await` / `.then` |
| 3106 | continuation goes on it. An exception thrown in a microtask follows the measured Bare |
| 3107 | behaviour: held, then delivered through `hc::uncaughtException` at the end of the iteration |
| 3108 | (one call per throw). |
| 3109 | - **`rt::setTimeout` / `clearTimeout` / `setInterval` / `clearInterval`** (`refresh`, `unref`) |
| 3110 | are thin wrappers over `uv_timer_t`, one call per JS call. They promise only what the audit |
| 3111 | shows is relied on: fire after ≥ ms, cancellable, refreshable. |
| 3112 | - **`rt::setImmediate`** runs in the check phase (`uv_check_t`), after I/O polling. Its only |
| 3113 | user is `yieldToLoop`. |
| 3114 | - **`rt::now()` / `rt::random()`** are `Date.now` / `Math.random`. |
| 3115 | - **I/O:** a raw stream over a `uv_stream_t` (pipe / TCP) and UDX, all on the same loop. |
| 3116 | - **Gates:** the same `hc::rt` API with a VIRTUAL clock (timers fire when the script advances |
| 3117 | time; `now` / `random` pinned). That is the seam a JS test fakes: the timer globals, |
| 3118 | `Math.random`, `Date.now`. The existing secret-stream trace (fake timeout-refresh) maps onto |
| 3119 | it directly. |
| 3120 | |
| 3121 | **What goes:** |
| 3122 | - `repl::Clock` (and NullClock) and `Core::onsettimeout`: the replicator and the Core call `rt`. |
| 3123 | - `mux::Host::defer`: the mux calls `rt::queueMicrotask`; the Host keeps stream I/O only. |
| 3124 | - `resumeVia_`: a resume is `rt::queueMicrotask`. |
| 3125 | - `ss::Loop` and `ss::Timers`: ss calls `rt`. |
| 3126 | |
| 3127 | **Open decision — storage await.** |
| 3128 | - **JS:** hypercore-storage awaits rocksdb-native, whose reads and writes complete on libuv's |
| 3129 | thread pool, as callbacks on the loop. So storage completions interleave with network I/O. |
| 3130 | - **keetd:** storage is synchronous. That is a documented divergence ("wire order may differ"), |
| 3131 | and it does not affect content. |
| 3132 | - **The choice:** |
| 3133 | - (a) Keep it synchronous, documented. Simpler; the composed order still differs from JS |
| 3134 | wherever JS would await storage. |
| 3135 | - (b) Async completion through the libuv thread pool, as rocksdb-native does. Faithful |
| 3136 | interleaving, but every storage call site becomes a continuation. |
| 3137 | |
| 3138 | The choice decides how much of the composed order the gates can compare. |
| 3139 | |
| 3140 | **Migration (each step re-runs every gate and drill):** |
| 3141 | - **R1:** `hc::rt`, both the real libuv runtime and the virtual one. Gate: Bare probe scripts |
| 3142 | (microtask vs timer vs immediate vs I/O vs uncaught ordering, as in the 2026-10-04 probe) |
| 3143 | must give the same trace on `rt`. |
| 3144 | - **R2:** ss onto `rt` (`ss::Loop` and `ss::Timers` removed). The secret-stream trace and |
| 3145 | interop gates are unchanged. |
| 3146 | - **R3:** mux / replicator / Core onto `rt` (Clock, `onsettimeout`, `Host::defer`, |
| 3147 | `resumeVia_` removed). All repl / store / core gates and every drill re-run. |
| 3148 | - **R4:** storage, per the decision above. |
| 3149 | - **Then 4e-4** (`createProtocolStream` / `replicate`): the first gate that composes the stack |
| 3150 | is a COMPOSED trace against JS (secret-stream + protomux + replicator), now meaningful. |
| 3151 | |
| 3057 | 3152 | ## 10. Risks and open questions |
| 3058 | 3153 | |
| 3059 | 3154 | - **Autobee is young and moving** (2.11.8, hyperbee2 2.18 in Keet 4.22.5). It is now the |