| … | … | @@ -2455,6 +2455,111 @@ SessionState's truncate / reorg / named-session machinery. |
| 2455 | 2455 | - **4d:** marks (lib/mark-bitfield.js, `markBlock` / `startMarking` / `clearMarkings` / |
| 2456 | 2456 | `sweep`), `purge`, `compact`, `audit`. |
| 2457 | 2457 | - **4e:** `registerExtension` (with the replicator's extension messages), `inspect`. |
| 2458 | - **M5g-4e design (2026-10-04).** |
| 2459 | - **Read in full:** |
| 2460 | - hypercore 11.37: the statics (`enable`, `getProtocolMuxer`, `createProtocolStream`), |
| 2461 | `replicate` / `_isAttached` / `_attachToMuxer`, `registerExtension`, extensions in |
| 2462 | `_close`; `lib/inspect.js`, `lib/wants.js`, `lib/feature-flags.js`; the replicator's |
| 2463 | extension and wants call sites (`extension` / `onextension`, `_onpeerupdate`, every |
| 2464 | `w.WANT_BATCH` read); |
| 2465 | - @hyperswarm/secret-stream 6.9.2 (`index.js`, `lib/handshake.js`, `lib/bridge.js`), |
| 2466 | noise-handshake 4.2.0 (all files) and noise-curve-ed 2.1.0. |
| 2467 | - **Keet's usage:** |
| 2468 | - keet-core calls `h.enable(h.SMALL_WANTS)` and `h.setRecoveryPeers([])` when its module |
| 2469 | loads. **Keet replicates with NON-COMPAT wants:** 4096-block batches, and "any" ranges |
| 2470 | sent unrounded with `any: true`. keetd's wants port has been COMPAT only (2^21-block |
| 2471 | batches, any ranges rounded out to whole batches and sent as `any: false`), so until |
| 2472 | 4e-1 lands its want traffic is not the traffic a Keet peer sends. RemoteWants is the |
| 2473 | same in both modes, so the two modes interoperate. |
| 2474 | - corestore 7.12.6 `replicate` → `Hypercore.createProtocolStream(isInitiator, { ondiscoverykey })`. |
| 2475 | - keet-call 10.4.0 (presence, webrtc): `getProtocolMuxer(stream) || createProtocolStream(stream)`. |
| 2476 | - hyperbee 2.27.3 `lib/extension.js`: `registerExtension('hyperbee', …)` (M5h). |
| 2477 | - **hyperswarm-peer audit → FRESH PORT (user decision 2026-10-04).** |
| 2478 | - **What matches JS:** the uint24 framing, the stream id derivation, the 56-byte header |
| 2479 | message, libsodium's secretstream (17-byte ABYTES), and the handshake's |
| 2480 | HMAC / HKDF / mixHash / AEAD nonce layout. This is the path M3d proved. |
| 2481 | - **What doesn't** (gaps in hyperswarm-peer itself, recorded for it): |
| 2482 | 1. Its XX is X25519 (`crypto_kx`, `Noise_XX_25519_…`) with a private `0xFE` type |
| 2483 | byte; JS's is Ed25519 (`noise-curve-ed`, `Noise_XX_Ed25519_…`). They cannot complete |
| 2484 | a handshake. |
| 2485 | 2. The header message: no exact-56 check, no stream id check, and an `init_rx` |
| 2486 | failure is ignored. JS destroys the stream on each of these. |
| 2487 | 3. A failed decrypt or a short message is dropped; JS destroys the stream. |
| 2488 | 4. Zero-byte frames are always eaten; JS eats them only with `_filterZeroByteMessages`. |
| 2489 | 5. Its keepalive is its own 5 s / 20 s sweep; JS has `setKeepAlive` (refreshed by |
| 2490 | writes) and `setTimeout` (destroy). |
| 2491 | 6. No `send` / `trySend` secretbox channel. |
| 2492 | 7. No standalone stream object: it is all inside `UdxTransport`'s `PeerConnection`. |
| 2493 | - The primitives that match are carried over as reference logic. The stream and |
| 2494 | handshake are ported from the JS. |
| 2495 | - **Order** (each sub-milestone ends at its gate and drill): |
| 2496 | - **4e-1, wants non-COMPAT.** |
| 2497 | - A process-wide `wants::enable(flags)` mirrors `UPDATE_COMPAT`, plus |
| 2498 | `Hypercore::enable(flag)` and `SMALL_WANTS = 1`. |
| 2499 | - `WANT_BATCH` becomes a runtime value, read at every use, as the replicator reads |
| 2500 | `w.WANT_BATCH` at call time. |
| 2501 | - In non-COMPAT `addAnyRange` does not round, and add / remove return `any: !COMPAT`. |
| 2502 | - Gates: |
| 2503 | - the m3c-wants vectors in BOTH modes. The generator's second pass calls |
| 2504 | `UPDATE_COMPAT(false)` and must read `exports.WANT_BATCH` afterwards, never a |
| 2505 | destructured const; |
| 2506 | - pipe scenarios with both sides on SMALL_WANTS (peer.js `smallWants` → |
| 2507 | `Hypercore.enable`): a range download crossing 4096-block batches and a sparse |
| 2508 | "any" range, with the uploader's received want / unwant messages compared exactly; |
| 2509 | - one mixed-mode scenario (COMPAT ↔ SMALL_WANTS) for interop. |
| 2510 | - TRAP: the flag is module-global in JS and process-global in keetd. Set it before any |
| 2511 | replicator exists, and reset it between tests. |
| 2512 | - **4e-2, extensions + `inspect`.** |
| 2513 | - Per-session `extensions` (re-registering replaces the handlers, encoding and |
| 2514 | session). |
| 2515 | - `send` / `broadcast` / `destroy` / `_onmessage`. |
| 2516 | - The default encoding is `c.buffer`, so the message is LENGTH-PREFIXED. Encodings are |
| 2517 | byte-level hooks (the 4a decision), so the bytes on the wire include that prefix. |
| 2518 | - `addMonitor` (or `_monitorIndex = -2` before open); `peer.extensions` filled at |
| 2519 | peer-add from every monitor's extensions. |
| 2520 | - The wire `extension` message with the name dedup (`lastExtensionSent` / |
| 2521 | `lastExtensionRecv`). |
| 2522 | - Extensions destroyed on session close. |
| 2523 | - `inspect(depth, { indentationLvl, stylize })`, lib/inspect.js's text exactly; the gate |
| 2524 | uses an identity `stylize`. |
| 2525 | - Gates: pipe scenarios (register before and after the peer opens, a repeated name, an |
| 2526 | unknown name, broadcast to 2 peers, close), and a core-gate `inspect` op. |
| 2527 | - **4e-3, the secret-stream layer** (a new `ss/` library, transport-agnostic: bytes in, |
| 2528 | bytes out). |
| 2529 | - The whole noise-handshake surface: NN / NNpsk0 / XX / XXpsk0 / IK / XK, psk, |
| 2530 | prologue and preshared keys. |
| 2531 | - noise-curve-ed: an Ed25519 sign keypair, a SHA-512 clamped scalar, and |
| 2532 | `scalarmult_ed25519_noclamp`. |
| 2533 | - HKDF-expand's first block hashes `info` twice (`prev` starts as `info`); a no-op |
| 2534 | for the empty `info` secret-stream uses, ported anyway. |
| 2535 | - The 65535 message cap. |
| 2536 | - `Handshake` (framed `send` / `recv`; null on failure). |
| 2537 | - `NoiseSecretStream`: |
| 2538 | - `isInitiator` type check; |
| 2539 | - the `handshake` and `keyPair` options and `pattern`, `remotePublicKey`; |
| 2540 | - the `_onrawdata` framing state machine; |
| 2541 | - header verification (exactly 56 bytes, stream id); |
| 2542 | - Push / Pull; |
| 2543 | - `_filterZeroByteMessages`; |
| 2544 | - `setKeepAlive` / `setTimeout` on the Clock; |
| 2545 | - the `send` / `trySend` secretbox with nonce exhaustion; |
| 2546 | - `alloc`, the MAX_ATOMIC_WRITE refusal; |
| 2547 | - `opened` / `connect` / `handshake`, the destroy paths, `toJSON`; |
| 2548 | - the statics `keyPair` / `id`; |
| 2549 | - a Bridge for the no-raw-stream case. |
| 2550 | - Gates: |
| 2551 | - byte-exact vectors with a fixed static key and a preset `noise.e` (the JS harness |
| 2552 | sets it before `send`); |
| 2553 | - JS ↔ keetd over a pipe in both directions: XX, the `handshake` option, keepalive, |
| 2554 | timeout, a bad header, a tampered frame, `send`. |
| 2555 | - **4e-4, `createProtocolStream` / `getProtocolMuxer` / `replicate`.** |
| 2556 | - The four `isInitiator` forms: a muxer, an existing stream, `opts.stream`, or a bool |
| 2557 | → a new NoiseSecretStream. |
| 2558 | - `Protomux.from` once per stream (`userData`), `setKeepAlive(5000)` unless |
| 2559 | `keepAlive: false`, and `ondiscoverykey` pairing with its 32-byte check. |
| 2560 | - corestore's `replicate(isInitiator)` over it. |
| 2561 | - Gates: pipe scenarios where JS and keetd each call `replicate(true|false)` on a raw |
| 2562 | byte pipe; the M3c / M4c pipe sets re-run over the ported stream. |
| 2458 | 2563 | - **M5g-4a design.** |
| 2459 | 2564 | - **Encodings are byte-level hooks:** |
| 2460 | 2565 | - A `ValueEncoding` is `encode(Buf) → Buf` / `decode(Buf) → Buf` over the bytes of JS's |
| … | … | @@ -2535,6 +2640,26 @@ SessionState's truncate / reorg / named-session machinery. |
| 2535 | 2640 | - **OWED:** "bytes: no prefetch" is observable only as download requests to a peer, so it |
| 2536 | 2641 | needs a pipe scenario. |
| 2537 | 2642 | - ctest 30/30. |
| 2643 | - **M5g-4d DONE 2026-10-03** (keetd ff3c4bf + c4984d0): marks / sweep / compact / audit / purge. |
| 2644 | - **MarkBitfield** (lib/mark-bitfield.js): lazily loaded 32768-bit pages over the mark |
| 2645 | records, LSB-first; `set(false)` never loads a page; the reverse mark stream. |
| 2646 | - **Sessions** share `_marking` / `_marks` with the sessions made from them; `get` marks |
| 2647 | while marking; `markBlock`, `clearMarkings`, `startMarking` (its three refusals), and |
| 2648 | `sweep` (the unmarked ranges, from the end; a snapshot is refused). |
| 2649 | - **`Core::compact`** (every state's storage) and **`Core::audit`** (lib/audit.js: the tree / |
| 2650 | blocks / bitfield passes, dryRun). 11.37's missing-root branch is dead: `getRootsFromStorage` |
| 2651 | throws first. |
| 2652 | - **`purge()`:** 11.37 calls an undefined `this._closeAllSessions`, which is a TypeError. |
| 2653 | Ported as such. |
| 2654 | - **Core gate `marks` (81 ops):** |
| 2655 | - marks in two pages of a 33000-block core (the reverse stream walks pages from the end); |
| 2656 | - a valid block whose bit was cleared (raw bitfield page + reopen: audit's block pass |
| 2657 | drops it); |
| 2658 | - corrupting twins (raw block / tree-node / bitfield-page writes). |
| 2659 | - The streams script lost its non-snapshot mid-stream append case. JS's read-ahead may end |
| 2660 | the stream before the append (6 or 7 values); it turned a drill control red. |
| 2661 | - **Drill `m5g4d`: 19/20 red**, both controls ok (full rerun 2026-10-04 on the fixed gate). |
| 2662 | "set(false) loads a page" is equivalent: 11.37 never calls `set(false)`. |
| 2538 | 2663 | - **M5g-4 — the rest of the surface:** |
| 2539 | 2664 | - value encodings and `encodeBatch` (DECODING_ERROR); `postappend`, `onseq`, `onwait`; |
| 2540 | 2665 | - `setEncryption` / `setEncryptionKey` / `setKeyPair` / `setGroup`; |