| … | … | @@ -3365,6 +3365,19 @@ injected per layer: |
| 3365 | 3365 | and a cached one counts as a miss, so a poisoned cache heals itself. |
| 3366 | 3366 | - Not in R3 (noted): `Core::now` stays an injected `Date.now` (gates pin it); l2 storage has |
| 3367 | 3367 | its own `now_ms`. |
| 3368 | - **R3c DONE 2026-10-06** (keetd 89ab960 on main; R3 is COMPLETE). As designed above: the |
| 3369 | iterator, the scan + ≥ 40-peer bump, the offsets, `NOT_DOWNLOADING_SLACK`, rt's random |
| 3370 | source. |
| 3371 | - Drill comparison vs the R3b tree (8ce4cc2). Both trees were re-drilled after a reboot wiped |
| 3372 | /tmp, lane-matched so that both runs of a set saw the same load: |
| 3373 | - 16 of 18 sets are identical (same mutants red, same survivors by name). |
| 3374 | - m5g2d and m5g4e1 differed only in load-flaky controls (the reference tree's m5g2d s14, |
| 3375 | and R3c's m5g4e1 s19 "uploader's received report missing"). |
| 3376 | - Both are clean in quiet reruns with fresh scratch: m5g2d controls ok / ok with 10/10 red; |
| 3377 | m5g4e1 controls ok / ok with 8/8 red. |
| 3378 | - The new r3c set: 17/17 red. |
| 3379 | - Branch `r4` was rebased onto it. The trees are byte-identical, and the R4 hashes are |
| 3380 | updated below. |
| 3368 | 3381 | - **R4 DESIGN (2026-10-06; approved by the user 2026-10-06).** Storage completes asynchronously, |
| 3369 | 3382 | as in JS, and every JS `await` is a continuation at the same place. |
| 3370 | 3383 | - **User decisions (2026-10-06):** |
| … | … | @@ -3514,8 +3527,8 @@ injected per layer: |
| 3514 | 3527 | the size (about 120 sites plus l3, L2 and the tools); compile-time growth. |
| 3515 | 3528 | - **Not R4:** network-side awaits that are already continuations (wire, timers); |
| 3516 | 3529 | 4e-4 `replicate()` comes after R4i. |
| 3517 | | - **R4a DONE 2026-10-06** (keetd 43440f8 on branch `r4`, on top of R3c's branch copy 4e1b3b5; |
| 3518 | | the branch rebases onto main once R3c lands there). |
| 3530 | - **R4a DONE 2026-10-06** (keetd 0eb1bda on branch `r4`, on top of R3c 89ab960; |
| 3531 | rebased 2026-10-06; first committed as 43440f8 on R3c's branch copy 4e1b3b5). |
| 3519 | 3532 | - **C++20.** nlohmann 3.11.3 switches to its three-way comparison set under C++20 and drops |
| 3520 | 3533 | `operator==(json, json)`; 50 test comparisons (`std::string == json`) stopped compiling. |
| 3521 | 3534 | `JSON_HAS_THREE_WAY_COMPARISON=0` keeps the C++17 operators, so every gate compares json |
| … | … | @@ -3600,7 +3613,7 @@ injected per layer: |
| 3600 | 3613 | - **Order:** R4b-1 streamx generalized (gates unchanged + object mode) → R4b-2 rt completion |
| 3601 | 3614 | FIFO + State / batches / sessions / snapshots / CFs → R4b-3 Iterator, `keys`, `peek` → |
| 3602 | 3615 | R4b-4 gate + drill. L2 keeps its synchronous Db until R4c. |
| 3603 | | - **R4b-1 DONE 2026-10-06** (keetd c613b42 on `r4`). `ss::BasicDuplex<T>` (hc/basic_duplex.h): the |
| 3616 | - **R4b-1 DONE 2026-10-06** (keetd cfa5507 on `r4`; was c613b42 before the rebase). `ss::BasicDuplex<T>` (hc/basic_duplex.h): the |
| 3604 | 3617 | streamx core over any element type, with `defaultByteLength` (a typed array's length, else |
| 3605 | 3618 | 1024), `map` / `mapReadable` / `mapWritable` (a readable map returning null skips; `unshift` |
| 3606 | 3619 | maps, then `push` maps the last pending again — streamx's own code), the byteLength options, |
| … | … | @@ -3609,7 +3622,7 @@ injected per layer: |
| 3609 | 3622 | scripts; drills r4b1 9/9 (two first-run survivors were script gaps: a Readable's lone 'drain' |
| 3610 | 3623 | listener; a `_read` that ends synchronously during the open callback), m5g4e3b 20/20 on the |
| 3611 | 3624 | moved code. ctest 83/83. |
| 3612 | | - **R4b-2 DONE 2026-10-06** (keetd fb50896). rn: rocksdb-native 3.17.4 in full (above), plus |
| 3625 | - **R4b-2 DONE 2026-10-06** (keetd a24b57a; was fb50896). rn: rocksdb-native 3.17.4 in full (above), plus |
| 3613 | 3626 | `rt::complete` (parked completions: Real = a `uv_async`, referenced only while pending; Virtual |
| 3614 | 3627 | = the poll phase; one native callback each, a pass delivers what was queued before it began) |
| 3615 | 3628 | and `js::asyncFn`. Gate `tools/rn`: Bare at `UV_THREADPOOL_SIZE=1` vs rn on rt real AND virtual, |
| … | … | @@ -3618,7 +3631,7 @@ injected per layer: |
| 3618 | 3631 | - MEASURED (the gate's traces): a read batch's per-op promises settle in index order and its |
| 3619 | 3632 | flush promise one tick later; separate `get`s are separate completions; an un-destroyed batch |
| 3620 | 3633 | blocks close forever (its handle ref), and Bare exits with that promise pending. |
| 3621 | | - **R4b-3 (iterators, peek, keys).** |
| 3634 | - **R4b-3 DONE 2026-10-06** (keetd f1ca9a3; iterators, peek, keys). |
| 3622 | 3635 | - `js::AsyncGenerator<T>`: ECMA-262 §27.9 step for step (the five states, the request queue, |
| 3623 | 3636 | next / return / throw, `co_yield` = Await then AsyncGeneratorYield, a return at a yield |
| 3624 | 3637 | unwrapped by Await then unwound with `js::GeneratorReturn` so the body's destructors run as |
| … | … | @@ -3637,6 +3650,153 @@ injected per layer: |
| 3637 | 3650 | - Drills: r4a-gen 10/10 (survivor "drain treats a return as a next" → script genQueuedReturn), |
| 3638 | 3651 | r4b3 12/12 (survivors → scripts: a negative limit over 20 entries; peek's close seen by |
| 3639 | 3652 | `diagOn`; an inline open seen by gets deferred past the open's submission). |
| 3653 | - **R4c DESIGN (DRAFT 2026-10-06, for the user's review).** Read in full first: hypercore-storage |
| 3654 | 3.3.0 (index.js, lib/{tx,view,streams,keys,block-dependency-stream,close-error-stream}.js, |
| 3655 | migrations/1), scope-lock 1.2.4, device-file 2.3.1 (the wait / suspend paths), fd-lock 2.1.1, |
| 3656 | and keetd's L2 (l2.h, db / view / tx / storage .cpp, 2577 lines). migrations/0 (index.js 823 + |
| 3657 | messages.js 1071) is read at R4c-6, its own step. |
| 3658 | - **What changes in kind.** L2 today is "synchronous by design": every read is a direct |
| 3659 | RocksDB get, every store op writes at once, and the l2.h header argues that the JS |
| 3660 | interleavings cannot happen. With L2 on rn they DO happen, and they are the point of R4c: |
| 3661 | - the store's `_enter` / `_exit` group commit (`enters`, the shared `view`, a flush when |
| 3662 | `enters` reaches 0 or `view.size() > 128`, the `flushing` resolver every exiting op waits |
| 3663 | on); |
| 3664 | - a second `_create` reading the head from the unflushed shared view; |
| 3665 | - the tree cache's shared pending promise (`entry.pending`, the `parallel` stat); |
| 3666 | - `Atom already flushing`; |
| 3667 | - a stream reading a View while changes are applied (OverlayStream's sorted snapshot); |
| 3668 | - `CloseErrorStream` when the session is closing. |
| 3669 | The header's argument is deleted along with the code it describes. |
| 3670 | - **The scaffold problem (found while designing; it changes the R4 plan).** The plan said |
| 3671 | unconverted callers would settle L2's promises "by draining microtasks". That cannot work: |
| 3672 | an rn completion is parked and only runs in a later poll phase, and libuv forbids |
| 3673 | re-entering its loop from inside a callback. A plain drain therefore never settles a |
| 3674 | read. A naive "run the loop until it settles" is no better: it would run foreign work |
| 3675 | (timers, wire callbacks, other microtasks) in the middle of a caller's synchronous |
| 3676 | section. The scaffold has to be ISOLATED: |
| 3677 | - **`rt::scaffold::Sync`** (RAII). Construction pushes a scope with its own microtask |
| 3678 | queue, its own unhandled-rejection list, and a completion tag. |
| 3679 | - `rt::queueMicrotask` and `rt::complete` inside the scope go to that scope. |
| 3680 | - `s.wait(p)` loop: |
| 3681 | 1. drain the scope's microtasks; each checkpoint processes its rejection list; |
| 3682 | 2. if `p` has settled, return its value or rethrow its reason; |
| 3683 | 3. otherwise deliver the scope's parked completions, one pass, each through |
| 3684 | `callback()`; |
| 3685 | 4. if steps 1–3 made no progress, throw `Error("scaffold::Sync: no progress")` |
| 3686 | (a wait that depends on something outside the scope). |
| 3687 | - Destruction runs the scope to quiescence, then pops it. Every leftover microtask and |
| 3688 | completion of the call finishes before the caller continues, just as with today's |
| 3689 | synchronous L2. |
| 3690 | - Scopes nest: a microtask inside a scope may open its own. |
| 3691 | - `wait` attaches its handler at once, so the awaited promise is never reported as |
| 3692 | unhandled. |
| 3693 | - Outer microtasks, completions submitted before the scope, timers and I/O never run |
| 3694 | inside a scope. The outer world therefore sees an L2 call as one atomic step, which |
| 3695 | is exactly today's behaviour, so the composed traces must stay identical. Any |
| 3696 | difference is a finding, not noise. |
| 3697 | - The call pattern keeps JS's batching (one CoreRX = one read batch): |
| 3698 | `Sync s; auto rx = st.read(); auto h = rx.getHead(); auto a = rx.getAuth(); rx.tryFlush(); |
| 3699 | use(s.wait(h), s.wait(a));` |
| 3700 | - Lint: `scaffold::` anywhere in the tree fails from R4i on (as planned). The R4 plan's |
| 3701 | R4b line ("by draining microtasks") is corrected to this. |
| 3702 | - **What gets ported (the 100 % rule), under JS's names, returning `js::Promise`:** |
| 3703 | - **lib/view.js.** |
| 3704 | - `View.get` returns `read.get(key)` or `_indexAndGet`, where a hit is |
| 3705 | `Promise.resolve(change[1])`. `_readAndMaybeDrop` is async. |
| 3706 | - `View.iterator` returns an rn Iterator, or an `OverlayStream` (an `ss::BasicReadable` |
| 3707 | over the rn Iterator, with `_read` / `_predestroy`, `'readable'` / `'close'` |
| 3708 | listeners, `getStreamError`, and Overlay's sorted, reversed arrays), or a |
| 3709 | `CloseErrorStream` when `dbClosing(db)`. |
| 3710 | - `View.flush` is async: one rn write batch (`autoDestroy`), `await w.flush()`. |
| 3711 | - **Atom.** `flush()` is async and throws `Atom already flushing`. Preflushes run, then |
| 3712 | `View.flush`, then the onflush handlers (called, then `Promise.all`), and the `finally` |
| 3713 | (reset, `flushing = false`, resolve `flushedPromise`). `flushed()` resolves via |
| 3714 | resolve-reject-promise. |
| 3715 | - Handlers become `std::function<js::Promise<void> ()>`. |
| 3716 | - core's `SessionState::commit_` is wrapped once (`withResolvers`) until R4e converts |
| 3717 | it. |
| 3718 | - hypercore.cpp's repair flush (no handlers) waits in a `Sync`. |
| 3719 | - **lib/tx.js.** |
| 3720 | - CoreRX / CorestoreRX each own one rn read batch (`autoDestroy`). Their getters are |
| 3721 | `async`, except where JS returns the view's promise directly (getBlock, |
| 3722 | getBitfieldPage, getUserData, getMark, getLocal, getCoreByAlias), so the hop counts |
| 3723 | follow JS. |
| 3724 | - `_resolveTreeCache` and the shared pending entry (the promise itself is cached; |
| 3725 | `parallel` returns it). |
| 3726 | - The static `getAuth` / `getHead` / `getHints(db, c)` read through a db. |
| 3727 | - CoreTX `flush()`: the three JS outcomes, `Promise.resolve(!view)`, |
| 3728 | `Promise.resolve(false)` and `View.flush`. The constructor's |
| 3729 | `Cannot open core tx on snapshot`. |
| 3730 | - **The streams** (lib/streams.js) are REAL streams, with `setReadableMap` set to JS's map |
| 3731 | functions. This includes BlockDependencyStream (`_update`, `_findDependencyIndex`, |
| 3732 | `_onclose` chaining) and the `reverse` flags that bitfield / tree / user-data / local |
| 3733 | ignore. |
| 3734 | - **index.js.** |
| 3735 | - CorestoreStorage: the ScopeLock (scope-lock 1.2.4, literal: `flush` / `lock` / |
| 3736 | `unlock` / `destroy` / debounce `skip`), `_enter` / `_exit` / `_flush`, |
| 3737 | `_migrateStore` (with `_preopen`), `_migrateCore` (the `locked` path that re-puts |
| 3738 | into the view), `_allocData`, `_getHead`, every public op, `suspend({ log })` / |
| 3739 | `resume`, `onresume`, `getInfos` with getInfoFromBatch's `.catch(noop)`, `export`, |
| 3740 | `deleteCore`, `audit`, `clear`, `setRecovering` / `_flushRecovering` (16 k batches |
| 3741 | through the discovery-key stream), `_maybeRecover` (uv_fs stat / unlink, as |
| 3742 | `fs.promises`), `_maybeRecoverLater` (synchronous, as JS), `tmpFixStorage`, |
| 3743 | createColumnFamily's option overrides, `close` (`_flush`, db, rocks, the tree cache, |
| 3744 | the device file). |
| 3745 | - The non-bootstrap constructor (`new CorestoreStorage(rocksDb)`), which keetd never |
| 3746 | had, becomes possible on rn and is ported. |
| 3747 | - HypercoreStorage: every method, including `close()`, which returns the session's |
| 3748 | `db.close()`. |
| 3749 | - **device-file 2.3.1 + fd-lock 2.1.1 as `rn::Lock`.** |
| 3750 | - ready-resource-based: `ready` / `transfer` / `close` / `suspend` / `resume`. |
| 3751 | - File ops through uv_fs, inline then parked, like rn in R4b. |
| 3752 | - `wait: true` (fsx.waitForLock blocks a pool thread) cannot run inline without |
| 3753 | blocking the loop. It keeps throwing until R4i, when the pool is real. |
| 3754 | - **L2's sync `Db` / `DbSnapshot` go.** rn's session, snapshot and column family replace |
| 3755 | them. `test_after_write` (the M4d kill-point hook) moves into rn's write completion. |
| 3756 | - **Hops: MEASURED first (R4c-0), per the R4a method.** Use a microtask ruler beside each |
| 3757 | primitive, on Bare at `UV_THREADPOOL_SIZE=1`. The primitives: every getter; a view hit vs |
| 3758 | miss; CoreTX.flush in its three forms; View.flush; Atom.flush and flushed; ScopeLock |
| 3759 | lock / unlock / flush; `_enter` / `_exit` alone and under group commit; a stream's first |
| 3760 | entry. Port against the measured numbers, not derived ones. |
| 3761 | - **Gates.** |
| 3762 | 1. The M4a storage gate (tools/m4 storage-ops.js vs `l2_ops`) becomes an async-native |
| 3763 | trace gate. `l2_ops` drives L2 with no scaffold, and both sides print every settlement |
| 3764 | with its ruler hop, then the dumped store. It runs on Bare at pool=1 vs rt real AND |
| 3765 | virtual. |
| 3766 | - New concurrency scripts: overlapping store ops (group commit; > 128 changes forces |
| 3767 | an early flush), two `_create` on one key, parallel getTreeNode (the `parallel` |
| 3768 | stat), atom flush re-entry, `flushed()` waiters, a stream over a view mutated |
| 3769 | mid-read, close during a read (CloseErrorStream), suspend / resume, `onresume`, |
| 3770 | `getInfos` with missing cores, export with batches, `_maybeRecover`. |
| 3771 | - The existing scripts keep their outputs. |
| 3772 | 2. The rt gate gains scaffold scripts: isolation (an outer microtask and an outer |
| 3773 | completion never run inside), nesting, the no-progress throw, and a destructor that |
| 3774 | runs to quiescence. |
| 3775 | 3. Everything above L2 stays identical: ctest, the store gate, core-ops, the m3c / m4 / |
| 3776 | m5 pipes, the replicate gates, P1's generators. |
| 3777 | - **Drills.** The l2, m5g1a, m5g3a and storage sets are re-anchored onto the new code. A new |
| 3778 | r4c set covers group commit, tree-cache sharing, overlay streams, the scaffold isolation, |
| 3779 | and the device-file lock. |
| 3780 | - **Order (each step green):** |
| 3781 | - **R4c-0:** measure. |
| 3782 | - **R4c-1:** `scaffold::Sync` plus its rt gate. |
| 3783 | - **R4c-2:** View / Atom / tx / tree cache / streams on rn. |
| 3784 | - **R4c-3:** CorestoreStorage / HypercoreStorage / ScopeLock / device-file on rn, plus the |
| 3785 | async `l2_ops` gate. |
| 3786 | - **R4c-4:** switch every caller to the new L2 through `Sync` (about 150 sites in core, |
| 3787 | store, repl, the tools and the tests), then delete the old synchronous L2. |
| 3788 | - **R4c-5:** drills. |
| 3789 | - **R4c-6:** migrations/0 (the hypercore-10 import, which L2 refuses today). The |
| 3790 | fixture comes from hypercore 10.38.2 + corestore 6, both found in the tree. This is a |
| 3791 | separate decision (below). |
| 3792 | - During R4c-2..3 the old synchronous L2 stays, renamed `l2sync` and frozen, so every |
| 3793 | commit builds. R4c-4 removes it. |
| 3794 | - **Open for the user:** |
| 3795 | 1. **migrations/0.** It is owed under the 100 % rule. It is also about 1900 lines that |
| 3796 | only a pre-2023 hypercore-10 store needs. Is it in R4c (R4c-6), or its own milestone |
| 3797 | after R4? |
| 3798 | 2. **Device-file `wait: true`.** It stays refused until R4i, which is when it becomes |
| 3799 | portable. OK? |
| 3640 | 3800 | - **Then 4e-4** (`createProtocolStream` / `replicate`): the first gate that composes the stack |
| 3641 | 3801 | is a COMPOSED trace against JS (secret-stream + protomux + replicator), now meaningful. |
| 3642 | 3802 | |