OverviewFilesCommitsIssuesDocsReleasesPeers

keet-daemon commits / 2e29e55

CPP-PORT-DESIGN: R4c-4 design (bottom-up, scaffold at the frontier) + call-site research record
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2e29e559a1bc5e9416d5d178888c312758a7e95d
by plankatron <fleeky@gmail.com> · 2026-10-07T13:19:10.000Z

Changed files (5) +1318 -0

docs/CPP-PORT-DESIGN.md +127 -0
……@@ -4032,6 +4032,133 @@ injected per layer:
40324032 - the `_preopen` hop and getInfoFromBatch's first `await null` both precede the first completion;
40334033 - a version-0 head is never length 0.
40344034 - ctest 89/89.
4035- **R4c-4 DESIGN (2026-10-07; approved by the user 2026-10-07 — direction: convert
4036 bottom-up, the scaffold only at the frontier; M4a's coverage moves into tools/st).**
4037 - **Research record** (read in full, every JS line verified; keet-daemon `docs/research/`):
4038 `r4c4-inv-core.md` (core/: ~275 call rows, 30 JS functions, hazards H1–H12),
4039 `r4c4-inv-store-repl.md` (store/, repl/, l3/: hazards H1–H16), `r4c4-inv-tools.md` (tools, tests,
4040 ctest, drills, what outlives l2/). Below, `core:Hn` / `sr:Hn` cite those hazards.
4041 - **Why the planned R4c-4 is replaced.** The plan wrapped every storage call (~275 in core/, 236 `l2::`
4042 references overall) in `scaffold::Sync` and kept the callers synchronous. That cannot follow JS:
4043 - `merkle::Storage::get_tree_node` (l3 merkle.h:36) is one synchronous read. JS has TWO shapes
4044 that interface erases: functions that take an **rx** and put every read in ONE batch
4045 (`getRootsFromStorage` mt:579-591, `downgrade`, `upgradeable`, `proof`, `byteRange` /
4046 `byteOffset`), and functions that take a **storage** and open one rx per node, awaited in
4047 sequence (`truncate`, `reorg`, `verify`, `get`, `missingNodes`, `ByteSeeker._seek`,
4048 `MerkleTreeBatch.get` mt:166). Under Sync every read stays its own batch (core:H1, core:H5).
4049 - Proofs: JS reads the block in the SAME batch as the proof's nodes (ix:1195, rep:918) and
4050 `destroy()`s the batch on a generation error (rep:928); keetd reads it from a second rx
4051 (core:H9, sr:H15). A synchronous `getTreeNode` on the new layer deadlocks proof generation,
4052 since a read resolves only at the caller's `tryFlush` (sr:H5).
4053 - R4d / R4e would then rewrite the same sites a second time.
4054 - **The frontier rule.** Layers convert bottom-up into JS's async shape. Code above the converted
4055 layer (the FRONTIER callers) is unchanged and enters converted code through `scaffold::Sync`:
4056 - every frontier entry runs to completion inside ONE scope, so the outer world still sees each
4057 call as one atomic step (today's behaviour; composed traces stay identical, any difference is
4058 a finding);
4059 - so converted code is never mid-operation when the next frontier call arrives: the
4060 Promise Mutex is always free at a frontier entry, as today's idle asserts require;
4061 - **exceptions:** an entry whose JS waits on a TIMER or the network cannot run in a scope
4062 (`NoProgress`). Today: `SessionState::waitForRecovery` (5 s / 250 ms timers, session_state.cpp
4063 :923-958), already callback-shaped. Such entries stay continuations; each one's held state
4064 across the wait (mutex, active batch, atom) is checked against JS and listed in the step
4065 that converts it. Any further one found during conversion is added to this list, not wrapped;
4066 - the frontier moves up as R4f (replicator), R4g (corestore), R4h (bee, blobs) convert; R4i
4067 deletes the scaffold (lint-guarded), unchanged.
4068 - **What is ported (the 100 % rule; JS names; each `async` a `js::Promise` coroutine):**
4069 - **l3 (hypercore 11.37 lib/merkle-tree.js).** `merkle::Storage` goes. Readers take what JS
4070 takes: an rx (`hc::storage::CoreRX`-shaped: getters returning `js::Promise`, `tryFlush`,
4071 `destroy`) or a storage (whose `read()` opens a fresh rx). `getTreeNodeOrError` /
4072 `getTreeNodeFromStorage(OrError)` / `hasTreeNode` as mt:1265-1292. Proof generation is
4073 JS's protocol: `generateProof` pushes reads into the caller's rx (seek reads, when present,
4074 through the storage in their own batches, mt:1348-1355), the caller `tryFlush`es, then
4075 `settleProof` (`Promise.all` per list, `allSettled` on failure, mt:1294-1311). ByteSeeker
4076 (mt:429-490). `ReorgBatch._update`'s `startingDiff !== this.diff` guard (mt:400) returns,
4077 since it becomes reachable.
4078 - **core (session-state.js, core.js, index.js, mutex.js, audit.js, copy-prologue.js,
4079 mark-bitfield.js, fully-remote-proof.js, bitfield.js).**
4080 - `Mutex` as lib/mutex.js: `lock()` is a promise (one hop even when free), FIFO, the waiter
4081 resumes in a microtask. The 8 "while the state mutex is held" throws go (core:H2), and
4082 `locked_` becomes `await lock; try { … } finally { _unlock() }` with JS's settle order
4083 (core:H3).
4084 - `Core` opens as JS does: `this.opening = this._open(opts)` (cj:67, 148-163); the
4085 constructor does no I/O (core:H6). Session's open awaits `core.ready()` (ix:390).
4086 - Core's `_verifyExclusive` / `_verifyShared` with `_verifies` / `_verified` batching
4087 (cj:514-603, 792-832), even though only R4f's concurrent replicator exercises sharing; the
4088 gate drives it directly.
4089 - Every read in JS's batch shape, issue → `tryFlush` → await, including the sites that today
4090 read without a `tryFlush` or use a value before it (core:H4). `CoreTX` held by `shared_ptr`
4091 across its `flush` (core:H12).
4092 - Atom: `repairTreeNodes_` awaits `Atom::flush` (core:H10).
4093 - Streams consumed as JS consumes them: `for await` with the body's awaits interleaved
4094 (`copyPrologue` cp:52, `audit` au:87-92, `sweep`'s `Promise.all` per `batchSize`
4095 ix:1063-1077, MarkBitfield's Transform mb:218-241) (core:H8).
4096 - The ordering deviations that become observable (core:H11): `markBlock`'s `Promise.all`
4097 under MarkBitfield's mutex (mb:162-207), the MarkPage inserted before its read (mb:173),
4098 awaited `preappend` / `postappend` and `encrypt`, `_get`'s re-checks after its awaits
4099 (ix:949-968), `compact` returning `core.compact()` (ix:235), the open's userData flush
4100 awaited before `ready` (ix:489), `treeHashFromStorage` async (ix:225).
4101 - **Storage close policy = JS's, literally (core:H7):** keetd closes a storage exactly where
4102 JS does (`SessionState.close` returns `storage.close()`, ss:195, 203-207; `unref`'s
4103 `.catch(noop)`), and never where JS never does (`_repairTreeNodes`' atomized storage
4104 cj:727, fully-remote-proof's resumed core frp:42): those sessions live until the db
4105 closes, as in JS. Destructors close nothing.
4106 - **Shared definitions out of l2.** `l2::keys` moves to `storage/` as `hc::storage::keys`; the
4107 `hs` alias and `K32`, `Change`, `Dependency`, `CorePtr`, `TreeCacheStats` become
4108 `hc::storage`'s own; `hc_storage` links `hc_schema_hypercore_storage` directly and storage.h
4109 stops including l2.h. `encode_core_auth` / `encode_core_head` are deleted (storage has
4110 `encodeAuth` / `encodeHead`).
4111 - **To measure on Bare before porting (the R4a method; never derived):**
4112 - JS quirk core:H9: with `seek`, `generateProof` awaits the seek BEFORE pushing its rx reads;
4113 by then `Hypercore#proof` has `tryFlush`ed (view nulled, tx.js:331-337). Measure what JS
4114 does (throw? rejection? which message?) and port that.
4115 - Hop rulers at every native completion for each converted core op (append, truncate, clear,
4116 verify, reorg, commit, catchup, createSession, open, proof, get, audit, copyPrologue, marks),
4117 as tools/st's `hops` does.
4118 - **Gates.**
4119 - **tools/st absorbs M4a** (user decision): M4a's 10 scripts (basic, ranges, reopen, cross,
4120 snapshots, atoms, treecache, deps, sessions, storeops), cross-open of each other's store, the
4121 RocksDB OPTIONS comparison, plus the cases no gate covers today (`onresume`, `alwaysRecover`,
4122 `_maybeRecover`, close during a read, `setRecovering`, `readOnly`). Bare at pool = 1 vs rt
4123 real + virtual, traces + hex stores. Then `l2_ops`, `storage-ops.js`, `gen-storage-ops.js`,
4124 `storage-gate.sh` and `m4a_storage_gate` are deleted.
4125 - **l3:** the merkle vectors and tests unchanged in value; a merkle hop trace vs hypercore's
4126 own merkle-tree.js over hypercore-storage (Bare, pool = 1), covering both reader shapes and
4127 proof generate / tryFlush / settle, including a generation error (`destroy`).
4128 - **core:** `m4b_core_gate` (core-ops.js vs core_ops: results, dumped stores, OPTIONS,
4129 cross-reopen) stays byte-identical and gains a `hops` mode (rulers at every native
4130 completion, both sides); new scripts for concurrency JS allows and keetd used to refuse:
4131 two appends queued on the mutex, verify during an append, shared verifies batched, close
4132 during an op, `waitForRecovery` with a pending op. Fuzz corpus + campaign unchanged.
4133 - **Everything above:** ctest, store gate, m3c / m4 / m5 pipes, replicate gates, P1
4134 generators — identical (frontier rule).
4135 - **M4d:** the kill-point hook moves into rn's write completion (`rn::native::write`'s
4136 on_write, rn/src/native.cpp:279-283). rn writes an empty batch and store ops group-commit,
4137 so write numbers shift; the BOUNDARY phase uses consecutive N and stays valid.
4138 - **Drills.** 75 anchors live in l2/ (l2 5, m5g1a 34, m5g3a 30, m5g3c 6) and ~69 more sit on
4139 core/repl lines this step rewrites. `drill_lint` fails the moment an anchor file disappears, so
4140 each sub-step re-anchors the sets it touches IN THE SAME COMMIT (onto storage/, l3/, core/;
4141 gates st / hs / core-gate) and re-runs them red; the old l2 sets retire with l2. Survivors are
4142 gate questions first (#107 lesson 5).
4143 - **Order (each sub-step green: ctest, every gate, its drills):**
4144 - **R4c-4a** shared definitions out of l2 (no behaviour change); M4a's coverage into tools/st
4145 and the M4a gate deleted; the M4d hook into rn.
4146 - **R4c-4b** measurements (the seek-proof quirk; merkle and core hop traces on Bare).
4147 - **R4c-4c** l3 async in JS's two reader shapes + the proof protocol, gated by the merkle hop
4148 trace. Core still runs on l2 for this one step and reaches l3 through ONE adapter (an
4149 l2-backed rx whose promises are already settled), with Sync at core's l3 calls; the adapter
4150 is deleted in R4c-4d.
4151 - **R4c-4d** core onto hc::storage in JS's shape (everything in "core" above), its frontier
4152 Syncs in store/, repl/, bee/, blobs/, tools and tests, the adapter gone; core-gate with hops.
4153 One commit for the storage-type switch itself (core holds one storage type); the Mutex,
4154 the open and the per-group conversions land as preceding commits on the adapter where they
4155 build green.
4156 - **R4c-4e** l2/ deleted with its CMake targets; drill_lint and ctest clean; R4c-5 drills.
4157 - **Consequence for the R4 plan:** R4d (l3) and R4e (core) are done inside R4c-4. R4f
4158 (replicator: `_rxLock` / `_txLock`, protomux `_track` sr:H8 FIRST in mux/, the post-await
4159 re-checks sr:H6 / H10 / H12, `_resolveBlocksLocally` sr:H16), R4g (corestore: async
4160 ready / close / get with `preload`, the GC close race sr:H1-H4, H13) and R4h (bee / blobs'
4161 synchronous-completion assumptions sr:H14) follow, each moving the frontier up.
40354162- **Then 4e-4** (`createProtocolStream` / `replicate`): the first gate that composes the stack
40364163 is a COMPOSED trace against JS (secret-stream + protomux + replicator), now meaningful.
40374164
docs/research/drills.py +33 -0
……@@ -0,0 +1,33 @@
1import os, re, sys, collections
2os.chdir('/storage/projects/code/hypercore-kb/hyper-fleeky/.scope/keetd')
3src = open('tools/m5e/drill.py').read()
4g = {}
5exec(src[:src.index('MUTANTS, FILES, TARGET, GATE = SETS[sys.argv[1]]')], g)
6# storage-call tokens in core/store/repl
7TOK = re.compile(r'l2::|\bhs::|storage|Storage|\btx\b|\brx\b|CoreTX|CoreRX|\.read\(\)|\.write\(\)|\bAtom\b|atom|\bflush\b|flushed|createBlockStream|createTreeNodeStream|createBitfieldStream|createMarkStream|createUserDataStream|createLocalStream|resumeCore|createCore|getInfo|hasCore|getAlias|createGroup|exportCore|deleteCore|setRecovering|getSeed|setSeed|DefaultDiscoveryKey|createDiscoveryKeyStream|createAliasStream|createCoreStream|createGroupUpdateStream|resumeSession|createSession|createAtomicSession|deleteSessions|setDependencyHead|updateDependencyLength|getDependency|dependencies\(\)|putBlock|putTreeNode|putBitfieldPage|putMark|putUserData|putLocal|deleteBlockRange|deleteTreeNodeRange|deleteBitfieldPageRange|deleteMarkRange|getTreeNode|getBitfieldPage|getMark\(|getLocal|setHead|setHints|setAuth|setSessions|setDependency|setGroup|deleteHead|putGroupUpdate')
8def where(f, anchor):
9 s = open(f).read()
10 i = s.find(anchor)
11 if i < 0: return None
12 return s[:i].count('\n') + 1
13res = {}
14for name, (mutants, files, target, gate) in g['SETS'].items():
15 c = collections.Counter(); hits = []
16 for m in mutants:
17 f, label, anchor = m[0], m[1], m[2]
18 top = f.split('/')[0]
19 line = where(f, anchor)
20 if top == 'l2': c['l2'] += 1; hits.append(('l2', f, line, label))
21 elif top == 'storage': c['storage'] += 1; hits.append(('storage', f, line, label))
22 elif top in ('core', 'store', 'repl'):
23 if TOK.search(anchor) or (len(m) > 3 and TOK.search(m[3] or '')):
24 c['callsite'] += 1; hits.append(('call', f, line, label))
25 else: c['other-csr'] += 1
26 else: c['other'] += 1
27 res[name] = (len(mutants), c, target, gate, hits)
28mode = sys.argv[1] if len(sys.argv) > 1 else 'summary'
29for name, (n, c, target, gate, hits) in res.items():
30 if c['l2'] or c['storage'] or c['callsite']:
31 print(f"{name}\t{n}\tl2={c['l2']}\tstorage={c['storage']}\tcallsite={c['callsite']}\tother-core/store/repl={c['other-csr']}\tother={c['other']}\ttarget={target}\tgate={gate}")
32 if mode == 'detail':
33 for h in hits: print(' ', h[0], f'{h[1]}:{h[2]}', '-', h[3])
docs/research/r4c4-inv-core.md +599 -0
……@@ -0,0 +1,599 @@
1# R4c-4 call-site inventory — core/ (l2 → hc::storage)
2
3Worktree `hyper-fleeky/.scope/keetd` @ 9bc081d (branch r4). Read in full: `l2/include/hc/l2.h`,
4`storage/include/hc/storage.h`, `storage/include/hc/corestore_storage.h`, and every file under `core/`.
5Spec: hypercore 11.37.0 and hypercore-storage 3.3.0 in `packages/keet-daemon/node_modules/`.
6
7Files under core/ that touch l2: `session_state.{h,cpp}`, `hypercore.{h,cpp}`, `session.{h,cpp}`,
8`marks.{h,cpp}`, `fully_remote_proof.{h,cpp}`, plus `byte_seeker.cpp` (indirectly, through
9`SessionState::get_tree_node`). `streams.*`, `requests.*`, `value_encoding.*`, `charset.c` have no l2 use.
10
11Abbreviations:
12- C++: **SS** `core/src/session_state.cpp`, **SSh** `core/include/hc/session_state.h`, **HC** `core/src/hypercore.cpp`,
13 **HCh** `core/include/hc/hypercore.h`, **SE** `core/src/session.cpp`, **SEh** `core/include/hc/session.h`,
14 **MK** `core/src/marks.cpp`, **MKh** `core/include/hc/marks.h`, **FRP** `core/src/fully_remote_proof.cpp`,
15 **FRPh** `core/include/hc/fully_remote_proof.h`, **BS** `core/src/byte_seeker.cpp`.
16- JS (hypercore 11.37.0): **ss** `lib/session-state.js`, **cj** `lib/core.js`, **ix** `index.js`, **mt** `lib/merkle-tree.js`,
17 **au** `lib/audit.js`, **cp** `lib/copy-prologue.js`, **mb** `lib/mark-bitfield.js`, **frp** `lib/fully-remote-proof.js`,
18 **bf** `lib/bitfield.js`. hypercore-storage 3.3.0: **hs** `index.js`, **tx** `lib/tx.js`.
19- New API shorthand (namespace `hc::storage`): `HS` = `HypercoreStorage`, `HSP` = `HypercoreStoragePtr`
20 (`shared_ptr<HS>`), `CS` = `CorestoreStorage`, `RX` = `CoreRXPtr` (`shared_ptr<CoreRX>`), `TX` = `shared_ptr<CoreTX>`,
21 `P<T>` = `js::Promise<T>`.
22- "await / Promise.all / sync / fire" = what the JS does with the value.
23
24Signature changes that apply everywhere (not repeated per row):
25- `l2::HypercoreStorage::read()` → by-value `CoreRX` ⇒ `RX HS::read(int64_t fork = -1)` (shared_ptr). Every getter
26 becomes `P<std::optional<…>> CoreRX::getX(…)` and **resolves only after `rx->tryFlush()`** (tx.js:179-338:
27 `this.read = db.read({ autoDestroy: true })`; `tryFlush()` flushes the ReadBatch and `_free()`s the view, so a
28 getter called AFTER tryFlush hits `this.view === null`). l2's `~CoreRX` (which settled pending reads) has no
29 counterpart: `CoreRX::destroy()` must be called explicitly when a batch is abandoned.
30- `l2::HypercoreStorage::write()` → by-value `CoreTX` ⇒ `TX HS::write()`; every mutator stays `void` (sync, as JS);
31 `bool CoreTX::flush()` ⇒ `P<bool> CoreTX::flush()`.
32- `HS::core()` (accessor, `const CorePtr&`) ⇒ public member `std::shared_ptr<CorePtr> HS::core` (`st->core->dataPointer`).
33- `HS::atom()` (accessor) ⇒ public member `AtomPtr HS::atom`.
34- Streams: l2 returns collected `std::vector`s; new returns `Stream` (= `shared_ptr<ss::BasicReadable<rn::Entry>>`),
35 values read via `storage::mapped<BlockEntry|PageEntry|UserDataEntry|…>(entry)`; range args become a struct
36 (`IndexRange{gte, lt, reverse}`, `UserDataRange`).
37- `hs::` schema types, `Dependency`, `CorePtr`, `Change`, `TreeCacheStats`, `K32` are aliases of l2's in storage.h
38 (`namespace hs = l2::hs; using Dependency = l2::Dependency; …`): type-only renames, no behaviour change.
39- No core/ code passes a fork to `read()` ⇒ every core read is `fork = -1` ⇒ the store tree cache is **skipped**
40 in every row below (tx.js `getTreeNode`: `if (!this.treeCache || this.fork < 0) … skips++`). Only
41 replicator.js:1027/1059 read with a fork (out of scope here).
42
43---
44
45## A. Calls (one row per use)
46
47### A1. session_state.cpp / session_state.h
48
49| # | file:line | C++ fn | l2 call / use | new hc::storage equivalent | P? | JS counterpart | JS | notes |
50|---|---|---|---|---|---|---|---|---|
51| 1 | SS:118 | `SessionState::isAtomicDefault` | `storage->atom()` | `storage->atom` (member `AtomPtr`) | n | ss:69-71 | sync | rename only |
52| 2 | SSh:75 | `SessionState::isSnapshot` | `storage->snapshotted()` | `bool HS::snapshotted() const` (now `db->snapshotted()`) | n | ss:61-63 | sync | |
53| 3 | SS:139 | `SessionState::flushedLength` | `storage->dependencies()` | `const std::vector<Dependency>& HS::dependencies() const` | n | ss:92-97 | sync | |
54| 4 | SS:173-182 | `SessionState::close` | (no call) "storage goes with its last holder"; `lingers.clear()` | `P<void> HS::close()` on `storage`, then on each `lingers` entry | **y** | ss:195 `const closing = this.storage.close()`; ss:203-205 `for (…lingers) await storage.close()`; ss:207 `return closing` | returns promise / await | **Deviation**: C++ never calls close (l2's `~HS` did). New HS has no destructor ⇒ without an explicit close the view keeps a reader (`readStop` never runs) and the db session is never closed (hs:439-446). `close()` becomes async; callers `unref()` (ss:104-107 fires it with `.catch(noop)`), `~SessionState`, `Core::close`/`~Core`. |
55| 5 | SS:186 | `SessionState::snapshot` | `storage->snapshot()` → `unique_ptr` | `HSP HS::snapshot()` | n | ss:211 | sync | |
56| 6 | SS:189-192 | `SessionState::snapshot` | callback `done(Fail, shared_ptr<SessionState>)` wrapper | `P<std::shared_ptr<SessionState>>` coroutine | y | ss:210-215 (`async snapshot`, `await this._updateSnapshotStorage`) | await | existing callback-async wrapper → direct Promise |
57| 7 | SS:196 | `updateSnapshotStorage_` | `atomized->flushing` (l2::Atom field) | `Atom::flushing` | n | ss:129 | sync | |
58| 8 | SS:199-221 | `updateSnapshotStorage_` | `atomized->flushed(fn)` loop (weak self-referencing lambda) | `P<void> Atom::flushed()`: `while (atomized->flushing) co_await atomized->flushed();` | y | ss:130 | await (in while) | existing callback wrapper (M5g-3c) → literal while/co_await. C++'s extra "atomized became null" TypeError branch (SS:203-205) is what JS does when `this.atomized` is nulled mid-wait (ss:193) — keep. |
59| 9 | SS:225 | `readSnapshotTree_` | `l2::CoreRX rx = st.read()` | `RX rx = st->read()` | n | ss:132 | sync | |
60| 10 | SS:226 | `readSnapshotTree_` | `rx.getHead()` | `P<optional<hs::CoreHead>> CoreRX::getHead()` | y | ss:133 | Promise.all (ss:138) | issue all three, tryFlush, then `co_await all` — C++ order head/auth/dep matches JS |
61| 11 | SS:227 | `readSnapshotTree_` | `rx.getAuth()` | `P<optional<hs::CoreAuth>> CoreRX::getAuth()` | y | ss:134 | Promise.all (ss:138) | |
62| 12 | SS:228 | `readSnapshotTree_` | `rx.getDependency()` | `P<optional<hs::CoreDependency>> CoreRX::getDependency()` | y | ss:135 | Promise.all (ss:138) | |
63| 13 | SS:229 | `readSnapshotTree_` | `rx.tryFlush()` | `rx->tryFlush()` | n | ss:137 | sync | |
64| 14 | SS:231 | `readSnapshotTree_` | `st.setDependencyHead(…)` | `void HS::setDependencyHead(const optional<Dependency>&)` | n | ss:140 | sync | |
65| 15 | SS:237 | `readSnapshotTree_` | `st.read()` (2nd batch) | `st->read()` | n | ss:145 | sync | |
66| 16 | SS:239-243 | `readSnapshotTree_` | `rx2.getTreeNode(r)` per root, **null → throw inside the loop** | `P<optional<hs::CoreTreeNode>> CoreRX::getTreeNode(int64_t)` | y | ss:146-149, ss:153 `Promise.all(rootPromises)`, null check ss:156-167 | Promise.all | **Deviation / hazard**: C++ consumes each node before `rx2.tryFlush()` (SS:246) and throws mid-loop (leaving rx2 un-flushed). New layer: issue all, tryFlush, `co_await all`, then check nulls (JS order). |
67| 17 | SS:246 | `readSnapshotTree_` | `rx2.tryFlush()` | `rx2->tryFlush()` | n | ss:151 | sync | must move before the null check (see #16) |
68| 18 | SS:248 | `readSnapshotTree_` | `auth->manifest->prologue` | same (hs type) | n | ss:173 | sync | JS reads `auth.manifest` without a null check on `auth` (TypeError when auth is null); C++ guards `auth &&` — pre-existing, not storage |
69| 19 | SS:265-268 | `SessionState::get_tree_node` (merkle::Storage override) | `storage->read().getTreeNode(i)` (temporary rx, **no tryFlush**) | `RX rx = storage->read(); auto p = rx->getTreeNode(i); rx->tryFlush(); co_return co_await p;` → interface must become `P<optional<Node>>` | y | mt:1280-1285 `getTreeNodeFromStorage(storage, index)` (rx, getTreeNode, tryFlush, return promise) | returns promise (callers await) | **Hazard H1**: merkle::Storage is synchronous (l3/include/hc/merkle.h:36). Used by every merkle fn that reads (truncate, restore, downgrade, reorg, verify, getRootsFromStorage, missingNodes, proof, get) and by ByteSeeker. |
70| 20 | SS:271 | `SessionState::getBlock` | `storage->read().getBlock(i)` (temporary, no tryFlush) | `P<optional<Buf>>` via rx/getBlock/tryFlush | y | ix:1386-1390 `readBlock(rx, index)` | returns promise (awaited at ix:945/956/965) | used by SE:470 (`readBlock_`) and SE:1353 (`proof`) |
71| 21 | SS:280-283 | `createWriteBatch` | `activeTx_.emplace(storage->write())`; returns `l2::CoreTX&` | `activeTx_ = storage->write()` (`TX`); return `CoreTX&` | n | ss:236-241 | sync | `std::optional<l2::CoreTX> activeTx_` (SSh:206) → `std::shared_ptr<CoreTX>` |
72| 22 | SS:281 | `createWriteBatch` | `storage->snapshotted()` | `HS::snapshotted()` | n | ss:237 | sync | |
73| 23 | SS:298-304 | `SessionState::flush` | `tx->flush()` → bool | `co_await tx->flush()` → `P<bool>`; `flush()` becomes `P<bool>` | **y** | ss:259-271 `async flush()`: `await tx.flush()` in try/finally `_clearActiveBatch` | await | every caller below must `co_await` it |
74| 24 | SS:275-276 | `updateGroupIndex` | `tx.deleteGroupUpdate`, `tx.putGroupUpdate` | `CoreTX::deleteGroupUpdate/putGroupUpdate` (void) | n | ss:226-227 | sync | |
75| 25 | SS:306-320 | `locked_` | callback mutex wrapper; `body()` must return synchronously | `P<…>` coroutine: `co_await mutex.lock(); try { … } finally { _unlock(); }` | y | (pattern of every `await this.mutex.lock(); try {…} finally { this._unlock() }`) | await | **Hazard H3**: every body below gains awaits ⇒ `locked_`'s synchronous `body()` contract cannot survive; Mutex needs a Promise-returning `lock()` |
76| 26 | SS:324 | `setUserData` | `createWriteBatch()` | `storage->write()` | n | ss:351 | sync | |
77| 27 | SS:325-326 | `setUserData` | `tx.putUserData` / `tx.deleteUserData` | same (void) | n | ss:352 (`putUserData(key, null)` pushes `[key, null, null]` = a delete, tx.js:114-121) | sync | equivalent |
78| 28 | SS:327 | `setUserData` | `flush()` | `co_await flush()` | y | ss:354 `return await this.flush()` | await | |
79| 29 | SS:336 | `verifyBlock` | `core->downgrade(batch)` → `merkle::downgrade(batch, Core&)` (Core::get_tree_node per root) | async merkle downgrade over one rx | y | ss:364-366 `await batch.downgrade()` → mt:301-325 (ONE rx, all roots, tryFlush, Promise.all) | await | **Deviation**: C++ = one rx per root; JS = one batch (H1, H5) |
80| 30 | SS:342 | `verifyBlock` | `createWriteBatch()` | `storage->write()` | n | ss:376 | sync | |
81| 31 | SS:343 | `verifyBlock` | `tx.putBlock` | void | n | ss:379 | sync | |
82| 32 | SS:344 | `verifyBlock` | `storeBitfieldRange_(tx, …)` | async (see #66-#69) | y | ss:382-384 `await storeBitfieldRange(...)` | await | |
83| 33 | SS:345 | `verifyBlock` | `core->setManifest_(tx, …)` → `tx.setAuth` (HC:492) | void | n | ss:386 → cj:416-441 | sync | |
84| 34 | SS:348 | `verifyBlock` | `commitBatch(tx, batch)` → put/delete tree nodes | void | n | ss:389 → mt:232-270 | sync | |
85| 35 | SS:352 | `verifyBlock` | `tx.setHead` | void | n | ss:400 | sync | |
86| 36 | SS:353 | `verifyBlock` | `updateGroupIndex(tx, head)` | void | n | ss:403 | sync | |
87| 37 | SS:356 | `verifyBlock` | `flush()` | `co_await flush()` | y | ss:407 | await | |
88| 38 | SS:369 | `verifyBlock` | `core->flushHints()` (HC:1065-1071 writes) | `co_await` (see #121) | y | ss:419 `await this.core.flushHints()` | await | |
89| 39 | SS:332-373 | `verifyBlock` | whole op is **synchronous**, asserts `mutex.idle()` (SS:333) instead of awaiting the lock; returns bool | `P<bool>`; real `co_await mutex.lock()` | y | ss:360-425 `async _verifyBlock`, `await this.mutex.lock()` (ss:361), `this.core.preupdate` await (ss:372-374) | await | **Hazard H2** |
90| 40 | SS:377 | `clear` | `createWriteBatch()` | write | n | ss:562 | sync | |
91| 41 | SS:397-398 | `clear` | `tx.deleteTreeNode` ×2 | void | n | ss:576-577 | sync | |
92| 42 | SS:402 | `clear` | `storeBitfieldRange_` | async | y | ss:581 | await | |
93| 43 | SS:404 | `clear` | `tx.setHints` | void | n | ss:583-586 | sync | |
94| 44 | SS:408-409 | `clear` | `tx.deleteBlock` / `tx.deleteBlockRange` | void | n | ss:590-591 | sync | |
95| 45 | SS:412 | `clear` | `updateDependency_` → `storage->findDependencyIndex`, `dependencies()` | sync | n | ss:593 → ss:1344-1352 | sync | |
96| 46 | SS:414 | `clear` | `flush()` | `co_await flush()` | y | ss:595 | await | |
97| 47 | SS:416 | `clear` | `storage->setDependencyHead` | sync | n | ss:597 | sync | |
98| 48 | SS:424 | `clear` | `core->flushHints()` | async | y | ss:607 | await | |
99| 49 | SS:434-445 | `append` | `waitForRecovery(cb)` then re-enter `append` with `recovered = true` | `co_await waitForRecovery()` inline | y | ss:664 | await | existing callback wrapper; `AppendOpts::recovered` (SSh:123, keetd-internal) disappears |
100| 50 | SS:456 | `append` | `createWriteBatch()` | write | n | ss:673 | sync | |
101| 51 | SS:459 | `append` | `core->setManifest_` (tx.setAuth) | void | n | ss:676 | sync | |
102| 52 | SS:461 | `append` | `opts.preappend(vals)` (sync encryption) | if providers become async: `co_await` | y? | ss:678 `if (preappend) await preappend(values)` → ix:1359-1366 (awaits `encryption.encrypt` each) | await | not storage; but JS hop between setManifest_ and the empty check. Hop parity only. |
103| 53 | SS:464 | `append` | `flush()` (empty append) | `co_await flush()` | y | ss:681 | await | |
104| 54 | SS:487 | `append` | `commitBatch(tx, batch)` | void | n | ss:700 | sync | |
105| 55 | SS:490 | `append` | `tx.setHead` | void | n | ss:710 | sync | |
106| 56 | SS:493 | `append` | `storeBitfieldRange_` | async | y | ss:713 | await | |
107| 57 | SS:495 | `append` | `tx.setHints` | void | n | ss:715-718 | sync | |
108| 58 | SS:497 | `append` | `updateGroupIndex` | void | n | ss:722 | sync | |
109| 59 | SS:500 | `append` | `tx.putBlock` ×n | void | n | ss:726-728 | sync | |
110| 60 | SS:504 | `append` | `flush()` | `co_await flush()` | y | ss:736 | await | |
111| 61 | SS:512 | `append` | `opts.postappend(vals)` (sync) | `co_await` if made async | y? | ss:744 `await postappend(values)` | await | not storage; hop parity |
112| 62 | SS:516 | `append` | `core->flushHints()` | async | y | ss:748 | await | |
113| 63 | SS:536 | `truncate` | `merkle::truncate(*this, len, batch, f)` (SessionState::get_tree_node per missing root) | async merkle truncate | y | ss:444 `await MerkleTree.truncate(this, …)` → mt:619-641 (`await getTreeNodeFromStorageOrError(session.storage, root)` **one rx per root, sequential**) | await (per node, sequential) | 1:1 with JS's per-root rx once get_tree_node is async (H1) |
114| 64 | SS:548 | `truncate` | `createWriteBatch()` | write | n | ss:453 | sync | |
115| 65 | SS:551 | `truncate` | `core->setManifest_` | void | n | ss:456 | sync | |
116| 66a | SS:553 | `truncate` | `truncate_(tx, batch)` (sync) | `P<Truncated>` | y | ss:458 `await this._truncate(tx, batch)` | await | |
117| 66b | SS:561 | `truncate` | `flush()` | `co_await flush()` | y | ss:470 | await | |
118| 66c | SS:570 | `truncate` | `storage->setDependencyHead` | sync | n | ss:478 | sync | |
119| 66d | SS:574 | `truncate` | `core->flushHints()` | async | y | ss:482 | await | |
120| 67 | SS:580-606 | `reorg` | sync, asserts mutex idle (SS:580) | `P<bool>`, `co_await mutex.lock()` | y | ss:488-514 | await | H2. JS creates the batch (ss:491) OUTSIDE its try (a throw there leaks the lock) — C++ mirrors inside `locked_` |
121| 68 | SS:585 | `reorg` | `createWriteBatch()` | write | n | ss:491 | sync | |
122| 69 | SS:589 | `reorg` | `truncate_` | async | y | ss:496 | await | |
123| 70 | SS:591 | `reorg` | `flush()` | async | y | ss:498 | await | |
124| 71 | SS:599 | `reorg` | `setDependencyHead` | sync | n | ss:506 | sync | |
125| 72 | SS:603 | `reorg` | `core->flushHints()` | async | y | ss:510 | await | |
126| 73 | SS:610 | `truncate_` | `tx.deleteBlockRange` | void | n | ss:517 | sync | |
127| 74 | SS:616 | `truncate_` | `updateGroupIndex` | void | n | ss:529-531 | sync | |
128| 75 | SS:618 | `truncate_` | `tx.setHead` | void | n | ss:533 | sync | |
129| 76 | SS:619 | `truncate_` | `commitBatch` | void | n | ss:534 | sync | |
130| 77 | SS:622 | `truncate_` | `updateDependency_` | sync | n | ss:537 | sync | |
131| 78 | SS:624 | `truncate_` | `tx.setDependency` | void | n | ss:539 | sync | |
132| 79 | SS:627 | `truncate_` | `storeBitfieldRange_` | async | y | ss:542 | await | the ONLY await in `_truncate`: makes `_truncate` async in JS |
133| 80 | SS:630 | `truncate_` | `tx.setHints` | void | n | ss:548-551 | sync | |
134| 81 | SS:638, 640 | `updateDependency_` | `storage->findDependencyIndex`, `storage->dependencies()` | sync | n | ss:1344-1352 | sync | |
135| 82 | SS:722 | `storeBitfieldRange_` | `l2::CoreRX rx = storage->read()` | `RX rx = storage->read()` | n | ss:1307 | sync | |
136| 83 | SS:724 | `storeBitfieldRange_` | `rx.getBitfieldPage(p)` **used at once in the loop** | `P<optional<Buf>> CoreRX::getBitfieldPage(int64_t)` | y | ss:1309-1312 (all pages issued), ss:1314 tryFlush, ss:1316 `await Promise.all` | Promise.all | **Deviation / hazard H4**: C++ never calls `rx.tryFlush()` and fills/puts each page as it reads it. New: issue all, tryFlush, `co_await all`, then fill + `tx.putBitfieldPage` in page order (ss:1319-1325) |
137| 84 | SS:727 | `storeBitfieldRange_` | `tx.putBitfieldPage` | void | n | ss:1324 | sync | |
138| 85 | SS:735-743 | `StorageNodes::get_tree_node` (anon) | `s.read().getTreeNode(i)` temporary, no tryFlush | async (H1) | y | (used by `rootsFromStorage`, below) | — | |
139| 86 | SS:744-751 | `rootsFromStorage` → `merkle::getRootsFromStorage(StorageNodes&)` | one rx PER root | `MerkleTree.getRootsFromStorage` port: ONE rx, all roots via `getTreeNodeOrError`, tryFlush, `Promise.all` | y | mt:579-591 | returns Promise.all | **Deviation H5**: hop/batch count differs (N batches vs 1). Callers: SS:755, SS:790, SS:1075 |
140| 87 | SS:755 | `getTreeHeadAt_` | `rootsFromStorage(*storage, len)` | async | y | ss:1114 `await MerkleTree.getRootsFromStorage(this.storage, length)` | await | `_getTreeHeadAt` is async in JS (ss:1107) |
141| 88 | SS:771 | `createSession` | `storage->resumeSession(name)` → `unique_ptr` | `P<HSP> HS::resumeSession(std::string)` (null when none) | y | ss:1224 | await | |
142| 89 | SS:773 | `createSession` | `st->read().getHead()` (temporary, no tryFlush) | `getCoreHead`: rx, `getHead()`, `tryFlush()`, `co_await` | y | ss:1226 `(await getCoreHead(storage)) \|\| getDefaultTree()` → ss:1363-1368 | await | **Deviation**: missing tryFlush (H4-class) |
143| 90 | SS:781 | `createSession` | `getTreeHeadAt_(len)` | async | y | ss:1232 | await | |
144| 91 | SS:782 | `createSession` | `storage->createAtomicSession(atom, treeInfo)` | `P<HSP> HS::createAtomicSession(AtomPtr, optional<hs::CoreHead>)` | y | ss:1235 | await | |
145| 92 | SS:783 | `createSession` | `storage->createSession(name, treeInfo)` | `P<HSP> HS::createSession(std::string, optional<hs::CoreHead>)` | y | ss:1237 | await | |
146| 93 | SS:790 | `createSession` | `rootsFromStorage(*st, len)` | async | y | ss:1250 `await MerkleTree.getRootsFromStorage(storage, length)` | await | |
147| 94 | SS:798 | `createSession` | `make_shared<SessionState>(…, st, …)` | ctor takes `HSP` | n | ss:1256-1262 | sync | |
148| 95 | SS:802 | `createSession` | `atomized = atom` (`shared_ptr<l2::Atom>`) | `AtomPtr` | n | ss:1265 | sync | |
149| 96 | SS:805-809 | `createSession` | `atom->preflush(fn)` | `Atom::preflush(std::function<void()>)` | n | ss:1266 | sync | weak-ptr guard is keetd-internal (JS binds the objects) |
150| 97 | SS:810-814 | `createSession` | `atom->onflush([](l2::Atom::Done d){ c->commit_(d) })` | `Atom::onflush(Handler)`, `Handler = std::function<P<void>()>` returning `child->commit_()` | y | ss:1267 `atom.onflush(state._commit.bind(state))`; hs:76-78 `promises.push(this.flushes[i]())`, `await Promise.all` | Promise.all | existing Done-callback wrapper (M5g-3c) → direct Promise. Gone child: `co_return` a rejected `Error("Closed")` (JS: closed state's destroyed mutex rejects) |
151| 98 | SS:765-819 | `createSession` | callback `done(Fail, state)` wrapper | `P<std::shared_ptr<SessionState>>` | y | ss:1216-1274 | await (ix:444/449) | existing wrapper |
152| 99 | SS:824 | `precommit_` | `state.storage->core().dataPointer`, `storage->core().dataPointer` | `storage->core->dataPointer` | n | ss:274-277 | sync | |
153| 100 | SS:829-847 | `commit_` | callback `mutex.lock` + `parent->oncommit_(…, cb)` | `P<void>`: `co_await mutex.lock(); try { co_await parent->oncommit_(…) } finally { mutex.unlock() }` | y | ss:280-291 | await | existing wrapper; note `_commit` uses `mutex.unlock()` not `_unlock()` (ss:289) — C++ matches |
154| 101 | SS:854 | `oncommit_` | `storage->read()` | `RX` | n | ss:300 | sync | |
155| 102 | SS:855 | `oncommit_` | `rx.getDependency()` | `P<optional<hs::CoreDependency>>` | y | ss:301, awaited ss:305 | await | |
156| 103 | SS:856 | `oncommit_` | `rx.tryFlush()` | same | n | ss:303 | sync | mechanical: getter, tryFlush, await — already in JS order |
157| 104 | SS:867 | `oncommit_` | `createWriteBatch()` | write | n | ss:322 | sync | |
158| 105 | SS:868 | `oncommit_` | `updateGroupIndex` | void | n | ss:323 | sync | |
159| 106 | SS:869 | `oncommit_` | `flush()` | async | y | ss:324 | await | |
160| 107 | SS:873 | `oncommit_` | `storage->setDependencyHead` | sync | n | ss:328 | sync | |
161| 108 | SS:888 | `oncommit_` | `core->flushHints()` | async | y | ss:341 | await | |
162| 109 | SS:976-986 | `waitForRecovery` (Run::round) | `self->mutex.lock(cb)`, `core->flushHints()` | `co_await mutex.lock(); try { co_await core->flushHints() } finally { mutex.unlock() }` | y | ss:649-655 | await | existing callback wrapper over rt timers; loop itself is timer-driven (not storage) |
163| 110 | SS:998 | `truncateAndFlush_` | `merkle::truncate(*this, …)` | async (H1) | y | ss:1330 | await | |
164| 111 | SS:1002 | `truncateAndFlush_` | `createWriteBatch()` | write | n | ss:1331 | sync | |
165| 112 | SS:1004 | `truncateAndFlush_` | `truncate_` | async | y | ss:1333 | await | |
166| 113 | SS:1005 | `truncateAndFlush_` | `flush()` | async | y | ss:1334 | await | |
167| 114 | SS:1048 | `moveTo` | `lingers.push_back(storage)` | `std::vector<HSP>` | n | ss:1158-1159 | sync | lingers must be `close()`d (row 4) |
168| 115 | SS:1053 | `moveTo` | `state->storage->resumeSession(n)` | `P<HSP>` | y | ss:1161 | await | |
169| 116 | SS:1056 | `moveTo` | `truncateAndFlush_(len)` | async | y | ss:1163 | await | |
170| 117 | SS:1070 | `moveTo` | `state->getTreeHeadAt_(length)` | async | y | ss:1168 | await | |
171| 118 | SS:1072 | `moveTo` | `state->storage->createSession(n, treeInfo)` | `P<HSP>` | y | ss:1180 | await | |
172| 119 | SS:1075 | `moveTo` | `rootsFromStorage(*st, len)` | async | y | ss:1183 | await | |
173| 120 | SS:1085 | `moveTo` | `storage->setDependencyHead` | sync | n | ss:1195 | sync | |
174| 120a | SS:1035-1103 | `moveTo` | callback wrapper via `locked_` | `P<void>` | y | ss:1142-1214 | await | |
175| 120b | SS:1108 | `overwrite_` | `source.storage->read()` | `RX` | n | ss:917 | sync | |
176| 120c | SS:1112 | `overwrite_` | `rx.getTreeNode(root)` ×roots | `P<optional<hs::CoreTreeNode>>` | y | ss:919-921 → Promise.all ss:935-939 | Promise.all (nested: blocks, nodes, roots) | C++ issue order roots → patch → per-i (node,node,block) = JS order; values consumed only after tryFlush ⇒ mechanical (collect promises, tryFlush, `co_await all` ×3) |
177| 120d | SS:1113 | `overwrite_` | `rx.getTreeNode(index)` ×patch | same | y | ss:923-925 | Promise.all | |
178| 120e | SS:1115-1116 | `overwrite_` | `rx.getTreeNode(i*2)`, `(i*2+1)` | same | y | ss:928-929 | Promise.all | |
179| 120f | SS:1117 | `overwrite_` | `rx.getBlock(i)` | `P<optional<Buf>> CoreRX::getBlock(int64_t)` | y | ss:930 | Promise.all | |
180| 120g | SS:1119 | `overwrite_` | `rx.tryFlush()` | same | n | ss:933 | sync | |
181| 120h | SS:1135 | `overwrite_` | `createWriteBatch()` | write | n | ss:956 | sync | |
182| 120i | SS:1138 | `overwrite_` | `tx.deleteBlockRange` | void | n | ss:959-961 | sync | |
183| 120j | SS:1140, 1143 | `overwrite_` | `tx.putTreeNode` (roots, nodes) | void | n | ss:963, 966-968 | sync | |
184| 120k | SS:1147 | `overwrite_` | `tx.putBlock` | void | n | ss:970-973 | sync | |
185| 120l | SS:1156 | `overwrite_` | `storage->read().getBitfieldPage(firstPage)` (temporary, **no tryFlush**) | rx, getBitfieldPage, tryFlush, `co_await` | y | ss:981-985 | await | **Deviation**: tryFlush missing (H4-class) |
186| 120m | SS:1169, 1174 | `overwrite_` | `tx.putBitfieldPage` (C++ puts the FILLED page; JS puts first, mutates after, ss:990-1004) | void | n | ss:991 | sync | C++ deliberately re-ordered (same bytes since tx keeps the buffer by reference in JS). New CoreTX `push_(Buf …)` copies like l2 ⇒ keep C++'s fill-then-put |
187| 120n | SS:1180 | `overwrite_` | `updateGroupIndex` | void | n | ss:1015-1017 | sync | |
188| 120o | SS:1184 | `overwrite_` | `tx.setHead` | void | n | ss:1021 | sync | |
189| 120p | SS:1186 | `overwrite_` | `flush()` | async | y | ss:1023 | await | |
190| 121a | SS:1200 | `isParent` | `l2::HypercoreStorage::isParentStorage` | `static bool HS::isParentStorage(const HS&, const HS&)` (or `CS::isParentStorage`) | n | ss:1038 | sync | |
191| 121b | SS:1214-1222 | `commit` | `waitForRecovery(cb)` | `co_await` | y | ss:1053 | await | existing wrapper |
192| 121c | SS:1233-1234 | `commitLocked_` | nested callback `mutex.lock` / `state->mutex.lock` | two `co_await lock()` with `srcLocked` flag | y | ss:1056, 1059-1060 | await | existing wrapper |
193| 121d | SS:1240 | `commitLocked_` | `core->validateCommit_(*state, treeLength)` (sync; reads via get_tree_node) | `P<bool>` | y | ss:1064 `await this.core._validateCommit(...)` → cj:466-485 | await | |
194| 121e | SS:1248 | `commitLocked_` | `merkle::restore(batch, *state, len)` (per-root rx) | async restore over one rx | y | ss:1070 `await batch.restore(length)` → mt:327-330 `getRootsFromStorage(this.storage, length)` (one rx) | await | H5 |
195| 121f | SS:1255 | `commitLocked_` | `overwrite_` | `P<Overwritten>` | y | ss:1074-1080 | await | |
196| 121g | SS:1259 | `commitLocked_` | `state->createWriteBatch()` | write | n | ss:1084 | sync | |
197| 121h | SS:1260 | `commitLocked_` | `tx.deleteBlockRange` | void | n | ss:1086 | sync | |
198| 121i | SS:1261-1262 | `commitLocked_` | `state->updateDependency_` + `tx.setDependency` | sync/void | n | ss:1087 → ss:217-221 `updateDependency(tx, length)` | sync | |
199| 121j | SS:1263 | `commitLocked_` | `state->flush()` | async | y | ss:1089 `await state.flush(tx)` | await | |
200| 121k | SS:1264 | `commitLocked_` | `state->storage->setDependencyHead` | sync | n | ss:1091 | sync | |
201| 122a | SS:1300 | `catchup` | `storage->dependencies()` | sync | n | ss:840-841 | sync | |
202| 122b | SS:1301 | `catchup` | `core->state->storage->core().dataPointer` | `->core->dataPointer` | n | ss:842 | sync | |
203| 122c | SS:1306 | `catchup` | `createWriteBatch()` | write | n | ss:848 | sync | |
204| 122d | SS:1307 | `catchup` | `core->state->storage->read()` | `RX` | n | ss:849 | sync | |
205| 122e | SS:1309 | `catchup` | `rx.getTreeNode(root)` ×roots | `P<…>` | y | ss:852-854 → ss:858 `await Promise.all` | Promise.all | mechanical (values used only after tryFlush) |
206| 122f | SS:1310 | `catchup` | `rx.tryFlush()` | same | n | ss:856 | sync | |
207| 122g | SS:1323-1325 | `catchup` | `tx.deleteBlockRange/deleteTreeNodeRange/deleteBitfieldPageRange(0, -1)` | void | n | ss:873-875 | sync | |
208| 122h | SS:1330 | `catchup` | `tx.setHead` | void | n | ss:886 | sync | |
209| 122i | SS:1333, 1337 | `catchup` | `updateDependency_`, `tx.setDependency` | sync/void | n | ss:889-892 | sync | |
210| 122j | SS:1339 | `catchup` | `flush()` | async | y | ss:894 | await | |
211| 122k | SS:1341 | `catchup` | `storage->setDependencyHead` | sync | n | ss:896 | sync | |
212| 122l | SS:1351 | `catchup` | `core->flushHints()` | async | y | ss:906 | await | |
213| 123 | SS:1367, 1375, 1383 | `commitBatch` | `tx.deleteTreeNodeRange`, `tx.deleteTreeNode`, `tx.putTreeNode` | void | n | mt:232-270 (`commit(tx)`, `_commitUpgrade(tx)`) | sync | |
214
215### A2. hypercore.cpp / hypercore.h
216
217| # | file:line | C++ fn | l2 call / use | new hc::storage equivalent | P? | JS counterpart | JS | notes |
218|---|---|---|---|---|---|---|---|---|
219| 124 | HC:266-431 | `Core::Core` | whole `_tryOpen` in a **constructor** | `P<void> Core::open_()` (JS `this.opening = this._open(opts)`, cj:67) | y | cj:148-163 `_open`, cj:165-366 `_tryOpen` | await | **Hazard H6**: a constructor cannot await. Callers outside core/: store/src/corestore.cpp:420, repl/test/r3c_walk_test.cpp:91, repl/test/r3c_scan_test.cpp:73. SE:195 assumes "the Core is open (corestore opened it synchronously)". |
220| 125 | HC:273 | `Core::Core` | `db.resumeCore(dk)` → `unique_ptr` | `P<HSP> CS::resumeCore(std::optional<K32>)` | y | cj:168 | await | |
221| 126 | HC:283 | `Core::Core` | `storage_->read()` | `RX` | n | cj:1050 (`getCoreInfo`) | sync | |
222| 127 | HC:284 | `Core::Core` | `rx.getAuth()` | `P<optional<hs::CoreAuth>>` | y | cj:1052 | Promise.all (cj:1059) | |
223| 128 | HC:285 | `Core::Core` | `rx.getHead()` | `P<optional<hs::CoreHead>>` | y | cj:1053 | Promise.all | |
224| 129 | HC:286 | `Core::Core` | `rx.getHints()` | `P<optional<hs::CoreHints>>` | y | cj:1054 | Promise.all | |
225| 130 | HC:287 | `Core::Core` | `rx.getGroup()` | `P<optional<hs::CoreGroup>>` | y | cj:1055 | Promise.all | **Deviation**: C++ never calls `rx.tryFlush()` (cj:1057 does) — H4-class; values used at once |
226| 131 | HC:334-341 | `Core::Core` | `l2::CreateCoreArgs`, `db.createCore(c)` | `storage::CreateCoreArgs`, `P<HSP> CS::createCore(CreateCoreArgs)` | y | cj:238-246 | await | JS passes `userData: opts.userData \|\| []` and `alias: opts.alias \|\| null` — new struct's comment "empty = absent" matches |
227| 132 | HC:367 | `Core::Core` (Bitfield.open) | `storage_->createBitfieldStream(0, pages)` (vector) | `Stream HS::createBitfieldStream(IndexRange{0, pages})`, `mapped<PageEntry>` | y (stream) | cj:291 → bf:536-548 `for await (const { index, page } of stream)` | for-await | JS: `{ lt: pages }` |
228| 133 | HC:377 | `Core::Core` | `storage_->read()` | `RX` | n | cj:297 → cj:1033-1041 `maybeGetRootsFromStorage` → mt:579-591 | sync | |
229| 134 | HC:379-381 | `Core::Core` | `rx.getTreeNode(r)` used at once; null → `treeRoots.clear(); break` | `getRootsFromStorage` port: all `getTreeNodeOrError`, tryFlush, `Promise.all`; INVALID_OPERATION → `[]` | y | mt:579-591 + cj:1036-1040 catch | await (Promise.all) | **Deviation**: no tryFlush; early break. Same RESULT (first null in index order → []), different batch shape |
230| 135 | HC:389→791 | `setGroupLocked_` (from ctor) | `storage_->store->createGroup(group)` | `P<std::pair<K32,uint64_t>> CS::createGroup(K32)` | y | cj:308 `await this.db.createGroup(opts.group)` | await | JS uses `this.db` (the CorestoreStorage the core was opened with); C++ `storage_->store` is the same object |
231| 136 | HC:798-803 | `setGroupLocked_` | `storage_->write()`, `tx.setGroup`, `tx.flush()` | write / `setGroup` / `co_await tx->flush()` | y (flush) | cj:315-317 (open) / cj:388-390 (setGroup) | await | |
232| 137 | HC:392-396 | `Core::Core` (overwrite) | write, `deleteTreeNodeRange`, `deleteBlockRange`, `deleteBitfieldPageRange`, `flush` | same, `co_await flush` | y | cj:321-325 (`bitfield.clear(tx)` = bf:526-528 `tx.deleteBitfieldPageRange(0, -1)`) | await | |
233| 138 | HC:400 | `Core::Core` | `storage_->readOnly()` | `bool HS::readOnly() const` | n | cj:329 | sync | |
234| 139 | HC:402-404 | `Core::Core` | write, `setHints`, `flush` | same | y | cj:331-333 | await | |
235| 140 | HC:410-417 | `Core::Core` | write, `setAuth`, `flush` | same | y | cj:340-347 | await | |
236| 141 | HC:427 | `Core::Core` | `make_shared<SessionState>(…, storage_, …)` | `HSP` | n | cj:360 | sync | |
237| 142 | HC:433-437 | `Core::~Core` | `state->close()` (sync) | `P<void>` (row 4) — cannot await in a destructor | y | cj:942-955 `destroy` → `await this.state.close()` (cj:951) | await | H6 / H7 |
238| 143 | HC:445-449 | `Core::get_tree_node` (merkle::Storage) | `storage_->read().getTreeNode(i)` temporary, no tryFlush | async (H1) | y | mt:1280-1285 `getTreeNodeFromStorage` | returns promise | used via `Core::session()` (HC:439-443) by `merkle::verify` (HC:1083), `merkle::downgrade` (HC:1074), `merkle::missingNodes` (HC:1393) |
239| 144 | HC:451 | `Core::get_block` | `storage_->read().getBlock(i)` temporary, no tryFlush | rx/getBlock/tryFlush | y | ix:956 `readBlock(this.core.state.storage.read(), index)` | await | used by SE:934 |
240| 145 | HC:457-475 | `Core::setManifest` | sync; asserts mutex idle (HC:464) | `P<void>`, `co_await state->mutex.lock()` | y | cj:396-414 | await | H2 |
241| 146 | HC:467 | `Core::setManifest` | `state->createWriteBatch()` | write | n | cj:407 | sync | |
242| 147 | HC:469 | `Core::setManifest` | `state->flush()` | async | y | cj:409 | await | |
243| 148 | HC:492 | `Core::setManifest_` | `tx.setAuth(a)` | void | n | cj:428-434 | sync | |
244| 149 | HC:600-602 | `flushPrologueBatch` | `src.storage->read()`, `getTreeNode` ×tree, `tryFlush` | RX / `P<…>` / tryFlush | y | cp:67-73 (`await Promise.all(nodePromises)`) | Promise.all | mechanical |
245| 150 | HC:605-618 | `flushPrologueBatch` | `dst.storage().read()`, `getHead()` (first batch only), `getBitfieldPage` ×pages, `tryFlush` | same | y | cp:76-92 | Promise.all + await head | **Ordering**: JS awaits the src nodes (cp:73) BEFORE issuing the dst reads (cp:75-89); C++ also flushes src first (HC:602) then dst. OK. JS `headPromise.catch(noop)` (cp:79) — reproduce |
246| 151 | HC:633 | `flushPrologueBatch` | `src.storage->createUserDataStream()` (vector) | `Stream HS::createUserDataStream(UserDataRange{})`, `mapped<UserDataEntry>` | y (stream) | cp:108-110 `for await (...) userData.push(data)` | for-await | |
247| 152 | HC:637 | `flushPrologueBatch` | `dst.storage().write()` | write | n | cp:116 | sync | |
248| 153 | HC:641 | `flushPrologueBatch` | `tx.putTreeNode` | void | n | cp:118 | sync | JS `putTreeNode(null)` TypeError mirrored (HC:640) |
249| 154 | HC:661, 663 | `flushPrologueBatch` | `tx.putBitfieldPage(pageIndex, …)`, `tx.putBlock` | void | n | cp:130, cp:135 | sync | C++ sets bits first then puts (tx keeps buffer by ref in JS) — keep |
250| 155 | HC:667 | `flushPrologueBatch` | `tx.putUserData` | void | n | cp:138-140 | sync | |
251| 156 | HC:670 | `flushPrologueBatch` | `tx.setHead` | void | n | cp:143-145 | sync | |
252| 157 | HC:672 | `flushPrologueBatch` | `tx.flush()` | async | y | cp:147 | await | |
253| 158 | HC:734 | `copyPrologueInto` | `src.storage->createBlockStream(0, len, true)` (collected FIRST) | `Stream HS::createBlockStream(IndexRange{0, len, true})`, `mapped<BlockEntry>` | y (stream) | cp:32-56 `for await (… createBlockStream({gte:0, lt, reverse:true}))` with `await flushBatch(...)` INSIDE the loop (cp:52) | for-await + await in body | **Deviation H8**: C++ drains the whole stream before any flush; JS interleaves stream reads with dst writes (backpressure) |
254| 159 | HC:752, 761 | `copyPrologueInto` | `flushPrologueBatch` (sync) | `P<void>` | y | cp:52, cp:61 | await | |
255| 160 | HC:766-786 | `Core::copyPrologue` | callback mutex wrapper | `P<void>` | y | cj:443-460 | await | existing wrapper |
256| 161 | HC:809 | `Core::compact` | `s->storage->compact()` (void, sync) | `P<std::vector<js::Undefined>> HS::compact()` | y | cj:124-130 `compacting.push(s.storage.compact()); return Promise.all(compacting)` | returns Promise.all | `Core::compact` → `P<…>`; SE:1147 must await it (ix:234-236 returns it) |
257| 162 | HC:842 | `Core::audit` | `merkle::getRootsFromStorage(*state, length)` (per-root rx via SessionState) | getRootsFromStorage over `state->storage` (one rx) | y | au:25 | await | H5. JS's `r === null` → `deleteCore` branch (au:28-38) is unreachable in JS too (getTreeNodeOrError throws first, mt:1265-1271) |
258| 163 | HC:860-863 | `Core::audit` | `state->storage->read()`, `getTreeNode(l)`, `getTreeNode(r)`, `tryFlush` | RX / 2×`P` / tryFlush / `co_await all` | y | au:51-57 | Promise.all | mechanical |
259| 164 | HC:866, 869 | `Core::audit` | `tx.emplace(state->storage->write())`, `tx->deleteTreeNodeRange` | write / void | n | au:60-62 | sync | `std::optional<l2::CoreTX>` → `TX` (null) |
260| 165 | HC:881 | `Core::audit` | `tx->flush()` | async | y | au:73 | await | |
261| 166 | HC:887 | `Core::audit` | `state->storage->createBlockStream()` (collected) | `Stream`, `mapped<BlockEntry>` | y (stream) | au:80 `for await` with awaits inside (au:92) | for-await + await in body | H8-class; reads (tree nodes) during iteration, writes only after the loop |
262| 167 | HC:890-891, 900-901 | `Core::audit` | write, `tx->deleteBlock` | void | n | au:82-83, au:95-96 | sync | |
263| 168 | HC:895-897 | `Core::audit` | `read()`, `getTreeNode(2*index)`, `tryFlush` | RX/P/tryFlush | y | au:87-92 | await | mechanical |
264| 169 | HC:909 | `Core::audit` | `tx->flush()` | async | y | au:104 | await | |
265| 170 | HC:923-925 | `Core::audit` | `read()`, `getBlock(index)`, `tryFlush` | RX/P/tryFlush | y | au:111-116 | await | mechanical |
266| 171 | HC:930, 939, 941 | `Core::audit` | write, `putBitfieldPage`, `deleteBitfieldPage` | void | n | au:121, 126-127 | sync | |
267| 172 | HC:949 | `Core::audit` | `tx->flush()` | async | y | au:134 | await | |
268| 173 | HC:828-959 | `Core::audit` | callback wrapper (`state->mutex.lock(cb)`, `done(err, stats)`) | `P<std::optional<AuditStats>>` | y | cj:368-376 | await | existing wrapper |
269| 174 | HC:961-973 | `Core::setGroup` | callback wrapper; `setGroupLocked_` | `P<void>` | y | cj:378-394 | await | uses `state->mutex.unlock()` (cj:392), not `_unlock` — C++ matches |
270| 175 | HC:982 | `Core::validateCommit_` | `s.get_tree_node(root.index)` per root | `co_await MerkleTree.get(state, index)` (one rx per root, sequential) | y | cj:473 → mt:615-617 → mt:1280-1285 | await (sequential per root) | 1:1 once get_tree_node is async |
271| 176 | HC:1065-1071 | `Core::flushHints` | `storage_->write()`, `setHints`, `flush()` | `P<void>` | y | cj:605-618 (`this.state.storage.write()`) | await | storage_ == state->storage for the default state |
272| 177 | HC:1074 | `Core::downgrade` | `merkle::downgrade(batch, *this)` (per-root rx) | one rx, Promise.all | y | mt:301-325 | await | H5 |
273| 178 | HC:1083 | `Core::verify` | `merkle::verify(session(), …)` → reads the unverified node through `Core::get_tree_node` | async | y | cj:798 `await MerkleTree.verify(this.state, proof)` → mt:704-709 (`getTreeNodeFromStorageOrError`) | await | |
274| 179 | HC:1088 | `Core::verify` | `downgrade(batch)` | async | y | cj:800-802 | await | |
275| 180 | HC:1099 | `Core::verify` (exclusive) | `state->verifyBlock(...)` | `P<bool>` | y | cj:514-533 `_verifyExclusive` → cj:518 `await this.state._verifyBlock` | await | |
276| 181 | HC:1107-1108 | `Core::verify` (shared) | asserts mutex idle; `mutex.lock([]{})` | `co_await state->mutex.lock()` | y | cj:538 | await | **Hazard H2**: JS batches concurrent shared verifies (`_verifies`, cj:821-831); keetd's "one verify at a time" assumption breaks once verify suspends |
277| 182 | HC:1110 | `Core::verify` (shared) | `state->createWriteBatch()` | write | n | cj:540 | sync | |
278| 183 | HC:1111 | `Core::verify` (shared) | `tx.putBlock` | void | n | cj:550-552 | sync | |
279| 184 | HC:1116 | `Core::verify` (shared) | `setManifest_` (setAuth) | void | n | cj:570-575 | sync | |
280| 185 | HC:1118 | `Core::verify` (shared) | `state->commitBatch` | void | n | cj:577 | sync | |
281| 186 | HC:1129 | `Core::verify` (shared) | `tx.putBitfieldPage` (in-memory page + range) | void | n | cj:580 `bits.flush(tx, this.bitfield)` (bit-interlude.js) | sync | no storage read |
282| 187 | HC:1131 | `Core::verify` (shared) | `state->flush()` | async | y | cj:582 | await | |
283| 188 | HC:1138 | `Core::verify` (shared) | `flushHints()` | async | y | cj:596 | await | |
284| 189 | HC:1140-1145 | `Core::verify` (shared) | `clearActiveBatch`, `mutex.unlock` | same | n | cj:597-600 (finally) | sync | |
285| 190 | HC:1150-1152 | `Core::verifyReorg` | `merkle::reorgBatch(session())`, `merkle::reorg(*state, proof, batch)` (per-root `get_tree_node`, `update`) | async | y | cj:772-773 → mt:644-670 (`await getTreeNodeFromStorage(session.storage, root.index)` per root, `await batch._update`, mt:378-390) | await | |
286| 191 | HC:1160-1161 | `Core::verifyReorg` | asserts mutex idle | `co_await mutex.lock()` | y | cj:777 | await | H2 |
287| 192 | HC:1164, 1166 | `Core::verifyReorg` | `createWriteBatch`, `state->flush()` | write / async | y | cj:780, cj:782 | await | |
288| 193 | HC:1203 | `Core::checkConflict` | `merkle::getRootsFromStorage(*state, len)` | one rx (maybeGetRootsFromStorage) | y | cj:639 `await maybeGetRootsFromStorage(this.storage, …)` | await | H5 |
289| 194 | HC:1216 | `Core::checkConflict` | `merkle::proof(session(), *state, req, …)` — rx = the state (a fresh CoreRX per node) | `RX rx = state->storage->read(); p = MerkleTree.proof(state, rx, …); rx->tryFlush(); co_await p; co_await settle()` | y | cj:644-659 | await | **Hazard H9**: JS proof generation pushes all reads into ONE rx, then tryFlush, then `settle()`. The C++ merkle proof reads synchronously node by node. |
290| 195 | HC:1231-1232 | `Core::checkConflict` | asserts mutex idle | `co_await mutex.lock()` | y | cj:670 | await | H2 |
291| 196 | HC:1234, 1245, 1247 | `Core::checkConflict` | `createWriteBatch`, `tx.setAuth`, `state->flush()` | write / void / async | y | cj:673, 677-683, 685 | await | |
292| 197 | HC:1261 | `Core::checkConflict` | `onconflict_` (sync) | `co_await this._onconflict(proof)` | y | cj:701 | await | not storage |
293| 198 | HC:1269-1270 | `Core::repairTreeNodes_` | asserts mutex idle | `co_await mutex.lock()` | y | cj:708 | await | H2 |
294| 199 | HC:1290 | `Core::repairTreeNodes_` | `storage_->createAtom()` → `shared_ptr<l2::Atom>` | `AtomPtr HS::createAtom()` | n | cj:726 | sync | |
295| 200 | HC:1291 | `Core::repairTreeNodes_` | `storage_->atomize(atom)` → `unique_ptr` | `HSP HS::atomize(const AtomPtr&)` | n | cj:727 | sync | the atomized HS is never closed in JS either (its db session leaks to GC); C++ must decide (row 4) |
296| 201 | HC:1293-1295 | `Core::repairTreeNodes_` | `atomized->write()`, `putTreeNode` ×n, `tx.flush()` | write / void / `co_await flush()` (answers false: atom view) | y | cj:728-732 | await | |
297| 202 | HC:1299-1312 | `Core::repairTreeNodes_` | `AtomizedReader::get_tree_node` (fresh `s.read()` per node) + `merkle::proof(session(), rx, …)` | ONE `atomized->read()`, proof, tryFlush, settle | y | cj:736-751 | await | H9 |
298| 203 | HC:1322-1324 | `Core::repairTreeNodes_` | `atom->flush(cb)` **assumed to settle synchronously** (`flushError` read on the next line) | `co_await Atom::flush(atom)` | y | cj:761 | await | **Hazard H10**: an existing callback wrapper that only works because l2's Atom flush with no handlers settles inline |
299| 204 | HC:1363 | `Core::reorg` | `state->reorg(batch)` | `P<bool>` | y | cj:840 | await | |
300| 205 | HC:1386 | `Core::close` | `state->close()` | `P<void>` | y | cj:951 (via `destroy`, cj:963) | await | H7 |
301| 206 | HC:1391-1397 | `Core::missingNodes` | `merkle::missingNodes(*this, …)` (Core::get_tree_node per step) | async, `hasTreeNode(storage, idx)` one rx per step (sequential) | y | mt:743-763 (mt:754 `await hasTreeNode(...)` → mt:1287-1292 `rx.hasTreeNode`) | await (sequential) | new `CoreRX::hasTreeNode` exists (`P<bool>`) |
302
303### A3. session.cpp / session.h
304
305| # | file:line | C++ fn | l2 call / use | new hc::storage equivalent | P? | JS counterpart | JS | notes |
306|---|---|---|---|---|---|---|---|---|
307| 207 | SE:179 | `openSession_` | `core->setGroup(…, cb)` | `co_await core->setGroup(group)` | y | ix:392-394 | await | existing wrapper |
308| 208 | SE:214 | `openSessionGroup_` | `ps->snapshot(cb)` | `co_await ps->snapshot()` | y | ix:426 | await | existing wrapper |
309| 209 | SE:271-272 | `openSessionLocked_` | `parentState->createSession(…, cb)` | `co_await` | y | ix:444, ix:449 | await | existing wrapper |
310| 210 | SE:292 | `openCheckout_` | `state_->truncate(…, cb)` | `co_await` | y | ix:469 | await | |
311| 211 | SE:316 | `openRest_` | `state_->storage->write()` | write | n | ix:485 | sync | |
312| 212 | SE:317 | `openRest_` | `tx.putUserData` ×n | void | n | ix:486-488 | sync | |
313| 213 | SE:318 | `openRest_` | `tx.flush()` | async | y | ix:489 | await | currently not awaited by anything after it (sync) — the open's tail must move behind it |
314| 214 | SE:321 | `openRest_` | `core->setManifest(m)` (sync) | `co_await` | y | ix:492-494 | await | |
315| 215 | SE:470 | `readBlock_` | `state_->getBlock(i)` (SS:271) | `P<optional<Buf>>` | y | ix:1386-1390 | await | |
316| 216 | SE:534 | `close` body | `state_->unref()` → may `SessionState::close()` | fire-and-forget (`close().catch(noop)`) | y (ignored) | ix:570 → ss:104-107 | not awaited | |
317| 217 | SE:874 | `has` | `readBlock_(start)` | `co_await` | y | ix:864-869 (rx, getBlock, tryFlush, await) | await | |
318| 218 | SE:875-876 | `has` | `state_->storage->createBlockStream(start, end)` (vector, `.size()`) | `Stream`, count entries | y (stream) | ix:874-880 `for await`, counting | for-await | |
319| 219 | SE:928, 939 | `get_` | `readBlock_(index)` | `co_await` | y | ix:945, ix:965 | await | **Note**: JS re-checks `this.closing` (ix:949), snapshot (ix:954-960), bitfield (ix:964) AFTER awaits — with async reads the "during the above async calls" comment becomes real |
320| 220 | SE:934 | `get_` | `core->get_block(index)` (HC:451) | `co_await` read of `core->state->storage` | y | ix:956 | await | |
321| 221 | SE:1074 | `setupMarks_` | `make_shared<MarkBitfield>(… storage …)` | ctor takes `HSP` | n | ix:1007-1012 | sync | |
322| 222 | SE:893 | `get` | `(*marks_)->set(index, true)` (sync) | `co_await markBlock(index)` | y | ix:895 `if (this._marking) await this.markBlock(index)` | await | |
323| 223 | SE:1080 | `markBlock` | `(*marks_)->set(i, true)` sequentially | `Promise.all` of `set` | y | ix:1020-1025 `setPromises.push(this._marks.set(i, true)); return Promise.all(...)` | Promise.all | **Deviation**: C++ sequential; JS concurrent sets serialized only by MarkBitfield's mutex around the write (mb:203-207) |
324| 224 | SE:1088 | `clearMarkings` | `(*marks_)->clear()` | `co_await` | y | ix:1033 | await | |
325| 225 | SE:1098 | `startMarking` | `state_->storage->atom()` | `->atom` | n | ix:1044 | sync | |
326| 226 | SE:1120 | `sweep` | `(*marks_)->createMarkStream(true)` (vector) | Stream (piped Transform) | y (stream) | ix:1063 `for await (…createMarkStream({reverse:true}))` | for-await | **Deviation H8**: JS pushes `this.clear()` promises while iterating, awaiting `Promise.all` every `batchSize` (ix:1068-1072); C++ collects all ranges, then clears sequentially (SE:1113 comment). |
327| 227 | SE:1136 | `sweep` | `clear(…, cb)` chain | `co_await` / `Promise.all` batches | y | ix:1068, 1076-1077 | Promise.all | |
328| 228 | SE:1147 | `compact` | `core->compact()` (sync) | returns `P` | y | ix:234-236 `return this.core.compact()` | returns promise | |
329| 229 | SE:1165 | `rootsHash` | `merkle::getRootsFromStorage(state, len)` (per-root rx) | one rx | y | ix:1182 / ix:226 `await MerkleTree.getRoots(this.state, length)` → mt:575-591 | await | H5. used by `treeHash` (SE:1185) and `treeHashFromStorage` (SE:1177) — the latter is a sync static in C++, `static async` in JS (ix:225) |
330| 230 | SE:1220 | `missingNodes` | `core->missingNodes(...)` | async | y | ix:1188 `await MerkleTree.missingNodes(this.core.state, …)` | await | |
331| 231 | SE:1224, 1345-1357 | `Session::proof` | `merkle::proof(state_->merkleSession(), *state_, req, …)` (per-node rx) THEN `state_->getBlock(index)` (another rx) | ONE rx: `MerkleTree.proof(state, rx, opts)` + `rx->getBlock(idx)`, tryFlush, `Promise.all`, then `settle()` | y | ix:1191-1201 | Promise.all + await settle | **Deviation H9**: JS reads the block in the SAME batch as the proof nodes; C++ reads it afterwards from a fresh rx. Sync `proof(req)` overload (SE:1345) is used by `fully_remote_proof::proof` (FRP:381) |
332| 232 | SE:1228 | `applyProof` | `core->verify(p)` (sync) | `co_await` | y | ix:1203-1206 | returns promise | |
333| 233 | SE:1235 | `verifyFullyRemote` | `merkle::verify_fully_remote(...)` (no storage read) | — | n | ix:1210 `await MerkleTree.verifyFullyRemote` (sync fn, mt:672-691; the await is one hop) | await | hop parity only |
334| 234 | SE:1373 | `recoverFromRemoteProof` | asserts mutex idle; lock | `co_await core->state->mutex.lock()` | y | ix:1228 | await | H2 |
335| 235 | SE:1377 | `recoverFromRemoteProof` | `fully_remote_proof::verify(*core->db, …)` (sync) | `P<optional<Result>>` | y | ix:1231 | await | |
336| 236 | SE:1381-1383 | `recoverFromRemoteProof` | `core->storage().write()`, `putTreeNode` ×n, `tx.flush()` | write / void / async | y | ix:1234-1238 | await | |
337| 237 | SE:1405 | `setUserData` | `state_->storage->read().getUserData(key)` temporary, no tryFlush | `co_await getUserData(key)` | y | ix:719 `await this.getUserData(key)` → ix:724-730 | await | missing tryFlush (H4-class) |
338| 238 | SE:1407 | `setUserData` | `state_->setUserData(…, done)` | `co_await` | y | ix:721 | await | |
339| 239 | SE:1413 | `getUserData` | `state_->storage->read().getUserData(key)` temporary, no tryFlush | rx, `getUserData`, tryFlush, return promise | y | ix:726-729 | returns promise | missing tryFlush |
340| 240 | SE:1439-1442 | `seek` | `ByteSeeker(state_, …)`, `s->update()` (sync; reads via `state->get_tree_node`, BS:38) | `co_await s->update()` | y | ix:826-828 | await | see row 247 |
341| 241 | SE:1491 | `clear` | `state_->clear(…, cb)` | `co_await` | y | ix:934 | await | |
342| 242 | SE:1338 | `commit` | `state_->commit(src->state_, c, done)` | `co_await` / return | y | ix:591 `return this.state.commit(...)` | returns promise | |
343| 243 | SE:1068 | `recover` | `state_->waitForRecovery(done)` | return promise | y | ix:714 | returns promise | |
344| 244 | SE:1058 | `truncate` | `st->truncate(…, cb)` | `co_await` | y | ix:1109-1128 (`await this.state.truncate`) | await | |
345| 245 | SE:1040 | `append_` | `state_->append(…, done)` | return promise | y | ix:1166 `return this.state.append(...)` | returns promise | |
346
347### A4. marks.cpp / marks.h
348
349| # | file:line | C++ fn | l2 call / use | new hc::storage equivalent | P? | JS counterpart | JS | notes |
350|---|---|---|---|---|---|---|---|---|
351| 246a | MK:12 | `MarkBitfield::loadPage_` | `storage_->read()` | `RX` | n | mb:174 | sync | |
352| 246b | MK:13 | `loadPage_` | `rx.getMark(page)` | `P<optional<Buf>> CoreRX::getMark(int64_t)` | y | mb:175, awaited mb:177 | await | |
353| 246c | MK:14 | `loadPage_` | `rx.tryFlush()` | same | n | mb:176 | sync | mechanical. **But**: JS inserts the (unloaded) `MarkPage` into `_pages` BEFORE the read (mb:173) and `set` awaits `p.loaded` (mb:201); C++ inserts after. Concurrency-visible once async (a concurrent `get` on a loading page reads `null.bitfield` in JS, mb:186-189) |
354| 246d | MK:37-39 | `MarkBitfield::set` | `storage_->write()`, `putMark`, `flush()` | write / void / `co_await flush()` | y | mb:203-207 (`await this.mutex.lock()` around the write) | await | **Deviation**: C++ has no MarkBitfield mutex ("keetd's writes are synchronous", MK:37); JS's `Mutex` must come back |
355| 246e | MK:43-45 | `MarkBitfield::clear` | write, `deleteMarkRange(0,-1)`, `flush()` | same | y | mb:211-216 | await | |
356| 246f | MK:51 | `createMarkStream` | `storage_->createMarkStream(0, -1, reverse)` (vector) | `Stream HS::createMarkStream(IndexRange{0, -1, reverse})` + Transform to block indices | y (stream) | mb:218-241 (`.pipe(new Transform(...))`) | stream | C++ returns a collected `vector<uint64_t>`. JS reverse starts `quickbit.findLast(page, true, BITS_PER_PAGE)` |
357| 246g | MKh:23, 32 | `MarkBitfield` | ctor / member `std::shared_ptr<l2::HypercoreStorage>` | `HSP` | n | mb:166-170 | — | type |
358
359### A5. fully_remote_proof.cpp / .h
360
361| # | file:line | C++ fn | l2 call / use | new hc::storage equivalent | P? | JS counterpart | JS | notes |
362|---|---|---|---|---|---|---|---|---|
363| 247 | BS:38 | `ByteSeeker::seek_` | `state->get_tree_node(ite.left_child())` (SS:265) | `co_await` per step (sequential, one rx each) | y | mt:457 `await getTreeNodeFromStorage(this.session.storage, ite.leftChild())` | await (sequential) | 1:1 once async; `ByteSeeker::update` → `P<…>` (mt:478-489). The replicator also calls `update()` (out of scope) |
364| 248 | FRP:18-23 | `StorageReader::get_tree_node` | `s.read().getTreeNode(i)` temporary, no tryFlush | async (H1) | y | (SlimSession storage = the resumed core; read only by `verifyFullyRemote`, which never touches storage, mt:672-691) | — | effectively dead: `merkle::verify_fully_remote` reads no stored node |
365| 249 | FRP:34 / FRPh:30 | `verify` | param `l2::CorestoreStorage &` | `CS &` (or `CorestoreStoragePtr`) | n | frp:27 | — | type |
366| 250 | FRP:51 | `verify` | `storage.resumeCore(dk)` → `unique_ptr` | `P<HSP> CS::resumeCore(optional<K32>)` | y | frp:42 | await | JS never closes this storage (leak to GC) — the C++ `unique_ptr` dtor did; decide with row 4 |
367| 251 | FRP:58-61 | `verify` | `core->read()`, `getAuth`, `getHead`, `getUserData("referrer")` — **no tryFlush** (scope dtor) | RX, 3×`P`, tryFlush, `co_await all` | y | frp:45-52 | Promise.all | missing tryFlush |
368| 252 | FRP:70-76 | `verify` | `core->read()`, `getTreeNode` per root, null → `roots.clear(); break` — no tryFlush | RX, all `P`, tryFlush, `Promise.all`, `roots.some(isNull) → []` | y | frp:58-69 | Promise.all | early break + missing tryFlush |
369| 253 | FRP:345 | `verify` | `merkle::verify_fully_remote(slim, …)` | — | n | frp:75 `await MerkleTree.verifyFullyRemote(...)` (sync fn) | await | hop only |
370| 254 | FRP:381 | `proof` | `sender.proof(req)` (sync SE:1345) | `co_await sender.proof(req)` | y | frp:106-109 | await | |
371
372---
373
374## B. Type-only uses (members, parameters, typedefs) — no call
375
376| file:line | declaration | l2 type | new type |
377|---|---|---|---|
378| SSh:34, SS:15, HC:24, FRP:14 | `namespace hs = l2::hs` | alias | `storage::hs` (same namespace) |
379| SSh:47, SS:97 | `SessionState(…, std::shared_ptr<l2::HypercoreStorage> storage, …)` | HS | `HSP` |
380| SSh:54 | `std::shared_ptr<l2::HypercoreStorage> storage` | HS | `HSP` |
381| SSh:60 | `std::shared_ptr<l2::Atom> atomized` | Atom | `AtomPtr` |
382| SSh:99, SS:280 | `l2::CoreTX &createWriteBatch()` | CoreTX | `CoreTX &` (storage) |
383| SSh:104, SS:273 | `updateGroupIndex(l2::CoreTX &, …)` | CoreTX | `CoreTX &` |
384| SSh:108, SS:1362 | `commitBatch(l2::CoreTX &, …)` | CoreTX | `CoreTX &` |
385| SSh:140 | `std::vector<std::shared_ptr<l2::HypercoreStorage>> lingers` | HS | `std::vector<HSP>` |
386| SSh:166 | `createSession(…, std::shared_ptr<l2::Atom> atom, …)` | Atom | `AtomPtr` |
387| SSh:185 | `TruncFlush::dependency : std::optional<l2::Dependency>` | Dependency | `storage::Dependency` (alias) |
388| SSh:191-192 | `updateSnapshotStorage_(const std::shared_ptr<l2::HypercoreStorage>&, …)`, `readSnapshotTree_(l2::HypercoreStorage&)` | HS | `HSP` |
389| SSh:202 | `Truncated::dependency : std::optional<l2::Dependency>` | Dependency | alias |
390| SSh:203 | `truncate_(l2::CoreTX &, …)` | CoreTX | `CoreTX &` |
391| SSh:205 | `std::optional<l2::Dependency> updateDependency_(…)` | Dependency | alias |
392| SSh:206 | `std::optional<l2::CoreTX> activeTx_` | CoreTX (by value) | `std::shared_ptr<CoreTX>` |
393| SSh:209 | `storeBitfieldRange_(l2::CoreTX &, …)` | CoreTX | `CoreTX &` |
394| SS:736-737 | `StorageNodes { l2::HypercoreStorage &s; }` | HS | `HS &` / `HSP` |
395| SS:744 | `rootsFromStorage(l2::HypercoreStorage &, …)` | HS | `HS &` |
396| SS:767, 1062 | locals `std::shared_ptr<l2::HypercoreStorage> st` | HS | `HSP` |
397| SS:1053 | local `resumed` | HS | `HSP` |
398| SS:1109-1110 | `std::vector<std::optional<hs::CoreTreeNode>>`, `std::vector<std::optional<Buf>>` | hs | → `std::vector<P<…>>` before the await |
399| SS:1308 | `std::vector<std::optional<hs::CoreTreeNode>> nodes` | hs | → `std::vector<P<…>>` |
400| SS:810 | `l2::Atom::Done` | Atom::Done | gone (`Handler` returns `P<void>`) |
401| HCh:16 | `#include "hc/l2.h"` | — | `hc/corestore_storage.h` |
402| HCh:54 | `OpenOptions::alias : std::optional<l2::hs::CorestoreAlias>` | hs | alias |
403| HCh:121, HC:266 | `Core(l2::CorestoreStorage &db, …)` | CS | `CS &` / `CorestoreStoragePtr` (and async open, H6) |
404| HCh:156 | `l2::CorestoreStorage *db` | CS | `CS *` |
405| HCh:217 | `l2::HypercoreStorage &storage()` | HS | `HS &` |
406| HCh:262, HC:477 | `setManifest_(l2::CoreTX &, …)` | CoreTX | `CoreTX &` |
407| HCh:294 | `std::shared_ptr<l2::HypercoreStorage> storage_` | HS | `HSP` |
408| HCh:334-335 | `to_stored` / `from_stored` with `l2::hs::CoreManifest` | hs | alias |
409| HC:334 | `l2::CreateCoreArgs c` | CreateCoreArgs | `storage::CreateCoreArgs` (same fields; `userData` empty = absent) |
410| HC:598, 606 | `std::optional<l2::hs::CoreTreeNode>`, `std::optional<l2::hs::CoreHead>` | hs | alias (+ `P<…>` vectors) |
411| HC:813, 819, 846, 853 | `isBadBlock` / `isBadTree` / audit stack `l2::hs::CoreTreeNode` | hs | alias |
412| HC:839, 886, 913 | `std::optional<l2::CoreTX> tx` | CoreTX | `std::shared_ptr<CoreTX>` |
413| HC:1290-1291 | `std::shared_ptr<l2::Atom>`, `std::unique_ptr<l2::HypercoreStorage>` | Atom, HS | `AtomPtr`, `HSP` |
414| HC:1299-1301 | `AtomizedReader { l2::HypercoreStorage &s; }` | HS | `HS &` |
415| SEh:99 | `SessionOptions::atom : std::shared_ptr<l2::Atom>` | Atom | `AtomPtr` |
416| SE:1382 | `l2::hs::CoreTreeNode` | hs | alias |
417| MKh:16 | `using l2::Buf` | Buf | `storage::Buf` (= `rn::Buf`) |
418| MKh:23, 32 | `std::shared_ptr<l2::HypercoreStorage>` | HS | `HSP` |
419| FRPh:30, FRP:34 | `l2::CorestoreStorage &storage` | CS | `CS &` |
420| FRP:18-19 | `StorageReader { l2::HypercoreStorage &s; }` | HS | `HS &` |
421| FRP:51 | `std::unique_ptr<l2::HypercoreStorage> core` | HS | `HSP` |
422
423Not used anywhere in core/: `l2::Db`, `l2::DbSnapshot`, `l2::View`, `l2::Change`, `l2::TreeCache` /
424`TreeCacheStats` / `TreeCacheEntry`, `l2::keys::*`, `l2::DeviceFile`, `l2::StoreHead`, `l2::Info`, `l2::ExportData` /
425`CoreExport`, `l2::blockStream`, `l2::encode_core_auth` / `encode_core_head`, `CorestoreStorage::{getSeed, setSeed,
426getInfo(s), getAuth, getAlias, hasCore, deleteCore, exportCore, audit, clear, flush, setRecovering, getGroup, streams}`,
427`HS::{deleteSessions, exportCore, addDependency_, getDependencyLength, getDependency, updateDependencyLength,
428createTreeNodeStream, createLocalStream}`, `CoreRX::{getSessions, getHints(static), getLocal}`.
429
430---
431
432## C. Callback-style async wrappers that already exist (direct Promise conversions)
433
434| where | wrapper | JS it stands for | notes |
435|---|---|---|---|
436| SS:185-193 `snapshot(done)` | `done(Fail, state)` | `async snapshot()` ss:210-215 | |
437| SS:195-222 `updateSnapshotStorage_(st, done)` | self-referencing weak lambda over `Atom::flushed(fn)` | `while (this.atomized.flushing) await this.atomized.flushed()` ss:130 | → `co_await atom->flushed()` loop |
438| SS:306-320 `locked_(body, fail)` | sync body returning a resolver | `await mutex.lock(); try {…} finally { _unlock() }` | must become a coroutine scope (H3) |
439| SS:322-330 `setUserData`, SS:375-428 `clear`, SS:430-521 `append`, SS:523-577 `truncate`, SS:765-819 `createSession`, SS:849-891 `oncommit_`, SS:1035-1103 `moveTo`, SS:1288-1354 `catchup` | `std::function<void (Fail …)> done` via `locked_` | the async methods | |
440| SS:829-847 `commit_(done)` | `mutex.lock(cb)` + `parent->oncommit_(…, cb)` | `async _commit()` ss:280-291 | |
441| SS:810-814 `atom->onflush([](l2::Atom::Done d){…})` | M5g-3c Done handler | `atom.onflush(state._commit.bind(state))` ss:1267 | new `Atom::Handler` returns `P<void>` directly |
442| SS:923-991 `waitForRecovery(done)` | rt timers + `mutex.lock(cb)` | ss:615-659 | timers stay; tail becomes `co_await` |
443| SS:1203-1286 `commit` / `commitLocked_` | nested `mutex.lock` callbacks | ss:1041-1105 | |
444| HC:766-786 `copyPrologue(done)` | nested `mutex.lock` callbacks | cj:443-460 | |
445| HC:828-959 `audit(opts, done)` | `mutex.lock(cb)` | cj:368-376 | |
446| HC:961-973 `setGroup(group, done)` | `mutex.lock(cb)` | cj:378-394 | |
447| HC:1322-1324 `atom->flush(cb)` in `repairTreeNodes_` | **assumes inline settle** | `await atom.flush()` cj:761 | H10 |
448| SE: every public method (`ready`, `get`, `has`, `append`, `truncate`, `commit`, `setUserData`, `getUserData`, `seek`, `clear`, `markBlock`, `clearMarkings`, `startMarking`, `sweep`, `treeHash`, `signable`, `missingNodes`, `proof`, `applyProof`, `verifyFullyRemote`, `info`, `setGroup`, `recover`) via `whenOpened_(body, fail)` (SE:446-457) | `done(Fail, T)` | `if (this.opened === false) await this.opening` + async body | the open itself is a callback chain (SE:175-335) |
449| SE:631-730 `Download` | `whenOpened_` + `Gen` waiters | lib/download.js | no storage |
450
451---
452
453## D. Groups (rows clustered by JS function)
454
455- **SessionState#close / unref / lingers** (ss:104-107, 188-208): 4, 114, 216, 142, 205.
456- **SessionState#snapshot / _updateSnapshotStorage** (ss:128-176, 210-215): 5-18, 208.
457- **SessionState#createWriteBatch / flush / _unlock** (ss:232-271): 21-23, and every "flush()" row: 28, 37, 46, 53, 60, 66b, 70, 106, 113, 120p, 121j, 122j, 147, 187, 192, 196.
458- **storeBitfieldRange** (ss:1299-1326): 82-84; callers 32, 42, 56, 79.
459- **SessionState#setUserData** (ss:347-358): 26-28; **Hypercore#setUserData / getUserData** (ix:717-730): 237-239.
460- **SessionState#_verifyBlock** (ss:360-425): 29-39; **Core#verify / _verifyExclusive / _verifyShared** (cj:514-603, 792-832): 178-189.
461- **SessionState#truncate / _truncate / reorg** (ss:427-556): 63-81; **Core#verifyReorg / reorg** (cj:771-790, 834-846): 190-192, 204.
462- **SessionState#clear** (ss:558-612): 40-48; **Hypercore#clear** (ix:914-937): 241.
463- **SessionState#append** (ss:661-754): 49-62; **Hypercore#append** (ix:1131-1167): 245.
464- **SessionState#_commit / _oncommit / _precommit** (ss:273-345): 95-108.
465- **SessionState#createSession / _getTreeHeadAt / getCoreHead** (ss:1107-1123, 1216-1274, 1363-1368): 85-98; **Hypercore#_openSession** (ix:389-499): 207-214.
466- **SessionState#moveTo / truncateAndFlush** (ss:1142-1214, 1328-1342): 110-120a.
467- **SessionState#commit / _overwrite / Core#_validateCommit** (ss:912-1105, cj:466-485): 120b-121k, 175; **Hypercore#commit** (ix:587-592): 242.
468- **SessionState#catchup** (ss:831-910): 122a-122l.
469- **SessionState#waitForRecovery** (ss:615-659): 109, 49, 121b, 243.
470- **MerkleTreeBatch#commit** (mt:232-270): 123, 34, 54, 76, 185.
471- **MerkleTree storage readers** (mt:579-591, 615-641, 644-670, 693-712, 743-763, 1265-1292; MerkleTreeBatch#downgrade/restore mt:301-330): 19, 85-87, 93, 119, 29, 63, 110, 121e, 143, 162, 175, 177-179, 190, 193, 206, 229, 230.
472- **MerkleTree.proof / settle** (mt:714-716, 1294-1360): 194, 202, 231, 254.
473- **ByteSeeker** (mt:429-490): 240, 247.
474- **Core#_tryOpen / getCoreInfo / maybeGetRootsFromStorage / Bitfield.open** (cj:165-366, 1033-1066; bf:536-548): 124-141.
475- **Core#setGroup** (cj:307-318, 378-394): 135, 136, 174, 207.
476- **Core#setManifest / _setManifest** (cj:396-441): 145-148, 214.
477- **Core#flushHints** (cj:605-618): 176 + callers 38, 48, 62, 66d, 72, 108, 109, 122l, 188.
478- **Core#checkConflict / _repairTreeNodes** (cj:620-769): 193-203.
479- **Core#copyPrologue + copy-prologue.js** (cj:443-460, cp:14-188): 149-160.
480- **Core#compact** (cj:124-130, ix:234-236): 161, 228.
481- **Core#audit + audit.js** (cj:368-376, au:6-138): 162-173.
482- **Core#destroy / _close** (cj:942-967): 142, 205.
483- **Hypercore#_get / get / has** (ix:851-912, 944-997): 215, 217-220, 222.
484- **marks: Hypercore#markBlock / clearMarkings / startMarking / sweep + mark-bitfield.js** (ix:1007-1081; mb:162-241): 221-227, 246a-246g.
485- **Hypercore#treeHash / treeHashFromStorage / missingNodes / proof / applyProof / verifyFullyRemote** (ix:225-228, 1169-1213): 229-233.
486- **Hypercore#recoverFromRemoteProof + fully-remote-proof.js** (ix:1224-1248; frp:27-126): 234-236, 248-254.
487
488---
489
490## E. Hazards (cannot be a mechanical swap)
491
492**H1. merkle::Storage is synchronous.** `merkle::Storage::get_tree_node` (l3/include/hc/merkle.h:36) is a sync
493virtual. core/ implements it five times — `SessionState` (SS:265), `Core` (HC:445), `StorageNodes` (SS:735),
494`AtomizedReader` (HC:1299), `StorageReader` (FRP:18) — and every merkle function that reads storage goes through it:
495`truncate`, `restore`, `downgrade`, `reorg`/`update`, `verify`, `getRootsFromStorage`, `missingNodes`, `get`, `proof`,
496`byteOffset`/`byteRange`/`batchSeek`, `upgradeable`, plus `ByteSeeker::seek_` (BS:38). With CoreRX getters resolving
497only at tryFlush, none of these can return a value synchronously. R4c-4 needs the merkle layer (l3, outside this
498inventory) ported to `P<…>` too, in JS's shape: some JS readers take an **rx** and batch (`getRootsFromStorage`,
499`downgrade`, `upgradeable`, `proof`, `byteRange`/`byteOffset`), others take a **storage** and open one rx per node,
500awaited in sequence (`truncate`, `reorg`, `verify`, `get`, `missingNodes`, `ByteSeeker._seek`, `MerkleTreeBatch.get`
501mt:166). The single `get_tree_node(index)` interface erases that difference.
502
503**H2. The verify family is synchronous and refuses to wait on the mutex.** `Core::verify` (HC:1107),
504`SessionState::verifyBlock` (SS:333), `SessionState::reorg` (SS:580), `Core::setManifest` (HC:464),
505`Core::verifyReorg` (HC:1160), `Core::checkConflict` (HC:1231), `Core::repairTreeNodes_` (HC:1269) and
506`Session::recoverFromRemoteProof` (SE:1372) throw "keetd: … while the state mutex is held" where JS does
507`await this.state.mutex.lock()`. That only works because nothing suspends today. Once flush or reads await, a second
508proof (or an append) can arrive mid-verify. So these need a real Promise-based `Mutex::lock()`, and Core needs JS's
509`_verifies` / `_verified` batching of shared verifies (cj:821-831, 535-603). C++ currently handles one op at a time
510(HC:1104-1106). Their callers (the replicator, outside core/) must await the results.
511
512**H3. `locked_` assumes a synchronous body.** `SessionState::locked_` (SS:306-320) runs `body()` to completion, then
513`_unlock`, then the resolver. Every body above gains `co_await`s (flush, storeBitfieldRange, `_truncate`, flushHints,
514reads), so it must become `co_await mutex.lock(); try { … } finally { _unlock(); }`. JS's ordering ("unlock in finally,
515then the op's promise settles; a queued waiter resumes first") has to be re-proved with js::Promise hops. The
516`Mutex` (SS:53-93) hands the lock over inside `unlock()` (`next.granted()` runs synchronously). JS's mutex.js resumes
517the waiter in a microtask, which the header comment already acknowledges ("callers unlock LAST").
518
519**H4. Reads with a missing tryFlush, or values used before tryFlush.** These rely on l2 getters returning at once,
520or on `~CoreRX`. In the new layer each one either hangs (no tryFlush) or must be restructured into "issue, tryFlush,
521await":
522- `storeBitfieldRange_` (SS:722-728): no tryFlush; each page is filled and put as it is read.
523- `readSnapshotTree_` (SS:239-243): throws on a null root before `rx2.tryFlush()`.
524- `Core::Core` getCoreInfo (HC:283-287): no tryFlush. Root loop (HC:377-383): no tryFlush, early `break`.
525- `createSession` getCoreHead (SS:773), `overwrite_` bitfield page (SS:1156): no tryFlush.
526- All five `get_tree_node` impls, plus `SessionState::getBlock` (SS:271) and `Core::get_block` (HC:451): use
527 temporaries, no tryFlush.
528- `Session::setUserData` / `getUserData` (SE:1405, 1413): no tryFlush.
529- `fully_remote_proof::verify` (FRP:58-61, 70-76): no tryFlush, early `break`.
530
531Already JS-shaped, so a mechanical swap: `oncommit_` (SS:854-856), `overwrite_` main batch (SS:1108-1119), `catchup`
532(SS:1307-1310), `flushPrologueBatch` (HC:600-618), `audit` (HC:860-863, 895-897, 923-925), `loadPage_` (MK:12-14).
533
534**H5. Batch shape and hop count.** Where JS reads all of a set through ONE rx and `Promise.all`, C++ goes through
535`get_tree_node` with one rx per node: `getRootsFromStorage` (rows 86, 87, 93, 119, 162, 193, 229, 121e/restore),
536`downgrade` (29, 177), the proofs (194, 202, 231). The values are the same, but the number of ReadBatches and
537microtask hops differs. That matters to R4c's "every await costs what it costs in JS" goal (tools/l2/hops.js). In
538`getRootsFromStorage`, JS's `Promise.all` over `getTreeNodeOrError` rejects with the first rejection in settle order
539(all settle in one flush, in issue order). That gives the same error C++'s first-missing check gives.
540
541**H6. Core opens in a constructor.** `Core::Core` (HC:266-431) is `_tryOpen` and has 9 awaits in JS
542(resumeCore, getCoreInfo, createCore, Bitfield.open, maybeGetRootsFromStorage, createGroup, three `tx.flush()`s).
543It needs JS's `this.opening = this._open(opts)` (cj:67, 148-163). That ripples to every constructor site
544(store/src/corestore.cpp:420, repl/test/r3c_walk_test.cpp:91, repl/test/r3c_scan_test.cpp:73). It also ripples to
545Session's open, which assumes "the Core is open (corestore opened it synchronously)" (SE:195) and has no
546`await this.core.ready()` (ix:390).
547
548**H7. Storage lifetime: `close()` is async and nothing calls it.** l2's `HypercoreStorage` closed itself in its
549destructor (l2.h:384). The new `HypercoreStorage` has no destructor and `P<void> close()` (`view.readStop()` +
550`db.close()`, hs:439-446).
551- `SessionState::close` (SS:166-183) must call `storage->close()` and await each linger. JS returns the storage's
552 close promise (ss:195, 203-207).
553- `~SessionState`, `~Core` (HC:433-437) and `Core::close` (HC:1380-1388) cannot await.
554- JS itself never closes some storages: the atomized storage in `_repairTreeNodes` (cj:727), the resumed core in
555 frp `verify` (frp:42), and snapshot storages whose state is never unref'd. keetd must pick an explicit policy
556 (fire-and-forget close, as `unref` does with `.catch(noop)`), or rn db sessions leak.
557
558**H8. Streams drained before use vs `for await` with awaits in the body.**
559- `copyPrologueInto` (HC:734): JS awaits `flushBatch` inside the block-stream loop (cp:52), writing dst while src is
560 still streaming. C++ collects the whole stream first.
561- `audit` blocks (HC:887): JS reads a tree node per block inside the loop (au:87-92).
562- `sweep` (SE:1120-1141): JS starts `clear()`s during iteration and awaits `Promise.all` every `batchSize`
563 (ix:1063-1077). C++ collects the ranges, then clears them one after another; `batchSize` is ignored (SE:1113).
564- `Bitfield.open` (HC:367), `has` (SE:875), `createUserDataStream` (HC:633), `createMarkStream` (MK:51): plain
565 drains, so the order is preserved, but they become streamx iteration (hops).
566- `MarkBitfield::createMarkStream` returns a `vector<uint64_t>`. JS pipes a streamx `Transform` (mb:218-241).
567
568**H9. Proof generation needs JS's rx + settle protocol.** `MerkleTree.proof(session, rx, req)` pushes every node
569read into the caller's rx, the caller calls `tryFlush()`, then awaits the proof and `settle()`s it
570(ix:1193-1200, cj:644-659, cj:736-751). `Hypercore#proof` puts the block read (`rx.getBlock`) in the SAME batch
571(ix:1195). C++ `Session::proof` reads it afterwards from a new rx (SE:1353). C++ `merkle::proof` is synchronous over
572`Storage&`.
573
574JS quirk to check against the spec before porting: when `seek` is set, `generateProof` awaits
575`seekUntrustedTree` / `seekFromHead` (mt:1348-1355) BEFORE it pushes the rx reads. By then the caller has already
576called `rx.tryFlush()`, which nulls `rx.view` (tx.js:331-337), so the later `rx.getTreeNode` → `this.view.get` would
577throw. Verify this against JS behaviour before deciding what the port does.
578
579**H10. `repairTreeNodes_` reads an Atom flush result synchronously** (HC:1322-1324): it reads `flushError` on the
580line after `atom->flush(cb)`. That only holds because l2's handler-less flush settles inline. The new
581`static P<void> Atom::flush(AtomPtr)` never does (JS: `await View.flush(...)` + `Promise.all`), so this must become
582`co_await Atom::flush(atom)`.
583
584**H11. Ordering deviations already in the sync C++ (not storage calls, but visible once things suspend):**
585- `markBlock` (SE:1080) sets marks one after another. JS runs `Promise.all` over `_marks.set`, which share
586 MarkBitfield's mutex (mb:203-207). C++ dropped that mutex (MK:37).
587- `MarkBitfield::loadPage_` inserts the page after the read. JS inserts the not-yet-loaded `MarkPage` first (mb:173).
588- `append`: `preappend` and `postappend` are synchronous in C++ but awaited in JS (ss:678, 744; ix:1359-1366 awaits
589 each `encryption.encrypt`). These are hop-parity points, not correctness ones.
590- `_get` (SE:927-940): JS's "during the above async calls" re-checks (closing, snapshot, bitfield, ix:949-968)
591 become live races.
592- `Session::compact` (SE:1145-1153) calls `done` at once. JS returns `core.compact()`'s `Promise.all` (ix:235).
593- `SE:316-318`: the open's userData flush is not awaited before `opened = true` / 'ready' (JS awaits it, ix:489).
594- `treeHashFromStorage` (SE:1177) is a sync static. JS's is `static async` (ix:225).
595
596**H12. CoreTX ownership.** `activeTx_` is `std::optional<l2::CoreTX>`, a move-only value whose `changes_` points at
597its own vector (l2.h:267-272). The new `CoreTX` is held by `shared_ptr` with `ChangesPtr changes` (null once
598flushed). `createWriteBatch` returns a reference into `activeTx_`. `flush()` moves it out (SS:299-300), so in the new
599layer it should keep the `TX` alive across the `co_await tx->flush()`.
docs/research/r4c4-inv-store-repl.md +211 -0
……@@ -0,0 +1,211 @@
1# R4c-4 call-site inventory — store/, repl/, l3/ (+ mux/bee/blobs/wakeup sweep)
2
3Worktree `/storage/projects/code/hypercore-kb/hyper-fleeky/.scope/keetd` (branch r4). Read in full:
4`l2/include/hc/l2.h`, `storage/include/hc/storage.h`, `storage/include/hc/corestore_storage.h`,
5`store/include/hc/corestore.h`, `store/src/corestore.cpp`, `repl/include/hc/replicator.h`,
6`repl/src/replicator.cpp`, `l3/include/hc/merkle.h`, `l3/src/merkle.cpp`.
7Swept with grep, then the matching regions read: `repl/src/{wants,bitfield}.cpp`, `mux/`, `bee/`, `blobs/`, `wakeup/`, `l3/src/{wire,verifier}.cpp`.
8None of these touches an l2 object directly. bee and blobs do have sync-completion assumptions, listed under Hazards H14.
9
10JS reference root: `hyper-fleeky/packages/keet-daemon/node_modules/`. The abbreviations used below are
11`cs` = corestore 7.12.6, `hc` = hypercore 11.37.0, `hs` = hypercore-storage 3.3.0,
12`rep` = hc/lib/replicator.js, `mt` = hc/lib/merkle-tree.js, `core.js` = hc/lib/core.js.
13I read every JS line number cited here.
14
15New-API line refs: `S:` = storage/include/hc/storage.h, `CS:` = storage/include/hc/corestore_storage.h.
16
17## Counts
18
19| file | type-only uses | direct l2 calls | indirect storage reads/writes (via Core / merkle::Storage) |
20|---|---|---|---|
21| store/include/hc/corestore.h | 1 | 0 | 0 |
22| store/src/corestore.cpp | 6 | 11 | 3 (Core ctor = open, Core::close x2) |
23| repl/src/replicator.cpp | 3 | 3 | 14 |
24| repl/include/hc/replicator.h | 0 | 0 | 0 (only `hcore::Core`) |
25| l3/include/hc/merkle.h | 2 | 0 | (interface: see L-rows) |
26| l3/src/merkle.cpp | 1 | 0 | 28 rows (every `Storage::get_tree_node` reach) |
27| mux/ bee/ blobs/ wakeup/ repl/{wants,bitfield} | 0 | 0 | 0 direct; bee/blobs have 7 sync-completion sites (H14) |
28
29The total is 13 type-only uses, 14 direct l2 calls and 45 indirect sites.
30
31---
32
33## A. store/ — corestore 7.12.6 over l2::CorestoreStorage
34
35### A1. Type-only uses
36
37| # | file:line | C++ enclosing | l2 type use | new equivalent | notes |
38|---|---|---|---|---|---|
39| T1 | store/include/hc/corestore.h:129 | `Corestore::getAuth` decl | `std::optional<l2::hs::CoreAuth>` return | `Promise<std::optional<hs::CoreAuth>>` (CS:185); `storage::hs` aliases `l2::hs` (S:34) so the schema type is unchanged | the return type becomes a promise. JS cs index.js:384-386 returns the promise un-awaited |
40| T2 | store/src/corestore.cpp:105 | `struct Tracked` | `std::shared_ptr<l2::CorestoreStorage> storage` | `CorestoreStoragePtr` (CS:31, `shared_ptr<CorestoreStorage>`) | the comment says "the Core's db outlives it". With async close the continuation must also keep a ref (H9) |
41| T3 | store/src/corestore.cpp:116 | `Corestore::Shared` | `std::shared_ptr<l2::CorestoreStorage> storage` | `CorestoreStoragePtr` | must be created with `make_shared`, because `enable_shared_from_this` is used (CS:122) |
42| T4 | store/src/corestore.cpp:242 | `Corestore::Corestore(path, opts)` | `l2::CorestoreStorage::Options so` | `CorestoreStorage::Options` (CS:124): adds alwaysRecover, onresume, treeCacheMaxSize, CF knobs | JS hc index.js:208-212 `defaultStorage` gets `{id, allowBackup, readOnly, wait, treeCache}` from cs index.js:243-249. keetd sets only readOnly, as it does today |
43| T5 | store/src/corestore.cpp:357 | `Corestore::get` | `l2::hs::CorestoreAlias{name, ns}` | `hs::CorestoreAlias` (same type via alias) | type construction only |
44| T6 | store/src/corestore.cpp:408 | `Corestore::openCore_` | `t->storage = shared_->storage` (shared ownership copy) | same, with `CorestoreStoragePtr` | keetd-internal lifetime pin; there is no JS counterpart (the GC keeps it alive) |
45| T7 | store/src/corestore.cpp:418 | `Corestore::openCore_` | `l2::hs::CorestoreAlias{...}` into `OpenOptions.alias` | `hs::CorestoreAlias` | `hcore::OpenOptions` (core/) carries `l2::hs` types. It is a core/ inventory item |
46
47### A2. Calls
48
49| # | file:line | C++ enclosing function | l2 call | new hc::storage equivalent (exact) | Promise? | JS counterpart (verified) | JS awaits it? | notes |
50|---|---|---|---|---|---|---|---|---|
51| C1 | corestore.cpp:244 | `Corestore::Corestore(path, opts)` | `std::make_shared<l2::CorestoreStorage>(path, so)` | `CorestoreStorage (std::string storage, Options opts)` (CS:135) via `std::make_shared` | no (ctor) | cs index.js:241-249 → hc index.js:208-212 `new CoreStorage(directory, opts)` | sync | Both constructors are sync. The new ctor runs `construct_` (bootstrap, rocks opened lazily?). I have not verified whether rn opens in the ctor or in `ready()` |
52| C2 | corestore.cpp:282 | `Corestore::ready` (JS `_open`) | `shared_->storage->ready()` | `Promise<void> ready ()` (CS:155) = hs index.js:550-553 (`_migrateStore` if version 0, then `db.ready()`) | **yes** | **No JS counterpart.** cs `_open` (index.js:398-416) never calls `storage.ready()`. Migration happens lazily inside `getSeed` (hs index.js:857 `if (this.version === 0) await this._migrateStore()`) | n/a | **keetd-internal deviation.** JS's first storage op is `getSeed`. Dropping this call and letting getSeed migrate is literal. Keeping it adds `db.ready()` plus a migrate before getSeed. That is observably the same, but the hop count differs |
53| C3 | corestore.cpp:283 | `Corestore::ready` (`_getOrSetSeed`) | `getSeed()` → `optional<K32>` | `Promise<std::optional<K32>> getSeed ()` (CS:177) | **yes** | cs index.js:393 `const seed = await this.storage.getSeed()`; hs index.js:856-866 | await | |
54| C4 | corestore.cpp:288 | `Corestore::ready` (`_getOrSetSeed`) | `setSeed(fresh)` | `Promise<std::optional<K32>> setSeed (std::optional<K32> seed, bool overwrite = true)` (CS:179) | **yes** | cs index.js:395 `return await this.storage.setSeed(this.primaryKey \|\| crypto.randomBytes(32))`; hs index.js:877-899 | await | `Corestore::ready()` is `void` and sync today. JS `_open` is async (ReadyResource). Every caller of `ready()` must await it (H1) |
55| C5 | corestore.cpp:310 | `Corestore::close` (root path, JS `_close`) | `shared_->storage->close()` | `Promise<void> close ()` (CS:170) = hs index.js:808-815 | **yes** | cs index.js:446 `await this.storage.close()` | await | JS first awaits `Promise.all(closing)` (sessions + child stores, :434/:443), then `await this.cores.close()` (:445), then closes storage. C++ does all of that synchronously. `~Corestore` (corestore.cpp:267-273) calls `close()` from a destructor, which can't await (H9) |
56| C6 | corestore.cpp:330 | `Corestore::list` | `createDiscoveryKeyStream(n)` → `vector<Buf>` (collected) | `Stream createDiscoveryKeyStream (const std::optional<K32> &ns = std::nullopt)` (CS:174) | no (a streamx Readable) | cs index.js:380-382 `return this.storage.createDiscoveryKeyStream(namespace)`; hs index.js:839-841 | sync (returns stream) | **The return type changes** from `vector<Buf>` to `Stream`. Callers of `Corestore::list` must consume a stream (H13) |
57| C7 | corestore.cpp:332 | `Corestore::getAuth` | `getAuth(dk)` | `Promise<std::optional<hs::CoreAuth>> getAuth (K32 discoveryKey)` (CS:185) | **yes** | cs index.js:384-386 `return this.storage.getAuth(discoveryKey)`; hs index.js:984-1001 | returns the promise | The public signature changes (T1) |
58| C8 | corestore.cpp:357 | `Corestore::get` (JS `_preload`) | `getAlias({name, ns})` | `Promise<std::optional<Buf>> getAlias (hs::CorestoreAlias alias)` (CS:176) | **yes** | cs index.js:633-635 `await this.storage.getAlias({ name: opts.name, namespace: this.ns })`, then `_maybeClosed()` :636; hs index.js:847-854 | await | JS `get()` returns the session **synchronously** with `conf.preload = this._preload(opts)` (cs:590-592). The await is inside the preload promise. C++ `get` returns an opened session or throws (H2). JS `_preload` also awaits `this.ready()` first (:631) |
59| C9 | corestore.cpp:363 | `Corestore::get` (JS `_preloadCheckIfExists`) | `hasCore(dk)` | `Promise<bool> hasCore (K32 discoveryKey, bool ifMigrated = false)` (CS:184) | **yes** | cs index.js:623-627 `const has = await this.storage.hasCore(opts.discoveryKey)`, then throws STORAGE_EMPTY, then `return this._preload(opts)`; hs index.js:968-982 | await | JS rejects `session.ready()` with STORAGE_EMPTY. C++ `hcore::raise` throws from `get`. JS then goes through `_preload` (awaits `ready()` and runs `_maybeClosed()` with a null alias). C++ skips straight to `openCore_` (H2) |
60| C10 | corestore.cpp:555 | `Corestore::attachMaybe_` (JS `_attachMaybe`) | `hasCore(dk, true)` | `hasCore(dk, /*ifMigrated*/ true)` (CS:184) | **yes** | cs index.js:449-466: `:450 await this.ready()`, `:453 !(await this.storage.hasCore(discoveryKey, { ifMigrated: true }))`, `:456 if (this.closing) return`, `:462 if (!core.opened) await core.ready()` | await | **Runs inside the protomux pair (ondiscoverykey) callback** (corestore.cpp:523-530). It is already continuation-shaped (`done`), so it can complete later. But the `closing` re-check and the `core.ready()` await must follow the hasCore await. The C++ order (closing check after hasCore) is already right |
61| C11 | corestore.cpp:216 | `GroupNotifyHandle::updates` | `getGroup(topic)` (throws → mapped to "Invalid number undefined") | `Promise<std::optional<uint64_t>> getGroup (K32 topic)` (CS:178) | **yes** | cs lib/notify.js:12-18: `Readable.deferred(async () => { const groupId = await this._store.storage.getGroup(this._topic).catch(noop); if (groupId === null) return null; return ...createGroupUpdateStream(groupId, opts) })`; hs index.js:868-875 | await (inside a deferred stream) | **The return type changes**: JS returns a Readable synchronously and the await happens inside it. C++ returns `vector<Buf>` eagerly. `.catch(noop)` yields `undefined`, so `undefined !== null` and `createGroupUpdateStream(undefined)` is called. That error must surface as a **stream error**, not a throw from `updates()` (H13) |
62| C12 | corestore.cpp:221 | `GroupNotifyHandle::updates` | `createGroupUpdateStream(gid, since, reverse)` → `vector<Buf>` | `Stream createGroupUpdateStream (uint64_t group, double since = 0, bool reverse = true)` (CS:175) | no (stream) | notify.js:16; hs index.js:843-845 `createGroupUpdateStream(this.db, EMPTY, group, { gte: since, reverse })` | sync (stream) | `undefined` since/reverse fall to their defaults (0, true), which matches the C++ `value_or` |
63| C13 | corestore.cpp:420 | `Corestore::openCore_` | `std::make_unique<hcore::Core>(*shared_->storage, oo)`: the Core ctor resumes/creates the core on l2 (sync reads + writes) | `hcore::Core` must take `CorestoreStoragePtr` and open via `Promise<HypercoreStoragePtr> resumeCore (std::optional<K32>)` / `createCore (CreateCoreArgs)` (CS:191,195) | **yes** (indirect) | cs index.js:698-720 `Hypercore.createCore(this.storage, {...})` (sync ctor). The open happens in `core.ready()` (awaited at cs:462 and by sessions) | ctor sync, open awaited | **Core open becomes async.** `openCore_` currently returns a Tracked whose core is open or `openError`. JS registers the core in the tracker before it opens and sessions await `ready`. `openCore_`'s tail (corestore.cpp:450-454) reads `core->header` / `state->length` right after construction. In JS that is Core._open's tail, which runs after the open (H3). The swap itself is a core/ inventory item |
64| C14 | corestore.cpp:188 | `Shared::gcCycle` | `t->core->close()` (Core closes its l2 HypercoreStorage synchronously) | Core::close → `Promise<void> HypercoreStorage::close ()` (CS:97) | **yes** (indirect) | cs index.js:156-164 `core.close().then(gc, gc)`; `_gc` :151-154 deletes **only if `map.get(id) === core`, after close settles** | `.then` | C++ erases at once (:189). JS keeps a closing core in the map until close resolves. `_openCore` (cs:695-696) then sees `existing.closing` and opens a new core with `preopen: existing.closing` (H4) |
65| C15 | corestore.cpp:201 | `Shared::closeAll` | `t->core->close()` per core | as C14 | **yes** (indirect) | cs index.js:186-197 `close(){ ...; all.push(core.close()); this.map.clear(); return Promise.all(all) }` | returned, awaited by `_close` :445 | C++ discards the result. It must become `Promise.all` and be awaited before C5 |
66
67---
68
69## B. repl/src/replicator.cpp — hypercore 11.37 lib/replicator.js
70
71### B1. Type-only uses
72
73| # | file:line | C++ enclosing | type use | new equivalent | notes |
74|---|---|---|---|---|---|
75| T8 | replicator.cpp:558 | `Peer::_handleRequest` (local struct) | `struct CachedRx : merkle::Storage` | goes away: the batch becomes a `storage::CoreRXPtr` (S:228) passed to an async `merkle::proof` | the adapter exists only because merkle::Storage is sync (H5) |
76| T9 | replicator.cpp:559 | same | `l2::CoreRX rx;` member | `storage::CoreRXPtr` | |
77| T10 | replicator.cpp:560 | same | `explicit CachedRx (l2::CoreRX &&r)` | — | |
78
79replicator.h has no l2 types. It only uses `hcore::Core` (replicator.h:46). Its header comment (:9-11) states the sync-storage divergence ("wire ORDER may differ from JS"). R4c-4 is what retires it.
80
81### B2. Direct l2 calls
82
83| # | file:line | C++ enclosing | l2 call | new equivalent | Promise? | JS counterpart | JS awaits? | notes |
84|---|---|---|---|---|---|---|---|---|
85| R1 | replicator.cpp:566 | `Peer::_handleRequest` | `core->storage().read(int64_t(core->state->fork))` → `l2::CoreRX` (tree cache on, since fork ≥ 0) | `CoreRXPtr HypercoreStorage::read (int64_t fork = -1)` (CS:95) | no (sync ctor) | rep:1059 `const batch = this.core.storage.read(this.core.state.fork)`; hs index.js:428-433 (treeCache only when `!this.atom`) | sync | `core->storage()` returns `l2::HypercoreStorage&` (core/include/hc/hypercore.h:217). That is a core/ swap. The batch is taken **before** the fork comparison, as in JS |
86| R2 | replicator.cpp:562 | `CachedRx::get_tree_node` (called by merkle::proof's pending/settle) | `rx.getTreeNode(int64_t(index))` → sync optional | `Promise<std::optional<hs::CoreTreeNode>> CoreRX::getTreeNode (int64_t index)` (S:217) | **yes** | hs lib/tx.js:273-303 (tree cache: a miss caches the **pending** promise, parallel readers share it). Called via mt:1076/1081/1093/1104/1136/1157/1186/1199 `getTreeNodeOrError(rx, idx)` (mt:1265-1271) inside `generateProof` (mt:1314-1358) | the promises are **created** at generation time and **awaited** in `TreeProof.settle()` (mt:510-520 → `settleProof` mt:1294-1311: `Promise.all` per list, `allSettled` on failure) **after** the caller's `tryFlush` | **A mechanical swap deadlocks.** A JS rx read resolves only at `tryFlush`. merkle::proof currently reads synchronously during generation. It must be split into a generation phase that collects promises, then `tryFlush`, then settle (H5) |
87| R3 | replicator.cpp:569 | `Peer::_handleRequest` | `batch.rx.tryFlush()` (always, even on a failed proof) | `void CoreRX::tryFlush ()` (S:222); on the generation-error path `void CoreRX::destroy ()` (S:223) | no | rep:1071-1072 `batch.tryFlush()` only after a non-null req. On a generation error, rep:928 `batch.destroy()` inside `_getProof` | sync | C++ always calls tryFlush. JS **destroys** the batch when `MerkleTree.proof` throws (mt:921-935) |
88
89### B3. Indirect storage reads/writes (through Core / merkle::Storage-on-Core / SessionState)
90
91These sites reach storage through Core (`core/src/hypercore.cpp`: `get_tree_node` :445, `get_block` :451, `flushHints` :1065, `verify` :1077, `verifyReorg` :1149, `checkConflict` :1178, `reorg` :1358) or through `ByteSeeker` (core/src/byte_seeker.cpp:38). The core/ inventory owns the swap inside Core. The replicator-side consequence is listed here: the call becomes an await, inside a handler.
92
93| # | file:line | C++ enclosing | call (what it hits) | new form | Promise? | JS counterpart | JS awaits? | notes |
94|---|---|---|---|---|---|---|---|---|
95| R4 | replicator.cpp:540 | `Peer::_handleRequest` | `core->get_block(idx)` → `storage_->read().getBlock` (a **separate** rx, fork -1) | `batch->getBlock(int64_t idx)` on the **same** CoreRX as R1: `Promise<std::optional<Buf>> CoreRX::getBlock (int64_t)` (S:215) | **yes** | rep:918-919 `block = batch.getBlock(index); block.catch(noop)`, queued **before** `MerkleTree.proof` (rep:925); joined in `ProofRequest.fulfill` rep:480 `await Promise.all([this.proof.settle(), this.block])` | created early, awaited in fulfill | **Deviation:** C++ reads the block from a different read batch than the nodes. JS reads both from one rocks read batch, flushed together. Use the R1 batch |
96| R5 | replicator.cpp:568 | `Peer::_handleRequest` | `merkle::proof(s, batch, …)` with `s = core->session()`, `s.storage = Core`. The seek paths read via Core (L3–L6) | async `merkle::proof` (H5) | **yes** | rep:925 `const proof = await MerkleTree.proof(this.core.state, batch, msg)`. `generateProof` awaits `seekUntrustedTree` / `seekFromHead` (mt:1345-1354) through `session.storage` = **state.storage** | await | JS's session is `core.state` (SessionState) and C++'s is Core. For the default session both are the core's storage. I have not checked this for named/atomic sessions. While this awaits, other messages are processed. `remoteRequests` is re-checked after fulfill (C++ :602-603 is already there) |
97| R6 | replicator.cpp:480 | `Peer::_canUpgrade` | `merkle::upgradeable(*core, rLength)` → Core::get_tree_node per root (one rx per node, stops at the first missing) | async upgradeable over ONE rx (L24) | **yes** | rep:890-906: `:898 await MerkleTree.upgradeable(this.core.state, remoteLength)`, then **`:900 if (remoteFork !== this.core.state.fork) return false`**; mt:593-609 (one rx, every root queued, `tryFlush`, `Promise.all`) | await | The re-check after the await is **missing in C++** (H6). The read is batched in JS and per-node in C++ |
98| R7 | replicator.cpp:439 (→480) | `Peer::onsync` | `_canUpgrade(msg.length, msg.fork)` | await of R6 | **yes** | rep:790-848, `:828 : await this._canUpgrade(length, fork)`; `syncsProcessing` guards the latest-wins logic (:820, :838) | await | **This is a wire message handler.** The await sits between `syncsProcessing++` and `--syncsProcessing`. A second SYNC can arrive meanwhile, which is why the counter exists. The C++ code already has the counter, so semantics carry over |
99| R8 | replicator.cpp:461 (→480) | `Peer::_updateCanUpgradeAndSync` | `_canUpgrade(remoteLength, remoteFork)` | await of R6 | **yes** | rep:859-884: captures `length`, `fork`, `remoteLength`, `remoteFork` **before** the await (:860-863). After it, it bails if `syncsProcessing > 0` **or any of the four changed** (:866-875) | await | C++ checks only `syncsProcessing` (:462). The **four post-await staleness checks must be added** (H6). Callers: `signalUpgrade` :291 and `_handleData` :746. JS calls it un-awaited at rep:1290 (fire and forget) |
100| R9 | replicator.cpp:678 | `Peer::_handleData` | `core->checkConflict(data)` (merkle verify_fully_remote + `getRootsFromStorage` via `maybeGetRootsFromStorage`) | async Core::checkConflict | **yes** | rep:1214 `if (await this.core.checkConflict(data, this)) return`; core.js:620-… (`await maybeGetRootsFromStorage(this.storage, …)`) | await | Reached only under `_rxLock` (rep:1200-1209: lock when `request === 0 \|\| upgrade`). C++ :673 says "the rx lock is moot: synchronous" (H7) |
101| R10 | replicator.cpp:718 | `Peer::_handleData` | `core->verify(data)` (merkle verify reads `getTreeNodeOrError(*session.storage)` L10, downgrade L14, then a CoreTX write + `state->flush`, and `flushHints`) | async Core::verify | **yes** | rep:1258 `!(await this.core.verify(data, this))`; core.js:792-832. Concurrent shared verifies are **batched** (`_verifies` / `_verified`, :817-829) | await | C++ Core::verify **throws on re-entry** ("keetd: Core.verify while the state mutex is held", hypercore.cpp:1107). In JS, non-upgrade data (`request !== 0 && !upgrade`) skips `_rxLock` (rep:1201), so concurrent verifies are normal (H7). `_markProcessing` / `_markProcessed` + `dataProcessing` bracket the await (rep:1254-1287), and C++ keeps them |
102| R11 | replicator.cpp:876 | `Peer::onrange` | `core->flushHints()` (`storage_->write()` + `tx.setHints` + `tx.flush`) | async Core::flushHints → `std::shared_ptr<CoreTX> HypercoreStorage::write ()` + `Promise<bool> CoreTX::flush ()` (S:191) | **yes** | rep:1440 `if (this.core.hintsChanged) await this.core.flushHints()`, then `:1441 _update()` and `:1443 emitRemoteContiguousLength()` run **after** the write; core.js:605-618 (`hintsChanged = false` before the await, as a debounce) | await | **A message handler that awaits a write.** `_update()` and the emit must move after the await. Ordering against the next RANGE message changes (H8) |
103| R12 | replicator.cpp:1276 | `Peer::_send` | `merkle::missingNodes(*core, 2*idx, length)` (one Core::get_tree_node per step) | async missingNodes (L25) | **yes** | rep:1936-1941 `req.block.nodes = await MerkleTree.missingNodes(this.core.state, …)`, **then `if (req.priority === PRIORITY.CANCELLED) return`**; mt:743-762 (`await hasTreeNode(session.storage, ite.index)` sequentially, mt:754; hasTreeNode = new rx per node mt:1287-1292) | await | The **post-await CANCELLED check is missing** (H10). `_send` is called from the update walks (`_requestBlock` / `_requestRange` / …), which in JS fire it **un-awaited** and immediately count the request as sent (`inflight++` and the `_inflight.add` happen **before** the await, :1927-1928) |
104| R13 | replicator.cpp:1279 | `Peer::_send` | `merkle::missingNodes(*core, hash.index, length)` | as R12 | **yes** | rep:1944-1957 (await, CANCELLED check :1949, then the `nodes === 0` local resolve) | await | as R12 |
105| R14 | replicator.cpp:1712 | `Replicator::_resolveBlocksLocally` | `core->get_block(b->index)` per block: **one rx per block**, resolved synchronously and in order | ONE `CoreRXPtr reader = storage->read()`, then `reader->getBlock(i)` per block, then `reader->tryFlush()`, then `Promise.all` | **yes** | rep:2500-2523: one reader (:2506); a per-block `_resolveLocalBlock` (:2525-2534) with try/catch → `b.reject(err)`; `reader.tryFlush()` (:2511); `await Promise.all(blocks)` (:2513); the remove loop runs **after** | Promise.all; the caller **does not await** (`onupgrade` rep:2189) | **Ordering deviation:** in JS `onupgrade` continues synchronously into `_resolveUpgradeRequest` (:2190) and `_updateNonPrimary` (:2191-2193) **before** any block resolves. C++ resolves the blocks first. JS also has a reject path (a storage error rejects the block), and C++ has none |
106| R15 | replicator.cpp:1970 | `Replicator::_updateNonPrimary` | `s->seeker->update()` (ByteSeeker → `state->get_tree_node` per step, byte_seeker.cpp:38) | async ByteSeeker::update (core/) | **yes** | rep:2704 `res = await s.seeker.update()`; mt:441-480 (`await getTreeNodeFromStorage(this.session.storage, ite.leftChild())` :457) | await | `_updatingSeeks = true` spans the awaits (rep:2696 → :2721). `_requestSeek` bails while it is set (C++ :1088). There is also an `await yieldToLoop()` (rep:2729) that C++ omits (comment :1995) (H11) |
107| R16 | replicator.cpp:2217 | `Replicator::_onreorgdata` | `core->verifyReorg(data)` (`merkle::reorg(*state, …)`, which reads storage L22/L23, plus an optional manifest write) | async Core::verifyReorg | **yes** | rep:2904 `const newBatch = data.upgrade && (await this.core.verifyReorg(data))`; core.js:771-790 | await | Reached from `_handleData` (C++ :703) under `_rxLock` in JS (reorg data has `upgrade`) |
108| R17 | replicator.cpp:2229 | `Replicator::_onreorgdata` | `merkle::update(*f->batch, *core->state, data)` → updateNodes → `state.get_tree_node` (L13/L20) | async ReorgBatch update | **yes** | rep:2915 `await f.batch.update(data)`; mt:366-376 → `_update` mt:378-403 (sequential `await getTreeNodeFromStorage(this.session.storage, …)` :390; **`if (startingDiff !== this.diff) return false` :400**) | await | the `startingDiff` guard is skipped in C++ (merkle.cpp:728 comment). It must come back (H12) |
109| R18 | replicator.cpp:2259 | `Replicator::_applyReorg` | `core->reorg(*f->batch)` (state reorg: storage writes) | async Core::reorg | **yes** | rep:2933-2967: `this._applyingReorg = this.core.reorg(f.batch, null)` (:2939, **a promise**), `await this._applyingReorg` (:2942), catch → `u.reject` | await | C++ `_applyingReorg` is a `bool`. JS holds the promise: `applyPendingReorg` awaits it (rep:2205-2209), and C++ answers `true` without waiting (:2162). `_updateFork` (:2284), `updatePeer` (:2061) and `updateAll` (:2095) gate on it while it is pending (H12). `_applyReorg` is awaited from `_onreorgdata` (rep:2925) and `applyPendingReorg` (:2214) |
110
111Some calls on this path don't touch storage. `core->updateRemoteContiguousLength` (:858; core.js:876-880) and `core->openSkipBitfield` (:1817; core.js:848-855) are memory-only. `Peer::_onconflict` (:1255) calls `onrequest`, which reaches R1–R5. JS awaits that at rep:1459 before `channel.close()`, but C++ closes synchronously after the call (H10). `Replicator::onconflict` is `Promise.allSettled` in JS (rep:2197-2203).
112
113---
114
115## C. l3/ — merkle.cpp: hypercore lib/merkle-tree.js over the sync `merkle::Storage` interface
116
117l3 has no l2 dependency. It reads through `struct Storage { virtual std::optional<Node> get_tree_node (uint64_t) = 0; }` (merkle.h:36-39). Three classes implement it: `hcore::Core` (hypercore.h:118/220), `hcore::SessionState` (session_state.h:36, session_state.cpp:265 and :738), and the replicator's `CachedRx` (T8).
118
119JS draws a distinction that the C++ interface erases. Some JS functions take an **rx** (a CoreRX batch): every read is queued synchronously, the *caller* tryFlushes, and then they are awaited (often in `Promise.all`). Other functions take a **storage** (`getTreeNodeFromStorage` mt:1280-1285 / `…OrError` mt:1273-1278 / `hasTreeNode` mt:1287-1292). Those create a new rx per node, tryFlush at once, and await sequentially. The async port has to restore this distinction (H5).
120
121New primitives: `CoreRXPtr HypercoreStorage::read(int64_t fork = -1)` (CS:95), `Promise<std::optional<hs::CoreTreeNode>> CoreRX::getTreeNode(int64_t)` (S:217), `Promise<bool> CoreRX::hasTreeNode(int64_t)` (S:218), `void CoreRX::tryFlush()` (S:222), `void CoreRX::destroy()` (S:223).
122
123### C1. Type-only
124
125| # | file:line | use | notes |
126|---|---|---|---|
127| T11 | l3/include/hc/merkle.h:36-39 | `struct Storage` (sync `get_tree_node`) | becomes async (an rx- or storage-shaped interface returning `Promise<optional<Node>>`) |
128| T12 | l3/include/hc/merkle.h:51 | `merkle::Session::storage` (`Storage *`) | JS `session.storage` = a HypercoreStorage (state.storage) |
129| T13 | l3/src/merkle.cpp:75-78 | `struct Gen { Storage &rx; Storage &storage(); }` | rx = the passed batch; storage() = session.storage (the seek paths) |
130
131### C2. Reads (every reach of `Storage::get_tree_node`)
132
133| # | merkle.cpp line | C++ function | read | JS counterpart (verified) | JS shape | Promise? | notes |
134|---|---|---|---|---|---|---|---|
135| L1 | 51 | `has` | `s.get_tree_node(i)` | `getTreeNodeFromStorage` mt:1280 / `rx.getTreeNode` | either | yes | a helper |
136| L2 | 58 | `getTreeNodeOrError` | via `has` | `getTreeNodeOrError(rx, i)` mt:1265-1271 / `getTreeNodeFromStorageOrError` mt:1273-1278 | either | yes | the error text matches mt:1268 |
137| L3 | 106 | `Gen::getByteOffsetBatchSum` (via `getByteOffsetSession` :113) | `getTreeNodeOrError(storage(), i)` per lookup | `getByteOffsetSession(session, root, null)` mt:800-812 → `getByteOffsetBatchFlush` mt:825-829 (ONE `session.storage.read()`, every lookup queued in `getByteOffsetBatch` mt:831-865, `tryFlush`, `Promise.all`) | rx-batched | yes | C++ does one rx per node. JS does one batch. The NaN quirk (mt:845 pushes `node.size`) is preserved |
138| L4 | 124 | `Gen::seekTrustedTree` | `has(storage(), leftChild)` per step | `seekTrustedTree` mt:1023-1047, `await getTreeNodeFromStorage(session.storage, ite.leftChild())` mt:1031 | storage, sequential awaits | yes | |
139| L5 | 143 | `Gen::seekUntrustedTree` | `getTreeNodeOrError(storage(), root)` | mt:1050-1065: `await getByteOffsetSession(…)` (L3) then `await getTreeNodeFromStorageOrError(session.storage, root)` mt:1060 | storage | yes | |
140| L6 | 151 | `Gen::seekFromHead` | `has(storage(), root, n)` per root | mt:1001-1019, `await getTreeNodeFromStorage(session.storage, root)` mt:1006 | storage, sequential | yes | a null node gives the TypeError at getUnpaddedSize (mt:1361-1365). C++ :81 matches |
141| L7 | 258-262 | `resolve(rx, list)` | `getTreeNodeOrError(rx, i)` per index | the pending arrays filled by seekProof / blockAndSeekProof / upgradeProof / additionalUpgradeProof (mt:1071, 1086, 1111, 1161: `p.x.push(getTreeNodeOrError(rx, ite.index))`), resolved in `settleProof` mt:1294-1311 | rx: queued at generation, awaited at settle | yes | **The reads must be ISSUED during generation (before the caller's tryFlush) and awaited after it** |
142| L8 | 513 | `proof` (settle check) | `getTreeNodeOrError(rx, i)` over node/seek/upgrade/additionalUpgrade, in order | `settleProof`: `Promise.all([all(node), all(seek), all(upgrade), all(additional)])`, and on failure `allSettled` of each (mt:1302-1310) | rx | yes | `atSettle` (merkle.h:63-67) maps the settle-phase rejection to `ProofRequest.fulfill` / `_fulfillRequest` (rep:477-489, :1096) |
143| L9 | 518, 521, 524, 530, 531 | `proof` | `resolve(rx, *p.node / *p.seek / *p.upgrade / *p.additionalUpgrade)` | `TreeProof.settle` mt:510-560 (it builds the result from pNode/pSeek/pUpgrade/pAdditional) | rx | yes | They collapse into L8 once the reads are promises |
144| L10 | 557 | `verify` | `getTreeNodeOrError(*session.storage, unverified.index)` | `MerkleTree.verify` mt:693-712, `await getTreeNodeFromStorageOrError(session.storage, unverified.index)` mt:705 | storage | yes | reached from Core::verify (R10) |
145| L11 | 647 | `offsetSum` | `getTreeNodeOrError(rx, i)` per lookup | `getByteOffset(tree, index, rx)` mt:814-823 → `getByteOffsetBatch` mt:831-865 (rx, `Promise.all`) | rx (caller flushes) | yes | used by L17/L18/L26/L27 |
146| L12 | 662 | `rangeOf` | `getTreeNodeOrError(rx, index)` (size) | `getNodeSize` mt:796-798 inside `getByteRange` mt:867-876 (`Promise.all([offset, size])`) | rx | yes | The C++ comment explains the rejection order. A real `Promise.all` will reproduce it |
147| L13 | 716 | `updateNodes` (= `ReorgBatch._update`) | `storage.get_tree_node(left.index)` per step | mt:378-403, `await getTreeNodeFromStorage(this.session.storage, left.index)` mt:390; **`startingDiff !== this.diff` → false** mt:400 | storage, sequential | yes | the guard is dropped in C++ (:728) (H12) |
148| L14 | 832 | `downgrade` | `storage.get_tree_node(root.index)` per root, with an early return | `MerkleTreeBatch.downgrade` mt:301-325 (ONE `this.storage.read()`, all roots queued, `tryFlush`, `Promise.all`, then compare) | rx-batched | yes | C++ stops at the first miss. JS reads all of them (no observable difference besides read count) |
149| L15 | 842 | `restore` | `getRootsFromStorage(storage, length)` | `MerkleTreeBatch.restore` mt:327-340, `await MerkleTree.getRootsFromStorage(this.storage, length)` | rx-batched (L28) | yes | |
150| L16 | 853 | `get(Batch, …)` | `storage.get_tree_node(index)` | `MerkleTreeBatch.get` mt:157-167 → `getTreeNodeFromStorage(this.session.storage, index)` | storage | yes (the return is a promise) | |
151| L17 | 858 | `byteOffset(Batch, …)` | `byteOffsetOf(…, storage, …)` → L11 | `MerkleTreeBatch.byteOffset` mt:289-298 (`this.storage.read()`, `getByteOffset`, `tryFlush`) | rx | yes | |
152| L18 | 865 | `byteRange(Batch, …)` | `rangeOf(…, storage, …)` → L11+L12 | `MerkleTreeBatch.byteRange` mt:281-287 (rx, `getByteRange`, `tryFlush`) | rx | yes | `getByteRange`'s BAD_ARGUMENT is a **synchronous throw** in JS (mt:869-871; byteRange is not async), not a rejection |
153| L19 | 893 | `batchSeek` | `storage.get_tree_node(leftChild)` per step | `ByteSeeker._seek` mt:441-475 over `new ByteSeeker(this, this, …)` (mt:279), `await getTreeNodeFromStorage(this.session.storage, ite.leftChild())` mt:457 | storage, sequential | yes | |
154| L20 | 952 | `update(ReorgBatch, …)` | → `updateNodes` (L13) | `ReorgBatch.update` mt:366-376 (`return this._update(nodes)`) | storage | yes | R17 |
155| L21 | 966 | `truncate` | `getTreeNodeOrError(storage, root)` per missing root | `MerkleTree.truncate` mt:619-642, `await getTreeNodeFromStorageOrError(session.storage, root)` mt:628 | storage, sequential | yes | caller is core/ (SessionState truncate) |
156| L22 | 988 | `reorg` | `storage.get_tree_node(root.index)` per root until the first diff | `MerkleTree.reorg` mt:644-670, `await getTreeNodeFromStorage(session.storage, root.index)` mt:656 | storage, sequential | yes | R16 |
157| L23 | 996 | `reorg` | → `updateNodes` (L13) | mt:663 `await batch._update(batch.nodes)` | storage | yes | |
158| L24 | 1006 | `upgradeable` | `storage.get_tree_node(i)` per full root, stops at the first miss | mt:593-609 (one rx, all queued, `tryFlush`, `await Promise.all`) | rx-batched | yes | R6 |
159| L25 | 1044 | `missingNodes` | `storage.get_tree_node(ite.index())` per step | mt:743-762, `await hasTreeNode(session.storage, ite.index)` mt:754 | storage, sequential | yes | R12/R13 |
160| L26 | 1054 | `byteOffset(Session, …)` | → L11 | `MerkleTree.byteOffset` mt:765-767 → `getByteOffsetSession(session, index, null)` → BatchFlush (mt:825) | rx-batched (own rx) | yes | |
161| L27 | 1058 | `byteRange(Session, …)` | → L11+L12 | `MerkleTree.byteRange` mt:769-775 (one rx, `getByteOffsetSession(…, rx)` + `getNodeSize`, `tryFlush`, `Promise.all`) | rx | yes | |
162| L28 | 1070 | `getRootsFromStorage` | `getTreeNodeOrError(storage, i)` per root | mt:579-591 (one rx, all queued via `getTreeNodeOrError(rx, …)`, `tryFlush`, `Promise.all`) | rx-batched | yes | used by L15 and core/ (`maybeGetRootsFromStorage`, checkConflict R9) |
163
164---
165
166## Groups (by JS function)
167
168- **cs `_open` / `_getOrSetSeed`** (cs:392-416): C2 (keetd-only), C3, C4.
169- **cs `_close` + `CoreTracker.close`** (cs:418-447, :186-197): C5, C15.
170- **cs `CoreTracker._gcCycle` / `_gc`** (cs:151-164): C14.
171- **cs `list` / `getAuth`** (cs:380-386): C6, C7 (+T1).
172- **cs `get` → `_preload` / `_preloadCheckIfExists`** (cs:563-640): C8, C9, T5.
173- **cs `_openCore`** (cs:692-740) → Core open: C13, T6, T7.
174- **cs `_attachMaybe`** (cs:449-466): C10.
175- **cs lib/notify.js `updates`** (notify.js:12-18): C11, C12.
176- **cs ctor → `Hypercore.defaultStorage`** (cs:241-249, hc index.js:208-212): C1, T2–T4.
177- **rep `Peer._handleRequest` / `_getProof` / `ProofRequest.fulfill` / `_fulfillRequest`** (rep:477-489, 908-937, 1044-1078, 1080-…): R1, R2, R3, R4, R5, T8–T10, L7, L8, L9 (+ L3–L6 for seeks).
178- **rep `Peer._canUpgrade` / `onsync` / `_updateCanUpgradeAndSync`** (rep:790-906): R6, R7, R8, L24.
179- **rep `Peer.ondata` / `_handleData`** (rep:1200-1292): R9, R10 (+ core.js verify → L10, L14, L28).
180- **rep `Peer.onrange`** (rep:1408-1444): R11.
181- **rep `Peer._send`** (rep:1924-1965): R12, R13, L25.
182- **rep `Replicator._resolveBlocksLocally` / `_resolveLocalBlock`** (rep:2500-2534), reached from `onupgrade` (rep:2187-2194): R14.
183- **rep `Replicator._updateNonPrimary`** (rep:2658-2733): R15, L19-shape (ByteSeeker).
184- **rep `_onreorgdata` / `_applyReorg` / `applyPendingReorg`** (rep:2205-2218, 2903-2967): R16, R17, R18, L13, L20, L22, L23.
185- **mt batch API used by core/** (get / byteOffset / byteRange / downgrade / restore / truncate / seek): L14–L19, L21, L26–L28.
186
187## Hazards (not a mechanical swap)
188
189- **H1 — `Corestore::ready()` / `close()` are void+sync.** C2–C5 become awaits, so both methods must return `js::Promise<void>`. Every keetd caller of `Corestore::ready` has to await it (daemon boot, bee/blobs test helpers). The extra `storage->ready()` (C2) is keetd-only. To be literal, drop it and let `getSeed` migrate (hs:857).
190- **H2 — `Corestore::get` is synchronous and throws.** JS returns a session at once (cs:590-602) and defers the alias/hasCore lookups into `conf.preload`. STORAGE_EMPTY and alias failures then reject `session.ready()`. Today `makeSession_` (:503-509) throws when `ready` settled synchronously with an error. After R4c-4 nothing settles synchronously. Either `get` returns the unopened session (JS) or it becomes `Promise<unique_ptr<Session>>`. Its callers include bee.cpp:394 and :649, which use the session immediately (H14).
191- **H3 — Core open becomes async (C13).** `openCore_` reads `t->core->header` / `state->length` right after the ctor (corestore.cpp:450-454, Core._open's tail). `attachMaybe_` checks `openError` synchronously (:567). JS awaits `core.ready()` (cs:462). The Tracked must exist before the open settles (JS `cores.set` happens before the open; C++ already pushes at :409). Every consumer of `t->core` state has to move behind the open.
192- **H4 — the GC/close race (C14).** JS deletes the core from the tracker only **after** `core.close()` resolves, and only if it is still the mapped one. `_openCore` must then honour `existing.closing` / `preopen` (cs:695-700). C++ `resume` (:157-166) has no `closing` check and `gcCycle` erases at once. With async close, a `get` during the close window must open a new Core chained after the old one's close.
193- **H5 — merkle::Storage is synchronous, and JS's rx contract is "issue → tryFlush → await".** A promise-returning `getTreeNode` that is waited on during proof generation **deadlocks**, because the read only resolves at the caller's `tryFlush` (hs lib/tx.js:321-324 → rn read batch). merkle::proof (merkle.cpp:462-543) has to become generate (push promises into pending; may await **storage-shaped** seek reads L3–L6 via `session.storage`, each with its own rx) → caller tryFlush → settle (`Promise.all` per list, `allSettled` on failure, mt:1294-1311). Restore the rx-vs-storage split across L1–L28. Sites JS batches that C++ reads per node: L3, L14, L24, L26, L28. Sites JS reads sequentially with one rx each: L4–L6, L10, L13, L16, L19, L21, L22, L25. The tree cache's shared pending promise (S:132-137, hs tx.js:273-303) makes parallel proofs share in-flight node reads. That is only observable once reads are async.
194- **H6 — post-await staleness re-checks that C++ skipped.** `_canUpgrade` needs `if (remoteFork !== this.core.state.fork) return false` after the await (rep:900). `_updateCanUpgradeAndSync` captures length/fork/remoteLength/remoteFork and bails if any changed (rep:860-875). C++ (:461-471) checks only `syncsProcessing`.
195- **H7 — `_rxLock` and concurrent `Core.verify`.** C++ `Peer::ondata` drops `_rxLock` ("moot: synchronous", :673). JS takes it only for `request === 0 || upgrade` data (rep:1200-1209). It also has `_txLock` (rep:2032-2033), used only for push and locked requests, and push is not ported. Plain block data verifies **concurrently**, and Core batches those writes via `_verifies` / `_verified` (core.js:817-829). C++ Core::verify **throws on re-entry** (core/src/hypercore.cpp:1107). Before replicator handlers can await, the replicator needs `_rxLock` and Core needs the `_verifyShared` batching (core/ owns that).
196- **H8 — handlers that await a write (R11).** In `onrange`, `_update()` and `emitRemoteContiguousLength()` run after `await flushHints()` (rep:1440-1443). Moving them behind the await changes the order relative to the next decoded message on the channel. protomux JS tracks the handler promise (`_track`, protomux index.js:184-191: `_active++`, and a rejection destroys the channel). keetd's Protomux has no `_track` (mux/include/hc/protomux.h:15-17: "Handlers are synchronous … _active not modelled"). An async rejection (settle failure in `_handleRequest`, verify throws after the await) has no path to destroy the channel, and `ondestroy` (→ `onpeerdestroy` → `core->checkIfIdle` → corestore GC) is no longer delayed until handlers finish. **mux/ needs `_track` before repl handlers go async.**
197- **H9 — destructors and lifetimes.** `~Corestore` (:267) calls `close()`. `Tracked::~Tracked` resets replicator then core. `Corestore::Shared::~Shared` stops the GC. None of these can await `storage->close()` / `core->close()`. Any coroutine still in flight (a proof, verify or flushHints) holds raw `Peer *` / `Replicator *` / `Core *`. Replicator already has `life_` (replicator.h:326) for timers, and continuations need the same guard. `GroupNotifyHandle` holds `shared_` (type-erased): if `updates()` becomes deferred, the stream must keep `shared_` alive.
198- **H10 — cancellation and ordering inside `_send` and `_onconflict`.** After each `missingNodes` await, JS returns if `req.priority === PRIORITY.CANCELLED` (rep:1941, :1949). C++ (:1274-1291) has no such check, because nothing could cancel mid-call. JS calls `_send` un-awaited from the update walks, so `inflight` / `_inflight` / `_addUpgrade` happen synchronously and the REQUEST goes on the wire later. Wire order of REQUEST frames versus other sends then changes (the replicator.h:10 divergence note becomes JS-exact). `_onconflict` awaits `onrequest` before `channel.close()` (rep:1456-1471). If C++ closes synchronously, the conflict proof is lost.
199- **H11 — `_updateNonPrimary` yields.** JS awaits `seeker.update()` and `yieldToLoop()` per round (rep:2704, :2729). `_updatesRunning` / `_updatesQueued` / `_updatingSeeks` then become real cross-task guards. C++ already models the flags. The `destroyed` checks after the yield (rep:2730, :2733) start to matter.
200- **H12 — reorg is a promise in JS.** `_applyingReorg` is the `core.reorg()` promise (rep:2939). `applyPendingReorg` awaits it (rep:2206-2208), and C++ answers `true` immediately (:2162), so Hypercore#update callers will race. `ReorgBatch._update` must restore `if (startingDiff !== this.diff) return false` (mt:400; C++ merkle.cpp:728 removed it as unreachable).
201- **H13 — return-type changes on the public store API.** `Corestore::list` changes `vector<Buf>` → `Stream`. `Corestore::getAuth` changes `optional` → `Promise<optional>`. `GroupNotifyHandle::updates` changes `vector<Buf>` → a deferred Readable. Its "Invalid number undefined" failure (G2) must become a **stream error** from `createGroupUpdateStream(undefined)`, not a synchronous throw. Callers outside these dirs (daemon / godot-keet bindings) will need updating, and I did not inventory them.
202- **H14 — transitive sync-completion assumptions (not l2 uses, but they break the moment storage is async).** Each of these reads a callback result immediately after the call:
203 - `blobs/src/blobs.cpp:43-51` `now<T>()` **throws** "did not complete synchronously". It is used at :58 get, :67 seek, :73 append, :77 clear, :84 and :564 session ready.
204 - `bee/src/bee.cpp:47-55` `getLocal` (`s.get(…, [&]…)`, then it reads `out`).
205 - `bee/src/bee.cpp:409` `core->getUserData(…, [&]…)`, then it reads `buf`.
206 - `bee/src/bee.cpp:1129` and `:1693` `append(…, [&]…)`, then it reads `appendError`.
207 - `bee/src/bee.cpp:394` and `:649` `store.get(g)` (H2).
208
209 All of these resolve today only because Session → Core → l2 is synchronous.
210- **H15 — the block and its nodes must come from one read batch (R4).** Today `get_block` uses a second rx without a fork. Use `batch->getBlock` on the R1 rx, issued before `merkle::proof`, as JS does (rep:918). On a generation-phase proof error, `destroy()` the rx instead of calling `tryFlush()` (rep:928).
211- **H16 — `_resolveBlocksLocally` ordering (R14).** JS fires it un-awaited from `onupgrade` and resolves the blocks later in microtasks, after `_resolveUpgradeRequest` / `_updateNonPrimary` have run. JS also rejects the block on a storage error. C++ resolves first and has no reject path. The `inOndata_` microtask emulation (replicator.h:429-433) should go away once every resolve really is a continuation.
docs/research/r4c4-inv-tools.md +348 -0
……@@ -0,0 +1,348 @@
1# R4c-4 inventory: tools, tests, CMake, drills, keys/codecs (read-only)
2
3Worktree: `/storage/projects/code/hypercore-kb/hyper-fleeky/.scope/keetd` (branch `r4`, HEAD `9bc081d` = R4c-3b-3).
4All paths below are relative to the worktree. Line numbers are taken from the files at that HEAD.
5
6OLD = `hc::l2` (`l2/include/hc/l2.h` 523 lines; `l2/src/{db,keys,storage,tx,view}.cpp`), synchronous.
7NEW = `hc::storage` (`storage/include/hc/{storage,corestore_storage}.h`), async `js::Promise`, on `rn`.
8Settling NEW from synchronous callers = `rt::scaffold::Sync` (`rt/include/hc/scaffold.h`, R4c-1):
9`Sync s; auto rx = st->read(); auto p = rx->getX(); rx->tryFlush(); use(s.wait(p));`
10
11---
12
13## 0. Summary of findings
14
151. **The M4a storage gate has NOT been converted.** `m4a_storage_gate` still runs `l2_ops` (linked to `hc_l2`) against `tools/m4/storage-ops.js` (CMakeLists.txt:268-274). R4c-3b-3 added a **separate** gate, `tools/st` (`r4c_st`), instead of turning `l2_ops` into the "async l2_ops gate" that the design describes (§9.4 R4c "Gates" 1, and R4c-3 / R4c-3b-3). M4a's 10 scripts, its cross-open check and its RocksDB-OPTIONS check exist only on hc::l2 today. See §5.
162. **The M4d kill-point hook has no write-only equivalent in rn.** `l2::test_after_write` is called once per non-empty write batch, synchronously, after `rocksdb_write` (l2/src/db.cpp:220,240). rn has `rn::testOnCompletion` (rn/include/hc/rn.h:40-42, rn/src/native.cpp:13,18-22), but it fires at the top of EVERY parked binding completion (open, close, read, write, iterator, flush, compact, …), so it cannot count write batches as it is. The natural place for the hook is the `rocksdb_native__on_write` completion in `rn::native::write` (rn/src/native.cpp:279-283). See §1.6.
173. **Deleting `l2/` breaks 62 of the 89 ctests at build time.** `hc_core` links `hc_l2` PUBLIC, and `hc_storage` links `hc_l2` PUBLIC and includes `hc/l2.h`. Everything above core, and every storage gate, depends on it. See §2.
184. **75 drill anchors point into `l2/`:** l2 5, m5g1a 34, m5g3a 30, m5g3c 6. The l2, m5g1a and m5g3a sets also gate on `l2_ops` + `storage-gate.sh`. Another 69 anchors in core/repl sit on storage-call lines that R4c-4 rewrites. `drill_lint` (ctest) fails as soon as any anchor file disappears or an anchored line changes. See §3.
195. **What has to outlive `l2/`:** `l2::keys` (keys.cpp, 121 lines; used by storage/src/{streams,tx,hypercore_storage,migrations0}.cpp); the type aliases `hs`, `K32`, `Change`, `Dependency`, `CorePtr`, `TreeCacheStats`; and hc_storage's direct link to `hc_schema_hypercore_storage`. `encode_core_auth` / `encode_core_head` are used only inside l2 and can go with it; storage already has `encodeAuth` / `encodeHead`. See §4.
20
21---
22
23## 1. Every tool and test that uses l2 or exercises storage
24
25### 1.1 `tools/l2-ops/main.cpp` (416 lines): the M4a gate's C++ side (l2's only direct test)
26
27- **Uses from l2 (everything; `using namespace hc::l2`, :17):**
28 - `CorestoreStorage(dir, Options{readOnly,id})` + `ready()` (:153-157); `close()` (:144).
29 - Store calls: `setSeed` / `getSeed` / `setDefaultDiscoveryKey` / `getDefaultDiscoveryKey` (:160-167); `createCore(CreateCoreArgs)` (:168-179); `resumeCore` (:181); `createAtom` (:191); `stats` (TreeCacheStats, :221).
30 - More store calls: `exportCore(dk, batches)` → `CoreExport` (:257-273); `deleteCore` / `audit` / `clear` / `flush` (:274-281); `getInfo` / `getInfos` → `Info` (:283, :297-304).
31 - `setRecovering` with the injectable `now` clock (:285-290); `hasCore` / `getAuth` / `getAlias` / `createGroup` / `getGroup` (:291-310).
32 - Store streams, all collected vectors: `createDiscoveryKeyStream` / `createAliasStream` / `createCoreStream` / `createGroupUpdateStream` (:366-379).
33 - `HypercoreStorage`: `snapshot`, `snapshotted`, `close`, `atomize` (:183-194); the dependency calls (`getDependencyLength`, `getDependency`, `findDependencyIndex`, `setDependencyHead`, `updateDependencyLength`, :223-236); `core()` (`CorePtr`, :238).
34 - More `HypercoreStorage`: `createSession` / `resumeSession` / `createAtomicSession` / `deleteSessions` / `compact` / `isParentStorage` (:239-255); per-core streams `createBlockStream` / `createBitfieldStream` / `createTreeNodeStream` / `createUserDataStream` / `createMarkStream` / `createLocalStream`, all collected (:380-403).
35 - `Atom`: `preflush`, `onflush(Handler(Done))` with re-entry, `flush(Done)`, which throws "did not complete" if it is not synchronous (:195-216).
36 - `CoreTX`: every setter / put / delete, then `flush()` → bool (:311-344). `CoreRX`: every getter, then `tryFlush()` (:345-365).
37 - `l2::Error` (:235, :342, :363, :405).
38- **Gate / comparison:** `tools/m4/storage-gate.sh` (M4a) runs `bare tools/m4/gen-storage-ops.js` to produce 10 scripts: basic, ranges, reopen, cross, snapshots, atoms, treecache, deps, sessions, storeops (gen-storage-ops.js:506). Each script runs on `bare tools/m4/storage-ops.js` (hypercore-storage 3.3.0, keet-daemon's node_modules) and on `$L2_OPS`. The check is:
39 - op-result lines identical (`cmp`);
40 - `tools/store-dump/dump.js` dumps of both stores identical, `versions` popped (storage-gate.sh:25-30);
41 - RocksDB `OPTIONS-*` identical (:31-34);
42 - cross-open: each side opens the OTHER's `basic` store writable, which verifies the device file and takes the lock (:37-48).
43- **ctest:** `m4a_storage_gate` (CMakeLists.txt:273-274, env `L2_OPS`).
44- **Drill target:** sets `l2`, `m5g1a`, `m5g3a` (tools/m5e/drill.py:1098, 1105, 1116).
45- **Under hc::storage it must become one of these (the design names (a); R4c-3b-3 effectively started (b)):**
46 - (a) **Retarget** `l2_ops` onto hc::storage. The design (R4c "Gates" 1) says: "`l2_ops` drives L2 with no scaffold, both sides print every settlement with its ruler hop … Bare at pool=1 vs rt real AND virtual … The existing scripts keep their outputs." That makes it a coroutine driver like st_trace, and storage-ops.js needs settlement tracing too.
47 - (b) **Delete** `l2_ops`, `storage-ops.js`, `gen-storage-ops.js`, `storage-gate.sh` and the `m4a_storage_gate` test, after porting M4a's 10 scripts plus cross-open plus the OPTIONS check into `tools/st` (see §5 for what st lacks).
48 - (c) Minimal path: rewrite `l2_ops` over hc::storage with `Sync` around every op. The outputs are results only, so they stay byte-identical, but no hops are gated, and the design deletes `scaffold::` usage at R4i.
49
50### 1.2 `tools/core-fixture/fixture.h` (69 lines): the M3c/M3d upload fixture writer
51- Not a CMake target. Included by `repl/test/m3c_pipe_test.cpp:8` and `tools/keetd-replicate/main.cpp:18`.
52- **Uses:** `write_core(l2::CorestoreStorage &db, const Source &)` (:31). It calls `l2::CreateCoreArgs` (:32), `db.createCore(c)` → `unique_ptr<HypercoreStorage>` (:38), `l2::CoreTX tx = st->write()` (:40), `putTreeNode(l2::hs::CoreTreeNode)` (:41), `putBlock` / `putBitfieldPage` (:46, :51), `setHead(l2::hs::CoreHead)` (:60), `setHints(l2::hs::CoreHints)` (:64) and `tx.flush()` (:65). It also uses `hcore::to_stored` (core/include/hc/hypercore.h:334, which returns `l2::hs::CoreManifest`).
53- **Compares against:** nothing itself. Its data is vectors/data/m3b.json.
54- **Becomes:** a rewrite taking `storage::CorestoreStorage &` (or `CorestoreStoragePtr`). Inside one `Sync`: `auto st = s.wait(db.createCore(c)); auto tx = st->write(); …; s.wait(tx->flush());` (`tx` is `shared_ptr<CoreTX>`; `flush()` returns `Promise<bool>`). The written bytes are unchanged.
55
56### 1.3 `tools/core-ops/main.cpp` (936 lines): the M4b core gate's C++ side
57- **Uses from l2:**
58 - `namespace hs = hc::l2::hs` (:25).
59 - Store map `std::map<std::string, std::unique_ptr<l2::CorestoreStorage>>` (:154). The `storage(name)` factory does `make_unique<l2::CorestoreStorage>(dir/name, Options{})` + `ready()` (:174-181). `st->close()` runs on reopen (:237) and at exit (:934).
60 - `Core(storage(...), oo, clock)` takes `l2::CorestoreStorage&` (:226), as does `fully_remote_proof::verify(storage(o["store"]), …)` (:500).
61 - Raw tx ops, each `l2::CoreTX tx = core.storage().write(); …; tx.flush();`: `setHints` (:257-260), `corruptHeadSignature` (:466-472), `deleteTreeNodes` (:478-481), `rawPutBitfieldPage` (:571-574), `rawPutBlock` / `rawDeleteBlock` / `rawPutTreeNode` (:575-580).
62 - Raw reads: `state->storage->read().getUserData` (:266); `l2::CoreRX rx = c.storage().read()` with getBlock / getTreeNode / getBitfieldPage / getMark / getHead / getHints / getAuth (:286-303); `c.storage().read().getHead()` (:468). Note that `rx.tryFlush()` is never called here, so the RX destructor frees.
63 - `st.storage->dependencies()`, `st.storage->core().dataPointer` (:532, :535).
64 - Atoms: `std::map<…, std::shared_ptr<l2::Atom>>` (:203); `storage(...).createAtom()` (:899); `atoms[...]->flush(Done)` + `take` (:900-903); `->flushing` (:905). Atoms are also passed as `SessionOptions::atom` (:322).
65- **Gate:** `tools/m4/core-gate.sh` vs `tools/m4/core-ops.js` (hypercore + hypercore-storage). Per script it checks results identical, every store dumped identically through dump.js, identical OPTIONS, and cross reopen (core-gate.sh:1-13). Also `fuzz_core_corpus` / `fuzz_core_campaign` (`tools/fuzz/run_corpus.py`, `fuzz.py`, `gen-core.js`).
66- **ctests:** `m4b_core_gate` (CMakeLists.txt:288-289), `fuzz_core_corpus`, `fuzz_core_campaign` (:294-299).
67- **Drill target:** core, m5g1c, m5g1d, m5g3b, m5g3c, m5g3d, m5g3e1, m5g3e2, m5g4a, m5g4b, m5g4c, m5g4d, m5g4e2.
68- **Becomes:** a rewrite of the l2 call sites only:
69 - `std::shared_ptr<storage::CorestoreStorage>` with `Sync`-waited `ready()` / `close()`;
70 - raw tx/rx as `Sync` + `s.wait(tx->flush())` / `s.wait(rx->getX())` + `rx->tryFlush()`;
71 - `storage->core->dataPointer` (`core` is now a `shared_ptr<CorePtr>` field);
72 - atoms as `storage::AtomPtr`, with `atomFlush` = `s.wait(storage::Atom::flush(atom))` or `.then` + `rth::act`.
73
74 Whatever Core's new constructor takes (`CorestoreStorage&` vs `Ptr`) decides :226 and :500. The JS twin (core-ops.js) is unchanged. The outputs must stay byte-identical.
75 - **Check:** an `atomFlush` inside `Sync` runs SessionState's onflush handler (commit_, wrapped once with `withResolvers` until R4e) inside the scope. If that handler waits on anything outside the scope (a core mutex held by another op, a timer), `Sync::wait` throws `NoProgress`. That needs a test before it is assumed safe.
76
77### 1.4 `tools/store-ops/main.cpp` (234 lines): the M4c-1 store gate's C++ side
78- **Uses from l2:** only `namespace hs = hc::l2::hs` (:18), for `j_stored_manifest(const hs::CoreManifest&)` (:47) and the `hs::CoreAuth` that `store::Corestore::getAuth` returns (:189; store/src/corestore.cpp:332). Every storage call goes through `store::Corestore`: `ready` (:118), `get` (:151), `list` (:160, :195), `close` (:202), `notifyGroup` / `updates` (:204-214).
79- **Gate:** `tools/m4/store-gate.sh` vs `tools/m4/store-ops.js` (corestore 7.12.6): results, dumps, OPTIONS, cross reopen. Also `fuzz_store_*` and m4d_gate's LOCK / CRASH `ops_run` (tools/m4d/gate.py:49-54).
80- **ctests:** `m4c_store_gate` (:326-327), `fuzz_store_corpus` / `fuzz_store_campaign` (:329-332), `m4d_gate` (:341).
81- **Becomes:** a one-line retarget of the `hs` alias (to `hc::storage::hs` or `hc::schema::hypercore_storage`), plus whatever Corestore's R4c-4 API change is (if `ready()` becomes async the harness waits). The outputs must be identical.
82
83### 1.5 `tools/m4d/appender.cpp` (91 lines) + `tools/m4d/gate.py` (148 lines): durability
84- **Uses from l2:** `l2::test_after_write = [] { if (++writes >= limit) { fflush; _exit(137); } }`, set after `st.ready()` (appender.cpp:58-62; doc :6-7). Everything else goes through `store::Corestore` (:49-68) and `Session::append` (:78).
85- **Gate:** gate.py runs three phases.
86 - LOCK: 4 holder × contender pairings, keetd (`m4d_appender`, `store_ops`) vs JS (`appender.js`, `store-ops.js`). Every pairing must refuse with the JS↔JS message (:68-86).
87 - CRASH: a random SIGKILL, then `verify.js` (JS reference) on the store, then keetd reopens it and appends, then verify.js again (:88-124).
88 - BOUNDARY: `--exit-after-writes N` for consecutive N from a random base. verify.js must pass and the exit code must be 137 (:126-143).
89- **ctest:** `m4d_gate` (CMakeLists.txt:339-341, target `m4d_appender`, links `hc_store`).
90- **Where the hook must move:**
91 - l2 today calls it synchronously at the end of `Db::write`, after `rocksdb_write` + cleanup, ONLY for non-empty change lists (`if (changes.empty()) return;` l2/src/db.cpp:224, hook :240).
92 - rn has `rn::testOnCompletion` (rn/include/hc/rn.h:40-42; defined rn/src/native.cpp:13; called in `complete()` :18-22). It fires for every binding completion, not only writes. st_trace uses it as a ruler (tools/st/st-trace.cpp:390, :395).
93 - The write path is `storage::View::flush` (storage/src/view.cpp:365-381) → `rn::WriteBatch::flushWrite_` (rn/src/batch.cpp:244-264) → `rn::native::write` (rn/src/native.cpp:265-283). There `rocksdb_write` runs synchronously at submit (:277, the R4b inline mode) and the `rocksdb_native__on_write` completion is parked (:279-283) and then delivered to `WriteBatch::onwrite_` (batch.cpp:266).
94 - **Proposed:** a write-only gate hook (e.g. `rn::testOnWrite`) called at the top of the on_write completion lambda (native.cpp:279). Under R4b the batch is already in RocksDB at that point, and under R4i the completion still follows the pool's write, so "after the batch reached RocksDB" holds in both modes. An alternative is to have the appender filter `testOnCompletion` by kind, but that needs a kind argument.
95 - **Semantic shifts the gate must allow for:**
96 - rn writes a batch even for an empty non-null change list (View.flush only short-circuits `changes == nullptr`, view.cpp:366), while l2 skipped empty lists.
97 - Store ops now group-commit (one write for several ops; corestore_storage.h:289-291).
98
99 "Write N" therefore numbers differently. The BOUNDARY phase uses consecutive N from a random base, so it stays meaningful, but the old/new counts are not comparable.
100 - The appender also has to update its comment (:6-7), and gate.py its comment (:126).
101
102### 1.6 `tools/keetd-replicate/main.cpp` (259 lines): M3d, the real transport
103- **Uses from l2:**
104 - `auto store = std::make_unique<l2::CorestoreStorage>(storeDir, Options{})` + `ready()` (:140-143);
105 - `fixture::write_core(*store, src)` (:144);
106 - `hcore::Core(*store, oo)` (:150);
107 - `store->close()` at exit (:254-257);
108 - doc comment :10.
109- **Gate:** `tools/m3d/gate.py`, which is manual, NOT a ctest. It runs keetd ↔ hypercore on the PUBLIC DHT. When keetd downloaded, the JS reference reopens keetd's store and reads the core back (gate.py:15-17).
110- **Build:** only when the hyperswarm-peer sibling exists (CMakeLists.txt:435-456, target `keetd-replicate`, links `hc_repl hsp_transport`).
111- **Becomes:** a rewrite like m3c_pipe's `CppSide`: a `storage::CorestoreStorage` (shared_ptr) with `Sync`-waited `ready` / `close`, and a fixture via the new `write_core`.
112 - **Check:** the store is opened on the MAIN thread before the transport thread starts. All later rt work runs on the transport loop thread (pump at :218-233). With rn, the store open parks `rt::complete` completions. Inside a `Sync` scope they are scope-local and delivered before `wait` returns, so nothing should be left for the other thread, but this has not been run.
113
114### 1.7 `repl/test/m3c_pipe_test.cpp` (2347 lines): M3c, keetd ↔ JS hypercore over a pipe
115- **Uses from l2:**
116 - `CppSide::store` is a `std::unique_ptr<hc::l2::CorestoreStorage>` (:128). Three constructors each do `make_unique<l2::CorestoreStorage>(…, Options{})` + `ready()`:
117 - upload / download (:136-147, with `fixture::write_core` at :141);
118 - reopen an existing store (:149-159);
119 - keetd writer (:161-173).
120
121 `close()` does `store->close()` (:176-180).
122 - `using Source = hc::fixture::Source` (:196).
123 - Scenario s11 ("delete every tree node through L2", :880-891): `hc::l2::CorestoreStorage st(...)` + `ready()`, `hc::l2::K32` (:885), `resumeCore`, `hc::l2::CoreTX tx = core->write(); tx.deleteTreeNodeRange(0, -1); tx.flush(); st.close()`.
124 - :959-960 only mention "from storage" in an error string, with no l2 use.
125- **Gate:** JS peers `tools/m3c-pipe/peer.js` (hypercore). The controls are cached in `m3c-controls`.
126- **ctests:** 35 × `m3c_pipe_<sc>` (CMakeLists.txt:419-431). The drill sets m5g1e, m5g2a, m5g2b, m5g2c, m5g2d, m5g3e3, m5g4e1, m5g4e2 and r3c use it as a gate.
127- **Becomes:** a rewrite of the three constructors, `close()` and the s11 block (`Sync` + `s.wait(st->resumeCore(d))`, `auto tx = core->write(); tx->deleteTreeNodeRange(0,-1); s.wait(tx->flush()); s.wait(core->close()); s.wait(st->close())`). The `Core(*store, …)` lines follow Core's new constructor.
128
129### 1.8 `repl/test/r3c_scan_test.cpp` (137 lines) and `repl/test/r3c_walk_test.cpp` (161 lines)
130- **Uses:** a stack `l2::CorestoreStorage store(dir/store, Options{})` + `store.ready()`, then `hcore::Core core(store, oo)` (scan :67-73; walk :85-91). That is the backing for a stub Core and nothing else.
131- **Gate:** vectors/data/r3c-scan.json and r3c-walk.json, generated by vectors/gen/r3c-{scan,walk}.js from hypercore 11.37's own methods.
132- **ctests:** `r3c_scan` (:410-413), `r3c_walk` (:415-418). The drill set r3c lists them as TARGET.
133- **Becomes:** a 2-line rewrite each: `auto store = std::make_shared<storage::CorestoreStorage>(…); { Sync s; s.wait(store->ready()); }`, plus Core's new constructor.
134
135### 1.9 Gates on hc::storage already (they keep working but include l2 transitively)
136
137| tool | file | ctest | JS reference | l2 dependency |
138|---|---|---|---|---|
139| hs_trace | tools/hs/hs-trace.cpp (+ hs-trace.js, gen-hs.js, hs-gate.sh, hs-dump.js) | `r4c_hs` (CMakeLists.txt:242-247) | hypercore-storage lib/ on Bare, pool=1 | via `hc/storage.h` → `hc/l2.h` and hc_storage → hc_l2 |
140| df_trace | tools/df/df-trace.cpp (+ df-trace.js, gen-df.js, df-gate.sh) | `r4c_df` (:249-254) | device-file / fd-lock / bare-fs on Bare | same |
141| r4c_oplog_test | storage/test/r4c_oplog_test.cpp | `r4c_oplog` (:256-260) | vectors/data/r4c-oplog.json (vectors/gen/r4c-oplog.js, legacy-cores.js) | same |
142| st_trace | tools/st/st-trace.cpp (+ st-trace.js, gen-st.js, st-gate.sh, make-v0cores.js) | `r4c_st` (:262-267) | CorestoreStorage / HypercoreStorage + migrations on Bare | same |
143
144These need no source change for R4c-4 beyond whatever replaces `hc/l2.h` in `storage.h`.
145
146### 1.10 Storage exercised only through `store::Corestore` / `hcore::Core` (no l2 symbol; affected through the link and API)
147- `store/test/m4c_pipe_test.cpp`, ctest `m4c_pipe` (:334-337). Drill sets events, m5g2a-store and m5g2b-store use it.
148- `tools/bee-read/main.cpp`, ctests `m5b_bee_read`, `fuzz_bee_*`, `m5c_bee_write`, `m5d_bee_reindex` (:347-366).
149- `tools/blobs-ops/main.cpp`, ctests `m5e_blobs`, `fuzz_blobs_*` (:374-384). `blobs/test/monitor_pipe_test.cpp`, `m5f_blobs_monitor` (:386-393).
150- `wakeup/test/wakeup_pipe_test.cpp`, `m5f_wakeup` (:400-403). `wakeup/test/autobee_wakeup_pipe_test.cpp`, `m5f_autobee_wakeup` (:404-407).
151- `tools/keetd-store/main.cpp` (M4c-3, built only with hyperswarm-peer, :451-453; no ctest).
152
153All of them must stay identical. They change only if Corestore's or Core's public API changes.
154
155### 1.11 Storage-adjacent tools that do NOT touch l2 (system librocksdb + schema only)
156- `tools/store-dump/store_dump.cpp` + `store_reader.h` (its own `namespace hs = hc::schema::hypercore_storage`, store_reader.h:29), `tools/store-verify/store_verify.cpp`, `tools/store-proof/store_proof.cpp`. They link `${ROCKSDB_LIB}` (`find_library(rocksdb)`, CMakeLists.txt:115-118, :124-126, :191-193) and `hc_schema_hypercore_storage`.
157- ctests `store_dump_gate`, `store_verify_gate`, `store_proof_gate`. **They are unaffected.**
158- `tools/l2/hops.js` is the JS-only R4c-0 hop measurement, cited by storage.h:3 and storage/src/tx.cpp:2. It has no C++ and survives as it is; the directory name is historical.
159
160### 1.12 l2's own tests
161There is no `l2/test/`. l2 is tested only by `m4a_storage_gate` through `l2_ops` (§1.1) and by the drill sets l2, m5g1a, m5g3a (and m5g3c through core_ops).
162
163---
164
165## 2. CMakeLists.txt: targets and ctests that depend on hc_l2
166
167### 2.1 Direct
168| line | target | relation |
169|---|---|---|
170| 219-222 | `hc_l2` (STATIC keys/db/tx/storage/view) | PUBLIC `hc_schema_hypercore_storage hc_l0 uv_a`, PRIVATE `rocksdb_static` |
171| 238-241 | `hc_storage` | `target_link_libraries(hc_storage PUBLIC hc_rn hc_l2)`. Its header includes `hc/l2.h` (storage/include/hc/storage.h:19) |
172| 268-270 | `l2_ops` | PRIVATE hc_l2 |
173| 278-282 | `hc_core` | `PUBLIC hc_l2 hc_l3 hc_repl_bits hc_ss` (:281). core/include/hc/hypercore.h:16 and core/include/hc/marks.h:7 include `hc/l2.h` |
174
175### 2.2 Transitive (everything above hc_core or hc_storage)
176- Libraries: `hc_repl` (:302-305), `hc_store` (:317-320), `hc_bee_read` (:344-346), `hc_blobs` (:369-372), `hc_wakeup` (:396-399).
177- Executables: `hs_trace`, `df_trace`, `r4c_oplog_test`, `st_trace` (via hc_storage); `core_ops`, `store_ops`, `m4c_pipe_test`, `m4d_appender`, `bee_read`, `blobs_ops`, `blobs_monitor_pipe_test`, `wakeup_pipe_test`, `autobee_wakeup_pipe_test`, `r3c_scan_test`, `r3c_walk_test`, `m3c_pipe_test`, `keetd-replicate`, `keetd-store` (via hc_core).
178
179### 2.3 ctests: 89 registered (`ctest -N` in build/). 62 depend on hc_l2.
180
181**Direct l2 gate (removed or retargeted):** `m4a_storage_gate` (:273-274). The C++ side is `l2_ops`; the JS side is `storage-ops.js` (hypercore-storage).
182
183**R4c gates on hc::storage (JS = Bare's own modules at pool=1; these are THE storage gates going forward):** `r4c_hs`, `r4c_df`, `r4c_oplog` (vectors from JS), `r4c_st`.
184
185**Gates vs JS above storage (must stay identical through R4c-4; design "Gates" 3):**
186- `m4b_core_gate` (core-ops.js / hypercore), `fuzz_core_corpus`, `fuzz_core_campaign`
187- `m4c_store_gate` (store-ops.js / corestore 7.12.6), `fuzz_store_corpus`, `fuzz_store_campaign`
188- `m4c_pipe` (JS corestore peers)
189- `m4d_gate` (JS appender / store-ops / verify.js; the LOCK, CRASH and BOUNDARY phases)
190- `m5b_bee_read`, `fuzz_bee_corpus`, `fuzz_bee_campaign`, `m5c_bee_write`, `m5d_bee_reindex` (hyperbee2 2.18.0)
191- `m5e_blobs`, `fuzz_blobs_corpus`, `fuzz_blobs_campaign`, `m5f_blobs_monitor`, `m5f_wakeup`, `m5f_autobee_wakeup`
192- `r3c_scan`, `r3c_walk` (vectors from hypercore 11.37)
193- `m3c_pipe_s0 … m3c_pipe_s20b` (35)
194
195**Lint:** `drill_lint` (:293). It fails on any stale anchor (§3). At HEAD it passes, 879/879 (run 2026-10-07). The saved `ctest-r4c3b3.log` shows it failing at 21:28:41, and drill.py was modified 21:28:55, so that failure was transient.
196
197**Unaffected (27):**
198- l0 ×4, l3_verifier, m3a_wire, m3b_merkle, m5g1b_merkle, m3c_caps, schema_hypercore_storage
199- store_dump_gate, store_verify_gate, store_proof_gate
200- r1_rt, r4a_js, r4c_scaffold
201- m5g4e3a_noise, m5g4e3c_interop, m5g4e3b_duplex, m5g4e3c_secret_stream, protomux
202- m3c_bitfield, m3c_wants, m3c_wants_small, r3c_random_iterator, r4b_rn, m5a_codec
203
204### 2.4 CMake edits R4c-4 needs
205- Delete the `hc_l2` block (:218-222). The header comment at :30-34 ("link it PRIVATE so only L2 compiles without RTTI") is about l2 and must be reworded for rn.
206- `hc_storage`: replace `hc_l2` with `hc_schema_hypercore_storage` (PUBLIC; the codecs used to come through hc_l2). `hc_l0` + `uv_a` already arrive through `hc_rn`. Add the moved keys source (§4) to its sources.
207- `hc_core`: `hc_l2` → `hc_storage` (:281).
208- `l2_ops` + `m4a_storage_gate` (:268-274): delete, or retarget to `hc_storage` (§1.1).
209- Comments at :236-237 ("built beside the synchronous hc_l2 (frozen until R4c-4…)") go.
210
211---
212
213## 3. Drills
214
215### 3.1 Machinery
216- `tools/m5e/drill.py` (161 KB) holds every set. `SETS` is at :1097-1145: `name: (MUTANTS, FILES, TARGET, GATE)`. A mutant is `(file, label, anchor, replacement)` and must match exactly once. The drill builds `TARGET` with `npx cmake --build build --target …` (≈:1185) and runs `GATE`, which is a gate script or a named mode (EVENTS / M3C / M4C / R4C* …, ≈:1194-1240).
217- `tools/m5e/drill_lint.py` (21 lines) execs the tables and requires every anchor to count == 1 in its file. It runs as ctest `drill_lint`. If a file is missing, `open()` raises, so deleting `l2/` without editing the tables fails ctest.
218- There is NO `run_all.py` or `compare_runs.py` in the tree (`find . -name run_all.py -o -name compare_runs.py`: nothing). Other drill scripts are `tools/m5b/drill.py` and `tools/m5c/drill.py`; neither references l2 or storage.
219- Every set's mutants were counted per location by a script that execs the tables like drill_lint (scratchpad `drills.py`). "Call-site" means an anchor in core/store/repl whose anchor or replacement text contains a storage API token (`l2::`, `hs::`, `storage`, `tx`, `rx`, `CoreTX`, `.read()`, `.write()`, `Atom`, `flush`, the tx/rx/store method names, …). This is a heuristic for "lines R4c-4 rewrites", listed below so each one can be checked.
220
221### 3.2 Sets with anchors in `l2/` (they must move to `storage/` or `rn/`, and change gate)
222
223| set | mutants | in l2/ | per file | TARGET / GATE today | after R4c-4 |
224|---|---|---|---|---|---|
225| `l2` (M5e-1, snapshots) | 5 | 5 | db.cpp 2 (:188 point reads ignore snapshot, :254 iterators ignore snapshot); storage.cpp 3 (:183, :200, :213) | `l2_ops` / `tools/m4/storage-gate.sh` | the db.cpp snapshot mutants move to rn (snapshot read options in rn/src/native.cpp / rn's Snapshot); the storage.cpp ones to storage/src/hypercore_storage.cpp; gate becomes st (or the converted l2_ops) |
226| `m5g1a` (View, Atom, tree cache, deps) | 34 | 34 | view.cpp 19, tx.cpp 6, storage.cpp 9 | `l2_ops` / storage-gate.sh | storage/src/view.cpp (View, Overlay, Atom, TreeCache), storage/src/tx.cpp (CoreTX/CoreRX/tree cache), storage/src/hypercore_storage.cpp (deps, atomize, snapshot, close); gate hs and/or st |
227| `m5g3a` (sessions, dep streams, store ops) | 30 | 30 | storage.cpp 30 | `l2_ops` / storage-gate.sh | storage/src/hypercore_storage.cpp (sessions, export, isParentStorage), storage/src/streams.cpp (BlockDependencyStream), storage/src/corestore_storage.cpp (deleteCore, audit, clear, setRecovering, allocData); gate st |
228| `m5g3c` (async atom flush, atomic sessions) | 39 | 6 | view.cpp 6 (:265, :275 ×2, :293, :308, :312: flushing / flushed() / rejection handling) | `core_ops` / `tools/m4/core-gate.sh` | the 6 move to `storage::Atom` (storage/src/view.cpp:383+); the gate stays core-gate (13 more anchors are call-site, §3.3) |
229
230**Total: 75 anchors in `l2/`** (storage.cpp 42, view.cpp 25, tx.cpp 6, db.cpp 2).
231
232The full l2 anchor list, with labels and l2 line numbers, is reproducible with `python3 <scratchpad>/drills.py detail`. Labels per set:
233- **l2:** point reads ignore the snapshot; iterators ignore the snapshot; a snapshot of a snapshot takes a new one; tx allowed on a snapshot; streams ignore the snapshot.
234- **m5g1a, view.cpp:** attached snapshot not followed; a frozen view still links its snapshot; snapshot not counted as a reader; readStop never releases the parent; cleared reads not dropped; cleared read end inclusive; a range delete leaves mapped keys; no copy-on-write under readers; overlay keeps db entries in cleared ranges; overlay emits a deleted key from the db; overlay loses trailing changes; overlay not reversed for reverse streams; map lookup ignored; atom flush skips preflushes; atom flush not refused while flushing; atom view not reset; atom flush writes nothing; cache never rotates; cache get from oldest does not promote.
235- **m5g1a, tx.cpp:** atom tx flush answers true; pending entries never settle; missing nodes stay cached; repeats counted as hits; fork -1 goes through the cache; cache key ignores the fork.
236- **m5g1a, storage.cpp:** atomized reads use the cache; atomize allows a new atom; atom writes go to the db; replace-all makes a new core object; update path mutates the shared core; findDependencyIndex truncated off by one; getDependency inclusive; snapshot reads the live view; close keeps the reader.
237- **m5g3a, storage.cpp:** resumeSession ignores the dependency; createSession allows a head ahead of the core; createSession reuses the core data pointer; a reused session is not nuked; createSession sets no head; createSession sets no dependency; createSession depends at length 0; createSession ignores the stored head; addDependency keeps longer parents; addDependency clips its own pointer; addDependency appends a duplicate; atomic session writes the head to the db; deleteSessions keeps the record; deleteSessions keeps the data; deleteSessions with none does not throw; block stream ignores dependencies; reverse dependency stream reads every dependency; dependency stream start not clipped; dependency stream end not clipped; export keeps the key pair; export ignores batches; export ignores dependencies; isParentStorage compares the first dependency; deleteCore keeps the store record; deleteCore keeps the sessions data; audit keeps linked; audit rewrites v1 manifests; clear does nothing; setRecovering stamps nothing; allocData does not persist the head.
238- **m5g3c, view.cpp:** flushing cleared before the handlers; flushed() waiters never run; flushed() waiters run after done; a rejection waits for every handler; a rejection inside the loop is dropped; flushed() resolves at once while flushing.
239
240Many of these behaviours already have NEW-code mutants in `r4c2` (20, on hs) and `r4c3b3` (19, on st), whose anchors are in storage/. Re-anchoring should drop exact duplicates and keep the rest. The design's R4c-5 "re-anchored onto the new code" covers this. Every re-anchored mutant has to be re-run red against its new gate.
241
242### 3.3 Sets with anchors on core/store/repl storage call sites (stale if R4c-4 rewrites those lines)
243
244The heuristic counts, with line numbers at HEAD:
245
246| set | mutants | call-site anchors | where | TARGET / GATE |
247|---|---|---|---|---|
248| core | 18 | 3 | session.cpp:1405 (snapshot setUserData refuses before comparing: anchor is `state_->storage->read().getUserData(key)`); session_state.cpp:404 (clear keeps the stored hints: `tx.setHints`), :409 (clear range end off by one: `tx.deleteBlockRange`) | core_ops / core-gate |
249| events | 16 | 1 | session_state.cpp:366 (no append on a replicated upgrade) | m4c_pipe_test / EVENTS |
250| m5g1c | 29 | 6 | session_state.cpp:545, :610, :616, :618, :627, :630 (truncate: signs / old blocks / group index / head / bitfield / hints) | core_ops / core-gate |
251| m5g1d | 34 | 6 | hypercore.cpp:1163 (verifyReorg stores manifest), :1244 (conflict auth keyPair), :1312 (repair re-proof reads live store), :1323 (repair atom never flushed); session_state.cpp:601; session.cpp:1382 (recover puts nothing) | core_ops / core-gate |
252| m5g1e | 29 | 1 | repl/src/replicator.cpp:566 (proofs served without the tree cache) | m3c_pipe_test / M3C |
253| m5g3b | 16 | 5 | session_state.cpp:414, :770, :771, :790; session.cpp:492 | core_ops / core-gate |
254| m5g3c | 39 | 13 (+6 in l2/) | session_state.cpp:118, :171, :196, :231, :786, :796, :798, :802, :866, :873; session.cpp:271, :282, :492 | core_ops / core-gate |
255| m5g3d | 37 | 17 | session_state.cpp:1113, :1116, :1138, :1180, :1184, :1200, :1238, :1260, :1262, :1264, :1267, :1323, :1325, :1337, :1341, :1348, :1349 (overwrite / commit / catchup) | core_ops / core-gate |
256| m5g3e1 | 16 | 3 | hypercore.cpp:633, :661, :788 (prologue) | core_ops / core-gate |
257| m5g3e2 | 23 | 3 | session_state.cpp:1047, :1053, :1085 (moveTo) | core_ops / core-gate |
258| m5g4a | 25 | 1 | hypercore.cpp:802 (setGroup writes no record) | core_ops / core-gate |
259| m5g4d | 20 | 7 | marks.cpp:38, :44, :51; session.cpp:1098; hypercore.cpp:869, :881, :889 (marks, audit) | core_ops / core-gate |
260
261**Total: 66 call-site anchors in core + 1 in repl (+ events' 1 in core).** The heuristic can over- or under-count. The authoritative check after the rewrite is `drill_lint` plus re-running each affected set red. Sets with 0 call-site anchors (m5g2a/b/c/d, m5g2a-store, m5g2b-store, m5g3e3, m5g4b, m5g4c, m5g4e1, m5g4e2, r3c, …) can still go stale if a rewrite touches neighbouring text that an anchor includes. drill_lint will show that.
262
263### 3.4 Sets on hc::storage today (no l2 anchor, but storage/ files R4c-4 edits)
264
265| set | mutants | storage/ files | TARGET / GATE |
266|---|---|---|---|
267| r4c2 | 20 | view.cpp, tx.cpp, streams.cpp, storage.h | hs_trace / hs-gate |
268| r4c3a | 19 | device_file.cpp, bare_fs.cpp (+1 ss/include/hc/basic_duplex.h) | df_trace / df-gate |
269| r4c3b2 | 11 | migrations0.cpp | r4c_oplog_test |
270| r4c3b3 | 19 | corestore_storage.cpp, hypercore_storage.cpp, migrations0.cpp | st_trace / st-gate |
271
272Per-file anchor counts in storage/: storage.h 1, bare_fs.cpp 6, corestore_storage.cpp 9, device_file.cpp 12, hypercore_storage.cpp 3, migrations0.cpp 18, streams.cpp 3, tx.cpp 6, view.cpp 10.
273
274Moving the keys (`namespace keys = l2::keys`, streams.cpp:12; `l2::keys::…` in tx.cpp:70-77 etc.) and the aliases in storage.h:34-38, :65-66, :153 can invalidate those anchors that include these lines. The r4c2 anchor in storage.h is the one most at risk.
275
276---
277
278## 4. Keys and schema codecs: where they live and who uses them
279
280### 4.1 `l2::keys` (lib/keys.js)
281- Declared at l2/include/hc/l2.h:43-79. Defined in `l2/src/keys.cpp` (121 lines), which includes `hc/l2.h` (for `Buf`, `K32`, `hs::CorestoreAlias`) and `hc/index_encoder.h` (hc_l0).
282- **Users outside l2:**
283 - `storage/src/streams.cpp`: `namespace keys = l2::keys;` (:12); 25 uses (:63, :133-262).
284 - `storage/src/tx.cpp`: `l2::keys::core_*` in every CoreTX setter (:70-77 …, about 55 `l2::` refs).
285 - `storage/src/hypercore_storage.cpp`: :147-148 (compactRange keys), :312-313 (deleteSessions range).
286 - `storage/src/migrations0.cpp`: :1099 (`l2::keys::core_auth`).
287 - `storage/src/corestore_storage.cpp`: 3 `l2::` refs.
288- core/store/repl/tools: NO direct `keys::` use.
289- **Must survive.** It could go to `storage/src/keys.cpp` + a header `storage/include/hc/storage_keys.h` (or inside storage.h), namespace `hc::storage::keys`. Then `namespace keys = l2::keys` → `storage::keys` and the `l2::keys::` qualifiers drop. It is a pure move, so its outputs are byte-identical; the r4c_hs and r4c_st dumps prove it.
290
291### 4.2 Schema codecs (`hs::`)
292- They live in `schema/generated/hypercore_storage.{h,cpp}` + `schema/external/hypercore_storage_external.cpp`. That is the library `hc_schema_hypercore_storage` (CMakeLists.txt:101-112), with ctest `schema_hypercore_storage`, and it is independent of l2.
293- l2 only provides the alias: `namespace hs = hc::schema::hypercore_storage;` (l2.h:35), via `#include "hypercore_storage.h"` (l2.h:19). `hc_l2` links the library PUBLIC (:221), which is how hc_storage and hc_core get it today.
294- **Alias users that must be retargeted:**
295 - storage/include/hc/storage.h:34 `namespace hs = l2::hs;`
296 - core/include/hc/hypercore.h:54, :334-335 (`l2::hs::CorestoreAlias`, `to_stored` / `from_stored`)
297 - core/include/hc/session_state.h:34
298 - core/src/{hypercore.cpp:24, :598, :606, :670, :813-853; session_state.cpp:15; session.cpp:1382; fully_remote_proof.cpp:14}
299 - store/src/corestore.cpp:332, :357, :418
300 - tools/core-ops/main.cpp:25, tools/store-ops/main.cpp:18, tools/core-fixture/fixture.h:41, :60, :64
301
302 After the move these become `hc::storage::hs` (defined in storage.h) or `hc::schema::hypercore_storage`.
303- **`encode_core_auth` / `encode_core_head`** (l2.h:518-520, l2/src/tx.cpp:43-44) are used ONLY inside l2 (tx.cpp:68, :70). They die with l2. NEW already has `storage::encodeAuth` / `encodeHead` (storage.h:279-281, storage/src/tx.cpp:54-55).
304
305### 4.3 Other l2 types hc::storage borrows (storage.h currently includes hc/l2.h, :19)
306- `K32` (storage.h:36), `Change` (:38), `Dependency` (:65), `CorePtr` (:66), `TreeCacheStats` (:153).
307- To survive, they must be DEFINED in storage.h, copied verbatim from l2.h:37, :82-86, :254-259, :235.
308- Their outside users: core (`l2::Dependency` in session_state.h:185, :202, :205 and session_state.cpp:231, :412, :622, :637-640, :873, :1261, :1333; `l2::Buf` in marks.h:16; `l2::CreateCoreArgs` in hypercore.cpp:334), tools (`l2::CreateCoreArgs` in fixture.h:32), repl/test (`l2::K32` in m3c_pipe_test.cpp:885).
309- `Buf` is already `rn::Buf` in storage (:35). l2's `Buf`, `K32` and `Error` are otherwise only used inside l2.
310- **Deleted with l2:** `Db`, `DbSnapshot`, `test_after_write`, l2's `View` / `Atom` / `TreeCache` / `TreeCacheEntry`, `DeviceFile`, `CoreTX` / `CoreRX`, `ExportData` / `CoreExport`, `blockStream`, `HypercoreStorage`, `StoreHead`, `Info`, `CreateCoreArgs`, `CorestoreStorage`, `Error`. NEW has its own `StoreHead`, `Info`, `CreateCoreArgs`, `ExportData`, `CoreExport` (corestore_storage.h:318-404).
311 - Behavioural notes: NEW's `CoreExport` adds `hasAuth` (:340) and its blocks are `BlockEntry`/`PageEntry`, not pairs (:333-336). NEW's `Info` drops `hasAuth` / `hasHead` / `hasHints` (l2.h:446 vs corestore_storage.h:386-392) and NEW takes `InfoOptions`.
312
313---
314
315## 5. Has the M4a storage gate become the async trace gate? No
316
317| | tools/l2-ops + tools/m4 storage gate | tools/st (R4c-3b-3) | tools/hs (R4c-2) |
318|---|---|---|---|
319| C++ under test | `hc::l2` (sync) via `l2_ops` | `hc::storage` CorestoreStorage / HypercoreStorage via `st_trace` | `hc::storage` lib/ layer (View, Atom, tx, streams) via `hs_trace` |
320| JS reference | `storage-ops.js`: hypercore-storage 3.3.0, result lines only | `st-trace.js`: same module, Bare at `UV_THREADPOOL_SIZE=1`, every settlement + ruler hops | `hs-trace.js`: hypercore-storage lib/, Bare at pool=1 |
321| Runtimes | none (sync) | rt real AND virtual | rt real AND virtual |
322| Comparison | result lines; `store-dump/dump.js` record hashes (`versions` popped); RocksDB OPTIONS; cross-open of the other's `basic` store | full trace (+ file listing, rc); `hs-dump.js` hex dump of the `corestore` CF | trace; `hs-dump.js` hex dump |
323| Scripts | basic, ranges, reopen, cross, snapshots, atoms, treecache, deps, sessions, storeops (gen-storage-ops.js:506) | storeOps, groupCommit, coreOps, fresh, primaryKey (fixture pk), legacy (hypercore-10), v0cores, v0coresCreate, hops (gen-st.js:26-161) | getters, rxLifecycle, treeCache, coreTx, deletes, viewReads, viewSnapshots, atomFlush, streamsPlain, overlay, overlayTail, overlayMany, dependencies, treeDependencies, treeHops, dependencyBounds, store, closed (gen-hs.js:30-464) |
324| ctest | `m4a_storage_gate` (CMakeLists.txt:268-274) | `r4c_st` (:262-267) | `r4c_hs` (:242-247) |
325
326**What M4a covers that st/hs do not (to port before M4a is deleted, or to keep by retargeting l2_ops):**
327- **Cross-open:** each implementation opens the OTHER's store writable, verifying the device file and taking the lock (storage-gate.sh:37-48). st only opens fixtures without their CORESTORE file (st-gate.sh:5-7, :22). df covers the device file at the lib level, not across implementations.
328- **RocksDB OPTIONS-file identity** (storage-gate.sh:31-34). Neither st-gate.sh nor hs-gate.sh checks it. core-gate and store-gate do check OPTIONS, but only through Core / Corestore.
329- **Reopen persistence** (the `reopen` script): close then open the same store mid-script. st scripts open once.
330- **Stream range variants** (the `ranges` script: gt / gte / lte / lt / reverse per stream, and `locals` with a range). st's `stream` op takes a `range`, but no st script passes one, and hs covers ranges only at the lib level.
331- **Direct dependency calls** `getDependencyLength` / `getDependency` / `findDependencyIndex` (the `deps` script, l2-ops main.cpp:223-236). st has only `depHead` / `depLength`.
332- **`isParentStorage`, `coreOf`.** No st op exists for either.
333- **Atom re-entry through an onflush handler** (`atomHooks` with `reenter`, l2-ops :195-210, "Atom already flushing"). hs has onflush / preflush / flushed at the lib level; st's `atomFlush` has no hooks.
334- **`setRecovering` with a pinned time.** st-trace supports the op (st-trace.js:173, st-trace.cpp:275) but no gen-st script uses it.
335- **`readOnly` opens.** M4a `open` takes `readOnly`, and st-trace.cpp:242 supports it, but no script sets it.
336
337**Items from the design's list of new concurrency scripts that are still missing in st** (supported by the harness, used by no script):
338- `onresume` and `alwaysRecover` (st-trace.cpp:244, :246);
339- `_maybeRecover`;
340- a close during a read (CloseErrorStream). hs has a `closed` script at the lib level, but there is nothing at the store level.
341
342**Conclusion:** R4c-3b-3 delivered the async trace gate as `tools/st`, a new gate, not as a converted `l2_ops`. "The existing scripts keep their outputs" (the design's M4a promise) is not yet met on hc::storage.
343
344R4c-4 must either:
345- port M4a's 10 scripts plus cross-open plus OPTIONS into st (st-gate.sh, gen-st.js), then delete `l2_ops` / `storage-ops.js` / `gen-storage-ops.js` / `storage-gate.sh` / `m4a_storage_gate` and retarget the l2 / m5g1a / m5g3a drills to `st_trace` / `tools/st/st-gate.sh`; or
346- rewrite `l2_ops` over hc::storage, which keeps `m4a_storage_gate` and those drills' gate.
347
348The design text (R4c "Gates" 1) describes the second option. The code so far went the first way.