| … | … | @@ -0,0 +1,783 @@ |
| 1 | # R4c-4d inventory — core/ off hc::l2 onto hc::storage, in JS's async shape |
| 2 | |
| 3 | Worktree `hyper-fleeky/.scope/keetd`, branch r4, HEAD **bb707dd** (R4c-4c-4). Written 2026-10-07 for the R4c-4d step |
| 4 | (CPP-PORT-DESIGN.md §9.4, "R4c-4 DESIGN", "R4c-4c DONE"). |
| 5 | |
| 6 | **Read in full for this file:** |
| 7 | - C++, every file under `core/`: `core/src/{session_state,hypercore,session,marks,fully_remote_proof,streams,requests}.cpp` |
| 8 | and `core/include/hc/{hypercore,session,session_state,marks,fully_remote_proof,streams,requests,merkle_l2}.h`. |
| 9 | `value_encoding.*` and `charset.c` have no storage use (checked by grep only). |
| 10 | - Supporting C++: `rt/include/hc/scaffold.h`, `storage/include/hc/{storage,corestore_storage}.h` (the API), |
| 11 | `storage/src/tx.cpp` (`CoreTX::flush`, `CoreRX::tryFlush/destroy/free_`), `storage/src/hypercore_storage.cpp` |
| 12 | (`snapshot`, `compact`, `atomize`, `read`, `write`, `close`), `l3/include/hc/merkle.h` (`Rx` / `TreeStorage` / |
| 13 | `SessionView`), `tools/mt/mt-trace.cpp:92-118`, `tools/m4d/appender.cpp`, `l2/src/db.cpp:220-241`, |
| 14 | `rn/src/native.cpp:264-284`. |
| 15 | - JS, in full: hypercore 11.37.0 `lib/session-state.js`, `lib/core.js`, `lib/mutex.js`, `lib/audit.js`, |
| 16 | `lib/copy-prologue.js`, `lib/mark-bitfield.js`, `lib/fully-remote-proof.js`. `index.js` was read at 200-620, |
| 17 | 700-1260 and 1340-1400 (every async method of Hypercore). `lib/bitfield.js` was read at 515-549 (clear / open). |
| 18 | |
| 19 | **Abbreviations.** |
| 20 | - C++: **SS** `core/src/session_state.cpp`, **SSh** `core/include/hc/session_state.h`, **HC** `core/src/hypercore.cpp`, |
| 21 | **HCh** `core/include/hc/hypercore.h`, **SE** `core/src/session.cpp`, **SEh** `core/include/hc/session.h`, |
| 22 | **MK** `core/src/marks.cpp`, **MKh** `core/include/hc/marks.h`, **FRP** `core/src/fully_remote_proof.cpp`, |
| 23 | **FRPh** `core/include/hc/fully_remote_proof.h`, **ML2** `core/include/hc/merkle_l2.h`. |
| 24 | - JS: **ss** session-state.js, **cj** core.js, **ix** index.js, **mx** mutex.js, **au** audit.js, **cp** copy-prologue.js, |
| 25 | **mb** mark-bitfield.js, **frp** fully-remote-proof.js, **bf** bitfield.js. |
| 26 | - New API (`namespace hc::storage`): **S:** `storage/include/hc/storage.h`, **CS:** `storage/include/hc/corestore_storage.h`. |
| 27 | `P<T>` = `js::Promise<T>`; `HSP` = `HypercoreStoragePtr`; `RX` = `CoreRXPtr`; `TX` = `std::shared_ptr<CoreTX>`. |
| 28 | |
| 29 | **Correction to the earlier inventory.** Every `mb:` line in `r4c4-inv-core.md` is wrong by +161. mark-bitfield.js |
| 30 | has 112 lines. The old numbers came from a concatenated listing that started with audit.js (161 lines). The right |
| 31 | lines are MarkPage 10-30 (`loaded` promise 13-15), `loadPage` 42-50 (the page goes into `_pages` at 43, BEFORE the |
| 32 | read at 44-47), `get` 52-60, `set` 62-79 (`await p.loaded` 71, `mutex.lock` 73, `tx.flush` 76, `unlock` 77), |
| 33 | `clear` 81-86, `createMarkStream` 88-111. The `au:` numbers in the old file are right. |
| 34 | |
| 35 | --- |
| 36 | |
| 37 | ## A. Every remaining l2 use in core/ (at bb707dd) |
| 38 | |
| 39 | ### A0. Exact hc::storage signatures (the right-hand column below refers to these) |
| 40 | |
| 41 | | what | signature | where | |
| 42 | |---|---|---| |
| 43 | | read batch | `CoreRXPtr HypercoreStorage::read (int64_t fork = -1)` | CS:95 | |
| 44 | | getters | `P<std::optional<hs::CoreAuth>> getAuth ()`, `P<…CoreHead> getHead ()`, `P<…CoreDependency> getDependency ()`, `P<…CoreGroup> getGroup ()`, `P<…CoreHints> getHints ()`, `P<std::optional<Buf>> getBlock (int64_t)`, `P<std::optional<Buf>> getBitfieldPage (int64_t)`, `P<std::optional<hs::CoreTreeNode>> getTreeNode (int64_t)`, `P<bool> hasTreeNode (int64_t)`, `P<std::optional<Buf>> getUserData (const std::string &)`, `P<std::optional<Buf>> getMark (int64_t)` | S:203-214 | |
| 45 | | batch end | `void CoreRX::tryFlush ()`, `void CoreRX::destroy ()`. Both readStop the view (`free_`, tx.cpp:253-267). An rx that gets neither leaks a view reader | S:216-217 | |
| 46 | | write batch | `std::shared_ptr<CoreTX> HypercoreStorage::write ()` (`atom ? view : nullptr`, hypercore_storage.cpp:332-334) | CS:96 | |
| 47 | | mutators | `void setAuth/setHead/setDependency/setGroup/setHints (const hs::…&)`, `putBlock (int64_t, const Buf&)`, `deleteBlock`, `deleteBlockRange (int64_t, int64_t)`, `putBitfieldPage`, `deleteBitfieldPage`, `deleteBitfieldPageRange`, `putTreeNode (const hs::CoreTreeNode&)`, `deleteTreeNode`, `deleteTreeNodeRange`, `putUserData (const std::string&, const Buf&)`, `deleteUserData`, `putMark`, `deleteMarkRange`, `putGroupUpdate (uint64_t, double, const Buf&)`, `deleteGroupUpdate (uint64_t, double)` — all `void`, as JS | S:158-184 | |
| 48 | | write flush | `P<bool> CoreTX::flush ()`. NOT a coroutine: it takes `changes` and returns `View::flush(c, db)` or `js::resolve(false)` on an atom view, so the TX need not outlive the call (tx.cpp:101-113) | S:185 | |
| 49 | | storage | `bool readOnly () const`, `const std::vector<Dependency> &dependencies () const`, `void setDependencyHead (const std::optional<Dependency>&)`, `int64_t findDependencyIndex (uint64_t, bool) const`, `bool snapshotted () const`, `HSP snapshot ()`, `P<std::vector<js::Undefined>> compact ()`, `HSP atomize (const AtomPtr&)`, `AtomPtr createAtom ()`, `Stream createBlockStream / createBitfieldStream / createUserDataStream / createMarkStream (const IndexRange& / UserDataRange&)`, `P<HSP> resumeSession (std::string)` (null when none), `P<HSP> createSession (std::string, std::optional<hs::CoreHead>)`, `P<HSP> createAtomicSession (AtomPtr, std::optional<hs::CoreHead>)`, `P<void> close ()` (readStop + `db->close()`), `static bool isParentStorage (const HS&, const HS&)`; members `CorestoreStorage *store`, `std::shared_ptr<CorePtr> core`, `AtomPtr atom` | CS:62-100 | |
| 50 | | store | `P<HSP> resumeCore (std::optional<K32>)`, `P<HSP> createCore (CreateCoreArgs)`, `P<std::pair<K32,uint64_t>> createGroup (K32)`; `CreateCoreArgs` | CS:191, 195, 197; 111-120 | |
| 51 | | Atom | `bool flushing`, `P<void> flushed ()`, `static P<void> flush (std::shared_ptr<Atom>)`, `preflush (std::function<void ()>)`, `onflush (Handler)` where `Handler = std::function<P<void> ()>` | S:108-125 | |
| 52 | | stream value | `storage::mapped<BlockEntry/PageEntry/UserDataEntry>(entry)`; `IndexRange{ gte, lt, reverse }` | S:51-56, 258 | |
| 53 | |
| 54 | **No bridge exists in core/ yet.** `hc::storage::CoreRX` does not implement `merkle::Rx`: the index is `int64_t`, not |
| 55 | `uint64_t`, and l3 does not link storage. R4c-4d needs a permanent `merkle::Rx` / `merkle::TreeStorage` over |
| 56 | `CoreRXPtr` / `HSP` in core, in place of ML2's L2Rx / L2TreeStorage. `tools/mt/mt-trace.cpp:92-105` already has it |
| 57 | (`StorageRx`, `StorageTree`), and its getters return the storage promise itself (one node type since 4c-1, so there is |
| 58 | no extra hop). `Session::proof` and the replicator also need the bridge to expose the underlying rx's `getBlock`, so |
| 59 | the block lands in the proof's batch. |
| 60 | |
| 61 | ### A1. session_state.cpp / session_state.h |
| 62 | |
| 63 | "Old #" refers to `r4c4-inv-core.md`. **gone** means the row no longer exists. **changed** means it is still there in a |
| 64 | different shape, usually since R4c-4c. |
| 65 | |
| 66 | | file:line | C++ fn | l2 call / type now | hc::storage equivalent | JS (await?) | old # / status | |
| 67 | |---|---|---|---|---|---| |
| 68 | | SSh:34, SS:17 | ns | `namespace hs = l2::hs` | `storage::hs` (S:34 area alias; R4c-4a made l2 alias storage_keys) | — | type | |
| 69 | | SSh:47, SS:99 | ctor | `std::shared_ptr<l2::HypercoreStorage>` param | `HSP` | ss:21 | B-row | |
| 70 | | SSh:54 | member `storage` | `std::shared_ptr<l2::HypercoreStorage>` | `HSP` | ss:25 | B-row | |
| 71 | | SSh:60 | member `atomized` | `std::shared_ptr<l2::Atom>` | `AtomPtr` | ss:34 | B-row | |
| 72 | | SSh:75 | `isSnapshot` | `storage->snapshotted()` | `HS::snapshotted()` | ss:61-63 sync | #2 same | |
| 73 | | SS:120 | `isAtomicDefault` | `storage->atom()` (accessor) | member `storage->atom` | ss:69-71 sync | #1 same | |
| 74 | | SS:141 | `flushedLength` | `storage->dependencies()` | `HS::dependencies()` | ss:92-97 sync | #3 same | |
| 75 | | SS:151-154 | `unref` | `close()` (sync) | `close()` → `P<void>`, fired with a handled no-op catch | ss:104-107 **not awaited** (`.catch(noop)`) | #216 | |
| 76 | | SS:168-185 | `close` | no storage close; `lingers.clear()` (SS:184) | `const P<void> closing = storage->close()` before the sessionStates pop (ss:195), `co_await` each linger's `close()` (ss:203-205), `co_return co_await closing` (ss:207). `mutex.destroy(Error("Closed"))` returns a promise in JS (mx:33-46, `.catch(noop)` ss:192) | ss:188-208 async | #4 same (still the H7 deviation) | |
| 77 | | SS:116 | `~SessionState` | `close()` | the design says destructors close nothing. The bookkeeping (sessionStates swap-pop) must stay, because a raw `SessionState*` is in `core->sessionStates` | — (GC) | #142-class | |
| 78 | | SS:187-195 | `snapshot` | `storage->snapshot()` → shared_ptr; callback `done` | `HSP HS::snapshot()`; `P<std::shared_ptr<SessionState>>` | ss:210-215, `await _updateSnapshotStorage` (212) | #5-6 same | |
| 79 | | SS:197-224 | `updateSnapshotStorage_` | `atomized->flushing`, `atomized->flushed(fn)` weak loop | `while (atomized->flushing) co_await atomized->flushed()` (keep the null check SS:205-207, JS reads `null.flushing`) | ss:129-130 await in loop | #7-8 same | |
| 80 | | SS:226-253 | `readSnapshotTree_` | `st.read()`; `getHead/getAuth/getDependency`; `tryFlush`; `setDependencyHead`; 2nd `read()`; `getTreeNode` per root **consumed before** `rx2.tryFlush()`, throws mid-loop (SS:239-248) | issue head/auth/dep, `tryFlush`, `co_await allOf` (ss:132-138); `setDependencyHead(dep)` (140); 2nd rx: all roots, `tryFlush`, `co_await all`, THEN the null check (145-167). JS reads `auth.manifest` with no null guard (173) | ss:132-176 Promise.all ×2 | #9-18 same (H4) | |
| 81 | | SS:255 | `sessionStorage` (merkle::SessionView) | `make_shared<L2TreeStorage>(storage)` (ML2) | `StorageTree(storage)` bridge (A0) | mt reads `session.storage` | **new since 4c-3** (replaces old #19 `get_tree_node`, **gone**) | |
| 82 | | SS:257 | `getBlock` | `storage->read().getBlock(i)` temp rx, **no tryFlush** | `readBlock(rx, i)`: `getBlock`, `tryFlush`, return the promise | ix:1386-1390; awaited ix:945/956/965 | #20 same | |
| 83 | | SSh:106, SS:266-270 | `createWriteBatch` | `l2::CoreTX&`, `activeTx_.emplace(storage->write())`, `storage->snapshotted()` | `TX` (keep the returned shared_ptr; `activeTx_` becomes `TX`) | ss:236-241 sync | #21-22 same | |
| 84 | | SSh:213 | member `activeTx_` | `std::optional<l2::CoreTX>` | `TX` | ss:53 | H12 | |
| 85 | | SS:259-264, SSh:111 | `updateGroupIndex` | `tx.deleteGroupUpdate`, `tx.putGroupUpdate` | same, void | ss:223-230 sync | #24 same | |
| 86 | | SS:284-290 | `flush` | `std::optional<l2::CoreTX> tx = move(activeTx_)`; `tx->flush()` → bool | `P<bool>`: `TX tx = activeTx_; activeTx_ = nullptr; try { if (!co_await tx->flush()) co_return false; } finally-equivalent clearActiveBatch; lastTruncation = null; co_return true` | ss:259-271 async (`await tx.flush()` 264) | #23 same | |
| 87 | | SS:292-306 | `locked_` | callback mutex, synchronous `body()` | gone: each op is `co_await mutex.lock(); try {…} finally { _unlock(); }` | the pattern | #25 same (H3) | |
| 88 | | SS:308-316 | `setUserData` | `createWriteBatch`, `putUserData`/`deleteUserData`, `flush()` | `co_await flush()`. JS returns `await this.flush()` (bool) | ss:347-358 | #26-28 same | |
| 89 | | SS:318-359 | `verifyBlock` | idle-assert throw (SS:319); `core->downgrade(batch)` (Sync inside, HC:1071); `createWriteBatch`; `tx.putBlock`; `storeBitfieldRange_`; `core->setManifest_(tx…)`; `commitBatch`; `tx.setHead`; `updateGroupIndex`; `flush()`; `core->flushHints()` | `P<bool>`, real `co_await mutex.lock()`; `co_await batchDowngrade(tb)` (one rx); `co_await storeBitfieldRange`; `co_await flush()`; `co_await core->flushHints()` | ss:360-425: lock 361, downgrade 365, preupdate 373 (unset), storeBitfieldRange 383, flush 407, flushHints 419 | #29-39 **changed**: the downgrade is now one rx via l3 + Sync (old: per-root `get_tree_node`) | |
| 90 | | SS:361-414 | `clear` | `createWriteBatch`, `tx.deleteTreeNode ×2`, `storeBitfieldRange_`, `tx.setHints`, `tx.deleteBlock`/`deleteBlockRange`, `updateDependency_`, `flush()`, `storage->setDependencyHead`, `core->flushHints()` | as A0; `co_await` storeBitfieldRange / flush / flushHints | ss:558-612: lock 559, storeBitfieldRange 581, flush 595, flushHints 607 | #40-48 same | |
| 91 | | SS:416-508 | `append` | `waitForRecovery(cb)` re-entry with `AppendOpts::recovered` (SSh:215); `createWriteBatch`; `setManifest_`; `opts.preappend` (sync); `flush()` (empty); `commitBatch`; `tx.setHead`; `storeBitfieldRange_`; `tx.setHints`; `updateGroupIndex`; `tx.putBlock ×n`; `flush()`; `opts.postappend` (sync); `flushHints` | `co_await waitForRecovery()` inline (the `recovered` flag goes); preappend / postappend awaited (hop parity: ix:1359-1366 awaits each `encrypt`) | ss:661-754: waitForRecovery 664, lock 666, preappend 678, flush 681, storeBitfieldRange 713, flush 736, postappend 744, flushHints 748 | #49-62 same | |
| 92 | | SS:510-566 | `truncate` | `Sync` around `merkle::truncate` (SS:524-525); `createWriteBatch`; `setManifest_`; `truncate_`; `flush()`; `setDependencyHead`; `flushHints` | `co_await merkle::truncate(…)` (Sync removed); `co_await truncate_`; `co_await flush()`; `co_await flushHints()` | ss:427-486: lock 430, MerkleTree.truncate 444, _truncate 458, flush 470, flushHints 482 | #63-66d **changed** (l3 via Sync since 4c-3) | |
| 93 | | SS:568-596 | `reorg` | idle-assert (SS:569); `createWriteBatch` (inside locked_; JS creates it OUTSIDE its try, ss:491); `truncate_`; `flush()`; `setDependencyHead`; `flushHints` | `P<bool>` (JS returns undefined on success, false when not commitable); real lock | ss:488-514: lock 489, _truncate 496, flush 498, flushHints 510 | #67-72 same | |
| 94 | | SS:598-624, SSh:210 | `truncate_` | `l2::CoreTX&`; deleteBlockRange, setHead, commitBatch, setDependency, `storeBitfieldRange_`, setHints | `P<Truncated>` over `CoreTX&` / TX | ss:516-556, only await `storeBitfieldRange` 542 | #73-80 same | |
| 95 | | SS:626-630, SSh:212 | `updateDependency_` | `storage->findDependencyIndex`, `dependencies()`, `l2::Dependency` | `HS::findDependencyIndex`, `dependencies()`, `storage::Dependency` | ss:1344-1352 sync | #81 same | |
| 96 | | SS:706-718, SSh:216 | `storeBitfieldRange_` | `l2::CoreRX rx = storage->read()`; `rx.getBitfieldPage(p)` **used at once, no tryFlush**; `tx.putBitfieldPage` per page | issue every page, `tryFlush`, `co_await all`, then fill + put in page order | ss:1299-1326 (Promise.all 1316) | #82-84 same (H4) | |
| 97 | | SS:724-733 | `rootsFromStorage` (anon) | non-owning `L2TreeStorage` (aliasing shared_ptr, SS:726) + `Sync` + `merkle::getRootsFromStorage` (one rx) | `co_await merkle::getRootsFromStorage(StorageTree(st), len)`. The storage must be OWNED across the await (the aliasing ptr is only safe because the scope is synchronous) | mt:579-591; callers ss:1114, 1183, 1250 | #85-86 **changed** (now one rx; per-root rx gone) | |
| 98 | | SS:736-745 | `getTreeHeadAt_` | `rootsFromStorage(*storage, len)` | `P<hs::CoreHead>` | ss:1107-1123, await 1114 | #87 changed | |
| 99 | | SS:747-801, SSh:173 | `createSession` | `storage->resumeSession(n)`; `st->read().getHead()` temp rx **no tryFlush** (SS:755); `getTreeHeadAt_`; `createAtomicSession` / `createSession`; `rootsFromStorage(*st, len)`; `std::shared_ptr<l2::Atom>`; `atom->preflush`; `atom->onflush([](l2::Atom::Done d){…})` | `co_await resumeSession`; `getCoreHead` (rx, getHead, tryFlush, await); `co_await createAtomicSession / createSession`; `co_await getRootsFromStorage`; `onflush` returns `P<void>` (`c->commit_()`; gone child: a rejected `Error("Closed")`) | ss:1216-1274: lock 1217, resumeSession 1224, getCoreHead 1226, _getTreeHeadAt 1232, createAtomicSession 1235 / createSession 1237, getRootsFromStorage 1250 | #88-98 same | |
| 100 | | SS:805-809 | `precommit_` | `storage->core().dataPointer` | `storage->core->dataPointer` | ss:273-278 sync | #99 same | |
| 101 | | SS:811-829 | `commit_` | callback `mutex.lock` + `parent->oncommit_(cb)`; `mutex.unlock()` (not `_unlock`, ss:289) | `P<void>` | ss:280-291: lock 281, `_oncommit` 287 | #100 same | |
| 102 | | SS:831-873 | `oncommit_` | `storage->read()`, `getDependency()`, `tryFlush()` (JS-shaped); `createWriteBatch`; `updateGroupIndex`; `flush()`; `setDependencyHead`; `flushHints` | `co_await` dependency, flush, flushHints | ss:293-345: lock 294, dependency 305, flush 324, flushHints 341 | #101-108 same | |
| 103 | | SS:905-973 | `waitForRecovery` | rt timers (5000 / 250 ms, SS:930/933/940); `mutex.lock(cb)`; `core->flushHints()` | stays timer-driven (a continuation, never under a scope wait); tail `co_await mutex.lock(); try { co_await core->flushHints(); } finally { mutex.unlock(); }` | ss:615-659: timers 623 / 630, lock 649, flushHints 651 | #109 same | |
| 104 | | SS:977-992 | `truncateAndFlush_` | `Sync` + `merkle::truncate`; `createWriteBatch`; `truncate_`; `flush()` | `P<TruncFlush>` | ss:1328-1342: truncate 1330, _truncate 1333, flush 1334 | #110-113 **changed** (l3 via Sync) | |
| 105 | | SS:1019-1087 | `moveTo` | `lingers.push_back(storage)`; `state->storage->resumeSession(n)`; `truncateAndFlush_`; `getTreeHeadAt_`; `state->storage->createSession`; `rootsFromStorage(*st, len)`; `setDependencyHead` | each awaited | ss:1142-1214: lock 1145, resumeSession 1161, truncateAndFlush 1163, _getTreeHeadAt 1168, createSession 1180, getRootsFromStorage 1183 | #114-120a same | |
| 106 | | SS:1091-1179 | `overwrite_` | `source.storage->read()` + getTreeNode (roots / patch / pairs) + getBlock, `tryFlush` (JS-shaped); `createWriteBatch`; deleteBlockRange, putTreeNode, putBlock; `storage->read().getBitfieldPage(firstPage)` temp **no tryFlush** (SS:1140); putBitfieldPage; setHead; `flush()` | 3 promise vectors, `tryFlush`, `co_await` them; `srx` rx + tryFlush + await (ss:981-985); `co_await flush()` | ss:912-1032: Promise.all 935-939, bitfieldPage 985, flush 1023 | #120b-120p same | |
| 107 | | SS:1181-1185 | `isParent` | `l2::HypercoreStorage::isParentStorage` | `HS::isParentStorage` (CS:99) | ss:1034-1039 sync | #121a same | |
| 108 | | SS:1187-1212 | `commit` | `waitForRecovery(cb)` | `co_await waitForRecovery()` | ss:1041-1053 | #121b same | |
| 109 | | SS:1214-1272 | `commitLocked_` | nested `mutex.lock` callbacks; `core->validateCommit_` (Sync inside, HC:977); `Sync` + `merkle::batchRestore` (SS:1233-1234); `overwrite_`; `state->createWriteBatch`; deleteBlockRange; `updateDependency_` + setDependency; `state->flush()`; `setDependencyHead` | `co_await mutex.lock(); srcLocked; co_await state->mutex.lock()`; `co_await validateCommit_`; `co_await batchRestore`; `co_await overwrite_`; `co_await state->flush()` | ss:1055-1104: lock 1056, src lock 1059, _validateCommit 1064, restore 1070, _overwrite 1074, state.flush 1089 | #121c-121k **changed** (restore is one rx via l3 + Sync) | |
| 110 | | SS:1274-1340 | `catchup` | `dependencies()`; `core->state->storage->core().dataPointer`; `createWriteBatch`; `core->state->storage->read()` + roots + `tryFlush` (JS-shaped); delete*Range(0, -1); setHead; `updateDependency_`; setDependency; `flush()`; `setDependencyHead`; `flushHints` | as A0, awaited | ss:831-910: lock 834, Promise.all 858, flush 894, flushHints 906 | #122a-122l same | |
| 111 | | SS:1348-1372, SSh:115 | `commitBatch` | `l2::CoreTX&`; deleteTreeNodeRange / deleteTreeNode / putTreeNode | `CoreTX&`, void | mt:232-270 sync | #123 same | |
| 112 | | SSh:147 | `lingers` | `std::vector<std::shared_ptr<l2::HypercoreStorage>>` | `std::vector<HSP>` (JS: `null` until the first move, ss:31, 1158) | — | type | |
| 113 | | SSh:192, 198-199, 209-212 | `TruncFlush`, `updateSnapshotStorage_`, `readSnapshotTree_`, `Truncated`, `truncate_`, `updateDependency_` | `l2::Dependency`, `l2::HypercoreStorage`, `l2::CoreTX` | `storage::Dependency`, `HSP`, `CoreTX&` | — | type | |
| 114 | |
| 115 | ### A2. hypercore.cpp / hypercore.h |
| 116 | |
| 117 | | file:line | C++ fn | l2 call / type now | hc::storage equivalent | JS (await?) | old # / status | |
| 118 | |---|---|---|---|---|---| |
| 119 | | HCh:16 | include | `#include "hc/l2.h"` | `hc/corestore_storage.h` | — | type | |
| 120 | | HCh:54 | `OpenOptions::alias` | `std::optional<l2::hs::CorestoreAlias>` | `hs::CorestoreAlias` | cj:245 | type | |
| 121 | | HCh:121, HC:268 | `Core::Core` | `l2::CorestoreStorage &db` | `CorestoreStorage &` (or `CorestoreStoragePtr`) | cj:24 | type + H6 | |
| 122 | | HC:275 | ctor | `db.resumeCore(dk)` | `co_await db.resumeCore(dk)` | cj:168 await | #125 same | |
| 123 | | HC:285-301 | ctor | `storage_->read()`; getAuth / getHead / getHints / getGroup, **no tryFlush** | `getCoreInfo`: 4 getters, `tryFlush`, `co_await allOf` | cj:180 → cj:1049-1066 (Promise.all 1059) | #126-130 same (H4) | |
| 124 | | HC:336-343 | ctor | `l2::CreateCoreArgs c`; `db.createCore(c)` | `storage::CreateCoreArgs`; `co_await db.createCore(c)` | cj:238-246 await | #131 same | |
| 125 | | HC:366-374 | ctor (Bitfield.open) | `storage_->createBitfieldStream(0, pages)` → vector | `for await` over `Stream createBitfieldStream(IndexRange{0, pages})`, `mapped<PageEntry>` | cj:291 → bf:536-548 (`for await` 543) | #132 same | |
| 126 | | HC:377-385 | ctor (maybeGetRootsFromStorage) | `storage_->read()`, `getTreeNode` per root, **no tryFlush**, early `break` | `merkle::getRootsFromStorage(StorageTree(storage), len)`; INVALID_OPERATION → `[]` | cj:297 → cj:1033-1041 → mt:579-591 | #133-134 same (still raw l2 in the ctor; 4c-3 did not touch the ctor) | |
| 127 | | HC:391 → 782-796 | `setGroupLocked_` | `storage_->store->createGroup(group)` (sync pair); `storage_->write()`, `tx.setGroup`, `tx.flush()` | `co_await this->db->createGroup(group)` (JS uses `this.db`, cj:308 / 381); `co_await tx->flush()` | cj:307-318 (open), cj:378-394 (setGroup) | #135-136 same | |
| 128 | | HC:393-399 | ctor (overwrite) | write; deleteTreeNodeRange / deleteBlockRange / deleteBitfieldPageRange; `tx.flush()` | `co_await tx->flush()` | cj:320-326 (flush 325) | #137 same | |
| 129 | | HC:401-407 | ctor (hints) | `storage_->readOnly()`; write; setHints; flush | `HS::readOnly()`; `co_await flush` | cj:328-334 (flush 333) | #138-139 same | |
| 130 | | HC:409-421 | ctor (auth) | write; setAuth; flush | `co_await flush` | cj:337-349 (flush 347) | #140 same | |
| 131 | | HC:429 | ctor | `make_shared<SessionState>(…, storage_, …)` | `HSP` | cj:360 | #141 same | |
| 132 | | HC:435-439 | `~Core` | `state->close()` | destructors close nothing (design); keep the bookkeeping | (GC) | #142 same (H7) | |
| 133 | | HC:441 | `Core::session` | returns `state` (SessionView) | unchanged | — | 4c-3 | |
| 134 | | HC:443 | `Core::get_block` | `storage_->read().getBlock(i)`, no tryFlush | `readBlock(core->state->storage->read(), i)` | ix:956 (`_get`'s snapshot branch) | #144 same | |
| 135 | | HC:449-467 | `setManifest` | early `if (header.manifest) return` BEFORE the lock (HC:450); idle-assert (456); `mutex.lock([]{})`; `createWriteBatch`; `setManifest_`; `state->flush()` | `P<void>`; JS takes the lock first, checks `manifest && header.manifest === null` and validates inside it, and always `_unlock`s (cj:396-414) | cj:396-414: lock 397, flush 409 | #145-147 same | |
| 136 | | HCh:261, HC:469-489 | `setManifest_` | `l2::CoreTX &tx`; `tx.setAuth` | `CoreTX &` | cj:416-441 sync | #148 same | |
| 137 | | HC:589-705 | `flushPrologueBatch` (anon) | src `read()` + getTreeNode ×tree + `tryFlush`; dst `dst.storage().read()` + getHead (first) + getBitfieldPage ×pages + `tryFlush`; `src.storage->createUserDataStream()` (vector); `dst.storage().write()`; putTreeNode / putBitfieldPage / putBlock / putUserData / setHead; `tx.flush()` | JS shape: `co_await all(nodes)` BEFORE the dst rx (cp:73), head `.catch(noop)` (cp:79), pages awaited then head (91-92), `for await` user data (109), `co_await tx->flush()` (147) | cp:64-188 | #149-157 same | |
| 138 | | HC:707-754 | `copyPrologueInto` (anon) | `src.storage->createBlockStream(0, len, true)` **drained first**, then `flushPrologueBatch` (sync) inside the loop | `for await` over the stream with `co_await flushBatch(…)` INSIDE the loop (cp:32-56, 52), then the last flush (61) | cp:14-62 | #158-159 same (H8) | |
| 139 | | HC:758-778 | `Core::copyPrologue` | nested callback `mutex.lock`s | `P<void>`: lock; src lock (on failure unlock + rethrow, cj:447-451); `co_await copyPrologue`; finally src unlock, unlock, `checkIfIdle` | cj:443-460 | #160 same | |
| 140 | | HC:800-802 | `compact` | `s->storage->compact()` (void) per state | returns `js::all` of each state's `P<std::vector<js::Undefined>>` | cj:124-130 (returns Promise.all, not async) | #161 same | |
| 141 | | HC:820-952 | `audit` | callback `mutex.lock`; `Sync` + `merkle::getRootsFromStorage(state->sessionStorage(), length)` (HC:834-835); per node `state->storage->read()` + 2 getTreeNode + tryFlush (JS-shaped); `std::optional<l2::CoreTX> tx` (HC:831/879/906); `tx->flush()` (874/902/942); `state->storage->createBlockStream()` drained (880); per block rx + getTreeNode + tryFlush (888-890); per bit rx + getBlock + tryFlush (916-918); putBitfieldPage / deleteBitfieldPage / deleteBlock / deleteTreeNodeRange | `P<std::optional<AuditStats>>`; audit.js literally: `for await` the block stream with the per-block read awaited inside (au:80-92) | cj:368-376 (lock 369, `await audit` 372) + au:6-138 | #162-173 **changed** (tree roots: one rx via l3 + Sync) | |
| 142 | | HC:954-966 | `setGroup` | callback `mutex.lock` → `setGroupLocked_` | `P<void>`; `mutex.unlock()` not `_unlock` (cj:392) | cj:378-394 | #174 same | |
| 143 | | HC:968-990 | `validateCommit_` | `Sync` + `merkle::get(s.merkleSession(), root.index)` per root (HC:977-978) | `P<bool>`; `co_await merkle::get(…)` per root, in sequence | cj:466-485 (await 473) | #175 **changed** | |
| 144 | | HC:1062-1068 | `flushHints` | `storage_->write()`; setHints; `tx.flush()` | `P<void>`; JS writes `this.state.storage.write()` (cj:609); `hintsChanged = false` before the await, the "debounce" (cj:607) | cj:605-618 (flush 617) | #176 same | |
| 145 | | HC:1071-1074 | `downgrade` | `Sync` + `merkle::batchDowngrade(batch)` | `co_await merkle::batchDowngrade(batch)` (or inline at the two callers) | mt:301-325 via ss:365 / cj:801 | #177 **changed** | |
| 146 | | HC:1077-1149 | `verify` | `Sync` + `merkle::verify(state, proof)` (HC:1083-1084); `downgrade(tb)`; exclusive → `state->verifyBlock`; shared: idle-assert (1109), `mutex.lock([]{})`, `createWriteBatch`, putBlock, `setManifest_`, `commitBatch`, putBitfieldPage, `state->flush()`, `flushHints()`, `clearActiveBatch` + `mutex.unlock()` | `P<bool>`: `co_await merkle::verify`; `co_await batchDowngrade`; `_verifyExclusive` / `_verifyShared` with `_verifies` / `_verified` (cj:821-831) and BitInterlude | cj:792-832, 514-533, 535-603 | #178-189 **changed** (verify via l3 + Sync) | |
| 147 | | HC:1151-1179 | `verifyReorg` | `merkle::treeReorgBatch(state)`; `Sync` + `merkle::reorg` (1154-1155); idle-assert (1163); `createWriteBatch`; `setManifest_`; `state->flush()` | `P<TreeBatchPtr>`; `co_await merkle::reorg`; real lock; `_unlock` in finally (cj:785) | cj:771-790: reorg 773, lock 777, flush 782 | #190-192 **changed** | |
| 148 | | HC:1181-1279 | `checkConflict` | `merkle::verifyFullyRemote` (sync); `Sync` + `getRootsFromStorage(state->sessionStorage(), …)` (1209-1210); `Sync` + `proveAndSettle(state, state->sessionStorage()->read(), req)` (1224-1225, rx = L2Rx); idle-assert (1247); `createWriteBatch`; `tx.setAuth`; `state->flush()`; `mutex.unlock()`; `onconflict_` (sync) | `P<bool>`; `co_await` the roots (maybeGetRootsFromStorage, INVALID_OPERATION → []); rx = `StorageRx(state->storage->read())`, `proof`, `tryFlush`, `co_await` proof, `co_await settle`; `co_await repairTreeNodes_`; real lock; `co_await state->flush()`; `co_await onconflict_` | cj:620-703: roots 639, proof 657, settle 659, _repairTreeNodes 664, lock 670, flush 685, _onconflict 701 | #193-197 **changed** (proof protocol done in 4c-3) | |
| 149 | | HC:1281-1337 | `repairTreeNodes_` | idle-assert (1285); `mutex.lock([]{})`; `storage_->createAtom()` (`std::shared_ptr<l2::Atom>`); `storage_->atomize(atom)` (`std::unique_ptr<l2::HypercoreStorage>`); `atomized->write()`, putTreeNode ×n, `tx.flush()`; `Sync` + `proveAndSettle(state, make_shared<L2Rx>(atomized->read()), req)` (1318-1319); `atom->flush(cb)` **assumed inline** (1329-1331) | `AtomPtr HS::createAtom()`; `HSP HS::atomize(atom)`, never closed (as JS, cj:727); `co_await tx->flush()` (answers false: atom view); rx = `StorageRx(atomized->read())`; `co_await Atom::flush(atom)` (H10) | cj:705-769: lock 708, flush 732, proof 749, settle 751, atom.flush 761 | #198-203 **changed** | |
| 150 | | HC:1339-1349 | `onconflict_` | none (replicator hook, emit, `closeAllSessions`) | `P<void>`: JS awaits `replicator.onconflict()` (cj:921) and `closeAllSessions` (929, `Promise.allSettled` of the closes, 932-940) | cj:920-930 | #197 note | |
| 151 | | HC:1365-1378 | `reorg` | `state->reorg(batch)` (sync) | `P<bool>`; `truncating--` in a finally | cj:834-846 (await 840) | #204 same | |
| 152 | | HC:1387-1395 | `close` | `state->close()` (sync) | `P<void>`: JS `close()` memoizes `_close` (cj:862-865): `if (!opened) await opening` (958); throw while sessions (959); `await replicator.close()` (961); `await destroy()` (963) → `await state.close()` (951); `autoClose` (corestore passes false) | cj:862-865, 942-967 | #205 same (H7) | |
| 153 | | HC:1398-1405 | `missingNodes` | `Sync` + `merkle::missingNodes(state, …)` | `P<uint64_t>` | mt:743-763 via ix:1188 / replicator | #206 **changed** | |
| 154 | | HCh:156 | member `db` | `l2::CorestoreStorage *` | `CorestoreStorage *` | cj:25 | type | |
| 155 | | HCh:217 | `storage()` | `l2::HypercoreStorage &` | `HS &` (callers outside core: see C) | cj `this.storage` | type | |
| 156 | | HCh:293 | member `storage_` | `std::shared_ptr<l2::HypercoreStorage>` | `HSP` | cj:26 / 355 | type | |
| 157 | | HCh:333-334 | `to_stored` / `from_stored` | `l2::hs::CoreManifest` | alias | — | type | |
| 158 | | HC:26, 590, 598, 662, 805, 811, 839-846 | locals | `l2::hs::CoreTreeNode` / `CoreHead` | alias | — | type | |
| 159 | |
| 160 | ### A3. session.cpp / session.h |
| 161 | |
| 162 | | file:line | C++ fn | l2 call / type now | hc::storage equivalent | JS (await?) | old # / status | |
| 163 | |---|---|---|---|---|---| |
| 164 | | SEh:99 | `SessionOptions::atom` | `std::shared_ptr<l2::Atom>` | `AtomPtr` (callers outside core build these: see C) | ix:443 | type | |
| 165 | | SE:316-320 | `openRest_` | `state_->storage->write()`; putUserData ×n; `tx.flush()` **not awaited before `opened = true`** | `co_await tx->flush()` before setManifest / opened (ix:484-490) | ix:489 await | #211-213 same | |
| 166 | | SE:322 | `openRest_` | `core->setManifest(m)` (sync) | `co_await core->setManifest(m)` | ix:492-494 | #214 same | |
| 167 | | SE:471 | `readBlock_` | `state_->getBlock(i)` (SS:257) | `P<std::optional<Buf>>` | ix:1386-1390 | #215 same | |
| 168 | | SE:872-885 | `has` | `readBlock_(start)` (non-default state); `state_->storage->createBlockStream(start, end)` → vector `.size()` | rx / getBlock / tryFlush / await (ix:865-869); `for await`, counting (874-880) | ix:851-881 | #217-218 same | |
| 169 | | SE:928-982 | `get_` | `readBlock_` (929, 940); `core->get_block` (935) | `co_await` each; JS's "during the above async calls" re-checks then run on live state (ix:949, 954-960, 964) | ix:944-997 | #219-220 same | |
| 170 | | SE:887-926 | `get` | `(*marks_)->set(index, true)` (894, sync) | `co_await markBlock(index)` (ix:895 → `Promise.all` of `_marks.set`) | ix:883-912 | #222 same | |
| 171 | | SE:1074-1076 | `setupMarks_` | `make_shared<MarkBitfield>(… storage)` | `HSP` | ix:1007-1012 | #221 same | |
| 172 | | SE:1078-1084 | `markBlock` | `(*marks_)->set(i, true)` one after another | `Promise.all` of `set` (ix:1020-1025) | ix:1014-1026 | #223 same | |
| 173 | | SE:1086-1093 | `clearMarkings` | `(*marks_)->clear()` | `co_await` | ix:1028-1035 (1033) | #224 same | |
| 174 | | SE:1099 | `startMarking` | `state_->storage->atom()` | `->atom` | ix:1044 | #225 same | |
| 175 | | SE:1113-1144 | `sweep` | `(*marks_)->createMarkStream(true)` (vector), `batchSize` ignored (1114), clears one after another | `for await` the Transform stream; `clearing.push(this.clear(…))`, `Promise.all` every `batchSize`, a final `Promise.all`, then `clearMarkings` | ix:1053-1081 | #226-227 same (H8) | |
| 176 | | SE:1146-1154 | `compact` | `core->compact()` (sync void), then `done` at once | returns `core->compact()`'s promise (ix:234-236) | ix:234-236 | #228 same | |
| 177 | | SE:1162-1179 | `rootsHash` / `treeHashFromStorage` | `Sync` + `merkle::getRoots(state.merkleSession(), length)` (1166-1167); `treeHashFromStorage` is a sync static | `static P<K32> treeHashFromStorage` (ix:225-228 `static async`) | ix:225-228, 1182 | #229 **changed** | |
| 178 | | SE:1221-1223 | `missingNodes` | `core->missingNodes(…)` (Sync inside) | `co_await` | ix:1186-1189 | #230 changed | |
| 179 | | SE:1347-1367 | `proofOf` / `proof` | `Sync` + `proofOf(state, make_shared<L2Rx>(state_->storage->read()), req)`; the block is read in the proof's batch through `rx->getBlock` | rx = `StorageRx(state_->storage->read())`; block via the bridge's underlying `getBlock`; `allOf`; `settle` | ix:1191-1201 | #231 **changed** (H9 shape fixed in 4c-3) | |
| 180 | | SE:1229-1231 | `applyProof` | `core->verify(p)` (sync) | returns `core->verify` (ix:1205) | ix:1203-1206 | #232 same | |
| 181 | | SE:1233-1244 | `verifyFullyRemote` | none (`merkle::verifyFullyRemote` sync) | hop parity only: JS awaits a sync fn (1210) and `_verifyBatchUpgrade` (1211) | ix:1208-1213 | #233 same | |
| 182 | | SE:1375-1403 | `recoverFromRemoteProof` | idle-assert (1382); `mutex.lock([]{})`; `fully_remote_proof::verify(*core->db, …)` (sync); `core->storage().write()`, putTreeNode ×n, `tx.flush()` | `P<bool>`; real lock; `co_await verify`; `co_await tx->flush()`; JS `return false` inside the try still unlocks in finally (ix:1232, 1245-1247) | ix:1224-1248: lock 1228, verify 1231, flush 1238 | #234-236 same | |
| 183 | | SE:1413-1419 | `setUserData` | `state_->storage->read().getUserData(key)` temp, **no tryFlush** | `co_await getUserData(key)` (rx, getUserData, tryFlush) | ix:717-722 (719, 721) | #237-238 same (H4) | |
| 184 | | SE:1421-1425 | `getUserData` | same temp rx, no tryFlush | rx, getUserData, tryFlush, return the promise | ix:724-730 | #239 same (H4) | |
| 185 | | SE:1429-1493 | `seek` | `merkle::seek(state_->merkleSession(), …)`; `Sync` + `s->update()` (1453-1456) | `co_await s->update()` | ix:809-849 (826-828) | #240 **changed** (keetd's ByteSeeker deleted) | |
| 186 | |
| 187 | ### A4. marks.cpp / marks.h |
| 188 | |
| 189 | | file:line | C++ fn | l2 call / type now | hc::storage equivalent | JS (await?) | old # / status | |
| 190 | |---|---|---|---|---|---| |
| 191 | | MKh:7, 16 | include / `using l2::Buf` | — | `storage::Buf` | — | type | |
| 192 | | MKh:23, 32 | ctor / `storage_` | `std::shared_ptr<l2::HypercoreStorage>` | `HSP`; plus JS's `mutex` (mb:38) and `_pages` of MarkPage (mb:39) | mb:36-40 | #246g | |
| 193 | | MK:11-18 | `loadPage_` | `storage_->read()`, `rx.getMark`, `rx.tryFlush()` (JS-shaped), the page inserted AFTER the read | `_pages.set(i, MarkPage)` FIRST (mb:43), rx / getMark / tryFlush, `co_await`, `setBitfield` | mb:42-50 | #246a-c (**JS lines corrected**) | |
| 194 | | MK:20-26 | `get` | `loadPage_` | `P<bool>` | mb:52-60 | — | |
| 195 | | MK:28-40 | `set` | `storage_->write()`, `putMark`, `tx.flush()`; no mutex ("keetd's writes are synchronous", MK:37) | `co_await loadPage` (when absent and val); `co_await p.loaded`; set the bit; `co_await mutex.lock()`; write / putMark / `co_await flush`; unlock (mb:62-79). JS has NO try/finally: a flush rejection leaves the mutex locked | mb:62-79 | #246d same | |
| 196 | | MK:42-47 | `clear` | write, `deleteMarkRange(0, -1)`, `flush()` | `co_await flush()`; `_pages` reset AFTER the flush | mb:81-86 | #246e same | |
| 197 | | MK:49-65 | `createMarkStream` | `storage_->createMarkStream(0, -1, reverse)` → collected `vector<uint64_t>` | `Stream` piped through a streamx Transform (findLast / findFirst) | mb:88-111 | #246f same | |
| 198 | |
| 199 | ### A5. fully_remote_proof.cpp / .h |
| 200 | |
| 201 | | file:line | C++ fn | l2 call / type now | hc::storage equivalent | JS (await?) | old # / status | |
| 202 | |---|---|---|---|---|---| |
| 203 | | FRP:15 | ns | `namespace hs = l2::hs` | alias | — | type | |
| 204 | | FRPh:30, FRP:41 | `verify` | param `l2::CorestoreStorage &` | `CorestoreStorage &`; returns `P<std::optional<Result>>` | frp:27 async | #249 same | |
| 205 | | FRP:58-59 | `verify` | `storage.resumeCore(dk)` → `std::unique_ptr<l2::HypercoreStorage>` | `co_await resumeCore` → `HSP`; never closed (as JS, frp:42) | frp:42-43 await | #250 same | |
| 206 | | FRP:61-71 | `verify` | `core->read()`; getAuth / getHead / getUserData("referrer"); **no tryFlush** (scope destructor) | 3 getters, `tryFlush`, `co_await allOf` | frp:45-52 Promise.all | #251 same (H4) | |
| 207 | | FRP:75-83 | `verify` | `core->read()`, getTreeNode per root, early `break`, no tryFlush | all roots, `tryFlush`, `co_await all`, `roots.some(isNull) → []` | frp:58-69 | #252 same | |
| 208 | | FRP:97 | `verify` | `make_shared<L2TreeStorage>(non-owning aliasing ptr to core.get())` | `StorageTree(core)` OWNING (the SlimSession's storage is read only by verifyFullyRemote, which reads nothing, mt:672-691) | frp:10-23, 75 | **new since 4c-3** (old #248 `StorageReader` **gone**) | |
| 209 | | FRP:101 | `verify` | `merkle::verifyFullyRemote(slim, proof)` (sync) | hop only: JS `await`s a sync fn (frp:75) | frp:75 | #253 same | |
| 210 | | FRP:136-152 | `proof` | `sender.proof(req)` (sync overload SE:1360) | `co_await sender.proof(req)`; `proof` returns `P<Buf>` (frp:105 async); `Session::generateRemoteProofForTreeNode` (SE:1369-1373) returns that promise (ix:1215-1222) | frp:105-126 (await 106) | #254 same | |
| 211 | |
| 212 | ### A6. merkle_l2.h (the R4c-4c-3 adapter: deleted at R4c-4d) |
| 213 | |
| 214 | | file:line | what | replaced by | |
| 215 | |---|---|---| |
| 216 | | ML2:23-65 | `L2Rx : merkle::Rx` (getTreeNode / hasTreeNode / getBlock QUEUE in call order, `tryFlush` resolves in order, a getter after the flush rejects with the freed view's TypeError, `destroy` drops the queue) | the `StorageRx` bridge over `CoreRXPtr` (A0). The real CoreRX resolves at the rn read completion, not at `tryFlush`: one more completion hop per batch than L2Rx (the first hop measurement is the core gate's `hops` mode) | |
| 217 | | ML2:68-72 | `L2TreeStorage : merkle::TreeStorage` (`read()` = fresh L2Rx, fork -1) | `StorageTree` over `HSP` | |
| 218 | | ML2:77-82 | `proveAndSettle` (proof, `tryFlush`, await, settle in ONE chain) | kept as is (it is JS's cj:644-659 / 736-751 shape), over StorageRx | |
| 219 | | users | SS:4, 255, 726; HC:7, 1225, 1319; SE:8, 1349-1363; FRP:71/97 | — | |
| 220 | |
| 221 | **Counts at bb707dd.** `grep -c 'l2::\|hc/l2.h'`: SS 44, SSh 17, HC 35, HCh 9, SE 3, SEh 1, MK 3, MKh 4, FRP 6, FRPh 1, ML2 8. |
| 222 | `scaffold::Sync` scopes inside core/ (the R4c-4c-3 frontier, all deleted at R4c-4d): SS:524, 728, 981, 1233; |
| 223 | HC:834, 977, 1072, 1083, 1154, 1209, 1224, 1318, 1400; SE:1166, 1362, 1454. That is **16**. |
| 224 | |
| 225 | --- |
| 226 | |
| 227 | ## B. Core functions whose JS counterpart is async (or returns a promise) |
| 228 | |
| 229 | Shape key: **sync** = returns its value; **cb** = completion callback (`std::function<void (Fail…)>`); **cb+locked_** = |
| 230 | callback through `SessionState::locked_`; **Sync** = contains a `scaffold::Sync` scope; **assert** = one of the 8 |
| 231 | "while the state mutex is held" throws. Each row becomes a `js::Promise` coroutine with JS's awaits in JS's order. |
| 232 | Brackets mark conditional awaits. |
| 233 | |
| 234 | ### B1. lib/session-state.js → SessionState (SS) |
| 235 | |
| 236 | | JS fn (lines) | C++ now (lines) | shape | JS awaits, in order | becomes | |
| 237 | |---|---|---|---|---| |
| 238 | | `unref` (104-107, sync, fires `close().catch(noop)`) | `unref` SS:151-154 | sync, calls sync close | — | sync; fires `P<void> close()` and marks it handled | |
| 239 | | `_updateSnapshotStorage` (128-176) | `updateSnapshotStorage_` SS:197-224 + `readSnapshotTree_` SS:226-253 | cb (weak self-loop) + sync l2 | [`while flushing: atomized.flushed()` 130] → `Promise.all(head, auth, dep)` 138 → `Promise.all(roots)` 153 | `P<TreeInfo>` | |
| 240 | | `close` (188-208) | `close` SS:168-185 | sync | (`mutex.destroy().catch` 192 is not awaited) → `for lingers: storage.close()` 204 → returns `closing` (195) | `P<void>` | |
| 241 | | `snapshot` (210-215) | SS:187-195 | cb | `_updateSnapshotStorage` 212 | `P<std::shared_ptr<SessionState>>` | |
| 242 | | `flush` (259-271) | SS:284-290 | sync bool | `tx.flush()` 264 | `P<bool>` | |
| 243 | | `_commit` (280-291) | `commit_` SS:811-829 | cb | `mutex.lock` 281 → `parent._oncommit` 287 | `P<void>` (the Atom Handler) | |
| 244 | | `_oncommit` (293-345) | `oncommit_` SS:831-873 | cb+locked_ | lock 294 → dependency 305 → [flush 324] → [flushHints 341] | `P<void>` | |
| 245 | | `setUserData` (347-358) | SS:308-316 | cb+locked_ | lock 348 → flush 354 | `P<bool>` | |
| 246 | | `_verifyBlock` (360-425) | `verifyBlock` SS:318-359 | sync + **assert** (319) + Sync (via `core->downgrade`) | lock 361 → [downgrade 365] → [preupdate 373: unset in keetd] → [storeBitfieldRange 383] → flush 407 → [flushHints 419] | `P<bool>` | |
| 247 | | `truncate` (427-486) | SS:510-566 | cb+locked_ + Sync | lock 430 → MerkleTree.truncate 444 → `_truncate` 458 → flush 470 → [flushHints 482] | `P<void>` | |
| 248 | | `reorg` (488-514) | SS:568-596 | sync + **assert** (569) | lock 489 → `_truncate` 496 → flush 498 → [flushHints 510] | `P<bool>` (JS: `undefined` / `false`) | |
| 249 | | `_truncate` (516-556) | `truncate_` SS:598-624 | sync | [storeBitfieldRange 542] | `P<Truncated>` | |
| 250 | | `clear` (558-612) | SS:361-414 | cb+locked_ | lock 559 → [storeBitfieldRange 581] → flush 595 → [flushHints 607] | `P<void>` | |
| 251 | | `waitForRecovery` (615-659) | SS:905-973 | cb over rt timers | loop: [setTimeout 5000 623] / 32 × setTimeout 250 630 → lock 649 → flushHints 651 | `P<void>`, still timer-driven. **Never** under a `scope.wait` (D4) | |
| 252 | | `append` (661-754) | SS:416-508 | cb (+ waitForRecovery re-entry) + locked_ | [waitForRecovery 664] → lock 666 → [preappend 678] → [empty: flush 681] → [storeBitfieldRange 713] → flush 736 → [postappend 744] → [flushHints 748] | `P<AppendResult>` | |
| 253 | | `catchup` (831-910) | SS:1274-1340 | cb+locked_ | lock 834 → `Promise.all(roots)` 858 → flush 894 → [flushHints 906] | `P<void>` | |
| 254 | | `_overwrite` (912-1032) | `overwrite_` SS:1091-1179 | sync | `Promise.all([blocks, nodes, roots])` 935 → [bitfieldPage 985] → flush 1023 | `P<Overwritten>` | |
| 255 | | `commit` (1041-1105) | `commit` SS:1187-1212 + `commitLocked_` SS:1214-1272 | cb (nested locks) + Sync | [waitForRecovery 1053] → lock 1056 → state lock 1059 → `_validateCommit` 1064 → [restore 1070] → `_overwrite` 1074 → [state.flush 1089] | `P<std::optional<AppendResult>>` | |
| 256 | | `_getTreeHeadAt` (1107-1123) | `getTreeHeadAt_` SS:736-745 | sync + Sync | getRootsFromStorage 1114 | `P<hs::CoreHead>` | |
| 257 | | `moveTo` (1142-1214) | SS:1019-1087 | cb+locked_ + Sync | lock 1145 → resumeSession 1161 → [truncateAndFlush 1163] → [_getTreeHeadAt 1168] → [createSession 1180] → getRootsFromStorage 1183 | `P<void>` | |
| 258 | | `createSession` (1216-1274) | SS:747-801 | cb+locked_ + Sync | lock 1217 → [resumeSession 1224 → getCoreHead 1226] → [_getTreeHeadAt 1232 → createAtomicSession 1235 / createSession 1237] → [getRootsFromStorage 1250] | `P<std::shared_ptr<SessionState>>` | |
| 259 | | `storeBitfieldRange` (1299-1326) | `storeBitfieldRange_` SS:706-718 | sync | `Promise.all(pages)` 1316 | `P<void>` | |
| 260 | | `truncateAndFlush` (1328-1342) | `truncateAndFlush_` SS:977-992 | sync + Sync | MerkleTree.truncate 1330 → `_truncate` 1333 → flush 1334 | `P<TruncFlush>` | |
| 261 | | `getCoreHead` (1363-1368, returns a promise) | inline SS:755 | sync | — (returns `p`) | returns `P<std::optional<hs::CoreHead>>` | |
| 262 | |
| 263 | ### B2. lib/core.js → Core (HC) |
| 264 | |
| 265 | | JS fn (lines) | C++ now (lines) | shape | JS awaits, in order | becomes | |
| 266 | |---|---|---|---|---| |
| 267 | | constructor `this.opening = this._open(opts)` (67-68) + `ready()` (75-77) | `Core::Core` HC:268-433 | sync ctor doing all the I/O | — | ctor without I/O; `P<void> opening`; `ready()` returns it | |
| 268 | | `_open` (148-163) | — (in the ctor) | — | `_tryOpen` 150 (catch → `onidle()`, rethrow) → `opened = true` → setDownloading pair | `P<void> open_()` | |
| 269 | | `_tryOpen` (165-366) | ctor | sync | [preopen 166] → resumeCore 168 → [getCoreInfo 180] → [createCore 238] → Bitfield.open 291 → maybeGetRootsFromStorage 297 → [group: createGroup 308, flush 317] → [overwrite flush 325] → [hints flush 333] → [auth flush 347] | `P<void>` | |
| 270 | | `compact` (124-130, returns `Promise.all`) | HC:800-802 | sync void | — | returns `js::all(…)` | |
| 271 | | `lockExclusive` (139-142) | HCh:254 `lockExclusive(granted)` | cb | `exclusive.lock()` 141 | `P<void>` (one hop even when free) | |
| 272 | | `audit` (368-376) + audit.js | HC:820-952 | cb + Sync | lock 369 → `audit()` 372 (see B5) | `P<std::optional<AuditStats>>` | |
| 273 | | `setGroup` (378-394) | HC:954-966 + `setGroupLocked_` 782-796 | cb (and sync from the ctor) | lock 379 → [createGroup 381] → flush 390 | `P<void>` | |
| 274 | | `setManifest` (396-414) | HC:449-467 | sync + **assert** (456) | lock 397 → [flush 409] | `P<void>` | |
| 275 | | `copyPrologue` (443-460) + copy-prologue.js | HC:758-778, 589-754 | cb | lock 444 → src lock 447 → `copyPrologue` 454 (see B6) | `P<void>` | |
| 276 | | `_validateCommit` (466-485) | `validateCommit_` HC:968-990 | sync + Sync | per root `MerkleTree.get` 473 | `P<bool>` | |
| 277 | | `_verifyExclusive` (514-533) | inline HC:1098-1104 | sync | `state._verifyBlock` 518 | `P<bool>` | |
| 278 | | `_verifyShared` (535-603) | inline HC:1106-1148 | sync + **assert** (1109) | lock 538 → flush 582 → [flushHints 596] | `P<bool>` + `_verifies` / `_verified` | |
| 279 | | `flushHints` (605-618) | HC:1062-1068 | sync | flush 617 | `P<void>` | |
| 280 | | `checkConflict` (620-703) | HC:1181-1279 | sync + Sync ×2 + **assert** (1247) | maybeGetRootsFromStorage 639 → treeProofPromise 657 → settle 659 → [`_repairTreeNodes` 664] → lock 670 → flush 685 → `_onconflict` 701 | `P<bool>` | |
| 281 | | `_repairTreeNodes` (705-769) | `repairTreeNodes_` HC:1281-1337 | sync + Sync + **assert** (1285) | lock 708 → flush 732 → proof 749 → settle 751 → atom.flush 761 | `P<bool>` | |
| 282 | | `verifyReorg` (771-790) | HC:1151-1179 | sync + Sync + **assert** (1163) | MerkleTree.reorg 773 → [lock 777 → flush 782] | `P<TreeBatchPtr>` | |
| 283 | | `verify` (792-832) | HC:1077-1149 | sync + Sync | MerkleTree.verify 798 → [downgrade 801] → `_verifyExclusive` (returned, 818) / [`await this._verified` 824] / `_verifyShared` (returned, 831) | `P<bool>` | |
| 284 | | `reorg` (834-846) | HC:1365-1378 | sync | `state.reorg` 840 | `P<bool>` | |
| 285 | | `_onconflict` (920-930) | `onconflict_` HC:1339-1349 | sync | `replicator.onconflict()` 921 → `closeAllSessions` 929 | `P<void>` | |
| 286 | | `closeAllSessions` (932-940) | HC:1359-1363 | sync | `Promise.allSettled(closes)` 939 | `P<void>` | |
| 287 | | `destroy` (942-955) | in `close` HC:1392-1393 | sync | `state.close()` 951 | `P<void>` | |
| 288 | | `close` (862-865) / `_close` (957-967) | HC:1387-1395 | sync | [opening 958] → replicator.close 961 → destroy 963 → [store.close 964, autoClose: false under corestore] | `P<void>`, memoized as `closing` | |
| 289 | | `maybeGetRootsFromStorage` (1033-1041), `getCoreInfo` (1049-1066) | inline in the ctor | sync | getRootsFromStorage 1035 / Promise.all 1059 | `P<…>` helpers | |
| 290 | |
| 291 | Also: `Core::downgrade` (HC:1071, Sync) → inlined `co_await batchDowngrade`; `Core::missingNodes` (HC:1398, Sync) → |
| 292 | `P<uint64_t>`; `Core::get_block` (HC:443) → `readBlock` promise. |
| 293 | |
| 294 | ### B3. index.js → Session (SE) |
| 295 | |
| 296 | | JS fn (lines) | C++ now (lines) | shape | JS awaits, in order | becomes | |
| 297 | |---|---|---|---|---| |
| 298 | | `_open` (324-379) | ctor SE:139-162 + `settle_` 431-445 | callback chain | [preload 330] → `_openSession` 350 (catch: [core.close 352], unlock, remove, `closed`, `'close'`) → `'ready'` | `P<void> opening` started in the ctor | |
| 299 | | `_openSession` (389-499) | `openSession_` → `openSessionGroup_` → `openSessionState_` → `openSessionLocked_` → `openCheckout_` → `openRest_` SE:176-336 | callback chain | [core.ready 390] → [core.setGroup 393] → [parent.ready 419] → [ps.snapshot 426] → [lockExclusive 436] → [createSession 444 / 449] → [state.truncate 469] → [tx.flush 489] → [core.setManifest 493] | one coroutine | |
| 300 | | `static treeHashFromStorage` (225-228) | SE:1179 | sync static + Sync | getRoots 226 | `static P<K32>` | |
| 301 | | `compact` (234-236) | SE:1146-1154 | cb, done at once | returns `core.compact()` | `P<…>` | |
| 302 | | `setEncryptionKey` / `setEncryption` / `setGroup` / `setAlwaysLatestBlock` (273-322) | SE:1269-1284, 619-624 | cb via whenOpened_ | [opening] (+ `return core.setGroup` 296) | `P<void>` | |
| 303 | | `_close` (539-585) | `close` SE:509-553 | cb via whenOpened_ | [opening 542] → [core.close 581, autoClose] | `P<void>` | |
| 304 | | `commit` (587-592) | SE:1329-1343 | cb | ready 588 → session.ready 589 → returns `state.commit` | `P<std::optional<AppendResult>>` | |
| 305 | | `recover` (712-715) | SE:1068-1070 | cb | [opening] → returns `waitForRecovery()` | `P<void>` | |
| 306 | | `setUserData` (717-722) | SE:1413-1419 | cb, sync l2 read | [opening] → getUserData 719 → state.setUserData 721 | `P<void>` | |
| 307 | | `getUserData` (724-730) | SE:1421-1425 | cb, sync l2 read | [opening] → returns `p` (after tryFlush) | `P<std::optional<Buf>>` | |
| 308 | | `info` (770-774) | SE:1286-1298 | cb | [opening] | `P<Info>` | |
| 309 | | `update` (776-807) | SE:557-594 | cb | [opening] → `replicator.applyPendingReorg()` 787 → [`req.promise` 798] | `P<bool>`; `applyPendingReorg` is R4f's (replicator) | |
| 310 | | `seek` (809-849) | SE:1429-1493 | cb + Sync | [opening] → [addUpgrade `req.promise` 819] → `s.update()` 828 → [`req.promise` 844] | `P<std::optional<std::pair<…>>>` | |
| 311 | | `has` (851-881) | SE:872-885 | cb, sync l2 | [opening] → [block 869] / [`for await` 875] | `P<bool>` | |
| 312 | | `get` (883-912) | SE:887-926 | cb | [opening] → [markBlock 895] → `_get` 899 → [encryption.decrypt 908] | `P<std::optional<Buf>>` | |
| 313 | | `clear` (914-937) | SE:1497-1511 | cb | [opening] → `state.clear` 934 | `P<std::optional<uint64_t>>` | |
| 314 | | `_get` (944-997) | `get_` SE:928-982 | sync + cb (network) | readBlock 945 → [readBlock core 956] → [readBlock 965] → [`req.promise` 989] | `P<std::optional<Buf>>` | |
| 315 | | `markBlock` (1014-1026) | SE:1078-1084 | cb, sequential sets | [opening] → returns `Promise.all(sets)` | `P<void>` | |
| 316 | | `clearMarkings` (1028-1035) | SE:1086-1093 | cb | [opening] → `_marks.clear()` 1033 | `P<void>` | |
| 317 | | `startMarking` (1037-1051) | SE:1095-1111 | cb | [opening 1047] → clearMarkings 1048 | `P<void>` | |
| 318 | | `sweep` (1053-1081) | SE:1113-1144 | cb chain | [opening] → `for await` markStream 1063 (`Promise.all` 1070 per batchSize) → `Promise.all` 1077 → clearMarkings 1080 | `P<void>` | |
| 319 | | `truncate` (1109-1129) | SE:1045-1066 | cb | [opening] → `state.truncate` 1125 | `P<void>` | |
| 320 | | `append` (1131-1167) | `append_` SE:1002-1043 | cb | [opening] → returns `state.append` | `P<AppendResult>` | |
| 321 | | `signable` (1169-1175) | SE:1207-1219 | cb | [opening] → treeHash 1174 | `P<Buf>` | |
| 322 | | `treeHash` (1177-1184) | SE:1181-1205 | cb + Sync | [opening] → [has 1180] → [get 1180] → getRoots 1182 | `P<K32>` | |
| 323 | | `missingNodes` (1186-1189) | SE:1221-1223 | cb + Sync | [opening] → MerkleTree.missingNodes 1188 | `P<uint64_t>` | |
| 324 | | `proof` (1191-1201) | SE:1225-1227 (cb) + 1360-1367 (sync, Sync) | cb / sync + Sync | [opening] → `Promise.all([proof, block])` 1197 → settle 1198 | `P<wire::Data>` (one overload) | |
| 325 | | `applyProof` (1203-1206) | SE:1229-1231 | cb | [opening] → returns `core.verify` | `P<bool>` | |
| 326 | | `verifyFullyRemote` (1208-1213) | SE:1233-1244 | cb | [opening] → verifyFullyRemote 1210 (sync fn) → `_verifyBatchUpgrade` 1211 (sync fn) | `P<Batch>` (2 hops) | |
| 327 | | `generateRemoteProofForTreeNode` (1215-1222, returns `proof(...)`'s promise) | SE:1369-1373 | sync | — | `P<Buf>` | |
| 328 | | `recoverFromRemoteProof` (1224-1248) | SE:1375-1403 | sync + **assert** (1382) | lock 1228 → verify 1231 → [flush 1238] | `P<bool>` | |
| 329 | | `preappend` (1359-1366) | lambda SE:1034-1038 | sync | per block `encryption.encrypt` 1364 | `P<void>` (hop parity) | |
| 330 | | `readBlock` (1386-1390, returns a promise) | `readBlock_` SE:471 / SS:257 | sync | — | returns `P<std::optional<Buf>>` | |
| 331 | |
| 332 | ### B4. lib/mark-bitfield.js → MarkBitfield (MK) |
| 333 | |
| 334 | | JS fn | C++ | shape | JS awaits | becomes | |
| 335 | |---|---|---|---|---| |
| 336 | | `loadPage` (42-50) | `loadPage_` MK:11-18 | sync | pageBuf 47 → `p.setBitfield` 48 (the inserted MarkPage's `loaded` resolves) | `P<MarkPagePtr>` | |
| 337 | | `get` (52-60) | MK:20-26 | sync | [loadPage 57] | `P<bool>` | |
| 338 | | `set` (62-79) | MK:28-40 | sync, no mutex | [loadPage 68] → `p.loaded` 71 → mutex.lock 73 → flush 76 | `P<void>` (with the MarkBitfield Mutex) | |
| 339 | | `clear` (81-86) | MK:42-47 | sync | flush 84 | `P<void>` | |
| 340 | | `createMarkStream` (88-111) | MK:49-65 | sync vector | stream + Transform | `Stream` | |
| 341 | |
| 342 | ### B5. lib/audit.js (6-138) → the body of `Core::audit` (HC:820-952) |
| 343 | |
| 344 | Awaits in order: `getRootsFromStorage` 25 → [deleteCore 32: unreachable, getTreeNodeOrError throws first] → per stack |
| 345 | node `Promise.all([left, right])` 57 → [tx.flush 73] → `for await` block stream 80 with, per block, `treeNodePromise` |
| 346 | 92 → [tx.flush 104] → per set bit `blockPromise` 116 → [tx.flush 134]. Now: a callback + Sync + sync l2 reads, with the |
| 347 | block stream collected first (HC:880). Becomes one coroutine (`auditCore`) with the stream consumed by `for await`. |
| 348 | |
| 349 | ### B6. lib/copy-prologue.js → `copyPrologueInto` / `flushPrologueBatch` (HC:589-754) |
| 350 | |
| 351 | `copyPrologue` (14-62): `for await` block stream (reverse) 32, with [flushBatch 52] inside the loop → flushBatch 61. |
| 352 | `flushBatch` (64-188): `Promise.all(nodes)` 73 → `Promise.all(pages)` 91 → [head 92] → [`for await` user data 109] → |
| 353 | `tx.flush()` 147. Now: sync, the stream drained first. Becomes two coroutines. |
| 354 | |
| 355 | ### B7. lib/fully-remote-proof.js → fully_remote_proof (FRP) |
| 356 | |
| 357 | `verify` (27-103): resumeCore 42 → `Promise.all(auth, head, ref)` 52 → `Promise.all(roots)` 68 → verifyFullyRemote 75 |
| 358 | (a sync function: one hop). Now sync (FRP:41-134). Becomes `P<std::optional<Result>>`. `proof` (105-126): |
| 359 | `sender.proof` 106. Now sync (FRP:136-152). Becomes `P<Buf>`. |
| 360 | |
| 361 | ### B8. lib/bitfield.js → Bitfield.open (in the Core ctor, HC:366-374) |
| 362 | |
| 363 | `static async open` (536-548): `for await` the bitfield stream 543. Becomes part of `open_()`. `clear(tx)` (526-528) |
| 364 | is sync. |
| 365 | |
| 366 | ### B9. lib/mutex.js → Mutex (SS:57-95, HCh:81-94) |
| 367 | |
| 368 | `lock()` (16-23) returns `Promise.resolve()` when free (ONE hop even then), `new Promise(enqueue)` when locked, and a |
| 369 | rejection when destroyed. `unlock()` (25-31) resolves the head waiter (it resumes in a microtask). `destroy(err)` |
| 370 | (33-46) returns `_destroying` (when locked, a `lock().catch()`) and rejects every queued waiter. The C++ grants |
| 371 | synchronously inside `lock` / `unlock` (SS:57-77) and has no `_destroying`. `cancel(ticket)` (SS:91-95) is |
| 372 | keetd-only: `~Session` uses it for an exclusive waiter (SE:165). JS never cancels; it relies on GC. Becomes |
| 373 | `P<void> lock()`. |
| 374 | |
| 375 | --- |
| 376 | |
| 377 | ## C. The frontier after R4c-4d (callers outside core/) |
| 378 | |
| 379 | How this was checked: a sweep of every non-core directory (store, repl src + test, bee, blobs, wakeup, mux, tools, |
| 380 | store/blobs/wakeup tests), with each caller read around the call. The rows marked ✓ I re-read myself at bb707dd. |
| 381 | There is no daemon directory in the worktree, and mux/ calls nothing in core. |
| 382 | |
| 383 | Legend for "after R4c-4d": |
| 384 | - **wait**: the caller uses the value or state at once → `{ Sync s; s.wait(p); }`. |
| 385 | - **scope**: callback-shaped and read at once → a bare Sync scope around the call; its destructor drains the scope. |
| 386 | - **chain**: already continuation-shaped → `.then` / a callback, no scope. |
| 387 | - **net**: may wait on the network or a timer → never `wait` (D4). |
| 388 | |
| 389 | ### C0. `scaffold::Sync` already outside core (the R4c-4c-3 frontier) |
| 390 | |
| 391 | | file:line | caller | wraps | |
| 392 | |---|---|---| |
| 393 | | repl/src/replicator.cpp:482 ✓ | `Peer::_canUpgrade` | `merkle::upgradeable(core->session(), …)` | |
| 394 | | repl/src/replicator.cpp:562 ✓ (waits 566, 589) | `Peer::_handleRequest` | `merkle::proof(core->session(), batch, req)` + `allOf(settle, block)`; batch = `L2Rx(core->storage().read(fork))` (540) | |
| 395 | | repl/src/replicator.cpp:1984 ✓ | `Replicator::_updateNonPrimary` | `s->seeker->update()` | |
| 396 | | repl/src/replicator.cpp:2245 ✓ | `Replicator::_onreorgdata` | `merkle::reorgUpdate(f->batch, data)` | |
| 397 | | tools/core-ops/main.cpp:292, 435, 460, 481 | ops proof / reorg serve / reorg loop / conflict | `merkle::proof` + `rx->getBlock`, `hcore::proveAndSettle`, `reorgUpdate` | |
| 398 | |
| 399 | Lines 540 and 1984 reach core-owned l2 through `core->storage()` (HCh:217) wrapped in `L2Rx`. They switch to the |
| 400 | StorageRx bridge at E5. |
| 401 | |
| 402 | ### C1. store/src/corestore.cpp |
| 403 | |
| 404 | | line | caller | core call | shape today | done synchronously after | after R4c-4d | |
| 405 | |---|---|---|---|---|---| |
| 406 | | 420 ✓ | `Corestore::openCore_` | `make_unique<hcore::Core>(*shared_->storage, oo)` (throw → `openError` + `gc`) | ctor does all the I/O | 427: `Replicator(core)` reads `core->repairMode`; 450-451 ✓ read `header.keyPair`, `state->length` (Core._open's tail) | **wait** on `core->ready()` in a scope (the rejection goes into `openError`). The Replicator's construction stays after the open, as JS's `_open` tail | |
| 407 | | 501 / 507-508 ✓ | `makeSession_` | `make_unique<Session>(*t->core, so)`; `s->ready([&]…)` | ctor + cb; rethrows only if settled inline | returns `s` | **scope** around ctor + ready (never `wait`: an exclusive session may queue, D4); the inline-error rethrow is kept | |
| 408 | | 188 ✓ | `Shared::gcCycle` | `t->core->close()` (exceptions swallowed) | sync | `erase(t)` at once | **wait** on `Core::close()` (`P<void>`). JS erases only after the close settles (cs:151-164) — R4g | |
| 409 | | 201 ✓ | `Shared::closeAll` | `t->core->close()` per core | sync | `cores.clear()`; then `Corestore::close` closes the storage (310) | **wait** on each close (JS `Promise.all`, cs:186-197) before the storage close | |
| 410 | | 304 | `Corestore::close` | `s->close()` per session | cb, no done | — | **scope** | |
| 411 | | 574 | `attachMaybe_` | `t->core->checkIfIdle()` | sync | `done("")` | unchanged (sync in JS) | |
| 412 | | 140, 515 | field reads | `core->closed` | — | — | unchanged | |
| 413 | | (E5) 105, 116, 242-244, 282-288, 310, 330, 332, 357, 363, 555, 216-221 | the store's own l2 calls | (store's, not core's) | — | — | switch type at E5, each **wait**ed in a scope until R4g | |
| 414 | |
| 415 | ### C2. repl/src/replicator.cpp (the 11.37 JS lines are the old inventory's `rep:` rows) |
| 416 | |
| 417 | | line | caller | core call | today | done synchronously after | after R4c-4d | |
| 418 | |---|---|---|---|---|---| |
| 419 | | 540 | `_handleRequest` | `core->storage().read(fork)` | accessor + l2 | wrapped in L2Rx | `StorageRx(core->storage().read(fork))`, already in the 562 scope | |
| 420 | | 692 ✓ | `Peer::_handleData` | `core->checkConflict(data)` | sync bool | `return` when true | **wait** (JS awaits rep:1214). The `_rxLock` returns at R4f, not here (sr:H7) | |
| 421 | | 732 ✓ | `_handleData` | `core->verify(data)` | sync bool, throws caught | 733-760: `_onnodata` / `_markProcessed` / `_ondata` / `_updateCanUpgradeAndSync` on `applied` | **wait** (JS rep:1258) | |
| 422 | | 890 ✓ | `Peer::onrange` | `core->flushHints()` | sync write | `_update()`, `emitRemoteContiguousLength` (893) | **wait** (JS awaits, rep:1440) | |
| 423 | | 1290, 1293 ✓ | `Peer::_send` | `core->missingNodes(…)` | sync (Sync inside core) | value into `req->…nodes`; `nodes == 0` → local resolve; `send` | **wait** (the CANCELLED re-check comes with R4f, sr:H10) | |
| 424 | | 1388, 2323 | `onpeerdestroy`, `attachTo` | `core->checkIfIdle()` | sync | — | unchanged | |
| 425 | | 1726 ✓ | `_resolveBlocksLocally` (from the core hook `onupgrade`, 1488) | `core->get_block(i)` | sync l2 read | `b->resolve` inline, then `_blocks.remove` | **wait** on `readBlock`. It runs from inside a core op (`onupgrade` fires in `onappend`), so it is a NESTED scope inside that op's coroutine: it must not wait on the state mutex (it does not; it reads only). R4f gives it one reader + `Promise.all` (sr:H16) | |
| 426 | | 2233 ✓ | `_onreorgdata` | `core->verifyReorg(data)` | sync `TreeBatchPtr` | `_addReorg`, maybe `_applyReorg` | **wait** | |
| 427 | | 2276 ✓ | `_applyReorg` (from 2258, and `applyPendingReorg` 2183, which core reaches via `onapplypendingreorg`, 1348) | `core->reorg(*f->batch)` | sync | 2282-2297 resolve upgrade / reorgs | **wait**. Re-entrancy hazard: `Session::update` (SE:569) → `onapplypendingreorg` → `core->reorg` → `state->reorg` → `mutex.lock()`. Today update is not under the mutex, so the lock is free. Keep it that way (update must stay outside any locked section) | |
| 428 | | 483 | `_canUpgrade` | `core->session()` | accessor | (in C0's scope) | unchanged | |
| 429 | | 638, 1874; hdr 453, 458; 2405; 2424 | emitTransfer / emitMonitors / `Extension::_onmessage` / `Session::emit` | — | sync events | — | unchanged (sync in JS) | |
| 430 | | 872, 1831 | `updateRemoteContiguousLength`, `openSkipBitfield` | — | memory | — | unchanged | |
| 431 | | 1323-1384, 1412-1434 | hook install / reset; reads `core->repairMode` (1384) | — | — | — | the ctor must run after the open (C1 420) | |
| 432 | |
| 433 | ### C3. bee/src/bee.cpp — all inline-completion, none checked (failures are SILENT) |
| 434 | |
| 435 | | line | caller | core call | done synchronously after | after | |
| 436 | |---|---|---|---|---| |
| 437 | | 52 ✓ | `getLocal` (used by `inflateCheckpointFromLength_` 420, `inflateCheckpoint_` 432, `Context::getBlock` 523 → most bee reads) | `Session::get(seq, {wait:false}, [&]…)` | reads `err` / `out` (53-54); a late callback becomes a false BLOCK_NOT_AVAILABLE | **scope** (wait:false never goes to the network) | |
| 438 | | 394 | `Context::openCore_` | `store.get(g)` | returns the session | via C1 | |
| 439 | | 409 ✓ | `Context::updateMaybe` | `core->getUserData(CHECKPOINT_KEY, [&]…)` | reads `buf` at 410 | **scope** | |
| 440 | | 446 | `inflateCheckpoint_` | `core->setUserData(…, noop)` | fire and forget | **scope** (keeps today's atomicity) | |
| 441 | | 649 | `Hyperbee::Hyperbee` | `st.get(g)` | 651 reads `core->key` | via C1 | |
| 442 | | 1129 ✓ | `copyChanges_` (from `reindex` 985) | `core->append(…, [&]…)` | `appendError` checked inline (1130); `reindex` 1001 reads `length()` | **scope**; but `net`-class when the core is recovering (D4: waitForRecovery) — must not be `wait` | |
| 443 | | 1693 ✓ | `WriteBatch::flush_` (from `flush` 1277) | `core->append(…, [&]…)` | inline error check (1694); `setRoot` 1303 | as 1129 | |
| 444 | |
| 445 | ### C4. blobs/src/blobs.cpp |
| 446 | |
| 447 | `now<T>` (43-51 ✓) **throws** "did not complete synchronously" when the callback has not fired. Wrapped calls: 58 |
| 448 | `get` (wait:false), 67 `seek`, 73 `append`, 77 `clear`, 84 `session` + ready, 564 `snapshot` + ready. They are used |
| 449 | inline at 142, 158, 168, 239-244, 266, 296, 308-317, 327, 388, 393, 408-426, 476, 507-510, 556-559. **After:** |
| 450 | `now<T>` opens a Sync scope around `call` (the destructor drains it), and its throw stays the guard. `seek` can reach |
| 451 | the replicator (`onaddseek`) when the block is not local, and `append` can reach `waitForRecovery`. Those stay loud |
| 452 | (the throw), as today. `Session::close()` at 343 and 568 is fire and forget → **scope**. |
| 453 | |
| 454 | ### C5. wakeup/ |
| 455 | |
| 456 | No storage-reaching core call: `autobee_wakeup.cpp` 15, 25, 50, 54, 106 are accessors and listeners. wakeup.cpp has |
| 457 | none. Nothing changes. |
| 458 | |
| 459 | ### C6. Tests and tools |
| 460 | |
| 461 | | file:line | what | today | after | |
| 462 | |---|---|---|---| |
| 463 | | repl/test/r3c_walk_test.cpp:91 ✓, r3c_scan_test.cpp:73 ✓ | `hcore::Core core(store, oo)` on the stack (+ a Replicator) | ctor opens | **wait** `ready()`; switch `l2::CorestoreStorage` (85 / 67) at E5 | |
| 464 | | repl/test/m3c_pipe_test.cpp:145, 157, 170 | `make_unique<Core>` (+ Replicator 146 / 158 / 171) | ctor | **wait** `ready()`; switch the store type (128-164, 881-887) at E5 | |
| 465 | | m3c_pipe_test.cpp:180 | `core->get_block` | sync | **wait** | |
| 466 | | m3c_pipe_test.cpp:1483 ✓ | `s->has(…, [&h]…)` read at once | inline | **scope** | |
| 467 | | m3c_pipe_test.cpp: gets / update / append / commit / close / `state->append` / `truncate` (784…1553; 1114-1117, 1336) | settled by polling `run()` / `settle()` | chain | unchanged (1116-1117 rely on the state mutex for FIFO order, which the Promise mutex keeps) | |
| 468 | | store/test/m4c_pipe_test.cpp:219-220 ✓ | `append` then reads `ok` and `treeHash()` | inline | **scope** | |
| 469 | | m4c_pipe_test.cpp:323 | a `wait:false` get asserted "at once" | inline | **scope** | |
| 470 | | blobs/test/monitor_pipe_test.cpp:109, 123; wakeup/test/autobee_wakeup_pipe_test.cpp:120 | `store->get` | via C1 | — | |
| 471 | | tools/core-ops/main.cpp:228-229 ✓ | `make_unique<Core>(…)` then `summary(*c)` | ctor | **wait** `ready()` | |
| 472 | | core-ops: 270 setManifest, 277 missingNodes, 305 verify, 446 verifyReorg, 470-471 reorg, 489 checkConflict, 506 / 714 / 719 `Session::proof` (sync overload), 520 generateRemoteProofForTreeNode, 524 / 529 fully_remote_proof proof / verify, 535 recoverFromRemoteProof, 709 treeHashFromStorage, 908 flushed | sync returns | sync | **wait** each | |
| 473 | | core-ops: 256 `state->append`, 265 `state->setUserData`, 712 `s.missingNodes` | callback read at once | inline | **scope** | |
| 474 | | core-ops: `take()` (135-142 ✓) used at 173, 386-428, 543, 562, 580-585, 594, 654, 705-724, 782, 795-848, 904, 911-926, 948; stream ops 636-643, 687, 693 | `rth::act([]{})` (harness.h:22 = `rt::callback`: one callback + a microtask drain, **no storage completions**), then "did not complete" | inline after one tick | becomes "deliver until settled": a Sync scope around the call + `wait` on a promise. Once the ops can suspend on rn completions, `act` alone never sees them. The hops mode (E6) drives these with `rulers` / `nowait` / `waitAll` instead | |
| 475 | | core-ops: 205, 927-931 | `l2::Atom` (`storage.createAtom()`, `atom->flush(cb)`) | cb | `AtomPtr`; `Atom::flush(atom)` → **wait** or a slot; E5 type switch | |
| 476 | | core-ops: 156-179, 260, 309, 499, 508, 601, 605 | direct `l2::CorestoreStorage` / `CoreRX` / `CoreTX` | l2 | E5: storage API, each read / flush **wait**ed | |
| 477 | | tools/store-ops/main.cpp:151, 166 / 179-181 / 154, 168, 229 | `Corestore::get`; append + `act` + "did not complete"; close | inline | **scope** / via C1 | |
| 478 | | tools/keetd-store/main.cpp:151, 223; 157-159 ✓ | `st->get`; append with an err read at once, then `length()` / `treeHash()` | inline | **scope** | |
| 479 | | tools/keetd-replicate/main.cpp:150 ✓ (Replicator 155), 211, 245, 212, 223-227 | `make_unique<Core>(*store, oo)`; `get_block`; treeHash / state reads | ctor + sync | **wait** `ready()` / `readBlock`; store type (142) at E5 | |
| 480 | | tools/blobs-ops/main.cpp:129, 143-145 | `st.get`; put + `act` + "did not settle" | inline | **scope** | |
| 481 | | tools/bee-read/main.cpp:102, 106, 118, 129 | `st.get(g)->core->key` / `length()`; `Hyperbee::ready` | inline | via C1 / C3 | |
| 482 | | tools/m4d/appender.cpp:57, 68, 70-71, 76-83 ✓ | `st.ready()`, `st.get`, `length()`, append with `err` read at once (virtual runtime) | inline | **scope** + the rn hook (D7) | |
| 483 | | tools/core-fixture/fixture.h:31-64 | direct l2 writes (`CreateCoreArgs`, `CoreTX`) + `hcore::to_stored` | l2 | E5 switch, flush **wait**ed | |
| 484 | |
| 485 | ### C7. Counts |
| 486 | |
| 487 | store/src 7 core call sites (+ the store's own ~11 l2 calls); repl/src 21 calls + 4 existing scopes; repl/test ~38; |
| 488 | bee 7; blobs ~14 (6 through `now`); wakeup 0; store / blobs / wakeup tests ~20 / 3 / 2; tools: core-ops ~60 (4 |
| 489 | scopes), store-ops 5, keetd-store 4, keetd-replicate 5, blobs-ops 2, bee-read 4, m4d 3+hook, core-fixture (l2 only). |
| 490 | mt-trace / st-trace / hs-trace / store-dump / l2-ops never touch core. |
| 491 | |
| 492 | --- |
| 493 | |
| 494 | ## D. Hazards specific to R4c-4d (re-checked at bb707dd) |
| 495 | |
| 496 | **D1. Promise Mutex, the 8 asserts and `locked_` (core:H2/H3).** The 8 "while the state mutex is held" throws are |
| 497 | still there: SS:319 (`verifyBlock`), SS:569 (`reorg`), HC:456 (`setManifest`), HC:1109 (`verify` shared), HC:1163 |
| 498 | (`verifyReorg`), HC:1247 (`checkConflict`), HC:1285 (`repairTreeNodes_`), SE:1382 (`recoverFromRemoteProof`). The |
| 499 | callback mutex users are `locked_` (SS:292-306), `commit_` (SS:813), `waitForRecovery` (SS:958), |
| 500 | `commitLocked_` (SS:1217-1218), `copyPrologue` (HC:760-761), `audit` (HC:821), `setGroup` (HC:955) and the exclusive |
| 501 | lock (HCh:254-257; Session open SE:240; cancel in `~Session` SE:165). |
| 502 | - **A Promise `lock()` costs one hop even when free (mx:22).** Every callback-shaped op then completes one microtask |
| 503 | LATER. Callers that assume inline completion therefore break the moment the Mutex converts, unless they already sit |
| 504 | in a frontier `Sync` scope. Those callers are bee's `[&]` callbacks, blobs' `now<T>()`, tools/m4d, core_ops and the |
| 505 | r3c tests. **So the frontier scopes must land BEFORE the Mutex** (E0). |
| 506 | - **JS's finally order.** `_unlock()` in the finally, THEN the async function's promise settles, so a queued waiter's |
| 507 | resumption microtask is queued before the caller's reaction. `locked_` encodes this with "body returns a resolver" |
| 508 | (SS:292-306). With coroutines it is automatic, provided the coroutine's own completion is the promise settle (no |
| 509 | callback `done` invoked inside the body). |
| 510 | - **Different unlock calls, kept as JS has them.** `_commit` (ss:289), `waitForRecovery` (ss:654), `setGroup` |
| 511 | (cj:392), `copyPrologue` (cj:456-457), `checkConflict` (cj:687) and `_verifyShared` (cj:599) use |
| 512 | `mutex.unlock()`, not `_unlock()`, so they skip `checkIfIdle`. `copyPrologue` calls `checkIfIdle` itself (458). |
| 513 | The C++ already matches each one. |
| 514 | - **`Mutex::cancel` (SS:91-95) has no JS counterpart.** With a Promise mutex, an exclusive waiter whose Session is |
| 515 | destroyed is a suspended coroutine whose frame holds a dangling `this` once resumed. Keep the waiter's frame |
| 516 | alive-checked (`alive_` token, SE:367) instead of cancelling, or keep `cancel` as a keetd-only removal of the |
| 517 | queued resolver. A JS mutex never drops a waiter, so the next unlock passes the lock to the dead waiter, which then |
| 518 | must unlock at once. Decide and record which. |
| 519 | - **`Mutex::destroy` must return `_destroying` (mx:33-46).** `SessionState::close` fires it with `.catch(noop)` |
| 520 | (ss:192). |
| 521 | |
| 522 | **D2. Core opens in the constructor (core:H6).** The ctor (HC:268-433) does 4 l2 reads and up to 4 writes, plus |
| 523 | `createGroup` and the stream. JS: `this.opening = this._open(opts); this.opening.catch(noop)` (cj:67-68), and |
| 524 | `_open`'s catch calls `onidle()` (cj:151-153). Session's open has no `await core.ready()` (SE:196 comment "corestore |
| 525 | opened it synchronously"; JS ix:390). `Core::checkIfIdle` (HC:1380-1385) lacks JS's `!this.opened` guard (cj:133), |
| 526 | which becomes reachable. Constructor sites outside core: see C. Every caller that reads `core->header` / |
| 527 | `state->length` / `key` right after construction must `wait` on `ready()` in a frontier scope. The `key` / |
| 528 | `discoveryKey` assignment (HC:431-432) moves to the end of `_tryOpen` (cj:362-365). JS also sets them in the |
| 529 | constructor when `opts.key` is given (cj:33-35). |
| 530 | |
| 531 | **D3. Storage lifetime and close policy (core:H7).** |
| 532 | - l2's HypercoreStorage closed itself in its destructor. The new one does not. `HypercoreStorage::close()` is |
| 533 | `view->readStop(); return db->close()` (hypercore_storage.cpp:337-344), and nothing in core calls it today |
| 534 | (SS:175 comment). |
| 535 | - Port JS's sites literally: |
| 536 | - `SessionState::close` returns `storage.close()` (ss:195, 207) and awaits each linger (ss:203-205); |
| 537 | - `unref` fires it (ss:106); |
| 538 | - `Core.destroy` awaits `state.close()` (cj:951). |
| 539 | - Never close where JS does not: `_repairTreeNodes`' atomized storage (cj:727), frp's resumed core (frp:42), |
| 540 | `snapshot()`'s storage when its state is GC'd. |
| 541 | - **What this costs:** the rn root's close closes every remaining session (rn/src/rocksdb.cpp:78-79). An unclosed |
| 542 | session therefore leaks only until the store closes, and never hangs `CorestoreStorage::close`. |
| 543 | - **Destructors close nothing** (design). But `~SessionState` (SS:116) and `~Core` (HC:435-439) must keep the C++ |
| 544 | bookkeeping: the swap-pop out of `core->sessionStates`, which holds raw pointers. |
| 545 | - **Non-owning storage handles that rely on synchronous scopes:** `rootsFromStorage`'s aliasing ptr (SS:726) and |
| 546 | FRP:97's aliasing ptr to `unique_ptr::get()`. Both become owning `HSP`s, because a coroutine suspends. |
| 547 | - **Every CoreRX must end in `tryFlush()` or `destroy()`.** It `readStart`s the view in its constructor |
| 548 | (tx.cpp:117-120), and only `free_` stops it. Sites that today rely on l2's temporary-rx destructor: SS:257, 755, |
| 549 | 1140; HC:285-288, 379-384, 443; SE:1415, 1423; FRP:65-69, 77-83. |
| 550 | |
| 551 | **D4. Waits on something outside the scope (NoProgress).** `scaffold::Sync::wait` throws NoProgress when the promise |
| 552 | depends on work outside the scope: an outer microtask, a completion submitted earlier, a timer or I/O |
| 553 | (scaffold.h:12-22). The loop never runs inside a scope (rt.h:95). At R4c-4d this can happen in these places: |
| 554 | - **`waitForRecovery`** (SS:905-973) arms rt timers (5 s / 250 ms). Any frontier `wait` on an op that reaches it |
| 555 | throws NoProgress: `Session::append` / `appendValues` when `header.hints.recovering != 0` (SS:420), `Session::commit` |
| 556 | (SS:1198), `Session::recover`. Today these return with the callback pending, and a caller that reads its |
| 557 | `[&]` result at once silently sees "no error". bee's append sites are an example (see C). The design already |
| 558 | lists `waitForRecovery` as the one continuation exception. |
| 559 | - **A timer-resumed continuation holds the mutex outside every scope.** After the wait, the rest of `append` / |
| 560 | `commit` runs from a timer callback at the root, not in a scope. Once it is a coroutine, it holds the state mutex |
| 561 | across a storage completion that is parked in the ROOT queue. A frontier entry that arrives meanwhile and waits on |
| 562 | `mutex.lock()` would wait on the root, which is NoProgress. Examples: a replicator handler's `core->verify` in a |
| 563 | Sync scope, or a bee append. **Fix:** the timer callback opens its own Sync scope and runs the continuation to |
| 564 | quiescence (the timer is a frontier entry). Then the mutex is free again whenever any other entry runs, as the |
| 565 | frontier rule requires. |
| 566 | - **Network waits.** `Session::get` / `seek` / `update` with wait, and `Download`, resolve through the replicator. |
| 567 | Frontier callers must keep these callback-shaped. They may open a scope around the call (the local path completes |
| 568 | inside it), but must not `wait` on them. |
| 569 | - **The exclusive lock.** A Session opened with `exclusive` while another exclusive session holds the lock |
| 570 | (SE:238-249) suspends until that session closes. A scope around the Session constructor is fine: the scope pops |
| 571 | when it is quiescent, and a pending promise is not work. A `wait(session.ready())` would throw NoProgress. |
| 572 | - **Nested scopes.** Core's 16 internal Sync scopes (A6) nest inside the frontier ones. Microtasks and completions go |
| 573 | to the innermost scope (rt.h:86-90). An inner scope's `wait` on a promise whose resolution is queued in an OUTER |
| 574 | scope throws NoProgress. One example is a Mutex waiter resolved by an unlock that ran in the outer scope. This |
| 575 | appears when an inner scope is reached from a converted coroutine that resumed in an outer scope. **Rule for the |
| 576 | transition: a function is converted with every Sync inside it removed in the same commit.** No coroutine may call a |
| 577 | function that still waits in an inner scope on the mutex or on a flush the coroutine started. |
| 578 | |
| 579 | **D5. CoreTX ownership (core:H12): mostly retired.** |
| 580 | - `CoreTX::flush` is not a coroutine. It moves `changes` out and returns `View::flush(c, db)` (tx.cpp:101-113), so |
| 581 | the TX object need not live across the await. |
| 582 | - What remains is the reference pattern. `createWriteBatch` returns `l2::CoreTX&` into `std::optional activeTx_` |
| 583 | (SS:266-270), and `flush` moves it out (SS:285). With `activeTx_` as `TX`, hold the `TX` (shared_ptr) in locals. |
| 584 | Today, callers hold a `CoreTX&` across `storeBitfieldRange_` / `truncate_`, which will be awaits. That is safe only |
| 585 | because the mutex is held. |
| 586 | - **Also:** `audit`'s three local `std::optional<l2::CoreTX>` (HC:831/879/906) become `TX` (null), and |
| 587 | `copyPrologue`'s / `setGroupLocked_`'s / `flushHints`' by-value l2 TXs become `TX`. |
| 588 | - `reorg`'s `createWriteBatch` sits outside JS's try (ss:491). A throw there leaks the lock in JS too. Port it |
| 589 | literally. |
| 590 | |
| 591 | **D6. Coroutine frame lifetime (new for R4c-4d).** js::Promise coroutines are eager (`initial_suspend` = |
| 592 | `suspend_never`, js.h:252), like an async function. |
| 593 | - **By-reference parameters dangle after the first suspension.** Examples: `const wire::Data &proof` (Core::verify, |
| 594 | checkConflict), `const std::string &key`, `merkle::Batch &batch` (SessionState::reorg, commitBatch callers), |
| 595 | `SessionState &src` (copyPrologue), `Session &target` (moveTo), `const std::shared_ptr<SessionState> &`. Take them |
| 596 | by value / shared_ptr, as l3 did (`merkle::verify(SessionPtr, wire::Data)`). |
| 597 | - **A raw `this` in a suspended member coroutine** (Core, SessionState, Session, MarkBitfield) dangles if the owner is |
| 598 | destroyed while it is suspended. JS's GC keeps the object. Examples: |
| 599 | - store's Tracked owns `std::unique_ptr<Core>`; |
| 600 | - `~Session` closes but does not wait; |
| 601 | - `Download` already uses `alive_`. |
| 602 | |
| 603 | Each coroutine pins its object: SessionState has `shared_from_this`, and Core / Session need an equivalent pin or |
| 604 | an `alive_` check after every `co_await`, as the replicator's `life_` does. |
| 605 | - **What keeps this out of reach today:** the frontier scope makes every entry complete inside the scope, so |
| 606 | dangling becomes possible only at the D4 exceptions (timers, network) and in R4f/R4g, when the frontier moves up. |
| 607 | It must still be designed in now. In particular, `waitForRecovery`'s Ctx holds `shared_from_this` (SS:919), and an |
| 608 | `append` suspended behind it holds the Session. |
| 609 | |
| 610 | **D7. The M4d kill-point hook (tools/m4d).** |
| 611 | - Today `l2::test_after_write` is declared at l2.h:50 and defined at l2/src/db.cpp:220. `Db::write` calls it after a |
| 612 | synchronous `rocksdb_write` (db.cpp:240), but only for non-empty batches (`if (changes.empty()) return`, db.cpp:223). |
| 613 | `tools/m4d/appender.cpp:58-62` installs it after `st.ready()`, counting writes to `--exit-after-writes N`. |
| 614 | - It must move into rn's write completion. That is `rn::native::write` (rn/src/native.cpp:264-284), whose |
| 615 | `complete([&req, onwrite] { … onwrite(error); })` is at 279-283. Call it in that lambda before `onwrite`. |
| 616 | - **Write numbering shifts.** rn issues writes for EMPTY batches too, and store ops group-commit (design). |
| 617 | `gate.py`'s BOUNDARY phase (gate.py:126-142) iterates consecutive N and stays valid. The other phases kill at |
| 618 | random times and are unaffected. |
| 619 | - **The appender's own frontier.** `st.ready()` (57), `st.get(g)` (68), `->length()` reads (70-71, 76, 83), and |
| 620 | `append` with an `[&]` callback whose `err` is read at once (78-82). All need Sync scopes. The appender runs on |
| 621 | `rt::makeVirtual()` (44), and a virtual runtime has no loop to fall back on. |
| 622 | |
| 623 | **D8. `_verifies` / `_verified` batching (cj:821-831).** |
| 624 | - Today it is unreachable: frontier entries are atomic, so no two shared verifies overlap. Port it anyway (design), |
| 625 | and drive it from the core gate (`tools/r4c4b` measured blocks 9 / 11 settling before 1 / 5). |
| 626 | - `_verifyShared` creates its write batch BEFORE its `try` (cj:540), outside the finally's `_clearActiveBatch`. That |
| 627 | ordering matters if `createWriteBatch` asserts. |
| 628 | |
| 629 | **D9. Session open ordering.** JS checks `this.core.closing` after `'ready'` (ix:378) and runs `_open`'s catch with |
| 630 | `await this.core.close()` when `autoClose` is set (ix:352; corestore passes `autoClose: false`). The C++ `settle_` |
| 631 | (SE:431-445) emits `'ready'` before settling. In JS, `'ready'` is emitted inside `_open` (375), and `opening` |
| 632 | resolves after it. Keep that order. `whenOpened_` waiters become `if (!opened) co_await opening` (JS's per-method |
| 633 | guard). Its callers' order on settle changes from "queued vector in FIFO order" to "promise reactions in |
| 634 | registration order", which is the same FIFO. |
| 635 | |
| 636 | **D10. Ordering deviations that become observable (core:H11, re-checked).** |
| 637 | - `markBlock` is sequential (SE:1081), against JS's `Promise.all` under MarkBitfield's mutex (mb:73-77). The |
| 638 | MarkPage is inserted after the read (MK:15), against before it (mb:43). |
| 639 | - `preappend` / `postappend` are synchronous (SS:447, 499), against awaited (ss:678, 744). |
| 640 | - `_get`'s re-checks (SE:930-941) run against state that is now live. |
| 641 | - `compact` calls `done` at once (SE:1146-1154). |
| 642 | - The open's userData flush is not awaited (SE:316-320). |
| 643 | - `treeHashFromStorage` is a sync static (SE:1179). |
| 644 | - `setManifest` checks `header.manifest` before the lock (HC:450). JS does it inside, under `_unlock` (cj:396-414), |
| 645 | so JS always costs a lock hop and a `checkIfIdle`. |
| 646 | - `copyPrologue` drains its block stream first (HC:726). `audit` collects blocks first (HC:880). `sweep` serializes |
| 647 | its clears (SE:1128-1142). |
| 648 | |
| 649 | **D11. Atom handlers.** |
| 650 | - `l2::Atom::Handler` is `void(Done)`. `storage::Atom::Handler` is `std::function<P<void> ()>` (S:110). |
| 651 | `createSession`'s onflush (SS:792-796) becomes `return c->commit_()`, and a gone child becomes a rejected |
| 652 | `Error("Closed")`. |
| 653 | - `repairTreeNodes_` reads `flushError` on the line after `atom->flush(cb)` (HC:1329-1331). That is core:H10, and it |
| 654 | becomes `co_await Atom::flush(atom)`. |
| 655 | - Atoms are created and flushed OUTSIDE core (`SessionOptions::atom`, SEh:99): `tools/core-ops/main.cpp:205, 927-931`. |
| 656 | See C. |
| 657 | |
| 658 | --- |
| 659 | |
| 660 | ## E. Proposed order of sub-steps for R4c-4d (each builds, ctest green, gates identical) |
| 661 | |
| 662 | Gates referenced: |
| 663 | - **ctest**: all 90 at R4c-4c (CMakeLists `add_test`s). |
| 664 | - **m4b_core_gate**: core-ops.js vs core_ops — results, dumped stores, OPTIONS, cross-reopen. |
| 665 | - **m4c_store_gate**, **m4c_pipe**, **m4d_gate**, **m3c_pipe_\***, **r3c_scan / r3c_walk**. |
| 666 | - **m5b-f**: bee, blobs, wakeup, autobee_wakeup. |
| 667 | - **fuzz_\*_corpus**. |
| 668 | - **r4c_mt**, **r4c_st**, **drill_lint**. |
| 669 | - Replicate gates and P1 generators (design). |
| 670 | |
| 671 | "Identical" means byte-identical traces and stores against the previous commit. |
| 672 | |
| 673 | **E0. Frontier scopes first, core unchanged.** Wrap every frontier call listed in C in `rt::scaffold::Sync`: |
| 674 | - `wait` where the caller uses the result synchronously; |
| 675 | - a bare scope (run to quiescence) around callback-shaped calls whose callback the caller reads at once; |
| 676 | - nothing around network-waiting calls. |
| 677 | |
| 678 | Core still completes inline, so each scope is a no-op. Proof: every gate is identical, and r4c_scaffold stays green. |
| 679 | This commit is what lets E1 delay completion by a hop without breaking callers. |
| 680 | - **Gate:** full ctest + m4b/m4c/m4d/m3c/m5* identical. |
| 681 | - **Drill:** add a set that removes a frontier scope and expects red. The scope must be load-bearing, at the latest |
| 682 | after E1. |
| 683 | |
| 684 | **E1. Extend the adapter into an async-l2 facade with hc::storage's exact API.** Widen ML2 into one temporary |
| 685 | header that gives l2 the S:/CS: signatures of A0, so later commits compile against the final API: |
| 686 | - CoreRX getters return `P<std::optional<…>>`, queued and resolved at `tryFlush`, as L2Rx does; |
| 687 | - `CoreTX::flush` returns `P<bool>` (l2 writes synchronously, so it resolves at the call); |
| 688 | - `resumeSession` / `createSession` / `createAtomicSession` / `close` / `resumeCore` / `createCore` / |
| 689 | `createGroup` / `compact` return `P`; |
| 690 | - the Atom has `flushed()` / `flush()` → `P<void>`. |
| 691 | |
| 692 | Nothing uses it yet. |
| 693 | - **Gate:** ctest (build) + a small unit test of the facade's queue / flush order. |
| 694 | |
| 695 | **E2. The Promise Mutex (lib/mutex.js literally, incl. `_destroying`).** |
| 696 | - `locked_` becomes a coroutine shell around the still-synchronous bodies: `co_await lock`; run the body; `_unlock`; |
| 697 | settle. |
| 698 | - The callback users `commit_`, `commitLocked_`, `copyPrologue`, `audit`, `setGroup` and `waitForRecovery`'s tail |
| 699 | `co_await` the lock. |
| 700 | - The 8 asserts become `{ Sync s; s.wait(mutex.lock()); }` for now. That is equivalent: NoProgress replaces the keetd |
| 701 | throw, and they are unreachable while frontier entries are atomic. |
| 702 | - `lockExclusive` returns `P<void>`. Decide `cancel` (D1). |
| 703 | - **Gate:** ctest; m4b identical in results. Hop-insensitive gates stay identical because every entry completes in |
| 704 | its scope. Drill the mutex (FIFO, microtask resume, destroy rejects waiters). |
| 705 | |
| 706 | **E3. Async Core open.** `Core::Core` does no I/O. `opening = open_()` (`_open` + `_tryOpen` + `getCoreInfo` + |
| 707 | `maybeGetRootsFromStorage` + Bitfield.open) runs over the facade, with `ready()` and the `checkIfIdle` `opened` guard. |
| 708 | - The constructor sites outside core `wait` on `ready()` in their E0 scope. |
| 709 | - `Session`'s open awaits `core->ready()` first (ix:390). |
| 710 | - **Gate:** ctest, m4b (open / reopen / cross-reopen scripts), m4c, fuzz_core_corpus, r3c. |
| 711 | |
| 712 | **E4. Per-JS-function groups, bottom-up, each on the facade, each removing its own internal Sync scopes (D4 rule).** |
| 713 | 1. **Write primitives:** |
| 714 | - `SessionState::flush` → `P<bool>` and `activeTx_` → `TX`; |
| 715 | - `storeBitfieldRange` in JS's batch shape; |
| 716 | - `_truncate` → `P`; |
| 717 | - `Core::flushHints` → `P<void>`; |
| 718 | - `updateGroupIndex` and `commitBatch` take `CoreTX&`. |
| 719 | |
| 720 | Every locked body that calls these becomes a coroutine body in the same commit, because they are leaves. |
| 721 | 2. **session-state.js ops:** `setUserData`, `clear`, `append` (with waitForRecovery inlined and preappend / |
| 722 | postappend awaited), `truncate` (+ `truncateAndFlush`), `reorg` (assert → real lock), `catchup`, `_verifyBlock` |
| 723 | (assert → real lock), `_commit` / `_oncommit` (Atom Handler → `P<void>`), `commit` / `_overwrite` (+ Core |
| 724 | `_validateCommit`), `_getTreeHeadAt` / `createSession` / `moveTo`, `snapshot` / `_updateSnapshotStorage`, `close` / |
| 725 | `unref` / lingers (D3). |
| 726 | 3. **core.js:** `setManifest` (lock-first order), `setGroup` (and from `_tryOpen`), `verify` with `_verifyExclusive` / |
| 727 | `_verifyShared` and `_verifies` batching (D8), `verifyReorg`, `reorg`, `checkConflict` / `_repairTreeNodes` (atom |
| 728 | flush awaited, D11), `_onconflict` / `closeAllSessions`, `compact`, `audit` + audit.js, `copyPrologue` + |
| 729 | copy-prologue.js (except the stream interleaving: see E5), `close` / `_close` / `destroy`, `missingNodes`. |
| 730 | 4. **mark-bitfield.js:** MarkPage + `loaded`, the Mutex, `loadPage` insert-first; `createMarkStream` waits for E5. |
| 731 | 5. **index.js:** |
| 732 | - `_open` / `_openSession` as one coroutine; |
| 733 | - `whenOpened_` → `if (!opened) co_await opening`; |
| 734 | - `get` / `_get` / `has` / `seek` / `clear` / `truncate` / `append` / `commit` / `setUserData` / `getUserData` |
| 735 | (tryFlush) / `treeHash` / `treeHashFromStorage` / `signable` / `missingNodes` / `proof` (one overload) / |
| 736 | `applyProof` / `verifyFullyRemote` / `recoverFromRemoteProof` / `markBlock` / `clearMarkings` / |
| 737 | `startMarking` / `sweep` / `compact` / `close`. |
| 738 | 6. **fully-remote-proof.js:** `verify` / `proof`, and `generateRemoteProofForTreeNode` → `P<Buf>`. |
| 739 | |
| 740 | Each group is one commit (or a few). As each converts, the matching frontier sites switch from a bare scope to |
| 741 | `wait`, or from a callback to `.then`. |
| 742 | - **Gate per group:** ctest; m4b identical; the group's drill set re-anchored and run red in the same commit |
| 743 | (drill_lint). Groups 2-3 also run m4c / m3c pipes / replicate gates; group 5 also runs m5* (bee / blobs read and |
| 744 | write through Session). |
| 745 | |
| 746 | **E5. The storage-type switch (ONE commit).** The facade and ML2 are deleted. |
| 747 | - **Core:** `HSP` / `CorestoreStorage` / `AtomPtr` / `TX` / `RX` replace the facade aliases. The `StorageRx` / |
| 748 | `StorageTree` bridge goes into core (A0). |
| 749 | - **Streams consumed as JS:** `Bitfield.open` `for await`; `has` counting; `copyPrologue` `for await` with |
| 750 | `flushBatch` inside; audit blocks `for await` with the per-block read; the user-data stream; MarkBitfield's |
| 751 | Transform; `sweep`'s `Promise.all` per batchSize. |
| 752 | - **The external l2 users switch in the same commit, because they share the on-disk store and one RocksDB lock:** |
| 753 | - `store/` (`Tracked::storage`, `Shared::storage`, the ctor's `make_shared`, `getAuth`, `getAlias`, `hasCore`, |
| 754 | `getSeed` / `setSeed` / `ready` / `close`, `createDiscoveryKeyStream`, `getGroup` / `createGroupUpdateStream`): |
| 755 | each wrapped in Sync at store, since R4g converts store later; |
| 756 | - `repl/test/{r3c_walk_test.cpp, r3c_scan_test.cpp, m3c_pipe_test.cpp}`; |
| 757 | - `tools/core-fixture/fixture.h`, `tools/core-ops/main.cpp`, `tools/store-ops/main.cpp`, |
| 758 | `tools/keetd-replicate/main.cpp`; |
| 759 | - `tools/m4d/appender.cpp`, with `test_after_write` moved into rn (D7). |
| 760 | - **What stays on l2 until R4c-4e:** `tools/l2-ops` and l2's own gates and drill sets. |
| 761 | - **Gate:** full ctest; m4b byte-identical (results, stores, OPTIONS, cross-reopen against JS); m4c; m4d (BOUNDARY |
| 762 | on the new numbering); m3c pipes; r3c; m5*; fuzz corpora + campaigns; r4c_st / r4c_mt; replicate gates; P1 |
| 763 | generators; drill sets re-anchored onto storage / core. |
| 764 | |
| 765 | **E6. Core gate `hops` mode + the concurrency scripts.** These are the R4c-4b JS side (`tools/r4c4b/gen-core-hops.js`, |
| 766 | core-ops.js `rulers` / `nowait` / `waitAll`): single ops one at a time; two appends queued on the mutex; four shared |
| 767 | verifies batched (blocks 9 / 11 before 1 / 5); verify against a mutex held by `setUserData`; close during an append |
| 768 | (rejects `RocksDB session is closed`). |
| 769 | - **The C++ side:** `core_ops` gains the same rulers on the virtual runtime at every rn completion. Add |
| 770 | `waitForRecovery` with a pending op once R4f's pipe scenarios exist (not measurable here, R4c-4b). |
| 771 | - **Gate:** m4b hops mode line-for-line against Bare at pool = 1. Any difference is a finding (hop or ordering), and |
| 772 | it is fixed in core. |
| 773 | |
| 774 | Why this order: |
| 775 | - E0 must precede E2: the first hop that delays completion would otherwise break every inline-completion caller. |
| 776 | - E1 lets E2-E4 build against the final API without touching store/. |
| 777 | - E2 precedes E3 / E4: every op's first await is the lock. |
| 778 | - E4's groups go leaf-first. Converting a callee before its callers keeps every intermediate commit buildable: an |
| 779 | unconverted caller reaches a converted callee through a temporary Sync, and the D4 rule says that Sync must not wait |
| 780 | on the mutex the caller holds. |
| 781 | - E5 is the one place where the storage type changes. The design requires that to be a single commit, because core |
| 782 | holds one storage type and store / tests share the files. |
| 783 | - E6 can only be measured on real rn completions. |