| … | … | @@ -0,0 +1,102 @@ |
| 1 | # HyperStorageNode — MVP scope |
| 2 | |
| 3 | Status: SCOPE / DESIGN (2026-07-05, unbuilt). Rationale in `activegraph-synthesis.md` |
| 4 | + `atomspace` org issues #1 (NEXT STEP #4) and #3. This is the smallest artifact that |
| 5 | makes Nakajima's replay / fork / lineage real for an AtomSpace, over P2P. |
| 6 | |
| 7 | ## Goal |
| 8 | A `HyperStorageNode : public StorageNode` — a sibling of the `GpuStorageNode`, |
| 9 | `RocksStorageNode`, `CogStorageNode` in the StorageNode family — whose **op-log is a |
| 10 | Hypercore**, and whose AtomSpace projection is **deterministically replayed from that |
| 11 | log**. Each device's log→projection is one agent (Linas's 5-tuple); log replication |
| 12 | between devices is inter-agent communication. |
| 13 | |
| 14 | The MVP proves ONE claim: *an AtomSpace is a deterministic projection of an |
| 15 | append-only log, and that log can be a Hypercore.* |
| 16 | |
| 17 | ## Precedent (grounded — not slop) |
| 18 | - `GpuStorageNode.h` (this repo): `class GpuStorageNode : public StorageNode` with |
| 19 | `storeAtom / getAtom / loadAtomSpace / storeAtomSpace / barrier / |
| 20 | fetchIncomingByType / storeValue / loadValue / runQuery`. HyperStorageNode |
| 21 | implements the **same BackingStore contract** — copy the shape, swap the backend. |
| 22 | - `CogStorageNode` (repos/atomspace-cog): a StorageNode that serializes atoms as |
| 23 | **s-expressions over TCP** to a cogserver. HyperStorageNode is the same idea with a |
| 24 | different backend: s-expressions **appended to a Hypercore** instead of sent to a |
| 25 | cogserver. |
| 26 | - s-expr codec: `SexprAST` (atomspace core) + the atomspace-cog wire codec. **Reuse |
| 27 | it** — do not invent a serialization. Linas's rule: ASCII over binary. |
| 28 | |
| 29 | ## Architecture (MVP) — UDS bridge to a Bare hypercore daemon |
| 30 | opencog is C++, hypercore is Bare/JS. Mirror the CogStorageNode split rather than |
| 31 | embed a hypercore client in C++: |
| 32 | |
| 33 | ``` |
| 34 | C++ HyperStorageNode Bare `hyper-atomspace` daemon (owns the Hypercore) |
| 35 | ───────────────────── ──────────────────────────────────────────────── |
| 36 | storeAtom(h) ──sexpr op──▶ core.append( sexpr ) (log grows) |
| 37 | loadAtomSpace(as) ─replay─▶ for seq 0..len: yield sexpr ──▶ parse → as.add_atom |
| 38 | barrier() ──flush────▶ await core.update() |
| 39 | (line-oriented s-expr protocol over UDS, hyper-svc pattern) |
| 40 | ``` |
| 41 | |
| 42 | - **C++ side**: thin StorageNode. Emits one s-expr per atom on store; on load, |
| 43 | requests the replay stream and re-parses each line into the target AtomSpace. |
| 44 | `storeAtom` is recursive outgoing-first (as GpuStorageNode already does) → a link's |
| 45 | outgoing atoms are always logged before the link → referential integrity by |
| 46 | construction, no forward references. |
| 47 | - **Bare side**: owns the Hypercore. `append` on store; streamed `get(0..len)` on |
| 48 | replay; `truncate`/checkout for fork. This daemon is the **agent boundary** (inside |
| 49 | = its store, outside = the swarm) — which is exactly the "device = agent" step. |
| 50 | - Reuse `hyper-svc` UDS transport (already in the stack; note the half-open leak fix). |
| 51 | |
| 52 | ## What the MVP proves (three gold checks, GpuBasicSaveUTest discipline) |
| 53 | 1. **Deterministic replay.** Build AtomSpace A → `storeAtomSpace(A)` → fresh AtomSpace |
| 54 | B → `loadAtomSpace(B)` (replays log 0..len) → assert **A ≡ B** (same atoms, same |
| 55 | set). Byte-identical, same bar as the GPU gold check. |
| 56 | 2. **Cheap fork.** Log has length N. Fork = a Hypercore checkout truncated to seq K → |
| 57 | replay into AtomSpace C → assert C is the graph as-of event K and diverges from B. |
| 58 | 3. **Complete lineage.** Each atom's append-seq is its provenance. "What produced atom |
| 59 | X" = the log prefix ending at X's seq → reconstructable; expose as a query. |
| 60 | |
| 61 | ## Scope |
| 62 | **IN (MVP):** |
| 63 | - `storeAtom / getAtom / loadAtomSpace / storeAtomSpace / barrier`. |
| 64 | - Structure only: Nodes (type+name) and Links (type+outgoing). N-ary; link-of-links. |
| 65 | - s-expr op encoding via the existing codec. |
| 66 | - Single-writer Hypercore. |
| 67 | - `fetchIncomingByType` MAY be deferred (rebuild from replay) or added later. |
| 68 | |
| 69 | **OUT (deferred, by design):** |
| 70 | - **Values.** `storeValue`/`loadValue` stay no-ops (as in GpuStorageNode). Per the |
| 71 | synthesis rule *event-source the graph, checkpoint the Values* — Values are the |
| 72 | mutable projection state (Linas Step 4 slice-table), snapshotted separately, NOT |
| 73 | logged. Logging every Value write would detonate the log. |
| 74 | - **Multi-writer.** Swap Hypercore → **Autobase** (keet pattern: `optimistic:true, |
| 75 | ackInterval:1000, ackThreshold:4`) so two daemons write one linearized log = two |
| 76 | agents entraining. This is the P2P generalization; MVP is single-writer first. |
| 77 | - **GPU residency / compute.** End state = a **ProxyNode stack**: `GpuStorageNode` |
| 78 | (compute) atop `HyperStorageNode` (durable P2P log), write-through/read-through. |
| 79 | MVP is HyperStorageNode standalone. |
| 80 | - Attention allocation / GC / conflict policy. |
| 81 | |
| 82 | ## Milestones |
| 83 | - **M0** — s-expr-over-UDS echo: C++ StorageNode ↔ Bare daemon, in-memory array (no |
| 84 | Hypercore). Proves the bridge + codec round-trip. |
| 85 | - **M1** — append + replay to a real Hypercore; gold check #1 (deterministic replay). |
| 86 | - **M2** — fork via Hypercore checkout/truncate; gold check #2. |
| 87 | - **M3** — lineage query; gold check #3. |
| 88 | - **M4** (deferred) — Autobase backend: two daemons, one log = the first real |
| 89 | two-agent test. `HyperBasicSaveUTest` (C++, mirrors GpuBasicSaveUTest) + a Bare |
| 90 | brittle test for append/replay/fork. |
| 91 | |
| 92 | ## Open design question (decide at M0) |
| 93 | UDS-to-Bare-daemon (recommended: reuses the whole stack, matches CogStorageNode |
| 94 | precedent, makes the Bare daemon the agent boundary) **vs** an in-process C++ |
| 95 | hypercore client (fewer moving parts, but no C++ hypercore exists → would mean |
| 96 | porting or FFI, and loses the device-as-agent framing). Recommendation: UDS bridge. |
| 97 | |
| 98 | ## Provenance |
| 99 | VERIFIED: the StorageNode/BackingStore contract (GpuStorageNode.h in this repo), the |
| 100 | CogStorageNode s-expr-over-network precedent, the SexprAST codec, Autobase keet |
| 101 | pattern, hyper-svc UDS. INFERRED/unbuilt: HyperStorageNode itself and the Bare |
| 102 | `hyper-atomspace` daemon. |