| … | … | @@ -0,0 +1,167 @@ |
| 1 | --- |
| 2 | name: atproto-room |
| 3 | description: The room engine under hyper-proto-peer and favela — StateBaseV3 (attestation-gated encrypted multi-writer autobase whose view IS the DID's repo), the roomKey/roomSeed/encryptionKey derivation that makes a DID a room, proof-gated in-apply admission, the presence-based single-signer election that replaced the lease, and the four sign() gates that exist because each one was an outage. Read this BEFORE reasoning about pairing, joining, admission, who signs, or why a chain froze. |
| 4 | category: project |
| 5 | priority: high |
| 6 | location: hyper-fleeky/atproto-room (consumed by hyper-proto-peer, favela-web, favela-android) |
| 7 | updated: 2026-08-09 |
| 8 | --- |
| 9 | |
| 10 | # atproto-room |
| 11 | |
| 12 | The shared room engine. `hyper-proto-peer` composes it into a daemon; `favela-android` bundles that |
| 13 | same daemon in-process. Everything below is grounded in source read on 2026-08-09 — line refs are exact. |
| 14 | |
| 15 | ## ⛔ THE ONE THING TO GET RIGHT: a DID *is* a room, by derivation |
| 16 | |
| 17 | `hyper-proto-peer/lib/device-node.js:125-127`: |
| 18 | |
| 19 | ```js |
| 20 | const roomKey = crypto.hash([NS('room-key'), b4a.from(did)]) // PUBLIC — anyone with the DID |
| 21 | const roomSeed = crypto.hash([NS('room-seed'), b4a.from(sharedKey)]) // SECRET — signing-key holders |
| 22 | const encryptionKey = crypto.hash([NS('enc'), b4a.from(sharedKey)]) // SECRET — signing-key holders |
| 23 | ``` |
| 24 | |
| 25 | `roomSeed` → `deriveGsKeyPair` → `deriveBaseKey` (`state-base.js:95-96, 190-191`), and the base is opened |
| 26 | **encrypted** with `encryptionKey` (`:432`). A non-holder can neither derive the base key nor read a byte. |
| 27 | **Membership secrecy is encryption, not an ACL.** |
| 28 | |
| 29 | So **joining is a derivation, not a handshake**. `bootSwarmDevice` requires exactly four inputs |
| 30 | (`device-node.js:116`) — `did`, `signingKey`, `deviceProof`, `rootIdentityKey` — plus a local |
| 31 | `deviceMnemonic` the device generates and never transmits. Hold those, derive the room, open it. |
| 32 | |
| 33 | ⛔ **Common wrong turns** (all made, all cost time): |
| 34 | - "The invite/bundle must carry the DID." The DID is an *input to a derivation*, not payload. What gates |
| 35 | entry is the **signing key**. |
| 36 | - "Some node must mint/authorize the join." No. See admission below — it is proof-gated **in apply**, |
| 37 | not owner-mediated (`state-base.js:5-8`). |
| 38 | - "The phone should get a delegation instead." Delegation is for something holding **no** key (a browser). |
| 39 | A device holding the signing key derives the room and joins directly. Different products. |
| 40 | |
| 41 | ## Admission — cryptographic, in-apply, not owner-mediated |
| 42 | |
| 43 | Writing needs admission; admission needs no owner: |
| 44 | |
| 45 | 1. `requestWriter()` (`:593-630`) appends a **content-free** optimistic doorbell carrying an admission |
| 46 | proof, then re-rings every 250ms until `writable` (a single optimistic block can be rolled back |
| 47 | before an intermittent indexer acks it). |
| 48 | 2. `_buildAdmissionProof()` (`:1067`) signs `localKey ‖ base.key` — either keet-writer-control |
| 49 | (`{manifest, sig}`) or a keet attestation (`{identity, proof}`). |
| 50 | 3. `_verifyAdmissionInApply()` (`:1094`) re-derives the device key, verifies against the DID's **root |
| 51 | identity** (`membership.js:39-50` → `IdentityKey.verify` with `expectedIdentity`/`expectedDevice`), |
| 52 | resolves device→member→role via `__devices__`/`__members__`, rejects anything in `__removed__`, |
| 53 | enforces the minimum-epoch `__receipt__`. Verified → `addWriter`. |
| 54 | |
| 55 | The only online requirement is **an indexer** to ack. `_ensureWritable` **throws** rather than losing a |
| 56 | write silently. |
| 57 | |
| 58 | ⚠ `__members__` must be projected before a doorbell arrives or the admit verifies with `role=null` and the |
| 59 | member never re-rings. `ready()` awaits `_ensureMembersProjected()` for exactly this (`:269-286`) — the |
| 60 | wj62 fork root. |
| 61 | |
| 62 | ## Roles: creator ≠ indexer ≠ signer |
| 63 | |
| 64 | - **Creator** = lowest-keyed eligible member (`_amCreator:1190-1202`). Emergent from a *mutable* roster, |
| 65 | so adding a device can silently flip it — the loser-club fork. Hence the explicit pins: `joiner: true` |
| 66 | (⇒ `forceNonCreator` + converge-only boot) and `forceCreator` on the incumbent |
| 67 | (`device-node.js:97-112, 178`). Exactly one node holds `gsKeyPair` at a time. |
| 68 | - **Indexer** = autobase checkpointing only. A joiner is admitted a **plain writer**; two indexers is the |
| 69 | N=2 quorum trap. Only a designated anchor grows the set, and autobase only promotes a **fresh |
| 70 | (length-0)** writer (`:352-374`). |
| 71 | - **Signer** needs the `#atproto` key, **not** indexer status. |
| 72 | |
| 73 | ⛔ **An indexer's writer key can never be ROTATED** — so "creator authors genesis, then rotates off |
| 74 | `gsKeyPair` so nobody can collide with a live writer" does NOT work. autobase's `setLocal` asserts on an |
| 75 | indexer, and even fixed it would drop the sole indexer to a plain writer and freeze `indexedLength` with |
| 76 | no way back. The `forceCreator`/`joiner` pins above remain the ONLY thing keeping exactly one holder of |
| 77 | `gsKeyPair` — convention, not cryptography. See [[autobase]] and |
| 78 | `memory/reference_autobase-setlocal-writer-rotation`. |
| 79 | |
| 80 | ## Who signs: presence, not the lease |
| 81 | |
| 82 | `signer-presence.js` **replaced** `lease.js`. The lease rewrote a wall-clock `ts` into an append-only log |
| 83 | every ~8s — ~20MB/day, unreclaimable. The election is now the **lowest-keyed connected co-owner**, |
| 84 | computed in RAM, appending nothing, with a 15s `graceMs` blip tolerance. |
| 85 | |
| 86 | **Currency-aware** (`currentSigner:90`): a co-owner whose announced head is behind the network head is |
| 87 | excluded — otherwise it wins the role while the forward-only gate blocks it and the chain **freezes**. |
| 88 | |
| 89 | ⚠ **Diagnostic trap:** `SwarmDevice.state()` still returns `holdsLease`/`leaseHolder`/`leaseEpoch`, and |
| 90 | `swarm-device.js:739-741` says they are legacy and **`holdsLease` is false on healthy signing nodes**. |
| 91 | Read `holdsSigner`. Likewise use `sb._isCreator`, never `sb._amCreator` (a method — `!!` is always true). |
| 92 | |
| 93 | ## `sign()` — four gates, each one an incident (`swarm-device.js:343`) |
| 94 | |
| 95 | 1. **Lazy `restore()` first**, so the currency-aware election knows if *we* are caught up. |
| 96 | 2. **Election gate** — `holdsSigner(undefined, net.rev)`; behind ⇒ return, let the caught-up peer sign. |
| 97 | 3. **Anti-wipe, two arms** (`:371-404`). A repo with a head but an empty record set is never "everything |
| 98 | was deleted": arm 1 catches record *files* present with pointers not yet backfilled (the wj62 wipe); |
| 99 | arm 2 catches *inline* records in a legacy repo flipped onto the pointer substrate (e2jev), where both |
| 100 | the pointer set and the file set read empty. Either would emit a delete-every-record commit. |
| 101 | 4. **Forward-only gate** (`:405-470`). Adopt the highest-rev `__commit__` frame as head (kills rev |
| 102 | regression), replay every op up to it, refuse to sign if our view is missing any key (kills content |
| 103 | regression). Three states, only the first may be signed forward: |
| 104 | - explicit **tombstone** → genuinely deleted → sign the delete forward |
| 105 | - live pointer, bytes unresolved → **behind** → skip |
| 106 | - nothing at all → **behind** → skip (misreading this as a delete is how a lagging co-owner published |
| 107 | deletes for live records) |
| 108 | `__tombstone_epoch__` carves out legacy hard-deletes so they don't wedge the chain (delete-tip wedge). |
| 109 | |
| 110 | `_nextRev()` is `nextRev(head.rev)` — deterministic prev+1µs, not wall-clock — so two co-owners continuing |
| 111 | the same chain emit **byte-identical** commits (materialize is RFC6979-deterministic, `signer.js:1-5`). |
| 112 | Because revs stopped being wall-clock, `serializeEntry` records a **real** `time` once at the signer; |
| 113 | deriving it from the rev would freeze near the first commit and relays read `time` as liveness. |
| 114 | |
| 115 | ## Records + firehose layout |
| 116 | |
| 117 | `repo-base.js`: key = `collection/rkey`. Reserved = single-segment keys and any collection starting `__`, |
| 118 | which is what keeps `__commit__/<rev>`, `__members__`, `__lease__`, `__prefs__`, `__device__/*` out of the |
| 119 | signed repo. Deletes write an explicit **tombstone** — absence is ambiguous in a replicated log, and |
| 120 | absence-as-deleted makes a lagging device publish deletes for live records. |
| 121 | |
| 122 | With an `OverlayRepoStore`, `PointerSubstrate` puts a `{$oid,size}` **pointer in the autobase** (so the |
| 123 | record *set* converges) and the **bytes in deduped overlay files**. The firehose lives in the **same** |
| 124 | StateBase under `__commit__/<rev>` — no second Hypercore, continuous across signer hand-offs. |
| 125 | |
| 126 | ## Non-blocking discipline (this is load-bearing, not style) |
| 127 | |
| 128 | `applyWrite` and `state()` count **raw pointer keys** and never call `repo.listRecords()` — under |
| 129 | `PointerSubstrate` that resolves bytes via `ensureLocal`, an overlay fetch-by-oid that blocks when the |
| 130 | node lacks the bytes. That was a 20s `apply-write` timeout and an 8s+ hang on the liveness probe |
| 131 | (`swarm-device.js:662-671, 717-734`). Snapshot reads (`listNow`/`getNow`) everywhere hot; |
| 132 | `_updateBounded` (`state-base.js:646`) bounds every `base.update()` await. |
| 133 | |
| 134 | ## Serving (`serve-membrane.js`) |
| 135 | |
| 136 | Each device projects the shared `__commit__` log into its own FirehoseSequencer + repo core and |
| 137 | **push-announces** `atproto/did-announce/v1` on connect with `{did, handle, repoKey, firehoseKey, |
| 138 | lengths}` — no round trip, no directionality. `__commit__` is rev-ordered and deterministic, so every |
| 139 | device projects the **same** firehose: the keyless bridge can front any device and reconnect to another. |
| 140 | The repo core advances first and the firehose only if that succeeded (`:256-274`) — they are read as a |
| 141 | pair, so on failure neither advances and the previously-served head stays consistent instead of 500ing. |
| 142 | `swarm.join(discoveryTopic(did))` — the DID is itself a swarm topic. |
| 143 | |
| 144 | ⚠ **Two RPC surfaces exist** — `serve-membrane.js` (what the bridge talks to) and hpp's |
| 145 | `serve-device.js`. `:325-328` records the symptom of patching only one: `getBlob` worked while |
| 146 | `listBlobs` answered UNKNOWN_METHOD with the handler demonstrably deployed. Keep them in step. |
| 147 | |
| 148 | ## Reading map |
| 149 | |
| 150 | - `state-base.js` — `ready`/`_open`/`_openBase`, `requestWriter`, `_buildAdmissionProof`, |
| 151 | `_verifyAdmissionInApply`, `_amCreator`, `_maybePromote`, `_updateBounded` |
| 152 | - `membership.js` — `makeAttest`/`makeVerifyAttest`, epoch receipts |
| 153 | - `signer.js` — `materialize`, `buildCommitWrite` (+ the covering proof that asserts invertibility) |
| 154 | - `signer-presence.js` — the election; `lease.js` is superseded, kept for continuity |
| 155 | - `repo-base.js` — key convention, reserved namespace, tombstones |
| 156 | - `serve-membrane.js` — projection pair, announce-push, the bridge-facing RPC set |
| 157 | - `hyper-proto-peer/lib/swarm-device.js` — the device object: `sign`, `applyWrite`, `delegate`, `state` |
| 158 | - `hyper-proto-peer/lib/device-node.js:96-215` — derivation + boot posture flags |
| 159 | |
| 160 | ## Related |
| 161 | |
| 162 | - [[hyper-proto-peer]] — the daemon that composes this |
| 163 | - [[autobase]] — admission doorbell, N=2 quorum trap, fresh-writer promotion |
| 164 | - [[atproto-store]] — `PointerSubstrate`, `OverlayRepoStore`, `FirehoseSequencer` |
| 165 | - [[atproto-did-peer]] / [[atproto-bridge]] — the keyless consumers |
| 166 | - memory: [[reference_statebase-indexing-vs-did-signing]], [[project_wj62-write-path-saga-2026-07-24]], |
| 167 | [[reference_autobase-quorum-2f1]], [[project_retention-clobber-wj62-2026-07-31]] |