OverviewFilesCommitsIssuesDocsReleasesPeers

atproto-base commits / 6a79bfb

atproto-base: README + consumer-extraction session plan
README.md — what it is (deterministic projection of one autobase → single-DID atproto), the reconciliation (indexer-agnostic signing, no fork), API surface, modules, and what it's grounded in (keet handlers, atproto-repo/store codecs). EXTRACTION-PLAN.md — the offline-first, staged plan to wire it into the live stack: Phase 0 closes the deliberate simplifications (deterministic base key, encryption, keet-attestation admission, repo-for-bridge projection, migration reader) BEFORE any cutover; Phase 1 unifies the 3x consumer primitive; Phase 2 rehearses migration on a snapshot; Phase 3 stages cutover (throwaway DID → loser-club → wj62 last); Phase 4 is the favela shared-appview feed-room. Leads with the risk: touches live accounts, and clean-testnet passes are not proof for flaky-link/live scenarios. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
6a79bfb156578f80779f3ff5ec2fb79f262324eb
by plankatron <fleeky@gmail.com> · 2026-07-23T19:17:36.000Z

Changed files (2) +184 -0

EXTRACTION-PLAN.md +82 -0
……@@ -0,0 +1,82 @@
1# Consumer extraction — session plan
2
3Repoint hpp / atproto-room / atproto-bridge / favela onto `atproto-base`, retire the 3× duplicate
4consumer primitive, and (separately) fix favela's per-node-appview divergence with a shared feed-room.
5This TOUCHES LIVE ACCOUNTS (wj62 `p2p.7rnx.net`, loser-club, ashatron) — **offline-first, staged, with
6a rehearsed migration.** Do NOT cut a live DID over on a clean-testnet's word (every autobase failure
7this session was flaky-link/concurrency-specific — see the session memories).
8
9## Where atproto-base stands (start point)
1019 tests / 93 asserts green: deterministic projection · record engine · admission · finalized-view ·
11indexer topology (keet-correct) · revocation · firehose (`#commit` frames + `subscribeRepos`). But it
12was built with SIMPLIFICATIONS that must close before it can replace StateBaseV3 for a REAL co-owned DID.
13
14## PHASE 0 — close the atproto-base gaps (all OFFLINE, extend the test suite). BLOCKING.
15Grounded in atproto-room/lib/state-base.js (the god-class being replaced):
161. **Deterministic base key.** StateBaseV3 derives the autobase base key from a members-only
17 `roomSeed` (`deriveGsKeyPair(roomSeed, GS_NAMESPACE)` → `deriveBaseKey`) so every co-owner opens the
18 SAME base. atproto-base currently uses corestore-derived keys + explicit `bootstrap`. Port the
19 deterministic derivation + the `forceCreator`/`forceNonCreator` resync postures (needed for the
20 fresh-corestore heal). Test: two co-owners with the same roomSeed open the same base with no shared bootstrap.
212. **Encryption.** Co-owned DIDs run an ENCRYPTED base keyed by a members-only key. atproto-base takes
22 `encryptionKey` but wire the members-only derivation + `encrypted:true`. Test: a non-member can't read.
233. **keet-attestation admission proof.** atproto-base admits via a raw ed25519 `sign(localKey‖base.key)`;
24 real co-owner devices use keet-writer-control (manifest + device→member→role, `_verifyAdmissionInApply`).
25 Add the keet-attestation proof path alongside the raw one (injectable `attest`/`verifyAttest`). Test
26 against a real device chain (reuse hpp's identity fixtures).
274. **Repo projection for the bridge.** The bridge's PeerFront reads a `RepoStore` + `FirehoseSequencer`
28 (getRepo assembles a CAR from the RepoStore; the sig comes off the firehose). atproto-base emits the
29 firehose; confirm/ADD a repo projection (an atproto-store RepoStore fed from the finalized record set)
30 so `getRepo` works, OR decide the bridge reads the firehose and reconstructs. Ground this by reading
31 PeerFront.getRepo (`atproto-bridge/pds/lib/peer-front.js`) end-to-end first.
325. **Records migration reader.** A function to read an EXISTING StateBaseV3 DID's records (the
33 `@state/entries`/PointerSubstrate view) and import them into atproto-base storage — the migration input.
34
35## PHASE 1 — unify the consumer primitive (offline)
36The "replicate membrane + follow firehose + verify head" primitive exists 3×: `PeerFront`/`PeerRouter`
37(bridge, best — multi-front + freshest-by-rev), `atproto-did-peer` `DidPeer` (favela, behind — single
38front, no failover), and a dormant bridge `did-peer.js`. Pick ONE canonical (extend PeerRouter's
39multi-front + freshest-live logic — it already handles co-owned multi-membrane), make it read
40atproto-base's firehose/repo, and repoint favela's `atproto-did-peer` at it. Retire the other two.
41Fold in the bridge front-liveness fix from this session's audit (staleStrikes on `rpcConnected:false`,
42never relay from a dead front) — that was a separate open bug (hyper-social #10/#11).
43
44## PHASE 2 — migration, rehearsed on a SNAPSHOT (never hot)
45Per the wj62 discipline (`reconcileFromCar` on FRESH storage; rocksdb hot-copy is inconsistent):
46stop a daemon, snapshot its storage, run the Phase-0 migration reader → atproto-base on a THROWAWAY
47copy, boot it, verify the projected head matches the DID's current bsky head (`getRepo` byte-compare).
48Only after a clean offline rehearsal on real data do we touch the live daemon.
49
50## PHASE 3 — staged cutover (lowest-risk DID first)
511. A THROWAWAY test DID (no bsky account) end-to-end: hpp on atproto-base → bridge → `requestCrawl` → bsky.
522. **loser-club** (single-creator, no co-owner quorum → lowest risk) — migrate, boot-test the BUILT
53 `--production` closure (`npm run smoke`, not just units — test-green ≠ deploy-ready), deploy via
54 `pgh publish` (NOT git-push CI — hpp fleet has autoupdate off; `pgh app update`+restart), verify e2e.
553. **wj62** (co-owned, quorum, HIGHEST risk — currently in a gen-2 resync/heal state) LAST, only after 1+2
56 are stable for a soak. Rehearse the co-owned handoff offline (both co-owners on atproto-base, converge,
57 post, verify no fork) before touching prod.
58
59## PHASE 4 — favela shared-appview (the original divergence, separate track)
60Replace favela-web's per-node local `Appview` (never replicated → tabertron≠ashatron) with a **co-owned
61atproto-base feed-room**: the community feed is an atproto-base instance whose `apply()` folds member
62posts (favela's already-deterministic `applyCommit`), that ALL favela nodes REPLICATE and read — one
63shared view, no per-node re-index. "Multiple feed-rooms = multiple feeds." Depends on Phase 0 (shared
64base key) + Phase 1 (one consumer primitive). See [[reference_shared-autobase-view-keet-blueprint]].
65
66## Verification discipline (every phase)
67- Offline test FIRST; a clean-testnet pass is NOT proof for a flaky-link/live scenario.
68- Boot-test the actual `--production` closure before any deploy (declare-your-deps footgun, 2 past outages).
69- e2e = post → bridge → bsky visible; `getRepo` byte-matches; no fork (`wj62-state.mjs`-style diagnose).
70- Rollback: keep the old StateBaseV3 storage untouched (nothing deleted); a bad cutover reverts by
71 pointing the daemon back at the old app version.
72
73## Risks / open decisions to settle at session start
74- Does the bridge consume atproto-base's FIREHOSE directly, or a membrane repo projection? (Phase-0 #4 read decides.)
75- Migration determinism: does re-projecting existing records reproduce the SAME `rev` chain the relay
76 already has? (Forward-only relay cursor — a rev regression parks the host Idle, federation-wall #3.)
77 Likely need to continue the chain from the current head (adopt), not re-derive from rev 0.
78- Multi-indexer reliability at N≥3 under real links is still uncharacterized — wj62 co-owned cutover
79 should stay N=1 (single designated indexer) + notary-revival recovery until N≥3 is soak-tested.
80
81Related session artifacts: `project_atproto-base-deterministic-projection-2026-07-23`,
82`reference_pptaproto-architecture-holes`, `project_atproto-storage-consensus-redesign-2026-07-19`.
README.md +102 -0
……@@ -0,0 +1,102 @@
1# atproto-base
2
3**A co-owned atproto DID as a deterministic projection of one autobase.**
4
5atproto wants a single-DID, single-signing-key, append-only *signed* log. Autobase is a
6multi-writer log whose **indexer set changes** (writers are added/removed/promoted, indexers move,
7migrations happen). Every previous attempt to marry them tried to elect a *stable single signer* — but
8a P2P multi-writer system has no stable authority, so a handoff would either stall (0 signers) or fork
9(2 signers). atproto-base resolves this differently:
10
11> **The atproto commit chain is a PURE DETERMINISTIC PROJECTION of the autobase INDEXED view, signed
12> with RFC6979-deterministic ECDSA. Every co-owner independently computes the byte-identical signed
13> commit — so *who* signs is irrelevant. No signer election, no possible fork, indexer-change-agnostic.**
14
15- The indexed prefix `[0, indexedLength)` is immutable even across an indexer change — so a commit
16 signed against it can never be rewritten.
17- Liveness (advancing `indexedLength`) is autobase's quorum job: **N=1 sole indexer, or N≥3, never N=2.**
18- A group/team **labeler** is the same primitive projecting `com.atproto.label` instead of repo commits.
19
20See [`DESIGN.md`](./DESIGN.md) for the full rationale, and [`EXTRACTION-PLAN.md`](./EXTRACTION-PLAN.md)
21for wiring it into the live stack.
22
23## Install
24
25```sh
26npm install
27npm test # 19 tests / 93 asserts, all offline
28```
29
30## Quick start
31
32```js
33import { AtprotoBase } from 'atproto-base'
34import { deriveAtprotoKey, createAtprotoIdentity } from 'atproto-identity'
35import crypto from 'hypercore-crypto'
36
37const id = createAtprotoIdentity(mnemonic, { handle, pdsUrl })
38const key = deriveAtprotoKey(mnemonic) // the shared #atproto signing key
39const me = crypto.keyPair() // this co-owner's membership key
40
41// Creator (bootstrap sole indexer). members = the co-owner roster (role 1 = admin).
42const base = new AtprotoBase({
43 path: './data', did: id.did, privateKey: key.privateKey,
44 identity: me, members: [{ key: me.publicKey.toString('hex'), role: 1 }]
45})
46await base.ready()
47
48await base.putRecord('app.bsky.feed.post', '3labc...', { $type: 'app.bsky.feed.post', text: 'hi', createdAt, langs: ['en'] })
49
50const head = await (await base.settled(), base.head()) // the finalized, relay-valid signed commit (CAR)
51const write = await base.emitFirehose() // emit a #commit frame; subscribeRepos() serves it
52```
53
54A joining co-owner opens with `{ bootstrap: creatorBaseKey, identity }`, is admitted via the
55block-until-admitted doorbell (`ensureWritable()`), and reads/writes the *same* shared view.
56
57## API (`AtprotoBase`)
58
59| | |
60|---|---|
61| `new AtprotoBase({ path\|store, bootstrap?, keyPair?, encryptionKey?, did, privateKey, identity?, members?, ackInterval? })` | one autobase per DID |
62| `ready()` / `close()` | lifecycle |
63| `putRecord(collection, rkey, record)` / `delRecord(collection, rkey)` | typed record ops (block-until-admitted) |
64| `head({ prevRev?, prev? })` | project the finalized head → `{ carBytes, commitCid, dataCid, signedCommit }` (relay-valid) |
65| `indexedRecords()` | the FINALIZED record set (reads at the view core's `signedLength`, never the optimistic tip) |
66| `settled()` | await finalization (the chain advances on finalization, not on write) |
67| `emitFirehose()` | emit a `#commit` frame for a finalized-head advance (no-op if unchanged) |
68| `subscribeRepos(cursor)` / `firehoseKey` / `firehoseLength` | the `com.atproto.sync.subscribeRepos` firehose |
69| `projectMembers(roster)` | admin/bootstrap: project `__members__`/`__devices__` |
70| `ensureWritable()` | ring the admission doorbell; throws if no indexer admits (never a silent loss) |
71| `requestIndexer()` / `stopIndexing()` | keet-aligned self-promote/demote (admin-gated) — grow N=1→N=3 |
72| `ban(identityHex)` | keet remove-member: removeWriter + block re-admission (⚠ not a PLC key rotation) |
73
74## Modules
75
76- **`lib/project.js`** — the deterministic commit projection (`projectCommit`, prev-chaining; reuses
77 atproto-repo's byte-exact MST/commit primitives).
78- **`lib/rev.js`** — deterministic `rev` (a log-derived TID, *never* wall-clock — the load-bearing edge).
79- **`lib/sign.js`** / **`lib/label.js`** — generic DAG-CBOR signer + the label projection (labeler).
80- **`lib/base.js`** — `AtprotoBase`: the autobase record engine + typed governance ops + firehose.
81- **`lib/firehose.js`** — `buildCommitWrite`: diffs consecutive heads into a `#commit` (ops + minimal
82 MST proof blocks + changed record blocks + prevData); reuses atproto-store's byte-exact codec.
83
84## What it's grounded in
85
86Every governance op mirrors keet's real contract handlers (read in full from keet-core 4.16
87`members/index.js`): `start-indexing`/`stop-indexing` (self-promote if `canAdmin`),
88`remove-member` (ban). Deliberately **not** ported: keet's `move-indexer` (a same-member device→device
89handoff, misread by the predecessor as a cross-member creator-demotion). Commit/firehose codecs reuse
90`atproto-repo` (verified byte-exact against real relay-accepted repos) and `atproto-store` (what the
91bridge already consumes).
92
93## Status
94
95All correctness-critical pieces are proven offline (19 tests / 93 asserts). It is **not yet wired into
96the live stack** — that is the consumer extraction ([`EXTRACTION-PLAN.md`](./EXTRACTION-PLAN.md)), which
97first closes a few deliberate simplifications (deterministic base key, encryption, keet-attestation
98admission, repo-for-bridge projection) before any live cutover.
99
100## License
101
102Apache-2.0