OverviewFilesCommitsIssuesDocsReleasesPeers

atproto-store commits / f0677e3

0.0.18: migrateInline — inline records in the autobase become fetchable pointers
The inverse gap to backfill(). backfill covers "bytes on disk, no entry in the autobase"; this covers "an entry that is a full RECORD where a { $oid, size } pointer belongs" — written through the legacy substrate against a DID that now runs the pointer design. Such a repo reads perfectly on the node holding it and is UNFETCHABLE BY EVERY OTHER NODE: _resolve passes an undefined $oid to ensureLocal, which falls back to the local file, so records resolve locally while no peer has an oid to fetch by. A co-owner converges the keys and can never turn them into bytes. Confirmed live on loser-club, whose three entries are all inline records (pointerFields ["$type","did","displayName",...] etc. via hyper-proto-peer 0.3.78) — so that DID could not be co-owned at all regardless of roster or config work. backfill() cannot do it: it skips any key that already has a value, correctly (never clobber a converged pointer or tombstone), and an inline record is such a value. CONTENT IS PRESERVED, NOT CHOSEN. What the repo publishes today is the LOCAL FILE, not the inline copy, so the pointer must address the file's bytes or the signed repo would change underneath the migration. The record is re-put from disk — which registers it with the overlay, the whole point, and yields its oid — and the result is ASSERTED against what the file already hashed to: on a mismatch the key is skipped and counted (`mismatched`) rather than silently republishing different content. Only with no local file at all does the inline copy become the source, since then it is the only copy in existence. Idempotent, reserved keys untouched, tombstones never resurrected, nothing deleted (set appends; the old value stays in oplog history). Writable-only — the caller gates on base.writable. 5 tests / 24 asserts, including a control proving a peer CANNOT resolve an inline entry and can once migrated. 64/64. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
f0677e372559e83acb27d71ca24b169a000ceddd
by plankatron <fleeky@gmail.com> · 2026-07-27T20:07:55.000Z

Changed files (3) +232 -1

lib/pointer-substrate.js +53 -0
……@@ -116,6 +116,59 @@ export class PointerSubstrate {
116116 // and yields the oid without trusting stale overlay index state. WRITABLE-ONLY: statebase.set appends
117117 // to the autobase, so only an admitted writer (the creator) may run this — the caller gates on
118118 // base.writable. Returns { written, skipped, total }.
119 // ── MIGRATION: INLINE RECORD → POINTER (idempotent) ────────────────────────────────────────────
120 // The inverse gap to backfill(). backfill covers "bytes on disk, no entry in the autobase"; this
121 // covers "an entry in the autobase that is a full RECORD instead of a { $oid, size } pointer" —
122 // written through the LEGACY substrate against a DID that now runs the pointer design.
123 //
124 // Such a repo reads perfectly on the node that holds it and is UNFETCHABLE BY EVERY OTHER NODE:
125 // _resolve passes ptr.$oid (undefined) to ensureLocal, which falls back to the local file, so the
126 // records resolve locally and no peer has an oid to fetch by. A co-owner would converge N keys it can
127 // never turn into bytes. Live on loser-club: three records, all inline, so that DID cannot be
128 // co-owned at all — the config and roster work is wasted until this is fixed.
129 //
130 // backfill() cannot do it: it SKIPS any key that already has a value (`if (existing != null)`), which
131 // is correct for its job — never clobber a converged pointer or tombstone — but an inline record IS
132 // such a value.
133 //
134 // CONTENT IS PRESERVED, NOT CHOSEN. What the repo currently publishes is the LOCAL FILE (that is what
135 // ensureLocal returns today), not the inline copy in the log — so the pointer must address the file's
136 // bytes or the signed repo would change underneath us. We therefore re-put the record read back from
137 // disk, which registers it with the overlay (making it fetchable, the whole point) and returns its
138 // oid; DAG-CBOR is canonical so the rewrite is byte-identical. That is ASSERTED, not assumed: if the
139 // resulting oid differs from what the file already hashed to, the key is skipped and counted rather
140 // than silently republishing different content. Only when there is no local file at all do we fall
141 // back to the inline record — then it is the only copy in existence.
142 //
143 // Nothing is deleted (statebase.set appends; the old value stays in oplog history) and files are
144 // never modified beyond a byte-identical rewrite. WRITABLE-ONLY: the caller gates on base.writable.
145 // Returns { converted, alreadyPointer, tombstones, fromInline, mismatched, failed, total }.
146 async migrateInline () {
147 const raw = await this.statebase.list('')
148 let converted = 0; let alreadyPointer = 0; let tombstones = 0; let fromInline = 0; let mismatched = 0; let failed = 0; let total = 0
149 for (const key of Object.keys(raw)) {
150 const s = recSplit(key)
151 if (!s) continue // reserved/control key — never part of the record pointer set
152 total++
153 const v = raw[key]
154 if (isTombstone(v)) { tombstones++; continue }
155 if (v && v.$oid) { alreadyPointer++; continue } // already the right shape — idempotent
156 if (!v || typeof v !== 'object') { failed++; continue }
157 try {
158 const before = (typeof this.store.localOid === 'function') ? await this.store.localOid(s.collection, s.rkey) : null
159 const onDisk = await this.store.get(s.collection, s.rkey)
160 const record = onDisk != null ? onDisk : v // the file is what resolves today; the inline copy is the fallback
161 if (onDisk == null) fromInline++
162 const { oid, size } = await this.store.put(s.collection, s.rkey, record)
163 if (!oid) { failed++; continue } // no overlay → no oid to publish; leave the entry alone
164 if (before && oid !== before.oid) { mismatched++; continue } // would change published content — refuse
165 await this.statebase.set(key, { $oid: oid, size })
166 converted++
167 } catch { failed++ }
168 }
169 return { converted, alreadyPointer, tombstones, fromInline, mismatched, failed, total }
170 }
171
119172 async backfill () {
120173 if (typeof this.store.records !== 'function') {
121174 throw new Error('PointerSubstrate.backfill: the store must enumerate records() to migrate files → pointers')
package.json +1 -1
……@@ -1,6 +1,6 @@
11{
22 "name": "atproto-store",
3 "version": "0.0.17",
3 "version": "0.0.18",
44 "description": "Holepunch-native atproto repo store — records in a classic Hyperbee (collection/rkey -> DAG-CBOR). The P2P home; the atproto-repo module is the bridge serializer over it.",
55 "type": "module",
66 "main": "lib/repo-store.js",
test/migrate-inline.test.js +178 -0
……@@ -0,0 +1,178 @@
1// PointerSubstrate.migrateInline — convert INLINE RECORDS in the autobase into fetchable pointers.
2//
3// A DID written through the LEGACY substrate but now running the pointer design has full records where
4// { $oid, size } pointers belong. It reads perfectly on the node holding it — _resolve passes an
5// undefined $oid to ensureLocal, which falls back to the local file — and is UNFETCHABLE BY EVERY OTHER
6// NODE, because there is no oid to fetch by. A co-owner converges the keys and can never turn them into
7// bytes.
8//
9// Live on loser-club (2026-07-27): all three records inline
10// (app.bsky.feed.generator/loser-club, app.bsky.feed.post/self.hello, net.favela.federation/self),
11// so that DID cannot be co-owned at all regardless of roster or config.
12//
13// backfill() cannot fix it: it skips any key that already has a value, which is correct for its job
14// (never clobber a converged pointer or tombstone) but an inline record is such a value.
15//
16// The load-bearing property is that CONTENT IS PRESERVED. What the repo publishes today is the LOCAL
17// FILE, not the inline copy, so the pointer must address the file's bytes — otherwise the signed repo
18// changes underneath the migration.
19//
20// Run: brittle-node test/migrate-inline.test.js
21import test from 'brittle'
22import fs from 'fs'
23import os from 'os'
24import path from 'path'
25import crypto from 'hypercore-crypto'
26import * as dagCbor from '@ipld/dag-cbor'
27import { OverlayRepoStore } from '../lib/overlay-repo-store.js'
28import { PointerSubstrate } from '../lib/pointer-substrate.js'
29
30const tmp = () => fs.mkdtempSync(path.join(os.tmpdir(), 'mig-'))
31const GEN = 'app.bsky.feed.generator'
32const POST = 'app.bsky.feed.post'
33
34function memKV () {
35 const m = new Map()
36 return {
37 _m: m,
38 async set (k, v) { m.set(k, v) },
39 async del (k) { m.delete(k) },
40 async get (k) { return m.has(k) ? m.get(k) : null },
41 async list (prefix = '') { const o = {}; for (const [k, v] of m) if (k.startsWith(prefix)) o[k] = v; return o },
42 async listNow (prefix = '') { return this.list(prefix) }
43 }
44}
45
46function mkOverlay (net) {
47 return {
48 async registerFile (overlayPath, diskPath) {
49 const bytes = fs.readFileSync(diskPath)
50 const oid = crypto.data(bytes).toString('hex')
51 net.set(oid, bytes)
52 return { contentHash: oid }
53 },
54 async fetchFile (oid, o = {}) {
55 const bytes = net.get(oid)
56 if (!bytes) return null
57 fs.writeFileSync(o.destPath, bytes)
58 return { destPath: o.destPath, size: bytes.length }
59 },
60 async hashFile (p) {
61 let bytes
62 try { bytes = fs.readFileSync(p) } catch { return null }
63 return { contentHash: crypto.data(bytes).toString('hex'), size: bytes.length }
64 }
65 }
66}
67
68const GEN_REC = { $type: GEN, did: 'did:web:x', displayName: 'loser-club', description: 'a feed', createdAt: '2026-07-01T00:00:00.000Z' }
69const POST_REC = { $type: POST, text: 'hello from loser-club', createdAt: '2026-07-01T00:00:00.000Z' }
70
71// Reproduce the live shape: the INLINE RECORD is in the autobase and the record FILE is on disk.
72async function seedInline (sb, store, collection, rkey, record) {
73 fs.mkdirSync(path.join(store._recordsDir, collection), { recursive: true })
74 fs.writeFileSync(path.join(store._recordsDir, collection, rkey + '.dagcbor'), Buffer.from(dagCbor.encode(record)))
75 await sb.set(collection + '/' + rkey, record) // the legacy substrate wrote the whole record here
76}
77
78test('inline records become pointers that address the SAME bytes (published content unchanged)', async (t) => {
79 const net = new Map()
80 const root = tmp()
81 const store = await new OverlayRepoStore(root, { overlay: mkOverlay(net), did: 'did:plc:lc' }).ready()
82 const sb = memKV()
83 const sub = new PointerSubstrate(sb, store)
84
85 await seedInline(sb, store, GEN, 'loser-club', GEN_REC)
86 await seedInline(sb, store, POST, 'self.hello', POST_REC)
87
88 // Precondition: the entries are records, not pointers, and nothing is fetchable by oid.
89 t.absent((await sb.get(GEN + '/loser-club')).$oid, 'precondition: the autobase holds a record, not a pointer')
90 const beforeOid = (await store.localOid(GEN, 'loser-club')).oid
91 const beforeRec = await sub.get(GEN + '/loser-club')
92 t.alike(beforeRec, GEN_REC, 'precondition: it resolves locally (via the file fallback)')
93
94 const r = await sub.migrateInline()
95 t.is(r.converted, 2, 'both inline records converted')
96 t.is(r.mismatched, 0, 'no key was refused for changing content')
97 t.is(r.failed, 0, 'and none failed')
98
99 const ptr = await sb.get(GEN + '/loser-club')
100 t.ok(ptr.$oid, 'the entry is now a pointer')
101 t.is(ptr.$oid, beforeOid, 'addressing EXACTLY the bytes that were already resolving — the repo does not change')
102 t.is(ptr.size, Buffer.from(dagCbor.encode(GEN_REC)).length, 'and carries the record\'s real byte length')
103 t.alike(await sub.get(GEN + '/loser-club'), GEN_REC, 'the record still resolves, byte-identical')
104 t.ok(net.has(ptr.$oid), 'the bytes are now registered with the overlay — a PEER can fetch them by oid')
105})
106
107test('a peer holding only the pointer can now resolve the record (the whole point)', async (t) => {
108 const net = new Map()
109 const store = await new OverlayRepoStore(tmp(), { overlay: mkOverlay(net), did: 'did:plc:lc' }).ready()
110 const sb = memKV()
111 const sub = new PointerSubstrate(sb, store)
112 await seedInline(sb, store, POST, 'self.hello', POST_REC)
113
114 // BEFORE: a co-owner replicates the entry. It has no oid, so it can never become bytes.
115 const peerStore = await new OverlayRepoStore(tmp(), { overlay: mkOverlay(net), did: 'did:plc:lc' }).ready()
116 const peerSb = memKV()
117 await peerSb.set(POST + '/self.hello', await sb.get(POST + '/self.hello'))
118 const peerSub = new PointerSubstrate(peerSb, peerStore)
119 t.is(await peerSub.get(POST + '/self.hello'), null, 'CONTROL: the peer cannot resolve an inline entry — no oid to fetch by')
120
121 await sub.migrateInline()
122
123 const peerSb2 = memKV()
124 await peerSb2.set(POST + '/self.hello', await sb.get(POST + '/self.hello')) // replicate the POINTER
125 const peerSub2 = new PointerSubstrate(peerSb2, await new OverlayRepoStore(tmp(), { overlay: mkOverlay(net), did: 'did:plc:lc' }).ready())
126 t.alike(await peerSub2.get(POST + '/self.hello'), POST_REC, 'after migration the peer fetches the record by oid')
127})
128
129test('idempotent: already-correct pointers and tombstones are left alone', async (t) => {
130 const net = new Map()
131 const store = await new OverlayRepoStore(tmp(), { overlay: mkOverlay(net), did: 'did:plc:lc' }).ready()
132 const sb = memKV()
133 const sub = new PointerSubstrate(sb, store)
134
135 await sub.set(POST + '/proper', POST_REC) // a real pointer
136 await sb.set(POST + '/dead', { $deleted: true }) // a tombstone
137 await seedInline(sb, store, GEN, 'loser-club', GEN_REC) // one inline record
138
139 const first = await sub.migrateInline()
140 t.is(first.converted, 1, 'only the inline record was converted')
141 t.is(first.alreadyPointer, 1, 'the healthy pointer was skipped')
142 t.is(first.tombstones, 1, 'the tombstone was skipped — never resurrected')
143
144 const second = await sub.migrateInline()
145 t.is(second.converted, 0, 'a second run converts nothing')
146 t.is(second.alreadyPointer, 2, 'everything is now a pointer')
147 t.alike(await sb.get(POST + '/dead'), { $deleted: true }, 'the tombstone is untouched')
148})
149
150test('reserved keys are never touched', async (t) => {
151 const net = new Map()
152 const store = await new OverlayRepoStore(tmp(), { overlay: mkOverlay(net), did: 'did:plc:lc' }).ready()
153 const sb = memKV()
154 const sub = new PointerSubstrate(sb, store)
155
156 await sb.set('__commit__/3kzzz', { rev: '3kzzz', ops: [] })
157 await sb.set('__members__', { a: 1 })
158 await sb.set('__blob__/bafkreiX', { $oid: 'abc', size: 1 })
159
160 const r = await sub.migrateInline()
161 t.is(r.total, 0, 'no reserved key was considered a record')
162 t.is(r.converted, 0, 'and nothing was rewritten')
163 t.alike(await sb.get('__commit__/3kzzz'), { rev: '3kzzz', ops: [] }, 'the firehose frame is intact')
164})
165
166test('with no local file the inline record itself is used (it is the only copy)', async (t) => {
167 const net = new Map()
168 const store = await new OverlayRepoStore(tmp(), { overlay: mkOverlay(net), did: 'did:plc:lc' }).ready()
169 const sb = memKV()
170 const sub = new PointerSubstrate(sb, store)
171
172 await sb.set(POST + '/orphan', POST_REC) // inline entry, NO file on disk
173
174 const r = await sub.migrateInline()
175 t.is(r.converted, 1, 'converted')
176 t.is(r.fromInline, 1, 'and reports that the content came from the inline copy, not a file')
177 t.alike(await sub.get(POST + '/orphan'), POST_REC, 'the record resolves afterwards')
178})