OverviewFilesCommitsIssuesDocsReleasesPeers

atproto-store commits / dc2e6d7

0.0.19: migrateInline materializes ALL files before publishing ANY pointer
The order is a safety property, not tidiness. It previously interleaved per record — put the file, set the pointer, next record — which leaves the set only partly protected while it runs. Why that matters: a legacy DID being flipped to storeBackend:'overlay' reads as an EMPTY record set (_resolve has no $oid and no file to fall back to), and sign()'s anti-wipe guard aborts only when record FILES exist. With no files it does not fire, so the first sign publishes a delete-every-record commit. That is e2jev's exact state today (hyper-social#16 P1). Materializing the whole set FIRST arms that guard before a single autobase entry is touched: if the publish pass never lands — crash, lost writability, autobase error — the files are on disk and sign() refuses to wipe instead of wiping. Pass 1 is purely local (content-addressed files, nothing published, no writability needed, nothing to roll back); pass 2 is the only part that appends. Returns `materialized` alongside the existing counters so an operator can see the two halves separately — "files written" and "pointers published" are now distinct facts rather than one number. Behaviour is otherwise unchanged: still idempotent (a converted entry has $oid and is skipped), still refuses any key whose oid would change, still never resurrects a tombstone. Proven in hyper-proto-peer/test/inline-migration-boot.test.mjs, including the ordering property itself — with the publish half forced to throw, all three records still materialize to disk and the guard's precondition holds. atproto-store 64/64, 187/187. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dc2e6d735087eece3a50907a039e2378cdbc5c8b
by plankatron <fleeky@gmail.com> · 2026-07-28T19:00:08.000Z

Changed files (2) +21 -4

lib/pointer-substrate.js +20 -3
……@@ -143,9 +143,23 @@ export class PointerSubstrate {
143143 // Nothing is deleted (statebase.set appends; the old value stays in oplog history) and files are
144144 // never modified beyond a byte-identical rewrite. WRITABLE-ONLY: the caller gates on base.writable.
145145 // Returns { converted, alreadyPointer, tombstones, fromInline, mismatched, failed, total }.
146 // TWO PASSES, and the order is a SAFETY property, not tidiness.
147 //
148 // PASS 1 MATERIALIZES every inline record to a file. PASS 2 publishes the pointers. Interleaving them
149 // (put/set per record, as this did originally) leaves the set only partly protected while it runs.
150 //
151 // Why that matters: a legacy repo being flipped to this substrate reads as an EMPTY record set (_resolve
152 // has no $oid and no file to fall back to), and sign()'s anti-wipe guard aborts only when record FILES
153 // exist. With no files it does not fire, and the first sign publishes a delete-every-record commit.
154 // Materializing the whole set FIRST arms that guard before a single autobase entry is touched — so if
155 // pass 2 never runs (crash, lost writability, autobase error), the repo is protected rather than wiped.
156 //
157 // Pass 1 is also purely local: content-addressed files, nothing published, no writability needed, and
158 // nothing to roll back. Pass 2 is the only part that appends.
146159 async migrateInline () {
147160 const raw = await this.statebase.list('')
148161 let converted = 0; let alreadyPointer = 0; let tombstones = 0; let fromInline = 0; let mismatched = 0; let failed = 0; let total = 0
162 const staged = []
149163 for (const key of Object.keys(raw)) {
150164 const s = recSplit(key)
151165 if (!s) continue // reserved/control key — never part of the record pointer set
……@@ -162,11 +176,14 @@ export class PointerSubstrate {
162176 const { oid, size } = await this.store.put(s.collection, s.rkey, record)
163177 if (!oid) { failed++; continue } // no overlay → no oid to publish; leave the entry alone
164178 if (before && oid !== before.oid) { mismatched++; continue } // would change published content — refuse
165 await this.statebase.set(key, { $oid: oid, size })
166 converted++
179 staged.push({ key, oid, size })
167180 } catch { failed++ }
168181 }
169 return { converted, alreadyPointer, tombstones, fromInline, mismatched, failed, total }
182 const materialized = staged.length
183 for (const { key, oid, size } of staged) {
184 try { await this.statebase.set(key, { $oid: oid, size }); converted++ } catch { failed++ }
185 }
186 return { converted, materialized, alreadyPointer, tombstones, fromInline, mismatched, failed, total }
170187 }
171188
172189 async backfill () {
package.json +1 -1
……@@ -1,6 +1,6 @@
11{
22 "name": "atproto-store",
3 "version": "0.0.18",
3 "version": "0.0.19",
44 "description": "Holepunch-native atproto repo store — records in a classic Hyperbee (collection/rkey -> DAG-CBOR). The P2P home; the atproto-repo module is the bridge serializer over it.",
55 "type": "module",
66 "main": "lib/repo-store.js",