OverviewFilesCommitsIssuesDocsReleasesPeers

pear-git-v3 commits / 9a01e78

Release the drive before republishing it, and report the split
Two defects found by testing the second release rather than the first. The seeder holds the drive's store with an exclusive lock for as long as it is announced, and a second opener — even in-process — gets "File descriptor could not be locked". So the SECOND publish for a vdrive repo could never write its drive. The release itself would still succeed, because vdrive failures are caught, making the symptom a drive that silently stops updating after its first release. The orchestrator now drops the drive, publishes, and re-announces; the re-announce is unconditional so a failed publish puts back what the drop took away. The test proves the drop is necessary, not merely sufficient: publishing under an active seeder fails on the lock, and succeeds after the drop. And the hook's log line now carries the pointer/literal split, the locations count and the store size. The old line said only "6 files, 1835 blocks", which left a 76MB drive ambiguous between "the redirect failed and everything is literal" and "it just needs compacting" — opposite problems, and telling them apart meant reproducing the publish. 76/76 under bare across vdrive, gip and release suites. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
9a01e7805fdb25b57cc7880da01f22b899b830ff
by plankatron <fleeky@gmail.com> · 2026-08-25T00:07:27.000Z

Changed files (7) +371 -13

lib/ci-orchestrator.js +20 -11
……@@ -128,6 +128,13 @@ export async function buildAndPublish ({
128128 let vdrive = null
129129 if (config.vdrive && config.vdrive.enabled && releaseManager.releaseDir) {
130130 try {
131 // The seeder holds the drive's store with an exclusive lock for as long
132 // as it is announced, and a second opener gets "File descriptor could
133 // not be locked". So the SECOND publish for a repo cannot write its
134 // drive unless the seeder lets go first. Drop, publish, re-announce.
135 // The drive is unserved for that window, which is the cost of writing it.
136 if (vdriveSeeder) await vdriveSeeder.dropRepo(repo)
137
131138 vdrive = await publishVersionToDrive({
132139 releaseDir: releaseManager.releaseDir,
133140 repo,
……@@ -137,21 +144,23 @@ export async function buildAndPublish ({
137144 onLog
138145 })
139146 if (vdrive.skipped) log(onLog, `[ci] vdrive skipped: ${vdrive.reason}`)
140 // Announce it NOW. The seeder's boot scan cannot have seen a drive that
141 // did not exist yet, so without this a freshly published drive stays
142 // unreachable until the next daemon restart.
143 if (vdrive.ok && vdriveSeeder) {
144 try {
145 const seeded = await vdriveSeeder.announceRepo(repo)
146 if (seeded) log(onLog, `[ci] vdrive seeding ${repo} -> ${seeded.link}`)
147 } catch (err) {
148 log(onLog, `[ci] vdrive announce failed (release unaffected): ${err.message}`)
149 }
150 }
151147 } catch (err) {
152148 vdrive = { ok: false, error: err.message }
153149 log(onLog, `[ci] vdrive failed (release unaffected): ${err.message}`)
154150 }
151
152 // Re-announce unconditionally. On success this is what makes a freshly
153 // published drive reachable — the boot scan cannot have seen a drive that
154 // did not exist yet. On FAILURE it puts back what the drop above took
155 // away, so a bad publish does not also stop serving the previous release.
156 if (vdriveSeeder) {
157 try {
158 const seeded = await vdriveSeeder.announceRepo(repo)
159 if (seeded) log(onLog, `[ci] vdrive seeding ${repo} -> ${seeded.link}`)
160 } catch (err) {
161 log(onLog, `[ci] vdrive announce failed (release unaffected): ${err.message}`)
162 }
163 }
155164 }
156165
157166 return {
lib/services/ref-diff.js +67 -0
……@@ -0,0 +1,67 @@
1/**
2 * ref-diff — compare a repo's ref-VIEW (the replicated autobase source of
3 * truth) against its bare repo on DISK, and name the disagreements.
4 *
5 * Pure: no I/O, no network. The `repo-refs` handler supplies both maps.
6 *
7 * WHY THIS EXISTS. Until 2026-08-08 nothing in pear-git-v3 could show the view's
8 * refs at all — `pgh repo branches` goes to `syncServer.getBranches()`, which
9 * reads DISK. That invisibility produced a concrete failure: a `release` ref was
10 * deleted on the origin and believed to have propagated fleet-wide, when for 3 of
11 * 5 repos the deletion silently never entered the view (`gipPublishRef` returns
12 * without a word when the tracker is not writable, or when the view never carried
13 * the ref) — while `ref-update-observed` still answered `ok: true`.
14 *
15 * The four states are the whole point; each one means something operationally:
16 *
17 * in-sync view and disk agree. Nothing to do.
18 * divergent both have it, at different oids. The view wins: RefReconciler
19 * rolls disk BACK to the view's oid (that is how a locally-accepted
20 * but globally-non-FF push gets undone).
21 * viewOnly the view carries it, disk does not. RefReconciler will materialize
22 * it — so deleting it from disk alone does NOT stick.
23 * diskOnly disk carries it, the view does not. THE RESURRECTION HAZARD:
24 * ref-publish-reconciler reads this as "disk is ahead" and
25 * re-announces, which undoes a deletion made elsewhere. It is also
26 * what an out-of-band ref (never propagated) looks like, so the two
27 * cases are indistinguishable from this side alone — which is
28 * exactly the ambiguity that follows from deleting a key rather than
29 * tombstoning it.
30 */
31
32/** @typedef {{ ref: string, view: string|null, disk: string|null, state: string }} RefRow */
33
34export const IN_SYNC = 'in-sync'
35export const DIVERGENT = 'divergent'
36export const VIEW_ONLY = 'viewOnly'
37export const DISK_ONLY = 'diskOnly'
38
39/**
40 * @param {Map<string,string>|Iterable<[string,string]>} viewRefs ref -> oid
41 * @param {Map<string,string>|Iterable<[string,string]>} diskRefs ref -> oid
42 * @returns {RefRow[]} sorted by ref name
43 */
44export function classifyRefs (viewRefs, diskRefs) {
45 const view = viewRefs instanceof Map ? viewRefs : new Map(viewRefs || [])
46 const disk = diskRefs instanceof Map ? diskRefs : new Map(diskRefs || [])
47
48 const rows = []
49 for (const ref of new Set([...view.keys(), ...disk.keys()])) {
50 const v = view.get(ref) || null
51 const d = disk.get(ref) || null
52 let state
53 if (v && d) state = v === d ? IN_SYNC : DIVERGENT
54 else if (v) state = VIEW_ONLY
55 else state = DISK_ONLY
56 rows.push({ ref, view: v, disk: d, state })
57 }
58 return rows.sort((a, b) => (a.ref < b.ref ? -1 : a.ref > b.ref ? 1 : 0))
59}
60
61/**
62 * Rows worth an operator's attention — everything except in-sync.
63 * @param {RefRow[]} rows
64 */
65export function refDisagreements (rows) {
66 return rows.filter((r) => r.state !== IN_SYNC)
67}
lib/vdrive-publish.js +26 -1
……@@ -28,6 +28,16 @@ export function vdriveStoreDir (releaseDir, repo) {
2828 return path.join(releaseDir, repo, '.vdrive')
2929}
3030
31async function dirSize (dir) {
32 let total = 0
33 for (const e of await fsp.readdir(dir, { withFileTypes: true })) {
34 const full = path.join(dir, e.name)
35 if (e.isDirectory()) total += await dirSize(full)
36 else if (e.isFile()) total += (await fsp.stat(full)).size
37 }
38 return total
39}
40
3141async function walk (dir, base = dir, out = []) {
3242 for (const entry of await fsp.readdir(dir, { withFileTypes: true })) {
3343 const full = path.join(dir, entry.name)
……@@ -98,7 +108,22 @@ export async function publishVersionToDrive ({
98108 })
99109
100110 if (onLog) {
101 onLog(`[ci] vdrive ${repo} ${version}: ${result.files} files, ${result.blocks} blocks -> ${result.link}`)
111 // Report the pointer/literal split and the store size. Without them a large
112 // .vdrive is ambiguous — it reads the same whether the redirect failed (all
113 // literal) or the store merely needs compacting, and those want opposite
114 // responses. Once the seeder holds the lock you cannot go back and measure.
115 let storeBytes = null
116 try {
117 storeBytes = await dirSize(vdriveStoreDir(releaseDir, repo))
118 } catch {}
119 onLog(
120 `[ci] vdrive ${repo} ${version}: ${result.files} files, ${result.blocks} blocks ` +
121 `(${result.pointerBlocks} pointer, ${result.literalBlocks} literal), ` +
122 `${result.locations} locations` +
123 (storeBytes === null ? '' : `, store ${(storeBytes / 1048576).toFixed(1)}MB`) +
124 (result.compacted === false ? ' [compaction failed]' : '') +
125 ` -> ${result.link}`
126 )
102127 }
103128
104129 return { ok: true, ...result }
package.json +1 -1
……@@ -1,6 +1,6 @@
11{
22 "name": "pear-git-v3",
3 "version": "0.1.229",
3 "version": "0.1.230",
44 "description": "P2P git forge \u2014 multi-writer issue/org/release/peer-mirror daemon. Built bottom-up after the failed v2.",
55 "type": "module",
66 "main": "index.js",
test/identity-apply-determinism.test.js +114 -0
……@@ -0,0 +1,114 @@
1// Determinism test for the IdentityRegistry apply function.
2//
3// Autobase requires apply to be DETERMINISTIC — it is replayed on every boot
4// and every reorg, and each peer runs it independently to build its own view.
5// If apply writes non-deterministic values, two peers replaying the SAME ops
6// produce DIFFERENT view bytes, so their view cores and checkpoints can never
7// match.
8//
9// identity-apply.js writes Date.now() into the view in three places:
10// :437 ADD_WRITER -> { addedAt: Date.now() }
11// :474 REGISTER_MIRROR_WRITERS -> { recordedAt: Date.now() }
12// :428 SET_USERNAME -> user.updatedAt = Date.now()
13//
14// Claim under test: replaying an identical op batch twice yields different
15// stored bytes. The CONTROL uses REGISTER_IDENTITY, which stores only
16// op-provided fields — it must come out byte-identical, proving any
17// difference in the other cases is attributable to apply itself and not to
18// the harness.
19
20import test from 'brittle'
21import Corestore from 'corestore'
22import Hyperbee from 'hyperbee'
23import b4a from 'b4a'
24import os from '#node:os'
25import path from '#node:path'
26import fs from '#node:fs'
27import { createApplyFunction } from '../node_modules/hyper-identity/lib/auth/identity-apply.js'
28import { encodeOp, OpTypes } from '../node_modules/hyper-identity/lib/auth/identity-operations.js'
29
30const TMP = fs.mkdtempSync(path.join(os.tmpdir(), 'apply-determinism-'))
31let n = 0
32
33async function freshView () {
34 const store = new Corestore(path.join(TMP, 'store-' + (n++)))
35 await store.ready()
36 // Raw Corestore takes { name }; the string form in identity-registry.js:77
37 // is autobase's view-store API, not this one.
38 const bee = new Hyperbee(store.get({ name: 'identity-view' }), {
39 extension: false, keyEncoding: 'utf-8', valueEncoding: 'json'
40 })
41 await bee.ready()
42 return { store, bee }
43}
44
45// A writer whose addWriter never fails, so the ADD_WRITER path completes.
46const stubHost = { addWriter: async () => {}, removeWriter: async () => {} }
47
48const node = (op) => ({ value: encodeOp(op), from: { key: b4a.alloc(32) } })
49
50// Run the REAL apply over a batch, return the stored value at `key`.
51async function applyAndRead (batch, key) {
52 const { store, bee } = await freshView()
53 const apply = createApplyFunction({})
54 await apply(batch, bee, stubHost)
55 const entry = await bee.get(key)
56 const value = entry ? entry.value : null
57 await bee.close(); await store.close()
58 return value
59}
60
61const sleep = (ms) => new Promise(r => setTimeout(r, ms))
62
63test('CONTROL: an op storing only op-provided fields replays byte-identically', async (t) => {
64 const identity = {
65 identityId: 'ident-control',
66 method: 'web-identity',
67 publicKey: 'ff'.repeat(32),
68 userId: 'user-control',
69 createdAt: 1700000000000 // fixed by the op, not by apply
70 }
71 const batch = [node({ type: OpTypes.REGISTER_IDENTITY, identity })]
72
73 const a = await applyAndRead(batch, 'identity:ident-control')
74 await sleep(25)
75 const b = await applyAndRead(batch, 'identity:ident-control')
76
77 t.ok(a && b, 'both replays stored the record')
78 t.alike(a, b, 'CONTROL replays byte-identical — the harness itself is deterministic')
79})
80
81test('ADD_WRITER: apply stamps Date.now() -> replays DIVERGE', async (t) => {
82 const writerKey = 'ab'.repeat(32)
83 const batch = [node({
84 type: OpTypes.ADD_WRITER, writerKey, role: 'writer', addedBy: 'test'
85 })]
86
87 const a = await applyAndRead(batch, 'writer:' + writerKey)
88 await sleep(25)
89 const b = await applyAndRead(batch, 'writer:' + writerKey)
90
91 t.ok(a && b, 'both replays stored the writer record')
92 t.comment('replay A addedAt=' + (a && a.addedAt) + ' replay B addedAt=' + (b && b.addedAt))
93 t.unlike(a, b,
94 'identical ADD_WRITER op produces DIFFERENT view bytes on replay — ' +
95 'apply is NON-DETERMINISTIC, so peers cannot converge on a shared view')
96})
97
98test('REGISTER_MIRROR_WRITERS: apply stamps Date.now() -> replays DIVERGE', async (t) => {
99 const devicePubkey = 'cd'.repeat(32)
100 const batch = [node({
101 type: OpTypes.REGISTER_MIRROR_WRITERS,
102 devicePubkey,
103 keys: { orgs: 'ef'.repeat(32) }
104 })]
105
106 const a = await applyAndRead(batch, 'mirror-writers:' + devicePubkey)
107 await sleep(25)
108 const b = await applyAndRead(batch, 'mirror-writers:' + devicePubkey)
109
110 t.ok(a && b, 'both replays stored the announcement')
111 t.comment('replay A recordedAt=' + (a && a.recordedAt) + ' replay B recordedAt=' + (b && b.recordedAt))
112 t.unlike(a, b,
113 'identical REGISTER_MIRROR_WRITERS op produces DIFFERENT view bytes on replay')
114})
test/ref-diff.test.js +82 -0
……@@ -0,0 +1,82 @@
1// ref-diff — the view-vs-disk comparison that makes ref replication observable.
2//
3// Grounded in a real 2026-08-08 failure: `refs/heads/release` was deleted on the
4// origin for 5 repos and believed to have propagated fleet-wide. It propagated
5// for 2. For the other 3 the deletion never entered the view (gipPublishRef
6// returns silently when the tracker is not writable, or when the view never had
7// the ref) while `ref-update-observed` still answered ok:true. Nothing could show
8// the view's refs, so the no-op was indistinguishable from success.
9
10import test from 'brittle'
11
12import {
13 classifyRefs, refDisagreements, IN_SYNC, DIVERGENT, VIEW_ONLY, DISK_ONLY
14} from '../lib/services/ref-diff.js'
15
16const A = 'a'.repeat(40)
17const B = 'b'.repeat(40)
18
19test('agreeing refs are in-sync', (t) => {
20 const rows = classifyRefs(new Map([['refs/heads/main', A]]), new Map([['refs/heads/main', A]]))
21 t.is(rows.length, 1)
22 t.is(rows[0].state, IN_SYNC)
23 t.alike(refDisagreements(rows), [], 'nothing for an operator to look at')
24})
25
26test('same ref at different oids is divergent', (t) => {
27 // The view wins: RefReconciler rolls disk BACK to the view's oid. That is how a
28 // push accepted locally but non-FF in the linearized order gets undone.
29 const rows = classifyRefs(new Map([['refs/heads/main', A]]), new Map([['refs/heads/main', B]]))
30 t.is(rows[0].state, DIVERGENT)
31 t.is(rows[0].view, A)
32 t.is(rows[0].disk, B)
33})
34
35test('viewOnly means deleting from disk alone will NOT stick', (t) => {
36 // RefReconciler materializes view refs onto disk, so a local `update-ref -d`
37 // gets undone on the next reconcile.
38 const rows = classifyRefs(new Map([['refs/heads/release', A]]), new Map())
39 t.is(rows[0].state, VIEW_ONLY)
40 t.is(rows[0].disk, null)
41 t.is(refDisagreements(rows).length, 1)
42})
43
44test('diskOnly is the resurrection hazard — the exact 2026-08-08 state', (t) => {
45 // stortron after the origin-side delete silently failed to publish: disk still
46 // has the ref, the view does not. ref-publish-reconciler reads that as "disk
47 // ahead of view" and re-announces, undoing the deletion fleet-wide.
48 const rows = classifyRefs(new Map(), new Map([['refs/heads/release', A]]))
49 t.is(rows[0].state, DISK_ONLY)
50 t.is(rows[0].view, null)
51 t.is(rows[0].disk, A)
52})
53
54test('diskOnly cannot distinguish "never propagated" from "deleted elsewhere"', (t) => {
55 // Both an out-of-band ref and a ref deleted on another node look identical from
56 // here, because deletion removes the key rather than leaving a tombstone. This
57 // pins the ambiguity as a known property, so a future tombstone can be judged
58 // against it rather than added speculatively.
59 const outOfBand = classifyRefs(new Map(), new Map([['refs/heads/internal', A]]))
60 const deletedElsewhere = classifyRefs(new Map(), new Map([['refs/heads/release', A]]))
61 t.is(outOfBand[0].state, deletedElsewhere[0].state,
62 'indistinguishable by state alone — only intent separates them')
63})
64
65test('a full repo picture sorts and covers every combination', (t) => {
66 const rows = classifyRefs(
67 new Map([['refs/heads/main', A], ['refs/heads/dev', A], ['refs/heads/gone-locally', A]]),
68 new Map([['refs/heads/main', A], ['refs/heads/dev', B], ['refs/heads/release', A]])
69 )
70 t.alike(rows.map(r => r.ref), [
71 'refs/heads/dev', 'refs/heads/gone-locally', 'refs/heads/main', 'refs/heads/release'
72 ], 'sorted by ref name')
73 t.alike(rows.map(r => r.state), [DIVERGENT, VIEW_ONLY, IN_SYNC, DISK_ONLY])
74 t.is(refDisagreements(rows).length, 3, 'only the in-sync one is filtered out')
75})
76
77test('empty and missing inputs are not errors', (t) => {
78 t.alike(classifyRefs(new Map(), new Map()), [])
79 t.alike(classifyRefs(null, null), [], 'null tolerated — a repo with no tracker yet')
80 t.alike(classifyRefs([['refs/heads/main', A]], [['refs/heads/main', A]])[0].state, IN_SYNC,
81 'accepts plain entry arrays as well as Maps')
82})
test/vdrive-seeder.test.js +61 -0
……@@ -158,3 +158,64 @@ test('CI announces a freshly published drive — boot scan cannot have seen it',
158158 await s.close()
159159 await fsp.rm(releaseDir, { recursive: true, force: true })
160160})
161
162test('a second publish succeeds while the drive is being seeded', async (t) => {
163 // The defect: the seeder holds the drive's store with an exclusive lock while
164 // announced, so a later publish cannot open it — "File descriptor could not be
165 // locked". The release still succeeds (vdrive failures are caught), so the
166 // symptom is a drive that silently stops updating after its FIRST release.
167 const releaseDir = await tmpdir('secondpublish')
168 const { chunk } = await import('hyper-overlay/chunker')
169
170 await seedOneRepo(releaseDir, 'demo', '1.0.0', Buffer.alloc(128 * 1024, 1))
171
172 const s = new VdriveSeeder({ releaseDir, wire: () => {} })
173 await s.start()
174 t.is(s.count, 1, 'seeding, so the store is locked')
175
176 // Prove the drop is NECESSARY, not just sufficient: publishing while the
177 // seeder still holds the store must fail on the lock.
178 const blockedDir = path.join(releaseDir, 'demo', 'versions', '1.5.0')
179 await fsp.mkdir(blockedDir, { recursive: true })
180 await fsp.writeFile(path.join(blockedDir, 'app.bin'), Buffer.alloc(64 * 1024, 9))
181 let blocked = null
182 try {
183 await publishVersionToDrive({
184 releaseDir,
185 repo: 'demo',
186 version: '1.5.0',
187 channel: 'production',
188 vdrive: { enabled: true, channels: null },
189 chunk
190 })
191 } catch (err) {
192 blocked = err.message
193 }
194 t.ok(blocked, 'publishing under an active seeder fails: ' + blocked)
195 t.ok(/lock/i.test(blocked || ''), 'and it fails on the store lock')
196
197 // The orchestrator therefore drops, publishes, re-announces.
198 await s.dropRepo('demo')
199
200 const versionDir = path.join(releaseDir, 'demo', 'versions', '2.0.0')
201 await fsp.mkdir(versionDir, { recursive: true })
202 await fsp.writeFile(path.join(versionDir, 'app.bin'), Buffer.alloc(128 * 1024, 2))
203 const second = await publishVersionToDrive({
204 releaseDir,
205 repo: 'demo',
206 version: '2.0.0',
207 channel: 'production',
208 vdrive: { enabled: true, channels: null },
209 chunk
210 })
211
212 t.is(second.ok, true, 'second publish wrote the drive')
213 t.ok(second.priorLocations > 0, 'and it resolved the first release via the index')
214
215 const again = await s.announceRepo('demo')
216 t.ok(again, 're-announced after the publish')
217 t.is(s.count, 1)
218
219 await s.close()
220 await fsp.rm(releaseDir, { recursive: true, force: true })
221})