OverviewFilesCommitsIssuesDocsReleasesPeers

atproto-bridge commits / 8c3a22e

0.9.37: seq-guard — notice when a relay has parked us, and lift past it
THE OUTAGE. Our emitted seq is a Hypercore length, so a storage rebuild restarts it near zero. atproto relays keep a strictly-forward per-host cursor and never roll back: once our seq sits below their cursor they treat us as a future-cursor and park the host. Measured 2026-07-31 — our firehose held 79 frames, bsky's relay sat at cursor 9834 with status "offline", and federation had been dead ~20 hours. Every local surface reported health the entire time; the only symptom was that the outside world stopped seeing us, which nothing local can observe. THE FIX ALREADY EXISTED AND WAS NEVER CALLED. FleetRelay.setSeqBase() raises the emitted floor monotonically and persists it to the durable `relay-seq-base` file kept beside the front-store. control.js exposes `set-seq-base`. fleet-relay.js documents this precise failure verbatim. What was missing was the part that NOTICES: recovery needed a human to know to read getHostStatus and hand-call a UDS RPC, so it stayed parked until someone went looking. A mechanism nothing invokes is not a mechanism. seq-guard adds the trigger. Once at boot and every 15 minutes it reads the relay's own view of us and lifts the floor past its cursor when the arithmetic says the relay can never reach us, then requests a crawl. decideLift is pure, so the policy is tested without a network: it keys on `emitted <= cursor`, NOT on status === 'offline'. A relay can report active while holding a cursor we cannot reach — the arithmetic is the condition, the status is a symptom. Equality still lifts, because the relay asks for seq > cursor. Read-only and silent on a healthy pass — no outward call is made when there is nothing wrong. Never throws: an unreachable relay is reported, not escalated into taking the membrane down. OUTWARD-FACING so it inherits the reactor's opt-in gate, and PDS_CRAWL_RELAY is now set in pgh.defaults.env — without that it would ship inert, which is how requestCrawl has sat unused. pds/test/seq-guard.test.js: 8 tests / 18 asserts, including the real outage numbers (79 vs 9834 → lift to 9835). Full pds suite 61/61. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
8c3a22e53a4ade9a5bd2a6928362ae2ccc5d642c
by plankatron <fleeky@gmail.com> · 2026-07-31T19:43:09.000Z

Changed files (4) +218 -3

package.json +4 -3
……@@ -1,7 +1,7 @@
11{
22 "name": "atproto-bridge",
3 "version": "0.9.36",
4 "description": "Stateless keyless atproto MEMBRANE — fronts self-custody hyper-proto-peers to bluesky over HTTP/WS, holds no keys and never signs. Bare-runnable. Deploy root / flat closure over the identity+repo+store+pds packages.",
3 "version": "0.9.37",
4 "description": "Stateless keyless atproto MEMBRANE \u2014 fronts self-custody hyper-proto-peers to bluesky over HTTP/WS, holds no keys and never signs. Bare-runnable. Deploy root / flat closure over the identity+repo+store+pds packages.",
55 "type": "module",
66 "main": "index.js",
77 "bin": {
……@@ -90,7 +90,8 @@
9090 "PDS_PORT": "3399",
9191 "PDS_STORAGE": "${HOME}/.config/atproto-bridge/data",
9292 "PDS_SOCK": "${HOME}/.config/atproto-bridge/svc.sock",
93 "FAVELA_WEB_UPSTREAM": "http://10.22.0.1:4041"
93 "FAVELA_WEB_UPSTREAM": "http://10.22.0.1:4041",
94 "PDS_CRAWL_RELAY": "https://bsky.network"
9495 }
9596 },
9697 "readiness": {
pds/bin/pds-serve.js +32 -0
……@@ -107,6 +107,38 @@ for (const did of cfg.frontedDids) {
107107 .catch((e) => console.error(JSON.stringify({ msg: 'front failed (will retry on demand)', did, error: e.message })))
108108}
109109
110// 3b. Seq guard — the half of the seq-floor mechanism that was missing: something that NOTICES.
111//
112// FleetRelay.setSeqBase + the durable `relay-seq-base` file (above) have always been able to lift our
113// emitted seq past a relay's stored cursor. Nothing ever called them. Recovery required a human to know
114// to read com.atproto.sync.getHostStatus and hand-call the `set-seq-base` UDS RPC — so on 2026-07-31 the
115// bridge sat parked at cursor 9834 against our 79 frames for ~20 hours, with every local surface healthy.
116// A mechanism nothing invokes is not a mechanism; this is the trigger.
117//
118// Runs once at boot and every 15 minutes. Read-only unless the arithmetic says the relay can never reach
119// us. OUTWARD-FACING, so it inherits the same opt-in gate as the reactor's requestCrawl below.
120if (cfg.crawlRelay && cfg.crawlHostname) {
121 const { runSeqGuard } = await import('../lib/seq-guard.js')
122 const crawl = async () => {
123 const res = await fetch(`${cfg.crawlRelay.replace(/\/$/, '')}/xrpc/com.atproto.sync.requestCrawl`, {
124 method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ hostname: cfg.crawlHostname })
125 })
126 if (!res.ok) throw new Error(`requestCrawl ${res.status}`)
127 }
128 const pass = () => runSeqGuard({
129 relay: peers.relay,
130 relayBase: cfg.crawlRelay,
131 hostname: cfg.crawlHostname,
132 requestCrawl: crawl,
133 log: (o) => console.log(JSON.stringify(o))
134 }).catch((e) => console.error(JSON.stringify({ msg: 'seq-guard pass failed', error: e && e.message })))
135 pass() // boot pass, backgrounded — must never delay listen or the readiness probe
136 const seqGuardTimer = setInterval(pass, 15 * 60 * 1000)
137 if (seqGuardTimer.unref) seqGuardTimer.unref()
138} else {
139 console.log(JSON.stringify({ msg: 'seq-guard OFF (no crawlRelay/crawlHostname) — a relay that parks us will stay parked' }))
140}
141
110142// 4. Feed-registry reactor (community-ux.md §111) — OFF unless cfg.feedRegistries is set. Watch each keyless
111143// FeedRegistry (the durable P2P allow-set) and, per feed entry, front its home DID + requestCrawl once, with
112144// no per-community config and no restart. Gated + guarded: favela-community is dynamically imported only when
pds/lib/seq-guard.js +97 -0
……@@ -0,0 +1,97 @@
1// seq-guard — detect that a downstream relay has parked us, and lift our emitted-seq floor past it.
2//
3// THE FAILURE. Our emitted `seq` is a Hypercore length (FleetSequencer/FirehoseSequencer): the block
4// index IS the seq. A storage rebuild resets that core to 0, so our emitted sequence restarts near zero.
5// atproto relays (indigo) keep a STRICTLY FORWARD per-host cursor and never roll it back: once our seq
6// is below their stored cursor they treat us as a future-cursor, park the host, and never read again.
7//
8// Measured 2026-07-31 on pds.7rnx.net: our firehose held 79 frames while bsky's relay sat at cursor 9834
9// with status "offline". Federation had been dead for ~20 hours. NOTHING on our side reported a problem —
10// the membrane was healthy, the peers were healthy, the repo signed correctly. The only symptom was that
11// the outside world stopped seeing us, which no local surface can observe.
12//
13// THE MECHANISM TO FIX IT ALREADY EXISTED and was never invoked: FleetRelay.setSeqBase(n) raises the
14// emitted floor monotonically and persists it (pds-serve's `relay-seq-base` file, kept BESIDE the
15// front-store so it survives a corestore reset). control.js exposes it as `set-seq-base`. What was
16// missing is the part that NOTICES — recovery required a human to know to read getHostStatus and hand-call
17// a UDS RPC. So it stayed parked until someone went looking.
18//
19// This closes that: read the relay's own view of us, and if it can never advance, lift the floor above its
20// cursor and ask it to crawl. Read-only on every healthy pass.
21//
22// ⚠ OUTWARD-FACING and therefore OPT-IN. It talks to bsky's relay, so it stays inert unless crawlRelay +
23// hostname are configured — the same gating the feed-registry reactor uses for requestCrawl.
24
25// Pure decision, so the policy is testable without a network or a relay.
26//
27// emitted — our current emitted seq (relay.length = seqBase + internal)
28// cursor — the relay's stored cursor for our host (getHostStatus.seq)
29//
30// Lift when `emitted <= cursor`: the relay will only ever ask for frames after `cursor`, and we cannot
31// produce a seq that high, so it is wedged by arithmetic regardless of what we sign. Lifting to
32// `cursor + margin` makes our next emitted frame land strictly above its cursor.
33//
34// Deliberately NOT keyed on status === 'offline'. A relay can report active while still holding a cursor
35// we can never reach (it reconnects, asks for >cursor, gets nothing, idles). The arithmetic is the real
36// condition; the status is a symptom of it.
37export function decideLift ({ emitted, cursor, margin = 1 } = {}) {
38 const e = Number(emitted)
39 const c = Number(cursor)
40 if (!Number.isFinite(e) || !Number.isFinite(c)) return { lift: false, reason: 'unknown-seq' }
41 if (c <= 0) return { lift: false, reason: 'relay-has-no-cursor' }
42 if (e > c) return { lift: false, reason: 'ahead-of-cursor' }
43 return { lift: true, to: c + Math.max(1, Number(margin) || 1), reason: 'behind-relay-cursor', emitted: e, cursor: c }
44}
45
46export function hostStatusUrl (relayBase, hostname) {
47 return `${String(relayBase).replace(/\/$/, '')}/xrpc/com.atproto.sync.getHostStatus?hostname=${encodeURIComponent(hostname)}`
48}
49
50// One pass. Returns a result object describing what it saw and did — never throws: a guard that can take
51// the membrane down is worse than the parked state it repairs.
52export async function runSeqGuard ({
53 relay,
54 relayBase,
55 hostname,
56 fetchImpl = null,
57 margin = 1,
58 requestCrawl = null,
59 log = null
60} = {}) {
61 const emit = (o) => { if (log) { try { log(o) } catch {} } return o }
62 if (!relay || !relayBase || !hostname) return { ok: false, skipped: 'not-configured' }
63 const doFetch = fetchImpl || (typeof globalThis.fetch === 'function' ? globalThis.fetch.bind(globalThis) : null)
64 if (!doFetch) return { ok: false, skipped: 'no-fetch' }
65
66 let status
67 try {
68 const res = await doFetch(hostStatusUrl(relayBase, hostname))
69 if (!res || !res.ok) return { ok: false, error: 'getHostStatus ' + (res && res.status) }
70 status = await res.json()
71 } catch (e) {
72 // The relay being unreachable is not our emergency — try again next pass.
73 return { ok: false, error: (e && e.message) || 'fetch failed' }
74 }
75
76 const emitted = relay.length
77 const d = decideLift({ emitted, cursor: status && status.seq, margin })
78 if (!d.lift) return { ok: true, lifted: false, emitted, cursor: status && status.seq, relayStatus: status && status.status, reason: d.reason }
79
80 // LOUD: this is a real repair of a state nothing else reports. The 20-hour outage was invisible
81 // precisely because no line like this existed.
82 emit({
83 msg: 'seq-guard: relay cursor is AHEAD of our emitted seq — we can never satisfy it; lifting the floor',
84 hostname,
85 relayStatus: status.status,
86 relayCursor: status.seq,
87 ourEmitted: emitted,
88 liftingTo: d.to
89 })
90 const seqBase = await relay.setSeqBase(d.to)
91 let crawled = null
92 if (requestCrawl) {
93 try { await requestCrawl(); crawled = true } catch (e) { crawled = false; emit({ msg: 'seq-guard: requestCrawl failed', error: e && e.message }) }
94 }
95 emit({ msg: 'seq-guard: floor lifted', seqBase, emitted: relay.length, crawled })
96 return { ok: true, lifted: true, seqBase, emitted: relay.length, cursor: status.seq, crawled }
97}
pds/test/seq-guard.test.js +85 -0
……@@ -0,0 +1,85 @@
1// seq-guard — the policy that stops a relay parking us forever. Run: brittle-node pds/test/seq-guard.test.js
2//
3// Pins the arithmetic that caused a 20-hour silent federation outage on 2026-07-31: our firehose held 79
4// frames while bsky's relay sat at cursor 9834, so the relay could only ever ask for frames we cannot
5// produce. Everything local reported health.
6import test from 'brittle'
7import { decideLift, hostStatusUrl, runSeqGuard } from '../lib/seq-guard.js'
8
9test('decideLift: the real outage numbers', (t) => {
10 const d = decideLift({ emitted: 79, cursor: 9834 })
11 t.ok(d.lift, 'emitted 79 vs cursor 9834 → must lift')
12 t.is(d.to, 9835, 'lifts to cursor + 1 so our next frame lands strictly above it')
13 t.is(d.reason, 'behind-relay-cursor')
14})
15
16test('decideLift: no lift when we are ahead — the healthy steady state', (t) => {
17 t.absent(decideLift({ emitted: 9900, cursor: 9834 }).lift, 'ahead of the cursor → nothing to do')
18 // Equality is NOT safe: the relay asks for seq > cursor, so emitted === cursor can never advance it.
19 t.ok(decideLift({ emitted: 9834, cursor: 9834 }).lift, 'equal is still wedged — the relay asks for > cursor')
20})
21
22test('decideLift: refuses to act on nonsense rather than guessing', (t) => {
23 t.absent(decideLift({ emitted: 10, cursor: 0 }).lift, 'a relay with no cursor for us is not a parked relay')
24 t.absent(decideLift({ emitted: undefined, cursor: 5 }).lift, 'unknown emitted → no action')
25 t.absent(decideLift({ emitted: 5, cursor: null }).lift, 'unknown cursor → no action')
26})
27
28test('hostStatusUrl encodes the hostname and tolerates a trailing slash', (t) => {
29 t.is(hostStatusUrl('https://bsky.network/', 'pds.7rnx.net'),
30 'https://bsky.network/xrpc/com.atproto.sync.getHostStatus?hostname=pds.7rnx.net')
31})
32
33// A fake relay with just the surface the guard touches.
34const fakeRelay = (length) => ({
35 length,
36 _base: 0,
37 async setSeqBase (n) { if (n > this._base) { this._base = n; this.length = n } ; return this._base }
38})
39const fakeFetch = (body, ok = true) => async () => ({ ok, status: ok ? 200 : 500, json: async () => body })
40
41test('runSeqGuard lifts and crawls when parked', async (t) => {
42 const relay = fakeRelay(79)
43 let crawled = false
44 const r = await runSeqGuard({
45 relay,
46 relayBase: 'https://bsky.network',
47 hostname: 'pds.7rnx.net',
48 fetchImpl: fakeFetch({ hostname: 'pds.7rnx.net', seq: 9834, status: 'offline' }),
49 requestCrawl: async () => { crawled = true }
50 })
51 t.ok(r.lifted, 'lifted')
52 t.is(r.seqBase, 9835, 'floor set above the relay cursor')
53 t.ok(crawled, 'and asked the relay to crawl — lifting alone leaves it parked')
54})
55
56test('runSeqGuard is read-only on a healthy pass', async (t) => {
57 const relay = fakeRelay(9900)
58 let crawled = false
59 const r = await runSeqGuard({
60 relay,
61 relayBase: 'https://bsky.network',
62 hostname: 'pds.7rnx.net',
63 fetchImpl: fakeFetch({ seq: 9834, status: 'active' }),
64 requestCrawl: async () => { crawled = true }
65 })
66 t.absent(r.lifted, 'nothing lifted')
67 t.absent(crawled, 'and NO outward call made — a guard must be silent when there is nothing wrong')
68 t.is(relay._base, 0, 'the floor is untouched')
69})
70
71test('runSeqGuard never throws — an unreachable relay is not our emergency', async (t) => {
72 const r = await runSeqGuard({
73 relay: fakeRelay(79),
74 relayBase: 'https://bsky.network',
75 hostname: 'pds.7rnx.net',
76 fetchImpl: async () => { throw new Error('ECONNREFUSED') }
77 })
78 t.absent(r.ok, 'reports failure')
79 t.ok(r.error, 'with the reason, rather than taking the membrane down')
80})
81
82test('runSeqGuard stays inert when unconfigured — outward-facing means opt-in', async (t) => {
83 const r = await runSeqGuard({ relay: fakeRelay(79) })
84 t.is(r.skipped, 'not-configured')
85})