OverviewFilesCommitsIssuesDocsReleasesPeers

P2P-Hiverelay commits / 5f9a5df

fix: pin drifted fleet nodes safely
5f9a5df0225626124ea93c69e15dfc8713a3c56a
by bigdestiny2 <33146784+bigdestiny2@users.noreply.github.com> · 2026-07-29T19:52:16.000Z

Changed files (10) +140 -29

CHANGELOG.md +3 -0
……@@ -52,6 +52,9 @@ deployment.**
5252- Fleet health and service probes support exact include/exclude scopes before
5353 SSH, report exact prerelease versions and updater/channel drift, inspect
5454 signed capability evidence, and avoid exposing private inventory/key paths.
55- Fleet updater config supports a validated per-host `PINNED_TAG`, allowing a
56 remigrated box to restore its timer without following an older shared channel
57 target. Signature verification now also precedes up-to-date and dry-run exits.
5558- StartOS and Umbrel metadata, release-evidence tests, and operator docs are
5659 synchronized to the current prerelease baseline and distinguish configured
5760 defaults from signed runtime proof.
docs/FLEET-PLAN-2026-07-28.md +1 -1
……@@ -121,7 +121,7 @@ exists, every enablement is a manual, unrepeatable, undocumented act.
121121| 0.1 | `kill 51423`; bind `127.0.0.1`, add a shared secret, replace `exec` with `execFile(argv)`, drop wildcard CORS | R-1 — remote root on 13 relays from any network the laptop joins |
122122| 0.2 | Recover **utah-0.5gb**: `npm rebuild` (or reinstall deps) and confirm the addon resolves; if it will not, revert it to `v0.24.3` | Box is down; it is a seed holder |
123123| 0.3 | Fix **C-1**: `config/default.js:205` `serviceDefaultPeerRole: 'authenticated-user'` → `'anonymous'` | Silently reverts commit `9125f3c`; opens `arbitration.submit` (uncapped Map) to any anonymous swarm peer on 10 boxes |
124| 0.4 | Pin **utah-8gb** to `hold` before its timer is ever armed | R-3 — it runs rc.4 on `channel: stable` (pinned 0.24.3); arming converges it *backwards*, logged as success |
124| 0.4 | Pin **utah-8gb** to exact signed tag `v0.25.0-rc.4` with `PINNED_TAG` before its timer is ever armed; retain `CHANNEL=stable` for its eventual destination | R-3 — it runs rc.4 while stable and the shared `hold` target are both 0.24.3; arming either unpinned channel converges it *backwards* across its Corestore-7 remigration, logged as success |
125125
126126**Gate out of Phase 0:** utah-0.5gb green, admin operator on loopback with auth, C-1 shipped.
127127
docs/LADDER-SHIP-MAP.md +3 -2
……@@ -440,8 +440,9 @@ You can call the **product ladder complete for honest marketing** when:
440440
4414411. **Name and ship the repair candidate:** finish release gates, cut a new signed
442442 post-RC7 prerelease, and deploy it to the permitted canary only. Correct
443 Utah-8GB's disabled stable updater separately; do not fold a downgrade into
444 a canary promotion.
443 Utah-8GB's disabled stable updater separately: install the repaired updater,
444 set `CHANNEL=stable` plus `PINNED_TAG=v0.25.0-rc.4`, prove a no-op dry run,
445 then arm its timer. Do not fold a downgrade into a canary promotion.
4454462. **Prove the claims on live canary:** configure operator-owned push egress if
446447 wake is a release claim; verify exact-lane wake, outboxlog persistence,
447448 restricted-Tor negative probe, and the 100 MB onion transfer. Observe for
fleet/README.md +20 -5
……@@ -102,9 +102,23 @@ allowlist. Invalid list syntax fails closed before any SSH connection. The
102102runtime service-catalogue probe accepts the same environment variables, or
103103equivalent `--include` / `--exclude` options.
104104
105To **hold** a box, point its channel at the version it already runs. To
106**roll the whole channel back**, set the tag back — boxes check out the
107older tag exactly like a forward update (health-gated the same way).
105To **hold one box** on the exact signed tag it already runs, set a per-host
106pin while preserving its intended channel:
107
108```bash
109sudo bash fleet/install-updater.sh stable v0.25.0-rc.4
110hiverelay-updater --dry-run
111```
112
113The installer writes `PINNED_TAG=` as data in
114`/etc/hiverelay-updater.conf`. A valid pin bypasses the moving channel target,
115but still goes through fetch, signed-tag verification, runtime preflight, and
116the health gate. Remove the `PINNED_TAG` line only when that box is ready to
117rejoin its channel. Do not use the shared `hold` channel to freeze a box that
118runs a different version: `hold` has one fleet-wide target and may serve other
119operator-owned lanes. To **roll a whole channel back**, set its shared tag back
120— boxes check out the older tag exactly like a forward update (health-gated the
121same way).
108122
109123## Safety properties
110124
……@@ -117,8 +131,9 @@ older tag exactly like a forward update (health-gated the same way).
117131 version. See "Signed releases" above and `docs/SUPPLY-CHAIN.md`.
118132- **Dirty-tree guard:** the agent refuses to act if the repo has
119133 uncommitted changes — it never clobbers a hand-edit.
120- **Config treated as data:** `/etc/hiverelay-updater.conf` is parsed for a
121 single validated `CHANNEL=` value; the updater does not source it as shell.
134- **Config treated as data:** `/etc/hiverelay-updater.conf` is parsed for
135 validated `CHANNEL=` and optional `PINNED_TAG=` values; the updater does not
136 source it as shell.
122137- **Single-flight:** `flock`; overlapping ticks can't collide.
123138- **Jittered:** boxes update on randomized offsets — no thundering herd
124139 on GitHub and no fleet-wide simultaneous restart.
fleet/fleet-status.sh +2 -0
……@@ -112,6 +112,8 @@ U_ACTIVE="$(systemctl is-active hiverelay-updater.timer 2>/dev/null || true)"
112112[ -n "$U_ACTIVE" ] || U_ACTIVE=inactive
113113C=$(awk -F= '/^[[:space:]]*CHANNEL[[:space:]]*=/ { sub(/^[^=]*=/,""); sub(/^[[:space:]]*/,""); sub(/[[:space:]]*$/,""); print; exit }' /etc/hiverelay-updater.conf 2>/dev/null || true)
114114[ -n "$C" ] || C=?
115P=$(awk -F= '/^[[:space:]]*PINNED_TAG[[:space:]]*=/ { sub(/^[^=]*=/,""); sub(/^[[:space:]]*/,""); sub(/[[:space:]]*$/,""); print; exit }' /etc/hiverelay-updater.conf 2>/dev/null || true)
116[ -z "$P" ] || C="$C@$P"
115117echo "$V|$R|$D|$U_ENABLED/$U_ACTIVE|$C"
116118REMOTE_EOF
117119
fleet/install-updater.sh +21 -3
……@@ -6,11 +6,14 @@
66# curl -fsSL .../fleet/install-updater.sh | bash -s -- <channel>
77# or, from a checked-out repo on the box:
88# sudo bash fleet/install-updater.sh stable
9# sudo bash fleet/install-updater.sh stable v0.25.0-rc.4 # exact signed-tag pin
910#
10# <channel> must match a key in fleet/channels.json (stable | canary).
11# <channel> must match a key in fleet/channels.json (stable | canary | hold).
12# [pinned-tag] overrides the moving channel target until an operator removes it.
1113set -euo pipefail
1214
1315CHANNEL="${1:-stable}"
16PINNED_TAG="${2:-}"
1417REPO_DIR="${HIVERELAY_REPO_DIR:-$HOME/hiverelay}"
1518SRC="$REPO_DIR/fleet"
1619
……@@ -18,16 +21,27 @@ if [[ ! "$CHANNEL" =~ ^[A-Za-z0-9._-]{1,32}$ ]]; then
1821 echo "Invalid channel '$CHANNEL' — use a key from fleet/channels.json (for example stable or canary)" >&2
1922 exit 1
2023fi
24if [ -n "$PINNED_TAG" ] && [[ ! "$PINNED_TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]]; then
25 echo "Invalid pinned tag '$PINNED_TAG' — use a release tag such as v0.25.0-rc.4" >&2
26 exit 1
27fi
2128
2229[ -d "$SRC" ] || { echo "fleet/ not found in $REPO_DIR — clone/pull the repo first"; exit 1; }
2330
24echo "Installing hiverelay-updater (channel=$CHANNEL)…"
31if [ -n "$PINNED_TAG" ]; then
32 echo "Installing hiverelay-updater (channel=$CHANNEL, pinned-tag=$PINNED_TAG)…"
33else
34 echo "Installing hiverelay-updater (channel=$CHANNEL)…"
35fi
2536
2637install -m 0755 "$SRC/updater.sh" /usr/local/bin/hiverelay-updater
2738install -m 0644 "$SRC/hiverelay-updater.service" /etc/systemd/system/hiverelay-updater.service
2839install -m 0644 "$SRC/hiverelay-updater.timer" /etc/systemd/system/hiverelay-updater.timer
2940
3041printf 'CHANNEL=%s\n' "$CHANNEL" > /etc/hiverelay-updater.conf
42if [ -n "$PINNED_TAG" ]; then
43 printf 'PINNED_TAG=%s\n' "$PINNED_TAG" >> /etc/hiverelay-updater.conf
44fi
3145chmod 0644 /etc/hiverelay-updater.conf
3246
3347systemctl daemon-reload
……@@ -50,7 +64,11 @@ if [ -f "$SRC/harden-box.sh" ]; then
5064 bash "$SRC/harden-box.sh" || echo "WARN harden-box.sh failed (non-fatal)"
5165fi
5266
53echo "Installed. Channel=$CHANNEL. Timer:"
67if [ -n "$PINNED_TAG" ]; then
68 echo "Installed. Channel=$CHANNEL pinned-tag=$PINNED_TAG. Timer:"
69else
70 echo "Installed. Channel=$CHANNEL. Timer:"
71fi
5472systemctl status hiverelay-updater.timer --no-pager | sed -n '1,4p' || true
5573echo
5674echo "Dry-run the agent now (no changes):"
fleet/updater.sh +37 -15
……@@ -5,8 +5,9 @@
55# Run by a systemd timer (every ~15 min). Each box owns its own lifecycle:
66# no inbound SSH, no orchestrator, works behind NAT. The flow:
77#
8# 1. Read this box's channel (/etc/hiverelay-updater.conf, default stable).
9# 2. Fetch fleet/channels.json from the repo and resolve the target tag.
8# 1. Read this box's channel and optional exact signed-tag pin
9# (/etc/hiverelay-updater.conf, default stable).
10# 2. Resolve the pin, or fetch fleet/channels.json and resolve the channel.
1011# 3. If already on the target -> no-op.
1112# 4. Otherwise: snapshot current SHA, fetch + checkout the target tag,
1213# reinstall deps only if package-lock changed, restart the relay.
……@@ -147,6 +148,7 @@ fi
147148
148149# ── channel ────────────────────────────────────────────────────────
149150CHANNEL="stable"
151PINNED_TAG=""
150152if [ -r "$CONF" ]; then
151153 # Treat config as data, not shell. The installer writes CHANNEL=<name>;
152154 # sourcing this file would make a writable config path code-executable.
……@@ -156,16 +158,29 @@ if [ -r "$CONF" ]; then
156158 CONF_CHANNEL="${CONF_CHANNEL%\'}"
157159 CONF_CHANNEL="${CONF_CHANNEL#\'}"
158160 [ -n "$CONF_CHANNEL" ] && CHANNEL="$CONF_CHANNEL"
161 CONF_PINNED_TAG="$(sed -n 's/^[[:space:]]*PINNED_TAG[[:space:]]*=[[:space:]]*//p' "$CONF" | head -n 1)"
162 CONF_PINNED_TAG="${CONF_PINNED_TAG%\"}"
163 CONF_PINNED_TAG="${CONF_PINNED_TAG#\"}"
164 CONF_PINNED_TAG="${CONF_PINNED_TAG%\'}"
165 CONF_PINNED_TAG="${CONF_PINNED_TAG#\'}"
166 [ -n "$CONF_PINNED_TAG" ] && PINNED_TAG="$CONF_PINNED_TAG"
159167fi
160168if [[ ! "$CHANNEL" =~ ^[A-Za-z0-9._-]{1,32}$ ]]; then
161169 die "invalid channel '$CHANNEL' in $CONF"
162170fi
171if [ -n "$PINNED_TAG" ] && [[ ! "$PINNED_TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]]; then
172 die "invalid pinned tag '$PINNED_TAG' in $CONF"
173fi
163174
164175# ── resolve target tag ─────────────────────────────────────────────
165JSON="$(curl -fsS --max-time 20 "$CHANNELS_URL")" || die "cannot fetch channels.json"
166TARGET="$(printf '%s' "$JSON" | CHANNEL="$CHANNEL" python3 -c 'import os,sys,json; print(json.load(sys.stdin).get(os.environ["CHANNEL"], ""))')" \
167 || die "cannot parse channels.json"
168[ -n "$TARGET" ] || die "no target tag for channel '$CHANNEL'"
176if [ -n "$PINNED_TAG" ]; then
177 TARGET="$PINNED_TAG"
178else
179 JSON="$(curl -fsS --max-time 20 "$CHANNELS_URL")" || die "cannot fetch channels.json"
180 TARGET="$(printf '%s' "$JSON" | CHANNEL="$CHANNEL" python3 -c 'import os,sys,json; print(json.load(sys.stdin).get(os.environ["CHANNEL"], ""))')" \
181 || die "cannot parse channels.json"
182 [ -n "$TARGET" ] || die "no target tag for channel '$CHANNEL'"
183fi
169184if [[ ! "$TARGET" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]]; then
170185 die "invalid target tag '$TARGET' for channel '$CHANNEL'"
171186fi
……@@ -183,10 +198,24 @@ CUR_SHA="$(git rev-parse HEAD)"
183198git fetch --tags --quiet origin || die "git fetch failed"
184199TARGET_SHA="$(git rev-parse -q --verify "refs/tags/$TARGET^{}")" || die "target tag $TARGET not found after fetch"
185200
201# Verify even when the target already equals HEAD or this is a dry run. A
202# no-op decision proves only commit equality; it must not let an unsigned or
203# untrusted pin masquerade as a safe installed release.
204verify_tag "$TARGET" || die "refusing to update: tag $TARGET is not signed by a trusted key (see fleet/README.md 'Signed releases')"
205
186206if [ "$CUR_SHA" = "$TARGET_SHA" ]; then
187 log "channel=$CHANNEL up-to-date at $TARGET ($TARGET_SHA)"; exit 0
207 if [ -n "$PINNED_TAG" ]; then
208 log "channel=$CHANNEL pinned=$PINNED_TAG up-to-date at $TARGET ($TARGET_SHA)"
209 else
210 log "channel=$CHANNEL up-to-date at $TARGET ($TARGET_SHA)"
211 fi
212 exit 0
213fi
214if [ -n "$PINNED_TAG" ]; then
215 log "channel=$CHANNEL pinned=$PINNED_TAG current=$CUR_VER/$CUR_SHA target=$TARGET/$TARGET_SHA"
216else
217 log "channel=$CHANNEL current=$CUR_VER/$CUR_SHA target=$TARGET/$TARGET_SHA"
188218fi
189log "channel=$CHANNEL current=$CUR_VER/$CUR_SHA target=$TARGET/$TARGET_SHA"
190219if [ "$DRY_RUN" = 1 ]; then log "dry-run: would update $CUR_SHA -> $TARGET_SHA"; exit 0; fi
191220
192221# ── helpers ────────────────────────────────────────────────────────
……@@ -300,13 +329,6 @@ rollback_to_previous() {
300329}
301330
302331# ── update ─────────────────────────────────────────────────────────
303# SUPPLY-CHAIN GATE (fail closed): never check out a tag we can't verify was
304# signed by a trusted key. This runs AFTER fetch (so the tag object + its
305# signature are local) and BEFORE checkout (so a forged/moved tag can never
306# reach the working tree, deps install, or a service restart). A failure
307# here leaves the box exactly where it was — no rollback needed.
308verify_tag "$TARGET" || die "refusing to update: tag $TARGET is not signed by a trusted key (see fleet/README.md 'Signed releases')"
309
310332git checkout --quiet "$TARGET" || die "checkout $TARGET failed"
311333deps_if_changed "$CUR_SHA" "$TARGET_SHA" || rollback_to_previous "dependency install failed on $TARGET"
312334preflight_runtime || rollback_to_previous "runtime preflight failed on $TARGET (native addon or Node ABI)"
scripts/audit-workspace-alignment.mjs +8 -2
……@@ -9185,11 +9185,17 @@ if (
91859185 fleetUpdaterScript.includes(updaterExpectedVersionArg) &&
91869186 fleetUpdaterScript.includes(updaterTargetHealthGate) &&
91879187 fleetUpdaterScript.includes(updaterRollbackHealthGate) &&
9188 fleetUpdaterScript.includes('TARGET="$PINNED_TAG"') &&
9189 fleetUpdaterScript.indexOf('verify_tag "$TARGET" ||') <
9190 fleetUpdaterScript.indexOf('if [ "$CUR_SHA" = "$TARGET_SHA" ]') &&
9191 fleetUpdaterScript.indexOf('verify_tag "$TARGET" ||') <
9192 fleetUpdaterScript.indexOf('if [ "$DRY_RUN" = 1 ]') &&
9193 fleetShellSafetyTest.includes('fleet updater supports an exact signed-tag pin') &&
91889194 fleetShellSafetyTest.includes('fleet updater health-gates target and rollback runtime versions')
91899195) {
9190 pass('fleet updater rolls back dependency-install failures, stops on failed rollback checkout, and verifies runtime versions')
9196 pass('fleet updater verifies exact pins before early exits, rolls back dependency-install failures, stops on failed rollback checkout, and verifies runtime versions')
91919197} else {
9192 fail('fleet updater can exit before rollback, continue after failed rollback checkout, or accept a stale runtime version')
9198 fail('fleet updater can accept an unverified pin, exit before rollback, continue after failed rollback checkout, or accept a stale runtime version')
91939199}
91949200
91959201if (
test/unit/fleet-shell-safety.test.js +13 -0
……@@ -19,6 +19,19 @@ test('fleet scripts pass channel names into JSON parsing as data', (t) => {
1919 t.absent(fleetStatus.includes('python3 -c "import sys,json;print(json.load(sys.stdin).get('))
2020})
2121
22test('fleet updater supports an exact signed-tag pin without changing shared channels', (t) => {
23 t.ok(updater.includes('PINNED_TAG=""'))
24 t.ok(updater.includes('PINNED_TAG[[:space:]]*=[[:space:]]*'))
25 t.ok(updater.includes("invalid pinned tag '$PINNED_TAG' in $CONF"))
26 t.ok(updater.includes('TARGET="$PINNED_TAG"'))
27 t.ok(updater.includes('pinned=$PINNED_TAG'))
28 t.ok(updater.indexOf('verify_tag "$TARGET"') > updater.indexOf('TARGET="$PINNED_TAG"'),
29 'a pinned tag still crosses the signed-tag gate')
30 t.absent(updater.includes('source "$CONF"'))
31 t.ok(fleetStatus.includes('PINNED_TAG[[:space:]]*='))
32 t.ok(fleetStatus.includes('C="$C@$P"'))
33})
34
2235test('fleet updater routes dependency-install failures through rollback', (t) => {
2336 t.ok(updater.includes('rollback_to_previous'))
2437 t.ok(updater.includes('deps_if_changed "$CUR_SHA" "$TARGET_SHA" || rollback_to_previous'))
test/unit/fleet-updater-verify-tag.test.js +32 -1
……@@ -31,6 +31,10 @@ test('updater defines a tag-signature gate wired before checkout', (t) => {
3131 t.ok(gateIdx !== -1, 'target tag is gated by verify_tag')
3232 t.ok(checkoutIdx !== -1, 'target checkout still present')
3333 t.ok(gateIdx < checkoutIdx, 'verify_tag gate precedes the checkout')
34 t.ok(gateIdx < updater.indexOf('if [ "$CUR_SHA" = "$TARGET_SHA" ]'),
35 'signature gate also precedes the up-to-date early exit')
36 t.ok(gateIdx < updater.indexOf('if [ "$DRY_RUN" = 1 ]'),
37 'signature gate also precedes the dry-run early exit')
3438
3539 // Fail-closed building blocks.
3640 t.ok(updater.includes('ALLOWED_SIGNERS='), 'allowed-signers path is configurable')
……@@ -75,7 +79,8 @@ test('updater --verify-only accepts a trusted-signed tag and rejects everything
7579 git(['config', 'user.email', 'good@hiverelay'])
7680 git(['config', 'gpg.format', 'ssh'])
7781 writeFileSync(path.join(repo, 'a'), 'hi\n')
78 git(['add', 'a'])
82 writeFileSync(path.join(repo, 'package.json'), '{"version":"1.0.0"}\n')
83 git(['add', 'a', 'package.json'])
7984 git(['commit', '-qm', 'init'])
8085
8186 // 1. annotated tag signed by the TRUSTED key
……@@ -104,4 +109,30 @@ test('updater --verify-only accepts a trusted-signed tag and rejects everything
104109 // Break-glass override is honored but loud (documented emergency escape).
105110 t.is(verify('v2.0.0-unsigned', { HIVERELAY_REQUIRE_SIGNED_TAGS: '0' }).status, 0,
106111 'HIVERELAY_REQUIRE_SIGNED_TAGS=0 disables the gate (break-glass)')
112
113 // A per-host pin must bypass the moving channel document, but it must not
114 // bypass trust. Use a bare origin because the normal updater always fetches
115 // tags before deciding whether the installed commit is already current.
116 const origin = path.join(work, 'origin.git')
117 run('git', ['clone', '--bare', '-q', repo, origin])
118 git(['remote', 'add', 'origin', origin])
119 const conf = path.join(work, 'updater.conf')
120 const pinnedDryRun = (tag) => {
121 writeFileSync(conf, `CHANNEL=stable\nPINNED_TAG=${tag}\n`)
122 return spawnSync('bash', [updaterPath, '--dry-run'], {
123 encoding: 'utf8',
124 env: {
125 ...process.env,
126 HIVERELAY_REPO_DIR: repo,
127 HIVERELAY_UPDATER_CONF: conf,
128 HIVERELAY_ALLOWED_SIGNERS: allowedSigners,
129 HIVERELAY_CHANNELS_URL: 'file:///definitely-not-a-channel-document'
130 }
131 })
132 }
133
134 const trustedPin = pinnedDryRun('v1.0.0-trusted')
135 t.is(trustedPin.status, 0, 'trusted exact pin succeeds without reading the channel document')
136 t.ok(trustedPin.stdout.includes('pinned=v1.0.0-trusted'), 'pin decision is visible to the operator')
137 t.not(pinnedDryRun('v9.9.9-forged').status, 0, 'untrusted exact pin fails even when it resolves to current HEAD')
107138})