OverviewFilesCommitsIssuesDocsReleasesPeers

P2P-Hiverelay commits / 506616c

fleet: make update failures detectable before they reach a live relay
Three blockers today were each found by hitting them in production. All three were mechanically detectable first. These are the gates. 1. Runtime preflight before the service restart. `npm ci` succeeding does not mean the relay can boot: utah-0.5gb installed cleanly and then crash-looped because require-addon resolved the package root to "/" and looked for /prebuilds/linux-x64/sodium-native.node, while the binary sat correct inside node_modules. Same tag, same install — the box that worked runs Node v22.22.0 and the box that failed runs v22.22.2, and `engines: >=20.0.0` admits both. Loading hyperswarm walks the exact chain that broke (hyperswarm -> hyperdht -> dht-rpc -> udx-native -> require-addon -> sodium-native), so it catches native-addon breakage of any origin without pinning a version we would then have to chase. A failure now rolls back instead of leaving the box crash-looping. Verified: passes on a healthy tree, and rejects a stub that throws the observed MODULE_NOT_FOUND. 2. The agent self-updates. It runs from /usr/local/bin and nothing reinstalled it, so a bug in the update path could only be fixed by hand-visiting every box — which is why rc.6's --force rollback fix is currently on no relay at all. Installs after verify_tag and after a proven-good deps install, before the restart, atomically (temp + rename, with `bash -n` between) so a crash cannot leave a partial script at a path systemd executes. The running shell is unaffected; it takes effect next tick. 3. The watchdog no longer kills a relay for answering honestly. It dropped `curl -f`, which discards the body on non-2xx — so a 503 {"ok":false,"reason":"disk-critical"} drain (PRODUCTION.md:433-436) was indistinguishable from a hang, and since a restart frees no disk the box looped SIGKILL every ~4 minutes forever. The rule is now: this watchdog exists for event-loop hangs, and any structured reply proves the loop is turning. Latent until now only because diskHealthGate defaults off — it armed the moment an operator followed the docs. Also surfaces storageAdmission.failClosed() on /health as `reason: "storage-fail-closed"`. It set _fatalReason with no emit, log or metric and nothing read the snapshot; meanwhile canAcknowledge() returned false so every gateway drive read 404'd as "still replicating" — actively misleading. Reported ahead of the disk gate: it is the more fundamental state, and unlike a drain a restart will not clear it. The generalised watchdog rule above is what keeps this from becoming a new kill loop. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
506616c63ee0b47d6cc898e8c0ff333c11a00390
by bigdestiny2 <33146784+bigdestiny2@users.noreply.github.com> · 2026-07-28T14:02:33.000Z

Changed files (3) +101 -1

fleet/health-watchdog.sh +23 -1
……@@ -42,12 +42,34 @@ if [ -n "$MainPID" ] && [ "$MainPID" != "0" ] && [ -r "/proc/$MainPID" ]; then
4242 fi
4343fi
4444
45body=$(curl -fsS --max-time "$TIMEOUT" "$URL" 2>/dev/null || true)
45# NOTE: deliberately not `curl -f`. -f discards the response body on a non-2xx
46# status, and the relay answers 503 with {"ok":false,"reason":"disk-critical"}
47# as a DELIBERATE drain signal (PRODUCTION.md:433-436). Without the body this
48# watchdog cannot tell "draining on purpose" from "event-loop hung", so it
49# counted the drain as a failure and SIGKILLed — and since a restart frees no
50# disk, the box then looped every ~4 minutes forever.
51body=$(curl -sS --max-time "$TIMEOUT" "$URL" 2>/dev/null || true)
4652if echo "$body" | grep -q '"ok"[[:space:]]*:[[:space:]]*true'; then
4753 echo 0 >"$STATE_FILE"
4854 exit 0
4955fi
5056
57# This watchdog exists for ONE failure mode: an event-loop hang that systemd
58# cannot see, where the unit is active and :9100 is listening but /health never
59# answers. A structured JSON answer — whatever it says — proves the loop is
60# turning, so it is by definition not the thing we are here to kill.
61#
62# Deliberate not-ok states (disk-critical drain, storage-fail-closed) are
63# reported precisely so an operator can act. SIGKILLing them destroys the signal
64# and fixes nothing: a restart frees no disk and does not clear a fail-closed
65# storage authority. Treat any reasoned reply as "responsive" and stand down.
66if echo "$body" | grep -q '"reason"[[:space:]]*:'; then
67 reason=$(echo "$body" | sed -n 's/.*"reason"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p')
68 log "relay responsive but not ok (reason=${reason:-unknown}) — reporting, not a hang; not restarting"
69 echo 0 >"$STATE_FILE"
70 exit 0
71fi
72
5173fails=0
5274if [ -f "$STATE_FILE" ]; then
5375 fails=$(cat "$STATE_FILE" 2>/dev/null || echo 0)
fleet/updater.sh +54 -0
……@@ -30,6 +30,14 @@ REPO_DIR="${HIVERELAY_REPO_DIR:-$HOME/hiverelay}"
3030CONF="${HIVERELAY_UPDATER_CONF:-/etc/hiverelay-updater.conf}"
3131CHANNELS_URL="${HIVERELAY_CHANNELS_URL:-https://raw.githubusercontent.com/bigdestiny2/P2P-Hiverelay/main/fleet/channels.json}"
3232SERVICE="${HIVERELAY_SERVICE:-hiverelay}"
33# Where this agent is installed, for the post-update self-refresh. Resolved from
34# $0 so a non-standard install path still refreshes itself; falls back to the
35# canonical location when $0 is not a readable file (e.g. piped from stdin).
36SELF_PATH="${HIVERELAY_UPDATER_PATH:-}"
37if [ -z "$SELF_PATH" ]; then
38 if [ -f "$0" ]; then SELF_PATH="$(cd "$(dirname "$0")" && pwd)/$(basename "$0")"
39 else SELF_PATH="/usr/local/bin/hiverelay-updater"; fi
40fi
3341API="${HIVERELAY_API:-http://127.0.0.1:9100}"
3442ENV_FILE="${HIVERELAY_ENV_FILE:-/etc/hiverelay/hiverelay.env}"
3543HEALTH_TIMEOUT="${HIVERELAY_HEALTH_TIMEOUT:-120}"
……@@ -242,6 +250,24 @@ deps_if_changed() { # $1=from-ref $2=to-ref
242250 fi
243251}
244252
253# Prove the runtime can actually boot BEFORE restarting the live service.
254#
255# `npm ci` succeeding does not mean the relay can start. On 2026-07-28
256# utah-0.5gb installed cleanly and then crash-looped, because require-addon
257# resolved the package root to "/" and looked for
258# /prebuilds/linux-x64/sodium-native.node — the binary was present and correct
259# inside node_modules. Same tag, same install; the difference was Node
260# v22.22.2 vs v22.22.0 on the box that worked. `engines: >=20.0.0` admits both.
261#
262# Loading hyperswarm walks the exact chain that failed
263# (hyperswarm -> hyperdht -> dht-rpc -> udx-native -> require-addon ->
264# sodium-native), so this catches native-addon breakage of any origin — bad
265# prebuild, wrong libc, Node ABI drift — without pinning a version we would
266# then have to chase. Cheap: a require, no network, no listeners.
267preflight_runtime() {
268 node -e 'require("hyperswarm")' >/dev/null 2>&1
269}
270
245271rollback_to_previous() {
246272 local reason="$1"
247273 log "FAIL $reason — ROLLING BACK to $CUR_VER ($CUR_SHA)"
……@@ -278,6 +304,34 @@ verify_tag "$TARGET" || die "refusing to update: tag $TARGET is not signed by a
278304
279305git checkout --quiet "$TARGET" || die "checkout $TARGET failed"
280306deps_if_changed "$CUR_SHA" "$TARGET_SHA" || rollback_to_previous "dependency install failed on $TARGET"
307preflight_runtime || rollback_to_previous "runtime preflight failed on $TARGET (native addon or Node ABI)"
308
309# SELF-UPDATE. The agent is the one component an update could never fix: it
310# runs from /usr/local/bin and nothing here reinstalled it, so a bug in the
311# update path could only be repaired by hand-visiting every box. That is
312# exactly what happened with the missing --force on the rollback checkout —
313# the fix shipped in the tag and reached no relay.
314#
315# Ordering matters. This runs AFTER verify_tag and AFTER the deps install, so
316# the script being installed comes from a signature-verified tree that has
317# already proven it can install. It runs BEFORE the service restart so a
318# failed restart rolls back with the NEW agent's logic.
319#
320# The running shell is unaffected: bash has already read this file, and the
321# replacement takes effect on the next tick. Install atomically (write a temp
322# beside the target, then rename) so a crash mid-copy cannot leave a partial
323# interpreter script at a path systemd will execute.
324if [ -f "$REPO_DIR/fleet/updater.sh" ] && ! cmp -s "$REPO_DIR/fleet/updater.sh" "$SELF_PATH"; then
325 if install -m 0755 "$REPO_DIR/fleet/updater.sh" "$SELF_PATH.next" 2>/dev/null &&
326 bash -n "$SELF_PATH.next" 2>/dev/null &&
327 mv -f "$SELF_PATH.next" "$SELF_PATH" 2>/dev/null; then
328 log "agent self-updated from $TARGET — active next tick"
329 else
330 rm -f "$SELF_PATH.next" 2>/dev/null || true
331 log "WARN agent self-update failed (non-fatal); continuing with the running agent"
332 fi
333fi
334
281335systemctl restart "$SERVICE"
282336
283337if healthy "${TARGET#v}"; then
packages/core/core/relay-node/api-health.js +24 -0
……@@ -71,6 +71,30 @@ export function buildHealthResponse ({ node, version }) {
7171 const diskSummary = diskHealthSummary(disk)
7272 const uptime = node && node.metrics ? node.metrics.getSummary().uptime : null
7373 const running = node ? node.running : false
74 const storageFatal = node && node.storageAdmission ? node.storageAdmission.fatalReason : null
75
76 // A fail-closed storage authority is a terminal state that used to be
77 // invisible everywhere: failClosed() sets _fatalReason with no emit, log or
78 // metric, and nothing reads the snapshot. Worse than silent — canAcknowledge()
79 // then returns false, so every gateway drive read 404s as "still replicating",
80 // which actively misdirects whoever is debugging it.
81 //
82 // Report it before the disk gate: if storage has failed closed, that is the
83 // more fundamental fact, and unlike a disk drain a restart will not clear it.
84 if (storageFatal) {
85 return {
86 status: 503,
87 payload: {
88 ok: false,
89 reason: 'storage-fail-closed',
90 storageFatalReason: String(storageFatal),
91 version,
92 uptime,
93 running,
94 disk: diskSummary
95 }
96 }
97 }
7498
7599 if (
76100 node &&