OverviewFilesCommitsIssuesDocsReleasesPeers

P2P-Hiverelay commits / 40ea4bf

ops(fleet): health watchdog, admin operator dashboard, stability writeup
Systemd alone cannot recover an event-loop hang: the main PID stays alive so Restart= never fires. The watchdog timer probes /health every 2 minutes and SIGKILLs + restarts after two consecutive failures, with a 90s grace after process start for boot reseed. Also lands the CS7 remigrate runbook (utah-8gb's storage/db had bloated to 8.6GB; quarantining it dropped RSS from ~5.5GB to ~110MB), the fleet stability writeup, and the Dashboard Admin Operator. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
40ea4bf0656550ec67b8a2740c4a07c622fbce29
by bigdestiny2 <33146784+bigdestiny2@users.noreply.github.com> · 2026-07-28T00:19:42.000Z

Changed files (22) +4172 -4

dashboard-admin-operator/README.md +106 -0
……@@ -0,0 +1,106 @@
1# Dashboard Admin Operator
2
3Operator-facing **HiveRelay fleet control plane** — live health, feature matrix, restart / update / doctor, feature toggles, and inventory management.
4
5This is the manage UI used for day-to-day fleet ops (not the public `dashboard/` pages or the static status HTML).
6
7## Quick start
8
9From the monorepo root:
10
11```bash
12npm run dashboard:admin-operator
13```
14
15Open **http://localhost:3458/**
16
17Faster probes while debugging:
18
19```bash
20npm run dashboard:admin-operator:dev
21```
22
23## What it does
24
25| Capability | How |
26|---|---|
27| Live fleet matrix | SSH probe → `/status` + `/health` every 60s (SSE) |
28| **Healthy ranges** | Policy by tier (small/medium/large): mem, disk, swap, watchdog |
29| **Auto-control** | After each probe: force-restart hung/degraded, doctor disk, ensure swap/watchdog |
30| Restart / update / doctor | Per-node manual actions |
31| Feature toggles | DHT WS, outboxlog, forward relay, services, eviction, … |
32| Stabilize all | Install local health watchdog + swap on every reachable box |
33| Add / remove relays | Writes inventory JSON |
34
35### Control APIs
36
37| Endpoint | Purpose |
38|---|---|
39| `GET /api/control` | Policy + last control cycle |
40| `POST /api/control` | `{ "autoControl": true/false }` |
41| `POST /api/control/run` | Run evaluation + remediation now |
42| `POST /api/control/stabilize-all` | Baseline all reachable nodes |
43
44Policy file: `control-policy.json` (cooldowns, tier limits, auto on/off).
45
46## Inventory
47
48Default fleet directory: `../fleet` (repo `fleet/`).
49
50Resolution order for the relays file:
51
521. `HIVERELAY_FLEET_INVENTORY` — explicit path
532. `fleet/relays.local.json` — **preferred** (local SSH key paths; usually gitignored)
543. `fleet/relays.json` — committed template
55
56Channels: `fleet/channels.json` (`stable` / `canary` tags).
57
58### Env
59
60| Variable | Default | Meaning |
61|---|---|---|
62| `PORT` | `3458` | HTTP listen port |
63| `FLEET_INTERVAL` | `60` | Probe interval (seconds) |
64| `HIVERELAY_FLEET_DIR` | `<repo>/fleet` | Fleet directory |
65| `HIVERELAY_FLEET_INVENTORY` | *(auto)* | Override relays JSON path |
66
67Example with an explicit inventory:
68
69```bash
70HIVERELAY_FLEET_INVENTORY=./fleet/relays.local.json \
71 npm run dashboard:admin-operator
72```
73
74## Layout
75
76```
77dashboard-admin-operator/
78 server.cjs # HTTP + SSE + SSH probe/actions
79 public/
80 index.html # shell
81 app.js # UI (table, KPIs, actions)
82 styles.css
83 README.md
84```
85
86## SSH keys
87
88Each inventory entry needs a working `sshKey` for that host, e.g.:
89
90- `utah`, `dubai` → `~/.ssh/id_ed25519`
91- other cloudzy boxes → `~/.ssh/cloudzy_hiverelay` (or `~/.ssh/hiverelay_fleet` symlink)
92
93Without the right key the node shows **degraded** / **offline** even if the process is healthy.
94
95## Relation to other dashboards
96
97| Surface | Role |
98|---|---|
99| **`dashboard-admin-operator/`** | This — manage + control the fleet |
100| `dashboard/` | Public product UI (catalog, network, …) |
101| `docs/html/FLEET-OPS-LIVE.html` | Static snapshot status page |
102| `scripts/local-fleet-dashboard.mjs` | Proxied single-relay `/dashboard` with API key |
103
104## Security note
105
106This server runs **locally** and executes SSH as your user against the fleet. Do not expose port `3458` to the public internet.
dashboard-admin-operator/control-policy.json +46 -0
……@@ -0,0 +1,46 @@
1{
2 "_doc": "Fleet control plane policy — healthy ranges + auto-remediation. Used by Dashboard Admin Operator.",
3 "autoControl": true,
4 "cooldownsSec": {
5 "forceRestart": 480,
6 "doctor": 1800,
7 "ensureBaseline": 3600,
8 "ensureSwap": 7200
9 },
10 "global": {
11 "requireHealthOk": true,
12 "requireRunning": true,
13 "requireWatchdog": true,
14 "minSwapMB": 256,
15 "diskPctWarn": 75,
16 "diskPctCritical": 90,
17 "forceRestartOnDegraded": true,
18 "forceRestartOnHung": true,
19 "maxConcurrentRemediations": 3
20 },
21 "tiers": {
22 "small": {
23 "ramGBMax": 1.1,
24 "memMBMax": 420,
25 "memMBWarn": 340,
26 "diskPctMax": 85,
27 "plugins": ["outboxlog", "notify", "vrf"],
28 "description": "≤1GB boxes — light plugins only"
29 },
30 "medium": {
31 "ramGBMin": 1.1,
32 "ramGBMax": 4,
33 "memMBMax": 1600,
34 "memMBWarn": 1200,
35 "diskPctMax": 85,
36 "description": "2–4GB seed boxes"
37 },
38 "large": {
39 "ramGBMin": 4,
40 "memMBMax": 4800,
41 "memMBWarn": 3500,
42 "diskPctMax": 90,
43 "description": "≥4GB canaries / capacity nodes"
44 }
45 }
46}
dashboard-admin-operator/controller.cjs +410 -0
……@@ -0,0 +1,410 @@
1'use strict'
2
3/**
4 * Fleet control plane — evaluate healthy ranges + auto-remediate.
5 * Called after each probe cycle when policy.autoControl is true.
6 */
7
8const fs = require('fs')
9const path = require('path')
10
11function defaultPolicy () {
12 return {
13 autoControl: true,
14 cooldownsSec: { forceRestart: 480, doctor: 1800, ensureBaseline: 3600, ensureSwap: 7200 },
15 global: {
16 requireHealthOk: true,
17 requireRunning: true,
18 requireWatchdog: true,
19 minSwapMB: 256,
20 diskPctWarn: 75,
21 diskPctCritical: 90,
22 forceRestartOnDegraded: true,
23 forceRestartOnHung: true,
24 maxConcurrentRemediations: 3
25 },
26 tiers: {
27 small: { ramGBMax: 1.1, memMBMax: 300, memMBWarn: 240, diskPctMax: 85 },
28 medium: { ramGBMin: 1.1, ramGBMax: 4, memMBMax: 1600, memMBWarn: 1200, diskPctMax: 85 },
29 large: { ramGBMin: 4, memMBMax: 4800, memMBWarn: 3500, diskPctMax: 90 }
30 }
31 }
32}
33
34function loadPolicy (policyPath) {
35 try {
36 if (fs.existsSync(policyPath)) {
37 return { ...defaultPolicy(), ...JSON.parse(fs.readFileSync(policyPath, 'utf8')) }
38 }
39 } catch (e) {
40 console.error('[controller] policy load failed:', e.message)
41 }
42 return defaultPolicy()
43}
44
45function savePolicy (policyPath, policy) {
46 const out = { ...policy }
47 delete out._loadedAt
48 fs.writeFileSync(policyPath, JSON.stringify(out, null, 2) + '\n')
49}
50
51function tierFor (node, policy) {
52 const ram = Number(node.ramGB) || 2
53 const tiers = policy.tiers || {}
54 if (ram <= (tiers.small?.ramGBMax ?? 1.1)) return { name: 'small', ...tiers.small }
55 if (ram < (tiers.large?.ramGBMin ?? 4)) return { name: 'medium', ...tiers.medium }
56 return { name: 'large', ...tiers.large }
57}
58
59/**
60 * @returns {{ violations: object[], actions: object[] }}
61 */
62function evaluateNode (node, policy) {
63 const violations = []
64 const tier = tierFor(node, policy)
65 const g = policy.global || {}
66
67 if (node.status === 'offline') {
68 if (node.reachable) {
69 violations.push({ code: 'degraded_ssh_ok', severity: 'critical', msg: 'SSH ok but probe failed', auto: 'forceRestart' })
70 } else {
71 violations.push({ code: 'offline', severity: 'critical', msg: 'SSH unreachable', auto: null })
72 }
73 return { violations, tier: tier.name, actions: suggestActions(violations) }
74 }
75
76 // healthOk from enhanced probe; fall back to running/status
77 const healthOk = node.healthOk === true || (node.running === true && node.status === 'healthy')
78 if (g.requireHealthOk && !healthOk) {
79 violations.push({ code: 'health_fail', severity: 'critical', msg: ' /health not ok', auto: 'forceRestart' })
80 }
81 if (g.requireRunning && node.running !== true && healthOk) {
82 // still warming or status lag — soft
83 if (node.status === 'degraded') {
84 violations.push({ code: 'not_running', severity: 'warning', msg: 'process not fully running', auto: 'forceRestart' })
85 }
86 }
87 if (node.status === 'degraded' && g.forceRestartOnDegraded) {
88 if (!violations.some(v => v.code === 'health_fail' || v.code === 'not_running')) {
89 violations.push({ code: 'degraded', severity: 'critical', msg: 'node marked degraded', auto: 'forceRestart' })
90 }
91 }
92
93 const mem = Number(node.memCurrentMB) || 0
94 if (mem > 0 && tier.memMBMax && mem > tier.memMBMax) {
95 violations.push({
96 code: 'mem_over_max',
97 severity: 'critical',
98 msg: `mem ${mem}MB > tier max ${tier.memMBMax}MB (${tier.name})`,
99 auto: 'forceRestart'
100 })
101 } else if (mem > 0 && tier.memMBWarn && mem > tier.memMBWarn) {
102 violations.push({
103 code: 'mem_warn',
104 severity: 'warning',
105 msg: `mem ${mem}MB > warn ${tier.memMBWarn}MB (${tier.name})`,
106 auto: null
107 })
108 }
109
110 const disk = Number(node.diskPct) || 0
111 if (disk >= (g.diskPctCritical || 90)) {
112 violations.push({ code: 'disk_critical', severity: 'critical', msg: `disk ${disk}%`, auto: 'doctor' })
113 } else if (disk >= (g.diskPctWarn || 75) || (tier.diskPctMax && disk >= tier.diskPctMax)) {
114 violations.push({ code: 'disk_warn', severity: 'warning', msg: `disk ${disk}%`, auto: null })
115 }
116
117 const swap = Number(node.swapMB) || 0
118 if (g.minSwapMB != null && swap < g.minSwapMB) {
119 violations.push({ code: 'no_swap', severity: 'warning', msg: `swap ${swap}MB < ${g.minSwapMB}MB`, auto: 'ensureSwap' })
120 }
121
122 if (g.requireWatchdog && node.watchdog !== 'enabled' && node.watchdog !== true) {
123 violations.push({ code: 'no_watchdog', severity: 'warning', msg: 'health watchdog timer missing', auto: 'ensureBaseline' })
124 }
125
126 return { violations, tier: tier.name, actions: suggestActions(violations) }
127}
128
129function suggestActions (violations) {
130 const seen = new Set()
131 const actions = []
132 for (const v of violations) {
133 if (v.auto && !seen.has(v.auto)) {
134 seen.add(v.auto)
135 actions.push({ type: v.auto, reason: v.code, severity: v.severity })
136 }
137 }
138 // prefer forceRestart first
139 actions.sort((a, b) => {
140 const order = { forceRestart: 0, doctor: 1, ensureSwap: 2, ensureBaseline: 3 }
141 return (order[a.type] ?? 9) - (order[b.type] ?? 9)
142 })
143 return actions
144}
145
146function evaluateFleet (nodes, policy) {
147 const results = []
148 for (const n of nodes) {
149 const ev = evaluateNode(n, policy)
150 results.push({
151 name: n.name,
152 status: n.status,
153 tier: ev.tier,
154 violations: ev.violations,
155 actions: ev.actions,
156 memCurrentMB: n.memCurrentMB,
157 diskPct: n.diskPct,
158 swapMB: n.swapMB,
159 watchdog: n.watchdog,
160 healthOk: n.healthOk,
161 running: n.running
162 })
163 }
164 const critical = results.filter(r => r.violations.some(v => v.severity === 'critical')).length
165 const warnings = results.filter(r => r.violations.some(v => v.severity === 'warning') && !r.violations.some(v => v.severity === 'critical')).length
166 const inRange = results.filter(r => r.violations.length === 0 && r.status === 'healthy').length
167 return {
168 evaluatedAt: new Date().toISOString(),
169 nodes: results,
170 summary: {
171 total: results.length,
172 inRange,
173 critical,
174 warnings,
175 offline: results.filter(r => r.status === 'offline').length
176 }
177 }
178}
179
180/** Remote shell snippets for remediation */
181const REMOTE = {
182 forceRestart: [
183 'set +e',
184 'systemctl kill -s SIGKILL hiverelay 2>/dev/null',
185 'systemctl stop hiverelay 2>/dev/null',
186 'fuser -k -9 9100/tcp 2>/dev/null',
187 'sleep 2',
188 'systemctl reset-failed hiverelay 2>/dev/null',
189 'systemctl start hiverelay',
190 'for i in $(seq 1 20); do',
191 ' sleep 3',
192 ' H=$(curl -sS --max-time 4 http://127.0.0.1:9100/health 2>/dev/null)',
193 ' if echo "$H" | grep -q ok; then echo RESTART_OK; exit 0; fi',
194 'done',
195 'echo RESTART_PENDING',
196 'exit 0'
197 ].join('\n'),
198
199 doctor: [
200 'set +e',
201 'echo "=== doctor ==="',
202 'SW=$(free -m | awk "/Swap/{print \\$2}")',
203 'if [ "${SW:-0}" = "0" ]; then',
204 ' fallocate -l 2G /swapfile 2>/dev/null || dd if=/dev/zero of=/swapfile bs=1M count=2048',
205 ' chmod 600 /swapfile; mkswap /swapfile; swapon /swapfile',
206 ' grep -q /swapfile /etc/fstab || echo "/swapfile none swap sw 0 0" >> /etc/fstab',
207 ' sysctl -w vm.swappiness=10 >/dev/null',
208 'fi',
209 'rm -f /root/.hiverelay/storage/*.lock 2>/dev/null',
210 'journalctl --vacuum-size=80M 2>/dev/null',
211 'df -h / | tail -1',
212 'echo DOCTOR_OK'
213 ].join('\n'),
214
215 ensureSwap: [
216 'set +e',
217 'if [ "$(swapon --show --noheadings 2>/dev/null | wc -l)" -eq 0 ]; then',
218 ' fallocate -l 2G /swapfile 2>/dev/null || dd if=/dev/zero of=/swapfile bs=1M count=2048',
219 ' chmod 600 /swapfile; mkswap /swapfile; swapon /swapfile',
220 ' grep -q /swapfile /etc/fstab || echo "/swapfile none swap sw 0 0" >> /etc/fstab',
221 ' sysctl -w vm.swappiness=10 >/dev/null',
222 ' echo SWAP_ADDED',
223 'else',
224 ' echo SWAP_PRESENT',
225 'fi'
226 ].join('\n'),
227
228 ensureBaseline: [
229 'set +e',
230 '# health watchdog install (inline)',
231 'if [ ! -f /usr/local/bin/hiverelay-health-watchdog.sh ]; then',
232 ' cat > /usr/local/bin/hiverelay-health-watchdog.sh << "WATCH"',
233 '#!/usr/bin/env bash',
234 'set -uo pipefail',
235 'URL="${HIVERELAY_HEALTH_URL:-http://127.0.0.1:9100/health}"',
236 'TIMEOUT="${HIVERELAY_HEALTH_TIMEOUT:-5}"',
237 'NEED_FAILS="${HIVERELAY_HEALTH_FAILS:-2}"',
238 'STATE_FILE=/run/hiverelay-health-watchdog.failcount',
239 'UNIT=hiverelay',
240 'active=$(systemctl is-active "$UNIT" 2>/dev/null || echo inactive)',
241 '[ "$active" = "active" ] || [ "$active" = "activating" ] || { echo 0 >"$STATE_FILE"; exit 0; }',
242 'body=$(curl -fsS --max-time "$TIMEOUT" "$URL" 2>/dev/null || true)',
243 'if echo "$body" | grep -q \'"ok"[[:space:]]*:[[:space:]]*true\'; then echo 0 >"$STATE_FILE"; exit 0; fi',
244 'fails=0; [ -f "$STATE_FILE" ] && fails=$(cat "$STATE_FILE" 2>/dev/null || echo 0)',
245 'fails=$((fails+1)); echo "$fails" >"$STATE_FILE"',
246 '[ "$fails" -lt "$NEED_FAILS" ] && exit 0',
247 'systemctl kill -s SIGKILL "$UNIT" 2>/dev/null; sleep 1; fuser -k -9 9100/tcp 2>/dev/null; sleep 1',
248 'systemctl reset-failed "$UNIT" 2>/dev/null; systemctl start "$UNIT" 2>/dev/null',
249 'echo 0 >"$STATE_FILE"',
250 'WATCH',
251 ' chmod 0755 /usr/local/bin/hiverelay-health-watchdog.sh',
252 'fi',
253 'cat > /etc/systemd/system/hiverelay-health-watchdog.service << "EOF"',
254 '[Unit]',
255 'Description=HiveRelay health watchdog',
256 '[Service]',
257 'Type=oneshot',
258 'ExecStart=/usr/local/bin/hiverelay-health-watchdog.sh',
259 'TimeoutStartSec=30',
260 'EOF',
261 'cat > /etc/systemd/system/hiverelay-health-watchdog.timer << "EOF"',
262 '[Unit]',
263 'Description=HiveRelay health watchdog every 2m',
264 '[Timer]',
265 'OnBootSec=3min',
266 'OnUnitActiveSec=2min',
267 'AccuracySec=30s',
268 'Persistent=true',
269 'Unit=hiverelay-health-watchdog.service',
270 '[Install]',
271 'WantedBy=timers.target',
272 'EOF',
273 'mkdir -p /etc/systemd/system/hiverelay.service.d',
274 'cat > /etc/systemd/system/hiverelay.service.d/stabilize-restart.conf << "EOF"',
275 '[Service]',
276 'Restart=always',
277 'RestartSec=15',
278 'TimeoutStopSec=12',
279 'KillMode=mixed',
280 'FinalKillSignal=SIGKILL',
281 'SendSIGKILL=yes',
282 'EOF',
283 'systemctl daemon-reload',
284 'systemctl enable --now hiverelay-health-watchdog.timer 2>/dev/null',
285 'echo BASELINE_OK'
286 ].join('\n')
287}
288
289function createController ({ policyPath, sshAction, logAction, probeNode, loadRelays }) {
290 let policy = loadPolicy(policyPath)
291 policy._loadedAt = new Date().toISOString()
292 const lastActionAt = new Map() // key: `${node}:${type}` -> epoch ms
293 let lastCycle = null
294 let controlling = false
295
296 function cooldownOk (node, type) {
297 const sec = (policy.cooldownsSec && policy.cooldownsSec[type]) || 600
298 const key = `${node}:${type}`
299 const prev = lastActionAt.get(key) || 0
300 return Date.now() - prev >= sec * 1000
301 }
302
303 function markAction (node, type) {
304 lastActionAt.set(`${node}:${type}`, Date.now())
305 }
306
307 function getPolicy () { return policy }
308
309 function setAutoControl (enabled) {
310 policy.autoControl = !!enabled
311 try { savePolicy(policyPath, policy) } catch {}
312 logAction('fleet', 'auto-control', enabled ? 'ENABLED' : 'DISABLED')
313 return policy.autoControl
314 }
315
316 function reloadPolicy () {
317 policy = loadPolicy(policyPath)
318 policy._loadedAt = new Date().toISOString()
319 return policy
320 }
321
322 /**
323 * Run evaluation + optional auto-remediation against current fleetState nodes.
324 * @param {object[]} nodes
325 */
326 async function runControlCycle (nodes, { force = false } = {}) {
327 if (controlling) return lastCycle
328 controlling = true
329 try {
330 const evaluation = evaluateFleet(nodes, policy)
331 const remediations = []
332 const auto = force || policy.autoControl
333 const maxN = policy.global?.maxConcurrentRemediations || 3
334
335 if (auto) {
336 const plan = []
337 for (const r of evaluation.nodes) {
338 for (const a of r.actions) {
339 if (!cooldownOk(r.name, a.type)) continue
340 plan.push({ name: r.name, ...a })
341 }
342 }
343 // one action per node, critical first
344 const byNode = new Map()
345 for (const p of plan) {
346 if (!byNode.has(p.name)) byNode.set(p.name, p)
347 }
348 const batch = [...byNode.values()].slice(0, maxN)
349 const relays = loadRelays()
350
351 await Promise.all(batch.map(async (p) => {
352 const relay = relays.find(x => x.name === p.name)
353 if (!relay) return
354 const cmd = REMOTE[p.type]
355 if (!cmd) return
356 try {
357 logAction(p.name, `auto:${p.type}`, `reason=${p.reason}`)
358 markAction(p.name, p.type)
359 const out = await sshAction(relay, cmd, `auto:${p.type}`)
360 remediations.push({ node: p.name, type: p.type, reason: p.reason, ok: true, detail: String(out).slice(0, 200) })
361 // re-probe node after remediation
362 try {
363 const updated = await probeNode(relay)
364 const idx = nodes.findIndex(n => n.name === p.name)
365 if (idx >= 0) nodes[idx] = updated
366 } catch {}
367 } catch (err) {
368 remediations.push({ node: p.name, type: p.type, reason: p.reason, ok: false, detail: err.message?.slice(0, 200) })
369 }
370 }))
371 }
372
373 // re-evaluate after remediations
374 const after = remediations.length ? evaluateFleet(nodes, policy) : evaluation
375 lastCycle = {
376 ...after,
377 autoControl: !!policy.autoControl,
378 remediations,
379 forced: !!force
380 }
381 return lastCycle
382 } finally {
383 controlling = false
384 }
385 }
386
387 function getLastCycle () { return lastCycle }
388
389 return {
390 getPolicy,
391 setAutoControl,
392 reloadPolicy,
393 runControlCycle,
394 getLastCycle,
395 evaluateFleet: (nodes) => evaluateFleet(nodes, policy),
396 evaluateNode: (node) => evaluateNode(node, policy),
397 REMOTE
398 }
399}
400
401module.exports = {
402 loadPolicy,
403 savePolicy,
404 defaultPolicy,
405 tierFor,
406 evaluateNode,
407 evaluateFleet,
408 createController,
409 REMOTE
410}
dashboard-admin-operator/public/app.js +495 -0
……@@ -0,0 +1,495 @@
1/* HiveRelay · Dashboard Admin Operator — live fleet control plane */
2
3let state = null
4let evtSource = null
5let expandedNode = null
6
7const REGION_NAMES = { NA: 'North America', EU: 'Europe', APAC: 'Asia-Pacific', ME: 'Middle East' }
8const FEATURE_LABELS = {
9 coreRelay: 'Core Relay', seeding: 'Seeding', services: 'Service Layer',
10 dhtRelayWs: 'DHT Relay (WS)', tor: 'Tor Transport', holesail: 'Holesail',
11 outboxlog: 'OutboxLog', forwardRelay: 'Forward Relay', signedDirectory: 'Signed Directory',
12 vrf: 'VRF (RFC 9381)', poker: 'Poker Substrate', zk: 'ZK Proofs',
13 arbitration: 'Arbitration', payment: 'Payments', eviction: 'Auto-Eviction',
14 autoHeal: 'Auto-Heal', distributedDrive: 'Distributed Drive'
15}
16const TOGGLEABLE = ['dhtRelayWs', 'outboxlog', 'forwardRelay', 'signedDirectory', 'services', 'eviction']
17
18async function fetchState () {
19 try { state = await (await fetch('/api/state')).json(); render() }
20 catch { document.getElementById('app').innerHTML = '<div class="loading">Connection lost...</div>' }
21}
22
23function connectSSE () {
24 if (evtSource) evtSource.close()
25 evtSource = new EventSource('/api/events')
26 evtSource.onmessage = (e) => {
27 try {
28 const msg = JSON.parse(e.data)
29 if (msg.type === 'fleet') {
30 if (!state) { fetchState(); return }
31 state.nodes = msg.data; state.lastProbe = msg.timestamp
32 state.summary = computeSummary(msg.data)
33 render()
34 } else if (msg.type === 'action') {
35 if (state) { state.actions = [msg.data, ...(state.actions || [])] }
36 showToast(msg.data)
37 } else if (msg.type === 'control') {
38 if (state) {
39 state.control = {
40 ...(state.control || {}),
41 lastCycle: msg.data,
42 autoControl: msg.data.autoControl,
43 inRange: msg.data.summary?.inRange,
44 critical: msg.data.summary?.critical,
45 warnings: msg.data.summary?.warnings
46 }
47 render()
48 }
49 }
50 } catch {}
51 }
52 evtSource.onerror = () => { evtSource.close(); setTimeout(() => { connectSSE(); fetchState() }, 5000) }
53}
54
55function computeSummary (nodes) {
56 const regions = {}
57 for (const n of nodes) {
58 const r = n.region || 'Unknown'
59 if (!regions[r]) regions[r] = { total: 0, healthy: 0, degraded: 0, offline: 0 }
60 regions[r].total++
61 if (n.status === 'healthy') regions[r].healthy++
62 else if (n.status === 'offline') regions[r].offline++
63 else regions[r].degraded++
64 }
65 return {
66 total: nodes.length, healthy: nodes.filter(n => n.status === 'healthy').length,
67 degraded: nodes.filter(n => n.status === 'degraded').length,
68 offline: nodes.filter(n => n.status === 'offline').length,
69 running: nodes.filter(n => n.running === true).length,
70 diskWarnings: nodes.filter(n => n.diskPct && n.diskPct > 75).length,
71 totalRamGB: Math.round(nodes.reduce((s, n) => s + (n.ramGB || 0), 0)),
72 totalDiskGB: nodes.reduce((s, n) => s + (n.diskGB || 0), 0),
73 totalSeeded: nodes.reduce((s, n) => s + (n.seededApps || 0), 0),
74 totalConns: nodes.reduce((s, n) => s + (n.connections || 0), 0),
75 regions
76 }
77}
78
79function fmtBytes (b) {
80 if (!b || b === 0) return '0 B'
81 const u = ['B','KB','MB','GB','TB','PB']
82 const i = Math.floor(Math.log(b) / Math.log(1024))
83 return (b / Math.pow(1024, i)).toFixed(1) + ' ' + u[i]
84}
85
86function esc (s) { if (s == null) return ''; const d = document.createElement('div'); d.textContent = String(s); return d.innerHTML }
87
88function render () {
89 if (!state || !state.nodes) return
90 const s = state.summary
91 const probeAgo = state.lastProbe ? Math.round((Date.now() - new Date(state.lastProbe).getTime()) / 1000) : '?'
92
93 document.getElementById('app').innerHTML = `
94 <div class="header">
95 <div class="header-left">
96 <div class="logo-mark">H</div>
97 <div><h1>Dashboard Admin Operator</h1>
98 <div class="header-meta">Fleet control · ${state.nodes.length} nodes · ${s.totalSeeded} apps seeded · ${s.totalConns} connections · probed ${probeAgo}s ago</div></div>
99 </div>
100 <div class="header-right">
101 <div class="live-indicator"><span class="live-pulse"></span><span>LIVE</span></div>
102 <button class="btn" onclick="runControlNow()" title="Evaluate ranges + remediate now">Control cycle</button>
103 <button class="btn" onclick="stabilizeAll()" title="Install watchdog + swap on all reachable">Stabilize all</button>
104 <button class="btn" onclick="reprobe()">Re-probe</button>
105 </div>
106 </div>
107 <div class="container">
108 ${renderKPIs(s)}
109 ${renderControlPanel(state.control)}
110 ${renderRegions(s.regions)}
111 ${renderNodeTable(state.nodes)}
112 ${renderActionLog(state.actions)}
113 </div>
114 <div id="toast-container"></div>
115 <div class="footer"><div class="footer-meta">Admin Operator · full fleet control · probe ${state.interval||60}s · auto-control ${state.control?.autoControl ? 'ON' : 'OFF'} · ${new Date().toISOString()}</div></div>
116 `
117}
118
119function renderControlPanel (ctrl) {
120 if (!ctrl) {
121 return `<div class="control-panel"><div class="control-panel-head"><strong>Fleet control</strong><span class="dim">waiting for first probe…</span></div></div>`
122 }
123 const auto = !!ctrl.autoControl
124 const cycle = ctrl.lastCycle
125 const sum = cycle?.summary || {}
126 const nodes = cycle?.nodes || []
127 const violators = nodes.filter(n => (n.violations || []).length > 0)
128 const remediations = cycle?.remediations || []
129 return `
130 <div class="control-panel">
131 <div class="control-panel-head">
132 <div>
133 <strong>Fleet control plane</strong>
134 <span class="dim"> · healthy ranges by tier · auto-remediate hangs/OOM/disk</span>
135 </div>
136 <div class="control-panel-actions">
137 <label class="toggle-auto" title="When on, each probe auto-restarts hung nodes and enforces baseline">
138 <input type="checkbox" ${auto ? 'checked' : ''} onchange="setAutoControl(this.checked)">
139 <span>Auto-control ${auto ? 'ON' : 'OFF'}</span>
140 </label>
141 </div>
142 </div>
143 <div class="control-kpis">
144 <div class="ckpi ok"><div class="ckpi-v">${sum.inRange ?? '—'}</div><div class="ckpi-l">In range</div></div>
145 <div class="ckpi warn"><div class="ckpi-v">${sum.warnings ?? 0}</div><div class="ckpi-l">Warnings</div></div>
146 <div class="ckpi crit"><div class="ckpi-v">${sum.critical ?? 0}</div><div class="ckpi-l">Critical</div></div>
147 <div class="ckpi"><div class="ckpi-v">${sum.offline ?? 0}</div><div class="ckpi-l">Offline</div></div>
148 <div class="ckpi"><div class="ckpi-v">${remediations.length}</div><div class="ckpi-l">Last fixes</div></div>
149 </div>
150 <div class="control-ranges">
151 <span class="range-chip">small ≤1GB: mem≤300MB light plugins</span>
152 <span class="range-chip">medium 2–4GB: mem≤1.6GB</span>
153 <span class="range-chip">large ≥4GB: mem≤4.8GB</span>
154 <span class="range-chip">swap ≥256MB · watchdog required · disk warn 75%</span>
155 </div>
156 ${violators.length ? `
157 <div class="violations">
158 <div class="section-title" style="margin:8px 0 6px">Out of range <span class="count">${violators.length}</span></div>
159 ${violators.map(v => `
160 <div class="violation-row">
161 <strong>${esc(v.name)}</strong>
162 <span class="badge ${esc(v.tier||'')}">${esc(v.tier||'?')}</span>
163 ${(v.violations||[]).map(x => `<span class="vchip ${x.severity}">${esc(x.code)}: ${esc(x.msg)}</span>`).join('')}
164 ${(v.actions||[]).map(a => `<span class="auto-act">→ ${esc(a.type)}</span>`).join('')}
165 </div>`).join('')}
166 </div>` : `<div class="in-range-ok">All reachable nodes within healthy ranges (or offline only).</div>`}
167 ${remediations.length ? `
168 <div class="remed-log">
169 ${remediations.map(r => `<div class="remed-item ${r.ok?'ok':'fail'}">${esc(r.node)} · auto:${esc(r.type)} · ${esc(r.reason)} · ${r.ok?'ok':'fail'} ${esc(r.detail||'')}</div>`).join('')}
170 </div>` : ''}
171 </div>`
172}
173
174async function setAutoControl (enabled) {
175 try {
176 const res = await fetch('/api/control', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ autoControl: enabled }) })
177 const data = await res.json()
178 if (state) {
179 state.control = state.control || {}
180 state.control.autoControl = data.autoControl
181 }
182 showToast({ node: 'fleet', action: 'auto-control', detail: enabled ? 'ENABLED' : 'DISABLED' })
183 render()
184 } catch (e) { alert(e.message) }
185}
186
187async function runControlNow () {
188 showToast({ node: 'fleet', action: 'control cycle', detail: 'running…' })
189 try {
190 const res = await fetch('/api/control/run', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ force: true }) })
191 const data = await res.json()
192 if (data.error) alert(data.error)
193 else showToast({ node: 'fleet', action: 'control cycle', detail: `done · ${data.cycle?.remediations?.length||0} remediations` })
194 await fetchState()
195 } catch (e) { alert(e.message) }
196}
197
198async function stabilizeAll () {
199 if (!confirm('Install health watchdog + ensure swap on all reachable nodes?')) return
200 showToast({ node: 'fleet', action: 'stabilize-all', detail: 'running…' })
201 try {
202 const res = await fetch('/api/control/stabilize-all', { method: 'POST' })
203 const data = await res.json()
204 if (data.error) alert(data.error)
205 else {
206 const ok = (data.results || []).filter(r => r.ok).length
207 showToast({ node: 'fleet', action: 'stabilize-all', detail: `${ok} nodes ok` })
208 }
209 await fetchState()
210 } catch (e) { alert(e.message) }
211}
212
213function renderKPIs (s) {
214 return `<div class="kpis">
215 <div class="kpi neutral"><div class="kpi-value">${s.total}</div><div class="kpi-label">Nodes</div></div>
216 <div class="kpi success"><div class="kpi-value">${s.healthy}</div><div class="kpi-label">Healthy</div></div>
217 <div class="kpi warning"><div class="kpi-value">${s.degraded}</div><div class="kpi-label">Degraded</div></div>
218 <div class="kpi critical"><div class="kpi-value">${s.offline}</div><div class="kpi-label">Offline</div></div>
219 <div class="kpi success"><div class="kpi-value">${s.running}</div><div class="kpi-label">Relay Active</div></div>
220 <div class="kpi neutral"><div class="kpi-value">${fmtBytes(s.totalDiskGB * 1e9)}</div><div class="kpi-label">Total Disk</div></div>
221 <div class="kpi neutral"><div class="kpi-value">${s.totalRamGB} GB</div><div class="kpi-label">Total RAM</div></div>
222 </div>`
223}
224
225function renderRegions (regions) {
226 if (!regions) return ''
227 return `<div class="regions">${Object.entries(regions).map(([code, r]) => `
228 <div class="region-card"><div class="region-name">${REGION_NAMES[code]||code}</div>
229 <div class="region-stats"><span class="up">${r.healthy} up</span>${r.degraded?`<span class="deg">${r.degraded} deg</span>`:''}${r.offline?`<span class="down">${r.offline} down</span>`:''}<span class="total">${r.total}</span></div></div>`).join('')}</div>`
230}
231
232function renderNodeTable (nodes) {
233 const sorted = [...nodes].sort((a, b) => {
234 const order = { offline: 3, degraded: 2, healthy: 1 }
235 return (order[a.status]||3) - (order[b.status]||3) || a.name.localeCompare(b.name)
236 })
237 return `
238 <div class="section">
239 <div class="section-title">All Nodes <span class="count">${nodes.length}</span> <button class="btn add-btn" onclick="showAddRelay()">+ Add Relay</button></div>
240 <table class="tbl"><thead><tr>
241 <th></th><th>Node</th><th>Address</th><th>Version</th><th>Relay</th>
242 <th>Disk</th><th>Mem</th><th>RAM%</th><th>Load</th><th>Apps</th><th>Conns</th>
243 <th>WD</th><th>Services</th><th>Channel</th><th>Region</th><th>Spec</th>
244 </tr></thead><tbody>
245 ${sorted.map(n => renderRow(n)).join('')}
246 ${expandedNode ? renderDetailRow(sorted.find(n => n.name === expandedNode)) : ''}
247 </tbody></table></div>`
248}
249
250function renderRow (n) {
251 const isExpanded = expandedNode === n.name
252 const verOk = n.version && n.version !== 'none' && n.targetVersion && n.version === n.targetVersion
253 const svcCount = n.features?.services?.count || n.services?.length || 0
254 const wdOk = n.watchdog === 'enabled'
255 const mem = n.memCurrentMB
256 return `<tr class="${n.status} ${isExpanded ? 'expanded' : ''}" onclick="toggleDetail('${esc(n.name)}')">
257 <td><span class="chevron ${isExpanded ? 'open' : ''}">&#9654;</span></td>
258 <td><span class="indicator"><span class="dot ${n.status}"></span></span> <strong>${esc(n.name)}</strong></td>
259 <td class="mono">${esc(n.publicIp)}</td>
260 <td class="${verOk?'version-ok':'version-bad'} mono-val">${esc(n.version||'—')}</td>
261 <td>${n.running&&n.healthOk!==false?'<span class="ok-text">running</span>':(n.running?'<span class="err-text">hung?</span>':'<span class="err-text">stopped</span>')}</td>
262 <td>${renderDiskBar(n)}</td>
263 <td class="mono-val">${mem!=null?mem+'M':'—'}</td>
264 <td>${renderRamBar(n)}</td>
265 <td class="mono">${n.loadAvg!=null?Number(n.loadAvg).toFixed(2):'—'}</td>
266 <td class="mono-val">${n.seededApps||'—'}</td>
267 <td class="mono-val">${n.connections||'—'}</td>
268 <td>${wdOk?'<span class="ok-text">on</span>':'<span class="err-text">off</span>'}</td>
269 <td>${svcCount?`<span class="svc-count">${svcCount}</span>`:'—'}</td>
270 <td><span class="badge ${esc(n.channel||'stable')}">${esc(n.channel||'stable')}</span></td>
271 <td>${esc(n.region||'—')}</td>
272 <td class="mono">${n.vcpu||'?'}c/${n.ramGB||'?'}G/${n.diskGB||'?'}G</td>
273 </tr>`
274}
275
276function renderDetailRow (n) {
277 if (!n) return ''
278 const f = n.features || {}
279 const offline = n.status === 'offline'
280
281 return `<tr class="detail-row"><td colspan="16"><div class="detail-panel">
282 <div class="detail-grid">
283 <div class="detail-col">
284 <div class="detail-section-title">Features & Services</div>
285 <div class="feature-grid">
286 ${Object.entries(FEATURE_LABELS).map(([key, label]) => {
287 const feat = f[key] || {}
288 const isOn = feat.enabled === true
289 const canToggle = TOGGLEABLE.includes(key) && !offline
290 return `<div class="feature-item ${isOn?'on':'off'}">
291 <span class="feat-status ${isOn?'on':'off'}">${isOn?'ON':'OFF'}</span>
292 <span class="feat-label">${label}</span>
293 ${feat.running != null ? `<span class="feat-extra">${feat.running?'running':'stopped'}</span>` : ''}
294 ${feat.count != null ? `<span class="feat-extra">${feat.count} svc</span>` : ''}
295 ${feat.coresSeeded != null ? `<span class="feat-extra">${feat.coresSeeded} cores</span>` : ''}
296 ${canToggle ? `<button class="feat-toggle ${isOn?'off':'on'}" onclick="toggleFeature('${esc(n.name)}','${key}',${!isOn})">${isOn?'Disable':'Enable'}</button>` : ''}
297 </div>`
298 }).join('')}
299 </div>
300 </div>
301 <div class="detail-col">
302 <div class="detail-section-title">Runtime Stats</div>
303 <div class="stats-grid">
304 ${renderStat('Version', n.version)}
305 ${renderStat('Target', n.targetVersion)}
306 ${renderStat('Uptime since', n.uptime)}
307 ${renderStat('Restarts', n.restarts)}
308 ${renderStat('Memory', n.memCurrentMB != null ? n.memCurrentMB + ' MB' : '—')}
309 ${renderStat('Tasks', n.tasks)}
310 ${renderStat('Swap', n.swapMB ? n.swapMB + ' MB' : 'none')}
311 ${renderStat('Apps seeded', n.seededApps)}
312 ${renderStat('Connections', n.connections)}
313 ${renderStat('Cores seeded', n.features?.seeding?.coresSeeded)}
314 ${renderStat('Bytes served', n.bytesServed ? fmtBytes(n.bytesServed) : '—')}
315 ${renderStat('Blocks served', n.blocksServed)}
316 </div>
317
318 ${n.services && n.services.length > 0 ? `
319 <div class="detail-section-title" style="margin-top:16px">Service Layer (${n.services.length})</div>
320 <div class="svc-list">
321 ${n.services.map(s => `<div class="svc-item"><span class="svc-name">${esc(s.name)}</span><span class="svc-ver">v${esc(s.version)}</span><span class="svc-caps">${(s.capabilities||[]).length} caps</span></div>`).join('')}
322 </div>` : ''}
323
324 ${n.transports ? `
325 <div class="detail-section-title" style="margin-top:16px">Transports</div>
326 <div class="stats-grid">
327 ${renderTransport('DHT WS', n.transports.dhtRelayWs)}
328 ${renderTransport('Tor', n.transports.tor)}
329 ${renderTransport('Holesail', n.transports.holesail)}
330 </div>` : ''}
331 </div>
332 <div class="detail-col">
333 <div class="detail-section-title">Controls</div>
334 <div class="control-list">
335 <button class="ctrl-btn" onclick="nodeAction('${esc(n.name)}','restart')" ${offline?'disabled':''}>Restart Relay</button>
336 <button class="ctrl-btn" onclick="nodeAction('${esc(n.name)}','update')" ${offline?'disabled':''}>Update (git pull + restart)</button>
337 <button class="ctrl-btn" onclick="nodeAction('${esc(n.name)}','doctor')" ${offline?'disabled':''}>Doctor (swap + locks + cleanup)</button>
338 </div>
339
340 <div class="detail-section-title" style="margin-top:16px">Environment</div>
341 <div class="env-list">
342 ${Object.entries(n.env||{}).map(([k,v]) => `<div class="env-item"><span class="env-key">${esc(k)}</span><span class="env-val">${esc(k.includes('KEY')?'[redacted]':v)}</span></div>`).join('') || '<div class="env-empty">No env vars</div>'}
343 </div>
344 </div>
345 </div>
346 </div></td></tr>`
347}
348
349function renderStat (label, value) {
350 return `<div class="stat-item"><span class="stat-label">${esc(label)}</span><span class="stat-value">${esc(value != null ? value : '—')}</span></div>`
351}
352
353function renderTransport (name, t) {
354 if (!t) return `<div class="stat-item"><span class="stat-label">${name}</span><span class="stat-value">disabled</span></div>`
355 return `<div class="stat-item"><span class="stat-label">${name}</span><span class="stat-value">${t.running?'running':'stopped'} ${t.activeConnections!=null?`(${t.activeConnections} active)`:''}</span></div>`
356}
357
358function renderActionLog (actions) {
359 if (!actions || actions.length === 0) return ''
360 return `<div class="section"><div class="section-title">Action Log <span class="count">last ${Math.min(actions.length,10)}</span></div>
361 <div class="action-log">${actions.slice(0,10).map(a => `<div class="action-item"><span class="action-time">${new Date(a.timestamp).toLocaleTimeString()}</span><span class="action-node">${esc(a.node)}</span><span class="action-type">${esc(a.action)}</span><span class="action-detail">${esc(a.detail||'')}</span></div>`).join('')}</div></div>`
362}
363
364function renderDiskBar (n) {
365 if (n.diskPct == null) return '—'
366 const pct = Math.min(n.diskPct, 100)
367 const cls = n.diskPct > 85 ? 'crit' : n.diskPct > 70 ? 'warn' : 'norm'
368 const totalGB = n.diskGB || 0
369 const usedGB = Math.round(totalGB * n.diskPct / 100)
370 const freeGB = totalGB - usedGB
371 const title = `${usedGB} GB used / ${freeGB} GB free / ${totalGB} GB total`
372 return `<div class="bar-cell" title="${title}"><span class="bar"><span class="fill ${cls}" style="width:${pct}%"></span></span><span class="bar-pct">${n.diskPct}%</span></div>`
373}
374
375function renderRamBar (n) {
376 if (n.ramUsedPct == null) return '—'
377 const pct = Math.min(n.ramUsedPct, 100)
378 const cls = n.ramUsedPct > 85 ? 'crit' : n.ramUsedPct > 70 ? 'warn' : 'norm'
379 const totalGB = n.ramGB || 0
380 const usedGB = (totalGB * n.ramUsedPct / 100).toFixed(1)
381 const title = `${usedGB} GB used / ${totalGB} GB total`
382 return `<div class="bar-cell" title="${title}"><span class="bar ram"><span class="fill ${cls}" style="width:${pct}%"></span></span><span class="bar-pct">${n.ramUsedPct}%</span></div>`
383}
384
385function toggleDetail (name) { expandedNode = expandedNode === name ? null : name; render() }
386
387async function nodeAction (name, action) {
388 if (!confirm(`${action} on ${name}?`)) return
389 try {
390 showToast({ node: name, action, detail: 'executing...' })
391 const res = await fetch(`/api/node/${name}/${action}`, { method: 'POST' })
392 const data = await res.json()
393 if (data.error) alert(data.error)
394 else showToast({ node: name, action, detail: 'completed' })
395 await fetchState()
396 } catch (e) { alert(e.message) }
397}
398
399async function toggleFeature (name, feature, enabled) {
400 if (!confirm(`${enabled?'Enable':'Disable'} ${feature} on ${name}?`)) return
401 try {
402 showToast({ node: name, action: `${enabled?'enable':'disable'} ${feature}`, detail: 'applying...' })
403 const res = await fetch(`/api/node/${name}/toggle`, { method: 'POST', headers: {'content-type':'application/json'}, body: JSON.stringify({ feature, enabled }) })
404 const data = await res.json()
405 if (data.error) alert(data.error)
406 await fetchState()
407 } catch (e) { alert(e.message) }
408}
409
410async function reprobe () { try { await fetch('/api/reprobe', { method: 'POST' }) } catch {} }
411
412// --- Add Relay ---
413function showAddRelay () {
414 const overlay = document.createElement('div')
415 overlay.className = 'modal-overlay active'
416 overlay.id = 'modal-add-relay'
417 overlay.innerHTML = `
418 <div class="modal">
419 <h2>Add Relay to Fleet</h2>
420 <div class="modal-field"><label>Name</label><input type="text" id="relay-name" placeholder="e.g. tokyo-1"></div>
421 <div class="modal-field"><label>IP Address</label><input type="text" id="relay-ip" placeholder="1.2.3.4"></div>
422 <div class="modal-field">
423 <label>SSH Key</label>
424 <select id="relay-key">
425 <option value="~/.ssh/cloudzy_hiverelay">cloudzy_hiverelay</option>
426 <option value="~/.ssh/id_ed25519">id_ed25519 (utah/dubai)</option>
427 <option value="~/.ssh/hiverelay_fleet">hiverelay_fleet</option>
428 </select>
429 </div>
430 <div class="modal-field-row">
431 <div class="modal-field"><label>Region</label>
432 <select id="relay-region">
433 <option value="NA">North America</option>
434 <option value="EU">Europe</option>
435 <option value="APAC">Asia-Pacific</option>
436 <option value="ME">Middle East</option>
437 </select>
438 </div>
439 <div class="modal-field"><label>Channel</label>
440 <select id="relay-channel">
441 <option value="stable">stable</option>
442 <option value="canary">canary</option>
443 </select>
444 </div>
445 </div>
446 <div class="modal-field-row">
447 <div class="modal-field"><label>vCPU</label><input type="number" id="relay-vcpu" value="1" min="1" max="16"></div>
448 <div class="modal-field"><label>RAM (GB)</label><input type="number" id="relay-ram" value="2" step="0.5"></div>
449 <div class="modal-field"><label>Disk (GB)</label><input type="number" id="relay-disk" value="60"></div>
450 </div>
451 <div class="modal-actions">
452 <button class="btn ghost" onclick="closeModal()">Cancel</button>
453 <button class="btn primary" onclick="submitAddRelay()">Add & Probe</button>
454 </div>
455 </div>`
456 document.body.appendChild(overlay)
457}
458
459function closeModal () {
460 const m = document.getElementById('modal-add-relay')
461 if (m) m.remove()
462}
463
464async function submitAddRelay () {
465 const relay = {
466 name: document.getElementById('relay-name').value.trim(),
467 publicIp: document.getElementById('relay-ip').value.trim(),
468 sshKey: document.getElementById('relay-key').value,
469 region: document.getElementById('relay-region').value,
470 channel: document.getElementById('relay-channel').value,
471 vcpu: parseInt(document.getElementById('relay-vcpu').value) || 1,
472 ramGB: parseFloat(document.getElementById('relay-ram').value) || 2,
473 diskGB: parseInt(document.getElementById('relay-disk').value) || 60
474 }
475 if (!relay.name || !relay.publicIp) { alert('Name and IP required'); return }
476 closeModal()
477 showToast({ node: relay.name, action: 'add relay', detail: 'adding...' })
478 try {
479 const res = await fetch('/api/relay', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(relay) })
480 const data = await res.json()
481 if (data.error) { alert(data.error); return }
482 showToast({ node: relay.name, action: 'add relay', detail: 'added' })
483 await fetchState()
484 } catch (e) { alert(e.message) }
485}
486
487function showToast (data) {
488 const el = document.createElement('div')
489 el.className = 'toast'
490 el.innerHTML = `<strong>${esc(data.node)}</strong> · ${esc(data.action)} · ${esc(data.detail||'')}`
491 document.getElementById('toast-container')?.appendChild(el)
492 setTimeout(() => el.remove(), 5000)
493}
494
495fetchState().then(() => connectSSE())
dashboard-admin-operator/public/fleet.css +171 -0
……@@ -0,0 +1,171 @@
1:root {
2 --bg:#080a0d; --bg-panel:#0e1116; --bg-elevated:#141821; --bg-row:#0c0f14;
3 --border:#1e242e; --border-light:#2a313d;
4 --text:#d4dae0; --text-primary:#f0f3f6; --muted:#6b7684; --dim:#4a5260;
5 --accent:#4493f8; --success:#2ecc71; --warning:#f0a020; --critical:#e74c3c;
6 --mono:"JetBrains Mono","SF Mono",ui-monospace,Menlo,monospace;
7 --sans:-apple-system,"SF Pro Display","Segoe UI",system-ui,sans-serif;
8}
9*{box-sizing:border-box;margin:0;padding:0} html,body{height:100%}
10body{font-family:var(--sans);background:var(--bg);color:var(--text);font-size:13px}
11.loading{display:flex;align-items:center;justify-content:center;height:100vh;color:var(--dim);font-size:15px}
12
13.header{display:flex;align-items:center;justify-content:space-between;padding:14px 24px;border-bottom:1px solid var(--border);background:var(--bg-panel);position:sticky;top:0;z-index:100}
14.header-left{display:flex;align-items:center;gap:12px}
15.logo-mark{width:30px;height:30px;border-radius:6px;background:linear-gradient(135deg,var(--accent),#6d5cf6);display:flex;align-items:center;justify-content:center;font-weight:800;font-size:14px;color:#fff}
16.header h1{font-size:15px;font-weight:600;color:var(--text-primary);letter-spacing:-0.02em}
17.header-meta{font-size:11px;color:var(--muted);font-family:var(--mono);margin-top:1px}
18.header-right{display:flex;gap:12px;align-items:center}
19.live-indicator{display:flex;align-items:center;gap:6px;font-size:11px;color:var(--muted)}
20.live-pulse{width:7px;height:7px;border-radius:50%;background:var(--success);animation:pulse 2s infinite}
21@keyframes pulse{0%,100%{opacity:1;box-shadow:0 0 4px var(--success)}50%{opacity:.4}}
22.btn{background:var(--bg-elevated);border:1px solid var(--border-light);color:var(--text);border-radius:6px;padding:5px 12px;font-size:11px;cursor:pointer}
23.btn:hover{background:var(--border-light)}
24
25/* Fleet control plane */
26.control-panel{background:var(--bg-panel);border:1px solid var(--border);border-radius:10px;padding:14px 16px;margin-bottom:18px}
27.control-panel-head{display:flex;align-items:center;justify-content:space-between;gap:12px;margin-bottom:10px}
28.control-panel-head strong{color:var(--text-primary);font-size:13px}
29.control-panel-head .dim,.dim{color:var(--muted);font-size:11px}
30.control-panel-actions{display:flex;align-items:center;gap:10px}
31.toggle-auto{display:flex;align-items:center;gap:8px;font-size:11px;color:var(--text);cursor:pointer;user-select:none}
32.toggle-auto input{accent-color:var(--success)}
33.control-kpis{display:grid;grid-template-columns:repeat(5,1fr);gap:8px;margin-bottom:10px}
34.ckpi{background:var(--bg-elevated);border:1px solid var(--border);border-radius:8px;padding:8px 10px}
35.ckpi-v{font-size:18px;font-weight:700;letter-spacing:-.02em}
36.ckpi-l{font-size:9px;text-transform:uppercase;letter-spacing:.08em;color:var(--muted);margin-top:2px}
37.ckpi.ok .ckpi-v{color:var(--success)} .ckpi.warn .ckpi-v{color:var(--warning)} .ckpi.crit .ckpi-v{color:var(--critical)}
38.control-ranges{display:flex;flex-wrap:wrap;gap:6px;margin-bottom:8px}
39.range-chip{font-size:10px;font-family:var(--mono);color:var(--muted);background:var(--bg-elevated);border:1px solid var(--border);border-radius:999px;padding:3px 8px}
40.violations{margin-top:6px}
41.violation-row{display:flex;flex-wrap:wrap;align-items:center;gap:6px;padding:6px 0;border-top:1px solid var(--border);font-size:11px}
42.vchip{font-family:var(--mono);font-size:10px;padding:2px 6px;border-radius:4px;border:1px solid var(--border)}
43.vchip.critical{color:var(--critical);border-color:rgba(231,76,60,.35);background:rgba(231,76,60,.08)}
44.vchip.warning{color:var(--warning);border-color:rgba(240,160,32,.35);background:rgba(240,160,32,.08)}
45.auto-act{font-family:var(--mono);font-size:10px;color:var(--accent)}
46.in-range-ok{font-size:11px;color:var(--success);padding:6px 0}
47.remed-log{margin-top:8px;border-top:1px solid var(--border);padding-top:6px}
48.remed-item{font-family:var(--mono);font-size:10px;color:var(--muted);padding:2px 0}
49.remed-item.ok{color:var(--success)} .remed-item.fail{color:var(--critical)}
50.badge.small,.badge.medium,.badge.large{background:rgba(68,147,248,.1);color:var(--accent);border:1px solid rgba(68,147,248,.2)}
51
52.container{padding:20px 24px;max-width:1700px}
53.kpis{display:grid;grid-template-columns:repeat(7,1fr);gap:1px;background:var(--border);margin-bottom:20px;border-radius:8px;overflow:hidden}
54.kpi{background:var(--bg-panel);padding:12px 16px}
55.kpi-value{font-size:22px;font-weight:700;letter-spacing:-0.03em}
56.kpi-label{font-size:9px;text-transform:uppercase;letter-spacing:.08em;color:var(--muted);margin-top:2px}
57.kpi.success .kpi-value{color:var(--success)} .kpi.critical .kpi-value{color:var(--critical)} .kpi.warning .kpi-value{color:var(--warning)} .kpi.neutral .kpi-value{color:var(--accent)}
58
59.regions{display:flex;gap:8px;margin-bottom:20px;flex-wrap:wrap}
60.region-card{flex:1;min-width:120px;background:var(--bg-panel);border:1px solid var(--border);border-radius:8px;padding:10px 14px}
61.region-name{font-size:10px;text-transform:uppercase;letter-spacing:.06em;color:var(--muted)}
62.region-stats{display:flex;gap:10px;font-size:12px}
63.region-stats .up{color:var(--success)} .region-stats .down{color:var(--critical)} .region-stats .deg{color:var(--warning)} .region-stats .total{color:var(--muted);margin-left:auto}
64
65.section{margin-bottom:24px}
66.section-title{font-size:11px;text-transform:uppercase;letter-spacing:.1em;color:var(--muted);margin-bottom:10px;display:flex;align-items:center;gap:8px}
67.section-title .count{background:var(--bg-elevated);padding:1px 7px;border-radius:10px;font-size:10px}
68
69.tbl{width:100%;border-collapse:collapse;background:var(--bg-panel);border-radius:8px;overflow:hidden}
70.tbl th{text-align:left;padding:9px 12px;font-size:10px;font-weight:600;text-transform:uppercase;letter-spacing:.06em;color:var(--muted);border-bottom:1px solid var(--border);background:var(--bg-elevated);white-space:nowrap}
71.tbl td{padding:9px 12px;border-bottom:1px solid var(--border);font-size:12px;cursor:pointer}
72.tbl tr:last-child td{border-bottom:none}
73.tbl tr.offline td{opacity:.4}
74.tbl tr.degraded td{background:rgba(240,160,32,.03)}
75.tbl tr:hover td{background:var(--bg-row)}
76.tbl tr.expanded td{background:var(--bg-row)}
77.tbl tr.detail-row td{padding:0;cursor:default}
78.chevron{display:inline-block;font-size:9px;color:var(--dim);transition:transform .2s;cursor:pointer;width:12px}
79.chevron.open{transform:rotate(90deg)}
80
81.indicator{display:inline-flex;align-items:center;gap:6px}
82.dot{width:7px;height:7px;border-radius:50%;flex-shrink:0}
83.dot.healthy{background:var(--success);box-shadow:0 0 6px rgba(46,204,113,.4)}
84.dot.offline{background:var(--critical)} .dot.degraded{background:var(--warning);box-shadow:0 0 4px rgba(240,160,32,.3)}
85
86.badge{display:inline-block;padding:2px 7px;border-radius:4px;font-size:10px;font-weight:600}
87.badge.stable{background:rgba(46,204,113,.1);color:var(--success);border:1px solid rgba(46,204,113,.15)}
88.badge.canary{background:rgba(240,160,32,.1);color:var(--warning);border:1px solid rgba(240,160,32,.15)}
89
90.mono{font-family:var(--mono);font-size:11px;color:var(--muted)}
91.mono-val{font-family:var(--mono);font-size:11px;color:var(--text)}
92.ok-text{color:var(--success)} .err-text{color:var(--critical)}
93.version-ok{color:var(--success)} .version-bad{color:var(--critical)}
94.svc-count{background:rgba(68,147,248,.12);color:var(--accent);padding:1px 6px;border-radius:4px;font-size:11px;font-weight:600}
95
96.bar-cell{display:flex;align-items:center;gap:6px}
97.bar{width:50px;height:5px;background:rgba(255,255,255,0.06);border-radius:3px;overflow:hidden;display:inline-block;vertical-align:middle}
98.bar.ram{width:40px;height:4px}
99.fill{height:100%;border-radius:3px;transition:width .5s,background .5s;display:block}
100.fill.norm{background:var(--success)} .fill.warn{background:var(--warning)} .fill.crit{background:var(--critical)}
101.bar-pct{font-family:var(--mono);font-size:10px;color:var(--muted);min-width:28px}
102
103/* Detail panel */
104.detail-panel{padding:20px 24px;background:var(--bg-row);border-top:1px solid var(--border)}
105.detail-grid{display:grid;grid-template-columns:1fr 1fr 1fr;gap:24px}
106.detail-section-title{font-size:11px;text-transform:uppercase;letter-spacing:.08em;color:var(--accent);margin-bottom:10px}
107
108.feature-grid{display:flex;flex-direction:column;gap:4px}
109.feature-item{display:flex;align-items:center;gap:8px;padding:6px 8px;border-radius:6px;background:var(--bg-panel);border:1px solid var(--border)}
110.feature-item.on{border-color:rgba(46,204,113,.2)} .feature-item.off{opacity:.6}
111.feat-status{font-size:9px;font-weight:700;padding:2px 5px;border-radius:3px}
112.feat-status.on{background:rgba(46,204,113,.15);color:var(--success)} .feat-status.off{background:rgba(90,107,126,.15);color:var(--dim)}
113.feat-label{font-size:12px;flex:1}
114.feat-extra{font-size:10px;color:var(--muted);font-family:var(--mono)}
115.feat-toggle{background:var(--bg-elevated);border:1px solid var(--border-light);color:var(--text);border-radius:4px;padding:3px 8px;font-size:10px;cursor:pointer}
116.feat-toggle:hover{background:var(--border-light)} .feat-toggle.on{color:var(--success);border-color:rgba(46,204,113,.3)} .feat-toggle.off{color:var(--critical);border-color:rgba(231,76,60,.3)}
117
118.stats-grid{display:grid;grid-template-columns:1fr 1fr;gap:4px}
119.stat-item{display:flex;justify-content:space-between;padding:4px 8px;border-radius:4px;background:var(--bg-panel)}
120.stat-label{font-size:10px;color:var(--muted)} .stat-value{font-size:11px;color:var(--text);font-family:var(--mono)}
121
122.svc-list{display:flex;flex-direction:column;gap:4px}
123.svc-item{display:flex;align-items:center;gap:8px;padding:5px 8px;background:var(--bg-panel);border-radius:4px}
124.svc-name{font-weight:600;font-size:12px;color:var(--accent)} .svc-ver{font-size:10px;color:var(--muted);font-family:var(--mono)} .svc-caps{font-size:10px;color:var(--dim)}
125
126.control-list{display:flex;flex-direction:column;gap:6px}
127.ctrl-btn{background:var(--bg-panel);border:1px solid var(--border-light);color:var(--text);border-radius:6px;padding:8px 14px;font-size:12px;cursor:pointer;text-align:left;transition:all .15s}
128.ctrl-btn:hover:not(:disabled){background:var(--bg-elevated);border-color:var(--accent)}
129.ctrl-btn:disabled{opacity:.4;cursor:not-allowed}
130
131.env-list{display:flex;flex-direction:column;gap:3px;max-height:200px;overflow-y:auto}
132.env-item{display:flex;justify-content:space-between;padding:3px 8px;background:var(--bg-panel);border-radius:4px}
133.env-key{font-size:10px;color:var(--muted);font-family:var(--mono)} .env-val{font-size:10px;color:var(--text);font-family:var(--mono)}
134.env-empty{color:var(--dim);font-size:11px;padding:8px}
135
136/* Action log */
137.action-log{background:var(--bg-panel);border:1px solid var(--border);border-radius:8px;padding:8px 12px;max-height:200px;overflow-y:auto}
138.action-item{display:flex;gap:10px;padding:4px 0;border-bottom:1px solid #1a1e26;font-size:11px}
139.action-item:last-child{border-bottom:none}
140.action-time{color:var(--dim);font-family:var(--mono);min-width:70px}
141.action-node{color:var(--accent);font-weight:600;min-width:80px}
142.action-type{color:var(--text);min-width:120px}
143.action-detail{color:var(--muted);flex:1}
144
145/* Toast */
146#toast-container{position:fixed;bottom:20px;right:20px;z-index:200;display:flex;flex-direction:column;gap:8px}
147.toast{background:var(--bg-elevated);border:1px solid var(--border-light);border-radius:8px;padding:10px 16px;font-size:12px;color:var(--text);animation:slideIn .3s ease;box-shadow:0 4px 12px rgba(0,0,0,.3)}
148@keyframes slideIn{from{transform:translateX(100%);opacity:0}to{transform:none;opacity:1}}
149
150.footer{padding:14px 24px;border-top:1px solid var(--border);margin-top:24px}
151.footer-meta{font-size:10px;color:var(--dim);font-family:var(--mono)}
152
153@media(max-width:1200px){.kpis{grid-template-columns:repeat(4,1fr)}.detail-grid{grid-template-columns:1fr}.regions{flex-wrap:wrap}}
154
155/* Buttons */
156.btn.primary{background:linear-gradient(180deg,var(--accent),#3672d4);color:#fff;border-color:transparent;font-weight:600}
157.btn.ghost{background:transparent}
158.add-btn{font-size:10px;padding:3px 10px;margin-left:auto}
159
160/* Modal */
161.modal-overlay{position:fixed;inset:0;background:rgba(0,0,0,.7);display:none;align-items:center;justify-content:center;z-index:200}
162.modal-overlay.active{display:flex}
163.modal{background:var(--bg-panel);border:1px solid var(--border-light);border-radius:12px;padding:24px;min-width:440px;max-width:500px;box-shadow:0 8px 32px rgba(0,0,0,.4)}
164.modal h2{font-size:16px;margin-bottom:16px;color:var(--text-primary)}
165.modal-field{margin-bottom:10px}
166.modal-field label{display:block;font-size:11px;color:var(--muted);margin-bottom:3px}
167.modal-field input,.modal-field select{width:100%;background:var(--bg-elevated);border:1px solid var(--border);border-radius:6px;color:var(--text);padding:7px 10px;font:inherit;font-size:13px}
168.modal-field input:focus,.modal-field select:focus{outline:none;border-color:var(--accent)}
169.modal-field-row{display:flex;gap:10px}
170.modal-field-row .modal-field{flex:1}
171.modal-actions{display:flex;gap:8px;justify-content:flex-end;margin-top:16px}
dashboard-admin-operator/public/fleet.html +18 -0
……@@ -0,0 +1,18 @@
1<!DOCTYPE html>
2<html lang="en">
3<head>
4<meta charset="UTF-8">
5<meta name="viewport" content="width=device-width, initial-scale=1.0">
6<meta name="description" content="HiveRelay Dashboard Admin Operator — fleet health, features, restart/update/doctor">
7<title>Dashboard Admin Operator — HiveRelay</title>
8<link rel="stylesheet" href="styles.css">
9</head>
10<body>
11
12<div id="app">
13 <div class="loading">Connecting to Admin Operator…</div>
14</div>
15
16<script src="app.js"></script>
17</body>
18</html>
dashboard-admin-operator/public/fleet.js +495 -0
……@@ -0,0 +1,495 @@
1/* HiveRelay · Dashboard Admin Operator — live fleet control plane */
2
3let state = null
4let evtSource = null
5let expandedNode = null
6
7const REGION_NAMES = { NA: 'North America', EU: 'Europe', APAC: 'Asia-Pacific', ME: 'Middle East' }
8const FEATURE_LABELS = {
9 coreRelay: 'Core Relay', seeding: 'Seeding', services: 'Service Layer',
10 dhtRelayWs: 'DHT Relay (WS)', tor: 'Tor Transport', holesail: 'Holesail',
11 outboxlog: 'OutboxLog', forwardRelay: 'Forward Relay', signedDirectory: 'Signed Directory',
12 vrf: 'VRF (RFC 9381)', poker: 'Poker Substrate', zk: 'ZK Proofs',
13 arbitration: 'Arbitration', payment: 'Payments', eviction: 'Auto-Eviction',
14 autoHeal: 'Auto-Heal', distributedDrive: 'Distributed Drive'
15}
16const TOGGLEABLE = ['dhtRelayWs', 'outboxlog', 'forwardRelay', 'signedDirectory', 'services', 'eviction']
17
18async function fetchState () {
19 try { state = await (await fetch('/api/state')).json(); render() }
20 catch { document.getElementById('app').innerHTML = '<div class="loading">Connection lost...</div>' }
21}
22
23function connectSSE () {
24 if (evtSource) evtSource.close()
25 evtSource = new EventSource('/api/events')
26 evtSource.onmessage = (e) => {
27 try {
28 const msg = JSON.parse(e.data)
29 if (msg.type === 'fleet') {
30 if (!state) { fetchState(); return }
31 state.nodes = msg.data; state.lastProbe = msg.timestamp
32 state.summary = computeSummary(msg.data)
33 render()
34 } else if (msg.type === 'action') {
35 if (state) { state.actions = [msg.data, ...(state.actions || [])] }
36 showToast(msg.data)
37 } else if (msg.type === 'control') {
38 if (state) {
39 state.control = {
40 ...(state.control || {}),
41 lastCycle: msg.data,
42 autoControl: msg.data.autoControl,
43 inRange: msg.data.summary?.inRange,
44 critical: msg.data.summary?.critical,
45 warnings: msg.data.summary?.warnings
46 }
47 render()
48 }
49 }
50 } catch {}
51 }
52 evtSource.onerror = () => { evtSource.close(); setTimeout(() => { connectSSE(); fetchState() }, 5000) }
53}
54
55function computeSummary (nodes) {
56 const regions = {}
57 for (const n of nodes) {
58 const r = n.region || 'Unknown'
59 if (!regions[r]) regions[r] = { total: 0, healthy: 0, degraded: 0, offline: 0 }
60 regions[r].total++
61 if (n.status === 'healthy') regions[r].healthy++
62 else if (n.status === 'offline') regions[r].offline++
63 else regions[r].degraded++
64 }
65 return {
66 total: nodes.length, healthy: nodes.filter(n => n.status === 'healthy').length,
67 degraded: nodes.filter(n => n.status === 'degraded').length,
68 offline: nodes.filter(n => n.status === 'offline').length,
69 running: nodes.filter(n => n.running === true).length,
70 diskWarnings: nodes.filter(n => n.diskPct && n.diskPct > 75).length,
71 totalRamGB: Math.round(nodes.reduce((s, n) => s + (n.ramGB || 0), 0)),
72 totalDiskGB: nodes.reduce((s, n) => s + (n.diskGB || 0), 0),
73 totalSeeded: nodes.reduce((s, n) => s + (n.seededApps || 0), 0),
74 totalConns: nodes.reduce((s, n) => s + (n.connections || 0), 0),
75 regions
76 }
77}
78
79function fmtBytes (b) {
80 if (!b || b === 0) return '0 B'
81 const u = ['B','KB','MB','GB','TB','PB']
82 const i = Math.floor(Math.log(b) / Math.log(1024))
83 return (b / Math.pow(1024, i)).toFixed(1) + ' ' + u[i]
84}
85
86function esc (s) { if (s == null) return ''; const d = document.createElement('div'); d.textContent = String(s); return d.innerHTML }
87
88function render () {
89 if (!state || !state.nodes) return
90 const s = state.summary
91 const probeAgo = state.lastProbe ? Math.round((Date.now() - new Date(state.lastProbe).getTime()) / 1000) : '?'
92
93 document.getElementById('app').innerHTML = `
94 <div class="header">
95 <div class="header-left">
96 <div class="logo-mark">H</div>
97 <div><h1>Dashboard Admin Operator</h1>
98 <div class="header-meta">Fleet control · ${state.nodes.length} nodes · ${s.totalSeeded} apps seeded · ${s.totalConns} connections · probed ${probeAgo}s ago</div></div>
99 </div>
100 <div class="header-right">
101 <div class="live-indicator"><span class="live-pulse"></span><span>LIVE</span></div>
102 <button class="btn" onclick="runControlNow()" title="Evaluate ranges + remediate now">Control cycle</button>
103 <button class="btn" onclick="stabilizeAll()" title="Install watchdog + swap on all reachable">Stabilize all</button>
104 <button class="btn" onclick="reprobe()">Re-probe</button>
105 </div>
106 </div>
107 <div class="container">
108 ${renderKPIs(s)}
109 ${renderControlPanel(state.control)}
110 ${renderRegions(s.regions)}
111 ${renderNodeTable(state.nodes)}
112 ${renderActionLog(state.actions)}
113 </div>
114 <div id="toast-container"></div>
115 <div class="footer"><div class="footer-meta">Admin Operator · full fleet control · probe ${state.interval||60}s · auto-control ${state.control?.autoControl ? 'ON' : 'OFF'} · ${new Date().toISOString()}</div></div>
116 `
117}
118
119function renderControlPanel (ctrl) {
120 if (!ctrl) {
121 return `<div class="control-panel"><div class="control-panel-head"><strong>Fleet control</strong><span class="dim">waiting for first probe…</span></div></div>`
122 }
123 const auto = !!ctrl.autoControl
124 const cycle = ctrl.lastCycle
125 const sum = cycle?.summary || {}
126 const nodes = cycle?.nodes || []
127 const violators = nodes.filter(n => (n.violations || []).length > 0)
128 const remediations = cycle?.remediations || []
129 return `
130 <div class="control-panel">
131 <div class="control-panel-head">
132 <div>
133 <strong>Fleet control plane</strong>
134 <span class="dim"> · healthy ranges by tier · auto-remediate hangs/OOM/disk</span>
135 </div>
136 <div class="control-panel-actions">
137 <label class="toggle-auto" title="When on, each probe auto-restarts hung nodes and enforces baseline">
138 <input type="checkbox" ${auto ? 'checked' : ''} onchange="setAutoControl(this.checked)">
139 <span>Auto-control ${auto ? 'ON' : 'OFF'}</span>
140 </label>
141 </div>
142 </div>
143 <div class="control-kpis">
144 <div class="ckpi ok"><div class="ckpi-v">${sum.inRange ?? '—'}</div><div class="ckpi-l">In range</div></div>
145 <div class="ckpi warn"><div class="ckpi-v">${sum.warnings ?? 0}</div><div class="ckpi-l">Warnings</div></div>
146 <div class="ckpi crit"><div class="ckpi-v">${sum.critical ?? 0}</div><div class="ckpi-l">Critical</div></div>
147 <div class="ckpi"><div class="ckpi-v">${sum.offline ?? 0}</div><div class="ckpi-l">Offline</div></div>
148 <div class="ckpi"><div class="ckpi-v">${remediations.length}</div><div class="ckpi-l">Last fixes</div></div>
149 </div>
150 <div class="control-ranges">
151 <span class="range-chip">small ≤1GB: mem≤300MB light plugins</span>
152 <span class="range-chip">medium 2–4GB: mem≤1.6GB</span>
153 <span class="range-chip">large ≥4GB: mem≤4.8GB</span>
154 <span class="range-chip">swap ≥256MB · watchdog required · disk warn 75%</span>
155 </div>
156 ${violators.length ? `
157 <div class="violations">
158 <div class="section-title" style="margin:8px 0 6px">Out of range <span class="count">${violators.length}</span></div>
159 ${violators.map(v => `
160 <div class="violation-row">
161 <strong>${esc(v.name)}</strong>
162 <span class="badge ${esc(v.tier||'')}">${esc(v.tier||'?')}</span>
163 ${(v.violations||[]).map(x => `<span class="vchip ${x.severity}">${esc(x.code)}: ${esc(x.msg)}</span>`).join('')}
164 ${(v.actions||[]).map(a => `<span class="auto-act">→ ${esc(a.type)}</span>`).join('')}
165 </div>`).join('')}
166 </div>` : `<div class="in-range-ok">All reachable nodes within healthy ranges (or offline only).</div>`}
167 ${remediations.length ? `
168 <div class="remed-log">
169 ${remediations.map(r => `<div class="remed-item ${r.ok?'ok':'fail'}">${esc(r.node)} · auto:${esc(r.type)} · ${esc(r.reason)} · ${r.ok?'ok':'fail'} ${esc(r.detail||'')}</div>`).join('')}
170 </div>` : ''}
171 </div>`
172}
173
174async function setAutoControl (enabled) {
175 try {
176 const res = await fetch('/api/control', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ autoControl: enabled }) })
177 const data = await res.json()
178 if (state) {
179 state.control = state.control || {}
180 state.control.autoControl = data.autoControl
181 }
182 showToast({ node: 'fleet', action: 'auto-control', detail: enabled ? 'ENABLED' : 'DISABLED' })
183 render()
184 } catch (e) { alert(e.message) }
185}
186
187async function runControlNow () {
188 showToast({ node: 'fleet', action: 'control cycle', detail: 'running…' })
189 try {
190 const res = await fetch('/api/control/run', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ force: true }) })
191 const data = await res.json()
192 if (data.error) alert(data.error)
193 else showToast({ node: 'fleet', action: 'control cycle', detail: `done · ${data.cycle?.remediations?.length||0} remediations` })
194 await fetchState()
195 } catch (e) { alert(e.message) }
196}
197
198async function stabilizeAll () {
199 if (!confirm('Install health watchdog + ensure swap on all reachable nodes?')) return
200 showToast({ node: 'fleet', action: 'stabilize-all', detail: 'running…' })
201 try {
202 const res = await fetch('/api/control/stabilize-all', { method: 'POST' })
203 const data = await res.json()
204 if (data.error) alert(data.error)
205 else {
206 const ok = (data.results || []).filter(r => r.ok).length
207 showToast({ node: 'fleet', action: 'stabilize-all', detail: `${ok} nodes ok` })
208 }
209 await fetchState()
210 } catch (e) { alert(e.message) }
211}
212
213function renderKPIs (s) {
214 return `<div class="kpis">
215 <div class="kpi neutral"><div class="kpi-value">${s.total}</div><div class="kpi-label">Nodes</div></div>
216 <div class="kpi success"><div class="kpi-value">${s.healthy}</div><div class="kpi-label">Healthy</div></div>
217 <div class="kpi warning"><div class="kpi-value">${s.degraded}</div><div class="kpi-label">Degraded</div></div>
218 <div class="kpi critical"><div class="kpi-value">${s.offline}</div><div class="kpi-label">Offline</div></div>
219 <div class="kpi success"><div class="kpi-value">${s.running}</div><div class="kpi-label">Relay Active</div></div>
220 <div class="kpi neutral"><div class="kpi-value">${fmtBytes(s.totalDiskGB * 1e9)}</div><div class="kpi-label">Total Disk</div></div>
221 <div class="kpi neutral"><div class="kpi-value">${s.totalRamGB} GB</div><div class="kpi-label">Total RAM</div></div>
222 </div>`
223}
224
225function renderRegions (regions) {
226 if (!regions) return ''
227 return `<div class="regions">${Object.entries(regions).map(([code, r]) => `
228 <div class="region-card"><div class="region-name">${REGION_NAMES[code]||code}</div>
229 <div class="region-stats"><span class="up">${r.healthy} up</span>${r.degraded?`<span class="deg">${r.degraded} deg</span>`:''}${r.offline?`<span class="down">${r.offline} down</span>`:''}<span class="total">${r.total}</span></div></div>`).join('')}</div>`
230}
231
232function renderNodeTable (nodes) {
233 const sorted = [...nodes].sort((a, b) => {
234 const order = { offline: 3, degraded: 2, healthy: 1 }
235 return (order[a.status]||3) - (order[b.status]||3) || a.name.localeCompare(b.name)
236 })
237 return `
238 <div class="section">
239 <div class="section-title">All Nodes <span class="count">${nodes.length}</span> <button class="btn add-btn" onclick="showAddRelay()">+ Add Relay</button></div>
240 <table class="tbl"><thead><tr>
241 <th></th><th>Node</th><th>Address</th><th>Version</th><th>Relay</th>
242 <th>Disk</th><th>Mem</th><th>RAM%</th><th>Load</th><th>Apps</th><th>Conns</th>
243 <th>WD</th><th>Services</th><th>Channel</th><th>Region</th><th>Spec</th>
244 </tr></thead><tbody>
245 ${sorted.map(n => renderRow(n)).join('')}
246 ${expandedNode ? renderDetailRow(sorted.find(n => n.name === expandedNode)) : ''}
247 </tbody></table></div>`
248}
249
250function renderRow (n) {
251 const isExpanded = expandedNode === n.name
252 const verOk = n.version && n.version !== 'none' && n.targetVersion && n.version === n.targetVersion
253 const svcCount = n.features?.services?.count || n.services?.length || 0
254 const wdOk = n.watchdog === 'enabled'
255 const mem = n.memCurrentMB
256 return `<tr class="${n.status} ${isExpanded ? 'expanded' : ''}" onclick="toggleDetail('${esc(n.name)}')">
257 <td><span class="chevron ${isExpanded ? 'open' : ''}">&#9654;</span></td>
258 <td><span class="indicator"><span class="dot ${n.status}"></span></span> <strong>${esc(n.name)}</strong></td>
259 <td class="mono">${esc(n.publicIp)}</td>
260 <td class="${verOk?'version-ok':'version-bad'} mono-val">${esc(n.version||'—')}</td>
261 <td>${n.running&&n.healthOk!==false?'<span class="ok-text">running</span>':(n.running?'<span class="err-text">hung?</span>':'<span class="err-text">stopped</span>')}</td>
262 <td>${renderDiskBar(n)}</td>
263 <td class="mono-val">${mem!=null?mem+'M':'—'}</td>
264 <td>${renderRamBar(n)}</td>
265 <td class="mono">${n.loadAvg!=null?Number(n.loadAvg).toFixed(2):'—'}</td>
266 <td class="mono-val">${n.seededApps||'—'}</td>
267 <td class="mono-val">${n.connections||'—'}</td>
268 <td>${wdOk?'<span class="ok-text">on</span>':'<span class="err-text">off</span>'}</td>
269 <td>${svcCount?`<span class="svc-count">${svcCount}</span>`:'—'}</td>
270 <td><span class="badge ${esc(n.channel||'stable')}">${esc(n.channel||'stable')}</span></td>
271 <td>${esc(n.region||'—')}</td>
272 <td class="mono">${n.vcpu||'?'}c/${n.ramGB||'?'}G/${n.diskGB||'?'}G</td>
273 </tr>`
274}
275
276function renderDetailRow (n) {
277 if (!n) return ''
278 const f = n.features || {}
279 const offline = n.status === 'offline'
280
281 return `<tr class="detail-row"><td colspan="16"><div class="detail-panel">
282 <div class="detail-grid">
283 <div class="detail-col">
284 <div class="detail-section-title">Features & Services</div>
285 <div class="feature-grid">
286 ${Object.entries(FEATURE_LABELS).map(([key, label]) => {
287 const feat = f[key] || {}
288 const isOn = feat.enabled === true
289 const canToggle = TOGGLEABLE.includes(key) && !offline
290 return `<div class="feature-item ${isOn?'on':'off'}">
291 <span class="feat-status ${isOn?'on':'off'}">${isOn?'ON':'OFF'}</span>
292 <span class="feat-label">${label}</span>
293 ${feat.running != null ? `<span class="feat-extra">${feat.running?'running':'stopped'}</span>` : ''}
294 ${feat.count != null ? `<span class="feat-extra">${feat.count} svc</span>` : ''}
295 ${feat.coresSeeded != null ? `<span class="feat-extra">${feat.coresSeeded} cores</span>` : ''}
296 ${canToggle ? `<button class="feat-toggle ${isOn?'off':'on'}" onclick="toggleFeature('${esc(n.name)}','${key}',${!isOn})">${isOn?'Disable':'Enable'}</button>` : ''}
297 </div>`
298 }).join('')}
299 </div>
300 </div>
301 <div class="detail-col">
302 <div class="detail-section-title">Runtime Stats</div>
303 <div class="stats-grid">
304 ${renderStat('Version', n.version)}
305 ${renderStat('Target', n.targetVersion)}
306 ${renderStat('Uptime since', n.uptime)}
307 ${renderStat('Restarts', n.restarts)}
308 ${renderStat('Memory', n.memCurrentMB != null ? n.memCurrentMB + ' MB' : '—')}
309 ${renderStat('Tasks', n.tasks)}
310 ${renderStat('Swap', n.swapMB ? n.swapMB + ' MB' : 'none')}
311 ${renderStat('Apps seeded', n.seededApps)}
312 ${renderStat('Connections', n.connections)}
313 ${renderStat('Cores seeded', n.features?.seeding?.coresSeeded)}
314 ${renderStat('Bytes served', n.bytesServed ? fmtBytes(n.bytesServed) : '—')}
315 ${renderStat('Blocks served', n.blocksServed)}
316 </div>
317
318 ${n.services && n.services.length > 0 ? `
319 <div class="detail-section-title" style="margin-top:16px">Service Layer (${n.services.length})</div>
320 <div class="svc-list">
321 ${n.services.map(s => `<div class="svc-item"><span class="svc-name">${esc(s.name)}</span><span class="svc-ver">v${esc(s.version)}</span><span class="svc-caps">${(s.capabilities||[]).length} caps</span></div>`).join('')}
322 </div>` : ''}
323
324 ${n.transports ? `
325 <div class="detail-section-title" style="margin-top:16px">Transports</div>
326 <div class="stats-grid">
327 ${renderTransport('DHT WS', n.transports.dhtRelayWs)}
328 ${renderTransport('Tor', n.transports.tor)}
329 ${renderTransport('Holesail', n.transports.holesail)}
330 </div>` : ''}
331 </div>
332 <div class="detail-col">
333 <div class="detail-section-title">Controls</div>
334 <div class="control-list">
335 <button class="ctrl-btn" onclick="nodeAction('${esc(n.name)}','restart')" ${offline?'disabled':''}>Restart Relay</button>
336 <button class="ctrl-btn" onclick="nodeAction('${esc(n.name)}','update')" ${offline?'disabled':''}>Update (git pull + restart)</button>
337 <button class="ctrl-btn" onclick="nodeAction('${esc(n.name)}','doctor')" ${offline?'disabled':''}>Doctor (swap + locks + cleanup)</button>
338 </div>
339
340 <div class="detail-section-title" style="margin-top:16px">Environment</div>
341 <div class="env-list">
342 ${Object.entries(n.env||{}).map(([k,v]) => `<div class="env-item"><span class="env-key">${esc(k)}</span><span class="env-val">${esc(k.includes('KEY')?'[redacted]':v)}</span></div>`).join('') || '<div class="env-empty">No env vars</div>'}
343 </div>
344 </div>
345 </div>
346 </div></td></tr>`
347}
348
349function renderStat (label, value) {
350 return `<div class="stat-item"><span class="stat-label">${esc(label)}</span><span class="stat-value">${esc(value != null ? value : '—')}</span></div>`
351}
352
353function renderTransport (name, t) {
354 if (!t) return `<div class="stat-item"><span class="stat-label">${name}</span><span class="stat-value">disabled</span></div>`
355 return `<div class="stat-item"><span class="stat-label">${name}</span><span class="stat-value">${t.running?'running':'stopped'} ${t.activeConnections!=null?`(${t.activeConnections} active)`:''}</span></div>`
356}
357
358function renderActionLog (actions) {
359 if (!actions || actions.length === 0) return ''
360 return `<div class="section"><div class="section-title">Action Log <span class="count">last ${Math.min(actions.length,10)}</span></div>
361 <div class="action-log">${actions.slice(0,10).map(a => `<div class="action-item"><span class="action-time">${new Date(a.timestamp).toLocaleTimeString()}</span><span class="action-node">${esc(a.node)}</span><span class="action-type">${esc(a.action)}</span><span class="action-detail">${esc(a.detail||'')}</span></div>`).join('')}</div></div>`
362}
363
364function renderDiskBar (n) {
365 if (n.diskPct == null) return '—'
366 const pct = Math.min(n.diskPct, 100)
367 const cls = n.diskPct > 85 ? 'crit' : n.diskPct > 70 ? 'warn' : 'norm'
368 const totalGB = n.diskGB || 0
369 const usedGB = Math.round(totalGB * n.diskPct / 100)
370 const freeGB = totalGB - usedGB
371 const title = `${usedGB} GB used / ${freeGB} GB free / ${totalGB} GB total`
372 return `<div class="bar-cell" title="${title}"><span class="bar"><span class="fill ${cls}" style="width:${pct}%"></span></span><span class="bar-pct">${n.diskPct}%</span></div>`
373}
374
375function renderRamBar (n) {
376 if (n.ramUsedPct == null) return '—'
377 const pct = Math.min(n.ramUsedPct, 100)
378 const cls = n.ramUsedPct > 85 ? 'crit' : n.ramUsedPct > 70 ? 'warn' : 'norm'
379 const totalGB = n.ramGB || 0
380 const usedGB = (totalGB * n.ramUsedPct / 100).toFixed(1)
381 const title = `${usedGB} GB used / ${totalGB} GB total`
382 return `<div class="bar-cell" title="${title}"><span class="bar ram"><span class="fill ${cls}" style="width:${pct}%"></span></span><span class="bar-pct">${n.ramUsedPct}%</span></div>`
383}
384
385function toggleDetail (name) { expandedNode = expandedNode === name ? null : name; render() }
386
387async function nodeAction (name, action) {
388 if (!confirm(`${action} on ${name}?`)) return
389 try {
390 showToast({ node: name, action, detail: 'executing...' })
391 const res = await fetch(`/api/node/${name}/${action}`, { method: 'POST' })
392 const data = await res.json()
393 if (data.error) alert(data.error)
394 else showToast({ node: name, action, detail: 'completed' })
395 await fetchState()
396 } catch (e) { alert(e.message) }
397}
398
399async function toggleFeature (name, feature, enabled) {
400 if (!confirm(`${enabled?'Enable':'Disable'} ${feature} on ${name}?`)) return
401 try {
402 showToast({ node: name, action: `${enabled?'enable':'disable'} ${feature}`, detail: 'applying...' })
403 const res = await fetch(`/api/node/${name}/toggle`, { method: 'POST', headers: {'content-type':'application/json'}, body: JSON.stringify({ feature, enabled }) })
404 const data = await res.json()
405 if (data.error) alert(data.error)
406 await fetchState()
407 } catch (e) { alert(e.message) }
408}
409
410async function reprobe () { try { await fetch('/api/reprobe', { method: 'POST' }) } catch {} }
411
412// --- Add Relay ---
413function showAddRelay () {
414 const overlay = document.createElement('div')
415 overlay.className = 'modal-overlay active'
416 overlay.id = 'modal-add-relay'
417 overlay.innerHTML = `
418 <div class="modal">
419 <h2>Add Relay to Fleet</h2>
420 <div class="modal-field"><label>Name</label><input type="text" id="relay-name" placeholder="e.g. tokyo-1"></div>
421 <div class="modal-field"><label>IP Address</label><input type="text" id="relay-ip" placeholder="1.2.3.4"></div>
422 <div class="modal-field">
423 <label>SSH Key</label>
424 <select id="relay-key">
425 <option value="~/.ssh/cloudzy_hiverelay">cloudzy_hiverelay</option>
426 <option value="~/.ssh/id_ed25519">id_ed25519 (utah/dubai)</option>
427 <option value="~/.ssh/hiverelay_fleet">hiverelay_fleet</option>
428 </select>
429 </div>
430 <div class="modal-field-row">
431 <div class="modal-field"><label>Region</label>
432 <select id="relay-region">
433 <option value="NA">North America</option>
434 <option value="EU">Europe</option>
435 <option value="APAC">Asia-Pacific</option>
436 <option value="ME">Middle East</option>
437 </select>
438 </div>
439 <div class="modal-field"><label>Channel</label>
440 <select id="relay-channel">
441 <option value="stable">stable</option>
442 <option value="canary">canary</option>
443 </select>
444 </div>
445 </div>
446 <div class="modal-field-row">
447 <div class="modal-field"><label>vCPU</label><input type="number" id="relay-vcpu" value="1" min="1" max="16"></div>
448 <div class="modal-field"><label>RAM (GB)</label><input type="number" id="relay-ram" value="2" step="0.5"></div>
449 <div class="modal-field"><label>Disk (GB)</label><input type="number" id="relay-disk" value="60"></div>
450 </div>
451 <div class="modal-actions">
452 <button class="btn ghost" onclick="closeModal()">Cancel</button>
453 <button class="btn primary" onclick="submitAddRelay()">Add & Probe</button>
454 </div>
455 </div>`
456 document.body.appendChild(overlay)
457}
458
459function closeModal () {
460 const m = document.getElementById('modal-add-relay')
461 if (m) m.remove()
462}
463
464async function submitAddRelay () {
465 const relay = {
466 name: document.getElementById('relay-name').value.trim(),
467 publicIp: document.getElementById('relay-ip').value.trim(),
468 sshKey: document.getElementById('relay-key').value,
469 region: document.getElementById('relay-region').value,
470 channel: document.getElementById('relay-channel').value,
471 vcpu: parseInt(document.getElementById('relay-vcpu').value) || 1,
472 ramGB: parseFloat(document.getElementById('relay-ram').value) || 2,
473 diskGB: parseInt(document.getElementById('relay-disk').value) || 60
474 }
475 if (!relay.name || !relay.publicIp) { alert('Name and IP required'); return }
476 closeModal()
477 showToast({ node: relay.name, action: 'add relay', detail: 'adding...' })
478 try {
479 const res = await fetch('/api/relay', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(relay) })
480 const data = await res.json()
481 if (data.error) { alert(data.error); return }
482 showToast({ node: relay.name, action: 'add relay', detail: 'added' })
483 await fetchState()
484 } catch (e) { alert(e.message) }
485}
486
487function showToast (data) {
488 const el = document.createElement('div')
489 el.className = 'toast'
490 el.innerHTML = `<strong>${esc(data.node)}</strong> · ${esc(data.action)} · ${esc(data.detail||'')}`
491 document.getElementById('toast-container')?.appendChild(el)
492 setTimeout(() => el.remove(), 5000)
493}
494
495fetchState().then(() => connectSSE())
dashboard-admin-operator/public/index.html +18 -0
……@@ -0,0 +1,18 @@
1<!DOCTYPE html>
2<html lang="en">
3<head>
4<meta charset="UTF-8">
5<meta name="viewport" content="width=device-width, initial-scale=1.0">
6<meta name="description" content="HiveRelay Dashboard Admin Operator — fleet health, features, restart/update/doctor">
7<title>Dashboard Admin Operator — HiveRelay</title>
8<link rel="stylesheet" href="styles.css">
9</head>
10<body>
11
12<div id="app">
13 <div class="loading">Connecting to Admin Operator…</div>
14</div>
15
16<script src="app.js"></script>
17</body>
18</html>
dashboard-admin-operator/public/styles.css +171 -0
……@@ -0,0 +1,171 @@
1:root {
2 --bg:#080a0d; --bg-panel:#0e1116; --bg-elevated:#141821; --bg-row:#0c0f14;
3 --border:#1e242e; --border-light:#2a313d;
4 --text:#d4dae0; --text-primary:#f0f3f6; --muted:#6b7684; --dim:#4a5260;
5 --accent:#4493f8; --success:#2ecc71; --warning:#f0a020; --critical:#e74c3c;
6 --mono:"JetBrains Mono","SF Mono",ui-monospace,Menlo,monospace;
7 --sans:-apple-system,"SF Pro Display","Segoe UI",system-ui,sans-serif;
8}
9*{box-sizing:border-box;margin:0;padding:0} html,body{height:100%}
10body{font-family:var(--sans);background:var(--bg);color:var(--text);font-size:13px}
11.loading{display:flex;align-items:center;justify-content:center;height:100vh;color:var(--dim);font-size:15px}
12
13.header{display:flex;align-items:center;justify-content:space-between;padding:14px 24px;border-bottom:1px solid var(--border);background:var(--bg-panel);position:sticky;top:0;z-index:100}
14.header-left{display:flex;align-items:center;gap:12px}
15.logo-mark{width:30px;height:30px;border-radius:6px;background:linear-gradient(135deg,var(--accent),#6d5cf6);display:flex;align-items:center;justify-content:center;font-weight:800;font-size:14px;color:#fff}
16.header h1{font-size:15px;font-weight:600;color:var(--text-primary);letter-spacing:-0.02em}
17.header-meta{font-size:11px;color:var(--muted);font-family:var(--mono);margin-top:1px}
18.header-right{display:flex;gap:12px;align-items:center}
19.live-indicator{display:flex;align-items:center;gap:6px;font-size:11px;color:var(--muted)}
20.live-pulse{width:7px;height:7px;border-radius:50%;background:var(--success);animation:pulse 2s infinite}
21@keyframes pulse{0%,100%{opacity:1;box-shadow:0 0 4px var(--success)}50%{opacity:.4}}
22.btn{background:var(--bg-elevated);border:1px solid var(--border-light);color:var(--text);border-radius:6px;padding:5px 12px;font-size:11px;cursor:pointer}
23.btn:hover{background:var(--border-light)}
24
25/* Fleet control plane */
26.control-panel{background:var(--bg-panel);border:1px solid var(--border);border-radius:10px;padding:14px 16px;margin-bottom:18px}
27.control-panel-head{display:flex;align-items:center;justify-content:space-between;gap:12px;margin-bottom:10px}
28.control-panel-head strong{color:var(--text-primary);font-size:13px}
29.control-panel-head .dim,.dim{color:var(--muted);font-size:11px}
30.control-panel-actions{display:flex;align-items:center;gap:10px}
31.toggle-auto{display:flex;align-items:center;gap:8px;font-size:11px;color:var(--text);cursor:pointer;user-select:none}
32.toggle-auto input{accent-color:var(--success)}
33.control-kpis{display:grid;grid-template-columns:repeat(5,1fr);gap:8px;margin-bottom:10px}
34.ckpi{background:var(--bg-elevated);border:1px solid var(--border);border-radius:8px;padding:8px 10px}
35.ckpi-v{font-size:18px;font-weight:700;letter-spacing:-.02em}
36.ckpi-l{font-size:9px;text-transform:uppercase;letter-spacing:.08em;color:var(--muted);margin-top:2px}
37.ckpi.ok .ckpi-v{color:var(--success)} .ckpi.warn .ckpi-v{color:var(--warning)} .ckpi.crit .ckpi-v{color:var(--critical)}
38.control-ranges{display:flex;flex-wrap:wrap;gap:6px;margin-bottom:8px}
39.range-chip{font-size:10px;font-family:var(--mono);color:var(--muted);background:var(--bg-elevated);border:1px solid var(--border);border-radius:999px;padding:3px 8px}
40.violations{margin-top:6px}
41.violation-row{display:flex;flex-wrap:wrap;align-items:center;gap:6px;padding:6px 0;border-top:1px solid var(--border);font-size:11px}
42.vchip{font-family:var(--mono);font-size:10px;padding:2px 6px;border-radius:4px;border:1px solid var(--border)}
43.vchip.critical{color:var(--critical);border-color:rgba(231,76,60,.35);background:rgba(231,76,60,.08)}
44.vchip.warning{color:var(--warning);border-color:rgba(240,160,32,.35);background:rgba(240,160,32,.08)}
45.auto-act{font-family:var(--mono);font-size:10px;color:var(--accent)}
46.in-range-ok{font-size:11px;color:var(--success);padding:6px 0}
47.remed-log{margin-top:8px;border-top:1px solid var(--border);padding-top:6px}
48.remed-item{font-family:var(--mono);font-size:10px;color:var(--muted);padding:2px 0}
49.remed-item.ok{color:var(--success)} .remed-item.fail{color:var(--critical)}
50.badge.small,.badge.medium,.badge.large{background:rgba(68,147,248,.1);color:var(--accent);border:1px solid rgba(68,147,248,.2)}
51
52.container{padding:20px 24px;max-width:1700px}
53.kpis{display:grid;grid-template-columns:repeat(7,1fr);gap:1px;background:var(--border);margin-bottom:20px;border-radius:8px;overflow:hidden}
54.kpi{background:var(--bg-panel);padding:12px 16px}
55.kpi-value{font-size:22px;font-weight:700;letter-spacing:-0.03em}
56.kpi-label{font-size:9px;text-transform:uppercase;letter-spacing:.08em;color:var(--muted);margin-top:2px}
57.kpi.success .kpi-value{color:var(--success)} .kpi.critical .kpi-value{color:var(--critical)} .kpi.warning .kpi-value{color:var(--warning)} .kpi.neutral .kpi-value{color:var(--accent)}
58
59.regions{display:flex;gap:8px;margin-bottom:20px;flex-wrap:wrap}
60.region-card{flex:1;min-width:120px;background:var(--bg-panel);border:1px solid var(--border);border-radius:8px;padding:10px 14px}
61.region-name{font-size:10px;text-transform:uppercase;letter-spacing:.06em;color:var(--muted)}
62.region-stats{display:flex;gap:10px;font-size:12px}
63.region-stats .up{color:var(--success)} .region-stats .down{color:var(--critical)} .region-stats .deg{color:var(--warning)} .region-stats .total{color:var(--muted);margin-left:auto}
64
65.section{margin-bottom:24px}
66.section-title{font-size:11px;text-transform:uppercase;letter-spacing:.1em;color:var(--muted);margin-bottom:10px;display:flex;align-items:center;gap:8px}
67.section-title .count{background:var(--bg-elevated);padding:1px 7px;border-radius:10px;font-size:10px}
68
69.tbl{width:100%;border-collapse:collapse;background:var(--bg-panel);border-radius:8px;overflow:hidden}
70.tbl th{text-align:left;padding:9px 12px;font-size:10px;font-weight:600;text-transform:uppercase;letter-spacing:.06em;color:var(--muted);border-bottom:1px solid var(--border);background:var(--bg-elevated);white-space:nowrap}
71.tbl td{padding:9px 12px;border-bottom:1px solid var(--border);font-size:12px;cursor:pointer}
72.tbl tr:last-child td{border-bottom:none}
73.tbl tr.offline td{opacity:.4}
74.tbl tr.degraded td{background:rgba(240,160,32,.03)}
75.tbl tr:hover td{background:var(--bg-row)}
76.tbl tr.expanded td{background:var(--bg-row)}
77.tbl tr.detail-row td{padding:0;cursor:default}
78.chevron{display:inline-block;font-size:9px;color:var(--dim);transition:transform .2s;cursor:pointer;width:12px}
79.chevron.open{transform:rotate(90deg)}
80
81.indicator{display:inline-flex;align-items:center;gap:6px}
82.dot{width:7px;height:7px;border-radius:50%;flex-shrink:0}
83.dot.healthy{background:var(--success);box-shadow:0 0 6px rgba(46,204,113,.4)}
84.dot.offline{background:var(--critical)} .dot.degraded{background:var(--warning);box-shadow:0 0 4px rgba(240,160,32,.3)}
85
86.badge{display:inline-block;padding:2px 7px;border-radius:4px;font-size:10px;font-weight:600}
87.badge.stable{background:rgba(46,204,113,.1);color:var(--success);border:1px solid rgba(46,204,113,.15)}
88.badge.canary{background:rgba(240,160,32,.1);color:var(--warning);border:1px solid rgba(240,160,32,.15)}
89
90.mono{font-family:var(--mono);font-size:11px;color:var(--muted)}
91.mono-val{font-family:var(--mono);font-size:11px;color:var(--text)}
92.ok-text{color:var(--success)} .err-text{color:var(--critical)}
93.version-ok{color:var(--success)} .version-bad{color:var(--critical)}
94.svc-count{background:rgba(68,147,248,.12);color:var(--accent);padding:1px 6px;border-radius:4px;font-size:11px;font-weight:600}
95
96.bar-cell{display:flex;align-items:center;gap:6px}
97.bar{width:50px;height:5px;background:rgba(255,255,255,0.06);border-radius:3px;overflow:hidden;display:inline-block;vertical-align:middle}
98.bar.ram{width:40px;height:4px}
99.fill{height:100%;border-radius:3px;transition:width .5s,background .5s;display:block}
100.fill.norm{background:var(--success)} .fill.warn{background:var(--warning)} .fill.crit{background:var(--critical)}
101.bar-pct{font-family:var(--mono);font-size:10px;color:var(--muted);min-width:28px}
102
103/* Detail panel */
104.detail-panel{padding:20px 24px;background:var(--bg-row);border-top:1px solid var(--border)}
105.detail-grid{display:grid;grid-template-columns:1fr 1fr 1fr;gap:24px}
106.detail-section-title{font-size:11px;text-transform:uppercase;letter-spacing:.08em;color:var(--accent);margin-bottom:10px}
107
108.feature-grid{display:flex;flex-direction:column;gap:4px}
109.feature-item{display:flex;align-items:center;gap:8px;padding:6px 8px;border-radius:6px;background:var(--bg-panel);border:1px solid var(--border)}
110.feature-item.on{border-color:rgba(46,204,113,.2)} .feature-item.off{opacity:.6}
111.feat-status{font-size:9px;font-weight:700;padding:2px 5px;border-radius:3px}
112.feat-status.on{background:rgba(46,204,113,.15);color:var(--success)} .feat-status.off{background:rgba(90,107,126,.15);color:var(--dim)}
113.feat-label{font-size:12px;flex:1}
114.feat-extra{font-size:10px;color:var(--muted);font-family:var(--mono)}
115.feat-toggle{background:var(--bg-elevated);border:1px solid var(--border-light);color:var(--text);border-radius:4px;padding:3px 8px;font-size:10px;cursor:pointer}
116.feat-toggle:hover{background:var(--border-light)} .feat-toggle.on{color:var(--success);border-color:rgba(46,204,113,.3)} .feat-toggle.off{color:var(--critical);border-color:rgba(231,76,60,.3)}
117
118.stats-grid{display:grid;grid-template-columns:1fr 1fr;gap:4px}
119.stat-item{display:flex;justify-content:space-between;padding:4px 8px;border-radius:4px;background:var(--bg-panel)}
120.stat-label{font-size:10px;color:var(--muted)} .stat-value{font-size:11px;color:var(--text);font-family:var(--mono)}
121
122.svc-list{display:flex;flex-direction:column;gap:4px}
123.svc-item{display:flex;align-items:center;gap:8px;padding:5px 8px;background:var(--bg-panel);border-radius:4px}
124.svc-name{font-weight:600;font-size:12px;color:var(--accent)} .svc-ver{font-size:10px;color:var(--muted);font-family:var(--mono)} .svc-caps{font-size:10px;color:var(--dim)}
125
126.control-list{display:flex;flex-direction:column;gap:6px}
127.ctrl-btn{background:var(--bg-panel);border:1px solid var(--border-light);color:var(--text);border-radius:6px;padding:8px 14px;font-size:12px;cursor:pointer;text-align:left;transition:all .15s}
128.ctrl-btn:hover:not(:disabled){background:var(--bg-elevated);border-color:var(--accent)}
129.ctrl-btn:disabled{opacity:.4;cursor:not-allowed}
130
131.env-list{display:flex;flex-direction:column;gap:3px;max-height:200px;overflow-y:auto}
132.env-item{display:flex;justify-content:space-between;padding:3px 8px;background:var(--bg-panel);border-radius:4px}
133.env-key{font-size:10px;color:var(--muted);font-family:var(--mono)} .env-val{font-size:10px;color:var(--text);font-family:var(--mono)}
134.env-empty{color:var(--dim);font-size:11px;padding:8px}
135
136/* Action log */
137.action-log{background:var(--bg-panel);border:1px solid var(--border);border-radius:8px;padding:8px 12px;max-height:200px;overflow-y:auto}
138.action-item{display:flex;gap:10px;padding:4px 0;border-bottom:1px solid #1a1e26;font-size:11px}
139.action-item:last-child{border-bottom:none}
140.action-time{color:var(--dim);font-family:var(--mono);min-width:70px}
141.action-node{color:var(--accent);font-weight:600;min-width:80px}
142.action-type{color:var(--text);min-width:120px}
143.action-detail{color:var(--muted);flex:1}
144
145/* Toast */
146#toast-container{position:fixed;bottom:20px;right:20px;z-index:200;display:flex;flex-direction:column;gap:8px}
147.toast{background:var(--bg-elevated);border:1px solid var(--border-light);border-radius:8px;padding:10px 16px;font-size:12px;color:var(--text);animation:slideIn .3s ease;box-shadow:0 4px 12px rgba(0,0,0,.3)}
148@keyframes slideIn{from{transform:translateX(100%);opacity:0}to{transform:none;opacity:1}}
149
150.footer{padding:14px 24px;border-top:1px solid var(--border);margin-top:24px}
151.footer-meta{font-size:10px;color:var(--dim);font-family:var(--mono)}
152
153@media(max-width:1200px){.kpis{grid-template-columns:repeat(4,1fr)}.detail-grid{grid-template-columns:1fr}.regions{flex-wrap:wrap}}
154
155/* Buttons */
156.btn.primary{background:linear-gradient(180deg,var(--accent),#3672d4);color:#fff;border-color:transparent;font-weight:600}
157.btn.ghost{background:transparent}
158.add-btn{font-size:10px;padding:3px 10px;margin-left:auto}
159
160/* Modal */
161.modal-overlay{position:fixed;inset:0;background:rgba(0,0,0,.7);display:none;align-items:center;justify-content:center;z-index:200}
162.modal-overlay.active{display:flex}
163.modal{background:var(--bg-panel);border:1px solid var(--border-light);border-radius:12px;padding:24px;min-width:440px;max-width:500px;box-shadow:0 8px 32px rgba(0,0,0,.4)}
164.modal h2{font-size:16px;margin-bottom:16px;color:var(--text-primary)}
165.modal-field{margin-bottom:10px}
166.modal-field label{display:block;font-size:11px;color:var(--muted);margin-bottom:3px}
167.modal-field input,.modal-field select{width:100%;background:var(--bg-elevated);border:1px solid var(--border);border-radius:6px;color:var(--text);padding:7px 10px;font:inherit;font-size:13px}
168.modal-field input:focus,.modal-field select:focus{outline:none;border-color:var(--accent)}
169.modal-field-row{display:flex;gap:10px}
170.modal-field-row .modal-field{flex:1}
171.modal-actions{display:flex;gap:8px;justify-content:flex-end;margin-top:16px}
dashboard-admin-operator/server.cjs +636 -0
……@@ -0,0 +1,636 @@
1'use strict'
2
3/**
4 * HiveRelay · Dashboard Admin Operator
5 *
6 * Full fleet control plane:
7 * - Live probe (SSH /status + /health)
8 * - Healthy-range policy (memory/disk/swap/watchdog by tier)
9 * - Auto-remediation (force restart, doctor, swap, baseline watchdog)
10 * - Manual actions: restart / update / doctor / feature toggles / inventory
11 *
12 * Inventory:
13 * HIVERELAY_FLEET_DIR — fleet dir (default: <repo>/fleet)
14 * HIVERELAY_FLEET_INVENTORY — explicit relays JSON path
15 * Prefers fleet/relays.local.json when present (correct SSH keys).
16 *
17 * Run:
18 * npm run dashboard:admin-operator
19 * # → http://localhost:3458
20 */
21
22const http = require('http')
23const { exec } = require('child_process')
24const fs = require('fs')
25const path = require('path')
26const { createController } = require('./controller.cjs')
27
28const INTERVAL_S = parseInt(process.env.FLEET_INTERVAL || '60', 10)
29const PORT = parseInt(process.env.PORT || '3458', 10)
30const POLICY_PATH = process.env.HIVERELAY_CONTROL_POLICY
31 || path.join(__dirname, 'control-policy.json')
32
33function expandHome (p) {
34 if (!p) return p
35 return p.replace(/^~(?=\/|$)/, process.env.HOME || '')
36}
37
38function resolveFleetDir () {
39 if (process.env.HIVERELAY_FLEET_DIR) {
40 return path.resolve(expandHome(process.env.HIVERELAY_FLEET_DIR))
41 }
42 // <repo>/dashboard-admin-operator → <repo>/fleet
43 return path.resolve(__dirname, '../fleet')
44}
45
46function resolveRelaysJson (fleetDir) {
47 if (process.env.HIVERELAY_FLEET_INVENTORY) {
48 return path.resolve(expandHome(process.env.HIVERELAY_FLEET_INVENTORY))
49 }
50 const local = path.join(fleetDir, 'relays.local.json')
51 if (fs.existsSync(local)) return local
52 return path.join(fleetDir, 'relays.json')
53}
54
55function loadChannels (fleetDir) {
56 const p = path.join(fleetDir, 'channels.json')
57 if (fs.existsSync(p)) return JSON.parse(fs.readFileSync(p, 'utf8'))
58 return { stable: 'v0.24.3', canary: 'v0.25.0-rc.4' }
59}
60
61const FLEET_DIR = resolveFleetDir()
62const RELAYS_JSON = resolveRelaysJson(FLEET_DIR)
63const channels = loadChannels(FLEET_DIR)
64
65let fleetState = []
66let lastProbe = null
67let probing = false
68const sseClients = []
69const actionLog = []
70
71// --- Probe logic ----------------------------------------------------------
72
73function loadRelays () {
74 return JSON.parse(fs.readFileSync(RELAYS_JSON, 'utf8')).relays
75}
76
77function probeNode (relay) {
78 return new Promise((resolve) => {
79 const ip = relay.publicIp
80 const keyArg = relay.sshKey && relay.sshKey !== 'default'
81 ? ['-i', relay.sshKey.replace(/^~/, process.env.HOME)]
82 : []
83
84 const remote = [
85 'V=$(grep -m1 \'"version"\' ~/hiverelay/package.json 2>/dev/null | tr -dc \'0-9.\' || echo \'none\')',
86 'D=$(df -h / | awk \'NR==2{print $5}\' | tr -dc \'0-9\')',
87 'DG=$(df -h / | awk \'NR==2{print $2}\' | tr -dc \'0-9.\')',
88 'SW=$(free -m 2>/dev/null | awk \'/Swap/{print $2}\' | head -1)',
89 'RM=$(free -m 2>/dev/null | awk \'/Mem/{printf "%d", $3/$2*100}\')',
90 'LA=$(cat /proc/loadavg | awk \'{print $1}\')',
91 'UP=$(uptime -s 2>/dev/null || echo \'?\')',
92 'RT=$(systemctl show hiverelay -p NRestarts --value 2>/dev/null || echo \'?\')',
93 // Prefer process RSS (MB) — MemoryCurrent cgroup often includes cache and looks multi-GB wrongly
94 'PID=$(systemctl show hiverelay -p MainPID --value 2>/dev/null)',
95 'MC=$(ps -o rss= -p "$PID" 2>/dev/null | awk \'{printf "%d",$1/1024}\')',
96 'TS=$(systemctl show hiverelay -p TasksCurrent --value 2>/dev/null || echo \'?\')',
97 'ENV=$(systemctl show hiverelay -p Environment --value 2>/dev/null | tr \'\' \';\' | tr \' \' \';\')',
98 'CFG=$(cat ~/hiverelay/config.json 2>/dev/null || cat ~/.hiverelay/config.json 2>/dev/null || echo \'{}\')',
99 'STATUS=$(curl -fsS --max-time 8 http://127.0.0.1:9100/status 2>/dev/null || echo \'{}\')',
100 'HEALTH=$(curl -fsS --max-time 4 http://127.0.0.1:9100/health 2>/dev/null || echo \'\')',
101 'WD=$(systemctl is-enabled hiverelay-health-watchdog.timer 2>/dev/null || echo missing)',
102 'HO=0; echo "$HEALTH" | grep -q \'"ok"[[:space:]]*:[[:space:]]*true\' && HO=1',
103 'printf \'{"version":"%s","diskPct":%s,"diskGB":%s,"swapMB":%s,"ramUsedPct":%s,"loadAvg":%s,"uptime":"%s","restarts":%s,"memCurrentMB":%s,"tasks":%s,"env":"%s","watchdog":"%s","healthOk":%s,"config":%s,"status":%s}\\n\' "$V" "${D:-0}" "${DG:-0}" "${SW:-0}" "${RM:-0}" "${LA:-0}" "$UP" "${RT:-0}" "${MC:-0}" "${TS:-0}" "$ENV" "$WD" "$HO" "$CFG" "$STATUS"'
104 ].join('\n')
105
106 // timeout is required: macOS nc -w is unreliable and can hang forever on dead hosts
107 exec(`nc -z -G 3 -w 3 ${ip} 22 2>&1`, { timeout: 6000 }, (tcpErr) => {
108 if (tcpErr) {
109 return resolve(makeOffline(relay))
110 }
111
112 const sshBase = `ssh ${keyArg.join(' ')} -o IdentitiesOnly=yes -o ConnectTimeout=8 -o BatchMode=yes -o StrictHostKeyChecking=accept-new root@${ip}`
113 exec(`${sshBase} '${remote.replace(/'/g, "'\\''")}' 2>/dev/null`, { timeout: 25000 }, (err, stdout) => {
114 if (err || !stdout || !stdout.trim().startsWith('{')) {
115 return resolve({ ...makeOffline(relay), reachable: true, status: 'degraded' })
116 }
117
118 try {
119 const d = JSON.parse(stdout.trim())
120 const st = d.status || {}
121 const health = st.health || {}
122 const healthOk = d.healthOk === 1 || d.healthOk === true || health.healthy === true
123 const running = st.running === true || health.healthy === true
124 // Prefer health endpoint: LISTEN-but-hung nodes show degraded
125 let status = 'degraded'
126 if (running && healthOk) status = 'healthy'
127 else if (!running && !healthOk) status = 'degraded'
128
129 // Extract features from config + env
130 const config = d.config || {}
131 const env = {}
132 for (const pair of (d.env || '').split(';')) {
133 const [k, v] = pair.split('=')
134 if (k) env[k.trim()] = (v || '').trim()
135 }
136
137 const features = extractFeatures(config, env, st)
138
139 resolve({
140 ...relay,
141 status,
142 reachable: true,
143 version: d.version || 'none',
144 running,
145 healthOk,
146 watchdog: d.watchdog || 'missing',
147 diskPct: d.diskPct,
148 diskGB: parseFloat(d.diskGB) || relay.diskGB,
149 swapMB: d.swapMB,
150 ramUsedPct: d.ramUsedPct,
151 loadAvg: d.loadAvg,
152 uptime: d.uptime,
153 restarts: d.restarts,
154 memCurrentMB: d.memCurrentMB,
155 tasks: d.tasks,
156 targetVersion: channels[relay.channel] || '?',
157 probedAt: new Date().toISOString(),
158 // Rich status data
159 seededApps: st.seededApps,
160 connections: st.connections,
161 coresSeeded: st.seeder?.coresSeeded,
162 bytesServed: st.served?.totalBytesServed,
163 blocksServed: st.served?.totalBlocksServed,
164 replication: st.replication,
165 transports: st.transports,
166 appRegistry: st.appRegistry,
167 services: st.services?.services || [],
168 payment: st.payment,
169 // Config
170 config,
171 env,
172 features
173 })
174 } catch {
175 resolve({ ...makeOffline(relay), reachable: true, status: 'degraded' })
176 }
177 })
178 })
179 })
180}
181
182function makeOffline (relay) {
183 return {
184 ...relay,
185 status: 'offline',
186 reachable: false,
187 version: null,
188 running: false,
189 diskPct: null,
190 swapMB: null,
191 ramUsedPct: null,
192 loadAvg: null,
193 restarts: null,
194 features: {},
195 config: {},
196 env: {},
197 services: [],
198 probedAt: new Date().toISOString()
199 }
200}
201
202function extractFeatures (config, env, status) {
203 return {
204 coreRelay: { enabled: true, running: status.running === true },
205 seeding: { enabled: true, coresSeeded: status.seeder?.coresSeeded || 0 },
206 services: { enabled: config.enableServices === true, count: status.services?.count || 0 },
207 dhtRelayWs: { enabled: config.transports?.dhtRelayWs === true || env.HIVERELAY_DHT_RELAY_WS === '1', running: status.transports?.dhtRelayWs?.running },
208 tor: { enabled: !!status.transports?.tor, running: !!status.transports?.tor },
209 holesail: { enabled: !!status.transports?.holesail, running: !!status.transports?.holesail },
210 outboxlog: { enabled: config.plugins?.includes('outboxlog') || env.HIVERELAY_OUTBOXLOG === '1', namespaces: config.outboxlog?.namespaces || {} },
211 forwardRelay: { enabled: config.forwardRelay?.enabled === true },
212 signedDirectory: { enabled: config.signedDirectory?.enabled === true },
213 vrf: { enabled: (status.services?.services || []).some(s => s.name === 'vrf') },
214 poker: { enabled: config.plugins?.includes('poker') === true },
215 zk: { enabled: (status.services?.services || []).some(s => s.name === 'zk') },
216 arbitration: { enabled: (status.services?.services || []).some(s => s.name === 'arbitration') },
217 payment: { enabled: status.payment?.enabled === true, experimental: status.payment?.experimental },
218 eviction: { enabled: config.eviction?.enabled === true },
219 autoHeal: { enabled: true }, // from CLI flag
220 distributedDrive: { enabled: status.distributedDrive?.enabled === true, available: status.distributedDrive?.moduleAvailable }
221 }
222}
223
224async function probeAll () {
225 if (probing) return
226 probing = true
227 const relays = loadRelays()
228 const results = await Promise.allSettled(relays.map(r => probeNode(r)))
229 fleetState = results.map((r, i) =>
230 r.status === 'fulfilled' ? r.value : makeOffline(relays[i])
231 )
232 lastProbe = new Date().toISOString()
233 probing = false
234 broadcast({ type: 'fleet', data: fleetState, timestamp: lastProbe })
235
236 // Control plane: evaluate healthy ranges + auto-remediate
237 try {
238 const cycle = await controller.runControlCycle(fleetState)
239 if (cycle) {
240 broadcast({ type: 'control', data: cycle })
241 if (cycle.remediations && cycle.remediations.length) {
242 // Re-broadcast fleet after remediations updated nodes in place
243 broadcast({ type: 'fleet', data: fleetState, timestamp: new Date().toISOString() })
244 }
245 }
246 } catch (err) {
247 console.error('[admin-operator] control cycle error:', err.message)
248 }
249}
250
251// --- Control plane --------------------------------------------------------
252
253const controller = createController({
254 policyPath: POLICY_PATH,
255 sshAction: (relay, command, label) => sshAction(relay, command, label),
256 logAction,
257 probeNode,
258 loadRelays
259})
260
261function broadcast (msg) {
262 const data = `data: ${JSON.stringify(msg)}\n\n`
263 for (const res of sseClients) { try { res.write(data) } catch {} }
264}
265
266function logAction (node, action, detail) {
267 const entry = { node, action, detail, timestamp: new Date().toISOString() }
268 actionLog.unshift(entry)
269 if (actionLog.length > 100) actionLog.pop()
270 broadcast({ type: 'action', data: entry })
271}
272
273// --- SSH action executor --------------------------------------------------
274
275function sshAction (relay, command, label) {
276 return new Promise((resolve, reject) => {
277 const keyArg = relay.sshKey && relay.sshKey !== 'default'
278 ? ['-i', relay.sshKey.replace(/^~/, process.env.HOME)]
279 : []
280 const fullCmd = `ssh ${keyArg.join(' ')} -o IdentitiesOnly=yes -o ConnectTimeout=12 -o BatchMode=yes -o StrictHostKeyChecking=accept-new root@${relay.publicIp} '${command.replace(/'/g, "'\\''")}' 2>&1`
281 logAction(relay.name, label, 'started')
282 exec(fullCmd, { timeout: 60000 }, (err, stdout, stderr) => {
283 const output = (stdout || '') + (stderr || '')
284 if (err) {
285 logAction(relay.name, label, `FAILED: ${output.slice(0, 200)}`)
286 reject(new Error(output.slice(0, 500)))
287 } else {
288 logAction(relay.name, label, `done: ${output.slice(0, 100)}`)
289 resolve(output.trim())
290 }
291 })
292 })
293}
294
295// --- HTTP server ----------------------------------------------------------
296
297const server = http.createServer(async (req, res) => {
298 if (req.method === 'OPTIONS') {
299 res.writeHead(204, { 'access-control-allow-origin': '*', 'access-control-allow-methods': 'GET, POST, OPTIONS', 'access-control-allow-headers': 'content-type' })
300 return res.end()
301 }
302
303 // SSE
304 if (req.url === '/api/events') {
305 res.writeHead(200, { 'content-type': 'text/event-stream', 'cache-control': 'no-cache', 'connection': 'keep-alive', 'access-control-allow-origin': '*' })
306 res.write(`data: ${JSON.stringify({ type: 'fleet', data: fleetState, timestamp: lastProbe })}\n\n`)
307 res.write(`data: ${JSON.stringify({ type: 'actions', data: actionLog })}\n\n`)
308 sseClients.push(res)
309 req.on('close', () => { const i = sseClients.indexOf(res); if (i >= 0) sseClients.splice(i, 1) })
310 return
311 }
312
313 if (req.url === '/api/state') {
314 const cycle = controller.getLastCycle()
315 return sendJson(res, 200, {
316 nodes: fleetState,
317 summary: computeSummary(),
318 lastProbe,
319 channels,
320 interval: INTERVAL_S,
321 actions: actionLog.slice(0, 30),
322 control: {
323 autoControl: controller.getPolicy().autoControl,
324 policy: controller.getPolicy(),
325 lastCycle: cycle,
326 inRange: cycle?.summary?.inRange,
327 critical: cycle?.summary?.critical,
328 warnings: cycle?.summary?.warnings
329 }
330 })
331 }
332
333 if (req.url === '/api/reprobe' && req.method === 'POST') {
334 probeAll()
335 return sendJson(res, 200, { ok: true })
336 }
337
338 // Control plane API
339 if (req.url === '/api/control' && req.method === 'GET') {
340 return sendJson(res, 200, {
341 autoControl: controller.getPolicy().autoControl,
342 policy: controller.getPolicy(),
343 lastCycle: controller.getLastCycle()
344 })
345 }
346
347 if (req.url === '/api/control' && req.method === 'POST') {
348 try {
349 const body = await readBody(req)
350 if (typeof body.autoControl === 'boolean') {
351 controller.setAutoControl(body.autoControl)
352 }
353 if (body.reload) controller.reloadPolicy()
354 return sendJson(res, 200, { ok: true, autoControl: controller.getPolicy().autoControl, policy: controller.getPolicy() })
355 } catch (err) {
356 return sendJson(res, 400, { error: err.message })
357 }
358 }
359
360 // Force a control/remediation cycle now (respects cooldowns unless forceAll)
361 if (req.url === '/api/control/run' && req.method === 'POST') {
362 try {
363 const body = await readBody(req).catch(() => ({}))
364 const cycle = await controller.runControlCycle(fleetState, { force: body.force !== false })
365 broadcast({ type: 'control', data: cycle })
366 broadcast({ type: 'fleet', data: fleetState, timestamp: new Date().toISOString() })
367 return sendJson(res, 200, { ok: true, cycle })
368 } catch (err) {
369 return sendJson(res, 500, { error: err.message })
370 }
371 }
372
373 // Stabilize entire fleet: ensure baseline + swap on all reachable nodes
374 if (req.url === '/api/control/stabilize-all' && req.method === 'POST') {
375 const relays = loadRelays()
376 const results = []
377 for (const relay of relays) {
378 const node = fleetState.find(n => n.name === relay.name)
379 if (node && node.status === 'offline' && !node.reachable) {
380 results.push({ name: relay.name, skipped: true, reason: 'offline' })
381 continue
382 }
383 try {
384 await sshAction(relay, controller.REMOTE.ensureBaseline + '\n' + controller.REMOTE.ensureSwap, 'stabilize-all')
385 results.push({ name: relay.name, ok: true })
386 } catch (err) {
387 results.push({ name: relay.name, ok: false, error: err.message?.slice(0, 200) })
388 }
389 }
390 probeAll()
391 return sendJson(res, 200, { ok: true, results })
392 }
393
394 // Add a new relay to the fleet
395 if (req.url === '/api/relay' && req.method === 'POST') {
396 const body = await readBody(req)
397 if (!body.name || !body.publicIp) return sendJson(res, 400, { error: 'name and publicIp required' })
398
399 const relays = loadRelays()
400 if (relays.find(r => r.name === body.name)) {
401 return sendJson(res, 409, { error: 'relay name already exists' })
402 }
403 if (relays.find(r => r.publicIp === body.publicIp)) {
404 return sendJson(res, 409, { error: 'IP already in fleet' })
405 }
406
407 const newRelay = {
408 name: body.name,
409 publicIp: body.publicIp,
410 tailnet: null,
411 sshKey: body.sshKey || 'default',
412 channel: body.channel || 'stable',
413 region: body.region || 'NA',
414 diskGB: body.diskGB || 60,
415 ramGB: body.ramGB || 2,
416 vcpu: body.vcpu || 1,
417 role: body.role || 'seed'
418 }
419 relays.push(newRelay)
420 fs.writeFileSync(RELAYS_JSON, JSON.stringify({ _doc: 'Fleet inventory. Updated via dashboard.', relays }, null, 2))
421 logAction(body.name, 'add relay', `added to fleet (${body.region}, ${body.channel})`)
422
423 // Probe the new node immediately
424 const probed = await probeNode(newRelay)
425 fleetState.push(probed)
426 broadcast({ type: 'fleet', data: fleetState, timestamp: new Date().toISOString() })
427 return sendJson(res, 200, { ok: true, relay: newRelay, state: probed })
428 }
429
430 // Remove a relay from the fleet config
431 if (req.url.match(/^\/api\/relay\/[^/]+$/) && req.method === 'DELETE') {
432 const nodeName = decodeURIComponent(req.url.split('/').slice(-1)[0])
433 const relays = loadRelays()
434 const filtered = relays.filter(r => r.name !== nodeName)
435 if (filtered.length === relays.length) return sendJson(res, 404, { error: 'not found' })
436 fs.writeFileSync(RELAYS_JSON, JSON.stringify({ _doc: 'Fleet inventory. Updated via dashboard.', relays: filtered }, null, 2))
437 fleetState = fleetState.filter(n => n.name !== nodeName)
438 broadcast({ type: 'fleet', data: fleetState, timestamp: new Date().toISOString() })
439 logAction(nodeName, 'remove relay', 'removed from fleet config')
440 return sendJson(res, 200, { ok: true })
441 }
442
443 // Node detail
444 // Node detail (GET only — POST routes to actions below)
445 const nodeMatch = req.url.match(/^\/api\/node\/([^/]+)$/)
446 if (nodeMatch && req.method === 'GET') {
447 const nodeName = decodeURIComponent(nodeMatch[1])
448 const relay = loadRelays().find(r => r.name === nodeName)
449 const state = fleetState.find(n => n.name === nodeName)
450 if (!relay) return sendJson(res, 404, { error: 'node not found' })
451 return sendJson(res, 200, { relay, state })
452 }
453
454 // Actions: restart, update, doctor, toggle feature
455 const actionMatch = req.url.match(/^\/api\/node\/([^/]+)\/(restart|update|doctor|toggle)$/)
456 if (actionMatch && req.method === 'POST') {
457 const [, nodeName, action] = actionMatch
458 const relay = loadRelays().find(r => r.name === nodeName)
459 if (!relay) return sendJson(res, 404, { error: 'node not found' })
460
461 try {
462 if (action === 'restart') {
463 await sshAction(relay, 'systemctl restart hiverelay; sleep 3; systemctl is-active hiverelay', 'restart relay')
464 // Re-probe this node
465 const updated = await probeNode(relay)
466 const idx = fleetState.findIndex(n => n.name === nodeName)
467 if (idx >= 0) fleetState[idx] = updated
468 broadcast({ type: 'fleet', data: fleetState, timestamp: new Date().toISOString() })
469 return sendJson(res, 200, { ok: true, state: updated })
470
471 } else if (action === 'update') {
472 const output = await sshAction(relay, 'cd ~/hiverelay && git pull --ff-only 2>&1 && npm install --production 2>&1 | tail -5 && systemctl restart hiverelay && sleep 5 && systemctl is-active hiverelay', 'update relay')
473 const updated = await probeNode(relay)
474 const idx = fleetState.findIndex(n => n.name === nodeName)
475 if (idx >= 0) fleetState[idx] = updated
476 broadcast({ type: 'fleet', data: fleetState, timestamp: new Date().toISOString() })
477 return sendJson(res, 200, { ok: true, output: output.slice(0, 500), state: updated })
478
479 } else if (action === 'doctor') {
480 const cmds = [
481 'echo "=== Swap ===" && free -m | grep Swap',
482 'SW=$(free -m | awk "/Swap/{print $2}") && if [ "$SW" = "0" ]; then echo "Adding 2G swap..." && fallocate -l 2G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile && echo "/swapfile none swap sw 0 0" >> /etc/fstab && sysctl vm.swappiness=10; fi',
483 'echo "=== Locks ===" && rm -f /root/.hiverelay/storage/*.lock 2>/dev/null && echo "cleared"',
484 'echo "=== Journal ===" && journalctl --vacuum-size=50M 2>/dev/null',
485 'echo "=== Disk ===" && df -h / | tail -1',
486 'echo "=== Relay ===" && systemctl is-active hiverelay && curl -fsS --max-time 5 http://127.0.0.1:9100/health 2>/dev/null | python3 -c "import sys,json;d=json.load(sys.stdin);print(d.get(\'running\'))" 2>/dev/null || echo "?"'
487 ].join('; ')
488 const output = await sshAction(relay, cmds, 'doctor relay')
489 return sendJson(res, 200, { ok: true, output })
490
491 } else if (action === 'toggle') {
492 const body = await readBody(req)
493 const { feature, enabled } = body
494 const result = await toggleFeature(relay, feature, enabled)
495 return sendJson(res, 200, { ok: true, ...result })
496 }
497 } catch (err) {
498 return sendJson(res, 500, { error: err.message })
499 }
500 }
501
502 // Static files (Admin Operator UI)
503 const publicDir = path.join(__dirname, 'public')
504 const urlPath = (req.url || '/').split('?')[0]
505 let filePath = path.join(publicDir, urlPath === '/' ? 'index.html' : urlPath)
506 // prevent path escape
507 if (!filePath.startsWith(publicDir) || !fs.existsSync(filePath) || fs.statSync(filePath).isDirectory()) {
508 filePath = path.join(publicDir, 'index.html')
509 }
510 serveStatic(res, filePath)
511})
512
513async function toggleFeature (relay, feature, enabled) {
514 // Map feature name to config/env changes
515 const featureMap = {
516 dhtRelayWs: { type: 'env', envVar: 'HIVERELAY_DHT_RELAY_WS', dropin: 'dht-relay-ws' },
517 outboxlog: { type: 'env', envVar: 'HIVERELAY_OUTBOXLOG', dropin: 'outboxlog', extraEnv: 'HIVERELAY_OUTBOXLOG_NAMESPACE=peerit' },
518 forwardRelay: { type: 'config', path: ['forwardRelay', 'enabled'] },
519 signedDirectory: { type: 'config', path: ['signedDirectory', 'enabled'] },
520 services: { type: 'config', path: ['enableServices'] },
521 eviction: { type: 'config', path: ['eviction', 'enabled'] }
522 }
523
524 const fm = featureMap[feature]
525 if (!fm) throw new Error(`Unknown feature: ${feature}`)
526
527 if (fm.type === 'env') {
528 const dropinPath = `/etc/systemd/system/hiverelay.service.d/${fm.dropin}.conf`
529 if (enabled) {
530 let envLines = `Environment=${fm.envVar}=1`
531 if (fm.extraEnv) envLines += `\nEnvironment=${fm.extraEnv}`
532 await sshAction(relay, `mkdir -p /etc/systemd/system/hiverelay.service.d && printf '[Service]\\n${envLines}\\n' > ${dropinPath} && systemctl daemon-reload && systemctl restart hiverelay`, `enable ${feature}`)
533 } else {
534 await sshAction(relay, `rm -f ${dropinPath} && systemctl daemon-reload && systemctl restart hiverelay`, `disable ${feature}`)
535 }
536 } else if (fm.type === 'config') {
537 // Edit config.json using python — creates the file if it doesn't exist
538 const [section, key] = fm.path.length === 2 ? fm.path : [null, fm.path[0]]
539 const valStr = enabled ? 'true' : 'false'
540 const pyCode = section
541 ? `d.setdefault('${section}',{})['${key}']=${valStr}`
542 : `d['${key}']=${valStr}`
543
544 await sshAction(relay,
545 [
546 'CFG_PATH=$( [ -f ~/hiverelay/config.json ] && echo ~/hiverelay/config.json || echo ~/.hiverelay/config.json )',
547 'mkdir -p ~/.hiverelay',
548 `python3 -c "import json; d=json.load(open('$CFG_PATH')) if __import__('os').path.exists('$CFG_PATH') else {}; ${pyCode}; json.dump(d, open('/tmp/cfg.json','w'), indent=2)"`,
549 'cp /tmp/cfg.json "$CFG_PATH"',
550 'systemctl restart hiverelay'
551 ].join(' && '),
552 `${enabled ? 'enable' : 'disable'} ${feature}`)
553 }
554
555 // Re-probe
556 const updated = await probeNode(relay)
557 const idx = fleetState.findIndex(n => n.name === relay.name)
558 if (idx >= 0) fleetState[idx] = updated
559 broadcast({ type: 'fleet', data: fleetState, timestamp: new Date().toISOString() })
560 return { state: updated }
561}
562
563function readBody (req) {
564 return new Promise((resolve, reject) => {
565 let d = ''
566 req.on('data', c => { d += c; if (d.length > 1e6) reject(new Error('too large')) })
567 req.on('end', () => { try { resolve(JSON.parse(d)) } catch { reject(new Error('bad json')) } })
568 req.on('error', reject)
569 })
570}
571
572function sendJson (res, code, data) {
573 const body = JSON.stringify(data)
574 res.writeHead(code, { 'content-type': 'application/json', 'access-control-allow-origin': '*' })
575 res.end(body)
576}
577
578function serveStatic (res, filePath) {
579 const types = { '.html': 'text/html', '.css': 'text/css', '.js': 'application/javascript', '.svg': 'image/svg+xml' }
580 const ct = types[path.extname(filePath)] || 'application/octet-stream'
581 try {
582 const data = fs.readFileSync(filePath)
583 res.writeHead(200, { 'content-type': ct })
584 res.end(data)
585 } catch { sendJson(res, 404, { error: 'not found' }) }
586}
587
588function computeSummary () {
589 const nodes = fleetState
590 const totalRamGB = nodes.reduce((s, n) => s + (n.ramGB || 0), 0)
591 const totalDiskGB = nodes.reduce((s, n) => s + (n.diskGB || 0), 0)
592 const totalSeeded = nodes.reduce((s, n) => s + (n.seededApps || 0), 0)
593 const totalConns = nodes.reduce((s, n) => s + (n.connections || 0), 0)
594 const regions = {}
595 for (const n of nodes) {
596 const r = n.region || 'Unknown'
597 if (!regions[r]) regions[r] = { total: 0, healthy: 0, degraded: 0, offline: 0 }
598 regions[r].total++
599 if (n.status === 'healthy') regions[r].healthy++
600 else if (n.status === 'offline') regions[r].offline++
601 else regions[r].degraded++
602 }
603 return {
604 total: nodes.length, healthy: nodes.filter(n => n.status === 'healthy').length,
605 degraded: nodes.filter(n => n.status === 'degraded').length,
606 offline: nodes.filter(n => n.status === 'offline').length,
607 running: nodes.filter(n => n.running === true).length,
608 diskWarnings: nodes.filter(n => n.diskPct && n.diskPct > 75).length,
609 totalRamGB: Math.round(totalRamGB), totalDiskGB, totalSeeded, totalConns, regions
610 }
611}
612
613// --- Bootstrap ------------------------------------------------------------
614
615async function main () {
616 console.log(`[admin-operator] Fleet dir: ${FLEET_DIR}`)
617 console.log(`[admin-operator] Inventory: ${RELAYS_JSON}`)
618 const relays = loadRelays()
619 console.log(`[admin-operator] ${relays.length} nodes configured`)
620 // Listen first so UI is available while the (slow) fleet probe runs
621 await new Promise((resolve, reject) => {
622 server.listen(PORT, () => {
623 console.log(`\n [admin-operator] Dashboard Admin Operator → http://localhost:${PORT}\n`)
624 resolve()
625 })
626 server.on('error', reject)
627 })
628 console.log(`[admin-operator] Running initial probe...`)
629 await probeAll()
630 const s = computeSummary()
631 console.log(`[admin-operator] Done: ${s.healthy} healthy, ${s.degraded} degraded, ${s.offline} offline`)
632 setInterval(() => { probeAll().catch(() => {}) }, INTERVAL_S * 1000)
633 console.log(`[admin-operator] Polling every ${INTERVAL_S}s`)
634}
635
636main().catch(err => { console.error('[admin-operator] Fatal:', err); process.exit(1) })
docs/CORESTORE7-MIGRATE-RUNBOOK.md +98 -0
……@@ -0,0 +1,98 @@
1# Corestore 7 production migration runbook
2
3**Date:** 2026-07-26
4**Status:** Proven on utah + utah-8gb for `v0.25.0-rc.4`
5
6## Root cause of the 0.25 “hang”
7
8Opening a **Corestore 6** production tree with **Corestore 7** is fine when RocksDB migration is clean. Earlier rc canaries left a **partial / corrupt `storage/db/`** next to legacy `storage/cores/`.
9
10In that mixed state:
11
12| Operation | Result |
13|-----------|--------|
14| `store.ready()` | Succeeds quickly |
15| `store.get({ name }).ready()` (any named core, even fresh) | **Hangs forever** |
16| Same tree under CS6 | Named cores ready in tens of ms |
17
18Empty CS7 stores work immediately. Removing the bad `db/` and letting CS7 rebuild from `cores/` restores `core.ready()` (~100ms for `app-registry-v1`).
19
20This is **not** “Corestore cannot open production data” — it is **incomplete CS7 RocksDB state from aborted canaries**.
21
22## Fix on a box (operator)
23
241. **Stop updater** so canary hold does not immediately pull back to 0.24.3:
25
26 ```bash
27 systemctl stop hiverelay-updater.timer
28 systemctl disable hiverelay-updater.timer
29 ```
30
312. Stop the relay and free `:9100` (avoid `pkill -f` patterns that match the SSH command line):
32
33 ```bash
34 systemctl kill -s SIGKILL hiverelay || true
35 fuser -k -9 9100/tcp 2>/dev/null || true
36 ```
37
383. **Quarantine** the bad RocksDB (keep for forensics; do not delete until soak is green):
39
40 ```bash
41 STORAGE=/root/.hiverelay/storage
42 if [ -d "$STORAGE/cores" ] && [ -d "$STORAGE/db" ]; then
43 mv "$STORAGE/db" "$STORAGE/db.corrupt-pre-cs7-$(date +%Y%m%d%H%M%S)"
44 fi
45 ```
46
474. Checkout **0.25** and install CS7 deps:
48
49 ```bash
50 cd /root/hiverelay
51 git fetch --tags origin
52 git checkout -f v0.25.0-rc.4
53 npm ci # need corestore@7; omit=dev can skip patch-package if you rely on patches
54 ```
55
565. Ensure full features + Tor config (`services.json` plugins + `config.transports.tor` / `dhtRelayWs`), host Tor daemon running.
57
586. Start and **wait** — first boot reseeds many apps; health may take **60–120s**:
59
60 ```bash
61 systemctl start hiverelay
62 # poll /health until version shows 0.25.x
63 ```
64
657. Verify:
66
67 ```bash
68 curl -sS http://127.0.0.1:9100/health
69 curl -sS http://127.0.0.1:9100/status # transports.tor.running, dhtRelayWs
70 curl -sS http://127.0.0.1:9100/api/v1/services
71 ```
72
73## Rollback to 0.24.3
74
751. Stop relay, `git checkout -f v0.24.3`, reinstall CS6 deps (`npm ci --omit=dev`).
762. Restore a **CS6-era** `db/` only if you had one that worked under 0.24; CS6 primarily uses `cores/`.
773. Re-enable updater only after channel points at the intended tag.
78
79## Do not
80
81- Point CS7 at a tree with a half-written `db/` from failed experiments without quarantine.
82- Leave canary channel at `v0.24.3` while manually running 0.25 — the fleet updater will **pull you back** (observed 2026-07-26 on utah).
83- Use `pkill -f 'cli/index.js'` over SSH — it can match and kill the remote shell.
84
85## Fleet status (post-fix, 2026-07-26 evening)
86
87| Box | Version | Tor | Notes |
88|-----|---------|-----|--------|
89| utah | `0.25.0-rc.4` | host + transport on | Updater timer disabled |
90| utah-8gb | `0.25.0-rc.4` | host + transport on | Updater timer disabled |
91| Other raw VPS | `0.24.3` | mostly off | Utilities rolled where SSH worked |
92| bern, utah-0.5gb | — | — | SSH key / reachability broken |
93
94## Code follow-ups (for rc.5)
95
961. Startup guard: if `cores/` exists and `db/` fails `get({name}).ready()` within N seconds, emit a clear error / optional auto-quarantine of `db/`.
972. Point canary channel at `v0.25.0-rc.4` (or later) once soak is accepted; re-enable updater.
983. Investigate **outboxlog** missing from `/api/v1/services` while listed in `services.json` (other utilities load).
docs/FLEET-STABILITY-2026-07-27.md +55 -0
……@@ -0,0 +1,55 @@
1# Fleet stability — why nodes were dropping (2026-07-27)
2
3## Failure modes (observed)
4
5| Mode | Symptom | Root cause | Nodes hit |
6|------|---------|------------|-----------|
7| **Event-loop hang** | `systemctl active`, `:9100` LISTEN, `/health` times out | Process stuck; often memory cgroup reclaim (`wchan=mem_cgroup_handle_ov`) or CS7 bad `storage/db` | utah-8gb, miami, others intermittently |
8| **OOM / cgroup kill** | journal `oom-kill`, `MemoryMax` | Working set exceeds cgroup cap; no swap on large boxes | utah-8gb (5.8G peak), sing-1 |
9| **ABRT / core-dump** | `status=6/ABRT` every few hours | Heap/native pressure on **0.5–1GB** boxes running full plugin suite | miami, utah-0.5gb, sing-1 |
10| **CS7 bloated RocksDB** | RSS multi-GB, hang within minutes of start | Partial/corrupt `storage/db` growing to multi-GB blobs; healthy utah has ~186MB db | utah-8gb (**8.6GB** db quarantined) |
11| **SSH offline** | connect timeout | Host/network unreachable | bern |
12
13Systemd alone cannot fix hangs: the main PID stays alive, so `Restart=` never fires.
14
15## Fixes applied
16
17### 1. Local health watchdog (all reachable boxes)
18
19- `fleet/health-watchdog.sh` + `hiverelay-health-watchdog.{service,timer}`
20- Install: `ssh root@box 'bash -s' < fleet/install-health-watchdog.sh`
21- Every **2 minutes**: if unit is active but `/health` fails **twice** → `SIGKILL` + free `:9100` + `systemctl start`
22- 90s grace after process start (boot reseed)
23
24### 2. utah-8gb remigrate + memory
25
26- Quarantined `storage/db` (8.6GB) → `db.bloated-pre-stabilize-*`
27- CS7 rebuilds from `cores/`; RSS dropped **~5.5GB → ~110MB**
28- Added **4G swap**, disabled thrashing `MemoryHigh` soft-lock (`MemoryHigh=infinity`, keep `MemoryMax=6G`)
29- Full feature plugins retained
30
31### 3. Small boxes (≤1GB RAM)
32
33- Plugin set trimmed to `outboxlog`, `notify`, `vrf` (full suite belongs on ≥2GB)
34- Restart + existing `MemoryMax` caps + swap kept
35
36### 4. Fleet-wide hardening drop-ins
37
38- `Restart=always`, `TimeoutStopSec=12`, `KillMode=mixed`, `FinalKillSignal=SIGKILL`
39- Swap added where missing (utah, dallas, utah-us, utah-2gb-a, …)
40
41## Post-fix snapshot
42
43**12/13 healthy**, bern offline (SSH timeout — infrastructure, not process).
44
45| Box | Version | ~RSS | Watchdog |
46|-----|---------|------|----------|
47| utah, utah-8gb | 0.25.0-rc.4 | ~110–210MB | enabled |
48| rest (reachable) | 0.24.3 | 130–780MB | enabled |
49
50## Ops notes
51
52- Do **not** use `pkill -f hiverelay` over SSH (kills the SSH session). Prefer `systemctl kill -s SIGKILL hiverelay` + `fuser -k 9100/tcp`.
53- If a 0.25 box hangs again with multi-GB RSS and large `storage/db`, re-quarantine db per `docs/CORESTORE7-MIGRATE-RUNBOOK.md`.
54- 0.5GB boxes will still be fragile under load; prefer seed-only light plugins.
55- Bern needs provider-side recovery (power/network/console).
docs/HIVERELAY-0.25-LAUNCH.html +559 -0
……@@ -0,0 +1,559 @@
1<!DOCTYPE html>
2<html lang="en">
3<head>
4<meta charset="UTF-8">
5<meta name="viewport" content="width=device-width, initial-scale=1.0">
6<title>HiveRelay 0.25 — what we're standing up</title>
7<link rel="icon" type="image/svg+xml" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'%3E%3Crect width='100' height='100' fill='%2316213e'/%3E%3Cpath d='M50 10 C 54 38 60 44 88 50 C 60 56 54 62 50 90 C 46 62 40 56 12 50 C 40 44 46 38 50 10 Z' fill='%23f5c842'/%3E%3C/svg%3E">
8<style>
9*,*::before,*::after{box-sizing:border-box;margin:0;padding:0}
10:root{
11 --bg:#0e1118;--bg-glow:#131a2b;
12 --card:#151a26;--card-hi:#1a2132;--card-border:#222b3e;--card-border-hi:#2e3b56;
13 --green:#3fb950;--orange:#f0a64a;--cyan:#58a6ff;--red:#f47067;--accent:#f5c842;--purple:#a371f7;
14 --text:#d6dde8;--text-dim:#8493a8;--text-faint:#7c89a0;--text-bright:#fff;
15 --mono:'SF Mono','Cascadia Code','Fira Code','Consolas',monospace;
16 --sans:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Helvetica,Arial,sans-serif;
17 --radius:16px;--radius-sm:10px;
18}
19html{font-size:16px;scroll-behavior:smooth}
20body{
21 background:var(--bg);
22 background-image:
23 radial-gradient(1200px 600px at 50% -200px,var(--bg-glow),transparent),
24 radial-gradient(900px 500px at 100% 0%,rgba(88,166,255,.05),transparent);
25 color:var(--text);font-family:var(--sans);line-height:1.55;
26 min-height:100vh;-webkit-font-smoothing:antialiased;
27}
28a{color:var(--cyan);text-decoration:none}
29a:hover{text-decoration:underline}
30code{font-family:var(--mono);font-size:.88em}
31:focus-visible{outline:2px solid var(--cyan);outline-offset:2px;border-radius:4px}
32@media(prefers-reduced-motion:reduce){*,*::before,*::after{transition:none!important;animation:none!important}}
33.wrap{max-width:1080px;margin:0 auto;padding:0 1.5rem}
34
35/* ── Top bar ──────────────────────────────────────────────────── */
36.topbar{
37 position:sticky;top:0;z-index:50;
38 backdrop-filter:blur(14px);-webkit-backdrop-filter:blur(14px);
39 background:rgba(14,17,24,.78);border-bottom:1px solid var(--card-border);
40}
41.topbar-inner{display:flex;align-items:center;gap:.8rem;padding:.8rem 0}
42.brand{display:flex;align-items:center;gap:.65rem}
43.spark{width:26px;height:26px;flex-shrink:0;filter:drop-shadow(0 0 7px rgba(245,200,66,.35))}
44.brand-name{font-size:1rem;font-weight:650;color:var(--text-bright);letter-spacing:.2px}
45.brand-sub{font-size:.62rem;color:var(--text-dim);letter-spacing:1px;text-transform:uppercase;margin-left:.1rem}
46.topbar-nav{margin-left:auto;display:flex;gap:1.3rem}
47.topbar-nav a{color:var(--text-dim);font-size:.83rem;transition:color .15s}
48.topbar-nav a:hover{color:var(--text-bright);text-decoration:none}
49@media(max-width:680px){.topbar-nav{display:none}}
50
51/* ── Hero ─────────────────────────────────────────────────────── */
52.hero{padding:5rem 0 3.2rem;text-align:center}
53.eyebrow{
54 display:inline-flex;align-items:center;gap:.5rem;
55 font-family:var(--mono);font-size:.72rem;letter-spacing:1.4px;text-transform:uppercase;
56 color:var(--accent);background:rgba(245,200,66,.08);
57 border:1px solid rgba(245,200,66,.25);padding:.4rem .9rem;border-radius:999px;margin-bottom:1.6rem;
58}
59.eyebrow .dot{width:6px;height:6px;border-radius:50%;background:var(--accent);box-shadow:0 0 6px var(--accent)}
60.hero h1{
61 font-size:clamp(2.2rem,6vw,3.7rem);line-height:1.05;font-weight:700;
62 color:var(--text-bright);letter-spacing:-1px;margin-bottom:1.1rem;
63}
64.hero h1 .grad{
65 background:linear-gradient(120deg,var(--accent),var(--orange));
66 -webkit-background-clip:text;background-clip:text;-webkit-text-fill-color:transparent;
67}
68.hero p.lead{font-size:1.12rem;color:var(--text-dim);max-width:640px;margin:0 auto 2rem}
69.chans{display:flex;flex-wrap:wrap;gap:.7rem;justify-content:center;margin-bottom:2.2rem}
70.chip{
71 display:inline-flex;align-items:center;gap:.45rem;font-family:var(--mono);font-size:.78rem;
72 padding:.4rem .85rem;border-radius:999px;border:1px solid var(--card-border);background:var(--card);color:var(--text);
73}
74.chip .dot{width:7px;height:7px;border-radius:50%;flex-shrink:0}
75.chip.canary .dot{background:var(--accent);box-shadow:0 0 6px var(--accent)}
76.chip.stable .dot{background:var(--green);box-shadow:0 0 6px var(--green)}
77.chip.track .dot{background:var(--purple);box-shadow:0 0 6px var(--purple)}
78.cta-row{display:flex;gap:.8rem;justify-content:center;flex-wrap:wrap}
79.btn{
80 font-family:inherit;font-size:.9rem;font-weight:550;padding:.72rem 1.3rem;border-radius:10px;
81 border:1px solid transparent;transition:transform .12s,background .15s,border-color .15s;cursor:pointer;
82}
83.btn:active{transform:translateY(1px)}
84.btn-primary{background:var(--accent);color:#1a1400;border-color:var(--accent)}
85.btn-primary:hover{background:#ffcf52}
86.btn-ghost{background:transparent;color:var(--text);border-color:var(--card-border-hi)}
87.btn-ghost:hover{border-color:var(--cyan);color:var(--cyan)}
88
89/* ── Section scaffold ─────────────────────────────────────────── */
90section{padding:3.4rem 0}
91.sec-head{margin-bottom:2rem;max-width:720px}
92.sec-head.center{margin-left:auto;margin-right:auto;text-align:center}
93.sec-kicker{font-family:var(--mono);font-size:.72rem;letter-spacing:1.3px;text-transform:uppercase;color:var(--cyan);margin-bottom:.6rem}
94.sec-head h2{font-size:clamp(1.6rem,3.5vw,2.2rem);font-weight:650;color:var(--text-bright);letter-spacing:-.4px;margin-bottom:.6rem}
95.sec-head p{color:var(--text-dim);font-size:1.02rem}
96
97/* ── Status strip ─────────────────────────────────────────────── */
98.status-strip{
99 display:grid;grid-template-columns:repeat(auto-fit,minmax(180px,1fr));gap:1px;
100 background:var(--card-border);border:1px solid var(--card-border);border-radius:var(--radius);overflow:hidden;
101 margin-top:-1rem;
102}
103.stat{background:var(--card);padding:1.3rem 1.4rem}
104.stat .k{font-size:.7rem;letter-spacing:.5px;text-transform:uppercase;color:var(--text-faint);margin-bottom:.5rem}
105.stat .v{font-family:var(--mono);font-size:1.35rem;font-weight:600;color:var(--text-bright)}
106.stat .sub{font-size:.78rem;color:var(--text-dim);margin-top:.3rem}
107.stat .v.green{color:var(--green)} .stat .v.accent{color:var(--accent)} .stat .v.orange{color:var(--orange)}
108
109/* ── Feature cards ────────────────────────────────────────────── */
110.grid{display:grid;grid-template-columns:repeat(auto-fill,minmax(320px,1fr));gap:1.1rem}
111.card{
112 background:linear-gradient(180deg,var(--card-hi),var(--card));
113 border:1px solid var(--card-border);border-radius:var(--radius);padding:1.5rem;
114 transition:border-color .18s,transform .18s;display:flex;flex-direction:column;
115}
116.card:hover{border-color:var(--card-border-hi);transform:translateY(-2px)}
117.card .ico{
118 width:38px;height:38px;border-radius:10px;display:flex;align-items:center;justify-content:center;
119 background:rgba(88,166,255,.1);color:var(--cyan);margin-bottom:1rem;font-size:1.15rem;
120}
121.card.h-accent .ico{background:rgba(245,200,66,.1);color:var(--accent)}
122.card.h-orange .ico{background:rgba(240,166,74,.12);color:var(--orange)}
123.card.h-green .ico{background:rgba(63,185,80,.12);color:var(--green)}
124.card.h-purple .ico{background:rgba(163,113,247,.12);color:var(--purple)}
125.card h3{font-size:1.06rem;color:var(--text-bright);font-weight:600;margin-bottom:.5rem;letter-spacing:.1px}
126.card p{font-size:.92rem;color:var(--text-dim);margin-bottom:.85rem}
127.card .unlocks{
128 margin-top:auto;padding-top:.9rem;border-top:1px dashed var(--card-border);font-size:.83rem;
129}
130.card .unlocks .lbl{color:var(--text-faint);text-transform:uppercase;letter-spacing:.6px;font-size:.66rem;display:block;margin-bottom:.4rem}
131.card .unlocks .rung{display:inline-flex;align-items:center;gap:.3rem;font-family:var(--mono);font-size:.74rem;padding:.18rem .5rem;border-radius:6px;background:rgba(88,166,255,.08);color:var(--cyan);margin-right:.25rem;margin-bottom:.25rem}
132.card ul.tags{display:flex;flex-wrap:wrap;gap:.35rem;margin-top:.4rem}
133.card ul.tags li{font-family:var(--mono);font-size:.72rem;color:var(--text-dim);background:rgba(255,255,255,.04);padding:.15rem .45rem;border-radius:5px;list-style:none}
134code.endpoint{display:inline-block;font-family:var(--mono);font-size:.8rem;background:rgba(255,255,255,.05);color:var(--green);padding:.12rem .42rem;border-radius:5px;margin-top:.2rem}
135
136/* ── Honesty ladder ───────────────────────────────────────────── */
137.ladder{display:grid;grid-template-columns:repeat(auto-fit,minmax(260px,1fr));gap:1rem}
138.rung{
139 background:var(--card);border:1px solid var(--card-border);border-left-width:4px;border-radius:var(--radius-sm);
140 padding:1.2rem 1.3rem;display:flex;flex-direction:column;
141}
142.rung.live{border-left-color:var(--green)}
143.rung.partial{border-left-color:var(--orange)}
144.rung.path{border-left-color:var(--cyan)}
145.rung.future{border-left-color:var(--card-border-hi);opacity:.82}
146.rung.never{border-left-color:var(--red);opacity:.78}
147.rung .top{display:flex;align-items:center;justify-content:space-between;margin-bottom:.45rem}
148.rung .code{font-family:var(--mono);font-size:.95rem;font-weight:700;color:var(--text-bright)}
149.badge{font-family:var(--mono);font-size:.64rem;letter-spacing:.5px;text-transform:uppercase;padding:.22rem .55rem;border-radius:5px}
150.badge.live{background:rgba(63,185,80,.14);color:var(--green)}
151.badge.partial{background:rgba(240,166,74,.14);color:var(--orange)}
152.badge.path{background:rgba(88,166,255,.14);color:var(--cyan)}
153.badge.future{background:rgba(255,255,255,.05);color:var(--text-dim)}
154.badge.never{background:rgba(244,112,103,.14);color:var(--red)}
155.rung .name{font-size:.95rem;color:var(--text);font-weight:550;margin-bottom:.35rem}
156.rung .desc{font-size:.84rem;color:var(--text-dim)}
157
158/* ── Tracks ───────────────────────────────────────────────────── */
159.tracks{display:grid;grid-template-columns:repeat(auto-fit,minmax(280px,1fr));gap:1.1rem}
160.track{
161 background:linear-gradient(180deg,var(--card-hi),var(--card));
162 border:1px solid var(--card-border);border-radius:var(--radius);padding:1.6rem;position:relative;overflow:hidden;
163}
164.track::before{content:'';position:absolute;top:0;left:0;right:0;height:3px}
165.track.a::before{background:linear-gradient(90deg,var(--green),transparent)}
166.track.b::before{background:linear-gradient(90deg,var(--purple),transparent)}
167.track.c::before{background:linear-gradient(90deg,var(--accent),transparent)}
168.track .label{font-family:var(--mono);font-size:.7rem;letter-spacing:1px;text-transform:uppercase;color:var(--text-faint);margin-bottom:.4rem}
169.track h3{font-size:1.1rem;color:var(--text-bright);font-weight:600;margin-bottom:.3rem}
170.track .ver{font-family:var(--mono);font-size:.82rem;color:var(--cyan);margin-bottom:.9rem}
171.track p{font-size:.9rem;color:var(--text-dim);margin-bottom:.8rem}
172.track ul{list-style:none;font-size:.85rem;color:var(--text)}
173.track ul li{padding:.2rem 0;padding-left:1rem;position:relative}
174.track ul li::before{content:'›';position:absolute;left:0;color:var(--text-faint)}
175
176/* ── Roadmap ──────────────────────────────────────────────────── */
177.road{position:relative;padding-left:2.2rem}
178.road::before{content:'';position:absolute;left:.5rem;top:.4rem;bottom:.4rem;width:2px;background:var(--card-border)}
179.ship{position:relative;padding:1.1rem 0 1.1rem 0;border-bottom:1px solid var(--card-border)}
180.ship:last-child{border-bottom:none}
181.ship::before{
182 content:'';position:absolute;left:-1.85rem;top:1.45rem;width:14px;height:14px;border-radius:50%;
183 background:var(--bg);border:2px solid var(--card-border-hi);
184}
185.ship.now::before{background:var(--accent);border-color:var(--accent);box-shadow:0 0 10px rgba(245,200,66,.5)}
186.ship.next::before{border-color:var(--cyan)}
187.ship .row{display:flex;align-items:baseline;gap:.7rem;flex-wrap:wrap;margin-bottom:.3rem}
188.ship .tag{font-family:var(--mono);font-size:.72rem;font-weight:600;color:var(--text-bright);
189 background:rgba(255,255,255,.05);padding:.2rem .55rem;border-radius:6px}
190.ship.now .tag{background:rgba(245,200,66,.16);color:var(--accent)}
191.ship .when{font-size:.82rem;color:var(--text-faint)}
192.ship h4{font-size:1rem;color:var(--text-bright);font-weight:550;margin-bottom:.3rem}
193.ship p{font-size:.9rem;color:var(--text-dim);max-width:680px}
194
195/* ── Non-claims callout ───────────────────────────────────────── */
196.nonclaim{
197 background:rgba(244,112,103,.04);border:1px solid rgba(244,112,103,.22);border-radius:var(--radius);
198 padding:1.8rem;
199}
200.nonclaim h3{color:var(--red);font-size:1.05rem;font-weight:600;margin-bottom:1rem;display:flex;align-items:center;gap:.5rem}
201.nc-list{display:grid;grid-template-columns:repeat(auto-fit,minmax(300px,1fr));gap:1rem}
202.nc{background:var(--card);border:1px solid var(--card-border);border-radius:var(--radius-sm);padding:1rem 1.1rem}
203.nc .dont{font-family:var(--mono);font-size:.83rem;color:var(--red);margin-bottom:.3rem}
204.nc .why{font-size:.85rem;color:var(--text-dim)}
205
206/* ── Footer ───────────────────────────────────────────────────── */
207footer{border-top:1px solid var(--card-border);padding:2.5rem 0;margin-top:3rem;color:var(--text-dim);font-size:.85rem}
208.foot-grid{display:flex;justify-content:space-between;gap:1.5rem;flex-wrap:wrap}
209.foot-grid a{color:var(--cyan)}
210.spark-foot{display:flex;align-items:center;gap:.55rem;color:var(--text)}
211.spark-foot .spark{width:20px;height:20px}
212
213@media(max-width:560px){
214 .hero{padding:3.5rem 0 2.5rem}
215 section{padding:2.6rem 0}
216}
217</style>
218</head>
219<body>
220
221<!-- ─── Top bar ─── -->
222<header class="topbar">
223 <div class="wrap topbar-inner">
224 <div class="brand">
225 <svg class="spark" viewBox="0 0 100 100" xmlns="http://www.w3.org/2000/svg"><rect width="100" height="100" rx="14" fill="#16213e"/><path d="M50 10 C 54 38 60 44 88 50 C 60 56 54 62 50 90 C 46 62 40 56 12 50 C 40 44 46 38 50 10 Z" fill="#f5c842"/></svg>
226 <span class="brand-name">HiveRelay</span>
227 <span class="brand-sub">0.25</span>
228 </div>
229 <nav class="topbar-nav">
230 <a href="#standing-up">Features</a>
231 <a href="#capabilities">Capabilities</a>
232 <a href="#tracks">Tracks</a>
233 <a href="#whats-next">Roadmap</a>
234 </nav>
235 </div>
236</header>
237
238<!-- ─── Hero ─── -->
239<div class="wrap">
240 <section class="hero">
241 <span class="eyebrow"><span class="dot"></span> Canary · v0.25.0-rc.4</span>
242 <h1>The utility relay,<br><span class="grad">made useful — honestly.</span></h1>
243 <p class="lead">HiveRelay 0.25 turns the raw DHT seeder into a real substrate operators and apps can rely on: Tor + utility services on by default, app-reachable service HTTP, fleet stability hardening — and a published ladder that keeps every claim bounded.</p>
244 <div class="chans">
245 <span class="chip canary"><span class="dot"></span> Canary → rc.4</span>
246 <span class="chip stable"><span class="dot"></span> Stable → 0.24.3</span>
247 <span class="chip track"><span class="dot"></span> Track A (fleet)</span>
248 </div>
249 <div class="cta-row">
250 <a class="btn btn-primary" href="https://github.com/bigdestiny2/P2P-Hiverelay/releases" target="_blank" rel="noopener">View release</a>
251 <a class="btn btn-ghost" href="#standing-up">See what's new ↓</a>
252 </div>
253 </section>
254
255 <!-- Status strip -->
256 <div class="status-strip">
257 <div class="stat"><div class="k">Fleet healthy</div><div class="v green">12 / 13</div><div class="sub">bern offline (infra, not process)</div></div>
258 <div class="stat"><div class="k">Canary tag</div><div class="v accent">0.25.0-rc.4</div><div class="sub">signed, health-gated updater</div></div>
259 <div class="stat"><div class="k">Stable tag</div><div class="v">0.24.3</div><div class="sub">unchanged through soak</div></div>
260 <div class="stat"><div class="k">utah-8gb RSS</div><div class="v green">~110 MB</div><div class="sub">down from 5.5 GB after remigrate</div></div>
261 </div>
262</div>
263
264<!-- ─── What's being stood up ─── -->
265<div class="wrap">
266<section id="standing-up">
267 <div class="sec-head">
268 <div class="sec-kicker">What we're standing up</div>
269 <h2>Ten workstreams, one coherent ship</h2>
270 <p>0.25 isn't a single feature — it's the wiring that turns a bare seeder into an always-on utility relay apps can call, operators can monitor, and the network can depend on.</p>
271 </div>
272
273 <div class="grid">
274
275 <div class="card h-accent">
276 <div class="ico">🧅</div>
277 <h3>Tor as a packaging default</h3>
278 <p>Docker Compose, the systemd unit, and the Umbrel appliance now ship with <code>HIVERELAY_TOR=1</code> and full SOCKS / control / key-file wiring. The path to onion-hidden relays is on by default — live endpoints still need host Tor ≥ 0.4.9 with cookie control.</p>
279 <div class="unlocks">
280 <span class="lbl">Capability path</span>
281 <span class="rung">G4-T route separation</span>
282 </div>
283 </div>
284
285 <div class="card h-green">
286 <div class="ico">⚙️</div>
287 <h3>Utility services, on by default</h3>
288 <p>A fresh appliance is no longer an empty seeder. Outboxlog, notify, storage-proof, VRF, witnesslog, and repairticket are enabled in the unit / compose path. <code>shard-store</code> stays opt-in for disks that can carry it.</p>
289 <ul class="tags"><li>outboxlog</li><li>notify</li><li>vrf</li><li>witnesslog</li><li>repairticket</li><li>storage-proof</li></ul>
290 </div>
291
292 <div class="card">
293 <div class="ico">🎲</div>
294 <h3>VRF HTTP surface</h3>
295 <p>Verifiable randomness is now reachable over HTTP for apps that need leaderless sortition, fair ordering, or unbiasable beacons.</p>
296 <code class="endpoint">GET /api/v1/vrf/beacon-info</code><br>
297 <code class="endpoint">GET /api/v1/vrf/beacon-latest</code><br>
298 <code class="endpoint">POST /api/v1/vrf/select</code>
299 <div class="unlocks"><span class="lbl">Unlocks</span><span class="rung">on-chain-style fairness for p2p apps</span></div>
300 </div>
301
302 <div class="card h-purple">
303 <div class="ico">🔔</div>
304 <h3>Notify push provider resolution</h3>
305 <p>The relay-hosted encrypted wake-up service now resolves a webpush descriptor to a live, non-memory provider — and fails closed on bad config so a misconfigured box never silently drops wakes.</p>
306 <div class="unlocks"><span class="lbl">Unlocks</span><span class="rung">asynchronous p2p apps that wake on event</span></div>
307 </div>
308
309 <div class="card">
310 <div class="ico">🧰</div>
311 <h3>Client SDK helpers</h3>
312 <p><code>HiveRelayClient</code> gains first-class methods for the new surfaces — notify send/watch, shard put/get, witness append, and VRF beacon-latest — so app developers stop reimplementing transport.</p>
313 <ul class="tags"><li>notify</li><li>shard</li><li>witness</li><li>vrf</li></ul>
314 </div>
315
316 <div class="card h-orange">
317 <div class="ico">📊</div>
318 <h3>Dashboard panels</h3>
319 <p>Operators get visibility into the two new live surfaces: a notify manage-status aggregate panel and a VRF panel. What the relay is actually doing is now on the dashboard, not buried in logs.</p>
320 <div class="unlocks"><span class="lbl">Unlocks</span><span class="rung">operator observability for G1 services</span></div>
321 </div>
322
323 <div class="card">
324 <div class="ico">🛡️</div>
325 <h3>Outboxlog durability & lifecycle</h3>
326 <p>Journal and torn-tail audit paths are covered, <code>loadRegistry()</code> is split from <code>reseedDrives()</code> so recovery can seal storage-admission before reseed, and bytes-per-day metering is honest under restart.</p>
327 <ul class="tags"><li>journal</li><li>takedown audit</li><li>storage-authority close</li></ul>
328 </div>
329
330 <div class="card h-green">
331 <div class="ico">🩺</div>
332 <h3>Fleet health watchdog</h3>
333 <p>systemd couldn't recover an event-loop hang — the main PID stayed alive, so <code>Restart=</code> never fired. A new local watchdog pings <code>/health</code> every 2 min and SIGKILLs + restarts on two consecutive failures, now rolled across every reachable box.</p>
334 <div class="unlocks"><span class="lbl">Unlocks</span><span class="rung">12/13 fleet healthy vs repeated hangs</span></div>
335 </div>
336
337 <div class="card h-accent">
338 <div class="ico">🧱</div>
339 <h3>utah-8gb remigrate + memory</h3>
340 <p>The 8.6 GB bloated RocksDB was quarantined and rebuilt from <code>cores/</code>; RSS fell from 5.5 GB to ~110 MB. Added 4 GB swap, dropped the thrashing <code>MemoryHigh</code> soft-lock, and kept the full feature plugin set.</p>
341 <ul class="tags"><li>CS7 rebuild</li><li>swap</li><li>MemoryMax=6G</li></ul>
342 </div>
343
344 <div class="card h-purple">
345 <div class="ico">📐</div>
346 <h3>Ladder ship map</h3>
347 <p>The defining artifact of 0.25: <code>docs/LADDER-SHIP-MAP.md</code> documents Track A (fleet), Track B (blind public-test), and Track C (merge) on one chronological honesty ladder, so fleet vs blind never get mashed into one claim.</p>
348 <div class="unlocks"><span class="lbl">Unlocks</span><span class="rung">bounded, evidence-graded marketing</span></div>
349 </div>
350
351 </div>
352</section>
353</div>
354
355<!-- ─── Capabilities unlocked ─── -->
356<div class="wrap">
357<section id="capabilities">
358 <div class="sec-head center">
359 <div class="sec-kicker">The honesty ladder</div>
360 <h2>Capabilities, by rung</h2>
361 <p>Every feature maps to a privacy / integrity rung. We badge only what's evidenced — never what's "coming soon."</p>
362 </div>
363
364 <div class="ladder">
365 <div class="rung live">
366 <div class="top"><span class="code">G0</span><span class="badge live">Live</span></div>
367 <div class="name">Integrity</div>
368 <div class="desc">Can't forge bytes or signatures. Hypercore Merkle roots + signed envelopes. The foundation the whole stack is built on.</div>
369 </div>
370 <div class="rung live">
371 <div class="top"><span class="code">G1</span><span class="badge live">Live</span></div>
372 <div class="name">Payload opacity</div>
373 <div class="desc">Encrypted, signed envelopes; utility services HTTP. <strong>0.25 makes this fleet-wide and app-reachable.</strong></div>
374 </div>
375 <div class="rung partial">
376 <div class="top"><span class="code">G2-S</span><span class="badge partial">Partial</span></div>
377 <div class="name">Storage-schema opacity</div>
378 <div class="desc">Blind cells — no app semantics at rest. Code exists; blind public-test pilot runs in parallel (Track B).</div>
379 </div>
380 <div class="rung partial">
381 <div class="top"><span class="code">G3</span><span class="badge partial">Partial</span></div>
382 <div class="name">At-rest unlinkability</div>
383 <div class="desc">BlindShard / cells + multi-relay cohort. shard-store code complete; not fleet-wide until 0.26.</div>
384 </div>
385 <div class="rung partial">
386 <div class="top"><span class="code">G4-T</span><span class="badge partial">Partial</span></div>
387 <div class="name">Route separation</div>
388 <div class="desc">Tor onion path. <strong>0.25 ships the packaging default</strong>; host deploy + negative-probe still uneven.</div>
389 </div>
390 <div class="rung future">
391 <div class="top"><span class="code">G2-W</span><span class="badge future">Spec'd</span></div>
392 <div class="name">Wire opacity</div>
393 <div class="desc">Split transport (OHTTP + Protomux). Spec exists; code does not. Lands after 0.26.</div>
394 </div>
395 <div class="rung future">
396 <div class="top"><span class="code">G4-I</span><span class="badge future">Research</span></div>
397 <div class="name">Read-interest privacy</div>
398 <div class="desc">Bucket / PIR patterns for "can't tell which post I read." Optional research track, not on the near map.</div>
399 </div>
400 <div class="rung never">
401 <div class="top"><span class="code">G5</span><span class="badge never">Never (public)</span></div>
402 <div class="name">Active-reader resistance</div>
403 <div class="desc">Impossible for public data. We document this on every public explainer rather than claim it.</div>
404 </div>
405 </div>
406</section>
407</div>
408
409<!-- ─── Tracks ─── -->
410<div class="wrap">
411<section id="tracks">
412 <div class="sec-head">
413 <div class="sec-kicker">Two tracks, one ladder</div>
414 <h2>Version tags stay separate until a deliberate merge</h2>
415 <p>0.25 is deliberately scoped. The blind substrate is a parallel public-test version line, not a redefinition of the fleet tag.</p>
416 </div>
417
418 <div class="tracks">
419 <div class="track a">
420 <div class="label">Track A · this release</div>
421 <h3>Fleet channel</h3>
422 <div class="ver">v0.25.x → rc.4 now, 0.25.0 stable after soak</div>
423 <p>All raw fleet relays (systemd / Docker / Umbrel via <code>channels.json</code>).</p>
424 <ul>
425 <li>Tor + utility defaults in packaging</li>
426 <li>App-reachable service HTTP</li>
427 <li>G0 + G1 + partial G3 (shard-store opt-in) + partial G4-T path</li>
428 </ul>
429 </div>
430 <div class="track b">
431 <div class="label">Track B · parallel</div>
432 <h3>Blind public-test</h3>
433 <div class="ver">1.0.0-rc.1.public-test.1</div>
434 <p>syd-1 first, then a second owner-operated failure domain (Dallas / Bern).</p>
435 <ul>
436 <li>blind-edge :443 + blind-daemon (Unix socket)</li>
437 <li>LIVE_PUBLIC_TEST_ONLY lease — no GA, no stable channel</li>
438 <li>Peerit binds as client only after two-FD acceptance</li>
439 </ul>
440 </div>
441 <div class="track c">
442 <div class="label">Track C · later</div>
443 <h3>Merge + promote</h3>
444 <div class="ver">v0.26.x</div>
445 <p>When Track B is accepted and boot-restore lands on main.</p>
446 <ul>
447 <li>blind packages in the monorepo tree</li>
448 <li>opt-in blind-cell relay profile</li>
449 <li>First time "blind" is a fleet version story</li>
450 </ul>
451 </div>
452 </div>
453</section>
454</div>
455
456<!-- ─── What's next ─── -->
457<div class="wrap">
458<section id="whats-next">
459 <div class="sec-head">
460 <div class="sec-kicker">What's next</div>
461 <h2>The ship ladder</h2>
462 <p>A chronological map from the canary we just cut to the rungs we'll never claim. Each ship advances one ladder rung — and earns its badge.</p>
463 </div>
464
465 <div class="road">
466 <div class="ship now">
467 <div class="row"><span class="tag">Ship 1</span><span class="when">now · canary</span></div>
468 <h4>v0.25.0-rc.1 → rc.4 · Fleet canary (Track A)</h4>
469 <p>Service HTTP wiring, dashboard panels, Tor + utilities default-on. Gate to promote: canary green 24–48h, no OOM thrash.</p>
470 </div>
471 <div class="ship next">
472 <div class="row"><span class="tag">Ship 2</span><span class="when">after soak</span></div>
473 <h4>v0.25.0 · Fleet stable</h4>
474 <p>Roll all raw relays. Large disks (utah-8gb, bern, dubai) get optional shard-store → partial G3. Small boxes stay utilities-only.</p>
475 </div>
476 <div class="ship">
477 <div class="row"><span class="tag">Ship 3</span><span class="when">parallel</span></div>
478 <h4>Blind public-test live · syd → dal</h4>
479 <p>DNS + ACME on syd-1, qualify DESCRIBE / CELL / INBOX / CORE / FORWARD, then a second failure domain. Two owner-operated FDs, not independent operators.</p>
480 </div>
481 <div class="ship">
482 <div class="row"><span class="tag">Ship 4</span><span class="when">after Track B accept</span></div>
483 <h4>Boot-restore → main merge</h4>
484 <p><code>packages/blind-daemon</code>, blind-edge, IPC, boot floors, sealed-manifest recovery land on main. Service-http-wiring rebases on top.</p>
485 </div>
486 <div class="ship">
487 <div class="row"><span class="tag">Ship 5–6</span><span class="when">post-merge</span></div>
488 <h4>v0.26.0-rc.1 → 0.26.0 · Blind-capable monorepo</h4>
489 <p>First tag where blind is in the tree. Operators can run a verified blind-cell relay profile on large seeds. Code-complete G0–G3 for the GA path.</p>
490 </div>
491 <div class="ship">
492 <div class="row"><span class="tag">Ship 7</span><span class="when">overlaps late 5–6</span></div>
493 <h4>App-origin HTTPS gateway evidence</h4>
494 <p>Signed-tag digests, two-operator gateway evidence, public read plane honesty. Public content gets an evidenced HTTPS plane; private/blind still fail closed.</p>
495 </div>
496 <div class="ship">
497 <div class="row"><span class="tag">Ship 8</span><span class="when">after Ship 2</span></div>
498 <h4>Tor hardening RC · full G4-T evidence</h4>
499 <p>Persistent onion keys, negative-probe on restricted discovery, 100 MB bulk-over-onion measurement. "Onion-hidden with measured limits" — never mixnet-level anonymity.</p>
500 </div>
501 <div class="ship">
502 <div class="row"><span class="tag">Ship 9</span><span class="when">after 0.26 stable</span></div>
503 <h4>Split transport · G2-W</h4>
504 <p>OHTTP ingress + Protomux data plane per the split-transport spec. Wire opacity under stated non-collusion assumptions.</p>
505 </div>
506 <div class="ship">
507 <div class="row"><span class="tag">Ship 11</span><span class="when">never</span></div>
508 <h4>G5 for public content</h4>
509 <p>Documented as unachievable and badged on every public explainer. Some rungs you earn; this one you respect by not claiming it.</p>
510 </div>
511 </div>
512</section>
513</div>
514
515<!-- ─── Honest non-claims ─── -->
516<div class="wrap">
517<section id="non-claims">
518 <div class="nonclaim">
519 <h3>⚠️ What 0.25 is NOT</h3>
520 <p style="color:var(--text-dim);font-size:.92rem;margin-bottom:1.4rem">We hold ourselves to evidence-graded badges. These are the claims you will <em>not</em> hear from us about this release:</p>
521 <div class="nc-list">
522 <div class="nc">
523 <div class="dont">"Stable fleet is 0.25"</div>
524 <div class="why">Stable remains <strong>0.24.3</strong> until canary soaks green.</div>
525 </div>
526 <div class="nc">
527 <div class="dont">"Tor is live on every relay"</div>
528 <div class="why">Status shows no onion until host Tor + cookie control + upgrade evidence. Packaging is the path, not the proof.</div>
529 </div>
530 <div class="nc">
531 <div class="dont">"Blind substrate is GA / in 0.25"</div>
532 <div class="why">Blind public-test is a <strong>parallel</strong> version line and lease, on ≤2 pilot nodes.</div>
533 </div>
534 <div class="nc">
535 <div class="dont">"The browser is a full DHT peer"</div>
536 <div class="why">Still false. We use honest stack language — the browser is a client of the relay.</div>
537 </div>
538 </div>
539 </div>
540</section>
541</div>
542
543<!-- ─── Footer ─── -->
544<footer>
545 <div class="wrap foot-grid">
546 <div class="spark-foot">
547 <svg class="spark" viewBox="0 0 100 100" xmlns="http://www.w3.org/2000/svg"><rect width="100" height="100" rx="14" fill="#16213e"/><path d="M50 10 C 54 38 60 44 88 50 C 60 56 54 62 50 90 C 46 62 40 56 12 50 C 40 44 46 38 50 10 Z" fill="#f5c842"/></svg>
548 <span>HiveRelay / Blindspark — v0.25 canary</span>
549 </div>
550 <div>
551 <a href="https://github.com/bigdestiny2/P2P-Hiverelay" target="_blank" rel="noopener">GitHub</a> ·
552 <a href="https://github.com/bigdestiny2/P2P-Hiverelay/releases" target="_blank" rel="noopener">Releases</a> ·
553 <a href="#" onclick="return false">Ladder ship map</a>
554 </div>
555 </div>
556</footer>
557
558</body>
559</html>
docs/html/FLEET-OPS-LIVE.html +53 -0
……@@ -0,0 +1,53 @@
1<!doctype html>
2<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
3<title>HiveRelay · Fleet ops dashboard — live</title>
4<style>
5:root{--bg:#0a0e14;--panel:#111821;--ink:#e8eef6;--dim:#8b9bb0;--faint:#5b6b80;--line:#1f2a3a;--line-2:#152028;--accent:#4fd1c5;--accent-2:#5b9fd4;--gold:#e0a54b;--rose:#e0533d;--green:#3ecf8e;--cutover:#22d3ee}
6*{box-sizing:border-box}html,body{margin:0;padding:0;background:var(--bg);color:var(--ink);font:15px/1.55 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif}
7a{color:var(--accent-2)}.mono,code{font-family:ui-monospace,Menlo,Consolas,monospace;font-size:.86em}
8.wrap{max-width:1240px;margin:0 auto;padding:0 1.4em}
9header{padding:2.4em 0 1.6em;border-bottom:1px solid var(--line);background:radial-gradient(ellipse 70% 50% at 15% -10%,rgba(62,207,142,.12),transparent 55%)}
10.eyebrow{color:var(--gold);font-size:.75em;font-weight:700;letter-spacing:.16em;text-transform:uppercase}
11h1{font-size:clamp(1.6em,3.5vw,2.15em);margin:.2em 0;font-weight:800}
12h1 .grad{background:linear-gradient(120deg,var(--gold),var(--green),var(--accent));-webkit-background-clip:text;background-clip:text;color:transparent}
13.lede{color:var(--dim);max-width:70ch}.meta{margin-top:1em;display:flex;flex-wrap:wrap;gap:.45em;font-size:.78em;color:var(--faint)}
14.meta span{padding:.28em .65em;border:1px solid var(--line);border-radius:999px;background:var(--panel)}
15.cards{display:grid;grid-template-columns:repeat(auto-fit,minmax(140px,1fr));gap:.75em;margin:1.4em 0}
16.card{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:.9em 1em}
17.card .label{font-size:.7em;text-transform:uppercase;letter-spacing:.08em;color:var(--faint);font-weight:700}
18.card .value{font-size:1.55em;font-weight:800;margin-top:.15em}
19.live-dot{width:8px;height:8px;border-radius:50%;background:var(--green);display:inline-block;box-shadow:0 0 8px var(--green);animation:pulse 2s ease-in-out infinite;margin-right:.35em}
20@keyframes pulse{0%,100%{opacity:1}50%{opacity:.35}}
21section{padding:1.6em 0 2em}h2{font-size:1.25em;margin:0 0 .8em}
22table{border-collapse:collapse;width:100%;font-size:.88em}th,td{border:1px solid var(--line);padding:.45em .55em;text-align:left;vertical-align:top}
23th{background:#0a0f16;color:var(--dim);font-size:.72em;text-transform:uppercase}
24tr.row-dn td{background:rgba(224,83,61,.04)}.sub{font-size:.78em;color:var(--faint)}.err{color:var(--rose);font-size:.75em}
25.dim{color:var(--faint);font-size:.75em}
26.status{display:inline-block;padding:.1em .45em;border-radius:4px;font-size:.7em;font-weight:700;text-transform:uppercase;border:1px solid var(--line)}
27.s-done{color:var(--green);border-color:#1d3a2c;background:rgba(62,207,142,.1)}
28.s-stale{color:var(--rose);border-color:#3a1d18;background:rgba(224,83,61,.1)}
29.s-off{color:var(--faint);border-color:#2a3340}
30.chip{display:inline-block;padding:.08em .38em;border-radius:4px;font-size:.68em;font-weight:700;margin:.08em .1em;border:1px solid var(--line);font-family:ui-monospace,Menlo,monospace}
31.chip.on{color:var(--green);border-color:#1d3a2c;background:rgba(62,207,142,.12)}.chip.off{color:#3a4455;border-color:#1a2230}
32.ver-025{color:var(--cutover);font-weight:700}.feats{line-height:1.7;max-width:28em}
33footer{padding:1.5em 0 2.5em;color:var(--faint);font-size:.82em}
34</style></head><body>
35<header><div class="wrap">
36<p class="eyebrow"><span class="live-dot"></span> Live fleet · recovered dubai + utah-8gb</p>
37<h1>HiveRelay <span class="grad">fleet ops</span> dashboard</h1>
38<p class="lede">Health, version, Tor host, and utility feature matrix. Healthy <strong>11/12</strong> · down: <strong>bern only</strong>.</p>
39<div class="meta"><span>Probed 2026-07-27 10:17 UTC</span><span>10/12 healthy</span><span>0.25: 2</span><span>Tor host: 3</span></div>
40</div></header>
41<main class="wrap">
42<div class="cards">
43<div class="card"><div class="label">Total</div><div class="value">12</div></div>
44<div class="card"><div class="label">Healthy</div><div class="value" style="color:var(--green)">11</div></div>
45<div class="card"><div class="label">Down</div><div class="value" style="color:var(--rose)">1</div></div>
46<div class="card"><div class="label">On 0.25</div><div class="value" style="color:var(--cutover)">2</div></div>
47<div class="card"><div class="label">Tor host</div><div class="value">3</div></div>
48</div>
49<section><h2>Fleet matrix</h2>
50<table><thead><tr><th>Box</th><th>Region</th><th>Channel</th><th>Health</th><th>Version</th><th>Tor</th><th>Disk</th><th>Apps</th><th>Conn</th><th>Features</th></tr></thead>
51<tbody><tr class="row-up"><td><strong>amsterdam</strong><div class="sub mono">153.75.89.206</div></td><td>EU</td><td>stable</td><td><span class="status s-done">healthy</span></td><td><code class="ver-ok">0.24.3</code></td><td><span class="status s-off">off</span></td><td class="mono">43%</td><td class="mono">—</td><td class="mono">—</td><td class="feats"><span class="chip on">outbox</span><span class="chip on">poker</span><span class="chip on">vrf</span><span class="chip on">arb</span><span class="chip on">zk</span><span class="chip on">notify</span><span class="chip on">s-proof</span><span class="chip on">witness</span><span class="chip on">repair</span><span class="chip on">shard</span></td></tr><tr class="row-up"><td><strong>dubai</strong><div class="sub mono">172.86.76.209</div></td><td>ME</td><td>stable</td><td><span class="status s-done">healthy</span></td><td><code class="ver-ok">0.24.3</code></td><td><span class="status s-done">host</span></td><td class="mono">—</td><td class="mono">—</td><td class="mono">—</td><td class="feats"><span class="chip on">outbox</span><span class="chip on">poker</span><span class="chip on">vrf</span><span class="chip on">arb</span><span class="chip on">zk</span><span class="chip on">notify</span><span class="chip on">s-proof</span><span class="chip on">witness</span><span class="chip on">repair</span><span class="chip on">shard</span></td></tr><tr class="row-up"><td><strong>miami</strong><div class="sub mono">144.172.111.80</div></td><td>NA</td><td>stable</td><td><span class="status s-done">healthy</span></td><td><code class="ver-ok">0.24.3</code></td><td><span class="status s-off">off</span></td><td class="mono">20%</td><td class="mono">—</td><td class="mono">—</td><td class="feats"><span class="chip on">outbox</span><span class="chip on">poker</span><span class="chip on">vrf</span><span class="chip on">arb</span><span class="chip on">zk</span><span class="chip on">notify</span><span class="chip on">s-proof</span><span class="chip on">witness</span><span class="chip on">repair</span><span class="chip off">shard</span></td></tr><tr class="row-up"><td><strong>sing-1</strong><div class="sub mono">153.75.94.187</div></td><td>APAC</td><td>stable</td><td><span class="status s-done">healthy</span></td><td><code class="ver-ok">0.24.3</code></td><td><span class="status s-off">off</span></td><td class="mono">66%</td><td class="mono">—</td><td class="mono">—</td><td class="feats"><span class="chip on">outbox</span><span class="chip on">poker</span><span class="chip on">vrf</span><span class="chip on">arb</span><span class="chip on">zk</span><span class="chip on">notify</span><span class="chip on">s-proof</span><span class="chip on">witness</span><span class="chip on">repair</span><span class="chip off">shard</span></td></tr><tr class="row-up"><td><strong>sing-2</strong><div class="sub mono">153.75.94.144</div></td><td>APAC</td><td>stable</td><td><span class="status s-done">healthy</span></td><td><code class="ver-ok">0.24.3</code></td><td><span class="status s-off">off</span></td><td class="mono">9%</td><td class="mono">—</td><td class="mono">—</td><td class="feats"><span class="chip on">outbox</span><span class="chip on">poker</span><span class="chip on">vrf</span><span class="chip on">arb</span><span class="chip on">zk</span><span class="chip on">notify</span><span class="chip on">s-proof</span><span class="chip on">witness</span><span class="chip on">repair</span><span class="chip off">shard</span></td></tr><tr class="row-up"><td><strong>sydney</strong><div class="sub mono">104.194.135.205</div></td><td>APAC</td><td>stable</td><td><span class="status s-done">healthy</span></td><td><code class="ver-ok">0.24.3</code></td><td><span class="status s-off">off</span></td><td class="mono">46%</td><td class="mono">—</td><td class="mono">—</td><td class="feats"><span class="chip on">outbox</span><span class="chip on">poker</span><span class="chip on">vrf</span><span class="chip on">arb</span><span class="chip on">zk</span><span class="chip on">notify</span><span class="chip on">s-proof</span><span class="chip on">witness</span><span class="chip on">repair</span><span class="chip off">shard</span></td></tr><tr class="row-up"><td><strong>utah</strong><div class="sub mono">144.172.116.110</div></td><td>NA</td><td>canary</td><td><span class="status s-done">healthy</span></td><td><code class="ver-025">0.25.0-rc.4</code></td><td><span class="status s-done">host</span></td><td class="mono">21%</td><td class="mono">—</td><td class="mono">—</td><td class="feats"><span class="chip on">outbox</span><span class="chip on">poker</span><span class="chip on">vrf</span><span class="chip on">arb</span><span class="chip on">zk</span><span class="chip on">notify</span><span class="chip on">s-proof</span><span class="chip on">witness</span><span class="chip on">repair</span><span class="chip on">shard</span></td></tr><tr class="row-up"><td><strong>utah-0.5gb</strong><div class="sub mono">144.172.101.215</div></td><td>NA</td><td>canary</td><td><span class="status s-done">healthy</span></td><td><code class="ver-ok">0.24.3</code></td><td><span class="status s-off">off</span></td><td class="mono">43%</td><td class="mono">—</td><td class="mono">—</td><td class="feats"><span class="chip on">outbox</span><span class="chip off">poker</span><span class="chip off">vrf</span><span class="chip off">arb</span><span class="chip off">zk</span><span class="chip off">notify</span><span class="chip off">s-proof</span><span class="chip off">witness</span><span class="chip off">repair</span><span class="chip off">shard</span></td></tr><tr class="row-up"><td><strong>utah-2gb-a</strong><div class="sub mono">216.126.237.3</div></td><td>NA</td><td>stable</td><td><span class="status s-done">healthy</span></td><td><code class="ver-ok">0.24.3</code></td><td><span class="status s-off">off</span></td><td class="mono">21%</td><td class="mono">—</td><td class="mono">—</td><td class="feats"><span class="chip on">outbox</span><span class="chip on">poker</span><span class="chip on">vrf</span><span class="chip on">arb</span><span class="chip on">zk</span><span class="chip on">notify</span><span class="chip on">s-proof</span><span class="chip on">witness</span><span class="chip on">repair</span><span class="chip off">shard</span></td></tr><tr class="row-up"><td><strong>utah-8gb</strong><div class="sub mono">144.172.97.68</div></td><td>NA</td><td>stable</td><td><span class="status s-done">healthy</span></td><td><code class="ver-025">0.25.0-rc.4</code></td><td><span class="status s-done">host</span></td><td class="mono">—</td><td class="mono">—</td><td class="mono">—</td><td class="feats"><span class="chip on">outbox</span><span class="chip on">poker</span><span class="chip on">vrf</span><span class="chip on">arb</span><span class="chip on">zk</span><span class="chip on">notify</span><span class="chip on">s-proof</span><span class="chip on">witness</span><span class="chip on">repair</span><span class="chip on">shard</span></td></tr><tr class="row-up"><td><strong>utah-us</strong><div class="sub mono">144.172.91.26</div></td><td>NA</td><td>stable</td><td><span class="status s-done">healthy</span></td><td><code class="ver-ok">0.24.3</code></td><td><span class="status s-off">off</span></td><td class="mono">42%</td><td class="mono">—</td><td class="mono">—</td><td class="feats"><span class="chip on">outbox</span><span class="chip on">poker</span><span class="chip on">vrf</span><span class="chip on">arb</span><span class="chip on">zk</span><span class="chip on">notify</span><span class="chip on">s-proof</span><span class="chip on">witness</span><span class="chip on">repair</span><span class="chip off">shard</span></td></tr><tr class="row-dn"><td><strong>bern</strong><div class="sub mono">45.59.123.112</div></td><td>EU</td><td>canary</td><td><span class="status s-stale">down</span></td><td><code class="ver-miss">—</code></td><td><span class="status s-off">off</span></td><td class="mono">—</td><td class="mono">—</td><td class="mono">—</td><td class="feats"><span class="dim">none</span></td></tr></tbody></table></section>
52<footer><p>docs/html/FLEET-OPS-LIVE.html · 2026-07-27 10:17 UTC</p></footer>
53</main></body></html>
docs/html/STATUS-RELEASE-FLEET-PEERIT-2026-07-26.html +4 -4
……@@ -89,13 +89,13 @@
8989 <h1>The <span class="grad">0.25 canary saga</span> — fixed, held, documented</h1>
9090 <p class="lede">
9191 We cut a Track A prerelease, canary crashed, we fixed modules, fixed ETIMEDOUT handling,
92 cut a correct tree as <strong>v0.25.0-rc.4</strong>, then discovered a
93 <strong>production Corestore open hang</strong> on utah. Fleet stays healthy on
94 <strong>0.24.3</strong> with Tor + utilities on two large boxes. Peerit blind bind remains scaffold.
92 cut a correct tree as <strong>v0.25.0-rc.4</strong>, then discovered a production hang from <strong>corrupt partial CS7 <code>db/</code></strong>.
93 <strong>Fixed 2026-07-26 evening:</strong> quarantine bad <code>db/</code>, remigrate from <code>cores/</code>.
94 <strong>utah + utah-8gb live on <code>v0.25.0-rc.4</code></strong> with Tor + utilities. Peerit blind bind remains scaffold.
9595 </p>
9696 <div class="meta">
9797 <span>Updated 2026-07-26 (evening)</span>
98 <span>Live fleet: <code>v0.24.3</code></span>
98 <span>Live cohort: <code>v0.25.0-rc.4</code> (utah, utah-8gb) · rest <code>v0.24.3</code></span>
9999 <span>Channels: canary+stable → <code>v0.24.3</code></span>
100100 <span>Preferred tag for lab: <code>v0.25.0-rc.4</code></span>
101101 </div>
docs/html/fleet-live.json +512 -0
……@@ -0,0 +1,512 @@
1{
2 "generatedAt": "2026-07-27 10:17 UTC",
3 "relays": [
4 {
5 "name": "amsterdam",
6 "ip": "153.75.89.206",
7 "region": "EU",
8 "channel": "stable",
9 "diskGB": 60,
10 "role": "seed",
11 "ssh": true,
12 "health_ok": true,
13 "version": "0.24.3",
14 "running": true,
15 "disk_pct": 43,
16 "git": "v0.24.3",
17 "active": "active",
18 "tor_host": "inactive\ninactive",
19 "tor_transport": null,
20 "dht_relay": null,
21 "services": [
22 "outboxlog",
23 "poker",
24 "vrf",
25 "arbitration",
26 "zk",
27 "notify",
28 "storage-proof",
29 "witnesslog",
30 "repairticket",
31 "shard-store"
32 ],
33 "cfg_plugins": [
34 "outboxlog",
35 "poker",
36 "vrf",
37 "arbitration",
38 "zk",
39 "notify",
40 "storage-proof",
41 "witnesslog",
42 "repairticket",
43 "shard-store"
44 ],
45 "seededApps": null,
46 "connections": null,
47 "error": null,
48 "services_source": "config"
49 },
50 {
51 "name": "dubai",
52 "ip": "172.86.76.209",
53 "region": "ME",
54 "channel": "stable",
55 "diskGB": 120,
56 "role": "seed",
57 "ssh": true,
58 "health_ok": true,
59 "version": "0.24.3",
60 "running": true,
61 "disk_pct": null,
62 "git": "v0.24.3",
63 "active": "active",
64 "tor_host": "active",
65 "tor_transport": null,
66 "dht_relay": null,
67 "services": [
68 "outboxlog",
69 "poker",
70 "vrf",
71 "arbitration",
72 "zk",
73 "notify",
74 "storage-proof",
75 "witnesslog",
76 "repairticket",
77 "shard-store"
78 ],
79 "cfg_plugins": [
80 "outboxlog",
81 "poker",
82 "vrf",
83 "arbitration",
84 "zk",
85 "notify",
86 "storage-proof",
87 "witnesslog",
88 "repairticket",
89 "shard-store"
90 ],
91 "seededApps": null,
92 "connections": null,
93 "error": null,
94 "services_source": "config"
95 },
96 {
97 "name": "miami",
98 "ip": "144.172.111.80",
99 "region": "NA",
100 "channel": "stable",
101 "diskGB": 20,
102 "role": "seed",
103 "ssh": true,
104 "health_ok": true,
105 "version": "0.24.3",
106 "running": true,
107 "disk_pct": 20,
108 "git": "v0.24.3",
109 "active": "active",
110 "tor_host": "inactive\ninactive",
111 "tor_transport": null,
112 "dht_relay": null,
113 "services": [
114 "outboxlog",
115 "poker",
116 "vrf",
117 "arbitration",
118 "zk",
119 "notify",
120 "storage-proof",
121 "witnesslog",
122 "repairticket"
123 ],
124 "cfg_plugins": [
125 "outboxlog",
126 "poker",
127 "vrf",
128 "arbitration",
129 "zk",
130 "notify",
131 "storage-proof",
132 "witnesslog",
133 "repairticket"
134 ],
135 "seededApps": null,
136 "connections": null,
137 "error": null,
138 "services_source": "config"
139 },
140 {
141 "name": "sing-1",
142 "ip": "153.75.94.187",
143 "region": "APAC",
144 "channel": "stable",
145 "diskGB": 25,
146 "role": "seed",
147 "ssh": true,
148 "health_ok": true,
149 "version": "0.24.3",
150 "running": true,
151 "disk_pct": 66,
152 "git": "v0.24.3",
153 "active": "active",
154 "tor_host": "inactive\ninactive",
155 "tor_transport": null,
156 "dht_relay": null,
157 "services": [
158 "outboxlog",
159 "poker",
160 "vrf",
161 "arbitration",
162 "zk",
163 "notify",
164 "storage-proof",
165 "witnesslog",
166 "repairticket"
167 ],
168 "cfg_plugins": [
169 "outboxlog",
170 "poker",
171 "vrf",
172 "arbitration",
173 "zk",
174 "notify",
175 "storage-proof",
176 "witnesslog",
177 "repairticket"
178 ],
179 "seededApps": null,
180 "connections": null,
181 "error": null,
182 "services_source": "config"
183 },
184 {
185 "name": "sing-2",
186 "ip": "153.75.94.144",
187 "region": "APAC",
188 "channel": "stable",
189 "diskGB": 65,
190 "role": "seed",
191 "ssh": true,
192 "health_ok": true,
193 "version": "0.24.3",
194 "running": true,
195 "disk_pct": 9,
196 "git": "v0.24.3",
197 "active": "active",
198 "tor_host": "inactive\ninactive",
199 "tor_transport": null,
200 "dht_relay": null,
201 "services": [
202 "outboxlog",
203 "poker",
204 "vrf",
205 "arbitration",
206 "zk",
207 "notify",
208 "storage-proof",
209 "witnesslog",
210 "repairticket"
211 ],
212 "cfg_plugins": [
213 "outboxlog",
214 "poker",
215 "vrf",
216 "arbitration",
217 "zk",
218 "notify",
219 "storage-proof",
220 "witnesslog",
221 "repairticket"
222 ],
223 "seededApps": null,
224 "connections": null,
225 "error": null,
226 "services_source": "config"
227 },
228 {
229 "name": "sydney",
230 "ip": "104.194.135.205",
231 "region": "APAC",
232 "channel": "stable",
233 "diskGB": 60,
234 "role": "seed",
235 "ssh": true,
236 "health_ok": true,
237 "version": "0.24.3",
238 "running": true,
239 "disk_pct": 46,
240 "git": "v0.24.3",
241 "active": "active",
242 "tor_host": "inactive\ninactive",
243 "tor_transport": null,
244 "dht_relay": null,
245 "services": [
246 "outboxlog",
247 "poker",
248 "vrf",
249 "arbitration",
250 "zk",
251 "notify",
252 "storage-proof",
253 "witnesslog",
254 "repairticket"
255 ],
256 "cfg_plugins": [
257 "outboxlog",
258 "poker",
259 "vrf",
260 "arbitration",
261 "zk",
262 "notify",
263 "storage-proof",
264 "witnesslog",
265 "repairticket"
266 ],
267 "seededApps": null,
268 "connections": null,
269 "error": null,
270 "services_source": "config"
271 },
272 {
273 "name": "utah",
274 "ip": "144.172.116.110",
275 "region": "NA",
276 "channel": "canary",
277 "diskGB": 350,
278 "role": "canary",
279 "ssh": true,
280 "health_ok": true,
281 "version": "0.25.0-rc.4",
282 "running": true,
283 "disk_pct": 21,
284 "git": "v0.25.0-rc.4",
285 "active": "active",
286 "tor_host": "active",
287 "tor_transport": null,
288 "dht_relay": null,
289 "services": [
290 "outboxlog",
291 "poker",
292 "vrf",
293 "arbitration",
294 "zk",
295 "notify",
296 "storage-proof",
297 "witnesslog",
298 "repairticket",
299 "shard-store"
300 ],
301 "cfg_plugins": [
302 "outboxlog",
303 "poker",
304 "vrf",
305 "arbitration",
306 "zk",
307 "notify",
308 "storage-proof",
309 "witnesslog",
310 "repairticket",
311 "shard-store"
312 ],
313 "seededApps": null,
314 "connections": null,
315 "error": null,
316 "services_source": "config"
317 },
318 {
319 "name": "utah-0.5gb",
320 "ip": "144.172.101.215",
321 "region": "NA",
322 "channel": "canary",
323 "diskGB": 20,
324 "role": "canary",
325 "ssh": true,
326 "health_ok": true,
327 "version": "0.24.3",
328 "running": true,
329 "disk_pct": 43,
330 "git": "v0.24.3",
331 "active": "active",
332 "tor_host": "inactive",
333 "tor_transport": null,
334 "dht_relay": null,
335 "services": [
336 "outboxlog"
337 ],
338 "cfg_plugins": [
339 "outboxlog"
340 ],
341 "seededApps": null,
342 "connections": null,
343 "error": null,
344 "services_source": "live"
345 },
346 {
347 "name": "utah-2gb-a",
348 "ip": "216.126.237.3",
349 "region": "NA",
350 "channel": "stable",
351 "diskGB": 60,
352 "role": "seed",
353 "ssh": true,
354 "health_ok": true,
355 "version": "0.24.3",
356 "running": true,
357 "disk_pct": 21,
358 "git": "v0.24.3",
359 "active": "active",
360 "tor_host": "inactive\ninactive",
361 "tor_transport": null,
362 "dht_relay": null,
363 "services": [
364 "outboxlog",
365 "poker",
366 "vrf",
367 "arbitration",
368 "zk",
369 "notify",
370 "storage-proof",
371 "witnesslog",
372 "repairticket"
373 ],
374 "cfg_plugins": [
375 "outboxlog",
376 "poker",
377 "vrf",
378 "arbitration",
379 "zk",
380 "notify",
381 "storage-proof",
382 "witnesslog",
383 "repairticket"
384 ],
385 "seededApps": null,
386 "connections": null,
387 "error": null,
388 "services_source": "config"
389 },
390 {
391 "name": "utah-8gb",
392 "ip": "144.172.97.68",
393 "region": "NA",
394 "channel": "stable",
395 "diskGB": 960,
396 "role": "seed",
397 "ssh": true,
398 "health_ok": true,
399 "version": "0.25.0-rc.4",
400 "running": true,
401 "disk_pct": null,
402 "git": "v0.25.0-rc.4",
403 "active": "active",
404 "tor_host": "active",
405 "tor_transport": null,
406 "dht_relay": null,
407 "services": [
408 "outboxlog",
409 "poker",
410 "vrf",
411 "arbitration",
412 "zk",
413 "notify",
414 "storage-proof",
415 "witnesslog",
416 "repairticket",
417 "shard-store"
418 ],
419 "cfg_plugins": [
420 "outboxlog",
421 "poker",
422 "vrf",
423 "arbitration",
424 "zk",
425 "notify",
426 "storage-proof",
427 "witnesslog",
428 "repairticket",
429 "shard-store"
430 ],
431 "seededApps": null,
432 "connections": null,
433 "error": null,
434 "services_source": "config"
435 },
436 {
437 "name": "utah-us",
438 "ip": "144.172.91.26",
439 "region": "NA",
440 "channel": "stable",
441 "diskGB": 58,
442 "role": "seed",
443 "ssh": true,
444 "health_ok": true,
445 "version": "0.24.3",
446 "running": true,
447 "disk_pct": 42,
448 "git": "v0.24.3",
449 "active": "active",
450 "tor_host": "inactive\ninactive",
451 "tor_transport": null,
452 "dht_relay": null,
453 "services": [
454 "outboxlog",
455 "poker",
456 "vrf",
457 "arbitration",
458 "zk",
459 "notify",
460 "storage-proof",
461 "witnesslog",
462 "repairticket"
463 ],
464 "cfg_plugins": [
465 "outboxlog",
466 "poker",
467 "vrf",
468 "arbitration",
469 "zk",
470 "notify",
471 "storage-proof",
472 "witnesslog",
473 "repairticket"
474 ],
475 "seededApps": null,
476 "connections": null,
477 "error": null,
478 "services_source": "config"
479 },
480 {
481 "name": "bern",
482 "ip": "45.59.123.112",
483 "region": "EU",
484 "channel": "canary",
485 "diskGB": 500,
486 "role": "seed",
487 "ssh": false,
488 "health_ok": false,
489 "version": null,
490 "running": null,
491 "disk_pct": null,
492 "git": null,
493 "active": null,
494 "tor_host": null,
495 "tor_transport": null,
496 "dht_relay": null,
497 "services": [],
498 "cfg_plugins": [],
499 "seededApps": null,
500 "connections": null,
501 "error": "ssh/unreachable"
502 }
503 ],
504 "summary": {
505 "total": 12,
506 "up": 11,
507 "down": 1,
508 "v025": 2,
509 "tor_host": 3,
510 "tor_transport": 0
511 }
512}
fleet/README.md +4 -0
……@@ -30,6 +30,10 @@ fleet/channels.json ──(raw.githubusercontent)──▶ each box's updater
3030|---|---|
3131| `channels.json` | **The control plane.** Target tag per channel. Edit + commit to release. |
3232| `relays.json` | Inventory: the boxes, their channel, SSH key, tailnet name. |
33| `relays.local.json` | Local inventory with real SSH key paths (gitignored). Preferred by Admin Operator. |
34| `../dashboard-admin-operator/` | **Dashboard Admin Operator** — live manage UI (`npm run dashboard:admin-operator`). |
35| `health-watchdog.sh` + `hiverelay-health-watchdog.{service,timer}` | Local timer: if `/health` fails twice, SIGKILL+restart (fixes event-loop hangs systemd cannot see). |
36| `install-health-watchdog.sh` | Install the timer on a box (`ssh root@box 'bash -s' < fleet/install-health-watchdog.sh`). |
3337| `updater.sh` | The agent. Resolve target → checkout → restart → health-gate → rollback. |
3438| `hiverelay-updater.{service,timer}` | systemd units (every 15 min, 5 min jitter). |
3539| `install-updater.sh` | Install the agent on a box, set its channel. |
fleet/health-watchdog.sh +73 -0
……@@ -0,0 +1,73 @@
1#!/usr/bin/env bash
2# health-watchdog.sh — local systemd timer target.
3# If HiveRelay is "active" but /health fails repeatedly, force-kill and restart.
4# Covers event-loop hangs (LISTEN on :9100, no response) that systemd cannot see.
5#
6# State: /run/hiverelay-health-watchdog.failcount
7# Env (optional drop-in):
8# HIVERELAY_HEALTH_URL default http://127.0.0.1:9100/health
9# HIVERELAY_HEALTH_TIMEOUT curl max-time seconds (default 5)
10# HIVERELAY_HEALTH_FAILS consecutive failures before restart (default 2)
11set -uo pipefail
12
13URL="${HIVERELAY_HEALTH_URL:-http://127.0.0.1:9100/health}"
14TIMEOUT="${HIVERELAY_HEALTH_TIMEOUT:-5}"
15NEED_FAILS="${HIVERELAY_HEALTH_FAILS:-2}"
16STATE_FILE="${HIVERELAY_HEALTH_STATE:-/run/hiverelay-health-watchdog.failcount}"
17UNIT="${HIVERELAY_UNIT:-hiverelay}"
18LOG_TAG="hiverelay-health-watchdog"
19
20log() { logger -t "$LOG_TAG" -- "$*"; echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) $*"; }
21
22active=$(systemctl is-active "$UNIT" 2>/dev/null || echo inactive)
23if [ "$active" != "active" ] && [ "$active" != "activating" ]; then
24 # Unit not expected to answer health; clear streak
25 echo 0 >"$STATE_FILE" 2>/dev/null || true
26 exit 0
27fi
28
29# Skip while still starting (boot grace)
30MainPID=$(systemctl show "$UNIT" -p MainPID --value 2>/dev/null || echo 0)
31if [ -n "$MainPID" ] && [ "$MainPID" != "0" ] && [ -r "/proc/$MainPID" ]; then
32 # seconds of process life
33 start_ticks=$(awk '{print $22}' /proc/"$MainPID"/stat 2>/dev/null || echo 0)
34 hz=$(getconf CLK_TCK 2>/dev/null || echo 100)
35 uptime_s=$(awk '{print int($1)}' /proc/uptime)
36 start_s=$(( start_ticks / hz ))
37 # approximate process age from boot - starttime is jiffies since boot
38 age=$(( uptime_s - start_s ))
39 if [ "$age" -ge 0 ] && [ "$age" -lt 90 ]; then
40 log "grace period age=${age}s pid=$MainPID — skip"
41 exit 0
42 fi
43fi
44
45body=$(curl -fsS --max-time "$TIMEOUT" "$URL" 2>/dev/null || true)
46if echo "$body" | grep -q '"ok"[[:space:]]*:[[:space:]]*true'; then
47 echo 0 >"$STATE_FILE"
48 exit 0
49fi
50
51fails=0
52if [ -f "$STATE_FILE" ]; then
53 fails=$(cat "$STATE_FILE" 2>/dev/null || echo 0)
54fi
55fails=$((fails + 1))
56echo "$fails" >"$STATE_FILE"
57log "health FAIL $fails/$NEED_FAILS body=${body:0:80}"
58
59if [ "$fails" -lt "$NEED_FAILS" ]; then
60 exit 0
61fi
62
63log "forcing restart of $UNIT after $fails consecutive health failures"
64# Force-kill hung event loops (SIGTERM often does nothing when loop is stuck)
65systemctl kill -s SIGKILL "$UNIT" 2>/dev/null || true
66sleep 1
67fuser -k -9 9100/tcp 2>/dev/null || true
68sleep 1
69systemctl reset-failed "$UNIT" 2>/dev/null || true
70systemctl start "$UNIT" 2>/dev/null || systemctl restart "$UNIT" 2>/dev/null || true
71echo 0 >"$STATE_FILE"
72log "restart issued"
73exit 0
fleet/hiverelay-health-watchdog.service +11 -0
……@@ -0,0 +1,11 @@
1[Unit]
2Description=HiveRelay health watchdog (one-shot probe)
3After=network-online.target
4Wants=network-online.target
5
6[Service]
7Type=oneshot
8ExecStart=/usr/local/bin/hiverelay-health-watchdog.sh
9Nice=10
10# Do not block the host if curl hangs longer than expected
11TimeoutStartSec=30
fleet/hiverelay-health-watchdog.timer +13 -0
……@@ -0,0 +1,13 @@
1[Unit]
2Description=HiveRelay health watchdog every 2 minutes
3Requires=hiverelay-health-watchdog.service
4
5[Timer]
6OnBootSec=3min
7OnUnitActiveSec=2min
8AccuracySec=30s
9Persistent=true
10Unit=hiverelay-health-watchdog.service
11
12[Install]
13WantedBy=timers.target
fleet/install-health-watchdog.sh +100 -0
……@@ -0,0 +1,100 @@
1#!/usr/bin/env bash
2# install-health-watchdog.sh — install local health restart timer on a relay box.
3# Run as root on the box, or via: ssh root@box 'bash -s' < fleet/install-health-watchdog.sh
4set -euo pipefail
5
6SCRIPT_SRC="${1:-}"
7if [ -n "$SCRIPT_SRC" ] && [ -f "$SCRIPT_SRC" ]; then
8 install -m 0755 "$SCRIPT_SRC" /usr/local/bin/hiverelay-health-watchdog.sh
9else
10 # Embedded install when piped over SSH without file args
11 cat > /usr/local/bin/hiverelay-health-watchdog.sh <<'WATCH'
12#!/usr/bin/env bash
13set -uo pipefail
14URL="${HIVERELAY_HEALTH_URL:-http://127.0.0.1:9100/health}"
15TIMEOUT="${HIVERELAY_HEALTH_TIMEOUT:-5}"
16NEED_FAILS="${HIVERELAY_HEALTH_FAILS:-2}"
17STATE_FILE="${HIVERELAY_HEALTH_STATE:-/run/hiverelay-health-watchdog.failcount}"
18UNIT="${HIVERELAY_UNIT:-hiverelay}"
19LOG_TAG="hiverelay-health-watchdog"
20log() { logger -t "$LOG_TAG" -- "$*"; echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) $*"; }
21active=$(systemctl is-active "$UNIT" 2>/dev/null || echo inactive)
22if [ "$active" != "active" ] && [ "$active" != "activating" ]; then
23 echo 0 >"$STATE_FILE" 2>/dev/null || true
24 exit 0
25fi
26MainPID=$(systemctl show "$UNIT" -p MainPID --value 2>/dev/null || echo 0)
27if [ -n "$MainPID" ] && [ "$MainPID" != "0" ] && [ -r "/proc/$MainPID" ]; then
28 start_ticks=$(awk '{print $22}' /proc/"$MainPID"/stat 2>/dev/null || echo 0)
29 hz=$(getconf CLK_TCK 2>/dev/null || echo 100)
30 uptime_s=$(awk '{print int($1)}' /proc/uptime)
31 start_s=$(( start_ticks / hz ))
32 age=$(( uptime_s - start_s ))
33 if [ "$age" -ge 0 ] && [ "$age" -lt 90 ]; then
34 log "grace period age=${age}s pid=$MainPID — skip"
35 exit 0
36 fi
37fi
38body=$(curl -fsS --max-time "$TIMEOUT" "$URL" 2>/dev/null || true)
39if echo "$body" | grep -q '"ok"[[:space:]]*:[[:space:]]*true'; then
40 echo 0 >"$STATE_FILE"
41 exit 0
42fi
43fails=0
44if [ -f "$STATE_FILE" ]; then
45 fails=$(cat "$STATE_FILE" 2>/dev/null || echo 0)
46fi
47fails=$((fails + 1))
48echo "$fails" >"$STATE_FILE"
49log "health FAIL $fails/$NEED_FAILS body=${body:0:80}"
50if [ "$fails" -lt "$NEED_FAILS" ]; then
51 exit 0
52fi
53log "forcing restart of $UNIT after $fails consecutive health failures"
54systemctl kill -s SIGKILL "$UNIT" 2>/dev/null || true
55sleep 1
56fuser -k -9 9100/tcp 2>/dev/null || true
57sleep 1
58systemctl reset-failed "$UNIT" 2>/dev/null || true
59systemctl start "$UNIT" 2>/dev/null || systemctl restart "$UNIT" 2>/dev/null || true
60echo 0 >"$STATE_FILE"
61log "restart issued"
62exit 0
63WATCH
64 chmod 0755 /usr/local/bin/hiverelay-health-watchdog.sh
65fi
66
67cat > /etc/systemd/system/hiverelay-health-watchdog.service <<'EOF'
68[Unit]
69Description=HiveRelay health watchdog (one-shot probe)
70After=network-online.target
71Wants=network-online.target
72
73[Service]
74Type=oneshot
75ExecStart=/usr/local/bin/hiverelay-health-watchdog.sh
76Nice=10
77TimeoutStartSec=30
78EOF
79
80cat > /etc/systemd/system/hiverelay-health-watchdog.timer <<'EOF'
81[Unit]
82Description=HiveRelay health watchdog every 2 minutes
83Requires=hiverelay-health-watchdog.service
84
85[Timer]
86OnBootSec=3min
87OnUnitActiveSec=2min
88AccuracySec=30s
89Persistent=true
90Unit=hiverelay-health-watchdog.service
91
92[Install]
93WantedBy=timers.target
94EOF
95
96systemctl daemon-reload
97systemctl enable --now hiverelay-health-watchdog.timer
98systemctl start hiverelay-health-watchdog.service || true
99echo "installed hiverelay-health-watchdog.timer"
100systemctl list-timers --all | grep hiverelay-health || true
scripts/local-fleet-dashboard.mjs +124 -0
……@@ -0,0 +1,124 @@
1#!/usr/bin/env node
2/**
3 * Local operator proxy for the HiveRelay fleet dashboard.
4 *
5 * Serves dashboard HTML with an injected management token and proxies
6 * /api/* + static dashboard assets to a remote relay. Keeps the API key
7 * off the public :9100 surface (do not enable ui.exposeToken on open VPS).
8 *
9 * HIVERELAY_DASHBOARD_UPSTREAM=http://144.172.116.110:9100 \
10 * HIVERELAY_API_KEY=... \
11 * node scripts/local-fleet-dashboard.mjs
12 *
13 * Then open http://127.0.0.1:9191/fleet
14 */
15import http from 'node:http'
16import { readFileSync, existsSync } from 'node:fs'
17import { join, extname } from 'node:path'
18import { fileURLToPath } from 'node:url'
19import { dirname } from 'node:path'
20
21const __dirname = dirname(fileURLToPath(import.meta.url))
22const ROOT = join(__dirname, '..')
23const DASH = join(ROOT, 'dashboard')
24
25const PORT = Number(process.env.HIVERELAY_DASHBOARD_PORT || 9191)
26const UPSTREAM = (process.env.HIVERELAY_DASHBOARD_UPSTREAM || 'http://144.172.116.110:9100').replace(/\/$/, '')
27const API_KEY = process.env.HIVERELAY_API_KEY || ''
28
29if (!API_KEY) {
30 console.error('HIVERELAY_API_KEY is required')
31 process.exit(1)
32}
33
34const MIME = {
35 '.html': 'text/html; charset=utf-8',
36 '.js': 'application/javascript; charset=utf-8',
37 '.css': 'text/css; charset=utf-8',
38 '.json': 'application/json',
39 '.svg': 'image/svg+xml',
40 '.png': 'image/png',
41 '.ico': 'image/x-icon'
42}
43
44function injectToken (html) {
45 const tag = `<meta name="hiverelay-ui-token" content="${API_KEY.replace(/"/g, '')}">`
46 if (html.includes('hiverelay-ui-token')) {
47 return html.replace(/<meta name="hiverelay-ui-token"[^>]*>/i, tag)
48 }
49 if (html.includes('</head>')) return html.replace('</head>', ` ${tag}\n</head>`)
50 return tag + html
51}
52
53function mapPath (urlPath) {
54 // Relay serves /fleet, /dashboard, /network, … — map to dashboard/*.html
55 const p = urlPath.split('?')[0]
56 const map = {
57 '/': 'index.html',
58 '/dashboard': 'index.html',
59 '/fleet': 'fleet.html',
60 '/network': 'network.html',
61 '/catalog': 'catalog.html',
62 '/payments': 'payments.html',
63 '/calculator': 'calculator.html',
64 '/leaderboard': 'leaderboard.html',
65 '/docs': 'docs.html',
66 '/wizard': 'wizard.html'
67 }
68 if (map[p]) return join(DASH, map[p])
69 if (p.startsWith('/dashboard/')) return join(DASH, p.slice('/dashboard/'.length))
70 // bare filename under dashboard
71 if (/^\/[a-z0-9_.-]+\.html$/i.test(p)) return join(DASH, p.slice(1))
72 return null
73}
74
75async function proxy (req, res) {
76 const target = UPSTREAM + req.url
77 const headers = { ...req.headers, host: new URL(UPSTREAM).host }
78 delete headers['content-length']
79 if (req.url.startsWith('/api/')) {
80 headers.authorization = `Bearer ${API_KEY}`
81 }
82 const init = { method: req.method, headers }
83 if (req.method !== 'GET' && req.method !== 'HEAD') {
84 const chunks = []
85 for await (const c of req) chunks.push(c)
86 init.body = Buffer.concat(chunks)
87 }
88 const upstream = await fetch(target, init)
89 res.writeHead(upstream.status, Object.fromEntries(upstream.headers))
90 const buf = Buffer.from(await upstream.arrayBuffer())
91 res.end(buf)
92}
93
94const server = http.createServer(async (req, res) => {
95 try {
96 const urlPath = req.url.split('?')[0]
97 if (urlPath.startsWith('/api/')) {
98 await proxy(req, res)
99 return
100 }
101 const file = mapPath(urlPath)
102 if (file && existsSync(file)) {
103 let body = readFileSync(file)
104 const ext = extname(file)
105 if (ext === '.html') body = Buffer.from(injectToken(body.toString('utf8')), 'utf8')
106 res.writeHead(200, {
107 'content-type': MIME[ext] || 'application/octet-stream',
108 'cache-control': 'no-store'
109 })
110 res.end(body)
111 return
112 }
113 // fall through proxy for other paths (status, health, …)
114 await proxy(req, res)
115 } catch (err) {
116 res.writeHead(502, { 'content-type': 'text/plain' })
117 res.end('proxy error: ' + (err && err.message ? err.message : String(err)))
118 }
119})
120
121server.listen(PORT, '127.0.0.1', () => {
122 console.log(`Fleet dashboard proxy on http://127.0.0.1:${PORT}/fleet`)
123 console.log(`Upstream ${UPSTREAM}`)
124})