| … | … | @@ -0,0 +1,154 @@ |
| 1 | # Release runbook — v0.25.0-rc.5 (canary, Track A) |
| 2 | |
| 3 | **Prepared:** 2026-07-28 · **Branch:** `feat/service-http-wiring` · **Status:** ready for maintainer signing |
| 4 | |
| 5 | Everything up to the signed tag is done. The remaining steps need your signing key |
| 6 | and a push, so they are yours. |
| 7 | |
| 8 | --- |
| 9 | |
| 10 | ## 0. What is already prepared |
| 11 | |
| 12 | | Item | State | |
| 13 | |------|-------| |
| 14 | | Code fixes + restored features | committed (`2fa58c1`) | |
| 15 | | Doc corrections (pilot hostnames, Ship 9 state) | committed (`aae40ff`) | |
| 16 | | Version bump to `0.25.0-rc.5` + CHANGELOG | committed (`59d2362`) | |
| 17 | | Fleet watchdog / admin operator / stability writeup | committed (`40ea4bf`) | |
| 18 | | `hold` channel added to `channels.json` | committed, pinned at `v0.24.3` | |
| 19 | | `canary` promotion in `channels.json` | **NOT applied** — do it after the tag exists | |
| 20 | |
| 21 | Unit suite at time of prep: **3510 tests, 3 known failures** (serve-only recovery |
| 22 | mode; two startup-rollback cases blocked on the Corestore-7 storage-sweep |
| 23 | question). Previously the suite aborted at test 727 on a hard `TypeError`. |
| 24 | |
| 25 | --- |
| 26 | |
| 27 | ## 1. Blockers you must clear |
| 28 | |
| 29 | ### 1a. The branch is not on `main` |
| 30 | |
| 31 | Every box resolves its target from |
| 32 | `https://raw.githubusercontent.com/bigdestiny2/P2P-Hiverelay/main/fleet/channels.json`. |
| 33 | Nothing ships until this branch lands on `main`. |
| 34 | |
| 35 | ```bash |
| 36 | git rev-list --count origin/main..HEAD |
| 37 | ``` |
| 38 | |
| 39 | At prep time this was **124 commits**. Open a PR from `feat/service-http-wiring` |
| 40 | rather than fast-forwarding — `rc.3` was previously cut from `main` by mistake |
| 41 | without the feature tree, and that is exactly the failure this gap invites. |
| 42 | |
| 43 | ### 1b. No signing key is configured |
| 44 | |
| 45 | The updater refuses any tag that is not an annotated tag signed by a key in |
| 46 | `/etc/hiverelay/allowed-signers`. It fails closed: a box that cannot verify the |
| 47 | tag stays on its current version. This repo currently has no signing config: |
| 48 | |
| 49 | ```bash |
| 50 | git config --get user.signingkey # empty |
| 51 | git config --get gpg.format # empty |
| 52 | git config --get tag.gpgsign # empty |
| 53 | ``` |
| 54 | |
| 55 | Configure it, then tag. **An unsigned tag is refused by every box.** |
| 56 | |
| 57 | --- |
| 58 | |
| 59 | ## 2. Cut the release |
| 60 | |
| 61 | ```bash |
| 62 | git tag -s v0.25.0-rc.5 -m "v0.25.0-rc.5 — restore orphaned lifecycle/gateway features" |
| 63 | ``` |
| 64 | |
| 65 | Verify before pushing — this is the exact check each box performs: |
| 66 | |
| 67 | ```bash |
| 68 | git tag -v v0.25.0-rc.5 |
| 69 | ``` |
| 70 | |
| 71 | Then: |
| 72 | |
| 73 | ```bash |
| 74 | git push origin v0.25.0-rc.5 |
| 75 | ``` |
| 76 | |
| 77 | --- |
| 78 | |
| 79 | ## 3. Hold sydney and dallas OFF this rollout |
| 80 | |
| 81 | **dallas needs nothing.** It has no `/etc/hiverelay-updater.conf` and no updater |
| 82 | installed, so it is excluded by construction and will not move. |
| 83 | |
| 84 | **sydney is `CHANNEL=stable`.** Today that is harmless — this release only bumps |
| 85 | `canary`, and sydney is not a canary box. It becomes load-bearing the moment you |
| 86 | promote `stable`. Pin it now so that promotion cannot sweep it in: |
| 87 | |
| 88 | ```bash |
| 89 | ssh -i ~/.ssh/cloudzy_hiverelay root@104.194.135.205 'printf "CHANNEL=hold\n" > /etc/hiverelay-updater.conf && cat /etc/hiverelay-updater.conf' |
| 90 | ``` |
| 91 | |
| 92 | Confirm the box resolves the held target rather than a moving one: |
| 93 | |
| 94 | ```bash |
| 95 | ssh -i ~/.ssh/cloudzy_hiverelay root@104.194.135.205 'hiverelay-updater --dry-run' |
| 96 | ``` |
| 97 | |
| 98 | Both hosts run the Track B blind public-test stack on `:443`. Track A never |
| 99 | touches `:443`, the blind compose, or the blind roots — only the fleet relay on |
| 100 | `:9100` — but keeping them off the rollout entirely is the safer reading of the |
| 101 | `LIVE_PUBLIC_TEST_ONLY` lease. |
| 102 | |
| 103 | --- |
| 104 | |
| 105 | ## 4. Promote the canary channel |
| 106 | |
| 107 | Only after the signed tag is pushed: |
| 108 | |
| 109 | ```bash |
| 110 | # fleet/channels.json — canary only; stable stays on v0.24.3 |
| 111 | # "canary": "v0.25.0-rc.4" -> "canary": "v0.25.0-rc.5" |
| 112 | git commit -am 'chore(fleet): promote canary to v0.25.0-rc.5' && git push |
| 113 | ``` |
| 114 | |
| 115 | Canary boxes are **utah** and **utah-0.5gb** (and bern, currently down). They |
| 116 | self-update within ~15 minutes on a jittered timer, health-gate, and roll back |
| 117 | automatically if `/health` does not report `running:true` at the target version |
| 118 | within 120s. |
| 119 | |
| 120 | Verify convergence against the release tag SHA — not just the package version: |
| 121 | |
| 122 | ```bash |
| 123 | npm run fleet:check-rollout -- --target v0.25.0-rc.5 --channel canary |
| 124 | ``` |
| 125 | |
| 126 | ⚠️ **utah-0.5gb has 0.5 GB RAM.** Per `docs/FLEET-STABILITY-2026-07-27.md` it runs |
| 127 | a trimmed plugin set (`outboxlog`, `notify`, `vrf`) and was ABRT-looping under the |
| 128 | full suite. Watch it specifically after the tick. |
| 129 | |
| 130 | --- |
| 131 | |
| 132 | ## 5. Fleet state at prep time |
| 133 | |
| 134 | 12/13 up. bern is a provider-side outage (SSH + ping dead), not a process fault. |
| 135 | |
| 136 | | Box | Channel | Version | Note | |
| 137 | |-----|---------|---------|------| |
| 138 | | utah | canary | 0.25.0-rc.4 | canary target; live onion advertised | |
| 139 | | utah-0.5gb | canary | 0.24.3 | 0.5 GB — trimmed plugins, watch closely | |
| 140 | | utah-8gb | stable | 0.25.0-rc.4 | manually placed after CS7 remigrate | |
| 141 | | utah-us, utah-2gb-a, miami, sing-1, sing-2, amsterdam, dubai | stable | 0.24.3 | | |
| 142 | | **sydney** | stable | 0.24.3 | Track B host — pin to `hold` (§3) | |
| 143 | | **dallas** | *(no updater)* | 0.24.3 | Track B host — excluded by construction | |
| 144 | | bern | canary | — | provider outage; needs console/power | |
| 145 | |
| 146 | --- |
| 147 | |
| 148 | ## 6. Do NOT promote `stable` from this runbook |
| 149 | |
| 150 | `npm run release:check-blockers` reports the full public-release evidence chain as |
| 151 | open: image manifest, digest-pinned image smoke, Umbrel package smoke, official |
| 152 | Umbrel PR, Umbrel runtime review, StartOS registry, and fleet rollout evidence. |
| 153 | Those gate the *public full release*, not a canary tick. Close them before |
| 154 | `stable` moves off `v0.24.3`. |