| … | … | @@ -0,0 +1,198 @@ |
| 1 | # Runbook: migrating an Autobase to autobee 2 |
| 2 | |
| 3 | **Status:** validated by spike, **not yet run on production data.** Every claim below marked ✅ was |
| 4 | measured on 2026-09-02 against real code (`pgh/lib/name-registry.js` and an |
| 5 | `atproto-room`-shaped encrypted multi-writer HyperDB base). Claims marked ⚠ are reasoned, not measured. |
| 6 | |
| 7 | autobee 2 replaces autobase's indexer/quorum model with provable sort **weights** |
| 8 | (`WEIGHTS.md`) — the machinery `atproto-room/lib/state-base.js` spends ~1391 lines working around. |
| 9 | It is a ground-up replacement (no autobase dependency) but it **adopts an existing autobase log in |
| 10 | place, preserving the key**. |
| 11 | |
| 12 | --- |
| 13 | |
| 14 | ## 0. Preconditions |
| 15 | |
| 16 | | requirement | our state (2026-09-02) | |
| 17 | |---|---| |
| 18 | | `hypercore ^11.33.2` | ✅ atproto-room 11.34.0, pgh 11.33.1, mirall-cli 11.33.5; ⚠ `.scope/hyper-rooms` 11.28.1 needs a bump | |
| 19 | | view is Hyperbee | port to hyperbee2 (see §2) | |
| 20 | | view is HyperDB | ✅ `HyperDB.bee2` already exists in our hyperdb **6.7.0** — no upgrade | |
| 21 | | autobee status | **`2.0.0-rc.30`**, README says "expect breaking changes"; upstream still marks the migration path `// @todo migration` | |
| 22 | |
| 23 | ⚠ Do not start on a base whose writers you do not control (see §4 — the writer set is a flag day). |
| 24 | |
| 25 | --- |
| 26 | |
| 27 | ## 1. The order that matters |
| 28 | |
| 29 | ``` |
| 30 | 1. ship guard-aware code fleet-wide (inert; no behaviour change) |
| 31 | 2. confirm adoption on EVERY peer |
| 32 | 3. announce requireVersion in the OLD format |
| 33 | 4. confirm every peer halted or upgraded |
| 34 | 5. migrate the writers (flag day) |
| 35 | 6. re-grant indexer capability, per writer |
| 36 | 7. verify FROM AN OLD PEER |
| 37 | ``` |
| 38 | |
| 39 | **Steps 1–4 must happen before step 5.** Once a writer emits autobee nodes, old peers cannot parse |
| 40 | anything from it, so a guard sent afterwards is never seen. |
| 41 | |
| 42 | --- |
| 43 | |
| 44 | ## 2. Code changes (the port) |
| 45 | |
| 46 | Measured on `pgh/lib/name-registry.js`: **46 changed lines, 33 of which are a codec shim** — only 9 |
| 47 | lines of the registry proper. The apply body, every op handler, every read method and both |
| 48 | `addWriter`/`removeWriter` calls port **verbatim** (autobee passes `host` in the same positional slot |
| 49 | with the same method names). |
| 50 | |
| 51 | ```js |
| 52 | - import Autobase from 'autobase' |
| 53 | - import Hyperbee from 'hyperbee' |
| 54 | + import Autobee from 'autobee' |
| 55 | |
| 56 | - this._base = new Autobase(this._store, this._bootstrap, { |
| 57 | - open: (store) => new Hyperbee(store.get('view'), { keyEncoding: 'utf-8', valueEncoding: 'json' }), |
| 58 | - apply, |
| 59 | - valueEncoding: 'binary' |
| 60 | - }) |
| 61 | + this._base = new Autobee(this._store, this._bootstrap, { |
| 62 | + open: (bee) => new Utf8JsonBee(bee), // hyperbee2 has NO encodings and no put/del |
| 63 | + legacyViews: ['view'], // autobase's view core NAME |
| 64 | + migrate: (views) => views.get('view'), // ⛔ REQUIRED — see §5 |
| 65 | + apply |
| 66 | + }) |
| 67 | ``` |
| 68 | |
| 69 | **The one real API break:** hyperbee2 has **no `keyEncoding`/`valueEncoding` and no `put`/`del`.** |
| 70 | Keys and values are raw Buffers; every mutation goes through `bee.write()` → |
| 71 | `w.tryPut/tryDelete` → `await w.flush()`; `get()` returns `{key, value}` as Buffers or null. |
| 72 | Either hand-encode at every call site or wrap once in a codec (the 33 lines). |
| 73 | |
| 74 | **HyperDB views** (StateBaseV3): autobase's `open` receives a *store*, autobee's receives a |
| 75 | *hyperbee2 instance*, so: |
| 76 | |
| 77 | ```js |
| 78 | - open: (store) => HyperDB.bee(store.get('view'), def, { autoUpdate: true }) |
| 79 | + open: (bee) => HyperDB.bee2(bee, def, { autoUpdate: true }) |
| 80 | ``` |
| 81 | |
| 82 | **Encrypted bases:** pass `encryptionKey` + `encrypted: true`. The migration additionally |
| 83 | self-discovers the legacy key from `autobase/encryption` userData. |
| 84 | |
| 85 | --- |
| 86 | |
| 87 | ## 3. The guard (steps 1–4) |
| 88 | |
| 89 | Both stacks expose `interrupt(reason)` to apply (autobase 7.28 `lib/apply-calls.js:87`; autobee's |
| 90 | equivalent). Add to apply, in the **old** codebase, ahead of any migration: |
| 91 | |
| 92 | ```js |
| 93 | if (op.requireVersion !== undefined) { |
| 94 | if (op.requireVersion > MY_VERSION) { |
| 95 | base.interrupt('under-versioned: autobee migration requires v' + op.requireVersion) |
| 96 | return |
| 97 | } |
| 98 | continue |
| 99 | } |
| 100 | ``` |
| 101 | |
| 102 | ✅ Measured both arms: an under-versioned peer fires `'interrupt'`, sets `base.interrupted` to the |
| 103 | reason, and **stops applying** (no silent divergence); an up-to-date peer is untouched and keeps |
| 104 | applying. This is keet's "Autobase is closing"/unsupportedABI shape and matches pgv3's |
| 105 | `lib/services/schema-guard.js`. |
| 106 | |
| 107 | ⚠ **The guard only protects peers that already ship the guard clause.** A peer whose apply has no |
| 108 | `requireVersion` branch treats it as an unknown op and skips it — diverging silently exactly as |
| 109 | before. That is why step 1 exists and why step 2 is a hard gate. (Reasoned from the apply shape; |
| 110 | a guard-unaware peer was not tested.) |
| 111 | |
| 112 | --- |
| 113 | |
| 114 | ## 4. Migrate (step 5) — this is a FLAG DAY |
| 115 | |
| 116 | ✅ Measured: adoption preserves the key, all pre-migration state, and both writer records. |
| 117 | |
| 118 | ⛔ **Mixed fleets are ONE-WAY.** With A migrated and B left on autobase: |
| 119 | |
| 120 | | direction | result | |
| 121 | |---|---| |
| 122 | | old autobase write → new autobee peer | ✅ seen | |
| 123 | | new autobee write → old autobase peer | ⛔ **never** seen (confirmed with a 45s wait) | |
| 124 | |
| 125 | The pair still connects, replicates, and raises **no error on either side**. So: |
| 126 | |
| 127 | - read-only followers/mirrors may migrate at any time (they only consume); |
| 128 | - **every writer of a given base must migrate together.** |
| 129 | |
| 130 | --- |
| 131 | |
| 132 | ## 5. Three silent traps |
| 133 | |
| 134 | 1. **Missing `migrate` handler = total state loss that looks like success.** |
| 135 | `autobee/index.js:410-413` — without it the view is `EMPTY_HEAD`. Observed: key preserved, |
| 136 | `writable: true`, migration ran, apply never fired, `bee.core.length: 0`, every record gone. |
| 137 | Fix: `migrate: (views) => views.get('view')`. |
| 138 | 2. **Asserting before `update()` = false alarm.** The migrated head carries only the *indexed* |
| 139 | state; `migration.catchup` nodes replay on the first drain. Observed: 2 of 3 entries right after |
| 140 | `ready()` (the second writer's was missing); a plain `update()`, no write, recovered it. |
| 141 | **Always `await db.update()` after `ready()` before judging the view.** |
| 142 | 3. **Mixed-fleet divergence is invisible from the migrated node** (§4). It sees a healthy, |
| 143 | fully-converged room. Only old peers are missing data, and they get no error. |
| 144 | |
| 145 | --- |
| 146 | |
| 147 | ## 6. Re-grant indexer capability (step 6) |
| 148 | |
| 149 | ✅ Migration carries the **sort weight** (`weight=2`, `currentWeight=2` — ordering preserved) but |
| 150 | **not** the granted capability: `maxWeight` comes back unset, so `db.isIndexer` reports `false`. |
| 151 | autobee treats these as distinct — `writers.js:467` `this.weight = info.maxWeight` is commented |
| 152 | *"capability (isIndexer etc), NOT the resolved sort weight"*, and `isIndexer` is `weight === 2`. |
| 153 | |
| 154 | Fix, **once per writer that should hold indexer capability**: |
| 155 | |
| 156 | ```js |
| 157 | host.addWriter(writerKey, { isIndexer: true }) // sets maxWeight = 2 |
| 158 | ``` |
| 159 | |
| 160 | ✅ Measured: `maxWeight undefined → 2`, `isIndexer false → true`, writes landing throughout. |
| 161 | ⚠ Re-granting only self leaves peers at `maxWeight = 0`. |
| 162 | ⚠ Impact of skipping this not measured — but `writers.js:406` `detachAndClose()`s a non-indexer, |
| 163 | non-bootstrap, non-local writer once caught up, so peer-writer retention may change. |
| 164 | |
| 165 | --- |
| 166 | |
| 167 | ## 7. Verify (step 7) |
| 168 | |
| 169 | **Run the check on an OLD peer, not the migrated one.** A check run on the migrated node is |
| 170 | structurally incapable of catching the §4 failure. |
| 171 | |
| 172 | ``` |
| 173 | [ ] migrated node: key unchanged (compare to the pre-migration key you recorded) |
| 174 | [ ] migrated node: await update(), THEN count records vs the pre-migration count |
| 175 | [ ] migrated node: a record authored by a DIFFERENT writer is present |
| 176 | [ ] migrated node: isIndexer true for every writer that should have it (after §6) |
| 177 | [ ] OLD peer: does it see a write the migrated node made AFTER migration? |
| 178 | if no -> that peer has not migrated and is silently diverging |
| 179 | [ ] a post-migration write lands and is visible on every migrated peer |
| 180 | ``` |
| 181 | |
| 182 | Record the pre-migration key and record count **before** step 5; you cannot recover them afterwards. |
| 183 | |
| 184 | --- |
| 185 | |
| 186 | ## 8. Not yet verified |
| 187 | |
| 188 | - StateBaseV3's real attestation-gated admission on top of a migrated base |
| 189 | - a guard-unaware peer (§3) |
| 190 | - rollback from autobee back to autobase — **assume there is none** |
| 191 | - performance (the §2 codec shim flushes a batch per mutation; never benchmarked) |
| 192 | - any of this on production data |
| 193 | |
| 194 | ## Provenance |
| 195 | |
| 196 | Six spikes, 2026-09-02. Detail + exact file:line citations in memory: |
| 197 | `reference_autobee2-vs-our-autobase.md`. Throwaway spike code (control + port + 4 migration |
| 198 | scenarios) lived in the session scratchpad and was not kept. |