| … | … | @@ -0,0 +1,164 @@ |
| 1 | # autobase-doctor | |
| 2 | ||
| 3 | Diagnose and repair corrupted [Autobase](https://github.com/holepunchto/autobase) instances. | |
| 4 | ||
| 5 | ## Install | |
| 6 | ||
| 7 | ```bash | |
| 8 | cd autobase-doctor && npm install | |
| 9 | ``` | |
| 10 | ||
| 11 | ## Usage | |
| 12 | ||
| 13 | ```bash | |
| 14 | # Scan all Autobases in a storage directory for corruption | |
| 15 | bare bin/autobase-doctor.js scan ./pear-git-storage | |
| 16 | ||
| 17 | # Repair (dry run first) | |
| 18 | bare bin/autobase-doctor.js repair ./pear-git-storage --dry-run | |
| 19 | bare bin/autobase-doctor.js repair ./pear-git-storage | |
| 20 | ||
| 21 | # Inspect a specific Autobase's ops | |
| 22 | bare bin/autobase-doctor.js inspect ./pear-git-storage --namespace linked:myrepo | |
| 23 | ||
| 24 | # Check system state health | |
| 25 | bare bin/autobase-doctor.js check ./pear-git-storage --namespace linked:myrepo | |
| 26 | ``` | |
| 27 | ||
| 28 | **Must stop any process using the storage first** — RocksDB doesn't allow concurrent access. | |
| 29 | ||
| 30 | ## Autobase Failure Modes & Pitfalls | |
| 31 | ||
| 32 | ### 1. Poisoned `host.addWriter()` — Wrong Key Type | |
| 33 | ||
| 34 | **What happens:** An apply function calls `host.addWriter(key)` with a key that isn't a real Hypercore key (e.g., an Ed25519 identity key, a random hash, or any 32-byte buffer that doesn't correspond to an actual Hypercore writer core). | |
| 35 | ||
| 36 | **Why it's fatal:** | |
| 37 | - `host.addWriter()` calls `system.add(key)` which persists the key in the system Hyperbee | |
| 38 | - Then calls `_addWriter(key)` which creates a writer for it | |
| 39 | - The writer core has no data, no manifest — it's an empty stub | |
| 40 | - If registered as an indexer: `getIndexerManifests()` skips cores without manifests, producing a shorter array than the indexers array. `findViewByKey()` then crashes on `indexerManifests[0].signers[0].namespace` (TypeError) | |
| 41 | - If registered as a non-indexer writer: the drain loop tries to load data from it, enters wakeup/catchup loops, and `update()` hangs forever | |
| 42 | ||
| 43 | **Why it's permanent:** | |
| 44 | - The `add-writer` op is in the append-only writer log — it replays on every boot | |
| 45 | - Even if you catch the error in your apply function, `system.add()` already ran and poisoned the system state | |
| 46 | - The system core persists the bogus writer registration | |
| 47 | - Truncating only the writer core leaves stale system state | |
| 48 | - Clearing only `autobase/boot` doesn't help — the system core key is deterministic (same manifest → same key → same core → same poisoned data) | |
| 49 | ||
| 50 | **Recovery (what this tool does):** | |
| 51 | 1. Truncate the writer core at the poisoned op (removes it from the log) | |
| 52 | 2. Truncate the system core's batch session (removes persisted poisoned state) | |
| 53 | 3. Truncate the view core's batch session (removes derived view data) | |
| 54 | 4. Clear `autobase/boot` userData (forces re-bootstrap) | |
| 55 | 5. On next boot, Autobase sees empty system → bootstraps fresh → replays clean ops | |
| 56 | ||
| 57 | **Prevention:** | |
| 58 | - Validate keys before calling `host.addWriter()` — check that a core with that key has a manifest or data | |
| 59 | - Never pass server identity keys, SSH keys, or other non-Hypercore keys | |
| 60 | - For federation, exchange actual Hypercore local core keys, not server identity keys | |
| 61 | ||
| 62 | ### 2. System Core Is Not Writable | |
| 63 | ||
| 64 | **What happens:** The Autobase system core exists but `writable` is false, even on the server that created it. | |
| 65 | ||
| 66 | **Why it happens:** | |
| 67 | - `base.core` is a **read-only session** — Autobase explicitly creates it with `writable: false` in `ViewCore.createSession()` | |
| 68 | - Opening the core directly by key (without the manifest) makes the corestore treat it as a remote core | |
| 69 | - The actual writable access is through `ViewCore.batch` — a session created with `{ name: 'batch', writable: true }` | |
| 70 | ||
| 71 | **Pitfall:** Checking `base.core.writable` and concluding the core can't be fixed is wrong. The core IS writable through the batch session. The recovery must access `base._viewStore.getSystemView().batch`. | |
| 72 | ||
| 73 | ### 3. System Core Length Mismatch | |
| 74 | ||
| 75 | **What happens:** The system core shows `length: 0` when opened directly by key, but `length: N` when accessed through the Autobase session. | |
| 76 | ||
| 77 | **Why:** The Autobase creates system state during `ready()` in an in-memory batch. These blocks haven't been flushed to disk yet. The `length: 0` is the on-disk state; the `length: N` is the in-memory batch state. Both are real — the batch will be flushed on the next successful `update()`. | |
| 78 | ||
| 79 | **Pitfall:** Inspecting cores by name in the corestore namespace gives different results than what Autobase sees. The system core's key is derived from the indexer manifests via `_createManifest()`, not from the namespace name `_system`. Opening `ns.get({ name: '_system' })` gives a different core than `base.core`. | |
| 80 | ||
| 81 | ### 4. Mirror Replication Re-poisons | |
| 82 | ||
| 83 | **What happens:** You fix server A, but when it reconnects to server B (mirror), the corrupted state replicates back. | |
| 84 | ||
| 85 | **Why:** Each server has its own writer core in the Autobase. If server B's writer core still has the poisoned op, corestore replication syncs it to server A. Server A's Autobase now has two writer cores — its own (clean) and B's (poisoned). The apply function replays B's poisoned op and re-corrupts. | |
| 86 | ||
| 87 | **Fix:** Run recovery on ALL mirror peers, not just one. Stop all peers first, truncate all, then restart. | |
| 88 | ||
| 89 | ### 5. `autobase/boot` Gets Re-set on Every Boot | |
| 90 | ||
| 91 | **What happens:** You clear `autobase/boot` from the local core's userData, but after booting, it's set again. | |
| 92 | ||
| 93 | **Why:** Autobase writes `autobase/boot` during normal operation as a checkpoint. Clearing it before boot makes Autobase start fresh, but it immediately writes a new boot record after processing. If the system still has poisoned state when it writes, the new boot record points to poisoned state. | |
| 94 | ||
| 95 | **Fix:** Clear `autobase/boot` AND truncate the system batch in the same operation. Don't clear boot without also resetting the system. | |
| 96 | ||
| 97 | ### 6. `referrer` userData Gets Re-set | |
| 98 | ||
| 99 | **What happens:** You clear the `referrer` from the local core, but boot.js sets it again. | |
| 100 | ||
| 101 | **Why:** `boot.js` lines 98-102 set referrer on every boot when a key or keyPair is provided. For self-bootstrapped Autobases (key=null), lines 88-95 set `result.key = result.local.key` but DON'T re-set referrer. However, if the Autobase was previously bootstrapped with a key, the referrer gets re-set. | |
| 102 | ||
| 103 | **Implication:** Clearing referrer alone is not a fix. The core data and system state must also be addressed. | |
| 104 | ||
| 105 | ### 7. Truncating Writer Core Loses Ops After the Poison | |
| 106 | ||
| 107 | **What happens:** If the poisoned `add-writer` op is at index N and there are ops after it (at N+1, N+2, ...), truncating to N removes those too. | |
| 108 | ||
| 109 | **Why:** Hypercore truncation removes all blocks from the specified index onward. It's not selective — you can't remove just one block from the middle. | |
| 110 | ||
| 111 | **Mitigation:** The poisoned op is usually the last (or near-last) op appended. Check what's after it before truncating. For pear-git, the poisoned op was always the last or second-to-last op (only 1-2 ops lost per repo). | |
| 112 | ||
| 113 | ### 8. Autobase apply Must Be Deterministic | |
| 114 | ||
| 115 | **What happens:** Adding validation logic to the apply function (e.g., checking core existence before `host.addWriter()`) works for preventing NEW corruption, but doesn't fix existing poisoned logs. | |
| 116 | ||
| 117 | **Why:** Apply runs deterministically on every boot, replaying the entire log. If the poisoned op is in the log, apply processes it every time. Even with validation that skips the `host.addWriter()` call, the op still gets decoded and the metadata still gets stored in the Hyperbee view. The validation prevents Autobase-level corruption but the op remains in the log forever. | |
| 118 | ||
| 119 | **Implication:** Apply-level fixes are defense-in-depth, not recovery. Recovery requires modifying the log (truncation) and/or the system state. | |
| 120 | ||
| 121 | ## Architecture | |
| 122 | ||
| 123 | ``` | |
| 124 | autobase-doctor/ | |
| 125 | ├── bin/autobase-doctor.js # CLI entry point | |
| 126 | ├── lib/ | |
| 127 | │ ├── scanner.js # Find poisoned ops in writer cores | |
| 128 | │ └── repair.js # Truncate cores and reset system state | |
| 129 | ├── package.json | |
| 130 | └── README.md | |
| 131 | ``` | |
| 132 | ||
| 133 | ### How It Works | |
| 134 | ||
| 135 | 1. **Scan:** Opens each writer core as raw binary, searches for `{"type":"add-writer"` marker, decodes the JSON, checks if `serverKey` is present (indicator of the poison pattern). | |
| 136 | ||
| 137 | 2. **Repair — Writer Truncation:** Opens the local core by name, calls `core.truncate(poisonIndex)` to remove the poisoned op and everything after it. | |
| 138 | ||
| 139 | 3. **Repair — System Reset:** Opens the Autobase with a no-op apply function, accesses `_viewStore.getSystemView().batch` (the writable batch session), truncates it to 0. Also truncates the view batch. Clears `autobase/boot` userData. | |
| 140 | ||
| 141 | 4. **Check:** Opens the Autobase and inspects `writable`, `system.members`, `system.indexers` to determine if the system state is healthy. A self-bootstrapped Autobase with `writable: false` and `members > 0` indicates poisoned state. | |
| 142 | ||
| 143 | ## Customization | |
| 144 | ||
| 145 | The scanner and repair functions accept custom options: | |
| 146 | ||
| 147 | ```javascript | |
| 148 | import { scanWriterCore } from './lib/scanner.js' | |
| 149 | import { resetSystemState } from './lib/repair.js' | |
| 150 | ||
| 151 | // Custom poison detection | |
| 152 | const result = await scanWriterCore(ns, { | |
| 153 | marker: Buffer.from('"type":"my-bad-op"'), | |
| 154 | isPoisoned: (op) => op.type === 'my-bad-op' && op.badField | |
| 155 | }) | |
| 156 | ||
| 157 | // Custom Autobase open/apply for repair | |
| 158 | await resetSystemState(ns, { | |
| 159 | open: (store) => new MyCustomView(store), | |
| 160 | apply: myCustomApply | |
| 161 | }) | |
| 162 | ``` | |
| 163 | ||
| 164 | The `--registry` and `--registry-prefix` CLI flags allow using different namespace discovery sources (not just pear-git's `config:issueAutobase:` prefix). |
| … | … | @@ -0,0 +1,191 @@ |
| 1 | #!/usr/bin/env bare | |
| 2 | ||
| 3 | /** | |
| 4 | * autobase-doctor — Diagnose and repair corrupted Autobase instances. | |
| 5 | * | |
| 6 | * Usage: | |
| 7 | * autobase-doctor scan <storage> [--namespace <ns>] | |
| 8 | * autobase-doctor repair <storage> [--namespace <ns>] [--dry-run] | |
| 9 | * autobase-doctor inspect <storage> --namespace <ns> | |
| 10 | * autobase-doctor check <storage> --namespace <ns> | |
| 11 | */ | |
| 12 | ||
| 13 | import Corestore from 'corestore' | |
| 14 | import Hyperbee from 'hyperbee' | |
| 15 | import path from 'bare-path' | |
| 16 | import process from 'bare-process' | |
| 17 | import { scanWriterCore, listOps } from '../lib/scanner.js' | |
| 18 | import { truncateWriterCore, resetSystemState, hasPoisonedSystemState } from '../lib/repair.js' | |
| 19 | ||
| 20 | const args = process.argv.slice(2) | |
| 21 | const command = args[0] | |
| 22 | const storagePath = args[1] | |
| 23 | ||
| 24 | if (!command || !storagePath || command === '--help' || command === '-h') { | |
| 25 | console.log(`autobase-doctor — Diagnose and repair corrupted Autobase instances | |
| 26 | ||
| 27 | Usage: | |
| 28 | autobase-doctor scan <storage> Scan all namespaces for poisoned ops | |
| 29 | autobase-doctor repair <storage> [--dry-run] Fix all poisoned Autobases | |
| 30 | autobase-doctor inspect <storage> --namespace <ns> Dump all ops in a namespace | |
| 31 | autobase-doctor check <storage> --namespace <ns> Check system state health | |
| 32 | ||
| 33 | Options: | |
| 34 | --namespace <ns> Target a specific corestore namespace (e.g. "linked:myrepo") | |
| 35 | --registry <name> Registry core name for namespace discovery (default: "registry") | |
| 36 | --registry-prefix Registry key prefix (default: "config:issueAutobase:") | |
| 37 | --dry-run Show what would be done without making changes | |
| 38 | ||
| 39 | Examples: | |
| 40 | autobase-doctor scan ./pear-git-storage | |
| 41 | autobase-doctor repair ./pear-git-storage --dry-run | |
| 42 | autobase-doctor inspect ./pear-git-storage --namespace linked:pear-git | |
| 43 | autobase-doctor check ./pear-git-storage --namespace linked:myrepo`) | |
| 44 | process.exit(0) | |
| 45 | } | |
| 46 | ||
| 47 | let namespace = null | |
| 48 | let registryName = 'registry' | |
| 49 | let registryPrefix = 'config:issueAutobase:' | |
| 50 | let dryRun = false | |
| 51 | ||
| 52 | for (let i = 2; i < args.length; i++) { | |
| 53 | if (args[i] === '--namespace' && args[i + 1]) { namespace = args[i + 1]; i++ } | |
| 54 | else if (args[i] === '--registry' && args[i + 1]) { registryName = args[i + 1]; i++ } | |
| 55 | else if (args[i] === '--registry-prefix' && args[i + 1]) { registryPrefix = args[i + 1]; i++ } | |
| 56 | else if (args[i] === '--dry-run') { dryRun = true } | |
| 57 | } | |
| 58 | ||
| 59 | const storageDir = path.resolve(storagePath) | |
| 60 | const store = new Corestore(storageDir) | |
| 61 | await store.ready() | |
| 62 | ||
| 63 | /** | |
| 64 | * Discover namespaces from registry or use explicit --namespace | |
| 65 | */ | |
| 66 | async function getNamespaces () { | |
| 67 | if (namespace) return [namespace] | |
| 68 | ||
| 69 | const core = store.get({ name: registryName }) | |
| 70 | const db = new Hyperbee(core, { keyEncoding: 'utf-8', valueEncoding: 'json' }) | |
| 71 | await db.ready() | |
| 72 | ||
| 73 | const namespaces = [] | |
| 74 | for await (const entry of db.createReadStream({ | |
| 75 | gt: registryPrefix, lt: registryPrefix + '\xff' | |
| 76 | })) { | |
| 77 | const name = entry.key.slice(registryPrefix.length) | |
| 78 | namespaces.push('linked:' + name) | |
| 79 | } | |
| 80 | ||
| 81 | await db.close() | |
| 82 | return namespaces | |
| 83 | } | |
| 84 | ||
| 85 | // ── Commands ────────────────────────────────────────────── | |
| 86 | ||
| 87 | if (command === 'scan') { | |
| 88 | const namespaces = await getNamespaces() | |
| 89 | console.log('Scanning ' + namespaces.length + ' namespaces in ' + storageDir + '\n') | |
| 90 | ||
| 91 | let poisoned = 0 | |
| 92 | for (const ns of namespaces) { | |
| 93 | const result = await scanWriterCore(store.namespace(ns)) | |
| 94 | if (result) { | |
| 95 | poisoned++ | |
| 96 | const label = ns.replace('linked:', '') | |
| 97 | console.log(label + ': POISONED at op ' + result.poisonIndex + '/' + result.totalOps) | |
| 98 | console.log(' ' + result.op.type + ' serverKey=' + (result.op.serverKey || '').slice(0, 24) + '...') | |
| 99 | } | |
| 100 | } | |
| 101 | ||
| 102 | console.log('\n' + poisoned + '/' + namespaces.length + ' poisoned') | |
| 103 | ||
| 104 | } else if (command === 'repair') { | |
| 105 | const namespaces = await getNamespaces() | |
| 106 | console.log('Repairing ' + namespaces.length + ' namespaces in ' + storageDir) | |
| 107 | console.log('Dry run: ' + dryRun + '\n') | |
| 108 | ||
| 109 | let fixed = 0 | |
| 110 | for (const nsName of namespaces) { | |
| 111 | const ns = store.namespace(nsName) | |
| 112 | const label = nsName.replace('linked:', '') | |
| 113 | ||
| 114 | // Step 1: Check writer core for poison | |
| 115 | const scan = await scanWriterCore(ns) | |
| 116 | if (scan) { | |
| 117 | console.log(label + ': poison at op ' + scan.poisonIndex + '/' + scan.totalOps) | |
| 118 | if (!dryRun) { | |
| 119 | const { before, after } = await truncateWriterCore(ns, scan.poisonIndex) | |
| 120 | console.log(' ✓ writer truncated ' + before + ' → ' + after) | |
| 121 | } else { | |
| 122 | console.log(' [DRY RUN] would truncate writer ' + scan.totalOps + ' → ' + scan.poisonIndex) | |
| 123 | } | |
| 124 | } | |
| 125 | ||
| 126 | // Step 2: Check system state (even if writer was already clean) | |
| 127 | const sysInfo = await hasPoisonedSystemState(ns) | |
| 128 | if (sysInfo.poisoned) { | |
| 129 | console.log((scan ? '' : label + ': ') + 'system poisoned (writable=' + sysInfo.writable + ' members=' + sysInfo.members + ')') | |
| 130 | if (!dryRun) { | |
| 131 | const result = await resetSystemState(ns) | |
| 132 | if (result.systemTruncated) console.log(' ✓ system batch reset (' + result.systemTruncated + ' blocks)') | |
| 133 | if (result.viewTruncated) console.log(' ✓ view batch reset (' + result.viewTruncated + ' blocks)') | |
| 134 | if (result.bootCleared) console.log(' ✓ cleared autobase/boot') | |
| 135 | } else { | |
| 136 | console.log(' [DRY RUN] would reset system batch (' + sysInfo.systemBatchLength + ' blocks)') | |
| 137 | } | |
| 138 | } | |
| 139 | ||
| 140 | if (scan || sysInfo.poisoned) fixed++ | |
| 141 | } | |
| 142 | ||
| 143 | console.log('\nFixed: ' + fixed + '/' + namespaces.length) | |
| 144 | if (dryRun) console.log('(Dry run — re-run without --dry-run to apply)') | |
| 145 | else if (fixed > 0) console.log('Restart Autobase consumers to rebuild clean state.') | |
| 146 | ||
| 147 | } else if (command === 'inspect') { | |
| 148 | if (!namespace) { | |
| 149 | console.error('--namespace required for inspect') | |
| 150 | process.exit(1) | |
| 151 | } | |
| 152 | ||
| 153 | const ns = store.namespace(namespace) | |
| 154 | const ops = await listOps(ns) | |
| 155 | console.log(namespace + ': ' + ops.length + ' ops\n') | |
| 156 | ||
| 157 | for (const { index, op, size, error } of ops) { | |
| 158 | if (error) { | |
| 159 | console.log('[' + index + '] ERROR: ' + error) | |
| 160 | } else if (op) { | |
| 161 | const summary = op.type + (op.title ? ': ' + op.title.slice(0, 60) : '') + | |
| 162 | (op.serverKey ? ' key=' + op.serverKey.slice(0, 24) + '...' : '') | |
| 163 | console.log('[' + index + '] ' + summary) | |
| 164 | } else { | |
| 165 | console.log('[' + index + '] (undecoded, ' + size + ' bytes)') | |
| 166 | } | |
| 167 | } | |
| 168 | ||
| 169 | } else if (command === 'check') { | |
| 170 | if (!namespace) { | |
| 171 | console.error('--namespace required for check') | |
| 172 | process.exit(1) | |
| 173 | } | |
| 174 | ||
| 175 | const ns = store.namespace(namespace) | |
| 176 | const info = await hasPoisonedSystemState(ns) | |
| 177 | console.log(namespace + ':') | |
| 178 | console.log(' writable: ' + info.writable) | |
| 179 | console.log(' members: ' + info.members) | |
| 180 | console.log(' indexers: ' + info.indexers) | |
| 181 | console.log(' local length: ' + info.localLength) | |
| 182 | console.log(' system batch: ' + info.systemBatchLength) | |
| 183 | console.log(' poisoned: ' + info.poisoned) | |
| 184 | ||
| 185 | } else { | |
| 186 | console.error('Unknown command: ' + command) | |
| 187 | console.error('Run autobase-doctor --help for usage') | |
| 188 | process.exit(1) | |
| 189 | } | |
| 190 | ||
| 191 | await store.close() |
| … | … | @@ -0,0 +1,144 @@ |
| 1 | /** | |
| 2 | * Repair — fix corrupted Autobase instances. | |
| 3 | * | |
| 4 | * Two repair operations: | |
| 5 | * 1. truncateWriterCore — remove poisoned ops from the writer log | |
| 6 | * 2. resetSystemState — truncate the internal system + view batches | |
| 7 | * so Autobase re-bootstraps from the clean writer log | |
| 8 | * | |
| 9 | * Both are needed because: | |
| 10 | * - The writer core has the poisoned op in the append-only log | |
| 11 | * - The system core has persisted state built from the poisoned op | |
| 12 | * - Truncating only the writer core leaves stale system state | |
| 13 | * - Resetting only the system core causes it to replay the poison | |
| 14 | */ | |
| 15 | ||
| 16 | import Autobase from 'autobase' | |
| 17 | import Hyperbee from 'hyperbee' | |
| 18 | ||
| 19 | /** | |
| 20 | * Truncate a writer core at the poisoned op index. | |
| 21 | * All ops before the poison are preserved. | |
| 22 | * | |
| 23 | * @param {Corestore} ns - Namespaced corestore | |
| 24 | * @param {number} poisonIndex - Index of the poisoned op | |
| 25 | * @returns {{ before: number, after: number }} | |
| 26 | */ | |
| 27 | export async function truncateWriterCore (ns, poisonIndex) { | |
| 28 | const local = ns.get({ name: 'local' }) | |
| 29 | await local.ready() | |
| 30 | ||
| 31 | const before = local.length | |
| 32 | await local.truncate(poisonIndex) | |
| 33 | const after = local.length | |
| 34 | ||
| 35 | await local.close() | |
| 36 | return { before, after } | |
| 37 | } | |
| 38 | ||
| 39 | /** | |
| 40 | * Reset Autobase system state so it re-bootstraps from the writer log. | |
| 41 | * | |
| 42 | * Opens the Autobase, accesses the internal ViewCore batch sessions | |
| 43 | * (which are writable even when the read-only session is not), and | |
| 44 | * truncates them to 0. Also clears the autobase/boot userData. | |
| 45 | * | |
| 46 | * Why the batch, not the core directly: | |
| 47 | * - base.core is a read-only session (Autobase creates it with writable: false) | |
| 48 | * - Opening the core by key doesn't provide the manifest, so corestore | |
| 49 | * treats it as remote (not writable) | |
| 50 | * - The ViewCore's batch session IS writable — it was created with | |
| 51 | * { name: 'batch', writable: true } by AutoStore | |
| 52 | * | |
| 53 | * @param {Corestore} ns - Namespaced corestore | |
| 54 | * @param {Object} opts | |
| 55 | * @param {Function} opts.open - Autobase open handler (creates the view) | |
| 56 | * @param {Function} opts.apply - Autobase apply handler | |
| 57 | * @returns {{ systemTruncated: number, viewTruncated: number, bootCleared: boolean }} | |
| 58 | */ | |
| 59 | export async function resetSystemState (ns, opts = {}) { | |
| 60 | const { | |
| 61 | open = defaultOpen, | |
| 62 | apply = defaultApply | |
| 63 | } = opts | |
| 64 | ||
| 65 | const base = new Autobase(ns, null, { | |
| 66 | open, | |
| 67 | apply, | |
| 68 | valueEncoding: 'binary' | |
| 69 | }) | |
| 70 | await base.ready() | |
| 71 | ||
| 72 | const result = { systemTruncated: 0, viewTruncated: 0, bootCleared: false } | |
| 73 | ||
| 74 | // Truncate system batch | |
| 75 | const sysView = base._viewStore.getSystemView() | |
| 76 | if (sysView.batch && sysView.batch.writable && sysView.batch.length > 0) { | |
| 77 | result.systemTruncated = sysView.batch.length | |
| 78 | await sysView.batch.truncate(0) | |
| 79 | } | |
| 80 | ||
| 81 | // Truncate view batch | |
| 82 | const viewView = base._viewStore.byName.get('view') | |
| 83 | if (viewView && viewView.batch && viewView.batch.writable && viewView.batch.length > 0) { | |
| 84 | result.viewTruncated = viewView.batch.length | |
| 85 | await viewView.batch.truncate(0) | |
| 86 | } | |
| 87 | ||
| 88 | // Clear boot record | |
| 89 | const boot = await base.local.getUserData('autobase/boot') | |
| 90 | if (boot) { | |
| 91 | await base.local.setUserData('autobase/boot', null) | |
| 92 | result.bootCleared = true | |
| 93 | } | |
| 94 | ||
| 95 | await base.close() | |
| 96 | return result | |
| 97 | } | |
| 98 | ||
| 99 | /** | |
| 100 | * Check if an Autobase has poisoned system state. | |
| 101 | * Signs: writable=false with members > 0 on a self-bootstrapped Autobase. | |
| 102 | */ | |
| 103 | export async function hasPoisonedSystemState (ns, opts = {}) { | |
| 104 | const { | |
| 105 | open = defaultOpen, | |
| 106 | apply = defaultApply | |
| 107 | } = opts | |
| 108 | ||
| 109 | const base = new Autobase(ns, null, { | |
| 110 | open, | |
| 111 | apply, | |
| 112 | valueEncoding: 'binary' | |
| 113 | }) | |
| 114 | await base.ready() | |
| 115 | ||
| 116 | const poisoned = !base.writable && base.system.members > 0 | |
| 117 | const info = { | |
| 118 | poisoned, | |
| 119 | writable: base.writable, | |
| 120 | members: base.system.members, | |
| 121 | indexers: base.system.indexers.length, | |
| 122 | localLength: base.local.length, | |
| 123 | systemBatchLength: 0 | |
| 124 | } | |
| 125 | ||
| 126 | const sysView = base._viewStore.getSystemView() | |
| 127 | if (sysView.batch) { | |
| 128 | info.systemBatchLength = sysView.batch.length | |
| 129 | } | |
| 130 | ||
| 131 | await base.close() | |
| 132 | return info | |
| 133 | } | |
| 134 | ||
| 135 | function defaultOpen (store) { | |
| 136 | return new Hyperbee(store.get('view', { valueEncoding: 'json' }), { | |
| 137 | keyEncoding: 'utf-8', | |
| 138 | valueEncoding: 'json' | |
| 139 | }) | |
| 140 | } | |
| 141 | ||
| 142 | async function defaultApply (nodes, view, host) { | |
| 143 | await view.batch().flush() | |
| 144 | } |
| … | … | @@ -0,0 +1,116 @@ |
| 1 | /** | |
| 2 | * Scanner — find poisoned ops in Autobase writer cores. | |
| 3 | * | |
| 4 | * Walks every core in a corestore namespace looking for ops that match | |
| 5 | * a poison pattern (default: add-writer ops with non-Hypercore keys). | |
| 6 | * | |
| 7 | * Returns a list of { repoName, coreKey, poisonIndex, totalOps, op } | |
| 8 | */ | |
| 9 | ||
| 10 | export const DEFAULT_POISON_MARKER = Buffer.from('"type":"add-writer"') | |
| 11 | ||
| 12 | /** | |
| 13 | * Scan a single writer core for poisoned ops. | |
| 14 | * | |
| 15 | * @param {Corestore} ns - Namespaced corestore (e.g. store.namespace('linked:repo')) | |
| 16 | * @param {Object} opts | |
| 17 | * @param {Buffer} opts.marker - Binary marker to search for in raw blocks | |
| 18 | * @param {Function} opts.isPoisoned - (op) => boolean, validates decoded op | |
| 19 | * @returns {{ poisonIndex: number, op: object } | null} | |
| 20 | */ | |
| 21 | export async function scanWriterCore (ns, opts = {}) { | |
| 22 | const { | |
| 23 | marker = DEFAULT_POISON_MARKER, | |
| 24 | isPoisoned = defaultIsPoisoned | |
| 25 | } = opts | |
| 26 | ||
| 27 | const local = ns.get({ name: 'local' }) | |
| 28 | await local.ready() | |
| 29 | ||
| 30 | if (local.length === 0) { | |
| 31 | await local.close() | |
| 32 | return null | |
| 33 | } | |
| 34 | ||
| 35 | // Open raw session to read actual bytes | |
| 36 | const raw = ns.get({ key: local.key, valueEncoding: 'binary' }) | |
| 37 | await raw.ready() | |
| 38 | ||
| 39 | let result = null | |
| 40 | ||
| 41 | for (let i = 0; i < raw.length; i++) { | |
| 42 | try { | |
| 43 | const block = await raw.get(i) | |
| 44 | if (block.indexOf(marker) === -1) continue | |
| 45 | ||
| 46 | const op = extractJson(block) | |
| 47 | if (op && isPoisoned(op)) { | |
| 48 | result = { poisonIndex: i, totalOps: raw.length, op } | |
| 49 | break | |
| 50 | } | |
| 51 | } catch {} | |
| 52 | } | |
| 53 | ||
| 54 | await raw.close() | |
| 55 | await local.close() | |
| 56 | return result | |
| 57 | } | |
| 58 | ||
| 59 | /** | |
| 60 | * Default poison check: any add-writer op with a serverKey field. | |
| 61 | * In a healthy system, add-writer ops should use Hypercore local core keys, | |
| 62 | * not server identity keys. | |
| 63 | */ | |
| 64 | function defaultIsPoisoned (op) { | |
| 65 | return op.type === 'add-writer' && !!op.serverKey | |
| 66 | } | |
| 67 | ||
| 68 | /** | |
| 69 | * Extract JSON from an Autobase OplogMessage block. | |
| 70 | * The block has compact-encoded headers before the JSON payload. | |
| 71 | */ | |
| 72 | export function extractJson (block) { | |
| 73 | const start = block.indexOf(Buffer.from('{"type":')) | |
| 74 | if (start === -1) return null | |
| 75 | ||
| 76 | let end = block.length - 1 | |
| 77 | while (end > start && block[end] !== 0x7d) end-- | |
| 78 | if (end <= start) return null | |
| 79 | ||
| 80 | try { | |
| 81 | return JSON.parse(block.slice(start, end + 1).toString()) | |
| 82 | } catch { | |
| 83 | return null | |
| 84 | } | |
| 85 | } | |
| 86 | ||
| 87 | /** | |
| 88 | * List all ops in a writer core as decoded JSON objects. | |
| 89 | */ | |
| 90 | export async function listOps (ns) { | |
| 91 | const local = ns.get({ name: 'local' }) | |
| 92 | await local.ready() | |
| 93 | ||
| 94 | if (local.length === 0) { | |
| 95 | await local.close() | |
| 96 | return [] | |
| 97 | } | |
| 98 | ||
| 99 | const raw = ns.get({ key: local.key, valueEncoding: 'binary' }) | |
| 100 | await raw.ready() | |
| 101 | ||
| 102 | const ops = [] | |
| 103 | for (let i = 0; i < raw.length; i++) { | |
| 104 | try { | |
| 105 | const block = await raw.get(i) | |
| 106 | const op = extractJson(block) | |
| 107 | ops.push({ index: i, op, size: block.length }) | |
| 108 | } catch (err) { | |
| 109 | ops.push({ index: i, op: null, error: err.message }) | |
| 110 | } | |
| 111 | } | |
| 112 | ||
| 113 | await raw.close() | |
| 114 | await local.close() | |
| 115 | return ops | |
| 116 | } |
| … | … | @@ -0,0 +1,17 @@ |
| 1 | { | |
| 2 | "name": "autobase-doctor", | |
| 3 | "version": "1.0.0", | |
| 4 | "description": "Diagnose and repair corrupted Autobase instances", | |
| 5 | "type": "module", | |
| 6 | "main": "index.js", | |
| 7 | "bin": { | |
| 8 | "autobase-doctor": "./bin/autobase-doctor.js" | |
| 9 | }, | |
| 10 | "dependencies": { | |
| 11 | "autobase": "^7.27.3", | |
| 12 | "corestore": "^7.9.2", | |
| 13 | "hyperbee": "^2.22.5", | |
| 14 | "hypercore": "^11.7.3", | |
| 15 | "paparam": "^2.0.1" | |
| 16 | } | |
| 17 | } |