4fb0bca · hypercore-kb

Changed files (2) +392 -0

.claude/skills/autobee/SKILL.md +134 -0

…	…	@@ -0,0 +1,134 @@
	1	---
	2	name: autobee
	3	description: Multi-writer KV built on hyperbee2 — autobase's successor, with provable sort WEIGHTS instead of indexers and quorum. Read before adopting it, porting an autobase off it, or debugging a migration. Keywords: autobee, multiwriter, weights, quorum, indexer, hyperbee2, migration, optimistic, interrupt, apply.
	4	---
	5
	6	# autobee
	7
	8	Verified against the installed `autobee@2.1.0` + `hyperbee2@2.13.2` on 2026-09-06 (README + source
	9	under `node_modules/`). Anything sourced from the repo clone or the 2026-09-02 spike instead of the
	10	published package is marked ⚠ REMEMBERED.
	11
	12	`"A multi-writer KV-store built with Autobase and Hyperbee"` — but it has **zero autobase
	13	dependency**. It is a ground-up replacement built on `hyperbee2` + `hypercore ^11.33.2`.
	14
	15	## Why it exists — weights replace indexers and quorum
	16
	17	⚠ REMEMBERED (repo `WEIGHTS.md`, not shipped in the npm tarball; audited 2026-09-02 at rc.6):
	18	`lib/system.js` has `indexers: null // legacy` and the anchor manifest sets `quorum: 0`. In their
	19	place is a provable sort weight: `weight` (a monotone resolved floor), `maxWeight` (a ceiling
	20	granted by `addWriter`), and `isGenesis` (a self-certifying trust root at bootstrap). Heavier writers
	21	win reorgs. Weight is earned — backers sign `(NS_WITNESS ‖ backer ‖ grantee ‖ position ‖ weight)`
	22	attestations that ride the flush tail and are harvested during ordinary ingest, so verification is a
	23	pure function of a node's own bytes and needs no probing or remote reads.
	24
	25	Why that matters to us: autobase's majority is `floor(N/2)+1`, so N=2 needs both indexers —
	26	one goes offline and `indexedLength` freezes while the view keeps growing, and you cannot self-heal
	27	by dropping 2→1. Anywhere our writer set is naturally two (a person's two devices), N=2 is the
	28	normal posture rather than an edge case. Weights retire that whole class.
	29
	30	VERIFIED in 2.1.0: `isIndexer` survives as sugar over weight —
	31	`addWriter(key, { isIndexer = true, weight = isIndexer ? 2 : 1 })` (`lib/apply-calls.js:33`).
	32
	33	## Constructor
	34
	35	```js
	36	const db = new Autobee(store, key /* or null to create */, {
	37	apply (nodes, view, host) {}, // batches of new nodes → update the view
	38	open (bee, db) {}, // ⛔ receives the BEE, not a store. Return your view
	39	close (view) {},
	40	update (view, changes) {}, // after apply, when the view moved
	41	migrate (views, system) {}, // ⛔ REQUIRED when adopting an autobase log — see Migration
	42	encryptionKey: Buffer, // 32 bytes, at rest
	43	encrypted: false, // set true when passing encryptionKey
	44	keyPair, // custom local-writer signing key
	45	optimistic: true, // allow appends from not-yet-writers
	46	isTrusted (key, reference) {}, // fast-forward
	47	mostRecentTrusted (target, reference) {},
	48	fastForward: {}
	49	})
	50	```
	51
	52	`db.key` `db.discoveryKey` `db.id` `db.local` (the local writer core) `db.view` / `db.bee` (a
	53	read-only snapshot) `db.writable` `db.isIndexer` · `append(value\|values, [{optimistic}])`
	54	`update()` `updated()` `flush()` `replicate(isInitiator)` `wakeup({key,length})` `setLocal(key)`
	55	`views()` `moveTo(head)` · statics `Autobee.isAutobee`, `encodeValue`, `decodeValue`,
	56	`GENESIS = { length: 0, key: null }`.
	57
	58	## ⛔ Three API traps when porting from autobase
	59
	60	1. `node.key`, not `node.from.key`. In apply, a node is `{ key, value, length }` where `key` is
	61	the writer's public key and `length` its position in that writer's core. autobase puts the writer
	62	under `node.from`. Attribution is the one thing you must never take from the op body, so getting this
	63	wrong is silent misattribution, not a crash.
	64
	65	2. The `view` in apply is a WRITABLE hyperbee2 batch; `db.view` is a read-only snapshot. Two
	66	different objects with the same name.
	67
	68	3. hyperbee2 has no `put`/`del` and no `keyEncoding`/`valueEncoding`. Keys and values are raw
	69	Buffers. Every mutation is a batch:
	70
	71	```js
	72	const w = view.write()
	73	w.tryPut(Buffer.from(key), Buffer.from(JSON.stringify(value)))
	74	w.tryDelete(Buffer.from(key))
	75	await w.flush()
	76	```
	77
	78	and `await bee.get(Buffer.from(key))` returns `{ key, value }` as Buffers, or `null`. A codec shim is
	79	the price every autobase→autobee port pays once per view — 33 of the 46 lines in the NameRegistry
	80	spike. ⚠ NOT MEASURED: that shim flushes a batch per mutation, and nobody has benchmarked it. Inside
	81	a hot apply loop that is a flush per op.
	82
	83	## host, inside apply
	84
	85	- `host.addWriter(key, { isIndexer = true })` — weight 2 when `isIndexer`, else 1.
	86	- `host.removeWriter(key)` · `host.ackWriter(key)` — ack confirms an optimistic writer's block
	87	without granting permissions.
	88	- `host.interrupt(reason)` — halts the apply cycle and emits `'interrupt'`. This is the in-band
	89	loud-halt guard, available natively rather than hand-rolled.
	90	- `host.createAnchor()` → `{ key, length }` — a verifiable checkpoint later writers can use to prove
	91	causal ordering.
	92	- ★ `host.genesis` — `true` while the system has processed no nodes. **Use this to bootstrap the
	93	first writer** instead of inferring genesis from an empty roster.
	94
	95	## ⛔ Migration — the trap that looks like success
	96
	97	⚠ REMEMBERED (spike, 2026-09-02, measured at rc.30). `checkAutobaseMigration()` adopts an existing
	98	autobase log in place — the key is preserved, the same writer core continues, and a mixed fleet
	99	is tolerated by majority vote across indexers.
	100
	101	> A missing `migrate` handler is silent TOTAL STATE LOSS. With `legacyViews: ['view']` and no
	102	> `migrate`, the run reports key preserved, `writable: true`, apply never fires, and
	103	> `bee.core.length: 0`. It looks like a clean migration. For a single-view consumer the fix is one
	104	> line: `migrate: (views) => views.get('view')`.
	105
	106	⛔ Mixed fleet is ONE-WAY and silent — old→new is fine, new→old never works. Writer sets need a
	107	flag day, though announcing an `interrupt` on guard-aware peers before migrating turns it into a
	108	loud halt. Migration keeps sort weight 2 but not the `maxWeight` capability, so writers come back
	109	`isIndexer: false`; the fix is one `addWriter({ isIndexer: true })` re-grant per writer.
	110
	111	★ Every hazard above is a MIGRATION hazard. A greenfield log pays none of them — which is why
	112	adopting autobee is cheap at the start of a project and expensive later.
	113
	114	## What autobee does NOT solve
	115
	116	It is a KV view. Multi-writer settles which records exist, never where the bytes are. A
	117	shared-folder design still needs pointers in the log and content-addressed bytes outside it — and the
	118	byte layer is where our production incidents actually lived.
	119
	120	## Port cost, measured
	121
	122	pgh `NameRegistry`, autobase → autobee: 46 changed lines, 33 of them the codec shim, 7/7 tests
	123	green before and after. The entire apply body, all op handlers, every read method and both
	124	`addWriter`/`removeWriter` calls ported verbatim — `host` sits in the same positional slot with
	125	the same method names.
	126
	127	⇒ If your apply is written against an abstract async `{ get, set }` view and your KV is injected,
	128	the port is confined to the one file that constructs the base.
	129
	130	## Related
	131
	132	`hyperbee2` (the view, and its batch API) · `autobase` (what this replaces) ·
	133	`autobee-wakeup` · memory: `reference_autobee2-vs-our-autobase`,
	134	`project_autobee-migration-todo-2026-09-02`, `reference_autobase-quorum-2f1`.

.claude/skills/hyperbee2/SKILL.md +258 -0

…	…	@@ -0,0 +1,258 @@
	1	---
	2	name: hyperbee2
	3	description: Next-gen append-only B-tree on Hypercore — Buffers only (no keyEncoding/valueEncoding), writes go through a WriteBatch, and it is the view autobee hands you. Read before porting hyperbee 1.x code, debugging a close() throw, or writing an autobee apply. Keywords: hyperbee2, writebatch, tryPut, flush, autobee, diff stream, changes stream, checkout, move, undo.
	4	---
	5
	6	# hyperbee2
	7
	8	Next-generation append-only B-tree on top of a [Hypercore][hypercore].
	9	Will eventually merge back into mainline hyperbee as a new major. Today
	10	it ships as its own package — used by [bundlebee](../bundlebee/SKILL.md),
	11	@holepunchto/keet-core (transitively, via 4.14.x), and anywhere a
	12	hyperbee with batched writes + diff streams + cohorted nodes is wanted.
	13
	14	Source: `holepunchto/repos/tier3/hyperbee2/`. Auto-extracted stub:
	15	`holepunchto/skills/hyperbee2/`.
	16
	17	Installed 2.13.2, checked 2026-09-06. The body of this skill was written
	18	against 2.9.x on 2026-05-13 and ⚠ **the 2.9 → 2.13 delta has not been
	19	audited** — the API shown below was re-confirmed against 2.13.2's README and
	20	`lib/write.js`, but changes between those points were not reviewed. Treat
	21	anything here that a source read did not touch as REMEMBERED.
	22
	23	★ It is now also autobee's view. `autobee@2.x` is built on hyperbee2 and
	24	hands it to you directly — see "With autobee" below and the
	25	[autobee](../autobee/SKILL.md) skill.
	26
	27	## When to read this skill
	28
	29	- Touching anything that imports `hyperbee2` directly (rare — most
	30	consumers see it through bundlebee or keet-core)
	31	- Debugging a hyperbee close() that throws — `close()` calls a
	32	specific static helper on Hypercore (see "Close path" below) and
	33	the wrong polyfill / pinned-version mismatch surfaces here first
	34	- Writing or porting code that used hyperbee 1.x semantics — write
	35	batching, `view`, `move`, `unbatch`, encoding, and the change-stream
	36	shape are all different
	37	- Investigating "bundled hyperbee2 inside keet-core" issues — the
	38	baked copy may pin a different point release than the on-disk
	39	`node_modules/hyperbee2` and the two can diverge on internal API names
	40
	41	## Core idea (vs hyperbee 1.x)
	42
	43	\| \| hyperbee 1.x \| hyperbee2 \|
	44	\|---\|---\|---\|
	45	\| Write API \| per-call `put`/`del` \| `WriteBatch` (deferred-lock by default) — `db.write()`, `batch.tryPut`, `batch.flush` \|
	46	\| Views \| `snapshot()`, `sub()` \| `snapshot()`, `checkout({length, key})`, `move({length, key})`, `undo(n)`, `_makeView` for views \|
	47	\| Diff \| `createDiffStream` (legacy) \| `createDiffStream(right)` over `RangeStream` with shared inflate path \|
	48	\| Change history \| n/a \| `createChangesStream` walks block-by-block, yields `{ head, tail, batch: [{tree, keys, values, cohorts, metadata}] }` \|
	49	\| Node cache \| implicit \| explicit `NodeCache` (default 4096) — `db.cache` exposes it \|
	50	\| Activity tracking \| n/a \| `config.activeRequests` — passed down so close() can cancel in-flight reads \|
	51	\| Encoding \| self-describing \| versioned via `encoding.TYPE_LATEST`, `compat: true` writes hyperbee-1 blocks \|
	52	\| Encryption \| per-bee key \| per-key provider — `getEncryptionProvider(key)` resolves at session time \|
	53	\| Inline vs pointer values \| always inline \| `inlineValueSize` (default 1024) — bigger values get pointered \|
	54
	55	## Close path — the load-bearing detail
	56
	57	`async close()` (current source, `index.js:166-173`):
	58
	59	```js
	60	async close() {
	61	if (!this.root) await this.ready()
	62	if (this.config.activeRequests.length) Hypercore.clearRequests(this.config.activeRequests)
	63	if (this.view) return
	64	if (this.store.closing) return this.store.close()
	65	await this.store.close()
	66	this.emit('close')
	67	}
	68	```
	69
	70	**The cleanup call is `Hypercore.clearRequests(activeRequests)` — single
	71	argument, static method on the Hypercore class.** It exists natively on
	72	hypercore 11.x at `index.js:194` (`static clearRequests(session, err)`).
	73
	74	⚠ It is NOT `Hypercore.destroyRequests(activeRequests, null)` — that
	75	name doesn't appear anywhere in upstream hyperbee2 (verified across
	76	2.9.1 + 2.9.3 + installed copies). If you find a polyfill or bundle
	77	rewrite targeting `destroyRequests`, that polyfill is a no-op for
	78	modern hyperbee2 — the rename either happened upstream or the comment
	79	in your codebase is stale. Check the actual node_modules/hyperbee2
	80	source for the method name before patching.
	81
	82	Closing a `view: true` Hyperbee is a no-op (it shares storage with the
	83	parent). Closing a writer-Hyperbee also closes its `Corestore`.
	84
	85	## ⛔ The first thing to know: Buffers only
	86
	87	There is no `keyEncoding` and no `valueEncoding`, and no `put` / `del`.
	88	Keys and values are raw Buffers, and every mutation goes through a WriteBatch.
	89	That is the single biggest porting cost from hyperbee 1.x, and from anything
	90	built on it — a codec shim was 33 of the 46 changed lines when pgh's
	91	`NameRegistry` moved from autobase+hyperbee to autobee+hyperbee2.
	92
	93	⚠ Not measured: the obvious shim flushes a batch per mutation. Nobody has
	94	benchmarked that, and inside a hot loop it is a flush per operation.
	95
	96	## Core API (verified against 2.13.2)
	97
	98	```js
	99	const Hyperbee = require('hyperbee2')
	100	const Corestore = require('corestore')
	101
	102	const db = new Hyperbee(new Corestore('./store'))
	103	await db.ready()
	104
	105	// Write: build a batch, flush atomically
	106	const w = db.write()
	107	w.tryPut(Buffer.from('a'), Buffer.from('1'))
	108	w.tryPut(Buffer.from('b'), Buffer.from('2'))
	109	w.tryDelete(Buffer.from('old'))
	110	await w.flush()
	111
	112	// Read: single key
	113	const entry = await db.get(Buffer.from('a'))
	114	// entry = { core, offset, seq, key, value } or null
	115
	116	// Range scan
	117	for await (const { key, value } of db.createReadStream({ gte, lt, limit })) {}
	118
	119	// Diff between this and right tree
	120	for await (const { left, right } of db.createDiffStream(otherDb)) {}
	121
	122	// Walk every batch that ever wrote to this tree
	123	for await (const { head, tail, batch } of db.createChangesStream()) {}
	124
	125	// Read-only snapshot at current head
	126	const snap = db.snapshot()
	127
	128	// Read-only view at historical length
	129	const past = db.checkout({ length: 42 })
	130
	131	// Replace root with a new core (e.g. after a move/migration)
	132	db.move({ length: newLength, key: newKey })
	133
	134	// Roll back the last N write batches without rewriting blocks
	135	const undone = db.undo(2)
	136
	137	await db.close()
	138	```
	139
	140	## Constructor options that bite
	141
	142	```js
	143	new Hyperbee(store, {
	144	key: null, // load hypercore at this key, else `name: 'bee'`
	145	core: existingCore, // bring your own Hypercore inside the store
	146	view: false, // true = don't close store on close()
	147	writable: true, // throws on db.write() when false
	148	unbatch: 0, // rollback N batches at bootstrap
	149	autoUpdate: false, // re-root on remote append. Forced to !writable && !view
	150	preload: null, // async hook run inside ready()
	151	encryption: null, // or `getEncryptionProvider` for per-key
	152	maxCacheSize: 4096, // NodeCache capacity
	153	activeRequests: [], // session-config tracked array — close() drains this
	154	timeout: 0, wait: true, trace: null
	155	})
	156	```
	157
	158	`activeRequests` is shared via `SessionConfig` — child sessions
	159	inherit the same array unless they pass their own. This is the array
	160	`close()` hands to `Hypercore.clearRequests` to abort in-flight reads.
	161
	162	**`db.opening` / `db.opened` / `db.closing` / `db.closed` are
	163	pass-throughs to `db.store` (the Corestore)** — not the Hyperbee
	164	itself. Naming preserved from hyperbee 1.x but the semantics are
	165	store-level.
	166
	167	## WriteBatch — the bite-size details
	168
	169	`db.write([opts])` returns a `WriteBatch` (`lib/write.js`). Built up
	170	in memory with `tryPut`/`tryDelete`/`tryClear`, then `flush()` takes
	171	the exclusive lock and writes one block. Until `flush()` runs, the
	172	batch holds no lock — two concurrent batches can interleave unless
	173	one of them calls `await batch.lock()` first.
	174
	175	⚠ Calling `flush()` twice on the same batch is destructive. The
	176	second flush will roll back blocks the first flush wrote — see the
	177	"flushing w1 again will unapply w2" warning in the upstream README. If
	178	you need to extend a batch after flush, build a new one.
	179
	180	Block size options on `db.write`:
	181	- `inlineValueSize: 1024` — values smaller stay inline in the node;
	182	larger values get written as a separate pointer block
	183	- `preferredBlockSize: 4096` — flush splits the batch to approximate
	184	this target
	185	- `compat: false` — write hyperbee-1-compatible blocks (round-tripable
	186	with the old library)
	187	- `type: encoding.TYPE_LATEST` — explicit block format override
	188
	189	## Module layout (`lib/`)
	190
	191	\| File \| Purpose \|
	192	\|------\|---------\|
	193	\| `index.js` \| Hyperbee class + `WriteBatch` wiring + close path \|
	194	\| `lib/context.js` \| `CoreContext` — owns the Hypercore + NodeCache + per-key encryption resolver \|
	195	\| `lib/session-config.js` \| `SessionConfig` — `activeRequests` lifecycle, sub-session inheritance \|
	196	\| `lib/cache.js` \| `NodeCache` — LRU of decoded tree nodes \|
	197	\| `lib/inflate.js` \| Decode tree nodes / values from raw blocks \|
	198	\| `lib/tree.js` \| `EMPTY` sentinel, tree-node creation, `createTreeNode` \|
	199	\| `lib/ranges.js` \| `RangeStream` — drives `createReadStream` and `peek`/`download` \|
	200	\| `lib/diff.js` \| `DiffStream` — drives `createDiffStream` \|
	201	\| `lib/changes.js` \| `ChangesStream` — drives `createChangesStream` \|
	202	\| `lib/write.js` \| `WriteBatch` \|
	203	\| `lib/encoding.js` \| Block format / `TYPE_LATEST` \|
	204	\| `lib/compression.js` \| Per-key cohort compression \|
	205	\| `lib/compat.js` \| Hyperbee-1 round-trip helpers \|
	206
	207	## Events
	208
	209	- `ready` — emitted once at the end of `ready()`
	210	- `update` — fired on every flush (when `autoUpdate: true` on the bee
	211	AND/OR on the WriteBatch). ⚠ can fire twice for one update if
	212	both flags are set
	213	- `close` — emitted at the end of `close()` (skipped for views)
	214
	215	## Common gotchas
	216
	217	- Don't reuse a flushed WriteBatch. See above.
	218	- `autoUpdate` is two flags — one on the Hyperbee (re-root on
	219	remote append), one on the WriteBatch (emit `update` after flush).
	220	They are independent.
	221	- `store.close()` cascades to every Hyperbee on that store. If you
	222	want close() to leave the store open, construct with `view: true`.
	223	- `checkout({ length })` is read-only by default — pass
	224	`writable: true` to make a writable historical fork (rare).
	225	- `activeRequests` is an array, not a Map. `close()` checks
	226	`.length` and passes the whole array to `Hypercore.clearRequests`.
	227	- Bundled vs npm-installed copies can diverge. When hyperbee2 is
	228	baked into a larger bundle (e.g. via [bundlebee](../bundlebee/SKILL.md)
	229	inside `@holepunchto/keet-core`), the host `node_modules/hyperbee2`
	230	is NOT what runs. Patches against the wrong copy silently no-op.
	231
	232	## With autobee
	233
	234	`autobee` calls `open(bee, db)` with a hyperbee2 instance — not a corestore,
	235	which is what an autobase `open(store)` handler receives. Two objects share the
	236	name `view` and they are different things:
	237
	238	\| \| what it is \|
	239	\|---\|---\|
	240	\| the `view` argument inside `apply(nodes, view, host)` \| a writable hyperbee2 batch — `view.write()` → `tryPut` / `tryDelete` → `await flush()` \|
	241	\| `db.view` (alias `db.bee`) \| a read-only snapshot, updated after each apply cycle \|
	242
	243	Writing to the snapshot, or expecting the apply-time view to be readable
	244	elsewhere, is the mistake this table exists to prevent.
	245
	246	## Related
	247
	248	- [bundlebee](../bundlebee/SKILL.md) — uses a Hyperbee2 as its backing
	249	store
	250	- [autobee](../autobee/SKILL.md) — multi-writer on top of this; weights
	251	instead of indexers/quorum, and the source of the `open(bee, db)` hook
	252	- [hyperbee](../hyperbee/SKILL.md) — the stable 1.x version still in
	253	wide use; hyperbee2 is its forward port
	254	- [corestore](../corestore/SKILL.md) — wraps the underlying Hypercore
	255	- hypercore 11.x — owns `static clearRequests(session, err)`
	256	(`node_modules/hypercore/index.js:194`)
	257
	258	[hypercore]: https://github.com/holepunchto/hypercore

hypercore-kb commits / 4fb0bca

Changed files (2) +392 -0