3a0100d · hyper-vdrive

Changed files (5) +552 -427

README.md +40 -128

…	…	@@ -2,172 +2,84 @@
2	2
3	3	A real Hyperdrive that stores no blob bytes.
4	4
5		The drive is genuine all the way up — real Hyperbee metadata, real Hyperblobs core,
6		real merkle tree, real signatures, stock `hyperdrive` / `hyperblobs` / `hypercore`
7		with no patches. Only the storage layer underneath is swapped: block bytes are never
8		written to RocksDB. Instead each block record holds a 32-byte content hash, and the
9		bytes are read on demand from a content-addressed store already on disk.
	5	Real Hyperbee metadata, real Hyperblobs core, real merkle tree, real signatures,
	6	stock `hyperdrive` / `hyperblobs` / `hypercore` unpatched. Only the storage layer
	7	underneath is swapped: block bytes are never written to RocksDB. Each block record
	8	holds a 32-byte content hash instead, and the bytes are read on demand from a
	9	content-addressed store already on disk.
10	10
11		Peers can't tell. `pear install pear://<key>` works against it.
	11	Peers cannot tell. `pear install pear://<key>` works against it.
12	12
13	13	## Why
14	14
15	15	We already store every file once, chunked and content-addressed, in `hyper-overlay`.
16		Publishing the same content as a Hyperdrive meant a second full copy in RocksDB. This
17		removes the second copy while keeping `pear://` compatibility.
18
19		Measured on ten consecutive real 367 MB game-pack releases:
	16	Publishing the same content as a Hyperdrive meant a second full copy in RocksDB.
	17	This removes the second copy and keeps `pear://` compatibility.
20	18
21	19	\| \| stock drive \| virtual drive \|
22	20	\|---\|---\|---\|
23		\| store after 10 releases \| 580.8 MB \| 0.88 MB \|
24		\| publish time per release \| 3.6 s \| 2.4 s \|
	21	\| ten 367 MB releases \| 580.8 MB \| 0.88 MB \|
	22	\| app tree, 976 files / 209 MB \| 210.1 MB \| 1.38 MB \|
	23	\| three 49 MB AppImages \| 148 MB \| 0.52 MB \|
	24	\| publish time \| 3.6 s \| 2.4 s \|
25	25	\| drive key \| — \| identical \|
26	26
27		And on an app tree — 976 files, 209 MB, median 2 KB — 210.1 MB becomes 1.38 MB
28		(152×). Small files dilute the win because metadata, not bytes, becomes the cost:
29		the floor is ~1.4 KB per file.
30
31		See [RESULTS.md](RESULTS.md) for the full measurements and method. Measure stores
32		compacted (`test/compact-size.mjs`) — RocksDB's raw footprint includes write churn
33		that dominates a small store.
34
35		## How
	27	Proven end to end: a real `pear install` on another machine, over the public DHT,
	28	byte-identical, from a drive holding zero blobs.
36	29
37		`Core` takes its per-core storage from exactly two calls
38		(`hypercore/lib/core.js`): `db.resumeCore(discoveryKey)` and `db.createCore({...})`.
39		Subclassing `CorestoreStorage` and wrapping those two return values puts us underneath
40		every block read and write, with nothing above aware.
41
42		Stored block records carry a one-byte tag:
	30	## Install
43	31
44	32	```
45		0x00 \| <original bytes> literal — bee blocks, and anything we don't hold
46		0x01 \| <32-byte hash> pointer — bytes live in the content store
	33	npm install hyper-vdrive
47	34	```
48	35
49		Because the pointer is the block value, it commits in the same `CoreTX` as the tree
50		node — tree and pointer land atomically, and `deleteBlock` / truncate free it with no
51		extra bookkeeping. Cost: 1 byte per literal block, 33 bytes per virtualised block.
52
53		The chunk hash is `hypercore-crypto`'s `data()` — the same construction hypercore uses
54		for a merkle leaf (`merkle-tree.js`). So our existing FastCDC chunk hashes are
55		already the leaf hashes, and a chunk maps directly onto a block.
56
57		## Usage
58
59		Against a real hyper-overlay index:
	36	## Use
60	37
61	38	```js
62	39	const { build } = require('hyper-vdrive')
63	40	const { OverlayContentStore } = require('hyper-vdrive/overlay')
64	41
65		const content = new OverlayContentStore(fileIndex, {
66		roots: [repoRoot], // working tree — bytes at <root>/<path>
67		syncBaseDirs: [spoolDir] // content spool — bytes at <dir>/<oid>, for mirrors
68		})
69		await content.indexAll() // or indexPath(p) / indexContent(oid) per artifact
	42	const content = new OverlayContentStore(fileIndex, { roots: [repoRoot] })
	43	await content.indexAll()
70	44
71		const store = new Corestore(new build(content)(dir))
	45	const store = new Corestore(new (build(content))(dir))
72	46	const drive = new Hyperdrive(store)
73	47	```
74	48
75		Or standalone, if you are managing locations yourself:
	49	Publishing and serving, which is what most callers want:
76	50
77	51	```js
78		const { ContentStore, build } = require('hyper-vdrive')
79		const content = new ContentStore()
80		for (const c of chunk(buf)) content.add(c.hash, file, c.offset, c.length)
81		```
82
83		Note on the reverse index. `FileIndex` maps path → chunks and oid → chunks; it
84		has no chunk hash → location index. `OverlayContentStore` builds that in memory by
85		reading the chunk maps of the paths you register, which is O(total chunks) — fine for a
86		publish or a seeder boot, not something to redo per request. A persistent reverse index
87		in the overlay would remove it.
	52	const { publishTree } = require('hyper-vdrive/publish')
	53	const { openDrive } = require('hyper-vdrive/serve')
88	54
89		Write files chunk-by-chunk so block boundaries are your chunk boundaries:
90
91		```js
92		const ws = drive.createWriteStream('/app.bin', { dedup: true })
93		for (const c of chunks) ws.write(c.data) // honour backpressure
94		ws.end()
	55	await publishTree({ storeDir, root, chunk, files }) // writes + compacts
	56	const opened = await openDrive({ storeDir, root }) // reopen to serve
95	57	```
96	58
97	59	## CLI
98	60
99	61	```
100		vdrive publish <srcDir> --store <driveDir> [--only a,b]
	62	vdrive publish <srcDir> --store <driveDir> [--root <dir>] [--only a,b]
101	63	vdrive info --store <driveDir>
102	64	vdrive seed --store <driveDir> --root <srcDir> [--timeout <seconds>]
103	65	```
104	66
105		The source tree is the byte store — the drive holds pointers, and reads are served
106		from the files under `<srcDir>` (or `--root` when seeding). Publishing is incremental:
107		re-publishing into the same `--store` writes only what changed.
108
109		End-to-end, verified against the stock `pear` CLI:
	67	The source tree is the byte store — the drive holds pointers, and reads are
	68	served from the files beneath it. Republishing into the same store is incremental.
110	69
111		```
112		$ vdrive publish ./app --store ./appdrive
113		link: pear://1a1kwj1u8ygcnpw1czu57wya6ym7fpp7ji93gy5qapnau43ubgjo
114		files: 2 bytes: 4194432 blocks: 53
115		pointerBlocks: 51 literalBlocks: 5 driveStoreBytes: 133277
116
117		$ vdrive seed --store ./appdrive --root ./app &
118		$ pear install pear://1a1kwj1u8ygcn... --to ./installed
119		Installed # byte-identical, executable, runs
120		```
	70	## Docs
121	71
122		For a `pear install`-able drive the tree needs `/package.json` (with `name` and `bin`)
123		and `/by-arch/<platform>-<arch>/app/<bin>`. `vdrive` does not impose that layout — it
124		publishes whatever tree you point it at.
	72	- [docs/design.md](docs/design.md) — how it works: the storage seam, the tagged
	73	block records, the locations index, and why `pear seed` cannot serve one of these.
	74	- [docs/results.md](docs/results.md) — every measurement, with method and source
	75	citations.
	76	- [docs/gaps.md](docs/gaps.md) — what is unproven or known-broken. Read this
	77	before relying on it.
125	78
126		Note: `seed` prints its served-bytes line on SIGINT/SIGTERM, but that is best-effort —
127		under a hard kill or a process-group teardown it will not appear.
	79	## Status
128	80
129		## Tests
130
131		Tests import the FastCDC chunker from a sibling `hyper-overlay` checkout
132		(`../../hyper-overlay/lib/chunker.js`) rather than vendoring a copy that could drift.
133
134		```
135		npm install
136		node test/pass-through.mjs ./work/passthrough # seam holds, identical drive key
137		node test/redirect.mjs ./work/redirect <f1> <f2>
138		node test/releases.mjs ./work/releases <f...> # N-release size curve
139		node test/gc-bitfield.mjs ./work/gc # GC / bitfield pairing
140		node example/publish.mjs ./work/app # pear-install-shaped drive
141		node example/seed.mjs ./work/app # serve it on the DHT
142		```
	81	A working spike, running in `pear-git-v3` behind an opt-in `vdrive:` key in
	82	`ci.yaml`. It publishes, seeds and installs for real. The gaps are honest ones,
	83	and encryption defeats it entirely — see `docs/gaps.md`.
143	84
144		## Status — spike
145
146		Proven: identical drive key/blobs key/root hash, reopen from disk, replication to a
147		stock peer, a real `pear install` from a drive holding zero blobs, parity through
148		clear/truncate, and crash-during-append leaving no tree node without its pointer.
149
150		Known gaps, in the order they'd bite:
151
152		1. GC should be paired, but need not be atomic. Dropping bytes does not update the
153		bitfield, so the core goes on advertising blocks it cannot produce. Measured with two
154		seeders, that is survivable — the replicator routes around it and a second source
155		serves the data in the same ~120 ms. So clearing the block range is hygiene rather
156		than a correctness requirement, and can be done opportunistically. What does matter:
157		with `dedup: true`, old releases share blocks with new ones, so ranges must be
158		computed from the content store's refcounts, never from a version's block range.
159		2. Block streams (`createBlockStream`, used by audit/copy-prologue) yield raw tagged
160		records; they need decoding like `getBlock` does.
161		3. Encryption defeats virtualisation — measured: 0 of 105 blocks redirect, the store
162		goes from 0.14 MB to 8.53 MB. Correctness is unaffected (fall-through-by-hash
163		degrades to a normal drive). Recoverable in principle: the cipher is deterministic
164		per (index, fork, key), so an adapter holding the key could decrypt on write and
165		re-encrypt on read, with dedup surviving because hyperblobs hashes plaintext. Until
166		then, publish public apps unencrypted.
167		4. Unsupported seam. `hypercore-storage` has no public pluggable-storage contract
168		since `random-access-storage` was dropped. Pin versions and run the tests as a canary.
169		5. clear-then-truncate is unsupported upstream — `clear()` prunes merkle tree nodes,
170		so a later `truncate()` fails with `Expected tree node N from storage, got (nil)`.
171		Same on a stock drive. And `clear()` reclaims no disk promptly: on a virtual drive
172		that does not matter, because the bytes are the overlay's to free.
173		6. Untested: power loss — SIGKILL is covered and passes, fsync durability is not.
	85	AGPL-3.0-or-later.

RESULTS.md +0 -299

…	…	@@ -1,299 +0,0 @@
1		# Measurements
2
3		> Measure compacted. RocksDB's on-disk footprint includes uncompacted WAL/SST
4		> churn, which is noise on a big store and dominant on a small one. The small-file
5		> store below read 14.4 MB before compaction and 1.38 MB after. The large-file
6		> stores barely moved (±0.05 MB), so those figures were already steady-state.
7		> `test/compact-size.mjs` does the compact-then-measure.
8
9		All runs on tabertron, 2026-08-24. Source data: ten consecutive real `hypertps.pck`
10		release artifacts (~367 MB each) from `~/.local/share/store/dependencies`, and
11		synthetic files where noted. Chunker: `hyper-overlay/lib/chunker.js` (FastCDC,
12		tier 2 → ~320 KB average chunk).
13
14		## T0 — does stock Pear read chunk-sized blocks?
15
16		The installed `pear` 3.0.1 bundle (`~/.config/pear/current/boot.bundle`) contains
17		`node_modules/hyperblobs/lib/block-map.js` and the marker strings `Block map is too
18		large`, `Unsupported block map version`, `_dedup`, `getBlockMap`. So the shipped
19		reader supports block maps and variable-size blocks. Pass.
20
21		## T1 — storage cost, one release pair
22
23		Two consecutive releases, 367.06 MB each, into one drive at the same path.
24
25		\| Arm \| after v1 \| after v2 \| v2 delta \|
26		\|---\|---\|---\|---\|
27		\| overlay (control) \| 360.1 MB \| 378.3 MB \| 18.2 MB \|
28		\| hyperdrive, no dedup \| 368.3 MB \| 736.4 MB \| 368.1 MB \|
29		\| hyperdrive, `dedup:true`, fixed 64 KB \| 367.6 MB \| 488.7 MB \| 121.2 MB \|
30		\| hyperdrive, `dedup:true`, FastCDC \| 360.5 MB \| 378.7 MB \| 18.2 MB \|
31
32		Two corrections to earlier assumptions:
33
34		- RocksDB per-byte overhead is 0.3%, not a multiplier (368.3 MB stored for
35		367.06 MB in). The corestore column family enables blob files with GC.
36		- Hyperblobs does dedup, at block level, via `{dedup:true}` + block maps
37		(`hyperblobs/lib/streams.js:86-101`, `hyperdrive/index.js:591-596`). A stock drive
38		fed FastCDC chunks matched the overlay to within 0.1%.
39
40		## T1c — ten releases, stock drive vs overlay
41
42		\| Release \| drive \| overlay \| gap \|
43		\|---\|---\|---\|---\|
44		\| 1 \| 360.5 MB \| 360.1 MB \| +0.4 \|
45		\| 5 \| 456.1 MB \| 453.2 MB \| +2.9 \|
46		\| 6 \| 484.8 MB \| 471.4 MB \| +13.4 \|
47		\| 10 \| 580.8 MB \| 563.3 MB \| +17.5 (3.1%) \|
48
49		The gap appears at release 6 — a chunk set that vanished and returned. `dedup:true`
50		seeds its hash table from only the previous entry at that path
51		(`hyperdrive/index.js:592`), so resurrected chunks are re-appended. Publish time was
52		flat at ~3.6 s: the seeding re-read is O(previous release), not O(history).
53
54		## T2 — does the storage seam hold?
55
56		Pass-through wrapper (delegates everything, redirects nothing):
57
58		```
59		identical drive key / blobs key / root hash / digests true
60		reopen from disk true
61		replication to a stock peer true
62		intercepted 111 getBlock, 111 putBlock
63		```
64
65		Nothing escaped the wrapper. Pass.
66
67		## T3 — redirect, two releases
68
69		```
70		sameDriveKey / sameBlobsKey / sameRootHash true
71		blobsLength stock 2244 virtual 2244
72		input 734.0 MB
73		stock store 720.8 MB
74		virtual store 0.58 MB (compacted; tagged-record scheme, no side file)
75		peer store 720.7 MB (stock peer replicated from the virtual drive)
76		blocks 2242 redirected, 5 literal (the bee's own)
77		hash failures 0
78		```
79
80		## T3n — ten releases, virtual
81
82		\| Release \| blocks \| store \| stock equivalent \|
83		\|---\|---\|---\|---\|
84		\| 1 \| 1122 \| 0.50 MB \| 360.5 MB \|
85		\| 5 \| 1368 \| 0.50 MB \| 456.1 MB \|
86		\| 10 \| 1682 \| 0.88 MB \| 580.8 MB \|
87
88		3.67 GB of releases → 0.88 MB compacted (657×). Publish ~2.4 s/release, faster than stock
89		because the RocksDB block writes disappear. Pointer overhead is 36 bytes per block —
90		≈0.011% of content at tier-2 chunk sizes.
91
92		## T5 — real `pear install`
93
94		Built a `/by-arch/linux-x64/app/` drive on virtual storage (6.29 MB payload, 120 KB
95		store), seeded it on the public DHT, and ran the stock `pear` 3.0.1 CLI:
96
97		```
98		$ pear install pear://uimennwjwyot5cinekhb348kawufit1eq4ctyf5irurn1nfmo3so --to …
99		[ Peers: 1 ] [ ⬇ 2.8MB - 2.8MB/s ]
100		App: t5app Version: 1.0.0
101		Target: …/t5/installed/t5app
102		Installed
103		```
104
105		Installed file: 6,291,480 bytes, mode 755, hash identical to source, runs. Seeder
106		store unchanged at 139 KB; it served 6,291,560 bytes as 89 redirected block reads
107		with 0 hash failures. Pass.
108
109		## Many small files
110
111		976 files, 209.1 MB, median 2 KB, largest 11 MB (this repo's own `node_modules`),
112		plus three synthetic 0-byte files to cover that case.
113
114		\| \| stock \| virtual \|
115		\|---\|---\|---\|
116		\| store (compacted) \| 210.1 MB \| 1.38 MB \|
117		\| ratio \| \| 152× \|
118		\| per file \| \| 1,415 bytes \|
119		\| drive key \| identical \| identical \|
120		\| readback mismatches \| 0 \| 0 \|
121		\| publish \| 4.2 s \| 2.7 s \|
122
123		Blocks: 4,590 content chunks redirected, 1,950 stored literally. The literal ones
124		are the bee's own blocks (977) plus roughly one block-map block per file — with
125		`dedup: true` hyperblobs appends an encoded block map after each file's data, and that
126		map is generated metadata, not content we hold, so it correctly falls through to real
127		storage.
128
129		So the floor for an app tree is ~1.4 KB per file regardless of file size: bee entry +
130		block-map block + tree nodes + pointer record. A 10,000-file tree costs ~14 MB.
131		Small files dilute the win (152× rather than the 657× of large artifacts) because
132		metadata, not bytes, becomes the cost.
133
134		## truncate / clear
135
136		Three files (v1 and v2 of the same path, which share blocks under `dedup:true`, plus
137		an unrelated file), then `drive.clear('/app.bin')`, then a truncate of the blobs core.
138		Sizes are compacted.
139
140		\| \| stock \| virtual \|
141		\|---\|---\|---\|
142		\| after publish \| 6.81 MB \| 0.13 MB \|
143		\| after `clear('/app.bin')` \| 6.90 MB \| 0.22 MB \|
144		\| after attempted truncate \| 6.98 MB \| 0.30 MB \|
145		\| `has()` on a cleared block \| false \| false \|
146		\| read the cleared entry \| REQUEST_TIMEOUT \| REQUEST_TIMEOUT \|
147		\| read the untouched entry \| ok \| ok \|
148
149		Parity: the virtual drive behaves exactly like the stock one. Two findings, both
150		upstream and both reproduced on stock:
151
152		1. `clear()` reclaims no disk. Clearing ~4 MB of blocks left the stock store
153		larger (6.81 → 6.90 MB) even after `compactRange`. The corestore column family
154		sets `blobGarbageCollectionAgeCutOff: 0.25`, so blob-file space is collected lazily
155		— a plausible explanation I did not verify. Either way, "clear to reclaim space" is
156		not a thing you can rely on promptly.
157
158		This is an argument for virtualising: on a virtual drive, clear frees only pointer
159		records, and the bytes are freed by the overlay's own refcount and `unlink` —
160		immediately, and verifiably.
161
162		2. clear-then-truncate is not a supported sequence. After a clear, truncating the
163		blobs core fails with `INVALID_OPERATION: Expected tree node 71 from storage, got
164		(nil)` — on stock and virtual identically. Mechanism: `clear()` deletes merkle tree
165		nodes for fully-cleared spans (`hypercore/lib/session-state.js:576-577`), and
166		truncate then needs one of them. Same-session clear-then-truncate happens to work
167		(the node is still cached); across sessions it fails. Pick one operation, not both.
168
169		## Crash during append
170
171		SIGKILL a publish in flight, reopen, and look for the one state that would prove the
172		tagged-record scheme broken: the tree claims a block whose bytes cannot be produced.
173		Eight trials (four kill points × stock and virtual), one 367 MB artifact.
174
175		```
176		trials 8
177		killed mid-publish 7
178		all stores reopened true
179		tree claims a block it cannot produce 0
180		verdict PASS
181		```
182
183		Every killed publish left `entry: none` — hyperdrive writes the bee entry only after
184		the blob stream closes, so a crash can never expose a half-written file. The one trial
185		that completed read back byte-identical. Parity with stock throughout.
186
187		Two things worth noting:
188
189		- Virtual publishes get further per unit time. At the 950 ms kill point the virtual
190		drive had committed 816 blocks to stock's 272 — the same ~3× that shows up as faster
191		publish wall-clock, because the RocksDB block writes are gone.
192		- A crashed publish leaks orphaned blocks, and that is where the two diverge
193		sharply. Compacted, a crashed stock publish left 182.5 MB of blocks with no entry
194		pointing at them; the virtual equivalent — further along, 816 blocks vs 560 — left
195		0.28 MB. Combined with `clear()` not reclaiming promptly, aborted publishes on a
196		stock drive accumulate garbage that is awkward to get rid of.
197
198		Caveat: SIGKILL ends the process but the OS still flushes its page cache. This is not a
199		power-loss test, and nothing here should be read as an fsync-durability claim.
200
201		## Encryption
202
203		Same 8 MB payload, same adapter, with and without `encryptionKey`. Sizes compacted.
204
205		\| \| plaintext \| encrypted \|
206		\|---\|---\|---\|
207		\| blocks redirected \| 102 \| 0 \|
208		\| blocks stored literally \| 3 \| 105 \|
209		\| store \| 0.14 MB \| 8.53 MB \|
210		\| readback \| ok \| ok \|
211		\| drive key \| identical \| identical \|
212
213		Virtualisation is lost completely under encryption, and correctness is unaffected.
214		Hypercore encrypts in `_preappend`, above the storage layer, so the bytes reaching
215		`putBlock` are ciphertext with 8 bytes of prepended padding. Their hashes are not chunk
216		hashes, so every block falls through to real storage — the fall-through-by-hash design
217		degrades safely to a normal drive rather than corrupting anything.
218
219		It is recoverable. The cipher is a keyed stream XOR nonced by block index and
220		blinded fork (`hypercore/lib/default-encryption.js:49-69`), and the probe confirms it:
221		same plaintext at the same index gives identical ciphertext, a different index gives
222		different ciphertext. So an adapter holding the encryption key could decrypt in
223		`putBlock` to recover the plaintext hash, and re-encrypt in `getBlock` from the stored
224		plaintext chunk. Dedup survives that, because hyperblobs hashes what the caller
225		writes — plaintext — and only storage ever sees ciphertext.
226
227		Two things to get right if that is built: `CoreRX` already carries `fork`, but `CoreTX`
228		does not obviously, and the fork participates in the nonce. And it hands the storage
229		layer a key it currently never sees, which is a real trust-boundary change.
230
231		Until then: publish public apps unencrypted — which is what `pear://` distribution
232		is anyway — and treat encrypted private repos as a separate track.
233
234		## GC / bitfield
235
236		```
237		host still advertises after bytes deleted TRUE ← the lie
238		peer requesting it REQUEST_TIMEOUT after 5.0 s
239		after core.clear(range): host advertises FALSE
240		unrelated file still fetchable TRUE
241		```
242
243		`core.clear()` on the affected range makes the advertisement truthful again. One run of
244		the probe died with an uncaught `REQUEST_TIMEOUT` thrown from
245		`hypercore/lib/replicator.js:3371` rather than rejecting `get()`; the rerun did not
246		reproduce it. Racy, but a long-lived seeder in that state could take an uncaught
247		exception.
248
249		### Two seeders — the lie is survivable (corrects the framing above)
250
251		A (virtual) publishes, B (stock) mirrors it fully, then A's bytes are GC'd. A fresh
252		client connected to both asks for the file.
253
254		\| \| outcome \| elapsed \| blocks served by B \|
255		\|---\|---\|---\|---\|
256		\| A lies (GC, no clear) \| ok \| 117–135 ms \| 93–109 of ~110 \|
257		\| A cleared the range \| ok \| 104–133 ms \| 92–109 of ~110 \|
258
259		Four runs each; the distributions overlap, so the elapsed difference is noise, not
260		signal. Both recover, and the second seeder serves essentially all of it.
261
262		So an un-cleared GC is not the correctness hazard the single-seeder probe implied.
263		Hypercore's replicator routes around a peer that fails to deliver, and a second source
264		covers it transparently. With no second source, cleared or not, the client times out
265		identically — the bytes are gone either way.
266
267		Clearing after GC is therefore hygiene, not a requirement: it keeps advertisement
268		honest and avoids wasted requests, and can be done opportunistically rather than
269		atomically with the byte drop.
270
271		## Bound to hyper-overlay's FileIndex
272
273		A vdrive published with bytes resolved through a real `FileIndex`, exercising both
274		byte sources — a working-tree file and a content-addressed spool blob (what a mirror
275		holds, having no working tree).
276
277		```
278		indexed 2 files -> 1 path + 1 spool blob, 114 chunks
279		blocks 114 redirected, 5 literal
280		readback working tree ok, spool ok
281		hash failures 0
282		input 8.39 MB
283		drive store 0.14 MB
284		```
285
286		`indexAll()` walks `listFiles()`, tries `getChunkMap(path)` against the working-tree
287		roots first, then falls back to `getChunkMapByHash(contentHash)` against the spool dirs.
288
289		One thing to know when populating an index by hand: `chunk()` returns `data` subarrays
290		alongside the locations, and putting those into the bee blows past Hypercore's 15 MiB
291		block cap. Store `{hash, offset, length}` only.
292
293		## Upstream bug found
294
295		A 0-byte file written with `dedup: true` cannot be read back: hyperblobs writes an
296		empty block map, then `seekBlockMap` dereferences `map.blocks === null`
297		(`hyperblobs/lib/streams.js:327`). Reproduced on a plain store, so it is upstream.
298		Matters because app trees are full of empty files. Workaround: write 0-byte files
299		without dedup.

docs/design.md +107 -0

…	…	@@ -0,0 +1,107 @@
	1	# How it works
	2
	3	## The seam
	4
	5	`Core` obtains its per-core storage from exactly two calls (`hypercore/lib/core.js`):
	6
	7	```js
	8	:162 let storage = await this.db.resumeCore(this.discoveryKey)
	9	:232 storage = await this.db.createCore({ ... })
	10	```
	11
	12	Subclassing `CorestoreStorage` and wrapping what those two return puts us underneath
	13	every block read and write, with nothing above aware. `Hypercore.defaultStorage`
	14	accepts any object that duck-types as core storage (`isCorestoreStorage` checks for
	15	`setDefaultDiscoveryKey`), so a subclass instance passes straight through Corestore.
	16
	17	Objects that construct more of themselves must be re-wrapped — `snapshot()` and
	18	`atomize()` — and `read()`/`write()` return wrapped RX/TX. That is the whole surface:
	19	five methods.
	20
	21	**Getters must be read with `Reflect.get(target, prop)` and methods bound to the
	22	target, not the proxy.** Hypercore's storage objects hold state in plain properties
	23	that break if `this` becomes the proxy.
	24
	25	## Block records
	26
	27	Every stored record carries a one-byte tag:
	28
	29	```
	30	0x00 \| <original bytes> literal — bee blocks, and anything we do not hold
	31	0x01 \| <32-byte hash> pointer — bytes live in the content store
	32	```
	33
	34	Because the pointer is the block value, it commits in the same `CoreTX` as the tree
	35	node — tree and pointer land atomically — and `deleteBlock` / truncate free it with no
	36	extra bookkeeping. Cost: 1 byte per literal block, 33 bytes per virtualised block.
	37
	38	Fall-through is by content hash: a block whose hash the store does not recognise is
	39	written literally. That is what makes the failure mode safe — an unrecognised block is
	40	always stored, never assumed. It is also why encryption degrades to an ordinary drive
	41	rather than corrupting one.
	42
	43	## Chunks are hypercore leaves
	44
	45	`hyper-overlay`'s chunk hash is `hypercore-crypto`'s `data()` (`chunker.js:171`), which
	46	is the same construction hypercore uses for a merkle leaf
	47	(`hypercore/lib/merkle-tree.js:1227`). So an existing FastCDC chunk hash is the leaf
	48	hash, and a chunk maps directly onto a block with no translation.
	49
	50	Write files chunk-by-chunk and the block boundaries become the chunk boundaries —
	51	`BlobWriteStream` makes one block per `write()`. With `dedup: true` hyperblobs records
	52	a block map of `{index, byteLength}`, so blocks may be variable-sized and repeated.
	53
	54	## The locations index
	55
	56	A drive stores `index → contentHash`. Serving needs the other half,
	57	`contentHash → (file, offset, length)`, and `hyper-overlay`'s `FileIndex` has no such
	58	reverse index — it maps path → chunks and oid → chunks.
	59
	60	Rebuilding it by re-chunking every published file is O(all bytes) on every boot, which
	61	is untenable for a node holding a repo's release history. So `publishTree` writes it
	62	once, at publish time, when both halves are already known: `locations.json` beside the
	63	drive, paths relative to a `root` so it survives a move and never bakes a local layout
	64	into a shared artifact. Roughly 90 bytes per unique chunk.
	65
	66	It does two jobs:
	67
	68	1. Serving — `openDrive()` reads it and returns a drive ready to serve, with no
	69	chunker involved.
	70	2. Republishing — `dedup: true` seeds its hash table by reading back the previous
	71	entry's blocks, and those reads come through the content store. Without the index
	72	those chunks resolve to `null` and the write stalls on an unsettled promise.
	73
	74	A persistent reverse index in `hyper-overlay` itself would remove the need (its issue
	75	#140).
	76
	77	## Why not `pear seed`
	78
	79	`pear seed` opens the drive by key in PEAR's own corestore and downloads it
	80	(`pear/subsystems/sidecar/ops/seed.js`):
	81
	82	```js
	83	const corestore = this.sidecar.getCorestore()
	84	const drive = await session.add(new Hyperdrive(corestore, key))
	85	await replicator.join(this.sidecar.swarm, { server: true, client: true })
	86	drive.db.core.download({ start: 0, end: -1 })
	87	```
	88
	89	To seed a virtual drive it would have to fetch every byte from us and materialise a
	90	full second copy — exactly the copy this package exists to avoid. It is a re-seeder,
	91	not a source: something must already be serving before it has anything to offer.
	92
	93	So the origin serves its own drives, out of its own store. In a daemon that also buys
	94	one swarm for every repo rather than a process per link, supervision, re-announcement
	95	on boot, and serving through the redirect.
	96
	97	## Compaction timing
	98
	99	RocksDB's footprint carries the write churn of a publish — 76 MB against a 0.58 MB
	100	steady state, for the same publish. `publishTree` compacts in its `finally`, after the
	101	store closes and before anything else opens it.
	102
	103	That window matters: a seeder takes an exclusive lock the moment the drive is
	104	announced, and a second opener gets `File descriptor could not be locked`. After that
	105	point the churn cannot be reclaimed without stopping the daemon.
	106
	107	The same lock is why a republish must drop the drive first, publish, then re-announce.

docs/gaps.md +72 -0

…	…	@@ -0,0 +1,72 @@
	1	# Gaps
	2
	3	What is unproven or known-broken. Read this before relying on any of it.
	4
	5	## Encryption defeats it entirely
	6
	7	Hypercore encrypts in `_preappend`, above the storage layer, so `putBlock` sees
	8	ciphertext with 8 bytes of prepended padding whose hash is not a chunk hash. Measured:
	9	0 of 105 blocks redirect, and the store goes from 0.14 MB to 8.53 MB. Readback
	10	stays correct — fall-through-by-hash degrades to an ordinary drive rather than
	11	corrupting one.
	12
	13	Recoverable in principle. The cipher is a keyed stream XOR nonced by block index and
	14	blinded fork (`hypercore/lib/default-encryption.js:49-69`), and it is deterministic per
	15	index, so an adapter holding the key could decrypt on write and re-encrypt on read.
	16	Dedup would survive, since hyperblobs hashes what the caller writes — plaintext — and
	17	only storage sees ciphertext. Two catches: `CoreTX` does not obviously carry the fork
	18	that participates in the nonce, and it hands the storage layer a key it never sees today.
	19
	20	Until then: publish public apps unencrypted, which is what `pear://` distribution
	21	is anyway. Whether this is ever worth building depends on which layer owns encryption
	22	— if `hyper-overlay` encrypts its own content, the adapter needs no changes at all.
	23
	24	## The seam is unsupported
	25
	26	`hypercore-storage` has had no public pluggable-storage contract since
	27	`random-access-storage` was dropped. Everything here wedges into an internal shape.
	28
	29	Mitigated, not solved, by `test/canary.mjs`, which asserts every assumption the adapter
	30	makes and prints resolved versions. Required checks pass against hypercore-storage
	31	2.9.0, 3.0.2, 3.1.2 and 3.2.1 — stable across that range. `CoreRX` gained its `fork`
	32	field only in 3.2.x, flagged advisory since it matters only for encryption work.
	33
	34	Run the canary on any dependency bump.
	35
	36	## Block streams are not decoded
	37
	38	`createBlockStream` — used by `audit.js` and `copy-prologue.js` — yields raw tagged
	39	records rather than decoded bytes. Anything walking blocks that way sees the pointer
	40	framing. Not hit in the publish/serve path, but it would surprise an audit.
	41
	42	## Untested
	43
	44	- Power loss. SIGKILL is covered and passes: eight trials, zero cases of the tree
	45	claiming a block it could not produce. But SIGKILL leaves the OS page cache to flush,
	46	so nothing here is an fsync-durability result.
	47	- Whether a published AppImage actually launches. Installs are verified
	48	byte-identical; both test machines are headless.
	49
	50	## Sharp edges worth knowing
	51
	52	0-byte files must not use `dedup: true`. Upstream bug: hyperblobs writes an empty
	53	block map, then `seekBlockMap` dereferences `map.blocks === null`
	54	(`hyperblobs/lib/streams.js:327`). Reproduces on a plain store. `publishTree` already
	55	routes empty files around it; anything else writing drives must too.
	56
	57	Dedup only seeds from the previous entry at the same path. A chunk that vanishes
	58	and returns is re-appended — worth 3.1% over ten releases against a store that dedups
	59	against everything. It also means the same file published at two paths is written
	60	twice, though both redirect to the same bytes on disk.
	61
	62	GC has to be paired, but not atomically. Dropping bytes does not update the
	63	bitfield, so the core keeps advertising blocks it cannot produce. Measured with two
	64	seeders, that is survivable: the replicator routes around it and a second source serves
	65	in the same ~120 ms. So clearing the range is hygiene, and can be opportunistic. What
	66	does matter: under `dedup: true` old releases share blocks with new ones, so ranges
	67	must be computed from the content store's refcounts, never from a version's block range.
	68
	69	`clear()` reclaims no disk promptly, and clear-then-truncate is unsupported —
	70	`clear()` prunes merkle tree nodes (`hypercore/lib/session-state.js:576-577`), so a
	71	later truncate fails with `Expected tree node N from storage, got (nil)`. Both
	72	reproduce on a stock drive.

docs/results.md +333 -0

…	…	@@ -0,0 +1,333 @@
	1	# Measurements
	2
	3	> Measure compacted. RocksDB's on-disk footprint includes uncompacted WAL/SST
	4	> churn, which is noise on a big store and dominant on a small one. The small-file
	5	> store below read 14.4 MB before compaction and 1.38 MB after. The large-file
	6	> stores barely moved (±0.05 MB), so those figures were already steady-state.
	7	> `test/compact-size.mjs` does the compact-then-measure.
	8
	9	All runs on tabertron, 2026-08-24. Source data: ten consecutive real `hypertps.pck`
	10	release artifacts (~367 MB each) from `~/.local/share/store/dependencies`, and
	11	synthetic files where noted. Chunker: `hyper-overlay/lib/chunker.js` (FastCDC,
	12	tier 2 → ~320 KB average chunk).
	13
	14	## T0 — does stock Pear read chunk-sized blocks?
	15
	16	The installed `pear` 3.0.1 bundle (`~/.config/pear/current/boot.bundle`) contains
	17	`node_modules/hyperblobs/lib/block-map.js` and the marker strings `Block map is too
	18	large`, `Unsupported block map version`, `_dedup`, `getBlockMap`. So the shipped
	19	reader supports block maps and variable-size blocks. Pass.
	20
	21	## T1 — storage cost, one release pair
	22
	23	Two consecutive releases, 367.06 MB each, into one drive at the same path.
	24
	25	\| Arm \| after v1 \| after v2 \| v2 delta \|
	26	\|---\|---\|---\|---\|
	27	\| overlay (control) \| 360.1 MB \| 378.3 MB \| 18.2 MB \|
	28	\| hyperdrive, no dedup \| 368.3 MB \| 736.4 MB \| 368.1 MB \|
	29	\| hyperdrive, `dedup:true`, fixed 64 KB \| 367.6 MB \| 488.7 MB \| 121.2 MB \|
	30	\| hyperdrive, `dedup:true`, FastCDC \| 360.5 MB \| 378.7 MB \| 18.2 MB \|
	31
	32	Two corrections to earlier assumptions:
	33
	34	- RocksDB per-byte overhead is 0.3%, not a multiplier (368.3 MB stored for
	35	367.06 MB in). The corestore column family enables blob files with GC.
	36	- Hyperblobs does dedup, at block level, via `{dedup:true}` + block maps
	37	(`hyperblobs/lib/streams.js:86-101`, `hyperdrive/index.js:591-596`). A stock drive
	38	fed FastCDC chunks matched the overlay to within 0.1%.
	39
	40	## T1c — ten releases, stock drive vs overlay
	41
	42	\| Release \| drive \| overlay \| gap \|
	43	\|---\|---\|---\|---\|
	44	\| 1 \| 360.5 MB \| 360.1 MB \| +0.4 \|
	45	\| 5 \| 456.1 MB \| 453.2 MB \| +2.9 \|
	46	\| 6 \| 484.8 MB \| 471.4 MB \| +13.4 \|
	47	\| 10 \| 580.8 MB \| 563.3 MB \| +17.5 (3.1%) \|
	48
	49	The gap appears at release 6 — a chunk set that vanished and returned. `dedup:true`
	50	seeds its hash table from only the previous entry at that path
	51	(`hyperdrive/index.js:592`), so resurrected chunks are re-appended. Publish time was
	52	flat at ~3.6 s: the seeding re-read is O(previous release), not O(history).
	53
	54	## T2 — does the storage seam hold?
	55
	56	Pass-through wrapper (delegates everything, redirects nothing):
	57
	58	```
	59	identical drive key / blobs key / root hash / digests true
	60	reopen from disk true
	61	replication to a stock peer true
	62	intercepted 111 getBlock, 111 putBlock
	63	```
	64
	65	Nothing escaped the wrapper. Pass.
	66
	67	## T3 — redirect, two releases
	68
	69	```
	70	sameDriveKey / sameBlobsKey / sameRootHash true
	71	blobsLength stock 2244 virtual 2244
	72	input 734.0 MB
	73	stock store 720.8 MB
	74	virtual store 0.58 MB (compacted; tagged-record scheme, no side file)
	75	peer store 720.7 MB (stock peer replicated from the virtual drive)
	76	blocks 2242 redirected, 5 literal (the bee's own)
	77	hash failures 0
	78	```
	79
	80	## T3n — ten releases, virtual
	81
	82	\| Release \| blocks \| store \| stock equivalent \|
	83	\|---\|---\|---\|---\|
	84	\| 1 \| 1122 \| 0.50 MB \| 360.5 MB \|
	85	\| 5 \| 1368 \| 0.50 MB \| 456.1 MB \|
	86	\| 10 \| 1682 \| 0.88 MB \| 580.8 MB \|
	87
	88	3.67 GB of releases → 0.88 MB compacted (657×). Publish ~2.4 s/release, faster than stock
	89	because the RocksDB block writes disappear. Pointer overhead is 36 bytes per block —
	90	≈0.011% of content at tier-2 chunk sizes.
	91
	92	## T5 — real `pear install`
	93
	94	Built a `/by-arch/linux-x64/app/` drive on virtual storage (6.29 MB payload, 120 KB
	95	store), seeded it on the public DHT, and ran the stock `pear` 3.0.1 CLI:
	96
	97	```
	98	$ pear install pear://uimennwjwyot5cinekhb348kawufit1eq4ctyf5irurn1nfmo3so --to …
	99	[ Peers: 1 ] [ ⬇ 2.8MB - 2.8MB/s ]
	100	App: t5app Version: 1.0.0
	101	Target: …/t5/installed/t5app
	102	Installed
	103	```
	104
	105	Installed file: 6,291,480 bytes, mode 755, hash identical to source, runs. Seeder
	106	store unchanged at 139 KB; it served 6,291,560 bytes as 89 redirected block reads
	107	with 0 hash failures. Pass.
	108
	109	## Many small files
	110
	111	976 files, 209.1 MB, median 2 KB, largest 11 MB (this repo's own `node_modules`),
	112	plus three synthetic 0-byte files to cover that case.
	113
	114	\| \| stock \| virtual \|
	115	\|---\|---\|---\|
	116	\| store (compacted) \| 210.1 MB \| 1.38 MB \|
	117	\| ratio \| \| 152× \|
	118	\| per file \| \| 1,415 bytes \|
	119	\| drive key \| identical \| identical \|
	120	\| readback mismatches \| 0 \| 0 \|
	121	\| publish \| 4.2 s \| 2.7 s \|
	122
	123	Blocks: 4,590 content chunks redirected, 1,950 stored literally. The literal ones
	124	are the bee's own blocks (977) plus roughly one block-map block per file — with
	125	`dedup: true` hyperblobs appends an encoded block map after each file's data, and that
	126	map is generated metadata, not content we hold, so it correctly falls through to real
	127	storage.
	128
	129	So the floor for an app tree is ~1.4 KB per file regardless of file size: bee entry +
	130	block-map block + tree nodes + pointer record. A 10,000-file tree costs ~14 MB.
	131	Small files dilute the win (152× rather than the 657× of large artifacts) because
	132	metadata, not bytes, becomes the cost.
	133
	134	## truncate / clear
	135
	136	Three files (v1 and v2 of the same path, which share blocks under `dedup:true`, plus
	137	an unrelated file), then `drive.clear('/app.bin')`, then a truncate of the blobs core.
	138	Sizes are compacted.
	139
	140	\| \| stock \| virtual \|
	141	\|---\|---\|---\|
	142	\| after publish \| 6.81 MB \| 0.13 MB \|
	143	\| after `clear('/app.bin')` \| 6.90 MB \| 0.22 MB \|
	144	\| after attempted truncate \| 6.98 MB \| 0.30 MB \|
	145	\| `has()` on a cleared block \| false \| false \|
	146	\| read the cleared entry \| REQUEST_TIMEOUT \| REQUEST_TIMEOUT \|
	147	\| read the untouched entry \| ok \| ok \|
	148
	149	Parity: the virtual drive behaves exactly like the stock one. Two findings, both
	150	upstream and both reproduced on stock:
	151
	152	1. `clear()` reclaims no disk. Clearing ~4 MB of blocks left the stock store
	153	larger (6.81 → 6.90 MB) even after `compactRange`. The corestore column family
	154	sets `blobGarbageCollectionAgeCutOff: 0.25`, so blob-file space is collected lazily
	155	— a plausible explanation I did not verify. Either way, "clear to reclaim space" is
	156	not a thing you can rely on promptly.
	157
	158	This is an argument for virtualising: on a virtual drive, clear frees only pointer
	159	records, and the bytes are freed by the overlay's own refcount and `unlink` —
	160	immediately, and verifiably.
	161
	162	2. clear-then-truncate is not a supported sequence. After a clear, truncating the
	163	blobs core fails with `INVALID_OPERATION: Expected tree node 71 from storage, got
	164	(nil)` — on stock and virtual identically. Mechanism: `clear()` deletes merkle tree
	165	nodes for fully-cleared spans (`hypercore/lib/session-state.js:576-577`), and
	166	truncate then needs one of them. Same-session clear-then-truncate happens to work
	167	(the node is still cached); across sessions it fails. Pick one operation, not both.
	168
	169	## Crash during append
	170
	171	SIGKILL a publish in flight, reopen, and look for the one state that would prove the
	172	tagged-record scheme broken: the tree claims a block whose bytes cannot be produced.
	173	Eight trials (four kill points × stock and virtual), one 367 MB artifact.
	174
	175	```
	176	trials 8
	177	killed mid-publish 7
	178	all stores reopened true
	179	tree claims a block it cannot produce 0
	180	verdict PASS
	181	```
	182
	183	Every killed publish left `entry: none` — hyperdrive writes the bee entry only after
	184	the blob stream closes, so a crash can never expose a half-written file. The one trial
	185	that completed read back byte-identical. Parity with stock throughout.
	186
	187	Two things worth noting:
	188
	189	- Virtual publishes get further per unit time. At the 950 ms kill point the virtual
	190	drive had committed 816 blocks to stock's 272 — the same ~3× that shows up as faster
	191	publish wall-clock, because the RocksDB block writes are gone.
	192	- A crashed publish leaks orphaned blocks, and that is where the two diverge
	193	sharply. Compacted, a crashed stock publish left 182.5 MB of blocks with no entry
	194	pointing at them; the virtual equivalent — further along, 816 blocks vs 560 — left
	195	0.28 MB. Combined with `clear()` not reclaiming promptly, aborted publishes on a
	196	stock drive accumulate garbage that is awkward to get rid of.
	197
	198	Caveat: SIGKILL ends the process but the OS still flushes its page cache. This is not a
	199	power-loss test, and nothing here should be read as an fsync-durability claim.
	200
	201	## Encryption
	202
	203	Same 8 MB payload, same adapter, with and without `encryptionKey`. Sizes compacted.
	204
	205	\| \| plaintext \| encrypted \|
	206	\|---\|---\|---\|
	207	\| blocks redirected \| 102 \| 0 \|
	208	\| blocks stored literally \| 3 \| 105 \|
	209	\| store \| 0.14 MB \| 8.53 MB \|
	210	\| readback \| ok \| ok \|
	211	\| drive key \| identical \| identical \|
	212
	213	Virtualisation is lost completely under encryption, and correctness is unaffected.
	214	Hypercore encrypts in `_preappend`, above the storage layer, so the bytes reaching
	215	`putBlock` are ciphertext with 8 bytes of prepended padding. Their hashes are not chunk
	216	hashes, so every block falls through to real storage — the fall-through-by-hash design
	217	degrades safely to a normal drive rather than corrupting anything.
	218
	219	It is recoverable. The cipher is a keyed stream XOR nonced by block index and
	220	blinded fork (`hypercore/lib/default-encryption.js:49-69`), and the probe confirms it:
	221	same plaintext at the same index gives identical ciphertext, a different index gives
	222	different ciphertext. So an adapter holding the encryption key could decrypt in
	223	`putBlock` to recover the plaintext hash, and re-encrypt in `getBlock` from the stored
	224	plaintext chunk. Dedup survives that, because hyperblobs hashes what the caller
	225	writes — plaintext — and only storage ever sees ciphertext.
	226
	227	Two things to get right if that is built: `CoreRX` already carries `fork`, but `CoreTX`
	228	does not obviously, and the fork participates in the nonce. And it hands the storage
	229	layer a key it currently never sees, which is a real trust-boundary change.
	230
	231	Until then: publish public apps unencrypted — which is what `pear://` distribution
	232	is anyway — and treat encrypted private repos as a separate track.
	233
	234	## GC / bitfield
	235
	236	```
	237	host still advertises after bytes deleted TRUE ← the lie
	238	peer requesting it REQUEST_TIMEOUT after 5.0 s
	239	after core.clear(range): host advertises FALSE
	240	unrelated file still fetchable TRUE
	241	```
	242
	243	`core.clear()` on the affected range makes the advertisement truthful again. One run of
	244	the probe died with an uncaught `REQUEST_TIMEOUT` thrown from
	245	`hypercore/lib/replicator.js:3371` rather than rejecting `get()`; the rerun did not
	246	reproduce it. Racy, but a long-lived seeder in that state could take an uncaught
	247	exception.
	248
	249	### Two seeders — the lie is survivable (corrects the framing above)
	250
	251	A (virtual) publishes, B (stock) mirrors it fully, then A's bytes are GC'd. A fresh
	252	client connected to both asks for the file.
	253
	254	\| \| outcome \| elapsed \| blocks served by B \|
	255	\|---\|---\|---\|---\|
	256	\| A lies (GC, no clear) \| ok \| 117–135 ms \| 93–109 of ~110 \|
	257	\| A cleared the range \| ok \| 104–133 ms \| 92–109 of ~110 \|
	258
	259	Four runs each; the distributions overlap, so the elapsed difference is noise, not
	260	signal. Both recover, and the second seeder serves essentially all of it.
	261
	262	So an un-cleared GC is not the correctness hazard the single-seeder probe implied.
	263	Hypercore's replicator routes around a peer that fails to deliver, and a second source
	264	covers it transparently. With no second source, cleared or not, the client times out
	265	identically — the bytes are gone either way.
	266
	267	Clearing after GC is therefore hygiene, not a requirement: it keeps advertisement
	268	honest and avoids wasted requests, and can be done opportunistically rather than
	269	atomically with the byte drop.
	270
	271	## Bound to hyper-overlay's FileIndex
	272
	273	A vdrive published with bytes resolved through a real `FileIndex`, exercising both
	274	byte sources — a working-tree file and a content-addressed spool blob (what a mirror
	275	holds, having no working tree).
	276
	277	```
	278	indexed 2 files -> 1 path + 1 spool blob, 114 chunks
	279	blocks 114 redirected, 5 literal
	280	readback working tree ok, spool ok
	281	hash failures 0
	282	input 8.39 MB
	283	drive store 0.14 MB
	284	```
	285
	286	`indexAll()` walks `listFiles()`, tries `getChunkMap(path)` against the working-tree
	287	roots first, then falls back to `getChunkMapByHash(contentHash)` against the spool dirs.
	288
	289	One thing to know when populating an index by hand: `chunk()` returns `data` subarrays
	290	alongside the locations, and putting those into the bee blows past Hypercore's 15 MiB
	291	block cap. Store `{hash, offset, length}` only.
	292
	293
	294	## End to end, from CI (2026-08-24)
	295
	296	`ifsrenderer-linux`, opted in with `vdrive: true` in its `.pear-git/ci.yaml`, built
	297	and published by pear-git-v3, seeded by the daemon, installed by the stock `pear`
	298	CLI on a second machine over the public DHT.
	299
	300	\| \| release 1 \| release 2 \|
	301	\|---\|---\|---\|
	302	\| CI build \| 76 s \| 80 s \|
	303	\| blocks \| 1835 \| 2478 \|
	304	\| new pointer blocks \| 1829 \| 637 \|
	305	\| prior locations loaded \| 0 \| 1216 \|
	306	\| drive store (compacted) \| 0.58 MB \| 5.1 MB* \|
	307	\| installed sha \| `87b4487d…` \| `7bf5023c…` \|
	308	\| link \| \multicolumn — unchanged across both \| \|
	309
	310	\* the live drive was written by 0.1.229, before compaction moved into publishTree;
	311	the same publish into a fresh store compacts to 0.58 MB.
	312
	313	284 MB of release artifacts across three versions, served by one drive under one
	314	stable `pear://` link. Both installs byte-identical to what CI built, mode 755, with
	315	`pear`'s Linux desktop-entry and icon install completing.
	316
	317	The AppImage result also corrects an earlier generalisation in this document: three
	318	different 49 MB AppImage builds deduped at 97.7% (614 chunks, then 14, then 12).
	319	Compression does not zero dedup — stream compression does. gzip over a tarball is
	320	one continuous stream, so a single byte shifts everything after it; a squashfs (and so
	321	an AppImage) compresses per block, so a change disturbs only the blocks it touches.
	322
	323	Note that the claim "git bundles do not dedup because they are compressed", made
	324	elsewhere in our notes, is an inference from the tarball measurement and has never
	325	been measured on bundles.
	326
	327	## Upstream bug found
	328
	329	A 0-byte file written with `dedup: true` cannot be read back: hyperblobs writes an
	330	empty block map, then `seekBlockMap` dereferences `map.blocks === null`
	331	(`hyperblobs/lib/streams.js:327`). Reproduced on a plain store, so it is upstream.
	332	Matters because app trees are full of empty files. Workaround: write 0-byte files
	333	without dedup.

hyper-vdrive commits / 3a0100d

Changed files (5) +552 -427