d8c1abb · hyper-vdrive

Changed files (4) +189 -6

README.md +9 -3

…	…	@@ -20,11 +20,17 @@ Measured on ten consecutive real 367 MB game-pack releases:
20	20
21	21	\| \| stock drive \| virtual drive \|
22	22	\|---\|---\|---\|
23		\| store after 10 releases \| 580.8 MB \| 0.95 MB \|
	23	\| store after 10 releases \| 580.8 MB \| 0.88 MB \|
24	24	\| publish time per release \| 3.6 s \| 2.4 s \|
25	25	\| drive key \| — \| identical \|
26	26
27		See [RESULTS.md](RESULTS.md) for the full measurements and method.
	27	And on an app tree — 976 files, 209 MB, median 2 KB — 210.1 MB becomes 1.38 MB
	28	(152×). Small files dilute the win because metadata, not bytes, becomes the cost:
	29	the floor is ~1.4 KB per file.
	30
	31	See [RESULTS.md](RESULTS.md) for the full measurements and method. Measure stores
	32	compacted (`test/compact-size.mjs`) — RocksDB's raw footprint includes write churn
	33	that dominates a small store.
28	34
29	35	## How
30	36
…	…	@@ -102,4 +108,4 @@ Known gaps, in the order they'd bite:
102	108	equalling chunk bytes. Run unencrypted, or teach the adapter to re-derive.
103	109	4. Unsupported seam. `hypercore-storage` has no public pluggable-storage contract
104	110	since `random-access-storage` was dropped. Pin versions and run the tests as a canary.
105		5. Untested: many small files, truncate/clear, crash during append.
	111	5. Untested: truncate/clear, crash during append.

RESULTS.md +34 -3

…	…	@@ -1,5 +1,11 @@
1	1	# Measurements
2	2
	3	> Measure compacted. RocksDB's on-disk footprint includes uncompacted WAL/SST
	4	> churn, which is noise on a big store and dominant on a small one. The small-file
	5	> store below read 14.4 MB before compaction and 1.38 MB after. The large-file
	6	> stores barely moved (±0.05 MB), so those figures were already steady-state.
	7	> `test/compact-size.mjs` does the compact-then-measure.
	8
3	9	All runs on tabertron, 2026-08-24. Source data: ten consecutive real `hypertps.pck`
4	10	release artifacts (~367 MB each) from `~/.local/share/store/dependencies`, and
5	11	synthetic files where noted. Chunker: `hyper-overlay/lib/chunker.js` (FastCDC,
…	…	@@ -65,7 +71,7 @@ sameDriveKey / sameBlobsKey / sameRootHash true
65	71	blobsLength stock 2244 virtual 2244
66	72	input 734.0 MB
67	73	stock store 720.8 MB
68		virtual store 1.03 MB (tagged-record scheme, no side file)
	74	virtual store 0.58 MB (compacted; tagged-record scheme, no side file)
69	75	peer store 720.7 MB (stock peer replicated from the virtual drive)
70	76	blocks 2242 redirected, 5 literal (the bee's own)
71	77	hash failures 0
…	…	@@ -77,9 +83,9 @@ hash failures 0
77	83	\|---\|---\|---\|---\|
78	84	\| 1 \| 1122 \| 0.50 MB \| 360.5 MB \|
79	85	\| 5 \| 1368 \| 0.50 MB \| 456.1 MB \|
80		\| 10 \| 1682 \| 0.82 MB \| 580.8 MB \|
	86	\| 10 \| 1682 \| 0.88 MB \| 580.8 MB \|
81	87
82		3.67 GB of releases → 0.95 MB (610×). Publish ~2.4 s/release, faster than stock
	88	3.67 GB of releases → 0.88 MB compacted (657×). Publish ~2.4 s/release, faster than stock
83	89	because the RocksDB block writes disappear. Pointer overhead is 36 bytes per block —
84	90	≈0.011% of content at tier-2 chunk sizes.
85	91
…	…	@@ -100,6 +106,31 @@ Installed file: 6,291,480 bytes, mode 755, hash identical to source, runs. Seede
100	106	store unchanged at 139 KB; it served 6,291,560 bytes as 89 redirected block reads
101	107	with 0 hash failures. Pass.
102	108
	109	## Many small files
	110
	111	976 files, 209.1 MB, median 2 KB, largest 11 MB (this repo's own `node_modules`),
	112	plus three synthetic 0-byte files to cover that case.
	113
	114	\| \| stock \| virtual \|
	115	\|---\|---\|---\|
	116	\| store (compacted) \| 210.1 MB \| 1.38 MB \|
	117	\| ratio \| \| 152× \|
	118	\| per file \| \| 1,415 bytes \|
	119	\| drive key \| identical \| identical \|
	120	\| readback mismatches \| 0 \| 0 \|
	121	\| publish \| 4.2 s \| 2.7 s \|
	122
	123	Blocks: 4,590 content chunks redirected, 1,950 stored literally. The literal ones
	124	are the bee's own blocks (977) plus roughly one block-map block per file — with
	125	`dedup: true` hyperblobs appends an encoded block map after each file's data, and that
	126	map is generated metadata, not content we hold, so it correctly falls through to real
	127	storage.
	128
	129	So the floor for an app tree is ~1.4 KB per file regardless of file size: bee entry +
	130	block-map block + tree nodes + pointer record. A 10,000-file tree costs ~14 MB.
	131	Small files dilute the win (152× rather than the 657× of large artifacts) because
	132	metadata, not bytes, becomes the cost.
	133
103	134	## GC / bitfield
104	135
105	136	```

test/compact-size.mjs +19 -0

…	…	@@ -0,0 +1,19 @@
	1	// Steady-state size: RocksDB reports its uncompacted footprint, which flatters
	2	// nothing and misleads on small stores. Compact, then measure.
	3	// node test/compact-size.mjs <storeDir...>
	4	import { createRequire } from 'module'
	5	const require = createRequire(import.meta.url)
	6	const { execFileSync } = require('child_process')
	7	const CorestoreStorage = require('hypercore-storage')
	8
	9	const du = (p) => Number(execFileSync('du', ['-sb', p]).toString().split('\t')[0])
	10
	11	for (const dir of process.argv.slice(2)) {
	12	const before = du(dir)
	13	const s = new CorestoreStorage(dir)
	14	await s.ready()
	15	await s.compact()
	16	await s.close()
	17	const after = du(dir)
	18	console.log(JSON.stringify({ dir, before, afterCompact: after, freed: before - after }))
	19	}

test/small-files.mjs +127 -0

…	…	@@ -0,0 +1,127 @@
	1	// Many small files — an app tree, not one big artifact.
	2	// With blobs virtualised, what remains is bee metadata + one blob id per file.
	3	// node test/small-files.mjs <workDir> <treeDir>
	4	import { createRequire } from 'module'
	5	const require = createRequire(import.meta.url)
	6
	7	const fs = require('fs')
	8	const path = require('path')
	9	const { execFileSync } = require('child_process')
	10	const crypto = require('hypercore-crypto')
	11	const Corestore = require('corestore')
	12	const Hyperdrive = require('hyperdrive')
	13	const { ContentStore, build, stats } = require('../lib/storage.cjs')
	14	const { chunk } = await import('../../hyper-overlay/lib/chunker.js')
	15
	16	const ROOT = process.argv[2]
	17	const TREE = process.argv[3]
	18	const PRIMARY = Buffer.alloc(32, 7)
	19	const du = (p) => Number(execFileSync('du', ['-sb', p]).toString().split('\t')[0])
	20
	21	function walk (dir, base = dir, out = []) {
	22	for (const e of fs.readdirSync(dir, { withFileTypes: true })) {
	23	const full = path.join(dir, e.name)
	24	if (e.isDirectory()) walk(full, base, out)
	25	else if (e.isFile()) out.push({ file: full, name: '/' + path.relative(base, full) })
	26	}
	27	return out
	28	}
	29
	30	const files = walk(TREE)
	31	// force coverage of the 0-byte case even if the tree has none
	32	const emptyDir = ROOT + '/empties'
	33	fs.mkdirSync(emptyDir, { recursive: true })
	34	for (const n of ['a.stamp', 'b.keep', 'c.empty']) {
	35	const f = path.join(emptyDir, n)
	36	fs.writeFileSync(f, '')
	37	files.push({ file: f, name: '/__empty__/' + n })
	38	}
	39
	40	const content = new ContentStore()
	41	let inputBytes = 0
	42	let chunkCount = 0
	43	for (const f of files) {
	44	f.bytes = fs.readFileSync(f.file)
	45	f.chunks = chunk(f.bytes)
	46	inputBytes += f.bytes.length
	47	chunkCount += f.chunks.length
	48	for (const c of f.chunks) content.add(c.hash, f.file, c.offset, c.length)
	49	}
	50
	51	function write (ws, buf) {
	52	return new Promise((resolve) => {
	53	if (ws.write(buf)) return resolve()
	54	ws.once('drain', resolve)
	55	})
	56	}
	57
	58	async function publish (dir, virtual) {
	59	const Storage = virtual ? build(content) : null
	60	const store = new Corestore(virtual ? new Storage(dir) : dir, {
	61	primaryKey: PRIMARY,
	62	unsafe: true
	63	})
	64	const drive = new Hyperdrive(store)
	65	await drive.ready()
	66
	67	for (const f of files) {
	68	// 0-byte + dedup is an upstream crash on read (hyperblobs seekBlockMap, null map)
	69	const ws = drive.createWriteStream(f.name, f.bytes.length === 0 ? {} : { dedup: true })
	70	for (const c of f.chunks) await write(ws, c.data)
	71	await new Promise((resolve, reject) => {
	72	ws.once('error', reject)
	73	ws.once('close', resolve)
	74	ws.end()
	75	})
	76	}
	77
	78	await drive.getBlobs()
	79	const out = {
	80	driveKey: drive.key.toString('hex'),
	81	metaLength: drive.core.length,
	82	blobsLength: drive.blobs.core.length
	83	}
	84
	85	let mismatches = 0
	86	for (const f of files) {
	87	const got = await drive.get(f.name)
	88	const bytes = got === null ? Buffer.alloc(0) : got
	89	if (!bytes.equals(f.bytes)) mismatches++
	90	}
	91	out.mismatches = mismatches
	92
	93	await drive.close()
	94	await store.close()
	95	out.storeBytes = du(dir)
	96	return out
	97	}
	98
	99	const t0 = Date.now()
	100	const stock = await publish(ROOT + '/stock', false)
	101	const tStock = Date.now() - t0
	102
	103	const t1 = Date.now()
	104	const virt = await publish(ROOT + '/virtual', true)
	105	const tVirt = Date.now() - t1
	106	content.close()
	107
	108	console.log(
	109	JSON.stringify(
	110	{
	111	files: files.length,
	112	inputBytes,
	113	chunks: chunkCount,
	114	sameDriveKey: stock.driveKey === virt.driveKey,
	115	metaLength: { stock: stock.metaLength, virtual: virt.metaLength },
	116	blobsLength: { stock: stock.blobsLength, virtual: virt.blobsLength },
	117	mismatches: { stock: stock.mismatches, virtual: virt.mismatches },
	118	storeBytes: { stock: stock.storeBytes, virtual: virt.storeBytes },
	119	ratio: +(stock.storeBytes / virt.storeBytes).toFixed(1),
	120	bytesPerFileVirtual: Math.round(virt.storeBytes / files.length),
	121	elapsedMs: { stock: tStock, virtual: tVirt },
	122	stats
	123	},
	124	null,
	125	2
	126	)
	127	)

hyper-vdrive commits / d8c1abb

Changed files (4) +189 -6