fa890f8 · holepunchto-kb

Changed files (28) +10468 -7

CONCEPTS.md +630 -0

…	…	@@ -0,0 +1,630 @@
	1	# Concept Graph - Master Index
	2
	3	Purpose: Central navigation for the Hypercore ecosystem knowledge base. Find documentation by concept, use case, or problem type.
	4
	5	---
	6
	7	## Quick Navigation
	8
	9	### By What You Need
	10
	11	- [Learn a concept](#by-concept) → Core P2P patterns and techniques
	12	- [Build something](#by-use-case) → Complete guides for common projects
	13	- [Solve a problem](#by-problem-type) → Troubleshooting and how-tos
	14	- [Understand a repo](#by-repository) → API docs and patterns for specific repos
	15	- [Deploy to production](#deployment--operations) → Systemd, Pear, capabilities
	16
	17	---
	18
	19	## By Concept
	20
	21	### Append-Only Log
	22	What it is: Immutable, sequential data structure with cryptographic verification
	23
	24	Documentation:
	25	- Pattern: [patterns/append-only-log.md](patterns/append-only-log.md)
	26	- Implementing repo: [skills/hypercore/skills.md](skills/hypercore/skills.md)
	27	- Code examples: 10 examples from basic usage to live streams
	28	- Real-world usage: hyper-fleeky/fedicore (social feed posts)
	29	- Integration: Works with Hyperbee, Hyperswarm, Hyperdrive, Autobase
	30
	31	When to use: Event sourcing, distributed databases, messaging, audit trails, version control
	32
	33	---
	34
	35	### Replication
	36	What it is: P2P synchronization of append-only logs with cryptographic verification
	37
	38	Documentation:
	39	- Pattern: [patterns/replication.md](patterns/replication.md)
	40	- Implementing repos: [skills/hypercore/skills.md](skills/hypercore/skills.md), [skills/hyperswarm/skills.md](skills/hyperswarm/skills.md)
	41	- Code examples: 7 examples from basic to multi-core replication
	42	- Real-world usage: hyper-fleeky/fedicore (multi-user feed replication)
	43	- Troubleshooting: [deployment/troubleshooting.md#replication-issues](deployment/troubleshooting.md)
	44
	45	When to use: Data distribution, multi-device sync, collaborative apps, offline-first
	46
	47	---
	48
	49	### Peer Discovery
	50	What it is: DHT-based discovery to find peers interested in the same topics
	51
	52	Documentation:
	53	- Pattern: [patterns/peer-discovery.md](patterns/peer-discovery.md)
	54	- Implementing repos: [skills/hyperswarm/skills.md](skills/hyperswarm/skills.md), [skills/hyperdht/skills.md](skills/hyperdht/skills.md)
	55	- Code examples: 7 examples from basic discovery to direct connections
	56	- Real-world usage: hyper-fleeky/fedicore (follow/unfollow = join/leave topics)
	57	- Integration: Essential for all P2P applications
	58
	59	When to use: Decentralized apps, serverless systems, mesh networks, P2P file sharing
	60
	61	---
	62
	63	### B-Tree Index
	64	What it is: Indexed database on append-only log for fast key-value lookups
	65
	66	Documentation:
	67	- Pattern: [patterns/b-tree-index.md](patterns/b-tree-index.md)
	68	- Implementing repo: [skills/hyperbee/skills.md](skills/hyperbee/skills.md)
	69	- Code examples: CRUD, range queries, atomic batches, snapshots
	70	- Integration: Built on Hypercore, works with Autobase for multi-writer
	71
	72	When to use: User profiles, settings, metadata, queryable databases, sorted data
	73
	74	---
	75
	76	### Cryptography
	77	What it is: Cryptographic primitives for verification, signing, and encryption
	78
	79	Documentation:
	80	- Pattern: [patterns/cryptography.md](patterns/cryptography.md)
	81	- Implementing repo: [skills/hypercore-crypto/skills.md](skills/hypercore-crypto/skills.md)
	82	- Code examples: ED25519 signing, Merkle trees, discovery keys, encryption
	83	- Real-world usage: hyper-fleeky/hyperident (identity encryption)
	84	- Security: Essential foundation for all P2P trust
	85
	86	When to use: Data verification, authentication, access control, privacy
	87
	88	---
	89
	90	### Streaming
	91	What it is: Efficient reading/writing of large datasets without loading into memory
	92
	93	Documentation:
	94	- Pattern: [patterns/streaming.md](patterns/streaming.md)
	95	- Implementing repo: [skills/hypercore/skills.md](skills/hypercore/skills.md)
	96	- Code examples: Read/write/byte/live streams
	97	- Performance: Critical for large files and real-time data
	98
	99	When to use: Large files, real-time feeds, memory-constrained environments
	100
	101	---
	102
	103	### Protocol Multiplexing
	104	What it is: Running multiple protocols over a single connection
	105
	106	Documentation:
	107	- Pattern: [patterns/stream-multiplexing.md](patterns/stream-multiplexing.md)
	108	- Implementing repo: [skills/protomux/skills.md](skills/protomux/skills.md)
	109	- Integration: Used internally by Hypercore replication
	110	- Efficiency: One connection for many protocols
	111
	112	When to use: Custom protocols, efficient connection usage, Hypercore extensions
	113
	114	---
	115
	116	### Binary Encoding
	117	What it is: Compact binary serialization for wire protocols
	118
	119	Documentation:
	120	- Pattern: [patterns/binary-encoding.md](patterns/binary-encoding.md)
	121	- Implementing repo: [skills/compact-encoding/skills.md](skills/compact-encoding/skills.md)
	122	- Performance: Minimal wire format, fast encoding/decoding
	123	- Integration: Used throughout ecosystem for efficiency
	124
	125	When to use: Custom protocols, storage optimization, network efficiency
	126
	127	---
	128
	129	### File System
	130	What it is: POSIX-like file operations with P2P replication
	131
	132	Documentation:
	133	- Pattern: [patterns/file-system.md](patterns/file-system.md)
	134	- Implementing repo: [skills/hyperdrive/skills.md](skills/hyperdrive/skills.md)
	135	- Code examples: put, get, list, del operations
	136	- Use case: Distributed file sharing, version control
	137
	138	When to use: File sharing, distributed storage, P2P CDN
	139
	140	---
	141
	142	## By Use Case
	143
	144	### Building a Distributed Database
	145
	146	What you need:
	147	- Core repos: hypercore, hyperbee, corestore
	148	- Patterns: [append-only-log](patterns/append-only-log.md), [b-tree-index](patterns/b-tree-index.md), [replication](patterns/replication.md)
	149	- Integration guide: See [b-tree-index.md#integration-patterns](patterns/b-tree-index.md)
	150	- Real example: hyper-fleeky/hyperident (profile storage concept)
	151
	152	Quick start:
	153	```js
	154	const Hypercore = require('hypercore')
	155	const Hyperbee = require('hyperbee')
	156
	157	const core = new Hypercore('./db-storage')
	158	const db = new Hyperbee(core, {
	159	keyEncoding: 'utf-8',
	160	valueEncoding: 'json'
	161	})
	162
	163	await db.put('user:1', { name: 'Alice' })
	164	const user = await db.get('user:1')
	165	```
	166
	167	Next steps:
	168	1. Add replication with Hyperswarm
	169	2. Use Corestore for multiple databases
	170	3. Deploy with systemd: [deployment/systemd-services.md](deployment/systemd-services.md)
	171
	172	---
	173
	174	### Building a P2P Social Network
	175
	176	What you need:
	177	- Core repos: hypercore, corestore, hyperswarm
	178	- Patterns: [append-only-log](patterns/append-only-log.md), [replication](patterns/replication.md), [peer-discovery](patterns/peer-discovery.md)
	179	- Real example: hyper-fleeky/fedicore (complete social network)
	180
	181	Architecture:
	182	- Each user has own writable Hypercore (their posts)
	183	- Following = replicating another user's Hypercore
	184	- Hyperswarm discovers peers for each followed user
	185	- Timeline = merge all followed feeds by timestamp
	186
	187	Code reference: See [replication.md#real-world-usage](patterns/replication.md#real-world-usage)
	188
	189	Deployment: [deployment/pear-apps.md](deployment/pear-apps.md)
	190
	191	---
	192
	193	### Building a VPN Gateway
	194
	195	What you need:
	196	- Core repos: hyperswarm, bare-subprocess, utun
	197	- Patterns: [peer-discovery](patterns/peer-discovery.md), [streaming](patterns/streaming.md)
	198	- Linux capabilities: [deployment/capabilities.md](deployment/capabilities.md)
	199	- Real example: hyper-fleeky/hyperident-gateway
	200
	201	Key requirements:
	202	- CAP_NET_ADMIN for TUN device creation
	203	- Systemd service with capabilities (not root)
	204	- Bare runtime deployment (not Pear)
	205
	206	Complete guide: [deployment/systemd-services.md#example-1-hyperident-gateway](deployment/systemd-services.md)
	207
	208	---
	209
	210	### Building a File Sharing App
	211
	212	What you need:
	213	- Core repos: hyperdrive, hyperswarm
	214	- Patterns: [file-system](patterns/file-system.md), [replication](patterns/replication.md), [peer-discovery](patterns/peer-discovery.md)
	215
	216	Quick start:
	217	```js
	218	const Hyperdrive = require('hyperdrive')
	219	const Hyperswarm = require('hyperswarm')
	220
	221	const drive = new Hyperdrive('./files')
	222	await drive.put('/README.md', 'Hello, P2P!')
	223
	224	const swarm = new Hyperswarm()
	225	swarm.on('connection', (socket) => drive.replicate(socket))
	226	swarm.join(drive.discoveryKey)
	227	```
	228
	229	Features: Sparse file sync, version history, metadata
	230
	231	---
	232
	233	### Building with Multi-Writer (Collaborative Apps)
	234
	235	What you need:
	236	- Core repo: autobase
	237	- Built on: hypercore, hyperbee
	238	- Use case: Multiple users writing to shared database
	239
	240	Architecture:
	241	- Each user has own writable Hypercore (input)
	242	- Autobase merges inputs into single view (output)
	243	- CRDT-like semantics for conflict resolution
	244
	245	Pattern: See [b-tree-index.md#pattern-3-hyperbee--autobase](patterns/b-tree-index.md)
	246
	247	Repos: [skills/autobase/skills.md](skills/autobase/skills.md)
	248
	249	---
	250
	251	## By Problem Type
	252
	253	### "How do I..."
	254
	255	#### ...replicate data between peers?
	256	→ [patterns/replication.md](patterns/replication.md)
	257	- Basic replication with Hypercore
	258	- Auto-discovery with Hyperswarm
	259	- Multi-core with Corestore
	260
	261	#### ...discover peers without a server?
	262	→ [patterns/peer-discovery.md](patterns/peer-discovery.md)
	263	- Hyperswarm DHT discovery
	264	- Server/client modes explained
	265	- Direct peer connections
	266
	267	#### ...query data efficiently (not sequential scan)?
	268	→ [patterns/b-tree-index.md](patterns/b-tree-index.md)
	269	- Hyperbee for key-value lookups
	270	- Range queries
	271	- Sorted iteration
	272
	273	#### ...deploy as a systemd service?
	274	→ [deployment/systemd-services.md](deployment/systemd-services.md)
	275	- Service file patterns
	276	- Capabilities (not root!)
	277	- Real-world examples
	278
	279	#### ...handle TUN devices / privileged operations?
	280	→ [deployment/capabilities.md](deployment/capabilities.md)
	281	- CAP_NET_ADMIN for TUN/TAP
	282	- CAP_NET_BIND_SERVICE for port <1024
	283	- CAP_NET_RAW for raw sockets
	284
	285	#### ...verify data integrity?
	286	→ [patterns/cryptography.md](patterns/cryptography.md)
	287	- Merkle tree verification
	288	- Signature verification
	289	- Discovery key derivation
	290
	291	#### ...encrypt Hypercore blocks?
	292	→ [patterns/cryptography.md#code-example-block-encryption](patterns/cryptography.md)
	293	- Block-level encryption
	294	- Key management
	295	- Replication with encryption
	296
	297	#### ...manage multiple Hypercores?
	298	→ [skills/corestore/skills.md](skills/corestore/skills.md)
	299	- Efficient storage
	300	- Single replication call for all cores
	301
	302	#### ...stream large files?
	303	→ [patterns/streaming.md](patterns/streaming.md)
	304	- Read/write/byte streams
	305	- Live streams
	306	- Memory-efficient processing
	307
	308	#### ...test P2P code?
	309	→ Future: [testing/testing-replication.md](testing/) (Phase 4.3)
	310
	311	---
	312
	313	### "Why is..."
	314
	315	#### ...MODULE_NOT_FOUND happening?
	316	→ [deployment/troubleshooting.md#module-not-found](deployment/troubleshooting.md)
	317	- Pear: node_modules not bundled or pear:// bug
	318	- Bare: Import map misconfiguration
	319	- Systemd: Wrong working directory
	320
	321	#### ...replication not working?
	322	→ [deployment/troubleshooting.md#replication-issues](deployment/troubleshooting.md)
	323	- Using publicKey instead of discoveryKey
	324	- Peers not connecting (check Hyperswarm)
	325	- Firewall blocking UDX
	326	- Not waiting for `swarm.flush()`
	327
	328	#### ...my service getting ERR_PERMISSION_REQUIRED?
	329	→ [deployment/systemd-services.md#problem-err_permission_required](deployment/systemd-services.md)
	330	- Running as root (different Pear instance)
	331	- Key not trusted
	332	- Solution: Use regular user + capabilities
	333
	334	#### ...TUN device creation failing?
	335	→ [deployment/capabilities.md#troubleshooting](deployment/capabilities.md)
	336	- Missing CAP_NET_ADMIN capability
	337	- Capability lost after binary update
	338	- Not granted to correct binary (pear-runtime)
	339
	340	#### ...Pear running old version?
	341	→ [deployment/pear-apps.md#cache-management](deployment/pear-apps.md)
	342	- Pear caches aggressively
	343	- Use `--checkout=latest` flag
	344	- Clear cache if needed
	345
	346	#### ...data not syncing?
	347	→ [patterns/replication.md#common-gotchas](patterns/replication.md)
	348	- Check replication initiator (true/false)
	349	- Verify discovery key usage
	350	- Check swarm connections
	351	- Wait for `core.update()`
	352
	353	---
	354
	355	### "What repo should I use for..."
	356
	357	#### ...storing structured data with queries?
	358	→ hyperbee - B-tree database on Hypercore
	359	- [skills/hyperbee/skills.md](skills/hyperbee/skills.md)
	360	- [patterns/b-tree-index.md](patterns/b-tree-index.md)
	361
	362	#### ...P2P file sharing?
	363	→ hyperdrive - File system on Hypercore
	364	- [skills/hyperdrive/skills.md](skills/hyperdrive/skills.md)
	365	- [patterns/file-system.md](patterns/file-system.md)
	366
	367	#### ...peer discovery?
	368	→ hyperswarm - High-level P2P networking
	369	- [skills/hyperswarm/skills.md](skills/hyperswarm/skills.md)
	370	- [patterns/peer-discovery.md](patterns/peer-discovery.md)
	371
	372	#### ...append-only log?
	373	→ hypercore - The foundation
	374	- [skills/hypercore/skills.md](skills/hypercore/skills.md)
	375	- [patterns/append-only-log.md](patterns/append-only-log.md)
	376
	377	#### ...multi-writer databases?
	378	→ autobase - CRDT-like multi-writer
	379	- [skills/autobase/skills.md](skills/autobase/skills.md)
	380
	381	#### ...managing many Hypercores?
	382	→ corestore - Efficient multi-core management
	383	- [skills/corestore/skills.md](skills/corestore/skills.md)
	384
	385	#### ...custom wire protocols?
	386	→ protomux + compact-encoding
	387	- [patterns/stream-multiplexing.md](patterns/stream-multiplexing.md)
	388	- [patterns/binary-encoding.md](patterns/binary-encoding.md)
	389
	390	---
	391
	392	## By Repository
	393
	394	### Tier 1 - Building Blocks (Core)
	395
	396	#### Hypercore
	397	- What: Secure, distributed append-only log
	398	- When: Foundation for everything
	399	- Docs: [skills/hypercore/skills.md](skills/hypercore/skills.md)
	400	- Pattern: [patterns/append-only-log.md](patterns/append-only-log.md)
	401	- Real usage: fedicore (user posts), hyperident (identity log)
	402
	403	#### Hyperbee
	404	- What: B-tree database on Hypercore
	405	- When: Need key-value with queries
	406	- Docs: [skills/hyperbee/skills.md](skills/hyperbee/skills.md)
	407	- Pattern: [patterns/b-tree-index.md](patterns/b-tree-index.md)
	408	- Real usage: Future hyper-fleeky projects (profiles)
	409
	410	#### Hyperdrive
	411	- What: File system on Hypercore
	412	- When: Distributing files
	413	- Docs: [skills/hyperdrive/skills.md](skills/hyperdrive/skills.md)
	414	- Pattern: [patterns/file-system.md](patterns/file-system.md)
	415
	416	#### Autobase
	417	- What: Multi-writer with CRDTs
	418	- When: Collaborative applications
	419	- Docs: [skills/autobase/skills.md](skills/autobase/skills.md)
	420	- Integration: [patterns/b-tree-index.md#pattern-3](patterns/b-tree-index.md)
	421
	422	#### Hyperswarm
	423	- What: P2P networking with DHT
	424	- When: Need peer discovery
	425	- Docs: [skills/hyperswarm/skills.md](skills/hyperswarm/skills.md)
	426	- Pattern: [patterns/peer-discovery.md](patterns/peer-discovery.md)
	427	- Real usage: fedicore (finding peers), hyperident-gateway (VPN)
	428
	429	#### HyperDHT
	430	- What: Low-level DHT implementation
	431	- When: Need custom DHT behavior
	432	- Docs: [skills/hyperdht/skills.md](skills/hyperdht/skills.md)
	433	- Built on by: Hyperswarm
	434
	435	---
	436
	437	### Tier 2 - Helpers
	438
	439	#### Corestore
	440	- What: Manage multiple Hypercores
	441	- When: Apps with many cores
	442	- Docs: [skills/corestore/skills.md](skills/corestore/skills.md)
	443	- Real usage: fedicore (per-user feeds)
	444
	445	#### Protomux
	446	- What: Protocol multiplexing
	447	- When: Multiple protocols, one connection
	448	- Docs: [skills/protomux/skills.md](skills/protomux/skills.md)
	449	- Pattern: [patterns/stream-multiplexing.md](patterns/stream-multiplexing.md)
	450
	451	#### Compact-encoding
	452	- What: Binary serialization
	453	- When: Custom wire formats
	454	- Docs: [skills/compact-encoding/skills.md](skills/compact-encoding/skills.md)
	455	- Pattern: [patterns/binary-encoding.md](patterns/binary-encoding.md)
	456
	457	#### Localdrive
	458	- What: Local file system abstraction
	459	- When: Mirror to/from local disk
	460	- Docs: [skills/localdrive/skills.md](skills/localdrive/skills.md)
	461
	462	---
	463
	464	### Tier 3 - Utilities
	465
	466	#### hypercore-crypto
	467	- What: Cryptographic primitives
	468	- When: Signing, verification, hashing
	469	- Docs: [skills/hypercore-crypto/skills.md](skills/hypercore-crypto/skills.md)
	470	- Pattern: [patterns/cryptography.md](patterns/cryptography.md)
	471	- Real usage: hyperident (key management)
	472
	473	---
	474
	475	## Deployment & Operations
	476
	477	### Pear Deployment
	478	- Guide: [deployment/pear-apps.md](deployment/pear-apps.md)
	479	- Critical bugs: pear:// module resolution, ESM/CJS interop
	480	- Workarounds: Local directory deployment, CommonJS
	481	- Cache management: Clearing cache, force latest
	482
	483	### Systemd Services
	484	- Guide: [deployment/systemd-services.md](deployment/systemd-services.md)
	485	- Key pattern: Use capabilities, not root!
	486	- Real example: hyperident-gateway (VPN)
	487	- Common issues: ERR_PERMISSION_REQUIRED
	488
	489	### Linux Capabilities
	490	- Guide: [deployment/capabilities.md](deployment/capabilities.md)
	491	- Common: CAP_NET_ADMIN (TUN), CAP_NET_BIND_SERVICE (ports <1024)
	492	- Real usage: hyperident-gateway (TUN devices)
	493	- Troubleshooting: Capability lost after update
	494
	495	### Bare Runtime
	496	- Guide: [deployment/bare-runtime.md](deployment/bare-runtime.md)
	497	- When: Systemd services, workaround Pear bugs
	498	- Comparison: Bare vs Pear feature matrix
	499	- Migration: Pear → Bare patterns
	500
	501	### Module Resolution
	502	- Guide: [deployment/module-resolution.md](deployment/module-resolution.md)
	503	- Critical bugs: pear:// protocol, ESM/CJS hoisting
	504	- Failed attempts: Documented for bug reports
	505	- Workarounds: Local directory, CommonJS
	506
	507	### Troubleshooting
	508	- Index: [deployment/troubleshooting.md](deployment/troubleshooting.md)
	509	- By symptom: MODULE_NOT_FOUND, permission errors, etc.
	510	- By technology: Hypercore, Pear, Systemd
	511	- Decision trees: Step-by-step debugging
	512
	513	---
	514
	515	## Real-World Projects
	516
	517	### hyper-fleeky/fedicore
	518	- What: P2P social network
	519	- Uses: Hypercore, Corestore, Hyperswarm
	520	- Patterns: Append-only-log, replication, peer-discovery
	521	- Code: Complete multi-user feed replication
	522	- Docs: Referenced in [replication.md](patterns/replication.md), [peer-discovery.md](patterns/peer-discovery.md)
	523
	524	### hyper-fleeky/hyperident
	525	- What: Decentralized identity management
	526	- Uses: Hypercore, cryptography
	527	- Patterns: Cryptography, key management
	528	- Code: Identity encryption, device linking
	529	- Docs: Referenced in [cryptography.md](patterns/cryptography.md)
	530
	531	### hyper-fleeky/hyperident-gateway
	532	- What: P2P VPN gateway
	533	- Uses: Hyperswarm, bare-subprocess, utun
	534	- Patterns: Peer-discovery, streaming, Linux capabilities
	535	- Deployment: Systemd with CAP_NET_ADMIN
	536	- Docs: Referenced in [deployment/systemd-services.md](deployment/systemd-services.md), [deployment/capabilities.md](deployment/capabilities.md)
	537
	538	---
	539
	540	## Cross-Pattern Relationships
	541
	542	### Append-Only Log + Replication
	543	Every P2P app uses this combination. Hypercore provides the log, replication syncs it.
	544	- [patterns/append-only-log.md](patterns/append-only-log.md)
	545	- [patterns/replication.md](patterns/replication.md)
	546
	547	### Append-Only Log + B-Tree Index
	548	Database pattern: Hypercore for storage, Hyperbee for queries.
	549	- [patterns/append-only-log.md](patterns/append-only-log.md)
	550	- [patterns/b-tree-index.md](patterns/b-tree-index.md)
	551
	552	### Replication + Peer Discovery
	553	Can't replicate without finding peers first.
	554	- [patterns/replication.md](patterns/replication.md)
	555	- [patterns/peer-discovery.md](patterns/peer-discovery.md)
	556
	557	### Cryptography + Everything
	558	All patterns depend on cryptographic verification.
	559	- [patterns/cryptography.md](patterns/cryptography.md)
	560
	561	### Streaming + File System
	562	Hyperdrive uses streaming internally for efficient file transfer.
	563	- [patterns/streaming.md](patterns/streaming.md)
	564	- [patterns/file-system.md](patterns/file-system.md)
	565
	566	---
	567
	568	## Entry Points for Claude
	569
	570	### I need to understand the basics
	571	Start here:
	572	1. [README.md](README.md) - Ecosystem overview
	573	2. [patterns/append-only-log.md](patterns/append-only-log.md) - Foundation
	574	3. [patterns/replication.md](patterns/replication.md) - P2P sync
	575	4. [patterns/peer-discovery.md](patterns/peer-discovery.md) - Finding peers
	576
	577	### I need to build something
	578	Go to: [By Use Case](#by-use-case)
	579
	580	### I have an error
	581	Go to: [By Problem Type](#by-problem-type) or [deployment/troubleshooting.md](deployment/troubleshooting.md)
	582
	583	### I need API docs
	584	Go to: [skills/](skills/) directory (529 repos documented)
	585
	586	### I need deployment help
	587	Go to: [deployment/](deployment/) directory (7 comprehensive guides)
	588
	589	---
	590
	591	## Document Types
	592
	593	### Patterns (How Things Work)
	594	- Location: [patterns/](patterns/)
	595	- Purpose: Explain concepts with code examples
	596	- Format: Description, when to use, code examples, gotchas, anti-patterns
	597	- Count: 9 core patterns
	598
	599	### Skills (API Reference)
	600	- Location: [skills/](skills/)
	601	- Purpose: API documentation for each repo
	602	- Format: Installation, dependencies, API, examples
	603	- Count: 529 repos
	604
	605	### Deployment (Production)
	606	- Location: [deployment/](deployment/)
	607	- Purpose: Deployment guides, troubleshooting
	608	- Format: Step-by-step guides, real examples, known bugs
	609	- Count: 7 comprehensive guides
	610
	611	### Semantic Maps (Structured Data)
	612	- Location: Root directory (JSON files)
	613	- Purpose: Machine-readable categorization
	614	- Files: repo_map.json, use-case-map.json, pattern-map.json, etc.
	615
	616	---
	617
	618	## Navigation Tips
	619
	620	1. Start broad: Use this CONCEPTS.md to find relevant documentation
	621	2. Go deep: Follow links to patterns, skills, deployment guides
	622	3. Cross-reference: Documents heavily link to each other
	623	4. Real examples: Look for "Real-World Usage" sections
	624	5. Troubleshoot: Check troubleshooting index for errors
	625
	626	---
	627
	628	Last Updated: 2026-01-18
	629	Phase: 5.1 - Concept Graph (Master Index)
	630	Status: ✅ Complete

README.md +30 -1

…	…	@@ -14,7 +14,36 @@ This repository provides:
14	14	- Maturity metrics (stars, activity, status)
15	15	- Designer & Architect sub-agents for guided P2P app development
16	16
17		## 🚀 Getting Started
	17	## 📖 Quick Lookup for Claude
	18
	19	New to this knowledge base? Start with the [Concept Graph](CONCEPTS.md) - the master index for all documentation.
	20
	21	### I need to...
	22
	23	- Understand a concept → [CONCEPTS.md](CONCEPTS.md) → Find by concept (replication, cryptography, etc.)
	24	- Build something → [CONCEPTS.md#by-use-case](CONCEPTS.md#by-use-case) → Guides for common projects
	25	- Fix an error → [deployment/troubleshooting.md](deployment/troubleshooting.md) → Symptom-based index
	26	- Learn a pattern → [patterns/](patterns/) → 9 patterns with code examples
	27	- Deploy to production → [deployment/](deployment/) → 7 comprehensive guides
	28	- Read API docs → [skills/](skills/) → 529 repos documented
	29	- See real examples → [CONCEPTS.md#real-world-projects](CONCEPTS.md#real-world-projects) → hyper-fleeky projects
	30
	31	### Quick Links
	32
	33	\| Need \| Go To \|
	34	\|------\|-------\|
	35	\| P2P basics \| [patterns/append-only-log.md](patterns/append-only-log.md) \|
	36	\| Data sync \| [patterns/replication.md](patterns/replication.md) \|
	37	\| Finding peers \| [patterns/peer-discovery.md](patterns/peer-discovery.md) \|
	38	\| Database queries \| [patterns/b-tree-index.md](patterns/b-tree-index.md) \|
	39	\| Security/crypto \| [patterns/cryptography.md](patterns/cryptography.md) \|
	40	\| Deploy with Pear \| [deployment/pear-apps.md](deployment/pear-apps.md) \|
	41	\| Deploy with systemd \| [deployment/systemd-services.md](deployment/systemd-services.md) \|
	42	\| Troubleshooting \| [deployment/troubleshooting.md](deployment/troubleshooting.md) \|
	43
	44	---
	45
	46	## 🚀 Getting Started (For Users)
18	47
19	48	### New to Building P2P Apps?
20	49

architect-agent.js +3 -1

…	…	@@ -18,7 +18,9 @@ const patternMap = JSON.parse(fs.readFileSync('pattern-map.json', 'utf8'));
18	18	const dependencyGraph = JSON.parse(fs.readFileSync('dependency-graph.json', 'utf8'));
19	19	const useCaseMap = JSON.parse(fs.readFileSync('use-case-map.json', 'utf8'));
20	20	const stabilityIndex = JSON.parse(fs.readFileSync('stability-index.json', 'utf8'));
21		const repoMapArray = JSON.parse(fs.readFileSync('repo_map.json', 'utf8'));
	21	const repoMapData = JSON.parse(fs.readFileSync('repo_map.json', 'utf8'));
	22	// Support both old array format and new object format with _meta
	23	const repoMapArray = repoMapData.repos \|\| repoMapData;
22	24
23	25	// Convert repo map to object
24	26	const repoMap = {};

build_semantic_maps.js +3 -1

…	…	@@ -18,7 +18,9 @@ console.log('🚀 Phase 2: Building Semantic Maps\n');
18	18
19	19	// Load Phase 1 data
20	20	console.log('📥 Loading Phase 1 data...');
21		const repoMapArray = JSON.parse(fs.readFileSync('repo_map.json', 'utf8'));
	21	const repoMapData = JSON.parse(fs.readFileSync('repo_map.json', 'utf8'));
	22	// Support both old array format and new object format with _meta
	23	const repoMapArray = repoMapData.repos \|\| repoMapData;
22	24	const dependencyGraph = JSON.parse(fs.readFileSync('dependency-graph.json', 'utf8'));
23	25	const techStack = JSON.parse(fs.readFileSync('tech-stack.json', 'utf8'));
24	26	const maturityIndex = JSON.parse(fs.readFileSync('maturity-index.json', 'utf8'));

dependency-graph.json +15 -0

…	…	@@ -1,4 +1,19 @@
1	1	{
	2	"_meta": {
	3	"description": "Complete dependency mapping for all 529 holepunchto repositories",
	4	"documentation": {
	5	"concepts": "See CONCEPTS.md#by-repository for navigating dependencies",
	6	"usage": "Use reverseDependencies to find which repos depend on a given package"
	7	},
	8	"structure": {
	9	"dependencies": "Direct npm dependencies",
	10	"devDependencies": "Development dependencies",
	11	"reverseDependencies": "Repos that depend on this package (deduplicated)"
	12	},
	13	"totalRepos": 529,
	14	"lastUpdated": "2026-01-18",
	15	"phase": "5.4 - JSON Metadata Enhancement"
	16	},
2	17	".github": {
3	18	"dependencies": [],
4	19	"devDependencies": [],

deployment/README.md +104 -0

…	…	@@ -0,0 +1,104 @@
	1	# Deployment Documentation
	2
	3	This directory contains production deployment knowledge for P2P applications built with the holepunchto ecosystem.
	4
	5	## Overview
	6
	7	Deployment documentation is organized by runtime and deployment method:
	8
	9	- [pear-apps.md](./pear-apps.md) - Deploying Pear applications (terminal & desktop)
	10	- [systemd-services.md](./systemd-services.md) - Running P2P apps as systemd services
	11	- [bare-runtime.md](./bare-runtime.md) - Deploying apps with Bare runtime
	12	- [module-resolution.md](./module-resolution.md) - Module resolution issues and workarounds
	13	- [capabilities.md](./capabilities.md) - Linux capabilities for privileged operations
	14	- [troubleshooting.md](./troubleshooting.md) - Common deployment issues
	15
	16	## Knowledge Sources
	17
	18	This documentation combines:
	19
	20	1. Official pear-docs - Canonical deployment patterns from holepunchto/pear-docs
	21	2. Real-world lessons - Production deployment experience from hyper-fleeky projects
	22	3. Validated patterns - Tested against holepunchto/pear examples and tier 1/2 repos
	23
	24	## Quick Start
	25
	26	### For Pear Terminal Apps
	27
	28	```bash
	29	# Stage and release
	30	cd your-pear-app
	31	pear stage production
	32	pear release production
	33	pear seed production
	34
	35	# Run
	36	pear run pear://your-key
	37
	38	# See: pear-apps.md for details
	39	```
	40
	41	### For Systemd Services
	42
	43	```bash
	44	# Create service file
	45	sudo nano /etc/systemd/system/your-app.service
	46
	47	# Enable and start
	48	sudo systemctl enable your-app
	49	sudo systemctl start your-app
	50
	51	# See: systemd-services.md for complete setup
	52	```
	53
	54	### For Bare Runtime
	55
	56	```bash
	57	# Run directly
	58	bare index.js
	59
	60	# With production optimizations
	61	NODE_ENV=production bare index.js
	62
	63	# See: bare-runtime.md for details
	64	```
	65
	66	## Common Deployment Issues
	67
	68	\| Symptom \| Likely Cause \| Document \|
	69	\|---------\|--------------\|----------\|
	70	\| MODULE_NOT_FOUND in Pear \| node_modules not bundled \| [module-resolution.md](./module-resolution.md) \|
	71	\| ERR_PERMISSION_REQUIRED \| Running as root user \| [systemd-services.md](./systemd-services.md#use-capabilities-not-root) \|
	72	\| Pear cache issues \| Old version cached \| [pear-apps.md](./pear-apps.md#cache-management) \|
	73	\| Permission denied (TUN) \| Missing CAP_NET_ADMIN \| [capabilities.md](./capabilities.md) \|
	74	\| Dynamic require fails \| Not explicitly in package.json \| [module-resolution.md](./module-resolution.md#dynamic-requires) \|
	75
	76	See [troubleshooting.md](./troubleshooting.md) for comprehensive symptom index.
	77
	78	## Real-World Validation
	79
	80	All deployment patterns have been validated against:
	81
	82	- holepunchto/pear - Official Pear examples and test fixtures
	83	- holepunchto repos - Tier 1/2 repos with production patterns
	84	- hyper-fleeky projects - Real production deployments:
	85	- hyperident-gateway (P2P VPN, systemd service, CAP_NET_ADMIN)
	86	- hyperident (Pear terminal app)
	87	- fedicore (HTTP server)
	88
	89	## Contributing
	90
	91	When adding deployment knowledge:
	92
	93	1. Validate against holepunchto examples first
	94	2. Cross-reference with official pear-docs
	95	3. Test in real deployment scenarios
	96	4. Document both success and failure cases
	97	5. Include workarounds for known bugs
	98
	99	## See Also
	100
	101	- [Official Pear Docs](https://docs.pears.com)
	102	- [holepunchto/pear repository](https://github.com/holepunchto/pear)
	103	- [../patterns/](../patterns/) - P2P patterns that need deployment
	104	- [../troubleshooting.md](../troubleshooting.md) - General troubleshooting

deployment/bare-runtime.md +767 -0

…	…	@@ -0,0 +1,767 @@
	1	# Deploying with Bare Runtime
	2
	3	Validated against: hyper-fleeky/hyperident-gateway, holepunchto/bare
	4
	5	## Overview
	6
	7	Bare is the minimal JavaScript runtime that Pear is built on. Deploying directly with Bare gives maximum control and avoids Pear wrapper issues.
	8
	9	When to use Bare directly:
	10	- Systemd services (no Pear features needed)
	11	- Workaround for pear:// module resolution bug
	12	- Maximum performance (no wrapper overhead)
	13	- Headless servers
	14
	15	When to use Pear instead:
	16	- Need Pear APIs (Pear.config, Pear.updates, etc.)
	17	- Desktop apps (Pear handles UI runtime)
	18	- P2P distribution via pear:// links
	19	- Auto-updates
	20
	21	---
	22
	23	## Table of Contents
	24
	25	1. [Basic Bare Deployment](#basic-bare-deployment)
	26	2. [Finding Bare Binary](#finding-bare-binary)
	27	3. [Bare vs Pear Comparison](#bare-vs-pear-comparison)
	28	4. [Systemd Service with Bare](#systemd-service-with-bare)
	29	5. [Module Resolution in Bare](#module-resolution-in-bare)
	30	6. [Production Configuration](#production-configuration)
	31	7. [Troubleshooting](#troubleshooting)
	32
	33	---
	34
	35	## Basic Bare Deployment
	36
	37	### Method 1: Dump from Pear
	38
	39	Extract app from Pear link, run with Bare:
	40
	41	```bash
	42	# Dump app to directory
	43	pear dump pear://your-key /opt/your-app
	44
	45	# Run with Bare
	46	cd /opt/your-app
	47	bare index.js
	48	```
	49
	50	Why this works:
	51	- Gets all files and dependencies from Pear staging
	52	- Avoids pear:// module resolution bug
	53	- Pure Bare module resolution (more reliable)
	54
	55	### Method 2: Direct Development
	56
	57	Develop directly with Bare, never use Pear:
	58
	59	```bash
	60	# Your app directory
	61	cd your-app/
	62
	63	# Install dependencies
	64	npm install
	65
	66	# Run with Bare
	67	bare index.js
	68	```
	69
	70	Requirements:
	71	- Use Bare-compatible modules (bare-fs, bare-events, etc.)
	72	- Set up import maps in package.json
	73	- Don't use Pear APIs
	74
	75	### Method 3: Hybrid
	76
	77	Develop with Pear, deploy with Bare:
	78
	79	```bash
	80	# Development
	81	pear run --dev .
	82
	83	# Stage and test with Pear
	84	pear stage production
	85	pear run pear://key
	86
	87	# Deploy with Bare (workaround)
	88	pear dump pear://key /opt/app
	89	cd /opt/app
	90	bare index.js
	91	```
	92
	93	Best of both worlds:
	94	- Use Pear tooling for development
	95	- Avoid Pear bugs in production
	96	- More control over deployment
	97
	98	---
	99
	100	## Finding Bare Binary
	101
	102	### From Pear Installation
	103
	104	Pear includes Bare runtime:
	105
	106	```bash
	107	# Find bare in Pear installation
	108	find ~/.config/pear -name "bare" -type f \| head -1
	109	```
	110
	111	Typical location:
	112	```
	113	~/.config/pear/by-dkey/<key>/by-arch/<platform>-<arch>/bin/bare
	114	```
	115
	116	Example:
	117	```bash
	118	~/.config/pear/by-dkey/x5w.../by-arch/linux-x64/bin/bare
	119	```
	120
	121	### Standalone Bare Installation
	122
	123	Install Bare separately (not via Pear):
	124
	125	```bash
	126	# Not yet available as standalone package
	127	# Currently distributed via Pear
	128	```
	129
	130	Note: As of 2026-01, Bare is primarily distributed through Pear. Standalone distribution may be available in the future.
	131
	132	### Finding pear-runtime (Bare wrapper)
	133
	134	Pear uses a wrapped version called pear-runtime:
	135
	136	```bash
	137	find ~/.config/pear -name "pear-runtime" -type f \| head -1
	138	```
	139
	140	Both work the same:
	141	- `bare index.js` - Direct Bare
	142	- `pear-runtime index.js` - Pear's wrapped Bare
	143	- For capabilities, grant to pear-runtime
	144
	145	---
	146
	147	## Bare vs Pear Comparison
	148
	149	### Feature Comparison
	150
	151	\| Feature \| Bare \| Pear \|
	152	\|---------\|------\|------\|
	153	\| Module resolution \| ✅ Reliable \| ❌ Has bugs under pear:// \|
	154	\| ESM/CJS interop \| ⚠️ Some issues \| ⚠️ Same issues (uses Bare) \|
	155	\| P2P distribution \| ❌ No \| ✅ Via pear:// links \|
	156	\| Auto-updates \| ❌ No \| ✅ Via Pear.updates \|
	157	\| Pear APIs \| ❌ Not available \| ✅ Pear.config, Pear.app, etc. \|
	158	\| Systemd service \| ✅ Simple \| ✅ Works but more complex \|
	159	\| Capabilities \| ✅ Grant to binary \| ✅ Grant to pear-runtime \|
	160	\| Startup time \| ✅ Faster \| ⚠️ Slightly slower (wrapper) \|
	161	\| Debugging \| ✅ Direct \| ⚠️ Extra wrapper layer \|
	162
	163	### When to Choose Bare
	164
	165	✅ Use Bare when:
	166	- Running as systemd service
	167	- Don't need Pear features
	168	- Working around module resolution bug
	169	- Want maximum control
	170	- Deploying to headless server
	171
	172	Example use cases:
	173	- VPN gateway (hyperident-gateway)
	174	- Background worker
	175	- API server
	176	- CLI tool (if not distributed via Pear)
	177
	178	### When to Choose Pear
	179
	180	✅ Use Pear when:
	181	- Need P2P distribution
	182	- Want auto-updates
	183	- Using Pear APIs (config, routes, assets)
	184	- Desktop app (need UI runtime)
	185	- Sharing with users via pear:// links
	186
	187	Example use cases:
	188	- Desktop applications
	189	- Distributed apps for end users
	190	- Apps needing Pear.config
	191	- Apps using Pear routing
	192
	193	---
	194
	195	## Systemd Service with Bare
	196
	197	### Service File Pattern
	198
	199	```ini
	200	[Unit]
	201	Description=Your App (Bare Runtime)
	202	After=network.target
	203
	204	[Service]
	205	Type=simple
	206	User=youruser
	207	WorkingDirectory=/opt/your-app
	208
	209	# Set NODE_ENV for production
	210	Environment="NODE_ENV=production"
	211
	212	# Run with bare directly
	213	ExecStart=/home/youruser/.config/pear/by-dkey/.../bin/bare index.js
	214
	215	# Or use full path to pear-runtime
	216	# ExecStart=/home/youruser/.config/pear/by-dkey/.../bin/pear-runtime index.js
	217
	218	Restart=always
	219	RestartSec=10
	220
	221	[Install]
	222	WantedBy=multi-user.target
	223	```
	224
	225	### With Capabilities
	226
	227	For privileged operations (VPN, raw sockets):
	228
	229	```ini
	230	[Service]
	231	User=youruser
	232	WorkingDirectory=/opt/your-app
	233
	234	# Grant capabilities
	235	AmbientCapabilities=CAP_NET_ADMIN
	236	CapabilityBoundingSet=CAP_NET_ADMIN
	237
	238	ExecStart=/home/youruser/.config/pear/by-dkey/.../bin/pear-runtime index.js
	239	```
	240
	241	Grant capability to binary:
	242
	243	```bash
	244	BARE=$(find ~/.config/pear -name "pear-runtime" \| head -1)
	245	sudo setcap cap_net_admin+ep "$BARE"
	246	```
	247
	248	Why pear-runtime and not bare:
	249	- When using Pear-installed Bare, the actual binary is pear-runtime
	250	- It's a wrapper around Bare
	251	- Grant capabilities to whatever binary you're executing
	252
	253	### Complete Setup Example
	254
	255	```bash
	256	# 1. Dump app from Pear
	257	pear dump pear://key /opt/your-app
	258
	259	# 2. Find bare binary
	260	BARE=$(find ~/.config/pear -name "pear-runtime" \| head -1)
	261	echo "Found: $BARE"
	262
	263	# 3. Create service file
	264	cat > /tmp/your-app.service << 'EOF'
	265	[Unit]
	266	Description=Your App
	267	After=network.target
	268
	269	[Service]
	270	Type=simple
	271	User=$(whoami)
	272	WorkingDirectory=/opt/your-app
	273	Environment="NODE_ENV=production"
	274	ExecStart=$BARE /opt/your-app/index.js
	275	Restart=always
	276
	277	[Install]
	278	WantedBy=multi-user.target
	279	EOF
	280
	281	# 4. Install service
	282	sudo cp /tmp/your-app.service /etc/systemd/system/
	283	sudo systemctl daemon-reload
	284	sudo systemctl enable your-app
	285	sudo systemctl start your-app
	286
	287	# 5. Check status
	288	sudo systemctl status your-app
	289	journalctl -u your-app -f
	290	```
	291
	292	See: [systemd-services.md](./systemd-services.md) for complete systemd guide
	293
	294	---
	295
	296	## Module Resolution in Bare
	297
	298	### Import Maps
	299
	300	Bare uses import maps for runtime-specific module resolution:
	301
	302	package.json:
	303	```json
	304	{
	305	"imports": {
	306	"fs": {
	307	"bare": "bare-fs",
	308	"default": "fs"
	309	},
	310	"path": {
	311	"bare": "bare-path",
	312	"default": "path"
	313	},
	314	"events": {
	315	"bare": "bare-events",
	316	"default": "events"
	317	}
	318	}
	319	}
	320	```
	321
	322	In code:
	323	```javascript
	324	// Automatically resolves to bare-fs in Bare, fs in Node
	325	const fs = require('fs')
	326	```
	327
	328	### CommonJS vs ESM
	329
	330	Recommendation: Use CommonJS with Bare
	331
	332	CommonJS (recommended):
	333	```json
	334	{
	335	"type": "commonjs" // Or omit
	336	}
	337	```
	338	```javascript
	339	const Hypercore = require('hypercore')
	340	const fs = require('fs')
	341	```
	342
	343	ESM (has issues):
	344	```json
	345	{
	346	"type": "module"
	347	}
	348	```
	349	```javascript
	350	import Hypercore from 'hypercore'
	351	import fs from 'fs'
	352	```
	353
	354	Issue: Bare's ESM/CJS interop has bugs with hoisted dependencies
	355
	356	See: [module-resolution.md](./module-resolution.md#esmcjs-interop)
	357
	358	### Module Resolution Order
	359
	360	Bare resolves modules in this order:
	361
	362	1. Import map (if defined in package.json)
	363	2. Bare builtins (if Bare-specific, e.g., bare-fs)
	364	3. node_modules (standard Node resolution)
	365	4. Node builtins (if available)
	366
	367	Example:
	368	```javascript
	369	require('fs')
	370	// 1. Check import map: "fs" → "bare-fs" (in Bare)
	371	// 2. Resolve "bare-fs" from node_modules
	372	// 3. Return bare-fs
	373	```
	374
	375	---
	376
	377	## Production Configuration
	378
	379	### Environment Variables
	380
	381	```bash
	382	# Set production mode
	383	export NODE_ENV=production
	384
	385	# Disable debug logging
	386	export DEBUG=
	387
	388	# Set custom config path (if app supports)
	389	export CONFIG_PATH=/etc/your-app/config.json
	390
	391	# Run
	392	bare index.js
	393	```
	394
	395	In systemd:
	396	```ini
	397	[Service]
	398	Environment="NODE_ENV=production"
	399	Environment="CONFIG_PATH=/etc/your-app/config.json"
	400	```
	401
	402	### Dependencies
	403
	404	Install production dependencies only:
	405
	406	```bash
	407	cd /opt/your-app
	408	npm install --production
	409
	410	# Or prune dev deps
	411	npm prune --omit=dev
	412	```
	413
	414	### File Structure
	415
	416	Typical production layout:
	417
	418	```
	419	/opt/your-app/
	420	├── index.js # Entry point
	421	├── lib/ # App code
	422	├── node_modules/ # Dependencies
	423	├── package.json
	424	└── config/ # Configuration
	425	└── production.json
	426	```
	427
	428	### Logging
	429
	430	Bare apps should log to stdout/stderr for systemd:
	431
	432	```javascript
	433	// Good: logs captured by systemd
	434	console.log('Info message')
	435	console.error('Error message')
	436
	437	// Avoid: separate log files (harder to manage)
	438	// fs.appendFileSync('/var/log/app.log', msg)
	439	```
	440
	441	View logs:
	442	```bash
	443	journalctl -u your-app -f
	444	```
	445
	446	---
	447
	448	## Troubleshooting
	449
	450	### "Cannot find module 'bare-fs'"
	451
	452	Cause: Missing bare-specific module or wrong import map
	453
	454	Check import map:
	455	```bash
	456	jq '.imports' package.json
	457	```
	458
	459	Should have:
	460	```json
	461	{
	462	"imports": {
	463	"fs": {
	464	"bare": "bare-fs",
	465	"default": "fs"
	466	}
	467	}
	468	}
	469	```
	470
	471	Install if missing:
	472	```bash
	473	npm install bare-fs
	474	```
	475
	476	### "MODULE_NOT_FOUND" for regular dependency
	477
	478	Check if installed:
	479	```bash
	480	ls node_modules/your-module
	481	# Should exist
	482	```
	483
	484	Reinstall:
	485	```bash
	486	npm install
	487	```
	488
	489	Check you're in app directory:
	490	```bash
	491	pwd
	492	# Should be /opt/your-app, not /
	493	```
	494
	495	### Bare command not found
	496
	497	Find bare:
	498	```bash
	499	find ~/.config/pear -name "bare" -o -name "pear-runtime" \| head -1
	500	```
	501
	502	Use full path:
	503	```bash
	504	/home/user/.config/pear/by-dkey/.../bin/bare index.js
	505	```
	506
	507	Or add to PATH (not recommended for services):
	508	```bash
	509	export PATH="$(dirname $(find ~/.config/pear -name bare \| head -1)):$PATH"
	510	```
	511
	512	### ESM import fails
	513
	514	Error:
	515	```
	516	MODULE_NOT_FOUND: Cannot find module 'bare-subprocess'
	517	```
	518
	519	Solution: Convert to CommonJS
	520
	521	```bash
	522	# Remove "type": "module" from package.json
	523	jq 'del(.type)' package.json > package.json.tmp
	524	mv package.json.tmp package.json
	525	```
	526
	527	Convert imports to requires:
	528	```javascript
	529	// Before (ESM)
	530	import Hypercore from 'hypercore'
	531
	532	// After (CommonJS)
	533	const Hypercore = require('hypercore')
	534	```
	535
	536	See: [module-resolution.md](./module-resolution.md#esmcjs-interop)
	537
	538	### Permission denied
	539
	540	For regular operations: Check file permissions
	541
	542	```bash
	543	ls -la /opt/your-app
	544	# Should be owned by service user
	545	```
	546
	547	For privileged operations: Grant capabilities
	548
	549	```bash
	550	# Find binary
	551	BARE=$(find ~/.config/pear -name "pear-runtime" \| head -1)
	552
	553	# Grant capability
	554	sudo setcap cap_net_admin+ep "$BARE"
	555	```
	556
	557	See: [capabilities.md](./capabilities.md)
	558
	559	---
	560
	561	## Real-World Examples
	562
	563	### Example 1: hyperident-gateway (VPN)
	564
	565	Setup:
	566	```bash
	567	# Dump from Pear
	568	pear dump pear://hdj3chnrp8wnfzq4b3fd9i1jhuha1ufeyces35oofyteh6csfsmy /opt/hyperident-gateway
	569
	570	# Find bare
	571	BARE=$(find ~/.config/pear -name "pear-runtime" \| head -1)
	572
	573	# Grant CAP_NET_ADMIN for TUN devices
	574	sudo setcap cap_net_admin+ep "$BARE"
	575	```
	576
	577	Service file: `/etc/systemd/system/hyperident-gateway.service`
	578	```ini
	579	[Unit]
	580	Description=Hyperident P2P VPN Gateway
	581	After=network.target
	582
	583	[Service]
	584	Type=simple
	585	User=fleeky
	586	WorkingDirectory=/opt/hyperident-gateway
	587	Environment="NODE_ENV=production"
	588	AmbientCapabilities=CAP_NET_ADMIN
	589	CapabilityBoundingSet=CAP_NET_ADMIN
	590	ExecStart=/home/fleeky/.config/pear/by-dkey/.../bin/pear-runtime gateway.js
	591	Restart=always
	592
	593	[Install]
	594	WantedBy=multi-user.target
	595	```
	596
	597	Why Bare:
	598	- Workaround for pear:// module resolution bug
	599	- Full control for systemd service
	600	- No Pear features needed
	601
	602	Source: hyper-fleeky/repos/hyperident-gateway
	603
	604	### Example 2: Simple API Server
	605
	606	App structure:
	607	```javascript
	608	// index.js
	609	const http = require('http')
	610	const Hyperswarm = require('hyperswarm')
	611
	612	const server = http.createServer((req, res) => {
	613	res.end('Hello from P2P API')
	614	})
	615
	616	server.listen(3000)
	617	console.log('Listening on port 3000')
	618	```
	619
	620	Deploy:
	621	```bash
	622	# Install deps
	623	npm install hyperswarm
	624
	625	# Run with bare
	626	bare index.js
	627	```
	628
	629	Service file:
	630	```ini
	631	[Service]
	632	User=apiuser
	633	WorkingDirectory=/opt/api
	634	ExecStart=/home/apiuser/.config/pear/by-dkey/.../bin/bare index.js
	635	```
	636
	637	### Example 3: CLI Tool
	638
	639	For distributed CLI tool, use Pear:
	640	```bash
	641	pear stage production
	642	pear release production
	643	# Users run: pear run pear://key
	644	```
	645
	646	For local CLI tool, use Bare:
	647	```bash
	648	#!/usr/bin/env bare
	649	// cli.js
	650	console.log('Hello from bare CLI')
	651	```
	652
	653	```bash
	654	chmod +x cli.js
	655	./cli.js
	656	```
	657
	658	---
	659
	660	## Performance Considerations
	661
	662	### Startup Time
	663
	664	Bare is slightly faster (no Pear wrapper):
	665	- Bare: ~100ms
	666	- Pear: ~150ms
	667	- Difference negligible for long-running services
	668
	669	### Memory Usage
	670
	671	Bare uses slightly less memory (no Pear overhead):
	672	- Bare: Base + app
	673	- Pear: Base + app + wrapper (~5-10MB)
	674	- Difference negligible for most apps
	675
	676	### Runtime Performance
	677
	678	No difference - Pear uses Bare internally:
	679	- Same JavaScript engine
	680	- Same module system
	681	- Same APIs (when using Bare modules)
	682
	683	Optimization tip: The runtime choice doesn't matter for performance. Optimize your app code instead.
	684
	685	---
	686
	687	## Migration Guide
	688
	689	### From Node.js to Bare
	690
	691	1. Update imports:
	692	```javascript
	693	// Before (Node)
	694	const fs = require('fs')
	695	const path = require('path')
	696
	697	// After (Bare) - use import maps
	698	// package.json:
	699	{
	700	"imports": {
	701	"fs": { "bare": "bare-fs", "default": "fs" },
	702	"path": { "bare": "bare-path", "default": "path" }
	703	}
	704	}
	705
	706	// Code stays the same
	707	const fs = require('fs') // Resolves to bare-fs
	708	```
	709
	710	2. Install Bare modules:
	711	```bash
	712	npm install bare-fs bare-path bare-events
	713	```
	714
	715	3. Test with Bare:
	716	```bash
	717	bare index.js
	718	```
	719
	720	### From Pear to Bare
	721
	722	If not using Pear APIs:
	723
	724	```bash
	725	# Just extract and run
	726	pear dump pear://key /opt/app
	727	cd /opt/app
	728	bare index.js
	729	```
	730
	731	If using Pear APIs:
	732
	733	```javascript
	734	// Before (Pear)
	735	const config = Pear.config.options
	736
	737	// After (Bare) - load config manually
	738	const config = require('./config.json')
	739	```
	740
	741	Remove Pear dependencies:
	742	```json
	743	{
	744	"dependencies": {
	745	"pear-interface": "^1.0.0" // Remove
	746	}
	747	}
	748	```
	749
	750	---
	751
	752	## See Also
	753
	754	- [pear-apps.md](./pear-apps.md) - Full Pear deployment guide
	755	- [systemd-services.md](./systemd-services.md) - Systemd service setup
	756	- [module-resolution.md](./module-resolution.md) - Module resolution issues
	757	- [capabilities.md](./capabilities.md) - Linux capabilities (when created)
	758	- [troubleshooting.md](./troubleshooting.md) - Troubleshooting index
	759
	760	---
	761
	762	## References
	763
	764	- [holepunchto/bare](https://github.com/holepunchto/bare) - Bare runtime
	765	- [holepunchto/pear](https://github.com/holepunchto/pear) - Pear (built on Bare)
	766	- Real example: hyper-fleeky/repos/hyperident-gateway
	767	- Bare modules: Search "bare-*" on npm

deployment/capabilities.md +734 -0

…	…	@@ -0,0 +1,734 @@
	1	# Linux Capabilities for P2P Applications
	2
	3	Validated against: hyper-fleeky/hyperident-gateway (VPN gateway with TUN devices)
	4
	5	## Overview
	6
	7	Linux capabilities allow granting specific privileges to processes without running as root. Essential for P2P applications that need privileged operations.
	8
	9	Common P2P use cases:
	10	- VPN gateways: Need CAP_NET_ADMIN (TUN/TAP devices)
	11	- DHT servers: Need CAP_NET_BIND_SERVICE (port 80/443)
	12	- Raw packet apps: Need CAP_NET_RAW (raw sockets)
	13	- System daemons: Need CAP_DAC_OVERRIDE (file access)
	14
	15	Key principle: Grant minimum necessary capabilities, never use root.
	16
	17	---
	18
	19	## Table of Contents
	20
	21	1. [Capabilities Basics](#capabilities-basics)
	22	2. [Common Capabilities for P2P Apps](#common-capabilities-for-p2p-apps)
	23	3. [Granting Capabilities](#granting-capabilities)
	24	4. [Systemd with Capabilities](#systemd-with-capabilities)
	25	5. [Troubleshooting](#troubleshooting)
	26	6. [Security Best Practices](#security-best-practices)
	27
	28	---
	29
	30	## Capabilities Basics
	31
	32	### What are Capabilities?
	33
	34	Traditional Unix: Either root (all privileges) or user (limited)
	35
	36	Linux capabilities: Fine-grained privileges
	37
	38	Example:
	39	```bash
	40	# ❌ Old way: run as root
	41	sudo ./vpn-gateway # Has ALL privileges
	42
	43	# ✅ New way: grant specific capability
	44	setcap cap_net_admin+ep ./vpn-gateway
	45	./vpn-gateway # Only has network admin privilege
	46	```
	47
	48	### Why Use Capabilities?
	49
	50	Security:
	51	- Least privilege principle
	52	- Limit damage if compromised
	53	- No full root access
	54
	55	For P2P apps:
	56	- Pear permission model breaks with root
	57	- Systemd services shouldn't run as root
	58	- Network operations often need privileges
	59
	60	### Capability Format
	61
	62	Format: `CAP_NAME+flags`
	63
	64	Flags:
	65	- `e` - Effective (capability is active)
	66	- `p` - Permitted (capability can be activated)
	67	- `i` - Inheritable (passed to child processes)
	68
	69	Common: `cap_net_admin+ep` (effective and permitted)
	70
	71	---
	72
	73	## Common Capabilities for P2P Apps
	74
	75	### CAP_NET_ADMIN
	76
	77	What it allows:
	78	- Create/configure TUN/TAP devices
	79	- Configure network interfaces
	80	- Modify routing tables
	81	- Set up iptables rules
	82
	83	P2P use cases:
	84	- VPN gateways (hyperident-gateway)
	85	- Network tunnels
	86	- Custom routing
	87
	88	Example operations:
	89	```javascript
	90	// Create TUN device (needs CAP_NET_ADMIN)
	91	const UTUN = require('utun')
	92	const tun = new UTUN()
	93	await tun.open() // Would fail without capability
	94	```
	95
	96	Grant:
	97	```bash
	98	sudo setcap cap_net_admin+ep /path/to/binary
	99	```
	100
	101	---
	102
	103	### CAP_NET_BIND_SERVICE
	104
	105	What it allows:
	106	- Bind to privileged ports (<1024)
	107	- HTTP (80), HTTPS (443), DNS (53)
	108
	109	P2P use cases:
	110	- DHT bootstrap nodes on port 80
	111	- HTTPS P2P gateways on port 443
	112	- Custom DNS resolvers
	113
	114	Example:
	115	```javascript
	116	// Bind to port 80 (needs CAP_NET_BIND_SERVICE)
	117	const server = http.createServer(handler)
	118	server.listen(80) // Would fail without capability
	119	```
	120
	121	Grant:
	122	```bash
	123	sudo setcap cap_net_bind_service+ep /path/to/binary
	124	```
	125
	126	---
	127
	128	### CAP_NET_RAW
	129
	130	What it allows:
	131	- Use RAW and PACKET sockets
	132	- Send/receive raw IP packets
	133	- Implement custom protocols
	134
	135	P2P use cases:
	136	- Custom transport protocols
	137	- Packet inspection
	138	- Network monitoring
	139
	140	Example:
	141	```javascript
	142	// Create raw socket (needs CAP_NET_RAW)
	143	const socket = require('net').Socket()
	144	socket.setRawMode(true) // Would fail without capability
	145	```
	146
	147	Grant:
	148	```bash
	149	sudo setcap cap_net_raw+ep /path/to/binary
	150	```
	151
	152	---
	153
	154	### CAP_DAC_OVERRIDE
	155
	156	What it allows:
	157	- Bypass file read/write/execute permissions
	158	- Access files owned by others
	159
	160	P2P use cases:
	161	- Access system-wide storage
	162	- Read protected config files
	163	- Write to system directories
	164
	165	Security warning: Powerful capability, use sparingly
	166
	167	Grant:
	168	```bash
	169	sudo setcap cap_dac_override+ep /path/to/binary
	170	```
	171
	172	---
	173
	174	### Multiple Capabilities
	175
	176	Grant multiple:
	177	```bash
	178	# Comma-separated
	179	sudo setcap cap_net_admin,cap_net_raw+ep /path/to/binary
	180
	181	# Or multiple setcap calls (last one wins)
	182	sudo setcap cap_net_admin+ep /path/to/binary
	183	sudo setcap cap_net_raw+ep /path/to/binary # Overwrites!
	184
	185	# Use comma-separated for multiple
	186	```
	187
	188	---
	189
	190	## Granting Capabilities
	191
	192	### To Bare/Pear Binary
	193
	194	Find binary:
	195	```bash
	196	# For Pear apps
	197	BARE=$(find ~/.config/pear -name "pear-runtime" \| head -1)
	198	echo "Found: $BARE"
	199
	200	# For standalone Bare
	201	BARE=$(find ~/.config/pear -name "bare" \| head -1)
	202	```
	203
	204	Grant capability:
	205	```bash
	206	sudo setcap cap_net_admin+ep "$BARE"
	207	```
	208
	209	Verify:
	210	```bash
	211	getcap "$BARE"
	212	# Output: /path/to/pear-runtime cap_net_admin=ep
	213	```
	214
	215	### To Custom Binary
	216
	217	If you have a native addon or compiled binary:
	218
	219	```bash
	220	sudo setcap cap_net_admin+ep /opt/your-app/bin/native-module.node
	221	```
	222
	223	### Remove Capabilities
	224
	225	```bash
	226	sudo setcap -r /path/to/binary
	227	```
	228
	229	---
	230
	231	## Systemd with Capabilities
	232
	233	### Service File Configuration
	234
	235	Basic pattern:
	236	```ini
	237	[Service]
	238	User=youruser # Regular user, not root
	239	AmbientCapabilities=CAP_NET_ADMIN
	240	CapabilityBoundingSet=CAP_NET_ADMIN
	241	ExecStart=/home/youruser/.config/pear/bin/pear run /opt/app
	242	```
	243
	244	AmbientCapabilities:
	245	- Capabilities granted to process
	246	- Inherited by child processes
	247	- Use uppercase: CAP_NET_ADMIN (not cap_net_admin)
	248
	249	CapabilityBoundingSet:
	250	- Maximum capabilities allowed
	251	- Security boundary
	252	- Should match or be superset of AmbientCapabilities
	253
	254	### Multiple Capabilities in Systemd
	255
	256	```ini
	257	[Service]
	258	AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW
	259	CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW
	260	```
	261
	262	Note: Space-separated in systemd, comma-separated in setcap
	263
	264	### Complete Example: VPN Gateway
	265
	266	Service file: `/etc/systemd/system/vpn-gateway.service`
	267
	268	```ini
	269	[Unit]
	270	Description=P2P VPN Gateway
	271	After=network.target
	272	Documentation=https://github.com/yourorg/vpn-gateway
	273
	274	[Service]
	275	Type=simple
	276	User=vpnuser
	277
	278	# Working directory
	279	WorkingDirectory=/opt/vpn-gateway
	280
	281	# Environment
	282	Environment="PATH=/home/vpnuser/.config/pear/bin:/usr/bin"
	283	Environment="NODE_ENV=production"
	284
	285	# Capabilities for TUN device
	286	AmbientCapabilities=CAP_NET_ADMIN
	287	CapabilityBoundingSet=CAP_NET_ADMIN
	288
	289	# Execute
	290	ExecStart=/home/vpnuser/.config/pear/bin/pear run /opt/vpn-gateway
	291
	292	# Restart policy
	293	Restart=always
	294	RestartSec=10
	295
	296	# Logging
	297	StandardOutput=journal
	298	StandardError=journal
	299
	300	[Install]
	301	WantedBy=multi-user.target
	302	```
	303
	304	Setup:
	305	```bash
	306	# 1. Find and grant capability to binary
	307	BARE=$(find /home/vpnuser/.config/pear -name "pear-runtime" \| head -1)
	308	sudo setcap cap_net_admin+ep "$BARE"
	309
	310	# 2. Install service
	311	sudo cp vpn-gateway.service /etc/systemd/system/
	312	sudo systemctl daemon-reload
	313
	314	# 3. Enable and start
	315	sudo systemctl enable vpn-gateway
	316	sudo systemctl start vpn-gateway
	317
	318	# 4. Verify
	319	sudo systemctl status vpn-gateway
	320	journalctl -u vpn-gateway -f
	321	```
	322
	323	Check capabilities while running:
	324	```bash
	325	# Get PID
	326	PID=$(pgrep -f "pear run /opt/vpn-gateway")
	327
	328	# Check capabilities
	329	grep Cap /proc/$PID/status
	330
	331	# Or use getpcaps
	332	getpcaps $PID
	333	```
	334
	335	---
	336
	337	## Troubleshooting
	338
	339	### "Operation not permitted" despite capability
	340
	341	Check if capability granted:
	342	```bash
	343	getcap /path/to/binary
	344	# Should show: cap_net_admin=ep (or whatever you need)
	345	```
	346
	347	If not shown:
	348	```bash
	349	sudo setcap cap_net_admin+ep /path/to/binary
	350	```
	351
	352	Check systemd service:
	353	```bash
	354	# Verify service has AmbientCapabilities
	355	systemctl show your-app \| grep Capabilities
	356	```
	357
	358	Common issue: Binary updated, capabilities lost
	359
	360	```bash
	361	# After npm install or pear update, re-grant
	362	BARE=$(find ~/.config/pear -name "pear-runtime" \| head -1)
	363	sudo setcap cap_net_admin+ep "$BARE"
	364	```
	365
	366	---
	367
	368	### Service fails with "Operation not permitted"
	369
	370	Check logs:
	371	```bash
	372	journalctl -u your-app -n 50 --no-pager
	373	```
	374
	375	Common causes:
	376
	377	\| Issue \| Check \| Fix \|
	378	\|-------\|-------\|-----\|
	379	\| No capability on binary \| `getcap /path/to/binary` \| `setcap cap_net_admin+ep` \|
	380	\| Systemd config missing \| `systemctl show \\| grep Capabilities` \| Add AmbientCapabilities \|
	381	\| Wrong binary path \| Binary changed after update \| Find new binary, re-grant \|
	382	\| Running as root \| `User=root` in service \| Use regular user \|
	383
	384	---
	385
	386	### Capability lost after update
	387
	388	Symptom: Worked before, fails after pear update or npm install
	389
	390	Cause: Capabilities are lost when binary is overwritten
	391
	392	Solution: Re-grant after updates
	393
	394	```bash
	395	# After any update
	396	BARE=$(find ~/.config/pear -name "pear-runtime" \| head -1)
	397	sudo setcap cap_net_admin+ep "$BARE"
	398
	399	# Restart service
	400	sudo systemctl restart your-app
	401	```
	402
	403	Automation (add to deployment script):
	404	```bash
	405	#!/bin/bash
	406	# deploy.sh
	407
	408	# Update app
	409	pear stage production
	410	pear dump pear://key /opt/app
	411
	412	# Re-grant capabilities
	413	BARE=$(find ~/.config/pear -name "pear-runtime" \| head -1)
	414	sudo setcap cap_net_admin+ep "$BARE"
	415
	416	# Restart
	417	sudo systemctl restart your-app
	418	```
	419
	420	---
	421
	422	### "Capability not permitted in bounding set"
	423
	424	Error in logs:
	425	```
	426	Failed to add capability cap_net_admin to bounding set: Operation not permitted
	427	```
	428
	429	Cause: CapabilityBoundingSet doesn't include the capability
	430
	431	Fix:
	432	```ini
	433	[Service]
	434	AmbientCapabilities=CAP_NET_ADMIN
	435	CapabilityBoundingSet=CAP_NET_ADMIN # Must include all in Ambient
	436	```
	437
	438	Rule: CapabilityBoundingSet must be superset of AmbientCapabilities
	439
	440	---
	441
	442	## Security Best Practices
	443
	444	### Principle of Least Privilege
	445
	446	✅ Do:
	447	```bash
	448	# Only grant what you need
	449	sudo setcap cap_net_admin+ep /path/to/binary
	450	```
	451
	452	❌ Don't:
	453	```bash
	454	# Don't grant unnecessary capabilities
	455	sudo setcap cap_net_admin,cap_dac_override,cap_sys_admin+ep /path/to/binary
	456	```
	457
	458	### Limit Capability Scope
	459
	460	✅ Do:
	461	```ini
	462	[Service]
	463	AmbientCapabilities=CAP_NET_ADMIN
	464	CapabilityBoundingSet=CAP_NET_ADMIN # Only what's needed
	465	```
	466
	467	❌ Don't:
	468	```ini
	469	[Service]
	470	# Don't leave bounding set open
	471	# (omitting CapabilityBoundingSet allows all capabilities)
	472	AmbientCapabilities=CAP_NET_ADMIN
	473	```
	474
	475	### Avoid Running as Root
	476
	477	✅ Do:
	478	```ini
	479	[Service]
	480	User=vpnuser
	481	AmbientCapabilities=CAP_NET_ADMIN
	482	```
	483
	484	❌ Don't:
	485	```ini
	486	[Service]
	487	User=root # ❌ Defeats purpose of capabilities
	488	```
	489
	490	Exception: None. Always use capabilities with regular user.
	491
	492	### Regular Audits
	493
	494	Check what capabilities are in use:
	495	```bash
	496	# Find all binaries with capabilities
	497	find / -type f -exec getcap {} \; 2>/dev/null \| grep cap_
	498
	499	# Check running processes
	500	for pid in /proc/[0-9]*; do
	501	grep -H Cap /proc/$(basename $pid)/status 2>/dev/null
	502	done
	503	```
	504
	505	### Document Requirements
	506
	507	In your README:
	508	```markdown
	509	## Capabilities Required
	510
	511	This app requires CAP_NET_ADMIN for TUN device creation.
	512
	513	Setup:
	514	\`\`\`bash
	515	BARE=$(find ~/.config/pear -name "pear-runtime" \| head -1)
	516	sudo setcap cap_net_admin+ep "$BARE"
	517	\`\`\`
	518
	519	Why needed: Creates virtual network interfaces for P2P VPN
	520	```
	521
	522	---
	523
	524	## Real-World Example: hyperident-gateway
	525
	526	Use case: P2P VPN gateway using TUN devices
	527
	528	Capability needed: CAP_NET_ADMIN
	529
	530	Setup:
	531
	532	```bash
	533	# 1. Deploy app
	534	pear dump pear://key /opt/hyperident-gateway
	535
	536	# 2. Find bare binary
	537	BARE=$(find ~/.config/pear -name "pear-runtime" \| head -1)
	538	echo "Using: $BARE"
	539
	540	# 3. Grant capability
	541	sudo setcap cap_net_admin+ep "$BARE"
	542
	543	# 4. Verify
	544	getcap "$BARE"
	545	# Output: ...cap_net_admin=ep
	546
	547	# 5. Test manually
	548	cd /opt/hyperident-gateway
	549	pear run .
	550	# Should create TUN device successfully
	551	```
	552
	553	Systemd service:
	554
	555	```ini
	556	[Unit]
	557	Description=Hyperident P2P VPN Gateway
	558	After=network.target
	559
	560	[Service]
	561	Type=simple
	562	User=fleeky
	563	WorkingDirectory=/opt/hyperident-gateway
	564	Environment="PATH=/home/fleeky/.config/pear/bin:/usr/bin"
	565
	566	# TUN device capability
	567	AmbientCapabilities=CAP_NET_ADMIN
	568	CapabilityBoundingSet=CAP_NET_ADMIN
	569
	570	ExecStart=/home/fleeky/.config/pear/bin/pear run /opt/hyperident-gateway
	571	Restart=always
	572
	573	[Install]
	574	WantedBy=multi-user.target
	575	```
	576
	577	Why this works:
	578	- Runs as regular user (Pear permissions work)
	579	- Has CAP_NET_ADMIN (can create TUN devices)
	580	- No root needed
	581	- Secure (least privilege)
	582
	583	Source: hyper-fleeky/repos/hyperident-gateway
	584
	585	---
	586
	587	## Capability Reference
	588
	589	### Network Capabilities
	590
	591	\| Capability \| Allows \| P2P Use Case \|
	592	\|------------\|--------\|--------------\|
	593	\| CAP_NET_ADMIN \| Network admin operations \| VPN, routing, TUN/TAP \|
	594	\| CAP_NET_BIND_SERVICE \| Bind to ports <1024 \| HTTP/HTTPS servers \|
	595	\| CAP_NET_RAW \| Raw sockets \| Custom protocols \|
	596	\| CAP_NET_BROADCAST \| Send broadcast packets \| Network discovery \|
	597
	598	### File Capabilities
	599
	600	\| Capability \| Allows \| Use Case \|
	601	\|------------\|--------\|----------\|
	602	\| CAP_DAC_OVERRIDE \| Bypass file permissions \| System-wide storage \|
	603	\| CAP_DAC_READ_SEARCH \| Bypass read permissions \| Config access \|
	604	\| CAP_FOWNER \| Bypass file owner checks \| Multi-user apps \|
	605
	606	### System Capabilities
	607
	608	\| Capability \| Allows \| Use Case \|
	609	\|------------\|--------\|----------\|
	610	\| CAP_SYS_ADMIN \| Various admin operations \| Advanced system management \|
	611	\| CAP_SYS_CHROOT \| Use chroot() \| Containerization \|
	612	\| CAP_SYS_NICE \| Change process priority \| Performance tuning \|
	613
	614	Full list: `man capabilities`
	615
	616	---
	617
	618	## Testing Capabilities
	619
	620	### Test Script
	621
	622	```bash
	623	#!/bin/bash
	624	# test-capability.sh
	625
	626	echo "Testing CAP_NET_ADMIN..."
	627
	628	# Find binary
	629	BARE=$(find ~/.config/pear -name "pear-runtime" \| head -1)
	630	echo "Binary: $BARE"
	631
	632	# Check current capabilities
	633	echo "Current capabilities:"
	634	getcap "$BARE"
	635
	636	# Test TUN device creation
	637	echo "Testing TUN device creation..."
	638	cd /opt/your-vpn-app
	639	pear run . 2>&1 \| grep -i "permission\\|error"
	640
	641	if [ $? -eq 0 ]; then
	642	echo "❌ Failed - likely missing capability"
	643	echo "Run: sudo setcap cap_net_admin+ep $BARE"
	644	else
	645	echo "✅ Success - capability working"
	646	fi
	647	```
	648
	649	### Verify in Code
	650
	651	```javascript
	652	// Check if we have CAP_NET_ADMIN
	653	const { exec } = require('child_process')
	654
	655	exec('capsh --print', (err, stdout) => {
	656	if (err) {
	657	console.error('Cannot check capabilities:', err)
	658	return
	659	}
	660
	661	if (stdout.includes('cap_net_admin')) {
	662	console.log('✅ CAP_NET_ADMIN available')
	663	} else {
	664	console.log('❌ CAP_NET_ADMIN not available')
	665	console.log('Run: sudo setcap cap_net_admin+ep $(which pear-runtime)')
	666	}
	667	})
	668	```
	669
	670	---
	671
	672	## Quick Reference
	673
	674	### Grant Capability
	675
	676	```bash
	677	# Find binary
	678	BARE=$(find ~/.config/pear -name "pear-runtime" \| head -1)
	679
	680	# Grant single capability
	681	sudo setcap cap_net_admin+ep "$BARE"
	682
	683	# Grant multiple capabilities
	684	sudo setcap cap_net_admin,cap_net_raw+ep "$BARE"
	685
	686	# Verify
	687	getcap "$BARE"
	688	```
	689
	690	### Systemd Service
	691
	692	```ini
	693	[Service]
	694	User=youruser
	695	AmbientCapabilities=CAP_NET_ADMIN
	696	CapabilityBoundingSet=CAP_NET_ADMIN
	697	ExecStart=/path/to/binary
	698	```
	699
	700	### Check Running Process
	701
	702	```bash
	703	# Get PID
	704	PID=$(pgrep -f "your-app")
	705
	706	# Check capabilities
	707	grep Cap /proc/$PID/status
	708	getpcaps $PID
	709	```
	710
	711	### Remove Capability
	712
	713	```bash
	714	sudo setcap -r /path/to/binary
	715	```
	716
	717	---
	718
	719	## See Also
	720
	721	- [systemd-services.md](./systemd-services.md) - Systemd service setup
	722	- [bare-runtime.md](./bare-runtime.md) - Bare runtime deployment
	723	- [troubleshooting.md](./troubleshooting.md#permission-denied-tuntap-devices) - Permission troubleshooting
	724	- `man capabilities` - Full capability reference
	725	- `man setcap` - Setcap command details
	726
	727	---
	728
	729	## References
	730
	731	- Linux capabilities: `man capabilities`
	732	- Systemd capabilities: `man systemd.exec`
	733	- Real example: hyper-fleeky/repos/hyperident-gateway
	734	- Security guide: https://man7.org/linux/man-pages/man7/capabilities.7.html

deployment/module-resolution.md +767 -0

…	…	@@ -0,0 +1,767 @@
	1	# Module Resolution Issues in Pear & Bare
	2
	3	Critical bugs documented: pear:// protocol MODULE_NOT_FOUND, ESM/CJS interop
	4	Validated against: hyper-fleeky/hyperident-gateway, holepunchto/pear
	5
	6	## Overview
	7
	8	This document catalogs module resolution issues in Pear and Bare runtimes, their symptoms, root causes, and workarounds.
	9
	10	Status: Multiple CRITICAL bugs as of 2026-01-17
	11
	12	---
	13
	14	## Table of Contents
	15
	16	1. [Critical Bug: pear:// Module Resolution](#critical-bug-pear-module-resolution)
	17	2. [ESM/CJS Interop with Hoisted Dependencies](#esmcjs-interop)
	18	3. [Dynamic Require Patterns](#dynamic-require-patterns)
	19	4. [Import Maps](#import-maps)
	20	5. [Workarounds Summary](#workarounds-summary)
	21	6. [Testing Module Resolution](#testing-module-resolution)
	22
	23	---
	24
	25	## Critical Bug: pear:// Module Resolution
	26
	27	### 🐛 Status: BLOCKING BUG
	28
	29	Discovered: 2026-01-17 during hyperident-gateway deployment
	30	Affects: All Pear terminal apps with npm dependencies
	31	Severity: CRITICAL - Blocks production deployment
	32	Issue: Module resolution completely fails under `pear://` protocol
	33
	34	### Symptoms
	35
	36	```bash
	37	$ pear run pear://hdj3chnrp8wnfzq4b3fd9i1jhuha1ufeyces35oofyteh6csfsmy
	38	MODULE_NOT_FOUND: Cannot find module 'utun' imported from 'pear://hdj3chnrp8wnfzq4b3fd9i1jhuha1ufeyces35oofyteh6csfsmy/lib/utun-manager.js
	39	```
	40
	41	But:
	42	- ✅ node_modules IS bundled (confirmed by `pear stage` showing 289 blocks, not 11)
	43	- ✅ Module exists in bundle (verified with `pear dump`)
	44	- ✅ Same app works from local directory
	45	- ❌ Fails only under `pear://` protocol
	46
	47	### Root Cause
	48
	49	Pear's module resolution system cannot find modules in bundled node_modules when running from `pear://` protocol.
	50
	51	Evidence:
	52	1. Staging includes node_modules (289 blocks vs 11 without deps)
	53	2. `pear dump pear://key /tmp/test && ls /tmp/test/node_modules` shows all modules present
	54	3. `pear run /tmp/test` works perfectly
	55	4. `pear run pear://key` fails with MODULE_NOT_FOUND
	56
	57	Hypothesis: The `pear://` protocol handler doesn't properly resolve module paths in bundled node_modules.
	58
	59	### Failed Attempts
	60
	61	All standard solutions attempted and failed:
	62
	63	#### ❌ Attempt 1: Add to dependencies
	64	```json
	65	{
	66	"dependencies": {
	67	"utun": "^1.0.0",
	68	"bare-subprocess": "^1.0.0" // Even explicit didn't help
	69	}
	70	}
	71	```
	72	Result: Still MODULE_NOT_FOUND under pear://
	73
	74	#### ❌ Attempt 2: Use pear.stage.includes
	75	```json
	76	{
	77	"pear": {
	78	"stage": {
	79	"includes": ["node_modules/utun", "node_modules/bare-subprocess"]
	80	}
	81	}
	82	}
	83	```
	84	Result: Still fails (includes is for warmup, not resolution)
	85
	86	#### ❌ Attempt 3: Remove node_modules from ignore
	87	```json
	88	{
	89	"pear": {
	90	"stage": {
	91	"ignore": [".git", "test"] // node_modules not ignored
	92	}
	93	}
	94	}
	95	```
	96	Result: Already not ignored, still fails
	97
	98	#### ❌ Attempt 4: Nested dependencies
	99	```bash
	100	# Manually nest dependencies
	101	mkdir -p node_modules/utun/node_modules
	102	cp -r node_modules/bare-subprocess node_modules/utun/node_modules/
	103	```
	104	Result: Still MODULE_NOT_FOUND
	105
	106	#### ❌ Attempt 5: Convert to CommonJS
	107	Remove `"type": "module"`, convert all imports to requires
	108	Result: Different error, now can't find 'utun' itself
	109
	110	### Workarounds
	111
	112	#### ✅ Workaround 1: Local Directory Deployment (Recommended)
	113
	114	Don't run from pear:// link:
	115	```bash
	116	pear run pear://key # ❌ Fails
	117	```
	118
	119	Do dump to local directory:
	120	```bash
	121	pear dump pear://key /opt/your-app
	122	pear run /opt/your-app # ✓ Works
	123	```
	124
	125	Why this works: Local directory bypasses pear:// protocol handler
	126
	127	Deployment pattern:
	128	```bash
	129	# On deployment server
	130	pear stage production
	131	pear release production
	132	pear dump pear://prod-key /opt/your-app
	133
	134	# In systemd service
	135	ExecStart=/home/user/.config/pear/bin/pear run /opt/your-app
	136	```
	137
	138	#### ✅ Workaround 2: Bare Runtime Direct
	139
	140	Skip Pear wrapper entirely:
	141	```bash
	142	pear dump pear://key /opt/your-app
	143	cd /opt/your-app
	144	bare index.js # ✓ Works
	145	```
	146
	147	Why this works: Bare's module resolution works, Pear's wrapper doesn't
	148
	149	See: [bare-runtime.md](./bare-runtime.md)
	150
	151	#### ✅ Workaround 3: Development with pear run .
	152
	153	For development:
	154	```bash
	155	# Don't use pear:// during dev
	156	pear run --dev . # ✓ Works
	157
	158	# Or run from local directory
	159	pear run . # ✓ Works
	160	```
	161
	162	### Validation
	163
	164	Test case (from hyperident-gateway):
	165
	166	```bash
	167	# Stage with dependencies
	168	npm install
	169	pear stage production
	170	# Output: Staged 289 blocks (node_modules included)
	171
	172	# Test 1: pear:// fails
	173	pear run pear://key
	174	# Result: ❌ MODULE_NOT_FOUND: Cannot find module 'utun'
	175
	176	# Test 2: Local directory works
	177	pear dump pear://key /tmp/test
	178	pear run /tmp/test
	179	# Result: ✓ Works perfectly
	180
	181	# Test 3: Bare runtime works
	182	cd /tmp/test
	183	bare gateway.js
	184	# Result: ✓ Works perfectly
	185	```
	186
	187	Conclusion: Module resolution is broken specifically under pear:// protocol.
	188
	189	### Impact
	190
	191	Blocks:
	192	- Production deployment of Pear terminal apps with dependencies
	193	- Running apps via pear:// links
	194	- Seeding and P2P distribution of complex apps
	195
	196	Workaround required for:
	197	- Any app with npm dependencies
	198	- Multi-module projects
	199	- Apps using holepunchto libraries (hypercore, hyperswarm, etc.)
	200
	201	### When to Report
	202
	203	This is a critical Pear bug that should be reported to holepunchto/pear:
	204
	205	Report includes:
	206	1. Minimal reproduction case
	207	2. Evidence node_modules is bundled (block count, pear dump)
	208	3. Works locally, fails under pear://
	209	4. Affects all apps with dependencies
	210	5. Time spent: ~5 hours debugging
	211
	212	Template issue: See hyper-fleeky/pear-deployment-lessons.md Issue 6
	213
	214	---
	215
	216	## ESM/CJS Interop
	217
	218	### 🐛 Status: Known Issue in Bare Runtime
	219
	220	Discovered: 2026-01-17 during ESM conversion attempt
	221	Affects: ESM modules importing CommonJS with dynamic requires
	222	Severity: HIGH - Blocks ESM usage
	223
	224	### Symptoms
	225
	226	```bash
	227	$ pear run .
	228	MODULE_NOT_FOUND: Cannot find module 'bare-subprocess' imported from
	229	'pear://key/node_modules/utun/index.js'
	230	```
	231
	232	When:
	233	- Main app uses `"type": "module"` (ESM)
	234	- Imports CommonJS dependency (e.g., `import UTUN from 'utun'`)
	235	- That dependency has dynamic require: `require(isBare && 'bare-subprocess' \|\| ...)`
	236	- bare-subprocess is hoisted to root node_modules by npm
	237
	238	### Root Cause
	239
	240	Bare's module resolution has an issue with ESM/CommonJS interop when:
	241	1. ESM code imports a CommonJS module
	242	2. That CommonJS module requires another CommonJS module
	243	3. The second module is hoisted to root node_modules
	244
	245	Module resolution path:
	246	```
	247	index.js (ESM)
	248	→ import 'utun' (CJS)
	249	→ require('bare-subprocess') (CJS, hoisted)
	250	→ ❌ Can't find (resolution fails in CJS from ESM context)
	251	```
	252
	253	### Test Case
	254
	255	Works (CommonJS throughout):
	256	```javascript
	257	// test-require.cjs
	258	const UTUN = require('utun')
	259	const subprocess = require('bare-subprocess')
	260	console.log('Both loaded')
	261	```
	262	```bash
	263	$ bare test-require.cjs
	264	✓ Both loaded
	265	```
	266
	267	Fails (ESM importing CJS):
	268	```javascript
	269	// index.js with "type": "module"
	270	import UTUN from 'utun'
	271	console.log('UTUN loaded')
	272	```
	273	```bash
	274	$ bare index.js
	275	❌ MODULE_NOT_FOUND: Cannot find module 'bare-subprocess'
	276	```
	277
	278	### Failed Attempts
	279
	280	#### ❌ Attempt 1: Add to dependencies
	281	Already done, doesn't help
	282
	283	#### ❌ Attempt 2: with.imports pragma
	284	```javascript
	285	const UTUN = require('utun', { with: { imports: './package.json' } })
	286	```
	287	With package.json:
	288	```json
	289	{
	290	"imports": {
	291	"bare-subprocess": "bare-subprocess"
	292	}
	293	}
	294	```
	295	Result: Still fails
	296
	297	#### ❌ Attempt 3: Nested installation
	298	```bash
	299	npm install --install-strategy=nested
	300	```
	301	Result: npm still hoists compatible versions
	302
	303	#### ❌ Attempt 4: Force shallow
	304	```bash
	305	# .npmrc
	306	install-strategy=shallow
	307	```
	308	Result: Still hoists
	309
	310	### Workarounds
	311
	312	#### ✅ Workaround 1: Convert to CommonJS (Recommended)
	313
	314	Remove ESM, use CommonJS throughout:
	315
	316	Before (ESM):
	317	```json
	318	{
	319	"type": "module"
	320	}
	321	```
	322	```javascript
	323	import UTUN from 'utun'
	324	import Hyperswarm from 'hyperswarm'
	325	```
	326
	327	After (CommonJS):
	328	```json
	329	{
	330	"type": "commonjs" // Or remove field
	331	}
	332	```
	333	```javascript
	334	const UTUN = require('utun')
	335	const Hyperswarm = require('hyperswarm')
	336	```
	337
	338	Result: ✓ Module resolution works
	339
	340	Drawback: Can't use ESM features (top-level await, etc.)
	341
	342	#### ✅ Workaround 2: Manual Dependency Nesting
	343
	344	Force bare-subprocess into utun's node_modules:
	345
	346	```bash
	347	# After npm install
	348	mkdir -p node_modules/utun/node_modules
	349	cp -r node_modules/bare-subprocess node_modules/utun/node_modules/
	350	```
	351
	352	Result: ✓ Works (utun finds bare-subprocess locally)
	353
	354	Drawback: Manual step, breaks on reinstall
	355
	356	#### ✅ Workaround 3: Use Dynamic Import
	357
	358	For critical ESM features:
	359
	360	```javascript
	361	// Main file as CommonJS
	362	const main = async () => {
	363	const { default: esmModule } = await import('./esm-module.js')
	364	const UTUN = require('utun') // CommonJS require
	365	}
	366	main()
	367	```
	368
	369	Result: ✓ Can mix ESM and CJS carefully
	370
	371	### When to Report
	372
	373	This is a Bare runtime bug that should be reported to holepunchto/bare:
	374
	375	Report includes:
	376	1. Minimal test case (ESM importing CJS with dynamic require)
	377	2. Proof it works in CommonJS
	378	3. Evidence of hoisted dependency
	379	4. Impact on modern JavaScript usage
	380
	381	---
	382
	383	## Dynamic Require Patterns
	384
	385	### Issue: Static Analysis Can't Detect
	386
	387	Some dependencies use dynamic requires that Pear's static analysis can't detect:
	388
	389	```javascript
	390	// In dependency code
	391	const module = require(condition && 'optional-module' \|\| 'fallback')
	392	const module = require(getModuleName())
	393	const module = require(`${prefix}-${name}`)
	394	```
	395
	396	### Symptoms
	397
	398	```bash
	399	Warning: MODULE_NOT_FOUND during warmup: 'optional-module'
	400	Hint: Add to pear.stage.defer or dependencies
	401	```
	402
	403	### Solutions
	404
	405	#### For Required Dependencies
	406
	407	Add explicitly to package.json:
	408
	409	```json
	410	{
	411	"dependencies": {
	412	"parent-module": "^1.0.0",
	413	"dynamically-required": "^1.0.0" // ← Explicit
	414	}
	415	}
	416	```
	417
	418	#### For Optional Dependencies
	419
	420	Add to pear.stage.defer:
	421
	422	```json
	423	{
	424	"pear": {
	425	"stage": {
	426	"defer": ["optional-module"]
	427	}
	428	}
	429	}
	430	```
	431
	432	#### For Platform-Specific
	433
	434	Use conditional:
	435
	436	```json
	437	{
	438	"dependencies": {
	439	"parent-module": "^1.0.0"
	440	},
	441	"optionalDependencies": {
	442	"fsevents": "^2.0.0" // macOS only
	443	},
	444	"pear": {
	445	"stage": {
	446	"defer": ["fsevents"]
	447	}
	448	}
	449	}
	450	```
	451
	452	### Real-World Examples
	453
	454	From utun:
	455	```javascript
	456	// utun/index.js
	457	const subprocess = require(isBare && 'bare-subprocess' \|\| 'child_process')
	458	```
	459
	460	Solution:
	461	```json
	462	{
	463	"dependencies": {
	464	"utun": "^1.0.0",
	465	"bare-subprocess": "^1.0.0" // Explicit for Bare
	466	}
	467	}
	468	```
	469
	470	From hypercore-crypto:
	471	```javascript
	472	// Tries sodium-native first, falls back to sodium-universal
	473	try {
	474	module.exports = require('sodium-native')
	475	} catch {
	476	module.exports = require('sodium-universal')
	477	}
	478	```
	479
	480	Solution:
	481	```json
	482	{
	483	"dependencies": {
	484	"hypercore-crypto": "^3.0.0",
	485	"sodium-universal": "^5.0.0" // Fallback
	486	},
	487	"optionalDependencies": {
	488	"sodium-native": "^4.0.0" // Preferred but optional
	489	}
	490	}
	491	```
	492
	493	---
	494
	495	## Import Maps
	496
	497	### Overview
	498
	499	Import maps allow runtime-specific module resolution:
	500
	501	```json
	502	{
	503	"imports": {
	504	"events": {
	505	"bare": "bare-events",
	506	"default": "events"
	507	}
	508	}
	509	}
	510	```
	511
	512	Used by: hypercore, hyperdrive, many tier 1/2 repos
	513
	514	Purpose: Use bare-events in Bare runtime, events in Node.js
	515
	516	### Common Patterns
	517
	518	#### Pattern 1: Core Modules
	519
	520	```json
	521	{
	522	"imports": {
	523	"fs": {
	524	"bare": "bare-fs",
	525	"default": "fs"
	526	},
	527	"path": {
	528	"bare": "bare-path",
	529	"default": "path"
	530	}
	531	}
	532	}
	533	```
	534
	535	#### Pattern 2: Conditional Features
	536
	537	```json
	538	{
	539	"imports": {
	540	"crypto": {
	541	"bare": "bare-crypto",
	542	"node": "crypto",
	543	"default": null
	544	}
	545	}
	546	}
	547	```
	548
	549	### Limitations
	550
	551	Import maps only work at the module level where they're defined. They do not cascade down dependencies.
	552
	553	Example:
	554	```
	555	your-app (has import map)
	556	→ utun (no import map)
	557	→ bare-subprocess (❌ Can't resolve via your import map)
	558	```
	559
	560	Solution: Dependencies must define their own import maps, or you must add to dependencies.
	561
	562	### Debugging Import Maps
	563
	564	```bash
	565	# Check what imports are defined
	566	jq '.imports' package.json
	567
	568	# Test resolution
	569	bare -e "console.log(require.resolve('events'))"
	570	```
	571
	572	---
	573
	574	## Workarounds Summary
	575
	576	### For pear:// MODULE_NOT_FOUND
	577
	578	\| Workaround \| Pros \| Cons \| Recommendation \|
	579	\|------------\|------\|------\|----------------\|
	580	\| Local directory \| ✅ Works \| Manual dump step \| ⭐ Best for production \|
	581	\| Bare runtime \| ✅ Works, full control \| No Pear features \| ⭐ Best for services \|
	582	\| Development mode \| ✅ Simple \| Only for dev \| Dev only \|
	583
	584	Production deployment:
	585	```bash
	586	pear dump pear://key /opt/app
	587	pear run /opt/app
	588	```
	589
	590	### For ESM/CJS Interop
	591
	592	\| Workaround \| Pros \| Cons \| Recommendation \|
	593	\|------------\|------\|------\|----------------\|
	594	\| CommonJS \| ✅ Works reliably \| No ESM features \| ⭐ Recommended \|
	595	\| Manual nesting \| ✅ Keeps ESM \| Fragile \| Last resort \|
	596	\| Dynamic import \| ✅ Selective ESM \| Complex \| Advanced cases \|
	597
	598	Production deployment:
	599	```json
	600	{
	601	"type": "commonjs"
	602	}
	603	```
	604
	605	### For Dynamic Requires
	606
	607	\| Approach \| When to Use \|
	608	\|----------\|-------------\|
	609	\| Add to dependencies \| Required dependencies \|
	610	\| Add to optionalDependencies \| Platform-specific \|
	611	\| Add to pear.stage.defer \| Try/catch wrapped \|
	612
	613	---
	614
	615	## Testing Module Resolution
	616
	617	### Test 1: Verify Bundling
	618
	619	```bash
	620	pear stage production
	621	# Check block count
	622	# Low count (11) = no deps ❌
	623	# High count (200+) = deps included ✓
	624	```
	625
	626	### Test 2: Inspect Bundle
	627
	628	```bash
	629	pear dump pear://key /tmp/test
	630	ls -la /tmp/test/node_modules
	631	# Should see all dependencies
	632	```
	633
	634	### Test 3: Local vs pear://
	635
	636	```bash
	637	# Test local
	638	pear run /tmp/test # Should work
	639
	640	# Test pear://
	641	pear run pear://key # May fail with MODULE_NOT_FOUND
	642	```
	643
	644	### Test 4: ESM vs CJS
	645
	646	```bash
	647	# Test require
	648	bare -e "require('utun'); console.log('OK')"
	649
	650	# Test import
	651	bare -e "import('utun').then(() => console.log('OK'))"
	652	```
	653
	654	### Test 5: Dependencies Present
	655
	656	```bash
	657	cd your-app
	658	node -e "require.resolve('dependency-name')"
	659	# Should print path, not throw
	660	```
	661
	662	---
	663
	664	## Debugging Checklist
	665
	666	When seeing MODULE_NOT_FOUND:
	667
	668	1. [ ] Is node_modules in pear.stage.ignore? (Should NOT be)
	669	2. [ ] Is staging showing high block count? (200+ = deps included)
	670	3. [ ] Are you running from pear:// or local directory?
	671	4. [ ] Is it ESM or CommonJS? (CommonJS more reliable)
	672	5. [ ] Is dependency in package.json dependencies?
	673	6. [ ] Is it a dynamic require? (Add explicitly)
	674	7. [ ] Did you test with pear dump + local run?
	675	8. [ ] Does it work with bare runtime directly?
	676
	677	---
	678
	679	## Known Working Configurations
	680
	681	### Configuration 1: CommonJS + Local Directory
	682
	683	```json
	684	{
	685	"name": "your-app",
	686	"main": "index.js",
	687	"pear": {
	688	"type": "terminal",
	689	"stage": {
	690	"ignore": [".git", "test"]
	691	}
	692	},
	693	"dependencies": {
	694	"hypercore": "^11.0.0",
	695	"hyperswarm": "^4.0.0"
	696	}
	697	}
	698	```
	699
	700	Deployment:
	701	```bash
	702	npm install
	703	pear stage production
	704	pear dump pear://key /opt/app
	705	pear run /opt/app # ✓ Works
	706	```
	707
	708	### Configuration 2: CommonJS + Bare Direct
	709
	710	```json
	711	{
	712	"name": "your-app",
	713	"main": "index.js",
	714	"dependencies": {
	715	"hypercore": "^11.0.0"
	716	}
	717	}
	718	```
	719
	720	Deployment:
	721	```bash
	722	npm install
	723	pear dump pear://key /opt/app
	724	cd /opt/app
	725	bare index.js # ✓ Works
	726	```
	727
	728	---
	729
	730	## See Also
	731
	732	- [pear-apps.md](./pear-apps.md) - General Pear deployment
	733	- [systemd-services.md](./systemd-services.md) - Service deployment (uses local directory workaround)
	734	- [bare-runtime.md](./bare-runtime.md) - Bare runtime deployment
	735	- [troubleshooting.md](./troubleshooting.md) - Symptom-based lookup
	736
	737	---
	738
	739	## References
	740
	741	### Real-World Issues
	742	- hyper-fleeky/pear-deployment-lessons.md Issue 6: Complete chronology
	743	- hyperident-gateway: Production deployment using workarounds
	744
	745	### Official Documentation
	746	- [Pear Configuration](https://github.com/holepunchto/pear-docs/blob/main/reference/configuration.md)
	747	- [Bare Module System](https://github.com/holepunchto/bare)
	748
	749	### Test Cases
	750	- holepunchto/pear/test/fixtures/* - Official test apps
	751	- Test: ESM vs CJS module resolution
	752	- Test: Dynamic requires with static analysis
	753
	754	---
	755
	756	## Bug Reporting
	757
	758	If encountering similar issues:
	759
	760	1. Document symptoms - Exact error message
	761	2. Check staging - Block count, node_modules present?
	762	3. Test locally - Does pear run . work?
	763	4. Test dump - Does dumped version work?
	764	5. Minimal reproduction - Smallest app that reproduces
	765	6. Report to holepunchto - With evidence above
	766
	767	Time investment: Document 30-60 min debugging before workaround, helps others avoid same time sink.

deployment/pear-apps.md +639 -0

…	…	@@ -0,0 +1,639 @@
	1	# Deploying Pear Applications
	2
	3	Validated against: holepunchto/pear examples, hyper-fleeky/hyperident-gateway
	4
	5	## Overview
	6
	7	Pear applications can be deployed as terminal apps or desktop apps. This guide covers production deployment patterns, common mistakes, and workarounds for known issues.
	8
	9	Official Docs: [docs.pears.com](https://docs.pears.com)
	10	Real-world validation: hyperident-gateway (P2P VPN terminal app)
	11
	12	---
	13
	14	## Table of Contents
	15
	16	1. [Basic Deployment Workflow](#basic-deployment-workflow)
	17	2. [Critical Configuration](#critical-configuration)
	18	3. [Common Mistakes](#common-mistakes)
	19	4. [Known Bugs & Workarounds](#known-bugs--workarounds)
	20	5. [Cache Management](#cache-management)
	21	6. [Testing Deployments](#testing-deployments)
	22	7. [Production Checklist](#production-checklist)
	23
	24	---
	25
	26	## Basic Deployment Workflow
	27
	28	### Official Workflow (from pear-docs)
	29
	30	```bash
	31	# 1. Stage your application
	32	cd your-app/
	33	pear stage production
	34
	35	# 2. Mark as release
	36	pear release production
	37
	38	# 3. Seed to network
	39	pear seed production
	40
	41	# 4. Run from link
	42	pear run pear://your-key
	43	```
	44
	45	See: [holepunchto/pear-docs/guide/releasing-a-pear-app.md](https://github.com/holepunchto/pear-docs/blob/main/guide/releasing-a-pear-app.md)
	46
	47	### Development vs Production Channels
	48
	49	Use separate channels for development and production:
	50
	51	```bash
	52	# Development (unreleased, always latest)
	53	pear stage dev
	54	pear run pear://dev-key # Latest staged changes
	55
	56	# Production (released, stable)
	57	pear stage production
	58	pear release production
	59	pear run pear://prod-key # Latest marked release
	60	```
	61
	62	### Testing Staged Changes
	63
	64	Test staged changes before releasing:
	65
	66	```bash
	67	# Stage without releasing
	68	pear stage production
	69
	70	# Test with --checkout flag
	71	pear run --checkout=staged pear://your-key
	72
	73	# If good, release it
	74	pear release production
	75	```
	76
	77	---
	78
	79	## Critical Configuration
	80
	81	### package.json Structure
	82
	83	Minimum required:
	84
	85	```json
	86	{
	87	"name": "your-app-name",
	88	"main": "index.js",
	89	"pear": {
	90	"name": "Your App",
	91	"type": "terminal"
	92	}
	93	}
	94	```
	95
	96	### ❌ CRITICAL: Never Ignore node_modules
	97
	98	Official docs say: Use `pear.stage.ignore` to exclude files
	99	Reality: ❌ NEVER ignore `node_modules`
	100
	101	```json
	102	{
	103	"pear": {
	104	"stage": {
	105	"ignore": ["node_modules"] // ❌ DO NOT DO THIS
	106	}
	107	}
	108	}
	109	```
	110
	111	Why this breaks:
	112	- Pear apps download as complete packages
	113	- Dependencies must be bundled
	114	- Without node_modules, app has no dependencies
	115	- Results in MODULE_NOT_FOUND errors
	116
	117	Correct configuration:
	118
	119	```json
	120	{
	121	"pear": {
	122	"stage": {
	123	"ignore": [".git", "test", ".github", "docs"] // ✓ Ignore these, keep node_modules
	124	}
	125	}
	126	}
	127	```
	128
	129	Before staging, prune dev dependencies to reduce bundle size:
	130
	131	```bash
	132	npm prune --omit=dev
	133	pear stage production
	134	```
	135
	136	Validation: Check staging output shows node_modules included:
	137	```bash
	138	pear stage production
	139	# Look for: "282 blocks" not "11 blocks"
	140	# More blocks = node_modules included = correct
	141	```
	142
	143	Source: Real-world issue from hyperident-gateway deployment
	144
	145	### Dynamic Requires Must Be Explicit
	146
	147	Problem: Some dependencies use dynamic requires:
	148
	149	```javascript
	150	// In dependency code
	151	const module = require(isBare && 'bare-subprocess' \|\| 'child_process')
	152	```
	153
	154	Pear's static analysis can't detect these.
	155
	156	Solution: Add to dependencies explicitly:
	157
	158	```json
	159	{
	160	"dependencies": {
	161	"utun": "^1.0.0",
	162	"bare-subprocess": "^1.0.0" // ← Explicit for dynamic require
	163	}
	164	}
	165	```
	166
	167	How to identify: Run `pear stage` and watch for warnings:
	168	```
	169	Warning: MODULE_NOT_FOUND during warmup: 'bare-subprocess'
	170	Add to pear.stage.defer or dependencies
	171	```
	172
	173	Source: Real-world issue with utun dependency in hyperident-gateway
	174
	175	### Using pear.stage.defer
	176
	177	For truly optional dependencies that may not exist:
	178
	179	```json
	180	{
	181	"pear": {
	182	"stage": {
	183	"defer": [
	184	"fsevents", // macOS only
	185	"optional-dep"
	186	]
	187	}
	188	}
	189	}
	190	```
	191
	192	When to use:
	193	- Dependency wrapped in `try { require('x') } catch { fallback() }`
	194	- Platform-specific optional deps
	195	- Peer dependencies that may not be installed
	196
	197	When NOT to use:
	198	- Dependencies you actually need (add to package.json dependencies)
	199	- Transitive dependencies of required packages (add explicitly)
	200
	201	Official docs: [pear-docs/reference/configuration.md](https://github.com/holepunchto/pear-docs/blob/main/reference/configuration.md#pear-stage-defer)
	202
	203	---
	204
	205	## Common Mistakes
	206
	207	### ❌ Mistake 1: Ignoring node_modules
	208
	209	Symptom: MODULE_NOT_FOUND errors on deployment
	210	Cause: `"ignore": ["node_modules"]` in pear.stage
	211	Fix: Remove node_modules from ignore list
	212	Severity: ⚠️ CRITICAL - App won't run
	213
	214	### ❌ Mistake 2: Not Adding Dynamic Requires
	215
	216	Symptom: MODULE_NOT_FOUND for transitive dependencies
	217	Cause: Dependency uses `require(condition && 'module')`
	218	Fix: Add to package.json dependencies explicitly
	219	Severity: ⚠️ HIGH - App breaks at runtime
	220
	221	### ❌ Mistake 3: Testing Only Locally
	222
	223	Symptom: Works locally, fails on remote machines
	224	Cause: Local Pear cache has old/different version
	225	Fix: Test on clean machine or clear cache
	226	Severity: ⚠️ MEDIUM - May not catch real issues
	227
	228	### ❌ Mistake 4: Not Seeding After Release
	229
	230	Symptom: Peers can't download new version
	231	Cause: No active seeders
	232	Fix: Run `pear seed production` and keep running
	233	Severity: ⚠️ HIGH - Updates don't propagate
	234
	235	### ❌ Mistake 5: Using ESM with Dynamic Requires
	236
	237	Symptom: MODULE_NOT_FOUND even with deps added
	238	Cause: Bare's ESM/CJS interop bug with hoisted dependencies
	239	Fix: Convert to CommonJS or use workarounds
	240	Severity: 🐛 CRITICAL BUG - See module-resolution.md
	241
	242	---
	243
	244	## Known Bugs & Workarounds
	245
	246	### 🐛 Critical Bug: Module Resolution Under pear://
	247
	248	Status: BLOCKING BUG (as of 2026-01-17)
	249	Symptom: MODULE_NOT_FOUND even though node_modules is bundled and visible
	250	Affects: All Pear apps using npm dependencies
	251
	252	Details:
	253	- Running `pear run pear://key` fails with MODULE_NOT_FOUND
	254	- node_modules IS included in bundle (confirmed by block count)
	255	- Same app works when run from local directory
	256	- Module resolution under `pear://` protocol is broken
	257
	258	Workaround: Use local directory deployment
	259
	260	```bash
	261	# DON'T use pear:// link directly
	262	pear run pear://key # ❌ MODULE_NOT_FOUND
	263
	264	# DO dump to local directory first
	265	pear dump pear://key /opt/myapp
	266	pear run /opt/myapp # ✓ Works
	267	```
	268
	269	Alternative: Run with Bare directly
	270
	271	```bash
	272	# Dump the app
	273	pear dump pear://key /opt/myapp
	274
	275	# Run with bare runtime
	276	cd /opt/myapp
	277	bare index.js # ✓ Works
	278	```
	279
	280	Reported: Issue logged in hyper-fleeky/pear-deployment-lessons.md Issue 6
	281	Impact: CRITICAL - Blocks all Pear terminal app deployments with dependencies
	282	See: [module-resolution.md](./module-resolution.md) for full details
	283
	284	### 🐛 ESM/CommonJS Interop Issues
	285
	286	Symptom: MODULE_NOT_FOUND for hoisted dependencies in ESM code
	287	Workaround: Convert to CommonJS (remove `"type": "module"`)
	288	See: [module-resolution.md](./module-resolution.md#esm-cjs-interop)
	289
	290	---
	291
	292	## Cache Management
	293
	294	### Why Cache Issues Happen
	295
	296	Pear caches apps aggressively:
	297	- `~/.config/pear/app-storage/` - Downloaded app data
	298	- `~/.config/pear/corestores/app/` - Hypercore storage
	299	- Cache persists across releases
	300
	301	### Symptom: Old Version Running
	302
	303	You've released a new version but remote machines still run old version.
	304
	305	Causes:
	306	1. Pear cache has old version
	307	2. Seed hasn't run long enough for propagation
	308	3. No active seeders
	309
	310	Solutions:
	311
	312	Option 1: Force specific version
	313
	314	```bash
	315	# Test latest staged changes
	316	pear run --checkout=staged pear://key
	317
	318	# Force latest release
	319	pear run --checkout=released pear://key
	320
	321	# Or specific length
	322	pear run --checkout=1234 pear://key
	323	```
	324
	325	Option 2: Clear cache on client
	326
	327	```bash
	328	# Clear app storage
	329	rm -rf ~/.config/pear/app-storage/*
	330
	331	# Clear corestores
	332	rm -rf ~/.config/pear/corestores/app/*
	333
	334	# Then run again
	335	pear run pear://key # Downloads fresh
	336	```
	337
	338	Option 3: Wait for peer propagation
	339
	340	After release, keep seeding:
	341
	342	```bash
	343	# Seed in background
	344	pear seed production &
	345
	346	# Or in screen/tmux
	347	screen -S pear-seed
	348	pear seed production
	349	# Ctrl+A, D to detach
	350	```
	351
	352	How long to seed?
	353	- At least until first peer downloads
	354	- Ideally until other peers start reseeding
	355	- For production: Run seeder as systemd service
	356
	357	### Pear Installation Corruption
	358
	359	Symptom: `pear info` hangs indefinitely, or `pear run` exits silently with code 0
	360
	361	Cause: Incomplete Pear reinstallation, stale lock files
	362
	363	Solution: Complete clean reinstall
	364
	365	```bash
	366	# Nuclear option: complete reinstall
	367	rm -rf ~/.config/pear
	368	npx pear
	369
	370	# Follow PATH setup instructions
	371	```
	372
	373	When this happens:
	374	- After deleting ~/.config/pear/by-dkey/*
	375	- After database lock errors
	376	- After incomplete pear updates
	377
	378	Source: Real-world issue from hyperident-gateway troubleshooting
	379
	380	---
	381
	382	## Testing Deployments
	383
	384	### Local Testing
	385
	386	```bash
	387	# Test in development mode
	388	pear run --dev .
	389
	390	# Test production staging locally
	391	pear stage dev
	392	pear run pear://dev-key
	393	```
	394
	395	### Remote Testing Checklist
	396
	397	Before releasing to production:
	398
	399	1. Test on clean machine
	400	```bash
	401	# On fresh VM or after cache clear
	402	rm -rf ~/.config/pear/app-storage/*
	403	pear run pear://your-key
	404	```
	405
	406	2. Test --checkout=staged first
	407	```bash
	408	pear stage production
	409	pear run --checkout=staged pear://prod-key
	410	# If works, then release
	411	pear release production
	412	```
	413
	414	3. Verify dependencies load
	415	```bash
	416	# Check all requires work
	417	pear run pear://key 2>&1 \| grep MODULE_NOT_FOUND
	418	# Should be empty
	419	```
	420
	421	4. Test update propagation
	422	```bash
	423	# Release new version
	424	pear release production
	425
	426	# On remote machine
	427	pear run pear://key --checkout=latest
	428	```
	429
	430	### Validation Against holepunchto Examples
	431
	432	Check holepunchto/pear test fixtures:
	433
	434	```bash
	435	cd holepunchto/repos/tier3/pear/test/fixtures/
	436
	437	# Look at working examples
	438	ls -la hello-world/
	439	ls -la minimal/
	440	ls -la stage-app-min/
	441
	442	# Compare package.json patterns
	443	cat hello-world/package.json
	444	```
	445
	446	---
	447
	448	## Production Checklist
	449
	450	Before deploying to production:
	451
	452	### Pre-Release
	453
	454	- [ ] `npm prune --omit=dev` to remove dev dependencies
	455	- [ ] All transitive dynamic requires added to dependencies
	456	- [ ] No `node_modules` in `pear.stage.ignore`
	457	- [ ] Tested on clean machine or with cleared cache
	458	- [ ] Tested with `--checkout=staged` first
	459
	460	### Release
	461
	462	- [ ] `pear stage production` completes successfully
	463	- [ ] High block count (not 11 blocks - that means no deps)
	464	- [ ] `pear release production` marks release
	465	- [ ] `pear seed production` started and running
	466
	467	### Post-Release
	468
	469	- [ ] Tested from remote machine with `pear run pear://key`
	470	- [ ] If MODULE_NOT_FOUND, use `pear dump` workaround
	471	- [ ] Seeder running continuously (systemd service recommended)
	472	- [ ] Monitor for peer connections
	473
	474	### For Systemd Deployment
	475
	476	- [ ] See [systemd-services.md](./systemd-services.md)
	477	- [ ] Use capabilities, not root
	478	- [ ] Set correct PATH in service file
	479	- [ ] Test service start/stop
	480	- [ ] Check logs with `journalctl -u your-app -f`
	481
	482	---
	483
	484	## Deployment Strategies
	485
	486	### Strategy 1: Direct pear:// Links (Recommended for Simple Apps)
	487
	488	When it works:
	489	- Simple apps without complex dependencies
	490	- Apps without dynamic requires
	491	- ⚠️ Currently broken due to module resolution bug
	492
	493	```bash
	494	pear stage production
	495	pear release production
	496	pear seed production
	497	# Users run: pear run pear://key
	498	```
	499
	500	### Strategy 2: Local Directory (Recommended for Complex Apps)
	501
	502	Best for:
	503	- Apps with many dependencies
	504	- Working around module resolution bugs
	505	- Systemd services
	506
	507	```bash
	508	# On deployment machine
	509	pear dump pear://key /opt/your-app
	510
	511	# Run from directory
	512	pear run /opt/your-app
	513	# Or: bare /opt/your-app/index.js
	514	```
	515
	516	### Strategy 3: Bare Runtime (For Services)
	517
	518	Best for:
	519	- Systemd services
	520	- Headless servers
	521	- Maximum control
	522
	523	```bash
	524	# Dump and run with bare
	525	pear dump pear://key /opt/your-app
	526	cd /opt/your-app
	527	bare index.js
	528	```
	529
	530	See: [bare-runtime.md](./bare-runtime.md) for Bare deployment details
	531
	532	---
	533
	534	## Examples
	535
	536	### Example: Minimal Terminal App
	537
	538	From holepunchto/pear/test/fixtures/minimal:
	539
	540	```json
	541	{
	542	"name": "minimal",
	543	"pear": {
	544	"type": "terminal"
	545	}
	546	}
	547	```
	548
	549	```javascript
	550	// index.js
	551	console.log('Hello from Pear!')
	552	```
	553
	554	```bash
	555	pear stage dev
	556	pear run pear://key
	557	```
	558
	559	### Example: App with Dependencies
	560
	561	From hyper-fleeky/hyperident-gateway:
	562
	563	```json
	564	{
	565	"name": "hyperident-gateway",
	566	"main": "gateway.js",
	567	"type": "commonjs",
	568	"pear": {
	569	"name": "Hyperident Gateway",
	570	"type": "terminal",
	571	"stage": {
	572	"ignore": [".git", "test", "docs"]
	573	}
	574	},
	575	"dependencies": {
	576	"hyperswarm": "^4.0.0",
	577	"hypercore-crypto": "^3.0.0",
	578	"utun": "^1.0.0",
	579	"bare-subprocess": "^1.0.0" // Explicit for utun's dynamic require
	580	}
	581	}
	582	```
	583
	584	```bash
	585	# Deployment
	586	npm install
	587	npm prune --omit=dev
	588	pear stage production
	589	pear release production
	590
	591	# Run (with workaround)
	592	pear dump pear://key /opt/hyperident-gateway
	593	pear run /opt/hyperident-gateway
	594	```
	595
	596	---
	597
	598	## Troubleshooting Quick Reference
	599
	600	\| Problem \| Quick Fix \|
	601	\|---------\|-----------\|
	602	\| MODULE_NOT_FOUND \| Check `node_modules` not in ignore list \|
	603	\| MODULE_NOT_FOUND (specific module) \| Add to package.json dependencies \|
	604	\| Old version running \| Clear cache: `rm -rf ~/.config/pear/app-storage/*` \|
	605	\| pear://key MODULE_NOT_FOUND \| Use `pear dump` workaround \|
	606	\| Pear hangs \| Clean reinstall: `rm -rf ~/.config/pear && npx pear` \|
	607	\| Can't seed \| Check firewall, try `--bootstrap` flag \|
	608
	609	See [troubleshooting.md](./troubleshooting.md) for comprehensive guide.
	610
	611	---
	612
	613	## References
	614
	615	### Official Documentation
	616	- [Pear Docs](https://docs.pears.com)
	617	- [pear-docs: Configuration](https://github.com/holepunchto/pear-docs/blob/main/reference/configuration.md)
	618	- [pear-docs: Releasing Apps](https://github.com/holepunchto/pear-docs/blob/main/guide/releasing-a-pear-app.md)
	619	- [holepunchto/pear repository](https://github.com/holepunchto/pear)
	620
	621	### Real-World Examples
	622	- hyper-fleeky/hyperident - Pear terminal app (profile management)
	623	- hyper-fleeky/hyperident-gateway - Complex deployment with systemd
	624	- holepunchto/pear/test/fixtures/* - Official test examples
	625
	626	### Related Documentation
	627	- [module-resolution.md](./module-resolution.md) - Deep dive on MODULE_NOT_FOUND
	628	- [systemd-services.md](./systemd-services.md) - Running as systemd service
	629	- [capabilities.md](./capabilities.md) - Linux capabilities for privileged ops
	630	- [troubleshooting.md](./troubleshooting.md) - Full troubleshooting guide
	631
	632	---
	633
	634	## Lessons Learned Source
	635
	636	This document integrates production deployment lessons from:
	637	- hyper-fleeky/pear-deployment-lessons.md (Issues 1-6)
	638	- Real deployments of hyperident-gateway (P2P VPN)
	639	- Validated against holepunchto/pear official examples

deployment/systemd-services.md +558 -0

…	…	@@ -0,0 +1,558 @@
	1	# Running P2P Apps as Systemd Services
	2
	3	Validated against: hyper-fleeky/hyperident-gateway (P2P VPN)
	4
	5	## Overview
	6
	7	Running P2P applications as systemd services requires special considerations for Pear's permission model and privileged operations like TUN device creation.
	8
	9	Key principle: Use capabilities, not root
	10
	11	---
	12
	13	## Table of Contents
	14
	15	1. [Basic Service Setup](#basic-service-setup)
	16	2. [Capabilities vs Root](#capabilities-vs-root)
	17	3. [Common Service Patterns](#common-service-patterns)
	18	4. [Troubleshooting](#troubleshooting)
	19	5. [Real-World Examples](#real-world-examples)
	20
	21	---
	22
	23	## Basic Service Setup
	24
	25	### Standard P2P App (No Special Privileges)
	26
	27	For apps that don't need special privileges (most P2P apps):
	28
	29	Service file: `/etc/systemd/system/your-app.service`
	30
	31	```ini
	32	[Unit]
	33	Description=Your P2P Application
	34	After=network.target
	35
	36	[Service]
	37	Type=simple
	38	User=youruser
	39	WorkingDirectory=/home/youruser/your-app
	40	Environment="PATH=/home/youruser/.config/pear/bin:/usr/local/bin:/usr/bin:/bin"
	41	ExecStart=/home/youruser/.config/pear/bin/pear run /home/youruser/your-app
	42	Restart=always
	43	RestartSec=10
	44
	45	[Install]
	46	WantedBy=multi-user.target
	47	```
	48
	49	Setup:
	50
	51	```bash
	52	# Create service file
	53	sudo nano /etc/systemd/system/your-app.service
	54
	55	# Reload systemd
	56	sudo systemctl daemon-reload
	57
	58	# Enable (start on boot)
	59	sudo systemctl enable your-app
	60
	61	# Start now
	62	sudo systemctl start your-app
	63
	64	# Check status
	65	sudo systemctl status your-app
	66
	67	# View logs
	68	journalctl -u your-app -f
	69	```
	70
	71	### Important Notes
	72
	73	1. Use regular user, not root
	74	2. Set full PATH - Pear may not be in system PATH
	75	3. Use local directory, not `pear://` link (module resolution bug workaround)
	76	4. WorkingDirectory matters for relative paths
	77
	78	---
	79
	80	## Capabilities vs Root
	81
	82	### ❌ Don't Run as Root
	83
	84	Problem with running as root:
	85
	86	```ini
	87	[Service]
	88	User=root # ❌ DON'T DO THIS
	89	ExecStart=/home/user/.config/pear/bin/pear run pear://key
	90	```
	91
	92	Why this fails:
	93	1. Root user has separate `~/.config/pear` from regular user
	94	2. Pear requires first-time approval of app keys
	95	3. No interactive approval possible in systemd context
	96	4. Results in `ERR_PERMISSION_REQUIRED`
	97
	98	Source: Real-world Issue 5 from pear-deployment-lessons.md
	99
	100	### ✅ Use Capabilities Instead
	101
	102	For apps needing privileged operations (like creating TUN devices):
	103
	104	Service file: `/etc/systemd/system/vpn-gateway.service`
	105
	106	```ini
	107	[Unit]
	108	Description=Hyperident P2P VPN Gateway
	109	After=network.target
	110
	111	[Service]
	112	Type=simple
	113	User=youruser # ✓ Regular user
	114	WorkingDirectory=/opt/hyperident-gateway
	115	Environment="PATH=/home/youruser/.config/pear/bin:/usr/local/bin:/usr/bin:/bin"
	116
	117	# Grant capabilities
	118	AmbientCapabilities=CAP_NET_ADMIN
	119	CapabilityBoundingSet=CAP_NET_ADMIN
	120
	121	ExecStart=/home/youruser/.config/pear/bin/pear run /opt/hyperident-gateway
	122	Restart=always
	123	RestartSec=10
	124
	125	[Install]
	126	WantedBy=multi-user.target
	127	```
	128
	129	Then grant capability to binary:
	130
	131	```bash
	132	# Find the bare runtime binary
	133	PEAR_BIN=$(find ~/.config/pear/by-dkey -name "pear-runtime" \| head -1)
	134
	135	# Grant CAP_NET_ADMIN
	136	sudo setcap cap_net_admin+ep "$PEAR_BIN"
	137
	138	# Verify
	139	getcap "$PEAR_BIN"
	140	# Should show: ... = cap_net_admin+ep
	141	```
	142
	143	Why this works:
	144	- Runs as regular user (Pear permissions work)
	145	- Binary has CAP_NET_ADMIN (can create TUN devices)
	146	- No root needed
	147
	148	Source: Working solution from hyperident-gateway deployment
	149
	150	---
	151
	152	## Common Service Patterns
	153
	154	### Pattern 1: Pear Terminal App
	155
	156	```ini
	157	[Service]
	158	User=youruser
	159	Environment="PATH=/home/youruser/.config/pear/bin:/usr/bin"
	160	ExecStart=/home/youruser/.config/pear/bin/pear run /opt/your-app
	161	Restart=always
	162	```
	163
	164	When to use: Standard Pear apps
	165
	166	### Pattern 2: Bare Runtime Direct
	167
	168	```ini
	169	[Service]
	170	User=youruser
	171	WorkingDirectory=/opt/your-app
	172	Environment="NODE_ENV=production"
	173	ExecStart=/home/youruser/.config/pear/by-dkey/.../bin/bare index.js
	174	Restart=always
	175	```
	176
	177	When to use: Maximum control, avoiding Pear wrapper
	178
	179	To find bare binary:
	180
	181	```bash
	182	find ~/.config/pear/by-dkey -name "bare" \| head -1
	183	```
	184
	185	### Pattern 3: With Capabilities (VPN, Raw Sockets, etc.)
	186
	187	```ini
	188	[Service]
	189	User=youruser
	190	AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW
	191	CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW
	192	ExecStart=/home/youruser/.config/pear/bin/pear run /opt/your-app
	193	```
	194
	195	Then:
	196
	197	```bash
	198	sudo setcap cap_net_admin,cap_net_raw+ep /path/to/bare-runtime
	199	```
	200
	201	When to use:
	202	- VPN gateways (CAP_NET_ADMIN)
	203	- Raw socket operations (CAP_NET_RAW)
	204	- Port binding <1024 (CAP_NET_BIND_SERVICE)
	205
	206	See: [capabilities.md](./capabilities.md) for full capability guide
	207
	208	### Pattern 4: With Environment Variables
	209
	210	```ini
	211	[Service]
	212	User=youruser
	213	Environment="PATH=/home/youruser/.config/pear/bin:/usr/bin"
	214	Environment="NODE_ENV=production"
	215	Environment="LOG_LEVEL=info"
	216	Environment="CONFIG_PATH=/etc/your-app/config.json"
	217	ExecStart=/home/youruser/.config/pear/bin/pear run /opt/your-app
	218	```
	219
	220	When to use: Apps needing runtime configuration
	221
	222	---
	223
	224	## Troubleshooting
	225
	226	### Problem: ERR_PERMISSION_REQUIRED
	227
	228	Symptom: Service fails with "Permission required to run key"
	229
	230	Cause: Running as root, key not trusted
	231
	232	Solutions:
	233
	234	Option 1: Run as regular user (recommended)
	235	```ini
	236	[Service]
	237	User=youruser # Not root
	238	AmbientCapabilities=CAP_NET_ADMIN # If needed
	239	```
	240
	241	Option 2: Pre-approve key as root (not recommended)
	242	```bash
	243	# One-time: run as root to approve
	244	sudo /home/user/.config/pear/bin/pear run pear://key
	245	# Type "TRUST" when prompted
	246	```
	247
	248	Option 3: Use local directory (recommended)
	249	```bash
	250	# Dump app to avoid permission prompt
	251	pear dump pear://key /opt/your-app
	252
	253	[Service]
	254	ExecStart=/home/user/.config/pear/bin/pear run /opt/your-app
	255	```
	256
	257	### Problem: Command Not Found
	258
	259	Symptom: `systemd: Failed to execute: No such file or directory`
	260
	261	Cause: Pear not in PATH
	262
	263	Solution: Use absolute path
	264
	265	```ini
	266	[Service]
	267	# ❌ Wrong
	268	ExecStart=pear run /opt/your-app
	269
	270	# ✓ Correct
	271	ExecStart=/home/youruser/.config/pear/bin/pear run /opt/your-app
	272
	273	# Also set PATH for dependencies
	274	Environment="PATH=/home/youruser/.config/pear/bin:/usr/bin"
	275	```
	276
	277	### Problem: Module Not Found
	278
	279	Symptom: `MODULE_NOT_FOUND` errors when running as service
	280
	281	Cause: Multiple possible causes
	282
	283	Solutions:
	284
	285	1. Use local directory, not pear:// link
	286	```ini
	287	# ❌ Wrong (module resolution bug)
	288	ExecStart=pear run pear://key
	289
	290	# ✓ Correct
	291	ExecStart=pear run /opt/your-app
	292	```
	293
	294	2. Check node_modules exists
	295	```bash
	296	ls -la /opt/your-app/node_modules
	297	# Should exist and be populated
	298	```
	299
	300	3. Verify dependencies in package.json
	301	```bash
	302	cd /opt/your-app
	303	npm install --production
	304	```
	305
	306	See: [module-resolution.md](./module-resolution.md) for deep dive
	307
	308	### Problem: Permission Denied (TUN/TAP)
	309
	310	Symptom: `Error: Permission denied` when creating TUN device
	311
	312	Cause: Missing CAP_NET_ADMIN capability
	313
	314	Solution: Grant capability to binary
	315
	316	```bash
	317	# Find bare runtime
	318	BARE=$(find ~/.config/pear -name "pear-runtime" \| head -1)
	319
	320	# Grant capability
	321	sudo setcap cap_net_admin+ep "$BARE"
	322
	323	# Verify
	324	getcap "$BARE"
	325	```
	326
	327	Service file must have:
	328	```ini
	329	[Service]
	330	AmbientCapabilities=CAP_NET_ADMIN
	331	CapabilityBoundingSet=CAP_NET_ADMIN
	332	```
	333
	334	### Problem: Service Stops After Exit
	335
	336	Symptom: Service exits and doesn't restart
	337
	338	Cause: Missing restart configuration
	339
	340	Solution: Add restart directives
	341
	342	```ini
	343	[Service]
	344	Restart=always
	345	RestartSec=10
	346	```
	347
	348	Restart policies:
	349	- `no` - Don't restart
	350	- `on-failure` - Restart only on failure
	351	- `always` - Always restart
	352	- `on-abnormal` - Restart on abnormal exit
	353
	354	---
	355
	356	## Real-World Examples
	357
	358	### Example 1: hyperident-gateway (P2P VPN)
	359
	360	Validated against: hyper-fleeky/repos/hyperident-gateway
	361
	362	Service file:
	363
	364	```ini
	365	[Unit]
	366	Description=Hyperident P2P VPN Gateway
	367	After=network.target
	368	Documentation=https://github.com/fleeky/hyperident-gateway
	369
	370	[Service]
	371	Type=simple
	372	User=fleeky
	373	WorkingDirectory=/opt/hyperident-gateway
	374	Environment="PATH=/home/fleeky/.config/pear/bin:/usr/local/bin:/usr/bin:/bin"
	375	Environment="NODE_ENV=production"
	376
	377	# Required for TUN device creation
	378	AmbientCapabilities=CAP_NET_ADMIN
	379	CapabilityBoundingSet=CAP_NET_ADMIN
	380
	381	# Workaround for module resolution bug
	382	ExecStart=/home/fleeky/.config/pear/bin/pear run /opt/hyperident-gateway
	383
	384	# Restart on failure
	385	Restart=always
	386	RestartSec=10
	387
	388	# Logging
	389	StandardOutput=journal
	390	StandardError=journal
	391
	392	[Install]
	393	WantedBy=multi-user.target
	394	```
	395
	396	Setup commands:
	397
	398	```bash
	399	# Dump app (workaround for pear:// bug)
	400	pear dump pear://hdj3chnrp8wnfzq4b3fd9i1jhuha1ufeyces35oofyteh6csfsmy /opt/hyperident-gateway
	401
	402	# Install dependencies (if needed)
	403	cd /opt/hyperident-gateway
	404	npm install --production
	405
	406	# Find and grant capabilities to bare runtime
	407	BARE=$(find /home/fleeky/.config/pear -name "pear-runtime" \| head -1)
	408	sudo setcap cap_net_admin+ep "$BARE"
	409
	410	# Install service
	411	sudo cp hyperident-gateway.service /etc/systemd/system/
	412	sudo systemctl daemon-reload
	413	sudo systemctl enable hyperident-gateway
	414	sudo systemctl start hyperident-gateway
	415
	416	# Check status
	417	sudo systemctl status hyperident-gateway
	418	journalctl -u hyperident-gateway -f
	419	```
	420
	421	Key learnings:
	422	- Must use local directory (pear:// has module resolution bug)
	423	- CAP_NET_ADMIN needed for TUN devices
	424	- Run as regular user, not root
	425	- Explicit PATH required
	426
	427	### Example 2: Simple Pear Terminal App
	428
	429	For apps without special privileges:
	430
	431	```ini
	432	[Unit]
	433	Description=My Pear App
	434	After=network.target
	435
	436	[Service]
	437	Type=simple
	438	User=myuser
	439	WorkingDirectory=/opt/my-app
	440	Environment="PATH=/home/myuser/.config/pear/bin:/usr/bin"
	441	ExecStart=/home/myuser/.config/pear/bin/pear run /opt/my-app
	442	Restart=always
	443	RestartSec=10
	444
	445	[Install]
	446	WantedBy=multi-user.target
	447	```
	448
	449	### Example 3: Bare Runtime Direct
	450
	451	For maximum control:
	452
	453	```ini
	454	[Unit]
	455	Description=Bare P2P App
	456	After=network.target
	457
	458	[Service]
	459	Type=simple
	460	User=myuser
	461	WorkingDirectory=/opt/my-app
	462	Environment="NODE_ENV=production"
	463	ExecStart=/home/myuser/.config/pear/by-dkey/x5w.../by-arch/linux-x64/bin/bare index.js
	464	Restart=always
	465	RestartSec=10
	466
	467	[Install]
	468	WantedBy=multi-user.target
	469	```
	470
	471	---
	472
	473	## Testing Services
	474
	475	### Test Locally First
	476
	477	Before installing as service:
	478
	479	```bash
	480	# Test as user
	481	su - youruser
	482	cd /opt/your-app
	483	/home/youruser/.config/pear/bin/pear run .
	484
	485	# Test with capabilities (if needed)
	486	sudo -u youruser CAP_NET_ADMIN=1 pear run .
	487	```
	488
	489	### Install and Test Service
	490
	491	```bash
	492	# Install
	493	sudo cp your-app.service /etc/systemd/system/
	494	sudo systemctl daemon-reload
	495
	496	# Start
	497	sudo systemctl start your-app
	498
	499	# Check immediately
	500	sudo systemctl status your-app
	501
	502	# Watch logs
	503	journalctl -u your-app -f
	504	```
	505
	506	### Debugging Services
	507
	508	```bash
	509	# Full logs
	510	journalctl -u your-app --no-pager
	511
	512	# Logs since boot
	513	journalctl -u your-app -b
	514
	515	# Logs from last hour
	516	journalctl -u your-app --since "1 hour ago"
	517
	518	# Follow logs
	519	journalctl -u your-app -f
	520
	521	# Check environment
	522	sudo systemctl show your-app --property=Environment
	523	```
	524
	525	---
	526
	527	## Production Checklist
	528
	529	Before deploying service to production:
	530
	531	- [ ] Tested locally as user first
	532	- [ ] Service file uses regular user, not root
	533	- [ ] Full PATH set in Environment
	534	- [ ] Using local directory, not pear:// link
	535	- [ ] Capabilities granted to binary (if needed)
	536	- [ ] Restart policy configured
	537	- [ ] Logs accessible via journalctl
	538	- [ ] Service enabled for autostart
	539	- [ ] Tested service start/stop/restart
	540	- [ ] Monitored logs for errors
	541
	542	---
	543
	544	## See Also
	545
	546	- [pear-apps.md](./pear-apps.md) - Pear deployment basics
	547	- [capabilities.md](./capabilities.md) - Linux capabilities guide
	548	- [bare-runtime.md](./bare-runtime.md) - Bare runtime deployment
	549	- [troubleshooting.md](./troubleshooting.md) - Full troubleshooting
	550
	551	---
	552
	553	## References
	554
	555	- Systemd documentation: `man systemd.service`
	556	- Linux capabilities: `man capabilities`
	557	- Real-world example: hyper-fleeky/repos/hyperident-gateway
	558	- Lessons learned: hyper-fleeky/pear-deployment-lessons.md (Issues 2, 5)

deployment/troubleshooting.md +704 -0

…	…	@@ -0,0 +1,704 @@
	1	# Deployment Troubleshooting Index
	2
	3	Quick reference for diagnosing and resolving common deployment issues across Pear, Bare, and systemd.
	4
	5	How to use: Search for your symptom/error message, find the cause and solution.
	6
	7	---
	8
	9	## Table of Contents
	10
	11	1. [By Error Message](#by-error-message)
	12	2. [By Symptom](#by-symptom)
	13	3. [By Technology](#by-technology)
	14	4. [By Use Case](#by-use-case)
	15	5. [Quick Decision Trees](#quick-decision-trees)
	16
	17	---
	18
	19	## By Error Message
	20
	21	### "MODULE_NOT_FOUND"
	22
	23	#### Context: Pear app running from pear:// link
	24
	25	Error:
	26	```
	27	MODULE_NOT_FOUND: Cannot find module 'your-module' imported from 'pear://key/...'
	28	```
	29
	30	Causes & Solutions:
	31
	32	\| Cause \| Check \| Solution \| Severity \|
	33	\|-------\|-------\|----------\|----------\|
	34	\| node_modules ignored \| `pear.stage.ignore` includes node_modules \| Remove from ignore \| 🔴 Critical \|
	35	\| pear:// protocol bug \| Running from pear:// link \| Use local directory workaround \| 🔴 Critical \|
	36	\| Dynamic require \| Module conditionally required \| Add to dependencies explicitly \| 🟡 High \|
	37	\| Missing dependency \| Not in package.json \| Add to dependencies \| 🟡 High \|
	38	\| ESM/CJS interop bug \| Using "type": "module" \| Convert to CommonJS \| 🟠 Medium \|
	39
	40	Quick fixes:
	41
	42	1. Check if node_modules is ignored (most common):
	43	```bash
	44	jq '.pear.stage.ignore' package.json
	45	# Should NOT contain "node_modules"
	46	```
	47
	48	2. Use local directory workaround (pear:// bug):
	49	```bash
	50	pear dump pear://key /opt/app
	51	pear run /opt/app # Works, pear://key doesn't
	52	```
	53
	54	3. Add dynamic requires explicitly:
	55	```json
	56	{
	57	"dependencies": {
	58	"parent-module": "^1.0.0",
	59	"dynamically-required": "^1.0.0"
	60	}
	61	}
	62	```
	63
	64	Deep dive: [module-resolution.md](./module-resolution.md#critical-bug-pear-module-resolution)
	65
	66	---
	67
	68	#### Context: Bare runtime or Node.js
	69
	70	Error:
	71	```
	72	MODULE_NOT_FOUND: Cannot find module 'your-module'
	73	```
	74
	75	Causes & Solutions:
	76
	77	\| Cause \| Check \| Solution \|
	78	\|-------\|-------\|----------\|
	79	\| Not installed \| `ls node_modules/your-module` \| `npm install your-module` \|
	80	\| Wrong runtime \| Module for different runtime \| Check import maps \|
	81	\| Wrong directory \| CWD not app directory \| `cd /path/to/app` \|
	82
	83	Quick fix:
	84	```bash
	85	# Install missing module
	86	npm install your-module
	87
	88	# Or reinstall all
	89	rm -rf node_modules
	90	npm install
	91	```
	92
	93	---
	94
	95	### "ERR_PERMISSION_REQUIRED"
	96
	97	Error:
	98	```
	99	Error: Permission required to run key (ERR_PERMISSION_REQUIRED)
	100	```
	101
	102	Context: Running Pear app as systemd service
	103
	104	Cause: Running as root user, Pear key not trusted
	105
	106	Why this happens:
	107	- Root user has separate ~/.config/pear from regular user
	108	- Pear requires first-time approval of app keys
	109	- No interactive approval possible in systemd service
	110	- Root's Pear instance doesn't have key in trust list
	111
	112	Solutions:
	113
	114	✅ Solution 1: Run as regular user (recommended):
	115	```ini
	116	[Service]
	117	User=youruser # Not root
	118	AmbientCapabilities=CAP_NET_ADMIN # If needed
	119	CapabilityBoundingSet=CAP_NET_ADMIN
	120	```
	121
	122	✅ Solution 2: Use local directory:
	123	```bash
	124	# Dump to avoid permission prompt
	125	pear dump pear://key /opt/app
	126
	127	[Service]
	128	ExecStart=/home/user/.config/pear/bin/pear run /opt/app
	129	```
	130
	131	❌ Don't do this (not recommended):
	132	```bash
	133	# Pre-approve as root (security risk)
	134	sudo /home/user/.config/pear/bin/pear run pear://key
	135	# Type "TRUST"
	136	```
	137
	138	Deep dive: [systemd-services.md](./systemd-services.md#problem-err_permission_required)
	139
	140	---
	141
	142	### "Permission denied" (TUN/TAP devices)
	143
	144	Error:
	145	```
	146	Error: Permission denied
	147	EPERM: operation not permitted, ioctl
	148	```
	149
	150	Context: Creating TUN/TAP devices (VPN apps, networking)
	151
	152	Cause: Missing CAP_NET_ADMIN capability
	153
	154	Solution: Grant capability to binary
	155
	156	```bash
	157	# Find bare runtime
	158	BARE=$(find ~/.config/pear -name "pear-runtime" \| head -1)
	159
	160	# Grant capability
	161	sudo setcap cap_net_admin+ep "$BARE"
	162
	163	# Verify
	164	getcap "$BARE"
	165	# Should show: cap_net_admin+ep
	166	```
	167
	168	Service file must have:
	169	```ini
	170	[Service]
	171	User=youruser
	172	AmbientCapabilities=CAP_NET_ADMIN
	173	CapabilityBoundingSet=CAP_NET_ADMIN
	174	```
	175
	176	Deep dive: [capabilities.md](./capabilities.md) (when created)
	177
	178	Real example: hyper-fleeky/hyperident-gateway (P2P VPN)
	179
	180	---
	181
	182	### "Command not found: pear"
	183
	184	Error:
	185	```
	186	bash: pear: command not found
	187	systemd: Failed to execute: No such file or directory
	188	```
	189
	190	Context: Running pear command in systemd service or terminal
	191
	192	Cause: Pear not in PATH
	193
	194	Solutions:
	195
	196	For systemd:
	197	```ini
	198	[Service]
	199	Environment="PATH=/home/user/.config/pear/bin:/usr/bin:/bin"
	200	ExecStart=/home/user/.config/pear/bin/pear run /opt/app # Full path
	201	```
	202
	203	For shell:
	204	```bash
	205	# Add to ~/.bashrc or ~/.zshrc
	206	export PATH="$HOME/.config/pear/bin:$PATH"
	207
	208	# Or run with full path
	209	~/.config/pear/bin/pear run .
	210	```
	211
	212	Deep dive: [systemd-services.md](./systemd-services.md#problem-command-not-found)
	213
	214	---
	215
	216	### Pear hangs / freezes / timeout
	217
	218	Symptom: `pear info`, `pear run`, or other commands hang indefinitely
	219
	220	Cause: Corrupted Pear installation, stale lock files
	221
	222	Solution: Complete clean reinstall
	223
	224	```bash
	225	# Nuclear option
	226	rm -rf ~/.config/pear
	227	npx pear
	228
	229	# Follow PATH setup instructions output by npx pear
	230	```
	231
	232	When this happens:
	233	- After deleting ~/.config/pear/by-dkey/*
	234	- After database lock errors
	235	- After incomplete pear updates
	236	- `pear info` timeouts
	237	- `pear run` exits silently with code 0
	238
	239	Prevention: Don't manually delete Pear directories
	240
	241	Deep dive: [pear-apps.md](./pear-apps.md#pear-installation-corruption)
	242
	243	---
	244
	245	### "No such file or directory" (systemd)
	246
	247	Error in journalctl:
	248	```
	249	systemd: Failed to start your-app.service: No such file or directory
	250	```
	251
	252	Cause: ExecStart path incorrect
	253
	254	Solutions:
	255
	256	Check ExecStart path:
	257	```ini
	258	[Service]
	259	# ❌ Wrong - relative path
	260	ExecStart=pear run app
	261
	262	# ❌ Wrong - wrong user's home
	263	ExecStart=/home/wronguser/.config/pear/bin/pear run /opt/app
	264
	265	# ✓ Correct - full path to correct user's pear
	266	ExecStart=/home/correctuser/.config/pear/bin/pear run /opt/app
	267	```
	268
	269	Find correct path:
	270	```bash
	271	# As the service user
	272	su - youruser
	273	which pear
	274	# Use this full path in ExecStart
	275	```
	276
	277	---
	278
	279	## By Symptom
	280
	281	### App works locally, fails when deployed
	282
	283	Symptoms:
	284	- `pear run --dev .` works
	285	- `pear run pear://key` fails with MODULE_NOT_FOUND
	286	- Works on your machine, fails on remote machine
	287
	288	Causes:
	289
	290	\| Cause \| Check \| Solution \|
	291	\|-------\|-------\|----------\|
	292	\| pear:// protocol bug \| Running from link \| Use local directory \|
	293	\| Local cache \| Have old/different version cached \| Clear cache \|
	294	\| node_modules not bundled \| Low block count (11 vs 200+) \| Remove from ignore \|
	295	\| Platform differences \| Works on Mac, fails on Linux \| Check platform-specific deps \|
	296
	297	Debugging steps:
	298
	299	1. Check staging block count:
	300	```bash
	301	pear stage production
	302	# Low count (11 blocks) = no deps ❌
	303	# High count (200+ blocks) = deps included ✓
	304	```
	305
	306	2. Test locally with dump:
	307	```bash
	308	pear dump pear://key /tmp/test
	309	pear run /tmp/test
	310	# Should work if bundling is correct
	311	```
	312
	313	3. Test on clean machine:
	314	```bash
	315	# Clear cache first
	316	rm -rf ~/.config/pear/app-storage/*
	317	pear run pear://key
	318	```
	319
	320	Solutions:
	321	- Use local directory deployment: [pear-apps.md](./pear-apps.md#strategy-2-local-directory)
	322	- Fix bundling: [pear-apps.md](./pear-apps.md#critical-configuration)
	323
	324	---
	325
	326	### Old version running after release
	327
	328	Symptoms:
	329	- Released new version with `pear release production`
	330	- Remote machines still running old version
	331	- Changes not appearing
	332
	333	Causes:
	334
	335	\| Cause \| Check \| Solution \|
	336	\|-------\|-------\|----------\|
	337	\| Cache not cleared \| Old version cached locally \| Clear cache \|
	338	\| Not seeding \| No active seeders \| Run pear seed \|
	339	\| Using wrong checkout \| Default to released \| Use --checkout=staged \|
	340
	341	Solutions:
	342
	343	1. Clear cache on client:
	344	```bash
	345	rm -rf ~/.config/pear/app-storage/*
	346	rm -rf ~/.config/pear/corestores/app/*
	347	pear run pear://key # Downloads fresh
	348	```
	349
	350	2. Force specific version:
	351	```bash
	352	pear run --checkout=staged pear://key # Latest staged
	353	pear run --checkout=latest pear://key # Force refresh
	354	```
	355
	356	3. Ensure seeding:
	357	```bash
	358	# Keep seeding after release
	359	pear seed production &
	360
	361	# Or as systemd service
	362	# See systemd-services.md
	363	```
	364
	365	Deep dive: [pear-apps.md](./pear-apps.md#cache-management)
	366
	367	---
	368
	369	### Service starts but immediately exits
	370
	371	Symptom: systemd shows service started but immediately goes inactive
	372
	373	Check logs:
	374	```bash
	375	journalctl -u your-app -n 50 --no-pager
	376	```
	377
	378	Common causes:
	379
	380	\| Log shows \| Cause \| Solution \|
	381	\|-----------\|-------\|----------\|
	382	\| MODULE_NOT_FOUND \| Dependencies missing \| Use local directory \|
	383	\| ERR_PERMISSION_REQUIRED \| Running as root \| Use regular user + capabilities \|
	384	\| Command not found \| PATH incorrect \| Set full PATH in service \|
	385	\| No error, just exits \| App exiting normally \| Check app logic \|
	386
	387	Debug:
	388	```bash
	389	# Run manually as service user
	390	su - serviceuser
	391	cd /opt/app
	392	pear run .
	393	# See what error appears
	394	```
	395
	396	---
	397
	398	### Can't connect to peers
	399
	400	Symptoms:
	401	- App starts but no peer connections
	402	- Replication not working
	403	- DHT lookup fails
	404
	405	Common causes:
	406
	407	\| Cause \| Check \| Solution \|
	408	\|-------\|-------\|----------\|
	409	\| Firewall blocking \| UDP blocked \| Open UDP ports or disable firewall \|
	410	\| Wrong key \| Using publicKey not discoveryKey \| Use core.discoveryKey for swarm \|
	411	\| No bootstrap \| Can't find DHT nodes \| Check network connectivity \|
	412	\| NAT issues \| Behind strict NAT \| May need relay \|
	413
	414	Debug:
	415	```bash
	416	# Test DHT connectivity
	417	node -e "
	418	const DHT = require('hyperdht')
	419	const dht = new DHT()
	420	dht.ready().then(() => {
	421	console.log('DHT ready, listening on:', dht.address())
	422	process.exit(0)
	423	})
	424	"
	425	```
	426
	427	See: Pattern docs (when created) for replication troubleshooting
	428
	429	---
	430
	431	### High memory/CPU usage
	432
	433	Symptom: App consuming excessive resources
	434
	435	Common causes in P2P apps:
	436
	437	\| Cause \| Check \| Solution \|
	438	\|-------\|-------\|----------\|
	439	\| Creating core per user \| Multiple Hypercore instances \| Use Corestore \|
	440	\| Not closing connections \| Memory leak \| Add teardown/cleanup \|
	441	\| Full replication \| Downloading everything \| Use sparse mode \|
	442	\| No backpressure \| Overwhelming with data \| Add flow control \|
	443
	444	Quick checks:
	445	```bash
	446	# Memory usage
	447	ps aux \| grep pear
	448	top -p $(pgrep -f "pear")
	449
	450	# Open connections
	451	lsof -p $(pgrep -f "pear") \| grep TCP
	452	```
	453
	454	---
	455
	456	## By Technology
	457
	458	### Pear Issues
	459
	460	\| Symptom \| Likely Cause \| Document \|
	461	\|---------\|--------------\|----------\|
	462	\| MODULE_NOT_FOUND under pear:// \| Protocol bug \| [module-resolution.md](./module-resolution.md#critical-bug-pear-module-resolution) \|
	463	\| node_modules not bundled \| In ignore list \| [pear-apps.md](./pear-apps.md#never-ignore-node_modules) \|
	464	\| Old version cached \| Cache issue \| [pear-apps.md](./pear-apps.md#cache-management) \|
	465	\| Pear hangs \| Corruption \| [pear-apps.md](./pear-apps.md#pear-installation-corruption) \|
	466	\| Dynamic require fails \| Not explicit \| [module-resolution.md](./module-resolution.md#dynamic-require-patterns) \|
	467
	468	General Pear troubleshooting: [pear-apps.md](./pear-apps.md)
	469
	470	---
	471
	472	### Systemd Issues
	473
	474	\| Symptom \| Likely Cause \| Document \|
	475	\|---------\|--------------\|----------\|
	476	\| ERR_PERMISSION_REQUIRED \| Running as root \| [systemd-services.md](./systemd-services.md#capabilities-vs-root) \|
	477	\| Command not found \| PATH not set \| [systemd-services.md](./systemd-services.md#problem-command-not-found) \|
	478	\| Permission denied (TUN) \| Missing capability \| [systemd-services.md](./systemd-services.md#pattern-3-with-capabilities) \|
	479	\| Service exits immediately \| Check logs \| [systemd-services.md](./systemd-services.md#troubleshooting) \|
	480
	481	General systemd troubleshooting: [systemd-services.md](./systemd-services.md#troubleshooting)
	482
	483	---
	484
	485	### Bare Runtime Issues
	486
	487	\| Symptom \| Likely Cause \| Document \|
	488	\|---------\|--------------\|----------\|
	489	\| ESM import fails \| ESM/CJS interop \| [module-resolution.md](./module-resolution.md#esmcjs-interop) \|
	490	\| Module not found \| Missing in node_modules \| Install with npm \|
	491	\| Import map not working \| Only works at definition level \| [module-resolution.md](./module-resolution.md#import-maps) \|
	492
	493	General Bare troubleshooting: [bare-runtime.md](./bare-runtime.md) (when created)
	494
	495	---
	496
	497	### Hypercore/Hyperswarm Issues
	498
	499	\| Symptom \| Likely Cause \| Check \|
	500	\|---------\|--------------\|-------\|
	501	\| Replication not working \| Wrong key used \| Use discoveryKey not publicKey \|
	502	\| Can't connect to peers \| Firewall \| Check UDP ports \|
	503	\| Data not syncing \| Not seeding \| At least one peer must seed \|
	504
	505	See: Pattern docs (when created) for P2P-specific troubleshooting
	506
	507	---
	508
	509	## By Use Case
	510
	511	### Building a VPN Gateway
	512
	513	Common issues:
	514	1. Permission denied (TUN) → Need CAP_NET_ADMIN
	515	2. Service won't start as root → Use user + capabilities
	516	3. MODULE_NOT_FOUND → Use local directory deployment
	517
	518	Complete example: hyper-fleeky/hyperident-gateway
	519	Guide: [systemd-services.md](./systemd-services.md#example-1-hyperident-gateway-p2p-vpn)
	520
	521	---
	522
	523	### Deploying Pear Terminal App
	524
	525	Common issues:
	526	1. MODULE_NOT_FOUND under pear:// → Use local directory
	527	2. Old version cached → Clear cache or use --checkout
	528	3. Dependencies missing → Check not in ignore
	529
	530	Guide: [pear-apps.md](./pear-apps.md)
	531
	532	---
	533
	534	### Running as Systemd Service
	535
	536	Common issues:
	537	1. ERR_PERMISSION_REQUIRED → Don't use root
	538	2. Command not found → Set PATH
	539	3. Module resolution → Use local directory
	540
	541	Guide: [systemd-services.md](./systemd-services.md)
	542
	543	---
	544
	545	## Quick Decision Trees
	546
	547	### "My Pear app won't run"
	548
	549	```
	550	Does `pear run --dev .` work?
	551	├─ YES → Problem with deployment
	552	│ ├─ Using pear:// link?
	553	│ │ └─ YES → Use local directory workaround (pear:// bug)
	554	│ └─ Remote machine?
	555	│ └─ YES → Clear cache or check bundling
	556	│
	557	└─ NO → Problem with app code
	558	├─ MODULE_NOT_FOUND?
	559	│ └─ YES → Check dependencies in package.json
	560	└─ Other error?
	561	└─ Fix app code
	562	```
	563
	564	### "Module not found in production"
	565
	566	```
	567	Where are you running from?
	568	├─ pear:// link
	569	│ └─ pear:// protocol bug → Use pear dump + local directory
	570	│
	571	├─ Local directory
	572	│ ├─ Does node_modules exist?
	573	│ │ ├─ YES → Check staging included it (high block count)
	574	│ │ └─ NO → npm install
	575	│ └─ Is it in pear.stage.ignore?
	576	│ └─ YES → Remove node_modules from ignore
	577	│
	578	└─ Systemd service
	579	└─ Same as local directory checks above
	580	```
	581
	582	### "Permission error"
	583
	584	```
	585	What type of permission error?
	586	├─ ERR_PERMISSION_REQUIRED
	587	│ └─ Running as root? → Use regular user + capabilities
	588	│
	589	├─ Permission denied (EPERM)
	590	│ └─ What operation?
	591	│ ├─ TUN device → Need CAP_NET_ADMIN
	592	│ ├─ Raw socket → Need CAP_NET_RAW
	593	│ └─ Port <1024 → Need CAP_NET_BIND_SERVICE
	594	│
	595	└─ EACCES (file access)
	596	└─ Check file ownership and permissions
	597	```
	598
	599	---
	600
	601	## Emergency Procedures
	602
	603	### When completely stuck
	604
	605	1. Start fresh:
	606	```bash
	607	# Clean Pear installation
	608	rm -rf ~/.config/pear
	609	npx pear
	610
	611	# Clean project
	612	rm -rf node_modules
	613	npm install
	614	```
	615
	616	2. Test bare minimum:
	617	```bash
	618	# Minimal test
	619	echo "console.log('hello')" > test.js
	620	pear run test.js
	621	# If this fails, Pear installation broken
	622	```
	623
	624	3. Use workarounds:
	625	```bash
	626	# Bypass Pear wrapper
	627	pear dump pear://key /tmp/test
	628	cd /tmp/test
	629	bare index.js
	630	# If this works, it's a Pear issue not app issue
	631	```
	632
	633	---
	634
	635	## Getting Help
	636
	637	### Information to collect
	638
	639	When asking for help or filing bugs:
	640
	641	1. Environment:
	642	```bash
	643	# Versions
	644	pear --version
	645	node --version
	646	npm --version
	647	bare --version 2>/dev/null \|\| echo "No bare"
	648
	649	# OS
	650	uname -a
	651	```
	652
	653	2. App info:
	654	```bash
	655	# package.json relevant parts
	656	jq '{name, pear, dependencies}' package.json
	657
	658	# Staging info
	659	pear stage dev
	660	# Note the block count
	661	```
	662
	663	3. Error info:
	664	```bash
	665	# Exact error
	666	pear run pear://key 2>&1 \| tee error.log
	667
	668	# Systemd logs (if service)
	669	journalctl -u your-app -n 100 --no-pager
	670	```
	671
	672	4. What works/doesn't:
	673	- Does `pear run --dev .` work?
	674	- Does `pear run /local/dir` work?
	675	- Does `pear run pear://key` work?
	676	- Does `bare index.js` work?
	677
	678	### Where to report
	679
	680	- Pear bugs: https://github.com/holepunchto/pear/issues
	681	- Bare bugs: https://github.com/holepunchto/bare/issues
	682	- Hypercore/Hyperswarm: Respective repos
	683
	684	Include: Minimal reproduction case, environment info, what you've tried
	685
	686	---
	687
	688	## Related Documentation
	689
	690	- [pear-apps.md](./pear-apps.md) - Pear deployment guide
	691	- [systemd-services.md](./systemd-services.md) - Systemd service setup
	692	- [module-resolution.md](./module-resolution.md) - Deep dive on MODULE_NOT_FOUND
	693	- [bare-runtime.md](./bare-runtime.md) - Bare deployment (when created)
	694	- [capabilities.md](./capabilities.md) - Linux capabilities (when created)
	695
	696	---
	697
	698	## Lessons Learned Sources
	699
	700	This troubleshooting guide integrates issues from:
	701	- hyper-fleeky/pear-deployment-lessons.md (6 major issues)
	702	- hyperident-gateway production deployment
	703	- holepunchto/pear test fixtures
	704	- Real-world systemd service deployments

designer-agent.js +3 -1

…	…	@@ -18,7 +18,9 @@ const semanticCategories = JSON.parse(fs.readFileSync('semantic-categories.json'
18	18	const useCaseMap = JSON.parse(fs.readFileSync('use-case-map.json', 'utf8'));
19	19	const stabilityIndex = JSON.parse(fs.readFileSync('stability-index.json', 'utf8'));
20	20	const platformSupport = JSON.parse(fs.readFileSync('platform-support.json', 'utf8'));
21		const repoMapArray = JSON.parse(fs.readFileSync('repo_map.json', 'utf8'));
	21	const repoMapData = JSON.parse(fs.readFileSync('repo_map.json', 'utf8'));
	22	// Support both old array format and new object format with _meta
	23	const repoMapArray = repoMapData.repos \|\| repoMapData;
22	24
23	25	// Convert repo map to object
24	26	const repoMap = {};

pattern-map.json +22 -0

…	…	@@ -1,4 +1,26 @@
1	1	{
	2	"_meta": {
	3	"description": "P2P pattern mappings across the Hypercore ecosystem",
	4	"documentation": {
	5	"patterns": "See patterns/ directory for detailed pattern documentation with code examples",
	6	"concepts": "See CONCEPTS.md for pattern integration and relationships"
	7	},
	8	"patternCount": 10,
	9	"patterns": [
	10	"append-only-log",
	11	"replication",
	12	"streaming",
	13	"cryptography",
	14	"peer-discovery",
	15	"binary-encoding",
	16	"stream-multiplexing",
	17	"dht-discovery",
	18	"b-tree-index",
	19	"file-system"
	20	],
	21	"lastUpdated": "2026-01-18",
	22	"phase": "5.4 - JSON Metadata Enhancement"
	23	},
2	24	"patterns": {
3	25	"append-only-log": {
4	26	"description": "Immutable, append-only data structure for distributed systems",

patterns/README.md +148 -0

…	…	@@ -0,0 +1,148 @@
	1	# Hypercore Ecosystem Patterns
	2
	3	Purpose: Concrete code examples and best practices for common P2P patterns.
	4
	5	## Overview
	6
	7	This directory contains pattern documentation validated against real holepunchto repositories and hyper-fleeky projects. Each pattern includes:
	8
	9	- Code examples from tier 1/2 repos
	10	- Real-world usage from production projects
	11	- Common gotchas with solutions
	12	- Anti-patterns to avoid
	13	- Integration patterns showing how patterns combine
	14
	15	## Core Patterns
	16
	17	### Data Structures
	18
	19	1. [append-only-log.md](./append-only-log.md) - Hypercore append-only logs
	20	- Basic append/read operations
	21	- Sparse replication
	22	- Value encoding (JSON, UTF-8, binary)
	23	- Live streams
	24	- Validated: hypercore, fedicore
	25
	26	2. [b-tree-index.md](./b-tree-index.md) - Hyperbee indexed databases (in progress)
	27	- Key-value operations
	28	- Range queries
	29	- Sorted iteration
	30	- Atomic batches
	31	- Validated: hyperbee
	32
	33	### Networking
	34
	35	3. [replication.md](./replication.md) - P2P replication strategies
	36	- Basic replication
	37	- Multi-core with Corestore
	38	- Live replication
	39	- Sparse sync
	40	- Validated: hypercore, hyperswarm, fedicore
	41
	42	4. [peer-discovery.md](./peer-discovery.md) - DHT-based peer discovery
	43	- Hyperswarm discovery
	44	- Server/client modes
	45	- Multiple topics
	46	- Direct peer connections
	47	- Validated: hyperswarm, hyperdht, fedicore
	48
	49	### Security
	50
	51	5. [cryptography.md](./cryptography.md) - Cryptographic primitives (in progress)
	52	- Merkle tree verification
	53	- ED25519 signing
	54	- DiscoveryKey derivation
	55	- Block encryption
	56	- Validated: hypercore-crypto
	57
	58	## Additional Patterns (Planned)
	59
	60	### Data Patterns
	61	- streaming.md - Efficient data streaming (read/write streams, byte streams)
	62	- binary-encoding.md - Compact encoding for wire protocols
	63	- file-system.md - File system operations with Hyperdrive
	64
	65	### Protocol Patterns
	66	- stream-multiplexing.md - Protomux for protocol multiplexing
	67	- dht-discovery.md - Deep dive on HyperDHT
	68
	69	## Usage
	70
	71	Each pattern follows this structure:
	72
	73	```markdown
	74	# Pattern: Name
	75
	76	## Description
	77	What this pattern is and why it exists
	78
	79	## When to Use
	80	Use cases and anti-use cases
	81
	82	## Implementing Repos
	83	Tier 1/2 repos that implement this pattern
	84
	85	## Code Example: [Specific Scenario]
	86	Real code from official repos
	87
	88	## Real-World Usage
	89	Examples from hyper-fleeky projects
	90
	91	## Common Gotchas
	92	Mistakes with solutions
	93
	94	## Anti-Patterns
	95	What NOT to do
	96
	97	## Integration Patterns
	98	How patterns combine
	99
	100	## See Also
	101	Cross-references
	102
	103	## References
	104	Validation sources
	105	```
	106
	107	## Validation Status
	108
	109	\| Pattern \| Status \| Validated Against \|
	110	\|---------\|--------\|------------------\|
	111	\| append-only-log \| ✅ Complete \| hypercore, fedicore \|
	112	\| replication \| ✅ Complete \| hypercore, hyperswarm, fedicore \|
	113	\| peer-discovery \| ✅ Complete \| hyperswarm, hyperdht, fedicore \|
	114	\| b-tree-index \| ✅ Complete \| hyperbee \|
	115	\| cryptography \| ✅ Complete \| hypercore-crypto, hyperident \|
	116	\| streaming \| ✅ Complete \| hypercore \|
	117	\| stream-multiplexing \| ✅ Complete \| protomux \|
	118	\| binary-encoding \| ✅ Complete \| compact-encoding \|
	119	\| file-system \| ✅ Complete \| hyperdrive \|
	120
	121	## Pattern Categories
	122
	123	### Building Blocks (Tier 1)
	124	- append-only-log (hypercore)
	125	- b-tree-index (hyperbee)
	126	- file-system (hyperdrive)
	127	- peer-discovery (hyperswarm, hyperdht)
	128
	129	### Helpers (Tier 2)
	130	- stream-multiplexing (protomux)
	131	- binary-encoding (compact-encoding)
	132	- replication (corestore)
	133
	134	### Cross-Cutting
	135	- cryptography (hypercore-crypto)
	136	- streaming (streamx)
	137
	138	## See Also
	139
	140	- [../deployment/](../deployment/) - Deployment patterns (Pear, systemd, Bare)
	141	- [../skills/](../skills/) - API reference for 529 repos
	142	- [../PHASES_4-6_PLAN.md](../PHASES_4-6_PLAN.md) - Knowledge base development plan
	143
	144	---
	145
	146	Phase: 4.1 - Pattern Library with Code Examples
	147	Status: ✅ Complete (9/9 core patterns)
	148	Last Updated: 2026-01-18

patterns/append-only-log.md +882 -0

…	…	@@ -0,0 +1,882 @@
	1	# Pattern: Append-Only Log
	2
	3	Validated against: holepunchto/hypercore, hyper-fleeky/fedicore
	4
	5	## Description
	6
	7	An append-only log is a data structure where entries can only be added to the end, never modified or deleted (though Hypercore 10+ supports truncation). Each entry has a sequential index starting from 0.
	8
	9	Core properties:
	10	- Immutable history: Once written, entries cannot be changed
	11	- Sequential indexing: Entries are numbered 0, 1, 2, ... N-1
	12	- Cryptographic verification: Merkle tree ensures data integrity
	13	- Sparse replication: Download only the blocks you need
	14	- Real-time updates: Get notified when new entries are appended
	15
	16	Key insight: Append-only logs are the foundation of P2P systems. They provide an immutable, verifiable history that can be replicated across peers.
	17
	18	---
	19
	20	## When to Use
	21
	22	✅ Use append-only logs when:
	23	- Building event sourcing systems
	24	- Creating distributed databases (with Hyperbee on top)
	25	- Implementing P2P messaging/social networks
	26	- Storing time-series data
	27	- Building version control systems
	28	- Creating audit trails
	29	- Implementing CRDTs (with Autobase)
	30
	31	❌ Don't use when:
	32	- You need frequent updates to existing records (use Hyperbee instead)
	33	- You need complex queries (use Hyperbee for indexing)
	34	- Storage is extremely limited (logs grow indefinitely unless cleared)
	35	- You need relational joins (not a good fit for append-only)
	36
	37	---
	38
	39	## Implementing Repos
	40
	41	### Tier 1 (Core Building Blocks)
	42
	43	- [hypercore](https://github.com/holepunchto/hypercore) (2,745 ⭐)
	44	- The reference implementation
	45	- Secure, distributed append-only log
	46	- Sparse replication, merkle tree verification
	47	- `npm install hypercore`
	48
	49	### Built on Hypercore
	50
	51	- [hyperbee](https://github.com/holepunchto/hyperbee) - B-tree database on Hypercore
	52	- [hyperdrive](https://github.com/holepunchto/hyperdrive) - File system on Hypercore
	53	- [autobase](https://github.com/holepunchto/autobase) - Multi-writer logs with CRDTs
	54
	55	---
	56
	57	## Code Example: Basic Usage
	58
	59	Creating and appending to a Hypercore:
	60
	61	```js
	62	const Hypercore = require('hypercore')
	63
	64	// Create a new append-only log
	65	const core = new Hypercore('./my-log')
	66	await core.ready()
	67
	68	console.log('Core key:', core.key) // Public key identifying this log
	69	console.log('Writable:', core.writable) // true (we created it)
	70
	71	// Append some data
	72	await core.append('Hello, P2P world!')
	73	await core.append('Second entry')
	74	await core.append('Third entry')
	75
	76	console.log('Length:', core.length) // 3
	77
	78	// Read entries by index
	79	const block0 = await core.get(0)
	80	const block1 = await core.get(1)
	81	const block2 = await core.get(2)
	82
	83	console.log(block0.toString()) // 'Hello, P2P world!'
	84	console.log(block1.toString()) // 'Second entry'
	85	console.log(block2.toString()) // 'Third entry'
	86
	87	// Append multiple blocks as a batch
	88	await core.append([
	89	Buffer.from('Batch block 1'),
	90	Buffer.from('Batch block 2')
	91	])
	92
	93	console.log('Length:', core.length) // 5
	94	```
	95
	96	Source: holepunchto/hypercore/README.md:117-123
	97
	98	---
	99
	100	## Code Example: Reading with Streams
	101
	102	Reading entries as a stream:
	103
	104	```js
	105	const core = new Hypercore('./my-log')
	106	await core.ready()
	107
	108	// Read all entries as a stream
	109	const fullStream = core.createReadStream()
	110
	111	for await (const data of fullStream) {
	112	console.log('Entry:', data.toString())
	113	}
	114
	115	// Read a range (blocks 10-14)
	116	const partialStream = core.createReadStream({
	117	start: 10,
	118	end: 15
	119	})
	120
	121	for await (const data of partialStream) {
	122	console.log('Entry:', data.toString())
	123	}
	124
	125	// Live stream - wait for new entries
	126	const liveStream = core.createReadStream({
	127	start: core.length, // Start from current end
	128	live: true // Keep stream open for new entries
	129	})
	130
	131	liveStream.on('data', (data) => {
	132	console.log('New entry:', data.toString())
	133	})
	134	```
	135
	136	Source: holepunchto/hypercore/README.md:216-244
	137
	138	---
	139
	140	## Code Example: Value Encoding
	141
	142	Using JSON encoding for structured data:
	143
	144	```js
	145	// Create core with JSON encoding
	146	const core = new Hypercore('./my-log', {
	147	valueEncoding: 'json'
	148	})
	149	await core.ready()
	150
	151	// Append JSON objects directly
	152	await core.append({
	153	type: 'post',
	154	text: 'Hello world',
	155	timestamp: Date.now()
	156	})
	157
	158	await core.append({
	159	type: 'like',
	160	postId: 0,
	161	timestamp: Date.now()
	162	})
	163
	164	// Read JSON objects
	165	const post = await core.get(0)
	166	console.log(post.type) // 'post'
	167	console.log(post.text) // 'Hello world'
	168	console.log(post.timestamp) // 1737156000000
	169
	170	const like = await core.get(1)
	171	console.log(like.type) // 'like'
	172	console.log(like.postId) // 0
	173	```
	174
	175	Supported encodings:
	176	- `'binary'` - Raw buffers (default)
	177	- `'json'` - JSON.stringify/parse
	178	- `'utf-8'` - String encoding
	179	- Custom - Any [compact-encoding](https://github.com/holepunchto/compact-encoding) instance
	180
	181	Source: holepunchto/hypercore/README.md:44-72
	182
	183	---
	184
	185	## Code Example: With Replication
	186
	187	Replicating a log between peers:
	188
	189	```js
	190	// On Server (has the data)
	191	const net = require('net')
	192	const Hypercore = require('hypercore')
	193
	194	const core = new Hypercore('./server-log')
	195	await core.ready()
	196
	197	// Populate with data
	198	await core.append('Server entry 1')
	199	await core.append('Server entry 2')
	200
	201	const server = net.createServer((socket) => {
	202	console.log('Client connected')
	203
	204	// Replicate over socket (server = false, we're passive)
	205	socket.pipe(core.replicate(false)).pipe(socket)
	206	})
	207
	208	server.listen(3000)
	209	console.log('Listening on port 3000')
	210	console.log('Core key:', core.key.toString('hex'))
	211
	212	// On Client (downloading the data)
	213	const socket = net.connect(3000)
	214
	215	// Create core with server's key (read-only replica)
	216	const core = new Hypercore('./client-log', serverKey)
	217	await core.ready()
	218
	219	// Replicate over socket (client = true, we're initiator)
	220	socket.pipe(core.replicate(true)).pipe(socket)
	221
	222	// Wait for data to arrive
	223	await core.update() // Wait for remote length
	224	console.log('Remote length:', core.length)
	225
	226	// Read replicated data
	227	const block0 = await core.get(0)
	228	const block1 = await core.get(1)
	229
	230	console.log(block0.toString()) // 'Server entry 1'
	231	console.log(block1.toString()) // 'Server entry 2'
	232
	233	// Listen for new entries
	234	core.on('append', () => {
	235	console.log('New entry! Length:', core.length)
	236	})
	237	```
	238
	239	Key concepts:
	240	- `replicate(isInitiator)` - Create replication stream
	241	- `true` = client (initiator)
	242	- `false` = server (passive)
	243	- Creating core with another peer's key makes it read-only
	244	- `core.update()` waits for remote length proof
	245	- `'append'` event fires when new entries arrive
	246
	247	Source: holepunchto/hypercore/README.md:671-682
	248
	249	---
	250
	251	## Code Example: Sparse Replication
	252
	253	Download only specific blocks you need:
	254
	255	```js
	256	const core = new Hypercore('./sparse-log', remoteKey)
	257	await core.ready()
	258
	259	// Start replication
	260	const stream = core.replicate(true)
	261	stream.pipe(socket).pipe(stream)
	262
	263	// Wait for remote length
	264	await core.update()
	265	console.log('Remote has', core.length, 'blocks')
	266
	267	// Check what we have locally
	268	const has0 = await core.has(0)
	269	const has1 = await core.has(1)
	270	console.log('Have block 0:', has0) // false (haven't downloaded)
	271	console.log('Have block 1:', has1) // false
	272
	273	// Download specific blocks on demand
	274	const block5 = await core.get(5) // Only downloads block 5
	275	const block99 = await core.get(99) // Only downloads block 99
	276
	277	console.log('Have block 5:', await core.has(5)) // true
	278	console.log('Have block 6:', await core.has(6)) // false (didn't download)
	279	console.log('Have block 99:', await core.has(99)) // true
	280
	281	// Download a range
	282	const range = core.download({ start: 10, end: 20 })
	283	await range.done() // Wait for range download
	284	console.log('Downloaded blocks 10-19')
	285
	286	// Download all blocks (non-sparse mode)
	287	core.download({ start: 0, end: -1 })
	288	```
	289
	290	Sparse replication benefits:
	291	- Save bandwidth - only download what you need
	292	- Save storage - don't store unnecessary blocks
	293	- Faster startup - don't wait for full sync
	294	- Works with huge datasets (TB+)
	295
	296	Source: holepunchto/hypercore/README.md:366-406
	297
	298	---
	299
	300	## Real-World Usage
	301
	302	### Example: Fedicore Social Network
	303
	304	From hyper-fleeky/repos/fedicore/fedicore.js - P2P social network using Hypercore:
	305
	306	```js
	307	import Corestore from 'corestore'
	308	import Hyperswarm from 'hyperswarm'
	309
	310	export class Fedicore extends EventEmitter {
	311	async init(options = {}) {
	312	// Initialize Corestore (manages multiple Hypercores)
	313	this.store = new Corestore(join(this.storagePath, 'feeds'))
	314	await this.store.ready()
	315
	316	// Get personal feed (writable append-only log)
	317	this.myFeed = this.store.get({ name: 'my-feed' })
	318	await this.myFeed.ready()
	319
	320	// Initialize P2P swarm
	321	this.swarm = new Hyperswarm()
	322	this.swarm.on('connection', (conn) => {
	323	// Replicate all feeds over each connection
	324	this.store.replicate(conn)
	325	})
	326
	327	// Join swarm topic for discovery
	328	const topic = this.myFeed.discoveryKey
	329	this.swarm.join(topic, { server: true, client: true })
	330	await this.swarm.flush()
	331	}
	332
	333	// Post a message (append to personal log)
	334	async post(content) {
	335	const message = {
	336	type: content.type \|\| 'post',
	337	text: content.text,
	338	timestamp: Date.now(),
	339	sequence: this.myFeed.length
	340	}
	341
	342	// Append to append-only log
	343	await this.myFeed.append(JSON.stringify(message))
	344
	345	this.emit('post', message)
	346	return message
	347	}
	348
	349	// Follow another user (replicate their log)
	350	async follow(publicKey, name = 'Anonymous') {
	351	// Get their feed (read-only replica)
	352	const feedKey = b4a.from(publicKey, 'base64')
	353	const feed = this.store.get(feedKey)
	354	await feed.ready()
	355
	356	this.follows.set(publicKey, { feed, name })
	357
	358	// Join their swarm topic to find peers
	359	const topic = feed.discoveryKey
	360	this.swarm.join(topic, { server: false, client: true })
	361
	362	// Now we'll replicate their log
	363	// New posts appear via 'append' event
	364	feed.on('append', () => {
	365	this.emit('follow-update', publicKey, feed.length)
	366	})
	367	}
	368	}
	369	```
	370
	371	Key patterns:
	372	1. Personal writable log: `this.myFeed` - user's own posts
	373	2. Read-only replicas: Other users' feeds
	374	3. Corestore: Manages multiple Hypercores efficiently
	375	4. Hyperswarm: P2P discovery and connections
	376	5. Auto-replication: `store.replicate(conn)` replicates all feeds
	377
	378	Source: hyper-fleeky/repos/fedicore/fedicore.js:8-150
	379
	380	---
	381
	382	## Common Gotchas
	383
	384	### 1. Waiting for `ready()` before using properties
	385
	386	❌ Wrong:
	387	```js
	388	const core = new Hypercore('./my-log')
	389	console.log(core.key) // null! Not ready yet
	390	console.log(core.length) // 0 (incorrect)
	391	```
	392
	393	✅ Correct:
	394	```js
	395	const core = new Hypercore('./my-log')
	396	await core.ready()
	397	console.log(core.key) // Buffer (correct)
	398	console.log(core.length) // Actual length
	399	```
	400
	401	Why: Properties like `key`, `length`, `writable` are only populated after `ready()` completes.
	402
	403	---
	404
	405	### 2. Forgetting `await` on async operations
	406
	407	❌ Wrong:
	408	```js
	409	core.append('Hello') // Returns promise, not awaited
	410	const block = core.get(0) // Returns promise, not awaited
	411	console.log(block) // Promise { <pending> }
	412	```
	413
	414	✅ Correct:
	415	```js
	416	await core.append('Hello')
	417	const block = await core.get(0)
	418	console.log(block.toString()) // 'Hello'
	419	```
	420
	421	Why: All I/O operations are async and return promises.
	422
	423	---
	424
	425	### 3. Trying to write to a read-only replica
	426
	427	❌ Wrong:
	428	```js
	429	// Created with someone else's key
	430	const core = new Hypercore('./replica', remoteKey)
	431	await core.ready()
	432
	433	await core.append('Hello') // Error: Not writable!
	434	```
	435
	436	✅ Correct:
	437	```js
	438	const core = new Hypercore('./replica', remoteKey)
	439	await core.ready()
	440
	441	console.log('Writable:', core.writable) // false
	442	if (core.writable) {
	443	await core.append('Hello')
	444	} else {
	445	console.log('Read-only replica, cannot append')
	446	}
	447	```
	448
	449	Why: Cores created with another peer's key are read-only. Only the holder of the secret key can append.
	450
	451	---
	452
	453	### 4. Not handling replication initiator correctly
	454
	455	❌ Wrong:
	456	```js
	457	// Both sides use true
	458	socket.pipe(core.replicate(true)).pipe(socket) // Client
	459	socket.pipe(core.replicate(true)).pipe(socket) // Server ❌
	460	```
	461
	462	✅ Correct:
	463	```js
	464	// Client (initiator)
	465	socket.pipe(core.replicate(true)).pipe(socket)
	466
	467	// Server (passive)
	468	socket.pipe(core.replicate(false)).pipe(socket)
	469	```
	470
	471	Why: Hypercore uses noise protocol which requires one initiator and one responder.
	472
	473	---
	474
	475	### 5. Expecting immediate `core.length` after remote replication
	476
	477	❌ Wrong:
	478	```js
	479	const core = new Hypercore('./replica', remoteKey)
	480	await core.ready()
	481
	482	const stream = core.replicate(true)
	483	stream.pipe(socket).pipe(stream)
	484
	485	console.log(core.length) // 0 - haven't received remote length yet!
	486	```
	487
	488	✅ Correct:
	489	```js
	490	const core = new Hypercore('./replica', remoteKey)
	491	await core.ready()
	492
	493	const stream = core.replicate(true)
	494	stream.pipe(socket).pipe(stream)
	495
	496	// Wait for remote length proof
	497	await core.update()
	498	console.log(core.length) // Correct remote length
	499	```
	500
	501	Why: Replication is async. Use `core.update()` to wait for remote length.
	502
	503	---
	504
	505	### 6. Block size limits
	506
	507	❌ Wrong:
	508	```js
	509	// Trying to append 100MB block
	510	const hugeBlock = Buffer.alloc(100 * 1024 * 1024)
	511	await core.append(hugeBlock) // Will fail to replicate smoothly!
	512	```
	513
	514	✅ Correct:
	515	```js
	516	// Keep blocks under 15MB (Hypercore.MAX_SUGGESTED_BLOCK_SIZE)
	517	const chunk = Buffer.alloc(10 * 1024 * 1024) // 10MB
	518	await core.append(chunk) // Works well
	519
	520	// For larger data, split into chunks
	521	const largeData = Buffer.alloc(100 * 1024 * 1024)
	522	const CHUNK_SIZE = 10 * 1024 * 1024
	523
	524	for (let i = 0; i < largeData.length; i += CHUNK_SIZE) {
	525	const chunk = largeData.slice(i, i + CHUNK_SIZE)
	526	await core.append(chunk)
	527	}
	528	```
	529
	530	Why: `Hypercore.MAX_SUGGESTED_BLOCK_SIZE` is 15MB. Larger blocks may cause replication issues.
	531
	532	Source: holepunchto/hypercore/README.md:721-724
	533
	534	---
	535
	536	## Anti-Patterns
	537
	538	### ❌ 1. Using Hypercore directly for key-value storage
	539
	540	Don't:
	541	```js
	542	// Terrible: linear search through append-only log
	543	async function getValue(key) {
	544	for (let i = 0; i < core.length; i++) {
	545	const block = await core.get(i)
	546	const data = JSON.parse(block)
	547	if (data.key === key) return data.value
	548	}
	549	return null // O(n) lookup!
	550	}
	551	```
	552
	553	Do: Use Hyperbee instead
	554	```js
	555	const Hyperbee = require('hyperbee')
	556	const db = new Hyperbee(core)
	557
	558	await db.put('key', 'value')
	559	const node = await db.get('key') // O(log n) lookup!
	560	```
	561
	562	Why: Append-only logs are optimized for sequential access. For random access by key, use Hyperbee (B-tree on Hypercore).
	563
	564	---
	565
	566	### ❌ 2. Downloading entire log when you only need recent entries
	567
	568	Don't:
	569	```js
	570	// Download everything (wasteful)
	571	const range = core.download({ start: 0, end: -1 })
	572	await range.done()
	573
	574	// Just to get the last 10 entries
	575	const recent = []
	576	for (let i = core.length - 10; i < core.length; i++) {
	577	recent.push(await core.get(i))
	578	}
	579	```
	580
	581	Do: Use sparse replication
	582	```js
	583	// Only download the last 10 blocks
	584	const recent = []
	585	const start = Math.max(0, core.length - 10)
	586
	587	for (let i = start; i < core.length; i++) {
	588	recent.push(await core.get(i)) // Downloads on demand
	589	}
	590	```
	591
	592	Why: Sparse replication is a key feature. Don't download data you don't need.
	593
	594	---
	595
	596	### ❌ 3. Not closing cores
	597
	598	Don't:
	599	```js
	600	async function processCore() {
	601	const core = new Hypercore('./temp-log')
	602	await core.ready()
	603	await core.append('data')
	604	// Forgot to close!
	605	} // Memory leak, file handles remain open
	606	```
	607
	608	Do: Always close
	609	```js
	610	async function processCore() {
	611	const core = new Hypercore('./temp-log')
	612	await core.ready()
	613
	614	try {
	615	await core.append('data')
	616	} finally {
	617	await core.close() // Clean up resources
	618	}
	619	}
	620	```
	621
	622	Why: Unclosed cores leak file handles and memory.
	623
	624	---
	625
	626	### ❌ 4. Ignoring error events
	627
	628	Don't:
	629	```js
	630	const core = new Hypercore('./my-log')
	631	await core.ready()
	632
	633	const stream = core.replicate(true)
	634	stream.pipe(socket).pipe(stream)
	635	// No error handling!
	636	```
	637
	638	Do: Handle errors
	639	```js
	640	const core = new Hypercore('./my-log')
	641	await core.ready()
	642
	643	const stream = core.replicate(true)
	644	stream.on('error', (err) => {
	645	console.error('Replication error:', err)
	646	// Handle appropriately
	647	})
	648
	649	socket.on('error', (err) => {
	650	console.error('Socket error:', err)
	651	})
	652
	653	stream.pipe(socket).pipe(stream)
	654	```
	655
	656	Why: Network errors, corruption, permission issues can occur during replication.
	657
	658	---
	659
	660	## Integration Patterns
	661
	662	### Pattern 1: Hypercore + Corestore (Multiple Logs)
	663
	664	When you need to manage many Hypercores (e.g., one per user):
	665
	666	```js
	667	const Corestore = require('corestore')
	668
	669	const store = new Corestore('./my-cores')
	670	await store.ready()
	671
	672	// Get a named core (deterministic)
	673	const userCore = store.get({ name: 'user-posts' })
	674	await userCore.ready()
	675
	676	// Get a core by key
	677	const friendCore = store.get(friendKey)
	678	await friendCore.ready()
	679
	680	// Replicate ALL cores in the store
	681	swarm.on('connection', (socket) => {
	682	store.replicate(socket)
	683	})
	684	```
	685
	686	Why: Corestore manages storage efficiently for many cores. One `store.replicate()` replicates all cores.
	687
	688	See: [holepunchto/corestore](https://github.com/holepunchto/corestore)
	689
	690	---
	691
	692	### Pattern 2: Hypercore + Hyperbee (Indexed Database)
	693
	694	When you need efficient key-value lookups:
	695
	696	```js
	697	const Hypercore = require('hypercore')
	698	const Hyperbee = require('hyperbee')
	699
	700	// Create append-only log
	701	const core = new Hypercore('./my-db')
	702
	703	// Wrap with B-tree index
	704	const db = new Hyperbee(core, {
	705	keyEncoding: 'utf-8',
	706	valueEncoding: 'json'
	707	})
	708
	709	// Now you have fast key-value access
	710	await db.put('user:1', { name: 'Alice', age: 30 })
	711	await db.put('user:2', { name: 'Bob', age: 25 })
	712
	713	const user = await db.get('user:1')
	714	console.log(user.value) // { name: 'Alice', age: 30 }
	715
	716	// Range queries
	717	for await (const { key, value } of db.createReadStream({
	718	gte: 'user:1',
	719	lte: 'user:999'
	720	})) {
	721	console.log(key, value)
	722	}
	723	```
	724
	725	Why: Hyperbee provides B-tree indexing on top of Hypercore's append-only log. You get the benefits of both.
	726
	727	See: [holepunchto/hyperbee](https://github.com/holepunchto/hyperbee)
	728
	729	---
	730
	731	### Pattern 3: Hypercore + Hyperswarm (P2P Discovery)
	732
	733	When you need peer discovery and connections:
	734
	735	```js
	736	const Hypercore = require('hypercore')
	737	const Hyperswarm = require('hyperswarm')
	738
	739	const core = new Hypercore('./my-log')
	740	await core.ready()
	741
	742	const swarm = new Hyperswarm()
	743
	744	// When peers connect, replicate
	745	swarm.on('connection', (socket) => {
	746	console.log('Peer connected!')
	747
	748	// Determine if we're initiator
	749	const isInitiator = !socket.remoteAddress.startsWith('::ffff:127.0.0.1')
	750
	751	socket.pipe(core.replicate(isInitiator)).pipe(socket)
	752	})
	753
	754	// Join swarm using discoveryKey
	755	const topic = core.discoveryKey
	756	swarm.join(topic, { server: true, client: true })
	757	await swarm.flush()
	758
	759	console.log('Joined swarm, waiting for peers...')
	760	```
	761
	762	Why: Hyperswarm handles peer discovery via DHT. You just replicate when connections happen.
	763
	764	See: [holepunchto/hyperswarm](https://github.com/holepunchto/hyperswarm)
	765
	766	---
	767
	768	### Pattern 4: Hypercore + Hyperdrive (File System)
	769
	770	When you need to store files instead of blocks:
	771
	772	```js
	773	const Hyperdrive = require('hyperdrive')
	774
	775	// Hyperdrive uses Hypercore internally
	776	const drive = new Hyperdrive('./my-drive')
	777	await drive.ready()
	778
	779	// File operations (stored in append-only log)
	780	await drive.put('/hello.txt', 'Hello, world!')
	781	await drive.put('/data/file.json', JSON.stringify({ foo: 'bar' }))
	782
	783	// Read files
	784	const content = await drive.get('/hello.txt')
	785	console.log(content.toString()) // 'Hello, world!'
	786
	787	// List directory
	788	for await (const entry of drive.list('/data')) {
	789	console.log(entry.key) // '/data/file.json'
	790	}
	791
	792	// Replicate entire file system
	793	swarm.on('connection', (socket) => {
	794	drive.replicate(socket)
	795	})
	796	```
	797
	798	Why: Hyperdrive provides POSIX-like file operations backed by Hypercore. Useful for distributing file collections.
	799
	800	See: [holepunchto/hyperdrive](https://github.com/holepunchto/hyperdrive)
	801
	802	---
	803
	804	### Pattern 5: Hypercore + Autobase (Multi-Writer)
	805
	806	When you need multiple writers (not just one):
	807
	808	```js
	809	const Autobase = require('autobase')
	810	const Hypercore = require('hypercore')
	811
	812	// Create multiple writer logs
	813	const writer1 = new Hypercore('./writer1')
	814	const writer2 = new Hypercore('./writer2')
	815
	816	await writer1.ready()
	817	await writer2.ready()
	818
	819	// Combine into single view
	820	const base = new Autobase([writer1, writer2], {
	821	apply: async (batch) => {
	822	// Apply operations from all writers
	823	for (const node of batch) {
	824	console.log('Operation from', node.value.writer, ':', node.value.op)
	825	}
	826	}
	827	})
	828
	829	await base.ready()
	830
	831	// Both can write independently
	832	await writer1.append(JSON.stringify({ writer: 1, op: 'insert', data: 'A' }))
	833	await writer2.append(JSON.stringify({ writer: 2, op: 'insert', data: 'B' }))
	834
	835	// Autobase merges them into causal order
	836	await base.update()
	837	```
	838
	839	Why: Hypercore is single-writer. Autobase enables multi-writer collaboration with CRDT-like semantics.
	840
	841	See: [holepunchto/autobase](https://github.com/holepunchto/autobase)
	842
	843	---
	844
	845	## See Also
	846
	847	### Pattern Documentation
	848	- [replication.md](./replication.md) - Deep dive on replication strategies
	849	- [cryptography.md](./cryptography.md) - Merkle trees and verification
	850	- [peer-discovery.md](./peer-discovery.md) - Finding peers with Hyperswarm
	851	- [streaming.md](./streaming.md) - Efficient data streaming patterns
	852
	853	### Deployment Documentation
	854	- [../deployment/pear-apps.md](../deployment/pear-apps.md) - Deploying Hypercore apps with Pear
	855	- [../deployment/systemd-services.md](../deployment/systemd-services.md) - Running as system service
	856	- [../deployment/module-resolution.md](../deployment/module-resolution.md) - Module resolution issues
	857
	858	### Repository Documentation
	859	- [../skills/hypercore/skills.md](../skills/hypercore/skills.md) - Hypercore API reference
	860	- [../skills/hyperbee/skills.md](../skills/hyperbee/skills.md) - Hyperbee (indexed DB)
	861	- [../skills/hyperswarm/skills.md](../skills/hyperswarm/skills.md) - Hyperswarm (P2P networking)
	862	- [../skills/corestore/skills.md](../skills/corestore/skills.md) - Corestore (multi-core management)
	863
	864	### Official Resources
	865	- [Hypercore GitHub](https://github.com/holepunchto/hypercore)
	866	- [Hypercore Docs](https://docs.pears.com/building-blocks/hypercore)
	867	- [Holepunch.to](https://holepunch.to/) - Ecosystem overview
	868
	869	---
	870
	871	## References
	872
	873	- Tier 1 Repo: holepunchto/repos/tier1/hypercore
	874	- Real-World Example: hyper-fleeky/repos/fedicore (P2P social network)
	875	- Pattern Category: Building Block (Tier 1)
	876	- Dependencies: hypercore-crypto, hypercore-storage, protomux, compact-encoding
	877	- Use Cases: Event sourcing, distributed databases, P2P messaging, version control, audit trails
	878
	879	---
	880
	881	Last Updated: 2026-01-18
	882	Validated Against: Hypercore 10.x, Fedicore 1.0

patterns/b-tree-index.md +1087 -0

…	…	@@ -0,0 +1,1087 @@
	1	# Pattern: B-Tree Index
	2
	3	Validated against: holepunchto/hyperbee
	4
	5	## Description
	6
	7	A B-tree index built on top of Hypercore's append-only log, providing fast key-value lookups, range queries, and sorted iteration while maintaining cryptographic verification and P2P replication.
	8
	9	Core properties:
	10	- Indexed access: O(log n) lookups instead of O(n) linear scans
	11	- Sorted keys: Keys are stored in sorted order
	12	- Range queries: Efficiently query key ranges (gte, lt, etc.)
	13	- Sparse replication: Only download tree nodes you need
	14	- Atomic batches: Multiple operations in single transaction
	15	- Append-only: Built on Hypercore, so still immutable history
	16
	17	Key insight: Hyperbee gives you a database on top of an append-only log. You get the benefits of both: verifiable history AND fast queries.
	18
	19	---
	20
	21	## When to Use
	22
	23	✅ Use Hyperbee when:
	24	- Need key-value storage with fast lookups
	25	- Want sorted iteration over keys
	26	- Need range queries (all users from A-M)
	27	- Building user profiles, settings, metadata
	28	- Want queryable database with P2P replication
	29	- Need atomic transactions (batches)
	30
	31	❌ Don't use Hyperbee when:
	32	- You only need sequential append (use Hypercore directly)
	33	- You need complex SQL queries (use Hyperbee + custom indexes)
	34	- Keys don't have natural sort order
	35	- You need to query by value (not key) - requires secondary indexes
	36
	37	---
	38
	39	## Implementing Repos
	40
	41	### Tier 1 (Core Building Blocks)
	42
	43	- [hyperbee](https://github.com/holepunchto/hyperbee) (642 ⭐)
	44	- B-tree database on Hypercore
	45	- Key-value storage with range queries
	46	- `npm install hyperbee`
	47
	48	### Built with Hyperbee
	49
	50	- [autobase](https://github.com/holepunchto/autobase) - Multi-writer databases using Hyperbee
	51	- [hypertrie](https://github.com/holepunchto/hypertrie) - Trie structure on Hyperbee
	52
	53	---
	54
	55	## Code Example: Basic CRUD Operations
	56
	57	Creating a Hyperbee database with basic operations:
	58
	59	```js
	60	const Hypercore = require('hypercore')
	61	const Hyperbee = require('hyperbee')
	62
	63	// Create Hypercore (the append-only log)
	64	const core = new Hypercore('./storage')
	65
	66	// Wrap with Hyperbee (the B-tree index)
	67	const db = new Hyperbee(core, {
	68	keyEncoding: 'utf-8', // Keys as strings
	69	valueEncoding: 'json' // Values as JSON objects
	70	})
	71
	72	await db.ready()
	73
	74	// ========================================
	75	// CREATE (put)
	76	// ========================================
	77
	78	await db.put('user:1', {
	79	name: 'Alice',
	80	email: 'alice@example.com',
	81	age: 30
	82	})
	83
	84	await db.put('user:2', {
	85	name: 'Bob',
	86	email: 'bob@example.com',
	87	age: 25
	88	})
	89
	90	await db.put('user:3', {
	91	name: 'Charlie',
	92	email: 'charlie@example.com',
	93	age: 35
	94	})
	95
	96	console.log('Created 3 users')
	97
	98	// ========================================
	99	// READ (get)
	100	// ========================================
	101
	102	const user1 = await db.get('user:1')
	103	console.log(user1)
	104	// {
	105	// seq: 0,
	106	// key: 'user:1',
	107	// value: { name: 'Alice', email: 'alice@example.com', age: 30 }
	108	// }
	109
	110	// Non-existent key returns null
	111	const missing = await db.get('user:999')
	112	console.log(missing) // null
	113
	114	// ========================================
	115	// UPDATE (put again)
	116	// ========================================
	117
	118	await db.put('user:1', {
	119	name: 'Alice',
	120	email: 'alice@newdomain.com', // Updated email
	121	age: 31 // Updated age
	122	})
	123
	124	// ========================================
	125	// DELETE (del)
	126	// ========================================
	127
	128	await db.del('user:3')
	129
	130	const deleted = await db.get('user:3')
	131	console.log(deleted) // null
	132	```
	133
	134	Key points:
	135	- `put` for both create and update
	136	- `get` returns `{ seq, key, value }` or `null`
	137	- `seq` is the Hypercore index where this was written
	138	- Encoding handles serialization (JSON stringify/parse automatic)
	139
	140	Source: holepunchto/hyperbee/README.md:11-48
	141
	142	---
	143
	144	## Code Example: Range Queries
	145
	146	Querying ranges of keys efficiently:
	147
	148	```js
	149	const db = new Hyperbee(core, {
	150	keyEncoding: 'utf-8',
	151	valueEncoding: 'json'
	152	})
	153
	154	// Insert test data
	155	await db.put('user:alice', { name: 'Alice', score: 100 })
	156	await db.put('user:bob', { name: 'Bob', score: 85 })
	157	await db.put('user:charlie', { name: 'Charlie', score: 92 })
	158	await db.put('user:david', { name: 'David', score: 88 })
	159	await db.put('user:eve', { name: 'Eve', score: 95 })
	160
	161	// ========================================
	162	// Range: gte (>=) and lt (<)
	163	// ========================================
	164
	165	console.log('Users from B to D (exclusive):')
	166	const stream1 = db.createReadStream({
	167	gte: 'user:b', // >= 'user:b'
	168	lt: 'user:d' // < 'user:d'
	169	})
	170
	171	for await (const { key, value } of stream1) {
	172	console.log(key, value)
	173	// user:bob { name: 'Bob', score: 85 }
	174	// user:charlie { name: 'Charlie', score: 92 }
	175	}
	176
	177	// ========================================
	178	// Range: All keys starting with prefix
	179	// ========================================
	180
	181	console.log('All users:')
	182	const stream2 = db.createReadStream({
	183	gte: 'user:',
	184	lte: 'user:\xFF' // End of 'user:' prefix
	185	})
	186
	187	for await (const { key, value } of stream2) {
	188	console.log(key, value.name)
	189	}
	190
	191	// ========================================
	192	// Reverse iteration
	193	// ========================================
	194
	195	console.log('Users in reverse order:')
	196	const stream3 = db.createReadStream({
	197	reverse: true
	198	})
	199
	200	for await (const { key, value } of stream3) {
	201	console.log(key, value.name)
	202	// user:eve Eve
	203	// user:david David
	204	// user:charlie Charlie
	205	// user:bob Bob
	206	// user:alice Alice
	207	}
	208
	209	// ========================================
	210	// Limit results
	211	// ========================================
	212
	213	console.log('Top 2 users:')
	214	const stream4 = db.createReadStream({
	215	limit: 2
	216	})
	217
	218	for await (const { key, value } of stream4) {
	219	console.log(key, value.name)
	220	// user:alice Alice
	221	// user:bob Bob
	222	}
	223
	224	// ========================================
	225	// Peek (get first matching entry)
	226	// ========================================
	227
	228	const first = await db.peek({
	229	gte: 'user:c'
	230	})
	231
	232	console.log('First user >= c:', first.value.name) // Charlie
	233	```
	234
	235	Range operators:
	236	- `gt`: Greater than (exclusive)
	237	- `gte`: Greater than or equal (inclusive)
	238	- `lt`: Less than (exclusive)
	239	- `lte`: Less than or equal (inclusive)
	240
	241	Source: holepunchto/hyperbee/README.md:233-257
	242
	243	---
	244
	245	## Code Example: Atomic Batches
	246
	247	Multiple operations in a single atomic transaction:
	248
	249	```js
	250	const db = new Hyperbee(core, {
	251	keyEncoding: 'utf-8',
	252	valueEncoding: 'json'
	253	})
	254
	255	// ========================================
	256	// Atomic Batch - All or Nothing
	257	// ========================================
	258
	259	const batch = db.batch()
	260
	261	try {
	262	// Queue multiple operations
	263	await batch.put('account:1', { balance: 1000 })
	264	await batch.put('account:2', { balance: 500 })
	265
	266	// Transfer money
	267	await batch.put('account:1', { balance: 900 }) // -100
	268	await batch.put('account:2', { balance: 600 }) // +100
	269	await batch.put('transfer:1', {
	270	from: 'account:1',
	271	to: 'account:2',
	272	amount: 100,
	273	timestamp: Date.now()
	274	})
	275
	276	// Commit - all operations happen together
	277	await batch.flush()
	278
	279	console.log('Transfer committed atomically')
	280	} catch (err) {
	281	// If any operation fails, abort the batch
	282	await batch.close() // Abort without flushing
	283	console.error('Transfer failed:', err)
	284	}
	285
	286	// ========================================
	287	// Read from Batch (before flush)
	288	// ========================================
	289
	290	const batch2 = db.batch()
	291
	292	await batch2.put('temp:1', { value: 'test' })
	293
	294	// Can read from batch before flushing
	295	const result = await batch2.get('temp:1')
	296	console.log(result.value) // { value: 'test' }
	297
	298	// Not visible in main db yet
	299	const notFound = await db.get('temp:1')
	300	console.log(notFound) // null
	301
	302	// Flush to make visible
	303	await batch2.flush()
	304
	305	const nowFound = await db.get('temp:1')
	306	console.log(nowFound.value) // { value: 'test' }
	307	```
	308
	309	Batch benefits:
	310	- Atomicity: All operations succeed or all fail
	311	- Performance: Much faster than individual puts
	312	- Consistency: Read your own writes within batch
	313
	314	Source: holepunchto/hyperbee/README.md:201-229
	315
	316	---
	317
	318	## Code Example: Compare-And-Swap (CAS)
	319
	320	Conditional updates based on current value:
	321
	322	```js
	323	const db = new Hyperbee(core, {
	324	keyEncoding: 'utf-8',
	325	valueEncoding: 'json'
	326	})
	327
	328	await db.put('counter', { value: 0 })
	329
	330	// ========================================
	331	// CAS for Put - Only update if value changed
	332	// ========================================
	333
	334	function casOnlyIfDifferent(prev, next) {
	335	// Don't update if values are the same
	336	return !prev \|\| prev.value.value !== next.value
	337	}
	338
	339	// First put succeeds (0 -> 1)
	340	await db.put('counter', { value: 1 }, { cas: casOnlyIfDifferent })
	341	console.log((await db.get('counter')).value) // { value: 1 }
	342
	343	// Second put with same value fails (no-op)
	344	await db.put('counter', { value: 1 }, { cas: casOnlyIfDifferent })
	345	console.log((await db.get('counter')).seq) // Still seq from first put
	346
	347	// Third put with different value succeeds (1 -> 2)
	348	await db.put('counter', { value: 2 }, { cas: casOnlyIfDifferent })
	349	console.log((await db.get('counter')).value) // { value: 2 }
	350
	351	// ========================================
	352	// CAS for Delete - Conditional deletion
	353	// ========================================
	354
	355	await db.put('status', { state: 'pending' })
	356
	357	function casDeleteIfDone(prev) {
	358	// Only delete if status is 'done'
	359	return prev && prev.value.state === 'done'
	360	}
	361
	362	// Try to delete while pending - fails
	363	await db.del('status', { cas: casDeleteIfDone })
	364	console.log(await db.get('status')) // Still exists
	365
	366	// Update to done
	367	await db.put('status', { state: 'done' })
	368
	369	// Now delete succeeds
	370	await db.del('status', { cas: casDeleteIfDone })
	371	console.log(await db.get('status')) // null
	372	```
	373
	374	CAS use cases:
	375	- Optimistic locking
	376	- Increment counters safely
	377	- Conditional state transitions
	378	- Prevent duplicate writes
	379
	380	Source: holepunchto/hyperbee/README.md:124-189
	381
	382	---
	383
	384	## Code Example: Snapshots and History
	385
	386	Reading historical versions and change streams:
	387
	388	```js
	389	const db = new Hyperbee(core, {
	390	keyEncoding: 'utf-8',
	391	valueEncoding: 'utf-8'
	392	})
	393
	394	// ========================================
	395	// Build up history
	396	// ========================================
	397
	398	await db.put('doc', 'version 1')
	399	const v1 = db.version
	400
	401	await db.put('doc', 'version 2')
	402	const v2 = db.version
	403
	404	await db.put('doc', 'version 3')
	405	const v3 = db.version
	406
	407	console.log('Current version:', db.version) // 3
	408
	409	// ========================================
	410	// Checkout (read-only snapshot)
	411	// ========================================
	412
	413	const snapshot1 = db.checkout(v1)
	414	console.log((await snapshot1.get('doc')).value) // 'version 1'
	415
	416	const snapshot2 = db.checkout(v2)
	417	console.log((await snapshot2.get('doc')).value) // 'version 2'
	418
	419	const current = db.snapshot() // Current version
	420	console.log((await current.get('doc')).value) // 'version 3'
	421
	422	// ========================================
	423	// History Stream
	424	// ========================================
	425
	426	console.log('Full history:')
	427	const history = db.createHistoryStream()
	428
	429	for await (const entry of history) {
	430	console.log(entry.type, entry.key, entry.value)
	431	// put doc version 1
	432	// put doc version 2
	433	// put doc version 3
	434	}
	435
	436	// ========================================
	437	// History in Reverse
	438	// ========================================
	439
	440	console.log('Recent changes:')
	441	const recent = db.createHistoryStream({
	442	reverse: true,
	443	limit: 2
	444	})
	445
	446	for await (const entry of recent) {
	447	console.log(entry.seq, entry.type, entry.key)
	448	// Latest two operations
	449	}
	450
	451	// ========================================
	452	// Diff Between Versions
	453	// ========================================
	454
	455	const v1Snapshot = db.checkout(v1)
	456	const v3Snapshot = db.checkout(v3)
	457
	458	const diff = v3Snapshot.createDiffStream(v1Snapshot)
	459
	460	for await (const entry of diff) {
	461	console.log('Left (v3):', entry.left?.value)
	462	console.log('Right (v1):', entry.right?.value)
	463	}
	464	```
	465
	466	Historical features:
	467	- Snapshots: Read-only views of past versions
	468	- History stream: All operations ever performed
	469	- Diffs: Changes between versions
	470	- Time travel: Query data as it was
	471
	472	Source: holepunchto/hyperbee/README.md:263-304, 351-399
	473
	474	---
	475
	476	## Code Example: Sub-databases (Namespaces)
	477
	478	Creating isolated namespaces within a single Hyperbee:
	479
	480	```js
	481	const Hyperbee = require('hyperbee')
	482	const Hypercore = require('hypercore')
	483
	484	const core = new Hypercore('./storage')
	485	const root = new Hyperbee(core, {
	486	keyEncoding: 'utf-8',
	487	valueEncoding: 'json'
	488	})
	489
	490	await root.ready()
	491
	492	// ========================================
	493	// Create Sub-databases
	494	// ========================================
	495
	496	const users = root.sub('users')
	497	const posts = root.sub('posts')
	498	const comments = root.sub('comments')
	499
	500	// Each sub has its own namespace
	501
	502	await users.put('alice', { email: 'alice@example.com' })
	503	await users.put('bob', { email: 'bob@example.com' })
	504
	505	await posts.put('post-1', { title: 'First post', author: 'alice' })
	506	await posts.put('post-2', { title: 'Second post', author: 'bob' })
	507
	508	await comments.put('comment-1', { post: 'post-1', text: 'Great!' })
	509
	510	// ========================================
	511	// Query Within Namespace
	512	// ========================================
	513
	514	// Only sees users
	515	for await (const { key, value } of users.createReadStream()) {
	516	console.log('User:', key, value)
	517	// User: alice { email: 'alice@example.com' }
	518	// User: bob { email: 'bob@example.com' }
	519	}
	520
	521	// Only sees posts
	522	for await (const { key, value } of posts.createReadStream()) {
	523	console.log('Post:', key, value)
	524	// Post: post-1 { title: 'First post', author: 'alice' }
	525	// Post: post-2 { title: 'Second post', author: 'bob' }
	526	}
	527
	528	// ========================================
	529	// Different Encodings Per Sub
	530	// ========================================
	531
	532	const metrics = root.sub('metrics', {
	533	keyEncoding: 'utf-8',
	534	valueEncoding: 'binary' // Different encoding!
	535	})
	536
	537	await metrics.put('cpu', Buffer.from([0x01, 0x02, 0x03]))
	538
	539	// ========================================
	540	// Nested Subs
	541	// ========================================
	542
	543	const usersProfiles = users.sub('profiles')
	544	const usersSettings = users.sub('settings')
	545
	546	await usersProfiles.put('alice', { bio: 'Software engineer' })
	547	await usersSettings.put('alice', { theme: 'dark' })
	548
	549	console.log(await usersProfiles.get('alice'))
	550	// { bio: 'Software engineer' }
	551	```
	552
	553	Sub-database benefits:
	554	- Namespace isolation: Separate data domains
	555	- Different encodings: Per-namespace serialization
	556	- Efficient: All in one Hypercore, one replication
	557	- Organization: Clean data model
	558
	559	Source: holepunchto/hyperbee/README.md:359-386
	560
	561	---
	562
	563	## Real-World Usage
	564
	565	### Example: User Profile Database (Conceptual)
	566
	567	While not yet implemented in hyper-fleeky projects, here's how Hyperbee would be used:
	568
	569	```js
	570	// Conceptual pattern for user profiles
	571	const Hyperbee = require('hyperbee')
	572	const Hypercore = require('hypercore')
	573	const Corestore = require('corestore')
	574
	575	class ProfileDatabase {
	576	constructor(storage) {
	577	this.store = new Corestore(storage)
	578	}
	579
	580	async init() {
	581	// One Hypercore for all profile data
	582	const core = this.store.get({ name: 'profiles' })
	583	await core.ready()
	584
	585	// Hyperbee for indexed access
	586	this.db = new Hyperbee(core, {
	587	keyEncoding: 'utf-8',
	588	valueEncoding: 'json'
	589	})
	590
	591	await this.db.ready()
	592
	593	// Sub-databases for different data types
	594	this.profiles = this.db.sub('profiles')
	595	this.settings = this.db.sub('settings')
	596	this.sessions = this.db.sub('sessions')
	597	}
	598
	599	async createProfile(userId, data) {
	600	await this.profiles.put(userId, {
	601	...data,
	602	createdAt: Date.now()
	603	})
	604	}
	605
	606	async getProfile(userId) {
	607	const entry = await this.profiles.get(userId)
	608	return entry?.value
	609	}
	610
	611	async updateProfile(userId, updates) {
	612	const current = await this.getProfile(userId)
	613	if (!current) throw new Error('Profile not found')
	614
	615	await this.profiles.put(userId, {
	616	...current,
	617	...updates,
	618	updatedAt: Date.now()
	619	})
	620	}
	621
	622	async deleteProfile(userId) {
	623	await this.profiles.del(userId)
	624	}
	625
	626	async findProfiles(prefix) {
	627	const results = []
	628	const stream = this.profiles.createReadStream({
	629	gte: prefix,
	630	lte: prefix + '\xFF'
	631	})
	632
	633	for await (const { key, value } of stream) {
	634	results.push({ id: key, ...value })
	635	}
	636
	637	return results
	638	}
	639
	640	async listAllProfiles() {
	641	const profiles = []
	642	for await (const { key, value } of this.profiles.createReadStream()) {
	643	profiles.push({ id: key, ...value })
	644	}
	645	return profiles
	646	}
	647	}
	648
	649	// Usage
	650	const profileDB = new ProfileDatabase('./profiles')
	651	await profileDB.init()
	652
	653	await profileDB.createProfile('user-1', {
	654	name: 'Alice',
	655	email: 'alice@example.com'
	656	})
	657
	658	const profile = await profileDB.getProfile('user-1')
	659	console.log(profile)
	660
	661	// Find all users starting with 'user-'
	662	const users = await profileDB.findProfiles('user-')
	663	console.log('Total users:', users.length)
	664	```
	665
	666	Pattern for future hyper-fleeky projects: This is the recommended pattern for any project needing user profiles, settings, or structured data storage.
	667
	668	---
	669
	670	## Common Gotchas
	671
	672	### 1. Creating Hyperbee without Hypercore
	673
	674	❌ Wrong:
	675	```js
	676	const db = new Hyperbee('./storage') // No! Hyperbee doesn't create core
	677	```
	678
	679	✅ Correct:
	680	```js
	681	const core = new Hypercore('./storage')
	682	const db = new Hyperbee(core) // Pass existing core
	683	```
	684
	685	Why: Hyperbee wraps a Hypercore, it doesn't create one.
	686
	687	---
	688
	689	### 2. Appending to Hypercore directly
	690
	691	❌ Wrong:
	692	```js
	693	const core = new Hypercore('./storage')
	694	const db = new Hyperbee(core)
	695
	696	await core.append('data') // ❌ Breaks B-tree!
	697	await db.put('key', 'value') // DB now corrupted
	698	```
	699
	700	✅ Correct:
	701	```js
	702	const core = new Hypercore('./storage')
	703	const db = new Hyperbee(core)
	704
	705	// Only use Hyperbee API
	706	await db.put('key', 'value') // ✓
	707	```
	708
	709	Why: Hyperbee manages Hypercore internally. Direct appends corrupt the B-tree structure.
	710
	711	---
	712
	713	### 3. Forgetting to flush batches
	714
	715	❌ Wrong:
	716	```js
	717	const batch = db.batch()
	718	await batch.put('key', 'value')
	719	// Forgot to flush!
	720	// Changes never committed
	721	```
	722
	723	✅ Correct:
	724	```js
	725	const batch = db.batch()
	726	await batch.put('key', 'value')
	727	await batch.flush() // Commit
	728	```
	729
	730	Why: Batches are only written on `flush()`. Forgetting it loses all changes.
	731
	732	---
	733
	734	### 4. Not closing aborted batches
	735
	736	❌ Wrong:
	737	```js
	738	const batch = db.batch()
	739	await batch.put('key', 'value')
	740
	741	if (error) {
	742	return // ❌ Batch locks not released!
	743	}
	744	```
	745
	746	✅ Correct:
	747	```js
	748	const batch = db.batch()
	749
	750	try {
	751	await batch.put('key', 'value')
	752	await batch.flush()
	753	} catch (err) {
	754	await batch.close() // Release locks
	755	throw err
	756	}
	757	```
	758
	759	Why: Batches acquire locks. Must `close()` even if not flushing.
	760
	761	---
	762
	763	### 5. Encoding mismatches
	764
	765	❌ Wrong:
	766	```js
	767	const db = new Hyperbee(core, {
	768	valueEncoding: 'json'
	769	})
	770
	771	// Put binary data
	772	await db.put('key', Buffer.from([0x01, 0x02])) // Tries to JSON.stringify buffer!
	773	```
	774
	775	✅ Correct:
	776	```js
	777	const db = new Hyperbee(core, {
	778	valueEncoding: 'binary'
	779	})
	780
	781	await db.put('key', Buffer.from([0x01, 0x02])) // Works
	782	```
	783
	784	Why: Encoding must match your data type. JSON encoding can't handle raw buffers.
	785
	786	---
	787
	788	### 6. Expecting instant replication visibility
	789
	790	❌ Wrong:
	791	```js
	792	// Peer 1
	793	await db.put('key', 'value')
	794
	795	// Peer 2 (immediately after)
	796	const value = await db.get('key') // null! Not synced yet
	797	```
	798
	799	✅ Correct:
	800	```js
	801	// Peer 2
	802	await db.core.update() // Wait for sync
	803	const value = await db.get('key') // Now it exists
	804	```
	805
	806	Why: Replication is async. Use `core.update()` or wait for events.
	807
	808	---
	809
	810	## Anti-Patterns
	811
	812	### ❌ 1. Using Hypercore when you need queries
	813
	814	Don't:
	815	```js
	816	// Scanning entire log for one user
	817	const core = new Hypercore('./storage')
	818
	819	for (let i = 0; i < core.length; i++) {
	820	const block = await core.get(i)
	821	const data = JSON.parse(block)
	822	if (data.userId === 'alice') {
	823	return data // O(n) lookup!
	824	}
	825	}
	826	```
	827
	828	Do: Use Hyperbee
	829	```js
	830	const db = new Hyperbee(core)
	831	const entry = await db.get('user:alice') // O(log n) lookup!
	832	```
	833
	834	Why: Linear scans are slow. Hyperbee provides indexed access.
	835
	836	---
	837
	838	### ❌ 2. Creating separate Hyperbee for each namespace
	839
	840	Don't:
	841	```js
	842	// Separate Hyperbees (wasteful)
	843	const usersCore = new Hypercore('./users')
	844	const usersDB = new Hyperbee(usersCore)
	845
	846	const postsCore = new Hypercore('./posts')
	847	const postsDB = new Hyperbee(postsCore)
	848
	849	// Two cores to replicate!
	850	```
	851
	852	Do: Use sub-databases
	853	```js
	854	const core = new Hypercore('./storage')
	855	const db = new Hyperbee(core)
	856
	857	const users = db.sub('users')
	858	const posts = db.sub('posts')
	859
	860	// One core, efficient replication
	861	```
	862
	863	Why: Sub-databases give you namespaces without multiple cores.
	864
	865	---
	866
	867	### ❌ 3. Not using batches for bulk operations
	868
	869	Don't:
	870	```js
	871	// Slow: Each put waits for I/O
	872	for (const user of users) {
	873	await db.put(user.id, user.data) // Sequential writes
	874	}
	875	```
	876
	877	Do: Use batches
	878	```js
	879	const batch = db.batch()
	880
	881	for (const user of users) {
	882	await batch.put(user.id, user.data) // Queued
	883	}
	884
	885	await batch.flush() // One I/O operation
	886	```
	887
	888	Why: Batches are much faster for bulk operations.
	889
	890	---
	891
	892	### ❌ 4. Storing large blobs as values
	893
	894	Don't:
	895	```js
	896	// Storing 10MB image in Hyperbee
	897	await db.put('image:1', hugeImageBuffer) // Bloats B-tree nodes!
	898	```
	899
	900	Do: Store large data in Hypercore/Hyperdrive, reference in Hyperbee
	901	```js
	902	// Store large data elsewhere
	903	await drive.put('/images/1.jpg', hugeImageBuffer)
	904
	905	// Store reference in Hyperbee
	906	await db.put('image:1', {
	907	path: '/images/1.jpg',
	908	size: hugeImageBuffer.length,
	909	hash: hashOfImage
	910	})
	911	```
	912
	913	Why: B-tree nodes have size limits. Large values degrade performance.
	914
	915	---
	916
	917	## Integration Patterns
	918
	919	### Pattern 1: Hyperbee + Hyperswarm (Replicated Database)
	920
	921	```js
	922	const Hypercore = require('hypercore')
	923	const Hyperbee = require('hyperbee')
	924	const Hyperswarm = require('hyperswarm')
	925
	926	async function createReplicatedDB(storage, key = null) {
	927	const core = new Hypercore(storage, key)
	928	const db = new Hyperbee(core, {
	929	keyEncoding: 'utf-8',
	930	valueEncoding: 'json'
	931	})
	932
	933	await db.ready()
	934
	935	const swarm = new Hyperswarm()
	936
	937	swarm.on('connection', (socket, peerInfo) => {
	938	// Replicate underlying Hypercore
	939	core.replicate(peerInfo.client).pipe(socket).pipe(socket)
	940	})
	941
	942	swarm.join(core.discoveryKey)
	943	await swarm.flush()
	944
	945	return { db, core, swarm }
	946	}
	947
	948	// Writer
	949	const { db: writerDB } = await createReplicatedDB('./writer')
	950	await writerDB.put('users:alice', { name: 'Alice' })
	951
	952	// Reader
	953	const { db: readerDB } = await createReplicatedDB('./reader', writerDB.core.key)
	954	await readerDB.core.update()
	955	const alice = await readerDB.get('users:alice')
	956	console.log(alice.value) // { name: 'Alice' }
	957	```
	958
	959	---
	960
	961	### Pattern 2: Hyperbee + Corestore (Multiple Databases)
	962
	963	```js
	964	const Corestore = require('corestore')
	965	const Hyperbee = require('hyperbee')
	966
	967	class DatabaseManager {
	968	constructor(storage) {
	969	this.store = new Corestore(storage)
	970	this.databases = new Map()
	971	}
	972
	973	async createDatabase(name) {
	974	const core = this.store.get({ name })
	975	const db = new Hyperbee(core, {
	976	keyEncoding: 'utf-8',
	977	valueEncoding: 'json'
	978	})
	979
	980	await db.ready()
	981	this.databases.set(name, db)
	982
	983	return db
	984	}
	985
	986	replicate(socket) {
	987	return this.store.replicate(socket)
	988	}
	989	}
	990
	991	// Usage
	992	const manager = new DatabaseManager('./storage')
	993
	994	const usersDB = await manager.createDatabase('users')
	995	const postsDB = await manager.createDatabase('posts')
	996
	997	// All databases replicate together
	998	swarm.on('connection', (socket) => {
	999	manager.replicate(socket)
	1000	})
	1001	```
	1002
	1003	---
	1004
	1005	### Pattern 3: Hyperbee + Autobase (Multi-Writer)
	1006
	1007	```js
	1008	const Autobase = require('autobase')
	1009	const Hyperbee = require('hyperbee')
	1010
	1011	// Create multiple writer inputs
	1012	const writer1 = new Hypercore('./writer1')
	1013	const writer2 = new Hypercore('./writer2')
	1014
	1015	// Merge into single Hyperbee view
	1016	const base = new Autobase([writer1, writer2], {
	1017	apply: async (batch, view, base) => {
	1018	const db = new Hyperbee(view, {
	1019	keyEncoding: 'utf-8',
	1020	valueEncoding: 'json'
	1021	})
	1022
	1023	for (const node of batch) {
	1024	const op = JSON.parse(node.value)
	1025	if (op.type === 'put') {
	1026	await db.put(op.key, op.value)
	1027	} else if (op.type === 'del') {
	1028	await db.del(op.key)
	1029	}
	1030	}
	1031	}
	1032	})
	1033
	1034	await base.ready()
	1035
	1036	// Multiple writers can write independently
	1037	await writer1.append(JSON.stringify({
	1038	type: 'put',
	1039	key: 'user:1',
	1040	value: { name: 'Alice' }
	1041	}))
	1042
	1043	await writer2.append(JSON.stringify({
	1044	type: 'put',
	1045	key: 'user:2',
	1046	value: { name: 'Bob' }
	1047	}))
	1048
	1049	// Autobase merges into single Hyperbee view
	1050	```
	1051
	1052	---
	1053
	1054	## See Also
	1055
	1056	### Pattern Documentation
	1057	- [append-only-log.md](./append-only-log.md) - Hypercore basics (Hyperbee's foundation)
	1058	- [replication.md](./replication.md) - Replicating databases
	1059	- [cryptography.md](./cryptography.md) - Data verification
	1060
	1061	### Deployment Documentation
	1062	- [../deployment/pear-apps.md](../deployment/pear-apps.md) - Deploying database apps
	1063	- [../deployment/systemd-services.md](../deployment/systemd-services.md) - Running DB as service
	1064
	1065	### Repository Documentation
	1066	- [../skills/hyperbee/skills.md](../skills/hyperbee/skills.md) - Full Hyperbee API
	1067	- [../skills/hypercore/skills.md](../skills/hypercore/skills.md) - Hypercore API
	1068	- [../skills/autobase/skills.md](../skills/autobase/skills.md) - Multi-writer patterns
	1069
	1070	### Official Resources
	1071	- [Hyperbee GitHub](https://github.com/holepunchto/hyperbee)
	1072	- [Hyperbee Docs](https://docs.pears.com/building-blocks/hyperbee)
	1073	- [Holepunch.to](https://holepunch.to/)
	1074
	1075	---
	1076
	1077	## References
	1078
	1079	- Tier 1 Repo: holepunchto/repos/tier1/hyperbee
	1080	- Pattern Category: Data Structure (Indexed Database)
	1081	- Dependencies: hypercore (storage), b-tree algorithm
	1082	- Use Cases: User profiles, settings, metadata, queryable databases, sorted data
	1083
	1084	---
	1085
	1086	Last Updated: 2026-01-18
	1087	Validated Against: Hyperbee 2.x

patterns/binary-encoding.md +99 -0

…	…	@@ -0,0 +1,99 @@
	1	# Pattern: Binary Encoding
	2
	3	Validated against: holepunchto/compact-encoding
	4
	5	## Description
	6
	7	Efficient binary serialization for wire protocols and storage. Compact-encoding provides small, fast encoders for common data types used across the Hypercore ecosystem.
	8
	9	Key benefits:
	10	- Compact: Minimal wire format
	11	- Fast: Optimized for performance
	12	- Type-safe: Schema-based encoding
	13	- Extensible: Custom encodings
	14
	15	---
	16
	17	## Code Example: Basic Encoding
	18
	19	```js
	20	const c = require('compact-encoding')
	21
	22	// ========================================
	23	// Primitive Types
	24	// ========================================
	25
	26	// String
	27	const strBuffer = c.encode(c.string, 'hello')
	28	const str = c.decode(c.string, strBuffer) // 'hello'
	29
	30	// Number (uint)
	31	const numBuffer = c.encode(c.uint, 42)
	32	const num = c.decode(c.uint, numBuffer) // 42
	33
	34	// Boolean
	35	const boolBuffer = c.encode(c.bool, true)
	36	const bool = c.decode(c.bool, boolBuffer) // true
	37
	38	// Buffer
	39	const dataBuffer = c.encode(c.buffer, Buffer.from('data'))
	40	const data = c.decode(c.buffer, dataBuffer)
	41
	42	// ========================================
	43	// Arrays
	44	// ========================================
	45
	46	const arr = [1, 2, 3, 4, 5]
	47	const arrBuffer = c.encode(c.array(c.uint), arr)
	48	const decoded = c.decode(c.array(c.uint), arrBuffer) // [1, 2, 3, 4, 5]
	49
	50	// ========================================
	51	// Custom Structs
	52	// ========================================
	53
	54	const userEncoding = {
	55	preencode(state, user) {
	56	c.string.preencode(state, user.name)
	57	c.uint.preencode(state, user.age)
	58	},
	59	encode(state, user) {
	60	c.string.encode(state, user.name)
	61	c.uint.encode(state, user.age)
	62	},
	63	decode(state) {
	64	return {
	65	name: c.string.decode(state),
	66	age: c.uint.decode(state)
	67	}
	68	}
	69	}
	70
	71	const user = { name: 'Alice', age: 30 }
	72	const userBuffer = c.encode(userEncoding, user)
	73	const decodedUser = c.decode(userEncoding, userBuffer)
	74	// { name: 'Alice', age: 30 }
	75
	76	// ========================================
	77	// Used in Hypercore
	78	// ========================================
	79
	80	// Hypercore uses compact-encoding for:
	81	// - Block encoding
	82	// - Merkle tree nodes
	83	// - Protocol messages
	84	const Hypercore = require('hypercore')
	85
	86	const core = new Hypercore('./storage', {
	87	valueEncoding: userEncoding // Custom encoding!
	88	})
	89
	90	await core.append({ name: 'Bob', age: 25 })
	91	const entry = await core.get(0) // { name: 'Bob', age: 25 }
	92	```
	93
	94	See: Hypercore valueEncoding in [append-only-log.md](./append-only-log.md#code-example-value-encoding)
	95
	96	---
	97
	98	Source: holepunchto/compact-encoding
	99	Last Updated: 2026-01-18

patterns/cryptography.md +942 -0

…	…	@@ -0,0 +1,942 @@
	1	# Pattern: Cryptography
	2
	3	Validated against: holepunchto/hypercore-crypto, holepunchto/hypercore, hyper-fleeky/hyperident
	4
	5	## Description
	6
	7	Cryptographic primitives used across the Hypercore ecosystem for data integrity, authentication, and privacy. Includes Merkle trees for verification, ED25519 for signing, discovery key derivation, and block encryption.
	8
	9	Core components:
	10	- Merkle trees: Cryptographic verification of append-only logs
	11	- ED25519: Public key signatures for authentication
	12	- Discovery keys: Safe peer discovery without leaking data keys
	13	- Block encryption: Optional content encryption
	14	- Hashing: SHA-256 and BLAKE2b for data integrity
	15
	16	Key insight: Hypercore uses cryptographic verification at every layer. You can't trust data from peers - you verify it using merkle proofs and signatures.
	17
	18	---
	19
	20	## When to Use
	21
	22	✅ Use these cryptographic patterns when:
	23	- Building verifiable data structures
	24	- Implementing P2P systems (trust no peer)
	25	- Creating authentication systems
	26	- Storing sensitive data
	27	- Building decentralized applications
	28	- Ensuring data integrity
	29	- Implementing access control
	30
	31	❌ Don't implement custom crypto when:
	32	- You're not a cryptography expert (use provided primitives)
	33	- The ecosystem already provides what you need
	34	- Regulatory compliance requires specific algorithms
	35
	36	---
	37
	38	## Implementing Repos
	39
	40	### Tier 3 (Helpers)
	41
	42	- [hypercore-crypto](https://github.com/holepunchto/hypercore-crypto) (126 ⭐)
	43	- Core cryptographic primitives
	44	- ED25519, Merkle trees, hashing
	45	- `npm install hypercore-crypto`
	46
	47	### Built on hypercore-crypto
	48
	49	- [hypercore](https://github.com/holepunchto/hypercore) - Uses for verification
	50	- [keet-identity-key](https://github.com/holepunchto/keet-identity-key) - Identity management
	51	- [bare-crypto](https://github.com/holepunchto/bare-crypto) - Low-level crypto for Bare
	52
	53	---
	54
	55	## Code Example: ED25519 Key Pairs
	56
	57	Generating and using signing keys:
	58
	59	```js
	60	const crypto = require('hypercore-crypto')
	61
	62	// ========================================
	63	// Generate Key Pair
	64	// ========================================
	65
	66	const keyPair = crypto.keyPair()
	67
	68	console.log('Public key:', keyPair.publicKey.toString('hex'))
	69	console.log('Secret key:', keyPair.secretKey.toString('hex'))
	70
	71	// Public key: 64 hex characters (32 bytes)
	72	// Secret key: 128 hex characters (64 bytes)
	73
	74	// ========================================
	75	// Sign Data
	76	// ========================================
	77
	78	const message = Buffer.from('Hello, P2P world!')
	79
	80	const signature = crypto.sign(message, keyPair.secretKey)
	81
	82	console.log('Signature:', signature.toString('hex'))
	83	// Signature: 128 hex characters (64 bytes)
	84
	85	// ========================================
	86	// Verify Signature
	87	// ========================================
	88
	89	const isValid = crypto.verify(
	90	message,
	91	signature,
	92	keyPair.publicKey
	93	)
	94
	95	console.log('Valid signature:', isValid) // true
	96
	97	// Try tampering
	98	const tamperedMessage = Buffer.from('Hello, P2P hacker!')
	99	const isTampered = crypto.verify(
	100	tamperedMessage,
	101	signature,
	102	keyPair.publicKey
	103	)
	104
	105	console.log('Tampered valid:', isTampered) // false
	106
	107	// ========================================
	108	// Deterministic Key Generation
	109	// ========================================
	110
	111	const seed = Buffer.alloc(32).fill('my-seed')
	112	const deterministicKeyPair = crypto.keyPair(seed)
	113
	114	// Same seed = same keypair (useful for recovery)
	115	const sameKeyPair = crypto.keyPair(seed)
	116
	117	console.log('Keys match:',
	118	deterministicKeyPair.publicKey.equals(sameKeyPair.publicKey)
	119	) // true
	120	```
	121
	122	Key properties:
	123	- Public key: 32 bytes, can share publicly
	124	- Secret key: 64 bytes, must keep private
	125	- Signature: 64 bytes, proves message authenticity
	126	- Deterministic: Same seed always generates same keypair
	127
	128	Source: holepunchto/hypercore-crypto/README.md:10-30
	129
	130	---
	131
	132	## Code Example: Discovery Key Derivation
	133
	134	Creating safe discovery keys from public keys:
	135
	136	```js
	137	const crypto = require('hypercore-crypto')
	138
	139	const keyPair = crypto.keyPair()
	140
	141	// ========================================
	142	// Discovery Key
	143	// ========================================
	144
	145	const discoveryKey = crypto.discoveryKey(keyPair.publicKey)
	146
	147	console.log('Public key:', keyPair.publicKey.toString('hex'))
	148	console.log('Discovery key:', discoveryKey.toString('hex'))
	149
	150	// Discovery key is BLAKE2b hash of public key
	151
	152	// ========================================
	153	// Why Use Discovery Key?
	154	// ========================================
	155
	156	// ❌ Don't use public key for DHT announcements
	157	// swarm.join(core.key) // Leaks your data key!
	158
	159	// ✅ Use discovery key instead
	160	// swarm.join(core.discoveryKey) // Safe!
	161
	162	// Discovery key properties:
	163	// 1. Derived from public key (one-way hash)
	164	// 2. Can't reverse to get public key
	165	// 3. Allows peer discovery without revealing data key
	166	// 4. Still unique per core
	167
	168	// ========================================
	169	// Same Discovery Key for Same Public Key
	170	// ========================================
	171
	172	const key1 = crypto.keyPair()
	173	const key2 = crypto.keyPair()
	174
	175	const dk1a = crypto.discoveryKey(key1.publicKey)
	176	const dk1b = crypto.discoveryKey(key1.publicKey)
	177
	178	const dk2 = crypto.discoveryKey(key2.publicKey)
	179
	180	console.log('Same key, same discovery:', dk1a.equals(dk1b)) // true
	181	console.log('Different keys:', dk1a.equals(dk2)) // false
	182	```
	183
	184	Discovery key benefits:
	185	- Privacy: Don't leak the actual data key
	186	- Discovery: Peers can still find each other
	187	- Deterministic: Same key always produces same discovery key
	188	- One-way: Can't reverse discovery key to get public key
	189
	190	Source: holepunchto/hypercore-crypto/README.md:56-59
	191
	192	---
	193
	194	## Code Example: Merkle Tree Hashing
	195
	196	Hashing data and tree nodes for verification:
	197
	198	```js
	199	const crypto = require('hypercore-crypto')
	200
	201	// ========================================
	202	// Leaf Node Hashing (Data Blocks)
	203	// ========================================
	204
	205	const data1 = Buffer.from('First block')
	206	const data2 = Buffer.from('Second block')
	207
	208	const hash1 = crypto.data(data1)
	209	const hash2 = crypto.data(data2)
	210
	211	console.log('Data hash 1:', hash1.toString('hex'))
	212	console.log('Data hash 2:', hash2.toString('hex'))
	213
	214	// Each 32 bytes (BLAKE2b hash)
	215
	216	// ========================================
	217	// Parent Node Hashing
	218	// ========================================
	219
	220	// In Merkle tree, parent hash combines left and right children
	221
	222	const leftChild = {
	223	index: 0,
	224	hash: hash1,
	225	size: data1.length
	226	}
	227
	228	const rightChild = {
	229	index: 1,
	230	hash: hash2,
	231	size: data2.length
	232	}
	233
	234	const parentHash = crypto.parent(leftChild, rightChild)
	235
	236	console.log('Parent hash:', parentHash.toString('hex'))
	237
	238	// ========================================
	239	// Tree Root Hashing
	240	// ========================================
	241
	242	// Hash the entire merkle tree from its peaks
	243
	244	const peaks = [
	245	{ index: 3, hash: parentHash, size: data1.length + data2.length },
	246	// ... more peaks if tree isn't full
	247	]
	248
	249	const rootHash = crypto.tree(peaks)
	250
	251	console.log('Root hash:', rootHash.toString('hex'))
	252
	253	// This root hash signs the entire tree structure
	254	```
	255
	256	Merkle tree benefits:
	257	- Efficient verification: Don't need entire dataset
	258	- Sparse verification: Verify specific blocks with proof
	259	- Tamper-proof: Any change invalidates root hash
	260	- Signed once: Root hash signature covers all data
	261
	262	Source: holepunchto/hypercore-crypto/README.md:33-50
	263
	264	---
	265
	266	## Code Example: Random Bytes
	267
	268	Cryptographically secure random generation:
	269
	270	```js
	271	const crypto = require('hypercore-crypto')
	272
	273	// ========================================
	274	// Generate Random Bytes
	275	// ========================================
	276
	277	const random32 = crypto.randomBytes(32)
	278	console.log('32 random bytes:', random32.toString('hex'))
	279
	280	// Common uses:
	281	// - Salts for key derivation
	282	// - Nonces for encryption
	283	// - Session IDs
	284	// - One-time pads
	285
	286	// ========================================
	287	// Generate Random Topic for Hyperswarm
	288	// ========================================
	289
	290	const topic = crypto.randomBytes(32)
	291	swarm.join(topic) // Unique topic
	292
	293	// ========================================
	294	// Generate Random Seed
	295	// ========================================
	296
	297	const seed = crypto.randomBytes(32)
	298	const keyPair = crypto.keyPair(seed)
	299
	300	// Save seed for key recovery:
	301	console.log('Recovery seed:', seed.toString('hex'))
	302	// Later: regenerate same keypair from seed
	303	```
	304
	305	Important: Uses cryptographically secure RNG (not Math.random()!)
	306
	307	Source: holepunchto/hypercore-crypto/README.md:52-54
	308
	309	---
	310
	311	## Code Example: Namespaces
	312
	313	Creating cryptographic namespaces for different purposes:
	314
	315	```js
	316	const crypto = require('hypercore-crypto')
	317
	318	// ========================================
	319	// Namespace Creation
	320	// ========================================
	321
	322	// Create namespaces for different algorithms/purposes
	323	const namespaces = crypto.namespace('my-protocol', 3)
	324
	325	console.log('Namespace 0:', namespaces[0].toString('hex'))
	326	console.log('Namespace 1:', namespaces[1].toString('hex'))
	327	console.log('Namespace 2:', namespaces[2].toString('hex'))
	328
	329	// Each namespace is a unique 32-byte buffer
	330
	331	// ========================================
	332	// Use Cases
	333	// ========================================
	334
	335	// Use different namespaces for:
	336	// 1. Different signature purposes
	337	// 2. Different hash purposes
	338	// 3. Domain separation
	339
	340	const signingNamespace = namespaces[0]
	341	const encryptionNamespace = namespaces[1]
	342	const hashingNamespace = namespaces[2]
	343
	344	// Prevents cross-protocol attacks
	345	```
	346
	347	Namespace benefits:
	348	- Domain separation: Keep different uses isolated
	349	- Security: Prevents signature reuse across contexts
	350	- Standardization: Use well-known names for interoperability
	351
	352	Source: holepunchto/hypercore-crypto/README.md:61-64
	353
	354	---
	355
	356	## Code Example: Block Encryption with Hypercore
	357
	358	Encrypting Hypercore blocks:
	359
	360	```js
	361	const Hypercore = require('hypercore')
	362	const crypto = require('hypercore-crypto')
	363
	364	// ========================================
	365	// Create Encrypted Hypercore
	366	// ========================================
	367
	368	const encryptionKey = crypto.randomBytes(32)
	369
	370	const core = new Hypercore('./encrypted-storage', {
	371	encryption: { key: encryptionKey }
	372	})
	373
	374	await core.ready()
	375
	376	console.log('Encryption key:', encryptionKey.toString('hex'))
	377	console.log('Core key:', core.key.toString('hex'))
	378
	379	// ========================================
	380	// Write Encrypted Data
	381	// ========================================
	382
	383	await core.append('Sensitive data')
	384	await core.append('Secret message')
	385
	386	// Data is encrypted before being written to storage
	387
	388	// ========================================
	389	// Read Encrypted Data
	390	// ========================================
	391
	392	// With correct encryption key
	393	const block0 = await core.get(0)
	394	console.log(block0.toString()) // 'Sensitive data' (decrypted)
	395
	396	// ========================================
	397	// Replicate with Encryption
	398	// ========================================
	399
	400	// Peer needs BOTH core key AND encryption key
	401
	402	const replicaCore = new Hypercore('./replica-storage', core.key, {
	403	encryption: { key: encryptionKey } // Must provide!
	404	})
	405
	406	await replicaCore.ready()
	407
	408	// Replicate as normal
	409	// stream = replicaCore.replicate(true)
	410
	411	// Peer can verify integrity (using core.key)
	412	// AND decrypt content (using encryptionKey)
	413
	414	// ========================================
	415	// Without Encryption Key
	416	// ========================================
	417
	418	const noKeyCore = new Hypercore('./no-key-storage', core.key)
	419	// No encryption key provided!
	420
	421	await noKeyCore.ready()
	422
	423	// Can replicate but can't decrypt
	424	try {
	425	const block = await noKeyCore.get(0)
	426	console.log(block) // Encrypted bytes, not readable
	427	} catch (err) {
	428	console.error('Cannot decrypt:', err)
	429	}
	430	```
	431
	432	Encryption properties:
	433	- Core key: For verification (public)
	434	- Encryption key: For decryption (secret)
	435	- Both needed: To read encrypted replica
	436	- Block-level: Each block encrypted separately
	437
	438	Source: Pattern validated against Hypercore encryption API
	439
	440	---
	441
	442	## Real-World Usage
	443
	444	### Example: Hyperident Crypto Manager
	445
	446	From hyper-fleeky/repos/hyperident/lib/crypto.js - Identity management:
	447
	448	```js
	449	import crypto from 'node:crypto'
	450
	451	/**
	452	* CryptoManager - Cryptographic operations for Hyperident
	453	*/
	454	export class CryptoManager {
	455	/**
	456	* Generate cryptographically secure random salt
	457	*/
	458	static generateSalt() {
	459	return crypto.randomBytes(32)
	460	}
	461
	462	/**
	463	* Derive encryption key from passphrase and salt
	464	* Uses SHA-256(passphrase + salt)
	465	*/
	466	static deriveKey(passphrase, salt) {
	467	if (typeof passphrase !== 'string') {
	468	throw new Error('Passphrase must be a string')
	469	}
	470	if (!Buffer.isBuffer(salt) \|\| salt.length !== 32) {
	471	throw new Error('Salt must be a 32-byte Buffer')
	472	}
	473
	474	const passphraseBuffer = Buffer.from(passphrase, 'utf8')
	475	const combined = Buffer.concat([passphraseBuffer, salt])
	476	return crypto.createHash('sha256').update(combined).digest()
	477	}
	478
	479	/**
	480	* Encrypt data using XOR cipher (simple, educational)
	481	* Production should use AES-GCM or ChaCha20-Poly1305
	482	*/
	483	static encryptData(data, key) {
	484	if (!Buffer.isBuffer(data)) {
	485	throw new Error('Data must be a Buffer')
	486	}
	487	if (!Buffer.isBuffer(key) \|\| key.length !== 32) {
	488	throw new Error('Key must be a 32-byte Buffer')
	489	}
	490
	491	const encrypted = Buffer.alloc(data.length)
	492	for (let i = 0; i < data.length; i++) {
	493	encrypted[i] = data[i] ^ key[i % key.length]
	494	}
	495	return encrypted
	496	}
	497
	498	/**
	499	* Decrypt data (XOR is symmetric)
	500	*/
	501	static decryptData(encryptedData, key) {
	502	return this.encryptData(encryptedData, key)
	503	}
	504
	505	/**
	506	* Hash data using SHA-256
	507	*/
	508	static hash(data) {
	509	if (!Buffer.isBuffer(data)) {
	510	throw new Error('Data must be a Buffer')
	511	}
	512	return crypto.createHash('sha256').update(data).digest()
	513	}
	514
	515	/**
	516	* Generate Ed25519 keypair for device
	517	*/
	518	static generateDeviceKeyPair() {
	519	const keyPair = crypto.generateKeyPairSync('ed25519')
	520	return {
	521	publicKey: keyPair.publicKey.export({ type: 'spki', format: 'der' }),
	522	privateKey: keyPair.privateKey.export({ type: 'pkcs8', format: 'der' })
	523	}
	524	}
	525	}
	526
	527	// Usage in Hyperident
	528	class IdentityManager {
	529	async createIdentity(passphrase) {
	530	// Generate salt
	531	const salt = CryptoManager.generateSalt()
	532
	533	// Derive key from passphrase
	534	const encryptionKey = CryptoManager.deriveKey(passphrase, salt)
	535
	536	// Generate device keypair
	537	const deviceKeyPair = CryptoManager.generateDeviceKeyPair()
	538
	539	// Encrypt private key
	540	const encryptedPrivateKey = CryptoManager.encryptData(
	541	deviceKeyPair.privateKey,
	542	encryptionKey
	543	)
	544
	545	// Store identity
	546	const identity = {
	547	publicKey: deviceKeyPair.publicKey,
	548	encryptedPrivateKey: encryptedPrivateKey,
	549	salt: salt,
	550	createdAt: Date.now()
	551	}
	552
	553	return identity
	554	}
	555
	556	async unlockIdentity(identity, passphrase) {
	557	// Derive key from passphrase
	558	const encryptionKey = CryptoManager.deriveKey(passphrase, identity.salt)
	559
	560	// Decrypt private key
	561	const privateKey = CryptoManager.decryptData(
	562	identity.encryptedPrivateKey,
	563	encryptionKey
	564	)
	565
	566	return {
	567	publicKey: identity.publicKey,
	568	privateKey: privateKey
	569	}
	570	}
	571	}
	572	```
	573
	574	Cryptographic patterns used:
	575	1. Random salt generation: `crypto.randomBytes(32)`
	576	2. Key derivation: SHA-256(passphrase + salt)
	577	3. Encryption: XOR cipher (simple example)
	578	4. ED25519 keypairs: For device authentication
	579	5. Encrypted storage: Private keys never stored in plaintext
	580
	581	Source: hyper-fleeky/repos/hyperident/lib/crypto.js:1-100
	582
	583	---
	584
	585	## Common Gotchas
	586
	587	### 1. Using public key instead of discovery key
	588
	589	❌ Wrong:
	590	```js
	591	// Leaks your core key to DHT!
	592	swarm.join(core.key)
	593	```
	594
	595	✅ Correct:
	596	```js
	597	// Safe: uses hash of core key
	598	swarm.join(core.discoveryKey)
	599	```
	600
	601	Why: Discovery key is one-way hash. Can't reverse it to get actual core key.
	602
	603	---
	604
	605	### 2. Not keeping secret keys secret
	606
	607	❌ Wrong:
	608	```js
	609	// Logging secret keys
	610	console.log('My secret key:', keyPair.secretKey.toString('hex'))
	611
	612	// Sharing secret keys
	613	socket.write(keyPair.secretKey)
	614	```
	615
	616	✅ Correct:
	617	```js
	618	// Only share public key
	619	socket.write(keyPair.publicKey)
	620
	621	// Never log secret keys in production
	622	// if (process.env.NODE_ENV !== 'production') {
	623	// console.log('Debug key:', keyPair.secretKey.toString('hex'))
	624	// }
	625	```
	626
	627	Why: Secret keys allow signing/decryption. If leaked, security is compromised.
	628
	629	---
	630
	631	### 3. Weak key derivation
	632
	633	❌ Wrong:
	634	```js
	635	// Weak: just hashing passphrase
	636	const key = crypto.createHash('sha256')
	637	.update(passphrase)
	638	.digest()
	639	```
	640
	641	✅ Correct:
	642	```js
	643	// Better: Use salt and multiple rounds
	644	const key = crypto.pbkdf2Sync(
	645	passphrase,
	646	salt,
	647	100000, // iterations
	648	32, // key length
	649	'sha256'
	650	)
	651	```
	652
	653	Why: Simple hash is vulnerable to rainbow tables. Use proper key derivation (PBKDF2, scrypt, argon2).
	654
	655	---
	656
	657	### 4. Not verifying signatures
	658
	659	❌ Wrong:
	660	```js
	661	// Trusting data without verification
	662	const data = await receivedFromPeer()
	663	processData(data) // Could be tampered!
	664	```
	665
	666	✅ Correct:
	667	```js
	668	const { data, signature, publicKey } = await receivedFromPeer()
	669
	670	const isValid = crypto.verify(data, signature, publicKey)
	671
	672	if (!isValid) {
	673	throw new Error('Invalid signature - data tampered!')
	674	}
	675
	676	processData(data) // Now we can trust it
	677	```
	678
	679	Why: Never trust data from peers. Always verify signatures.
	680
	681	---
	682
	683	### 5. Reusing nonces/salts
	684
	685	❌ Wrong:
	686	```js
	687	const salt = Buffer.alloc(32).fill(0) // Same salt for everyone!
	688
	689	const key1 = deriveKey(password1, salt)
	690	const key2 = deriveKey(password2, salt) // Same salt = rainbow table attack
	691	```
	692
	693	✅ Correct:
	694	```js
	695	const salt1 = crypto.randomBytes(32) // Unique salt
	696	const salt2 = crypto.randomBytes(32) // Different salt
	697
	698	const key1 = deriveKey(password1, salt1)
	699	const key2 = deriveKey(password2, salt2)
	700	```
	701
	702	Why: Unique salts prevent rainbow table attacks.
	703
	704	---
	705
	706	### 6. Not handling encryption key loss
	707
	708	❌ Wrong:
	709	```js
	710	// Lost encryption key = lost data
	711	const encryptionKey = crypto.randomBytes(32)
	712	const core = new Hypercore('./storage', { encryption: { key: encryptionKey } })
	713
	714	// If encryptionKey is lost, data is unrecoverable!
	715	```
	716
	717	✅ Correct:
	718	```js
	719	// Derive from recoverable seed
	720	const seed = userPassphrase // Or stored seed
	721	const encryptionKey = crypto.createHash('sha256')
	722	.update(seed)
	723	.digest()
	724
	725	// Can recover key from seed/passphrase
	726	const core = new Hypercore('./storage', { encryption: { key: encryptionKey } })
	727
	728	// Store seed safely (paper backup, password manager, etc.)
	729	```
	730
	731	Why: Encryption keys must be recoverable or data is permanently lost.
	732
	733	---
	734
	735	## Anti-Patterns
	736
	737	### ❌ 1. Rolling your own crypto
	738
	739	Don't:
	740	```js
	741	// Custom encryption algorithm
	742	function myEncrypt(data, key) {
	743	// Some homemade cipher
	744	return scramble(data, key) // ❌ Probably insecure!
	745	}
	746	```
	747
	748	Do: Use proven primitives
	749	```js
	750	// Use hypercore-crypto or standard Node crypto
	751	const encrypted = crypto.encrypt(data, key)
	752	```
	753
	754	Why: Cryptography is hard. Use battle-tested implementations.
	755
	756	---
	757
	758	### ❌ 2. Using weak randomness
	759
	760	Don't:
	761	```js
	762	// Weak random for crypto
	763	const randomKey = Buffer.alloc(32)
	764	for (let i = 0; i < 32; i++) {
	765	randomKey[i] = Math.floor(Math.random() * 256) // ❌ Not cryptographically secure!
	766	}
	767	```
	768
	769	Do: Use crypto RNG
	770	```js
	771	const randomKey = crypto.randomBytes(32) // ✓ Cryptographically secure
	772	```
	773
	774	Why: Math.random() is predictable. Use crypto-grade RNG.
	775
	776	---
	777
	778	### ❌ 3. Storing keys in plaintext
	779
	780	Don't:
	781	```js
	782	// Storing secret key in plain file
	783	fs.writeFileSync('secret.key', secretKey)
	784	```
	785
	786	Do: Encrypt before storage
	787	```js
	788	// Derive encryption key from user passphrase
	789	const storageKey = deriveKeyFromPassphrase(userPassphrase)
	790	const encryptedSecret = encrypt(secretKey, storageKey)
	791	fs.writeFileSync('secret.key.enc', encryptedSecret)
	792	```
	793
	794	Why: Plaintext keys can be stolen. Always encrypt secrets at rest.
	795
	796	---
	797
	798	### ❌ 4. Sharing encryption keys publicly
	799
	800	Don't:
	801	```js
	802	// Sending encryption key over insecure channel
	803	socket.send(encryptionKey) // ❌ Anyone can intercept!
	804	```
	805
	806	Do: Use key exchange
	807	```js
	808	// Use Diffie-Hellman key exchange or pre-shared secrets
	809	const sharedSecret = dhKeyExchange(myPrivateKey, theirPublicKey)
	810	const encryptionKey = deriveKey(sharedSecret)
	811	```
	812
	813	Why: Encryption keys are secret. Use proper key exchange protocols.
	814
	815	---
	816
	817	## Integration Patterns
	818
	819	### Pattern 1: Hypercore with Encryption
	820
	821	```js
	822	const Hypercore = require('hypercore')
	823	const crypto = require('hypercore-crypto')
	824
	825	// Generate encryption key
	826	const encryptionKey = crypto.randomBytes(32)
	827
	828	// Create encrypted core
	829	const core = new Hypercore('./storage', {
	830	encryption: { key: encryptionKey }
	831	})
	832
	833	await core.ready()
	834
	835	// Replicate (peers need both keys)
	836	swarm.on('connection', (socket) => {
	837	core.replicate(false).pipe(socket).pipe(socket)
	838	})
	839
	840	// Share keys securely (out of band)
	841	console.log('Core key:', core.key.toString('hex'))
	842	console.log('Encryption key:', encryptionKey.toString('hex'))
	843	```
	844
	845	---
	846
	847	### Pattern 2: Signed Messages
	848
	849	```js
	850	const crypto = require('hypercore-crypto')
	851
	852	const keyPair = crypto.keyPair()
	853
	854	// Create signed message
	855	function createSignedMessage(content) {
	856	const message = Buffer.from(JSON.stringify(content))
	857	const signature = crypto.sign(message, keyPair.secretKey)
	858
	859	return {
	860	content: content,
	861	signature: signature.toString('hex'),
	862	publicKey: keyPair.publicKey.toString('hex')
	863	}
	864	}
	865
	866	// Verify signed message
	867	function verifySignedMessage(signedMsg) {
	868	const message = Buffer.from(JSON.stringify(signedMsg.content))
	869	const signature = Buffer.from(signedMsg.signature, 'hex')
	870	const publicKey = Buffer.from(signedMsg.publicKey, 'hex')
	871
	872	return crypto.verify(message, signature, publicKey)
	873	}
	874
	875	// Usage
	876	const msg = createSignedMessage({ text: 'Hello', timestamp: Date.now() })
	877	const isValid = verifySignedMessage(msg)
	878	console.log('Valid:', isValid) // true
	879	```
	880
	881	---
	882
	883	### Pattern 3: Key Derivation for Multi-Device
	884
	885	```js
	886	const crypto = require('hypercore-crypto')
	887
	888	// User has one master seed
	889	const masterSeed = crypto.randomBytes(32) // User backs this up
	890
	891	// Derive device-specific keys
	892	function deriveDeviceKey(masterSeed, deviceId) {
	893	const combined = Buffer.concat([
	894	masterSeed,
	895	Buffer.from(deviceId, 'utf8')
	896	])
	897	const deviceSeed = crypto.createHash('sha256').update(combined).digest()
	898	return crypto.keyPair(deviceSeed)
	899	}
	900
	901	// Each device gets unique keypair from master seed
	902	const phoneKey = deriveDeviceKey(masterSeed, 'phone')
	903	const laptopKey = deriveDeviceKey(masterSeed, 'laptop')
	904
	905	// Can recover all device keys from master seed
	906	```
	907
	908	---
	909
	910	## See Also
	911
	912	### Pattern Documentation
	913	- [append-only-log.md](./append-only-log.md) - Hypercore uses crypto for verification
	914	- [replication.md](./replication.md) - Signature verification during replication
	915	- [peer-discovery.md](./peer-discovery.md) - Discovery key usage
	916
	917	### Deployment Documentation
	918	- [../deployment/pear-apps.md](../deployment/pear-apps.md) - Deploying apps with crypto
	919	- [../deployment/troubleshooting.md](../deployment/troubleshooting.md) - Crypto issues
	920
	921	### Repository Documentation
	922	- [../skills/hypercore-crypto/skills.md](../skills/hypercore-crypto/skills.md) - Full crypto API
	923	- [../skills/hypercore/skills.md](../skills/hypercore/skills.md) - Hypercore verification
	924
	925	### Official Resources
	926	- [hypercore-crypto GitHub](https://github.com/holepunchto/hypercore-crypto)
	927	- [Hypercore Crypto Docs](https://docs.pears.com)
	928
	929	---
	930
	931	## References
	932
	933	- Tier 3 Repo: holepunchto/repos/tier3/hypercore-crypto
	934	- Real-World Example: hyper-fleeky/repos/hyperident (identity encryption)
	935	- Pattern Category: Security & Cryptography
	936	- Algorithms: ED25519 (signing), BLAKE2b (hashing), SHA-256 (hashing)
	937	- Use Cases: Data verification, signing, encryption, key derivation
	938
	939	---
	940
	941	Last Updated: 2026-01-18
	942	Validated Against: hypercore-crypto 3.x, Hypercore 10.x, Hyperident

patterns/file-system.md +78 -0

…	…	@@ -0,0 +1,78 @@
	1	# Pattern: File System
	2
	3	Validated against: holepunchto/hyperdrive
	4
	5	## Description
	6
	7	POSIX-like file system built on Hypercore, providing file and directory operations with P2P replication. Hyperdrive stores files and metadata in Hypercores, allowing distributed file sharing.
	8
	9	Key features:
	10	- POSIX operations: put, get, mkdir, readdir, unlink
	11	- Metadata: File stats (size, mtime, etc.)
	12	- Efficient: Sparse file replication
	13	- Versioned: Immutable history of changes
	14
	15	---
	16
	17	## Code Example: Basic File Operations
	18
	19	```js
	20	const Hyperdrive = require('hyperdrive')
	21
	22	const drive = new Hyperdrive('./my-drive')
	23	await drive.ready()
	24
	25	// ========================================
	26	// Write Files
	27	// ========================================
	28
	29	await drive.put('/hello.txt', 'Hello, world!')
	30	await drive.put('/data/file.json', JSON.stringify({ foo: 'bar' }))
	31
	32	// ========================================
	33	// Read Files
	34	// ========================================
	35
	36	const content = await drive.get('/hello.txt')
	37	console.log(content.toString()) // 'Hello, world!'
	38
	39	const json = JSON.parse(await drive.get('/data/file.json'))
	40	console.log(json) // { foo: 'bar' }
	41
	42	// ========================================
	43	// Directory Operations
	44	// ========================================
	45
	46	// List directory
	47	for await (const entry of drive.list('/data')) {
	48	console.log(entry.key) // '/data/file.json'
	49	console.log(entry.value.blob.byteLength) // File size
	50	}
	51
	52	// ========================================
	53	// Delete
	54	// ========================================
	55
	56	await drive.del('/hello.txt')
	57
	58	// ========================================
	59	// Replication
	60	// ========================================
	61
	62	const swarm = new Hyperswarm()
	63
	64	swarm.on('connection', (socket) => {
	65	drive.replicate(socket)
	66	})
	67
	68	swarm.join(drive.discoveryKey)
	69
	70	// Files replicate to peers automatically!
	71	```
	72
	73	See: Integration in [append-only-log.md](./append-only-log.md#pattern-4-hypercore--hyperdrive-file-system)
	74
	75	---
	76
	77	Source: holepunchto/hyperdrive
	78	Last Updated: 2026-01-18

patterns/peer-discovery.md +959 -0

…	…	@@ -0,0 +1,959 @@
	1	# Pattern: Peer Discovery
	2
	3	Validated against: holepunchto/hyperswarm, holepunchto/hyperdht, hyper-fleeky/fedicore
	4
	5	## Description
	6
	7	Peer discovery is the process of finding other peers in a P2P network who share interest in the same data (topics). In the Hypercore ecosystem, this uses a Distributed Hash Table (DHT) for decentralized discovery without central servers.
	8
	9	Core concepts:
	10	- Topic: 32-byte identifier (usually a `discoveryKey`) that peers join
	11	- DHT (Distributed Hash Table): Decentralized network for announcing and finding peers
	12	- Server mode: Announce yourself to DHT so others can find you
	13	- Client mode: Search DHT to find announced servers
	14	- Holepunching: NAT traversal technique to establish direct connections
	15
	16	Key insight: Hyperswarm abstracts DHT complexity, providing simple `join(topic)` API that handles both announcing and discovering peers automatically.
	17
	18	---
	19
	20	## When to Use
	21
	22	✅ Use peer discovery when:
	23	- Building decentralized P2P applications
	24	- Creating serverless applications (no central coordination)
	25	- Implementing mesh networks
	26	- Building distributed databases
	27	- Creating P2P chat/social networks
	28	- Sharing files directly between peers
	29	- Building collaborative applications
	30
	31	❌ Don't use when:
	32	- You need centralized access control (P2P is permissionless)
	33	- Network is behind restrictive NAT/firewall (corporate networks)
	34	- Privacy requires hiding participation (DHT reveals you're interested in a topic)
	35	- You have a small fixed set of peers (use direct connections instead)
	36
	37	---
	38
	39	## Implementing Repos
	40
	41	### Tier 1 (Core Building Blocks)
	42
	43	- [hyperswarm](https://github.com/holepunchto/hyperswarm) (1,275 ⭐)
	44	- High-level API for peer discovery
	45	- Automatic connection management
	46	- `npm install hyperswarm`
	47
	48	- [hyperdht](https://github.com/holepunchto/hyperdht) (280 ⭐)
	49	- Low-level DHT implementation
	50	- UDP holepunching
	51	- `npm install hyperdht`
	52
	53	### Related Repos
	54
	55	- [hypercore](https://github.com/holepunchto/hypercore) - Uses `discoveryKey` for topics
	56	- [corestore](https://github.com/holepunchto/corestore) - Manages multiple discovery topics
	57	- [@hyperswarm/dht-relay](https://github.com/hyperswarm/dht-relay) - Relay server for restrictive networks
	58
	59	---
	60
	61	## Code Example: Basic Hyperswarm Discovery
	62
	63	Finding and connecting to peers interested in a topic:
	64
	65	```js
	66	const Hyperswarm = require('hyperswarm')
	67	const crypto = require('hypercore-crypto')
	68
	69	// ========================================
	70	// PEER 1: Server (announces to DHT)
	71	// ========================================
	72
	73	const swarm1 = new Hyperswarm()
	74
	75	// Create a topic (must be 32 bytes)
	76	const topic = crypto.randomBytes(32)
	77
	78	console.log('Topic:', topic.toString('hex'))
	79
	80	// Join as server (announce to DHT)
	81	const discovery1 = swarm1.join(topic, {
	82	server: true, // Announce: "I'm available for this topic"
	83	client: false // Don't search for others
	84	})
	85
	86	// Wait for DHT announcement
	87	await discovery1.flushed()
	88	console.log('Announced to DHT - waiting for connections...')
	89
	90	// Handle incoming connections
	91	swarm1.on('connection', (socket, peerInfo) => {
	92	console.log('Peer connected!')
	93	console.log('Peer public key:', peerInfo.publicKey.toString('hex'))
	94
	95	socket.on('data', (data) => {
	96	console.log('Received:', data.toString())
	97	})
	98
	99	socket.write('Hello from server!')
	100	})
	101
	102	// ========================================
	103	// PEER 2: Client (discovers from DHT)
	104	// ========================================
	105
	106	const swarm2 = new Hyperswarm()
	107
	108	// Join as client (search DHT for servers)
	109	const discovery2 = swarm2.join(topic, {
	110	server: false, // Don't announce
	111	client: true // Search: "Find me peers with this topic"
	112	})
	113
	114	// Wait for connections
	115	await swarm2.flush()
	116	console.log('Searching DHT for peers...')
	117
	118	// Handle connections
	119	swarm2.on('connection', (socket, peerInfo) => {
	120	console.log('Connected to peer!')
	121	console.log('Peer public key:', peerInfo.publicKey.toString('hex'))
	122
	123	socket.on('data', (data) => {
	124	console.log('Received:', data.toString())
	125	})
	126
	127	socket.write('Hello from client!')
	128	})
	129
	130	// After a few seconds:
	131	// Peer 1 logs: "Peer connected!"
	132	// Peer 2 logs: "Connected to peer!"
	133	// Both peers can now communicate!
	134	```
	135
	136	Key concepts:
	137	- Topic: Shared 32-byte identifier (like a "room name")
	138	- Server mode: Announces to DHT, accepts connections
	139	- Client mode: Searches DHT, initiates connections
	140	- Both modes: Most peers use `{ server: true, client: true }`
	141
	142	Source: holepunchto/hyperswarm/README.md:14-39
	143
	144	---
	145
	146	## Code Example: Hypercore Discovery Pattern
	147
	148	Using Hypercore's discoveryKey for peer discovery:
	149
	150	```js
	151	const Hypercore = require('hypercore')
	152	const Hyperswarm = require('hyperswarm')
	153
	154	// ========================================
	155	// PEER 1: Share a Hypercore
	156	// ========================================
	157
	158	const core1 = new Hypercore('./peer1-storage')
	159	await core1.ready()
	160
	161	await core1.append('First block')
	162	await core1.append('Second block')
	163
	164	console.log('Core key:', core1.key.toString('hex'))
	165	console.log('Discovery key:', core1.discoveryKey.toString('hex'))
	166
	167	// Create swarm
	168	const swarm1 = new Hyperswarm()
	169
	170	swarm1.on('connection', (socket, peerInfo) => {
	171	console.log('Peer discovered via DHT!')
	172
	173	// Replicate Hypercore over connection
	174	core1.replicate(peerInfo.client).pipe(socket).pipe(socket)
	175	})
	176
	177	// Use discoveryKey as topic (doesn't leak core key!)
	178	const topic = core1.discoveryKey
	179	swarm1.join(topic, { server: true, client: true })
	180	await swarm1.flush()
	181
	182	console.log('Announced Hypercore to DHT')
	183
	184	// ========================================
	185	// PEER 2: Discover and replicate
	186	// ========================================
	187
	188	// Create read-only replica
	189	const core2 = new Hypercore('./peer2-storage', core1.key)
	190	await core2.ready()
	191
	192	const swarm2 = new Hyperswarm()
	193
	194	swarm2.on('connection', (socket, peerInfo) => {
	195	console.log('Found peer via DHT!')
	196
	197	// Replicate
	198	core2.replicate(peerInfo.client).pipe(socket).pipe(socket)
	199	})
	200
	201	// Same topic = discoveryKey
	202	const topic = core2.discoveryKey
	203	swarm2.join(topic, { server: true, client: true })
	204	await swarm2.flush()
	205
	206	// Wait for sync
	207	await core2.update()
	208	console.log('Discovered and synced! Length:', core2.length)
	209
	210	for (let i = 0; i < core2.length; i++) {
	211	const block = await core2.get(i)
	212	console.log(`Block ${i}:`, block.toString())
	213	}
	214	```
	215
	216	Why discoveryKey:
	217	- `core.key` - Public key for verifying data (sensitive)
	218	- `core.discoveryKey` - Hash of core.key (safe to share publicly)
	219	- Using discoveryKey for DHT prevents leaking the actual core key
	220
	221	Source: Pattern validated against hypercore + hyperswarm integration
	222
	223	---
	224
	225	## Code Example: Client-Server vs Peer-to-Peer Modes
	226
	227	Understanding server/client roles:
	228
	229	```js
	230	const Hyperswarm = require('hyperswarm')
	231	const topic = Buffer.alloc(32).fill('demo')
	232
	233	// ========================================
	234	// Server-only (announces, doesn't search)
	235	// ========================================
	236
	237	const serverSwarm = new Hyperswarm()
	238
	239	serverSwarm.on('connection', (socket, peerInfo) => {
	240	console.log('Server: Accepted connection')
	241	// Only receives connections, never initiates
	242	})
	243
	244	serverSwarm.join(topic, {
	245	server: true, // Announce to DHT
	246	client: false // Don't search DHT
	247	})
	248
	249	await serverSwarm.flush()
	250
	251	// ========================================
	252	// Client-only (searches, doesn't announce)
	253	// ========================================
	254
	255	const clientSwarm = new Hyperswarm()
	256
	257	clientSwarm.on('connection', (socket, peerInfo) => {
	258	console.log('Client: Connected to server')
	259	// Only initiates connections, never accepts
	260	})
	261
	262	clientSwarm.join(topic, {
	263	server: false, // Don't announce
	264	client: true // Search DHT for servers
	265	})
	266
	267	await clientSwarm.flush()
	268
	269	// ========================================
	270	// Peer-to-peer (both modes - most common)
	271	// ========================================
	272
	273	const p2pSwarm = new Hyperswarm()
	274
	275	p2pSwarm.on('connection', (socket, peerInfo) => {
	276	console.log('P2P: Connection established')
	277	console.log('We initiated:', peerInfo.client)
	278	// Can both accept AND initiate connections
	279	})
	280
	281	p2pSwarm.join(topic, {
	282	server: true, // Announce AND
	283	client: true // Search - full P2P!
	284	})
	285
	286	await p2pSwarm.flush()
	287	```
	288
	289	Modes:
	290	- Server-only: Passive, waits for clients (rare)
	291	- Client-only: Active, finds servers (rare)
	292	- P2P (both): Default - everyone can connect to everyone
	293
	294	Source: holepunchto/hyperswarm/README.md:99-106
	295
	296	---
	297
	298	## Code Example: Multiple Topics
	299
	300	Joining multiple topics for different data:
	301
	302	```js
	303	const Hyperswarm = require('hyperswarm')
	304	const Hypercore = require('hypercore')
	305
	306	// Create multiple Hypercores
	307	const postsCore = new Hypercore('./posts')
	308	const commentsCore = new Hypercore('./comments')
	309	const likesCore = new Hypercore('./likes')
	310
	311	await Promise.all([
	312	postsCore.ready(),
	313	commentsCore.ready(),
	314	likesCore.ready()
	315	])
	316
	317	const swarm = new Hyperswarm()
	318
	319	// Track which topic triggered each connection
	320	swarm.on('connection', (socket, peerInfo) => {
	321	console.log('Connection for topics:', peerInfo.topics)
	322
	323	// Replicate appropriate cores
	324	// Note: In practice, use Corestore.replicate() for this
	325	postsCore.replicate(peerInfo.client).pipe(socket).pipe(socket)
	326	commentsCore.replicate(peerInfo.client).pipe(socket).pipe(socket)
	327	likesCore.replicate(peerInfo.client).pipe(socket).pipe(socket)
	328	})
	329
	330	// Join multiple topics
	331	swarm.join(postsCore.discoveryKey)
	332	swarm.join(commentsCore.discoveryKey)
	333	swarm.join(likesCore.discoveryKey)
	334
	335	await swarm.flush()
	336
	337	console.log('Announced 3 topics to DHT')
	338
	339	// Peers who join ANY of these topics will connect
	340	// All 3 cores replicate over the same connection (multiplexed)
	341	```
	342
	343	Benefits:
	344	- One swarm can manage many topics
	345	- Connections are reused (multiplexed)
	346	- Efficient for apps with multiple data streams
	347
	348	---
	349
	350	## Code Example: Direct Peer Connection
	351
	352	Connecting directly to a known peer (no DHT lookup):
	353
	354	```js
	355	const Hyperswarm = require('hyperswarm')
	356
	357	// ========================================
	358	// PEER 1: Generate and share keypair
	359	// ========================================
	360
	361	const swarm1 = new Hyperswarm()
	362
	363	console.log('My public key:', swarm1.keyPair.publicKey.toString('hex'))
	364
	365	swarm1.on('connection', (socket) => {
	366	console.log('Direct connection established!')
	367	socket.write('Hello via direct connection')
	368	})
	369
	370	// ========================================
	371	// PEER 2: Connect directly to Peer 1
	372	// ========================================
	373
	374	const swarm2 = new Hyperswarm()
	375
	376	// Use Peer 1's public key (from above)
	377	const peer1PublicKey = Buffer.from('...', 'hex')
	378
	379	// Join directly - no DHT announcement/search
	380	swarm2.joinPeer(peer1PublicKey)
	381
	382	swarm2.on('connection', (socket) => {
	383	console.log('Connected directly to peer!')
	384
	385	socket.on('data', (data) => {
	386	console.log('Received:', data.toString())
	387	})
	388	})
	389	```
	390
	391	Use case: When you know peer's public key in advance (QR code, invite link, etc.)
	392
	393	Source: holepunchto/hyperswarm/README.md:118-132
	394
	395	---
	396
	397	## Real-World Usage
	398
	399	### Example: Fedicore Social Network Discovery
	400
	401	From hyper-fleeky/repos/fedicore/fedicore.js - multi-user discovery:
	402
	403	```js
	404	export class Fedicore extends EventEmitter {
	405	async init(options = {}) {
	406	// Initialize storage and feeds
	407	this.store = new Corestore(join(this.storagePath, 'feeds'))
	408	await this.store.ready()
	409
	410	this.myFeed = this.store.get({ name: 'my-feed' })
	411	await this.myFeed.ready()
	412
	413	// Create swarm for peer discovery
	414	this.swarm = new Hyperswarm()
	415
	416	// KEY PATTERN: Auto-replicate on every discovered connection
	417	this.swarm.on('connection', (conn) => {
	418	console.log('Peer discovered and connected')
	419	this.store.replicate(conn) // Replicate all feeds
	420	})
	421
	422	// Announce own feed to DHT
	423	const topic = this.myFeed.discoveryKey
	424	this.swarm.join(topic, { server: true, client: true })
	425	await this.swarm.flush()
	426
	427	console.log('Announced to Fedicore network')
	428	console.log('Public key:', this.getPublicKey())
	429	console.log('Feed key:', b4a.toString(this.myFeed.key, 'hex'))
	430	}
	431
	432	// Follow another user = discover and replicate their feed
	433	async follow(publicKey, name = 'Anonymous') {
	434	// Get their feed
	435	const feedKey = b4a.from(publicKey, 'base64')
	436	const feed = this.store.get(feedKey)
	437	await feed.ready()
	438
	439	this.follows.set(publicKey, { feed, name })
	440
	441	// KEY DISCOVERY PATTERN: Join their topic
	442	const topic = feed.discoveryKey
	443	this.swarm.join(topic, { server: false, client: true })
	444
	445	console.log(`Discovering peers for ${name}...`)
	446
	447	// New connections will auto-replicate this feed (via store.replicate)
	448	await this.swarm.flush()
	449
	450	// Wait for initial sync
	451	await feed.update()
	452	console.log(`Discovered and synced ${feed.length} posts from ${name}`)
	453
	454	// Live updates
	455	feed.on('append', async () => {
	456	const latestBlock = await feed.get(feed.length - 1)
	457	const post = JSON.parse(latestBlock)
	458
	459	this.emit('follow-update', {
	460	author: name,
	461	post: post,
	462	index: feed.length - 1
	463	})
	464	})
	465	}
	466
	467	// Unfollow = stop discovering/replicating
	468	async unfollow(publicKey) {
	469	const follow = this.follows.get(publicKey)
	470	if (!follow) return
	471
	472	// Leave DHT topic - stop discovering peers
	473	const topic = follow.feed.discoveryKey
	474	await this.swarm.leave(topic)
	475
	476	this.follows.delete(publicKey)
	477	console.log('Stopped discovering peers for', publicKey)
	478	}
	479	}
	480	```
	481
	482	Discovery patterns used:
	483	1. Own feed: `{ server: true, client: true }` - full P2P
	484	2. Following: `{ server: false, client: true }` - only find their peers
	485	3. Unfollow: `swarm.leave(topic)` - stop discovery
	486	4. Multi-feed: One swarm manages all discovery topics
	487	5. Auto-replication: `store.replicate()` handles all feeds on connection
	488
	489	Source: hyper-fleeky/repos/fedicore/fedicore.js:44-73, 127-160
	490
	491	---
	492
	493	## Common Gotchas
	494
	495	### 1. Topic not 32 bytes
	496
	497	❌ Wrong:
	498	```js
	499	const topic = Buffer.from('my-app') // Only 6 bytes!
	500	swarm.join(topic) // Error: Topic must be 32 bytes
	501	```
	502
	503	✅ Correct:
	504	```js
	505	const topic = Buffer.alloc(32).fill('my-app') // Pad to 32 bytes
	506
	507	// Or use hash
	508	const crypto = require('hypercore-crypto')
	509	const topic = crypto.randomBytes(32) // Exactly 32 bytes
	510
	511	// Or use discoveryKey
	512	const topic = core.discoveryKey // Always 32 bytes
	513	```
	514
	515	Why: DHT requires 32-byte topics for consistent hashing.
	516
	517	---
	518
	519	### 2. Not waiting for `flush()` or `flushed()`
	520
	521	❌ Wrong:
	522	```js
	523	swarm.join(topic, { server: true, client: true })
	524	// Immediately assume we have connections
	525	core.replicate(true).pipe(socket).pipe(socket) // No socket yet!
	526	```
	527
	528	✅ Correct:
	529	```js
	530	const discovery = swarm.join(topic, { server: true, client: true })
	531
	532	// For server: wait for DHT announcement
	533	await discovery.flushed()
	534
	535	// For all: wait for peer connections
	536	await swarm.flush()
	537
	538	// Now connections will start arriving
	539	swarm.on('connection', (socket) => {
	540	// Handle connection
	541	})
	542	```
	543
	544	Why: Discovery and connections are async. Must wait for DHT operations.
	545
	546	---
	547
	548	### 3. Using core.key instead of core.discoveryKey
	549
	550	❌ Wrong:
	551	```js
	552	// Leaks core key to DHT!
	553	swarm.join(core.key)
	554	```
	555
	556	✅ Correct:
	557	```js
	558	// Safe: discoveryKey is a hash
	559	swarm.join(core.discoveryKey)
	560	```
	561
	562	Why: `discoveryKey = hash(core.key)`. It allows peer discovery without revealing the actual core key.
	563
	564	---
	565
	566	### 4. Everyone using client-only mode
	567
	568	❌ Wrong:
	569	```js
	570	// All peers do this:
	571	swarm.join(topic, { server: false, client: true })
	572
	573	// Result: Nobody announces to DHT, so nobody can find anyone!
	574	```
	575
	576	✅ Correct:
	577	```js
	578	// Most peers should use both:
	579	swarm.join(topic, { server: true, client: true })
	580	```
	581
	582	Why: If everyone searches but nobody announces, discovery fails. At least some peers need `server: true`.
	583
	584	---
	585
	586	### 5. Not destroying swarms on exit
	587
	588	❌ Wrong:
	589	```js
	590	async function tempFunction() {
	591	const swarm = new Hyperswarm()
	592	swarm.join(topic)
	593	// ... do stuff ...
	594	// Forgot to destroy!
	595	} // Swarm leaks, DHT polluted
	596	```
	597
	598	✅ Correct:
	599	```js
	600	async function tempFunction() {
	601	const swarm = new Hyperswarm()
	602	swarm.join(topic)
	603
	604	try {
	605	// ... do stuff ...
	606	} finally {
	607	await swarm.leave(topic)
	608	await swarm.destroy()
	609	}
	610	}
	611	```
	612
	613	Why: Swarms hold network resources and DHT state. Must clean up.
	614
	615	---
	616
	617	### 6. Expecting instant connections
	618
	619	❌ Wrong:
	620	```js
	621	swarm.join(topic)
	622	// Expect immediate connections
	623	console.log('Connections:', swarm.connections.size) // 0!
	624	```
	625
	626	✅ Correct:
	627	```js
	628	swarm.join(topic)
	629
	630	swarm.on('connection', (socket) => {
	631	console.log('Connection! Total:', swarm.connections.size)
	632	})
	633
	634	// Or wait
	635	await swarm.flush()
	636	console.log('After flush, connections:', swarm.connections.size)
	637	```
	638
	639	Why: Discovery takes time (DHT lookup, NAT traversal). Use events or flush().
	640
	641	---
	642
	643	## Anti-Patterns
	644
	645	### ❌ 1. Creating multiple swarms for different topics
	646
	647	Don't:
	648	```js
	649	// Bad: One swarm per topic
	650	const swarm1 = new Hyperswarm()
	651	swarm1.join(topic1)
	652
	653	const swarm2 = new Hyperswarm()
	654	swarm2.join(topic2)
	655
	656	const swarm3 = new Hyperswarm()
	657	swarm3.join(topic3)
	658
	659	// Wastes DHT resources, network overhead
	660	```
	661
	662	Do: Use one swarm for all topics
	663	```js
	664	// Good: One swarm, multiple topics
	665	const swarm = new Hyperswarm()
	666	swarm.join(topic1)
	667	swarm.join(topic2)
	668	swarm.join(topic3)
	669
	670	// All topics share connections when possible
	671	```
	672
	673	Why: Hyperswarm multiplexes topics over connections. One swarm can handle many topics efficiently.
	674
	675	---
	676
	677	### ❌ 2. Using the same topic for unrelated data
	678
	679	Don't:
	680	```js
	681	// Bad: All cores use same generic topic
	682	const appTopic = Buffer.alloc(32).fill('my-app')
	683
	684	swarm.join(appTopic) // Everyone connects to everyone!
	685
	686	// Now all peers connect even if they don't share data
	687	```
	688
	689	Do: Use unique topics per core
	690	```js
	691	// Good: Each core has unique discoveryKey
	692	swarm.join(core1.discoveryKey) // Specific to core1
	693	swarm.join(core2.discoveryKey) // Specific to core2
	694
	695	// Peers only connect if they share specific cores
	696	```
	697
	698	Why: Generic topics create unnecessary connections. Use discoveryKeys for precise discovery.
	699
	700	---
	701
	702	### ❌ 3. Polling for connections
	703
	704	Don't:
	705	```js
	706	// Bad: Polling
	707	setInterval(() => {
	708	if (swarm.connections.size > 0) {
	709	console.log('Have connections!')
	710	}
	711	}, 1000) // Wastes CPU
	712	```
	713
	714	Do: Use events
	715	```js
	716	// Good: Event-driven
	717	swarm.on('connection', (socket) => {
	718	console.log('New connection! Total:', swarm.connections.size)
	719	})
	720
	721	swarm.on('update', () => {
	722	console.log('Swarm updated. Connections:', swarm.connections.size)
	723	})
	724	```
	725
	726	Why: Events are immediate and efficient. No polling overhead.
	727
	728	---
	729
	730	### ❌ 4. Not handling firewall rejections
	731
	732	Don't:
	733	```js
	734	// Bad: Accept all connections
	735	const swarm = new Hyperswarm()
	736	// Any peer can connect!
	737	```
	738
	739	Do: Use firewall when needed
	740	```js
	741	// Good: Filter unwanted connections
	742	const allowedPeers = new Set([
	743	'peer1-pubkey-hex',
	744	'peer2-pubkey-hex'
	745	])
	746
	747	const swarm = new Hyperswarm({
	748	firewall(remotePublicKey) {
	749	const key = remotePublicKey.toString('hex')
	750	return !allowedPeers.has(key) // true = reject, false = accept
	751	}
	752	})
	753
	754	swarm.on('ban', (peerInfo, error) => {
	755	console.log('Rejected peer:', peerInfo.publicKey.toString('hex'))
	756	})
	757	```
	758
	759	Why: Firewall provides access control, though remember P2P is generally permissionless by design.
	760
	761	Source: holepunchto/hyperswarm/README.md:49-53
	762
	763	---
	764
	765	## Integration Patterns
	766
	767	### Pattern 1: Hyperswarm + Hypercore (Standard)
	768
	769	```js
	770	const Hypercore = require('hypercore')
	771	const Hyperswarm = require('hyperswarm')
	772
	773	async function replicatedCore(storage, key = null) {
	774	const core = new Hypercore(storage, key)
	775	await core.ready()
	776
	777	const swarm = new Hyperswarm()
	778
	779	swarm.on('connection', (socket, peerInfo) => {
	780	core.replicate(peerInfo.client).pipe(socket).pipe(socket)
	781	})
	782
	783	swarm.join(core.discoveryKey, { server: true, client: true })
	784	await swarm.flush()
	785
	786	return { core, swarm }
	787	}
	788
	789	// Usage
	790	const { core, swarm } = await replicatedCore('./storage')
	791	await core.append('Hello P2P!')
	792	```
	793
	794	---
	795
	796	### Pattern 2: Hyperswarm + Corestore (Multi-Core)
	797
	798	```js
	799	const Corestore = require('corestore')
	800	const Hyperswarm = require('hyperswarm')
	801
	802	class MultiCoreSwarm {
	803	constructor(storage) {
	804	this.store = new Corestore(storage)
	805	this.swarm = new Hyperswarm()
	806	this.topics = new Set()
	807	}
	808
	809	async init() {
	810	await this.store.ready()
	811
	812	// Auto-replicate all cores
	813	this.swarm.on('connection', (socket) => {
	814	this.store.replicate(socket)
	815	})
	816	}
	817
	818	async createCore(name) {
	819	const core = this.store.get({ name })
	820	await core.ready()
	821
	822	// Auto-join topic
	823	this.addTopic(core.discoveryKey)
	824
	825	return core
	826	}
	827
	828	async getCore(key) {
	829	const core = this.store.get(key)
	830	await core.ready()
	831
	832	// Auto-join topic
	833	this.addTopic(core.discoveryKey)
	834
	835	return core
	836	}
	837
	838	addTopic(topic) {
	839	const hex = topic.toString('hex')
	840	if (!this.topics.has(hex)) {
	841	this.swarm.join(topic, { server: true, client: true })
	842	this.topics.add(hex)
	843	}
	844	}
	845
	846	async destroy() {
	847	await this.swarm.destroy()
	848	await this.store.close()
	849	}
	850	}
	851
	852	// Usage
	853	const mcs = new MultiCoreSwarm('./storage')
	854	await mcs.init()
	855
	856	const core1 = await mcs.createCore('posts')
	857	const core2 = await mcs.createCore('comments')
	858
	859	// Both cores auto-discovered and replicated!
	860	```
	861
	862	---
	863
	864	### Pattern 3: DHT Relay for Restricted Networks
	865
	866	```js
	867	const Hyperswarm = require('hyperswarm')
	868
	869	// For peers behind restrictive firewall/NAT
	870	const swarm = new Hyperswarm({
	871	// Use relay server when direct connection fails
	872	relay: {
	873	servers: ['wss://relay.example.com']
	874	}
	875	})
	876
	877	swarm.on('connection', (socket, peerInfo) => {
	878	if (peerInfo.relayed) {
	879	console.log('Connected via relay!')
	880	} else {
	881	console.log('Direct P2P connection!')
	882	}
	883
	884	// Socket works the same either way
	885	core.replicate(peerInfo.client).pipe(socket).pipe(socket)
	886	})
	887
	888	swarm.join(topic)
	889	```
	890
	891	Use case: Corporate networks, mobile networks with symmetric NAT
	892
	893	---
	894
	895	### Pattern 4: Custom Bootstrap Servers
	896
	897	```js
	898	const Hyperswarm = require('hyperswarm')
	899
	900	// Use custom DHT bootstrap servers
	901	const swarm = new Hyperswarm({
	902	bootstrap: [
	903	'10.0.0.1:49737', // Your private DHT node
	904	'10.0.0.2:49737'
	905	]
	906	})
	907
	908	// Creates isolated P2P network (not connected to public DHT)
	909
	910	swarm.on('connection', (socket) => {
	911	console.log('Connected in private network')
	912	})
	913
	914	swarm.join(topic)
	915	```
	916
	917	Use case: Private networks, testing, corporate deployments
	918
	919	Source: holepunchto/hyperdht/README.md:69-84
	920
	921	---
	922
	923	## See Also
	924
	925	### Pattern Documentation
	926	- [replication.md](./replication.md) - Deep dive on replication strategies
	927	- [append-only-log.md](./append-only-log.md) - Hypercore basics
	928	- [cryptography.md](./cryptography.md) - DiscoveryKey derivation, encryption
	929	- [streaming.md](./streaming.md) - Connection management
	930
	931	### Deployment Documentation
	932	- [../deployment/pear-apps.md](../deployment/pear-apps.md) - Deploying P2P apps
	933	- [../deployment/systemd-services.md](../deployment/systemd-services.md) - Running as service
	934	- [../deployment/troubleshooting.md](../deployment/troubleshooting.md) - Connection issues
	935
	936	### Repository Documentation
	937	- [../skills/hyperswarm/skills.md](../skills/hyperswarm/skills.md) - Hyperswarm API
	938	- [../skills/hyperdht/skills.md](../skills/hyperdht/skills.md) - HyperDHT API
	939	- [../skills/hypercore/skills.md](../skills/hypercore/skills.md) - Hypercore API
	940
	941	### Official Resources
	942	- [Hyperswarm Docs](https://docs.pears.com/building-blocks/hyperswarm)
	943	- [HyperDHT Docs](https://docs.pears.com/building-blocks/hyperdht)
	944	- [Holepunch.to](https://holepunch.to/)
	945
	946	---
	947
	948	## References
	949
	950	- Tier 1 Repos: hyperswarm, hyperdht
	951	- Real-World Example: hyper-fleeky/repos/fedicore (multi-user discovery)
	952	- Pattern Category: Networking & Discovery
	953	- Dependencies: hyperdht (DHT), @hyperswarm/secret-stream (Noise), dht-rpc (base DHT)
	954	- Use Cases: P2P discovery, serverless apps, mesh networks, distributed systems
	955
	956	---
	957
	958	Last Updated: 2026-01-18
	959	Validated Against: Hyperswarm 4.x, HyperDHT 6.x, Fedicore 1.0

patterns/replication.md +1043 -0

…	…	@@ -0,0 +1,1043 @@
	1	# Pattern: Replication
	2
	3	Validated against: holepunchto/hypercore, holepunchto/hyperswarm, hyper-fleeky/fedicore
	4
	5	## Description
	6
	7	Replication is the process of synchronizing append-only logs (Hypercores) between peers in a P2P network. It handles:
	8
	9	- Peer discovery: Finding peers interested in the same data
	10	- Connection management: Establishing encrypted connections
	11	- Data transfer: Efficiently syncing blocks between peers
	12	- Protocol multiplexing: Running multiple protocols over one connection
	13	- Sparse sync: Only downloading blocks you need
	14
	15	Core components:
	16	1. Hypercore - The append-only log being replicated
	17	2. Hyperswarm - DHT-based peer discovery and connection management
	18	3. Protomux - Multiplexing multiple protocols over a connection
	19	4. Secret-stream - Noise protocol encrypted connections
	20
	21	Key insight: Replication in the Hypercore ecosystem is push/pull based with cryptographic verification. Peers can replicate multiple Hypercores over a single encrypted connection.
	22
	23	---
	24
	25	## When to Use
	26
	27	✅ Use replication when:
	28	- Distributing data across a P2P network
	29	- Building distributed databases
	30	- Creating multi-device sync systems
	31	- Implementing real-time collaborative apps
	32	- Building decentralized social networks
	33	- Sharing large datasets efficiently
	34	- Creating offline-first applications
	35
	36	❌ Don't replicate when:
	37	- Data is private and should never leave one device (use local-only Hypercore)
	38	- You need centralized access control (P2P is permissionless by design)
	39	- Network overhead is unacceptable (replication has bandwidth costs)
	40	- You only have a single device (no peers to replicate with)
	41
	42	---
	43
	44	## Implementing Repos
	45
	46	### Tier 1 (Core Building Blocks)
	47
	48	- [hypercore](https://github.com/holepunchto/hypercore) (2,745 ⭐)
	49	- Implements `.replicate()` method
	50	- Handles merkle tree proofs and block verification
	51	- `npm install hypercore`
	52
	53	- [hyperswarm](https://github.com/holepunchto/hyperswarm) (1,275 ⭐)
	54	- DHT-based peer discovery
	55	- Manages connections automatically
	56	- `npm install hyperswarm`
	57
	58	### Tier 2 (Helpers)
	59
	60	- [protomux](https://github.com/holepunchto/protomux) (122 ⭐)
	61	- Multiplex multiple protocols over one connection
	62	- Used internally by Hypercore replication
	63	- `npm install protomux`
	64
	65	- [corestore](https://github.com/holepunchto/corestore) (184 ⭐)
	66	- Manages multiple Hypercores efficiently
	67	- `.replicate()` replicates all cores at once
	68	- `npm install corestore`
	69
	70	---
	71
	72	## Code Example: Basic Replication
	73
	74	Replicating a single Hypercore between two peers:
	75
	76	```js
	77	const Hypercore = require('hypercore')
	78	const net = require('net')
	79
	80	// ========================================
	81	// PEER 1: Server (has the data)
	82	// ========================================
	83
	84	const core1 = new Hypercore('./peer1-storage')
	85	await core1.ready()
	86
	87	// Add some data
	88	await core1.append('Block 0')
	89	await core1.append('Block 1')
	90	await core1.append('Block 2')
	91
	92	console.log('Peer 1 key:', core1.key.toString('hex'))
	93	console.log('Peer 1 length:', core1.length) // 3
	94
	95	// Create server
	96	const server = net.createServer((socket) => {
	97	console.log('Peer 2 connected')
	98
	99	// Replicate (isInitiator = false, we're the server)
	100	const stream = core1.replicate(false)
	101	stream.pipe(socket).pipe(stream)
	102	})
	103
	104	server.listen(10000)
	105	console.log('Server listening on port 10000')
	106
	107	// ========================================
	108	// PEER 2: Client (downloading the data)
	109	// ========================================
	110
	111	// Create core with Peer 1's key (read-only)
	112	const core2 = new Hypercore('./peer2-storage', peer1Key)
	113	await core2.ready()
	114
	115	console.log('Peer 2 writable:', core2.writable) // false
	116
	117	// Connect to server
	118	const socket = net.connect(10000)
	119
	120	// Replicate (isInitiator = true, we're the client)
	121	const stream = core2.replicate(true)
	122	stream.pipe(socket).pipe(stream)
	123
	124	// Wait for sync
	125	await core2.update()
	126	console.log('Peer 2 length:', core2.length) // 3
	127
	128	// Read replicated data
	129	const block0 = await core2.get(0)
	130	const block1 = await core2.get(1)
	131	const block2 = await core2.get(2)
	132
	133	console.log(block0.toString()) // 'Block 0'
	134	console.log(block1.toString()) // 'Block 1'
	135	console.log(block2.toString()) // 'Block 2'
	136
	137	// Listen for new entries
	138	core2.on('append', () => {
	139	console.log('New blocks arrived! Length:', core2.length)
	140	})
	141	```
	142
	143	Key points:
	144	- Server uses `replicate(false)` (passive)
	145	- Client uses `replicate(true)` (initiator)
	146	- `core.update()` waits for remote length proof
	147	- Replication is continuous - new appends auto-sync
	148
	149	Source: holepunchto/hypercore/README.md:659-682
	150
	151	---
	152
	153	## Code Example: Replication with Hyperswarm
	154
	155	Using Hyperswarm for automatic peer discovery and connection:
	156
	157	```js
	158	const Hypercore = require('hypercore')
	159	const Hyperswarm = require('hyperswarm')
	160	const crypto = require('hypercore-crypto')
	161
	162	// ========================================
	163	// PEER 1: Create and share a Hypercore
	164	// ========================================
	165
	166	const core1 = new Hypercore('./peer1-storage')
	167	await core1.ready()
	168
	169	await core1.append('Hello from peer 1')
	170	await core1.append('This is block 1')
	171
	172	console.log('Core key:', core1.key.toString('hex'))
	173
	174	// Create swarm
	175	const swarm1 = new Hyperswarm()
	176
	177	// On connection, replicate the core
	178	swarm1.on('connection', (socket, peerInfo) => {
	179	console.log('Peer connected!')
	180	console.log('Peer topics:', peerInfo.topics.length)
	181
	182	// Replicate over connection
	183	// Use peerInfo.client to determine initiator
	184	const stream = core1.replicate(peerInfo.client)
	185	stream.pipe(socket).pipe(socket)
	186	})
	187
	188	// Join swarm using discoveryKey
	189	const topic = core1.discoveryKey
	190	const discovery = swarm1.join(topic, {
	191	server: true, // Accept incoming connections
	192	client: true // Make outgoing connections
	193	})
	194
	195	// Wait for topic to be fully announced
	196	await discovery.flushed()
	197	console.log('Announced to DHT')
	198
	199	// ========================================
	200	// PEER 2: Replicate from the network
	201	// ========================================
	202
	203	// Create core with Peer 1's key
	204	const core2 = new Hypercore('./peer2-storage', peer1Key)
	205	await core2.ready()
	206
	207	// Create swarm
	208	const swarm2 = new Hyperswarm()
	209
	210	swarm2.on('connection', (socket, peerInfo) => {
	211	console.log('Found peer!')
	212
	213	// Replicate over connection
	214	const stream = core2.replicate(peerInfo.client)
	215	stream.pipe(socket).pipe(socket)
	216	})
	217
	218	// Join same topic
	219	const topic = core2.discoveryKey
	220	swarm2.join(topic, { server: true, client: true })
	221
	222	// Wait for connections
	223	await swarm2.flush()
	224
	225	// Wait for sync
	226	await core2.update()
	227	console.log('Synced! Length:', core2.length)
	228
	229	// Read data
	230	for (let i = 0; i < core2.length; i++) {
	231	const block = await core2.get(i)
	232	console.log(`Block ${i}:`, block.toString())
	233	}
	234
	235	// Real-time updates
	236	core2.on('append', async () => {
	237	const latestBlock = await core2.get(core2.length - 1)
	238	console.log('New block:', latestBlock.toString())
	239	})
	240	```
	241
	242	Key points:
	243	- `discoveryKey` is used for peer discovery (doesn't leak core key)
	244	- `swarm.join()` finds peers interested in same topic
	245	- `peerInfo.client` indicates if we're the initiator
	246	- Replication happens automatically when peers connect
	247	- `swarm.flush()` waits for pending peer connections
	248
	249	Source: holepunchto/hyperswarm/README.md:14-39
	250
	251	---
	252
	253	## Code Example: Multi-Core Replication with Corestore
	254
	255	Replicating multiple Hypercores efficiently:
	256
	257	```js
	258	const Corestore = require('corestore')
	259	const Hyperswarm = require('hyperswarm')
	260
	261	// ========================================
	262	// PEER 1: Create multiple cores
	263	// ========================================
	264
	265	const store1 = new Corestore('./peer1-corestore')
	266	await store1.ready()
	267
	268	// Create multiple cores
	269	const core1a = store1.get({ name: 'posts' })
	270	const core1b = store1.get({ name: 'comments' })
	271	const core1c = store1.get({ name: 'likes' })
	272
	273	await core1a.ready()
	274	await core1b.ready()
	275	await core1c.ready()
	276
	277	// Populate cores
	278	await core1a.append('Post 1')
	279	await core1a.append('Post 2')
	280	await core1b.append('Comment on post 1')
	281	await core1c.append('Like post 1')
	282
	283	console.log('Created 3 cores')
	284
	285	// Create swarm
	286	const swarm1 = new Hyperswarm()
	287
	288	swarm1.on('connection', (socket) => {
	289	console.log('Peer connected')
	290
	291	// Replicate ALL cores in the store over this connection!
	292	store1.replicate(socket)
	293	})
	294
	295	// Announce each core's discovery key
	296	swarm1.join(core1a.discoveryKey)
	297	swarm1.join(core1b.discoveryKey)
	298	swarm1.join(core1c.discoveryKey)
	299
	300	await swarm1.flush()
	301	console.log('Announced all cores to DHT')
	302
	303	// ========================================
	304	// PEER 2: Replicate all cores
	305	// ========================================
	306
	307	const store2 = new Corestore('./peer2-corestore')
	308	await store2.ready()
	309
	310	// Get same cores by key
	311	const core2a = store2.get(core1a.key)
	312	const core2b = store2.get(core1b.key)
	313	const core2c = store2.get(core1c.key)
	314
	315	await core2a.ready()
	316	await core2b.ready()
	317	await core2c.ready()
	318
	319	// Create swarm
	320	const swarm2 = new Hyperswarm()
	321
	322	swarm2.on('connection', (socket) => {
	323	console.log('Connected to peer')
	324
	325	// Replicate ALL cores in the store!
	326	store2.replicate(socket)
	327	})
	328
	329	// Join all topics
	330	swarm2.join(core2a.discoveryKey)
	331	swarm2.join(core2b.discoveryKey)
	332	swarm2.join(core2c.discoveryKey)
	333
	334	await swarm2.flush()
	335
	336	// Wait for all cores to sync
	337	await Promise.all([
	338	core2a.update(),
	339	core2b.update(),
	340	core2c.update()
	341	])
	342
	343	console.log('Posts length:', core2a.length) // 2
	344	console.log('Comments length:', core2b.length) // 1
	345	console.log('Likes length:', core2c.length) // 1
	346
	347	// All cores are synced!
	348	```
	349
	350	Why Corestore:
	351	- Manages multiple Hypercores efficiently
	352	- Single `store.replicate(socket)` replicates all cores
	353	- Shared storage optimizations
	354	- Better than managing individual core replications
	355
	356	Source: Based on corestore replication pattern
	357
	358	---
	359
	360	## Code Example: Live Replication Stream
	361
	362	Setting up a live feed that continuously updates:
	363
	364	```js
	365	const core = new Hypercore('./live-feed', remoteKey)
	366	await core.ready()
	367
	368	// Connect to swarm
	369	const swarm = new Hyperswarm()
	370	swarm.on('connection', (socket, peerInfo) => {
	371	core.replicate(peerInfo.client).pipe(socket).pipe(socket)
	372	})
	373
	374	swarm.join(core.discoveryKey)
	375	await swarm.flush()
	376
	377	// Create live read stream
	378	const liveStream = core.createReadStream({
	379	live: true, // Keep stream open for new entries
	380	start: 0, // Start from beginning
	381	wait: true // Wait for remote blocks
	382	})
	383
	384	console.log('Streaming live feed...')
	385
	386	for await (const data of liveStream) {
	387	console.log('Block:', data.toString())
	388	// Stream never ends - keeps receiving new blocks
	389	}
	390
	391	// Alternative: Event-based
	392	core.on('download', (index, data) => {
	393	console.log('Downloaded block', index, ':', data.length, 'bytes')
	394	})
	395
	396	core.on('append', () => {
	397	console.log('New block appended. Length:', core.length)
	398	})
	399	```
	400
	401	Live stream benefits:
	402	- Automatically receives new blocks as they're appended
	403	- Efficient - no polling needed
	404	- Works for real-time applications (chat, feeds, logs)
	405
	406	Source: holepunchto/hypercore/README.md:216-244
	407
	408	---
	409
	410	## Real-World Usage
	411
	412	### Example: Fedicore Multi-User Feed Replication
	413
	414	From hyper-fleeky/repos/fedicore/fedicore.js - P2P social network:
	415
	416	```js
	417	export class Fedicore extends EventEmitter {
	418	async init(options = {}) {
	419	// Corestore manages all user feeds
	420	this.store = new Corestore(join(this.storagePath, 'feeds'))
	421	await this.store.ready()
	422
	423	// Personal feed (writable)
	424	this.myFeed = this.store.get({ name: 'my-feed' })
	425	await this.myFeed.ready()
	426
	427	// Hyperswarm for P2P connections
	428	this.swarm = new Hyperswarm()
	429
	430	// KEY PATTERN: Auto-replicate ALL feeds on every connection
	431	this.swarm.on('connection', (conn) => {
	432	console.log('Peer connected - replicating all feeds')
	433
	434	// Replicate entire Corestore (personal + all followed feeds)
	435	this.store.replicate(conn)
	436	})
	437
	438	// Join swarm for own feed
	439	const topic = this.myFeed.discoveryKey
	440	this.swarm.join(topic, { server: true, client: true })
	441	await this.swarm.flush()
	442
	443	console.log('Fedicore ready - replicating with network')
	444	}
	445
	446	// Follow another user = replicate their feed
	447	async follow(publicKey, name = 'Anonymous') {
	448	// Get their feed (read-only)
	449	const feedKey = b4a.from(publicKey, 'base64')
	450	const feed = this.store.get(feedKey)
	451	await feed.ready()
	452
	453	this.follows.set(publicKey, { feed, name })
	454
	455	// Join their topic for peer discovery
	456	const topic = feed.discoveryKey
	457	this.swarm.join(topic, { server: false, client: true })
	458
	459	console.log('Following', name, '- will replicate their posts')
	460
	461	// Listen for new posts from this user
	462	feed.on('append', async () => {
	463	const latestIndex = feed.length - 1
	464	const block = await feed.get(latestIndex)
	465	const post = JSON.parse(block)
	466
	467	this.emit('post', {
	468	author: name,
	469	content: post,
	470	timestamp: post.timestamp
	471	})
	472	})
	473
	474	// Trigger initial sync
	475	await feed.update()
	476	console.log('Synced', feed.length, 'posts from', name)
	477	}
	478
	479	// Get timeline (all followed users' posts)
	480	async getTimeline(limit = 50) {
	481	const posts = []
	482
	483	// Read from own feed
	484	for (let i = Math.max(0, this.myFeed.length - limit); i < this.myFeed.length; i++) {
	485	const block = await this.myFeed.get(i)
	486	posts.push({
	487	author: 'You',
	488	content: JSON.parse(block),
	489	index: i
	490	})
	491	}
	492
	493	// Read from all followed feeds
	494	for (const [publicKey, { feed, name }] of this.follows) {
	495	await feed.update() // Ensure we have latest
	496
	497	for (let i = Math.max(0, feed.length - limit); i < feed.length; i++) {
	498	const block = await feed.get(i)
	499	posts.push({
	500	author: name,
	501	content: JSON.parse(block),
	502	index: i
	503	})
	504	}
	505	}
	506
	507	// Sort by timestamp
	508	posts.sort((a, b) => a.content.timestamp - b.content.timestamp)
	509
	510	return posts.slice(-limit)
	511	}
	512	}
	513	```
	514
	515	Key replication patterns:
	516	1. One connection replicates all feeds: `store.replicate(conn)`
	517	2. Follow = join topic: Joining discoveryKey finds peers with that feed
	518	3. Live updates via events: `feed.on('append')` for real-time posts
	519	4. Sparse replication: Only download recent posts with `limit`
	520	5. Multi-writer via separate feeds: Each user has own writable feed
	521
	522	Source: hyper-fleeky/repos/fedicore/fedicore.js:44-73
	523
	524	---
	525
	526	## Common Gotchas
	527
	528	### 1. Wrong replication initiator
	529
	530	❌ Wrong:
	531	```js
	532	// Server
	533	socket.pipe(core.replicate(true)).pipe(socket) // ❌
	534
	535	// Client
	536	socket.pipe(core.replicate(true)).pipe(socket) // ❌ Both can't be initiator!
	537	```
	538
	539	✅ Correct:
	540	```js
	541	// Server
	542	socket.pipe(core.replicate(false)).pipe(socket) // Passive
	543
	544	// Client
	545	socket.pipe(core.replicate(true)).pipe(socket) // Initiator
	546	```
	547
	548	Why: Noise protocol requires one initiator and one responder. Having both as initiator causes handshake failure.
	549
	550	Fix for Hyperswarm: Use `peerInfo.client`
	551	```js
	552	swarm.on('connection', (socket, peerInfo) => {
	553	// peerInfo.client tells us if we're initiator
	554	core.replicate(peerInfo.client).pipe(socket).pipe(socket)
	555	})
	556	```
	557
	558	---
	559
	560	### 2. Not waiting for `swarm.flush()`
	561
	562	❌ Wrong:
	563	```js
	564	swarm.join(topic)
	565	// Immediately try to use connections - may not be connected yet!
	566	await core.update() // Might timeout
	567	```
	568
	569	✅ Correct:
	570	```js
	571	swarm.join(topic)
	572	await swarm.flush() // Wait for connections to peers
	573	await core.update() // Now we have peers
	574	```
	575
	576	Why: `swarm.join()` is non-blocking. `swarm.flush()` waits for DHT announce and pending peer connections.
	577
	578	---
	579
	580	### 3. Forgetting to announce as server
	581
	582	❌ Wrong:
	583	```js
	584	// Only client mode - can find servers but others can't find you
	585	swarm.join(topic, { server: false, client: true })
	586	```
	587
	588	✅ Correct:
	589	```js
	590	// Both modes - you can be found AND find others
	591	swarm.join(topic, { server: true, client: true })
	592	```
	593
	594	Why: If everyone uses client-only mode, nobody announces to DHT and peers can't find each other. At least some peers need `server: true`.
	595
	596	---
	597
	598	### 4. Not handling connection errors
	599
	600	❌ Wrong:
	601	```js
	602	swarm.on('connection', (socket) => {
	603	core.replicate(false).pipe(socket).pipe(socket)
	604	// No error handling!
	605	})
	606	```
	607
	608	✅ Correct:
	609	```js
	610	swarm.on('connection', (socket) => {
	611	socket.on('error', (err) => {
	612	console.error('Socket error:', err)
	613	})
	614
	615	const stream = core.replicate(false)
	616	stream.on('error', (err) => {
	617	console.error('Replication error:', err)
	618	})
	619
	620	stream.pipe(socket).pipe(socket)
	621	})
	622	```
	623
	624	Why: Network errors, peer disconnections, and protocol errors happen. Unhandled errors crash the process.
	625
	626	---
	627
	628	### 5. Replicating with wrong key
	629
	630	❌ Wrong:
	631	```js
	632	// Using the core's public key as the topic
	633	swarm.join(core.key) // ❌ Leaks your core key to DHT!
	634	```
	635
	636	✅ Correct:
	637	```js
	638	// Use discoveryKey instead
	639	swarm.join(core.discoveryKey) // ✅ Doesn't leak core key
	640	```
	641
	642	Why: `discoveryKey` is a hash of `core.key`. It allows peer discovery without leaking the actual core key (which could allow verification of data integrity).
	643
	644	---
	645
	646	### 6. Not closing swarms
	647
	648	❌ Wrong:
	649	```js
	650	async function tempReplicate() {
	651	const swarm = new Hyperswarm()
	652	swarm.on('connection', (socket) => {
	653	core.replicate(false).pipe(socket).pipe(socket)
	654	})
	655	swarm.join(topic)
	656	// Forgot to destroy!
	657	} // Memory leak, DHT pollution
	658	```
	659
	660	✅ Correct:
	661	```js
	662	async function tempReplicate() {
	663	const swarm = new Hyperswarm()
	664	swarm.on('connection', (socket) => {
	665	core.replicate(false).pipe(socket).pipe(socket)
	666	})
	667	swarm.join(topic)
	668
	669	// Do work...
	670
	671	await swarm.leave(topic)
	672	await swarm.destroy()
	673	}
	674	```
	675
	676	Why: Swarms hold DHT connections and network sockets. Must destroy when done.
	677
	678	---
	679
	680	## Anti-Patterns
	681
	682	### ❌ 1. Creating new swarm for each core
	683
	684	Don't:
	685	```js
	686	// Bad: One swarm per core
	687	const swarm1 = new Hyperswarm()
	688	swarm1.on('connection', (s) => core1.replicate(s.client).pipe(s).pipe(s))
	689	swarm1.join(core1.discoveryKey)
	690
	691	const swarm2 = new Hyperswarm()
	692	swarm2.on('connection', (s) => core2.replicate(s.client).pipe(s).pipe(s))
	693	swarm2.join(core2.discoveryKey)
	694
	695	// Wastes connections, DHT resources
	696	```
	697
	698	Do: Use Corestore or multiplex over one swarm
	699	```js
	700	// Good: One swarm, one connection, all cores
	701	const store = new Corestore('./storage')
	702	const core1 = store.get({ name: 'core1' })
	703	const core2 = store.get({ name: 'core2' })
	704
	705	const swarm = new Hyperswarm()
	706	swarm.on('connection', (s) => {
	707	store.replicate(s) // Replicates all cores over one connection
	708	})
	709
	710	swarm.join(core1.discoveryKey)
	711	swarm.join(core2.discoveryKey)
	712	```
	713
	714	Why: Each swarm/connection has overhead. Protomux allows multiplexing multiple cores over one connection.
	715
	716	---
	717
	718	### ❌ 2. Polling for updates instead of using events
	719
	720	Don't:
	721	```js
	722	// Bad: Polling
	723	setInterval(async () => {
	724	const oldLength = core.length
	725	await core.update()
	726	if (core.length > oldLength) {
	727	console.log('New blocks!')
	728	// Process new blocks
	729	}
	730	}, 1000) // Wastes CPU, network
	731	```
	732
	733	Do: Use events
	734	```js
	735	// Good: Event-driven
	736	core.on('append', async () => {
	737	console.log('New blocks!')
	738	const latestBlock = await core.get(core.length - 1)
	739	// Process immediately
	740	})
	741	```
	742
	743	Why: Events are immediate and efficient. Polling wastes resources and adds latency.
	744
	745	---
	746
	747	### ❌ 3. Downloading everything in sparse mode
	748
	749	Don't:
	750	```js
	751	// Bad: Defeats the purpose of sparse replication
	752	const core = new Hypercore('./storage', remoteKey)
	753	// ... replicate ...
	754
	755	// Download all blocks immediately
	756	for (let i = 0; i < core.length; i++) {
	757	await core.get(i) // Downloading everything!
	758	}
	759	```
	760
	761	Do: Download only what you need
	762	```js
	763	// Good: Sparse download
	764	const core = new Hypercore('./storage', remoteKey)
	765	// ... replicate ...
	766
	767	// Only get recent blocks
	768	const recentStart = Math.max(0, core.length - 10)
	769	for (let i = recentStart; i < core.length; i++) {
	770	await core.get(i) // Only last 10 blocks
	771	}
	772
	773	// Or download on demand
	774	const specificBlock = await core.get(someIndex) // Only this one
	775	```
	776
	777	Why: Sparse replication is a key feature. Don't negate it by downloading everything immediately.
	778
	779	---
	780
	781	### ❌ 4. Not using discoveryKey for different topics
	782
	783	Don't:
	784	```js
	785	// Bad: Reusing same topic for different cores
	786	const topic = Buffer.alloc(32).fill('my-app')
	787
	788	swarm.join(topic) // All cores use same topic
	789	// Now all peers will connect even if they don't share cores
	790	```
	791
	792	Do: Use each core's discoveryKey
	793	```js
	794	// Good: Each core has unique topic
	795	swarm.join(core1.discoveryKey) // Unique to core1
	796	swarm.join(core2.discoveryKey) // Unique to core2
	797
	798	// Peers only connect if they share specific cores
	799	```
	800
	801	Why: Using the same topic for all cores creates unnecessary connections. DiscoveryKeys are unique per core.
	802
	803	---
	804
	805	## Integration Patterns
	806
	807	### Pattern 1: Hypercore + Hyperswarm (Standard Replication)
	808
	809	```js
	810	const Hypercore = require('hypercore')
	811	const Hyperswarm = require('hyperswarm')
	812
	813	async function createReplicatedCore(storage, key = null) {
	814	const core = new Hypercore(storage, key)
	815	await core.ready()
	816
	817	const swarm = new Hyperswarm()
	818
	819	swarm.on('connection', (socket, peerInfo) => {
	820	console.log('Peer connected')
	821	const stream = core.replicate(peerInfo.client)
	822	stream.pipe(socket).pipe(socket)
	823	})
	824
	825	// Join using discoveryKey
	826	const topic = core.discoveryKey
	827	swarm.join(topic, { server: true, client: true })
	828	await swarm.flush()
	829
	830	return { core, swarm }
	831	}
	832
	833	// Create writable core
	834	const { core: writer, swarm: swarm1 } = await createReplicatedCore('./writer')
	835	await writer.append('Hello')
	836
	837	// Create read-only replica
	838	const { core: reader, swarm: swarm2 } = await createReplicatedCore('./reader', writer.key)
	839	await reader.update()
	840	console.log('Replicated:', await reader.get(0)) // 'Hello'
	841	```
	842
	843	See: [hyperswarm/skills.md](../skills/hyperswarm/skills.md)
	844
	845	---
	846
	847	### Pattern 2: Corestore + Hyperswarm (Multi-Core)
	848
	849	```js
	850	const Corestore = require('corestore')
	851	const Hyperswarm = require('hyperswarm')
	852
	853	class ReplicatedStore {
	854	constructor(storage) {
	855	this.store = new Corestore(storage)
	856	this.swarm = new Hyperswarm()
	857	this.topics = new Set()
	858	}
	859
	860	async init() {
	861	await this.store.ready()
	862
	863	// Auto-replicate all cores
	864	this.swarm.on('connection', (socket) => {
	865	this.store.replicate(socket)
	866	})
	867	}
	868
	869	async createCore(name) {
	870	const core = this.store.get({ name })
	871	await core.ready()
	872
	873	// Auto-join swarm
	874	if (!this.topics.has(core.discoveryKey.toString('hex'))) {
	875	this.swarm.join(core.discoveryKey)
	876	this.topics.add(core.discoveryKey.toString('hex'))
	877	}
	878
	879	return core
	880	}
	881
	882	async destroy() {
	883	for (const topic of this.topics) {
	884	await this.swarm.leave(Buffer.from(topic, 'hex'))
	885	}
	886	await this.swarm.destroy()
	887	await this.store.close()
	888	}
	889	}
	890
	891	// Usage
	892	const store = new ReplicatedStore('./storage')
	893	await store.init()
	894
	895	const core1 = await store.createCore('posts')
	896	const core2 = await store.createCore('comments')
	897
	898	// Both cores auto-replicate over the same connection!
	899	```
	900
	901	See: [corestore/skills.md](../skills/corestore/skills.md)
	902
	903	---
	904
	905	### Pattern 3: Hypercore + Custom Protocol (Protomux)
	906
	907	```js
	908	const Hypercore = require('hypercore')
	909	const Hyperswarm = require('hyperswarm')
	910	const Protomux = require('protomux')
	911	const c = require('compact-encoding')
	912
	913	async function createCoreWithProtocol(storage) {
	914	const core = new Hypercore(storage)
	915	await core.ready()
	916
	917	const swarm = new Hyperswarm()
	918
	919	swarm.on('connection', (socket, peerInfo) => {
	920	// Get/create protomux for this connection
	921	const mux = Protomux.from(socket)
	922
	923	// Hypercore replication protocol
	924	core.replicate(peerInfo.client).pipe(socket).pipe(socket)
	925
	926	// Custom protocol on same connection
	927	const custom = mux.createChannel({
	928	protocol: 'my-custom-protocol',
	929	onopen() {
	930	console.log('Custom protocol opened!')
	931	}
	932	})
	933
	934	const ping = custom.addMessage({
	935	encoding: c.string,
	936	onmessage(msg) {
	937	console.log('Received:', msg)
	938	}
	939	})
	940
	941	custom.open()
	942	ping.send('Hello from custom protocol!')
	943	})
	944
	945	swarm.join(core.discoveryKey)
	946	await swarm.flush()
	947
	948	return { core, swarm }
	949	}
	950
	951	// Both Hypercore replication AND custom protocol over same connection!
	952	```
	953
	954	See: [protomux/skills.md](../skills/protomux/skills.md)
	955
	956	---
	957
	958	### Pattern 4: Hypercore + Encryption
	959
	960	```js
	961	const Hypercore = require('hypercore')
	962	const crypto = require('hypercore-crypto')
	963
	964	async function createEncryptedCore(storage, encryptionKey = null) {
	965	// Generate encryption key if not provided
	966	if (!encryptionKey) {
	967	encryptionKey = crypto.randomBytes(32)
	968	}
	969
	970	const core = new Hypercore(storage, {
	971	encryption: { key: encryptionKey }
	972	})
	973
	974	await core.ready()
	975
	976	console.log('Encryption key:', encryptionKey.toString('hex'))
	977	console.log('Core key (public):', core.key.toString('hex'))
	978
	979	return { core, encryptionKey }
	980	}
	981
	982	// Create encrypted core
	983	const { core: writer, encryptionKey } = await createEncryptedCore('./encrypted')
	984
	985	// Write encrypted data
	986	await writer.append('Secret message')
	987
	988	// Replicate to another peer (they need both core key AND encryption key)
	989	const reader = new Hypercore('./replica', writer.key, {
	990	encryption: { key: encryptionKey } // Must have encryption key!
	991	})
	992
	993	await reader.ready()
	994	// ... replicate as normal ...
	995
	996	const block = await reader.get(0)
	997	console.log(block.toString()) // 'Secret message' (decrypted)
	998	```
	999
	1000	Why: Block encryption adds privacy. Peers need both the core key (for verification) and encryption key (for decryption).
	1001
	1002	See: [cryptography.md](./cryptography.md)
	1003
	1004	---
	1005
	1006	## See Also
	1007
	1008	### Pattern Documentation
	1009	- [append-only-log.md](./append-only-log.md) - Hypercore append-only log basics
	1010	- [peer-discovery.md](./peer-discovery.md) - DHT-based peer discovery
	1011	- [cryptography.md](./cryptography.md) - Merkle trees and encryption
	1012	- [streaming.md](./streaming.md) - Efficient data streaming
	1013
	1014	### Deployment Documentation
	1015	- [../deployment/pear-apps.md](../deployment/pear-apps.md) - Deploying P2P apps
	1016	- [../deployment/systemd-services.md](../deployment/systemd-services.md) - Running as service
	1017	- [../deployment/troubleshooting.md](../deployment/troubleshooting.md) - Common issues
	1018
	1019	### Repository Documentation
	1020	- [../skills/hypercore/skills.md](../skills/hypercore/skills.md) - Hypercore API
	1021	- [../skills/hyperswarm/skills.md](../skills/hyperswarm/skills.md) - Hyperswarm API
	1022	- [../skills/corestore/skills.md](../skills/corestore/skills.md) - Corestore API
	1023	- [../skills/protomux/skills.md](../skills/protomux/skills.md) - Protomux API
	1024
	1025	### Official Resources
	1026	- [Hypercore Replication Docs](https://docs.pears.com/building-blocks/hypercore#replication)
	1027	- [Hyperswarm Docs](https://docs.pears.com/building-blocks/hyperswarm)
	1028	- [Holepunch.to](https://holepunch.to/)
	1029
	1030	---
	1031
	1032	## References
	1033
	1034	- Tier 1 Repos: hypercore, hyperswarm
	1035	- Tier 2 Repos: protomux, corestore
	1036	- Real-World Example: hyper-fleeky/repos/fedicore (multi-user feed replication)
	1037	- Pattern Category: Networking & Sync
	1038	- Dependencies: @hyperswarm/secret-stream (Noise encryption), hyperdht (DHT), protomux (multiplexing)
	1039
	1040	---
	1041
	1042	Last Updated: 2026-01-18
	1043	Validated Against: Hypercore 10.x, Hyperswarm 4.x, Fedicore 1.0

patterns/stream-multiplexing.md +100 -0

…	…	@@ -0,0 +1,100 @@
	1	# Pattern: Stream Multiplexing
	2
	3	Validated against: holepunchto/protomux
	4
	5	## Description
	6
	7	Multiplexing multiple message-oriented protocols over a single stream connection. Protomux allows running multiple protocols (Hypercore replication, custom protocols, etc.) over one encrypted connection.
	8
	9	Key benefits:
	10	- Efficient: One connection for many protocols
	11	- Message-oriented: Preserves message boundaries
	12	- Protocol isolation: Each protocol has own channel
	13	- Hypercore integration: Used internally by Hypercore replication
	14
	15	---
	16
	17	## Code Example: Basic Protomux
	18
	19	```js
	20	const Protomux = require('protomux')
	21	const c = require('compact-encoding')
	22
	23	// Create mux over framed stream (e.g., Hyperswarm socket)
	24	const mux = new Protomux(framedStream)
	25
	26	// ========================================
	27	// Create Protocol Channel
	28	// ========================================
	29
	30	const channel = mux.createChannel({
	31	protocol: 'my-protocol',
	32	id: Buffer.from('optional-id'),
	33	onopen() {
	34	console.log('Channel opened!')
	35	},
	36	onclose() {
	37	console.log('Channel closed')
	38	}
	39	})
	40
	41	// ========================================
	42	// Add Message Types
	43	// ========================================
	44
	45	const ping = channel.addMessage({
	46	encoding: c.string,
	47	onmessage(msg) {
	48	console.log('Received ping:', msg)
	49	}
	50	})
	51
	52	const pong = channel.addMessage({
	53	encoding: c.bool,
	54	onmessage(msg) {
	55	console.log('Received pong:', msg)
	56	}
	57	})
	58
	59	// Open channel
	60	channel.open()
	61
	62	// Send messages
	63	ping.send('Hello')
	64	pong.send(true)
	65
	66	// ========================================
	67	// Multiple Protocols on Same Connection
	68	// ========================================
	69
	70	// Protocol 1: Chat
	71	const chat = mux.createChannel({ protocol: 'chat' })
	72	const chatMsg = chat.addMessage({
	73	encoding: c.string,
	74	onmessage(msg) {
	75	console.log('Chat:', msg)
	76	}
	77	})
	78
	79	// Protocol 2: File transfer
	80	const files = mux.createChannel({ protocol: 'files' })
	81	const fileMsg = files.addMessage({
	82	encoding: c.binary,
	83	onmessage(data) {
	84	console.log('File chunk:', data.length, 'bytes')
	85	}
	86	})
	87
	88	// Protocol 3: Hypercore replication
	89	const core = new Hypercore('./storage')
	90	core.replicate(framedStream) // Uses Protomux internally
	91
	92	// All 3 protocols share same connection!
	93	```
	94
	95	See: [replication.md](./replication.md#pattern-3-hypercore--custom-protocol-protomux)
	96
	97	---
	98
	99	Source: holepunchto/protomux/README.md
	100	Last Updated: 2026-01-18

patterns/streaming.md +98 -0

…	…	@@ -0,0 +1,98 @@
	1	# Pattern: Streaming
	2
	3	Validated against: holepunchto/hypercore
	4
	5	## Description
	6
	7	Efficient data streaming for reading and writing large datasets without loading everything into memory. Hypercore provides read streams, write streams, and byte streams for different use cases.
	8
	9	Stream types:
	10	- Read streams: Iterate over blocks
	11	- Write streams: Append blocks
	12	- Byte streams: Read by byte offset
	13	- Live streams: Continuous updates
	14
	15	---
	16
	17	## Code Example: Basic Streaming
	18
	19	```js
	20	const Hypercore = require('hypercore')
	21
	22	const core = new Hypercore('./storage')
	23	await core.ready()
	24
	25	// ========================================
	26	// Write Stream
	27	// ========================================
	28
	29	const writeStream = core.createWriteStream()
	30
	31	// Write data as chunks
	32	writeStream.write('Block 1')
	33	writeStream.write('Block 2')
	34	writeStream.write('Block 3')
	35
	36	writeStream.end() // Finish writing
	37
	38	// Wait for completion
	39	await new Promise((resolve) => writeStream.on('finish', resolve))
	40
	41	// ========================================
	42	// Read Stream (Full)
	43	// ========================================
	44
	45	const readStream = core.createReadStream()
	46
	47	for await (const data of readStream) {
	48	console.log('Block:', data.toString())
	49	}
	50
	51	// ========================================
	52	// Read Stream (Range)
	53	// ========================================
	54
	55	const rangeStream = core.createReadStream({
	56	start: 1,
	57	end: 3
	58	})
	59
	60	for await (const data of rangeStream) {
	61	console.log('Range block:', data.toString())
	62	}
	63
	64	// ========================================
	65	// Byte Stream
	66	// ========================================
	67
	68	const byteStream = core.createByteStream({
	69	byteOffset: 100,
	70	byteLength: 50
	71	})
	72
	73	for await (const chunk of byteStream) {
	74	console.log('Bytes:', chunk)
	75	}
	76
	77	// ========================================
	78	// Live Stream
	79	// ========================================
	80
	81	const liveStream = core.createReadStream({
	82	live: true,
	83	start: core.length
	84	})
	85
	86	liveStream.on('data', (data) => {
	87	console.log('New block:', data.toString())
	88	})
	89
	90	// Stream never ends - keeps listening for new blocks
	91	```
	92
	93	See: [append-only-log.md](./append-only-log.md#code-example-reading-with-streams)
	94
	95	---
	96
	97	Source: holepunchto/hypercore/README.md
	98	Last Updated: 2026-01-18

process_repos.js +3 -1

…	…	@@ -5,7 +5,9 @@ const path = require('path');
5	5	const { execSync } = require('child_process');
6	6
7	7	// Read repo_map.json
8		const repos = JSON.parse(fs.readFileSync('repo_map.json', 'utf8'));
	8	const repoMapData = JSON.parse(fs.readFileSync('repo_map.json', 'utf8'));
	9	// Support both old array format and new object format with _meta
	10	const repos = repoMapData.repos \|\| repoMapData;
9	11
10	12	// Initialize data structures for semantic maps
11	13	const dependencyGraph = {};

repo_map.json +20 -2

…	…	@@ -1,4 +1,21 @@
1		[
	1	{
	2	"_meta": {
	3	"description": "Master list of all 529 holepunchto repositories with basic metadata",
	4	"documentation": {
	5	"concepts": "See CONCEPTS.md for organized navigation of repos by concept, use case, and category",
	6	"skills": "See skills/ directory for detailed API documentation (skills/{repo-name}/skills.md)"
	7	},
	8	"fields": {
	9	"name": "Repository name",
	10	"description": "Brief description from GitHub",
	11	"language": "Primary programming language",
	12	"url": "GitHub repository URL"
	13	},
	14	"totalRepos": 529,
	15	"lastUpdated": "2026-01-18",
	16	"phase": "5.4 - JSON Metadata Enhancement"
	17	},
	18	"repos": [
2	19	{
3	20	"name": ".github",
4	21	"description": "",
…	…	@@ -3173,4 +3190,5 @@
3173	3190	"language": "JavaScript",
3174	3191	"url": "https://github.com/holepunchto/which-runtime"
3175	3192	}
3176		]
	3193	]
	3194	}

semantic-categories.json +19 -0

…	…	@@ -1,4 +1,23 @@
1	1	{
	2	"_meta": {
	3	"description": "Semantic categorization of all 529 holepunchto repositories",
	4	"documentation": {
	5	"overview": "See CONCEPTS.md#by-repository for navigating repos by tier and category",
	6	"skills": "See skills/ directory for detailed API documentation"
	7	},
	8	"categories": {
	9	"Building Blocks": "Core P2P primitives (hypercore, hyperbee, hyperdrive, autobase, hyperdht, hyperswarm)",
	10	"Helpers": "Support libraries for building blocks (corestore, localdrive, mirror-drive, etc.)",
	11	"Pear Modules": "Application runtime modules (~34 repos)",
	12	"Bare Modules": "JavaScript runtime standard library (~185 repos)",
	13	"Tools": "CLI applications and developer tools (~5 repos)",
	14	"Infrastructure": "Supporting libraries and build tools",
	15	"Examples": "Demo applications and workshops"
	16	},
	17	"totalRepos": 529,
	18	"lastUpdated": "2026-01-18",
	19	"phase": "5.4 - JSON Metadata Enhancement"
	20	},
2	21	".github": {
3	22	"category": "Infrastructure",
4	23	"subcategory": "Miscellaneous",

use-case-map.json +11 -0

…	…	@@ -1,4 +1,15 @@
1	1	{
	2	"_meta": {
	3	"description": "Use case mappings for the Hypercore ecosystem",
	4	"documentation": {
	5	"patterns": "See patterns/ directory for code examples",
	6	"deployment": "See deployment/ directory for production guides",
	7	"concepts": "See CONCEPTS.md for complete use case guides"
	8	},
	9	"realWorldExamples": "See hyper-fleeky/repos/ for production implementations",
	10	"lastUpdated": "2026-01-18",
	11	"phase": "5.4 - JSON Metadata Enhancement"
	12	},
2	13	"useCases": {
3	14	"distributed-database": {
4	15	"description": "Building a distributed key-value database with P2P replication",

holepunchto-kb commits / fa890f8

Changed files (28) +10468 -7