8c1e7e2 · hyperswarm-peer

Changed files (1) +161 -0

docs/SWARMD-DESIGN.md +161 -0

…	…	@@ -0,0 +1,161 @@
	1	# swarmd — a generic C/C++ swarm daemon
	2
	3	Status: DESIGN. No code exists. Written 2026-09-10 so the design survives the
	4	session it was worked out in. Every upstream claim here was read on disk or via
	5	the GitHub API that day; where something is unverified it says so.
	6
	7	## 1. What it is, and what it is NOT
	8
	9	⛔ NOT a hypercore daemon. It cannot offer feeds, Hyperbee, Corestore or
	10	Autobase — those exist only in JS. Verified: `libhc` has
	11	`core/crypto/encodings/hashes/keys/merkle_tree/store` and its only `replicate`
	12	symbol is `hc_hashes_replicate`, a namespace-hash derivation for a handshake.
	13	There is no Hypercore replication in C anywhere. A C program needing a real
	14	hypercore must talk to a Bare daemon.
	15
	16	It IS a swarm daemon, exposing the three things hyperswarm-peer owns:
	17
	18	- peer connectivity (HyperDHT, NAT traversal, Noise)
	19	- multiplexed protocol channels (Protomux)
	20	- content-addressed bulk transfer (liboverlay, hyper-overlay/v2)
	21
	22	Pure C/C++. No JS at runtime. (Code generators run under Node at BUILD time;
	23	their output links C libraries only.)
	24
	25	## 2. Why it exists
	26
	27	`session-daemon` is the embryo and its own header already says the right thing:
	28	*"NOT BLENDER-SPECIFIC ON PURPOSE — exposing a Protomux channel over a swarm
	29	topic to a local socket has nothing to do with Blender."* But it is hardcoded to
	30	one topic, one protocol name, one slot, and it has no control plane.
	31
	32	The daemon is the generic seam: Blender must have one (a shipped Blender
	33	extension is pure Python — `multi-user` 0.8.2 deleted all nine of its pyzmq
	34	wheels for exactly this reason). Godot editor and runtime don't need one but can
	35	use one. Any authoring tool with a Python-only plugin API uses it unchanged.
	36
	37	## 3. ⛔ Do not hand-roll the protocol — generate it
	38
	39	The first sketch of this design hand-rolled length-prefixed frames, a
	40	hand-written type+id header, and hand-written compact-encoding. All three
	41	already exist:
	42
	43	\| layer \| use \| not \|
	44	\|---\|---\|---\|
	45	\| codec \| `libcompact` — ABI-identical to the JS codec \| our hand-written `src/compact_encoding.h` \|
	46	\| structs \| `hyperschema-c` — emits `preencode/encode/decode(compact_state_t*)` \| hand-typed message structs \|
	47	\| RPC \| `hrpc-c` — typed sans-io C over the bare-rpc wire, five handler kinds \| a hand-rolled type/id header \|
	48	\| runtime \| `librpc` \| — \|
	49
	50	`hrpc-c` is sans-io: "it never owns a socket, an event loop, or stream state",
	51	so it composes with a UDS, a Protomux channel, a pipe.
	52
	53	⛔ These generate protocols WE define. They cannot generate `dht_messages.h` —
	54	hyperdht has no `spec/`, no hyperschema dependency, and `lib/messages.js` opens
	55	`const c = require('compact-encoding')`. It hand-writes its wire exactly as we
	56	do. Matching hyperdht stays a hand-written port held to vectors.
	57
	58	⛔ Two memory rules from hrpc-c: every `_encode_` heap-allocates and the
	59	CALLER frees; decoded views point INTO the inbound frame buffer and must be
	60	copied out before it is reused.
	61
	62	## 4. Architecture
	63
	64	main thread UDS accept, per-client I/O, event fan-out, status
	65	\| queues (no lock held across a write)
	66	UDX thread uv loop, udx, protomux, DHT, SwarmManager, overlay
	67
	68	⛔ The UDX thread must never write to a client socket. `session-daemon` does
	69	today, unchecked, on non-blocking fds:
	70
	71	if (write(fd, hdr, 4) != 4) continue;
	72	if (write(fd, data, len) != (ssize_t)len) continue;
	73
	74	A partial write there does not drop a frame — it desynchronises the client's
	75	stream, and the client then reads a garbage length and closes. A slow local
	76	client silently corrupts its own session. The UDX thread enqueues; the main
	77	thread owns every fd, buffers per client, and drops a client that exceeds its
	78	high-water mark rather than writing partially.
	79
	80	## 5. Surface
	81
	82	Two planes over one socket. Control is request/response (hrpc-c unary); data is
	83	broadcast (hrpc-c events) — a grab is not a call, nobody replies.
	84
	85	Session: `HELLO {name, app_identity(32), version} -> {client_id, daemon_pubkey}`,
	86	`STATUS_GET -> {dht_state, bootstrapped, peers_known, peers_connected, connecting, queued, local_clients}`
	87
	88	Swarm: `SWARM_JOIN/LEAVE {topic(32), server, client}`, `PEER_ADD/REMOVE {pubkey(32)}`
	89
	90	Channels: `CHANNEL_OPEN {protocol, id, slots} -> {channel}`, `CHANNEL_CLOSE`,
	91	`SEND {channel, slot, target, bytes} -> {accepted, writable}`
	92
	93	Content: `CONTENT_HASH {path} -> {hash}`, `CONTENT_SERVE {hash, path}`,
	94	`CONTENT_FETCH {hash, dest_path}`, `CONTENT_CANCEL {hash, discard}`
	95
	96	DHT storage: `DHT_PUT/GET`, `DHT_MUTABLE_PUT/GET`
	97
	98	Events: `PEER_UP {peer, pubkey, topics[]}`, `PEER_DOWN`,
	99	`MESSAGE {channel, slot, origin_kind, origin_id, bytes}`, `DRAIN {channel}`,
	100	`STATUS`, `CONTENT_PROGRESS/DONE/FAILED`, `DHT_RESULT`
	101
	102	`origin_kind` separates a remote peer from another local client — that is what
	103	lets a client suppress its own echo, which is impossible today.
	104
	105	## 6. Bulk bytes never cross the socket
	106
	107	A `.res` is 108 MB. Content moves as files: the overlay writes to a path, the
	108	client opens the path, the socket carries a 32-byte hash. So the daemon owns a
	109	content directory, which is also what makes a fetch resumable across restarts.
	110
	111	## 7. What session-daemon lacks
	112
	113	1. per-client output buffering + a high-water policy (§4)
	114	2. backpressure that survives the trip — Protomux already returns writability and
	115	its header says discarding that answer "left every protocol above with no way
	116	to avoid burying the connection". `SEND_RESULT.writable` + `DRAIN` carry it
	117	3. liveness — heartbeat `STATUS` plus socket close. #72: the panel said Connected
	118	while the daemon was gone
	119	4. client ids and app identity, for echo suppression and membership later
	120	5. a stable transport identity (a stored seed, not a fresh keypair per start)
	121	6. discoverability — register the socket by name, as pgh-sidecar's `app-lookup` does
	122	7. a C++ client library. "Any program in C++" is the goal; today the only client
	123	is Python
	124	8. Zig-built, pgh-published multi-arch. A plain g++ build will not load on ashatron
	125
	126	## 8. Gates — all headless, no Blender, no Godot
	127
	128	\| # \| asserts \|
	129	\|---\|---\|
	130	\| D1 \| two daemons on the PUBLIC DHT, two C++ clients, a frame both ways \|
	131	\| D2 \| a client that stops reading is dropped; survivors' streams stay intact \|
	132	\| D3 \| `SEND` past the high-water mark returns `writable=false`, `DRAIN` follows \|
	133	\| D4 \| two channels on one connection, neither starves the other \|
	134	\| D5 \| `PEER_DOWN` on channel close, not on timeout \|
	135	\| D6 \| a 100 MB file crosses by hash and no frame exceeds the cap \|
	136	\| D7 \| `STATUS` reports `dht=down` when the DHT is down \|
	137
	138	D1 and D7 are the gates #68 proved the suite lacked: every existing
	139	cross-machine test passes `--bootstrap` at a local testnet, the one configuration
	140	the products never use.
	141
	142	## 9. Order
	143
	144	1. `libcompact` replaces `src/compact_encoding.h` — useful alone, no schema needed
	145	2. define the schema; generate structs (`hyperschema-c`) and RPC (`hrpc-c`)
	146	3. the daemon skeleton: threads, UDS, per-client buffering (§4)
	147	4. control plane, then channels, then content
	148	5. the C++ client library
	149	6. port the Blender addon off `session-daemon`, and retire it
	150
	151	⚠ Open decision: whether `swarmd` lives in hyperswarm-peer (where its includes
	152	are, making it that repo's first product rather than a library) or its own repo.
	153
	154	## 10. Ground it stands on
	155
	156	The transport was not trustworthy when this was first sketched. As of
	157	2026-09-10 (see `PARITY-hyperswarm.md`): the #72 wall is closed and verified on
	158	the public DHT by mutation drill; the announce/unannounce signable, a
	159	libudx back-pointer corruption, a 30s connect hold and the self-announce are
	160	fixed and gated; holder recording and liveness are verified against a real
	161	testnet; and our connect-failure rate is measured equal to hyperswarm's own.

hyperswarm-peer commits / 8c1e7e2

Changed files (1) +161 -0