6fb11ed · hyperswarm-peer

Changed files (9) +269 -1

docs/SWARMD-DESIGN.md +44 -1

…	…	@@ -131,7 +131,7 @@ content directory, which is also what makes a fetch resumable across restarts.
131	131	\| D2 \| a client that stops reading is dropped; survivors' streams stay intact \|
132	132	\| D3 \| `SEND` past the high-water mark returns `writable=false`, `DRAIN` follows \|
133	133	\| D4 \| two channels on one connection, neither starves the other \|
134		\| D5 \| `PEER_DOWN` on channel close, not on timeout \|
	134	\| D5 \| `PEER_DOWN` on channel close, not on timeout — written, RED, see below \|
135	135	\| D6 \| a 100 MB file crosses by hash and no frame exceeds the cap \|
136	136	\| D7 \| `STATUS` reports `dht=down` when the DHT is down \|
137	137
…	…	@@ -139,6 +139,49 @@ D1 and D7 are the gates #68 proved the suite lacked: every existing
139	139	cross-machine test passes `--bootstrap` at a local testnet, the one configuration
140	140	the products never use.
141	141
	142	D1, D2, D3, D4, D6 and D7 pass. `test/test_peer_down.py` is **D5, and it is
	143	RED** — deliberately kept that way, because what it found is a real hole and not
	144	a gate bug.
	145
	146	### 8.1 D5: nothing detects a silently dead peer on an idle stream
	147
	148	Kill one of two meshed daemons with SIGKILL and the survivor reports `peers=1`
	149	for the whole window. Measured at 10s, 45s, 60s and 90s deadlines: never.
	150
	151	Three layers each look like they handle it and none does.
	152
	153	1. DhtConnector's keepalive is unreachable. `_tick_connection` has a
	154	`DHT_CONN_CONNECTED` branch that probes every `DHT_KEEPALIVE_INTERVAL_MS`
	155	(10s) and fails the connection after `DHT_KEEPALIVE_TIMEOUT_MS` (30s). But
	156	`_finalize_connection` calls `_free_connection(conn)` the instant the
	157	connection reaches CONNECTED — "UDX stream takes over with its own
	158	keepalive". So that branch cannot run, and its two constants describe
	159	liveness that happens nowhere.
	160
	161	2. libudx's keepalive was never switched on. Nothing called
	162	`udx_stream_set_keepalive`, so `keepalive_timeout_ms` was 0 and an idle
	163	stream said nothing at all. It is called now.
	164
	165	3. ⛔ AND IT STILL IS NOT LIVENESS. `send_probe` writes a bare
	166	`UDX_HEADER_HEARTBEAT` with `uv_udp_try_send`, outside the packet and RTO
	167	machinery. libudx says so at that line: `// todo: send a data packet with
	168	seq=remote_acked-1 instead`. An unanswered heartbeat therefore produces no
	169	retransmit, never reaches `UDX_MAX_RTO_TIMEOUTS` (6), and never calls
	170	`close_stream(UV_ETIMEDOUT)` — which is the only thing that would reach
	171	`_on_stream_close` → `on_mux_gone` → `PEER_DOWN`.
	172
	173	Only a stream with real data outstanding dies. An idle one is immortal.
	174
	175	⚠ Two fixes were tried and neither worked, which is why this is written down
	176	instead of a third: propagating `DhtConnector::on_failed` to `disconnect_peer`
	177	(dead code — the connector never reaches `on_failed` for an established peer),
	178	and switching the libudx keepalive on (real, useful for NAT, and not liveness).
	179
	180	What it needs: an application-level ping at our layer — a small periodic
	181	message on a reserved channel, a timeout, and `disconnect_peer` when it lapses —
	182	so the close path that already exists gets driven. That is a component, not an
	183	edit, and it should be designed rather than patched in.
	184
142	185	## 9. Order
143	186
144	187	1. ✅ `libcompact` replaces `src/compact_encoding.h` — useful alone, no schema needed

src/udx_transport.cpp +22 -0

…	…	@@ -1027,6 +1027,28 @@ PeerConnection *UdxTransport::_add_peer(int32_t peer_id, uint32_t local_sid,
1027	1027	udx_socket_t *use_socket = socket_override ? socket_override : &_socket;
1028	1028	udx_stream_connect(&peer->stream, use_socket, remote_sid, addr);
1029	1029
	1030	// Keep an idle stream's NAT mapping open. Nothing called this before, and
	1031	// DhtConnector's comment at the point it frees a CONNECTED connection —
	1032	// "UDX stream takes over with its own keepalive" — described something that
	1033	// was therefore not happening.
	1034	//
	1035	// ⛔ IT IS NOT LIVENESS, AND IT CANNOT BE. libudx's send_probe writes a bare
	1036	// UDX_HEADER_HEARTBEAT with uv_udp_try_send; it never enters the packet or
	1037	// RTO machinery, and libudx says so itself at that line:
	1038	//
	1039	// // todo: send a data packet with seq=remote_acked-1 instead
	1040	//
	1041	// So an unanswered heartbeat produces no retransmit, never reaches
	1042	// UDX_MAX_RTO_TIMEOUTS, and never calls close_stream(UV_ETIMEDOUT). Only a
	1043	// stream with real data outstanding dies that way. An IDLE stream to a
	1044	// process that has been killed stays "connected" for ever — measured, in
	1045	// swarmd's gate D5: kill one of two meshed daemons and the survivor still
	1046	// reports peers=1 ninety seconds later.
	1047	//
	1048	// Detecting that is OURS to build, above this layer, and it is not built.
	1049	// See docs/SWARMD-DESIGN.md §8 (D5) and PARITY-hyperswarm.md §17.
	1050	udx_stream_set_keepalive(&peer->stream, (uint32_t)DHT_KEEPALIVE_INTERVAL_MS);
	1051
1030	1052	// Enable keepalive probes every 10s to detect dead connections.
1031	1053	udx_stream_set_keepalive(&peer->stream, 10000);
1032	1054

swarmd/test/test_client.cpp +8 -0

…	…	@@ -114,10 +114,18 @@ int main() {
114	114	"send() on a channel no peer has open is REFUSED, not silently dropped");
115	115	check(!writable, "and the channel is not reported writable");
116	116
	117	// ⛔ GATE D7, the half that can be asserted without a network: STATUS must
	118	// say the DHT is NOT up when it is not. #72's panel said Connected while
	119	// the daemon was gone, and a status that only ever reports health is
	120	// worse than no status — it is a reason to stop looking. This daemon is
	121	// pointed at a bootstrap that answers nothing, so `up` here would be a
	122	// lie with nothing behind it.
117	123	swarmd::Status st;
118	124	check(c.status(st), "status() answered");
119	125	check(st.local_clients >= 1, "and counts this client");
120	126	check(st.peers_connected == 0, "and reports no peers, which is the truth here");
	127	check(!st.bootstrapped, "D7: it does NOT claim to be bootstrapped against a dead bootstrap");
	128	check(st.dht_state != 2, "D7: and does not report the DHT up");
121	129
122	130	// ---- content ----
123	131	uint8_t hash[32], hash2[32];

swarmd/test/test_peer_down.py +168 -0

…	…	@@ -0,0 +1,168 @@
	1	#!/usr/bin/env python3
	2	"""Gate D5: a peer that goes away is reported on CLOSE, not on a timeout.
	3
	4	⛔ THIS GATE IS RED, AND THE DEFECT IT FOUND IS REAL. Nothing detects a silently
	5	dead peer on an IDLE stream. Traced 2026-09-10:
	6
	7	* DhtConnector has a DHT_CONN_CONNECTED keepalive branch with a 30s timeout —
	8	unreachable, because _finalize_connection frees the connection the instant it
	9	reaches CONNECTED, "UDX stream takes over with its own keepalive".
	10	* libudx's keepalive was never switched on. It is now, and it still is not
	11	liveness: send_probe writes a bare UDX_HEADER_HEARTBEAT with
	12	uv_udp_try_send, outside the packet and RTO machinery — libudx's own TODO at
	13	that line says the packet that would fix this is not written. So no
	14	retransmit, no UDX_MAX_RTO_TIMEOUTS, no close_stream(UV_ETIMEDOUT).
	15	* Only a stream with real data outstanding dies. An idle one never does.
	16
	17	Measured: kill one of two meshed daemons and the survivor reports peers=1 for
	18	the whole window — 10s, 45s, 60s, 90s, all the same. The fix is an
	19	application-level liveness ping with its own timeout and teardown, above the
	20	transport. It is not built. See docs/SWARMD-DESIGN.md §8.
	21
	22
	23	python3 -B test/test_peer_down.py [--bootstrap host:port] [--seconds N]
	24
	25	Two daemons meet on a topic; one is killed; the survivor's client must be told.
	26
	27	⛔ THE NUMBER THAT MAKES THIS A GATE IS THE DEADLINE, and it is not arbitrary.
	28	DHT_KEEPALIVE_TIMEOUT_MS is 30 seconds (src/dht_connector.h): if PEER_DOWN only
	29	ever arrived because a keepalive lapsed, it would still arrive — thirty seconds
	30	late, with the application meanwhile addressing a peer that is not there. So the
	31	assertion is that it lands WELL inside that, which a timeout-driven
	32	implementation cannot do.
	33
	34	⛔ SIGKILL, NOT SIGTERM. A clean shutdown is the easy case and not the one that
	35	hurts: a process that was killed never gets to say goodbye, and the stream close
	36	is the only signal there is.
	37
	38	⛔ IT ASSERTS ITS OWN BASELINE. If the two daemons never meet, "no PEER-DOWN"
	39	means nothing, so a swarm that never meshed exits INCONCLUSIVE rather than FAIL.
	40	"""
	41
	42	import os, re, signal, subprocess, sys, tempfile, time
	43
	44	HERE = os.path.dirname(os.path.realpath(__file__))
	45	ROOT = os.path.dirname(HERE)
	46	SWARMD = os.path.join(ROOT, "swarmd")
	47	PROBE = os.path.join(ROOT, "tools", "swarmd-probe")
	48	EXIT_PASS, EXIT_FAIL, EXIT_INCONCLUSIVE = 0, 1, 2
	49
	50	TOPIC = "d5" * 32
	51	PROTOCOL = "swarmd-peerdown/v1"
	52
	53	# Comfortably inside DHT_KEEPALIVE_TIMEOUT_MS (30s). A timeout-driven report
	54	# cannot land here; a close-driven one lands almost immediately.
	55	DOWN_DEADLINE_S = int(os.environ.get("DOWN_DEADLINE_S", "10"))
	56
	57
	58	def arg(name, default=None):
	59	return sys.argv[sys.argv.index(name) + 1] if name in sys.argv else default
	60
	61
	62	def read(path):
	63	try:
	64	with open(path, "rb") as fh:
	65	return fh.read().decode("utf-8", "replace")
	66	except OSError:
	67	return ""
	68
	69
	70	def wait_for(path, needle, timeout, proc=None):
	71	end = time.time() + timeout
	72	while time.time() < end:
	73	if proc is not None and proc.poll() is not None:
	74	return False
	75	if needle in read(path):
	76	return True
	77	time.sleep(0.2)
	78	return False
	79
	80
	81	def main():
	82	bootstrap = arg("--bootstrap")
	83	window = int(arg("--seconds", "90" if bootstrap else "240"))
	84
	85	for path in (SWARMD, PROBE):
	86	if not os.path.isfile(path):
	87	print("missing %s — run ./build.sh" % path)
	88	return EXIT_INCONCLUSIVE
	89
	90	tmp = tempfile.mkdtemp(prefix="swarmd-peerdown-")
	91	procs, logs = [], {}
	92
	93	def spawn(argv, tag):
	94	logs[tag] = os.path.join(tmp, tag + ".log")
	95	fh = open(logs[tag], "wb")
	96	p = subprocess.Popen(argv, stdout=fh, stderr=subprocess.STDOUT)
	97	procs.append(p)
	98	return p
	99
	100	try:
	101	daemons = {}
	102	for tag in ("a", "b"):
	103	argv = [SWARMD, "--socket", os.path.join(tmp, tag + ".sock")]
	104	if bootstrap:
	105	argv += ["--bootstrap", bootstrap]
	106	daemons[tag] = spawn(argv, tag)
	107
	108	for tag in ("a", "b"):
	109	if not wait_for(logs[tag], "dht=up", 90, daemons[tag]):
	110	print("\nINCONCLUSIVE: daemon %s never reached dht=up." % tag)
	111	return EXIT_INCONCLUSIVE
	112	print(" both daemons are on the DHT")
	113
	114	watchers = {}
	115	for tag in ("a", "b"):
	116	watchers[tag] = spawn([PROBE, "--socket", os.path.join(tmp, tag + ".sock"),
	117	"--topic-hex", TOPIC, "--protocol", PROTOCOL,
	118	"--watch-peers", "--seconds", str(window)],
	119	"watch-" + tag)
	120
	121	# ---- baseline: they have to meet before a departure means anything ----
	122	if not wait_for(logs["watch-a"], "PEER-UP", window, watchers["a"]):
	123	print("\nINCONCLUSIVE: daemon A never saw a peer, so its silence "
	124	"afterwards says nothing.")
	125	return EXIT_INCONCLUSIVE
	126	print(" A saw the peer arrive")
	127
	128	# D7's other half, read from a client rather than from a log line.
	129	status = re.search(r"STATUS dht_state=(\d+) bootstrapped=(\d+)", read(logs["watch-a"]))
	130	if status:
	131	print(" A's STATUS: dht_state=%s bootstrapped=%s"
	132	% (status.group(1), status.group(2)))
	133
	134	# ---- the departure ----
	135	killed_at = time.time()
	136	daemons["b"].send_signal(signal.SIGKILL)
	137	daemons["b"].wait(timeout=10)
	138	print(" B killed with SIGKILL")
	139
	140	seen = wait_for(logs["watch-a"], "PEER-DOWN", DOWN_DEADLINE_S, watchers["a"])
	141	elapsed = time.time() - killed_at
	142
	143	if not seen:
	144	print("\nFAIL: A was not told its peer had gone within %ds. "
	145	"DHT_KEEPALIVE_TIMEOUT_MS is 30s, so a report driven by a "
	146	"timeout rather than by the close would look exactly like this."
	147	% DOWN_DEADLINE_S)
	148	print("--- A ---\n%s" % read(logs["watch-a"])[-1500:])
	149	return EXIT_FAIL
	150
	151	print("\nPASS: PEER-DOWN reached the client %.1fs after the peer was "
	152	"killed (deadline %ds, keepalive timeout 30s)." % (elapsed, DOWN_DEADLINE_S))
	153	if status and status.group(1) != "2":
	154	print(" ⚠ but STATUS reported dht_state=%s while the daemon's own log "
	155	"said dht=up — those disagree." % status.group(1))
	156	return EXIT_FAIL
	157	return EXIT_PASS
	158	finally:
	159	for p in procs:
	160	if p.poll() is None:
	161	p.terminate()
	162	try: p.wait(timeout=5)
	163	except subprocess.TimeoutExpired: p.kill()
	164	print("(logs: %s)" % tmp)
	165
	166
	167	if __name__ == "__main__":
	168	sys.exit(main())

swarmd/tools/swarmd_probe.cpp +27 -0

…	…	@@ -46,6 +46,7 @@ int main(int argc, char **argv) {
46	46	const char sock = nullptr, topic_hex = nullptr, *proto = "swarmd-probe/v1";
47	47	const char send_text = nullptr, serve_file = nullptr;
48	48	const char fetch_hex = nullptr, dest = nullptr;
	49	bool watch_peers = false;
49	50	int seconds = 30;
50	51	for (int i = 1; i < argc; i++) {
51	52	if (!strcmp(argv[i], "--socket") && i + 1 < argc) sock = argv[++i];
…	…	@@ -55,6 +56,7 @@ int main(int argc, char **argv) {
55	56	else if (!strcmp(argv[i], "--serve-file") && i + 1 < argc) serve_file = argv[++i];
56	57	else if (!strcmp(argv[i], "--fetch-hex") && i + 1 < argc) fetch_hex = argv[++i];
57	58	else if (!strcmp(argv[i], "--dest") && i + 1 < argc) dest = argv[++i];
	59	else if (!strcmp(argv[i], "--watch-peers")) watch_peers = true;
58	60	else if (!strcmp(argv[i], "--seconds") && i + 1 < argc) seconds = atoi(argv[++i]);
59	61	}
60	62	uint8_t topic[32];
…	…	@@ -83,6 +85,18 @@ int main(int argc, char **argv) {
83	85	fail_reason = why;
84	86	failed = true;
85	87	};
	88	// Gate D5 reads these: a peer going away must be reported when the channel
	89	// closes, not when something times out.
	90	if (watch_peers) {
	91	c.on.on_peer_up = [](uint64_t peer, const uint8_t *) {
	92	printf("PEER-UP %llu\n", (unsigned long long)peer);
	93	fflush(stdout);
	94	};
	95	c.on.on_peer_down = [](uint64_t peer) {
	96	printf("PEER-DOWN %llu\n", (unsigned long long)peer);
	97	fflush(stdout);
	98	};
	99	}
86	100	c.on.on_disconnect = [&](const std::string &why) {
87	101	fprintf(stderr, "swarmd went away: %s\n", why.c_str());
88	102	};
…	…	@@ -96,6 +110,16 @@ int main(int argc, char **argv) {
96	110	const uint64_t ch = c.open_channel(proto, {}, 2);
97	111	if (ch == 0) { fprintf(stderr, "channel failed: %s\n", c.last_error().c_str()); return 2; }
98	112	printf("joined topic, opened %s\n", proto);
	113	{
	114	// D7's other half: what the daemon says about itself, from a client
	115	// rather than from its own log line.
	116	swarmd::Status st;
	117	if (c.status(st)) {
	118	printf("STATUS dht_state=%llu bootstrapped=%d peers=%llu\n",
	119	(unsigned long long)st.dht_state, st.bootstrapped ? 1 : 0,
	120	(unsigned long long)st.peers_connected);
	121	}
	122	}
99	123	fflush(stdout);
100	124
101	125	// ---- content ----
…	…	@@ -135,6 +159,9 @@ int main(int argc, char **argv) {
135	159	time_t last_try = 0;
136	160
137	161	while (time(nullptr) < deadline && !received && !fetched && !failed) {
	162	if (watch_peers && !send_text && !fetch_hex) {
	163	// A watcher has nothing of its own to finish; it runs the window out.
	164	}
138	165	if (send_text && !sent && time(nullptr) != last_try) {
139	166	last_try = time(nullptr);
140	167	attempts++;

test/test_bootstrap_ownership +0 -0

(no content changes — binary or rename)

test/test_regression +0 -0

(no content changes — binary or rename)

test/test_session_channel +0 -0

(no content changes — binary or rename)

test/test_three_peer_swarm +0 -0

(no content changes — binary or rename)

hyperswarm-peer commits / 6fb11ed

Changed files (9) +269 -1