ec7b8ae · hypercompute

Changed files (17) +2591 -0

.gitignore +7 -0

…	…	@@ -0,0 +1,7 @@
	1	node_modules/
	2	*.log
	3	.DS_Store
	4	# local test/dev storage
	5	storage/
	6	*-storage/
	7	.tmp/

README.md +155 -0

…	…	@@ -0,0 +1,155 @@
	1	# hypercompute
	2
	3	> *Hypercore turned data* into a shared, verifiable, append-only log.
	4	> hypercompute turns computation into shared, verifiable, content-addressed
	5	> chunks over that log — one P2P substrate with two lanes**: a CPU lane that
	6	> is deterministic, sandboxed, memoized and trustless, and a GPU lane that is
	7	> native, partitioned and trusted. Determinism is the line between them.
	8
	9	Status: design + P0 scaffold. The core (content-addressed submit → execute →
	10	memoized result over a deterministic fold) is real and tested; networking,
	11	sandbox, and the GPU lane are scaffolded/specified. Nothing is deployed.
	12	Full design in [SPEC.md](SPEC.md); evidence base in [research/](research/).
	13
	14	---
	15
	16	## The one idea
	17
	18	A Hyperdrive file is chunked: `bytes → hash` — immutable, fetchable from
	19	anyone, verifiable against a Merkle root, shareable. A computation chunk is
	20	`f(code, input) → output`, named by `hash(code) + hash(input)`. The first peer to
	21	compute it shares it with everyone; anyone can re-check it. That single move
	22	generates the whole design.
	23
	24	## The one law
	25
	26	Data chunks are verifiable for free (data is static — its hash is fixed). A
	27	computed chunk becomes a real, reusable, verifiable chunk **only if the
	28	computation is deterministic.** Data answers only "are these the right bytes?"
	29	(integrity). Computation also asks "was it computed correctly?" (correctness) —
	30	and correctness is what needs determinism. Determinism is the toll that
	31	promotes a computed result into a Hyperdrive-style chunk.
	32
	33	## The two lanes (the backbone)
	34
	35	One substrate, two lanes, split by determinism:
	36
	37	\| \| Lane A — CPU Verifiable \| Lane B — GPU Orchestration \|
	38	\|---\|---\|---\|
	39	\| Unit \| content-addressed, deterministic, sandboxed task \| a native GPU job (model shard, render tile, training step) \|
	40	\| Executor \| WASM now → libriscv/C++ later \| native CUDA/Vulkan/Metal; cross-vendor via Slang \|
	41	\| Superpowers \| both — partition and memoize/verify \| partition only (capacity) \|
	42	\| Trust \| T1 re-execution voting → trustless \| T0 trust / T2 statistical (TOPLOC) \|
	43	\| Use \| CI/builds, data processing, sci kernels, sims, pure serverless \| distributed inference, low-comm training, remote-GPU offload \|
	44
	45	The bridge: `RepOps` (fixed FP order) makes GPU compute bitwise-reproducible
	46	across hardware — letting Lane B reach Lane A's verifiability, at a 30–370%
	47	cost. It's the one mechanism connecting the lanes.
	48
	49	Honest GPU framing (a correction worth stating loudly): GPU is not
	50	impossible — it's a full first-class lane. Distributing a model is a **capacity
	51	win, not a speed win** (you run something too big for one machine; a single
	52	request gets slower). What's genuinely impossible over P2P is **fine-grained,
	53	tightly-coupled** work — tensor-parallel, collective all-reduce, per-instruction
	54	sharing — whether CPU or GPU. The wall is about coupling, not hardware.
	55
	56	## The shared substrate (both lanes use it)
	57
	58	- Durable plane — an optimistic Autobase: the shared append-only log of chunks
	59	+ the deterministic fold that materializes current truth (mutability = a view
	60	over the log). Writes don't block; reads are local.
	61	- Hot path — datagrams: fast, lossy hints ("chunk ready") that front-run the
	62	durable truth; reconcile against the fold.
	63	- Blob plane — big chunk payloads (artifacts, weights, frames) by hash. This is
	64	essentially pgh + hyper-overlay — which already exists.
	65	- Stream plane — real-time media via Keet's pattern (WebRTC-over-udx +
	66	signaling on the control plane).
	67	- Enrollment / membership — blind-pairing mints a peer into a membership view;
	68	the hot path then just checks "are you a member?"
	69
	70	## The general-compute unit
	71
	72	The chunk is the computational unit: a **content-addressed, deterministic,
	73	sandboxed task** — the same unit that makes Lambda, WASM platforms, Bazel and Ray
	74	general because it's crisp, not despite it. So hypercompute is a general P2P
	75	compute substrate:
	76
	77	- *general and verifiable*** over the deterministic class (Lane A);
	78	- *general and trusted*** (capacity-pooling) over everything else (Lane B).
	79
	80	Both are general; they differ in trust, not in scope.
	81
	82	## The vision — the swarm as one device (Hyperdevice)
	83
	84	A peer programs against one virtual device; the sandbox's gated syscall boundary
	85	is the seam where "run locally vs route to the swarm" is decided. Three honest
	86	tiers of invisibility:
	87	- invisible — Lane A (deterministic CPU): the host routes every call to the
	88	swarm, results are memoized + verified, the program never knows it's distributed;
	89	- pooled-but-explicit — Lane B (GPU): poolable, not invisible (no interception
	90	seam — native code talks to hardware directly);
	91	- impossible — fine-grained / tightly-coupled work (the ~10⁶× latency wall →
	92	chunks must be coarse).
	93
	94	## Apps are DAGs of chunks (the layering)
	95
	96	- hypercompute = the chunk substrate (this repo, base lib).
	97	- Apps on top = domain DAGs of chunks: hyperactions (CI — build steps as
	98	chunks), data pipelines, scientific compute, distributed inference.
	99	- Built app-first so generality stays honest (prove it with ≥2 distinct apps,
	100	don't ship a formless fabric).
	101
	102	## What's real vs honest limits
	103
	104	Real / strong (research-verified):
	105	- The metaphor is already half-built: **pgh is a content-addressed
	106	computation-output store distributed P2P** — the chunk store exists.
	107	- Autobase/Hyperbee (the fold) are mature in JS-on-Bare.
	108	- Async GPU remoting ~2% (compute-bound); DiLoCo trains across continents
	109	(OpenDiLoCo: 1.1B, 3 countries, 127–935 Mbit/s, 90–95% util); RepOps makes GPU
	110	verifiable at 30–370%; the interconnect wall is absolute (NVLink 600–1800 GB/s/GPU
	111	vs inter-node ~12–27 GB/s).
	112
	113	Honest boundaries (named, not hidden):
	114	- Verifiable generality stops at determinism — Lane B is trusted, not verified
	115	(unless it pays the RepOps tax).
	116	- Physics caps it at coarse grain — no fine-grained distribution.
	117	- Two open problems: incentives (why would a peer contribute? → today:
	118	your-own-devices / trusted club) and malicious owner (the trust anchor isn't
	119	trustless).
	120
	121	## Run
	122
	123	```bash
	124	npm install
	125	npm test # brittle: submit -> execute -> result + memoization (Lane A)
	126	node index.js demo # (or: bare index.js demo) local in-memory walkthrough
	127	```
	128
	129	## Layout
	130
	131	```
	132	SPEC.md full specification
	133	README.md this synthesis
	134	index.js CLI / local demo
	135	lib/
	136	pool.js durable plane: optimistic Autobase + Hyperbee view + apply()
	137	membership.js hot-path gate (remotePublicKey ∈ member/*)
	138	node.js ComputeNode: corestore + base + executors + submit/run/memoize
	139	enroll.js blind-pairing enrollment ceremony (scaffold)
	140	transport.js hyperswarm dual-plane (durable + datagram hot path) (scaffold)
	141	test/e2e.js P0 proof over replicateAndSync
	142	research/ the deep-research evidence base (reports + raw outputs)
	143	```
	144
	145	## Roadmap (SPEC §9)
	146
	147	- P0 ✅ membership + durable plane + content-addressed memoization (Lane A core).
	148	- P1 datagram hot path + prediction/reconcile; real blind-pairing enrollment.
	149	- P2 WASM sandbox + fuel/timeout limits + content-addressed code.
	150	- P3 T1 replication+vote + dispute hook (the trustless lane).
	151	- P4 blob plane; Lane B (GPU orchestration: native + Slang + RepOps for verify).
	152
	153	## License
	154
	155	MIT

SPEC.md +553 -0

…	…	@@ -0,0 +1,553 @@
	1	# hypercompute — Specification (draft 0.1)
	2
	3	A low-latency P2P compute pool on the Holepunch stack. Enrolled peers share
	4	computation; tasks are content-addressed and memoized; dispatch + results travel
	5	a datagram hot path while an optimistic Autobase is the durable, deterministically
	6	-folded source of truth.
	7
	8	Status: design draft. Code sketches are illustrative. Claims marked
	9	(VERIFIED) were read from shipped source this session; (DESIGN) are proposals;
	10	(REMEMBERED) need a source re-read before implementation.
	11
	12	---
	13
	14	## Conceptual foundation — the two axes of chunking
	15
	16	Everything below rests on one distinction. "Chunking" (the thing Hyperdrive does
	17	to files) gives two independent superpowers, and they have completely
	18	different requirements. Conflating them is the classic mistake.
	19
	20	Superpower A — PARTITION (Hyperdrive's actual killer feature).
	21	Split the whole into distinct pieces, each on different devices; assemble the
	22	whole from parts no single device holds. The payoff is **capacity, parallelism,
	23	locality** — different devices do/hold different things. Sharded storage,
	24	pipeline-parallel inference, MapReduce. Requires NO determinism.
	25
	26	Superpower B — MEMOIZE/VERIFY (trustless reuse).
	27	Make a computed result behave like a fixed, fetch-from-anyone, re-checkable
	28	chunk: stable hash, cached forever, verifiable by re-running. The payoff is
	29	redundancy and trust — different devices can do the same work and
	30	compare/cache. Requires determinism (§3).
	31
	32	These are orthogonal. Hyperdrive appears to get both for free only because its
	33	chunks are static data: a fixed hash forever, so partitioning is trivial and
	34	the hash does double duty as both name (locate/dedupe) and integrity check
	35	(Merkle-verify against the author's signed root).
	36
	37	The shift when chunks are computation, not data: data only ever has an
	38	integrity question ("are these the bytes the author committed?" — solved by
	39	signatures, no determinism). Computation adds a correctness question ("did the
	40	worker compute the RIGHT answer, or cheat/err?"). Integrity of a computed piece
	41	still needs only a signature; **correctness is the only thing that needs
	42	determinism** (so you can re-run and compare) — or a proof, or trust.
	43
	44	> **Determinism is the toll paid ONLY to promote a computed result into a
	45	> verifiable chunk (Superpower B). Partitioning a computation across devices
	46	> (Superpower A) needs no determinism whatsoever.**
	47
	48	### The two-axis map (this subsumes the trust tiers and the GPU lane)
	49
	50	\| \| Trust the producer (no determinism) \| Verify the output (needs determinism) \|
	51	\|---\|---\|---\|
	52	\| Replicate (same work) \| seeding / availability \| T1 voting, re-execution (§4.4) \|
	53	\| Partition (different pieces — killer feature) \| pipeline-parallel AI, sharded storage ✅ \| content-addressed task graph (verifiable MapReduce) \|
	54
	55	Consequences that drive the rest of the spec:
	56	- GPU/AI is NOT "plumbing-only." It gets Superpower A in full — pipeline-parallel
	57	inference (exo) runs happily on nondeterministic GPUs precisely because
	58	partitioning ignores determinism. It simply forgoes Superpower B. Capacity
	59	unlock, not a consolation prize.
	60	- Deterministic CPU work gets BOTH A and B — the only lane that does.
	61	- The trust tiers (§4.4) and the determinism gate (§3) govern the *right column
	62	only*. Anything in the left column (incl. GPU) is available regardless.
	63	- So the system has two earnings: partition anything for capacity (left), and
	64	memoize/verify deterministic CPU work for trustless reuse (right). Most
	65	prior designs see only one column; hypercompute is explicit that it spans both.
	66
	67	---
	68
	69	## 0. Scope & non-goals
	70
	71	In scope
	72	- Enrollment of peers into a compute group (trust bootstrap).
	73	- A two-plane runtime: ephemeral datagram hot path + durable optimistic-autobase log.
	74	- A constrained task model that makes computation content-addressable and verifiable.
	75	- Trust tiers from "trust the author" to "replicate and vote."
	76
	77	Non-goals (v0.1)
	78	- Truly arbitrary, unsandboxed, nondeterministic native code in the trustless tiers
	79	(see §3 — it is neither memoizable nor verifiable; it is RCE). Allowed only in
	80	the explicit trusted-author tier (T0).
	81	- Succinct proofs (zkVM) of execution — left as a future tier (T3, §6).
	82	- A token/payment/marketplace layer.
	83	- Scheduling optimality (we do greedy claim-based dispatch, not a solver).
	84
	85	---
	86
	87	## 1. Do we need a protocol? — Yes. Three surfaces.
	88
	89	A "protocol" here means: **versioned, schema-defined messages with stable wire
	90	encodings**, not ad-hoc JSON. We need it because (a) peers run independently
	91	-deployed versions and must stay wire-compatible; (b) the durable log is
	92	append-only and replayed forever, so its op encoding can never break; (c)
	93	content-addressing requires byte-stable encodings — the same task must hash
	94	identically on every node.
	95
	96	There are exactly three protocol surfaces:
	97
	98	\| Surface \| Transport \| Encoding \| Append-only? \|
	99	\|---\|---\|---\|---\|
	100	\| P1 Enrollment \| blind-pairing over HyperDHT \| blind-pairing-core messages + our `PeerClaim`/`GrantCap` \| n/a (ceremony) \|
	101	\| P2 Durable plane \| Autobase op log → Hyperbee view \| `compact-encoding` structs, schema-versioned \| yes — never break \|
	102	\| P3 Data plane \| secret-stream unordered datagrams \| `compact-encoding` structs \| no (ephemeral, lossy) \|
	103
	104	Encoding choice (DESIGN): define all P2/P3 structs in a `hyperschema`
	105	registry and (optionally) generate with `hrpc` for the control channel.
	106	Rationale: hyperschema gives append-only, versioned struct evolution — the same
	107	discipline the rest of our stack uses. Hand-rolled JSON would break
	108	content-addressing (key order / number formatting differences change the hash).
	109
	110	Versioning rule: every P2 op carries a `v` (uint). `apply()` must handle all
	111	historical `v` values forever (append-only). P3 datagrams carry `v` too but may be
	112	dropped if unknown (they're hints, not truth).
	113
	114	---
	115
	116	## 2. Architecture (two planes, one membership)
	117
	118	```
	119	P1 enroll ──(once)──▶ writes member/* + add-writer into the durable plane
	120	│
	121	P3 data plane (conn.trySend) ─────┼──▶ predictions: task-available / claiming / result-ready
	122	ephemeral, lossy, low-latency │ act NOW, ahead of the fold
	123	│
	124	P2 durable plane (optimistic base) │
	125	member/* task/* claim/* result/* ──▶ deterministic fold = TRUTH; reconcile predictions
	126	```
	127
	128	- Hot path = P3 datagrams. No head-of-line blocking, no ack, no quorum.
	129	- Truth = P2 fold. Reads are local (0 network). Writes are optimistic
	130	(`optimistic:true` — no quorum wait). (VERIFIED: keet pattern, autobase source.)
	131	- Auth after enrollment = `remotePublicKey ∈ member/*`, a local view read (~µs).
	132
	133	---
	134
	135	## 3. Trust & determinism model — the load-bearing constraint
	136
	137	The whole task model is dictated by one fact established in research and in the
	138	autobase source: **memoization and re-execution verification both require
	139	deterministic execution.** A `(codeHash, inputHash)` pair can only have a
	140	canonical output if running `code` on `input` yields the same bytes everywhere.
	141
	142	Sources of nondeterminism that MUST be excluded (or quarantined to T0):
	143	wall-clock, RNG, threads/scheduling, floating-point reordering across hardware,
	144	network/syscalls, filesystem, locale, uninitialized memory.
	145
	146	This gives the task purity contract (§4.2). Tasks that satisfy it are
	147	content-addressable and verifiable. Tasks that don't are confined to T0 (trusted
	148	author) or T2 (statistical check) — they can never be memoized as canonical.
	149
	150	---
	151
	152	## 4. The task model — how we handle tasks, including "arbitrary" ones
	153
	154	### 4.1 A task is content-addressed (not "a function call")
	155
	156	The unit of work is a pure, addressed computation, mirroring how Hyperdrive
	157	addresses file chunks:
	158
	159	```
	160	task = {
	161	codeHash: hash of the executable artifact (WASM module / addressed bundle)
	162	inputHash: hash of the canonical-encoded input
	163	input: the bytes (or a hyperdrive/blob pointer if large)
	164	limits: { fuel, wallMs, memPages } // resource bounds
	165	tier: T0 \| T1 \| T2 // trust requirement
	166	}
	167	id = hash(codeHash ‖ inputHash ‖ limits)
	168	result key = "result/" + codeHash + "/" + inputHash
	169	```
	170
	171	Because the output is keyed by content, the first correct computation is
	172	shared by everyone — recompute cost drops to zero (Layer-2 memoization). This is
	173	literally "chunking computation": the canonical output of `(codeHash,inputHash)`
	174	is a reusable chunk.
	175
	176	### 4.2 Execution targets — how arbitrary code runs safely
	177
	178	"Arbitrary task" does not mean "arbitrary native binary." It means *any
	179	computation expressible under the purity contract*, run in a sandbox. Ranked by
	180	fit:
	181
	182	1. WASM (PRIMARY) (DESIGN). Portable, sandboxed-by-default (no ambient
	183	syscalls), content-addressable, and *deterministic if you forbid
	184	nondeterministic imports*. The host exposes only a whitelisted import surface
	185	(no clock/RNG/net unless the tier allows). Fuel metering for resource bounds.
	186	This is the sweet spot: arbitrary logic, bounded, deterministic, hashable.
	187	- Runtime options to evaluate: a WASM engine available under Bare (native
	188	addon) — REMEMBERED that the ecosystem has WASM execution paths (Bacalhau
	189	uses WASM for exactly this reason, per the research); the specific Bare WASM
	190	runtime must be chosen + verified.
	191	2. bare-worker thread (DESIGN). Run JS in an isolated worker with a frozen
	192	global, no network, killed on timeout. Weaker isolation than WASM (shares the
	193	JS heap engine), acceptable for T0/trusted authors. Good for our own code.
	194	3. Subprocess / OS sandbox (DESIGN). `bare-subprocess` into a jailed process
	195	(seccomp/namespaces/container). Strongest for genuinely arbitrary native
	196	workloads, but heaviest and OS-specific. T0 only.
	197
	198	Decision (resolved — see executor research below): v0.1 targets **WASM
	199	(Wasmtime or WAMR) as the canonical sandbox**, with bare-worker as a T0 fast path
	200	for first-party code. Native/subprocess is out of v0.1. **libriscv/RISC-V is the
	201	strategic P3+ target, not v0.1** (rationale below).
	202
	203	### 4.2.1 Executor research verdict — WASM now, libriscv later
	204
	205	Evaluated libriscv (+ godot-sandbox) vs WASM as the executor (deep-research, all
	206	findings 3-0 from primary sources unless noted).
	207
	208	libriscv confirmed strengths (VERIFIED):
	209	- Fuel metering: `set_max_instructions()` / `simulate(limit)`, resumable, default-on.
	210	- Deny-by-default syscalls: unhandled → `-ENOSYS`; narrow ECALL ABI; host installs
	211	every handler explicitly — exactly how you forbid clock/RNG/net.
	212	- Memory bound via `memory_max` (NOT the encompassing arena — arena is fully R/W,
	213	"should not be used when sandboxing is necessary" until userfaultfd lands).
	214	- Embeddable standalone (header-only C++ + `libriscv_capi`); godot-sandbox is the
	215	WRONG path (Godot-coupled). Guests from C/C++/Rust/Zig via stock riscv toolchains.
	216
	217	libriscv blocking gap (VERIFIED): it makes **NO guarantee of bit-exact
	218	cross-hardware determinism or IEEE-754 FP reproducibility, and has no audit.**
	219	Determinism evidence comes from a different system (Stanford DeCl, SIB'24/OSDI'25):
	220	a restricted deterministic ISA subset incl. FP is bit-exact across ARM64 µarches —
	221	validates the principle but does not certify libriscv. DeCl also shows native
	222	instruction-metering ~16-21% overhead vs Wasmtime fuel ~90% vs WASM interp ~1920%.
	223
	224	Two unresolved questions block libriscv (must answer empirically before P3):
	225	1. Does binary-translation/JIT preserve bit-exactness AND keep enforcing the
	226	budget? (Interpreter is the only currently-trustworthy det+metered path — slow.)
	227	2. Host-FPU (must pin FPCR) or soft-float? Undocumented. Decisive for memoization.
	228
	229	Why WASM for v0.1: mature embedding + first-class bindings; documented
	230	determinism knobs (NaN canonicalization, deterministic fuel); avoids libriscv's
	231	unknowns. The ~90% metering overhead is acceptable — determinism + embedding
	232	maturity are the gate, not metering speed.
	233
	234	Why libriscv stays on the roadmap: large metering-cost advantage, and the
	235	trustless-compute field is converging on RISC-V (PolkaVM, Cartesi, RISC Zero) —
	236	signal that RISC-V is a clean determinism substrate.
	237
	238	### 4.2.2 Determinism is never automatic (applies to EITHER executor)
	239
	240	Regardless of WASM or RISC-V, to make `(codeHash,inputHash)` memoizable you MUST:
	241	(a) restrict to a deterministic instruction/feature subset; (b) pin the FP env
	242	(rounding/NaN/flush-to-zero) or use soft-float; (c) forbid all nondeterministic
	243	imports/syscalls (no clock/RNG/net/thread-scheduling); (d) control the JIT path;
	244	(e) empirically validate cross-hardware bit-exactness before trusting the
	245	cache. Same lesson as ccache/Bazel/Nix: nondeterminism silently poisons the cache.
	246
	247	### 4.3 Resource bounds (mandatory — anti-DoS)
	248
	249	Every task declares `limits`; the worker enforces them or rejects the task:
	250	- fuel — instruction/gas budget (WASM fuel metering). Halts runaway loops.
	251	- wallMs — hard timeout; worker kills + reports `timeout`.
	252	- memPages — max linear memory.
	253	Unbounded tasks are invalid. A worker that can't honor the limits declines (it
	254	doesn't partially run).
	255
	256	### 4.4 Trust tiers — verifying the result
	257
	258	\| Tier \| What it means \| Verification \| Cost \| Use \|
	259	\|---\|---\|---\|---\|---\|
	260	\| T0 \| trust the author/worker \| none (signed by worker) \| 1× \| first-party / own devices \|
	261	\| T1 \| replicate + vote \| run on k workers, accept if ≥⌈k/2⌉+1 agree on output hash \| k× \| default for shared pure tasks \|
	262	\| T2 \| statistical \| TOPLOC-style fingerprint over execution (REMEMBERED, research) \| ~1.x× \| nondeterministic/ML, can't bit-match \|
	263	\| T3 \| succinct proof \| zkVM proof of execution (FUTURE) \| ~10^5× prove \| high-value, non-interactive \|
	264
	265	v0.1 ships T0 + T1. T1 is the honest "cheap trust" answer: because pure tasks
	266	are deterministic, k independent workers must produce the same output hash;
	267	disagreement triggers the dispute hook. T2/T3 are stubs with defined seams.
	268
	269	### 4.5 Why not "just run anything"
	270
	271	Running an unconstrained peer-supplied binary = (a) RCE on the worker,
	272	(b) un-memoizable (nondeterministic), (c) un-verifiable (can't re-execute to
	273	compare), (d) un-bounded (DoS). The constrained model trades "literally any
	274	binary" for "any logic, sandboxed + bounded + addressable" — which is what
	275	makes the trustless tiers possible at all.
	276
	277	### 4.6 The GPU / native lane (synthesized from GPU research)
	278
	279	> Provenance: salvaged from a hung research run, then ADVERSARIALLY RE-VERIFIED
	280	> (2-vote refute-biased, wf wly8cxk91). Core mechanics CONFIRMED against primary
	281	> sources (arXiv 2502.19405, 2401.13354, 2407.07852). Corrections applied below:
	282	> NVLink bandwidth (was stale/understated), io.net endpoint (600k→100k, not 10k),
	283	> WebGPU device (RTX 5080; dropped an unsourced "% of CUDA" figure), and the
	284	> GPU-determinism framing (same-hardware repro is largely achievable — the hard
	285	> part is CROSS-hardware, which is what RepOps solves).
	286
	287	GPU work runs native (no WASM/RISC-V sandbox — sandboxes are CPU-only and
	288	cannot touch the GPU). Under the two-axis foundation it is natively LEFT-COLUMN
	289	(partition + trust). Correction to an earlier too-absolute claim: GPU CAN
	290	reach the right column (verify) — but only by paying a determinism toll (RepOps,
	291	below) that CPU work never pays.
	292
	293	(A) Determinism — SAME-hardware is achievable; CROSS-hardware is the hard part.
	294	Causes of nondeterminism: atomic ops, FP reduction order (non-associative), thread
	295	scheduling. BUT (verified correction): on a fixed GPU, most ops CAN be made
	296	bit-reproducible across runs — cuDNN documents most routines as bit-identical when
	297	configured, and PyTorch's deterministic mode forces it (some ops have no
	298	deterministic impl and throw). The genuinely hard, unsolved-by-default problem is
	299	CROSS-hardware bit-exactness (T4 vs A100, NVIDIA vs AMD) — FP non-associativity
	300	+ differing reduction order across devices. Even Vulkan/SPIR-V diverges
	301	cross-device (NVIDIA-desktop denorm-flush, ~0.2 accumulated error; FP-control flags
	302	didn't fix it). **The bridge — RepOps (below) — is specifically a cross-hardware
	303	fix. The bridge — Gensyn RepOps** (Verde,
	304	arXiv 2502.19405): re-implement ML operators to enforce fixed FP order →
	305	bitwise-reproducible across T4/RTX3090/A100. Cost: matmul <30% (large) to
	306	~200% (256-dim); end-to-end 60–374% by model/GPU (cheaper on bigger VRAM).
	307	Single-GPU deterministic reductions (pairwise/tree) are nearly free (<0.2% V100,
	308	7.8% GH200); the expensive part is CROSS-hardware bit-exactness.
	309
	310	(B) Verification — enabled by RepOps via refereed delegation. Verde: hash
	311	checkpoints; on disagreement a referee re-executes only the SINGLE first-divergent
	312	operator (2 OOM < re-running; <1 OOM total overhead vs 4 OOM for naive full
	313	re-exec). Honest if ≥1 verifier honest. This is the §Layer-3 TrueBit fraud-proof
	314	model applied to ML — working ONLY because RepOps re-engineers determinism in.
	315	Without RepOps: T0 trust or T2 statistical (TOPLOC) only.
	316
	317	(C) Remote-GPU over a network — fine for compute-bound, fails for chatty.
	318	API remoting = scuda/LUPINE (LD_PRELOAD shim of libcuda.so.1 over HTTP/2),
	319	Thunder Compute (CUDA→TCP, ~10–20ms connect). Dominant cost = host↔device transfer
	320	(PCIe-over-network); per-call compute overhead small once the model is GPU-resident.
	321	Key result (arXiv 2401.13354): with ASYNC remoting, **GPT-2 inference is only ~2%
	322	slower at 10µs RTT + 1 Gbps** — GPU exec (10–100ms) dwarfs network delay (1–100µs);
	323	5–20µs → <5% overhead. BUT chatty/fine-grained workloads are "terrible even with
	324	100/400Gb ethernet," and large host transfers collapse.
	325
	326	(D) The interconnect wall (why tensor-parallel can't cross machines).
	327	Intra-node NVLink: 600 / 900 / 1800 GB/s per GPU (A100 / H100 / Blackwell).
	328	Inter-node practical GPUDirect-RDMA: ~12 GB/s per ~100Gb NIC; NCCL inter-node
	329	all-reduce busbw ~15–27 GB/s on 200–400Gb IB. So the intra-vs-inter gap is roughly
	330	an order of magnitude (larger than first written — NVLink was understated). WAN: ~25–45 Gbit/s
	331	cross-continent; consumer ~60 Mbps upload → naive data-parallel 671B = "5,000
	332	years." NCCL LL128 (95% bw) needs atomic-128B-write HW guarantees unavailable over
	333	commodity transport. ⟹ **tensor-parallel + collective all-reduce are IMPOSSIBLE
	334	over P2P/WAN; only communication-avoiding training survives — DiLoCo** (500×
	335	less comm; OpenDiLoCo trained 1.1B across 3 countries on 127–935 Mbit/s at 90–95%
	336	utilization, all-reduce just 6.9% of time).
	337
	338	(E) Portable kernels for a heterogeneous swarm (cross-vendor). Slang
	339	(§4.2.x): one source → SPIR-V/Metal/CUDA/CPU — cleanest authoring layer. Vulkan
	340	compute: 80–95% of native, genuinely cross-vendor. SYCL: matches/exceeds CUDA on
	341	NVIDIA, near-native AMD/Intel (Intel Xe ~23%, driver immaturity). WebGPU/WGSL:
	342	widest reach (LlamaWeb: 16 devices/8 vendors incl. mobile) but heavy tax — up to
	343	10× prefill / 2.5× decode slower than native (RTX 5080), per-dispatch overhead
	344	24–71µs dominates, browser variance (1 vs 52 tok/s). Reach, not throughput.
	345	(Vulkan's "80–95% of native" is single-source/weak — treat as indicative.)
	346
	347	(F) Decentralized GPU markets — hype gap is real. io.net: a Sybil attack
	348	spoofed ~1.8M fake GPUs; after a security upgrade real connections fell 600k→100k.
	349	Pooling primitives (MIG/MPS/time-slice) only fractionalize ONE GPU among tenants;
	350	none lets one job span GPUs on different hosts without the datacenter interconnect.
	351
	352	### 4.6.1 GPU-lane feasibility verdict (offload over hole-punched P2P today)
	353
	354	\| Workload \| Verdict \|
	355	\|---\|---\|
	356	\| Coarse containerized batch on a remote GPU \| ✅ works (latency-tolerant) \|
	357	\| Compute-bound async API remoting (inference / a training step) \| ✅ ~2–5% if compute-bound + low host transfer \|
	358	\| Pipeline-parallel inference (tiny activations) \| ✅ partition lane, no determinism needed \|
	359	\| Low-communication training (DiLoCo family) \| ✅ the only training that survives WAN \|
	360	\| Verified GPU compute by re-execution \| ⚠️ only via RepOps (30–370% tax) + refereed delegation \|
	361	\| Tensor-parallel / collective all-reduce \| ❌ needs datacenter NVLink/IB fabric \|
	362	\| Chatty/fine-grained API remoting, big host transfers \| ❌ collapses on commodity links \|
	363
	364	Net: the GPU/native lane is a genuine partition + trust lane (capacity
	365	unlock — exactly what the two-axis model predicts). It reaches verify only by
	366	paying the RepOps determinism tax — the GPU analogue of the CPU sandbox's
	367	determinism requirement, just far more expensive.
	368
	369	---
	370
	371	## 4.9 The Hyperdevice abstraction (capstone — the swarm as one device)
	372
	373	The unifying goal: a peer programs against one virtual device that is actually
	374	the union of the swarm. Lineage: Plan 9 ("the network is the computer" — mount
	375	remote resources into your namespace), MOSIX/Kerrighed (transparent process
	376	migration), Ray/exo (cluster-as-one-endpoint). The novelty here is that the
	377	swarm device is verifiable (content-addressed tasks) over untrusted peers —
	378	which the prior art assumed away.
	379
	380	The enabling seam. Sandboxed computation touches the outside world ONLY through
	381	a gated host-call ABI (VERIFIED for libriscv: deny-by-default, host installs every
	382	syscall; the guest cannot tell whether a call was serviced locally or routed to a
	383	peer). That ABI boundary IS the virtualization layer — the host intercepts each
	384	guest call and decides: run locally, or fan out to the swarm. The guest is oblivious.
	385
	386	Components (all already specified above):
	387	- Capability namespace / registry = the membership Autobase (`member/*` + a
	388	`caps/*` view of what each peer offers: cpu, gpu:vendor, model:X, storage).
	389	- ABI router = the virtualization seam: intercept each sandbox host-call →
	390	local-or-route via the scheduler.
	391	- Scheduler / placement = locality + capability-match + memory-weighted (exo-
	392	style) layer assignment. Invisible to the guest.
	393	- Four planes carry it: datagram hot path (P3, activations/signals), durable
	394	Autobase (P2, coordination/results/verdicts), **blob plane (P4, weights/datasets/
	395	frames by hash), and a stream plane** (Keet's decomposition: real-time media
	396	over WebRTC/udx with signaling on the control plane — §Keet finding).
	397
	398	**Tiers of invisibility (set by the latency wall — local memory ~100ns vs remote
	399	~ms is 10^6x, so the abstraction MUST be coarse-grained):**
	400
	401	\| Tier \| What \| How visible \|
	402	\|---\|---\|---\|
	403	\| Fully invisible \| deterministic CPU compute in the sandbox \| guest is an ordinary program; host routes every syscall to the swarm — and results are memoized + T1-verified. The clean win, unique to this lane. \|
	404	\| Semi-visible \| pooled resources (a model, TBs of storage) \| addressed by handle; sharding/placement hidden, but you know it's pooled \|
	405	\| Explicit \| GPU/native + tightly-coupled \| no interception seam (native code talks to hardware directly) → explicit remote-GPU / pipeline protocol; poolable, not invisible \|
	406
	407	What it unifies — every use case is "a program on `/dev/swarm`":
	408	- basic compute → spawn content-addressed tasks (fully invisible, memoized).
	409	- AI inference → call a pipeline-sharded model (semi-visible; partition lane).
	410	- AI training → a job whose DiLoCo sync hides under the abstraction.
	411	- GPU/RTX → render/compute calls to a pooled remote GPU (explicit; native, T0/RepOps).
	412	- storage → a pooled drive/blob namespace.
	413
	414	Honest limits (the wall that sank 30 years of SSI projects): invisible only at
	415	COARSE grain (whole tasks, model calls, blob reads, streams) — NEVER at fine grain
	416	(per-instruction distributed shared memory over WAN is a performance disaster).
	417	Clean+verifiable for sandboxed CPU; pooled-but-explicit for GPU/native; impossible
	418	for fine-grained tightly-coupled work over P2P (the interconnect wall, §4.6 D).
	419
	420	The one-line statement: *a verifiable single-system-image over a P2P swarm —
	421	the sandbox syscall boundary is the virtualization layer, the two-axis model says
	422	which calls can be made invisible-and-verifiable (deterministic CPU) vs only
	423	pooled-and-trusted (GPU/native), and physics caps it at coarse grain.*
	424
	425	---
	426
	427	## 5. Protocol messages (schemas)
	428
	429	### P1 — enrollment (over blind-pairing) (VERIFIED API)
	430	```
	431	PeerClaim { identityPubKey: fixed32, writerKey: fixed32, attestationProof?: buffer }
	432	GrantCap { role: uint, exp: uint, sig: fixed64 } // signed by group owner
	433	```
	434	- Candidate sends `PeerClaim` as blind-pairing `userData`.
	435	- Owner `request.open(invitePub)` → verifies → writes `grant` + `add-writer` →
	436	`request.confirm({ key, encryptionKey, additional: GrantCap })`.
	437
	438	### P2 — durable ops (Autobase log; append-only, versioned)
	439	```
	440	Op = variant on .type:
	441	add-writer { v, writerKey }
	442	grant { v, identityPubKey, role, exp, sig }
	443	task { v, id, codeHash, inputHash, input\|inputRef, limits, tier }
	444	claim { v, id, worker, leaseMs } // optimistic soft-lock
	445	result { v, codeHash, inputHash, output\|outputRef, worker, sig }
	446	verdict { v, id, accepted, votes[] } // T1 outcome
	447	```
	448	View keys: `member/<idPub>`, `task/<id>`, `claim/<id>`, `result/<codeHash>/<inputHash>`, `verdict/<id>`.
	449
	450	### P3 — datagrams (secret-stream unordered; hints only, lossy)
	451	```
	452	TaskDgram = variant on .kind:
	453	task-available { v, id, codeHash, inputHash }
	454	claiming { v, id, worker }
	455	progress { v, id, pct }
	456	result-ready { v, id, codeHash, inputHash, output\|outputRef }
	457	```
	458	Datagrams are never authoritative — they front-run the fold for latency. The
	459	fold confirms or overrides.
	460
	461	---
	462
	463	## 6. Task lifecycle (state machine)
	464
	465	```
	466	submitted ──▶ (memo hit? ──▶ done)
	467	│ no
	468	▼
	469	announced (P2 task + P3 task-available)
	470	▼
	471	claimed (P2 claim + P3 claiming) ── lease expires ──▶ re-announced
	472	▼
	473	executing (sandbox, bounded) ── timeout/fuel ──▶ failed ──▶ re-announced
	474	▼
	475	produced (P2 result + P3 result-ready)
	476	▼
	477	[T1] verifying (k-of-n) ── disagree ──▶ disputed ──▶ trustPolicy
	478	▼
	479	accepted (verdict) ──▶ memoized as canonical result/*
	480	```
	481
	482	Claims are leased (`leaseMs`) and soft — two workers can race; the duplicate
	483	is wasted, not corrupt, because results are content-keyed and verified. Optimistic
	484	by design (matches `optimistic:true` writes).
	485
	486	---
	487
	488	## 7. Security model
	489
	490	- Sandbox escape → WASM host exposes only whitelisted imports; no ambient
	491	authority. T0-only for weaker sandboxes.
	492	- DoS → mandatory `limits`; workers decline unbounded tasks.
	493	- Result poisoning → T1 voting; results signed by worker pubkey; disagreement
	494	= dispute, not silent accept.
	495	- Sybil (fake votes) → votes only count from enrolled `member/*` writers;
	496	enrollment is gated by the owner. (Does not defend against a malicious owner —
	497	trust root is the group owner / invite issuer.)
	498	- Replay / MITM at enrollment → handled by blind-pairing: discoveryKey-bound
	499	response + invite-secret signature (VERIFIED, blind-pairing-core §_openResponse).
	500	- Code provenance → `codeHash` is the artifact identity; workers may pin an
	501	allowlist of codeHashes (refuse unknown code) — strongly recommended.
	502
	503	---
	504
	505	## 8. Honest limits (what this spec does NOT solve)
	506
	507	- Nondeterministic tasks can never be canonical-memoized — only T0-trusted or
	508	T2-fingerprinted. The float-reordering problem (research §determinism gate) is
	509	real; cross-hardware bit-exactness for FP/ML is not assumed.
	510	- No defense against a dishonest group owner — the owner is the trust root.
	511	- T1 voting assumes ≥⌈k/2⌉+1 honest enrolled workers — same 1-of-N-class
	512	assumption family as optimistic systems, weaker (majority, not 1-of-N).
	513	- Large inputs/outputs need a blob channel (hyperdrive / hyper-overlay /
	514	hyperblobs) referenced by hash; inline bytes are for small payloads only.
	515
	516	---
	517
	518	## 9. Phased roadmap
	519
	520	- P0 Enrollment + membership gate + durable plane (T0, bare-worker). Prove
	521	submit→result over `replicateAndSync` (brittle).
	522	- P1 Datagram hot path + prediction/reconcile. Measure latency (hyper-lab).
	523	- P2 WASM sandbox + fuel/timeout limits. Content-addressed code allowlist.
	524	- P3 T1 replication+vote + dispute hook + `verdict` ops.
	525	- P4 Large I/O via blob channel; T2 statistical seam.
	526	- P5 (research) T3 zkVM proof tier.
	527
	528	---
	529
	530	## 10. Open questions
	531
	532	1. RESOLVED (§4.2.1): WASM (Wasmtime/WAMR) for v0.1; libriscv/RISC-V as P3+ target
	533	pending its two unresolved determinism questions. Remaining: confirm the exact
	534	Bare-compatible WASM runtime + native-addon embedding cost, and validate its
	535	deterministic-fuel + NaN-canonicalization config (rate-limited out of this round).
	536	2. Datagram passthrough: does the hyperswarm `connection` forward
	537	`trySend()`/`'message'` to secret-stream's unordered path un-wrapped? (Flagged
	538	in the prototype — verify before relying on it.)
	539	3. Separate membership autobase vs namespaced keys in one base? (v0.1: one base.)
	540	4. Lease length + re-announce backoff tuning.
	541	5. keet-identity attestation (T0 own-device flavor) — confirm `verifyIdentityProofs`
	542	API against keet-identity-key source; pin exact 3.1.0 (ce v2/v3 boundary).
	543
	544	---
	545
	546	## Dependencies (pinned per ecosystem rundown #76/#77/#78)
	547	```
	548	autobase 7.28.1 · hyperbee 2.27.3 (classic, NOT hyperbee2) · hyperdht ≥6.32.0
	549	hyperswarm · blind-pairing · corestore · compact-encoding · hyperschema
	550	keet-identity-key 3.1.0 (EXACT) # ce v2/v3 split at attestation boundary
	551	```
	552	```
	553	```

index.js +44 -0

…	…	@@ -0,0 +1,44 @@
	1	// index.js — hypercompute entry point. P0: a local, in-memory demo of the core
	2	// (content-addressed submit -> execute -> memoized result). Networked
	3	// own/join modes are scaffolded in lib/enroll.js + lib/transport.js (P1).
	4	//
	5	// Run: bare index.js demo (or: node index.js demo)
	6
	7	const b4a = require('b4a')
	8	const crypto = require('hypercore-crypto')
	9	const { ComputeNode } = require('./lib/node')
	10
	11	function inputHashOf (input) {
	12	return b4a.toString(crypto.hash(b4a.from(JSON.stringify(input))), 'hex')
	13	}
	14
	15	async function demo () {
	16	const codeHash = 'square-v1'
	17	const executors = { [codeHash]: async (i) => i.n * i.n }
	18
	19	const node = await ComputeNode.create(undefined, null, { executors })
	20	await node.grantSelf('owner')
	21
	22	const input = { n: 7 }
	23	const ih = inputHashOf(input)
	24
	25	console.log('submit n=7 ...')
	26	const a = await node.submit(codeHash, ih, input)
	27	console.log(' ->', a.memoHit ? 'memo hit' : 'memo miss (task queued: ' + a.id.slice(0, 12) + ')')
	28
	29	await node.runPendingTasks()
	30	console.log('result (via the fold):', await node.lookup(codeHash, ih))
	31
	32	const b = await node.submit(codeHash, ih, input)
	33	console.log('re-submit n=7 ->', b.memoHit ? 'memo HIT, output ' + b.output : 'miss')
	34
	35	await node.close()
	36	}
	37
	38	const cmd = (typeof Bare !== 'undefined' ? Bare.argv : process.argv).slice(2)[0] \|\| 'demo'
	39
	40	if (cmd === 'demo') {
	41	demo().catch((err) => { console.error(err); (typeof Bare !== 'undefined' ? Bare : process).exit(1) })
	42	} else {
	43	console.log('hypercompute (P0)\n usage: bare index.js demo\n own/join (networked) — see lib/enroll.js + lib/transport.js (P1)\n see SPEC.md for the full design.')
	44	}

lib/enroll.js +71 -0

…	…	@@ -0,0 +1,71 @@
	1	// lib/enroll.js — enrollment ceremony (SPEC §4-enroll). SCAFFOLD (P1).
	2	//
	3	// The one-time "secret handshake" that mints a peer into the membership view.
	4	// Built on blind-pairing (API VERIFIED against shipped source this session):
	5	// BlindPairing.createInvite(key, opts) -> { id, invite, publicKey: invitePub }
	6	// blind.addMember({ discoveryKey, onadd }) // owner; request.open()/confirm()/deny()
	7	// blind.addCandidate({ invite, userData, onadd }) // joiner; onadd(paired)
	8	//
	9	// The ceremony's ONLY durable output is a `grant` (+ add-writer) in the pool —
	10	// after that the hot path authenticates by membership lookup (membership.js).
	11	//
	12	// NOT wired into P0's test (which uses direct grant/add-writer). Kept as the
	13	// real shape to build out in P1.
	14
	15	const BlindPairing = require('blind-pairing')
	16	const crypto = require('hypercore-crypto')
	17	const b4a = require('b4a')
	18
	19	// OWNER: accept candidates against an invite, enroll them into the pool.
	20	// `verify(claim)` returns a role string to grant, or null to deny.
	21	function acceptMembers (swarm, node, { verify } = {}) {
	22	const blind = new BlindPairing(swarm)
	23	const inviteStore = new Map()
	24
	25	blind.addMember({
	26	discoveryKey: crypto.discoveryKey(node.base.key),
	27	onadd: async (request) => {
	28	const rec = inviteStore.get(b4a.toString(request.inviteId, 'hex'))
	29	if (!rec \|\| rec.used) return request.deny({ status: 2 }) // INVITE_USED
	30
	31	const userData = request.open(rec.invitePub) // verifies invite-holding
	32	const claim = JSON.parse(b4a.toString(userData)) // P0: JSON; P1: compact-encoding
	33	const role = verify ? await verify(claim) : 'worker'
	34	if (!role) return request.deny()
	35
	36	// CONTROL-PLANE WRITE = the enrollment
	37	await node.grant(claim.identityPubKey, role)
	38	if (claim.writerKey) await node.addWriter(claim.writerKey)
	39
	40	rec.used = true
	41	request.confirm({ key: node.base.key, encryptionKey: node.base.encryptionKey })
	42	}
	43	})
	44
	45	return {
	46	blind,
	47	mintInvite () {
	48	const { id, invite, publicKey } = BlindPairing.createInvite(node.base.key)
	49	inviteStore.set(b4a.toString(id, 'hex'), { invitePub: publicKey, used: false })
	50	return b4a.toString(invite, 'hex')
	51	}
	52	}
	53	}
	54
	55	// JOINER: present a claim, receive the pool key.
	56	function join (swarm, node, inviteHex) {
	57	const blind = new BlindPairing(swarm)
	58	const userData = b4a.from(JSON.stringify({
	59	identityPubKey: node.idHex,
	60	writerKey: b4a.toString(node.writerKey, 'hex')
	61	}))
	62	return new Promise((resolve, reject) => {
	63	blind.addCandidate({
	64	invite: b4a.from(inviteHex, 'hex'),
	65	userData,
	66	onadd: async (paired) => resolve(paired) // { key, encryptionKey, data }
	67	}).pairing.catch(reject)
	68	})
	69	}
	70
	71	module.exports = { acceptMembers, join }

lib/membership.js +22 -0

…	…	@@ -0,0 +1,22 @@
	1	// lib/membership.js — the hot-path gate (SPEC §2, §4-enroll).
	2	//
	3	// After the one-time enrollment ceremony writes a `grant` into the durable
	4	// plane, every connection check is just "is this peer's key in member/*" —
	5	// a local Hyperbee read (~µs, zero network). Enrollment is expensive and rare;
	6	// this check is cheap and per-connection.
	7
	8	const b4a = require('b4a')
	9
	10	async function isMember (base, publicKey) {
	11	const hex = b4a.isBuffer(publicKey) ? b4a.toString(publicKey, 'hex') : publicKey
	12	const node = await base.view.get('member/' + hex)
	13	return !!node
	14	}
	15
	16	async function memberRole (base, publicKey) {
	17	const hex = b4a.isBuffer(publicKey) ? b4a.toString(publicKey, 'hex') : publicKey
	18	const node = await base.view.get('member/' + hex)
	19	return node ? node.value.role : null
	20	}
	21
	22	module.exports = { isMember, memberRole }

lib/node.js +98 -0

…	…	@@ -0,0 +1,98 @@
	1	// lib/node.js — ComputeNode: owns the corestore + durable-plane Autobase and
	2	// the T0 (trusted, native) executor table. SPEC §2, §4.
	3	//
	4	// P0 scope: enrollment/transport are scaffolded (see enroll.js / transport.js)
	5	// but the core — content-addressed submit -> execute -> result over the
	6	// deterministic fold, with memoization — is real and exercised by test/e2e.js
	7	// via replicateAndSync (no network).
	8
	9	const Corestore = require('corestore')
	10	const ram = require('random-access-memory')
	11	const b4a = require('b4a')
	12	const crypto = require('hypercore-crypto')
	13	const { openPool, taskId, resultKey } = require('./pool')
	14
	15	class ComputeNode {
	16	constructor (store, bootstrap, { identity, executors } = {}) {
	17	this.store = store
	18	this.identity = identity \|\| crypto.keyPair()
	19	this.executors = executors \|\| {} // codeHash -> async (input) => output (T0 native)
	20	this.base = openPool(store, bootstrap)
	21	}
	22
	23	// storage: a Corestore, a path string, or undefined (=> in-memory)
	24	static async create (storage, bootstrap = null, opts = {}) {
	25	let store = storage
	26	if (!(storage instanceof Corestore)) {
	27	store = new Corestore(storage \|\| ram)
	28	}
	29	const node = new ComputeNode(store, bootstrap, opts)
	30	await node.base.ready()
	31	return node
	32	}
	33
	34	get key () { return this.base.key } // pool bootstrap key (share to join)
	35	get writerKey () { return this.base.local.key } // this node's autobase writer key
	36	get idHex () { return b4a.toString(this.identity.publicKey, 'hex') }
	37
	38	// --- membership (P0: direct; real path is enroll.js blind-pairing) ----------
	39	async grantSelf (role = 'worker') {
	40	await this.base.append({ type: 'grant', identityPubKey: this.idHex, role })
	41	}
	42
	43	async grant (identityPubKey, role = 'worker') {
	44	const hex = b4a.isBuffer(identityPubKey) ? b4a.toString(identityPubKey, 'hex') : identityPubKey
	45	await this.base.append({ type: 'grant', identityPubKey: hex, role })
	46	}
	47
	48	async addWriter (writerKey, indexer = false) {
	49	const hex = b4a.isBuffer(writerKey) ? b4a.toString(writerKey, 'hex') : writerKey
	50	await this.base.append({ type: 'add-writer', writerKey: hex, indexer })
	51	}
	52
	53	// --- content-addressed compute (the core, SPEC §4) --------------------------
	54	async lookup (codeHash, inputHash) {
	55	const node = await this.base.view.get(resultKey(codeHash, inputHash))
	56	return node ? node.value.output : null
	57	}
	58
	59	// submit returns { output, memoHit:true } on a cache hit, else { id, memoHit:false }
	60	async submit (codeHash, inputHash, input) {
	61	const cached = await this.lookup(codeHash, inputHash)
	62	if (cached !== null && cached !== undefined) return { output: cached, memoHit: true }
	63	const id = taskId(codeHash, inputHash)
	64	await this.base.append({ type: 'task', id, codeHash, inputHash, input })
	65	return { id, memoHit: false }
	66	}
	67
	68	// worker loop: execute every task that has a registered executor and no result yet.
	69	// Returns the number of tasks executed.
	70	async runPendingTasks () {
	71	let executed = 0
	72	const tasks = []
	73	for await (const { value: task } of this.base.view.createReadStream({ gte: 'task/', lt: 'task0' })) {
	74	tasks.push(task)
	75	}
	76	for (const task of tasks) {
	77	const existing = await this.base.view.get(resultKey(task.codeHash, task.inputHash))
	78	if (existing) continue
	79	const exec = this.executors[task.codeHash]
	80	if (!exec) continue
	81	await this.base.append({ type: 'claim', id: task.id, worker: this.idHex })
	82	const output = await exec(task.input) // T0 native execution
	83	await this.base.append({
	84	type: 'result',
	85	codeHash: task.codeHash,
	86	inputHash: task.inputHash,
	87	output,
	88	worker: this.idHex
	89	})
	90	executed++
	91	}
	92	return executed
	93	}
	94
	95	close () { return this.base.close() }
	96	}
	97
	98	module.exports = { ComputeNode }

lib/pool.js +68 -0

…	…	@@ -0,0 +1,68 @@
	1	// lib/pool.js — the DURABLE plane (SPEC §2, §4).
	2	//
	3	// One optimistic Autobase whose view is a Hyperbee. Mutable state = a
	4	// deterministic fold (`apply`) over the append-only op log. View keys:
	5	// member/<idPub> -> { role }
	6	// task/<id> -> task op
	7	// claim/<id> -> { worker } (optimistic soft-lock)
	8	// result/<codeHash>/<input> -> result op (the memoized chunk)
	9	//
	10	// apply() MUST stay deterministic (no Date.now/random/external reads) — SPEC §3.
	11
	12	const Autobase = require('autobase')
	13	const Hyperbee = require('hyperbee')
	14	const b4a = require('b4a')
	15	const crypto = require('hypercore-crypto')
	16
	17	function hash (str) {
	18	return b4a.toString(crypto.hash(b4a.from(str)), 'hex')
	19	}
	20
	21	// content-addressed identity of a unit of work (SPEC §4.1)
	22	function taskId (codeHash, inputHash) {
	23	return hash(codeHash + ':' + inputHash)
	24	}
	25
	26	function resultKey (codeHash, inputHash) {
	27	return 'result/' + codeHash + '/' + inputHash
	28	}
	29
	30	function openPool (store, bootstrap) {
	31	return new Autobase(store, bootstrap, {
	32	valueEncoding: 'json',
	33	open (s) {
	34	return new Hyperbee(s.get('view'), {
	35	keyEncoding: 'utf-8',
	36	valueEncoding: 'json'
	37	})
	38	},
	39	// the keet pattern (SPEC §2): optimistic writes never block on quorum
	40	optimistic: true,
	41	ackInterval: 1000,
	42	ackThreshold: 4,
	43	async apply (nodes, view, host) {
	44	for (const node of nodes) {
	45	const op = node.value
	46	switch (op && op.type) {
	47	case 'add-writer':
	48	await host.addWriter(b4a.from(op.writerKey, 'hex'), { indexer: !!op.indexer })
	49	break
	50	case 'grant': // enrollment result (SPEC §4-enroll)
	51	await view.put('member/' + op.identityPubKey, { role: op.role })
	52	break
	53	case 'task': // content-addressed work unit
	54	await view.put('task/' + op.id, op)
	55	break
	56	case 'claim': // optimistic soft-lock to dedupe workers
	57	await view.put('claim/' + op.id, { worker: op.worker })
	58	break
	59	case 'result': // memoized output, keyed by content
	60	await view.put(resultKey(op.codeHash, op.inputHash), op)
	61	break
	62	}
	63	}
	64	}
	65	})
	66	}
	67
	68	module.exports = { openPool, taskId, resultKey, hash }

lib/transport.js +46 -0

…	…	@@ -0,0 +1,46 @@
	1	// lib/transport.js — the two communication planes (SPEC §2, §3). SCAFFOLD (P1).
	2	//
	3	// DURABLE : replicate the pool Autobase over each hyperswarm connection.
	4	// HOT PATH : unordered datagrams (conn.trySend / 'message') for ephemeral
	5	// task-available / claiming / result-ready signals. No HOL, no ack.
	6	//
	7	// HOT-PATH GATE: drop any connection whose remotePublicKey is not in member/*.
	8	//
	9	// ⚠️ OPEN (SPEC §10): verify hyperswarm `connection` forwards trySend()/'message'
	10	// to secret-stream's unordered path un-wrapped. If not, drop to a dedicated
	11	// secret-stream or the raw udx stream. Test before relying on it.
	12
	13	const { isMember } = require('./membership')
	14
	15	function startTransport (swarm, node, { gate = true } = {}) {
	16	swarm.join(node.base.discoveryKey, { server: true, client: true })
	17	node.peers = new Set()
	18
	19	swarm.on('connection', async (conn) => {
	20	// 1) replicate the durable plane
	21	node.store.replicate(conn)
	22
	23	// 2) hot-path membership gate
	24	if (gate && !(await isMember(node.base, conn.remotePublicKey))) {
	25	// conn.destroy(); return // enable once enrollment is the only join path
	26	}
	27
	28	// 3) DATA PLANE: unordered datagrams (the latency win) — see OPEN note above
	29	conn.on('message', (buf) => {
	30	try { node.onDatagram && node.onDatagram(conn, JSON.parse(buf.toString())) } catch {}
	31	})
	32
	33	node.peers.add(conn)
	34	conn.on('close', () => node.peers.delete(conn))
	35	})
	36	}
	37
	38	// fire-and-forget datagram to all connected peers (the hot path)
	39	function broadcast (node, dgram) {
	40	const buf = Buffer.from(JSON.stringify(dgram))
	41	for (const conn of node.peers \|\| []) {
	42	if (typeof conn.trySend === 'function') conn.trySend(buf)
	43	}
	44	}
	45
	46	module.exports = { startTransport, broadcast }

package.json +22 -0

…	…	@@ -0,0 +1,22 @@
	1	{
	2	"name": "hypercompute",
	3	"version": "0.0.1",
	4	"description": "Verifiable P2P compute pool on the Holepunch stack — see SPEC.md",
	5	"main": "index.js",
	6	"scripts": {
	7	"test": "brittle test/*.js"
	8	},
	9	"dependencies": {
	10	"autobase": "^7.28.1",
	11	"b4a": "^1.6.7",
	12	"corestore": "^6.18.4",
	13	"hyperbee": "^2.27.3",
	14	"hypercore-crypto": "^3.6.1",
	15	"random-access-memory": "^6.2.1"
	16	},
	17	"devDependencies": {
	18	"autobase-test-helpers": "^2.2.0",
	19	"brittle": "^3.7.0"
	20	},
	21	"license": "MIT"
	22	}

research/README.md +33 -0

…	…	@@ -0,0 +1,33 @@
	1	# Research
	2
	3	The evidence base behind [SPEC.md](../SPEC.md). Each report was produced by a
	4	fan-out → fetch → adversarial-verify deep-research pass. Raw machine outputs are
	5	in [`raw/`](raw/); distilled findings are folded directly into the spec.
	6
	7	> Verification caveat: several runs hit server-side rate limits in the
	8	> adversarial-verification phase. Where that happened it is flagged below and in
	9	> the spec's provenance notes. The GPU claims were re-verified in a separate
	10	> 2-vote pass (`raw/05-gpu-verification.raw.json`) — that pass is what corrected
	11	> the NVLink bandwidth, the io.net endpoint (600k→100k), the WebGPU device
	12	> (RTX 5080), and the GPU-determinism framing.
	13
	14	## Reports
	15
	16	\| # \| Topic \| Raw \| Status \|
	17	\|---\|---\|---\|---\|
	18	\| 01 \| P2P / decentralized computation — state of the art. Three tiers: incremental deterministic computation over logs (Autobase) = production; trustless verification (fraud proofs, zkVMs) = the hard problem; DePIN/decentralized-AI = real at orchestration. \| `raw/01-p2p-computation.raw.json` \| verified (22/25 confirmed) \|
	19	\| 02 \| libriscv / godot-sandbox as the executor. Confirmed: fuel metering + deny-by-default sandbox. NOT guaranteed: cross-hardware bit-exact determinism (must be enforced). Verdict: WASM now, libriscv as P3+ target. \| `raw/02-libriscv-executor.raw.json` \| verified (13/25; key gaps noted) \|
	20	\| 03 \| Distributed AI (inference + training). exo pipeline-parallel; Petals; DiLoCo/INTELLECT-2/Nous-Psyche. Splitting a model is a capacity win, not a speed win. \| `raw/03-distributed-ai.raw.json` \| ⚠️ verification rate-limited (primary-source extractions; INTELLECT-2 corroborated in #01) \|
	21	\| 04 \| Distributed GPU usage (salvaged). API remoting, the interconnect wall, cross-vendor portable compute (Slang/Vulkan/SYCL/WebGPU), GPU determinism (RepOps), DePIN markets. Salvaged from a hung run. \| `raw/04-distributed-gpu-salvaged.raw.txt` \| salvaged → re-verified in #05 \|
	22	\| 05 \| GPU-claims verification. 2-vote adversarial re-verification of the 8 load-bearing GPU claims. 5 confirmed, 3 corrected (interconnect numbers, io.net endpoint, WebGPU device, determinism framing). \| `raw/05-gpu-verification.raw.json` \| verified \|
	23
	24	## How these map into the spec
	25
	26	- #01 → §0 foundation, §3 determinism, §4.4 trust tiers.
	27	- #02 → §4.2.1 executor verdict (WASM now / libriscv later) + §4.2.2 determinism-enforcement.
	28	- #03 → §4.6 (D) communication wall, the AI-as-partition framing.
	29	- #04 + #05 → §4.6 GPU/native lane + §4.6.1 feasibility verdict.
	30
	31	Methodology note: the GPU verification pass is itself a small live instance of
	32	the spec's own T1/refereed-delegation pattern — an adversarial check caught real
	33	errors before they entered the durable artifact.

research/raw/01-p2p-computation.raw.json +331 -0

…	…	@@ -0,0 +1,331 @@
	1	{
	2	"summary": "Deep research harness — fan-out web searches, fetch sources, adversarially verify claims, synthesize a cited report.",
	3	"agentCount": 112,
	4	"logs": [
	5	"Q: Research the state of the art in peer-to-peer (P2P) / decentralized computation:…",
	6	"Decomposed into 6 angles: Verifiable computation / zkVM state of the art (technical+benchmarks), Decentralized GPU/AI compute markets — real vs hype (practitioner/skeptical), P2P compute grids & content-addressed compute-over-data (systems mechanics), Trustless verification trust models — TEEs, optimistic fraud proofs, refereed delegation, Deterministic computation over append-only logs / local-first foundations (primary sources), Academic foundations: interactive proofs, MPC, FHE, BFT (key papers)",
	7	"Decentralized GPU/AI compute markets — real vs hype (practitioner/skeptical): 6 results",
	8	"Deterministic computation over append-only logs / local-first foundations (primary sources): 6 results",
	9	"Verifiable computation / zkVM state of the art (technical+benchmarks): 6 results",
	10	"Verifiable computation / zkVM state of the art (technical+benchmarks): 4 novel (2 filtered)",
	11	"Academic foundations: interactive proofs, MPC, FHE, BFT (key papers): 6 results",
	12	"Academic foundations: interactive proofs, MPC, FHE, BFT (key papers): 4 novel (2 filtered)",
	13	"Trustless verification trust models — TEEs, optimistic fraud proofs, refereed delegation: 6 results",
	14	"Trustless verification trust models — TEEs, optimistic fraud proofs, refereed delegation: 5 novel (1 filtered)",
	15	"P2P compute grids & content-addressed compute-over-data (systems mechanics): 6 results",
	16	"P2P compute grids & content-addressed compute-over-data (systems mechanics): 4 novel (2 filtered)",
	17	"Fetched 29 sources → 128 claims → verifying top 25",
	18	"\"Jolt is a zkVM based on the Lasso lookup argument …\": 3-0 ✓",
	19	"\"Prime Intellect trained INTELLECT-2, a 32-billion-…\": 3-0 ✓",
	20	"\"Jolt reduces zkVM proving overhead to below 6 orde…\": 3-0 ✓",
	21	"\"Jolt proves over 5x faster than RISC Zero and up t…\": 0-3 ✗",
	22	"\"Trustless verification of computation from untrust…\": 3-0 ✓",
	23	"\"INTELLECT-2 is a 32B-parameter model trained via g…\": 3-0 ✓",
	24	"\"Bacalhau is a distributed compute orchestration fr…\": 3-0 ✓",
	25	"\"Distributed compute over untrusted nodes is verifi…\": 3-0 ✓",
	26	"\"Bacalhau executes jobs in standardized runtimes (D…\": 2-1 ✓",
	27	"\"Bacalhau does not provide a trustless verification…\": 3-0 ✓",
	28	"\"TrueBit verifies arbitrary off-chain computation v…\": 3-0 ✓",
	29	"\"Arbitrum's interactive fraud proof system resolves…\": 3-0 ✓",
	30	"\"The verification game's binary-search dispute idea…\": 3-0 ✓",
	31	"\"The on-chain verification cost is logarithmic in c…\": 3-0 ✓",
	32	"\"On-chain verification on Ethereum is limited to va…\": 3-0 ✓",
	33	"\"The fraud-proof trust model is a 1-of-N honest-par…\": 3-0 ✓",
	34	"\"CRDTs automatically merge concurrent state changes…\": 1-2 ✗",
	35	"\"The only class of concurrent change a CRDT cannot …\": 0-3 ✗",
	36	"\"Autobase's view is produced by deterministically a…\": 3-0 ✓",
	37	"\"Writer nodes carry explicit causal references to p…\": 3-0 ✓",
	38	"\"CRDTs are a theoretical technique for implementing…\": 3-0 ✓",
	39	"\"Autobase coordinates multiple writers via a vote-a…\": 3-0 ✓",
	40	"[stall] agent \"v2:Autobase is a multiwriter data structure\" stalled (no progress) after 194s — retrying (1/5)",
	41	"\"Autobase is a multiwriter data structure that comb…\": 3-0 ✓",
	42	"\"Autobase produces a deterministic total ordering (…\": 3-0 ✓",
	43	"\"Correct deterministic computation in Autobase requ…\": 3-0 ✓",
	44	"Verify done: 25 claims → 22 confirmed, 3 killed"
	45	],
	46	"result": {
	47	"question": "Research the state of the art in peer-to-peer (P2P) / decentralized computation: how computation (not just data/files) is distributed, coordinated, and verified across untrusted or trust-minimized peers.\n\nCover these dimensions and name concrete systems, papers, and projects with their tradeoffs:\n\n1. INCREMENTAL / DETERMINISTIC COMPUTATION OVER LOGS: event-sourcing / deterministic fold over append-only logs as a computation model. How CRDTs, Autobase (Holepunch/Hypercore), and local-first systems (Ink & Switch) treat \"mutable state = deterministic reduction of an immutable event log.\" Where this scales and where it breaks.\n\n2. DISTRIBUTED TASK EXECUTION / COMPUTE GRIDS (P2P): volunteer/grid computing (BOINC), serverless-over-P2P, content-addressed compute and memoization (e.g. keyed by code hash + input hash), Bacalhau / compute-over-data (IPFS/Filecoin), Golem, iExec, Nunet, Ray-style distributed execution but decentralized. Deterministic/reproducible builds (Nix) as content-addressed computation.\n\n3. TRUSTLESS / TRUST-MINIMIZED VERIFICATION OF COMPUTATION: the core hard problem. Verifiable computation, deterministic re-execution, replication+voting, trusted execution environments (TEEs/SGX), optimistic fraud proofs (optimistic rollups, Truebit, interactive verification games / refereed delegation), and succinct proofs: zkVMs and the 2023-2026 wave (RISC Zero, SP1/Succinct, Jolt, Cairo/StarkWare, zkWASM, Nexus). Proof generation cost vs verification cost tradeoffs and where they are practical today.\n\n4. DECENTRALIZED PHYSICAL/AI COMPUTE NETWORKS (DePIN): the recent wave of decentralized GPU/AI training & inference markets — Akash, io.net, Render, Gensyn (verifiable ML training), Bittensor, Prime Intellect (decentralized training / DiLoCo / OpenDiLoCo), Petals-style distributed inference. What is real vs hype.\n\n5. ACADEMIC FOUNDATIONS: verifiable computation history (Goldwasser-Kalai-Rothblum / interactive proofs, PCP), secure multiparty computation (MPC), homomorphic encryption (FHE) for private computation, BFT consensus as computation agreement, and how these map onto P2P settings.\n\n6. WHERE THE HOLEPUNCH / HYPERCORE ECOSYSTEM SITS: given Hypercore (append-only logs), Autobase (multi-writer deterministic views), Hyperswarm (DHT/transport), and content-addressed stores — what's natively solved (incremental deterministic computation, content-addressed output caching) vs what would require building (spatial task distribution, trustless verification of others' compute). What an architect building \"chunked computation\" on this stack realistically can and cannot do today.\n\nPrioritize: concrete named systems with how-they-work mechanics, honest maturity assessment (production vs research vs vaporware) as of 2025-2026, and the fundamental tradeoffs (especially proof cost, determinism requirements, and trust models). Surface key papers and primary sources.",
	48	"summary": "As of 2025-2026, P2P/decentralized computation splits cleanly into three maturity tiers. (1) Incremental deterministic computation over append-only logs is PRODUCTION-READY and is exactly what the Holepunch/Hypercore stack natively solves: Autobase materializes mutable state as a deterministic fold over immutable, causally-linked logs, with multi-writer ordering resolved by DAG linearization + vote-quorum, and CRDTs provide the same coordination-free strong-eventual-consistency model in local-first systems. (2) Trustless verification of others' computation remains the hard, unsolved-natively problem; the field has converged on a small set of mechanisms with sharply different cost/trust profiles — interactive fraud proofs (TrueBit, Arbitrum) that bisect a disputed computation down to a single re-executable step giving logarithmic on-chain verification cost under a 1-of-N honest-party assumption, succinct proofs/zkVMs (Jolt's sumcheck/Lasso lineage vs RISC Zero/SP1's FRI-STARK lineage) whose proving overhead has dropped from ~8 to below 6 orders of magnitude but is still enormous, and lightweight statistical checks like locality-sensitive-hashing verification (TOPLOC). (3) Decentralized physical/AI compute is REAL at the orchestration and even training layer — Bacalhau moves containerized/WASM compute to data (but uses a trusted coordinated orchestrator, not trustless verification), and Prime Intellect actually trained and Apache-2.0-released INTELLECT-2, a 32B model, via permissionless globally-distributed async RL with on-chain-triggered TOPLOC validation of untrusted workers. The fundamental tradeoff throughout is that strong trustlessness (succinct proofs) costs orders of magnitude of compute, cheaper interactive/optimistic verification needs a determinism precondition plus a synchronous dispute window, and the cheapest schemes (replication, LSH fingerprinting, trusted orchestration) weaken the trust model.",
	49	"findings": [
	50	{
	51	"claim": "Autobase (Holepunch/Hypercore) is a production multi-writer data structure implementing event sourcing: mutable state is a deterministic reduction/fold over immutable, causally-linked append-only logs, with a deterministic total ordering derived from DAG analysis and a vote-quorum mechanism.",
	52	"confidence": "high",
	53	"vote": "merged 5 unanimous claims (15,16,17,18,19,20)",
	54	"sources": [
	55	"https://github.com/holepunchto/autobase/blob/main/README.md",
	56	"https://github.com/holepunchto/autobase/blob/main/DESIGN.md"
	57	],
	58	"evidence": "README: 'A multiwriter data structure... Using the event sourcing pattern, writers append blocks which are linearized into an eventually consistent order' and 'A view is one or more hypercores whose contents are created by deterministically applying the linearized nodes from writers.' DESIGN.md: ordering is 'a deterministic linearisation... achieved by traversing the graph according to clocks,' tie-broken by key comparison; 'each node in the graph accumulates votes from other writers (max 1 per writer)' and reaches quorum once 'referenced by a majority of writers.' Determinism is a hard requirement: apply() must mutate only its view argument and open() derive only from its store, because reordered messages are 'undone and reapplied internally' and external state cannot be rolled back. Verified against shipped source (lib/linearizer.js, topolist.js, clock.js — logical/Lamport clock, not wall-clock; no external consensus)."
	59	},
	60	{
	61	"claim": "CRDTs are the local-first / collaborative-app analog of this model: a coordination-free technique providing strong eventual consistency without a central server.",
	62	"confidence": "high",
	63	"vote": "3-0 (claim 21); note two related Ink&Switch over-generalizations were refuted",
	64	"sources": [
	65	"https://arxiv.org/pdf/2212.02618"
	66	],
	67	"evidence": "Collabs paper (Weidner et al.): 'Conflict-free Replicated Data Types (CRDTs) are a promising theoretical technique for implementing collaboration frameworks... while still providing strong eventual consistency.' SEC = replicas converge automatically without consensus/coordination (Shapiro et al. 2011 foundation). CAVEAT: adversarial verification REFUTED two stronger local-first framings — that CRDTs make 'all concurrent changes take effect' and that 'the only unresolvable conflict is concurrent updates to the same property of the same object' — so do not overstate CRDT automatic-merge guarantees."
	68	},
	69	{
	70	"claim": "Interactive fraud proofs verify arbitrary off-chain computation by bisecting a disputed execution (binary search) down to a single instruction step that a bounded on-chain arbiter re-executes directly — giving verification cost logarithmic in computation length, under a 1-of-N honest-party trust model that depends on deterministic state transitions.",
	71	"confidence": "high",
	72	"vote": "merged 6 unanimous claims (9,10,11,12,13,14)",
	73	"sources": [
	74	"https://people.cs.uchicago.edu/~teutsch/papers/truebit.pdf",
	75	"https://docs.arbitrum.io/how-arbitrum-works/interactive-fraud-proofs"
	76	],
	77	"evidence": "TrueBit (Teutsch & Reitwiessner, arXiv:1908.04756): a Solver/Challenger 'verification game' narrows the disputed portion over log t / log c rounds until 'the loop converges to the first, disputed computational step, and the Judges explicitly check this step'; total Judge work is O[(log t/log c)(cσ+log c)+2((2σ+1)log s+5+p)] — logarithmic in t, exponentially smaller than the t-step task. The bisection idea is attributed to Canetti, Riva & Rothblum (refereed delegation), which assumed >=1 of k clouds honest — an assumption TrueBit rejects as too strong for a purely rational network. Arbitrum Nitro implements the same: parties 'take turns bisecting their history commitments until they arrive at a single step of instruction' then submit a one-step proof to the OneStepProof contract on Ethereum, which executes only that step. Trust model: 'only a single honest party is required to defend Arbitrum' precisely because 'state transitions are entirely deterministic.' NUANCE: the 1-of-N safety guarantee presupposes L1 censorship-resistance/liveness, a bounded challenge window, and bonding capital; criticisms (verifier's dilemma, economic censorship) target the incentive layer, not the bisection mechanics."
	78	},
	79	{
	80	"claim": "Succinct-proof zkVMs are the strongest (trustless, non-interactive) verification approach and split into two theoretical lineages — Jolt (sumcheck/Lasso lookup argument) vs RISC Zero and SP1 (FRI-based STARKs) — with proving overhead now driven below 6 orders of magnitude, down from a historical ~8.",
	81	"confidence": "high",
	82	"vote": "merged claims 0 (3-0) and 1 (3-0)",
	83	"sources": [
	84	"https://a16zcrypto.com/posts/article/building-jolt/"
	85	],
	86	"evidence": "a16z 'Building Jolt': Jolt is 'based on the Lasso lookup argument and other sumcheck-based techniques' and 'emerged from a different theoretical lineage than STARK-based incumbents' (RISC Zero uses DEEP-ALI + batched FRI; SP1 is FRI/Poseidon2 over BabyBear — both independently confirmed). zkVMs historically added 'about 8 orders of magnitude of overhead compared to native execution'; 'the current version of Jolt brings this overhead below 6 orders of magnitude.' CAVEATS: vendor-authored (a16z built Jolt); baseline is native execution, not peer-reviewed independent benchmark; a separate stronger head-to-head claim ('Jolt 5x faster than RISC Zero / 2x faster than SP1') was REFUTED 0-3. The architectural lineage distinction and the order-of-magnitude overhead figure are sound; the relative speed ranking is not established. Practical takeaway: even 'below 6 OOM' means ~<100,000x native compute to prove — verification is cheap/succinct, proving remains the dominant cost."
	87	},
	88	{
	89	"claim": "Lightweight statistical verification of distributed AI inference is viable: TOPLOC, a locality-sensitive hashing scheme over intermediate activations, detects tampering/precision changes from untrusted workers across heterogeneous GPU hardware, far cheaper than zk-proofs or full re-execution.",
	90	"confidence": "high",
	91	"vote": "merged claims 3 (3-0) and 5 (3-0)",
	92	"sources": [
	93	"https://arxiv.org/abs/2505.07291",
	94	"https://www.primeintellect.ai/blog/intellect-2-release"
	95	],
	96	"evidence": "INTELLECT-2 paper: 'TOPLOC, which verifies rollouts from untrusted inference workers.' Mechanism (its own ICML 2025 paper, arXiv:2501.16007): LSH over intermediate activations that 'detects unauthorized modifications to models, prompts, or precision' robustly 'across diverse hardware configurations, GPU types, and algebraic reorderings.' Operational loop (Prime Intellect blog): workers upload signed rollout files; 'an on-chain event triggers TOPLOC validators to check them; accepted files feed the trainer, while invalid ones remove the submitting node from the pool.' SCOPING NUANCE: TOPLOC verifies inference/rollout generation, not arbitrary computation, and works by tamper-detection fingerprinting rather than soundness-proven verifiable computation — it is a trust-minimized statistical check, weaker than fraud/validity proofs but vastly cheaper."
	97	},
	98	{
	99	"claim": "Decentralized AI TRAINING at frontier-relevant scale is real, not vaporware: Prime Intellect trained and Apache-2.0-released INTELLECT-2, a 32B-parameter reasoning model, via fully asynchronous globally-distributed RL across a permissionless, heterogeneous swarm of compute contributors.",
	100	"confidence": "high",
	101	"vote": "merged claims 2 (3-0) and 4 (3-0)",
	102	"sources": [
	103	"https://arxiv.org/abs/2505.07291",
	104	"https://www.primeintellect.ai/blog/intellect-2-release"
	105	],
	106	"evidence": "'The first globally distributed reinforcement learning (RL) training run of a 32 billion parameter language model... across a dynamic, heterogeneous swarm of permissionless compute contributors' using 'fully asynchronous RL.' Strongest anti-vaporware evidence: open weights under Apache 2.0 with full codebase and training logs; improves on QwQ-32B on reasoning benchmarks; corroborated by InfoQ, MarkTechPost. Named infrastructure: PRIME-RL (async RL framework decoupling rollout/training/weight-broadcast), TOPLOC (untrusted-worker verification), SHARDCAST (weight distribution). HONEST QUALIFICATION: not every layer is fully P2P — weight distribution used trusted relay servers, with DHT-based full decentralization planned; the permissionless/untrusted part is the RL rollout/inference worker pool that TOPLOC guards."
	107	},
	108	{
	109	"claim": "Compute-over-data orchestration (Bacalhau) is production but operates a TRUSTED coordinated model, not trustless peer verification — it brings containerized/WASM compute to where data lives, with no fraud/validity proofs or cross-node result verification.",
	110	"confidence": "high",
	111	"vote": "merged claims 6 (3-0), 7 (2-1), 8 (3-0)",
	112	"sources": [
	113	"https://github.com/bacalhau-project/bacalhau"
	114	],
	115	"evidence": "Bacalhau README: 'an open-source distributed compute orchestration framework designed to bring compute to the data'; 'Instead of moving large datasets around networks, Bacalhau makes it easy to execute jobs close to the data's location.' Execution substrate: 'Support for multiple execution engines (Docker, WebAssembly)' — managed/sandboxed runtimes (the WASM path is the basis for content-addressable, deterministic compute-over-data per project reports), not direct host-binary invocation. Trust model: an 'Orchestrator-Compute Model' where 'a dedicated orchestrator coordinates job scheduling, while compute nodes run tasks'; the README addresses data access/permissions but contains NO fraud proofs, validity proofs, result-voting, or deterministic re-execution verification — i.e., trusted/coordinated, not trustless. NUANCE (one dissenting vote on the WASM substrate claim; and historical context): early 0.x 'Compute Over Data' Bacalhau documented determinism as a precondition for possible future verification, but the current production system does not deploy trustless cross-node verification."
	116	}
	117	],
	118	"caveats": "SOURCE QUALITY / BIAS: The zkVM overhead and lineage claims rest on a single vendor source (a16z, who built Jolt); the order-of-magnitude figures and architectural lineage are credible and corroborated, but relative-speed rankings against RISC Zero/SP1 were explicitly refuted (0-3) and should not be repeated. Prime Intellect / TOPLOC claims lean partly on a company blog, but are anchored by two independent arXiv papers (2505.07291, 2501.16007 / ICML 2025) and an Apache-2.0 open-weights artifact, so maturity is well-evidenced.\\n\\nTRUST-MODEL PRECISION: 'Trustless' is used loosely across the field. Properly: succinct proofs (zkVMs) = cryptographically trustless but proving-cost-heavy; interactive/optimistic fraud proofs = trust-minimized (1-of-N honest party) but require determinism + a synchronous dispute window + bonding capital + L1 liveness; TOPLOC = statistical tamper-detection, not soundness-proven; Bacalhau = trusted coordinator. Do not flatten these into one category.\\n\\nDETERMINISM IS THE COMMON GATE: Both Autobase's correctness and fraud-proof verifiability depend on fully deterministic execution; nondeterminism (e.g., GPU floating-point reordering) is precisely why AI compute needs LSH-style schemes (TOPLOC) instead of bit-exact re-execution.\\n\\nTIME-SENSITIVITY: zkVM performance and DePIN/decentralized-training projects move fast (2023-2026 wave); the below-6-OOM figure and INTELLECT-2 'first/at-scale' framing are mid-2025 snapshots likely to be superseded.\\n\\nCOVERAGE GAPS: Several dimensions requested in the brief were NOT covered by surviving verified claims — BOINC/volunteer grid computing, Nix/content-addressed reproducible builds, Golem/iExec/Nunet/Ray, DePIN markets (Akash, io.net, Render, Gensyn, Bittensor, Petals), and the academic MPC/FHE/GKR-PCP/BFT foundations. Treat their absence here as 'not verified in this round,' not 'unimportant.'\"",
	119	"openQuestions": [
	120	"What can an architect realistically build for 'chunked computation' on the Holepunch/Hypercore stack TODAY? The verified claims confirm Autobase natively solves incremental deterministic computation and content-addressed output caching, but trustless verification of OTHERS' compute and spatial/task distribution have no native primitive — would such a system need to bolt on a TrueBit-style dispute game or a TOPLOC-style statistical check, and is the determinism constraint compatible with heterogeneous worker hardware?",
	121	"What is the honest maturity and mechanics of the DePIN GPU/AI markets not verified this round (Akash, io.net, Render, Gensyn's verifiable ML training, Bittensor, Petals-style distributed inference, Prime Intellect's OpenDiLoCo/DiLoCo)? Which are production marketplaces vs research vs token-driven hype?",
	122	"How do the academic foundations (GKR/interactive proofs, PCP, secure multiparty computation, FHE, BFT-as-computation-agreement) actually map onto unstructured P2P settings, and which are practical vs purely theoretical as of 2026?",
	123	"Where exactly does content-addressed memoization (code-hash + input-hash keying) and reproducible builds (Nix) sit relative to verifiable computation — is content-addressing sufficient for trust when builds are deterministic, or only for caching/dedup without solving the trust problem?"
	124	],
	125	"refuted": [
	126	{
	127	"claim": "Jolt proves over 5x faster than RISC Zero and up to 2x faster than SP1 in the authors' benchmarks.",
	128	"vote": "0-3",
	129	"source": "https://a16zcrypto.com/posts/article/building-jolt/"
	130	},
	131	{
	132	"claim": "CRDTs automatically merge concurrent state changes from different devices so that all changes take effect, without requiring a central server to resolve them.",
	133	"vote": "1-2",
	134	"source": "https://www.inkandswitch.com/essay/local-first/"
	135	},
	136	{
	137	"claim": "The only class of concurrent change a CRDT cannot automatically resolve deterministically is when multiple users concurrently update the same property of the same object.",
	138	"vote": "0-3",
	139	"source": "https://www.inkandswitch.com/essay/local-first/"
	140	}
	141	],
	142	"sources": [
	143	{
	144	"url": "https://blog.alignedlayer.com/the-year-of-zkvm-real-time-proving-milestones-present-and-future/",
	145	"quality": "blog",
	146	"angle": "Verifiable computation / zkVM state of the art (technical+benchmarks)",
	147	"claimCount": 5
	148	},
	149	{
	150	"url": "https://a16zcrypto.com/posts/article/building-jolt/",
	151	"quality": "primary",
	152	"angle": "Verifiable computation / zkVM state of the art (technical+benchmarks)",
	153	"claimCount": 5
	154	},
	155	{
	156	"url": "https://medium.com/@gwrx2005/comparative-analysis-of-sp1-and-risc-zero-zero-knowledge-virtual-machines-4abf806daa70",
	157	"quality": "blog",
	158	"angle": "Verifiable computation / zkVM state of the art (technical+benchmarks)",
	159	"claimCount": 5
	160	},
	161	{
	162	"url": "https://truebit.io/truebit-announces-general-availability-of-verify-platform-the-verification-layer-for-tokenized-assets/",
	163	"quality": "blog",
	164	"angle": "Verifiable computation / zkVM state of the art (technical+benchmarks)",
	165	"claimCount": 5
	166	},
	167	{
	168	"url": "https://arxiv.org/abs/2505.07291",
	169	"quality": "primary",
	170	"angle": "Decentralized GPU/AI compute markets — real vs hype (practitioner/skeptical)",
	171	"claimCount": 5
	172	},
	173	{
	174	"url": "https://www.chakra.dev/research/the-third-epoch-of-ai-decentralizing-the-training-stack",
	175	"quality": "unreliable",
	176	"angle": "Decentralized GPU/AI compute markets — real vs hype (practitioner/skeptical)",
	177	"claimCount": 0
	178	},
	179	{
	180	"url": "https://medium.com/@0xjacobzhao/the-holy-grail-of-crypto-ai-frontier-exploration-on-decentralized-training-88584f6422a6",
	181	"quality": "blog",
	182	"angle": "Decentralized GPU/AI compute markets — real vs hype (practitioner/skeptical)",
	183	"claimCount": 5
	184	},
	185	{
	186	"url": "https://www.reflexivityresearch.com/free-reports/overview-of-decentralized-compute",
	187	"quality": "secondary",
	188	"angle": "Decentralized GPU/AI compute markets — real vs hype (practitioner/skeptical)",
	189	"claimCount": 5
	190	},
	191	{
	192	"url": "https://research.zengenlabs.com/p/can-decentralized-compute-solve-ais",
	193	"quality": "secondary",
	194	"angle": "Decentralized GPU/AI compute markets — real vs hype (practitioner/skeptical)",
	195	"claimCount": 5
	196	},
	197	{
	198	"url": "https://www.primeintellect.ai/blog/intellect-2-release",
	199	"quality": "primary",
	200	"angle": "Decentralized GPU/AI compute markets — real vs hype (practitioner/skeptical)",
	201	"claimCount": 4
	202	},
	203	{
	204	"url": "https://github.com/bacalhau-project/bacalhau",
	205	"quality": "primary",
	206	"angle": "P2P compute grids & content-addressed compute-over-data (systems mechanics)",
	207	"claimCount": 5
	208	},
	209	{
	210	"url": "https://blog.ipfs.tech/2022-12-13-bacalhau-beta-v1/",
	211	"quality": "blog",
	212	"angle": "P2P compute grids & content-addressed compute-over-data (systems mechanics)",
	213	"claimCount": 4
	214	},
	215	{
	216	"url": "https://medium.com/iex-ec/iexec-end-to-end-sgx-solution-fee1e63297b2",
	217	"quality": "blog",
	218	"angle": "P2P compute grids & content-addressed compute-over-data (systems mechanics)",
	219	"claimCount": 4
	220	},
	221	{
	222	"url": "https://www.tweag.io/blog/2021-12-02-nix-cas-4/",
	223	"quality": "secondary",
	224	"angle": "P2P compute grids & content-addressed compute-over-data (systems mechanics)",
	225	"claimCount": 4
	226	},
	227	{
	228	"url": "https://people.cs.uchicago.edu/~teutsch/papers/truebit.pdf",
	229	"quality": "primary",
	230	"angle": "Trustless verification trust models — TEEs, optimistic fraud proofs, refereed delegation",
	231	"claimCount": 5
	232	},
	233	{
	234	"url": "https://www.cs.tau.ac.il/~canetti/CRR11.pdf",
	235	"quality": "unreliable",
	236	"angle": "Trustless verification trust models — TEEs, optimistic fraud proofs, refereed delegation",
	237	"claimCount": 0
	238	},
	239	{
	240	"url": "https://docs.arbitrum.io/how-arbitrum-works/interactive-fraud-proofs",
	241	"quality": "primary",
	242	"angle": "Trustless verification trust models — TEEs, optimistic fraud proofs, refereed delegation",
	243	"claimCount": 5
	244	},
	245	{
	246	"url": "https://www.alchemy.com/overviews/validity-proof-vs-fraud-proof",
	247	"quality": "secondary",
	248	"angle": "Trustless verification trust models — TEEs, optimistic fraud proofs, refereed delegation",
	249	"claimCount": 5
	250	},
	251	{
	252	"url": "https://medium.com/ternoa/sgx-proofs-alternative-proof-system-for-ethereum-l2s-31682c0e5b93",
	253	"quality": "blog",
	254	"angle": "Trustless verification trust models — TEEs, optimistic fraud proofs, refereed delegation",
	255	"claimCount": 5
	256	},
	257	{
	258	"url": "https://www.inkandswitch.com/essay/local-first/",
	259	"quality": "primary",
	260	"angle": "Deterministic computation over append-only logs / local-first foundations (primary sources)",
	261	"claimCount": 5
	262	},
	263	{
	264	"url": "https://github.com/holepunchto/autobase/blob/main/DESIGN.md",
	265	"quality": "primary",
	266	"angle": "Deterministic computation over append-only logs / local-first foundations (primary sources)",
	267	"claimCount": 4
	268	},
	269	{
	270	"url": "https://github.com/holepunchto/autobase/blob/main/README.md",
	271	"quality": "primary",
	272	"angle": "Deterministic computation over append-only logs / local-first foundations (primary sources)",
	273	"claimCount": 5
	274	},
	275	{
	276	"url": "https://arxiv.org/pdf/2212.02618",
	277	"quality": "primary",
	278	"angle": "Deterministic computation over append-only logs / local-first foundations (primary sources)",
	279	"claimCount": 4
	280	},
	281	{
	282	"url": "https://martin.kleppmann.com/papers/local-first.pdf",
	283	"quality": "primary",
	284	"angle": "Deterministic computation over append-only logs / local-first foundations (primary sources)",
	285	"claimCount": 5
	286	},
	287	{
	288	"url": "https://arxiv.org/pdf/2305.00583",
	289	"quality": "primary",
	290	"angle": "Deterministic computation over append-only logs / local-first foundations (primary sources)",
	291	"claimCount": 5
	292	},
	293	{
	294	"url": "https://dl.acm.org/doi/10.1145/2699436",
	295	"quality": "primary",
	296	"angle": "Academic foundations: interactive proofs, MPC, FHE, BFT (key papers)",
	297	"claimCount": 4
	298	},
	299	{
	300	"url": "https://en.wikipedia.org/wiki/Verifiable_computing",
	301	"quality": "secondary",
	302	"angle": "Academic foundations: interactive proofs, MPC, FHE, BFT (key papers)",
	303	"claimCount": 5
	304	},
	305	{
	306	"url": "https://www.di.ens.fr/~nitulesc/files/Survey-SNARKs.pdf",
	307	"quality": "primary",
	308	"angle": "Academic foundations: interactive proofs, MPC, FHE, BFT (key papers)",
	309	"claimCount": 5
	310	},
	311	{
	312	"url": "https://eprint.iacr.org/2011/535.pdf",
	313	"quality": "primary",
	314	"angle": "Academic foundations: interactive proofs, MPC, FHE, BFT (key papers)",
	315	"claimCount": 5
	316	}
	317	],
	318	"stats": {
	319	"angles": 6,
	320	"sourcesFetched": 29,
	321	"claimsExtracted": 128,
	322	"claimsVerified": 25,
	323	"confirmed": 22,
	324	"killed": 3,
	325	"afterSynthesis": 7,
	326	"urlDupes": 0,
	327	"budgetDropped": 7,
	328	"agentCalls": 112
	329	}
	330	}
	331	}

research/raw/02-libriscv-executor.raw.json +365 -0

…	…	@@ -0,0 +1,365 @@
	1	{
	2	"summary": "Deep research harness — fan-out web searches, fetch sources, adversarially verify claims, synthesize a cited report.",
	3	"agentCount": 106,
	4	"logs": [
	5	"Q: Evaluate libriscv (fwsGonzo's C++ RISC-V emulator) and godot-sandbox (the Godot …",
	6	"Decomposed into 6 angles: primary/determinism, metering/sandbox, embeddability/godot-sandbox, performance/JIT, comparative/trustless-VM, reproducible-builds",
	7	"reproducible-builds: 6 results",
	8	"primary/determinism: 6 results",
	9	"performance/JIT: 6 results",
	10	"performance/JIT: 3 novel (3 filtered)",
	11	"embeddability/godot-sandbox: 6 results",
	12	"embeddability/godot-sandbox: 2 novel (4 filtered)",
	13	"comparative/trustless-VM: 6 results",
	14	"comparative/trustless-VM: 3 novel (3 filtered)",
	15	"metering/sandbox: 6 results",
	16	"metering/sandbox: 3 novel (3 filtered)",
	17	"Fetched 23 sources → 101 claims → verifying top 25",
	18	"\"libriscv supports configurable guest virtual-memor…\": 3-0 ✓",
	19	"\"libriscv handles no system calls by default — unha…\": 3-0 ✓",
	20	"\"libriscv supports a hard instruction-count budget …\": 3-0 ✓",
	21	"\"libriscv markets itself as a secure, low-attack-su…\": 3-0 ✓",
	22	"\"The guest invokes the host via the standard RISC-V…\": 3-0 ✓",
	23	"\"libriscv exposes a memory_max configuration parame…\": 3-0 ✓",
	24	"\"libriscv embeds into other C++ host projects via C…\": 3-0 ✓",
	25	"\"The host retains complete discretion over how (or …\": 0-3 ✗",
	26	"\"Performance is 2.5-10x faster than alternatives by…\": 1-2 ✗",
	27	"\"libriscv supports an optional encompassing arena (…\": 3-0 ✓",
	28	"\"libriscv performs instruction counting as its exec…\": 0-3 ✗",
	29	"\"Host-side syscall handlers must be explicitly inst…\": 3-0 ✓",
	30	"\"libriscv syscalls are deny-by-default: any system …\": 3-0 ✓",
	31	"[v0:godot-sandbox supports execution timeout] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	32	"[v2:godot-sandbox is a Godot engine addon ti] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	33	"[v0:Deterministic metering must use an expli] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	34	"\"The host is explicitly under no obligation to perf…\": 3-0 ✓",
	35	"[v2:godot-sandbox supports execution timeout] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	36	"[v1:godot-sandbox supports execution timeout] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	37	"\"godot-sandbox supports execution timeout via instr…\": 0-0 (3 abstain) ✗",
	38	"[v0:WebAssembly is only \"mostly\" determinist] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	39	"[v1:WebAssembly is only \"mostly\" determinist] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	40	"[v1:Deterministic metering must use an expli] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	41	"[v0:Floating-point NaN nondeterminism in Was] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	42	"[v1:Floating-point NaN nondeterminism in Was] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	43	"[v1:godot-sandbox is a Godot engine addon ti] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	44	"[v2:WebAssembly is only \"mostly\" determinist] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	45	"\"WebAssembly is only \"mostly\" deterministic; achiev…\": 0-0 (3 abstain) ✗",
	46	"[v0:godot-sandbox is a Godot engine addon ti] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	47	"\"godot-sandbox is a Godot engine addon tightly coup…\": 0-0 (3 abstain) ✗",
	48	"[v0:Wasmtime recommends deterministic fuel-b] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	49	"[v1:Wasmtime recommends deterministic fuel-b] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	50	"[v2:Floating-point NaN nondeterminism in Was] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	51	"\"Floating-point NaN nondeterminism in Wasmtime can …\": 0-0 (3 abstain) ✗",
	52	"[v2:Wasmtime recommends deterministic fuel-b] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	53	"\"Wasmtime recommends deterministic fuel-based inter…\": 0-0 (3 abstain) ✗",
	54	"[v0:Wasmtime's fuel metering provides precis] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	55	"[v2:Wasmtime's fuel metering provides precis] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	56	"[v1:Wasmtime's fuel metering provides precis] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	57	"\"Wasmtime's fuel metering provides precise instruct…\": 0-0 (3 abstain) ✗",
	58	"[v2:Wasmtime's precise fuel metering imposes] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	59	"[v0:Wasmtime's precise fuel metering imposes] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	60	"[v2:libriscv supports instruction counting w] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	61	"[v1:Wasmtime's precise fuel metering imposes] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	62	"\"Wasmtime's precise fuel metering imposes a signifi…\": 0-0 (3 abstain) ✗",
	63	"[v0:libriscv supports instruction counting w] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	64	"[v1:libriscv supports instruction counting w] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	65	"\"libriscv supports instruction counting with a conf…\": 0-0 (3 abstain) ✗",
	66	"\"Deterministic metering must use an explicit per-ba…\": 1-0 (2 abstain) ✗",
	67	"\"Instruction-counting deterministic metering (gas d…\": 3-0 ✓",
	68	"\"The Arm floating-point instructions are determinis…\": 3-0 ✓",
	69	"Verify done: 25 claims → 13 confirmed, 12 killed"
	70	],
	71	"result": {
	72	"question": "Evaluate libriscv (fwsGonzo's C++ RISC-V emulator) and godot-sandbox (the Godot GDExtension that wraps libriscv) as a sandboxed execution target for a peer-to-peer distributed computation system. The system needs to run untrusted, content-addressed compute tasks where the SAME code+input must produce the SAME output bytes on every peer, so results can be memoized by hash and verified by independent re-execution.\n\nAssess libriscv and godot-sandbox specifically on these decision criteria, with concrete mechanics, current (2025-2026) maturity, and primary sources:\n\n1. DETERMINISM: Is libriscv execution bit-exact deterministic across different host hardware/OS (x86-64, arm64)? How is floating-point handled (soft-float vs host FPU, IEEE-754 reproducibility)? Any sources of nondeterminism (uninitialized memory, syscall results, host time)? This is the single most important criterion — without cross-hardware determinism it cannot memoize or verify.\n\n2. RESOURCE METERING / SANDBOXING: Does libriscv support instruction counting / \"fuel\" / execution-budget limits to bound runaway computation? Memory limits (sandboxed guest address space)? Wall-clock timeout? How strong is the isolation (can guest code escape, make syscalls, touch the host)? What is the host<->guest call/syscall ABI and how is it gated?\n\n3. EMBEDDABILITY: libriscv is header-only C++. Can it be embedded as a native addon in a non-Godot host runtime (specifically Bare / Node.js via N-API, or a C ABI)? godot-sandbox is Godot-specific — how coupled is it to the engine, and could libriscv be used standalone? What languages compile to its RISC-V guest (C/C++/Rust/Zig via standard riscv toolchains)?\n\n4. PERFORMANCE: libriscv's interpreter vs binary-translation/JIT modes — throughput vs native, startup/load latency, and the determinism implications of JIT (does binary translation preserve bit-exactness?). How does it compare to WASM runtimes (wasmtime/wasmer) on speed and on fuel-metering overhead.\n\n5. CONTENT-ADDRESSABILITY & REPRODUCIBLE BUILDS: Can a RISC-V ELF binary be built reproducibly (same source -> same bytes -> stable codeHash)? How does this compare to reproducible WASM modules.\n\n6. libriscv vs WASM (wasmtime/wasmer/WAMR) HEAD-TO-HEAD as the executor for trustless distributed compute: determinism guarantees, metering maturity, ecosystem/toolchain, embedding cost, security track record. Which is the better default and in what scenarios is the other preferred? Surface any other contenders used in blockchain/trustless-compute settings (e.g. RISC Zero's RISC-V zkVM, Cartesi's RISC-V Linux, PolkaVM/RISC-V, Fuel/FuelVM, eBPF) and what their choice of ISA says about RISC-V-for-determinism.\n\nPrioritize primary sources (libriscv and godot-sandbox GitHub repos/docs/wiki, fwsGonzo's writeups, benchmarks), honest maturity assessment, and the determinism + metering criteria above all else, since those are what make trustless memoized re-execution possible.",
	73	"summary": "libriscv is a viable sandboxed execution target for trustless distributed compute, with strong, primary-source-confirmed support for the two most critical criteria: fuel-style instruction-count metering (set_max_instructions / simulate with an instruction-limit timeout) and a deny-by-default sandbox (no syscalls unless the host explicitly installs handlers; -ENOSYS otherwise; narrow ECALL ABI; configurable memory_max and address-space arena caps). However, the libriscv project itself makes NO explicit guarantee of bit-exact cross-hardware determinism or IEEE-754 floating-point reproducibility, and has no formal audit — so determinism is plausible-but-unproven from libriscv's own documentation. The strongest determinism evidence comes from an adjacent academic system (DeCl, Stanford, SIB'24/OSDI'25), which establishes that a restricted deterministic instruction subset (including FP) is bit-exact across ARM64 microarchitectures via ISA analysis and fuzzing, and that deterministic native instruction-count metering is far cheaper than WASM metering (~16-21% overhead vs Wasmtime/Cranelift fuel at ~90% and a WASM interpreter at ~1920%). Net: RISC-V/native sandboxing offers a metering-cost advantage and a clean opt-in ABI, but to use libriscv for memoizable re-execution you must yourself restrict the guest to a deterministic instruction subset, pin the FP environment, control the binary-translation/JIT path, and validate cross-hardware bit-exactness — none of which libriscv guarantees out of the box.",
	74	"findings": [
	75	{
	76	"claim": "libriscv provides production-grade fuel-style resource metering: a hard instruction-count budget (set_max_instructions / simulate(limit)) that deterministically halts runaway computation, with detection (instruction_limit_reached) and resumability (reset_instruction_counter).",
	77	"confidence": "high",
	78	"sources": [
	79	"https://github.com/libriscv/libriscv",
	80	"https://github.com/libriscv/libriscv/blob/master/docs/INTEGRATION.md"
	81	],
	82	"evidence": "README shows machine.set_max_instructions(50'000'000ull) and simulate(5'000'000'000ull) running until exit syscall or the instruction counter reaches the limit; instruction_limit_reached() detects timeout, reset_instruction_counter() resumes. The budget is resumable and can be disabled in an 'inaccurate' fast mode (simulate_inaccurate) but is enforced by default. This directly satisfies the 'fuel'/execution-budget criterion for bounding untrusted compute.",
	83	"vote": "3-0"
	84	},
	85	{
	86	"claim": "libriscv provides hard sandbox memory bounds: a memory_max config caps how much memory a guest can consume (lazy physical allocation underneath), and an optional encompassing arena (RISCV_ENCOMPASSING_ARENA / _BITS) bounds the guest's entire address space to a fixed bit-width (e.g. 32 bits = 4GB, 30 bits = 1GB).",
	87	"confidence": "high",
	88	"sources": [
	89	"https://github.com/libriscv/libriscv",
	90	"https://github.com/libriscv/libriscv/blob/master/docs/INTEGRATION.md"
	91	],
	92	"evidence": "README: 'You can limit the amount of (virtual) memory the machine can use' via MachineOptions{.memory_max=...}. INTEGRATION.md: memory_max sets the upper limit a guest can consume and the guest 'may never access memory outside of this area'; physical pages allocated lazily on write. The arena bounds all memory operations to an N-bit space. IMPORTANT caveat from the docs: the arena is currently fully readable/writable and 'should not be used when sandboxing is necessary' (full isolation needs userfaultfd, not yet wired up) — so for untrusted code use memory_max-based isolation, not the arena, for the security boundary.",
	93	"vote": "3-0"
	94	},
	95	{
	96	"claim": "libriscv is a deny-by-default sandbox with a narrow, fully host-controlled syscall ABI: no syscalls are serviced unless the host explicitly installs handlers (install_syscall_handler), unhandled calls return -ENOSYS, and the host is under no obligation to honor any request.",
	97	"confidence": "high",
	98	"sources": [
	99	"https://libriscv.no/docs/concepts/syscalls/",
	100	"https://github.com/libriscv/libriscv/blob/master/docs/SYSCALLS.md"
	101	],
	102	"evidence": "SYSCALLS.md and libriscv.no both state 'Unhandled system calls return -ENOSYS by default' and 'To implement missing system calls, you have to set a handler for it' via machine.install_syscall_handler(number, handler). Even exit(93)/exit_group(94) must be installed. The guest invokes the host purely via the RISC-V ECALL instruction (args in A0-A6, syscall number in A7, return in A0). 'The host system is under no obligation to do anything.' This gives the embedder a small, fully gated host<->guest boundary — exactly what is needed to forbid host time, network, randomness, and any other nondeterminism source in untrusted compute. Convenience batch-installers (e.g. setup_linux_syscalls) exist but require explicit host opt-in.",
	103	"vote": "3-0"
	104	},
	105	{
	106	"claim": "libriscv itself does NOT guarantee bit-exact cross-platform determinism or IEEE-754 floating-point reproducibility, and has no formal third-party audit; it only markets a low-attack-surface (~20k LOC) speculation-safe sandbox.",
	107	"confidence": "high",
	108	"sources": [
	109	"https://github.com/libriscv/libriscv",
	110	"https://github.com/libriscv/libriscv/blob/master/SECURITY.md"
	111	],
	112	"evidence": "README Goals state 'Low attack surface, only 20k LOC' and 'Secure speculation-safe sandbox' but make NO claim of bit-exact cross-host (x86-64/arm64) execution or FP IEEE-754 reproducibility. SECURITY.md likewise mentions no cross-platform determinism, no FP reproducibility, and no formal audit. CRITICAL for the use case: this is the single most important criterion for memoized re-execution, and libriscv's own docs do not promise it. Absence of a guarantee is not evidence of non-determinism, but it means determinism must be established/enforced by the embedder (restrict instruction subset, pin FP rounding/NaN behavior, validate cross-hardware) rather than assumed.",
	113	"vote": "3-0"
	114	},
	115	{
	116	"claim": "A restricted deterministic instruction subset — including floating-point — is empirically bit-exact across different ARM64 hardware, supporting the feasibility of trustless re-execution on RISC-style native sandboxes.",
	117	"confidence": "high",
	118	"sources": [
	119	"https://www.scs.stanford.edu/~zyedidia/docs/papers/decl_sib24.pdf"
	120	],
	121	"evidence": "DeCl (Stanford, SIB'24, also OSDI'25): 'According to our analysis of the Arm ISA, the floating point instructions are deterministic,' corroborated by fuzzing on Apple silicon, Arm Neoverse, Ampere Altra, and QEMU, generating multi-megabyte branchless snapshots and confirming end-state (memory + registers) is consistent across microarchitectures — when restricted to a deterministic subset (excluding CONSTRAINED UNPREDICTABLE, certain atomics, UNDEFINED encodings). NOTE this is about ARM64 native, not libriscv specifically, and is engineering-grade corroboration via fuzzing, not a formal proof. It validates the general principle (deterministic ISA subset + pinned FP env => reproducible end-state) that libriscv's RISC-V interpreter would need to inherit, but does not itself certify libriscv. FP determinism also depends on per-process FPCR config (flush-to-zero, default-NaN, rounding) being initialized identically on all peers.",
	122	"vote": "3-0"
	123	},
	124	{
	125	"claim": "Deterministic native instruction-count metering is dramatically cheaper than WASM metering — a core argument for native/RISC-style sandboxes over WASM in metered trustless compute.",
	126	"confidence": "high",
	127	"sources": [
	128	"https://www.scs.stanford.edu/~zyedidia/docs/papers/decl_sib24.pdf"
	129	],
	130	"evidence": "DeCl (SPEC 2017, Apple M2, ARM64) reports geomean overhead vs native: DeCl timer-based 16.3%, branch-based 21.2% (gas debited at basic-block ends via a reserved meter register), versus iwasm WASM interpreter ~1920% and Wasmtime/Cranelift JIT with fuel metering ~90.0%. Versions: iwasm v1.3.2, Wasmtime v24.0.0. This quantifies a large metering-overhead gap favoring native sandboxing. Caveat: DeCl is a different system (LFI-based native sandbox), not libriscv; libriscv's own interpreter overhead vs native is higher than DeCl's instrumented-native approach, and libriscv's competitive performance story relies on binary translation (whose determinism interaction must be separately verified). The number to take away is the metering-cost comparison, not a direct libriscv-vs-Wasmtime throughput figure.",
	131	"vote": "3-0"
	132	},
	133	{
	134	"claim": "libriscv is cleanly embeddable in non-Godot native hosts: header-only C++ via CMake FetchContent/add_subdirectory, plus a C API (libriscv_capi) for integration from other languages.",
	135	"confidence": "high",
	136	"sources": [
	137	"https://fwsgonzo.medium.com/libriscv-version-1-0-594985712e2e"
	138	],
	139	"evidence": "fwsGonzo's v1.0 writeup: 'Embedding a project with CMake should be really straight-forward. I support both FetchContent and add_subdirectory()' and 'There is now a C API ... CMake to build a new library called libriscv_capi ... presents a public C header. The C API will become more fleshed out over time, but for now it does the job.' For the target architecture (a Bare/Node.js host), this means libriscv could be wrapped as a native addon over its C ABI or C++ API (e.g. via bare-addon / N-API / require-addon). godot-sandbox, by contrast, is the Godot-engine wrapper and is not the right integration path for a non-Godot runtime — libriscv standalone is. Guests can be compiled from C/C++/Rust/Zig via standard riscv toolchains. Maturity caveat: the author explicitly hedges the C API is not yet fully fleshed out.",
	140	"vote": "3-0"
	141	}
	142	],
	143	"caveats": "DETERMINISM IS THE KEY UNPROVEN POINT. libriscv's own documentation does NOT guarantee bit-exact cross-hardware execution or FP IEEE-754 reproducibility (claim 3 above). The strong determinism evidence (claim 4) and the metering-overhead numbers (claim 5) come from DeCl, a SEPARATE academic system (native ARM64 sandbox via LFI), not from libriscv itself — they validate the general RISC/native-deterministic-subset principle but do not certify libriscv. Treat libriscv determinism as 'plausible and likely achievable with discipline' rather than 'guaranteed.'\n\nSEVERAL HEAD-TO-HEAD CLAIMS WERE REFUTED OR UNVERIFIED, leaving real gaps: (a) the claim that libriscv binary translation/JIT bypasses instruction metering (translate_ignore_instruction_limit) was REFUTED 0-3, but so was the counter-claim implicitly — the precise metering behavior under binary translation remains unconfirmed and is critical because JIT could threaten bit-exactness; (b) godot-sandbox's coupling and its own timeout/memory-safety claims went unvoted (0-0), so its standalone-extractability is uncorroborated here; (c) all the Wasmtime determinism/fuel claims (NaN canonicalization, deterministic fuel vs epoch, fuel overhead) were 0-0 unvoted — the WASM side of the head-to-head is weakly sourced in this dataset; (d) the performance claim '2.5-10x faster, 5-50x with binary translation, cross-platform translation without JIT' was REFUTED 1-2, so libriscv throughput claims are unreliable as stated.\n\nSOURCE QUALITY: libriscv claims rest on the official repo, libriscv.no docs, and fwsGonzo's Medium writeup (primary, current to master branch). The determinism + metering-overhead evidence rests on a single peer-reviewed paper (DeCl). No independent benchmark of libriscv-vs-Wasmtime for this specific trustless-compute workload was verified. No formal security audit of libriscv exists.\n\nTIME-SENSITIVITY: API surface (set_max_instructions, memory_max, install_syscall_handler, RISCV_ENCOMPASSING_ARENA) is current master (2025-2026). The arena's lack of full isolation (needs userfaultfd) and the C API's incompleteness are explicitly evolving — re-verify before committing.",
	144	"openQuestions": [
	145	"Does libriscv's binary-translation/JIT mode preserve bit-exact determinism AND continue to enforce the instruction-count budget? The 'translate_ignore_instruction_limit' option and the refuted-but-unresolved metering-under-translation claim mean the interpreter is the only currently-trustworthy deterministic+metered path — but the interpreter is the slow path, undermining the performance argument. Empirical cross-hardware (x86-64 vs arm64) bit-exactness testing of libriscv specifically is required.",
	146	"How does libriscv handle floating-point internally — does its RISC-V FP execution use the host FPU (inheriting host FPCR/rounding/NaN state, which DeCl shows must be pinned identically on all peers) or a soft-float path? Soft-float would give the strongest cross-hardware reproducibility guarantee; this is undocumented in the verified sources and is decisive for the memoization use case.",
	147	"Can a RISC-V ELF guest be built reproducibly (same source -> same bytes -> stable codeHash) with standard riscv toolchains, and how does that compare to reproducible WASM modules? Criterion 5 (content-addressability / reproducible builds) was not covered by any surviving verified claim and needs direct investigation.",
	148	"For the actual target host (Bare/Node.js), what is the real embedding cost and stability of wrapping libriscv's C API/C++ as a native addon, given fwsGonzo's own hedge that the C API 'will become more fleshed out over time'? And how does this compare to embedding a mature WASM runtime (wasmtime/wasmer/WAMR) which already ships first-class language bindings — i.e., is libriscv's metering/determinism edge worth the heavier embedding and self-imposed-determinism burden versus an off-the-shelf metered WASM runtime?"
	149	],
	150	"refuted": [
	151	{
	152	"claim": "The host retains complete discretion over how (or whether) to service any syscall, which is the mechanism enforcing the sandbox boundary against untrusted guest code.",
	153	"vote": "0-3",
	154	"source": "https://libriscv.no/docs/concepts/syscalls/"
	155	},
	156	{
	157	"claim": "Performance is 2.5-10x faster than alternatives by default and 5-50x with binary translation, with optional JIT builds; binary translation works across all platforms without requiring JIT.",
	158	"vote": "1-2",
	159	"source": "https://github.com/libriscv/godot-sandbox/blob/main/README.md"
	160	},
	161	{
	162	"claim": "libriscv performs instruction counting as its execution-budget mechanism, but this counting is NOT performed during binary translation (the translate_ignore_instruction_limit option), meaning the fuel/instruction-limit metering is bypassed in translated/JIT mode and execution must be stopped by another external method.",
	163	"vote": "0-3",
	164	"source": "https://github.com/libriscv/libriscv/blob/master/docs/INTEGRATION.md"
	165	},
	166	{
	167	"claim": "Deterministic metering must use an explicit per-basic-block instruction count (a 'gas' counter in a reserved register decremented and checked at branches), NOT timer interrupts or the hardware PMU instruction counter, because timers/PMU preempt at non-deterministic instruction boundaries and would cause replicas to diverge on side-effecting calls — establishing the design requirement for any memoizable re-execution metering scheme.",
	168	"vote": "1-0",
	169	"source": "https://www.scs.stanford.edu/~zyedidia/docs/papers/decl_sib24.pdf"
	170	},
	171	{
	172	"claim": "godot-sandbox is a Godot engine addon tightly coupled to the engine: it exposes a Sandbox node, loads ELF binaries, and implements its API as Godot syscalls in src/sandbox_syscalls.cpp, with libriscv embedded underneath.",
	173	"vote": "0-0",
	174	"source": "https://github.com/libriscv/godot-sandbox/blob/main/CLAUDE.md"
	175	},
	176	{
	177	"claim": "godot-sandbox supports execution timeout via instruction counting (a metering mechanism to bound runaway computation), and is memory safe.",
	178	"vote": "0-0",
	179	"source": "https://github.com/libriscv/godot-sandbox/blob/main/CLAUDE.md"
	180	},
	181	{
	182	"claim": "WebAssembly is only \"mostly\" deterministic; achieving determinism requires explicit configuration because several language features are inherently nondeterministic by spec.",
	183	"vote": "0-0",
	184	"source": "https://docs.wasmtime.dev/examples-deterministic-wasm-execution.html"
	185	},
	186	{
	187	"claim": "Floating-point NaN nondeterminism in Wasmtime can be eliminated via Cranelift NaN canonicalization, but at a performance cost.",
	188	"vote": "0-0",
	189	"source": "https://docs.wasmtime.dev/examples-deterministic-wasm-execution.html"
	190	},
	191	{
	192	"claim": "Wasmtime recommends deterministic fuel-based interruption (rather than epoch-based) for bounding execution in a reproducible way, providing a metering mechanism for trustless compute.",
	193	"vote": "0-0",
	194	"source": "https://docs.wasmtime.dev/examples-deterministic-wasm-execution.html"
	195	},
	196	{
	197	"claim": "Wasmtime's fuel metering provides precise instruction-execution control at basic-block granularity, implemented via compare-and-conditional-jump checks.",
	198	"vote": "0-0",
	199	"source": "https://github.com/bytecodealliance/wasmtime/issues/4109"
	200	},
	201	{
	202	"claim": "Wasmtime's precise fuel metering imposes a significant runtime performance penalty.",
	203	"vote": "0-0",
	204	"source": "https://github.com/bytecodealliance/wasmtime/issues/4109"
	205	},
	206	{
	207	"claim": "libriscv supports instruction counting with a configurable hard execution budget (e.g. set_max_instructions(50'000'000)), allowing computation to be paused or aborted when an instruction-count limit is reached — providing deterministic 'fuel'-style metering for bounding runaway compute.",
	208	"vote": "0-0",
	209	"source": "https://github.com/fwsGonzo/libriscv"
	210	}
	211	],
	212	"sources": [
	213	{
	214	"url": "https://github.com/libriscv/libriscv",
	215	"quality": "primary",
	216	"angle": "primary/determinism",
	217	"claimCount": 5
	218	},
	219	{
	220	"url": "https://libriscv.no/docs/concepts/syscalls/",
	221	"quality": "primary",
	222	"angle": "primary/determinism",
	223	"claimCount": 4
	224	},
	225	{
	226	"url": "https://github.com/libriscv/godot-sandbox/blob/main/README.md",
	227	"quality": "primary",
	228	"angle": "primary/determinism",
	229	"claimCount": 4
	230	},
	231	{
	232	"url": "https://fwsgonzo.medium.com/libriscv-risc-v-binary-translation-part-2-deb3589375ad",
	233	"quality": "blog",
	234	"angle": "primary/determinism",
	235	"claimCount": 5
	236	},
	237	{
	238	"url": "https://fwsgonzo.medium.com/libriscv-version-1-0-594985712e2e",
	239	"quality": "primary",
	240	"angle": "primary/determinism",
	241	"claimCount": 4
	242	},
	243	{
	244	"url": "https://www.chciken.com/simulation/2023/11/12/fast-floating-point-simulation.html",
	245	"quality": "blog",
	246	"angle": "primary/determinism",
	247	"claimCount": 5
	248	},
	249	{
	250	"url": "https://github.com/libriscv/libriscv/blob/master/docs/INTEGRATION.md",
	251	"quality": "primary",
	252	"angle": "metering/sandbox",
	253	"claimCount": 5
	254	},
	255	{
	256	"url": "https://github.com/libriscv/libriscv/blob/master/docs/SYSCALLS.md",
	257	"quality": "primary",
	258	"angle": "metering/sandbox",
	259	"claimCount": 5
	260	},
	261	{
	262	"url": "https://www.scs.stanford.edu/~zyedidia/docs/papers/decl_sib24.pdf",
	263	"quality": "primary",
	264	"angle": "metering/sandbox",
	265	"claimCount": 5
	266	},
	267	{
	268	"url": "https://github.com/libriscv/godot-sandbox/blob/main/CLAUDE.md",
	269	"quality": "primary",
	270	"angle": "embeddability/godot-sandbox",
	271	"claimCount": 5
	272	},
	273	{
	274	"url": "https://libriscv.no/docs/host_langs/godot_integration/godot_docs/cmake/",
	275	"quality": "primary",
	276	"angle": "embeddability/godot-sandbox",
	277	"claimCount": 3
	278	},
	279	{
	280	"url": "https://libriscv.no/docs/intro/",
	281	"quality": "primary",
	282	"angle": "performance/JIT",
	283	"claimCount": 4
	284	},
	285	{
	286	"url": "https://docs.wasmtime.dev/examples-deterministic-wasm-execution.html",
	287	"quality": "primary",
	288	"angle": "performance/JIT",
	289	"claimCount": 5
	290	},
	291	{
	292	"url": "https://github.com/bytecodealliance/wasmtime/issues/4109",
	293	"quality": "primary",
	294	"angle": "performance/JIT",
	295	"claimCount": 5
	296	},
	297	{
	298	"url": "https://github.com/fwsGonzo/libriscv",
	299	"quality": "primary",
	300	"angle": "comparative/trustless-VM",
	301	"claimCount": 5
	302	},
	303	{
	304	"url": "https://forum.polkadot.network/t/announcing-polkavm-a-new-risc-v-based-vm-for-smart-contracts-and-possibly-more/3811",
	305	"quality": "primary",
	306	"angle": "comparative/trustless-VM",
	307	"claimCount": 5
	308	},
	309	{
	310	"url": "https://github.com/cartesi/machine-emulator",
	311	"quality": "primary",
	312	"angle": "comparative/trustless-VM",
	313	"claimCount": 5
	314	},
	315	{
	316	"url": "https://github.com/risc0/risc0/issues/1481",
	317	"quality": "primary",
	318	"angle": "reproducible-builds",
	319	"claimCount": 3
	320	},
	321	{
	322	"url": "https://docs.rs/risc0-build/latest/risc0_build/",
	323	"quality": "primary",
	324	"angle": "reproducible-builds",
	325	"claimCount": 3
	326	},
	327	{
	328	"url": "https://github.com/project-oak/oak/issues/865",
	329	"quality": "primary",
	330	"angle": "reproducible-builds",
	331	"claimCount": 3
	332	},
	333	{
	334	"url": "https://github.com/rust-lang/rust/issues/117597",
	335	"quality": "forum",
	336	"angle": "reproducible-builds",
	337	"claimCount": 3
	338	},
	339	{
	340	"url": "https://docs.internetcomputer.org/building-apps/best-practices/reproducible-builds",
	341	"quality": "primary",
	342	"angle": "reproducible-builds",
	343	"claimCount": 5
	344	},
	345	{
	346	"url": "https://reproducible-builds.org/docs/deterministic-build-systems/",
	347	"quality": "primary",
	348	"angle": "reproducible-builds",
	349	"claimCount": 5
	350	}
	351	],
	352	"stats": {
	353	"angles": 6,
	354	"sourcesFetched": 23,
	355	"claimsExtracted": 101,
	356	"claimsVerified": 25,
	357	"confirmed": 13,
	358	"killed": 12,
	359	"afterSynthesis": 7,
	360	"urlDupes": 9,
	361	"budgetDropped": 4,
	362	"agentCalls": 106
	363	}
	364	}
	365	}

research/raw/03-distributed-ai.raw.json +397 -0

…	…	@@ -0,0 +1,397 @@
	1	{
	2	"summary": "Deep research harness — fan-out web searches, fetch sources, adversarially verify claims, synthesize a cited report.",
	3	"agentCount": 109,
	4	"logs": [
	5	"Q: Research the state of the art (2024-2026) in distributed AI across peers/devices…",
	6	"Decomposed into 6 angles: Primary — exo + heterogeneous consumer-device inference, Distributed inference systems + parallelism mechanics, Decentralized training — DiLoCo / Prime Intellect / Nous, Technical — communication bottleneck and bandwidth math, Skeptical / maturity verdict — production vs research vs hype, Mapping to P2P — Hyperswarm/udx feasibility for activations",
	7	"Primary — exo + heterogeneous consumer-device inference: 6 results",
	8	"Technical — communication bottleneck and bandwidth math: 6 results",
	9	"Mapping to P2P — Hyperswarm/udx feasibility for activations: 6 results",
	10	"Mapping to P2P — Hyperswarm/udx feasibility for activations: 3 novel (3 filtered)",
	11	"Decentralized training — DiLoCo / Prime Intellect / Nous: 6 results",
	12	"Decentralized training — DiLoCo / Prime Intellect / Nous: 3 novel (3 filtered)",
	13	"Skeptical / maturity verdict — production vs research vs hype: 6 results",
	14	"Skeptical / maturity verdict — production vs research vs hype: 4 novel (2 filtered)",
	15	"Distributed inference systems + parallelism mechanics: 6 results",
	16	"Distributed inference systems + parallelism mechanics: 5 novel (1 filtered)",
	17	"Fetched 27 sources → 131 claims → verifying top 25",
	18	"[v0:exo supports both pipeline and tensor pa] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	19	"[v1:exo supports both pipeline and tensor pa] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	20	"[v2:For exo's pipeline-parallel inference th] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	21	"[v2:exo supports both pipeline and tensor pa] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	22	"\"exo supports both pipeline and tensor parallelism …\": 0-0 (3 abstain) ✗",
	23	"[v0:Adding devices increases aggregate (mult] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	24	"[v1:Adding devices increases aggregate (mult] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	25	"[v2:Adding devices increases aggregate (mult] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	26	"\"Adding devices increases aggregate (multi-request)…\": 0-0 (3 abstain) ✗",
	27	"[v1:exo targets Mac clusters with hardware i] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	28	"[v2:exo runs a model across multiple devices] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	29	"[v1:vLLM recommends tensor parallelism only ] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	30	"[v0:vLLM recommends tensor parallelism only ] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	31	"[v1:exo runs a model across multiple devices] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	32	"[v0:exo runs a model across multiple devices] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	33	"\"exo runs a model across multiple devices using pip…\": 0-0 (3 abstain) ✗",
	34	"[v2:vLLM recommends tensor parallelism only ] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	35	"\"vLLM recommends tensor parallelism only when a mod…\": 0-0 (3 abstain) ✗",
	36	"[v0:exo automatically discovers devices on t] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	37	"[v2:exo targets Mac clusters with hardware i] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	38	"[v0:Tensor parallelism requires fast interno] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	39	"[v1:exo automatically discovers devices on t] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	40	"[v2:Tensor parallelism requires fast interno] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	41	"[v0:exo targets Mac clusters with hardware i] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	42	"\"exo targets Mac clusters with hardware interconnec…\": 0-0 (3 abstain) ✗",
	43	"[v2:exo automatically discovers devices on t] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	44	"\"exo automatically discovers devices on the network…\": 0-0 (3 abstain) ✗",
	45	"[v1:Even within a single node, when GPUs lac] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	46	"[v1:Tensor parallelism requires fast interno] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	47	"\"Tensor parallelism requires fast internode communi…\": 0-0 (3 abstain) ✗",
	48	"[v2:Even within a single node, when GPUs lac] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	49	"[v0:distributed-llama accelerates LLM infere] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	50	"[v1:distributed-llama accelerates LLM infere] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	51	"[v0:The architecture is split into a root no] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	52	"[v2:distributed-llama accelerates LLM infere] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	53	"\"distributed-llama accelerates LLM inference by con…\": 0-0 (3 abstain) ✗",
	54	"[v0:Even within a single node, when GPUs lac] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	55	"\"Even within a single node, when GPUs lack NVLink i…\": 0-0 (3 abstain) ✗",
	56	"[v0:For exo's pipeline-parallel inference th] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	57	"[v1:The architecture is split into a root no] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	58	"[v1:It connects multiple devices over a stan] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	59	"[v0:It connects multiple devices over a stan] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	60	"[v2:The architecture is split into a root no] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	61	"\"The architecture is split into a root node (loads …\": 0-0 (3 abstain) ✗",
	62	"[v2:It connects multiple devices over a stan] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	63	"\"It connects multiple devices over a standard Ether…\": 0-0 (3 abstain) ✗",
	64	"[v1:For exo's pipeline-parallel inference th] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	65	"\"For exo's pipeline-parallel inference the bottlene…\": 0-0 (3 abstain) ✗",
	66	"[v1:Petals runs LLMs BitTorrent-style: each ] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	67	"[v0:Petals runs LLMs BitTorrent-style: each ] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	68	"[v2:Single-batch inference on Petals runs at] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	69	"[v1:Single-batch inference on Petals runs at] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	70	"[v0:Single-batch inference on Petals runs at] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	71	"\"Single-batch inference on Petals runs at up to 6 t…\": 0-0 (3 abstain) ✗",
	72	"[v0:DiLoCo enables training language models ] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	73	"[v2:DiLoCo enables training language models ] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	74	"[v1:DiLoCo is a variant of federated averagi] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	75	"[v1:DiLoCo enables training language models ] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	76	"\"DiLoCo enables training language models across isl…\": 0-0 (3 abstain) ✗",
	77	"[v2:Petals runs LLMs BitTorrent-style: each ] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	78	"\"Petals runs LLMs BitTorrent-style: each user loads…\": 0-0 (3 abstain) ✗",
	79	"[v2:DiLoCo is a variant of federated averagi] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	80	"[v0:DiLoCo is a variant of federated averagi] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	81	"\"DiLoCo is a variant of federated averaging that us…\": 0-0 (3 abstain) ✗",
	82	"[v1:Prime Intellect trained INTELLECT-2, a 3] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	83	"[v2:Prime Intellect trained INTELLECT-2, a 3] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	84	"[v0:Decentralized RL training over untrusted] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	85	"[v1:Decentralized RL training over untrusted] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	86	"[v2:Decentralized RL training over untrusted] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	87	"\"Decentralized RL training over untrusted permissio…\": 0-0 (3 abstain) ✗",
	88	"[v0:Prime Intellect trained INTELLECT-2, a 3] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	89	"\"Prime Intellect trained INTELLECT-2, a 32-billion-…\": 0-0 (3 abstain) ✗",
	90	"[v1:INTELLECT-2 and its full code and data w] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	91	"[v0:INTELLECT-2 and its full code and data w] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	92	"[v2:INTELLECT-2 and its full code and data w] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	93	"\"INTELLECT-2 and its full code and data were open-s…\": 0-0 (3 abstain) ✗",
	94	"[v0:Psyche's DisTrO optimizer reduces inter-] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	95	"[v2:Psyche's DisTrO optimizer reduces inter-] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	96	"[v1:Psyche's DisTrO optimizer reduces inter-] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	97	"\"Psyche's DisTrO optimizer reduces inter-node commu…\": 0-0 (3 abstain) ✗",
	98	"[v0:DisTrO/DeMo communicates only the sign o] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	99	"[v1:DisTrO/DeMo communicates only the sign o] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	100	"[v2:DisTrO/DeMo communicates only the sign o] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	101	"\"DisTrO/DeMo communicates only the sign of the DCT …\": 0-0 (3 abstain) ✗",
	102	"[v0:Psyche uses the Iroh P2P library to esta] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	103	"[v1:Psyche uses the Iroh P2P library to esta] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	104	"[v0:Streaming DiLoCo reduces total bits exch] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	105	"[v1:The Psyche network is training a 40-bill] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	106	"[v0:The Psyche network is training a 40-bill] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	107	"[v2:The Psyche network is training a 40-bill] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	108	"\"The Psyche network is training a 40-billion-parame…\": 0-0 (3 abstain) ✗",
	109	"[v1:Streaming DiLoCo reduces total bits exch] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	110	"[v2:Psyche uses the Iroh P2P library to esta] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	111	"\"Psyche uses the Iroh P2P library to establish dire…\": 0-0 (3 abstain) ✗",
	112	"[v0:Streaming DiLoCo synchronizes only every] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	113	"[v1:Streaming DiLoCo synchronizes only every] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	114	"[v2:Streaming DiLoCo reduces total bits exch] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	115	"\"Streaming DiLoCo reduces total bits exchanged by ~…\": 0-0 (3 abstain) ✗",
	116	"[v2:Streaming DiLoCo synchronizes only every] failed: API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited",
	117	"\"Streaming DiLoCo synchronizes only every H=100 inn…\": 0-0 (3 abstain) ✗",
	118	"Verify done: 25 claims → 0 confirmed, 25 killed"
	119	],
	120	"result": {
	121	"question": "Research the state of the art (2024-2026) in distributed AI across peers/devices — distributed inference (running ONE model split across multiple machines), distributed/decentralized training, and distributed GPU use — with concrete named systems, how-they-work mechanics, honest maturity (production vs research vs hype), and the fundamental bottlenecks. The context is evaluating what is realistically buildable over a P2P (Holepunch/Hyperswarm) stack.\n\nCover and name concrete systems with mechanics, hardware/bandwidth requirements, and tradeoffs:\n\n1. exo (exolabs): how it actually runs \"multiple peers as one model\" — device discovery, model PARTITIONING strategy (ring/pipeline parallelism, memory-weighted partitioning across heterogeneous devices), supported backends (MLX, tinygrad), what hardware it targets (Mac clusters, Thunderbolt/LAN), throughput/latency reality, and limitations. Is it production or experimental?\n\n2. DISTRIBUTED INFERENCE generally: pipeline parallelism vs tensor parallelism vs data parallelism — what each demands of the network. Named systems: Petals (BitTorrent-style LLM inference), llama.cpp RPC backend, distributed-llama, vLLM + Ray, GPUStack, Cake. LAN vs WAN feasibility, why tensor-parallel needs NVLink/fast interconnect and pipeline-parallel tolerates slower links. Real tokens/sec numbers where available.\n\n3. DISTRIBUTED / DECENTRALIZED TRAINING: low-communication training algorithms that make WAN training feasible — DiLoCo / OpenDiLoCo (Prime Intellect), INTELLECT-1 and INTELLECT-2 (32B, async RL), Nous Research DisTrO / Psyche, Gensyn, Templar, Photon. How they cut communication (local steps + infrequent sync, gradient compression), what was actually trained and released, and verification of untrusted workers (TOPLOC). Real vs vaporware.\n\n4. THE COMMUNICATION BOTTLENECK: why distributed AI is network-bound — NCCL/RCCL collectives (all-reduce, ring), bandwidth/latency requirements of each parallelism strategy, the gap between datacenter interconnect (InfiniBand/NVLink, 100s of GB/s) and consumer LAN/WAN (1-10 Gbps LAN, much less WAN). What bandwidth each approach actually needs and how low-comm methods (DiLoCo) change the math.\n\n5. HETEROGENEOUS / CONSUMER-DEVICE distributed AI: running big models on clusters of consumer machines (Mac Studios via MLX, mixed GPU/CPU, Thunderbolt bridges). Memory-weighted layer assignment, KV-cache distribution, quantization's role. What's genuinely possible on a home/office network today.\n\n6. MAPPING TO P2P: given a Holepunch-style stack (Hyperswarm/HyperDHT discovery, udx transport, hole-punched direct connections), what parts of distributed inference/training are feasible over P2P vs require datacenter interconnect. Which parallelism strategy fits P2P best (pipeline, because it only passes activations point-to-point between consecutive stages), what the activation-transfer bandwidth looks like, and where WAN latency kills it.\n\nPrioritize primary sources (exo GitHub/docs, Petals + DiLoCo papers, Prime Intellect/Nous writeups, llama.cpp RPC docs), real throughput/bandwidth numbers, and an honest production-vs-research-vs-hype verdict for each system. The key question throughout: what can a peer-to-peer network of ordinary machines actually do for running and training AI models today.",
	122	"summary": "All 25 claims refuted by adversarial verification. Research inconclusive — sources may be low-quality or claims overstated.",
	123	"findings": [],
	124	"refuted": [
	125	{
	126	"claim": "exo automatically discovers devices on the network with no manual configuration, using libp2p for peer discovery and cluster formation, and figures out how to split the model based on a realtime view of device topology (Topology-Aware Auto Parallel).",
	127	"vote": "0-0",
	128	"source": "https://github.com/exo-explore/exo"
	129	},
	130	{
	131	"claim": "exo supports both pipeline and tensor parallelism for sharding a single model across multiple devices, achieving up to 1.8x speedup on 2 devices and 3.2x speedup on 4 devices.",
	132	"vote": "0-0",
	133	"source": "https://github.com/exo-explore/exo"
	134	},
	135	{
	136	"claim": "exo targets Mac clusters with hardware interconnect: it provides day-0 support for RDMA over Thunderbolt 5, claiming a 99% reduction in inter-device latency, and benchmarks run on 4x M3 Ultra Mac Studio clusters running models like DeepSeek v3.1 671B and Qwen3-235B.",
	137	"vote": "0-0",
	138	"source": "https://github.com/exo-explore/exo"
	139	},
	140	{
	141	"claim": "exo runs a model across multiple devices using pipeline parallel inference, splitting the model into contiguous slices of layers ('shards'), each shard assigned to a different device (which can be separate accelerators on one machine or physically separate machines over a network).",
	142	"vote": "0-0",
	143	"source": "https://blog.exolabs.net/day-1/"
	144	},
	145	{
	146	"claim": "For exo's pipeline-parallel inference the bottleneck is inter-device latency, not bandwidth, because per-token activations transferred between stages are tiny (under 4KB for Llama 3.2 3B, scaling roughly linearly with layer size).",
	147	"vote": "0-0",
	148	"source": "https://blog.exolabs.net/day-1/"
	149	},
	150	{
	151	"claim": "Adding devices increases aggregate (multi-request) throughput but reduces single-request throughput: Llama 3.2 3B on M4 Pro went from 49.3 TPS single-request on one device to 39.7 TPS on three devices, while multi-request throughput rose from 49.3 to 108.8 TPS (a 3-device cluster processes 2.2x as many tokens as a single device).",
	152	"vote": "0-0",
	153	"source": "https://blog.exolabs.net/day-1/"
	154	},
	155	{
	156	"claim": "vLLM recommends tensor parallelism only when a model fits within a single node's multiple GPUs, and pipeline parallelism (combined with tensor parallelism) when the model is too large for a single node — directly establishing pipeline parallelism as the cross-node strategy.",
	157	"vote": "0-0",
	158	"source": "https://docs.vllm.ai/en/stable/serving/parallelism_scaling/"
	159	},
	160	{
	161	"claim": "Tensor parallelism requires fast internode communication (preferably InfiniBand high-speed network adapters), which is why it is unsuitable for slow/consumer interconnects.",
	162	"vote": "0-0",
	163	"source": "https://docs.vllm.ai/en/stable/serving/parallelism_scaling/"
	164	},
	165	{
	166	"claim": "Even within a single node, when GPUs lack NVLink interconnect (e.g. L40S), vLLM recommends pipeline parallelism over tensor parallelism for higher throughput — confirming tensor parallelism's dependence on high-bandwidth links like NVLink.",
	167	"vote": "0-0",
	168	"source": "https://docs.vllm.ai/en/stable/serving/parallelism_scaling/"
	169	},
	170	{
	171	"claim": "distributed-llama accelerates LLM inference by connecting home devices into a cluster using tensor parallelism with synchronization over Ethernet.",
	172	"vote": "0-0",
	173	"source": "https://github.com/b4rtaz/distributed-llama"
	174	},
	175	{
	176	"claim": "The architecture is split into a root node (loads model/weights, synchronizes neural-network state, forwards to workers) and worker nodes (each processes its own slice of the network, requiring no model configuration), with RAM split across all nodes.",
	177	"vote": "0-0",
	178	"source": "https://github.com/b4rtaz/distributed-llama"
	179	},
	180	{
	181	"claim": "It connects multiple devices over a standard Ethernet switch or router (LAN), not a specialized datacenter interconnect, targeting Linux/macOS/Windows on ARM and x86_64 AVX2 CPUs.",
	182	"vote": "0-0",
	183	"source": "https://github.com/b4rtaz/distributed-llama"
	184	},
	185	{
	186	"claim": "Petals runs LLMs BitTorrent-style: each user loads a small part of the model and joins a network of peers serving the other parts, distributing transformer layers across volunteer peers.",
	187	"vote": "0-0",
	188	"source": "https://github.com/bigscience-workshop/petals"
	189	},
	190	{
	191	"claim": "Single-batch inference on Petals runs at up to 6 tokens/sec for Llama 2 (70B) and up to 4 tokens/sec for Falcon (180B), i.e. interactive but slow speeds for very large models over distributed peers.",
	192	"vote": "0-0",
	193	"source": "https://github.com/bigscience-workshop/petals"
	194	},
	195	{
	196	"claim": "DiLoCo enables training language models across islands of poorly-connected devices while communicating 500 times less than fully synchronous (standard) distributed training, with equivalent final performance on 8 workers.",
	197	"vote": "0-0",
	198	"source": "https://arxiv.org/abs/2311.08105"
	199	},
	200	{
	201	"claim": "DiLoCo is a variant of federated averaging that uses a large number of inner steps with AdamW as the inner optimizer and Nesterov momentum as the outer optimizer (i.e., local steps + infrequent outer sync).",
	202	"vote": "0-0",
	203	"source": "https://arxiv.org/abs/2311.08105"
	204	},
	205	{
	206	"claim": "Prime Intellect trained INTELLECT-2, a 32-billion-parameter reasoning language model, via globally distributed reinforcement learning across a dynamic, heterogeneous, permissionless swarm of compute contributors.",
	207	"vote": "0-0",
	208	"source": "https://arxiv.org/abs/2505.07291"
	209	},
	210	{
	211	"claim": "Decentralized RL training over untrusted permissionless workers required a custom verification component, TOPLOC, to validate rollouts/inference outputs from untrusted workers before they are used.",
	212	"vote": "0-0",
	213	"source": "https://arxiv.org/abs/2505.07291"
	214	},
	215	{
	216	"claim": "INTELLECT-2 and its full code and data were open-sourced, and the model improves on QwQ-32B in the 32B reasoning category, establishing a concrete released artifact rather than a vaporware claim.",
	217	"vote": "0-0",
	218	"source": "https://arxiv.org/abs/2505.07291"
	219	},
	220	{
	221	"claim": "Psyche's DisTrO optimizer reduces inter-node communication via 1-bit quantization of DCT coefficients, compressing results by more than 3x on top of DeMo's order-of-magnitude reduction.",
	222	"vote": "0-0",
	223	"source": "https://nousresearch.com/nous-psyche"
	224	},
	225	{
	226	"claim": "DisTrO/DeMo communicates only the sign of the DCT coefficient to reconstruct the momentum matrix, which is the mechanism enabling low-communication WAN training.",
	227	"vote": "0-0",
	228	"source": "https://nousresearch.com/nous-psyche"
	229	},
	230	{
	231	"claim": "The Psyche network is training a 40-billion-parameter dense model (Consilience) on 20 trillion tokens over a decentralized network.",
	232	"vote": "0-0",
	233	"source": "https://nousresearch.com/nous-psyche"
	234	},
	235	{
	236	"claim": "Psyche uses the Iroh P2P library to establish direct peer-to-peer connections over UDP/QUIC, succeeding with direct (hole-punched) connections in roughly 90% of cases and falling back to relays otherwise.",
	237	"vote": "0-0",
	238	"source": "https://nousresearch.com/nous-psyche"
	239	},
	240	{
	241	"claim": "Streaming DiLoCo reduces total bits exchanged by ~400x and peak bandwidth by ~8x versus standard data-parallel training, while reaching similar loss and downstream accuracy.",
	242	"vote": "0-0",
	243	"source": "https://arxiv.org/pdf/2501.18512"
	244	},
	245	{
	246	"claim": "Streaming DiLoCo synchronizes only every H=100 inner steps and synchronizes parameter subsets sequentially rather than all at once, which is what slashes peak bandwidth.",
	247	"vote": "0-0",
	248	"source": "https://arxiv.org/pdf/2501.18512"
	249	}
	250	],
	251	"sources": [
	252	{
	253	"url": "https://github.com/exo-explore/exo",
	254	"quality": "primary",
	255	"claimCount": 5
	256	},
	257	{
	258	"url": "https://blog.exolabs.net/day-1/",
	259	"quality": "primary",
	260	"claimCount": 4
	261	},
	262	{
	263	"url": "https://www.virge.io/en/blog/exo-mac-studio-cluster-llm/",
	264	"quality": "blog",
	265	"claimCount": 5
	266	},
	267	{
	268	"url": "https://aicoolies.com/reviews/exo-review",
	269	"quality": "secondary",
	270	"claimCount": 5
	271	},
	272	{
	273	"url": "https://www.gkservis.com/case-studies/llm-inference-cluster.html",
	274	"quality": "unreliable",
	275	"claimCount": 5
	276	},
	277	{
	278	"url": "https://michalasobczak.pl/ai-ml/2025/03/exo-the-gpu-cluster-tinygrad-mlx/",
	279	"quality": "blog",
	280	"claimCount": 5
	281	},
	282	{
	283	"url": "https://docs.vllm.ai/en/stable/serving/parallelism_scaling/",
	284	"quality": "primary",
	285	"claimCount": 5
	286	},
	287	{
	288	"url": "https://github.com/b4rtaz/distributed-llama",
	289	"quality": "primary",
	290	"claimCount": 5
	291	},
	292	{
	293	"url": "https://github.com/bigscience-workshop/petals",
	294	"quality": "primary",
	295	"claimCount": 4
	296	},
	297	{
	298	"url": "https://www.spheron.network/blog/what-is-nvlink-gpu-interconnect-bandwidth-explained/",
	299	"quality": "blog",
	300	"claimCount": 5
	301	},
	302	{
	303	"url": "https://github.com/ggml-org/llama.cpp/discussions/9136",
	304	"quality": "forum",
	305	"claimCount": 5
	306	},
	307	{
	308	"url": "https://arxiv.org/abs/2311.08105",
	309	"quality": "primary",
	310	"claimCount": 5
	311	},
	312	{
	313	"url": "https://arxiv.org/abs/2505.07291",
	314	"quality": "primary",
	315	"claimCount": 5
	316	},
	317	{
	318	"url": "https://nousresearch.com/nous-psyche",
	319	"quality": "primary",
	320	"claimCount": 5
	321	},
	322	{
	323	"url": "https://arxiv.org/pdf/2501.18512",
	324	"quality": "primary",
	325	"claimCount": 5
	326	},
	327	{
	328	"url": "https://www.primeintellect.ai/blog/intellect-1",
	329	"quality": "primary",
	330	"claimCount": 5
	331	},
	332	{
	333	"url": "https://www.primeintellect.ai/blog/opendiloco",
	334	"quality": "primary",
	335	"claimCount": 5
	336	},
	337	{
	338	"url": "https://arxiv.org/pdf/2507.14392",
	339	"quality": "primary",
	340	"claimCount": 5
	341	},
	342	{
	343	"url": "https://www.abhik.ai/concepts/gpu/nccl-communication",
	344	"quality": "blog",
	345	"claimCount": 5
	346	},
	347	{
	348	"url": "https://corti.com/borrowing-memory-not-speed-clustering-a-mac-studio-and-a-dgx-spark-with-exo/",
	349	"quality": "blog",
	350	"claimCount": 5
	351	},
	352	{
	353	"url": "https://epoch.ai/gradient-updates/how-far-can-decentralized-training-over-the-internet-scale",
	354	"quality": "primary",
	355	"claimCount": 5
	356	},
	357	{
	358	"url": "https://sharedllm.org/blog/sharedllm-vs-petals-vs-exo.html",
	359	"quality": "blog",
	360	"claimCount": 5
	361	},
	362	{
	363	"url": "https://medium.com/@leif.markthaler/deep-dive-exo-distributed-ai-inference-on-consumer-hardware-068e341d8e3c",
	364	"quality": "blog",
	365	"claimCount": 5
	366	},
	367	{
	368	"url": "https://www.ahmadosman.com/blog/do-not-use-llama-cpp-or-ollama-on-multi-gpus-setups-use-vllm-or-exllamav2/",
	369	"quality": "blog",
	370	"claimCount": 4
	371	},
	372	{
	373	"url": "https://arxiv.org/abs/2509.26182",
	374	"quality": "primary",
	375	"claimCount": 4
	376	},
	377	{
	378	"url": "https://research.yandex.com/blog/petals-decentralized-inference-and-finetuning-of-large-language-models",
	379	"quality": "primary",
	380	"claimCount": 5
	381	},
	382	{
	383	"url": "https://arxiv.org/html/2603.05692v1",
	384	"quality": "primary",
	385	"claimCount": 5
	386	}
	387	],
	388	"stats": {
	389	"angles": 6,
	390	"sources": 27,
	391	"claims": 131,
	392	"verified": 25,
	393	"confirmed": 0,
	394	"killed": 25
	395	}
	396	}
	397	}

research/raw/04-distributed-gpu-salvaged.raw.txt +134 -0

…	…	@@ -0,0 +1,134 @@
	1
	2	========== RESULT 1 ==========
	3	question: Research the state of the art (2023-2026) in distributed GPU usage — using GPUs on other machines, pooling/sharing GPUs across a network, and making GPU compute portable and deterministic — to inform a P2P system where a peer could transparently use GPUs elsewhere in a swarm. Prioritize mechanics, real latency/bandwidth numbers, maturity (production vs research vs hype), and feasibility over commodity LAN/WAN vs datacenter interconnect.
	4	summary: The question spans 7 layers from low-level GPU API remoting up to decentralized GPU markets and P2P feasibility. I split into five non-overlapping angles: (1) GPU-over-IP/API-remoting mechanics and per-call overhead; (2) the interconnect bandwidth wall (NVLink/InfiniBand/NCCL) and the datacenter-vs-Ethernet/WAN cliff that bounds tensor-parallel feasibility; (3) cross-vendor portable compute (Vulkan/WebGPU/SYCL) for heterogeneous swarms; (4) GPU determinism/bit-exact reproducibility, which decides whether results can be memoized/verified by re-execution; and (5) the skeptical reality of decentralized GPU markets plus pooling/disaggregation. Each query targets primary sources and hard numbers (us-per-call, GB/s, overhead %, tokens/sec).
	5	angles: [{"label": "GPU-over-IP / API remoting mechanics + overhead", "query": "rCUDA scuda Juice Labs Thundercompute CUDA API remoting latency overhead per-call benchmark GPU-over-IP", "rationale": "Primary-source mechanics and real per-call latency/bandwidth numbers for driver-call interception+forwarding (sub-topic 1), plus honest maturity of each named system."}, {"label": "Interconnect bandwidth wall (datacenter vs Ethernet/WAN cliff)", "query": "NCCL all-reduce NVLink InfiniBand GPUDirect RDMA bandwidth latency tensor parallelism Ethernet vs datacenter fabric benchmark", "rationale": "Quantifies why cross-GPU compute is interconnect-bound (sub-topics 2-3) and the concrete cliff between 100s-of-GB/s fabric and commodity Ethernet/WAN that kills fine-grained distribution."}, {"label": "Cross-vendor portable GPU compute for heterogeneous hardware", "query": "WebGPU WGSL wgpu Vulkan compute SPIR-V SYCL portable GPU kernel cross-vendor NVIDIA AMD Apple performance cost portability", "rationale": "Covers sub-topic 4: which APIs give a single portable kernel across mixed GPUs in a swarm and the measured performance penalty of portability."}, {"label": "GPU determinism / bit-exact reproducibility across hardware", "query": "GPU floating point nondeterminism reproducibility deterministic cuDNN cuBLAS PyTorch atomic reduction order bit-exact cross-GPU verification", "rationale": "Sub-topic 5 is decisive: whether distributed GPU results can be verified by re-execution or only trusted/stati
	6
	7	========== RESULT 2 ==========
	8	results: [{"url": "https://arxiv.org/pdf/2502.19405", "title": "Verde: Verification via Refereed Delegation for Machine Learning Programs (+ RepOps)", "relevance": "high", "snippet": "THE decisive source for sub-topic 5 mapped to P2P. Gensyn's Verde shows that ML compute over UNTRUSTED nodes cannot be verified by naive re-execution because results are not bitwise-reproducible across hardware -- even A100 vs H100 from the same vendor differ, since IEEE-754 FP ops are non-associative and execution order varies. Their fix, RepOps, ENFORCES a fixed FP operation order (e.g. in matmul reductions) so honest providers yield bitwise-identical output, enabling a refereed-delegation bisection game where a referee re-runs a SINGLE op rather than the whole task. This is exactly the 'can distributed GPU results be verified by re-execution or only trusted' question, with a concrete mechanism and the honest verdict: only feasible if you constrain kernels to a canonical order at a real performance cost."}, {"url": "https://docs.pytorch.org/docs/stable/notes/randomness.html", "title": "Reproducibility \u2014 PyTorch documentation", "relevance": "high", "snippet": "Primary-source enumeration of the actual nondeterminism knobs: torch.use_deterministic_algorithms(True), torch.backends.cudnn.deterministic=True, and the CUBLAS_WORKSPACE_CONFIG env var required for cuBLAS determinism (CUDA >=10.2). Documents that some cuDNN/atomicAdd-based ops have no deterministic implementation and will throw. Establishes
	9
	10	========== RESULT 3 ==========
	11	results: [{"url": "https://arxiv.org/html/2605.20706v1", "title": "Llamas on the Web: Memory-Efficient, Performance-Portable, Multi-Precision LLM Inference with WebGPU", "relevance": "high", "snippet": "Primary-source benchmark quantifying the COST OF PORTABILITY directly relevant to a heterogeneous swarm. Concrete numbers: WebGPU per-dispatch API overhead 24-36 us on Vulkan, 32-71 us on Metal (~95 us total per-op with framework); on RTX 5090/Dawn/Vulkan WebGPU reaches only 11-12% of CUDA throughput for LLM inference, and dispatch overhead (not compute) is the dominant differentiator -- kernel fusion (876->564 dispatches) gives +53% while CUDA fusion gives nothing. This is the honest portability tax for a single WGSL kernel running across NVIDIA/AMD/Apple."}, {"url": "https://arxiv.org/abs/2309.09609", "title": "Comparing Performance and Portability between CUDA and SYCL for Protein Database Search on NVIDIA, AMD, and Intel GPUs", "relevance": "high", "snippet": "Peer-reviewed cross-vendor performance-portability study: SYCL matches or exceeds CUDA on NVIDIA A100 across six workloads (negligible diff elsewhere) and is comparable to HIP on AMD MI100 -- i.e. a single SYCL source can run across NVIDIA/AMD/Intel at near-native speed. Quantifies the central claim that portable kernels need not pay a large performance penalty on datacenter GPUs, while noting Intel Xe efficiency drops to ~23% due to immature drivers. Directly answers 'which APIs give one portable kernel across mixed GPUs and
	12
	13	========== RESULT 4 ==========
	14	results: [{"url": "https://arxiv.org/pdf/2401.13354", "title": "Characterizing Network Requirements for GPU API Remoting in AI Applications (arXiv 2401.13354)", "relevance": "high", "snippet": "PRIMARY-SOURCE benchmark paper, the single best source for this angle. Quantifies API-remoting overhead: <5% achievable, but ONLY under datacenter networking (200 Gbps RDMA, 2-5 us RTT). Key mechanic: a single RDMA MMIO posts in 400-800 cycles, so remoting can avoid batching; over commodity TCP/Ethernet the per-call round-trip dominates and latency-sensitive workloads collapse. Directly answers 'per-call latency, bandwidth needed, which workloads tolerate it' and the LAN/WAN-vs-datacenter cliff central to the P2P mapping."}, {"url": "https://github.com/kevmo314/scuda", "title": "kevmo314/scuda \u2014 GPU over IP bridge (project repo + HN discussion)", "relevance": "high", "snippet": "Primary repo for scuda, the modern open-source GPU-over-IP bridge. Mechanic: codegen reads CUDA dependency headers to auto-generate RPC stubs; intercepts PyTorch CUDA calls and routes them over TCP to a remote GPU, attaching a remote GPU to a CPU-only host. The linked Hacker News thread (news.ycombinator.com/item?id=41787547) gives the honest maturity/physics verdict: practitioners note performance 'will be terrible even with 100/400gb ethernet' for chatty workloads \u2014 exactly the per-call-overhead reality a P2P/WAN design must reckon with."}, {"url": "https://medium.com/@carl_56793/how-thunder-compute-works-gp
	15
	16	========== RESULT 5 ==========
	17	results: [{"url": "https://epoch.ai/gradient-updates/how-far-can-decentralized-training-over-the-internet-scale", "title": "How far can decentralized training over the internet scale? \u2014 Epoch AI", "relevance": "high", "snippet": "Directly on-point for the P2P/WAN throughline. Quantifies the bandwidth cliff: cross-continental WAN links efficiently utilize only ~45 Gbit/s across Europe and ~25 Gbit/s across NA+EU \u2014 orders of magnitude below NVLink's ~900 GB/s intra-node fabric. Explains WHY commodity internet links force communication-avoiding schemes (DiLoCo-style) rather than tensor-parallel/all-reduce, and where the physics ceiling sits for swarm-scale GPU pooling. Best single source on the datacenter-vs-WAN cliff verdict."}, {"url": "https://www.primeintellect.ai/blog/opendiloco", "title": "OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training \u2014 Prime Intellect", "relevance": "high", "snippet": "Primary source showing the only form of distributed GPU training that survives the bandwidth wall over commodity links. Workers sync every ~500 inner steps (minutes apart, not every few hundred ms like data/tensor parallelism), with int8 all-reduce giving ~400x bandwidth reduction. Demonstrated across 2 continents / 3 countries at 90-95% compute utilization. Defines the production-vs-research line for what a P2P swarm can realistically do: coarse data-parallel, NOT fine-grained collective ops."}, {"url": "https://arxiv.org/pdf/1903.04611", "t
	18
	19	========== RESULT 6 ==========
	20	claims: [{"claim": "Vulkan compute achieves roughly 80-95% of the vendor's native compute API performance for typical bandwidth-bound and compute-bound kernels.", "quote": "80-95% of the vendor's native compute API for typical bandwidth-bound and compute-bound kernels", "importance": "central"}, {"claim": "Vulkan is genuinely cross-vendor with drivers for NVIDIA, AMD, Intel, Apple Silicon, and mobile GPUs, with no API-level lock-in.", "quote": "Vulkan is genuinely cross-vendor \u2014 drivers exist for NVIDIA, AMD, Intel, Apple Silicon, mobile GPUs", "importance": "central"}, {"claim": "OpenCL is in decline for ML/AI: cross-vendor but retreating from the ML ecosystem and not keeping pace with post-2020 hardware features.", "quote": "cross-vendor but is in a slow retreat from the ML ecosystem ... The standard has not kept pace with hardware features introduced after roughly 2020", "importance": "supporting"}, {"claim": "SYCL is the recommended path for a single codebase running across AMD, Intel, and NVIDIA, relying on vendor-specific back-ends.", "quote": "when the workload must run on AMD, Intel, and NVIDIA from a single codebase ... across vendors with vendor-specific back-ends doing the heavy lifting", "importance": "supporting"}, {"claim": "CUDA is not portable and is exclusive to NVIDIA hardware.", "quote": "Not portable\" and \"NVIDIA exclusivity", "importance": "supporting"}]
	21	publishDate: 2026-01-08
	22	sourceQuality: blog
	23
	24	========== RESULT 7 ==========
	25	claims: [{"claim": "The same floating-point compute kernel produces different results on Vulkan/SPIR-V vs OpenCL on the same NVIDIA desktop GPU (RTX 3090), while OpenCL and CPU backends matched each other.", "quote": "CL (desktop) = CPU (desktop) = CL (mobile) = CPU (mobile) = VK (mobile); VK (desktop) Different!", "importance": "central"}, {"claim": "The Vulkan/NVIDIA-desktop divergence accumulates into a meaningful numerical error (~0.2 difference), not just last-bit noise.", "quote": "Vulkan results diverged on NVIDIA desktop hardware, accumulating errors with \"~0.2 difference\"", "importance": "central"}, {"claim": "Standard SPIR-V floating-point control mechanisms (NoContraction, RoundingModeRTE, SignedZeroInfNanPreserve, FloatControl2, FPFastMathDefault) failed to make NVIDIA Vulkan results match the other backends.", "quote": "None of this worked, they didn't even change the result.", "importance": "central"}, {"claim": "The poster suspected NVIDIA's Vulkan driver flushes float32 denormals to zero with no available way to preserve them, though this was not confirmed as the sole cause.", "quote": "the different stands in Vulkan on NV driver that is flushing float32 denorms to zero. Unfortunately there seems to be no way to preserve denorms on NV hardware", "importance": "supporting"}, {"claim": "The same three backends (OpenCL, CPU, Vulkan) produced identical results on Qualcomm Adreno 830 mobile hardware, indicating the discrepancy is hardware/driver-specific rather than inhe
	26	publishDate: 2026-03-14
	27	sourceQuality: forum
	28
	29	========== RESULT 8 ==========
	30	results: [{"url": "https://arxiv.org/html/2502.19405v1", "title": "Verde: Verification via Refereed Delegation for Machine Learning Programs (Gensyn paper)", "relevance": "high", "snippet": "Primary source for the ONLY decentralized network with a real cryptographic-style verification design. Verde verifies ML training over untrusted nodes via refereed-delegation: trainers store/hash intermediate checkpoints, a verifier re-runs; on disagreement a referee (smart contract or jury) recomputes only the single first-divergent operator in the compute graph, slashing verification overhead vs full re-execution. Honest if >=1 verifier is honest. Crucially depends on RepOps (bitwise-reproducible operators enforcing fixed FP reduction order) to defeat hardware non-determinism \u2014 directly answers sub-topic 5 (determinism is the precondition for verification-by-re-execution) and 6 (real verification mechanics). This is the load-bearing primary source for the skeptical verdict: verification is feasible but only because they re-engineered determinism into the kernels."}, {"url": "https://blog.gensyn.ai/verde-verification-system-in-production/", "title": "Verde Verification System In Production \u2014 Gensyn blog", "relevance": "high", "snippet": "Vendor primary source describing Verde's production deployment and RepOps (bitwise-reproducible ML operators that fix floating-point execution order so honest providers yield bit-identical outputs). Establishes maturity: proof-of-learning/refereed-deleg
	31
	32	========== RESULT 9 ==========
	33	claims: [{"claim": "gpu.cpp uses the WebGPU specification as a portable low-level GPU interface, enabling the same C++ GPU code to run across NVIDIA, Intel, AMD, and other GPUs via Vulkan, Metal, or DirectX backends.", "quote": "leveraging the WebGPU specification as a portable low-level GPU interface ... we can drop in GPU code in C++ projects and have it run on Nvidia, Intel, AMD, and other GPUs ... The same C++ code can work on a wide variety of laptops, workstations, mobile devices or virtually any hardware with Vulkan, Metal, or DirectX support.", "importance": "central"}, {"claim": "gpu.cpp's WebGPU implementation is built on Google's Dawn native backend, with plans to also support browser WebGPU and wgpu as alternative implementations.", "quote": "support other targets and WebGPU implementations (web browsers or other native implementations such as wgpu)", "importance": "supporting"}, {"claim": "A portable WebGPU matrix-multiplication kernel achieves roughly 1.2 TFLOPs on an Apple M1 MacBook Pro, against a theoretical peak of 10.4 TFLOPs (about 12% of peak), illustrating the performance cost of portability.", "quote": "an estimated ~ 1.2+ TFLOPs on a Macbook Pro M1 laptop, which has a theoretical peak of 10.4 TFLOPs", "importance": "central"}, {"claim": "The core gpu.cpp library is a header-only implementation of around 1000 lines of code, with compile/run cycles under 5 seconds, demonstrating WebGPU compute can be embedded with minimal footprint.", "quote": "The core library
	34	publishDate: 2024-07-13
	35	sourceQuality: primary
	36
	37	========== RESULT 10 ==========
	38	claims: [{"claim": "SYCL and CUDA achieve similar performance on NVIDIA GPUs for a Smith-Waterman protein database search workload, meaning the portability layer (SYCL) does not impose a meaningful slowdown vs native CUDA on NVIDIA hardware.", "quote": "while both CUDA and SYCL versions achieve similar performance on NVIDIA devices, the latter demonstrated remarkable code portability to other GPU architectures, such as AMD and Intel.", "importance": "central"}, {"claim": "A single SYCL codebase is portable across NVIDIA, AMD, and Intel GPUs (single and multi-GPU configs), whereas CUDA is NVIDIA-specific \u2014 directly relevant to running portable kernels across a heterogeneous swarm of mixed-vendor GPUs.", "quote": "the latter demonstrated remarkable code portability to other GPU architectures, such as AMD and Intel", "importance": "central"}, {"claim": "SYCL did not just match but exceeded architectural efficiency in 3 of 4 non-NVIDIA cases tested, indicating portability did not always come at a performance cost on AMD/Intel hardware.", "quote": "the architectural efficiency rates achieved on these devices were superior in 3 of the 4 cases tested", "importance": "supporting"}, {"claim": "The study is empirical and peer-reviewed, evaluating real GPUs from three vendors (NVIDIA, Intel, AMD) in both single- and multi-GPU configurations, lending production-relevant evidence for cross-vendor portable compute.", "quote": "This study evaluates the portability and performance of the SYCL a
	39	publishDate: 2023-09-18
	40	sourceQuality: primary
	41
	42	========== RESULT 11 ==========
	43	claims: [{"claim": "Bit-exact reproducibility from PyTorch GPU computation is not guaranteed across PyTorch releases, individual commits, or different hardware platforms.", "quote": "Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms.", "importance": "central"}, {"claim": "Even with identical RNG seeds, results may differ between CPU and GPU execution, meaning re-execution on a different device cannot be assumed to reproduce a prior result.", "quote": "Results may not be reproducible between CPU and GPU executions, even when using identical seeds.", "importance": "central"}, {"claim": "Some CUDA operations have no deterministic implementation at all; with torch.use_deterministic_algorithms(True), PyTorch raises an error rather than producing a nondeterministic result.", "quote": "throw an error if an operation is known to be nondeterministic (and without a deterministic alternative)", "importance": "central"}, {"claim": "Determinism in PyTorch carries a measurable performance cost: forcing deterministic algorithms is generally slower than the default nondeterministic ones.", "quote": "Deterministic operations are often slower than nondeterministic operations.", "importance": "supporting"}, {"claim": "Disabling cuDNN autotuning alone does not guarantee determinism; the selected convolution algorithm can itself be nondeterministic unless deterministic mode is explicitly enabled.", "quote": "that algorithm itself may be
	44	publishDate: 2025-10-03
	45	sourceQuality: primary
	46
	47	========== RESULT 12 ==========
	48	claims: [{"claim": "GPUs are inherently non-deterministic, making bit-exact reproducibility across runs impossible, which directly breaks verification schemes that rely on re-executing a computation to check its result.", "quote": "GPUs are inherently non-deterministic, making it impossible to achieve bit-exact reproducibility", "importance": "central"}, {"claim": "Bit-exact reproducibility is a prerequisite for verifying GPU computations in security-critical settings such as untrusted or remote GPUs running ML models \u2014 exactly the proof-of-computation/integrity-check pattern a P2P GPU swarm would need.", "quote": "Bit-exact reproducibility is essential for verifying GPU computations in security-critical applications", "importance": "central"}, {"claim": "The primary causes of GPU non-determinism are atomic operations, floating-point reduction operations whose results depend on summation order, and unspecified scheduling of concurrent GPU threads.", "quote": "atomic operations", "importance": "supporting"}, {"claim": "Hawkeye is a GPU simulator that reproduces and detects non-deterministic behavior rather than enforcing determinism, letting researchers study how different execution orderings change computation outcomes.", "quote": "https://github.com/badasherez/gpu-simulator", "importance": "supporting"}, {"claim": "The work explicitly ties GPU non-determinism to distributed ML training and zero-knowledge proofs for ML (citing Megatron-LM, ZeRO, and cloud ML platforms), framing
	49	publishDate: 2026-05-19
	50	sourceQuality: primary
	51
	52	========== RESULT 13 ==========
	53	claims: [{"claim": "A single WebGPU-based LLM inference framework (LlamaWeb) runs portably across 16 devices from 8 different GPU vendors (NVIDIA, AMD, Intel, Apple, Qualcomm, Samsung, ARM, Imagination), demonstrating WebGPU/WGSL as a genuine cross-vendor portable GPU compute substrate for heterogeneous hardware.", "quote": "GPU Vendors Tested (16 devices across 8 vendors): NVIDIA (3 devices: RTX 5080, RTX 5070, Tesla T4), AMD (1 device: RX 7900 XT), Intel (2 devices: Arc B580, Iris Xe), Apple (5 devices...), Qualcomm, Samsung, ARM, Imagination Tech", "importance": "central"}, {"claim": "Portable WebGPU compute carries a large performance penalty versus native CUDA/Vulkan: native backends outperform WebGPU by up to 10x during prefill and 2.5x during decode, with Metal beating WebGPU by over 2x prefill and 50% decode.", "quote": "The CUDA and Vulkan backends still significantly outperform WebGPU by up to 10\u00d7 during prefill and 2.5\u00d7 during decode.", "importance": "central"}, {"claim": "WebGPU portability is not uniform: the same WGSL kernels show order-of-magnitude implementation-dependent variance across browser backends, e.g. ~1 tok/s on Firefox vs ~52 tok/s on Chrome on the same Apple M3 device.", "quote": "substantial implementation-dependent performance differences across browsers,\" with Firefox performing at \"~1 tok/s on Firefox, but ~52 tok/s on Chrome\" on Apple M3.", "importance": "supporting"}, {"claim": "WebGPU is not always slower than native portable APIs: the
	54	publishDate: 2026-05-20
	55	sourceQuality: primary
	56
	57	========== RESULT 14 ==========
	58	claims: [{"claim": "GPU floating-point computations can be made bit-exact across heterogeneous hardware (NVIDIA T4, RTX 3090, A100) by controlling FP operation order; the RepOps library re-implements ML operators (and exp/sin/cos/tanh) to enforce a fixed order, eliminating IEEE-754 non-associativity nondeterminism. This makes re-execution-based verification possible.", "quote": "RepOps re-implements common ML operators ... in a way that controls the order of floating point operators across hardware setups ... eliminating hardware non-determinism.", "importance": "central"}, {"claim": "Enforcing reproducibility carries a real performance cost: RepOps matrix multiplication incurs ~30-60% overhead for matrices larger than 2^10, with a steady-state ~60-70% overhead on RTX 3090 (24GB) and ~35% on T4 (16GB); overhead worsens for small matrices (~200% for 256-dim).", "quote": "RepOps matrix multiplications (of dimensions larger than 2^10) incur less than 30% overhead when tested on various GPUs.", "importance": "central"}, {"claim": "End-to-end reproducible model overhead is large and hardware-dependent: DistilBERT training incurred 258% (T4) and 312% (A100-40GB) overhead; Llama-3.1-1B training 374% (T4) and 67% (A100); Llama-3.1-8B 98% inference / 126% LoRA fine-tuning on A100-80GB. Overhead is lower on larger-memory GPUs.", "quote": "both inference and training on Llama-3.1-1B incur around 60% overhead RepOps on an A100 GPU (with 40 GB VRAM).", "importance": "central"}, {"claim": "RepOps
	59	publishDate: 2025-02-26
	60	sourceQuality: primary
	61
	62	========== RESULT 15 ==========
	63	claims: [{"claim": "GPU floating-point non-associativity makes parallel reductions run-to-run non-deterministic, but deterministic alternatives (pairwise/tree reductions avoiding atomicAdd) can match or only slightly underperform the non-deterministic versions depending on GPU type.", "quote": "Our results show that the performance of a deterministic implementation can be faster or only slightly slower than its non-deterministic counterpart, depending on the GPU type.", "importance": "central"}, {"claim": "The naive deterministic single-atomicAdd approach is catastrophically slow: atomicAdd-only (AO) is two orders of magnitude slower than the fastest parallel-sum implementations, so determinism-via-serialization is impractical.", "quote": "We find that AO is 2 orders of magnitude slower than the fastest implementations.", "importance": "supporting"}, {"claim": "Well-designed deterministic GPU reduction kernels carry only a small overhead versus non-deterministic ones: under 0.2% difference on V100, up to 7.8% on GH200, with most V100 implementations within 0.5-1% of each other.", "quote": "The difference in timings between SPA and SPTR is less than 0.2% for the V100 GPU but can reach up to 7.8% on the GH200... Most implementations on V100 are within 0.5% to 1% of each other.", "importance": "supporting"}, {"claim": "FPNA-induced variability accumulates in iterative solvers and can approach or exceed 20% after six or seven iterations even in double precision, exceeding tolerances (as
	64	publishDate: 2024-10-30
	65	sourceQuality: primary
	66
	67	========== RESULT 16 ==========
	68	claims: [{"claim": "Heterogeneous GPU hardware has a fundamental binary compatibility problem: a kernel compiled for one vendor's GPU cannot run on another due to divergent instruction sets, execution models, and driver stacks.", "quote": "Heterogeneous GPU infrastructures present a binary compatibility challenge: code compiled for one vendor's GPU will not run on another due to divergent instruction sets, execution models, and driver stacks.", "importance": "central"}, {"claim": "HetGPU proposes a single-GPU-binary portability stack (compiler + runtime + abstraction layer) that enables one binary to execute across NVIDIA, AMD, Intel, and Tenstorrent hardware via an architecture-agnostic intermediate representation translated to native code at execution time.", "quote": "We propose hetGPU, a new system comprising a compiler, runtime, and abstraction layer that together enable a single GPU binary to execute on NVIDIA, AMD, Intel, and Tenstorrent hardware.", "importance": "central"}, {"claim": "Achieving portability requires bridging fundamentally different GPU execution models, including warp-centric SIMT designs (NVIDIA/AMD) versus core-centric MIMD approaches (Tenstorrent RISC-V), unifying thread, memory, and synchronization abstractions.", "quote": "Bridges conceptual differences between warp-centric designs (NVIDIA/AMD SIMT) and core-centric approaches (Tenstorrent RISC-V MIMD), unifying thread, memory, and synchronization abstractions.", "importance": "supporting"}, {"claim": "He
	69	publishDate: 2025-06-19
	70	sourceQuality: primary
	71
	72	========== RESULT 17 ==========
	73	claims: [{"claim": "Thunder Compute intercepts CUDA API calls on a virtual machine and translates them into network messages forwarded over TCP to a remote GPU, presenting a remote GPU as if it were local.", "quote": "Your virtual machine communicates with the GPU over TCP \u2014 the same protocol your browser uses. ... Behind the scenes, our instance translates those calls into network messages.", "importance": "central"}, {"claim": "Thunder Compute claims the GPU-over-TCP connection overhead is roughly 10-20 milliseconds for the initial connection, plus a scaled runtime increase, with GPU (re)attachment on the order of double-digit milliseconds.", "quote": "~10\u201320 milliseconds for the initial connection (blinking is ~200 milliseconds), and a scaled runtime increase.", "importance": "central"}, {"claim": "Thunder Compute asserts the GPU-over-TCP approach is viable because ML workloads are compute-bound rather than data-transfer-bound, and they optimize program execution to hide network latency.", "quote": "Most ML jobs spend far more time computing than waiting for data. By strategically optimizing the way your program runs behind the scenes, we can prevent network latency from affecting your GPU computation.", "importance": "central"}, {"claim": "In Thunder Compute's model a running process owns the entire physical GPU (full VRAM and compute), rather than fractionally sharing it among tenants simultaneously.", "quote": "When your process runs, it owns the whole GPU. You have a
	74	publishDate: 2025-01-07
	75	sourceQuality: blog
	76
	77	========== RESULT 18 ==========
	78	claims: [{"claim": "Typical consumer internet upload bandwidth (~60 Mbps) makes naive data-parallel training infeasible: training a 671B-parameter model that way would take roughly 5,000 years, and the largest trainable model in a 10-minute sync window is only ~600M parameters with 32-bit gradients.", "quote": "Typical internet upload bandwidths for consumers are around 60 Mbps ... training a 671B parameter model would require \"5,000 years\" ... Maximum trainable model with 10-minute synchronization window at consumer speeds: \"600M parameter model with 32-bit gradients\"", "importance": "central"}, {"claim": "DiLoCo reduces communication-bandwidth requirements by ~500x by lowering synchronization frequency (up to 500 inner steps between syncs) with only a small performance penalty.", "quote": "Enables \"500x reduction in bandwidth requirements\" through reduced synchronization frequency ... Permits \"up to 500 inner steps with only a small reduction in performance\"", "importance": "central"}, {"claim": "Combined techniques (Streaming DiLoCo with quantization/sparsification) achieve roughly 100x reduction in bandwidth strain versus naive data parallelism; 8-bit gradients give 2-4x savings, and 25% sparsification (SparseLoCo) allows training models 8x larger.", "quote": "Combined techniques achieve \"100x reduction in bandwidth strain compared to naive data parallelism\" ... 8-bit gradient reduction: 2-4x bandwidth savings versus 32-bit ... SparseLoCo with 25% sparsification: enable
	79	publishDate: 2025-12-29
	80	sourceQuality: primary
	81
	82	========== RESULT 19 ==========
	83	claims: []
	84	sourceQuality: unreliable
	85
	86	========== RESULT 20 ==========
	87	claims: [{"claim": "DiLoCo reduces inter-node communication frequency by 500x for a 150M model (and 125x in billion-parameter runs) by synchronizing pseudo-gradients only every ~500 local steps instead of every step, making globally distributed training feasible over commodity internet links.", "quote": "500x lower communication requirement compared to baseline with same compute budget (150M model); 125x reduction achieved in billion-parameter experiments with local step size of 125; DiLoCo synchronizes pseudo-gradients every ~500 steps versus traditional approaches", "importance": "central"}, {"claim": "OpenDiLoCo successfully trained a 1.1B-parameter model across four H100 clusters in three countries (Canada, Finland, Texas, Delaware) connected by links of only 127 to 935 Mbit/s, i.e. commodity-WAN bandwidth rather than datacenter fabric.", "quote": "Four H100 GPU clusters across three countries... Network bandwidth: \"between 127 to 935 Mbit/s\" across worker pairs", "importance": "central"}, {"claim": "Despite WAN-scale links spanning two continents, the system maintained 90-95% compute utilization, with the all-reduce synchronization consuming only 6.9% of total training time even though each all-reduce averaged 300 seconds.", "quote": "\"90-95% compute utilization\" maintained during globally distributed training across two continents and three countries; All-reduce bottleneck represented only \"6.9% of the training time\"; \"average all-reduce time between the workers of 300 s
	88	publishDate: 2024-07-11
	89	sourceQuality: primary
	90
	91	========== RESULT 21 ==========
	92	claims: [{"claim": "With careful (asynchronous) remoting design, unmodified AI applications can run over GPU API remoting using commodity networking hardware with no overhead or even better performance and low network demands, contradicting the assumption that GPU-over-IP requires datacenter interconnect.", "quote": "with careful remoting design, unmodified AI applications can run on the remoting setup using commodity networking hardware without any overhead or even with better performance, with low network demands.", "importance": "central"}, {"claim": "Even at 10 microsecond network latency and just 1 Gbps bandwidth, GPT-2 inference under GPU API remoting suffers only ~2% performance degradation, because GPU execution time (10-100 ms) dwarfs network delay (1-100 us).", "quote": "At one extreme, GPT-2 only experiences a 2% performance degradation under a 10 \u00b5s RTT and 1 Gbps setup. The main reason is that the execution time of AI applications on GPUs (10\u2013100 ms) is typically significantly longer than the network delay", "importance": "central"}, {"claim": "Network latency in the 5-20 microsecond range is sufficient for most inference applications to benefit from API remoting with less than 5% overhead; achieving this is trivial if the proxy is co-located, and easy in modern datacenter networks.", "quote": "Latency in the range of 5\u201320 \u00b5s is sufficient for most inference applications to benefit from API remoting with less than 5% overhead.", "importance": "central
	93	publishDate: 2024-01-24
	94	sourceQuality: primary
	95
	96	========== RESULT 22 ==========
	97	claims: [{"claim": "scuda (now renamed LUPINE, transferred from kevmo314 to lupinemachines) is a GPU-over-IP bridge that lets remote-machine GPUs be attached to CPU-only machines by intercepting CUDA driver API and NVML calls via LD_PRELOAD shim libraries (libcuda.so.1 and libnvidia-ml.so.1).", "quote": "LUPINE is a GPU over IP bridge allowing GPUs on remote machines to be attached to CPU-only machines. ... CUDA driver users pick up the LUPINE libcuda.so.1 shim and NVML users such as nvidia-smi pick up the LUPINE libnvidia-ml.so.1 shim automatically.", "importance": "central"}, {"claim": "The remoting transport is plain HTTP/2 (not RDMA), which is feasible to run over commodity/WAN networks and can be fronted with arbitrary TLS/auth.", "quote": "It's a plain HTTP/2 server, so you can front it with whatever TLS/auth server you want.", "importance": "central"}, {"claim": "The dominant overhead of GPU-over-IP remoting is slower host-device data transfers (because it bottlenecks the PCIe link over the network), but per-call compute overhead is small; once a model is resident on the GPU, little host transfer occurs, so latency-tolerant workloads with low host-device traffic perform acceptably.", "quote": "You will see device transfers get slower because this is basically bottlenecking a PCIe link over the network, but there is very little overhead besides that. ... once the model is on the GPU very little data transfer happens to the host. As a result, it might be faster than you expect."
	98	publishDate: 2026-06-03
	99	sourceQuality: primary
	100
	101	========== RESULT 23 ==========
	102	claims: [{"claim": "Exact re-execution cannot verify GPU computation in decentralized networks because GPU operations are non-deterministic across nodes, making bitwise comparison of redundant computations ineffective even when hardware/software variables are controlled.", "quote": "The non-deterministic nature of GPU operations precludes the implementation of exact recomputation as a verification mechanism. While theoretical approaches suggest bitwise comparison of redundant computations as an optimal verification strategy, the architectural characteristics of GPU computing render such methodologies ineffective.", "importance": "central"}, {"claim": "GPU non-determinism stems from differences in GPU architecture, driver versions, CUDA implementations, cuDNN libraries, and framework distributions, but persists even when all of these are strictly controlled, due to the parallel execution nature of GPU operations.", "quote": "This non-determinism stems from multiple sources of hardware and software variability, including differences in GPU architecture, driver versions, CUDA implementations, cuDNN libraries, and framework distributions. However, even in environments where these variables are strictly controlled, fundamental non-determinism persists due to the parallel execution nature of GPU operations", "importance": "central"}, {"claim": "In LLM inference, non-determinism arises specifically from the non-guaranteed ordering of floating-point arithmetic in parallel matrix operations;
	103	publishDate: 2025-01-09
	104	sourceQuality: primary
	105
	106	========== RESULT 24 ==========
	107	claims: [{"claim": "NCCL's three protocols trade latency for bandwidth: the LL (Low-Latency) protocol achieves ~1 microsecond per-hop latency but only 25-50% of peak bandwidth, LL128 achieves ~2 microsecond latency at ~95% of peak bandwidth, and the Simple protocol reaches near-peak bandwidth for large messages but with higher synchronization overhead.", "quote": "LL Protocol: \"~1 \u03bcs\" latency per-hop, Achieves \"25\u223c50% of peak\" bandwidth ... LL128 Protocol: \"~2 \u03bcs\" latency per-hop, Achieves \"\u223c95% of peak\" bandwidth ... Simple ... \"achieves near-peak bandwidth for large messages\"", "importance": "central"}, {"claim": "There is a roughly 6x bandwidth gap between intra-node and inter-node interconnect on a modern GPU supercomputer (Alps): 150 GB/s intra-node high-bandwidth interconnect versus 25 GB/s per-direction inter-node network links (Cray Slingshot).", "quote": "\"150GB/s high-bandwidth intra-node interconnect\" ... \"25GB/s per-direction network link\" (Cray Slingshot)", "importance": "central"}, {"claim": "LL128's high-bandwidth efficiency depends on atomic 128-byte writes (120B data + 8B flag per 128-byte unit), a hardware-level guarantee not available over commodity network transports.", "quote": "LL128 Protocol: ... 120B data + 8B flag per 128-byte unit, \"depends on atomic 128-byte writes\"", "importance": "supporting"}, {"claim": "Collective algorithm choice is message-size dependent: ring algorithms are optimal for large messages while tree alg
	108	publishDate: 2025-07-07
	109	sourceQuality: primary
	110
	111	========== RESULT 25 ==========
	112	claims: [{"claim": "GPU floating-point computation can be made bit-exact across different GPUs by fixing the order of floating-point operations, because FP ops are non-associative even under IEEE-754; Verde's RepOps library does this for ML operators.", "quote": "RepOps (Reproducible Operators), a library that implements bitwise reproducible versions of popular ML operators. It works by ensuring that floating point operations are performed in the same order", "importance": "central"}, {"claim": "Forcing reproducible (fixed-order) floating-point operations carries a real but bounded performance cost: under ~30% overhead for large matrix multiplications, ~58-98% inference and 67-312% training overhead depending on model, on A100 GPUs.", "quote": "RepOps matrix multiplications (of dimensions larger than 2^10) incur less than 30% overhead when tested on various GPUs", "importance": "central"}, {"claim": "Verifying an untrusted ML computation by refereed delegation requires the referee to compute only a single diverging operator, not re-run the whole model, costing two orders of magnitude less compute than running the model.", "quote": "The referee's only needs to compute a single operator in the computational graph, which can be performed with two orders of magnitude less compute resources than it takes to run the model itself", "importance": "central"}, {"claim": "Refereed-delegation verification of ML adds less than one order of magnitude total overhead, versus four orders of magnitude
	113	publishDate: 2025-02-26
	114	sourceQuality: primary
	115
	116	========== RESULT 26 ==========
	117	claims: [{"claim": "Gensyn's Verde achieves bitwise-reproducible GPU/ML results across diverse hardware using RepOps, which enforces a fixed ordering of operations for common ML operators to overcome floating-point non-associativity.", "quote": "RepOps (reproducible operators) enforces a fixed ordering of operations for common ML operators (such as matrix multiplication) to overcome the non-associativity problem... With RepOps, we achieve bitwise-reproducible results across diverse hardware.", "importance": "central"}, {"claim": "Verde verifies untrusted GPU computation via refereed delegation: when two providers disagree, a referee resolves it by re-executing a single operation rather than the entire task, located via a two-level (iteration then operation) bisection game.", "quote": "if two compute providers disagree on an ML execution, a referee can determine which provider is honest by executing a single operation rather than running the entire task.", "importance": "central"}, {"claim": "Verde reduces verification overhead by re-executing only a randomly selected single checkpoint interval for training (not the whole task), and a single forward pass for autoregressive inference via causal masking.", "quote": "only a single checkpoint interval (randomly selected)", "importance": "supporting"}, {"claim": "Naive hashing of outputs fails for verifying ML training because floating-point non-associativity differences propagate and amplify into significant deviations, which is why fixed
	118	publishDate: 2025-11-24
	119	sourceQuality: primary
	120
	121	========== RESULT 27 ==========
	122	claims: [{"claim": "NVIDIA MIG partitions a single supported GPU into up to seven hardware-isolated GPU instances, each with dedicated memory and compute and hardware-level fault isolation, but only on Ampere+ (compute capability 8.0+) datacenter GPUs (A100, A30, H100, H200, B200, GB200).", "quote": "Multi-Instance GPU (MIG) partitions a supported Ampere-generation (compute capability 8.0+) NVIDIA GPU into up to seven hardware-isolated GPU instances", "importance": "central"}, {"claim": "MIG is static hardware partitioning using predefined profiles that must be planned in advance, and reconfiguring requires the GPU to be free of running workloads \u2014 so it fractionalizes one GPU among tenants but cannot dynamically adapt.", "quote": "Static, hardware-level partitioning method, MIG uses predefined profiles, so you must plan slice sizes in advance", "importance": "supporting"}, {"claim": "MPS gives more explicit memory/compute partitioning than time-slicing but weaker isolation than MIG; a fatal fault in one client can put the MPS server into a FAULT state and affect other clients, and NVIDIA marks MPS device-plugin support as experimental.", "quote": "A fatal fault from one client can cause the MPS server to enter a FAULT state and affect other clients", "importance": "central"}, {"claim": "Time-slicing shares a single physical GPU via CUDA context switching across processes taking turns; it does not isolate memory or faults and contention can impact latency, but it works across a
	123	publishDate: 2026-05-26
	124	sourceQuality: blog
	125
	126	========== RESULT 28 ==========
	127	claims: [{"claim": "The io.net attack was a SQL injection that exploited exposed user ID tokens to make unauthorized changes to device metadata in the GPU network; the GPUs' actual hardware was NOT compromised due to permission layers.", "quote": "Malicious users exploited exposed user ID tokens to execute a system query language (SQL) injection attack, which led to unauthorized changes in device metadata within the graphics processing unit (GPU) network ... The attack did not compromise the GPUs' actual hardware, which remains secure due to robust permission layers.", "importance": "central"}, {"claim": "io.net suffered a Sybil attack in which roughly 1.8 million fake/spoofed GPUs attempted to connect to the network to solicit rewards.", "quote": "the network saw an enormous spike of ~1.8M fake GPUs attempting to connect, as users were spoofing GPUs to solicit rewards from the network in what was characterized as a Sybil attack", "importance": "central"}, {"claim": "After security upgrades, active GPU connections on io.net dropped dramatically from 600,000 to 10,000 because valid GPUs that did not restart and upgrade could not access the uptime API.", "quote": "valid GPUs that did not restart and upgrade could not access the uptime API, causing a dramatic decline in active GPU connections from 600,000 to 10,000", "importance": "supporting"}, {"claim": "The metadata breach was detected via monitoring of a surge in write operations to the GPU metadata API, triggering alerts at 1:05 am
	128	publishDate: 2024-04-28
	129	sourceQuality: secondary
	130
	131	========== RESULT 29 ==========
	132	claims: [{"claim": "Intra-node NVSwitch in the DGX-2 delivers 50 GB/s bidirectional per link, ~300 GB/s per GPU via a single switch hop, and a baseboard-to-baseboard raw bisection bandwidth of 2.4 TB/s \u2014 establishing that datacenter GPU fabric operates in the hundreds-of-GB/s-to-TB/s range.", "quote": "each of the GPUs on one baseboard can talk to any GPUs on the other baseboard also at a full bandwidth of 300 GB/s, but through 2 NVSwitch hops. The baseboard-to-baseboard raw bisection bandwidth is thus 25 GB/s x 8 links/switch x 6 switches=2.4 TB/s", "importance": "central"}, {"claim": "Inter-node GPU communication over InfiniBand with GPUDirect-RDMA peaks at only ~12 GB/s (at 64MB messages) on SummitDev, with pinned-memory GPUDirect reaching >14 GB/s \u2014 i.e. inter-node bandwidth is more than an order of magnitude below intra-node NVLink/NVSwitch.", "quote": "From 4MB on, GPUDirect-RDMA shows its advantage on bandwidth and obtains its optimal bandwidth -- 12GB/s at 64MB. However, this is still lower than the PinnedMem-GPUDirect scheme, which demonstrates more than 14GB/s sustainable bandwidth", "importance": "central"}, {"claim": "Inter-node collective (all-reduce etc.) bandwidth across IB-connected nodes saturates at only ~16 GB/s on SummitDev and ~32 GB/s on Summit, and only at very large 32/64MB message sizes \u2014 small messages get poor efficiency.", "quote": "the bandwidth for the five CL operations converge and saturate around 32/64MB message size, demonstrating near
	133	publishDate: 2019-03-11
	134	sourceQuality: primary

research/raw/05-gpu-verification.raw.json +167 -0

…	…	@@ -0,0 +1,167 @@
	1	{
	2	"summary": "Adversarially verify the salvaged GPU-research claims (2-vote refute-biased)",
	3	"agentCount": 16,
	4	"logs": [],
	5	"result": [
	6	{
	7	"i": 0,
	8	"claim": "Gensyn RepOps makes GPU ML operators BITWISE-reproducible across different GPU hardware (e",
	9	"verdict": "CONFIRMED",
	10	"refutes": 0,
	11	"votes": [
	12	{
	13	"refuted": false,
	14	"confidence": "high",
	15	"finding": "The Verde paper (arXiv:2502.19405) PDF directly states RepOps gives bitwise-reproducible operators by fixing floating-point op order, tested on T4/RTX 3090/A100, with large matmuls (>2^10) under 30% overhead and end-to-end training spanning ~60-67% (Llama-1B on A100) up to 374% (Llama-1B on T4), matching the claim faithfully.",
	16	"source_checked": "Extracted full text of arXiv:2502.19405 PDF via pdftotext (lines 49, 108-118, 600-820 incl. Tables 1-2: matmul <30%, Llama-1B ~60% A100 / 374% T4, DistilBERT 258%/312%); corroborated by paper abstract and gensyn-ai/repops-demo GitHub README listing T4/RTX 3090/A100 as supported targets."
	17	},
	18	{
	19	"refuted": false,
	20	"confidence": "high",
	21	"finding": "The Verde paper (arXiv:2502.19405v1) explicitly describes RepOps as making ML operators bitwise-reproducible across GPUs (T4, RTX 3090, A100 40/80GB) by fixing FP operation order, with large matmuls (>2^10) incurring <30% overhead and end-to-end training overheads spanning ~67% (Llama-1B/A100-40GB) to 374% (Llama-1B/T4), matching the claim closely.",
	22	"source_checked": "Primary source: arXiv:2502.19405v1 full HTML (Verde, Arun et al.), RepOps evaluation section and Figure 3; cross-checked against Gensyn blog and arXiv abstract. Numbers are author-reported, not independently reproduced."
	23	}
	24	]
	25	},
	26	{
	27	"i": 1,
	28	"claim": "With asynchronous GPU API remoting, GPT-2 inference suffers only ~2% performance degradati",
	29	"verdict": "CONFIRMED",
	30	"refutes": 0,
	31	"votes": [
	32	{
	33	"refuted": false,
	34	"confidence": "high",
	35	"finding": "The primary source states near-verbatim that GPT-2 sees only ~2% (contribution bullet says 2.5%) degradation at 10 us RTT / 1 Gbps because GPU execution (10-100ms) dwarfs network delay (1-100us), and uses a <5% overhead budget; under 2-5us RTT modern networking it reports a max ~6% overhead across apps with some 1-14% gains.",
	36	"source_checked": "arXiv:2401.13354 \"Characterizing Network Requirements for GPU API Remoting in AI Applications\" (Wang et al., 2024) — fetched abstract page and converted the full PDF to text, reading the abstract/contributions and the per-model requirements section."
	37	},
	38	{
	39	"refuted": false,
	40	"confidence": "high",
	41	"finding": "All three sub-claims are corroborated near-verbatim by arXiv:2401.13354 (\"Characterizing Network Requirements for GPU API Remoting in AI Applications\"): the paper states GPT-2 sees only ~2% degradation at 10us RTT / 1 Gbps, that GPU execution (10-100ms) dwarfs network delay (1-100us), and that 5-20us latency yields <5% overhead for most inference.",
	42	"source_checked": "Primary source — arXiv:2401.13354 abstract (arxiv.org/abs) and full HTML (arxiv.org/html/2401.13354); confirmed exact quotes including \"GPT-2 only experiences a 2% performance degradation under a 10 us RTT and 1 Gbps setup\" and the 5-20us / <5% abstract statement."
	43	}
	44	]
	45	},
	46	{
	47	"i": 2,
	48	"claim": "OpenDiLoCo trained a 1.1B-parameter model across four H100 clusters in three countries ove",
	49	"verdict": "CONFIRMED",
	50	"refutes": 0,
	51	"votes": [
	52	{
	53	"refuted": false,
	54	"confidence": "high",
	55	"finding": "Prime Intellect's official OpenDiLoCo blog and arXiv paper confirm all specifics: 1.1B model across four 8xH100 workers in Canada/Finland/USA (3 countries), 127-935 Mbit/s WAN, 90-95% compute utilization, and all-reduce at exactly 6.9% of training time.",
	56	"source_checked": "Primary source — Prime Intellect OpenDiLoCo blog (primeintellect.ai/blog/opendiloco), cross-checked via WebSearch against the arXiv paper (2407.07852)."
	57	},
	58	{
	59	"refuted": false,
	60	"confidence": "high",
	61	"finding": "The primary OpenDiLoCo paper states verbatim all five figures: a 1.1B model on four 8xH100 workers in Canada/Finland/two US states, 127-935 Mbit/s WAN, 90-95% compute utilization, and all-reduce accounting for 6.9% of training time.",
	62	"source_checked": "OpenDiLoCo paper, arXiv 2407.07852v1 (Prime Intellect), distributed-training experiment section; corroborated by the Prime Intellect blog post."
	63	}
	64	]
	65	},
	66	{
	67	"i": 3,
	68	"claim": "Inter-node GPU interconnect (InfiniBand + GPUDirect-RDMA) peaks at only ~12-14 GB/s and co",
	69	"verdict": "CONFIRMED",
	70	"refutes": 0,
	71	"votes": [
	72	{
	73	"refuted": false,
	74	"confidence": "high",
	75	"finding": "The qualitative thesis holds — inter-node InfiniBand is ~an order of magnitude below intra-node NVLink and tensor-parallel all-reduce needs RDMA datacenter fabric — but the specific numbers are stale EDR-era figures: current HDR is ~25 GB/s/port and NDR ~50 GB/s/port (with ~70 GB/s effective multi-HCA all-reduce per NVIDIA), and NVLink is actually 600 (A100) / 900 (H100) / 1800 (Blackwell) GB/s per GPU, not ~300.",
	76	"source_checked": "NVIDIA ConnectX-7 datasheet (NDR 400Gb/s / HDR 200Gb/s), Wikipedia NVLink (per-GPU aggregate by generation), and NVIDIA-reported ~70.3 GB/s NCCL all-reduce on multi-HCA HDR with SHARP/GPUDirect-RDMA."
	77	},
	78	{
	79	"refuted": false,
	80	"confidence": "high",
	81	"finding": "Multiple sources corroborate the claim: a single ~100Gb NIC delivers ~12 GB/s practical GPUDirect-RDMA bandwidth and NCCL inter-node all-reduce busbw measures ~15-27 GB/s on 200-400Gb IB, versus NVLink/NVSwitch ~300 GB/s/GPU unidirectional (600 GB/s A100, 900 GB/s H100 bidirectional) — roughly an order of magnitude gap — and the standard practice of placing tensor-parallel/all-reduce traffic on NVLink/datacenter fabric (TP intra-node, DP/PP inter-node) is widely documented; numbers mix hardware generations but none overstate the gap (NVLink is if anything understated at 300).",
	82	"source_checked": "NVIDIA HGX A100/H100 NVLink specs (fibermall, Introl, NVSwitch HotChips 2022), Oracle GPUDirect-RDMA IB write-bw benchmark, NVIDIA/nccl GitHub issues #489/#595/#949 on inter-node all-reduce busbw, and arXiv survey 2407.20018 / parallelism best-practice writeups."
	83	}
	84	]
	85	},
	86	{
	87	"i": 4,
	88	"claim": "io.net suffered a Sybil attack with roughly 1.8 million fake/spoofed GPUs attempting to co",
	89	"verdict": "CONTESTED",
	90	"refutes": 1,
	91	"votes": [
	92	{
	93	"refuted": false,
	94	"confidence": "medium",
	95	"finding": "io.net's April 2024 Sybil/GPU-metadata attack is well-documented, with ~1.8M fake GPUs attempting to connect and the widely-syndicated Cointelegraph report stating active GPU connections fell from 600,000 to 10,000, matching the claim.",
	96	"source_checked": "Cointelegraph \"Io.net responds to GPU metadata attack\" (states 600,000→10,000), plus io.net's own Medium incident report (25 April 2024), The Block postmortem, and Cryptopolitan; numbers are io.net/CEO-reported, not independently audited, and Cryptopolitan gives an outlier 600,000→100,000 for the drop."
	97	},
	98	{
	99	"refuted": true,
	100	"confidence": "high",
	101	"finding": "The ~1.8M fake-GPU figure and the ~600,000 starting point are corroborated, but multiple sources covering io.net CEO Ahmad Shadid's April 2024 postmortem state active GPU connections dropped from 0.6 million to 0.1 million (100,000), not to ~10,000 — the claim's endpoint is 10x too low, materially overstating the collapse.",
	102	"source_checked": "io.net post-attack postmortem coverage: cryptopolitan.com (quoted \"decreased from 0.6 million to 0.1 million\"), The Block (#291315), cryptonews.com, coincu.com, and io.net's own X postmortem; April 2024 Sybil/GPU-metadata attack."
	103	}
	104	]
	105	},
	106	{
	107	"i": 5,
	108	"claim": "Vulkan compute achieves roughly 80-95% of native vendor-API performance and is genuinely c",
	109	"verdict": "CONTESTED",
	110	"refutes": 1,
	111	"votes": [
	112	{
	113	"refuted": true,
	114	"confidence": "high",
	115	"finding": "The 16-devices/8-vendors, ~10x-prefill and ~2.5x-decode figures match the LlamaWeb paper exactly, but the named GPU is wrong (it is an RTX 5080, not a 5090) and the precise \"~11-12% of CUDA throughput\" figure does not appear in the source — the paper reports slowdown multiples, not that percentage — so a load-bearing specific is materially overstated/fabricated.",
	116	"source_checked": "Primary source: arXiv:2605.20706v1 \"Llamas on the Web\" (LlamaWeb) full HTML, Section 6.2 + device table; plus secondary search results for the Vulkan 80-95% general claim (uncorroborated by a strong independent source)."
	117	},
	118	{
	119	"refuted": false,
	120	"confidence": "high",
	121	"finding": "Primary source (arXiv 2605.20706, \"Llamas on the Web\") confirms 16 devices/8 vendors and that CUDA/Vulkan outperform WebGPU \"by up to 10x during prefill and 2.5x during decode,\" and TechnoLynx confirms the verbatim Vulkan \"80-95% of native, genuinely cross-vendor\" claim; the only errors are the GPU model (paper uses RTX 5080, not 5090) and the \"~11-12% of CUDA throughput\" figure, which is an inference not stated in the paper, neither of which inverts the claim's substance.",
	122	"source_checked": "arXiv:2605.20706 \"Llamas on the Web\" (HTML full text, incl. Section 6.2 and device Table 4) for the LlamaWeb/WebGPU numbers; TechnoLynx GPU-compute API comparison article for the Vulkan 80-95% native cross-vendor claim."
	123	}
	124	]
	125	},
	126	{
	127	"i": 6,
	128	"claim": "Gensyn Verde verifies untrusted ML computation via refereed delegation: on disagreement a ",
	129	"verdict": "CONFIRMED",
	130	"refutes": 0,
	131	"votes": [
	132	{
	133	"refuted": false,
	134	"confidence": "high",
	135	"finding": "The primary source (arXiv:2502.19405, \"Verde: Verification via Refereed Delegation for Machine Learning Programs,\" Arun, Bonneau, Gensyn et al.) confirms all elements: refereed delegation for untrusted ML, a bisection game locating the first-divergent operator, the referee recomputing only that single operator, and a stated ~two-orders-of-magnitude reduction in referee compute/communication versus re-running the whole task.",
	136	"source_checked": "arXiv abstract page 2502.19405 (title/authors/abstract verified) plus corroborating WebSearch results from gensyn.ai and the arXiv PDF; note the 100x figure is author/vendor-reported for the dispute-resolution step, not an independent benchmark, but it is the claim the paper actually makes and is not materially overstated."
	137	},
	138	{
	139	"refuted": false,
	140	"confidence": "high",
	141	"finding": "The Verde paper (arXiv:2502.19405) explicitly states the referee re-executes only \"a single operator in the computational graph...performed with two orders of magnitude less compute resources than it takes to run the model itself,\" confirming both the single-first-divergent-operator and ~100x claims; the only minor imprecision is \"bisection\" — the paper presents binary search as the canonical method but footnote 2 says their own protocol eschews it (N<100 checkpoints sent in one round), which is not material.",
	142	"source_checked": "Primary source: full PDF of arXiv:2502.19405 (extracted via pdftotext), verified against the abstract page; quoted intro/Section 2 text on two-phase narrowing to a single operator and the explicit \"two orders of magnitude\" compute reduction."
	143	}
	144	]
	145	},
	146	{
	147	"i": 7,
	148	"claim": "GPU computation is inherently non-deterministic (atomic operations, floating-point reducti",
	149	"verdict": "REFUTED",
	150	"refutes": 2,
	151	"votes": [
	152	{
	153	"refuted": true,
	154	"confidence": "high",
	155	"finding": "The mechanisms cited (atomics, FP reduction non-associativity, warp scheduling) are real and some cuDNN routines are non-reproducible even on identical hardware, but the categorical claim that GPU computation is \"inherently non-deterministic\" and reproducibility \"fails\" with versions controlled is materially overstated: NVIDIA's own cuDNN docs state most routines produce bit-wise identical results across runs on the same architecture, and bit-exact reproducibility IS achievable by selecting deterministic algorithms (at a ~2x perf cost), so non-determinism is an opt-out default for specific ops, not an unavoidable property.",
	156	"source_checked": "NVIDIA cuDNN backend docs (docs.nvidia.com/deeplearning/cudnn/backend/latest/developer/misc.html) directly fetched; corroborated by Hawkeye arXiv 2603.20421 and FP non-associativity arXiv 2408.05148."
	157	},
	158	{
	159	"refuted": true,
	160	"confidence": "high",
	161	"finding": "The non-deterministic sources cited (atomic ops, FP non-associativity) are real, but the claim that bit-exact reproducibility \"fails\" once architecture/driver/CUDA/cuDNN are controlled is materially overstated: NVIDIA's own cuDNN docs say MOST routines DO produce identical bit-wise results across runs on the same architecture, with non-determinism confined to a small enumerated set of atomicAdd-based fast paths that can be disabled (e.g. torch.use_deterministic_algorithms(True)), so it is opt-in for speed and avoidable, not inherent.",
	162	"source_checked": "NVIDIA cuDNN Backend developer docs (\"Odds and Ends\" / Reproducibility section, docs.nvidia.com) and PyTorch Reproducibility + torch.use_deterministic_algorithms documentation"
	163	}
	164	]
	165	}
	166	]
	167	}

test/e2e.js +78 -0

…	…	@@ -0,0 +1,78 @@
	1	// test/e2e.js — P0 proof: the spec's core claim, no network.
	2	//
	3	// Two ComputeNodes share one optimistic Autobase (compute pool). The owner
	4	// submits a content-addressed task; the worker executes it (T0 native); the
	5	// result converges on BOTH peers via the deterministic fold; and a re-submit
	6	// is a memo hit. Sync is driven by replicateAndSync (SPEC §P0).
	7
	8	const test = require('brittle')
	9	const Corestore = require('corestore')
	10	const ram = require('random-access-memory')
	11	const { replicateAndSync } = require('autobase-test-helpers')
	12	const b4a = require('b4a')
	13	const crypto = require('hypercore-crypto')
	14	const { ComputeNode } = require('../lib/node')
	15
	16	function inputHashOf (input) {
	17	return b4a.toString(crypto.hash(b4a.from(JSON.stringify(input))), 'hex')
	18	}
	19
	20	test('P0: submit -> execute -> result converges + memoizes', async (t) => {
	21	// a DETERMINISTIC CPU task (square). codeHash stands in for the content hash
	22	// of the executable artifact (SPEC §4.1).
	23	const codeHash = 'square-v1'
	24	const square = async (input) => input.n * input.n
	25	const executors = { [codeHash]: square }
	26
	27	const owner = await ComputeNode.create(new Corestore(ram), null, { executors })
	28	await owner.grantSelf('owner')
	29
	30	const worker = await ComputeNode.create(new Corestore(ram), owner.key, { executors })
	31
	32	// P0 membership: owner adds the worker as an autobase writer + grants it.
	33	await owner.addWriter(worker.writerKey)
	34	await owner.grant(worker.identity.publicKey, 'worker')
	35	await replicateAndSync([owner.base, worker.base])
	36
	37	t.ok(worker.base.writable, 'worker became a writer after add-writer synced')
	38
	39	const input = { n: 7 }
	40	const ih = inputHashOf(input)
	41
	42	// first submit = memo MISS -> a task op
	43	const sub = await owner.submit(codeHash, ih, input)
	44	t.is(sub.memoHit, false, 'first submit is a memo miss')
	45	await replicateAndSync([owner.base, worker.base])
	46
	47	// worker sees the task via the fold, executes it (T0), appends the result
	48	const n = await worker.runPendingTasks()
	49	t.is(n, 1, 'worker executed exactly one pending task')
	50	await replicateAndSync([owner.base, worker.base])
	51
	52	// result converged on BOTH peers (the deterministic fold = shared truth)
	53	t.is(await owner.lookup(codeHash, ih), 49, 'owner sees result via the fold')
	54	t.is(await worker.lookup(codeHash, ih), 49, 'worker sees result via the fold')
	55
	56	// re-submit = memo HIT (compute once, reuse everywhere — SPEC §4.1)
	57	const again = await owner.submit(codeHash, ih, input)
	58	t.is(again.memoHit, true, 're-submit is a memo hit')
	59	t.is(again.output, 49, 'memoized output returned without recompute')
	60
	61	await owner.close()
	62	await worker.close()
	63	})
	64
	65	test('P0: a different input is a separate content-addressed chunk', async (t) => {
	66	const codeHash = 'square-v1'
	67	const executors = { [codeHash]: async (i) => i.n * i.n }
	68
	69	const owner = await ComputeNode.create(new Corestore(ram), null, { executors })
	70	await owner.grantSelf('owner')
	71
	72	const ih = inputHashOf({ n: 9 })
	73	t.is((await owner.submit(codeHash, ih, { n: 9 })).memoHit, false, 'n=9 is a fresh task')
	74	await owner.runPendingTasks()
	75	t.is(await owner.lookup(codeHash, ih), 81, 'n=9 -> 81 computed and memoized')
	76
	77	await owner.close()
	78	})

hypercompute commits / ec7b8ae

Changed files (17) +2591 -0