{
"summary": "Adversarially verify the salvaged GPU-research claims (2-vote refute-biased)",
"agentCount": 16,
"logs": [],
"result": [
{
"i": 0,
"claim": "Gensyn RepOps makes GPU ML operators BITWISE-reproducible across different GPU hardware (e",
"verdict": "CONFIRMED",
"refutes": 0,
"votes": [
{
"refuted": false,
"confidence": "high",
"finding": "The Verde paper (arXiv:2502.19405) PDF directly states RepOps gives bitwise-reproducible operators by fixing floating-point op order, tested on T4/RTX 3090/A100, with large matmuls (>2^10) under 30% overhead and end-to-end training spanning ~60-67% (Llama-1B on A100) up to 374% (Llama-1B on T4), matching the claim faithfully.",
"source_checked": "Extracted full text of arXiv:2502.19405 PDF via pdftotext (lines 49, 108-118, 600-820 incl. Tables 1-2: matmul <30%, Llama-1B ~60% A100 / 374% T4, DistilBERT 258%/312%); corroborated by paper abstract and gensyn-ai/repops-demo GitHub README listing T4/RTX 3090/A100 as supported targets."
},
{
"refuted": false,
"confidence": "high",
"finding": "The Verde paper (arXiv:2502.19405v1) explicitly describes RepOps as making ML operators bitwise-reproducible across GPUs (T4, RTX 3090, A100 40/80GB) by fixing FP operation order, with large matmuls (>2^10) incurring <30% overhead and end-to-end training overheads spanning ~67% (Llama-1B/A100-40GB) to 374% (Llama-1B/T4), matching the claim closely.",
"source_checked": "Primary source: arXiv:2502.19405v1 full HTML (Verde, Arun et al.), RepOps evaluation section and Figure 3; cross-checked against Gensyn blog and arXiv abstract. Numbers are author-reported, not independently reproduced."
}
]
},
{
"i": 1,
"claim": "With asynchronous GPU API remoting, GPT-2 inference suffers only ~2% performance degradati",
"verdict": "CONFIRMED",
"refutes": 0,
"votes": [
{
"refuted": false,
"confidence": "high",
"finding": "The primary source states near-verbatim that GPT-2 sees only ~2% (contribution bullet says 2.5%) degradation at 10 us RTT / 1 Gbps because GPU execution (10-100ms) dwarfs network delay (1-100us), and uses a <5% overhead budget; under 2-5us RTT modern networking it reports a max ~6% overhead across apps with some 1-14% gains.",
"source_checked": "arXiv:2401.13354 \"Characterizing Network Requirements for GPU API Remoting in AI Applications\" (Wang et al., 2024) — fetched abstract page and converted the full PDF to text, reading the abstract/contributions and the per-model requirements section."
},
{
"refuted": false,
"confidence": "high",
"finding": "All three sub-claims are corroborated near-verbatim by arXiv:2401.13354 (\"Characterizing Network Requirements for GPU API Remoting in AI Applications\"): the paper states GPT-2 sees only ~2% degradation at 10us RTT / 1 Gbps, that GPU execution (10-100ms) dwarfs network delay (1-100us), and that 5-20us latency yields <5% overhead for most inference.",
"source_checked": "Primary source — arXiv:2401.13354 abstract (arxiv.org/abs) and full HTML (arxiv.org/html/2401.13354); confirmed exact quotes including \"GPT-2 only experiences a 2% performance degradation under a 10 us RTT and 1 Gbps setup\" and the 5-20us / <5% abstract statement."
}
]
},
{
"i": 2,
"claim": "OpenDiLoCo trained a 1.1B-parameter model across four H100 clusters in three countries ove",
"verdict": "CONFIRMED",
"refutes": 0,
"votes": [
{
"refuted": false,
"confidence": "high",
"finding": "Prime Intellect's official OpenDiLoCo blog and arXiv paper confirm all specifics: 1.1B model across four 8xH100 workers in Canada/Finland/USA (3 countries), 127-935 Mbit/s WAN, 90-95% compute utilization, and all-reduce at exactly 6.9% of training time.",
"source_checked": "Primary source — Prime Intellect OpenDiLoCo blog (primeintellect.ai/blog/opendiloco), cross-checked via WebSearch against the arXiv paper (2407.07852)."
},
{
"refuted": false,
"confidence": "high",
"finding": "The primary OpenDiLoCo paper states verbatim all five figures: a 1.1B model on four 8xH100 workers in Canada/Finland/two US states, 127-935 Mbit/s WAN, 90-95% compute utilization, and all-reduce accounting for 6.9% of training time.",
"source_checked": "OpenDiLoCo paper, arXiv 2407.07852v1 (Prime Intellect), distributed-training experiment section; corroborated by the Prime Intellect blog post."
}
]
},
{
"i": 3,
"claim": "Inter-node GPU interconnect (InfiniBand + GPUDirect-RDMA) peaks at only ~12-14 GB/s and co",
"verdict": "CONFIRMED",
"refutes": 0,
"votes": [
{
"refuted": false,
"confidence": "high",
"finding": "The qualitative thesis holds — inter-node InfiniBand is ~an order of magnitude below intra-node NVLink and tensor-parallel all-reduce needs RDMA datacenter fabric — but the specific numbers are stale EDR-era figures: current HDR is ~25 GB/s/port and NDR ~50 GB/s/port (with ~70 GB/s effective multi-HCA all-reduce per NVIDIA), and NVLink is actually 600 (A100) / 900 (H100) / 1800 (Blackwell) GB/s per GPU, not ~300.",
"source_checked": "NVIDIA ConnectX-7 datasheet (NDR 400Gb/s / HDR 200Gb/s), Wikipedia NVLink (per-GPU aggregate by generation), and NVIDIA-reported ~70.3 GB/s NCCL all-reduce on multi-HCA HDR with SHARP/GPUDirect-RDMA."
},
{
"refuted": false,
"confidence": "high",
"finding": "Multiple sources corroborate the claim: a single ~100Gb NIC delivers ~12 GB/s practical GPUDirect-RDMA bandwidth and NCCL inter-node all-reduce busbw measures ~15-27 GB/s on 200-400Gb IB, versus NVLink/NVSwitch ~300 GB/s/GPU unidirectional (600 GB/s A100, 900 GB/s H100 bidirectional) — roughly an order of magnitude gap — and the standard practice of placing tensor-parallel/all-reduce traffic on NVLink/datacenter fabric (TP intra-node, DP/PP inter-node) is widely documented; numbers mix hardware generations but none overstate the gap (NVLink is if anything understated at 300).",
"source_checked": "NVIDIA HGX A100/H100 NVLink specs (fibermall, Introl, NVSwitch HotChips 2022), Oracle GPUDirect-RDMA IB write-bw benchmark, NVIDIA/nccl GitHub issues #489/#595/#949 on inter-node all-reduce busbw, and arXiv survey 2407.20018 / parallelism best-practice writeups."
}
]
},
{
"i": 4,
"claim": "io.net suffered a Sybil attack with roughly 1.8 million fake/spoofed GPUs attempting to co",
"verdict": "CONTESTED",
"refutes": 1,
"votes": [
{
"refuted": false,
"confidence": "medium",
"finding": "io.net's April 2024 Sybil/GPU-metadata attack is well-documented, with ~1.8M fake GPUs attempting to connect and the widely-syndicated Cointelegraph report stating active GPU connections fell from 600,000 to 10,000, matching the claim.",
"source_checked": "Cointelegraph \"Io.net responds to GPU metadata attack\" (states 600,000→10,000), plus io.net's own Medium incident report (25 April 2024), The Block postmortem, and Cryptopolitan; numbers are io.net/CEO-reported, not independently audited, and Cryptopolitan gives an outlier 600,000→100,000 for the drop."
},
{
"refuted": true,
"confidence": "high",
"finding": "The ~1.8M fake-GPU figure and the ~600,000 starting point are corroborated, but multiple sources covering io.net CEO Ahmad Shadid's April 2024 postmortem state active GPU connections dropped from 0.6 million to 0.1 million (100,000), not to ~10,000 — the claim's endpoint is 10x too low, materially overstating the collapse.",
"source_checked": "io.net post-attack postmortem coverage: cryptopolitan.com (quoted \"decreased from 0.6 million to 0.1 million\"), The Block (#291315), cryptonews.com, coincu.com, and io.net's own X postmortem; April 2024 Sybil/GPU-metadata attack."
}
]
},
{
"i": 5,
"claim": "Vulkan compute achieves roughly 80-95% of native vendor-API performance and is genuinely c",
"verdict": "CONTESTED",
"refutes": 1,
"votes": [
{
"refuted": true,
"confidence": "high",
"finding": "The 16-devices/8-vendors, ~10x-prefill and ~2.5x-decode figures match the LlamaWeb paper exactly, but the named GPU is wrong (it is an RTX 5080, not a 5090) and the precise \"~11-12% of CUDA throughput\" figure does not appear in the source — the paper reports slowdown multiples, not that percentage — so a load-bearing specific is materially overstated/fabricated.",
"source_checked": "Primary source: arXiv:2605.20706v1 \"Llamas on the Web\" (LlamaWeb) full HTML, Section 6.2 + device table; plus secondary search results for the Vulkan 80-95% general claim (uncorroborated by a strong independent source)."
},
{
"refuted": false,
"confidence": "high",
"finding": "Primary source (arXiv 2605.20706, \"Llamas on the Web\") confirms 16 devices/8 vendors and that CUDA/Vulkan outperform WebGPU \"by up to 10x during prefill and 2.5x during decode,\" and TechnoLynx confirms the verbatim Vulkan \"80-95% of native, genuinely cross-vendor\" claim; the only errors are the GPU model (paper uses RTX 5080, not 5090) and the \"~11-12% of CUDA throughput\" figure, which is an inference not stated in the paper, neither of which inverts the claim's substance.",
"source_checked": "arXiv:2605.20706 \"Llamas on the Web\" (HTML full text, incl. Section 6.2 and device Table 4) for the LlamaWeb/WebGPU numbers; TechnoLynx GPU-compute API comparison article for the Vulkan 80-95% native cross-vendor claim."
}
]
},
{
"i": 6,
"claim": "Gensyn Verde verifies untrusted ML computation via refereed delegation: on disagreement a ",
"verdict": "CONFIRMED",
"refutes": 0,
"votes": [
{
"refuted": false,
"confidence": "high",
"finding": "The primary source (arXiv:2502.19405, \"Verde: Verification via Refereed Delegation for Machine Learning Programs,\" Arun, Bonneau, Gensyn et al.) confirms all elements: refereed delegation for untrusted ML, a bisection game locating the first-divergent operator, the referee recomputing only that single operator, and a stated ~two-orders-of-magnitude reduction in referee compute/communication versus re-running the whole task.",
"source_checked": "arXiv abstract page 2502.19405 (title/authors/abstract verified) plus corroborating WebSearch results from gensyn.ai and the arXiv PDF; note the 100x figure is author/vendor-reported for the dispute-resolution step, not an independent benchmark, but it is the claim the paper actually makes and is not materially overstated."
},
{
"refuted": false,
"confidence": "high",
"finding": "The Verde paper (arXiv:2502.19405) explicitly states the referee re-executes only \"a single operator in the computational graph...performed with two orders of magnitude less compute resources than it takes to run the model itself,\" confirming both the single-first-divergent-operator and ~100x claims; the only minor imprecision is \"bisection\" — the paper presents binary search as the canonical method but footnote 2 says their own protocol eschews it (N<100 checkpoints sent in one round), which is not material.",
"source_checked": "Primary source: full PDF of arXiv:2502.19405 (extracted via pdftotext), verified against the abstract page; quoted intro/Section 2 text on two-phase narrowing to a single operator and the explicit \"two orders of magnitude\" compute reduction."
}
]
},
{
"i": 7,
"claim": "GPU computation is inherently non-deterministic (atomic operations, floating-point reducti",
"verdict": "REFUTED",
"refutes": 2,
"votes": [
{
"refuted": true,
"confidence": "high",
"finding": "The mechanisms cited (atomics, FP reduction non-associativity, warp scheduling) are real and some cuDNN routines are non-reproducible even on identical hardware, but the categorical claim that GPU computation is \"inherently non-deterministic\" and reproducibility \"fails\" with versions controlled is materially overstated: NVIDIA's own cuDNN docs state most routines produce bit-wise identical results across runs on the same architecture, and bit-exact reproducibility IS achievable by selecting deterministic algorithms (at a ~2x perf cost), so non-determinism is an opt-out default for specific ops, not an unavoidable property.",
"source_checked": "NVIDIA cuDNN backend docs (docs.nvidia.com/deeplearning/cudnn/backend/latest/developer/misc.html) directly fetched; corroborated by Hawkeye arXiv 2603.20421 and FP non-associativity arXiv 2408.05148."
},
{
"refuted": true,
"confidence": "high",
"finding": "The non-deterministic sources cited (atomic ops, FP non-associativity) are real, but the claim that bit-exact reproducibility \"fails\" once architecture/driver/CUDA/cuDNN are controlled is materially overstated: NVIDIA's own cuDNN docs say MOST routines DO produce identical bit-wise results across runs on the same architecture, with non-determinism confined to a small enumerated set of atomicAdd-based fast paths that can be disabled (e.g. torch.use_deterministic_algorithms(True)), so it is opt-in for speed and avoidable, not inherent.",
"source_checked": "NVIDIA cuDNN Backend developer docs (\"Odds and Ends\" / Reproducibility section, docs.nvidia.com) and PyTorch Reproducibility + torch.use_deterministic_algorithms documentation"
}
]
}
]
}