========== RESULT 1 ==========
question: Research the state of the art (2023-2026) in distributed GPU usage — using GPUs on other machines, pooling/sharing GPUs across a network, and making GPU compute portable and deterministic — to inform a P2P system where a peer could transparently use GPUs elsewhere in a swarm. Prioritize mechanics, real latency/bandwidth numbers, maturity (production vs research vs hype), and feasibility over commodity LAN/WAN vs datacenter interconnect.
summary: The question spans 7 layers from low-level GPU API remoting up to decentralized GPU markets and P2P feasibility. I split into five non-overlapping angles: (1) GPU-over-IP/API-remoting mechanics and per-call overhead; (2) the interconnect bandwidth wall (NVLink/InfiniBand/NCCL) and the datacenter-vs-Ethernet/WAN cliff that bounds tensor-parallel feasibility; (3) cross-vendor portable compute (Vulkan/WebGPU/SYCL) for heterogeneous swarms; (4) GPU determinism/bit-exact reproducibility, which decides whether results can be memoized/verified by re-execution; and (5) the skeptical reality of decentralized GPU markets plus pooling/disaggregation. Each query targets primary sources and hard numbers (us-per-call, GB/s, overhead %, tokens/sec).
angles: [{"label": "GPU-over-IP / API remoting mechanics + overhead", "query": "rCUDA scuda Juice Labs Thundercompute CUDA API remoting latency overhead per-call benchmark GPU-over-IP", "rationale": "Primary-source mechanics and real per-call latency/bandwidth numbers for driver-call interception+forwarding (sub-topic 1), plus honest maturity of each named system."}, {"label": "Interconnect bandwidth wall (datacenter vs Ethernet/WAN cliff)", "query": "NCCL all-reduce NVLink InfiniBand GPUDirect RDMA bandwidth latency tensor parallelism Ethernet vs datacenter fabric benchmark", "rationale": "Quantifies why cross-GPU compute is interconnect-bound (sub-topics 2-3) and the concrete cliff between 100s-of-GB/s fabric and commodity Ethernet/WAN that kills fine-grained distribution."}, {"label": "Cross-vendor portable GPU compute for heterogeneous hardware", "query": "WebGPU WGSL wgpu Vulkan compute SPIR-V SYCL portable GPU kernel cross-vendor NVIDIA AMD Apple performance cost portability", "rationale": "Covers sub-topic 4: which APIs give a single portable kernel across mixed GPUs in a swarm and the measured performance penalty of portability."}, {"label": "GPU determinism / bit-exact reproducibility across hardware", "query": "GPU floating point nondeterminism reproducibility deterministic cuDNN cuBLAS PyTorch atomic reduction order bit-exact cross-GPU verification", "rationale": "Sub-topic 5 is decisive: whether distributed GPU results can be verified by re-execution or only trusted/stati
========== RESULT 2 ==========
results: [{"url": "https://arxiv.org/pdf/2502.19405", "title": "Verde: Verification via Refereed Delegation for Machine Learning Programs (+ RepOps)", "relevance": "high", "snippet": "THE decisive source for sub-topic 5 mapped to P2P. Gensyn's Verde shows that ML compute over UNTRUSTED nodes cannot be verified by naive re-execution because results are not bitwise-reproducible across hardware -- even A100 vs H100 from the same vendor differ, since IEEE-754 FP ops are non-associative and execution order varies. Their fix, RepOps, ENFORCES a fixed FP operation order (e.g. in matmul reductions) so honest providers yield bitwise-identical output, enabling a refereed-delegation bisection game where a referee re-runs a SINGLE op rather than the whole task. This is exactly the 'can distributed GPU results be verified by re-execution or only trusted' question, with a concrete mechanism and the honest verdict: only feasible if you constrain kernels to a canonical order at a real performance cost."}, {"url": "https://docs.pytorch.org/docs/stable/notes/randomness.html", "title": "Reproducibility \u2014 PyTorch documentation", "relevance": "high", "snippet": "Primary-source enumeration of the actual nondeterminism knobs: torch.use_deterministic_algorithms(True), torch.backends.cudnn.deterministic=True, and the CUBLAS_WORKSPACE_CONFIG env var required for cuBLAS determinism (CUDA >=10.2). Documents that some cuDNN/atomicAdd-based ops have no deterministic implementation and will throw. Establishes
========== RESULT 3 ==========
results: [{"url": "https://arxiv.org/html/2605.20706v1", "title": "Llamas on the Web: Memory-Efficient, Performance-Portable, Multi-Precision LLM Inference with WebGPU", "relevance": "high", "snippet": "Primary-source benchmark quantifying the COST OF PORTABILITY directly relevant to a heterogeneous swarm. Concrete numbers: WebGPU per-dispatch API overhead 24-36 us on Vulkan, 32-71 us on Metal (~95 us total per-op with framework); on RTX 5090/Dawn/Vulkan WebGPU reaches only 11-12% of CUDA throughput for LLM inference, and dispatch overhead (not compute) is the dominant differentiator -- kernel fusion (876->564 dispatches) gives +53% while CUDA fusion gives nothing. This is the honest portability tax for a single WGSL kernel running across NVIDIA/AMD/Apple."}, {"url": "https://arxiv.org/abs/2309.09609", "title": "Comparing Performance and Portability between CUDA and SYCL for Protein Database Search on NVIDIA, AMD, and Intel GPUs", "relevance": "high", "snippet": "Peer-reviewed cross-vendor performance-portability study: SYCL matches or exceeds CUDA on NVIDIA A100 across six workloads (negligible diff elsewhere) and is comparable to HIP on AMD MI100 -- i.e. a single SYCL source can run across NVIDIA/AMD/Intel at near-native speed. Quantifies the central claim that portable kernels need not pay a large performance penalty on datacenter GPUs, while noting Intel Xe efficiency drops to ~23% due to immature drivers. Directly answers 'which APIs give one portable kernel across mixed GPUs and
========== RESULT 4 ==========
results: [{"url": "https://arxiv.org/pdf/2401.13354", "title": "Characterizing Network Requirements for GPU API Remoting in AI Applications (arXiv 2401.13354)", "relevance": "high", "snippet": "PRIMARY-SOURCE benchmark paper, the single best source for this angle. Quantifies API-remoting overhead: <5% achievable, but ONLY under datacenter networking (200 Gbps RDMA, 2-5 us RTT). Key mechanic: a single RDMA MMIO posts in 400-800 cycles, so remoting can avoid batching; over commodity TCP/Ethernet the per-call round-trip dominates and latency-sensitive workloads collapse. Directly answers 'per-call latency, bandwidth needed, which workloads tolerate it' and the LAN/WAN-vs-datacenter cliff central to the P2P mapping."}, {"url": "https://github.com/kevmo314/scuda", "title": "kevmo314/scuda \u2014 GPU over IP bridge (project repo + HN discussion)", "relevance": "high", "snippet": "Primary repo for scuda, the modern open-source GPU-over-IP bridge. Mechanic: codegen reads CUDA dependency headers to auto-generate RPC stubs; intercepts PyTorch CUDA calls and routes them over TCP to a remote GPU, attaching a remote GPU to a CPU-only host. The linked Hacker News thread (news.ycombinator.com/item?id=41787547) gives the honest maturity/physics verdict: practitioners note performance 'will be terrible even with 100/400gb ethernet' for chatty workloads \u2014 exactly the per-call-overhead reality a P2P/WAN design must reckon with."}, {"url": "https://medium.com/@carl_56793/how-thunder-compute-works-gp
========== RESULT 5 ==========
results: [{"url": "https://epoch.ai/gradient-updates/how-far-can-decentralized-training-over-the-internet-scale", "title": "How far can decentralized training over the internet scale? \u2014 Epoch AI", "relevance": "high", "snippet": "Directly on-point for the P2P/WAN throughline. Quantifies the bandwidth cliff: cross-continental WAN links efficiently utilize only ~45 Gbit/s across Europe and ~25 Gbit/s across NA+EU \u2014 orders of magnitude below NVLink's ~900 GB/s intra-node fabric. Explains WHY commodity internet links force communication-avoiding schemes (DiLoCo-style) rather than tensor-parallel/all-reduce, and where the physics ceiling sits for swarm-scale GPU pooling. Best single source on the datacenter-vs-WAN cliff verdict."}, {"url": "https://www.primeintellect.ai/blog/opendiloco", "title": "OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training \u2014 Prime Intellect", "relevance": "high", "snippet": "Primary source showing the only form of distributed GPU training that survives the bandwidth wall over commodity links. Workers sync every ~500 inner steps (minutes apart, not every few hundred ms like data/tensor parallelism), with int8 all-reduce giving ~400x bandwidth reduction. Demonstrated across 2 continents / 3 countries at 90-95% compute utilization. Defines the production-vs-research line for what a P2P swarm can realistically do: coarse data-parallel, NOT fine-grained collective ops."}, {"url": "https://arxiv.org/pdf/1903.04611", "t
========== RESULT 6 ==========
claims: [{"claim": "Vulkan compute achieves roughly 80-95% of the vendor's native compute API performance for typical bandwidth-bound and compute-bound kernels.", "quote": "80-95% of the vendor's native compute API for typical bandwidth-bound and compute-bound kernels", "importance": "central"}, {"claim": "Vulkan is genuinely cross-vendor with drivers for NVIDIA, AMD, Intel, Apple Silicon, and mobile GPUs, with no API-level lock-in.", "quote": "Vulkan is genuinely cross-vendor \u2014 drivers exist for NVIDIA, AMD, Intel, Apple Silicon, mobile GPUs", "importance": "central"}, {"claim": "OpenCL is in decline for ML/AI: cross-vendor but retreating from the ML ecosystem and not keeping pace with post-2020 hardware features.", "quote": "cross-vendor but is in a slow retreat from the ML ecosystem ... The standard has not kept pace with hardware features introduced after roughly 2020", "importance": "supporting"}, {"claim": "SYCL is the recommended path for a single codebase running across AMD, Intel, and NVIDIA, relying on vendor-specific back-ends.", "quote": "when the workload must run on AMD, Intel, and NVIDIA from a single codebase ... across vendors with vendor-specific back-ends doing the heavy lifting", "importance": "supporting"}, {"claim": "CUDA is not portable and is exclusive to NVIDIA hardware.", "quote": "Not portable\" and \"NVIDIA exclusivity", "importance": "supporting"}]
publishDate: 2026-01-08
sourceQuality: blog
========== RESULT 7 ==========
claims: [{"claim": "The same floating-point compute kernel produces different results on Vulkan/SPIR-V vs OpenCL on the same NVIDIA desktop GPU (RTX 3090), while OpenCL and CPU backends matched each other.", "quote": "CL (desktop) = CPU (desktop) = CL (mobile) = CPU (mobile) = VK (mobile); VK (desktop) Different!", "importance": "central"}, {"claim": "The Vulkan/NVIDIA-desktop divergence accumulates into a meaningful numerical error (~0.2 difference), not just last-bit noise.", "quote": "Vulkan results diverged on NVIDIA desktop hardware, accumulating errors with \"~0.2 difference\"", "importance": "central"}, {"claim": "Standard SPIR-V floating-point control mechanisms (NoContraction, RoundingModeRTE, SignedZeroInfNanPreserve, FloatControl2, FPFastMathDefault) failed to make NVIDIA Vulkan results match the other backends.", "quote": "None of this worked, they didn't even change the result.", "importance": "central"}, {"claim": "The poster suspected NVIDIA's Vulkan driver flushes float32 denormals to zero with no available way to preserve them, though this was not confirmed as the sole cause.", "quote": "the different stands in Vulkan on NV driver that is flushing float32 denorms to zero. Unfortunately there seems to be no way to preserve denorms on NV hardware", "importance": "supporting"}, {"claim": "The same three backends (OpenCL, CPU, Vulkan) produced identical results on Qualcomm Adreno 830 mobile hardware, indicating the discrepancy is hardware/driver-specific rather than inhe
publishDate: 2026-03-14
sourceQuality: forum
========== RESULT 8 ==========
results: [{"url": "https://arxiv.org/html/2502.19405v1", "title": "Verde: Verification via Refereed Delegation for Machine Learning Programs (Gensyn paper)", "relevance": "high", "snippet": "Primary source for the ONLY decentralized network with a real cryptographic-style verification design. Verde verifies ML training over untrusted nodes via refereed-delegation: trainers store/hash intermediate checkpoints, a verifier re-runs; on disagreement a referee (smart contract or jury) recomputes only the single first-divergent operator in the compute graph, slashing verification overhead vs full re-execution. Honest if >=1 verifier is honest. Crucially depends on RepOps (bitwise-reproducible operators enforcing fixed FP reduction order) to defeat hardware non-determinism \u2014 directly answers sub-topic 5 (determinism is the precondition for verification-by-re-execution) and 6 (real verification mechanics). This is the load-bearing primary source for the skeptical verdict: verification is feasible but only because they re-engineered determinism into the kernels."}, {"url": "https://blog.gensyn.ai/verde-verification-system-in-production/", "title": "Verde Verification System In Production \u2014 Gensyn blog", "relevance": "high", "snippet": "Vendor primary source describing Verde's production deployment and RepOps (bitwise-reproducible ML operators that fix floating-point execution order so honest providers yield bit-identical outputs). Establishes maturity: proof-of-learning/refereed-deleg
========== RESULT 9 ==========
claims: [{"claim": "gpu.cpp uses the WebGPU specification as a portable low-level GPU interface, enabling the same C++ GPU code to run across NVIDIA, Intel, AMD, and other GPUs via Vulkan, Metal, or DirectX backends.", "quote": "leveraging the WebGPU specification as a portable low-level GPU interface ... we can drop in GPU code in C++ projects and have it run on Nvidia, Intel, AMD, and other GPUs ... The same C++ code can work on a wide variety of laptops, workstations, mobile devices or virtually any hardware with Vulkan, Metal, or DirectX support.", "importance": "central"}, {"claim": "gpu.cpp's WebGPU implementation is built on Google's Dawn native backend, with plans to also support browser WebGPU and wgpu as alternative implementations.", "quote": "support other targets and WebGPU implementations (web browsers or other native implementations such as wgpu)", "importance": "supporting"}, {"claim": "A portable WebGPU matrix-multiplication kernel achieves roughly 1.2 TFLOPs on an Apple M1 MacBook Pro, against a theoretical peak of 10.4 TFLOPs (about 12% of peak), illustrating the performance cost of portability.", "quote": "an estimated ~ 1.2+ TFLOPs on a Macbook Pro M1 laptop, which has a theoretical peak of 10.4 TFLOPs", "importance": "central"}, {"claim": "The core gpu.cpp library is a header-only implementation of around 1000 lines of code, with compile/run cycles under 5 seconds, demonstrating WebGPU compute can be embedded with minimal footprint.", "quote": "The core library
publishDate: 2024-07-13
sourceQuality: primary
========== RESULT 10 ==========
claims: [{"claim": "SYCL and CUDA achieve similar performance on NVIDIA GPUs for a Smith-Waterman protein database search workload, meaning the portability layer (SYCL) does not impose a meaningful slowdown vs native CUDA on NVIDIA hardware.", "quote": "while both CUDA and SYCL versions achieve similar performance on NVIDIA devices, the latter demonstrated remarkable code portability to other GPU architectures, such as AMD and Intel.", "importance": "central"}, {"claim": "A single SYCL codebase is portable across NVIDIA, AMD, and Intel GPUs (single and multi-GPU configs), whereas CUDA is NVIDIA-specific \u2014 directly relevant to running portable kernels across a heterogeneous swarm of mixed-vendor GPUs.", "quote": "the latter demonstrated remarkable code portability to other GPU architectures, such as AMD and Intel", "importance": "central"}, {"claim": "SYCL did not just match but exceeded architectural efficiency in 3 of 4 non-NVIDIA cases tested, indicating portability did not always come at a performance cost on AMD/Intel hardware.", "quote": "the architectural efficiency rates achieved on these devices were superior in 3 of the 4 cases tested", "importance": "supporting"}, {"claim": "The study is empirical and peer-reviewed, evaluating real GPUs from three vendors (NVIDIA, Intel, AMD) in both single- and multi-GPU configurations, lending production-relevant evidence for cross-vendor portable compute.", "quote": "This study evaluates the portability and performance of the SYCL a
publishDate: 2023-09-18
sourceQuality: primary
========== RESULT 11 ==========
claims: [{"claim": "Bit-exact reproducibility from PyTorch GPU computation is not guaranteed across PyTorch releases, individual commits, or different hardware platforms.", "quote": "Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms.", "importance": "central"}, {"claim": "Even with identical RNG seeds, results may differ between CPU and GPU execution, meaning re-execution on a different device cannot be assumed to reproduce a prior result.", "quote": "Results may not be reproducible between CPU and GPU executions, even when using identical seeds.", "importance": "central"}, {"claim": "Some CUDA operations have no deterministic implementation at all; with torch.use_deterministic_algorithms(True), PyTorch raises an error rather than producing a nondeterministic result.", "quote": "throw an error if an operation is known to be nondeterministic (and without a deterministic alternative)", "importance": "central"}, {"claim": "Determinism in PyTorch carries a measurable performance cost: forcing deterministic algorithms is generally slower than the default nondeterministic ones.", "quote": "Deterministic operations are often slower than nondeterministic operations.", "importance": "supporting"}, {"claim": "Disabling cuDNN autotuning alone does not guarantee determinism; the selected convolution algorithm can itself be nondeterministic unless deterministic mode is explicitly enabled.", "quote": "that algorithm itself may be
publishDate: 2025-10-03
sourceQuality: primary
========== RESULT 12 ==========
claims: [{"claim": "GPUs are inherently non-deterministic, making bit-exact reproducibility across runs impossible, which directly breaks verification schemes that rely on re-executing a computation to check its result.", "quote": "GPUs are inherently non-deterministic, making it impossible to achieve bit-exact reproducibility", "importance": "central"}, {"claim": "Bit-exact reproducibility is a prerequisite for verifying GPU computations in security-critical settings such as untrusted or remote GPUs running ML models \u2014 exactly the proof-of-computation/integrity-check pattern a P2P GPU swarm would need.", "quote": "Bit-exact reproducibility is essential for verifying GPU computations in security-critical applications", "importance": "central"}, {"claim": "The primary causes of GPU non-determinism are atomic operations, floating-point reduction operations whose results depend on summation order, and unspecified scheduling of concurrent GPU threads.", "quote": "atomic operations", "importance": "supporting"}, {"claim": "Hawkeye is a GPU simulator that reproduces and detects non-deterministic behavior rather than enforcing determinism, letting researchers study how different execution orderings change computation outcomes.", "quote": "https://github.com/badasherez/gpu-simulator", "importance": "supporting"}, {"claim": "The work explicitly ties GPU non-determinism to distributed ML training and zero-knowledge proofs for ML (citing Megatron-LM, ZeRO, and cloud ML platforms), framing
publishDate: 2026-05-19
sourceQuality: primary
========== RESULT 13 ==========
claims: [{"claim": "A single WebGPU-based LLM inference framework (LlamaWeb) runs portably across 16 devices from 8 different GPU vendors (NVIDIA, AMD, Intel, Apple, Qualcomm, Samsung, ARM, Imagination), demonstrating WebGPU/WGSL as a genuine cross-vendor portable GPU compute substrate for heterogeneous hardware.", "quote": "GPU Vendors Tested (16 devices across 8 vendors): NVIDIA (3 devices: RTX 5080, RTX 5070, Tesla T4), AMD (1 device: RX 7900 XT), Intel (2 devices: Arc B580, Iris Xe), Apple (5 devices...), Qualcomm, Samsung, ARM, Imagination Tech", "importance": "central"}, {"claim": "Portable WebGPU compute carries a large performance penalty versus native CUDA/Vulkan: native backends outperform WebGPU by up to 10x during prefill and 2.5x during decode, with Metal beating WebGPU by over 2x prefill and 50% decode.", "quote": "The CUDA and Vulkan backends still significantly outperform WebGPU by up to 10\u00d7 during prefill and 2.5\u00d7 during decode.", "importance": "central"}, {"claim": "WebGPU portability is not uniform: the same WGSL kernels show order-of-magnitude implementation-dependent variance across browser backends, e.g. ~1 tok/s on Firefox vs ~52 tok/s on Chrome on the same Apple M3 device.", "quote": "substantial implementation-dependent performance differences across browsers,\" with Firefox performing at \"~1 tok/s on Firefox, but ~52 tok/s on Chrome\" on Apple M3.", "importance": "supporting"}, {"claim": "WebGPU is not always slower than native portable APIs: the
publishDate: 2026-05-20
sourceQuality: primary
========== RESULT 14 ==========
claims: [{"claim": "GPU floating-point computations can be made bit-exact across heterogeneous hardware (NVIDIA T4, RTX 3090, A100) by controlling FP operation order; the RepOps library re-implements ML operators (and exp/sin/cos/tanh) to enforce a fixed order, eliminating IEEE-754 non-associativity nondeterminism. This makes re-execution-based verification possible.", "quote": "RepOps re-implements common ML operators ... in a way that controls the order of floating point operators across hardware setups ... eliminating hardware non-determinism.", "importance": "central"}, {"claim": "Enforcing reproducibility carries a real performance cost: RepOps matrix multiplication incurs ~30-60% overhead for matrices larger than 2^10, with a steady-state ~60-70% overhead on RTX 3090 (24GB) and ~35% on T4 (16GB); overhead worsens for small matrices (~200% for 256-dim).", "quote": "RepOps matrix multiplications (of dimensions larger than 2^10) incur less than 30% overhead when tested on various GPUs.", "importance": "central"}, {"claim": "End-to-end reproducible model overhead is large and hardware-dependent: DistilBERT training incurred 258% (T4) and 312% (A100-40GB) overhead; Llama-3.1-1B training 374% (T4) and 67% (A100); Llama-3.1-8B 98% inference / 126% LoRA fine-tuning on A100-80GB. Overhead is lower on larger-memory GPUs.", "quote": "both inference and training on Llama-3.1-1B incur around 60% overhead RepOps on an A100 GPU (with 40 GB VRAM).", "importance": "central"}, {"claim": "RepOps
publishDate: 2025-02-26
sourceQuality: primary
========== RESULT 15 ==========
claims: [{"claim": "GPU floating-point non-associativity makes parallel reductions run-to-run non-deterministic, but deterministic alternatives (pairwise/tree reductions avoiding atomicAdd) can match or only slightly underperform the non-deterministic versions depending on GPU type.", "quote": "Our results show that the performance of a deterministic implementation can be faster or only slightly slower than its non-deterministic counterpart, depending on the GPU type.", "importance": "central"}, {"claim": "The naive deterministic single-atomicAdd approach is catastrophically slow: atomicAdd-only (AO) is two orders of magnitude slower than the fastest parallel-sum implementations, so determinism-via-serialization is impractical.", "quote": "We find that AO is 2 orders of magnitude slower than the fastest implementations.", "importance": "supporting"}, {"claim": "Well-designed deterministic GPU reduction kernels carry only a small overhead versus non-deterministic ones: under 0.2% difference on V100, up to 7.8% on GH200, with most V100 implementations within 0.5-1% of each other.", "quote": "The difference in timings between SPA and SPTR is less than 0.2% for the V100 GPU but can reach up to 7.8% on the GH200... Most implementations on V100 are within 0.5% to 1% of each other.", "importance": "supporting"}, {"claim": "FPNA-induced variability accumulates in iterative solvers and can approach or exceed 20% after six or seven iterations even in double precision, exceeding tolerances (as
publishDate: 2024-10-30
sourceQuality: primary
========== RESULT 16 ==========
claims: [{"claim": "Heterogeneous GPU hardware has a fundamental binary compatibility problem: a kernel compiled for one vendor's GPU cannot run on another due to divergent instruction sets, execution models, and driver stacks.", "quote": "Heterogeneous GPU infrastructures present a binary compatibility challenge: code compiled for one vendor's GPU will not run on another due to divergent instruction sets, execution models, and driver stacks.", "importance": "central"}, {"claim": "HetGPU proposes a single-GPU-binary portability stack (compiler + runtime + abstraction layer) that enables one binary to execute across NVIDIA, AMD, Intel, and Tenstorrent hardware via an architecture-agnostic intermediate representation translated to native code at execution time.", "quote": "We propose hetGPU, a new system comprising a compiler, runtime, and abstraction layer that together enable a single GPU binary to execute on NVIDIA, AMD, Intel, and Tenstorrent hardware.", "importance": "central"}, {"claim": "Achieving portability requires bridging fundamentally different GPU execution models, including warp-centric SIMT designs (NVIDIA/AMD) versus core-centric MIMD approaches (Tenstorrent RISC-V), unifying thread, memory, and synchronization abstractions.", "quote": "Bridges conceptual differences between warp-centric designs (NVIDIA/AMD SIMT) and core-centric approaches (Tenstorrent RISC-V MIMD), unifying thread, memory, and synchronization abstractions.", "importance": "supporting"}, {"claim": "He
publishDate: 2025-06-19
sourceQuality: primary
========== RESULT 17 ==========
claims: [{"claim": "Thunder Compute intercepts CUDA API calls on a virtual machine and translates them into network messages forwarded over TCP to a remote GPU, presenting a remote GPU as if it were local.", "quote": "Your virtual machine communicates with the GPU over TCP \u2014 the same protocol your browser uses. ... Behind the scenes, our instance translates those calls into network messages.", "importance": "central"}, {"claim": "Thunder Compute claims the GPU-over-TCP connection overhead is roughly 10-20 milliseconds for the initial connection, plus a scaled runtime increase, with GPU (re)attachment on the order of double-digit milliseconds.", "quote": "~10\u201320 milliseconds for the initial connection (blinking is ~200 milliseconds), and a scaled runtime increase.", "importance": "central"}, {"claim": "Thunder Compute asserts the GPU-over-TCP approach is viable because ML workloads are compute-bound rather than data-transfer-bound, and they optimize program execution to hide network latency.", "quote": "Most ML jobs spend far more time computing than waiting for data. By strategically optimizing the way your program runs behind the scenes, we can prevent network latency from affecting your GPU computation.", "importance": "central"}, {"claim": "In Thunder Compute's model a running process owns the entire physical GPU (full VRAM and compute), rather than fractionally sharing it among tenants simultaneously.", "quote": "When your process runs, it owns the whole GPU. You have a
publishDate: 2025-01-07
sourceQuality: blog
========== RESULT 18 ==========
claims: [{"claim": "Typical consumer internet upload bandwidth (~60 Mbps) makes naive data-parallel training infeasible: training a 671B-parameter model that way would take roughly 5,000 years, and the largest trainable model in a 10-minute sync window is only ~600M parameters with 32-bit gradients.", "quote": "Typical internet upload bandwidths for consumers are around 60 Mbps ... training a 671B parameter model would require \"5,000 years\" ... Maximum trainable model with 10-minute synchronization window at consumer speeds: \"600M parameter model with 32-bit gradients\"", "importance": "central"}, {"claim": "DiLoCo reduces communication-bandwidth requirements by ~500x by lowering synchronization frequency (up to 500 inner steps between syncs) with only a small performance penalty.", "quote": "Enables \"500x reduction in bandwidth requirements\" through reduced synchronization frequency ... Permits \"up to 500 inner steps with only a small reduction in performance\"", "importance": "central"}, {"claim": "Combined techniques (Streaming DiLoCo with quantization/sparsification) achieve roughly 100x reduction in bandwidth strain versus naive data parallelism; 8-bit gradients give 2-4x savings, and 25% sparsification (SparseLoCo) allows training models 8x larger.", "quote": "Combined techniques achieve \"100x reduction in bandwidth strain compared to naive data parallelism\" ... 8-bit gradient reduction: 2-4x bandwidth savings versus 32-bit ... SparseLoCo with 25% sparsification: enable
publishDate: 2025-12-29
sourceQuality: primary
========== RESULT 19 ==========
claims: []
sourceQuality: unreliable
========== RESULT 20 ==========
claims: [{"claim": "DiLoCo reduces inter-node communication frequency by 500x for a 150M model (and 125x in billion-parameter runs) by synchronizing pseudo-gradients only every ~500 local steps instead of every step, making globally distributed training feasible over commodity internet links.", "quote": "500x lower communication requirement compared to baseline with same compute budget (150M model); 125x reduction achieved in billion-parameter experiments with local step size of 125; DiLoCo synchronizes pseudo-gradients every ~500 steps versus traditional approaches", "importance": "central"}, {"claim": "OpenDiLoCo successfully trained a 1.1B-parameter model across four H100 clusters in three countries (Canada, Finland, Texas, Delaware) connected by links of only 127 to 935 Mbit/s, i.e. commodity-WAN bandwidth rather than datacenter fabric.", "quote": "Four H100 GPU clusters across three countries... Network bandwidth: \"between 127 to 935 Mbit/s\" across worker pairs", "importance": "central"}, {"claim": "Despite WAN-scale links spanning two continents, the system maintained 90-95% compute utilization, with the all-reduce synchronization consuming only 6.9% of total training time even though each all-reduce averaged 300 seconds.", "quote": "\"90-95% compute utilization\" maintained during globally distributed training across two continents and three countries; All-reduce bottleneck represented only \"6.9% of the training time\"; \"average all-reduce time between the workers of 300 s
publishDate: 2024-07-11
sourceQuality: primary
========== RESULT 21 ==========
claims: [{"claim": "With careful (asynchronous) remoting design, unmodified AI applications can run over GPU API remoting using commodity networking hardware with no overhead or even better performance and low network demands, contradicting the assumption that GPU-over-IP requires datacenter interconnect.", "quote": "with careful remoting design, unmodified AI applications can run on the remoting setup using commodity networking hardware without any overhead or even with better performance, with low network demands.", "importance": "central"}, {"claim": "Even at 10 microsecond network latency and just 1 Gbps bandwidth, GPT-2 inference under GPU API remoting suffers only ~2% performance degradation, because GPU execution time (10-100 ms) dwarfs network delay (1-100 us).", "quote": "At one extreme, GPT-2 only experiences a 2% performance degradation under a 10 \u00b5s RTT and 1 Gbps setup. The main reason is that the execution time of AI applications on GPUs (10\u2013100 ms) is typically significantly longer than the network delay", "importance": "central"}, {"claim": "Network latency in the 5-20 microsecond range is sufficient for most inference applications to benefit from API remoting with less than 5% overhead; achieving this is trivial if the proxy is co-located, and easy in modern datacenter networks.", "quote": "Latency in the range of 5\u201320 \u00b5s is sufficient for most inference applications to benefit from API remoting with less than 5% overhead.", "importance": "central
publishDate: 2024-01-24
sourceQuality: primary
========== RESULT 22 ==========
claims: [{"claim": "scuda (now renamed LUPINE, transferred from kevmo314 to lupinemachines) is a GPU-over-IP bridge that lets remote-machine GPUs be attached to CPU-only machines by intercepting CUDA driver API and NVML calls via LD_PRELOAD shim libraries (libcuda.so.1 and libnvidia-ml.so.1).", "quote": "LUPINE is a GPU over IP bridge allowing GPUs on remote machines to be attached to CPU-only machines. ... CUDA driver users pick up the LUPINE libcuda.so.1 shim and NVML users such as nvidia-smi pick up the LUPINE libnvidia-ml.so.1 shim automatically.", "importance": "central"}, {"claim": "The remoting transport is plain HTTP/2 (not RDMA), which is feasible to run over commodity/WAN networks and can be fronted with arbitrary TLS/auth.", "quote": "It's a plain HTTP/2 server, so you can front it with whatever TLS/auth server you want.", "importance": "central"}, {"claim": "The dominant overhead of GPU-over-IP remoting is slower host-device data transfers (because it bottlenecks the PCIe link over the network), but per-call compute overhead is small; once a model is resident on the GPU, little host transfer occurs, so latency-tolerant workloads with low host-device traffic perform acceptably.", "quote": "You will see device transfers get slower because this is basically bottlenecking a PCIe link over the network, but there is very little overhead besides that. ... once the model is on the GPU very little data transfer happens to the host. As a result, it might be faster than you expect."
publishDate: 2026-06-03
sourceQuality: primary
========== RESULT 23 ==========
claims: [{"claim": "Exact re-execution cannot verify GPU computation in decentralized networks because GPU operations are non-deterministic across nodes, making bitwise comparison of redundant computations ineffective even when hardware/software variables are controlled.", "quote": "The non-deterministic nature of GPU operations precludes the implementation of exact recomputation as a verification mechanism. While theoretical approaches suggest bitwise comparison of redundant computations as an optimal verification strategy, the architectural characteristics of GPU computing render such methodologies ineffective.", "importance": "central"}, {"claim": "GPU non-determinism stems from differences in GPU architecture, driver versions, CUDA implementations, cuDNN libraries, and framework distributions, but persists even when all of these are strictly controlled, due to the parallel execution nature of GPU operations.", "quote": "This non-determinism stems from multiple sources of hardware and software variability, including differences in GPU architecture, driver versions, CUDA implementations, cuDNN libraries, and framework distributions. However, even in environments where these variables are strictly controlled, fundamental non-determinism persists due to the parallel execution nature of GPU operations", "importance": "central"}, {"claim": "In LLM inference, non-determinism arises specifically from the non-guaranteed ordering of floating-point arithmetic in parallel matrix operations;
publishDate: 2025-01-09
sourceQuality: primary
========== RESULT 24 ==========
claims: [{"claim": "NCCL's three protocols trade latency for bandwidth: the LL (Low-Latency) protocol achieves ~1 microsecond per-hop latency but only 25-50% of peak bandwidth, LL128 achieves ~2 microsecond latency at ~95% of peak bandwidth, and the Simple protocol reaches near-peak bandwidth for large messages but with higher synchronization overhead.", "quote": "LL Protocol: \"~1 \u03bcs\" latency per-hop, Achieves \"25\u223c50% of peak\" bandwidth ... LL128 Protocol: \"~2 \u03bcs\" latency per-hop, Achieves \"\u223c95% of peak\" bandwidth ... Simple ... \"achieves near-peak bandwidth for large messages\"", "importance": "central"}, {"claim": "There is a roughly 6x bandwidth gap between intra-node and inter-node interconnect on a modern GPU supercomputer (Alps): 150 GB/s intra-node high-bandwidth interconnect versus 25 GB/s per-direction inter-node network links (Cray Slingshot).", "quote": "\"150GB/s high-bandwidth intra-node interconnect\" ... \"25GB/s per-direction network link\" (Cray Slingshot)", "importance": "central"}, {"claim": "LL128's high-bandwidth efficiency depends on atomic 128-byte writes (120B data + 8B flag per 128-byte unit), a hardware-level guarantee not available over commodity network transports.", "quote": "LL128 Protocol: ... 120B data + 8B flag per 128-byte unit, \"depends on atomic 128-byte writes\"", "importance": "supporting"}, {"claim": "Collective algorithm choice is message-size dependent: ring algorithms are optimal for large messages while tree alg
publishDate: 2025-07-07
sourceQuality: primary
========== RESULT 25 ==========
claims: [{"claim": "GPU floating-point computation can be made bit-exact across different GPUs by fixing the order of floating-point operations, because FP ops are non-associative even under IEEE-754; Verde's RepOps library does this for ML operators.", "quote": "RepOps (Reproducible Operators), a library that implements bitwise reproducible versions of popular ML operators. It works by ensuring that floating point operations are performed in the same order", "importance": "central"}, {"claim": "Forcing reproducible (fixed-order) floating-point operations carries a real but bounded performance cost: under ~30% overhead for large matrix multiplications, ~58-98% inference and 67-312% training overhead depending on model, on A100 GPUs.", "quote": "RepOps matrix multiplications (of dimensions larger than 2^10) incur less than 30% overhead when tested on various GPUs", "importance": "central"}, {"claim": "Verifying an untrusted ML computation by refereed delegation requires the referee to compute only a single diverging operator, not re-run the whole model, costing two orders of magnitude less compute than running the model.", "quote": "The referee's only needs to compute a single operator in the computational graph, which can be performed with two orders of magnitude less compute resources than it takes to run the model itself", "importance": "central"}, {"claim": "Refereed-delegation verification of ML adds less than one order of magnitude total overhead, versus four orders of magnitude
publishDate: 2025-02-26
sourceQuality: primary
========== RESULT 26 ==========
claims: [{"claim": "Gensyn's Verde achieves bitwise-reproducible GPU/ML results across diverse hardware using RepOps, which enforces a fixed ordering of operations for common ML operators to overcome floating-point non-associativity.", "quote": "RepOps (reproducible operators) enforces a fixed ordering of operations for common ML operators (such as matrix multiplication) to overcome the non-associativity problem... With RepOps, we achieve bitwise-reproducible results across diverse hardware.", "importance": "central"}, {"claim": "Verde verifies untrusted GPU computation via refereed delegation: when two providers disagree, a referee resolves it by re-executing a single operation rather than the entire task, located via a two-level (iteration then operation) bisection game.", "quote": "if two compute providers disagree on an ML execution, a referee can determine which provider is honest by executing a single operation rather than running the entire task.", "importance": "central"}, {"claim": "Verde reduces verification overhead by re-executing only a randomly selected single checkpoint interval for training (not the whole task), and a single forward pass for autoregressive inference via causal masking.", "quote": "only a single checkpoint interval (randomly selected)", "importance": "supporting"}, {"claim": "Naive hashing of outputs fails for verifying ML training because floating-point non-associativity differences propagate and amplify into significant deviations, which is why fixed
publishDate: 2025-11-24
sourceQuality: primary
========== RESULT 27 ==========
claims: [{"claim": "NVIDIA MIG partitions a single supported GPU into up to seven hardware-isolated GPU instances, each with dedicated memory and compute and hardware-level fault isolation, but only on Ampere+ (compute capability 8.0+) datacenter GPUs (A100, A30, H100, H200, B200, GB200).", "quote": "Multi-Instance GPU (MIG) partitions a supported Ampere-generation (compute capability 8.0+) NVIDIA GPU into up to seven hardware-isolated GPU instances", "importance": "central"}, {"claim": "MIG is static hardware partitioning using predefined profiles that must be planned in advance, and reconfiguring requires the GPU to be free of running workloads \u2014 so it fractionalizes one GPU among tenants but cannot dynamically adapt.", "quote": "Static, hardware-level partitioning method, MIG uses predefined profiles, so you must plan slice sizes in advance", "importance": "supporting"}, {"claim": "MPS gives more explicit memory/compute partitioning than time-slicing but weaker isolation than MIG; a fatal fault in one client can put the MPS server into a FAULT state and affect other clients, and NVIDIA marks MPS device-plugin support as experimental.", "quote": "A fatal fault from one client can cause the MPS server to enter a FAULT state and affect other clients", "importance": "central"}, {"claim": "Time-slicing shares a single physical GPU via CUDA context switching across processes taking turns; it does not isolate memory or faults and contention can impact latency, but it works across a
publishDate: 2026-05-26
sourceQuality: blog
========== RESULT 28 ==========
claims: [{"claim": "The io.net attack was a SQL injection that exploited exposed user ID tokens to make unauthorized changes to device metadata in the GPU network; the GPUs' actual hardware was NOT compromised due to permission layers.", "quote": "Malicious users exploited exposed user ID tokens to execute a system query language (SQL) injection attack, which led to unauthorized changes in device metadata within the graphics processing unit (GPU) network ... The attack did not compromise the GPUs' actual hardware, which remains secure due to robust permission layers.", "importance": "central"}, {"claim": "io.net suffered a Sybil attack in which roughly 1.8 million fake/spoofed GPUs attempted to connect to the network to solicit rewards.", "quote": "the network saw an enormous spike of ~1.8M fake GPUs attempting to connect, as users were spoofing GPUs to solicit rewards from the network in what was characterized as a Sybil attack", "importance": "central"}, {"claim": "After security upgrades, active GPU connections on io.net dropped dramatically from 600,000 to 10,000 because valid GPUs that did not restart and upgrade could not access the uptime API.", "quote": "valid GPUs that did not restart and upgrade could not access the uptime API, causing a dramatic decline in active GPU connections from 600,000 to 10,000", "importance": "supporting"}, {"claim": "The metadata breach was detected via monitoring of a surge in write operations to the GPU metadata API, triggering alerts at 1:05 am
publishDate: 2024-04-28
sourceQuality: secondary
========== RESULT 29 ==========
claims: [{"claim": "Intra-node NVSwitch in the DGX-2 delivers 50 GB/s bidirectional per link, ~300 GB/s per GPU via a single switch hop, and a baseboard-to-baseboard raw bisection bandwidth of 2.4 TB/s \u2014 establishing that datacenter GPU fabric operates in the hundreds-of-GB/s-to-TB/s range.", "quote": "each of the GPUs on one baseboard can talk to any GPUs on the other baseboard also at a full bandwidth of 300 GB/s, but through 2 NVSwitch hops. The baseboard-to-baseboard raw bisection bandwidth is thus 25 GB/s x 8 links/switch x 6 switches=2.4 TB/s", "importance": "central"}, {"claim": "Inter-node GPU communication over InfiniBand with GPUDirect-RDMA peaks at only ~12 GB/s (at 64MB messages) on SummitDev, with pinned-memory GPUDirect reaching >14 GB/s \u2014 i.e. inter-node bandwidth is more than an order of magnitude below intra-node NVLink/NVSwitch.", "quote": "From 4MB on, GPUDirect-RDMA shows its advantage on bandwidth and obtains its optimal bandwidth -- 12GB/s at 64MB. However, this is still lower than the PinnedMem-GPUDirect scheme, which demonstrates more than 14GB/s sustainable bandwidth", "importance": "central"}, {"claim": "Inter-node collective (all-reduce etc.) bandwidth across IB-connected nodes saturates at only ~16 GB/s on SummitDev and ~32 GB/s on Summit, and only at very large 32/64MB message sizes \u2014 small messages get poor efficiency.", "quote": "the bandwidth for the five CL operations converge and saturate around 32/64MB message size, demonstrating near
publishDate: 2019-03-11
sourceQuality: primary