| … | … | @@ -0,0 +1,108 @@ |
| 1 | # Reproducible builds → recovering verifiability for build chunks |
| 2 | |
| 3 | > Research run 2026-06-17 (deep-research harness, run `wf_16b48908-b21`). |
| 4 | > **Verification caveat:** the adversarial-verify phase was starved by API |
| 5 | > rate-limiting — 18 of 25 claims scored `0-0 (3 abstain)` because the verifier |
| 6 | > votes *errored*, NOT because the claims were refuted. Treat those as |
| 7 | > **unconfirmed-this-round, not disproven.** Only the 5 findings below survived |
| 8 | > genuine 3-0 / 2-0 voting from primary sources. Re-run the open items when the |
| 9 | > API is not rate-limited. |
| 10 | |
| 11 | ## The problem (recap) |
| 12 | |
| 13 | A build chunk = `run(build action, content-addressed inputs) → output`, memoized |
| 14 | by hash. Non-reproducible builds produce different output bytes on different |
| 15 | peers, which (a) breaks content-addressed memoization/dedup and (b) makes |
| 16 | multi-peer re-execution voting (T1) impossible — forcing the chunk down to |
| 17 | **T0 = trust the builder**. Goal: peer-side steps that claw back verifiability. |
| 18 | |
| 19 | ## Confirmed findings (survived adversarial verification) |
| 20 | |
| 21 | 1. **Playbook kills the two biggest non-determinism classes** (primary: |
| 22 | reproducible-builds.org/docs/source-date-epoch, /build-path). |
| 23 | - `SOURCE_DATE_EPOCH` — standardized env var; tools read it instead of calling |
| 24 | system time → eliminates embedded-timestamp non-determinism (the single |
| 25 | biggest class). |
| 26 | - `-ffile-prefix-map=OLD=NEW` (alias of `-fdebug-prefix-map` + |
| 27 | `-fmacro-prefix-map`; GCC 8 / Clang 10) → normalizes embedded build paths in |
| 28 | debug info and `__FILE__` → eliminates the build-path class for C/C++ at |
| 29 | compile time. |
| 30 | |
| 31 | 2. **Normalize-then-hash is real and tooled** (primary: reproducible-builds.org/tools; |
| 32 | github.com/keszybz/add-determinism). |
| 33 | - `strip-nondeterminism` normalizes gzip / ZIP / Jar. |
| 34 | - `add-determinism` extends to static archives, gzip, and rewrites `.pyc` |
| 35 | bytecode (`.a` resets embedded mod times to `SOURCE_DATE_EPOCH`; `.gz` clamps |
| 36 | mtime; `.pyc` cleans flag refs). Active in Fedora post-install. |
| 37 | - **Nuance:** these tools normalize *in-place during the build* to emit |
| 38 | identical bytes, not "hash a post-build normalized form." Same mechanism; |
| 39 | for us, lift their per-format logic into a **peer-side post-step before |
| 40 | writing the `result` op**. |
| 41 | |
| 42 | 3. **`diffoscope`** (primary: reproducible-builds.org/tools) — recursively unpacks |
| 43 | archives + renders binaries human-readable to explain *why* two builds diverge. |
| 44 | The diagnostic that tells us which fields to normalize. |
| 45 | |
| 46 | 4. **Rebuilder voting = our T1, production-proven** (primary: |
| 47 | github.com/kpcyrd/rebuilderd). `rebuilderd` (Arch) monitors a repo and uses |
| 48 | backend rebuilders to verify published binaries reproduce from source — the |
| 49 | independent-rebuild trust model, 1:1 with our multi-writer pool. Trust gained = |
| 50 | **quorum-of-rebuilders** (shifts trust to the rebuilder set you choose, doesn't |
| 51 | eliminate it). |
| 52 | |
| 53 | 5. **Layered recommendation (synthesis, medium confidence):** |
| 54 | `playbook (#1) → normalize-then-hash (#2) → vote on normalized hash (T1) → T0 |
| 55 | fallback for the residual tail.` |
| 56 | |
| 57 | ## The design-critical ordering: normalize BEFORE vote |
| 58 | |
| 59 | rebuilderd's own framing: **a failed rebuild is far more likely non-determinism |
| 60 | than an actual compromise.** So if you vote on raw bytes, T1 drowns in |
| 61 | false-positive "disagreements" that are just noise. Normalization must run first; |
| 62 | voting happens on the normalized hash. (This claim was in a rate-killed vote but |
| 63 | is in the canonical rebuilderd README — reported at reduced weight, but it's the |
| 64 | load-bearing ordering constraint.) |
| 65 | |
| 66 | ## The boundary between #1 and #2 (the question we set out to answer) |
| 67 | |
| 68 | - **Normalize-then-hash (#2) absorbs *packaging-level* noise** — archive |
| 69 | timestamps, file ordering, gzip/zip/jar metadata, `.pyc`. Strippable post-hoc. |
| 70 | This is the cheap initial win and covers a large fraction of real cases. |
| 71 | - **Only the compile-time playbook (#1) fixes *in-binary* noise** — paths in |
| 72 | debug info, build-ids, codegen/LTO ordering. Cannot be cheaply stripped from an |
| 73 | arbitrary binary after the fact; must be compiled out. |
| 74 | |
| 75 | They cover **different byte classes** — not redundant. #2 first (cheap), #1 as the |
| 76 | long-term proper fix. |
| 77 | |
| 78 | ## Trust-tier mapping |
| 79 | |
| 80 | | Mitigation | Trust it buys | Tier | Maturity | |
| 81 | |---|---|---|---| |
| 82 | | Reproducible-builds playbook | none shifted — restores true determinism | enables T1 | production | |
| 83 | | Normalize-then-hash | none shifted — converges known-noise bytes | enables T1 | production (tools exist) | |
| 84 | | Rebuilder / re-exec voting | quorum-of-rebuilders (shifts to the set) | **T1** | production (Arch) | |
| 85 | | TEE-attested build | shifts to TEE vendor | ~T2.5 | **UNCONFIRMED** | |
| 86 | | zkVM proof-of-build | shifts to proof system | **T3** | **UNCONFIRMED** | |
| 87 | | Statistical / refereed (TOPLOC, Verde) | bounded / fraud-proof | **T2** | **UNCONFIRMED** (ML, maybe not builds) | |
| 88 | |
| 89 | ## Open questions (rate-limited out — re-run) |
| 90 | |
| 91 | 1. **Residual-tail size per toolchain** after playbook + normalize (C/C++, Rust, |
| 92 | Go, JVM, Python wheels) — the most decision-relevant number. Reported but |
| 93 | unverified: Debian ~94–95.5%, Guix ~77.6%, Nix 69–91%. Do NOT quote as settled. |
| 94 | 2. **TEE-attested builds** (SGX / SEV-SNP / Nitro) — verifiable without |
| 95 | reproducibility? Threat model (trust → TEE vendor), overhead, maturity. |
| 96 | 3. **zkVM proof-of-execution** (RISC Zero / SP1 / Jolt) for compile steps — |
| 97 | feasible today? Cost in orders of magnitude. |
| 98 | 4. **Refereed delegation / statistical** (Gensyn Verde, TrueBit, TOPLOC) — do they |
| 99 | apply to deterministic-but-not-bit-reproducible builds, or only ML inference? |
| 100 | |
| 101 | ## Implications for the hypercompute code |
| 102 | |
| 103 | - The `result`-op handling must become **write-once** (`pool.js:60` currently does |
| 104 | an unconditional `view.put` — last-write-wins). Combined with normalize-then-hash, |
| 105 | the *first accepted normalized result* becomes canonical instead of clobberable. |
| 106 | - Normalization is a **peer-side post-step** between executing the build and |
| 107 | appending the `result` op — reuse `strip-nondeterminism` / `add-determinism` |
| 108 | per-format logic. The chunk is keyed by the **normalized** output hash. |