| … | … | @@ -0,0 +1,47 @@ |
| 1 | # Jigsaw transformer v3 — split words into senses by their disjuncts. Pre-registration |
| 2 | |
| 3 | 2026-10-18. Written and committed **before any v3 code or run**. Follows `RESULTS-v2.md`: per-occurrence context |
| 4 | has a small order-dependent effect, but every learner's *collapsed* sense accuracy is ~0.63 against MAJ 0.886 — |
| 5 | polysemous words sit in wrong classes at the type level, because exchange gives each word one class from its |
| 6 | mixed contexts. Linas's answer is to cluster **word-disjunct pairs (Sections)**, so a word's Sections can go to |
| 7 | different classes. v3 tests that, with one split round. |
| 8 | |
| 9 | ## The learner (per checkpoint, everything re-learned from the text so far) |
| 10 | |
| 11 | 1. **V2-DJ** as pre-registered (unchanged), giving final parameters. |
| 12 | 2. **Sections.** E-step on every training sentence; occurrence t's Section = (word, its disjunct's connectors |
| 13 | (side, class)). |
| 14 | 3. **Split test per word** with ≥ 20 training occurrences. Model the word's occurrences as bags of connectors. |
| 15 | Initialise a 2-way split on the single connector whose presence/absence gives the largest G-statistic |
| 16 | (deterministic); refine with 20 EM passes of a 2-component mixture of multinomials (add-0.1). Occurrences |
| 17 | with no connectors are left out of the fit and go to the larger component. **Accept** the split if |
| 18 | G = 2·(LL₂ − LL₁) exceeds the χ² 0.999 quantile with df = number of distinct connectors the word uses |
| 19 | (Wilson–Hilferty), and the smaller component has weight ≥ 0.05. Otherwise one sense. |
| 20 | 4. **Sense units.** Training occurrences in the smaller component of a split word get a new unit id. Layer 0 |
| 21 | (word-pair PMI attention) and the exchange clustering are re-run over **units**; each training occurrence |
| 22 | starts one-hot in its unit's class, so a split word's prior π(z | w) has two modes. |
| 23 | 5. **EM** over words exactly as V2-DJ (held-out text has words only; the context picks the sense). |
| 24 | |
| 25 | ## Arms |
| 26 | |
| 27 | **JT-V3** and **JT-V3-COLL**; **JT-V2D** and **JT-V2D-COLL** in the same job; MAJ. Ordered and shuffled-within- |
| 28 | sentence; 5 seeds; tiers 3K/10K/30K; audit as v2 (the state hash also covers the units). |
| 29 | |
| 30 | **Split diagnostics** (final checkpoint): eligible words, words split, **split precision** (share of split |
| 31 | words that are gold-polysemous), **split recall** (share of eligible gold-polysemous words that are split), |
| 32 | base rate (share of eligible words that are polysemous). |
| 33 | |
| 34 | ## Predictions (paired over 5 seeds at 30K; "beats" = 95% CI of the paired difference excludes 0) |
| 35 | |
| 36 | - **S1 (senses are found):** split precision ≥ 2 × base rate (mean over seeds), and split recall ≥ 0.3. |
| 37 | - **S2 (type-level classes improve):** JT-V3-COLL beats JT-V2D-COLL on sense accuracy. |
| 38 | - **S3 (senses improve):** JT-V3 beats JT-V2D on sense accuracy. |
| 39 | - **S4 (control):** on shuffled text, words split ≤ 0.5 × words split on ordered text (mean), and JT-V3-SHUF |
| 40 | does not beat JT-V2D-SHUF on sense accuracy. |
| 41 | - **S5 (no class damage):** token V-measure JT-V3 − JT-V2D has 95% CI lower bound > −0.02. |
| 42 | - **S6 (audit):** replay and mutation checks all PASS. |
| 43 | |
| 44 | ## Procedure |
| 45 | |
| 46 | One smoke run (seed 1, 3K) only to check it runs and the audit passes; no parameter changes after it. Scored by |
| 47 | `summarize-v3.py`, written before the full run. |