| … | … | @@ -0,0 +1,45 @@ |
| 1 | # Jigsaw transformer v1 — classes emerge from word-level PMI. Pre-registration |
| 2 | |
| 3 | 2026-10-18. Written and committed **before any v1 code or run**. Replaces v0's random start (`SMOKE.md`: |
| 4 | attention over random classes had PMI ≈ 0, so nothing could organise). Same corpus, metrics, seeds and tiers |
| 5 | as `PREREG.md` unless stated. |
| 6 | |
| 7 | ## The learner — one pipeline, everything from the training text |
| 8 | |
| 9 | **Layer 0 (words).** Ordered word pairs in the window (W = 6) → gated PMI (PMI > 0, G > 10.83; `ac.c`). |
| 10 | Each occurrence attends to its window neighbours in proportion to the gated PMI of the word pair: |
| 11 | α_tj = S(w_left, w_right) / Σ_j S (no attention if no neighbour passes). This is a soft Linas MI parse. |
| 12 | |
| 13 | **Classes emerge.** Attended pairs are accumulated as weighted word-pair counts (each pair gets α_tj + α_jt). |
| 14 | Words are clustered into K classes by the exchange algorithm maximising the MI between the classes of attended |
| 15 | pairs (Martin/Liermann/Ney objective over these weighted pairs; deterministic, frequency-rank start). Nothing |
| 16 | is given except K = 30 (declared shortcut, as in v0). |
| 17 | |
| 18 | **Layers 1–2 (classes) and EM.** Every occurrence starts as its word's class (one-hot). One pass of |
| 19 | accumulation with the word-level attention initialises π, Φ_L, Φ_R and the class-pair counts N; then v0's |
| 20 | count-based layers (class-pair PMI⁺ attention, messages, per-occurrence class update) and 10 EM passes refine |
| 21 | each occurrence's class distribution from its context. |
| 22 | |
| 23 | **Continuous.** Checkpoints at 1K, 2K, 4K, 8K, 16K and the tier size (the doubling schedule). Word-pair counts |
| 24 | are only ever added; at each checkpoint everything else is re-learned from all text so far, which is the |
| 25 | "re-learn from scratch each sleep" design. Each checkpoint is a hash-chained event (sentence-range hash, |
| 26 | pair-count mass added, state hash). This makes v0's J4 trivially true (the final checkpoint equals batch), so |
| 27 | J4 is replaced by J4′ below. |
| 28 | |
| 29 | ## Arms and baselines |
| 30 | |
| 31 | **JT-v1** (full); **TYPE-A** (the layer-0 classes alone, one class per word: the learner's own starting |
| 32 | point); **TYPE** (exchange over adjacent bigrams, from `ac.c`); **MAJ** (majority gold class per word). |
| 33 | |
| 34 | ## Predictions (paired over 5 seeds; "beats" = 95% CI of the paired difference excludes 0) |
| 35 | |
| 36 | - **J1 (context refines senses):** at 30K, JT-v1 beats **TYPE-A** on sense accuracy for polysemous words. That is, |
| 37 | the attention layers add something one-class-per-word cannot. |
| 38 | - **J1+ (strong form):** at 30K, JT-v1 beats **MAJ** on sense accuracy. |
| 39 | - **J2 (good classes from PMI):** at 30K, JT-v1 token V-measure ≥ 0.7 and beats TYPE. |
| 40 | - **J3 (soft parse):** at every tier, the class-level attention parse beats the word-level MI-tree parse on link F1. |
| 41 | - **J4′ (continuous learning curve):** token V-measure at the checkpoints is non-decreasing from 4K on, in |
| 42 | mean over seeds. |
| 43 | - **J5 (audit):** replay reproduces every checkpoint hash; altering one checkpoint's count delta is detected at |
| 44 | that checkpoint. |
| 45 | - **J6 (control):** with words shuffled within each sentence, JT-v1 does not beat TYPE-A on sense accuracy. |