| … | … | @@ -0,0 +1,66 @@ |
| 1 | # Jigsaw transformer v0 — count-based attention over jigsaws. Pre-registration |
| 2 | |
| 3 | 2026-10-18. Written and committed **before any code or run**. |
| 4 | |
| 5 | ## Idea |
| 6 | |
| 7 | Linas (diary ten-I:11698-11705): "invent the transformer, but for jigsaws". Our synthesis keeps what a |
| 8 | transformer layer does — **context-dependent representations, soft attention, stacked layers** — but every |
| 9 | number is a **count** or a statistic of counts, so every decision traces to observed data (**auditable**) and |
| 10 | learning is additive (**continuous**). v0 is one level: words → context-dependent classes for every occurrence. |
| 11 | |
| 12 | ## Model (all quantities are counts or ratios of counts) |
| 13 | |
| 14 | Each occurrence t carries a distribution q_t over K classes. Parameters, all from (soft) counts with add-λ |
| 15 | smoothing: π(z | word), left/right neighbour-class profiles Φ_L(y | z), Φ_R(y | z), and class-pair counts |
| 16 | N(x, y) over the window (W = 6) giving positive PMI⁺(x, y) (Linas's pair MI, over classes). |
| 17 | |
| 18 | **One layer (E-step for a sentence):** |
| 19 | 1. **Attention = soft connector mating.** For tokens t, j in the window: |
| 20 | s_tj = max(0, Σ_x Σ_y q_left(x) q_right(y) PMI⁺(x, y)) (left/right by position); α_tj = s_tj / Σ_j s_tj |
| 21 | (uniform if all zero). This is Linas's MI-weighted link, kept soft instead of the argmax tree. |
| 22 | 2. **Messages:** m^L_t(y) = Σ_{j<t} α_tj q_j(y), m^R_t(y) = Σ_{j>t} α_tj q_j(y) — the occurrence's attended |
| 23 | connector surface. |
| 24 | 3. **Class update:** q_t(z) ∝ π(z | w_t) · exp(Σ_y m^L_t(y) log Φ_L(y | z) + Σ_y m^R_t(y) log Φ_R(y | z)). |
| 25 | |
| 26 | Two layers (two sweeps) per sentence. **M-step:** add the soft counts (q_t into π; q_t ⊗ m_t into Φ; |
| 27 | q_i ⊗ q_j into N) and renormalise. Initialisation: random (seeded), K = 30 (the gold count; declared shortcut, |
| 28 | with K ∈ {15, 60} reported as a sweep). |
| 29 | |
| 30 | - **Batch mode:** 10 EM passes over the training prefix. |
| 31 | - **Continuous mode:** the stream in batches of 1K sentences, each seen once: E-step with current |
| 32 | parameters, add its counts (never deleted), M-step. Same code. |
| 33 | |
| 34 | ## Auditability |
| 35 | |
| 36 | Every M-step is an **event**: batch id, hash of its sentences, the summary of the count deltas, and the hash |
| 37 | of the resulting parameters, chained (each event includes the previous event's hash). Replaying the inputs |
| 38 | must reproduce every hash. `explain(t)`: for any occurrence, the attended neighbours with their α, the |
| 39 | class-pair counts behind each PMI, and the profile counts behind q_t. |
| 40 | |
| 41 | ## Corpus, baselines, metrics |
| 42 | |
| 43 | Planted grammar of `across-cluster/ac.c` (30 classes, ~1,600 words, 10% polysemy, gold sense of every |
| 44 | occurrence, gold trees). Tiers 3K, 10K, 30K training sentences; 2K held-out; 5 seeds. |
| 45 | |
| 46 | - **Token V-measure** and many-to-one accuracy of argmax q_t vs gold token classes (held-out, E-step only). |
| 47 | - **Sense accuracy** on occurrences of polysemous words, after many-to-one mapping of learned classes to |
| 48 | gold. Baselines: **MAJ** = each word always gets its majority gold class (the best any one-class-per-word |
| 49 | method can do); **TYPE** = exchange-algorithm word classes, K = 30 (from `ac.c`). |
| 50 | - **Attention parse:** Eisner over α_tj as edge weights; link F1 vs gold trees. Baseline: word-level MI-tree |
| 51 | parse (`ac.c` arm U) and gold-class MI-tree parse (arm G). |
| 52 | |
| 53 | ## Predictions (paired over 5 seeds; "beats" = 95% CI of the paired difference excludes 0) |
| 54 | |
| 55 | - **J1 (context helps senses):** at 30K, batch JT beats **MAJ** on sense accuracy — i.e. it does better on |
| 56 | polysemous words than any one-class-per-word method could. |
| 57 | - **J2 (good classes):** at 30K, batch JT token V-measure ≥ 0.7 and beats TYPE. |
| 58 | - **J3 (soft parse):** at every tier, the attention parse beats the word-level MI-tree parse on link F1. |
| 59 | - **J4 (continuous):** at 30K, continuous JT is within 0.03 token V-measure of batch JT. |
| 60 | - **J5 (audit):** replay reproduces every event hash exactly; replay of a log with one event's count delta |
| 61 | altered fails at that event. |
| 62 | - **J6 (control):** with words shuffled within each sentence, JT's sense accuracy does not beat MAJ. |
| 63 | |
| 64 | ## Not tested here |
| 65 | |
| 66 | Levels beyond 1 (collapse-and-repeat), real text, AtomSpace storage, GPU. |