OverviewFilesCommitsIssuesDocsReleasesPeers

atomspace-layer-discovery commits / 3ffb8cb

experiments/jigsaw-transformer: v1 results — PMI-emergent classes + soft parse work; sense claim not supported
J3 (attention parse 0.625 vs word MI 0.378), J4' (monotone learning curve), J5 (60/60 audit) PASS; J1 PASS but J6 FAIL (same gain with order shuffled), J1+ and J2 FAIL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3ffb8cb0b69becdd26ced56df762c886586d2d58
by plankatron <fleeky@gmail.com> · 2026-10-18T21:07:11.000Z

Changed files (15) +474 -0

experiments/jigsaw-transformer/RESULTS-v1.md +27 -0
……@@ -0,0 +1,27 @@
1# Jigsaw transformer v1 — results
2
3Run `results-v1/1792357299` (code `964b7e1`, 5 seeds, tiers 3K/10K/30K, plus word-order-shuffled control). Scored against
4PREREG-v1 (`fc8ed86`, before any v1 code) by `summarize-v1.py`; full output in `results-v1/1792357299/summary.txt`.
5
6| | prediction | verdict |
7|---|---|---|
8| J1 | context layers beat the learner's own starting classes on polysemous words | **PASS** (+0.044 [+0.030, +0.058]) — but see J6 |
9| J1+ | beat the one-class-per-word ceiling (MAJ) | **FAIL** (0.637 vs 0.886) |
10| J2 | V-measure ≥ 0.7 and beats adjacent-bigram exchange (TYPE) | **FAIL**: 0.842 passes the bar, but TYPE is 0.901 (−0.059, CI crosses 0) |
11| J3 | attention parse beats the word-level MI tree at every tier | **PASS** (0.485 / 0.572 / 0.625 vs 0.158 / 0.272 / 0.378) |
12| J4′ | continuous checkpoints non-decreasing from 4K | **PASS** (0.653 → 0.746 → 0.803 → 0.842) |
13| J5 | replay reproduces every checkpoint; one altered checkpoint detected there | **PASS** (60/60) |
14| J6 | no sense gain with words shuffled within sentences | **FAIL**: the same +0.045 gain appears shuffled |
15
16## What it says
17
181. **Classes emerge from PMI and improve steadily with data** (V-measure 0.52 at 1K → 0.84 at 30K), with an
19 auditable, replayable, tamper-evident record (J4′, J5). Many-to-one accuracy beats TYPE at 30K (0.829 vs
20 0.771), though V-measure does not (not pre-registered; noted only).
212. **The attention parse is the clearest win:** link F1 0.625 vs the word-level MI tree's 0.378 at 30K. It
22 approaches the gold-class MI tree (0.734): count-based soft attention over discovered classes pools evidence
23 that word-level PMI cannot.
243. **The polysemy gain is not from word order.** J6 shows the same +0.045 when order is destroyed, so the gain
25 over the starting classes comes from which words co-occur in a sentence (or from EM refinement itself),
26 not from attending to positional connectors. The contextual-sense claim is **not supported**.
274. Still behind the simple adjacent-bigram clusterer on V-measure, and far from the one-class ceiling on senses.
experiments/jigsaw-transformer/results-v1/1792357299/commit.txt +1 -0
……@@ -0,0 +1 @@
1964b7e17152602fbf9669ccb26b4418cd886be8c
experiments/jigsaw-transformer/results-v1/1792357299/seed-1.tsv +41 -0
……@@ -0,0 +1,41 @@
1C 1 3000 30 JT-V1 1000 0.51007 0.52897 0.57673
2C 1 3000 30 JT-V1 2000 0.61088 0.61267 0.49182
3C 1 3000 30 JT-V1 3000 0.59293 0.59555 0.46415
4V 1 3000 30 JT-V1 0.59293 0.59555 0.46415
5L 1 3000 30 ATTN-V1 0.48016
6V 1 3000 30 TYPE-A 0.49386 0.55137 0.45220
7A 1 3000 30 replay PASS events=3 first_mismatch=-1
8A 1 3000 30 mutation PASS altered=1 detected_at=1
9V 1 3000 30 TYPE 0.77499 0.61384 0.63396
10V 1 3000 30 MAJ 0.97629 0.98795 0.88931
11L 1 3000 30 WORD-MI 0.14961
12L 1 3000 30 GOLD-MI 0.70421
13C 1 10000 30 JT-V1 1000 0.51007 0.52897 0.57673
14C 1 10000 30 JT-V1 2000 0.61088 0.61267 0.49182
15C 1 10000 30 JT-V1 4000 0.64708 0.64247 0.55409
16C 1 10000 30 JT-V1 8000 0.72887 0.73178 0.64654
17C 1 10000 30 JT-V1 10000 0.75161 0.75637 0.65346
18V 1 10000 30 JT-V1 0.75161 0.75637 0.65346
19L 1 10000 30 ATTN-V1 0.57421
20V 1 10000 30 TYPE-A 0.64821 0.69973 0.59371
21A 1 10000 30 replay PASS events=5 first_mismatch=-1
22A 1 10000 30 mutation PASS altered=2 detected_at=2
23V 1 10000 30 TYPE 0.90545 0.78986 0.76164
24V 1 10000 30 MAJ 0.97820 0.98904 0.89937
25L 1 10000 30 WORD-MI 0.26366
26L 1 10000 30 GOLD-MI 0.72903
27C 1 30000 30 JT-V1 1000 0.51007 0.52897 0.57673
28C 1 30000 30 JT-V1 2000 0.61088 0.61267 0.49182
29C 1 30000 30 JT-V1 4000 0.64708 0.64247 0.55409
30C 1 30000 30 JT-V1 8000 0.72887 0.73178 0.64654
31C 1 30000 30 JT-V1 16000 0.80335 0.79726 0.75283
32C 1 30000 30 JT-V1 30000 0.84553 0.82226 0.73836
33V 1 30000 30 JT-V1 0.84553 0.82226 0.73836
34L 1 30000 30 ATTN-V1 0.62905
35V 1 30000 30 TYPE-A 0.79801 0.79651 0.69811
36A 1 30000 30 replay PASS events=6 first_mismatch=-1
37A 1 30000 30 mutation PASS altered=3 detected_at=3
38V 1 30000 30 TYPE 0.92877 0.84308 0.76289
39V 1 30000 30 MAJ 0.97776 0.98870 0.89623
40L 1 30000 30 WORD-MI 0.37422
41L 1 30000 30 GOLD-MI 0.73022
experiments/jigsaw-transformer/results-v1/1792357299/seed-2.tsv +41 -0
……@@ -0,0 +1,41 @@
1C 2 3000 30 JT-V1 1000 0.53185 0.55463 0.42227
2C 2 3000 30 JT-V1 2000 0.55622 0.58562 0.43245
3C 2 3000 30 JT-V1 3000 0.62410 0.64242 0.59674
4V 2 3000 30 JT-V1 0.62410 0.64242 0.59674
5L 2 3000 30 ATTN-V1 0.49492
6V 2 3000 30 TYPE-A 0.52169 0.58030 0.53360
7A 2 3000 30 replay PASS events=3 first_mismatch=-1
8A 2 3000 30 mutation PASS altered=1 detected_at=1
9V 2 3000 30 TYPE 0.81337 0.67858 0.55126
10V 2 3000 30 MAJ 0.97729 0.98834 0.88527
11L 2 3000 30 WORD-MI 0.16166
12L 2 3000 30 GOLD-MI 0.69330
13C 2 10000 30 JT-V1 1000 0.53185 0.55463 0.42227
14C 2 10000 30 JT-V1 2000 0.55622 0.58562 0.43245
15C 2 10000 30 JT-V1 4000 0.65251 0.65746 0.40394
16C 2 10000 30 JT-V1 8000 0.74463 0.74153 0.62933
17C 2 10000 30 JT-V1 10000 0.77409 0.77141 0.74406
18V 2 10000 30 JT-V1 0.77409 0.77141 0.74406
19L 2 10000 30 ATTN-V1 0.58379
20V 2 10000 30 TYPE-A 0.67791 0.72137 0.69111
21A 2 10000 30 replay PASS events=5 first_mismatch=-1
22A 2 10000 30 mutation PASS altered=2 detected_at=2
23V 2 10000 30 TYPE 0.90581 0.80019 0.60353
24V 2 10000 30 MAJ 0.97879 0.98923 0.89409
25L 2 10000 30 WORD-MI 0.26885
26L 2 10000 30 GOLD-MI 0.72498
27C 2 30000 30 JT-V1 1000 0.53185 0.55463 0.42227
28C 2 30000 30 JT-V1 2000 0.55622 0.58562 0.43245
29C 2 30000 30 JT-V1 4000 0.65251 0.65746 0.40394
30C 2 30000 30 JT-V1 8000 0.74463 0.74153 0.62933
31C 2 30000 30 JT-V1 16000 0.79642 0.78121 0.59470
32C 2 30000 30 JT-V1 30000 0.84617 0.83670 0.56008
33V 2 30000 30 JT-V1 0.84617 0.83670 0.56008
34L 2 30000 30 ATTN-V1 0.62671
35V 2 30000 30 TYPE-A 0.79663 0.81006 0.50509
36A 2 30000 30 replay PASS events=6 first_mismatch=-1
37A 2 30000 30 mutation PASS altered=3 detected_at=3
38V 2 30000 30 TYPE 0.92451 0.82138 0.60421
39V 2 30000 30 MAJ 0.97775 0.98875 0.88934
40L 2 30000 30 WORD-MI 0.38086
41L 2 30000 30 GOLD-MI 0.73453
experiments/jigsaw-transformer/results-v1/1792357299/seed-3.tsv +41 -0
……@@ -0,0 +1,41 @@
1C 3 3000 30 JT-V1 1000 0.53489 0.54102 0.27537
2C 3 3000 30 JT-V1 2000 0.56374 0.57191 0.32476
3C 3 3000 30 JT-V1 3000 0.61895 0.62236 0.49256
4V 3 3000 30 JT-V1 0.61895 0.62236 0.49256
5L 3 3000 30 ATTN-V1 0.47409
6V 3 3000 30 TYPE-A 0.51535 0.56967 0.45061
7A 3 3000 30 replay PASS events=3 first_mismatch=-1
8A 3 3000 30 mutation PASS altered=1 detected_at=1
9V 3 3000 30 TYPE 0.79529 0.61468 0.75710
10V 3 3000 30 MAJ 0.97468 0.98651 0.86942
11L 3 3000 30 WORD-MI 0.15552
12L 3 3000 30 GOLD-MI 0.70371
13C 3 10000 30 JT-V1 1000 0.53489 0.54102 0.27537
14C 3 10000 30 JT-V1 2000 0.56374 0.57191 0.32476
15C 3 10000 30 JT-V1 4000 0.65698 0.66436 0.52233
16C 3 10000 30 JT-V1 8000 0.73919 0.73620 0.50744
17C 3 10000 30 JT-V1 10000 0.76423 0.74158 0.58390
18V 3 10000 30 JT-V1 0.76423 0.74158 0.58390
19L 3 10000 30 ATTN-V1 0.56515
20V 3 10000 30 TYPE-A 0.67024 0.68435 0.55210
21A 3 10000 30 replay PASS events=5 first_mismatch=-1
22A 3 10000 30 mutation PASS altered=2 detected_at=2
23V 3 10000 30 TYPE 0.92204 0.84151 0.77470
24V 3 10000 30 MAJ 0.97641 0.98763 0.88024
25L 3 10000 30 WORD-MI 0.27664
26L 3 10000 30 GOLD-MI 0.74009
27C 3 30000 30 JT-V1 1000 0.53489 0.54102 0.27537
28C 3 30000 30 JT-V1 2000 0.56374 0.57191 0.32476
29C 3 30000 30 JT-V1 4000 0.65698 0.66436 0.52233
30C 3 30000 30 JT-V1 8000 0.73919 0.73620 0.50744
31C 3 30000 30 JT-V1 16000 0.80245 0.79720 0.65562
32C 3 30000 30 JT-V1 30000 0.83872 0.81670 0.75034
33V 3 30000 30 JT-V1 0.83872 0.81670 0.75034
34L 3 30000 30 ATTN-V1 0.61535
35V 3 30000 30 TYPE-A 0.79603 0.79385 0.71854
36A 3 30000 30 replay PASS events=6 first_mismatch=-1
37A 3 30000 30 mutation PASS altered=3 detected_at=3
38V 3 30000 30 TYPE 0.92078 0.81209 0.56631
39V 3 30000 30 MAJ 0.97727 0.98819 0.88566
40L 3 30000 30 WORD-MI 0.37845
41L 3 30000 30 GOLD-MI 0.72961
experiments/jigsaw-transformer/results-v1/1792357299/seed-4.tsv +41 -0
……@@ -0,0 +1,41 @@
1C 4 3000 30 JT-V1 1000 0.50291 0.53564 0.44068
2C 4 3000 30 JT-V1 2000 0.60121 0.62236 0.41492
3C 4 3000 30 JT-V1 3000 0.61743 0.62383 0.52475
4V 4 3000 30 JT-V1 0.61743 0.62383 0.52475
5L 4 3000 30 ATTN-V1 0.48578
6V 4 3000 30 TYPE-A 0.51036 0.56611 0.47525
7A 4 3000 30 replay PASS events=3 first_mismatch=-1
8A 4 3000 30 mutation PASS altered=1 detected_at=1
9V 4 3000 30 TYPE 0.78722 0.61768 0.57763
10V 4 3000 30 MAJ 0.97339 0.98581 0.86237
11L 4 3000 30 WORD-MI 0.16105
12L 4 3000 30 GOLD-MI 0.71224
13C 4 10000 30 JT-V1 1000 0.50291 0.53564 0.44068
14C 4 10000 30 JT-V1 2000 0.60121 0.62236 0.41492
15C 4 10000 30 JT-V1 4000 0.65731 0.66806 0.55932
16C 4 10000 30 JT-V1 8000 0.75745 0.75856 0.51119
17C 4 10000 30 JT-V1 10000 0.76652 0.77456 0.50508
18V 4 10000 30 JT-V1 0.76652 0.77456 0.50508
19L 4 10000 30 ATTN-V1 0.56231
20V 4 10000 30 TYPE-A 0.67055 0.71817 0.45288
21A 4 10000 30 replay PASS events=5 first_mismatch=-1
22A 4 10000 30 mutation PASS altered=2 detected_at=2
23V 4 10000 30 TYPE 0.77178 0.53529 0.55458
24V 4 10000 30 MAJ 0.97560 0.98707 0.87458
25L 4 10000 30 WORD-MI 0.27889
26L 4 10000 30 GOLD-MI 0.73639
27C 4 30000 30 JT-V1 1000 0.50291 0.53564 0.44068
28C 4 30000 30 JT-V1 2000 0.60121 0.62236 0.41492
29C 4 30000 30 JT-V1 4000 0.65731 0.66806 0.55932
30C 4 30000 30 JT-V1 8000 0.75745 0.75856 0.51119
31C 4 30000 30 JT-V1 16000 0.80220 0.79713 0.77220
32C 4 30000 30 JT-V1 30000 0.85019 0.84458 0.59254
33V 4 30000 30 JT-V1 0.85019 0.84458 0.59254
34L 4 30000 30 ATTN-V1 0.62900
35V 4 30000 30 TYPE-A 0.80053 0.81824 0.55729
36A 4 30000 30 replay PASS events=6 first_mismatch=-1
37A 4 30000 30 mutation PASS altered=3 detected_at=3
38V 4 30000 30 TYPE 0.81074 0.54787 0.52203
39V 4 30000 30 MAJ 0.97478 0.98665 0.87051
40L 4 30000 30 WORD-MI 0.37526
41L 4 30000 30 GOLD-MI 0.73533
experiments/jigsaw-transformer/results-v1/1792357299/seed-5.tsv +41 -0
……@@ -0,0 +1,41 @@
1C 5 3000 30 JT-V1 1000 0.50511 0.53814 0.38505
2C 5 3000 30 JT-V1 2000 0.56858 0.58961 0.44424
3C 5 3000 30 JT-V1 3000 0.61451 0.61246 0.53707
4V 5 3000 30 JT-V1 0.61451 0.61246 0.53707
5L 5 3000 30 ATTN-V1 0.48871
6V 5 3000 30 TYPE-A 0.51737 0.56564 0.49657
7A 5 3000 30 replay PASS events=3 first_mismatch=-1
8A 5 3000 30 mutation PASS altered=1 detected_at=1
9V 5 3000 30 TYPE 0.87463 0.79210 0.75576
10V 5 3000 30 MAJ 0.97317 0.98562 0.87103
11L 5 3000 30 WORD-MI 0.16322
12L 5 3000 30 GOLD-MI 0.67959
13C 5 10000 30 JT-V1 1000 0.50511 0.53814 0.38505
14C 5 10000 30 JT-V1 2000 0.56858 0.58961 0.44424
15C 5 10000 30 JT-V1 4000 0.64889 0.66025 0.49844
16C 5 10000 30 JT-V1 8000 0.75895 0.77383 0.67913
17C 5 10000 30 JT-V1 10000 0.74958 0.74055 0.66417
18V 5 10000 30 JT-V1 0.74958 0.74055 0.66417
19L 5 10000 30 ATTN-V1 0.57325
20V 5 10000 30 TYPE-A 0.65556 0.68318 0.59502
21A 5 10000 30 replay PASS events=5 first_mismatch=-1
22A 5 10000 30 mutation PASS altered=2 detected_at=2
23V 5 10000 30 TYPE 0.91059 0.81057 0.78069
24V 5 10000 30 MAJ 0.97412 0.98618 0.87601
25L 5 10000 30 WORD-MI 0.27297
26L 5 10000 30 GOLD-MI 0.73756
27C 5 30000 30 JT-V1 1000 0.50511 0.53814 0.38505
28C 5 30000 30 JT-V1 2000 0.56858 0.58961 0.44424
29C 5 30000 30 JT-V1 4000 0.64889 0.66025 0.49844
30C 5 30000 30 JT-V1 8000 0.75895 0.77383 0.67913
31C 5 30000 30 JT-V1 16000 0.81288 0.78730 0.52710
32C 5 30000 30 JT-V1 30000 0.83115 0.82662 0.54455
33V 5 30000 30 JT-V1 0.83115 0.82662 0.54455
34L 5 30000 30 ATTN-V1 0.62343
35V 5 30000 30 TYPE-A 0.78598 0.80272 0.48847
36A 5 30000 30 replay PASS events=6 first_mismatch=-1
37A 5 30000 30 mutation PASS altered=3 detected_at=3
38V 5 30000 30 TYPE 0.92034 0.82919 0.75576
39V 5 30000 30 MAJ 0.97578 0.98729 0.88598
40L 5 30000 30 WORD-MI 0.38075
41L 5 30000 30 GOLD-MI 0.74128
experiments/jigsaw-transformer/results-v1/1792357299/shuf-1.tsv +29 -0
……@@ -0,0 +1,29 @@
1C 1 3000 30 JT-V1-SHUF 1000 0.41373 0.45952 0.31824
2C 1 3000 30 JT-V1-SHUF 2000 0.47039 0.51630 0.34717
3C 1 3000 30 JT-V1-SHUF 3000 0.50072 0.53363 0.44403
4V 1 3000 30 JT-V1-SHUF 0.50072 0.53363 0.44403
5L 1 3000 30 ATTN-V1-SHUF 0.36698
6V 1 3000 30 TYPE-A-SHUF 0.45467 0.51144 0.46604
7A 1 3000 30 replay PASS events=3 first_mismatch=-1
8A 1 3000 30 mutation PASS altered=1 detected_at=1
9C 1 10000 30 JT-V1-SHUF 1000 0.41373 0.45952 0.31824
10C 1 10000 30 JT-V1-SHUF 2000 0.47039 0.51630 0.34717
11C 1 10000 30 JT-V1-SHUF 4000 0.51285 0.55068 0.51132
12C 1 10000 30 JT-V1-SHUF 8000 0.56313 0.59144 0.57233
13C 1 10000 30 JT-V1-SHUF 10000 0.61289 0.62760 0.36981
14V 1 10000 30 JT-V1-SHUF 0.61289 0.62760 0.36981
15L 1 10000 30 ATTN-V1-SHUF 0.37373
16V 1 10000 30 TYPE-A-SHUF 0.53248 0.58014 0.31698
17A 1 10000 30 replay PASS events=5 first_mismatch=-1
18A 1 10000 30 mutation PASS altered=2 detected_at=2
19C 1 30000 30 JT-V1-SHUF 1000 0.41373 0.45952 0.31824
20C 1 30000 30 JT-V1-SHUF 2000 0.47039 0.51630 0.34717
21C 1 30000 30 JT-V1-SHUF 4000 0.51285 0.55068 0.51132
22C 1 30000 30 JT-V1-SHUF 8000 0.56313 0.59144 0.57233
23C 1 30000 30 JT-V1-SHUF 16000 0.65941 0.67281 0.43648
24C 1 30000 30 JT-V1-SHUF 30000 0.71109 0.72205 0.63459
25V 1 30000 30 JT-V1-SHUF 0.71109 0.72205 0.63459
26L 1 30000 30 ATTN-V1-SHUF 0.36524
27V 1 30000 30 TYPE-A-SHUF 0.64847 0.68637 0.60189
28A 1 30000 30 replay PASS events=6 first_mismatch=-1
29A 1 30000 30 mutation PASS altered=3 detected_at=3
experiments/jigsaw-transformer/results-v1/1792357299/shuf-2.tsv +29 -0
……@@ -0,0 +1,29 @@
1C 2 3000 30 JT-V1-SHUF 1000 0.43946 0.47954 0.39308
2C 2 3000 30 JT-V1-SHUF 2000 0.46175 0.48602 0.32587
3C 2 3000 30 JT-V1-SHUF 3000 0.49510 0.53965 0.48608
4V 2 3000 30 JT-V1-SHUF 0.49510 0.53965 0.48608
5L 2 3000 30 ATTN-V1-SHUF 0.37593
6V 2 3000 30 TYPE-A-SHUF 0.45275 0.52626 0.47794
7A 2 3000 30 replay PASS events=3 first_mismatch=-1
8A 2 3000 30 mutation PASS altered=1 detected_at=1
9C 2 10000 30 JT-V1-SHUF 1000 0.43946 0.47954 0.39308
10C 2 10000 30 JT-V1-SHUF 2000 0.46175 0.48602 0.32587
11C 2 10000 30 JT-V1-SHUF 4000 0.47883 0.51204 0.45485
12C 2 10000 30 JT-V1-SHUF 8000 0.54044 0.56809 0.34148
13C 2 10000 30 JT-V1-SHUF 10000 0.56113 0.59162 0.45418
14V 2 10000 30 JT-V1-SHUF 0.56113 0.59162 0.45418
15L 2 10000 30 ATTN-V1-SHUF 0.37769
16V 2 10000 30 TYPE-A-SHUF 0.49933 0.55270 0.46436
17A 2 10000 30 replay PASS events=5 first_mismatch=-1
18A 2 10000 30 mutation PASS altered=2 detected_at=2
19C 2 30000 30 JT-V1-SHUF 1000 0.43946 0.47954 0.39308
20C 2 30000 30 JT-V1-SHUF 2000 0.46175 0.48602 0.32587
21C 2 30000 30 JT-V1-SHUF 4000 0.47883 0.51204 0.45485
22C 2 30000 30 JT-V1-SHUF 8000 0.54044 0.56809 0.34148
23C 2 30000 30 JT-V1-SHUF 16000 0.63690 0.65505 0.42566
24C 2 30000 30 JT-V1-SHUF 30000 0.73193 0.73794 0.44399
25V 2 30000 30 JT-V1-SHUF 0.73193 0.73794 0.44399
26L 2 30000 30 ATTN-V1-SHUF 0.37553
27V 2 30000 30 TYPE-A-SHUF 0.65724 0.68983 0.39986
28A 2 30000 30 replay PASS events=6 first_mismatch=-1
29A 2 30000 30 mutation PASS altered=3 detected_at=3
experiments/jigsaw-transformer/results-v1/1792357299/shuf-3.tsv +29 -0
……@@ -0,0 +1,29 @@
1C 3 3000 30 JT-V1-SHUF 1000 0.41258 0.39085 0.17321
2C 3 3000 30 JT-V1-SHUF 2000 0.45359 0.49029 0.35047
3C 3 3000 30 JT-V1-SHUF 3000 0.46840 0.50049 0.35724
4V 3 3000 30 JT-V1-SHUF 0.46840 0.50049 0.35724
5L 3 3000 30 ATTN-V1-SHUF 0.37230
6V 3 3000 30 TYPE-A-SHUF 0.42339 0.48106 0.35995
7A 3 3000 30 replay PASS events=3 first_mismatch=-1
8A 3 3000 30 mutation PASS altered=1 detected_at=1
9C 3 10000 30 JT-V1-SHUF 1000 0.41258 0.39085 0.17321
10C 3 10000 30 JT-V1-SHUF 2000 0.45359 0.49029 0.35047
11C 3 10000 30 JT-V1-SHUF 4000 0.49433 0.52313 0.30311
12C 3 10000 30 JT-V1-SHUF 8000 0.56200 0.58574 0.34844
13C 3 10000 30 JT-V1-SHUF 10000 0.58060 0.60594 0.48038
14V 3 10000 30 JT-V1-SHUF 0.58060 0.60594 0.48038
15L 3 10000 30 ATTN-V1-SHUF 0.37506
16V 3 10000 30 TYPE-A-SHUF 0.51740 0.56471 0.45061
17A 3 10000 30 replay PASS events=5 first_mismatch=-1
18A 3 10000 30 mutation PASS altered=2 detected_at=2
19C 3 30000 30 JT-V1-SHUF 1000 0.41258 0.39085 0.17321
20C 3 30000 30 JT-V1-SHUF 2000 0.45359 0.49029 0.35047
21C 3 30000 30 JT-V1-SHUF 4000 0.49433 0.52313 0.30311
22C 3 30000 30 JT-V1-SHUF 8000 0.56200 0.58574 0.34844
23C 3 30000 30 JT-V1-SHUF 16000 0.65097 0.66737 0.38160
24C 3 30000 30 JT-V1-SHUF 30000 0.69793 0.70594 0.50947
25V 3 30000 30 JT-V1-SHUF 0.69793 0.70594 0.50947
26L 3 30000 30 ATTN-V1-SHUF 0.38652
27V 3 30000 30 TYPE-A-SHUF 0.62813 0.66632 0.46211
28A 3 30000 30 replay PASS events=6 first_mismatch=-1
29A 3 30000 30 mutation PASS altered=3 detected_at=3
experiments/jigsaw-transformer/results-v1/1792357299/shuf-4.tsv +29 -0
……@@ -0,0 +1,29 @@
1C 4 3000 30 JT-V1-SHUF 1000 0.46325 0.49672 0.38847
2C 4 3000 30 JT-V1-SHUF 2000 0.45417 0.49560 0.37627
3C 4 3000 30 JT-V1-SHUF 3000 0.46159 0.49092 0.30644
4V 4 3000 30 JT-V1-SHUF 0.46159 0.49092 0.30644
5L 4 3000 30 ATTN-V1-SHUF 0.37539
6V 4 3000 30 TYPE-A-SHUF 0.42244 0.47939 0.29356
7A 4 3000 30 replay PASS events=3 first_mismatch=-1
8A 4 3000 30 mutation PASS altered=1 detected_at=1
9C 4 10000 30 JT-V1-SHUF 1000 0.46325 0.49672 0.38847
10C 4 10000 30 JT-V1-SHUF 2000 0.45417 0.49560 0.37627
11C 4 10000 30 JT-V1-SHUF 4000 0.49922 0.54633 0.42915
12C 4 10000 30 JT-V1-SHUF 8000 0.56928 0.59287 0.33559
13C 4 10000 30 JT-V1-SHUF 10000 0.57176 0.60035 0.52203
14V 4 10000 30 JT-V1-SHUF 0.57176 0.60035 0.52203
15L 4 10000 30 ATTN-V1-SHUF 0.37929
16V 4 10000 30 TYPE-A-SHUF 0.51118 0.56660 0.49424
17A 4 10000 30 replay PASS events=5 first_mismatch=-1
18A 4 10000 30 mutation PASS altered=2 detected_at=2
19C 4 30000 30 JT-V1-SHUF 1000 0.46325 0.49672 0.38847
20C 4 30000 30 JT-V1-SHUF 2000 0.45417 0.49560 0.37627
21C 4 30000 30 JT-V1-SHUF 4000 0.49922 0.54633 0.42915
22C 4 30000 30 JT-V1-SHUF 8000 0.56928 0.59287 0.33559
23C 4 30000 30 JT-V1-SHUF 16000 0.64122 0.65234 0.46712
24C 4 30000 30 JT-V1-SHUF 30000 0.70106 0.72243 0.61831
25V 4 30000 30 JT-V1-SHUF 0.70106 0.72243 0.61831
26L 4 30000 30 ATTN-V1-SHUF 0.37734
27V 4 30000 30 TYPE-A-SHUF 0.62732 0.67862 0.55864
28A 4 30000 30 replay PASS events=6 first_mismatch=-1
29A 4 30000 30 mutation PASS altered=3 detected_at=3
experiments/jigsaw-transformer/results-v1/1792357299/shuf-5.tsv +29 -0
……@@ -0,0 +1,29 @@
1C 5 3000 30 JT-V1-SHUF 1000 0.43279 0.47256 0.39377
2C 5 3000 30 JT-V1-SHUF 2000 0.46174 0.50069 0.38754
3C 5 3000 30 JT-V1-SHUF 3000 0.46241 0.49562 0.22555
4V 5 3000 30 JT-V1-SHUF 0.46241 0.49562 0.22555
5L 5 3000 30 ATTN-V1-SHUF 0.37020
6V 5 3000 30 TYPE-A-SHUF 0.42190 0.48104 0.22741
7A 5 3000 30 replay PASS events=3 first_mismatch=-1
8A 5 3000 30 mutation PASS altered=1 detected_at=1
9C 5 10000 30 JT-V1-SHUF 1000 0.43279 0.47256 0.39377
10C 5 10000 30 JT-V1-SHUF 2000 0.46174 0.50069 0.38754
11C 5 10000 30 JT-V1-SHUF 4000 0.49062 0.50611 0.35078
12C 5 10000 30 JT-V1-SHUF 8000 0.54986 0.58308 0.41994
13C 5 10000 30 JT-V1-SHUF 10000 0.58301 0.61066 0.40810
14V 5 10000 30 JT-V1-SHUF 0.58301 0.61066 0.40810
15L 5 10000 30 ATTN-V1-SHUF 0.37665
16V 5 10000 30 TYPE-A-SHUF 0.51517 0.57210 0.38567
17A 5 10000 30 replay PASS events=5 first_mismatch=-1
18A 5 10000 30 mutation PASS altered=2 detected_at=2
19C 5 30000 30 JT-V1-SHUF 1000 0.43279 0.47256 0.39377
20C 5 30000 30 JT-V1-SHUF 2000 0.46174 0.50069 0.38754
21C 5 30000 30 JT-V1-SHUF 4000 0.49062 0.50611 0.35078
22C 5 30000 30 JT-V1-SHUF 8000 0.54986 0.58308 0.41994
23C 5 30000 30 JT-V1-SHUF 16000 0.64654 0.65240 0.54829
24C 5 30000 30 JT-V1-SHUF 30000 0.70267 0.72020 0.54081
25V 5 30000 30 JT-V1-SHUF 0.70267 0.72020 0.54081
26L 5 30000 30 ATTN-V1-SHUF 0.37932
27V 5 30000 30 TYPE-A-SHUF 0.63816 0.67936 0.50156
28A 5 30000 30 replay PASS events=6 first_mismatch=-1
29A 5 30000 30 mutation PASS altered=3 detected_at=3
experiments/jigsaw-transformer/results-v1/1792357299/src.sha256 +2 -0
……@@ -0,0 +1,2 @@
1cd923c3330538767cbd05bebf36646ecc4f104beaa915230fe57d165c01cc62c jt.c
22f4343faaa8bd25592fdadbc5bd38807f63d6823bcf85aff9eaeda50e96b8396 ../across-cluster/ac.c
experiments/jigsaw-transformer/results-v1/1792357299/summary.txt +39 -0
……@@ -0,0 +1,39 @@
1## Token classes (mean): V-measure / many-to-one / sense accuracy
2tier 3000
3 JT-V1 0.614 / 0.619 / 0.523
4 TYPE-A 0.512 / 0.567 / 0.482
5 TYPE 0.809 / 0.663 / 0.655
6 MAJ 0.975 / 0.987 / 0.875
7 JT-V1-SHUF 0.478 / 0.512 / 0.364
8 TYPE-A-SHUF 0.435 / 0.496 / 0.365
9tier 10000
10 JT-V1 0.761 / 0.757 / 0.630
11 TYPE-A 0.664 / 0.701 / 0.577
12 TYPE 0.883 / 0.755 / 0.695
13 MAJ 0.977 / 0.988 / 0.885
14 JT-V1-SHUF 0.582 / 0.607 / 0.447
15 TYPE-A-SHUF 0.515 / 0.567 / 0.422
16tier 30000
17 JT-V1 0.842 / 0.829 / 0.637
18 TYPE-A 0.795 / 0.804 / 0.593
19 TYPE 0.901 / 0.771 / 0.642
20 MAJ 0.977 / 0.988 / 0.886
21 JT-V1-SHUF 0.709 / 0.722 / 0.549
22 TYPE-A-SHUF 0.640 / 0.680 / 0.505
23
24## Parse link F1 (mean)
253000: ATTN-V1 0.485 WORD-MI 0.158 GOLD-MI 0.699
2610000: ATTN-V1 0.572 WORD-MI 0.272 GOLD-MI 0.734
2730000: ATTN-V1 0.625 WORD-MI 0.378 GOLD-MI 0.734
28
29## Continuous checkpoints, tier 30000 (mean V-measure)
30 1000:0.517 2000:0.580 4000:0.653 8000:0.746 16000:0.803 30000:0.842
31
32## Predictions
33J1: PASS — sense JT-V1 − TYPE-A at 30000: +0.044 [+0.030, +0.058]
34J1+: FAIL — sense JT-V1 − MAJ at 30000: -0.248 [-0.368, -0.129]
35J2: FAIL — V-measure JT-V1 0.842 (bar 0.7); − TYPE -0.059 [-0.127, +0.010]
36J3: PASS — ATTN-V1 − WORD-MI 3000: +0.327 [+0.319]; 10000: +0.300 [+0.283]; 30000: +0.247 [+0.237]
37J4': PASS — checkpoints from 4K: 0.653 0.746 0.803 0.842
38J5: PASS — 60/60 audit checks PASS
39J6: FAIL — shuffled sense JT-V1 − TYPE-A at 30000: +0.045 [+0.032, +0.057]
experiments/jigsaw-transformer/summarize-v1.py +55 -0
……@@ -0,0 +1,55 @@
1#!/usr/bin/env python3
2# Score a results-v1/<stamp>/ directory against PREREG-v1.md. Usage: summarize-v1.py results-v1/<stamp>
3import glob, math, os, sys
4
5T975 = {4: 2.776}
6def mean(x): return sum(x) / len(x)
7def ci(d):
8 n = len(d); m = mean(d); sd = math.sqrt(sum((v - m) ** 2 for v in d) / (n - 1))
9 h = T975.get(n - 1, 1.96) * sd / math.sqrt(n)
10 return m, m - h, m + h
11
12V, L, C, A = {}, {}, {}, []
13for f in sorted(glob.glob(os.path.join(sys.argv[1], '*.tsv'))):
14 for line in open(f):
15 p = line.rstrip('\n').split('\t')
16 if p[0] == 'V': V[(int(p[2]), p[4], int(p[1]))] = tuple(map(float, p[5:8]))
17 elif p[0] == 'L': L[(int(p[2]), p[4], int(p[1]))] = float(p[5])
18 elif p[0] == 'C': C[(int(p[2]), p[4], int(p[5]), int(p[1]))] = float(p[6])
19 elif p[0] == 'A': A.append((p[4], p[5]))
20seeds = sorted({k[2] for k in V}); tiers = sorted({k[0] for k in V}); top = max(tiers)
21out = []
22def say(s=''): out.append(s); print(s)
23def v(t, m, i): return [V[(t, m, s)][i] for s in seeds] # i: 0 vmeasure, 1 m1, 2 sense
24def l(t, m): return [L[(t, m, s)] for s in seeds]
25
26say('## Token classes (mean): V-measure / many-to-one / sense accuracy')
27for t in tiers:
28 say(f'tier {t}')
29 for m in ['JT-V1', 'TYPE-A', 'TYPE', 'MAJ', 'JT-V1-SHUF', 'TYPE-A-SHUF']:
30 if (t, m, seeds[0]) in V: say(f' {m:12s} ' + ' / '.join(f'{mean(v(t, m, i)):.3f}' for i in range(3)))
31say('\n## Parse link F1 (mean)')
32for t in tiers: say(f'{t}: ' + ' '.join(f'{m} {mean(l(t, m)):.3f}' for m in ['ATTN-V1', 'WORD-MI', 'GOLD-MI'] if (t, m, seeds[0]) in L))
33say('\n## Continuous checkpoints, tier ' + str(top) + ' (mean V-measure)')
34cks = sorted({k[2] for k in C if k[0] == top and k[1] == 'JT-V1'})
35say(' ' + ' '.join(f'{c}:{mean([C[(top, "JT-V1", c, s)] for s in seeds]):.3f}' for c in cks))
36
37say('\n## Predictions')
38def verdict(n, ok, d): say(f'{n}: {"PASS" if ok else "FAIL"} — {d}')
39c = ci([a - b for a, b in zip(v(top, 'JT-V1', 2), v(top, 'TYPE-A', 2))])
40verdict('J1', c[1] > 0, f'sense JT-V1 − TYPE-A at {top}: {c[0]:+.3f} [{c[1]:+.3f}, {c[2]:+.3f}]')
41c = ci([a - b for a, b in zip(v(top, 'JT-V1', 2), v(top, 'MAJ', 2))])
42verdict('J1+', c[1] > 0, f'sense JT-V1 − MAJ at {top}: {c[0]:+.3f} [{c[1]:+.3f}, {c[2]:+.3f}]')
43c = ci([a - b for a, b in zip(v(top, 'JT-V1', 0), v(top, 'TYPE', 0))]); m = mean(v(top, 'JT-V1', 0))
44verdict('J2', m >= 0.7 and c[1] > 0, f'V-measure JT-V1 {m:.3f} (bar 0.7); − TYPE {c[0]:+.3f} [{c[1]:+.3f}, {c[2]:+.3f}]')
45ok = True; cells = []
46for t in tiers:
47 c = ci([a - b for a, b in zip(l(t, 'ATTN-V1'), l(t, 'WORD-MI'))]); ok &= c[1] > 0
48 cells.append(f'{t}: {c[0]:+.3f} [{c[1]:+.3f}]')
49verdict('J3', ok, 'ATTN-V1 − WORD-MI ' + '; '.join(cells))
50seq = [mean([C[(top, 'JT-V1', ck, s)] for s in seeds]) for ck in cks if ck >= 4000]
51verdict("J4'", all(seq[i] <= seq[i + 1] for i in range(len(seq) - 1)), 'checkpoints from 4K: ' + ' '.join(f'{x:.3f}' for x in seq))
52verdict('J5', all(r == 'PASS' for _, r in A) and len(A) > 0, f'{sum(r == "PASS" for _, r in A)}/{len(A)} audit checks PASS')
53c = ci([a - b for a, b in zip(v(top, 'JT-V1-SHUF', 2), v(top, 'TYPE-A-SHUF', 2))])
54verdict('J6', not c[1] > 0, f'shuffled sense JT-V1 − TYPE-A at {top}: {c[0]:+.3f} [{c[1]:+.3f}, {c[2]:+.3f}]')
55open(os.path.join(sys.argv[1], 'summary.txt'), 'w').write('\n'.join(out) + '\n')