OverviewFilesCommitsIssuesDocsReleasesPeers

atomspace-layer-discovery commits / ae1fc3a

experiments/jigsaw-transformer: v3 diagnosis — gold parse makes the split test exact; the learned parse breaks it
D1 FAIL on recall only (0.213 vs 0.3; precision 1.000), D2 PASS (0.97/0.97). Gold tree with learned classes ~ gold; learned parse with gold classes collapses to below base rate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ae1fc3afb082993ac7df3c86b03f62ab0981dc8d
by plankatron <fleeky@gmail.com> · 2026-10-18T22:21:06.000Z

Changed files (9) +103 -0

experiments/jigsaw-transformer/RESULTS-v3diag.md +26 -0
……@@ -0,0 +1,26 @@
1# v3 diagnosis — results
2
3Run `results-v3diag/1792361993` (code `7918707`, 5 seeds, 30K). Scored against PREREG-v3diag (`c0a1768`) by `summarize-v3diag.py`.
4
5| cell (parse × connector classes) | words split | split precision | split recall | minority-part precision | minority-part recall |
6|---|---|---|---|---|---|
7| GG gold tree × gold | 31.8 | **1.000** | 0.213 | 0.970 | 0.974 |
8| GJ gold tree × JT labels | 28.0 | **0.993** | 0.186 | 0.976 | 0.981 |
9| JG JT parse × gold | 34.2 | 0.064 | 0.015 | 0.145 | 0.413 |
10| JJ JT parse × JT labels | 17.6 | 0.111 | 0.013 | 0.230 | 0.350 |
11
12Base rate (eligible words that are polysemous): 0.133.
13
14**Verdicts:** D1 **FAIL** (GG precision 1.000 passes its bar of 0.266; recall 0.213 misses 0.3). D2 **PASS**
15(0.970 / 0.974). D3, by the pre-registered rule: "the method fails".
16
17## What it says
18
191. **By the letter, the method fails D1 — on recall only.** With gold inputs it finds about a fifth of the
20 polysemous words. The χ² 0.999 gate over ~30–50 df has too little power for rare minority senses.
212. **What it finds is exactly right.** Every word it splits is polysemous, and the minority part is the
22 minority sense 97% of the time with 97% coverage.
233. **Not pre-registered (the D3 rule did not cover a recall-only D1 failure), but plain in the table: the parse is
24 what breaks it.** Swapping in the learned classes barely matters (GJ ≈ GG). Swapping in the learned parse
25 collapses precision to below the base rate (JG ≈ JJ). The v3 failure came from the JT attention parse, not
26 from the class connectors and not mainly from the method.
experiments/jigsaw-transformer/results-v3diag/1792361993/commit.txt +1 -0
……@@ -0,0 +1 @@
179187077d309de236308e0c1f12a4128550ae0ef
experiments/jigsaw-transformer/results-v3diag/1792361993/seed-1.tsv +13 -0
……@@ -0,0 +1,13 @@
1C 1 30000 30 JT-V2D 1000 0.45557 0.51938 0.42579
2C 1 30000 30 JT-V2D 2000 0.58058 0.61082 0.54717
3C 1 30000 30 JT-V2D 4000 0.67755 0.69027 0.62642
4C 1 30000 30 JT-V2D 8000 0.80670 0.78610 0.73270
5C 1 30000 30 JT-V2D 16000 0.84560 0.81596 0.75220
6C 1 30000 30 JT-V2D 30000 0.86165 0.82274 0.73585
7V 1 30000 30 JT-V2D 0.86165 0.82274 0.73585
8V 1 30000 30 JT-V2D-COLL 0.86595 0.82555 0.72390
9L 1 30000 30 ATTN-V2D 0.64111
10D 1 30000 30 GG 1139 32 32 150 0.96875 0.96875 32
11D 1 30000 30 GJ 1139 29 28 150 0.96230 0.99301 28
12D 1 30000 30 JG 1139 36 2 150 0.41099 0.52083 2
13D 1 30000 30 JJ 1139 16 2 150 0.42234 0.50926 2
experiments/jigsaw-transformer/results-v3diag/1792361993/seed-2.tsv +13 -0
……@@ -0,0 +1,13 @@
1C 2 30000 30 JT-V2D 1000 0.47334 0.53896 0.37814
2C 2 30000 30 JT-V2D 2000 0.53048 0.57892 0.40054
3C 2 30000 30 JT-V2D 4000 0.69388 0.68990 0.43585
4C 2 30000 30 JT-V2D 8000 0.79827 0.76693 0.63001
5C 2 30000 30 JT-V2D 16000 0.83613 0.80364 0.55329
6C 2 30000 30 JT-V2D 30000 0.86232 0.84202 0.56687
7V 2 30000 30 JT-V2D 0.86232 0.84202 0.56687
8V 2 30000 30 JT-V2D-COLL 0.86938 0.84719 0.55193
9L 2 30000 30 ATTN-V2D 0.64337
10D 2 30000 30 GG 1123 34 34 148 0.92402 0.93048 34
11D 2 30000 30 GJ 1123 31 31 148 0.96452 0.96226 31
12D 2 30000 30 JG 1123 37 1 148 0.10112 1.00000 1
13D 2 30000 30 JJ 1123 16 1 148 0.00645 0.26667 1
experiments/jigsaw-transformer/results-v3diag/1792361993/seed-3.tsv +13 -0
……@@ -0,0 +1,13 @@
1C 3 30000 30 JT-V2D 1000 0.46120 0.51020 0.26049
2C 3 30000 30 JT-V2D 2000 0.52876 0.56303 0.34574
3C 3 30000 30 JT-V2D 4000 0.67645 0.67582 0.49391
4C 3 30000 30 JT-V2D 8000 0.80803 0.78393 0.52977
5C 3 30000 30 JT-V2D 16000 0.84891 0.82264 0.68065
6C 3 30000 30 JT-V2D 30000 0.86702 0.82425 0.76252
7V 3 30000 30 JT-V2D 0.86702 0.82425 0.76252
8V 3 30000 30 JT-V2D-COLL 0.87397 0.82788 0.75372
9L 3 30000 30 ATTN-V2D 0.62998
10D 3 30000 30 GG 1123 28 28 149 1.00000 1.00000 28
11D 3 30000 30 GJ 1123 29 29 149 0.99666 0.99836 29
12D 3 30000 30 JG 1123 39 2 149 0.00786 0.11667 2
13D 3 30000 30 JJ 1123 21 2 149 0.46430 0.51071 2
experiments/jigsaw-transformer/results-v3diag/1792361993/seed-4.tsv +13 -0
……@@ -0,0 +1,13 @@
1C 4 30000 30 JT-V2D 1000 0.43152 0.50664 0.32407
2C 4 30000 30 JT-V2D 2000 0.58560 0.61635 0.44475
3C 4 30000 30 JT-V2D 4000 0.70651 0.70224 0.57017
4C 4 30000 30 JT-V2D 8000 0.81443 0.78421 0.68881
5C 4 30000 30 JT-V2D 16000 0.84054 0.82089 0.77695
6C 4 30000 30 JT-V2D 30000 0.86424 0.84626 0.58780
7V 4 30000 30 JT-V2D 0.86424 0.84626 0.58780
8V 4 30000 30 JT-V2D-COLL 0.87032 0.84899 0.58034
9L 4 30000 30 ATTN-V2D 0.64647
10D 4 30000 30 GG 1116 32 32 150 0.95565 0.96875 32
11D 4 30000 30 GJ 1116 27 27 150 0.96296 0.95600 27
12D 4 30000 30 JG 1116 25 1 150 0.00059 0.05556 1
13D 4 30000 30 JJ 1116 16 1 150 0.00144 0.11111 1
experiments/jigsaw-transformer/results-v3diag/1792361993/seed-5.tsv +13 -0
……@@ -0,0 +1,13 @@
1C 5 30000 30 JT-V2D 1000 0.44416 0.52313 0.45171
2C 5 30000 30 JT-V2D 2000 0.55967 0.60058 0.38069
3C 5 30000 30 JT-V2D 4000 0.70990 0.70916 0.61433
4C 5 30000 30 JT-V2D 8000 0.81669 0.80765 0.61308
5C 5 30000 30 JT-V2D 16000 0.84918 0.81016 0.71776
6C 5 30000 30 JT-V2D 30000 0.86272 0.83752 0.56137
7V 5 30000 30 JT-V2D 0.86272 0.83752 0.56137
8V 5 30000 30 JT-V2D-COLL 0.86348 0.83718 0.53894
9L 5 30000 30 ATTN-V2D 0.64747
10D 5 30000 30 GG 1122 33 33 151 1.00000 1.00000 33
11D 5 30000 30 GJ 1122 24 24 151 0.99588 0.99781 24
12D 5 30000 30 JG 1122 34 5 151 0.20446 0.37273 5
13D 5 30000 30 JJ 1122 19 4 151 0.25507 0.35281 4
experiments/jigsaw-transformer/results-v3diag/1792361993/src.sha256 +2 -0
……@@ -0,0 +1,2 @@
1cd2533ebbf28fc1a4f4fd9e318afcc7450663bd1d34985715edcefeedd338f6e jt.c
22f4343faaa8bd25592fdadbc5bd38807f63d6823bcf85aff9eaeda50e96b8396 ../across-cluster/ac.c
experiments/jigsaw-transformer/results-v3diag/1792361993/summary.txt +9 -0
……@@ -0,0 +1,9 @@
1cell eligible split precision recall base minor-prec minor-rec (words)
2GG 1124.6 31.8 1.000 0.213 0.133 0.970 0.974 (31.8)
3GJ 1124.6 28.0 0.993 0.186 0.133 0.976 0.981 (27.8)
4JG 1124.6 34.2 0.064 0.015 0.133 0.145 0.413 (2.2)
5JJ 1124.6 17.6 0.111 0.013 0.133 0.230 0.350 (2.0)
6
7D1: FAIL — GG split precision 1.000 (bar 0.266), recall 0.213 (bar 0.3)
8D2: PASS — GG minor-component precision 0.970, recall 0.974 (bars 0.5)
9D3: the method fails (D1 FAIL with gold inputs)