OverviewFilesCommitsIssuesDocsReleasesPeers

atomspace-layer-discovery commits / 4deb612

experiments/level2: Test DICT results — dictionary lifts the MST by +0.003; candidates hold +0.10 it cannot find
D1, D2 PASS (tiny), D3, D4 FAIL. Oracle of 31 candidates 0.837 vs 0.738 chosen. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4deb612ffd270cf08634c21892976f3eca1f8350
by plankatron <fleeky@gmail.com> · 2026-10-18T23:58:19.000Z

Changed files (14) +338 -0

experiments/level2/RESULTS-dict.md +22 -0
……@@ -0,0 +1,22 @@
1# Test DICT — parse with the learned dictionary — results
2
3Run `results-dict/1792367793` (code `62f0642`, 10 seeds). Scored against PREREG-dict (`fd3faf3`) by `summarize-dict.py`.
4
5| | prediction (30K) | verdict |
6|---|---|---|
7| D1 | with gold classes, the dictionary beats the pair-MI MST | **PASS**, but +0.0025 (0.7377 vs 0.7352) |
8| D2 | with learned classes (RP1), it beats their MST | **PASS**, but +0.0028 (0.7173 vs 0.7145) |
9| D3 | re-estimation helps (iteration 3 vs 1) | **FAIL** +0.0001 |
10| D4 | learned classes + dictionary beat gold classes + MST | **FAIL** −0.018 |
11
12Search ceiling (oracle: the best of the 31 candidates by gold, per sentence) at 30K: **0.837** with G, 0.815 with RP1.
13
14## What it says
15
161. **The dictionary does lift the pair-MI ceiling, but by a quarter of a point.** Both effects are reliable over 10
17 seeds and negligible in size.
182. **Search is not the limit; scoring is.** The candidate sets contain parses 0.10 better (0.837 vs 0.738), and the
19 dictionary cannot find them. Its disjunct statistics are learned from MST parses, so it mostly re-prefers what
20 the MST already chose, and Viterbi re-estimation then reinforces that (D3: no change over 3 iterations).
213. Open: a dictionary learned from *several* parses per sentence (Linas counts over several linkages, and Test A-soft
22 showed six parses beat one), or with word rather than class connectors, may not inherit the MST's choices.
experiments/level2/results-dict/1792367793/commit.txt +1 -0
……@@ -0,0 +1 @@
162f064268f26aaa9c2a5bbb70b241abb48ceec10
experiments/level2/results-dict/1792367793/seed-1.tsv +30 -0
……@@ -0,0 +1,30 @@
1# seed=1 V=1607 heldout_sents=2000
2GUARD C1 PASS tp=5002 kept=5002 gold=5002
3GUARD C5 PASS gold=5002 conn=5002 comp=5002
4# tier=3000 train_toks=22033
5DP 1 3000 G 0 0.72817 0.72817
6DP 1 3000 G 1 0.73119 0.83381
7DP 1 3000 G 2 0.73246 0.83246
8DP 1 3000 G 3 0.73151 0.83738
9DP 1 3000 RP1 0 0.60635 0.60635
10DP 1 3000 RP1 1 0.61119 0.76960
11DP 1 3000 RP1 2 0.61143 0.76810
12DP 1 3000 RP1 3 0.61103 0.77016
13# tier=10000 train_toks=72741
14DP 1 10000 G 0 0.72754 0.72754
15DP 1 10000 G 1 0.72929 0.83381
16DP 1 10000 G 2 0.73111 0.83230
17DP 1 10000 G 3 0.72976 0.83214
18DP 1 10000 RP1 0 0.63849 0.63849
19DP 1 10000 RP1 1 0.64341 0.77492
20DP 1 10000 RP1 2 0.64365 0.77595
21DP 1 10000 RP1 3 0.64254 0.77897
22# tier=30000 train_toks=217958
23DP 1 30000 G 0 0.73048 0.73048
24DP 1 30000 G 1 0.73325 0.83579
25DP 1 30000 G 2 0.73294 0.83325
26DP 1 30000 G 3 0.73254 0.83111
27DP 1 30000 RP1 0 0.70960 0.70960
28DP 1 30000 RP1 1 0.71175 0.81675
29DP 1 30000 RP1 2 0.71190 0.81722
30DP 1 30000 RP1 3 0.71262 0.81714
experiments/level2/results-dict/1792367793/seed-10.tsv +30 -0
……@@ -0,0 +1,30 @@
1# seed=10 V=1607 heldout_sents=2000
2GUARD C1 PASS tp=4978 kept=4978 gold=4978
3GUARD C5 PASS gold=4978 conn=4978 comp=4978
4# tier=3000 train_toks=21781
5DP 10 3000 G 0 0.70860 0.70860
6DP 10 3000 G 1 0.71195 0.82977
7DP 10 3000 G 2 0.71219 0.82873
8DP 10 3000 G 3 0.71171 0.82778
9DP 10 3000 RP1 0 0.59237 0.59237
10DP 10 3000 RP1 1 0.59309 0.76922
11DP 10 3000 RP1 2 0.59588 0.77082
12DP 10 3000 RP1 3 0.59509 0.77018
13# tier=10000 train_toks=72856
14DP 10 10000 G 0 0.72248 0.72248
15DP 10 10000 G 1 0.72559 0.83264
16DP 10 10000 G 2 0.72559 0.83360
17DP 10 10000 G 3 0.72551 0.83551
18DP 10 10000 RP1 0 0.63465 0.63465
19DP 10 10000 RP1 1 0.63856 0.77601
20DP 10 10000 RP1 2 0.63896 0.77497
21DP 10 10000 RP1 3 0.63848 0.77521
22# tier=30000 train_toks=218274
23DP 10 30000 G 0 0.72846 0.72846
24DP 10 30000 G 1 0.73101 0.83567
25DP 10 30000 G 2 0.73046 0.83496
26DP 10 30000 G 3 0.72990 0.83384
27DP 10 30000 RP1 0 0.70373 0.70373
28DP 10 30000 RP1 1 0.70772 0.80831
29DP 10 30000 RP1 2 0.70716 0.81270
30DP 10 30000 RP1 3 0.70748 0.81039
experiments/level2/results-dict/1792367793/seed-2.tsv +30 -0
……@@ -0,0 +1,30 @@
1# seed=2 V=1607 heldout_sents=2000
2GUARD C1 PASS tp=5023 kept=5023 gold=5023
3GUARD C5 PASS gold=5023 conn=5023 comp=5023
4# tier=3000 train_toks=21780
5DP 2 3000 G 0 0.71127 0.71127
6DP 2 3000 G 1 0.71295 0.82697
7DP 2 3000 G 2 0.71367 0.82304
8DP 2 3000 G 3 0.71447 0.82945
9DP 2 3000 RP1 0 0.57194 0.57194
10DP 2 3000 RP1 1 0.57354 0.76203
11DP 2 3000 RP1 2 0.57563 0.75859
12DP 2 3000 RP1 3 0.57587 0.76059
13# tier=10000 train_toks=72490
14DP 2 10000 G 0 0.72232 0.72232
15DP 2 10000 G 1 0.72504 0.82585
16DP 2 10000 G 2 0.72536 0.83289
17DP 2 10000 G 3 0.72464 0.83097
18DP 2 10000 RP1 0 0.70798 0.70798
19DP 2 10000 RP1 1 0.71055 0.81320
20DP 2 10000 RP1 2 0.71167 0.81544
21DP 2 10000 RP1 3 0.71167 0.81472
22# tier=30000 train_toks=218306
23DP 2 30000 G 0 0.73433 0.73433
24DP 2 30000 G 1 0.73721 0.83385
25DP 2 30000 G 2 0.73561 0.83313
26DP 2 30000 G 3 0.73673 0.83393
27DP 2 30000 RP1 0 0.71623 0.71623
28DP 2 30000 RP1 1 0.71863 0.81384
29DP 2 30000 RP1 2 0.71743 0.80927
30DP 2 30000 RP1 3 0.71799 0.81288
experiments/level2/results-dict/1792367793/seed-3.tsv +30 -0
……@@ -0,0 +1,30 @@
1# seed=3 V=1607 heldout_sents=2000
2GUARD C1 PASS tp=4908 kept=4908 gold=4908
3GUARD C5 PASS gold=4908 conn=4908 comp=4908
4# tier=3000 train_toks=21946
5DP 3 3000 G 0 0.70414 0.70414
6DP 3 3000 G 1 0.70796 0.83152
7DP 3 3000 G 2 0.71040 0.82998
8DP 3 3000 G 3 0.70837 0.83014
9DP 3 3000 RP1 0 0.56125 0.56125
10DP 3 3000 RP1 1 0.56588 0.74151
11DP 3 3000 RP1 2 0.56336 0.74151
12DP 3 3000 RP1 3 0.56377 0.73932
13# tier=10000 train_toks=72679
14DP 3 10000 G 0 0.73997 0.73997
15DP 3 10000 G 1 0.74484 0.83859
16DP 3 10000 G 2 0.74379 0.83924
17DP 3 10000 G 3 0.74314 0.84078
18DP 3 10000 RP1 0 0.71552 0.71552
19DP 3 10000 RP1 1 0.71690 0.81950
20DP 3 10000 RP1 2 0.71909 0.82104
21DP 3 10000 RP1 3 0.71649 0.81974
22# tier=30000 train_toks=218379
23DP 3 30000 G 0 0.72941 0.72941
24DP 3 30000 G 1 0.73274 0.83859
25DP 3 30000 G 2 0.73225 0.83818
26DP 3 30000 G 3 0.73241 0.83769
27DP 3 30000 RP1 0 0.70690 0.70690
28DP 3 30000 RP1 1 0.70950 0.80950
29DP 3 30000 RP1 2 0.70950 0.81202
30DP 3 30000 RP1 3 0.70926 0.81056
experiments/level2/results-dict/1792367793/seed-4.tsv +30 -0
……@@ -0,0 +1,30 @@
1# seed=4 V=1607 heldout_sents=2000
2GUARD C1 PASS tp=4900 kept=4900 gold=4900
3GUARD C5 PASS gold=4900 conn=4900 comp=4900
4# tier=3000 train_toks=21802
5DP 4 3000 G 0 0.73054 0.73054
6DP 4 3000 G 1 0.73266 0.83826
7DP 4 3000 G 2 0.73266 0.83656
8DP 4 3000 G 3 0.73266 0.83810
9DP 4 3000 RP1 0 0.57254 0.57254
10DP 4 3000 RP1 1 0.57514 0.74752
11DP 4 3000 RP1 2 0.57693 0.74744
12DP 4 3000 RP1 3 0.57717 0.75020
13# tier=10000 train_toks=72633
14DP 4 10000 G 0 0.74273 0.74273
15DP 4 10000 G 1 0.74435 0.83786
16DP 4 10000 G 2 0.74500 0.83729
17DP 4 10000 G 3 0.74452 0.83688
18DP 4 10000 RP1 0 0.64297 0.64297
19DP 4 10000 RP1 1 0.64768 0.78188
20DP 4 10000 RP1 2 0.64630 0.78229
21DP 4 10000 RP1 3 0.64647 0.77636
22# tier=30000 train_toks=217946
23DP 4 30000 G 0 0.73420 0.73420
24DP 4 30000 G 1 0.73794 0.83842
25DP 4 30000 G 2 0.73786 0.83656
26DP 4 30000 G 3 0.73753 0.83623
27DP 4 30000 RP1 0 0.71543 0.71543
28DP 4 30000 RP1 1 0.71901 0.81933
29DP 4 30000 RP1 2 0.71901 0.82055
30DP 4 30000 RP1 3 0.71917 0.81860
experiments/level2/results-dict/1792367793/seed-5.tsv +30 -0
……@@ -0,0 +1,30 @@
1# seed=5 V=1607 heldout_sents=2000
2GUARD C1 PASS tp=4953 kept=4953 gold=4953
3GUARD C5 PASS gold=4953 conn=4953 comp=4953
4# tier=3000 train_toks=21900
5DP 5 3000 G 0 0.70402 0.70402
6DP 5 3000 G 1 0.70773 0.83430
7DP 5 3000 G 2 0.70749 0.83237
8DP 5 3000 G 3 0.70773 0.83027
9DP 5 3000 RP1 0 0.56478 0.56478
10DP 5 3000 RP1 1 0.56857 0.75290
11DP 5 3000 RP1 2 0.56994 0.75347
12DP 5 3000 RP1 3 0.57018 0.75194
13# tier=10000 train_toks=73066
14DP 5 10000 G 0 0.73637 0.73637
15DP 5 10000 G 1 0.73935 0.83745
16DP 5 10000 G 2 0.73895 0.83704
17DP 5 10000 G 3 0.73959 0.83422
18DP 5 10000 RP1 0 0.70055 0.70055
19DP 5 10000 RP1 1 0.70587 0.80720
20DP 5 10000 RP1 2 0.70708 0.80752
21DP 5 10000 RP1 3 0.70741 0.80720
22# tier=30000 train_toks=218533
23DP 5 30000 G 0 0.74040 0.74040
24DP 5 30000 G 1 0.74298 0.83898
25DP 5 30000 G 2 0.74274 0.83737
26DP 5 30000 G 3 0.74322 0.84100
27DP 5 30000 RP1 0 0.72677 0.72677
28DP 5 30000 RP1 1 0.72975 0.81680
29DP 5 30000 RP1 2 0.72911 0.81696
30DP 5 30000 RP1 3 0.73048 0.81631
experiments/level2/results-dict/1792367793/seed-6.tsv +30 -0
……@@ -0,0 +1,30 @@
1# seed=6 V=1607 heldout_sents=2000
2GUARD C1 PASS tp=5030 kept=5030 gold=5030
3GUARD C5 PASS gold=5030 conn=5030 comp=5030
4# tier=3000 train_toks=22127
5DP 6 3000 G 0 0.71320 0.71320
6DP 6 3000 G 1 0.71597 0.82534
7DP 6 3000 G 2 0.71724 0.82582
8DP 6 3000 G 3 0.71605 0.82724
9DP 6 3000 RP1 0 0.57360 0.57360
10DP 6 3000 RP1 1 0.57779 0.74414
11DP 6 3000 RP1 2 0.57898 0.74541
12DP 6 3000 RP1 3 0.57906 0.74470
13# tier=10000 train_toks=73282
14DP 6 10000 G 0 0.72665 0.72665
15DP 6 10000 G 1 0.72895 0.82977
16DP 6 10000 G 2 0.72990 0.83048
17DP 6 10000 G 3 0.73014 0.83080
18DP 6 10000 RP1 0 0.69167 0.69167
19DP 6 10000 RP1 1 0.69405 0.79535
20DP 6 10000 RP1 2 0.69547 0.79717
21DP 6 10000 RP1 3 0.69389 0.79820
22# tier=30000 train_toks=217426
23DP 6 30000 G 0 0.73417 0.73417
24DP 6 30000 G 1 0.73670 0.83262
25DP 6 30000 G 2 0.73623 0.83476
26DP 6 30000 G 3 0.73631 0.83246
27DP 6 30000 RP1 0 0.69911 0.69911
28DP 6 30000 RP1 1 0.70109 0.80595
29DP 6 30000 RP1 2 0.70141 0.80571
30DP 6 30000 RP1 3 0.70173 0.80714
experiments/level2/results-dict/1792367793/seed-7.tsv +30 -0
……@@ -0,0 +1,30 @@
1# seed=7 V=1607 heldout_sents=2000
2GUARD C1 PASS tp=4942 kept=4942 gold=4942
3GUARD C5 PASS gold=4942 conn=4942 comp=4942
4# tier=3000 train_toks=22049
5DP 7 3000 G 0 0.74028 0.74028
6DP 7 3000 G 1 0.74213 0.83756
7DP 7 3000 G 2 0.74245 0.83353
8DP 7 3000 G 3 0.74116 0.83643
9DP 7 3000 RP1 0 0.62278 0.62278
10DP 7 3000 RP1 1 0.62761 0.76959
11DP 7 3000 RP1 2 0.62487 0.76717
12DP 7 3000 RP1 3 0.62342 0.76742
13# tier=10000 train_toks=73005
14DP 7 10000 G 0 0.73770 0.73770
15DP 7 10000 G 1 0.73875 0.83692
16DP 7 10000 G 2 0.73842 0.83523
17DP 7 10000 G 3 0.73834 0.83482
18DP 7 10000 RP1 0 0.71185 0.71185
19DP 7 10000 RP1 1 0.71354 0.81606
20DP 7 10000 RP1 2 0.71386 0.81735
21DP 7 10000 RP1 3 0.71515 0.81598
22# tier=30000 train_toks=217853
23DP 7 30000 G 0 0.74100 0.74100
24DP 7 30000 G 1 0.74334 0.83788
25DP 7 30000 G 2 0.74309 0.83515
26DP 7 30000 G 3 0.74414 0.83619
27DP 7 30000 RP1 0 0.73319 0.73319
28DP 7 30000 RP1 1 0.73633 0.81928
29DP 7 30000 RP1 2 0.73585 0.81751
30DP 7 30000 RP1 3 0.73488 0.81976
experiments/level2/results-dict/1792367793/seed-8.tsv +30 -0
……@@ -0,0 +1,30 @@
1# seed=8 V=1607 heldout_sents=2000
2GUARD C1 PASS tp=4825 kept=4825 gold=4825
3GUARD C5 PASS gold=4825 conn=4825 comp=4825
4# tier=3000 train_toks=21908
5DP 8 3000 G 0 0.71360 0.71360
6DP 8 3000 G 1 0.71747 0.83585
7DP 8 3000 G 2 0.71779 0.83347
8DP 8 3000 G 3 0.71714 0.83733
9DP 8 3000 RP1 0 0.60762 0.60762
10DP 8 3000 RP1 1 0.61411 0.77744
11DP 8 3000 RP1 2 0.61140 0.77777
12DP 8 3000 RP1 3 0.61264 0.77662
13# tier=10000 train_toks=72661
14DP 8 10000 G 0 0.73439 0.73439
15DP 8 10000 G 1 0.73751 0.84037
16DP 8 10000 G 2 0.73727 0.84086
17DP 8 10000 G 3 0.73603 0.84095
18DP 8 10000 RP1 0 0.70605 0.70605
19DP 8 10000 RP1 1 0.70884 0.81425
20DP 8 10000 RP1 2 0.70917 0.81704
21DP 8 10000 RP1 3 0.71024 0.81531
22# tier=30000 train_toks=217378
23DP 8 30000 G 0 0.73012 0.73012
24DP 8 30000 G 1 0.73176 0.83758
25DP 8 30000 G 2 0.73225 0.83684
26DP 8 30000 G 3 0.73357 0.83939
27DP 8 30000 RP1 0 0.73217 0.73217
28DP 8 30000 RP1 1 0.73513 0.82895
29DP 8 30000 RP1 2 0.73718 0.82690
30DP 8 30000 RP1 3 0.73488 0.82542
experiments/level2/results-dict/1792367793/seed-9.tsv +30 -0
……@@ -0,0 +1,30 @@
1# seed=9 V=1607 heldout_sents=2000
2GUARD C1 PASS tp=4964 kept=4964 gold=4964
3GUARD C5 PASS gold=4964 conn=4964 comp=4964
4# tier=3000 train_toks=22082
5DP 9 3000 G 0 0.73285 0.73285
6DP 9 3000 G 1 0.73630 0.83670
7DP 9 3000 G 2 0.73807 0.83879
8DP 9 3000 G 3 0.73598 0.83646
9DP 9 3000 RP1 0 0.57783 0.57783
10DP 9 3000 RP1 1 0.57719 0.75368
11DP 9 3000 RP1 2 0.57534 0.75352
12DP 9 3000 RP1 3 0.57662 0.75505
13# tier=10000 train_toks=73202
14DP 9 10000 G 0 0.74274 0.74274
15DP 9 10000 G 1 0.74491 0.83943
16DP 9 10000 G 2 0.74419 0.84289
17DP 9 10000 G 3 0.74427 0.84120
18DP 9 10000 RP1 0 0.69471 0.69471
19DP 9 10000 RP1 1 0.69552 0.81055
20DP 9 10000 RP1 2 0.69447 0.81160
21DP 9 10000 RP1 3 0.69608 0.81305
22# tier=30000 train_toks=217891
23DP 9 30000 G 0 0.74942 0.74942
24DP 9 30000 G 1 0.75038 0.83992
25DP 9 30000 G 2 0.75014 0.84136
26DP 9 30000 G 3 0.75038 0.84064
27DP 9 30000 RP1 0 0.70228 0.70228
28DP 9 30000 RP1 1 0.70413 0.81313
29DP 9 30000 RP1 2 0.70437 0.81466
30DP 9 30000 RP1 3 0.70582 0.81176
experiments/level2/results-dict/1792367793/src.sha256 +2 -0
……@@ -0,0 +1,2 @@
1c5d77d6ed343a2928e677d52ea1acc2245df33cc7f4bb0e9e4cd4c7c50e70c7c l2.c
22f4343faaa8bd25592fdadbc5bd38807f63d6823bcf85aff9eaeda50e96b8396 ../across-cluster/ac.c
experiments/level2/results-dict/1792367793/summary.txt +13 -0
……@@ -0,0 +1,13 @@
1## Link F1 (mean): MST (iteration 0), DICT iterations 1-3, [oracle of the candidate set]
23000 G: MST 0.7187 it1 0.7216 [0.833] it2 0.7224 [0.831] it3 0.7217 [0.833]
33000 RP1: MST 0.5851 it1 0.5884 [0.759] it2 0.5884 [0.758] it3 0.5885 [0.759]
410000 G: MST 0.7333 it1 0.7359 [0.835] it2 0.7360 [0.836] it3 0.7356 [0.836]
510000 RP1: MST 0.6844 it1 0.6875 [0.801] it2 0.6880 [0.802] it3 0.6878 [0.801]
630000 G: MST 0.7352 it1 0.7377 [0.837] it2 0.7374 [0.836] it3 0.7377 [0.836]
730000 RP1: MST 0.7145 it1 0.7173 [0.815] it2 0.7173 [0.815] it3 0.7174 [0.815]
8
9## Predictions (at 30000)
10D1: PASS — G: DICT it1 − G-MST +0.0025 [+0.0020, +0.0031]
11D2: PASS — RP1: DICT it1 − RP1-MST +0.0028 [+0.0023, +0.0033]
12D3: FAIL — RP1: it3 − it1 +0.0001 [-0.0005, +0.0008]
13D4: FAIL — RP1 it3 − G-MST -0.0178 [-0.0273, -0.0082]