c0a1768 · atomspace-layer-discovery

Changed files (2) +62 -0

experiments/jigsaw-transformer/PREREG-v3diag.md +30 -0

…	…	@@ -0,0 +1,30 @@
	1	# v3 diagnosis — can the split test find senses with good inputs? Pre-registration
	2
	3	2026-10-18. Written and committed before any code or run. `RESULTS-v3.md`: the v3 split test found no senses
	4	(split precision at base rate). Either the method (2-component mixture of connector bags, χ² 0.999 gate,
	5	minority ≥ 0.05, ≥ 20 occurrences — unchanged from PREREG-v3) cannot find them, or its inputs (the learned
	6	parse and the learned connector classes) are too poor. This test swaps the inputs and keeps the method.
	7
	8	## Cells (training text at 30K, 5 seeds; V2-DJ trained as pre-registered for the learned inputs)
	9
	10	\| cell \| parse giving each occurrence's links \| connector class of each linked word \|
	11	\|---\|---\|---\|
	12	\| GG \| gold tree \| gold class \|
	13	\| GJ \| gold tree \| JT label (argmax q, final E-step) \|
	14	\| JG \| JT attention parse (Eisner over α, final E-step) \| gold class \|
	15	\| JJ \| JT attention parse \| JT label (≈ the v3 input, recomputed) \|
	16
	17	## Measures per cell
	18
	19	Eligible words, words split, split precision / recall / base rate (as PREREG-v3). For split polysemous words
	20	(each has exactly two gold classes; the minority sense = the less frequent one in training): **minor-component
	21	precision (share of the smaller component's occurrences that are the minority sense) and minor-component
	22	recall** (share of the minority-sense occurrences that land in the smaller component), averaged over words.
	23
	24	## Predictions
	25
	26	- D1 (the method works with good inputs): in GG, split precision ≥ 2 × base rate and split recall ≥ 0.3
	27	(the S1 bar), means over seeds.
	28	- D2 (the splits are the senses): in GG, mean minor-component precision ≥ 0.5 and recall ≥ 0.5.
	29	- D3 (attribution, no pass/fail): if D1 passes, the input whose substitution (GJ or JG vs GG) lowers split
	30	recall more is named the main failing input. If D1 fails, the method is named as failing.

experiments/level2/PREREG-soft.md +32 -0

…	…	@@ -0,0 +1,32 @@
	1	# Level 2, Test A-soft — rank by assembly-ness over several parses instead of deleting. Pre-registration
	2
	3	2026-10-18. Written and committed before any code or run. Test A and A-JT: a hard "complete assembly of the one
	4	layer-1 parse" filter deletes two thirds of true phrases (gold classes keep 0.343), because one parse's errors
	5	compound. Here the constraint orders the search instead of deleting from it, and evidence is pooled over several
	6	parses (Linas counts over several parses per sentence; `run-config` uses 6).
	7
	8	## Method (same corpus, tiers, 10 seeds, held-out set, gold NP/PP spans, span lengths 2–6 as Test A)
	9
	10	For layer-1 source x ∈ {G (gold classes), L (exchange classes)}: window-PMI gated scores as Test A. Each
	11	training sentence gets the max parse (as Test A) and k = 6 perturbed parses: every gated edge score gets
	12	independent Gumbel(0, 1) noise (in bits; perturb-and-MAP), root edges stay 0, Eisner as before. Per span type:
	13	total count n, hard count h (instances complete in the max parse), soft count a (sum over instances of the share
	14	of the 6 perturbed parses in which the instance is complete).
	15
	16	Rankings of all types seen in training (ties broken by n, then by first appearance):
	17	- CNT: by n (pooling only, no mating);
	18	- HARD: by h (Test A's M-x-comp as a ranking instead of a filter);
	19	- SOFT: by a.
	20
	21	Measures at budgets B ∈ {1K, 3K, 10K} types: R@B = share of held-out gold spans (length ≤ 6) whose type
	22	is in the top B; P@B = among all held-out spans (length 2–6) whose type is in the top B, the share that are
	23	gold. No held-out parse is used.
	24
	25	## Predictions (paired over 10 seeds; "beats" = 95% CI of the paired difference excludes 0)
	26
	27	- E1 (mating informs the order): for x ∈ {G, L}, at every tier, SOFT beats CNT on R@3K.
	28	- E2 (several parses beat one): for x ∈ {G, L}, at 30K, SOFT beats HARD on R@3K.
	29	- E3 (focus): for x ∈ {G, L}, at 30K, SOFT beats CNT on P@3K.
	30
	31	Control R0: all Test A lines reproduce `results/1792340082` exactly. Control Z: with noise scale 0, SOFT's
	32	ranking equals HARD's (checked once on seed 1, 3K, before the run).

atomspace-layer-discovery commits / c0a1768

Changed files (2) +62 -0