| … | … | @@ -0,0 +1,29 @@ |
| 1 | # Sense split with connectors pooled over several parses. Pre-registration | |
| 2 | ||
| 3 | 2026-10-18. Written and committed **before any code or run**. `RESULTS-v3diag.md`: the v3 split test is exact with | |
| 4 | the gold tree (precision 1.000, minority part 97% right) and collapses with the JT attention parse (precision | |
| 5 | 0.064, below the 0.133 base rate); the connector classes barely matter. `level2/RESULTS-soft.md`: pooling over 6 | |
| 6 | perturbed parses beats one parse. Question: does pooling the parse rescue the split test? | |
| 7 | ||
| 8 | ## Cells (training text at 30K, 5 seeds; split test exactly as PREREG-v3; connector classes = **gold** in every cell) | |
| 9 | ||
| 10 | | cell | links of each occurrence | | |
| 11 | |---|---| | |
| 12 | | **GG** | gold tree (reference; must reproduce `results-v3diag`) | | |
| 13 | | **JG1** | JT attention parse, max (Eisner over α_tj + α_jt; must reproduce `results-v3diag` JG) | | |
| 14 | | **JG6** | JT attention parse, **consensus of 6 perturbed parses** | | |
| 15 | | **PG1** | window-PMI parse over the JT labels (l2's layer-1 parse; gated PMI edge scores), max | | |
| 16 | | **PG6** | that PMI parse, consensus of 6 perturbed parses | | |
| 17 | ||
| 18 | **Perturbation:** each scored edge gets independent Gumbel(0, 1) noise times **0.3 × the mean positive edge | |
| 19 | score of that parser over the training text** (a relative scale, since attention scores lie in [0, 2] and PMI | |
| 20 | scores are in bits). Root edges stay 0. **Consensus:** a link is kept if it appears in ≥ 4 of the 6 parses. | |
| 21 | ||
| 22 | ## Predictions (means over seeds) | |
| 23 | ||
| 24 | - **F0 (control):** GG and JG1 reproduce the `results-v3diag` GG and JG rows exactly. | |
| 25 | - **F1:** JG6 split precision ≥ 2 × base rate. | |
| 26 | - **F2:** PG6 split precision ≥ 2 × base rate. | |
| 27 | - **F3:** JG6 split precision > JG1, and PG6 > PG1 (pooling raises precision for both parsers). | |
| 28 | ||
| 29 | Reported, no prediction: split recall and minority-part precision/recall per cell; PG1 vs JG1. |
| … | … | @@ -0,0 +1,32 @@ |
| 1 | # Level 2, Test B — choosing units. Pre-registration | |
| 2 | ||
| 3 | 2026-10-18. Written and committed **before any code or run**. Test A-soft: frequency already puts 95% of true phrase | |
| 4 | types in the top 3K; the open step is **choosing** which spans are units. The withdrawn amendment-1 scorer | |
| 5 | (weakest-split PMI) chose spans that cross phrases. Here the unit criterion is **assembly reliability**: a span type | |
| 6 | is a unit if its instances are usually complete assemblies across parses. | |
| 7 | ||
| 8 | ## Method (corpus, tiers, 10 seeds, held-out set, sources G and L, the 6 perturbed parses and counts n, h, a exactly | |
| 9 | as Test A-soft) | |
| 10 | ||
| 11 | For each held-out sentence, every span of length 2–6 whose type was seen in training with n ≥ 3 is a candidate | |
| 12 | under the arm's keep rule; candidates are chosen by the existing greedy selector (highest score first, then longer, | |
| 13 | nesting allowed, no crossing). Scored by bracket precision / recall / F1 against gold NP/PP spans of length ≤ 6. | |
| 14 | ||
| 15 | | arm | score | keep if | | |
| 16 | |---|---|---| | |
| 17 | | **CNT** | n | n ≥ 3 | | |
| 18 | | **HARD-R** | h / n | h / n > 0.5 | | |
| 19 | | **SOFT-R** | a / n | a / n > 0.5 | | |
| 20 | | **SOFT-RI** | (a / n) · i | a / n > 0.5 and i > 0.5 | | |
| 21 | ||
| 22 | i = share of 6 perturbed parses of the **held-out** sentence (same perturbation, separate random stream) in which | |
| 23 | this instance is complete. | |
| 24 | ||
| 25 | ## Predictions (paired over 10 seeds; "beats" = 95% CI of the paired difference excludes 0) | |
| 26 | ||
| 27 | - **G1 (reliability is a unit criterion):** for x ∈ {G, L}, SOFT-R beats CNT on F1 at every tier. | |
| 28 | - **G2 (several parses beat one):** for x ∈ {G, L}, SOFT-R beats HARD-R on F1 at 30K. | |
| 29 | - **G3 (the instance's own parses help):** for x ∈ {G, L}, SOFT-RI beats SOFT-R on F1 at 30K. | |
| 30 | - **G4 (useful):** SOFT-RI with G has F1 ≥ 0.5 at 30K (mean). | |
| 31 | ||
| 32 | **Control R0:** Test A lines and the Test A-soft R rows reproduce `results-soft/<stamp>` exactly. |