| … | … | @@ -0,0 +1,89 @@ |
| 1 | # Audit of the layer-discovery experiments against Linas's design — 2026-10-18 |
| 2 | |
| 3 | Scope: `experiments/across-cluster`, `level2` (Tests A, A-JT, A-soft, B), `jigsaw-transformer` (v0–v3, v3diag, |
| 4 | v3pool), checked against Linas's source (`repos/learn`, `repos/lg-atomese`, `repos/link-grammar`, |
| 5 | `repos/atomspace/opencog/sheaf`). Quotes verified on disk where marked ✓. |
| 6 | |
| 7 | ## 1. Why Linas's links are undirected |
| 8 | |
| 9 | - **He follows Yuret:** "it does not make a head-dependent distinction (all dependencies are bi-directional)… |
| 10 | the words w_l and w_r are connected by an unlabelled, un-directed edge" ✓ (`learn-lang-diary-part-one.lyx:381-399`). |
| 11 | - **Link Grammar links are un-oriented.** `+`/`-` on a connector means *word order* (the partner is to the |
| 12 | right/left), not head/dependent. Optional `h`/`d` head markers exist (LG ≥ 5.1) for languages that need them, not |
| 13 | English (`link-grammar-5.9.0/README.md:782-791`). |
| 14 | - **Jigsaw view:** connectors have polarity only so they mate correctly; "the sense of direction that they offer is |
| 15 | immaterial to the final construction" (`sheaf/docs/connectors-and-variables.lyx:~366`). A link is the *event* of |
| 16 | two connectors mating, symmetric between the pieces. |
| 17 | - **What is not symmetric is word order.** Pair MI is computed on ordered (left, right) pairs and is "manifestly not |
| 18 | symmetric in w_l and w_r" (`part-one.lyx:~266-283`; `matrix/frequency.scm:57`). |
| 19 | - INFERRED reason it is the right choice for unsupervised learning: co-occurrence statistics say *that* two words |
| 20 | attach, never *which* is the head. Head direction is a linguistic convention the data does not carry; adding it |
| 21 | invents a variable with no evidence (which is exactly what our parser did, §3.1). |
| 22 | |
| 23 | ## 2. How Linas's pipeline actually works (verified ✓ where cited) |
| 24 | |
| 25 | | stage | Linas | source | |
| 26 | |---|---|---| |
| 27 | | pair counts | from random planar parses (`LgDict "any"`) | `scm/pipe-parse/pipe-count.scm:203` | |
| 28 | | parse | `LgDict "dict-pair"`, cost = scale·MI + offset; all-positive costs give an **MST**, negative costs add links → **MPG, a planar graph with loops**; undirected, unlabeled; 3 linkages counted | `pipe-count.scm:214-243`; `dict-atomese/README.md:209-221`; `word-pairs.cc:271` | |
| 29 | | disjuncts | Section(germ word, ConnectorSeq of **linked words** with `-`/`+` by position) | ✓ `LGParseLink.cc:594-628` | |
| 30 | | classes | cluster word-disjunct vectors; a merge moves only the **shared** disjuncts into the class and leaves the rest of the word to join other classes — **senses are factored inside the merge** | ✓ `gram-class/README.md:190-194`; ✓ `agglo-mi-rank.scm:461-466` | |
| 31 | | next level | merge connectors along with words (Shapes/CrossSections); **re-parse with the classes** and re-classify; sense tagging (word + subscript); later idioms, entities, multi-sentence | `gram-class/README.md:219-221`; `part-seven.lyx:~351, 380-405`; `version-one/README.md:141, 228` | |
| 32 | | evaluation | **link** precision/recall/F1 against the LG English dictionary or a gold corpus; MI/entropy distributions; dupes (multi-class words) | `scm/attic/lg-compare.scm:446-506`; `part-one.lyx:~18919` | |
| 33 | |
| 34 | He never evaluates phrase brackets, and favours dependency over constituency (`part-one.lyx:~3738-3800`). He planned but |
| 35 | never ran "MST with and without clusters, see which is better" (`part-nine.lyx:275`). |
| 36 | |
| 37 | ## 3. Where our implementation deviates |
| 38 | |
| 39 | ### 3.1 Errors (wrong, not just different) |
| 40 | |
| 41 | 1. **Directed parse from symmetric scores.** Our layer-1 parser is Eisner (directed, projective) over symmetric edge |
| 42 | scores with every root edge 0, so all orientations of a tree score the same and direction is a tie-break: |
| 43 | 76–94% of arcs head-left vs 34% gold (`L2_DIR_DEBUG`, `level2/RESULTS-B.md`). |
| 44 | 2. **Directional "complete assembly" rule.** Test A's `comp` rule ("no outside token has its head inside") reads that |
| 45 | artifact. Every learned-parse `comp`/HARD/SOFT number in Tests A, A-JT, A-soft, B is biased. The Test A pre- |
| 46 | registration called the rule "exactly one free connector — Linas's surface"; that attribution was **mine and |
| 47 | wrong**: Linas's links have no head, so his free connector is undirected. |
| 48 | 3. **The gold-tree control (C5) could not catch it**, because gold trees carry real directions. |
| 49 | |
| 50 | Not affected: `conn` numbers, gold-tree controls, link-F1 parse scores (scored undirected), the jt disjuncts and the |
| 51 | v3/v3diag/v3pool split tests (links taken in both directions). |
| 52 | |
| 53 | ### 3.2 Deviations from Linas (each a different model, not a bug) |
| 54 | |
| 55 | | ours | Linas | effect | |
| 56 | |---|---|---| |
| 57 | | window pairs (±6) | pairs from random planar parses | different pair statistics (the port A–D reproduces Linas exactly, but no level experiment uses it) | |
| 58 | | gate: drop PMI ≤ 0 and G ≤ 10.83 | every pair, MI as cost (negative MI allowed) | our parse is a **forest** (unlinked words go to root); his spans the sentence | |
| 59 | | one max tree per sentence | MST or MPG with loops, 3 linkages | more links per word, richer disjuncts | |
| 60 | | jt disjuncts with **class** connectors | **word** connectors, merged with the words later | v3's split test fed already-mixed classes | |
| 61 | | exchange clustering (one class per word) | disjunct-vector clustering with partial merges | senses: ours bolted on (v3), his built in. Linas's clusterer is ported (`port/cluster.c`) but parked at the round-2 divergence and never used as layer 1 | |
| 62 | | level 2 = NP/PP spans, bracket F1 | connector classes + re-parse with classes; link F1 | we measured a target he does not have | |
| 63 | | K = 30 given | number of classes emerges | declared shortcut | |
| 64 | |
| 65 | ## 4. What we are missing |
| 66 | |
| 67 | 1. **The test Linas planned and never ran is the cleanest test of the core hypothesis.** "Does the discovered level |
| 68 | constrain the next search?" in his terms is: *do classes make the next parse better?* We already have half the |
| 69 | evidence: in `across-cluster`, the MI parse with gold classes scores link F1 0.734 vs 0.378 with words, and the JT |
| 70 | attention parse over learned classes 0.642 (`RESULTS-v2.md`). Classes do constrain the parse. What is missing is |
| 71 | the **loop**: parse → disjuncts → classes → re-parse with classes → re-cluster, measured by link F1 at each |
| 72 | round, with learned classes. |
| 73 | 2. **Constrain by re-expressing, not by filtering.** Our level-2 tests filter span types with one parse. Linas |
| 74 | constrains by re-expressing counts in class terms (pooling) and re-parsing. Our own earlier principle said the |
| 75 | same; the level-2 tests did not follow it. |
| 76 | 3. **The parse is the common bottleneck.** It cost most level-2 recall, and with a correct tree the split test is |
| 77 | exact (precision 1.000) while with a learned parse it is below base rate. Every result points at layer-1 parse |
| 78 | quality, which is what the re-parse loop targets. |
| 79 | 4. **Senses belong inside the clustering**, per disjunct. The split test works with good inputs but has low recall |
| 80 | (0.213) under a strict gate. |
| 81 | |
| 82 | ## 5. Recommendations (each needs its own pre-registration) |
| 83 | |
| 84 | 1. **Fix the parser to be Linas-faithful:** undirected; MI as cost without the sign/G gate; spanning MST, then an MPG |
| 85 | variant; counts over several linkages. Re-score the level-2 `comp` arms with the undirected rule (connected by |
| 86 | internal links, exactly one link crossing the boundary). |
| 87 | 2. **Run the re-parse loop** (Linas's planned, untested experiment) and measure link F1 and total MI per round. |
| 88 | 3. **Use Linas's clusterer as layer 1** (resolve the parked round-2 divergence), with word connectors. |
| 89 | 4. Keep NP/PP brackets as a secondary diagnostic only; add VP/S spans if brackets are kept. |