| … | … | @@ -0,0 +1,290 @@ |
| 1 | # Audit 2026-04-10 — Classifier Recovery + Rule Quality |
| 2 | |
| 3 | First-hand audit of the ratchet codebase and data, followed by two targeted |
| 4 | changes. This doc captures what we found, what we changed, what we deferred, |
| 5 | and where to pick up next. |
| 6 | |
| 7 | ## Scope |
| 8 | |
| 9 | Read the source of: |
| 10 | |
| 11 | - `pipeline.js` (fast end-of-session orchestrator) |
| 12 | - `ratchet.js` (forward-only git guard for principle changes) |
| 13 | - `llm-analyzer.js` (batch LLM classification of corrections via `claude --print --model haiku`) |
| 14 | - `loop-briefing.js` (SessionStart injection point) |
| 15 | - `rule-extractor.js` (regex-based rule distillation) |
| 16 | - `loop-detector.js` (loop extraction + `findSimilarLoops`) |
| 17 | - `fuse-signals.js` (18-sensor scoring, first ~450 lines) |
| 18 | |
| 19 | Plus data sampling from `loops.json` (307 loops), `learned-rules.json` |
| 20 | (953 rules pre-change, 1048 post-change), and `benchmark-history.json` |
| 21 | (5 daily snapshots). |
| 22 | |
| 23 | ## Findings |
| 24 | |
| 25 | ### Infrastructure that already existed and was being rebuilt from a bad summary |
| 26 | |
| 27 | An earlier planning session generated a set of "improvements" based on a |
| 28 | secondhand audit rather than first-hand reading. Three of those proposed |
| 29 | improvements turned out to already exist: |
| 30 | |
| 31 | - **LLM-augmented correction classification.** `llm-analyzer.js` (394 LOC) |
| 32 | batches user messages through Claude Haiku, tracks regex-vs-LLM |
| 33 | disagreements explicitly, maintains per-file cursors to avoid re-processing, |
| 34 | and records classifications in `corrections.log` with source tags |
| 35 | (`regex`/`llm`/`auto`) and confidence scores. The fast pipeline |
| 36 | (`pipeline.js`) does **not** run it — it's batch-only by design. |
| 37 | |
| 38 | - **Relevance-scored loop retrieval.** `loop-detector.js:220–250` |
| 39 | exposes `findSimilarLoops(currentFingerprint, pastLoops)` with project |
| 40 | weight 3, type weight 2, file-overlap weight 2× each, threshold ≥3, |
| 41 | top-3 matches. Exported. Called only from the CLI debug path |
| 42 | (`--match=...`) and nowhere else in the runtime. |
| 43 | |
| 44 | - **Auto-tuned regex patterns.** `tune-regex.js` produces `tuned-patterns.json`, |
| 45 | consumed by `fuse-signals.js:106` as the primary pattern source with |
| 46 | hardcoded fallbacks. The fuse-signals regex is not static. |
| 47 | |
| 48 | ### What's actually broken (from the data, not from theory) |
| 49 | |
| 50 | 1. **Project classifier silently drops ~15% of work.** |
| 51 | `classifyProject` was duplicated between `loop-detector.js` and |
| 52 | `loop-briefing.js` as a hardcoded substring allowlist of ~11 project names. |
| 53 | Anything not on the list returned `'unknown'` / `null`, and `loop-briefing.js` |
| 54 | exited early on unknown → no briefing. |
| 55 | - 51 of 305 loops (16%) were in the `unknown` bucket. |
| 56 | - 108 of 953 rules (11%) belonged to `unknown` and were dead-on-arrival. |
| 57 | - Unknown loops collided on fingerprint (`unknown:needs-rethink:` ×9, |
| 58 | `unknown:other:` ×8, `unknown:fix-failed:` ×8), so the dedup logic in |
| 59 | `rule-extractor.js` ate most of them — 8 of every 9 collapses produced |
| 60 | zero rules. |
| 61 | - Affected cwds included `/hyper-fleeky/hyper-deploy`, `/opencog-kb/repos`, |
| 62 | and the hypercore-kb root itself. |
| 63 | |
| 64 | 2. **35% of rules have the generic fallback antipattern.** |
| 65 | 336 of 953 rules carry the literal string `"Slow down and verify each step"`, |
| 66 | which is the default `buildAntipattern` returns when none of the 5 hardcoded |
| 67 | keyword checks match. |
| 68 | |
| 69 | 3. **34% of rules have no fix files.** |
| 70 | 324 of 953 rules have empty `action.fixFiles`. Their action description |
| 71 | falls back to `"Read the target files completely before making changes"`. |
| 72 | |
| 73 | 4. **14% of rules have BOTH problems.** |
| 74 | 134 rules are generic antipattern AND no fix files — truly unsalvageable |
| 75 | as-is. This is the `needs-investigation` tier. |
| 76 | |
| 77 | 5. **`findSimilarLoops` exists but is unwired from the runtime path.** |
| 78 | `loop-briefing.js` filters rules by exact project string equality and |
| 79 | sorts by `source.duration` — no relevance scoring. The relevance function |
| 80 | is imported only by the CLI debug mode. |
| 81 | |
| 82 | 6. **Correction regex over-fires on planning messages.** |
| 83 | Real loops in the data include "ok let's go over and audit the modules" — |
| 84 | those are instructions, not corrections. The `\b(stop)\b` pattern catches |
| 85 | phrases like "let's stop and rethink." The LLM analyzer catches this |
| 86 | asymmetrically (only in batch mode). |
| 87 | |
| 88 | 7. **Rule extractor id collisions on re-run.** |
| 89 | `rule-${project}-${type}-${rules.length}` uses the extraction-time index, |
| 90 | so re-running produces different IDs for equivalent rules, which the |
| 91 | id-based merge treats as new. The rule corpus grows on every pipeline |
| 92 | run even when no new loops occurred. Pre-existing issue, not caused by |
| 93 | the changes here, but worth fixing later. |
| 94 | |
| 95 | ## Changes shipped |
| 96 | |
| 97 | ### Change 1 — Shared project classifier with git-root walk |
| 98 | |
| 99 | **Files:** |
| 100 | - **new** `classify-project.js` (147 LOC) |
| 101 | - **mod** `loop-detector.js` — imports from shared classifier, local copy removed |
| 102 | - **mod** `loop-briefing.js` — imports from shared classifier, local copy removed |
| 103 | |
| 104 | **Strategy** (ordered): |
| 105 | 1. Hardcoded substring match — preserves historical classifications and |
| 106 | handles names that don't match a directory basename (`nvidia-rtx`). |
| 107 | 2. Walk parent directories from cwd looking for `.git`. Use that directory's |
| 108 | basename. If the basename is generic (`repos`, `src`, `code`, `build`, |
| 109 | `dist`, `lib`, `project`, `projects`, `work`, `workspace`, `tmp`, `temp`, |
| 110 | `test`, `tests`, `app`, `apps`), walk up one more level. |
| 111 | 3. Fall back to `'unknown'`. |
| 112 | |
| 113 | **Self-test: 12/12 cases passing**, including the previously-unknown cwds |
| 114 | (hyper-deploy, opencog-kb/repos, hypercore-kb root) and all the hardcoded |
| 115 | backcompat cases. |
| 116 | |
| 117 | **Dry-run against real data (before applying):** 48 of 51 unknown loops would |
| 118 | reclassify. Loops gained new buckets: `hypercore-kb` (+40), `hyper-deploy` (+4), |
| 119 | `hyper-identity` (+2), `opencog-kb` (+2). Three stubborn unknowns remained |
| 120 | (empty cwd fields in source data). |
| 121 | |
| 122 | **After pipeline run:** confirmed 48/51 recovery. The reclassified loops |
| 123 | generated 21 new rules in the new buckets. The 108 old `unknown` rules |
| 124 | remained in `learned-rules.json` as dead weight — expected, cleanup is a |
| 125 | separate migration pass. |
| 126 | |
| 127 | ### Change 2 — Rule quality field + injection priority |
| 128 | |
| 129 | **Files:** |
| 130 | - **mod** `rule-extractor.js` — adds `computeQuality(rule)`, exports it |
| 131 | alongside `QUALITY_RANK` and `GENERIC_ANTIPATTERN`. Computes quality at |
| 132 | rule creation time AND backfills all existing rules in the merged output |
| 133 | at save time, so pre-Change-2 rules get tagged on the next pipeline run. |
| 134 | - **mod** `loop-briefing.js` — imports `computeQuality`, sorts rules by |
| 135 | `(quality tier, loop duration)`, filters `needs-investigation` out of the |
| 136 | injectable list, surfaces held-back count in the briefing header, shows |
| 137 | a "caution" fallback message when every rule for a project is |
| 138 | `needs-investigation`. |
| 139 | |
| 140 | **Quality tiers:** |
| 141 | |
| 142 | ``` |
| 143 | specific — concrete symptom AND concrete fix files |
| 144 | partial — one of the two is missing, still injectable |
| 145 | needs-investigation — both antipattern and fixFiles are fallback values |
| 146 | ``` |
| 147 | |
| 148 | **Computed purely from existing fields:** |
| 149 | |
| 150 | ```js |
| 151 | function computeQuality (rule) { |
| 152 | const hasGenericAntipattern = rule.antipattern?.description === GENERIC_ANTIPATTERN |
| 153 | const hasNoFixFiles = !rule.action?.fixFiles || rule.action.fixFiles.length === 0 |
| 154 | if (hasGenericAntipattern && hasNoFixFiles) return 'needs-investigation' |
| 155 | if (hasGenericAntipattern || hasNoFixFiles) return 'partial' |
| 156 | return 'specific' |
| 157 | } |
| 158 | ``` |
| 159 | |
| 160 | **After backfill on 1048 rules:** |
| 161 | |
| 162 | | Quality | Count | Percentage | |
| 163 | |---|---|---| |
| 164 | | specific | 466 | 44.5% | |
| 165 | | partial | 439 | 41.9% | |
| 166 | | needs-investigation | 143 | 13.6% | |
| 167 | |
| 168 | **86.4% of rules are injectable**, 13.6% held back. Per-project ratios vary: |
| 169 | `proc-nodes` is 10/10 `needs-investigation` (all rules held back, caution |
| 170 | fallback will fire). `nvidia-rtx` is 63% `needs-investigation`. |
| 171 | `hypercore-kb` is 6/15 (40%). |
| 172 | |
| 173 | **Design principle applied:** reclassification preserves information, |
| 174 | suppression discards it. Held-back rules remain in `learned-rules.json` |
| 175 | and are counted in the briefing header ("N actionable, M need investigation"), |
| 176 | so the signal "historical friction exists here" is preserved even when |
| 177 | directives aren't. |
| 178 | |
| 179 | **Verified end-to-end:** ran `loop-briefing.js` from the `hypercore-kb` |
| 180 | directory after backfill. The briefing header reads |
| 181 | `LEARNED RULES (9 actionable, 6 need investigation):` and the 5 injected |
| 182 | rules all have at least one concrete signal (concrete fix files or a |
| 183 | concrete antipattern — not both missing). |
| 184 | |
| 185 | ## Changes deferred |
| 186 | |
| 187 | ### Change 3 — Wire `findSimilarLoops` into `loop-briefing.js` |
| 188 | |
| 189 | Not done. The function exists (`loop-detector.js:220`) and works. The wire-up |
| 190 | is ~5 lines (import and call after the project filter). Deferred because |
| 191 | after Change 1, cross-project retrieval is less urgent — most loops now have |
| 192 | specific project tags and the `hypercore-kb` bucket surfaces the work we're |
| 193 | actually doing. Revisit once we see whether cross-project rule transfer is |
| 194 | still a measurable gap. |
| 195 | |
| 196 | ### Cleanup of 108 dead `unknown` rules |
| 197 | |
| 198 | Not done. The rules remain in `learned-rules.json` with `quality: |
| 199 | needs-investigation` (most of them) and `project: unknown`, so they're held |
| 200 | back from injection anyway. A cleanup migration would drop them entirely but |
| 201 | loses the audit trail. Scoped separately; decide later whether to purge or |
| 202 | keep as history. |
| 203 | |
| 204 | ### Rule extractor id collision fix |
| 205 | |
| 206 | Not done. Pre-existing issue: `rule-${project}-${type}-${rules.length}` uses |
| 207 | extraction-time index, which changes on re-run, causing id mismatches and |
| 208 | duplicate accumulation. The rule count grew from ~462 to 1048 over 5 days |
| 209 | largely from this churn. A content-addressed id |
| 210 | (`rule-${hash(project+type+symptom+fingerprint)}`) would fix it. Separate |
| 211 | commit when we need it. |
| 212 | |
| 213 | ## Measurement caveats |
| 214 | |
| 215 | - **5 days of benchmark history total.** The ratchet's benchmarks |
| 216 | (`benchmark-history.json`) only go back to 2026-04-07. There is |
| 217 | statistically no room to measure Change 2's effect yet — tomorrow's |
| 218 | snapshot is the first one fully post-change. |
| 219 | - **Core metrics are essentially flat over 5 days.** novelRate 0.868–0.870, |
| 220 | density 0.209–0.212, avgResolution exactly 8.2 every day. Any movement |
| 221 | we'd attribute to Change 2 is within daily noise. |
| 222 | - **The fundamental counterfactual is unavailable.** N=1 per session, no |
| 223 | true A/B. We can only reason about distributions and populations. |
| 224 | - **Change 1 and Change 2 shipped on the same day.** Effects are |
| 225 | confounded until we add a shadow-harness or wait for weekly aggregates. |
| 226 | - **This audit session is itself being captured as a loop.** The live |
| 227 | session score was ~0.50 (yellow zone) because planning/audit conversations |
| 228 | with multiple "that's off, distill, rethink" exchanges match correction |
| 229 | regex. Not a Change-2 regression; just conversation shape. |
| 230 | |
| 231 | ## What we cannot yet tell from the dashboard |
| 232 | |
| 233 | The weekly chart's rightmost label is `2026-04-06`, which looks 5 days stale |
| 234 | but is actually the Sunday-start week that contains the current day. The |
| 235 | dashboard data IS current (generated 2026-04-10 21:43 EDT, same second as |
| 236 | the pipeline run). The label is a pre-existing UX artifact in `dashboard.js`'s |
| 237 | week-bucketing — unrelated to these changes. |
| 238 | |
| 239 | The live session card is measuring the audit session itself, not the effect |
| 240 | of the changes on future sessions. |
| 241 | |
| 242 | ## Next actions |
| 243 | |
| 244 | ### Immediate (cheap, bounded) |
| 245 | |
| 246 | - **Spot-check 20 random `needs-investigation` rules** (~15 min). Read them, |
| 247 | ask "would injecting this have helped?" If ≥10% are useful, the filter is |
| 248 | too aggressive. If <5%, it's well-calibrated. |
| 249 | - **Pin today's benchmark values as a baseline** (30 seconds). Compare in |
| 250 | 7 days. |
| 251 | |
| 252 | ### Short-term (next session if we resume) |
| 253 | |
| 254 | - **Wire `findSimilarLoops`** (Change 3) — 5 LOC, targets cross-project |
| 255 | transfer, easy to ship and revert. |
| 256 | - **Fix the rule extractor id collision** — content-addressed ids, prevents |
| 257 | the rule corpus growing on every re-run. |
| 258 | - **Clean up the 108 dead unknown rules** — either purge (cleaner) or |
| 259 | re-classify retroactively using the new classifier (preserves history). |
| 260 | |
| 261 | ### Medium-term (only if needed) |
| 262 | |
| 263 | - **Shadow harness** — dual-compute old and new briefings, log both, don't |
| 264 | affect injection. Real before/after measurement. ~60 LOC, 1-2 hours. |
| 265 | Skip unless the cheap measurements above are inconclusive. |
| 266 | |
| 267 | ### Not doing |
| 268 | |
| 269 | - AtomSpace integration |
| 270 | - Graphify-style idiom catalog over opencog repos |
| 271 | - Lexentry tool for Linas' 02-memory experiment |
| 272 | - Tree-sitter anything in the ratchet (natural language, no grammar) |
| 273 | - ollama / nomic-embed-text embeddings (concept tags + file tokens are cheaper) |
| 274 | |
| 275 | All of these were explored in the conversation and deferred. See the |
| 276 | conversation transcript if anyone wants the reasoning in detail. |
| 277 | |
| 278 | ## Files touched this session |
| 279 | |
| 280 | **Changed:** |
| 281 | - `knowledge-ratchet/classify-project.js` (new, 147 LOC) |
| 282 | - `knowledge-ratchet/loop-detector.js` (import shared classifier, -15 LOC local) |
| 283 | - `knowledge-ratchet/loop-briefing.js` (import shared classifier, add quality-aware injection) |
| 284 | - `knowledge-ratchet/rule-extractor.js` (add `computeQuality` + backfill + exports) |
| 285 | |
| 286 | **Not changed (but modified in working tree by other work — not staged):** |
| 287 | - `realtime-score.js` — pre-existing uncommitted fix for correction-burst counting |
| 288 | - Various data files under `data/` — pipeline artifacts |
| 289 | |
| 290 | **Total net: ~230 LOC across 4 files.** |