e7ed82d · knowledge-ratchet

Changed files (3) +163 -44

README.md +2 -2

…	…	@@ -50,8 +50,8 @@ node benchmark-ratchet.js
50	50	\| Novel correction rate \| 86.9% \| >90% \| Old mistakes eliminated (higher = better) \|
51	51	\| Correction density \| 21.2% \| <15% \| Corrections per turn (lower = better) \|
52	52	\| Loop resolution \| 8.2 turns \| <6 turns \| How fast loops resolve (lower = better) \|
53		\| Learned rules \| 1048 \| growing \| Auto-extracted from resolved loops \|
54		\| Rule quality \| 44.5% specific / 41.9% partial / 13.6% needs-investigation \| more specific \| Quality-aware injection prioritizes actionable rules \|
	53	\| Learned rules \| 271 \| growing \| Auto-extracted from resolved loops (content-addressed, idempotent across runs) \|
	54	\| Rule quality \| 47.2% specific / 39.9% partial / 12.9% needs-investigation \| more specific \| Quality-aware injection prioritizes actionable rules \|
55	55
56	56	## Architecture
57	57

docs/audit-2026-04-10.md +82 -32

…	…	@@ -182,33 +182,83 @@ directory after backfill. The briefing header reads
182	182	rules all have at least one concrete signal (concrete fix files or a
183	183	concrete antipattern — not both missing).
184	184
185		## Changes deferred
186
187		### Change 3 — Wire `findSimilarLoops` into `loop-briefing.js`
188
189		Not done. The function exists (`loop-detector.js:220`) and works. The wire-up
190		is ~5 lines (import and call after the project filter). Deferred because
191		after Change 1, cross-project retrieval is less urgent — most loops now have
192		specific project tags and the `hypercore-kb` bucket surfaces the work we're
193		actually doing. Revisit once we see whether cross-project rule transfer is
194		still a measurable gap.
195
196		### Cleanup of 108 dead `unknown` rules
197
198		Not done. The rules remain in `learned-rules.json` with `quality:
199		needs-investigation` (most of them) and `project: unknown`, so they're held
200		back from injection anyway. A cleanup migration would drop them entirely but
201		loses the audit trail. Scoped separately; decide later whether to purge or
202		keep as history.
203
204		### Rule extractor id collision fix
205
206		Not done. Pre-existing issue: `rule-${project}-${type}-${rules.length}` uses
207		extraction-time index, which changes on re-run, causing id mismatches and
208		duplicate accumulation. The rule count grew from ~462 to 1048 over 5 days
209		largely from this churn. A content-addressed id
210		(`rule-${hash(project+type+symptom+fingerprint)}`) would fix it. Separate
211		commit when we need it.
	185	## Change 3 — Wire `findSimilarLoops` into `loop-briefing.js`
	186
	187	Still deferred. The function exists (`loop-detector.js:220`) and works.
	188	The wire-up is ~5 lines (import and call after the project filter).
	189	After Change 1, cross-project retrieval is less urgent — most loops
	190	now have specific project tags. Revisit once we see whether cross-project
	191	rule transfer is still a measurable gap.
	192
	193	## Change 4 — Content-addressed rule IDs + dedup pass
	194
	195	Added after a spot-check sample of 20 `needs-investigation` rules
	196	revealed they were only 4 distinct underlying loops duplicated 3–6 times
	197	each. Deeper analysis showed the entire corpus was inflated: **1048 rule
	198	entries corresponded to only 270 distinct loop fingerprints** (avg 3.88x
	199	duplication, with some loops having up to 9 copies).
	200
	201	### Root cause
	202
	203	`rule-extractor.js` generated IDs as `rule-${project}-${type}-${rules.length}`
	204	where `rules.length` was the extraction-time index. Every pipeline run
	205	produced different indexes (loops get reordered, new ones appear), so the
	206	same underlying loop generated a new rule ID on every run. The id-based
	207	merge in `mergeRules` treated each as a new rule, so the corpus grew
	208	linearly with pipeline invocations.
	209
	210	### Fix
	211
	212	- New `ruleId(loop)` helper in `rule-extractor.js`: derives the ID from
	213	`md5(loop.fingerprint).slice(0,8)`, giving a stable content-addressed
	214	ID. Format: `rule-${project}-${type}-${8-hex-hash}`. Same loop → same
	215	ID → merge is idempotent.
	216	- Dedup pass in `main()` groups the merged rule list by
	217	`source.loopFingerprint`, keeps one representative per fingerprint
	218	(tiebreak: most recent `source.date`), and normalizes all IDs to the
	219	new content-addressed form. This handles both one-shot migration of
	220	the pre-Change-4 corpus and defensive future-proofing.
	221
	222	### Verification
	223
	224	- Run 1 (one-shot collapse): 1048 rules → 271 rules (988 duplicates
	225	dropped). Pipeline took 12.1s due to initial collapse workload.
	226	- Run 2 (idempotence check): 271 → 271 (0 duplicates dropped).
	227	Pipeline back to 2.6s. Content-addressed IDs mean the same corpus
	228	reproduces exactly across re-runs.
	229	- Quality ratios preserved: 128 specific (47.2%), 108 partial (39.9%),
	230	35 needs-investigation (12.9%). Ratios are essentially identical to
	231	pre-collapse because duplication was uniform across tiers.
	232	- hypercore-kb briefing unchanged: still shows "LEARNED RULES
	233	(9 actionable, 6 need investigation)" — proof that Change 2's filter
	234	isn't distorted by the dedup.
	235
	236	### Side effects
	237
	238	- The README's "1048 learned rules" (from before Change 4) is obsolete.
	239	Actual count is 271, which is the real knowledge base size.
	240	- The dashboard's "Learned Rules" trend line will show a dramatic drop
	241	on the next snapshot. This is not a regression — it's the corpus being
	242	truthful for the first time.
	243	- The benchmark history still shows old inflated numbers (462 → 498 →
	244	648 → 953 → 1048 → 271). Future entries will be stable.
	245
	246	## Deferred
	247
	248	### Cleanup of 21 remaining `unknown` rules
	249
	250	The 108-rule unknown bucket has collapsed to 21 after Change 4's dedup
	251	(most of the 108 were duplicates of the same underlying unknown loops).
	252	They remain in the corpus as `needs-investigation` and are held back from
	253	injection by Change 2. A cleanup migration would drop them entirely but
	254	loses the audit trail. Scoped separately.
	255
	256	### Shadow harness for real measurement
	257
	258	Still not built. The dashboard's benchmark trend is only 5 days old and
	259	now has a synthetic discontinuity from Change 4's collapse. Real before/
	260	after measurement of the changes requires either waiting 1-2 weeks for a
	261	clean history or building the shadow harness.
212	262
213	263	## Measurement caveats
214	264
…	…	@@ -278,13 +328,13 @@ conversation transcript if anyone wants the reasoning in detail.
278	328	## Files touched this session
279	329
280	330	Changed:
281		- `knowledge-ratchet/classify-project.js` (new, 147 LOC)
282		- `knowledge-ratchet/loop-detector.js` (import shared classifier, -15 LOC local)
283		- `knowledge-ratchet/loop-briefing.js` (import shared classifier, add quality-aware injection)
284		- `knowledge-ratchet/rule-extractor.js` (add `computeQuality` + backfill + exports)
	331	- `knowledge-ratchet/classify-project.js` (new, 147 LOC) — Change 1
	332	- `knowledge-ratchet/loop-detector.js` — imports shared classifier
	333	- `knowledge-ratchet/loop-briefing.js` — imports shared classifier + quality-aware injection
	334	- `knowledge-ratchet/rule-extractor.js` — `computeQuality` + backfill + content-addressed IDs + dedup pass
285	335
286	336	Not changed (but modified in working tree by other work — not staged):
287	337	- `realtime-score.js` — pre-existing uncommitted fix for correction-burst counting
288	338	- Various data files under `data/` — pipeline artifacts
289	339
290		Total net: ~230 LOC across 4 files.
	340	Total net: ~280 LOC across 4 files across 3 code commits.

rule-extractor.js +79 -10

…	…	@@ -19,11 +19,34 @@
19	19
20	20	const fs = require('fs')
21	21	const path = require('path')
	22	const crypto = require('crypto')
22	23
23	24	const DATA_DIR = path.join(__dirname, 'data')
24	25	const LOOPS_FILE = path.join(DATA_DIR, 'loops.json')
25	26	const RULES_FILE = path.join(DATA_DIR, 'learned-rules.json')
26	27
	28	// ─── Content-addressed rule IDs ───
	29	// Previously: `rule-${project}-${type}-${rules.length}` — index-based,
	30	// unstable across extraction runs (rules.length shifts when loops are
	31	// reordered or new ones appear). This caused merge-based dedup to
	32	// fail: the same underlying loop produced a new rule ID on every
	33	// pipeline run, and mergeRules treated each as a new rule. Result:
	34	// corpus inflation to ~4x the actual number of distinct loops.
	35	//
	36	// Fix: derive the ID from the loop's fingerprint (which is stable for
	37	// a given loop). Same loop → same ID → merge is idempotent.
	38	//
	39	// The fingerprint is already generated by loop-detector.js:buildFingerprint
	40	// and stored on every rule as source.loopFingerprint.
	41
	42	function ruleId (ruleOrLoop) {
	43	const fp = ruleOrLoop.fingerprint \|\| ruleOrLoop.source?.loopFingerprint \|\| ''
	44	const project = ruleOrLoop.project \|\| ruleOrLoop.trigger?.project \|\| 'unknown'
	45	const type = ruleOrLoop.type \|\| ruleOrLoop.trigger?.type \|\| 'other'
	46	const hash = crypto.createHash('md5').update(fp).digest('hex').slice(0, 8)
	47	return `rule-${project}-${type}-${hash}`
	48	}
	49
27	50	function loadJSON (file) {
28	51	if (!fs.existsSync(file)) return null
29	52	try { return JSON.parse(fs.readFileSync(file, 'utf8')) }
…	…	@@ -104,9 +127,10 @@ function extractRules (loops) {
104	127	const fixFiles = (loop.resolution.files \|\| []).map(f => path.basename(f))
105	128	const approval = loop.resolution.approval \|\| null
106	129
107		// Build the rule
	130	// Build the rule — id is content-addressed via ruleId(loop) so the
	131	// same loop always produces the same id across extraction runs
108	132	const rule = {
109		id: `rule-${loop.project}-${loop.type}-${rules.length}`,
	133	id: ruleId(loop),
110	134	trigger: {
111	135	project: loop.project,
112	136	type: loop.type,
…	…	@@ -213,17 +237,60 @@ function main () {
213	237	const newRules = extractRules(loopsData.loops \|\| [])
214	238	const { merged, added } = mergeRules(newRules, existingRules)
215	239
216		// Backfill quality on every rule in the merged output. This catches
	240	// ─── Content-addressed dedup pass ───
	241	// Group by loop fingerprint, collapse duplicates, and normalize all
	242	// IDs to the content-addressed form. This handles:
	243	// - One-shot migration: pre-Change-4 corpora have index-based IDs
	244	// like `rule-dc-terrain-other-3` which don't survive re-extraction,
	245	// so the corpus accumulates ~4x duplicates. This pass collapses
	246	// them to one representative per fingerprint.
	247	// - Defensive: future pipeline runs stay idempotent. If anything
	248	// ever produces a duplicate again, it gets collapsed here.
	249	// Rules without a fingerprint are kept as-is (can't dedup them safely).
	250	// When duplicates exist, the rule with the most recent source.date wins.
	251	const byFingerprint = new Map()
	252	const noFingerprint = []
	253	let duplicatesDropped = 0
	254
	255	for (const rule of merged) {
	256	const fp = rule.source?.loopFingerprint
	257	if (!fp) {
	258	noFingerprint.push(rule)
	259	continue
	260	}
	261	// Normalize id to content-addressed form
	262	rule.id = ruleId({
	263	project: rule.trigger.project,
	264	type: rule.trigger.type,
	265	fingerprint: fp
	266	})
	267	const existing = byFingerprint.get(fp)
	268	if (!existing) {
	269	byFingerprint.set(fp, rule)
	270	} else {
	271	duplicatesDropped++
	272	// Tiebreak: keep the one with the most recent source.date
	273	const existingDate = existing.source?.date \|\| ''
	274	const newDate = rule.source?.date \|\| ''
	275	if (newDate > existingDate) {
	276	byFingerprint.set(fp, rule)
	277	}
	278	}
	279	}
	280
	281	const deduped = [...Array.from(byFingerprint.values()), ...noFingerprint]
	282
	283	// Backfill quality on every rule in the deduped output. This catches
217	284	// pre-Change 2 rules that lack a .quality field, and also refreshes
218	285	// the tag for any rule whose antipattern or fixFiles changed since
219	286	// the last extraction.
220		for (const rule of merged) {
	287	for (const rule of deduped) {
221	288	rule.quality = computeQuality(rule)
222	289	}
223	290
224	291	// Group by project
225	292	const byProject = {}
226		for (const rule of merged) {
	293	for (const rule of deduped) {
227	294	const proj = rule.trigger.project
228	295	if (!byProject[proj]) byProject[proj] = []
229	296	byProject[proj].push(rule)
…	…	@@ -235,7 +302,9 @@ function main () {
235	302	console.log(` Resolved loops analyzed: ${loopsData.loops.filter(l => l.resolved).length}`)
236	303	console.log(` Rules extracted: ${newRules.length}`)
237	304	console.log(` New rules added: ${added}`)
238		console.log(` Total rules: ${merged.length}`)
	305	console.log(` Pre-dedup total: ${merged.length}`)
	306	console.log(` Duplicates dropped: ${duplicatesDropped}`)
	307	console.log(` Total rules: ${deduped.length}`)
239	308
240	309	console.log('\n BY PROJECT')
241	310	console.log(' ' + '-'.repeat(56))
…	…	@@ -253,7 +322,7 @@ function main () {
253	322	}
254	323
255	324	// Show top rules by severity (longest loops)
256		const bySeverity = [...merged].sort((a, b) => b.source.duration - a.source.duration)
	325	const bySeverity = [...deduped].sort((a, b) => b.source.duration - a.source.duration)
257	326
258	327	console.log('\n HIGHEST SEVERITY RULES (longest loops)')
259	328	console.log(' ' + '-'.repeat(56))
…	…	@@ -265,14 +334,14 @@ function main () {
265	334	// Save
266	335	const report = {
267	336	generated: new Date().toISOString(),
268		totalRules: merged.length,
269		rules: merged,
	337	totalRules: deduped.length,
	338	rules: deduped,
270	339	}
271	340	fs.writeFileSync(RULES_FILE, JSON.stringify(report, null, 2))
272	341	console.log(`\nSaved: ${RULES_FILE}`)
273	342	console.log('═'.repeat(60))
274	343	}
275	344
276		module.exports = { extractRules, mergeRules, computeQuality, QUALITY_RANK, GENERIC_ANTIPATTERN }
	345	module.exports = { extractRules, mergeRules, computeQuality, QUALITY_RANK, GENERIC_ANTIPATTERN, ruleId }
277	346
278	347	if (require.main === module) main()

knowledge-ratchet commits / e7ed82d

Changed files (3) +163 -44