ea76327 · knowledge-ratchet

Changed files (1) +149 -106

README.md +149 -106

…	…	@@ -1,143 +1,186 @@
1	1	# Knowledge Ratchet
2	2
3		A meta-learning system for Claude Code that extracts principles from user corrections, validates them retrospectively, and enforces them via hooks. Knowledge can only improve, never degrade.
4
5		## The Problem
6
7		You correct Claude. It saves a memory file. Next session, maybe it loads, maybe it doesn't. You correct it again. Knowledge leaks between sessions.
8
9		## The Solution
10
11		Every correction gets captured, grouped into patterns, crystallized into principles, and enforced through progressively stronger mechanisms. Principles live forever until contradicted.
	3	Self-improving quality measurement system for Claude Code. Extracts principles from corrections, detects failure loops, auto-generates prevention rules, and tracks intelligence improvement over time.
12	4
13	5	## How It Works
14	6
15	7	```
16		YOU → correct Claude naturally
17		↓
18		CAPTURE → hooks detect corrections in real-time
19		↓
20		EXTRACT → group corrections by topic, find patterns
21		↓
22		VALIDATE → retrospective test: would principles prevent past corrections?
23		↓
24		PROMOTE → passing principles enter rules (always loaded)
25		↓
26		ENFORCE → hooks block violations, surface principles at moment of action
27		↓
28		REGRESS → weekly: are correction rates dropping?
29		↓
30		CONTRADICT → new correction conflicts with principle? ask the user
	8	Session starts
	9	→ Loop briefing: past failures + 221 learned rules for this project
	10	→ Preflight hook: blocks edits to danger files without reading first
	11
	12	During session
	13	→ Realtime scorer: tracks corrections, approvals, tool calls per turn
	14	→ Loop detector: 2+ corrections → injects learned rules as context
	15	→ Status bar: ▬ 0.72 │ 3 approved 1 corrected │ 45 tools
	16
	17	Session ends (2.7s pipeline)
	18	→ Sensor fusion: scores session with 16 per-turn + 5 session-level signals
	19	→ Meta-ratchet: adjusts sensor weights from predictive accuracy
	20	→ Loop extraction: finds correction→resolution patterns
	21	→ Rule extraction: auto-generates rules from resolved loops
	22	→ Project mapper: autodiscovers subsystems + danger levels
	23	→ Benchmarks: tracks novel/repeat ratio, density, resolution speed
	24	→ Dashboard: regenerates HTML visualization
	25
	26	Next session
	27	→ Briefing includes new rules — each session makes the next one smarter
31	28	```
32	29
33		## Three Layers of Knowledge
	30	## Quick Start
34	31
35		```
36		PRINCIPLES "Every storage decision must answer: what's the deletion story?"
37		↑ extracted from patterns, prunes entire categories of bad decisions
38		\|
39		PATTERNS "Append-only structures grow without bound — plan GC from day one"
40		↑ extracted from facts, predicts future corrections
41		\|
42		FACTS "Hyperdrive caused 56GB balloon in pear-git"
43		↑ your direct corrections, the raw signal
44		```
	32	```bash
	33	# Fast pipeline (runs automatically at session end)
	34	node pipeline.js
45	35
46		## How Corrections Become Constraints
	36	# Full batch analysis (includes LLM semantic classification)
	37	node analyze-all.js --limit=50
47	38
48		Principles migrate down through enforcement layers:
	39	# Dashboard (port 9077)
	40	node dashboard.js --serve
49	41
50		\| Corrected... \| Level \| Mechanism \|
51		\|---\|---\|---\|
52		\| Once \| Context \| Principle in rules file (advisory) \|
53		\| Twice \| Reminder \| Hook surfaces principle at moment of action \|
54		\| 3+ times \| Enforced \| Hook blocks the action \|
	42	# Benchmarks
	43	node benchmark-ratchet.js
	44	```
55	45
56		## Quick Start
	46	## Key Metrics
57	47
58		```bash
59		# See current state
60		node run.js status
	48	\| Metric \| Current \| Target \| What it means \|
	49	\|--------\|---------\|--------\|---------------\|
	50	\| Novel correction rate \| 87% \| >90% \| Old mistakes eliminated (higher = better) \|
	51	\| Correction density \| 20.8% \| <15% \| Corrections per turn (lower = better) \|
	52	\| Loop resolution \| 8.4 turns \| <6 turns \| How fast loops resolve (lower = better) \|
	53	\| Learned rules \| 221 \| growing \| Auto-extracted from resolved loops \|
61	54
62		# Run full pipeline
63		node run.js full
	55	## Architecture
	56
	57	### Two Pipelines
64	58
65		# Scan transcripts for corrections (LLM-augmented)
66		node llm-analyzer.js --limit=50
	59	Fast pipeline (`pipeline.js`, 2.7s, runs at Stop hook):
	60	```
	61	Sensor Fusion → Meta-Ratchet → Loop Detector → Rule Extractor
	62	→ Project Mapper → Benchmarks → Dashboard
	63	```
67	64
68		# Deep sentiment analysis
69		node analyze-transcripts.js
	65	Full batch (`analyze-all.js`, includes LLM calls):
	66	```
	67	LLM Analyzer → Regex Tuner → Sensor Fusion → Meta-Ratchet
	68	→ Coverage Map → Ratchet Pipeline → Deep Sentiment
70	69	```
71	70
72		## Pipeline Commands
	71	### Hooks
73	72
74		```bash
75		node run.js full # Run everything
76		node run.js extract # Extract patterns from feedback files
77		node run.js validate # Retrospective split test
78		node run.js regress # Score principles against recent corrections
79		node run.js migrate # Check enforcement level escalation
80		node run.js questions # List pending questions for user
81		node run.js status # Show current state
	73	\| Hook \| Script \| Mode \| Purpose \|
	74	\|------\|--------\|------\|---------\|
	75	\| SessionStart \| loop-briefing.js \| sync \| Past loops + learned rules for current project \|
	76	\| SessionStart \| realtime-score.js reset \| sync \| Reset session state \|
	77	\| UserPromptSubmit \| loop-intervention.js \| sync \| Inject rules when loop detected \|
	78	\| UserPromptSubmit \| detect-correction.js \| async \| Log corrections \|
	79	\| UserPromptSubmit \| realtime-score.js prompt \| async \| Update live score \|
	80	\| PostToolUse \| realtime-score.js tool \| async \| Track tool calls, errors, edits \|
	81	\| PreToolUse (Edit/Write) \| preflight-review.js \| sync \| Block edits to danger files \|
	82	\| Stop \| realtime-score.js stop \| sync \| Finalize session score \|
	83	\| Stop \| pipeline.js \| async \| Run full pipeline \|
	84
	85	### Self-Improving Feedback Loops
	86
	87	1. Sensor weight learning (meta-ratchet.js):
	88	```
	89	fuse-signals scores turns → meta-ratchet checks which sensors
	90	predicted corrections → adjusts weights → next run uses new weights
82	91	```
83	92
84		## Individual Tools
	93	2. Pattern discovery (tune-regex.js):
	94	```
	95	LLM classifies corrections → tune-regex finds which regex patterns
	96	match → discovers new patterns → fuse-signals uses tuned patterns
	97	```
85	98
86		```bash
87		node scan-transcripts.js # Regex scan of session transcripts
88		node llm-analyzer.js --limit=20 # LLM analysis of subtle corrections
89		node llm-analyzer.js status # Check processing cursor
90		node llm-analyzer.js reset # Reset cursor (reprocess everything)
91		node contradiction-detector.js check "text" # Check for contradictions
92		node contradiction-detector.js pending # List pending questions
93		node promote.js list # Show principles and candidates
94		node promote.js sync # Write principles to rules file
95		node ratchet.js commit "message" # Git-guarded principle update
96		node ratchet.js score # Current validation score
	99	3. Rule accumulation (rule-extractor.js):
	100	```
	101	loop resolves → rule-extractor captures what fixed it → rule saved
	102	→ next loop briefing includes the rule → prevents repeat
97	103	```
98	104
99		## Architecture
	105	## Files
100	106
101	107	```
102	108	knowledge-ratchet/
103		├── run.js # Pipeline orchestrator
104		├── detect-correction.js # Hook: captures corrections in real-time
105		├── preflight-review.js # Hook: blocks shared module edits with briefing
106		├── reset-session.js # Hook: clears per-session state
107		├── scan-transcripts.js # Regex scanner for session transcripts
108		├── llm-analyzer.js # LLM-augmented semantic analysis (Haiku)
109		├── analyze-transcripts.js # Deep sentiment and flow analysis
110		├── extract-patterns.js # Groups feedback files into patterns/principles
111		├── validate-principles.js # Retrospective split test
112		├── contradiction-detector.js # Checks new corrections against principles
113		├── promote.js # Moves candidates to active, syncs to rules
114		├── regress.js # Scores principles by correction rate
115		├── migrate-down.js # Tracks violations, escalates enforcement
116		├── ratchet.js # Git-guarded principle commits
117		├── active-principles.md # Current active principles
	109	├── pipeline.js # Fast pipeline (2.7s at session end)
	110	├── analyze-all.js # Full batch pipeline (7 steps)
	111	├── dashboard.js # HTML dashboard on port 9077
	112	│
	113	├── Measurement
	114	│ ├── fuse-signals.js # 16 sensors + 5 session signals → fused score
	115	│ ├── scan-sessions.js # Single-pass JSONL reader with results cache
	116	│ ├── realtime-score.js # Live session scoring via hooks
	117	│ ├── meta-ratchet.js # Auto-adjust sensor weights
	118	│ ├── tune-regex.js # LLM-driven regex pattern discovery
	119	│ └── coverage-map.js # Topic coverage + failure risk
	120	│
	121	├── Loop System
	122	│ ├── loop-detector.js # Extract correction loops from turn data
	123	│ ├── loop-intervention.js # Inject rules when loop forms (non-blocking)
	124	│ ├── loop-briefing.js # Session-start briefing with past loops
	125	│ └── rule-extractor.js # Auto-generate rules from resolutions
	126	│
	127	├── Prevention
	128	│ ├── preflight-review.js # Block edits to danger files
	129	│ ├── project-mapper.js # Autodiscover subsystems + danger levels
	130	│ └── detect-correction.js # Real-time correction capture
	131	│
	132	├── Phase 1 Pipeline
	133	│ ├── extract-patterns.js # Group corrections into patterns
	134	│ ├── validate-principles.js # Retrospective split test
	135	│ ├── contradiction-detector.js
	136	│ ├── promote.js # Candidate → active principle
	137	│ ├── regress.js # Score principles + fused quality trend
	138	│ ├── migrate-down.js # Escalate enforcement level
	139	│ ├── ratchet.js # Git-guarded principle commits
	140	│ └── run.js # Phase 1 pipeline orchestrator
	141	│
	142	├── Analysis
	143	│ ├── llm-analyzer.js # LLM semantic classification (Haiku)
	144	│ ├── scan-transcripts.js # Batch regex scanner
	145	│ ├── analyze-transcripts.js # Deep sentiment/flow analysis
	146	│ └── benchmark-ratchet.js # T3, R1, R2, R3, R4 benchmarks + history
	147	│
	148	├── active-principles.md # 5 validated principles
118	149	└── data/
119		├── patterns.json # Extracted topic groups
120		├── principles.json # Structured principles with metadata
121		└── validation-report.json # Test results
	150	├── fused-scores.json # Per-session fusion scores
	151	├── fused-turns-detail.json # Per-turn sensor data
	152	├── loops.json # 287 loops with corrections + resolutions
	153	├── learned-rules.json # 221 auto-extracted rules
	154	├── project-maps.json # Autodiscovered subsystem maps
	155	├── sensor-weights.json # Meta-ratchet learned weights
	156	├── tuned-patterns.json # LLM-discovered regex patterns
	157	├── benchmark-history.json # Daily benchmark snapshots
	158	├── coverage-map.json # Topic coverage + failure risk
	159	├── results-cache/ # Scored session cache (0.4s vs 10s)
	160	└── dashboard.html # Generated visualization
122	161	```
123	162
124		## Design Decisions
	163	## Pear-Git Analysis (worst project: 58 loops)
	164
	165	Top failure patterns:
	166	- "Do an audit / enter planning mode" — 68 corrections (33%)
	167	- "Stop" — 34 corrections (17%)
	168	- "Same error again" — 28 corrections (14%)
125	169
126		- No decay: Principles live forever until contradicted. Silence is confidence, not staleness.
127		- LLM over regex: Regex catches obvious corrections. LLM catches "the mesh is invisible?" and "sounds like a timer to me." The LLM is the source of truth.
128		- Contradictions ask, don't auto-update: When your thinking changes, the system asks you to confirm — it doesn't silently overwrite.
129		- Forward-only ratchet: Principle changes are git-committed with validation. If the score drops, the commit is reverted.
130		- Dedup via cursor: LLM analyzer tracks what it's already processed. No re-reading.
	170	Danger subsystems (auto-discovered):
	171	- `cli` — 51 files, 36 loops (⚠ high)
	172	- `server-core` — 9 files, 16 loops (⚠ high)
	173	- `p2p` — 13 files, 14 loops (⚠ high)
	174	- `routes` — 23 files, 13 loops (⚠ high)
131	175
132		## Validated Results
	176	## Design Decisions
133	177
134		- 30 feedback files analyzed
135		- 5 principles extracted
136		- 89% prevention rate on held-out test set
137		- 1,704 data points from 108 session transcripts
138		- LLM catches corrections regex misses (e.g., "you flipped x not y", "why are you advocating for polling")
	178	- Context injection over blocking: Loop intervention adds rules to the conversation, doesn't reject prompts. Claude reads rules AND responds to the user.
	179	- Results cache: Scored sessions cached by file mtime + weights hash. 140 files (1.1GB) → 0.4s scan.
	180	- LLM is ground truth: Regex has 97% precision but 25% recall. LLM catches the other 75% — subtle corrections like "hmm", "it is the same", "why are we doing X".
	181	- Rules accumulate, never delete: Each resolved loop generates a rule. Rules live forever unless superseded.
	182	- Autodiscovery over hardcoding: Project mapper scans file trees + require() graphs to find subsystems. Preflight hook reads the map, not a hardcoded list.
139	183
140	184	## Inspired By
141	185
142		Karpathy's [autoresearch](https://github.com/karpathy/autoresearch) — automated experiment loops with a forward-only ratchet. Adapted from scalar metric optimization to domain knowledge improvement. The metric isn't val_bpb — it's "did the user have to correct me?"
143		# trigger
	186	Karpathy's [autoresearch](https://github.com/karpathy/autoresearch) — forward-only ratchet applied to domain knowledge instead of scalar metrics. The metric isn't val_bpb — it's correction density and loop resolution speed.

knowledge-ratchet commits / ea76327

Changed files (1) +149 -106