05b5094 · knowledge-ratchet

Changed files (3) +332 -0

README.md +142 -0

…	…	@@ -0,0 +1,142 @@
	1	# Knowledge Ratchet
	2
	3	A meta-learning system for Claude Code that extracts principles from user corrections, validates them retrospectively, and enforces them via hooks. Knowledge can only improve, never degrade.
	4
	5	## The Problem
	6
	7	You correct Claude. It saves a memory file. Next session, maybe it loads, maybe it doesn't. You correct it again. Knowledge leaks between sessions.
	8
	9	## The Solution
	10
	11	Every correction gets captured, grouped into patterns, crystallized into principles, and enforced through progressively stronger mechanisms. Principles live forever until contradicted.
	12
	13	## How It Works
	14
	15	```
	16	YOU → correct Claude naturally
	17	↓
	18	CAPTURE → hooks detect corrections in real-time
	19	↓
	20	EXTRACT → group corrections by topic, find patterns
	21	↓
	22	VALIDATE → retrospective test: would principles prevent past corrections?
	23	↓
	24	PROMOTE → passing principles enter rules (always loaded)
	25	↓
	26	ENFORCE → hooks block violations, surface principles at moment of action
	27	↓
	28	REGRESS → weekly: are correction rates dropping?
	29	↓
	30	CONTRADICT → new correction conflicts with principle? ask the user
	31	```
	32
	33	## Three Layers of Knowledge
	34
	35	```
	36	PRINCIPLES "Every storage decision must answer: what's the deletion story?"
	37	↑ extracted from patterns, prunes entire categories of bad decisions
	38	\|
	39	PATTERNS "Append-only structures grow without bound — plan GC from day one"
	40	↑ extracted from facts, predicts future corrections
	41	\|
	42	FACTS "Hyperdrive caused 56GB balloon in pear-git"
	43	↑ your direct corrections, the raw signal
	44	```
	45
	46	## How Corrections Become Constraints
	47
	48	Principles migrate down through enforcement layers:
	49
	50	\| Corrected... \| Level \| Mechanism \|
	51	\|---\|---\|---\|
	52	\| Once \| Context \| Principle in rules file (advisory) \|
	53	\| Twice \| Reminder \| Hook surfaces principle at moment of action \|
	54	\| 3+ times \| Enforced \| Hook blocks the action \|
	55
	56	## Quick Start
	57
	58	```bash
	59	# See current state
	60	node run.js status
	61
	62	# Run full pipeline
	63	node run.js full
	64
	65	# Scan transcripts for corrections (LLM-augmented)
	66	node llm-analyzer.js --limit=50
	67
	68	# Deep sentiment analysis
	69	node analyze-transcripts.js
	70	```
	71
	72	## Pipeline Commands
	73
	74	```bash
	75	node run.js full # Run everything
	76	node run.js extract # Extract patterns from feedback files
	77	node run.js validate # Retrospective split test
	78	node run.js regress # Score principles against recent corrections
	79	node run.js migrate # Check enforcement level escalation
	80	node run.js questions # List pending questions for user
	81	node run.js status # Show current state
	82	```
	83
	84	## Individual Tools
	85
	86	```bash
	87	node scan-transcripts.js # Regex scan of session transcripts
	88	node llm-analyzer.js --limit=20 # LLM analysis of subtle corrections
	89	node llm-analyzer.js status # Check processing cursor
	90	node llm-analyzer.js reset # Reset cursor (reprocess everything)
	91	node contradiction-detector.js check "text" # Check for contradictions
	92	node contradiction-detector.js pending # List pending questions
	93	node promote.js list # Show principles and candidates
	94	node promote.js sync # Write principles to rules file
	95	node ratchet.js commit "message" # Git-guarded principle update
	96	node ratchet.js score # Current validation score
	97	```
	98
	99	## Architecture
	100
	101	```
	102	knowledge-ratchet/
	103	├── run.js # Pipeline orchestrator
	104	├── detect-correction.js # Hook: captures corrections in real-time
	105	├── preflight-review.js # Hook: blocks shared module edits with briefing
	106	├── reset-session.js # Hook: clears per-session state
	107	├── scan-transcripts.js # Regex scanner for session transcripts
	108	├── llm-analyzer.js # LLM-augmented semantic analysis (Haiku)
	109	├── analyze-transcripts.js # Deep sentiment and flow analysis
	110	├── extract-patterns.js # Groups feedback files into patterns/principles
	111	├── validate-principles.js # Retrospective split test
	112	├── contradiction-detector.js # Checks new corrections against principles
	113	├── promote.js # Moves candidates to active, syncs to rules
	114	├── regress.js # Scores principles by correction rate
	115	├── migrate-down.js # Tracks violations, escalates enforcement
	116	├── ratchet.js # Git-guarded principle commits
	117	├── active-principles.md # Current active principles
	118	└── data/
	119	├── patterns.json # Extracted topic groups
	120	├── principles.json # Structured principles with metadata
	121	└── validation-report.json # Test results
	122	```
	123
	124	## Design Decisions
	125
	126	- No decay: Principles live forever until contradicted. Silence is confidence, not staleness.
	127	- LLM over regex: Regex catches obvious corrections. LLM catches "the mesh is invisible?" and "sounds like a timer to me." The LLM is the source of truth.
	128	- Contradictions ask, don't auto-update: When your thinking changes, the system asks you to confirm — it doesn't silently overwrite.
	129	- Forward-only ratchet: Principle changes are git-committed with validation. If the score drops, the commit is reverted.
	130	- Dedup via cursor: LLM analyzer tracks what it's already processed. No re-reading.
	131
	132	## Validated Results
	133
	134	- 30 feedback files analyzed
	135	- 5 principles extracted
	136	- 89% prevention rate on held-out test set
	137	- 1,704 data points from 108 session transcripts
	138	- LLM catches corrections regex misses (e.g., "you flipped x not y", "why are you advocating for polling")
	139
	140	## Inspired By
	141
	142	Karpathy's [autoresearch](https://github.com/karpathy/autoresearch) — automated experiment loops with a forward-only ratchet. Adapted from scalar metric optimization to domain knowledge improvement. The metric isn't val_bpb — it's "did the user have to correct me?"

docs/architecture.md +116 -0

…	…	@@ -0,0 +1,116 @@
	1	# Architecture
	2
	3	## Core Concept
	4
	5	The knowledge ratchet is a meta-learning system that sits between Claude Code and the user. It observes user corrections, extracts generalizable principles, and progressively enforces them — from advisory context to mechanical blocks.
	6
	7	## Data Flow
	8
	9	```
	10	Session Transcripts (.jsonl)
	11	│
	12	├──→ scan-transcripts.js (regex, fast, bulk)
	13	│ │
	14	│ └──→ corrections.log (corrections + approvals)
	15	│
	16	└──→ llm-analyzer.js (Haiku, semantic, batched)
	17	│
	18	├──→ corrections.log (subtle corrections regex missed)
	19	└──→ disagreements (where regex was wrong)
	20
	21	corrections.log
	22	│
	23	├──→ extract-patterns.js ──→ patterns.json ──→ principles.json
	24	│
	25	├──→ regress.js ──→ scorecard.tsv (per-principle correction rate)
	26	│
	27	├──→ migrate-down.js ──→ migration-state.json (enforcement levels)
	28	│
	29	└──→ contradiction-detector.js ──→ questions/ (asks user)
	30
	31	principles.json
	32	│
	33	├──→ promote.js ──→ active-principles.md
	34	│ ──→ .claude/rules/09-ratchet-principles.md
	35	│
	36	└──→ ratchet.js ──→ git commit (reverts if validation drops)
	37	```
	38
	39	## Hooks (Real-Time)
	40
	41	Three hooks fire during Claude Code sessions:
	42
	43	### SessionStart
	44	Clears per-session state (`/tmp/claude-preflight-state.json`). Each session starts fresh — files that were reviewed in a previous session get reviewed again.
	45
	46	### UserPromptSubmit (async)
	47	Runs `detect-correction.js` on every user message. Detects correction/approval signals and appends to `corrections.log`. Also runs contradiction detection against active principles. Async so it doesn't block the conversation.
	48
	49	### PreToolUse (Edit\|Write)
	50	Runs `preflight-review.js` when Claude edits shared modules (multiwriter, auth, replication). Blocks the edit with a briefing showing downstream impact, relevant principles, and past corrections. Fires once per file per session — after review, the file is unlocked.
	51
	52	## Enforcement Stack
	53
	54	```
	55	MECHANICAL (can't be overridden)
	56	└── deny rules in settings.json
	57	└── PreToolUse exit(2) blocks
	58
	59	INTELLIGENT (can reason about novel cases)
	60	└── prompt hooks (Haiku evaluation)
	61	└── agent hooks (full sub-agent)
	62
	63	CONTEXTUAL (shapes reasoning)
	64	└── SessionStart context injection
	65	└── .claude/rules/ (always loaded)
	66	└── .claude/skills/ (loaded when relevant)
	67	```
	68
	69	## Deduplication Strategy
	70
	71	### Transcript Scanner (scan-transcripts.js)
	72	Deduplicates by checking if a timestamp already exists in `corrections.log`.
	73
	74	### LLM Analyzer (llm-analyzer.js)
	75	Maintains `data/llm-cursor.json` — tracks the last-processed timestamp per session file. Only messages after the cursor are sent to the LLM. Cursor advances only for messages that were actually processed (pre-classified + LLM-analyzed batches), not for messages that were seen but skipped due to batch limits.
	76
	77	### Corrections Log
	78	Append-only. Entries tagged with source (`regex`, `llm`, `auto`) and confidence scores. Disagreements between regex and LLM are tagged with `DISAGREE:regex-said-X`.
	79
	80	## Principle Lifecycle
	81
	82	```
	83	Feedback file (manual)
	84	or
	85	Corrections log (automatic)
	86	│
	87	↓
	88	CANDIDATE (extracted, not yet validated)
	89	│
	90	↓ validate: prevention rate > 50%
	91	ACTIVE (in rules file, loaded every session)
	92	│
	93	├──→ CONTRADICTED (user's thinking changed → question asked)
	94	│ ├──→ UPDATED (revised principle)
	95	│ ├──→ KEPT (exception, not contradiction)
	96	│ └──→ REPLACED (new principle)
	97	│
	98	└──→ OVERRIDDEN 3x (hook fired, user proceeded anyway)
	99	├──→ RETIRED (no longer relevant)
	100	├──→ REVISED (made more specific)
	101	└──→ KEPT (overrides were exceptions)
	102	```
	103
	104	No decay. Principles live until contradicted or explicitly retired.
	105
	106	## Enforcement Migration
	107
	108	Principles start at `context` level and migrate down based on violation count:
	109
	110	\| Violations \| Level \| Mechanism \|
	111	\|---\|---\|---\|
	112	\| 0-1 \| context \| In rules file, advisory \|
	113	\| 2-3 \| reminder \| Hook warns at moment of action \|
	114	\| 4+ \| enforced \| Hook blocks until confirmed \|
	115
	116	Tracked in `data/migration-state.json`. The migration is automatic — `migrate-down.js` counts violations and escalates.

docs/theory.md +74 -0

…	…	@@ -0,0 +1,74 @@
	1	# Theory — The Intelligence Ratchet
	2
	3	## Origin
	4
	5	Inspired by Karpathy's autoresearch — an automated experiment loop that trains ML models overnight. The agent modifies code, runs a 5-minute experiment, checks if the metric improved, keeps or reverts. Repeat forever.
	6
	7	Autoresearch works because `val_bpb` (validation bits per byte) is cheap, objective, and directly measures model quality. One number, computed in seconds, ungameable.
	8
	9	## The Problem
	10
	11	Domain knowledge has no `val_bpb`. You can't run a function that returns "how well does Claude understand P2P architecture?" The metric is the user — their corrections, approvals, frustrations, and silences.
	12
	13	## The Adaptation
	14
	15	Instead of optimizing a training script against a scalar metric, optimize a knowledge base against user corrections:
	16
	17	\| Autoresearch \| Knowledge Ratchet \|
	18	\|---\|---\|
	19	\| `train.py` (the code) \| Skills, rules, principles (the knowledge) \|
	20	\| `val_bpb` (the metric) \| User correction rate (the metric) \|
	21	\| `prepare.py` (immutable evaluator) \| User's judgment (immutable ground truth) \|
	22	\| `program.md` (agent instructions) \| Pipeline scripts (extraction logic) \|
	23	\| git commit/revert \| git commit/revert \|
	24	\| 5-minute experiment \| Conversation → correction → extraction cycle \|
	25
	26	## Meta-Ratcheting
	27
	28	Small provable values (facts) build into patterns, which crystallize into principles. The layers constrain each other bidirectionally:
	29
	30	Bottom-up: Facts validate patterns validate principles.
	31	If a principle stops predicting corrections, it gets flagged.
	32
	33	Top-down: Principles constrain which patterns apply, which constrain which concrete actions are considered.
	34	"Fix don't skip" eliminates an entire category of workarounds before any concrete decision is made.
	35
	36	## Why Not Decay
	37
	38	The first design included time-based decay — principles lose confidence unless reinforced. But stable truths ("never delete user storage") shouldn't decay just because the user hasn't needed to correct that specific mistake recently.
	39
	40	Revised design: principles live until contradicted. Silence is confidence. The only signals that change a principle:
	41
	42	1. A new correction contradicts it → ask the user
	43	2. The user overrides the enforcement hook 3+ times → ask the user
	44	3. The user explicitly revises it
	45
	46	## The Vague Constraining the Concrete
	47
	48	A concern: vague principles can't constrain concrete actions. But they can — not by specifying what to do, but by eliminating what not to consider.
	49
	50	"Every storage decision must answer: what's the deletion story?" doesn't tell you which database to use. But it prunes Hyperdrive (no GC), prunes raw Autobase (append-only growth), and forces the question before any implementation begins.
	51
	52	The pruning IS the intelligence. Each principle eliminates a class of mistakes across all future decisions.
	53
	54	## Co-Evolution
	55
	56	The user's understanding evolves. A correction from three months ago might reflect thinking they've moved past. The system must co-evolve:
	57
	58	- Contradictions are signal, not noise — they mean the user learned something new
	59	- The system asks about contradictions rather than auto-updating
	60	- Repeated overrides suggest a principle is stale, not that the user is wrong
	61
	62	## Limits
	63
	64	The ratchet can't:
	65	- Manufacture judgment for novel domains (no corrections = no signal)
	66	- Guarantee Claude follows advisory principles (context is not enforcement)
	67	- Replace the user's role as ground truth
	68
	69	The ratchet can:
	70	- Prevent repeated concrete mistakes via enforcement hooks
	71	- Build patterns that predict and prevent categories of mistakes
	72	- Mechanically block known-bad actions (deny rules, PreToolUse hooks)
	73	- Detect when the user's thinking has changed (contradictions)
	74	- Track whether quality is actually improving (sentiment trends, correction rates)

knowledge-ratchet commits / 05b5094

Changed files (3) +332 -0