| … | … | @@ -0,0 +1,291 @@ |
| 1 | # CLAUDE.md |
| 2 | |
| 3 | This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. |
| 4 | |
| 5 | ## Repository Purpose |
| 6 | |
| 7 | This is a **knowledge base and analysis system** for the holepunchto P2P ecosystem, containing documentation and tooling for all 529 repositories in the holepunchto GitHub organization. It provides: |
| 8 | |
| 9 | - **Designer & Architect agents**: CLI tools that help users build P2P applications |
| 10 | - **Skills documentation**: API docs for all 529 repos (133 with deep analysis) |
| 11 | - **Semantic mappings**: Categorization, use cases, patterns, dependencies |
| 12 | - **Analysis automation**: Scripts to update the knowledge base |
| 13 | |
| 14 | ## Key Commands |
| 15 | |
| 16 | ### Using the Sub-Agents (Primary Use Case) |
| 17 | |
| 18 | ```bash |
| 19 | # Get repo recommendations for a use case |
| 20 | node designer-agent.js "I want to build a distributed database" |
| 21 | node designer-agent.js --use-case p2p-file-sharing |
| 22 | node designer-agent.js --list-use-cases |
| 23 | |
| 24 | # Design system architecture for selected repos |
| 25 | node architect-agent.js hypercore hyperbee hyperswarm |
| 26 | node architect-agent.js --use-case distributed-database |
| 27 | ``` |
| 28 | |
| 29 | ### Updating the Knowledge Base |
| 30 | |
| 31 | ```bash |
| 32 | # Phase 1A: Analyze all 529 repos via GitHub API |
| 33 | node process_repos.js |
| 34 | |
| 35 | # Phase 1B: Deep code analysis for specific tier |
| 36 | node clone_and_analyze_tier.js 1 # Tier 1 (building blocks) |
| 37 | node clone_and_analyze_tier.js 2 # Tier 2 (helpers) |
| 38 | node clone_and_analyze_tier.js 3 # Tier 3 (top 50 popular) |
| 39 | node clone_and_analyze_tier.js 4 # Tier 4 (selective 150) |
| 40 | |
| 41 | # Phase 2: Build semantic mappings |
| 42 | node build_semantic_maps.js |
| 43 | ``` |
| 44 | |
| 45 | ### Working with Git Submodules |
| 46 | |
| 47 | ```bash |
| 48 | # Initialize specific submodules (repos are optional, ~121MB if all loaded) |
| 49 | git submodule update --init repos/tier1/hypercore |
| 50 | git submodule update --init pear-docs |
| 51 | |
| 52 | # Update all initialized submodules |
| 53 | git submodule update --remote |
| 54 | |
| 55 | # Check submodule status |
| 56 | git submodule status |
| 57 | ``` |
| 58 | |
| 59 | ## Architecture Overview |
| 60 | |
| 61 | ### Three-Layer System |
| 62 | |
| 63 | 1. **Data Layer** (JSON files) |
| 64 | - `repo_map.json`: All 529 repos with metadata |
| 65 | - `dependency-graph.json`: Complete dependency relationships |
| 66 | - `semantic-categories.json`: 7 categories (Building Blocks, Bare Modules, etc.) |
| 67 | - `use-case-map.json`: 11 P2P use cases with repo recommendations |
| 68 | - `pattern-map.json`: 10 P2P patterns mapped to repos |
| 69 | - `stability-index.json`: Production readiness classification |
| 70 | - `platform-support.json`: Platform compatibility matrix |
| 71 | |
| 72 | 2. **Documentation Layer** (skills/ directory) |
| 73 | - `skills/{repo-name}/skills.md`: API docs, usage examples, patterns (529 files) |
| 74 | - `skills/{repo-name}/api-extract.json`: Structured API data (133 files with deep analysis) |
| 75 | |
| 76 | 3. **Agent Layer** (executable scripts) |
| 77 | - `designer-agent.js`: Requirements → repo recommendations |
| 78 | - `architect-agent.js`: Repos → system architecture design |
| 79 | |
| 80 | ### Analysis Pipeline |
| 81 | |
| 82 | ``` |
| 83 | Phase 1A (process_repos.js) |
| 84 | → Fetch metadata via GitHub API |
| 85 | → Build dependency graphs |
| 86 | → Generate initial skills.md files |
| 87 | → Output: repo_map.json, dependency-graph.json, tech-stack.json |
| 88 | |
| 89 | Phase 1B (clone_and_analyze_tier.js) |
| 90 | → Clone repos as git submodules |
| 91 | → Parse package.json and index.js |
| 92 | → Extract API surfaces |
| 93 | → Detect P2P patterns |
| 94 | → Output: api-extract.json, enhanced skills.md |
| 95 | |
| 96 | Phase 2 (build_semantic_maps.js) |
| 97 | → Load Phase 1 data |
| 98 | → Apply categorization rules |
| 99 | → Map use cases and patterns |
| 100 | → Output: semantic-categories.json, use-case-map.json, pattern-map.json |
| 101 | |
| 102 | Phase 3 (agents) |
| 103 | → Use semantic maps to guide users |
| 104 | → Designer: natural language → use case → repos |
| 105 | → Architect: repos → layers → dependencies → integration points |
| 106 | ``` |
| 107 | |
| 108 | ### Tier System |
| 109 | |
| 110 | Repos are organized into priority tiers for analysis: |
| 111 | |
| 112 | - **Tier 1** (6 repos): Core building blocks (hypercore, hyperbee, hyperdrive, autobase, hyperdht, hyperswarm) |
| 113 | - **Tier 2** (6 repos): Essential helpers (corestore, localdrive, compact-encoding, protomux, etc.) |
| 114 | - **Tier 3** (50 repos): Most popular by stars |
| 115 | - **Tier 4** (150 repos): Selective additional repos |
| 116 | - **Remaining** (317 repos): Low-priority infrastructure/tools |
| 117 | |
| 118 | Currently analyzed: 133/529 repos (25.1%) - covers all critical repos. |
| 119 | |
| 120 | ## Key Data Structures |
| 121 | |
| 122 | ### repo_map.json |
| 123 | Array of 529 repos with: `name`, `description`, `language`, `url` |
| 124 | |
| 125 | ### dependency-graph.json |
| 126 | Object keyed by repo name with: |
| 127 | - `dependencies`: npm dependencies |
| 128 | - `devDependencies`: npm dev dependencies |
| 129 | - `reverseDependencies`: which repos depend on this one (deduplicated) |
| 130 | |
| 131 | ### use-case-map.json |
| 132 | Defines 11 use cases, each with: |
| 133 | - `description`: What the use case is for |
| 134 | - `requiredRepos`: Essential repos |
| 135 | - `optionalRepos`: Enhancement repos |
| 136 | - `patterns`: P2P patterns involved |
| 137 | |
| 138 | ### pattern-map.json |
| 139 | Defines 10 P2P patterns with: |
| 140 | - `patterns.{name}.description`: Pattern description |
| 141 | - `patterns.{name}.implementedBy`: Repos that implement this pattern |
| 142 | - `repoPatterns.{repo}.implements`: Patterns this repo implements |
| 143 | - `repoPatterns.{repo}.uses`: Patterns this repo uses |
| 144 | |
| 145 | ## Important Conventions |
| 146 | |
| 147 | ### Git Commits |
| 148 | All commits must include co-authorship: |
| 149 | ``` |
| 150 | Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> |
| 151 | ``` |
| 152 | |
| 153 | Use heredoc for commit messages with proper formatting. |
| 154 | |
| 155 | ### Node.js Compatibility |
| 156 | Avoid optional chaining (`?.`) - not supported in all Node.js versions. Use explicit checks: |
| 157 | ```javascript |
| 158 | // DON'T: packageJson.dependencies?.hypercore |
| 159 | // DO: packageJson.dependencies && packageJson.dependencies.hypercore |
| 160 | ``` |
| 161 | |
| 162 | ### Skills Documentation Format |
| 163 | Each `skills/{repo}/skills.md` follows this structure: |
| 164 | ```markdown |
| 165 | # {Repo Name} |
| 166 | |
| 167 | **Repository**: https://github.com/holepunchto/{repo} |
| 168 | **Category**: {category} |
| 169 | |
| 170 | ## Overview |
| 171 | {description} |
| 172 | |
| 173 | ## Installation |
| 174 | npm install {repo} |
| 175 | |
| 176 | ## Dependencies |
| 177 | {list of dependencies} |
| 178 | |
| 179 | ## API Documentation (if deeply analyzed) |
| 180 | {extracted from api-extract.json} |
| 181 | |
| 182 | ## P2P Patterns |
| 183 | {detected patterns} |
| 184 | ``` |
| 185 | |
| 186 | ### API Extract Format |
| 187 | `api-extract.json` structure: |
| 188 | ```json |
| 189 | { |
| 190 | "repo": "repo-name", |
| 191 | "entryPoint": "index.js", |
| 192 | "exports": [...], |
| 193 | "api": { |
| 194 | "classes": [{ |
| 195 | "name": "ClassName", |
| 196 | "methods": [...] |
| 197 | }], |
| 198 | "functions": [...] |
| 199 | }, |
| 200 | "readme": { |
| 201 | "sections": [...], |
| 202 | "codeExamples": [...] |
| 203 | }, |
| 204 | "patterns": ["append-only-log", "replication", ...] |
| 205 | } |
| 206 | ``` |
| 207 | |
| 208 | ## Working with This Codebase |
| 209 | |
| 210 | ### Adding Analysis for New Repos |
| 211 | |
| 212 | 1. Update `repo_map.json` if new repos were added to holepunchto org |
| 213 | 2. Run `node process_repos.js` to regenerate Phase 1A data |
| 214 | 3. Update `tiers.json` if you want to deeply analyze new repos |
| 215 | 4. Run `node clone_and_analyze_tier.js {tier}` for deep analysis |
| 216 | 5. Run `node build_semantic_maps.js` to update semantic mappings |
| 217 | 6. Commit with proper co-authorship |
| 218 | |
| 219 | ### Modifying Agents |
| 220 | |
| 221 | Both agents (`designer-agent.js`, `architect-agent.js`) are standalone Node.js scripts that: |
| 222 | - Can be invoked from command line |
| 223 | - Export functions for programmatic use |
| 224 | - Load semantic map JSON files |
| 225 | - Output formatted text to console |
| 226 | |
| 227 | When modifying, maintain: |
| 228 | - CLI argument parsing (`--help`, `--use-case`, etc.) |
| 229 | - Formatted output (box-drawing characters, clear sections) |
| 230 | - Exported functions for library use |
| 231 | |
| 232 | ### Data Consistency |
| 233 | |
| 234 | When updating semantic maps, ensure consistency across: |
| 235 | - `use-case-map.json` lists repos that must exist in `repo_map.json` |
| 236 | - `pattern-map.json` references repos that exist |
| 237 | - `stability-index.json` includes all repos from `repo_map.json` |
| 238 | - Repo counts match across files (529 total) |
| 239 | |
| 240 | ## File Organization |
| 241 | |
| 242 | ``` |
| 243 | / |
| 244 | ├── skills/ # 529 directories, one per repo |
| 245 | │ └── {repo}/ |
| 246 | │ ├── skills.md # Always present |
| 247 | │ └── api-extract.json # Only for 133 deeply analyzed repos |
| 248 | ├── repos/ # Git submodules (optional) |
| 249 | │ ├── tier1/ # 6 repos |
| 250 | │ ├── tier2/ # 6 repos |
| 251 | │ ├── tier3/ # 50 repos |
| 252 | │ └── tier4/ # 150 repos |
| 253 | ├── {phase}-data.json # 10 JSON files with analysis data |
| 254 | ├── {agent}.js # 6 Node.js scripts (automation + agents) |
| 255 | └── docs/ # Progress reports and guides |
| 256 | ``` |
| 257 | |
| 258 | ## Current Status (as of 2026-01-16) |
| 259 | |
| 260 | - ✅ Phase 1A: Complete (529/529 repos) |
| 261 | - ✅ Phase 1B: Selective complete (133/529 repos analyzed) |
| 262 | - ✅ Phase 2: Complete (all semantic mappings) |
| 263 | - ✅ Phase 3: Complete (designer + architect agents functional) |
| 264 | - ⏳ Phase 4-6: Pending (additional skills, enhanced analysis, integration) |
| 265 | |
| 266 | The system is **production-ready** for helping users build P2P applications. The designer and architect agents work with existing data. |
| 267 | |
| 268 | ## User Workflow |
| 269 | |
| 270 | Users interact with this knowledge base through: |
| 271 | |
| 272 | 1. **QUICK_START.md**: 5-minute introduction |
| 273 | 2. **AI_WORKFLOW_GUIDE.md**: Complete guide for AI-assisted development |
| 274 | 3. **Designer agent**: Discovers appropriate repos for requirements |
| 275 | 4. **Architect agent**: Designs system architecture |
| 276 | 5. **Skills docs**: Reads API documentation for implementation |
| 277 | 6. **Optional AI**: Can read all docs and generate code |
| 278 | |
| 279 | Typical flow: |
| 280 | ```bash |
| 281 | node designer-agent.js "user requirement" |
| 282 | → Get repo recommendations |
| 283 | |
| 284 | node architect-agent.js repo1 repo2 repo3 |
| 285 | → Get architectural design |
| 286 | |
| 287 | cat skills/repo1/skills.md |
| 288 | → Read API documentation |
| 289 | |
| 290 | # Build the app (with or without AI assistance) |
| 291 | ``` |