OverviewFilesCommitsIssuesDocsReleasesPeers

wirecore commits / 43de5c2

Create comprehensive refactor plan for ANProto alignment
Full refactor plan to align Pear app with original Wirecore design: Key Finding: Original Wirecore design IS Hypercore best practice - Hypercore blocks = MINIMUM needed (signatures, 132 bytes) - Content stored separately = Sparse replication - Lazy fetching = Efficient bandwidth usage Design Criteria Met: ✓ ANProto signatures → Small blocks (132 bytes vs KB) ✓ Content-by-hash → Deduplication + sparse downloads ✓ YAML metadata → Structured format in cache ✓ Wiredove compatible → Same ANProto + YAML ✓ Efficient replication → Download signatures first ✓ Lazy content → Fetch via Protomux on-demand Implementation Phases: - Phase 1: Add ANProto layer (non-breaking) - Phase 2: Add content.json storage (non-breaking) - Phase 3: Add YAML format (non-breaking, dual-format reader) - Phase 4: Switch to signature-only blocks (BREAKING) - Phase 5: Add ContentSync + Protomux transport - Phase 6: Wiredove bridge integration Migration Strategy: - Option A: Fresh start (new identities) - Option B: Dual-format reader (gradual) - Option C: Migration command (convert old feeds) Timeline: 6 weeks for full implementation Risk: Medium (breaking change in Phase 4) Rollback: Each phase independent until Phase 4 Open Questions: 1. Storage location for anproto-keypair? 2. Hyperdrive vs content.json for images? 3. Force migration or support dual-format? 4. Keep IdentityKey alongside ANProto? 5. Should relay serve content-by-hash? Reference: pear-app/ARCHITECTURE.md, docs/CLAUDE-SKILLS.md Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
43de5c203bc24217c51a818238de6441fd221cbb
by plankatron <fleeky@gmail.com> · 2026-01-21T15:00:51.000Z

Changed files (1) +629 -0

pear-app/REFACTOR_PLAN.md +629 -0
……@@ -0,0 +1,629 @@
1# Wirecore Pear App - Full Refactor Plan
2
3**Goal**: Align Pear app with original Wirecore design while maintaining Hypercore best practices and maximum efficiency.
4
5**Status**: Planning Phase
6**Date**: 2026-01-21
7
8---
9
10## Executive Summary
11
12The refactor will transform the Pear app from storing full messages in Hypercore to using:
13- **ANProto signatures** in Hypercore blocks (132 bytes)
14- **Content-by-hash** storage in local cache
15- **Lazy content fetching** via Protomux channel
16- **YAML format** with metadata (name, image, previous)
17
18This achieves all design criteria while following Hypercore best practices for efficiency.
19
20---
21
22## Design Criteria Analysis
23
24### ✓ How Each Criterion Uses Hypercore Efficiently
25
26| Criterion | Implementation | Hypercore Best Practice |
27|-----------|----------------|------------------------|
28| **ANProto Signatures** | Store 132-byte signatures in blocks | Small blocks = fast replication ✓ |
29| **Content-by-Hash** | Separate content.json cache | Sparse replication ✓ |
30| **Wiredove Compatible** | Same ANProto + YAML format | Transport-agnostic ✓ |
31| **Efficient Replication** | Download signatures first, content on-demand | Hypercore's sparse mode ✓ |
32| **YAML Metadata** | Structured format in content cache | Keep structured data out of blocks ✓ |
33| **Lazy Content** | Fetch via Protomux channel | Multiplex over existing connections ✓ |
34
35**Key Insight**: The original Wirecore design IS Hypercore best practice. Hypercore blocks should store the MINIMUM needed for ordering and authentication (signatures), and everything else should be fetched on-demand.
36
37---
38
39## Architecture Overview
40
41### Message Flow (Refactored)
42
43```javascript
44// POST FLOW
45async function post(text) {
46 // 1. Create YAML message with metadata
47 const yaml = YAMLHelper.create({
48 name: await getUsername(),
49 image: await getProfileImageHash(),
50 previous: await getLatestMessageHash()
51 }, text)
52
53 // 2. Hash the content (SHA256 → base64)
54 const contentHash = ANProto.hashData(yaml)
55 // → "pZGm1Av0IEBKARczz7exkNYsZb8LzaMrV7J32a2fFG4="
56
57 // 3. Store content in local cache
58 await contentStore.put(contentHash, yaml)
59 // → Writes to {storage}/content.json
60
61 // 4. Sign the hash with ANProto
62 const signature = ANProto.signHash(contentHash, anprotoKeypair)
63 // → "BSY7/er4VJIu08o39...wnlzLSoZ1QC7rHJc..." (132 bytes)
64 // → Format: publicKey(44) + signed(timestamp + hash)(88+)
65
66 // 5. Append ONLY signature to Hypercore
67 await myFeed.append(Buffer.from(signature))
68 // → Hypercore block = 132 bytes, not KB!
69
70 return { hash: contentHash, signature, timestamp }
71}
72
73// READ FLOW
74async function getTimeline(limit) {
75 const posts = []
76
77 // 1. Read signatures from Hypercore feeds
78 for (const feed of [myFeed, ...followedFeeds]) {
79 for (let i = 0; i < feed.length; i++) {
80 const block = await feed.get(i)
81 const signature = block.toString()
82
83 // 2. Verify signature and extract hash
84 const { timestamp, hash } = ANProto.verifySignature(signature)
85
86 // 3. Try to get content from local cache
87 let content = await contentStore.get(hash)
88
89 // 4. If not cached, request from peers
90 if (!content) {
91 content = await contentSync.fetch(hash)
92 await contentStore.put(hash, content)
93 }
94
95 // 5. Parse YAML and extract metadata
96 const { metadata, text } = YAMLHelper.parse(content)
97
98 posts.push({
99 signature,
100 hash,
101 timestamp,
102 author: signature.substring(0, 44),
103 name: metadata.name,
104 image: metadata.image,
105 text
106 })
107 }
108 }
109
110 // Sort by timestamp and return
111 return posts.sort((a, b) => b.timestamp - a.timestamp).slice(0, limit)
112}
113```
114
115### Storage Architecture (Refactored)
116
117```
118{storage-path}/
119├── anproto-keypair.json # ANProto ed25519 keypair (132 chars)
120│ {
121│ "keypair": "BSY7/er4VJIu08o39NaRAiPY/MAvd7oQhlGCRDABjYU=tQa03kqUWG..."
122│ }
123│
124├── content.json # Content-by-hash cache
125│ {
126│ "pZGm1Av0...": "---\nname: fleeky\nimage: hash...\n---\nHello!",
127│ "anotherHash...": "---\nname: fleeky\n---\nAnother post",
128│ "imageHash...": "base64-encoded-image-data..."
129│ }
130│
131├── cores/ # Corestore (Hypercore feeds)
132│ ├── identity/ # IdentityKey seed (for display name)
133│ │ └── (kept for compatibility)
134│ │
135│ ├── my-feed/ # Personal feed (writable)
136│ │ └── data # Contains ONLY ANProto signatures!
137│ │ # Each block: 132 bytes
138│ │
139│ ├── {followed-feed-1}/ # Followed user's feed (read-only)
140│ │ └── data # Signatures only, sparse download
141│ │
142│ └── {followed-feed-N}/ # More followed feeds
143│
144└── metadata/ # Hyperbee for user metadata
145 ├── profile # { username, bio, avatar }
146 ├── follow:{pubkey} # Follow list
147 └── config # App settings
148```
149
150### Network Architecture
151
152```
153┌─────────────────────────────────────────────────────────────┐
154│ Hyperswarm Connection │
155├─────────────────────────────────────────────────────────────┤
156│ │
157│ Channel: "hypercore/replication" │
158│ ├─ Replicate feed signatures (132 bytes each) │
159│ └─ Sparse mode: Download only what's needed │
160│ │
161│ Channel: "wirecore-content-sync" (Protomux) │
162│ ├─ content-request: { hash } │
163│ ├─ content-response: { hash, content } │
164│ └─ Multiplexed over same connection │
165│ │
166└─────────────────────────────────────────────────────────────┘
167```
168
169**Efficiency**:
170- Hypercore replication: 132 bytes per message
171- Content requests: Only for posts being viewed
172- Multiplexed: No additional connections needed
173- Cached: content.json stores fetched content
174
175---
176
177## Components to Implement
178
179### 1. ANProto Module (`lib/anproto.js`)
180
181**Copy from**: `lib/core/anproto.js`
182
183**API**:
184```javascript
185class ANProto {
186 // Generate new keypair
187 static gen() → { publicKey, secretKey, keypair }
188
189 // Load keypair from 132-char string
190 static loadKeypair(keypairString) → { publicKey, secretKey }
191
192 // Hash data (SHA256)
193 static hashData(data) → base64Hash
194
195 // Sign hash with keypair
196 static signHash(hash, keypair) → signature
197
198 // Verify signature
199 static verifySignature(signature) → { timestamp, hash }
200
201 // Create complete signed message
202 static createMessage(content, keypair) → { anproto, content, timestamp, hash }
203
204 // Verify complete message
205 static verifyMessage(message) → { author, timestamp, hash, verified }
206}
207```
208
209### 2. YAML Helper (`lib/yaml-helper.js`)
210
211**Copy from**: `lib/core/yaml-helper.js`
212
213**API**:
214```javascript
215class YAMLHelper {
216 // Create YAML document with metadata
217 static create(metadata, body) → yamlString
218 // Example:
219 // create({ name: 'fleeky', image: 'hash...' }, 'Hello!')
220 // → "---\nname: fleeky\nimage: hash...\n---\nHello!"
221
222 // Parse YAML document
223 static parse(yamlString) → { metadata, text }
224 // Example:
225 // parse("---\nname: fleeky\n---\nHello!")
226 // → { metadata: { name: 'fleeky' }, text: 'Hello!' }
227
228 // Extract name from YAML (for directory compatibility)
229 static extractName(yamlString) → name
230}
231```
232
233### 3. Content Store (`lib/content-store.js`)
234
235**Copy from**: `lib/core/content-store.js`
236
237**API**:
238```javascript
239class ContentStore {
240 constructor(storagePath)
241
242 async init() // Load content.json
243 async get(hash) → content | null
244 async put(hash, content) → void
245 async has(hash) → boolean
246 async delete(hash) → void
247 async clear() → void
248 async size() → number
249}
250```
251
252**File Format** (`content.json`):
253```json
254{
255 "hashABC...": "yaml content here",
256 "hashDEF...": "another yaml content",
257 "hashGHI...": "base64-image-data..."
258}
259```
260
261### 4. Content Sync Manager (`lib/content-sync.js`)
262
263**Copy from**: `lib/core/content-sync.js`
264
265**API**:
266```javascript
267class ContentSyncManager {
268 constructor(contentStore, options)
269
270 // Register a transport (Hyperswarm, HTTP, WebRTC)
271 addTransport(transport)
272
273 // Fetch content by hash (tries all transports)
274 async fetch(hash) → content | null
275
276 // Respond to content requests (for serving to peers)
277 async handleRequest(hash, peerId) → content | null
278}
279```
280
281### 5. Hyperswarm Content Transport (`lib/hyperswarm-content-transport.js`)
282
283**Copy from**: `lib/core/hyperswarm-content-transport.js`
284
285**API**:
286```javascript
287class HyperswarmContentTransport {
288 constructor(swarm, contentSync)
289
290 // Attach to a connection (add Protomux channel)
291 attachToConnection(conn)
292
293 // Request content from peers
294 async request(hash) → content | null
295
296 // Respond to peer requests
297 async serve(hash) → content | null
298}
299```
300
301**Protocol** (Protomux channel):
302```
303Channel: "wirecore-content-sync"
304
305Message Types:
3061. content-request: { id, hash }
3072. content-response: { id, hash, content, found }
308```
309
310---
311
312## Implementation Phases
313
314### Phase 0: Preparation (No Code Changes)
315- ✅ Review original wirecore.js design
316- ✅ Document architectural deviation
317- ✅ Create refactor plan
318- ⏳ Get user approval on approach
319
320### Phase 1: Add ANProto Layer (Non-Breaking)
321**Goal**: Add ANProto alongside existing code, no breaking changes
322
323**Tasks**:
3241. Copy `anproto.js` to pear-app
3252. Add ANProto keypair generation/loading
326 - Store in `anproto-keypair.json`
327 - Keep IdentityKey for backwards compat
3283. Add ANProto signing to post()
329 - Store signature in message metadata
330 - Keep full JSON in Hypercore (for now)
3314. Add ANProto verification to timeline reading
332 - Verify signatures when reading
333 - Log warnings if verification fails
334
335**Testing**:
336- Existing posts still work
337- New posts have ANProto signatures
338- Signatures verify correctly
339
340**Rollback**: Remove anproto.js, revert changes
341
342### Phase 2: Add Content Storage (Non-Breaking)
343**Goal**: Add content.json cache, still store full messages in Hypercore
344
345**Tasks**:
3461. Copy `content-store.js` to pear-app
3472. Initialize ContentStore on startup
3483. Store content in content.json when posting
349 - Hash content
350 - Store in content.json
351 - Keep storing full message in Hypercore
3524. Read from content.json when available
353 - Try content cache first
354 - Fall back to Hypercore
355
356**Testing**:
357- Posts work as before
358- Content cached in content.json
359- Timeline reads from cache
360
361**Rollback**: Remove content-store.js, revert changes
362
363### Phase 3: Add YAML Format (Non-Breaking)
364**Goal**: Convert messages to YAML, dual-format reader
365
366**Tasks**:
3671. Copy `yaml-helper.js` to pear-app
3682. Store username in metadata (from Hyperbee)
3693. Convert new posts to YAML format
370 - Add name, image, previous fields
371 - Store YAML in content.json
372 - Store signature in Hypercore
373 - **DUAL FORMAT**: Also store JSON for backwards compat
3744. Read both YAML and JSON formats
375 - Try parsing as YAML first
376 - Fall back to JSON for old posts
377
378**Testing**:
379- New posts use YAML
380- Old posts still readable (JSON)
381- Timeline shows both formats
382
383**Rollback**: Remove yaml-helper.js, revert to JSON only
384
385### Phase 4: Change to Signature-Only Hypercore (BREAKING)
386**Goal**: Store only signatures in Hypercore, content in cache
387
388**Tasks**:
3891. Modify post() to store ONLY signature
390 - Remove JSON storage from Hypercore
391 - Store signature (132 bytes) only
392 - Content already in content.json
3932. Modify timeline reading
394 - Read signatures from Hypercore
395 - Extract hash from signature
396 - Fetch content from cache
3973. Add migration tool
398 - Convert old feeds to new format
399 - Extract content from old blocks
400 - Store in content.json
401 - Create new feed with signatures
402
403**Testing**:
404- New posts are signature-only
405- Timeline works with new format
406- Migration tool converts old data
407
408**Rollback**: BREAKING - need to restore from backup
409
410### Phase 5: Add Content Sync (Feature Complete)
411**Goal**: Fetch content from peers on-demand
412
413**Tasks**:
4141. Copy `content-sync.js` to pear-app
4152. Copy `hyperswarm-content-transport.js` to pear-app
4163. Initialize ContentSync with Hyperswarm transport
4174. Attach Protomux channel to Hyperswarm connections
4185. Request content from peers when not cached
419 - Check content.json first
420 - If missing, request via ContentSync
421 - Cache response
4226. Respond to peer content requests
423 - Listen for content-request messages
424 - Serve from content.json if available
425
426**Testing**:
427- Content requests work over Hyperswarm
428- Peers can fetch missing content
429- Content cached after fetch
430
431**Feature Complete**: Full ANProto + content-by-hash implementation
432
433### Phase 6: Wiredove Bridge (Optional)
434**Goal**: Bridge with wiredove.net users
435
436**Tasks**:
4371. Relay can translate between formats
4382. Directory API already working
4393. Add wiredove feed replication
440 - Follow wiredove users
441 - Convert their YAML to Wirecore format
442 - Replicate via relay
443
444**Testing**:
445- Can follow wiredove.net users
446- Their posts appear in timeline
447- Can interact with them
448
449---
450
451## Migration Strategy
452
453### For Existing Users
454
455**Option A: Fresh Start (Recommended for Alpha)**
456- Ship new version as separate release
457- Users create new ANProto identity
458- Old data stays in old format
459- Export/import tools for manual migration
460
461**Option B: Dual-Format Reader**
462- New code reads both old and new formats
463- Old posts: Full JSON in Hypercore blocks
464- New posts: Signatures + content.json
465- Gradual migration as users post
466
467**Option C: Migration Command**
468```bash
469pear run . cli migrate-to-anproto
470
471# Converts:
472# - Creates anproto-keypair.json
473# - Extracts content from old Hypercore blocks
474# - Stores in content.json
475# - Creates new feed with signatures
476# - Preserves identity (exports/imports keypair)
477```
478
479### For New Users
480
481- Initialize with ANProto from start
482- No migration needed
483- Clean implementation
484
485---
486
487## Testing Strategy
488
489### Unit Tests
490```bash
491pear-app/test/
492├── anproto.test.js # ANProto signing/verification
493├── yaml-helper.test.js # YAML parsing/creation
494├── content-store.test.js # Content storage
495├── content-sync.test.js # Content fetching
496└── integration.test.js # Full message flow
497```
498
499### Integration Tests
5001. **Post Flow**:
501 - Create post → Verify signature in Hypercore
502 - Check content in content.json
503 - Verify YAML format
504
5052. **Timeline Flow**:
506 - Read signatures from feeds
507 - Fetch content from cache
508 - Parse YAML metadata
509 - Verify ANProto signatures
510
5113. **Content Sync**:
512 - Post on peer A
513 - Request from peer B
514 - Verify content received
515 - Check caching works
516
5174. **Wiredove Compat**:
518 - Post in Wirecore
519 - Read via wiredove.net
520 - Post in wiredove
521 - Read in Wirecore
522
523### Performance Tests
524- Measure signature-only vs full-message replication
525- Benchmark content fetching latency
526- Test with 1000+ posts in timeline
527- Measure cache hit rates
528
529---
530
531## Rollout Plan
532
533### Week 1: Phase 0-1 (ANProto Layer)
534- Copy ANProto module
535- Add keypair generation
536- Add signing to posts
537- Test signatures verify
538
539### Week 2: Phase 2-3 (Content Store + YAML)
540- Add content.json storage
541- Convert to YAML format
542- Test dual-format reading
543
544### Week 3: Phase 4 (Signature-Only Breaking Change)
545- Switch to signatures only
546- Build migration tool
547- Test migration
548
549### Week 4: Phase 5 (Content Sync)
550- Add ContentSync layer
551- Implement Protomux transport
552- Test peer content requests
553
554### Week 5: Phase 6 (Wiredove Bridge)
555- Test wiredove compatibility
556- Build relay bridge
557- Integration testing
558
559### Week 6: Testing & Documentation
560- Full integration tests
561- Performance testing
562- Update documentation
563- User migration guide
564
565---
566
567## Success Criteria
568
569### Functional
570- ✅ Posts use ANProto signatures
571- ✅ Content stored by hash
572- ✅ YAML format with metadata
573- ✅ Lazy content fetching works
574- ✅ Wiredove compatible
575- ✅ Existing users can migrate
576
577### Performance
578- ✅ Signature-only blocks < 200 bytes
579- ✅ Content fetch < 1 second
580- ✅ Timeline load < 2 seconds (100 posts)
581- ✅ Cache hit rate > 80%
582
583### Compatibility
584- ✅ Works with lib/core/wirecore.js
585- ✅ Works with wiredove.net
586- ✅ Works with relay bridge
587- ✅ Works in Pear runtime
588
589---
590
591## Open Questions
592
5931. **Storage Location**: Should anproto-keypair.json be separate or in Hyperbee?
594 - Separate: Easier to export/import
595 - Hyperbee: More integrated with other metadata
596
5972. **Image Storage**: Should large images use Hyperdrive or content.json?
598 - content.json: Simpler, consistent
599 - Hyperdrive: Better for large files, sparse downloads
600
6013. **Migration**: Force migration or support dual-format forever?
602 - Force: Cleaner codebase
603 - Dual: Better UX, no breaking change
604
6054. **Identity**: Keep both IdentityKey and ANProto keypair?
606 - Both: Backwards compatible
607 - ANProto only: Simpler, cleaner
608
6095. **Content Sync**: Should relay serve content-by-hash for web users?
610 - Yes: Better for web users (no P2P content transport)
611 - No: Keep relay simple (signatures only)
612
613---
614
615## Next Steps
616
6171. **Get Approval**: Review this plan with user
6182. **Answer Open Questions**: Make architectural decisions
6193. **Start Phase 1**: Copy ANProto module and begin implementation
6204. **Commit Frequently**: Small, incremental commits with clear messages
6215. **Test Continuously**: Test each phase before moving to next
622
623---
624
625**Status**: Awaiting approval to proceed
626**Estimated Timeline**: 6 weeks for full implementation
627**Risk Level**: Medium (breaking changes in Phase 4)
628**Rollback Plan**: Each phase can be reverted independently until Phase 4
629