OverviewFilesCommitsIssuesDocsReleasesPeers

wirecore commits / 7784789

Add prebuild pruning optimization - reduce package from 304MB to 68MB
Created prune-prebuilds.js script that removes platform-specific prebuilds for all platforms except the current one (e.g., linux-x64), achieving a 77.6% size reduction from 304MB to 68MB. Changes: - Add prune-prebuilds.js: Recursive prebuild pruning script - Detects current platform (os.platform() + os.arch()) - Scans all node_modules including deeply nested packages - Removes prebuilds for 12 other platforms (Android, iOS, macOS, Windows, etc.) - Compatible with Node v10+ (no fs.rmSync dependency) - Processes 33 native packages, removes 276 platform directories - Update package.json: Integrate pruning into staging workflow - Add "prune" script: node prune-prebuilds.js - Update "stage" script: npm run prune && pear stage wirecore . - Automatic optimization before each staging - Update .pearignore: Exclude unnecessary files in node_modules - Remove docs, tests, examples, benchmarks - Remove license files, changelogs, config files - Remove source maps - Keep actual module code and binaries - Update IMPLEMENTATION_PROGRESS.md: Document Phase 1.9 - Add package size optimization results - Document breakdown of remaining 68MB - Add lessons learned about prebuild optimization - Update deployment process with new version (v2774) Results: - Original size: 304 MB (2212 files) - Optimized size: 68 MB (reduced by 236 MB / 77.6%) - Remaining breakdown: - App code: ~300 KB - JavaScript: ~7 MB - Native binaries (linux-x64): ~60 MB - bare-tls: 11 MB - bare-crypto: 5.2 MB - sodium-native (10 copies): ~22 MB - Other native modules: ~21 MB Staged as version 2774 to pear://zupx3qzpwj61hdt1bxanabctcyxqqu3arogctwim4qt3wfbzn4bo Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
778478964326468dd45581487c6616f457951fcf
by plankatron <fleeky@gmail.com> · 2026-01-20T21:18:29.000Z

Changed files (4) +291 -8

IMPLEMENTATION_PROGRESS.md +54 -3
……@@ -149,7 +149,7 @@ Researched 72 Pear-related tools in holepunchto ecosystem:
1491491. **Staging**: `pear stage wirecore .`
150150 - Created DHT channel "wirecore"
151151 - Generated immutable key
152 - Uploaded 2212 files (304MB with node_modules)
152 - ~~Uploaded 2212 files (304MB with node_modules)~~ → Optimized to 68MB (v2774)
153153
1541542. **Seeding**: `pear seed wirecore .` (background process)
155155 - Makes app available on P2P network
……@@ -160,6 +160,51 @@ Researched 72 Pear-related tools in holepunchto ecosystem:
160160 - Confirmed module imports work in Bare runtime
161161 - Validated across different directories
162162
163#### Package Size Optimization (Phase 1.9)
164
165**Problem**: Initial package size was 304MB (99% from node_modules prebuilds)
166
167**Root Cause**: Native modules ship binaries for 13 platforms:
168- Android (4 architectures)
169- iOS (3 variants)
170- macOS (2 architectures)
171- Linux (2 architectures)
172- Windows (2 architectures)
173
174**Solution**: Created `prune-prebuilds.js` script that:
1751. Detects current platform (e.g., linux-x64)
1762. Recursively scans all node_modules (including nested)
1773. Removes prebuilds for other 12 platforms
1784. Runs automatically as part of `npm run stage`
179
180**Results**:
181- Original size: 304 MB
182- Optimized size: 68 MB
183- **Saved: 236 MB (77.6% reduction!)**
184- Processed 33 native packages
185- Removed 276 platform directories
186
187**Breakdown of remaining 68MB**:
188- App code: ~300 KB
189- JavaScript modules: ~7 MB
190- Native binaries (linux-x64 only): ~60 MB
191 - bare-tls: 11 MB (TLS/SSL library)
192 - bare-crypto: 5.2 MB (crypto primitives)
193 - sodium-native (10 copies): ~22 MB
194 - Other native modules: ~21 MB
195
196**Integration**:
197```json
198{
199 "scripts": {
200 "prune": "node prune-prebuilds.js",
201 "stage": "npm run prune && pear stage wirecore ."
202 }
203}
204```
205
206**Current Version**: v2774 (optimized)
207
163208#### Documentation Created
164209
165210- **PEAR_STAGING_GUIDE.md** - Complete reference documentation:
……@@ -201,9 +246,12 @@ test-storage/
201246#### Lessons Learned
202247
2032481. **Module Resolution**: Bare/Pear doesn't support dynamic imports from Node.js builtins
2042. **node_modules Required**: Must include in staging despite size (304MB) due to module resolution bugs
2492. **node_modules Required**: Must include in staging ~~despite size (304MB)~~ but can optimize (68MB after pruning)
2052503. **Background Seeding**: Seed process must run continuously for DHT availability
2062514. **Try-Catch Pattern**: Better than filesystem existence checks for Bare compatibility
2525. **Prebuild Optimization**: Native modules ship 13 platform variants - prune before staging
2536. **Recursive Pruning**: Must check ALL directories at all depths, not just top-level packages
2547. **Platform Detection**: Use `os.platform()` + `os.arch()` to keep only current platform's binaries
207255
208256#### Production Considerations
209257
……@@ -360,12 +408,15 @@ All Phase 1 objectives achieved:
360408- Phase 1.6: Testing and validation ✅
361409- Phase 1.7: P2P networking validation ✅
362410- **Phase 1.8: Pear packaging & distribution** ✅
411- **Phase 1.9: Package size optimization** ✅
363412
364### Phase 1.8 Highlights - Pear Packaging
413### Phase 1.8 + 1.9 Highlights - Pear Packaging & Optimization
365414- ✅ Successfully staged to DHT
366415- ✅ Seeding active (background process)
367416- ✅ Fixed module resolution bugs for Bare/Pear runtime
368417- ✅ Installation link: `pear://zupx3qzpwj61hdt1bxanabctcyxqqu3arogctwim4qt3wfbzn4bo`
418- ✅ Optimized package size: **304 MB → 68 MB** (77.6% reduction!)
419- ✅ Automated prebuild pruning via `prune-prebuilds.js`
369420- ✅ Tested installation, init, post, timeline commands
370421- ✅ Comprehensive documentation created
371422
pear-app/.pearignore +28 -3
……@@ -1,5 +1,6 @@
11# Pear stage ignore file
22# CRITICAL: Do NOT ignore node_modules (module resolution bug under pear://)
3# But we CAN ignore unnecessary files WITHIN node_modules
34
45# Development files
56.git
……@@ -9,16 +10,40 @@
910
1011# Test files
1112test/
13**/test/
14**/*.test.js
15**/*.test.mjs
1216
13# Documentation
14README.md
17# Documentation (except our own README)
18/README.md
1519docs/
20**/docs/
21**/*.md
22!README.md
1623
1724# Build artifacts
1825*.swp
1926*.swo
2027*~
2128
22# Local data (should be per-installation)
29# Local data (per-installation)
2330wirecore-storage/
2431test-storage/
32
33# Unnecessary files in node_modules
34node_modules/**/.git
35node_modules/**/.github
36node_modules/**/test
37node_modules/**/tests
38node_modules/**/example
39node_modules/**/examples
40node_modules/**/benchmark
41node_modules/**/*.md
42node_modules/**/LICENSE
43node_modules/**/CHANGELOG*
44node_modules/**/.npmignore
45node_modules/**/.editorconfig
46node_modules/**/.eslintrc*
47node_modules/**/.travis.yml
48node_modules/**/tsconfig.json
49node_modules/**/*.map
pear-app/package.json +3 -2
……@@ -13,8 +13,9 @@
1313 },
1414 "scripts": {
1515 "dev": "pear run -d .",
16 "stage": "pear stage .",
17 "seed": "pear seed .",
16 "prune": "node prune-prebuilds.js",
17 "stage": "npm run prune && pear stage wirecore .",
18 "seed": "pear seed wirecore .",
1819 "test": "brittle test/*.js"
1920 },
2021 "dependencies": {
pear-app/prune-prebuilds.js +206 -0
……@@ -0,0 +1,206 @@
1#!/usr/bin/env node
2/**
3 * Prune platform-specific prebuilds from node_modules
4 *
5 * This removes prebuilds for platforms other than the current one,
6 * reducing package size from ~300MB to ~40MB (85% reduction)
7 *
8 * Run after: npm install
9 * Run before: pear stage
10 */
11
12const fs = require('fs')
13const path = require('path')
14const os = require('os')
15
16// Detect current platform
17const platform = os.platform() // 'linux', 'darwin', 'win32'
18const arch = os.arch() // 'x64', 'arm64', etc.
19const currentPlatform = `${platform}-${arch}`
20
21console.log(`🔍 Current platform: ${currentPlatform}`)
22console.log(`🗑️ Removing prebuilds for other platforms...\n`)
23
24let totalRemoved = 0
25let totalKept = 0
26let packagesProcessed = 0
27
28/**
29 * Recursively find and prune prebuilds directories
30 */
31function pruneDirectory(dir, depth = 0) {
32 if (depth > 10) return // Prevent infinite recursion
33
34 try {
35 const entries = fs.readdirSync(dir, { withFileTypes: true })
36
37 for (const entry of entries) {
38 const fullPath = path.join(dir, entry.name)
39
40 if (entry.isDirectory()) {
41 if (entry.name === 'prebuilds') {
42 // Found a prebuilds directory - prune it
43 prunePrebuild(fullPath)
44 packagesProcessed++
45 } else if (entry.name === 'node_modules') {
46 // Recurse into nested node_modules
47 pruneDirectory(fullPath, depth + 1)
48 } else {
49 // Check every directory for prebuilds and nested node_modules
50 const prebuildsPath = path.join(fullPath, 'prebuilds')
51 if (fs.existsSync(prebuildsPath)) {
52 prunePrebuild(prebuildsPath)
53 packagesProcessed++
54 }
55 // Also check nested node_modules
56 const nestedNodeModules = path.join(fullPath, 'node_modules')
57 if (fs.existsSync(nestedNodeModules)) {
58 pruneDirectory(nestedNodeModules, depth + 1)
59 }
60 }
61 }
62 }
63 } catch (err) {
64 // Ignore permission errors, etc.
65 console.error(`⚠️ Error reading ${dir}: ${err.message}`)
66 }
67}
68
69/**
70 * Recursively remove directory (compatible with Node v10+)
71 */
72function removeDir(dir) {
73 if (!fs.existsSync(dir)) return
74
75 const entries = fs.readdirSync(dir, { withFileTypes: true })
76
77 for (const entry of entries) {
78 const fullPath = path.join(dir, entry.name)
79
80 if (entry.isDirectory()) {
81 removeDir(fullPath)
82 } else {
83 fs.unlinkSync(fullPath)
84 }
85 }
86
87 fs.rmdirSync(dir)
88}
89
90/**
91 * Prune a specific prebuilds directory
92 */
93function prunePrebuild(prebuildsPath) {
94 try {
95 const platforms = fs.readdirSync(prebuildsPath)
96 const packageName = path.basename(path.dirname(prebuildsPath))
97
98 let removedPlatforms = []
99 let keptPlatforms = []
100
101 for (const platform of platforms) {
102 const platformPath = path.join(prebuildsPath, platform)
103
104 let stat
105 try {
106 stat = fs.statSync(platformPath)
107 } catch (err) {
108 continue
109 }
110
111 if (!stat.isDirectory()) continue
112
113 if (platform === currentPlatform) {
114 // Keep current platform
115 keptPlatforms.push(platform)
116 totalKept++
117 } else {
118 // Remove other platforms
119 removeDir(platformPath)
120 removedPlatforms.push(platform)
121 totalRemoved++
122 }
123 }
124
125 if (removedPlatforms.length > 0) {
126 console.log(`📦 ${packageName}:`)
127 console.log(` ✅ Kept: ${keptPlatforms.join(', ') || 'none'}`)
128 console.log(` ❌ Removed: ${removedPlatforms.join(', ')}`)
129 }
130 } catch (err) {
131 console.error(`⚠️ Error processing ${prebuildsPath}: ${err.message}`)
132 }
133}
134
135/**
136 * Get directory size
137 */
138function getDirSize(dir) {
139 let size = 0
140
141 try {
142 const entries = fs.readdirSync(dir, { withFileTypes: true })
143
144 for (const entry of entries) {
145 const fullPath = path.join(dir, entry.name)
146
147 if (entry.isDirectory()) {
148 size += getDirSize(fullPath)
149 } else {
150 const stat = fs.statSync(fullPath)
151 size += stat.size
152 }
153 }
154 } catch (err) {
155 // Ignore errors
156 }
157
158 return size
159}
160
161/**
162 * Format bytes to human-readable
163 */
164function formatBytes(bytes) {
165 if (bytes < 1024) return bytes + ' B'
166 if (bytes < 1024 * 1024) return (bytes / 1024).toFixed(1) + ' KB'
167 return (bytes / (1024 * 1024)).toFixed(1) + ' MB'
168}
169
170// Main execution
171const nodeModulesPath = path.join(__dirname, 'node_modules')
172
173if (!fs.existsSync(nodeModulesPath)) {
174 console.error('❌ node_modules not found. Run npm install first.')
175 process.exit(1)
176}
177
178// Measure before
179const sizeBefore = getDirSize(nodeModulesPath)
180console.log(`📊 Size before: ${formatBytes(sizeBefore)}\n`)
181
182// Prune
183const startTime = Date.now()
184pruneDirectory(nodeModulesPath)
185const duration = ((Date.now() - startTime) / 1000).toFixed(2)
186
187// Measure after
188const sizeAfter = getDirSize(nodeModulesPath)
189const saved = sizeBefore - sizeAfter
190const percentSaved = ((saved / sizeBefore) * 100).toFixed(1)
191
192// Summary
193console.log(`\n✅ Pruning complete in ${duration}s`)
194console.log(`\n📊 Summary:`)
195console.log(` Packages processed: ${packagesProcessed}`)
196console.log(` Platforms kept: ${totalKept}`)
197console.log(` Platforms removed: ${totalRemoved}`)
198console.log(` Size before: ${formatBytes(sizeBefore)}`)
199console.log(` Size after: ${formatBytes(sizeAfter)}`)
200console.log(` Saved: ${formatBytes(saved)} (${percentSaved}%)`)
201
202if (saved > 1024 * 1024) {
203 console.log(`\n🎉 Saved ${formatBytes(saved)}! Ready for staging.`)
204} else {
205 console.log(`\n⚠️ Minimal savings. Prebuilds may already be pruned.`)
206}