OverviewFilesCommitsIssuesDocsReleasesPeers

pear-git commits / a4b4b94

Partial clone + sparse checkout for pgh app install
Major install optimization: download only the current platform's artifacts instead of the entire release branch. Changes: - repo-registry: ensurePartialCloneConfig() enables uploadpack.allowFilter on all bare repos during scan (required for --filter to work) - app install: uses --filter=blob:none --sparse for initial clone (~10 MB tree metadata), then sparse-checkout for by-arch/{host}/ blobs only - finishInstall: detects by-arch entry point, launcher cd's into the by-arch directory for correct relative requires - Design doc: docs/plans/2026-04-07-partial-clone-install.md Verified over punch:// P2P transport: Before: 980 MB full clone After: 9.8 MB tree + 566 MB platform blobs = 576 MB total For repos without by-arch (e.g., hyper-overlay), falls back to full checkout automatically. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
a4b4b947a4d5f57f8717ca4a79c0d6cc8899edb8
by plankatron <fleeky@gmail.com> · 2026-04-07T15:12:01.000Z

Changed files (3) +196 -4

docs/plans/2026-04-07-partial-clone-install.md +150 -0
……@@ -0,0 +1,150 @@
1# Partial Clone App Install
2
3## Problem
4
5`pgh app install` downloads the ENTIRE release branch (~1.4 GB for hyper-tps) even though the client only needs one platform's artifacts (~566 MB). Federation sync also transfers full packs because orphan release commits share no object history with peers.
6
7Current flow:
8```
9git clone --depth 1 -b release punch://key/repo # downloads ALL blobs (all platforms)
10```
11
12## Solution
13
14Use git's partial clone + sparse checkout to download only what the current platform needs.
15
16New flow:
17```
18git clone --filter=blob:none --sparse --depth 1 -b release punch://key/repo # 9.8 MB (trees only)
19cd repo
20git sparse-checkout init --cone
21git sparse-checkout set by-arch/{host}/app/{name} # fetches only this platform's blobs (~566 MB)
22```
23
24### Verified
25
26Tested 2026-04-07 on tabertron → stortron over punch:// P2P transport:
27- `--filter=blob:none` works over git-remote-punch (transparent proxy)
28- Clone with filter: 9.8 MB, ~2 seconds (vs 980 MB without filter)
29- Sparse checkout of linux-x64: 566 MB, ~55 seconds
30- All files present: index.js, hypertps binary, .pck, .so files, node_modules
31
32### Server Requirement
33
34Bare repos must have partial clone enabled:
35```bash
36git -C ~/git-repos/REPO.git config uploadpack.allowFilter true
37git -C ~/git-repos/REPO.git config uploadpack.allowAnySHA1InWant true
38```
39
40Without this, server responds "filtering not recognized" and falls back to full clone.
41
42## Implementation
43
44### 1. Enable partial clone on all bare repos
45
46**File:** `lib/repo-registry.js` — `ensurePostReceiveHook()` or new `ensureRepoConfig()`
47
48After scanning/creating a bare repo, set:
49```javascript
50await execGit(repoPath, ['config', 'uploadpack.allowFilter', 'true'])
51await execGit(repoPath, ['config', 'uploadpack.allowAnySHA1InWant', 'true'])
52```
53
54### 2. Update `pgh app install` git clone path
55
56**File:** `lib/cli/commands/app.js` — `appInstallGit()`
57
58Replace:
59```javascript
60await execGitInDir(parentDir, [
61 'clone', '--depth', '1', '-b', 'release', cloneUrl, name
62])
63```
64
65With:
66```javascript
67const host = detectHost() // e.g. 'linux-x64'
68
69// Step 1: partial clone (trees only, no blobs — ~10 MB)
70await execGitInDir(parentDir, [
71 'clone', '--filter=blob:none', '--sparse', '--depth', '1',
72 '-b', 'release', cloneUrl, name
73])
74
75// Step 2: sparse checkout (fetch only this platform's blobs)
76await execGitInDir(installDir, ['sparse-checkout', 'init', '--cone'])
77await execGitInDir(installDir, [
78 'sparse-checkout', 'set',
79 `by-arch/${host}/app/${repoName}`
80])
81```
82
83### 3. Update `pgh app install` update path
84
85**File:** `lib/cli/commands/app.js` — existing git-installed update
86
87Replace:
88```javascript
89await execGitInDir(installDir, ['fetch', 'origin', 'release', '--depth', '1'])
90await execGitInDir(installDir, ['reset', '--hard', 'origin/release'])
91```
92
93With:
94```javascript
95await execGitInDir(installDir, ['fetch', '--filter=blob:none', 'origin', 'release', '--depth', '1'])
96await execGitInDir(installDir, ['reset', '--hard', 'origin/release'])
97// Sparse checkout config persists — only fetches blobs for configured paths
98```
99
100### 4. Update `finishInstall` launcher
101
102**File:** `lib/cli/commands/app.js` — `finishInstall()`
103
104Detect by-arch layout and set entry point accordingly:
105```javascript
106const host = detectHost()
107const byArchEntry = path.join(installDir, 'by-arch', host, 'app', name, pkg.main || 'index.js')
108const rootEntry = path.join(installDir, pkg.main || 'index.js')
109
110const entry = existsSync(byArchEntry) ? byArchEntry : rootEntry
111// Launcher: exec bare {entry} "$@"
112```
113
114### 5. Federation sync — enable partial clone for syncRepo
115
116**File:** `lib/p2p/sync/git-sync-client-manager.js` — `_syncRepo()`
117
118The federation `git fetch` should also use filters when the remote supports it:
119```javascript
120const child = spawn('git', [
121 '-C', repoPath, 'fetch', '--filter=blob:none', fetchUrl,
122 'refs/*:refs/*', '--no-tags'
123])
124```
125
126This reduces federation sync bandwidth from full pack to tree-only deltas. Blobs are fetched on demand when clients request them via sparse checkout.
127
128## Fallback
129
130If the server doesn't support `--filter` (old git or missing config), git silently falls back to full clone. No breakage.
131
132The P2P connectivity check (`git ls-remote` ping) runs before clone, so connection issues are caught early regardless of filter support.
133
134## Size Comparison
135
136| Method | Download Size | Time |
137|--------|-------------|------|
138| Current: `git clone --depth 1` | 980 MB | ~4 min |
139| Partial clone (trees only) | 9.8 MB | ~2 sec |
140| + sparse checkout (linux-x64) | +566 MB | ~55 sec |
141| **Total with partial clone** | **576 MB** | **~1 min** |
142| Delta fetch (subsequent update) | ~few MB | ~seconds |
143
144## Future: Proper Delta Updates
145
146Current release commits are orphans (fresh `git commit-tree` from temp index). This defeats git's delta compression — every deploy transfers full blobs even for unchanged files.
147
148Fix: use `git read-tree release` to load the previous release tree into the index before `git add -A`. Only changed files create new blobs. Tree objects reuse unchanged subtrees. This would make delta fetches (test 5 in p2p-fetch-size.js) transfer only the actual changes — a few MB instead of hundreds.
149
150**File:** `lib/ci/deploy-hook.js` — `_commitToReleaseBranch()`
lib/cli/commands/app.js +28 -4
……@@ -238,13 +238,30 @@ async function appInstallGit (repoName, flags) {
238238 if (p2pReachable) {
239239 if (flags.force) cleanup(installDir)
240240
241 log('Cloning ' + repoName + ' (release branch) from ' + cloneUrl + '...')
241 const host = detectHost()
242 log('Cloning ' + repoName + ' (release branch, partial clone) from ' + cloneUrl + '...')
242243
243244 try {
244245 fs.mkdirSync(path.dirname(installDir), { recursive: true })
246
247 // Partial clone: download tree metadata only (~10 MB), no blobs
245248 await execGitInDir(path.dirname(installDir), [
246 'clone', '--depth', '1', '-b', 'release', cloneUrl, name
249 'clone', '--filter=blob:none', '--sparse', '--depth', '1',
250 '-b', 'release', cloneUrl, name
247251 ])
252
253 // Sparse checkout: fetch only this platform's blobs
254 const byArchPath = `by-arch/${host}/app/${repoName}`
255 try {
256 await execGitInDir(installDir, ['sparse-checkout', 'init', '--cone'])
257 await execGitInDir(installDir, ['sparse-checkout', 'set', byArchPath])
258 log('Sparse checkout: ' + byArchPath)
259 } catch {
260 // No by-arch layout — fetch all blobs (small repos like hyper-overlay)
261 await execGitInDir(installDir, ['sparse-checkout', 'disable'])
262 log('No by-arch layout, full checkout')
263 }
264
248265 return finishInstall(repoName, name, installDir, flags, 'git')
249266 } catch (err) {
250267 log('P2P git clone failed (' + err.message + '), falling back to HTTP archive...')
……@@ -701,7 +718,11 @@ async function finishInstall (repoName, name, installDir, flags, method) {
701718 try {
702719 pkg = JSON.parse(fs.readFileSync(path.join(installDir, 'package.json'), 'utf-8'))
703720 } catch {}
704 const resolvedEntry = pkg.main || 'index.js'
721 // Resolve entry point — prefer by-arch layout if present
722 const host = detectHost()
723 const byArchEntry = path.join('by-arch', host, 'app', repoName, pkg.main || 'index.js')
724 const rootEntry = pkg.main || 'index.js'
725 const resolvedEntry = existsSync(path.join(installDir, byArchEntry)) ? byArchEntry : rootEntry
705726
706727 // Store deploy info for self-update
707728 try {
……@@ -740,7 +761,10 @@ async function finishInstall (repoName, name, installDir, flags, method) {
740761 // Create launcher script
741762 fs.mkdirSync(BIN_DIR, { recursive: true })
742763 const launcherPath = path.join(BIN_DIR, name)
743 const launcherContent = `#!/bin/sh\nexec bare ${path.join(installDir, resolvedEntry)} "$@"\n`
764 const entryAbsolute = path.join(installDir, resolvedEntry)
765 const entryDir = path.dirname(entryAbsolute)
766 const entryFile = path.basename(entryAbsolute)
767 const launcherContent = `#!/bin/sh\ncd ${entryDir}\nexec bare ${entryFile} "$@"\n`
744768 fs.writeFileSync(launcherPath, launcherContent)
745769 fs.chmodSync(launcherPath, 0o755)
746770
lib/repo-registry.js +18 -0
……@@ -272,6 +272,7 @@ export class RepoRegistry {
272272
273273 // Auto-install post-receive hook if missing
274274 await this.ensurePostReceiveHook(fullPath)
275 await this.ensurePartialCloneConfig(fullPath)
275276
276277 const repoInfo = await this.getRepoInfo(fullPath, repoName)
277278
……@@ -310,6 +311,23 @@ export class RepoRegistry {
310311 * Without this hook, pushes to the bare repo won't trigger federation
311312 * sync notifications and peers won't learn about new refs.
312313 */
314 /**
315 * Enable partial clone support on a bare repo.
316 * Required for --filter=blob:none to work during fetch/clone.
317 * Without this, git ignores filter requests and sends full packs.
318 */
319 async ensurePartialCloneConfig (repoPath) {
320 try {
321 const current = await this.execGit(repoPath, ['config', '--get', 'uploadpack.allowFilter']).catch(() => '')
322 if (current.trim() === 'true') return
323 await this.execGit(repoPath, ['config', 'uploadpack.allowFilter', 'true'])
324 await this.execGit(repoPath, ['config', 'uploadpack.allowAnySHA1InWant', 'true'])
325 console.log(` ✓ Enabled partial clone: ${path.basename(repoPath, '.git')}`)
326 } catch (err) {
327 console.error(` ✗ Failed to enable partial clone for ${path.basename(repoPath)}: ${err.message}`)
328 }
329 }
330
313331 async ensurePostReceiveHook (repoPath) {
314332 const hookPath = path.join(repoPath, 'hooks', 'post-receive')
315333