OverviewFilesCommitsIssuesDocsReleasesPeers

hyper-sync commits / fc12a48

pull: pre-fetch tmp_pack cleanup + retry loop
Addresses two related failure modes observed on stortron: 1. Abandoned tmp_pack files accumulate indefinitely. An interrupted git fetch leaves `objects/pack/tmp_pack_XXX` orphaned; these never get cleaned up. Seen 37 accumulated on hyper-overlay.git alone. Waste disk and make repo operations slower over time. 2. Transient failures weren't retried. A single network hiccup during a large-repo pull would mark the repo as failed permanently until the next manual resync. Fix: before each pull attempt, unlink any tmp_pack_* in the pack dir so the state is deterministic. Wrap the fetch in a bounded retry loop (3 attempts, exponential backoff) with re-cleanup between attempts. A failed attempt's own garbage can't contaminate the next. Does NOT fix the underlying bloat problem — hyper-overlay is still 13 GB, so pulls still take minutes. The real fix for that is `git filter-repo --path node_modules --invert-paths` on the publisher side, which shrinks the repo ~30x. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fc12a48ca959e2187130b65c65896a40e81364c0
by plankatron <fleeky@gmail.com> · 2026-04-21T19:52:10.000Z

Changed files (1) +47 -4

index.js +47 -4
……@@ -204,16 +204,59 @@ module.exports = class HyperSync extends ReadyResource {
204204 }
205205 }
206206
207 // Pre-fetch cleanup: git-receive-pack (and git-fetch in the
208 // remote-punch transport) writes incoming data to
209 // `objects/pack/tmp_pack_XXX` and only renames to a finalized
210 // `pack-<sha>.pack` on success. An interrupted transfer leaves
211 // the tmp file orphaned; these accumulate indefinitely (we've
212 // seen 37 on hyper-overlay) and waste disk + slow operations.
213 // Clean them before starting each fetch so the state is
214 // deterministic and retries aren't contaminated by stale data.
215 const packDir = path.join(repoPath, 'objects', 'pack');
216 try {
217 for (const f of fs.readdirSync(packDir)) {
218 if (f.startsWith('tmp_pack_')) {
219 try { fs.unlinkSync(path.join(packDir, f)) } catch {}
220 }
221 }
222 } catch {}
223
207224 const fetchUrl = 'punch://' + remote.serverKey + '/' + name;
208225
209226 this.emit('pull', remoteName, name, 'start');
210227
211 try {
212 await this._gitFetch(repoPath, fetchUrl);
228 // Retry on transient failures. The punch:// transport is stateless
229 // per-attempt — pre-fetch cleanup above gives each retry a clean
230 // slate. Exponential backoff bounds total time.
231 const maxAttempts = 3;
232 let lastErr = null;
233 let synced = false;
234 for (let attempt = 1; attempt <= maxAttempts; attempt++) {
235 try {
236 await this._gitFetch(repoPath, fetchUrl);
237 synced = true;
238 break;
239 } catch (err) {
240 lastErr = err;
241 if (attempt < maxAttempts) {
242 // Re-clean tmp_packs left by this failed attempt before retry
243 try {
244 for (const f of fs.readdirSync(packDir)) {
245 if (f.startsWith('tmp_pack_')) {
246 try { fs.unlinkSync(path.join(packDir, f)) } catch {}
247 }
248 }
249 } catch {}
250 await new Promise(r => setTimeout(r, 1000 * attempt));
251 }
252 }
253 }
254
255 if (synced) {
213256 result.synced.push(name);
214257 this.emit('pull', remoteName, name, 'complete');
215 } catch (err) {
216 result.failed.push({ name, error: err.message });
258 } else {
259 result.failed.push({ name, error: lastErr ? lastErr.message : 'unknown error', attempts: maxAttempts });
217260 this.emit('pull', remoteName, name, 'failed');
218261 }
219262 }