mirror of
https://github.com/RayLabsHQ/gitea-mirror.git
synced 2026-10-30 22:21:58 +01:00
Sean Mousseau
6979c3bb32
The previous middleware logic gated recovery behind a once-per-process flag. After the first request handled the boot-time recovery check, the resume codepath never fired again — even when new interruptions appeared later in the process lifetime. Symptom: a sync that started after server boot, crashed mid-flight (deadlock retry hitting maxRetries, network blip, container restart of an upstream service, etc.) and never reached the resume codepath would sit at `inProgress=true, lastCheckpoint=never` forever. The periodic detector kept finding the stuck row and logging `Found 1 interrupted jobs:` on every poll (driven by the health endpoint via `hasJobsNeedingRecovery`), but the resumer (`resumeInterruptedJob`) was only invoked from `initializeRecovery` which the middleware never re-called. Fix: replace the one-shot gate with an in-flight latch (`recoveryInFlight`) that's released in a `finally` block. Throttling of actual recovery work is delegated to the existing 5-minute `skipIfRecentAttempt` check inside `initializeRecovery()`, which is the right place for it. `recoveryInitialized` is kept and only used to control whether the "first run" log lines fire. Secondary fix: `findInterruptedJobs` previously logged `Found N interrupted jobs:` unconditionally on every call, including from passive polls in `hasJobsNeedingRecovery` (which the health endpoint and middleware probe call frequently). That produced one log line per poll per stuck job for as long as the job stayed stuck. Make logging opt-in via a `logFound` parameter, default off; the active recovery cycle in `initializeRecovery()` opts in so the operator-facing log still surfaces which jobs are being worked on. Related: #268 (partially addressed). The PR #280 / v3.15.7 fix was about a JS scoping bug that made the *initial* migrate call's catch path crash before transitioning the repo to `failed`. That was one cause of stuck `mirroring` state; this PR addresses the follow-on issue that even when a job *is* correctly detectable as interrupted, the post-startup recovery path never re-engages. Adds `orchestrator-resume-after-startup.test.ts` using the structural-source test pattern from `gitea-mirror-failure-recovery.test.ts` so the four guarantees (no static gate; in-flight latch released in finally; logging opt-in; active path opts in) are enforced without heavy mocks.