The previous middleware logic gated recovery behind a
once-per-process flag. After the first request handled the boot-time
recovery check, the resume codepath never fired again — even when
new interruptions appeared later in the process lifetime.
Symptom: a sync that started after server boot, crashed mid-flight
(deadlock retry hitting maxRetries, network blip, container
restart of an upstream service, etc.) and never reached the resume
codepath would sit at `inProgress=true, lastCheckpoint=never`
forever. The periodic detector kept finding the stuck row and
logging `Found 1 interrupted jobs:` on every poll (driven by the
health endpoint via `hasJobsNeedingRecovery`), but the resumer
(`resumeInterruptedJob`) was only invoked from `initializeRecovery`
which the middleware never re-called.
Fix: replace the one-shot gate with an in-flight latch
(`recoveryInFlight`) that's released in a `finally` block. Throttling
of actual recovery work is delegated to the existing 5-minute
`skipIfRecentAttempt` check inside `initializeRecovery()`, which is
the right place for it. `recoveryInitialized` is kept and only used
to control whether the "first run" log lines fire.
Secondary fix: `findInterruptedJobs` previously logged
`Found N interrupted jobs:` unconditionally on every call, including
from passive polls in `hasJobsNeedingRecovery` (which the health
endpoint and middleware probe call frequently). That produced one
log line per poll per stuck job for as long as the job stayed stuck.
Make logging opt-in via a `logFound` parameter, default off; the
active recovery cycle in `initializeRecovery()` opts in so the
operator-facing log still surfaces which jobs are being worked on.
Related: #268 (partially addressed). The PR #280 / v3.15.7 fix was
about a JS scoping bug that made the *initial* migrate call's
catch path crash before transitioning the repo to `failed`. That
was one cause of stuck `mirroring` state; this PR addresses the
follow-on issue that even when a job *is* correctly detectable as
interrupted, the post-startup recovery path never re-engages.
Adds `orchestrator-resume-after-startup.test.ts` using the
structural-source test pattern from
`gitea-mirror-failure-recovery.test.ts` so the four guarantees
(no static gate; in-flight latch released in finally; logging
opt-in; active path opts in) are enforced without heavy mocks.
mirrorGitRepoIssuesToGitea and mirrorGitRepoPullRequestsToGitea both
had two compounding bugs that produced duplicate Gitea issues/PR-as-
issue rows on every sync against any non-SQLite backend.
(1) Pagination on existing-issues / existing-PRs / per-issue-comments
was wrong.
The loops paginated with `limit=100` and broke on `pageX.length <
itemsPerPage`. Gitea caps response size at server-side
[api].MAX_RESPONSE_ITEMS (default 50), so the very first page
already looks "short" and pagination terminated after one page. The
existing-issue and existing-PR maps were built from only ~50 items
per repo, so every issue/PR past page 1 was treated as new on every
sync and re-created via the CREATE branch.
Naive removal of the short-page break (relying only on "break on
empty page") doesn't work either: for some Gitea endpoints Gitea
returns the same data on every page when the page is past the
actual end instead of returning [], so the loop runs forever.
Fix: use the Link header (RFC 5988). If `rel="next"` is absent,
terminate pagination. Applied to: existing-issues pre-fetch,
existing-PRs pre-fetch, and per-issue comments fetch.
(2) Retry-after-deadlock duplicated issues/PRs even when the map
was correct.
Gitea's CreateIssue handler commits the issue insert in one
transaction and then deadlocks on the subsequent addLabel /
UPDATE repository transaction. The issue row is committed and
visible, but the in-memory dedup maps are never refreshed between
retries — processWithRetry re-invokes the callback, sees
existingIssue === undefined from the stale map, and creates a fresh
duplicate via httpPost.
Reproduces deterministically on MySQL (Error 1213 / 40001) and
PostgreSQL (40P01); SQLite escapes because writes serialize globally.
Fix: defensive recheck via httpGet by [GH-ISSUE #N] (issues) or
[PR #N] (pull requests) before the create call, with PATCH
fall-through when found. Applied to the issues create path AND
both create paths in the PR mirror (enriched + basic-fallback).
Also cache freshly-created items into the dedup maps after a
successful create so subsequent retries of the same per-item
callback don't lose track of it.
Adds gitea-issue-dedup-on-retry.test.ts using the structural-source
test pattern from gitea-mirror-failure-recovery.test.ts so all
guarantees are enforced without heavy module mocks (8 tests).
Sean Mousseau