Adds a Pre-releases switch (stable only), include and exclude tag patterns (globs or /regex/) with a preview of what a save keeps or removes, per organization and per repository overrides, and RELEASE_TYPE, RELEASE_INCLUDE_TAGS and RELEASE_EXCLUDE_TAGS. Releases in the destination that stop matching are removed on the next sync, and a warning is raised when that empties a destination.
Three things combined to make crash recovery restart live jobs:
- isJobInterrupted treated any job older than two hours as interrupted before
looking at its checkpoint, so a large job that heartbeated every two minutes
was resumed while it was still running. The age rule is gone: a checkpoint
is the job's proof of life, and only a missing or stale one (ten minutes)
marks a job interrupted.
- processWithRetry only checkpointed the item that tripped the interval, so a
resume re-processed up to N-1 finished items. Every completed item is now
recorded; the interval only throttles the published progress event.
- updateMirrorJobProgress was a read-modify-write of completedItemIds with no
lock, so two items finishing in the same tick could drop each other. Writes
are serialized per job with the keyed lock from #419.
A resumed job also records a fresh start time. Found while investigating #417.
A job that had just started, with in_progress=1 and no checkpoint yet,
matched findInterruptedJobs immediately. Since #297 the middleware runs
that check on every request, so a live job was "resumed" by recovery
while the original process was still working on the same repositories.
The request-level 15 second timeout then released the in-flight latch
without cancelling recovery, and later requests logged misleading
"already in progress" and "completed with some issues" lines.
- Move the liveness rule into interrupted-job-detection.ts, as both a
predicate and the SQL condition used by findInterruptedJobs. A job
with no checkpoint is only interrupted once it is older than the 10
minute checkpoint window (or has no recorded start at all).
- Stamp an initial checkpoint on in-progress jobs at creation.
- Refresh the checkpoint every 2 minutes from processWithResilience so a
single long item cannot make a live job look interrupted. The refresh
only touches rows still in progress.
- Keep the middleware recovery latch held until the recovery promise
settles, not until the request stops waiting, and clear the timeout
timer so it no longer leaves a dangling rejection.
Tests cover the predicate, the SQL against an in-memory SQLite database,
and the wiring into helpers, concurrency, and middleware.
Claude-Session: https://claude.ai/code/session_01Tp9pmi65a8k5jLMQFLf4JX
The previous middleware logic gated recovery behind a
once-per-process flag. After the first request handled the boot-time
recovery check, the resume codepath never fired again — even when
new interruptions appeared later in the process lifetime.
Symptom: a sync that started after server boot, crashed mid-flight
(deadlock retry hitting maxRetries, network blip, container
restart of an upstream service, etc.) and never reached the resume
codepath would sit at `inProgress=true, lastCheckpoint=never`
forever. The periodic detector kept finding the stuck row and
logging `Found 1 interrupted jobs:` on every poll (driven by the
health endpoint via `hasJobsNeedingRecovery`), but the resumer
(`resumeInterruptedJob`) was only invoked from `initializeRecovery`
which the middleware never re-called.
Fix: replace the one-shot gate with an in-flight latch
(`recoveryInFlight`) that's released in a `finally` block. Throttling
of actual recovery work is delegated to the existing 5-minute
`skipIfRecentAttempt` check inside `initializeRecovery()`, which is
the right place for it. `recoveryInitialized` is kept and only used
to control whether the "first run" log lines fire.
Secondary fix: `findInterruptedJobs` previously logged
`Found N interrupted jobs:` unconditionally on every call, including
from passive polls in `hasJobsNeedingRecovery` (which the health
endpoint and middleware probe call frequently). That produced one
log line per poll per stuck job for as long as the job stayed stuck.
Make logging opt-in via a `logFound` parameter, default off; the
active recovery cycle in `initializeRecovery()` opts in so the
operator-facing log still surfaces which jobs are being worked on.
Related: #268 (partially addressed). The PR #280 / v3.15.7 fix was
about a JS scoping bug that made the *initial* migrate call's
catch path crash before transitioning the repo to `failed`. That
was one cause of stuck `mirroring` state; this PR addresses the
follow-on issue that even when a job *is* correctly detectable as
interrupted, the post-startup recovery path never re-engages.
Adds `orchestrator-resume-after-startup.test.ts` using the
structural-source test pattern from
`gitea-mirror-failure-recovery.test.ts` so the four guarantees
(no static gate; in-flight latch released in finally; logging
opt-in; active path opts in) are enforced without heavy mocks.
* feat: add notification system with Ntfy.sh and Apprise providers (#231)
Add push notification support for mirror job events with two providers:
- Ntfy.sh: direct HTTP POST to ntfy topics with priority/tag support
- Apprise API: aggregator gateway supporting 100+ notification services
Includes database migration (0010), settings UI tab, test endpoint,
auto-save integration, token encryption, and comprehensive tests.
Notifications are fire-and-forget and never block the mirror flow.
* fix: address review findings for notification system
- Fix silent catch in GET handler that returned ciphertext to UI,
causing double-encryption on next save. Now clears token to ""
on decryption failure instead.
- Add Zod schema validation to test notification endpoint, following
project API route pattern guidelines.
- Mark notifyOnNewRepo toggle as "coming soon" with disabled state,
since the backend doesn't yet emit new_repo events. The schema
and type support is in place for when it's implemented.
* fix notification gating and config validation
* trim sync notification details
- Added a startup recovery script to handle interrupted jobs before application startup.
- Enhanced recovery system with database connection validation and stale job cleanup.
- Improved middleware to check for recovery needs and handle recovery during requests.
- Updated health check endpoint to include recovery system status and metrics.
- Introduced test scripts for verifying recovery functionality and job state management.
- Enhanced logging and error handling throughout the recovery process.
- Added new fields to the mirror_jobs table for job resilience, including job_type, batch_id, total_items, completed_items, item_ids, completed_item_ids, in_progress, started_at, completed_at, and last_checkpoint.
- Implemented database migration scripts to update the mirror_jobs table schema.
- Introduced processWithResilience utility for handling item processing with checkpointing and recovery capabilities.
- Updated API routes for mirroring organizations and repositories to utilize the new resilience features.
- Created recovery system to detect and resume interrupted jobs on application startup.
- Added middleware to initialize the recovery system when the server starts.
Florian Zevedei