[BUG] Scheduler remains stuck as "already running" after GitHub rate-limit backoff #174

New Issue

2026-10-05T01:45:50+02:00

adam commented

2026-10-05 01:45:50 +02:00

Description

When Gitea-Mirror hits the GitHub API rate limit during a scheduled metadata sync, it waits until the GitHub rate limit resets:

[GitHub] Waiting 3263s before retry...

During this wait, the scheduler continues to run its periodic checks, but every new cycle is skipped:

[Scheduler] Processing 1 valid configurations (out of 1 with scheduling enabled)
[Scheduler] Skipping user <user-id> - next run at 2026-09-16T09:00:00.000Z

After the configured nextRun time has passed, the scheduler does not resume normally.

Instead, subsequent scheduler attempts eventually report that the scheduler is already running, and no further repositories are synchronized.

The only way I have found to resume synchronization is to restart the Gitea-Mirror service/container. I also need to mark failed repositories for retry again.

Environment

Gitea-Mirror: v3.36.2
Destination: Forgejo
Source: GitHub

Steps to reproduce

Enable scheduled synchronization for a GitHub source.
Have enough repositories/metadata to exhaust the GitHub API rate limit.
Let Gitea-Mirror reach the GitHub API limit.
Observe:

[GitHub] Rate limit hit for GET https://api.github.com/user/repos?per_page=100&affiliation=owner%2Corganization_member. Retry 1/3
Publishing event to channel rate-limit for user <user-id>
Event published successfully with ID <ID-uuid>
[GitHub] Waiting 3263s before retry...
[Scheduler] Processing 1 valid configurations (out of 1 with scheduling enabled)
[Scheduler] Skipping user <user-id> - next run at 2026-09-16T09:00:00.000Z
[Scheduler] Processing 1 valid configurations (out of 1 with scheduling enabled)
[Scheduler] Skipping user <user-id> - next run at 2026-09-16T09:00:00.000Z
[Scheduler] Processing 1 valid configurations (out of 1 with scheduling enabled)
[Scheduler] Skipping user <user-id> - next run at 2026-09-16T09:00:00.000Z
[Scheduler] Processing 1 valid configurations (out of 1 with scheduling enabled)

Wait until the configured scheduler nextRun time has passed.
Gitea-Mirror no longer starts a new synchronization run.
Logs show that the scheduler is already running.

[Scheduler] Scheduler is already running, skipping this cycle
[Scheduler] Scheduler is already running, skipping this cycle
[Scheduler] Scheduler is already running, skipping this cycle

Restarting the service/container allows synchronization to continue again.

Expected behavior

When GitHub's API rate limit is reached, Gitea-Mirror should recover automatically after the rate limit resets.

Ideally one of these approaches could be used:

abort the current scheduler run when the primary GitHub rate limit is reached, release the scheduler lock, and retry during a later scheduler cycle; or
allow the existing request to wait for the GitHub reset, but guarantee that the scheduler run completes and releases its running state afterward.

A GitHub rate-limit event should not require restarting the Gitea-Mirror service.

Actual behavior

The scheduled synchronization becomes stuck after a GitHub rate-limit wait.

The process/container itself remains healthy, but the scheduler appears to remain in a running state indefinitely and refuses to start another synchronization cycle.

Restarting the service resets the condition and allows another batch of repositories to synchronize.

Additional observation

The current scheduler has a global isSchedulerRunning state, while GitHub rate-limit handling can wait for a long period inside the Octokit request.

It may be that the long-running rate-limit retry keeps the scheduler invocation alive and prevents later scheduler cycles from running.

This is particularly problematic during initial metadata synchronization of many repositories, where hitting GitHub's API rate limit is expected and may happen repeatedly.

### Description When Gitea-Mirror hits the GitHub API rate limit during a scheduled metadata sync, it waits until the GitHub rate limit resets: ```text [GitHub] Waiting 3263s before retry... ``` During this wait, the scheduler continues to run its periodic checks, but every new cycle is skipped: ```text [Scheduler] Processing 1 valid configurations (out of 1 with scheduling enabled) [Scheduler] Skipping user <user-id> - next run at 2026-09-16T09:00:00.000Z ``` After the configured `nextRun` time has passed, the scheduler does not resume normally. Instead, subsequent scheduler attempts eventually report that the scheduler is already running, and no further repositories are synchronized. The only way I have found to resume synchronization is to restart the Gitea-Mirror service/container. I also need to mark failed repositories for retry again. ### Environment - Gitea-Mirror: v3.36.2 - Destination: Forgejo - Source: GitHub ### Steps to reproduce 1. Enable scheduled synchronization for a GitHub source. 2. Have enough repositories/metadata to exhaust the GitHub API rate limit. 3. Let Gitea-Mirror reach the GitHub API limit. 4. Observe: ```text [GitHub] Rate limit hit for GET https://api.github.com/user/repos?per_page=100&affiliation=owner%2Corganization_member. Retry 1/3 Publishing event to channel rate-limit for user <user-id> Event published successfully with ID <ID-uuid> [GitHub] Waiting 3263s before retry... [Scheduler] Processing 1 valid configurations (out of 1 with scheduling enabled) [Scheduler] Skipping user <user-id> - next run at 2026-09-16T09:00:00.000Z [Scheduler] Processing 1 valid configurations (out of 1 with scheduling enabled) [Scheduler] Skipping user <user-id> - next run at 2026-09-16T09:00:00.000Z [Scheduler] Processing 1 valid configurations (out of 1 with scheduling enabled) [Scheduler] Skipping user <user-id> - next run at 2026-09-16T09:00:00.000Z [Scheduler] Processing 1 valid configurations (out of 1 with scheduling enabled) ``` 5. Wait until the configured scheduler `nextRun` time has passed. 6. Gitea-Mirror no longer starts a new synchronization run. 7. Logs show that the scheduler is already running. ```text [Scheduler] Scheduler is already running, skipping this cycle [Scheduler] Scheduler is already running, skipping this cycle [Scheduler] Scheduler is already running, skipping this cycle ``` 8. Restarting the service/container allows synchronization to continue again. ### Expected behavior When GitHub's API rate limit is reached, Gitea-Mirror should recover automatically after the rate limit resets. Ideally one of these approaches could be used: - abort the current scheduler run when the primary GitHub rate limit is reached, release the scheduler lock, and retry during a later scheduler cycle; or - allow the existing request to wait for the GitHub reset, but guarantee that the scheduler run completes and releases its running state afterward. A GitHub rate-limit event should not require restarting the Gitea-Mirror service. ### Actual behavior The scheduled synchronization becomes stuck after a GitHub rate-limit wait. The process/container itself remains healthy, but the scheduler appears to remain in a running state indefinitely and refuses to start another synchronization cycle. Restarting the service resets the condition and allows another batch of repositories to synchronize. ### Additional observation The current scheduler has a global `isSchedulerRunning` state, while GitHub rate-limit handling can wait for a long period inside the Octokit request. It may be that the long-running rate-limit retry keeps the scheduler invocation alive and prevents later scheduler cycles from running. This is particularly problematic during initial metadata synchronization of many repositories, where hitting GitHub's API rate limit is expected and may happen repeatedly.

adam closed this issue

2026-10-05 01:45:50 +02:00

adam commented

2026-10-05 01:45:51 +02:00

Thanks for the detailed report, the observation at the end was exactly right.

One scheduler run holds a global lock for the whole sync, and the GitHub throttle handler answered every primary rate limit by sleeping for the full reset window inside the request, up to three times per request. Each repository whose calls hit the exhausted limit added up to an hour of sleeping inside the lock, so with enough repositories the run never ended in practice. The boot auto-start does the same work outside the lock, which is where your first Waiting 3263s came from; the 09:00 run then hit the same exhausted limit and held the lock from there.

Fixed in #443, released in v3.37.0:

A wait of up to two minutes is still slept off inside the request. Anything longer marks the source as paused until the reset, the request fails immediately, and the run stops at the next batch or repository and releases the lock.
The next scheduled run is moved to one minute after the reset, so the remaining repositories are picked up then rather than at the next cron slot.
A repository whose attempt failed only because of the rate limit is put back to the status it had before, so it is retried automatically instead of needing a manual retry.
The dashboard rate-limit event now carries the reset time and says the work resumes on its own.

For a first metadata sync of many repositories you will still see the limit hit and the run pause, but it now comes back by itself after each reset instead of needing a restart.

Thanks for the detailed report, the observation at the end was exactly right. One scheduler run holds a global lock for the whole sync, and the GitHub throttle handler answered every primary rate limit by sleeping for the full reset window inside the request, up to three times per request. Each repository whose calls hit the exhausted limit added up to an hour of sleeping inside the lock, so with enough repositories the run never ended in practice. The boot auto-start does the same work outside the lock, which is where your first `Waiting 3263s` came from; the 09:00 run then hit the same exhausted limit and held the lock from there. Fixed in #443, released in v3.37.0: - A wait of up to two minutes is still slept off inside the request. Anything longer marks the source as paused until the reset, the request fails immediately, and the run stops at the next batch or repository and releases the lock. - The next scheduled run is moved to one minute after the reset, so the remaining repositories are picked up then rather than at the next cron slot. - A repository whose attempt failed only because of the rate limit is put back to the status it had before, so it is retried automatically instead of needing a manual retry. - The dashboard rate-limit event now carries the reset time and says the work resumes on its own. For a first metadata sync of many repositories you will still see the limit hit and the run pause, but it now comes back by itself after each reset instead of needing a restart.

adam commented

2026-10-05 01:45:52 +02:00

@arunavo4 With v3.38.2, When rate limit is hit, I keeps hammering Github with seemly what is already scheduled to sync and exist in queue to process. If the rate limit is hit, ALL in queue and current queries needs to abort immediately and wait for the given GitHub timestamp has been expired before next try may occur.
Due this hammering my account has been blocked completely by GitHub now.

@arunavo4 With v3.38.2, When rate limit is hit, I keeps hammering Github with seemly what is already scheduled to sync and exist in queue to process. If the rate limit is hit, ALL in queue and current queries needs to abort immediately and wait for the given GitHub timestamp has been expired before next try may occur. Due this hammering my account has been blocked completely by GitHub now.

adam commented

2026-10-05 01:45:52 +02:00

@hunter-nl You are right, and I am sorry about the account. Reopening this.

The v3.37.0 change stopped the scheduler run from holding its lock through the wait, but it only affected the one request that was refused. Nothing sat in front of the GitHub client, so every other call already queued in the run still went out and came back 403, and the shared retry helper sent each of them three more times with backoff. A repository with a thousand issues turned into thousands of refused requests in a few minutes. GitHub counts requests made while limited toward abuse detection, which is what suspended your account.

The fix is in progress and covers every path, not only the scheduler:

every GitHub client holds its requests while the source is paused: a reset within two minutes is waited out inside the request, a longer one fails the request at once without contacting GitHub, and only the rate limit probe endpoint is exempt
the retry helper no longer retries a rate limit refusal and stops starting the remaining items
the issues, pull request, release, label and milestone passes fail the repository at the first refusal instead of moving on to the next component; the scheduler already puts such a repository back to its previous status and retries it after the reset
a secondary rate limit ("slow down") pauses the whole client for its retry-after window

On the block itself: that is on GitHub's side. A new token will not help while the suspension lasts; GitHub Support lifts those, and pointing them at this issue and the fix release should help. I will post here when the release is out.

@hunter-nl You are right, and I am sorry about the account. Reopening this. The v3.37.0 change stopped the scheduler run from holding its lock through the wait, but it only affected the one request that was refused. Nothing sat in front of the GitHub client, so every other call already queued in the run still went out and came back 403, and the shared retry helper sent each of them three more times with backoff. A repository with a thousand issues turned into thousands of refused requests in a few minutes. GitHub counts requests made while limited toward abuse detection, which is what suspended your account. The fix is in progress and covers every path, not only the scheduler: - every GitHub client holds its requests while the source is paused: a reset within two minutes is waited out inside the request, a longer one fails the request at once without contacting GitHub, and only the rate limit probe endpoint is exempt - the retry helper no longer retries a rate limit refusal and stops starting the remaining items - the issues, pull request, release, label and milestone passes fail the repository at the first refusal instead of moving on to the next component; the scheduler already puts such a repository back to its previous status and retries it after the reset - a secondary rate limit ("slow down") pauses the whole client for its retry-after window On the block itself: that is on GitHub's side. A new token will not help while the suspension lasts; GitHub Support lifts those, and pointing them at this issue and the fix release should help. I will post here when the release is out.

adam commented

2026-10-05 01:45:53 +02:00

The fix is out in v3.40.1: https://github.com/RayLabsHQ/gitea-mirror/releases/tag/v3.40.1

While GitHub is refusing requests, the app no longer sends anything to it apart from the rate limit check. Every GitHub request now goes through the pause. A limit that resets within two minutes is waited out. For a longer one, the request fails before it is sent. The secondary limit also triggers the pause. Rate limit refusals are no longer retried, and the remaining work in the batch stops. Issue, PR, release, label and milestone sync no longer skip past a refusal and move on to the next request. Scheduled runs pick up again after the reset, as before.

Sorry about the suspension. Only GitHub Support can lift it. If anything comes up once your account is back, please reopen this issue.

The fix is out in v3.40.1: https://github.com/RayLabsHQ/gitea-mirror/releases/tag/v3.40.1 While GitHub is refusing requests, the app no longer sends anything to it apart from the rate limit check. Every GitHub request now goes through the pause. A limit that resets within two minutes is waited out. For a longer one, the request fails before it is sent. The secondary limit also triggers the pause. Rate limit refusals are no longer retried, and the remaining work in the batch stops. Issue, PR, release, label and milestone sync no longer skip past a refusal and move on to the next request. Scheduled runs pick up again after the reset, as before. Sorry about the suspension. Only GitHub Support can lift it. If anything comes up once your account is back, please reopen this issue.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: adam/gitea-mirror#174