Incremental GitHub metadata sync request #178

New Issue

Closed

opened 2026-10-05 01:46:00 +02:00 by adam · 5 comments

adam commented 2026-10-05 01:46:00 +02:00

Owner

Copy link

Copy source

Could Gitea-Mirror use GitHub’s repository-wide incremental endpoints for scheduled metadata syncs instead of checking comments on every historical issue/PR?

GET /repos/{owner}/{repo}/issues?state=all&since=<last_sync>&sort=updated
GET /repos/{owner}/{repo}/issues/comments?since=<last_sync>&sort=updated
GET /repos/{owner}/{repo}/pulls/comments?since=<last_sync>&sort=updated

These endpoints allow only changed/added issues, PRs, and comments since the previous successful sync to be fetched.

For large repositories, the current per-issue comment scanning causes thousands of API calls and can exhaust GitHub’s 5,000 requests/hour limit before the sync completes.

An incremental approach should reduce API usage dramatically and make recurring metadata syncs practical.

Could Gitea-Mirror use GitHub’s repository-wide incremental endpoints for scheduled metadata syncs instead of checking comments on every historical issue/PR? ```http GET /repos/{owner}/{repo}/issues?state=all&since=<last_sync>&sort=updated GET /repos/{owner}/{repo}/issues/comments?since=<last_sync>&sort=updated GET /repos/{owner}/{repo}/pulls/comments?since=<last_sync>&sort=updated ``` These endpoints allow only changed/added issues, PRs, and comments since the previous successful sync to be fetched. For large repositories, the current per-issue comment scanning causes thousands of API calls and can exhaust GitHub’s 5,000 requests/hour limit before the sync completes. An incremental approach should reduce API usage dramatically and make recurring metadata syncs practical. <!-- github-sync-id:5554868325 -->

adam closed this issue 2026-10-05 01:46:00 +02:00

adam commented 2026-10-05 01:46:01 +02:00

Author

Owner

Copy link

Copy source

Thanks, this is accurate and worth doing. Today every metadata sync lists all issues and all pull requests and then makes one comment call per issue and four calls per pull request, so a large repository does go past the hourly limit. The client already sends ETags, and a 304 does not count against the limit, but that cache lives in memory, is capped, and every 304 is still a round trip, so it only softens the problem.

The plan: keep the first sync full, then on later syncs list issues with since set to the last completed pass (with a safety margin) and only reconcile the issues and pull requests that changed, which is also the only set whose comments and details get fetched. A periodic full pass and the existing reset metadata action stay as the fallback for anything since cannot see. Work is in progress; it will land in the next minor release.

Thanks, this is accurate and worth doing. Today every metadata sync lists all issues and all pull requests and then makes one comment call per issue and four calls per pull request, so a large repository does go past the hourly limit. The client already sends ETags, and a 304 does not count against the limit, but that cache lives in memory, is capped, and every 304 is still a round trip, so it only softens the problem. The plan: keep the first sync full, then on later syncs list issues with `since` set to the last completed pass (with a safety margin) and only reconcile the issues and pull requests that changed, which is also the only set whose comments and details get fetched. A periodic full pass and the existing reset metadata action stay as the fallback for anything `since` cannot see. Work is in progress; it will land in the next minor release. <!-- github-comment-id:5799462846 -->

adam commented 2026-10-05 01:46:02 +02:00

Author

Owner

Copy link

Copy source

Done in #450 and released in v3.39.0.

After the first complete pass, the issues and pull request passes now ask GitHub only for items updated since the previous run (minus a 10 minute margin), and fetch comments, PR details, commits and files only for those. Pull requests go through the issues endpoint for this since pulls.list has no since filter.

A full pass still runs on the first sync after upgrading, after Reset metadata, when the last full pass is 7 or more days old, and when the destination has no mirrored issues or PRs. The watermark only advances when a pass finishes with no failed item, so nothing gets skipped because of a transient error.

No action needed on existing installs. If you see the request count not dropping after the first sync, the [Issues] and [Pull Requests] log lines say which mode each pass ran in and why.

Done in #450 and released in v3.39.0. After the first complete pass, the issues and pull request passes now ask GitHub only for items updated since the previous run (minus a 10 minute margin), and fetch comments, PR details, commits and files only for those. Pull requests go through the issues endpoint for this since `pulls.list` has no `since` filter. A full pass still runs on the first sync after upgrading, after Reset metadata, when the last full pass is 7 or more days old, and when the destination has no mirrored issues or PRs. The watermark only advances when a pass finishes with no failed item, so nothing gets skipped because of a transient error. No action needed on existing installs. If you see the request count not dropping after the first sync, the `[Issues]` and `[Pull Requests]` log lines say which mode each pass ran in and why. <!-- github-comment-id:5799798875 -->

adam commented 2026-10-05 01:46:03 +02:00

Author

Owner

Copy link

Copy source

A full pass still runs on the first sync after upgrading, after Reset metadata, when the last full pass is 7 or more days old, and when the destination has no mirrored issues or PRs.
Does this mean that in all these cases, always a full pass will be done? If yes, then this is still an issue for me. With many large repos, it will never get to the state of a full pass for all repos within that 7 days, and then it will already start again for a new full pass.
And with upgrading, you mean upgrade of gitea-mirror?

And I get these errors:

[Batch] Stopping before item 165 of 245: source is rate limited
[Sync] Error while syncing repository <REDACTED>: 126 | 			}
127 | 		}
128 | 		throw lastError || /* @__PURE__ */ new Error("Unknown error occurred");
129 | 	};
130 | 	const results = await processInParallel(items, processWithRetryLogic, concurrencyLimit, onProgress, () => rateLimitError !== null);
131 | 	if (rateLimitError) throw new RateLimitedBatchError(rateLimitError, results, items.length - results.length);
                                 ^
RateLimitedBatchError: Rate limit hit while processing items; stopped with 84 item(s) left (API rate limit exceeded for user ID <REDACTED>. If you reach out to GitHub Support for help, please include the request ID <REDACTED> and timestamp 2026-09-24 15:06:32 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting)
  status: 403,
 results: [
  undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined,
  undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined,
  undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined,
  undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined,
  undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined,
  undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined,
  undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined,
  undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined,
  undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined,
  undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined,
  undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined,
  undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined,
  undefined, undefined, undefined, undefined,
  ... 61 more items
],

      at processWithRetry (/app/dist/server/chunks/concurrency_CO4qU4tK.mjs:131:28)

119 |       request: requestOptions
120 |     });
121 |   }
122 |   if (status >= 400) {
123 |     octokitResponse.data = await getResponseData(fetchResponse);
124 |     throw new RequestError(toErrorMessage(octokitResponse.data), status, {
                    ^
HttpError: API rate limit exceeded for user ID <REDACTED>. If you reach out to GitHub Support for help, please include the request ID <REDACTED> and timestamp 2026-09-24 15:06:32 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting
   status: 403,
  request: {
  method: "GET",
  url: "https://api.github.com/repos/<REDACTED>/pulls/218",
  headers: [Object ...],
  request: [Object ...],
},
 response: {
  url: "https://api.github.com/repos/<REDACTED>/pulls/218",
  status: 403,
  headers: [Object ...],
  data: [Object ...],
},

      at fetchWrapper (/app/node_modules/@octokit/request/dist-bundle/index.js:124:15)
      at processTicksAndRejections (unknown:7:39)

`A full pass still runs on the first sync after upgrading, after Reset metadata, when the last full pass is 7 or more days old, and when the destination has no mirrored issues or PRs.` Does this mean that in all these cases, always a full pass will be done? If yes, then this is still an issue for me. With many large repos, it will never get to the state of a full pass for all repos within that 7 days, and then it will already start again for a new full pass. And with `upgrading`, you mean upgrade of gitea-mirror? And I get these errors: ``` [Batch] Stopping before item 165 of 245: source is rate limited [Sync] Error while syncing repository <REDACTED>: 126 | } 127 | } 128 | throw lastError || /* @__PURE__ */ new Error("Unknown error occurred"); 129 | }; 130 | const results = await processInParallel(items, processWithRetryLogic, concurrencyLimit, onProgress, () => rateLimitError !== null); 131 | if (rateLimitError) throw new RateLimitedBatchError(rateLimitError, results, items.length - results.length); ^ RateLimitedBatchError: Rate limit hit while processing items; stopped with 84 item(s) left (API rate limit exceeded for user ID <REDACTED>. If you reach out to GitHub Support for help, please include the request ID <REDACTED> and timestamp 2026-09-24 15:06:32 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting) status: 403, results: [ undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, ... 61 more items ], at processWithRetry (/app/dist/server/chunks/concurrency_CO4qU4tK.mjs:131:28) 119 | request: requestOptions 120 | }); 121 | } 122 | if (status >= 400) { 123 | octokitResponse.data = await getResponseData(fetchResponse); 124 | throw new RequestError(toErrorMessage(octokitResponse.data), status, { ^ HttpError: API rate limit exceeded for user ID <REDACTED>. If you reach out to GitHub Support for help, please include the request ID <REDACTED> and timestamp 2026-09-24 15:06:32 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting status: 403, request: { method: "GET", url: "https://api.github.com/repos/<REDACTED>/pulls/218", headers: [Object ...], request: [Object ...], }, response: { url: "https://api.github.com/repos/<REDACTED>/pulls/218", status: 403, headers: [Object ...], data: [Object ...], }, at fetchWrapper (/app/node_modules/@octokit/request/dist-bundle/index.js:124:15) at processTicksAndRejections (unknown:7:39) ``` <!-- github-comment-id:5816986635 -->

adam commented 2026-10-05 01:46:03 +02:00

Author

Owner

Copy link

Copy source

Thanks, both points are fair.

Yes, "upgrading" means upgrading gitea-mirror. A repository synced by an older version has no watermark yet, so its first sync on 3.39.0 or later is a full pass.

You are right that in 3.39.0 a full pass was expensive every time. It fetched comments for every issue, and details, commits and files for every pull request. A pass stopped by the rate limit also started over on the next run, so on large repositories it might never finish. #453 changes both:

• A pass stopped by the rate limit saves the items it finished. The next run continues from there and only redoes an item if it changed in the meantime.
• The weekly full pass still lists everything, but it only fetches details for items that are missing in Gitea or changed since the last completed pass. For a large repository that is about one request per hundred items instead of several per item.

The log you posted is the new rate limit handling from 3.40.1 doing its job. It stopped at item 165 instead of sending the remaining 84 requests. The error dump was too noisy though. It is now a single line, and the activity entry says the sync was paused instead of failed.

This will be in the next release, and I will post here when it is out.

Thanks, both points are fair. Yes, "upgrading" means upgrading gitea-mirror. A repository synced by an older version has no watermark yet, so its first sync on 3.39.0 or later is a full pass. You are right that in 3.39.0 a full pass was expensive every time. It fetched comments for every issue, and details, commits and files for every pull request. A pass stopped by the rate limit also started over on the next run, so on large repositories it might never finish. #453 changes both: - A pass stopped by the rate limit saves the items it finished. The next run continues from there and only redoes an item if it changed in the meantime. - The weekly full pass still lists everything, but it only fetches details for items that are missing in Gitea or changed since the last completed pass. For a large repository that is about one request per hundred items instead of several per item. The log you posted is the new rate limit handling from 3.40.1 doing its job. It stopped at item 165 instead of sending the remaining 84 requests. The error dump was too noisy though. It is now a single line, and the activity entry says the sync was paused instead of failed. This will be in the next release, and I will post here when it is out. <!-- github-comment-id:5825738226 -->

adam commented 2026-10-05 01:46:04 +02:00

Author

Owner

Copy link

Copy source

@hunter-nl this is out in v3.40.2: https://github.com/RayLabsHQ/gitea-mirror/releases/tag/v3.40.2

After upgrading, a pass that stops at the rate limit continues where it left off on the next run. The weekly full pass also only fetches details for items that are missing or changed. The log should show lines like Resuming an unfinished full pass and unchanged since the last pass. If a pass still never completes on your larger repositories, please post the [Issues] and [Pull Requests] lines from a couple of runs here.

@hunter-nl this is out in v3.40.2: https://github.com/RayLabsHQ/gitea-mirror/releases/tag/v3.40.2 After upgrading, a pass that stops at the rate limit continues where it left off on the next run. The weekly full pass also only fetches details for items that are missing or changed. The log should show lines like `Resuming an unfinished full pass` and `unchanged since the last pass`. If a pass still never completes on your larger repositories, please post the `[Issues]` and `[Pull Requests]` lines from a couple of runs here. <!-- github-comment-id:5825813858 -->

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

fix/archived-sync-label

chore/zod-dependabot

feat/449-incremental-metadata-sync

fix/node-adapter-pipeline

fix/release-asset-duplicates

v3.50.0

v3.49.0

v3.48.0

v3.47.0

v3.46.0

v3.45.0

v3.44.0

v3.43.0

v3.42.0

v3.41.0

v3.40.5

v3.40.4

v3.40.3

v3.40.2

v3.40.1

v3.40.0

v3.39.0

v3.38.2

v3.38.1

v3.38.0

v3.37.0

v3.36.2

v3.36.1

v3.36.0

v3.35.1

v3.35.0

v3.34.0

v3.33.0

v3.32.1

v3.32.0

v3.31.0

v3.30.0

v3.29.1

v3.29.0

v3.28.2

v3.28.1

v3.28.0

v3.27.2

v3.27.1

v3.27.0

v3.26.2

v3.26.1

v3.26.0

v3.25.0

v3.24.0

v3.23.0

v3.22.0

v3.21.0

v3.20.4

v3.20.3

v3.20.2

v3.20.1

v3.20.0

v3.19.1

v3.19.0

v3.18.0

v3.17.1

v3.17.0

v3.16.3

v3.16.2

v3.16.1

v3.16.0

v3.15.12

v3.15.11

v3.15.10

v3.15.9

v3.15.8

v3.15.7

v3.15.6

v3.15.5

v3.15.4

v3.15.3

v3.15.2

v3.15.1

v3.15.0

v3.14.2

v3.14.1

v3.14.0

v3.13.4

v3.13.3

v3.13.2

v3.13.1

v3.13.0

v3.12.5

v3.12.4

v3.12.3

v3.12.2

v3.12.1

v3.12.0

v3.11.0

v3.10.1

v3.10.0

v3.9.6

v3.9.5

v3.9.4

v3.9.3

v3.9.2

v3.9.0

v3.8.11

v3.8.10

v3.8.9

v3.8.8

v3.8.7

v3.8.6

v3.8.5

v3.8.4

v3.8.3

v3.8.2

v3.8.1

v3.8.0

v3-sso

v3.7.2

v3.7.1

v3.7.0

v3.6.0

v3.5.4

v3.5.3

v3.5.2

v3.5.1

v3.5.0

v3.4.0

v3.3.0

v3.2.6

v3.2.5

v3.2.4

v3.2.3

v3.2.2

v3.2.1

v3.2.0

v3.1.1

v3.1.0

v3.0.1

v3.0.0

v2.22.0

v2.21.0

v2.20.1

v2.20.0

v2.18.0

v2.17.0

v2.16.3

v2.16.2

v2.16.1

v2.16.0

v2.15.0

v2.14.0

v2.13.2

v2.13.1

v2.13.0

v2.12.0

v2.11.2

v2.11.1

v2.11.0

v2.10.0

v2.9.3

v2.9.2

v2.9.1

v2.9.0

v2.8.0

v2.7.0

v2.6.0

v2.5.4

v2.5.3

v2.5.2

v2.5.1

v2.5.0

v2.1.0

v2.0.0

v1.0.0

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

adam (Adam Wojcieszonek)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: adam/gitea-mirror#178