PR mirroring re-fetches all pull requests each sync without conditional requests (ETag) #145

New Issue

2026-10-05T01:44:32+02:00

adam commented

2026-10-05 01:44:32 +02:00

Summary

On each scheduled sync, mirrorGitHubPullRequestsToGitea re-lists a repository's pull requests with state: "all" and then fetches per-PR details, with no If-None-Match/ETag conditional requests. Unchanged pull request lists are re-downloaded in full every run.

Where

The Octokit client is created with the throttling plugin but no response cache / conditional-request layer: src/lib/github.ts (createGitHubClient).
The full PR list is paginated every sync: src/lib/gitea.ts — octokit.paginate(octokit.rest.pulls.list, { state: "all", per_page: 100, ... }), followed by pulls.get / pulls.listCommits / pulls.listFiles per PR.
The scheduler runs on an interval (default 3600s) — src/lib/env-config-loader.ts.

Impact on users

GitHub's REST API supports conditional requests: sending the previously returned ETag as If-None-Match yields a 304 Not Modified when nothing changed, and 304 responses do not count against the token's primary rate limit. Because the mirror doesn't use them, every sync spends full rate-limit budget re-downloading unchanged PR data. For users with many mirrored repositories or short sync intervals, this makes it easier to hit the token's hourly rate limit (leading to throttling and slower/failed syncs) and uses more bandwidth than necessary.

Suggested change

Store the ETag returned for each (repo, endpoint) and send it as If-None-Match on the next sync, reusing the cached result on 304. Octokit ships no cache by default. This can be done without new heavyweight dependencies — e.g. persist ETags in the existing SQLite database and set/read If-None-Match in an octokit.hook around the current client, keeping in line with the project's "minimal dependencies / SQLite" guidelines. Optionally, narrowing the list to state: "open" (or an incremental since query) where appropriate would further reduce redundant work.

This keeps sync results identical while lowering rate-limit pressure and bandwidth for self-hosted users.

### Summary On each scheduled sync, `mirrorGitHubPullRequestsToGitea` re-lists a repository's pull requests with `state: "all"` and then fetches per-PR details, with no `If-None-Match`/ETag conditional requests. Unchanged pull request lists are re-downloaded in full every run. ### Where - The Octokit client is created with the throttling plugin but no response cache / conditional-request layer: [`src/lib/github.ts`](https://github.com/RayLabsHQ/gitea-mirror/blob/main/src/lib/github.ts) (`createGitHubClient`). - The full PR list is paginated every sync: [`src/lib/gitea.ts`](https://github.com/RayLabsHQ/gitea-mirror/blob/main/src/lib/gitea.ts) — `octokit.paginate(octokit.rest.pulls.list, { state: "all", per_page: 100, ... })`, followed by `pulls.get` / `pulls.listCommits` / `pulls.listFiles` per PR. - The scheduler runs on an interval (default `3600`s) — [`src/lib/env-config-loader.ts`](https://github.com/RayLabsHQ/gitea-mirror/blob/main/src/lib/env-config-loader.ts). ### Impact on users GitHub's REST API supports conditional requests: sending the previously returned `ETag` as `If-None-Match` yields a `304 Not Modified` when nothing changed, and **`304` responses do not count against the token's primary rate limit**. Because the mirror doesn't use them, every sync spends full rate-limit budget re-downloading unchanged PR data. For users with many mirrored repositories or short sync intervals, this makes it easier to hit the token's hourly rate limit (leading to throttling and slower/failed syncs) and uses more bandwidth than necessary. ### Suggested change Store the `ETag` returned for each `(repo, endpoint)` and send it as `If-None-Match` on the next sync, reusing the cached result on `304`. Octokit ships no cache by default. This can be done without new heavyweight dependencies — e.g. persist ETags in the existing SQLite database and set/read `If-None-Match` in an `octokit.hook` around the current client, keeping in line with the project's "minimal dependencies / SQLite" guidelines. Optionally, narrowing the list to `state: "open"` (or an incremental `since` query) where appropriate would further reduce redundant work. This keeps sync results identical while lowering rate-limit pressure and bandwidth for self-hosted users.

adam closed this issue

2026-10-05 01:44:32 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: adam/gitea-mirror#145