feat: incremental issue and pull request sync (#449) (#450)

After a complete pass, the issues and pull request passes list only the
items GitHub reports as updated since the previous listing (minus a 10
minute margin), so per-issue comment listing and per-PR detail, commits
and files calls run only for what changed. Pull requests use the issues
endpoint for this since pulls.list has no since filter.

The watermark lives in the repository metadata state as syncCursors, one
per kind, and records when the listing started. It only moves when the
pass finishes with no failed item. A full pass still runs when the last
one is older than 7 days, when the destination has no mirrored issues or
pull requests, and after reset metadata, which clears the state.
This commit is contained in:
ARUNAVO RAY authored and GitHub committed 2026-09-23 23:07:23 +05:30
1 parent 8ae778d708
commit 133268e63a
10 files changed
+1003 -45

No files matched your search

+6
View File
@@ -152,6 +152,12 @@ Throughout, job status updates are published as server-sent events so the dashbo
Gitea's API cannot create a pull request through the migration path, so mirrored pull requests are created as issues instead. Each one is tagged with a `pull-request` label, its title is prefixed with the original number and status, and its body includes the commit history, changed files, and merge outcome. This is a deliberate workaround for an API limitation, not a temporary gap.
### Incremental issue and pull request sync
The first issues pass and the first pull request pass list everything on GitHub. After a pass completes, the repository's metadata state records when its listing started, and later passes ask GitHub only for items updated since then, minus a 10 minute safety margin. New comments, label, state, milestone, title and body changes all move an item's `updated_at`, so only those items are reconciled and only their comments, pull request details, commits and files are fetched. Pull requests are listed through the issues endpoint for this, because the pull request list has no `since` filter.
A full pass still runs when the last one is more than 7 days old, when the destination has no mirrored issues or pull requests (for example after it was recreated), and after Reset metadata. If any item fails, the watermark is not moved, so the next pass lists it again. Items deleted on GitHub are left on the destination, as before.
### Ordering and concurrency
Gitea assigns issue numbers on creation, so processing issues in parallel can produce numbering that does not match GitHub's. The concurrency settings default to 3 for issues and 5 for pull requests, trading some ordering fidelity for speed. Set both to 1 if you need the numbering to line up.
@@ -33,6 +33,8 @@ Gitea Mirror uses this on every GET. Each response's ETag is cached; every subse
The practical effect is that sync frequency mostly stops mattering. A `30m` interval on a stable account costs barely more than a `24h` one, because 47 of the 48 daily sweeps are nearly all cache hits. The requests you spend scale with how much changed, not how often you look.
Issues and pull requests go one step further. After the first complete pass, a scheduled sync asks GitHub only for the issues and pull requests updated since the previous one, and fetches comments, commits and changed files for those alone. A tracker with thousands of issues and a handful of changes costs a handful of requests. A full pass still runs once a week to catch anything the incremental listing cannot see.
## Configuration that stays inside the budget
A few settings still matter for large or busy accounts: