Push engine: GitHub and GitLab as mirror destinations #152

New Issue

2026-10-05T01:45:00+02:00

adam commented

2026-10-05 01:45:00 +02:00

Why

#26 asks for GitLab as a destination and #34 for GitHub. #131 and the feature/matrix branch (#368) want any source to any target. Today every mirror is a Gitea pull mirror: the app asks Gitea to fetch from the source and Gitea keeps it in sync. GitHub has no pull mirror API at all and GitLab only has one on paid tiers, so any destination other than Gitea/Forgejo needs a push engine on our side. This issue scopes that engine. Sources are covered by #375 and PR #376.

What the engine does

For each repository with a GitHub or GitLab destination:

Keep a bare clone on disk under data/mirrors/<source host>/<owner>/<repo>.git. The first run clones, later runs git fetch --prune from the source with the source token. This reuses the git process handling in src/lib/repo-backup.ts.
Make sure the target repository exists. GitHub: POST /user/repos or POST /orgs/{org}/repos. GitLab: POST /projects with the group as namespace_id. Visibility and the organization strategy apply the same way they do for Gitea today.
git push --mirror to the target with the target token. Credentials go through an askpass helper or GIT_CONFIG env, never into a URL that lands on disk.
Record the result like the Gitea path does: same statuses, same mirror jobs, same activity rows, mirroredLocation holds owner/repo on the target.

Data model

A destination config next to the source config: { provider, url, username, token, organization }. The current giteaConfig becomes the Gitea flavour of it. PR #376 already added provider (Gitea or Forgejo) to it.
repositories gets destination_provider and destination_url, the mirror image of the source_* columns from #376, so cleanup, the guard at the mirror sites and the list views all know where a row lives.
No change to mirror jobs or activities.

Scheduling, concurrency, recovery

The scheduler's sync phase calls the push engine for repositories whose destination is GitHub or GitLab, and syncGiteaRepo for the rest. Auto import and auto mirror are unchanged.
One git process per repository at a time, a lock file per clone directory, and a small global concurrency limit (git is heavy, unlike API calls).
Crash recovery: fetch and push are both idempotent, so an interrupted run just reruns. A half written clone directory is detected and recloned.
Cleanup: when the source deletes a repository, archive or delete on the target through its API, same options as today.

Force pushes, deletions, backups

git push --mirror replicates force pushes and deleted branches to the target, which is exactly what the Destructive Update Protection strategies exist to catch. The engine compares refs between the last push and the fresh fetch before pushing, so the Smart and Block strategies keep working. The bare clone is itself the snapshot, which makes the Always Backup strategy cheap here.

Milestones

Manual push to GitHub. Bare clone, target creation, push --mirror, statuses and activity, triggered from the Mirror button. Force push detection off.
Scheduled. Scheduler integration, concurrency limit, lock files, recovery, cleanup on the target.
GitLab target. Project creation in groups, token handling, same tests.
Extras. LFS (git lfs fetch --all then git lfs push --all), wikis (GitLab .wiki.git can be pushed, GitHub wikis must be created by hand first), the backup strategies through ref comparison.

Not in scope

Bidirectional mirroring. Not planned, ever.
Several destinations per repository. That is the matrix follow up once one target per repository works.
Issues, pull requests, releases and other metadata on push targets.
Gitea or Forgejo as a push target. The pull mirror stays the transport for them.

Open questions

Disk: bare clones double the storage of the Gitea install. Where they live, and when they are removed once a repository is deleted, needs a decision before milestone 1.
Token scopes on the target. GitHub needs repo, and workflow as soon as a repository contains workflow files, or the push is rejected. GitLab needs api and write_repository.
Large first pushes. GitHub rejects very large pushes; the first push may have to go in ref batches instead of one --mirror.
Whether an install can run without any Gitea at all, or whether Gitea stays the default destination and push targets are additional.

Refs #26 #34 #131 #368 #375

## Why #26 asks for GitLab as a destination and #34 for GitHub. #131 and the `feature/matrix` branch (#368) want any source to any target. Today every mirror is a Gitea pull mirror: the app asks Gitea to fetch from the source and Gitea keeps it in sync. GitHub has no pull mirror API at all and GitLab only has one on paid tiers, so any destination other than Gitea/Forgejo needs a push engine on our side. This issue scopes that engine. Sources are covered by #375 and PR #376. ## What the engine does For each repository with a GitHub or GitLab destination: 1. Keep a bare clone on disk under `data/mirrors/<source host>/<owner>/<repo>.git`. The first run clones, later runs `git fetch --prune` from the source with the source token. This reuses the git process handling in `src/lib/repo-backup.ts`. 2. Make sure the target repository exists. GitHub: `POST /user/repos` or `POST /orgs/{org}/repos`. GitLab: `POST /projects` with the group as `namespace_id`. Visibility and the organization strategy apply the same way they do for Gitea today. 3. `git push --mirror` to the target with the target token. Credentials go through an askpass helper or `GIT_CONFIG` env, never into a URL that lands on disk. 4. Record the result like the Gitea path does: same statuses, same mirror jobs, same activity rows, `mirroredLocation` holds `owner/repo` on the target. ## Data model - A destination config next to the source config: `{ provider, url, username, token, organization }`. The current `giteaConfig` becomes the Gitea flavour of it. PR #376 already added `provider` (Gitea or Forgejo) to it. - `repositories` gets `destination_provider` and `destination_url`, the mirror image of the `source_*` columns from #376, so cleanup, the guard at the mirror sites and the list views all know where a row lives. - No change to mirror jobs or activities. ## Scheduling, concurrency, recovery - The scheduler's sync phase calls the push engine for repositories whose destination is GitHub or GitLab, and `syncGiteaRepo` for the rest. Auto import and auto mirror are unchanged. - One git process per repository at a time, a lock file per clone directory, and a small global concurrency limit (git is heavy, unlike API calls). - Crash recovery: fetch and push are both idempotent, so an interrupted run just reruns. A half written clone directory is detected and recloned. - Cleanup: when the source deletes a repository, archive or delete on the target through its API, same options as today. ## Force pushes, deletions, backups `git push --mirror` replicates force pushes and deleted branches to the target, which is exactly what the Destructive Update Protection strategies exist to catch. The engine compares refs between the last push and the fresh fetch before pushing, so the Smart and Block strategies keep working. The bare clone is itself the snapshot, which makes the Always Backup strategy cheap here. ## Milestones 1. **Manual push to GitHub.** Bare clone, target creation, `push --mirror`, statuses and activity, triggered from the Mirror button. Force push detection off. 2. **Scheduled.** Scheduler integration, concurrency limit, lock files, recovery, cleanup on the target. 3. **GitLab target.** Project creation in groups, token handling, same tests. 4. **Extras.** LFS (`git lfs fetch --all` then `git lfs push --all`), wikis (GitLab `.wiki.git` can be pushed, GitHub wikis must be created by hand first), the backup strategies through ref comparison. ## Not in scope - Bidirectional mirroring. Not planned, ever. - Several destinations per repository. That is the matrix follow up once one target per repository works. - Issues, pull requests, releases and other metadata on push targets. - Gitea or Forgejo as a push target. The pull mirror stays the transport for them. ## Open questions - Disk: bare clones double the storage of the Gitea install. Where they live, and when they are removed once a repository is deleted, needs a decision before milestone 1. - Token scopes on the target. GitHub needs `repo`, and `workflow` as soon as a repository contains workflow files, or the push is rejected. GitLab needs `api` and `write_repository`. - Large first pushes. GitHub rejects very large pushes; the first push may have to go in ref batches instead of one `--mirror`. - Whether an install can run without any Gitea at all, or whether Gitea stays the default destination and push targets are additional. Refs #26 #34 #131 #368 #375

adam closed this issue

2026-10-05 01:45:01 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: adam/gitea-mirror#152