diff --git a/docs/API.md b/docs/API.md index 6124da0..cdc1055 100644 --- a/docs/API.md +++ b/docs/API.md @@ -25,7 +25,7 @@ A missing, malformed, expired or revoked key gets `401 Unauthorized`: { "success": false, "error": "Unauthorized" } ``` -If the app is served under a base path (`BASE_PATH=/mirror`), prefix every URL with it: `/mirror/api/...`. +If the app is served under a path prefix (`BASE_URL=/mirror`), prefix every URL with it: `/mirror/api/...`. ## Endpoints diff --git a/docs/README.md b/docs/README.md index d705add..00138e5 100644 --- a/docs/README.md +++ b/docs/README.md @@ -17,12 +17,15 @@ This folder contains engineering and operations references for the open-source G ### Authentication - **[SSO-OIDC-SETUP.md](./SSO-OIDC-SETUP.md)** – Configure OIDC/SSO providers through the admin UI. - **[SSO_TESTING.md](./SSO_TESTING.md)** – Recipes for local and staging SSO testing (Google, Keycloak, mock providers). + +### API - **[API.md](./API.md)** – Create API keys and call the app from scripts, CI pipelines and workflow tools. -### Destinations +### Sources and destinations +- **[SOURCE_PROVIDERS.md](./SOURCE_PROVIDERS.md)** – GitHub, GitLab and Gitea/Forgejo as sources: tokens, what each can mirror, the source lock. - **[PUSH_TARGETS.md](./PUSH_TARGETS.md)** – GitHub and GitLab as destinations: how the push engine works, token scopes, disk usage, what is not mirrored. -If you are looking for customer-facing playbooks, see the MDX use cases under `www/src/pages/use-cases/`. +The user-facing documentation lives on the website at , with its sources under `www/src/pages/docs/`. If you are looking for playbooks, see the MDX use cases under `www/src/pages/use-cases/`. ## Quick start for local development diff --git a/docs/SOURCE_PROVIDERS.md b/docs/SOURCE_PROVIDERS.md index 549b8c1..5746715 100644 --- a/docs/SOURCE_PROVIDERS.md +++ b/docs/SOURCE_PROVIDERS.md @@ -66,4 +66,4 @@ See [ENVIRONMENT_VARIABLES.md](ENVIRONMENT_VARIABLES.md) for the full list. ## Not yet -Several sources at once (the tracking issue is #375, the follow up is a list of sources in the card), targets other than Gitea/Forgejo (see the `feature/matrix` branch), and hosts without a usable API. +Several sources at once (the tracking issue is #375, the follow up is a list of sources in the card), hosts without a usable API, and issues, merge requests and releases from GitLab, Gitea or Forgejo sources. GitHub and GitLab as destinations shipped in 3.31.0 through the push engine, see [PUSH_TARGETS.md](PUSH_TARGETS.md). diff --git a/www/public/assets/api-keys.png b/www/public/assets/api-keys.png new file mode 100644 index 0000000..662d46e Binary files /dev/null and b/www/public/assets/api-keys.png differ diff --git a/www/public/assets/configuration-2.png b/www/public/assets/configuration-2.png index c37d213..0e1f638 100644 Binary files a/www/public/assets/configuration-2.png and b/www/public/assets/configuration-2.png differ diff --git a/www/public/assets/configuration.png b/www/public/assets/configuration.png index a430f14..2d71e4a 100644 Binary files a/www/public/assets/configuration.png and b/www/public/assets/configuration.png differ diff --git a/www/public/assets/configuration_mobile.png b/www/public/assets/configuration_mobile.png index 06617cd..2191b90 100644 Binary files a/www/public/assets/configuration_mobile.png and b/www/public/assets/configuration_mobile.png differ diff --git a/www/public/assets/destinations.png b/www/public/assets/destinations.png new file mode 100644 index 0000000..a9d2904 Binary files /dev/null and b/www/public/assets/destinations.png differ diff --git a/www/src/lib/docs-nav.ts b/www/src/lib/docs-nav.ts index 64ec3b9..7505564 100644 --- a/www/src/lib/docs-nav.ts +++ b/www/src/lib/docs-nav.ts @@ -20,12 +20,15 @@ export const docsNav: DocsNavSection[] = [ title: 'Configuration', items: [ { title: 'Configuration Guide', href: '/docs/configuration/' }, + { title: 'Sources', href: '/docs/sources/' }, + { title: 'Destinations', href: '/docs/destinations/' }, { title: 'Environment Variables', href: '/docs/environment-variables/' }, ], }, { title: 'Features', items: [ + { title: 'Cleanup & Reconcile', href: '/docs/cleanup-and-reconcile/' }, { title: 'Notifications', href: '/docs/notifications/' }, { title: 'Authentication & SSO', href: '/docs/authentication/' }, { title: 'Force-Push Protection', href: '/docs/force-push-protection/' }, @@ -34,6 +37,7 @@ export const docsNav: DocsNavSection[] = [ { title: 'Reference', items: [ + { title: 'API & API Keys', href: '/docs/api/' }, { title: 'Architecture', href: '/docs/architecture/' }, { title: 'Advanced', href: '/docs/advanced/' }, { title: 'Custom CA Certificates', href: '/docs/ca-certificates/' }, diff --git a/www/src/pages/docs/advanced.mdx b/www/src/pages/docs/advanced.mdx index d42602b..2d7270c 100644 --- a/www/src/pages/docs/advanced.mdx +++ b/www/src/pages/docs/advanced.mdx @@ -22,8 +22,8 @@ A healthy instance responds with HTTP 200 and a body like this: { "status": "ok", "timestamp": "2026-08-03T12:00:00.000Z", - "version": "3.24.0", - "latestVersion": "3.24.0", + "version": "3.31.0", + "latestVersion": "3.31.0", "updateAvailable": false, "database": { "connected": true diff --git a/www/src/pages/docs/api.mdx b/www/src/pages/docs/api.mdx new file mode 100644 index 0000000..e931e3a --- /dev/null +++ b/www/src/pages/docs/api.mdx @@ -0,0 +1,210 @@ +--- +layout: '../../layouts/DocsLayout.astro' +title: API and API Keys +description: Create API keys and call the same JSON endpoints the web interface uses from scripts, CI pipelines and workflow tools. +--- + +# API and API Keys + +The web interface talks to a set of JSON endpoints under `/api`. With an API key you can call the same endpoints from scripts, CI pipelines and workflow tools such as n8n, without a browser session. + +## Create a key + +1. Sign in and open the Configuration page, then the **Authentication** tab. +2. In the **API Keys** card click **Create key**, give it a name that tells you where it is used, and pick an expiry: never, 30 days, 90 days or a year. +3. Copy the key. It is shown once; afterwards only its first characters are visible in the list. + +![The API Keys card with one key created](/assets/api-keys.png) + +Keys start with `gm_`, belong to the user who created them and carry the same rights as that user's session. Revoke a key from the same list at any time. Keys are stored hashed. They are not rate limited by the app, so a runaway script is stopped by revoking its key. + +## Send the key + +Put the key in the `x-api-key` header. Every endpoint that accepts a session cookie accepts the header instead. + +```bash +curl -sS "https://mirror.example.com/api/github/repositories" \ + -H "x-api-key: gm_..." +``` + +A missing, malformed, expired or revoked key gets `401 Unauthorized`: + +```json +{ "success": false, "error": "Unauthorized" } +``` + +If the app is served under a path prefix (`BASE_URL=/mirror`), prefix every URL with it: `/mirror/api/...`. + +## Endpoints + +The calls below cover the common automation cases. Response bodies show the fields you are most likely to use; the app returns more. + +### Health + +`GET /api/health` needs no key. It reports overall status, database connectivity and the recovery state, and is what the container health check polls. The full response is described in [Advanced topics](./advanced.md#health-endpoint). + +### List tracked repositories + +`GET /api/github/repositories` + +Despite the name it lists every repository the account tracks, whatever the configured source. Requires a saved configuration, otherwise `404`. + +```json +{ + "success": true, + "message": "Repositories fetched successfully", + "repositories": [ + { + "id": "9b2f...", + "name": "hello-world", + "fullName": "octocat/hello-world", + "owner": "octocat", + "organization": null, + "status": "mirrored", + "mirroredLocation": "github-mirrors/hello-world", + "lastMirrored": "2026-09-02T08:00:00.000Z", + "errorMessage": null, + "destinationOrg": null, + "sourceProvider": "github", + "isPrivate": false + } + ] +} +``` + +`status` is one of `imported`, `mirroring`, `mirrored`, `failed`, `syncing`, `synced`, `skipped`, `deleted`, `archived`. + +### Add a repository + +`POST /api/sync/repository` + +```json +{ "owner": "octocat", "repo": "hello-world" } +``` + +Optional fields: `destinationOrg` to override the destination organization for this repository, and `force: true` to refresh the metadata of a repository that is already tracked. + +The repository is looked up on the configured source and added with status `imported`. It is not mirrored until you start a mirror (next call) or the scheduler picks it up. + +| Status | Meaning | +| --- | --- | +| `200` | Added. `repository.id` is what the mirror call needs. | +| `400` | `owner` or `repo` missing. | +| `404` | No configuration for this account, or the repository does not exist on the source. | +| `409` | Already tracked. Send `force: true` to refresh it instead. | + +### Start a mirror + +`POST /api/job/mirror-repo` + +```json +{ "repositoryIds": ["9b2f..."] } +``` + +Starts mirroring in the background and returns straight away. Poll the list endpoint to watch `status` move from `mirroring` to `mirrored` or `failed`. + +| Status | Meaning | +| --- | --- | +| `200` | Job started. `repositories` lists what was queued. | +| `400` | `repositoryIds` missing or empty, or no source token saved. | +| `404` | None of the ids belong to this account. | + +`POST /api/job/sync-repo` takes the same body and re-syncs repositories that are already mirrored. + +### Organizations + +`POST /api/sync/organization` adds an organization or group the same way: + +```json +{ "org": "my-org", "role": "member" } +``` + +`role` is the account's role in that organization (`member`, `admin`, `owner` or `billing_manager`). `409` means it is already tracked; `force: true` refreshes it. Then `POST /api/job/mirror-org` with `{ "organizationIds": ["..."] }` mirrors its repositories. + +### Run cleanup + +`POST /api/cleanup/trigger` runs the [repository cleanup](./cleanup-and-reconcile.md) now, with the account's saved cleanup settings, and returns how many orphaned repositories were found and processed and any errors. `GET /api/cleanup/trigger` returns the cleanup configuration and the last and next run times. + +### Reconcile the destination + +`POST /api/cleanup/reconcile` + +Compares what the destination (Gitea or Forgejo) holds with the repositories this account tracks. See [Cleanup and reconcile](./cleanup-and-reconcile.md#reconcile) for what the groups mean. + +```json +{ "dryRun": true, "adoptUntracked": false, "resetMissing": false } +``` + +The body is optional and every field defaults to the value above, so an empty `POST` is a dry run. With `dryRun: false`: + +- `adoptUntracked: true` creates a row for each untracked mirror from its source URL, so scheduled sync and the cleanup service include it from then on. The row keeps the mirror where it is, even when the strategy would put it elsewhere. +- `resetMissing: true` sets each missing row back to `imported` so the next mirror run recreates the mirror. Rows are never deleted. + +Nothing is deleted or archived on either side by this call. + +```json +{ + "success": true, + "dryRun": false, + "report": { + "untracked": [{ "location": "github-mirrors/hello-world", "originalUrl": "https://github.com/octocat/hello-world.git", "sourcePath": "octocat/hello-world", "isPrivate": false }], + "missing": [{ "id": "9b2f...", "fullName": "octocat/gone", "location": "github-mirrors/gone" }], + "notManaged": [{ "location": "me/notes", "reason": "not a mirror" }], + "unverified": [], + "healthyCount": 41, + "elsewhereCount": 0, + "scannedOwners": ["admin", "github-mirrors"], + "skippedOwners": ["starred"], + "totalOnDestination": 43 + }, + "applied": { "adopted": 1, "reset": 1, "skipped": 0 } +} +``` + +`applied` is `null` on a dry run. `scannedOwners` are the destination users and organizations that were listed; `skippedOwners` are ones the configuration or the database point at but the destination does not have. A row lands in `unverified` when the presence check itself failed, and is left alone. `elsewhereCount` counts rows mirrored to a previous destination, which are left out of the comparison. + +| Status | Meaning | +| --- | --- | +| `200` | Report computed, and applied when asked. | +| `400` | A field has the wrong type, the destination is not configured yet, or the destination is a push target. | +| `404` | No configuration for this account. | + +## A complete example + +Add a repository and mirror it in one go: + +```bash +#!/usr/bin/env bash +set -euo pipefail + +BASE="https://mirror.example.com" +KEY="gm_..." + +added=$(curl -sS -X POST "$BASE/api/sync/repository" \ + -H "x-api-key: $KEY" \ + -H "Content-Type: application/json" \ + -d '{"owner":"octocat","repo":"hello-world"}') + +id=$(echo "$added" | jq -r '.repository.id') + +curl -sS -X POST "$BASE/api/job/mirror-repo" \ + -H "x-api-key: $KEY" \ + -H "Content-Type: application/json" \ + -d "{\"repositoryIds\":[\"$id\"]}" +``` + +## Managing keys from the API + +Manage keys through the settings card or with a browser session. The key endpoints need a session cookie, not a key, so a leaked key cannot mint more keys. Like every cookie-authenticated `POST` to `/api/auth/*` they also need an `Origin` header; browsers send it on their own, a script must add `Origin: https://mirror.example.com`. The endpoints live under `/api/auth/api-key/`: + +| Call | Body | Result | +| --- | --- | --- | +| `POST /api/auth/api-key/create` | `{ "name": "ci", "expiresIn": 2592000 }` (`expiresIn` in seconds, omit for no expiry) | The new key, including `key`, shown only here. | +| `GET /api/auth/api-key/list` | | `{ "apiKeys": [...], "total": n }`. Each entry has `id`, `name`, `start`, `createdAt`, `lastRequest` and `expiresAt`, never the secret. | +| `POST /api/auth/api-key/delete` | `{ "keyId": "..." }` | Revokes the key. | + +## Notes + +- A key can only act on the account that created it. Each account keeps its own source, destination and repositories. +- Keys are not rate limited by the app. Source hosts still apply their own limits, and the app tracks GitHub's. +- Deleting a user deletes their keys. diff --git a/www/src/pages/docs/architecture.mdx b/www/src/pages/docs/architecture.mdx index 0e5fad9..00e2113 100644 --- a/www/src/pages/docs/architecture.mdx +++ b/www/src/pages/docs/architecture.mdx @@ -6,7 +6,7 @@ description: How Gitea Mirror is put together, from the request path through the # Architecture -Gitea Mirror is a single self-hosted process. It serves a web interface, talks to the GitHub and Gitea APIs, keeps state in a local SQLite file, and runs background schedulers in the same process. There is no separate worker, queue server, or external database to operate. +Gitea Mirror is a single self-hosted process. It serves a web interface, talks to the source and destination APIs, keeps state in a local SQLite file, and runs background schedulers in the same process. There is no separate worker, queue server, or external database to operate. ## Stack @@ -14,7 +14,7 @@ The frontend is Astro 5 in server-side rendering mode, with React 19 islands for The backend is Astro API routes served through the `@astrojs/node` adapter in standalone mode. Everything runs on the Bun runtime, which requires version 1.2.9 or newer. -Persistence is SQLite accessed through Drizzle ORM, using Bun's native SQLite driver. Authentication is Better Auth with session cookies. GitHub access goes through Octokit with the throttling plugin, and Gitea access goes through its REST API directly. +Persistence is SQLite accessed through Drizzle ORM, using Bun's native SQLite driver. Authentication is Better Auth with session cookies, plus API keys through the Better Auth API key plugin. GitHub access goes through Octokit with the throttling plugin; GitLab, Gitea and Forgejo are reached through their REST APIs directly. The push engine for GitHub and GitLab destinations drives the `git` binary as a child process. ## Request path @@ -76,8 +76,13 @@ src/ │ ├── modules/ Module registry │ ├── polyfills/ Runtime shims │ ├── github.ts GitHub client -│ ├── gitea.ts Gitea client +│ ├── gitea.ts Gitea client and pull mirror path │ ├── gitea-enhanced.ts Metadata mirroring +│ ├── source-providers/ GitHub, GitLab and Gitea source adapters +│ ├── push-engine/ Bare clones, git runner, lock, limiter, GitHub and GitLab targets +│ ├── mirror-dispatch.ts Routes each repository to the pull mirror path or the push engine +│ ├── destination-kinds.ts Destination catalogue and beta flags +│ ├── destination-reconcile*.ts Reconcile the destination with the database │ ├── scheduler-service.ts │ ├── cleanup-service.ts │ ├── repository-cleanup-service.ts @@ -95,7 +100,7 @@ src/ Outside `src/`: -- `drizzle/` holds the generated SQL migrations and their journal. Migrations run automatically on startup. +- `drizzle/` holds the generated SQL migrations and their journal. Migrations run automatically on startup. Recent ones: 0015 added the source host columns on repositories, 0016 carried the Better Auth 1.7 schema, 0017 added API keys, and 0018 added the destination host columns on repositories with a backfill from each account's destination settings. - `scripts/` holds the runtime server, database management CLI, recovery and repair scripts, and the environment configuration loader entry point. - `helm/gitea-mirror/` is the Kubernetes Helm chart. - `www/` is the marketing and documentation site, a separate Astro project with its own dependencies. @@ -107,11 +112,13 @@ Tests are colocated with the code they cover, as `*.test.ts` files next to the i Everything lives in one SQLite database. The tables that matter: -`configs` holds per-user configuration as JSON columns validated by Zod schemas: `githubConfig`, `giteaConfig`, `scheduleConfig`, `cleanupConfig`, and `notificationConfig`. A user can have several configuration rows but one is marked active. +`configs` holds per-user configuration as JSON columns validated by Zod schemas: `githubConfig`, `giteaConfig`, `scheduleConfig`, `cleanupConfig`, and `notificationConfig`. The source host lives in `githubConfig` (`provider` and `url`) and the destination host in `giteaConfig` (`provider`), the column names being historical. A user can have several configuration rows but one is marked active. -`repositories` tracks every discovered repository with its GitHub metadata, its mirror status, its destination in Gitea, and any per-repository overrides. +`repositories` tracks every discovered repository with its source metadata, its mirror status, its location on the destination, and any per-repository overrides. Each row also records which source host it came from and which destination host it was mirrored to, so a row is never mirrored with the wrong token after a host switch. -`organizations` tracks GitHub organizations, including destination overrides that let one organization go somewhere different from what the global strategy would choose. +`organizations` tracks source organizations and GitLab groups, including destination overrides that let one organization go somewhere different from what the global strategy would choose. + +`api_keys` holds API keys, hashed, with their name, expiry and last use. `mirrorJobs` is the job record and history. Jobs move through `pending`, `mirroring`, and then `success` or `failed`. @@ -127,13 +134,17 @@ GitHub and Gitea tokens are encrypted with AES-256-GCM before being written, usi A mirror operation runs in stages. -Discovery queries GitHub for repositories matching the configured selection rules, filtered by visibility, fork status, archive status, collaborator affiliation, and organization allowlists and denylists. Results are written to the `repositories` table. +Discovery queries the configured source for repositories matching the configured selection rules, filtered by visibility, fork status, archive status, collaborator affiliation, and organization allowlists and denylists. Each source has an adapter under `src/lib/source-providers/` that maps its API onto the same repository shape. Results are written to the `repositories` table. Destination resolution decides where each repository goes, based on the mirror strategy plus any per-repository or per-organization override. The four strategies are `preserve`, `single-org`, `flat-user`, and `mixed`, described in the [configuration guide](./configuration.md). -Repository creation calls Gitea's migration API to create a pull mirror, creating the destination organization first if it does not exist. Organization creation is sequential by default to avoid races when many repositories target the same new organization. +The dispatcher in `src/lib/mirror-dispatch.ts` then picks one of two paths by destination kind, and every entry point goes through it: the mirror, sync and retry endpoints, the scheduler's auto mirror and sync phases, boot recovery and the organization mirror loop. -Metadata mirroring, if enabled, follows separately. Issues are recreated with their comments, labels, assignees, and milestones. Releases are recreated with their assets. Wiki content is cloned as a separate repository. +**Pull mirror path** (Gitea, Forgejo). Repository creation calls the migration API to create a pull mirror, creating the destination organization first if it does not exist. Organization creation is sequential by default to avoid races when many repositories target the same new organization. The destination fetches from the source itself on its mirror interval. + +**Push engine path** (GitHub, GitLab). The app keeps a bare clone per repository under the data directory, fetches it with prune, and pushes branches and tags to the target with force and prune. A target adapter creates, archives and deletes repositories through the host's API. A lock file per clone and a global concurrency limit keep the number of git processes bounded, and tokens reach git only through an inline credential helper fed from the child process environment. The [Destinations](./destinations.md) page describes the engine in detail. + +Metadata mirroring, if enabled and the source is GitHub and the destination a pull mirror, follows separately. Issues are recreated with their comments, labels, assignees, and milestones. Releases are recreated with their assets, subject to the release and asset limits. Wiki content is cloned as a separate repository. Throughout, job status updates are published as server-sent events so the dashboard reflects progress live without polling. @@ -157,13 +168,13 @@ The **scheduler** performs periodic mirroring, repository discovery, and auto-mi The **cleanup service** trims the activity log according to the configured retention period. -The **repository cleanup service** handles repositories that have disappeared from GitHub, archiving or deleting them according to the configured action. +The **repository cleanup service** handles repositories that have disappeared from the source, archiving or deleting them according to the configured action, on the destination through its API when allowed to. The related reconcile pass runs on demand rather than on a schedule. The **shutdown manager** tracks running jobs and persists their state when a termination signal arrives, so an interrupted job can be resumed on the next boot instead of being lost. ## Authentication -Better Auth handles sessions, with three sign-in paths. +Better Auth handles sessions, with three sign-in paths, plus API keys for scripts. Email and password authentication is always available. The first account to sign up becomes the admin. @@ -171,13 +182,13 @@ OIDC and SSO providers are registered through the Authentication tab rather than Header authentication trusts identity headers set by an upstream reverse proxy, for deployments where something like Authentik or Authelia already handles login. It can auto-provision accounts, optionally restricted to an email domain allowlist. -Sessions are cookie-based and validated in the middleware on every request. +Sessions are cookie-based and validated in the middleware on every request. An API key in the `x-api-key` header is accepted wherever a session cookie is, carries the rights of the user who created it, and is refused by the key management endpoints themselves so a leaked key cannot mint more keys. ## Deployment options **Docker** is the primary path. A multi-architecture image is published to `ghcr.io/raylabshq/gitea-mirror`, built from a multi-stage `Dockerfile` on the official Bun base image, with a statically compiled `git-lfs` binary included for LFS mirroring. Three compose files ship with the repository: `docker-compose.alt.yml` for a minimal prebuilt deployment, `docker-compose.yml` for a full environment-driven one, and `docker-compose.dev.yml` for development with hot reload. -**Kubernetes** is supported through the Helm chart in `helm/gitea-mirror`. It packages a Deployment, Service, ConfigMap, Secret, optional PersistentVolumeClaim, optional ServiceAccount, and a choice of Ingress or Gateway API HTTPRoute. It needs Kubernetes 1.23 or newer and Helm 3.8 or newer. +**Kubernetes** is supported through the Helm chart, published as an OCI package at `oci://ghcr.io/raylabshq/charts/gitea-mirror` with every release and kept in source at `helm/gitea-mirror`. It packages a Deployment, Service, ConfigMap, Secret, optional PersistentVolumeClaim, optional ServiceAccount, and a choice of Ingress or Gateway API HTTPRoute. It needs Kubernetes 1.23 or newer and Helm 3.8 or newer. **Nix** deployment works either as a flake package or as a NixOS module. `nix run github:RayLabsHQ/gitea-mirror` starts it with no setup, and the NixOS module handles secret generation and database initialization declaratively. diff --git a/www/src/pages/docs/cleanup-and-reconcile.mdx b/www/src/pages/docs/cleanup-and-reconcile.mdx new file mode 100644 index 0000000..3e60b97 --- /dev/null +++ b/www/src/pages/docs/cleanup-and-reconcile.mdx @@ -0,0 +1,78 @@ +--- +layout: '../../layouts/DocsLayout.astro' +title: Cleanup and Reconcile +description: What happens to mirrors whose source repository disappears, and how to bring the destination and the database back in step after a lost database or a manual change. +--- + +# Cleanup and Reconcile + +Two maintenance features live on the **Repository Cleanup** card of the Automation tab. Repository cleanup handles mirrors whose source repository is gone. Reconcile compares what the destination holds with what the database knows, and offers to fix the difference. + +## Repository cleanup + +Cleanup looks for repositories that the database tracks but that no longer exist on the source: a repository you deleted or unstarred, or one moved out of the account's reach. When it finds one it applies the configured action. + +### Settings + +| Setting | Environment variable | Default | Meaning | +| --- | --- | --- | --- | +| Enabled | `CLEANUP_ENABLED` | off | Runs as part of each scheduled sync. | +| Action | `CLEANUP_ORPHANED_REPO_ACTION` | `archive` | `skip` leaves the mirror untouched, `archive` keeps it read only with an archived prefix, `delete` removes it. | +| Dry run | `CLEANUP_DRY_RUN` | off | Log the planned actions without changing anything. | +| Delete from destination | `CLEANUP_DELETE_FROM_GITEA` | `false` | Apply the action on the destination too. Off means only the database is updated. | +| Protected repositories | `CLEANUP_PROTECTED_REPOS` | none | Comma-separated repository names cleanup never touches. | +| Batch size and pause | `CLEANUP_BATCH_SIZE`, `CLEANUP_PAUSE_BETWEEN_DELETES` | `10`, `2000` | Items per batch, and the pause between deletions in milliseconds. | + +The first three are on the card. The others are environment variables; see the [reference](./environment-variables.md#cleanup). + +Cleanup can also run on demand with `POST /api/cleanup/trigger`, see the [API page](./api.md#run-cleanup). + +### What each action does + +`archive` never deletes. With **Delete from destination** off, the default, the row is marked archived in the database and the destination is left alone. With it on: + +- On Gitea and Forgejo, a regular repository gets the native archive flag, which makes it read only while keeping every byte. A mirror repository cannot be archived through the Gitea API, so it is renamed with an `archived-` prefix, given a description that says why, and has its mirror interval pushed out to a year so no more sync attempts run. It remains fully cloneable. +- On GitHub and GitLab, the repository is archived through the host's API. + +`delete` removes the row from the database and, with **Delete from destination** on, removes the repository from the destination. Deleting on GitHub needs a token with the `delete_repo` scope; without it the run fails with a message naming the scope and the repository stays. On gitlab.com and instances with delayed deletion, a deleted project is renamed and kept for the configured number of days. For push targets the bare clone on disk is removed as well. + +If archiving fails for any reason, the repository is left alone and stays accessible. + +### How cleanup decides + +Cleanup is careful about what it calls an orphan, because the cost of a false positive is an archived or deleted backup. + +- It first lists everything the source token can see, including collaborator repositories and every organization regardless of the import filters, so narrowing a filter later cannot make old mirrors look orphaned. +- A repository missing from that list is only a candidate. It counts as orphaned when a direct lookup of that repository on the source returns a clean 404 as well. Any other outcome, including a rate limit, a network error or a 5xx, leaves it alone. +- If the source API cannot be reached at all, or answers 403 or 404 for the account itself, the whole run is skipped. +- Repositories imported from a previous source, and rows mirrored to a previous destination, are not touched. + +The **Database Maintenance** card above cleanup is a different thing: it trims the activity log after the configured retention period (`CLEANUP_RETENTION_DAYS`) and has nothing to do with repositories. + +## Reconcile + +Cleanup only looks at the database, so a mirror that exists on the destination without a row is invisible to every maintenance feature, and a row whose mirror was deleted by hand keeps saying "mirrored". Both happen after a lost or restored database, or after someone tidied up on the destination directly. Reconcile finds them. + +Click **Reconcile with destination** on the Repository Cleanup card. The dialog runs a dry run as soon as it opens and sorts what it finds into groups: + +- **On the destination, not in the database.** Mirrors of your source that this app does not know about. Adopting them adds a row so scheduled sync and cleanup include them. +- **In the database, gone from the destination.** Rows marked mirrored whose repository is no longer there. Resetting them makes the next mirror run recreate the mirror. +- **Not managed by gitea-mirror.** Native repositories and mirrors of other hosts. These are listed for information and never touched. +- **Could not be verified.** The destination did not answer the presence check for these rows. They were left alone. + +The dialog also reports a healthy count, and, when it applies, how many rows are mirrored to a previous destination and were left out of the comparison. + +Tick **Adopt** or **Reset** for the groups you want fixed and click **Apply**. The lists refresh afterwards, and one activity entry records what changed. + +### What counts as this app's mirror + +Only mirrors whose original URL points at the configured source count. The hosts that qualify are the source itself, the API host it is reached through, and the host of every clone URL already stored, which covers GitHub Enterprise and setups where the clone host differs from the nominal source. Everything else, including mirrors of other hosts and repositories that are not mirrors at all, lands in the not managed group. + +### Safety rules + +- A row is only called missing after a direct lookup of its expected location returns 404. A failed check leaves the row alone and lists it as unverified. +- Nothing is deleted or archived on either side, and rows are never deleted. Adopting inserts rows; resetting sets a row back to imported with a note in its error field. +- An adopted row records where the mirror already lives and keeps it there, even when the mirror strategy would put a new mirror elsewhere, so the next sync does not create a second copy. +- Adoption uses the existing unique index on account and full name, so it cannot create duplicates. + +Reconcile works with Gitea and Forgejo destinations. It is not available for GitHub and GitLab, which have no server side mirror state to compare. The same comparison is available as `POST /api/cleanup/reconcile`; see the [API page](./api.md#reconcile-the-destination). diff --git a/www/src/pages/docs/configuration.mdx b/www/src/pages/docs/configuration.mdx index 5ca9956..a5ca1ae 100644 --- a/www/src/pages/docs/configuration.mdx +++ b/www/src/pages/docs/configuration.mdx @@ -18,23 +18,33 @@ The Configuration page is organized into four tabs. ### Connections -This is where the two ends of the mirror are set up. It contains two cards: +This is where the two ends of the mirror are set up. It contains two cards, one per end, and each starts with a dropdown that picks the host. -**GitHub Connection** holds your GitHub username, personal access token, and account type (personal or organization). The token needs the `repo` scope for private repositories and `admin:org` to read organization data. This card also carries the repository selection options: whether to include private repositories, public repositories, archived repositories, forks, repositories where you are only a collaborator, and starred repositories. +**The source card** starts with the **Source** dropdown: GitHub, GitLab or Gitea / Forgejo. GitHub is the default. The other two carry a BETA pill and show an Instance URL field for self-hosted instances. Below that come the username and personal access token for the selected host, with the scopes the card needs listed next to the token field, and the repository selection options: whether to include private repositories, public repositories, archived repositories, forks, repositories where you are only a collaborator, and starred repositories. The [Sources](./sources.md) page covers each host. -**Gitea Connection** holds the destination. Gitea is the default; Forgejo, GitHub and GitLab are available from the Destination dropdown and marked beta, as are the GitLab and Gitea / Forgejo sources on the GitHub card. GitHub to Gitea is the supported path. The card holds the instance URL, an access token, and the default owner. It also carries the destination settings: the mirror strategy, default organization, organization visibility, and the per-repository mirror interval that Gitea itself uses to re-pull from GitHub. +**The destination card** starts with the **Destination** dropdown: Gitea, Forgejo, GitHub or GitLab. Gitea is the default; the other three carry a BETA pill. For Gitea and Forgejo the card holds the server URL, an optional external URL, an access token and the username that acts as default owner. For GitHub and GitLab it holds the instance URL, the account and a token, and a notice explains that those targets are overwritten on every sync. The card also carries the organization settings: the mirror strategy, default organization, organization visibility, and the per-repository mirror interval that Gitea itself uses to re-pull from the source. The [Destinations](./destinations.md) page covers each host. -Both cards have a test button that performs a live authenticated call, so you can confirm credentials before saving. +GitHub to Gitea is the supported path. Every other source and destination is beta: tested end to end against real hosts, with less time in the field. + +Both cards have a **Test** button that performs a live authenticated call, so you can confirm credentials before importing anything. + +**Locks.** Once repositories are imported the source locks, and once anything is mirrored the destination locks. The dropdown and the URL field then show a lock note with a **Change** button, and changing goes through a confirmation that spells out what happens to the existing repositories. Environment values that disagree with a locked host are ignored on boot with a warning. If your Gitea instance is reachable at one URL from the Gitea Mirror process and at a different URL from your browser, for example an internal service name versus a public hostname, set the external URL as well. Dashboard links use the external URL while all API and mirroring traffic keeps using the primary URL. +### Mirror settings + +Below the connection cards, the mirror settings decide what travels with the code: issues, pull requests, labels, milestones, releases, wiki and LFS objects, plus the force push protection strategy. Switches that the current source or destination cannot serve are disabled and say why. + +Releases have two numbers next to the switch. **Latest** is how many releases are mirrored per repository, newest first, ten by default. **Assets for latest** is how many of those get their assets uploaded: leave it empty to upload assets for every mirrored release, set it to `0` for release notes and tags only, or set it to a number to keep assets for the newest N releases while older ones keep their notes and tag. The limit never deletes assets that were already uploaded. Both values can be overridden per organization and per repository through the mirror overrides dialog, and the asset limit is `RELEASE_ASSET_LIMIT` in the environment. + ### Automation This tab controls scheduled mirroring and cleanup. Scheduled syncing runs on an interval or a cron schedule. When enabled it re-syncs existing mirrors, and optionally discovers newly created GitHub repositories and mirrors them without any manual action. Two separate switches govern that: auto-import decides whether new repositories are discovered and added to the database, and auto-mirror decides whether they are then actually mirrored to Gitea. -The same tab holds repository cleanup, which handles repositories that have disappeared from GitHub. The action for an orphaned repository is one of `skip`, `archive`, or `delete`, and `archive` is the default because it preserves the mirrored data. There is a dry-run mode that logs what would happen without touching anything, and a list of protected repository names that cleanup will never act on. +The same tab holds repository cleanup, which handles repositories that have disappeared from the source. The action for an orphaned repository is one of `skip`, `archive`, or `delete`, and `archive` is the default because it preserves the mirrored data. There is a dry-run mode that logs what would happen without touching anything. The card's **Reconcile with destination** button compares a Gitea or Forgejo server with the database and offers to adopt mirrors the app does not know about or reset rows whose mirror is gone. Both are described on the [Cleanup and reconcile](./cleanup-and-reconcile.md) page. ### Notifications @@ -48,19 +58,21 @@ This tab configures how people sign in. Email and password authentication is alw Header authentication reads the username, email, and display name from configurable request headers. It can optionally auto-provision accounts on first sight of a new user, restricted to an allowlist of email domains. +The **API Keys** card on the same tab creates keys for scripts and CI pipelines. A key is sent in the `x-api-key` header and can call every endpoint the web interface uses. See [API and API keys](./api.md). + > **Warning:** Only enable header authentication when the app is genuinely behind a trusted proxy that strips those headers from inbound client requests. If clients can set the headers themselves, anyone can sign in as anyone. ## Mirror strategies -The mirror strategy decides where each repository lands on the Gitea side. There are four, and they are set on the Gitea Connection card. +The mirror strategy decides where each repository lands on the destination. There are four, and they are set on the destination card. The examples below say Gitea and organization; on GitLab the equivalent is a group, and the GitHub target uses organizations that must already exist. ### preserve -Keeps the same organization structure as GitHub. A repository at `acme/api` on GitHub becomes `acme/api` in Gitea, with the organization created if it does not exist. This is the default and the right choice when you want the mirror to be recognizable to anyone who knows the GitHub side. +Keeps the same organization structure as the source. A repository at `acme/api` on GitHub becomes `acme/api` in Gitea, with the organization created if it does not exist. This is the default and the right choice when you want the mirror to be recognizable to anyone who knows the source side. ### single-org -Consolidates every repository into one Gitea organization, regardless of where it came from on GitHub. The target organization defaults to `github-mirrors`. Useful when you want all mirrored content in one clearly separated namespace. +Consolidates every repository into one Gitea organization, regardless of where it came from on the source. The target organization defaults to `github-mirrors`. Useful when you want all mirrored content in one clearly separated namespace. ### flat-user @@ -68,7 +80,7 @@ Places every repository directly under your Gitea user account, with no organiza ### mixed -Splits the difference. Your personal GitHub repositories go into a single named organization, while repositories owned by GitHub organizations preserve their original structure. This keeps organization repositories where people expect them while corralling your personal ones. +Splits the difference. Your personal repositories go into a single named organization, while repositories owned by organizations on the source preserve their original structure. This keeps organization repositories where people expect them while corralling your personal ones. > **Note:** Repositories mirrored because you starred them are handled separately from the strategy above. They can either go into a dedicated organization, which is named `starred` by default, or preserve their original owner's path. When two starred repositories from different owners share a name, a duplicate strategy setting decides whether the name gets a suffix, a prefix, or is placed in an owner-named organization. @@ -94,6 +106,8 @@ The second group seeds the database configuration. On startup, if any of `GITHUB Tokens supplied through the environment are encrypted with AES-256-GCM before they are stored, the same as tokens entered through the UI. +The source and destination hosts are seeded the same way: `SOURCE_PROVIDER` and `SOURCE_URL` pick the source, `DESTINATION_PROVIDER` picks the destination, and the `GITHUB_*` and `GITEA_*` names keep working for every host. A value that disagrees with a locked host is ignored with a warning rather than applied. + ## Core environment variables These are the ones that matter for nearly every deployment. The [environment variables reference](./environment-variables.md) has the complete list. diff --git a/www/src/pages/docs/destinations.mdx b/www/src/pages/docs/destinations.mdx new file mode 100644 index 0000000..a90768d --- /dev/null +++ b/www/src/pages/docs/destinations.mdx @@ -0,0 +1,134 @@ +--- +layout: '../../layouts/DocsLayout.astro' +title: Destinations +description: Where mirrors are created. Gitea is the default; Forgejo, GitHub and GitLab are available as beta destinations, the last two through a push engine. +--- + +# Destinations + +A destination is the host mirrors are created on. It is chosen from the **Destination** dropdown at the top of the second card on the Connections tab of the Configuration page. + +| Destination | How mirrors are kept up to date | Status | +| --- | --- | --- | +| Gitea | Pull mirror: Gitea fetches from the source on its own interval | default, supported | +| Forgejo | Pull mirror, same API as Gitea | beta | +| GitHub | Push engine: the app pushes branches and tags | beta | +| GitLab | Push engine | beta | + +GitHub to Gitea is the supported path. The other three carry a BETA pill in the dropdown. They are tested end to end against real hosts, including github.com and gitlab.com, but have had less time in the field. + +![The Destination dropdown open, showing the beta pills](/assets/destinations.png) + +## Gitea and Forgejo + +These are pull mirrors. For each repository the app calls the migrate API with the source clone URL and the source token, and the destination creates a mirror that it refreshes itself on the interval set on the card (the per-repository mirror interval, `GITEA_MIRROR_INTERVAL` in the environment). Everything beyond the git data, meaning issues, pull requests, labels, milestones, releases, wiki and LFS objects, is mirrored by the app through the destination's API, subject to what the [source](./sources.md) offers. + +The card holds: + +- **Server URL** of the instance, used for all API and mirroring traffic. +- **External URL**, optional, used for the links on the dashboard when your browser reaches the instance at a different address than the app does. +- **Username** and **access token**. The username is the default owner for mirrors that the mirror strategy does not send elsewhere. +- The organization settings: mirror strategy, default organization, organization visibility, and the mirror interval. See [Mirror strategies](./configuration.md#mirror-strategies). + +Forgejo shares Gitea's API, so everything Gitea supports works there. One thing to know: Forgejo releases before 15.0.0 do not persist the credentials sent through the migrate API, so later syncs of private repositories fail with `terminal prompts disabled`. The card shows a warning when the Test button detects such a version. Upgrade Forgejo, then delete and re-mirror the affected repositories, or re-enter the source token once in each repository's mirror settings on Forgejo. + +## GitHub and GitLab + +GitHub has no pull mirror API and GitLab only offers one on paid tiers, so for these two the app runs a push engine. When you pick one of them the card changes shape: the URL field becomes **Instance URL**, prefilled with `https://github.com` or `https://gitlab.com` and editable for GitHub Enterprise Server or a self-hosted GitLab, the username field becomes **Account**, and a notice explains that the target is overwritten on every sync. + +### How the push engine works + +For every repository whose destination is GitHub or GitLab: + +1. **A bare clone is kept on disk** under `mirrors///.git` inside the data directory, next to the SQLite database, so a backup of the data directory carries the clones. The first run clones; later runs fetch with prune, so a branch or tag deleted on the source disappears from the clone. +2. **The target repository is created when it is missing.** On GitHub it goes under the account, or under an organization the mirror strategy picks. On GitLab it goes under the user, or in a group; a missing top level group is created, a nested one is not. +3. **Branches and tags are pushed** with force and prune, so the target ends up identical to the source, including rewritten history and deleted branches. Pull request refs, merge request refs and other host-owned namespaces are not pushed, because hosts refuse writes to them. +4. **The default branch** on the target is set to match the source, best effort. +5. **Statuses, jobs and activity** are the same as for a Gitea mirror, so the dashboard, the scheduler, retry, recovery and cleanup treat both kinds the same way. The repository table shows the destination's icon and links to the target repository. + +Scheduled sync runs the same fetch and push again. Both steps are idempotent, so an interrupted run simply runs again. + +Right after creating a repository, GitLab (and occasionally GitHub) answers the first push with "not found" for a second or two. The engine retries that first push up to six times, two seconds apart, before it counts as a failure. + +### The target is overwritten + +Every push uses force and prune. Anything committed directly on the target, a branch created there or a tag moved there is replaced by what the source has on the next sync. Treat a push target as read only, the same way a Gitea pull mirror is. + +### What is not mirrored to push targets + +Push targets receive **branches and tags only**. These switches are disabled with a reason in the mirror settings when a push target is selected: + +- issues, pull requests, labels and milestones +- releases and release assets (tags do travel) +- wiki +- LFS objects +- force push protection and its backup strategies + +Several destinations per repository, and pulling anything back from the target, are out of scope. Mirroring is one way. + +### Tokens + +Tokens never appear in a URL and are never written to disk. Each git process gets an inline credential helper that reads the token from an environment variable set on that process only: the source token for the fetch, the target token for the push. The remote URL stored in the bare clone is the plain source URL. + +| Target | Scopes | Notes | +| --- | --- | --- | +| GitHub | `repo`, `workflow` | GitHub rejects a push that adds GitHub Actions workflow files unless the token has `workflow`. Add `delete_repo` only if cleanup should delete repositories on GitHub. | +| GitLab | `api`, `write_repository` | Pushes use the `oauth2` user name with the token, as GitLab documents. | + +GitHub organizations cannot be created through the API. Create the organization first and make the token's user a member with permission to create repositories. A private GitLab group forces its projects to be private, whatever the source visibility. + +### Disk usage and concurrency + +A bare clone takes roughly the size of the repository's git objects, so the data directory grows by about the size of everything you mirror this way. A clone is removed when its repository row is deleted, and by cleanup when it deletes an orphaned repository. + +| Variable | Default | Meaning | +| --- | --- | --- | +| `MIRROR_CLONE_DIR` | `mirrors` inside the data directory | Where the bare clones live. | +| `PUSH_CONCURRENCY` | `2` | How many git fetch or push operations run at the same time across all repositories. API calls are cheap; a clone or a mirror push is not. | + +One git process per repository at a time is enforced with a lock file next to the clone. A lock older than an hour belongs to a run that died and is taken over. A half written clone, one without a `HEAD`, is removed and cloned again. + +GitHub limits the size of a single push. When one push of all refs is refused, the engine retries in batches of 50 refs, branches first, then tags, followed by a prune. If a batch is still refused, the mirror fails with git's message in the repository's status. + +### Cleanup on push targets + +When the source deletes a repository, the [cleanup service](./cleanup-and-reconcile.md) applies the configured action to the target through its API: archive, or delete. Deleting on GitHub needs the `delete_repo` scope; without it the cleanup run fails with a message naming the scope and the repository stays. On gitlab.com and instances with delayed deletion, a deleted project is renamed and kept for the configured number of days before it disappears. The bare clone is removed with the row. + +**Reconcile with destination** compares a Gitea or Forgejo server with the database. It is not available for push targets, which have no server side mirror state to reconcile. + +> **Warning:** With a push target, every repository the app imports gets created on the target once it is mirrored. If auto import and auto mirror are both on, that includes every repository the source token can see. Check the import list before enabling automatic mirroring against a GitHub organization or a GitLab group. + +## Switching destinations + +Once anything has been mirrored the destination locks. The dropdown and the URL field show a lock note with the number of mirrored repositories and a **Change** button, and changing goes through a confirmation that spells out what happens: + +- Existing mirrors stay on the current server and are not moved. +- Syncing them from here fails until they exist on the new server or are removed and mirrored again. +- New mirrors go to the new server only. + +Each repository row records the host it was mirrored to. A row mirrored to one host is refused on another until it is removed and mirrored again, so a token is never sent to the wrong server. Reconcile reports such rows as mirrored to a previous destination and leaves them alone. A `DESTINATION_PROVIDER` or `GITEA_URL` value that disagrees with a locked destination is ignored on boot with a warning. + +## Environment variables + +The `GITEA_*` names are kept for every destination kind, so an existing deployment only has to add `DESTINATION_PROVIDER`. + +| Variable | Default | Description | +| --- | --- | --- | +| `DESTINATION_PROVIDER` | `gitea` | `gitea`, `forgejo`, `github` or `gitlab`. | +| `GITEA_URL` | none | Instance URL. Optional for GitHub and GitLab, which default to github.com and gitlab.com; set it for GitHub Enterprise Server or a self-hosted GitLab. | +| `GITEA_EXTERNAL_URL` | none | Browser-facing URL for dashboard links, when it differs from `GITEA_URL`. | +| `GITEA_USERNAME` | none | Account on the destination. | +| `GITEA_TOKEN` | none | Token with the scopes above. | +| `GITEA_ORGANIZATION` | `github-mirrors` | Default organization or group for the `single-org` strategy. | +| `MIRROR_CLONE_DIR` | `mirrors` inside the data directory | Push engine clone directory. | +| `PUSH_CONCURRENCY` | `2` | Concurrent git operations for the push engine. | + +A GitHub organization as destination: + +```bash +DESTINATION_PROVIDER=github +GITEA_USERNAME=your-github-account +GITEA_TOKEN=${GITHUB_TARGET_TOKEN} +GITEA_ORGANIZATION=your-org +MIRROR_STRATEGY=single-org +``` diff --git a/www/src/pages/docs/environment-variables.mdx b/www/src/pages/docs/environment-variables.mdx index 261c5f5..6678e7e 100644 --- a/www/src/pages/docs/environment-variables.mdx +++ b/www/src/pages/docs/environment-variables.mdx @@ -60,14 +60,18 @@ For deployments behind a reverse proxy that already authenticates users. > **Warning:** Enable this only when a trusted proxy strips these headers from inbound client requests. Otherwise any client can set them and sign in as any user. -## GitHub +## Source ### Connection +The variable names are historical: `GITHUB_USERNAME` and `GITHUB_TOKEN` hold the credentials for whichever source is selected. See [Sources](./sources.md). + | Variable | Default | Description | | --- | --- | --- | -| `GITHUB_USERNAME` | none | Your GitHub username. | -| `GITHUB_TOKEN` | none | Personal access token. Needs `repo` for private repositories and `admin:org` to read organization data. | +| `SOURCE_PROVIDER` | `github` | Which host repositories are pulled from: `github`, `gitlab` or `gitea`. `gitea` also covers Forgejo and Codeberg. Anything other than `github` is beta. | +| `SOURCE_URL` | `https://gitlab.com` or `https://codeberg.org` | Base URL of the GitLab or Gitea / Forgejo instance. Ignored for GitHub, which uses `GH_API_URL`. | +| `GITHUB_USERNAME` | none | Your username on the source host. | +| `GITHUB_TOKEN` | none | Personal access token for the source host. GitHub needs `repo` and `admin:org`, GitLab `read_api` and `read_repository`, Gitea / Forgejo `read:repository`, `read:user` and `read:organization`. | | `GITHUB_TYPE` | `personal` | Account type: `personal` or `organization`. | | `GH_API_URL` | `https://api.github.com` | GitHub API base URL. Point this at GitHub Enterprise Server (`https://ghe.example.com/api/v3`) or Enterprise Cloud with data residency (`https://api.TENANT.ghe.com`). | | `GITHUB_API_URL` | none | Alternate name for `GH_API_URL`, checked second. Prefer `GH_API_URL`, since `GITHUB_API_URL` is set automatically inside GitHub Actions and can collide. | @@ -102,19 +106,31 @@ Standard GitHub Enterprise Cloud on `github.com` needs no override. Use a token | `PRESERVE_ORG_STRUCTURE` | `false` | Keep the GitHub organization structure in Gitea. Shorthand for the `preserve` strategy. | | `MIRROR_STRATEGY` | `preserve` | Destination layout: `preserve`, `single-org`, `flat-user`, or `mixed`. Takes precedence over `PRESERVE_ORG_STRUCTURE`. | -## Gitea +## Destination ### Connection +The `GITEA_*` names are kept for every destination kind, so an existing deployment only has to add `DESTINATION_PROVIDER`. See [Destinations](./destinations.md). + | Variable | Default | Description | | --- | --- | --- | -| `GITEA_URL` | none | Gitea or Forgejo instance URL used for all API and mirroring traffic. | +| `DESTINATION_PROVIDER` | `gitea` | Which host mirrors are created on: `gitea`, `forgejo`, `github` or `gitlab`. Gitea and Forgejo are pull mirrors; GitHub and GitLab receive pushes of branches and tags from the app. Anything other than `gitea` is beta. | +| `GITEA_URL` | none | Destination instance URL used for all API and mirroring traffic. Optional for GitHub and GitLab, which default to github.com and gitlab.com; set it for GitHub Enterprise Server or a self-hosted GitLab. | | `GITEA_EXTERNAL_URL` | none | Browser-facing URL used for dashboard links, when it differs from `GITEA_URL`. | -| `GITEA_TOKEN` | none | Gitea access token. | -| `GITEA_USERNAME` | none | Gitea username, used as the default owner. | -| `GITEA_ORGANIZATION` | `github-mirrors` | Default destination organization for the `single-org` strategy. | +| `GITEA_TOKEN` | none | Destination access token. GitHub push targets need `repo` and `workflow`, plus `delete_repo` if cleanup should delete; GitLab needs `api` and `write_repository`. | +| `GITEA_USERNAME` | none | Destination username, used as the default owner. | +| `GITEA_ORGANIZATION` | `github-mirrors` | Default destination organization or group for the `single-org` strategy. | | `GITEA_SKIP_TLS_VERIFY` | `false` | Disable TLS certificate verification. Sets `NODE_TLS_REJECT_UNAUTHORIZED=0` process-wide. | +### Push engine + +Only used when `DESTINATION_PROVIDER` is `github` or `gitlab`. + +| Variable | Default | Description | +| --- | --- | --- | +| `MIRROR_CLONE_DIR` | `mirrors` inside the data directory | Where the push engine keeps its bare clones, one per repository. | +| `PUSH_CONCURRENCY` | `2` | How many git fetch or push operations run at once across all repositories, 1 to 32. | + > **Warning:** `GITEA_SKIP_TLS_VERIFY=true` disables certificate checking for every outbound HTTPS request the process makes, not just to Gitea. Use [CA certificates](./ca-certificates.md) instead for self-signed or privately signed instances. ### Repository settings @@ -142,13 +158,16 @@ Standard GitHub Enterprise Cloud on `github.com` needs no override. Use a token | `MIRROR_LABELS` | `false` | Mirror repository labels. | | `MIRROR_MILESTONES` | `false` | Mirror milestones. | | `MIRROR_RELEASES` | `false` | Mirror GitHub releases along with their assets. | -| `RELEASE_LIMIT` | `10` | Maximum releases mirrored per repository. | +| `RELEASE_LIMIT` | `10` | Maximum releases mirrored per repository, newest first. | +| `RELEASE_ASSET_LIMIT` | unset | Upload release assets only for the newest N mirrored releases; older releases still get their notes and tag. Unset means assets for every mirrored release, `0` means notes only. Never deletes assets already uploaded. | | `MIRROR_WIKI` | `false` | Clone wiki content. | | `MIRROR_ISSUE_CONCURRENCY` | `3` | Issues processed in parallel. | | `MIRROR_PULL_REQUEST_CONCURRENCY` | `5` | Pull requests processed in parallel. | > **Note:** Concurrency above `1` speeds up metadata mirroring but issues and pull requests may be created out of chronological order, since Gitea assigns numbers on creation. Set both to `1` if exact ordering matters more than throughput. +Issues, pull requests, labels, milestones and releases are read from the GitHub API and need a GitHub source. With a GitHub or GitLab destination none of the metadata options apply, and neither do wiki and LFS: push targets receive branches and tags only. + ## Scheduling | Variable | Default | Description | @@ -175,7 +194,7 @@ Standard GitHub Enterprise Cloud on `github.com` needs no override. Use a token | Variable | Default | Description | | --- | --- | --- | -| `AUTO_IMPORT_REPOS` | `true` | Discover and import newly created GitHub repositories during scheduled syncs. Set to `false` to disable. | +| `AUTO_IMPORT_REPOS` | `true` | Discover and import newly created repositories on the source during scheduled syncs. Set to `false` to disable. | | `AUTO_MIRROR_REPOS` | `false` | Mirror newly imported repositories automatically, without a manual "Mirror All". | | `SCHEDULE_ONLY_MIRROR_UPDATED` | `false` | Only mirror repositories that have changed since the last run. | | `SCHEDULE_UPDATE_INTERVAL` | `86400000` | How often to check for updates, in milliseconds. | @@ -194,15 +213,15 @@ Notification providers themselves (ntfy, Apprise, Gotify, and generic webhook) a ## Cleanup -Cleanup covers both trimming the activity log and handling repositories that no longer exist on GitHub. +Cleanup covers both trimming the activity log and handling repositories that no longer exist on the source. See [Cleanup and reconcile](./cleanup-and-reconcile.md). | Variable | Default | Description | | --- | --- | --- | | `CLEANUP_ENABLED` | `false` | Enable automatic cleanup. Also enabled implicitly when `CLEANUP_DELETE_IF_NOT_IN_GITHUB=true`. | | `CLEANUP_RETENTION_DAYS` | `7` | How long activity events are kept, in days. | -| `CLEANUP_DELETE_IF_NOT_IN_GITHUB` | `true` | Act on repositories that no longer exist on GitHub. | +| `CLEANUP_DELETE_IF_NOT_IN_GITHUB` | `true` | Act on repositories that no longer exist on the source. | | `CLEANUP_ORPHANED_REPO_ACTION` | `archive` | What to do with an orphaned repository: `skip`, `archive`, or `delete`. | -| `CLEANUP_DELETE_FROM_GITEA` | `false` | Permit deletion of the repository from Gitea, not just from the local database. | +| `CLEANUP_DELETE_FROM_GITEA` | `false` | Apply the orphan action on the destination too, not just in the local database. Deleting on a GitHub push target needs the `delete_repo` scope. | | `CLEANUP_PROTECTED_REPOS` | none | Comma-separated repository names cleanup will never touch. | | `CLEANUP_DRY_RUN` | `false` | Log what cleanup would do without doing it. | | `CLEANUP_BATCH_SIZE` | `10` | Items processed per batch. | @@ -210,7 +229,7 @@ Cleanup covers both trimming the activity log and handling repositories that no Cleanup has several safety behaviors worth knowing: -- If the GitHub API returns an error, cleanup is skipped for that run. A 404 or a rate limit never causes a mirror to be treated as orphaned. +- If the source API returns an error, cleanup is skipped for that run. A rate limit or a failed direct lookup never causes a mirror to be treated as orphaned; only a clean 404 on the repository itself does. - The `archive` action never deletes. Regular repositories use Gitea's native archive flag, which makes them read-only while preserving all data. Mirror repositories cannot be archived through the Gitea API, so they are renamed with an `archived-` prefix, given a description noting the preservation, and have their mirror interval pushed out to a year to stop sync attempts. They remain fully cloneable. - If archiving fails for any reason, the repository is left alone and stays accessible. diff --git a/www/src/pages/docs/quickstart.mdx b/www/src/pages/docs/quickstart.mdx index ec20e3f..ca04f0e 100644 --- a/www/src/pages/docs/quickstart.mdx +++ b/www/src/pages/docs/quickstart.mdx @@ -10,6 +10,8 @@ description: Get Gitea Mirror running with Docker in a few minutes, or build it Gitea Mirror ships as a prebuilt multi-architecture container image. The fastest path is to pull that image and configure everything else through the web interface after it starts. This page covers the Docker path first, then building from source, then running directly on Bun. +This guide walks the supported path, GitHub to Gitea. GitLab and Gitea / Forgejo as sources, and Forgejo, GitHub and GitLab as destinations, are available as beta options in the same two dropdowns; see [Sources](./sources.md) and [Destinations](./destinations.md) once the app is running. + ## Requirements Before you start, you need: @@ -148,9 +150,9 @@ Whichever install path you took, the first-run flow is the same: 1. Open the app at `http://localhost:4321`, or at whatever URL you configured. 2. Sign up. The first account created becomes the admin account. -3. Go to the Configuration page and fill in the **GitHub Connection** and **Gitea Connection** cards on the Connections tab. Use the test buttons to confirm both sides authenticate. +3. Go to the Configuration page and fill in the two connection cards on the Connections tab. Leave the Source dropdown on GitHub and the Destination dropdown on Gitea, enter the username and token for each, and use the Test buttons to confirm both sides authenticate. 4. Choose a mirror strategy, which decides where repositories land in Gitea. -5. Import your GitHub repositories, then mirror them. +5. Import your GitHub repositories, then mirror them. Once repositories are imported the source locks, and once one is mirrored the destination locks; changing either later asks for confirmation. 6. Optionally, turn on scheduled syncing on the Automation tab so new commits and new repositories are picked up without manual action. ## Verifying the deployment @@ -166,6 +168,8 @@ The container image also has a built-in `HEALTHCHECK` that polls this endpoint, ## Where to go next - [Configuration](./configuration.md) covers the Configuration page tab by tab, plus environment-based pre-configuration. +- [Sources](./sources.md) and [Destinations](./destinations.md) cover the hosts beyond GitHub and Gitea. +- [API and API keys](./api.md) is for driving the app from scripts and CI. - [Environment variables](./environment-variables.md) is the complete reference. - [CA certificates](./ca-certificates.md) explains how to trust a self-signed or privately signed Gitea instance. - The [use-case guides](/use-cases/) cover platform-specific installs (Proxmox, Unraid, Helm) and deeper topics like [Git LFS](/use-cases/mirror-git-lfs-repositories/), [issue and PR mirroring](/use-cases/mirror-github-issues-pull-requests/), and a [full migration to Gitea](/use-cases/migrate-github-to-gitea/). diff --git a/www/src/pages/docs/sources.mdx b/www/src/pages/docs/sources.mdx new file mode 100644 index 0000000..4b0f411 --- /dev/null +++ b/www/src/pages/docs/sources.mdx @@ -0,0 +1,116 @@ +--- +layout: '../../layouts/DocsLayout.astro' +title: Sources +description: Where Gitea Mirror pulls repositories from. GitHub is the default; GitLab and Gitea or Forgejo are available as beta sources. +--- + +# Sources + +A source is the host Gitea Mirror pulls repositories from. Each account has one source at a time, chosen from the **Source** dropdown at the top of the first card on the Connections tab of the Configuration page. + +| Source | Covers | Default instance | +| --- | --- | --- | +| GitHub | github.com and GitHub Enterprise | `https://github.com` | +| GitLab (beta) | gitlab.com and self-hosted GitLab | `https://gitlab.com` | +| Gitea / Forgejo (beta) | Codeberg and any self-hosted Gitea or Forgejo | `https://codeberg.org` | + +GitHub to Gitea is the supported path. GitLab and Gitea / Forgejo carry a BETA pill in the dropdown: they are tested end to end against real hosts, but have had less time in the field. Sources other than GitHub also carry a short note under the dropdown saying what they cannot mirror. + +![The Configuration page with the Source and Destination dropdowns](/assets/configuration.png) + +## Choosing a source + +Pick the host in the dropdown. The rest of the card adapts to it: + +- **Instance URL** appears for GitLab and Gitea / Forgejo. Leave it empty for gitlab.com or codeberg.org, or enter the base URL of your own instance, for example `https://gitlab.example.com` or `http://gitea.local:3000/gitea`. Values without a scheme get `https://` added. +- **Username** and **Personal access token** hold the account and token on the selected host. +- The **Test** button in the card header makes a live authenticated call against the selected host, so you can confirm the token before importing anything. + +GitHub Enterprise Server and Enterprise Cloud with data residency are configured through the `GH_API_URL` environment variable rather than the Instance URL field, which only shows for the other two sources. See [Environment variables](./environment-variables.md#source). + +## Tokens + +The card lists the scopes it needs and links to the token page of the selected instance. + +| Source | Where to create the token | Scopes | +| --- | --- | --- | +| GitHub | Settings, Developer settings, Personal access tokens (classic) | `repo`, `admin:org` | +| GitLab | Preferences, Access tokens | `read_api`, `read_repository` | +| Gitea / Forgejo | Settings, Applications | `read:repository`, `read:user`, `read:organization` | + +Without a token, GitLab and Gitea / Forgejo only show the configured user's public repositories and starred repositories. Adding a public repository by URL works without a token on every source. + +Tokens are encrypted with AES-256-GCM before they are stored, whether they come from the form or from the environment. + +## What each source can mirror + +The code itself is copied by the destination: Gitea and Forgejo pull from the source as a plain git remote, and the push engine used for GitHub and GitLab destinations clones and pushes the same way. Sources therefore differ only in what their API offers on top of git. + +| | GitHub | GitLab | Gitea / Forgejo | +| --- | :---: | :---: | :---: | +| Code, branches, tags | yes | yes | yes | +| Wiki | yes | yes | yes | +| LFS objects | yes | yes | yes | +| Scheduled sync | yes | yes | yes | +| Auto import of new repositories | yes | yes | yes | +| Cleanup of repositories deleted upstream | yes | yes | yes | +| Starred repositories | yes | yes | yes | +| Private repositories | yes | yes | yes | +| Add a single repository by URL | yes | yes | yes | +| Issues, pull requests, labels, milestones | yes | no | no | +| Releases with assets | yes | no | no | +| Star lists | yes | no | no | +| Force push detection | yes | no | no | + +The GitHub only rows read the GitHub API. With another source the corresponding switches in the mirror settings are disabled and say why. Wiki and LFS also depend on the destination: they travel to Gitea and Forgejo, not to GitHub or GitLab. See [Destinations](./destinations.md). + +## Adding repositories + +Repositories reach the database in two ways. + +**Import** discovers what the token can see on the source: the account's own repositories, the organizations or groups it belongs to, and its starred repositories, filtered by the selection options on the card (private, public, archived, forks, collaborator repositories, starred). Scheduled syncing can repeat the discovery so new repositories are picked up on their own. + +**Add by URL** takes a single repository from the Repositories page. The dialog accepts what you are likely to have on the clipboard: a browser URL, a clone URL, an SSH remote, the `owner/repo` shorthand, or a deep link into the repository such as a file or an issue. Nested GitLab groups are fine, so `https://gitlab.com/group/subgroup/project` resolves to the project in the subgroup. The Organizations page accepts a profile URL or a bare name the same way. The server re-resolves the reference against the configured source, so a URL for a different host than the configured one is refused. + +## Behaviour to know about + +**The source locks once repositories are imported.** The dropdown and the Instance URL then show a lock note with the number of repositories imported from that host, and a **Change** button. Changing the source is still possible, but only after a confirmation that spells out what happens: + +- Repositories already imported stay tied to the current source and keep syncing. +- Cleanup ignores them, and mirroring one of them again is refused until it is removed and added from the new source. +- New imports come from the new source only. + +A save that tries to switch a locked source without that confirmation is refused by the API, and a `SOURCE_PROVIDER` or `SOURCE_URL` value that disagrees with a locked source is ignored on boot with a warning in the log. + +**Each repository remembers its host.** A row imported from one host is never mirrored with another host's token, so switching sources cannot send a token to the wrong server. + +**GitLab groups are flattened.** A project under `group/subgroup/project` lands in the organization `group` with the preserve strategy and keeps its full path as the repository's full name. The organization allowlist matches the top level group. + +**GitLab internal projects are treated as private** when the mirror is created, because anonymous users cannot see them on the source either. + +**Repository names come from the URL path**, not the display name. A GitLab project called "My Widget" with path `my-widget` mirrors as `my-widget`. + +## Environment variables + +| Variable | Default | Description | +| --- | --- | --- | +| `SOURCE_PROVIDER` | `github` | `github`, `gitlab` or `gitea`. `gitea` also covers Forgejo and Codeberg. | +| `SOURCE_URL` | per source | Base URL of the GitLab or Gitea / Forgejo instance. Ignored for GitHub. | +| `GITHUB_USERNAME` | none | Username on the selected host. The name is kept for every source. | +| `GITHUB_TOKEN` | none | Token for the selected host, with the scopes listed above. | +| `GH_API_URL` | `https://api.github.com` | GitHub API base URL, for GitHub Enterprise. | + +A self-hosted GitLab source declared through the environment: + +```bash +SOURCE_PROVIDER=gitlab +SOURCE_URL=https://gitlab.example.com +GITHUB_USERNAME=your-gitlab-username +GITHUB_TOKEN=${GITLAB_TOKEN} +``` + +The [environment variables reference](./environment-variables.md) has the complete list, and the [configuration guide](./configuration.md) explains how seeded values relate to the form. + +## Not yet + +Several sources at once is tracked in [issue #375](https://github.com/RayLabsHQ/gitea-mirror/issues/375). Issues, merge requests and releases from GitLab and Gitea sources are not mirrored.