diff --git a/.github/screenshots/multi-source/sources-add-editor.png b/.github/screenshots/multi-source/sources-add-editor.png new file mode 100644 index 0000000..e3fdf01 Binary files /dev/null and b/.github/screenshots/multi-source/sources-add-editor.png differ diff --git a/.github/screenshots/multi-source/sources-card.png b/.github/screenshots/multi-source/sources-card.png new file mode 100644 index 0000000..586ddea Binary files /dev/null and b/.github/screenshots/multi-source/sources-card.png differ diff --git a/CHANGELOG.md b/CHANGELOG.md index 22cfb9e..cba9012 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] ### Added +- Multi-source support: connect multiple source services per user (#375 follow-up) + - Sources list on the Configuration page with add, edit and remove per source + - Per-repository source attribution, with credentials resolved from each repository's own source + - Per-source locking, discovery, auto import and cleanup - Git LFS (Large File Storage) support for mirroring (#74) - New UI checkbox "Mirror LFS" in Mirror Options - Automatic LFS object transfer when enabled diff --git a/README.md b/README.md index edbcbfa..686425e 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ First user signup becomes admin. Configure GitHub and Gitea/Forgejo through the ## ✨ Features - 🔁 Mirror public, private, and starred GitHub repos to Gitea/Forgejo -- 🦊 **GitLab and Gitea/Forgejo sources (beta)** - Pick the source in the configuration card. gitlab.com, Codeberg and self hosted instances mirror code, tags, wiki and LFS (issues, pull requests and releases need a GitHub source). GitHub to Gitea is the supported default; every other source or destination is beta. See [docs/SOURCE_PROVIDERS.md](docs/SOURCE_PROVIDERS.md) +- 🦊 **Multiple sources (GitLab and Gitea/Forgejo beta)** - Connect several sources at once: GitHub and GitHub Enterprise, GitLab, Gitea/Forgejo, and each repository keeps its source. gitlab.com, Codeberg and self hosted instances mirror code, tags, wiki and LFS (issues, pull requests and releases need a GitHub source). GitHub to Gitea is the supported default; every other source or destination is beta. See [docs/SOURCE_PROVIDERS.md](docs/SOURCE_PROVIDERS.md) - 🏛️ **GitHub Enterprise support** - Works with GHES and GHEC with data residency via `GH_API_URL` - 🚀 **Forgejo, GitHub and GitLab destinations (beta)** - Forgejo works like Gitea. Pick GitHub or GitLab and the app keeps a bare clone of each repository and pushes its branches and tags there. Code only, and the target is overwritten on every sync. See [docs/PUSH_TARGETS.md](docs/PUSH_TARGETS.md) - 🏢 Mirror entire organizations with flexible strategies diff --git a/docs/SOURCE_PROVIDERS.md b/docs/SOURCE_PROVIDERS.md index e8934dd..530c7a9 100644 --- a/docs/SOURCE_PROVIDERS.md +++ b/docs/SOURCE_PROVIDERS.md @@ -2,7 +2,7 @@ GitHub to Gitea is the supported default. Every other source and every other destination is marked **beta** in the dropdowns: they are tested end to end against real hosts, but have had less time in the field. The destination card has a matching **Destination** dropdown with Gitea, Forgejo (beta, same API as Gitea), GitHub (beta) and GitLab (beta); `DESTINATION_PROVIDER` sets it from the environment, and [PUSH_TARGETS.md](PUSH_TARGETS.md) covers the GitHub and GitLab targets. -Gitea Mirror pulls repositories from one source host per user. GitHub is the default. GitLab and Gitea/Forgejo are available from the **Source** dropdown at the top of the connection card on the Configuration page. +Gitea Mirror pulls repositories from as many source hosts as the user connects, github.com and a GitHub Enterprise instance included. The connection card on the Configuration page is a list of sources with add, edit and remove. GitHub is the default, and GitLab and Gitea/Forgejo are available from the **Source** dropdown when a source is added or edited. | Source | Covers | Default instance URL | |--------|--------|----------------------| @@ -47,8 +47,8 @@ The GitHub only rows read the GitHub API. For other sources the corresponding sw ## Behaviour to know about -- **The source locks once repositories are imported, and the destination locks once anything is mirrored.** The Source dropdown, the instance URL and the Gitea server URL then show a lock note and a **Change** button. Changing one is still possible, but only after confirming a dialog that spells out what happens to the existing repositories. Saves that try to switch a locked host without that confirmation are refused by the API, and an environment variable that disagrees with a locked host is ignored on boot with a warning. -- **One source at a time.** Switching the source does not touch repositories that were imported from the previous host. They keep syncing, because Gitea already holds their clone credentials. The cleanup service ignores them, and mirroring one of them again is refused with an error naming both hosts. Remove the repository and add it again from the current source if you need to re-mirror it. +- **Each source locks once repositories are imported from it, and the destination locks once anything is mirrored.** A locked source's **Source** dropdown and instance URL, and the Gitea server URL, then show a lock note and a **Change** button. Changing them is still possible, but only after confirming a dialog that spells out what happens to the existing repositories. Removing a source that has repositories needs an explicit confirmation too. Saves that try to switch a locked source without that confirmation are refused by the API, and an environment variable that disagrees with a locked source is ignored on boot with a warning. +- **Sources run side by side.** Every connected source discovers, imports and mirrors independently, and each repository remembers which source it came from. Cleanup checks each source's own repositories, and mirroring a repository uses its own source's credentials. Repositories whose source was removed keep syncing on the Gitea side, because Gitea holds the stored clone credentials, but re-mirroring, metadata and cleanup skip them. - **GitLab groups are flattened.** A project under `group/subgroup/project` lands in the Gitea organization `group` (with the preserve strategy) and keeps its full path as the repository's full name. The "Limit to specific groups" filter matches the top level group. - **GitLab internal projects are treated as private** when the mirror is created, because they are not visible to anonymous users on the source either. - **Repository names come from the URL path**, not the display name, so a GitLab project called "My Widget" with path `my-widget` mirrors as `my-widget`. @@ -63,8 +63,10 @@ The GitHub only rows read the GitHub API. For other sources the corresponding sw | `GITHUB_USERNAME` | Username on the selected host | | `GITHUB_TOKEN` | Token for the selected host | +These variables initialize or update the first source connection, creating it when none exists, and never switch a source that is locked. A `GH_API_URL` set for a single GitHub Enterprise deployment keeps applying to sources that point at github.com, so adding a github.com source in such a deployment sends its token to the configured Enterprise API; connect the Enterprise host as its own source instead. + See [ENVIRONMENT_VARIABLES.md](ENVIRONMENT_VARIABLES.md) for the full list. ## Not yet -Several sources at once (the tracking issue is #375, the follow up is a list of sources in the card), hosts without a usable API, and issues, merge requests and releases from GitLab, Gitea or Forgejo sources. GitHub and GitLab as destinations shipped in 3.31.0 through the push engine, see [PUSH_TARGETS.md](PUSH_TARGETS.md). +Hosts without a usable API, and issues, merge requests and releases from GitLab, Gitea or Forgejo sources. Discovery preferences are shared, so filters apply to all sources rather than per source, and the GitHub and GitLab rate limit buckets are shared per provider. GitHub and GitLab as destinations shipped in 3.31.0 through the push engine, see [PUSH_TARGETS.md](PUSH_TARGETS.md). diff --git a/drizzle/0019_multi_source.sql b/drizzle/0019_multi_source.sql new file mode 100644 index 0000000..63ac897 --- /dev/null +++ b/drizzle/0019_multi_source.sql @@ -0,0 +1,82 @@ +CREATE TABLE `sources` ( + `id` text PRIMARY KEY NOT NULL, + `user_id` text NOT NULL, + `name` text NOT NULL, + `provider` text NOT NULL, + `url` text NOT NULL, + `username` text DEFAULT '' NOT NULL, + `token` text, + `enabled` integer DEFAULT true NOT NULL, + `created_at` integer DEFAULT (unixepoch()) NOT NULL, + `updated_at` integer DEFAULT (unixepoch()) NOT NULL, + FOREIGN KEY (`user_id`) REFERENCES `users`(`id`) ON UPDATE no action ON DELETE no action +); +--> statement-breakpoint +CREATE INDEX `idx_sources_user_id` ON `sources` (`user_id`);--> statement-breakpoint +CREATE UNIQUE INDEX `uniq_sources_user_provider_url_username` ON `sources` (`user_id`,`provider`,`url`,`username`);--> statement-breakpoint +-- Seed one source row per ACTIVE config that carries connection fields. The +-- token is copied as-is: it is already encrypted at rest in the config. +-- INSERT OR IGNORE guards the unique index for accounts that somehow kept +-- multiple config rows with identical connection values (configs has no +-- per-user unique constraint). Active configs only: the app reads the +-- active config, so seeding from a stale inactive row would elect a primary +-- source the user no longer uses. +INSERT OR IGNORE INTO `sources` (`id`, `user_id`, `name`, `provider`, `url`, `username`, `token`, `enabled`, `created_at`, `updated_at`) +WITH `connection` AS ( + SELECT + `configs`.`user_id` AS `user_id`, + CASE json_extract(`configs`.`github_config`, '$.provider') + WHEN 'gitlab' THEN 'gitlab' + WHEN 'gitea' THEN 'gitea' + WHEN 'forgejo' THEN 'gitea' + WHEN 'codeberg' THEN 'gitea' + WHEN 'github' THEN 'github' + ELSE 'github' + END AS `provider`, + rtrim(COALESCE(json_extract(`configs`.`github_config`, '$.url'), ''), '/') AS `url`, + COALESCE(json_extract(`configs`.`github_config`, '$.owner'), '') AS `username`, + COALESCE(json_extract(`configs`.`github_config`, '$.token'), '') AS `token` + FROM `configs` + WHERE `configs`.`is_active` = 1 + AND ( + COALESCE(json_extract(`configs`.`github_config`, '$.token'), '') <> '' + OR COALESCE(json_extract(`configs`.`github_config`, '$.owner'), '') <> '' + ) +) +SELECT + lower(hex(randomblob(4))) || '-' || lower(hex(randomblob(2))) || '-' || lower(hex(randomblob(2))) || '-' || lower(hex(randomblob(2))) || '-' || lower(hex(randomblob(6))), + `user_id`, + CASE `provider` + WHEN 'github' THEN 'GitHub' + WHEN 'gitlab' THEN 'GitLab' + ELSE 'Gitea / Forgejo' + END || (CASE WHEN `username` = '' THEN '' ELSE ' (' || `username` || ')' END), + `provider`, + CASE + WHEN `url` <> '' THEN `url` + WHEN `provider` = 'github' THEN 'https://github.com' + WHEN `provider` = 'gitlab' THEN 'https://gitlab.com' + ELSE 'https://codeberg.org' + END, + `username`, + NULLIF(`token`, ''), + 1, + unixepoch(), + unixepoch() +FROM `connection`;--> statement-breakpoint +ALTER TABLE `repositories` ADD `source_id` text;--> statement-breakpoint +-- Link each repository to its user's source row with the same provider and +-- host. MIN() keeps the subquery scalar; rows without a match keep NULL +-- ("switched source" legacy rows that deliberately match no source). +UPDATE `repositories` SET `source_id` = ( + SELECT MIN(`sources`.`id`) + FROM `sources` + WHERE `sources`.`user_id` = `repositories`.`user_id` + AND `sources`.`provider` = `repositories`.`source_provider` + AND rtrim(`sources`.`url`, '/') = rtrim(`repositories`.`source_url`, '/') +) +WHERE `source_id` IS NULL;--> statement-breakpoint +DROP INDEX `uniq_repositories_user_full_name`;--> statement-breakpoint +CREATE UNIQUE INDEX `uniq_repositories_user_full_name` ON `repositories` (`user_id`,`source_id`,`full_name`);--> statement-breakpoint +DROP INDEX `uniq_repositories_user_normalized_full_name`;--> statement-breakpoint +CREATE UNIQUE INDEX `uniq_repositories_user_normalized_full_name` ON `repositories` (`user_id`,`source_id`,`normalized_full_name`); diff --git a/drizzle/meta/0019_snapshot.json b/drizzle/meta/0019_snapshot.json new file mode 100644 index 0000000..b4be405 --- /dev/null +++ b/drizzle/meta/0019_snapshot.json @@ -0,0 +1,3138 @@ +{ + "version": "6", + "dialect": "sqlite", + "id": "01e538bc-a176-4a53-a834-4212cd9045e1", + "prevId": "82ddc08f-0de7-43ec-8525-91f4706a2b35", + "tables": { + "accounts": { + "name": "accounts", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "account_id": { + "name": "account_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "provider_id": { + "name": "provider_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "issuer": { + "name": "issuer", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "provider_user_id": { + "name": "provider_user_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "access_token": { + "name": "access_token", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "refresh_token": { + "name": "refresh_token", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "id_token": { + "name": "id_token", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "access_token_expires_at": { + "name": "access_token_expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "refresh_token_expires_at": { + "name": "refresh_token_expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "scope": { + "name": "scope", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "password": { + "name": "password", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_accounts_account_id": { + "name": "idx_accounts_account_id", + "columns": [ + "account_id" + ], + "isUnique": false + }, + "idx_accounts_user_id": { + "name": "idx_accounts_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + }, + "idx_accounts_provider": { + "name": "idx_accounts_provider", + "columns": [ + "provider_id", + "provider_user_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "accounts_user_id_users_id_fk": { + "name": "accounts_user_id_users_id_fk", + "tableFrom": "accounts", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "api_keys": { + "name": "api_keys", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "config_id": { + "name": "config_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'default'" + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "start": { + "name": "start", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "prefix": { + "name": "prefix", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "key": { + "name": "key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "reference_id": { + "name": "reference_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "refill_interval": { + "name": "refill_interval", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "refill_amount": { + "name": "refill_amount", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_refill_at": { + "name": "last_refill_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "rate_limit_enabled": { + "name": "rate_limit_enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "rate_limit_time_window": { + "name": "rate_limit_time_window", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "rate_limit_max": { + "name": "rate_limit_max", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "request_count": { + "name": "request_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "remaining": { + "name": "remaining", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_request": { + "name": "last_request", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "permissions": { + "name": "permissions", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "metadata": { + "name": "metadata", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "idx_api_keys_reference_id": { + "name": "idx_api_keys_reference_id", + "columns": [ + "reference_id" + ], + "isUnique": false + }, + "idx_api_keys_key": { + "name": "idx_api_keys_key", + "columns": [ + "key" + ], + "isUnique": false + } + }, + "foreignKeys": { + "api_keys_reference_id_users_id_fk": { + "name": "api_keys_reference_id_users_id_fk", + "tableFrom": "api_keys", + "tableTo": "users", + "columnsFrom": [ + "reference_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "configs": { + "name": "configs", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "is_active": { + "name": "is_active", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "github_config": { + "name": "github_config", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "gitea_config": { + "name": "gitea_config", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "include": { + "name": "include", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[\"*\"]'" + }, + "exclude": { + "name": "exclude", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[]'" + }, + "schedule_config": { + "name": "schedule_config", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "cleanup_config": { + "name": "cleanup_config", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "notification_config": { + "name": "notification_config", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{\"enabled\":false,\"provider\":\"ntfy\",\"notifyOnSyncError\":true,\"notifyOnSyncSuccess\":false,\"notifyOnNewRepo\":false}'" + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": {}, + "foreignKeys": { + "configs_user_id_users_id_fk": { + "name": "configs_user_id_users_id_fk", + "tableFrom": "configs", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "events": { + "name": "events", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "channel": { + "name": "channel", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "payload": { + "name": "payload", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "read": { + "name": "read", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_events_user_channel": { + "name": "idx_events_user_channel", + "columns": [ + "user_id", + "channel" + ], + "isUnique": false + }, + "idx_events_created_at": { + "name": "idx_events_created_at", + "columns": [ + "created_at" + ], + "isUnique": false + }, + "idx_events_read": { + "name": "idx_events_read", + "columns": [ + "read" + ], + "isUnique": false + } + }, + "foreignKeys": { + "events_user_id_users_id_fk": { + "name": "events_user_id_users_id_fk", + "tableFrom": "events", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "jwks": { + "name": "jwks", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "public_key": { + "name": "public_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "private_key": { + "name": "private_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "alg": { + "name": "alg", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "crv": { + "name": "crv", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "mirror_jobs": { + "name": "mirror_jobs", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "repository_id": { + "name": "repository_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "repository_name": { + "name": "repository_name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "organization_name": { + "name": "organization_name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "details": { + "name": "details", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'imported'" + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "timestamp": { + "name": "timestamp", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "job_type": { + "name": "job_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'mirror'" + }, + "batch_id": { + "name": "batch_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "total_items": { + "name": "total_items", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "completed_items": { + "name": "completed_items", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": 0 + }, + "item_ids": { + "name": "item_ids", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "completed_item_ids": { + "name": "completed_item_ids", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "'[]'" + }, + "in_progress": { + "name": "in_progress", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "started_at": { + "name": "started_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "completed_at": { + "name": "completed_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_checkpoint": { + "name": "last_checkpoint", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "idx_mirror_jobs_user_id": { + "name": "idx_mirror_jobs_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + }, + "idx_mirror_jobs_batch_id": { + "name": "idx_mirror_jobs_batch_id", + "columns": [ + "batch_id" + ], + "isUnique": false + }, + "idx_mirror_jobs_in_progress": { + "name": "idx_mirror_jobs_in_progress", + "columns": [ + "in_progress" + ], + "isUnique": false + }, + "idx_mirror_jobs_job_type": { + "name": "idx_mirror_jobs_job_type", + "columns": [ + "job_type" + ], + "isUnique": false + }, + "idx_mirror_jobs_timestamp": { + "name": "idx_mirror_jobs_timestamp", + "columns": [ + "timestamp" + ], + "isUnique": false + } + }, + "foreignKeys": { + "mirror_jobs_user_id_users_id_fk": { + "name": "mirror_jobs_user_id_users_id_fk", + "tableFrom": "mirror_jobs", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "oauth_access_tokens": { + "name": "oauth_access_tokens", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "token": { + "name": "token", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "client_id": { + "name": "client_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "session_id": { + "name": "session_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "reference_id": { + "name": "reference_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "refresh_id": { + "name": "refresh_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + }, + "scopes": { + "name": "scopes", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "authorization_code_id": { + "name": "authorization_code_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "resources": { + "name": "resources", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "requested_user_info_claims": { + "name": "requested_user_info_claims", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "revoked": { + "name": "revoked", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "confirmation": { + "name": "confirmation", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "oauth_access_tokens_token_unique": { + "name": "oauth_access_tokens_token_unique", + "columns": [ + "token" + ], + "isUnique": true + }, + "idx_oauth_access_tokens_token": { + "name": "idx_oauth_access_tokens_token", + "columns": [ + "token" + ], + "isUnique": false + }, + "idx_oauth_access_tokens_client_id": { + "name": "idx_oauth_access_tokens_client_id", + "columns": [ + "client_id" + ], + "isUnique": false + }, + "idx_oauth_access_tokens_user_id": { + "name": "idx_oauth_access_tokens_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "oauth_access_tokens_user_id_users_id_fk": { + "name": "oauth_access_tokens_user_id_users_id_fk", + "tableFrom": "oauth_access_tokens", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "oauth_client_assertions": { + "name": "oauth_client_assertions", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "oauth_client_resources": { + "name": "oauth_client_resources", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "client_id": { + "name": "client_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "resource_id": { + "name": "resource_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "metadata": { + "name": "metadata", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_oauth_client_resources_client_id": { + "name": "idx_oauth_client_resources_client_id", + "columns": [ + "client_id" + ], + "isUnique": false + }, + "idx_oauth_client_resources_resource_id": { + "name": "idx_oauth_client_resources_resource_id", + "columns": [ + "resource_id" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "oauth_clients": { + "name": "oauth_clients", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "client_id": { + "name": "client_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "client_secret": { + "name": "client_secret", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "disabled": { + "name": "disabled", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": false + }, + "skip_consent": { + "name": "skip_consent", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "enable_end_session": { + "name": "enable_end_session", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "subject_type": { + "name": "subject_type", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "scopes": { + "name": "scopes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "uri": { + "name": "uri", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "icon": { + "name": "icon", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "contacts": { + "name": "contacts", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tos": { + "name": "tos", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "policy": { + "name": "policy", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "software_id": { + "name": "software_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "software_version": { + "name": "software_version", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "software_statement": { + "name": "software_statement", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "redirect_uris": { + "name": "redirect_uris", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "post_logout_redirect_uris": { + "name": "post_logout_redirect_uris", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "token_endpoint_auth_method": { + "name": "token_endpoint_auth_method", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "grant_types": { + "name": "grant_types", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "response_types": { + "name": "response_types", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "public": { + "name": "public", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "require_pkce": { + "name": "require_pkce", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "client_discovery_id": { + "name": "client_discovery_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "client_credentials_scopes": { + "name": "client_credentials_scopes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "backchannel_logout_uri": { + "name": "backchannel_logout_uri", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "backchannel_logout_session_required": { + "name": "backchannel_logout_session_required", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "application_type": { + "name": "application_type", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "jwks": { + "name": "jwks", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "jwks_uri": { + "name": "jwks_uri", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "dpop_bound_access_tokens": { + "name": "dpop_bound_access_tokens", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "reference_id": { + "name": "reference_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "metadata": { + "name": "metadata", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "oauth_clients_client_id_unique": { + "name": "oauth_clients_client_id_unique", + "columns": [ + "client_id" + ], + "isUnique": true + }, + "idx_oauth_clients_client_id": { + "name": "idx_oauth_clients_client_id", + "columns": [ + "client_id" + ], + "isUnique": false + }, + "idx_oauth_clients_user_id": { + "name": "idx_oauth_clients_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "oauth_clients_user_id_users_id_fk": { + "name": "oauth_clients_user_id_users_id_fk", + "tableFrom": "oauth_clients", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "oauth_consents": { + "name": "oauth_consents", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "client_id": { + "name": "client_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "reference_id": { + "name": "reference_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "scopes": { + "name": "scopes", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "resources": { + "name": "resources", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "requested_user_info_claims": { + "name": "requested_user_info_claims", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_oauth_consents_client_id": { + "name": "idx_oauth_consents_client_id", + "columns": [ + "client_id" + ], + "isUnique": false + }, + "idx_oauth_consents_user_id": { + "name": "idx_oauth_consents_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "oauth_consents_user_id_users_id_fk": { + "name": "oauth_consents_user_id_users_id_fk", + "tableFrom": "oauth_consents", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "oauth_refresh_tokens": { + "name": "oauth_refresh_tokens", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "token": { + "name": "token", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "client_id": { + "name": "client_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "session_id": { + "name": "session_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "reference_id": { + "name": "reference_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + }, + "revoked": { + "name": "revoked", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "auth_time": { + "name": "auth_time", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "scopes": { + "name": "scopes", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "authorization_code_id": { + "name": "authorization_code_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "resources": { + "name": "resources", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "requested_user_info_claims": { + "name": "requested_user_info_claims", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "rotated_at": { + "name": "rotated_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "rotation_replay_response": { + "name": "rotation_replay_response", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "rotation_replay_expires_at": { + "name": "rotation_replay_expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "confirmation": { + "name": "confirmation", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "oauth_refresh_tokens_token_unique": { + "name": "oauth_refresh_tokens_token_unique", + "columns": [ + "token" + ], + "isUnique": true + }, + "idx_oauth_refresh_tokens_token": { + "name": "idx_oauth_refresh_tokens_token", + "columns": [ + "token" + ], + "isUnique": false + }, + "idx_oauth_refresh_tokens_client_id": { + "name": "idx_oauth_refresh_tokens_client_id", + "columns": [ + "client_id" + ], + "isUnique": false + }, + "idx_oauth_refresh_tokens_user_id": { + "name": "idx_oauth_refresh_tokens_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "oauth_refresh_tokens_user_id_users_id_fk": { + "name": "oauth_refresh_tokens_user_id_users_id_fk", + "tableFrom": "oauth_refresh_tokens", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "oauth_resources": { + "name": "oauth_resources", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "identifier": { + "name": "identifier", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "access_token_ttl": { + "name": "access_token_ttl", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "refresh_token_ttl": { + "name": "refresh_token_ttl", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "signing_algorithm": { + "name": "signing_algorithm", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "signing_key_id": { + "name": "signing_key_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "allowed_scopes": { + "name": "allowed_scopes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "custom_claims": { + "name": "custom_claims", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "dpop_bound_access_tokens_required": { + "name": "dpop_bound_access_tokens_required", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "disabled": { + "name": "disabled", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "policy_version": { + "name": "policy_version", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "metadata": { + "name": "metadata", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "oauth_resources_identifier_unique": { + "name": "oauth_resources_identifier_unique", + "columns": [ + "identifier" + ], + "isUnique": true + }, + "idx_oauth_resources_identifier": { + "name": "idx_oauth_resources_identifier", + "columns": [ + "identifier" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "organizations": { + "name": "organizations", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "config_id": { + "name": "config_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "normalized_name": { + "name": "normalized_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "avatar_url": { + "name": "avatar_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "membership_role": { + "name": "membership_role", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'member'" + }, + "is_included": { + "name": "is_included", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "destination_org": { + "name": "destination_org", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "mirror_overrides": { + "name": "mirror_overrides", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'imported'" + }, + "last_mirrored": { + "name": "last_mirrored", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "error_message": { + "name": "error_message", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "repository_count": { + "name": "repository_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "public_repository_count": { + "name": "public_repository_count", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "private_repository_count": { + "name": "private_repository_count", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "fork_repository_count": { + "name": "fork_repository_count", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_organizations_user_id": { + "name": "idx_organizations_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + }, + "idx_organizations_config_id": { + "name": "idx_organizations_config_id", + "columns": [ + "config_id" + ], + "isUnique": false + }, + "idx_organizations_status": { + "name": "idx_organizations_status", + "columns": [ + "status" + ], + "isUnique": false + }, + "idx_organizations_is_included": { + "name": "idx_organizations_is_included", + "columns": [ + "is_included" + ], + "isUnique": false + }, + "uniq_organizations_user_normalized_name": { + "name": "uniq_organizations_user_normalized_name", + "columns": [ + "user_id", + "normalized_name" + ], + "isUnique": true + } + }, + "foreignKeys": { + "organizations_user_id_users_id_fk": { + "name": "organizations_user_id_users_id_fk", + "tableFrom": "organizations", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "organizations_config_id_configs_id_fk": { + "name": "organizations_config_id_configs_id_fk", + "tableFrom": "organizations", + "tableTo": "configs", + "columnsFrom": [ + "config_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "rate_limits": { + "name": "rate_limits", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'github'" + }, + "limit": { + "name": "limit", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "remaining": { + "name": "remaining", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "used": { + "name": "used", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "reset": { + "name": "reset", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "retry_after": { + "name": "retry_after", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'ok'" + }, + "last_checked": { + "name": "last_checked", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_rate_limits_user_provider": { + "name": "idx_rate_limits_user_provider", + "columns": [ + "user_id", + "provider" + ], + "isUnique": false + }, + "idx_rate_limits_status": { + "name": "idx_rate_limits_status", + "columns": [ + "status" + ], + "isUnique": false + } + }, + "foreignKeys": { + "rate_limits_user_id_users_id_fk": { + "name": "rate_limits_user_id_users_id_fk", + "tableFrom": "rate_limits", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "repositories": { + "name": "repositories", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "config_id": { + "name": "config_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "full_name": { + "name": "full_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "normalized_full_name": { + "name": "normalized_full_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "url": { + "name": "url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "clone_url": { + "name": "clone_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "source_provider": { + "name": "source_provider", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'github'" + }, + "source_url": { + "name": "source_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'https://github.com'" + }, + "source_id": { + "name": "source_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "destination_provider": { + "name": "destination_provider", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'gitea'" + }, + "destination_url": { + "name": "destination_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "owner": { + "name": "owner", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "organization": { + "name": "organization", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "mirrored_location": { + "name": "mirrored_location", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "''" + }, + "is_private": { + "name": "is_private", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "is_fork": { + "name": "is_fork", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "forked_from": { + "name": "forked_from", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "has_issues": { + "name": "has_issues", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "is_starred": { + "name": "is_starred", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "is_archived": { + "name": "is_archived", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "size": { + "name": "size", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "has_lfs": { + "name": "has_lfs", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "has_submodules": { + "name": "has_submodules", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "language": { + "name": "language", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "default_branch": { + "name": "default_branch", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "visibility": { + "name": "visibility", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'public'" + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'imported'" + }, + "last_mirrored": { + "name": "last_mirrored", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "error_message": { + "name": "error_message", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "destination_org": { + "name": "destination_org", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "mirror_overrides": { + "name": "mirror_overrides", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "metadata": { + "name": "metadata", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "imported_at": { + "name": "imported_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_repositories_user_id": { + "name": "idx_repositories_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + }, + "idx_repositories_config_id": { + "name": "idx_repositories_config_id", + "columns": [ + "config_id" + ], + "isUnique": false + }, + "idx_repositories_status": { + "name": "idx_repositories_status", + "columns": [ + "status" + ], + "isUnique": false + }, + "idx_repositories_owner": { + "name": "idx_repositories_owner", + "columns": [ + "owner" + ], + "isUnique": false + }, + "idx_repositories_organization": { + "name": "idx_repositories_organization", + "columns": [ + "organization" + ], + "isUnique": false + }, + "idx_repositories_is_fork": { + "name": "idx_repositories_is_fork", + "columns": [ + "is_fork" + ], + "isUnique": false + }, + "idx_repositories_is_starred": { + "name": "idx_repositories_is_starred", + "columns": [ + "is_starred" + ], + "isUnique": false + }, + "idx_repositories_user_imported_at": { + "name": "idx_repositories_user_imported_at", + "columns": [ + "user_id", + "imported_at" + ], + "isUnique": false + }, + "uniq_repositories_user_full_name": { + "name": "uniq_repositories_user_full_name", + "columns": [ + "user_id", + "source_id", + "full_name" + ], + "isUnique": true + }, + "uniq_repositories_user_normalized_full_name": { + "name": "uniq_repositories_user_normalized_full_name", + "columns": [ + "user_id", + "source_id", + "normalized_full_name" + ], + "isUnique": true + }, + "idx_repositories_mirrored_location": { + "name": "idx_repositories_mirrored_location", + "columns": [ + "user_id", + "mirrored_location" + ], + "isUnique": false + } + }, + "foreignKeys": { + "repositories_user_id_users_id_fk": { + "name": "repositories_user_id_users_id_fk", + "tableFrom": "repositories", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "repositories_config_id_configs_id_fk": { + "name": "repositories_config_id_configs_id_fk", + "tableFrom": "repositories", + "tableTo": "configs", + "columnsFrom": [ + "config_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "sessions": { + "name": "sessions", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "token": { + "name": "token", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ip_address": { + "name": "ip_address", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "user_agent": { + "name": "user_agent", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "sessions_token_unique": { + "name": "sessions_token_unique", + "columns": [ + "token" + ], + "isUnique": true + }, + "idx_sessions_user_id": { + "name": "idx_sessions_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + }, + "idx_sessions_token": { + "name": "idx_sessions_token", + "columns": [ + "token" + ], + "isUnique": false + }, + "idx_sessions_expires_at": { + "name": "idx_sessions_expires_at", + "columns": [ + "expires_at" + ], + "isUnique": false + } + }, + "foreignKeys": { + "sessions_user_id_users_id_fk": { + "name": "sessions_user_id_users_id_fk", + "tableFrom": "sessions", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "sources": { + "name": "sources", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "url": { + "name": "url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "username": { + "name": "username", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "token": { + "name": "token", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_sources_user_id": { + "name": "idx_sources_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + }, + "uniq_sources_user_provider_url_username": { + "name": "uniq_sources_user_provider_url_username", + "columns": [ + "user_id", + "provider", + "url", + "username" + ], + "isUnique": true + } + }, + "foreignKeys": { + "sources_user_id_users_id_fk": { + "name": "sources_user_id_users_id_fk", + "tableFrom": "sources", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "sso_providers": { + "name": "sso_providers", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "issuer": { + "name": "issuer", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "domain": { + "name": "domain", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "oidc_config": { + "name": "oidc_config", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "saml_config": { + "name": "saml_config", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "domain_verified": { + "name": "domain_verified", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "provider_id": { + "name": "provider_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "sso_providers_provider_id_unique": { + "name": "sso_providers_provider_id_unique", + "columns": [ + "provider_id" + ], + "isUnique": true + }, + "idx_sso_providers_provider_id": { + "name": "idx_sso_providers_provider_id", + "columns": [ + "provider_id" + ], + "isUnique": false + }, + "idx_sso_providers_domain": { + "name": "idx_sso_providers_domain", + "columns": [ + "domain" + ], + "isUnique": false + }, + "idx_sso_providers_issuer": { + "name": "idx_sso_providers_issuer", + "columns": [ + "issuer" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "users": { + "name": "users", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "email": { + "name": "email", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "email_verified": { + "name": "email_verified", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "image": { + "name": "image", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "username": { + "name": "username", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "users_email_unique": { + "name": "users_email_unique", + "columns": [ + "email" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "verification_tokens": { + "name": "verification_tokens", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "token": { + "name": "token", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "identifier": { + "name": "identifier", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "verification_tokens_token_unique": { + "name": "verification_tokens_token_unique", + "columns": [ + "token" + ], + "isUnique": true + }, + "idx_verification_tokens_token": { + "name": "idx_verification_tokens_token", + "columns": [ + "token" + ], + "isUnique": false + }, + "idx_verification_tokens_identifier": { + "name": "idx_verification_tokens_identifier", + "columns": [ + "identifier" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "verifications": { + "name": "verifications", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "identifier": { + "name": "identifier", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "value": { + "name": "value", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_verifications_identifier": { + "name": "idx_verifications_identifier", + "columns": [ + "identifier" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + } + }, + "views": {}, + "enums": {}, + "_meta": { + "schemas": {}, + "tables": {}, + "columns": {} + }, + "internal": { + "indexes": {} + } +} \ No newline at end of file diff --git a/drizzle/meta/_journal.json b/drizzle/meta/_journal.json index 766be79..109d40f 100644 --- a/drizzle/meta/_journal.json +++ b/drizzle/meta/_journal.json @@ -134,6 +134,13 @@ "when": 1788350408052, "tag": "0018_destination_columns", "breakpoints": true + }, + { + "idx": 19, + "version": "6", + "when": 1788350408053, + "tag": "0019_multi_source", + "breakpoints": true } ] } \ No newline at end of file diff --git a/scripts/validate-migrations.ts b/scripts/validate-migrations.ts index 27f29fd..d2e4dd8 100644 --- a/scripts/validate-migrations.ts +++ b/scripts/validate-migrations.ts @@ -657,6 +657,149 @@ function verify0018Migration(db: any) { ); } +function seedPre0019Database(db: any) { + // Migrations 0000-0018 have run; the sources table does not exist yet. + // Seed one user per connection flavor: a plain GitHub config, a GitLab + // config without an instance URL (provider default must apply), a + // "forgejo" alias with a trailing-slash URL (must normalize to the gitea + // kind and the slashed URL), and an empty config (must create no source). + for (const [id, email, name] of [ + ["u-gh", "gh19@example.com", "GitHub Nineteen"], + ["u-gl", "gl19@example.com", "GitLab Nineteen"], + ["u-fj", "fj19@example.com", "Forgejo Nineteen"], + ["u-none", "none19@example.com", "No Source Nineteen"], + ] as const) { + db.run("INSERT INTO users (id, email, username, name) VALUES (?, ?, ?, ?)", [id, email, id.slice(2), name]); + } + + db.run( + "INSERT INTO configs (id, user_id, name, is_active, github_config, gitea_config, schedule_config, cleanup_config) " + + "VALUES ('cfg-gh19', 'u-gh', 'Default', 1, '{\"provider\":\"github\",\"owner\":\"octocat\",\"token\":\"gh-encrypted-token\",\"type\":\"personal\"}', '{}', '{}', '{}')", + ); + db.run( + "INSERT INTO configs (id, user_id, name, is_active, github_config, gitea_config, schedule_config, cleanup_config) " + + "VALUES ('cfg-gl19', 'u-gl', 'Default', 1, '{\"provider\":\"gitlab\",\"owner\":\"greta\",\"token\":\"gl-encrypted-token\",\"type\":\"personal\"}', '{}', '{}', '{}')", + ); + db.run( + "INSERT INTO configs (id, user_id, name, is_active, github_config, gitea_config, schedule_config, cleanup_config) " + + "VALUES ('cfg-fj19', 'u-fj', 'Default', 1, '{\"provider\":\"forgejo\",\"owner\":\"cbuser\",\"url\":\"https://codeberg.org/\",\"token\":\"\",\"type\":\"personal\"}', '{}', '{}', '{}')", + ); + db.run( + "INSERT INTO configs (id, user_id, name, is_active, github_config, gitea_config, schedule_config, cleanup_config) " + + "VALUES ('cfg-none19', 'u-none', 'Default', 1, '{\"owner\":\"\",\"token\":\"\"}', '{}', '{}', '{}')", + ); + + const insertRepo = (id: string, userId: string, configId: string, fullName: string, provider: string, url: string) => + db.run( + "INSERT INTO repositories (id, user_id, config_id, name, full_name, normalized_full_name, url, clone_url, owner, default_branch, source_provider, source_url) " + + `VALUES ('${id}', '${userId}', '${configId}', '${fullName.split("/")[1]}', '${fullName}', '${fullName}', 'https://example.com/${fullName}', 'https://example.com/${fullName}.git', '${fullName.split("/")[0]}', 'main', '${provider}', '${url}')`, + ); + + // Matches the config's connection -> backfilled. + insertRepo("repo-gh19", "u-gh", "cfg-gh19", "octocat/tool", "github", "https://github.com"); + // From a host the user has no source for -> stays NULL ("switched source"). + insertRepo("repo-other19", "u-gh", "cfg-gh19", "octocat/legacy", "gitlab", "https://gitlab.com"); + // Trailing slash in the repo's source_url still matches the trimmed source url. + insertRepo("repo-cb19", "u-fj", "cfg-fj19", "cbuser/tool", "gitea", "https://codeberg.org/"); +} + +function verify0019Migration(db: any) { + // The sources table and its indexes. + const sourceCount = db.query("SELECT COUNT(*) AS count FROM sources").get() as { count: number }; + assert(sourceCount.count === 3, `Expected 3 seeded source rows, got ${sourceCount.count}`); + for (const indexName of ["idx_sources_user_id", "uniq_sources_user_provider_url_username"]) { + const index = db + .query("SELECT name FROM sqlite_master WHERE type='index' AND name = ?") + .get(indexName) as { name: string } | null; + assert(index, `Expected ${indexName} index to exist after migration`); + } + + const byUser = Object.fromEntries( + (db.query("SELECT user_id, name, provider, url, username, token, enabled FROM sources").all() as Array<{ + user_id: string; + name: string; + provider: string; + url: string; + username: string; + token: string | null; + enabled: number; + }>).map((row) => [row.user_id, row]), + ); + + assert( + byUser["u-gh"]?.provider === "github" && + byUser["u-gh"].url === "https://github.com" && + byUser["u-gh"].username === "octocat" && + byUser["u-gh"].token === "gh-encrypted-token" && + byUser["u-gh"].name === "GitHub (octocat)" && + byUser["u-gh"].enabled === 1, + `Expected the GitHub config to seed a matching source row, got ${JSON.stringify(byUser["u-gh"])}`, + ); + assert( + byUser["u-gl"]?.provider === "gitlab" && byUser["u-gl"].url === "https://gitlab.com", + `Expected a GitLab config without a URL to take the provider default, got ${JSON.stringify(byUser["u-gl"])}`, + ); + assert( + byUser["u-fj"]?.provider === "gitea" && + byUser["u-fj"].url === "https://codeberg.org" && + byUser["u-fj"].token === null && + byUser["u-fj"].name === "Gitea / Forgejo (cbuser)", + `Expected the forgejo alias to normalize to gitea with a trimmed URL and NULL token, got ${JSON.stringify(byUser["u-fj"])}`, + ); + assert(!byUser["u-none"], "Expected an empty config to seed no source row"); + + // The repositories.source_id column exists, nullable, no default. + const repoCols = db.query("PRAGMA table_info(repositories)").all() as TableInfoRow[]; + const sourceIdCol = repoCols.find((column) => column.name === "source_id"); + assert(sourceIdCol, "Expected repositories.source_id column to exist after migration"); + assert(sourceIdCol.notnull === 0, "Expected repositories.source_id to be nullable"); + assert(sourceIdCol.dflt_value === null, `Expected repositories.source_id to have no default, got ${sourceIdCol.dflt_value}`); + + // Backfill: matching repos linked, unmatched legacy repo left NULL. + const linked = db + .query( + "SELECT repositories.source_id FROM repositories JOIN sources ON sources.id = repositories.source_id WHERE repositories.id = 'repo-gh19'", + ) + .get() as { source_id: string } | null; + assert(linked, "Expected the GitHub repository to be linked to its source row"); + const cbLinked = db + .query( + "SELECT repositories.source_id FROM repositories JOIN sources ON sources.id = repositories.source_id WHERE repositories.id = 'repo-cb19'", + ) + .get() as { source_id: string } | null; + assert(cbLinked, "Expected the trailing-slash Codeberg repository to be linked to its source row"); + const unlinked = db + .query("SELECT source_id FROM repositories WHERE id = 'repo-other19'") + .get() as { source_id: string | null } | null; + assert(unlinked?.source_id === null, `Expected a repository from an unconnected host to keep NULL source_id, got ${JSON.stringify(unlinked)}`); + + // Full names are now unique per source: the same owner/repo can exist once + // per connected source, and still only once per source. + db.run( + "INSERT INTO sources (id, user_id, name, provider, url, username) VALUES ('src-gl19', 'u-gh', 'GitLab (octocat)', 'gitlab', 'https://gitlab.com', 'octocat')", + ); + const ghSourceId = (db.query("SELECT id FROM sources WHERE user_id = 'u-gh' AND provider = 'github'").get() as { id: string }).id; + db.run( + "INSERT INTO repositories (id, user_id, config_id, name, full_name, normalized_full_name, url, clone_url, owner, default_branch, source_id) " + + `VALUES ('repo-dup-gh', 'u-gh', 'cfg-gh19', 'dup-tool', 'octocat/dup-tool', 'octocat/dup-tool', 'https://github.com/octocat/dup-tool', 'https://github.com/octocat/dup-tool.git', 'octocat', 'main', '${ghSourceId}')`, + ); + db.run( + "INSERT INTO repositories (id, user_id, config_id, name, full_name, normalized_full_name, url, clone_url, owner, default_branch, source_id) " + + `VALUES ('repo-dup-gl', 'u-gh', 'cfg-gh19', 'dup-tool', 'octocat/dup-tool', 'octocat/dup-tool', 'https://gitlab.com/octocat/dup-tool', 'https://gitlab.com/octocat/dup-tool.git', 'octocat', 'main', 'src-gl19')`, + ); + + let rejected = false; + try { + db.run( + "INSERT INTO repositories (id, user_id, config_id, name, full_name, normalized_full_name, url, clone_url, owner, default_branch, source_id) " + + `VALUES ('repo-dup-gh2', 'u-gh', 'cfg-gh19', 'dup-tool', 'octocat/dup-tool', 'octocat/dup-tool', 'https://github.com/octocat/dup-tool', 'https://github.com/octocat/dup-tool.git', 'octocat', 'main', '${ghSourceId}')`, + ); + } catch { + rejected = true; + } + assert(rejected, "Expected the same full name under the same source to still be rejected"); +} + const MIGRATION_0012_TIMESTAMP = 1774062000000; const MIGRATION_0013_TIMESTAMP = 1780377747526; @@ -856,6 +999,10 @@ const latestUpgradeFixtures: Record = { seed: seedPre0018Database, verify: verify0018Migration, }, + "0019_multi_source": { + seed: seedPre0019Database, + verify: verify0019Migration, + }, }; function lintMigrations(selectedMigrations: Migration[]) { diff --git a/src/components/config/ConfigTabs.tsx b/src/components/config/ConfigTabs.tsx index 0752945..30b2eca 100644 --- a/src/components/config/ConfigTabs.tsx +++ b/src/components/config/ConfigTabs.tsx @@ -1,6 +1,7 @@ import { useEffect, useState, useCallback, useRef } from 'react'; import { GitHubConfigForm } from './GitHubConfigForm'; import { GiteaConfigForm } from './GiteaConfigForm'; +import { SourcesCard } from './SourcesCard'; import { GitHubMirrorSettings } from './GitHubMirrorSettings'; import { AutomationSettings } from './AutomationSettings'; import { SSOSettings } from './SSOSettings'; @@ -18,6 +19,7 @@ import type { AdvancedOptions, NotificationConfig, ConfigLockState, + SourceApiRecord, } from '@/types/config'; import { Button } from '../ui/button'; import { useAuth } from '@/hooks/useAuth'; @@ -113,6 +115,8 @@ export function ConfigTabs() { const [isSyncing, setIsSyncing] = useState(false); // Source and destination locks from the API; null until the config is loaded. const [locks, setLocks] = useState(null); + // Connected sources; the source of truth for the source side of the page. + const [sources, setSources] = useState([]); const [isAutoSavingSchedule, setIsAutoSavingSchedule] = useState(false); const [isAutoSavingCleanup, setIsAutoSavingCleanup] = useState(false); @@ -125,22 +129,28 @@ export function ConfigTabs() { const autoSaveGiteaTimeoutRef = useRef(null); const autoSaveNotificationTimeoutRef = useRef(null); + // Sources are the source of truth for the source connection; the legacy + // githubConfig check only covers APIs that predate multi-source support. + const isSourceConnected = (): boolean => { + if (sources.length > 0) { + return sources.some((source) => Boolean(source.token && source.token.trim())); + } + const { githubConfig } = config; + return !!(githubConfig.username.trim() && githubConfig.token.trim()); + }; + const isConfigFormValid = (): boolean => { - const { githubConfig, giteaConfig } = config; - const isGitHubValid = !!( - githubConfig.username.trim() && githubConfig.token.trim() - ); + const { giteaConfig } = config; const isGiteaValid = !!( giteaConfig.url.trim() && giteaConfig.username.trim() && giteaConfig.token.trim() ); - return isGitHubValid && isGiteaValid; + return isSourceConnected() && isGiteaValid; }; const isGitHubConfigValid = (): boolean => { - const { githubConfig } = config; - return !!(githubConfig.username.trim() && githubConfig.token.trim()); + return isSourceConnected(); }; // Removed the problematic useEffect that was causing circular dependencies @@ -565,53 +575,58 @@ export function ConfigTabs() { }; }, []); + // Loads (or reloads) the config: fills the forms, locks and the sources + // list. SourcesCard calls this after every source mutation; only the + // initial load shows the page skeleton. + const loadConfig = useCallback(async () => { + if (!user?.id) return; + + try { + const response = await apiRequest( + `/config?userId=${user.id}`, + { method: 'GET' }, + ); + if (response && !response.error) { + setLocks(response.locks ?? null); + setSources(response.sources ?? []); + setConfig(prev => ({ + githubConfig: { + ...prev.githubConfig, + ...response.githubConfig, // Merge so a response without provider keeps the default + }, + giteaConfig: + response.giteaConfig || prev.giteaConfig, + scheduleConfig: + response.scheduleConfig || prev.scheduleConfig, + cleanupConfig: { + ...prev.cleanupConfig, + ...response.cleanupConfig, // Merge to preserve all fields + }, + mirrorOptions: { + ...prev.mirrorOptions, + ...response.mirrorOptions, // Merge to preserve all fields including new mirrorLFS + }, + advancedOptions: + response.advancedOptions || prev.advancedOptions, + notificationConfig: + (response as any).notificationConfig || prev.notificationConfig, + })); + + } + } catch (error) { + console.warn( + 'Could not fetch configuration, using defaults:', + error, + ); + } + }, [user?.id]); + useEffect(() => { if (!user?.id) return; - const fetchConfig = async () => { - setIsLoading(true); - try { - const response = await apiRequest( - `/config?userId=${user.id}`, - { method: 'GET' }, - ); - if (response && !response.error) { - setLocks(response.locks ?? null); - setConfig({ - githubConfig: { - ...config.githubConfig, - ...response.githubConfig, // Merge so a response without provider keeps the default - }, - giteaConfig: - response.giteaConfig || config.giteaConfig, - scheduleConfig: - response.scheduleConfig || config.scheduleConfig, - cleanupConfig: { - ...config.cleanupConfig, - ...response.cleanupConfig, // Merge to preserve all fields - }, - mirrorOptions: { - ...config.mirrorOptions, - ...response.mirrorOptions, // Merge to preserve all fields including new mirrorLFS - }, - advancedOptions: - response.advancedOptions || config.advancedOptions, - notificationConfig: - (response as any).notificationConfig || config.notificationConfig, - }); - - } - } catch (error) { - console.warn( - 'Could not fetch configuration, using defaults:', - error, - ); - } - setIsLoading(false); - }; - - fetchConfig(); - }, [user?.id]); // Only depend on user.id, not the entire user object + setIsLoading(true); + loadConfig().finally(() => setIsLoading(false)); + }, [loadConfig]); // Only depend on user.id, not the entire user object function ConfigCardSkeleton() { return ( @@ -799,7 +814,7 @@ export function ConfigTabs() { <> {/* Connection cards share a stretched row so they stay equal height */}
- +
diff --git a/src/components/config/SourcesCard.tsx b/src/components/config/SourcesCard.tsx new file mode 100644 index 0000000..009cbb5 --- /dev/null +++ b/src/components/config/SourcesCard.tsx @@ -0,0 +1,722 @@ +import React, { useState } from "react"; +import { Button } from "@/components/ui/button"; +import { Input } from "../ui/input"; +import { Label } from "../ui/label"; +import { + Dialog, + DialogContent, + DialogDescription, + DialogFooter, + DialogHeader, + DialogTitle, +} from "@/components/ui/dialog"; +import { + Activity, + ExternalLink, + KeyRound, + Layers, + Lock, + Pencil, + Plus, + PlugZap, + Trash2, +} from "lucide-react"; +import { SiGitea, SiGithub, SiGitlab } from "react-icons/si"; +import { + Select, + SelectContent, + SelectItem, + SelectTrigger, + SelectValue, +} from "@/components/ui/select"; +import { toast } from "sonner"; +import { githubApi } from "@/lib/api"; +import { withBase } from "@/lib/base-path"; +import type { SourceApiRecord, SourceProvider } from "@/types/config"; +import { + SOURCE_PROVIDER_DEFAULT_URLS, + SOURCE_PROVIDER_LABELS, + isBetaSourceProvider, +} from "@/lib/source-providers/kinds"; +import { invalidateConfigCache } from "@/hooks/useConfigStatus"; +import { HostLockNotice } from "./HostLockNotice"; +import { CardDivider, CardSection, SettingsCard } from "./settings-ui"; + +const SOURCES_API_PATH = withBase("/api/sources"); + +type SourceProviderMeta = { + value: SourceProvider; + label: string; + icon: React.ComponentType<{ className?: string }>; + usernamePlaceholder: string; + tokenPlaceholder: string; + tokenHint: string; + /** Appended to the instance URL to reach the token settings page. */ + tokenSettingsPath: string; + tokenSteps: string[]; + scopes: string[]; + /** Short pill shown next to the option, e.g. BETA. */ + badge?: string; +}; + +const sourceProviders: SourceProviderMeta[] = [ + { + value: "github", + label: SOURCE_PROVIDER_LABELS.github, + icon: SiGithub, + usernamePlaceholder: "Your GitHub username", + tokenPlaceholder: "Your GitHub token (classic)", + tokenHint: "Needed for private repos, organizations, and starred repositories", + tokenSettingsPath: "/settings/tokens", + tokenSteps: [ + "GitHub → Settings → Developer settings", + "Personal access tokens → Generate new token (classic)", + "Select the scopes below and paste the token here", + ], + scopes: ["repo", "admin:org"], + }, + { + value: "gitlab", + label: SOURCE_PROVIDER_LABELS.gitlab, + icon: SiGitlab, + badge: isBetaSourceProvider("gitlab") ? "BETA" : undefined, + usernamePlaceholder: "Your GitLab username", + tokenPlaceholder: "Your GitLab personal access token", + tokenHint: "Needed for private projects, groups, and starred projects", + tokenSettingsPath: "/-/user_settings/personal_access_tokens", + tokenSteps: [ + "GitLab → Preferences → Access tokens", + "Add new token with the scopes below", + "Paste the token here", + ], + scopes: ["read_api", "read_repository"], + }, + { + value: "gitea", + label: SOURCE_PROVIDER_LABELS.gitea, + icon: SiGitea, + badge: isBetaSourceProvider("gitea") ? "BETA" : undefined, + usernamePlaceholder: "Your Gitea or Forgejo username", + tokenPlaceholder: "Your access token", + tokenHint: "Needed for private repos, organizations, and starred repositories", + tokenSettingsPath: "/user/settings/applications", + tokenSteps: [ + "Gitea → Settings → Applications", + "Generate a token with the permissions below", + "Paste the token here", + ], + scopes: ["read:repository", "read:user", "read:organization"], + }, +]; + +function providerMetaFor(provider: SourceProvider): SourceProviderMeta { + return sourceProviders.find((option) => option.value === provider) ?? sourceProviders[0]; +} + +function providerHelp(provider: SourceProvider): string { + if (provider === "github") return "Where your repositories are pulled from"; + if (provider === "gitlab") + return "Beta. Code, tags, wiki and LFS are mirrored. Issues, merge requests and releases need a GitHub source."; + return "Beta. Code, tags, wiki and LFS are mirrored. Issues, pull requests and releases need a GitHub source."; +} + +function sourceHost(source: SourceApiRecord): string { + const url = (source.url ?? "").trim() || SOURCE_PROVIDER_DEFAULT_URLS[source.provider]; + return url.replace(/^https?:\/\//, "").replace(/\/+$/, ""); +} + +function repositoryNoun(count: number): string { + return `${count} ${count === 1 ? "repository" : "repositories"}`; +} + +const sourceChangeConsequences = [ + "Repositories already imported stay tied to the current source and keep syncing through Gitea.", + "Cleanup ignores them, and mirroring one of them again is refused until it is removed and added from the new source.", + "New imports come from the new source only.", +]; + +interface SourcesCardProps { + /** Connected sources from the config API; empty until one is added. */ + sources: SourceApiRecord[]; + /** Refetches the config so the list, locks and counts stay fresh. */ + onRefresh: () => Promise; +} + +type SourcesMutationResponse = { + success?: boolean; + message?: string; + /** Present when a locked source change needs explicit confirmation. */ + lock?: string; + source?: SourceApiRecord; +}; + +type EditorState = { + /** Source being edited; null when adding a new one. */ + source: SourceApiRecord | null; + provider: SourceProvider; + url: string; + username: string; + token: string; + /** Set once the lock dialog confirmed changing provider or instance URL. */ + sourceUnlocked: boolean; +}; + +/** + * The Connections tab source card: one row per connected source, with the + * add/edit form reusing the connection fields of the single-source era. + */ +export function SourcesCard({ sources, onRefresh }: SourcesCardProps) { + const [editor, setEditor] = useState(null); + const [isTesting, setIsTesting] = useState(false); + const [isSaving, setIsSaving] = useState(false); + const [isRemoving, setIsRemoving] = useState(false); + const [isRefreshing, setIsRefreshing] = useState(false); + const [confirmChangeMessage, setConfirmChangeMessage] = useState(null); + const [removeTarget, setRemoveTarget] = useState(null); + + const editing = editor?.source ?? null; + const editingLocked = Boolean(editing?.locked) && !editor?.sourceUnlocked; + const provider = editor?.provider ?? "github"; + const providerMeta = providerMetaFor(provider); + const providerLabel = providerMeta.label; + const defaultInstanceUrl = SOURCE_PROVIDER_DEFAULT_URLS[provider]; + const instanceUrl = (editor?.url ?? "").trim() || defaultInstanceUrl; + const tokenSettingsUrl = `${instanceUrl.replace(/\/+$/, "")}${providerMeta.tokenSettingsPath}`; + // An empty token field while editing keeps the stored one, so testing + // and saving both fall back to it. + const effectiveToken = editor?.token.trim() || editing?.token || ""; + + const refreshSources = async () => { + setIsRefreshing(true); + try { + await onRefresh(); + } finally { + setIsRefreshing(false); + } + }; + + // Raw fetch (like ConfigTabs's config save) so 409 bodies with a lock + // flag can be distinguished from plain validation errors. + const requestMutation = async ( + path: string, + init: RequestInit + ): Promise<{ ok: boolean; status: number; body: SourcesMutationResponse }> => { + const response = await fetch(`${SOURCES_API_PATH}${path}`, { + headers: { "Content-Type": "application/json" }, + ...init, + }); + const body = await response.json().catch(() => ({})); + return { ok: response.ok, status: response.status, body }; + }; + + const openAddEditor = () => { + setEditor({ + source: null, + provider: "github", + url: "", + username: "", + token: "", + sourceUnlocked: false, + }); + }; + + const openEditEditor = (source: SourceApiRecord) => { + setEditor({ + source, + provider: source.provider, + url: source.url ?? "", + username: source.username ?? "", + token: "", + sourceUnlocked: false, + }); + }; + + const handleProviderChange = (value: string) => { + setEditor((prev) => { + if (!prev) return prev; + const nextProvider = value as SourceProvider; + return { + ...prev, + provider: nextProvider, + }; + }); + }; + + const handleFieldChange = (e: React.ChangeEvent) => { + const { name, value } = e.target; + setEditor((prev) => (prev ? { ...prev, [name]: value } : prev)); + }; + + const handleTestConnection = async () => { + if (!editor) return; + if (!effectiveToken) { + toast.error(`${providerLabel} token is required to test the connection`); + return; + } + + setIsTesting(true); + + try { + const result = await githubApi.testConnection(effectiveToken, { + provider, + url: editor.url.trim() || undefined, + username: editor.username.trim() || undefined, + }); + if (result.success) { + toast.success(result.message || `Successfully connected to ${providerLabel}!`); + } else { + toast.error( + result.message || `Failed to connect to ${providerLabel}. Please check your token.` + ); + } + } catch (error) { + toast.error( + error instanceof Error ? error.message : "An unknown error occurred" + ); + } finally { + setIsTesting(false); + } + }; + + const handleSave = async (confirmSourceChange: boolean) => { + if (!editor) return; + setIsSaving(true); + + try { + const body: Record = { + provider, + username: editor.username.trim(), + }; + if (editor.url.trim()) { + body.url = editor.url.trim(); + } + // Empty token on edit keeps the stored one (existing save convention). + if (!editing || editor.token.trim()) { + body.token = editor.token.trim(); + } + if (editing && (confirmSourceChange || editor.sourceUnlocked)) { + body.confirmSourceChange = true; + } + + const { ok, status, body: result } = await requestMutation( + editing ? `/${editing.id}` : "", + { method: editing ? "PUT" : "POST", body: JSON.stringify(body) } + ); + + if (ok) { + toast.success(editing ? "Source updated" : "Source added"); + setEditor(null); + invalidateConfigCache(); + await refreshSources(); + return; + } + + if (status === 409 && result.lock === "source") { + setConfirmChangeMessage( + result.message || + `${repositoryNoun(editing?.repositoryCount ?? 0)} were imported from this source.` + ); + return; + } + + toast.error( + result.message || `Failed to ${editing ? "update" : "add"} source. Please try again.` + ); + } catch (error) { + toast.error( + error instanceof Error ? error.message : "An unknown error occurred" + ); + } finally { + setIsSaving(false); + } + }; + + const handleRemove = async () => { + if (!removeTarget) return; + setIsRemoving(true); + + try { + const { ok, body: result } = await requestMutation(`/${removeTarget.id}`, { + method: "DELETE", + body: JSON.stringify({ confirmRemoveSource: true }), + }); + + if (ok) { + toast.success("Source removed"); + setRemoveTarget(null); + invalidateConfigCache(); + await refreshSources(); + return; + } + + toast.error(result.message || "Failed to remove source. Please try again."); + } catch (error) { + toast.error( + error instanceof Error ? error.message : "An unknown error occurred" + ); + } finally { + setIsRemoving(false); + } + }; + + const isBusy = isSaving || isRemoving || isRefreshing; + + return ( + + {isBusy && ( + + )} + +
+ } + > + {editor && ( + <> + +
+ + +

+ {providerHelp(provider)} +

+ {editing?.locked && ( + + setEditor((prev) => + prev ? { ...prev, sourceUnlocked: true } : prev + ) + } + /> + )} +
+ +
+ + +

+ {`Leave empty for ${defaultInstanceUrl.replace(/^https?:\/\//, "")}, or enter the base URL of your own instance`} +

+
+ +
+ + +
+ +
+ + +

+ {providerMeta.tokenHint} +

+
+ +
+
+
+ + + Creating your token + +
+ + + +
+
    + {providerMeta.tokenSteps.map((step) => ( +
  1. {step}
    2. + ))} +
+
+ {providerMeta.scopes.map((scope) => ( + + {scope} + + ))} +
+
+ +
+ +
+ + +
+
+ + {sources.length > 0 && } + + )} + + + {sources.length === 0 && !editor ? ( +
+

+ No sources connected yet — add one to start mirroring. +

+ +
+ ) : ( +
+ {sources.map((source) => { + const meta = providerMetaFor(source.provider); + const ProviderIcon = meta.icon; + return ( +
+
+
+ + + {source.name || meta.label} + + + {meta.label} + + {meta.badge && ( + + {meta.badge} + + )} + {source.locked && ( + + + Locked + + )} +
+

+ {[ + sourceHost(source), + source.username, + repositoryNoun(source.repositoryCount), + ] + .filter(Boolean) + .join(" · ")} +

+
+
+ + +
+
+ ); + })} +
+ )} + + + !open && setConfirmChangeMessage(null)}> + + + Change the source? + {confirmChangeMessage} + +
    + {sourceChangeConsequences.map((line) => ( +
  • {line}
    • + ))} +
+ + + + + + + + !open && setRemoveTarget(null)}> + + + Remove source? + + {removeTarget && removeTarget.repositoryCount > 0 + ? `${repositoryNoun(removeTarget.repositoryCount)} were imported from this source. They keep syncing on the destination with their stored credentials.` + : "No repositories were imported from this source. Removing it only disconnects the credentials."} + + + + + + + + + + ); +} diff --git a/src/components/repositories/Repository.tsx b/src/components/repositories/Repository.tsx index bc44df6..89a7d2d 100644 --- a/src/components/repositories/Repository.tsx +++ b/src/components/repositories/Repository.tsx @@ -1,4 +1,4 @@ -import { useCallback, useEffect, useState } from "react"; +import { useCallback, useEffect, useMemo, useState } from "react"; import RepositoryTable, { REPOSITORY_SORT_OPTIONS } from "./RepositoryTable"; import type { MirrorJob, Repository } from "@/lib/db/schema"; import { useAuth } from "@/hooks/useAuth"; @@ -51,13 +51,14 @@ import { useLiveRefresh } from "@/hooks/useLiveRefresh"; import { useConfigStatus } from "@/hooks/useConfigStatus"; import { useNavigation } from "@/components/layout/MainLayout"; import { withBase } from "@/lib/base-path"; +import { getRepositorySource, normalizeSourceUrl } from "@/lib/source-providers/kinds"; export default function Repository() { const [repositories, setRepositories] = useState([]); const [isInitialLoading, setIsInitialLoading] = useState(true); const { user } = useAuth(); const { registerRefreshCallback, isLiveEnabled } = useLiveRefresh(); - const { isGitHubConfigured, isFullyConfigured, autoMirrorStarred, githubOwner } = useConfigStatus(); + const { isGitHubConfigured, isFullyConfigured, autoMirrorStarred, githubOwner, sources } = useConfigStatus(); const { navigationKey } = useNavigation(); const { filter, setFilter } = useFilterParams({ searchTerm: "", @@ -228,6 +229,27 @@ export default function Repository() { } }; + const usernameBySourceId = useMemo( + () => new Map(sources.map((source) => [source.id, source.username.toLowerCase()])), + [sources] + ); + + // Mirrors the scheduler's starred classification: a starred repo counts as + // "own" when its owner matches the username of the repository's own source + // (falling back to the configured owner only when no source resolves). + const sourceOwnerOf = useCallback( + (repo: Repository): string => { + const byId = repo.sourceId ? usernameBySourceId.get(repo.sourceId) : undefined; + if (byId !== undefined) return byId; + const repoSource = getRepositorySource({ sourceProvider: repo.sourceProvider, sourceUrl: repo.sourceUrl }); + const matched = sources.find( + (source) => source.provider === repoSource.provider && normalizeSourceUrl(source.url, source.provider) === repoSource.url + ); + return matched?.username.toLowerCase() ?? githubOwner.toLowerCase(); + }, + [usernameBySourceId, sources, githubOwner] + ); + const handleMirrorAllRepos = async () => { try { if (!user || !user.id || repositories.length === 0) { @@ -242,7 +264,7 @@ export default function Repository() { repo.status !== "ignored" && // Skip ignored repositories repo.id && // Skip starred repos from other owners when autoMirrorStarred is disabled - !(repo.isStarred && !autoMirrorStarred && repo.owner !== githubOwner) + !(repo.isStarred && !autoMirrorStarred && repo.owner.toLowerCase() !== sourceOwnerOf(repo)) ); if (eligibleRepos.length === 0) { diff --git a/src/hooks/useConfigStatus.ts b/src/hooks/useConfigStatus.ts index 04425a7..8b62643 100644 --- a/src/hooks/useConfigStatus.ts +++ b/src/hooks/useConfigStatus.ts @@ -1,6 +1,7 @@ import { useCallback, useEffect, useState, useRef } from 'react'; import { useAuth } from './useAuth'; import { apiRequest } from '@/lib/utils'; +import type { ConfigApiResponse, SourceApiRecord } from '@/types/config'; import type { ConfigApiResponse } from '@/types/config'; import { DEFAULT_SOURCE_PROVIDER, @@ -16,6 +17,7 @@ interface ConfigStatus { error: string | null; autoMirrorStarred: boolean; githubOwner: string; + sources: SourceApiRecord[]; sourceProvider: SourceProviderKind; } @@ -43,6 +45,7 @@ export function useConfigStatus(): ConfigStatus { error: null, autoMirrorStarred: false, githubOwner: '', + sources: [], sourceProvider: DEFAULT_SOURCE_PROVIDER, }); @@ -59,6 +62,7 @@ export function useConfigStatus(): ConfigStatus { error: 'No user found', autoMirrorStarred: false, githubOwner: '', + sources: [], sourceProvider: DEFAULT_SOURCE_PROVIDER, }); return; @@ -96,6 +100,7 @@ export function useConfigStatus(): ConfigStatus { error: null, autoMirrorStarred: configResponse?.advancedOptions?.autoMirrorStarred ?? false, githubOwner: configResponse?.githubConfig?.username ?? '', + sources: configResponse?.sources ?? [], sourceProvider: normalizeSourceProviderKind(configResponse?.githubConfig?.provider), }); return; @@ -142,6 +147,7 @@ export function useConfigStatus(): ConfigStatus { error: null, autoMirrorStarred: configResponse?.advancedOptions?.autoMirrorStarred ?? false, githubOwner: configResponse?.githubConfig?.username ?? '', + sources: configResponse?.sources ?? [], sourceProvider: normalizeSourceProviderKind(configResponse?.githubConfig?.provider), }); @@ -155,6 +161,7 @@ export function useConfigStatus(): ConfigStatus { error: error instanceof Error ? error.message : 'Failed to check configuration', autoMirrorStarred: false, githubOwner: '', + sources: [], sourceProvider: DEFAULT_SOURCE_PROVIDER, }); hasCheckedRef.current = true; diff --git a/src/lib/config-locks.test.ts b/src/lib/config-locks.test.ts index e68d41a..6e44670 100644 --- a/src/lib/config-locks.test.ts +++ b/src/lib/config-locks.test.ts @@ -40,11 +40,19 @@ describe("hasSourceChanged", () => { expect(hasSourceChanged({ provider: "github" }, { provider: "gitea" })).toBe(true); }); - test("sourceIdentity describes GitHub with its fixed URL", () => { + test("sourceIdentity defaults GitHub to github.com and preserves a GitHub Enterprise URL", () => { expect(sourceIdentity({ provider: "github", url: "https://ghe.example.com" })).toEqual({ + provider: "github", + url: "https://ghe.example.com", + }); + expect(sourceIdentity({ provider: "github" })).toEqual({ provider: "github", url: "https://github.com", }); + expect(hasSourceChanged( + { provider: "github", url: "https://ghe1.example.com" }, + { provider: "github", url: "https://ghe2.example.com" } + )).toBe(true); }); }); diff --git a/src/lib/config-locks.ts b/src/lib/config-locks.ts index ffe909c..7d3b16e 100644 --- a/src/lib/config-locks.ts +++ b/src/lib/config-locks.ts @@ -11,7 +11,6 @@ */ import type { ConfigLockState } from "@/types/config"; import { - SOURCE_PROVIDER_DEFAULT_URLS, SOURCE_PROVIDER_LABELS, normalizeSourceProviderKind, normalizeSourceUrl, @@ -49,15 +48,17 @@ export const UNLOCKED_CONFIG: ConfigLockState = computeConfigLocks({ mirroredCount: 0, }); -/** The host a source config points at, normalized so cosmetic edits do not count as a change. */ +/** + * The host a source config points at, normalized so cosmetic edits do not + * count as a change. An empty or missing URL resolves to the provider's + * default instance; a custom GitHub Enterprise URL is preserved so host + * switches on locked sources still need confirmation. + */ export function sourceIdentity(config: SourceLike | null | undefined): SourceIdentity { const provider = normalizeSourceProviderKind(config?.provider); return { provider, - url: - provider === "github" - ? SOURCE_PROVIDER_DEFAULT_URLS.github - : normalizeSourceUrl(config?.url, provider), + url: normalizeSourceUrl(config?.url, provider), }; } @@ -184,10 +185,27 @@ export async function loadConfigLocks(userId: string): Promise ) ); - return computeConfigLocks({ - repositoryCount: Number(all?.count ?? 0), - mirroredCount: Number(mirrored?.count ?? 0), + // Per-source lock state: how many imported repositories belong to each + // connected source row. The null bucket (repositories whose sourceId is + // missing or dangling) counts toward no source's lock, exactly like the + // "switched source" rows it contains. + const perSourceRows = await db + .select({ sourceId: repositories.sourceId, count: sql`count(*)` }) + .from(repositories) + .where(eq(repositories.userId, userId)) + .groupBy(repositories.sourceId); + const perSource = perSourceRows.map((row) => { + const count = Number(row.count ?? 0); + return { sourceId: row.sourceId ?? null, locked: count > 0, repositoryCount: count }; }); + + return { + ...computeConfigLocks({ + repositoryCount: Number(all?.count ?? 0), + mirroredCount: Number(mirrored?.count ?? 0), + }), + sources: perSource, + }; } catch (error) { console.warn( `[Config] Could not compute source/destination locks for user ${userId}; treating as unlocked: ${ diff --git a/src/lib/db/index.ts b/src/lib/db/index.ts index 4fafdad..e19632a 100644 --- a/src/lib/db/index.ts +++ b/src/lib/db/index.ts @@ -143,12 +143,13 @@ if (process.env.NODE_ENV !== "test") { export { db }; // Export all table definitions from schema -export { - users, - events, - configs, - repositories, - mirrorJobs, +export { + users, + events, + configs, + sources, + repositories, + mirrorJobs, organizations, sessions, accounts, diff --git a/src/lib/db/schema.ts b/src/lib/db/schema.ts index 971b7a8..ba5a9f5 100644 --- a/src/lib/db/schema.ts +++ b/src/lib/db/schema.ts @@ -238,6 +238,27 @@ export const configSchema = z.object({ updatedAt: z.coerce.date(), }); +// One connected source host per row of the sources table. Parses leniently +// for reads (defaults like githubConfigSchema); the write path in +// src/lib/sources.ts validates strictly. +export const sourceSchema = z.object({ + id: z.string(), + userId: z.string(), + name: z.string(), + // Which host the repositories come from. "gitea" also covers Forgejo. + provider: z.enum(SOURCE_PROVIDER_KINDS).default("github"), + // Normalized base URL of the instance; the provider default is resolved + // on write so the empty string is never stored. + url: z.string().default(""), + username: z.string().default(""), + // AES-256-GCM encrypted at rest, same convention as + // configs.githubConfig.token. Null means public-only access. + token: z.string().nullable().optional(), + enabled: z.boolean().default(true), + createdAt: z.coerce.date(), + updatedAt: z.coerce.date(), +}); + export const repositorySchema = z.object({ id: z.string(), userId: z.string(), @@ -250,6 +271,7 @@ export const repositorySchema = z.object({ owner: z.string(), sourceProvider: z.string().optional(), sourceUrl: z.string().optional(), + sourceId: z.string().optional().nullable(), destinationProvider: z.string().optional(), destinationUrl: z.string().optional(), organization: z.string().optional().nullable(), @@ -458,6 +480,44 @@ export const configs = sqliteTable("configs", { .default(sql`(unixepoch())`), }, (_table) => []); +// One connected source host per user. Multiple rows per user are allowed +// (github.com plus a GHES instance, or GitLab plus Gitea, ...); the first +// row by createdAt is the "primary" source mirrored into the legacy +// configs.githubConfig connection fields for backward compatibility. +export const sources = sqliteTable("sources", { + id: text("id").primaryKey(), + userId: text("user_id") + .notNull() + .references(() => users.id), + // Display label for the sources list. Auto-derived ("GitHub (octocat)") + // by src/lib/sources.ts when the caller leaves it blank. + name: text("name").notNull(), + provider: text("provider") + .$type() + .notNull(), + // Normalized instance base URL. The provider default is resolved on write + // so the empty string is never stored, which keeps the unique index below + // meaningful for default-instance rows. + url: text("url").notNull(), + // The account the token belongs to (the legacy config's "owner"). + username: text("username").notNull().default(""), + // AES-256-GCM encrypted at rest, same convention as + // configs.githubConfig.token. Null means public-only access. + token: text("token"), + enabled: integer("enabled", { mode: "boolean" }) + .notNull() + .default(true), + createdAt: integer("created_at", { mode: "timestamp" }) + .notNull() + .default(sql`(unixepoch())`), + updatedAt: integer("updated_at", { mode: "timestamp" }) + .notNull() + .default(sql`(unixepoch())`), +}, (table) => [ + index("idx_sources_user_id").on(table.userId), + uniqueIndex("uniq_sources_user_provider_url_username").on(table.userId, table.provider, table.url, table.username), +]); + export const repositories = sqliteTable("repositories", { id: text("id").primaryKey(), userId: text("user_id") @@ -479,6 +539,11 @@ export const repositories = sqliteTable("repositories", { .notNull() .default("github"), sourceUrl: text("source_url").notNull().default("https://github.com"), + // The sources row this repository was imported from. Nullable and + // deliberately without a foreign key: legacy rows may reference sources + // that were deleted later ("switched source" state), and those rows keep + // their dangling id on purpose. NULL means "not linked (yet)". + sourceId: text("source_id"), // Where the mirror lives: the destination kind and base URL recorded when // the row was imported or mirrored. An empty URL means "not recorded yet". destinationProvider: text("destination_provider") @@ -547,8 +612,12 @@ export const repositories = sqliteTable("repositories", { index("idx_repositories_is_fork").on(table.isForked), index("idx_repositories_is_starred").on(table.isStarred), index("idx_repositories_user_imported_at").on(table.userId, table.importedAt), - uniqueIndex("uniq_repositories_user_full_name").on(table.userId, table.fullName), - uniqueIndex("uniq_repositories_user_normalized_full_name").on(table.userId, table.normalizedFullName), + // Full names are unique per source: the same "owner/repo" can legitimately + // exist on two connected hosts (e.g. github.com and a GHES instance). + // SQLite treats NULLs as distinct in unique indexes, so unlinked legacy + // rows (source_id NULL) never collide with each other. + uniqueIndex("uniq_repositories_user_full_name").on(table.userId, table.sourceId, table.fullName), + uniqueIndex("uniq_repositories_user_normalized_full_name").on(table.userId, table.sourceId, table.normalizedFullName), index("idx_repositories_mirrored_location").on(table.userId, table.mirroredLocation), ]); @@ -1031,6 +1100,7 @@ export const rateLimits = sqliteTable("rate_limits", { // Export type definitions export type User = z.infer; export type Config = z.infer; +export type Source = z.infer; export type Repository = z.infer; export type MirrorJob = z.infer; export type Organization = z.infer; diff --git a/src/lib/destination-reconcile-service.ts b/src/lib/destination-reconcile-service.ts index fa687e8..835afae 100644 --- a/src/lib/destination-reconcile-service.ts +++ b/src/lib/destination-reconcile-service.ts @@ -15,8 +15,18 @@ import type { Config } from "@/types/config"; import type { GitRepo } from "@/types/Repository"; import { HttpError, httpGet } from "@/lib/http-client"; import { getDecryptedGiteaToken } from "@/lib/utils/config-encryption"; -import { createSourceProviderFromConfig, resolveSourceConnection } from "@/lib/source-providers"; +import { + createSourceProviderFromSource, + resolveSourceConnection, + sourceHostOf, + type SourceProvider, +} from "@/lib/source-providers"; import { githubApiBaseUrl } from "@/lib/source-providers/github-source"; +import { + listSources, + resolveGitHubApiBaseUrl, + type SourceRecord, +} from "@/lib/sources"; import { getGiteaRepoOwnerAsync } from "@/lib/gitea"; import { resolveDestinationIdentity } from "@/lib/destination-connection"; import { normalizeGitRepoToInsert } from "@/lib/repo-utils"; @@ -192,7 +202,30 @@ export async function reconcileDestination( const baseUrl = giteaConfig.url.replace(/\/+$/, ""); const headers = { Authorization: `token ${getDecryptedGiteaToken(config)}` }; + // The configured connection is only the last-resort identity for mirrors + // whose host no connected source owns; adoption resolves each mirror to + // its own source row below. const connection = resolveSourceConnection(config); + const sources = await listSources(userId); + // Host a mirror's original URL can point at and still count as ours: any + // connected source, plus the API alias it is reached through. Stored + // clone URL hosts are added once the rows are loaded below. + const sourceByHost = new Map(); + const knownHosts = new Set(); + for (const source of sources) { + const apiBaseUrl = + source.provider === "github" + ? resolveGitHubApiBaseUrl(source.url) ?? githubApiBaseUrl() + : null; + for (const host of [sourceHostOf(source.url), apiBaseUrl ? sourceHostOf(apiBaseUrl) : ""]) { + if (!host) continue; + knownHosts.add(host); + if (!sourceByHost.has(host)) sourceByHost.set(host, source); + } + for (const host of knownSourceHosts({ sourceUrl: source.url, apiUrl: apiBaseUrl, cloneUrls: [] })) { + knownHosts.add(host); + } + } const starredMode = githubConfig?.starredReposMode || "dedicated-org"; const starredOrg = githubConfig?.starredReposOrg || "starred"; const strategy = @@ -204,6 +237,14 @@ export async function reconcileDestination( .from(organizations) .where(eq(organizations.userId, userId)); + for (const host of knownSourceHosts({ + sourceUrl: sources[0]?.url ?? "", + apiUrl: null, + cloneUrls: rows.map((row) => row.cloneUrl), + })) { + knownHosts.add(host); + } + // Every owner the strategy or the database could have put a mirror under. const owners = collectDestinationOwners([ giteaConfig.defaultOwner, @@ -231,12 +272,6 @@ export async function reconcileDestination( destinationRepos.push(...repos); } - const knownHosts = knownSourceHosts({ - sourceUrl: connection.url, - apiUrl: connection.provider === "github" ? githubApiBaseUrl() : null, - cloneUrls: rows.map((row) => row.cloneUrl), - }); - // Rows mirrored to a previous destination are neither healthy nor missing // here; they are left out of the comparison and only counted. const { here: rowsHere, elsewhere } = splitRowsByDestination(rows, resolveDestinationIdentity(config)); @@ -322,16 +357,43 @@ export async function reconcileDestination( let relocated = 0; if (options.adoptUntracked && classified.untracked.length > 0) { - const sourceProvider = createSourceProviderFromConfig(config, { userId }); + // One provider instance per source row (the Gitea provider keeps an + // instance-local org cache), and one identity per mirror: the source + // its original URL points at. + const providersBySourceId = new Map(); + const providerForSource = (source: SourceRecord): SourceProvider => { + let provider = providersBySourceId.get(source.id); + if (!provider) { + provider = createSourceProviderFromSource(source, { userId }); + providersBySourceId.set(source.id, provider); + } + return provider; + }; await processInParallel( classified.untracked, async (repo) => { try { + const origin = parseRepoUrl(repo.originalUrl); + const repoSource = origin ? sourceByHost.get(origin.host) ?? null : null; + if (!repoSource) { + // Same skip the cleanup service applies to rows outside the + // configured source: without the source row there is no token + // to check the upstream with, so the destination's own report + // is all the adoption can rely on. + console.log( + `[Reconcile] Skipping the source check for ${repo.fullName} - no connected source owns its host` + ); + } const outcome = await adoptUntrackedMirror({ config, repo, - lookup: (owner, name) => sourceProvider.getRepository(owner, name), - connection, + sourceId: repoSource?.id ?? null, + lookup: repoSource + ? (owner: string, name: string) => providerForSource(repoSource).getRepository(owner, name) + : async () => null, + connection: repoSource + ? { provider: repoSource.provider, url: repoSource.url } + : connection, starredOrg: starredMode === "preserve-owner" ? null : starredOrg, }); if (outcome === "adopted") adopted += 1; @@ -430,12 +492,17 @@ export async function reconcileDestination( async function adoptUntrackedMirror({ config, repo, + sourceId, lookup, connection, starredOrg, }: { config: Config; repo: DestinationRepo; + // The connected source row the mirror's host belongs to, or null when no + // connected source owns it. Adoption must never attribute a repo to the + // config's source on a host mismatch — null keeps the row unattributed. + sourceId: string | null; lookup: (owner: string, name: string) => Promise; connection: { provider: GitRepo["sourceProvider"]; url: string }; starredOrg: string | null; @@ -491,7 +558,7 @@ async function adoptUntrackedMirror({ importedAt: now, }; - const insert = normalizeGitRepoToInsert(gitRepo, { userId: config.userId!, configId: config.id! }); + const insert = normalizeGitRepoToInsert(gitRepo, { userId: config.userId!, configId: config.id!, sourceId }); // The normalizer always starts rows as imported; this one is already mirrored. insert.status = "mirrored"; insert.mirroredLocation = repo.fullName; @@ -515,7 +582,7 @@ async function adoptUntrackedMirror({ const inserted = await db .insert(repositories) .values(insert) - .onConflictDoNothing({ target: [repositories.userId, repositories.normalizedFullName] }) + .onConflictDoNothing({ target: [repositories.userId, repositories.sourceId, repositories.normalizedFullName] }) .returning({ id: repositories.id }); return inserted.length > 0 ? "adopted" : "skipped"; diff --git a/src/lib/env-config-loader.ts b/src/lib/env-config-loader.ts index d75a520..39387bd 100644 --- a/src/lib/env-config-loader.ts +++ b/src/lib/env-config-loader.ts @@ -3,13 +3,14 @@ * Loads configuration from environment variables and populates the database */ -import { db, configs, users } from '@/lib/db'; +import { db, configs, users, repositories } from '@/lib/db'; import { eq, and, sql } from 'drizzle-orm'; import { v4 as uuidv4 } from 'uuid'; import { encrypt } from '@/lib/utils/encryption'; import { isSourceProviderKind, type SourceProviderKind } from '@/lib/source-providers/kinds'; import { DESTINATION_PROVIDER_DEFAULT_URLS, isDestinationProviderKind, type DestinationProviderKind } from '@/lib/destination-kinds'; import { hasDestinationChanged, hasSourceChanged, loadConfigLocks } from '@/lib/config-locks'; +import type { SourceUpdateInput } from '@/lib/sources'; import { normalizeReleaseAssetLimit } from '@/lib/utils/mirror-overrides'; interface EnvConfig { @@ -476,9 +477,87 @@ export async function initializeConfigFromEnv(): Promise { }); } + await syncFirstSourceFromEnv(userId, envConfig); + console.log('[ENV Config Loader] Configuration initialized successfully from environment variables'); } catch (error) { console.error('[ENV Config Loader] Failed to initialize configuration from environment:', error); // Don't throw - this is a non-critical initialization } } + +/** + * Keep the user's first (primary) source row in sync with the environment + * connection variables. The sources table is authoritative for connections; + * createSource/updateSource mirror the primary source back into the config's + * githubConfig connection fields (write-through). Never clears values the + * env did not set. + */ +async function syncFirstSourceFromEnv(userId: string, envConfig: EnvConfig): Promise { + try { + const { createSource, listSources, updateSource } = await import('@/lib/sources'); + const existingSources = await listSources(userId); + + if (existingSources.length === 0) { + // Only mint a source row when the environment actually carries a + // connection; preference-only envs must not create an empty source. + if (envConfig.github.token || envConfig.github.username) { + await createSource(userId, { + provider: envConfig.github.provider ?? 'github', + url: envConfig.github.url, + username: envConfig.github.username ?? '', + token: envConfig.github.token ?? '', + }); + console.log('[ENV Config Loader] Created the primary source from environment variables'); + } + return; + } + + const primary = existingSources[0]; + + const updates: SourceUpdateInput = {}; + if (envConfig.github.provider !== undefined) { + updates.provider = envConfig.github.provider; + } + if (envConfig.github.url !== undefined) { + updates.url = envConfig.github.url; + } + if (envConfig.github.username !== undefined) { + updates.username = envConfig.github.username; + } + if (envConfig.github.token) { + updates.token = envConfig.github.token; + } + + // A source with imported repositories is locked: environment variables + // cannot switch its host, mirroring the githubConfig check above. + const sourceRepoCount = await countRepositoriesForSource(userId, primary.id); + const incomingSource = { + provider: envConfig.github.provider ?? primary.provider, + url: envConfig.github.url ?? primary.url, + }; + const lockedHostChange = + sourceRepoCount > 0 && + (updates.provider !== undefined || updates.url !== undefined) && + hasSourceChanged({ provider: primary.provider, url: primary.url }, incomingSource); + if (lockedHostChange) { + console.warn( + `[ENV Config Loader] Ignoring SOURCE_PROVIDER/SOURCE_URL: the source "${primary.name}" is locked because ${sourceRepoCount} repositories were imported from it. Change it on the Configuration page.` + ); + delete updates.provider; + delete updates.url; + } + + await updateSource(userId, primary.id, updates); + } catch (error) { + console.error('[ENV Config Loader] Failed to sync the primary source from environment variables:', error); + } +} + +async function countRepositoriesForSource(userId: string, sourceId: string): Promise { + const [row] = await db + .select({ count: sql`count(*)` }) + .from(repositories) + .where(and(eq(repositories.userId, userId), eq(repositories.sourceId, sourceId))); + return Number(row?.count ?? 0); +} diff --git a/src/lib/gitea-enhanced.test.ts b/src/lib/gitea-enhanced.test.ts index d376449..cb3cd78 100644 --- a/src/lib/gitea-enhanced.test.ts +++ b/src/lib/gitea-enhanced.test.ts @@ -1,5 +1,6 @@ import { describe, test, expect, mock, beforeEach, afterEach } from "bun:test"; import { createMockResponse, mockFetch } from "@/tests/mock-fetch"; +import { getRepositorySource } from "@/lib/source-providers/kinds"; // Mock the helpers module before importing gitea-enhanced const mockCreateMirrorJob = mock(() => Promise.resolve("mock-job-id")); @@ -56,6 +57,48 @@ mock.module("@/lib/utils/config-encryption", () => ({ getDecryptedGiteaToken: (config: any) => config.giteaConfig?.token || "" })); +// The sync path resolves the GitHub client from the repository's own source +// row. One GitHub source mirrors the pre-multi-source fixtures (same token, +// same account), with the real provider+host matching semantics. +const sourceRows = [ + { + id: "source-1", + userId: "user123", + name: "GitHub (testuser)", + provider: "github", + url: "https://github.com", + username: "testuser", + token: "github-token", + enabled: true, + createdAt: new Date(0), + updatedAt: new Date(0), + }, +]; + +mock.module("@/lib/sources", () => ({ + listSources: mock(async (userId: string) => + sourceRows.filter((source) => source.userId === userId) + ), + findSourceForRepository: (repo: any, list: any[]) => { + if (repo.sourceId) { + const byId = list.find((source) => source.id === repo.sourceId); + if (byId) return byId; + } + const source = getRepositorySource(repo); + return ( + list.find( + (candidate: any) => candidate.provider === source.provider && candidate.url === source.url + ) ?? null + ); + }, + decryptSourceToken: (token: string | null | undefined) => token ?? "", + resolveGitHubApiBaseUrl: (url: string | null | undefined) => { + const trimmed = url?.trim().replace(/\/+$/, "") ?? ""; + if (!trimmed || trimmed === "https://github.com") return undefined; + return `${trimmed}/api/v3`; + }, +})); + // Mock http-client class MockHttpError extends Error { constructor(message: string, public status: number, public statusText: string, public response?: string) { diff --git a/src/lib/gitea-enhanced.ts b/src/lib/gitea-enhanced.ts index 9f071e5..e66d4bc 100644 --- a/src/lib/gitea-enhanced.ts +++ b/src/lib/gitea-enhanced.ts @@ -7,7 +7,13 @@ */ import type { Config } from "@/types/config"; -import { normalizeSourceProviderKind } from "@/lib/source-providers/kinds"; +import { getRepositorySource } from "@/lib/source-providers/kinds"; +import { + decryptSourceToken, + findSourceForRepository, + listSources, + resolveGitHubApiBaseUrl, +} from "@/lib/sources"; import type { Repository } from "./db/schema"; import type { Octokit } from "@octokit/rest"; import { createGitHubClient } from "./github"; @@ -451,6 +457,14 @@ export async function syncGiteaRepoEnhanced({ const decryptedConfig = decryptConfigTokens(config as Config); + // Metadata and force-push detection talk to the repository's own source, + // so the user's sources are loaded once and this repository's row drives + // every GitHub client below. + const sources = config.userId ? await listSources(config.userId) : []; + const repoSource = findSourceForRepository(repository, sources); + const repoSourceToken = repoSource ? decryptSourceToken(repoSource.token) : ""; + const repoIsGitHub = getRepositorySource(repository).provider === "github"; + if (usesPushEngine(config)) { throw new Error("The configured destination is a push target; syncing goes through the push engine."); } @@ -671,11 +685,16 @@ export async function syncGiteaRepoEnhanced({ // (skip when called from approve-sync to avoid re-blocking) if (strategyNeedsDetection(backupStrategy) && !skipForcePushDetection) { try { - const decryptedGithubToken = decryptedConfig.githubConfig?.token; // Force-push detection compares branches through the GitHub API, - // so it only runs for GitHub sources. - if (decryptedGithubToken && normalizeSourceProviderKind(config.githubConfig?.provider) === "github") { - const fpOctokit = createGitHubClient(decryptedGithubToken); + // so it only runs for GitHub sources, with the repository's own + // source token. + if (repoSource && repoSourceToken && repoIsGitHub) { + const fpOctokit = createGitHubClient( + repoSourceToken, + config.userId || undefined, + repoSource.username || undefined, + resolveGitHubApiBaseUrl(repoSource.url) + ); const detectionResult = await detectForcePush({ giteaUrl: config.giteaConfig.url, giteaToken: decryptedConfig.giteaConfig.token, @@ -897,15 +916,17 @@ export async function syncGiteaRepoEnhanced({ if (metadataOctokit) { return metadataOctokit; } - // The token belongs to whichever host is configured; only GitHub's - // token can drive Octokit. - if ( - !decryptedConfig.githubConfig?.token || - normalizeSourceProviderKind(config.githubConfig?.provider) !== "github" - ) { + // Only the token of the repository's own GitHub source can drive + // Octokit; other hosts and public-only sources skip metadata. + if (!repoSource || !repoSourceToken || !repoIsGitHub) { return null; } - metadataOctokit = createGitHubClient(decryptedConfig.githubConfig.token); + metadataOctokit = createGitHubClient( + repoSourceToken, + config.userId || undefined, + repoSource.username || undefined, + resolveGitHubApiBaseUrl(repoSource.url) + ); return metadataOctokit; }; diff --git a/src/lib/gitea-org-mirror-destination.test.ts b/src/lib/gitea-org-mirror-destination.test.ts index 60b437b..1de61ca 100644 --- a/src/lib/gitea-org-mirror-destination.test.ts +++ b/src/lib/gitea-org-mirror-destination.test.ts @@ -11,6 +11,7 @@ * where the mocks are safely contained. */ import { describe, test, expect, mock, beforeEach } from "bun:test"; +import { getRepositorySource } from "@/lib/source-providers/kinds"; const CHILD_FLAG = "GITEA_ORG_MIRROR_DEST_ISOLATED"; const isChild = !!process.env[CHILD_FLAG]; @@ -115,6 +116,49 @@ mock.module("@/lib/helpers", () => ({ createMirrorJob: mock(async () => "job-id"), })); +// The migrate paths resolve credentials from the repository's own source +// row and refuse rows no connected source owns. One GitHub source mirrors +// the pre-multi-source fixtures (same token, same account); matching keeps +// the real provider+host semantics via getRepositorySource. +mock.module("@/lib/sources", () => ({ + listSources: mock(async (userId: string) => + userId === "user-1" + ? [ + { + id: "source-1", + userId, + name: "GitHub (me)", + provider: "github", + url: "https://github.com", + username: "me", + token: "gh-token", + enabled: true, + createdAt: new Date(0), + updatedAt: new Date(0), + }, + ] + : [] + ), + findSourceForRepository: (repo: any, list: any[]) => { + if (repo.sourceId) { + const byId = list.find((source) => source.id === repo.sourceId); + if (byId) return byId; + } + const source = getRepositorySource(repo); + return ( + list.find( + (candidate: any) => candidate.provider === source.provider && candidate.url === source.url + ) ?? null + ); + }, + decryptSourceToken: (token: string | null | undefined) => token ?? "", + resolveGitHubApiBaseUrl: (url: string | null | undefined) => { + const trimmed = url?.trim().replace(/\/+$/, "") ?? ""; + if (!trimmed || trimmed === "https://github.com") return undefined; + return `${trimmed}/api/v3`; + }, +})); + const actualHttp = await import("./http-client"); const actualEnhanced = await import("./gitea-enhanced"); const actualConfigEncryption = await import("./utils/config-encryption"); diff --git a/src/lib/gitea.ts b/src/lib/gitea.ts index cbc91c2..8b9edb4 100644 --- a/src/lib/gitea.ts +++ b/src/lib/gitea.ts @@ -16,9 +16,15 @@ import { decryptConfigTokens } from "./utils/config-encryption"; import { formatDateShort } from "./utils"; import { buildSourceAuthPayload } from "./utils/mirror-source-auth"; import { - assertRepositoryMatchesConfiguredSource, + assertRepositoryMatchesConnectedSource, resolveSourceConnection, + sourceConnectionFromSource, } from "./source-providers"; +import { + decryptSourceToken, + findSourceForRepository, + listSources, +} from "./sources"; import { parseRepositoryMetadataState, serializeRepositoryMetadataState, @@ -77,9 +83,12 @@ export const getOrganizationConfig = async ({ export const getGiteaRepoOwnerAsync = async ({ config, repository, + sourceUsername, }: { config: Partial; repository: Repository; + /** Account name of the repository's own source, when already resolved. */ + sourceUsername?: string; }): Promise => { if (!config.githubConfig || !config.giteaConfig) { throw new Error("GitHub or Gitea config is required."); @@ -124,15 +133,18 @@ export const getGiteaRepoOwnerAsync = async ({ // For personal repos (not organization repos), fall back to the default strategy // Fall back to existing strategy logic - return getGiteaRepoOwner({ config, repository }); + return getGiteaRepoOwner({ config, repository, sourceUsername }); }; export const getGiteaRepoOwner = ({ config, repository, + sourceUsername, }: { config: Partial; repository: Repository; + /** Account name of the repository's own source, when already resolved. */ + sourceUsername?: string; }): string => { if (!config.githubConfig || !config.giteaConfig) { throw new Error("GitHub or Gitea config is required."); @@ -154,14 +166,17 @@ export const getGiteaRepoOwner = ({ // Get the mirror strategy - use preserveOrgStructure for backward compatibility const mirrorStrategy = config.githubConfig.mirrorStrategy || (config.giteaConfig.preserveOrgStructure ? "preserve" : "flat-user"); - const configuredGitHubOwner = - ( - config.githubConfig.owner || - (config.githubConfig as typeof config.githubConfig & { username?: string }).username || - "" - ) - .trim() - .toLowerCase(); + // The account the repository's own source belongs to. Callers that + // resolved the source row pass its username; everyone else keeps the + // configured owner. + const configuredGitHubOwner = ( + sourceUsername ?? + config.githubConfig.owner ?? + (config.githubConfig as typeof config.githubConfig & { username?: string }).username ?? + "" + ) + .trim() + .toLowerCase(); switch (mirrorStrategy) { case "preserve": @@ -652,24 +667,34 @@ export const mirrorGithubRepoToGitea = async ({ // Decrypt config tokens for API usage const decryptedConfig = decryptConfigTokens(config as Config); - // Never send another host's token: the repository must come from the - // source that is configured now. - assertRepositoryMatchesConfiguredSource({ repository, config }); + // Never send another host's token: the repository must come from one of + // the connected sources, and its own source provides the credentials. + const sources = await listSources(config.userId); + assertRepositoryMatchesConnectedSource({ repository, sources }); if (usesPushEngine(config)) { throw new Error("The configured destination is a push target; the Gitea mirror path cannot serve it."); } // Nor to a different destination than the one the row was mirrored to. assertRepositoryMatchesConfiguredDestination({ repository, config }); + // The source this repository came from. The assert above refused rows + // no connected source owns, so this practically always resolves; the + // config fallback only keeps the types honest. + const repoSource = findSourceForRepository(repository, sources); + const sourceConnection = repoSource + ? sourceConnectionFromSource(repoSource, { userId: config.userId }) + : resolveSourceConnection(config); + const repoSourceToken = repoSource ? decryptSourceToken(repoSource.token) : ""; + // Get the correct owner based on the strategy (with organization overrides) - let repoOwner = await getGiteaRepoOwnerAsync({ config, repository }); + let repoOwner = await getGiteaRepoOwnerAsync({ + config, + repository, + sourceUsername: repoSource?.username, + }); const mirrorStrategy = config.githubConfig.mirrorStrategy || (config.giteaConfig.preserveOrgStructure ? "preserve" : "flat-user"); - const configuredGitHubOwner = ( - config.githubConfig.owner || - (config.githubConfig as typeof config.githubConfig & { username?: string }).username || - "" - ) + const configuredGitHubOwner = (repoSource?.username ?? "") .trim() .toLowerCase(); const normalizedRepoOwner = repository.owner.trim().toLowerCase(); @@ -988,14 +1013,14 @@ export const mirrorGithubRepoToGitea = async ({ // Always send authentication credentials so Gitea/Forgejo stores them // for subsequent mirror fetches. This prevents "terminal prompts disabled" - // errors on public repos and raises GitHub API rate limits. + // errors on public repos and raises GitHub API rate limits. The token and + // account name belong to the repository's own source. { - const sourceConnection = resolveSourceConnection(config); Object.assign( migratePayload, buildSourceAuthPayload({ provider: sourceConnection.provider, - token: decryptedConfig.githubConfig?.token, + token: repoSourceToken, username: sourceConnection.username, repositoryOwner: repository.owner, }) @@ -1508,15 +1533,25 @@ export async function mirrorGitHubRepoToGiteaOrg({ // Decrypt config tokens for API usage const decryptedConfig = decryptConfigTokens(config as Config); - // Never send another host's token: the repository must come from the - // source that is configured now. - assertRepositoryMatchesConfiguredSource({ repository, config }); + // Never send another host's token: the repository must come from one of + // the connected sources, and its own source provides the credentials. + const sources = await listSources(config.userId); + assertRepositoryMatchesConnectedSource({ repository, sources }); if (usesPushEngine(config)) { throw new Error("The configured destination is a push target; the Gitea mirror path cannot serve it."); } // Nor to a different destination than the one the row was mirrored to. assertRepositoryMatchesConfiguredDestination({ repository, config }); + // The source this repository came from. The assert above refused rows + // no connected source owns, so this practically always resolves; the + // config fallback only keeps the types honest. + const repoSource = findSourceForRepository(repository, sources); + const sourceConnection = repoSource + ? sourceConnectionFromSource(repoSource, { userId: config.userId }) + : resolveSourceConnection(config); + const repoSourceToken = repoSource ? decryptSourceToken(repoSource.token) : ""; + // Determine the actual repository name to use (handle duplicates for starred repos) let targetRepoName = repository.name; // The org we will record/reuse for. Stays === orgName on the create path @@ -1754,14 +1789,14 @@ export async function mirrorGitHubRepoToGiteaOrg({ // Always send authentication credentials so Gitea/Forgejo stores them // for subsequent mirror fetches. This prevents "terminal prompts disabled" - // errors on public repos and raises GitHub API rate limits. + // errors on public repos and raises GitHub API rate limits. The token and + // account name belong to the repository's own source. { - const sourceConnection = resolveSourceConnection(config); Object.assign( migratePayload, buildSourceAuthPayload({ provider: sourceConnection.provider, - token: decryptedConfig.githubConfig?.token, + token: repoSourceToken, username: sourceConnection.username, repositoryOwner: repository.owner, }) @@ -2066,10 +2101,18 @@ export async function mirrorGitHubOrgToGitea({ organization, octokit, config, + resolveRepositoryOctokit, }: { organization: Organization; octokit: Octokit | null; config: Partial; + /** + * GitHub client for one repository, built from that repository's own + * source. Organizations are shared across sources, so the repos inside + * one org can come from different hosts; without a resolver every repo + * gets the single passed-in client. + */ + resolveRepositoryOctokit?: (repository: Repository) => Octokit | null; }) { try { if ( @@ -2216,18 +2259,25 @@ export async function mirrorGitHubOrgToGitea({ return repo; } + // Repos in one organization can come from different sources, so + // the GitHub client is resolved per repository when a resolver is + // available, and shared otherwise. + const repoOctokit = resolveRepositoryOctokit + ? resolveRepositoryOctokit(repoData) + : octokit; + // Resolve per repo with the canonical precedence const owner = await getGiteaRepoOwnerAsync({ config, repository: repoData }); if (owner === config.giteaConfig?.defaultOwner) { await mirrorGithubRepoToGitea({ - octokit, + octokit: repoOctokit, repository: repoData, config, }); } else if (owner === targetOrgName && giteaOrgId !== undefined) { await mirrorGitHubRepoToGiteaOrg({ - octokit, + octokit: repoOctokit, config, repository: repoData, giteaOrgId, @@ -2239,7 +2289,7 @@ export async function mirrorGitHubOrgToGitea({ config, }); await mirrorGitHubRepoToGiteaOrg({ - octokit, + octokit: repoOctokit, config, repository: repoData, giteaOrgId: ownerOrgId, diff --git a/src/lib/github.ts b/src/lib/github.ts index 7502fee..719cc9a 100644 --- a/src/lib/github.ts +++ b/src/lib/github.ts @@ -36,6 +36,7 @@ export function createGitHubClient( token: string, userId?: string, username?: string, + apiBaseUrl?: string, ): Octokit { // Create a proper User-Agent to identify our application // This helps GitHub understand our traffic patterns and can provide better rate limits @@ -43,9 +44,11 @@ export function createGitHubClient( ? `gitea-mirror/3.5.4 (user:${username})` : "gitea-mirror/3.5.4"; - // Support GH_API_URL (preferred) or GITHUB_API_URL (may conflict with GitHub Actions) - // GitHub Actions sets GITHUB_API_URL to https://api.github.com by default - const baseUrl = process.env.GH_API_URL || process.env.GITHUB_API_URL || "https://api.github.com"; + // An explicit apiBaseUrl (a GitHub Enterprise source's /api/v3) replaces + // the env default; otherwise support GH_API_URL (preferred) or + // GITHUB_API_URL (may conflict with GitHub Actions, which sets it to + // https://api.github.com by default). + const baseUrl = apiBaseUrl || process.env.GH_API_URL || process.env.GITHUB_API_URL || "https://api.github.com"; const octokit = new MyOctokit({ auth: token, // Always use token for authentication (5000 req/hr vs 60 for unauthenticated) diff --git a/src/lib/push-engine/mirror.ts b/src/lib/push-engine/mirror.ts index 1ca2931..1fb818d 100644 --- a/src/lib/push-engine/mirror.ts +++ b/src/lib/push-engine/mirror.ts @@ -11,10 +11,15 @@ import type { Config, Repository } from "@/lib/db/schema"; import { createMirrorJob } from "@/lib/helpers"; import { getGiteaRepoOwnerAsync } from "@/lib/gitea"; import { - assertRepositoryMatchesConfiguredSource, - resolveSourceConnection, + assertRepositoryMatchesConnectedSource, + sourceConnectionFromSource, } from "@/lib/source-providers"; -import { getDecryptedGitHubToken } from "@/lib/utils/config-encryption"; +import type { SourceRecord } from "@/lib/sources"; +import { + decryptSourceToken, + findSourceForRepository, + listSources, +} from "@/lib/sources"; import { buildSourceAuthPayload } from "@/lib/utils/mirror-source-auth"; import { assertRepositoryMatchesConfiguredDestination, @@ -48,13 +53,22 @@ export function parseMirroredLocation(location: string | null | undefined): { ow return { owner: trimmed.slice(0, slash), name: trimmed.slice(slash + 1) }; } -/** The credentials git uses to fetch from the source, per source host. */ -export function resolveSourceCredentials(config: Partial, repository: Repository): GitCredentials | null { - const token = config.githubConfig?.token ? getDecryptedGitHubToken(config as Config) : ""; - const connection = resolveSourceConnection(config, { token }); +/** + * The credentials git uses to fetch from the source, from the repository's + * own source row. Sources are loaded once per push by the caller and passed + * down so batches never query per repository. + */ +export function resolveSourceCredentials( + config: Partial, + repository: Repository, + sources: SourceRecord[] +): GitCredentials | null { + const source = findSourceForRepository(repository, sources); + if (!source) return null; + const connection = sourceConnectionFromSource(source, { userId: config.userId }); const payload = buildSourceAuthPayload({ provider: connection.provider, - token, + token: decryptSourceToken(source.token), username: connection.username, repositoryOwner: repository.owner, }); @@ -89,8 +103,11 @@ export async function pushMirrorRepository({ const label = DESTINATION_PROVIDER_LABELS[kind]; const destination = resolveDestinationIdentity(config); - // Never send another host's token in either direction. - assertRepositoryMatchesConfiguredSource({ repository, config }); + // Never send another host's token in either direction: the repository + // must come from one of the connected sources, and its own source's + // credentials are used below. + const sources = await listSources(config.userId); + assertRepositoryMatchesConnectedSource({ repository, sources }); assertRepositoryMatchesConfiguredDestination({ repository, config }); const recorded = parseMirroredLocation(repository.mirroredLocation); @@ -131,7 +148,7 @@ export async function pushMirrorRepository({ const outcome = await runPushMirror({ clonePath: clonePathForRepository(repository), sourceCloneUrl: repository.cloneUrl, - sourceCredentials: resolveSourceCredentials(config, repository), + sourceCredentials: resolveSourceCredentials(config, repository, sources), target: target ?? createPushTargetFromConfig(config), owner, name, diff --git a/src/lib/recovery.ts b/src/lib/recovery.ts index 29748ec..b61d8c9 100644 --- a/src/lib/recovery.ts +++ b/src/lib/recovery.ts @@ -10,11 +10,15 @@ import { eq, and, lt, inArray, sql } from 'drizzle-orm'; import { mirrorRepositoryToDestination, syncRepositoryOnDestination } from './mirror-dispatch'; import { createGitHubClient } from './github'; import type { Octokit } from '@octokit/rest'; -import { resolveSourceProviderKind } from './source-providers'; +import { + decryptSourceToken, + findSourceForRepository, + listSources, + resolveGitHubApiBaseUrl, +} from './sources'; import { processWithResilience } from './utils/concurrency'; import { repositoryVisibilityEnum, repoStatusEnum } from '@/types/Repository'; import type { Repository } from './db/schema'; -import { getDecryptedGitHubToken } from './utils/config-encryption'; // Recovery state tracking let recoveryInProgress = false; @@ -275,17 +279,9 @@ async function recoverMirrorJob(job: any, remainingItemIds: string[]) { // Only GitHub sources use an API client while mirroring (metadata). // Other hosts get code-only mirrors through Gitea, so no client is built. - let octokit: Octokit | null = null; - if (resolveSourceProviderKind(config) === 'github') { - try { - const decryptedToken = getDecryptedGitHubToken(config); - const githubUsername = config.githubConfig?.owner || undefined; - const userId = config.userId || undefined; - octokit = createGitHubClient(decryptedToken, userId, githubUsername); - } catch (error) { - throw new Error(`Failed to create GitHub client: ${error instanceof Error ? error.message : String(error)}`); - } - } + // Repositories picked by bare id can span sources, so the sources are + // loaded once here and the client is resolved per repository below. + const sources = await listSources(job.userId); // Process repositories with resilience and reduced concurrency for recovery await processWithResilience( @@ -303,6 +299,24 @@ async function recoverMirrorJob(job: any, remainingItemIds: string[]) { mirroredLocation: repo.mirroredLocation || "", }; + let octokit: Octokit | null = null; + const repoSource = findSourceForRepository(repo, sources); + if (repoSource?.provider === 'github') { + try { + const decryptedToken = decryptSourceToken(repoSource.token); + octokit = decryptedToken + ? createGitHubClient( + decryptedToken, + config.userId || undefined, + repoSource.username || undefined, + resolveGitHubApiBaseUrl(repoSource.url) + ) + : null; + } catch (error) { + throw new Error(`Failed to create GitHub client: ${error instanceof Error ? error.message : String(error)}`); + } + } + await mirrorRepositoryToDestination({ octokit, repository: repoData, diff --git a/src/lib/repo-utils.ts b/src/lib/repo-utils.ts index 0671c74..2042947 100644 --- a/src/lib/repo-utils.ts +++ b/src/lib/repo-utils.ts @@ -27,12 +27,14 @@ export function normalizeGitRepoToInsert( { userId, configId, - }: { userId: string; configId: string } + sourceId = null, + }: { userId: string; configId: string; sourceId?: string | null } ): RepoInsert { return { id: uuidv4(), userId, configId, + sourceId, name: repo.name, fullName: repo.fullName, normalizedFullName: repo.fullName.toLowerCase(), diff --git a/src/lib/repository-cleanup-service.ts b/src/lib/repository-cleanup-service.ts index 426732e..77678d0 100644 --- a/src/lib/repository-cleanup-service.ts +++ b/src/lib/repository-cleanup-service.ts @@ -7,7 +7,6 @@ import { db, configs, repositories } from '@/lib/db'; import { eq, and, or, sql, not, inArray } from 'drizzle-orm'; import { - createSourceProviderFromConfig, describeSource, getRepositorySource, isRepositoryFromConfiguredSource, @@ -104,161 +103,190 @@ export function splitFullName( async function identifyOrphanedRepositories(config: any): Promise { const userId = config.userId; - + try { - // The configured source host (GitHub honors GH_API_URL and tracks rate limits) - const sourceProvider = createSourceProviderFromConfig(config, { userId }); - const sourceConnection = sourceProvider.connection; - - let allGithubRepos = []; - let githubApiAccessible = true; - - try { - // Fetch GitHub data. Always include collaborator repos and bypass the - // organization allowlist here regardless of the user's import filters, - // otherwise repos previously mirrored as a collaborator or from an org the - // user later removed from the allowlist would be flagged as orphaned and - // archived/deleted as soon as the user narrows those filters. - const [basicAndForkedRepos, starredRepos] = await Promise.all([ - sourceProvider.listRepositories(config, { - includeCollaboratorReposOverride: true, - includeAllOrgsOverride: true, - }), - config.githubConfig?.includeStarred - ? sourceProvider.listStarredRepositories(config) - : Promise.resolve([]), - ]); - - allGithubRepos = [...basicAndForkedRepos, ...starredRepos]; - } catch (githubError: any) { - // Handle GitHub API errors gracefully - console.warn(`[Repository Cleanup] GitHub API error for user ${userId}: ${githubError.message}`); - - // Check if it's a critical error (like account deleted/banned) - if (githubError.status === 404 || githubError.status === 403) { - console.error(`[Repository Cleanup] CRITICAL: GitHub account may be deleted/banned. Skipping cleanup to prevent data loss.`); - console.error(`[Repository Cleanup] Consider using CLEANUP_ORPHANED_REPO_ACTION=archive instead of delete for safety.`); - - // Return empty array to skip cleanup entirely when GitHub account is inaccessible - return []; - } - - // For other errors, also skip cleanup to be safe - console.error(`[Repository Cleanup] Skipping cleanup due to GitHub API error. This prevents accidental deletion of backups.`); - return []; - } - - const githubReposByFullName = new Map( - allGithubRepos.map((repo) => [repo.fullName, repo] as const) - ); - - // Get all repositories from our database + // Multi-source: each enabled source vouches for the repositories that + // were imported from it. Orphans found by the sources are aggregated + // (deduped by repository id). + const { listSources } = await import('@/lib/sources'); + const { createSourceProviderFromSource } = await import('@/lib/source-providers'); + const sources = (await listSources(userId)).filter(source => source.enabled); + const knownSourceIds = new Set(sources.map(source => source.id)); + const dbRepos = await db .select() .from(repositories) .where(eq(repositories.userId, userId)); - - // Only identify repositories as orphaned if we successfully accessed GitHub - // This prevents false positives when GitHub is down or account is inaccessible. - // - // First pass (sync, cheap): filter down to repos that merely *look* - // orphaned based on map membership against the single bulk fetch above. - // This is the false-positive-prone signal — see resolveOrphanVerdict. - const candidateOrphans = dbRepos.filter(repo => { - // Skip repositories we've already archived/preserved - if (repo.status === 'archived' || repo.isArchived) { - console.log(`[Repository Cleanup] Skipping ${repo.fullName} - already archived`); + + // Legacy rows without a source, and rows whose source was deleted + // (dangling source id), keep mirroring on the destination side but + // cleanup leaves them alone (the documented "switched source" behavior). + const sourceOwnedRepos = dbRepos.filter(repo => { + if (!repo.sourceId) { + console.log(`[Repository Cleanup] Skipping ${repo.fullName} - not tied to any source`); return false; } - - // Only the configured source can vouch for a repository. Rows imported - // from another host (the source was switched since) are left alone. - if (!isRepositoryFromConfiguredSource(repo, sourceConnection)) { - console.log( - `[Repository Cleanup] Skipping ${repo.fullName} - imported from ${describeSource(getRepositorySource(repo))}, not the configured source` - ); + if (!knownSourceIds.has(repo.sourceId)) { + console.log(`[Repository Cleanup] Skipping ${repo.fullName} - its source no longer exists`); return false; } - - // If starred repos are not being fetched from GitHub, we can't determine - // if a starred repo is orphaned - skip it to prevent data loss - if (repo.isStarred && !config.githubConfig?.includeStarred) { - console.log(`[Repository Cleanup] Skipping starred repo ${repo.fullName} - starred repos not being fetched from GitHub`); - return false; - } - - const githubRepo = githubReposByFullName.get(repo.fullName); - if (!githubRepo) { - // Missing from the bulk list — candidate for direct confirmation below, - // not yet a confirmed orphan. - return true; - } - - if (!isMirrorableGitHubRepo(githubRepo)) { - console.log(`[Repository Cleanup] Preserving ${repo.fullName} - repository is disabled on GitHub`); - return false; - } - - return false; + return true; }); - if (candidateOrphans.length === 0) { - return []; - } + const orphanedById = new Map(); - // Second pass (async, targeted): confirm each candidate directly against - // GitHub before finalizing it as orphaned. This only adds extra API calls - // for the (presumably small) set of repos that look orphaned, not for - // every repo, so it shouldn't meaningfully increase rate-limit pressure - // in the common case (few or no orphans per run). Promise.allSettled so - // one repo's verification failure can't block the others. - const verificationOutcomes = await Promise.allSettled( - candidateOrphans.map(async (repo) => { - // Look the repository up by its full name: a GitLab project under a - // subgroup is stored with the top level group as owner, so owner/name - // would not resolve on the source host. - const upstreamPath = splitFullName(repo.fullName, repo.owner, repo.name); + for (const source of sources) { + // The source host with its token decrypted (GitHub honors GH_API_URL and tracks rate limits) + const sourceProvider = createSourceProviderFromSource(source, { userId }); + const sourceConnection = sourceProvider.connection; + + let allGithubRepos: any[] = []; + + try { + // Fetch source data. Always include collaborator repos and bypass the + // organization allowlist here regardless of the user's import filters, + // otherwise repos previously mirrored as a collaborator or from an org the + // user later removed from the allowlist would be flagged as orphaned and + // archived/deleted as soon as the user narrows those filters. + const [basicAndForkedRepos, starredRepos] = await Promise.all([ + sourceProvider.listRepositories(config, { + includeCollaboratorReposOverride: true, + includeAllOrgsOverride: true, + }), + config.githubConfig?.includeStarred + ? sourceProvider.listStarredRepositories(config) + : Promise.resolve([]), + ]); + + allGithubRepos = [...basicAndForkedRepos, ...starredRepos]; + } catch (githubError: any) { + // Handle source API errors gracefully + console.warn(`[Repository Cleanup] Source API error for user ${userId} on source ${source.name}: ${githubError.message}`); + + // Check if it's a critical error (like account deleted/banned) + if (githubError.status === 404 || githubError.status === 403) { + console.error(`[Repository Cleanup] CRITICAL: the account on source ${source.name} may be deleted/banned. Skipping cleanup for this source to prevent data loss.`); + console.error(`[Repository Cleanup] Consider using CLEANUP_ORPHANED_REPO_ACTION=archive instead of delete for safety.`); + } else { + // For other errors, also skip this source to be safe + console.error(`[Repository Cleanup] Skipping cleanup of source ${source.name} due to a source API error. This prevents accidental deletion of backups.`); + } + continue; + } + + const githubReposByFullName = new Map( + allGithubRepos.map((repo) => [repo.fullName, repo] as const) + ); + + // Only identify repositories as orphaned if we successfully accessed the + // source. This prevents false positives when the source is down or the + // account is inaccessible. + // + // First pass (sync, cheap): filter down to repos that merely *look* + // orphaned based on map membership against the single bulk fetch above. + // This is the false-positive-prone signal — see resolveOrphanVerdict. + const candidateOrphans = sourceOwnedRepos.filter(repo => { + // Skip repositories we've already archived/preserved + if (repo.status === 'archived' || repo.isArchived) { + console.log(`[Repository Cleanup] Skipping ${repo.fullName} - already archived`); + return false; + } + + // Only this source can vouch for its own rows; other sources' rows + // are handled by their source's iteration. + if (repo.sourceId !== source.id) { + return false; + } + + // The row must still match the source it was imported from (the + // source host cannot vouch for a repository imported from another + // provider/instance before it was switched). + if (!isRepositoryFromConfiguredSource(repo, sourceConnection)) { + console.log( + `[Repository Cleanup] Skipping ${repo.fullName} - imported from ${describeSource(getRepositorySource(repo))}, not from source ${source.name}` + ); + return false; + } + + // If starred repos are not being fetched from the source, we can't determine + // if a starred repo is orphaned - skip it to prevent data loss + if (repo.isStarred && !config.githubConfig?.includeStarred) { + console.log(`[Repository Cleanup] Skipping starred repo ${repo.fullName} - starred repos not being fetched from the source`); + return false; + } + + const githubRepo = githubReposByFullName.get(repo.fullName); + if (!githubRepo) { + // Missing from the bulk list — candidate for direct confirmation below, + // not yet a confirmed orphan. + return true; + } + + if (!isMirrorableGitHubRepo(githubRepo)) { + console.log(`[Repository Cleanup] Preserving ${repo.fullName} - repository is disabled on the source`); + return false; + } + + return false; + }); + + if (candidateOrphans.length === 0) { + continue; + } + + // Second pass (async, targeted): confirm each candidate directly against + // the source before finalizing it as orphaned. This only adds extra API + // calls for the (presumably small) set of repos that look orphaned, not + // for every repo, so it shouldn't meaningfully increase rate-limit + // pressure in the common case (few or no orphans per run). + // Promise.allSettled so one repo's verification failure can't block the + // others. + const verificationOutcomes = await Promise.allSettled( + candidateOrphans.map(async (repo) => { + // Look the repository up by its full name: a GitLab project under a + // subgroup is stored with the top level group as owner, so owner/name + // would not resolve on the source host. + const upstreamPath = splitFullName(repo.fullName, repo.owner, repo.name); + + if (repo.isStarred) { + try { + const stillStarred = await sourceProvider.isRepositoryStarred( + upstreamPath.owner, + upstreamPath.name + ); + // Still starred => the bulk star fetch missed it. Fail safe: do + // not treat as orphaned. A clean "not starred" confirms it. + return { repo, directCheckConfirmsGone: !stillStarred }; + } catch (starError: any) { + console.warn( + `[Repository Cleanup] Direct star-check for ${repo.fullName} failed with a non-404 error; skipping this cycle to be safe: ${ + starError instanceof Error ? starError.message : String(starError) + }` + ); + return { repo, directCheckConfirmsGone: false }; + } + } - if (repo.isStarred) { try { - const stillStarred = await sourceProvider.isRepositoryStarred( + const upstream = await sourceProvider.getRepository( upstreamPath.owner, upstreamPath.name ); - // Still starred => the bulk star fetch missed it. Fail safe: do - // not treat as orphaned. A clean "not starred" confirms it. - return { repo, directCheckConfirmsGone: !stillStarred }; - } catch (starError: any) { + // Present => the bulk fetch missed it (e.g. an org-allowlist edge + // case). Fail safe: not orphaned. A clean 404 (null) confirms it. + return { repo, directCheckConfirmsGone: upstream === null }; + } catch (repoError: any) { console.warn( - `[Repository Cleanup] Direct star-check for ${repo.fullName} failed with a non-404 error; skipping this cycle to be safe: ${ - starError instanceof Error ? starError.message : String(starError) + `[Repository Cleanup] Direct existence check for ${repo.fullName} failed with a non-404 error; skipping this cycle to be safe: ${ + repoError instanceof Error ? repoError.message : String(repoError) }` ); return { repo, directCheckConfirmsGone: false }; } - } + }) + ); - try { - const upstream = await sourceProvider.getRepository( - upstreamPath.owner, - upstreamPath.name - ); - // Present => the bulk fetch missed it (e.g. an org-allowlist edge - // case). Fail safe: not orphaned. A clean 404 (null) confirms it. - return { repo, directCheckConfirmsGone: upstream === null }; - } catch (repoError: any) { - console.warn( - `[Repository Cleanup] Direct existence check for ${repo.fullName} failed with a non-404 error; skipping this cycle to be safe: ${ - repoError instanceof Error ? repoError.message : String(repoError) - }` - ); - return { repo, directCheckConfirmsGone: false }; - } - }) - ); - - const orphanedRepos = verificationOutcomes - .map((outcome, index) => { + verificationOutcomes.forEach((outcome, index) => { if (outcome.status !== 'fulfilled') { const repo = candidateOrphans[index]; console.warn( @@ -266,7 +294,7 @@ async function identifyOrphanedRepositories(config: any): Promise { outcome.reason instanceof Error ? outcome.reason.message : String(outcome.reason) }` ); - return null; + return; } const { repo, directCheckConfirmsGone } = outcome.value; @@ -276,16 +304,17 @@ async function identifyOrphanedRepositories(config: any): Promise { }); if (!isOrphaned) { - return null; + return; } console.log( - `[Repository Cleanup] Confirmed orphaned via direct GitHub check: ${repo.fullName}` + `[Repository Cleanup] Confirmed orphaned via direct source check: ${repo.fullName}` ); - return repo; - }) - .filter((repo): repo is (typeof dbRepos)[number] => repo !== null); + orphanedById.set(repo.id, repo); + }); + } + const orphanedRepos = Array.from(orphanedById.values()); if (orphanedRepos.length > 0) { console.log(`[Repository Cleanup] Found ${orphanedRepos.length} orphaned repositories for user ${userId}`); } diff --git a/src/lib/scheduler-service.ts b/src/lib/scheduler-service.ts index 96e90f0..29f6fc7 100644 --- a/src/lib/scheduler-service.ts +++ b/src/lib/scheduler-service.ts @@ -19,6 +19,7 @@ import { loadOrganizationForkPolicies, orgForkSkipDecision } from '@/lib/utils/m import { createMirrorJob } from '@/lib/helpers'; import { getNextScheduledRun, isCronExpression, normalizeTimezone } from '@/lib/utils/schedule-utils'; import { resetStuckMirrorStatuses } from '@/lib/stuck-status-recovery'; +import type { SourceRecord } from '@/lib/sources'; let schedulerInterval: NodeJS.Timeout | null = null; let isSchedulerRunning = false; @@ -78,6 +79,166 @@ async function persistScheduleRunState(config: any, currentTime: Date, forceEnab return nextRun; } +/** + * Auto-discovery: import new repositories from every enabled source. Shared + * by the scheduled-sync phase and the boot auto-start phase. A failing + * source is logged and skipped so the others still import. + */ +async function importRepositoriesFromSources( + config: any, + userId: string, + phase: 'scheduled sync' | 'auto-start' +): Promise { + const { listSources, ensureSourcesFromConfig } = await import('@/lib/sources'); + const { createSourceProviderFromSource } = await import('@/lib/source-providers'); + // Configs written outside the sources API must still import: seed the + // first source row when the config carries connection fields but no + // source exists yet. + await ensureSourcesFromConfig(userId); + const sources = (await listSources(userId)).filter(source => source.enabled); + + // Per-organization fork pins, so orgs opted out of forks stay that way + // during auto import. Loaded once; the pins apply to every source. + const orgForkOverrides = await loadOrganizationForkPolicies({ userId }); + + const existingRepos = await db + .select({ normalizedFullName: repositories.normalizedFullName, sourceId: repositories.sourceId }) + .from(repositories) + .where(eq(repositories.userId, userId)); + // Repositories are unique per source: the same full name under two sources + // counts as two repositories. + const existingRepoKeys = new Set( + existingRepos.map(r => `${r.sourceId ?? ''}|${r.normalizedFullName}`) + ); + + for (const source of sources) { + try { + const sourceProvider = createSourceProviderFromSource(source, { userId }); + + const [basicAndForkedRepos, starredRepos] = await Promise.all([ + sourceProvider.listRepositories(config, { orgForkOverrides }), + config.githubConfig?.includeStarred + ? sourceProvider.listStarredRepositories(config) + : Promise.resolve([]), + ]); + const allGithubRepos = mergeGitReposPreferStarred(basicAndForkedRepos, starredRepos); + const mirrorableGithubRepos = allGithubRepos.filter(isMirrorableGitHubRepo); + + const newRepos = mirrorableGithubRepos.filter( + r => !existingRepoKeys.has(`${source.id}|${r.fullName.toLowerCase()}`) + ); + + if (newRepos.length > 0) { + console.log(`[Scheduler] Found ${newRepos.length} new repositories for user ${userId} on source ${source.name}`); + + const reposToInsert = newRepos.map(repo => + normalizeGitRepoToInsert(repo, { userId, configId: config.id, sourceId: source.id }) + ); + + // Batch insert to avoid SQLite parameter limit + const sample = reposToInsert[0]; + const columnCount = Object.keys(sample ?? {}).length || 1; + const BATCH_SIZE = calcBatchSizeForInsert(columnCount); + for (let i = 0; i < reposToInsert.length; i += BATCH_SIZE) { + const batch = reposToInsert.slice(i, i + BATCH_SIZE); + await db + .insert(repositories) + .values(batch) + .onConflictDoNothing({ target: [repositories.userId, repositories.sourceId, repositories.normalizedFullName] }); + } + console.log(`[Scheduler] Successfully imported ${newRepos.length} new repositories for user ${userId} from source ${source.name}`); + + for (const repo of newRepos) { + const sourceLabel = repo.isStarred ? 'starred' : 'owned'; + await createMirrorJob({ + userId, + repositoryName: repo.fullName, + message: `Auto-imported ${sourceLabel} repository: ${repo.fullName}`, + details: `Repository ${repo.fullName} was discovered and imported during ${phase}.`, + status: 'imported', + skipDuplicateEvent: true, + }); + } + } else { + console.log(`[Scheduler] No new repositories found for user ${userId} on source ${source.name}`); + } + const skippedDisabledCount = allGithubRepos.length - mirrorableGithubRepos.length; + if (skippedDisabledCount > 0) { + console.log(`[Scheduler] Skipped ${skippedDisabledCount} disabled repositories for user ${userId} on source ${source.name}`); + } + } catch (sourceError) { + console.error(`[Scheduler] Failed to import repositories from source ${source.name} for user ${userId} during ${phase}:`, sourceError); + } + } +} + +/** + * Resolve the GitHub API client for a repository's own source. Only GitHub + * sources need a client while mirroring (metadata); other hosts get + * code-only mirrors. Repositories whose source row no longer resolves keep + * the pre-multi-source behavior: the client built from the config, when the + * configured provider is GitHub. Clients are cached per source for the + * batch loop. + */ +async function createOctokitResolverForSources( + config: any, + userId: string, + sources: SourceRecord[] +): Promise<(repo: { sourceId?: string | null }) => Promise> { + const { decryptSourceToken, resolveGitHubApiBaseUrl } = await import('@/lib/sources'); + const { createGitHubClient } = await import('@/lib/github'); + + const sourcesById = new Map(sources.map(source => [source.id, source])); + const clientsBySourceId = new Map(); + let configClient: Octokit | null | undefined; + + return async (repo) => { + const source = sourcesById.get(repo.sourceId ?? ''); + if (source) { + if (!clientsBySourceId.has(source.id)) { + if (source.provider === 'github') { + // A tokenless source has no usable client: auth:"" 401s on every + // metadata call. Return null so the loop skips these repos, the + // same way the job and recovery resolvers do. + const sourceToken = decryptSourceToken(source.token); + clientsBySourceId.set( + source.id, + sourceToken + ? createGitHubClient( + sourceToken, + userId, + source.username, + resolveGitHubApiBaseUrl(source.url) + ) + : null + ); + } else { + clientsBySourceId.set(source.id, null); + } + } + return clientsBySourceId.get(source.id) ?? null; + } + + if (configClient === undefined) { + const { resolveSourceProviderKind } = await import('@/lib/source-providers'); + if (resolveSourceProviderKind(config) === 'github') { + configClient = createGitHubClient(getDecryptedGitHubToken(config), userId, config.githubConfig?.owner); + } else { + configClient = null; + } + } + return configClient; + }; +} + +/** + * The lower-cased username of each source, for classifying starred repos by + * the owner of the repository's own source. + */ +function sourceUsernamesBySourceId(sources: SourceRecord[]): Map { + return new Map(sources.map(source => [source.id, (source.username || '').toLowerCase()])); +} + /** * Run scheduled mirror sync for a single user configuration */ @@ -99,79 +260,11 @@ async function runScheduledSync(config: any): Promise { console.log(`[Scheduler] Using schedule source for user ${userId}: ${String(source)} (timezone=${timezone})`); await persistScheduleRunState(config, currentTime); - // Auto-discovery: Check for new GitHub repositories + // Auto-discovery: Check for new repositories on every enabled source if (scheduleConfig.autoImport !== false) { console.log(`[Scheduler] Checking for new source repositories for user ${userId}...`); try { - const { createSourceProviderFromConfig } = await import('@/lib/source-providers'); - const { v4: uuidv4 } = await import('uuid'); - - // The configured source host (GitHub honors GH_API_URL for GHES / GHEC) - const sourceProvider = createSourceProviderFromConfig(config, { userId }); - - // Per-organization fork pins, so orgs opted out of forks stay that - // way during scheduled auto-import. - const orgForkOverrides = await loadOrganizationForkPolicies({ userId }); - - // Fetch source data - const [basicAndForkedRepos, starredRepos] = await Promise.all([ - sourceProvider.listRepositories(config, { orgForkOverrides }), - config.githubConfig?.includeStarred - ? sourceProvider.listStarredRepositories(config) - : Promise.resolve([]), - ]); - const allGithubRepos = mergeGitReposPreferStarred(basicAndForkedRepos, starredRepos); - const mirrorableGithubRepos = allGithubRepos.filter(isMirrorableGitHubRepo); - - // Check for new repositories - const existingRepos = await db - .select({ normalizedFullName: repositories.normalizedFullName }) - .from(repositories) - .where(eq(repositories.userId, userId)); - - const existingRepoNames = new Set(existingRepos.map(r => r.normalizedFullName)); - const newRepos = mirrorableGithubRepos.filter(r => !existingRepoNames.has(r.fullName.toLowerCase())); - - if (newRepos.length > 0) { - console.log(`[Scheduler] Found ${newRepos.length} new repositories for user ${userId}`); - - // Insert new repositories - const reposToInsert = newRepos.map(repo => - normalizeGitRepoToInsert(repo, { userId, configId: config.id }) - ); - - // Batch insert to avoid SQLite parameter limit - const sample = reposToInsert[0]; - const columnCount = Object.keys(sample ?? {}).length || 1; - const BATCH_SIZE = calcBatchSizeForInsert(columnCount); - for (let i = 0; i < reposToInsert.length; i += BATCH_SIZE) { - const batch = reposToInsert.slice(i, i + BATCH_SIZE); - await db - .insert(repositories) - .values(batch) - .onConflictDoNothing({ target: [repositories.userId, repositories.normalizedFullName] }); - } - console.log(`[Scheduler] Successfully imported ${newRepos.length} new repositories for user ${userId}`); - - // Log activity for each newly imported repo - for (const repo of newRepos) { - const sourceLabel = repo.isStarred ? 'starred' : 'owned'; - await createMirrorJob({ - userId, - repositoryName: repo.fullName, - message: `Auto-imported ${sourceLabel} repository: ${repo.fullName}`, - details: `Repository ${repo.fullName} was discovered and imported during scheduled sync.`, - status: 'imported', - skipDuplicateEvent: true, - }); - } - } else { - console.log(`[Scheduler] No new repositories found for user ${userId}`); - } - const skippedDisabledCount = allGithubRepos.length - mirrorableGithubRepos.length; - if (skippedDisabledCount > 0) { - console.log(`[Scheduler] Skipped ${skippedDisabledCount} disabled GitHub repositories for user ${userId}`); - } + await importRepositoriesFromSources(config, userId, 'scheduled sync'); } catch (error) { console.error(`[Scheduler] Failed to auto-import repositories for user ${userId}:`, error); } @@ -231,11 +324,21 @@ async function runScheduledSync(config: any): Promise { ) ); - const githubOwner = (config.githubConfig?.owner || '').toLowerCase(); + const { listSources, findSourceForRepository } = await import('@/lib/sources'); + const userSources = await listSources(userId); + const usernameBySourceId = sourceUsernamesBySourceId(userSources); + const configOwner = (config.githubConfig?.owner || '').toLowerCase(); const beforeCount = reposNeedingMirror.length; reposNeedingMirror = reposNeedingMirror.filter(repo => { - // GitHub usernames are case-insensitive; lowercase both sides to avoid misclassifying self-starred repos. - const isStarredFromOther = repo.isStarred && repo.owner.toLowerCase() !== githubOwner; + // Starred repos are classified by the owner of the repository's own + // source; rows without a resolvable source keep the configured + // owner, as before multiple sources. Usernames are lower-cased on + // both sides because they are case-insensitive. + const sourceOwner = + usernameBySourceId.get(repo.sourceId ?? '') ?? + findSourceForRepository(repo, userSources)?.username.toLowerCase() ?? + configOwner; + const isStarredFromOther = repo.isStarred && repo.owner.toLowerCase() !== sourceOwner; return isStarredFromOther ? autoMirrorStarred : autoMirrorOwned; }); const skippedCount = beforeCount - reposNeedingMirror.length; @@ -258,15 +361,8 @@ async function runScheduledSync(config: any): Promise { if (reposNeedingMirror.length > 0) { console.log(`[Scheduler] Found ${reposNeedingMirror.length} repositories that need initial mirroring`); - // Only GitHub sources need an API client while mirroring (metadata). - // Other hosts get code-only mirrors through Gitea. - const { resolveSourceProviderKind } = await import('@/lib/source-providers'); - let octokit: Octokit | null = null; - if (resolveSourceProviderKind(config) === 'github') { - const decryptedToken = getDecryptedGitHubToken(config); - const { createGitHubClient } = await import('@/lib/github'); - octokit = createGitHubClient(decryptedToken, userId, config.githubConfig?.owner); - } + // Each repository is mirrored with the API client of its own source. + const resolveOctokit = await createOctokitResolverForSources(config, userId, userSources); // Process repositories in batches const batchSize = scheduleConfig.batchSize || 10; @@ -310,7 +406,7 @@ async function runScheduledSync(config: any): Promise { } } - await mirrorRepositoryToDestination({ octokit, repository, config }); + await mirrorRepositoryToDestination({ octokit: await resolveOctokit(repo), repository, config }); console.log(`[Scheduler] Auto-mirrored repository: ${repo.fullName}`); } catch (error) { console.error(`[Scheduler] Failed to auto-mirror repository ${repo.fullName}:`, error); @@ -531,77 +627,9 @@ async function performInitialAutoStart(): Promise { console.log(`[Scheduler] Auto-starting for user ${config.userId}...`); try { - // Step 1: Import repositories from GitHub - console.log(`[Scheduler] Step 1: Importing repositories from the configured source for user ${config.userId}...`); - const { createSourceProviderFromConfig } = await import('@/lib/source-providers'); - const { v4: uuidv4 } = await import('uuid'); - - // The configured source host (GitHub honors GH_API_URL for GHES / GHEC) - const sourceProvider = createSourceProviderFromConfig(config, { userId: config.userId }); - - // Per-organization fork pins, so orgs opted out of forks stay that - // way during boot auto-start imports. - const orgForkOverrides = await loadOrganizationForkPolicies({ userId: config.userId }); - - // Fetch source data - const [basicAndForkedRepos, starredRepos] = await Promise.all([ - sourceProvider.listRepositories(config, { orgForkOverrides }), - config.githubConfig?.includeStarred - ? sourceProvider.listStarredRepositories(config) - : Promise.resolve([]), - ]); - const allGithubRepos = mergeGitReposPreferStarred(basicAndForkedRepos, starredRepos); - const mirrorableGithubRepos = allGithubRepos.filter(isMirrorableGitHubRepo); - - // Check for new repositories - const existingRepos = await db - .select({ normalizedFullName: repositories.normalizedFullName }) - .from(repositories) - .where(eq(repositories.userId, config.userId)); - - const existingRepoNames = new Set(existingRepos.map(r => r.normalizedFullName)); - const reposToImport = mirrorableGithubRepos.filter(r => !existingRepoNames.has(r.fullName.toLowerCase())); - - if (reposToImport.length > 0) { - console.log(`[Scheduler] Importing ${reposToImport.length} repositories for user ${config.userId}...`); - - // Insert new repositories - const reposToInsert = reposToImport.map(repo => - normalizeGitRepoToInsert(repo, { userId: config.userId, configId: config.id }) - ); - - // Batch insert to avoid SQLite parameter limit - const sample = reposToInsert[0]; - const columnCount = Object.keys(sample ?? {}).length || 1; - const BATCH_SIZE = calcBatchSizeForInsert(columnCount); - for (let i = 0; i < reposToInsert.length; i += BATCH_SIZE) { - const batch = reposToInsert.slice(i, i + BATCH_SIZE); - await db - .insert(repositories) - .values(batch) - .onConflictDoNothing({ target: [repositories.userId, repositories.normalizedFullName] }); - } - console.log(`[Scheduler] Successfully imported ${reposToImport.length} repositories`); - - // Log activity for each newly imported repo - for (const repo of reposToImport) { - const sourceLabel = repo.isStarred ? 'starred' : 'owned'; - await createMirrorJob({ - userId: config.userId, - repositoryName: repo.fullName, - message: `Auto-imported ${sourceLabel} repository: ${repo.fullName}`, - details: `Repository ${repo.fullName} was discovered and imported during auto-start.`, - status: 'imported', - skipDuplicateEvent: true, - }); - } - } else { - console.log(`[Scheduler] No new repositories to import for user ${config.userId}`); - } - const skippedDisabledCount = allGithubRepos.length - mirrorableGithubRepos.length; - if (skippedDisabledCount > 0) { - console.log(`[Scheduler] Skipped ${skippedDisabledCount} disabled GitHub repositories for user ${config.userId}`); - } + // Step 1: Import repositories from every enabled source + console.log(`[Scheduler] Step 1: Importing repositories from the configured sources for user ${config.userId}...`); + await importRepositoriesFromSources(config, config.userId, 'auto-start'); // Check if we already have mirrored repositories (indicating this isn't first run) const mirroredRepos = await db @@ -659,11 +687,21 @@ async function performInitialAutoStart(): Promise { ) ); - const githubOwner = (config.githubConfig?.owner || '').toLowerCase(); + const { listSources, findSourceForRepository } = await import('@/lib/sources'); + const userSources = await listSources(config.userId); + const usernameBySourceId = sourceUsernamesBySourceId(userSources); + const configOwner = (config.githubConfig?.owner || '').toLowerCase(); const beforeCount = reposNeedingMirror.length; reposNeedingMirror = reposNeedingMirror.filter(repo => { - // GitHub usernames are case-insensitive; lowercase both sides to avoid misclassifying self-starred repos. - const isStarredFromOther = repo.isStarred && repo.owner.toLowerCase() !== githubOwner; + // Starred repos are classified by the owner of the repository's own + // source; rows without a resolvable source keep the configured + // owner, as before multiple sources. Usernames are lower-cased on + // both sides because they are case-insensitive. + const sourceOwner = + usernameBySourceId.get(repo.sourceId ?? '') ?? + findSourceForRepository(repo, userSources)?.username.toLowerCase() ?? + configOwner; + const isStarredFromOther = repo.isStarred && repo.owner.toLowerCase() !== sourceOwner; return isStarredFromOther ? autoMirrorStarred : autoMirrorOwned; }); const skippedCount = beforeCount - reposNeedingMirror.length; @@ -673,23 +711,16 @@ async function performInitialAutoStart(): Promise { if (reposNeedingMirror.length > 0) { console.log(`[Scheduler] Found ${reposNeedingMirror.length} repositories that need mirroring`); - - // Only GitHub sources need an API client while mirroring (metadata). - // Other hosts get code-only mirrors through Gitea. - const { resolveSourceProviderKind } = await import('@/lib/source-providers'); - let octokit: Octokit | null = null; - if (resolveSourceProviderKind(config) === 'github') { - const decryptedToken = getDecryptedGitHubToken(config); - const { createGitHubClient } = await import('@/lib/github'); - octokit = createGitHubClient(decryptedToken, config.userId, config.githubConfig?.owner); - } - + + // Each repository is mirrored with the API client of its own source. + const resolveOctokit = await createOctokitResolverForSources(config, config.userId, userSources); + // Process repositories in batches const batchSize = config.scheduleConfig?.batchSize || 5; for (let i = 0; i < reposNeedingMirror.length; i += batchSize) { const batch = reposNeedingMirror.slice(i, Math.min(i + batchSize, reposNeedingMirror.length)); console.log(`[Scheduler] Processing batch ${Math.floor(i / batchSize) + 1} of ${Math.ceil(reposNeedingMirror.length / batchSize)} (${batch.length} repos)`); - + await Promise.all( batch.map(async (repo) => { try { @@ -703,9 +734,9 @@ async function performInitialAutoStart(): Promise { forkedFrom: repo.forkedFrom ?? undefined, visibility: repositoryVisibilityEnum.parse(repo.visibility), }; - - await mirrorRepositoryToDestination({ - octokit, + + await mirrorRepositoryToDestination({ + octokit: await resolveOctokit(repo), repository, config }); diff --git a/src/lib/source-provider-wiring.test.ts b/src/lib/source-provider-wiring.test.ts index 2ae3404..6c4e822 100644 --- a/src/lib/source-provider-wiring.test.ts +++ b/src/lib/source-provider-wiring.test.ts @@ -2,11 +2,12 @@ * Source regression tests for the source provider wiring (issue #375). * * The mirror pipeline, the scheduler, the cleanup service and the discovery - * routes must go through the SourceProvider built from the configuration - * instead of calling GitHub directly, and the two migrate sites must pick - * clone credentials per provider and refuse a repository from another host. - * These files need process-wide module mocks to exercise behaviorally, so - * the wiring is asserted by reading the source (same convention as + * routes must go through the SourceProvider built from the user's connected + * source rows instead of calling GitHub directly, and the two migrate sites + * must pick clone credentials from the repository's own source and refuse a + * repository from a host that is not connected. These files need + * process-wide module mocks to exercise behaviorally, so the wiring is + * asserted by reading the source (same convention as * stuck-status-recovery.test.ts). */ import { describe, expect, test } from "bun:test"; @@ -27,15 +28,20 @@ describe("gitea.ts migrate sites", () => { expect(count(source, "provider: sourceConnection.provider,")).toBe(2); }); - test("both mirror paths refuse a repository from a different source before any side effect", () => { - expect(count(source, "assertRepositoryMatchesConfiguredSource({ repository, config });")).toBe(2); + test("both mirror paths refuse a repository from an unconnected source before any side effect", () => { + expect(count(source, "assertRepositoryMatchesConnectedSource({ repository, sources });")).toBe(2); const userPath = source.indexOf("export const mirrorGithubRepoToGitea"); - const guard = source.indexOf("assertRepositoryMatchesConfiguredSource({ repository, config });", userPath); + const guard = source.indexOf("assertRepositoryMatchesConnectedSource({ repository, sources });", userPath); const mirroringWrite = source.indexOf('repoStatusEnum.parse("mirroring")', userPath); expect(guard).toBeGreaterThan(userPath); expect(guard).toBeLessThan(mirroringWrite); }); + test("both mirror paths resolve the repository's own source for clone credentials", () => { + expect(count(source, "findSourceForRepository(repository, sources)")).toBe(2); + expect(count(source, "sourceConnectionFromSource(repoSource, { userId: config.userId })")).toBe(2); + }); + test("the mirror entry points accept a null GitHub client and skip metadata without one", () => { expect(count(source, "octokit: Octokit | null;")).toBeGreaterThanOrEqual(4); expect(count(source, "&& octokit !== null;")).toBe(10); @@ -43,18 +49,21 @@ describe("gitea.ts migrate sites", () => { }); describe("discovery and housekeeping go through the source provider", () => { - test("scheduler auto import, auto mirror and boot auto start", () => { + test("scheduler auto import, auto mirror and boot auto start iterate the connected sources", () => { const source = read("scheduler-service.ts"); - expect(count(source, "createSourceProviderFromConfig(config")).toBe(2); - expect(count(source, "sourceProvider.listRepositories(")).toBe(2); + expect(source).toContain("createSourceProviderFromSource(source, { userId })"); + // Auto import passes the per-organization fork pins through every source. + expect(source).toContain("sourceProvider.listRepositories(config, { orgForkOverrides })"); expect(source).not.toContain("getGithubRepositories("); - // A GitHub client is built only for GitHub sources, in both mirror phases. - expect(count(source, "resolveSourceProviderKind(config) === 'github'")).toBe(2); + // Auto import, auto mirror and boot auto start each load the user's sources. + expect(count(source, "listSources(")).toBeGreaterThanOrEqual(3); + // A GitHub client is built per source (only for GitHub sources), cached per batch. + expect(source).toContain("source.provider === 'github'"); }); test("cleanup lists, filters and verifies through the provider", () => { const source = read("repository-cleanup-service.ts"); - expect(source).toContain("createSourceProviderFromConfig(config, { userId })"); + expect(source).toContain("createSourceProviderFromSource(source, { userId })"); expect(source).toContain("isRepositoryFromConfiguredSource(repo, sourceConnection)"); expect(source).toContain("sourceProvider.getRepository("); expect(source).toContain("sourceProvider.isRepositoryStarred("); @@ -63,10 +72,13 @@ describe("discovery and housekeeping go through the source provider", () => { expect(source).toContain("splitFullName(repo.fullName, repo.owner, repo.name)"); }); - test("recovery and the sync path only build a GitHub client for GitHub sources", () => { - expect(read("recovery.ts")).toContain("resolveSourceProviderKind(config) === 'github'"); + test("recovery and the sync path resolve the GitHub client from the repository's own source", () => { + const recovery = read("recovery.ts"); + expect(recovery).toContain("findSourceForRepository(repo, sources)"); + expect(recovery).toContain("repoSource?.provider === 'github'"); const enhanced = read("gitea-enhanced.ts"); - expect(count(enhanced, 'normalizeSourceProviderKind(config.githubConfig?.provider)')).toBe(2); + expect(enhanced).toContain("getRepositorySource(repository).provider === \"github\""); + expect(enhanced).toContain("findSourceForRepository(repository, sources)"); }); test("the metadata resolver clamps GitHub only options for other sources", () => { @@ -75,12 +87,16 @@ describe("discovery and housekeeping go through the source provider", () => { expect(source).toContain('normalizeSourceProviderKind(repository.sourceProvider) !== "github"'); }); - test("the discovery routes use the provider and stamp the source on inserted rows", () => { + test("the discovery routes use the provider, match the pasted host against connected sources and stamp the source on inserted rows", () => { for (const route of ["index.ts", "organization.ts", "repository.ts"]) { const source = read("..", "pages", "api", "sync", route); - expect(source).toContain("createSourceProviderFromConfig(config, { userId })"); + expect(source).toContain("createSourceProviderFromSource(source, { userId })"); expect(source).not.toContain("createGitHubClient("); } + expect(read("..", "pages", "api", "sync", "repository.ts")).toContain( + "sourceHostOf(s.url) === pastedHost" + ); + expect(read("..", "pages", "api", "sync", "index.ts")).toContain("sourceId: source.id"); expect(read("repo-utils.ts")).toContain("repositorySourceColumns(repo)"); }); }); diff --git a/src/lib/source-providers/github-source.ts b/src/lib/source-providers/github-source.ts index 21e3689..862d54f 100644 --- a/src/lib/source-providers/github-source.ts +++ b/src/lib/source-providers/github-source.ts @@ -14,6 +14,7 @@ import { getGithubRepositories, getGithubStarredRepositories, } from "@/lib/github"; +import { resolveGitHubApiBaseUrl } from "@/lib/sources"; import type { GitRepo, RepositoryVisibility } from "@/types/Repository"; import type { GitOrg } from "@/types/organizations"; import { toDate } from "./http"; @@ -112,13 +113,17 @@ export class GitHubSourceProvider implements SourceProvider { readonly octokit: Octokit; constructor(readonly connection: SourceConnection) { + // A GitHub Enterprise source URL drives the API base; github.com (the + // default) keeps the GH_API_URL env default. + const apiBaseUrl = resolveGitHubApiBaseUrl(connection.url); this.octokit = connection.token.trim() ? createGitHubClient( connection.token, connection.userId, - connection.username || undefined + connection.username || undefined, + apiBaseUrl ) - : new Octokit({ baseUrl: githubApiBaseUrl() }); + : new Octokit({ baseUrl: apiBaseUrl || githubApiBaseUrl() }); } private stamp(repos: GitRepo[]): GitRepo[] { diff --git a/src/lib/source-providers/index.ts b/src/lib/source-providers/index.ts index b25b72d..6180811 100644 --- a/src/lib/source-providers/index.ts +++ b/src/lib/source-providers/index.ts @@ -8,6 +8,11 @@ */ import type { Config, Repository } from "@/lib/db/schema"; import { getDecryptedGitHubToken } from "@/lib/utils/config-encryption"; +import { + decryptSourceToken, + findSourceForRepository, + type SourceRecord, +} from "@/lib/sources"; import { GiteaSourceProvider } from "./gitea-source"; import { GitHubSourceProvider } from "./github-source"; import { GitLabSourceProvider } from "./gitlab"; @@ -69,6 +74,28 @@ export function createSourceProviderFromConfig( return createSourceProvider(resolveSourceConnection(config, { token, userId })); } +/** Build the connection details for a stored source row. The token is decrypted. */ +export function sourceConnectionFromSource( + source: SourceRecord, + { userId }: { userId?: string } = {} +): SourceConnection { + return { + provider: source.provider, + url: source.url, + username: source.username, + token: decryptSourceToken(source.token), + userId: userId ?? source.userId, + }; +} + +/** Build the provider for a stored source row, decrypting its token. */ +export function createSourceProviderFromSource( + source: SourceRecord, + { userId }: { userId?: string } = {} +): SourceProvider { + return createSourceProvider(sourceConnectionFromSource(source, { userId })); +} + /** * Refuse to mirror a repository with credentials for a different host. * @@ -93,3 +120,28 @@ export function assertRepositoryMatchesConfiguredSource({ "Switch the source back, or remove the repository and add it again from the current source." ); } + +/** + * Multi-source form of assertRepositoryMatchesConfiguredSource: refuse a + * repository whose host matches none of the connected source rows. The + * credentials of the matched source are the ones the pipeline must use. + */ +export function assertRepositoryMatchesConnectedSource({ + repository, + sources, +}: { + repository: Pick & RepositorySourceFields & { sourceId?: string | null }; + sources: SourceRecord[]; +}): void { + if (findSourceForRepository(repository, sources)) return; + + const connected = sources.length + ? sources.map((source) => describeSource(source)).join(", ") + : "no source"; + throw new Error( + `Repository ${repository.fullName} was imported from ${describeSource( + getRepositorySource(repository) + )} but the connected sources are ${connected}. ` + + "Add a source for that host, or remove the repository and add it again from a connected source." + ); +} diff --git a/src/lib/sources-api.ts b/src/lib/sources-api.ts new file mode 100644 index 0000000..3ab1fe9 --- /dev/null +++ b/src/lib/sources-api.ts @@ -0,0 +1,102 @@ +/** + * Shared shaping for the /api/sources endpoints and the sources array the + * configuration GET returns: source rows from the service layer joined with + * per-source repository counts, tokens decrypted like every other config + * endpoint. + * + * The database and service modules are imported lazily (the same pattern as + * loadConfigLocks) so per-test-file module mocks keep working. + */ +import { eq, sql } from "drizzle-orm"; +import type { SourceProviderKind } from "@/lib/source-providers/kinds"; + +/** One source as the API returns it: the stored row plus its lock state. */ +export interface SourceApiItem { + id: string; + name: string; + provider: SourceProviderKind; + url: string; + username: string; + token: string; + enabled: boolean; + repositoryCount: number; + locked: boolean; +} + +/** The source fields the service layer hands back (the SourceRecord shape). */ +export interface SourceRowLike { + id: string; + name: string; + provider: SourceProviderKind; + url: string | null; + username: string; + token: string | null; + enabled: boolean; +} + +export async function toSourceApiItem( + source: SourceRowLike, + repositoryCount: number +): Promise { + const { decryptSourceToken } = await import("@/lib/sources"); + return { + id: source.id, + name: source.name, + provider: source.provider, + url: source.url ?? "", + username: source.username, + token: decryptSourceToken(source.token), + enabled: source.enabled, + repositoryCount, + locked: repositoryCount > 0, + }; +} + +/** + * All of a user's sources with the number of repositories imported from + * each. A source with repositories is locked (see config-locks). + */ +export async function loadSourceApiItems(userId: string): Promise { + const [{ listSources }, { db, repositories }] = await Promise.all([ + import("@/lib/sources"), + import("@/lib/db"), + ]); + + const sources = await listSources(userId); + + const countRows = await db + .select({ sourceId: repositories.sourceId, count: sql`count(*)` }) + .from(repositories) + .where(eq(repositories.userId, userId)) + .groupBy(repositories.sourceId); + + const counts = new Map(); + for (const row of countRows) { + if (row.sourceId) { + counts.set(row.sourceId, Number(row.count)); + } + } + + return Promise.all( + sources.map((source) => toSourceApiItem(source, counts.get(source.id) ?? 0)) + ); +} + +/** + * Whether the service refused a create because the user already has this + * (provider, url, username) combination. Checked structurally so it works + * with the service's typed duplicate error. + */ +export function isDuplicateSourceError(error: unknown): boolean { + if (!error || typeof error !== "object") return false; + const { code, name, message } = error as { + code?: unknown; + name?: unknown; + message?: unknown; + }; + if (typeof code === "string" && /duplicate/i.test(code)) return true; + if (typeof name === "string" && /duplicate/i.test(name)) return true; + return ( + typeof message === "string" && /already (connected|exists|added)/i.test(message) + ); +} diff --git a/src/lib/sources.test.ts b/src/lib/sources.test.ts new file mode 100644 index 0000000..8273b2f --- /dev/null +++ b/src/lib/sources.test.ts @@ -0,0 +1,163 @@ +import { describe, expect, test } from "bun:test"; +import { + decryptSourceToken, + deriveSourceName, + findSourceForRepository, + primarySource, + resolveGitHubApiBaseUrl, + selectRepositoriesToRelink, + type SourceRecord, +} from "./sources"; +import { encrypt } from "./utils/encryption"; + +function source(overrides: Partial = {}): SourceRecord { + return { + id: "src-1", + userId: "user-1", + name: "GitHub (octocat)", + provider: "github", + url: "https://github.com", + username: "octocat", + token: null, + enabled: true, + createdAt: new Date("2026-01-01T00:00:00Z"), + updatedAt: new Date("2026-01-01T00:00:00Z"), + ...overrides, + }; +} + +describe("deriveSourceName", () => { + test("appends the username when present", () => { + expect(deriveSourceName("github", "octocat")).toBe("GitHub (octocat)"); + expect(deriveSourceName("gitea", "me")).toBe("Gitea / Forgejo (me)"); + }); + + test("falls back to the provider label without a username", () => { + expect(deriveSourceName("gitlab", "")).toBe("GitLab"); + }); +}); + +describe("primarySource", () => { + test("returns the first entry", () => { + const list = [source({ id: "a" }), source({ id: "b" })]; + expect(primarySource(list)?.id).toBe("a"); + }); + + test("returns null when nothing is connected", () => { + expect(primarySource([])).toBeNull(); + }); +}); + +describe("findSourceForRepository", () => { + const github = source(); + const gitlab = source({ + id: "src-2", + provider: "gitlab", + url: "https://gitlab.com", + username: "me", + }); + + test("matches by the stored source id first", () => { + const repo = { sourceId: "src-2", sourceProvider: "github", sourceUrl: "https://github.com" }; + expect(findSourceForRepository(repo, [github, gitlab])?.id).toBe("src-2"); + }); + + test("falls back to provider and normalized url when the id dangles", () => { + const repo = { + sourceId: "deleted-source", + sourceProvider: "gitlab", + sourceUrl: "https://gitlab.com", + }; + expect(findSourceForRepository(repo, [github, gitlab])?.id).toBe("src-2"); + }); + + test("matches by provider and url for legacy rows without a source id", () => { + const repo = { sourceProvider: "gitea", sourceUrl: "https://codeberg.org" }; + const codeberg = source({ + id: "src-3", + provider: "gitea", + url: "https://codeberg.org", + username: "me", + }); + expect(findSourceForRepository(repo, [github, codeberg])?.id).toBe("src-3"); + }); + + test("returns null for a host that is not connected", () => { + const repo = { sourceProvider: "gitlab", sourceUrl: "https://gitlab.example.com" }; + expect(findSourceForRepository(repo, [github, gitlab])).toBeNull(); + }); + + test("returns null when nothing is connected", () => { + const repo = { sourceProvider: "github", sourceUrl: "https://github.com" }; + expect(findSourceForRepository(repo, [])).toBeNull(); + }); +}); + +describe("selectRepositoriesToRelink", () => { + const connected = new Set(["src-1"]); + const base = { + sourceProvider: "github" as const, + sourceUrl: "https://github.com", + normalizedFullName: "octocat/tool", + updatedAt: new Date("2026-01-01T00:00:00Z"), + }; + + test("collects dangling and null-id rows for the same provider and host", () => { + const rows = [ + { id: "r-1", sourceId: "deleted-source", ...base }, + { id: "r-2", sourceId: null, ...base, normalizedFullName: "octocat/second" }, + { id: "r-3", ...base, normalizedFullName: "octocat/third" }, + ]; + const result = selectRepositoriesToRelink(rows, connected, "github", "https://github.com"); + expect(result.relinkIds).toEqual(["r-1", "r-2", "r-3"]); + expect(result.duplicateIds).toEqual([]); + }); + + test("keeps rows owned by a live source or another host", () => { + const rows = [ + { id: "r-1", sourceId: "src-1", ...base }, + { id: "r-2", sourceId: "deleted-source", ...base, sourceProvider: "gitlab" as const, sourceUrl: "https://gitlab.com" }, + { id: "r-3", sourceId: "deleted-source", ...base, sourceUrl: "https://ghe.example.com" }, + ]; + const result = selectRepositoriesToRelink(rows, connected, "github", "https://github.com"); + expect(result.relinkIds).toEqual([]); + expect(result.duplicateIds).toEqual([]); + }); + + test("same-name stale rows collapse to the newest one so the relink cannot violate the unique index", () => { + const rows = [ + { id: "r-old", sourceId: "dead-incarnation-1", ...base, updatedAt: new Date("2026-01-01T00:00:00Z") }, + { id: "r-new", sourceId: "dead-incarnation-2", ...base, updatedAt: new Date("2026-02-01T00:00:00Z") }, + { id: "r-other", sourceId: null, ...base, normalizedFullName: "octocat/other" }, + ]; + const result = selectRepositoriesToRelink(rows, connected, "github", "https://github.com"); + expect(result.relinkIds).toEqual(["r-new", "r-other"]); + expect(result.duplicateIds).toEqual(["r-old"]); + }); +}); + +describe("decryptSourceToken", () => { + test("round-trips an encrypted token", () => { + expect(decryptSourceToken(encrypt("ghp-secret"))).toBe("ghp-secret"); + }); + + test("empty means public-only and never throws", () => { + expect(decryptSourceToken(null)).toBe(""); + expect(decryptSourceToken(undefined)).toBe(""); + expect(decryptSourceToken("")).toBe(""); + }); +}); + +describe("resolveGitHubApiBaseUrl", () => { + test("github.com and empty keep the environment default", () => { + expect(resolveGitHubApiBaseUrl("https://github.com")).toBeUndefined(); + expect(resolveGitHubApiBaseUrl("")).toBeUndefined(); + expect(resolveGitHubApiBaseUrl(null)).toBeUndefined(); + expect(resolveGitHubApiBaseUrl(undefined)).toBeUndefined(); + }); + + test("a GitHub Enterprise instance is addressed at /api/v3", () => { + expect(resolveGitHubApiBaseUrl("https://ghe.example.com")).toBe("https://ghe.example.com/api/v3"); + expect(resolveGitHubApiBaseUrl("https://ghe.example.com/")).toBe("https://ghe.example.com/api/v3"); + }); +}); diff --git a/src/lib/sources.ts b/src/lib/sources.ts new file mode 100644 index 0000000..6adfbb0 --- /dev/null +++ b/src/lib/sources.ts @@ -0,0 +1,441 @@ +/** + * Source rows: the connected hosts a user pulls repositories from. + * + * The sources table is the authoritative home for source connections + * (provider, instance URL, username, token). The legacy + * configs.githubConfig connection fields stay synchronized with the FIRST + * source row (the "primary" source, oldest first) so un-migrated readers and + * the existing truthiness gates keep working: zero sources means empty + * connection fields, which reads as "not configured". + * + * Tokens are stored AES-256-GCM encrypted, the same convention as + * configs.githubConfig.token, and copied between the two storages without + * re-encrypting. The database module is imported lazily (the same pattern as + * loadConfigLocks) so per-test-file module mocks keep working. + */ +import { and, eq, inArray, ne, sql } from "drizzle-orm"; +import { v4 as uuidv4 } from "uuid"; +import { sources } from "@/lib/db/schema"; +import { decrypt, encrypt } from "@/lib/utils/encryption"; +import { + SOURCE_PROVIDER_KINDS, + SOURCE_PROVIDER_LABELS, + getRepositorySource, + isSourceProviderKind, + normalizeSourceProviderKind, + normalizeSourceUrl, + type RepositorySourceFields, + type SourceProviderKind, +} from "@/lib/source-providers/kinds"; + +/** A row of the sources table, as stored (token still encrypted). */ +export type SourceRecord = typeof sources.$inferSelect; + +/** The result of findSourceForRepository: the matched row, or null when none matches. */ +export type SourceConnectionForRepo = SourceRecord | null; + +/** Input for createSource. The provider is validated strictly, not coerced. */ +export interface SourceInput { + provider: unknown; + url?: string | null; + username?: string | null; + token?: string | null; + name?: string | null; + enabled?: boolean; +} + +/** Input for updateSource: every field optional, absent means "keep". */ +export type SourceUpdateInput = Partial; + +/** Create refused because this (provider, url, username) is already connected. */ +export class DuplicateSourceError extends Error { + readonly code = "duplicate_source"; + + constructor(label: string) { + super(`${label} is already connected. Remove it first to re-add or change it.`); + this.name = "DuplicateSourceError"; + } +} + +/** The display label for a source: "GitHub (octocat)", or just the provider label without a username. */ +export function deriveSourceName(provider: SourceProviderKind, username: string): string { + const label = SOURCE_PROVIDER_LABELS[provider]; + return username ? `${label} (${username})` : label; +} + +function requireProviderKind(value: unknown, label: string): SourceProviderKind { + if (!isSourceProviderKind(value)) { + throw new Error( + `Unknown source provider ${JSON.stringify(value)} for ${label}. Supported providers: ${SOURCE_PROVIDER_KINDS.join(", ")}.` + ); + } + return value; +} + +/** All of a user's sources, oldest first (index 0 is the primary source). */ +export async function listSources(userId: string): Promise { + const { db } = await import("@/lib/db"); + const rows = await db + .select() + .from(sources) + .where(eq(sources.userId, userId)); + // Sorted in JS rather than SQL so shallow test doubles of the database + // (which resolve the query at .where()) keep working. + const list = Array.isArray(rows) ? rows : []; + return list.sort( + (a, b) => (+a.createdAt || 0) - (+b.createdAt || 0) || (a.id < b.id ? -1 : a.id > b.id ? 1 : 0) + ); +} + +/** The first entry of listSources(userId), or null when nothing is connected. */ +export function primarySource(list: SourceRecord[]): SourceRecord | null { + return list.length > 0 ? list[0] : null; +} + +function duplicateLabel(provider: SourceProviderKind, url: string, username: string): string { + return `${SOURCE_PROVIDER_LABELS[provider]} (${url})${username ? ` for ${username}` : ""}`; +} + +async function assertNoDuplicate( + userId: string, + provider: SourceProviderKind, + url: string, + username: string, + excludeId?: string +): Promise { + const { db } = await import("@/lib/db"); + const conditions = [ + eq(sources.userId, userId), + eq(sources.provider, provider), + eq(sources.url, url), + eq(sources.username, username), + ]; + if (excludeId) conditions.push(ne(sources.id, excludeId)); + + const [existing] = await db + .select({ id: sources.id }) + .from(sources) + .where(and(...conditions)) + .limit(1); + if (existing) { + throw new DuplicateSourceError(duplicateLabel(provider, url, username)); + } +} + +async function selectSource(userId: string, sourceId: string): Promise { + const { db } = await import("@/lib/db"); + const [row] = await db + .select() + .from(sources) + .where(and(eq(sources.id, sourceId), eq(sources.userId, userId))) + .limit(1); + if (!row) { + throw new Error(`Source ${sourceId} not found for user ${userId}`); + } + return row; +} + +export type DanglingRepositoryRow = { + id: string; + sourceId?: string | null; + sourceProvider?: string | null; + sourceUrl?: string | null; + normalizedFullName: string | null; + updatedAt: Date | string | null; +}; + +// Repositories whose sourceId no longer resolves (or was never set) but whose +// provider and host match the given source: exactly the rows a deleted and +// re-added source must take back. Without the re-link, the next import sees +// them as new (NULL and dangling ids never equal the new source's id, and +// SQLite treats NULLs as distinct in the unique index) and duplicates them. +// Rows are deduplicated by name: relinking two same-name rows onto one source +// would violate uniq_repositories_user_source_normalized_full_name, so only +// the most recently updated row per name is taken back and the losers are +// returned separately for manual cleanup. +export function selectRepositoriesToRelink( + rows: DanglingRepositoryRow[], + connectedSourceIds: Set, + provider: SourceProviderKind, + url: string +): { relinkIds: string[]; duplicateIds: string[] } { + const matches = rows.filter((row) => { + if (row.sourceId && connectedSourceIds.has(row.sourceId)) return false; + const normalized = getRepositorySource({ sourceProvider: row.sourceProvider, sourceUrl: row.sourceUrl }); + return normalized.provider === provider && normalized.url === url; + }); + + const winnerByName = new Map(); + for (const row of matches) { + const name = (row.normalizedFullName ?? "").trim(); + const key = name || `id:${row.id}`; + const current = winnerByName.get(key); + if (!current || +new Date(row.updatedAt ?? 0) > +new Date(current.updatedAt ?? 0)) { + winnerByName.set(key, row); + } + } + + const winnerIds = new Set([...winnerByName.values()].map((row) => row.id)); + return { + relinkIds: [...winnerIds], + duplicateIds: matches.filter((row) => !winnerIds.has(row.id)).map((row) => row.id), + }; +} + +async function relinkRepositoriesToSource( + userId: string, + sourceId: string, + provider: SourceProviderKind, + url: string +): Promise { + const { db, repositories } = await import("@/lib/db"); + const connected = await db.select({ id: sources.id }).from(sources).where(eq(sources.userId, userId)); + const connectedIds = new Set(connected.map((row) => row.id)); + const rows = await db + .select({ + id: repositories.id, + sourceId: repositories.sourceId, + sourceProvider: repositories.sourceProvider, + sourceUrl: repositories.sourceUrl, + normalizedFullName: repositories.normalizedFullName, + updatedAt: repositories.updatedAt, + }) + .from(repositories) + .where(eq(repositories.userId, userId)); + const { relinkIds, duplicateIds } = selectRepositoriesToRelink(rows, connectedIds, provider, url); + if (duplicateIds.length > 0) { + console.warn( + `[sources] ${duplicateIds.length} duplicate repositories already occupy a name on this source; leaving them dangling for manual cleanup` + ); + } + if (relinkIds.length === 0) return; + await db + .update(repositories) + .set({ sourceId, updatedAt: new Date() }) + .where(and(eq(repositories.userId, userId), inArray(repositories.id, [...relinkIds]))); +} + +export async function createSource(userId: string, input: SourceInput): Promise { + const provider = requireProviderKind(input.provider, "the new source"); + const url = normalizeSourceUrl(input.url, provider); + const username = (input.username ?? "").trim(); + const rawToken = (input.token ?? "").trim(); + const name = input.name?.trim() || deriveSourceName(provider, username); + + await assertNoDuplicate(userId, provider, url, username); + + const { db } = await import("@/lib/db"); + const id = uuidv4(); + await db.insert(sources).values({ + id, + userId, + name, + provider, + url, + username, + token: rawToken ? encrypt(rawToken) : null, + enabled: input.enabled ?? true, + }); + + await relinkRepositoriesToSource(userId, id, provider, url); + await syncPrimarySourceToConfig(userId); + return selectSource(userId, id); +} + +export async function updateSource( + userId: string, + sourceId: string, + input: SourceUpdateInput +): Promise { + const existing = await selectSource(userId, sourceId); + + const provider = + input.provider === undefined + ? existing.provider + : requireProviderKind(input.provider, `source ${sourceId}`); + // A provider switch re-normalizes the stored URL against the new provider + // (the default of the old provider must not carry over as a custom URL); + // an explicit URL always wins. + const url = + input.url !== undefined + ? normalizeSourceUrl(input.url, provider) + : provider === existing.provider + ? existing.url + : normalizeSourceUrl(existing.url, provider); + const username = (input.username ?? existing.username).trim(); + const rawToken = (input.token ?? "").trim(); + const name = + input.name === undefined + ? existing.name + : input.name?.trim() || deriveSourceName(provider, username); + const enabled = input.enabled ?? existing.enabled; + + if ( + provider !== existing.provider || + url !== existing.url || + username !== existing.username + ) { + await assertNoDuplicate(userId, provider, url, username, sourceId); + } + // Host edits deliberately do not re-link repositories (createSource does): + // a host switch is lock-confirmed, and rows of the old host intentionally + // stay dangling rather than silently following the source to the new host. + + const { db } = await import("@/lib/db"); + await db + .update(sources) + .set({ + name, + provider, + url, + username, + // An empty token on update keeps the stored one; only a non-empty + // token replaces it (the config save convention). + token: rawToken ? encrypt(rawToken) : existing.token, + enabled, + updatedAt: new Date(), + }) + .where(and(eq(sources.id, sourceId), eq(sources.userId, userId))); + + await syncPrimarySourceToConfig(userId); + return selectSource(userId, sourceId); +} + +export async function deleteSource(userId: string, sourceId: string): Promise { + await selectSource(userId, sourceId); + + const { db } = await import("@/lib/db"); + await db.delete(sources).where(and(eq(sources.id, sourceId), eq(sources.userId, userId))); + + // Repositories keep their sourceId on purpose (dangling by design). + await syncPrimarySourceToConfig(userId); +} + +/** + * Write-through for the legacy config: mirror the FIRST source's connection + * fields into configs.githubConfig, or clear them when nothing is connected. + * The token is copied as-is (it is stored encrypted in both places). + */ +export async function syncPrimarySourceToConfig(userId: string): Promise { + const { db, configs } = await import("@/lib/db"); + const list = await listSources(userId); + + // Prefer active and most-recently-updated config to avoid picking a stale + // inactive stub when multiple rows exist (see issue #271). + const [config] = await db + .select() + .from(configs) + .where(eq(configs.userId, userId)) + .orderBy(sql`${configs.isActive} desc`, sql`${configs.updatedAt} desc`) + .limit(1); + if (!config) return; + + const primary = primarySource(list); + const githubConfig = { ...config.githubConfig }; + if (primary) { + githubConfig.provider = primary.provider; + githubConfig.url = primary.url; + githubConfig.owner = primary.username; + githubConfig.token = primary.token ?? ""; + } else { + githubConfig.provider = "github"; + githubConfig.url = undefined; + githubConfig.owner = ""; + githubConfig.token = ""; + } + + await db + .update(configs) + .set({ githubConfig, updatedAt: new Date() }) + .where(eq(configs.id, config.id)); +} + +/** + * Idempotent bootstrap for configs written outside the sources API: when a + * user has no source rows but their config carries connection fields + * (POST /api/config from scripts, seed data), seed the first source row from + * them so discovery and the scheduler find something to iterate. Mirrors the + * 0019 migration: the token is copied encrypted as-is and the provider is + * normalized (config blobs may still hold "forgejo"/"codeberg"). + */ +export async function ensureSourcesFromConfig(userId: string): Promise { + const existing = await listSources(userId); + if (existing.length > 0) return; + + const { db, configs } = await import("@/lib/db"); + const [config] = await db + .select() + .from(configs) + .where(eq(configs.userId, userId)) + .orderBy(sql`${configs.isActive} desc`, sql`${configs.updatedAt} desc`) + .limit(1); + if (!config) return; + + const github = config.githubConfig; + const username = (github?.owner ?? "").trim(); + const token = (github?.token ?? "").trim(); + if (!username && !token) return; + + const provider = normalizeSourceProviderKind(github?.provider); + const url = normalizeSourceUrl(github?.url, provider); + const sourceId = uuidv4(); + await db.insert(sources).values({ + id: sourceId, + userId, + name: deriveSourceName(provider, username), + provider, + url, + username, + token: token || null, + enabled: true, + }); + await relinkRepositoriesToSource(userId, sourceId, provider, url); +} + +/** + * The source a repository belongs to: the stored sourceId first, then a + * provider + host match against the user's sources (so a deleted and + * re-added source is found again). Null means the repository's host is not + * connected, and mirroring it must be refused. + */ +export function findSourceForRepository( + repo: RepositorySourceFields & { sourceId?: string | null }, + list: SourceRecord[] +): SourceConnectionForRepo { + if (repo.sourceId) { + const byId = list.find((source) => source.id === repo.sourceId); + if (byId) return byId; + } + + const source = getRepositorySource(repo); + // Compare normalized URLs: rows seeded by the 0019 migration may hold + // non-canonical URLs (rtrim-only) that still resolve once normalized. + return ( + list.find( + (candidate) => + candidate.provider === source.provider && + normalizeSourceUrl(candidate.url, candidate.provider) === source.url + ) ?? null + ); +} + +/** + * Decrypt a source token for API use. Unlike getDecryptedGitHubToken this + * never throws: empty or missing means public-only access, which every + * source kind supports. + */ +export function decryptSourceToken(token: string | null | undefined): string { + return token ? decrypt(token) : ""; +} + +/** + * The REST API base for a GitHub-kind source. github.com (or empty) returns + * undefined so the GH_API_URL environment default applies (GHES/GHEC setups + * that predate multi-source keep working); any other URL is a GitHub + * Enterprise instance addressed at /api/v3. + */ +export function resolveGitHubApiBaseUrl(sourceUrl: string | null | undefined): string | undefined { + const trimmed = sourceUrl?.trim().replace(/\/+$/, "") ?? ""; + if (!trimmed || trimmed === "https://github.com") return undefined; + return `${trimmed}/api/v3`; +} diff --git a/src/pages/api/config/index.ts b/src/pages/api/config/index.ts index a5be8c1..d5fd373 100644 --- a/src/pages/api/config/index.ts +++ b/src/pages/api/config/index.ts @@ -28,6 +28,10 @@ import { encrypt, decrypt } from "@/lib/utils/encryption"; import { createDefaultConfig } from "@/lib/utils/config-defaults"; import { requireAuthenticatedUserId } from "@/lib/auth-guards"; import { notificationConfigSchema } from "@/lib/db/schema"; +import { + loadSourceApiItems, + type SourceApiItem, +} from "@/lib/sources-api"; export const POST: APIRoute = async ({ request, locals }) => { try { @@ -365,6 +369,24 @@ export const GET: APIRoute = async ({ request, locals }) => { const locks = await loadConfigLocks(userId); + // Sources for the Sources card: per-source repository counts, lock state + // and decrypted tokens, following the same convention as the config + // tokens below. A failure must not take the Configuration page down. + let sources: SourceApiItem[] = []; + try { + // Heal configs written outside the sources API (scripts, seeds): + // seed the first source row before listing. + const { ensureSourcesFromConfig } = await import("@/lib/sources"); + await ensureSourcesFromConfig(userId); + sources = await loadSourceApiItems(userId); + } catch (sourcesError) { + console.warn( + `[Config] Could not load sources for user ${userId}: ${ + sourcesError instanceof Error ? sourcesError.message : String(sourcesError) + }` + ); + } + if (config.length === 0) { // Create default configuration for the user const defaultConfig = await createDefaultConfig({ userId }); @@ -381,6 +403,7 @@ export const GET: APIRoute = async ({ request, locals }) => { scheduleConfig: uiScheduleConfig, cleanupConfig: uiCleanupConfig, locks, + sources, }), { status: 200, @@ -474,6 +497,7 @@ export const GET: APIRoute = async ({ request, locals }) => { }, notificationConfig, locks, + sources, }), { status: 200, headers: { "Content-Type": "application/json" }, @@ -500,6 +524,7 @@ export const GET: APIRoute = async ({ request, locals }) => { }, notificationConfig: dbConfig.notificationConfig, locks, + sources, }), { status: 200, headers: { "Content-Type": "application/json" }, diff --git a/src/pages/api/job/mirror-org.ts b/src/pages/api/job/mirror-org.ts index 6c50a6d..fcdad1a 100644 --- a/src/pages/api/job/mirror-org.ts +++ b/src/pages/api/job/mirror-org.ts @@ -1,6 +1,13 @@ import type { APIRoute } from "astro"; import type { Octokit } from "@octokit/rest"; -import { resolveSourceProviderKind } from "@/lib/source-providers"; +import type { Repository } from "@/lib/db/schema"; +import type { SourceRecord } from "@/lib/sources"; +import { + decryptSourceToken, + findSourceForRepository, + listSources, + resolveGitHubApiBaseUrl, +} from "@/lib/sources"; import type { MirrorOrgRequest, MirrorOrgResponse } from "@/types/mirror"; import { db, configs, organizations } from "@/lib/db"; import { and, eq, inArray, sql } from "drizzle-orm"; @@ -11,7 +18,6 @@ import { type MembershipRole } from "@/types/organizations"; import { createSecureErrorResponse } from "@/lib/utils"; import { processWithResilience } from "@/lib/utils/concurrency"; import { v4 as uuidv4 } from "uuid"; -import { getDecryptedGitHubToken } from "@/lib/utils/config-encryption"; import { requireAuthenticatedUserId } from "@/lib/auth-guards"; export const POST: APIRoute = async ({ request, locals }) => { @@ -86,13 +92,23 @@ export const POST: APIRoute = async ({ request, locals }) => { } // Only GitHub sources need an API client while mirroring (metadata). - // Other hosts get code-only mirrors through Gitea. - let octokit: Octokit | null = null; - if (resolveSourceProviderKind(config) === "github") { - const decryptedToken = getDecryptedGitHubToken(config); - const githubUsername = config.githubConfig?.owner || undefined; - octokit = createGitHubClient(decryptedToken, userId, githubUsername); - } + // Other hosts get code-only mirrors through Gitea. An organization is + // shared across sources, so its repositories can come from different + // hosts: the resolver hands each one a client built from its own + // source instead of one client for the whole batch. + const sources = await listSources(userId); + const resolveRepositoryOctokit = (repository: Repository): Octokit | null => { + const source = findSourceForRepository(repository, sources); + if (!source || source.provider !== "github") return null; + const token = decryptSourceToken(source.token); + if (!token) return null; + return createGitHubClient( + token, + userId, + source.username || undefined, + resolveGitHubApiBaseUrl(source.url) + ); + }; // Define the concurrency limit - adjust based on API rate limits // Using a lower concurrency for organizations since each org might contain many repos @@ -120,7 +136,8 @@ export const POST: APIRoute = async ({ request, locals }) => { // Mirror the organization await mirrorGitHubOrgToGitea({ config, - octokit, + octokit: null, + resolveRepositoryOctokit, organization: orgData, }); diff --git a/src/pages/api/job/mirror-repo.test.ts b/src/pages/api/job/mirror-repo.test.ts index 18dc384..1ff38ce 100644 --- a/src/pages/api/job/mirror-repo.test.ts +++ b/src/pages/api/job/mirror-repo.test.ts @@ -97,6 +97,43 @@ mock.module("@/lib/github", () => ({ createGitHubClient: mockCreateGitHubClient })); +// The route builds the GitHub client per repository from its own source row. +// One GitHub source mirrors the pre-multi-source fixtures (same token). +mock.module("@/lib/sources", () => ({ + listSources: mock(async (userId: string) => + userId === "user-id" + ? [ + { + id: "source-1", + userId, + name: "GitHub", + provider: "github", + url: "https://github.com", + username: "", + token: "github-token", + enabled: true, + createdAt: new Date(0), + updatedAt: new Date(0), + }, + ] + : [] + ), + findSourceForRepository: (repo: any, list: any[]) => { + if (repo.sourceId) { + const byId = list.find((source) => source.id === repo.sourceId); + if (byId) return byId; + } + const normalized = "https://github.com"; + return list.find((candidate: any) => candidate.provider === "github" && candidate.url === normalized) ?? null; + }, + decryptSourceToken: (token: string | null | undefined) => token ?? "", + resolveGitHubApiBaseUrl: (url: string | null | undefined) => { + const trimmed = url?.trim().replace(/\/+$/, "") ?? ""; + if (!trimmed || trimmed === "https://github.com") return undefined; + return `${trimmed}/api/v3`; + }, +})); + // Mock the concurrency module const mockProcessWithResilience = mock(() => Promise.resolve([])); diff --git a/src/pages/api/job/mirror-repo.ts b/src/pages/api/job/mirror-repo.ts index 0731fdc..b97e3fd 100644 --- a/src/pages/api/job/mirror-repo.ts +++ b/src/pages/api/job/mirror-repo.ts @@ -1,6 +1,12 @@ import type { APIRoute } from "astro"; import type { Octokit } from "@octokit/rest"; -import { resolveSourceProviderKind } from "@/lib/source-providers"; +import type { SourceRecord } from "@/lib/sources"; +import { + decryptSourceToken, + findSourceForRepository, + listSources, + resolveGitHubApiBaseUrl, +} from "@/lib/sources"; import type { MirrorRepoRequest, MirrorRepoResponse } from "@/types/mirror"; import { db, configs, repositories } from "@/lib/db"; import { and, eq, inArray, sql } from "drizzle-orm"; @@ -8,7 +14,6 @@ import { repositoryVisibilityEnum, repoStatusEnum } from "@/types/Repository"; import { getGiteaRepoOwnerAsync } from "@/lib/gitea"; import { mirrorRepositoryToDestination } from "@/lib/mirror-dispatch"; import { createGitHubClient } from "@/lib/github"; -import { getDecryptedGitHubToken } from "@/lib/utils/config-encryption"; import { processWithResilience } from "@/lib/utils/concurrency"; import { createSecureErrorResponse } from "@/lib/utils"; import { requireAuthenticatedUserId } from "@/lib/auth-guards"; @@ -85,13 +90,21 @@ export const POST: APIRoute = async ({ request, locals }) => { } // Only GitHub sources need an API client while mirroring (metadata). - // Other hosts get code-only mirrors through Gitea. - let octokit: Octokit | null = null; - if (resolveSourceProviderKind(config) === "github") { - const decryptedToken = getDecryptedGitHubToken(config); - const githubUsername = config.githubConfig?.owner || undefined; - octokit = createGitHubClient(decryptedToken, userId, githubUsername); - } + // Other hosts get code-only mirrors through Gitea. The selected + // repositories can come from different sources, so the client is + // built per repository from its own source, never one for the batch. + const sources = await listSources(userId); + const octokitForSource = (source: SourceRecord | null): Octokit | null => { + if (!source || source.provider !== "github") return null; + const token = decryptSourceToken(source.token); + if (!token) return null; + return createGitHubClient( + token, + userId, + source.username || undefined, + resolveGitHubApiBaseUrl(source.url) + ); + }; // Define the concurrency limit - adjust based on API rate limits const CONCURRENCY_LIMIT = 3; @@ -115,10 +128,13 @@ export const POST: APIRoute = async ({ request, locals }) => { // Log the start of mirroring console.log(`Starting mirror for repository: ${repo.name}`); + const repoSource = findSourceForRepository(repo, sources); + // Determine where the repository should be mirrored (with organization overrides) const owner = await getGiteaRepoOwnerAsync({ config, repository: repoData, + sourceUsername: repoSource?.username, }); console.log(`Repository ${repo.name} will be mirrored to owner: ${owner}`); @@ -136,7 +152,7 @@ export const POST: APIRoute = async ({ request, locals }) => { // pushed by the engine. The dispatcher picks the path. await mirrorRepositoryToDestination({ config, - octokit, + octokit: octokitForSource(repoSource), repository: repoData, orgName: shouldUseOrgMirror ? owner : undefined, }); diff --git a/src/pages/api/job/retry-repo.ts b/src/pages/api/job/retry-repo.ts index 0c4e9f6..c41dd4a 100644 --- a/src/pages/api/job/retry-repo.ts +++ b/src/pages/api/job/retry-repo.ts @@ -1,5 +1,11 @@ import type { APIRoute } from "astro"; -import { resolveSourceProviderKind } from "@/lib/source-providers"; +import type { SourceRecord } from "@/lib/sources"; +import { + decryptSourceToken, + findSourceForRepository, + listSources, + resolveGitHubApiBaseUrl, +} from "@/lib/sources"; import { db, configs, repositories } from "@/lib/db"; import { and, eq, inArray, sql } from "drizzle-orm"; import { getGiteaRepoOwnerAsync, isRepoPresentInGitea } from "@/lib/gitea"; @@ -11,7 +17,6 @@ import type { RetryRepoRequest, RetryRepoResponse } from "@/types/retry"; import { processWithRetry } from "@/lib/utils/concurrency"; import { createMirrorJob } from "@/lib/helpers"; import { createSecureErrorResponse } from "@/lib/utils"; -import { getDecryptedGitHubToken } from "@/lib/utils/config-encryption"; import { requireAuthenticatedUserId } from "@/lib/auth-guards"; export const POST: APIRoute = async ({ request, locals }) => { @@ -81,16 +86,22 @@ export const POST: APIRoute = async ({ request, locals }) => { // Start background retry with parallel processing setTimeout(async () => { - // Create a single Octokit instance to be reused if needed with rate limit tracking - const decryptedToken = config.githubConfig.token - ? getDecryptedGitHubToken(config) - : null; - const githubUsername = config.githubConfig?.owner || undefined; // Only GitHub sources need an API client while mirroring (metadata). - const octokit = - decryptedToken && resolveSourceProviderKind(config) === "github" - ? createGitHubClient(decryptedToken, userId, githubUsername) - : null; + // Other hosts get code-only mirrors through Gitea. The retried + // repositories can come from different sources, so the client is + // built per repository from its own source. + const sources = await listSources(userId); + const octokitForSource = (source: SourceRecord | null) => { + if (!source || source.provider !== "github") return null; + const token = decryptSourceToken(source.token); + if (!token) return null; + return createGitHubClient( + token, + userId, + source.username || undefined, + resolveGitHubApiBaseUrl(source.url) + ); + }; // Define the concurrency limit - adjust based on API rate limits const CONCURRENCY_LIMIT = 3; @@ -113,6 +124,9 @@ export const POST: APIRoute = async ({ request, locals }) => { mirroredLocation: repo.mirroredLocation || "", }; + const repoSource = findSourceForRepository(repo, sources); + const octokit = octokitForSource(repoSource); + // Log the start of retry operation console.log(`Starting retry for repository: ${repo.name}`); @@ -138,6 +152,7 @@ export const POST: APIRoute = async ({ request, locals }) => { let owner = await getGiteaRepoOwnerAsync({ config, repository: repoData, + sourceUsername: repoSource?.username, }); const present = await isRepoPresentInGitea({ diff --git a/src/pages/api/sources/[id].test.ts b/src/pages/api/sources/[id].test.ts new file mode 100644 index 0000000..66f2665 --- /dev/null +++ b/src/pages/api/sources/[id].test.ts @@ -0,0 +1,239 @@ +/** + * Route tests for /api/sources/:id: the per-source lock that guards host + * changes on PUT, the confirmation that overrides it, and the removal rules + * of DELETE (repositories keep their mirrors and their dangling sourceId). + */ + +import { describe, test, expect, mock, beforeEach } from "bun:test"; + +let sourceRows: any[] = []; +let repositoryCount = 0; + +function encrypted(token: string): string { + return `enc:${token}`; +} + +// The service layer syncs the legacy config itself on every write; the mock +// follows its contract (in-memory rows, tokens "encrypted" with a prefix, an +// empty token preserves the stored one) so these tests stay about the route. +mock.module("@/lib/sources", () => ({ + listSources: mock(async (userId: string) => + sourceRows.filter((source) => source.userId === userId) + ), + createSource: mock(async (userId: string, input: any) => { + const row = { + id: `source-${sourceRows.length + 1}`, + userId, + name: input.name || `${input.provider} source`, + provider: input.provider, + url: input.url ?? "", + username: input.username ?? "", + token: input.token ? encrypted(input.token) : null, + enabled: input.enabled ?? true, + createdAt: Date.now(), + updatedAt: Date.now(), + }; + sourceRows.push(row); + return { ...row }; + }), + updateSource: mock(async (userId: string, sourceId: string, input: any) => { + const row = sourceRows.find( + (source) => source.id === sourceId && source.userId === userId + ); + if (!row) throw new Error("Source not found"); + if (input.provider !== undefined) row.provider = input.provider; + if (input.url !== undefined) row.url = input.url; + if (input.name !== undefined) row.name = input.name; + if (input.username !== undefined) row.username = input.username; + if (input.enabled !== undefined) row.enabled = input.enabled; + if (input.token) row.token = encrypted(input.token); + return { ...row }; + }), + deleteSource: mock(async (userId: string, sourceId: string) => { + sourceRows = sourceRows.filter( + (source) => source.id !== sourceId || source.userId !== userId + ); + }), + syncPrimarySourceToConfig: mock(async () => {}), + decryptSourceToken: mock((token: string | null | undefined) => + typeof token === "string" && token.startsWith("enc:") ? token.slice(4) : token ?? "" + ), +})); + +const mockRepositories = {}; +mock.module("@/lib/db", () => ({ + db: { + select: () => ({ + from: () => ({ + where: async () => [{ count: repositoryCount }], + groupBy: async () => [], + }), + }), + insert: () => ({ values: async () => ({}) }), + update: () => ({ set: () => ({ where: async () => {} }) }), + delete: () => ({ where: async () => {} }), + }, + repositories: mockRepositories, + configs: {}, + users: {}, + organizations: {}, + events: {}, + mirrorJobs: {}, + sessions: {}, + accounts: {}, + verificationTokens: {}, + verifications: {}, + oauthClients: {}, + oauthAccessTokens: {}, + oauthRefreshTokens: {}, + oauthConsents: {}, + jwkss: {}, + ssoProviders: {}, + apikeys: {}, + rateLimits: {}, +})); + +const { PUT, DELETE } = await import("./[id]"); + +const githubSource = { + id: "source-1", + userId: "user-1", + name: "GitHub (octocat)", + provider: "github", + url: "", + username: "octocat", + token: encrypted("ghp_old"), + enabled: true, +}; + +function put(id: string, body: Record) { + return PUT({ + params: { id }, + request: new Request(`http://localhost/api/sources/${id}`, { + method: "PUT", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(body), + }), + locals: { session: { userId: "user-1" } }, + } as any); +} + +function remove(id: string, options?: { query?: string; body?: unknown }) { + return DELETE({ + params: { id }, + request: new Request( + `http://localhost/api/sources/${id}${options?.query ?? ""}`, + { + method: "DELETE", + headers: { "Content-Type": "application/json" }, + body: options?.body === undefined ? undefined : JSON.stringify(options.body), + } + ), + locals: { session: { userId: "user-1" } }, + } as any); +} + +beforeEach(() => { + sourceRows = [{ ...githubSource, token: encrypted("ghp_old") }]; + repositoryCount = 2; +}); + +describe("PUT /api/sources/:id", () => { + test("answers 404 for a source that is not the account's", async () => { + const response = await put("source-404", { name: "Renamed" }); + expect(response.status).toBe(404); + const data = await response.json(); + expect(data.success).toBe(false); + expect(data.message).toContain("Source not found"); + }); + + test("refuses a host change on a locked source without confirmation", async () => { + const response = await put("source-1", { + provider: "gitlab", + url: "https://gitlab.example.com", + }); + expect(response.status).toBe(409); + const data = await response.json(); + expect(data.success).toBe(false); + expect(data.lock).toBe("source"); + expect(data.message).toContain("2 repositories were imported"); + expect(data.message).toContain("https://github.com"); + expect(data.message).toContain("https://gitlab.example.com"); + expect(data.message).toContain("Confirm the change"); + expect(sourceRows[0].provider).toBe("github"); + }); + + test("renaming a locked source needs no confirmation", async () => { + const response = await put("source-1", { name: "My GitHub" }); + expect(response.status).toBe(200); + const data = await response.json(); + expect(data.success).toBe(true); + expect(data.source.name).toBe("My GitHub"); + }); + + test("switches the host with confirmSourceChange and preserves an empty token", async () => { + const response = await put("source-1", { + provider: "gitlab", + url: "https://gitlab.example.com", + token: "", + confirmSourceChange: true, + }); + expect(response.status).toBe(200); + const data = await response.json(); + expect(data.success).toBe(true); + expect(data.source).toMatchObject({ + id: "source-1", + provider: "gitlab", + url: "https://gitlab.example.com", + token: "ghp_old", + repositoryCount: 2, + locked: true, + }); + }); +}); + +describe("DELETE /api/sources/:id", () => { + test("refuses the removal of a source with repositories without confirmation", async () => { + const response = await remove("source-1"); + expect(response.status).toBe(409); + const data = await response.json(); + expect(data.success).toBe(false); + expect(data.lock).toBe("source"); + expect(data.message).toContain("2 repositories were imported from GitHub (https://github.com)"); + expect(data.message).toContain("Existing mirrors keep syncing"); + expect(sourceRows).toHaveLength(1); + }); + + test("removes the source with repositories once confirmed via query flag", async () => { + const response = await remove("source-1", { query: "?confirmRemoveSource=true" }); + expect(response.status).toBe(200); + const data = await response.json(); + expect(data.success).toBe(true); + expect(sourceRows).toHaveLength(0); + }); + + test("removes the source once confirmed via body flag", async () => { + const response = await remove("source-1", { body: { confirmRemoveSource: true } }); + expect(response.status).toBe(200); + const data = await response.json(); + expect(data.success).toBe(true); + expect(sourceRows).toHaveLength(0); + }); + + test("removes a source without repositories without confirmation", async () => { + repositoryCount = 0; + const response = await remove("source-1"); + expect(response.status).toBe(200); + const data = await response.json(); + expect(data.success).toBe(true); + expect(sourceRows).toHaveLength(0); + }); + + test("answers 404 for a source that is not the account's", async () => { + const response = await remove("source-404", { query: "?confirmRemoveSource=true" }); + expect(response.status).toBe(404); + const data = await response.json(); + expect(data.success).toBe(false); + expect(data.message).toContain("Source not found"); + }); +}); diff --git a/src/pages/api/sources/[id].ts b/src/pages/api/sources/[id].ts new file mode 100644 index 0000000..5a47831 --- /dev/null +++ b/src/pages/api/sources/[id].ts @@ -0,0 +1,190 @@ +import type { APIRoute } from "astro"; +import { and, eq, sql } from "drizzle-orm"; +import { db, repositories } from "@/lib/db"; +import { + SOURCE_PROVIDER_LABELS, + isSourceProviderKind, + isValidSourceUrl, + type SourceProviderKind, +} from "@/lib/source-providers/kinds"; +import { + computeConfigLocks, + describeSourceIdentity, + evaluateConfigChange, + sourceIdentity, +} from "@/lib/config-locks"; +import { requireAuthenticatedUserId } from "@/lib/auth-guards"; +import { createSecureErrorResponse } from "@/lib/utils"; +import { + deleteSource, + listSources, + updateSource, +} from "@/lib/sources"; +import { toSourceApiItem } from "@/lib/sources-api"; + +/** Repositories imported from this source; they are what locks it. */ +async function countRepositoriesForSource( + userId: string, + sourceId: string +): Promise { + const [row] = await db + .select({ count: sql`count(*)` }) + .from(repositories) + .where(and(eq(repositories.userId, userId), eq(repositories.sourceId, sourceId))); + return Number(row?.count ?? 0); +} + +function jsonError(message: string, status: number, extra?: Record) { + return new Response(JSON.stringify({ success: false, message, ...extra }), { + status, + headers: { "Content-Type": "application/json" }, + }); +} + +export const PUT: APIRoute = async ({ params, request, locals }) => { + try { + const authResult = await requireAuthenticatedUserId({ request, locals }); + if ("response" in authResult) return authResult.response; + const userId = authResult.userId; + + const sourceId = params.id; + if (!sourceId) { + return jsonError("Source ID is required.", 400); + } + + const body = (await request.json()) as { + name?: unknown; + provider?: unknown; + url?: unknown; + username?: unknown; + token?: unknown; + enabled?: unknown; + confirmSourceChange?: unknown; + }; + + const sources = await listSources(userId); + const existing = sources.find((source) => source.id === sourceId); + if (!existing) { + return jsonError("Source not found.", 404); + } + + if (body.provider !== undefined && !isSourceProviderKind(body.provider)) { + return jsonError("Unknown source provider.", 400); + } + const provider: SourceProviderKind = isSourceProviderKind(body.provider) + ? body.provider + : existing.provider; + + // GitLab and Gitea sources point at an instance; GitHub needs no URL. + if ( + provider !== "github" && + typeof body.url === "string" && + body.url.trim() && + !isValidSourceUrl(body.url) + ) { + return jsonError( + `${SOURCE_PROVIDER_LABELS[provider]} URL must be a valid http or https URL.`, + 400 + ); + } + + const repositoryCount = await countRepositoriesForSource(userId, sourceId); + + // A source repositories were imported from only switches host with an + // explicit confirmation (the same rule as the configuration card). + const verdict = evaluateConfigChange({ + locks: computeConfigLocks({ repositoryCount, mirroredCount: 0 }), + existingSource: { provider: existing.provider, url: existing.url }, + incomingSource: { + provider, + url: typeof body.url === "string" ? body.url : existing.url, + }, + existingDestinationUrl: "", + incomingDestinationUrl: "", + confirmSourceChange: body.confirmSourceChange === true, + }); + if (!verdict.ok) { + return jsonError(verdict.message, 409, { lock: verdict.lock }); + } + + const updated = await updateSource(userId, sourceId, { + name: typeof body.name === "string" ? body.name : undefined, + provider: body.provider === undefined ? undefined : provider, + url: typeof body.url === "string" ? body.url : undefined, + username: typeof body.username === "string" ? body.username : undefined, + // An empty token keeps the stored one (the service preserves it). + token: typeof body.token === "string" ? body.token : undefined, + enabled: typeof body.enabled === "boolean" ? body.enabled : undefined, + }); + + // updateSource syncs the legacy config's connection fields itself. + const source = await toSourceApiItem(updated, repositoryCount); + + return new Response(JSON.stringify({ success: true, source }), { + status: 200, + headers: { "Content-Type": "application/json" }, + }); + } catch (error) { + return createSecureErrorResponse(error, "source update", 500); + } +}; + +export const DELETE: APIRoute = async ({ params, request, locals }) => { + try { + const authResult = await requireAuthenticatedUserId({ request, locals }); + if ("response" in authResult) return authResult.response; + const userId = authResult.userId; + + const sourceId = params.id; + if (!sourceId) { + return jsonError("Source ID is required.", 400); + } + + const sources = await listSources(userId); + const existing = sources.find((source) => source.id === sourceId); + if (!existing) { + return jsonError("Source not found.", 404); + } + + const repositoryCount = await countRepositoriesForSource(userId, sourceId); + + // The confirmation flag may arrive as a query parameter or in the body. + let confirmRemoveSource = ["true", "1"].includes( + new URL(request.url).searchParams.get("confirmRemoveSource") ?? "" + ); + const rawBody = await request.text(); + if (rawBody.trim()) { + try { + const body = JSON.parse(rawBody) as { confirmRemoveSource?: unknown }; + confirmRemoveSource = confirmRemoveSource || body.confirmRemoveSource === true; + } catch { + // Not JSON: the flag was not sent in the body. + } + } + + if (repositoryCount > 0 && !confirmRemoveSource) { + const label = describeSourceIdentity( + sourceIdentity({ provider: existing.provider, url: existing.url }) + ); + return jsonError( + `${repositoryCount} ${ + repositoryCount === 1 ? "repository was" : "repositories were" + } imported from ${label}; confirm the removal to disconnect it. ` + + `Existing mirrors keep syncing on the destination with their stored credentials.`, + 409, + { lock: "source" } + ); + } + + await deleteSource(userId, sourceId); + + // deleteSource syncs the legacy config's connection fields itself; + // repositories keep their sourceId on purpose (dangling by design). + return new Response(JSON.stringify({ success: true }), { + status: 200, + headers: { "Content-Type": "application/json" }, + }); + } catch (error) { + return createSecureErrorResponse(error, "source delete", 500); + } +}; diff --git a/src/pages/api/sources/index.test.ts b/src/pages/api/sources/index.test.ts new file mode 100644 index 0000000..ba7b8ab --- /dev/null +++ b/src/pages/api/sources/index.test.ts @@ -0,0 +1,265 @@ +/** + * Route tests for /api/sources: the list with per-source repository counts + * and lock state, and the validation, duplicate and write-through rules for + * adding a source. + */ + +import { describe, test, expect, mock, beforeEach } from "bun:test"; + +let sourceRows: any[] = []; +let countRows: Array<{ sourceId: string | null; count: number }> = []; + +function encrypted(token: string): string { + return `enc:${token}`; +} + +// The service layer syncs the legacy config itself on every write; the mock +// follows its contract (in-memory rows, tokens "encrypted" with a prefix, +// DuplicateSourceError shape) so these tests stay about the route. +mock.module("@/lib/sources", () => ({ + listSources: mock(async (userId: string) => + sourceRows.filter((source) => source.userId === userId) + ), + createSource: mock(async (userId: string, input: any) => { + const url = input.url ?? ""; + const username = input.username ?? ""; + const duplicate = sourceRows.find( + (source) => + source.userId === userId && + source.provider === input.provider && + source.url === url && + source.username === username + ); + if (duplicate) { + const error = new Error("This source is already connected."); + (error as any).code = "duplicate_source"; + throw error; + } + const row = { + id: `source-${sourceRows.length + 1}`, + userId, + name: input.name || `${input.provider} source`, + provider: input.provider, + url, + username, + token: input.token ? encrypted(input.token) : null, + enabled: input.enabled ?? true, + createdAt: Date.now(), + updatedAt: Date.now(), + }; + sourceRows.push(row); + return { ...row }; + }), + updateSource: mock(async (userId: string, sourceId: string, input: any) => { + const row = sourceRows.find( + (source) => source.id === sourceId && source.userId === userId + ); + if (!row) throw new Error("Source not found"); + if (input.provider !== undefined) row.provider = input.provider; + if (input.url !== undefined) row.url = input.url; + if (input.name !== undefined) row.name = input.name; + if (input.username !== undefined) row.username = input.username; + if (input.enabled !== undefined) row.enabled = input.enabled; + if (input.token) row.token = encrypted(input.token); + return { ...row }; + }), + deleteSource: mock(async (userId: string, sourceId: string) => { + sourceRows = sourceRows.filter( + (source) => source.id !== sourceId || source.userId !== userId + ); + }), + syncPrimarySourceToConfig: mock(async () => {}), + decryptSourceToken: mock((token: string | null | undefined) => + typeof token === "string" && token.startsWith("enc:") ? token.slice(4) : token ?? "" + ), +})); + +const mockRepositories = {}; +mock.module("@/lib/db", () => ({ + db: { + select: () => ({ + from: () => ({ + where: () => ({ + groupBy: async () => countRows, + limit: async () => [], + }), + }), + }), + insert: () => ({ values: async () => ({}) }), + update: () => ({ set: () => ({ where: async () => {} }) }), + delete: () => ({ where: async () => {} }), + }, + repositories: mockRepositories, + configs: {}, + users: {}, + organizations: {}, + events: {}, + mirrorJobs: {}, + sessions: {}, + accounts: {}, + verificationTokens: {}, + verifications: {}, + oauthClients: {}, + oauthAccessTokens: {}, + oauthRefreshTokens: {}, + oauthConsents: {}, + jwkss: {}, + ssoProviders: {}, + apikeys: {}, + rateLimits: {}, +})); + +const { GET, POST } = await import("./index"); + +function get() { + return GET({ + request: new Request("http://localhost/api/sources"), + locals: { session: { userId: "user-1" } }, + } as any); +} + +function post(body: Record) { + return POST({ + request: new Request("http://localhost/api/sources", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(body), + }), + locals: { session: { userId: "user-1" } }, + } as any); +} + +beforeEach(() => { + sourceRows = []; + countRows = []; +}); + +describe("GET /api/sources", () => { + test("lists sources with repository counts, lock state and decrypted tokens", async () => { + sourceRows = [ + { + id: "source-1", + userId: "user-1", + name: "GitHub (octocat)", + provider: "github", + url: "", + username: "octocat", + token: encrypted("ghp_secret"), + enabled: true, + }, + { + id: "source-2", + userId: "user-1", + name: "GitLab", + provider: "gitlab", + url: "https://gitlab.example.com", + username: "octo", + token: null, + enabled: false, + }, + ]; + countRows = [ + { sourceId: "source-1", count: 2 }, + { sourceId: null, count: 5 }, + ]; + + const response = await get(); + expect(response.status).toBe(200); + const data = await response.json(); + expect(data.success).toBe(true); + expect(data.sources).toHaveLength(2); + expect(data.sources[0]).toMatchObject({ + id: "source-1", + name: "GitHub (octocat)", + provider: "github", + username: "octocat", + token: "ghp_secret", + enabled: true, + repositoryCount: 2, + locked: true, + }); + expect(data.sources[1]).toMatchObject({ + id: "source-2", + provider: "gitlab", + token: "", + enabled: false, + repositoryCount: 0, + locked: false, + }); + }); +}); + +describe("POST /api/sources", () => { + test("rejects an unknown source provider", async () => { + const response = await post({ provider: "bitbucket", username: "me", token: "t" }); + expect(response.status).toBe(400); + const data = await response.json(); + expect(data.success).toBe(false); + expect(data.message).toContain("Unknown source provider"); + }); + + test("rejects an instance URL that is not http(s) for a GitLab source", async () => { + const response = await post({ + provider: "gitlab", + url: "ftp://gitlab.example.com", + username: "me", + token: "glpat-x", + }); + expect(response.status).toBe(400); + const data = await response.json(); + expect(data.success).toBe(false); + expect(data.message).toContain("GitLab URL"); + }); + + test("answers 409 when the same source is already connected", async () => { + sourceRows = [ + { + id: "source-1", + userId: "user-1", + name: "GitLab", + provider: "gitlab", + url: "https://gitlab.example.com", + username: "me", + token: null, + enabled: true, + }, + ]; + + const response = await post({ + provider: "gitlab", + url: "https://gitlab.example.com", + username: "me", + token: "glpat-x", + }); + expect(response.status).toBe(409); + const data = await response.json(); + expect(data.success).toBe(false); + expect(data.message).toContain("already connected"); + expect(sourceRows).toHaveLength(1); + }); + + test("creates a source, echoes it decrypted", async () => { + const response = await post({ + name: "Work GitLab", + provider: "gitlab", + url: "https://gitlab.example.com", + username: "me", + token: "glpat-x", + }); + expect(response.status).toBe(201); + const data = await response.json(); + expect(data.success).toBe(true); + expect(data.source).toMatchObject({ + name: "Work GitLab", + provider: "gitlab", + url: "https://gitlab.example.com", + username: "me", + token: "glpat-x", + repositoryCount: 0, + locked: false, + }); + + expect(sourceRows).toHaveLength(1); + expect(sourceRows[0].token).toBe(encrypted("glpat-x")); + }); +}); diff --git a/src/pages/api/sources/index.ts b/src/pages/api/sources/index.ts new file mode 100644 index 0000000..adc14ea --- /dev/null +++ b/src/pages/api/sources/index.ts @@ -0,0 +1,110 @@ +import type { APIRoute } from "astro"; +import { + SOURCE_PROVIDER_LABELS, + isSourceProviderKind, + isValidSourceUrl, +} from "@/lib/source-providers/kinds"; +import { requireAuthenticatedUserId } from "@/lib/auth-guards"; +import { createSecureErrorResponse } from "@/lib/utils"; +import { createSource } from "@/lib/sources"; +import { + isDuplicateSourceError, + loadSourceApiItems, + toSourceApiItem, +} from "@/lib/sources-api"; + +export const GET: APIRoute = async ({ request, locals }) => { + try { + const authResult = await requireAuthenticatedUserId({ request, locals }); + if ("response" in authResult) return authResult.response; + const userId = authResult.userId; + + const sources = await loadSourceApiItems(userId); + + return new Response(JSON.stringify({ success: true, sources }), { + status: 200, + headers: { "Content-Type": "application/json" }, + }); + } catch (error) { + return createSecureErrorResponse(error, "sources fetch", 500); + } +}; + +export const POST: APIRoute = async ({ request, locals }) => { + try { + const authResult = await requireAuthenticatedUserId({ request, locals }); + if ("response" in authResult) return authResult.response; + const userId = authResult.userId; + + const body = (await request.json()) as { + name?: unknown; + provider?: unknown; + url?: unknown; + username?: unknown; + token?: unknown; + enabled?: unknown; + }; + + if (!isSourceProviderKind(body.provider)) { + return new Response( + JSON.stringify({ success: false, message: "Unknown source provider." }), + { + status: 400, + headers: { "Content-Type": "application/json" }, + } + ); + } + const provider = body.provider; + + // GitLab and Gitea sources point at an instance; GitHub needs no URL. + if (provider !== "github") { + const rawSourceUrl = typeof body.url === "string" ? body.url.trim() : ""; + if (rawSourceUrl && !isValidSourceUrl(rawSourceUrl)) { + return new Response( + JSON.stringify({ + success: false, + message: `${SOURCE_PROVIDER_LABELS[provider]} URL must be a valid http or https URL.`, + }), + { + status: 400, + headers: { "Content-Type": "application/json" }, + } + ); + } + } + + let created: Awaited>; + try { + created = await createSource(userId, { + name: typeof body.name === "string" ? body.name : undefined, + provider, + url: typeof body.url === "string" ? body.url : undefined, + username: typeof body.username === "string" ? body.username : undefined, + token: typeof body.token === "string" ? body.token : undefined, + enabled: typeof body.enabled === "boolean" ? body.enabled : undefined, + }); + } catch (error) { + if (isDuplicateSourceError(error)) { + const message = + error instanceof Error && error.message + ? error.message + : "This source is already connected."; + return new Response(JSON.stringify({ success: false, message }), { + status: 409, + headers: { "Content-Type": "application/json" }, + }); + } + throw error; + } + + // createSource syncs the legacy config's connection fields itself. + const source = await toSourceApiItem(created, 0); + + return new Response(JSON.stringify({ success: true, source }), { + status: 201, + headers: { "Content-Type": "application/json" }, + }); + } catch (error) { + return createSecureErrorResponse(error, "source create", 500); + } +}; diff --git a/src/pages/api/sync/index.ts b/src/pages/api/sync/index.ts index e04835d..e3c261c 100644 --- a/src/pages/api/sync/index.ts +++ b/src/pages/api/sync/index.ts @@ -3,7 +3,7 @@ import { db, organizations, repositories, configs } from "@/lib/db"; import { eq, and, sql } from "drizzle-orm"; import { v4 as uuidv4 } from "uuid"; import { createMirrorJob } from "@/lib/helpers"; -import { createSourceProviderFromConfig, SOURCE_PROVIDER_LABELS } from "@/lib/source-providers"; +import { SOURCE_PROVIDER_LABELS } from "@/lib/source-providers"; import { jsonResponse, createSecureErrorResponse } from "@/lib/utils"; import { mergeGitReposPreferStarred, @@ -36,17 +36,22 @@ export const POST: APIRoute = async ({ request, locals }) => { }); } - if (!config.githubConfig?.token) { + // Every enabled source the user connected, oldest first (the first entry + // is the primary source). A source without a token still lists public + // repositories through its provider. Configs written outside the sources + // API get their first source row seeded here. + const { listSources, ensureSourcesFromConfig } = await import("@/lib/sources"); + const { createSourceProviderFromSource } = await import("@/lib/source-providers"); + await ensureSourcesFromConfig(userId); + const sources = (await listSources(userId)).filter((source) => source.enabled); + + if (sources.length === 0) { return jsonResponse({ - data: { error: "Source token is missing in config" }, + data: { error: "No enabled source is configured for this user" }, status: 400, }); } - // The configured source host, with its token decrypted - const sourceProvider = createSourceProviderFromConfig(config, { userId }); - const sourceLabel = SOURCE_PROVIDER_LABELS[sourceProvider.kind]; - // Load ignored orgs from the DB so we can skip them during import const ignoredOrgRows = await db .select({ normalizedName: organizations.normalizedName }) @@ -56,192 +61,237 @@ export const POST: APIRoute = async ({ request, locals }) => { // Per-organization fork pins, so an org opted out of forks is not // re-imported by this bulk pass even when the global switch is off. + // Loaded once; the pins apply to every source. const orgForkOverrides = await loadOrganizationForkPolicies({ userId }); - // Fetch source data in parallel - const [basicAndForkedRepos, starredRepos, orgResult] = await Promise.all([ - sourceProvider.listRepositories(config, { orgForkOverrides }), - config.githubConfig?.includeStarred - ? sourceProvider.listStarredRepositories(config) - : Promise.resolve([]), - sourceProvider.listOrganizations(config, ignoredOrgNames), - ]); - const { organizations: gitOrgs, failedOrgs } = orgResult; + let totalInsertedRepos = 0; + let totalInsertedOrgs = 0; + let totalRecoveredOrgs = 0; + let totalSkippedDisabled = 0; + const failedOrgNames: string[] = []; + const failedSourceNames: string[] = []; - // Merge and de-duplicate by fullName, preferring starred variant when duplicated - const allGithubRepos = mergeGitReposPreferStarred(basicAndForkedRepos, starredRepos); - const mirrorableGithubRepos = allGithubRepos.filter(isMirrorableGitHubRepo); + for (const source of sources) { + try { + const sourceProvider = createSourceProviderFromSource(source, { userId }); + const sourceLabel = SOURCE_PROVIDER_LABELS[sourceProvider.kind]; - // Prepare full list of repos and orgs. The normalizer stamps the source - // provider and URL on every row. - const newRepos = mirrorableGithubRepos.map((repo) => - normalizeGitRepoToInsert(repo, { userId, configId: config.id }) - ); + // Fetch source data in parallel + const [basicAndForkedRepos, starredRepos, orgResult] = await Promise.all([ + sourceProvider.listRepositories(config, { orgForkOverrides }), + config.githubConfig?.includeStarred + ? sourceProvider.listStarredRepositories(config) + : Promise.resolve([]), + sourceProvider.listOrganizations(config, ignoredOrgNames), + ]); + const { organizations: gitOrgs, failedOrgs } = orgResult; - const newOrgs = gitOrgs.map((org) => ({ - id: uuidv4(), - userId, - configId: config.id, - name: org.name, - normalizedName: org.name.toLowerCase(), - avatarUrl: org.avatarUrl, - membershipRole: org.membershipRole, - isIncluded: false, - status: org.status, - repositoryCount: org.repositoryCount, - createdAt: new Date(), - updatedAt: new Date(), - })); + // Merge and de-duplicate by fullName, preferring starred variant when duplicated + const allGithubRepos = mergeGitReposPreferStarred(basicAndForkedRepos, starredRepos); + const mirrorableGithubRepos = allGithubRepos.filter(isMirrorableGitHubRepo); + totalSkippedDisabled += allGithubRepos.length - mirrorableGithubRepos.length; - // Prepare failed org records for DB insertion - const failedOrgRecords = failedOrgs.map((org) => ({ - id: uuidv4(), - userId, - configId: config.id, - name: org.name, - normalizedName: org.name.toLowerCase(), - avatarUrl: org.avatarUrl, - membershipRole: "member" as const, - isIncluded: false, - status: "failed" as const, - errorMessage: org.reason, - repositoryCount: 0, - createdAt: new Date(), - updatedAt: new Date(), - })); + // Prepare full list of repos and orgs. The normalizer stamps the source + // provider and URL on every row; sourceId ties the row to this source. + const newRepos = mirrorableGithubRepos.map((repo) => + normalizeGitRepoToInsert(repo, { userId, configId: config.id, sourceId: source.id }) + ); - let insertedRepos: typeof newRepos = []; - let insertedOrgs: typeof newOrgs = []; - let insertedFailedOrgs: typeof failedOrgRecords = []; - let recoveredOrgCount = 0; - - // Transaction to insert only new items - await db.transaction(async (tx) => { - const [existingRepos, existingOrgs] = await Promise.all([ - tx - .select({ normalizedFullName: repositories.normalizedFullName }) - .from(repositories) - .where(eq(repositories.userId, userId)), - tx - .select({ normalizedName: organizations.normalizedName, status: organizations.status }) - .from(organizations) - .where(eq(organizations.userId, userId)), - ]); - - const existingRepoNames = new Set(existingRepos.map((r) => r.normalizedFullName)); - const existingOrgMap = new Map(existingOrgs.map((o) => [o.normalizedName, o.status])); - - insertedRepos = newRepos.filter( - (r) => - !existingRepoNames.has(r.normalizedFullName) && - (!r.organization || !ignoredOrgNames.has(r.organization.toLowerCase())) - ); - insertedOrgs = newOrgs.filter((o) => !existingOrgMap.has(o.normalizedName)); - - // Update previously failed orgs that now succeeded - const recoveredOrgs = newOrgs.filter( - (o) => existingOrgMap.get(o.normalizedName) === "failed" - ); - for (const org of recoveredOrgs) { - await tx - .update(organizations) - .set({ - status: "imported", - errorMessage: null, - repositoryCount: org.repositoryCount, - avatarUrl: org.avatarUrl, - membershipRole: org.membershipRole, - updatedAt: new Date(), - }) - .where( - and( - eq(organizations.userId, userId), - eq(organizations.normalizedName, org.normalizedName), - ) - ); - } - recoveredOrgCount = recoveredOrgs.length; - - // Insert or update failed orgs (only update orgs already in "failed" state — don't overwrite good state) - insertedFailedOrgs = failedOrgRecords.filter((o) => !existingOrgMap.has(o.normalizedName)); - const stillFailedOrgs = failedOrgRecords.filter( - (o) => existingOrgMap.get(o.normalizedName) === "failed" - ); - for (const org of stillFailedOrgs) { - await tx - .update(organizations) - .set({ - errorMessage: org.errorMessage, - updatedAt: new Date(), - }) - .where( - and( - eq(organizations.userId, userId), - eq(organizations.normalizedName, org.normalizedName), - ) - ); - } - - // Batch insert repositories to avoid SQLite parameter limit (dynamic by column count) - const sample = newRepos[0]; - const columnCount = Object.keys(sample ?? {}).length || 1; - const REPO_BATCH_SIZE = calcBatchSizeForInsert(columnCount); - if (insertedRepos.length > 0) { - for (let i = 0; i < insertedRepos.length; i += REPO_BATCH_SIZE) { - const batch = insertedRepos.slice(i, i + REPO_BATCH_SIZE); - await tx - .insert(repositories) - .values(batch) - .onConflictDoNothing({ target: [repositories.userId, repositories.normalizedFullName] }); - } - } - - // Batch insert organizations (they have fewer fields, so we can use larger batches) - const ORG_BATCH_SIZE = 100; - const allNewOrgs = [...insertedOrgs, ...insertedFailedOrgs]; - if (allNewOrgs.length > 0) { - for (let i = 0; i < allNewOrgs.length; i += ORG_BATCH_SIZE) { - const batch = allNewOrgs.slice(i, i + ORG_BATCH_SIZE); - await tx.insert(organizations).values(batch); - } - } - }); - - // Create mirror jobs only for newly inserted items - const mirrorJobPromises = [ - ...insertedRepos.map((repo) => - createMirrorJob({ + const newOrgs = gitOrgs.map((org) => ({ + id: uuidv4(), userId, - repositoryId: repo.id, - repositoryName: repo.name, - status: "imported", - message: `Repository ${repo.name} fetched successfully`, - details: `Repository ${repo.name} was fetched from ${sourceLabel}`, - }) - ), - ...insertedOrgs.map((org) => - createMirrorJob({ - userId, - organizationId: org.id, - organizationName: org.name, - status: "imported", - message: `Organization ${org.name} fetched successfully`, - details: `Organization ${org.name} was fetched from ${sourceLabel}`, - }) - ), - ]; + configId: config.id, + name: org.name, + normalizedName: org.name.toLowerCase(), + avatarUrl: org.avatarUrl, + membershipRole: org.membershipRole, + isIncluded: false, + status: org.status, + repositoryCount: org.repositoryCount, + createdAt: new Date(), + updatedAt: new Date(), + })); - await Promise.all(mirrorJobPromises); + // Prepare failed org records for DB insertion + const failedOrgRecords = failedOrgs.map((org) => ({ + id: uuidv4(), + userId, + configId: config.id, + name: org.name, + normalizedName: org.name.toLowerCase(), + avatarUrl: org.avatarUrl, + membershipRole: "member" as const, + isIncluded: false, + status: "failed" as const, + errorMessage: org.reason, + repositoryCount: 0, + createdAt: new Date(), + updatedAt: new Date(), + })); + + let insertedRepos: typeof newRepos = []; + let insertedOrgs: typeof newOrgs = []; + let insertedFailedOrgs: typeof failedOrgRecords = []; + let recoveredOrgCount = 0; + + // Transaction to insert only new items + await db.transaction(async (tx) => { + const [existingRepos, existingOrgs] = await Promise.all([ + tx + .select({ + normalizedFullName: repositories.normalizedFullName, + sourceId: repositories.sourceId, + }) + .from(repositories) + .where(eq(repositories.userId, userId)), + tx + .select({ normalizedName: organizations.normalizedName, status: organizations.status }) + .from(organizations) + .where(eq(organizations.userId, userId)), + ]); + + // The same full name under two sources is two different repositories. + const existingRepoKeys = new Set( + existingRepos.map((r) => `${r.sourceId ?? ""}|${r.normalizedFullName}`) + ); + const existingOrgMap = new Map(existingOrgs.map((o) => [o.normalizedName, o.status])); + + insertedRepos = newRepos.filter( + (r) => + !existingRepoKeys.has(`${source.id}|${r.normalizedFullName}`) && + (!r.organization || !ignoredOrgNames.has(r.organization.toLowerCase())) + ); + insertedOrgs = newOrgs.filter((o) => !existingOrgMap.has(o.normalizedName)); + + // Update previously failed orgs that now succeeded + const recoveredOrgs = newOrgs.filter( + (o) => existingOrgMap.get(o.normalizedName) === "failed" + ); + for (const org of recoveredOrgs) { + await tx + .update(organizations) + .set({ + status: "imported", + errorMessage: null, + repositoryCount: org.repositoryCount, + avatarUrl: org.avatarUrl, + membershipRole: org.membershipRole, + updatedAt: new Date(), + }) + .where( + and( + eq(organizations.userId, userId), + eq(organizations.normalizedName, org.normalizedName), + ) + ); + } + recoveredOrgCount = recoveredOrgs.length; + + // Insert or update failed orgs (only update orgs already in "failed" state — don't overwrite good state) + insertedFailedOrgs = failedOrgRecords.filter((o) => !existingOrgMap.has(o.normalizedName)); + const stillFailedOrgs = failedOrgRecords.filter( + (o) => existingOrgMap.get(o.normalizedName) === "failed" + ); + for (const org of stillFailedOrgs) { + await tx + .update(organizations) + .set({ + errorMessage: org.errorMessage, + updatedAt: new Date(), + }) + .where( + and( + eq(organizations.userId, userId), + eq(organizations.normalizedName, org.normalizedName), + ) + ); + } + + // Batch insert repositories to avoid SQLite parameter limit (dynamic by column count) + const sample = newRepos[0]; + const columnCount = Object.keys(sample ?? {}).length || 1; + const REPO_BATCH_SIZE = calcBatchSizeForInsert(columnCount); + if (insertedRepos.length > 0) { + for (let i = 0; i < insertedRepos.length; i += REPO_BATCH_SIZE) { + const batch = insertedRepos.slice(i, i + REPO_BATCH_SIZE); + await tx + .insert(repositories) + .values(batch) + .onConflictDoNothing({ + target: [repositories.userId, repositories.sourceId, repositories.normalizedFullName], + }); + } + } + + // Batch insert organizations (they have fewer fields, so we can use larger batches) + const ORG_BATCH_SIZE = 100; + const allNewOrgs = [...insertedOrgs, ...insertedFailedOrgs]; + if (allNewOrgs.length > 0) { + for (let i = 0; i < allNewOrgs.length; i += ORG_BATCH_SIZE) { + const batch = allNewOrgs.slice(i, i + ORG_BATCH_SIZE); + await tx.insert(organizations).values(batch); + } + } + }); + + // Create mirror jobs only for newly inserted items + const mirrorJobPromises = [ + ...insertedRepos.map((repo) => + createMirrorJob({ + userId, + repositoryId: repo.id, + repositoryName: repo.name, + status: "imported", + message: `Repository ${repo.name} fetched successfully`, + details: `Repository ${repo.name} was fetched from ${sourceLabel}`, + }) + ), + ...insertedOrgs.map((org) => + createMirrorJob({ + userId, + organizationId: org.id, + organizationName: org.name, + status: "imported", + message: `Organization ${org.name} fetched successfully`, + details: `Organization ${org.name} was fetched from ${sourceLabel}`, + }) + ), + ]; + + await Promise.all(mirrorJobPromises); + + totalInsertedRepos += insertedRepos.length; + totalInsertedOrgs += insertedOrgs.length; + totalRecoveredOrgs += recoveredOrgCount; + failedOrgNames.push( + ...failedOrgs + .filter((o) => !ignoredOrgNames.has(o.name.toLowerCase())) + .map((o) => o.name) + ); + } catch (sourceError) { + console.error(`Sync failed for source ${source.name} (${source.id}):`, sourceError); + failedSourceNames.push(source.name); + } + } + + if (sources.length > 0 && failedSourceNames.length === sources.length) { + return createSecureErrorResponse( + new Error("every enabled source failed to sync"), + "source data sync", + 502 + ); + } return jsonResponse({ data: { success: true, message: "Repositories and organizations synced successfully", - newRepositories: insertedRepos.length, - newOrganizations: insertedOrgs.length, - skippedDisabledRepositories: allGithubRepos.length - mirrorableGithubRepos.length, - failedOrgs: failedOrgs.filter((o) => !ignoredOrgNames.has(o.name.toLowerCase())).map((o) => o.name), - recoveredOrgs: recoveredOrgCount, + newRepositories: totalInsertedRepos, + newOrganizations: totalInsertedOrgs, + skippedDisabledRepositories: totalSkippedDisabled, + failedOrgs: failedOrgNames, + failedSources: failedSourceNames, + recoveredOrgs: totalRecoveredOrgs, }, }); } catch (error) { diff --git a/src/pages/api/sync/organization.ts b/src/pages/api/sync/organization.ts index 0b2282b..e34f880 100644 --- a/src/pages/api/sync/organization.ts +++ b/src/pages/api/sync/organization.ts @@ -8,7 +8,7 @@ import type { } from "@/types/organizations"; import type { RepoStatus } from "@/types/Repository"; import { v4 as uuidv4 } from "uuid"; -import { createSourceProviderFromConfig, isValidSourceOrgName } from "@/lib/source-providers"; +import { isValidSourceOrgName } from "@/lib/source-providers"; import { normalizeGitRepoToInsert, calcBatchSizeForInsert } from "@/lib/repo-utils"; import { resolveOrganizationSkipForks } from "@/lib/utils/mirror-overrides"; import { requireAuthenticatedUserId } from "@/lib/auth-guards"; @@ -21,6 +21,8 @@ export const POST: APIRoute = async ({ request, locals }) => { const body: AddOrganizationApiRequest = await request.json(); const { role, org, force = false } = body; + // Optional body field: which connected source to import from; defaults to the primary source. + const sourceId = (body as AddOrganizationApiRequest & { sourceId?: string }).sourceId?.trim() || undefined; if (!org || !role) { return jsonResponse({ @@ -118,8 +120,34 @@ export const POST: APIRoute = async ({ request, locals }) => { }); } - // The configured source host, with its token decrypted - const sourceProvider = createSourceProviderFromConfig(config, { userId }); + const { listSources } = await import("@/lib/sources"); + const { createSourceProviderFromSource } = await import("@/lib/source-providers"); + const userSources = await listSources(userId); + const source = sourceId + ? userSources.find((s) => s.id === sourceId) + : userSources[0]; + + if (sourceId && !source) { + return jsonResponse({ + data: { + success: false, + error: `No source with id ${sourceId} belongs to this user`, + }, + status: 400, + }); + } + + if (!source) { + return jsonResponse({ + data: { + success: false, + error: "No source is configured for this user", + }, + status: 400, + }); + } + + const sourceProvider = createSourceProviderFromSource(source, { userId }); // Fetch org metadata const orgData = await sourceProvider.getOrganization(trimmedOrg); @@ -147,7 +175,7 @@ export const POST: APIRoute = async ({ request, locals }) => { const repoRecords = mirrorableRepos.map((repo) => normalizeGitRepoToInsert( { ...repo, organization: repo.organization ?? orgData.name }, - { userId, configId } + { userId, configId, sourceId: source.id } ) ); @@ -161,7 +189,9 @@ export const POST: APIRoute = async ({ request, locals }) => { await db .insert(repositories) .values(batch) - .onConflictDoNothing({ target: [repositories.userId, repositories.normalizedFullName] }); + .onConflictDoNothing({ + target: [repositories.userId, repositories.sourceId, repositories.normalizedFullName], + }); } // Insert organization metadata diff --git a/src/pages/api/sync/repository.ts b/src/pages/api/sync/repository.ts index b6b2473..da36314 100644 --- a/src/pages/api/sync/repository.ts +++ b/src/pages/api/sync/repository.ts @@ -12,8 +12,6 @@ import { createMirrorJob } from "@/lib/helpers"; import { requireAuthenticatedUserId } from "@/lib/auth-guards"; import { SOURCE_PROVIDER_LABELS, - createSourceProviderFromConfig, - describeSource, sourceHostOf, } from "@/lib/source-providers"; import { repositorySourceColumns } from "@/lib/repo-utils"; @@ -68,24 +66,44 @@ export const POST: APIRoute = async ({ request, locals }) => { const configId = config.id; - // The configured source host. Without a token GitHub still allows public - // lookups, so a missing token is not an error here. - const sourceProvider = createSourceProviderFromConfig(config, { userId }); - const sourceConnection = sourceProvider.connection; - const sourceLabel = SOURCE_PROVIDER_LABELS[sourceProvider.kind]; + // The source that serves this URL: when the request pastes a host, the + // connected source whose host matches it; otherwise the primary source. + // The URL shape cannot tell providers apart, so multiple sources on the + // same host resolve to the primary one. + const { listSources } = await import("@/lib/sources"); + const { createSourceProviderFromSource } = await import("@/lib/source-providers"); + const userSources = await listSources(userId); - // A pasted URL that names a different host cannot be served by this source. - const configuredHost = sourceHostOf(sourceConnection.url); - if (host && configuredHost && host.trim().toLowerCase() !== configuredHost) { + const pastedHost = host?.trim().toLowerCase() || ""; + let source = userSources[0]; + if (pastedHost) { + const hostMatches = userSources.filter((s) => sourceHostOf(s.url) === pastedHost); + if (hostMatches.length === 0) { + return jsonResponse({ + data: { + success: false, + error: `No connected source matches host ${pastedHost}; add it on the Configuration page first`, + }, + status: 400, + }); + } + source = hostMatches.includes(userSources[0]) ? userSources[0] : hostMatches[0]; + } + + if (!source) { return jsonResponse({ data: { success: false, - error: `That URL points at ${host}, but the configured source is ${describeSource(sourceConnection)}.`, + error: "No source is configured for this user; add one on the Configuration page first", }, status: 400, }); } + // A missing token still allows public lookups on every provider. + const sourceProvider = createSourceProviderFromSource(source, { userId }); + const sourceLabel = SOURCE_PROVIDER_LABELS[sourceProvider.kind]; + // Prefer the pasted path segments: the source host knows how to split // them (GitLab nests groups, GitHub and Gitea do not). const resolvedPath = @@ -95,13 +113,15 @@ export const POST: APIRoute = async ({ request, locals }) => { const normalizedFullName = `${resolvedPath.owner}/${resolvedPath.repo}`.toLowerCase(); - // Check if repository with the same owner, name, and userId already exists + // A repository is unique per source: the same full name under another + // source is a different repository and may coexist. const [existingRepo] = await db .select() .from(repositories) .where( and( eq(repositories.userId, userId), + eq(repositories.sourceId, source.id), eq(repositories.normalizedFullName, normalizedFullName) ) ) @@ -132,6 +152,7 @@ export const POST: APIRoute = async ({ request, locals }) => { const baseMetadata = { userId, configId, + sourceId: source.id, name: repoData.name, fullName: repoData.fullName, normalizedFullName: repoData.fullName.toLowerCase(), @@ -194,7 +215,9 @@ export const POST: APIRoute = async ({ request, locals }) => { await db .insert(repositories) .values(metadata) - .onConflictDoNothing({ target: [repositories.userId, repositories.normalizedFullName] }); + .onConflictDoNothing({ + target: [repositories.userId, repositories.sourceId, repositories.normalizedFullName], + }); createMirrorJob({ userId, diff --git a/src/tests/setup.bun.ts b/src/tests/setup.bun.ts index c9eb724..ddf553c 100644 --- a/src/tests/setup.bun.ts +++ b/src/tests/setup.bun.ts @@ -31,7 +31,11 @@ mock.module("@/lib/db", () => { select: () => ({ from: () => ({ where: () => ({ - limit: () => Promise.resolve([]) + limit: () => Promise.resolve([]), + // Sources queries order or group before awaiting; resolve to no + // rows so callers see an empty result. + orderBy: () => Promise.resolve([]), + groupBy: () => Promise.resolve([]) }) }) }), @@ -53,6 +57,7 @@ mock.module("@/lib/db", () => { users: {}, events: {}, configs: {}, + sources: {}, repositories: {}, mirrorJobs: {}, organizations: {}, diff --git a/src/types/Repository.ts b/src/types/Repository.ts index ac1061f..26fe5d0 100644 --- a/src/types/Repository.ts +++ b/src/types/Repository.ts @@ -55,6 +55,8 @@ export interface GitRepo { /** Host the repository came from. Absent means GitHub. */ sourceProvider?: SourceProviderKind; sourceUrl?: string; + /** Source row this repository belongs to; dangling (or null) after the source was removed. */ + sourceId?: string | null; destinationOrg?: string | null; isPrivate: boolean; diff --git a/src/types/config.ts b/src/types/config.ts index 531f47c..3f19daa 100644 --- a/src/types/config.ts +++ b/src/types/config.ts @@ -113,6 +113,26 @@ export interface AdvancedOptions { export interface ConfigLockState { source: { locked: boolean; repositoryCount: number }; destination: { locked: boolean; mirroredCount: number }; + /** + * Per-source lock state for multi-source: one entry per distinct + * repositories.source_id, plus a null entry for repositories whose source + * row is gone or was never linked (those count toward no source's lock). + * Absent when the counts could not be read. `source` above keeps + * reflecting the TOTAL repository count for backward compatibility. + */ + sources?: Array<{ sourceId: string | null; locked: boolean; repositoryCount: number }>; +} + +export interface SourceApiRecord { + id: string; + name: string; + provider: SourceProvider; + url: string; + username: string; + token: string; + enabled: boolean; + repositoryCount: number; + locked: boolean; } export interface SaveConfigApiRequest { @@ -194,5 +214,6 @@ export interface ConfigApiResponse { createdAt: Date; updatedAt: Date; locks?: ConfigLockState; + sources?: SourceApiRecord[]; error?: string; } diff --git a/www/src/pages/docs/sources.mdx b/www/src/pages/docs/sources.mdx index 4b0f411..55fa056 100644 --- a/www/src/pages/docs/sources.mdx +++ b/www/src/pages/docs/sources.mdx @@ -6,7 +6,7 @@ description: Where Gitea Mirror pulls repositories from. GitHub is the default; # Sources -A source is the host Gitea Mirror pulls repositories from. Each account has one source at a time, chosen from the **Source** dropdown at the top of the first card on the Connections tab of the Configuration page. +A source is the host Gitea Mirror pulls repositories from. Each account can connect several sources at once, managed as a list in the first card on the Connections tab of the Configuration page. | Source | Covers | Default instance | | --- | --- | --- | @@ -113,4 +113,4 @@ The [environment variables reference](./environment-variables.md) has the comple ## Not yet -Several sources at once is tracked in [issue #375](https://github.com/RayLabsHQ/gitea-mirror/issues/375). Issues, merge requests and releases from GitLab and Gitea sources are not mirrored. +Issues, merge requests and releases from GitLab and Gitea sources are not mirrored. diff --git a/www/src/pages/use-cases/mirror-gitlab-to-gitea.mdx b/www/src/pages/use-cases/mirror-gitlab-to-gitea.mdx index e65b933..9370386 100644 --- a/www/src/pages/use-cases/mirror-gitlab-to-gitea.mdx +++ b/www/src/pages/use-cases/mirror-gitlab-to-gitea.mdx @@ -59,11 +59,11 @@ The git side is complete. Code, every branch and tag, the wiki when enabled, and The metadata side is GitHub only for now. Issues, merge requests, labels, milestones and releases are not read from the GitLab API yet, so those switches are disabled on the Configuration page and the mirror step skips them. Force push detection also depends on the GitHub API and is unavailable for GitLab sources. The [issue and pull request playbook](/use-cases/mirror-github-issues-pull-requests/) only applies to GitHub sources. -## One source at a time +## Several sources at once -An account has one source. When you switch from GitHub to GitLab, the repositories imported from GitHub are not touched. They keep syncing, because Gitea already holds their clone configuration, but the cleanup service ignores them and mirroring one of them again is refused with an error that names both hosts. If you need a GitHub repository re-mirrored after the switch, remove it and add it again from the current source. +GitHub and GitLab sources can be connected at the same time, and each repository remembers the source it came from. Cleanup checks each source's own repositories, and mirroring a repository uses that source's credentials. Repositories whose source was later removed keep syncing on the Gitea side, because Gitea holds their clone configuration, but re-mirroring, metadata and cleanup skip them. -The same lock applies in the other direction. Once projects are imported from GitLab, the source and instance URL lock until you confirm a change. +Each source locks once repositories are imported from it. Its provider and instance URL stay fixed until you confirm a change. ## Environment variables