diff --git a/CHANGELOG.md b/CHANGELOG.md index 7cc6c12..2926ac9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Raised the dependency floors for the advisories published on 2026-09-08: Astro 7.2.8 (remote code execution through AVIF image optimization, and an authorization bypass when stripping the configured base), @xmldom/xmldom 0.8.15 (eight parser and serializer issues), sharp 0.35.4 (libheif), svgo 4.1.0 (removeScripts sanitization) and js-yaml 4.3.2 (merge-key CPU use). Applied to both the application and the documentation site. ### Added +- A Storage page that shows how much space the mirrors take on the destination (#479) + - Cards for the total, Git + LFS, release assets and how many mirrors are measured, bars by destination owner and for the largest repositories, and a sortable table with search and an owner filter + - Sizes come from responses a sync already fetches: the destination repository lookup before `mirror-sync` (Git plus LFS) and the release listing the retention pass reads (release assets). No extra requests to the source or the destination + - A repository whose release assets are most of its size gets a hint that a lower release limit removes the oldest releases + - The same numbers are available from `GET /api/storage` - Read only API keys - **Create key** in the **API Keys** card has an **Access** choice, and read only keys show a badge in the list - A read only key can call every `GET` endpoint; any `POST`, `PUT`, `PATCH` or `DELETE`, and every `/api/auth/*` call except `get-session`, gets `403` with the code `API_KEY_READ_ONLY` diff --git a/docs/API.md b/docs/API.md index 299b88d..2cf5f6b 100644 --- a/docs/API.md +++ b/docs/API.md @@ -156,6 +156,21 @@ When several sources are connected, an optional `sourceId` picks which one the o `PATCH /api/organizations/:id` with `{ "sourceId": "…" }` re-pins an existing organization to another source, and `{ "sourceId": null }` clears the pin so the organization follows every source's repositories of the same name again. A pin whose source was removed falls back the same way. +### Storage + +`GET /api/storage` returns what the Storage page shows: how much space the account's mirrors take on the destination (issue #479). + +```json +{ + "success": true, + "totals": { "total": 3554304, "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "measured": 1, "mirrored": 1 }, + "owners": [{ "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "total": 3554304, "repositories": 1 }], + "repositories": [{ "id": "…", "fullName": "upstream/app", "location": "archive/app", "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "total": 3554304, "checkedAt": "2026-10-29T03:32:01.000Z" }] +} +``` + +Sizes are bytes. `destinationSize` is Git and LFS together as Gitea or Forgejo reports it, and the release fields cover the assets on the destination's releases. Each sync records them from responses it already fetches, so this endpoint never contacts the destination. A repository is listed once a sync has measured it, and a field not measured yet is `null`. `mirrored` counts the repositories with a mirror on a Gitea or Forgejo destination and `measured` the ones with a size so far; push destinations (GitHub, GitLab) are not measured. + ### Export as CSV `GET /api/repositories/export` and `GET /api/organizations/export` return everything the account tracks as a CSV file, so the list can be shared with people who do not have a login here (issue #428). diff --git a/drizzle/0021_repository_storage.sql b/drizzle/0021_repository_storage.sql new file mode 100644 index 0000000..00ed650 --- /dev/null +++ b/drizzle/0021_repository_storage.sql @@ -0,0 +1,4 @@ +ALTER TABLE `repositories` ADD `destination_size` integer;--> statement-breakpoint +ALTER TABLE `repositories` ADD `release_assets_size` integer;--> statement-breakpoint +ALTER TABLE `repositories` ADD `release_assets_count` integer;--> statement-breakpoint +ALTER TABLE `repositories` ADD `storage_checked_at` integer; \ No newline at end of file diff --git a/drizzle/meta/0021_snapshot.json b/drizzle/meta/0021_snapshot.json new file mode 100644 index 0000000..e0fd4cb --- /dev/null +++ b/drizzle/meta/0021_snapshot.json @@ -0,0 +1,3173 @@ +{ + "version": "6", + "dialect": "sqlite", + "id": "a4340422-90af-409b-9dd3-6b8e316732ba", + "prevId": "06a2c1a2-856b-4831-82c0-996a707c220a", + "tables": { + "accounts": { + "name": "accounts", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "account_id": { + "name": "account_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "provider_id": { + "name": "provider_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "issuer": { + "name": "issuer", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "provider_user_id": { + "name": "provider_user_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "access_token": { + "name": "access_token", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "refresh_token": { + "name": "refresh_token", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "id_token": { + "name": "id_token", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "access_token_expires_at": { + "name": "access_token_expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "refresh_token_expires_at": { + "name": "refresh_token_expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "scope": { + "name": "scope", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "password": { + "name": "password", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_accounts_account_id": { + "name": "idx_accounts_account_id", + "columns": [ + "account_id" + ], + "isUnique": false + }, + "idx_accounts_user_id": { + "name": "idx_accounts_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + }, + "idx_accounts_provider": { + "name": "idx_accounts_provider", + "columns": [ + "provider_id", + "provider_user_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "accounts_user_id_users_id_fk": { + "name": "accounts_user_id_users_id_fk", + "tableFrom": "accounts", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "api_keys": { + "name": "api_keys", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "config_id": { + "name": "config_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'default'" + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "start": { + "name": "start", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "prefix": { + "name": "prefix", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "key": { + "name": "key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "reference_id": { + "name": "reference_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "refill_interval": { + "name": "refill_interval", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "refill_amount": { + "name": "refill_amount", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_refill_at": { + "name": "last_refill_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "rate_limit_enabled": { + "name": "rate_limit_enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "rate_limit_time_window": { + "name": "rate_limit_time_window", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "rate_limit_max": { + "name": "rate_limit_max", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "request_count": { + "name": "request_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "remaining": { + "name": "remaining", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_request": { + "name": "last_request", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "permissions": { + "name": "permissions", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "metadata": { + "name": "metadata", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "idx_api_keys_reference_id": { + "name": "idx_api_keys_reference_id", + "columns": [ + "reference_id" + ], + "isUnique": false + }, + "idx_api_keys_key": { + "name": "idx_api_keys_key", + "columns": [ + "key" + ], + "isUnique": false + } + }, + "foreignKeys": { + "api_keys_reference_id_users_id_fk": { + "name": "api_keys_reference_id_users_id_fk", + "tableFrom": "api_keys", + "tableTo": "users", + "columnsFrom": [ + "reference_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "configs": { + "name": "configs", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "is_active": { + "name": "is_active", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "github_config": { + "name": "github_config", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "gitea_config": { + "name": "gitea_config", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "include": { + "name": "include", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[\"*\"]'" + }, + "exclude": { + "name": "exclude", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[]'" + }, + "schedule_config": { + "name": "schedule_config", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "cleanup_config": { + "name": "cleanup_config", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "notification_config": { + "name": "notification_config", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{\"enabled\":false,\"provider\":\"ntfy\",\"notifyOnSyncError\":true,\"notifyOnSyncSuccess\":false,\"notifyOnNewRepo\":false}'" + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": {}, + "foreignKeys": { + "configs_user_id_users_id_fk": { + "name": "configs_user_id_users_id_fk", + "tableFrom": "configs", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "events": { + "name": "events", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "channel": { + "name": "channel", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "payload": { + "name": "payload", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "read": { + "name": "read", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_events_user_channel": { + "name": "idx_events_user_channel", + "columns": [ + "user_id", + "channel" + ], + "isUnique": false + }, + "idx_events_created_at": { + "name": "idx_events_created_at", + "columns": [ + "created_at" + ], + "isUnique": false + }, + "idx_events_read": { + "name": "idx_events_read", + "columns": [ + "read" + ], + "isUnique": false + } + }, + "foreignKeys": { + "events_user_id_users_id_fk": { + "name": "events_user_id_users_id_fk", + "tableFrom": "events", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "jwks": { + "name": "jwks", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "public_key": { + "name": "public_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "private_key": { + "name": "private_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "alg": { + "name": "alg", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "crv": { + "name": "crv", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "mirror_jobs": { + "name": "mirror_jobs", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "repository_id": { + "name": "repository_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "repository_name": { + "name": "repository_name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "organization_name": { + "name": "organization_name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "details": { + "name": "details", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'imported'" + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "timestamp": { + "name": "timestamp", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "job_type": { + "name": "job_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'mirror'" + }, + "batch_id": { + "name": "batch_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "total_items": { + "name": "total_items", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "completed_items": { + "name": "completed_items", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": 0 + }, + "item_ids": { + "name": "item_ids", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "completed_item_ids": { + "name": "completed_item_ids", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "'[]'" + }, + "in_progress": { + "name": "in_progress", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "started_at": { + "name": "started_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "completed_at": { + "name": "completed_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_checkpoint": { + "name": "last_checkpoint", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "idx_mirror_jobs_user_id": { + "name": "idx_mirror_jobs_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + }, + "idx_mirror_jobs_batch_id": { + "name": "idx_mirror_jobs_batch_id", + "columns": [ + "batch_id" + ], + "isUnique": false + }, + "idx_mirror_jobs_in_progress": { + "name": "idx_mirror_jobs_in_progress", + "columns": [ + "in_progress" + ], + "isUnique": false + }, + "idx_mirror_jobs_job_type": { + "name": "idx_mirror_jobs_job_type", + "columns": [ + "job_type" + ], + "isUnique": false + }, + "idx_mirror_jobs_timestamp": { + "name": "idx_mirror_jobs_timestamp", + "columns": [ + "timestamp" + ], + "isUnique": false + } + }, + "foreignKeys": { + "mirror_jobs_user_id_users_id_fk": { + "name": "mirror_jobs_user_id_users_id_fk", + "tableFrom": "mirror_jobs", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "oauth_access_tokens": { + "name": "oauth_access_tokens", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "token": { + "name": "token", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "client_id": { + "name": "client_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "session_id": { + "name": "session_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "reference_id": { + "name": "reference_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "refresh_id": { + "name": "refresh_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + }, + "scopes": { + "name": "scopes", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "authorization_code_id": { + "name": "authorization_code_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "resources": { + "name": "resources", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "requested_user_info_claims": { + "name": "requested_user_info_claims", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "revoked": { + "name": "revoked", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "confirmation": { + "name": "confirmation", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "oauth_access_tokens_token_unique": { + "name": "oauth_access_tokens_token_unique", + "columns": [ + "token" + ], + "isUnique": true + }, + "idx_oauth_access_tokens_token": { + "name": "idx_oauth_access_tokens_token", + "columns": [ + "token" + ], + "isUnique": false + }, + "idx_oauth_access_tokens_client_id": { + "name": "idx_oauth_access_tokens_client_id", + "columns": [ + "client_id" + ], + "isUnique": false + }, + "idx_oauth_access_tokens_user_id": { + "name": "idx_oauth_access_tokens_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "oauth_access_tokens_user_id_users_id_fk": { + "name": "oauth_access_tokens_user_id_users_id_fk", + "tableFrom": "oauth_access_tokens", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "oauth_client_assertions": { + "name": "oauth_client_assertions", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "oauth_client_resources": { + "name": "oauth_client_resources", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "client_id": { + "name": "client_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "resource_id": { + "name": "resource_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "metadata": { + "name": "metadata", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_oauth_client_resources_client_id": { + "name": "idx_oauth_client_resources_client_id", + "columns": [ + "client_id" + ], + "isUnique": false + }, + "idx_oauth_client_resources_resource_id": { + "name": "idx_oauth_client_resources_resource_id", + "columns": [ + "resource_id" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "oauth_clients": { + "name": "oauth_clients", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "client_id": { + "name": "client_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "client_secret": { + "name": "client_secret", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "disabled": { + "name": "disabled", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": false + }, + "skip_consent": { + "name": "skip_consent", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "enable_end_session": { + "name": "enable_end_session", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "subject_type": { + "name": "subject_type", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "scopes": { + "name": "scopes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "uri": { + "name": "uri", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "icon": { + "name": "icon", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "contacts": { + "name": "contacts", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tos": { + "name": "tos", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "policy": { + "name": "policy", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "software_id": { + "name": "software_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "software_version": { + "name": "software_version", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "software_statement": { + "name": "software_statement", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "redirect_uris": { + "name": "redirect_uris", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "post_logout_redirect_uris": { + "name": "post_logout_redirect_uris", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "token_endpoint_auth_method": { + "name": "token_endpoint_auth_method", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "grant_types": { + "name": "grant_types", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "response_types": { + "name": "response_types", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "public": { + "name": "public", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "require_pkce": { + "name": "require_pkce", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "client_discovery_id": { + "name": "client_discovery_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "client_credentials_scopes": { + "name": "client_credentials_scopes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "backchannel_logout_uri": { + "name": "backchannel_logout_uri", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "backchannel_logout_session_required": { + "name": "backchannel_logout_session_required", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "application_type": { + "name": "application_type", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "jwks": { + "name": "jwks", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "jwks_uri": { + "name": "jwks_uri", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "dpop_bound_access_tokens": { + "name": "dpop_bound_access_tokens", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "reference_id": { + "name": "reference_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "metadata": { + "name": "metadata", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "oauth_clients_client_id_unique": { + "name": "oauth_clients_client_id_unique", + "columns": [ + "client_id" + ], + "isUnique": true + }, + "idx_oauth_clients_client_id": { + "name": "idx_oauth_clients_client_id", + "columns": [ + "client_id" + ], + "isUnique": false + }, + "idx_oauth_clients_user_id": { + "name": "idx_oauth_clients_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "oauth_clients_user_id_users_id_fk": { + "name": "oauth_clients_user_id_users_id_fk", + "tableFrom": "oauth_clients", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "oauth_consents": { + "name": "oauth_consents", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "client_id": { + "name": "client_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "reference_id": { + "name": "reference_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "scopes": { + "name": "scopes", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "resources": { + "name": "resources", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "requested_user_info_claims": { + "name": "requested_user_info_claims", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_oauth_consents_client_id": { + "name": "idx_oauth_consents_client_id", + "columns": [ + "client_id" + ], + "isUnique": false + }, + "idx_oauth_consents_user_id": { + "name": "idx_oauth_consents_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "oauth_consents_user_id_users_id_fk": { + "name": "oauth_consents_user_id_users_id_fk", + "tableFrom": "oauth_consents", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "oauth_refresh_tokens": { + "name": "oauth_refresh_tokens", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "token": { + "name": "token", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "client_id": { + "name": "client_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "session_id": { + "name": "session_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "reference_id": { + "name": "reference_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + }, + "revoked": { + "name": "revoked", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "auth_time": { + "name": "auth_time", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "scopes": { + "name": "scopes", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "authorization_code_id": { + "name": "authorization_code_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "resources": { + "name": "resources", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "requested_user_info_claims": { + "name": "requested_user_info_claims", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "rotated_at": { + "name": "rotated_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "rotation_replay_response": { + "name": "rotation_replay_response", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "rotation_replay_expires_at": { + "name": "rotation_replay_expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "confirmation": { + "name": "confirmation", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "oauth_refresh_tokens_token_unique": { + "name": "oauth_refresh_tokens_token_unique", + "columns": [ + "token" + ], + "isUnique": true + }, + "idx_oauth_refresh_tokens_token": { + "name": "idx_oauth_refresh_tokens_token", + "columns": [ + "token" + ], + "isUnique": false + }, + "idx_oauth_refresh_tokens_client_id": { + "name": "idx_oauth_refresh_tokens_client_id", + "columns": [ + "client_id" + ], + "isUnique": false + }, + "idx_oauth_refresh_tokens_user_id": { + "name": "idx_oauth_refresh_tokens_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "oauth_refresh_tokens_user_id_users_id_fk": { + "name": "oauth_refresh_tokens_user_id_users_id_fk", + "tableFrom": "oauth_refresh_tokens", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "oauth_resources": { + "name": "oauth_resources", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "identifier": { + "name": "identifier", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "access_token_ttl": { + "name": "access_token_ttl", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "refresh_token_ttl": { + "name": "refresh_token_ttl", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "signing_algorithm": { + "name": "signing_algorithm", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "signing_key_id": { + "name": "signing_key_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "allowed_scopes": { + "name": "allowed_scopes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "custom_claims": { + "name": "custom_claims", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "dpop_bound_access_tokens_required": { + "name": "dpop_bound_access_tokens_required", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "disabled": { + "name": "disabled", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "policy_version": { + "name": "policy_version", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "metadata": { + "name": "metadata", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "oauth_resources_identifier_unique": { + "name": "oauth_resources_identifier_unique", + "columns": [ + "identifier" + ], + "isUnique": true + }, + "idx_oauth_resources_identifier": { + "name": "idx_oauth_resources_identifier", + "columns": [ + "identifier" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "organizations": { + "name": "organizations", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "config_id": { + "name": "config_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "normalized_name": { + "name": "normalized_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "avatar_url": { + "name": "avatar_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "source_id": { + "name": "source_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "membership_role": { + "name": "membership_role", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'member'" + }, + "is_included": { + "name": "is_included", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "destination_org": { + "name": "destination_org", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "mirror_overrides": { + "name": "mirror_overrides", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'imported'" + }, + "last_mirrored": { + "name": "last_mirrored", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "error_message": { + "name": "error_message", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "repository_count": { + "name": "repository_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "public_repository_count": { + "name": "public_repository_count", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "private_repository_count": { + "name": "private_repository_count", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "fork_repository_count": { + "name": "fork_repository_count", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_organizations_user_id": { + "name": "idx_organizations_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + }, + "idx_organizations_config_id": { + "name": "idx_organizations_config_id", + "columns": [ + "config_id" + ], + "isUnique": false + }, + "idx_organizations_status": { + "name": "idx_organizations_status", + "columns": [ + "status" + ], + "isUnique": false + }, + "idx_organizations_is_included": { + "name": "idx_organizations_is_included", + "columns": [ + "is_included" + ], + "isUnique": false + }, + "uniq_organizations_user_normalized_name": { + "name": "uniq_organizations_user_normalized_name", + "columns": [ + "user_id", + "normalized_name" + ], + "isUnique": true + } + }, + "foreignKeys": { + "organizations_user_id_users_id_fk": { + "name": "organizations_user_id_users_id_fk", + "tableFrom": "organizations", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "organizations_config_id_configs_id_fk": { + "name": "organizations_config_id_configs_id_fk", + "tableFrom": "organizations", + "tableTo": "configs", + "columnsFrom": [ + "config_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "rate_limits": { + "name": "rate_limits", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'github'" + }, + "limit": { + "name": "limit", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "remaining": { + "name": "remaining", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "used": { + "name": "used", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "reset": { + "name": "reset", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "retry_after": { + "name": "retry_after", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'ok'" + }, + "last_checked": { + "name": "last_checked", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_rate_limits_user_provider": { + "name": "idx_rate_limits_user_provider", + "columns": [ + "user_id", + "provider" + ], + "isUnique": false + }, + "idx_rate_limits_status": { + "name": "idx_rate_limits_status", + "columns": [ + "status" + ], + "isUnique": false + } + }, + "foreignKeys": { + "rate_limits_user_id_users_id_fk": { + "name": "rate_limits_user_id_users_id_fk", + "tableFrom": "rate_limits", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "repositories": { + "name": "repositories", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "config_id": { + "name": "config_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "full_name": { + "name": "full_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "normalized_full_name": { + "name": "normalized_full_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "url": { + "name": "url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "clone_url": { + "name": "clone_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "source_provider": { + "name": "source_provider", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'github'" + }, + "source_url": { + "name": "source_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'https://github.com'" + }, + "source_id": { + "name": "source_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "destination_provider": { + "name": "destination_provider", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'gitea'" + }, + "destination_url": { + "name": "destination_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "owner": { + "name": "owner", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "organization": { + "name": "organization", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "mirrored_location": { + "name": "mirrored_location", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "''" + }, + "is_private": { + "name": "is_private", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "is_fork": { + "name": "is_fork", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "forked_from": { + "name": "forked_from", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "has_issues": { + "name": "has_issues", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "is_starred": { + "name": "is_starred", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "is_archived": { + "name": "is_archived", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "size": { + "name": "size", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "has_lfs": { + "name": "has_lfs", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "has_submodules": { + "name": "has_submodules", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "destination_size": { + "name": "destination_size", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "release_assets_size": { + "name": "release_assets_size", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "release_assets_count": { + "name": "release_assets_count", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "storage_checked_at": { + "name": "storage_checked_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "language": { + "name": "language", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "default_branch": { + "name": "default_branch", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "visibility": { + "name": "visibility", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'public'" + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'imported'" + }, + "last_mirrored": { + "name": "last_mirrored", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "error_message": { + "name": "error_message", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "destination_org": { + "name": "destination_org", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "mirror_overrides": { + "name": "mirror_overrides", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "metadata": { + "name": "metadata", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "imported_at": { + "name": "imported_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_repositories_user_id": { + "name": "idx_repositories_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + }, + "idx_repositories_config_id": { + "name": "idx_repositories_config_id", + "columns": [ + "config_id" + ], + "isUnique": false + }, + "idx_repositories_status": { + "name": "idx_repositories_status", + "columns": [ + "status" + ], + "isUnique": false + }, + "idx_repositories_owner": { + "name": "idx_repositories_owner", + "columns": [ + "owner" + ], + "isUnique": false + }, + "idx_repositories_organization": { + "name": "idx_repositories_organization", + "columns": [ + "organization" + ], + "isUnique": false + }, + "idx_repositories_is_fork": { + "name": "idx_repositories_is_fork", + "columns": [ + "is_fork" + ], + "isUnique": false + }, + "idx_repositories_is_starred": { + "name": "idx_repositories_is_starred", + "columns": [ + "is_starred" + ], + "isUnique": false + }, + "idx_repositories_user_imported_at": { + "name": "idx_repositories_user_imported_at", + "columns": [ + "user_id", + "imported_at" + ], + "isUnique": false + }, + "uniq_repositories_user_full_name": { + "name": "uniq_repositories_user_full_name", + "columns": [ + "user_id", + "source_id", + "full_name" + ], + "isUnique": true + }, + "uniq_repositories_user_normalized_full_name": { + "name": "uniq_repositories_user_normalized_full_name", + "columns": [ + "user_id", + "source_id", + "normalized_full_name" + ], + "isUnique": true + }, + "idx_repositories_mirrored_location": { + "name": "idx_repositories_mirrored_location", + "columns": [ + "user_id", + "mirrored_location" + ], + "isUnique": false + } + }, + "foreignKeys": { + "repositories_user_id_users_id_fk": { + "name": "repositories_user_id_users_id_fk", + "tableFrom": "repositories", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "repositories_config_id_configs_id_fk": { + "name": "repositories_config_id_configs_id_fk", + "tableFrom": "repositories", + "tableTo": "configs", + "columnsFrom": [ + "config_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "sessions": { + "name": "sessions", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "token": { + "name": "token", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ip_address": { + "name": "ip_address", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "user_agent": { + "name": "user_agent", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "sessions_token_unique": { + "name": "sessions_token_unique", + "columns": [ + "token" + ], + "isUnique": true + }, + "idx_sessions_user_id": { + "name": "idx_sessions_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + }, + "idx_sessions_token": { + "name": "idx_sessions_token", + "columns": [ + "token" + ], + "isUnique": false + }, + "idx_sessions_expires_at": { + "name": "idx_sessions_expires_at", + "columns": [ + "expires_at" + ], + "isUnique": false + } + }, + "foreignKeys": { + "sessions_user_id_users_id_fk": { + "name": "sessions_user_id_users_id_fk", + "tableFrom": "sessions", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "sources": { + "name": "sources", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "url": { + "name": "url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "username": { + "name": "username", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "token": { + "name": "token", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_sources_user_id": { + "name": "idx_sources_user_id", + "columns": [ + "user_id" + ], + "isUnique": false + }, + "uniq_sources_user_provider_url_username": { + "name": "uniq_sources_user_provider_url_username", + "columns": [ + "user_id", + "provider", + "url", + "username" + ], + "isUnique": true + } + }, + "foreignKeys": { + "sources_user_id_users_id_fk": { + "name": "sources_user_id_users_id_fk", + "tableFrom": "sources", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "sso_providers": { + "name": "sso_providers", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "issuer": { + "name": "issuer", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "domain": { + "name": "domain", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "oidc_config": { + "name": "oidc_config", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "saml_config": { + "name": "saml_config", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "domain_verified": { + "name": "domain_verified", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "provider_id": { + "name": "provider_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "sso_providers_provider_id_unique": { + "name": "sso_providers_provider_id_unique", + "columns": [ + "provider_id" + ], + "isUnique": true + }, + "idx_sso_providers_provider_id": { + "name": "idx_sso_providers_provider_id", + "columns": [ + "provider_id" + ], + "isUnique": false + }, + "idx_sso_providers_domain": { + "name": "idx_sso_providers_domain", + "columns": [ + "domain" + ], + "isUnique": false + }, + "idx_sso_providers_issuer": { + "name": "idx_sso_providers_issuer", + "columns": [ + "issuer" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "users": { + "name": "users", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "email": { + "name": "email", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "email_verified": { + "name": "email_verified", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "image": { + "name": "image", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "username": { + "name": "username", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "users_email_unique": { + "name": "users_email_unique", + "columns": [ + "email" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "verification_tokens": { + "name": "verification_tokens", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "token": { + "name": "token", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "identifier": { + "name": "identifier", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "verification_tokens_token_unique": { + "name": "verification_tokens_token_unique", + "columns": [ + "token" + ], + "isUnique": true + }, + "idx_verification_tokens_token": { + "name": "idx_verification_tokens_token", + "columns": [ + "token" + ], + "isUnique": false + }, + "idx_verification_tokens_identifier": { + "name": "idx_verification_tokens_identifier", + "columns": [ + "identifier" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "verifications": { + "name": "verifications", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "identifier": { + "name": "identifier", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "value": { + "name": "value", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_verifications_identifier": { + "name": "idx_verifications_identifier", + "columns": [ + "identifier" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + } + }, + "views": {}, + "enums": {}, + "_meta": { + "schemas": {}, + "tables": {}, + "columns": {} + }, + "internal": { + "indexes": {} + } +} \ No newline at end of file diff --git a/drizzle/meta/_journal.json b/drizzle/meta/_journal.json index 81e346f..d01799f 100644 --- a/drizzle/meta/_journal.json +++ b/drizzle/meta/_journal.json @@ -148,6 +148,13 @@ "when": 1788805724170, "tag": "0020_org_source", "breakpoints": true + }, + { + "idx": 21, + "version": "6", + "when": 1793244040274, + "tag": "0021_repository_storage", + "breakpoints": true } ] } \ No newline at end of file diff --git a/scripts/validate-migrations.ts b/scripts/validate-migrations.ts index 83e4a7e..35cdc47 100644 --- a/scripts/validate-migrations.ts +++ b/scripts/validate-migrations.ts @@ -1007,8 +1007,55 @@ const latestUpgradeFixtures: Record = { seed: seedPre0020Database, verify: verify0020Migration, }, + "0021_repository_storage": { + seed: seedPre0021Database, + verify: verify0021Migration, + }, }; +function seedPre0021Database(db: any) { + // Migrations 0000-0020 have run: repositories have no storage columns yet. + db.run("INSERT INTO users (id, email, username, name) VALUES ('u-st21', 'st21@example.com', 'st21', 'Storage Twenty-One')"); + db.run( + "INSERT INTO configs (id, user_id, name, is_active, github_config, gitea_config, schedule_config, cleanup_config) " + + "VALUES ('cfg-st21', 'u-st21', 'Default', 1, '{\"provider\":\"github\",\"owner\":\"octocat\",\"token\":\"gh-token\",\"type\":\"personal\"}', '{}', '{}', '{}')", + ); + db.run( + "INSERT INTO repositories (id, user_id, config_id, name, full_name, normalized_full_name, url, clone_url, owner, default_branch, status, mirrored_location) " + + "VALUES ('repo-st21', 'u-st21', 'cfg-st21', 'app', 'octocat/app', 'octocat/app', 'https://github.com/octocat/app', 'https://github.com/octocat/app.git', 'octocat', 'main', 'synced', 'mirror/app')", + ); +} + +function verify0021Migration(db: any) { + const columns = db.query("PRAGMA table_info(repositories)").all() as TableInfoRow[]; + for (const name of ["destination_size", "release_assets_size", "release_assets_count", "storage_checked_at"]) { + const column = columns.find((c) => c.name === name); + assert(column, `Expected repositories.${name} to exist after migration`); + assert(column.notnull === 0, `Expected repositories.${name} to be nullable`); + assert(column.dflt_value === null, `Expected repositories.${name} to have no default, got ${column.dflt_value}`); + } + + const existing = db + .query("SELECT destination_size, release_assets_size, release_assets_count, storage_checked_at FROM repositories WHERE id = 'repo-st21'") + .get() as Record | null; + assert(existing, "Expected the existing repository to survive the migration"); + assert( + Object.values(existing).every((value) => value === null), + `Expected an existing repository to start unmeasured, got ${JSON.stringify(existing)}`, + ); + + db.run( + "UPDATE repositories SET destination_size = 1048576, release_assets_size = 2048, release_assets_count = 3, storage_checked_at = 1793000000 WHERE id = 'repo-st21'", + ); + const measured = db + .query("SELECT destination_size, release_assets_size, release_assets_count FROM repositories WHERE id = 'repo-st21'") + .get() as { destination_size: number; release_assets_size: number; release_assets_count: number } | null; + assert( + measured?.destination_size === 1048576 && measured.release_assets_size === 2048 && measured.release_assets_count === 3, + `Expected the storage columns to round-trip, got ${JSON.stringify(measured)}`, + ); +} + function seedPre0020Database(db: any) { // Migrations 0000-0019 have run: repositories carry source_id, // organizations have none yet. Seed two sources and three organizations: diff --git a/src/components/layout/Header.tsx b/src/components/layout/Header.tsx index ee68033..3780ecc 100644 --- a/src/components/layout/Header.tsx +++ b/src/components/layout/Header.tsx @@ -10,7 +10,7 @@ import { AccountMenu } from "@/components/auth/AccountMenu"; import { withBase } from "@/lib/base-path"; interface HeaderProps { - currentPage?: "dashboard" | "repositories" | "organizations" | "configuration" | "activity-log"; + currentPage?: "dashboard" | "repositories" | "organizations" | "storage" | "configuration" | "activity-log"; onNavigate?: (page: string) => void; onMenuClick: () => void; onToggleCollapse?: () => void; diff --git a/src/components/layout/MainLayout.tsx b/src/components/layout/MainLayout.tsx index 2cead78..d919550 100644 --- a/src/components/layout/MainLayout.tsx +++ b/src/components/layout/MainLayout.tsx @@ -7,6 +7,7 @@ import Providers from "./Providers"; import { ConfigTabs } from "../config/ConfigTabs"; import { ActivityLog } from "../activity/ActivityLog"; import { Organization } from "../organizations/Organization"; +import { Storage } from "../storage/Storage"; import { Toaster } from "@/components/ui/sonner"; import { useAuth } from "@/hooks/useAuth"; import { useRepoSync } from "@/hooks/useSyncRepo"; @@ -23,6 +24,7 @@ interface AppProps { | "dashboard" | "repositories" | "organizations" + | "storage" | "configuration" | "activity-log"; "client:load"?: boolean; @@ -77,6 +79,7 @@ function AppWithProviders({ page: initialPage }: AppProps) { '/': 'dashboard', '/repositories': 'repositories', '/organizations': 'organizations', + '/storage': 'storage', '/config': 'configuration', '/activity': 'activity-log' }; @@ -156,6 +159,7 @@ function AppWithProviders({ page: initialPage }: AppProps) { {currentPage === "dashboard" && } {currentPage === "repositories" && } {currentPage === "organizations" && } + {currentPage === "storage" && } {currentPage === "configuration" && } {currentPage === "activity-log" && } diff --git a/src/components/layout/Sidebar.tsx b/src/components/layout/Sidebar.tsx index 5996190..6fdbb61 100644 --- a/src/components/layout/Sidebar.tsx +++ b/src/components/layout/Sidebar.tsx @@ -55,6 +55,7 @@ export function Sidebar({ className, onNavigate, isOpen, isCollapsed = false, on '/': 'dashboard', '/repositories': 'repositories', '/organizations': 'organizations', + '/storage': 'storage', '/config': 'configuration', '/activity': 'activity-log' }; diff --git a/src/components/storage/Storage.tsx b/src/components/storage/Storage.tsx new file mode 100644 index 0000000..df44fc5 --- /dev/null +++ b/src/components/storage/Storage.tsx @@ -0,0 +1,456 @@ +import { useCallback, useEffect, useMemo, useState } from "react"; +import { + ArrowDown, + ArrowUp, + Database, + HardDrive, + Info, + Package, + ScanSearch, + Search, +} from "lucide-react"; +import { toast } from "sonner"; +import { StatusCard } from "@/components/dashboard/StatusCard"; +import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; +import { Skeleton } from "@/components/ui/skeleton"; +import { + Select, + SelectContent, + SelectItem, + SelectTrigger, + SelectValue, +} from "@/components/ui/select"; +import { + Tooltip, + TooltipContent, + TooltipProvider, + TooltipTrigger, +} from "@/components/ui/tooltip"; +import { useAuth } from "@/hooks/useAuth"; +import { useGiteaConfig } from "@/hooks/useGiteaConfig"; +import { useLiveRefresh } from "@/hooks/useLiveRefresh"; +import { useNavigation } from "@/components/layout/MainLayout"; +import { apiRequest, cn, formatLastSyncTime } from "@/lib/utils"; +import { buildGiteaWebUrl } from "@/lib/gitea-url"; +import { formatBytes } from "@/lib/utils/format-bytes"; +import type { StorageOverview, StorageRepository } from "@/lib/repository-storage"; + +type StorageApiResponse = ({ success: true } & StorageOverview) | { success: false; error?: string }; + +type SortKey = "fullName" | "owner" | "destinationSize" | "releaseAssetsSize" | "releaseAssetsCount" | "total" | "checkedAt"; + +const BAR_ROWS = 8; +// Repositories where release assets are at least half the space, and enough +// of it to matter, get a hint about the release limit. +const RELEASE_HEAVY_SHARE = 0.5; +const RELEASE_HEAVY_MIN_BYTES = 50 * 1024 * 1024; + +function percent(part: number, whole: number): number { + return whole > 0 ? Math.round((part / whole) * 100) : 0; +} + +function isReleaseHeavy(repo: StorageRepository): boolean { + const assets = repo.releaseAssetsSize ?? 0; + return assets >= RELEASE_HEAVY_MIN_BYTES && assets >= repo.total * RELEASE_HEAVY_SHARE; +} + +/** Git + LFS and release assets as one bar, scaled against the panel's largest row. */ +function SplitBar({ git, assets, max }: { git: number; assets: number; max: number }) { + const width = (value: number) => `${max > 0 ? (value / max) * 100 : 0}%`; + return ( +
+
+
+
+ ); +} + +function Legend() { + return ( +
+ + + Git + LFS + + + + Release assets + +
+ ); +} + +interface BarRow { + key: string; + label: string; + detail?: string; + git: number; + assets: number; + total: number; +} + +function BarPanel({ title, icon, rows }: { title: string; icon: React.ReactNode; rows: BarRow[] }) { + const max = rows.reduce((largest, row) => Math.max(largest, row.total), 0); + return ( + + + + {icon} + {title} + + + + + {rows.map((row) => ( +
+
+ + {row.label} + {row.detail && ( + {row.detail} + )} + + {formatBytes(row.total)} +
+ +
+ ))} + + + ); +} + +function SortHeader({ + label, + sortKey, + sort, + onSort, + className, +}: { + label: string; + sortKey: SortKey; + sort: { key: SortKey; descending: boolean }; + onSort: (key: SortKey) => void; + className?: string; +}) { + const active = sort.key === sortKey; + const Arrow = sort.descending ? ArrowDown : ArrowUp; + return ( + + ); +} + +function compareRepositories(a: StorageRepository, b: StorageRepository, key: SortKey): number { + switch (key) { + case "fullName": + return a.fullName.localeCompare(b.fullName); + case "owner": + return a.owner.localeCompare(b.owner) || a.fullName.localeCompare(b.fullName); + case "checkedAt": + return (a.checkedAt ?? "").localeCompare(b.checkedAt ?? ""); + default: + return (a[key] ?? -1) - (b[key] ?? -1); + } +} + +function StorageSkeleton() { + return ( +
+
+ {Array.from({ length: 4 }).map((_, i) => ( + + ))} +
+
+ + +
+ +
+ ); +} + +/** + * Storage the mirrors take up on the destination (#479). Every number was + * recorded by a sync from data it fetched anyway, so this page only reads the + * database. + */ +export function Storage() { + const { user } = useAuth(); + const { registerRefreshCallback } = useLiveRefresh(); + const { navigationKey } = useNavigation(); + const { giteaConfig } = useGiteaConfig(); + + const [overview, setOverview] = useState(null); + const [isLoading, setIsLoading] = useState(true); + const [search, setSearch] = useState(""); + const [ownerFilter, setOwnerFilter] = useState("all"); + const [sort, setSort] = useState<{ key: SortKey; descending: boolean }>({ key: "total", descending: true }); + + const fetchOverview = useCallback( + async (isLiveRefresh = false) => { + if (!user?.id) return; + try { + const response = await apiRequest("/storage", { method: "GET" }); + if (response.success) { + setOverview(response); + } else if (!isLiveRefresh) { + toast.error(response.error || "Error fetching storage"); + } + } catch (error) { + if (!isLiveRefresh) { + toast.error(error instanceof Error ? error.message : "Error fetching storage"); + } + } finally { + if (!isLiveRefresh) setIsLoading(false); + } + }, + [user?.id] + ); + + useEffect(() => { + setIsLoading(true); + fetchOverview(false); + }, [fetchOverview, navigationKey]); + + useEffect(() => registerRefreshCallback(() => fetchOverview(true)), [registerRefreshCallback, fetchOverview]); + + const handleSort = (key: SortKey) => + setSort((prev) => + prev.key === key + ? { key, descending: !prev.descending } + : { key, descending: key !== "fullName" && key !== "owner" } + ); + + const visibleRepositories = useMemo(() => { + if (!overview) return []; + const term = search.trim().toLowerCase(); + const rows = overview.repositories.filter( + (repo) => + (ownerFilter === "all" || repo.owner === ownerFilter) && + (!term || repo.fullName.toLowerCase().includes(term) || repo.location.toLowerCase().includes(term)) + ); + rows.sort((a, b) => { + const order = compareRepositories(a, b, sort.key); + return sort.descending ? -order : order; + }); + return rows; + }, [overview, search, ownerFilter, sort]); + + if (isLoading || !overview) return ; + + const { totals } = overview; + + if (totals.measured === 0) { + return ( + + + +

No sizes recorded yet

+

+ Each sync records how much space the mirror takes on the destination, from data the sync + already fetches. Sizes show up here after the next sync. +

+ + + ); + } + + const ownerRows: BarRow[] = overview.owners.slice(0, BAR_ROWS).map((owner) => ({ + key: owner.owner || "(unknown)", + label: owner.owner || "Unknown owner", + detail: `${owner.repositories} ${owner.repositories === 1 ? "repo" : "repos"}`, + git: owner.destinationSize, + assets: owner.releaseAssetsSize, + total: owner.total, + })); + const largestRows: BarRow[] = overview.repositories.slice(0, BAR_ROWS).map((repo) => ({ + key: repo.id, + label: repo.location || repo.fullName, + git: repo.destinationSize ?? 0, + assets: repo.releaseAssetsSize ?? 0, + total: repo.total, + })); + const unmeasured = Math.max(0, totals.mirrored - totals.measured); + + return ( + +
+
+ } + description={`Across ${totals.measured} ${totals.measured === 1 ? "repository" : "repositories"}`} + /> + } + description="As the destination reports it" + /> + } + description={`${percent(totals.releaseAssetsSize, totals.total)}% of the total, ${totals.releaseAssetsCount.toLocaleString()} files`} + /> + } + description={unmeasured > 0 ? "The rest are measured on their next sync" : "Every mirror is measured"} + /> +
+ +
+
+ } rows={ownerRows} /> +
+
+ } rows={largestRows} /> +
+
+ +
+
+
+ + setSearch(event.target.value)} + /> +
+ +
+ +
+
+
+
+ +
+
+ +
+
+ +
+
+ +
+
+ +
+
+ +
+
+ +
+
+ + {visibleRepositories.length === 0 && ( +
No repositories match.
+ )} + + {visibleRepositories.map((repo) => { + const destinationUrl = repo.location ? buildGiteaWebUrl(giteaConfig, repo.location) : null; + const heavy = isReleaseHeavy(repo); + return ( +
+
+
+ {destinationUrl ? ( + + {repo.location} + + ) : ( + {repo.location || repo.fullName} + )} + {repo.location && repo.location !== repo.fullName && ( + from {repo.fullName} + )} +
+
+
+ {repo.owner || "Unknown"} +
+
+ {repo.destinationSize === null ? - : formatBytes(repo.destinationSize)} +
+
+ {heavy && ( + + + + + + Release assets are {percent(repo.releaseAssetsSize ?? 0, repo.total)}% of this repository. A lower + release limit in its Mirror options removes the oldest releases on the next sync. + + + )} + {repo.releaseAssetsSize === null ? - : formatBytes(repo.releaseAssetsSize)} +
+
+ {repo.releaseAssetsCount === null ? "-" : repo.releaseAssetsCount.toLocaleString()} +
+
+ {formatBytes(repo.total)} +
+
+ {formatLastSyncTime(repo.checkedAt)} +
+
+ ); + })} +
+
+

+ Sizes are what the destination reports: Git and LFS together, plus the release assets it stores. Each sync + records them from data it already fetches. The Git size is read before the sync pulls, so it trails by one + sync. +

+
+
+ + ); +} diff --git a/src/data/Sidebar.ts b/src/data/Sidebar.ts index 2e1bee4..89b1d94 100644 --- a/src/data/Sidebar.ts +++ b/src/data/Sidebar.ts @@ -4,6 +4,7 @@ import { Settings, Activity, Building2, + HardDrive, } from "lucide-react"; import type { SidebarItem } from "@/types/Sidebar"; @@ -11,6 +12,7 @@ export const links: SidebarItem[] = [ { href: "/", label: "Dashboard", icon: LayoutDashboard }, { href: "/repositories", label: "Repositories", icon: GitFork }, { href: "/organizations", label: "Organizations", icon: Building2 }, + { href: "/storage", label: "Storage", icon: HardDrive }, { href: "/config", label: "Configuration", icon: Settings }, { href: "/activity", label: "Activity Log", icon: Activity }, ]; diff --git a/src/lib/db/schema.ts b/src/lib/db/schema.ts index b26e282..693c739 100644 --- a/src/lib/db/schema.ts +++ b/src/lib/db/schema.ts @@ -322,6 +322,10 @@ export const repositorySchema = z.object({ size: z.number().default(0), hasLFS: z.boolean().default(false), hasSubmodules: z.boolean().default(false), + destinationSize: z.number().optional().nullable(), + releaseAssetsSize: z.number().optional().nullable(), + releaseAssetsCount: z.number().optional().nullable(), + storageCheckedAt: z.coerce.date().optional().nullable(), language: z.string().optional().nullable(), description: z.string().optional().nullable(), defaultBranch: z.string(), @@ -623,6 +627,15 @@ export const repositories = sqliteTable("repositories", { .notNull() .default(false), + // What the mirror takes up on the destination, in bytes, recorded from + // responses a sync already fetches (#479). NULL until a sync has seen it. + // destinationSize is Git plus LFS as the destination reports it; the release + // asset totals come from the listing the release retention pass reads. + destinationSize: integer("destination_size"), + releaseAssetsSize: integer("release_assets_size"), + releaseAssetsCount: integer("release_assets_count"), + storageCheckedAt: integer("storage_checked_at", { mode: "timestamp" }), + language: text("language"), description: text("description"), defaultBranch: text("default_branch").notNull(), diff --git a/src/lib/gitea-enhanced.test.ts b/src/lib/gitea-enhanced.test.ts index 381b5da..dc22e60 100644 --- a/src/lib/gitea-enhanced.test.ts +++ b/src/lib/gitea-enhanced.test.ts @@ -998,6 +998,67 @@ describe("Enhanced Gitea Operations", () => { expect(String(mirrorSyncCalls[0][0])).not.toContain("/api/v1/repos/ceph/test-repo/mirror-sync"); }); + describe("destination size (#479)", () => { + const sizeConfig = { + userId: "user123", + githubConfig: { username: "testuser", token: "github-token", privateRepositories: false, mirrorStarred: true }, + giteaConfig: { url: "https://gitea.example.com", token: "encrypted-token", defaultOwner: "testuser", mirrorReleases: false }, + } as unknown as Partial; + const sizeRepository = (): Repository => ({ + id: "repo-size", name: "test-repo", fullName: "ceph/test-repo", owner: "ceph", + cloneUrl: "https://github.com/ceph/test-repo.git", isPrivate: false, isStarred: true, + status: repoStatusEnum.parse("mirrored"), visibility: "public", userId: "user123", + mirroredLocation: "starred/test-repo", createdAt: new Date(), updatedAt: new Date(), + } as Repository); + const sizeDependencies = { + getGiteaRepoOwnerAsync: mockGetGiteaRepoOwnerAsync, + mirrorGitHubReleasesToGitea: mockMirrorGitHubReleasesToGitea, + mirrorGitRepoIssuesToGitea: mockMirrorGitRepoIssuesToGitea, + mirrorGitRepoPullRequestsToGitea: mockMirrorGitRepoPullRequestsToGitea, + mirrorGitRepoLabelsToGitea: mockMirrorGitRepoLabelsToGitea, + mirrorGitRepoMilestonesToGitea: mockMirrorGitRepoMilestonesToGitea, + syncRepositoryMetadataToGitea: mockSyncRepositoryMetadataToGitea, + }; + const repoUrl = "https://gitea.example.com/api/v1/repos/starred/test-repo"; + + test("records the size from the lookup the sync already makes", async () => { + const defaultGet = mockHttpGet.getMockImplementation()!; + mockHttpGet.mockImplementation(async (url: string, headers?: any) => { + const response: any = await defaultGet(url, headers); + return url === repoUrl ? { ...response, data: { ...response.data, size: 2048 } } : response; + }); + + try { + const result = await syncGiteaRepoEnhanced( + { config: sizeConfig, repository: sizeRepository() }, + sizeDependencies + ); + expect(result).toEqual({ success: true }); + + const successUpdate = dbUpdateSetCalls.find((data) => "lastMirrored" in data); + expect(successUpdate.destinationSize).toBe(2048 * 1024); + expect(successUpdate.storageCheckedAt).toBeInstanceOf(Date); + // Read from the target lookup, not fetched again. + expect(mockHttpGet.mock.calls.filter((call) => call[0] === repoUrl)).toHaveLength(1); + } finally { + mockHttpGet.mockImplementation(defaultGet); + } + }); + + test("leaves the stored size alone when the destination reports none", async () => { + const result = await syncGiteaRepoEnhanced( + { config: sizeConfig, repository: sizeRepository() }, + sizeDependencies + ); + expect(result).toEqual({ success: true }); + + const successUpdate = dbUpdateSetCalls.find((data) => "lastMirrored" in data); + expect(successUpdate).toBeDefined(); + expect("destinationSize" in successUpdate).toBe(false); + expect("storageCheckedAt" in successUpdate).toBe(false); + }); + }); + describe("GitHub App sources", () => { const appConfig = { userId: "user123", diff --git a/src/lib/gitea-enhanced.ts b/src/lib/gitea-enhanced.ts index cb95515..e206954 100644 --- a/src/lib/gitea-enhanced.ts +++ b/src/lib/gitea-enhanced.ts @@ -44,6 +44,7 @@ import { strategyNeedsDetection, } from "./repo-backup"; import { detectForcePush } from "./utils/force-push-detection"; +import { destinationSizeColumns } from "./repository-storage"; import { sanitizeRepoNameAlphaDashDot, type GiteaMilestoneIds } from "./gitea"; import { parseRepositoryMetadataState, @@ -86,6 +87,8 @@ export interface GiteaRepoInfo { // skip writes that would not change anything. description?: string; topics?: string[]; + // KiB, Git plus LFS, recorded for the storage page (#479). + size?: number; // Set while a transfer waits for the new owner to accept it (issue #400). repo_transfer?: { recipient?: { login?: string; username?: string } | null } | null; } @@ -1292,6 +1295,9 @@ export async function syncGiteaRepoEnhanced({ metadata: metadataUpdated ? serializeRepositoryMetadataState(metadataState) : repository.metadata ?? null, + // From the GET that resolved the target, so before this sync's + // fetch: the size lags one sync and costs no request. + ...destinationSizeColumns(repoInfo), }) .where(eq(repositories.id, repository.id!)); diff --git a/src/lib/gitea-source-releases.test.ts b/src/lib/gitea-source-releases.test.ts index ae21a77..3a73e33 100644 --- a/src/lib/gitea-source-releases.test.ts +++ b/src/lib/gitea-source-releases.test.ts @@ -64,7 +64,22 @@ interface ActivityJob { } let activityJobs: ActivityJob[] = []; +// The release asset totals the retention pass records for the storage page. +interface StorageRecord { + repositoryId: string | null | undefined; + summary: { count: number; bytes: number }; +} +let storageRecords: StorageRecord[] = []; + if (isChild) { + mock.module("@/lib/repository-storage-store", () => ({ + recordReleaseAssetStorage: async ( + repositoryId: string | null | undefined, + summary: { count: number; bytes: number } + ) => { + storageRecords.push({ repositoryId, summary }); + }, + })); mock.module("@/lib/notification-service", () => ({ triggerAllReleasesRemovedNotification: async (notice: RemovalNotice) => { removalNotices.push(notice); @@ -94,6 +109,7 @@ afterEach(() => { globalThis.fetch = realFetch; removalNotices = []; activityJobs = []; + storageRecords = []; }); const config = { @@ -132,6 +148,10 @@ interface DestinationOptions { destinationTags?: string[]; /** Releases already on the destination, as the retention pass lists them. */ destinationReleases?: Array>; + /** The destination answers the retention listing with an error. */ + destinationListingFails?: boolean; + /** Destination release ids whose DELETE fails. */ + failDeleteIds?: number[]; } function fakeWorld(options: DestinationOptions) { @@ -205,12 +225,17 @@ function fakeWorld(options: DestinationOptions) { // The retention pass removing a release. if (method === "DELETE" && /\/releases\/\d+$/.test(url)) { - return new Response(null, { status: 204 }); + const id = Number(url.slice(url.lastIndexOf("/") + 1)); + return options.failDeleteIds?.includes(id) + ? new Response("boom", { status: 500 }) + : new Response(null, { status: 204 }); } // The retention pass lists everything the destination has. if (method === "GET" && url.startsWith(`${DEST_API}/releases?`)) { - return json(options.destinationReleases ?? []); + return options.destinationListingFails + ? new Response("boom", { status: 500 }) + : json(options.destinationReleases ?? []); } throw new Error(`unexpected request: ${method} ${url}`); @@ -507,6 +532,66 @@ describe.skipIf(!isChild)("release type filter", () => { }); }); +describe.skipIf(!isChild)("release asset storage (#479)", () => { + const withAssets = (release: Record, sizes: number[]) => ({ + ...release, + assets: sizes.map((size, index) => ({ id: index + 1, name: `file-${index}`, size })), + }); + + it("records the assets left on the destination after retention", async () => { + fakeWorld({ + sourceReleases: [giteaRelease("v2.0.0")], + destinationReleases: [ + withAssets(destinationRelease("v1.9.0-beta", 11, true), [100, 50]), + withAssets(destinationRelease("v1.8.0", 12, false), [1000]), + ], + }); + + await mirrorFromGitea(giteaConnection("source-token"), { releaseType: "stable" }); + + expect(storageRecords).toEqual([{ repositoryId: "repo-1", summary: { count: 1, bytes: 1000 } }]); + }); + + it("keeps counting a release whose delete failed", async () => { + fakeWorld({ + sourceReleases: [giteaRelease("v2.0.0")], + destinationReleases: [ + withAssets(destinationRelease("v1.9.0-beta", 11, true), [100, 50]), + withAssets(destinationRelease("v1.8.0", 12, false), [1000]), + ], + failDeleteIds: [11], + }); + + await mirrorFromGitea(giteaConnection("source-token"), { releaseType: "stable" }); + + expect(storageRecords).toEqual([{ repositoryId: "repo-1", summary: { count: 3, bytes: 1150 } }]); + }); + + it("records nothing when the listing fails", async () => { + fakeWorld({ + sourceReleases: [giteaRelease("v2.0.0")], + destinationListingFails: true, + }); + + await mirrorFromGitea(giteaConnection("source-token")); + + expect(storageRecords).toEqual([]); + }); + + it("makes no request of its own", async () => { + const world = fakeWorld({ + sourceReleases: [giteaRelease("v2.0.0")], + destinationReleases: [withAssets(destinationRelease("v2.0.0", 12, false), [10])], + }); + + await mirrorFromGitea(giteaConnection("source-token")); + + const listings = world.requests.filter((r) => r.url.startsWith(`${DEST_API}/releases?`)); + expect(listings).toHaveLength(1); + expect(storageRecords).toEqual([{ repositoryId: "repo-1", summary: { count: 1, bytes: 10 } }]); + }); +}); + describe.skipIf(!isChild)("all releases removed warning", () => { it("warns when the filters leave the destination with no releases", async () => { const world = fakeWorld({ diff --git a/src/lib/gitea.ts b/src/lib/gitea.ts index f4cf157..bac01e8 100644 --- a/src/lib/gitea.ts +++ b/src/lib/gitea.ts @@ -35,6 +35,8 @@ import { triggerAllReleasesRemovedNotification } from "./notification-service"; import { isRateLimitError } from "./rate-limit-gate"; import { assertMirrorCredentialUpdateSupported } from "./gitea-mirror-credentials"; import { persistMetadataPassProgress } from "./metadata-progress-store"; +import { summarizeReleaseAssets } from "./repository-storage"; +import { recordReleaseAssetStorage } from "./repository-storage-store"; import { advanceMetadataSyncCursor, buildPassProgress, @@ -4380,7 +4382,12 @@ async function mirrorSourceReleasesToGiteaLocked({ tag_name: string; created_at: string; prerelease?: boolean | null; + assets?: Array<{ size?: number }> | null; }> = []; + // A page that fails ends the listing early. Retention still runs on what + // was listed, as before, but the storage totals are only recorded from a + // complete listing. + let listingComplete = true; let cleanupPage = 1; while (true) { const pageResponse = await httpGet( @@ -4390,7 +4397,11 @@ async function mirrorSourceReleasesToGiteaLocked({ } ).catch(() => null); - if (!pageResponse?.data || !Array.isArray(pageResponse.data) || pageResponse.data.length === 0) { + if (!pageResponse?.data || !Array.isArray(pageResponse.data)) { + listingComplete = false; + break; + } + if (pageResponse.data.length === 0) { break; } @@ -4404,6 +4415,7 @@ async function mirrorSourceReleasesToGiteaLocked({ const giteaBaseUrl = config.giteaConfig.url; const giteaToken = decryptedConfig.giteaConfig.token; + const deletedReleaseIds = new Set(); const deleteGiteaRelease = async ( release: { id: number; tag_name: string }, @@ -4416,6 +4428,7 @@ async function mirrorSourceReleasesToGiteaLocked({ Authorization: `token ${giteaToken}`, } ); + deletedReleaseIds.add(release.id); console.log(`[Releases] Deleted ${reason} release: ${release.tag_name}`); } catch (deleteError) { console.error( @@ -4520,6 +4533,18 @@ async function mirrorSourceReleasesToGiteaLocked({ await deleteGiteaRelease(excess, "excess"); } } + + // The listing already carries every asset's size, so the storage page + // gets its release numbers without another request (#479). Releases whose + // delete failed are still on the destination and still count. + if (listingComplete) { + await recordReleaseAssetStorage( + repository.id, + summarizeReleaseAssets( + allGiteaReleases.filter((rel) => !deletedReleaseIds.has(rel.id)) + ) + ); + } } catch (cleanupError) { console.warn( `[Releases] Release retention cleanup failed: ${ diff --git a/src/lib/repository-storage-store.ts b/src/lib/repository-storage-store.ts new file mode 100644 index 0000000..f7b30d1 --- /dev/null +++ b/src/lib/repository-storage-store.ts @@ -0,0 +1,33 @@ +/** + * Write the release asset totals the retention pass measured (#479). The + * release pass has no write of its own at the end, so it records them here. + */ + +import { eq } from "drizzle-orm"; +import { db, repositories } from "./db"; +import type { ReleaseAssetSummary } from "./repository-storage"; + +/** Never throws: a failed write must not fail the release pass. */ +export async function recordReleaseAssetStorage( + repositoryId: string | null | undefined, + summary: ReleaseAssetSummary +): Promise { + if (!repositoryId) return; + + try { + await db + .update(repositories) + .set({ + releaseAssetsSize: summary.bytes, + releaseAssetsCount: summary.count, + storageCheckedAt: new Date(), + }) + .where(eq(repositories.id, repositoryId)); + } catch (error) { + console.warn( + `[Storage] Could not record release asset sizes for repository ${repositoryId}: ${ + error instanceof Error ? error.message : String(error) + }` + ); + } +} diff --git a/src/lib/repository-storage.test.ts b/src/lib/repository-storage.test.ts new file mode 100644 index 0000000..ae292c0 --- /dev/null +++ b/src/lib/repository-storage.test.ts @@ -0,0 +1,126 @@ +import { describe, expect, test } from "bun:test"; +import { + buildStorageOverview, + destinationSizeColumns, + summarizeReleaseAssets, + type StorageSourceRow, +} from "./repository-storage"; + +describe("destinationSizeColumns", () => { + test("turns the KiB Gitea reports into bytes", () => { + const now = new Date("2026-10-29T00:00:00Z"); + expect(destinationSizeColumns({ size: 1536 }, now)).toEqual({ + destinationSize: 1536 * 1024, + storageCheckedAt: now, + }); + }); + + test("records an empty repository as zero", () => { + expect(destinationSizeColumns({ size: 0 }).destinationSize).toBe(0); + }); + + test("writes nothing without a usable size", () => { + expect(destinationSizeColumns({})).toEqual({}); + expect(destinationSizeColumns(null)).toEqual({}); + expect(destinationSizeColumns({ size: "12" })).toEqual({}); + expect(destinationSizeColumns({ size: -1 })).toEqual({}); + expect(destinationSizeColumns({ size: Number.NaN })).toEqual({}); + }); +}); + +describe("summarizeReleaseAssets", () => { + test("counts every asset and adds up their sizes", () => { + expect( + summarizeReleaseAssets([ + { assets: [{ size: 100 }, { size: 50 }] }, + { assets: [] }, + { assets: null }, + {}, + { assets: [{ size: 1000 }] }, + ]) + ).toEqual({ count: 3, bytes: 1150 }); + }); + + test("counts an asset without a size but adds nothing for it", () => { + expect(summarizeReleaseAssets([{ assets: [{}, { size: "9" }, { size: 4 }] }])).toEqual({ + count: 3, + bytes: 4, + }); + }); +}); + +function row(overrides: Partial): StorageSourceRow { + return { + id: "r", + name: "repo", + fullName: "acme/repo", + mirroredLocation: "archive/repo", + status: "synced", + destinationProvider: "gitea", + destinationSize: null, + releaseAssetsSize: null, + releaseAssetsCount: null, + storageCheckedAt: null, + ...overrides, + }; +} + +describe("buildStorageOverview", () => { + const checkedAt = new Date("2026-10-29T08:00:00Z"); + const rows = [ + row({ id: "a", fullName: "acme/app", mirroredLocation: "archive/app", destinationSize: 300, releaseAssetsSize: 700, releaseAssetsCount: 4, storageCheckedAt: checkedAt }), + row({ id: "b", fullName: "acme/lib", mirroredLocation: "archive/lib", destinationSize: 200, storageCheckedAt: checkedAt }), + row({ id: "c", fullName: "me/dots", mirroredLocation: "me/dots", destinationSize: 50 }), + // Mirrored but not synced since the upgrade. + row({ id: "d", fullName: "acme/new", status: "mirrored" }), + // Imported only: neither measured nor mirrored. + row({ id: "e", fullName: "acme/later", status: "imported", mirroredLocation: "" }), + // A failed first mirror has nothing on the destination yet. + row({ id: "f", fullName: "acme/broken", status: "failed" }), + // Push destinations have no size API; they are left out entirely. + row({ id: "g", fullName: "acme/pushed", destinationProvider: "github", status: "synced", destinationSize: 999 }), + ]; + + test("adds up the measured repositories", () => { + expect(buildStorageOverview(rows).totals).toEqual({ + total: 1250, + destinationSize: 550, + releaseAssetsSize: 700, + releaseAssetsCount: 4, + measured: 3, + mirrored: 4, + }); + }); + + test("lists measured repositories largest first", () => { + const overview = buildStorageOverview(rows); + expect(overview.repositories.map((repo) => [repo.id, repo.total])).toEqual([ + ["a", 1000], + ["b", 200], + ["c", 50], + ]); + expect(overview.repositories[0]).toMatchObject({ + fullName: "acme/app", + location: "archive/app", + owner: "archive", + releaseAssetsCount: 4, + checkedAt: checkedAt.toISOString(), + }); + // An unmeasured release side stays null so the page can tell it from zero. + expect(overview.repositories[1].releaseAssetsSize).toBeNull(); + }); + + test("groups by destination owner", () => { + expect(buildStorageOverview(rows).owners).toEqual([ + { owner: "archive", destinationSize: 500, releaseAssetsSize: 700, total: 1200, repositories: 2 }, + { owner: "me", destinationSize: 50, releaseAssetsSize: 0, total: 50, repositories: 1 }, + ]); + }); + + test("is empty before anything was measured", () => { + const overview = buildStorageOverview([row({ status: "mirrored" })]); + expect(overview.repositories).toEqual([]); + expect(overview.owners).toEqual([]); + expect(overview.totals).toMatchObject({ total: 0, measured: 0, mirrored: 1 }); + }); +}); diff --git a/src/lib/repository-storage.ts b/src/lib/repository-storage.ts new file mode 100644 index 0000000..bb87c8d --- /dev/null +++ b/src/lib/repository-storage.ts @@ -0,0 +1,165 @@ +/** + * Storage a mirror takes up on its destination (#479). Measured only from + * responses a sync already fetches: the repository GET before `mirror-sync` + * carries `size`, and the release listing the retention pass reads carries + * every asset's size. Nothing here makes a request. + */ + +import { isPushDestinationKind } from "./destination-kinds"; + +/** + * Columns to record from a destination repository response. Gitea and Forgejo + * report `size` in KiB, Git and LFS together. Empty when the response has no + * usable size, so a missing field never overwrites a stored number. + */ +export function destinationSizeColumns( + info: { size?: unknown } | null | undefined, + now: Date = new Date() +): { destinationSize?: number; storageCheckedAt?: Date } { + const size = info?.size; + if (typeof size !== "number" || !Number.isFinite(size) || size < 0) return {}; + return { destinationSize: Math.round(size * 1024), storageCheckedAt: now }; +} + +export interface ReleaseAssetSummary { + count: number; + bytes: number; +} + +/** Count and total size of the assets across a destination release listing. */ +export function summarizeReleaseAssets( + releases: ReadonlyArray<{ assets?: ReadonlyArray<{ size?: unknown }> | null }> +): ReleaseAssetSummary { + let count = 0; + let bytes = 0; + for (const release of releases) { + for (const asset of release.assets ?? []) { + count += 1; + if (typeof asset.size === "number" && Number.isFinite(asset.size) && asset.size > 0) { + bytes += asset.size; + } + } + } + return { count, bytes }; +} + +/** Repository columns the storage overview reads. */ +export interface StorageSourceRow { + id: string; + name: string; + fullName: string; + mirroredLocation: string | null; + status: string; + destinationProvider: string; + destinationSize: number | null; + releaseAssetsSize: number | null; + releaseAssetsCount: number | null; + storageCheckedAt: Date | null; +} + +export interface StorageRepository { + id: string; + /** Source full name, as the Repositories page shows it. */ + fullName: string; + /** owner/name on the destination; empty when not recorded. */ + location: string; + /** Destination owner, the first half of `location`. */ + owner: string; + destinationSize: number | null; + releaseAssetsSize: number | null; + releaseAssetsCount: number | null; + total: number; + checkedAt: string | null; +} + +export interface StorageOwner { + owner: string; + destinationSize: number; + releaseAssetsSize: number; + total: number; + repositories: number; +} + +export interface StorageOverview { + totals: { + total: number; + destinationSize: number; + releaseAssetsSize: number; + releaseAssetsCount: number; + /** Repositories with at least one recorded number. */ + measured: number; + /** Repositories with a mirror on a Gitea or Forgejo destination. */ + mirrored: number; + }; + owners: StorageOwner[]; + repositories: StorageRepository[]; +} + +// Statuses that mean a mirror exists on the destination. A failed row may be a +// failed first mirror, so it only counts once it has been measured. +const MIRROR_EXISTS_STATUSES = new Set(["mirrored", "synced", "syncing", "archived", "pending-approval"]); + +function isMeasured(row: StorageSourceRow): boolean { + return row.destinationSize !== null || row.releaseAssetsSize !== null; +} + +/** The storage page's numbers: totals, per destination owner, and per repository, largest first. */ +export function buildStorageOverview(rows: ReadonlyArray): StorageOverview { + const repositories: StorageRepository[] = []; + const owners = new Map(); + let mirrored = 0; + + for (const row of rows) { + if (isPushDestinationKind(row.destinationProvider)) continue; + const measured = isMeasured(row); + if (measured || MIRROR_EXISTS_STATUSES.has(row.status)) mirrored += 1; + if (!measured) continue; + + const location = (row.mirroredLocation ?? "").trim(); + const owner = location.includes("/") ? location.split("/")[0] : ""; + const destinationSize = row.destinationSize ?? 0; + const releaseAssetsSize = row.releaseAssetsSize ?? 0; + const total = destinationSize + releaseAssetsSize; + + repositories.push({ + id: row.id, + fullName: row.fullName || row.name, + location, + owner, + destinationSize: row.destinationSize, + releaseAssetsSize: row.releaseAssetsSize, + releaseAssetsCount: row.releaseAssetsCount, + total, + checkedAt: row.storageCheckedAt ? row.storageCheckedAt.toISOString() : null, + }); + + const group = owners.get(owner) ?? { + owner, + destinationSize: 0, + releaseAssetsSize: 0, + total: 0, + repositories: 0, + }; + group.destinationSize += destinationSize; + group.releaseAssetsSize += releaseAssetsSize; + group.total += total; + group.repositories += 1; + owners.set(owner, group); + } + + repositories.sort((a, b) => b.total - a.total || a.fullName.localeCompare(b.fullName)); + const ownerList = [...owners.values()].sort((a, b) => b.total - a.total || a.owner.localeCompare(b.owner)); + + return { + totals: { + total: repositories.reduce((sum, repo) => sum + repo.total, 0), + destinationSize: repositories.reduce((sum, repo) => sum + (repo.destinationSize ?? 0), 0), + releaseAssetsSize: repositories.reduce((sum, repo) => sum + (repo.releaseAssetsSize ?? 0), 0), + releaseAssetsCount: repositories.reduce((sum, repo) => sum + (repo.releaseAssetsCount ?? 0), 0), + measured: repositories.length, + mirrored, + }, + owners: ownerList, + repositories, + }; +} diff --git a/src/lib/sources.test.ts b/src/lib/sources.test.ts index 2c601ee..175b342 100644 --- a/src/lib/sources.test.ts +++ b/src/lib/sources.test.ts @@ -12,6 +12,32 @@ import { } from "./sources"; import { encrypt } from "./utils/encryption"; +// --------------------------------------------------------------------------- +// These tests need the real module, but gitea-enhanced.test.ts and several +// route tests replace @/lib/sources process-wide (decryptSourceToken becomes +// `token ?? ""`), and bun's mock.module leaks into every file that runs after +// them. Which file runs first depends on the runner, so this file re-runs +// itself in an isolated child process where nothing else has been loaded. +// --------------------------------------------------------------------------- +const CHILD_FLAG = "SOURCES_TEST_ISOLATED"; +const isChild = !!process.env[CHILD_FLAG]; + +if (!isChild) { + test("sources - isolated child suite", () => { + const res = Bun.spawnSync({ + cmd: [process.execPath, "test", import.meta.path], + env: { ...process.env, [CHILD_FLAG]: "1" }, + stdout: "pipe", + stderr: "pipe", + }); + if (res.exitCode !== 0) { + console.error(res.stdout.toString()); + console.error(res.stderr.toString()); + } + expect(res.exitCode).toBe(0); + }, 60_000); +} + function source(overrides: Partial = {}): SourceRecord { return { id: "src-1", @@ -28,7 +54,7 @@ function source(overrides: Partial = {}): SourceRecord { }; } -describe("deriveSourceName", () => { +describe.skipIf(!isChild)("deriveSourceName", () => { test("appends the username when present", () => { expect(deriveSourceName("github", "octocat")).toBe("GitHub (octocat)"); expect(deriveSourceName("gitea", "me")).toBe("Gitea / Forgejo (me)"); @@ -39,7 +65,7 @@ describe("deriveSourceName", () => { }); }); -describe("primarySource", () => { +describe.skipIf(!isChild)("primarySource", () => { test("returns the first entry", () => { const list = [source({ id: "a" }), source({ id: "b" })]; expect(primarySource(list)?.id).toBe("a"); @@ -50,7 +76,7 @@ describe("primarySource", () => { }); }); -describe("findSourceForRepository", () => { +describe.skipIf(!isChild)("findSourceForRepository", () => { const github = source(); const gitlab = source({ id: "src-2", @@ -95,7 +121,7 @@ describe("findSourceForRepository", () => { }); }); -describe("findSourceForOrganization", () => { +describe.skipIf(!isChild)("findSourceForOrganization", () => { const github = source(); const gitlab = source({ id: "src-2", @@ -122,7 +148,7 @@ describe("findSourceForOrganization", () => { }); }); -describe("selectSameRunPinsToClear", () => { +describe.skipIf(!isChild)("selectSameRunPinsToClear", () => { const discovered = [{ normalizedName: "acme" }, { normalizedName: "tools" }]; test("clears a pin an earlier source in the same run created", () => { @@ -148,7 +174,7 @@ describe("selectSameRunPinsToClear", () => { }); }); -describe("selectRepositoriesToRelink", () => { +describe.skipIf(!isChild)("selectRepositoriesToRelink", () => { const connected = new Set(["src-1"]); const base = { sourceProvider: "github" as const, @@ -191,7 +217,7 @@ describe("selectRepositoriesToRelink", () => { }); }); -describe("decryptSourceToken", () => { +describe.skipIf(!isChild)("decryptSourceToken", () => { beforeAll(() => { // encrypt() needs a secret; do not depend on the environment or on test order. process.env.ENCRYPTION_SECRET ??= "sources-test-encryption-secret"; @@ -208,7 +234,7 @@ describe("decryptSourceToken", () => { }); }); -describe("resolveGitHubApiBaseUrl", () => { +describe.skipIf(!isChild)("resolveGitHubApiBaseUrl", () => { test("github.com and empty keep the environment default", () => { expect(resolveGitHubApiBaseUrl("https://github.com")).toBeUndefined(); expect(resolveGitHubApiBaseUrl("")).toBeUndefined(); diff --git a/src/lib/utils/format-bytes.test.ts b/src/lib/utils/format-bytes.test.ts new file mode 100644 index 0000000..db6d985 --- /dev/null +++ b/src/lib/utils/format-bytes.test.ts @@ -0,0 +1,18 @@ +import { describe, expect, test } from "bun:test"; +import { formatBytes } from "./format-bytes"; + +describe("formatBytes", () => { + test("uses binary units like Gitea", () => { + expect(formatBytes(512)).toBe("512 B"); + expect(formatBytes(1536)).toBe("1.5 KiB"); + expect(formatBytes(5 * 1024 ** 2)).toBe("5.0 MiB"); + expect(formatBytes(141 * 1024 ** 3)).toBe("141 GiB"); + expect(formatBytes(2.25 * 1024 ** 4)).toBe("2.3 TiB"); + }); + + test("shows nothing as zero", () => { + expect(formatBytes(0)).toBe("0 B"); + expect(formatBytes(-5)).toBe("0 B"); + expect(formatBytes(Number.NaN)).toBe("0 B"); + }); +}); diff --git a/src/lib/utils/format-bytes.ts b/src/lib/utils/format-bytes.ts new file mode 100644 index 0000000..e4e061f --- /dev/null +++ b/src/lib/utils/format-bytes.ts @@ -0,0 +1,19 @@ +const UNITS = ["B", "KiB", "MiB", "GiB", "TiB", "PiB"] as const; + +/** + * A byte count in binary units, the way Gitea and Forgejo show repository + * sizes: one decimal below 100 of a unit, none above. + */ +export function formatBytes(bytes: number): string { + if (!Number.isFinite(bytes) || bytes <= 0) return "0 B"; + + let value = bytes; + let unit = 0; + while (value >= 1024 && unit < UNITS.length - 1) { + value /= 1024; + unit += 1; + } + + if (unit === 0) return `${Math.round(value)} B`; + return `${value >= 100 ? Math.round(value) : value.toFixed(1)} ${UNITS[unit]}`; +} diff --git a/src/pages/api/storage/index.test.ts b/src/pages/api/storage/index.test.ts new file mode 100644 index 0000000..b63d411 --- /dev/null +++ b/src/pages/api/storage/index.test.ts @@ -0,0 +1,146 @@ +/** + * Route tests for GET /api/storage (#479). + * + * NOTE: these tests replace @/lib/db and @/lib/utils/auth-helpers with mocks. + * bun's mock.module is process-wide and leaks into other test files, so in + * the shared test process this file registers NOTHING: it re-runs itself in + * an isolated child process (the same harness as repositories/[id].test.ts). + */ +import { describe, test, expect, mock, beforeEach } from "bun:test"; + +const CHILD_FLAG = "GM_STORAGE_ROUTE_ISOLATED"; +const isChild = !!process.env[CHILD_FLAG]; + +if (!isChild) { + test("storage route - isolated child suite", () => { + const res = Bun.spawnSync({ + cmd: [process.execPath, "test", import.meta.path], + env: { ...process.env, [CHILD_FLAG]: "1" }, + stdout: "pipe", + stderr: "pipe", + }); + if (res.exitCode !== 0) { + console.error(res.stdout.toString()); + console.error(res.stderr.toString()); + } + expect(res.exitCode).toBe(0); + }, 60_000); +} + +let repoRows: any[] = []; +let authResult: { user: { id: string } | null; response: Response | null } = { + user: { id: "user-1" }, + response: null, +}; +/** Leaf values of each select's where() condition (drizzle conditions are cyclic). */ +let selectWhereValues: unknown[][] = []; + +function conditionLeafValues(node: any, depth = 0): unknown[] { + if (node == null || depth > 8) return []; + if (typeof node !== "object") return [node]; + const leaves: unknown[] = []; + if (Array.isArray(node.value)) leaves.push(...node.value); + for (const chunk of node.queryChunks ?? []) { + leaves.push(...conditionLeafValues(chunk, depth + 1)); + } + return leaves; +} + +if (isChild) { + mock.module("@/lib/utils/auth-helpers", () => ({ + requireAuth: mock(async () => authResult), + })); + + mock.module("@/lib/db", () => ({ + db: { + select: () => ({ + from: () => ({ + where: async (cond: any) => { + selectWhereValues.push(conditionLeafValues(cond)); + return repoRows; + }, + }), + }), + }, + organizations: {}, + repositories: {}, + configs: {}, + users: {}, + events: {}, + mirrorJobs: {}, + sessions: {}, + accounts: {}, + verificationTokens: {}, + verifications: {}, + oauthClients: {}, + oauthAccessTokens: {}, + oauthRefreshTokens: {}, + oauthConsents: {}, + jwkss: {}, + ssoProviders: {}, + apikeys: {}, + rateLimits: {}, + sources: {}, + })); +} + +const { GET } = isChild ? await import("./index") : { GET: undefined as any }; + +function get() { + return GET({ + params: {}, + request: new Request("http://localhost/api/storage"), + locals: { session: { userId: "user-1" } }, + } as any); +} + +beforeEach(() => { + selectWhereValues = []; + authResult = { user: { id: "user-1" }, response: null }; + repoRows = [ + { + id: "repo-1", + name: "app", + fullName: "acme/app", + mirroredLocation: "archive/app", + status: "synced", + destinationProvider: "gitea", + destinationSize: 4096, + releaseAssetsSize: 1024, + releaseAssetsCount: 2, + storageCheckedAt: new Date("2026-10-29T08:00:00Z"), + }, + ]; +}); + +describe.skipIf(!isChild)("GET /api/storage", () => { + test("returns the overview of this user's repositories", async () => { + const response = await get(); + + expect(response.status).toBe(200); + const data = await response.json(); + expect(data.success).toBe(true); + expect(data.totals).toMatchObject({ total: 5120, measured: 1, mirrored: 1 }); + expect(data.owners).toEqual([ + { owner: "archive", destinationSize: 4096, releaseAssetsSize: 1024, total: 5120, repositories: 1 }, + ]); + expect(data.repositories[0]).toMatchObject({ + id: "repo-1", + location: "archive/app", + checkedAt: "2026-10-29T08:00:00.000Z", + }); + expect(selectWhereValues[0]).toEqual(expect.arrayContaining(["user-1"])); + }); + + test("answers the auth helper's response when signed out", async () => { + authResult = { + user: null, + response: new Response(JSON.stringify({ error: "Unauthorized" }), { status: 401 }), + }; + + const response = await get(); + + expect(response.status).toBe(401); + expect(selectWhereValues).toEqual([]); + }); +}); diff --git a/src/pages/api/storage/index.ts b/src/pages/api/storage/index.ts new file mode 100644 index 0000000..e6dd2b6 --- /dev/null +++ b/src/pages/api/storage/index.ts @@ -0,0 +1,41 @@ +import type { APIRoute } from "astro"; +import { eq } from "drizzle-orm"; +import { db, repositories } from "@/lib/db"; +import { createSecureErrorResponse } from "@/lib/utils"; +import { requireAuth } from "@/lib/utils/auth-helpers"; +import { buildStorageOverview } from "@/lib/repository-storage"; + +/** + * Storage the user's mirrors take up on the destination (#479): totals, per + * destination owner and per repository. Reads only what syncs recorded, so it + * never contacts the destination or the source. + */ +export const GET: APIRoute = async (context) => { + try { + const { user, response } = await requireAuth(context); + if (response) return response; + + const rows = await db + .select({ + id: repositories.id, + name: repositories.name, + fullName: repositories.fullName, + mirroredLocation: repositories.mirroredLocation, + status: repositories.status, + destinationProvider: repositories.destinationProvider, + destinationSize: repositories.destinationSize, + releaseAssetsSize: repositories.releaseAssetsSize, + releaseAssetsCount: repositories.releaseAssetsCount, + storageCheckedAt: repositories.storageCheckedAt, + }) + .from(repositories) + .where(eq(repositories.userId, user!.id)); + + return new Response(JSON.stringify({ success: true, ...buildStorageOverview(rows) }), { + status: 200, + headers: { "Content-Type": "application/json" }, + }); + } catch (error) { + return createSecureErrorResponse(error, "storage overview", 500); + } +}; diff --git a/src/pages/storage.astro b/src/pages/storage.astro new file mode 100644 index 0000000..3645c99 --- /dev/null +++ b/src/pages/storage.astro @@ -0,0 +1,23 @@ +--- +import '../styles/global.css'; +import App from '@/components/layout/MainLayout'; +import ThemeScript from '@/components/theme/ThemeScript.astro'; +import { withBase } from '@/lib/base-path'; + +--- + + + + + + + + Storage - Gitea Mirror + + + + + + + + diff --git a/src/types/Sidebar.ts b/src/types/Sidebar.ts index bb31ffd..f79c26d 100644 --- a/src/types/Sidebar.ts +++ b/src/types/Sidebar.ts @@ -4,6 +4,7 @@ export type Paths = | "/" | "/repositories" | "/organizations" + | "/storage" | "/config" | "/activity"; diff --git a/www/src/pages/docs/api.mdx b/www/src/pages/docs/api.mdx index 0ea284d..fc8fc32 100644 --- a/www/src/pages/docs/api.mdx +++ b/www/src/pages/docs/api.mdx @@ -160,6 +160,21 @@ When several sources are connected, an optional `sourceId` picks which one the o `PATCH /api/organizations/:id` with `{ "sourceId": "…" }` re-pins an existing organization to another source, and `{ "sourceId": null }` clears the pin so the organization follows every source's repositories of the same name again. A pin whose source was removed falls back the same way. +### Storage + +`GET /api/storage` returns what the Storage page shows: how much space the account's mirrors take on the destination (issue #479). + +```json +{ + "success": true, + "totals": { "total": 3554304, "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "measured": 1, "mirrored": 1 }, + "owners": [{ "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "total": 3554304, "repositories": 1 }], + "repositories": [{ "id": "…", "fullName": "upstream/app", "location": "archive/app", "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "total": 3554304, "checkedAt": "2026-10-29T03:32:01.000Z" }] +} +``` + +Sizes are bytes. `destinationSize` is Git and LFS together as Gitea or Forgejo reports it, and the release fields cover the assets on the destination's releases. Each sync records them from responses it already fetches, so this endpoint never contacts the destination. A repository is listed once a sync has measured it, and a field not measured yet is `null`. `mirrored` counts the repositories with a mirror on a Gitea or Forgejo destination and `measured` the ones with a size so far; push destinations (GitHub, GitLab) are not measured. + ### Run cleanup `POST /api/cleanup/trigger` runs the [repository cleanup](./cleanup-and-reconcile.md) now, with the account's saved cleanup settings, and returns how many orphaned repositories were found and processed and any errors. `GET /api/cleanup/trigger` returns the cleanup configuration and the last and next run times.