Compare commits

...
13 Commits
Author SHA1 Message Date
Arunavo Ray 4a4550b94f chore: bump version to 3.50.0 2026-10-30 07:11:19 +05:30
cypherbits 5e95fa7d41 feat(repositories): filter by destination (#492)
The repositories API returns each repository's destination, resolved the way the mirror does it, and the page filters on it. The Organization column shows the same value.

The desktop filter sits in the second filter row so the toolbar still fits at 1024 and 1280px.
2026-10-30 07:10:43 +05:30
cypherbits f6342de307 feat(issues): issue state, comment and media filters (#491)
Adds three issue options under the metadata settings, each with organization and repository overrides: a state filter (all, open only or closed only), copying comments, and copying images and attachments.

The state filter only decides which issues get created. Mirrored issues keep following GitHub, and changing the filter runs a full issue pass. Media removal covers GitHub uploads only.
2026-10-30 07:10:15 +05:30
Arunavo Ray 35b275035d chore: bump version to 3.49.0 2026-10-30 00:41:25 +05:30
ARUNAVO RAY ecdba3cad7 fix(ui): sort the Activity Log name filter case-insensitively too (#490)
Follow-up to #487: the owner, organization and Activity Log name filters share one compareNames (an Intl.Collator built once), so the Activity Log filter no longer puts capitalized names first.
2026-10-30 00:40:56 +05:30
Florian Zevedei 2485513973 fix(activities): accept the warning status in the activity feed (#488)
The feed parsed job statuses with the repository status enum, which has no "warning", so one release filter warning turned GET /api/activities into a 500. It now parses with activityStatusEnum. Adds a route test with warning and mirrored rows.
2026-10-30 00:32:56 +05:30
Florian Zevedei 43581f3e8b feat(releases): opt-in pruning of release assets past the asset limit (#489)
A global Remove old release assets switch (PRUNE_OLD_RELEASE_ASSETS) deletes the attachments of mirrored releases ranked past each repository's asset limit on every sync. Releases, notes and tags stay. Off by default. A release whose attachments cannot be listed is left alone, and releases the listing already shows without attachments are skipped.

Closes #485
2026-10-30 00:30:56 +05:30
Florian Zevedei 7e70e1005d fix(ui): sort owner and organization filters case-insensitively (#487)
Owner and organization filter options are sorted with Intl.Collator, case and accent insensitive, instead of a plain sort that put "Zebra" before "apple".
2026-10-30 00:25:43 +05:30
Arunavo Ray 9784821690 chore: bump version to 3.48.0 2026-10-29 12:31:34 +05:30
ARUNAVO RAY d94196ff4e perf(ui): keep page data across navigation and virtualize long lists (#484)
- Dashboard, Repositories, Organizations, Storage and Activity Log keep what they last showed in this tab and refresh it in the background instead of starting from a skeleton. The cache is in memory and per user; nothing goes to browser storage.
- The Repositories page renders only the layout on screen, and the mobile cards and Storage rows are virtualized against the page scroll. With 3,007 repositories the page went from 124,174 elements to 880.
- The desktop table rows live in their own memoized component that owns the virtualizer, so a scroll renders only the rows coming into view. MarqueeText no longer forces a style recalculation on mount. With the CPU slowed 4x a scroll frame went from 30.7 ms to 12.6 ms.
2026-10-29 12:31:17 +05:30
ARUNAVO RAY 3f65a40a7f feat(storage): Refresh sizes on the Storage page (#483)
- Refresh sizes measures every mirror on the destination now instead of at its next sync. It only reads the destination: one repository listing per owner, the single repository GET for a mirror the listing does not have, and the release listing of repositories that mirror releases. POST /api/storage/refresh runs it in the background; GET /api/storage reports its progress.
- Destination release listings stopped after 30 releases: Gitea and Forgejo ignore per_page and answer 30 a page. The retention pass, the release filter preview and the refresh now share one pager that asks with limit and ends at X-Total-Count.
2026-10-29 12:15:06 +05:30
ARUNAVO RAY f2c689a402 fix(ui): label the archived repository sync button Sync (#482)
The archived row's button says Sync like every other row, with a tooltip that automatic syncs are off for archived mirrors and the button syncs it now. Settings text, README and env docs point to Sync.
2026-10-29 11:37:21 +05:30
ARUNAVO RAY e02856180a fix(deps): zod 4.6.7 and a Dependabot config for bun (#481)
- Raise the zod floor to 4.6.7 (GHSA-mrhx-c7xc-m3w5). Astro's nested 4.5.4 resolves to the same copy and an override keeps both there. No other package in bun.lock changes.
- Add .github/dependabot.yml: bun for the root, npm for www, open-pull-requests-limit 0 so only security updates open pull requests.
2026-10-29 11:33:19 +05:30
67 changed files with 3681 additions and 590 deletions

No files matched your search

+20
View File
@@ -0,0 +1,20 @@
# Dependabot reads this for security updates as well as version updates.
# Without it, security updates for the root ran the npm ecosystem, which
# cannot update bun.lock, and every security update job failed.
#
# open-pull-requests-limit: 0 turns off version update pull requests and
# leaves security updates on, which is how the repository ran before.
version: 2
updates:
- package-ecosystem: "bun"
directory: "/"
schedule:
interval: "weekly"
open-pull-requests-limit: 0
# The documentation site, locked with pnpm.
- package-ecosystem: "npm"
directory: "/www"
schedule:
interval: "weekly"
open-pull-requests-limit: 0
+16
View File
@@ -7,7 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- **Issue filters** narrow issue mirroring (#491). A state filter creates only open or only closed issues; an issue that is already mirrored keeps following GitHub, so closing it there still closes the copy. Comments can be skipped, which also skips the comment listing for each issue, and images, videos and attached files can be left out of issue bodies and new comments, replaced by `[media omitted]`. All three sit under the metadata options and can be overridden per organization and repository. With media off, mirrored issue bodies lose their media as they are next synced; comments already mirrored stay as they are. Changing the state filter runs a full issue pass on the next sync
- **Refresh sizes** on the Storage page measures every mirror now instead of waiting for its next sync, so the page fills in on a fresh install. It only reads the destination: one repository listing per destination owner, the single repository GET for a mirror the listing does not have, and the release listing of repositories that mirror releases. It runs in the background with its progress on the button (`POST /api/storage/refresh`)
- **Remove old release assets** brings the attachments of older releases under a release asset limit. Turning the limit on, or lowering it, used to leave every file uploaded earlier in place, so the limit only held for the future. With the switch on, a kept release past the newest N releases that get assets loses its attachments on the next sync; the release, its notes and its tag stay, and a release whose attachments cannot be listed is left alone. It is a global setting (`PRUNE_OLD_RELEASE_ASSETS`, off by default) because it deletes files from the destination, while the limit it works with stays configurable per organization and repository
- **Destination filter** on the Repositories page lists repositories by where they mirror to, worked out the way the mirror does it: the starred organization, the repository's own destination, its organization's destination, then the mirror strategy (#492). The Organization column names that same destination now. It used to work it out in the browser and got some cases wrong, such as personal repositories under the mixed strategy, which it showed under the Gitea user instead of the organization they mirror into
### Changed
- Pages keep what they last showed in this browser tab. Going back to the Dashboard, Repositories, Organizations, Storage or Activity Log shows that data at once and refreshes it in the background, instead of starting from a skeleton and waiting for the whole list again. Nothing is written to browser storage; a reload or a new tab starts fresh
### Fixed
- The Activity Log came back as an error instead of the feed once a sync had finished with a warning. A run whose release filter matched nothing on the destination leaves a job whose status is `warning`, and the feed validated the status with the repository status list, which has no such value
- Long lists no longer put every row in the page. The Repositories page built a card for every repository even on desktop, where the cards are hidden, and the Storage table rendered every row; with 3,000 repositories the Repositories page held over 120,000 elements. Only the layout on screen is rendered now, and the mobile cards and Storage rows are virtualized like the desktop table, so a few hundred elements are on the page whatever the count
- The owner and organization filters on the Repositories page and the name filter on the Activity Log list names in alphabetical order regardless of case (#487). A plain sort put every capitalized name before the lowercase ones and accented names last
- Scrolling the Repositories table on desktop dropped frames on slower machines. Each scroll step rendered the filters and every visible row again, and each row coming into view forced a style recalculation for its name. Only the rows coming into view render now; with the CPU slowed 4x, a scroll frame went from about 31 ms to about 13 ms
- Destination release listings stopped after 30 releases. Gitea and Forgejo ignore `per_page` and answer 30 a page, and the listing stopped on any page shorter than the 100 it asked for. The release limit only trimmed within the newest 30 on each sync, the Storage page counted at most 30 releases per repository, and the release filter preview showed only the newest 30 tags. Listings now page with `limit` until the total the destination reports
- The sync button on an archived repository says **Sync** like every other row instead of **Manual Sync**, so the wider label no longer pushes the row's status badge out of line. Hovering it explains that automatic syncs are off for archived mirrors
- The Activity Log summary no longer shows a repository as syncing after a quick sync finished (#454). Events are stored to the second, and a sync that starts and finishes in the same second could come back in the wrong order, so the start event was counted as the current state. Activities now come back newest first within the same second too, and the summary counts a finish over a start from the same second
- Issue and pull request passes over large repositories finish across rate limit resets (#449 follow-up)
- A pass stopped by the rate limit keeps the numbers it finished, and the next run repeats the same pass and skips them unless they changed since it started; before, every run started over from the first item, so a repository too large for one hour of budget never completed a pass
@@ -28,6 +43,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- SSO providers are scoped to the user who created them for listing, updating and deleting, and the OIDC client secret is no longer returned by the API (the form keeps the stored secret when the field is left blank)
- HTTP errors from Gitea and other hosts no longer carry the upstream response body in the message, so an error can no longer echo the content of a host the caller chose
- Requests to user supplied URLs (connection tests, OIDC discovery, ntfy, Gotify, Apprise and webhook notifications) refuse link local and cloud metadata addresses and no longer follow redirects; private networks stay allowed because mirroring to a LAN Gitea is the normal deployment
- Raised the zod floor to 4.6.7 in the application (GHSA-mrhx-c7xc-m3w5, memory exhaustion while validating an oversized array that has a length limit). Before 4.6.6 zod still checked every element of a list longer than its `.max()`; the MCP tools' repository id list is the one capped array that takes outside input, behind an API key. Astro's own nested zod 4.5.4 now resolves to the same 4.6.7
- Raised the devalue floor to 5.9.2 in the application and the documentation site (GHSA-9rgm-9g3h-6x36, denial of service through malformed input); both lockfiles resolve 5.9.4
- Raised the dependency floors for the advisories published on 2026-09-08: Astro 7.2.8 (remote code execution through AVIF image optimization, and an authorization bypass when stripping the configured base), @xmldom/xmldom 0.8.15 (eight parser and serializer issues), sharp 0.35.4 (libheif), svgo 4.1.0 (removeScripts sanitization) and js-yaml 4.3.2 (merge-key CPU use). Applied to both the application and the documentation site.
+2 -2
View File
@@ -242,7 +242,7 @@ Transfer complete repository metadata from GitHub to Gitea/Forgejo:
Releases have two limits and two filters. The release limit keeps only the newest N releases in Gitea/Forgejo. The release asset limit uploads assets only for the newest N of those, so a repository with years of releases can keep every release note and tag while only the recent binaries take up space. Leave the asset limit empty to upload assets for every mirrored release, or set it to 0 for release notes only. The **Pre-releases** switch skips everything the source marks as a pre-release, and **Custom tag rules** narrows releases by tag name (`v1.*`, `*-rc.*`, or a regular expression in slashes) for projects that ship candidates without marking them.
All of it can be overridden per organization and per repository in the **Mirror options** dialog, and lowering the asset limit never deletes assets that were already uploaded.
All of it can be overridden per organization and per repository in the **Mirror options** dialog, and lowering the asset limit never deletes assets that were already uploaded. Enable **Remove old release assets** if you want each sync to delete the attachments of releases past that count instead.
Enable in Settings → Mirror Options → Mirror metadata
@@ -309,7 +309,7 @@ CLEANUP_DRY_RUN=false # Set to true to test without changes
- Regular repositories: Made read-only using Gitea/Forgejo's archive feature
- Mirror repositories: Renamed with `archived-` prefix (Gitea/Forgejo API limitation prevents archiving mirrors)
- Failed operations: Repository remains fully accessible even if marking as archived fails
- **Manual Sync on Demand**: Archived mirrors stay in Gitea/Forgejo with automatic syncs disabled; trigger `Manual Sync` from the Repositories page whenever you need fresh data.
- **Sync on Demand**: Archived mirrors stay in Gitea/Forgejo with automatic syncs disabled; use `Sync` on the Repositories page whenever you need fresh data.
- **The Whole Point of Backups**: Your Gitea/Forgejo mirrors are preserved even when GitHub sources disappear - that's why you have backups!
- **Strongly Recommended**: Always use `CLEANUP_ORPHANED_REPO_ACTION=archive` (default) instead of `delete`
+3 -4
View File
@@ -61,7 +61,7 @@
"tw-animate-css": "^1.4.0",
"uuid": "^14.0.1",
"vaul": "^1.1.2",
"zod": "^4.4.3",
"zod": "^4.6.7",
},
"devDependencies": {
"@astrojs/check": "^0.9.9",
@@ -99,6 +99,7 @@
"source-map-js": "^1.2.2",
"svgo": "^4.1.0",
"vite": "~8.1.6",
"zod": "^4.6.7",
},
"packages": {
"@adobe/css-tools": ["@adobe/css-tools@4.4.4", "", {}, "sha512-Elp+iwUx5rN5+Y8xLt5/GRoG20WGoDCQ/1Fb+1LiGtvwbDavuSk0jhD/eZdckHAuzcDzccnkv+rEjyWfRx18gg=="],
@@ -1695,7 +1696,7 @@
"yocto-queue": ["yocto-queue@1.2.2", "", {}, "sha512-4LCcse/U2MHZ63HAJVE+v71o7yOdIe4cZ70Wpf8D/IyjDKYQLV5GD46B+hSTjJsvV5PztjvHoU580EftxjDZFQ=="],
"zod": ["zod@4.4.3", "", {}, "sha512-ytENFjIJFl2UwYglde2jchW2Hwm4GJFLDiSXWdTrJQBIN9Fcyp7n4DhxJEiWNAJMV1/BqWfW/kkg71UDcHJyTQ=="],
"zod": ["zod@4.6.7", "", {}, "sha512-g8W2D2kU6a8untc053fX1uVBKzM7yqAxW/3zwk83+6RyF2+SKOuV+c0wBDhUwzV1PaGapmZH+rmcUzPf+3CGwg=="],
"zwitch": ["zwitch@2.0.4", "", {}, "sha512-bXE4cR/kVZhKZX/RjPEflHaKVhUVl85noU3v6b8apfQEc1x4A+zBxjZ4lN8LqGd6WZ3dl98pY4o717VFmoPp+A=="],
@@ -1757,8 +1758,6 @@
"astro/obug": ["obug@3.0.0", "", {}, "sha512-5vvB5+W7ePv+p3uqxi+RcW1XAzLW0/hxt3/4X4Lc4qHudzOhmBiBwOY6DRob4WnanAEGvNcLjF+KNOufrUoEQw=="],
"astro/zod": ["zod@4.5.4", "", {}, "sha512-sC95tT5iHHH9gtpj6A81kh+NEaRAUFN+qlUPDUbRfOMvNf5QCBqsb3WgvnpVtK5Y+4UfA6KqufotuTvMGiTlsA=="],
"better-call/@better-auth/utils": ["@better-auth/utils@0.5.0", "", { "dependencies": { "@noble/hashes": "^2.0.1" } }, "sha512-BL8W4EfIZFwlu0r54m3v1ztjDhu6dDe/amLTm0xybmbZaNgYUqhD3SjpAsnq0q8YD6/ki4iwIgxJNLP/N3TxiA=="],
"bun-types/@types/node": ["@types/node@25.9.3", "", { "dependencies": { "undici-types": ">=7.24.0 <7.24.7" } }, "sha512-603BddQMv3pUcr4U2dhujk83N2tTDVr/34wII2B6bJy6g+8WD6yUb11jszNs0gdi4PesVWl7ABt8nYMVpnLUcg=="],
+3 -7
View File
@@ -3605,13 +3605,9 @@
url = "https://registry.npmjs.org/yocto-queue/-/yocto-queue-1.2.2.tgz";
hash = "sha512-4LCcse/U2MHZ63HAJVE+v71o7yOdIe4cZ70Wpf8D/IyjDKYQLV5GD46B+hSTjJsvV5PztjvHoU580EftxjDZFQ==";
};
"zod@4.4.3" = fetchurl {
url = "https://registry.npmjs.org/zod/-/zod-4.4.3.tgz";
hash = "sha512-ytENFjIJFl2UwYglde2jchW2Hwm4GJFLDiSXWdTrJQBIN9Fcyp7n4DhxJEiWNAJMV1/BqWfW/kkg71UDcHJyTQ==";
};
"zod@4.5.4" = fetchurl {
url = "https://registry.npmjs.org/zod/-/zod-4.5.4.tgz";
hash = "sha512-sC95tT5iHHH9gtpj6A81kh+NEaRAUFN+qlUPDUbRfOMvNf5QCBqsb3WgvnpVtK5Y+4UfA6KqufotuTvMGiTlsA==";
"zod@4.6.7" = fetchurl {
url = "https://registry.npmjs.org/zod/-/zod-4.6.7.tgz";
hash = "sha512-g8W2D2kU6a8untc053fX1uVBKzM7yqAxW/3zwk83+6RyF2+SKOuV+c0wBDhUwzV1PaGapmZH+rmcUzPf+3CGwg==";
};
"zwitch@2.0.4" = fetchurl {
url = "https://registry.npmjs.org/zwitch/-/zwitch-2.0.4.tgz";
+14 -2
View File
@@ -165,11 +165,23 @@ When several sources are connected, an optional `sourceId` picks which one the o
"success": true,
"totals": { "total": 3554304, "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "measured": 1, "mirrored": 1 },
"owners": [{ "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "total": 3554304, "repositories": 1 }],
"repositories": [{ "id": "…", "fullName": "upstream/app", "location": "archive/app", "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "total": 3554304, "checkedAt": "2026-10-29T03:32:01.000Z" }]
"repositories": [{ "id": "…", "fullName": "upstream/app", "location": "archive/app", "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "total": 3554304, "checkedAt": "2026-10-29T03:32:01.000Z" }],
"refresh": null
}
```
Sizes are bytes. `destinationSize` is Git and LFS together as Gitea or Forgejo reports it, and the release fields cover the assets on the destination's releases. Each sync records them from responses it already fetches, so this endpoint never contacts the destination. A repository is listed once a sync has measured it, and a field not measured yet is `null`. `mirrored` counts the repositories with a mirror on a Gitea or Forgejo destination and `measured` the ones with a size so far; push destinations (GitHub, GitLab) are not measured.
Sizes are bytes. `destinationSize` is Git and LFS together as Gitea or Forgejo reports it, and the release fields cover the assets on the destination's releases. Each sync records them from responses it already fetches, so this endpoint never contacts the destination. A repository is listed once a sync or Refresh sizes has measured it, and a field not measured yet is `null`. `mirrored` counts the repositories with a mirror on a Gitea or Forgejo destination and `measured` the ones with a size so far; push destinations (GitHub, GitLab) are not measured. `refresh` is `null` until Refresh sizes runs, then carries the latest run's progress.
`POST /api/storage/refresh` starts Refresh sizes, which reads every mirror's size from the destination now instead of at its next sync. It lists each destination owner's repositories once, uses the single repository GET for a mirror the listing does not have, and reads the release listing of repositories that mirror releases. The source is never contacted. It answers `202` and runs in the background, one run per user; a call while a run is going answers with that run.
```json
{
"success": true,
"refresh": { "running": true, "total": 7, "done": 0, "measured": 0, "missing": 0, "failed": 0, "startedAt": "2026-10-29T06:20:28.807Z", "finishedAt": null }
}
```
`missing` counts mirrors not found on the destination and `failed` the ones a request failed for; both keep their stored numbers. A read only API key cannot start a run.
### Export as CSV
+3 -2
View File
@@ -217,7 +217,8 @@ Control what content gets mirrored from GitHub to Gitea.
| `RELEASE_TYPE` | Which releases are candidates: `all` mirrors everything, `stable` skips what the source marks as a pre-release. With a filter active the release limit counts the releases that match, and releases of the other kind are removed from Gitea on the next sync | `all` | `all`, `stable` |
| `RELEASE_INCLUDE_TAGS` | Comma separated tag patterns; only releases whose tag matches are mirrored. A pattern is a glob over the whole tag (`v1.*`, `*-rc.*`, `?` for one character) or a regular expression in slashes (`/^v\d+\.\d+\.\d+$/`), matched case-insensitively. Commas inside the slashes belong to the expression (`/^v\d{1,3}\.\d+$/`) | empty | any list of globs or `/regex/` |
| `RELEASE_EXCLUDE_TAGS` | Comma separated tag patterns in the same syntax; releases whose tag matches are skipped, and an excluded tag wins over `RELEASE_INCLUDE_TAGS` | empty | any list of globs or `/regex/` |
| `RELEASE_ASSET_LIMIT` | Upload release assets only for the newest N mirrored releases. Older releases still get their notes and tag. Unset means assets for every mirrored release, `0` means release notes only. Lowering it never deletes assets already uploaded | unset (all) | Number (0 or more) |
| `RELEASE_ASSET_LIMIT` | Upload release assets only for the newest N mirrored releases. Older releases still get their notes and tag. Unset means assets for every mirrored release, `0` means release notes only. Lowering it never deletes assets already uploaded unless `PRUNE_OLD_RELEASE_ASSETS` is on | unset (all) | Number (0 or more) |
| `PRUNE_OLD_RELEASE_ASSETS` | Opt-in cleanup for what the asset limit left behind. Off, a sync only ever uploads and attachments already on Gitea stay. On, every sync also deletes the attachments of each mirrored release ranked past `RELEASE_ASSET_LIMIT`; the releases themselves and their notes stay | `false` | `true`, `false` |
| `MIRROR_WIKI` | Mirror wiki content | `false` | `true`, `false` |
| `MIRROR_METADATA` | Master toggle for metadata mirroring | `false` | `true`, `false` |
| `MIRROR_ISSUES` | Mirror issues (requires MIRROR_METADATA=true) | `false` | `true`, `false` |
@@ -333,7 +334,7 @@ Configure automatic cleanup of old events and data.
- Description updated with preservation notice and timestamp
- Mirror interval set to 8760h (1 year) to minimize sync attempts
- Repository remains fully accessible and cloneable
- **Manual Sync Option**: Archived mirrors are still available on the Repositories page with automatic syncs disabled—use the `Manual Sync` action to refresh them on demand.
- **Sync on Demand**: Archived mirrors are still available on the Repositories page with automatic syncs disabled; use the `Sync` action to refresh them on demand.
### Execution Settings
+2 -2
View File
@@ -2,8 +2,8 @@ apiVersion: v2
name: gitea-mirror
description: Kubernetes helm chart for gitea-mirror
type: application
version: 3.47.0
appVersion: "3.47.0"
version: 3.50.0
appVersion: "3.50.0"
icon: https://github.com/RayLabsHQ/gitea-mirror/blob/main/.github/assets/logo.png
keywords:
- git
+4 -3
View File
@@ -1,7 +1,7 @@
{
"name": "gitea-mirror",
"type": "module",
"version": "3.47.0",
"version": "3.50.0",
"engines": {
"bun": ">=1.2.9"
},
@@ -59,7 +59,8 @@
"svgo": "^4.1.0",
"source-map-js": "^1.2.2",
"http-cache-semantics": "^4.3.0",
"vite": "~8.1.6"
"vite": "~8.1.6",
"zod": "^4.6.7"
},
"dependencies": {
"@astrojs/mdx": "^7.0.3",
@@ -118,7 +119,7 @@
"tw-animate-css": "^1.4.0",
"uuid": "^14.0.1",
"vaul": "^1.1.2",
"zod": "^4.4.3"
"zod": "^4.6.7"
},
"devDependencies": {
"@astrojs/check": "^0.9.9",
+17 -5
View File
@@ -19,6 +19,7 @@ import {
DialogTrigger,
} from '../ui/dialog';
import { apiRequest, cn, formatDate, showErrorToast } from '@/lib/utils';
import { readPageCache, writePageCache } from '@/lib/page-cache';
import { useAuth } from '@/hooks/useAuth';
import type { MirrorJob } from '@/lib/db/schema';
import type { ActivityApiResponse } from '@/types/activities';
@@ -84,8 +85,12 @@ export function ActivityLog() {
const { isFullyConfigured } = useConfigStatus();
const { navigationKey } = useNavigation();
const [activities, setActivities] = useState<MirrorJobWithKey[]>([]);
// What this page showed last time in this tab, shown again while it refreshes.
const [cached] = useState(() => readPageCache<MirrorJobWithKey[]>(user?.id, 'activity-log'));
const [activities, setActivities] = useState<MirrorJobWithKey[]>(cached ?? []);
const [isInitialLoading, setIsInitialLoading] = useState(false);
// True once there are rows to show, from the cache or a fetch.
const hasLoadedRef = useRef(!!cached);
const [showCleanupDialog, setShowCleanupDialog] = useState(false);
const [showCancelPendingDialog, setShowCancelPendingDialog] = useState(false);
const [isCancelPendingLoading, setIsCancelPendingLoading] = useState(false);
@@ -192,6 +197,7 @@ export function ActivityLog() {
});
if (isMountedRef.current) {
hasLoadedRef.current = true;
setActivities(data);
}
return true;
@@ -211,11 +217,17 @@ export function ActivityLog() {
}, [user?.id]); // Only depend on user.id, not entire user object
useEffect(() => {
// Reset loading state when component becomes active
setIsInitialLoading(true);
fetchActivities(false); // Manual refresh, not live
// With rows from the last visit on screen, refresh them in the background
// instead of going back to the skeleton.
const showingRows = hasLoadedRef.current;
if (!showingRows) setIsInitialLoading(true);
fetchActivities(showingRows);
}, [fetchActivities, navigationKey]); // Include navigationKey to trigger on navigation
useEffect(() => {
if (hasLoadedRef.current) writePageCache(user?.id, 'activity-log', activities);
}, [user?.id, activities]);
// Register with global live refresh system
useEffect(() => {
// Only register for live refresh if configuration is complete
@@ -833,7 +845,7 @@ export function ActivityLog() {
<ActivityList
className="min-h-0 flex-1"
activities={visibleActivities}
isLoading={isInitialLoading || !connected}
isLoading={isInitialLoading || (!connected && !hasLoadedRef.current)}
filter={filter}
setFilter={setFilter}
/>
@@ -15,6 +15,7 @@ import {
} from "@/components/ui/popover";
import { selectTriggerClassName } from "@/components/ui/select";
import { cn } from "@/lib/utils";
import { compareNames } from "@/lib/utils/compare-names";
type ActivityNameComboboxProps = {
activities: any[];
@@ -39,7 +40,7 @@ export function ActivityNameCombobox({
if (a.repositoryName) set.add(a.repositoryName);
if (a.organizationName) set.add(a.organizationName);
});
return Array.from(set).sort();
return Array.from(set).sort(compareNames);
}, [activities]);
const [open, setOpen] = React.useState(false);
+1 -1
View File
@@ -117,7 +117,7 @@ const orphanActions = [
label: "Archive",
description: "Kept read-only with an archived prefix",
icon: Archive,
info: "Renames the mirror with an archived- prefix and disables automatic syncs. Nothing is lost; use Manual Sync to refresh.",
info: "Renames the mirror with an archived- prefix and disables automatic syncs. Nothing is lost; use Sync on the Repositories page to refresh.",
},
{
value: "delete" as const,
+3
View File
@@ -96,6 +96,9 @@ export function ConfigTabs() {
milestones: false,
wiki: false,
},
issueState: "all",
issueComments: true,
issueMedia: true,
},
advancedOptions: {
skipForks: false,
+77 -1
View File
@@ -39,6 +39,7 @@ import {
UserX,
X,
FlaskConical,
Trash2,
} from "lucide-react";
import type { GitHubConfig, MirrorOptions, AdvancedOptions, DuplicateNameStrategy, DestinationProvider } from "@/types/config";
import { DESTINATION_PROVIDER_LABELS, isPushDestinationKind } from "@/lib/destination-kinds";
@@ -479,7 +480,7 @@ export function GitHubMirrorSettings({
<ChevronDown className="ml-2 h-3 w-3 opacity-50" />
</Button>
</PopoverTrigger>
<PopoverContent align="end" className="w-64">
<PopoverContent align="end" className="w-72">
<div className="space-y-3">
<div className="flex items-center justify-between">
<div className="text-sm font-medium">Metadata to mirror</div>
@@ -528,6 +529,63 @@ export function GitHubMirrorSettings({
</Label>
</div>
{isGithubSource && mirrorOptions.metadataComponents.issues && (
<div className="ml-8 space-y-2 border-l pl-3">
<div className="flex items-center justify-between gap-2">
<Label htmlFor="issue-state-popup" className="text-xs font-normal">
State
</Label>
<Select
value={mirrorOptions.issueState ?? "all"}
onValueChange={(value) =>
handleMirrorChange("issueState", value)
}
>
<SelectTrigger id="issue-state-popup" className="h-8 w-28 text-xs">
<SelectValue />
</SelectTrigger>
<SelectContent align="end">
<SelectItem value="all" className="text-xs">All issues</SelectItem>
<SelectItem value="open" className="text-xs">Open only</SelectItem>
<SelectItem value="closed" className="text-xs">Closed only</SelectItem>
</SelectContent>
</Select>
</div>
<div className="flex items-center space-x-2">
<Checkbox
id="issue-comments-popup"
checked={mirrorOptions.issueComments ?? true}
onCheckedChange={(checked) =>
handleMirrorChange("issueComments", !!checked)
}
/>
<Label
htmlFor="issue-comments-popup"
className="text-xs font-normal cursor-pointer"
>
Copy comments
</Label>
</div>
<div className="flex items-center space-x-2">
<Checkbox
id="issue-media-popup"
checked={mirrorOptions.issueMedia ?? true}
onCheckedChange={(checked) =>
handleMirrorChange("issueMedia", !!checked)
}
/>
<Label
htmlFor="issue-media-popup"
className="text-xs font-normal cursor-pointer"
>
Copy images and attachments
</Label>
</div>
</div>
)}
<div className="flex items-center space-x-3 py-1 px-1 rounded hover:bg-accent">
<Checkbox
id="metadata-prs-popup"
@@ -1055,6 +1113,24 @@ export function GitHubMirrorSettings({
/>
)}
{/* Deleting what a lowered limit already uploaded is destructive, so it
stays off until the user asks for it. */}
{mirrorOptions.mirrorReleases && (
<SwitchRow
icon={Trash2}
className="pl-6"
label="Remove old release assets"
description={
isPushTarget
? pushTargetReason
: "Off by default: attachments stay on the destination even when a release falls outside \u201cAssets for latest\u201d. Turning this on deletes the attachments of every mirrored release past that count"
}
disabled={!isReleaseCapableSource}
checked={mirrorOptions.pruneOldReleaseAssets === true}
onCheckedChange={(checked) => handleMirrorChange('pruneOldReleaseAssets', checked)}
/>
)}
<SwitchRow
icon={HardDrive}
label="Git LFS objects"
@@ -20,11 +20,13 @@ import {
import { ChevronRight } from "lucide-react";
import { cn } from "@/lib/utils";
import { withBase } from "@/lib/base-path";
import type { MirrorOverrides, ReleaseType } from "@/lib/db/schema";
import type { IssueStateFilter, MirrorOverrides, ReleaseType } from "@/lib/db/schema";
import {
compileReleaseTagPattern,
DEFAULT_ISSUE_STATE,
getMirrorOverrideGating,
MIRROR_OVERRIDE_LABELS,
normalizeIssueState,
normalizeReleaseAssetLimit,
normalizeReleaseLimit,
normalizeReleaseTagPatterns,
@@ -48,6 +50,16 @@ type TriState = "inherit" | "on" | "off";
*/
type ReleaseTypeChoice = "inherit" | ReleaseType;
/** Issue state is a choice too, so "inherit" persists as absent. */
type IssueStateChoice = "inherit" | IssueStateFilter;
/** How the issue-state row describes the value it inherits. */
const ISSUE_STATE_LABELS: Record<IssueStateFilter, string> = {
all: "all issues",
open: "open only",
closed: "closed only",
};
/**
* What the single Releases row offers. It covers both stored fields at once:
* "off" pins the `mirrorReleases` flag false, the two kinds pin it true
@@ -203,6 +215,9 @@ export function MirrorOverridesDialog({
const [releaseTypeDraft, setReleaseTypeDraft] = useState<ReleaseTypeChoice>(
() => buildReleaseTypeDraft(value)
);
const [issueStateDraft, setIssueStateDraft] = useState<IssueStateChoice>(
() => buildIssueStateDraft(value)
);
const [includeTagsDraft, setIncludeTagsDraft] = useState(() =>
buildReleaseTagDraft(value, "releaseIncludeTags")
);
@@ -225,6 +240,7 @@ export function MirrorOverridesDialog({
setReleaseLimitDraft(buildReleaseLimitDraft(value));
setReleaseAssetLimitDraft(buildReleaseAssetLimitDraft(value));
setReleaseTypeDraft(buildReleaseTypeDraft(value));
setIssueStateDraft(buildIssueStateDraft(value));
setIncludeTagsDraft(buildReleaseTagDraft(value, "releaseIncludeTags"));
setExcludeTagsDraft(buildReleaseTagDraft(value, "releaseExcludeTags"));
}
@@ -317,6 +333,9 @@ export function MirrorOverridesDialog({
const pinnedReleaseType: ReleaseType | undefined =
releaseTypeDraft === "inherit" ? undefined : normalizeReleaseType(releaseTypeDraft);
const pinnedIssueState: IssueStateFilter | undefined =
issueStateDraft === "inherit" ? undefined : normalizeIssueState(issueStateDraft);
// One control drives two stored fields. A stored "releases on" with no kind
// pinned still reads as Inherit and keeps its pin on save, so an override set
// before the kind filter existed is never dropped by merely opening the dialog.
@@ -356,6 +375,7 @@ export function MirrorOverridesDialog({
(draft.mirrorReleases !== "inherit" || releaseTypeDraft !== "inherit"
? 1
: 0) +
(pinnedIssueState !== undefined ? 1 : 0) +
(pinnedReleaseLimit !== undefined ? 1 : 0) +
(pinnedReleaseAssetLimit !== undefined ? 1 : 0) +
(includeTags.patterns !== undefined ? 1 : 0) +
@@ -380,6 +400,9 @@ export function MirrorOverridesDialog({
next.releaseAssetLimit = pinnedReleaseAssetLimit;
}
if (pinnedReleaseType !== undefined) next.releaseType = pinnedReleaseType;
if (pinnedIssueState !== undefined) {
next.mirrorIssueState = pinnedIssueState;
}
if (includeTags.patterns !== undefined) {
next.releaseIncludeTags = includeTags.patterns;
}
@@ -398,6 +421,7 @@ export function MirrorOverridesDialog({
setReleaseLimitDraft("");
setReleaseAssetLimitDraft("");
setReleaseTypeDraft("inherit");
setIssueStateDraft("inherit");
setIncludeTagsDraft("");
setExcludeTagsDraft("");
};
@@ -469,6 +493,59 @@ export function MirrorOverridesDialog({
tagPatternCount === 0 &&
inheritedTagText !== "";
const issueStateReason = gating.mirrorIssueState;
const issueStateDisabled = !!issueStateReason;
const effectiveIssueState =
pinnedIssueState ?? inheritedFrom?.mirrorIssueState ?? DEFAULT_ISSUE_STATE;
const showIssueStateHint =
!inheritedLoading && !issueStateDisabled && issueStateDraft === "inherit";
// Rendered under the Issues row, with the other issue options.
const issueStateRow = (
<div key="mirrorIssueState" className="space-y-1">
<div className="flex items-center justify-between gap-4">
<Label
htmlFor="override-mirrorIssueState"
className={
issueStateDisabled ? "flex-1 text-muted-foreground" : "flex-1"
}
>
{MIRROR_OVERRIDE_LABELS.mirrorIssueState}
{showIssueStateHint && (
<span className="ml-2 text-xs font-normal text-muted-foreground">
currently {ISSUE_STATE_LABELS[effectiveIssueState]}
</span>
)}
</Label>
<Select
value={issueStateDraft}
disabled={issueStateDisabled}
onValueChange={(next) =>
setIssueStateDraft(next as IssueStateChoice)
}
>
<SelectTrigger
id="override-mirrorIssueState"
className="w-32"
>
<SelectValue />
</SelectTrigger>
<SelectContent>
<SelectItem value="inherit">Inherit</SelectItem>
<SelectItem value="all">All issues</SelectItem>
<SelectItem value="open">Open only</SelectItem>
<SelectItem value="closed">Closed only</SelectItem>
</SelectContent>
</Select>
</div>
{issueStateReason && (
<p className="text-xs text-muted-foreground">
{issueStateReason}
</p>
)}
</div>
);
return (
<Dialog open={open} onOpenChange={onOpenChange}>
<DialogContent className="sm:max-w-md">
@@ -668,7 +745,7 @@ export function MirrorOverridesDialog({
draft[key] === "inherit" &&
inherited !== undefined;
return (
const row = (
<div key={key} className="space-y-1">
<div className="flex items-center justify-between gap-4">
<Label
@@ -706,6 +783,7 @@ export function MirrorOverridesDialog({
)}
</div>
);
return key === "mirrorIssues" ? [row, issueStateRow] : row;
})}
<p className="text-xs text-muted-foreground">
@@ -793,3 +871,9 @@ function buildReleaseTypeDraft(
): ReleaseTypeChoice {
return normalizeReleaseType(value?.releaseType) ?? "inherit";
}
function buildIssueStateDraft(
value: MirrorOverrides | null | undefined
): IssueStateChoice {
return normalizeIssueState(value?.mirrorIssueState) ?? "inherit";
}
+41 -11
View File
@@ -4,6 +4,7 @@ import { RepositoryList } from "./RepositoryList";
import { GitFork, Clock, FlipHorizontal, Building2 } from "lucide-react";
import { useCallback, useEffect, useRef, useState } from "react";
import type { MirrorJob, Organization, Repository } from "@/lib/db/schema";
import { readPageCache, writePageCache } from "@/lib/page-cache";
import { useAuth } from "@/hooks/useAuth";
import { apiRequest, showErrorToast } from "@/lib/utils";
import type { DashboardApiResponse } from "@/types/dashboard";
@@ -54,8 +55,22 @@ function formatFullTimestamp(date: Date | null): string {
return formatShortDateTime(date).replace(',', '');
}
type DashboardPageCache = {
repositories: Repository[];
organizations: Organization[];
activities: MirrorJob[];
repoCount: number;
orgCount: number;
mirroredCount: number;
lastSync: Date | null;
};
export function Dashboard() {
const { user } = useAuth();
// What this page showed last time in this tab, shown again while it refreshes.
const [cached] = useState(() => readPageCache<DashboardPageCache>(user?.id, "dashboard"));
// True once there is data to show, from the cache or a fetch.
const hasLoadedRef = useRef(!!cached);
const { registerRefreshCallback } = useLiveRefresh();
const isPageVisible = usePageVisibility();
const { isFullyConfigured } = useConfigStatus();
@@ -63,14 +78,14 @@ export function Dashboard() {
// Re-render timestamps when the user changes the 12h/24h preference.
useTimeFormat();
const [repositories, setRepositories] = useState<Repository[]>([]);
const [organizations, setOrganizations] = useState<Organization[]>([]);
const [activities, setActivities] = useState<MirrorJob[]>([]);
const [isLoading, setIsLoading] = useState(true);
const [repoCount, setRepoCount] = useState<number>(0);
const [orgCount, setOrgCount] = useState<number>(0);
const [mirroredCount, setMirroredCount] = useState<number>(0);
const [lastSync, setLastSync] = useState<Date | null>(null);
const [repositories, setRepositories] = useState<Repository[]>(cached?.repositories ?? []);
const [organizations, setOrganizations] = useState<Organization[]>(cached?.organizations ?? []);
const [activities, setActivities] = useState<MirrorJob[]>(cached?.activities ?? []);
const [isLoading, setIsLoading] = useState(!cached);
const [repoCount, setRepoCount] = useState<number>(cached?.repoCount ?? 0);
const [orgCount, setOrgCount] = useState<number>(cached?.orgCount ?? 0);
const [mirroredCount, setMirroredCount] = useState<number>(cached?.mirroredCount ?? 0);
const [lastSync, setLastSync] = useState<Date | null>(cached?.lastSync ?? null);
// Dashboard auto-refresh timer (30 seconds)
const dashboardTimerRef = useRef<NodeJS.Timeout | null>(null);
@@ -176,6 +191,7 @@ export function Dashboard() {
);
if (response.success) {
hasLoadedRef.current = true;
setRepositories(response.repositories);
setOrganizations(response.organizations);
setActivities(response.activities);
@@ -202,11 +218,25 @@ export function Dashboard() {
// Initial data fetch and reset loading state when component becomes active
useEffect(() => {
// Reset loading state when component mounts or becomes active
setIsLoading(true);
// With data from the last visit on screen, refresh it in the background
// instead of going back to the skeleton.
if (!hasLoadedRef.current) setIsLoading(true);
fetchDashboardData();
}, [fetchDashboardData, navigationKey]); // Include navigationKey to trigger on navigation
useEffect(() => {
if (!hasLoadedRef.current) return;
writePageCache<DashboardPageCache>(user?.id, "dashboard", {
repositories,
organizations,
activities,
repoCount,
orgCount,
mirroredCount,
lastSync,
});
}, [user?.id, repositories, organizations, activities, repoCount, orgCount, mirroredCount, lastSync]);
// Setup dashboard auto-refresh (30 seconds) and register with live refresh
useEffect(() => {
// Clear any existing timer
@@ -263,7 +293,7 @@ export function Dashboard() {
);
}
return isLoading || !connected ? (
return isLoading || (!connected && !hasLoadedRef.current) ? (
<div className="flex flex-col gap-y-6">
<div className="grid grid-cols-2 lg:grid-cols-4 gap-3 sm:gap-4 md:gap-6">
<StatusCardSkeleton />
+9 -4
View File
@@ -14,8 +14,12 @@ import { useRepoSync } from "@/hooks/useSyncRepo";
import { useConfigStatus } from "@/hooks/useConfigStatus";
import { stripBasePath, withBase } from "@/lib/base-path";
// Navigation context to signal when navigation happens
const NavigationContext = createContext<{ navigationKey: number }>({ navigationKey: 0 });
// Navigation context to signal when navigation happens. It also carries the
// section the pages scroll in, for lists virtualized against the page scroll.
const NavigationContext = createContext<{ navigationKey: number; pageScrollElement: HTMLElement | null }>({
navigationKey: 0,
pageScrollElement: null,
});
export const useNavigation = () => useContext(NavigationContext);
@@ -47,6 +51,7 @@ function AppWithProviders({ page: initialPage }: AppProps) {
const { isLoading: configLoading } = useConfigStatus();
const [currentPage, setCurrentPage] = useState<AppProps['page']>(initialPage);
const [navigationKey, setNavigationKey] = useState(0);
const [pageScrollElement, setPageScrollElement] = useState<HTMLElement | null>(null);
const [sidebarOpen, setSidebarOpen] = useState(false);
const [sidebarCollapsed, setSidebarCollapsed] = useState(() => {
// Check if we're on medium screens (768px - 1280px)
@@ -135,7 +140,7 @@ function AppWithProviders({ page: initialPage }: AppProps) {
}
return (
<NavigationContext.Provider value={{ navigationKey }}>
<NavigationContext.Provider value={{ navigationKey, pageScrollElement }}>
<main className="flex h-dvh flex-col overflow-hidden">
<Header
currentPage={currentPage}
@@ -153,7 +158,7 @@ function AppWithProviders({ page: initialPage }: AppProps) {
onClose={() => setSidebarOpen(false)}
onToggleCollapse={() => setSidebarCollapsed(!sidebarCollapsed)}
/>
<section className={`flex-1 min-h-0 p-4 sm:p-6 overflow-y-auto w-full transition-all duration-200 ${
<section ref={setPageScrollElement} className={`flex-1 min-h-0 p-4 sm:p-6 overflow-y-auto w-full transition-all duration-200 ${
sidebarCollapsed ? 'md:w-[calc(100%-5rem)] xl:w-[calc(100%-16rem)]' : 'md:w-[calc(100%-16rem)]'
}`}>
{currentPage === "dashboard" && <Dashboard />}
+20 -8
View File
@@ -1,4 +1,4 @@
import { useCallback, useEffect, useState } from "react";
import { useCallback, useEffect, useRef, useState } from "react";
import { Button } from "@/components/ui/button";
import { Search, RefreshCw, FlipHorizontal, Filter, LoaderCircle, Trash2, Download, LayoutGrid, List } from "lucide-react";
import type { MirrorJob, Organization } from "@/lib/db/schema";
@@ -6,6 +6,7 @@ import { OrganizationList } from "./OrganizationsList";
import AddOrganizationDialog from "./AddOrganizationDialog";
import { useAuth } from "@/hooks/useAuth";
import { apiRequest, showErrorToast } from "@/lib/utils";
import { readPageCache, writePageCache } from "@/lib/page-cache";
import {
membershipRoleEnum,
type AddOrganizationApiRequest,
@@ -53,12 +54,16 @@ import {
} from "@/components/ui/dialog";
export function Organization() {
const [organizations, setOrganizations] = useState<Organization[]>([]);
const [isLoading, setIsLoading] = useState<boolean>(true);
const { user } = useAuth();
// What this page showed last time in this tab, shown again while it refreshes.
const [cached] = useState(() => readPageCache<Organization[]>(user?.id, "organizations"));
const [organizations, setOrganizations] = useState<Organization[]>(cached ?? []);
const [isLoading, setIsLoading] = useState<boolean>(!cached);
// True once there are organizations to show, from the cache or a fetch.
const hasLoadedRef = useRef(!!cached);
const [isDialogOpen, setIsDialogOpen] = useState<boolean>(false);
// Cards or the compact list (#428); remembered per browser.
const { view, setView } = useOrganizationsView();
const { user } = useAuth();
const { sourceProvider, sourceUrl, sources } = useConfigStatus();
const { navigationKey } = useNavigation();
const { registerRefreshCallback } = useLiveRefresh();
@@ -118,6 +123,7 @@ export function Organization() {
);
if (response.success) {
hasLoadedRef.current = true;
setOrganizations(response.organizations);
return true;
} else {
@@ -141,11 +147,17 @@ export function Organization() {
}, [user?.id]); // Only depend on user.id, not entire user object
useEffect(() => {
// Reset loading state when component becomes active
setIsLoading(true);
fetchOrganizations(false); // Manual refresh, not live
// With organizations from the last visit on screen, refresh them in the
// background instead of going back to the skeleton.
const showingRows = hasLoadedRef.current;
if (!showingRows) setIsLoading(true);
fetchOrganizations(showingRows);
}, [fetchOrganizations, navigationKey]); // Include navigationKey to trigger on navigation
useEffect(() => {
if (hasLoadedRef.current) writePageCache(user?.id, "organizations", organizations);
}, [user?.id, organizations]);
// Register with global live refresh system
useEffect(() => {
const unregister = registerRefreshCallback(() => {
@@ -985,7 +997,7 @@ export function Organization() {
<OrganizationList
organizations={organizations}
isLoading={isLoading || !connected}
isLoading={isLoading || (!connected && !hasLoadedRef.current)}
filter={filter}
setFilter={setFilter}
loadingOrgIds={loadingOrgIds}
@@ -31,6 +31,12 @@ interface InlineDestinationEditorProps {
* row would otherwise name the strategy's owner instead (#416).
*/
organizationDestination?: string | null;
/**
* Where the server resolved the mirror to go, with the same precedence and
* strategy rules the mirror itself uses. The local guess below gets some
* strategies wrong, such as personal repositories under "mixed".
*/
resolvedDestination?: string;
onUpdate: (
repoId: string,
newDestination: string | null,
@@ -48,6 +54,7 @@ export function InlineDestinationEditor({
repository,
giteaConfig,
organizationDestination,
resolvedDestination,
onUpdate,
isUpdating = false,
className,
@@ -105,7 +112,10 @@ export function InlineDestinationEditor({
}
};
const defaultDestination = getDefaultDestination();
// A repository pinned to its own destination resolves to that pin, so its
// default still comes from the local guess.
const defaultDestination =
(!repository.destinationOrg && resolvedDestination) || getDefaultDestination();
const currentDestination = repository.destinationOrg || defaultDestination;
// A stored destination is a pin even when it spells out the default, so the
// badge follows the stored value rather than comparing against it (#416).
+89 -14
View File
@@ -1,4 +1,4 @@
import { useCallback, useEffect, useMemo, useState } from "react";
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import RepositoryTable, { REPOSITORY_SORT_OPTIONS } from "./RepositoryTable";
import type { MirrorJob, Repository } from "@/lib/db/schema";
import { useAuth } from "@/hooks/useAuth";
@@ -10,6 +10,7 @@ import {
type RepoStatus,
} from "@/types/Repository";
import { apiRequest, showErrorToast, getStatusColor } from "@/lib/utils";
import { readPageCache, writePageCache } from "@/lib/page-cache";
import {
Select,
SelectContent,
@@ -42,7 +43,11 @@ import { useSSE } from "@/hooks/useSEE";
import { useFilterParams } from "@/hooks/useFilterParams";
import { toast } from "sonner";
import type { SyncRepoRequest, SyncRepoResponse } from "@/types/sync";
import { OwnerCombobox, OrganizationCombobox } from "./RepositoryComboboxes";
import {
OwnerCombobox,
OrganizationCombobox,
DestinationCombobox,
} from "./RepositoryComboboxes";
import type { RetryRepoRequest, RetryRepoResponse } from "@/types/retry";
import type { ResetMetadataRequest, ResetMetadataResponse } from "@/types/reset-metadata";
import AddRepositoryDialog from "./AddRepositoryDialog";
@@ -52,17 +57,32 @@ import { useConfigStatus } from "@/hooks/useConfigStatus";
import { useNavigation } from "@/components/layout/MainLayout";
import { withBase } from "@/lib/base-path";
import { getRepositorySource, normalizeSourceUrl } from "@/lib/source-providers/kinds";
import { compareNames } from "@/lib/utils/compare-names";
type RepositoriesPageCache = {
repositories: Repository[];
organizationDestinations: Record<string, string>;
destinationOwners: Record<string, string>;
};
export default function Repository() {
const [repositories, setRepositories] = useState<Repository[]>([]);
const { user } = useAuth();
// What this page showed last time in this tab, shown again while it refreshes.
const [cached] = useState(() => readPageCache<RepositoriesPageCache>(user?.id, "repositories"));
const [repositories, setRepositories] = useState<Repository[]>(cached?.repositories ?? []);
// Organization name -> its destination override, from the list response.
// Kept beside the rows rather than on them: the action endpoints return the
// repositories row alone, and replacing a row would drop the field (#416).
const [organizationDestinations, setOrganizationDestinations] = useState<
Record<string, string>
>({});
const [isInitialLoading, setIsInitialLoading] = useState(true);
const { user } = useAuth();
>(cached?.organizationDestinations ?? {});
// Repository id -> the destination owner/org the mirror will really use.
const [destinationOwners, setDestinationOwners] = useState<
Record<string, string>
>(cached?.destinationOwners ?? {});
const [isInitialLoading, setIsInitialLoading] = useState(!cached);
// True once there are rows to show, from the cache or a fetch.
const hasLoadedRef = useRef(!!cached);
const { registerRefreshCallback, isLiveEnabled } = useLiveRefresh();
const { isGitHubConfigured, isFullyConfigured, hasAnySource, autoMirrorStarred, githubOwner, sources } = useConfigStatus();
const { navigationKey } = useNavigation();
@@ -70,6 +90,7 @@ export default function Repository() {
searchTerm: "",
status: "",
organization: "",
destinationOrganization: "",
owner: "",
sort: "imported-desc",
});
@@ -145,8 +166,10 @@ export default function Repository() {
);
if (response.success) {
hasLoadedRef.current = true;
setRepositories(response.repositories);
setOrganizationDestinations(response.organizationDestinations ?? {});
setDestinationOwners(response.destinationOwners ?? {});
return true;
} else {
// Only show error toast for manual refreshes to avoid spam during live updates
@@ -169,11 +192,22 @@ export default function Repository() {
}, [user?.id, isGitHubConfigured, hasAnySource]); // Only depend on user.id, not entire user object
useEffect(() => {
// Reset loading state when component becomes active
setIsInitialLoading(true);
fetchRepositories(false); // Manual refresh, not live
// With rows from the last visit on screen, refresh them in the background
// instead of going back to the skeleton.
const showingRows = hasLoadedRef.current;
if (!showingRows) setIsInitialLoading(true);
fetchRepositories(showingRows);
}, [fetchRepositories, navigationKey]); // Include navigationKey to trigger on navigation
useEffect(() => {
if (!hasLoadedRef.current) return;
writePageCache<RepositoriesPageCache>(user?.id, "repositories", {
repositories,
organizationDestinations,
destinationOwners,
});
}, [user?.id, repositories, organizationDestinations, destinationOwners]);
// Register with global live refresh system
useEffect(() => {
// Only register for live refresh when a source can own repositories
@@ -899,14 +933,22 @@ export default function Repository() {
new Set(
repositories.map((repo) => repo.owner).filter((v): v is string => !!v)
)
).sort();
).sort(compareNames);
const orgOptions = Array.from(
new Set(
repositories
.map((repo) => repo.organization)
.filter((v): v is string => !!v)
)
).sort();
).sort(compareNames);
// The destination owner each row resolves to, not the source organization.
const destinationOptions = Array.from(
new Set(
repositories
.map((repo) => (repo.id ? destinationOwners[repo.id] : undefined))
.filter((v): v is string => !!v)
)
).sort(compareNames);
const handleConfirmDuplicateRepository = async () => {
if (!duplicateRepoCandidate) {
@@ -1057,8 +1099,19 @@ export default function Repository() {
const actionCounts = getActionCounts();
// Check if any filters are active
const hasActiveFilters = !!(filter.owner || filter.organization || filter.status);
const activeFilterCount = [filter.owner, filter.organization, filter.status, filter.hasOverrides].filter(Boolean).length;
const hasActiveFilters = !!(
filter.owner ||
filter.organization ||
filter.destinationOrganization ||
filter.status
);
const activeFilterCount = [
filter.owner,
filter.organization,
filter.destinationOrganization,
filter.status,
filter.hasOverrides,
].filter(Boolean).length;
// Clear all filters
const clearFilters = () => {
@@ -1066,6 +1119,7 @@ export default function Repository() {
searchTerm: filter.searchTerm,
status: "",
organization: "",
destinationOrganization: "",
owner: "",
hasOverrides: "",
sort: filter.sort || "imported-desc",
@@ -1171,6 +1225,25 @@ export default function Repository() {
/>
</div>
{/* Destination Filter */}
<div className="space-y-2">
<label className="text-sm font-medium flex items-center gap-2">
<span className="text-muted-foreground">By</span> Destination
{filter.destinationOrganization && (
<span className="ml-auto text-xs text-muted-foreground">
Selected
</span>
)}
</label>
<DestinationCombobox
options={destinationOptions}
value={filter.destinationOrganization || ""}
onChange={(destinationOrganization: string) =>
setFilter((prev) => ({ ...prev, destinationOrganization }))
}
/>
</div>
{/* Status Filter */}
<div className="space-y-2">
<label className="text-sm font-medium flex items-center gap-2">
@@ -1616,7 +1689,9 @@ export default function Repository() {
<RepositoryTable
repositories={repositories}
organizationDestinations={organizationDestinations}
isLoading={isInitialLoading || !connected}
destinationOwners={destinationOwners}
destinationOptions={destinationOptions}
isLoading={isInitialLoading || (!connected && !hasLoadedRef.current)}
filter={filter}
setFilter={setFilter}
onMirror={handleMirrorRepo}
@@ -22,6 +22,7 @@ type ComboboxProps = {
onChange: (value: string) => void;
placeholder?: string;
label?: string;
className?: string;
};
/** These stay comboboxes rather than Selects because the owner and
@@ -35,6 +36,7 @@ function FilterCombobox({
emptyLabel,
searchPlaceholder,
emptyMessage,
className,
}: ComboboxProps & {
emptyLabel: string;
searchPlaceholder: string;
@@ -52,7 +54,7 @@ function FilterCombobox({
type="button"
role="combobox"
aria-expanded={open}
className={cn(selectTriggerClassName, "h-10 w-full lg:w-50")}
className={cn(selectTriggerClassName, "h-10 w-full lg:w-50", className)}
>
<span className={cn("truncate", !value && "text-muted-foreground")}>
{value || emptyLabel}
@@ -129,3 +131,14 @@ export function OrganizationCombobox(props: ComboboxProps) {
/>
);
}
export function DestinationCombobox(props: ComboboxProps) {
return (
<FilterCombobox
{...props}
emptyLabel="All destinations"
searchPlaceholder="Search destinations..."
emptyMessage="No destinations found."
/>
);
}
+462 -255
View File
@@ -1,4 +1,4 @@
import { useEffect, useMemo, useRef, useState } from "react";
import { memo, useCallback, useEffect, useMemo, useRef, useState } from "react";
import { SourceIcon, repositorySourceInfo } from "@/components/source/SourceIcon";
import { DestinationIcon, destinationInfo } from "@/components/destination/DestinationIcon";
import {
@@ -21,6 +21,8 @@ import { buildGiteaWebUrl } from "@/lib/gitea-url";
import type { FilterParams } from "@/types/filter";
import { Skeleton } from "@/components/ui/skeleton";
import { useGiteaConfig } from "@/hooks/useGiteaConfig";
import { useMediaQuery } from "@/hooks/useMediaQuery";
import { usePageVirtualizer } from "@/hooks/usePageVirtualizer";
import { Checkbox } from "@/components/ui/checkbox";
import {
Tooltip,
@@ -30,11 +32,13 @@ import {
} from "@/components/ui/tooltip";
import { toast } from "sonner";
import { InlineDestinationEditor, type DestinationUpdateOptions } from "./InlineDestinationEditor";
import { DestinationCombobox } from "./RepositoryComboboxes";
import { MarqueeText, MarqueeTrigger } from "@/components/ui/marquee-text";
import { MirrorOverridesDialog } from "@/components/config/MirrorOverridesDialog";
import {
hasMirrorOverrides,
mirrorOptionsToFlags,
normalizeIssueState,
normalizeReleaseAssetLimit,
normalizeReleaseLimit,
normalizeReleaseTagPatterns,
@@ -72,6 +76,10 @@ interface RepositoryTableProps {
repositories: Repository[];
/** Organization name -> its destination override (#416). */
organizationDestinations?: Record<string, string>;
/** Repository id -> the destination owner/org the mirror will use. */
destinationOwners?: Record<string, string>;
/** The values of `destinationOwners`, sorted, for the destination filter. */
destinationOptions?: string[];
isLoading: boolean;
filter: FilterParams;
setFilter: (filter: FilterParams) => void;
@@ -115,6 +123,8 @@ function getTableSorting(sortOrder: string | undefined): SortingState {
export default function RepositoryTable({
repositories,
organizationDestinations,
destinationOwners,
destinationOptions = [],
isLoading,
filter,
setFilter,
@@ -130,7 +140,6 @@ export default function RepositoryTable({
onApproveSync,
onDismissSync,
}: RepositoryTableProps) {
const tableParentRef = useRef<HTMLDivElement>(null);
const lastSelectedIndexRef = useRef<number | null>(null);
const { giteaConfig, advancedOptions, mirrorOptions } = useGiteaConfig();
const [overridesTarget, setOverridesTarget] = useState<Repository | null>(null);
@@ -197,6 +206,8 @@ export default function RepositoryTable({
if (orgIncludeTags !== undefined) globals.releaseIncludeTags = orgIncludeTags;
const orgExcludeTags = normalizeReleaseTagPatterns(orgOverrides.releaseExcludeTags);
if (orgExcludeTags !== undefined) globals.releaseExcludeTags = orgExcludeTags;
const orgIssueState = normalizeIssueState(orgOverrides.mirrorIssueState);
if (orgIssueState !== undefined) globals.mirrorIssueState = orgIssueState;
return globals;
}, [mirrorOptions, orgOverrides]);
@@ -303,6 +314,7 @@ export default function RepositoryTable({
filter.status,
filter.owner,
filter.organization,
filter.destinationOrganization,
filter.hasOverrides,
// val?.toString() is undefined for an unset filter, and undefined !== ""
// is true, so this used to report a filter on a completely clean view.
@@ -323,12 +335,25 @@ export default function RepositoryTable({
next.push({ id: "organization", value: filter.organization });
}
if (filter.destinationOrganization) {
next.push({
id: "destinationOrganization",
value: filter.destinationOrganization,
});
}
if (filter.hasOverrides) {
next.push({ id: "hasOverrides", value: filter.hasOverrides });
}
return next;
}, [filter.status, filter.owner, filter.organization, filter.hasOverrides]);
}, [
filter.status,
filter.owner,
filter.organization,
filter.destinationOrganization,
filter.hasOverrides,
]);
const sorting = useMemo(() => getTableSorting(filter.sort), [filter.sort]);
@@ -348,6 +373,13 @@ export default function RepositoryTable({
accessorFn: (row) => row.organization ?? "",
filterFn: "equalsString",
},
{
id: "destinationOrganization",
accessorFn: (row) =>
row.id ? destinationOwners?.[row.id] ?? "" : "",
filterFn: "equalsString",
enableGlobalFilter: false,
},
{
id: "status",
accessorFn: (row) => row.status,
@@ -373,7 +405,7 @@ export default function RepositoryTable({
enableColumnFilter: false,
},
],
[]
[destinationOwners]
);
const table = useReactTable({
@@ -389,15 +421,26 @@ export default function RepositoryTable({
getSortedRowModel: getSortedRowModel(),
});
const visibleRepositories = table
.getRowModel()
.rows.map((row) => row.original);
// The row model only changes with the data, filters or sort, so the list
// keeps its identity across other renders.
const rowModel = table.getRowModel();
const visibleRepositories = useMemo(
() => rowModel.rows.map((row) => row.original),
[rowModel]
);
const rowVirtualizer = useVirtualizer({
count: visibleRepositories.length,
getScrollElement: () => tableParentRef.current,
estimateSize: () => 65,
overscan: 5,
// Only the layout on screen is rendered: a hidden list still built every
// card. The cards are virtualized against the page scroll like the rows.
const isDesktop = useMediaQuery("(min-width: 1024px)");
// A new getItemKey makes the virtualizer re-measure every card.
const getCardKey = useCallback(
(index: number) => visibleRepositories[index]?.id ?? index,
[visibleRepositories]
);
const { listRef: cardListRef, virtualizer: cardVirtualizer } = usePageVirtualizer({
count: isDesktop ? 0 : visibleRepositories.length,
estimateSize: () => 280,
getItemKey: getCardKey,
});
// Selection handlers
@@ -456,14 +499,34 @@ export default function RepositoryTable({
visibleRepositories.every((repo) => repo.id && selectedRepoIds.has(repo.id));
const isPartiallySelected = selectedRepoIds.size > 0 && !isAllSelected;
// Mobile card layout for repository
const RepositoryCard = ({ repo, index }: { repo: Repository; index: number }) => {
// What every desktop row shares. Rows re-render when the table does, but a
// scroll only renders the rows coming into view.
const rowContext: RepositoryRowContext = {
giteaConfig,
destination,
getGiteaRepoUrl,
onSelect: handleSelectRepo,
onShiftSelect: handleShiftRangeSelect,
onUpdateDestination: handleUpdateDestination,
onEditMirrorOptions: setOverridesTarget,
onMirror,
onSync,
onRetry,
onSkip,
onDelete,
onApproveSync,
onDismissSync,
};
// Mobile card layout for repository. A render function rather than a
// component declared in here, which React would remount on every render.
const renderRepositoryCard = (repo: Repository, index: number) => {
const isLoading = repo.id ? loadingRepoIds.has(repo.id) : false;
const isSelected = repo.id ? selectedRepoIds.has(repo.id) : false;
const giteaUrl = getGiteaRepoUrl(repo);
return (
<Card className="mb-3">
<Card>
<CardContent className="p-4">
<div className="flex flex-col gap-3">
{/* Header: checkbox, name, Custom badge and menu share one
@@ -751,7 +814,9 @@ export default function RepositoryTable({
searchTerm: "",
status: "",
organization: "",
destinationOrganization: "",
owner: "",
hasOverrides: "",
sort: filter.sort || "imported-desc",
})
}
@@ -762,6 +827,17 @@ export default function RepositoryTable({
</div>
<div className="hidden flex-wrap items-center gap-2 lg:flex">
{/* Here rather than beside the owner and organization filters, which
leave no room for it at 1280px and below. */}
<DestinationCombobox
className="h-9"
options={destinationOptions}
value={filter.destinationOrganization || ""}
onChange={(destinationOrganization: string) =>
setFilter({ ...filter, destinationOrganization })
}
/>
<Select
value={filter.status || "all"}
onValueChange={(value) =>
@@ -931,6 +1007,7 @@ export default function RepositoryTable({
) : (
<div className="flex min-h-0 flex-1 flex-col">
{/* Mobile card view: flows in the page scroll rather than its own. */}
{!isDesktop && (
<div className="lg:hidden pb-20">
{/* Select all checkbox */}
<div className="flex items-center gap-3 mb-3 p-3 bg-muted/50 rounded-md">
@@ -945,13 +1022,29 @@ export default function RepositoryTable({
</span>
</div>
{/* Repository cards */}
{visibleRepositories.map((repo, index) => (
<RepositoryCard key={repo.id} repo={repo} index={index} />
))}
{/* Repository cards, only the ones near the viewport */}
<div ref={cardListRef} className="relative" style={{ height: cardVirtualizer.getTotalSize() }}>
{cardVirtualizer.getVirtualItems().map((item) => {
const repo = visibleRepositories[item.index];
if (!repo) return null;
return (
<div
key={item.key}
data-index={item.index}
ref={cardVirtualizer.measureElement}
className="absolute left-0 top-0 w-full pb-3"
style={{ transform: `translateY(${item.start - cardVirtualizer.options.scrollMargin}px)` }}
>
{renderRepositoryCard(repo, item.index)}
</div>
);
})}
</div>
</div>
)}
{/* Desktop table view */}
{isDesktop && (
<div className="hidden lg:flex min-h-0 flex-1 flex-col border rounded-md">
{/* Table header */}
<div className="h-[45px] flex items-center justify-between border-b bg-muted/50">
@@ -981,224 +1074,16 @@ export default function RepositoryTable({
</div>
</div>
{/* Table body wrapper (for a parent in virtualization) */}
<div
ref={tableParentRef}
className="flex min-h-0 flex-1 flex-col overflow-y-auto"
>
<div
style={{
height: `${rowVirtualizer.getTotalSize()}px`,
position: "relative",
}}
>
{rowVirtualizer.getVirtualItems().map((virtualRow) => {
const repo = visibleRepositories[virtualRow.index];
if (!repo) return null;
const isLoading = loadingRepoIds.has(repo.id ?? "");
return (
<div
key={virtualRow.key}
ref={rowVirtualizer.measureElement}
style={{
position: "absolute",
top: 0,
left: 0,
transform: `translateY(${virtualRow.start}px)`,
width: "100%",
}}
data-index={virtualRow.index}
className="h-[65px] flex items-center justify-between bg-transparent border-b hover:bg-muted/50"
>
{/* Checkbox */}
<div
className="h-full p-3 flex items-center justify-center flex-[0.3]"
onClickCapture={(e) => {
if (e.shiftKey && repo.id) {
e.stopPropagation();
handleShiftRangeSelect(virtualRow.index);
}
}}
>
<Checkbox
checked={repo.id ? selectedRepoIds.has(repo.id) : false}
onCheckedChange={(checked) => repo.id && handleSelectRepo(repo.id, !!checked, virtualRow.index)}
aria-label={`Select ${repo.name}`}
/>
</div>
{/* Repository */}
<MarqueeTrigger className="h-full py-3 flex items-center gap-2 flex-[2.3] min-w-0">
<div className="flex-1 min-w-0">
<div className="font-medium flex items-center gap-1 min-w-0">
<MarqueeText>{repo.name}</MarqueeText>
{repo.isStarred && (
<Star className="h-3 w-3 shrink-0 fill-yellow-500 text-yellow-500" />
)}
</div>
<MarqueeText className="text-xs text-muted-foreground">
{repo.fullName}
</MarqueeText>
</div>
{repo.isPrivate && (
<span className="ml-2 shrink-0 rounded-full bg-muted px-2 py-0.5 text-xs">
Private
</span>
)}
{repo.isForked && (
<span className="ml-2 shrink-0 rounded-full bg-muted px-2 py-0.5 text-xs">
Fork
</span>
)}
{hasMirrorOverrides(repo.mirrorOverrides) && (
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<span className="ml-2 shrink-0 rounded-full bg-amber-500/10 px-2 py-0.5 text-xs text-amber-600 dark:text-amber-400">
Custom
</span>
</TooltipTrigger>
<TooltipContent>
This repository overrides the global mirror
options
</TooltipContent>
</Tooltip>
</TooltipProvider>
)}
</MarqueeTrigger>
{/* Owner */}
<div className="h-full p-3 flex items-center flex-[1]">
<p className="text-sm">{repo.owner}</p>
</div>
{/* Organization */}
<div className="h-full p-3 flex items-center flex-[1]">
<InlineDestinationEditor
repository={repo}
giteaConfig={giteaConfig}
organizationDestination={
repo.organization
? organizationDestinations?.[repo.organization]
: undefined
}
onUpdate={handleUpdateDestination}
isUpdating={loadingRepoIds.has(repo.id ?? "")}
/>
</div>
{/* Last Mirrored */}
<div className="h-full p-3 flex items-center flex-[1]">
<p className="text-sm">
{formatLastSyncTime(repo.lastMirrored ?? null)}
</p>
</div>
{/* Status */}
<div className="h-full p-3 flex items-center flex-[1]">
{repo.status === "failed" && repo.errorMessage ? (
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<Badge
variant="destructive"
className="cursor-help capitalize"
>
{repo.status}
</Badge>
</TooltipTrigger>
<TooltipContent className="max-w-xs">
<p className="text-sm">{repo.errorMessage}</p>
</TooltipContent>
</Tooltip>
</TooltipProvider>
) : (
<Badge
className={`capitalize
${repo.status === 'imported' ? 'bg-yellow-500/10 text-yellow-600 hover:bg-yellow-500/20 dark:text-yellow-400' :
repo.status === 'mirrored' || repo.status === 'synced' ? 'bg-green-500/10 text-green-600 hover:bg-green-500/20 dark:text-green-400' :
repo.status === 'mirroring' || repo.status === 'syncing' ? 'bg-blue-500/10 text-blue-600 hover:bg-blue-500/20 dark:text-blue-400' :
repo.status === 'failed' ? 'bg-red-500/10 text-red-600 hover:bg-red-500/20 dark:text-red-400' :
repo.status === 'ignored' ? 'bg-gray-500/10 text-gray-600 hover:bg-gray-500/20 dark:text-gray-400' :
repo.status === 'skipped' ? 'bg-orange-500/10 text-orange-600 hover:bg-orange-500/20 dark:text-orange-400' :
repo.status === 'pending-approval' ? 'bg-amber-500/10 text-amber-600 hover:bg-amber-500/20 dark:text-amber-400' :
'bg-muted hover:bg-muted/80'}`}
variant="secondary"
>
{repo.status}
</Badge>
)}
</div>
{/* Actions */}
<div className="h-full p-3 flex items-center justify-start flex-[1]">
<RepoActionButton
repo={{ id: repo.id ?? "", status: repo.status }}
isLoading={isLoading}
onMirror={() => onMirror({ repoId: repo.id ?? "" })}
onSync={() => onSync({ repoId: repo.id ?? "" })}
onRetry={() => onRetry({ repoId: repo.id ?? "" })}
onSkip={(skip) => onSkip({ repoId: repo.id ?? "", skip })}
onDelete={onDelete && repo.id ? () => onDelete(repo.id as string) : undefined}
onApproveSync={onApproveSync ? () => onApproveSync({ repoId: repo.id ?? "" }) : undefined}
onDismissSync={onDismissSync ? () => onDismissSync({ repoId: repo.id ?? "" }) : undefined}
onEditMirrorOptions={repo.id ? () => setOverridesTarget(repo) : undefined}
/>
</div>
{/* Links */}
<div className="h-full p-3 flex items-center justify-center gap-x-2 flex-[0.8]">
{(() => {
const giteaUrl = getGiteaRepoUrl(repo);
// Determine tooltip based on status and configuration
let tooltip: string;
if (!giteaConfig?.url) {
tooltip = `${destination.label} not configured`;
} else if (repo.status === 'imported') {
tooltip = `Repository not yet mirrored to ${destination.label}`;
} else if (repo.status === 'failed') {
tooltip = "Repository mirroring failed";
} else if (repo.status === 'mirroring') {
tooltip = `Repository is being mirrored to ${destination.label}`;
} else if (giteaUrl) {
tooltip = `View on ${destination.label}`;
} else {
tooltip = `${destination.label} repository not available`;
}
return giteaUrl ? (
<Button variant="ghost" size="icon" asChild>
<a
href={giteaUrl}
target="_blank"
rel="noopener noreferrer"
title={tooltip}
>
<DestinationIcon provider={destination.provider} className="h-4 w-4" />
</a>
</Button>
) : (
<Button variant="ghost" size="icon" disabled title={tooltip}>
<DestinationIcon provider={destination.provider} className="h-4 w-4" />
</Button>
);
})()}
<Button variant="ghost" size="icon" asChild>
<a
href={repo.url}
target="_blank"
rel="noopener noreferrer"
title={`View on ${repositorySourceInfo(repo).label}`}
>
<SourceIcon provider={repositorySourceInfo(repo).provider} className="h-4 w-4" />
</a>
</Button>
</div>
</div>
);
})}
</div>
</div>
<RepositoryTableRows
repositories={visibleRepositories}
selectedRepoIds={selectedRepoIds}
loadingRepoIds={loadingRepoIds}
organizationDestinations={organizationDestinations}
destinationOwners={destinationOwners}
context={rowContext}
/>
</div>
)}
</div>
)}
@@ -1231,6 +1116,308 @@ export default function RepositoryTable({
);
}
type RepositoryRowContext = {
giteaConfig: ReturnType<typeof useGiteaConfig>["giteaConfig"];
destination: ReturnType<typeof destinationInfo>;
getGiteaRepoUrl: (repository: Repository) => string | null;
onSelect: (repoId: string, checked: boolean, index: number) => void;
onShiftSelect: (index: number) => void;
onUpdateDestination: (
repoId: string,
newDestination: string | null,
options?: DestinationUpdateOptions
) => Promise<void>;
onEditMirrorOptions: (repository: Repository) => void;
} & Pick<
RepositoryTableProps,
"onMirror" | "onSync" | "onRetry" | "onSkip" | "onDelete" | "onApproveSync" | "onDismissSync"
>;
/**
* The desktop table body. It owns the virtualizer so a scroll renders only
* this list, not the filters and header above it, and the rows are memoized so
* only the ones scrolling into view render.
*/
function RepositoryTableRows({
repositories,
selectedRepoIds,
loadingRepoIds,
organizationDestinations,
destinationOwners,
context,
}: {
repositories: Repository[];
selectedRepoIds: Set<string>;
loadingRepoIds: Set<string>;
organizationDestinations?: Record<string, string>;
destinationOwners?: Record<string, string>;
context: RepositoryRowContext;
}) {
const scrollRef = useRef<HTMLDivElement>(null);
// A new getItemKey makes the virtualizer re-measure every row, so it only
// changes with the list.
const getItemKey = useCallback(
(index: number) => repositories[index]?.id ?? index,
[repositories]
);
const virtualizer = useVirtualizer({
count: repositories.length,
getScrollElement: () => scrollRef.current,
estimateSize: () => 65,
overscan: 5,
getItemKey,
});
return (
<div ref={scrollRef} className="flex min-h-0 flex-1 flex-col overflow-y-auto">
<div
style={{
height: `${virtualizer.getTotalSize()}px`,
position: "relative",
}}
>
{virtualizer.getVirtualItems().map((virtualRow) => {
const repo = repositories[virtualRow.index];
if (!repo) return null;
return (
<RepositoryTableRow
key={virtualRow.key}
repo={repo}
index={virtualRow.index}
start={virtualRow.start}
measureRef={virtualizer.measureElement}
selected={repo.id ? selectedRepoIds.has(repo.id) : false}
loading={loadingRepoIds.has(repo.id ?? "")}
organizationDestination={
repo.organization ? organizationDestinations?.[repo.organization] : undefined
}
resolvedDestination={repo.id ? destinationOwners?.[repo.id] : undefined}
context={context}
/>
);
})}
</div>
</div>
);
}
const RepositoryTableRow = memo(function RepositoryTableRow({
repo,
index,
start,
measureRef,
selected,
loading,
organizationDestination,
resolvedDestination,
context,
}: {
repo: Repository;
index: number;
start: number;
measureRef: (element: HTMLDivElement | null) => void;
selected: boolean;
loading: boolean;
organizationDestination?: string;
resolvedDestination?: string;
context: RepositoryRowContext;
}) {
return (
<div
ref={measureRef}
style={{
position: "absolute",
top: 0,
left: 0,
transform: `translateY(${start}px)`,
width: "100%",
}}
data-index={index}
className="h-[65px] flex items-center justify-between bg-transparent border-b hover:bg-muted/50"
>
{/* Checkbox */}
<div
className="h-full p-3 flex items-center justify-center flex-[0.3]"
onClickCapture={(e) => {
if (e.shiftKey && repo.id) {
e.stopPropagation();
context.onShiftSelect(index);
}
}}
>
<Checkbox
checked={selected}
onCheckedChange={(checked) => repo.id && context.onSelect(repo.id, !!checked, index)}
aria-label={`Select ${repo.name}`}
/>
</div>
{/* Repository */}
<MarqueeTrigger className="h-full py-3 flex items-center gap-2 flex-[2.3] min-w-0">
<div className="flex-1 min-w-0">
<div className="font-medium flex items-center gap-1 min-w-0">
<MarqueeText>{repo.name}</MarqueeText>
{repo.isStarred && (
<Star className="h-3 w-3 shrink-0 fill-yellow-500 text-yellow-500" />
)}
</div>
<MarqueeText className="text-xs text-muted-foreground">
{repo.fullName}
</MarqueeText>
</div>
{repo.isPrivate && (
<span className="ml-2 shrink-0 rounded-full bg-muted px-2 py-0.5 text-xs">
Private
</span>
)}
{repo.isForked && (
<span className="ml-2 shrink-0 rounded-full bg-muted px-2 py-0.5 text-xs">
Fork
</span>
)}
{hasMirrorOverrides(repo.mirrorOverrides) && (
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<span className="ml-2 shrink-0 rounded-full bg-amber-500/10 px-2 py-0.5 text-xs text-amber-600 dark:text-amber-400">
Custom
</span>
</TooltipTrigger>
<TooltipContent>
This repository overrides the global mirror
options
</TooltipContent>
</Tooltip>
</TooltipProvider>
)}
</MarqueeTrigger>
{/* Owner */}
<div className="h-full p-3 flex items-center flex-[1]">
<p className="text-sm">{repo.owner}</p>
</div>
{/* Organization */}
<div className="h-full p-3 flex items-center flex-[1]">
<InlineDestinationEditor
repository={repo}
giteaConfig={context.giteaConfig}
organizationDestination={organizationDestination}
resolvedDestination={resolvedDestination}
onUpdate={context.onUpdateDestination}
isUpdating={loading}
/>
</div>
{/* Last Mirrored */}
<div className="h-full p-3 flex items-center flex-[1]">
<p className="text-sm">
{formatLastSyncTime(repo.lastMirrored ?? null)}
</p>
</div>
{/* Status */}
<div className="h-full p-3 flex items-center flex-[1]">
{repo.status === "failed" && repo.errorMessage ? (
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<Badge
variant="destructive"
className="cursor-help capitalize"
>
{repo.status}
</Badge>
</TooltipTrigger>
<TooltipContent className="max-w-xs">
<p className="text-sm">{repo.errorMessage}</p>
</TooltipContent>
</Tooltip>
</TooltipProvider>
) : (
<Badge
className={`capitalize
${repo.status === 'imported' ? 'bg-yellow-500/10 text-yellow-600 hover:bg-yellow-500/20 dark:text-yellow-400' :
repo.status === 'mirrored' || repo.status === 'synced' ? 'bg-green-500/10 text-green-600 hover:bg-green-500/20 dark:text-green-400' :
repo.status === 'mirroring' || repo.status === 'syncing' ? 'bg-blue-500/10 text-blue-600 hover:bg-blue-500/20 dark:text-blue-400' :
repo.status === 'failed' ? 'bg-red-500/10 text-red-600 hover:bg-red-500/20 dark:text-red-400' :
repo.status === 'ignored' ? 'bg-gray-500/10 text-gray-600 hover:bg-gray-500/20 dark:text-gray-400' :
repo.status === 'skipped' ? 'bg-orange-500/10 text-orange-600 hover:bg-orange-500/20 dark:text-orange-400' :
repo.status === 'pending-approval' ? 'bg-amber-500/10 text-amber-600 hover:bg-amber-500/20 dark:text-amber-400' :
'bg-muted hover:bg-muted/80'}`}
variant="secondary"
>
{repo.status}
</Badge>
)}
</div>
{/* Actions */}
<div className="h-full p-3 flex items-center justify-start flex-[1]">
<RepoActionButton
repo={{ id: repo.id ?? "", status: repo.status }}
isLoading={loading}
onMirror={() => context.onMirror({ repoId: repo.id ?? "" })}
onSync={() => context.onSync({ repoId: repo.id ?? "" })}
onRetry={() => context.onRetry({ repoId: repo.id ?? "" })}
onSkip={(skip) => context.onSkip({ repoId: repo.id ?? "", skip })}
onDelete={context.onDelete && repo.id ? () => context.onDelete?.(repo.id as string) : undefined}
onApproveSync={context.onApproveSync ? () => context.onApproveSync?.({ repoId: repo.id ?? "" }) : undefined}
onDismissSync={context.onDismissSync ? () => context.onDismissSync?.({ repoId: repo.id ?? "" }) : undefined}
onEditMirrorOptions={repo.id ? () => context.onEditMirrorOptions(repo) : undefined}
/>
</div>
{/* Links */}
<div className="h-full p-3 flex items-center justify-center gap-x-2 flex-[0.8]">
{(() => {
const giteaUrl = context.getGiteaRepoUrl(repo);
// Determine tooltip based on status and configuration
let tooltip: string;
if (!context.giteaConfig?.url) {
tooltip = `${context.destination.label} not configured`;
} else if (repo.status === 'imported') {
tooltip = `Repository not yet mirrored to ${context.destination.label}`;
} else if (repo.status === 'failed') {
tooltip = "Repository mirroring failed";
} else if (repo.status === 'mirroring') {
tooltip = `Repository is being mirrored to ${context.destination.label}`;
} else if (giteaUrl) {
tooltip = `View on ${context.destination.label}`;
} else {
tooltip = `${context.destination.label} repository not available`;
}
return giteaUrl ? (
<Button variant="ghost" size="icon" asChild>
<a
href={giteaUrl}
target="_blank"
rel="noopener noreferrer"
title={tooltip}
>
<DestinationIcon provider={context.destination.provider} className="h-4 w-4" />
</a>
</Button>
) : (
<Button variant="ghost" size="icon" disabled title={tooltip}>
<DestinationIcon provider={context.destination.provider} className="h-4 w-4" />
</Button>
);
})()}
<Button variant="ghost" size="icon" asChild>
<a
href={repo.url}
target="_blank"
rel="noopener noreferrer"
title={`View on ${repositorySourceInfo(repo).label}`}
>
<SourceIcon provider={repositorySourceInfo(repo).provider} className="h-4 w-4" />
</a>
</Button>
</div>
</div>
);
});
function RepoActionButton({
repo,
isLoading,
@@ -1301,6 +1488,7 @@ function RepoActionButton({
let primaryIcon = <></>;
let primaryOnClick = () => {};
let primaryDisabled = isLoading;
let primaryTooltip: string | undefined;
let showPrimaryAction = true;
if (repo.status === "failed") {
@@ -1308,10 +1496,16 @@ function RepoActionButton({
primaryIcon = <RotateCcw className="h-4 w-4" />;
primaryOnClick = onRetry;
} else if (["mirrored", "synced", "syncing", "archived"].includes(repo.status)) {
primaryLabel = repo.status === "archived" ? "Manual Sync" : "Sync";
primaryLabel = "Sync";
primaryIcon = <RefreshCw className="h-4 w-4" />;
primaryOnClick = onSync;
primaryDisabled ||= repo.status === "syncing";
// Archived mirrors are left out of automatic syncs, so this button is the
// way to refresh one. The Archived badge already marks the row; the label
// stays "Sync" so the column lines up with the other rows.
if (repo.status === "archived") {
primaryTooltip = "Automatic syncs are off for archived mirrors. This syncs it now.";
}
} else if (["imported", "mirroring"].includes(repo.status)) {
primaryLabel = "Mirror";
primaryIcon = <FlipHorizontal className="h-4 w-4" />;
@@ -1336,28 +1530,41 @@ function RepoActionButton({
);
}
const primaryButton = (
<Button
variant="ghost"
disabled={primaryDisabled}
onClick={primaryOnClick}
className="min-w-[80px] justify-start rounded-r-none"
>
{isLoading ? (
<>
<RefreshCw className="h-4 w-4 animate-spin mr-1" />
{primaryLabel}
</>
) : (
<>
{primaryIcon}
<span className="ml-1">{primaryLabel}</span>
</>
)}
</Button>
);
// Show primary action with dropdown for additional actions
return (
<DropdownMenu>
<div className="flex">
<Button
variant="ghost"
disabled={primaryDisabled}
onClick={primaryOnClick}
className="min-w-[80px] justify-start rounded-r-none"
>
{isLoading ? (
<>
<RefreshCw className="h-4 w-4 animate-spin mr-1" />
{primaryLabel}
</>
) : (
<>
{primaryIcon}
<span className="ml-1">{primaryLabel}</span>
</>
)}
</Button>
{primaryTooltip ? (
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>{primaryButton}</TooltipTrigger>
<TooltipContent>{primaryTooltip}</TooltipContent>
</Tooltip>
</TooltipProvider>
) : (
primaryButton
)}
<DropdownMenuTrigger asChild>
<Button
variant="ghost"
+142 -13
View File
@@ -1,4 +1,4 @@
import { useCallback, useEffect, useMemo, useState } from "react";
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import {
ArrowDown,
ArrowUp,
@@ -6,11 +6,13 @@ import {
HardDrive,
Info,
Package,
RefreshCw,
ScanSearch,
Search,
} from "lucide-react";
import { toast } from "sonner";
import { StatusCard } from "@/components/dashboard/StatusCard";
import { Button } from "@/components/ui/button";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Skeleton } from "@/components/ui/skeleton";
import {
@@ -30,12 +32,24 @@ import { useAuth } from "@/hooks/useAuth";
import { useGiteaConfig } from "@/hooks/useGiteaConfig";
import { useLiveRefresh } from "@/hooks/useLiveRefresh";
import { useNavigation } from "@/components/layout/MainLayout";
import { apiRequest, cn, formatLastSyncTime } from "@/lib/utils";
import { usePageVirtualizer } from "@/hooks/usePageVirtualizer";
import { readPageCache, writePageCache } from "@/lib/page-cache";
import { apiRequest, cn, formatLastSyncTime, showErrorToast } from "@/lib/utils";
import { buildGiteaWebUrl } from "@/lib/gitea-url";
import { formatBytes } from "@/lib/utils/format-bytes";
import type { StorageOverview, StorageRepository } from "@/lib/repository-storage";
import type { StorageRefreshProgress } from "@/lib/storage-refresh";
type StorageApiResponse = ({ success: true } & StorageOverview) | { success: false; error?: string };
type StorageApiResponse =
| ({ success: true; refresh: StorageRefreshProgress | null } & StorageOverview)
| { success: false; error?: string };
type RefreshApiResponse = { success: true; refresh: StorageRefreshProgress } | { success: false; error?: string };
type StoragePageCache = { overview: StorageOverview; refresh: StorageRefreshProgress | null };
// While Refresh sizes runs, the page polls the overview for its progress.
const REFRESH_POLL_MS = 2000;
type SortKey = "fullName" | "owner" | "destinationSize" | "releaseAssetsSize" | "releaseAssetsCount" | "total" | "checkedAt";
@@ -160,6 +174,45 @@ function compareRepositories(a: StorageRepository, b: StorageRepository, key: So
}
}
function RefreshSizesButton({
refresh,
starting,
onClick,
className,
}: {
refresh: StorageRefreshProgress | null;
starting: boolean;
onClick: () => void;
className?: string;
}) {
const running = !!refresh?.running;
return (
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<Button variant="outline" className={cn("h-10", className)} onClick={onClick} disabled={running || starting}>
<RefreshCw className={cn("h-4 w-4", (running || starting) && "animate-spin")} />
{running ? `Measuring ${refresh.done} of ${refresh.total}` : "Refresh sizes"}
</Button>
</TooltipTrigger>
<TooltipContent className="max-w-xs">
Reads every mirror's size from the destination now instead of waiting for its next sync. Nothing is
asked of the source.
</TooltipContent>
</Tooltip>
</TooltipProvider>
);
}
function announceRefresh(progress: StorageRefreshProgress) {
const parts = [`Measured ${progress.measured} ${progress.measured === 1 ? "repository" : "repositories"}`];
if (progress.missing > 0) parts.push(`${progress.missing} not found on the destination`);
if (progress.failed > 0) parts.push(`${progress.failed} could not be read and kept their sizes`);
const message = parts.join(", ");
if (progress.missing > 0 || progress.failed > 0) toast.warning(message);
else toast.success(message);
}
function StorageSkeleton() {
return (
<div className="flex flex-col gap-y-6">
@@ -188,8 +241,16 @@ export function Storage() {
const { navigationKey } = useNavigation();
const { giteaConfig } = useGiteaConfig();
const [overview, setOverview] = useState<StorageOverview | null>(null);
const [isLoading, setIsLoading] = useState(true);
// What this page showed last time in this tab, shown again while it refreshes.
const [cached] = useState(() => readPageCache<StoragePageCache>(user?.id, "storage"));
const [overview, setOverview] = useState<StorageOverview | null>(cached?.overview ?? null);
const [refresh, setRefresh] = useState<StorageRefreshProgress | null>(cached?.refresh ?? null);
// True once there are numbers to show, from the cache or a fetch.
const hasLoadedRef = useRef(!!cached);
const [startingRefresh, setStartingRefresh] = useState(false);
// Set once this page has seen a run going, so its end is announced once.
const watchingRefreshRef = useRef(false);
const [isLoading, setIsLoading] = useState(!cached);
const [search, setSearch] = useState("");
const [ownerFilter, setOwnerFilter] = useState("all");
const [sort, setSort] = useState<{ key: SortKey; descending: boolean }>({ key: "total", descending: true });
@@ -200,7 +261,15 @@ export function Storage() {
try {
const response = await apiRequest<StorageApiResponse>("/storage", { method: "GET" });
if (response.success) {
hasLoadedRef.current = true;
setOverview(response);
setRefresh(response.refresh);
if (response.refresh?.running) {
watchingRefreshRef.current = true;
} else if (watchingRefreshRef.current && response.refresh) {
watchingRefreshRef.current = false;
announceRefresh(response.refresh);
}
} else if (!isLiveRefresh) {
toast.error(response.error || "Error fetching storage");
}
@@ -216,12 +285,42 @@ export function Storage() {
);
useEffect(() => {
setIsLoading(true);
fetchOverview(false);
// With numbers from the last visit on screen, refresh them in the
// background instead of going back to the skeleton.
const showingOverview = hasLoadedRef.current;
if (!showingOverview) setIsLoading(true);
fetchOverview(showingOverview);
}, [fetchOverview, navigationKey]);
useEffect(() => {
if (overview) writePageCache<StoragePageCache>(user?.id, "storage", { overview, refresh });
}, [user?.id, overview, refresh]);
useEffect(() => registerRefreshCallback(() => fetchOverview(true)), [registerRefreshCallback, fetchOverview]);
useEffect(() => {
if (!refresh?.running) return;
const timer = setInterval(() => fetchOverview(true), REFRESH_POLL_MS);
return () => clearInterval(timer);
}, [refresh?.running, fetchOverview]);
const startRefresh = async () => {
setStartingRefresh(true);
try {
const response = await apiRequest<RefreshApiResponse>("/storage/refresh", { method: "POST" });
if (response.success) {
watchingRefreshRef.current = true;
setRefresh(response.refresh);
} else {
toast.error(response.error || "Could not start Refresh sizes");
}
} catch (error) {
showErrorToast(error, toast);
} finally {
setStartingRefresh(false);
}
};
const handleSort = (key: SortKey) =>
setSort((prev) =>
prev.key === key
@@ -244,6 +343,18 @@ export function Storage() {
return rows;
}, [overview, search, ownerFilter, sort]);
// Only the rows near the viewport are in the DOM. A new getItemKey makes the
// virtualizer re-measure every row, so it only changes with the list.
const getRowKey = useCallback(
(index: number) => visibleRepositories[index]?.id ?? index,
[visibleRepositories]
);
const { listRef: rowListRef, virtualizer: rowVirtualizer } = usePageVirtualizer({
count: visibleRepositories.length,
estimateSize: () => 53,
getItemKey: getRowKey,
});
if (isLoading || !overview) return <StorageSkeleton />;
const { totals } = overview;
@@ -256,8 +367,10 @@ export function Storage() {
<h3 className="text-lg font-medium">No sizes recorded yet</h3>
<p className="text-sm text-muted-foreground mt-1 max-w-md">
Each sync records how much space the mirror takes on the destination, from data the sync
already fetches. Sizes show up here after the next sync.
already fetches. Sizes show up here after the next sync, or now with Refresh sizes, which only
reads the destination.
</p>
<RefreshSizesButton refresh={refresh} starting={startingRefresh} onClick={startRefresh} className="mt-6" />
</CardContent>
</Card>
);
@@ -346,6 +459,12 @@ export function Storage() {
))}
</SelectContent>
</Select>
<RefreshSizesButton
refresh={refresh}
starting={startingRefresh}
onClick={startRefresh}
className="w-full sm:w-auto shrink-0"
/>
</div>
<div className="overflow-x-auto" data-testid="storage-table">
@@ -378,14 +497,23 @@ export function Storage() {
<div className="p-6 text-center text-sm text-muted-foreground">No repositories match.</div>
)}
{visibleRepositories.map((repo) => {
<div ref={rowListRef} className="relative" style={{ height: rowVirtualizer.getTotalSize() }}>
{rowVirtualizer.getVirtualItems().map((item) => {
const repo = visibleRepositories[item.index];
if (!repo) return null;
const destinationUrl = repo.location ? buildGiteaWebUrl(giteaConfig, repo.location) : null;
const heavy = isReleaseHeavy(repo);
return (
<div
key={repo.id}
key={item.key}
data-index={item.index}
ref={rowVirtualizer.measureElement}
data-testid="storage-row"
className="group min-h-[48px] flex items-center border-b last:border-b-0 hover:bg-muted/50 text-sm"
className={cn(
"group absolute left-0 top-0 w-full min-h-[48px] flex items-center hover:bg-muted/50 text-sm",
item.index < visibleRepositories.length - 1 && "border-b"
)}
style={{ transform: `translateY(${item.start - rowVirtualizer.options.scrollMargin}px)` }}
>
<div className="h-full px-3 py-2 flex items-center flex-[2.4] min-w-0 sticky left-0 z-10 bg-background before:absolute before:inset-0 before:-z-10 group-hover:before:bg-muted/50">
<div className="min-w-0">
@@ -442,12 +570,13 @@ export function Storage() {
</div>
);
})}
</div>
</div>
</div>
<p className="text-xs text-muted-foreground">
Sizes are what the destination reports: Git and LFS together, plus the release assets it stores. Each sync
records them from data it already fetches. The Git size is read before the sync pulls, so it trails by one
sync.
records them from data it already fetches, and Refresh sizes reads them from the destination on demand. The
Git size a sync records is read before it pulls, so it trails by one sync.
</p>
</div>
</div>
+4
View File
@@ -77,6 +77,10 @@ export function MarqueeText({ children, className, speed = 60 }: MarqueeTextProp
}
setScrolling(false);
// Still resting (first render, or the text fits): nothing to undo. Reading
// the computed transform forces a style recalc, and a virtualized list
// mounts rows on every scroll.
if (resting) return;
// If the text never actually moved (left before the start delay ended),
// no transition runs and no transitionend will fire — rest immediately
const content = contentRef.current;
+1
View File
@@ -7,6 +7,7 @@ const FILTER_KEYS: (keyof FilterParams)[] = [
"membershipRole",
"owner",
"organization",
"destinationOrganization",
"sort",
"type",
"name",
+14
View File
@@ -0,0 +1,14 @@
import { useSyncExternalStore } from "react";
/** True while the media query matches; false when rendered on the server. */
export function useMediaQuery(query: string): boolean {
return useSyncExternalStore(
(onChange) => {
const list = window.matchMedia(query);
list.addEventListener("change", onChange);
return () => list.removeEventListener("change", onChange);
},
() => window.matchMedia(query).matches,
() => false
);
}
+49
View File
@@ -0,0 +1,49 @@
import { useLayoutEffect, useRef, useState } from "react";
import { useVirtualizer } from "@tanstack/react-virtual";
import { useNavigation } from "@/components/layout/MainLayout";
/**
* A virtualizer for a list that scrolls with the page (MainLayout's main
* section) instead of in a box of its own, so only the rows near the viewport
* are in the DOM. Put `listRef` on the list's container, give it the
* virtualizer's total size as its height, and place each row at
* `start - virtualizer.options.scrollMargin`.
*/
export function usePageVirtualizer({
count,
estimateSize,
overscan = 5,
getItemKey,
}: {
count: number;
estimateSize: (index: number) => number;
overscan?: number;
/** A stable key per row keeps measured heights with their row across filtering. */
getItemKey?: (index: number) => string | number;
}) {
const { pageScrollElement } = useNavigation();
const listRef = useRef<HTMLDivElement>(null);
const [scrollMargin, setScrollMargin] = useState(0);
// Where the list starts in the page's scroll content. Anything above it can
// change height, so it is measured after every render.
useLayoutEffect(() => {
const list = listRef.current;
if (!list || !pageScrollElement) return;
const next = Math.round(
list.getBoundingClientRect().top - pageScrollElement.getBoundingClientRect().top + pageScrollElement.scrollTop
);
if (next !== scrollMargin) setScrollMargin(next);
});
const virtualizer = useVirtualizer({
count,
getScrollElement: () => pageScrollElement,
estimateSize,
overscan,
scrollMargin,
...(getItemKey ? { getItemKey } : {}),
});
return { listRef, virtualizer };
}
+26
View File
@@ -58,6 +58,12 @@ export const backupStrategyEnum = z.enum([
export const releaseTypeEnum = z.enum(["all", "stable"]);
export type ReleaseType = z.infer<typeof releaseTypeEnum>;
// Which source issues are candidates for mirroring. "all" keeps the historical
// behavior; the other values narrow the GitHub listing before anything is
// created in the destination.
export const issueStateFilterEnum = z.enum(["all", "open", "closed"]);
export type IssueStateFilter = z.infer<typeof issueStateFilterEnum>;
/**
* Per-object overrides for the global mirror options in `giteaConfigSchema`.
*
@@ -94,6 +100,11 @@ export type ReleaseType = z.infer<typeof releaseTypeEnum>;
* list of strings and ignores everything else, and an empty list is not a pin:
* clearing the field means inherit.
*
* `mirrorIssueState` follows the same lenient-string rule as `releaseType`
* (`normalizeIssueState` sanitizes it). `mirrorIssueComments` and
* `mirrorIssueMedia` narrow what an issue pass copies; both default to on when
* no tier pins them.
*
* `skipForks` is organization-tier only: it opts an organization out of
* importing and mirroring forked repositories without touching the global
* `githubConfig.skipForks` switch. Null/absent inherits that global value.
@@ -107,6 +118,9 @@ export const mirrorOverridesSchema = z.object({
mirrorReleases: z.boolean().nullable().optional(),
mirrorMetadata: z.boolean().nullable().optional(),
mirrorIssues: z.boolean().nullable().optional(),
mirrorIssueComments: z.boolean().nullable().optional(),
mirrorIssueMedia: z.boolean().nullable().optional(),
mirrorIssueState: z.string().nullable().optional(),
mirrorPullRequests: z.boolean().nullable().optional(),
mirrorLabels: z.boolean().nullable().optional(),
mirrorMilestones: z.boolean().nullable().optional(),
@@ -153,6 +167,13 @@ export const giteaConfigSchema = z.object({
// Upload assets only for the newest N mirrored releases. null/absent means
// every mirrored release gets its assets; 0 means release notes only.
releaseAssetLimit: z.number().nullable().optional(),
// Opt-in cleanup for what that limit left behind. Off by default, because
// the limit alone only gates uploads: attachments a release already has are
// never removed. Turning this on deletes the attachments of every mirrored
// release ranked past `releaseAssetLimit`. Global only - there is no
// per-repository tier for it, the ranks still come from that repository's
// own effective asset limit.
pruneOldReleaseAssets: z.boolean().default(false),
// Which releases are candidates: every one, or only the ones not flagged as a
// pre-release. The limit is applied after filtering.
releaseType: releaseTypeEnum.default("all"),
@@ -164,6 +185,11 @@ export const giteaConfigSchema = z.object({
releaseExcludeTags: z.array(z.string()).default([]),
mirrorMetadata: z.boolean().default(false),
mirrorIssues: z.boolean().default(false),
// Which issues the pass lists, and how much of each one it copies. Defaults
// preserve the behavior from before these options existed.
mirrorIssueState: issueStateFilterEnum.default("all"),
mirrorIssueComments: z.boolean().default(true),
mirrorIssueMedia: z.boolean().default(true),
mirrorPullRequests: z.boolean().default(false),
mirrorLabels: z.boolean().default(false),
mirrorMilestones: z.boolean().default(false),
+152
View File
@@ -0,0 +1,152 @@
/**
* Tests for the destination pager. Gitea 1.27.3 answered `per_page=100` with
* 30 releases and `limit=50` with up to 50, plus `X-Total-Count`; the fake
* destination below pages the same way.
*
* Several test files mock @/lib/http-client, and bun's mock.module leaks across
* files, so this file re-runs itself in an isolated child to get the real one.
*/
import { describe, test, expect, afterEach } from "bun:test";
const CHILD_FLAG = "GM_DESTINATION_LISTING_ISOLATED";
const isChild = !!process.env[CHILD_FLAG];
if (!isChild) {
test("destination listing - isolated child suite", () => {
const res = Bun.spawnSync({
cmd: [process.execPath, "test", import.meta.path],
env: { ...process.env, [CHILD_FLAG]: "1" },
stdout: "pipe",
stderr: "pipe",
});
if (res.exitCode !== 0) {
console.error(res.stdout.toString());
console.error(res.stderr.toString());
}
expect(res.exitCode).toBe(0);
}, 60_000);
}
const { listDestinationPages, listDestinationReleases, listDestinationOwnerRepositories } = isChild
? await import("./destination-listing")
: ({} as typeof import("./destination-listing"));
const realFetch = globalThis.fetch;
let requestedUrls: string[] = [];
/**
* A destination holding `count` items that serves `pageCap` of them per page
* at most, whatever `limit` asks for.
*/
function serve(
count: number,
{ pageCap = 50, totalHeader = true, failPage }: { pageCap?: number; totalHeader?: boolean; failPage?: number } = {}
) {
requestedUrls = [];
globalThis.fetch = (async (input: RequestInfo | URL) => {
const url = String(input);
requestedUrls.push(url);
const params = new URL(url).searchParams;
const page = Number(params.get("page") ?? "1");
if (page === failPage) return new Response("boom", { status: 500, statusText: "Internal Server Error" });
const size = Math.min(Number(params.get("limit") ?? "30"), pageCap);
const items = Array.from({ length: count }, (_, i) => ({ id: i + 1 })).slice((page - 1) * size, page * size);
const headers: Record<string, string> = { "content-type": "application/json" };
if (totalHeader) headers["x-total-count"] = String(count);
return new Response(JSON.stringify(items), { status: 200, headers });
}) as typeof fetch;
}
afterEach(() => {
globalThis.fetch = realFetch;
});
describe.skipIf(!isChild)("listDestinationPages", () => {
test("asks with limit and follows X-Total-Count past pages shorter than asked", async () => {
serve(35, { pageCap: 30 });
const listing = await listDestinationPages({ url: "https://gitea.example.com/api/v1/repos/o/r/releases", token: "t" });
expect(listing.items).toHaveLength(35);
expect(listing.complete).toBe(true);
expect(requestedUrls).toEqual([
"https://gitea.example.com/api/v1/repos/o/r/releases?limit=50&page=1",
"https://gitea.example.com/api/v1/repos/o/r/releases?limit=50&page=2",
]);
});
test("ends on the page that reaches the total, without asking for an empty one", async () => {
serve(100);
const listing = await listDestinationPages({ url: "https://gitea.example.com/x", token: "t" });
expect(listing.items).toHaveLength(100);
expect(requestedUrls).toHaveLength(2);
});
test("without the total header, ends on a short page", async () => {
serve(70, { totalHeader: false });
const listing = await listDestinationPages({ url: "https://gitea.example.com/x", token: "t" });
expect(listing.items).toHaveLength(70);
expect(listing.complete).toBe(true);
expect(requestedUrls).toHaveLength(2);
});
test("ends on an empty page", async () => {
serve(0);
const listing = await listDestinationPages({ url: "https://gitea.example.com/x", token: "t" });
expect(listing).toEqual({ items: [], complete: true, truncated: false });
});
test("a failed page marks the listing incomplete and keeps what came before", async () => {
serve(120, { failPage: 2 });
const listing = await listDestinationPages({ url: "https://gitea.example.com/x", token: "t" });
expect(listing.items).toHaveLength(50);
expect(listing.complete).toBe(false);
});
test("stops at max and says whether more remained", async () => {
serve(120);
const capped = await listDestinationPages({ url: "https://gitea.example.com/x", token: "t", max: 60 });
expect(capped.items).toHaveLength(60);
expect(capped.truncated).toBe(true);
expect(requestedUrls).toHaveLength(2);
serve(60);
const exact = await listDestinationPages({ url: "https://gitea.example.com/x", token: "t", max: 60 });
expect(exact.items).toHaveLength(60);
expect(exact.truncated).toBe(false);
});
test("sends the token", async () => {
let authorization: string | null | undefined;
globalThis.fetch = (async (_input: RequestInfo | URL, init?: RequestInit) => {
authorization = new Headers(init?.headers).get("authorization");
return new Response("[]", { headers: { "content-type": "application/json" } });
}) as typeof fetch;
await listDestinationPages({ url: "https://gitea.example.com/x", token: "secret" });
expect(authorization).toBe("token secret");
});
});
describe.skipIf(!isChild)("listing URLs", () => {
test("releases of one repository", async () => {
serve(1);
await listDestinationReleases({ baseUrl: "https://gitea.example.com/", token: "t", owner: "my org", repo: "app" });
expect(requestedUrls[0]).toBe("https://gitea.example.com/api/v1/repos/my%20org/app/releases?limit=50&page=1");
});
test("repositories of one owner, user or organization", async () => {
serve(1);
await listDestinationOwnerRepositories({ baseUrl: "https://gitea.example.com", token: "t", owner: "archive" });
expect(requestedUrls[0]).toBe("https://gitea.example.com/api/v1/users/archive/repos?limit=50&page=1");
});
});
+108
View File
@@ -0,0 +1,108 @@
/**
* Paged listings from a Gitea or Forgejo destination.
*
* Both servers size pages by `limit`, capped at MAX_RESPONSE_ITEMS (50 unless
* an admin changed it), and ignore `per_page` on most endpoints. Asking for 100
* releases with `per_page` got 30 back, so a loop that stopped on a short page
* only ever saw the newest 30. Pages are asked for with `limit` and a listing
* ends once `X-Total-Count` is reached; without that header, on a short page.
*/
import { httpGet } from "./http-client";
export const DESTINATION_PAGE_SIZE = 50;
// A server that ignores `page` would otherwise be listed forever.
const MAX_PAGES = 1000;
export interface DestinationListing<T> {
items: T[];
/** False when a page failed; `items` then holds the pages before it. */
complete: boolean;
/** True when `max` stopped the listing while more items remained. */
truncated: boolean;
}
/** Every item of a paged destination endpoint. `url` must not carry a query. */
export async function listDestinationPages<T = unknown>({
url,
token,
max,
}: {
url: string;
token: string;
/** Stop once this many items are listed. */
max?: number;
}): Promise<DestinationListing<T>> {
const items: T[] = [];
for (let page = 1; page <= MAX_PAGES; page++) {
const response = await httpGet<T[]>(`${url}?limit=${DESTINATION_PAGE_SIZE}&page=${page}`, {
Authorization: `token ${token}`,
}).catch(() => null);
const batch = response?.data;
if (!Array.isArray(batch)) return { items, complete: false, truncated: false };
if (batch.length === 0) return { items, complete: true, truncated: false };
items.push(...batch);
const totalHeader = response?.headers?.get?.("x-total-count");
const total = totalHeader ? Number(totalHeader) : NaN;
const done = Number.isFinite(total) ? items.length >= total : batch.length < DESTINATION_PAGE_SIZE;
if (max !== undefined && items.length >= max) {
return {
items: items.slice(0, max),
complete: true,
truncated: done ? items.length > max : true,
};
}
if (done) return { items, complete: true, truncated: false };
}
return { items, complete: false, truncated: false };
}
function apiBase(baseUrl: string): string {
return `${baseUrl.replace(/\/+$/, "")}/api/v1`;
}
/** Every release the destination holds for one repository. */
export async function listDestinationReleases<T = unknown>({
baseUrl,
token,
owner,
repo,
max,
}: {
baseUrl: string;
token: string;
owner: string;
repo: string;
max?: number;
}): Promise<DestinationListing<T>> {
return listDestinationPages<T>({
url: `${apiBase(baseUrl)}/repos/${encodeURIComponent(owner)}/${encodeURIComponent(repo)}/releases`,
token,
max,
});
}
/**
* Every repository of one destination owner the token can see, private ones
* included. Works for users and organizations alike.
*/
export async function listDestinationOwnerRepositories<T = unknown>({
baseUrl,
token,
owner,
}: {
baseUrl: string;
token: string;
owner: string;
}): Promise<DestinationListing<T>> {
return listDestinationPages<T>({
url: `${apiBase(baseUrl)}/users/${encodeURIComponent(owner)}/repos`,
token,
});
}
+15
View File
@@ -66,6 +66,7 @@ interface EnvConfig {
mirrorMetadata?: boolean;
releaseLimit?: number;
releaseAssetLimit?: number;
pruneOldReleaseAssets?: boolean;
releaseType?: string;
releaseIncludeTags?: string[];
releaseExcludeTags?: string[];
@@ -197,6 +198,10 @@ function parseEnvConfig(): EnvConfig {
mirrorMetadata: process.env.MIRROR_METADATA === 'true',
releaseLimit: process.env.RELEASE_LIMIT ? parseInt(process.env.RELEASE_LIMIT, 10) : undefined,
releaseAssetLimit: process.env.RELEASE_ASSET_LIMIT ? parseInt(process.env.RELEASE_ASSET_LIMIT, 10) : undefined,
pruneOldReleaseAssets:
process.env.PRUNE_OLD_RELEASE_ASSETS === undefined
? undefined
: process.env.PRUNE_OLD_RELEASE_ASSETS === 'true',
releaseType: process.env.RELEASE_TYPE,
releaseIncludeTags,
releaseExcludeTags,
@@ -397,6 +402,12 @@ export async function initializeConfigFromEnv(): Promise<void> {
normalizeReleaseAssetLimit(envConfig.mirror.releaseAssetLimit) ??
existingConfig?.[0]?.giteaConfig?.releaseAssetLimit ??
null,
// Unset env var keeps what the UI stored; anything but "true" is false,
// matching how the other boolean mirror switches here are read.
pruneOldReleaseAssets:
envConfig.mirror.pruneOldReleaseAssets ??
existingConfig?.[0]?.giteaConfig?.pruneOldReleaseAssets ??
false,
// all | stable. An unusable value (a typo in the env var)
// falls through to the stored value rather than storing a filter.
releaseType:
@@ -421,6 +432,10 @@ export async function initializeConfigFromEnv(): Promise<void> {
: existingConfig?.[0]?.giteaConfig?.pullRequestConcurrency ?? 5,
mirrorMetadata: envConfig.mirror.mirrorMetadata ?? (envConfig.mirror.mirrorIssues || envConfig.mirror.mirrorPullRequests || envConfig.mirror.mirrorLabels || envConfig.mirror.mirrorMilestones) ?? existingConfig?.[0]?.giteaConfig?.mirrorMetadata ?? false,
mirrorIssues: envConfig.mirror.mirrorIssues ?? existingConfig?.[0]?.giteaConfig?.mirrorIssues ?? false,
// No env vars for these; keep what the UI stored across restarts.
mirrorIssueState: existingConfig?.[0]?.giteaConfig?.mirrorIssueState ?? 'all',
mirrorIssueComments: existingConfig?.[0]?.giteaConfig?.mirrorIssueComments ?? true,
mirrorIssueMedia: existingConfig?.[0]?.giteaConfig?.mirrorIssueMedia ?? true,
mirrorPullRequests: envConfig.mirror.mirrorPullRequests ?? existingConfig?.[0]?.giteaConfig?.mirrorPullRequests ?? false,
mirrorLabels: envConfig.mirror.mirrorLabels ?? existingConfig?.[0]?.giteaConfig?.mirrorLabels ?? false,
mirrorMilestones: envConfig.mirror.mirrorMilestones ?? existingConfig?.[0]?.giteaConfig?.mirrorMilestones ?? false,
+8 -5
View File
@@ -532,8 +532,8 @@ export async function syncGiteaRepoEnhanced({
// archiveGiteaRepo applies to mirror repos (see #331 follow-up).
// Repos archived before mirroredLocation was backfilled on rename, or
// any lingering false-positive orphan hit, would otherwise be
// unreachable from "Manual Sync" — the UI's documented way to refresh
// an archived mirror — because the recorded/expected name no longer
// unreachable from a manual Sync (the UI's documented way to refresh
// an archived mirror) because the recorded/expected name no longer
// exists and Gitea returns HTTP 405 for it.
const dependencies = deps ?? (await import("./gitea"));
const expectedOwner = await dependencies.getGiteaRepoOwnerAsync({ config, repository });
@@ -918,8 +918,8 @@ export async function syncGiteaRepoEnhanced({
//
// Skipped for archived repos: archiveGiteaRepo deliberately disabled
// Gitea's own periodic pulling, and the documented contract
// (AutomationSettings.tsx: "Archive ... disables automatic syncs—use
// Manual Sync when you want to refresh") says a Manual Sync refreshes
// (AutomationSettings.tsx: "Archive ... disables automatic syncs ... use
// Sync on the Repositories page to refresh") says a manual Sync refreshes
// once without re-enabling any automatic syncing.
if (config.giteaConfig?.mirrorInterval && !isArchivedRepo && !repoUsesGitHubApp) {
try {
@@ -1166,6 +1166,9 @@ export async function syncGiteaRepoEnhanced({
repository,
giteaOwner: repoOwner,
giteaRepoName: repoName,
issueState: mirrorOptions.mirrorIssueState,
includeComments: mirrorOptions.mirrorIssueComments,
includeMedia: mirrorOptions.mirrorIssueMedia,
syncCursor: metadataState.syncCursors.issues,
passProgress: metadataState.passProgress.issues,
onPassProgress: async (progress) => {
@@ -1277,7 +1280,7 @@ export async function syncGiteaRepoEnhanced({
// Mark repo as "synced" in DB — unless it's archived. The documented
// contract (AutomationSettings.tsx: "Archive ... disables automatic
// syncs—use Manual Sync when you want to refresh") means a Manual Sync
// syncs ... use Sync on the Repositories page to refresh") means a manual Sync
// of an archived repo must NOT re-enroll it into the scheduler's
// auto-sync pool (which selects mirrored/synced/failed/pending) nor
// clear the archived errorMessage annotation shown in the UI; it still
+94 -1
View File
@@ -195,7 +195,11 @@ function fakeOctokit(data: GitHubData) {
const issues = {
listForRepo: async (params: any) => {
calls.push({ endpoint: "issues.listForRepo", params });
return { data: data.issueListing };
// GitHub answers a state filter with only the issues in that state.
const state = params.state ?? "open";
return {
data: data.issueListing.filter((issue) => state === "all" || issue.state === state),
};
},
listComments: async (params: any) => {
calls.push({ endpoint: "issues.listComments", params });
@@ -983,3 +987,92 @@ describe.skipIf(!isChild)("milestone assignment (#460)", () => {
expect(cursor).toBeDefined();
});
});
describe.skipIf(!isChild)("mirrorGitRepoIssuesToGitea content options", () => {
const baseArgs = (octokit: any, extra: Record<string, unknown> = {}) => ({
config,
octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
...extra,
});
it("lists every state and creates only the issues the filter keeps", async () => {
const gitea = fakeGitea([]);
const github = fakeOctokit({
issueListing: [ghIssue(1), ghIssue(2, { state: "closed" })],
});
const cursor = await mirrorGitRepoIssuesToGitea(
baseArgs(github.octokit, { issueState: "open" })
);
expect(github.callsTo("issues.listForRepo")[0]?.params.state).toBe("all");
expect(gitea.createdTitles()).toEqual(["[GH-ISSUE #1] Issue 1"]);
expect(github.callsTo("issues.listComments").map((c) => c.params.issue_number)).toEqual([1]);
expect(cursor?.issueState).toBe("open");
});
it("closes the copy of a mirrored issue that was closed on GitHub", async () => {
const gitea = fakeGitea([{ number: 21, title: "[GH-ISSUE #1] Issue 1" }]);
const github = fakeOctokit({ issueListing: [ghIssue(1, { state: "closed" })] });
await mirrorGitRepoIssuesToGitea(baseArgs(github.octokit, { issueState: "open" }));
const edit = gitea.requests.find(
(r) => r.method === "PATCH" && r.url === `${DEST_API}/issues/21`
);
expect(JSON.parse(edit!.body!).state).toBe("closed");
});
it("runs a full pass when the filter differs from the last pass", async () => {
fakeGitea([{ number: 21, title: "[GH-ISSUE #1] Issue 1" }]);
const changed = fakeOctokit({ issueListing: [] });
await mirrorGitRepoIssuesToGitea(
baseArgs(changed.octokit, { issueState: "all", syncCursor: { ...recentCursor(), issueState: "open" } })
);
expect(changed.callsTo("issues.listForRepo")[0]?.params.since).toBeUndefined();
fakeGitea([{ number: 21, title: "[GH-ISSUE #1] Issue 1" }]);
const same = fakeOctokit({ issueListing: [] });
const cursor = recentCursor();
await mirrorGitRepoIssuesToGitea(
baseArgs(same.octokit, { issueState: "open", syncCursor: { ...cursor, issueState: "open" } })
);
expect(same.callsTo("issues.listForRepo")[0]?.params.since).toBe(expectedSince(cursor));
});
it("skips comments when the comments option is off", async () => {
fakeGitea([]);
const github = fakeOctokit({ issueListing: [ghIssue(1)] });
await mirrorGitRepoIssuesToGitea(baseArgs(github.octokit, { includeComments: false }));
expect(github.callsTo("issues.listComments")).toHaveLength(0);
});
it("keeps media by default and strips it when the media option is off", async () => {
const issue = ghIssue(1, {
body: "Before ![shot](https://user-images.githubusercontent.com/1/a.png) after",
});
const giteaWithMedia = fakeGitea([]);
const withMedia = fakeOctokit({ issueListing: [issue] });
await mirrorGitRepoIssuesToGitea(baseArgs(withMedia.octokit));
const giteaWithoutMedia = fakeGitea([]);
const withoutMedia = fakeOctokit({ issueListing: [issue] });
await mirrorGitRepoIssuesToGitea(
baseArgs(withoutMedia.octokit, { includeMedia: false })
);
expect(giteaWithMedia.createdBodies()[0]).toContain(
"user-images.githubusercontent.com"
);
expect(giteaWithoutMedia.createdBodies()[0]).toContain("[media omitted]");
expect(giteaWithoutMedia.createdBodies()[0]).not.toContain(
"user-images.githubusercontent.com"
);
});
});
+187 -4
View File
@@ -150,8 +150,19 @@ interface DestinationOptions {
destinationReleases?: Array<Record<string, unknown>>;
/** The destination answers the retention listing with an error. */
destinationListingFails?: boolean;
/**
* Serve the listing this many releases per page at most, with
* X-Total-Count, as Gitea does (30 when asked with `per_page`).
*/
destinationPageCap?: number;
/** Destination release ids whose DELETE fails. */
failDeleteIds?: number[];
/** Attachments a destination release has, keyed by release id. */
destinationAssets?: Record<string, Array<{ id: number; name: string; size: number }>>;
/** Attachment ids whose DELETE fails. */
failAssetDeleteIds?: number[];
/** Release ids whose attachment listing answers with an error. */
failAssetListingFor?: number[];
}
function fakeWorld(options: DestinationOptions) {
@@ -205,7 +216,19 @@ function fakeWorld(options: DestinationOptions) {
// Existing attachments of a release.
if (method === "GET" && /\/releases\/\d+\/assets$/.test(url)) {
return json([]);
const releaseId = Number(url.match(/\/releases\/(\d+)\/assets$/)?.[1]);
if (options.failAssetListingFor?.includes(releaseId)) {
return new Response("boom", { status: 500 });
}
return json(options.destinationAssets?.[String(releaseId)] ?? []);
}
// Pruning the attachments of a release that is past the asset limit.
if (method === "DELETE" && /\/releases\/\d+\/assets\/\d+$/.test(url)) {
const assetId = Number(url.slice(url.lastIndexOf("/") + 1));
return options.failAssetDeleteIds?.includes(assetId)
? new Response("boom", { status: 500 })
: new Response(null, { status: 204 });
}
// Uploading an attachment.
@@ -233,9 +256,15 @@ function fakeWorld(options: DestinationOptions) {
// The retention pass lists everything the destination has.
if (method === "GET" && url.startsWith(`${DEST_API}/releases?`)) {
return options.destinationListingFails
? new Response("boom", { status: 500 })
: json(options.destinationReleases ?? []);
if (options.destinationListingFails) return new Response("boom", { status: 500 });
const listed = options.destinationReleases ?? [];
if (!options.destinationPageCap) return json(listed);
const params = new URL(url).searchParams;
const size = Math.min(Number(params.get("limit") ?? "30"), options.destinationPageCap);
const page = Number(params.get("page") ?? "1");
return new Response(JSON.stringify(listed.slice((page - 1) * size, page * size)), {
headers: { "content-type": "application/json", "x-total-count": String(listed.length) },
});
}
throw new Error(`unexpected request: ${method} ${url}`);
@@ -256,6 +285,10 @@ function fakeWorld(options: DestinationOptions) {
requests.filter(
(r) => r.url.startsWith(`${SOURCE_URL}/attachments/`) || r.url.startsWith("https://github.com/")
),
assetListings: () =>
requests.filter((r) => r.method === "GET" && /\/releases\/\d+\/assets$/.test(r.url)),
assetDeletes: () =>
requests.filter((r) => r.method === "DELETE" && /\/releases\/\d+\/assets\/\d+$/.test(r.url)),
};
}
@@ -567,6 +600,39 @@ describe.skipIf(!isChild)("release asset storage (#479)", () => {
expect(storageRecords).toEqual([{ repositoryId: "repo-1", summary: { count: 3, bytes: 1150 } }]);
});
it("lists past the first page when the destination pages below what was asked", async () => {
// 35 releases, served 30 a page: the old per_page=100 loop stopped at 30.
const listed = Array.from({ length: 35 }, (_, i) =>
withAssets(destinationRelease(`v1.${i}.0`, 100 + i, false), [10])
);
const world = fakeWorld({
sourceReleases: [giteaRelease("v2.0.0")],
destinationReleases: listed,
destinationPageCap: 30,
});
await mirrorFromGitea(giteaConnection("source-token"), { releaseLimit: 50 });
expect(storageRecords).toEqual([{ repositoryId: "repo-1", summary: { count: 35, bytes: 350 } }]);
expect(world.requests.filter((r) => r.url.startsWith(`${DEST_API}/releases?`))).toHaveLength(2);
});
it("enforces the release limit across every page", async () => {
const listed = Array.from({ length: 35 }, (_, i) =>
withAssets(destinationRelease(`v1.${i}.0`, 100 + i, false), [10])
);
const world = fakeWorld({
sourceReleases: [giteaRelease("v2.0.0")],
destinationReleases: listed,
destinationPageCap: 30,
});
await mirrorFromGitea(giteaConnection("source-token"), { releaseLimit: 10 });
// The old loop saw 30 of the 35 and deleted 20, leaving 15.
expect(world.requests.filter((r) => r.method === "DELETE")).toHaveLength(25);
});
it("records nothing when the listing fails", async () => {
fakeWorld({
sourceReleases: [giteaRelease("v2.0.0")],
@@ -915,3 +981,120 @@ describe.skipIf(!isChild)("mirrorGitHubReleasesToGitea keeps its own header and
expect(download.authorization).toBe("token ghes-token");
});
});
describe.skipIf(!isChild)("pruning release assets past the asset limit", () => {
function configWithPruning(prune: boolean) {
return {
...config,
giteaConfig: {
...(config as any).giteaConfig,
pruneOldReleaseAssets: prune,
},
} as unknown as Partial<Config>;
}
const destinationReleases = [
{
id: 1,
tag_name: "v3.0.0",
created_at: "2026-03-01T00:00:00Z",
prerelease: false,
assets: [{ id: 11, name: "new.bin", size: 7 }],
},
{
id: 2,
tag_name: "v2.0.0",
created_at: "2026-02-01T00:00:00Z",
prerelease: false,
assets: [{ id: 21, name: "old.bin", size: 5 }],
},
{
id: 3,
tag_name: "v1.0.0",
created_at: "2026-01-01T00:00:00Z",
prerelease: false,
assets: [{ id: 31, name: "older.bin", size: 3 }],
},
];
/** Three releases on both sides; an asset limit of 1 covers only v3.0.0. */
function run(prune: boolean, extra: Partial<DestinationOptions> = {}) {
const world = fakeWorld({
sourceReleases: [
giteaRelease("v3.0.0"),
giteaRelease("v2.0.0"),
giteaRelease("v1.0.0"),
],
destinationReleases,
destinationAssets: {
"1": [{ id: 11, name: "new.bin", size: 7 }],
"2": [{ id: 21, name: "old.bin", size: 5 }],
"3": [{ id: 31, name: "older.bin", size: 3 }],
},
...extra,
});
return mirrorGiteaSourceReleasesToGitea({
config: configWithPruning(prune),
repository,
connection: giteaConnection("source-token"),
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
releaseLimit: 3,
releaseAssetLimit: 1,
}).then(() => world);
}
it("keeps attachments a looser limit uploaded before", async () => {
const world = await run(false);
expect(world.assetListings()).toHaveLength(0);
expect(world.assetDeletes()).toHaveLength(0);
expect(storageRecords.at(-1)?.summary).toEqual({ count: 3, bytes: 15 });
});
it("deletes the attachments of releases past the asset limit when enabled", async () => {
const world = await run(true);
expect(world.assetDeletes().map((r) => r.url)).toEqual([
`${DEST_API}/releases/2/assets/21`,
`${DEST_API}/releases/3/assets/31`,
]);
// The newest release stays inside the limit and keeps its file.
expect(world.assetListings().map((r) => r.url)).toEqual([
`${DEST_API}/releases/2/assets`,
`${DEST_API}/releases/3/assets`,
]);
// The storage summary follows the deletions instead of reporting the old total.
expect(storageRecords.at(-1)?.summary).toEqual({ count: 1, bytes: 7 });
});
it("skips releases the listing shows without attachments", async () => {
// v1.0.0 was pruned on an earlier sync: it is not listed or deleted again.
const world = await run(true, {
destinationReleases: destinationReleases.map((release) =>
release.id === 3 ? { ...release, assets: [] } : release
),
destinationAssets: {
"1": [{ id: 11, name: "new.bin", size: 7 }],
"2": [{ id: 21, name: "old.bin", size: 5 }],
"3": [],
},
});
expect(world.assetListings().map((r) => r.url)).toEqual([
`${DEST_API}/releases/2/assets`,
]);
expect(world.assetDeletes().map((r) => r.url)).toEqual([
`${DEST_API}/releases/2/assets/21`,
]);
expect(storageRecords.at(-1)?.summary).toEqual({ count: 1, bytes: 7 });
});
it("deletes nothing when a release's attachments cannot be listed", async () => {
const world = await run(true, { failAssetListingFor: [2, 3] });
expect(world.assetDeletes()).toHaveLength(0);
expect(storageRecords.at(-1)?.summary).toEqual({ count: 3, bytes: 15 });
});
});
+100 -1
View File
@@ -1,7 +1,12 @@
import { describe, test, expect, mock, beforeEach, afterEach } from "bun:test";
import { Octokit } from "@octokit/rest";
import { repoStatusEnum } from "@/types/Repository";
import { getOrCreateGiteaOrg, getGiteaRepoOwner, getGiteaRepoOwnerAsync } from "./gitea";
import {
getOrCreateGiteaOrg,
getGiteaRepoOwner,
getGiteaRepoOwnerAsync,
resolveDestinationOwner,
} from "./gitea";
import type { Config, Repository, Organization } from "./db/schema";
import { createMockResponse, mockFetch } from "@/tests/mock-fetch";
@@ -751,3 +756,97 @@ describe("getGiteaRepoOwner - Organization Override Tests", () => {
expect(result).toBe("nice-user");
});
});
describe("resolveDestinationOwner", () => {
const config = {
githubConfig: {
owner: "testuser",
username: "testuser",
token: "token",
starredReposOrg: "starred",
starredReposMode: "dedicated-org",
mirrorStrategy: "preserve",
},
giteaConfig: {
defaultOwner: "giteauser",
url: "https://gitea.example.com",
token: "gitea-token",
organization: "github-mirrors",
visibility: "public",
preserveVisibility: false,
},
} as unknown as Partial<Config>;
const repo = (overrides: Partial<Repository> = {}) =>
({
fullName: "acme/demo",
owner: "acme",
organization: "acme",
destinationOrg: null,
isStarred: false,
...overrides,
}) as Repository;
test("uses the starred organization in dedicated mode", () => {
expect(
resolveDestinationOwner({
config,
repository: repo({ isStarred: true }),
})
).toBe("starred");
});
test("preserves the owner namespace for starred repos in preserve-owner mode", () => {
expect(
resolveDestinationOwner({
config: {
...config,
githubConfig: {
...config.githubConfig!,
starredReposMode: "preserve-owner",
},
},
repository: repo({ isStarred: true, owner: "other", organization: undefined }),
})
).toBe("other");
});
test("a repository override wins over the organization override", () => {
expect(
resolveDestinationOwner({
config,
repository: repo({ destinationOrg: "repo-target" }),
organizationDestination: "org-target",
})
).toBe("repo-target");
});
test("the organization override wins over the strategy", () => {
expect(
resolveDestinationOwner({
config,
repository: repo(),
organizationDestination: "org-target",
})
).toBe("org-target");
});
test("preserve keeps the source organization when no override exists", () => {
expect(
resolveDestinationOwner({
config,
repository: repo(),
})
).toBe("acme");
});
test("personal repos of the configured source user go to defaultOwner", () => {
expect(
resolveDestinationOwner({
config,
repository: repo({ owner: "testuser", organization: undefined }),
sourceUsername: "testuser",
})
).toBe("giteauser");
});
});
+385 -154
View File
@@ -7,9 +7,10 @@ import {
import { membershipRoleEnum } from "@/types/organizations";
import { Octokit } from "@octokit/rest";
import type { Config } from "@/types/config";
import type { Organization, ReleaseType, Repository } from "./db/schema";
import type { IssueStateFilter, Organization, ReleaseType, Repository } from "./db/schema";
import { httpPost, httpGet, httpDelete, httpPut, httpPatch } from "./http-client";
import { createMirrorJob } from "./helpers";
import { applyIssueMediaOption, stripIssueMedia } from "./utils/issue-media";
import { db, organizations, repositories } from "./db";
import { eq, and, ne } from "drizzle-orm";
import { decryptConfigTokens } from "./utils/config-encryption";
@@ -37,6 +38,7 @@ import { assertMirrorCredentialUpdateSupported } from "./gitea-mirror-credential
import { persistMetadataPassProgress } from "./metadata-progress-store";
import { summarizeReleaseAssets } from "./repository-storage";
import { recordReleaseAssetStorage } from "./repository-storage-store";
import { listDestinationReleases } from "./destination-listing";
import {
advanceMetadataSyncCursor,
buildPassProgress,
@@ -217,38 +219,36 @@ export const getGiteaRepoOwnerAsync = async ({
throw new Error("User ID is required for organization overrides.");
}
// Check if repository is starred
if (repository.isStarred) {
const starredReposMode = config.githubConfig.starredReposMode || "dedicated-org";
if (starredReposMode === "preserve-owner") {
return repository.organization || repository.owner;
}
return config.githubConfig.starredReposOrg || "starred";
}
// Check for repository-specific override (second highest priority)
if (repository.destinationOrg) {
console.log(`Using repository override: ${repository.fullName} -> ${repository.destinationOrg}`);
return repository.destinationOrg;
}
// Check for organization-specific override
if (repository.organization) {
// The organization override is the only tier that needs a database read,
// and only when the starred mode or a repository override has not already
// decided. Everything else is shared with `resolveDestinationOwner` below,
// which the repositories list uses to filter by the real destination
// without one query per row.
let organizationDestination: string | undefined;
if (!repository.isStarred && !repository.destinationOrg && repository.organization) {
const orgConfig = await getOrganizationConfig({
orgName: repository.organization,
userId: config.userId,
});
if (orgConfig?.destinationOrg) {
console.log(`Using organization override: ${repository.organization} -> ${orgConfig.destinationOrg}`);
return orgConfig.destinationOrg;
}
organizationDestination = orgConfig?.destinationOrg ?? undefined;
}
// For personal repos (not organization repos), fall back to the default strategy
const owner = resolveDestinationOwner({
config,
repository,
organizationDestination,
sourceUsername,
});
// Fall back to existing strategy logic
return getGiteaRepoOwner({ config, repository, sourceUsername });
// Logged here rather than in the resolver, which the repositories list
// runs for every row on every load.
if (!repository.isStarred && repository.destinationOrg) {
console.log(`Using repository override: ${repository.fullName} -> ${repository.destinationOrg}`);
} else if (organizationDestination) {
console.log(`Using organization override: ${repository.organization} -> ${organizationDestination}`);
}
return owner;
};
export const getGiteaRepoOwner = ({
@@ -346,6 +346,53 @@ export const getGiteaRepoOwner = ({
}
};
/**
* Resolve the destination owner with the same precedence the mirror uses:
* starred mode > repository override > organization override > strategy.
*
* The organization override is passed in rather than fetched so callers that
* already hold the organization rows (the repositories list) can resolve many
* repositories without a database round-trip per row.
*/
export const resolveDestinationOwner = ({
config,
repository,
organizationDestination,
sourceUsername,
}: {
config: Partial<Config>;
repository: Repository;
organizationDestination?: string | null;
/** Account name of the repository's own source, when already resolved. */
sourceUsername?: string;
}): string => {
if (!config.githubConfig || !config.giteaConfig) {
throw new Error("GitHub or Gitea config is required.");
}
if (!config.giteaConfig.defaultOwner) {
throw new Error("Gitea username is required.");
}
if (repository.isStarred) {
const starredReposMode = config.githubConfig.starredReposMode || "dedicated-org";
if (starredReposMode === "preserve-owner") {
return repository.organization || repository.owner;
}
return config.githubConfig.starredReposOrg || "starred";
}
if (repository.destinationOrg) {
return repository.destinationOrg;
}
if (repository.organization && organizationDestination) {
return organizationDestination;
}
return getGiteaRepoOwner({ config, repository, sourceUsername });
};
export const isRepoPresentInGitea = async ({
config,
owner,
@@ -1271,6 +1318,9 @@ export const mirrorGithubRepoToGitea = async ({
repository,
giteaOwner: repoOwner,
giteaRepoName: targetRepoName,
issueState: mirrorOptions.mirrorIssueState,
includeComments: mirrorOptions.mirrorIssueComments,
includeMedia: mirrorOptions.mirrorIssueMedia,
syncCursor: metadataState.syncCursors.issues,
passProgress: metadataState.passProgress.issues,
onPassProgress: async (progress) => {
@@ -2094,6 +2144,9 @@ export async function mirrorGitHubRepoToGiteaOrg({
repository,
giteaOwner: orgName,
giteaRepoName: targetRepoName,
issueState: mirrorOptions.mirrorIssueState,
includeComments: mirrorOptions.mirrorIssueComments,
includeMedia: mirrorOptions.mirrorIssueMedia,
syncCursor: metadataState.syncCursors.issues,
passProgress: metadataState.passProgress.issues,
onPassProgress: async (progress) => {
@@ -2808,12 +2861,26 @@ export const mirrorGitRepoIssuesToGitea = async ({
passProgress,
onPassProgress,
milestoneIds,
issueState = "all",
includeComments = true,
includeMedia = true,
}: {
config: Partial<Config>;
octokit: Octokit;
repository: Repository;
giteaOwner: string;
giteaRepoName?: string;
/**
* Which source issues get created. "all" keeps the historical behavior.
* The listing still asks for every state, so an issue that is already
* mirrored keeps following GitHub when it leaves the filter: closing it
* there closes the copy. Nothing is deleted.
*/
issueState?: IssueStateFilter;
/** Copy issue comments. Off skips the per-issue comment listing entirely. */
includeComments?: boolean;
/** Keep images/videos/attachment links in issue bodies and comments. */
includeMedia?: boolean;
/**
* Gitea milestone ids by title from the milestones pass that ran just
* before (#460). Without it the issues' milestones are left untouched.
@@ -2887,9 +2954,19 @@ export const mirrorGitRepoIssuesToGitea = async ({
}
// GitHub numbers this pass has finished, kept if it does not complete.
const completedNumbers = new Set<number>();
// Issues an earlier state filter left out have not changed since, so an
// incremental listing would never return them. A pass under a different
// filter lists everything once.
if (plan.mode === "incremental" && (syncCursor?.issueState ?? "all") !== issueState) {
console.log(
`[Issues] The issue state filter for ${repository.fullName} changed; running a full pass`
);
plan = { mode: "full", reason: "the issue state filter changed" };
}
const finishPass = async () => {
if (resume) await onPassProgress?.(undefined);
return advanceMetadataSyncCursor(syncCursor, plan.mode, passStartedAt);
const cursor = advanceMetadataSyncCursor(syncCursor, plan.mode, passStartedAt);
return issueState === "all" ? cursor : { ...cursor, issueState };
};
const keepPassProgress = async () => {
await onPassProgress?.(buildPassProgress(plan, passStartedAt, completedNumbers, resume));
@@ -3067,6 +3144,7 @@ export const mirrorGitRepoIssuesToGitea = async ({
}
let unchangedIssueCount = 0;
let filteredOutIssueCount = 0;
// Process issues in parallel with concurrency control
let mirroredIssues: unknown[];
@@ -3087,6 +3165,17 @@ export const mirrorGitRepoIssuesToGitea = async ({
return issue;
}
// The state filter only decides what gets created. Not counted as done,
// so a resumed pass under another filter looks at the issue again.
if (
issueState !== "all" &&
issue.state !== issueState &&
!giteaIssueByGitHubNumber.has(issue.number)
) {
filteredOutIssueCount++;
return issue;
}
const githubLabelNames =
issue.labels
?.map((l) => (typeof l === "string" ? l : l.name))
@@ -3132,7 +3221,7 @@ export const mirrorGitRepoIssuesToGitea = async ({
}.`;
const issueMarker = `[GH-ISSUE #${issue.number}]`;
const mirroredTitle = `${issueMarker} ${issue.title}`;
const issueBody = `${issueOriginHeader}\nOriginal GitHub issue: ${issue.html_url}${originalAssignees}\n\n${issue.body ?? ""}`;
const issueBody = `${issueOriginHeader}\nOriginal GitHub issue: ${issue.html_url}${originalAssignees}\n\n${applyIssueMediaOption(issue.body, includeMedia)}`;
const wantedMilestoneId = resolveGiteaMilestoneId(milestoneIds, issue.milestone);
const issuePayload: any = {
@@ -3330,106 +3419,119 @@ export const mirrorGitRepoIssuesToGitea = async ({
});
}
// Clone comments
const comments = await octokit.paginate(
octokit.rest.issues.listComments,
{
owner,
repo,
issue_number: issue.number,
per_page: 100,
},
(res) => res.data
);
// Ensure comments are applied in chronological order to preserve discussion flow
const sortedComments = comments
.slice()
.sort(
(a, b) =>
new Date(a.created_at || 0).getTime() -
new Date(b.created_at || 0).getTime()
if (includeComments) {
// Clone comments
const comments = await octokit.paginate(
octokit.rest.issues.listComments,
{
owner,
repo,
issue_number: issue.number,
per_page: 100,
},
(res) => res.data
);
// Process comments sequentially to preserve historical ordering
if (sortedComments.length > 0) {
const existingComments: any[] = [];
let commentsPage = 1;
const commentsPerPage = 100;
while (true) {
const existingCommentsRes = await httpGet(
`${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues/${targetIssueNumber}/comments?page=${commentsPage}&limit=${commentsPerPage}`,
{
Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
}
// Ensure comments are applied in chronological order to preserve discussion flow
const sortedComments = comments
.slice()
.sort(
(a, b) =>
new Date(a.created_at || 0).getTime() -
new Date(b.created_at || 0).getTime()
);
const pageComments = Array.isArray(existingCommentsRes.data)
? existingCommentsRes.data
: [];
if (!pageComments.length) break;
existingComments.push(...pageComments);
// Use the Link header to decide whether more pages exist.
// See note on the existing-issues pagination above; the
// same Gitea behaviors (MAX_RESPONSE_ITEMS cap and
// repeated-data on out-of-bound pages) apply here.
const commentsLinkHeader =
existingCommentsRes.headers.get("link") || "";
if (!/\brel="next"/.test(commentsLinkHeader)) break;
commentsPage += 1;
}
const mirroredCommentIds = new Set<number>();
const existingCommentBodies = new Set<string>();
for (const existingComment of existingComments) {
const body = String(existingComment.body || "");
if (body) existingCommentBodies.add(body);
const marker = String(existingComment.body || "").match(
/<!--\s*gh-comment-id:(\d+)\s*-->/i
);
if (marker?.[1]) {
const parsed = Number.parseInt(marker[1], 10);
if (Number.isFinite(parsed)) mirroredCommentIds.add(parsed);
}
}
await processWithRetry(
sortedComments,
async (comment) => {
if (mirroredCommentIds.has(comment.id)) {
return comment;
}
const commenter = comment.user?.login ?? "unknown";
const commentDate = formatDateShort(comment.created_at);
const commentHeader = `@${commenter} commented on GitHub${
commentDate ? ` (${commentDate})` : ""
}:`;
const legacyBody = `${commentHeader}\n\n${comment.body ?? ""}`;
const markedBody = `<!-- gh-comment-id:${comment.id} -->\n${legacyBody}`;
if (existingCommentBodies.has(legacyBody) || existingCommentBodies.has(markedBody)) {
return comment;
}
await httpPost(
`${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues/${targetIssueNumber}/comments`,
{
body: markedBody,
},
// Process comments sequentially to preserve historical ordering
if (sortedComments.length > 0) {
const existingComments: any[] = [];
let commentsPage = 1;
const commentsPerPage = 100;
while (true) {
const existingCommentsRes = await httpGet(
`${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues/${targetIssueNumber}/comments?page=${commentsPage}&limit=${commentsPerPage}`,
{
Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
}
);
return comment;
},
{
concurrencyLimit: 1,
maxRetries: 2,
retryDelay: 1000,
onRetry: (_comment, error, attempt) => {
console.log(
`Retrying comment (attempt ${attempt}): ${error.message}`
);
},
const pageComments = Array.isArray(existingCommentsRes.data)
? existingCommentsRes.data
: [];
if (!pageComments.length) break;
existingComments.push(...pageComments);
// Use the Link header to decide whether more pages exist.
// See note on the existing-issues pagination above; the
// same Gitea behaviors (MAX_RESPONSE_ITEMS cap and
// repeated-data on out-of-bound pages) apply here.
const commentsLinkHeader =
existingCommentsRes.headers.get("link") || "";
if (!/\brel="next"/.test(commentsLinkHeader)) break;
commentsPage += 1;
}
);
const mirroredCommentIds = new Set<number>();
const existingCommentBodies = new Set<string>();
for (const existingComment of existingComments) {
const body = String(existingComment.body || "");
if (body) {
existingCommentBodies.add(body);
// When media is off, compare against the stripped form too so a
// comment mirrored before the option existed is not re-created
// as a duplicate with the same text and no media.
if (!includeMedia) {
existingCommentBodies.add(stripIssueMedia(body));
}
}
const marker = String(existingComment.body || "").match(
/<!--\s*gh-comment-id:(\d+)\s*-->/i
);
if (marker?.[1]) {
const parsed = Number.parseInt(marker[1], 10);
if (Number.isFinite(parsed)) mirroredCommentIds.add(parsed);
}
}
await processWithRetry(
sortedComments,
async (comment) => {
if (mirroredCommentIds.has(comment.id)) {
return comment;
}
const commenter = comment.user?.login ?? "unknown";
const commentDate = formatDateShort(comment.created_at);
const commentHeader = `@${commenter} commented on GitHub${
commentDate ? ` (${commentDate})` : ""
}:`;
const legacyBody = `${commentHeader}\n\n${applyIssueMediaOption(comment.body, includeMedia)}`;
const markedBody = `<!-- gh-comment-id:${comment.id} -->\n${legacyBody}`;
if (
existingCommentBodies.has(legacyBody) ||
existingCommentBodies.has(markedBody)
) {
return comment;
}
await httpPost(
`${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues/${targetIssueNumber}/comments`,
{
body: markedBody,
},
{
Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
}
);
return comment;
},
{
concurrencyLimit: 1,
maxRetries: 2,
retryDelay: 1000,
onRetry: (_comment, error, attempt) => {
console.log(
`Retrying comment (attempt ${attempt}): ${error.message}`
);
},
}
);
}
}
completedNumbers.add(issue.number);
@@ -3470,6 +3572,9 @@ export const mirrorGitRepoIssuesToGitea = async ({
`Completed mirroring ${filteredIssues.length} issues for ${repository.fullName}` +
(unchangedIssueCount > 0
? ` (${unchangedIssueCount} unchanged since the last pass, no GitHub calls)`
: "") +
(filteredOutIssueCount > 0
? ` (${filteredOutIssueCount} not ${issueState}, skipped)`
: "")
);
@@ -3806,6 +3911,90 @@ export async function reconcileReleaseAssets({
return { uploaded, failed, skipped };
}
/**
* Delete every attachment one release currently has on the destination.
*
* Used by the opt-in pruning in the retention pass, which removes what the
* asset limit left behind: the uploads a release got before the limit was set
* or lowered. Like `reconcileReleaseAssets` this reads before it writes, and a
* listing that cannot be read deletes nothing, so a transient API error never
* wipes a release that still had its assets.
*
* Returns what was removed so the caller can log it and keep the storage
* totals honest. A release whose deletes all succeeded reports `failed: 0`.
*/
export async function pruneReleaseAttachments({
baseUrl,
token,
repoOwner,
repoName,
giteaReleaseId,
tagName,
fetchImpl,
}: {
baseUrl: string;
token: string;
repoOwner: string;
repoName: string;
giteaReleaseId: number;
tagName: string;
/** Injected in tests; production uses the global fetch. */
fetchImpl?: typeof fetch;
}): Promise<{ removed: number; failed: number }> {
const doFetch: typeof fetch = fetchImpl ?? fetch;
const giteaAuth = { Authorization: `token ${token}` };
const assetsUrl = `${baseUrl}/api/v1/repos/${repoOwner}/${repoName}/releases/${giteaReleaseId}/assets`;
let existingAssets: Array<{ id: number; name: string }>;
try {
const listResponse = await doFetch(assetsUrl, { headers: giteaAuth });
if (!listResponse.ok) {
throw new Error(`HTTP ${listResponse.status} ${listResponse.statusText}`);
}
const body = await listResponse.json();
if (!Array.isArray(body)) {
throw new Error("the assets endpoint did not return a list");
}
existingAssets = body;
} catch (listError) {
console.error(
`[Releases] Could not list the attachments of ${tagName} to prune them: ${
listError instanceof Error ? listError.message : String(listError)
}. Deleted nothing; the next sync retries.`
);
return { removed: 0, failed: 0 };
}
let removed = 0;
let failed = 0;
for (const asset of existingAssets) {
try {
const deleteResponse = await doFetch(`${assetsUrl}/${asset.id}`, {
method: "DELETE",
headers: giteaAuth,
});
// 404 means it is already gone, which is the state we wanted.
if (!deleteResponse.ok && deleteResponse.status !== 404) {
throw new Error(`HTTP ${deleteResponse.status} ${deleteResponse.statusText}`);
}
removed++;
console.log(
`[Releases] Pruned attachment ${asset.name} (id ${asset.id}) from ${tagName} (past the release asset limit)`
);
} catch (deleteError) {
failed++;
console.error(
`[Releases] Failed to prune attachment ${asset.name} (id ${asset.id}) from ${tagName}: ${
deleteError instanceof Error ? deleteError.message : String(deleteError)
}`
);
}
}
return { removed, failed };
}
/**
* Where one release mirror pass gets its releases and its download credentials.
*
@@ -4092,13 +4281,19 @@ async function mirrorSourceReleasesToGiteaLocked({
// Which of those releases also get their assets (#311). Older releases
// past this point are still created with notes and tag, just without the
// uploads. Assets already in Gitea are never removed by this limit; it only
// decides what a sync uploads.
// uploads. The limit alone only decides what a sync uploads - attachments
// that already reached Gitea stay put, unless the opt-in pruning below is
// enabled and takes them away again.
const releaseAssetLimit =
releaseAssetLimitOverride !== undefined
? (normalizeReleaseAssetLimit(releaseAssetLimitOverride) ?? null)
: (await resolveOptions()).releaseAssetLimit;
// Opt-in retroactive pruning (see `pruneOldReleaseAssets` on the config).
// Global only; it prunes by the ranks this repository's own asset limit
// produces, so a per-repo asset limit of 3 prunes that repo's 4th release.
const pruneOldReleaseAssets = (await resolveOptions()).pruneOldReleaseAssets;
// Which releases are candidates at all: every one, or only the ones the source
// does not flag as a pre-release.
const releaseType =
@@ -4279,7 +4474,11 @@ async function mirrorSourceReleasesToGiteaLocked({
totalAssetsFailed += assetResult.failed;
} else if (release.assets && release.assets.length > 0) {
console.log(
`[Releases] Skipping ${release.assets.length} asset(s) for ${release.tag_name}: outside the newest ${releaseAssetLimit} release(s) that get assets (anything already uploaded is kept)`
`[Releases] Skipping ${release.assets.length} asset(s) for ${release.tag_name}: outside the newest ${releaseAssetLimit} release(s) that get assets${
pruneOldReleaseAssets
? " (their existing attachments are pruned by the retention pass)"
: " (anything already uploaded is kept)"
}`
);
releasesPastAssetLimit++;
}
@@ -4376,42 +4575,21 @@ async function mirrorSourceReleasesToGiteaLocked({
// Enforce release retention: first drop the releases the type filter excludes,
// then the oldest releases past the limit.
try {
// Paginate to fetch ALL Gitea releases (API max is 100 per page)
const allGiteaReleases: Array<{
// A page that fails ends the listing early. Retention still runs on what
// was listed, as before, but the storage totals are only recorded from a
// complete listing.
const { items: allGiteaReleases, complete: listingComplete } = await listDestinationReleases<{
id: number;
tag_name: string;
created_at: string;
prerelease?: boolean | null;
assets?: Array<{ size?: number }> | null;
}> = [];
// A page that fails ends the listing early. Retention still runs on what
// was listed, as before, but the storage totals are only recorded from a
// complete listing.
let listingComplete = true;
let cleanupPage = 1;
while (true) {
const pageResponse = await httpGet(
`${config.giteaConfig.url}/api/v1/repos/${repoOwner}/${repoName}/releases?per_page=100&page=${cleanupPage}`,
{
Authorization: `token ${decryptedConfig.giteaConfig.token}`,
}
).catch(() => null);
if (!pageResponse?.data || !Array.isArray(pageResponse.data)) {
listingComplete = false;
break;
}
if (pageResponse.data.length === 0) {
break;
}
allGiteaReleases.push(...pageResponse.data);
if (pageResponse.data.length < 100) {
break;
}
cleanupPage++;
}
}>({
baseUrl: config.giteaConfig.url,
token: decryptedConfig.giteaConfig.token,
owner: repoOwner,
repo: repoName,
});
const giteaBaseUrl = config.giteaConfig.url;
const giteaToken = decryptedConfig.giteaConfig.token;
@@ -4534,6 +4712,55 @@ async function mirrorSourceReleasesToGiteaLocked({
}
}
// Opt-in cleanup of what the asset limit left behind. The limit gates
// uploads, so releases that got their attachments before the limit was set
// or lowered still carry them. Ranked by the same newest-first order the
// uploads were gated on, so a destination release the source does not list
// keeps its attachments - retention removes the release itself if it is
// past the release limit.
const prunedReleaseIds = new Set<number>();
if (pruneOldReleaseAssets && releaseAssetLimit !== null) {
const rankByTag = new Map(
limitedReleases.map((rel, index) => [rel.tag_name, index])
);
const prunable = retainedReleases.filter((rel) => {
if (deletedReleaseIds.has(rel.id)) return false;
// The listing carries each release's attachments, so a release that
// has none (one already pruned on an earlier sync) costs no request.
if (Array.isArray(rel.assets) && rel.assets.length === 0) return false;
const rank = rankByTag.get(rel.tag_name);
return rank !== undefined && !releaseGetsAssets(rank, releaseAssetLimit);
});
if (prunable.length > 0) {
console.log(
`[Releases] Pruning release assets: ${prunable.length} kept release(s) sit past the newest ${releaseAssetLimit} release(s) that get assets`
);
let prunedAssets = 0;
for (const rel of prunable) {
const pruneResult = await pruneReleaseAttachments({
baseUrl: giteaBaseUrl,
token: giteaToken,
repoOwner,
repoName,
giteaReleaseId: rel.id,
tagName: rel.tag_name,
});
prunedAssets += pruneResult.removed;
// Only a release whose attachments actually went is worth zeroing out
// in the storage summary below. A failed listing removed nothing, and
// a partial failure left some behind; both keep the old totals, which
// are closer to the truth than assuming they are gone.
if (pruneResult.removed > 0 && pruneResult.failed === 0) {
prunedReleaseIds.add(rel.id);
}
}
console.log(
`[Releases] Removed ${prunedAssets} attachment(s) from releases past the asset limit for ${repository.fullName}`
);
}
}
// The listing already carries every asset's size, so the storage page
// gets its release numbers without another request (#479). Releases whose
// delete failed are still on the destination and still count.
@@ -4541,7 +4768,11 @@ async function mirrorSourceReleasesToGiteaLocked({
await recordReleaseAssetStorage(
repository.id,
summarizeReleaseAssets(
allGiteaReleases.filter((rel) => !deletedReleaseIds.has(rel.id))
allGiteaReleases
.filter((rel) => !deletedReleaseIds.has(rel.id))
.map((rel) =>
prunedReleaseIds.has(rel.id) ? { ...rel, assets: [] } : rel
)
)
);
}
+1
View File
@@ -44,6 +44,7 @@ describe("sync cursor parsing", () => {
state.syncCursors.issues = {
lastPassStartedAt: "2026-09-20T10:00:00.000Z",
lastFullPassStartedAt: "2026-09-18T10:00:00.000Z",
issueState: "open",
};
state.syncCursors.pullRequests = {
lastPassStartedAt: "2026-09-20T11:00:00.000Z",
+10 -2
View File
@@ -29,6 +29,11 @@ export interface MetadataSyncCursor {
lastPassStartedAt: string;
/** Start of the last full pass that completed cleanly. */
lastFullPassStartedAt: string;
/**
* Issue state filter the last completed issues pass ran with, when it was
* not "all". Issues a filter left out are only found again by a full pass.
*/
issueState?: string;
}
export interface MetadataSyncCursors {
@@ -115,14 +120,17 @@ function isValidTimestamp(value: unknown): value is string {
function parseSyncCursor(raw: unknown): MetadataSyncCursor | undefined {
if (!raw || typeof raw !== "object") return undefined;
const { lastPassStartedAt, lastFullPassStartedAt } = raw as {
const { lastPassStartedAt, lastFullPassStartedAt, issueState } = raw as {
lastPassStartedAt?: unknown;
lastFullPassStartedAt?: unknown;
issueState?: unknown;
};
if (!isValidTimestamp(lastPassStartedAt) || !isValidTimestamp(lastFullPassStartedAt)) {
return undefined;
}
return { lastPassStartedAt, lastFullPassStartedAt };
return typeof issueState === "string"
? { lastPassStartedAt, lastFullPassStartedAt, issueState }
: { lastPassStartedAt, lastFullPassStartedAt };
}
/**
+26
View File
@@ -0,0 +1,26 @@
/**
* The last data each page showed, kept in memory for this browser tab.
*
* The sidebar swaps pages without a reload, and every page used to start from
* a skeleton and fetch its whole list again. A page now starts from what it
* showed last time and refreshes behind it. Nothing goes to browser storage:
* a reload or a new tab starts empty. Entries are per user, so another account
* signed in on the same tab never sees them, and the server never reads or
* writes them.
*/
const entries = new Map<string, unknown>();
function key(userId: string, page: string): string {
return `${userId}\u0000${page}`;
}
export function readPageCache<T>(userId: string | null | undefined, page: string): T | undefined {
if (typeof window === "undefined" || !userId) return undefined;
return entries.get(key(userId, page)) as T | undefined;
}
export function writePageCache<T>(userId: string | null | undefined, page: string, value: T): void {
if (typeof window === "undefined" || !userId) return;
entries.set(key(userId, page), value);
}
+1 -1
View File
@@ -425,7 +425,7 @@ async function handleOrphanedRepository(
const { archivedName } = await archiveGiteaRepo(giteaClient, giteaOwner, giteaRepoName);
// If the archive call renamed the repo in Gitea (mirror path), persist
// the new location so a subsequent "Manual Sync" (the UI's documented
// the new location so a later manual Sync (the UI's documented
// path for refreshing an archived mirror) can find it by its actual
// current name instead of the stale pre-rename one — otherwise
// syncGiteaRepoEnhanced looks up a name that no longer exists and
+19 -8
View File
@@ -1,8 +1,9 @@
/**
* Storage a mirror takes up on its destination (#479). Measured only from
* responses a sync already fetches: the repository GET before `mirror-sync`
* Storage a mirror takes up on its destination (#479). A sync measures it from
* responses it already fetches: the repository GET before `mirror-sync`
* carries `size`, and the release listing the retention pass reads carries
* every asset's size. Nothing here makes a request.
* every asset's size. Refresh sizes (storage-refresh.ts) reads the same fields
* on request. Nothing here makes a request.
*/
import { isPushDestinationKind } from "./destination-kinds";
@@ -99,10 +100,21 @@ export interface StorageOverview {
// failed first mirror, so it only counts once it has been measured.
const MIRROR_EXISTS_STATUSES = new Set(["mirrored", "synced", "syncing", "archived", "pending-approval"]);
function isMeasured(row: StorageSourceRow): boolean {
function isMeasured(row: Pick<StorageSourceRow, "destinationSize" | "releaseAssetsSize">): boolean {
return row.destinationSize !== null || row.releaseAssetsSize !== null;
}
/**
* True for a row with a mirror on a Gitea or Forgejo destination: the rows
* the overview counts as mirrored and Refresh sizes measures.
*/
export function countsAsStorageMirror(
row: Pick<StorageSourceRow, "status" | "destinationProvider" | "destinationSize" | "releaseAssetsSize">
): boolean {
if (isPushDestinationKind(row.destinationProvider)) return false;
return isMeasured(row) || MIRROR_EXISTS_STATUSES.has(row.status);
}
/** The storage page's numbers: totals, per destination owner, and per repository, largest first. */
export function buildStorageOverview(rows: ReadonlyArray<StorageSourceRow>): StorageOverview {
const repositories: StorageRepository[] = [];
@@ -110,10 +122,9 @@ export function buildStorageOverview(rows: ReadonlyArray<StorageSourceRow>): Sto
let mirrored = 0;
for (const row of rows) {
if (isPushDestinationKind(row.destinationProvider)) continue;
const measured = isMeasured(row);
if (measured || MIRROR_EXISTS_STATUSES.has(row.status)) mirrored += 1;
if (!measured) continue;
if (!countsAsStorageMirror(row)) continue;
mirrored += 1;
if (!isMeasured(row)) continue;
const location = (row.mirroredLocation ?? "").trim();
const owner = location.includes("/") ? location.split("/")[0] : "";
+370
View File
@@ -0,0 +1,370 @@
/**
* Tests for Refresh sizes. The destination is a fake fetch that pages like
* Gitea; the database, the owner lookup, the mirror options and the token
* decryption are mocked.
*
* bun's mock.module is process-wide and leaks into other test files, so in the
* shared test process this file registers nothing: it re-runs itself in an
* isolated child process.
*/
import { describe, test, expect, mock, beforeEach, afterEach } from "bun:test";
const CHILD_FLAG = "GM_STORAGE_REFRESH_ISOLATED";
const isChild = !!process.env[CHILD_FLAG];
if (!isChild) {
test("storage refresh - isolated child suite", () => {
const res = Bun.spawnSync({
cmd: [process.execPath, "test", import.meta.path],
env: { ...process.env, [CHILD_FLAG]: "1" },
stdout: "pipe",
stderr: "pipe",
});
if (res.exitCode !== 0) {
console.error(res.stdout.toString());
console.error(res.stderr.toString());
}
expect(res.exitCode).toBe(0);
}, 60_000);
}
const DEST = "https://gitea.example.com";
let configRows: any[] = [];
let repoRows: any[] = [];
let updates: Array<{ id: unknown; values: Record<string, unknown> }> = [];
let releasesFor = new Set<string>();
const mockConfigs = { table: "configs" };
const mockRepositories = { table: "repositories" };
/** Leaf values of a drizzle condition (drizzle conditions are cyclic). */
function conditionLeafValues(node: any, depth = 0): unknown[] {
if (node == null || depth > 8) return [];
if (typeof node !== "object") return [node];
const leaves: unknown[] = [];
if (Array.isArray(node.value)) leaves.push(...node.value);
else if ("value" in node && typeof node.value !== "object") leaves.push(node.value);
for (const chunk of node.queryChunks ?? []) {
leaves.push(...conditionLeafValues(chunk, depth + 1));
}
return leaves;
}
const mockGetOwner = mock(async ({ repository }: any) => repository.expectedOwner ?? "");
if (isChild) {
mock.module("@/lib/db", () => ({
db: {
select: () => ({
from: (table: unknown) => ({
where: () =>
table === mockConfigs
? { orderBy: () => ({ limit: async () => configRows }) }
: Promise.resolve(repoRows),
}),
}),
update: () => ({
set: (values: Record<string, unknown>) => ({
where: async (cond: unknown) => {
// The id is the one leaf that is not an SQL fragment ("", " = ").
const id = conditionLeafValues(cond).find((v) => typeof v === "string" && !/^\s*=?\s*$/.test(v));
updates.push({ id, values });
},
}),
}),
},
configs: mockConfigs,
repositories: mockRepositories,
organizations: {},
users: {},
events: {},
mirrorJobs: {},
sessions: {},
accounts: {},
verificationTokens: {},
verifications: {},
oauthClients: {},
oauthAccessTokens: {},
oauthRefreshTokens: {},
oauthConsents: {},
jwkss: {},
ssoProviders: {},
apikeys: {},
rateLimits: {},
sources: {},
}));
mock.module("@/lib/gitea", () => ({
getGiteaRepoOwnerAsync: mockGetOwner,
}));
mock.module("@/lib/utils/mirror-overrides", () => ({
resolveMirrorOptionsForRepository: async ({ repository }: any) => ({
mirrorReleases: releasesFor.has(repository.id),
}),
}));
mock.module("@/lib/utils/config-encryption", () => ({
decryptConfigTokens: (config: any) => config,
}));
}
const { startStorageRefresh, getStorageRefreshProgress, waitForStorageRefresh } = isChild
? await import("./storage-refresh")
: ({} as typeof import("./storage-refresh"));
const realFetch = globalThis.fetch;
let requestedUrls: string[] = [];
interface FakeDestination {
/** Repositories per owner, as the owner listing returns them. */
owners?: Record<string, Array<Record<string, unknown>>>;
/** Single GET answers by "owner/name"; a number is an error status. */
single?: Record<string, Record<string, unknown> | number>;
/** Releases by "owner/name"; a number is an error status. */
releases?: Record<string, Array<Record<string, unknown>> | number>;
/** Held until released, to keep a run going. */
gate?: Promise<void>;
}
function json(data: unknown, total?: number) {
const headers: Record<string, string> = { "content-type": "application/json" };
if (total !== undefined) headers["x-total-count"] = String(total);
return new Response(JSON.stringify(data), { status: 200, headers });
}
function serve(world: FakeDestination) {
requestedUrls = [];
globalThis.fetch = (async (input: RequestInfo | URL) => {
const url = String(input);
requestedUrls.push(url);
await world.gate;
const { pathname, searchParams } = new URL(url);
const page = Number(searchParams.get("page") ?? "1");
const parts = pathname.split("/").map(decodeURIComponent);
// /api/v1/users/{owner}/repos
if (parts[3] === "users" && parts[5] === "repos") {
const all = world.owners?.[parts[4]];
if (!all) return new Response("not found", { status: 404, statusText: "Not Found" });
return json(all.slice((page - 1) * 50, page * 50), all.length);
}
// /api/v1/repos/{owner}/{name}/releases
if (parts[3] === "repos" && parts[6] === "releases") {
const answer = world.releases?.[`${parts[4]}/${parts[5]}`] ?? [];
if (typeof answer === "number") return new Response("boom", { status: answer, statusText: "Error" });
return json(answer.slice((page - 1) * 50, page * 50), answer.length);
}
// /api/v1/repos/{owner}/{name}
if (parts[3] === "repos" && parts.length === 6) {
const answer = world.single?.[`${parts[4]}/${parts[5]}`] ?? 404;
if (typeof answer === "number") return new Response("nope", { status: answer, statusText: "Error" });
return json(answer);
}
throw new Error(`unexpected request: ${url}`);
}) as typeof fetch;
}
function repo(id: string, overrides: Record<string, unknown> = {}) {
return {
id,
userId: "user-1",
name: id,
fullName: `octo/${id}`,
mirroredLocation: `archive/${id}`,
status: "mirrored",
destinationProvider: "gitea",
destinationSize: null,
releaseAssetsSize: null,
releaseAssetsCount: null,
...overrides,
};
}
function destRepo(name: string, sizeKiB: number, extra: Record<string, unknown> = {}) {
return { name, owner: { login: "archive" }, mirror: true, size: sizeKiB, ...extra };
}
async function refresh() {
const started = await startStorageRefresh("user-1");
await waitForStorageRefresh("user-1");
return { started, progress: getStorageRefreshProgress("user-1") };
}
function writesFor(id: string) {
return updates.filter((u) => u.id === id).map((u) => u.values);
}
beforeEach(() => {
configRows = [{ userId: "user-1", isActive: true, giteaConfig: { url: `${DEST}/`, token: "dest-token", provider: "gitea" } }];
repoRows = [];
updates = [];
releasesFor = new Set();
mockGetOwner.mockClear();
});
afterEach(() => {
globalThis.fetch = realFetch;
});
describe.skipIf(!isChild)("Refresh sizes", () => {
test("reads sizes from one listing per owner and releases only where they are mirrored", async () => {
repoRows = [repo("app"), repo("lib")];
releasesFor = new Set(["app"]);
serve({
owners: { archive: [destRepo("app", 2959), destRepo("lib", 10), destRepo("not-ours", 99)] },
releases: { "archive/app": [{ assets: [{ size: 1000 }, { size: 24 }] }, { assets: [] }] },
});
const { started, progress } = await refresh();
expect(started).toMatchObject({ ok: true, progress: { running: true, total: 2 } });
expect(progress).toMatchObject({ running: false, total: 2, done: 2, measured: 2, missing: 0, failed: 0 });
expect(writesFor("app")).toEqual([
expect.objectContaining({ destinationSize: 2959 * 1024, releaseAssetsSize: 1024, releaseAssetsCount: 2 }),
]);
expect(writesFor("lib")).toEqual([expect.objectContaining({ destinationSize: 10 * 1024 })]);
expect(writesFor("lib")[0]).not.toHaveProperty("releaseAssetsSize");
expect(writesFor("lib")[0].storageCheckedAt).toBeInstanceOf(Date);
const ownerListings = requestedUrls.filter((u) => u.includes("/users/"));
expect(ownerListings).toEqual([`${DEST}/api/v1/users/archive/repos?limit=50&page=1`]);
expect(requestedUrls.filter((u) => u.includes("/releases"))).toHaveLength(1);
// Only the destination is contacted.
expect(requestedUrls.every((u) => u.startsWith(`${DEST}/api/v1/`))).toBe(true);
});
test("pages through an owner with more repositories than one page", async () => {
const many = Array.from({ length: 120 }, (_, i) => destRepo(`r${i}`, i + 1));
repoRows = [repo("r119")];
serve({ owners: { archive: many } });
const { progress } = await refresh();
expect(progress?.measured).toBe(1);
expect(writesFor("r119")[0].destinationSize).toBe(120 * 1024);
expect(requestedUrls.filter((u) => u.includes("/users/"))).toHaveLength(3);
});
test("falls back to the single GET for a repository the listing does not have", async () => {
repoRows = [repo("renamed"), repo("gone")];
serve({
owners: { archive: [] },
single: { "archive/renamed": destRepo("new-name", 5) },
});
const { progress } = await refresh();
expect(progress).toMatchObject({ measured: 1, missing: 1, failed: 0 });
expect(writesFor("renamed")[0].destinationSize).toBe(5 * 1024);
expect(writesFor("gone")).toEqual([]);
});
test("reads releases at the name the destination reports", async () => {
repoRows = [repo("renamed")];
releasesFor = new Set(["renamed"]);
serve({
owners: { archive: [] },
single: { "archive/renamed": destRepo("new-name", 5) },
releases: { "archive/new-name": [{ assets: [{ size: 7 }] }] },
});
await refresh();
expect(writesFor("renamed")[0]).toMatchObject({ releaseAssetsSize: 7, releaseAssetsCount: 1 });
});
test("a failed owner listing leaves its repositories to the single GET", async () => {
repoRows = [repo("app")];
serve({ single: { "archive/app": destRepo("app", 3) } });
const { progress } = await refresh();
expect(progress?.measured).toBe(1);
expect(requestedUrls).toContain(`${DEST}/api/v1/repos/archive/app`);
});
test("without a recorded location, measures only a mirror where a sync would look", async () => {
repoRows = [
repo("placed", { mirroredLocation: null, expectedOwner: "me" }),
repo("someone-elses", { mirroredLocation: null, expectedOwner: "me" }),
];
serve({
owners: { me: [destRepo("placed", 4), destRepo("someone-elses", 8, { mirror: false })] },
});
const { progress } = await refresh();
expect(mockGetOwner).toHaveBeenCalledTimes(2);
expect(progress).toMatchObject({ measured: 1, missing: 1 });
expect(writesFor("placed")[0].destinationSize).toBe(4 * 1024);
expect(writesFor("someone-elses")).toEqual([]);
});
test("an incomplete release listing keeps the stored release numbers", async () => {
repoRows = [repo("app")];
releasesFor = new Set(["app"]);
serve({ owners: { archive: [destRepo("app", 2)] }, releases: { "archive/app": 500 } });
const { progress } = await refresh();
expect(progress?.measured).toBe(1);
expect(writesFor("app")[0]).toHaveProperty("destinationSize", 2048);
expect(writesFor("app")[0]).not.toHaveProperty("releaseAssetsSize");
});
test("counts a request that fails and writes nothing for it", async () => {
repoRows = [repo("app")];
serve({ owners: { archive: [] }, single: { "archive/app": 500 } });
const { progress } = await refresh();
expect(progress).toMatchObject({ running: false, done: 1, measured: 0, failed: 1 });
expect(updates).toEqual([]);
});
test("measures the rows the storage page counts as mirrors, and no others", async () => {
repoRows = [
repo("app"),
repo("pushed", { destinationProvider: "gitlab" }),
repo("imported", { status: "imported", mirroredLocation: null }),
repo("failed-but-measured", { status: "failed", destinationSize: 10 }),
];
serve({ owners: { archive: [destRepo("app", 1), destRepo("failed-but-measured", 2)] } });
const { progress } = await refresh();
expect(progress).toMatchObject({ total: 2, measured: 2 });
});
test("refuses without a destination token or with a push destination", async () => {
serve({});
configRows = [{ userId: "user-1", giteaConfig: { url: DEST, token: "", provider: "gitea" } }];
expect(await startStorageRefresh("user-1")).toMatchObject({ ok: false, status: 400 });
configRows = [{ userId: "user-1", giteaConfig: { url: "https://gitlab.com", token: "t", provider: "gitlab" } }];
expect(await startStorageRefresh("user-1")).toMatchObject({
ok: false,
status: 400,
error: expect.stringContaining("Gitea and Forgejo"),
});
expect(requestedUrls).toEqual([]);
});
test("a second start while a run is going reports that run instead of starting another", async () => {
repoRows = [repo("app")];
let open!: () => void;
const gate = new Promise<void>((resolve) => (open = resolve));
serve({ owners: { archive: [destRepo("app", 1)] }, gate });
const first = await startStorageRefresh("user-1");
const second = await startStorageRefresh("user-1");
expect(first).toMatchObject({ ok: true, progress: { running: true } });
expect(second).toMatchObject({ ok: true, progress: { running: true, startedAt: (first as any).progress.startedAt } });
open();
await waitForStorageRefresh("user-1");
expect(requestedUrls.filter((u) => u.includes("/users/"))).toHaveLength(1);
expect(getStorageRefreshProgress("user-1")).toMatchObject({ running: false, measured: 1 });
});
});
+284
View File
@@ -0,0 +1,284 @@
/**
* Refresh sizes: measure every mirror on the destination now instead of
* waiting for each one's next sync (#479 follow-up). It only reads the
* destination and never contacts the source:
*
* - one listing per destination owner carries every repository's `size`;
* - a repository missing from its owner's listing gets the single GET, which
* follows a rename or transfer the way a sync does;
* - repositories that mirror releases get their release listing read.
*
* Runs in the background, one run per user at a time. Progress lives in memory
* and the storage overview reports it.
*/
import { eq, sql } from "drizzle-orm";
import { db, configs, repositories } from "./db";
import type { Repository } from "./db/schema";
import type { Config } from "@/types/config";
import { HttpError, httpGet } from "./http-client";
import { decryptConfigTokens } from "./utils/config-encryption";
import { isPushDestinationKind, normalizeDestinationProviderKind } from "./destination-kinds";
import { getGiteaRepoOwnerAsync } from "./gitea";
import { resolveMirrorOptionsForRepository } from "./utils/mirror-overrides";
import { listDestinationOwnerRepositories, listDestinationReleases } from "./destination-listing";
import { countsAsStorageMirror, destinationSizeColumns, summarizeReleaseAssets } from "./repository-storage";
import { processInParallel } from "./utils/concurrency";
// Requests in flight against the destination at once.
const CONCURRENCY = 4;
export interface StorageRefreshProgress {
running: boolean;
/** Repositories this run measures. */
total: number;
/** Repositories finished, whatever the outcome. */
done: number;
measured: number;
/** Not found on the destination where the mirror should be. */
missing: number;
/** A request failed; the stored numbers were kept. */
failed: number;
startedAt: string;
finishedAt: string | null;
}
/** The destination's answer for one repository, the fields used here. */
interface DestinationRepository {
name?: string;
owner?: { login?: string } | string;
mirror?: boolean;
size?: number;
}
interface Target {
repository: Repository;
owner: string;
name: string;
/** False when the location was not recorded and is only where a sync would look. */
recorded: boolean;
}
const runs = new Map<string, StorageRefreshProgress>();
const inFlight = new Map<string, Promise<void>>();
export function getStorageRefreshProgress(userId: string): StorageRefreshProgress | null {
const run = runs.get(userId);
return run ? { ...run } : null;
}
/** Resolves when the user's current run ends. For tests. */
export function waitForStorageRefresh(userId: string): Promise<void> {
return inFlight.get(userId) ?? Promise.resolve();
}
export type StartStorageRefreshResult =
| { ok: true; progress: StorageRefreshProgress }
| { ok: false; status: number; error: string };
/** Start a run for the user, or report the one already running. */
export async function startStorageRefresh(userId: string): Promise<StartStorageRefreshResult> {
const previous = runs.get(userId);
if (previous?.running) return { ok: true, progress: { ...previous } };
// Claimed before the first await so a second click cannot start a twin.
const progress: StorageRefreshProgress = {
running: true,
total: 0,
done: 0,
measured: 0,
missing: 0,
failed: 0,
startedAt: new Date().toISOString(),
finishedAt: null,
};
runs.set(userId, progress);
const release = () => {
if (previous) runs.set(userId, previous);
else runs.delete(userId);
};
const refuse = (status: number, error: string): StartStorageRefreshResult => {
release();
return { ok: false, status, error };
};
let config: Partial<Config>;
let targets: Repository[];
try {
const [configRow] = await db
.select()
.from(configs)
.where(eq(configs.userId, userId))
.orderBy(sql`${configs.isActive} DESC`, sql`${configs.updatedAt} DESC`)
.limit(1);
if (!configRow?.giteaConfig?.url || !configRow.giteaConfig.token) {
return refuse(400, "Configure the destination URL and token first");
}
if (isPushDestinationKind(normalizeDestinationProviderKind(configRow.giteaConfig.provider))) {
return refuse(400, "Sizes are read from Gitea and Forgejo destinations only");
}
config = configRow as unknown as Partial<Config>;
const rows = await db.select().from(repositories).where(eq(repositories.userId, userId));
targets = (rows as Repository[]).filter((row) =>
countsAsStorageMirror({
status: row.status,
destinationProvider: row.destinationProvider ?? "gitea",
destinationSize: row.destinationSize ?? null,
releaseAssetsSize: row.releaseAssetsSize ?? null,
})
);
} catch (error) {
release();
throw error;
}
progress.total = targets.length;
const run = measureAll(config, targets, progress)
.catch((error) => {
console.error(
`[Storage] Refresh sizes stopped for user ${userId}: ${error instanceof Error ? error.message : String(error)}`
);
})
.finally(() => {
progress.failed += progress.total - progress.done;
progress.done = progress.total;
progress.running = false;
progress.finishedAt = new Date().toISOString();
inFlight.delete(userId);
console.log(
`[Storage] Refreshed sizes for user ${userId}: ${progress.measured} measured, ${progress.missing} not found, ${progress.failed} failed`
);
});
inFlight.set(userId, run);
return { ok: true, progress: { ...progress } };
}
function parseLocation(location: string | null | undefined): { owner: string; name: string } | null {
const trimmed = (location ?? "").trim();
const slash = trimmed.indexOf("/");
if (slash <= 0 || slash === trimmed.length - 1) return null;
return { owner: trimmed.slice(0, slash), name: trimmed.slice(slash + 1) };
}
async function measureAll(
config: Partial<Config>,
rows: Repository[],
progress: StorageRefreshProgress
): Promise<void> {
const baseUrl = config.giteaConfig!.url.replace(/\/+$/, "");
const token = decryptConfigTokens(config as Config).giteaConfig.token;
// Where each mirror is, or where a sync would look for it.
const targets: Target[] = [];
for (const repository of rows) {
const recorded = parseLocation(repository.mirroredLocation);
if (recorded) {
targets.push({ repository, ...recorded, recorded: true });
continue;
}
const owner = await getGiteaRepoOwnerAsync({ config, repository }).catch(() => "");
if (owner) {
targets.push({ repository, owner, name: repository.name, recorded: false });
} else {
progress.failed += 1;
progress.done += 1;
}
}
// One listing per owner. A failed listing leaves its repositories to the
// single GET below.
const owners = [...new Map(targets.map((t) => [t.owner.toLowerCase(), t.owner])).values()];
const listings = new Map<string, Map<string, DestinationRepository>>();
await processInParallel(
owners,
async (owner) => {
const listing = await listDestinationOwnerRepositories<DestinationRepository>({ baseUrl, token, owner });
if (!listing.complete) return;
const byName = new Map<string, DestinationRepository>();
for (const repo of listing.items) {
if (typeof repo?.name === "string") byName.set(repo.name.toLowerCase(), repo);
}
listings.set(owner.toLowerCase(), byName);
},
CONCURRENCY
);
await processInParallel(
targets,
async (target) => {
try {
const outcome = await measureOne({ config, baseUrl, token, target, listings });
progress[outcome] += 1;
} catch (error) {
progress.failed += 1;
console.warn(
`[Storage] Could not measure ${target.owner}/${target.name}: ${
error instanceof Error ? error.message : String(error)
}`
);
} finally {
progress.done += 1;
}
},
CONCURRENCY
);
}
async function measureOne({
config,
baseUrl,
token,
target,
listings,
}: {
config: Partial<Config>;
baseUrl: string;
token: string;
target: Target;
listings: Map<string, Map<string, DestinationRepository>>;
}): Promise<"measured" | "missing" | "failed"> {
let info = listings.get(target.owner.toLowerCase())?.get(target.name.toLowerCase()) ?? null;
if (!info) {
try {
const response = await httpGet<DestinationRepository>(
`${baseUrl}/api/v1/repos/${encodeURIComponent(target.owner)}/${encodeURIComponent(target.name)}`,
{ Authorization: `token ${token}` }
);
info = response.data;
} catch (error) {
if (error instanceof HttpError && error.status === 404) return "missing";
throw error;
}
}
// Without a recorded location, only a mirror at the expected place is ours.
if (!info || (!target.recorded && info.mirror !== true)) return "missing";
const now = new Date();
const columns: Partial<
Pick<Repository, "destinationSize" | "releaseAssetsSize" | "releaseAssetsCount" | "storageCheckedAt">
> = { ...destinationSizeColumns(info, now) };
const { mirrorReleases } = await resolveMirrorOptionsForRepository({ config, repository: target.repository });
if (mirrorReleases) {
const owner = (typeof info.owner === "string" ? info.owner : info.owner?.login) || target.owner;
const releases = await listDestinationReleases<{ assets?: Array<{ size?: unknown }> | null }>({
baseUrl,
token,
owner,
repo: info.name || target.name,
});
if (releases.complete) {
const summary = summarizeReleaseAssets(releases.items);
columns.releaseAssetsSize = summary.bytes;
columns.releaseAssetsCount = summary.count;
columns.storageCheckedAt = now;
}
}
if (Object.keys(columns).length === 0) return "failed";
await db.update(repositories).set(columns).where(eq(repositories.id, target.repository.id!));
return "measured";
}
+28
View File
@@ -0,0 +1,28 @@
import { describe, expect, test } from "bun:test";
import { compareNames } from "./compare-names";
describe("compareNames", () => {
test("ignores case", () => {
expect(["Zebra", "apple", "Mango"].sort(compareNames)).toEqual([
"apple",
"Mango",
"Zebra",
]);
});
test("sorts accented names with their base letter", () => {
expect(["Zeta", "Ärger", "Bravo"].sort(compareNames)).toEqual([
"Ärger",
"Bravo",
"Zeta",
]);
});
test("compares numbers by value", () => {
expect(["repo-10", "repo-2", "repo-1"].sort(compareNames)).toEqual([
"repo-1",
"repo-2",
"repo-10",
]);
});
});
+13
View File
@@ -0,0 +1,13 @@
/**
* Order for owner, organization and repository names in filter lists: case
* and accent insensitive, with numbers compared by value. A plain `.sort()`
* compares code units, so "Zebra" lands before "apple" and umlauts sort last.
*/
const nameCollator = new Intl.Collator(undefined, {
sensitivity: "base",
numeric: true,
});
export function compareNames(a: string, b: string): number {
return nameCollator.compare(a, b);
}
+3
View File
@@ -41,6 +41,9 @@ function buildMinimalUiConfigs(overrides: { includeCollaboratorRepos?: boolean }
milestones: false,
wiki: false,
},
issueState: "all",
issueComments: true,
issueMedia: true,
};
const advancedOptions: AdvancedOptions = {
skipForks: false,
+19 -1
View File
@@ -17,7 +17,14 @@ import { normalizeDestinationProviderKind } from "@/lib/destination-kinds";
import { z } from "zod";
import { githubConfigSchema, giteaConfigSchema, scheduleConfigSchema, cleanupConfigSchema } from "@/lib/db/schema";
import { parseInterval } from "@/lib/utils/duration-parser";
import { normalizeReleaseAssetLimit, normalizeReleaseTagPatterns, normalizeReleaseType, DEFAULT_RELEASE_TYPE } from "@/lib/utils/mirror-overrides";
import {
DEFAULT_ISSUE_STATE,
normalizeIssueState,
normalizeReleaseAssetLimit,
normalizeReleaseTagPatterns,
normalizeReleaseType,
DEFAULT_RELEASE_TYPE,
} from "@/lib/utils/mirror-overrides";
import { buildClockCronExpression, normalizeTimezone, parseClockCronExpression } from "@/lib/utils/schedule-utils";
// Use the actual database schema types
@@ -171,6 +178,9 @@ export function mapUiToDbConfig(
// null is a real value here ("assets for every mirrored release"), so
// it is stored as such rather than dropped.
releaseAssetLimit: normalizeReleaseAssetLimit(mirrorOptions.releaseAssetLimit) ?? null,
// Opt-in pruning: only ever a plain boolean, so an absent switch stores the
// safe default (never delete what the destination already has).
pruneOldReleaseAssets: mirrorOptions.pruneOldReleaseAssets === true,
// An unusable value falls back to "all" rather than storing a filter that
// would hide releases from the user.
releaseType: normalizeReleaseType(mirrorOptions.releaseType) ?? DEFAULT_RELEASE_TYPE,
@@ -180,6 +190,9 @@ export function mapUiToDbConfig(
releaseExcludeTags: normalizeReleaseTagPatterns(mirrorOptions.releaseExcludeTags) ?? [],
mirrorMetadata: mirrorOptions.mirrorMetadata,
mirrorIssues: mirrorOptions.mirrorMetadata && mirrorOptions.metadataComponents.issues,
mirrorIssueState: normalizeIssueState(mirrorOptions.issueState) ?? DEFAULT_ISSUE_STATE,
mirrorIssueComments: mirrorOptions.issueComments,
mirrorIssueMedia: mirrorOptions.issueMedia,
mirrorPullRequests: mirrorOptions.mirrorMetadata && mirrorOptions.metadataComponents.pullRequests,
mirrorLabels: mirrorOptions.mirrorMetadata && mirrorOptions.metadataComponents.labels,
mirrorMilestones: mirrorOptions.mirrorMetadata && mirrorOptions.metadataComponents.milestones,
@@ -251,6 +264,7 @@ export function mapDbToUiConfig(dbConfig: any): {
mirrorReleases: dbConfig.giteaConfig?.mirrorReleases || false,
releaseLimit: dbConfig.giteaConfig?.releaseLimit || 10,
releaseAssetLimit: normalizeReleaseAssetLimit(dbConfig.giteaConfig?.releaseAssetLimit) ?? null,
pruneOldReleaseAssets: dbConfig.giteaConfig?.pruneOldReleaseAssets === true,
releaseType: normalizeReleaseType(dbConfig.giteaConfig?.releaseType) ?? DEFAULT_RELEASE_TYPE,
releaseIncludeTags: normalizeReleaseTagPatterns(dbConfig.giteaConfig?.releaseIncludeTags) ?? [],
releaseExcludeTags: normalizeReleaseTagPatterns(dbConfig.giteaConfig?.releaseExcludeTags) ?? [],
@@ -263,6 +277,10 @@ export function mapDbToUiConfig(dbConfig: any): {
milestones: dbConfig.giteaConfig?.mirrorMilestones || false,
wiki: dbConfig.giteaConfig?.wiki || false,
},
issueState:
normalizeIssueState(dbConfig.giteaConfig?.mirrorIssueState) ?? DEFAULT_ISSUE_STATE,
issueComments: dbConfig.giteaConfig?.mirrorIssueComments ?? true,
issueMedia: dbConfig.giteaConfig?.mirrorIssueMedia ?? true,
};
// Map advanced options
+77
View File
@@ -0,0 +1,77 @@
import { describe, expect, test } from "bun:test";
import { applyIssueMediaOption, stripIssueMedia } from "./issue-media";
describe("stripIssueMedia", () => {
test("removes Markdown images but keeps the surrounding text", () => {
const body = "Before\n\n![screenshot](https://user-images.githubusercontent.com/1/a.png)\n\nAfter";
expect(stripIssueMedia(body)).toBe("Before\n\n[media omitted]\n\nAfter");
});
test("removes images nested inside links", () => {
const body = "[![preview](https://user-images.githubusercontent.com/1/a.png)](https://example.com)";
expect(stripIssueMedia(body)).toBe("[media omitted]");
});
test("removes HTML media tags", () => {
const body = 'Text\n<img src="https://example.com/a.png" />\n<video controls><source src="a.mp4"></video>\nEnd';
const result = stripIssueMedia(body);
expect(result).toContain("Text");
expect(result).toContain("End");
expect(result).not.toContain("<img");
expect(result).not.toContain("<video");
});
test("removes GitHub attachment links", () => {
const body = "See https://github.com/user-attachments/assets/abc123 and https://user-images.githubusercontent.com/2/b.png";
expect(stripIssueMedia(body)).toBe("See [media omitted] and [media omitted]");
});
test("removes attached files, on the current and the older upload paths", () => {
const body =
"Logs: [debug.log](https://github.com/user-attachments/files/123/debug.log) and [dump.zip](https://github.com/acme/demo/files/456/dump.zip)\nVideo: https://github.com/acme/demo/assets/789/0d1e2f3a-aaaa-bbbb-cccc-1234567890ab";
expect(stripIssueMedia(body)).toBe(
"Logs: [media omitted] and [media omitted]\nVideo: [media omitted]"
);
});
test("keeps links to repository content on other GitHub hosts", () => {
const body =
"Run https://raw.githubusercontent.com/acme/demo/main/install.sh, see [the config](https://raw.githubusercontent.com/acme/demo/main/config.yml) and https://gist.githubusercontent.com/u/abc/raw/log.txt";
expect(stripIssueMedia(body)).toBe(body);
});
test("keeps tilde fenced code blocks", () => {
const body = "~~~md\n![not-an-image](https://example.com/x.png)\n~~~";
expect(stripIssueMedia(body)).toBe(body);
});
test("keeps normal links and code blocks", () => {
const body = "See [the docs](https://example.com/docs).\n\n```\n![not-an-image](https://example.com/x.png)\n```";
expect(stripIssueMedia(body)).toBe(body);
});
test("returns an empty string for missing bodies", () => {
expect(stripIssueMedia(null)).toBe("");
expect(stripIssueMedia(undefined)).toBe("");
});
});
describe("applyIssueMediaOption", () => {
test("leaves the body untouched when media is enabled", () => {
const body = "![screenshot](https://user-images.githubusercontent.com/1/a.png)";
expect(applyIssueMediaOption(body, true)).toBe(body);
});
test("strips media when the option is disabled", () => {
const body = "![screenshot](https://user-images.githubusercontent.com/1/a.png)";
expect(applyIssueMediaOption(body, false)).toBe("[media omitted]");
});
});
+90
View File
@@ -0,0 +1,90 @@
const MEDIA_PLACEHOLDER = "[media omitted]";
/**
* Where GitHub keeps what people upload to issues and comments: images,
* videos and attached files, on the current hosts and the older per-repository
* paths. Other GitHub hosts such as raw.githubusercontent.com serve repository
* content, not uploads, and are left alone.
*/
const GITHUB_UPLOAD_URL =
String.raw`https?:\/\/(?:(?:private-)?user-images\.githubusercontent\.com\/|github\.com\/user-attachments\/(?:assets|files)\/|github\.com\/[\w.-]+\/[\w.-]+\/(?:assets|files)\/\d+\/)`;
const GITHUB_MEDIA_URL = new RegExp(String.raw`${GITHUB_UPLOAD_URL}[^\s)>\]]+`, "gi");
/**
* Markdown links to an upload. This runs before the bare URL pass so
* `[screenshot](https://user-images...)` becomes one placeholder instead of a
* broken empty link.
*/
const MARKDOWN_MEDIA_LINK = new RegExp(
String.raw`\[([^\]]*)\]\(${GITHUB_UPLOAD_URL}[^\s)]+(?:\s+"[^"]*")?\)`,
"gi"
);
/** Inline Markdown images, with optional title. */
const MARKDOWN_IMAGE = /!\[[^\]]*\]\([^)]*\)/g;
/** HTML media blocks/tags that can be embedded in Markdown. */
const HTML_MEDIA = [
/<picture\b[^>]*>[\s\S]*?<\/picture>/gi,
/<video\b[^>]*>[\s\S]*?<\/video>/gi,
/<audio\b[^>]*>[\s\S]*?<\/audio>/gi,
/<iframe\b[^>]*>[\s\S]*?<\/iframe>/gi,
/<(?:img|source)\b[^>]*\/?>/gi,
];
/**
* Remove images, videos and attachment links from an issue body or comment.
*
* With the "issue media" option off we still want the text of the issue, so
* this drops only the media references and leaves everything else intact. It
* does not download anything; it just prevents the mirrored copy from pointing
* at GitHub-hosted media that would be loaded every time the issue is opened.
*/
export function stripIssueMedia(value: string | null | undefined): string {
if (!value) return "";
// Code blocks may document Markdown/HTML media syntax. Protect them so the
// cleanup only touches content the author actually rendered as media.
const codeBlocks: string[] = [];
let result = value.replace(/(```|~~~)[\s\S]*?\1|`[^`\n]*`/g, (block) => {
const index = codeBlocks.push(block) - 1;
return `\u0000${index}\u0000`;
});
// Images first: their target URL is often a media host, and running the
// link pass first would leave the leading `![...]` behind.
result = result.replace(MARKDOWN_IMAGE, MEDIA_PLACEHOLDER);
result = result.replace(MARKDOWN_MEDIA_LINK, MEDIA_PLACEHOLDER);
for (const pattern of HTML_MEDIA) {
result = result.replace(pattern, MEDIA_PLACEHOLDER);
}
result = result.replace(GITHUB_MEDIA_URL, MEDIA_PLACEHOLDER);
// An image wrapped in a link leaves either `[](url)` or a link whose text is
// just the placeholder behind; collapse both so the body does not keep a
// meaningless link.
result = result.replace(
/\[\[media omitted\]\]\([^)]*\)/g,
MEDIA_PLACEHOLDER
);
result = result.replace(/\[\]\([^)]*\)/g, MEDIA_PLACEHOLDER);
// Removing block-level media leaves runs of blank lines behind. Keep the
// author's paragraph spacing but avoid growing the body with empty space.
result = result.replace(/\n{3,}/g, "\n\n");
result = result.replace(/\u0000(\d+)\u0000/g, (_match, index: string) => {
return codeBlocks[Number.parseInt(index, 10)] ?? "";
});
return result.trim();
}
/** Apply the media option without changing bodies when media is enabled. */
export function applyIssueMediaOption(
value: string | null | undefined,
includeMedia: boolean
): string {
return includeMedia ? value ?? "" : stripIssueMedia(value);
}
+65 -3
View File
@@ -16,6 +16,7 @@ import {
MIRROR_OVERRIDE_LIMIT_KEYS,
MIRROR_OVERRIDE_PATTERN_KEYS,
mirrorOptionsToFlags,
normalizeIssueState,
normalizeMirrorOverrides,
normalizeReleaseAssetLimit,
normalizeReleaseLimit,
@@ -446,6 +447,8 @@ describe("UI_MIRROR_OVERRIDE_KEYS", () => {
"lfs",
"wiki",
"mirrorIssues",
"mirrorIssueComments",
"mirrorIssueMedia",
"mirrorPullRequests",
"mirrorReleases",
"mirrorLabels",
@@ -1702,15 +1705,20 @@ describe("release type filter", () => {
});
test("the choice keys and labels agree", () => {
expect(MIRROR_OVERRIDE_CHOICE_KEYS).toEqual(["releaseType"]);
expect(MIRROR_OVERRIDE_CHOICE_KEYS).toEqual([
"releaseType",
"mirrorIssueState",
]);
for (const key of MIRROR_OVERRIDE_CHOICE_KEYS) {
expect(MIRROR_OVERRIDE_LABELS[key]).toBeTruthy();
}
});
test("the choice keys are not walked as flags or limits", () => {
expect(MIRROR_OVERRIDE_KEYS).not.toContain("releaseType");
expect(MIRROR_OVERRIDE_LIMIT_KEYS).not.toContain("releaseType");
for (const key of MIRROR_OVERRIDE_CHOICE_KEYS) {
expect(MIRROR_OVERRIDE_KEYS).not.toContain(key);
expect(MIRROR_OVERRIDE_LIMIT_KEYS).not.toContain(key);
}
});
});
@@ -1770,3 +1778,57 @@ describe("release type filter", () => {
});
});
});
describe("issue content options", () => {
test("normalizes the issue state filter", () => {
expect(normalizeIssueState(" OPEN ")).toBe("open");
expect(normalizeIssueState("closed")).toBe("closed");
expect(normalizeIssueState("all")).toBe("all");
expect(normalizeIssueState("merged")).toBeUndefined();
expect(normalizeIssueState(null)).toBeUndefined();
});
test("resolves issue state per tier, most specific first", () => {
const resolved = resolveMirrorOptions({
config: makeConfig({ mirrorIssueState: "closed" }),
repository: makeRepo({
mirrorOverrides: { mirrorIssueState: "all" },
}),
orgOverrides: { mirrorIssueState: "open" },
});
expect(resolved.mirrorIssueState).toBe("all");
});
test("inherits issue state from the organization when the repository has none", () => {
const resolved = resolveMirrorOptions({
config: makeConfig({ mirrorIssueState: "closed" }),
repository: makeRepo(),
orgOverrides: { mirrorIssueState: "open" },
});
expect(resolved.mirrorIssueState).toBe("open");
});
test("keeps comments and media on when no tier pins them", () => {
const resolved = resolveMirrorOptions({
config: makeConfig({}),
repository: makeRepo(),
});
expect(resolved.mirrorIssueComments).toBe(true);
expect(resolved.mirrorIssueMedia).toBe(true);
expect(resolved.mirrorIssueState).toBe("all");
});
test("the content options are disabled while issues are off", () => {
const gating = getMirrorOverrideGating({
targetKind: "repository",
effective: { mirrorIssues: false },
});
expect(gating.mirrorIssueComments).toBe(MIRROR_GATING_REASONS.issuesOff);
expect(gating.mirrorIssueMedia).toBe(MIRROR_GATING_REASONS.issuesOff);
expect(gating.mirrorIssueState).toBe(MIRROR_GATING_REASONS.issuesOff);
});
});
+101 -7
View File
@@ -1,5 +1,10 @@
import type { Config } from "@/types/config";
import type { MirrorOverrides, ReleaseType, Repository } from "@/lib/db/schema";
import type {
IssueStateFilter,
MirrorOverrides,
ReleaseType,
Repository,
} from "@/lib/db/schema";
import { mirrorOverridesSchema } from "@/lib/db/schema";
import {
normalizeSourceProviderKind,
@@ -18,6 +23,8 @@ export const MIRROR_OVERRIDE_KEYS = [
"mirrorReleases",
"mirrorMetadata",
"mirrorIssues",
"mirrorIssueComments",
"mirrorIssueMedia",
"mirrorPullRequests",
"mirrorLabels",
"mirrorMilestones",
@@ -44,9 +51,13 @@ export type MirrorOverrideLimitKey = (typeof MIRROR_OVERRIDE_LIMIT_KEYS)[number]
* tri-state toggles and the numeric rows in the dialog assume those shapes.
*
* `releaseType` decides which releases are candidates at all; `releaseLimit`
* then decides how many of the matching ones are kept.
* then decides how many of the matching ones are kept. `mirrorIssueState`
* selects which source issues the issue pass lists.
*/
export const MIRROR_OVERRIDE_CHOICE_KEYS = ["releaseType"] as const;
export const MIRROR_OVERRIDE_CHOICE_KEYS = [
"releaseType",
"mirrorIssueState",
] as const;
export type MirrorOverrideChoiceKey =
(typeof MIRROR_OVERRIDE_CHOICE_KEYS)[number];
@@ -86,6 +97,19 @@ export const DEFAULT_RELEASE_LIMIT = 10;
/** Matches `giteaConfigSchema.releaseType`'s default. */
export const DEFAULT_RELEASE_TYPE: ReleaseType = "all";
/** Matches `giteaConfigSchema.mirrorIssueState`'s default. */
export const DEFAULT_ISSUE_STATE: IssueStateFilter = "all";
/**
* Flags whose absent global value means "on". The schema defaults cover saved
* configs, but the resolver also runs against partial configs (tests, older
* callers), where an absent key must not silently turn issue content off.
*/
const FLAG_DEFAULTS: Partial<Record<MirrorOverrideKey, boolean>> = {
mirrorIssueComments: true,
mirrorIssueMedia: true,
};
/**
* How many releases the mirror looks at when any release filter is active.
*
@@ -112,9 +136,14 @@ export const MAX_RELEASE_TAG_PATTERN_LENGTH = 200;
export type ResolvedMirrorOptions = Record<MirrorOverrideKey, boolean> & {
releaseLimit: number;
releaseAssetLimit: number | null;
// Global only: a cleanup policy for the destination rather than something a
// single repository argues about. The ranks it prunes by still come from that
// repository's own effective `releaseAssetLimit`.
pruneOldReleaseAssets: boolean;
releaseType: ReleaseType;
releaseIncludeTags: string[];
releaseExcludeTags: string[];
mirrorIssueState: IssueStateFilter;
};
/**
@@ -128,6 +157,7 @@ export type InheritedMirrorOptions = Partial<Record<MirrorOverrideKey, boolean>>
releaseType?: ReleaseType;
releaseIncludeTags?: string[];
releaseExcludeTags?: string[];
mirrorIssueState?: IssueStateFilter;
skipForks?: boolean;
};
@@ -156,6 +186,20 @@ export function normalizeReleaseAssetLimit(value: unknown): number | undefined {
return limit >= 0 ? limit : undefined;
}
const ISSUE_STATES: readonly IssueStateFilter[] = ["all", "open", "closed"];
/**
* Sanitize an issue-state filter from any tier into one of the known values,
* or `undefined` when it is absent or unusable so the next tier out is
* consulted. Case is folded so an env var or a hand-edited row behaves like
* the value the UI writes.
*/
export function normalizeIssueState(value: unknown): IssueStateFilter | undefined {
if (typeof value !== "string") return undefined;
const state = value.trim().toLowerCase();
return ISSUE_STATES.find((candidate) => candidate === state);
}
const RELEASE_TYPES: readonly ReleaseType[] = ["all", "stable"];
/**
@@ -407,9 +451,10 @@ const LIMIT_NORMALIZERS: Record<
/** Per-key sanitizers for the choice overrides. */
const CHOICE_NORMALIZERS: Record<
MirrorOverrideChoiceKey,
(value: unknown) => ReleaseType | undefined
(value: unknown) => ReleaseType | IssueStateFilter | undefined
> = {
releaseType: normalizeReleaseType,
mirrorIssueState: normalizeIssueState,
};
/** Per-key sanitizers for the tag-pattern overrides. */
@@ -424,8 +469,10 @@ const PATTERN_NORMALIZERS: Record<
/**
* Whether the release at `index` in the newest-first list gets its assets
* uploaded. `null` means no limit. A release past the limit still gets its
* notes and tag, and assets that already reached Gitea are never removed
* because of this rule; it only decides whether a sync uploads.
* notes and tag, and assets that already reached Gitea are not removed by this
* rule on its own; it only decides whether a sync uploads. The opt-in
* `pruneOldReleaseAssets` config switch is what applies the same rank rule in
* reverse and deletes what an earlier, looser limit had uploaded.
*/
export function releaseGetsAssets(
index: number,
@@ -447,6 +494,8 @@ export const STARRED_CLAMPED_KEYS = [
"mirrorReleases",
"mirrorMetadata",
"mirrorIssues",
"mirrorIssueComments",
"mirrorIssueMedia",
"mirrorPullRequests",
"mirrorLabels",
"mirrorMilestones",
@@ -467,6 +516,8 @@ export const STARRED_CLAMPED_KEYS = [
export const GITHUB_ONLY_METADATA_KEYS = [
"mirrorMetadata",
"mirrorIssues",
"mirrorIssueComments",
"mirrorIssueMedia",
"mirrorPullRequests",
"mirrorLabels",
"mirrorMilestones",
@@ -485,6 +536,8 @@ export const PUSH_TARGET_CLAMPED_KEYS = [
"mirrorReleases",
"mirrorMetadata",
"mirrorIssues",
"mirrorIssueComments",
"mirrorIssueMedia",
"mirrorPullRequests",
"mirrorLabels",
"mirrorMilestones",
@@ -509,6 +562,8 @@ export const UI_MIRROR_OVERRIDE_KEYS: MirrorOverrideKey[] = [
"lfs",
"wiki",
"mirrorIssues",
"mirrorIssueComments",
"mirrorIssueMedia",
"mirrorPullRequests",
"mirrorReleases",
"mirrorLabels",
@@ -551,6 +606,9 @@ export function mirrorOptionsToFlags(
releaseIncludeTags?: string[];
releaseExcludeTags?: string[];
mirrorMetadata?: boolean;
issueState?: IssueStateFilter;
issueComments?: boolean;
issueMedia?: boolean;
metadataComponents?: {
issues?: boolean;
pullRequests?: boolean;
@@ -570,6 +628,10 @@ export function mirrorOptionsToFlags(
mirrorReleases: !!mirrorOptions?.mirrorReleases,
wiki: !!components?.wiki,
mirrorIssues: !!components?.issues,
mirrorIssueComments: mirrorOptions?.issueComments ?? true,
mirrorIssueMedia: mirrorOptions?.issueMedia ?? true,
mirrorIssueState:
normalizeIssueState(mirrorOptions?.issueState) ?? DEFAULT_ISSUE_STATE,
mirrorPullRequests: !!components?.pullRequests,
mirrorLabels: !!components?.labels,
mirrorMilestones: !!components?.milestones,
@@ -590,6 +652,7 @@ export const MIRROR_GATING_REASONS = {
starredCodeOnly:
"Starred repos mirror code only (Advanced Options > starred code only)",
labelsFollowIssues: "Issues mirroring already syncs labels",
issuesOff: "Issues are not being mirrored",
releasesOff: "Releases are not being mirrored",
pushTarget: "Not available for GitHub and GitLab destinations: only branches and tags are pushed",
} as const;
@@ -645,6 +708,7 @@ export function getMirrorOverrideGating({
gating.releaseLimit = MIRROR_GATING_REASONS.pushTarget;
gating.releaseAssetLimit = MIRROR_GATING_REASONS.pushTarget;
gating.releaseType = MIRROR_GATING_REASONS.pushTarget;
gating.mirrorIssueState = MIRROR_GATING_REASONS.pushTarget;
gating.releaseIncludeTags = MIRROR_GATING_REASONS.pushTarget;
gating.releaseExcludeTags = MIRROR_GATING_REASONS.pushTarget;
return gating;
@@ -666,6 +730,18 @@ export function getMirrorOverrideGating({
gating.mirrorLabels = MIRROR_GATING_REASONS.labelsFollowIssues;
}
// Issue content filters only mean something while issues are mirrored.
// Inherit the starred/push reason when that is what disabled issues.
if (gating.mirrorIssues) {
gating.mirrorIssueComments = gating.mirrorIssues;
gating.mirrorIssueMedia = gating.mirrorIssues;
gating.mirrorIssueState = gating.mirrorIssues;
} else if (effective.mirrorIssues === false) {
gating.mirrorIssueComments = MIRROR_GATING_REASONS.issuesOff;
gating.mirrorIssueMedia = MIRROR_GATING_REASONS.issuesOff;
gating.mirrorIssueState = MIRROR_GATING_REASONS.issuesOff;
}
// The release limit and the release filters ride along with releases:
// inherit the starred reason when that is what disabled releases, otherwise
// report releases being off. Only an explicit `false` counts; an unknown
@@ -698,6 +774,9 @@ export const MIRROR_OVERRIDE_LABELS: Record<
mirrorReleases: "Releases",
mirrorMetadata: "Metadata",
mirrorIssues: "Issues",
mirrorIssueComments: "Issue comments",
mirrorIssueMedia: "Issue images and attachments",
mirrorIssueState: "Issue state",
mirrorPullRequests: "Pull requests",
mirrorLabels: "Labels",
mirrorMilestones: "Milestones",
@@ -848,7 +927,9 @@ export function resolveMirrorOptions({
} else if (typeof orgValue === "boolean") {
resolved[key] = orgValue;
} else {
resolved[key] = !!globalConfig?.[key];
const globalValue = globalConfig?.[key];
resolved[key] =
typeof globalValue === "boolean" ? globalValue : FLAG_DEFAULTS[key] ?? !!globalValue;
}
}
@@ -879,6 +960,15 @@ export function resolveMirrorOptions({
normalizeReleaseType(globalConfig?.releaseType) ??
DEFAULT_RELEASE_TYPE;
// Issue state follows the same precedence and the same "an unusable value
// falls through to the next tier" rule. Nothing pinned means mirror every
// issue, like every configuration from before this option existed.
resolved.mirrorIssueState =
normalizeIssueState(repo?.mirrorIssueState) ??
normalizeIssueState(org?.mirrorIssueState) ??
normalizeIssueState(globalConfig?.mirrorIssueState) ??
DEFAULT_ISSUE_STATE;
// And the same for the tag lists. No tier pins one means no filter, which is
// what every configuration predating the option does. A tier that pins an
// empty or unusable list is not a pin either, so the next tier out still
@@ -895,6 +985,10 @@ export function resolveMirrorOptions({
normalizeReleaseTagPatterns(globalConfig?.releaseExcludeTags) ??
[];
// No per-repo or per-org tier for the pruning switch, and no stored value
// means the safe default: never delete what the destination already holds.
resolved.pruneOldReleaseAssets = globalConfig?.pruneOldReleaseAssets === true;
// Starred repos with starredCodeOnly mirror code and nothing else. This
// clamp intentionally outranks explicit per-repo overrides: the setting
// exists to stop starred repos from dragging in metadata wholesale.
+118
View File
@@ -0,0 +1,118 @@
/**
* GET /api/activities: the feed must accept every status a mirror job can hold.
*
* "warning" is an activity-only status (see mirrorJobSchema) written when all
* releases were filtered out of a repository. repoStatusEnum does not contain
* it, and parsing with it turned the whole feed into a 500.
*
* mock.module is process-wide in bun, so this file re-runs itself in an
* isolated child (same harness as ../dashboard/index.test.ts).
*/
import { expect, mock, test } from "bun:test";
import { mirrorJobs } from "@/lib/db/schema";
const CHILD_FLAG = "GM_ACTIVITIES_ROUTE_ISOLATED";
const isChild = !!process.env[CHILD_FLAG];
if (!isChild) {
test("activities route - isolated child suite", () => {
const res = Bun.spawnSync({
cmd: [process.execPath, "test", import.meta.path],
env: { ...process.env, [CHILD_FLAG]: "1" },
stdout: "pipe",
stderr: "pipe",
});
if (res.exitCode !== 0) {
console.error(res.stdout.toString());
console.error(res.stderr.toString());
}
expect(res.exitCode).toBe(0);
}, 120_000);
}
if (isChild) {
// Shaped like what the sync writes when the release filters empty a
// destination (gitea.ts), typed as table rows so they track the schema.
const rows: Array<typeof mirrorJobs.$inferSelect> = [
{
id: "job-warning",
userId: "user-1",
repositoryId: "repo-1",
repositoryName: "ollama",
organizationId: null,
organizationName: null,
details:
"12 releases deleted by the release filters; the destination now has none. Check the include/exclude tag patterns and the pre-release setting.",
status: "warning",
message: "All releases removed from mirrors/ollama",
timestamp: new Date("2026-10-29T12:00:05.000Z"),
jobType: "mirror",
batchId: null,
totalItems: null,
completedItems: 0,
itemIds: null,
completedItemIds: [],
inProgress: false,
startedAt: null,
completedAt: null,
lastCheckpoint: null,
},
{
id: "job-mirrored",
userId: "user-1",
repositoryId: "repo-2",
repositoryName: "other",
organizationId: null,
organizationName: null,
details: null,
status: "mirrored",
message: "Successfully mirrored repository: other",
timestamp: new Date("2026-10-29T11:00:02.000Z"),
jobType: "mirror",
batchId: null,
totalItems: null,
completedItems: 0,
itemIds: null,
completedItemIds: [],
inProgress: false,
startedAt: null,
completedAt: null,
lastCheckpoint: null,
},
];
mock.module("@/lib/db", () => ({
db: {
select: () => {
const query: any = {
from: () => query,
where: () => query,
orderBy: () => query,
limit: async () => rows,
then: (resolve: (value: unknown) => void) => resolve(rows),
};
return query;
},
},
mirrorJobs,
}));
mock.module("@/lib/auth-guards", () => ({
requireAuthenticatedUserId: async () => ({ userId: "user-1" }),
}));
const { GET } = await import("./index");
test("activity feed returns warning jobs instead of failing", async () => {
const response = await GET({
request: new Request("http://localhost/api/activities?userId=user-1"),
locals: {},
} as any);
expect(response.status).toBe(200);
const body = await response.json();
expect(body.success).toBe(true);
expect(body.activities.map((entry: any) => entry.status)).toEqual(["warning", "mirrored"]);
expect(body.activities[0].repositoryName).toBe("ollama");
expect(body.activities[0].message).toBe("All releases removed from mirrors/ollama");
});
}
+5 -2
View File
@@ -3,7 +3,7 @@ import { db, mirrorJobs } from "@/lib/db";
import { eq, sql } from "drizzle-orm";
import { createSecureErrorResponse } from "@/lib/utils";
import type { MirrorJob } from "@/lib/db/schema";
import { repoStatusEnum } from "@/types/Repository";
import { activityStatusEnum } from "@/types/Repository";
import { requireAuthenticatedUserId } from "@/lib/auth-guards";
export const GET: APIRoute = async ({ request, locals }) => {
@@ -28,7 +28,10 @@ export const GET: APIRoute = async ({ request, locals }) => {
repositoryName: job.repositoryName ?? undefined,
organizationId: job.organizationId ?? undefined,
organizationName: job.organizationName ?? undefined,
status: repoStatusEnum.parse(job.status),
// Job/activity statuses include the activity-only "warning" (see mirrorJobSchema).
// repoStatusEnum only covers repository statuses, so parsing "warning" with
// it threw a ZodError and turned the whole feed into a 500.
status: activityStatusEnum.parse(job.status),
details: job.details ?? undefined,
message: job.message,
timestamp: job.timestamp,
+57 -10
View File
@@ -1,6 +1,8 @@
import type { APIRoute } from "astro";
import { db, repositories, configs, organizations } from "@/lib/db";
import { db, repositories, configs, organizations, sources } from "@/lib/db";
import { and, eq, sql } from "drizzle-orm";
import { resolveDestinationOwner } from "@/lib/gitea";
import type { Repository } from "@/lib/db/schema";
import {
repositoryVisibilityEnum,
repoStatusEnum,
@@ -63,17 +65,54 @@ export const GET: APIRoute = async ({ request, locals }) => {
.where(eq(organizations.userId, userId));
const organizationDestinations: Record<string, string> = {};
const organizationDestinationByName = new Map<string, string>();
for (const row of orgDestinationRows) {
if (row.destinationOrg) {
organizationDestinations[row.name] = row.destinationOrg;
}
if (!row.destinationOrg) continue;
organizationDestinations[row.name] = row.destinationOrg;
organizationDestinationByName.set(row.name, row.destinationOrg);
}
const response: RepositoryApiResponse = {
success: true,
message: "Repositories fetched successfully",
organizationDestinations,
repositories: rawRepositories.map((repo) => ({
// The repository's own source username is part of the strategy fallback for
// personal repos, so resolve it up front instead of per row.
const sourceRows = await db
.select({ id: sources.id, username: sources.username })
.from(sources)
.where(eq(sources.userId, userId));
const sourceUsernames = new Map(sourceRows.map((row) => [row.id, row.username]));
// The real destination each row mirrors into. This is what the repositories
// page filters by; `organizationDestinations` above remains for the
// destination column's existing per-organization fallback (#416).
const destinationOwners: Record<string, string> = {};
const repositoriesWithDestination = rawRepositories.map((repo) => {
// Matched by exact name, like getOrganizationConfig on the mirror path.
const organizationDestination = repo.organization
? organizationDestinationByName.get(repo.organization)
: undefined;
try {
destinationOwners[repo.id] = resolveDestinationOwner({
config: {
userId,
githubConfig: config.githubConfig as any,
giteaConfig: config.giteaConfig as any,
},
repository: repo as unknown as Repository,
organizationDestination,
sourceUsername: repo.sourceId
? sourceUsernames.get(repo.sourceId)
: undefined,
});
} catch (error) {
// A row that cannot be resolved still belongs in the list; it simply
// has no destination filter value.
console.warn(
`[Repositories] Could not resolve destination owner for ${repo.fullName}:`,
error
);
}
return {
...repo,
organization: repo.organization ?? undefined,
lastMirrored: repo.lastMirrored ?? undefined,
@@ -81,7 +120,15 @@ export const GET: APIRoute = async ({ request, locals }) => {
forkedFrom: repo.forkedFrom ?? undefined,
status: repoStatusEnum.parse(repo.status),
visibility: repositoryVisibilityEnum.parse(repo.visibility),
})),
};
});
const response: RepositoryApiResponse = {
success: true,
message: "Repositories fetched successfully",
organizationDestinations,
destinationOwners,
repositories: repositoriesWithDestination,
};
return jsonResponse({
@@ -172,7 +172,7 @@ describe.skipIf(!isChild)("GET /api/repositories/:id/releases", () => {
{ tag: "v2.0-rc1", prerelease: true },
]);
expect(requestedUrls[0]).toContain("/api/v1/repos/mirrors/hello/releases");
expect(requestedUrls[0]).toContain("per_page=100");
expect(requestedUrls[0]).toContain("limit=50&page=1");
expect(mockGetOwner).not.toHaveBeenCalled();
});
@@ -233,6 +233,21 @@ describe.skipIf(!isChild)("GET /api/repositories/:id/releases", () => {
expect(requestedUrls[1]).toContain("page=2");
});
test("keeps paging while the destination's total says more remain", async () => {
// Gitea caps pages below what was asked for and reports the total.
const page = (count: number, offset: number) =>
new Response(JSON.stringify(batch(count, offset)), {
headers: { "content-type": "application/json", "x-total-count": "35" },
});
servePages([page(30, 0), page(5, 30)]);
const response = await call();
const data = await response.json();
expect(data.releases).toHaveLength(35);
expect(data.truncated).toBe(false);
expect(requestedUrls).toHaveLength(2);
});
test("stops at the cap and marks the list as shortened", async () => {
servePages([batch(100), batch(100, 100), batch(100, 200), batch(1, 300)]);
+28 -39
View File
@@ -7,10 +7,9 @@ import { requireAuthenticatedUserId } from "@/lib/auth-guards";
import { jsonResponse, createSecureErrorResponse } from "@/lib/utils";
import { getGiteaRepoOwnerAsync } from "@/lib/gitea";
import { decryptConfigTokens } from "@/lib/utils/config-encryption";
import { httpGet } from "@/lib/http-client";
import { listDestinationReleases } from "@/lib/destination-listing";
/** Gitea's per-page maximum; the cap below is how far the preview looks. */
const PAGE_SIZE = 100;
/** How far the preview looks. */
const MAX_RELEASES = 300;
type DestinationRelease = {
@@ -76,45 +75,35 @@ export const GET: APIRoute = async ({ params, request, locals }) => {
);
}
const token = decryptConfigTokens(config as Config).giteaConfig.token;
const base = `${config.giteaConfig.url.replace(/\/+$/, "")}/api/v1/repos/${encodeURIComponent(
owner
)}/${encodeURIComponent(repoName)}/releases`;
const listing = await listDestinationReleases<DestinationReleaseGitea>({
baseUrl: config.giteaConfig.url,
token: decryptConfigTokens(config as Config).giteaConfig.token,
owner,
repo: repoName,
max: MAX_RELEASES,
});
// Nothing listed and a failed page means the first page failed. A later
// failure keeps what was listed, as before.
if (!listing.complete && listing.items.length === 0) {
return jsonResponse({
data: {
success: false,
error: `No releases could be read for ${owner}/${repoName} in the destination`,
},
status: 404,
});
}
const releases: DestinationRelease[] = [];
let truncated = false;
for (let page = 1; ; page++) {
const response = await httpGet<DestinationReleaseGitea[]>(
`${base}?per_page=${PAGE_SIZE}&page=${page}`,
{ Authorization: `token ${token}` }
).catch(() => null);
const batch = response?.data;
if (!Array.isArray(batch)) {
if (page === 1) {
return jsonResponse({
data: {
success: false,
error: `No releases could be read for ${owner}/${repoName} in the destination`,
},
status: 404,
});
}
break;
}
for (const release of batch) {
const tag = typeof release.tag_name === "string" ? release.tag_name : "";
if (!tag) continue;
releases.push({
tag,
prerelease: typeof release.prerelease === "boolean" ? release.prerelease : null,
});
}
if (batch.length < PAGE_SIZE) break;
if (releases.length >= MAX_RELEASES) {
truncated = true;
break;
}
for (const release of listing.items) {
const tag = typeof release.tag_name === "string" ? release.tag_name : "";
if (!tag) continue;
releases.push({
tag,
prerelease: typeof release.prerelease === "boolean" ? release.prerelease : null,
});
}
const truncated = listing.truncated;
return jsonResponse({
data: {
+80 -2
View File
@@ -1,7 +1,8 @@
/**
* Route tests for GET /api/storage (#479).
* Route tests for GET /api/storage (#479) and POST /api/storage/refresh.
*
* NOTE: these tests replace @/lib/db and @/lib/utils/auth-helpers with mocks.
* NOTE: these tests replace @/lib/db, @/lib/utils/auth-helpers and
* @/lib/storage-refresh with mocks.
* bun's mock.module is process-wide and leaks into other test files, so in
* the shared test process this file registers NOTHING: it re-runs itself in
* an isolated child process (the same harness as repositories/[id].test.ts).
@@ -32,6 +33,9 @@ let authResult: { user: { id: string } | null; response: Response | null } = {
user: { id: "user-1" },
response: null,
};
let refreshProgress: unknown = null;
let startResult: unknown = null;
let startedFor: string[] = [];
/** Leaf values of each select's where() condition (drizzle conditions are cyclic). */
let selectWhereValues: unknown[][] = [];
@@ -51,6 +55,14 @@ if (isChild) {
requireAuth: mock(async () => authResult),
}));
mock.module("@/lib/storage-refresh", () => ({
getStorageRefreshProgress: () => refreshProgress,
startStorageRefresh: async (userId: string) => {
startedFor.push(userId);
return startResult;
},
}));
mock.module("@/lib/db", () => ({
db: {
select: () => ({
@@ -85,6 +97,7 @@ if (isChild) {
}
const { GET } = isChild ? await import("./index") : { GET: undefined as any };
const { POST } = isChild ? await import("./refresh") : { POST: undefined as any };
function get() {
return GET({
@@ -94,7 +107,29 @@ function get() {
} as any);
}
function post() {
return POST({
params: {},
request: new Request("http://localhost/api/storage/refresh", { method: "POST" }),
locals: { session: { userId: "user-1" } },
} as any);
}
const RUNNING = {
running: true,
total: 4,
done: 1,
measured: 1,
missing: 0,
failed: 0,
startedAt: "2026-10-29T08:00:00.000Z",
finishedAt: null,
};
beforeEach(() => {
refreshProgress = null;
startResult = { ok: true, progress: RUNNING };
startedFor = [];
selectWhereValues = [];
authResult = { user: { id: "user-1" }, response: null };
repoRows = [
@@ -144,3 +179,46 @@ describe.skipIf(!isChild)("GET /api/storage", () => {
expect(selectWhereValues).toEqual([]);
});
});
describe.skipIf(!isChild)("Refresh sizes progress", () => {
test("GET reports no run as null and a run's progress as is", async () => {
expect((await (await get()).json()).refresh).toBeNull();
refreshProgress = RUNNING;
expect((await (await get()).json()).refresh).toEqual(RUNNING);
});
});
describe.skipIf(!isChild)("POST /api/storage/refresh", () => {
test("starts a run for the signed in user and answers 202 with its progress", async () => {
const response = await post();
expect(response.status).toBe(202);
expect(await response.json()).toEqual({ success: true, refresh: RUNNING });
expect(startedFor).toEqual(["user-1"]);
});
test("passes a refusal through with its status", async () => {
startResult = { ok: false, status: 400, error: "Configure the destination URL and token first" };
const response = await post();
expect(response.status).toBe(400);
expect(await response.json()).toEqual({
success: false,
error: "Configure the destination URL and token first",
});
});
test("answers the auth helper's response when signed out", async () => {
authResult = {
user: null,
response: new Response(JSON.stringify({ error: "Unauthorized" }), { status: 401 }),
};
const response = await post();
expect(response.status).toBe(401);
expect(startedFor).toEqual([]);
});
});
+9 -3
View File
@@ -4,11 +4,13 @@ import { db, repositories } from "@/lib/db";
import { createSecureErrorResponse } from "@/lib/utils";
import { requireAuth } from "@/lib/utils/auth-helpers";
import { buildStorageOverview } from "@/lib/repository-storage";
import { getStorageRefreshProgress } from "@/lib/storage-refresh";
/**
* Storage the user's mirrors take up on the destination (#479): totals, per
* destination owner and per repository. Reads only what syncs recorded, so it
* never contacts the destination or the source.
* destination owner and per repository. Reads only what syncs and Refresh
* sizes recorded, so it never contacts the destination or the source. While a
* Refresh sizes run is going, `refresh` carries its progress.
*/
export const GET: APIRoute = async (context) => {
try {
@@ -31,7 +33,11 @@ export const GET: APIRoute = async (context) => {
.from(repositories)
.where(eq(repositories.userId, user!.id));
return new Response(JSON.stringify({ success: true, ...buildStorageOverview(rows) }), {
return new Response(JSON.stringify({
success: true,
...buildStorageOverview(rows),
refresh: getStorageRefreshProgress(user!.id),
}), {
status: 200,
headers: { "Content-Type": "application/json" },
});
+32
View File
@@ -0,0 +1,32 @@
import type { APIRoute } from "astro";
import { createSecureErrorResponse } from "@/lib/utils";
import { requireAuth } from "@/lib/utils/auth-helpers";
import { startStorageRefresh } from "@/lib/storage-refresh";
/**
* Refresh sizes: measure every mirror on the destination now. Starts a run in
* the background, or reports the one already running, and answers with its
* progress; GET /api/storage keeps reporting it until it ends. Only the
* destination is read.
*/
export const POST: APIRoute = async (context) => {
try {
const { user, response } = await requireAuth(context);
if (response) return response;
const result = await startStorageRefresh(user!.id);
if (!result.ok) {
return new Response(JSON.stringify({ success: false, error: result.error }), {
status: result.status,
headers: { "Content-Type": "application/json" },
});
}
return new Response(JSON.stringify({ success: true, refresh: result.progress }), {
status: 202,
headers: { "Content-Type": "application/json" },
});
} catch (error) {
return createSecureErrorResponse(error, "refresh storage sizes", 500);
}
};
+6
View File
@@ -44,6 +44,12 @@ export interface RepositoryApiSuccessResponse {
* list needs it to name a repository's real default (#416).
*/
organizationDestinations?: Record<string, string>;
/**
* Repository id -> the destination owner/org the mirror will actually use.
* Resolved with the same precedence as the mirror path, so the repositories
* page can filter by destination rather than by source organization.
*/
destinationOwners?: Record<string, string>;
}
export interface RepositoryApiErrorResponse {
+12 -1
View File
@@ -1,4 +1,8 @@
import { type Config as ConfigType, type ReleaseType } from "@/lib/db/schema";
import {
type Config as ConfigType,
type IssueStateFilter,
type ReleaseType,
} from "@/lib/db/schema";
export type GiteaOrgVisibility = "public" | "private" | "limited";
export type MirrorStrategy = "preserve" | "single-org" | "flat-user" | "mixed";
@@ -86,6 +90,7 @@ export interface MirrorOptions {
mirrorReleases: boolean;
releaseLimit?: number; // Limit number of releases to mirror (default: 10)
releaseAssetLimit?: number | null; // Upload assets only for the newest N releases; null or absent means all, 0 means none
pruneOldReleaseAssets?: boolean; // Also delete attachments already on releases past releaseAssetLimit (default: false)
releaseType?: ReleaseType; // "all" | "stable": which releases are mirror candidates before the limit applies
releaseIncludeTags?: string[]; // Glob or /regex/ tag patterns; when set, only matching tags are mirror candidates
releaseExcludeTags?: string[]; // Glob or /regex/ tag patterns; matching tags are skipped, and exclude wins over include
@@ -98,6 +103,12 @@ export interface MirrorOptions {
milestones: boolean;
wiki: boolean;
};
/** Which source issues are listed: all, open only, or closed only. */
issueState: IssueStateFilter;
/** Copy comments along with each issue. */
issueComments: boolean;
/** Keep images, videos and attachment links in issue bodies and comments. */
issueMedia: boolean;
}
export interface AdvancedOptions {
+2 -1
View File
@@ -6,7 +6,8 @@ export interface FilterParams {
status?: ActivityStatus | ""; // repos, activity and orgs status
membershipRole?: MembershipRole | ""; //membership role in orgs
owner?: string; // owner of the repos
organization?: string; // organization of the repos
organization?: string; // source organization of the repos
destinationOrganization?: string; // resolved organization/owner on the destination
sort?: string; // repository sort order
hasOverrides?: "overridden" | "default" | ""; // repos with custom mirror options
type?: string; //types in activity log
+1 -1
View File
@@ -157,7 +157,7 @@ export const useCases: UseCase[] = [
title: 'Control Release Asset Storage',
summary: 'Keep release notes for every mirrored release but upload binaries only for the newest few, globally or per organization and repository.',
painPoint: 'A project that ships builds for five platforms every fortnight turns a release mirror into gigabytes of installers nobody will run.',
outcome: 'Two independent limits decide how many releases exist on Gitea and how many of those carry assets, and nothing already uploaded is removed.',
outcome: 'Two independent limits decide how many releases exist on Gitea and how many of those carry assets, and an opt-in switch also deletes the attachments of the older ones.',
tags: ['Releases', 'Storage', 'Overrides'],
},
];
+14 -2
View File
@@ -169,11 +169,23 @@ When several sources are connected, an optional `sourceId` picks which one the o
"success": true,
"totals": { "total": 3554304, "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "measured": 1, "mirrored": 1 },
"owners": [{ "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "total": 3554304, "repositories": 1 }],
"repositories": [{ "id": "…", "fullName": "upstream/app", "location": "archive/app", "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "total": 3554304, "checkedAt": "2026-10-29T03:32:01.000Z" }]
"repositories": [{ "id": "…", "fullName": "upstream/app", "location": "archive/app", "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "total": 3554304, "checkedAt": "2026-10-29T03:32:01.000Z" }],
"refresh": null
}
```
Sizes are bytes. `destinationSize` is Git and LFS together as Gitea or Forgejo reports it, and the release fields cover the assets on the destination's releases. Each sync records them from responses it already fetches, so this endpoint never contacts the destination. A repository is listed once a sync has measured it, and a field not measured yet is `null`. `mirrored` counts the repositories with a mirror on a Gitea or Forgejo destination and `measured` the ones with a size so far; push destinations (GitHub, GitLab) are not measured.
Sizes are bytes. `destinationSize` is Git and LFS together as Gitea or Forgejo reports it, and the release fields cover the assets on the destination's releases. Each sync records them from responses it already fetches, so this endpoint never contacts the destination. A repository is listed once a sync or Refresh sizes has measured it, and a field not measured yet is `null`. `mirrored` counts the repositories with a mirror on a Gitea or Forgejo destination and `measured` the ones with a size so far; push destinations (GitHub, GitLab) are not measured. `refresh` is `null` until Refresh sizes runs, then carries the latest run's progress.
`POST /api/storage/refresh` starts Refresh sizes, which reads every mirror's size from the destination now instead of at its next sync. It lists each destination owner's repositories once, uses the single repository GET for a mirror the listing does not have, and reads the release listing of repositories that mirror releases. The source is never contacted. It answers `202` and runs in the background, one run per user; a call while a run is going answers with that run.
```json
{
"success": true,
"refresh": { "running": true, "total": 7, "done": 0, "measured": 0, "missing": 0, "failed": 0, "startedAt": "2026-10-29T06:20:28.807Z", "finishedAt": null }
}
```
`missing` counts mirrors not found on the destination and `failed` the ones a request failed for; both keep their stored numbers. A read only API key cannot start a run.
### Run cleanup
+2
View File
@@ -38,6 +38,8 @@ Below the connection cards, the mirror settings decide what travels with the cod
Releases have two numbers under the switch. **Latest** is how many releases are mirrored per repository, newest first, ten by default. **Assets for latest** is how many of those get their assets uploaded: leave it empty to upload assets for every mirrored release, set it to `0` for release notes and tags only, or set it to a number to keep assets for the newest N releases while older ones keep their notes and tag. The limit never deletes assets that were already uploaded.
The **Remove old release assets** switch under them turns that last sentence around. Off, which is the default, a sync only ever uploads. On, every sync also deletes the attachments of each mirrored release ranked past **Assets for latest**, so lowering the limit actually frees the disk. It is global rather than per repository, it never removes a release or its notes, and a release whose attachments cannot be listed is left alone and retried on the next sync.
The **Pre-releases** switch under them decides which releases are candidates at all: left on it mirrors every release, turned off it skips everything the source marks as a pre-release. With a filter active the **Latest** count applies to the releases that match, so ten stable releases are kept even when a project ships betas in between, and releases of the other kind are removed from Gitea on the next sync. A release the destination has no pre-release flag for is left alone. The per organization and per repository **Mirror options** dialog offers the same choice under its **Releases** row - **All**, **Stable only**, or **Inherit**, where **Inherit** means the level above - and the release type is `RELEASE_TYPE` in the environment.
**Custom tag rules...** beside that switch opens a dialog with **Include tags** and **Exclude tags**, two comma separated pattern lists that catch the projects shipping release candidates without marking them as pre-releases. The same lists sit under the **Releases** row in the per organization and per repository dialog, where leaving both lists empty means the level above, and in the environment they are `RELEASE_INCLUDE_TAGS` and `RELEASE_EXCLUDE_TAGS`.
+2 -1
View File
@@ -165,7 +165,8 @@ Only used when `DESTINATION_PROVIDER` is `github` or `gitlab`.
| `RELEASE_TYPE` | `all` | Which releases are candidates: `all` or `stable` (skip what the source marks as a pre-release). With a filter active the limit counts the releases that match, and releases of the other kind are removed from Gitea on the next sync. |
| `RELEASE_INCLUDE_TAGS` | empty | Comma separated tag patterns; only releases whose tag matches are mirrored. A pattern is a glob over the whole tag (`v1.*`, `*-rc.*`) or a regular expression in slashes (`/^v\d+\.\d+\.\d+$/`), matched case-insensitively. |
| `RELEASE_EXCLUDE_TAGS` | empty | Comma separated tag patterns in the same syntax; matching releases are skipped, and an excluded tag wins over `RELEASE_INCLUDE_TAGS`. |
| `RELEASE_ASSET_LIMIT` | unset | Upload release assets only for the newest N mirrored releases; older releases still get their notes and tag. Unset means assets for every mirrored release, `0` means notes only. Never deletes assets already uploaded. |
| `RELEASE_ASSET_LIMIT` | unset | Upload release assets only for the newest N mirrored releases; older releases still get their notes and tag. Unset means assets for every mirrored release, `0` means notes only. Lowering it never deletes assets already uploaded unless `PRUNE_OLD_RELEASE_ASSETS` is on. |
| `PRUNE_OLD_RELEASE_ASSETS` | `false` | Opt-in cleanup for what the asset limit left behind. On, every sync also deletes the attachments of each mirrored release ranked past `RELEASE_ASSET_LIMIT`; the releases and their notes stay. |
| `MIRROR_WIKI` | `false` | Clone wiki content. |
| `MIRROR_ISSUE_CONCURRENCY` | `3` | Issues processed in parallel. |
| `MIRROR_PULL_REQUEST_CONCURRENCY` | `5` | Pull requests processed in parallel. |
@@ -45,10 +45,12 @@ Patterns are comma separated, and each one is either a glob over the whole tag o
Matching ignores case, and a pattern that cannot compile is ignored rather than failing the run. A tag filter widens the search exactly like the type filter does, so **Latest 1** with `v*` still finds the newest real release behind a wall of nightlies. Releases whose tag no longer matches are removed from Gitea on the next sync, and unlike the type filter a tag pattern can clean up releases the destination has no pre-release flag for, because it only needs the tag name.
## Nothing already uploaded is deleted
## Nothing already uploaded is deleted, unless you ask
The asset limit only decides whether a sync uploads. Lowering it later never removes assets that already reached Gitea. If you mirrored everything for a year and then set the limit to 2, the old files stay until you delete them in Gitea yourself. That is the safe direction for a backup tool, and it means the setting is cheap to experiment with.
The **Remove old release assets** switch under the release settings changes that on purpose. With it on, a sync also prunes: the attachments of every mirrored release ranked past **Assets for latest** are deleted from Gitea, so lowering the limit actually shrinks what is on disk. It is a global switch, it stays off until you turn it on, and it only touches the files - the release and its notes stay. A release whose attachment listing cannot be read keeps its files and is retried on the next sync.
## Step-by-step
### 1. Set a global default
@@ -71,6 +73,7 @@ For deployments configured from the environment:
MIRROR_RELEASES=true
RELEASE_LIMIT=25
RELEASE_ASSET_LIMIT=3 # unset for all, 0 for notes only
PRUNE_OLD_RELEASE_ASSETS=true # also delete attachments past that count
RELEASE_TYPE=stable # all or stable
RELEASE_INCLUDE_TAGS=v* # comma separated globs or /regex/
RELEASE_EXCLUDE_TAGS=*-rc.*
@@ -82,6 +85,8 @@ RELEASE_EXCLUDE_TAGS=*-rc.*
Take a project that publishes a release every two weeks with five platform builds of 40 MB each. That is 200 MB per release and about 5 GB per year of assets, for one repository. With the release limit at 25 and no asset limit, the first mirror pulls about 5 GB and every sync afterwards adds the newest release's 200 MB. With **Assets for latest** at 3, the first mirror pulls 600 MB. Each later release still adds 200 MB, because a new release is inside the limit and gets its files, and the files of the release that dropped out of the top three stay on disk until you remove them. The ongoing growth is the same; what you save is the backfill of every historical build.
With **Remove old release assets** on, the picture changes: when a release drops out of the top three, its 200 MB go with it, so the destination hovers around 600 MB instead of growing without bound.
Multiply by the number of repositories that ship binaries and compare with the free space on the Gitea host. Gitea stores release attachments in its own attachment storage, separate from the git objects and from LFS, so check that path when you plan capacity.
## Where releases apply