mirror of
https://github.com/RayLabsHQ/gitea-mirror.git
synced 2026-10-30 22:21:58 +01:00
feat(storage): Refresh sizes on the Storage page (#483)
- Refresh sizes measures every mirror on the destination now instead of at its next sync. It only reads the destination: one repository listing per owner, the single repository GET for a mirror the listing does not have, and the release listing of repositories that mirror releases. POST /api/storage/refresh runs it in the background; GET /api/storage reports its progress. - Destination release listings stopped after 30 releases: Gitea and Forgejo ignore per_page and answer 30 a page. The retention pass, the release filter preview and the refresh now share one pager that asks with limit and ends at X-Total-Count.
This commit is contained in:
1 parent
f2c689a402
commit
3f65a40a7f
16 files changed
+1285
-97
No files matched your search
@@ -7,7 +7,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- **Refresh sizes** on the Storage page measures every mirror now instead of waiting for its next sync, so the page fills in on a fresh install. It only reads the destination: one repository listing per destination owner, the single repository GET for a mirror the listing does not have, and the release listing of repositories that mirror releases. It runs in the background with its progress on the button (`POST /api/storage/refresh`)
|
||||
|
||||
### Fixed
|
||||
- Destination release listings stopped after 30 releases. Gitea and Forgejo ignore `per_page` and answer 30 a page, and the listing stopped on any page shorter than the 100 it asked for. The release limit only trimmed within the newest 30 on each sync, the Storage page counted at most 30 releases per repository, and the release filter preview showed only the newest 30 tags. Listings now page with `limit` until the total the destination reports
|
||||
- The sync button on an archived repository says **Sync** like every other row instead of **Manual Sync**, so the wider label no longer pushes the row's status badge out of line. Hovering it explains that automatic syncs are off for archived mirrors
|
||||
- The Activity Log summary no longer shows a repository as syncing after a quick sync finished (#454). Events are stored to the second, and a sync that starts and finishes in the same second could come back in the wrong order, so the start event was counted as the current state. Activities now come back newest first within the same second too, and the summary counts a finish over a start from the same second
|
||||
- Issue and pull request passes over large repositories finish across rate limit resets (#449 follow-up)
|
||||
|
||||
+14
-2
@@ -165,11 +165,23 @@ When several sources are connected, an optional `sourceId` picks which one the o
|
||||
"success": true,
|
||||
"totals": { "total": 3554304, "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "measured": 1, "mirrored": 1 },
|
||||
"owners": [{ "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "total": 3554304, "repositories": 1 }],
|
||||
"repositories": [{ "id": "…", "fullName": "upstream/app", "location": "archive/app", "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "total": 3554304, "checkedAt": "2026-10-29T03:32:01.000Z" }]
|
||||
"repositories": [{ "id": "…", "fullName": "upstream/app", "location": "archive/app", "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "total": 3554304, "checkedAt": "2026-10-29T03:32:01.000Z" }],
|
||||
"refresh": null
|
||||
}
|
||||
```
|
||||
|
||||
Sizes are bytes. `destinationSize` is Git and LFS together as Gitea or Forgejo reports it, and the release fields cover the assets on the destination's releases. Each sync records them from responses it already fetches, so this endpoint never contacts the destination. A repository is listed once a sync has measured it, and a field not measured yet is `null`. `mirrored` counts the repositories with a mirror on a Gitea or Forgejo destination and `measured` the ones with a size so far; push destinations (GitHub, GitLab) are not measured.
|
||||
Sizes are bytes. `destinationSize` is Git and LFS together as Gitea or Forgejo reports it, and the release fields cover the assets on the destination's releases. Each sync records them from responses it already fetches, so this endpoint never contacts the destination. A repository is listed once a sync or Refresh sizes has measured it, and a field not measured yet is `null`. `mirrored` counts the repositories with a mirror on a Gitea or Forgejo destination and `measured` the ones with a size so far; push destinations (GitHub, GitLab) are not measured. `refresh` is `null` until Refresh sizes runs, then carries the latest run's progress.
|
||||
|
||||
`POST /api/storage/refresh` starts Refresh sizes, which reads every mirror's size from the destination now instead of at its next sync. It lists each destination owner's repositories once, uses the single repository GET for a mirror the listing does not have, and reads the release listing of repositories that mirror releases. The source is never contacted. It answers `202` and runs in the background, one run per user; a call while a run is going answers with that run.
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"refresh": { "running": true, "total": 7, "done": 0, "measured": 0, "missing": 0, "failed": 0, "startedAt": "2026-10-29T06:20:28.807Z", "finishedAt": null }
|
||||
}
|
||||
```
|
||||
|
||||
`missing` counts mirrors not found on the destination and `failed` the ones a request failed for; both keep their stored numbers. A read only API key cannot start a run.
|
||||
|
||||
### Export as CSV
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { useCallback, useEffect, useMemo, useState } from "react";
|
||||
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
|
||||
import {
|
||||
ArrowDown,
|
||||
ArrowUp,
|
||||
@@ -6,11 +6,13 @@ import {
|
||||
HardDrive,
|
||||
Info,
|
||||
Package,
|
||||
RefreshCw,
|
||||
ScanSearch,
|
||||
Search,
|
||||
} from "lucide-react";
|
||||
import { toast } from "sonner";
|
||||
import { StatusCard } from "@/components/dashboard/StatusCard";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
|
||||
import { Skeleton } from "@/components/ui/skeleton";
|
||||
import {
|
||||
@@ -30,12 +32,20 @@ import { useAuth } from "@/hooks/useAuth";
|
||||
import { useGiteaConfig } from "@/hooks/useGiteaConfig";
|
||||
import { useLiveRefresh } from "@/hooks/useLiveRefresh";
|
||||
import { useNavigation } from "@/components/layout/MainLayout";
|
||||
import { apiRequest, cn, formatLastSyncTime } from "@/lib/utils";
|
||||
import { apiRequest, cn, formatLastSyncTime, showErrorToast } from "@/lib/utils";
|
||||
import { buildGiteaWebUrl } from "@/lib/gitea-url";
|
||||
import { formatBytes } from "@/lib/utils/format-bytes";
|
||||
import type { StorageOverview, StorageRepository } from "@/lib/repository-storage";
|
||||
import type { StorageRefreshProgress } from "@/lib/storage-refresh";
|
||||
|
||||
type StorageApiResponse = ({ success: true } & StorageOverview) | { success: false; error?: string };
|
||||
type StorageApiResponse =
|
||||
| ({ success: true; refresh: StorageRefreshProgress | null } & StorageOverview)
|
||||
| { success: false; error?: string };
|
||||
|
||||
type RefreshApiResponse = { success: true; refresh: StorageRefreshProgress } | { success: false; error?: string };
|
||||
|
||||
// While Refresh sizes runs, the page polls the overview for its progress.
|
||||
const REFRESH_POLL_MS = 2000;
|
||||
|
||||
type SortKey = "fullName" | "owner" | "destinationSize" | "releaseAssetsSize" | "releaseAssetsCount" | "total" | "checkedAt";
|
||||
|
||||
@@ -160,6 +170,45 @@ function compareRepositories(a: StorageRepository, b: StorageRepository, key: So
|
||||
}
|
||||
}
|
||||
|
||||
function RefreshSizesButton({
|
||||
refresh,
|
||||
starting,
|
||||
onClick,
|
||||
className,
|
||||
}: {
|
||||
refresh: StorageRefreshProgress | null;
|
||||
starting: boolean;
|
||||
onClick: () => void;
|
||||
className?: string;
|
||||
}) {
|
||||
const running = !!refresh?.running;
|
||||
return (
|
||||
<TooltipProvider>
|
||||
<Tooltip>
|
||||
<TooltipTrigger asChild>
|
||||
<Button variant="outline" className={cn("h-10", className)} onClick={onClick} disabled={running || starting}>
|
||||
<RefreshCw className={cn("h-4 w-4", (running || starting) && "animate-spin")} />
|
||||
{running ? `Measuring ${refresh.done} of ${refresh.total}` : "Refresh sizes"}
|
||||
</Button>
|
||||
</TooltipTrigger>
|
||||
<TooltipContent className="max-w-xs">
|
||||
Reads every mirror's size from the destination now instead of waiting for its next sync. Nothing is
|
||||
asked of the source.
|
||||
</TooltipContent>
|
||||
</Tooltip>
|
||||
</TooltipProvider>
|
||||
);
|
||||
}
|
||||
|
||||
function announceRefresh(progress: StorageRefreshProgress) {
|
||||
const parts = [`Measured ${progress.measured} ${progress.measured === 1 ? "repository" : "repositories"}`];
|
||||
if (progress.missing > 0) parts.push(`${progress.missing} not found on the destination`);
|
||||
if (progress.failed > 0) parts.push(`${progress.failed} could not be read and kept their sizes`);
|
||||
const message = parts.join(", ");
|
||||
if (progress.missing > 0 || progress.failed > 0) toast.warning(message);
|
||||
else toast.success(message);
|
||||
}
|
||||
|
||||
function StorageSkeleton() {
|
||||
return (
|
||||
<div className="flex flex-col gap-y-6">
|
||||
@@ -189,6 +238,10 @@ export function Storage() {
|
||||
const { giteaConfig } = useGiteaConfig();
|
||||
|
||||
const [overview, setOverview] = useState<StorageOverview | null>(null);
|
||||
const [refresh, setRefresh] = useState<StorageRefreshProgress | null>(null);
|
||||
const [startingRefresh, setStartingRefresh] = useState(false);
|
||||
// Set once this page has seen a run going, so its end is announced once.
|
||||
const watchingRefreshRef = useRef(false);
|
||||
const [isLoading, setIsLoading] = useState(true);
|
||||
const [search, setSearch] = useState("");
|
||||
const [ownerFilter, setOwnerFilter] = useState("all");
|
||||
@@ -201,6 +254,13 @@ export function Storage() {
|
||||
const response = await apiRequest<StorageApiResponse>("/storage", { method: "GET" });
|
||||
if (response.success) {
|
||||
setOverview(response);
|
||||
setRefresh(response.refresh);
|
||||
if (response.refresh?.running) {
|
||||
watchingRefreshRef.current = true;
|
||||
} else if (watchingRefreshRef.current && response.refresh) {
|
||||
watchingRefreshRef.current = false;
|
||||
announceRefresh(response.refresh);
|
||||
}
|
||||
} else if (!isLiveRefresh) {
|
||||
toast.error(response.error || "Error fetching storage");
|
||||
}
|
||||
@@ -222,6 +282,29 @@ export function Storage() {
|
||||
|
||||
useEffect(() => registerRefreshCallback(() => fetchOverview(true)), [registerRefreshCallback, fetchOverview]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!refresh?.running) return;
|
||||
const timer = setInterval(() => fetchOverview(true), REFRESH_POLL_MS);
|
||||
return () => clearInterval(timer);
|
||||
}, [refresh?.running, fetchOverview]);
|
||||
|
||||
const startRefresh = async () => {
|
||||
setStartingRefresh(true);
|
||||
try {
|
||||
const response = await apiRequest<RefreshApiResponse>("/storage/refresh", { method: "POST" });
|
||||
if (response.success) {
|
||||
watchingRefreshRef.current = true;
|
||||
setRefresh(response.refresh);
|
||||
} else {
|
||||
toast.error(response.error || "Could not start Refresh sizes");
|
||||
}
|
||||
} catch (error) {
|
||||
showErrorToast(error, toast);
|
||||
} finally {
|
||||
setStartingRefresh(false);
|
||||
}
|
||||
};
|
||||
|
||||
const handleSort = (key: SortKey) =>
|
||||
setSort((prev) =>
|
||||
prev.key === key
|
||||
@@ -256,8 +339,10 @@ export function Storage() {
|
||||
<h3 className="text-lg font-medium">No sizes recorded yet</h3>
|
||||
<p className="text-sm text-muted-foreground mt-1 max-w-md">
|
||||
Each sync records how much space the mirror takes on the destination, from data the sync
|
||||
already fetches. Sizes show up here after the next sync.
|
||||
already fetches. Sizes show up here after the next sync, or now with Refresh sizes, which only
|
||||
reads the destination.
|
||||
</p>
|
||||
<RefreshSizesButton refresh={refresh} starting={startingRefresh} onClick={startRefresh} className="mt-6" />
|
||||
</CardContent>
|
||||
</Card>
|
||||
);
|
||||
@@ -346,6 +431,12 @@ export function Storage() {
|
||||
))}
|
||||
</SelectContent>
|
||||
</Select>
|
||||
<RefreshSizesButton
|
||||
refresh={refresh}
|
||||
starting={startingRefresh}
|
||||
onClick={startRefresh}
|
||||
className="w-full sm:w-auto shrink-0"
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div className="overflow-x-auto" data-testid="storage-table">
|
||||
@@ -446,8 +537,8 @@ export function Storage() {
|
||||
</div>
|
||||
<p className="text-xs text-muted-foreground">
|
||||
Sizes are what the destination reports: Git and LFS together, plus the release assets it stores. Each sync
|
||||
records them from data it already fetches. The Git size is read before the sync pulls, so it trails by one
|
||||
sync.
|
||||
records them from data it already fetches, and Refresh sizes reads them from the destination on demand. The
|
||||
Git size a sync records is read before it pulls, so it trails by one sync.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
/**
|
||||
* Tests for the destination pager. Gitea 1.27.3 answered `per_page=100` with
|
||||
* 30 releases and `limit=50` with up to 50, plus `X-Total-Count`; the fake
|
||||
* destination below pages the same way.
|
||||
*
|
||||
* Several test files mock @/lib/http-client, and bun's mock.module leaks across
|
||||
* files, so this file re-runs itself in an isolated child to get the real one.
|
||||
*/
|
||||
import { describe, test, expect, afterEach } from "bun:test";
|
||||
|
||||
const CHILD_FLAG = "GM_DESTINATION_LISTING_ISOLATED";
|
||||
const isChild = !!process.env[CHILD_FLAG];
|
||||
|
||||
if (!isChild) {
|
||||
test("destination listing - isolated child suite", () => {
|
||||
const res = Bun.spawnSync({
|
||||
cmd: [process.execPath, "test", import.meta.path],
|
||||
env: { ...process.env, [CHILD_FLAG]: "1" },
|
||||
stdout: "pipe",
|
||||
stderr: "pipe",
|
||||
});
|
||||
if (res.exitCode !== 0) {
|
||||
console.error(res.stdout.toString());
|
||||
console.error(res.stderr.toString());
|
||||
}
|
||||
expect(res.exitCode).toBe(0);
|
||||
}, 60_000);
|
||||
}
|
||||
|
||||
const { listDestinationPages, listDestinationReleases, listDestinationOwnerRepositories } = isChild
|
||||
? await import("./destination-listing")
|
||||
: ({} as typeof import("./destination-listing"));
|
||||
|
||||
const realFetch = globalThis.fetch;
|
||||
let requestedUrls: string[] = [];
|
||||
|
||||
/**
|
||||
* A destination holding `count` items that serves `pageCap` of them per page
|
||||
* at most, whatever `limit` asks for.
|
||||
*/
|
||||
function serve(
|
||||
count: number,
|
||||
{ pageCap = 50, totalHeader = true, failPage }: { pageCap?: number; totalHeader?: boolean; failPage?: number } = {}
|
||||
) {
|
||||
requestedUrls = [];
|
||||
globalThis.fetch = (async (input: RequestInfo | URL) => {
|
||||
const url = String(input);
|
||||
requestedUrls.push(url);
|
||||
const params = new URL(url).searchParams;
|
||||
const page = Number(params.get("page") ?? "1");
|
||||
if (page === failPage) return new Response("boom", { status: 500, statusText: "Internal Server Error" });
|
||||
const size = Math.min(Number(params.get("limit") ?? "30"), pageCap);
|
||||
const items = Array.from({ length: count }, (_, i) => ({ id: i + 1 })).slice((page - 1) * size, page * size);
|
||||
const headers: Record<string, string> = { "content-type": "application/json" };
|
||||
if (totalHeader) headers["x-total-count"] = String(count);
|
||||
return new Response(JSON.stringify(items), { status: 200, headers });
|
||||
}) as typeof fetch;
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
globalThis.fetch = realFetch;
|
||||
});
|
||||
|
||||
describe.skipIf(!isChild)("listDestinationPages", () => {
|
||||
test("asks with limit and follows X-Total-Count past pages shorter than asked", async () => {
|
||||
serve(35, { pageCap: 30 });
|
||||
|
||||
const listing = await listDestinationPages({ url: "https://gitea.example.com/api/v1/repos/o/r/releases", token: "t" });
|
||||
|
||||
expect(listing.items).toHaveLength(35);
|
||||
expect(listing.complete).toBe(true);
|
||||
expect(requestedUrls).toEqual([
|
||||
"https://gitea.example.com/api/v1/repos/o/r/releases?limit=50&page=1",
|
||||
"https://gitea.example.com/api/v1/repos/o/r/releases?limit=50&page=2",
|
||||
]);
|
||||
});
|
||||
|
||||
test("ends on the page that reaches the total, without asking for an empty one", async () => {
|
||||
serve(100);
|
||||
|
||||
const listing = await listDestinationPages({ url: "https://gitea.example.com/x", token: "t" });
|
||||
|
||||
expect(listing.items).toHaveLength(100);
|
||||
expect(requestedUrls).toHaveLength(2);
|
||||
});
|
||||
|
||||
test("without the total header, ends on a short page", async () => {
|
||||
serve(70, { totalHeader: false });
|
||||
|
||||
const listing = await listDestinationPages({ url: "https://gitea.example.com/x", token: "t" });
|
||||
|
||||
expect(listing.items).toHaveLength(70);
|
||||
expect(listing.complete).toBe(true);
|
||||
expect(requestedUrls).toHaveLength(2);
|
||||
});
|
||||
|
||||
test("ends on an empty page", async () => {
|
||||
serve(0);
|
||||
|
||||
const listing = await listDestinationPages({ url: "https://gitea.example.com/x", token: "t" });
|
||||
|
||||
expect(listing).toEqual({ items: [], complete: true, truncated: false });
|
||||
});
|
||||
|
||||
test("a failed page marks the listing incomplete and keeps what came before", async () => {
|
||||
serve(120, { failPage: 2 });
|
||||
|
||||
const listing = await listDestinationPages({ url: "https://gitea.example.com/x", token: "t" });
|
||||
|
||||
expect(listing.items).toHaveLength(50);
|
||||
expect(listing.complete).toBe(false);
|
||||
});
|
||||
|
||||
test("stops at max and says whether more remained", async () => {
|
||||
serve(120);
|
||||
const capped = await listDestinationPages({ url: "https://gitea.example.com/x", token: "t", max: 60 });
|
||||
expect(capped.items).toHaveLength(60);
|
||||
expect(capped.truncated).toBe(true);
|
||||
expect(requestedUrls).toHaveLength(2);
|
||||
|
||||
serve(60);
|
||||
const exact = await listDestinationPages({ url: "https://gitea.example.com/x", token: "t", max: 60 });
|
||||
expect(exact.items).toHaveLength(60);
|
||||
expect(exact.truncated).toBe(false);
|
||||
});
|
||||
|
||||
test("sends the token", async () => {
|
||||
let authorization: string | null | undefined;
|
||||
globalThis.fetch = (async (_input: RequestInfo | URL, init?: RequestInit) => {
|
||||
authorization = new Headers(init?.headers).get("authorization");
|
||||
return new Response("[]", { headers: { "content-type": "application/json" } });
|
||||
}) as typeof fetch;
|
||||
|
||||
await listDestinationPages({ url: "https://gitea.example.com/x", token: "secret" });
|
||||
|
||||
expect(authorization).toBe("token secret");
|
||||
});
|
||||
});
|
||||
|
||||
describe.skipIf(!isChild)("listing URLs", () => {
|
||||
test("releases of one repository", async () => {
|
||||
serve(1);
|
||||
await listDestinationReleases({ baseUrl: "https://gitea.example.com/", token: "t", owner: "my org", repo: "app" });
|
||||
expect(requestedUrls[0]).toBe("https://gitea.example.com/api/v1/repos/my%20org/app/releases?limit=50&page=1");
|
||||
});
|
||||
|
||||
test("repositories of one owner, user or organization", async () => {
|
||||
serve(1);
|
||||
await listDestinationOwnerRepositories({ baseUrl: "https://gitea.example.com", token: "t", owner: "archive" });
|
||||
expect(requestedUrls[0]).toBe("https://gitea.example.com/api/v1/users/archive/repos?limit=50&page=1");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,108 @@
|
||||
/**
|
||||
* Paged listings from a Gitea or Forgejo destination.
|
||||
*
|
||||
* Both servers size pages by `limit`, capped at MAX_RESPONSE_ITEMS (50 unless
|
||||
* an admin changed it), and ignore `per_page` on most endpoints. Asking for 100
|
||||
* releases with `per_page` got 30 back, so a loop that stopped on a short page
|
||||
* only ever saw the newest 30. Pages are asked for with `limit` and a listing
|
||||
* ends once `X-Total-Count` is reached; without that header, on a short page.
|
||||
*/
|
||||
|
||||
import { httpGet } from "./http-client";
|
||||
|
||||
export const DESTINATION_PAGE_SIZE = 50;
|
||||
|
||||
// A server that ignores `page` would otherwise be listed forever.
|
||||
const MAX_PAGES = 1000;
|
||||
|
||||
export interface DestinationListing<T> {
|
||||
items: T[];
|
||||
/** False when a page failed; `items` then holds the pages before it. */
|
||||
complete: boolean;
|
||||
/** True when `max` stopped the listing while more items remained. */
|
||||
truncated: boolean;
|
||||
}
|
||||
|
||||
/** Every item of a paged destination endpoint. `url` must not carry a query. */
|
||||
export async function listDestinationPages<T = unknown>({
|
||||
url,
|
||||
token,
|
||||
max,
|
||||
}: {
|
||||
url: string;
|
||||
token: string;
|
||||
/** Stop once this many items are listed. */
|
||||
max?: number;
|
||||
}): Promise<DestinationListing<T>> {
|
||||
const items: T[] = [];
|
||||
|
||||
for (let page = 1; page <= MAX_PAGES; page++) {
|
||||
const response = await httpGet<T[]>(`${url}?limit=${DESTINATION_PAGE_SIZE}&page=${page}`, {
|
||||
Authorization: `token ${token}`,
|
||||
}).catch(() => null);
|
||||
const batch = response?.data;
|
||||
if (!Array.isArray(batch)) return { items, complete: false, truncated: false };
|
||||
if (batch.length === 0) return { items, complete: true, truncated: false };
|
||||
|
||||
items.push(...batch);
|
||||
|
||||
const totalHeader = response?.headers?.get?.("x-total-count");
|
||||
const total = totalHeader ? Number(totalHeader) : NaN;
|
||||
const done = Number.isFinite(total) ? items.length >= total : batch.length < DESTINATION_PAGE_SIZE;
|
||||
|
||||
if (max !== undefined && items.length >= max) {
|
||||
return {
|
||||
items: items.slice(0, max),
|
||||
complete: true,
|
||||
truncated: done ? items.length > max : true,
|
||||
};
|
||||
}
|
||||
if (done) return { items, complete: true, truncated: false };
|
||||
}
|
||||
|
||||
return { items, complete: false, truncated: false };
|
||||
}
|
||||
|
||||
function apiBase(baseUrl: string): string {
|
||||
return `${baseUrl.replace(/\/+$/, "")}/api/v1`;
|
||||
}
|
||||
|
||||
/** Every release the destination holds for one repository. */
|
||||
export async function listDestinationReleases<T = unknown>({
|
||||
baseUrl,
|
||||
token,
|
||||
owner,
|
||||
repo,
|
||||
max,
|
||||
}: {
|
||||
baseUrl: string;
|
||||
token: string;
|
||||
owner: string;
|
||||
repo: string;
|
||||
max?: number;
|
||||
}): Promise<DestinationListing<T>> {
|
||||
return listDestinationPages<T>({
|
||||
url: `${apiBase(baseUrl)}/repos/${encodeURIComponent(owner)}/${encodeURIComponent(repo)}/releases`,
|
||||
token,
|
||||
max,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Every repository of one destination owner the token can see, private ones
|
||||
* included. Works for users and organizations alike.
|
||||
*/
|
||||
export async function listDestinationOwnerRepositories<T = unknown>({
|
||||
baseUrl,
|
||||
token,
|
||||
owner,
|
||||
}: {
|
||||
baseUrl: string;
|
||||
token: string;
|
||||
owner: string;
|
||||
}): Promise<DestinationListing<T>> {
|
||||
return listDestinationPages<T>({
|
||||
url: `${apiBase(baseUrl)}/users/${encodeURIComponent(owner)}/repos`,
|
||||
token,
|
||||
});
|
||||
}
|
||||
@@ -150,6 +150,11 @@ interface DestinationOptions {
|
||||
destinationReleases?: Array<Record<string, unknown>>;
|
||||
/** The destination answers the retention listing with an error. */
|
||||
destinationListingFails?: boolean;
|
||||
/**
|
||||
* Serve the listing this many releases per page at most, with
|
||||
* X-Total-Count, as Gitea does (30 when asked with `per_page`).
|
||||
*/
|
||||
destinationPageCap?: number;
|
||||
/** Destination release ids whose DELETE fails. */
|
||||
failDeleteIds?: number[];
|
||||
}
|
||||
@@ -233,9 +238,15 @@ function fakeWorld(options: DestinationOptions) {
|
||||
|
||||
// The retention pass lists everything the destination has.
|
||||
if (method === "GET" && url.startsWith(`${DEST_API}/releases?`)) {
|
||||
return options.destinationListingFails
|
||||
? new Response("boom", { status: 500 })
|
||||
: json(options.destinationReleases ?? []);
|
||||
if (options.destinationListingFails) return new Response("boom", { status: 500 });
|
||||
const listed = options.destinationReleases ?? [];
|
||||
if (!options.destinationPageCap) return json(listed);
|
||||
const params = new URL(url).searchParams;
|
||||
const size = Math.min(Number(params.get("limit") ?? "30"), options.destinationPageCap);
|
||||
const page = Number(params.get("page") ?? "1");
|
||||
return new Response(JSON.stringify(listed.slice((page - 1) * size, page * size)), {
|
||||
headers: { "content-type": "application/json", "x-total-count": String(listed.length) },
|
||||
});
|
||||
}
|
||||
|
||||
throw new Error(`unexpected request: ${method} ${url}`);
|
||||
@@ -567,6 +578,39 @@ describe.skipIf(!isChild)("release asset storage (#479)", () => {
|
||||
expect(storageRecords).toEqual([{ repositoryId: "repo-1", summary: { count: 3, bytes: 1150 } }]);
|
||||
});
|
||||
|
||||
it("lists past the first page when the destination pages below what was asked", async () => {
|
||||
// 35 releases, served 30 a page: the old per_page=100 loop stopped at 30.
|
||||
const listed = Array.from({ length: 35 }, (_, i) =>
|
||||
withAssets(destinationRelease(`v1.${i}.0`, 100 + i, false), [10])
|
||||
);
|
||||
const world = fakeWorld({
|
||||
sourceReleases: [giteaRelease("v2.0.0")],
|
||||
destinationReleases: listed,
|
||||
destinationPageCap: 30,
|
||||
});
|
||||
|
||||
await mirrorFromGitea(giteaConnection("source-token"), { releaseLimit: 50 });
|
||||
|
||||
expect(storageRecords).toEqual([{ repositoryId: "repo-1", summary: { count: 35, bytes: 350 } }]);
|
||||
expect(world.requests.filter((r) => r.url.startsWith(`${DEST_API}/releases?`))).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("enforces the release limit across every page", async () => {
|
||||
const listed = Array.from({ length: 35 }, (_, i) =>
|
||||
withAssets(destinationRelease(`v1.${i}.0`, 100 + i, false), [10])
|
||||
);
|
||||
const world = fakeWorld({
|
||||
sourceReleases: [giteaRelease("v2.0.0")],
|
||||
destinationReleases: listed,
|
||||
destinationPageCap: 30,
|
||||
});
|
||||
|
||||
await mirrorFromGitea(giteaConnection("source-token"), { releaseLimit: 10 });
|
||||
|
||||
// The old loop saw 30 of the 35 and deleted 20, leaving 15.
|
||||
expect(world.requests.filter((r) => r.method === "DELETE")).toHaveLength(25);
|
||||
});
|
||||
|
||||
it("records nothing when the listing fails", async () => {
|
||||
fakeWorld({
|
||||
sourceReleases: [giteaRelease("v2.0.0")],
|
||||
|
||||
+11
-31
@@ -37,6 +37,7 @@ import { assertMirrorCredentialUpdateSupported } from "./gitea-mirror-credential
|
||||
import { persistMetadataPassProgress } from "./metadata-progress-store";
|
||||
import { summarizeReleaseAssets } from "./repository-storage";
|
||||
import { recordReleaseAssetStorage } from "./repository-storage-store";
|
||||
import { listDestinationReleases } from "./destination-listing";
|
||||
import {
|
||||
advanceMetadataSyncCursor,
|
||||
buildPassProgress,
|
||||
@@ -4376,42 +4377,21 @@ async function mirrorSourceReleasesToGiteaLocked({
|
||||
// Enforce release retention: first drop the releases the type filter excludes,
|
||||
// then the oldest releases past the limit.
|
||||
try {
|
||||
// Paginate to fetch ALL Gitea releases (API max is 100 per page)
|
||||
const allGiteaReleases: Array<{
|
||||
// A page that fails ends the listing early. Retention still runs on what
|
||||
// was listed, as before, but the storage totals are only recorded from a
|
||||
// complete listing.
|
||||
const { items: allGiteaReleases, complete: listingComplete } = await listDestinationReleases<{
|
||||
id: number;
|
||||
tag_name: string;
|
||||
created_at: string;
|
||||
prerelease?: boolean | null;
|
||||
assets?: Array<{ size?: number }> | null;
|
||||
}> = [];
|
||||
// A page that fails ends the listing early. Retention still runs on what
|
||||
// was listed, as before, but the storage totals are only recorded from a
|
||||
// complete listing.
|
||||
let listingComplete = true;
|
||||
let cleanupPage = 1;
|
||||
while (true) {
|
||||
const pageResponse = await httpGet(
|
||||
`${config.giteaConfig.url}/api/v1/repos/${repoOwner}/${repoName}/releases?per_page=100&page=${cleanupPage}`,
|
||||
{
|
||||
Authorization: `token ${decryptedConfig.giteaConfig.token}`,
|
||||
}
|
||||
).catch(() => null);
|
||||
|
||||
if (!pageResponse?.data || !Array.isArray(pageResponse.data)) {
|
||||
listingComplete = false;
|
||||
break;
|
||||
}
|
||||
if (pageResponse.data.length === 0) {
|
||||
break;
|
||||
}
|
||||
|
||||
allGiteaReleases.push(...pageResponse.data);
|
||||
|
||||
if (pageResponse.data.length < 100) {
|
||||
break;
|
||||
}
|
||||
cleanupPage++;
|
||||
}
|
||||
}>({
|
||||
baseUrl: config.giteaConfig.url,
|
||||
token: decryptedConfig.giteaConfig.token,
|
||||
owner: repoOwner,
|
||||
repo: repoName,
|
||||
});
|
||||
|
||||
const giteaBaseUrl = config.giteaConfig.url;
|
||||
const giteaToken = decryptedConfig.giteaConfig.token;
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
/**
|
||||
* Storage a mirror takes up on its destination (#479). Measured only from
|
||||
* responses a sync already fetches: the repository GET before `mirror-sync`
|
||||
* Storage a mirror takes up on its destination (#479). A sync measures it from
|
||||
* responses it already fetches: the repository GET before `mirror-sync`
|
||||
* carries `size`, and the release listing the retention pass reads carries
|
||||
* every asset's size. Nothing here makes a request.
|
||||
* every asset's size. Refresh sizes (storage-refresh.ts) reads the same fields
|
||||
* on request. Nothing here makes a request.
|
||||
*/
|
||||
|
||||
import { isPushDestinationKind } from "./destination-kinds";
|
||||
@@ -99,10 +100,21 @@ export interface StorageOverview {
|
||||
// failed first mirror, so it only counts once it has been measured.
|
||||
const MIRROR_EXISTS_STATUSES = new Set(["mirrored", "synced", "syncing", "archived", "pending-approval"]);
|
||||
|
||||
function isMeasured(row: StorageSourceRow): boolean {
|
||||
function isMeasured(row: Pick<StorageSourceRow, "destinationSize" | "releaseAssetsSize">): boolean {
|
||||
return row.destinationSize !== null || row.releaseAssetsSize !== null;
|
||||
}
|
||||
|
||||
/**
|
||||
* True for a row with a mirror on a Gitea or Forgejo destination: the rows
|
||||
* the overview counts as mirrored and Refresh sizes measures.
|
||||
*/
|
||||
export function countsAsStorageMirror(
|
||||
row: Pick<StorageSourceRow, "status" | "destinationProvider" | "destinationSize" | "releaseAssetsSize">
|
||||
): boolean {
|
||||
if (isPushDestinationKind(row.destinationProvider)) return false;
|
||||
return isMeasured(row) || MIRROR_EXISTS_STATUSES.has(row.status);
|
||||
}
|
||||
|
||||
/** The storage page's numbers: totals, per destination owner, and per repository, largest first. */
|
||||
export function buildStorageOverview(rows: ReadonlyArray<StorageSourceRow>): StorageOverview {
|
||||
const repositories: StorageRepository[] = [];
|
||||
@@ -110,10 +122,9 @@ export function buildStorageOverview(rows: ReadonlyArray<StorageSourceRow>): Sto
|
||||
let mirrored = 0;
|
||||
|
||||
for (const row of rows) {
|
||||
if (isPushDestinationKind(row.destinationProvider)) continue;
|
||||
const measured = isMeasured(row);
|
||||
if (measured || MIRROR_EXISTS_STATUSES.has(row.status)) mirrored += 1;
|
||||
if (!measured) continue;
|
||||
if (!countsAsStorageMirror(row)) continue;
|
||||
mirrored += 1;
|
||||
if (!isMeasured(row)) continue;
|
||||
|
||||
const location = (row.mirroredLocation ?? "").trim();
|
||||
const owner = location.includes("/") ? location.split("/")[0] : "";
|
||||
|
||||
@@ -0,0 +1,370 @@
|
||||
/**
|
||||
* Tests for Refresh sizes. The destination is a fake fetch that pages like
|
||||
* Gitea; the database, the owner lookup, the mirror options and the token
|
||||
* decryption are mocked.
|
||||
*
|
||||
* bun's mock.module is process-wide and leaks into other test files, so in the
|
||||
* shared test process this file registers nothing: it re-runs itself in an
|
||||
* isolated child process.
|
||||
*/
|
||||
import { describe, test, expect, mock, beforeEach, afterEach } from "bun:test";
|
||||
|
||||
const CHILD_FLAG = "GM_STORAGE_REFRESH_ISOLATED";
|
||||
const isChild = !!process.env[CHILD_FLAG];
|
||||
|
||||
if (!isChild) {
|
||||
test("storage refresh - isolated child suite", () => {
|
||||
const res = Bun.spawnSync({
|
||||
cmd: [process.execPath, "test", import.meta.path],
|
||||
env: { ...process.env, [CHILD_FLAG]: "1" },
|
||||
stdout: "pipe",
|
||||
stderr: "pipe",
|
||||
});
|
||||
if (res.exitCode !== 0) {
|
||||
console.error(res.stdout.toString());
|
||||
console.error(res.stderr.toString());
|
||||
}
|
||||
expect(res.exitCode).toBe(0);
|
||||
}, 60_000);
|
||||
}
|
||||
|
||||
const DEST = "https://gitea.example.com";
|
||||
|
||||
let configRows: any[] = [];
|
||||
let repoRows: any[] = [];
|
||||
let updates: Array<{ id: unknown; values: Record<string, unknown> }> = [];
|
||||
let releasesFor = new Set<string>();
|
||||
const mockConfigs = { table: "configs" };
|
||||
const mockRepositories = { table: "repositories" };
|
||||
|
||||
/** Leaf values of a drizzle condition (drizzle conditions are cyclic). */
|
||||
function conditionLeafValues(node: any, depth = 0): unknown[] {
|
||||
if (node == null || depth > 8) return [];
|
||||
if (typeof node !== "object") return [node];
|
||||
const leaves: unknown[] = [];
|
||||
if (Array.isArray(node.value)) leaves.push(...node.value);
|
||||
else if ("value" in node && typeof node.value !== "object") leaves.push(node.value);
|
||||
for (const chunk of node.queryChunks ?? []) {
|
||||
leaves.push(...conditionLeafValues(chunk, depth + 1));
|
||||
}
|
||||
return leaves;
|
||||
}
|
||||
|
||||
const mockGetOwner = mock(async ({ repository }: any) => repository.expectedOwner ?? "");
|
||||
|
||||
if (isChild) {
|
||||
mock.module("@/lib/db", () => ({
|
||||
db: {
|
||||
select: () => ({
|
||||
from: (table: unknown) => ({
|
||||
where: () =>
|
||||
table === mockConfigs
|
||||
? { orderBy: () => ({ limit: async () => configRows }) }
|
||||
: Promise.resolve(repoRows),
|
||||
}),
|
||||
}),
|
||||
update: () => ({
|
||||
set: (values: Record<string, unknown>) => ({
|
||||
where: async (cond: unknown) => {
|
||||
// The id is the one leaf that is not an SQL fragment ("", " = ").
|
||||
const id = conditionLeafValues(cond).find((v) => typeof v === "string" && !/^\s*=?\s*$/.test(v));
|
||||
updates.push({ id, values });
|
||||
},
|
||||
}),
|
||||
}),
|
||||
},
|
||||
configs: mockConfigs,
|
||||
repositories: mockRepositories,
|
||||
organizations: {},
|
||||
users: {},
|
||||
events: {},
|
||||
mirrorJobs: {},
|
||||
sessions: {},
|
||||
accounts: {},
|
||||
verificationTokens: {},
|
||||
verifications: {},
|
||||
oauthClients: {},
|
||||
oauthAccessTokens: {},
|
||||
oauthRefreshTokens: {},
|
||||
oauthConsents: {},
|
||||
jwkss: {},
|
||||
ssoProviders: {},
|
||||
apikeys: {},
|
||||
rateLimits: {},
|
||||
sources: {},
|
||||
}));
|
||||
|
||||
mock.module("@/lib/gitea", () => ({
|
||||
getGiteaRepoOwnerAsync: mockGetOwner,
|
||||
}));
|
||||
|
||||
mock.module("@/lib/utils/mirror-overrides", () => ({
|
||||
resolveMirrorOptionsForRepository: async ({ repository }: any) => ({
|
||||
mirrorReleases: releasesFor.has(repository.id),
|
||||
}),
|
||||
}));
|
||||
|
||||
mock.module("@/lib/utils/config-encryption", () => ({
|
||||
decryptConfigTokens: (config: any) => config,
|
||||
}));
|
||||
}
|
||||
|
||||
const { startStorageRefresh, getStorageRefreshProgress, waitForStorageRefresh } = isChild
|
||||
? await import("./storage-refresh")
|
||||
: ({} as typeof import("./storage-refresh"));
|
||||
|
||||
const realFetch = globalThis.fetch;
|
||||
let requestedUrls: string[] = [];
|
||||
|
||||
interface FakeDestination {
|
||||
/** Repositories per owner, as the owner listing returns them. */
|
||||
owners?: Record<string, Array<Record<string, unknown>>>;
|
||||
/** Single GET answers by "owner/name"; a number is an error status. */
|
||||
single?: Record<string, Record<string, unknown> | number>;
|
||||
/** Releases by "owner/name"; a number is an error status. */
|
||||
releases?: Record<string, Array<Record<string, unknown>> | number>;
|
||||
/** Held until released, to keep a run going. */
|
||||
gate?: Promise<void>;
|
||||
}
|
||||
|
||||
function json(data: unknown, total?: number) {
|
||||
const headers: Record<string, string> = { "content-type": "application/json" };
|
||||
if (total !== undefined) headers["x-total-count"] = String(total);
|
||||
return new Response(JSON.stringify(data), { status: 200, headers });
|
||||
}
|
||||
|
||||
function serve(world: FakeDestination) {
|
||||
requestedUrls = [];
|
||||
globalThis.fetch = (async (input: RequestInfo | URL) => {
|
||||
const url = String(input);
|
||||
requestedUrls.push(url);
|
||||
await world.gate;
|
||||
const { pathname, searchParams } = new URL(url);
|
||||
const page = Number(searchParams.get("page") ?? "1");
|
||||
const parts = pathname.split("/").map(decodeURIComponent);
|
||||
|
||||
// /api/v1/users/{owner}/repos
|
||||
if (parts[3] === "users" && parts[5] === "repos") {
|
||||
const all = world.owners?.[parts[4]];
|
||||
if (!all) return new Response("not found", { status: 404, statusText: "Not Found" });
|
||||
return json(all.slice((page - 1) * 50, page * 50), all.length);
|
||||
}
|
||||
// /api/v1/repos/{owner}/{name}/releases
|
||||
if (parts[3] === "repos" && parts[6] === "releases") {
|
||||
const answer = world.releases?.[`${parts[4]}/${parts[5]}`] ?? [];
|
||||
if (typeof answer === "number") return new Response("boom", { status: answer, statusText: "Error" });
|
||||
return json(answer.slice((page - 1) * 50, page * 50), answer.length);
|
||||
}
|
||||
// /api/v1/repos/{owner}/{name}
|
||||
if (parts[3] === "repos" && parts.length === 6) {
|
||||
const answer = world.single?.[`${parts[4]}/${parts[5]}`] ?? 404;
|
||||
if (typeof answer === "number") return new Response("nope", { status: answer, statusText: "Error" });
|
||||
return json(answer);
|
||||
}
|
||||
throw new Error(`unexpected request: ${url}`);
|
||||
}) as typeof fetch;
|
||||
}
|
||||
|
||||
function repo(id: string, overrides: Record<string, unknown> = {}) {
|
||||
return {
|
||||
id,
|
||||
userId: "user-1",
|
||||
name: id,
|
||||
fullName: `octo/${id}`,
|
||||
mirroredLocation: `archive/${id}`,
|
||||
status: "mirrored",
|
||||
destinationProvider: "gitea",
|
||||
destinationSize: null,
|
||||
releaseAssetsSize: null,
|
||||
releaseAssetsCount: null,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function destRepo(name: string, sizeKiB: number, extra: Record<string, unknown> = {}) {
|
||||
return { name, owner: { login: "archive" }, mirror: true, size: sizeKiB, ...extra };
|
||||
}
|
||||
|
||||
async function refresh() {
|
||||
const started = await startStorageRefresh("user-1");
|
||||
await waitForStorageRefresh("user-1");
|
||||
return { started, progress: getStorageRefreshProgress("user-1") };
|
||||
}
|
||||
|
||||
function writesFor(id: string) {
|
||||
return updates.filter((u) => u.id === id).map((u) => u.values);
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
configRows = [{ userId: "user-1", isActive: true, giteaConfig: { url: `${DEST}/`, token: "dest-token", provider: "gitea" } }];
|
||||
repoRows = [];
|
||||
updates = [];
|
||||
releasesFor = new Set();
|
||||
mockGetOwner.mockClear();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
globalThis.fetch = realFetch;
|
||||
});
|
||||
|
||||
describe.skipIf(!isChild)("Refresh sizes", () => {
|
||||
test("reads sizes from one listing per owner and releases only where they are mirrored", async () => {
|
||||
repoRows = [repo("app"), repo("lib")];
|
||||
releasesFor = new Set(["app"]);
|
||||
serve({
|
||||
owners: { archive: [destRepo("app", 2959), destRepo("lib", 10), destRepo("not-ours", 99)] },
|
||||
releases: { "archive/app": [{ assets: [{ size: 1000 }, { size: 24 }] }, { assets: [] }] },
|
||||
});
|
||||
|
||||
const { started, progress } = await refresh();
|
||||
|
||||
expect(started).toMatchObject({ ok: true, progress: { running: true, total: 2 } });
|
||||
expect(progress).toMatchObject({ running: false, total: 2, done: 2, measured: 2, missing: 0, failed: 0 });
|
||||
expect(writesFor("app")).toEqual([
|
||||
expect.objectContaining({ destinationSize: 2959 * 1024, releaseAssetsSize: 1024, releaseAssetsCount: 2 }),
|
||||
]);
|
||||
expect(writesFor("lib")).toEqual([expect.objectContaining({ destinationSize: 10 * 1024 })]);
|
||||
expect(writesFor("lib")[0]).not.toHaveProperty("releaseAssetsSize");
|
||||
expect(writesFor("lib")[0].storageCheckedAt).toBeInstanceOf(Date);
|
||||
|
||||
const ownerListings = requestedUrls.filter((u) => u.includes("/users/"));
|
||||
expect(ownerListings).toEqual([`${DEST}/api/v1/users/archive/repos?limit=50&page=1`]);
|
||||
expect(requestedUrls.filter((u) => u.includes("/releases"))).toHaveLength(1);
|
||||
// Only the destination is contacted.
|
||||
expect(requestedUrls.every((u) => u.startsWith(`${DEST}/api/v1/`))).toBe(true);
|
||||
});
|
||||
|
||||
test("pages through an owner with more repositories than one page", async () => {
|
||||
const many = Array.from({ length: 120 }, (_, i) => destRepo(`r${i}`, i + 1));
|
||||
repoRows = [repo("r119")];
|
||||
serve({ owners: { archive: many } });
|
||||
|
||||
const { progress } = await refresh();
|
||||
|
||||
expect(progress?.measured).toBe(1);
|
||||
expect(writesFor("r119")[0].destinationSize).toBe(120 * 1024);
|
||||
expect(requestedUrls.filter((u) => u.includes("/users/"))).toHaveLength(3);
|
||||
});
|
||||
|
||||
test("falls back to the single GET for a repository the listing does not have", async () => {
|
||||
repoRows = [repo("renamed"), repo("gone")];
|
||||
serve({
|
||||
owners: { archive: [] },
|
||||
single: { "archive/renamed": destRepo("new-name", 5) },
|
||||
});
|
||||
|
||||
const { progress } = await refresh();
|
||||
|
||||
expect(progress).toMatchObject({ measured: 1, missing: 1, failed: 0 });
|
||||
expect(writesFor("renamed")[0].destinationSize).toBe(5 * 1024);
|
||||
expect(writesFor("gone")).toEqual([]);
|
||||
});
|
||||
|
||||
test("reads releases at the name the destination reports", async () => {
|
||||
repoRows = [repo("renamed")];
|
||||
releasesFor = new Set(["renamed"]);
|
||||
serve({
|
||||
owners: { archive: [] },
|
||||
single: { "archive/renamed": destRepo("new-name", 5) },
|
||||
releases: { "archive/new-name": [{ assets: [{ size: 7 }] }] },
|
||||
});
|
||||
|
||||
await refresh();
|
||||
|
||||
expect(writesFor("renamed")[0]).toMatchObject({ releaseAssetsSize: 7, releaseAssetsCount: 1 });
|
||||
});
|
||||
|
||||
test("a failed owner listing leaves its repositories to the single GET", async () => {
|
||||
repoRows = [repo("app")];
|
||||
serve({ single: { "archive/app": destRepo("app", 3) } });
|
||||
|
||||
const { progress } = await refresh();
|
||||
|
||||
expect(progress?.measured).toBe(1);
|
||||
expect(requestedUrls).toContain(`${DEST}/api/v1/repos/archive/app`);
|
||||
});
|
||||
|
||||
test("without a recorded location, measures only a mirror where a sync would look", async () => {
|
||||
repoRows = [
|
||||
repo("placed", { mirroredLocation: null, expectedOwner: "me" }),
|
||||
repo("someone-elses", { mirroredLocation: null, expectedOwner: "me" }),
|
||||
];
|
||||
serve({
|
||||
owners: { me: [destRepo("placed", 4), destRepo("someone-elses", 8, { mirror: false })] },
|
||||
});
|
||||
|
||||
const { progress } = await refresh();
|
||||
|
||||
expect(mockGetOwner).toHaveBeenCalledTimes(2);
|
||||
expect(progress).toMatchObject({ measured: 1, missing: 1 });
|
||||
expect(writesFor("placed")[0].destinationSize).toBe(4 * 1024);
|
||||
expect(writesFor("someone-elses")).toEqual([]);
|
||||
});
|
||||
|
||||
test("an incomplete release listing keeps the stored release numbers", async () => {
|
||||
repoRows = [repo("app")];
|
||||
releasesFor = new Set(["app"]);
|
||||
serve({ owners: { archive: [destRepo("app", 2)] }, releases: { "archive/app": 500 } });
|
||||
|
||||
const { progress } = await refresh();
|
||||
|
||||
expect(progress?.measured).toBe(1);
|
||||
expect(writesFor("app")[0]).toHaveProperty("destinationSize", 2048);
|
||||
expect(writesFor("app")[0]).not.toHaveProperty("releaseAssetsSize");
|
||||
});
|
||||
|
||||
test("counts a request that fails and writes nothing for it", async () => {
|
||||
repoRows = [repo("app")];
|
||||
serve({ owners: { archive: [] }, single: { "archive/app": 500 } });
|
||||
|
||||
const { progress } = await refresh();
|
||||
|
||||
expect(progress).toMatchObject({ running: false, done: 1, measured: 0, failed: 1 });
|
||||
expect(updates).toEqual([]);
|
||||
});
|
||||
|
||||
test("measures the rows the storage page counts as mirrors, and no others", async () => {
|
||||
repoRows = [
|
||||
repo("app"),
|
||||
repo("pushed", { destinationProvider: "gitlab" }),
|
||||
repo("imported", { status: "imported", mirroredLocation: null }),
|
||||
repo("failed-but-measured", { status: "failed", destinationSize: 10 }),
|
||||
];
|
||||
serve({ owners: { archive: [destRepo("app", 1), destRepo("failed-but-measured", 2)] } });
|
||||
|
||||
const { progress } = await refresh();
|
||||
|
||||
expect(progress).toMatchObject({ total: 2, measured: 2 });
|
||||
});
|
||||
|
||||
test("refuses without a destination token or with a push destination", async () => {
|
||||
serve({});
|
||||
configRows = [{ userId: "user-1", giteaConfig: { url: DEST, token: "", provider: "gitea" } }];
|
||||
expect(await startStorageRefresh("user-1")).toMatchObject({ ok: false, status: 400 });
|
||||
|
||||
configRows = [{ userId: "user-1", giteaConfig: { url: "https://gitlab.com", token: "t", provider: "gitlab" } }];
|
||||
expect(await startStorageRefresh("user-1")).toMatchObject({
|
||||
ok: false,
|
||||
status: 400,
|
||||
error: expect.stringContaining("Gitea and Forgejo"),
|
||||
});
|
||||
expect(requestedUrls).toEqual([]);
|
||||
});
|
||||
|
||||
test("a second start while a run is going reports that run instead of starting another", async () => {
|
||||
repoRows = [repo("app")];
|
||||
let open!: () => void;
|
||||
const gate = new Promise<void>((resolve) => (open = resolve));
|
||||
serve({ owners: { archive: [destRepo("app", 1)] }, gate });
|
||||
|
||||
const first = await startStorageRefresh("user-1");
|
||||
const second = await startStorageRefresh("user-1");
|
||||
expect(first).toMatchObject({ ok: true, progress: { running: true } });
|
||||
expect(second).toMatchObject({ ok: true, progress: { running: true, startedAt: (first as any).progress.startedAt } });
|
||||
|
||||
open();
|
||||
await waitForStorageRefresh("user-1");
|
||||
expect(requestedUrls.filter((u) => u.includes("/users/"))).toHaveLength(1);
|
||||
expect(getStorageRefreshProgress("user-1")).toMatchObject({ running: false, measured: 1 });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,284 @@
|
||||
/**
|
||||
* Refresh sizes: measure every mirror on the destination now instead of
|
||||
* waiting for each one's next sync (#479 follow-up). It only reads the
|
||||
* destination and never contacts the source:
|
||||
*
|
||||
* - one listing per destination owner carries every repository's `size`;
|
||||
* - a repository missing from its owner's listing gets the single GET, which
|
||||
* follows a rename or transfer the way a sync does;
|
||||
* - repositories that mirror releases get their release listing read.
|
||||
*
|
||||
* Runs in the background, one run per user at a time. Progress lives in memory
|
||||
* and the storage overview reports it.
|
||||
*/
|
||||
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { db, configs, repositories } from "./db";
|
||||
import type { Repository } from "./db/schema";
|
||||
import type { Config } from "@/types/config";
|
||||
import { HttpError, httpGet } from "./http-client";
|
||||
import { decryptConfigTokens } from "./utils/config-encryption";
|
||||
import { isPushDestinationKind, normalizeDestinationProviderKind } from "./destination-kinds";
|
||||
import { getGiteaRepoOwnerAsync } from "./gitea";
|
||||
import { resolveMirrorOptionsForRepository } from "./utils/mirror-overrides";
|
||||
import { listDestinationOwnerRepositories, listDestinationReleases } from "./destination-listing";
|
||||
import { countsAsStorageMirror, destinationSizeColumns, summarizeReleaseAssets } from "./repository-storage";
|
||||
import { processInParallel } from "./utils/concurrency";
|
||||
|
||||
// Requests in flight against the destination at once.
|
||||
const CONCURRENCY = 4;
|
||||
|
||||
export interface StorageRefreshProgress {
|
||||
running: boolean;
|
||||
/** Repositories this run measures. */
|
||||
total: number;
|
||||
/** Repositories finished, whatever the outcome. */
|
||||
done: number;
|
||||
measured: number;
|
||||
/** Not found on the destination where the mirror should be. */
|
||||
missing: number;
|
||||
/** A request failed; the stored numbers were kept. */
|
||||
failed: number;
|
||||
startedAt: string;
|
||||
finishedAt: string | null;
|
||||
}
|
||||
|
||||
/** The destination's answer for one repository, the fields used here. */
|
||||
interface DestinationRepository {
|
||||
name?: string;
|
||||
owner?: { login?: string } | string;
|
||||
mirror?: boolean;
|
||||
size?: number;
|
||||
}
|
||||
|
||||
interface Target {
|
||||
repository: Repository;
|
||||
owner: string;
|
||||
name: string;
|
||||
/** False when the location was not recorded and is only where a sync would look. */
|
||||
recorded: boolean;
|
||||
}
|
||||
|
||||
const runs = new Map<string, StorageRefreshProgress>();
|
||||
const inFlight = new Map<string, Promise<void>>();
|
||||
|
||||
export function getStorageRefreshProgress(userId: string): StorageRefreshProgress | null {
|
||||
const run = runs.get(userId);
|
||||
return run ? { ...run } : null;
|
||||
}
|
||||
|
||||
/** Resolves when the user's current run ends. For tests. */
|
||||
export function waitForStorageRefresh(userId: string): Promise<void> {
|
||||
return inFlight.get(userId) ?? Promise.resolve();
|
||||
}
|
||||
|
||||
export type StartStorageRefreshResult =
|
||||
| { ok: true; progress: StorageRefreshProgress }
|
||||
| { ok: false; status: number; error: string };
|
||||
|
||||
/** Start a run for the user, or report the one already running. */
|
||||
export async function startStorageRefresh(userId: string): Promise<StartStorageRefreshResult> {
|
||||
const previous = runs.get(userId);
|
||||
if (previous?.running) return { ok: true, progress: { ...previous } };
|
||||
|
||||
// Claimed before the first await so a second click cannot start a twin.
|
||||
const progress: StorageRefreshProgress = {
|
||||
running: true,
|
||||
total: 0,
|
||||
done: 0,
|
||||
measured: 0,
|
||||
missing: 0,
|
||||
failed: 0,
|
||||
startedAt: new Date().toISOString(),
|
||||
finishedAt: null,
|
||||
};
|
||||
runs.set(userId, progress);
|
||||
|
||||
const release = () => {
|
||||
if (previous) runs.set(userId, previous);
|
||||
else runs.delete(userId);
|
||||
};
|
||||
const refuse = (status: number, error: string): StartStorageRefreshResult => {
|
||||
release();
|
||||
return { ok: false, status, error };
|
||||
};
|
||||
|
||||
let config: Partial<Config>;
|
||||
let targets: Repository[];
|
||||
try {
|
||||
const [configRow] = await db
|
||||
.select()
|
||||
.from(configs)
|
||||
.where(eq(configs.userId, userId))
|
||||
.orderBy(sql`${configs.isActive} DESC`, sql`${configs.updatedAt} DESC`)
|
||||
.limit(1);
|
||||
if (!configRow?.giteaConfig?.url || !configRow.giteaConfig.token) {
|
||||
return refuse(400, "Configure the destination URL and token first");
|
||||
}
|
||||
if (isPushDestinationKind(normalizeDestinationProviderKind(configRow.giteaConfig.provider))) {
|
||||
return refuse(400, "Sizes are read from Gitea and Forgejo destinations only");
|
||||
}
|
||||
config = configRow as unknown as Partial<Config>;
|
||||
|
||||
const rows = await db.select().from(repositories).where(eq(repositories.userId, userId));
|
||||
targets = (rows as Repository[]).filter((row) =>
|
||||
countsAsStorageMirror({
|
||||
status: row.status,
|
||||
destinationProvider: row.destinationProvider ?? "gitea",
|
||||
destinationSize: row.destinationSize ?? null,
|
||||
releaseAssetsSize: row.releaseAssetsSize ?? null,
|
||||
})
|
||||
);
|
||||
} catch (error) {
|
||||
release();
|
||||
throw error;
|
||||
}
|
||||
|
||||
progress.total = targets.length;
|
||||
const run = measureAll(config, targets, progress)
|
||||
.catch((error) => {
|
||||
console.error(
|
||||
`[Storage] Refresh sizes stopped for user ${userId}: ${error instanceof Error ? error.message : String(error)}`
|
||||
);
|
||||
})
|
||||
.finally(() => {
|
||||
progress.failed += progress.total - progress.done;
|
||||
progress.done = progress.total;
|
||||
progress.running = false;
|
||||
progress.finishedAt = new Date().toISOString();
|
||||
inFlight.delete(userId);
|
||||
console.log(
|
||||
`[Storage] Refreshed sizes for user ${userId}: ${progress.measured} measured, ${progress.missing} not found, ${progress.failed} failed`
|
||||
);
|
||||
});
|
||||
inFlight.set(userId, run);
|
||||
|
||||
return { ok: true, progress: { ...progress } };
|
||||
}
|
||||
|
||||
function parseLocation(location: string | null | undefined): { owner: string; name: string } | null {
|
||||
const trimmed = (location ?? "").trim();
|
||||
const slash = trimmed.indexOf("/");
|
||||
if (slash <= 0 || slash === trimmed.length - 1) return null;
|
||||
return { owner: trimmed.slice(0, slash), name: trimmed.slice(slash + 1) };
|
||||
}
|
||||
|
||||
async function measureAll(
|
||||
config: Partial<Config>,
|
||||
rows: Repository[],
|
||||
progress: StorageRefreshProgress
|
||||
): Promise<void> {
|
||||
const baseUrl = config.giteaConfig!.url.replace(/\/+$/, "");
|
||||
const token = decryptConfigTokens(config as Config).giteaConfig.token;
|
||||
|
||||
// Where each mirror is, or where a sync would look for it.
|
||||
const targets: Target[] = [];
|
||||
for (const repository of rows) {
|
||||
const recorded = parseLocation(repository.mirroredLocation);
|
||||
if (recorded) {
|
||||
targets.push({ repository, ...recorded, recorded: true });
|
||||
continue;
|
||||
}
|
||||
const owner = await getGiteaRepoOwnerAsync({ config, repository }).catch(() => "");
|
||||
if (owner) {
|
||||
targets.push({ repository, owner, name: repository.name, recorded: false });
|
||||
} else {
|
||||
progress.failed += 1;
|
||||
progress.done += 1;
|
||||
}
|
||||
}
|
||||
|
||||
// One listing per owner. A failed listing leaves its repositories to the
|
||||
// single GET below.
|
||||
const owners = [...new Map(targets.map((t) => [t.owner.toLowerCase(), t.owner])).values()];
|
||||
const listings = new Map<string, Map<string, DestinationRepository>>();
|
||||
await processInParallel(
|
||||
owners,
|
||||
async (owner) => {
|
||||
const listing = await listDestinationOwnerRepositories<DestinationRepository>({ baseUrl, token, owner });
|
||||
if (!listing.complete) return;
|
||||
const byName = new Map<string, DestinationRepository>();
|
||||
for (const repo of listing.items) {
|
||||
if (typeof repo?.name === "string") byName.set(repo.name.toLowerCase(), repo);
|
||||
}
|
||||
listings.set(owner.toLowerCase(), byName);
|
||||
},
|
||||
CONCURRENCY
|
||||
);
|
||||
|
||||
await processInParallel(
|
||||
targets,
|
||||
async (target) => {
|
||||
try {
|
||||
const outcome = await measureOne({ config, baseUrl, token, target, listings });
|
||||
progress[outcome] += 1;
|
||||
} catch (error) {
|
||||
progress.failed += 1;
|
||||
console.warn(
|
||||
`[Storage] Could not measure ${target.owner}/${target.name}: ${
|
||||
error instanceof Error ? error.message : String(error)
|
||||
}`
|
||||
);
|
||||
} finally {
|
||||
progress.done += 1;
|
||||
}
|
||||
},
|
||||
CONCURRENCY
|
||||
);
|
||||
}
|
||||
|
||||
async function measureOne({
|
||||
config,
|
||||
baseUrl,
|
||||
token,
|
||||
target,
|
||||
listings,
|
||||
}: {
|
||||
config: Partial<Config>;
|
||||
baseUrl: string;
|
||||
token: string;
|
||||
target: Target;
|
||||
listings: Map<string, Map<string, DestinationRepository>>;
|
||||
}): Promise<"measured" | "missing" | "failed"> {
|
||||
let info = listings.get(target.owner.toLowerCase())?.get(target.name.toLowerCase()) ?? null;
|
||||
if (!info) {
|
||||
try {
|
||||
const response = await httpGet<DestinationRepository>(
|
||||
`${baseUrl}/api/v1/repos/${encodeURIComponent(target.owner)}/${encodeURIComponent(target.name)}`,
|
||||
{ Authorization: `token ${token}` }
|
||||
);
|
||||
info = response.data;
|
||||
} catch (error) {
|
||||
if (error instanceof HttpError && error.status === 404) return "missing";
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
// Without a recorded location, only a mirror at the expected place is ours.
|
||||
if (!info || (!target.recorded && info.mirror !== true)) return "missing";
|
||||
|
||||
const now = new Date();
|
||||
const columns: Partial<
|
||||
Pick<Repository, "destinationSize" | "releaseAssetsSize" | "releaseAssetsCount" | "storageCheckedAt">
|
||||
> = { ...destinationSizeColumns(info, now) };
|
||||
|
||||
const { mirrorReleases } = await resolveMirrorOptionsForRepository({ config, repository: target.repository });
|
||||
if (mirrorReleases) {
|
||||
const owner = (typeof info.owner === "string" ? info.owner : info.owner?.login) || target.owner;
|
||||
const releases = await listDestinationReleases<{ assets?: Array<{ size?: unknown }> | null }>({
|
||||
baseUrl,
|
||||
token,
|
||||
owner,
|
||||
repo: info.name || target.name,
|
||||
});
|
||||
if (releases.complete) {
|
||||
const summary = summarizeReleaseAssets(releases.items);
|
||||
columns.releaseAssetsSize = summary.bytes;
|
||||
columns.releaseAssetsCount = summary.count;
|
||||
columns.storageCheckedAt = now;
|
||||
}
|
||||
}
|
||||
|
||||
if (Object.keys(columns).length === 0) return "failed";
|
||||
await db.update(repositories).set(columns).where(eq(repositories.id, target.repository.id!));
|
||||
return "measured";
|
||||
}
|
||||
@@ -172,7 +172,7 @@ describe.skipIf(!isChild)("GET /api/repositories/:id/releases", () => {
|
||||
{ tag: "v2.0-rc1", prerelease: true },
|
||||
]);
|
||||
expect(requestedUrls[0]).toContain("/api/v1/repos/mirrors/hello/releases");
|
||||
expect(requestedUrls[0]).toContain("per_page=100");
|
||||
expect(requestedUrls[0]).toContain("limit=50&page=1");
|
||||
expect(mockGetOwner).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
@@ -233,6 +233,21 @@ describe.skipIf(!isChild)("GET /api/repositories/:id/releases", () => {
|
||||
expect(requestedUrls[1]).toContain("page=2");
|
||||
});
|
||||
|
||||
test("keeps paging while the destination's total says more remain", async () => {
|
||||
// Gitea caps pages below what was asked for and reports the total.
|
||||
const page = (count: number, offset: number) =>
|
||||
new Response(JSON.stringify(batch(count, offset)), {
|
||||
headers: { "content-type": "application/json", "x-total-count": "35" },
|
||||
});
|
||||
servePages([page(30, 0), page(5, 30)]);
|
||||
|
||||
const response = await call();
|
||||
const data = await response.json();
|
||||
expect(data.releases).toHaveLength(35);
|
||||
expect(data.truncated).toBe(false);
|
||||
expect(requestedUrls).toHaveLength(2);
|
||||
});
|
||||
|
||||
test("stops at the cap and marks the list as shortened", async () => {
|
||||
servePages([batch(100), batch(100, 100), batch(100, 200), batch(1, 300)]);
|
||||
|
||||
|
||||
@@ -7,10 +7,9 @@ import { requireAuthenticatedUserId } from "@/lib/auth-guards";
|
||||
import { jsonResponse, createSecureErrorResponse } from "@/lib/utils";
|
||||
import { getGiteaRepoOwnerAsync } from "@/lib/gitea";
|
||||
import { decryptConfigTokens } from "@/lib/utils/config-encryption";
|
||||
import { httpGet } from "@/lib/http-client";
|
||||
import { listDestinationReleases } from "@/lib/destination-listing";
|
||||
|
||||
/** Gitea's per-page maximum; the cap below is how far the preview looks. */
|
||||
const PAGE_SIZE = 100;
|
||||
/** How far the preview looks. */
|
||||
const MAX_RELEASES = 300;
|
||||
|
||||
type DestinationRelease = {
|
||||
@@ -76,45 +75,35 @@ export const GET: APIRoute = async ({ params, request, locals }) => {
|
||||
);
|
||||
}
|
||||
|
||||
const token = decryptConfigTokens(config as Config).giteaConfig.token;
|
||||
const base = `${config.giteaConfig.url.replace(/\/+$/, "")}/api/v1/repos/${encodeURIComponent(
|
||||
owner
|
||||
)}/${encodeURIComponent(repoName)}/releases`;
|
||||
const listing = await listDestinationReleases<DestinationReleaseGitea>({
|
||||
baseUrl: config.giteaConfig.url,
|
||||
token: decryptConfigTokens(config as Config).giteaConfig.token,
|
||||
owner,
|
||||
repo: repoName,
|
||||
max: MAX_RELEASES,
|
||||
});
|
||||
// Nothing listed and a failed page means the first page failed. A later
|
||||
// failure keeps what was listed, as before.
|
||||
if (!listing.complete && listing.items.length === 0) {
|
||||
return jsonResponse({
|
||||
data: {
|
||||
success: false,
|
||||
error: `No releases could be read for ${owner}/${repoName} in the destination`,
|
||||
},
|
||||
status: 404,
|
||||
});
|
||||
}
|
||||
|
||||
const releases: DestinationRelease[] = [];
|
||||
let truncated = false;
|
||||
for (let page = 1; ; page++) {
|
||||
const response = await httpGet<DestinationReleaseGitea[]>(
|
||||
`${base}?per_page=${PAGE_SIZE}&page=${page}`,
|
||||
{ Authorization: `token ${token}` }
|
||||
).catch(() => null);
|
||||
const batch = response?.data;
|
||||
if (!Array.isArray(batch)) {
|
||||
if (page === 1) {
|
||||
return jsonResponse({
|
||||
data: {
|
||||
success: false,
|
||||
error: `No releases could be read for ${owner}/${repoName} in the destination`,
|
||||
},
|
||||
status: 404,
|
||||
});
|
||||
}
|
||||
break;
|
||||
}
|
||||
for (const release of batch) {
|
||||
const tag = typeof release.tag_name === "string" ? release.tag_name : "";
|
||||
if (!tag) continue;
|
||||
releases.push({
|
||||
tag,
|
||||
prerelease: typeof release.prerelease === "boolean" ? release.prerelease : null,
|
||||
});
|
||||
}
|
||||
if (batch.length < PAGE_SIZE) break;
|
||||
if (releases.length >= MAX_RELEASES) {
|
||||
truncated = true;
|
||||
break;
|
||||
}
|
||||
for (const release of listing.items) {
|
||||
const tag = typeof release.tag_name === "string" ? release.tag_name : "";
|
||||
if (!tag) continue;
|
||||
releases.push({
|
||||
tag,
|
||||
prerelease: typeof release.prerelease === "boolean" ? release.prerelease : null,
|
||||
});
|
||||
}
|
||||
const truncated = listing.truncated;
|
||||
|
||||
return jsonResponse({
|
||||
data: {
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
/**
|
||||
* Route tests for GET /api/storage (#479).
|
||||
* Route tests for GET /api/storage (#479) and POST /api/storage/refresh.
|
||||
*
|
||||
* NOTE: these tests replace @/lib/db and @/lib/utils/auth-helpers with mocks.
|
||||
* NOTE: these tests replace @/lib/db, @/lib/utils/auth-helpers and
|
||||
* @/lib/storage-refresh with mocks.
|
||||
* bun's mock.module is process-wide and leaks into other test files, so in
|
||||
* the shared test process this file registers NOTHING: it re-runs itself in
|
||||
* an isolated child process (the same harness as repositories/[id].test.ts).
|
||||
@@ -32,6 +33,9 @@ let authResult: { user: { id: string } | null; response: Response | null } = {
|
||||
user: { id: "user-1" },
|
||||
response: null,
|
||||
};
|
||||
let refreshProgress: unknown = null;
|
||||
let startResult: unknown = null;
|
||||
let startedFor: string[] = [];
|
||||
/** Leaf values of each select's where() condition (drizzle conditions are cyclic). */
|
||||
let selectWhereValues: unknown[][] = [];
|
||||
|
||||
@@ -51,6 +55,14 @@ if (isChild) {
|
||||
requireAuth: mock(async () => authResult),
|
||||
}));
|
||||
|
||||
mock.module("@/lib/storage-refresh", () => ({
|
||||
getStorageRefreshProgress: () => refreshProgress,
|
||||
startStorageRefresh: async (userId: string) => {
|
||||
startedFor.push(userId);
|
||||
return startResult;
|
||||
},
|
||||
}));
|
||||
|
||||
mock.module("@/lib/db", () => ({
|
||||
db: {
|
||||
select: () => ({
|
||||
@@ -85,6 +97,7 @@ if (isChild) {
|
||||
}
|
||||
|
||||
const { GET } = isChild ? await import("./index") : { GET: undefined as any };
|
||||
const { POST } = isChild ? await import("./refresh") : { POST: undefined as any };
|
||||
|
||||
function get() {
|
||||
return GET({
|
||||
@@ -94,7 +107,29 @@ function get() {
|
||||
} as any);
|
||||
}
|
||||
|
||||
function post() {
|
||||
return POST({
|
||||
params: {},
|
||||
request: new Request("http://localhost/api/storage/refresh", { method: "POST" }),
|
||||
locals: { session: { userId: "user-1" } },
|
||||
} as any);
|
||||
}
|
||||
|
||||
const RUNNING = {
|
||||
running: true,
|
||||
total: 4,
|
||||
done: 1,
|
||||
measured: 1,
|
||||
missing: 0,
|
||||
failed: 0,
|
||||
startedAt: "2026-10-29T08:00:00.000Z",
|
||||
finishedAt: null,
|
||||
};
|
||||
|
||||
beforeEach(() => {
|
||||
refreshProgress = null;
|
||||
startResult = { ok: true, progress: RUNNING };
|
||||
startedFor = [];
|
||||
selectWhereValues = [];
|
||||
authResult = { user: { id: "user-1" }, response: null };
|
||||
repoRows = [
|
||||
@@ -144,3 +179,46 @@ describe.skipIf(!isChild)("GET /api/storage", () => {
|
||||
expect(selectWhereValues).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe.skipIf(!isChild)("Refresh sizes progress", () => {
|
||||
test("GET reports no run as null and a run's progress as is", async () => {
|
||||
expect((await (await get()).json()).refresh).toBeNull();
|
||||
|
||||
refreshProgress = RUNNING;
|
||||
expect((await (await get()).json()).refresh).toEqual(RUNNING);
|
||||
});
|
||||
});
|
||||
|
||||
describe.skipIf(!isChild)("POST /api/storage/refresh", () => {
|
||||
test("starts a run for the signed in user and answers 202 with its progress", async () => {
|
||||
const response = await post();
|
||||
|
||||
expect(response.status).toBe(202);
|
||||
expect(await response.json()).toEqual({ success: true, refresh: RUNNING });
|
||||
expect(startedFor).toEqual(["user-1"]);
|
||||
});
|
||||
|
||||
test("passes a refusal through with its status", async () => {
|
||||
startResult = { ok: false, status: 400, error: "Configure the destination URL and token first" };
|
||||
|
||||
const response = await post();
|
||||
|
||||
expect(response.status).toBe(400);
|
||||
expect(await response.json()).toEqual({
|
||||
success: false,
|
||||
error: "Configure the destination URL and token first",
|
||||
});
|
||||
});
|
||||
|
||||
test("answers the auth helper's response when signed out", async () => {
|
||||
authResult = {
|
||||
user: null,
|
||||
response: new Response(JSON.stringify({ error: "Unauthorized" }), { status: 401 }),
|
||||
};
|
||||
|
||||
const response = await post();
|
||||
|
||||
expect(response.status).toBe(401);
|
||||
expect(startedFor).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -4,11 +4,13 @@ import { db, repositories } from "@/lib/db";
|
||||
import { createSecureErrorResponse } from "@/lib/utils";
|
||||
import { requireAuth } from "@/lib/utils/auth-helpers";
|
||||
import { buildStorageOverview } from "@/lib/repository-storage";
|
||||
import { getStorageRefreshProgress } from "@/lib/storage-refresh";
|
||||
|
||||
/**
|
||||
* Storage the user's mirrors take up on the destination (#479): totals, per
|
||||
* destination owner and per repository. Reads only what syncs recorded, so it
|
||||
* never contacts the destination or the source.
|
||||
* destination owner and per repository. Reads only what syncs and Refresh
|
||||
* sizes recorded, so it never contacts the destination or the source. While a
|
||||
* Refresh sizes run is going, `refresh` carries its progress.
|
||||
*/
|
||||
export const GET: APIRoute = async (context) => {
|
||||
try {
|
||||
@@ -31,7 +33,11 @@ export const GET: APIRoute = async (context) => {
|
||||
.from(repositories)
|
||||
.where(eq(repositories.userId, user!.id));
|
||||
|
||||
return new Response(JSON.stringify({ success: true, ...buildStorageOverview(rows) }), {
|
||||
return new Response(JSON.stringify({
|
||||
success: true,
|
||||
...buildStorageOverview(rows),
|
||||
refresh: getStorageRefreshProgress(user!.id),
|
||||
}), {
|
||||
status: 200,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
});
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
import type { APIRoute } from "astro";
|
||||
import { createSecureErrorResponse } from "@/lib/utils";
|
||||
import { requireAuth } from "@/lib/utils/auth-helpers";
|
||||
import { startStorageRefresh } from "@/lib/storage-refresh";
|
||||
|
||||
/**
|
||||
* Refresh sizes: measure every mirror on the destination now. Starts a run in
|
||||
* the background, or reports the one already running, and answers with its
|
||||
* progress; GET /api/storage keeps reporting it until it ends. Only the
|
||||
* destination is read.
|
||||
*/
|
||||
export const POST: APIRoute = async (context) => {
|
||||
try {
|
||||
const { user, response } = await requireAuth(context);
|
||||
if (response) return response;
|
||||
|
||||
const result = await startStorageRefresh(user!.id);
|
||||
if (!result.ok) {
|
||||
return new Response(JSON.stringify({ success: false, error: result.error }), {
|
||||
status: result.status,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
});
|
||||
}
|
||||
|
||||
return new Response(JSON.stringify({ success: true, refresh: result.progress }), {
|
||||
status: 202,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
});
|
||||
} catch (error) {
|
||||
return createSecureErrorResponse(error, "refresh storage sizes", 500);
|
||||
}
|
||||
};
|
||||
@@ -169,11 +169,23 @@ When several sources are connected, an optional `sourceId` picks which one the o
|
||||
"success": true,
|
||||
"totals": { "total": 3554304, "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "measured": 1, "mirrored": 1 },
|
||||
"owners": [{ "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "total": 3554304, "repositories": 1 }],
|
||||
"repositories": [{ "id": "…", "fullName": "upstream/app", "location": "archive/app", "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "total": 3554304, "checkedAt": "2026-10-29T03:32:01.000Z" }]
|
||||
"repositories": [{ "id": "…", "fullName": "upstream/app", "location": "archive/app", "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "total": 3554304, "checkedAt": "2026-10-29T03:32:01.000Z" }],
|
||||
"refresh": null
|
||||
}
|
||||
```
|
||||
|
||||
Sizes are bytes. `destinationSize` is Git and LFS together as Gitea or Forgejo reports it, and the release fields cover the assets on the destination's releases. Each sync records them from responses it already fetches, so this endpoint never contacts the destination. A repository is listed once a sync has measured it, and a field not measured yet is `null`. `mirrored` counts the repositories with a mirror on a Gitea or Forgejo destination and `measured` the ones with a size so far; push destinations (GitHub, GitLab) are not measured.
|
||||
Sizes are bytes. `destinationSize` is Git and LFS together as Gitea or Forgejo reports it, and the release fields cover the assets on the destination's releases. Each sync records them from responses it already fetches, so this endpoint never contacts the destination. A repository is listed once a sync or Refresh sizes has measured it, and a field not measured yet is `null`. `mirrored` counts the repositories with a mirror on a Gitea or Forgejo destination and `measured` the ones with a size so far; push destinations (GitHub, GitLab) are not measured. `refresh` is `null` until Refresh sizes runs, then carries the latest run's progress.
|
||||
|
||||
`POST /api/storage/refresh` starts Refresh sizes, which reads every mirror's size from the destination now instead of at its next sync. It lists each destination owner's repositories once, uses the single repository GET for a mirror the listing does not have, and reads the release listing of repositories that mirror releases. The source is never contacted. It answers `202` and runs in the background, one run per user; a call while a run is going answers with that run.
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"refresh": { "running": true, "total": 7, "done": 0, "measured": 0, "missing": 0, "failed": 0, "startedAt": "2026-10-29T06:20:28.807Z", "finishedAt": null }
|
||||
}
|
||||
```
|
||||
|
||||
`missing` counts mirrors not found on the destination and `failed` the ones a request failed for; both keep their stored numbers. A read only API key cannot start a run.
|
||||
|
||||
### Run cleanup
|
||||
|
||||
|
||||
Reference in new issue
Block a user