feat: storage page with sizes recorded during sync (#480)

- Record each mirror's destination size (Git plus LFS) and its release asset bytes and count during sync, from responses the sync already fetches, so no new requests. Four nullable columns on repositories, migration 0021.
- GET /api/storage and a Storage page with cards, bars by destination owner and for the largest repositories, and a sortable table with search, an owner filter and a release limit hint.
- Run sources.test.ts in an isolated child process so leaked @/lib/sources mocks cannot reach it.

Closes #479

Co-authored-by: Florian Zevedei <florian.dahn@fdsoftware.de>
This commit is contained in:
ARUNAVO RAYandFlorian Zevedei authored and GitHub committed 2026-10-29 10:59:54 +05:30
1 parent 4e91237d99
commit 372d04dee2
27 files changed
+4529 -12

No files matched your search

+5
View File
@@ -32,6 +32,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Raised the dependency floors for the advisories published on 2026-09-08: Astro 7.2.8 (remote code execution through AVIF image optimization, and an authorization bypass when stripping the configured base), @xmldom/xmldom 0.8.15 (eight parser and serializer issues), sharp 0.35.4 (libheif), svgo 4.1.0 (removeScripts sanitization) and js-yaml 4.3.2 (merge-key CPU use). Applied to both the application and the documentation site.
### Added
- A Storage page that shows how much space the mirrors take on the destination (#479)
- Cards for the total, Git + LFS, release assets and how many mirrors are measured, bars by destination owner and for the largest repositories, and a sortable table with search and an owner filter
- Sizes come from responses a sync already fetches: the destination repository lookup before `mirror-sync` (Git plus LFS) and the release listing the retention pass reads (release assets). No extra requests to the source or the destination
- A repository whose release assets are most of its size gets a hint that a lower release limit removes the oldest releases
- The same numbers are available from `GET /api/storage`
- Read only API keys
- **Create key** in the **API Keys** card has an **Access** choice, and read only keys show a badge in the list
- A read only key can call every `GET` endpoint; any `POST`, `PUT`, `PATCH` or `DELETE`, and every `/api/auth/*` call except `get-session`, gets `403` with the code `API_KEY_READ_ONLY`
+15
View File
@@ -156,6 +156,21 @@ When several sources are connected, an optional `sourceId` picks which one the o
`PATCH /api/organizations/:id` with `{ "sourceId": "…" }` re-pins an existing organization to another source, and `{ "sourceId": null }` clears the pin so the organization follows every source's repositories of the same name again. A pin whose source was removed falls back the same way.
### Storage
`GET /api/storage` returns what the Storage page shows: how much space the account's mirrors take on the destination (issue #479).
```json
{
"success": true,
"totals": { "total": 3554304, "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "measured": 1, "mirrored": 1 },
"owners": [{ "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "total": 3554304, "repositories": 1 }],
"repositories": [{ "id": "…", "fullName": "upstream/app", "location": "archive/app", "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "total": 3554304, "checkedAt": "2026-10-29T03:32:01.000Z" }]
}
```
Sizes are bytes. `destinationSize` is Git and LFS together as Gitea or Forgejo reports it, and the release fields cover the assets on the destination's releases. Each sync records them from responses it already fetches, so this endpoint never contacts the destination. A repository is listed once a sync has measured it, and a field not measured yet is `null`. `mirrored` counts the repositories with a mirror on a Gitea or Forgejo destination and `measured` the ones with a size so far; push destinations (GitHub, GitLab) are not measured.
### Export as CSV
`GET /api/repositories/export` and `GET /api/organizations/export` return everything the account tracks as a CSV file, so the list can be shared with people who do not have a login here (issue #428).
+4
View File
@@ -0,0 +1,4 @@
ALTER TABLE `repositories` ADD `destination_size` integer;--> statement-breakpoint
ALTER TABLE `repositories` ADD `release_assets_size` integer;--> statement-breakpoint
ALTER TABLE `repositories` ADD `release_assets_count` integer;--> statement-breakpoint
ALTER TABLE `repositories` ADD `storage_checked_at` integer;
File diff suppressed because it is too large. Load diff
+7
View File
@@ -148,6 +148,13 @@
"when": 1788805724170,
"tag": "0020_org_source",
"breakpoints": true
},
{
"idx": 21,
"version": "6",
"when": 1793244040274,
"tag": "0021_repository_storage",
"breakpoints": true
}
]
}
+47
View File
@@ -1007,8 +1007,55 @@ const latestUpgradeFixtures: Record<string, UpgradeFixture> = {
seed: seedPre0020Database,
verify: verify0020Migration,
},
"0021_repository_storage": {
seed: seedPre0021Database,
verify: verify0021Migration,
},
};
function seedPre0021Database(db: any) {
// Migrations 0000-0020 have run: repositories have no storage columns yet.
db.run("INSERT INTO users (id, email, username, name) VALUES ('u-st21', 'st21@example.com', 'st21', 'Storage Twenty-One')");
db.run(
"INSERT INTO configs (id, user_id, name, is_active, github_config, gitea_config, schedule_config, cleanup_config) " +
"VALUES ('cfg-st21', 'u-st21', 'Default', 1, '{\"provider\":\"github\",\"owner\":\"octocat\",\"token\":\"gh-token\",\"type\":\"personal\"}', '{}', '{}', '{}')",
);
db.run(
"INSERT INTO repositories (id, user_id, config_id, name, full_name, normalized_full_name, url, clone_url, owner, default_branch, status, mirrored_location) " +
"VALUES ('repo-st21', 'u-st21', 'cfg-st21', 'app', 'octocat/app', 'octocat/app', 'https://github.com/octocat/app', 'https://github.com/octocat/app.git', 'octocat', 'main', 'synced', 'mirror/app')",
);
}
function verify0021Migration(db: any) {
const columns = db.query("PRAGMA table_info(repositories)").all() as TableInfoRow[];
for (const name of ["destination_size", "release_assets_size", "release_assets_count", "storage_checked_at"]) {
const column = columns.find((c) => c.name === name);
assert(column, `Expected repositories.${name} to exist after migration`);
assert(column.notnull === 0, `Expected repositories.${name} to be nullable`);
assert(column.dflt_value === null, `Expected repositories.${name} to have no default, got ${column.dflt_value}`);
}
const existing = db
.query("SELECT destination_size, release_assets_size, release_assets_count, storage_checked_at FROM repositories WHERE id = 'repo-st21'")
.get() as Record<string, unknown> | null;
assert(existing, "Expected the existing repository to survive the migration");
assert(
Object.values(existing).every((value) => value === null),
`Expected an existing repository to start unmeasured, got ${JSON.stringify(existing)}`,
);
db.run(
"UPDATE repositories SET destination_size = 1048576, release_assets_size = 2048, release_assets_count = 3, storage_checked_at = 1793000000 WHERE id = 'repo-st21'",
);
const measured = db
.query("SELECT destination_size, release_assets_size, release_assets_count FROM repositories WHERE id = 'repo-st21'")
.get() as { destination_size: number; release_assets_size: number; release_assets_count: number } | null;
assert(
measured?.destination_size === 1048576 && measured.release_assets_size === 2048 && measured.release_assets_count === 3,
`Expected the storage columns to round-trip, got ${JSON.stringify(measured)}`,
);
}
function seedPre0020Database(db: any) {
// Migrations 0000-0019 have run: repositories carry source_id,
// organizations have none yet. Seed two sources and three organizations:
+1 -1
View File
@@ -10,7 +10,7 @@ import { AccountMenu } from "@/components/auth/AccountMenu";
import { withBase } from "@/lib/base-path";
interface HeaderProps {
currentPage?: "dashboard" | "repositories" | "organizations" | "configuration" | "activity-log";
currentPage?: "dashboard" | "repositories" | "organizations" | "storage" | "configuration" | "activity-log";
onNavigate?: (page: string) => void;
onMenuClick: () => void;
onToggleCollapse?: () => void;
+4
View File
@@ -7,6 +7,7 @@ import Providers from "./Providers";
import { ConfigTabs } from "../config/ConfigTabs";
import { ActivityLog } from "../activity/ActivityLog";
import { Organization } from "../organizations/Organization";
import { Storage } from "../storage/Storage";
import { Toaster } from "@/components/ui/sonner";
import { useAuth } from "@/hooks/useAuth";
import { useRepoSync } from "@/hooks/useSyncRepo";
@@ -23,6 +24,7 @@ interface AppProps {
| "dashboard"
| "repositories"
| "organizations"
| "storage"
| "configuration"
| "activity-log";
"client:load"?: boolean;
@@ -77,6 +79,7 @@ function AppWithProviders({ page: initialPage }: AppProps) {
'/': 'dashboard',
'/repositories': 'repositories',
'/organizations': 'organizations',
'/storage': 'storage',
'/config': 'configuration',
'/activity': 'activity-log'
};
@@ -156,6 +159,7 @@ function AppWithProviders({ page: initialPage }: AppProps) {
{currentPage === "dashboard" && <Dashboard />}
{currentPage === "repositories" && <Repository />}
{currentPage === "organizations" && <Organization />}
{currentPage === "storage" && <Storage />}
{currentPage === "configuration" && <ConfigTabs />}
{currentPage === "activity-log" && <ActivityLog />}
</section>
+1
View File
@@ -55,6 +55,7 @@ export function Sidebar({ className, onNavigate, isOpen, isCollapsed = false, on
'/': 'dashboard',
'/repositories': 'repositories',
'/organizations': 'organizations',
'/storage': 'storage',
'/config': 'configuration',
'/activity': 'activity-log'
};
+456
View File
@@ -0,0 +1,456 @@
import { useCallback, useEffect, useMemo, useState } from "react";
import {
ArrowDown,
ArrowUp,
Database,
HardDrive,
Info,
Package,
ScanSearch,
Search,
} from "lucide-react";
import { toast } from "sonner";
import { StatusCard } from "@/components/dashboard/StatusCard";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Skeleton } from "@/components/ui/skeleton";
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select";
import {
Tooltip,
TooltipContent,
TooltipProvider,
TooltipTrigger,
} from "@/components/ui/tooltip";
import { useAuth } from "@/hooks/useAuth";
import { useGiteaConfig } from "@/hooks/useGiteaConfig";
import { useLiveRefresh } from "@/hooks/useLiveRefresh";
import { useNavigation } from "@/components/layout/MainLayout";
import { apiRequest, cn, formatLastSyncTime } from "@/lib/utils";
import { buildGiteaWebUrl } from "@/lib/gitea-url";
import { formatBytes } from "@/lib/utils/format-bytes";
import type { StorageOverview, StorageRepository } from "@/lib/repository-storage";
type StorageApiResponse = ({ success: true } & StorageOverview) | { success: false; error?: string };
type SortKey = "fullName" | "owner" | "destinationSize" | "releaseAssetsSize" | "releaseAssetsCount" | "total" | "checkedAt";
const BAR_ROWS = 8;
// Repositories where release assets are at least half the space, and enough
// of it to matter, get a hint about the release limit.
const RELEASE_HEAVY_SHARE = 0.5;
const RELEASE_HEAVY_MIN_BYTES = 50 * 1024 * 1024;
function percent(part: number, whole: number): number {
return whole > 0 ? Math.round((part / whole) * 100) : 0;
}
function isReleaseHeavy(repo: StorageRepository): boolean {
const assets = repo.releaseAssetsSize ?? 0;
return assets >= RELEASE_HEAVY_MIN_BYTES && assets >= repo.total * RELEASE_HEAVY_SHARE;
}
/** Git + LFS and release assets as one bar, scaled against the panel's largest row. */
function SplitBar({ git, assets, max }: { git: number; assets: number; max: number }) {
const width = (value: number) => `${max > 0 ? (value / max) * 100 : 0}%`;
return (
<div className="flex h-2 w-full overflow-hidden rounded-full bg-muted">
<div className="h-full bg-indigo-500" style={{ width: width(git) }} />
<div className="h-full bg-amber-500" style={{ width: width(assets) }} />
</div>
);
}
function Legend() {
return (
<div className="flex items-center gap-3 text-xs font-normal text-muted-foreground">
<span className="flex items-center gap-1.5">
<span className="h-2 w-2 rounded-full bg-indigo-500" />
Git + LFS
</span>
<span className="flex items-center gap-1.5">
<span className="h-2 w-2 rounded-full bg-amber-500" />
Release assets
</span>
</div>
);
}
interface BarRow {
key: string;
label: string;
detail?: string;
git: number;
assets: number;
total: number;
}
function BarPanel({ title, icon, rows }: { title: string; icon: React.ReactNode; rows: BarRow[] }) {
const max = rows.reduce((largest, row) => Math.max(largest, row.total), 0);
return (
<Card className="h-full w-full">
<CardHeader className="flex flex-row flex-wrap items-center justify-between gap-2">
<CardTitle className="flex items-center gap-3 text-base font-semibold">
{icon}
{title}
</CardTitle>
<Legend />
</CardHeader>
<CardContent className="flex flex-col gap-3.5">
{rows.map((row) => (
<div key={row.key} className="flex flex-col gap-1.5">
<div className="flex items-baseline justify-between gap-3 text-sm">
<span className="min-w-0 truncate font-medium" title={row.label}>
{row.label}
{row.detail && (
<span className="ml-2 text-xs font-normal text-muted-foreground">{row.detail}</span>
)}
</span>
<span className="shrink-0 tabular-nums text-muted-foreground">{formatBytes(row.total)}</span>
</div>
<SplitBar git={row.git} assets={row.assets} max={max} />
</div>
))}
</CardContent>
</Card>
);
}
function SortHeader({
label,
sortKey,
sort,
onSort,
className,
}: {
label: string;
sortKey: SortKey;
sort: { key: SortKey; descending: boolean };
onSort: (key: SortKey) => void;
className?: string;
}) {
const active = sort.key === sortKey;
const Arrow = sort.descending ? ArrowDown : ArrowUp;
return (
<button
type="button"
onClick={() => onSort(sortKey)}
className={cn("flex items-center gap-1 hover:text-foreground", active ? "text-foreground" : "text-muted-foreground", className)}
>
{label}
{active && <Arrow className="h-3.5 w-3.5" />}
</button>
);
}
function compareRepositories(a: StorageRepository, b: StorageRepository, key: SortKey): number {
switch (key) {
case "fullName":
return a.fullName.localeCompare(b.fullName);
case "owner":
return a.owner.localeCompare(b.owner) || a.fullName.localeCompare(b.fullName);
case "checkedAt":
return (a.checkedAt ?? "").localeCompare(b.checkedAt ?? "");
default:
return (a[key] ?? -1) - (b[key] ?? -1);
}
}
function StorageSkeleton() {
return (
<div className="flex flex-col gap-y-6">
<div className="grid grid-cols-2 lg:grid-cols-4 gap-3 sm:gap-4 md:gap-6">
{Array.from({ length: 4 }).map((_, i) => (
<Skeleton key={i} className="h-[104px] rounded-xl" />
))}
</div>
<div className="flex flex-col lg:flex-row gap-6">
<Skeleton className="h-72 w-full lg:w-1/2 rounded-xl" />
<Skeleton className="h-72 w-full lg:w-1/2 rounded-xl" />
</div>
<Skeleton className="h-96 w-full rounded-xl" />
</div>
);
}
/**
* Storage the mirrors take up on the destination (#479). Every number was
* recorded by a sync from data it fetched anyway, so this page only reads the
* database.
*/
export function Storage() {
const { user } = useAuth();
const { registerRefreshCallback } = useLiveRefresh();
const { navigationKey } = useNavigation();
const { giteaConfig } = useGiteaConfig();
const [overview, setOverview] = useState<StorageOverview | null>(null);
const [isLoading, setIsLoading] = useState(true);
const [search, setSearch] = useState("");
const [ownerFilter, setOwnerFilter] = useState("all");
const [sort, setSort] = useState<{ key: SortKey; descending: boolean }>({ key: "total", descending: true });
const fetchOverview = useCallback(
async (isLiveRefresh = false) => {
if (!user?.id) return;
try {
const response = await apiRequest<StorageApiResponse>("/storage", { method: "GET" });
if (response.success) {
setOverview(response);
} else if (!isLiveRefresh) {
toast.error(response.error || "Error fetching storage");
}
} catch (error) {
if (!isLiveRefresh) {
toast.error(error instanceof Error ? error.message : "Error fetching storage");
}
} finally {
if (!isLiveRefresh) setIsLoading(false);
}
},
[user?.id]
);
useEffect(() => {
setIsLoading(true);
fetchOverview(false);
}, [fetchOverview, navigationKey]);
useEffect(() => registerRefreshCallback(() => fetchOverview(true)), [registerRefreshCallback, fetchOverview]);
const handleSort = (key: SortKey) =>
setSort((prev) =>
prev.key === key
? { key, descending: !prev.descending }
: { key, descending: key !== "fullName" && key !== "owner" }
);
const visibleRepositories = useMemo(() => {
if (!overview) return [];
const term = search.trim().toLowerCase();
const rows = overview.repositories.filter(
(repo) =>
(ownerFilter === "all" || repo.owner === ownerFilter) &&
(!term || repo.fullName.toLowerCase().includes(term) || repo.location.toLowerCase().includes(term))
);
rows.sort((a, b) => {
const order = compareRepositories(a, b, sort.key);
return sort.descending ? -order : order;
});
return rows;
}, [overview, search, ownerFilter, sort]);
if (isLoading || !overview) return <StorageSkeleton />;
const { totals } = overview;
if (totals.measured === 0) {
return (
<Card>
<CardContent className="flex flex-col items-center justify-center py-16 text-center">
<HardDrive className="h-10 w-10 text-muted-foreground mb-4" />
<h3 className="text-lg font-medium">No sizes recorded yet</h3>
<p className="text-sm text-muted-foreground mt-1 max-w-md">
Each sync records how much space the mirror takes on the destination, from data the sync
already fetches. Sizes show up here after the next sync.
</p>
</CardContent>
</Card>
);
}
const ownerRows: BarRow[] = overview.owners.slice(0, BAR_ROWS).map((owner) => ({
key: owner.owner || "(unknown)",
label: owner.owner || "Unknown owner",
detail: `${owner.repositories} ${owner.repositories === 1 ? "repo" : "repos"}`,
git: owner.destinationSize,
assets: owner.releaseAssetsSize,
total: owner.total,
}));
const largestRows: BarRow[] = overview.repositories.slice(0, BAR_ROWS).map((repo) => ({
key: repo.id,
label: repo.location || repo.fullName,
git: repo.destinationSize ?? 0,
assets: repo.releaseAssetsSize ?? 0,
total: repo.total,
}));
const unmeasured = Math.max(0, totals.mirrored - totals.measured);
return (
<TooltipProvider>
<div className="flex flex-col gap-y-6">
<div className="grid grid-cols-2 lg:grid-cols-4 gap-3 sm:gap-4 md:gap-6">
<StatusCard
title="Total stored"
value={formatBytes(totals.total)}
icon={<HardDrive className="h-4 w-4" />}
description={`Across ${totals.measured} ${totals.measured === 1 ? "repository" : "repositories"}`}
/>
<StatusCard
title="Git + LFS"
value={formatBytes(totals.destinationSize)}
icon={<Database className="h-4 w-4" />}
description="As the destination reports it"
/>
<StatusCard
title="Release assets"
value={formatBytes(totals.releaseAssetsSize)}
icon={<Package className="h-4 w-4" />}
description={`${percent(totals.releaseAssetsSize, totals.total)}% of the total, ${totals.releaseAssetsCount.toLocaleString()} files`}
/>
<StatusCard
title="Measured"
value={`${totals.measured} / ${totals.mirrored}`}
icon={<ScanSearch className="h-4 w-4" />}
description={unmeasured > 0 ? "The rest are measured on their next sync" : "Every mirror is measured"}
/>
</div>
<div className="flex flex-col lg:flex-row gap-6">
<div className="w-full lg:w-1/2">
<BarPanel title="By owner" icon={<HardDrive className="h-5 w-5 text-muted-foreground" />} rows={ownerRows} />
</div>
<div className="w-full lg:w-1/2">
<BarPanel title="Largest repositories" icon={<Database className="h-5 w-5 text-muted-foreground" />} rows={largestRows} />
</div>
</div>
<div className="flex flex-col gap-3">
<div className="flex flex-col sm:flex-row gap-2">
<div className="relative flex-grow min-w-0">
<Search className="absolute left-3 top-1/2 -translate-y-1/2 h-4 w-4 text-muted-foreground" />
<input
type="text"
placeholder="Search repositories..."
className="pl-10 pr-3 h-10 w-full rounded-md border border-input bg-background text-sm shadow-sm transition-colors placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring"
value={search}
onChange={(event) => setSearch(event.target.value)}
/>
</div>
<Select value={ownerFilter} onValueChange={setOwnerFilter}>
<SelectTrigger className="h-10 w-full sm:w-52">
<SelectValue placeholder="All owners" />
</SelectTrigger>
<SelectContent>
<SelectItem value="all">All owners</SelectItem>
{overview.owners
.filter((owner) => owner.owner)
.map((owner) => (
<SelectItem key={owner.owner} value={owner.owner}>
{owner.owner}
</SelectItem>
))}
</SelectContent>
</Select>
</div>
<div className="overflow-x-auto" data-testid="storage-table">
<div className="min-w-[56rem] border rounded-md">
<div className="h-[45px] flex items-center border-b bg-muted/50 text-sm font-medium">
<div className="h-full p-3 flex items-center flex-[2.4] sticky left-0 z-10 bg-background before:absolute before:inset-0 before:-z-10 before:bg-muted/50">
<SortHeader label="Repository" sortKey="fullName" sort={sort} onSort={handleSort} />
</div>
<div className="h-full p-3 flex items-center flex-[1.1]">
<SortHeader label="Owner" sortKey="owner" sort={sort} onSort={handleSort} />
</div>
<div className="h-full p-3 flex items-center justify-end flex-[1]">
<SortHeader label="Git + LFS" sortKey="destinationSize" sort={sort} onSort={handleSort} />
</div>
<div className="h-full p-3 flex items-center justify-end flex-[1.1]">
<SortHeader label="Release assets" sortKey="releaseAssetsSize" sort={sort} onSort={handleSort} />
</div>
<div className="h-full p-3 flex items-center justify-end flex-[0.7]">
<SortHeader label="Files" sortKey="releaseAssetsCount" sort={sort} onSort={handleSort} />
</div>
<div className="h-full p-3 flex items-center justify-end flex-[1]">
<SortHeader label="Total" sortKey="total" sort={sort} onSort={handleSort} />
</div>
<div className="h-full p-3 flex items-center justify-end flex-[1]">
<SortHeader label="Checked" sortKey="checkedAt" sort={sort} onSort={handleSort} />
</div>
</div>
{visibleRepositories.length === 0 && (
<div className="p-6 text-center text-sm text-muted-foreground">No repositories match.</div>
)}
{visibleRepositories.map((repo) => {
const destinationUrl = repo.location ? buildGiteaWebUrl(giteaConfig, repo.location) : null;
const heavy = isReleaseHeavy(repo);
return (
<div
key={repo.id}
data-testid="storage-row"
className="group min-h-[48px] flex items-center border-b last:border-b-0 hover:bg-muted/50 text-sm"
>
<div className="h-full px-3 py-2 flex items-center flex-[2.4] min-w-0 sticky left-0 z-10 bg-background before:absolute before:inset-0 before:-z-10 group-hover:before:bg-muted/50">
<div className="min-w-0">
{destinationUrl ? (
<a
href={destinationUrl}
target="_blank"
rel="noopener noreferrer"
className="block font-medium truncate hover:underline"
title={`Open ${repo.location} on the destination`}
>
{repo.location}
</a>
) : (
<span className="block font-medium truncate">{repo.location || repo.fullName}</span>
)}
{repo.location && repo.location !== repo.fullName && (
<span className="block text-xs text-muted-foreground truncate">from {repo.fullName}</span>
)}
</div>
</div>
<div className="h-full px-3 py-2 flex items-center flex-[1.1] min-w-0 text-muted-foreground truncate">
{repo.owner || "Unknown"}
</div>
<div className="h-full px-3 py-2 flex items-center justify-end flex-[1] tabular-nums">
{repo.destinationSize === null ? <span className="text-muted-foreground">-</span> : formatBytes(repo.destinationSize)}
</div>
<div className="h-full px-3 py-2 flex items-center justify-end gap-1.5 flex-[1.1] tabular-nums">
{heavy && (
<Tooltip>
<TooltipTrigger asChild>
<Info className="h-3.5 w-3.5 shrink-0 text-amber-500" aria-label="Release assets take most of this repository" />
</TooltipTrigger>
<TooltipContent className="max-w-xs">
Release assets are {percent(repo.releaseAssetsSize ?? 0, repo.total)}% of this repository. A lower
release limit in its Mirror options removes the oldest releases on the next sync.
</TooltipContent>
</Tooltip>
)}
{repo.releaseAssetsSize === null ? <span className="text-muted-foreground">-</span> : formatBytes(repo.releaseAssetsSize)}
</div>
<div className="h-full px-3 py-2 flex items-center justify-end flex-[0.7] tabular-nums text-muted-foreground">
{repo.releaseAssetsCount === null ? "-" : repo.releaseAssetsCount.toLocaleString()}
</div>
<div className="h-full px-3 py-2 flex items-center justify-end flex-[1] tabular-nums font-medium">
{formatBytes(repo.total)}
</div>
<div
className="h-full px-3 py-2 flex items-center justify-end flex-[1] text-muted-foreground"
title={repo.checkedAt ? new Date(repo.checkedAt).toLocaleString() : undefined}
>
{formatLastSyncTime(repo.checkedAt)}
</div>
</div>
);
})}
</div>
</div>
<p className="text-xs text-muted-foreground">
Sizes are what the destination reports: Git and LFS together, plus the release assets it stores. Each sync
records them from data it already fetches. The Git size is read before the sync pulls, so it trails by one
sync.
</p>
</div>
</div>
</TooltipProvider>
);
}
+2
View File
@@ -4,6 +4,7 @@ import {
Settings,
Activity,
Building2,
HardDrive,
} from "lucide-react";
import type { SidebarItem } from "@/types/Sidebar";
@@ -11,6 +12,7 @@ export const links: SidebarItem[] = [
{ href: "/", label: "Dashboard", icon: LayoutDashboard },
{ href: "/repositories", label: "Repositories", icon: GitFork },
{ href: "/organizations", label: "Organizations", icon: Building2 },
{ href: "/storage", label: "Storage", icon: HardDrive },
{ href: "/config", label: "Configuration", icon: Settings },
{ href: "/activity", label: "Activity Log", icon: Activity },
];
+13
View File
@@ -322,6 +322,10 @@ export const repositorySchema = z.object({
size: z.number().default(0),
hasLFS: z.boolean().default(false),
hasSubmodules: z.boolean().default(false),
destinationSize: z.number().optional().nullable(),
releaseAssetsSize: z.number().optional().nullable(),
releaseAssetsCount: z.number().optional().nullable(),
storageCheckedAt: z.coerce.date().optional().nullable(),
language: z.string().optional().nullable(),
description: z.string().optional().nullable(),
defaultBranch: z.string(),
@@ -623,6 +627,15 @@ export const repositories = sqliteTable("repositories", {
.notNull()
.default(false),
// What the mirror takes up on the destination, in bytes, recorded from
// responses a sync already fetches (#479). NULL until a sync has seen it.
// destinationSize is Git plus LFS as the destination reports it; the release
// asset totals come from the listing the release retention pass reads.
destinationSize: integer("destination_size"),
releaseAssetsSize: integer("release_assets_size"),
releaseAssetsCount: integer("release_assets_count"),
storageCheckedAt: integer("storage_checked_at", { mode: "timestamp" }),
language: text("language"),
description: text("description"),
defaultBranch: text("default_branch").notNull(),
+61
View File
@@ -998,6 +998,67 @@ describe("Enhanced Gitea Operations", () => {
expect(String(mirrorSyncCalls[0][0])).not.toContain("/api/v1/repos/ceph/test-repo/mirror-sync");
});
describe("destination size (#479)", () => {
const sizeConfig = {
userId: "user123",
githubConfig: { username: "testuser", token: "github-token", privateRepositories: false, mirrorStarred: true },
giteaConfig: { url: "https://gitea.example.com", token: "encrypted-token", defaultOwner: "testuser", mirrorReleases: false },
} as unknown as Partial<Config>;
const sizeRepository = (): Repository => ({
id: "repo-size", name: "test-repo", fullName: "ceph/test-repo", owner: "ceph",
cloneUrl: "https://github.com/ceph/test-repo.git", isPrivate: false, isStarred: true,
status: repoStatusEnum.parse("mirrored"), visibility: "public", userId: "user123",
mirroredLocation: "starred/test-repo", createdAt: new Date(), updatedAt: new Date(),
} as Repository);
const sizeDependencies = {
getGiteaRepoOwnerAsync: mockGetGiteaRepoOwnerAsync,
mirrorGitHubReleasesToGitea: mockMirrorGitHubReleasesToGitea,
mirrorGitRepoIssuesToGitea: mockMirrorGitRepoIssuesToGitea,
mirrorGitRepoPullRequestsToGitea: mockMirrorGitRepoPullRequestsToGitea,
mirrorGitRepoLabelsToGitea: mockMirrorGitRepoLabelsToGitea,
mirrorGitRepoMilestonesToGitea: mockMirrorGitRepoMilestonesToGitea,
syncRepositoryMetadataToGitea: mockSyncRepositoryMetadataToGitea,
};
const repoUrl = "https://gitea.example.com/api/v1/repos/starred/test-repo";
test("records the size from the lookup the sync already makes", async () => {
const defaultGet = mockHttpGet.getMockImplementation()!;
mockHttpGet.mockImplementation(async (url: string, headers?: any) => {
const response: any = await defaultGet(url, headers);
return url === repoUrl ? { ...response, data: { ...response.data, size: 2048 } } : response;
});
try {
const result = await syncGiteaRepoEnhanced(
{ config: sizeConfig, repository: sizeRepository() },
sizeDependencies
);
expect(result).toEqual({ success: true });
const successUpdate = dbUpdateSetCalls.find((data) => "lastMirrored" in data);
expect(successUpdate.destinationSize).toBe(2048 * 1024);
expect(successUpdate.storageCheckedAt).toBeInstanceOf(Date);
// Read from the target lookup, not fetched again.
expect(mockHttpGet.mock.calls.filter((call) => call[0] === repoUrl)).toHaveLength(1);
} finally {
mockHttpGet.mockImplementation(defaultGet);
}
});
test("leaves the stored size alone when the destination reports none", async () => {
const result = await syncGiteaRepoEnhanced(
{ config: sizeConfig, repository: sizeRepository() },
sizeDependencies
);
expect(result).toEqual({ success: true });
const successUpdate = dbUpdateSetCalls.find((data) => "lastMirrored" in data);
expect(successUpdate).toBeDefined();
expect("destinationSize" in successUpdate).toBe(false);
expect("storageCheckedAt" in successUpdate).toBe(false);
});
});
describe("GitHub App sources", () => {
const appConfig = {
userId: "user123",
+6
View File
@@ -44,6 +44,7 @@ import {
strategyNeedsDetection,
} from "./repo-backup";
import { detectForcePush } from "./utils/force-push-detection";
import { destinationSizeColumns } from "./repository-storage";
import { sanitizeRepoNameAlphaDashDot, type GiteaMilestoneIds } from "./gitea";
import {
parseRepositoryMetadataState,
@@ -86,6 +87,8 @@ export interface GiteaRepoInfo {
// skip writes that would not change anything.
description?: string;
topics?: string[];
// KiB, Git plus LFS, recorded for the storage page (#479).
size?: number;
// Set while a transfer waits for the new owner to accept it (issue #400).
repo_transfer?: { recipient?: { login?: string; username?: string } | null } | null;
}
@@ -1292,6 +1295,9 @@ export async function syncGiteaRepoEnhanced({
metadata: metadataUpdated
? serializeRepositoryMetadataState(metadataState)
: repository.metadata ?? null,
// From the GET that resolved the target, so before this sync's
// fetch: the size lags one sync and costs no request.
...destinationSizeColumns(repoInfo),
})
.where(eq(repositories.id, repository.id!));
+87 -2
View File
@@ -64,7 +64,22 @@ interface ActivityJob {
}
let activityJobs: ActivityJob[] = [];
// The release asset totals the retention pass records for the storage page.
interface StorageRecord {
repositoryId: string | null | undefined;
summary: { count: number; bytes: number };
}
let storageRecords: StorageRecord[] = [];
if (isChild) {
mock.module("@/lib/repository-storage-store", () => ({
recordReleaseAssetStorage: async (
repositoryId: string | null | undefined,
summary: { count: number; bytes: number }
) => {
storageRecords.push({ repositoryId, summary });
},
}));
mock.module("@/lib/notification-service", () => ({
triggerAllReleasesRemovedNotification: async (notice: RemovalNotice) => {
removalNotices.push(notice);
@@ -94,6 +109,7 @@ afterEach(() => {
globalThis.fetch = realFetch;
removalNotices = [];
activityJobs = [];
storageRecords = [];
});
const config = {
@@ -132,6 +148,10 @@ interface DestinationOptions {
destinationTags?: string[];
/** Releases already on the destination, as the retention pass lists them. */
destinationReleases?: Array<Record<string, unknown>>;
/** The destination answers the retention listing with an error. */
destinationListingFails?: boolean;
/** Destination release ids whose DELETE fails. */
failDeleteIds?: number[];
}
function fakeWorld(options: DestinationOptions) {
@@ -205,12 +225,17 @@ function fakeWorld(options: DestinationOptions) {
// The retention pass removing a release.
if (method === "DELETE" && /\/releases\/\d+$/.test(url)) {
return new Response(null, { status: 204 });
const id = Number(url.slice(url.lastIndexOf("/") + 1));
return options.failDeleteIds?.includes(id)
? new Response("boom", { status: 500 })
: new Response(null, { status: 204 });
}
// The retention pass lists everything the destination has.
if (method === "GET" && url.startsWith(`${DEST_API}/releases?`)) {
return json(options.destinationReleases ?? []);
return options.destinationListingFails
? new Response("boom", { status: 500 })
: json(options.destinationReleases ?? []);
}
throw new Error(`unexpected request: ${method} ${url}`);
@@ -507,6 +532,66 @@ describe.skipIf(!isChild)("release type filter", () => {
});
});
describe.skipIf(!isChild)("release asset storage (#479)", () => {
const withAssets = (release: Record<string, unknown>, sizes: number[]) => ({
...release,
assets: sizes.map((size, index) => ({ id: index + 1, name: `file-${index}`, size })),
});
it("records the assets left on the destination after retention", async () => {
fakeWorld({
sourceReleases: [giteaRelease("v2.0.0")],
destinationReleases: [
withAssets(destinationRelease("v1.9.0-beta", 11, true), [100, 50]),
withAssets(destinationRelease("v1.8.0", 12, false), [1000]),
],
});
await mirrorFromGitea(giteaConnection("source-token"), { releaseType: "stable" });
expect(storageRecords).toEqual([{ repositoryId: "repo-1", summary: { count: 1, bytes: 1000 } }]);
});
it("keeps counting a release whose delete failed", async () => {
fakeWorld({
sourceReleases: [giteaRelease("v2.0.0")],
destinationReleases: [
withAssets(destinationRelease("v1.9.0-beta", 11, true), [100, 50]),
withAssets(destinationRelease("v1.8.0", 12, false), [1000]),
],
failDeleteIds: [11],
});
await mirrorFromGitea(giteaConnection("source-token"), { releaseType: "stable" });
expect(storageRecords).toEqual([{ repositoryId: "repo-1", summary: { count: 3, bytes: 1150 } }]);
});
it("records nothing when the listing fails", async () => {
fakeWorld({
sourceReleases: [giteaRelease("v2.0.0")],
destinationListingFails: true,
});
await mirrorFromGitea(giteaConnection("source-token"));
expect(storageRecords).toEqual([]);
});
it("makes no request of its own", async () => {
const world = fakeWorld({
sourceReleases: [giteaRelease("v2.0.0")],
destinationReleases: [withAssets(destinationRelease("v2.0.0", 12, false), [10])],
});
await mirrorFromGitea(giteaConnection("source-token"));
const listings = world.requests.filter((r) => r.url.startsWith(`${DEST_API}/releases?`));
expect(listings).toHaveLength(1);
expect(storageRecords).toEqual([{ repositoryId: "repo-1", summary: { count: 1, bytes: 10 } }]);
});
});
describe.skipIf(!isChild)("all releases removed warning", () => {
it("warns when the filters leave the destination with no releases", async () => {
const world = fakeWorld({
+26 -1
View File
@@ -35,6 +35,8 @@ import { triggerAllReleasesRemovedNotification } from "./notification-service";
import { isRateLimitError } from "./rate-limit-gate";
import { assertMirrorCredentialUpdateSupported } from "./gitea-mirror-credentials";
import { persistMetadataPassProgress } from "./metadata-progress-store";
import { summarizeReleaseAssets } from "./repository-storage";
import { recordReleaseAssetStorage } from "./repository-storage-store";
import {
advanceMetadataSyncCursor,
buildPassProgress,
@@ -4380,7 +4382,12 @@ async function mirrorSourceReleasesToGiteaLocked({
tag_name: string;
created_at: string;
prerelease?: boolean | null;
assets?: Array<{ size?: number }> | null;
}> = [];
// A page that fails ends the listing early. Retention still runs on what
// was listed, as before, but the storage totals are only recorded from a
// complete listing.
let listingComplete = true;
let cleanupPage = 1;
while (true) {
const pageResponse = await httpGet(
@@ -4390,7 +4397,11 @@ async function mirrorSourceReleasesToGiteaLocked({
}
).catch(() => null);
if (!pageResponse?.data || !Array.isArray(pageResponse.data) || pageResponse.data.length === 0) {
if (!pageResponse?.data || !Array.isArray(pageResponse.data)) {
listingComplete = false;
break;
}
if (pageResponse.data.length === 0) {
break;
}
@@ -4404,6 +4415,7 @@ async function mirrorSourceReleasesToGiteaLocked({
const giteaBaseUrl = config.giteaConfig.url;
const giteaToken = decryptedConfig.giteaConfig.token;
const deletedReleaseIds = new Set<number>();
const deleteGiteaRelease = async (
release: { id: number; tag_name: string },
@@ -4416,6 +4428,7 @@ async function mirrorSourceReleasesToGiteaLocked({
Authorization: `token ${giteaToken}`,
}
);
deletedReleaseIds.add(release.id);
console.log(`[Releases] Deleted ${reason} release: ${release.tag_name}`);
} catch (deleteError) {
console.error(
@@ -4520,6 +4533,18 @@ async function mirrorSourceReleasesToGiteaLocked({
await deleteGiteaRelease(excess, "excess");
}
}
// The listing already carries every asset's size, so the storage page
// gets its release numbers without another request (#479). Releases whose
// delete failed are still on the destination and still count.
if (listingComplete) {
await recordReleaseAssetStorage(
repository.id,
summarizeReleaseAssets(
allGiteaReleases.filter((rel) => !deletedReleaseIds.has(rel.id))
)
);
}
} catch (cleanupError) {
console.warn(
`[Releases] Release retention cleanup failed: ${
+33
View File
@@ -0,0 +1,33 @@
/**
* Write the release asset totals the retention pass measured (#479). The
* release pass has no write of its own at the end, so it records them here.
*/
import { eq } from "drizzle-orm";
import { db, repositories } from "./db";
import type { ReleaseAssetSummary } from "./repository-storage";
/** Never throws: a failed write must not fail the release pass. */
export async function recordReleaseAssetStorage(
repositoryId: string | null | undefined,
summary: ReleaseAssetSummary
): Promise<void> {
if (!repositoryId) return;
try {
await db
.update(repositories)
.set({
releaseAssetsSize: summary.bytes,
releaseAssetsCount: summary.count,
storageCheckedAt: new Date(),
})
.where(eq(repositories.id, repositoryId));
} catch (error) {
console.warn(
`[Storage] Could not record release asset sizes for repository ${repositoryId}: ${
error instanceof Error ? error.message : String(error)
}`
);
}
}
+126
View File
@@ -0,0 +1,126 @@
import { describe, expect, test } from "bun:test";
import {
buildStorageOverview,
destinationSizeColumns,
summarizeReleaseAssets,
type StorageSourceRow,
} from "./repository-storage";
describe("destinationSizeColumns", () => {
test("turns the KiB Gitea reports into bytes", () => {
const now = new Date("2026-10-29T00:00:00Z");
expect(destinationSizeColumns({ size: 1536 }, now)).toEqual({
destinationSize: 1536 * 1024,
storageCheckedAt: now,
});
});
test("records an empty repository as zero", () => {
expect(destinationSizeColumns({ size: 0 }).destinationSize).toBe(0);
});
test("writes nothing without a usable size", () => {
expect(destinationSizeColumns({})).toEqual({});
expect(destinationSizeColumns(null)).toEqual({});
expect(destinationSizeColumns({ size: "12" })).toEqual({});
expect(destinationSizeColumns({ size: -1 })).toEqual({});
expect(destinationSizeColumns({ size: Number.NaN })).toEqual({});
});
});
describe("summarizeReleaseAssets", () => {
test("counts every asset and adds up their sizes", () => {
expect(
summarizeReleaseAssets([
{ assets: [{ size: 100 }, { size: 50 }] },
{ assets: [] },
{ assets: null },
{},
{ assets: [{ size: 1000 }] },
])
).toEqual({ count: 3, bytes: 1150 });
});
test("counts an asset without a size but adds nothing for it", () => {
expect(summarizeReleaseAssets([{ assets: [{}, { size: "9" }, { size: 4 }] }])).toEqual({
count: 3,
bytes: 4,
});
});
});
function row(overrides: Partial<StorageSourceRow>): StorageSourceRow {
return {
id: "r",
name: "repo",
fullName: "acme/repo",
mirroredLocation: "archive/repo",
status: "synced",
destinationProvider: "gitea",
destinationSize: null,
releaseAssetsSize: null,
releaseAssetsCount: null,
storageCheckedAt: null,
...overrides,
};
}
describe("buildStorageOverview", () => {
const checkedAt = new Date("2026-10-29T08:00:00Z");
const rows = [
row({ id: "a", fullName: "acme/app", mirroredLocation: "archive/app", destinationSize: 300, releaseAssetsSize: 700, releaseAssetsCount: 4, storageCheckedAt: checkedAt }),
row({ id: "b", fullName: "acme/lib", mirroredLocation: "archive/lib", destinationSize: 200, storageCheckedAt: checkedAt }),
row({ id: "c", fullName: "me/dots", mirroredLocation: "me/dots", destinationSize: 50 }),
// Mirrored but not synced since the upgrade.
row({ id: "d", fullName: "acme/new", status: "mirrored" }),
// Imported only: neither measured nor mirrored.
row({ id: "e", fullName: "acme/later", status: "imported", mirroredLocation: "" }),
// A failed first mirror has nothing on the destination yet.
row({ id: "f", fullName: "acme/broken", status: "failed" }),
// Push destinations have no size API; they are left out entirely.
row({ id: "g", fullName: "acme/pushed", destinationProvider: "github", status: "synced", destinationSize: 999 }),
];
test("adds up the measured repositories", () => {
expect(buildStorageOverview(rows).totals).toEqual({
total: 1250,
destinationSize: 550,
releaseAssetsSize: 700,
releaseAssetsCount: 4,
measured: 3,
mirrored: 4,
});
});
test("lists measured repositories largest first", () => {
const overview = buildStorageOverview(rows);
expect(overview.repositories.map((repo) => [repo.id, repo.total])).toEqual([
["a", 1000],
["b", 200],
["c", 50],
]);
expect(overview.repositories[0]).toMatchObject({
fullName: "acme/app",
location: "archive/app",
owner: "archive",
releaseAssetsCount: 4,
checkedAt: checkedAt.toISOString(),
});
// An unmeasured release side stays null so the page can tell it from zero.
expect(overview.repositories[1].releaseAssetsSize).toBeNull();
});
test("groups by destination owner", () => {
expect(buildStorageOverview(rows).owners).toEqual([
{ owner: "archive", destinationSize: 500, releaseAssetsSize: 700, total: 1200, repositories: 2 },
{ owner: "me", destinationSize: 50, releaseAssetsSize: 0, total: 50, repositories: 1 },
]);
});
test("is empty before anything was measured", () => {
const overview = buildStorageOverview([row({ status: "mirrored" })]);
expect(overview.repositories).toEqual([]);
expect(overview.owners).toEqual([]);
expect(overview.totals).toMatchObject({ total: 0, measured: 0, mirrored: 1 });
});
});
+165
View File
@@ -0,0 +1,165 @@
/**
* Storage a mirror takes up on its destination (#479). Measured only from
* responses a sync already fetches: the repository GET before `mirror-sync`
* carries `size`, and the release listing the retention pass reads carries
* every asset's size. Nothing here makes a request.
*/
import { isPushDestinationKind } from "./destination-kinds";
/**
* Columns to record from a destination repository response. Gitea and Forgejo
* report `size` in KiB, Git and LFS together. Empty when the response has no
* usable size, so a missing field never overwrites a stored number.
*/
export function destinationSizeColumns(
info: { size?: unknown } | null | undefined,
now: Date = new Date()
): { destinationSize?: number; storageCheckedAt?: Date } {
const size = info?.size;
if (typeof size !== "number" || !Number.isFinite(size) || size < 0) return {};
return { destinationSize: Math.round(size * 1024), storageCheckedAt: now };
}
export interface ReleaseAssetSummary {
count: number;
bytes: number;
}
/** Count and total size of the assets across a destination release listing. */
export function summarizeReleaseAssets(
releases: ReadonlyArray<{ assets?: ReadonlyArray<{ size?: unknown }> | null }>
): ReleaseAssetSummary {
let count = 0;
let bytes = 0;
for (const release of releases) {
for (const asset of release.assets ?? []) {
count += 1;
if (typeof asset.size === "number" && Number.isFinite(asset.size) && asset.size > 0) {
bytes += asset.size;
}
}
}
return { count, bytes };
}
/** Repository columns the storage overview reads. */
export interface StorageSourceRow {
id: string;
name: string;
fullName: string;
mirroredLocation: string | null;
status: string;
destinationProvider: string;
destinationSize: number | null;
releaseAssetsSize: number | null;
releaseAssetsCount: number | null;
storageCheckedAt: Date | null;
}
export interface StorageRepository {
id: string;
/** Source full name, as the Repositories page shows it. */
fullName: string;
/** owner/name on the destination; empty when not recorded. */
location: string;
/** Destination owner, the first half of `location`. */
owner: string;
destinationSize: number | null;
releaseAssetsSize: number | null;
releaseAssetsCount: number | null;
total: number;
checkedAt: string | null;
}
export interface StorageOwner {
owner: string;
destinationSize: number;
releaseAssetsSize: number;
total: number;
repositories: number;
}
export interface StorageOverview {
totals: {
total: number;
destinationSize: number;
releaseAssetsSize: number;
releaseAssetsCount: number;
/** Repositories with at least one recorded number. */
measured: number;
/** Repositories with a mirror on a Gitea or Forgejo destination. */
mirrored: number;
};
owners: StorageOwner[];
repositories: StorageRepository[];
}
// Statuses that mean a mirror exists on the destination. A failed row may be a
// failed first mirror, so it only counts once it has been measured.
const MIRROR_EXISTS_STATUSES = new Set(["mirrored", "synced", "syncing", "archived", "pending-approval"]);
function isMeasured(row: StorageSourceRow): boolean {
return row.destinationSize !== null || row.releaseAssetsSize !== null;
}
/** The storage page's numbers: totals, per destination owner, and per repository, largest first. */
export function buildStorageOverview(rows: ReadonlyArray<StorageSourceRow>): StorageOverview {
const repositories: StorageRepository[] = [];
const owners = new Map<string, StorageOwner>();
let mirrored = 0;
for (const row of rows) {
if (isPushDestinationKind(row.destinationProvider)) continue;
const measured = isMeasured(row);
if (measured || MIRROR_EXISTS_STATUSES.has(row.status)) mirrored += 1;
if (!measured) continue;
const location = (row.mirroredLocation ?? "").trim();
const owner = location.includes("/") ? location.split("/")[0] : "";
const destinationSize = row.destinationSize ?? 0;
const releaseAssetsSize = row.releaseAssetsSize ?? 0;
const total = destinationSize + releaseAssetsSize;
repositories.push({
id: row.id,
fullName: row.fullName || row.name,
location,
owner,
destinationSize: row.destinationSize,
releaseAssetsSize: row.releaseAssetsSize,
releaseAssetsCount: row.releaseAssetsCount,
total,
checkedAt: row.storageCheckedAt ? row.storageCheckedAt.toISOString() : null,
});
const group = owners.get(owner) ?? {
owner,
destinationSize: 0,
releaseAssetsSize: 0,
total: 0,
repositories: 0,
};
group.destinationSize += destinationSize;
group.releaseAssetsSize += releaseAssetsSize;
group.total += total;
group.repositories += 1;
owners.set(owner, group);
}
repositories.sort((a, b) => b.total - a.total || a.fullName.localeCompare(b.fullName));
const ownerList = [...owners.values()].sort((a, b) => b.total - a.total || a.owner.localeCompare(b.owner));
return {
totals: {
total: repositories.reduce((sum, repo) => sum + repo.total, 0),
destinationSize: repositories.reduce((sum, repo) => sum + (repo.destinationSize ?? 0), 0),
releaseAssetsSize: repositories.reduce((sum, repo) => sum + (repo.releaseAssetsSize ?? 0), 0),
releaseAssetsCount: repositories.reduce((sum, repo) => sum + (repo.releaseAssetsCount ?? 0), 0),
measured: repositories.length,
mirrored,
},
owners: ownerList,
repositories,
};
}
+34 -8
View File
@@ -12,6 +12,32 @@ import {
} from "./sources";
import { encrypt } from "./utils/encryption";
// ---------------------------------------------------------------------------
// These tests need the real module, but gitea-enhanced.test.ts and several
// route tests replace @/lib/sources process-wide (decryptSourceToken becomes
// `token ?? ""`), and bun's mock.module leaks into every file that runs after
// them. Which file runs first depends on the runner, so this file re-runs
// itself in an isolated child process where nothing else has been loaded.
// ---------------------------------------------------------------------------
const CHILD_FLAG = "SOURCES_TEST_ISOLATED";
const isChild = !!process.env[CHILD_FLAG];
if (!isChild) {
test("sources - isolated child suite", () => {
const res = Bun.spawnSync({
cmd: [process.execPath, "test", import.meta.path],
env: { ...process.env, [CHILD_FLAG]: "1" },
stdout: "pipe",
stderr: "pipe",
});
if (res.exitCode !== 0) {
console.error(res.stdout.toString());
console.error(res.stderr.toString());
}
expect(res.exitCode).toBe(0);
}, 60_000);
}
function source(overrides: Partial<SourceRecord> = {}): SourceRecord {
return {
id: "src-1",
@@ -28,7 +54,7 @@ function source(overrides: Partial<SourceRecord> = {}): SourceRecord {
};
}
describe("deriveSourceName", () => {
describe.skipIf(!isChild)("deriveSourceName", () => {
test("appends the username when present", () => {
expect(deriveSourceName("github", "octocat")).toBe("GitHub (octocat)");
expect(deriveSourceName("gitea", "me")).toBe("Gitea / Forgejo (me)");
@@ -39,7 +65,7 @@ describe("deriveSourceName", () => {
});
});
describe("primarySource", () => {
describe.skipIf(!isChild)("primarySource", () => {
test("returns the first entry", () => {
const list = [source({ id: "a" }), source({ id: "b" })];
expect(primarySource(list)?.id).toBe("a");
@@ -50,7 +76,7 @@ describe("primarySource", () => {
});
});
describe("findSourceForRepository", () => {
describe.skipIf(!isChild)("findSourceForRepository", () => {
const github = source();
const gitlab = source({
id: "src-2",
@@ -95,7 +121,7 @@ describe("findSourceForRepository", () => {
});
});
describe("findSourceForOrganization", () => {
describe.skipIf(!isChild)("findSourceForOrganization", () => {
const github = source();
const gitlab = source({
id: "src-2",
@@ -122,7 +148,7 @@ describe("findSourceForOrganization", () => {
});
});
describe("selectSameRunPinsToClear", () => {
describe.skipIf(!isChild)("selectSameRunPinsToClear", () => {
const discovered = [{ normalizedName: "acme" }, { normalizedName: "tools" }];
test("clears a pin an earlier source in the same run created", () => {
@@ -148,7 +174,7 @@ describe("selectSameRunPinsToClear", () => {
});
});
describe("selectRepositoriesToRelink", () => {
describe.skipIf(!isChild)("selectRepositoriesToRelink", () => {
const connected = new Set(["src-1"]);
const base = {
sourceProvider: "github" as const,
@@ -191,7 +217,7 @@ describe("selectRepositoriesToRelink", () => {
});
});
describe("decryptSourceToken", () => {
describe.skipIf(!isChild)("decryptSourceToken", () => {
beforeAll(() => {
// encrypt() needs a secret; do not depend on the environment or on test order.
process.env.ENCRYPTION_SECRET ??= "sources-test-encryption-secret";
@@ -208,7 +234,7 @@ describe("decryptSourceToken", () => {
});
});
describe("resolveGitHubApiBaseUrl", () => {
describe.skipIf(!isChild)("resolveGitHubApiBaseUrl", () => {
test("github.com and empty keep the environment default", () => {
expect(resolveGitHubApiBaseUrl("https://github.com")).toBeUndefined();
expect(resolveGitHubApiBaseUrl("")).toBeUndefined();
+18
View File
@@ -0,0 +1,18 @@
import { describe, expect, test } from "bun:test";
import { formatBytes } from "./format-bytes";
describe("formatBytes", () => {
test("uses binary units like Gitea", () => {
expect(formatBytes(512)).toBe("512 B");
expect(formatBytes(1536)).toBe("1.5 KiB");
expect(formatBytes(5 * 1024 ** 2)).toBe("5.0 MiB");
expect(formatBytes(141 * 1024 ** 3)).toBe("141 GiB");
expect(formatBytes(2.25 * 1024 ** 4)).toBe("2.3 TiB");
});
test("shows nothing as zero", () => {
expect(formatBytes(0)).toBe("0 B");
expect(formatBytes(-5)).toBe("0 B");
expect(formatBytes(Number.NaN)).toBe("0 B");
});
});
+19
View File
@@ -0,0 +1,19 @@
const UNITS = ["B", "KiB", "MiB", "GiB", "TiB", "PiB"] as const;
/**
* A byte count in binary units, the way Gitea and Forgejo show repository
* sizes: one decimal below 100 of a unit, none above.
*/
export function formatBytes(bytes: number): string {
if (!Number.isFinite(bytes) || bytes <= 0) return "0 B";
let value = bytes;
let unit = 0;
while (value >= 1024 && unit < UNITS.length - 1) {
value /= 1024;
unit += 1;
}
if (unit === 0) return `${Math.round(value)} B`;
return `${value >= 100 ? Math.round(value) : value.toFixed(1)} ${UNITS[unit]}`;
}
+146
View File
@@ -0,0 +1,146 @@
/**
* Route tests for GET /api/storage (#479).
*
* NOTE: these tests replace @/lib/db and @/lib/utils/auth-helpers with mocks.
* bun's mock.module is process-wide and leaks into other test files, so in
* the shared test process this file registers NOTHING: it re-runs itself in
* an isolated child process (the same harness as repositories/[id].test.ts).
*/
import { describe, test, expect, mock, beforeEach } from "bun:test";
const CHILD_FLAG = "GM_STORAGE_ROUTE_ISOLATED";
const isChild = !!process.env[CHILD_FLAG];
if (!isChild) {
test("storage route - isolated child suite", () => {
const res = Bun.spawnSync({
cmd: [process.execPath, "test", import.meta.path],
env: { ...process.env, [CHILD_FLAG]: "1" },
stdout: "pipe",
stderr: "pipe",
});
if (res.exitCode !== 0) {
console.error(res.stdout.toString());
console.error(res.stderr.toString());
}
expect(res.exitCode).toBe(0);
}, 60_000);
}
let repoRows: any[] = [];
let authResult: { user: { id: string } | null; response: Response | null } = {
user: { id: "user-1" },
response: null,
};
/** Leaf values of each select's where() condition (drizzle conditions are cyclic). */
let selectWhereValues: unknown[][] = [];
function conditionLeafValues(node: any, depth = 0): unknown[] {
if (node == null || depth > 8) return [];
if (typeof node !== "object") return [node];
const leaves: unknown[] = [];
if (Array.isArray(node.value)) leaves.push(...node.value);
for (const chunk of node.queryChunks ?? []) {
leaves.push(...conditionLeafValues(chunk, depth + 1));
}
return leaves;
}
if (isChild) {
mock.module("@/lib/utils/auth-helpers", () => ({
requireAuth: mock(async () => authResult),
}));
mock.module("@/lib/db", () => ({
db: {
select: () => ({
from: () => ({
where: async (cond: any) => {
selectWhereValues.push(conditionLeafValues(cond));
return repoRows;
},
}),
}),
},
organizations: {},
repositories: {},
configs: {},
users: {},
events: {},
mirrorJobs: {},
sessions: {},
accounts: {},
verificationTokens: {},
verifications: {},
oauthClients: {},
oauthAccessTokens: {},
oauthRefreshTokens: {},
oauthConsents: {},
jwkss: {},
ssoProviders: {},
apikeys: {},
rateLimits: {},
sources: {},
}));
}
const { GET } = isChild ? await import("./index") : { GET: undefined as any };
function get() {
return GET({
params: {},
request: new Request("http://localhost/api/storage"),
locals: { session: { userId: "user-1" } },
} as any);
}
beforeEach(() => {
selectWhereValues = [];
authResult = { user: { id: "user-1" }, response: null };
repoRows = [
{
id: "repo-1",
name: "app",
fullName: "acme/app",
mirroredLocation: "archive/app",
status: "synced",
destinationProvider: "gitea",
destinationSize: 4096,
releaseAssetsSize: 1024,
releaseAssetsCount: 2,
storageCheckedAt: new Date("2026-10-29T08:00:00Z"),
},
];
});
describe.skipIf(!isChild)("GET /api/storage", () => {
test("returns the overview of this user's repositories", async () => {
const response = await get();
expect(response.status).toBe(200);
const data = await response.json();
expect(data.success).toBe(true);
expect(data.totals).toMatchObject({ total: 5120, measured: 1, mirrored: 1 });
expect(data.owners).toEqual([
{ owner: "archive", destinationSize: 4096, releaseAssetsSize: 1024, total: 5120, repositories: 1 },
]);
expect(data.repositories[0]).toMatchObject({
id: "repo-1",
location: "archive/app",
checkedAt: "2026-10-29T08:00:00.000Z",
});
expect(selectWhereValues[0]).toEqual(expect.arrayContaining(["user-1"]));
});
test("answers the auth helper's response when signed out", async () => {
authResult = {
user: null,
response: new Response(JSON.stringify({ error: "Unauthorized" }), { status: 401 }),
};
const response = await get();
expect(response.status).toBe(401);
expect(selectWhereValues).toEqual([]);
});
});
+41
View File
@@ -0,0 +1,41 @@
import type { APIRoute } from "astro";
import { eq } from "drizzle-orm";
import { db, repositories } from "@/lib/db";
import { createSecureErrorResponse } from "@/lib/utils";
import { requireAuth } from "@/lib/utils/auth-helpers";
import { buildStorageOverview } from "@/lib/repository-storage";
/**
* Storage the user's mirrors take up on the destination (#479): totals, per
* destination owner and per repository. Reads only what syncs recorded, so it
* never contacts the destination or the source.
*/
export const GET: APIRoute = async (context) => {
try {
const { user, response } = await requireAuth(context);
if (response) return response;
const rows = await db
.select({
id: repositories.id,
name: repositories.name,
fullName: repositories.fullName,
mirroredLocation: repositories.mirroredLocation,
status: repositories.status,
destinationProvider: repositories.destinationProvider,
destinationSize: repositories.destinationSize,
releaseAssetsSize: repositories.releaseAssetsSize,
releaseAssetsCount: repositories.releaseAssetsCount,
storageCheckedAt: repositories.storageCheckedAt,
})
.from(repositories)
.where(eq(repositories.userId, user!.id));
return new Response(JSON.stringify({ success: true, ...buildStorageOverview(rows) }), {
status: 200,
headers: { "Content-Type": "application/json" },
});
} catch (error) {
return createSecureErrorResponse(error, "storage overview", 500);
}
};
+23
View File
@@ -0,0 +1,23 @@
---
import '../styles/global.css';
import App from '@/components/layout/MainLayout';
import ThemeScript from '@/components/theme/ThemeScript.astro';
import { withBase } from '@/lib/base-path';
---
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width" />
<link rel="icon" type="image/svg+xml" href={withBase('/favicon.svg')} />
<meta name="generator" content={Astro.generator} />
<title>Storage - Gitea Mirror</title>
<ThemeScript />
</head>
<body>
<App page="storage" client:load />
</body>
</html>
+1
View File
@@ -4,6 +4,7 @@ export type Paths =
| "/"
| "/repositories"
| "/organizations"
| "/storage"
| "/config"
| "/activity";
+15
View File
@@ -160,6 +160,21 @@ When several sources are connected, an optional `sourceId` picks which one the o
`PATCH /api/organizations/:id` with `{ "sourceId": "…" }` re-pins an existing organization to another source, and `{ "sourceId": null }` clears the pin so the organization follows every source's repositories of the same name again. A pin whose source was removed falls back the same way.
### Storage
`GET /api/storage` returns what the Storage page shows: how much space the account's mirrors take on the destination (issue #479).
```json
{
"success": true,
"totals": { "total": 3554304, "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "measured": 1, "mirrored": 1 },
"owners": [{ "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "total": 3554304, "repositories": 1 }],
"repositories": [{ "id": "…", "fullName": "upstream/app", "location": "archive/app", "owner": "archive", "destinationSize": 3030016, "releaseAssetsSize": 524288, "releaseAssetsCount": 1, "total": 3554304, "checkedAt": "2026-10-29T03:32:01.000Z" }]
}
```
Sizes are bytes. `destinationSize` is Git and LFS together as Gitea or Forgejo reports it, and the release fields cover the assets on the destination's releases. Each sync records them from responses it already fetches, so this endpoint never contacts the destination. A repository is listed once a sync has measured it, and a field not measured yet is `null`. `mirrored` counts the repositories with a mirror on a Gitea or Forgejo destination and `measured` the ones with a size so far; push destinations (GitHub, GitLab) are not measured.
### Run cleanup
`POST /api/cleanup/trigger` runs the [repository cleanup](./cleanup-and-reconcile.md) now, with the account's saved cleanup settings, and returns how many orphaned repositories were found and processed and any errors. `GET /api/cleanup/trigger` returns the cleanup configuration and the last and next run times.