Compare commits

...
Author SHA1 Message Date
Arunavo Ray 41799154fc chore: bump version to 3.40.4 2026-10-14 17:41:32 +05:30
ARUNAVO RAY 8de3c0c091 fix: assign mirrored issues and pull requests to their milestones (#460) (#463)
* fix: assign mirrored issues and pull requests to their milestones (#460)

Milestones were created in Gitea but issues never pointed at them: the
issue and PR payloads had no milestone field, and milestones were
mirrored after issues anyway.

Milestones now sync first and return a title to Gitea id map, which the
issues and pull requests passes use to set the milestone on create and
to set, change or clear it on update. The edit only carries a milestone
when it differs, so Gitea writes no repeat timeline entries. Unchanged
items are fixed on the weekly full pass from the listing alone. When
milestones are not mirrored, or one is missing in Gitea, the issue is
left as it is.

* fix: review follow-ups for milestone assignment (#460)

- Never clear a milestone made in Gitea alone. The milestones pass now
  returns only the milestones that mirror a GitHub one, and an issue is
  only cleared when it sits on one of those.
- Match existing pull request issues by a [PR #N] prefix, so a mirrored
  issue that mentions [PR #N] in its title is not taken for that PR.
- The milestone edit on an unchanged pull request is best effort: a
  failure is logged instead of retried, and the pass still completes.
- Record a milestone edit on the cached Gitea issue so a retry does not
  send it again.
- Store 0 instead of undefined when a created milestone has no id.

* fix: keep mirrored milestones in line with GitHub (#460 follow-up)

Milestones were only ever created, so closing, reopening, moving the due
date or editing the description on GitHub never reached Gitea. The
milestones pass now edits an already mirrored milestone when its state,
due date or description differs.

Due dates compare as calendar days. Gitea moves an edited deadline to
the end of that day in its own time zone, so comparing instants would
edit the milestone on every sync. A removed due date is cleared with
Gitea's year 9999 value, since its edit endpoint ignores an empty one.
2026-10-14 17:41:10 +05:30
Arunavo Ray 89babdf461 test: isolate the reset-password CLI test and use the real schema (#461 follow-up)
The test failed in CI because another file replaces bun:sqlite with
mock.module, which leaks across files depending on order. It now runs in
an isolated child process like the other fetch and sqlite driven suites.

Each case builds its database with manage-db init, so the tables come
from drizzle/ instead of hand written DDL that could drift, and the
bundle is built once in beforeAll instead of at import time.

The Dockerfile rewrite of the manage-db scripts is now a one line sed.
2026-10-14 17:38:43 +05:30
Ayrat Belyaev 44d927e768 fix: restore password reset CLI in source and Docker installs (#461) 2026-10-14 17:37:35 +05:30
CrumblyLiquid 0f55d29298 fix: NixOS module warning (#459) 2026-10-14 17:35:37 +05:30
Arunavo Ray ba50792b0c chore: bump version to 3.40.3 2026-09-26 09:10:19 +05:30
ARUNAVO RAY 24fd26656e fix: stale syncing count in the Activity Log after a quick sync (#454) (#455)
Mirror job timestamps are stored to the second. A sync that starts and
finishes in the same second writes its "syncing" and "synced" events
with the same timestamp, and SQLite returned the tie oldest first, so
the summary took the start event as the repository's current state.

The activities and dashboard queries break ties on rowid, newest first,
and the summary counter prefers a finish over a start from the same
second whatever order they arrive in.
2026-09-26 09:10:08 +05:30
ARUNAVO RAY 34abda94fb fix(www): remove the stale bun.lock that fails the Cloudflare Pages build (#456)
The site moved to pnpm in July 2025 and pnpm-lock.yaml is the lockfile
kept up to date, but www/bun.lock was left behind and still pins Astro 5.
The Cloudflare Pages build image now installs with bun when it finds a
bun.lock, and bun install --frozen-lockfile stops on the stale file, so
every docs build has failed since.

Remove it, ignore it in www so a local bun install cannot bring it back,
and point the site README and CLAUDE.md at pnpm.
2026-09-26 09:10:03 +05:30
Arunavo Ray 1c1ff0b2fa chore: bump version to 3.40.2 2026-09-25 08:05:56 +05:30
ARUNAVO RAY e7e07d08f8 fix: resume issue and PR passes after a rate limit, cheap full passes (#449) (#453)
* fix: resume issue and PR passes after a rate limit, cheap full passes (#449)

A pass stopped by the rate limit now saves the numbers it finished with
the repository, and the next run repeats the same pass and skips them
unless they changed since it started. Before, every run started over, so
a repository too large for one hour of budget never completed a pass.

The weekly full pass no longer fetches comments, PR details, commits or
files for items already in the destination and unchanged since the last
completed pass. It corrects title, body, state and labels from the
listing alone.

A sync paused by the rate limit logs one line instead of the error
object with every partial result.

* chore: retrigger docs preview
2026-09-25 08:04:32 +05:30
Arunavo Ray fdb394942b chore: bump version to 3.40.1 2026-09-24 00:11:40 +05:30
ARUNAVO RAY 1d0afe2d2e fix: hold every GitHub request while the source is rate limited (#437) (#452)
The v3.37.0 change stopped the scheduler run from holding its lock
through a rate limit, but only the refused request stopped. Every other
call queued in the run still went to GitHub and came back 403, and the
retry helper sent each of them three more times. GitHub counts requests
made while limited toward abuse detection, and a reporter's account was
suspended that way.

Every GitHub client now carries a request wrap that reads the gate
before the network: a reset within two minutes is waited out inside the
request, a longer one fails at once with a RateLimitPausedError, and
only the rate limit probe endpoint is exempt. The retry helper does not
retry a rate limit refusal and stops starting the remaining items,
surfacing a RateLimitedBatchError. The metadata component catches in
both mirror paths rethrow rate limit errors so the repository fails and
the scheduler restores it, and the pull request detail fallback no
longer writes a stripped down issue after a refusal. The secondary rate
limit handler closes the gate for its retry-after window.
2026-09-24 00:11:30 +05:30
Arunavo Ray aaef5d6e5c chore: bump version to 3.40.0 2026-09-23 23:35:06 +05:30
ARUNAVO RAY 604ff3b6b3 feat: list view for the Organizations page (#428) (#451)
A cards/list switch in the toolbar. The list shows one row per
organization with its role, destination, repository counts, last
mirrored time, status, the same actions as the cards and the source
and destination links, so a few dozen organizations fit on one screen.
The choice is stored in localStorage next to the theme and time format
preferences.

The actions dropdown that existed twice in the card is now one shared
component used by the cards and the list, and the status badge, source
resolution and repository breakdown helpers moved to a shared module.
2026-09-23 23:35:01 +05:30
Arunavo Ray 935bc89e52 chore: bump version to 3.39.0 2026-09-23 23:07:34 +05:30
ARUNAVO RAY 133268e63a feat: incremental issue and pull request sync (#449) (#450)
After a complete pass, the issues and pull request passes list only the
items GitHub reports as updated since the previous listing (minus a 10
minute margin), so per-issue comment listing and per-PR detail, commits
and files calls run only for what changed. Pull requests use the issues
endpoint for this since pulls.list has no since filter.

The watermark lives in the repository metadata state as syncCursors, one
per kind, and records when the listing started. It only moves when the
pass finishes with no failed item. A full pass still runs when the last
one is older than 7 days, when the destination has no mirrored issues or
pull requests, and after reset metadata, which clears the state.
2026-09-23 23:07:23 +05:30
44 changed files with 4369 additions and 1463 deletions

No files matched your search

+20
View File
@@ -7,6 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Fixed
- The Activity Log summary no longer shows a repository as syncing after a quick sync finished (#454). Events are stored to the second, and a sync that starts and finishes in the same second could come back in the wrong order, so the start event was counted as the current state. Activities now come back newest first within the same second too, and the summary counts a finish over a start from the same second
- Issue and pull request passes over large repositories finish across rate limit resets (#449 follow-up)
- A pass stopped by the rate limit keeps the numbers it finished, and the next run repeats the same pass and skips them unless they changed since it started; before, every run started over from the first item, so a repository too large for one hour of budget never completed a pass
- The weekly full pass no longer fetches comments for issues, or the detail, commits and files of pull requests, that are already in the destination and have not changed since the last completed pass; it lists everything and corrects title, body, state and labels from the listing
- A sync paused by the rate limit logs one line instead of the error object with every partial result, and its activity entry says it was paused
- No request leaves the process while GitHub is rate limited (#437, second report)
- The v3.37.0 change stopped the scheduler run from holding its lock through a rate limit, but every other request already queued in the run still went to GitHub and came back 403, and the retry helper sent each of them three more times; GitHub counts requests made while limited toward abuse detection and one account was suspended that way
- Every GitHub client now holds its requests while the source is paused: a limit that resets within two minutes is waited out inside the request, a longer one fails the request at once without contacting GitHub, and the rate limit probe endpoint stays available so the pause can end early
- The retry helper no longer retries a rate limit refusal and stops starting the remaining items of the batch
- A rate limit refusal inside the issues, pull requests, releases, labels or milestones pass now fails the repository (which the scheduler already puts back to its previous status and retries after the reset) instead of moving on to the next component, and a pull request whose detail call was refused is no longer written as a stripped down issue
- A secondary rate limit pauses the whole client for its retry-after window, not only the request that was told to slow down
### Security
- Source URLs go through the outbound guard too (GHSA-p7w3-46pg-mv6h): `POST /api/sources`, `PUT /api/sources/:id` and the configuration save refuse link local and metadata addresses for source and destination URLs, the source fetch helper pins and never follows redirects like the other user supplied URLs, source errors no longer carry the upstream body in their message, and errors raised by a request to another host are never forwarded to API clients whatever words they contain
- Fixed four privately reported vulnerabilities (GHSA-9m33-xfrc-5jxw, GHSA-6m23-28hh-gjh2, GHSA-2hpx-83vg-gm45, GHSA-5pp8-r7f5-6q8p)
@@ -19,6 +32,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Raised the dependency floors for the advisories published on 2026-09-08: Astro 7.2.8 (remote code execution through AVIF image optimization, and an authorization bypass when stripping the configured base), @xmldom/xmldom 0.8.15 (eight parser and serializer issues), sharp 0.35.4 (libheif), svgo 4.1.0 (removeScripts sanitization) and js-yaml 4.3.2 (merge-key CPU use). Applied to both the application and the documentation site.
### Added
- List view for the Organizations page (#428)
- A cards/list switch in the toolbar; the list shows one row per organization with its role, destination, repository counts, last mirrored time, status, the same actions as the cards and the source and destination links, so a few dozen organizations fit on one screen
- The choice is remembered per browser, next to the theme and time format preferences
- Incremental issue and pull request sync (#449)
- After a complete pass, the issues and pull request passes ask GitHub only for items updated since the last one (with a 10 minute margin), so comments, pull request details, commits and files are fetched only for what changed instead of for every item on every sync
- The watermark is stored per repository in the metadata state and only moves when a pass finishes with no failed item
- A full pass still runs when the last one is older than 7 days, when the destination has no mirrored issues or pull requests, and after Reset metadata
- Releases mirror from Gitea and Forgejo sources, Codeberg included (#440)
- The release mirror lists releases through the repository's own source instead of always through the GitHub API, and keeps the same release limit, asset limit, per-destination lock, tag check and retention pruning
- Release assets are downloaded with the credentials of the host they live on, so a Gitea source uses its own token and a public one needs none
+3
View File
@@ -76,6 +76,9 @@ COPY --from=builder /app/docker-entrypoint.sh ./docker-entrypoint.sh
COPY --from=builder /app/scripts/runtime-server.ts ./scripts/runtime-server.ts
COPY --from=builder /app/drizzle ./drizzle
# Database commands must use the bundled script: sources are not shipped.
RUN sed -i 's#bun scripts/manage-db.ts#bun dist/scripts/manage-db.js#g' package.json
# Remove build-only packages that are not needed at runtime
# (esbuild, vite, rollup, tailwind, svgo — all only used during `astro build`)
RUN rm -rf node_modules/esbuild node_modules/@esbuild \
+1 -1
View File
@@ -303,7 +303,7 @@ EOF
package = mkOption {
type = types.package;
default = self.packages.${pkgs.system}.default;
default = self.packages.${pkgs.stdenv.hostPlatform.system}.default;
description = "The Gitea Mirror package to use";
};
+2 -2
View File
@@ -2,8 +2,8 @@ apiVersion: v2
name: gitea-mirror
description: Kubernetes helm chart for gitea-mirror
type: application
version: 3.38.2
appVersion: "3.38.2"
version: 3.40.4
appVersion: "3.40.4"
icon: https://github.com/RayLabsHQ/gitea-mirror/blob/main/.github/assets/logo.png
keywords:
- git
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "gitea-mirror",
"type": "module",
"version": "3.38.2",
"version": "3.40.4",
"engines": {
"bun": ">=1.2.9"
},
+1 -1
View File
@@ -221,7 +221,7 @@ async function resetPassword() {
}
const sqlite = new Database(dbPath);
const db = drizzle({ client: sqlite });
const db = drizzle({ client: sqlite, schema: { users, accounts, sessions } });
try {
const user = await db.query.users.findFirst({
+4 -22
View File
@@ -1,3 +1,4 @@
import { countLatestStatuses } from '@/lib/utils/activity-status';
import { useCallback, useEffect, useState, useRef, useMemo } from 'react';
import Fuse from 'fuse.js';
import { Button } from '@/components/ui/button';
@@ -407,28 +408,9 @@ export function ActivityLog() {
// Check if any filters are active
// Counts the current state of each repository or organization, not every
// event: a repo that failed and then synced is only "synced". Activities
// arrive newest first, so the first entry seen for a subject is its state.
// Taken from the unfiltered set so the chips stay put when one is clicked.
const statusCounts = useMemo(() => {
const latestBySubject = new Map<string, string>();
activities.forEach((a) => {
const subject =
a.repositoryId ||
a.organizationId ||
a.repositoryName ||
a.organizationName;
// Events with no subject have no state of their own to report.
if (!subject || latestBySubject.has(subject)) return;
latestBySubject.set(subject, a.status);
});
const counts = new Map<string, number>();
latestBySubject.forEach((status) => {
counts.set(status, (counts.get(status) ?? 0) + 1);
});
return counts;
}, [activities]);
// event: a repo that failed and then synced is only "synced". Taken from
// the unfiltered set so the chips stay put when one is clicked.
const statusCounts = useMemo(() => countLatestStatuses(activities), [activities]);
// Most actionable first, and only statuses that actually occurred.
const statChips = ['failed', 'syncing', 'mirroring', 'imported', 'synced', 'mirrored']
@@ -44,6 +44,11 @@ interface MirrorDestinationEditorProps {
destinationLabel?: string;
isUpdating?: boolean;
className?: string;
/**
* Show only the destination and the edit button. The list view (#428)
* already names the organization in its first column.
*/
compact?: boolean;
}
const MAX_LISTED = 8;
@@ -88,6 +93,7 @@ export function MirrorDestinationEditor({
destinationLabel = "Gitea",
isUpdating = false,
className,
compact = false,
}: MirrorDestinationEditorProps) {
const [isOpen, setIsOpen] = useState(false);
const [editValue, setEditValue] = useState(currentDestination || "");
@@ -185,14 +191,25 @@ export function MirrorDestinationEditor({
return (
<div className={cn("flex items-center gap-2 w-full", className)}>
<div className="flex items-center gap-1.5 text-xs text-muted-foreground min-w-0 flex-1">
<Building2 className="h-3 w-3 flex-shrink-0" />
<span className="font-medium truncate">{organizationName}</span>
<ArrowRight className="h-3 w-3 flex-shrink-0" />
<span className={cn(
"font-medium truncate",
hasOverride && "text-orange-600 dark:text-orange-400"
)}>
<div className={cn(
"flex items-center gap-1.5 text-muted-foreground min-w-0 flex-1",
compact ? "text-sm" : "text-xs"
)}>
{!compact && (
<>
<Building2 className="h-3 w-3 flex-shrink-0" />
<span className="font-medium truncate">{organizationName}</span>
<ArrowRight className="h-3 w-3 flex-shrink-0" />
</>
)}
<span
className={cn(
"font-medium truncate",
compact && "text-foreground",
hasOverride && "text-orange-600 dark:text-orange-400"
)}
title={effectiveDestination}
>
{effectiveDestination}
</span>
{hasOverride && (
+81 -1
View File
@@ -1,6 +1,6 @@
import { useCallback, useEffect, useState } from "react";
import { Button } from "@/components/ui/button";
import { Search, RefreshCw, FlipHorizontal, Filter, LoaderCircle, Trash2, Download } from "lucide-react";
import { Search, RefreshCw, FlipHorizontal, Filter, LoaderCircle, Trash2, Download, LayoutGrid, List } from "lucide-react";
import type { MirrorJob, Organization } from "@/lib/db/schema";
import { OrganizationList } from "./OrganizationsList";
import AddOrganizationDialog from "./AddOrganizationDialog";
@@ -30,6 +30,9 @@ import { invalidateConfigCache, useConfigStatus } from "@/hooks/useConfigStatus"
import { useNavigation } from "@/components/layout/MainLayout";
import { useLiveRefresh } from "@/hooks/useLiveRefresh";
import { withBase } from "@/lib/base-path";
import { useOrganizationsView } from "@/hooks/useOrganizationsView";
import type { OrganizationsViewMode } from "@/lib/utils/organizations-view";
import { cn } from "@/lib/utils";
import {
Drawer,
DrawerClose,
@@ -53,6 +56,8 @@ export function Organization() {
const [organizations, setOrganizations] = useState<Organization[]>([]);
const [isLoading, setIsLoading] = useState<boolean>(true);
const [isDialogOpen, setIsDialogOpen] = useState<boolean>(false);
// Cards or the compact list (#428); remembered per browser.
const { view, setView } = useOrganizationsView();
const { user } = useAuth();
const { sourceProvider, sourceUrl, sources } = useConfigStatus();
const { navigationKey } = useNavigation();
@@ -778,6 +783,8 @@ export function Organization() {
</DrawerContent>
</Drawer>
<OrganizationsViewToggle view={view} onChange={setView} compact />
<Button
variant="outline"
size="icon"
@@ -938,6 +945,8 @@ export function Organization() {
{/* Action buttons */}
<div className="flex items-center gap-2 ml-auto">
<OrganizationsViewToggle view={view} onChange={setView} />
<Button
variant="outline"
size="icon"
@@ -991,6 +1000,7 @@ export function Organization() {
sourceProvider={sourceProvider}
sourceUrl={sourceUrl}
sources={sources}
view={view}
/>
<AddOrganizationDialog
@@ -1066,3 +1076,73 @@ export function Organization() {
</div>
);
}
/**
* Cards or list switch for the Organizations page (#428). A segmented
* pair of icon buttons so it sits next to the other toolbar icons. The
* compact form is one button that flips to the other view, for the
* narrow toolbar where every icon costs search box width.
*/
function OrganizationsViewToggle({
view,
onChange,
compact = false,
}: {
view: OrganizationsViewMode;
onChange: (next: OrganizationsViewMode) => void;
compact?: boolean;
}) {
const options: { value: OrganizationsViewMode; label: string; Icon: typeof LayoutGrid }[] = [
{ value: "cards", label: "Cards", Icon: LayoutGrid },
{ value: "list", label: "List", Icon: List },
];
if (compact) {
const next = options.find((option) => option.value !== view) ?? options[0];
return (
<Button
type="button"
variant="outline"
size="icon"
onClick={() => onChange(next.value)}
title={`Switch to ${next.label.toLowerCase()} view`}
aria-label={`Switch to ${next.label.toLowerCase()} view`}
className="h-10 w-10 shrink-0"
data-testid="organizations-view-toggle"
>
<next.Icon className="h-4 w-4" />
</Button>
);
}
return (
<div
role="radiogroup"
aria-label="Organizations view"
className="flex items-center h-10 rounded-md border border-input bg-background shrink-0"
data-testid="organizations-view-toggle"
>
{options.map(({ value, label, Icon }, index) => (
<Button
key={value}
type="button"
variant="ghost"
size="icon"
role="radio"
aria-checked={view === value}
aria-label={`${label} view`}
title={`${label} view`}
data-state={view === value ? "on" : "off"}
onClick={() => onChange(value)}
className={cn(
"h-full w-10 rounded-none",
index === 0 ? "rounded-l-md" : "rounded-r-md border-l border-input",
view === value
? "bg-muted text-foreground hover:bg-muted"
: "text-muted-foreground hover:text-foreground"
)}
>
<Icon className="h-4 w-4" />
</Button>
))}
</div>
);
}
@@ -0,0 +1,90 @@
import { Button } from "@/components/ui/button";
import {
DropdownMenu,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuSeparator,
DropdownMenuTrigger,
} from "@/components/ui/dropdown-menu";
import { Ban, MoreVertical, RefreshCw, SlidersHorizontal, Trash2 } from "lucide-react";
import type { Organization } from "@/lib/db/schema";
import { cn } from "@/lib/utils";
export interface OrganizationActionsMenuProps {
org: Organization;
disabled?: boolean;
onSync?: ({ orgId }: { orgId: string }) => Promise<void>;
onIgnore?: ({ orgId, ignore }: { orgId: string; ignore: boolean }) => Promise<void>;
onDelete?: (orgId: string) => void;
onEditMirrorOptions: (org: Organization) => void;
triggerClassName?: string;
}
/**
* The secondary actions of an organization, shared by the cards and the
* list (#428). Hidden while the organization is being mirrored, like the
* primary action.
*/
export function OrganizationActionsMenu({
org,
disabled = false,
onSync,
onIgnore,
onDelete,
onEditMirrorOptions,
triggerClassName,
}: OrganizationActionsMenuProps) {
if (org.status === "mirroring") return null;
return (
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button
variant="ghost"
size="icon"
disabled={disabled}
className={cn(triggerClassName)}
title="More actions"
aria-label={`More actions for ${org.name}`}
>
<MoreVertical className="h-4 w-4" />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end">
{onSync && (org.status === "mirrored" || org.status === "failed") && (
<>
<DropdownMenuItem onClick={() => org.id && onSync({ orgId: org.id })}>
<RefreshCw className="h-4 w-4 mr-2" />
Sync Organization
</DropdownMenuItem>
<DropdownMenuSeparator />
</>
)}
<DropdownMenuItem onClick={() => onEditMirrorOptions(org)}>
<SlidersHorizontal className="h-4 w-4 mr-2" />
Mirror Options
</DropdownMenuItem>
{org.status !== "ignored" && (
<DropdownMenuItem
onClick={() => org.id && onIgnore && onIgnore({ orgId: org.id, ignore: true })}
>
<Ban className="h-4 w-4 mr-2" />
Ignore Organization
</DropdownMenuItem>
)}
{onDelete && (
<>
{org.status !== "ignored" && <DropdownMenuSeparator />}
<DropdownMenuItem
className="text-destructive focus:text-destructive"
onClick={() => org.id && onDelete(org.id)}
>
<Trash2 className="h-4 w-4 mr-2" />
Delete from Mirror
</DropdownMenuItem>
</>
)}
</DropdownMenuContent>
</DropdownMenu>
);
}
@@ -0,0 +1,88 @@
import { useState } from "react";
import { toast } from "sonner";
import { Layers } from "lucide-react";
import type { SourceApiRecord } from "@/types/config";
import { SOURCE_PROVIDER_ICONS } from "@/lib/source-providers/icons";
import { cn } from "@/lib/utils";
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select";
// Radix reserves the empty string as a Select value, so unpinned
// organizations use this sentinel instead.
const EVERY_SOURCE_VALUE = "every-source";
export interface OrganizationSourceEditorProps {
sources: SourceApiRecord[];
value?: string | null;
disabled?: boolean;
onUpdateSource: (sourceId: string | null) => Promise<void>;
/** Drop the "Source" label; the list view puts it in the column header. */
compact?: boolean;
className?: string;
}
/**
* Pin an organization to one connected source, or let it follow every
* source. Shown by the cards and the list only when more than one source
* is connected.
*/
export function OrganizationSourceEditor({
sources,
value,
disabled = false,
onUpdateSource,
compact = false,
className,
}: OrganizationSourceEditorProps) {
const [isUpdating, setIsUpdating] = useState(false);
const handleValueChange = async (next: string) => {
setIsUpdating(true);
try {
await onUpdateSource(next === EVERY_SOURCE_VALUE ? null : next);
} catch (error) {
toast.error(error instanceof Error ? error.message : "Failed to update source");
} finally {
setIsUpdating(false);
}
};
return (
<div className={cn("flex items-center gap-2 w-full", className)}>
{!compact && (
<span className="flex items-center gap-1.5 text-xs text-muted-foreground shrink-0">
<Layers className="h-3 w-3" />
Source
</span>
)}
<Select
value={value ?? EVERY_SOURCE_VALUE}
onValueChange={(next) => void handleValueChange(next)}
disabled={disabled || isUpdating}
>
<SelectTrigger className="h-8 min-w-0 flex-1 text-xs" aria-label="Source">
<SelectValue placeholder="Source" />
</SelectTrigger>
<SelectContent>
<SelectItem value={EVERY_SOURCE_VALUE}>Every source</SelectItem>
{sources.map((source) => {
const SourceIcon = SOURCE_PROVIDER_ICONS[source.provider];
return (
<SelectItem key={source.id} value={source.id}>
<span className="flex items-center gap-2">
<SourceIcon className="h-3.5 w-3.5" />
{source.name}
</span>
</SelectItem>
);
})}
</SelectContent>
</Select>
</div>
);
}
@@ -2,8 +2,7 @@ import { useMemo, useState } from "react";
import { Card } from "@/components/ui/card";
import { Button } from "@/components/ui/button";
import { Badge } from "@/components/ui/badge";
import { Plus, RefreshCw, Building2, Check, AlertCircle, Clock, MoreVertical, Ban, SlidersHorizontal, Trash2, Layers } from "lucide-react";
import { toast } from "sonner";
import { Plus, RefreshCw, Building2, Check, AlertCircle, Clock, Ban, Layers } from "lucide-react";
import type { MirrorOverrides, Organization } from "@/lib/db/schema";
import type { FilterParams } from "@/types/filter";
import type { SourceApiRecord } from "@/types/config";
@@ -26,21 +25,11 @@ import {
type SourceProviderKind,
} from "@/lib/source-providers/kinds";
import { SOURCE_PROVIDER_ICONS } from "@/lib/source-providers/icons";
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select";
import {
DropdownMenu,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuLabel,
DropdownMenuSeparator,
DropdownMenuTrigger,
} from "@/components/ui/dropdown-menu";
import type { OrganizationsViewMode } from "@/lib/utils/organizations-view";
import { OrganizationSourceEditor } from "./OrganizationSourceEditor";
import { OrganizationActionsMenu } from "./OrganizationActionsMenu";
import { OrganizationsTable } from "./OrganizationsTable";
import { getOrganizationStatusBadge, resolveOrganizationSource } from "./organization-presentation";
interface OrganizationListProps {
organizations: Organization[];
@@ -64,87 +53,18 @@ interface OrganizationListProps {
sourceUrl?: string;
/** Connected sources; with more than one, each card shows and edits its own source. */
sources?: SourceApiRecord[];
/** Cards (default) or the compact list (#428). */
view?: OrganizationsViewMode;
}
const STATUS_ICONS = { check: Check, alert: AlertCircle, clock: Clock, ban: Ban } as const;
// Helper function to get status badge variant and icon
const getStatusBadge = (status: string | null) => {
switch (status) {
case "imported":
return { variant: "secondary" as const, label: "Not Mirrored", icon: null };
case "mirroring":
return { variant: "outline" as const, label: "Mirroring", icon: Clock };
case "mirrored":
return { variant: "default" as const, label: "Mirrored", icon: Check };
case "failed":
return { variant: "destructive" as const, label: "Failed", icon: AlertCircle };
case "ignored":
return { variant: "outline" as const, label: "Ignored", icon: Ban };
default:
return { variant: "secondary" as const, label: "Unknown", icon: null };
}
const badge = getOrganizationStatusBadge(status);
return { ...badge, icon: badge.icon ? STATUS_ICONS[badge.icon] : null };
};
// Radix reserves the empty string as a Select value, so unpinned
// organizations use this sentinel instead.
const EVERY_SOURCE_VALUE = "every-source";
function OrganizationSourceEditor({
sources,
value,
disabled = false,
onUpdateSource,
}: {
sources: SourceApiRecord[];
value?: string | null;
disabled?: boolean;
onUpdateSource: (sourceId: string | null) => Promise<void>;
}) {
const [isUpdating, setIsUpdating] = useState(false);
const handleValueChange = async (next: string) => {
setIsUpdating(true);
try {
await onUpdateSource(next === EVERY_SOURCE_VALUE ? null : next);
} catch (error) {
toast.error(error instanceof Error ? error.message : "Failed to update source");
} finally {
setIsUpdating(false);
}
};
return (
<div className="flex items-center gap-2 w-full">
<span className="flex items-center gap-1.5 text-xs text-muted-foreground shrink-0">
<Layers className="h-3 w-3" />
Source
</span>
<Select
value={value ?? EVERY_SOURCE_VALUE}
onValueChange={(next) => void handleValueChange(next)}
disabled={disabled || isUpdating}
>
<SelectTrigger className="h-8 min-w-0 flex-1 text-xs">
<SelectValue placeholder="Source" />
</SelectTrigger>
<SelectContent>
<SelectItem value={EVERY_SOURCE_VALUE}>Every source</SelectItem>
{sources.map((source) => {
const SourceIcon = SOURCE_PROVIDER_ICONS[source.provider];
return (
<SelectItem key={source.id} value={source.id}>
<span className="flex items-center gap-2">
<SourceIcon className="h-3.5 w-3.5" />
{source.name}
</span>
</SelectItem>
);
})}
</SelectContent>
</Select>
</div>
);
}
export function OrganizationList({
organizations,
isLoading,
@@ -160,6 +80,7 @@ export function OrganizationList({
sourceProvider = "github",
sourceUrl: sourceUrlProp,
sources,
view = "cards",
}: OrganizationListProps) {
const { giteaConfig, mirrorOptions, advancedOptions } = useGiteaConfig();
const [overridesTarget, setOverridesTarget] = useState<Organization | null>(null);
@@ -339,12 +260,43 @@ export function OrganizationList({
return result;
}, [organizations, filter]);
const overridesDialog = (
<MirrorOverridesDialog
open={!!overridesTarget}
onOpenChange={(open) => {
if (!open) setOverridesTarget(null);
}}
targetKind="organization"
targetName={overridesTarget?.name ?? ""}
value={overridesTarget?.mirrorOverrides ?? null}
destinationProvider={giteaConfig?.provider}
inheritedFrom={{
...mirrorOptionsToFlags(mirrorOptions),
skipForks: !!advancedOptions?.skipForks,
}}
inheritedLabel="global settings"
onSave={async (overrides) => {
if (overridesTarget?.id) {
await handleUpdateMirrorOverrides(overridesTarget.id, overrides);
}
}}
/>
);
return isLoading ? (
<div className="grid grid-cols-1 sm:grid-cols-[repeat(auto-fill,minmax(27rem,1fr))] gap-4">
{Array.from({ length: 5 }).map((_, i) => (
<Skeleton key={i} className="h-[11.25rem] w-full" />
))}
</div>
view === "list" ? (
<div className="flex flex-col gap-2">
{Array.from({ length: 6 }).map((_, i) => (
<Skeleton key={i} className="h-14 w-full" />
))}
</div>
) : (
<div className="grid grid-cols-1 sm:grid-cols-[repeat(auto-fill,minmax(27rem,1fr))] gap-4">
{Array.from({ length: 5 }).map((_, i) => (
<Skeleton key={i} className="h-[11.25rem] w-full" />
))}
</div>
)
) : filteredOrganizations.length === 0 ? (
<div className="flex flex-col items-center justify-center py-12 text-center">
<Building2 className="h-12 w-12 text-muted-foreground mb-4" />
@@ -373,6 +325,29 @@ export function OrganizationList({
</Button>
)}
</div>
) : view === "list" ? (
<>
<OrganizationsTable
organizations={filteredOrganizations}
loadingOrgIds={loadingOrgIds}
sources={sources}
sourceProvider={sourceProvider}
sourceUrl={sourceUrl}
destination={destination}
getDestinationUrl={getGiteaOrgUrl}
destinationLinkTooltip={destinationLinkTooltip}
defaultDestinationFor={defaultDestinationFor}
onUpdateDestination={handleUpdateDestination}
onMoveMirrors={destination.isPushTarget ? undefined : handleMoveMirrors}
onUpdateSource={handleUpdateSource}
onMirror={onMirror}
onSync={onSync}
onIgnore={onIgnore}
onDelete={onDelete}
onEditMirrorOptions={setOverridesTarget}
/>
{overridesDialog}
</>
) : (
<div className="grid grid-cols-1 sm:grid-cols-[repeat(auto-fill,minmax(27rem,1fr))] gap-4 pb-20 sm:pb-0">
{filteredOrganizations.map((org, index) => {
@@ -380,16 +355,12 @@ export function OrganizationList({
const statusBadge = getStatusBadge(org.status);
const StatusIcon = statusBadge.icon;
const orgSources = sources ?? [];
const pinnedSource =
orgSources.find((source) => source.id === org.sourceId) ?? null;
const isPublicOnlySource = pinnedSource?.token === "";
const orgProvider: SourceProviderKind = pinnedSource?.provider ?? sourceProvider;
const orgSourceUrl = pinnedSource
? normalizeSourceUrl(pinnedSource.url, pinnedSource.provider)
: sourceUrl;
const orgSourceLabel = SOURCE_PROVIDER_LABELS[orgProvider];
const orgSourceShortLabel = orgProvider === "gitea" ? "Gitea" : orgSourceLabel;
const OrgSourceIcon = SOURCE_PROVIDER_ICONS[orgProvider];
const orgSource = resolveOrganizationSource(org, sources, sourceProvider, sourceUrl);
const { pinnedSource, isPublicOnlySource } = orgSource;
const orgSourceUrl = orgSource.sourceUrl;
const orgSourceLabel = orgSource.label;
const orgSourceShortLabel = orgSource.shortLabel;
const OrgSourceIcon = SOURCE_PROVIDER_ICONS[orgSource.provider];
return (
<Card
@@ -724,51 +695,15 @@ export function OrganizationList({
</>
)}
{/* Dropdown menu for additional actions */}
{org.status !== "mirroring" && (
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button variant="ghost" size="icon" disabled={isLoading} className="h-10 w-10">
<MoreVertical className="h-4 w-4" />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end">
{onSync && (org.status === "mirrored" || org.status === "failed") && (
<>
<DropdownMenuItem onClick={() => org.id && onSync({ orgId: org.id })}>
<RefreshCw className="h-4 w-4 mr-2" />
Sync Organization
</DropdownMenuItem>
<DropdownMenuSeparator />
</>
)}
<DropdownMenuItem onClick={() => setOverridesTarget(org)}>
<SlidersHorizontal className="h-4 w-4 mr-2" />
Mirror Options
</DropdownMenuItem>
{org.status !== "ignored" && (
<DropdownMenuItem
onClick={() => org.id && onIgnore && onIgnore({ orgId: org.id, ignore: true })}
>
<Ban className="h-4 w-4 mr-2" />
Ignore Organization
</DropdownMenuItem>
)}
{onDelete && (
<>
{org.status !== "ignored" && <DropdownMenuSeparator />}
<DropdownMenuItem
className="text-destructive focus:text-destructive"
onClick={() => org.id && onDelete(org.id)}
>
<Trash2 className="h-4 w-4 mr-2" />
Delete from Mirror
</DropdownMenuItem>
</>
)}
</DropdownMenuContent>
</DropdownMenu>
)}
<OrganizationActionsMenu
org={org}
disabled={isLoading}
onSync={onSync}
onIgnore={onIgnore}
onDelete={onDelete}
onEditMirrorOptions={setOverridesTarget}
triggerClassName="h-10 w-10"
/>
</div>
<div className="flex items-center gap-2 justify-center">
@@ -884,51 +819,14 @@ export function OrganizationList({
</>
)}
{/* Dropdown menu for additional actions */}
{org.status !== "mirroring" && (
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button variant="ghost" size="icon" disabled={isLoading}>
<MoreVertical className="h-4 w-4" />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end">
{onSync && (org.status === "mirrored" || org.status === "failed") && (
<>
<DropdownMenuItem onClick={() => org.id && onSync({ orgId: org.id })}>
<RefreshCw className="h-4 w-4 mr-2" />
Sync Organization
</DropdownMenuItem>
<DropdownMenuSeparator />
</>
)}
<DropdownMenuItem onClick={() => setOverridesTarget(org)}>
<SlidersHorizontal className="h-4 w-4 mr-2" />
Mirror Options
</DropdownMenuItem>
{org.status !== "ignored" && (
<DropdownMenuItem
onClick={() => org.id && onIgnore && onIgnore({ orgId: org.id, ignore: true })}
>
<Ban className="h-4 w-4 mr-2" />
Ignore Organization
</DropdownMenuItem>
)}
{onDelete && (
<>
<DropdownMenuSeparator />
<DropdownMenuItem
className="text-destructive focus:text-destructive"
onClick={() => org.id && onDelete(org.id)}
>
<Trash2 className="h-4 w-4 mr-2" />
Delete from Mirror
</DropdownMenuItem>
</>
)}
</DropdownMenuContent>
</DropdownMenu>
)}
<OrganizationActionsMenu
org={org}
disabled={isLoading}
onSync={onSync}
onIgnore={onIgnore}
onDelete={onDelete}
onEditMirrorOptions={setOverridesTarget}
/>
</div>
<div className="flex items-center gap-2">
@@ -988,26 +886,7 @@ export function OrganizationList({
);
})}
<MirrorOverridesDialog
open={!!overridesTarget}
onOpenChange={(open) => {
if (!open) setOverridesTarget(null);
}}
targetKind="organization"
targetName={overridesTarget?.name ?? ""}
value={overridesTarget?.mirrorOverrides ?? null}
destinationProvider={giteaConfig?.provider}
inheritedFrom={{
...mirrorOptionsToFlags(mirrorOptions),
skipForks: !!advancedOptions?.skipForks,
}}
inheritedLabel="global settings"
onSave={async (overrides) => {
if (overridesTarget?.id) {
await handleUpdateMirrorOverrides(overridesTarget.id, overrides);
}
}}
/>
{overridesDialog}
</div>
);
}
@@ -0,0 +1,348 @@
import { AlertCircle, Ban, Check, Clock, RefreshCw } from "lucide-react";
import { Badge } from "@/components/ui/badge";
import { Button } from "@/components/ui/button";
import {
Tooltip,
TooltipContent,
TooltipProvider,
TooltipTrigger,
} from "@/components/ui/tooltip";
import type { Organization } from "@/lib/db/schema";
import type { SourceApiRecord } from "@/types/config";
import type { OrganizationMoveResult } from "@/lib/destination-transfer";
import type { SourceProviderKind } from "@/lib/source-providers/kinds";
import { SOURCE_PROVIDER_ICONS } from "@/lib/source-providers/icons";
import { DestinationIcon, type destinationInfo } from "@/components/destination/DestinationIcon";
import { cn, formatLastSyncTime } from "@/lib/utils";
import { hasMirrorOverrides } from "@/lib/utils/mirror-overrides";
import { withBase } from "@/lib/base-path";
import { MirrorDestinationEditor } from "./MirrorDestinationEditor";
import { OrganizationSourceEditor } from "./OrganizationSourceEditor";
import { OrganizationActionsMenu } from "./OrganizationActionsMenu";
import {
describeRepositoryBreakdown,
getOrganizationStatusBadge,
resolveOrganizationSource,
} from "./organization-presentation";
const STATUS_ICONS = {
check: Check,
alert: AlertCircle,
clock: Clock,
ban: Ban,
} as const;
export interface OrganizationsTableProps {
organizations: Organization[];
loadingOrgIds: Set<string>;
sources?: SourceApiRecord[];
sourceProvider: SourceProviderKind;
sourceUrl: string;
destination: ReturnType<typeof destinationInfo>;
getDestinationUrl: (org: Organization) => string | null;
destinationLinkTooltip: (org: Organization, url: string | null) => string;
defaultDestinationFor: (orgName: string) => string;
onUpdateDestination: (orgId: string, destination: string | null) => Promise<void>;
onMoveMirrors?: (
orgId: string,
destination: string | null,
dryRun: boolean
) => Promise<OrganizationMoveResult>;
onUpdateSource: (orgId: string, sourceId: string | null) => Promise<void>;
onMirror: ({ orgId }: { orgId: string }) => Promise<void>;
onSync?: ({ orgId }: { orgId: string }) => Promise<void>;
onIgnore?: ({ orgId, ignore }: { orgId: string; ignore: boolean }) => Promise<void>;
onDelete?: (orgId: string) => void;
onEditMirrorOptions: (org: Organization) => void;
}
/**
* The compact list view of the Organizations page (#428): one row per
* organization so a few dozen fit on one screen. Same actions as the cards,
* with the secondary ones behind the menu.
*/
export function OrganizationsTable({
organizations,
loadingOrgIds,
sources,
sourceProvider,
sourceUrl,
destination,
getDestinationUrl,
destinationLinkTooltip,
defaultDestinationFor,
onUpdateDestination,
onMoveMirrors,
onUpdateSource,
onMirror,
onSync,
onIgnore,
onDelete,
onEditMirrorOptions,
}: OrganizationsTableProps) {
const hasMultipleSources = !!sources && sources.length > 1;
return (
<div className="overflow-x-auto pb-20 sm:pb-0" data-testid="organizations-table">
<div className="min-w-[64rem] border rounded-md">
{/* Header */}
<div className="h-[45px] flex items-center border-b bg-muted/50 text-sm font-medium">
<div className="h-full p-3 flex items-center flex-[2.2] sticky left-0 z-10 bg-background before:absolute before:inset-0 before:-z-10 before:bg-muted/50">
Organization
</div>
{hasMultipleSources && (
<div className="h-full p-3 flex items-center flex-[1.1]">Source</div>
)}
<div className="h-full p-3 flex items-center flex-[1.6]">Destination</div>
<div className="h-full p-3 flex items-center flex-[1]">Repositories</div>
<div className="h-full p-3 flex items-center flex-[1]">Last Mirrored</div>
<div className="h-full p-3 flex items-center flex-[0.9]">Status</div>
<div className="h-full p-3 flex items-center flex-[1.3]">Actions</div>
<div className="h-full p-3 flex items-center justify-center flex-[0.6]">Links</div>
</div>
{organizations.map((org) => {
const orgId = org.id ?? "";
const isLoading = loadingOrgIds.has(orgId);
const statusBadge = getOrganizationStatusBadge(org.status);
const StatusIcon = statusBadge.icon ? STATUS_ICONS[statusBadge.icon] : null;
const source = resolveOrganizationSource(org, sources, sourceProvider, sourceUrl);
const SourceIcon = SOURCE_PROVIDER_ICONS[source.provider];
const breakdown = describeRepositoryBreakdown(org, "long");
const destinationUrl = getDestinationUrl(org);
const destinationTooltip = destinationLinkTooltip(org, destinationUrl);
return (
<div
key={org.id ?? org.name}
data-testid="organization-row"
className={cn(
"group min-h-[48px] flex items-center border-b last:border-b-0 hover:bg-muted/50",
isLoading && "opacity-75"
)}
>
{/* Organization. Pinned to the left edge so the row stays
identifiable while a narrow screen scrolls the rest. */}
<div className="h-full px-3 py-2 flex items-center gap-2 flex-[2.2] min-w-0 sticky left-0 z-10 bg-background before:absolute before:inset-0 before:-z-10 group-hover:before:bg-muted/50">
<div className="min-w-0 flex-1">
<div className="flex items-center gap-2 min-w-0">
<a
href={`${withBase("/repositories")}?organization=${encodeURIComponent(org.name || "")}`}
className="font-medium hover:underline truncate"
title={`Show the repositories of ${org.name}`}
>
{org.name}
</a>
<Badge
variant={org.membershipRole === "member" ? "secondary" : "default"}
className="capitalize shrink-0 px-1.5 text-[11px] font-normal"
>
{org.membershipRole.replace(/_/g, " ")}
</Badge>
{source.isPublicOnlySource && (
<Badge
variant="secondary"
className="shrink-0 px-1.5 font-normal text-[11px]"
title="Mirrors public repositories only; its source has no account"
>
Public only
</Badge>
)}
{hasMirrorOverrides(org.mirrorOverrides) && (
<Badge
variant="outline"
className="shrink-0 px-1.5 text-[11px] font-normal text-amber-600 dark:text-amber-400 border-amber-500/40"
title="This organization overrides the global mirror options"
>
Custom
</Badge>
)}
</div>
{org.status === "failed" && org.errorMessage && (
<p
className="text-xs text-destructive truncate"
title={org.errorMessage}
>
{org.errorMessage}
</p>
)}
</div>
</div>
{/* Source (multi-source only) */}
{hasMultipleSources && (
<div className="h-full px-3 py-2 flex items-center flex-[1.1] min-w-0">
<OrganizationSourceEditor
sources={sources ?? []}
value={source.pinnedSource?.id ?? null}
disabled={isLoading}
onUpdateSource={(sourceId) => onUpdateSource(orgId, sourceId)}
compact
/>
</div>
)}
{/* Destination */}
<div className="h-full px-3 py-2 flex items-center flex-[1.6] min-w-0">
<MirrorDestinationEditor
organizationId={orgId}
organizationName={org.name!}
currentDestination={org.destinationOrg ?? undefined}
defaultDestination={defaultDestinationFor(org.name!)}
onUpdate={(next) => onUpdateDestination(orgId, next)}
onMoveMirrors={
onMoveMirrors
? (next, dryRun) => onMoveMirrors(orgId, next, dryRun)
: undefined
}
destinationLabel={destination.label}
isUpdating={isLoading}
compact
/>
</div>
{/* Repositories */}
<div className="h-full px-3 py-2 flex items-center flex-[1] min-w-0">
<div className="min-w-0">
<p className="text-sm">
<span className="font-semibold tabular-nums">{org.repositoryCount}</span>
<span className="text-muted-foreground ml-1">
{org.repositoryCount === 1 ? "repository" : "repositories"}
</span>
</p>
{breakdown.length > 0 && (
<p className="text-xs text-muted-foreground truncate" title={breakdown.join(", ")}>
{breakdown.join(" | ")}
</p>
)}
</div>
</div>
{/* Last Mirrored */}
<div className="h-full px-3 py-2 flex items-center flex-[1]">
<p className="text-sm">{formatLastSyncTime(org.lastMirrored ?? null)}</p>
</div>
{/* Status */}
<div className="h-full px-3 py-2 flex items-center flex-[0.9]">
{org.status === "failed" && org.errorMessage ? (
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<Badge variant="destructive" className="cursor-help flex items-center gap-1">
<AlertCircle className="h-3 w-3" />
{statusBadge.label}
</Badge>
</TooltipTrigger>
<TooltipContent className="max-w-xs">
<p className="text-sm">{org.errorMessage}</p>
</TooltipContent>
</Tooltip>
</TooltipProvider>
) : (
<Badge variant={statusBadge.variant} className="flex items-center gap-1">
{StatusIcon && (
<StatusIcon
className={cn("h-3 w-3", org.status === "mirroring" && "animate-pulse")}
/>
)}
{statusBadge.label}
</Badge>
)}
</div>
{/* Actions */}
<div className="h-full px-3 py-2 flex items-center gap-1 flex-[1.3]">
{org.status === "ignored" ? (
<Button
size="sm"
variant="outline"
onClick={() => orgId && onIgnore && onIgnore({ orgId, ignore: false })}
disabled={isLoading}
>
<Check className="h-4 w-4 mr-1.5" />
Include
</Button>
) : org.status === "imported" ? (
<Button size="sm" onClick={() => orgId && onMirror({ orgId })} disabled={isLoading}>
<RefreshCw className={cn("h-4 w-4 mr-1.5", isLoading && "animate-spin")} />
{isLoading ? "Starting" : "Mirror"}
</Button>
) : org.status === "mirroring" ? (
<Button size="sm" variant="outline" disabled>
<RefreshCw className="h-4 w-4 mr-1.5 animate-spin" />
Mirroring
</Button>
) : org.status === "failed" ? (
<Button
size="sm"
variant="destructive"
onClick={() => orgId && onMirror({ orgId })}
disabled={isLoading}
>
{isLoading ? (
<RefreshCw className="h-4 w-4 mr-1.5 animate-spin" />
) : (
<AlertCircle className="h-4 w-4 mr-1.5" />
)}
{isLoading ? "Retrying" : "Retry"}
</Button>
) : org.status === "mirrored" && onSync ? (
<Button
size="sm"
variant="outline"
onClick={() => orgId && onSync({ orgId })}
disabled={isLoading}
title="Re-sync this organization now"
>
<RefreshCw className={cn("h-4 w-4 mr-1.5", isLoading && "animate-spin")} />
Sync
</Button>
) : null}
<OrganizationActionsMenu
org={org}
disabled={isLoading}
onSync={onSync}
onIgnore={onIgnore}
onDelete={onDelete}
onEditMirrorOptions={onEditMirrorOptions}
triggerClassName="h-8 w-8"
/>
</div>
{/* Links */}
<div className="h-full px-3 py-2 flex items-center justify-center gap-x-1 flex-[0.6]">
{destinationUrl ? (
<Button variant="ghost" size="icon" asChild className="h-8 w-8">
<a
href={destinationUrl}
target="_blank"
rel="noopener noreferrer"
title={destinationTooltip}
>
<DestinationIcon provider={destination.provider} className="h-4 w-4" />
</a>
</Button>
) : (
<Button variant="ghost" size="icon" disabled title={destinationTooltip} className="h-8 w-8">
<DestinationIcon provider={destination.provider} className="h-4 w-4" />
</Button>
)}
<Button variant="ghost" size="icon" asChild className="h-8 w-8">
<a
href={`${source.sourceUrl}/${org.name}`}
target="_blank"
rel="noopener noreferrer"
title={`View on ${source.label}`}
>
<SourceIcon className="h-4 w-4" />
</a>
</Button>
</div>
</div>
);
})}
</div>
</div>
);
}
@@ -0,0 +1,89 @@
import type { Organization } from "@/lib/db/schema";
import type { SourceApiRecord } from "@/types/config";
import {
SOURCE_PROVIDER_LABELS,
normalizeSourceUrl,
type SourceProviderKind,
} from "@/lib/source-providers/kinds";
export interface OrganizationSourceInfo {
/** The source this organization is pinned to, if any. */
pinnedSource: SourceApiRecord | null;
/** A pinned source with no account mirrors public repositories only. */
isPublicOnlySource: boolean;
provider: SourceProviderKind;
/** Normalized instance URL of the source the organization lives on. */
sourceUrl: string;
/** Full provider label, for tooltips. */
label: string;
/** Shorter label for buttons: Gitea and Forgejo sources both read "Gitea". */
shortLabel: string;
}
/**
* Which source an organization belongs to, for the cards and the list.
* Unpinned organizations follow the configured (or first) source.
*/
export function resolveOrganizationSource(
org: Pick<Organization, "sourceId">,
sources: SourceApiRecord[] | undefined,
fallbackProvider: SourceProviderKind,
fallbackUrl: string
): OrganizationSourceInfo {
const pinnedSource = (sources ?? []).find((source) => source.id === org.sourceId) ?? null;
const provider: SourceProviderKind = pinnedSource?.provider ?? fallbackProvider;
const label = SOURCE_PROVIDER_LABELS[provider];
return {
pinnedSource,
isPublicOnlySource: pinnedSource?.token === "",
provider,
sourceUrl: pinnedSource
? normalizeSourceUrl(pinnedSource.url, pinnedSource.provider)
: fallbackUrl,
label,
shortLabel: provider === "gitea" ? "Gitea" : label,
};
}
export type OrganizationStatusBadge = {
variant: "default" | "secondary" | "outline" | "destructive";
label: string;
icon: "check" | "alert" | "clock" | "ban" | null;
};
/** Badge variant, label and icon for an organization status. */
export function getOrganizationStatusBadge(status: string | null | undefined): OrganizationStatusBadge {
switch (status) {
case "imported":
return { variant: "secondary", label: "Not Mirrored", icon: null };
case "mirroring":
return { variant: "outline", label: "Mirroring", icon: "clock" };
case "mirrored":
return { variant: "default", label: "Mirrored", icon: "check" };
case "failed":
return { variant: "destructive", label: "Failed", icon: "alert" };
case "ignored":
return { variant: "outline", label: "Ignored", icon: "ban" };
default:
return { variant: "secondary", label: "Unknown", icon: null };
}
}
/** "3 public | 2 private | 1 fork" style breakdown, only the non-zero parts. */
export function describeRepositoryBreakdown(
org: Pick<Organization, "publicRepositoryCount" | "privateRepositoryCount" | "forkRepositoryCount">,
style: "long" | "short" = "long"
): string[] {
const parts: string[] = [];
const pub = org.publicRepositoryCount ?? 0;
const priv = org.privateRepositoryCount ?? 0;
const forks = org.forkRepositoryCount ?? 0;
if (pub > 0) parts.push(style === "long" ? `${pub} public` : `${pub} pub`);
if (priv > 0) parts.push(style === "long" ? `${priv} private` : `${priv} priv`);
if (forks > 0) {
parts.push(
style === "long" ? `${forks} ${forks === 1 ? "fork" : "forks"}` : `${forks} fork`
);
}
return parts;
}
+28
View File
@@ -0,0 +1,28 @@
import { useCallback, useSyncExternalStore } from "react";
import {
DEFAULT_ORGANIZATIONS_VIEW,
getOrganizationsView,
setOrganizationsView,
subscribeToOrganizationsViewChange,
type OrganizationsViewMode,
} from "@/lib/utils/organizations-view";
const getServerSnapshot = (): OrganizationsViewMode => DEFAULT_ORGANIZATIONS_VIEW;
/**
* Cards or list on the Organizations page (#428). The choice is kept in
* localStorage and shared across tabs.
*/
export function useOrganizationsView() {
const view = useSyncExternalStore(
subscribeToOrganizationsViewChange,
getOrganizationsView,
getServerSnapshot
);
const setView = useCallback((next: OrganizationsViewMode) => {
setOrganizationsView(next);
}, []);
return { view, setView };
}
+106
View File
@@ -0,0 +1,106 @@
/**
* Regression test for the password reset CLI (#461).
*
* `manage-db reset-password` used a Drizzle client without a schema, so
* `db.query.users` was undefined and every reset failed. Each case builds a
* real database with `manage-db init` (the migrations in drizzle/), seeds it,
* runs the reset through the source script and through the bundle the Docker
* image ships, and checks the result with Better Auth.
*
* Other test files replace bun:sqlite with mock.module, which is process wide
* in bun and depends on file order, so the suite runs in an isolated child
* process (same harness as gitea-incremental-metadata.test.ts).
*/
import { afterAll, beforeAll, describe, expect, test } from "bun:test";
import { mkdtempSync, rmSync, symlinkSync } from "node:fs";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
const CHILD_FLAG = "GM_RESET_PASSWORD_CLI_ISOLATED";
const isChild = !!process.env[CHILD_FLAG];
if (!isChild) {
test("reset-password CLI - isolated child suite", () => {
const res = Bun.spawnSync({
cmd: [process.execPath, "test", import.meta.path],
env: { ...process.env, [CHILD_FLAG]: "1" },
stdout: "pipe",
stderr: "pipe",
});
if (res.exitCode !== 0) {
console.error(res.stdout.toString());
console.error(res.stderr.toString());
}
expect(res.exitCode).toBe(0);
}, 60_000);
}
const root = resolve(import.meta.dir, "../../..");
let temporaryRoot = "";
let bundle = "";
function run(cmd: string[], cwd: string) {
const result = Bun.spawnSync({ cmd, cwd, stdout: "pipe", stderr: "pipe" });
expect(result.exitCode, Buffer.from(result.stderr).toString()).toBe(0);
return Buffer.from(result.stdout).toString();
}
describe.skipIf(!isChild)("manage-db reset-password", () => {
beforeAll(() => {
temporaryRoot = mkdtempSync(join(tmpdir(), "gitea-mirror-reset-password-"));
bundle = join(temporaryRoot, "manage-db.js");
run([process.execPath, "build", "scripts/manage-db.ts", "--target=bun", `--outfile=${bundle}`], root);
});
afterAll(() => {
if (temporaryRoot) rmSync(temporaryRoot, { recursive: true, force: true });
});
for (const bundled of [false, true]) {
for (const existingCredential of [false, true]) {
test(`${bundled ? "bundled" : "source"} script, ${existingCredential ? "existing" : "new"} credential`, async () => {
const { Database } = await import("bun:sqlite");
const { verifyPassword } = await import("better-auth/crypto");
const script = bundled ? bundle : join(root, "scripts/manage-db.ts");
// A real schema: init runs the migrations from ./drizzle.
const cwd = mkdtempSync(join(temporaryRoot, "case-"));
symlinkSync(join(root, "drizzle"), join(cwd, "drizzle"));
run([process.execPath, script, "init"], cwd);
const db = new Database(join(cwd, "data/gitea-mirror.db"));
try {
db.exec(`
INSERT INTO users (id, name, email) VALUES ('target', 'Target', 'admin@example.com'), ('other', 'Other', 'other@example.com');
INSERT INTO accounts (id, account_id, user_id, provider_id, password)
VALUES ('oauth', 'remote', 'target', 'github', NULL), ('other-account', 'other', 'other', 'credential', 'unchanged');
INSERT INTO sessions (id, token, user_id, expires_at) VALUES ('one', 'one', 'target', 0), ('two', 'two', 'target', 0), ('other', 'other', 'other', 0);
`);
if (existingCredential) {
db.exec("INSERT INTO accounts (id, account_id, user_id, provider_id, password) VALUES ('credential', 'target', 'target', 'credential', 'old')");
}
const output = run(
[process.execPath, script, "reset-password", "--email=admin@example.com", "--new-password=new-secure-password"],
cwd
);
expect(output).toContain("Password reset for admin@example.com");
expect(output).toContain("Cleared 2 active session(s)");
const credentials = db
.query("SELECT * FROM accounts WHERE user_id = 'target' AND provider_id = 'credential'")
.all() as Array<{ password: string; issuer: string }>;
expect(credentials).toHaveLength(1);
expect(await verifyPassword({ hash: credentials[0].password, password: "new-secure-password" })).toBe(true);
if (!existingCredential) expect(credentials[0].issuer).toBe("local:credential");
expect(db.query("SELECT id FROM sessions").all()).toEqual([{ id: "other" }]);
expect(db.query("SELECT password FROM accounts WHERE id = 'other-account'").get()).toEqual({ password: "unchanged" });
expect(db.query("SELECT password FROM accounts WHERE id = 'oauth'").get()).toEqual({ password: null });
expect(db.query("SELECT id FROM users").all()).toHaveLength(2);
} finally {
db.close();
}
}, 15_000);
}
}
});
+10 -3
View File
@@ -9,10 +9,11 @@ mock.module("@/lib/helpers", () => ({
}));
const mockMirrorGitHubReleasesToGitea = mock(() => Promise.resolve());
const mockMirrorGitRepoIssuesToGitea = mock(() => Promise.resolve());
const mockMirrorGitRepoPullRequestsToGitea = mock(() => Promise.resolve());
const mockMirrorGitRepoIssuesToGitea = mock(() => Promise.resolve(undefined));
const mockMirrorGitRepoPullRequestsToGitea = mock(() => Promise.resolve(undefined));
const mockMirrorGitRepoLabelsToGitea = mock(() => Promise.resolve());
const mockMirrorGitRepoMilestonesToGitea = mock(() => Promise.resolve());
const mockMilestoneIds = new Map<string, number>([["v1.0", 7]]);
const mockMirrorGitRepoMilestonesToGitea = mock(() => Promise.resolve(mockMilestoneIds));
const mockSyncRepositoryMetadataToGitea = mock(() => Promise.resolve());
const mockGetGiteaRepoOwnerAsync = mock(() => Promise.resolve("starred"));
const mockCreatePreSyncBundleBackup = mock(() =>
@@ -1576,6 +1577,12 @@ describe("Enhanced Gitea Operations", () => {
expect(mockMirrorGitRepoMilestonesToGitea).toHaveBeenCalledTimes(1);
// Labels should be skipped because issues already import them
expect(mockMirrorGitRepoLabelsToGitea).not.toHaveBeenCalled();
// Milestones run first and their ids reach the issues and pull
// requests passes, which assign each item to its milestone (#460).
const issuesArgs = (mockMirrorGitRepoIssuesToGitea.mock.calls[0] as any[])[0];
const pullsArgs = (mockMirrorGitRepoPullRequestsToGitea.mock.calls[0] as any[])[0];
expect(issuesArgs.milestoneIds).toBe(mockMilestoneIds);
expect(pullsArgs.milestoneIds).toBe(mockMilestoneIds);
});
test("skips issues and PRs when metadata shows they were already synced", async () => {
+85 -37
View File
@@ -7,6 +7,8 @@
*/
import type { Config } from "@/types/config";
import { isRateLimitError } from "@/lib/rate-limit-gate";
import { persistMetadataPassProgress } from "./metadata-progress-store";
import {
getRepositorySource,
SOURCE_PROVIDER_LABELS,
@@ -40,7 +42,7 @@ import {
strategyNeedsDetection,
} from "./repo-backup";
import { detectForcePush } from "./utils/force-push-detection";
import { sanitizeRepoNameAlphaDashDot } from "./gitea";
import { sanitizeRepoNameAlphaDashDot, type GiteaMilestoneIds } from "./gitea";
import {
parseRepositoryMetadataState,
serializeRepositoryMetadataState,
@@ -1072,6 +1074,9 @@ export async function syncGiteaRepoEnhanced({
);
}
} catch (releaseError) {
// Rate limited: fail the sync so the scheduler restores the row and
// retries after the reset instead of running the next component (#437).
if (isRateLimitError(releaseError)) throw releaseError;
console.error(
`[Sync] Failed to mirror releases for ${repository.name}: ${
releaseError instanceof Error
@@ -1082,6 +1087,44 @@ export async function syncGiteaRepoEnhanced({
}
}
// Milestones go first so the issues and pull requests passes below
// can assign each item to its milestone (#460).
let milestoneIds: GiteaMilestoneIds | undefined;
if (shouldMirrorMilestones) {
const octokit = ensureOctokit();
if (!octokit) {
console.warn(
`[Sync] Skipping milestone mirroring for ${repository.name}: Missing GitHub token`
);
} else {
try {
milestoneIds = await dependencies.mirrorGitRepoMilestonesToGitea({
config,
octokit,
repository,
giteaOwner: repoOwner,
giteaRepoName: repoName,
});
metadataState.components.milestones = true;
metadataUpdated = true;
console.log(
`[Sync] Mirrored milestones for ${repository.name} after sync`
);
} catch (milestoneError) {
// Rate limited: fail the sync so the scheduler restores the row and
// retries after the reset instead of running the next component (#437).
if (isRateLimitError(milestoneError)) throw milestoneError;
console.error(
`[Sync] Failed to mirror milestones for ${repository.name}: ${
milestoneError instanceof Error
? milestoneError.message
: String(milestoneError)
}`
);
}
}
}
if (shouldMirrorIssuesThisRun) {
const octokit = ensureOctokit();
if (!octokit) {
@@ -1090,13 +1133,21 @@ export async function syncGiteaRepoEnhanced({
);
} else {
try {
await dependencies.mirrorGitRepoIssuesToGitea({
const issuesCursor = await dependencies.mirrorGitRepoIssuesToGitea({
config,
octokit,
milestoneIds,
repository,
giteaOwner: repoOwner,
giteaRepoName: repoName,
syncCursor: metadataState.syncCursors.issues,
passProgress: metadataState.passProgress.issues,
onPassProgress: async (progress) => {
metadataUpdated = true;
await persistMetadataPassProgress(repository.id, metadataState, "issues", progress);
},
});
if (issuesCursor) metadataState.syncCursors.issues = issuesCursor;
metadataState.components.issues = true;
metadataState.components.labels = true;
metadataUpdated = true;
@@ -1104,6 +1155,9 @@ export async function syncGiteaRepoEnhanced({
`[Sync] Mirrored issues for ${repository.name} after sync`
);
} catch (issueError) {
// Rate limited: fail the sync so the scheduler restores the row and
// retries after the reset instead of running the next component (#437).
if (isRateLimitError(issueError)) throw issueError;
console.error(
`[Sync] Failed to mirror issues for ${repository.name}: ${
issueError instanceof Error
@@ -1123,19 +1177,30 @@ export async function syncGiteaRepoEnhanced({
);
} else {
try {
await dependencies.mirrorGitRepoPullRequestsToGitea({
const pullRequestsCursor = await dependencies.mirrorGitRepoPullRequestsToGitea({
config,
octokit,
milestoneIds,
repository,
giteaOwner: repoOwner,
giteaRepoName: repoName,
syncCursor: metadataState.syncCursors.pullRequests,
passProgress: metadataState.passProgress.pullRequests,
onPassProgress: async (progress) => {
metadataUpdated = true;
await persistMetadataPassProgress(repository.id, metadataState, "pullRequests", progress);
},
});
if (pullRequestsCursor) metadataState.syncCursors.pullRequests = pullRequestsCursor;
metadataState.components.pullRequests = true;
metadataUpdated = true;
console.log(
`[Sync] Mirrored pull requests for ${repository.name} after sync`
);
} catch (prError) {
// Rate limited: fail the sync so the scheduler restores the row and
// retries after the reset instead of running the next component (#437).
if (isRateLimitError(prError)) throw prError;
console.error(
`[Sync] Failed to mirror pull requests for ${repository.name}: ${
prError instanceof Error ? prError.message : String(prError)
@@ -1166,6 +1231,9 @@ export async function syncGiteaRepoEnhanced({
`[Sync] Mirrored labels for ${repository.name} after sync`
);
} catch (labelError) {
// Rate limited: fail the sync so the scheduler restores the row and
// retries after the reset instead of running the next component (#437).
if (isRateLimitError(labelError)) throw labelError;
console.error(
`[Sync] Failed to mirror labels for ${repository.name}: ${
labelError instanceof Error
@@ -1177,38 +1245,6 @@ export async function syncGiteaRepoEnhanced({
}
}
if (shouldMirrorMilestones) {
const octokit = ensureOctokit();
if (!octokit) {
console.warn(
`[Sync] Skipping milestone mirroring for ${repository.name}: Missing GitHub token`
);
} else {
try {
await dependencies.mirrorGitRepoMilestonesToGitea({
config,
octokit,
repository,
giteaOwner: repoOwner,
giteaRepoName: repoName,
});
metadataState.components.milestones = true;
metadataUpdated = true;
console.log(
`[Sync] Mirrored milestones for ${repository.name} after sync`
);
} catch (milestoneError) {
console.error(
`[Sync] Failed to mirror milestones for ${repository.name}: ${
milestoneError instanceof Error
? milestoneError.message
: String(milestoneError)
}`
);
}
}
}
if (metadataUpdated) {
metadataState.lastSyncedAt = new Date().toISOString();
}
@@ -1301,7 +1337,17 @@ export async function syncGiteaRepoEnhanced({
throw syncError;
}
} catch (error) {
console.error(`[Sync] Error while syncing repository ${repository.name}:`, error);
if (isRateLimitError(error)) {
// Expected and handled: the scheduler puts the row back and retries
// after the reset. One line instead of the whole error object.
console.warn(
`[Sync] Sync of ${repository.name} paused by the source rate limit: ${
error instanceof Error ? error.message : String(error)
}`
);
} else {
console.error(`[Sync] Error while syncing repository ${repository.name}:`, error);
}
// Update repo with error status
await db
@@ -1318,7 +1364,9 @@ export async function syncGiteaRepoEnhanced({
userId: config.userId,
repositoryId: repository.id,
repositoryName: repository.name,
message: `Failed to sync repository: ${repository.name}`,
message: isRateLimitError(error)
? `Sync paused by the source rate limit: ${repository.name}`
: `Failed to sync repository: ${repository.name}`,
details: error instanceof Error ? error.message : "Unknown error",
status: "failed",
});
+985
View File
@@ -0,0 +1,985 @@
/**
* Behavioural tests for incremental issue and pull request sync (#449).
*
* With a watermark from a completed pass, the GitHub listing is asked only
* for what changed (`since`), and the per-issue comment listing and per-PR
* detail, commits and files calls run only for the items that came back.
* Without one, the pass is full and unchanged from before.
*
* The destination is a fake global fetch and GitHub is a fake Octokit. Other
* test files replace @/lib/http-client and friends with mock.module, which is
* process wide in bun and depends on file order, so the suites run in an
* isolated child process (same harness as gitea-source-releases.test.ts).
*/
import { afterEach, describe, expect, it, test } from "bun:test";
import type { Config } from "@/types/config";
import type { Repository } from "@/lib/db/schema";
import type { MetadataPassProgress, MetadataSyncCursor } from "@/lib/metadata-state";
const CHILD_FLAG = "GM_INCREMENTAL_METADATA_ISOLATED";
const isChild = !!process.env[CHILD_FLAG];
if (!isChild) {
test("incremental metadata sync - isolated child suite", () => {
const res = Bun.spawnSync({
cmd: [process.execPath, "test", import.meta.path],
env: { ...process.env, [CHILD_FLAG]: "1" },
stdout: "pipe",
stderr: "pipe",
});
if (res.exitCode !== 0) {
console.error(res.stdout.toString());
console.error(res.stderr.toString());
}
expect(res.exitCode).toBe(0);
}, 60_000);
}
const { mirrorGitRepoIssuesToGitea, mirrorGitRepoPullRequestsToGitea } = isChild
? await import("@/lib/gitea")
: ({} as typeof import("@/lib/gitea"));
const { INCREMENTAL_SYNC_SAFETY_MARGIN_MS } = isChild
? await import("@/lib/metadata-state")
: ({} as typeof import("@/lib/metadata-state"));
const GITEA_URL = "https://gitea.example.com";
const DEST_OWNER = "mirror-owner";
const DEST_REPO = "demo";
const DEST_API = `${GITEA_URL}/api/v1/repos/${DEST_OWNER}/${DEST_REPO}`;
const realFetch = globalThis.fetch;
afterEach(() => {
globalThis.fetch = realFetch;
});
const config = {
userId: "user-1",
githubConfig: { owner: "acme", token: "github-token" },
giteaConfig: {
url: GITEA_URL,
token: "gitea-token",
defaultOwner: DEST_OWNER,
issueConcurrency: 1,
pullRequestConcurrency: 1,
},
} as unknown as Partial<Config>;
const repository = {
id: "repo-1",
name: DEST_REPO,
fullName: "acme/demo",
owner: "acme",
isStarred: false,
mirrorOverrides: null,
organization: null,
} as unknown as Repository;
interface Recorded {
method: string;
url: string;
body?: string;
}
/** Destination Gitea with a fixed set of issues already present. */
function fakeGitea(
existingIssues: Array<{ number: number; title: string; milestone?: { id: number } | null }>
) {
const requests: Recorded[] = [];
let nextNumber = 1000;
globalThis.fetch = (async (input: any, init?: any) => {
const url = typeof input === "string" ? input : String(input);
const method = String(init?.method ?? "GET").toUpperCase();
const body = typeof init?.body === "string" ? init.body : undefined;
requests.push({ method, url, body });
const json = (payload: unknown, status = 200) =>
new Response(JSON.stringify(payload), {
status,
headers: { "content-type": "application/json" },
});
if (method === "GET" && url === DEST_API) return json({ id: 1, name: DEST_REPO });
if (method === "GET" && url === `${DEST_API}/labels`) return json([]);
if (method === "POST" && url === `${DEST_API}/labels`) return json({ id: 7 }, 201);
if (method === "GET" && url.startsWith(`${DEST_API}/issues?`)) {
// The defensive recheck searches by marker; nothing to recover.
if (url.includes("&q=")) return json([]);
const page = Number(new URL(url).searchParams.get("page") ?? "1");
return json(
page === 1
? existingIssues.map((issue) => ({ ...issue, body: "", state: "open" }))
: []
);
}
if (method === "POST" && url === `${DEST_API}/issues`) {
nextNumber += 1;
return json({ number: nextNumber, state: "open" }, 201);
}
if (method === "PATCH" && /\/issues\/\d+$/.test(url)) return json({});
if (method === "PUT" && /\/issues\/\d+\/labels$/.test(url)) return json([]);
if (method === "GET" && /\/issues\/\d+\/comments\?/.test(url)) return json([]);
if (method === "POST" && /\/issues\/\d+\/comments$/.test(url)) return json({ id: 1 }, 201);
throw new Error(`unexpected request: ${method} ${url}`);
}) as unknown as typeof fetch;
return {
requests,
createdTitles: () =>
requests
.filter((r) => r.method === "POST" && r.url === `${DEST_API}/issues`)
.map((r) => JSON.parse(r.body!).title as string),
createdBodies: () =>
requests
.filter((r) => r.method === "POST" && r.url === `${DEST_API}/issues`)
.map((r) => JSON.parse(r.body!).body as string),
};
}
function ghIssue(number: number, extra: Record<string, unknown> = {}) {
return {
number,
title: `Issue ${number}`,
body: `body ${number}`,
state: "open",
html_url: `https://github.com/acme/demo/issues/${number}`,
user: { login: "alice", html_url: "https://github.com/alice" },
labels: [],
assignees: [],
created_at: `2026-01-0${number % 9 || 1}T00:00:00Z`,
updated_at: "2026-09-23T00:00:00Z",
...extra,
};
}
function ghPull(number: number, extra: Record<string, unknown> = {}) {
return {
number,
title: `PR ${number}`,
body: `pr body ${number}`,
state: "open",
html_url: `https://github.com/acme/demo/pull/${number}`,
user: { login: "bob", html_url: "https://github.com/bob" },
created_at: "2026-01-01T00:00:00Z",
merged_at: null,
base: { ref: "main" },
head: { ref: `feature-${number}` },
...extra,
};
}
interface GitHubData {
/** What issues.listForRepo returns (issues and PR entries). */
issueListing: any[];
/** What pulls.list returns. */
pullListing?: any[];
/** When set, pulls.get throws for these numbers. */
failDetailFor?: number[];
/** GitHub refuses the comment listing of these issues with a rate limit. */
rateLimitCommentsFor?: number[];
/** GitHub refuses pulls.get for these numbers with a rate limit. */
rateLimitDetailFor?: number[];
}
function rateLimitRefusal() {
return Object.assign(new Error("API rate limit exceeded for user ID 1."), { status: 403 });
}
/** Fake Octokit that records every call made through it. */
function fakeOctokit(data: GitHubData) {
const calls: Array<{ endpoint: string; params: any }> = [];
const issues = {
listForRepo: async (params: any) => {
calls.push({ endpoint: "issues.listForRepo", params });
return { data: data.issueListing };
},
listComments: async (params: any) => {
calls.push({ endpoint: "issues.listComments", params });
if (data.rateLimitCommentsFor?.includes(params.issue_number)) throw rateLimitRefusal();
return { data: [] };
},
};
const pulls = {
list: async (params: any) => {
calls.push({ endpoint: "pulls.list", params });
return { data: data.pullListing ?? [] };
},
get: async (params: any) => {
calls.push({ endpoint: "pulls.get", params });
if (data.rateLimitDetailFor?.includes(params.pull_number)) throw rateLimitRefusal();
if (data.failDetailFor?.includes(params.pull_number)) {
throw new Error("detail unavailable");
}
return {
data: {
...ghPull(params.pull_number),
additions: 1,
deletions: 0,
changed_files: 1,
merged_by: null,
},
};
},
listCommits: async (params: any) => {
calls.push({ endpoint: "pulls.listCommits", params });
return { data: [] };
},
listFiles: async (params: any) => {
calls.push({ endpoint: "pulls.listFiles", params });
return { data: [] };
},
};
const octokit = {
rest: { issues, pulls },
paginate: async (fn: any, params: any, map?: (res: any) => any) => {
const res = await fn(params);
return map ? map(res) : res.data;
},
};
return {
octokit: octokit as any,
calls,
callsTo: (endpoint: string) => calls.filter((c) => c.endpoint === endpoint),
};
}
function recentCursor(): MetadataSyncCursor {
return {
lastPassStartedAt: new Date(Date.now() - 60 * 60 * 1000).toISOString(),
lastFullPassStartedAt: new Date(Date.now() - 2 * 24 * 60 * 60 * 1000).toISOString(),
};
}
function expectedSince(cursor: MetadataSyncCursor) {
return new Date(
Date.parse(cursor.lastPassStartedAt) - INCREMENTAL_SYNC_SAFETY_MARGIN_MS
).toISOString();
}
describe.skipIf(!isChild)("mirrorGitRepoIssuesToGitea incremental sync", () => {
it("runs a full pass without a watermark and records one", async () => {
fakeGitea([]);
const github = fakeOctokit({
issueListing: [ghIssue(1), ghIssue(2), ghPull(3, { pull_request: {} })],
});
const before = Date.now();
const cursor = await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
});
const after = Date.now();
const [listing] = github.callsTo("issues.listForRepo");
expect(listing.params.since).toBeUndefined();
expect(listing.params.sort).toBe("created");
expect(listing.params.direction).toBe("asc");
expect(github.callsTo("issues.listComments").map((c) => c.params.issue_number)).toEqual([1, 2]);
expect(cursor).toBeDefined();
const started = Date.parse(cursor!.lastPassStartedAt);
expect(started).toBeGreaterThanOrEqual(before);
expect(started).toBeLessThanOrEqual(after);
expect(cursor!.lastFullPassStartedAt).toBe(cursor!.lastPassStartedAt);
});
it("lists with since and only reconciles the returned issues", async () => {
fakeGitea([
{ number: 11, title: "[GH-ISSUE #1] Issue 1" },
{ number: 12, title: "[GH-ISSUE #2] Issue 2" },
{ number: 13, title: "[GH-ISSUE #3] Issue 3" },
]);
const github = fakeOctokit({ issueListing: [ghIssue(2)] });
const previous = recentCursor();
const cursor = await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: previous,
});
const listings = github.callsTo("issues.listForRepo");
expect(listings).toHaveLength(1);
expect(listings[0].params.since).toBe(expectedSince(previous));
expect(listings[0].params.sort).toBe("updated");
expect(github.callsTo("issues.listComments").map((c) => c.params.issue_number)).toEqual([2]);
// The watermark moves to this listing; the last full pass is kept.
expect(cursor).toBeDefined();
expect(Date.parse(cursor!.lastPassStartedAt)).toBeGreaterThan(
Date.parse(previous.lastPassStartedAt)
);
expect(cursor!.lastFullPassStartedAt).toBe(previous.lastFullPassStartedAt);
});
it("creates new issues from an incremental listing in creation order", async () => {
const gitea = fakeGitea([{ number: 11, title: "[GH-ISSUE #1] Issue 1" }]);
// GitHub returns them by updated_at: #5 changed before #4.
const github = fakeOctokit({
issueListing: [
ghIssue(5, { created_at: "2026-09-22T10:00:00Z", updated_at: "2026-09-23T01:00:00Z" }),
ghIssue(4, { created_at: "2026-09-22T09:00:00Z", updated_at: "2026-09-23T02:00:00Z" }),
],
});
await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: recentCursor(),
});
expect(gitea.createdTitles()).toEqual(["[GH-ISSUE #4] Issue 4", "[GH-ISSUE #5] Issue 5"]);
});
it("falls back to a full pass when the destination has no mirrored issues", async () => {
fakeGitea([]);
const github = fakeOctokit({ issueListing: [ghIssue(1)] });
const previous = recentCursor();
const cursor = await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: previous,
});
const listings = github.callsTo("issues.listForRepo");
expect(listings).toHaveLength(2);
expect(listings[0].params.since).toBe(expectedSince(previous));
expect(listings[1].params.since).toBeUndefined();
expect(cursor!.lastFullPassStartedAt).toBe(cursor!.lastPassStartedAt);
});
it("runs a full pass when the last full pass is older than 7 days", async () => {
fakeGitea([{ number: 11, title: "[GH-ISSUE #1] Issue 1" }]);
const github = fakeOctokit({ issueListing: [ghIssue(1), ghIssue(2)] });
await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: {
lastPassStartedAt: new Date(Date.now() - 60 * 60 * 1000).toISOString(),
lastFullPassStartedAt: new Date(Date.now() - 8 * 24 * 60 * 60 * 1000).toISOString(),
},
});
const listings = github.callsTo("issues.listForRepo");
expect(listings).toHaveLength(1);
expect(listings[0].params.since).toBeUndefined();
// #1 is in Gitea and has not changed since the last pass, so the full
// pass does not list its comments again; #2 is new.
expect(github.callsTo("issues.listComments").map((c) => c.params.issue_number)).toEqual([2]);
});
});
describe.skipIf(!isChild)("full passes that revisit unchanged items (#449 follow-up)", () => {
const weekOldFullPass = (): MetadataSyncCursor => ({
lastPassStartedAt: new Date(Date.now() - 60 * 60 * 1000).toISOString(),
lastFullPassStartedAt: new Date(Date.now() - 8 * 24 * 60 * 60 * 1000).toISOString(),
});
const recently = () => new Date(Date.now() - 5 * 60 * 1000).toISOString();
it("only lists comments for issues that changed or are missing", async () => {
const gitea = fakeGitea([
{ number: 11, title: "[GH-ISSUE #1] Issue 1" },
{ number: 12, title: "[GH-ISSUE #2] Issue 2" },
]);
const github = fakeOctokit({
issueListing: [ghIssue(1), ghIssue(2, { updated_at: recently() }), ghIssue(3)],
});
const cursor = await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: weekOldFullPass(),
});
expect(github.callsTo("issues.listComments").map((c) => c.params.issue_number)).toEqual([2, 3]);
// The unchanged issue is still brought in line from the listing.
expect(
gitea.requests.some((r) => r.method === "PATCH" && r.url === `${DEST_API}/issues/11`)
).toBe(true);
expect(gitea.createdTitles()).toEqual(["[GH-ISSUE #3] Issue 3"]);
expect(cursor!.lastFullPassStartedAt).toBe(cursor!.lastPassStartedAt);
});
it("leaves an unchanged issue alone when the Gitea copy already matches", async () => {
const gitea = fakeGitea([]);
const issue = ghIssue(1);
const matching = {
number: 11,
title: "[GH-ISSUE #1] Issue 1",
body:
`Originally created by @alice on GitHub (${new Intl.DateTimeFormat("en-US", {
year: "numeric",
month: "short",
day: "numeric",
}).format(new Date(issue.created_at))}).\n` +
`Original GitHub issue: ${issue.html_url}\n\n${issue.body}`,
state: "open",
labels: [],
};
// Serve an exact copy by overriding the listing response.
globalThis.fetch = ((original) =>
(async (input: any, init?: any) => {
const url = typeof input === "string" ? input : String(input);
const method = String(init?.method ?? "GET").toUpperCase();
if (method === "GET" && url.startsWith(`${DEST_API}/issues?`) && !url.includes("&q=")) {
const page = Number(new URL(url).searchParams.get("page") ?? "1");
return new Response(JSON.stringify(page === 1 ? [matching] : []), {
status: 200,
headers: { "content-type": "application/json" },
});
}
return original(input, init);
}) as unknown as typeof fetch)(globalThis.fetch);
const github = fakeOctokit({ issueListing: [issue] });
await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: weekOldFullPass(),
});
expect(github.callsTo("issues.listComments")).toHaveLength(0);
// Nothing to write either: no edit, no label reset, no create.
expect(gitea.requests.filter((r) => r.method !== "GET")).toEqual([]);
});
it("skips the detail, commits and files calls for unchanged pull requests", async () => {
fakeGitea([
{ number: 21, title: "[PR #1] PR 1" },
{ number: 22, title: "[PR #2] PR 2" },
]);
const github = fakeOctokit({
issueListing: [],
pullListing: [
ghPull(1, { updated_at: "2026-09-01T00:00:00Z" }),
ghPull(2, { updated_at: recently() }),
ghPull(3, { updated_at: "2026-09-01T00:00:00Z" }),
],
});
const cursor = await mirrorGitRepoPullRequestsToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: weekOldFullPass(),
});
// #1 is unchanged and mirrored; #2 changed; #3 is missing in Gitea.
expect(github.callsTo("pulls.get").map((c) => c.params.pull_number)).toEqual([2, 3]);
expect(github.callsTo("pulls.listCommits")).toHaveLength(2);
expect(github.callsTo("pulls.listFiles")).toHaveLength(2);
expect(cursor!.lastFullPassStartedAt).toBe(cursor!.lastPassStartedAt);
});
it("does not treat anything as unchanged on the first pass, when there is no watermark", async () => {
fakeGitea([{ number: 21, title: "[PR #1] PR 1" }]);
const github = fakeOctokit({
issueListing: [],
pullListing: [ghPull(1, { updated_at: "2026-09-01T00:00:00Z" })],
});
await mirrorGitRepoPullRequestsToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
});
expect(github.callsTo("pulls.get").map((c) => c.params.pull_number)).toEqual([1]);
});
});
describe.skipIf(!isChild)("passes stopped by the rate limit resume (#449 follow-up)", () => {
it("keeps the finished issues and continues from there on the next run", async () => {
fakeGitea([]);
const listing = [ghIssue(1), ghIssue(2), ghIssue(3), ghIssue(4)];
const saved: Array<MetadataPassProgress | undefined> = [];
const first = fakeOctokit({ issueListing: listing, rateLimitCommentsFor: [3] });
let caught: unknown;
try {
await mirrorGitRepoIssuesToGitea({
config,
octokit: first.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
onPassProgress: (progress) => void saved.push(progress),
});
} catch (error) {
caught = error;
}
expect(String(caught)).toContain("rate limit");
// #3 was refused and #4 never started.
expect(first.callsTo("issues.listComments").map((c) => c.params.issue_number)).toEqual([1, 2, 3]);
expect(saved).toHaveLength(1);
const progress = saved[0]!;
expect(progress.mode).toBe("full");
expect(progress.done).toEqual([[1, 2]]);
// Next run, budget back: #1 and #2 are not fetched again.
fakeGitea([
{ number: 11, title: "[GH-ISSUE #1] Issue 1" },
{ number: 12, title: "[GH-ISSUE #2] Issue 2" },
{ number: 13, title: "[GH-ISSUE #3] Issue 3" },
]);
const second = fakeOctokit({ issueListing: listing });
const cursor = await mirrorGitRepoIssuesToGitea({
config,
octokit: second.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
passProgress: progress,
onPassProgress: (next) => void saved.push(next),
});
const [relisting] = second.callsTo("issues.listForRepo");
expect(relisting.params.since).toBeUndefined();
expect(second.callsTo("issues.listComments").map((c) => c.params.issue_number)).toEqual([3, 4]);
// Completed: the progress is cleared and the watermark is the start of
// the first attempt, so nothing changed in between is missed.
expect(saved[saved.length - 1]).toBeUndefined();
expect(cursor!.lastPassStartedAt).toBe(progress.startedAt);
expect(cursor!.lastFullPassStartedAt).toBe(progress.startedAt);
});
it("redoes a finished issue that changed after the interrupted pass started", async () => {
fakeGitea([
{ number: 11, title: "[GH-ISSUE #1] Issue 1" },
{ number: 12, title: "[GH-ISSUE #2] Issue 2" },
]);
const startedAt = new Date(Date.now() - 2 * 60 * 60 * 1000).toISOString();
const github = fakeOctokit({
issueListing: [ghIssue(1), ghIssue(2, { updated_at: new Date().toISOString() })],
});
await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
passProgress: { mode: "full", startedAt, done: [[1, 2]] },
});
expect(github.callsTo("issues.listComments").map((c) => c.params.issue_number)).toEqual([2]);
});
it("resumes an incremental pass with the same since", async () => {
fakeGitea([{ number: 11, title: "[GH-ISSUE #1] Issue 1" }]);
const since = new Date(Date.now() - 3 * 60 * 60 * 1000).toISOString();
const startedAt = new Date(Date.now() - 2 * 60 * 60 * 1000).toISOString();
const github = fakeOctokit({ issueListing: [ghIssue(1), ghIssue(2)] });
const cursor = await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: recentCursor(),
passProgress: { mode: "incremental", since, startedAt, done: [[1, 1]] },
});
const [listing] = github.callsTo("issues.listForRepo");
expect(listing.params.since).toBe(since);
expect(github.callsTo("issues.listComments").map((c) => c.params.issue_number)).toEqual([2]);
expect(cursor!.lastPassStartedAt).toBe(startedAt);
});
it("keeps the finished pull requests when the detail call is refused", async () => {
fakeGitea([]);
const pullListing = [ghPull(1), ghPull(2), ghPull(3)].map((pr) => ({
...pr,
updated_at: "2026-09-01T00:00:00Z",
}));
const saved: Array<MetadataPassProgress | undefined> = [];
const first = fakeOctokit({ issueListing: [], pullListing, rateLimitDetailFor: [2] });
await expect(
mirrorGitRepoPullRequestsToGitea({
config,
octokit: first.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
onPassProgress: (progress) => void saved.push(progress),
})
).rejects.toThrow(/rate limit/i);
expect(saved[0]!.done).toEqual([[1, 1]]);
fakeGitea([{ number: 21, title: "[PR #1] PR 1" }]);
const second = fakeOctokit({ issueListing: [], pullListing });
await mirrorGitRepoPullRequestsToGitea({
config,
octokit: second.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
passProgress: saved[0],
});
expect(second.callsTo("pulls.get").map((c) => c.params.pull_number)).toEqual([2, 3]);
});
});
describe.skipIf(!isChild)("mirrorGitRepoPullRequestsToGitea incremental sync", () => {
it("runs a full pass with pulls.list without a watermark", async () => {
fakeGitea([]);
const github = fakeOctokit({
issueListing: [],
pullListing: [ghPull(1), ghPull(2)],
});
const cursor = await mirrorGitRepoPullRequestsToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
});
expect(github.callsTo("pulls.list")).toHaveLength(1);
expect(github.callsTo("issues.listForRepo")).toHaveLength(0);
expect(github.callsTo("pulls.get").map((c) => c.params.pull_number)).toEqual([1, 2]);
expect(github.callsTo("pulls.listCommits")).toHaveLength(2);
expect(github.callsTo("pulls.listFiles")).toHaveLength(2);
expect(cursor).toBeDefined();
expect(cursor!.lastFullPassStartedAt).toBe(cursor!.lastPassStartedAt);
});
it("lists changed pull requests through the issues endpoint with since", async () => {
const gitea = fakeGitea([
{ number: 21, title: "[PR #1] PR 1" },
{ number: 22, title: "[PR #2] PR 2" },
{ number: 23, title: "[PR #3] PR 3" },
]);
const github = fakeOctokit({
// Issue #9 changed too; it is not a pull request and is skipped here.
issueListing: [
ghIssue(9),
{ ...ghIssue(3), title: "PR 3", pull_request: { merged_at: null } },
{ ...ghIssue(4), title: "PR 4", pull_request: { merged_at: null } },
],
});
const previous = recentCursor();
const cursor = await mirrorGitRepoPullRequestsToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: previous,
});
expect(github.callsTo("pulls.list")).toHaveLength(0);
const [listing] = github.callsTo("issues.listForRepo");
expect(listing.params.since).toBe(expectedSince(previous));
expect(listing.params.sort).toBe("updated");
expect(github.callsTo("pulls.get").map((c) => c.params.pull_number).sort()).toEqual([3, 4]);
expect(github.callsTo("pulls.listCommits")).toHaveLength(2);
expect(github.callsTo("pulls.listFiles")).toHaveLength(2);
// The new PR is built from the detail response, which has the refs.
const [created] = gitea.createdBodies();
expect(created).toContain("feature-4");
expect(gitea.createdTitles()).toEqual(["[PR #4] PR 4"]);
expect(cursor!.lastFullPassStartedAt).toBe(previous.lastFullPassStartedAt);
});
it("keeps the previous watermark when a pull request fails", async () => {
fakeGitea([{ number: 21, title: "[PR #1] PR 1" }]);
// Detail fails and the basic fallback create also fails.
globalThis.fetch = ((original) =>
(async (input: any, init?: any) => {
const url = typeof input === "string" ? input : String(input);
if (String(init?.method ?? "GET").toUpperCase() === "POST" && url === `${DEST_API}/issues`) {
return new Response("boom", { status: 500 });
}
return original(input, init);
}) as unknown as typeof fetch)(globalThis.fetch);
const github = fakeOctokit({
issueListing: [{ ...ghIssue(5), pull_request: { merged_at: null } }],
failDetailFor: [5],
});
const cursor = await mirrorGitRepoPullRequestsToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: recentCursor(),
});
expect(cursor).toBeUndefined();
expect(github.callsTo("pulls.get").map((c) => c.params.pull_number)).toEqual([5]);
});
});
describe.skipIf(!isChild)("milestone assignment (#460)", () => {
const milestoneIds = new Map<string, number>([
["v1.0", 7],
["v2.0", 8],
]);
const weekOldFullPass = (): MetadataSyncCursor => ({
lastPassStartedAt: new Date(Date.now() - 60 * 60 * 1000).toISOString(),
lastFullPassStartedAt: new Date(Date.now() - 8 * 24 * 60 * 60 * 1000).toISOString(),
});
const patchesTo = (gitea: ReturnType<typeof fakeGitea>, number: number) =>
gitea.requests
.filter((r) => r.method === "PATCH" && r.url === `${DEST_API}/issues/${number}`)
.map((r) => JSON.parse(r.body!));
const created = (gitea: ReturnType<typeof fakeGitea>) =>
gitea.requests
.filter((r) => r.method === "POST" && r.url === `${DEST_API}/issues`)
.map((r) => JSON.parse(r.body!));
it("creates issues with their milestone", async () => {
const gitea = fakeGitea([]);
const github = fakeOctokit({
issueListing: [ghIssue(1, { milestone: { title: "v1.0" } }), ghIssue(2, { milestone: null })],
});
await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
milestoneIds,
});
const [first, second] = created(gitea);
expect(first.milestone).toBe(7);
expect(second.milestone).toBeUndefined();
});
it("assigns a milestone set on GitHub after the issue was mirrored", async () => {
const gitea = fakeGitea([{ number: 11, title: "[GH-ISSUE #1] Issue 1", milestone: null }]);
const github = fakeOctokit({ issueListing: [ghIssue(1, { milestone: { title: "v2.0" } })] });
await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: recentCursor(),
milestoneIds,
});
expect(patchesTo(gitea, 11).map((p) => p.milestone)).toEqual([8]);
});
it("fixes and clears milestones on unchanged issues during a full pass", async () => {
const gitea = fakeGitea([
// Mirrored before #460: no milestone in Gitea yet.
{ number: 11, title: "[GH-ISSUE #1] Issue 1", milestone: null },
// Milestone removed on GitHub.
{ number: 12, title: "[GH-ISSUE #2] Issue 2", milestone: { id: 7 } },
// Already right: the edit leaves it alone.
{ number: 13, title: "[GH-ISSUE #3] Issue 3", milestone: { id: 8 } },
// On a milestone made in Gitea alone: kept.
{ number: 14, title: "[GH-ISSUE #4] Issue 4", milestone: { id: 99 } },
]);
const github = fakeOctokit({
issueListing: [
ghIssue(1, { milestone: { title: "v1.0" } }),
ghIssue(2, { milestone: null }),
ghIssue(3, { milestone: { title: "v2.0" } }),
ghIssue(4, { milestone: null }),
],
});
await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: weekOldFullPass(),
milestoneIds,
});
// All three are unchanged, so no comment listing runs.
expect(github.callsTo("issues.listComments")).toHaveLength(0);
expect(patchesTo(gitea, 11).map((p) => p.milestone)).toEqual([7]);
expect(patchesTo(gitea, 12).map((p) => p.milestone)).toEqual([0]);
expect(patchesTo(gitea, 13).every((p) => !("milestone" in p))).toBe(true);
expect(patchesTo(gitea, 14).every((p) => !("milestone" in p))).toBe(true);
});
it("leaves milestones alone when they are not mirrored or not found", async () => {
const gitea = fakeGitea([{ number: 11, title: "[GH-ISSUE #1] Issue 1", milestone: { id: 7 } }]);
const github = fakeOctokit({
issueListing: [
ghIssue(1, { milestone: null }),
ghIssue(2, { milestone: { title: "v1.0" } }),
],
});
await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: recentCursor(),
});
expect(patchesTo(gitea, 11).every((p) => !("milestone" in p))).toBe(true);
expect(created(gitea)[0].milestone).toBeUndefined();
// A milestone Gitea does not have is not cleared from the issue either.
const gitea2 = fakeGitea([{ number: 11, title: "[GH-ISSUE #1] Issue 1", milestone: { id: 7 } }]);
await mirrorGitRepoIssuesToGitea({
config,
octokit: fakeOctokit({ issueListing: [ghIssue(1, { milestone: { title: "v9" } })] }).octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: recentCursor(),
milestoneIds,
});
expect(patchesTo(gitea2, 11).every((p) => !("milestone" in p))).toBe(true);
});
it("assigns milestones to pull requests, new and unchanged", async () => {
const gitea = fakeGitea([{ number: 21, title: "[PR #1] PR 1", milestone: null }]);
const github = fakeOctokit({
issueListing: [],
pullListing: [
ghPull(1, { updated_at: "2026-01-01T00:00:00Z", milestone: { title: "v1.0" } }),
ghPull(2, { milestone: { title: "v2.0" } }),
],
});
await mirrorGitRepoPullRequestsToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: weekOldFullPass(),
milestoneIds,
});
// #1 is unchanged: no detail calls, only the milestone is patched.
expect(github.callsTo("pulls.get").map((c) => c.params.pull_number)).toEqual([2]);
expect(patchesTo(gitea, 21)).toEqual([{ milestone: 7 }]);
expect(created(gitea).map((c) => c.milestone)).toEqual([8]);
});
it("carries the milestone through an incremental pull request listing", async () => {
const gitea = fakeGitea([{ number: 21, title: "[PR #1] PR 1", milestone: null }]);
const github = fakeOctokit({
issueListing: [
{ ...ghIssue(1), title: "PR 1", pull_request: { merged_at: null }, milestone: { title: "v2.0" } },
],
});
await mirrorGitRepoPullRequestsToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: recentCursor(),
milestoneIds,
});
expect(patchesTo(gitea, 21).map((p) => p.milestone)).toEqual([8]);
});
it("does not take a mirrored issue that mentions [PR #N] for that pull request", async () => {
const gitea = fakeGitea([
{ number: 21, title: "[PR #1] PR 1", milestone: null },
{ number: 40, title: "[GH-ISSUE #40] Revert [PR #1]", milestone: null },
]);
const github = fakeOctokit({
issueListing: [],
pullListing: [ghPull(1, { updated_at: "2026-01-01T00:00:00Z", milestone: { title: "v1.0" } })],
});
await mirrorGitRepoPullRequestsToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: weekOldFullPass(),
milestoneIds,
});
expect(patchesTo(gitea, 21)).toEqual([{ milestone: 7 }]);
expect(patchesTo(gitea, 40)).toEqual([]);
});
it("finishes the pass when the milestone edit of an unchanged pull request fails", async () => {
fakeGitea([{ number: 21, title: "[PR #1] PR 1", milestone: null }]);
let milestoneEdits = 0;
globalThis.fetch = ((original) =>
(async (input: any, init?: any) => {
const url = typeof input === "string" ? input : String(input);
if (String(init?.method ?? "GET").toUpperCase() === "PATCH" && url === `${DEST_API}/issues/21`) {
milestoneEdits += 1;
return new Response("boom", { status: 500 });
}
return original(input, init);
}) as unknown as typeof fetch)(globalThis.fetch);
const github = fakeOctokit({
issueListing: [],
pullListing: [ghPull(1, { updated_at: "2026-01-01T00:00:00Z", milestone: { title: "v1.0" } })],
});
const cursor = await mirrorGitRepoPullRequestsToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: weekOldFullPass(),
milestoneIds,
});
// Tried once, not retried, and the pass still completes.
expect(milestoneEdits).toBe(1);
expect(cursor).toBeDefined();
});
});
+91 -1
View File
@@ -14,7 +14,13 @@
*/
import { describe, test, expect } from "bun:test";
import { buildGiteaIssueEditPayload, buildGiteaIssueLabelsPayload } from "@/lib/gitea";
import {
buildGiteaIssueEditPayload,
buildGiteaIssueLabelsPayload,
buildGiteaMilestoneEditPayload,
giteaMilestoneEdit,
resolveGiteaMilestoneId,
} from "@/lib/gitea";
describe("buildGiteaIssueEditPayload (#334 sibling)", () => {
test("edit body never carries `labels` (Gitea's EditIssueOption ignores it)", () => {
@@ -37,6 +43,90 @@ describe("buildGiteaIssueEditPayload (#334 sibling)", () => {
});
});
describe("issue milestones (#460)", () => {
const ids = new Map([["v1.0", 7]]);
test("the edit body carries a milestone only when one is given", () => {
expect(buildGiteaIssueEditPayload({ title: "t", body: "b", closed: false })).not.toHaveProperty(
"milestone"
);
expect(
buildGiteaIssueEditPayload({ title: "t", body: "b", closed: false, milestone: 7 }).milestone
).toBe(7);
// 0 is how Gitea clears a milestone, so it has to survive.
expect(
buildGiteaIssueEditPayload({ title: "t", body: "b", closed: false, milestone: 0 }).milestone
).toBe(0);
});
test("resolves the Gitea id by title, 0 for none, undefined when unknown", () => {
expect(resolveGiteaMilestoneId(ids, { title: "v1.0" })).toBe(7);
expect(resolveGiteaMilestoneId(ids, null)).toBe(0);
expect(resolveGiteaMilestoneId(ids, { title: "missing" })).toBeUndefined();
expect(resolveGiteaMilestoneId(undefined, { title: "v1.0" })).toBeUndefined();
expect(resolveGiteaMilestoneId(undefined, null)).toBeUndefined();
});
test("only edits when the issue carries a different milestone", () => {
expect(giteaMilestoneEdit({ milestone: null }, ids, { title: "v1.0" })).toBe(7);
expect(giteaMilestoneEdit({ milestone: { id: 7 } }, ids, { title: "v1.0" })).toBeUndefined();
expect(giteaMilestoneEdit({ milestone: { id: 7 } }, ids, null)).toBe(0);
expect(giteaMilestoneEdit({}, ids, null)).toBeUndefined();
expect(giteaMilestoneEdit({ milestone: { id: 7 } }, undefined, null)).toBeUndefined();
});
test("never clears a milestone made in Gitea alone", () => {
// 42 is not in the map: no GitHub milestone has its title.
expect(giteaMilestoneEdit({ milestone: { id: 42 } }, ids, null)).toBeUndefined();
// GitHub does name a milestone, so it wins over the Gitea-only one.
expect(giteaMilestoneEdit({ milestone: { id: 42 } }, ids, { title: "v1.0" })).toBe(7);
});
});
describe("buildGiteaMilestoneEditPayload (#460 follow-up)", () => {
test("no edit when state, due day and description match", () => {
expect(
buildGiteaMilestoneEditPayload(
// Created from GitHub: Gitea keeps the instant, shown in its own zone.
{ state: "open", due_on: "2026-11-01T00:00:00-07:00", description: "d" },
{ state: "open", due_on: "2026-11-01T07:00:00Z", description: "d" }
)
).toBeNull();
expect(
buildGiteaMilestoneEditPayload(
// Edited through the API: moved to the end of the day.
{ state: "closed", due_on: "2026-12-15T23:59:59+09:00", description: null },
{ state: "closed", due_on: "2026-12-15T08:00:00Z", description: null }
)
).toBeNull();
// Year 9999 is how Gitea stores no deadline.
expect(
buildGiteaMilestoneEditPayload(
{ state: "open", due_on: "9999-12-31T23:59:59-08:00", description: "" },
{ state: "open", due_on: null, description: "" }
)
).toBeNull();
});
test("carries over a close, a due date and a description", () => {
expect(
buildGiteaMilestoneEditPayload(
{ state: "open", due_on: null, description: "old" },
{ state: "closed", due_on: "2026-12-15T08:00:00Z", description: "new" }
)
).toEqual({ state: "closed", due_on: "2026-12-15T12:00:00Z", description: "new" });
});
test("clears a removed due date with Gitea's no-deadline value", () => {
expect(
buildGiteaMilestoneEditPayload(
{ state: "open", due_on: "2026-11-01T00:00:00-07:00", description: "" },
{ state: "open", due_on: null, description: "" }
)
).toEqual({ due_on: "9999-12-31T00:00:00Z" });
});
});
describe("buildGiteaIssueLabelsPayload (#334 sibling)", () => {
test("replaces the full label set with the resolved Gitea label ids", () => {
expect(buildGiteaIssueLabelsPayload([7, 9])).toEqual({ labels: [7, 9] });
+2 -1
View File
@@ -146,7 +146,8 @@ describe("milestone dedup on sync", () => {
// Defensive: if `milestones` ever contains a same-named entry
// twice (unlikely but cheap to guard), we shouldn't POST it twice.
expect(
/existingMilestones\.add\(\s*milestone\.title\s*\)/.test(body),
// A map since #460 (title to id, for issue assignment).
/existingMilestones\.set\(\s*milestone\.title\b/.test(body),
"after a successful create, the new milestone title must be added to existingMilestones"
).toBe(true);
});
+777 -115
View File
File diff suppressed because it is too large. Load diff
+141
View File
@@ -0,0 +1,141 @@
/**
* The GitHub clients built by github.ts hold their requests while the gate
* is closed, and the secondary limit handler closes the gate for everyone
* (issue #437, second report). Uses a fake global fetch so nothing leaves
* the process; the point of these tests is that nothing does.
*/
import { afterEach, beforeEach, describe, expect, test } from "bun:test";
import {
createGitHubClient,
createPublicGitHubClient,
githubThrottleOptions,
} from "@/lib/github";
import {
RateLimitPausedError,
clearAllRateLimits,
isRateLimitError,
markRateLimited,
rateLimitGateKey,
rateLimitedUntil,
} from "@/lib/rate-limit-gate";
const realFetch = globalThis.fetch;
describe("GitHub clients while the gate is closed", () => {
let sent: string[] = [];
beforeEach(() => {
clearAllRateLimits();
sent = [];
globalThis.fetch = (async (input: any) => {
const url = typeof input === "string" ? input : input.url;
sent.push(url);
return new Response(JSON.stringify({ resources: {}, rate: { remaining: 1 } }), {
status: 200,
headers: {
"content-type": "application/json",
"x-ratelimit-remaining": "4999",
"x-ratelimit-reset": String(Math.floor(Date.now() / 1000) + 3600),
},
});
}) as unknown as typeof fetch;
});
afterEach(() => {
globalThis.fetch = realFetch;
clearAllRateLimits();
});
test("an authenticated client sends nothing during a long pause and fails with the paused error", async () => {
const octokit = createGitHubClient("ghp_test", "user-1");
markRateLimited(rateLimitGateKey("user-1"), Date.now() + 3_263_000);
let caught: unknown;
try {
await octokit.rest.repos.get({ owner: "a", repo: "b" });
} catch (error) {
caught = error;
}
expect(caught).toBeInstanceOf(RateLimitPausedError);
expect(isRateLimitError(caught)).toBe(true);
expect(sent).toEqual([]);
});
test("paginate stops at the gate too, before the first page", async () => {
const octokit = createGitHubClient("ghp_test", "user-2");
markRateLimited(rateLimitGateKey("user-2"), Date.now() + 3_600_000);
await expect(
octokit.paginate(octokit.rest.issues.listForRepo, { owner: "a", repo: "b", per_page: 100 })
).rejects.toBeInstanceOf(RateLimitPausedError);
expect(sent).toEqual([]);
});
test("the rate limit probe still goes out, so the pause can be confirmed or lifted", async () => {
const octokit = createGitHubClient("ghp_test", "user-3");
markRateLimited(rateLimitGateKey("user-3"), Date.now() + 3_600_000);
await octokit.rest.rateLimit.get();
expect(sent).toHaveLength(1);
expect(sent[0]).toContain("/rate_limit");
// The probe answered with budget left, which opens the gate again.
expect(rateLimitedUntil(rateLimitGateKey("user-3"))).toBeNull();
await octokit.rest.repos.get({ owner: "a", repo: "b" });
expect(sent).toHaveLength(2);
});
test("another user's pause does not hold this user's client", async () => {
const octokit = createGitHubClient("ghp_test", "user-4");
markRateLimited(rateLimitGateKey("user-5"), Date.now() + 3_600_000);
await octokit.rest.repos.get({ owner: "a", repo: "b" });
expect(sent).toHaveLength(1);
});
test("tokenless clients share the anonymous gate", async () => {
const octokit = createPublicGitHubClient();
markRateLimited(rateLimitGateKey(undefined), Date.now() + 3_600_000);
await expect(octokit.rest.repos.get({ owner: "a", repo: "b" })).rejects.toBeInstanceOf(
RateLimitPausedError
);
expect(sent).toEqual([]);
});
});
describe("secondary rate limit handler", () => {
beforeEach(() => {
clearAllRateLimits();
});
test("closes the gate for the retry-after window and still retries the request itself", async () => {
const before = Date.now();
const options = githubThrottleOptions("user-secondary");
const retry = await options.onSecondaryRateLimit(
60,
{ method: "GET", url: "/repos/a/b/issues" },
{},
0
);
expect(retry).toBe(true);
const until = rateLimitedUntil(rateLimitGateKey("user-secondary"));
expect(until).not.toBeNull();
const pausedFor = until!.getTime() - before;
expect(pausedFor).toBeGreaterThanOrEqual(59_000);
expect(pausedFor).toBeLessThanOrEqual(61_000);
});
test("gives up after two retries but the gate stays closed for the window", async () => {
const options = githubThrottleOptions("user-secondary-2");
expect(await options.onSecondaryRateLimit(30, { method: "GET", url: "/x" }, {}, 2)).toBe(false);
expect(rateLimitedUntil(rateLimitGateKey("user-secondary-2"))).not.toBeNull();
});
test("a tokenless client records the pause under the anonymous key", async () => {
const options = githubThrottleOptions();
await options.onSecondaryRateLimit(10, { method: "GET", url: "/x" }, {}, 0);
expect(rateLimitedUntil(rateLimitGateKey(undefined))).not.toBeNull();
});
});
+24 -2
View File
@@ -10,6 +10,7 @@ import {
} from "@/lib/github-conditional-requests";
import {
clearRateLimit,
installRateLimitGate,
markRateLimited,
rateLimitGateKey,
} from "@/lib/rate-limit-gate";
@@ -80,7 +81,7 @@ export function decideRateLimitRetry({
* handlers only log and schedule the retry — there is no per-user
* rate-limit state or UI event to update for a tokenless source.
*/
function githubThrottleOptions(userId?: string) {
export function githubThrottleOptions(userId?: string) {
return {
onRateLimit: async (
retryAfter: number,
@@ -161,6 +162,11 @@ function githubThrottleOptions(userId?: string) {
`[GitHub] Secondary rate limit hit for ${options.method} ${options.url}`,
);
// A secondary limit is GitHub asking the whole client to slow down,
// not only this request. Close the gate for the retry-after window so
// the other queued calls wait instead of piling on (issue #437).
markRateLimited(rateLimitGateKey(userId), Date.now() + retryAfter * 1000);
// Update status and notify UI (if available)
if (userId && publishEvent) {
await publishEvent({
@@ -306,6 +312,13 @@ export function createGitHubClient(
scope: userId ?? username ?? conditionalRequestTokenScope(token),
});
// Registered last so it runs first: a closed gate is answered before the
// conditional request cache and before anything reaches the network.
installRateLimitGate(octokit, {
key: rateLimitGateKey(userId),
maxWaitMs: RATE_LIMIT_FAST_FAIL_THRESHOLD_SECONDS * 1000,
});
return octokit;
}
@@ -328,7 +341,7 @@ export function createPublicGitHubClient(apiBaseUrl?: string): Octokit {
// with GitHub Actions, which sets it to https://api.github.com by default).
const baseUrl = apiBaseUrl || process.env.GH_API_URL || process.env.GITHUB_API_URL || "https://api.github.com";
return new MyOctokit({
const octokit = new MyOctokit({
userAgent: "gitea-mirror/3.5.4",
baseUrl, // Configurable for E2E testing
log: {
@@ -345,6 +358,15 @@ export function createPublicGitHubClient(apiBaseUrl?: string): Octokit {
},
throttle: githubThrottleOptions(),
});
// Anonymous clients share one gate key, so one exhausted 60 req/hr budget
// pauses every tokenless source together.
installRateLimitGate(octokit, {
key: rateLimitGateKey(undefined),
maxWaitMs: RATE_LIMIT_FAST_FAIL_THRESHOLD_SECONDS * 1000,
});
return octokit;
}
/**
+44
View File
@@ -0,0 +1,44 @@
/**
* Write the progress of an unfinished issues or pull request pass as soon
* as it is known (#449 follow-up). A rate limit ends the sync with a throw,
* before the usual metadata write at the end of it, so without this the
* next run would start the pass over from the first item.
*/
import { eq } from "drizzle-orm";
import { db, repositories } from "./db";
import {
serializeRepositoryMetadataState,
type MetadataPassProgress,
type MetadataPassProgressByKind,
type RepositoryMetadataState,
} from "./metadata-state";
export async function persistMetadataPassProgress(
repositoryId: string | null | undefined,
metadataState: RepositoryMetadataState,
kind: keyof MetadataPassProgressByKind,
progress: MetadataPassProgress | undefined
): Promise<void> {
if (progress) {
metadataState.passProgress[kind] = progress;
} else {
delete metadataState.passProgress[kind];
}
if (!repositoryId) return;
try {
await db
.update(repositories)
.set({ metadata: serializeRepositoryMetadataState(metadataState) })
.where(eq(repositories.id, repositoryId));
} catch (error) {
// The in-memory state still carries it for the write at the end of a
// sync that does not throw; only a rate limited run loses it.
console.warn(
`[Metadata] Could not save ${kind} pass progress: ${
error instanceof Error ? error.message : String(error)
}`
);
}
}
+265
View File
@@ -0,0 +1,265 @@
/**
* Incremental metadata sync watermarks (#449): parsing, serialisation and
* the full vs incremental decision.
*/
import { describe, expect, test } from "bun:test";
import {
FULL_METADATA_PASS_INTERVAL_MS,
INCREMENTAL_SYNC_SAFETY_MARGIN_MS,
advanceMetadataSyncCursor,
buildPassProgress,
classifyPassItem,
createDefaultMetadataState,
numberInRanges,
parseRepositoryMetadataState,
planMetadataPass,
planMetadataPassWithProgress,
serializeRepositoryMetadataState,
toNumberRanges,
} from "./metadata-state";
const HOUR = 60 * 60 * 1000;
const DAY = 24 * HOUR;
describe("sync cursor parsing", () => {
test("a row written before #449 loads with no cursors", () => {
const legacy = JSON.stringify({
components: { releases: true, issues: true, pullRequests: true, labels: true, milestones: true },
lastSyncedAt: "2026-09-01T00:00:00.000Z",
});
const state = parseRepositoryMetadataState(legacy);
expect(state.syncCursors).toEqual({});
expect(state.components.issues).toBe(true);
expect(state.lastSyncedAt).toBe("2026-09-01T00:00:00.000Z");
});
test("the default state has no cursors", () => {
expect(createDefaultMetadataState().syncCursors).toEqual({});
expect(parseRepositoryMetadataState(null).syncCursors).toEqual({});
expect(parseRepositoryMetadataState("not json").syncCursors).toEqual({});
});
test("cursors survive a round trip", () => {
const state = createDefaultMetadataState();
state.syncCursors.issues = {
lastPassStartedAt: "2026-09-20T10:00:00.000Z",
lastFullPassStartedAt: "2026-09-18T10:00:00.000Z",
};
state.syncCursors.pullRequests = {
lastPassStartedAt: "2026-09-20T11:00:00.000Z",
lastFullPassStartedAt: "2026-09-20T11:00:00.000Z",
};
const reparsed = parseRepositoryMetadataState(serializeRepositoryMetadataState(state));
expect(reparsed.syncCursors).toEqual(state.syncCursors);
});
test("malformed cursors are dropped instead of failing the row", () => {
const raw = {
components: {},
syncCursors: {
issues: { lastPassStartedAt: "yesterday", lastFullPassStartedAt: "2026-09-18T10:00:00.000Z" },
pullRequests: { lastPassStartedAt: "2026-09-20T11:00:00.000Z" },
},
};
expect(parseRepositoryMetadataState(raw).syncCursors).toEqual({});
expect(parseRepositoryMetadataState({ syncCursors: "nope" }).syncCursors).toEqual({});
expect(parseRepositoryMetadataState({ syncCursors: { issues: 42 } }).syncCursors).toEqual({});
});
});
describe("planMetadataPass", () => {
const now = new Date("2026-09-23T12:00:00.000Z");
test("no cursor means a full pass", () => {
const plan = planMetadataPass(undefined, now);
expect(plan.mode).toBe("full");
});
test("a recent cursor gives an incremental pass with the safety margin", () => {
const lastPass = new Date(now.getTime() - 2 * HOUR);
const plan = planMetadataPass(
{
lastPassStartedAt: lastPass.toISOString(),
lastFullPassStartedAt: new Date(now.getTime() - 3 * DAY).toISOString(),
},
now
);
expect(plan).toEqual({
mode: "incremental",
since: new Date(lastPass.getTime() - INCREMENTAL_SYNC_SAFETY_MARGIN_MS).toISOString(),
});
expect(INCREMENTAL_SYNC_SAFETY_MARGIN_MS).toBeGreaterThanOrEqual(5 * 60 * 1000);
});
test("a full pass older than the interval forces a full pass", () => {
const plan = planMetadataPass(
{
lastPassStartedAt: new Date(now.getTime() - HOUR).toISOString(),
lastFullPassStartedAt: new Date(now.getTime() - FULL_METADATA_PASS_INTERVAL_MS).toISOString(),
},
now
);
expect(plan.mode).toBe("full");
expect(FULL_METADATA_PASS_INTERVAL_MS).toBe(7 * DAY);
});
test("a full pass just inside the interval stays incremental", () => {
const plan = planMetadataPass(
{
lastPassStartedAt: new Date(now.getTime() - HOUR).toISOString(),
lastFullPassStartedAt: new Date(now.getTime() - FULL_METADATA_PASS_INTERVAL_MS + HOUR).toISOString(),
},
now
);
expect(plan.mode).toBe("incremental");
});
test("a watermark in the future is not trusted", () => {
const plan = planMetadataPass(
{
lastPassStartedAt: new Date(now.getTime() + DAY).toISOString(),
lastFullPassStartedAt: now.toISOString(),
},
now
);
expect(plan.mode).toBe("full");
});
});
describe("advanceMetadataSyncCursor", () => {
const started = new Date("2026-09-23T12:00:00.000Z");
test("a full pass moves both timestamps", () => {
expect(advanceMetadataSyncCursor(undefined, "full", started)).toEqual({
lastPassStartedAt: started.toISOString(),
lastFullPassStartedAt: started.toISOString(),
});
});
test("an incremental pass keeps the last full pass time", () => {
const previous = {
lastPassStartedAt: "2026-09-23T06:00:00.000Z",
lastFullPassStartedAt: "2026-09-20T06:00:00.000Z",
};
expect(advanceMetadataSyncCursor(previous, "incremental", started)).toEqual({
lastPassStartedAt: started.toISOString(),
lastFullPassStartedAt: "2026-09-20T06:00:00.000Z",
});
});
});
describe("unfinished pass progress (#449 follow-up)", () => {
const now = new Date("2026-09-25T12:00:00.000Z");
test("numbers collapse into ranges and can be looked up", () => {
const ranges = toNumberRanges([5, 1, 2, 3, 3, 9, 10, 7]);
expect(ranges).toEqual([[1, 3], [5, 5], [7, 7], [9, 10]]);
expect(numberInRanges(2, ranges)).toBe(true);
expect(numberInRanges(8, ranges)).toBe(false);
expect(toNumberRanges([])).toEqual([]);
});
test("progress survives a round trip and bad entries are dropped", () => {
const state = createDefaultMetadataState();
state.passProgress.issues = {
mode: "full",
startedAt: "2026-09-25T10:00:00.000Z",
done: [[1, 164]],
};
const reparsed = parseRepositoryMetadataState(serializeRepositoryMetadataState(state));
expect(reparsed.passProgress).toEqual(state.passProgress);
const junk = parseRepositoryMetadataState({
passProgress: {
issues: { mode: "sideways", startedAt: "2026-09-25T10:00:00.000Z", done: [] },
pullRequests: { mode: "incremental", startedAt: "2026-09-25T10:00:00.000Z", done: [] },
},
});
// Unknown mode, and an incremental pass without its since.
expect(junk.passProgress).toEqual({});
expect(parseRepositoryMetadataState(null).passProgress).toEqual({});
});
test("a stored pass is resumed with its mode, since and start", () => {
const cursor = {
lastPassStartedAt: "2026-09-25T09:00:00.000Z",
lastFullPassStartedAt: "2026-09-20T09:00:00.000Z",
};
const incremental = planMetadataPassWithProgress(
cursor,
{ mode: "incremental", since: "2026-09-25T08:50:00.000Z", startedAt: "2026-09-25T10:00:00.000Z", done: [] },
now
);
expect(incremental.plan).toEqual({ mode: "incremental", since: "2026-09-25T08:50:00.000Z" });
expect(incremental.startedAt.toISOString()).toBe("2026-09-25T10:00:00.000Z");
expect(incremental.resume).toBeDefined();
const full = planMetadataPassWithProgress(
undefined,
{ mode: "full", startedAt: "2026-09-25T10:00:00.000Z", done: [[1, 10]] },
now
);
expect(full.plan.mode).toBe("full");
const none = planMetadataPassWithProgress(cursor, undefined, now);
expect(none.plan.mode).toBe("incremental");
expect(none.resume).toBeUndefined();
expect(none.startedAt).toEqual(now);
});
test("progress stamped in the future is ignored", () => {
const planned = planMetadataPassWithProgress(
undefined,
{ mode: "full", startedAt: "2026-10-25T10:00:00.000Z", done: [[1, 10]] },
now
);
expect(planned.resume).toBeUndefined();
});
test("items are classified by what the pass already knows about them", () => {
const cursor = {
lastPassStartedAt: "2026-09-25T09:00:00.000Z",
lastFullPassStartedAt: "2026-09-17T09:00:00.000Z",
};
const fullPlan = { mode: "full" as const, reason: "test" };
const resume = { mode: "full" as const, startedAt: "2026-09-25T10:00:00.000Z", done: [[1, 5]] as Array<[number, number]> };
const base = { plan: fullPlan, cursor, resume, existsInDestination: true };
expect(classifyPassItem({ ...base, number: 3, updatedAt: "2026-09-24T00:00:00Z" })).toBe("skip");
// Changed after the interrupted pass started: done does not count.
expect(classifyPassItem({ ...base, number: 3, updatedAt: "2026-09-25T11:00:00Z" })).toBe("sync");
// Not done yet, but unchanged since the last completed pass.
expect(classifyPassItem({ ...base, number: 8, updatedAt: "2026-09-24T00:00:00Z" })).toBe("unchanged");
// Inside the safety margin of the last pass.
expect(
classifyPassItem({ ...base, number: 8, updatedAt: new Date(Date.parse(cursor.lastPassStartedAt) - INCREMENTAL_SYNC_SAFETY_MARGIN_MS + 1000).toISOString() })
).toBe("sync");
// Missing in the destination, no watermark, or no updated_at: full treatment.
expect(classifyPassItem({ ...base, number: 8, updatedAt: "2026-09-24T00:00:00Z", existsInDestination: false })).toBe("sync");
expect(classifyPassItem({ ...base, number: 8, updatedAt: "2026-09-24T00:00:00Z", cursor: undefined })).toBe("sync");
expect(classifyPassItem({ ...base, number: 8, updatedAt: null })).toBe("sync");
// An incremental pass never skips on the watermark alone.
expect(
classifyPassItem({ ...base, plan: { mode: "incremental", since: "x" }, resume: undefined, number: 8, updatedAt: "2026-09-24T00:00:00Z" })
).toBe("sync");
});
test("progress keeps what an earlier attempt of the same pass finished", () => {
const earlier = { mode: "full" as const, startedAt: "2026-09-25T10:00:00.000Z", done: [[1, 3]] as Array<[number, number]> };
const progress = buildPassProgress(
{ mode: "full", reason: "resume" },
new Date(earlier.startedAt),
[4, 5, 7],
earlier
);
expect(progress).toEqual({ mode: "full", startedAt: earlier.startedAt, done: [[1, 5], [7, 7]] });
const incremental = buildPassProgress(
{ mode: "incremental", since: "2026-09-25T08:50:00.000Z" },
new Date("2026-09-25T10:00:00.000Z"),
[2],
undefined
);
expect(incremental.since).toBe("2026-09-25T08:50:00.000Z");
});
});
+288
View File
@@ -18,12 +18,80 @@ export interface AcknowledgedDeletion {
giteaSha: string;
}
/**
* Watermark for one metadata kind that can be fetched incrementally
* from GitHub (#449). Both values are the time the GitHub listing
* started, not when the pass finished, so anything that changed while
* a long pass was running is picked up by the next one.
*/
export interface MetadataSyncCursor {
/** Start of the last pass (full or incremental) that completed cleanly. */
lastPassStartedAt: string;
/** Start of the last full pass that completed cleanly. */
lastFullPassStartedAt: string;
}
export interface MetadataSyncCursors {
issues?: MetadataSyncCursor;
pullRequests?: MetadataSyncCursor;
}
/**
* Progress of an issues or pull request pass that did not finish, most
* often because the source ran out of rate limit budget (#449 follow-up).
* The next run repeats the same pass and skips the items listed in
* `done`, as long as they have not changed since `startedAt`, so a pass
* over a large repository gets further every run instead of starting
* over each time.
*/
export interface MetadataPassProgress {
mode: MetadataPassPlan["mode"];
/** Listing start for an incremental pass; repeated as is on resume. */
since?: string;
/**
* When the listing of the first attempt started. Used as the watermark
* once the pass completes, so nothing that changed while it was
* interrupted is missed.
*/
startedAt: string;
/** GitHub numbers finished by this pass, as inclusive ranges. */
done: Array<[number, number]>;
}
export interface MetadataPassProgressByKind {
issues?: MetadataPassProgress;
pullRequests?: MetadataPassProgress;
}
export interface RepositoryMetadataState {
components: MetadataComponentsState;
lastSyncedAt?: string;
acknowledgedDeletions: AcknowledgedDeletion[];
syncCursors: MetadataSyncCursors;
passProgress: MetadataPassProgressByKind;
}
/**
* How far before the stored watermark an incremental listing starts.
* Covers clock skew between this host and GitHub and items whose
* updated_at was written just before the previous listing began.
* Reprocessing a few items twice is harmless: every write is matched
* by its [GH-ISSUE #N] / [PR #N] marker and comments by their id.
*/
export const INCREMENTAL_SYNC_SAFETY_MARGIN_MS = 10 * 60 * 1000;
/**
* A full pass still runs when the last one is older than this. It
* catches what `since` cannot see, such as a label renamed on GitHub
* (which does not touch the issues that carry it) or an issue removed
* on the Gitea side by hand.
*/
export const FULL_METADATA_PASS_INTERVAL_MS = 7 * 24 * 60 * 60 * 1000;
export type MetadataPassPlan =
| { mode: "full"; reason: string }
| { mode: "incremental"; since: string };
const defaultComponents: MetadataComponentsState = {
releases: false,
issues: false,
@@ -36,6 +104,212 @@ export function createDefaultMetadataState(): RepositoryMetadataState {
return {
components: { ...defaultComponents },
acknowledgedDeletions: [],
syncCursors: {},
passProgress: {},
};
}
function isValidTimestamp(value: unknown): value is string {
return typeof value === "string" && Number.isFinite(Date.parse(value));
}
function parseSyncCursor(raw: unknown): MetadataSyncCursor | undefined {
if (!raw || typeof raw !== "object") return undefined;
const { lastPassStartedAt, lastFullPassStartedAt } = raw as {
lastPassStartedAt?: unknown;
lastFullPassStartedAt?: unknown;
};
if (!isValidTimestamp(lastPassStartedAt) || !isValidTimestamp(lastFullPassStartedAt)) {
return undefined;
}
return { lastPassStartedAt, lastFullPassStartedAt };
}
/**
* Decide whether the next issues or pull request pass can ask GitHub
* only for what changed. Without a usable watermark, or when the last
* full pass is too old, the pass is full.
*/
export function planMetadataPass(
cursor: MetadataSyncCursor | undefined,
now: Date = new Date()
): MetadataPassPlan {
if (!cursor) {
return { mode: "full", reason: "no previous completed pass" };
}
const nowMs = now.getTime();
const lastPassMs = Date.parse(cursor.lastPassStartedAt);
const lastFullMs = Date.parse(cursor.lastFullPassStartedAt);
if (!Number.isFinite(lastPassMs) || !Number.isFinite(lastFullMs)) {
return { mode: "full", reason: "stored watermark is not a valid date" };
}
if (lastPassMs > nowMs + INCREMENTAL_SYNC_SAFETY_MARGIN_MS) {
return { mode: "full", reason: "stored watermark is in the future" };
}
if (nowMs - lastFullMs >= FULL_METADATA_PASS_INTERVAL_MS) {
return { mode: "full", reason: "last full pass is older than 7 days" };
}
return {
mode: "incremental",
since: new Date(lastPassMs - INCREMENTAL_SYNC_SAFETY_MARGIN_MS).toISOString(),
};
}
/**
* Collapse item numbers into sorted inclusive ranges, so a pass over
* thousands of issues stores a handful of pairs.
*/
export function toNumberRanges(numbers: Iterable<number>): Array<[number, number]> {
const sorted = [...new Set(numbers)]
.filter((n) => Number.isInteger(n))
.sort((a, b) => a - b);
const ranges: Array<[number, number]> = [];
for (const n of sorted) {
const last = ranges[ranges.length - 1];
if (last && n === last[1] + 1) {
last[1] = n;
} else {
ranges.push([n, n]);
}
}
return ranges;
}
export function numberInRanges(n: number, ranges: Array<[number, number]>): boolean {
return ranges.some(([from, to]) => n >= from && n <= to);
}
function parsePassProgress(raw: unknown): MetadataPassProgress | undefined {
if (!raw || typeof raw !== "object") return undefined;
const { mode, since, startedAt, done } = raw as {
mode?: unknown;
since?: unknown;
startedAt?: unknown;
done?: unknown;
};
if (mode !== "full" && mode !== "incremental") return undefined;
if (!isValidTimestamp(startedAt)) return undefined;
if (mode === "incremental" && !isValidTimestamp(since)) return undefined;
if (!Array.isArray(done)) return undefined;
const ranges = done.filter(
(entry): entry is [number, number] =>
Array.isArray(entry) &&
entry.length === 2 &&
Number.isInteger(entry[0]) &&
Number.isInteger(entry[1]) &&
entry[0] <= entry[1]
);
return mode === "incremental"
? { mode, since: since as string, startedAt, done: ranges }
: { mode, startedAt, done: ranges };
}
/**
* Plan a pass, picking up an unfinished one when there is one. A resumed
* pass keeps its mode and `since`, and reports its original start so the
* watermark stored at the end covers the whole interrupted window.
*/
export function planMetadataPassWithProgress(
cursor: MetadataSyncCursor | undefined,
progress: MetadataPassProgress | undefined,
now: Date = new Date()
): { plan: MetadataPassPlan; startedAt: Date; resume?: MetadataPassProgress } {
const startedAtMs = progress ? Date.parse(progress.startedAt) : NaN;
if (progress && Number.isFinite(startedAtMs) && startedAtMs <= now.getTime()) {
const plan: MetadataPassPlan =
progress.mode === "incremental" && progress.since
? { mode: "incremental", since: progress.since }
: { mode: "full", reason: "resuming a full pass that did not finish" };
return { plan, startedAt: new Date(startedAtMs), resume: progress };
}
return { plan: planMetadataPass(cursor, now), startedAt: now };
}
/**
* Decides how much work an item needs in the current pass.
*
* - "skip": a resumed pass already finished it and it has not changed
* since that pass started.
* - "unchanged": a full pass is revisiting an item that is already in
* Gitea and has not changed since the last completed pass. Its
* comments, commits and files cannot have changed either (each of
* those bumps updated_at), so no per-item source call is needed.
* - "sync": everything else gets the full treatment.
*/
export function classifyPassItem({
number,
updatedAt,
plan,
cursor,
resume,
existsInDestination,
}: {
number: number;
updatedAt: string | null | undefined;
plan: MetadataPassPlan;
cursor: MetadataSyncCursor | undefined;
resume: MetadataPassProgress | undefined;
existsInDestination: boolean;
}): "skip" | "unchanged" | "sync" {
const updatedMs = updatedAt ? Date.parse(updatedAt) : NaN;
if (!Number.isFinite(updatedMs)) return "sync";
if (resume && numberInRanges(number, resume.done)) {
const resumeStartMs = Date.parse(resume.startedAt);
if (updatedMs < resumeStartMs - INCREMENTAL_SYNC_SAFETY_MARGIN_MS) {
return "skip";
}
}
if (plan.mode === "full" && cursor && existsInDestination) {
const lastPassMs = Date.parse(cursor.lastPassStartedAt);
if (
Number.isFinite(lastPassMs) &&
updatedMs < lastPassMs - INCREMENTAL_SYNC_SAFETY_MARGIN_MS
) {
return "unchanged";
}
}
return "sync";
}
/**
* Progress to store for a pass that did not finish. Numbers finished by
* an earlier attempt of the same pass are kept.
*/
export function buildPassProgress(
plan: MetadataPassPlan,
startedAt: Date,
completed: Iterable<number>,
resume: MetadataPassProgress | undefined
): MetadataPassProgress {
const numbers = new Set<number>(completed);
for (const [from, to] of resume?.done ?? []) {
for (let n = from; n <= to; n++) numbers.add(n);
}
return {
mode: plan.mode,
...(plan.mode === "incremental" ? { since: plan.since } : {}),
startedAt: startedAt.toISOString(),
done: toNumberRanges(numbers),
};
}
/**
* The watermark to store after a pass that completed without failures.
* `startedAt` is the time the GitHub listing started.
*/
export function advanceMetadataSyncCursor(
previous: MetadataSyncCursor | undefined,
mode: MetadataPassPlan["mode"],
startedAt: Date
): MetadataSyncCursor {
const started = startedAt.toISOString();
return {
lastPassStartedAt: started,
lastFullPassStartedAt:
mode === "full" || !previous ? started : previous.lastFullPassStartedAt,
};
}
@@ -93,6 +367,20 @@ export function parseRepositoryMetadataState(
);
}
if (parsed.syncCursors && typeof parsed.syncCursors === "object") {
const issues = parseSyncCursor(parsed.syncCursors.issues);
const pullRequests = parseSyncCursor(parsed.syncCursors.pullRequests);
if (issues) base.syncCursors.issues = issues;
if (pullRequests) base.syncCursors.pullRequests = pullRequests;
}
if (parsed.passProgress && typeof parsed.passProgress === "object") {
const issues = parsePassProgress(parsed.passProgress.issues);
const pullRequests = parsePassProgress(parsed.passProgress.pullRequests);
if (issues) base.passProgress.issues = issues;
if (pullRequests) base.passProgress.pullRequests = pullRequests;
}
return base;
}
+125
View File
@@ -0,0 +1,125 @@
/**
* The request wrap that holds every GitHub call while the gate is closed
* (issue #437, second report). Tested against a hand made hook so the
* clock and the sleep are under control.
*/
import { beforeEach, describe, expect, test } from "bun:test";
import {
RateLimitPausedError,
clearAllRateLimits,
installRateLimitGate,
isRateLimitError,
markRateLimited,
} from "@/lib/rate-limit-gate";
type Wrapper = (request: (o: any) => Promise<any>, options: any) => Promise<any>;
function fakeOctokit() {
const holder: { wrapper?: Wrapper } = {};
return {
holder,
octokit: {
hook: {
wrap: (_name: string, wrapper: Wrapper) => {
holder.wrapper = wrapper;
},
},
},
};
}
describe("installRateLimitGate", () => {
const key = "user-gate";
const base = new Date("2026-09-24T10:00:00.000Z");
beforeEach(() => {
clearAllRateLimits();
});
test("an open gate passes the request straight through", async () => {
const { octokit, holder } = fakeOctokit();
const sleeps: number[] = [];
installRateLimitGate(octokit, { key, maxWaitMs: 120_000, sleep: async (ms) => void sleeps.push(ms), now: () => base });
const sent: any[] = [];
const result = await holder.wrapper!(async (o) => { sent.push(o); return "ok"; }, { method: "GET", url: "/repos/a/b" });
expect(result).toBe("ok");
expect(sent).toHaveLength(1);
expect(sleeps).toEqual([]);
});
test("a long pause fails the request without calling the inner request", async () => {
const { octokit, holder } = fakeOctokit();
installRateLimitGate(octokit, { key, maxWaitMs: 120_000, sleep: async () => {}, now: () => base });
markRateLimited(key, base.getTime() + 3_263_000);
let inner = 0;
let caught: unknown;
try {
await holder.wrapper!(async () => { inner += 1; return "no"; }, { method: "GET", url: "/repos/a/b/issues" });
} catch (error) {
caught = error;
}
expect(inner).toBe(0);
expect(caught).toBeInstanceOf(RateLimitPausedError);
expect((caught as RateLimitPausedError).status).toBe(403);
expect((caught as RateLimitPausedError).message).toContain("paused until 2026-09-24T10:54:23.000Z");
expect((caught as RateLimitPausedError).message).toContain("GET /repos/a/b/issues");
// The rest of the code base classifies it like GitHub's own refusal.
expect(isRateLimitError(caught)).toBe(true);
});
test("a short pause is slept off with a one second cushion, then the request goes out", async () => {
const { octokit, holder } = fakeOctokit();
const sleeps: number[] = [];
installRateLimitGate(octokit, { key, maxWaitMs: 120_000, sleep: async (ms) => void sleeps.push(ms), now: () => base });
markRateLimited(key, base.getTime() + 45_000);
let inner = 0;
const result = await holder.wrapper!(async () => { inner += 1; return "sent"; }, { method: "GET", url: "/repos/a/b" });
expect(result).toBe("sent");
expect(inner).toBe(1);
expect(sleeps).toEqual([46_000]);
});
test("exactly the threshold still waits; one millisecond more fails", async () => {
const { octokit, holder } = fakeOctokit();
const sleeps: number[] = [];
installRateLimitGate(octokit, { key, maxWaitMs: 120_000, sleep: async (ms) => void sleeps.push(ms), now: () => base });
markRateLimited(key, base.getTime() + 120_000);
await holder.wrapper!(async () => "a", { method: "GET", url: "/x" });
expect(sleeps).toEqual([121_000]);
clearAllRateLimits();
markRateLimited(key, base.getTime() + 120_001);
await expect(holder.wrapper!(async () => "b", { method: "GET", url: "/x" })).rejects.toBeInstanceOf(RateLimitPausedError);
});
test("the rate limit probe is exempt so the pause can be checked", async () => {
const { octokit, holder } = fakeOctokit();
installRateLimitGate(octokit, { key, maxWaitMs: 120_000, sleep: async () => {}, now: () => base });
markRateLimited(key, base.getTime() + 3_600_000);
let inner = 0;
await holder.wrapper!(async () => { inner += 1; return "probe"; }, { method: "GET", url: "/rate_limit" });
expect(inner).toBe(1);
await expect(holder.wrapper!(async () => "no", { method: "GET", url: "/rate_limited_looking/but/not" })).rejects.toBeInstanceOf(RateLimitPausedError);
});
test("keys are independent: another user's pause does not hold this client", async () => {
const { octokit, holder } = fakeOctokit();
installRateLimitGate(octokit, { key, maxWaitMs: 120_000, sleep: async () => {}, now: () => base });
markRateLimited("someone-else", base.getTime() + 3_600_000);
let inner = 0;
await holder.wrapper!(async () => { inner += 1; return "ok"; }, { method: "GET", url: "/repos/a/b" });
expect(inner).toBe(1);
});
test("a client without a hook system is left alone", () => {
expect(() => installRateLimitGate({}, { key, maxWaitMs: 1 })).not.toThrow();
expect(() => installRateLimitGate(undefined, { key, maxWaitMs: 1 })).not.toThrow();
});
});
+81
View File
@@ -112,3 +112,84 @@ export function isRateLimitError(error: unknown): boolean {
// all that is left to go on.
return status === undefined || status === 403;
}
/**
* The error a gated request fails with before anything is sent. Shaped like
* Octokit's own primary rate limit refusal (status 403, "rate limit" in the
* message) so every existing `isRateLimitError` check treats it the same.
*/
export class RateLimitPausedError extends Error {
readonly status = 403;
readonly resetAt: Date;
constructor(resetAt: Date, endpoint?: string) {
super(
`GitHub API rate limit exceeded; requests are paused until ${resetAt.toISOString()}` +
(endpoint ? ` (not sent: ${endpoint})` : "")
);
this.name = "RateLimitPausedError";
this.resetAt = resetAt;
}
}
/**
* Endpoints that never count against the limit and are needed to learn when
* it opens again. Everything else is held while the gate is closed.
*/
const GATE_EXEMPT_ROUTES = [/\/rate_limit(\?|$)/];
export interface RateLimitGateOptions {
/** Gate key of this client, from rateLimitGateKey(). */
key: string;
/**
* A closed gate that opens again within this many milliseconds is waited
* out inside the request instead of failing it, so a short primary or
* secondary limit stays invisible to the caller. Longer waits fail at once.
*/
maxWaitMs: number;
/** Injectable for tests. */
sleep?: (ms: number) => Promise<void>;
now?: () => Date;
}
/**
* Hold every request of an Octokit client while its gate is closed
* (issue #437, second report).
*
* The throttle handler records a reset time when a request is refused, but
* on its own that only affects the one request: every other call queued in
* the same run still went to GitHub and came back 403, and the retry helper
* sent each of them three more times. GitHub counts requests made while
* limited toward abuse detection, and one reporter's account was suspended
* that way. This wrap runs before the network: a closed gate either waits
* (short reset) or throws (long reset), and nothing leaves the process.
*/
export function installRateLimitGate(octokit: unknown, options: RateLimitGateOptions): void {
const hook = (octokit as { hook?: { wrap?: unknown } })?.hook;
if (typeof hook?.wrap !== "function") return;
const sleep = options.sleep ?? ((ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms)));
const now = options.now ?? (() => new Date());
(hook.wrap as (name: string, wrapper: (request: any, requestOptions: any) => Promise<any>) => void)(
"request",
async (request, requestOptions) => {
const url = String(requestOptions?.url ?? "");
if (GATE_EXEMPT_ROUTES.some((route) => route.test(url))) {
return request(requestOptions);
}
const until = rateLimitedUntil(options.key, now());
if (until) {
const waitMs = until.getTime() - now().getTime();
if (waitMs > options.maxWaitMs) {
throw new RateLimitPausedError(until, `${requestOptions?.method ?? "GET"} ${url}`);
}
// Small cushion: GitHub's reset is second precision.
await sleep(waitMs + 1000);
}
return request(requestOptions);
}
);
}
+40
View File
@@ -0,0 +1,40 @@
import { describe, expect, test } from "bun:test";
import { countLatestStatuses, latestStatusBySubject } from "./activity-status";
const at = (iso: string) => new Date(iso);
describe("Activity Log status chips (#454)", () => {
test("a start and finish in the same second count as finished, in either order", () => {
const start = { repositoryId: "r1", status: "syncing", timestamp: at("2026-09-26T00:29:48.000Z") };
const finish = { repositoryId: "r1", status: "synced", timestamp: at("2026-09-26T00:29:48.000Z") };
const aggregate = { status: "mirrored", timestamp: at("2026-09-26T00:29:48.000Z") };
// The order the reporter's export came back in.
expect(countLatestStatuses([aggregate, start, finish])).toEqual(new Map([["synced", 1]]));
expect(countLatestStatuses([finish, start])).toEqual(new Map([["synced", 1]]));
});
test("a sync that is really still running stays syncing", () => {
const earlier = { repositoryId: "r1", status: "synced", timestamp: at("2026-09-26T00:00:00.000Z") };
const running = { repositoryId: "r1", status: "syncing", timestamp: at("2026-09-26T01:00:00.000Z") };
expect(latestStatusBySubject([running, earlier]).get("r1")).toBe("syncing");
});
test("the newest event decides otherwise, and events without a subject are ignored", () => {
const counts = countLatestStatuses([
{ repositoryId: "r1", status: "synced", timestamp: "2026-09-26T02:00:00Z" },
{ repositoryId: "r1", status: "failed", timestamp: "2026-09-26T01:00:00Z" },
{ organizationId: "o1", status: "failed", timestamp: "2026-09-26T01:00:00Z" },
{ status: "mirrored", timestamp: "2026-09-26T01:00:00Z" },
]);
expect(counts).toEqual(new Map([["synced", 1], ["failed", 1]]));
});
test("a newer start never loses to an older finish from another second", () => {
const counts = countLatestStatuses([
{ repositoryName: "a", status: "mirroring", timestamp: "2026-09-26T00:00:02Z" },
{ repositoryName: "a", status: "mirrored", timestamp: "2026-09-26T00:00:01Z" },
]);
expect(counts).toEqual(new Map([["mirroring", 1]]));
});
});
+62
View File
@@ -0,0 +1,62 @@
/**
* Current state per repository or organization for the Activity Log
* summary chips. Activities arrive newest first, so the first event seen
* for a subject is its state.
*
* Timestamps are stored to the second, and a quick sync writes its
* "syncing" and "synced" events in the same second (#454). When the newest
* event for a subject says work started, an event from the same second
* that says it finished wins, whatever order the tie came back in.
*/
const IN_PROGRESS_STATUSES = new Set(["syncing", "mirroring", "deleting"]);
interface ActivityLike {
status: string;
timestamp: Date | string | number;
repositoryId?: string | null;
organizationId?: string | null;
repositoryName?: string | null;
organizationName?: string | null;
}
function toSecond(timestamp: ActivityLike["timestamp"]): number {
return Math.floor(new Date(timestamp).getTime() / 1000);
}
export function latestStatusBySubject(activities: ActivityLike[]): Map<string, string> {
const latest = new Map<string, { status: string; second: number }>();
for (const activity of activities) {
const subject =
activity.repositoryId ||
activity.organizationId ||
activity.repositoryName ||
activity.organizationName;
// Events with no subject have no state of their own to report.
if (!subject) continue;
const second = toSecond(activity.timestamp);
const seen = latest.get(subject);
if (!seen) {
latest.set(subject, { status: activity.status, second });
continue;
}
if (
IN_PROGRESS_STATUSES.has(seen.status) &&
!IN_PROGRESS_STATUSES.has(activity.status) &&
seen.second === second
) {
latest.set(subject, { status: activity.status, second });
}
}
return new Map([...latest].map(([subject, { status }]) => [subject, status]));
}
export function countLatestStatuses(activities: ActivityLike[]): Map<string, number> {
const counts = new Map<string, number>();
latestStatusBySubject(activities).forEach((status) => {
counts.set(status, (counts.get(status) ?? 0) + 1);
});
return counts;
}
@@ -0,0 +1,91 @@
/**
* processWithRetry and a rate limited source (issue #437, second report).
* A refusal used to be retried three times with backoff like any other
* error, and the items behind it were started anyway, so one exhausted
* budget turned into thousands of 403 responses.
*/
import { describe, expect, test } from "bun:test";
import { RateLimitedBatchError, processWithRetry } from "./concurrency";
import { isRateLimitError } from "@/lib/rate-limit-gate";
function rateLimitRefusal() {
const error = new Error("API rate limit exceeded for user ID 1.") as Error & { status: number };
error.status = 403;
return error;
}
describe("processWithRetry under a rate limit", () => {
test("a rate limit refusal is not retried and the remaining items are not started", async () => {
const attempts: number[] = [];
let caught: unknown;
try {
await processWithRetry(
[1, 2, 3, 4, 5, 6],
async (n) => {
attempts.push(n);
if (n === 2) throw rateLimitRefusal();
return n * 10;
},
{ concurrencyLimit: 2, maxRetries: 3, retryDelay: 1 }
);
} catch (error) {
caught = error;
}
expect(caught).toBeInstanceOf(RateLimitedBatchError);
const failure = caught as RateLimitedBatchError<number>;
// Item 2 was attempted exactly once; items 3 to 6 never started. The two
// items of the first batch run concurrently, so their order is not fixed.
expect([...attempts].sort()).toEqual([1, 2]);
expect(failure.results).toEqual([10]);
expect(failure.message).toContain("5 item(s) left");
expect(failure.status).toBe(403);
// Callers up the stack classify it like the refusal that caused it.
expect(isRateLimitError(failure)).toBe(true);
expect(isRateLimitError(failure.cause)).toBe(true);
});
test("a 429 from a paginated call counts as a rate limit too", async () => {
const attempts: number[] = [];
const tooMany = Object.assign(new Error("Too many requests"), { status: 429 });
await expect(
processWithRetry(
[1, 2, 3],
async (n) => {
attempts.push(n);
if (n === 1) throw tooMany;
return n;
},
{ concurrencyLimit: 1, maxRetries: 3, retryDelay: 1 }
)
).rejects.toBeInstanceOf(RateLimitedBatchError);
expect(attempts).toEqual([1]);
});
test("other errors keep their retries and do not stop the batch", async () => {
const attempts: number[] = [];
const results = await processWithRetry(
[1, 2, 3],
async (n) => {
attempts.push(n);
if (n === 2 && attempts.filter((a) => a === 2).length < 3) {
throw new Error("Gitea hiccup");
}
return n;
},
{ concurrencyLimit: 1, maxRetries: 3, retryDelay: 1 }
);
expect(results).toEqual([1, 2, 3]);
expect(attempts).toEqual([1, 2, 2, 2, 3]);
});
test("a batch that finishes without a refusal returns every result as before", async () => {
const results = await processWithRetry([1, 2, 3], async (n) => n + 1, {
concurrencyLimit: 3,
maxRetries: 0,
retryDelay: 1,
});
expect(results).toEqual([2, 3, 4]);
});
});
+56 -2
View File
@@ -1,4 +1,28 @@
import { JOB_HEARTBEAT_INTERVAL_MS } from "@/lib/interrupted-job-detection";
import { isRateLimitError } from "@/lib/rate-limit-gate";
/**
* Thrown by processWithRetry when a batch stopped because the source is rate
* limited. Carries the results gathered so far so a caller can still count
* what was done. Shaped like a GitHub refusal so isRateLimitError matches it.
*/
export class RateLimitedBatchError<R = unknown> extends Error {
readonly status = 403;
declare readonly results: R[];
declare readonly cause: unknown;
constructor(cause: unknown, results: R[], remaining: number) {
super(
`Rate limit hit while processing items; stopped with ${remaining} item(s) left ` +
`(${cause instanceof Error ? cause.message : String(cause)})`
);
this.name = "RateLimitedBatchError";
// Not enumerable, so logging the error does not print every partial
// result and the whole source response it carries.
Object.defineProperty(this, "cause", { value: cause, enumerable: false });
Object.defineProperty(this, "results", { value: results, enumerable: false });
}
}
/**
* Utility for processing items in parallel with concurrency control
@@ -13,7 +37,12 @@ export async function processInParallel<T, R>(
items: T[],
processItem: (item: T) => Promise<R>,
concurrencyLimit: number = 5, // Safe default for GitHub API (max 100 concurrent, but 5-10 recommended)
onProgress?: (completed: number, total: number, result?: R) => void
onProgress?: (completed: number, total: number, result?: R) => void,
/**
* Asked before each batch. Returning true ends the loop with the results
* gathered so far; the remaining items are not started.
*/
shouldStop?: () => boolean
): Promise<R[]> {
const results: R[] = [];
let completed = 0;
@@ -21,6 +50,11 @@ export async function processInParallel<T, R>(
// Process items in batches to control concurrency
for (let i = 0; i < total; i += concurrencyLimit) {
if (shouldStop?.()) {
console.warn(`[Batch] Stopping before item ${i} of ${total}: source is rate limited`);
break;
}
const batch = items.slice(i, i + concurrencyLimit);
const batchPromises = batch.map(async (item) => {
@@ -119,6 +153,11 @@ export async function processWithRetry<T, R>(
// Track checkpoint counter
let itemsProcessedSinceLastCheckpoint = 0;
// The first rate limit refusal ends the run: retrying it, or starting the
// items behind it, would only send more requests into the same wall
// (issue #437). The caller gets a RateLimitedBatchError with what finished.
let rateLimitError: unknown = null;
// Wrap the process function with retry logic
const processWithRetryLogic = async (item: T): Promise<R> => {
let lastError: Error | null = null;
@@ -153,6 +192,16 @@ export async function processWithRetry<T, R>(
} catch (error) {
lastError = error instanceof Error ? error : new Error(String(error));
if (isRateLimitError(error)) {
if (!rateLimitError) {
rateLimitError = error;
console.warn(
`[Batch] Rate limit hit on ${getItemId ? getItemId(item) : "an item"}; not retrying and not starting the remaining items`
);
}
throw lastError;
}
if (attempt <= maxRetries) {
if (onRetry) {
onRetry(item, lastError, attempt);
@@ -190,9 +239,14 @@ export async function processWithRetry<T, R>(
items,
processWithRetryLogic,
concurrencyLimit,
onProgress
onProgress,
() => rateLimitError !== null
);
if (rateLimitError) {
throw new RateLimitedBatchError(rateLimitError, results, items.length - results.length);
}
return results;
}
+127
View File
@@ -0,0 +1,127 @@
import { afterEach, beforeEach, describe, expect, it } from "bun:test";
import {
DEFAULT_ORGANIZATIONS_VIEW,
ORGANIZATIONS_VIEW_CHANGE_EVENT,
ORGANIZATIONS_VIEW_STORAGE_KEY,
getOrganizationsView,
isOrganizationsViewMode,
setOrganizationsView,
subscribeToOrganizationsViewChange,
} from "./organizations-view";
// A minimal window and localStorage so the helper can be exercised
// without a DOM library. Both are removed again afterwards.
function installBrowserGlobals() {
const store = new Map<string, string>();
const listeners = new Map<string, Set<(event: any) => void>>();
(globalThis as any).localStorage = {
getItem: (key: string) => store.get(key) ?? null,
setItem: (key: string, value: string) => void store.set(key, value),
removeItem: (key: string) => void store.delete(key),
};
(globalThis as any).window = {
addEventListener: (type: string, cb: (event: any) => void) => {
if (!listeners.has(type)) listeners.set(type, new Set());
listeners.get(type)!.add(cb);
},
removeEventListener: (type: string, cb: (event: any) => void) => {
listeners.get(type)?.delete(cb);
},
dispatchEvent: (event: any) => {
listeners.get(event.type)?.forEach((cb) => cb(event));
return true;
},
};
if (typeof (globalThis as any).CustomEvent === "undefined") {
(globalThis as any).CustomEvent = class CustomEvent {
type: string;
detail: unknown;
constructor(type: string, init?: { detail?: unknown }) {
this.type = type;
this.detail = init?.detail;
}
};
}
return { store, listeners };
}
describe("organizations view preference", () => {
const hadWindow = typeof (globalThis as any).window !== "undefined";
const hadStorage = typeof (globalThis as any).localStorage !== "undefined";
const savedWindow = (globalThis as any).window;
const savedStorage = (globalThis as any).localStorage;
beforeEach(() => {
delete (globalThis as any).window;
delete (globalThis as any).localStorage;
});
afterEach(() => {
if (hadWindow) (globalThis as any).window = savedWindow;
else delete (globalThis as any).window;
if (hadStorage) (globalThis as any).localStorage = savedStorage;
else delete (globalThis as any).localStorage;
});
it("falls back to cards without a browser", () => {
expect(getOrganizationsView()).toBe(DEFAULT_ORGANIZATIONS_VIEW);
expect(DEFAULT_ORGANIZATIONS_VIEW).toBe("cards");
// Setting without a window must not throw either.
expect(() => setOrganizationsView("list")).not.toThrow();
});
it("only accepts the two known modes", () => {
expect(isOrganizationsViewMode("cards")).toBe(true);
expect(isOrganizationsViewMode("list")).toBe(true);
expect(isOrganizationsViewMode("table")).toBe(false);
expect(isOrganizationsViewMode(null)).toBe(false);
});
it("persists the choice and ignores garbage in storage", () => {
const { store } = installBrowserGlobals();
expect(getOrganizationsView()).toBe("cards");
setOrganizationsView("list");
expect(store.get(ORGANIZATIONS_VIEW_STORAGE_KEY)).toBe("list");
expect(getOrganizationsView()).toBe("list");
store.set(ORGANIZATIONS_VIEW_STORAGE_KEY, "sideways");
expect(getOrganizationsView()).toBe("cards");
});
it("notifies subscribers in this tab and from the storage event", () => {
const { listeners } = installBrowserGlobals();
let calls = 0;
const unsubscribe = subscribeToOrganizationsViewChange(() => {
calls += 1;
});
setOrganizationsView("list");
expect(calls).toBe(1);
// Another tab changed it.
listeners.get("storage")?.forEach((cb) => cb({ key: ORGANIZATIONS_VIEW_STORAGE_KEY }));
expect(calls).toBe(2);
// An unrelated key does nothing.
listeners.get("storage")?.forEach((cb) => cb({ key: "timeFormat" }));
expect(calls).toBe(2);
unsubscribe();
(globalThis as any).window.dispatchEvent(new CustomEvent(ORGANIZATIONS_VIEW_CHANGE_EVENT));
expect(calls).toBe(2);
});
it("swallows a storage that throws", () => {
installBrowserGlobals();
(globalThis as any).localStorage = {
getItem: () => {
throw new Error("blocked");
},
setItem: () => {
throw new Error("blocked");
},
};
expect(getOrganizationsView()).toBe("cards");
expect(() => setOrganizationsView("list")).not.toThrow();
});
});
+55
View File
@@ -0,0 +1,55 @@
/**
* Cards or list on the Organizations page (#428).
*
* A pure display preference, so it lives in localStorage next to the theme
* and time format preferences rather than in the database. Reading is safe
* during SSR and in tests, where localStorage does not exist.
*/
export type OrganizationsViewMode = "cards" | "list";
export const ORGANIZATIONS_VIEW_STORAGE_KEY = "organizationsView";
export const ORGANIZATIONS_VIEW_CHANGE_EVENT = "gitea-mirror:organizations-view-change";
export const DEFAULT_ORGANIZATIONS_VIEW: OrganizationsViewMode = "cards";
export function isOrganizationsViewMode(value: unknown): value is OrganizationsViewMode {
return value === "cards" || value === "list";
}
export function getOrganizationsView(): OrganizationsViewMode {
if (typeof localStorage === "undefined") return DEFAULT_ORGANIZATIONS_VIEW;
try {
const stored = localStorage.getItem(ORGANIZATIONS_VIEW_STORAGE_KEY);
return isOrganizationsViewMode(stored) ? stored : DEFAULT_ORGANIZATIONS_VIEW;
} catch {
return DEFAULT_ORGANIZATIONS_VIEW;
}
}
export function setOrganizationsView(mode: OrganizationsViewMode): void {
try {
if (typeof localStorage !== "undefined") {
localStorage.setItem(ORGANIZATIONS_VIEW_STORAGE_KEY, mode);
}
} catch {
// Persisting is best effort; listeners are still told.
}
if (typeof window !== "undefined") {
window.dispatchEvent(new CustomEvent(ORGANIZATIONS_VIEW_CHANGE_EVENT, { detail: mode }));
}
}
/** Subscribe to changes from this tab and from other tabs. */
export function subscribeToOrganizationsViewChange(callback: () => void): () => void {
if (typeof window === "undefined") return () => {};
const onStorage = (event: StorageEvent) => {
if (event.key === null || event.key === ORGANIZATIONS_VIEW_STORAGE_KEY) callback();
};
window.addEventListener(ORGANIZATIONS_VIEW_CHANGE_EVENT, callback);
window.addEventListener("storage", onStorage);
return () => {
window.removeEventListener(ORGANIZATIONS_VIEW_CHANGE_EVENT, callback);
window.removeEventListener("storage", onStorage);
};
}
+3 -1
View File
@@ -17,7 +17,9 @@ export const GET: APIRoute = async ({ request, locals }) => {
.select()
.from(mirrorJobs)
.where(eq(mirrorJobs.userId, userId))
.orderBy(sql`${mirrorJobs.timestamp} DESC`);
// Timestamps are stored to the second; rowid keeps events written in
// the same second newest first too (#454).
.orderBy(sql`${mirrorJobs.timestamp} DESC, ${mirrorJobs}.rowid DESC`);
const activities: MirrorJob[] = jobs.map((job) => ({
id: job.id,
+3 -1
View File
@@ -36,7 +36,9 @@ export const GET: APIRoute = async ({ request, locals }) => {
.select()
.from(mirrorJobs)
.where(eq(mirrorJobs.userId, userId))
.orderBy(sql`${mirrorJobs.timestamp} DESC`)
// Timestamps are stored to the second; rowid keeps events written in
// the same second newest first too (#454).
.orderBy(sql`${mirrorJobs.timestamp} DESC, ${mirrorJobs}.rowid DESC`)
.limit(10),
db
.select()
+3
View File
@@ -76,6 +76,9 @@ export const POST: APIRoute = async ({ request, locals }) => {
);
}
// Clearing the whole state also drops the incremental sync
// watermarks (#449), so the next issues and pull request passes are
// full ones.
await db
.update(repositories)
.set({
+4
View File
@@ -5,6 +5,10 @@ dist/
# dependencies
node_modules/
# pnpm-lock.yaml is the lockfile. Cloudflare Pages installs with Bun when a
# bun.lock is present, and a stale one fails the build.
bun.lock
bun.lockb
# logs
npm-debug.log*
+6 -6
View File
@@ -11,18 +11,18 @@ This is the marketing website for Gitea Mirror, built with Astro and Tailwind CS
## Essential Commands
```bash
bun install # Install dependencies
bun run dev # Start development server (port 4321)
bun run build # Build for production
bun run preview # Preview production build
pnpm install # Install dependencies (pnpm-lock.yaml is the lockfile)
pnpm dev # Start development server (port 4321)
pnpm build # Build for production
pnpm preview # Preview production build
```
## Architecture & Key Concepts
### Technology Stack
- **Framework**: Astro (v5.0.5) - Static site generator with React integration
- **Framework**: Astro (v7) - Static site generator with React integration
- **UI**: React (v19.0.0) + Tailwind CSS v4
- **Runtime**: Bun
- **Package manager**: pnpm. Do not commit a bun.lock: Cloudflare Pages installs with Bun when it finds one
- **Styling**: Tailwind CSS v4 with Vite plugin
### Project Structure
+4 -4
View File
@@ -5,8 +5,8 @@ This Astro workspace powers the public marketing experience for Gitea Mirror. It
## Developing Locally
```bash
bun install
bun run dev
pnpm install
pnpm dev
```
The site is available at `http://localhost:4321`. Tailwind CSS v4 handles styling; classes can be used directly inside Astro, MDX, and React components.
@@ -23,8 +23,8 @@ The site is available at `http://localhost:4321`. Tailwind CSS v4 handles stylin
1. Add or update a record in `src/lib/use-cases.ts`. This keeps the landing page and library listing in sync.
2. Create a new MDX file in `src/pages/use-cases/<slug>.mdx` with the `UseCaseLayout` layout and descriptive frontmatter.
3. Run `bun run dev` to preview the layout and ensure the new guide inherits global styles.
3. Run `pnpm dev` to preview the layout and ensure the new guide inherits global styles.
## Deployment
The marketing site is built with the standard Astro pipeline. Use `bun run build` to generate a production build before deploying.
The marketing site is built with the standard Astro pipeline. Use `pnpm build` to generate a production build before deploying.
-1037
View File
File diff suppressed because it is too large. Load diff
+10
View File
@@ -152,6 +152,16 @@ Throughout, job status updates are published as server-sent events so the dashbo
Gitea's API cannot create a pull request through the migration path, so mirrored pull requests are created as issues instead. Each one is tagged with a `pull-request` label, its title is prefixed with the original number and status, and its body includes the commit history, changed files, and merge outcome. This is a deliberate workaround for an API limitation, not a temporary gap.
### Incremental issue and pull request sync
The first issues pass and the first pull request pass list everything on GitHub. After a pass completes, the repository's metadata state records when its listing started, and later passes ask GitHub only for items updated since then, minus a 10 minute safety margin. New comments, label, state, milestone, title and body changes all move an item's `updated_at`, so only those items are reconciled and only their comments, pull request details, commits and files are fetched. Pull requests are listed through the issues endpoint for this, because the pull request list has no `since` filter.
A full pass still runs when the last one is more than 7 days old, when the destination has no mirrored issues or pull requests (for example after it was recreated), and after Reset metadata. If any item fails, the watermark is not moved, so the next pass lists it again. Items deleted on GitHub are left on the destination, as before.
A full pass lists every issue and pull request, but only fetches comments, pull request details, commits and files for the items that are missing on the destination or changed since the last completed pass. Everything else is corrected from the listing alone, so a weekly full pass over a large tracker costs about one request per hundred items.
When a pass stops because the source is rate limited, the items it finished are saved with the repository. The next run repeats the same pass and skips them unless they changed in the meantime, so a pass that needs more than one hour of budget still completes over several runs. Once it completes, the watermark is set to the start of its first attempt.
### Ordering and concurrency
Gitea assigns issue numbers on creation, so processing issues in parallel can produce numbering that does not match GitHub's. The concurrency settings default to 3 for issues and 5 for pull requests, trading some ordering fidelity for speed. Set both to 1 if you need the numbering to line up.
@@ -33,6 +33,10 @@ Gitea Mirror uses this on every GET. Each response's ETag is cached; every subse
The practical effect is that sync frequency mostly stops mattering. A `30m` interval on a stable account costs barely more than a `24h` one, because 47 of the 48 daily sweeps are nearly all cache hits. The requests you spend scale with how much changed, not how often you look.
Issues and pull requests go one step further. After the first complete pass, a scheduled sync asks GitHub only for the issues and pull requests updated since the previous one, and fetches comments, commits and changed files for those alone. A tracker with thousands of issues and a handful of changes costs a handful of requests. A full pass still runs once a week to catch anything the incremental listing cannot see, and it only fetches details for items that are missing or changed. A pass that runs out of budget picks up where it stopped after the reset.
When the limit is hit anyway, nothing else is sent. Every GitHub client pauses until the reset GitHub reported: a reset within two minutes is waited out inside the request, a longer one fails the request at once without contacting GitHub, and the run ends so the next scheduled one starts a minute after the reset. Repositories that failed only because of the limit go back to their previous status and are retried then. This matters because GitHub counts requests made while limited toward abuse detection, which can suspend the account.
## Configuration that stays inside the budget
A few settings still matter for large or busy accounts: