Compare commits

...
Author SHA1 Message Date
Arunavo Ray 32bdae8d2a feat: incremental issue and pull request sync (#449)
After a complete pass, the issues and pull request passes list only the
items GitHub reports as updated since the previous listing (minus a 10
minute margin), so per-issue comment listing and per-PR detail, commits
and files calls run only for what changed. Pull requests use the issues
endpoint for this since pulls.list has no since filter.

The watermark lives in the repository metadata state as syncCursors, one
per kind, and records when the listing started. It only moves when the
pass finishes with no failed item. A full pass still runs when the last
one is older than 7 days, when the destination has no mirrored issues or
pull requests, and after reset metadata, which clears the state.
2026-09-23 22:57:11 +05:30
Arunavo Ray 8ae778d708 chore: bump version to 3.38.2 2026-09-23 18:27:26 +05:30
ARUNAVO RAY 32e40c9262 fix(security): apply the outbound guard to source and destination URLs (#448)
Fixes GHSA-p7w3-46pg-mv6h. The 3.38.1 guard covered the connection tests,
OIDC discovery and the notification providers, but a source URL stored
through POST /api/sources, PUT /api/sources/:id or the configuration save
was only checked for its scheme, and the source fetch helper called fetch
directly with the upstream body in the error message. The organization
sync route then forwarded that message when it matched a safe pattern.

- The three routes refuse link local and metadata addresses for source and
  destination URLs with the same message the connection tests give.
- sourceFetch goes through safeFetch: refused addresses, plain http pinned
  to the checked address, no redirects.
- SourceApiError messages carry the URL and status only; the body stays on
  the error and is logged.
- createSecureErrorResponse never forwards SourceApiError or HttpError
  messages, whatever words they contain.
2026-09-23 18:27:22 +05:30
Arunavo Ray c2bbcdf6ab chore: bump version to 3.38.1 2026-09-23 01:53:19 +05:30
ARUNAVO RAY c6157cdaad fix(security): guard connection tests, close sign-up and scope SSO providers (#447)
* fix(security): guard connection tests, close sign-up and scope SSO providers

Fixes the four privately reported issues GHSA-9m33-xfrc-5jxw,
GHSA-6m23-28hh-gjh2, GHSA-2hpx-83vg-gm45 and GHSA-5pp8-r7f5-6q8p.

- The Gitea and GitHub test-connection routes require a signed-in user.
  They were the only non-public API routes without a guard and made a
  server side request to any URL in the body.
- Email sign-up is refused server side once an account exists unless
  AUTH_ALLOW_SIGNUP=true. Only the signup page redirected before; the
  endpoint accepted new accounts from anyone reaching the instance.
- SSO providers are scoped to their owner for list, update and delete,
  and the client secret is never returned. An empty secret on update
  keeps the stored one, and the form says so.
- HttpError messages carry the status only; the upstream body stays on
  the response field for logs and never reaches API clients.
- A narrow outbound guard for user supplied URLs (connection tests, OIDC
  discovery, ntfy, Gotify, Apprise, webhook): link local and cloud
  metadata addresses are refused, host names are resolved and checked,
  and redirects are not followed. Private networks stay allowed since
  mirroring to a LAN Gitea is the normal deployment.

* fix(security): pin plain http requests to the checked address

The guard resolved a host name and checked its addresses, but fetch resolved
the name a second time, so a name that changes its answer between the two
lookups could still reach a link local address. Plain http requests now go
to the address the check saw, with the original name in the Host header.
https keeps the name: the certificate is validated against it and the
metadata services speak no TLS.
2026-09-23 01:53:15 +05:30
42 changed files with 2068 additions and 111 deletions

No files matched your search

+1
View File
@@ -216,6 +216,7 @@ DOCKER_TAG=latest
# Login page: tab that opens first when both email and SSO are available
# AUTH_DEFAULT_METHOD=email # email or sso
# AUTH_ALLOW_SIGNUP=false # true keeps email sign-up open after the first account
# Header Authentication (for Reverse Proxy SSO)
# Enable automatic authentication via reverse proxy headers
+11
View File
@@ -8,10 +8,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Security
- Source URLs go through the outbound guard too (GHSA-p7w3-46pg-mv6h): `POST /api/sources`, `PUT /api/sources/:id` and the configuration save refuse link local and metadata addresses for source and destination URLs, the source fetch helper pins and never follows redirects like the other user supplied URLs, source errors no longer carry the upstream body in their message, and errors raised by a request to another host are never forwarded to API clients whatever words they contain
- Fixed four privately reported vulnerabilities (GHSA-9m33-xfrc-5jxw, GHSA-6m23-28hh-gjh2, GHSA-2hpx-83vg-gm45, GHSA-5pp8-r7f5-6q8p)
- `POST /api/gitea/test-connection` and `POST /api/github/test-connection` now require a signed-in user; they were the only non-public API routes without a guard and made a server side request to any URL in the body
- The email sign-up endpoint is closed server side once an account exists (`AUTH_ALLOW_SIGNUP=true` reopens it); only the signup page redirected before, so anyone reaching the instance could create an account through the API
- SSO providers are scoped to the user who created them for listing, updating and deleting, and the OIDC client secret is no longer returned by the API (the form keeps the stored secret when the field is left blank)
- HTTP errors from Gitea and other hosts no longer carry the upstream response body in the message, so an error can no longer echo the content of a host the caller chose
- Requests to user supplied URLs (connection tests, OIDC discovery, ntfy, Gotify, Apprise and webhook notifications) refuse link local and cloud metadata addresses and no longer follow redirects; private networks stay allowed because mirroring to a LAN Gitea is the normal deployment
- Raised the devalue floor to 5.9.2 in the application and the documentation site (GHSA-9rgm-9g3h-6x36, denial of service through malformed input); both lockfiles resolve 5.9.4
- Raised the dependency floors for the advisories published on 2026-09-08: Astro 7.2.8 (remote code execution through AVIF image optimization, and an authorization bypass when stripping the configured base), @xmldom/xmldom 0.8.15 (eight parser and serializer issues), sharp 0.35.4 (libheif), svgo 4.1.0 (removeScripts sanitization) and js-yaml 4.3.2 (merge-key CPU use). Applied to both the application and the documentation site.
### Added
- Incremental issue and pull request sync (#449)
- After a complete pass, the issues and pull request passes ask GitHub only for items updated since the last one (with a 10 minute margin), so comments, pull request details, commits and files are fetched only for what changed instead of for every item on every sync
- The watermark is stored per repository in the metadata state and only moves when a pass finishes with no failed item
- A full pass still runs when the last one is older than 7 days, when the destination has no mirrored issues or pull requests, and after Reset metadata
- Releases mirror from Gitea and Forgejo sources, Codeberg included (#440)
- The release mirror lists releases through the repository's own source instead of always through the GitHub API, and keeps the same release limit, asset limit, per-destination lock, tag check and retention pruning
- Release assets are downloaded with the credentials of the host they live on, so a Gitea source uses its own token and a public one needs none
+1
View File
@@ -81,6 +81,7 @@ services:
# - GITEA_SKIP_TLS_VERIFY=${GITEA_SKIP_TLS_VERIFY:-false}
# Login page: which tab opens first when both methods are available
- AUTH_DEFAULT_METHOD=${AUTH_DEFAULT_METHOD:-email}
- AUTH_ALLOW_SIGNUP=${AUTH_ALLOW_SIGNUP:-false}
# Header Authentication (for Reverse Proxy SSO)
- HEADER_AUTH_ENABLED=${HEADER_AUTH_ENABLED:-false}
- HEADER_AUTH_USER_HEADER=${HEADER_AUTH_USER_HEADER:-X-Authentik-Username}
+1
View File
@@ -347,6 +347,7 @@ Configure authentication methods and SSO.
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `AUTH_DEFAULT_METHOD` | Tab the login page opens on when both email and SSO are available. Any other value falls back to `email`. | `email` | `email`, `sso` |
| `AUTH_ALLOW_SIGNUP` | Allow email and password sign-up after the first account exists. Off, the sign-up endpoint answers 403 once a user exists; the first account can always be created. | `false` | `true`, `false` |
The login page also remembers the method last used in that browser, which wins
over this default. Anything unrecognised falls back to `email`.
+2 -2
View File
@@ -2,8 +2,8 @@ apiVersion: v2
name: gitea-mirror
description: Kubernetes helm chart for gitea-mirror
type: application
version: 3.38.0
appVersion: "3.38.0"
version: 3.38.2
appVersion: "3.38.2"
icon: https://github.com/RayLabsHQ/gitea-mirror/blob/main/.github/assets/logo.png
keywords:
- git
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "gitea-mirror",
"type": "module",
"version": "3.38.0",
"version": "3.38.2",
"engines": {
"bun": ">=1.2.9"
},
+1
View File
@@ -454,6 +454,7 @@ export function SSOSettings() {
<Input
id="clientSecret"
type="password"
placeholder={editingProvider ? "Leave blank to keep the current secret" : ""}
value={providerForm.clientSecret}
onChange={e => setProviderForm(prev => ({ ...prev, clientSecret: e.target.value }))}
/>
+65
View File
@@ -0,0 +1,65 @@
import { describe, expect, it } from "bun:test";
import {
isOpenRegistration,
isSignupClosed,
isSignupRequest,
signupClosedError,
signupGuardPlugin,
} from "./auth-signup-guard";
describe("signup guard", () => {
it("only matches the email sign-up endpoint", () => {
expect(isSignupRequest("/sign-up/email")).toBe(true);
expect(isSignupRequest("/sign-in/email")).toBe(false);
expect(isSignupRequest("/sign-up/email/extra")).toBe(false);
expect(isSignupRequest(undefined)).toBe(false);
});
it("opens registration only for the literal true", () => {
expect(isOpenRegistration({ AUTH_ALLOW_SIGNUP: "true" } as NodeJS.ProcessEnv)).toBe(true);
expect(isOpenRegistration({ AUTH_ALLOW_SIGNUP: "TRUE" } as NodeJS.ProcessEnv)).toBe(false);
expect(isOpenRegistration({ AUTH_ALLOW_SIGNUP: "1" } as NodeJS.ProcessEnv)).toBe(false);
expect(isOpenRegistration({} as NodeJS.ProcessEnv)).toBe(false);
});
it("closes sign-up once a user exists unless registration is open", () => {
expect(isSignupClosed({ userCount: 0, allowSignup: false })).toBe(false);
expect(isSignupClosed({ userCount: 1, allowSignup: false })).toBe(true);
expect(isSignupClosed({ userCount: 5, allowSignup: true })).toBe(false);
});
it("answers 403 with a stable code", () => {
const error = signupClosedError();
expect(error.status).toBe("FORBIDDEN");
expect(error.body?.code).toBe("SIGNUP_CLOSED");
});
it("registers a before hook on the sign-up path that counts users", async () => {
let counted = 0;
const plugin = signupGuardPlugin({
countUsers: async () => {
counted += 1;
return 1;
},
});
const hook = plugin.hooks.before[0];
expect(hook.matcher({ path: "/sign-up/email" } as any)).toBe(true);
expect(hook.matcher({ path: "/sign-in/email" } as any)).toBe(false);
const previous = process.env.AUTH_ALLOW_SIGNUP;
delete process.env.AUTH_ALLOW_SIGNUP;
try {
await expect(hook.handler({ path: "/sign-up/email", context: {} } as any)).rejects.toMatchObject({
body: { code: "SIGNUP_CLOSED" },
});
expect(counted).toBe(1);
process.env.AUTH_ALLOW_SIGNUP = "true";
await hook.handler({ path: "/sign-up/email", context: {} } as any);
expect(counted).toBe(1);
} finally {
if (previous === undefined) delete process.env.AUTH_ALLOW_SIGNUP;
else process.env.AUTH_ALLOW_SIGNUP = previous;
}
});
});
+68
View File
@@ -0,0 +1,68 @@
import type { BetterAuthPlugin } from "better-auth";
import { APIError, createAuthMiddleware } from "better-auth/api";
/**
* Closes email and password sign-up once the first account exists.
*
* The signup page already redirects to the login page when a user exists,
* but that check lived only in the page. The Better Auth endpoint behind
* it, POST /api/auth/sign-up/email, accepted new accounts from anyone who
* could reach the instance, which turned every "signed in users only" API
* route into a public one. This plugin refuses the endpoint server side
* once a user exists, unless the operator opts into open registration with
* AUTH_ALLOW_SIGNUP=true.
*
* SSO sign-ins through a registered identity provider and header auth
* provisioning (HEADER_AUTH_AUTO_PROVISION) are separate flows and are not
* affected: the operator chose those providers.
*/
/** Endpoint path (without the auth base path) of email sign-up. */
export const SIGNUP_PATH = "/sign-up/email";
export function isSignupRequest(path: string | undefined): boolean {
return path === SIGNUP_PATH;
}
/** Reads the opt-in from the environment: only the literal "true" opens sign-up. */
export function isOpenRegistration(env: NodeJS.ProcessEnv = process.env): boolean {
return env.AUTH_ALLOW_SIGNUP === "true";
}
/** True when a sign-up must be refused: an account exists and registration is closed. */
export function isSignupClosed({
userCount,
allowSignup,
}: {
userCount: number;
allowSignup: boolean;
}): boolean {
return userCount > 0 && !allowSignup;
}
export function signupClosedError(): APIError {
return new APIError("FORBIDDEN", {
message:
"Sign-up is closed: an account already exists on this instance. Set AUTH_ALLOW_SIGNUP=true to allow more accounts.",
code: "SIGNUP_CLOSED",
});
}
export const signupGuardPlugin = (options: { countUsers: () => Promise<number> }) =>
({
id: "signup-guard",
hooks: {
before: [
{
matcher: (ctx) => isSignupRequest(ctx.path),
handler: createAuthMiddleware(async () => {
if (isOpenRegistration()) return;
const userCount = await options.countUsers();
if (isSignupClosed({ userCount, allowSignup: false })) {
throw signupClosedError();
}
}),
},
],
},
}) satisfies BetterAuthPlugin;
+13
View File
@@ -11,6 +11,7 @@ import { eq } from "drizzle-orm";
import { withBase } from "./base-path";
import { headerAuthPlugin } from "./auth-header-plugin";
import { apiKeyGuardPlugin } from "./auth-api-key-guard";
import { signupGuardPlugin } from "./auth-signup-guard";
/**
* Extracts the origins implied by registered SSO identity providers.
@@ -402,6 +403,18 @@ export const auth = betterAuth({
// can never create, list or revoke keys. See auth-api-key-guard.ts.
apiKeyGuardPlugin(),
// Refuses email sign-up once an account exists unless
// AUTH_ALLOW_SIGNUP=true. The signup page only redirected; the endpoint
// itself was open. See auth-signup-guard.ts.
signupGuardPlugin({
countUsers: async () => {
const { db, users } = await import("./db");
const { sql } = await import("drizzle-orm");
const [row] = await db.select({ count: sql<number>`count(*)` }).from(users);
return Number(row?.count ?? 0);
},
}),
// Header / forward authentication bridge. Exposes
// POST /api/auth/sign-in/header so the middleware can mint a real
// Better Auth session from trusted upstream headers (Authentik /
+2 -2
View File
@@ -9,8 +9,8 @@ mock.module("@/lib/helpers", () => ({
}));
const mockMirrorGitHubReleasesToGitea = mock(() => Promise.resolve());
const mockMirrorGitRepoIssuesToGitea = mock(() => Promise.resolve());
const mockMirrorGitRepoPullRequestsToGitea = mock(() => Promise.resolve());
const mockMirrorGitRepoIssuesToGitea = mock(() => Promise.resolve(undefined));
const mockMirrorGitRepoPullRequestsToGitea = mock(() => Promise.resolve(undefined));
const mockMirrorGitRepoLabelsToGitea = mock(() => Promise.resolve());
const mockMirrorGitRepoMilestonesToGitea = mock(() => Promise.resolve());
const mockSyncRepositoryMetadataToGitea = mock(() => Promise.resolve());
+6 -2
View File
@@ -1090,13 +1090,15 @@ export async function syncGiteaRepoEnhanced({
);
} else {
try {
await dependencies.mirrorGitRepoIssuesToGitea({
const issuesCursor = await dependencies.mirrorGitRepoIssuesToGitea({
config,
octokit,
repository,
giteaOwner: repoOwner,
giteaRepoName: repoName,
syncCursor: metadataState.syncCursors.issues,
});
if (issuesCursor) metadataState.syncCursors.issues = issuesCursor;
metadataState.components.issues = true;
metadataState.components.labels = true;
metadataUpdated = true;
@@ -1123,13 +1125,15 @@ export async function syncGiteaRepoEnhanced({
);
} else {
try {
await dependencies.mirrorGitRepoPullRequestsToGitea({
const pullRequestsCursor = await dependencies.mirrorGitRepoPullRequestsToGitea({
config,
octokit,
repository,
giteaOwner: repoOwner,
giteaRepoName: repoName,
syncCursor: metadataState.syncCursors.pullRequests,
});
if (pullRequestsCursor) metadataState.syncCursors.pullRequests = pullRequestsCursor;
metadataState.components.pullRequests = true;
metadataUpdated = true;
console.log(
+476
View File
@@ -0,0 +1,476 @@
/**
* Behavioural tests for incremental issue and pull request sync (#449).
*
* With a watermark from a completed pass, the GitHub listing is asked only
* for what changed (`since`), and the per-issue comment listing and per-PR
* detail, commits and files calls run only for the items that came back.
* Without one, the pass is full and unchanged from before.
*
* The destination is a fake global fetch and GitHub is a fake Octokit. Other
* test files replace @/lib/http-client and friends with mock.module, which is
* process wide in bun and depends on file order, so the suites run in an
* isolated child process (same harness as gitea-source-releases.test.ts).
*/
import { afterEach, describe, expect, it, test } from "bun:test";
import type { Config } from "@/types/config";
import type { Repository } from "@/lib/db/schema";
import type { MetadataSyncCursor } from "@/lib/metadata-state";
const CHILD_FLAG = "GM_INCREMENTAL_METADATA_ISOLATED";
const isChild = !!process.env[CHILD_FLAG];
if (!isChild) {
test("incremental metadata sync - isolated child suite", () => {
const res = Bun.spawnSync({
cmd: [process.execPath, "test", import.meta.path],
env: { ...process.env, [CHILD_FLAG]: "1" },
stdout: "pipe",
stderr: "pipe",
});
if (res.exitCode !== 0) {
console.error(res.stdout.toString());
console.error(res.stderr.toString());
}
expect(res.exitCode).toBe(0);
}, 60_000);
}
const { mirrorGitRepoIssuesToGitea, mirrorGitRepoPullRequestsToGitea } = isChild
? await import("@/lib/gitea")
: ({} as typeof import("@/lib/gitea"));
const { INCREMENTAL_SYNC_SAFETY_MARGIN_MS } = isChild
? await import("@/lib/metadata-state")
: ({} as typeof import("@/lib/metadata-state"));
const GITEA_URL = "https://gitea.example.com";
const DEST_OWNER = "mirror-owner";
const DEST_REPO = "demo";
const DEST_API = `${GITEA_URL}/api/v1/repos/${DEST_OWNER}/${DEST_REPO}`;
const realFetch = globalThis.fetch;
afterEach(() => {
globalThis.fetch = realFetch;
});
const config = {
userId: "user-1",
githubConfig: { owner: "acme", token: "github-token" },
giteaConfig: {
url: GITEA_URL,
token: "gitea-token",
defaultOwner: DEST_OWNER,
issueConcurrency: 1,
pullRequestConcurrency: 1,
},
} as unknown as Partial<Config>;
const repository = {
id: "repo-1",
name: DEST_REPO,
fullName: "acme/demo",
owner: "acme",
isStarred: false,
mirrorOverrides: null,
organization: null,
} as unknown as Repository;
interface Recorded {
method: string;
url: string;
body?: string;
}
/** Destination Gitea with a fixed set of issues already present. */
function fakeGitea(existingIssues: Array<{ number: number; title: string }>) {
const requests: Recorded[] = [];
let nextNumber = 1000;
globalThis.fetch = (async (input: any, init?: any) => {
const url = typeof input === "string" ? input : String(input);
const method = String(init?.method ?? "GET").toUpperCase();
const body = typeof init?.body === "string" ? init.body : undefined;
requests.push({ method, url, body });
const json = (payload: unknown, status = 200) =>
new Response(JSON.stringify(payload), {
status,
headers: { "content-type": "application/json" },
});
if (method === "GET" && url === DEST_API) return json({ id: 1, name: DEST_REPO });
if (method === "GET" && url === `${DEST_API}/labels`) return json([]);
if (method === "POST" && url === `${DEST_API}/labels`) return json({ id: 7 }, 201);
if (method === "GET" && url.startsWith(`${DEST_API}/issues?`)) {
// The defensive recheck searches by marker; nothing to recover.
if (url.includes("&q=")) return json([]);
const page = Number(new URL(url).searchParams.get("page") ?? "1");
return json(
page === 1
? existingIssues.map((issue) => ({ ...issue, body: "", state: "open" }))
: []
);
}
if (method === "POST" && url === `${DEST_API}/issues`) {
nextNumber += 1;
return json({ number: nextNumber, state: "open" }, 201);
}
if (method === "PATCH" && /\/issues\/\d+$/.test(url)) return json({});
if (method === "PUT" && /\/issues\/\d+\/labels$/.test(url)) return json([]);
if (method === "GET" && /\/issues\/\d+\/comments\?/.test(url)) return json([]);
if (method === "POST" && /\/issues\/\d+\/comments$/.test(url)) return json({ id: 1 }, 201);
throw new Error(`unexpected request: ${method} ${url}`);
}) as unknown as typeof fetch;
return {
requests,
createdTitles: () =>
requests
.filter((r) => r.method === "POST" && r.url === `${DEST_API}/issues`)
.map((r) => JSON.parse(r.body!).title as string),
createdBodies: () =>
requests
.filter((r) => r.method === "POST" && r.url === `${DEST_API}/issues`)
.map((r) => JSON.parse(r.body!).body as string),
};
}
function ghIssue(number: number, extra: Record<string, unknown> = {}) {
return {
number,
title: `Issue ${number}`,
body: `body ${number}`,
state: "open",
html_url: `https://github.com/acme/demo/issues/${number}`,
user: { login: "alice", html_url: "https://github.com/alice" },
labels: [],
assignees: [],
created_at: `2026-01-0${number % 9 || 1}T00:00:00Z`,
updated_at: "2026-09-23T00:00:00Z",
...extra,
};
}
function ghPull(number: number, extra: Record<string, unknown> = {}) {
return {
number,
title: `PR ${number}`,
body: `pr body ${number}`,
state: "open",
html_url: `https://github.com/acme/demo/pull/${number}`,
user: { login: "bob", html_url: "https://github.com/bob" },
created_at: "2026-01-01T00:00:00Z",
merged_at: null,
base: { ref: "main" },
head: { ref: `feature-${number}` },
...extra,
};
}
interface GitHubData {
/** What issues.listForRepo returns (issues and PR entries). */
issueListing: any[];
/** What pulls.list returns. */
pullListing?: any[];
/** When set, pulls.get throws for these numbers. */
failDetailFor?: number[];
}
/** Fake Octokit that records every call made through it. */
function fakeOctokit(data: GitHubData) {
const calls: Array<{ endpoint: string; params: any }> = [];
const issues = {
listForRepo: async (params: any) => {
calls.push({ endpoint: "issues.listForRepo", params });
return { data: data.issueListing };
},
listComments: async (params: any) => {
calls.push({ endpoint: "issues.listComments", params });
return { data: [] };
},
};
const pulls = {
list: async (params: any) => {
calls.push({ endpoint: "pulls.list", params });
return { data: data.pullListing ?? [] };
},
get: async (params: any) => {
calls.push({ endpoint: "pulls.get", params });
if (data.failDetailFor?.includes(params.pull_number)) {
throw new Error("detail unavailable");
}
return {
data: {
...ghPull(params.pull_number),
additions: 1,
deletions: 0,
changed_files: 1,
merged_by: null,
},
};
},
listCommits: async (params: any) => {
calls.push({ endpoint: "pulls.listCommits", params });
return { data: [] };
},
listFiles: async (params: any) => {
calls.push({ endpoint: "pulls.listFiles", params });
return { data: [] };
},
};
const octokit = {
rest: { issues, pulls },
paginate: async (fn: any, params: any, map?: (res: any) => any) => {
const res = await fn(params);
return map ? map(res) : res.data;
},
};
return {
octokit: octokit as any,
calls,
callsTo: (endpoint: string) => calls.filter((c) => c.endpoint === endpoint),
};
}
function recentCursor(): MetadataSyncCursor {
return {
lastPassStartedAt: new Date(Date.now() - 60 * 60 * 1000).toISOString(),
lastFullPassStartedAt: new Date(Date.now() - 2 * 24 * 60 * 60 * 1000).toISOString(),
};
}
function expectedSince(cursor: MetadataSyncCursor) {
return new Date(
Date.parse(cursor.lastPassStartedAt) - INCREMENTAL_SYNC_SAFETY_MARGIN_MS
).toISOString();
}
describe.skipIf(!isChild)("mirrorGitRepoIssuesToGitea incremental sync", () => {
it("runs a full pass without a watermark and records one", async () => {
fakeGitea([]);
const github = fakeOctokit({
issueListing: [ghIssue(1), ghIssue(2), ghPull(3, { pull_request: {} })],
});
const before = Date.now();
const cursor = await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
});
const after = Date.now();
const [listing] = github.callsTo("issues.listForRepo");
expect(listing.params.since).toBeUndefined();
expect(listing.params.sort).toBe("created");
expect(listing.params.direction).toBe("asc");
expect(github.callsTo("issues.listComments").map((c) => c.params.issue_number)).toEqual([1, 2]);
expect(cursor).toBeDefined();
const started = Date.parse(cursor!.lastPassStartedAt);
expect(started).toBeGreaterThanOrEqual(before);
expect(started).toBeLessThanOrEqual(after);
expect(cursor!.lastFullPassStartedAt).toBe(cursor!.lastPassStartedAt);
});
it("lists with since and only reconciles the returned issues", async () => {
fakeGitea([
{ number: 11, title: "[GH-ISSUE #1] Issue 1" },
{ number: 12, title: "[GH-ISSUE #2] Issue 2" },
{ number: 13, title: "[GH-ISSUE #3] Issue 3" },
]);
const github = fakeOctokit({ issueListing: [ghIssue(2)] });
const previous = recentCursor();
const cursor = await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: previous,
});
const listings = github.callsTo("issues.listForRepo");
expect(listings).toHaveLength(1);
expect(listings[0].params.since).toBe(expectedSince(previous));
expect(listings[0].params.sort).toBe("updated");
expect(github.callsTo("issues.listComments").map((c) => c.params.issue_number)).toEqual([2]);
// The watermark moves to this listing; the last full pass is kept.
expect(cursor).toBeDefined();
expect(Date.parse(cursor!.lastPassStartedAt)).toBeGreaterThan(
Date.parse(previous.lastPassStartedAt)
);
expect(cursor!.lastFullPassStartedAt).toBe(previous.lastFullPassStartedAt);
});
it("creates new issues from an incremental listing in creation order", async () => {
const gitea = fakeGitea([{ number: 11, title: "[GH-ISSUE #1] Issue 1" }]);
// GitHub returns them by updated_at: #5 changed before #4.
const github = fakeOctokit({
issueListing: [
ghIssue(5, { created_at: "2026-09-22T10:00:00Z", updated_at: "2026-09-23T01:00:00Z" }),
ghIssue(4, { created_at: "2026-09-22T09:00:00Z", updated_at: "2026-09-23T02:00:00Z" }),
],
});
await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: recentCursor(),
});
expect(gitea.createdTitles()).toEqual(["[GH-ISSUE #4] Issue 4", "[GH-ISSUE #5] Issue 5"]);
});
it("falls back to a full pass when the destination has no mirrored issues", async () => {
fakeGitea([]);
const github = fakeOctokit({ issueListing: [ghIssue(1)] });
const previous = recentCursor();
const cursor = await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: previous,
});
const listings = github.callsTo("issues.listForRepo");
expect(listings).toHaveLength(2);
expect(listings[0].params.since).toBe(expectedSince(previous));
expect(listings[1].params.since).toBeUndefined();
expect(cursor!.lastFullPassStartedAt).toBe(cursor!.lastPassStartedAt);
});
it("runs a full pass when the last full pass is older than 7 days", async () => {
fakeGitea([{ number: 11, title: "[GH-ISSUE #1] Issue 1" }]);
const github = fakeOctokit({ issueListing: [ghIssue(1), ghIssue(2)] });
await mirrorGitRepoIssuesToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: {
lastPassStartedAt: new Date(Date.now() - 60 * 60 * 1000).toISOString(),
lastFullPassStartedAt: new Date(Date.now() - 8 * 24 * 60 * 60 * 1000).toISOString(),
},
});
const listings = github.callsTo("issues.listForRepo");
expect(listings).toHaveLength(1);
expect(listings[0].params.since).toBeUndefined();
expect(github.callsTo("issues.listComments")).toHaveLength(2);
});
});
describe.skipIf(!isChild)("mirrorGitRepoPullRequestsToGitea incremental sync", () => {
it("runs a full pass with pulls.list without a watermark", async () => {
fakeGitea([]);
const github = fakeOctokit({
issueListing: [],
pullListing: [ghPull(1), ghPull(2)],
});
const cursor = await mirrorGitRepoPullRequestsToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
});
expect(github.callsTo("pulls.list")).toHaveLength(1);
expect(github.callsTo("issues.listForRepo")).toHaveLength(0);
expect(github.callsTo("pulls.get").map((c) => c.params.pull_number)).toEqual([1, 2]);
expect(github.callsTo("pulls.listCommits")).toHaveLength(2);
expect(github.callsTo("pulls.listFiles")).toHaveLength(2);
expect(cursor).toBeDefined();
expect(cursor!.lastFullPassStartedAt).toBe(cursor!.lastPassStartedAt);
});
it("lists changed pull requests through the issues endpoint with since", async () => {
const gitea = fakeGitea([
{ number: 21, title: "[PR #1] PR 1" },
{ number: 22, title: "[PR #2] PR 2" },
{ number: 23, title: "[PR #3] PR 3" },
]);
const github = fakeOctokit({
// Issue #9 changed too; it is not a pull request and is skipped here.
issueListing: [
ghIssue(9),
{ ...ghIssue(3), title: "PR 3", pull_request: { merged_at: null } },
{ ...ghIssue(4), title: "PR 4", pull_request: { merged_at: null } },
],
});
const previous = recentCursor();
const cursor = await mirrorGitRepoPullRequestsToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: previous,
});
expect(github.callsTo("pulls.list")).toHaveLength(0);
const [listing] = github.callsTo("issues.listForRepo");
expect(listing.params.since).toBe(expectedSince(previous));
expect(listing.params.sort).toBe("updated");
expect(github.callsTo("pulls.get").map((c) => c.params.pull_number).sort()).toEqual([3, 4]);
expect(github.callsTo("pulls.listCommits")).toHaveLength(2);
expect(github.callsTo("pulls.listFiles")).toHaveLength(2);
// The new PR is built from the detail response, which has the refs.
const [created] = gitea.createdBodies();
expect(created).toContain("feature-4");
expect(gitea.createdTitles()).toEqual(["[PR #4] PR 4"]);
expect(cursor!.lastFullPassStartedAt).toBe(previous.lastFullPassStartedAt);
});
it("keeps the previous watermark when a pull request fails", async () => {
fakeGitea([{ number: 21, title: "[PR #1] PR 1" }]);
// Detail fails and the basic fallback create also fails.
globalThis.fetch = ((original) =>
(async (input: any, init?: any) => {
const url = typeof input === "string" ? input : String(input);
if (String(init?.method ?? "GET").toUpperCase() === "POST" && url === `${DEST_API}/issues`) {
return new Response("boom", { status: 500 });
}
return original(input, init);
}) as unknown as typeof fetch)(globalThis.fetch);
const github = fakeOctokit({
issueListing: [{ ...ghIssue(5), pull_request: { merged_at: null } }],
failDetailFor: [5],
});
const cursor = await mirrorGitRepoPullRequestsToGitea({
config,
octokit: github.octokit,
repository,
giteaOwner: DEST_OWNER,
giteaRepoName: DEST_REPO,
syncCursor: recentCursor(),
});
expect(cursor).toBeUndefined();
expect(github.callsTo("pulls.get").map((c) => c.params.pull_number)).toEqual([5]);
});
});
+249 -41
View File
@@ -30,8 +30,11 @@ import {
listSources,
} from "./sources";
import {
advanceMetadataSyncCursor,
parseRepositoryMetadataState,
planMetadataPass,
serializeRepositoryMetadataState,
type MetadataSyncCursor,
} from "./metadata-state";
import {
normalizeReleaseAssetLimit,
@@ -1178,13 +1181,15 @@ export const mirrorGithubRepoToGitea = async ({
if (shouldMirrorIssuesThisRun) {
try {
await mirrorGitRepoIssuesToGitea({
const issuesCursor = await mirrorGitRepoIssuesToGitea({
config,
octokit,
repository,
giteaOwner: repoOwner,
giteaRepoName: targetRepoName,
syncCursor: metadataState.syncCursors.issues,
});
if (issuesCursor) metadataState.syncCursors.issues = issuesCursor;
metadataState.components.issues = true;
metadataState.components.labels = true;
metadataUpdated = true;
@@ -1209,13 +1214,15 @@ export const mirrorGithubRepoToGitea = async ({
if (shouldMirrorPullRequests) {
try {
await mirrorGitRepoPullRequestsToGitea({
const pullRequestsCursor = await mirrorGitRepoPullRequestsToGitea({
config,
octokit,
repository,
giteaOwner: repoOwner,
giteaRepoName: targetRepoName,
syncCursor: metadataState.syncCursors.pullRequests,
});
if (pullRequestsCursor) metadataState.syncCursors.pullRequests = pullRequestsCursor;
metadataState.components.pullRequests = true;
metadataUpdated = true;
console.log(
@@ -1950,13 +1957,15 @@ export async function mirrorGitHubRepoToGiteaOrg({
if (shouldMirrorIssuesThisRun) {
try {
await mirrorGitRepoIssuesToGitea({
const issuesCursor = await mirrorGitRepoIssuesToGitea({
config,
octokit,
repository,
giteaOwner: orgName,
giteaRepoName: targetRepoName,
syncCursor: metadataState.syncCursors.issues,
});
if (issuesCursor) metadataState.syncCursors.issues = issuesCursor;
metadataState.components.issues = true;
metadataState.components.labels = true;
metadataUpdated = true;
@@ -1981,13 +1990,15 @@ export async function mirrorGitHubRepoToGiteaOrg({
if (shouldMirrorPullRequests) {
try {
await mirrorGitRepoPullRequestsToGitea({
const pullRequestsCursor = await mirrorGitRepoPullRequestsToGitea({
config,
octokit,
repository,
giteaOwner: orgName,
giteaRepoName: targetRepoName,
syncCursor: metadataState.syncCursors.pullRequests,
});
if (pullRequestsCursor) metadataState.syncCursors.pullRequests = pullRequestsCursor;
metadataState.components.pullRequests = true;
metadataUpdated = true;
console.log(
@@ -2604,13 +2615,19 @@ export const mirrorGitRepoIssuesToGitea = async ({
repository,
giteaOwner,
giteaRepoName,
syncCursor,
}: {
config: Partial<Config>;
octokit: Octokit;
repository: Repository;
giteaOwner: string;
giteaRepoName?: string;
}) => {
/**
* Watermark from the previous completed issues pass (#449). When set,
* only issues updated since then are listed and reconciled.
*/
syncCursor?: MetadataSyncCursor;
}): Promise<MetadataSyncCursor | undefined> => {
//things covered here are- issue, title, body, labels, comments and assignees
// The source client arrives as `octokit` (anonymous for a public-only
// source), so only the destination has to be configured here.
@@ -2649,30 +2666,74 @@ export const mirrorGitRepoIssuesToGitea = async ({
const [owner, repo] = repository.fullName.split("/");
// Fetch GitHub issues
const issues = await octokit.paginate(
octokit.rest.issues.listForRepo,
{
owner,
repo,
state: "all",
per_page: 100,
sort: "created",
direction: "asc",
},
(res) => res.data
);
// Incremental sync (#449): once a pass has completed, ask GitHub only
// for issues updated since then. A new comment, a label, state,
// milestone, title or body change all bump updated_at, so this is the
// set that needs reconciling, and the per-issue comment listing below
// only runs for it. The watermark is the time this listing started.
let plan = planMetadataPass(syncCursor);
const listingStartedAt = new Date();
// Filter out pull requests
const filteredIssues = issues.filter((issue) => !(issue as any).pull_request);
const listGitHubIssues = async (since?: string) => {
const listed = await octokit.paginate(
octokit.rest.issues.listForRepo,
since
? {
owner,
repo,
state: "all",
per_page: 100,
sort: "updated",
direction: "asc",
since,
}
: {
owner,
repo,
state: "all",
per_page: 100,
sort: "created",
direction: "asc",
},
(res) => res.data
);
// Filter out pull requests
const onlyIssues = listed.filter((issue) => !(issue as any).pull_request);
if (since) {
// Process in creation order, like the full pass, so issues opened
// since the last pass are created in Gitea in their GitHub order.
onlyIssues.sort(
(a, b) =>
new Date(a.created_at).getTime() - new Date(b.created_at).getTime() ||
a.number - b.number
);
}
return onlyIssues;
};
let filteredIssues: Awaited<ReturnType<typeof listGitHubIssues>>;
if (plan.mode === "incremental") {
console.log(
`[Issues] Incremental pass for ${repository.fullName}: listing issues updated since ${plan.since}`
);
filteredIssues = await listGitHubIssues(plan.since);
console.log(
`[Issues] Incremental listing returned ${filteredIssues.length} changed issue(s) for ${repository.fullName}`
);
} else {
console.log(
`[Issues] Full pass for ${repository.fullName} (${plan.reason})`
);
filteredIssues = await listGitHubIssues();
}
console.log(
`Mirroring ${filteredIssues.length} issues from ${repository.fullName}`
);
if (filteredIssues.length === 0) {
if (filteredIssues.length === 0 && plan.mode === "full") {
console.log(`No issues to mirror for ${repository.fullName}`);
return;
return advanceMetadataSyncCursor(syncCursor, plan.mode, listingStartedAt);
}
const ghIssueMarkerRegex = /\[GH-ISSUE #(\d+)\]/i;
@@ -2736,6 +2797,25 @@ export const mirrorGitRepoIssuesToGitea = async ({
titleFallbackMap.set(title, existing);
}
// A watermark with no mirrored issue on the Gitea side means the
// destination was recreated or emptied since the last pass. Only a
// full listing can repopulate it.
if (plan.mode === "incremental" && giteaIssueByGitHubNumber.size === 0) {
console.log(
`[Issues] No mirrored issues found in Gitea for ${repoName}; ignoring the watermark and running a full pass`
);
plan = { mode: "full", reason: "destination has no mirrored issues" };
filteredIssues = await listGitHubIssues();
console.log(
`Mirroring ${filteredIssues.length} issues from ${repository.fullName}`
);
}
if (filteredIssues.length === 0) {
console.log(`No issues to mirror for ${repository.fullName}`);
return advanceMetadataSyncCursor(syncCursor, plan.mode, listingStartedAt);
}
// Get existing labels from Gitea
const giteaLabelsRes = await httpGet(
`${config.giteaConfig.url}/api/v1/repos/${giteaOwner}/${repoName}/labels`,
@@ -2765,7 +2845,7 @@ export const mirrorGitRepoIssuesToGitea = async ({
}
// Process issues in parallel with concurrency control
await processWithRetry(
const mirroredIssues = await processWithRetry(
filteredIssues,
async (issue) => {
const githubLabelNames =
@@ -3084,6 +3164,18 @@ export const mirrorGitRepoIssuesToGitea = async ({
console.log(
`Completed mirroring ${filteredIssues.length} issues for ${repository.fullName}`
);
// processWithRetry logs and drops items that still fail after their
// retries. Keep the old watermark so the next pass lists them again.
const failedIssueCount = filteredIssues.length - mirroredIssues.length;
if (failedIssueCount > 0) {
console.warn(
`[Issues] ${failedIssueCount} issue(s) failed for ${repository.fullName}; keeping the previous watermark so they are retried next run`
);
return undefined;
}
return advanceMetadataSyncCursor(syncCursor, plan.mode, listingStartedAt);
};
/**
@@ -3964,13 +4056,19 @@ export async function mirrorGitRepoPullRequestsToGitea({
repository,
giteaOwner,
giteaRepoName,
syncCursor,
}: {
config: Partial<Config>;
octokit: Octokit;
repository: Repository;
giteaOwner: string;
giteaRepoName?: string;
}) {
/**
* Watermark from the previous completed pull request pass (#449).
* When set, only pull requests updated since then are processed.
*/
syncCursor?: MetadataSyncCursor;
}): Promise<MetadataSyncCursor | undefined> {
// The source client arrives as `octokit`; see mirrorGitRepoIssuesToGitea.
if (
!config.giteaConfig?.token ||
@@ -4007,27 +4105,97 @@ export async function mirrorGitRepoPullRequestsToGitea({
const [owner, repo] = repository.fullName.split("/");
// Fetch GitHub pull requests
const pullRequests = await octokit.paginate(
octokit.rest.pulls.list,
{
owner,
repo,
state: "all",
per_page: 100,
sort: "created",
direction: "asc",
},
(res) => res.data
);
// Incremental sync (#449). pulls.list has no `since`, so an
// incremental pass lists issues updated since the watermark and keeps
// the pull request entries. The detail, commits and files calls below
// then only run for those. The watermark is the time this listing
// started.
let plan = planMetadataPass(syncCursor);
const listingStartedAt = new Date();
type ListedPullRequest = Awaited<
ReturnType<typeof octokit.rest.pulls.list>
>["data"][number];
const listAllPullRequests = () =>
octokit.paginate(
octokit.rest.pulls.list,
{
owner,
repo,
state: "all",
per_page: 100,
sort: "created",
direction: "asc",
},
(res) => res.data
);
const listChangedPullRequests = async (
since: string
): Promise<ListedPullRequest[]> => {
const changed = await octokit.paginate(
octokit.rest.issues.listForRepo,
{
owner,
repo,
state: "all",
per_page: 100,
sort: "updated",
direction: "asc",
since,
},
(res) => res.data
);
return changed
.filter((entry) => Boolean((entry as any).pull_request))
.sort(
(a, b) =>
new Date(a.created_at).getTime() - new Date(b.created_at).getTime() ||
a.number - b.number
)
.map(
(entry) =>
// The issues endpoint has no head/base refs. The per-PR detail
// call below replaces this with the full pull request; these
// fields are what the basic fallback needs if that call fails.
({
number: entry.number,
title: entry.title,
body: entry.body ?? null,
state: entry.state,
html_url: entry.html_url,
user: entry.user,
created_at: entry.created_at,
merged_at: (entry as any).pull_request?.merged_at ?? null,
}) as unknown as ListedPullRequest
);
};
let pullRequests: ListedPullRequest[];
let listedFromIssues = false;
if (plan.mode === "incremental") {
console.log(
`[Pull Requests] Incremental pass for ${repository.fullName}: listing pull requests updated since ${plan.since}`
);
pullRequests = await listChangedPullRequests(plan.since);
listedFromIssues = true;
console.log(
`[Pull Requests] Incremental listing returned ${pullRequests.length} changed pull request(s) for ${repository.fullName}`
);
} else {
console.log(
`[Pull Requests] Full pass for ${repository.fullName} (${plan.reason})`
);
pullRequests = await listAllPullRequests();
}
console.log(
`Mirroring ${pullRequests.length} pull requests from ${repository.fullName}`
);
if (pullRequests.length === 0) {
if (pullRequests.length === 0 && plan.mode === "full") {
console.log(`No pull requests to mirror for ${repository.fullName}`);
return;
return advanceMetadataSyncCursor(syncCursor, plan.mode, listingStartedAt);
}
// Note: Gitea doesn't have a direct API to create pull requests from external sources
@@ -4104,6 +4272,25 @@ export async function mirrorGitRepoPullRequestsToGitea({
prIssuesPage += 1;
}
// Same guard as the issues pass: a watermark with no mirrored pull
// request in Gitea means the destination was recreated or emptied.
if (plan.mode === "incremental" && existingPrIssuesByNumber.size === 0) {
console.log(
`[Pull Requests] No mirrored pull requests found in Gitea for ${repoName}; ignoring the watermark and running a full pass`
);
plan = { mode: "full", reason: "destination has no mirrored pull requests" };
pullRequests = await listAllPullRequests();
listedFromIssues = false;
console.log(
`Mirroring ${pullRequests.length} pull requests from ${repository.fullName}`
);
}
if (pullRequests.length === 0) {
console.log(`No pull requests to mirror for ${repository.fullName}`);
return advanceMetadataSyncCursor(syncCursor, plan.mode, listingStartedAt);
}
const { processWithRetry } = await import("@/lib/utils/concurrency");
const rawPullConcurrency = config.giteaConfig?.pullRequestConcurrency ?? 5;
@@ -4121,9 +4308,10 @@ export async function mirrorGitRepoPullRequestsToGitea({
let successCount = 0;
let failedCount = 0;
await processWithRetry(
const processedPullRequests = await processWithRetry(
pullRequests,
async (pr) => {
async (listedPr) => {
let pr = listedPr;
try {
// Fetch additional PR data for rich metadata
const [prDetail, commits, files] = await Promise.all([
@@ -4131,6 +4319,11 @@ export async function mirrorGitRepoPullRequestsToGitea({
octokit.rest.pulls.listCommits({ owner, repo, pull_number: pr.number, per_page: 10 }),
octokit.rest.pulls.listFiles({ owner, repo, pull_number: pr.number, per_page: 100 })
]);
// An incremental listing came from the issues endpoint; the
// detail response has every field the pulls.list entry has.
if (listedFromIssues) {
pr = prDetail.data as unknown as ListedPullRequest;
}
// Build rich PR body with metadata
let richBody = `## 📋 Pull Request Information\n\n`;
@@ -4408,6 +4601,21 @@ export async function mirrorGitRepoPullRequestsToGitea({
);
console.log(`✅ Mirrored ${successCount}/${pullRequests.length} pull requests to Gitea as enriched issues (${failedCount} failed)`);
// Keep the old watermark when any pull request failed so the next
// pass lists it again.
const unfinishedPullRequests = Math.max(
failedCount,
pullRequests.length - processedPullRequests.length
);
if (unfinishedPullRequests > 0) {
console.warn(
`[Pull Requests] ${unfinishedPullRequests} pull request(s) failed for ${repository.fullName}; keeping the previous watermark so they are retried next run`
);
return undefined;
}
return advanceMetadataSyncCursor(syncCursor, plan.mode, listingStartedAt);
}
export async function mirrorGitRepoLabelsToGitea({
+8 -16
View File
@@ -41,25 +41,17 @@ export async function httpRequest<T = any>(
const responseClone = response.clone();
if (!response.ok) {
let errorMessage = `HTTP ${response.status}: ${response.statusText}`;
// The message carries the status only. The upstream body is kept on
// `HttpError.response` for logs and for callers that inspect it, but
// it never goes into the message: messages can reach API clients
// through createSecureErrorResponse, and a URL under the user's
// control would otherwise turn any error into a readable proxy of
// whatever host it pointed at.
const errorMessage = `HTTP ${response.status}: ${response.statusText}`;
let responseText = '';
try {
responseText = await responseClone.text();
if (responseText) {
// Try to parse as JSON for better error messages
try {
const errorData = JSON.parse(responseText);
if (errorData.message) {
errorMessage = `HTTP ${response.status}: ${errorData.message}`;
} else {
errorMessage += ` - ${responseText}`;
}
} catch {
// Not JSON, use as-is
errorMessage += ` - ${responseText}`;
}
}
} catch {
// Ignore text parsing errors
}
@@ -104,7 +96,7 @@ export async function httpRequest<T = any>(
}
throw new HttpError(
`Failed to parse JSON response from ${url}: ${jsonError instanceof Error ? jsonError.message : String(jsonError)}. Response: ${responseText.substring(0, 200)}${responseText.length > 200 ? '...' : ''}`,
`Failed to parse JSON response from ${url}: ${jsonError instanceof Error ? jsonError.message : String(jsonError)}`,
response.status,
response.statusText,
responseText
+144
View File
@@ -0,0 +1,144 @@
/**
* Incremental metadata sync watermarks (#449): parsing, serialisation and
* the full vs incremental decision.
*/
import { describe, expect, test } from "bun:test";
import {
FULL_METADATA_PASS_INTERVAL_MS,
INCREMENTAL_SYNC_SAFETY_MARGIN_MS,
advanceMetadataSyncCursor,
createDefaultMetadataState,
parseRepositoryMetadataState,
planMetadataPass,
serializeRepositoryMetadataState,
} from "./metadata-state";
const HOUR = 60 * 60 * 1000;
const DAY = 24 * HOUR;
describe("sync cursor parsing", () => {
test("a row written before #449 loads with no cursors", () => {
const legacy = JSON.stringify({
components: { releases: true, issues: true, pullRequests: true, labels: true, milestones: true },
lastSyncedAt: "2026-09-01T00:00:00.000Z",
});
const state = parseRepositoryMetadataState(legacy);
expect(state.syncCursors).toEqual({});
expect(state.components.issues).toBe(true);
expect(state.lastSyncedAt).toBe("2026-09-01T00:00:00.000Z");
});
test("the default state has no cursors", () => {
expect(createDefaultMetadataState().syncCursors).toEqual({});
expect(parseRepositoryMetadataState(null).syncCursors).toEqual({});
expect(parseRepositoryMetadataState("not json").syncCursors).toEqual({});
});
test("cursors survive a round trip", () => {
const state = createDefaultMetadataState();
state.syncCursors.issues = {
lastPassStartedAt: "2026-09-20T10:00:00.000Z",
lastFullPassStartedAt: "2026-09-18T10:00:00.000Z",
};
state.syncCursors.pullRequests = {
lastPassStartedAt: "2026-09-20T11:00:00.000Z",
lastFullPassStartedAt: "2026-09-20T11:00:00.000Z",
};
const reparsed = parseRepositoryMetadataState(serializeRepositoryMetadataState(state));
expect(reparsed.syncCursors).toEqual(state.syncCursors);
});
test("malformed cursors are dropped instead of failing the row", () => {
const raw = {
components: {},
syncCursors: {
issues: { lastPassStartedAt: "yesterday", lastFullPassStartedAt: "2026-09-18T10:00:00.000Z" },
pullRequests: { lastPassStartedAt: "2026-09-20T11:00:00.000Z" },
},
};
expect(parseRepositoryMetadataState(raw).syncCursors).toEqual({});
expect(parseRepositoryMetadataState({ syncCursors: "nope" }).syncCursors).toEqual({});
expect(parseRepositoryMetadataState({ syncCursors: { issues: 42 } }).syncCursors).toEqual({});
});
});
describe("planMetadataPass", () => {
const now = new Date("2026-09-23T12:00:00.000Z");
test("no cursor means a full pass", () => {
const plan = planMetadataPass(undefined, now);
expect(plan.mode).toBe("full");
});
test("a recent cursor gives an incremental pass with the safety margin", () => {
const lastPass = new Date(now.getTime() - 2 * HOUR);
const plan = planMetadataPass(
{
lastPassStartedAt: lastPass.toISOString(),
lastFullPassStartedAt: new Date(now.getTime() - 3 * DAY).toISOString(),
},
now
);
expect(plan).toEqual({
mode: "incremental",
since: new Date(lastPass.getTime() - INCREMENTAL_SYNC_SAFETY_MARGIN_MS).toISOString(),
});
expect(INCREMENTAL_SYNC_SAFETY_MARGIN_MS).toBeGreaterThanOrEqual(5 * 60 * 1000);
});
test("a full pass older than the interval forces a full pass", () => {
const plan = planMetadataPass(
{
lastPassStartedAt: new Date(now.getTime() - HOUR).toISOString(),
lastFullPassStartedAt: new Date(now.getTime() - FULL_METADATA_PASS_INTERVAL_MS).toISOString(),
},
now
);
expect(plan.mode).toBe("full");
expect(FULL_METADATA_PASS_INTERVAL_MS).toBe(7 * DAY);
});
test("a full pass just inside the interval stays incremental", () => {
const plan = planMetadataPass(
{
lastPassStartedAt: new Date(now.getTime() - HOUR).toISOString(),
lastFullPassStartedAt: new Date(now.getTime() - FULL_METADATA_PASS_INTERVAL_MS + HOUR).toISOString(),
},
now
);
expect(plan.mode).toBe("incremental");
});
test("a watermark in the future is not trusted", () => {
const plan = planMetadataPass(
{
lastPassStartedAt: new Date(now.getTime() + DAY).toISOString(),
lastFullPassStartedAt: now.toISOString(),
},
now
);
expect(plan.mode).toBe("full");
});
});
describe("advanceMetadataSyncCursor", () => {
const started = new Date("2026-09-23T12:00:00.000Z");
test("a full pass moves both timestamps", () => {
expect(advanceMetadataSyncCursor(undefined, "full", started)).toEqual({
lastPassStartedAt: started.toISOString(),
lastFullPassStartedAt: started.toISOString(),
});
});
test("an incremental pass keeps the last full pass time", () => {
const previous = {
lastPassStartedAt: "2026-09-23T06:00:00.000Z",
lastFullPassStartedAt: "2026-09-20T06:00:00.000Z",
};
expect(advanceMetadataSyncCursor(previous, "incremental", started)).toEqual({
lastPassStartedAt: started.toISOString(),
lastFullPassStartedAt: "2026-09-20T06:00:00.000Z",
});
});
});
+111
View File
@@ -18,12 +18,52 @@ export interface AcknowledgedDeletion {
giteaSha: string;
}
/**
* Watermark for one metadata kind that can be fetched incrementally
* from GitHub (#449). Both values are the time the GitHub listing
* started, not when the pass finished, so anything that changed while
* a long pass was running is picked up by the next one.
*/
export interface MetadataSyncCursor {
/** Start of the last pass (full or incremental) that completed cleanly. */
lastPassStartedAt: string;
/** Start of the last full pass that completed cleanly. */
lastFullPassStartedAt: string;
}
export interface MetadataSyncCursors {
issues?: MetadataSyncCursor;
pullRequests?: MetadataSyncCursor;
}
export interface RepositoryMetadataState {
components: MetadataComponentsState;
lastSyncedAt?: string;
acknowledgedDeletions: AcknowledgedDeletion[];
syncCursors: MetadataSyncCursors;
}
/**
* How far before the stored watermark an incremental listing starts.
* Covers clock skew between this host and GitHub and items whose
* updated_at was written just before the previous listing began.
* Reprocessing a few items twice is harmless: every write is matched
* by its [GH-ISSUE #N] / [PR #N] marker and comments by their id.
*/
export const INCREMENTAL_SYNC_SAFETY_MARGIN_MS = 10 * 60 * 1000;
/**
* A full pass still runs when the last one is older than this. It
* catches what `since` cannot see, such as a label renamed on GitHub
* (which does not touch the issues that carry it) or an issue removed
* on the Gitea side by hand.
*/
export const FULL_METADATA_PASS_INTERVAL_MS = 7 * 24 * 60 * 60 * 1000;
export type MetadataPassPlan =
| { mode: "full"; reason: string }
| { mode: "incremental"; since: string };
const defaultComponents: MetadataComponentsState = {
releases: false,
issues: false,
@@ -36,6 +76,70 @@ export function createDefaultMetadataState(): RepositoryMetadataState {
return {
components: { ...defaultComponents },
acknowledgedDeletions: [],
syncCursors: {},
};
}
function isValidTimestamp(value: unknown): value is string {
return typeof value === "string" && Number.isFinite(Date.parse(value));
}
function parseSyncCursor(raw: unknown): MetadataSyncCursor | undefined {
if (!raw || typeof raw !== "object") return undefined;
const { lastPassStartedAt, lastFullPassStartedAt } = raw as {
lastPassStartedAt?: unknown;
lastFullPassStartedAt?: unknown;
};
if (!isValidTimestamp(lastPassStartedAt) || !isValidTimestamp(lastFullPassStartedAt)) {
return undefined;
}
return { lastPassStartedAt, lastFullPassStartedAt };
}
/**
* Decide whether the next issues or pull request pass can ask GitHub
* only for what changed. Without a usable watermark, or when the last
* full pass is too old, the pass is full.
*/
export function planMetadataPass(
cursor: MetadataSyncCursor | undefined,
now: Date = new Date()
): MetadataPassPlan {
if (!cursor) {
return { mode: "full", reason: "no previous completed pass" };
}
const nowMs = now.getTime();
const lastPassMs = Date.parse(cursor.lastPassStartedAt);
const lastFullMs = Date.parse(cursor.lastFullPassStartedAt);
if (!Number.isFinite(lastPassMs) || !Number.isFinite(lastFullMs)) {
return { mode: "full", reason: "stored watermark is not a valid date" };
}
if (lastPassMs > nowMs + INCREMENTAL_SYNC_SAFETY_MARGIN_MS) {
return { mode: "full", reason: "stored watermark is in the future" };
}
if (nowMs - lastFullMs >= FULL_METADATA_PASS_INTERVAL_MS) {
return { mode: "full", reason: "last full pass is older than 7 days" };
}
return {
mode: "incremental",
since: new Date(lastPassMs - INCREMENTAL_SYNC_SAFETY_MARGIN_MS).toISOString(),
};
}
/**
* The watermark to store after a pass that completed without failures.
* `startedAt` is the time the GitHub listing started.
*/
export function advanceMetadataSyncCursor(
previous: MetadataSyncCursor | undefined,
mode: MetadataPassPlan["mode"],
startedAt: Date
): MetadataSyncCursor {
const started = startedAt.toISOString();
return {
lastPassStartedAt: started,
lastFullPassStartedAt:
mode === "full" || !previous ? started : previous.lastFullPassStartedAt,
};
}
@@ -93,6 +197,13 @@ export function parseRepositoryMetadataState(
);
}
if (parsed.syncCursors && typeof parsed.syncCursors === "object") {
const issues = parseSyncCursor(parsed.syncCursors.issues);
const pullRequests = parseSyncCursor(parsed.syncCursors.pullRequests);
if (issues) base.syncCursors.issues = issues;
if (pullRequests) base.syncCursors.pullRequests = pullRequests;
}
return base;
}
+3 -2
View File
@@ -1,4 +1,5 @@
import type { AppriseConfig } from "@/types/config";
import { safeFetch } from "@/lib/utils/outbound-url";
import type { NotificationEvent } from "./ntfy";
export async function sendAppriseNotification(config: AppriseConfig, event: NotificationEvent): Promise<void> {
@@ -10,6 +11,6 @@ export async function sendAppriseNotification(config: AppriseConfig, event: Noti
type: event.type === "sync_error" ? "failure" : "success",
tag: config.tag || undefined,
});
const resp = await fetch(url, { method: "POST", body, headers });
if (!resp.ok) throw new Error(`Apprise error: ${resp.status} ${await resp.text()}`);
const resp = await safeFetch(url, { method: "POST", body, headers });
if (!resp.ok) throw new Error(`Apprise error: ${resp.status}`);
}
+3 -2
View File
@@ -1,4 +1,5 @@
import type { GotifyConfig } from "@/types/config";
import { safeFetch } from "@/lib/utils/outbound-url";
import type { NotificationEvent } from "./ntfy";
export async function sendGotifyNotification(config: GotifyConfig, event: NotificationEvent): Promise<void> {
@@ -12,6 +13,6 @@ export async function sendGotifyNotification(config: GotifyConfig, event: Notifi
message: event.message,
priority: event.type === "sync_error" ? 8 : (config.priority ?? 5),
});
const resp = await fetch(url, { method: "POST", body, headers });
if (!resp.ok) throw new Error(`Gotify error: ${resp.status} ${await resp.text()}`);
const resp = await safeFetch(url, { method: "POST", body, headers });
if (!resp.ok) throw new Error(`Gotify error: ${resp.status}`);
}
+3 -2
View File
@@ -1,4 +1,5 @@
import type { NtfyConfig } from "@/types/config";
import { safeFetch } from "@/lib/utils/outbound-url";
export interface NotificationEvent {
title: string;
@@ -16,6 +17,6 @@ export async function sendNtfyNotification(config: NtfyConfig, event: Notificati
if (config.token) {
headers["Authorization"] = `Bearer ${config.token}`;
}
const resp = await fetch(url, { method: "POST", body: event.message, headers });
if (!resp.ok) throw new Error(`Ntfy error: ${resp.status} ${await resp.text()}`);
const resp = await safeFetch(url, { method: "POST", body: event.message, headers });
if (!resp.ok) throw new Error(`Ntfy error: ${resp.status}`);
}
+3 -2
View File
@@ -1,4 +1,5 @@
import { createHmac } from "node:crypto";
import { safeFetch } from "@/lib/utils/outbound-url";
import type { WebhookConfig } from "@/types/config";
import type { NotificationEvent } from "./ntfy";
@@ -16,6 +17,6 @@ export async function sendWebhookNotification(config: WebhookConfig, event: Noti
const signature = createHmac("sha256", config.secret).update(body).digest("hex");
headers["X-Webhook-Signature"] = `sha256=${signature}`;
}
const resp = await fetch(config.url, { method: "POST", body, headers });
if (!resp.ok) throw new Error(`Webhook error: ${resp.status} ${await resp.text()}`);
const resp = await safeFetch(config.url, { method: "POST", body, headers });
if (!resp.ok) throw new Error(`Webhook error: ${resp.status}`);
}
+68
View File
@@ -0,0 +1,68 @@
/**
* sourceFetch goes through the outbound guard (GHSA-p7w3-46pg-mv6h): the
* source URL is stored by a signed in user, so it gets the same treatment
* as every other user supplied URL, and the upstream body never lands in
* the error message.
*/
import { afterEach, describe, expect, it } from "bun:test";
import { OutboundUrlError } from "@/lib/utils/outbound-url";
import { sourceFetch, SourceApiError } from "./http";
const realFetch = globalThis.fetch;
afterEach(() => {
globalThis.fetch = realFetch;
});
describe("sourceFetch", () => {
it("refuses a link local source before any request leaves", async () => {
let called = false;
globalThis.fetch = (async () => {
called = true;
return new Response("{}", { status: 200 });
}) as unknown as typeof fetch;
await expect(
sourceFetch("http://169.254.169.254/api/v1/orgs/x")
).rejects.toBeInstanceOf(OutboundUrlError);
expect(called).toBe(false);
});
it("never follows redirects", async () => {
let redirect: RequestRedirect | undefined;
globalThis.fetch = (async (_input: any, init?: RequestInit) => {
redirect = init?.redirect;
return new Response("[]", {
status: 200,
headers: { "content-type": "application/json" },
});
}) as typeof fetch;
const { data } = await sourceFetch<unknown[]>("http://192.168.1.10:3000/api/v1/user/repos");
expect(data).toEqual([]);
expect(redirect).toBe("manual");
});
it("keeps the upstream body out of the error message", async () => {
globalThis.fetch = (async () =>
new Response(JSON.stringify({ message: "Forbidden: token lacks scope. cluster=prod" }), {
status: 403,
headers: { "content-type": "application/json" },
})) as unknown as typeof fetch;
let caught: unknown;
try {
await sourceFetch("http://192.168.1.10:3000/api/v1/orgs/any/repos");
} catch (error) {
caught = error;
}
expect(caught).toBeInstanceOf(SourceApiError);
const error = caught as SourceApiError;
expect(error.status).toBe(403);
expect(error.message).toBe(
"Request to http://192.168.1.10:3000/api/v1/orgs/any/repos failed with status 403"
);
expect(error.body).toContain("token lacks scope");
});
});
+13 -2
View File
@@ -5,6 +5,8 @@
* of database imports so the adapters stay pure.
*/
import { safeFetch } from "@/lib/utils/outbound-url";
export const DEFAULT_SOURCE_TIMEOUT_MS = 30_000;
export class SourceApiError extends Error {
@@ -41,7 +43,11 @@ export async function sourceFetch<T>(
): Promise<SourceFetchResult<T>> {
const { method = "GET", headers = {}, timeoutMs = DEFAULT_SOURCE_TIMEOUT_MS } = init;
const response = await fetch(url, {
// Source URLs are stored by signed in users, so they go through the same
// outbound guard as every other user supplied URL: link local and
// metadata addresses are refused, plain http is pinned to the checked
// address and redirects are not followed (GHSA-p7w3-46pg-mv6h).
const response = await safeFetch(url, {
method,
headers: { Accept: "application/json", ...headers },
signal: AbortSignal.timeout(timeoutMs),
@@ -54,9 +60,14 @@ export async function sourceFetch<T>(
} catch {
// The status is what matters; the body is only for diagnostics.
}
// The body stays on the error for logs and callers that inspect it and
// never goes into the message, which can reach API clients.
const detail = summarizeErrorBody(body);
if (detail) {
console.warn(`[Source] ${method} ${url} failed with status ${response.status}: ${detail}`);
}
throw new SourceApiError(
`Request to ${url} failed with status ${response.status}${detail ? `: ${detail}` : ""}`,
`Request to ${url} failed with status ${response.status}`,
response.status,
url,
body
+26
View File
@@ -198,3 +198,29 @@ describe("showErrorToast", () => {
expect(calls[0][1].description).toContain("http://10.10.20.45:4321");
});
});
describe("createSecureErrorResponse and errors from other hosts", () => {
test("never forwards the message of an error raised by a request to another host", async () => {
const { createSecureErrorResponse } = await import("./utils");
const { HttpError } = await import("./http-client");
const { SourceApiError } = await import("./source-providers/http");
const remote = new SourceApiError(
"Request to http://10.0.0.9/api/v1/orgs/x failed with status 403: Forbidden",
403,
"http://10.0.0.9/api/v1/orgs/x"
);
const destination = new HttpError("HTTP 404: Not Found", 404, "Not Found");
const local = new Error("Organization not found");
for (const [error, forwarded] of [
[remote, false],
[destination, false],
[local, true],
] as const) {
const response = createSecureErrorResponse(error, "test");
const body = (await response.json()) as { error: string };
expect(body.error === error.message).toBe(forwarded);
}
});
});
+9 -3
View File
@@ -351,9 +351,15 @@ export function createSecureErrorResponse(
/use post method/i,
];
const isSafeError = safeErrorPatterns.some(pattern =>
pattern.test(error.message)
);
// An error raised by a request to another host carries that host's URL
// and status, and the words above are exactly what such hosts put in
// their bodies. Those errors are matched by type, never by text.
const describesRemoteHost =
error.name === "SourceApiError" || error.name === "HttpError";
const isSafeError =
!describesRemoteHost &&
safeErrorPatterns.some(pattern => pattern.test(error.message));
if (isSafeError) {
clientMessage = error.message;
+145
View File
@@ -0,0 +1,145 @@
import { afterEach, describe, expect, it } from "bun:test";
import {
assertSafeOutboundUrl,
isLinkLocalAddress,
OutboundUrlError,
resolveOutboundTarget,
safeFetch,
} from "./outbound-url";
const noResolve = async () => [];
describe("isLinkLocalAddress", () => {
it("matches the IPv4 link local range", () => {
expect(isLinkLocalAddress("169.254.169.254")).toBe(true);
expect(isLinkLocalAddress("169.254.0.1")).toBe(true);
expect(isLinkLocalAddress("169.253.1.1")).toBe(false);
expect(isLinkLocalAddress("10.0.0.1")).toBe(false);
expect(isLinkLocalAddress("192.168.1.10")).toBe(false);
expect(isLinkLocalAddress("127.0.0.1")).toBe(false);
});
it("matches IPv6 link local and the mapped IPv4 forms", () => {
expect(isLinkLocalAddress("fe80::1")).toBe(true);
expect(isLinkLocalAddress("FE9A::1")).toBe(true);
expect(isLinkLocalAddress("febf::1")).toBe(true);
expect(isLinkLocalAddress("fec0::1")).toBe(false);
expect(isLinkLocalAddress("::ffff:169.254.169.254")).toBe(true);
expect(isLinkLocalAddress("::ffff:a9fe:a9fe")).toBe(true);
expect(isLinkLocalAddress("::ffff:192.168.1.1")).toBe(false);
expect(isLinkLocalAddress("::1")).toBe(false);
expect(isLinkLocalAddress("not-an-ip")).toBe(false);
});
});
describe("assertSafeOutboundUrl", () => {
it("allows private networks, localhost and Docker service names", async () => {
for (const url of [
"http://192.168.1.10:3000",
"http://10.0.0.5",
"http://172.16.4.4:8080/api",
"http://localhost:3000",
"http://127.0.0.1:3000",
"http://gitea:3000",
"https://codeberg.org",
]) {
await expect(assertSafeOutboundUrl(url, noResolve)).resolves.toBeInstanceOf(URL);
}
});
it("refuses the metadata address in every spelling", async () => {
for (const url of [
"http://169.254.169.254/latest/meta-data/",
"http://[fe80::1]/",
"http://[::ffff:169.254.169.254]/",
"http://metadata.google.internal/computeMetadata/v1/",
"http://METADATA/",
]) {
await expect(assertSafeOutboundUrl(url, noResolve)).rejects.toBeInstanceOf(OutboundUrlError);
}
});
it("refuses a host name that resolves to a link local address", async () => {
const resolver = async () => ["93.184.216.34", "169.254.169.254"];
await expect(assertSafeOutboundUrl("http://evil.example/", resolver)).rejects.toThrow(
/link local/
);
});
it("leaves unresolvable names to fetch", async () => {
await expect(assertSafeOutboundUrl("http://nope.invalid/", noResolve)).resolves.toBeInstanceOf(
URL
);
});
it("refuses other schemes and malformed URLs", async () => {
await expect(assertSafeOutboundUrl("file:///etc/passwd", noResolve)).rejects.toThrow(
/http and https/
);
await expect(assertSafeOutboundUrl("gopher://x/", noResolve)).rejects.toThrow(/http and https/);
await expect(assertSafeOutboundUrl("not a url", noResolve)).rejects.toThrow(/Invalid URL/);
});
});
describe("resolveOutboundTarget", () => {
it("pins plain http to the checked address and keeps the name in Host", async () => {
const resolver = async () => ["10.0.0.5", "10.0.0.6"];
const target = await resolveOutboundTarget("http://ntfy.local:8080/topic?x=1", resolver);
expect(target.requestUrl).toBe("http://10.0.0.5:8080/topic?x=1");
expect(target.headers).toEqual({ Host: "ntfy.local:8080" });
});
it("brackets a pinned IPv6 address", async () => {
const resolver = async () => ["fd00::10"];
const target = await resolveOutboundTarget("http://gotify.local/message", resolver);
expect(target.requestUrl).toBe("http://[fd00::10]/message");
expect(target.headers).toEqual({ Host: "gotify.local" });
});
it("does not pin https, IP literals or unresolvable names", async () => {
const resolver = async () => ["10.0.0.5"];
expect((await resolveOutboundTarget("https://idp.example.com/x", resolver)).headers).toEqual({});
expect((await resolveOutboundTarget("http://192.168.1.10:3000/", resolver)).requestUrl).toBe(
"http://192.168.1.10:3000/"
);
const unresolved = await resolveOutboundTarget("http://nope.invalid/", noResolve);
expect(unresolved.requestUrl).toBe("http://nope.invalid/");
expect(unresolved.headers).toEqual({});
});
});
describe("safeFetch", () => {
const realFetch = globalThis.fetch;
afterEach(() => {
globalThis.fetch = realFetch;
});
it("never follows redirects, pins the address and refuses blocked hosts before fetching", async () => {
let seen: any = null;
globalThis.fetch = (async (input: any, init?: RequestInit) => {
const headers = new Headers(init?.headers);
seen = { url: String(input), redirect: init?.redirect, host: headers.get("host"), auth: headers.get("authorization") };
return new Response("", { status: 302, headers: { location: "http://169.254.169.254/" } });
}) as typeof fetch;
const resolver = async () => ["10.0.0.5"];
const response = await safeFetch(
"http://ntfy.local/topic",
{ method: "POST", headers: { Authorization: "Bearer t" } },
resolver
);
expect(response.status).toBe(302);
expect(seen).toEqual({
url: "http://10.0.0.5/topic",
redirect: "manual",
host: "ntfy.local",
auth: "Bearer t",
});
seen = null;
await expect(safeFetch("http://169.254.169.254/", {}, noResolve)).rejects.toBeInstanceOf(
OutboundUrlError
);
expect(seen).toBeNull();
});
});
+207
View File
@@ -0,0 +1,207 @@
/**
* Guard for outbound requests to user supplied URLs.
*
* Gitea Mirror talks to hosts on private networks by design: the Gitea it
* mirrors to usually lives on a LAN address, a Docker service name or
* localhost, and so do ntfy and Gotify. Blocking private ranges would break
* the normal deployment, so this guard is deliberately narrow. It refuses
* the addresses that are never a legitimate destination for this app and
* are the classic SSRF targets:
*
* - the IPv4 link local range 169.254.0.0/16, which carries the cloud
* instance metadata service on AWS, GCP, Azure, DigitalOcean and others
* - the IPv6 link local range fe80::/10 and the IPv4 mapped form of the
* same range
* - the well known metadata host names
* - anything that is not plain http or https
*
* Host names are resolved and every address they resolve to is checked, so
* a DNS name pointing at the metadata range is refused too. A plain http
* request is then sent to the address that was checked, with the original
* name in the Host header, so a name that changes its answer between the
* check and the request (DNS rebinding) gains nothing. An https request
* keeps the name: the certificate is validated against it, and the
* metadata services this guard exists for speak no TLS at all. Callers
* that follow redirects must run the check on every hop; `safeFetch` below
* does not follow redirects at all.
*/
import { isIP } from "node:net";
import { lookup } from "node:dns/promises";
export class OutboundUrlError extends Error {
constructor(message: string) {
super(message);
this.name = "OutboundUrlError";
}
}
const BLOCKED_HOSTNAMES = new Set([
"metadata.google.internal",
"metadata.goog",
"metadata",
"instance-data",
"instance-data.ec2.internal",
]);
/** True for an address in 169.254.0.0/16 or fe80::/10 (including v4 mapped). */
export function isLinkLocalAddress(address: string): boolean {
const version = isIP(address);
if (version === 4) {
const [a, b] = address.split(".").map(Number);
return a === 169 && b === 254;
}
if (version === 6) {
const lower = address.toLowerCase();
// ::ffff:169.254.x.y (dotted form)
const mapped = lower.match(/^(?:0*:)*ffff:(\d+\.\d+\.\d+\.\d+)$/);
if (mapped) return isLinkLocalAddress(mapped[1]);
// ::ffff:a9fe:xxxx (hex form of 169.254.0.0/16)
if (/^(?:0*:)*ffff:a9fe:[0-9a-f]{1,4}$/.test(lower)) return true;
// fe80::/10 covers fe80 to febf
return /^fe[89ab][0-9a-f]:/.test(lower);
}
return false;
}
function stripBrackets(hostname: string): string {
return hostname.startsWith("[") && hostname.endsWith("]")
? hostname.slice(1, -1)
: hostname;
}
export type AddressResolver = (hostname: string) => Promise<string[]>;
async function defaultResolver(hostname: string): Promise<string[]> {
try {
const records = await lookup(hostname, { all: true, verbatim: true });
return records.map((record) => record.address);
} catch {
// Unresolvable names are left to fetch, which fails on them anyway.
return [];
}
}
/**
* Parse a user supplied URL and refuse it when it points at a blocked
* destination. Returns the parsed URL on success.
*/
export async function assertSafeOutboundUrl(
rawUrl: string,
resolver: AddressResolver = defaultResolver
): Promise<URL> {
let url: URL;
try {
url = new URL(rawUrl);
} catch {
throw new OutboundUrlError("Invalid URL format");
}
if (url.protocol !== "http:" && url.protocol !== "https:") {
throw new OutboundUrlError("Only http and https URLs are allowed");
}
const hostname = stripBrackets(url.hostname).toLowerCase();
if (!hostname) {
throw new OutboundUrlError("Invalid URL format");
}
if (BLOCKED_HOSTNAMES.has(hostname)) {
throw new OutboundUrlError("Requests to the instance metadata service are not allowed");
}
if (isIP(hostname)) {
if (isLinkLocalAddress(hostname)) {
throw new OutboundUrlError("Requests to link local addresses are not allowed");
}
return url;
}
const addresses = await resolver(hostname);
if (addresses.some(isLinkLocalAddress)) {
throw new OutboundUrlError(
`${hostname} resolves to a link local address, which is not allowed`
);
}
return url;
}
/**
* For routes that store a URL for later use: the reason the URL would be
* refused by the guard, or null when it is fine. A missing scheme is read
* as https, matching `isValidSourceUrl`.
*/
export async function blockedOutboundUrlReason(
raw: string,
resolver: AddressResolver = defaultResolver
): Promise<string | null> {
const trimmed = raw.trim();
if (!trimmed) return null;
const withScheme = /^[a-z][a-z0-9+.-]*:\/\//i.test(trimmed) ? trimmed : `https://${trimmed}`;
try {
await assertSafeOutboundUrl(withScheme, resolver);
return null;
} catch (error) {
if (error instanceof OutboundUrlError) return error.message;
throw error;
}
}
export interface OutboundTarget {
/** The URL as given, parsed and checked. */
url: URL;
/** The URL to actually request: pinned to the checked address for plain http. */
requestUrl: string;
/** Headers the request must carry (the original Host when pinned). */
headers: Record<string, string>;
}
/**
* Check a user supplied URL and decide where the request goes. Plain http
* to a host name is pinned to the first address the check saw; everything
* else is requested as given.
*/
export async function resolveOutboundTarget(
rawUrl: string,
resolver: AddressResolver = defaultResolver
): Promise<OutboundTarget> {
const url = await assertSafeOutboundUrl(rawUrl, resolver);
const hostname = stripBrackets(url.hostname);
if (url.protocol !== "http:" || isIP(hostname)) {
return { url, requestUrl: url.toString(), headers: {} };
}
const [address] = await resolver(hostname.toLowerCase());
if (!address || isLinkLocalAddress(address)) {
// Unresolvable names are left to fetch, which fails on them anyway.
return { url, requestUrl: url.toString(), headers: {} };
}
const pinned = new URL(url.toString());
pinned.hostname = isIP(address) === 6 ? `[${address}]` : address;
return { url, requestUrl: pinned.toString(), headers: { Host: url.host } };
}
/**
* fetch for user supplied URLs: runs the guard first, pins plain http to
* the checked address and never follows a redirect, so a permitted host
* cannot bounce the request to a blocked one.
*/
export async function safeFetch(
rawUrl: string,
init: RequestInit = {},
resolver?: AddressResolver
): Promise<Response> {
const target = await resolveOutboundTarget(rawUrl, resolver);
let headers: HeadersInit;
if (init.headers instanceof Headers || Array.isArray(init.headers)) {
const merged = new Headers(init.headers);
for (const [name, value] of Object.entries(target.headers)) merged.set(name, value);
headers = merged;
} else {
headers = { ...(init.headers ?? {}), ...target.headers };
}
return fetch(target.requestUrl, { ...init, headers, redirect: "manual" });
}
+15
View File
@@ -27,6 +27,7 @@ import {
import { encrypt, decrypt } from "@/lib/utils/encryption";
import { createDefaultConfig } from "@/lib/utils/config-defaults";
import { requireAuthenticatedUserId } from "@/lib/auth-guards";
import { blockedOutboundUrlReason } from "@/lib/utils/outbound-url";
import { notificationConfigSchema } from "@/lib/db/schema";
import {
loadSourceApiItems,
@@ -100,6 +101,13 @@ export const POST: APIRoute = async ({ request, locals }) => {
{ status: 400, headers: { "Content-Type": "application/json" } }
);
}
const blockedDestination = await blockedOutboundUrlReason(giteaConfig.url);
if (blockedDestination) {
return new Response(
JSON.stringify({ success: false, message: blockedDestination }),
{ status: 400, headers: { "Content-Type": "application/json" } }
);
}
}
// Validate the destination provider. GitHub and GitLab targets default to
@@ -137,6 +145,13 @@ export const POST: APIRoute = async ({ request, locals }) => {
{ status: 400, headers: { "Content-Type": "application/json" } }
);
}
const blockedSource = rawSourceUrl ? await blockedOutboundUrlReason(rawSourceUrl) : null;
if (blockedSource) {
return new Response(
JSON.stringify({ success: false, message: blockedSource }),
{ status: 400, headers: { "Content-Type": "application/json" } }
);
}
}
// Fetch existing config — prefer the active config; fall back to most-recently-updated
+33 -2
View File
@@ -1,6 +1,8 @@
import type { APIRoute } from 'astro';
import { httpGet, HttpError } from '@/lib/http-client';
import { createSecureErrorResponse } from '@/lib/utils';
import { requireAuthenticatedUserId } from '@/lib/auth-guards';
import { assertSafeOutboundUrl, resolveOutboundTarget, OutboundUrlError } from '@/lib/utils/outbound-url';
import {
DESTINATION_PROVIDER_LABELS,
isPushDestinationKind,
@@ -81,13 +83,27 @@ async function testPushTarget(
}
}
export const POST: APIRoute = async ({ request }) => {
export const POST: APIRoute = async ({ request, locals }) => {
// The route makes a server side request to a URL from the body, so it is
// for signed in users only, like every other API route that reaches out.
const authResult = await requireAuthenticatedUserId({ request, locals });
if ('response' in authResult) return authResult.response;
try {
const body = await request.json();
const { url, token, username } = body;
const provider = normalizeDestinationProviderKind(body.provider);
if (isPushDestinationKind(provider)) {
const pushUrl = normalizeDestinationBaseUrl(url, provider);
try {
await assertSafeOutboundUrl(pushUrl);
} catch (error) {
if (error instanceof OutboundUrlError) {
return json({ success: false, message: error.message }, 400);
}
throw error;
}
return testPushTarget(provider, { url, token, username });
}
@@ -96,10 +112,24 @@ export const POST: APIRoute = async ({ request }) => {
}
// Normalize the URL (remove trailing slash if present)
const baseUrl = url.endsWith('/') ? url.slice(0, -1) : url;
const givenUrl = String(url).endsWith('/') ? String(url).slice(0, -1) : String(url);
// Checked and, for plain http, pinned to the checked address so the
// two requests below go where the check looked.
let target: Awaited<ReturnType<typeof resolveOutboundTarget>>;
try {
target = await resolveOutboundTarget(givenUrl);
} catch (error) {
if (error instanceof OutboundUrlError) {
return json({ success: false, message: error.message }, 400);
}
throw error;
}
const baseUrl = target.requestUrl.replace(/\/$/, '');
// Test the connection by fetching the authenticated user
const response = await httpGet(`${baseUrl}/api/v1/user`, {
...target.headers,
'Authorization': `token ${token}`,
'Accept': 'application/json',
});
@@ -114,6 +144,7 @@ export const POST: APIRoute = async ({ request }) => {
let serverInfo: ReturnType<typeof parseServerInfo> | undefined;
try {
const versionResp = await httpGet(`${baseUrl}/api/v1/version`, {
...target.headers,
'Accept': 'application/json',
});
if (typeof versionResp.data?.version === 'string') {
+4 -4
View File
@@ -55,7 +55,7 @@ describe("GitHub Test Connection API", () => {
body: JSON.stringify({})
});
const response = await POST({ request } as any);
const response = await POST({ request, locals: { user: { id: "user-1" } } } as any);
expect(response.status).toBe(400);
@@ -75,7 +75,7 @@ describe("GitHub Test Connection API", () => {
})
});
const response = await POST({ request } as any);
const response = await POST({ request, locals: { user: { id: "user-1" } } } as any);
expect(response.status).toBe(200);
@@ -101,7 +101,7 @@ describe("GitHub Test Connection API", () => {
})
});
const response = await POST({ request } as any);
const response = await POST({ request, locals: { user: { id: "user-1" } } } as any);
expect(response.status).toBe(400);
@@ -124,7 +124,7 @@ describe("GitHub Test Connection API", () => {
})
});
const response = await POST({ request } as any);
const response = await POST({ request, locals: { user: { id: "user-1" } } } as any);
expect(response.status).toBe(500);
+19 -2
View File
@@ -1,5 +1,7 @@
import type { APIRoute } from "astro";
import { createSecureErrorResponse } from "@/lib/utils";
import { requireAuthenticatedUserId } from "@/lib/auth-guards";
import { assertSafeOutboundUrl, OutboundUrlError } from "@/lib/utils/outbound-url";
import {
SOURCE_PROVIDER_LABELS,
createSourceProvider,
@@ -20,7 +22,12 @@ function json(body: unknown, status: number): Response {
* compatibility, but serves every source provider: the body carries the
* provider and, for GitLab and Gitea, the instance URL.
*/
export const POST: APIRoute = async ({ request }) => {
export const POST: APIRoute = async ({ request, locals }) => {
// The route makes a server side request to a URL from the body, so it is
// for signed in users only, like every other API route that reaches out.
const authResult = await requireAuthenticatedUserId({ request, locals });
if ("response" in authResult) return authResult.response;
const body = await request.json().catch(() => ({}));
const { token, username, provider: rawProvider, url } = body ?? {};
const provider = normalizeSourceProviderKind(rawProvider);
@@ -31,10 +38,20 @@ export const POST: APIRoute = async ({ request }) => {
return json({ success: false, message: `${label} token is required` }, 400);
}
const instanceUrl = normalizeSourceUrl(url, provider);
try {
await assertSafeOutboundUrl(instanceUrl);
} catch (error) {
if (error instanceof OutboundUrlError) {
return json({ success: false, message: error.message }, 400);
}
throw error;
}
// GitHub honors GH_API_URL / GITHUB_API_URL for GHES / GHEC inside the adapter.
const sourceProvider = createSourceProvider({
provider,
url: normalizeSourceUrl(url, provider),
url: instanceUrl,
username: typeof username === "string" ? username : "",
token,
});
+3
View File
@@ -76,6 +76,9 @@ export const POST: APIRoute = async ({ request, locals }) => {
);
}
// Clearing the whole state also drops the incremental sync
// watermarks (#449), so the next issues and pull request passes are
// full ones.
await db
.update(repositories)
.set({
+5
View File
@@ -14,6 +14,7 @@ import {
sourceIdentity,
} from "@/lib/config-locks";
import { requireAuthenticatedUserId } from "@/lib/auth-guards";
import { blockedOutboundUrlReason } from "@/lib/utils/outbound-url";
import { createSecureErrorResponse } from "@/lib/utils";
import {
deleteSource,
@@ -87,6 +88,10 @@ export const PUT: APIRoute = async ({ params, request, locals }) => {
400
);
}
if (provider !== "github" && typeof body.url === "string" && body.url.trim()) {
const blocked = await blockedOutboundUrlReason(body.url);
if (blocked) return jsonError(blocked, 400);
}
const repositoryCount = await countRepositoriesForSource(userId, sourceId);
+16
View File
@@ -211,6 +211,22 @@ describe("POST /api/sources", () => {
expect(data.message).toContain("GitLab URL");
});
test("refuses a source URL that points at the instance metadata service", async () => {
for (const url of ["http://169.254.169.254", "metadata.google.internal"]) {
const response = await post({
provider: "gitea",
url,
username: "me",
token: "tok",
});
expect(response.status).toBe(400);
const data = await response.json();
expect(data.success).toBe(false);
expect(data.message).toMatch(/link local|metadata service/);
}
expect(sourceRows).toHaveLength(0);
});
test("answers 409 when the same source is already connected", async () => {
sourceRows = [
{
+8
View File
@@ -5,6 +5,7 @@ import {
isValidSourceUrl,
} from "@/lib/source-providers/kinds";
import { requireAuthenticatedUserId } from "@/lib/auth-guards";
import { blockedOutboundUrlReason } from "@/lib/utils/outbound-url";
import { createSecureErrorResponse } from "@/lib/utils";
import { createSource } from "@/lib/sources";
import {
@@ -71,6 +72,13 @@ export const POST: APIRoute = async ({ request, locals }) => {
}
);
}
const blocked = rawSourceUrl ? await blockedOutboundUrlReason(rawSourceUrl) : null;
if (blocked) {
return new Response(
JSON.stringify({ success: false, message: blocked }),
{ status: 400, headers: { "Content-Type": "application/json" } }
);
}
}
let created: Awaited<ReturnType<typeof createSource>>;
+5 -1
View File
@@ -1,6 +1,7 @@
import type { APIContext } from "astro";
import { createSecureErrorResponse } from "@/lib/utils";
import { requireAuth } from "@/lib/utils/auth-helpers";
import { safeFetch, OutboundUrlError } from "@/lib/utils/outbound-url";
// POST /api/sso/discover - Discover OIDC configuration from issuer URL
export async function POST(context: APIContext) {
@@ -45,13 +46,16 @@ export async function POST(context: APIContext) {
let response: Response;
try {
response = await fetch(discoveryUrl, {
response = await safeFetch(discoveryUrl, {
signal: controller.signal,
headers: {
'Accept': 'application/json',
}
});
} catch (fetchError) {
if (fetchError instanceof OutboundUrlError) {
throw fetchError;
}
if (fetchError instanceof Error && fetchError.name === 'AbortError') {
throw new Error(`Request timeout: The OIDC provider at ${trimmedIssuer} did not respond within 10 seconds`);
}
+162
View File
@@ -0,0 +1,162 @@
/**
* SSO provider ownership (GHSA-2hpx-83vg-gm45).
*
* The routes authenticated the caller but read, updated and deleted
* providers by id across every user, and the list returned the OIDC client
* secret. Every query is now scoped to the caller and the secret stays on
* the server.
*
* Runs in an isolated child so the @/lib/db and auth mocks cannot leak into
* other files (same harness as ../organizations/[id]/status.test.ts).
*/
import { describe, test, expect, mock } from "bun:test";
import { Database } from "bun:sqlite";
import { drizzle } from "drizzle-orm/bun-sqlite";
import { readFileSync } from "node:fs";
import { join } from "node:path";
import * as schema from "@/lib/db/schema";
const CHILD_FLAG = "GM_SSO_PROVIDERS_ROUTE_ISOLATED";
const isChild = !!process.env[CHILD_FLAG];
if (!isChild) {
test("sso providers route (ownership) - isolated child suite", () => {
const res = Bun.spawnSync({
cmd: [process.execPath, "test", import.meta.path],
env: { ...process.env, [CHILD_FLAG]: "1" },
stdout: "pipe",
stderr: "pipe",
});
if (res.exitCode !== 0) {
console.error(res.stdout.toString());
console.error(res.stderr.toString());
}
expect(res.exitCode).toBe(0);
}, 60_000);
}
const sqlite = new Database(":memory:");
const db = drizzle({ client: sqlite });
function applyMigrations(): void {
const folder = join(process.cwd(), "drizzle");
const journal = JSON.parse(readFileSync(join(folder, "meta", "_journal.json"), "utf8")) as {
entries: { tag: string }[];
};
for (const entry of journal.entries) {
const sql = readFileSync(join(folder, `${entry.tag}.sql`), "utf8");
for (const statement of sql.split("--> statement-breakpoint")) {
if (statement.trim()) sqlite.run(statement);
}
}
}
let currentUserId = "alice";
if (isChild) {
applyMigrations();
sqlite.run("PRAGMA foreign_keys = OFF");
mock.module("@/lib/utils/auth-helpers", () => ({
requireAuth: mock(async () => ({ user: { id: currentUserId }, response: null })),
}));
mock.module("@/lib/db", () => ({ ...schema, db }));
}
const routes = isChild
? await import("./providers")
: ({} as typeof import("./providers"));
function seed(id: string, userId: string, providerId: string) {
sqlite.run(
`INSERT INTO sso_providers (id, issuer, domain, oidc_config, user_id, provider_id, created_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)`,
[
id,
"https://idp.example.com",
"example.com",
JSON.stringify({
clientId: `client-${id}`,
clientSecret: `secret-${id}`,
authorizationEndpoint: "https://idp.example.com/auth",
tokenEndpoint: "https://idp.example.com/token",
jwksEndpoint: "https://idp.example.com/jwks",
userInfoEndpoint: "https://idp.example.com/userinfo",
scopes: ["openid"],
}),
userId,
providerId,
Date.now(),
Date.now(),
]
);
}
function ctx(method: string, query = "", body?: unknown) {
return {
request: new Request(`http://localhost/api/sso/providers${query}`, {
method,
headers: { "Content-Type": "application/json" },
body: body === undefined ? undefined : JSON.stringify(body),
}),
} as any;
}
describe.skipIf(!isChild)("SSO providers ownership", () => {
test("GET lists only the caller's providers and never the client secret", async () => {
seed("p-alice", "alice", "alice-idp");
seed("p-bob", "bob", "bob-idp");
currentUserId = "alice";
const response = await routes.GET(ctx("GET"));
expect(response.status).toBe(200);
const list = (await response.json()) as any[];
expect(list.map((p) => p.id)).toEqual(["p-alice"]);
expect(JSON.stringify(list)).not.toContain("secret-");
expect(list[0].oidcConfig.hasClientSecret).toBe(true);
expect(list[0].oidcConfig.clientId).toBe("client-p-alice");
});
test("PUT on another user's provider answers 404 and changes nothing", async () => {
currentUserId = "alice";
const response = await routes.PUT(ctx("PUT", "?id=p-bob", { domain: "evil.example" }));
expect(response.status).toBe(404);
const row = sqlite.query("SELECT domain, oidc_config FROM sso_providers WHERE id = 'p-bob'").get() as any;
expect(row.domain).toBe("example.com");
expect(JSON.parse(row.oidc_config).clientSecret).toBe("secret-p-bob");
});
test("PUT with an empty secret keeps the stored one", async () => {
currentUserId = "alice";
const response = await routes.PUT(
ctx("PUT", "?id=p-alice", { domain: "alice.example", clientSecret: "" })
);
const body = await response.json();
expect([response.status, body]).toEqual([200, expect.anything()]);
expect(body.oidcConfig.clientSecret).toBeUndefined();
expect(body.oidcConfig.hasClientSecret).toBe(true);
const row = sqlite.query("SELECT domain, oidc_config FROM sso_providers WHERE id = 'p-alice'").get() as any;
expect(row.domain).toBe("alice.example");
expect(JSON.parse(row.oidc_config).clientSecret).toBe("secret-p-alice");
});
test("DELETE on another user's provider answers 404 and keeps the row", async () => {
currentUserId = "alice";
const response = await routes.DELETE(ctx("DELETE", "?id=p-bob"));
expect(response.status).toBe(404);
const count = sqlite.query("SELECT count(*) AS n FROM sso_providers WHERE id = 'p-bob'").get() as any;
expect(count.n).toBe(1);
});
test("DELETE on the caller's own provider works", async () => {
currentUserId = "bob";
const response = await routes.DELETE(ctx("DELETE", "?id=p-bob"));
expect(response.status).toBe(200);
const count = sqlite.query("SELECT count(*) AS n FROM sso_providers WHERE id = 'p-bob'").get() as any;
expect(count.n).toBe(0);
});
});
+41 -25
View File
@@ -3,23 +3,47 @@ import { createSecureErrorResponse } from "@/lib/utils";
import { requireAuth } from "@/lib/utils/auth-helpers";
import { db, ssoProviders } from "@/lib/db";
import { nanoid } from "nanoid";
import { eq } from "drizzle-orm";
import { and, eq } from "drizzle-orm";
import { normalizeOidcProviderConfig, OidcConfigError, type RawOidcConfig } from "@/lib/sso/oidc-config";
/**
* Providers are owned by the user who created them. Every read and write
* here is scoped to the caller, and the client secret never leaves the
* server: the API reports whether one is stored and the form keeps it by
* sending an empty value.
*/
export function formatProviderForClient(provider: {
oidcConfig: string | null;
[key: string]: unknown;
}) {
const oidcConfig = provider.oidcConfig ? JSON.parse(provider.oidcConfig) : undefined;
if (oidcConfig && typeof oidcConfig === "object") {
const { clientSecret, ...rest } = oidcConfig as { clientSecret?: unknown };
return {
...provider,
oidcConfig: { ...rest, hasClientSecret: typeof clientSecret === "string" && clientSecret.length > 0 },
samlConfig: (provider as any).samlConfig ? JSON.parse((provider as any).samlConfig) : undefined,
};
}
return {
...provider,
oidcConfig,
samlConfig: (provider as any).samlConfig ? JSON.parse((provider as any).samlConfig) : undefined,
};
}
// GET /api/sso/providers - List all SSO providers
export async function GET(context: APIContext) {
try {
const { user, response } = await requireAuth(context);
if (response) return response;
const providers = await db.select().from(ssoProviders);
const providers = await db
.select()
.from(ssoProviders)
.where(eq(ssoProviders.userId, user.id));
// Parse JSON fields before sending
const formattedProviders = providers.map(provider => ({
...provider,
oidcConfig: provider.oidcConfig ? JSON.parse(provider.oidcConfig) : undefined,
samlConfig: (provider as any).samlConfig ? JSON.parse((provider as any).samlConfig) : undefined,
}));
const formattedProviders = providers.map(formatProviderForClient);
return new Response(JSON.stringify(formattedProviders), {
status: 200,
@@ -142,12 +166,7 @@ export async function POST(context: APIContext) {
})
.returning();
// Parse JSON fields before sending
const formattedProvider = {
...newProvider,
oidcConfig: newProvider.oidcConfig ? JSON.parse(newProvider.oidcConfig) : undefined,
samlConfig: (newProvider as any).samlConfig ? JSON.parse((newProvider as any).samlConfig) : undefined,
};
const formattedProvider = formatProviderForClient(newProvider);
return new Response(JSON.stringify(formattedProvider), {
status: 201,
@@ -193,11 +212,11 @@ export async function PUT(context: APIContext) {
organizationId,
} = body;
// Get existing provider
// Get existing provider; another user's provider reads as not found
const [existingProvider] = await db
.select()
.from(ssoProviders)
.where(eq(ssoProviders.id, providerId))
.where(and(eq(ssoProviders.id, providerId), eq(ssoProviders.userId, user.id)))
.limit(1);
if (!existingProvider) {
@@ -228,7 +247,9 @@ export async function PUT(context: APIContext) {
const mergedConfig: RawOidcConfig = {
clientId: clientId ?? existingConfig.clientId,
clientSecret: clientSecret ?? existingConfig.clientSecret,
// The list endpoint never returns the secret, so the form sends an
// empty value to keep the stored one.
clientSecret: clientSecret ? clientSecret : existingConfig.clientSecret,
authorizationEndpoint: authorizationEndpoint ?? existingConfig.authorizationEndpoint,
tokenEndpoint: tokenEndpoint ?? existingConfig.tokenEndpoint,
jwksEndpoint: jwksEndpoint ?? existingConfig.jwksEndpoint,
@@ -269,15 +290,10 @@ export async function PUT(context: APIContext) {
organizationId: organizationId !== undefined ? organizationId : existingProvider.organizationId,
updatedAt: new Date(),
})
.where(eq(ssoProviders.id, providerId))
.where(and(eq(ssoProviders.id, providerId), eq(ssoProviders.userId, user.id)))
.returning();
// Parse JSON fields before sending
const formattedProvider = {
...updatedProvider,
oidcConfig: JSON.parse(updatedProvider.oidcConfig),
samlConfig: (updatedProvider as any).samlConfig ? JSON.parse((updatedProvider as any).samlConfig) : undefined,
};
const formattedProvider = formatProviderForClient(updatedProvider);
return new Response(JSON.stringify(formattedProvider), {
status: 200,
@@ -309,7 +325,7 @@ export async function DELETE(context: APIContext) {
const deleted = await db
.delete(ssoProviders)
.where(eq(ssoProviders.id, providerId))
.where(and(eq(ssoProviders.id, providerId), eq(ssoProviders.userId, user.id)))
.returning();
if (deleted.length === 0) {
+108
View File
@@ -0,0 +1,108 @@
/**
* The two connection test routes make a server side request to a URL from
* the request body. They were the only non-public API routes without an
* auth guard (GHSA-9m33-xfrc-5jxw, GHSA-6m23-28hh-gjh2). Both now require a
* signed in user and refuse link local and metadata addresses.
*
* Runs in an isolated child: the auth module is replaced here and bun's
* mock.module is process wide.
*/
import { describe, test, expect, mock } from "bun:test";
const CHILD_FLAG = "GM_TEST_CONNECTION_GUARD_ISOLATED";
const isChild = !!process.env[CHILD_FLAG];
if (!isChild) {
test("connection test routes (auth guard) - isolated child suite", () => {
const res = Bun.spawnSync({
cmd: [process.execPath, "test", import.meta.path],
env: { ...process.env, [CHILD_FLAG]: "1" },
stdout: "pipe",
stderr: "pipe",
});
if (res.exitCode !== 0) {
console.error(res.stdout.toString());
console.error(res.stderr.toString());
}
expect(res.exitCode).toBe(0);
}, 60_000);
}
if (isChild) {
// No session anywhere: the guard must answer 401 on its own.
mock.module("@/lib/auth", () => ({
auth: { api: { getSession: async () => null } },
}));
}
const gitea = isChild ? await import("./gitea/test-connection") : ({} as any);
const github = isChild ? await import("./github/test-connection") : ({} as any);
function post(path: string, body: unknown, locals?: unknown) {
return {
request: new Request(`http://localhost/api/${path}`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(body),
}),
locals: locals ?? {},
} as any;
}
describe.skipIf(!isChild)("connection test routes", () => {
const fetchCalls: string[] = [];
const realFetch = globalThis.fetch;
globalThis.fetch = (async (input: any) => {
fetchCalls.push(String(input));
return new Response(JSON.stringify({ login: "someone" }), {
status: 200,
headers: { "content-type": "application/json" },
});
}) as typeof fetch;
test("both routes answer 401 without a session and never fetch", async () => {
const giteaResponse = await gitea.POST(
post("gitea/test-connection", { url: "http://127.0.0.1:9999", token: "x" })
);
expect(giteaResponse.status).toBe(401);
const githubResponse = await github.POST(
post("github/test-connection", { provider: "gitea", url: "http://127.0.0.1:9999", token: "x" })
);
expect(githubResponse.status).toBe(401);
expect(fetchCalls).toEqual([]);
});
test("a signed in user cannot point either route at the metadata service", async () => {
const locals = { user: { id: "user-1" } };
const giteaResponse = await gitea.POST(
post("gitea/test-connection", { url: "http://169.254.169.254", token: "x" }, locals)
);
expect(giteaResponse.status).toBe(400);
expect((await giteaResponse.json()).message).toMatch(/link local/);
const githubResponse = await github.POST(
post(
"github/test-connection",
{ provider: "gitea", url: "http://169.254.169.254", token: "x" },
locals
)
);
expect(githubResponse.status).toBe(400);
expect((await githubResponse.json()).message).toMatch(/link local/);
expect(fetchCalls).toEqual([]);
});
test("a signed in user can still test a LAN Gitea", async () => {
const response = await gitea.POST(
post("gitea/test-connection", { url: "http://192.168.1.10:3000/", token: "x" }, {
user: { id: "user-1" },
})
);
expect(response.status).toBe(200);
expect(fetchCalls[0]).toBe("http://192.168.1.10:3000/api/v1/user");
globalThis.fetch = realFetch;
});
});
+6
View File
@@ -152,6 +152,12 @@ Throughout, job status updates are published as server-sent events so the dashbo
Gitea's API cannot create a pull request through the migration path, so mirrored pull requests are created as issues instead. Each one is tagged with a `pull-request` label, its title is prefixed with the original number and status, and its body includes the commit history, changed files, and merge outcome. This is a deliberate workaround for an API limitation, not a temporary gap.
### Incremental issue and pull request sync
The first issues pass and the first pull request pass list everything on GitHub. After a pass completes, the repository's metadata state records when its listing started, and later passes ask GitHub only for items updated since then, minus a 10 minute safety margin. New comments, label, state, milestone, title and body changes all move an item's `updated_at`, so only those items are reconciled and only their comments, pull request details, commits and files are fetched. Pull requests are listed through the issues endpoint for this, because the pull request list has no `since` filter.
A full pass still runs when the last one is more than 7 days old, when the destination has no mirrored issues or pull requests (for example after it was recreated), and after Reset metadata. If any item fails, the watermark is not moved, so the next pass lists it again. Items deleted on GitHub are left on the destination, as before.
### Ordering and concurrency
Gitea assigns issue numbers on creation, so processing issues in parallel can produce numbering that does not match GitHub's. The concurrency settings default to 3 for issues and 5 for pull requests, trading some ordering fidelity for speed. Set both to 1 if you need the numbering to line up.
@@ -43,6 +43,7 @@ Variables fall into two groups.
| `BETTER_AUTH_LOG_LEVEL` | `warn` | Better Auth logger verbosity: `debug`, `info`, `warn`, or `error`. Use `debug` to trace SSO and OIDC sign-in problems. Better Auth does not read the `DEBUG` variable. |
| `ENCRYPTION_SECRET` | none | Key for AES-256-GCM encryption of stored tokens. Generate with `openssl rand -base64 48`. |
| `AUTH_DEFAULT_METHOD` | `email` | Tab the login page opens on when both email and SSO are available: `email` or `sso`. The method last used in that browser wins over this default. |
| `AUTH_ALLOW_SIGNUP` | `false` | Keep email and password sign-up open after the first account exists. Off, the sign-up endpoint answers 403 once a user exists. |
In Docker, `BETTER_AUTH_SECRET` and `ENCRYPTION_SECRET` are generated on first boot if unset and saved to `/app/data/.better_auth_secret` and `/app/data/.encryption_secret` so they persist across restarts. Outside Docker there is no fallback: a missing `BETTER_AUTH_SECRET` logs a security warning in production and uses a known insecure development value.
@@ -33,6 +33,8 @@ Gitea Mirror uses this on every GET. Each response's ETag is cached; every subse
The practical effect is that sync frequency mostly stops mattering. A `30m` interval on a stable account costs barely more than a `24h` one, because 47 of the 48 daily sweeps are nearly all cache hits. The requests you spend scale with how much changed, not how often you look.
Issues and pull requests go one step further. After the first complete pass, a scheduled sync asks GitHub only for the issues and pull requests updated since the previous one, and fetches comments, commits and changed files for those alone. A tracker with thousands of issues and a handful of changes costs a handful of requests. A full pass still runs once a week to catch anything the incremental listing cannot see.
## Configuration that stays inside the budget
A few settings still matter for large or busy accounts: