commit b0cb018f7da5dc6f0b7d2eece09bae9b35ca6fab Author: omen <14905147+adamzend@users.noreply.github.com> Date: Sun Oct 4 23:18:05 2026 +0200 Release v1.0.0: synchronize GitHub mirror metadata in Gitea diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..f7e2def --- /dev/null +++ b/.gitattributes @@ -0,0 +1,2 @@ +* text=auto eol=lf +*.sh text eol=lf diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..e809c1e --- /dev/null +++ b/.gitignore @@ -0,0 +1,5 @@ +gitea-github-sync.conf +*.log +*.lock +*.bak +.env diff --git a/README.md b/README.md new file mode 100644 index 0000000..f19d968 --- /dev/null +++ b/README.md @@ -0,0 +1,103 @@ +# Gitea GitHub Sync + +Enrich all GitHub pull mirrors owned by a Gitea account with repository metadata, +topics, issues and comments that native Git mirroring does not synchronize. + +The script discovers repositories automatically and derives each GitHub source +from the mirror URL. Source and destination repository names can differ. + +## Features + +- Automatic discovery, including private mirrors visible to the Gitea token. +- Description, website, default branch and topic synchronization. +- Issue titles, bodies, open/closed states and comment synchronization. +- Existing synchronization markers prevent duplicate imports on repeated runs. +- Foreground preview, foreground apply and quiet background apply modes. +- GitHub rate-limit diagnostics, bounded waits and account-level stop conditions. +- Repository isolation, a shared process lock and per-repository summaries. +- Optional native Gitea mirror-sync requests. + +## Requirements + +Linux, Bash 4.3+, jq 1.6+, curl 7.55+, util-linux and coreutils. + +On Debian/Ubuntu: + +```bash +sudo apt-get install bash curl jq util-linux coreutils +``` + +## Configuration + +Use `gitea-github-sync.conf.example` as the template for +`/etc/gitea-github-sync.conf`. Update an existing configuration rather than +overwriting its tokens. Set your Gitea URL and token, the account owner, and a +GitHub token where needed. An empty `GITEA_OWNER` selects the token's account. + +The example contains deployment-specific defaults; replace them for your server. +Restrict the configuration to its owner: + +```bash +sudo chmod 600 /etc/gitea-github-sync.conf +``` + +Never commit the populated configuration or tokens to this repository. + +## Usage + +```bash +# Preview without API writes (the default) +sudo bash ./gitea-github-sync.sh --dry-run + +# Synchronize in the foreground +sudo bash ./gitea-github-sync.sh --apply + +# Synchronize in the background without terminal output +sudo bash ./gitea-github-sync.sh --apply-quiet + +# Enrich mirrors without queuing native Git synchronization +sudo bash ./gitea-github-sync.sh --apply --no-git +``` + +Use `--config FILE` for another configuration file and `--help` for all options. +Choose only one execution mode. + +Quiet mode appends progress and errors to `/var/log/gitea-github-sync.log`, +configurable through `QUIET_LOG_FILE`. Its launcher returns immediately; +check the log's final `exit_status` to determine the actual result. + +```bash +sudo tail -f /var/log/gitea-github-sync.log +``` + +**Depending on the number of repositories/mirrors and their issues/comments, +execution can take a very long time, potentially hours.** Quiet mode does not +speed up synchronization. GitHub limits may require a later run. + +## YunoHost deployment + +The original deployment uses Gitea installed through YunoHost. YunoHost manages +its own installation and data directories; repository storage paths can differ +from standalone Gitea examples. This script uses HTTP APIs and does not require +local repository paths. Include any web path prefix in `GITEA_URL`. + +See the [full English guide](README.txt) for locating the effective configuration, +YunoHost path details, all settings, all Bash functions and troubleshooting. + +## Scope and limitations + +Pull requests, releases/assets, labels, milestones, reactions, assignments, +original authorship and timestamps are not copied. Source deletions do not delete +Gitea records. Unmarked local records are untouched; previous imports without +markers require separate mapping. Local edits to synchronized fields are replaced +by the GitHub source. Disabled internal issue trackers receive metadata updates +only. Native mirror-sync runs asynchronously in Gitea. + +The script has been checked with Bash syntax validation, ShellCheck and local +mock API tests, including multiple mirrors, idempotency, throttling and quiet mode. + +## Downloads + +Download the script or the complete bundle from +[GitHub Releases](https://github.com/adamzend/gitea-github-sync/releases). +The first release is **v1.0.0**. diff --git a/README.txt b/README.txt new file mode 100644 index 0000000..f9eaf6d --- /dev/null +++ b/README.txt @@ -0,0 +1,269 @@ +GitHub mirror enrichment for a Gitea account +========================================== + +Purpose and workflow +-------------------- +This script complements Gitea's native GitHub pull mirrors. It discovers the +account's repositories through the Gitea REST API, identifies GitHub sources, +reads the source and destination data, compares their contents, and applies +missing or changed metadata, issues and comments. It processes one repository +at a time and can optionally queue Gitea's own background Git mirror job. + +IMPORTANT: Depending on the number of repositories/mirrors and the number of +issues and comments, execution can take a very long time. Large accounts may +require hours. Network latency, API pagination, server load, rate-limit waits +and the number of required writes all affect duration. Dry-run also reads the +APIs and can take a long time. There is no fixed completion-time guarantee. +Quiet mode changes execution/output handling; it does not make the work faster. + +Configuration +------------- +Linux requirements: Bash >= 4.3, jq >= 1.6, curl >= 7.55, flock (util-linux), +coreutils. On Debian/Ubuntu: + sudo apt-get install bash curl jq util-linux coreutils + +Save the example configuration as /etc/gitea-github-sync.conf, or update your +existing file. Set GITEA_URL, GITEA_TOKEN and GITEA_OWNER. An empty GITEA_OWNER +selects the account identified by GITEA_TOKEN. GITHUB_TOKEN is optional for +public sources and required for private sources. It is recommended for large +scans because of GitHub API rate limits. + +The Gitea token must allow account/repository reads and repository settings, +issues and comment writes. Its account must be able to edit imported comments. +The configuration is trusted Bash code. Restrict its permissions: + sudo chmod 600 /etc/gitea-github-sync.conf + +Execution +--------- +From the directory containing the downloaded script: + sudo bash ./gitea-github-sync.sh --dry-run + sudo bash ./gitea-github-sync.sh --apply + sudo bash ./gitea-github-sync.sh --apply-quiet + +Alternative configuration: + bash ./gitea-github-sync.sh --config ./my-config.conf --dry-run + +Dry-run is the default and makes no API writes. --no-git skips mirror-sync +requests while still enriching repositories. Non-root runs need LOCK_FILE in +a writable, trusted directory. All runs must use the same lock file. +No schedule is installed; the script can be invoked by cron or systemd. + +Complete command-line reference +------------------------------- +--dry-run + Foreground preview; reads APIs and reports planned changes. No API writes. + This is the default if no execution mode is provided. +--apply + Foreground synchronization. Logs are printed to the terminal. The command + returns when synchronization ends and exposes the final exit status. +--apply-quiet + Background synchronization with the same changes as --apply. The launcher + returns immediately. The script prints nothing to the terminal and appends + stdout/stderr, warnings and errors to QUIET_LOG_FILE. nohup protects the worker + from ordinary terminal hangups; stdin is disconnected. No extra '&' is needed. + Reboot, service/session policies that kill processes, and explicit termination + can still stop it. This is not a system service and has no automatic restart. +--config FILE + Use a different trusted Bash configuration file. Default: + /etc/gitea-github-sync.conf (or CONFIG_FILE from the environment). +--no-git + Do not queue native mirror-sync. Metadata, issue and comment enrichment still + runs. It works with any execution mode. +--help / -h + Display usage without synchronizing anything. + +Choose only one execution mode. For example: + sudo bash ./gitea-github-sync.sh --apply-quiet --no-git + +Quiet-mode logs and completion +----------------------------- +Default log: /var/log/gitea-github-sync.log +Override it in the configuration with an absolute path: + QUIET_LOG_FILE='/var/log/gitea-github-sync.log' + +The parent directory must already exist and be writable by the executing user. +The script creates/appends the log with mode 600. For non-root runs, use a log +in a directory owned by that user. The log grows across runs; arrange log +rotation separately if needed. It is separate from Gitea's application logs. + +Read progress when desired: + sudo tail -n 100 /var/log/gitea-github-sync.log + sudo tail -f /var/log/gitea-github-sync.log + +Each worker logs its PID and ends with: + Background worker finished: PID=12345, exit_status=0 +Zero means successful script completion; nonzero means an error or interruption. +The launcher's exit status 0 only means the background command was dispatched, +not that synchronization succeeded. Check the completion line in the log. +If it cannot read configuration or open the log, the launcher fails silently; +immediately use 'echo $?' to inspect its nonzero status. An uncatchable kill or +power loss can leave the log without a completion line. + +The worker uses the same LOCK_FILE as foreground runs. A second invocation +cannot synchronize concurrently; a rejected quiet worker records the lock error +in its log. For cron/systemd jobs that need the final exit status, prefer --apply +with output managed by the scheduler instead of detaching with --apply-quiet. + +Configuration reference +----------------------- +GITEA_URL: Base web URL, default https://git.an0n.eu. Include any path prefix, + for example https://example.org/gitea. Do not append /api/v1. +GITEA_OWNER: Owner to scan; empty means the authenticated token account. +GITEA_TOKEN: Required Gitea API token; distinct from the GitHub token. +GITHUB_TOKEN: Optional for public GitHub sources, required for private sources. +GITHUB_API: GitHub API base, default https://api.github.com. Source discovery + currently recognizes GitHub.com URLs, not arbitrary enterprise hosts. +GITHUB_API_VERSION: Default 2026-03-10. +LOCK_FILE: Default /var/run/gitea-github-sync.lock; shared by all execution modes. +MAX_PAGES: Default 10000; maximum pages per paginated API listing. +GITHUB_RATE_LIMIT_MAX_WAIT: Default 60; total wait budget in seconds per request. +GITHUB_RATE_LIMIT_RETRIES: Default 2; maximum 10; allowed throttling retries. +QUIET_LOG_FILE: Default /var/log/gitea-github-sync.log; --apply-quiet output. +TMPDIR: Optional temporary-directory base; defaults to /tmp. The script creates + a private working directory and removes it on normal exit or handled signals. +Legacy GITEA_REPO/GITHUB_OWNER/GITHUB_REPO values do not select a single mirror. + +Gitea installed through YunoHost +------------------------------- +In the deployment this script was prepared for, Gitea is installed through +YunoHost. Its managed installation and data directories differ from paths in +generic standalone Gitea guides. Do not assume repositories live in +/var/lib/gitea or /home/git/gitea-repositories. + +The YunoHost package's configuration template sets [repository] ROOT to +/repositories and APP_DATA_PATH to /data. The exact data_dir, +install_dir and service name depend on the installed app instance, package +version, and migration history. A second instance can have a different app ID. +Inspect the installed service to find its effective configuration: + sudo systemctl cat gitea +Use the actual instance's service name if it differs. If ExecStart specifies a +configuration file, inspect that file. The package installs app.ini at +/custom/conf/app.ini; install_dir is the directory containing the +Gitea executable in ExecStart. Read [repository] ROOT in the installed app.ini. +Do not publish the whole configuration; it contains credentials. YunoHost +manages this file and package upgrades may overwrite manual edits. + +No local repository path is needed by this script: it operates exclusively +through HTTP APIs and lets Gitea perform Git synchronization. Leave storage +directories and their ownership under YunoHost management. The script's own +/etc/gitea-github-sync.conf and /var/log/gitea-github-sync.log are independent +files, not the YunoHost Gitea application's app.ini or service log. + +If YunoHost serves Gitea below /gitea, use that prefix in GITEA_URL. The API must +be reachable through YunoHost's reverse proxy with token authentication. An +SSO redirect or access-denied HTML response prevents API use even with a valid +Gitea token; check app access settings without making private repositories public. + +Package references: +https://github.com/YunoHost-Apps/gitea_ynh/blob/master/conf/app.ini +https://github.com/YunoHost-Apps/gitea_ynh/blob/master/manifest.toml +https://github.com/YunoHost-Apps/gitea_ynh/blob/master/scripts/install + +Discovery +--------- +All pages of /api/v1/users/{owner}/repos are read. Only repositories owned by +that account with mirror=true and a GitHub.com original_url are selected. +Private repositories are included when visible to the token. Restricted token +access can limit discovery; inaccessible repositories cannot be detected. +Collaborator and organization repositories are not implicitly included. + +Supported sources: HTTPS/HTTP GitHub URLs, git@github.com:owner/repo.git and +ssh://git@github.com/owner/repo.git, with optional .git suffix and trailing slash. +HTTPS credentials and standard SSH port 22 are supported. Embedded credentials +are not logged or reused; configure GITHUB_TOKEN separately. +Each mirror's source is derived independently; the destination name can differ. +Legacy GITEA_REPO, GITHUB_OWNER and GITHUB_REPO settings are ignored. +Non-mirrors, non-GitHub sources and repositories owned by others are skipped. +Archived GitHub mirrors are reported as failures; remaining repositories run. + +Scope +----- +The original enrichment scope is preserved: description, website, default +branch, topics, issue titles/bodies/states and issue comment bodies. Mirrors +without an internal issue tracker receive metadata/topics updates and an +explicit warning that issue/comment synchronization was skipped. + +Pull requests, releases/assets, labels, milestones, assignments, reactions, +authorship and original timestamps are not copied. Attachments remain source +links. Records removed from GitHub are not deleted in Gitea. This is enrichment +within the original script's scope, not a complete GitHub migration. + +Existing V2.1 github-sync-id and github-comment-id markers are supported. +Unmarked local records remain untouched. Earlier imports without markers need +separate mapping to avoid creating additional records. Local edits to marked +fields are overwritten by GitHub. Marker names are reserved for the script. + +Errors and repeated runs +------------------------ +Repositories run sequentially with isolated state. A failure stops only the +current repository. The final summary reports discovered, selected, succeeded, +failed and skipped counts. Exit status is 1 if any selected repository fails. +Discovery failures abort before writes. Successful writes are not rolled back. + +After a failure, inspect the log, fix the cause, run dry-run and then apply. +POST is never automatically retried after an ambiguous response. Markers are +used to reconcile existing records on subsequent runs. Duplicate markers stop +the affected repository before its write phase. + +Mirror-sync is asynchronous. A missing default branch is deferred while other +fields are updated and mirror-sync is queued (unless --no-git is used). The +repository is reported as failed for that run; rerun after Git sync finishes. +Accepted mirror-sync requests do not confirm completion of the background job. +API reads are not a transactional snapshot; concurrent source changes may +require another run. + +GitHub HTTP 403 / 429 / 401 +------------------------- +The log reports whether a GitHub token is configured and includes the API error +message. Rate limits are recognized from response headers and API messages. +The script respects Retry-After and X-RateLimit-Reset. It retries within the +GITHUB_RATE_LIMIT_MAX_WAIT budget (default 60 seconds per request) and +GITHUB_RATE_LIMIT_RETRIES (default 2, maximum 10). Longer throttling stops the +entire account scan with exit status 1 instead of failing every repository. +HTTP 401 also stops the scan: correct GITHUB_TOKEN before rerunning. +An ordinary permission-related 403 fails only the affected repository. + +An anonymous quota exhaustion requires waiting for reset or configuring a valid +GITHUB_TOKEN. An authenticated quota can also be exhausted. Repeating the same +run immediately does not reset a quota. Permission errors require checking +token repository access and applicable organization policies. A proxy/firewall +may return non-JSON errors. Changing the script alone cannot grant API access. + +Reference: https://docs.github.com/en/rest/using-the-rest-api/troubleshooting-the-rest-api + +Validation +---------- +Bash syntax, ShellCheck and local mock API tests cover account pagination, +multiple sources, renamed/private mirrors, isolated issue/comment state, +repeat-run idempotency and continuation after repository failures. +No production API writes were performed during development. + +Internal function reference +--------------------------- +usage: Prints command-line help. +log: Writes timestamped messages, with the current repository label when set. +die: Logs an error and exits with failure. +on_error: Reports an unexpected failed command's line and exit status. +cleanup: Stops/waits for an active repository worker and removes temporary files. +finish: Runs cleanup at exit and records the quiet worker's final exit status. +response_header: Extracts a response header from the last HTTP response block. +request: Sends authenticated API requests, builds JSON request headers, handles + HTTP/transport errors and bounded GitHub throttling retries. It never retries + a write automatically. A shared stop marker halts scanning on GitHub rate-limit + exhaustion or invalid authentication. +validate: Checks an API response against a jq validation expression. +paginate: Collects every page until an empty page, detects repeated pages and + enforces MAX_PAGES even when the server caps the requested page size. +marker: Extracts an issue/comment synchronization marker and rejects ambiguity. +sync_repository: Runs the read/validation, comparison and update phases in an + isolated subshell/directory for one mirror, with per-repository counters. +write_api: Sends a Gitea mutation and counts successful write requests. +github_source: Parses supported GitHub clone URLs into source owner/repository. + +The main loop discovers and filters repositories, starts one worker at a time, +waits for its result, continues after repository-specific errors, and prints the +account summary. Quiet launching happens before this loop and reuses the same +foreground synchronization engine in a detached process. + +API reference: +https://docs.gitea.com/api/1.23/operations/user-list-repos/ diff --git a/gitea-github-sync.conf.example b/gitea-github-sync.conf.example new file mode 100644 index 0000000..e38a6ae --- /dev/null +++ b/gitea-github-sync.conf.example @@ -0,0 +1,18 @@ +# Trusted Bash configuration. Save as /etc/gitea-github-sync.conf (chmod 600). +GITEA_TOKEN='REPLACE_WITH_GITEA_TOKEN' +# Optional for public GitHub repositories; recommended to increase rate limits. +GITHUB_TOKEN='' +GITEA_URL='https://git.an0n.eu' +# Leave empty to discover the account from GITEA_TOKEN, or specify an owner. +# All repositories owned by this account are scanned, including private ones +# visible to the token. Collaborator repositories owned by others are excluded. +GITEA_OWNER='adam' +# Source owners and repository names are derived from each mirror's original_url. +GITHUB_API='https://api.github.com' +GITHUB_API_VERSION='2026-03-10' +# Maximum total waiting time per GitHub request; longer limits stop the scan. +GITHUB_RATE_LIMIT_MAX_WAIT=60 +GITHUB_RATE_LIMIT_RETRIES=2 +LOCK_FILE='/var/run/gitea-github-sync.lock' +# Used only by --apply-quiet. Parent directory must already exist and be writable. +QUIET_LOG_FILE='/var/log/gitea-github-sync.log' diff --git a/gitea-github-sync.sh b/gitea-github-sync.sh new file mode 100755 index 0000000..c2a17de --- /dev/null +++ b/gitea-github-sync.sh @@ -0,0 +1,523 @@ +#!/usr/bin/env bash +# GitHub -> Gitea: metadata, topics, issues and issue comments. +# Linux, Bash >= 4.3, curl, jq >= 1.6, flock (util-linux). +# Worker state is intentionally local to subshells; parent discovery state persists. +# shellcheck disable=SC2030,SC2031 +set -Eeuo pipefail +umask 077 + +CONFIG_FILE="${CONFIG_FILE:-/etc/gitea-github-sync.conf}" +MODE=dry-run +NO_GIT=0 +QUIET=0 +SELECTED_MODE= +# Silence even launcher/configuration errors when quiet mode was requested. +# Its caller can still inspect the launcher's exit status. +for argument in "$@"; do + if [[ "$argument" == --apply-quiet ]]; then exec >/dev/null 2>&1; fi +done +usage() { + cat <&2; } +die() { log "ERROR: $*"; exit 1; } +on_error() { local rc=$1 line=$2; log "Failure at line $line (exit $rc); synchronization incomplete."; exit "$rc"; } +trap 'on_error "$?" "$LINENO"' ERR +while (($#)); do + case "$1" in + --dry-run|--apply|--apply-quiet) + [[ -z "$SELECTED_MODE" || "$SELECTED_MODE" == "$1" ]] || die "Choose only one execution mode" + SELECTED_MODE=$1 ;; + esac + case "$1" in + --config) (($# >= 2)) || die "Missing path after --config"; CONFIG_FILE=$2; shift ;; + --dry-run) MODE=dry-run; QUIET=0 ;; + --apply) MODE=apply; QUIET=0 ;; + --apply-quiet) MODE=apply; QUIET=1 ;; + --no-git) NO_GIT=1 ;; + -h|--help) usage; exit 0 ;; + *) die "Unknown option: $1" ;; + esac + shift +done +((BASH_VERSINFO[0] > 4 || (BASH_VERSINFO[0] == 4 && BASH_VERSINFO[1] >= 3))) || + die "Bash >= 4.3 is required" +readonly MODE NO_GIT QUIET +[[ -r "$CONFIG_FILE" ]] || die "Configuration not found: $CONFIG_FILE" +# Configuration is trusted shell code; restrict write access to its owner. +# shellcheck disable=SC1090 +source "$CONFIG_FILE" +: "${QUIET_LOG_FILE:=/var/log/gitea-github-sync.log}" +if ((QUIET)); then + command -v nohup >/dev/null || die "Missing command: nohup" + [[ "$QUIET_LOG_FILE" == /* ]] || die "QUIET_LOG_FILE must be an absolute path" + # Open before spawning so an inaccessible log produces a failed launch. + exec >>"$QUIET_LOG_FILE" 2>&1 + chmod 600 "$QUIET_LOG_FILE" + quiet_args=(--config "$CONFIG_FILE" --apply) + if ((NO_GIT)); then quiet_args+=(--no-git); fi + GITEA_SYNC_QUIET_WORKER=1 nohup "$BASH" "$0" "${quiet_args[@]}" /dev/null || die "Missing command: $cmd" +done +GITEA_URL=${GITEA_URL%/} +GITHUB_API=${GITHUB_API%/} +[[ "$GITEA_URL" == http://* || "$GITEA_URL" == https://* ]] || die "Invalid GITEA_URL" +[[ "$GITHUB_API" == http://* || "$GITHUB_API" == https://* ]] || die "Invalid GITHUB_API" +exec 9>"$LOCK_FILE" || die "Cannot open lock: $LOCK_FILE" +flock -n 9 || die "Another synchronization is running" +WORK_DIR=$(mktemp -d "${TMPDIR:-/tmp}/gitea-github-sync.XXXXXX") +cleanup() { + if [[ -n "${worker_pid:-}" ]]; then + kill -TERM "$worker_pid" 2>/dev/null || true + wait "$worker_pid" 2>/dev/null || true + fi + rm -rf -- "$WORK_DIR" +} +finish() { + local rc=$? + cleanup + if [[ "${GITEA_SYNC_QUIET_WORKER:-}" == 1 ]]; then + log "Background worker finished: PID=$$, exit_status=$rc" + fi + return "$rc" +} +trap finish EXIT +trap 'exit 130' INT +trap 'exit 143' TERM +RUN_DIR=$WORK_DIR +HEADER_DIR=$RUN_DIR + +# Keep credentials out of curl's command line. curl >= 7.55 supports -H @file. +printf 'Authorization: token %s\nAccept: application/json\n' "$GITEA_TOKEN" > "$WORK_DIR/gt.headers" +printf 'Accept: application/vnd.github+json\nX-GitHub-Api-Version: %s\n' "$GITHUB_API_VERSION" > "$WORK_DIR/gh.headers" +if [[ -n "${GITHUB_TOKEN:-}" ]]; then + printf 'Authorization: Bearer %s\n' "$GITHUB_TOKEN" >> "$WORK_DIR/gh.headers" + log "GitHub authentication: token configured" +else + log "WARNING: GitHub authentication: anonymous. Set GITHUB_TOKEN in the configuration for a higher API quota" +fi + +response_header() { + awk -v name="$2" ' + /^HTTP\// { value="" } + tolower($1)==tolower(name ":") { + sub(/^[^:]*:[[:space:]]*/, ""); sub(/\r$/, ""); value=$0 + } + END { print value }' "$1" +} + +# GETs may retry. Never automatically repeat a mutation: a lost response can +# follow a successful POST, and retrying it would create duplicates. +request() { + local service=$1 method=$2 url=$3 output=$4 payload=${5:-} allow_missing=${6:-0} status + local headers="$WORK_DIR/response.headers" message remaining reset retry_after wait_seconds + local attempt=0 waited=0 now reset_wait + local -a args=(-q --silent --show-error --connect-timeout 15 --max-time 120 + --user-agent 'gitea-github-sync/3.1' -D "$headers" + -H "@$HEADER_DIR/$service.headers" -X "$method" -o "$output" -w '%{http_code}') + [[ "$method" == GET || "$MODE" == apply ]] || die "Write request blocked in dry-run" + # GitHub throttling must be inspected before any retry, including HTTP 429. + if [[ "$method" == GET && "$service" != gh ]]; then + args+=(--retry 3 --retry-delay 2 --retry-max-time 180) + fi + if [[ -n "$payload" ]]; then + args+=(-H 'Content-Type: application/json' --data-binary "@$payload") + fi + while :; do + status=$(curl "${args[@]}" "$url") || die "Transport error: $method $url; run dry-run before retrying APPLY" + if [[ "$status" == 2[0-9][0-9] ]]; then return 0; fi + if [[ "$status" == 404 && "$allow_missing" == 1 ]]; then return 4; fi + message=$(jq -r 'if type=="object" and (.message|type)=="string" then + .message | gsub("[[:cntrl:]]"; " ") | .[:1000] + else "No JSON error message in response" end' "$output" 2>/dev/null) || message='Non-JSON error response (possibly proxy or firewall)' + message=${message//"$GITEA_TOKEN"/[REDACTED]} + if [[ -n "${GITHUB_TOKEN:-}" ]]; then message=${message//"$GITHUB_TOKEN"/[REDACTED]}; fi + log "HTTP $status: $method $url; API message: $message" + remaining=$(response_header "$headers" x-ratelimit-remaining) + reset=$(response_header "$headers" x-ratelimit-reset) + retry_after=$(response_header "$headers" retry-after) + if [[ "$service" == gh && ( "$status" == 403 || "$status" == 429 ) ]] && + [[ "$remaining" == 0 || "$status" == 429 || -n "$retry_after" || "${message,,}" == *"rate limit"* ]]; then + log "GitHub rate limit: remaining=${remaining:-unknown}, reset_epoch=${reset:-unknown}, retry_after=${retry_after:-absent}" + now=$(date +%s) + wait_seconds=$((60 * (2 ** attempt))) + if [[ "$retry_after" =~ ^[0-9]{1,9}$ ]]; then + wait_seconds=$((10#$retry_after)) + elif [[ -n "$retry_after" ]]; then + reset_wait=$(date -d "$retry_after" +%s 2>/dev/null) || reset_wait=$((now + 60)) + wait_seconds=$((reset_wait - now)) + fi + if [[ "$remaining" == 0 && "$reset" =~ ^[0-9]{1,12}$ ]]; then + reset_wait=$((10#$reset - now + 1)) + if [[ -z "$retry_after" ]] || ((reset_wait > wait_seconds)); then wait_seconds=$reset_wait; fi + log "GitHub quota resets at $(date -u -d "@$reset" '+%Y-%m-%d %H:%M:%S UTC' 2>/dev/null || printf '%s' "$reset")" + fi + if ((wait_seconds < 1)); then wait_seconds=1; fi + if [[ "$method" == GET ]] && ((attempt < GITHUB_RATE_LIMIT_RETRIES && waited + wait_seconds <= GITHUB_RATE_LIMIT_MAX_WAIT)); then + log "Waiting $wait_seconds seconds before retrying GitHub (attempt $((attempt + 1)))" + sleep "$wait_seconds" + waited=$((waited + wait_seconds)) + attempt=$((attempt + 1)) + continue + fi + printf '%s\n' 'GitHub rate limit reached; wait for reset or correct GITHUB_TOKEN before rerunning' > "$RUN_DIR/github-stop" + die "GitHub rate limit wait/retry budget exceeded; stopping account scan" + fi + if [[ "$service" == gh && "$status" == 401 ]]; then + printf '%s\n' 'GitHub authentication failed; replace or correct GITHUB_TOKEN' > "$RUN_DIR/github-stop" + elif [[ "$service" == gh && "$status" == 403 ]]; then + log "GitHub access denied without a rate-limit signal. Check token repository access, organization/SSO policy, and any proxy/firewall" + fi + die "Request failed; see API message above" + done +} +validate() { jq -e "$2" "$1" >/dev/null || die "Invalid JSON data: ${1##*/}"; } +paginate() { + local service=$1 url=$2 output=$3 page count + : > "$output" + for ((page=1; page<=MAX_PAGES; page++)); do + request "$service" GET "$url&page=$page" "$WORK_DIR/page.json" + validate "$WORK_DIR/page.json" 'type == "array" and all(.[]; type == "object")' + count=$(jq length "$WORK_DIR/page.json") + ((count > 0)) || return 0 + if ((page > 1)) && cmp -s "$WORK_DIR/page.json" "$WORK_DIR/previous-page.json"; then + die "API returned a repeated page: $url" + fi + cat "$WORK_DIR/page.json" > "$WORK_DIR/previous-page.json" + jq -c '.[]' "$WORK_DIR/page.json" >> "$output" + # Continue to an empty page: servers may cap the requested page size. + done + die "MAX_PAGES exceeded for $url" +} +marker() { + # Existing V2.1 markers are supported. Ambiguous markers stop the run. + jq -r --arg kind "$2" ' + [(.body // "") | scan("") | .[0]] | + if length > 1 then error("Multiple markers in one record") + else .[0] // "" end' "$1" +} +issue_schema='def pos: type == "number" and . > 0 and floor == .; + all(.[]; (.id|pos) and (.number|pos) and (.title|type=="string") and + (.body == null or (.body|type=="string")) and (.state=="open" or .state=="closed"))' +comment_schema='def pos: type == "number" and . > 0 and floor == .; + all(.[]; (.id|pos) and (.body == null or (.body|type=="string")))' + +# Run each repository in its own subshell and directory. The parent waits for +# every worker and continues after a failure without disabling worker errexit. +sync_repository() ( +set -Eeuo pipefail +local entry=$1 repo_id source current_source +local GITEA_OWNER GITEA_REPO GITHUB_OWNER GITHUB_REPO WORK_DIR +repo_id=$(jq -r .id <<< "$entry") +GITEA_OWNER=$(jq -r .owner.login <<< "$entry") +GITEA_REPO=$(jq -r .name <<< "$entry") +REPO_LABEL="$GITEA_OWNER/$GITEA_REPO" +source=$(github_source "$(jq -r .original_url <<< "$entry")") +IFS=$'\t' read -r GITHUB_OWNER GITHUB_REPO <<< "$source" +WORK_DIR="$RUN_DIR/repo-$repo_id" +mkdir -- "$WORK_DIR" +# The parent owns cleanup; do not remove shared credentials from a worker. +trap - EXIT +trap 'exit 130' INT +trap 'exit 143' TERM +GT="$GITEA_URL/api/v1/repos/$GITEA_OWNER/$GITEA_REPO" +GH="$GITHUB_API/repos/$GITHUB_OWNER/$GITHUB_REPO" +log "GitHub $GITHUB_OWNER/$GITHUB_REPO -> Gitea $GITEA_OWNER/$GITEA_REPO; mode $MODE" +request gt GET "$GT" "$WORK_DIR/gt-repo.json" +request gh GET "$GH" "$WORK_DIR/gh-repo.json" +for service in gt gh; do + validate "$WORK_DIR/$service-repo.json" 'type=="object" and (.id|type=="number") and + (.default_branch|type=="string") and (.description==null or (.description|type=="string")) and + (.website==null or (.website|type=="string")) and (.homepage==null or (.homepage|type=="string"))' +done +validate "$WORK_DIR/gt-repo.json" '.archived != true and .mirror == true' +is_mirror=true +current_source=$(github_source "$(jq -r '.original_url // ""' "$WORK_DIR/gt-repo.json")") +[[ "${current_source,,}" == "${source,,}" ]] || die "Mirror source changed after discovery; rerun discovery" +sync_issues=$(jq -r '.has_issues == true and .external_tracker == null' "$WORK_DIR/gt-repo.json") +if [[ "$sync_issues" != true ]]; then + log "WARNING: Internal issues are disabled; synchronizing metadata and topics only" +fi +request gh GET "$GH/topics" "$WORK_DIR/gh-topics.json" +request gt GET "$GT/topics" "$WORK_DIR/gt-topics.json" +for service in gh gt; do + validate "$WORK_DIR/$service-topics.json" '(.topics // .names) | type=="array" and all(.[]; type=="string")' +done +jq '{topics:(.names | sort | unique)}' "$WORK_DIR/gh-topics.json" > "$WORK_DIR/topics-payload.json" +jq '{topics:((.topics // .names) | sort | unique)}' "$WORK_DIR/gt-topics.json" > "$WORK_DIR/topics-old.json" +jq '{description:(.description // ""), website:(.homepage // ""), default_branch}' "$WORK_DIR/gh-repo.json" > "$WORK_DIR/repo-payload.json" +jq '{description:(.description // ""), website:(.website // ""), default_branch}' "$WORK_DIR/gt-repo.json" > "$WORK_DIR/repo-old.json" + +# Complete the read/validation phase before any writes. +log "Reading issues and comments (excluding pull requests)..." +if [[ "$sync_issues" == true ]]; then +paginate gh "$GH/issues?state=all&sort=created&direction=asc&per_page=100" "$WORK_DIR/gh-all.jsonl" +paginate gt "$GT/issues?state=all&type=issues&limit=50" "$WORK_DIR/gt-all.jsonl" +else + : > "$WORK_DIR/gh-all.jsonl" + : > "$WORK_DIR/gt-all.jsonl" +fi +for service in gh gt; do + jq -s '[.[] | select(.pull_request == null)] | sort_by(.number)' "$WORK_DIR/$service-all.jsonl" > "$WORK_DIR/$service-issues.json" + validate "$WORK_DIR/$service-issues.json" "$issue_schema" + jq -c '.[]' "$WORK_DIR/$service-issues.json" > "$WORK_DIR/$service-issues.jsonl" +done +declare -A gh_numbers=() gt_numbers=() gt_comment_ids=() seen_gh=() +while IFS= read -r record; do + printf '%s\n' "$record" > "$WORK_DIR/record.json" + id=$(jq -r .id "$WORK_DIR/record.json") + number=$(jq -r .number "$WORK_DIR/record.json") + [[ -z "${seen_gh[$id]+x}" && -z "${gh_numbers[$number]+x}" ]] || die "Duplicate GitHub issue $id" + seen_gh[$id]=1 + gh_numbers[$number]=$id +done < "$WORK_DIR/gh-issues.jsonl" +while IFS= read -r record; do + printf '%s\n' "$record" > "$WORK_DIR/record.json" + id=$(marker "$WORK_DIR/record.json" github-sync-id) + [[ -n "$id" ]] || continue + [[ -z "${gt_numbers[$id]+x}" ]] || die "Duplicate github-sync-id:$id in Gitea" + number=$(jq -r .number "$WORK_DIR/record.json") + gt_numbers[$id]=$number + printf '%s\n' "$record" > "$WORK_DIR/gt-issue-$id.json" + paginate gt "$GT/issues/$number/comments?limit=50" "$WORK_DIR/comments.jsonl" + jq -s '.' "$WORK_DIR/comments.jsonl" > "$WORK_DIR/comments.json" + validate "$WORK_DIR/comments.json" "$comment_schema" + while IFS= read -r comment; do + printf '%s\n' "$comment" > "$WORK_DIR/record.json" + cid=$(marker "$WORK_DIR/record.json" github-comment-id) + [[ -n "$cid" ]] || continue + key="$number:$cid" + [[ -z "${gt_comment_ids[$key]+x}" ]] || die "Duplicate github-comment-id:$cid in Gitea #$number" + gt_comment_ids[$key]=$(jq -r .id "$WORK_DIR/record.json") + printf '%s\n' "$comment" > "$WORK_DIR/gt-comment-$number-$cid.json" + done < "$WORK_DIR/comments.jsonl" +done < "$WORK_DIR/gt-issues.jsonl" +if [[ "$sync_issues" == true ]]; then +paginate gh "$GH/issues/comments?sort=created&direction=asc&per_page=100" "$WORK_DIR/gh-comments.jsonl" +else + : > "$WORK_DIR/gh-comments.jsonl" +fi +jq -s '.' "$WORK_DIR/gh-comments.jsonl" > "$WORK_DIR/gh-comments.json" +validate "$WORK_DIR/gh-comments.json" "$comment_schema and all(.[]; (.issue_url|type==\"string\"))" +declare -A seen_comments=() +while IFS= read -r record; do + printf '%s\n' "$record" > "$WORK_DIR/record.json" + number=$(jq -r '.issue_url | split("/") | last' "$WORK_DIR/record.json") + [[ "$number" =~ ^[1-9][0-9]*$ ]] || die "Invalid issue number in GitHub comment" + [[ -n "${gh_numbers[$number]+x}" ]] || continue + cid=$(jq -r .id "$WORK_DIR/record.json") + [[ -z "${seen_comments[$cid]+x}" ]] || die "Duplicate GitHub comment $cid" + seen_comments[$cid]=1 + printf '%s\n' "$record" >> "$WORK_DIR/gh-comments-$number.jsonl" +done < "$WORK_DIR/gh-comments.jsonl" + +# Payloads are built from files, avoiding shell argument limits and retaining +# Unicode and trailing newlines. Ignore source copies of our reserved markers. +# shellcheck disable=SC2016 +body_filter='def marked($kind; $id): + (. // "" | gsub(""; "")) as $b | + $b + (if $b=="" then "" else "\n\n" end) + "";' +issues_create=0 issues_update=0 issues_same=0 +comments_create=0 comments_update=0 comments_same=0 +mutations=0 +deferred_branch=0 +write_api() { + request gt "$1" "$2" "$WORK_DIR/response.json" "${3:-}" + mutations=$((mutations + 1)) +} +if ! cmp -s "$WORK_DIR/repo-old.json" "$WORK_DIR/repo-payload.json"; then + log "Repository metadata: update planned" + # Mirror sync is asynchronous. Do not set a branch that does not exist yet. + branch=$(jq -r .default_branch "$WORK_DIR/repo-payload.json") + old_branch=$(jq -r .default_branch "$WORK_DIR/repo-old.json") + if [[ "$branch" != "$old_branch" ]]; then + log "Default branch must exist in Gitea: $branch" + encoded=$(jq -rn --arg v "$branch" '$v|@uri') + if request gt GET "$GT/branches/$encoded" "$WORK_DIR/branch.json" '' 1; then + validate "$WORK_DIR/branch.json" '(.name|type=="string")' + else + log "Branch is not available in Gitea; deferring default_branch. Retry after mirror-sync completes." + jq 'del(.default_branch)' "$WORK_DIR/repo-payload.json" > "$WORK_DIR/repo-without-branch.json" + cat "$WORK_DIR/repo-without-branch.json" > "$WORK_DIR/repo-payload.json" + deferred_branch=1 + fi + fi + if [[ "$MODE" == apply ]]; then write_api PATCH "$GT" "$WORK_DIR/repo-payload.json"; fi +fi +if ! cmp -s "$WORK_DIR/topics-old.json" "$WORK_DIR/topics-payload.json"; then + log "Repository topics: update planned" + if [[ "$MODE" == apply ]]; then write_api PUT "$GT/topics" "$WORK_DIR/topics-payload.json"; fi +fi +while IFS= read -r record; do + printf '%s\n' "$record" > "$WORK_DIR/source.json" + id=$(jq -r .id "$WORK_DIR/source.json") + number=$(jq -r .number "$WORK_DIR/source.json") + target=${gt_numbers[$id]:-} + jq "$body_filter"'{title, body:(.body | marked("github-sync-id"; $id)), state}' \ + --argjson id "$id" "$WORK_DIR/source.json" > "$WORK_DIR/issue-payload.json" + if [[ -z "$target" ]]; then + issues_create=$((issues_create + 1)) + log "CREATE issue GitHub #$number" + if [[ "$MODE" == apply ]]; then + jq '{title, body, closed:(.state=="closed")}' "$WORK_DIR/issue-payload.json" > "$WORK_DIR/create.json" + write_api POST "$GT/issues" "$WORK_DIR/create.json" + validate "$WORK_DIR/response.json" '(.number|type=="number" and .>0 and floor==.)' + target=$(jq -r .number "$WORK_DIR/response.json") + gt_numbers[$id]=$target + fi + else + jq '{title, body:(.body // ""), state}' "$WORK_DIR/gt-issue-$id.json" > "$WORK_DIR/old.json" + if cmp -s "$WORK_DIR/old.json" "$WORK_DIR/issue-payload.json"; then + issues_same=$((issues_same + 1)) + else + issues_update=$((issues_update + 1)) + log "UPDATE issue Gitea #$target <- GitHub #$number" + if [[ "$MODE" == apply ]]; then write_api PATCH "$GT/issues/$target" "$WORK_DIR/issue-payload.json"; fi + fi + fi + [[ -f "$WORK_DIR/gh-comments-$number.jsonl" ]] || continue + while IFS= read -r comment; do + printf '%s\n' "$comment" > "$WORK_DIR/source.json" + cid=$(jq -r .id "$WORK_DIR/source.json") + jq "$body_filter"'{body:(.body | marked("github-comment-id"; $id))}' \ + --argjson id "$cid" "$WORK_DIR/source.json" > "$WORK_DIR/comment-payload.json" + key="$target:$cid" + existing=${gt_comment_ids[$key]:-} + if [[ -z "$existing" ]]; then + comments_create=$((comments_create + 1)) + log "CREATE GitHub comment $cid on issue #$number" + if [[ "$MODE" == apply ]]; then write_api POST "$GT/issues/$target/comments" "$WORK_DIR/comment-payload.json"; fi + else + jq '{body:(.body // "")}' "$WORK_DIR/gt-comment-$target-$cid.json" > "$WORK_DIR/old.json" + if cmp -s "$WORK_DIR/old.json" "$WORK_DIR/comment-payload.json"; then + comments_same=$((comments_same + 1)) + else + comments_update=$((comments_update + 1)) + log "UPDATE Gitea comment $existing <- GitHub $cid" + if [[ "$MODE" == apply ]]; then write_api PATCH "$GT/issues/comments/$existing" "$WORK_DIR/comment-payload.json"; fi + fi + fi + done < "$WORK_DIR/gh-comments-$number.jsonl" +done < "$WORK_DIR/gh-issues.jsonl" +if ((NO_GIT)); then + log "Mirror-sync skipped (--no-git)" +elif [[ "$is_mirror" != true ]]; then + log "Mirror-sync skipped: repository is not a mirror" +elif [[ "$MODE" == apply ]]; then + write_api POST "$GT/mirror-sync" + log "Mirror-sync queued; check completion in Gitea" +else + log "DRY-RUN: mirror-sync would be queued" +fi +log "Issues: create=$issues_create, update=$issues_update, unchanged=$issues_same" +log "Comments: create=$comments_create, update=$comments_update, unchanged=$comments_same" +if [[ "$MODE" == apply ]] && ((deferred_branch)); then + die "Default branch update pending. Successful writes: $mutations. Wait for Git synchronization and rerun." +fi +log "Completed $MODE; successful write requests: $mutations" + +) + +# Parse only GitHub.com clone URLs. Do not log original_url: it can contain +# embedded credentials. Destination names may differ from source names. +github_source() { + jq -rn --arg url "$1" ' + ($url | capture("(?i)^(?:https?://(?:[^/@]+@)?github\\.com(?::443)?/|ssh://(?:git@)?github\\.com(?::22)?/|git@github\\.com:)(?[a-z0-9_.-]+)/(?[a-z0-9_.-]+)/?$")?) as $m | + if $m == null then "" else + ($m.repo | sub("(?i)\\.git$"; "")) as $repo | + if $m.owner == "." or $m.owner == ".." or $repo == "" or $repo == "." or $repo == ".." + then "" else [$m.owner, $repo] | @tsv end + end' +} + +if [[ -n "${GITEA_REPO:-}${GITHUB_OWNER:-}${GITHUB_REPO:-}" ]]; then + log "Legacy GITEA_REPO/GITHUB_OWNER/GITHUB_REPO settings are ignored; repositories are discovered automatically" +fi +if [[ -z "$GITEA_OWNER" ]]; then + request gt GET "$GITEA_URL/api/v1/user" "$WORK_DIR/user.json" + validate "$WORK_DIR/user.json" '(.login|type=="string" and length>0)' + GITEA_OWNER=$(jq -r .login "$WORK_DIR/user.json") +fi +[[ "$GITEA_OWNER" =~ ^[A-Za-z0-9_.-]+$ && "$GITEA_OWNER" != . && "$GITEA_OWNER" != .. ]] || die "Invalid GITEA_OWNER" +log "Discovering repositories owned by $GITEA_OWNER" +paginate gt "$GITEA_URL/api/v1/users/$GITEA_OWNER/repos?limit=50" "$WORK_DIR/repositories.jsonl" +jq -s '.' "$WORK_DIR/repositories.jsonl" > "$WORK_DIR/repositories.json" +validate "$WORK_DIR/repositories.json" 'all(.[]; + (.id|type=="number" and .>0 and floor==.) and + (.name|type=="string" and test("^[A-Za-z0-9_.-]+$") and .!="." and .!="..") and + (.owner.login|type=="string" and test("^[A-Za-z0-9_.-]+$") and .!="." and .!="..") and + (.mirror|type=="boolean") and (.original_url==null or (.original_url|type=="string")))' +declare -A discovered_ids=() +discovered=0 selected=0 skipped=0 succeeded=0 failed=0 +while IFS= read -r entry; do + repo_id=$(jq -r .id <<< "$entry") + [[ -z "${discovered_ids[$repo_id]+x}" ]] || continue + discovered_ids[$repo_id]=1 + discovered=$((discovered + 1)) + full_name=$(jq -r '.owner.login + "/" + .name' <<< "$entry") + owner=$(jq -r .owner.login <<< "$entry") + if [[ "${owner,,}" != "${GITEA_OWNER,,}" ]] || [[ $(jq -r .mirror <<< "$entry") != true ]]; then + skipped=$((skipped + 1)) + log "SKIP $full_name: not a pull mirror owned by $GITEA_OWNER" + continue + fi + source=$(github_source "$(jq -r '.original_url // ""' <<< "$entry")") + if [[ -z "$source" ]]; then + skipped=$((skipped + 1)) + log "SKIP $full_name: no supported GitHub.com source URL" + continue + fi + selected=$((selected + 1)) + if [[ $(jq -r '.archived // false' <<< "$entry") == true ]]; then + failed=$((failed + 1)) + log "FAILED $full_name: archived GitHub mirror cannot be updated" + continue + fi + sync_repository "$entry" & + worker_pid=$! + if wait "$worker_pid"; then + succeeded=$((succeeded + 1)) + log "OK $full_name" + else + failed=$((failed + 1)) + log "FAILED $full_name" + fi + worker_pid= + if [[ -f "$RUN_DIR/github-stop" ]]; then + log "Account scan aborted: $(cat "$RUN_DIR/github-stop"). Remaining repositories were not attempted" + break + fi +done < "$WORK_DIR/repositories.jsonl" +log "Repositories: discovered=$discovered, selected=$selected, succeeded=$succeeded, failed=$failed, skipped=$skipped" +((failed == 0)) || exit 1 +log "Account synchronization completed ($MODE)"