adam/gitea-github-sync
1
0 Fork 0
mirror of https://github.com/adamzend/gitea-github-sync.git synced 2026-10-30 19:01:59 +01:00
Code Issues Packages Projects Releases Wiki Activity

Release v1.0.0: synchronize GitHub mirror metadata in Gitea

Browse Source
This commit is contained in:
omen committed 2026-10-04 23:18:05 +02:00
commit b0cb018f7d
6 files changed
+920
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.gitattributes
+2
View File
@@ -0,0 +1,2 @@
* text=auto eol=lf
*.sh text eol=lf
.gitignore
+5
View File
@@ -0,0 +1,5 @@
gitea-github-sync.conf
*.log
*.lock
*.bak
.env
README.md
+103
View File
@@ -0,0 +1,103 @@
# Gitea GitHub Sync
Enrich all GitHub pull mirrors owned by a Gitea account with repository metadata,
topics, issues and comments that native Git mirroring does not synchronize.
The script discovers repositories automatically and derives each GitHub source
from the mirror URL. Source and destination repository names can differ.
## Features
- Automatic discovery, including private mirrors visible to the Gitea token.
- Description, website, default branch and topic synchronization.
- Issue titles, bodies, open/closed states and comment synchronization.
- Existing synchronization markers prevent duplicate imports on repeated runs.
- Foreground preview, foreground apply and quiet background apply modes.
- GitHub rate-limit diagnostics, bounded waits and account-level stop conditions.
- Repository isolation, a shared process lock and per-repository summaries.
- Optional native Gitea mirror-sync requests.
## Requirements
Linux, Bash 4.3+, jq 1.6+, curl 7.55+, util-linux and coreutils.
On Debian/Ubuntu:
```bash
sudo apt-get install bash curl jq util-linux coreutils
```
## Configuration
Use `gitea-github-sync.conf.example` as the template for
`/etc/gitea-github-sync.conf`. Update an existing configuration rather than
overwriting its tokens. Set your Gitea URL and token, the account owner, and a
GitHub token where needed. An empty `GITEA_OWNER` selects the token's account.
The example contains deployment-specific defaults; replace them for your server.
Restrict the configuration to its owner:
```bash
sudo chmod 600 /etc/gitea-github-sync.conf
```
Never commit the populated configuration or tokens to this repository.
## Usage
```bash
# Preview without API writes (the default)
sudo bash ./gitea-github-sync.sh --dry-run
# Synchronize in the foreground
sudo bash ./gitea-github-sync.sh --apply
# Synchronize in the background without terminal output
sudo bash ./gitea-github-sync.sh --apply-quiet
# Enrich mirrors without queuing native Git synchronization
sudo bash ./gitea-github-sync.sh --apply --no-git
```
Use `--config FILE` for another configuration file and `--help` for all options.
Choose only one execution mode.
Quiet mode appends progress and errors to `/var/log/gitea-github-sync.log`,
configurable through `QUIET_LOG_FILE`. Its launcher returns immediately;
check the log's final `exit_status` to determine the actual result.
```bash
sudo tail -f /var/log/gitea-github-sync.log
```
**Depending on the number of repositories/mirrors and their issues/comments,
execution can take a very long time, potentially hours.** Quiet mode does not
speed up synchronization. GitHub limits may require a later run.
## YunoHost deployment
The original deployment uses Gitea installed through YunoHost. YunoHost manages
its own installation and data directories; repository storage paths can differ
from standalone Gitea examples. This script uses HTTP APIs and does not require
local repository paths. Include any web path prefix in `GITEA_URL`.
See the [full English guide](README.txt) for locating the effective configuration,
YunoHost path details, all settings, all Bash functions and troubleshooting.
## Scope and limitations
Pull requests, releases/assets, labels, milestones, reactions, assignments,
original authorship and timestamps are not copied. Source deletions do not delete
Gitea records. Unmarked local records are untouched; previous imports without
markers require separate mapping. Local edits to synchronized fields are replaced
by the GitHub source. Disabled internal issue trackers receive metadata updates
only. Native mirror-sync runs asynchronously in Gitea.
The script has been checked with Bash syntax validation, ShellCheck and local
mock API tests, including multiple mirrors, idempotency, throttling and quiet mode.
## Downloads
Download the script or the complete bundle from
[GitHub Releases](https://github.com/adamzend/gitea-github-sync/releases).
The first release is **v1.0.0**.
README.txt
+269
View File
@@ -0,0 +1,269 @@
GitHub mirror enrichment for a Gitea account
==========================================
Purpose and workflow
--------------------
This script complements Gitea's native GitHub pull mirrors. It discovers the
account's repositories through the Gitea REST API, identifies GitHub sources,
reads the source and destination data, compares their contents, and applies
missing or changed metadata, issues and comments. It processes one repository
at a time and can optionally queue Gitea's own background Git mirror job.
IMPORTANT: Depending on the number of repositories/mirrors and the number of
issues and comments, execution can take a very long time. Large accounts may
require hours. Network latency, API pagination, server load, rate-limit waits
and the number of required writes all affect duration. Dry-run also reads the
APIs and can take a long time. There is no fixed completion-time guarantee.
Quiet mode changes execution/output handling; it does not make the work faster.
Configuration
-------------
Linux requirements: Bash >= 4.3, jq >= 1.6, curl >= 7.55, flock (util-linux),
coreutils. On Debian/Ubuntu:
sudo apt-get install bash curl jq util-linux coreutils
Save the example configuration as /etc/gitea-github-sync.conf, or update your
existing file. Set GITEA_URL, GITEA_TOKEN and GITEA_OWNER. An empty GITEA_OWNER
selects the account identified by GITEA_TOKEN. GITHUB_TOKEN is optional for
public sources and required for private sources. It is recommended for large
scans because of GitHub API rate limits.
The Gitea token must allow account/repository reads and repository settings,
issues and comment writes. Its account must be able to edit imported comments.
The configuration is trusted Bash code. Restrict its permissions:
sudo chmod 600 /etc/gitea-github-sync.conf
Execution
---------
From the directory containing the downloaded script:
sudo bash ./gitea-github-sync.sh --dry-run
sudo bash ./gitea-github-sync.sh --apply
sudo bash ./gitea-github-sync.sh --apply-quiet
Alternative configuration:
bash ./gitea-github-sync.sh --config ./my-config.conf --dry-run
Dry-run is the default and makes no API writes. --no-git skips mirror-sync
requests while still enriching repositories. Non-root runs need LOCK_FILE in
a writable, trusted directory. All runs must use the same lock file.
No schedule is installed; the script can be invoked by cron or systemd.
Complete command-line reference
-------------------------------
--dry-run
Foreground preview; reads APIs and reports planned changes. No API writes.
This is the default if no execution mode is provided.
--apply
Foreground synchronization. Logs are printed to the terminal. The command
returns when synchronization ends and exposes the final exit status.
--apply-quiet
Background synchronization with the same changes as --apply. The launcher
returns immediately. The script prints nothing to the terminal and appends
stdout/stderr, warnings and errors to QUIET_LOG_FILE. nohup protects the worker
from ordinary terminal hangups; stdin is disconnected. No extra '&' is needed.
Reboot, service/session policies that kill processes, and explicit termination
can still stop it. This is not a system service and has no automatic restart.
--config FILE
Use a different trusted Bash configuration file. Default:
/etc/gitea-github-sync.conf (or CONFIG_FILE from the environment).
--no-git
Do not queue native mirror-sync. Metadata, issue and comment enrichment still
runs. It works with any execution mode.
--help / -h
Display usage without synchronizing anything.
Choose only one execution mode. For example:
sudo bash ./gitea-github-sync.sh --apply-quiet --no-git
Quiet-mode logs and completion
-----------------------------
Default log: /var/log/gitea-github-sync.log
Override it in the configuration with an absolute path:
QUIET_LOG_FILE='/var/log/gitea-github-sync.log'
The parent directory must already exist and be writable by the executing user.
The script creates/appends the log with mode 600. For non-root runs, use a log
in a directory owned by that user. The log grows across runs; arrange log
rotation separately if needed. It is separate from Gitea's application logs.
Read progress when desired:
sudo tail -n 100 /var/log/gitea-github-sync.log
sudo tail -f /var/log/gitea-github-sync.log
Each worker logs its PID and ends with:
Background worker finished: PID=12345, exit_status=0
Zero means successful script completion; nonzero means an error or interruption.
The launcher's exit status 0 only means the background command was dispatched,
not that synchronization succeeded. Check the completion line in the log.
If it cannot read configuration or open the log, the launcher fails silently;
immediately use 'echo $?' to inspect its nonzero status. An uncatchable kill or
power loss can leave the log without a completion line.
The worker uses the same LOCK_FILE as foreground runs. A second invocation
cannot synchronize concurrently; a rejected quiet worker records the lock error
in its log. For cron/systemd jobs that need the final exit status, prefer --apply
with output managed by the scheduler instead of detaching with --apply-quiet.
Configuration reference
-----------------------
GITEA_URL: Base web URL, default https://git.an0n.eu. Include any path prefix,
for example https://example.org/gitea. Do not append /api/v1.
GITEA_OWNER: Owner to scan; empty means the authenticated token account.
GITEA_TOKEN: Required Gitea API token; distinct from the GitHub token.
GITHUB_TOKEN: Optional for public GitHub sources, required for private sources.
GITHUB_API: GitHub API base, default https://api.github.com. Source discovery
currently recognizes GitHub.com URLs, not arbitrary enterprise hosts.
GITHUB_API_VERSION: Default 2026-03-10.
LOCK_FILE: Default /var/run/gitea-github-sync.lock; shared by all execution modes.
MAX_PAGES: Default 10000; maximum pages per paginated API listing.
GITHUB_RATE_LIMIT_MAX_WAIT: Default 60; total wait budget in seconds per request.
GITHUB_RATE_LIMIT_RETRIES: Default 2; maximum 10; allowed throttling retries.
QUIET_LOG_FILE: Default /var/log/gitea-github-sync.log; --apply-quiet output.
TMPDIR: Optional temporary-directory base; defaults to /tmp. The script creates
a private working directory and removes it on normal exit or handled signals.
Legacy GITEA_REPO/GITHUB_OWNER/GITHUB_REPO values do not select a single mirror.
Gitea installed through YunoHost
-------------------------------
In the deployment this script was prepared for, Gitea is installed through
YunoHost. Its managed installation and data directories differ from paths in
generic standalone Gitea guides. Do not assume repositories live in
/var/lib/gitea or /home/git/gitea-repositories.
The YunoHost package's configuration template sets [repository] ROOT to
<data_dir>/repositories and APP_DATA_PATH to <data_dir>/data. The exact data_dir,
install_dir and service name depend on the installed app instance, package
version, and migration history. A second instance can have a different app ID.
Inspect the installed service to find its effective configuration:
sudo systemctl cat gitea
Use the actual instance's service name if it differs. If ExecStart specifies a
configuration file, inspect that file. The package installs app.ini at
<install_dir>/custom/conf/app.ini; install_dir is the directory containing the
Gitea executable in ExecStart. Read [repository] ROOT in the installed app.ini.
Do not publish the whole configuration; it contains credentials. YunoHost
manages this file and package upgrades may overwrite manual edits.
No local repository path is needed by this script: it operates exclusively
through HTTP APIs and lets Gitea perform Git synchronization. Leave storage
directories and their ownership under YunoHost management. The script's own
/etc/gitea-github-sync.conf and /var/log/gitea-github-sync.log are independent
files, not the YunoHost Gitea application's app.ini or service log.
If YunoHost serves Gitea below /gitea, use that prefix in GITEA_URL. The API must
be reachable through YunoHost's reverse proxy with token authentication. An
SSO redirect or access-denied HTML response prevents API use even with a valid
Gitea token; check app access settings without making private repositories public.
Package references:
https://github.com/YunoHost-Apps/gitea_ynh/blob/master/conf/app.ini
https://github.com/YunoHost-Apps/gitea_ynh/blob/master/manifest.toml
https://github.com/YunoHost-Apps/gitea_ynh/blob/master/scripts/install
Discovery
---------
All pages of /api/v1/users/{owner}/repos are read. Only repositories owned by
that account with mirror=true and a GitHub.com original_url are selected.
Private repositories are included when visible to the token. Restricted token
access can limit discovery; inaccessible repositories cannot be detected.
Collaborator and organization repositories are not implicitly included.
Supported sources: HTTPS/HTTP GitHub URLs, git@github.com:owner/repo.git and
ssh://git@github.com/owner/repo.git, with optional .git suffix and trailing slash.
HTTPS credentials and standard SSH port 22 are supported. Embedded credentials
are not logged or reused; configure GITHUB_TOKEN separately.
Each mirror's source is derived independently; the destination name can differ.
Legacy GITEA_REPO, GITHUB_OWNER and GITHUB_REPO settings are ignored.
Non-mirrors, non-GitHub sources and repositories owned by others are skipped.
Archived GitHub mirrors are reported as failures; remaining repositories run.
Scope
-----
The original enrichment scope is preserved: description, website, default
branch, topics, issue titles/bodies/states and issue comment bodies. Mirrors
without an internal issue tracker receive metadata/topics updates and an
explicit warning that issue/comment synchronization was skipped.
Pull requests, releases/assets, labels, milestones, assignments, reactions,
authorship and original timestamps are not copied. Attachments remain source
links. Records removed from GitHub are not deleted in Gitea. This is enrichment
within the original script's scope, not a complete GitHub migration.
Existing V2.1 github-sync-id and github-comment-id markers are supported.
Unmarked local records remain untouched. Earlier imports without markers need
separate mapping to avoid creating additional records. Local edits to marked
fields are overwritten by GitHub. Marker names are reserved for the script.
Errors and repeated runs
------------------------
Repositories run sequentially with isolated state. A failure stops only the
current repository. The final summary reports discovered, selected, succeeded,
failed and skipped counts. Exit status is 1 if any selected repository fails.
Discovery failures abort before writes. Successful writes are not rolled back.
After a failure, inspect the log, fix the cause, run dry-run and then apply.
POST is never automatically retried after an ambiguous response. Markers are
used to reconcile existing records on subsequent runs. Duplicate markers stop
the affected repository before its write phase.
Mirror-sync is asynchronous. A missing default branch is deferred while other
fields are updated and mirror-sync is queued (unless --no-git is used). The
repository is reported as failed for that run; rerun after Git sync finishes.
Accepted mirror-sync requests do not confirm completion of the background job.
API reads are not a transactional snapshot; concurrent source changes may
require another run.
GitHub HTTP 403 / 429 / 401
-------------------------
The log reports whether a GitHub token is configured and includes the API error
message. Rate limits are recognized from response headers and API messages.
The script respects Retry-After and X-RateLimit-Reset. It retries within the
GITHUB_RATE_LIMIT_MAX_WAIT budget (default 60 seconds per request) and
GITHUB_RATE_LIMIT_RETRIES (default 2, maximum 10). Longer throttling stops the
entire account scan with exit status 1 instead of failing every repository.
HTTP 401 also stops the scan: correct GITHUB_TOKEN before rerunning.
An ordinary permission-related 403 fails only the affected repository.
An anonymous quota exhaustion requires waiting for reset or configuring a valid
GITHUB_TOKEN. An authenticated quota can also be exhausted. Repeating the same
run immediately does not reset a quota. Permission errors require checking
token repository access and applicable organization policies. A proxy/firewall
may return non-JSON errors. Changing the script alone cannot grant API access.
Reference: https://docs.github.com/en/rest/using-the-rest-api/troubleshooting-the-rest-api
Validation
----------
Bash syntax, ShellCheck and local mock API tests cover account pagination,
multiple sources, renamed/private mirrors, isolated issue/comment state,
repeat-run idempotency and continuation after repository failures.
No production API writes were performed during development.
Internal function reference
---------------------------
usage: Prints command-line help.
log: Writes timestamped messages, with the current repository label when set.
die: Logs an error and exits with failure.
on_error: Reports an unexpected failed command's line and exit status.
cleanup: Stops/waits for an active repository worker and removes temporary files.
finish: Runs cleanup at exit and records the quiet worker's final exit status.
response_header: Extracts a response header from the last HTTP response block.
request: Sends authenticated API requests, builds JSON request headers, handles
HTTP/transport errors and bounded GitHub throttling retries. It never retries
a write automatically. A shared stop marker halts scanning on GitHub rate-limit
exhaustion or invalid authentication.
validate: Checks an API response against a jq validation expression.
paginate: Collects every page until an empty page, detects repeated pages and
enforces MAX_PAGES even when the server caps the requested page size.
marker: Extracts an issue/comment synchronization marker and rejects ambiguity.
sync_repository: Runs the read/validation, comparison and update phases in an
isolated subshell/directory for one mirror, with per-repository counters.
write_api: Sends a Gitea mutation and counts successful write requests.
github_source: Parses supported GitHub clone URLs into source owner/repository.
The main loop discovers and filters repositories, starts one worker at a time,
waits for its result, continues after repository-specific errors, and prints the
account summary. Quiet launching happens before this loop and reuses the same
foreground synchronization engine in a detached process.
API reference:
https://docs.gitea.com/api/1.23/operations/user-list-repos/
gitea-github-sync.conf.example
+18
View File
@@ -0,0 +1,18 @@
# Trusted Bash configuration. Save as /etc/gitea-github-sync.conf (chmod 600).
GITEA_TOKEN='REPLACE_WITH_GITEA_TOKEN'
# Optional for public GitHub repositories; recommended to increase rate limits.
GITHUB_TOKEN=''
GITEA_URL='https://git.an0n.eu'
# Leave empty to discover the account from GITEA_TOKEN, or specify an owner.
# All repositories owned by this account are scanned, including private ones
# visible to the token. Collaborator repositories owned by others are excluded.
GITEA_OWNER='adam'
# Source owners and repository names are derived from each mirror's original_url.
GITHUB_API='https://api.github.com'
GITHUB_API_VERSION='2026-03-10'
# Maximum total waiting time per GitHub request; longer limits stop the scan.
GITHUB_RATE_LIMIT_MAX_WAIT=60
GITHUB_RATE_LIMIT_RETRIES=2
LOCK_FILE='/var/run/gitea-github-sync.lock'
# Used only by --apply-quiet. Parent directory must already exist and be writable.
QUIET_LOG_FILE='/var/log/gitea-github-sync.log'
gitea-github-sync.sh
Executable
+523
View File
@@ -0,0 +1,523 @@
#!/usr/bin/env bash
# GitHub -> Gitea: metadata, topics, issues and issue comments.
# Linux, Bash >= 4.3, curl, jq >= 1.6, flock (util-linux).
# Worker state is intentionally local to subshells; parent discovery state persists.
# shellcheck disable=SC2030,SC2031
set -Eeuo pipefail
umask 077
CONFIG_FILE="${CONFIG_FILE:-/etc/gitea-github-sync.conf}"
MODE=dry-run
NO_GIT=0
QUIET=0
SELECTED_MODE=
# Silence even launcher/configuration errors when quiet mode was requested.
# Its caller can still inspect the launcher's exit status.
for argument in "$@"; do
if [[ "$argument" == --apply-quiet ]]; then exec >/dev/null 2>&1; fi
done
usage() {
cat <<USAGE
Usage: $0 [--config FILE] [--dry-run|--apply|--apply-quiet] [--no-git]
--dry-run Report changes without writing to APIs (default).
--apply Apply changes in the foreground.
--apply-quiet Apply in the background; write output to QUIET_LOG_FILE.
--config FILE Read a trusted Bash configuration file.
--no-git Skip native Gitea mirror-sync requests.
--help, -h Show this help.
USAGE
}
log() { printf '[%s] %s%s\n' "$(date '+%F %T')" "${REPO_LABEL:+[$REPO_LABEL] }" "$*" >&2; }
die() { log "ERROR: $*"; exit 1; }
on_error() { local rc=$1 line=$2; log "Failure at line $line (exit $rc); synchronization incomplete."; exit "$rc"; }
trap 'on_error "$?" "$LINENO"' ERR
while (($#)); do
case "$1" in
--dry-run|--apply|--apply-quiet)
[[ -z "$SELECTED_MODE" || "$SELECTED_MODE" == "$1" ]] || die "Choose only one execution mode"
SELECTED_MODE=$1 ;;
esac
case "$1" in
--config) (($# >= 2)) || die "Missing path after --config"; CONFIG_FILE=$2; shift ;;
--dry-run) MODE=dry-run; QUIET=0 ;;
--apply) MODE=apply; QUIET=0 ;;
--apply-quiet) MODE=apply; QUIET=1 ;;
--no-git) NO_GIT=1 ;;
-h|--help) usage; exit 0 ;;
*) die "Unknown option: $1" ;;
esac
shift
done
((BASH_VERSINFO[0] > 4 || (BASH_VERSINFO[0] == 4 && BASH_VERSINFO[1] >= 3))) ||
die "Bash >= 4.3 is required"
readonly MODE NO_GIT QUIET
[[ -r "$CONFIG_FILE" ]] || die "Configuration not found: $CONFIG_FILE"
# Configuration is trusted shell code; restrict write access to its owner.
# shellcheck disable=SC1090
source "$CONFIG_FILE"
: "${QUIET_LOG_FILE:=/var/log/gitea-github-sync.log}"
if ((QUIET)); then
command -v nohup >/dev/null || die "Missing command: nohup"
[[ "$QUIET_LOG_FILE" == /* ]] || die "QUIET_LOG_FILE must be an absolute path"
# Open before spawning so an inaccessible log produces a failed launch.
exec >>"$QUIET_LOG_FILE" 2>&1
chmod 600 "$QUIET_LOG_FILE"
quiet_args=(--config "$CONFIG_FILE" --apply)
if ((NO_GIT)); then quiet_args+=(--no-git); fi
GITEA_SYNC_QUIET_WORKER=1 nohup "$BASH" "$0" "${quiet_args[@]}" </dev/null &
exit 0
fi
if [[ "${GITEA_SYNC_QUIET_WORKER:-}" == 1 ]]; then
# Preserve the final worker result, including failures before acquiring a lock.
trap 'rc=$?; log "Background worker finished: PID=$$, exit_status=$rc"' EXIT
log "Background worker started: PID=$$"
fi
: "${GITEA_URL:=https://git.an0n.eu}"
: "${GITEA_OWNER:=}"
: "${GITHUB_API:=https://api.github.com}"
: "${GITHUB_API_VERSION:=2026-03-10}"
: "${LOCK_FILE:=/var/run/gitea-github-sync.lock}"
: "${MAX_PAGES:=10000}"
: "${GITHUB_RATE_LIMIT_MAX_WAIT:=60}"
: "${GITHUB_RATE_LIMIT_RETRIES:=2}"
[[ -n "${GITEA_TOKEN:-}" ]] || die "GITEA_TOKEN is missing from the configuration"
[[ "$MAX_PAGES" =~ ^[1-9][0-9]*$ ]] || die "Invalid MAX_PAGES"
for value in "$GITHUB_RATE_LIMIT_MAX_WAIT" "$GITHUB_RATE_LIMIT_RETRIES"; do
[[ "$value" =~ ^(0|[1-9][0-9]{0,5})$ ]] || die "Invalid GitHub rate limit setting"
done
((GITHUB_RATE_LIMIT_RETRIES <= 10)) || die "GITHUB_RATE_LIMIT_RETRIES must not exceed 10"
for cmd in curl jq flock mktemp rm cmp cat date mkdir awk sleep; do
command -v "$cmd" >/dev/null || die "Missing command: $cmd"
done
GITEA_URL=${GITEA_URL%/}
GITHUB_API=${GITHUB_API%/}
[[ "$GITEA_URL" == http://* || "$GITEA_URL" == https://* ]] || die "Invalid GITEA_URL"
[[ "$GITHUB_API" == http://* || "$GITHUB_API" == https://* ]] || die "Invalid GITHUB_API"
exec 9>"$LOCK_FILE" || die "Cannot open lock: $LOCK_FILE"
flock -n 9 || die "Another synchronization is running"
WORK_DIR=$(mktemp -d "${TMPDIR:-/tmp}/gitea-github-sync.XXXXXX")
cleanup() {
if [[ -n "${worker_pid:-}" ]]; then
kill -TERM "$worker_pid" 2>/dev/null || true
wait "$worker_pid" 2>/dev/null || true
fi
rm -rf -- "$WORK_DIR"
}
finish() {
local rc=$?
cleanup
if [[ "${GITEA_SYNC_QUIET_WORKER:-}" == 1 ]]; then
log "Background worker finished: PID=$$, exit_status=$rc"
fi
return "$rc"
}
trap finish EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
RUN_DIR=$WORK_DIR
HEADER_DIR=$RUN_DIR
# Keep credentials out of curl's command line. curl >= 7.55 supports -H @file.
printf 'Authorization: token %s\nAccept: application/json\n' "$GITEA_TOKEN" > "$WORK_DIR/gt.headers"
printf 'Accept: application/vnd.github+json\nX-GitHub-Api-Version: %s\n' "$GITHUB_API_VERSION" > "$WORK_DIR/gh.headers"
if [[ -n "${GITHUB_TOKEN:-}" ]]; then
printf 'Authorization: Bearer %s\n' "$GITHUB_TOKEN" >> "$WORK_DIR/gh.headers"
log "GitHub authentication: token configured"
else
log "WARNING: GitHub authentication: anonymous. Set GITHUB_TOKEN in the configuration for a higher API quota"
fi
response_header() {
awk -v name="$2" '
/^HTTP\// { value="" }
tolower($1)==tolower(name ":") {
sub(/^[^:]*:[[:space:]]*/, ""); sub(/\r$/, ""); value=$0
}
END { print value }' "$1"
}
# GETs may retry. Never automatically repeat a mutation: a lost response can
# follow a successful POST, and retrying it would create duplicates.
request() {
local service=$1 method=$2 url=$3 output=$4 payload=${5:-} allow_missing=${6:-0} status
local headers="$WORK_DIR/response.headers" message remaining reset retry_after wait_seconds
local attempt=0 waited=0 now reset_wait
local -a args=(-q --silent --show-error --connect-timeout 15 --max-time 120
--user-agent 'gitea-github-sync/3.1' -D "$headers"
-H "@$HEADER_DIR/$service.headers" -X "$method" -o "$output" -w '%{http_code}')
[[ "$method" == GET || "$MODE" == apply ]] || die "Write request blocked in dry-run"
# GitHub throttling must be inspected before any retry, including HTTP 429.
if [[ "$method" == GET && "$service" != gh ]]; then
args+=(--retry 3 --retry-delay 2 --retry-max-time 180)
fi
if [[ -n "$payload" ]]; then
args+=(-H 'Content-Type: application/json' --data-binary "@$payload")
fi
while :; do
status=$(curl "${args[@]}" "$url") || die "Transport error: $method $url; run dry-run before retrying APPLY"
if [[ "$status" == 2[0-9][0-9] ]]; then return 0; fi
if [[ "$status" == 404 && "$allow_missing" == 1 ]]; then return 4; fi
message=$(jq -r 'if type=="object" and (.message|type)=="string" then
.message | gsub("[[:cntrl:]]"; " ") | .[:1000]
else "No JSON error message in response" end' "$output" 2>/dev/null) || message='Non-JSON error response (possibly proxy or firewall)'
message=${message//"$GITEA_TOKEN"/[REDACTED]}
if [[ -n "${GITHUB_TOKEN:-}" ]]; then message=${message//"$GITHUB_TOKEN"/[REDACTED]}; fi
log "HTTP $status: $method $url; API message: $message"
remaining=$(response_header "$headers" x-ratelimit-remaining)
reset=$(response_header "$headers" x-ratelimit-reset)
retry_after=$(response_header "$headers" retry-after)
if [[ "$service" == gh && ( "$status" == 403 || "$status" == 429 ) ]] &&
[[ "$remaining" == 0 || "$status" == 429 || -n "$retry_after" || "${message,,}" == *"rate limit"* ]]; then
log "GitHub rate limit: remaining=${remaining:-unknown}, reset_epoch=${reset:-unknown}, retry_after=${retry_after:-absent}"
now=$(date +%s)
wait_seconds=$((60 * (2 ** attempt)))
if [[ "$retry_after" =~ ^[0-9]{1,9}$ ]]; then
wait_seconds=$((10#$retry_after))
elif [[ -n "$retry_after" ]]; then
reset_wait=$(date -d "$retry_after" +%s 2>/dev/null) || reset_wait=$((now + 60))
wait_seconds=$((reset_wait - now))
fi
if [[ "$remaining" == 0 && "$reset" =~ ^[0-9]{1,12}$ ]]; then
reset_wait=$((10#$reset - now + 1))
if [[ -z "$retry_after" ]] || ((reset_wait > wait_seconds)); then wait_seconds=$reset_wait; fi
log "GitHub quota resets at $(date -u -d "@$reset" '+%Y-%m-%d %H:%M:%S UTC' 2>/dev/null || printf '%s' "$reset")"
fi
if ((wait_seconds < 1)); then wait_seconds=1; fi
if [[ "$method" == GET ]] && ((attempt < GITHUB_RATE_LIMIT_RETRIES && waited + wait_seconds <= GITHUB_RATE_LIMIT_MAX_WAIT)); then
log "Waiting $wait_seconds seconds before retrying GitHub (attempt $((attempt + 1)))"
sleep "$wait_seconds"
waited=$((waited + wait_seconds))
attempt=$((attempt + 1))
continue
fi
printf '%s\n' 'GitHub rate limit reached; wait for reset or correct GITHUB_TOKEN before rerunning' > "$RUN_DIR/github-stop"
die "GitHub rate limit wait/retry budget exceeded; stopping account scan"
fi
if [[ "$service" == gh && "$status" == 401 ]]; then
printf '%s\n' 'GitHub authentication failed; replace or correct GITHUB_TOKEN' > "$RUN_DIR/github-stop"
elif [[ "$service" == gh && "$status" == 403 ]]; then
log "GitHub access denied without a rate-limit signal. Check token repository access, organization/SSO policy, and any proxy/firewall"
fi
die "Request failed; see API message above"
done
}
validate() { jq -e "$2" "$1" >/dev/null || die "Invalid JSON data: ${1##*/}"; }
paginate() {
local service=$1 url=$2 output=$3 page count
: > "$output"
for ((page=1; page<=MAX_PAGES; page++)); do
request "$service" GET "$url&page=$page" "$WORK_DIR/page.json"
validate "$WORK_DIR/page.json" 'type == "array" and all(.[]; type == "object")'
count=$(jq length "$WORK_DIR/page.json")
((count > 0)) || return 0
if ((page > 1)) && cmp -s "$WORK_DIR/page.json" "$WORK_DIR/previous-page.json"; then
die "API returned a repeated page: $url"
fi
cat "$WORK_DIR/page.json" > "$WORK_DIR/previous-page.json"
jq -c '.[]' "$WORK_DIR/page.json" >> "$output"
# Continue to an empty page: servers may cap the requested page size.
done
die "MAX_PAGES exceeded for $url"
}
marker() {
# Existing V2.1 markers are supported. Ambiguous markers stop the run.
jq -r --arg kind "$2" '
[(.body // "") | scan("<!-- " + $kind + ":([0-9]+) -->") | .[0]] |
if length > 1 then error("Multiple markers in one record")
else .[0] // "" end' "$1"
}
issue_schema='def pos: type == "number" and . > 0 and floor == .;
all(.[]; (.id|pos) and (.number|pos) and (.title|type=="string") and
(.body == null or (.body|type=="string")) and (.state=="open" or .state=="closed"))'
comment_schema='def pos: type == "number" and . > 0 and floor == .;
all(.[]; (.id|pos) and (.body == null or (.body|type=="string")))'
# Run each repository in its own subshell and directory. The parent waits for
# every worker and continues after a failure without disabling worker errexit.
sync_repository() (
set -Eeuo pipefail
local entry=$1 repo_id source current_source
local GITEA_OWNER GITEA_REPO GITHUB_OWNER GITHUB_REPO WORK_DIR
repo_id=$(jq -r .id <<< "$entry")
GITEA_OWNER=$(jq -r .owner.login <<< "$entry")
GITEA_REPO=$(jq -r .name <<< "$entry")
REPO_LABEL="$GITEA_OWNER/$GITEA_REPO"
source=$(github_source "$(jq -r .original_url <<< "$entry")")
IFS=$'\t' read -r GITHUB_OWNER GITHUB_REPO <<< "$source"
WORK_DIR="$RUN_DIR/repo-$repo_id"
mkdir -- "$WORK_DIR"
# The parent owns cleanup; do not remove shared credentials from a worker.
trap - EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
GT="$GITEA_URL/api/v1/repos/$GITEA_OWNER/$GITEA_REPO"
GH="$GITHUB_API/repos/$GITHUB_OWNER/$GITHUB_REPO"
log "GitHub $GITHUB_OWNER/$GITHUB_REPO -> Gitea $GITEA_OWNER/$GITEA_REPO; mode $MODE"
request gt GET "$GT" "$WORK_DIR/gt-repo.json"
request gh GET "$GH" "$WORK_DIR/gh-repo.json"
for service in gt gh; do
validate "$WORK_DIR/$service-repo.json" 'type=="object" and (.id|type=="number") and
(.default_branch|type=="string") and (.description==null or (.description|type=="string")) and
(.website==null or (.website|type=="string")) and (.homepage==null or (.homepage|type=="string"))'
done
validate "$WORK_DIR/gt-repo.json" '.archived != true and .mirror == true'
is_mirror=true
current_source=$(github_source "$(jq -r '.original_url // ""' "$WORK_DIR/gt-repo.json")")
[[ "${current_source,,}" == "${source,,}" ]] || die "Mirror source changed after discovery; rerun discovery"
sync_issues=$(jq -r '.has_issues == true and .external_tracker == null' "$WORK_DIR/gt-repo.json")
if [[ "$sync_issues" != true ]]; then
log "WARNING: Internal issues are disabled; synchronizing metadata and topics only"
fi
request gh GET "$GH/topics" "$WORK_DIR/gh-topics.json"
request gt GET "$GT/topics" "$WORK_DIR/gt-topics.json"
for service in gh gt; do
validate "$WORK_DIR/$service-topics.json" '(.topics // .names) | type=="array" and all(.[]; type=="string")'
done
jq '{topics:(.names | sort | unique)}' "$WORK_DIR/gh-topics.json" > "$WORK_DIR/topics-payload.json"
jq '{topics:((.topics // .names) | sort | unique)}' "$WORK_DIR/gt-topics.json" > "$WORK_DIR/topics-old.json"
jq '{description:(.description // ""), website:(.homepage // ""), default_branch}' "$WORK_DIR/gh-repo.json" > "$WORK_DIR/repo-payload.json"
jq '{description:(.description // ""), website:(.website // ""), default_branch}' "$WORK_DIR/gt-repo.json" > "$WORK_DIR/repo-old.json"
# Complete the read/validation phase before any writes.
log "Reading issues and comments (excluding pull requests)..."
if [[ "$sync_issues" == true ]]; then
paginate gh "$GH/issues?state=all&sort=created&direction=asc&per_page=100" "$WORK_DIR/gh-all.jsonl"
paginate gt "$GT/issues?state=all&type=issues&limit=50" "$WORK_DIR/gt-all.jsonl"
else
: > "$WORK_DIR/gh-all.jsonl"
: > "$WORK_DIR/gt-all.jsonl"
fi
for service in gh gt; do
jq -s '[.[] | select(.pull_request == null)] | sort_by(.number)' "$WORK_DIR/$service-all.jsonl" > "$WORK_DIR/$service-issues.json"
validate "$WORK_DIR/$service-issues.json" "$issue_schema"
jq -c '.[]' "$WORK_DIR/$service-issues.json" > "$WORK_DIR/$service-issues.jsonl"
done
declare -A gh_numbers=() gt_numbers=() gt_comment_ids=() seen_gh=()
while IFS= read -r record; do
printf '%s\n' "$record" > "$WORK_DIR/record.json"
id=$(jq -r .id "$WORK_DIR/record.json")
number=$(jq -r .number "$WORK_DIR/record.json")
[[ -z "${seen_gh[$id]+x}" && -z "${gh_numbers[$number]+x}" ]] || die "Duplicate GitHub issue $id"
seen_gh[$id]=1
gh_numbers[$number]=$id
done < "$WORK_DIR/gh-issues.jsonl"
while IFS= read -r record; do
printf '%s\n' "$record" > "$WORK_DIR/record.json"
id=$(marker "$WORK_DIR/record.json" github-sync-id)
[[ -n "$id" ]] || continue
[[ -z "${gt_numbers[$id]+x}" ]] || die "Duplicate github-sync-id:$id in Gitea"
number=$(jq -r .number "$WORK_DIR/record.json")
gt_numbers[$id]=$number
printf '%s\n' "$record" > "$WORK_DIR/gt-issue-$id.json"
paginate gt "$GT/issues/$number/comments?limit=50" "$WORK_DIR/comments.jsonl"
jq -s '.' "$WORK_DIR/comments.jsonl" > "$WORK_DIR/comments.json"
validate "$WORK_DIR/comments.json" "$comment_schema"
while IFS= read -r comment; do
printf '%s\n' "$comment" > "$WORK_DIR/record.json"
cid=$(marker "$WORK_DIR/record.json" github-comment-id)
[[ -n "$cid" ]] || continue
key="$number:$cid"
[[ -z "${gt_comment_ids[$key]+x}" ]] || die "Duplicate github-comment-id:$cid in Gitea #$number"
gt_comment_ids[$key]=$(jq -r .id "$WORK_DIR/record.json")
printf '%s\n' "$comment" > "$WORK_DIR/gt-comment-$number-$cid.json"
done < "$WORK_DIR/comments.jsonl"
done < "$WORK_DIR/gt-issues.jsonl"
if [[ "$sync_issues" == true ]]; then
paginate gh "$GH/issues/comments?sort=created&direction=asc&per_page=100" "$WORK_DIR/gh-comments.jsonl"
else
: > "$WORK_DIR/gh-comments.jsonl"
fi
jq -s '.' "$WORK_DIR/gh-comments.jsonl" > "$WORK_DIR/gh-comments.json"
validate "$WORK_DIR/gh-comments.json" "$comment_schema and all(.[]; (.issue_url|type==\"string\"))"
declare -A seen_comments=()
while IFS= read -r record; do
printf '%s\n' "$record" > "$WORK_DIR/record.json"
number=$(jq -r '.issue_url | split("/") | last' "$WORK_DIR/record.json")
[[ "$number" =~ ^[1-9][0-9]*$ ]] || die "Invalid issue number in GitHub comment"
[[ -n "${gh_numbers[$number]+x}" ]] || continue
cid=$(jq -r .id "$WORK_DIR/record.json")
[[ -z "${seen_comments[$cid]+x}" ]] || die "Duplicate GitHub comment $cid"
seen_comments[$cid]=1
printf '%s\n' "$record" >> "$WORK_DIR/gh-comments-$number.jsonl"
done < "$WORK_DIR/gh-comments.jsonl"
# Payloads are built from files, avoiding shell argument limits and retaining
# Unicode and trailing newlines. Ignore source copies of our reserved markers.
# shellcheck disable=SC2016
body_filter='def marked($kind; $id):
(. // "" | gsub("<!-- " + $kind + ":[0-9]+ -->"; "")) as $b |
$b + (if $b=="" then "" else "\n\n" end) + "<!-- " + $kind + ":" + ($id|tostring) + " -->";'
issues_create=0 issues_update=0 issues_same=0
comments_create=0 comments_update=0 comments_same=0
mutations=0
deferred_branch=0
write_api() {
request gt "$1" "$2" "$WORK_DIR/response.json" "${3:-}"
mutations=$((mutations + 1))
}
if ! cmp -s "$WORK_DIR/repo-old.json" "$WORK_DIR/repo-payload.json"; then
log "Repository metadata: update planned"
# Mirror sync is asynchronous. Do not set a branch that does not exist yet.
branch=$(jq -r .default_branch "$WORK_DIR/repo-payload.json")
old_branch=$(jq -r .default_branch "$WORK_DIR/repo-old.json")
if [[ "$branch" != "$old_branch" ]]; then
log "Default branch must exist in Gitea: $branch"
encoded=$(jq -rn --arg v "$branch" '$v|@uri')
if request gt GET "$GT/branches/$encoded" "$WORK_DIR/branch.json" '' 1; then
validate "$WORK_DIR/branch.json" '(.name|type=="string")'
else
log "Branch is not available in Gitea; deferring default_branch. Retry after mirror-sync completes."
jq 'del(.default_branch)' "$WORK_DIR/repo-payload.json" > "$WORK_DIR/repo-without-branch.json"
cat "$WORK_DIR/repo-without-branch.json" > "$WORK_DIR/repo-payload.json"
deferred_branch=1
fi
fi
if [[ "$MODE" == apply ]]; then write_api PATCH "$GT" "$WORK_DIR/repo-payload.json"; fi
fi
if ! cmp -s "$WORK_DIR/topics-old.json" "$WORK_DIR/topics-payload.json"; then
log "Repository topics: update planned"
if [[ "$MODE" == apply ]]; then write_api PUT "$GT/topics" "$WORK_DIR/topics-payload.json"; fi
fi
while IFS= read -r record; do
printf '%s\n' "$record" > "$WORK_DIR/source.json"
id=$(jq -r .id "$WORK_DIR/source.json")
number=$(jq -r .number "$WORK_DIR/source.json")
target=${gt_numbers[$id]:-}
jq "$body_filter"'{title, body:(.body | marked("github-sync-id"; $id)), state}' \
--argjson id "$id" "$WORK_DIR/source.json" > "$WORK_DIR/issue-payload.json"
if [[ -z "$target" ]]; then
issues_create=$((issues_create + 1))
log "CREATE issue GitHub #$number"
if [[ "$MODE" == apply ]]; then
jq '{title, body, closed:(.state=="closed")}' "$WORK_DIR/issue-payload.json" > "$WORK_DIR/create.json"
write_api POST "$GT/issues" "$WORK_DIR/create.json"
validate "$WORK_DIR/response.json" '(.number|type=="number" and .>0 and floor==.)'
target=$(jq -r .number "$WORK_DIR/response.json")
gt_numbers[$id]=$target
fi
else
jq '{title, body:(.body // ""), state}' "$WORK_DIR/gt-issue-$id.json" > "$WORK_DIR/old.json"
if cmp -s "$WORK_DIR/old.json" "$WORK_DIR/issue-payload.json"; then
issues_same=$((issues_same + 1))
else
issues_update=$((issues_update + 1))
log "UPDATE issue Gitea #$target <- GitHub #$number"
if [[ "$MODE" == apply ]]; then write_api PATCH "$GT/issues/$target" "$WORK_DIR/issue-payload.json"; fi
fi
fi
[[ -f "$WORK_DIR/gh-comments-$number.jsonl" ]] || continue
while IFS= read -r comment; do
printf '%s\n' "$comment" > "$WORK_DIR/source.json"
cid=$(jq -r .id "$WORK_DIR/source.json")
jq "$body_filter"'{body:(.body | marked("github-comment-id"; $id))}' \
--argjson id "$cid" "$WORK_DIR/source.json" > "$WORK_DIR/comment-payload.json"
key="$target:$cid"
existing=${gt_comment_ids[$key]:-}
if [[ -z "$existing" ]]; then
comments_create=$((comments_create + 1))
log "CREATE GitHub comment $cid on issue #$number"
if [[ "$MODE" == apply ]]; then write_api POST "$GT/issues/$target/comments" "$WORK_DIR/comment-payload.json"; fi
else
jq '{body:(.body // "")}' "$WORK_DIR/gt-comment-$target-$cid.json" > "$WORK_DIR/old.json"
if cmp -s "$WORK_DIR/old.json" "$WORK_DIR/comment-payload.json"; then
comments_same=$((comments_same + 1))
else
comments_update=$((comments_update + 1))
log "UPDATE Gitea comment $existing <- GitHub $cid"
if [[ "$MODE" == apply ]]; then write_api PATCH "$GT/issues/comments/$existing" "$WORK_DIR/comment-payload.json"; fi
fi
fi
done < "$WORK_DIR/gh-comments-$number.jsonl"
done < "$WORK_DIR/gh-issues.jsonl"
if ((NO_GIT)); then
log "Mirror-sync skipped (--no-git)"
elif [[ "$is_mirror" != true ]]; then
log "Mirror-sync skipped: repository is not a mirror"
elif [[ "$MODE" == apply ]]; then
write_api POST "$GT/mirror-sync"
log "Mirror-sync queued; check completion in Gitea"
else
log "DRY-RUN: mirror-sync would be queued"
fi
log "Issues: create=$issues_create, update=$issues_update, unchanged=$issues_same"
log "Comments: create=$comments_create, update=$comments_update, unchanged=$comments_same"
if [[ "$MODE" == apply ]] && ((deferred_branch)); then
die "Default branch update pending. Successful writes: $mutations. Wait for Git synchronization and rerun."
fi
log "Completed $MODE; successful write requests: $mutations"
)
# Parse only GitHub.com clone URLs. Do not log original_url: it can contain
# embedded credentials. Destination names may differ from source names.
github_source() {
jq -rn --arg url "$1" '
($url | capture("(?i)^(?:https?://(?:[^/@]+@)?github\\.com(?::443)?/|ssh://(?:git@)?github\\.com(?::22)?/|git@github\\.com:)(?<owner>[a-z0-9_.-]+)/(?<repo>[a-z0-9_.-]+)/?$")?) as $m |
if $m == null then "" else
($m.repo | sub("(?i)\\.git$"; "")) as $repo |
if $m.owner == "." or $m.owner == ".." or $repo == "" or $repo == "." or $repo == ".."
then "" else [$m.owner, $repo] | @tsv end
end'
}
if [[ -n "${GITEA_REPO:-}${GITHUB_OWNER:-}${GITHUB_REPO:-}" ]]; then
log "Legacy GITEA_REPO/GITHUB_OWNER/GITHUB_REPO settings are ignored; repositories are discovered automatically"
fi
if [[ -z "$GITEA_OWNER" ]]; then
request gt GET "$GITEA_URL/api/v1/user" "$WORK_DIR/user.json"
validate "$WORK_DIR/user.json" '(.login|type=="string" and length>0)'
GITEA_OWNER=$(jq -r .login "$WORK_DIR/user.json")
fi
[[ "$GITEA_OWNER" =~ ^[A-Za-z0-9_.-]+$ && "$GITEA_OWNER" != . && "$GITEA_OWNER" != .. ]] || die "Invalid GITEA_OWNER"
log "Discovering repositories owned by $GITEA_OWNER"
paginate gt "$GITEA_URL/api/v1/users/$GITEA_OWNER/repos?limit=50" "$WORK_DIR/repositories.jsonl"
jq -s '.' "$WORK_DIR/repositories.jsonl" > "$WORK_DIR/repositories.json"
validate "$WORK_DIR/repositories.json" 'all(.[];
(.id|type=="number" and .>0 and floor==.) and
(.name|type=="string" and test("^[A-Za-z0-9_.-]+$") and .!="." and .!="..") and
(.owner.login|type=="string" and test("^[A-Za-z0-9_.-]+$") and .!="." and .!="..") and
(.mirror|type=="boolean") and (.original_url==null or (.original_url|type=="string")))'
declare -A discovered_ids=()
discovered=0 selected=0 skipped=0 succeeded=0 failed=0
while IFS= read -r entry; do
repo_id=$(jq -r .id <<< "$entry")
[[ -z "${discovered_ids[$repo_id]+x}" ]] || continue
discovered_ids[$repo_id]=1
discovered=$((discovered + 1))
full_name=$(jq -r '.owner.login + "/" + .name' <<< "$entry")
owner=$(jq -r .owner.login <<< "$entry")
if [[ "${owner,,}" != "${GITEA_OWNER,,}" ]] || [[ $(jq -r .mirror <<< "$entry") != true ]]; then
skipped=$((skipped + 1))
log "SKIP $full_name: not a pull mirror owned by $GITEA_OWNER"
continue
fi
source=$(github_source "$(jq -r '.original_url // ""' <<< "$entry")")
if [[ -z "$source" ]]; then
skipped=$((skipped + 1))
log "SKIP $full_name: no supported GitHub.com source URL"
continue
fi
selected=$((selected + 1))
if [[ $(jq -r '.archived // false' <<< "$entry") == true ]]; then
failed=$((failed + 1))
log "FAILED $full_name: archived GitHub mirror cannot be updated"
continue
fi
sync_repository "$entry" &
worker_pid=$!
if wait "$worker_pid"; then
succeeded=$((succeeded + 1))
log "OK $full_name"
else
failed=$((failed + 1))
log "FAILED $full_name"
fi
worker_pid=
if [[ -f "$RUN_DIR/github-stop" ]]; then
log "Account scan aborted: $(cat "$RUN_DIR/github-stop"). Remaining repositories were not attempted"
break
fi
done < "$WORK_DIR/repositories.jsonl"
log "Repositories: discovered=$discovered, selected=$selected, succeeded=$succeeded, failed=$failed, skipped=$skipped"
((failed == 0)) || exit 1
log "Account synchronization completed ($MODE)"