BUG: Release assets are sometimes duplicated #166

New Issue

2026-10-05T01:45:31+02:00

adam commented

2026-10-05 01:45:31 +02:00

I noticed that on some repos I mirrored including the release assetsof the latest 1 releases, the release assets get mirrored more than once. On some repos they are mirrored 1:1, some are doubled, so 1:2 sometimes even trippled. The most amount of duplicates I saw on a repo was 1 asset being duplicated five times 1:5. Something seems to go on there, probably no duplication check is being done. Maybe some concurrencry that starts the download more than once and the assets get mirrored more than once as a result. Here an example of the assets getting doubled:

Original on github:

Forgejo after syncing:

I noticed that on some repos I mirrored including the release assetsof the latest 1 releases, the release assets get mirrored more than once. On some repos they are mirrored 1:1, some are doubled, so 1:2 sometimes even trippled. The most amount of duplicates I saw on a repo was 1 asset being duplicated five times 1:5. Something seems to go on there, probably no duplication check is being done. Maybe some concurrencry that starts the download more than once and the assets get mirrored more than once as a result. Here an example of the assets getting doubled: Original on github: <img width="1080" height="1258" alt="Image" src="https://github.com/user-attachments/assets/a4ab5dcc-7424-487b-9322-b7eb2adbb19c" /> Forgejo after syncing: <img width="1080" height="1889" alt="Image" src="https://github.com/user-attachments/assets/9b56bdde-9c5d-49e4-9f87-3092baab6c7b" />

adam closed this issue

2026-10-05 01:45:31 +02:00

adam commented

2026-10-05 01:45:32 +02:00

Thanks for the screenshots, they settled it. Both copies of every asset have the same size and the same creation time, so these are not two different syncs each adding a copy: two passes overlapped inside a single run. Fixed in v3.36.0 (#419).

What was happening: Gitea and Forgejo do not check attachment names on upload. Every upload POST appends a new attachment, so one release can hold any number of attachments called foo.zip. gitea-mirror listed the attachments a release already had, then downloaded from GitHub and uploaded whatever looked missing, which for large assets takes tens of seconds or minutes. Nothing prevented a second pass over the same repository from starting inside that window (the scheduler racing a manual run, crash recovery resuming a job that was still alive, or two repository rows that resolve to the same destination repository). Both passes saw the asset as missing, both uploaded it. Three overlapping passes gave three copies, and so on.

What changed:

Release reconciliation for a given destination repository runs one pass at a time. Overlapping runs queue instead of racing.
The asset comparison understands duplicates. For each upstream asset it keeps one copy on the destination and deletes the surplus ones; when no copy matches the upstream size it removes them all and uploads once.
It fails closed. If the destination cannot tell us which attachments a release already has, nothing is uploaded for that release and the next sync retries. If a stale copy cannot be deleted, the upload is skipped instead of adding one more copy.
A sync no longer starts on a repository that is already being mirrored or synced, and the mirror path claims the repository in one database statement instead of checking the status and then writing it.

About the duplicates you already have: the next sync of an affected repository cleans them up for every release that still gets its assets, leaving one copy of each. Releases that have since fallen outside the "newest N releases" window are no longer visited by a sync, so their duplicates stay until you delete them by hand (or raise that repository's asset limit, sync once, and lower it again).

Thanks for the screenshots, they settled it. Both copies of every asset have the same size and the same creation time, so these are not two different syncs each adding a copy: two passes overlapped inside a single run. Fixed in v3.36.0 (#419). What was happening: Gitea and Forgejo do not check attachment names on upload. Every upload POST appends a new attachment, so one release can hold any number of attachments called `foo.zip`. gitea-mirror listed the attachments a release already had, then downloaded from GitHub and uploaded whatever looked missing, which for large assets takes tens of seconds or minutes. Nothing prevented a second pass over the same repository from starting inside that window (the scheduler racing a manual run, crash recovery resuming a job that was still alive, or two repository rows that resolve to the same destination repository). Both passes saw the asset as missing, both uploaded it. Three overlapping passes gave three copies, and so on. What changed: - Release reconciliation for a given destination repository runs one pass at a time. Overlapping runs queue instead of racing. - The asset comparison understands duplicates. For each upstream asset it keeps one copy on the destination and deletes the surplus ones; when no copy matches the upstream size it removes them all and uploads once. - It fails closed. If the destination cannot tell us which attachments a release already has, nothing is uploaded for that release and the next sync retries. If a stale copy cannot be deleted, the upload is skipped instead of adding one more copy. - A sync no longer starts on a repository that is already being mirrored or synced, and the mirror path claims the repository in one database statement instead of checking the status and then writing it. About the duplicates you already have: the next sync of an affected repository cleans them up for every release that still gets its assets, leaving one copy of each. Releases that have since fallen outside the "newest N releases" window are no longer visited by a sync, so their duplicates stay until you delete them by hand (or raise that repository's asset limit, sync once, and lower it again).

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: adam/gitea-mirror#166