mirror of
https://github.com/RayLabsHQ/gitea-mirror.git
synced 2026-11-03 07:42:19 +01:00
Mirror makes many copies #77
Assignees
adam (Adam Wojcieszonek)
Clear assignees
adam (Adam Wojcieszonek)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: adam/gitea-mirror#77
Reference in new issue
Block a user
Hi
I haven't found anyone else with this issue, but it creates many copies of my repositories. When having LFS enabled i suspect they can take up lots of space if it continues like this.. Any idea why it does so? Example under here. When I check Gitea, I find many repositories like this:
starred/mame-mamedev-7
starred/mame-mamedev-6
starred/mame-mamedev-5
starred/mame-mamedev-4
starred/mame-mamedev-3
starred/mame-mamedev-2
starred/mame-mamedev-1
starred/mame-mamedev
@groke76 first are you running latest gitea and gitea-mirror cause I addressed similar race conditions earlier and if you can also send some docker logs would be great for some more context.
Similar:
https://github.com/RayLabsHQ/gitea-mirror/issues/115
Hi, yes I'm running the latest ones. I have logs for both Gitea and Gitea-mirror here. Woke up now to check and I have 113 repos downloaded, and I have no more than 80 of them. There are many duplicates that I can see here. It seems that it starts to download a new one if the previous one is falsly labeled with "synced" instead of "mirrored", and/or if they fail the first time, then creates a new one, and then tries the first one again later. Gitea also uses a lot of time to import (compared to GitLab for example), and maybe gitea-mirror becomes "impatient" and then just duplicates recursively instead
_gitea-mirror_logs.txt
_gitea_logs.txt
@groke76 Hey if the repo is too large then there is a timeout currently can you confirm that its only happening with very large repos?
Also the logs you pasted does not show any of the issues, I am guessing it was just tail of the logs if you can paste like part of the log where the duplicates happen that would be super helpful. and maybe for large repos I can expose a custom timeout that you can set using the env vars
I have the same problem and it only occures at large repos. Thats pretty sad. :/
@PascalH214 I think the issue is that your network is not fast enough with the current timeouts for large repos and thats whats casuing this. I will make the timeouts configurable for people with large repos or reimplement it maybe.
@arunavo4 Meanwhile, I think the problem lies with my Gitea setup. It is failing to migrate the repository. My server can clone the repository in just four seconds. However, Gitea takes two hours to migrate the same repository. I don't know where the problem lies, so configurable timeouts wouldn't solve it. Sorry for commenting here. I need to find out how to fix the problem within Gitea.
For me its the issues. They are mostly duplicate. How can i sovle this?
@emrebasarannn can you put some related logs here where there are duplicate issues. and also can you check if its happening for one repo or more than one?
@arunavo4 Hi,
Sharing logs might be overwhelming since i am mirroring more then +200 repos. Here is an example of my situation, we can see 3 different issue has been writed for 9 times in total.
@arunavo4 Hi,
I guess i figured out the problem. Since my repo's are big(some of them has +10k issue) and while mirroring for the first time the gitea-mirror's self schedule also running(i setted to 5 minute), the issues were duplicating. Giving more time for the initial mirror solved my problem.
Ahhh okay maybe we should put some guards but it will make it very complex, maybe i will document this somewhere in readme. Thanks for reporting back @emrebasarannn
Thanks for the detailed reports. I agree this should be handled operationally rather than by adding complex guards.
I opened #179 to document a safer first-run setup for large repositories:
5m) during initial bootstrap1hto8h(or temporarily disable scheduling)This addresses the duplicate-looking retry pattern seen when Gitea import/migration is slow.
Resolved by #179 (merged): documented initial large-repo sync scheduling guidance to prevent duplicate-looking retries during bootstrap.