adam/gitea-mirror
1
0 Fork 0
mirror of https://github.com/RayLabsHQ/gitea-mirror.git synced 2026-10-30 22:21:58 +01:00
Code Issues 4 Packages Projects Releases Wiki Activity

All data on the gitea-mirror web has disappeared #121

New Issue
Closed
opened 2026-10-05 01:42:45 +02:00 by adam · 11 comments
adam commented 2026-10-05 01:42:45 +02:00
Owner
Copy link
Copy source

Hi.
Gitea-mirror v3.15.2
Suddenly, all the data on the web disappeared.
Image
But all data it's still in the database.
Image
I see this in the logs for all repos

2026-04-17T07:17:32.475442401Z [Repository Cleanup] Skipping orphaned repository strijar/RNS-Gate
2026-04-17T07:17:34.479896996Z [Repository Cleanup] Skipping orphaned repository strijar/RNode-linux
2026-04-17T07:17:36.482713264Z [Repository Cleanup] Skipping orphaned repository strijar/RNode_Firmware
2026-04-17T07:17:38.488549382Z [Repository Cleanup] Skipping orphaned repository strijar/reticulum-network-stack
2026-04-17T07:17:40.488445080Z [Repository Cleanup] Skipping orphaned repository torlando-tech/columba
2026-04-17T07:17:42.495427281Z [Repository Cleanup] Skipping orphaned repository trindade99/FreeNetBrowser
2026-04-17T07:17:44.498522320Z [Repository Cleanup] Skipping orphaned repository tubearchivist/browser-extension
2026-04-17T07:17:46.503010049Z [Repository Cleanup] Skipping orphaned repository tubearchivist/discord-bot

How can I get it all back? Manually adding everything back to 800+ repositories is terrifying...

UPD:
New repo adding ok in db, but don't appear in any part of dashboard in web.
Image

Hi. Gitea-mirror v3.15.2 Suddenly, all the data on the web disappeared. <img width="1116" height="144" alt="Image" src="https://github.com/user-attachments/assets/139f8629-4c6f-46da-b8e4-0900c0f31449" /> But all data it's still in the database. <img width="555" height="323" alt="Image" src="https://github.com/user-attachments/assets/69673454-37cd-4c0c-9906-fe5d5de54fcf" /> I see this in the logs for all repos ``` 2026-04-17T07:17:32.475442401Z [Repository Cleanup] Skipping orphaned repository strijar/RNS-Gate 2026-04-17T07:17:34.479896996Z [Repository Cleanup] Skipping orphaned repository strijar/RNode-linux 2026-04-17T07:17:36.482713264Z [Repository Cleanup] Skipping orphaned repository strijar/RNode_Firmware 2026-04-17T07:17:38.488549382Z [Repository Cleanup] Skipping orphaned repository strijar/reticulum-network-stack 2026-04-17T07:17:40.488445080Z [Repository Cleanup] Skipping orphaned repository torlando-tech/columba 2026-04-17T07:17:42.495427281Z [Repository Cleanup] Skipping orphaned repository trindade99/FreeNetBrowser 2026-04-17T07:17:44.498522320Z [Repository Cleanup] Skipping orphaned repository tubearchivist/browser-extension 2026-04-17T07:17:46.503010049Z [Repository Cleanup] Skipping orphaned repository tubearchivist/discord-bot ``` How can I get it all back? Manually adding everything back to 800+ repositories is terrifying... UPD: New repo adding ok in db, but don't appear in any part of dashboard in web. <img width="382" height="147" alt="Image" src="https://github.com/user-attachments/assets/c93cd8c6-36a5-4013-88df-cbe3bd186efe" /> <!-- github-sync-id:4281012293 -->
adam closed this issue 2026-10-05 01:42:45 +02:00
adam commented 2026-10-05 01:42:46 +02:00
Author
Owner
Copy link
Copy source

@stephennord Hey looks like somehow ur user got orphaned. thats weird are u using user name password ? or sso? also are u sure the web container has the db file? and were u able to sign in with the same user?

@stephennord Hey looks like somehow ur user got orphaned. thats weird are u using user name password ? or sso? also are u sure the web container has the db file? and were u able to sign in with the same user? <!-- github-comment-id:4268227578 -->
adam commented 2026-10-05 01:42:47 +02:00
Author
Owner
Copy link
Copy source

Yes, i'm use local username and password and can login in web environment with it.
Also GitHub and Gitea config test Ok. DB present and i can add new repo from GitHub (screenshot in topic message) and can see it in DB. I can see all activity log in gitea-mirror web ui, but no all other data.
Here docker console log at sturtup

System CA bundle mounted, configuring Node.js to use it...
NODE_EXTRA_CA_CERTS set to: /etc/ssl/certs/ca-certificates.crt
Using previously generated ENCRYPTION_SECRET
ENCRYPTION_SECRET has been set to a secure random value
Database already exists, Drizzle will check for pending migrations on startup...
Setting application version: 3.15.2
Checking for environment configuration...
No GitHub/Gitea environment variables found, skipping env config initialization.
✅ Environment configuration loaded successfully
Running startup recovery...
Running startup recovery using compiled script...
Successfully connected to SQLite database using Bun's native driver
🔄 Checking for pending migrations...
✅ Database migrations completed successfully

and activity log in web from DB
Image

Yes, i'm use local username and password and can login in web environment with it. Also GitHub and Gitea config test Ok. DB present and i can add new repo from GitHub (screenshot in topic message) and can see it in DB. I can see all activity log in gitea-mirror web ui, but no all other data. Here docker console log at sturtup ``` System CA bundle mounted, configuring Node.js to use it... NODE_EXTRA_CA_CERTS set to: /etc/ssl/certs/ca-certificates.crt Using previously generated ENCRYPTION_SECRET ENCRYPTION_SECRET has been set to a secure random value Database already exists, Drizzle will check for pending migrations on startup... Setting application version: 3.15.2 Checking for environment configuration... No GitHub/Gitea environment variables found, skipping env config initialization. ✅ Environment configuration loaded successfully Running startup recovery... Running startup recovery using compiled script... Successfully connected to SQLite database using Bun's native driver 🔄 Checking for pending migrations... ✅ Database migrations completed successfully ``` and activity log in web from DB <img width="246" height="557" alt="Image" src="https://github.com/user-attachments/assets/a66711ee-c1da-4322-a7a6-0c64fc052f69" /> <!-- github-comment-id:4269850957 -->
adam commented 2026-10-05 01:42:47 +02:00
Author
Owner
Copy link
Copy source

Hey @stephennord, thanks for the extra info — that helps narrow it down a lot.

Since you can sign in fine with the same local password, can add new repos and see them write to the DB, and your startup log says env-loader didn't touch anything, the GitHub/Gitea side and the new session are healthy. The only thing that fits "rows are in the DB but the dashboard says 0" is that those ~800 rows are tied to a different user_id than the one your current session resolves to. The dashboard and repositories API both filter strictly by the logged-in user id, so any drift there gives exactly this empty UI.

Important

Before running anything below, stop the container and take a copy of the SQLite DB file. Something like cp /app/data/gitea-mirror.db /app/data/gitea-mirror.db.bak-2026-04-20 from inside the container, or copy it on the host from your mounted volume. Don't skip this — if anything goes sideways with the UPDATEs, the backup is what gets you back.

Once you have the backup, exec into the container and run these against the SQLite file to confirm what's going on:

sqlite3 /app/data/gitea-mirror.db

SELECT id, email, username, createdAt FROM users ORDER BY createdAt;
SELECT user_id, COUNT(*) AS n FROM repositories GROUP BY user_id;
SELECT id, user_id, isActive, name FROM configs;
SELECT user_id, COUNT(*) FROM mirror_jobs GROUP BY user_id;

If you see two user rows (or one row but a different user_id on the repos), that's the split. Re-point everything to your live user id:

BEGIN;
UPDATE repositories  SET user_id = '<live-id>' WHERE user_id = '<old-id>';
UPDATE organizations SET user_id = '<live-id>' WHERE user_id = '<old-id>';
UPDATE configs       SET user_id = '<live-id>' WHERE user_id = '<old-id>';
UPDATE mirror_jobs   SET user_id = '<live-id>' WHERE user_id = '<old-id>';
UPDATE events        SET user_id = '<live-id>' WHERE user_id = '<old-id>';
COMMIT;

Run the SELECTs again to confirm the counts moved over, then start the container and your dashboard should light up. You can drop the empty old user row afterwards.

If the query shows everything is already on a single user id and the count matches what you see in the DB, then we're chasing a different bug and I'd want to see the network response from /api/dashboard and /api/github/repositories in your browser devtools.

About the cleanup spam in the logs — please don't worry about that part, your data isn't going anywhere. With orphanedRepoAction: skip (which is what the log line means) nothing is being archived or deleted, it's just noting that those repos didn't show up in the GitHub fetch this cycle. After the user id is fixed I'd also double-check includeStarred and the is_starred column on those rows, since most of them look like starred repos and the protection only kicks in when both align.

Hey @stephennord, thanks for the extra info — that helps narrow it down a lot. Since you can sign in fine with the same local password, can add new repos and see them write to the DB, and your startup log says env-loader didn't touch anything, the GitHub/Gitea side and the new session are healthy. The only thing that fits "rows are in the DB but the dashboard says 0" is that those ~800 rows are tied to a different `user_id` than the one your current session resolves to. The dashboard and repositories API both filter strictly by the logged-in user id, so any drift there gives exactly this empty UI. > [!IMPORTANT] > **Before running anything below, stop the container and take a copy of the SQLite DB file.** Something like `cp /app/data/gitea-mirror.db /app/data/gitea-mirror.db.bak-2026-04-20` from inside the container, or copy it on the host from your mounted volume. Don't skip this — if anything goes sideways with the UPDATEs, the backup is what gets you back. Once you have the backup, exec into the container and run these against the SQLite file to confirm what's going on: ```bash sqlite3 /app/data/gitea-mirror.db ``` ```sql SELECT id, email, username, createdAt FROM users ORDER BY createdAt; SELECT user_id, COUNT(*) AS n FROM repositories GROUP BY user_id; SELECT id, user_id, isActive, name FROM configs; SELECT user_id, COUNT(*) FROM mirror_jobs GROUP BY user_id; ``` If you see two user rows (or one row but a different `user_id` on the repos), that's the split. Re-point everything to your live user id: ```sql BEGIN; UPDATE repositories SET user_id = '<live-id>' WHERE user_id = '<old-id>'; UPDATE organizations SET user_id = '<live-id>' WHERE user_id = '<old-id>'; UPDATE configs SET user_id = '<live-id>' WHERE user_id = '<old-id>'; UPDATE mirror_jobs SET user_id = '<live-id>' WHERE user_id = '<old-id>'; UPDATE events SET user_id = '<live-id>' WHERE user_id = '<old-id>'; COMMIT; ``` Run the SELECTs again to confirm the counts moved over, then start the container and your dashboard should light up. You can drop the empty old user row afterwards. If the query shows everything is already on a single user id and the count matches what you see in the DB, then we're chasing a different bug and I'd want to see the network response from `/api/dashboard` and `/api/github/repositories` in your browser devtools. About the cleanup spam in the logs — please don't worry about that part, your data isn't going anywhere. With `orphanedRepoAction: skip` (which is what the log line means) nothing is being archived or deleted, it's just noting that those repos didn't show up in the GitHub fetch this cycle. After the user id is fixed I'd also double-check `includeStarred` and the `is_starred` column on those rows, since most of them look like starred repos and the protection only kicks in when both align. <!-- github-comment-id:4277708396 -->
adam commented 2026-10-05 01:42:48 +02:00
Author
Owner
Copy link
Copy source

sql query shows everything is already on a single user id and the count matches what i see in the DB,
but some column has different name such "createdAt" -> "created_at" in "user" table and "isActive" -> "is_active" in "configs" table in my DB.
json file from api attached.

dashboard.json

repositories.json

sql query shows everything is already on a single user id and the count matches what i see in the DB, but some column has different name such "createdAt" -> "created_at" in "user" table and "isActive" -> "is_active" in "configs" table in my DB. json file from api attached. [dashboard.json](https://github.com/user-attachments/files/26933861/dashboard.json) [repositories.json](https://github.com/user-attachments/files/26933869/repositories.json) <!-- github-comment-id:4288752634 -->
adam commented 2026-10-05 01:42:49 +02:00
Author
Owner
Copy link
Copy source

@stephennord thanks for the json files. Looks like it's due to a small bug in dashboard. try v3.15.4 and try and reload the dashboard.

@stephennord thanks for the json files. Looks like it's due to a small bug in dashboard. try v3.15.4 and try and reload the dashboard. <!-- github-comment-id:4293289775 -->
adam commented 2026-10-05 01:42:49 +02:00
Author
Owner
Copy link
Copy source

updated to 3.15.4 clear cache but nothing changes, still no data at web ui.

updated to 3.15.4 clear cache but nothing changes, still no data at web ui. <!-- github-comment-id:4294909640 -->
adam commented 2026-10-05 01:42:50 +02:00
Author
Owner
Copy link
Copy source

Thanks @stephennord Can you try the below things-

  1. The startup log line showing the version. You should see Setting application version: 3.15.4 somewhere early in the container log. If it still says 3.15.2 or 3.15.3, the image didn't actually roll.
  2. The response body of GET /api/config?userId=... from your browser's Network tab (reload the dashboard, grab the one to /api/config, copy the JSON). Redact the token values with <redacted> but leave the field names so we can see what's present vs empty.
  3. The container logs from the moment you reloaded with something like docker logs gitea-mirror --since 5m. If there's an error (decryption failure, JSON parse error), it'll show up there but the browser wouldn't have surfaced it.

With those three we should be able to see exactly which field the UI thinks is empty and why.

Thanks @stephennord Can you try the below things- 1. The startup log line showing the version. You should see `Setting application version: 3.15.4` somewhere early in the container log. If it still says `3.15.2` or `3.15.3`, the image didn't actually roll. 2. The response body of `GET /api/config?userId=...` from your browser's Network tab (reload the dashboard, grab the one to `/api/config`, copy the JSON). Redact the `token` values with `<redacted>` but leave the field names so we can see what's present vs empty. 3. The container logs from the moment you reloaded with something like `docker logs gitea-mirror --since 5m`. If there's an error (decryption failure, JSON parse error), it'll show up there but the browser wouldn't have surfaced it. With those three we should be able to see exactly which field the UI thinks is empty and why. <!-- github-comment-id:4295470414 -->
adam commented 2026-10-05 01:42:51 +02:00
Author
Owner
Copy link
Copy source
  1. version is 3.15.4 This will be visible from the container log
  2. config.json change tokens and gitea url for security reason
  3. gitea-mirror_logs.txt
1. version is 3.15.4 This will be visible from the container log 2. [config.json](https://github.com/user-attachments/files/26968078/config.json) change tokens and gitea url for security reason 3. [gitea-mirror_logs.txt](https://github.com/user-attachments/files/26968129/gitea-mirror_logs.txt) <!-- github-comment-id:4295952249 -->
adam commented 2026-10-05 01:42:51 +02:00
Author
Owner
Copy link
Copy source

@stephennord looks like ur github username is misisng from the db. can you open the web ui and go to config and I think your username on the github config is empty somehow so please add that if empty and that should help.

@stephennord looks like ur github username is misisng from the db. can you open the web ui and go to config and I think your username on the github config is empty somehow so please add that if empty and that should help. <!-- github-comment-id:4296602682 -->
adam commented 2026-10-05 01:42:52 +02:00
Author
Owner
Copy link
Copy source

filled in the fields UserName at github and gitea configuration, and everything worked in web ui.
It's very strange, but I didn't do any manipulations with the config tab or db, and i check test connection button when there was a problem and it pass test without UserName.

filled in the fields UserName at github and gitea configuration, and everything worked in web ui. It's very strange, but I didn't do any manipulations with the config tab or db, and i check test connection button when there was a problem and it pass test without UserName. <!-- github-comment-id:4302052886 -->
adam commented 2026-10-05 01:42:53 +02:00
Author
Owner
Copy link
Copy source

@stephennord Glad its working again. The username might have got overridden in some migration that failed somehow maybe I am not entirely sure how it might have happened but the user name is not needed for mirroring jobs but for things like differnet. mirror strategy etc. and also used in some places of the web UI.

@stephennord Glad its working again. The username might have got overridden in some migration that failed somehow maybe I am not entirely sure how it might have happened but the user name is not needed for mirroring jobs but for things like differnet. mirror strategy etc. and also used in some places of the web UI. <!-- github-comment-id:4303218834 -->
Sign in to join this conversation.
No Branch/Tag Specified
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
adam (Adam Wojcieszonek)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: adam/gitea-mirror#121