- Add `skipPersonalRepos: z.boolean().default(false)` to githubConfigSchema
- Filter out user-owned repos in getGithubRepositories when flag is true
- Wire ONLY_MIRROR_ORGS env var to skipPersonalRepos in env-config-loader
- Add checkbox UI in GitHubMirrorSettings Filtering & Behavior section
- Round-trip skipPersonalRepos through config-mapper (UI ↔ DB)
- Add skipPersonalRepos to AdvancedOptions TypeScript type
- Mark include/exclude arrays in configSchema as unused/reserved
- Update ENVIRONMENT_VARIABLES.md to document ONLY_MIRROR_ORGS effect
Affiliation was set to "owner,collaborator", omitting repos owned
by orgs the user belongs to. As a result:
- main sync, scheduler, and cleanup never saw org repos
- orgs appeared empty unless manually re-added via /api/sync/organization
- restart archived previously-mirrored org repos as orphans
GitHub's API default is owner,collaborator,organization_member;
restoring it fixes both symptoms with no other code changes.
GitHub's listForAuthenticatedUser defaults to returning every repo the
user has access to (owner + collaborator + organization_member), which
imports a lot of noise for users who only want their own repos.
Adds an `includeCollaboratorRepos` toggle, defaulting to true to preserve
existing behavior. When disabled, the affiliation filter scopes the API
call to "owner" only.
The cleanup service overrides the filter to always include collaborator
repos when computing the "what's still on GitHub" list. Without this,
toggling the option off would mark previously-mirrored collab repos as
orphaned and archive/delete them from Gitea.
Wired through the schema, both UI<->DB mappers, the env-config loader
(with new INCLUDE_COLLABORATOR_REPOS env var), and the settings UI.
* feat: add E2E testing infrastructure with fake GitHub, Playwright, and CI workflow
- Add fake GitHub API server (tests/e2e/fake-github-server.ts) with
management API for seeding test data
- Add Playwright E2E test suite covering full mirror workflow:
service health checks, user registration, config, sync, verify
- Add Docker Compose for E2E Gitea instance
- Add orchestrator script (run-e2e.sh) with cleanup
- Add GitHub Actions workflow (e2e-tests.yml) with Gitea service container
- Make GITHUB_API_URL configurable via env var for testing
- Add npm scripts: test:e2e, test:e2e:ci, test:e2e:keep, test:e2e:cleanup
* feat: add real git repos + backup config testing to E2E suite
- Create programmatic test git repos (create-test-repos.ts) with real
commits, branches (main, develop, feature/*), and tags (v1.0.0, v1.1.0)
- Add git-server container to docker-compose serving bare repos via
dumb HTTP protocol so Gitea can actually clone them
- Update fake GitHub server to emit reachable clone_url fields pointing
to the git-server container (configurable via GIT_SERVER_URL env var)
- Add management endpoint POST /___mgmt/set-clone-url for runtime config
- Update E2E spec with real mirroring verification:
* Verify repos appear in Gitea with actual content
* Check branches, tags, commits, file content
* Verify 4/4 repos mirrored successfully
- Add backup configuration test suite:
* Enable/disable backupBeforeSync config
* Toggle blockSyncOnBackupFailure
* Trigger re-sync with backup enabled and verify activities
* Verify config persistence across changes
- Update CI workflow to use docker compose (not service containers)
matching the local run-e2e.sh approach
- Update cleanup.sh for git-repos directory and git-server port
- All 22 tests passing with real git content verification
* refactor: split E2E tests into focused files + add force-push tests
Split the monolithic e2e.spec.ts (1335 lines) into 5 focused spec files
and a shared helpers module:
helpers.ts — constants, GiteaAPI, auth, saveConfig, utilities
01-health.spec.ts — service health checks (4 tests)
02-mirror-workflow.spec.ts — full first-mirror journey (8 tests)
03-backup.spec.ts — backup config toggling (6 tests)
04-force-push.spec.ts — force-push simulation & backup verification (9 tests)
05-sync-verification.spec.ts — dynamic repos, content integrity, reset (5 tests)
The force-push tests are the critical addition:
F0: Record original state (commit SHAs, file content)
F1: Rewrite source repo history (simulate force-push)
F2: Sync to Gitea WITHOUT backup
F3: Verify data loss — LICENSE file gone, README overwritten
F4: Restore source, re-mirror to clean state
F5: Enable backup, force-push again, sync through app
F6: Verify Gitea reflects the force-push
F7: Verify backup system was invoked (snapshot activities logged)
F8: Restore source repo for subsequent tests
Also added to helpers.ts:
- GiteaAPI.getBranch(), .getCommit(), .triggerMirrorSync()
- getRepositoryIds(), triggerMirrorJobs(), triggerSyncRepo()
All 32 tests passing.
* Try to fix actions
* Try to fix the other action
* Add debug info to check why e2e action is failing
* More debug info
* Even more debug info
* E2E fix attempt #1
* E2E fix attempt #2
* more debug again
* E2E fix attempt #3
* E2E fix attempt #4
* Remove a bunch of debug info
* Hopefully fix backup bug
* Force backups to succeed
- Implemented comprehensive GitHub API rate limit handling:
- Integrated @octokit/plugin-throttling for automatic retry with exponential backoff
- Added RateLimitManager service to track and enforce rate limits
- Store rate limit status in database for persistence across restarts
- Automatic pause and resume when limits are exceeded
- Proper user identification for 5000 req/hr authenticated limit (vs 60 unauthenticated)
- Improved rate limit UI/UX:
- Removed intrusive rate limit card from dashboard
- Toast notifications only at critical thresholds (80% and 100% usage)
- All rate limit events logged for debugging
- Optimized for GitHub's API constraints:
- Reduced default batch size from 10 to 5 repositories
- Added documentation about GitHub's 100 concurrent request limit
- Better handling of repositories with many issues/PRs
- Add missing database fields (language, description, mirroredLocation, destinationOrg) to repository operations
- Add missing organization fields (publicRepositoryCount, privateRepositoryCount, forkRepositoryCount) to schema
- Update GitRepo interface to include all required database fields
- Fix GitHub data fetching functions to map all fields correctly
- Update all sync endpoints (main, repository, organization, scheduler) to handle new fields
This fixes the "SQLite query expected X values, received Y" error when importing
large numbers (4.6k+) of starred repositories by ensuring all database fields
are properly mapped from GitHub API responses through to database insertion.
Brendan Davidson