The route evaluator's fetch_candidate_paths triggered a 10-30s full-table
scan of packet_path_hops per route per sweep, flooding the slow-query
log and blocking route saves on the same scan. EXPLAIN analysis (Merge
Join vs forced Nested Loop, across query rewrites, ANALYZE, CTE
MATERIALIZED) confirmed both candidate plans cost ~10-15s at 6.4M rows;
no SQL rewrite or stats change could avoid the scan. The fix applies
four orthogonal data-volume pressures:
- ROUTE_EVALUATOR_INTERVAL_SECONDS default 60->300 (5x less frequent)
- Remove synchronous _reevaluate_route on POST/PUT; route_result and
recent_matches now refresh on the next background sweep instead of
inline (kills save latency)
- Route.window_hours default 48->6, max 12 (bounds the candidate set);
migration clamps pre-existing routes >12 to 12
- Covering index ix_packet_path_hops_raw_packet_id_position INCLUDE
(node_hash, packet_hash, event_hash, received_at, observer_node_id)
so the Merge Join outer scan becomes index-only
- RAW_PACKET_RETENTION_DAYS default 7->2 (hops cascade-delete with raw
packets; smaller table cuts the dominant outer scan)
Migration a59611449e2a clamps window_hours and rebuilds the index
CONCURRENTLY on PostgreSQL (SQLite is a no-op; no INCLUDE support).
DESTRUCTIVE on upgrade: the retention default change purges ~5 days of
raw packets + cascaded hops on the next cleanup run. See
docs/upgrading.md for the full change notes and the new save/sweep
contract.
Adds a checkbox filter to the Routes page that narrows the list to
routes owned by the current user. The filter is URL-driven (?mine=true),
cached server-side automatically via the existing key builder, and only
shown to operators/admins (members can't own routes).
Backend: mine query param on GET /api/v1/routes filters by created_by
matching the caller's X-User-Id. Legacy NULL routes are excluded.
Tests:
- Backend: 6 new TestRouteMineFilter tests (own/other/null/admin/default)
- Vitest: 6 new tests (param passing, role gating, checkbox state)
- E2E: new seed route owned by pw-operator + mine filter spec
A flake8 B007 fix at commit time renamed the whole observer-specs tuple
to (_area, _lat, _lon), but the Node() call below still referenced lat/lon.
Those names then resolved to the leaked values from the preceding
content_specs loop (last iteration = Delta's None/None), so all observer
nodes were seeded without coordinates and dropped from the map (7 -> 3
markers). Underscore only `area` (genuinely unused in that loop); lat/lon
are used in Node(), so they stay as loop vars.
Replaces the httpx-based smoke tests in tests/e2e/ with a browser E2E suite
under e2e/, running against a self-contained throwaway stack that never
touches the local development database.
Stack & data isolation (e2e/docker-compose.test.yml):
- Own ephemeral Postgres 17 (schema via the `migrate` service / Alembic),
distinct project name + named volumes, no host DB port, no \${VAR}
interpolation. `make e2e-down` destroys everything.
- OIDC enabled with a known session secret; WEB_AUTO_REFRESH_SECONDS=2 so
polling is assertable; CONTENT_HOME mounts a test markdown page.
- Deterministic seeder (e2e/seed_data.py) run via global setup inside the
collector container: nodes/observers with `area` tags, adverts, messages
on the public (17) + a custom channel, raw packets + path hops keyed to
node prefixes, a route + health/history, profiles + adoptions, and
event_observers rows so the observer filter/badges resolve.
Auth & tests:
- Forged signed `meshcore-session` cookies (e2e/mint_session.py,
itsdangerous) for admin/member identities -> storageState; no real IdP
needed. 31 specs across 12 files cover global nav/theme, profile
menu+edit, home hero, dashboard widgets, list filters/auto-refresh/row
actions, observer toggles, the path-node overlay, map filters +
show-labels, members, markdown pages, and routes add/edit/delete
(persistence, validation, confirm dialog). workers:1 / fullyParallel:false
against the single shared backend; targeted data-testids added to React
components for stable selectors.
Fixes surfaced while running the suite on fresh Postgres:
- Migration 5e3b712ccf10 aborted on Postgres: its route-health backfill
queries the live Route model (which now has max_path_length) before that
column exists. The swallowed error left the transaction aborted, killing
the subsequent alembic_version stamp. Wrapped the backfill in a SAVEPOINT
so a failure rolls back cleanly without blocking the migration.
- Restored the full ABUSE_* env set required by the MQTT broker, and set
the web service's API_KEY to the admin key (admin writes go through the
proxy as a Bearer token).
31/31 passing; tsc (frontend+e2e), pytest (1460), and pre-commit green.
Louis King