The route evaluator's fetch_candidate_paths triggered a 10-30s full-table
scan of packet_path_hops per route per sweep, flooding the slow-query
log and blocking route saves on the same scan. EXPLAIN analysis (Merge
Join vs forced Nested Loop, across query rewrites, ANALYZE, CTE
MATERIALIZED) confirmed both candidate plans cost ~10-15s at 6.4M rows;
no SQL rewrite or stats change could avoid the scan. The fix applies
four orthogonal data-volume pressures:
- ROUTE_EVALUATOR_INTERVAL_SECONDS default 60->300 (5x less frequent)
- Remove synchronous _reevaluate_route on POST/PUT; route_result and
recent_matches now refresh on the next background sweep instead of
inline (kills save latency)
- Route.window_hours default 48->6, max 12 (bounds the candidate set);
migration clamps pre-existing routes >12 to 12
- Covering index ix_packet_path_hops_raw_packet_id_position INCLUDE
(node_hash, packet_hash, event_hash, received_at, observer_node_id)
so the Merge Join outer scan becomes index-only
- RAW_PACKET_RETENTION_DAYS default 7->2 (hops cascade-delete with raw
packets; smaller table cuts the dominant outer scan)
Migration a59611449e2a clamps window_hours and rebuilds the index
CONCURRENTLY on PostgreSQL (SQLite is a no-op; no INCLUDE support).
DESTRUCTIVE on upgrade: the retention default change purges ~5 days of
raw packets + cascaded hops on the next cleanup run. See
docs/upgrading.md for the full change notes and the new save/sweep
contract.
Admins editing an operator-created route no longer steal ownership.
Ownership transfer now happens only for legacy (NULL created_by) routes.
This ensures operators retain edit access to their routes after an admin
makes a small tweak.
Replace visibility-tier-based write permissions with ownership-based:
operators can only edit/delete routes they created; admins can modify
any route and take ownership on edit. Each route stores the creator's
OIDC user_id (created_by column). Legacy routes with NULL created_by
are admin-only.
The creator's friendly name is resolved from UserProfile and displayed
on the route card with a profile link. Edit/delete buttons are hidden
per-route based on ownership rather than a flat role check.
- Migration: add nullable routes.created_by (batch mode, SQLite-safe)
- _assert_route_modifiable: dual-check visibility (404) + ownership (403)
- create_route: stamps created_by, ensures profile exists
- update_route: admin edits transfer ownership (logged)
- RouteOwner schema mirrors AdoptedByUser pattern
- Batch owner resolution in list endpoint (avoids N+1)
- Frontend: per-route canEdit gate, owner badge with profile link
- Tests: 20+ backend ownership tests, 5 frontend gating tests, e2e assertions
- Docs: routes.md + auth.md updated for ownership model
Operators can now create, edit, and delete routes (previously admin-only).
A user may never scope a route above their own role tier: an operator
creating/editing an admin-visibility route is rejected (403 on the
visibility value, 404 on touching an existing higher-visibility route),
preventing them from creating routes they could then never see or modify.
- routes.py: RequireAdmin -> RequireOperatorOrAdmin on create/update/delete;
add visibility-cap enforcement helpers reusing the existing
resolve_user_role / VISIBILITY_LEVELS ladder
- web/app.py: proxy access map admits operator for routes POST/PUT/DELETE
- Routes.tsx: canManage gate (admin||operator) on Add/Edit/Delete; visibility
<select> filters options by caller tier so operators never see 'admin'
- tests: operator-tier coverage (create/update/delete at/below/above level),
proxy access-map assertion, vitest role-gating + filtered select
- e2e: mint operator session + routes-operator spec
- docs: routes.md + auth.md operator/visibility-cap notes
Adds a new nullable per-route knob that caps the total number of hops
in a candidate packet's path. Packets whose path exceeds the cap are
dropped from matching consideration entirely (before the subsequence
matcher runs), so over-long paths never count toward
packet_count_threshold. Complements the existing max_hop_span, which
only constrains the gap between the first and last matched configured
node.
- Route model + migration (additive, nullable, default null = unlimited)
- Threaded through matcher chain (_subsequence_indices early-return)
- All 5 evaluate/preview/recent_matches call sites updated
- API serializer/create/update/preview passthrough
- CLI seed YAML import (update + create paths)
- Frontend: distinct icons for span (<-o->) vs path-length (|<->|),
always-rendered badges with infinity fallback, hover tooltips on
every stats row item, i18n keys (en + nl)
- Tests: matcher unit tests (within/exceeds cap), API round-trip,
CLI seed import
Change the new-route defaults so the create modal, REST API, YAML
import, and preview helper pre-fill packet_count_threshold=5 instead
of 3, and bump the effective-clear auto-multiplier from 2x to 3x so
the default comfort bar tracks to 15 for a default threshold of 5.
Schema/model/cli/preview:
- schemas/routes.py: RouteCreate + RoutePreviewRequest
packet_count_threshold default 3 -> 5 (clear_threshold stays None)
- models/route.py: Route INSERT default 3 -> 5 (clear_threshold stays
nullable, no default)
- collector/cli.py: YAML import fallbacks 3 -> 5
- collector/routes.py: preview helper fallback 3 -> 5; module constant
CLEAR_DEFAULT_MULTIPLIER 2 -> 3 (drives effective_clear_threshold)
- api/metrics.py: route_clear gauge help text 2x -> 3x
UI:
- web/spa/pages/routes.js: new-route modal pre-fills
packet_count_threshold=5; placeholder shows 3x threshold; parseInt
fallback 3 -> 5. clear_threshold field unchanged (clearing still
sends null for the auto-tracking behaviour).
Docs:
- docs/routes.md defaults table: window_hours 24->48, threshold 3->5,
clear (2x)->(3x), max_hop_span (unlimited)->8. Previous four
entries now match the shipped defaults (some were stale from #316).
- docs/seeding.md YAML example: window_hours 24->48, threshold 3->5,
commented clear_threshold example 10->15 with 3x note.
No migration: packet_count_threshold uses Python-side default= (no
server_default), so existing rows keep their stored values. Existing
routes with clear_threshold=NULL continue to track via the new 3x
multiplier at evaluation time.
Bundles four independent improvements to API caching and route evaluator
accuracy that all touch the same files in the cache/routes stack.
## HTTP Cache-Control headers (api_cache_control_enabled)
The API now emits HTTP Cache-Control on every /api/v1/* response and
handles ETag / If-None-Match -> 304 Not Modified on @cached endpoints.
- @cached GETs: private, max-age=<redis_ttl> + strong SHA-256 ETag.
Matching If-None-Match returns 304 with empty body.
- Uncached GETs under /api/v1/*: private, max-age=0, must-revalidate.
- POST/PUT/DELETE/PATCH and /health*: no-store.
- All private (several @cached endpoints are role-aware).
- @cached decorator stores new envelope {body, etag} in Redis; legacy
bare-JSON entries still read (hashed on the fly) and auto-migrate on
the next miss.
- X-Cache: HIT|MISS observability header is always emitted.
- Kill switch: API_CACHE_CONTROL_ENABLED (default true).
## Dashboard cache TTL bump (30s -> 300s)
REDIS_CACHE_TTL_DASHBOARD default raised from 30s to 300s. Covers all
/dashboard/* endpoints and /api/v1/routes/{id}/history. These return
trend/aggregation data where minute-level staleness is invisible but
every MISS costs seconds of SQL/aggregation. Also resolves reported
short-TTL behaviour on the per-route healthchart (which shares this
setting). Operators with explicit env var: unchanged. Old Redis entries
expire naturally within <=30s; no flush needed.
## Route recent-packets hop truncation
In the routes page expanded card, path hops are now truncated to
2 + ellipsis + 2 when length > 5, matching the packet-detail page
styling. Reuses the existing packets.hops_hidden i18n key as the
tooltip on the ellipsis badge.
## Route event-hash deduplication
A single advert/message/telemetry/trace retransmitted or flooded
through the mesh produces one RawPacket per on-air copy, each with a
fresh wire packet_hash. Counting those as distinct matches let a single
underlying event satisfy packet_count_threshold within seconds and
biased route health.
- Add nullable event_hash column to raw_packets and packet_path_hops
(alembic 6b3430fd84f4). No backfill; legacy rows keep NULL and the
evaluator falls back to wire packet_hash until they age out of
window_hours.
- Subscriber denormalizes the structured event's event_hash onto the
captured raw packet after handler dispatch.
- Route evaluator prefers event_hash when set, collapsing all
receptions of the same underlying event into one match.
Louis King