Files
Louis King c9b247c553 perf(routes): cut evaluator scan cost via window/retention/index/sweep
The route evaluator's fetch_candidate_paths triggered a 10-30s full-table
scan of packet_path_hops per route per sweep, flooding the slow-query
log and blocking route saves on the same scan. EXPLAIN analysis (Merge
Join vs forced Nested Loop, across query rewrites, ANALYZE, CTE
MATERIALIZED) confirmed both candidate plans cost ~10-15s at 6.4M rows;
no SQL rewrite or stats change could avoid the scan. The fix applies
four orthogonal data-volume pressures:

- ROUTE_EVALUATOR_INTERVAL_SECONDS default 60->300 (5x less frequent)
- Remove synchronous _reevaluate_route on POST/PUT; route_result and
  recent_matches now refresh on the next background sweep instead of
  inline (kills save latency)
- Route.window_hours default 48->6, max 12 (bounds the candidate set);
  migration clamps pre-existing routes >12 to 12
- Covering index ix_packet_path_hops_raw_packet_id_position INCLUDE
  (node_hash, packet_hash, event_hash, received_at, observer_node_id)
  so the Merge Join outer scan becomes index-only
- RAW_PACKET_RETENTION_DAYS default 7->2 (hops cascade-delete with raw
  packets; smaller table cuts the dominant outer scan)

Migration a59611449e2a clamps window_hours and rebuilds the index
CONCURRENTLY on PostgreSQL (SQLite is a no-op; no INCLUDE support).

DESTRUCTIVE on upgrade: the retention default change purges ~5 days of
raw packets + cascaded hops on the next cleanup run. See
docs/upgrading.md for the full change notes and the new save/sweep
contract.
2026-07-25 13:15:51 +01:00

8.4 KiB
Raw Permalink Blame History

Route Health Monitoring

The Routes page (/routes) lets operators define monitored multi-hop mesh routes and track each one's health over time. A background evaluator on the collector matches captured packet paths against each route's configured node sequence within a rolling time window and rolls the result up into a traffic-light status card, with a per-day history strip and a list of recent matching transmissions.

For the environment-variable reference, see configuration.md → Feature Flags (FEATURE_ROUTES) and → Collector (ROUTE_EVALUATOR_INTERVAL_SECONDS). To define routes from YAML, see seeding.md → Routes.

Prerequisite: route matching reads from the packet_path_hops table, which is populated by Raw Packet Capture. Keep FEATURE_PACKETS=true (the default) so packet paths continue to be captured; with capture off, route cards stay at unknown once the existing window of hops ages out.

How health is evaluated

A route is an ordered list of two or more nodes (the configured path). For each captured packet reception, the evaluator walks the reception's path-hash sequence and checks whether the route's nodes appear in order, as a subsequence (intermediate hops are allowed). When reversible is set (the default), the reverse-ordered path is also accepted, so a packet travelling B → ... → A counts toward an A → B route.

Matches are deduplicated by their underlying event identity, not per on-air transmission. The collector denormalizes the structured event's event_hash (the same key used to dedup advertisements, messages, telemetry, and traces at the structured-event layer) onto each captured raw packet at ingest time. The evaluator prefers event_hash when set, so retransmissions or floods of the same underlying advert/message count once toward packet_count_threshold instead of once per on-air copy. Packets captured before this column existed (and any unclassified wire packets) have event_hash IS NULL and fall back to the wire packet_hash, preserving the previous behaviour until they age out of the configured window_hours.

Each route carries these knobs:

Field Default Description
match_width 1 Path-hash prefix width in bytes (1/2/3). Higher widths disambiguate nodes that share a short public-key prefix.
window_hours 6 Rolling lookback window for the live status card (112 hours).
packet_count_threshold 5 Distinct matching packets at/above which the route is healthy. "Distinct" is per underlying event, not per transmission — see How health is evaluated above.
clear_threshold (3× threshold) Comfort bar for the clear/marginal split. Omit/null to use three times the threshold.
max_hop_span 8 Caps the position gap between the first and last matched node, to reject matches that wander too far.
max_path_length (unlimited) Caps the total number of hops in a candidate packet's full path; receptions whose path exceeds this are dropped from matching entirely (never counted toward packet_count_threshold). Useful to ignore wandering packets that happen to include the configured endpoints but traversed a long detour.
reversible true Also match the path in reverse direction.
enabled true When false, the route is skipped by the evaluator and reports unknown/no_coverage.

The result is reported on two axes:

  • Statehealthy (≥ threshold distinct matches in the window), unhealthy (some packets were observed but too few matched the configured path), or no_coverage (no in-scope packets at all in the window).
  • Qualityclear (≥ effective clear bar), marginal (healthy but below the comfort bar), failing (unhealthy), or unknown (no coverage / route disabled).

The state/quality split lets the dashboard render a single traffic-light band while keeping the underlying reason visible: a marginal route is technically up but losing margin, and a failing route has traffic in the window but the configured path isn't completing.

Observer scoping

By default every observer's receptions contribute to every route. A route may instead scope itself to an explicit observer allow-list (route_observers); when set, only receptions from those observer nodes are matched. Use this to ignore noisy or off-path observers that would otherwise drown out the signal. Observer nodes that don't yet exist in the database are skipped with a warning rather than failing the seed.

Background evaluator

The collector runs a background thread that re-evaluates every enabled route on a fixed cadence and upserts the result into route_results (one row per route, holding the latest state, quality, matched_count, and the threshold snapshots used). The cadence is controlled by ROUTE_EVALUATOR_INTERVAL_SECONDS (default 300); set it to 0 to disable the evaluator, in which case route cards remain at unknown until it is re-enabled. The per-route history strip and the /api/v1/routes/{id}/history endpoint evaluate on demand over the raw packet_path_hops table and are not dependent on the background evaluator.

Routes are not re-evaluated synchronously on create/update — the write handler returns immediately with the route's direct fields and whatever route_result snapshot the last sweep wrote (or null for a brand-new route). The next sweep (within ROUTE_EVALUATOR_INTERVAL_SECONDS) refreshes state, quality, matched_count, quality_avg, and the recent-matches card. Keep this in mind when changing window_hours / thresholds: the card reflects the new config on the next sweep, not on save.

The route evaluation window (window_hours) defaults to 6 hours and is capped at 12 hours (ge=1, le=12). A tight window bounds the candidate set fed to the route evaluator's packet_path_hops scan; raise it only if you accept the additional scan cost, and never above the configured RAW_PACKET_RETENTION_DAYS (the window would otherwise extend past purged data and falsely report no_coverage).

Visibility

Routes carry the same role-based visibility levels as channels — community, member, operator, admin. A user only sees routes whose visibility is at or below their role's maximum level. Seeded routes default to community (visible to everyone); set a higher level to restrict a route to operators/admins only. Visibility is enforced on both the list and detail endpoints, so a hidden route's existence is not leaked.

Both operators and admins can create routes. A user may never scope a route above their own role (e.g. an operator cannot create an admin-visibility route) — this is enforced on the write endpoints and prevents a user from creating a route they could then never see or modify.

Ownership-based editing: Each route records the user who created it (created_by). Operators can edit and delete only the routes they created. Admins can edit and delete any route; they claim ownership of legacy (unowned) routes — those with a NULL created_by — when they edit them, but do not displace an existing creator. Routes created before ownership tracking was introduced have a NULL created_by and are admin-only. The creator's display name is shown on each route card when available. Attempting to modify a route above the caller's visibility tier returns 404; modifying a visible route the caller does not own returns 403.

Defining routes

Routes are keyed by their from/to endpoint labels and upserted by that pair. There are two ways to create them:

  • Seed YAML — add a routes.yaml to your SEED_HOME and run the seed process. See seeding.md → Routes for the format and rules (path nodes must already exist in the database; the (from, to) pair must be unique).
  • APIPOST /api/v1/routes (operator or admin) creates a route, with a /preview endpoint that dry-runs matching against an unsaved configuration so you can tune thresholds before committing. See SCHEMAS.md for the request/response shapes.

The /routes page renders the live status card, the per-day history strip, recent matching transmissions (with observer attribution), the route owner's name, and inline edit/delete controls gated per-route by ownership (operators see controls only on their own routes; admins see them on all routes).