Files
meshcore-hub/docs/routes.md
Louis King c9b247c553 perf(routes): cut evaluator scan cost via window/retention/index/sweep
The route evaluator's fetch_candidate_paths triggered a 10-30s full-table
scan of packet_path_hops per route per sweep, flooding the slow-query
log and blocking route saves on the same scan. EXPLAIN analysis (Merge
Join vs forced Nested Loop, across query rewrites, ANALYZE, CTE
MATERIALIZED) confirmed both candidate plans cost ~10-15s at 6.4M rows;
no SQL rewrite or stats change could avoid the scan. The fix applies
four orthogonal data-volume pressures:

- ROUTE_EVALUATOR_INTERVAL_SECONDS default 60->300 (5x less frequent)
- Remove synchronous _reevaluate_route on POST/PUT; route_result and
  recent_matches now refresh on the next background sweep instead of
  inline (kills save latency)
- Route.window_hours default 48->6, max 12 (bounds the candidate set);
  migration clamps pre-existing routes >12 to 12
- Covering index ix_packet_path_hops_raw_packet_id_position INCLUDE
  (node_hash, packet_hash, event_hash, received_at, observer_node_id)
  so the Merge Join outer scan becomes index-only
- RAW_PACKET_RETENTION_DAYS default 7->2 (hops cascade-delete with raw
  packets; smaller table cuts the dominant outer scan)

Migration a59611449e2a clamps window_hours and rebuilds the index
CONCURRENTLY on PostgreSQL (SQLite is a no-op; no INCLUDE support).

DESTRUCTIVE on upgrade: the retention default change purges ~5 days of
raw packets + cascaded hops on the next cleanup run. See
docs/upgrading.md for the full change notes and the new save/sweep
contract.
2026-07-25 13:15:51 +01:00

63 lines
8.4 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Route Health Monitoring
The **Routes** page (`/routes`) lets operators define monitored multi-hop mesh routes and track each one's health over time. A background evaluator on the collector matches captured packet paths against each route's configured node sequence within a rolling time window and rolls the result up into a traffic-light status card, with a per-day history strip and a list of recent matching transmissions.
For the environment-variable reference, see [configuration.md → Feature Flags](configuration.md#feature-flags) (`FEATURE_ROUTES`) and [→ Collector](configuration.md#collector) (`ROUTE_EVALUATOR_INTERVAL_SECONDS`). To define routes from YAML, see [seeding.md → Routes](seeding.md#routes).
> **Prerequisite:** route matching reads from the `packet_path_hops` table, which is populated by **Raw Packet Capture**. Keep `FEATURE_PACKETS=true` (the default) so packet paths continue to be captured; with capture off, route cards stay at `unknown` once the existing window of hops ages out.
## How health is evaluated
A route is an ordered list of two or more nodes (the configured path). For each captured packet reception, the evaluator walks the reception's path-hash sequence and checks whether the route's nodes appear **in order, as a subsequence** (intermediate hops are allowed). When `reversible` is set (the default), the reverse-ordered path is also accepted, so a packet travelling `B → ... → A` counts toward an `A → B` route.
Matches are deduplicated by their **underlying event identity**, not per on-air transmission. The collector denormalizes the structured event's `event_hash` (the same key used to dedup advertisements, messages, telemetry, and traces at the structured-event layer) onto each captured raw packet at ingest time. The evaluator prefers `event_hash` when set, so retransmissions or floods of the same underlying advert/message count once toward `packet_count_threshold` instead of once per on-air copy. Packets captured before this column existed (and any unclassified wire packets) have `event_hash IS NULL` and fall back to the wire `packet_hash`, preserving the previous behaviour until they age out of the configured `window_hours`.
Each route carries these knobs:
| Field | Default | Description |
| --- | --- | --- |
| `match_width` | `1` | Path-hash prefix width in bytes (1/2/3). Higher widths disambiguate nodes that share a short public-key prefix. |
| `window_hours` | `6` | Rolling lookback window for the live status card (1–12 hours). |
| `packet_count_threshold` | `5` | Distinct matching packets at/above which the route is `healthy`. "Distinct" is per underlying event, not per transmission — see [How health is evaluated](#how-health-is-evaluated) above. |
| `clear_threshold` | _(3× threshold)_ | Comfort bar for the `clear`/`marginal` split. Omit/null to use three times the threshold. |
| `max_hop_span` | `8` | Caps the position gap between the first and last matched node, to reject matches that wander too far. |
| `max_path_length` | _(unlimited)_ | Caps the total number of hops in a candidate packet's full path; receptions whose path exceeds this are dropped from matching entirely (never counted toward `packet_count_threshold`). Useful to ignore wandering packets that happen to include the configured endpoints but traversed a long detour. |
| `reversible` | `true` | Also match the path in reverse direction. |
| `enabled` | `true` | When `false`, the route is skipped by the evaluator and reports `unknown`/`no_coverage`. |
The result is reported on two axes:
- **State** — `healthy` (≥ threshold distinct matches in the window), `unhealthy` (some packets were observed but too few matched the configured path), or `no_coverage` (no in-scope packets at all in the window).
- **Quality** — `clear` (≥ effective clear bar), `marginal` (healthy but below the comfort bar), `failing` (unhealthy), or `unknown` (no coverage / route disabled).
The state/quality split lets the dashboard render a single traffic-light band while keeping the underlying reason visible: a `marginal` route is technically up but losing margin, and a `failing` route has traffic in the window but the configured path isn't completing.
### Observer scoping
By default every observer's receptions contribute to every route. A route may instead scope itself to an explicit **observer allow-list** (`route_observers`); when set, only receptions from those observer nodes are matched. Use this to ignore noisy or off-path observers that would otherwise drown out the signal. Observer nodes that don't yet exist in the database are skipped with a warning rather than failing the seed.
### Background evaluator
The collector runs a background thread that re-evaluates every enabled route on a fixed cadence and upserts the result into `route_results` (one row per route, holding the latest `state`, `quality`, `matched_count`, and the threshold snapshots used). The cadence is controlled by `ROUTE_EVALUATOR_INTERVAL_SECONDS` (default `300`); set it to `0` to disable the evaluator, in which case route cards remain at `unknown` until it is re-enabled. The per-route history strip and the `/api/v1/routes/{id}/history` endpoint evaluate on demand over the raw `packet_path_hops` table and are not dependent on the background evaluator.
Routes are **not** re-evaluated synchronously on create/update — the write handler returns immediately with the route's direct fields and whatever `route_result` snapshot the last sweep wrote (or `null` for a brand-new route). The next sweep (within `ROUTE_EVALUATOR_INTERVAL_SECONDS`) refreshes `state`, `quality`, `matched_count`, `quality_avg`, and the recent-matches card. Keep this in mind when changing `window_hours` / thresholds: the card reflects the new config on the next sweep, not on save.
The route evaluation window (`window_hours`) defaults to **6 hours** and is capped at **12 hours** (`ge=1, le=12`). A tight window bounds the candidate set fed to the route evaluator's `packet_path_hops` scan; raise it only if you accept the additional scan cost, and never above the configured `RAW_PACKET_RETENTION_DAYS` (the window would otherwise extend past purged data and falsely report `no_coverage`).
## Visibility
Routes carry the same role-based visibility levels as channels — `community`, `member`, `operator`, `admin`. A user only sees routes whose visibility is at or below their role's maximum level. Seeded routes default to `community` (visible to everyone); set a higher level to restrict a route to operators/admins only. Visibility is enforced on both the list and detail endpoints, so a hidden route's existence is not leaked.
Both operators and admins can create routes. A user may never scope a route above their own role (e.g. an operator cannot create an `admin`-visibility route) — this is enforced on the write endpoints and prevents a user from creating a route they could then never see or modify.
**Ownership-based editing:** Each route records the user who created it (`created_by`). Operators can edit and delete only the routes they created. Admins can edit and delete any route; they claim ownership of legacy (unowned) routes — those with a `NULL` `created_by` — when they edit them, but do not displace an existing creator. Routes created before ownership tracking was introduced have a `NULL` `created_by` and are admin-only. The creator's display name is shown on each route card when available. Attempting to modify a route above the caller's visibility tier returns `404`; modifying a visible route the caller does not own returns `403`.
## Defining routes
Routes are keyed by their `from`/`to` endpoint labels and upserted by that pair. There are two ways to create them:
- **Seed YAML** — add a `routes.yaml` to your `SEED_HOME` and run the seed process. See [seeding.md → Routes](seeding.md#routes) for the format and rules (path nodes must already exist in the database; the `(from, to)` pair must be unique).
- **API** — `POST /api/v1/routes` (operator or admin) creates a route, with a `/preview` endpoint that dry-runs matching against an unsaved configuration so you can tune thresholds before committing. See `SCHEMAS.md` for the request/response shapes.
The `/routes` page renders the live status card, the per-day history strip, recent matching transmissions (with observer attribution), the route owner's name, and inline edit/delete controls gated per-route by ownership (operators see controls only on their own routes; admins see them on all routes).