docs: review link plan, add UI design, perf decisions, packet-detail swap

Review pass: fix stale citations (event_observer upsert path/method,
raw_packet hop-insert point, web path prefixes, VISIBILITY_LEVELS
location, features dict range); add Review section.

UI Design: status-board layout, four-state health badge
(healthy/unhealthy/no_coverage/disabled), inline accordion expand,
segmented match_width control, lazy GET /{id} detail, mesh_links.*
i18n namespace, POST /links/preview endpoint.

Performance: fetch-and-check engine over N-way self-join,
(node_hash, received_at) + (raw_packet_id, position) indexes,
preview candidate cap + debounce, evaluator threshold short-circuit.

Phase 9: consolidate packet-detail path read onto packet_path_hops;
delete duplicate _extract_path_hashes. Enabled Future Capabilities note
(path-node filtering, relay stats).
This commit is contained in:
Louis King
2026-07-07 00:07:46 +01:00
parent ae832fe73f
commit d60e2e8b1d
@@ -148,12 +148,14 @@ first bytes co-occur far apart on an unrelated long flood path.
- **F1 — Link configuration.** An operator with the `admin` role can create,
update, and delete Links. Each Link has: a unique name, optional description,
a `visibility` (community/member/operator/admin, default `community`), a `match_width` (1/2/3,
default 1), `window_hours`, `packet_count_threshold`, `max_hop_span`
(nullable int, default `null` = unlimited), an `enabled` flag, an ordered
default 1), `window_hours` (default 24, range 1..720),
`packet_count_threshold` (default 3, range 1..10000), `max_hop_span`
(nullable int, default `null` = unlimited), an `enabled` flag (default true), an ordered
list of **≥2** path node specs, and an optional observer scope.
- **F2 — Path node specs.** Each path entry selects a known Node (from
`nodes`); the system derives `expected_hash = public_key[:2*match_width]` at
save time. Entries are ordered; the subsequence match preserves that order
save time. Entries are ordered; entries must be **distinct** (the same node
twice in one link is invalid). The subsequence match preserves that order
with gaps allowed (intermediary nodes may sit between configured entries).
`match_width` is **per-link**: an operator who knows the traffic in a given
area is uniformly 2- or 3-byte can widen that link's width to drop from 256
@@ -176,15 +178,38 @@ first bytes co-occur far apart on an unrelated long flood path.
**distinct packets** (`packet_hash`) whose path, in at least one observer's
reception (within the observer scope, if set), contains the configured
ordered subsequence within the window and within `max_hop_span` (if set), is
greater than or equal to `packet_count_threshold`.
greater than or equal to `packet_count_threshold`. Each evaluation writes a
`link_result` row whose `state` is one of:
- **`healthy`** — `matched_count >= packet_count_threshold`.
- **`unhealthy`** — in-scope observers received packets in the window but
`matched_count < threshold` (route may be down).
- **`no_coverage`** — `matched_count == 0` **and** no in-scope observer
received any packet with a non-empty path in the window (cannot
distinguish route-down from no-listener; the operator action is to
add/widen observers, not assume the route failed). When the scope is "all
observers", `no_coverage` is only reachable when the whole mesh is silent.
The evaluator separates the latter two with one extra existence check (any
in-scope `packet_path_hops` row in the window). Disabled links are excluded
from evaluation entirely — they produce no `link_result`, are omitted from
Prometheus output, and render a gray **disabled** badge (that badge comes
from `link.enabled`, not a result state).
- **F5 — Visibility scoping.** The link list is filtered by the requesting
user's role exactly like channels (`VISIBILITY_LEVELS`). Reads are role-
scoped; writes are admin-only.
- **F6 — UI.** A dedicated `/links` page lists links with health badges,
grouped by visibility, with admin CRUD modals (mirror of `channels.js`). The
node picker shows prefix-collision badges and a live "matches in 24h"
preview; it warns on mixed-width intent and recommends 3-node paths.
- **F7 — Prometheus.** `/metrics` emits `meshcore_link_healthy{link}`,
user's role exactly like channels, using `VISIBILITY_LEVELS` from
`api/channel_visibility.py`. Reads are role-scoped; writes are admin-only.
- **F6 — UI.** A dedicated `/links` page (see **UI Design** below) behaves as
a status board: a health summary strip, cards grouped by visibility with
unhealthy/no_coverage sorted first, a four-state badge (healthy /
unhealthy / no_coverage / disabled), and an inline accordion expand
revealing the diagnosis, contributing observers, the latest matched path, a
config recap, and a deep-link to the packets view. Admin CRUD uses a wider
modal containing a node path-builder and observer picker with prefix-
collision badges, a live "matches in 24h" preview, and a segmented
`match_width` control. Mirrors `channels.js` structure throughout.
- **F7 — Prometheus.** `/metrics` emits `meshcore_link_healthy{link}` (1 if
`state == healthy` else 0), `meshcore_link_state{link}` (0=healthy,
1=unhealthy, 2=no_coverage — so the amber case is independently alertable:
`state==1` = route may be down, `state==2` = add observers),
`meshcore_link_matched_packets{link}`, and `meshcore_link_threshold{link}`
for **all** enabled links (no visibility filtering on the monitoring feed).
- **F8 — Seeding (no-auth provisioning).** Site operators can load Links from
@@ -232,18 +257,29 @@ first bytes co-occur far apart on an unrelated long flood path.
modeled on the spam re-scoring sweep (`subscriber.py:545-597`), runs at a
configurable interval (default 60s, `0` disables), performs an immediate
first run on startup, and upserts one row per link into `link_results` using
the existing dialect-specific `on_conflict_do_update` pattern
(`event_observer.py:143-158`).
- **T5 — Indexing.** `packet_path_hops` carries `INDEX (node_hash,
raw_packet_id, position)` (drives the chain), `INDEX (raw_packet_id)` (FK +
per-reception lookup + cascade), and the denormalized `packet_hash` for the
dedup count.
a dialect-specific `on_conflict_do_update` (postgresql
`pg_insert(...).on_conflict_do_update(...)` / sqlite
`sqlite_insert(...).on_conflict_do_update(...)`), modeled on the existing
dialect branch in `common/models/event_observer.py:143-158`. That branch
currently uses `on_conflict_do_nothing`; no `do_update` variant exists in
the codebase yet, so the evaluator authors it (standard SQLAlchemy idiom).
- **T5 — Indexing.** Every hop query is time-windowed, so `received_at`
belongs in the leading index, not as a post-filter. `packet_path_hops`
carries `INDEX (node_hash, received_at)` — drives the first-prefix + window
range scan in `fetch_candidate_paths` (a two-range seek: prefix then
recent) — and `INDEX (raw_packet_id, position)` — serves the per-reception
ordered-hop fetch, the FK lookup, and the `ON DELETE CASCADE` (leftmost-
prefix covers equality-on-`raw_packet_id`, so no separate FK index). The
denormalized `packet_hash`/`received_at` columns back the distinct count
and the window without a join back to `raw_packets`.
- **T6 — Backend-agnostic.** All DDL via Alembic **batch mode** (SQLite-safe);
queries use SQLAlchemy Core/ORM with a Python-computed `window_since`
datetime (never `NOW() - INTERVAL`).
- **T7 — Retention.** `packet_path_hops.raw_packet_id` uses `ON DELETE
CASCADE`, so the existing cleanup in `cleanup.py` removes hop rows for free
when aged `raw_packets` are deleted; no cleanup change required.
`link_results` and `link_nodes`/`link_observers` cascade-delete with their
parent `links` row (standard FK cascade).
- **T8 — Feature gating.** New `feature_links=True` UI flag and
`link_evaluator_interval_seconds=60` collector knob in `common/config.py`,
surfaced in `.env.example`. Hop extraction only runs when raw packet capture
@@ -262,6 +298,79 @@ first bytes co-occur far apart on an unrelated long flood path.
explicit value is honored, since the operator has filesystem access and links
carry no secret (unlike channel keys).
## UI Design
Decisions captured during plan review. The build lives in Phase 7; this
section is the single source of truth for the design.
### Mental model: status board, not catalog
Channels is a catalog (CRUD list of static keys). Links is a **status board**
— the page's primary job is glancing at health; configuration is secondary
admin work. Every layout choice below follows from that.
### List page (`/links`)
- **Summary strip** at the top: `● N healthy · ● M unhealthy · ◐ K no
coverage · ◌ D disabled` (live counts from the embedded results).
- **Cards grouped by visibility** (`VISIBILITY_ORDER`, like channels), but
within each group **sorted unhealthy / no_coverage first** so broken routes
surface immediately.
- Header + admin "Add link" button + empty state mirror `channels.js`.
### The link card
- **Health badge** — four states, color-coded: `healthy` (green ●),
`unhealthy` (red ●), `no_coverage` (amber ◐), `disabled` (gray ◌, from
`link.enabled`).
- **Path chips** — the configured nodes as `[Ipswich RP] → … → [Norwich RP]`,
conveying "ordered, gaps allowed".
- **Numbers line** — `matched / threshold · window · evaluated Xm ago`.
- Admin edit/delete buttons (channels pattern).
- **Click → inline accordion expand** (not a modal, not a separate page).
### Card expand contents (lazy `GET /api/v1/links/{id}`)
1. **Diagnosis line** — turns the amber/red split into a sentence.
2. **Contributing observers with counts** — an empty list *is* the
`no_coverage` signal.
3. **Latest match (~3) with the observed path** — configured nodes marked ✓,
intermediates shown, so the operator can verify a real match vs a
collision. Served by a new `recent_matches(link, limit)` engine helper.
4. **Config recap** (read-only) — width, span, window, observer scope.
5. **"View packets" deep-link** — to the existing packet-groups page filtered
to matched hashes + window (reuses built UI; no new packet browser).
### Create/edit modal (wider: `modal-box-lg`)
- `name`, `description`, `visibility` (select), `enabled` (checkbox) — as
channels.
- **`match_width`** — **segmented control** `[ 1 byte | 2 bytes | 3 bytes ]`
with a dynamic hint ("Matches all traffic · ~256 buckets" / "2-byte+ only ·
~65K" / "3-byte only · ~16M"). Chosen over a `<select>` because toggling it
live-updates the collision badges and preview — the explore-by-tapping
affordance is the point.
- **Path builder** (new shared component): search → `/api/v1/nodes`, selected
nodes as ordered draggable chips (↑↓ for a11y), enforce ≥2 + distinct, each
chip showing a **collision badge** ("unique" green / "N share prefix `a1`"
amber). Warn on mixed-width intent (a node never observed at the chosen
width) and suggest adding a 3rd node when collisions appear.
- **Observers** multi-picker (same component; empty = all observers).
- `window_hours`, `packet_count_threshold`, `max_hop_span` (empty =
unlimited) numeric fields.
- **Live "matches in 24h" preview** — debounced `POST /api/v1/links/preview`
as the path / width / observers change; shows `~N matches in 24h` plus the
per-node collision counts used by the chips.
### Data split (list vs detail vs preview)
- `GET /api/v1/links` embeds the **lightweight** result per card: `state`,
`matched_count`, `threshold`, `evaluated_at`. Keeps the list payload small.
- `GET /api/v1/links/{id}` returns the **full detail**: the lightweight result
plus contributing observers (with counts) and the latest ~3 matched paths.
Fetched lazily on first expand and cached in page state.
- `POST /api/v1/links/preview` (unsaved config → `{matched_count,
contributing_observers, collisions}`) powers the live preview + chip badges.
### i18n
- Feature strings under a new top-level **`mesh_links.*`** block (the existing
`links.*` block holds footer labels and must not be reused); nav label under
`entities.links`. See Phase 7.
## Implementation Plan
### Phase 1: Data model + migration + backfill
@@ -269,74 +378,165 @@ first bytes co-occur far apart on an unrelated long flood path.
`link.py` (with `LinkVisibility` mirroring the channel enum, plus config
columns `match_width` and nullable `max_hop_span`), `link_node.py`,
`link_observer.py`, `link_result.py`. Export all from `models/__init__.py`.
`link_result` carries: `link_id` (FK `links.id`, `ondelete=CASCADE`, unique
— one row per link), `state` (enum `healthy` / `unhealthy` /
`no_coverage`), `matched_count` (int), `threshold` (int, snapshot at eval
time for stable reporting), `evaluated_at` (datetime). Per-observer
breakdown and recent matched paths are **not** stored — they are computed
on demand by `GET /api/v1/links/{id}` (see UI Design → Data split).
- One Alembic revision (batch mode) creating the five tables + indexes.
- Backfill `packet_path_hops` from `raw_packets.decoded`, keyset-paginated
(batch 1000), reusing the **frozen dual-path extraction** copied from
migration `20260703_2250` (`_normalize_hash_list` + `decoded.path` →
`payload.decoded.pathHashes` fallback), emitting one row per
`(position, node_hash)` with `packet_hash`/`received_at` denormalized.
`payload.decoded.pathHashes` fallback), which yields a list of `node_hash`
strings ordered origin-to-observer. The backfill enumerates this list
(index = `position`) and emits one `PacketPathHop` row per
`(position, node_hash)` with `packet_hash`/`received_at` denormalized from
the source `raw_packet` row.
### Phase 2: Ingest hook + tests
- In `collector/handlers/raw_packet.py:138` (after `path_hashes` is computed at
lines 106-112), bulk-insert `PacketPathHop` rows from the normalized list.
Zero extra decode; gated by existing raw-capture flag.
- In `collector/handlers/raw_packet.py::store_raw_packet`, the normalized
`path_hashes` list is already computed at lines 106-111 (line 112 derives
`path_hash_bytes` from it — the citable "available" point is 106-111). The
hop bulk-insert goes inside the `with db.session_scope()` block (line 118)
**after** the `RawPacket` is added. The current inline
`session.add(RawPacket(...))` at lines 138-155 must be refactored to
`raw_packet = RawPacket(...); session.add(raw_packet); session.flush()` so
`raw_packet.id` is materialized, then bulk-insert one `PacketPathHop` per
`(position, node_hash)` from `path_hashes`, denormalizing
`packet_hash`/`received_at` from the same values. Zero extra decode; gated
by the existing raw-capture flag (the caller,
`Subscriber._maybe_capture_raw_packet`, already checks
`self._raw_packet_capture_enabled`).
- Extend `tests/test_collector/test_handlers/test_raw_packet.py` to assert
hops are inserted with correct positions/hashes and skipped when the path is
absent.
### Phase 3: Matching engine (pure, DB-tested)
- New `collector/links.py`: `build_subsequence_query(link, since)` (dynamic
chain of indexed self-joins on `raw_packet_id` with strictly-increasing
`position`, each filtering `node_hash LIKE expected_hash || '%'`, optional
observer filter, and optional `max_hop_span` predicate
`(ph_last.position ph0.position) <= max_hop_span` when set), `evaluate_link`,
`evaluate_all_links`, plus helpers `derive_expected_hash`,
`detect_observed_width`, and `prefix_collision_counts`
(`GROUP BY lower(public_key[:2])`) for UI badges.
- New `collector/links.py` using a **fetch-and-check** strategy (not an N-way
self-join). Rationale: the default 1-byte match width produces broad first-
prefix candidate sets where a self-join's cost scales with (candidates ×
depth); fetch-and-check scales with (candidates) only, on per-reception
paths that are ≤8 hops. Core functions:
- `fetch_candidate_paths(db, first_prefix, since, observer_ids=None,
limit=None)`: one statement — `SELECT raw_packet_id, position, node_hash,
packet_hash FROM packet_path_hops WHERE raw_packet_id IN (SELECT
raw_packet_id FROM packet_path_hops WHERE node_hash LIKE :p0 || '%' AND
received_at >= :since [...observer filter]) ORDER BY raw_packet_id,
position` — returns grouped ordered hop arrays. A subquery (not a client
`IN`-list) avoids the `SQLITE_MAX_VARIABLE_NUMBER` ceiling.
- `is_subsequence(path, expected, max_hop_span=None)`: pure two-pointer
prefix match (`node_hash.startswith(expected_hash)`), gaps allowed,
`position(last) position(first) <= max_hop_span` when set. ~8 lines,
unit-trivial.
- `evaluate_link(db, link, since)`: fetch candidates for the link's first
node prefix, run `is_subsequence` per reception, count **distinct**
`packet_hash`, and **short-circuit as soon as the count reaches
`packet_count_threshold`** (the evaluator only needs the threshold
crossing, not an exact count → `healthy` early-exit). Below threshold, run
**one existence check** (any in-scope `packet_path_hops` row in the
window) to choose `unhealthy` vs `no_coverage` per F4. Returns `(state,
matched_count)` (`matched_count` is `>= threshold` when short-circuited).
- `evaluate_all_links`: iterates enabled links, calls `evaluate_link`.
- `recent_matches(db, link, limit=3)`: same fetch + subsequence check,
returns the latest `limit` matching paths (positions/hashes) for the card
expand's ✓-marked path view.
- `preview_link(db, config, since)`: accepts an **unsaved** config (path
nodes by `node_id`, width, observers, span) and returns `{matched_count,
contributing_observers, collisions}`. Applies the **candidate cap**:
if `fetch_candidate_paths` exceeds the cap (default 5000), stops and
returns `{matched_count: null, truncated: true, candidate_count}` so no
preview call does unbounded work (see Phase 4).
- Helpers: `derive_expected_hash`, `detect_observed_width`,
`prefix_collision_counts` (`GROUP BY lower(public_key[:2])`).
- `tests/test_collector/test_links.py`: subsequence (gaps allowed, order
enforced), **per-reception isolation** (no cross-observer splice),
multi-observer dedup to distinct packets, observer-scope filter, **hop-span
cap rejects wide co-occurrences and keeps close ones on long packets**,
threshold boundary, prefix-match across widths.
enforced, span cap), **per-reception isolation** (no cross-observer
splice), multi-observer dedup to distinct packets, observer-scope filter,
**threshold short-circuit**, **`no_coverage` vs `unhealthy` separation**,
**`recent_matches` ordering/limit**, and **preview truncation**.
### Phase 4: CRUD API + schemas
- New `api/routes/links.py` + `common/schemas/links.py` mirroring
`api/routes/channels.py`: `GET /api/v1/links` (RequireRead, role-filtered,
`@cached`, embeds current `link_result`); `GET/POST/PUT/DELETE
/api/v1/links/{id}` (RequireAdmin writes; ≥2 `link_nodes` validated in
Pydantic; `expected_hash` auto-derived from `node_id` when omitted; observer
set managed inline). Register router in `api/app.py`.
`api/routes/channels.py` (which uses `@cached` from `api/cache.py`,
`RequireRead`/`RequireAdmin` from `api/auth.py`, and `DbSession` from
`api/dependencies.py`): `GET /api/v1/links` (RequireRead, role-filtered,
`@cached`, embeds current `link_result`); `POST /api/v1/links`
(collection-level, like channels) and `GET/PUT/DELETE /api/v1/links/{id}`
(RequireAdmin writes; ≥2 **distinct** `link_nodes` validated in Pydantic;
`expected_hash` auto-derived from `node_id` when omitted, and re-derived for
all path nodes when `match_width` changes; observer set managed inline).
Note: channels has no single-resource `GET`; links adds one to serve the
embedded result. Register router in `api/app.py`.
- `POST /api/v1/links/preview` (RequireRead — any authenticated user may
preview; it computes no saved state): accepts an unsaved link config (path
`node_id`s, `match_width`, observers, `max_hop_span`, `window_hours`) and
returns `{matched_count, contributing_observers, collisions}` by delegating
to `collector.links.preview_link`. Not cached (inputs are arbitrary).
`preview_link` applies a **candidate cap** (default 5000): on overflow it
returns `{matched_count: null, truncated: true, candidate_count}` and the
UI shows "~many — narrow your path to preview", bounding every call to one
scan. The client debounces (~400ms) and cancels in-flight via
`AbortController` (the `signal` pattern) so typing never stacks calls.
- `GET /api/v1/links/{id}` (RequireRead, role-scoped) returns the **full
detail** per UI Design → Data split: the lightweight result plus
contributing observers (with counts) and the latest ~3 matched paths (via
`recent_matches`). The list `GET /api/v1/links` embeds only the lightweight
result (`state`, `matched_count`, `threshold`, `evaluated_at`).
- `tests/test_api/test_links.py`: CRUD, role-scoping, visibility filter,
min-2-nodes rejection, result embedding.
min-2-nodes rejection, result embedding, **the preview endpoint**, and the
**`GET /{id}` detail shape** (observers + recent paths).
### Phase 5: Evaluator thread
- New `collector/link_evaluator.py` wrapping `collector/links.py`. In
- New `collector/link_evaluator.py` wrapping `collector/links.py`.
`evaluate_all_links` iterates only enabled links. In
`collector/subscriber.py`, add `_start_link_evaluator_scheduler` /
`_stop_link_evaluator_scheduler` (copy of the spam sweep), started in
`start()` (~line 667) and stopped in `stop()` (~line 708); thread attr near
line 114. Immediate first run, 60s loop, dialect upsert, per-iteration error
`start()` (after the spam scheduler at ~line 669) and stopped in `stop()`
(after the spam stop at ~line 710); thread attr near line 114. Immediate
first run on startup, 60s loop, dialect upsert, per-iteration error
logging.
- `tests/test_collector/test_link_evaluator.py`: upsert idempotency,
immediate-first-run, disabled when interval is 0.
### Phase 6: Prometheus
- In `api/metrics.py::collect_metrics`, read `link_results ⋈ links` and emit
the three gauges labelled by link name. Verify in
`meshcore_link_healthy`, `meshcore_link_state`, `meshcore_link_matched_packets`,
and `meshcore_link_threshold`, labelled by link name. Verify in
`tests/test_api/test_metrics.py`.
### Phase 7: Web UI + i18n
- New `web/static/js/spa/pages/links.js` (mirror `channels.js`): list grouped
by visibility with health badges; admin CRUD modal with the node multi-picker
(collision badges + observed width + live preview), observer multi-picker,
and an optional "within N hops" field (`max_hop_span`, empty = unlimited).
- Register route in `web/static/js/spa/app.js` (~lines 27, 90), add a nav card
in `home.js` (~line 99) and page titles; add `entities.links` + `links.*`
strings to `web/static/locales/en.json` and `nl.json`.
- Build per the **UI Design** section above. New
`src/meshcore_hub/web/static/js/spa/pages/links.js` (mirror `channels.js`):
summary strip + visibility-grouped cards sorted unhealthy/no_coverage first;
four-state health badge; **inline accordion expand** (toggle `expandedId` in
page state, lazy `GET /api/v1/links/{id}` on first expand, cached) showing
diagnosis / contributing observers / latest matched path (✓ markers via
`recent_matches`) / config recap / "View packets" deep-link; wider
(`modal-box-lg`) add/edit modal with the shared node path-builder +
observer picker, segmented `match_width` control, and a debounced
`POST /api/v1/links/preview` driving the live "matches in 24h" readout and
collision badges.
- Register route in `src/meshcore_hub/web/static/js/spa/app.js`: add
`links: () => import('./pages/links.js')` to the `pages` lazy-load map
(~line 27); add a `if (features.links !== false) { router.addRoute('/links',
pageHandler(pages.links)); }` block (~line 92, next to the channels guard);
add a `composePageTitle('entities.links')` title entry (~line 178). Add a
nav card in `src/meshcore_hub/web/static/js/spa/pages/home.js` (~line 99)
among the existing `renderNavCard` blocks in `renderHeroSection`.
- Add `entities.links` plus a new **`mesh_links.*`** top-level block to
`src/meshcore_hub/web/static/locales/en.json` and `nl.json`. The existing
top-level `links` block (`en.json:121-127`) holds footer labels
(website/github/discord/youtube/profile) and must **not** be reused — hence
`mesh_links.*` for the feature namespace.
### Phase 8: Config + seed loader + docs
- Add `feature_links=True` and `link_evaluator_interval_seconds=60` to
`common/config.py` (and the `features` dict ~line 611); add a `links_file`
property resolving to `$SEED_HOME/links.yaml`; update `.env.example`.
`common/config.py` (as `feature_*` `Field(...)` declarations in the
~569-602 block), and a `"links": self.feature_links` entry in the `features`
property's returned dict (dict body at ~lines 622-634); add a `links_file`
property mirroring the existing `channels_file` property at
`config.py:367-372` (resolves to `Path(self.effective_seed_home) /
"links.yaml"`); update `.env.example`.
- New `_import_links` in `collector/cli.py`, wired into `_run_seed_import` so
`meshcore-hub seed` and the compose `seed` profile pick up `links.yaml`
automatically. Idempotent upsert by `name`; resolves path/observer nodes by
@@ -348,6 +548,50 @@ first bytes co-occur far apart on an unrelated long flood path.
`docs/seeding.md` and `docs/letsmesh.md`. Optional `meshcore-hub links
list|delete` CLI (create/edit stays in the UI or seed).
### Phase 9: Consolidate packet-detail path read onto the hop table
- Opportunistic consolidation riding on the populated `packet_path_hops`
table. In `api/routes/packet_groups.py::get_packet_group`, replace the
per-reception `_extract_path_hashes(packet.decoded)` call (~line 292) with a
batched `SELECT raw_packet_id, position, node_hash FROM packet_path_hops
WHERE raw_packet_id IN (:ids) ORDER BY raw_packet_id, position` (one query
for the whole reception set), grouped into the same
`receptions[i].path_hashes` shape. **Zero client-visible payload change** —
`PacketReceptionInfo.path_hashes` stays `Optional[list[str]]`; the only
renderer (`packet-group-detail.js`) is untouched.
- Delete `_extract_path_hashes` (lines 36-51) — it becomes a dead third copy
of the dual-path extraction (the live normalizer in `letsmesh_normalizer.py`
and the frozen copy in migration `20260703_2250` remain the canonical
sources). Note the list endpoint and dashboard charts already bypass JSON
(they use the indexed `path_hash_bytes` column), so only the detail route
changes.
- Depends on the Phase 1 backfill being complete (every existing `raw_packet`
has its hops populated). For a row that somehow lacks hops, fall back to an
empty list (the renderer already handles a missing/empty path).
- `tests/test_api/test_packet_groups.py`: assert the detail endpoint returns
identical `path_hashes` per reception after the swap (golden-path parity),
including multi-observer divergence and packets with no path.
## Enabled Future Capabilities
The `packet_path_hops` index is built for Link matching, but it unlocks
packet-exploration features that are **impossible today** (each would require
a full-table JSON scan). This plan does **not** build them; they are noted
here as follow-on work, each likely its own plan:
- **Path-node filtering on `/packets`** — "show packets that passed through
node X" via an indexed `(node_hash, received_at)` lookup. Today's path-node
popover in `packet-group-detail.js` is read-only (resolves a hash to known
nodes); this would turn it into an interactive filter
(`/packets?path_node=…`) plus a new `list_packet_groups` query param.
- **Per-node relay statistics** — most active repeaters, common hop positions
(`GROUP BY node_hash`).
- **List-page hop preview** — the `/packets` list currently shows no path
content (per-row extraction in a GROUP BY is too costly); a cheap indexed
lookup could show first-3-hops previews per group.
Phase 9 captures only the low-risk consolidation (the packet-detail endpoint
reading from the hop table). The filtering/stats features above are deferred.
## Open Questions
- **Collision tolerance.** At 1-byte matching (256 buckets) on a busy mesh,
@@ -386,3 +630,83 @@ first bytes co-occur far apart on an unrelated long flood path.
(`0300609`), path-hash-bytes filter (`c029eae`), JSON tree / packet path
flow (`f845830`) — all building on the raw-packet foundation this plan
extends.
## Review
**Status**: Approved with Changes
**Reviewed**: 2026-07-06
### Resolutions
**Conflicts** — None. Cross-checked against all 45 plans under `docs/plans/`,
all 7 source files the plan modifies, and `git log --oneline -20` on `main`.
No existing `packet_path_hops` table, `Link` model, or `/api/v1/links`|
`/api/v1/routes` endpoint exists. The plan builds on (not duplicates)
spam-detection (`20260622-2243`), path-hash-bytes-filter (`20260703-2338`),
raw-packets-feature (`20260612-2014`), and channel-model-db-decrypt
(`20260519-2051`); their cited commits (`c029eae`, `0300609`, `f845830`) all
landed on `main`.
**First-pass resolutions (content):**
- **F1 — Defaults**: `window_hours` defaults to 24 (range 1..720),
`packet_count_threshold` to 3 (range 1..10000); `enabled` defaults `true`.
- **F2 — Duplicate path nodes**: entries must be distinct; validated in Pydantic.
- **F4 — Disabled links**: excluded from evaluation, produce no `link_result`,
omitted from Prometheus.
- **T7 — Cascade completeness**: `link_results`, `link_nodes`,
`link_observers` all cascade-delete with their parent `links` row.
- **Phase 1 — Backfill enumeration**: frozen extraction yields an ordered list;
backfill enumerates index = `position`.
**Second-pass resolutions (factual corrections verified against source):**
- **T4 — Upsert citation was wrong on two counts.** The cited path
`collector/handlers/event_observer.py` does not exist (the file is
`common/models/event_observer.py`), and the cited method
`on_conflict_do_update` does not exist anywhere in the repo — the branch at
`:143-158` uses `on_conflict_do_nothing`. Corrected to point at the real
dialect branch and clarify the evaluator authors the `do_update` variant.
- **Phase 2 — Insertion point was imprecise.** Line 138 is the start of
`session.add(RawPacket(...))` (spans 138-155), not the hop-insert point.
Hops need `raw_packet.id`, which requires assigning the inline `RawPacket`
to a variable and calling `session.flush()` before the bulk-insert.
`path_hashes` is computed at 106-111 (112 is the derived byte width).
Corrected in place.
- **Phase 7 / Phase 8 — Web paths were missing the `src/meshcore_hub/`
prefix.** There is no top-level `web/` directory; the real root is
`src/meshcore_hub/web/`. All `web/...` citations corrected.
- **F5 — `VISIBILITY_LEVELS` location.** It lives in
`api/channel_visibility.py:13`, not `models/channel.py`; import path added.
- **Phase 4 — Route shape clarified.** Channels has no single-resource `GET`
and its POST is collection-level (`""`); links' POST mirrors that, and links
adds a `GET /{id}` to serve the embedded result. `@cached`/`RequireRead`/
`RequireAdmin`/`DbSession` import paths confirmed.
- **Phase 8 — Config locations tightened.** Feature-flag `Field(...)` decls
live at ~569-602; the `features` property's dict body is at ~622-634 (not
~611); the `links_file` property mirrors `channels_file` at
`config.py:367-372`.
- **Phase 5 — Line numbers confirmed.** Spam scheduler calls occupy 667/708
exactly; ~669/~710 is the correct after-insertion point. Thread attr at 114
confirmed.
**Decisions:**
- **i18n namespace** — `mesh_links.*` chosen for the feature's page strings.
The existing top-level `links` block (`en.json:121-127`) holds footer labels
(website/github/discord/youtube/profile) and must not be reused; `mesh_links`
avoids the collision without renaming existing strings. `entities.links`
(the nav label) is unaffected.
### Remaining Action Items
- **Naming confirmation** — "Link" / `links` must be confirmed before Phase 1
(the migration authors the table; renaming later is costly).
- **Cap on configured nodes** — confirm ~8 is comfortable for the realistic
longest route (drives max self-join depth).
- **Collision tolerance** — validate the acceptable false-healthy rate with
live data once the hop table is populated.
- **Observer coverage guidance** — UI decision during Phase 7 (recommend
adding observers when a link reads unhealthy with zero contributing
observers).
These four items are the plan's existing Open Questions; none block starting
Phase 1 except naming.