The update_profile endpoint used RequireUserOwner which returns only the
caller's user_id — no role information. The ownership check blocked ALL
non-owner edits with 403, including admins. This regressed in d37b30a
when the old Member model (RequireAdmin) was replaced with UserProfile.
Backend: read X-User-Roles header directly in update_profile (same
pattern as node_tags.py / routes.py) and bypass the ownership check when
the admin role is present. Regular members editing their own profiles
are unaffected — RequireUserOwner stays as the dependency.
Frontend: extract ProfileEditForm component (with data-testids) from
OwnProfileView. PublicProfileView now shows an inline edit form when an
admin views another user's profile. Owner still gets the existing edit
link; non-admins see nothing.
Tests:
- Backend: test_update_profile_admin_can_edit_other (admin edits other
user's profile, asserts 200 + all fields updated)
- Vitest: 4 new tests — admin button visibility, non-admin hidden,
owner link vs admin button, form submission to correct endpoint
- E2E: admin-profile-edit.spec.ts (admin edits Mem South's profile,
verifies persistence; admin on own profile sees no admin button);
members.spec.ts negative assertion (member sees no admin button)
Adds a checkbox filter to the Routes page that narrows the list to
routes owned by the current user. The filter is URL-driven (?mine=true),
cached server-side automatically via the existing key builder, and only
shown to operators/admins (members can't own routes).
Backend: mine query param on GET /api/v1/routes filters by created_by
matching the caller's X-User-Id. Legacy NULL routes are excluded.
Tests:
- Backend: 6 new TestRouteMineFilter tests (own/other/null/admin/default)
- Vitest: 6 new tests (param passing, role gating, checkbox state)
- E2E: new seed route owned by pw-operator + mine filter spec
Admins editing an operator-created route no longer steal ownership.
Ownership transfer now happens only for legacy (NULL created_by) routes.
This ensures operators retain edit access to their routes after an admin
makes a small tweak.
Replace visibility-tier-based write permissions with ownership-based:
operators can only edit/delete routes they created; admins can modify
any route and take ownership on edit. Each route stores the creator's
OIDC user_id (created_by column). Legacy routes with NULL created_by
are admin-only.
The creator's friendly name is resolved from UserProfile and displayed
on the route card with a profile link. Edit/delete buttons are hidden
per-route based on ownership rather than a flat role check.
- Migration: add nullable routes.created_by (batch mode, SQLite-safe)
- _assert_route_modifiable: dual-check visibility (404) + ownership (403)
- create_route: stamps created_by, ensures profile exists
- update_route: admin edits transfer ownership (logged)
- RouteOwner schema mirrors AdoptedByUser pattern
- Batch owner resolution in list endpoint (avoids N+1)
- Frontend: per-route canEdit gate, owner badge with profile link
- Tests: 20+ backend ownership tests, 5 frontend gating tests, e2e assertions
- Docs: routes.md + auth.md updated for ownership model
Operators can now create, edit, and delete routes (previously admin-only).
A user may never scope a route above their own role tier: an operator
creating/editing an admin-visibility route is rejected (403 on the
visibility value, 404 on touching an existing higher-visibility route),
preventing them from creating routes they could then never see or modify.
- routes.py: RequireAdmin -> RequireOperatorOrAdmin on create/update/delete;
add visibility-cap enforcement helpers reusing the existing
resolve_user_role / VISIBILITY_LEVELS ladder
- web/app.py: proxy access map admits operator for routes POST/PUT/DELETE
- Routes.tsx: canManage gate (admin||operator) on Add/Edit/Delete; visibility
<select> filters options by caller tier so operators never see 'admin'
- tests: operator-tier coverage (create/update/delete at/below/above level),
proxy access-map assertion, vitest role-gating + filtered select
- e2e: mint operator session + routes-operator spec
- docs: routes.md + auth.md operator/visibility-cap notes
Replaces the httpx-based smoke tests in tests/e2e/ with a browser E2E suite
under e2e/, running against a self-contained throwaway stack that never
touches the local development database.
Stack & data isolation (e2e/docker-compose.test.yml):
- Own ephemeral Postgres 17 (schema via the `migrate` service / Alembic),
distinct project name + named volumes, no host DB port, no \${VAR}
interpolation. `make e2e-down` destroys everything.
- OIDC enabled with a known session secret; WEB_AUTO_REFRESH_SECONDS=2 so
polling is assertable; CONTENT_HOME mounts a test markdown page.
- Deterministic seeder (e2e/seed_data.py) run via global setup inside the
collector container: nodes/observers with `area` tags, adverts, messages
on the public (17) + a custom channel, raw packets + path hops keyed to
node prefixes, a route + health/history, profiles + adoptions, and
event_observers rows so the observer filter/badges resolve.
Auth & tests:
- Forged signed `meshcore-session` cookies (e2e/mint_session.py,
itsdangerous) for admin/member identities -> storageState; no real IdP
needed. 31 specs across 12 files cover global nav/theme, profile
menu+edit, home hero, dashboard widgets, list filters/auto-refresh/row
actions, observer toggles, the path-node overlay, map filters +
show-labels, members, markdown pages, and routes add/edit/delete
(persistence, validation, confirm dialog). workers:1 / fullyParallel:false
against the single shared backend; targeted data-testids added to React
components for stable selectors.
Fixes surfaced while running the suite on fresh Postgres:
- Migration 5e3b712ccf10 aborted on Postgres: its route-health backfill
queries the live Route model (which now has max_path_length) before that
column exists. The swallowed error left the transaction aborted, killing
the subsequent alembic_version stamp. Wrapped the backfill in a SAVEPOINT
so a failure rolls back cleanly without blocking the migration.
- Restored the full ABUSE_* env set required by the MQTT broker, and set
the web service's API_KEY to the admin key (admin writes go through the
proxy as a Bearer token).
31/31 passing; tsc (frontend+e2e), pytest (1460), and pre-commit green.
Migrate the React SPA off the bespoke useApiFetch hook and raw useEffect fetches to TanStack Query: useQuery/useQueries for reads, useMutation + invalidateQueries for writes; a central query-key factory and invalidation helpers mirroring the backend cache_invalidation prefixes (channels/routes/nodes/messages/profiles/dashboard/adoptions); polling via refetchInterval (useAutoRefresh now returns pause state only); RouteCard self-fetches its detail/history, dropping the hand-rolled caches. Adds QueryClientProvider in App, a renderWithProviders test helper, and deletes useApiFetch.
Also consolidates recurring UI into reusable components (Breadcrumbs, ListToolbar, Modal/ConfirmDialog, NotFoundState, TimeAgo, Definition, CopyableValue, MeshQrCode, Badges, SectionGroup, PageHeader) and fixes the FilterForm clear button, merged toggle wrappers, and role-aware channels/messages cache keys so client invalidation matches the server.
Verified: tsc --noEmit clean, vitest 113 passed, pytest test_web 251 + test_cache 119 passed, pre-commit --all-files green.
The React frontend is complete (Phases 1-3); the lit-html fallback is dead
(its vendor globals were removed in Phase 3). Delete it and the scaffolding:
- Delete the entire src/meshcore_hub/web/static/js/spa/ lit-html tree,
LitBridge.tsx, and legacy.d.ts.
- Remove the @legacy alias from vite.config.ts and tsconfig.json.
- Remove lit-html and qrcodejs from package.json (both unused now).
- Remove the lit-html fallback {% else %} branch from spa.html — the Vite
build is now required to serve the UI (no fallback bundle).
Tests (fallback no longer exists):
- test_home/advertisements/nodes/messages.py: assert the React mount point
(id="app") instead of a bundled-or-fallback script tag.
- test_caching.py: JS-cache tests are header-only (static JS is bundled into
dist/, absent in test env; the middleware sets headers on 404 too); the
dist-bundle HTML test drops its fallback branch.
Docs:
- AGENTS.md: new Frontend (React) section (host-run npm/vite/tsc toolchain,
react-chartjs-2/react-leaflet/react-qr-code, CSS load order); clarified the
compose-stack rule to exempt frontend tooling.
- REACT_MIGRATION.md: Phase 4 complete, final file structure, decisions.
Verified: tsc --noEmit clean, npm run build, full pytest (1463 passed,
22 skipped), pre-commit (passed).
Replace all window.Chart / window.L / window.QRCode globals and the
charts.js helper script with bundled React components:
- react-chartjs-2: typed config builders in utils/charts.ts
(buildLineChart, buildActivityChart, buildStackedBar, buildRoutesTrend,
buildRouteDetailStrip + ChartColors, averageRouteTier, routeQualityToTier)
and wrappers in components/charts/Charts.tsx (ActivityChart,
TrendLineChart, StackedBarChart, RoutesTrendChart, RouteDetailStrip).
utils/charts.ts imports chart.js/auto. Wired into Home, Dashboard, Routes.
- react-leaflet: MapPage rewritten with MapContainer/TileLayer/Marker/Popup
+ a useMap MapController for fit-bounds and memoized markers; NodeDetail
static hero map with divIcon marker + OffsetCenter. Both import
leaflet/dist/leaflet.css.
- react-qr-code: replaces window.QRCode in Channels and NodeDetail.
Bundling & shell:
- Chart.js, Leaflet (+CSS), react-qr-code now bundled by Vite; removed the
leaflet/chart.js/qrcodejs vendor <script>/<link> tags from spa.html and
their copy steps from build.js (fonts stay vendored). Deleted charts.js.
- Moved the Vite CSS bundle (asset_app_css) into <head> before app.css so
app.css dark-mode Leaflet overrides win over the bundled leaflet.css.
- Dropped the chart globals from the Window type declaration.
Tests/docs:
- test_caching.py: removed charts.js-specific tests; generic JS-cache tests
now target spa/app.js.
- Updated charts.js cross-references in collector/routes.py + test_routes.py
to point at spa-react/utils/charts.ts.
Verified: tsc --noEmit clean, npm run build (153 modules),
pytest tests/test_web (255 passed), pre-commit (passed).
Adds a new nullable per-route knob that caps the total number of hops
in a candidate packet's path. Packets whose path exceeds the cap are
dropped from matching consideration entirely (before the subsequence
matcher runs), so over-long paths never count toward
packet_count_threshold. Complements the existing max_hop_span, which
only constrains the gap between the first and last matched configured
node.
- Route model + migration (additive, nullable, default null = unlimited)
- Threaded through matcher chain (_subsequence_indices early-return)
- All 5 evaluate/preview/recent_matches call sites updated
- API serializer/create/update/preview passthrough
- CLI seed YAML import (update + create paths)
- Frontend: distinct icons for span (<-o->) vs path-length (|<->|),
always-rendered badges with infinity fallback, hover tooltips on
every stats row item, i18n keys (en + nl)
- Tests: matcher unit tests (within/exceeds cap), API round-trip,
CLI seed import
The route edit modal builds its PUT body with a ternary that collapses
an empty observer list to null:
observer_public_keys: observerPublicKeys.length > 0 ? observerPublicKeys : null,
Pydantic parses null as None, and the PUT handler's guard
if body.observer_public_keys is not None:
_sync_observers(session, route, observer_nodes)
skips the sync entirely when the field is None. So removing all
observers in the modal sent null -> no DB change. Adding observers
worked because a non-empty array passed the guard and _sync_observers
deleted + recreated.
The None-means-skip semantic is correct for true partial updates, so
the fix is on the frontend: the edit modal is a full-form PUT, so it
must always send the array. When empty, it sends [], which Pydantic
parses as [] (not None), the guard passes, and _sync_observers deletes
every existing RouteObserver row.
Tests: add test_update_clear_observers_with_empty_list next to the
existing test_update_observers as a regression guard. It seeds one
observer via PUT, then PUTs observer_public_keys: [] and asserts both
the response and a fresh GET come back with an empty list.
The Route Health and Routes Trend dashboard widgets pull from
GET /api/v1/dashboard/routes-overview, which previously filtered
routes by the caller's role tier (anonymous saw community, admin saw
all four tiers). Operators and admins therefore saw member/operator/
admin-tier routes mixed into the dashboard, even though the dedicated
/routes page is the working surface for managing those private tiers.
This change makes the dashboard widget surface ONLY community-tier
routes, regardless of the caller's role. Operators and admins still
see all four tiers on the /routes management page (unchanged).
Implementation:
- get_routes_overview: drop resolve_user_role/get_max_visibility_level;
push the filter into the SQL query
(where Route.visibility == RouteVisibility.COMMUNITY.value) so
higher-tier rows are no longer loaded just to be discarded.
- _dashboard_routes_overview_key_builder: drop the role dimension from
the cache key. The response is now identical across roles, so the
per-role cache slots were storing four identical copies. The prefix
is preserved so invalidate_routes' pattern invalidation still hits.
- Drop unused VISIBILITY_LEVELS import; add RouteVisibility to the
models import.
Tests:
- test_visibility_filter_hides_admin_routes renamed to
test_dashboard_only_shows_community_routes_regardless_of_role;
now seeds one route per visibility tier and asserts every role
(anonymous/member/operator/admin) sees only ['Public'].
- test_cache_key_is_role_scoped renamed to
test_cache_key_is_role_agnostic; asserts all four role-variants
produce the SAME cache key (no 'role=' dimension).
Change the new-route defaults so the create modal, REST API, YAML
import, and preview helper pre-fill packet_count_threshold=5 instead
of 3, and bump the effective-clear auto-multiplier from 2x to 3x so
the default comfort bar tracks to 15 for a default threshold of 5.
Schema/model/cli/preview:
- schemas/routes.py: RouteCreate + RoutePreviewRequest
packet_count_threshold default 3 -> 5 (clear_threshold stays None)
- models/route.py: Route INSERT default 3 -> 5 (clear_threshold stays
nullable, no default)
- collector/cli.py: YAML import fallbacks 3 -> 5
- collector/routes.py: preview helper fallback 3 -> 5; module constant
CLEAR_DEFAULT_MULTIPLIER 2 -> 3 (drives effective_clear_threshold)
- api/metrics.py: route_clear gauge help text 2x -> 3x
UI:
- web/spa/pages/routes.js: new-route modal pre-fills
packet_count_threshold=5; placeholder shows 3x threshold; parseInt
fallback 3 -> 5. clear_threshold field unchanged (clearing still
sends null for the auto-tracking behaviour).
Docs:
- docs/routes.md defaults table: window_hours 24->48, threshold 3->5,
clear (2x)->(3x), max_hop_span (unlimited)->8. Previous four
entries now match the shipped defaults (some were stale from #316).
- docs/seeding.md YAML example: window_hours 24->48, threshold 3->5,
commented clear_threshold example 10->15 with 3x note.
No migration: packet_count_threshold uses Python-side default= (no
server_default), so existing rows keep their stored values. Existing
routes with clear_threshold=NULL continue to track via the new 3x
multiplier at evaluation time.
Persist route health derivations so the API layer no longer recomputes
them on every request. Two new tables (route_result_history,
route_recent_matches) plus a quality_avg column on route_results back
the dashboard strip and per-route history endpoints.
Collector:
- run_evaluation (60s) writes snapshot + quality_avg + recent_matches
- run_history_backfill (hourly) recomputes completed-day buckets
- subscriber wires a dedicated backfill scheduler thread
API:
- dashboard routes-overview bulk-loads via single history SELECT
- routes detail/detail history read from precomputed tables with
live-compute fallback
- POST/PUT on routes upserts recent_matches and quality_avg inline
- dashboard cache TTL lowered from 1h to 5m (invalidation-aware)
Config: route_history_backfill_interval_seconds=3600,
redis_cache_ttl_dashboard default 3600 -> 300
The overall health badge on route cards now shows the rolling 7-day
average tier instead of the latest window-hours snapshot, so flapping
routes that are currently up still appear marginal/failing if the
week's mean warrants it. Same averaging drives the dashboard Route
Health widget's summary dot, the routes page summary strip counts,
and (already) the Route Trends chart line colors.
- compute_average_quality() in collector/routes.py (0/1/2 mean,
thresholds 1.5/0.75, empty-history fallback) — kept in sync with
the averageRouteTier JS helper in charts.js
- RouteRead / RouteDetail gain a 'quality_avg' field
- list/get/update handlers compute it per route; create skips (no
meaningful history yet) and the frontend falls back to
route_result.quality for brand-new routes
- diagnosis tooltip unchanged (still current-snapshot state text)
Adds Route Trends chart and Route Health strip grid to the dashboard,
backs them with a new GET /dashboard/routes-overview endpoint, and
consolidates the Redis cache layer (removes ROUTE_DETAIL TTL, raises
DASHBOARD default to 1h, splits recent-activity into its own short-TTL
endpoint so Recent panels stay fresh while aggregate counts cache 1h).
Reported symptom: user edits a Route (e.g. lowers packet_count_threshold
from 6 to 3), and the routes list card still shows the old threshold for
~30s. The cache stack was exonerated — x-cache: MISS on the stale GET,
response body had the new packet_count_threshold=3, but route_result.
threshold remained at 6.
Root cause was neither HTTP cache nor Redis cache. The list card's stats
row (renderStatsRow in routes.js:82) displays route_result.threshold /
effective_clear / matched_count, which are persisted by the background
route_evaluator on a 30-60s schedule — separate DB row from the route's
direct fields. The PUT handler updated the route row immediately but
never triggered a re-evaluation, so route_result carried the stale
snapshot from the prior evaluator cycle until the next sweep.
Fix: add _reevaluate_route(session, route) helper that runs evaluate_route
+ upsert_route_result synchronously after the route commit. Called from
create_route (initial evaluation) and update_route (refresh on every
config change). Disabled routes short-circuit (no point evaluating a
route that's turned off). One bounded DB scan per write — same cost as
a single-route evaluator tick.
After this fix, the PUT response itself carries a fresh route_result
reflecting the new packet_count_threshold / clear_threshold, and the
next GET /api/v1/routes returns it. The list card updates on the very
next render cycle.
Tests:
- test_update_threshold_immediately_reflects_in_route_result: seeds a
stale RouteResult with the OLD threshold, sends a PUT, asserts the
response's route_result.threshold/effective_clear now match the new
route config.
- test_disabled_route_does_not_trigger_evaluation: monkeypatches
evaluate_route to a spy, asserts it's never called for disabled routes.
After PR #311's invalidation wiring and HTTP policy fix, a user reported
the routes list still showed stale data for ~30s after a PUT. The HAR
they sent actually showed the fix working (x-cache: MISS after PUT,
fresh data returned) but had been captured with DevTools' 'Disable
cache' enabled — so it didn't represent normal browsing. We had four
competing hypotheses and no way to pick between them.
Add structured INFO logging at the two points that matter:
* meshcore_hub.api.cache_invalidation._drop now emits
'Cache invalidate start: prefix=... backend=...' and
'Cache invalidate ok: prefix=...'. The backend= field
distinguishes RedisCacheBackend from NullCache in one glance,
catching 'REDIS_ENABLED is actually false' cases.
* meshcore_hub.common.redis.RedisCacheBackend.delete now emits
'Redis cache delete: prefix=... full_prefix=... keys_deleted=N
scan_iterations=N'. keys_deleted=0 after a mutation that should
have invalidated entries is the smoking gun for a cache-key
mismatch between the store path (key_builder) and the delete
path (prefix glob).
* NullCache.delete emits a DEBUG line for the same reason.
The error paths are also enriched with full_prefix for greppability.
No behavioral changes — invalidation still swallows errors so cache
outages never break a write.
After deploy, a single route-edit repro will produce log output whose
shape (start/ok/warning/missing, keys_deleted count) unambiguously
identifies which of the four hypotheses is correct, so we can write
a targeted fix instead of guessing.
After PR #311 wired server-side invalidation, a user reported that editing
a Route still showed old values on the routes list for ~30s. Root cause:
the api_cache_middleware emitted 'Cache-Control: private, max-age=30' on
@cached GETs, which let the browser serve its local HTTP cache copy
without revalidating. The Redis invalidation fired correctly but never
mattered — the browser never asked the server.
Switch all /api/* GET responses to 'private, no-cache' (synonymous with
'max-age=0, must-revalidate'). The browser now sends If-None-Match on
every navigation; the server answers 304 when Redis is warm and unchanged
(cheap — no body) or 200 after an invalidation.
The per-endpoint Redis TTL (30s default, 300s dashboard/route-detail)
still bounds cache.set lifetime; only the HTTP-layer max-age disappears.
Most navigations are still 304s, so the cost is one tiny round-trip per
page load while guaranteeing freshness after any mutation.
Adds an end-to-end regression test (test_routes_list_refresh_after_mutation)
modelling the exact reported scenario: cache-fill, conditional 304,
PUT mutation, conditional 200 with fresh body.
Mutation handlers (POST/PUT/DELETE) on channels, routes, user profiles,
node tags, and adoptions now drop the corresponding Redis cache entries
after commit so the UI reflects changes on the next page load instead of
waiting for the 30s/300s TTL.
Adds meshcore_hub.api.cache_invalidation with seven helpers that
encapsulate the two cache-key formats (endpoint-name keys like 'nodes:'
vs URL-path keys like '/api/v1/channels:') and swallow backend errors
so a cache outage never breaks a successful write. The helper is a
no-op when Redis is disabled.
Cross-entity embeddings are covered: node-tag writes invalidate nodes,
messages, advertisements, and dashboard; adoption writes invalidate
nodes, profiles, advertisements, and dashboard.
CLI/collector mutations remain TTL-bound (infrequent, operator-driven).
Bundles four independent improvements to API caching and route evaluator
accuracy that all touch the same files in the cache/routes stack.
## HTTP Cache-Control headers (api_cache_control_enabled)
The API now emits HTTP Cache-Control on every /api/v1/* response and
handles ETag / If-None-Match -> 304 Not Modified on @cached endpoints.
- @cached GETs: private, max-age=<redis_ttl> + strong SHA-256 ETag.
Matching If-None-Match returns 304 with empty body.
- Uncached GETs under /api/v1/*: private, max-age=0, must-revalidate.
- POST/PUT/DELETE/PATCH and /health*: no-store.
- All private (several @cached endpoints are role-aware).
- @cached decorator stores new envelope {body, etag} in Redis; legacy
bare-JSON entries still read (hashed on the fly) and auto-migrate on
the next miss.
- X-Cache: HIT|MISS observability header is always emitted.
- Kill switch: API_CACHE_CONTROL_ENABLED (default true).
## Dashboard cache TTL bump (30s -> 300s)
REDIS_CACHE_TTL_DASHBOARD default raised from 30s to 300s. Covers all
/dashboard/* endpoints and /api/v1/routes/{id}/history. These return
trend/aggregation data where minute-level staleness is invisible but
every MISS costs seconds of SQL/aggregation. Also resolves reported
short-TTL behaviour on the per-route healthchart (which shares this
setting). Operators with explicit env var: unchanged. Old Redis entries
expire naturally within <=30s; no flush needed.
## Route recent-packets hop truncation
In the routes page expanded card, path hops are now truncated to
2 + ellipsis + 2 when length > 5, matching the packet-detail page
styling. Reuses the existing packets.hops_hidden i18n key as the
tooltip on the ellipsis badge.
## Route event-hash deduplication
A single advert/message/telemetry/trace retransmitted or flooded
through the mesh produces one RawPacket per on-air copy, each with a
fresh wire packet_hash. Counting those as distinct matches let a single
underlying event satisfy packet_count_threshold within seconds and
biased route health.
- Add nullable event_hash column to raw_packets and packet_path_hops
(alembic 6b3430fd84f4). No backfill; legacy rows keep NULL and the
evaluator falls back to wire packet_hash until they age out of
window_hours.
- Subscriber denormalizes the structured event's event_hash onto the
captured raw packet after handler dispatch.
- Route evaluator prefers event_hash when set, collapsing all
receptions of the same underlying event into one match.
Test data used a hardcoded _NOW (2026-07-12 12:00 UTC) while
run_evaluation used real datetime.now(), causing a time-window flake
once wall-clock time exceeded _NOW + window_hours (24h). Tests now
pass now=_NOW explicitly; production callers are unaffected (defaults
to datetime.now(timezone.utc) when not provided).
Route identity is now a (from_label, to_label) composite unique pair
instead of a single name column. The three route migrations (create
tables, add reversible, replace name) are collapsed into one clean
migration since the feature hasn't shipped yet — 8f2a3c4d5e6f creates
all five tables with from_label/to_label + reversible from the start,
no intermediate steps.
Engine: recent_matches now returns the sliced subpath between the
matched endpoints, not the full path. Diagnosis text moved to a hover
tooltip on the quality badge. Cards sorted by from_label. Seed YAML
switched to a list format with from/to keys; upsert by (from_label,
to_label). Plan/tasks docs updated to present the final schema.
Routes now default to bidirectional matching — packets traversing the
configured path in reverse (C->B->A instead of A->B->C) also count
towards health. A 'reversible' toggle lets users constrain a route
to one direction when needed.
- Route model: new 'reversible' column (default true, server_default 1)
- Migration: a1b2c3d4e5f6 (additive ALTER TABLE)
- Matching engine: _match_hops + _fetch_candidate_paths_maybe_bidirectional
helpers shared across evaluate_route, recent_matches, and preview_route
- API schemas: reversible on RouteCreate/Update/Read/Detail/PreviewRequest
- SPA: toggle in modal; card path chips show <-> for reversible, -> otherwise
- Seed loader + example YAML: reads 'reversible' key
- 4 new collector tests (reverse match, non-reversible, both-directions, API)
Add complete route health monitoring feature that tracks whether packets
traverse expected multi-hop paths through the mesh network.
Models & migration:
- 5 new models: PacketPathHop, Route, RouteNode, RouteObserver, RouteResult
- Alembic migration with keyset-paginated backfill of existing packets
Collector:
- store_raw_packet refactored to persist path hops via bulk insert
- Matching engine (collector/routes.py) with subsequence matching, quality
bands (clear/marginal/failing/no_coverage), and collision detection
- Background route evaluator (60s loop) wired into subscriber lifespan
- Route seed loader in CLI (resolves by public_key, matching YAML format)
API:
- 6 CRUD endpoints + preview endpoint under /api/v1/routes
- Schemas accept node_public_keys (64-char hex) instead of internal UUIDs
- 5 Prometheus gauges for route health metrics
- Packet groups endpoint reads from hop table
Web UI:
- Full SPA routes page with summary strip, grouped cards, expandable detail
- Routes nav entry in both desktop (spa.html) and mobile (app.js) navbars
- Home page nav card with feature gate
- API proxy access mapping for v1/routes endpoints
- Visual node-search path builder with autocomplete dropdown, ordered chips
with reorder/remove controls, and paste-64-char-key support
- Observer picker with same search UX (unordered chips)
- i18n strings in en.json and nl.json
Config:
- feature_routes flag (default: true)
- route_evaluator_interval_seconds (default: 60)
- routes_file seed path
- example/seed/routes.yaml
When an IdP name claim contains trailing whitespace (e.g. 'Matt '), every
proxied API request failed with 502 'Illegal header value' because httpx
enforces RFC 7230 which forbids leading/trailing OWS in header values.
Defense in depth:
- strip_userinfo() trims the IdP name at ingress (oidc.py)
- _sanitize_header_value() removes CTL/DEL chars at both header-injection
sites (API proxy + auth-callback bootstrap) in app.py
- update_profile() trims user-supplied names in PUT handler
X-User-Name is informational only (seeds non-unique UserProfile.name);
identity/auth are keyed on X-User-Id and X-User-Roles, so sanitization is
safe.
Add two horizontal 100% stacked-bar charts to the Dashboard, gated
behind the existing packets feature flag:
- Packet Types: top 6 event_type values + 'other' rollup
- Path Bytes: 1b/2b/3b path-hash width distribution (NULL excluded)
Driven by a single new cached endpoint
GET /api/v1/dashboard/packet-breakdown?days=7 returning pre-bucketed
counts. Frontend normalizes to percentages via a new
createStackedBarChart helper in charts.js with a 7-hue oklch palette.
Persist the path-hash byte width (1/2/3) as a nullable Integer column on
RawPacket, computed at ingest by the collector and backfilled for historical
rows via a self-contained Python migration. Replace the per-request Python
decode loop in the grouped-list route with a SQL MAX() aggregate + HAVING
filter, and add a discrete <select> filter (Any/1B/2B/3B) to the /packets
SPA page.
- Model: add path_hash_bytes column to RawPacket (nullable Integer)
- Migration: batch_alter_table add_column + keyset-paginated Python backfill
reading decoded via Core select() on sa.JSON column (portable across
SQLite and Postgres)
- Collector: compute path_hash_bytes at ingest via two-tier path-hash
extraction (decoded.path -> payload.decoded.pathHashes)
- API: add func.max() aggregate to group query, HAVING filter on
?path_hash_bytes=1|2|3 param; delete Phase 3 decode loop and dead helper
- Frontend: add path-width select filter wired through query/apiParams/
pagination/headerParams; add i18n keys (en/nl)
- Tests: 1186 passed, 22 skipped; collector + API + model coverage
Recent adverts card now shows route-type badges, an observer column
with three-way fallback (observers -> observed_by -> dash), and renders
all 10 rows instead of 5. The Type column hides on mobile to prevent
overflow, and the misleading cursor-help on observer badges is removed.
Also adds a Packets chart card to the dashboard and a packets stat to
the homepage, backed by the new dashboard packet-activity endpoint.
Promotes routeTypeBadge to a shared export in components.js (previously
local to advertisements.js), and enriches RecentAdvertisement with
route_type, observers, and observed_by fields.
Backend resolves observers via fetch_observers_for_events and
observer_node_id -> public_key in batched queries.
Replace the OS system font stack with IBM Plex: the variable-weight
sans (100-700, one file per subset) for UI and headings, and Plex Mono
400 for the public keys, packet hashes, and hex that font-mono renders
across the app. Latin + latin-ext subsets only (shipped locales are
en/nl); no italics; mono is never rendered bold here.
Fonts are self-hosted from @fontsource packages via the existing
build.js vendor pipeline (no CDN), copied to static/vendor/fonts/ with
a hard build failure on wrong filenames. Wiring:
- input.css: @theme --font-sans/--font-mono + @font-face rules with
unicode-ranges taken verbatim from the package CSS; Tailwind v4
derives the document default from --font-sans, re-fonting daisyUI
components with no other changes.
- spa.html: preload the latin sans variable woff2 (crossorigin, URL
identical to the @font-face src) to minimize FOUT.
- charts.js: Chart.defaults.font.family to match (Chart.js otherwise
uses Helvetica/Arial).
- error.html: name-prepend only; the page stays dependency-free.
- middleware.py: long-term immutable cache for /static/vendor/fonts/
(stable names referenced from CSS, so ?v= versioning can't apply),
with a matching test.
- app.css: slight hero-title letter-spacing tightening for Plex at
display sizes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FastAPI 0.137.0 refactored include_router to keep included routers as
nested objects rather than flattening their routes into app.routes, so
test_app_factory's `{route.path for route in app.routes}` no longer found
the /metrics route (the endpoint still serves; only this introspection
broke). FastAPI now treats router.routes as an internal implementation
detail.
Switch the metrics route checks to the public OpenAPI schema
(app.openapi()["paths"]), which is stable across versions and resolves
router prefixes correctly, and drop the <0.137.0 pin that was blocking
the upgrade.
Verified: app factory tests pass on both 0.136.3 and 0.137.2; full
tests/test_api suite (462 tests) passes on 0.137.2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Restrict which remote observers may ingest events, keyed on the observer's
public key (the <public_key> segment of its LetsMesh upload topic). Anyone
with broker access can publish as an observer via JWT auth, so operators can
now gate ingestion.
- New ObserverFilter (case-insensitive prefix matching, allowlist overrides
denylist, accept-all when both empty)
- New OBSERVER_ALLOWLIST / OBSERVER_DENYLIST collector settings, wired through
the CLI, run_collector, create_subscriber, and Subscriber
- Filter applied at the top of _handle_mqtt_message: blocked observers' packets
are dropped before any decode, raw-packet capture, or DB write; zero added
work on the default accept-all path
- Tests: ObserverFilter unit tests, subscriber drop/allow integration tests,
config parsing tests
- Docs: configuration.md, observer.md, upgrading.md (v0.16.0), .env.example,
docker-compose.yml
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Messages page badged a row as spam at a hardcoded score >= 0.6 while the
API hides rows at SPAM_SCORE_THRESHOLD (now 0.65), so messages scored in
[0.6, 0.65) appeared flagged but were never hidden by the "show potential
spam" toggle.
Expose spam_score_threshold in the SPA config and use it for the badge so a
row is badged exactly when the API would hide it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Guard the optional `_spam_rescore_thread` with an `is not None` assert before
calling `.is_alive()`, matching the sibling test. CI runs `pre-commit run
--all-files`, so mypy checks the test files too.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Raise diff/patch coverage above the Codecov gate (was 79.8%, target 80.29%)
by exercising the previously-untested spam paths:
- subscriber: TestSpamRescoreScheduler covers the disabled early-return, the
enabled thread spawning + one sweep + clean stop, and the swallowed-error
branch of the background re-scoring loop.
- handler: a contact-message scoring test covers the contact log branch.
- spam: get/reset_spam_config caching, the zero-weight combine path, the
default-`now` path, and the null-sender reset in rescore_recent.
Patch coverage for the change is now ~97%.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an optional, off-by-default spam-detection feature that scores each
message's spam likelihood at ingest, stores the score on the row, and lets
the display layer hide likely-spam by default behind a "show potential spam"
toggle. Nothing is ever dropped at ingest, so the threshold can be retuned
without reprocessing.
Scoring (collector/spam.py): windowed COUNT(*) over new
(path_prefix, received_at) and (sender_normalized, received_at) indexes —
joint path+sender signal plus a sender-name signal (trailing-digit suffix
stripped so bob1/bob2 collapse to bob). When the path is short/zero-hop or
absent, the name signal stands alone at full weight so local spam is still
flaggable. A background sweep re-scores recent rows with hindsight to catch
the leading edge of bursts. The collector logs each score (WARNING at/above
the threshold).
Display: the messages API gains include_spam and a master-switch-aware
hide-filter; the SPA shows the toggle + a badge only when the feature is on.
Config: FEATURE_SPAM_DETECTION is the single operator switch, bridged in
Compose to the backend SPAM_DETECTION_ENABLED for collector + api (mirrors
the FEATURE_PACKETS / RAW_PACKET_CAPTURE_ENABLED pattern). Both default off.
Works on SQLite and Postgres: DB-agnostic queries, an Alembic batch migration
for the three new columns + two indexes, and backend-aware collector test
fixtures (lifted db_backend/db_url into the shared conftest).
Also: move the meshcore-hub image pull_policy out of the base compose file.
It lived in docker-compose.yml as pull_policy: daily and made `make up` pull
the published image over a freshly built local one. Base is now policy-neutral
(default missing); dev sets pull_policy: build on the hub services so it only
ever uses local builds. Prod refreshes images via a manual `docker compose
... pull`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dashboard charts (activity, message-activity, node-count) rendered as
flat zeros on Postgres because func.date() returns a str on SQLite but
a datetime.date on Postgres — the dict lookup by string key always
missed. Fixed with a dialect-neutral _date_bucket_key() helper and
pinned the Postgres session timezone to UTC at the engine level.
Also adds dual-backend test infrastructure (TEST_DATABASE_BACKEND env
var), per-worker Postgres databases for pytest-xdist isolation, and
strengthened regression tests asserting non-zero date buckets.
After the Postgres migration, nodes with no last_seen timestamp floated
to the top of the default list because Postgres sorts NULLs first under
ORDER BY ... DESC, whereas SQLite (the previous backend) sorts them last.
Wrap the last_seen ORDER BY with SQLAlchemy nullslast() so NULL last_seen
nodes always sink to the end regardless of database backend or sort
direction. Adds three regression tests covering DESC, ASC, and all-NULL
cases.
The SPA reaches the backend via the web api_proxy, which forwarded query
params using dict(request.query_params). dict() on Starlette's QueryParams
multidict keeps only the last value of a repeated key, so a multi-valued
filter like ?observed_by=A&observed_by=B was forwarded to the backend as
observed_by=B only. The backend then filtered to B's events, making a
message observed only by A disappear as soon as B was also selected — the
reported "filters act like AND" symptom. This happens independently of the
Redis response cache.
Forward request.query_params.multi_items() (a list of (key, value) tuples)
so all repeated values reach the backend. Add web proxy regression tests
asserting both observed_by values are forwarded, and capture forwarded
params in the MockHttpClient.
This is the primary fix; the earlier cache-key change (multi_items in
sorted_query_string) addressed the same collapse pattern at the cache layer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The API response cache key was built from request.query_params.items(),
which in Starlette keeps only the last value of a repeated key. Repeated
params like observed_by (?observed_by=A&observed_by=B) therefore collapsed
to observed_by=B in the key, colliding with any other filter set sharing the
same last value. The first response cached under that key was then served
for the whole TTL, making observer-filtered Messages/Adverts return stale,
wrong results (e.g. an A-only message vanishing when B was also enabled).
Use multi_items() so all repeated values are included, sorted by the full
(key, value) tuple so order-independent filter sets map to one key. Add
regression tests covering preservation, order-independence, and the
{A,B} vs {B} collision case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Louis King