Routes now default to bidirectional matching — packets traversing the
configured path in reverse (C->B->A instead of A->B->C) also count
towards health. A 'reversible' toggle lets users constrain a route
to one direction when needed.
- Route model: new 'reversible' column (default true, server_default 1)
- Migration: a1b2c3d4e5f6 (additive ALTER TABLE)
- Matching engine: _match_hops + _fetch_candidate_paths_maybe_bidirectional
helpers shared across evaluate_route, recent_matches, and preview_route
- API schemas: reversible on RouteCreate/Update/Read/Detail/PreviewRequest
- SPA: toggle in modal; card path chips show <-> for reversible, -> otherwise
- Seed loader + example YAML: reads 'reversible' key
- 4 new collector tests (reverse match, non-reversible, both-directions, API)
Add complete route health monitoring feature that tracks whether packets
traverse expected multi-hop paths through the mesh network.
Models & migration:
- 5 new models: PacketPathHop, Route, RouteNode, RouteObserver, RouteResult
- Alembic migration with keyset-paginated backfill of existing packets
Collector:
- store_raw_packet refactored to persist path hops via bulk insert
- Matching engine (collector/routes.py) with subsequence matching, quality
bands (clear/marginal/failing/no_coverage), and collision detection
- Background route evaluator (60s loop) wired into subscriber lifespan
- Route seed loader in CLI (resolves by public_key, matching YAML format)
API:
- 6 CRUD endpoints + preview endpoint under /api/v1/routes
- Schemas accept node_public_keys (64-char hex) instead of internal UUIDs
- 5 Prometheus gauges for route health metrics
- Packet groups endpoint reads from hop table
Web UI:
- Full SPA routes page with summary strip, grouped cards, expandable detail
- Routes nav entry in both desktop (spa.html) and mobile (app.js) navbars
- Home page nav card with feature gate
- API proxy access mapping for v1/routes endpoints
- Visual node-search path builder with autocomplete dropdown, ordered chips
with reorder/remove controls, and paste-64-char-key support
- Observer picker with same search UX (unordered chips)
- i18n strings in en.json and nl.json
Config:
- feature_routes flag (default: true)
- route_evaluator_interval_seconds (default: 60)
- routes_file seed path
- example/seed/routes.yaml
Add F8/T9 + Phase 8 seed loader (links.yaml via existing meshcore-hub seed command, mirroring channels.yaml), referencing path nodes by public_key with expected_hash derived by the importer. Refine visibility to keep auth-scope but default to community (like channels).
Plan for a new Link entity that monitors whether packets traverse a configured ordered sequence of mesh nodes within a time window + count threshold, with background evaluation and Prometheus exposure. Includes shareable plain-language overview.
When an IdP name claim contains trailing whitespace (e.g. 'Matt '), every
proxied API request failed with 502 'Illegal header value' because httpx
enforces RFC 7230 which forbids leading/trailing OWS in header values.
Defense in depth:
- strip_userinfo() trims the IdP name at ingress (oidc.py)
- _sanitize_header_value() removes CTL/DEL chars at both header-injection
sites (API proxy + auth-callback bootstrap) in app.py
- update_profile() trims user-supplied names in PUT handler
X-User-Name is informational only (seeds non-unique UserProfile.name);
identity/auth are keyed on X-User-Id and X-User-Roles, so sanitization is
safe.
Add two horizontal 100% stacked-bar charts to the Dashboard, gated
behind the existing packets feature flag:
- Packet Types: top 6 event_type values + 'other' rollup
- Path Bytes: 1b/2b/3b path-hash width distribution (NULL excluded)
Driven by a single new cached endpoint
GET /api/v1/dashboard/packet-breakdown?days=7 returning pre-bucketed
counts. Frontend normalizes to percentages via a new
createStackedBarChart helper in charts.js with a 7-hue oklch palette.
The popover used position: fixed with viewport-relative coordinates, so
it stayed pinned to the viewport while the page scrolled away beneath
it. Switch to absolute positioning anchored to the document (adding
scrollX/scrollY) and read the badge's live rect on each reposition so
it tracks the anchor after the async node fetch resolves.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Charts jumped from a single column straight to N-across at the md
breakpoint, cramming them on landscape phones. Introduce a 2-column
layout at the sm breakpoint (2x2 for 4 charts; 3 charts flow to
2-on-top + 1 bottom-left with a gap on the right) and defer the full
N-across row to lg.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Persist the path-hash byte width (1/2/3) as a nullable Integer column on
RawPacket, computed at ingest by the collector and backfilled for historical
rows via a self-contained Python migration. Replace the per-request Python
decode loop in the grouped-list route with a SQL MAX() aggregate + HAVING
filter, and add a discrete <select> filter (Any/1B/2B/3B) to the /packets
SPA page.
- Model: add path_hash_bytes column to RawPacket (nullable Integer)
- Migration: batch_alter_table add_column + keyset-paginated Python backfill
reading decoded via Core select() on sa.JSON column (portable across
SQLite and Postgres)
- Collector: compute path_hash_bytes at ingest via two-tier path-hash
extraction (decoded.path -> payload.decoded.pathHashes)
- API: add func.max() aggregate to group query, HAVING filter on
?path_hash_bytes=1|2|3 param; delete Phase 3 decode loop and dead helper
- Frontend: add path-width select filter wired through query/apiParams/
pagination/headerParams; add i18n keys (en/nl)
- Tests: 1186 passed, 22 skipped; collector + API + model coverage
- Add reusable lit-html JSON tree component (json-tree.js) with
expand/collapse-all toolbar and type-coloured primitives; replace
flat <pre> decoded blocks on packet detail and packet group detail
- Render packet path as a complete flow with static sender (green dot)
and observer (satellite dish) terminators around hop badges
- Fill homepage chart areas and move legend to top-right to match
dashboard charts
- Add iconChevronRight helper and expand_all/collapse_all strings (en, nl)
Replace raw integer counts with Intl.NumberFormat()-grouped numbers
across all SPA pages — stat cards, dashboard stats, list-page total
badges, inline reception/observer counts, chart axis ticks and
tooltips, map counts, and packet-group-detail fields. Formatting uses
the visitor's browser locale (no explicit locale argument), decoupled
from the admin's datetime_locale.
Redesign the filter panel on all five filter-bearing pages (nodes,
packets, advertisements, messages, map): replace the heavy DaisyUI
collapse card with a compact right-aligned toggle slider plus bare
filter fields rendered below the control row. Filter open-state
survives auto-refresh and navigation via DOM-read of #filter-toggle.
Also fixes a pre-existing bug where pubkey_prefix was missing from
nodes.js hasActiveFilters, causing the filter to not default open
when only the public-key-prefix field was filled.
Recent adverts card now shows route-type badges, an observer column
with three-way fallback (observers -> observed_by -> dash), and renders
all 10 rows instead of 5. The Type column hides on mobile to prevent
overflow, and the misleading cursor-help on observer badges is removed.
Also adds a Packets chart card to the dashboard and a packets stat to
the homepage, backed by the new dashboard packet-activity endpoint.
Promotes routeTypeBadge to a shared export in components.js (previously
local to advertisements.js), and enriches RecentAdvertisement with
route_type, observers, and observed_by fields.
Backend resolves observers via fetch_observers_for_events and
observer_node_id -> public_key in batched queries.
The two-column mobile nav set display:block on #mobile-nav (an ID
selector) to override .menu's display:flex so column-count worked.
That ID specificity also beat DaisyUI's closed-state display:none
rule, so the menu stayed display:block while 'closed' (only opacity
dropped to 0). The invisible, two-column-wide overlay captured clicks
over underlying page content and fired SPA navigation.
Scope display:block (and the column layout) to the dropdown's open
state (.dropdown-open / :focus-within) so DaisyUI's display:none can
hide the menu and its links when closed.
- Replace panel-glow radial gradient with panel-accent flat tint + 5px
colored left border for section identity
- Add theme-aware --panel-tint-strength/--panel-tint-bg variables: dark
mode keeps 8% colored tint, light mode uses neutral grey fill (0%)
- Remove inline section color from stat-value numbers so they inherit
base-content (white/black) while icons and borders keep section colors
- Fix hero layout: center welcome text vertically, drop spacer div,
add flex-col to content wrapper
- Split mobile nav dropdown into two balanced columns (column-count: 2)
with responsive width to reduce vertical height on phones
- Soften flash banner in light mode with direct amber oklch values to
avoid hue drift from low-chroma color-mix
- Boost stat-title/stat-desc opacity from 60% to 80% in light mode for
better contrast on grey panels
- Dashboard chart cards now use section colors instead of neutral,
subtitles bumped from opacity-70 to opacity-80
Replace the OS system font stack with IBM Plex: the variable-weight
sans (100-700, one file per subset) for UI and headings, and Plex Mono
400 for the public keys, packet hashes, and hex that font-mono renders
across the app. Latin + latin-ext subsets only (shipped locales are
en/nl); no italics; mono is never rendered bold here.
Fonts are self-hosted from @fontsource packages via the existing
build.js vendor pipeline (no CDN), copied to static/vendor/fonts/ with
a hard build failure on wrong filenames. Wiring:
- input.css: @theme --font-sans/--font-mono + @font-face rules with
unicode-ranges taken verbatim from the package CSS; Tailwind v4
derives the document default from --font-sans, re-fonting daisyUI
components with no other changes.
- spa.html: preload the latin sans variable woff2 (crossorigin, URL
identical to the @font-face src) to minimize FOUT.
- charts.js: Chart.defaults.font.family to match (Chart.js otherwise
uses Helvetica/Arial).
- error.html: name-prepend only; the page stays dependency-free.
- middleware.py: long-term immutable cache for /static/vendor/fonts/
(stable names referenced from CSS, so ?v= versioning can't apply),
with a matching test.
- app.css: slight hero-title letter-spacing tightening for Plex at
display sizes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The channel cards, member node badges, and member URL text are
clickable divs/spans, so keyboard users could not reach or activate
them. Add role/tabindex, Enter/Space key handling, and a
focus-visible outline (invisible to mouse users). daisyUI btn/link/
input components already ship focus-visible outlines, so only these
custom elements needed it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Page <h1>s: text-3xl font-bold everywhere (packet-detail,
packet-group-detail, channels were text-2xl; channels icon bumped to
match).
- Page-header rows: mb-6 on all pages (list pages used mb-4).
- Empty states: text-center py-8 opacity-70 (members used py-12,
channels py-10 opacity-60).
- QR backings and adopted-node rows: rounded-box like other
panel-level surfaces; QR padding unified at p-2.
- Home hero welcome text: max-w-[90%] sm:max-w-[70%] so narrow screens
don't wrap into a skinny column (desktop unchanged).
- Document the styling conventions in the components.js header so
future pages don't drift.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Zero-visual-delta cleanups:
- Drop input-bordered/select-bordered (removed in daisyUI v5; emitted
no CSS) from all inputs and selects.
- Rename bare `shadow` to `shadow-sm` — in Tailwind v4 bare `shadow` is
a deprecated alias with the identical value.
- Hoist the hardcoded map marker hex colors into CSS variables in the
app.css palette block (same values; markers sit on map tiles and stay
theme-independent by design).
- Convert the five text-base-content/* outliers to the repo's dominant
opacity-* muted-text idiom (identical rendering on plain text).
- Remove a dead ternary in renderNodeDisplay and an unused iconLock
import.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The channels modal still used daisyUI v4's label-text class, which was
removed in v5, so its form labels rendered unstyled. Replace with the
Tailwind equivalent of what v4 produced (text-sm, muted).
.prose a:hover used --color-primary-content (the on-primary foreground),
making hovered markdown links nearly invisible against the page
background. Blend primary toward base-content instead so the hover
shade works in both themes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FastAPI 0.137.0 refactored include_router to keep included routers as
nested objects rather than flattening their routes into app.routes, so
test_app_factory's `{route.path for route in app.routes}` no longer found
the /metrics route (the endpoint still serves; only this introspection
broke). FastAPI now treats router.routes as an internal implementation
detail.
Switch the metrics route checks to the public OpenAPI schema
(app.openapi()["paths"]), which is stable across versions and resolves
router prefixes correctly, and drop the <0.137.0 pin that was blocking
the upgrade.
Verified: app factory tests pass on both 0.136.3 and 0.137.2; full
tests/test_api suite (462 tests) passes on 0.137.2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Restrict which remote observers may ingest events, keyed on the observer's
public key (the <public_key> segment of its LetsMesh upload topic). Anyone
with broker access can publish as an observer via JWT auth, so operators can
now gate ingestion.
- New ObserverFilter (case-insensitive prefix matching, allowlist overrides
denylist, accept-all when both empty)
- New OBSERVER_ALLOWLIST / OBSERVER_DENYLIST collector settings, wired through
the CLI, run_collector, create_subscriber, and Subscriber
- Filter applied at the top of _handle_mqtt_message: blocked observers' packets
are dropped before any decode, raw-packet capture, or DB write; zero added
work on the default accept-all path
- Tests: ObserverFilter unit tests, subscriber drop/allow integration tests,
config parsing tests
- Docs: configuration.md, observer.md, upgrading.md (v0.16.0), .env.example,
docker-compose.yml
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The spam-detection change wrapped the message table cell content in a
multi-line div/span so the spam badge could sit alongside the text, but left
white-space: pre-wrap on the td. The template literal's own indentation
(newlines + spaces around the div) then rendered as literal whitespace inside
the pre-wrap cell, padding out every row regardless of spam state.
Move pre-wrap onto the span that holds the message text so multi-line bodies
still wrap while the cell's layout whitespace collapses normally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make FEATURE_SPAM_DETECTION on by default with opt-out, mirroring
FEATURE_PACKETS: flip the web feature flag's Python default to true and the
Compose substitutions (collector/api/web) to :-true, so the shipped stack
scores and hides likely-spam without configuration. Opt out with
FEATURE_SPAM_DETECTION=false.
Update .env.example, docs/configuration.md and the v0.15 upgrade notes to
describe the feature as enabled-by-default with opt-out.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Messages page badged a row as spam at a hardcoded score >= 0.6 while the
API hides rows at SPAM_SCORE_THRESHOLD (now 0.65), so messages scored in
[0.6, 0.65) appeared flagged but were never hidden by the "show potential
spam" toggle.
Expose spam_score_threshold in the SPA config and use it for the badge so a
row is badged exactly when the API would hide it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adjust the default spam-scoring knobs across Python settings, SpamConfig,
docker-compose, .env.example and docs to reduce false positives on chatty
legitimate users:
SPAM_MIN_PATH_HOPS 5 -> 3
SPAM_PATH_THRESHOLD 5 -> 6
SPAM_NAME_THRESHOLD 5 -> 10
SPAM_WEIGHT_PATH 0.7 -> 0.75
SPAM_WEIGHT_NAME 0.3 -> 0.25
SPAM_SCORE_THRESHOLD 0.6 -> 0.65
Also document the spam-detection feature and the pull_policy change in a
new v0.15.0 section of docs/upgrading.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Guard the optional `_spam_rescore_thread` with an `is not None` assert before
calling `.is_alive()`, matching the sibling test. CI runs `pre-commit run
--all-files`, so mypy checks the test files too.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Raise diff/patch coverage above the Codecov gate (was 79.8%, target 80.29%)
by exercising the previously-untested spam paths:
- subscriber: TestSpamRescoreScheduler covers the disabled early-return, the
enabled thread spawning + one sweep + clean stop, and the swallowed-error
branch of the background re-scoring loop.
- handler: a contact-message scoring test covers the contact log branch.
- spam: get/reset_spam_config caching, the zero-weight combine path, the
default-`now` path, and the null-sender reset in rescore_recent.
Patch coverage for the change is now ~97%.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an optional, off-by-default spam-detection feature that scores each
message's spam likelihood at ingest, stores the score on the row, and lets
the display layer hide likely-spam by default behind a "show potential spam"
toggle. Nothing is ever dropped at ingest, so the threshold can be retuned
without reprocessing.
Scoring (collector/spam.py): windowed COUNT(*) over new
(path_prefix, received_at) and (sender_normalized, received_at) indexes —
joint path+sender signal plus a sender-name signal (trailing-digit suffix
stripped so bob1/bob2 collapse to bob). When the path is short/zero-hop or
absent, the name signal stands alone at full weight so local spam is still
flaggable. A background sweep re-scores recent rows with hindsight to catch
the leading edge of bursts. The collector logs each score (WARNING at/above
the threshold).
Display: the messages API gains include_spam and a master-switch-aware
hide-filter; the SPA shows the toggle + a badge only when the feature is on.
Config: FEATURE_SPAM_DETECTION is the single operator switch, bridged in
Compose to the backend SPAM_DETECTION_ENABLED for collector + api (mirrors
the FEATURE_PACKETS / RAW_PACKET_CAPTURE_ENABLED pattern). Both default off.
Works on SQLite and Postgres: DB-agnostic queries, an Alembic batch migration
for the three new columns + two indexes, and backend-aware collector test
fixtures (lifted db_backend/db_url into the shared conftest).
Also: move the meshcore-hub image pull_policy out of the base compose file.
It lived in docker-compose.yml as pull_policy: daily and made `make up` pull
the published image over a freshly built local one. Base is now policy-neutral
(default missing); dev sets pull_policy: build on the hub services so it only
ever uses local builds. Prod refreshes images via a manual `docker compose
... pull`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move scattered configuration tables and operational sections out of the
README into dedicated reference documents:
- docs/configuration.md: single source of truth for all environment
variables, grouped into 12 sections (Common, Database, Caching,
Collector, Webhooks, Auth, Data Retention, API, Web Dashboard,
Feature Flags, Traefik, Prometheus & Alertmanager)
- docs/deployment.md: production setup, reverse proxy, multi-instance,
API scaling, Redis caching
- docs/observer.md: remote observers plus PACKETCAPTURE_* and
SERIAL_PORT reference
- docs/maintenance.md: backup and restore
README is reduced from 712 to 385 lines; the ARM32/Raspberry Pi note
is dropped. database.md, auth.md, webhooks.md, and content.md have
their env-var tables removed and link back to configuration.md. Stale
cross-references in database.md, upgrading.md, and .env.example are
updated to point at the new locations.
Remove paths-ignore from the pull_request trigger so the CI workflow
always runs and the required Lint/Test/Build Package checks report on
docs-only PRs (previously skipped entirely, blocking merges).
Drop the github.event_name == 'push' gate on the build job so the
required Build Package check also reports on pull requests.
Replace the lingering v0.9 'Breaking Changes' alert with a concise v0.14
'DEPRECATION NOTICE' for SQLite (dual compatibility for ~3 months, then
PostgreSQL-only).
Move all database-specific instructions (SQLite + PostgreSQL) out of the
README into a new canonical docs/database.md covering:
- SQLite zero-config default (DATA_HOME / meshcore.db, WAL/single-host)
- PostgreSQL: DATABASE_* env vars, bundled Docker profile, production
role/database provisioning (mirrors the ipnet-mesh/infrastructure init
script), managed/external Postgres and DATABASE_URL
- Schema-per-instance (search_path) isolation for multiple instances on a
shared cluster
- Pointer to the SQLite->PostgreSQL migration runbook in upgrading.md
Update the README Multi-Instance Deployments and Scaling the API sections
to link to docs/database.md, and add the new doc to the docs list and
project tree. Add a pointer in .env.example and a Postgres note in
AGENTS.md.
Consolidate docs/upgrading.md v0.14: move the env-var/schema/provisioning
reference to docs/database.md (single source of truth) and keep only the
upgrade-time migration runbook and dashboard chart fix.
The packets feature was the only flag gated with a strict `=== true` check
in the SPA JavaScript while every other flag uses `!== false`. Align packets
to the shared pattern across route registration, mobile nav, page titles, the
home nav card, and the messages/advertisements packet-detail link gates.
Dashboard charts (activity, message-activity, node-count) rendered as
flat zeros on Postgres because func.date() returns a str on SQLite but
a datetime.date on Postgres — the dict lookup by string key always
missed. Fixed with a dialect-neutral _date_bucket_key() helper and
pinned the Postgres session timezone to UTC at the engine level.
Also adds dual-backend test infrastructure (TEST_DATABASE_BACKEND env
var), per-worker Postgres databases for pytest-xdist isolation, and
strengthened regression tests asserting non-zero date buckets.
After the Postgres migration, nodes with no last_seen timestamp floated
to the top of the default list because Postgres sorts NULLs first under
ORDER BY ... DESC, whereas SQLite (the previous backend) sorts them last.
Wrap the last_seen ORDER BY with SQLAlchemy nullslast() so NULL last_seen
nodes always sink to the end regardless of database backend or sort
direction. Adds three regression tests covering DESC, ASC, and all-NULL
cases.
Add actions/delete-package-versions@v5 to both Docker build pipelines to
clean up orphaned untagged manifests left behind by multi-arch rebuilds.
Uses delete-only-untagged-versions so tagged versions (v*, latest, main,
sha-*) are never deleted, with min-versions-to-keep: 10 to protect the
freshly-pushed image's own untagged child/attestation manifests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The SPA reaches the backend via the web api_proxy, which forwarded query
params using dict(request.query_params). dict() on Starlette's QueryParams
multidict keeps only the last value of a repeated key, so a multi-valued
filter like ?observed_by=A&observed_by=B was forwarded to the backend as
observed_by=B only. The backend then filtered to B's events, making a
message observed only by A disappear as soon as B was also selected — the
reported "filters act like AND" symptom. This happens independently of the
Redis response cache.
Forward request.query_params.multi_items() (a list of (key, value) tuples)
so all repeated values reach the backend. Add web proxy regression tests
asserting both observed_by values are forwarded, and capture forwarded
params in the MockHttpClient.
This is the primary fix; the earlier cache-key change (multi_items in
sorted_query_string) addressed the same collapse pattern at the cache layer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The API response cache key was built from request.query_params.items(),
which in Starlette keeps only the last value of a repeated key. Repeated
params like observed_by (?observed_by=A&observed_by=B) therefore collapsed
to observed_by=B in the key, colliding with any other filter set sharing the
same last value. The first response cached under that key was then served
for the whole TTL, making observer-filtered Messages/Adverts return stale,
wrong results (e.g. an A-only message vanishing when B was also enabled).
Use multi_items() so all repeated values are included, sorted by the full
(key, value) tuple so order-independent filter sets map to one key. Add
regression tests covering preservation, order-independence, and the
{A,B} vs {B} collision case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The org.opencontainers.image.source label was pointing at
meshcore-dev/meshcore-hub (the upstream MeshCore project's org)
instead of this repo's canonical location at ipnet-mesh/meshcore-hub.
Verify via: docker inspect ghcr.io/ipnet-mesh/meshcore-hub:main
Add concurrency (PR-cancel only), dependency caching, timeouts, and
path filters across all four workflows. Pin opencode action to v1.17.7
and tighten the /oc trigger. Skip MQTT broker rebuild when upstream
SHA is unchanged. Gate sdist/wheel build job to main-only pushes.
Louis King