Replaces ec40c67c8c83, 6b3430fd84f4 and cf8dd7eaba9b (never deployed
to production) with one migration that builds the final route-health
schema directly on top of the production head 57bb65130b97.
The consolidated migration creates the seven route tables, adds the
nullable event_hash column to raw_packets, and runs three idempotent
backfills: packet_path_hops from raw_packets.decoded, nodes dedup by
public_key, and route_result_history + quality_avg for enabled routes.
On a freshly-restored production snapshot the route-history backfill
is a no-op (no routes exist yet) but is retained for idempotency
against dev backups that do have routes.
Bundles four independent improvements to API caching and route evaluator
accuracy that all touch the same files in the cache/routes stack.
## HTTP Cache-Control headers (api_cache_control_enabled)
The API now emits HTTP Cache-Control on every /api/v1/* response and
handles ETag / If-None-Match -> 304 Not Modified on @cached endpoints.
- @cached GETs: private, max-age=<redis_ttl> + strong SHA-256 ETag.
Matching If-None-Match returns 304 with empty body.
- Uncached GETs under /api/v1/*: private, max-age=0, must-revalidate.
- POST/PUT/DELETE/PATCH and /health*: no-store.
- All private (several @cached endpoints are role-aware).
- @cached decorator stores new envelope {body, etag} in Redis; legacy
bare-JSON entries still read (hashed on the fly) and auto-migrate on
the next miss.
- X-Cache: HIT|MISS observability header is always emitted.
- Kill switch: API_CACHE_CONTROL_ENABLED (default true).
## Dashboard cache TTL bump (30s -> 300s)
REDIS_CACHE_TTL_DASHBOARD default raised from 30s to 300s. Covers all
/dashboard/* endpoints and /api/v1/routes/{id}/history. These return
trend/aggregation data where minute-level staleness is invisible but
every MISS costs seconds of SQL/aggregation. Also resolves reported
short-TTL behaviour on the per-route healthchart (which shares this
setting). Operators with explicit env var: unchanged. Old Redis entries
expire naturally within <=30s; no flush needed.
## Route recent-packets hop truncation
In the routes page expanded card, path hops are now truncated to
2 + ellipsis + 2 when length > 5, matching the packet-detail page
styling. Reuses the existing packets.hops_hidden i18n key as the
tooltip on the ellipsis badge.
## Route event-hash deduplication
A single advert/message/telemetry/trace retransmitted or flooded
through the mesh produces one RawPacket per on-air copy, each with a
fresh wire packet_hash. Counting those as distinct matches let a single
underlying event satisfy packet_count_threshold within seconds and
biased route health.
- Add nullable event_hash column to raw_packets and packet_path_hops
(alembic 6b3430fd84f4). No backfill; legacy rows keep NULL and the
evaluator falls back to wire packet_hash until they age out of
window_hours.
- Subscriber denormalizes the structured event's event_hash onto the
captured raw packet after handler dispatch.
- Route evaluator prefers event_hash when set, collapsing all
receptions of the same underlying event into one match.
Squashes four formerly-separate tail migrations into a single migration
(ec40c67c8c83) that builds the final schema directly off 57bb65130b97:
- 8f2a3c4d5e6f route health tables + packet_path_hops backfill
- c0d1e2f3a4b5 deduplicate nodes by public_key (idempotent data fix)
- 76513f4c57e9 ix_packet_path_hops_received_at index
- 71f6e01e4bf6 rename degraded_threshold -> clear_threshold
The rename runs as a no-op: routes.clear_threshold and
route_results.effective_clear are born with their final names. The
received_at index is created with the table, and the node dedup is
preserved verbatim (no-op on a clean DB).
Also drops the redundant uq_route_results_route_id unique constraint:
route_results.route_id now has a single unique index matching the model,
eliminating a pre-existing autogenerate drift.
Net schema is identical to running all four migrations. Verified with a
fresh alembic upgrade head + alembic check (no drift) on SQLite.
Production DB restore bypassed UNIQUE enforcement on nodes.public_key,
leaving duplicate rows that caused MultipleResultsFound in node lookups
(GET /api/v1/nodes/{public_key} and collector find-or-create handlers).
Migration merges duplicates (winner = earliest first_seen), re-points all
14 FK columns referencing nodes.id, merges scalar fields, and re-creates
the UNIQUE index. Superset of b1c2d3e4f5a6 which only covered 9 FKs.
Route identity is now a (from_label, to_label) composite unique pair
instead of a single name column. The three route migrations (create
tables, add reversible, replace name) are collapsed into one clean
migration since the feature hasn't shipped yet — 8f2a3c4d5e6f creates
all five tables with from_label/to_label + reversible from the start,
no intermediate steps.
Engine: recent_matches now returns the sliced subpath between the
matched endpoints, not the full path. Diagnosis text moved to a hover
tooltip on the quality badge. Cards sorted by from_label. Seed YAML
switched to a list format with from/to keys; upsert by (from_label,
to_label). Plan/tasks docs updated to present the final schema.
Routes now default to bidirectional matching — packets traversing the
configured path in reverse (C->B->A instead of A->B->C) also count
towards health. A 'reversible' toggle lets users constrain a route
to one direction when needed.
- Route model: new 'reversible' column (default true, server_default 1)
- Migration: a1b2c3d4e5f6 (additive ALTER TABLE)
- Matching engine: _match_hops + _fetch_candidate_paths_maybe_bidirectional
helpers shared across evaluate_route, recent_matches, and preview_route
- API schemas: reversible on RouteCreate/Update/Read/Detail/PreviewRequest
- SPA: toggle in modal; card path chips show <-> for reversible, -> otherwise
- Seed loader + example YAML: reads 'reversible' key
- 4 new collector tests (reverse match, non-reversible, both-directions, API)
Add complete route health monitoring feature that tracks whether packets
traverse expected multi-hop paths through the mesh network.
Models & migration:
- 5 new models: PacketPathHop, Route, RouteNode, RouteObserver, RouteResult
- Alembic migration with keyset-paginated backfill of existing packets
Collector:
- store_raw_packet refactored to persist path hops via bulk insert
- Matching engine (collector/routes.py) with subsequence matching, quality
bands (clear/marginal/failing/no_coverage), and collision detection
- Background route evaluator (60s loop) wired into subscriber lifespan
- Route seed loader in CLI (resolves by public_key, matching YAML format)
API:
- 6 CRUD endpoints + preview endpoint under /api/v1/routes
- Schemas accept node_public_keys (64-char hex) instead of internal UUIDs
- 5 Prometheus gauges for route health metrics
- Packet groups endpoint reads from hop table
Web UI:
- Full SPA routes page with summary strip, grouped cards, expandable detail
- Routes nav entry in both desktop (spa.html) and mobile (app.js) navbars
- Home page nav card with feature gate
- API proxy access mapping for v1/routes endpoints
- Visual node-search path builder with autocomplete dropdown, ordered chips
with reorder/remove controls, and paste-64-char-key support
- Observer picker with same search UX (unordered chips)
- i18n strings in en.json and nl.json
Config:
- feature_routes flag (default: true)
- route_evaluator_interval_seconds (default: 60)
- routes_file seed path
- example/seed/routes.yaml
Persist the path-hash byte width (1/2/3) as a nullable Integer column on
RawPacket, computed at ingest by the collector and backfilled for historical
rows via a self-contained Python migration. Replace the per-request Python
decode loop in the grouped-list route with a SQL MAX() aggregate + HAVING
filter, and add a discrete <select> filter (Any/1B/2B/3B) to the /packets
SPA page.
- Model: add path_hash_bytes column to RawPacket (nullable Integer)
- Migration: batch_alter_table add_column + keyset-paginated Python backfill
reading decoded via Core select() on sa.JSON column (portable across
SQLite and Postgres)
- Collector: compute path_hash_bytes at ingest via two-tier path-hash
extraction (decoded.path -> payload.decoded.pathHashes)
- API: add func.max() aggregate to group query, HAVING filter on
?path_hash_bytes=1|2|3 param; delete Phase 3 decode loop and dead helper
- Frontend: add path-width select filter wired through query/apiParams/
pagination/headerParams; add i18n keys (en/nl)
- Tests: 1186 passed, 22 skipped; collector + API + model coverage
Add an optional, off-by-default spam-detection feature that scores each
message's spam likelihood at ingest, stores the score on the row, and lets
the display layer hide likely-spam by default behind a "show potential spam"
toggle. Nothing is ever dropped at ingest, so the threshold can be retuned
without reprocessing.
Scoring (collector/spam.py): windowed COUNT(*) over new
(path_prefix, received_at) and (sender_normalized, received_at) indexes —
joint path+sender signal plus a sender-name signal (trailing-digit suffix
stripped so bob1/bob2 collapse to bob). When the path is short/zero-hop or
absent, the name signal stands alone at full weight so local spam is still
flaggable. A background sweep re-scores recent rows with hindsight to catch
the leading edge of bursts. The collector logs each score (WARNING at/above
the threshold).
Display: the messages API gains include_spam and a master-switch-aware
hide-filter; the SPA shows the toggle + a badge only when the feature is on.
Config: FEATURE_SPAM_DETECTION is the single operator switch, bridged in
Compose to the backend SPAM_DETECTION_ENABLED for collector + api (mirrors
the FEATURE_PACKETS / RAW_PACKET_CAPTURE_ENABLED pattern). Both default off.
Works on SQLite and Postgres: DB-agnostic queries, an Alembic batch migration
for the three new columns + two indexes, and backend-aware collector test
fixtures (lifted db_backend/db_url into the shared conftest).
Also: move the meshcore-hub image pull_policy out of the base compose file.
It lived in docker-compose.yml as pull_policy: daily and made `make up` pull
the published image over a freshly built local one. Base is now policy-neutral
(default missing); dev sets pull_policy: build on the hub services so it only
ever uses local builds. Prod refreshes images via a manual `docker compose
... pull`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The column was declared String(8) but stores 16-char hex signatures (and can
hold the up-to-32-char packet_hash fallback from the LetsMesh normalizer).
SQLite never enforced the length, so the undersized definition went unnoticed
until db migrate-to-postgres rejected the data with
StringDataRightTruncation (varchar(8)). Widen the model column and add an
Alembic migration (batch_alter_table, so it applies on both SQLite and
Postgres). Verified end-to-end migrating a 1.2GB live SQLite DB into Postgres
(all 12 tables reconcile).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- compose: add optional postgres service (postgres:17-alpine, profile
'postgres', healthcheck, postgres_data volume); POSTGRES_* derive from
DATABASE_* (single source of truth). DATABASE_* env added to migrate/
collector/api; migrate depends_on postgres with required:false so SQLite
deployments are unaffected.
- alembic/env: resolve the URL via CommonSettings.effective_database_url so
DATABASE_BACKEND=postgres is honoured (previously DATABASE_URL/DATA_HOME
only -> would silently migrate SQLite).
- migrations: normalize_public_key uses STRING_AGG + HAVING COUNT(*) on
Postgres (was SQLite GROUP_CONCAT + alias); raw_packets uses sa.JSON()
not the sqlite dialect type.
- database: fix _to_async_url to map postgresql+psycopg2:// (what the config
assembles) to asyncpg, so API async sessions work on Postgres; resolve the
search_path schema from DATABASE_SCHEMA env when not passed explicitly.
Validated against a live postgres:17: db upgrade builds all 13 tables in the
meshcorehub schema with correct native types (is_observer boolean, decoded
json) and alembic_version stamped; the upsert, JSON/timestamptz round-trip,
and asyncpg async sessions all work. SQLite suite still green (1061 passed).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- config: add DatabaseBackend enum + DATABASE_* component vars on
CommonSettings (host/port/name/schema/user/password, defaults
'meshcorehub'); single effective_database_url resolver with precedence
DATABASE_URL > postgres (assembled, fail-fast on missing vars) > SQLite
default. effective_database_schema returns the schema only for Postgres.
Collapses the duplicated resolver/field out of Collector/API settings.
- database: create_database_engine/DatabaseManager accept a schema arg and
scope Postgres connections via search_path (psycopg2 -c options; asyncpg
server_settings). No-op for SQLite.
- alembic/env: migrate into the instance schema (version_table_schema +
CREATE SCHEMA + search_path) for Postgres; fix online render_as_batch to
be SQLite-only (matching the offline path).
- .env.example: document the DATABASE_* block.
SQLite behaviour unchanged: default no-env path resolves to the same URL
(new regression tests), full suite green (1051 passed), fresh db upgrade OK.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dialect-neutral fixes that keep SQLite behaviour unchanged:
- event_observer: choose INSERT construct by bind dialect so the
on_conflict upsert works on both SQLite and Postgres
- database: map postgresql:// -> postgresql+asyncpg:// for async sessions
(was sqlite-only), via a _to_async_url helper
- models: import JSON from sqlalchemy (generic) not the sqlite dialect
- alembic/env: render_as_batch only for SQLite (its ALTER TABLE workaround)
Full SQLite test suite green (1045 passed) and fresh db upgrade builds all
tables; black/flake8/mypy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The packet-group detail observer table only ever showed the hop count
because _extract_path_hashes looked at decoded.payload.decoded.pathHashes,
but normal packets carry the routing path at the top level as decoded.path.
Read decoded.path first (falling back to the old pathHashes location).
Frontend: render each hop as its own badge joined by arrows, wrapping on
narrow screens, with centre-truncation for paths over 16 hops. Badges carry
a path-hash-badge class + data-path-hash hook for a future node lookup.
Also expose v1/packet-groups as an open endpoint in the web proxy mapping
(it is not a prefix of v1/packets) and rebase the packet_hash index
migration onto the current head.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- New GroupedPacketRead/PacketReceptionInfo/GroupedPacketList schemas
- GET /api/v1/packet-groups: two-phase GROUP BY query, 7-day default
window, lightweight Phase 2 (no raw_hex/decoded), role-aware cache
- GET /api/v1/packet-groups/{hash}: full reception list with path_hashes
extracted from decoded.payload.decoded.pathHashes
- New (packet_hash, received_at) composite index via Alembic migration
- i18n keys for receptions, path, observer counts
https://claude.ai/code/session_01NH2rZzuHzasJj12SZeRhbJ
Add a first-class Raw Packets feature that captures every inbound MeshCore
packet from the LetsMesh `packets` feed exactly as received, independent of
how the collector later classifies it.
Capture & storage
- New `RawPacket` model + migration (raw_packets table) with single and
composite indexes for the dominant filter-then-sort-by-newest queries.
- Collector-side `RAW_PACKET_CAPTURE_ENABLED` flag (default off); capture hook
reuses the decoder's per-hex cache (no second decode), one row per observer
reception, never blocks event dispatch.
- Separate `RAW_PACKET_RETENTION_DAYS` (falls back to DATA_RETENTION_DAYS);
cleanup runs regardless of capture so disabling drains the table. Raw-packet
observers retained in the is_observer recompute union.
API
- `GET /packets` and `/packets/{id}` with rich filtering, role-aware Redis
cache key, and channel-visibility redaction (restricted-channel packets are
returned metadata-only, not hidden, so pagination counts stay stable).
Web
- `FEATURE_PACKETS` flag (default off). Responsive Packets page (table desktop,
cards mobile) plus a Packet Detail page (breadcrumb nav, raw hex + decoded).
- Nav entry after Messages on all three surfaces; home.js reordered so Map
precedes Members; new packets icon + colour.
Finer-grained classification
- Replace the single `letsmesh_packet` catch-all with per-payload-type event
types (req, ack, encrypted_direct, encrypted_channel, grp_data, multipart,
control, raw_custom, ...); letsmesh_packet kept only as the unresolved-type
safety net.
Link from structured tables
- Add `packet_hash` to advertisements and messages (populated at ingest);
exact `packet_hash` filter on /packets; cube-icon link on the Adverts and
Messages lists -> /packets?packet_hash=<hash>, shown only when the feature is
on and the row has a stored hash.
Docs/config: .env.example, docker-compose (collector + web), AGENTS.md,
SCHEMAS.md, docs/letsmesh.md, docs/upgrading.md (## v0.13.0), en/nl i18n, and a
plan/tasks doc under docs/plans/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three related performance fixes for slow (>500ms) API responses.
1. Stop blocking the event loop. Route handlers were declared `async def`
but ran synchronous SQLAlchemy queries (and synchronous Redis calls via
the cache decorator) directly on the event loop, serializing requests.
Convert all handlers to sync `def` so FastAPI runs them in its
threadpool, and make the `@cached` decorator dual-mode (sync wrapper for
sync handlers, async wrapper preserved for a future async/Postgres path).
2. Tune SQLite for concurrency. Enable WAL, busy_timeout and
synchronous=NORMAL on every connection, and size the pool above the
threadpool so handlers don't wait on connections. In-memory SQLite is
guarded (no overflow-pool kwargs).
3. Precompute an indexed `nodes.is_observer` flag. The `observer=true`
filter scanned ~68k+ event rows to find a handful of observers (the
advertisements page calls it with limit=500). Replace the 5-way OR of
subqueries with `WHERE nodes.is_observer = ?`. The collector sets the
flag on first observation (in add_event_observer); the cleanup job
clears it once a node's events are all pruned; a migration adds the
column/index and backfills from the existing union.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Rename ChannelVisibility.PUBLIC to ChannelVisibility.COMMUNITY
- Update stored value from 'public' to 'community' across model, schema, API, CLI, and frontend
- Add Alembic migration to update existing database rows
- Consolidate upgrade docs: merge v0.11.0, v0.12.0, v0.13.0 into single v0.11.0 section
- Add i18n visibility level translation keys (en, nl)
- Update section headings on channels page to use t() for i18n
- Keep visibility badges lowercase per UI design
Replaces env-var channel keys with a Channel database model and periodic
DB refresh in the collector. Adds Channels dashboard page with QR codes,
channel visibility filtering on messages/dashboard APIs, and channel card
navigation to filtered messages view.
Track advertisement route type (flood/transport_flood/direct/transport_direct)
and node advert timestamp to distinguish zero-hop from flood adverts, improve
deduplication with 300s buckets, and default all dashboard/ad-API queries to
flood-only (including NULL for historical records).
Replace the role=infra NodeTag convention with UserProfileNode adoption
as the canonical infrastructure indicator across map, Prometheus metrics,
and alerting. Renames is_infra to is_adopted, infra_center to
adopted_center. Map icons change to blue (adopted) / green (normal),
with all adoption UI gated on OIDC_ENABLED. Adds meshcore_nodes_adopted
gauge and Alembic migration to clean up obsolete tags.
- Add description (Text) and url (String 2048) columns to user_profiles
- Expose in all API schemas (Read, Public, Update, ListItem) and list/get/profile endpoints
- Update profile.js form: add description/url inputs, render on view page
- Update members.js: render description and URL link in member tiles
- Fix update handler: use model_dump(exclude_unset=True) for nullable fields
while protecting name (set by IdP) from being cleared
- AnyUrl validation on update, converted to str for SQLite compatibility
- Add i18n keys (description_label/placeholder, url_label/placeholder)
- 7 new API tests covering description/url CRUD, URL validation, null-clearing,
and name non-nullability
Remove the static Member model/table, CRUD API, YAML seed files, and
admin UI. Replace with UserProfile-driven members page that reads roles
from OIDC identity provider. Key changes:
- Drop members table, add roles column to user_profiles (Alembic migration)
- Add GET /api/v1/user/profiles (paginated, no user_id exposed)
- Add GET /api/v1/user/profile/me (auto-creates profile for current user)
- Replace member_id node tag filter with adopted_by (profile UUID)
- Members page now shows profiles grouped by operator/member roles
- Profile page supports public view (/profile/:id) and owner edit (/profile)
- Node detail page shows adoption card side-by-side with public key card
- Auto-create user profile during OIDC login callback
- Hide Adopted Nodes section for non-operator/admin users
- Add member since date to profile cards
- Add role badges and adopted node badges to member tiles
- Add antenna/users icons to Members page group headers
- Add path_len column to event_observers table with Alembic migration
- Create shared observer_utils.py for building observer responses
- Extract SNR and path_len from LetsMesh uploads via normalizer
- Pass snr/path_len from all 4 handlers to add_event_observer()
- Add expandable observer detail sub-tables to messages and ads pages
- Rename Receivers column to Observers with updated i18n key
- Add satellite dish icon to observer badges and detail rows
- Add mobile card observer detail toggle
- Show full datetime tooltip on hover for relative times
- Hide Path column from advertisement observer tables (not populated)
The LetsMesh normalizer stored public keys as UPPERCASE while the tag
importer stored them as lowercase, creating duplicate nodes for the same
device. Normalize all public keys to lowercase throughout:
- MQTT topic parsing (event, command, LetsMesh upload)
- LetsMesh normalizer output
- Node model __init__ enforcement
- Alembic migration to merge duplicates and normalize existing data
Remove the meshcore_interface component in favor of external
meshcore-packet-capture for data ingestion. Rename receiver_node_id
to observer_node_id across all models, schemas, handlers, and API
routes. Add Alembic migration for the column/table renames. Fix
frontend JS property name mismatch that prevented the Receiver column
from displaying observer data.
Pin starlette<1.0.0 to avoid TemplateResponse breaking change, update
the TemplateResponse call to new-style request-first signature, and
bump Python version to 3.14 across Dockerfile, pyproject.toml, and
pre-commit hooks.
Replace presentation-layer deduplication with collector-level approach:
- Add event_hash column to messages, advertisements, trace_paths, telemetry tables
- Handlers compute content hashes and skip duplicate events at insertion time
- Use 5-minute time buckets for advertisements and telemetry
- Include Alembic migration for schema changes
Members can now have multiple associated nodes, each with a public_key
and node_role (e.g., 'chat', 'repeater'). This replaces the single
public_key field on members with a one-to-many relationship.
Changes:
- Add MemberNode model for member-node associations
- Update Member model to remove public_key, add nodes relationship
- Update Pydantic schemas with MemberNodeCreate/MemberNodeRead
- Update member_import.py to handle nodes list in seed files
- Update API routes to handle nodes in create/update/read operations
- Add Alembic migration to create member_nodes table and migrate data
- Update example seed file with new format
- Add Member database model with name, callsign, role, description, contact, and public_key fields
- Add Member Pydantic schemas (MemberCreate, MemberUpdate, MemberRead, MemberList)
- Add members table to initial migration
- Add members API endpoints (GET/POST/PUT/DELETE /api/v1/members)
- Add member_import.py for importing from JSON files
- Update web layer to fetch members from API instead of file
- Add SEED_HOME setting (defaults to ./seed) for seed data files
- Add 'collector seed' command to import node_tags.json and members.json
- Rename tags.json to node_tags.json for consistency
- Move example seed data from example/data/* to example/seed/
- Update tests and configuration
This commit establishes the complete foundation for the MeshCore Hub project:
- Project setup with pyproject.toml (Python 3.11+, all dependencies)
- Development tools: black, flake8, mypy, pytest configuration
- Pre-commit hooks for code quality
- Package structure with all components (interface, collector, api, web)
Common package includes:
- Pydantic settings for all component configurations
- SQLAlchemy models for nodes, messages, advertisements, traces, telemetry
- Pydantic schemas for events, API requests/responses, commands
- MQTT client utilities with topic builder
- Logging configuration
Database infrastructure:
- Alembic setup with initial migration for all tables
- Database manager with session handling
CLI entry point:
- Click-based CLI with subcommands for all components
- Database migration commands (upgrade, downgrade, revision)
Tests:
- Basic test suite for config and models
- pytest fixtures for in-memory database testing
Louis King