Remove the upgrade-compatibility items scheduled for v0.20 in the v0.19
release notes:
- the `meshcore-hub db migrate-to-postgres` command and its
`common/db_migrate.py` module (with its dedicated tests)
- the `database_backend` settings field and its legacy-backend validator
(a leftover DATABASE_BACKEND is now silently ignored)
- the `[postgres]` pip extra alias (Dockerfile installs `.` now)
`tests/test_main.py` is rewritten as a minimal CLI smoke test to keep
entry-point coverage. Docs updated: v0.20.0 upgrade section notes the
removals (SQLite deployments must upgrade through v0.19 first), README
and database.md pointers reworded, stale .env.example comment dropped.
Redis becomes mandatory infrastructure (PostgreSQL-parity): the
REDIS_ENABLED flag, NullCache backend, and every conditional caching
code path are removed. /health/ready reports not_ready (503) when
Redis is unreachable and cached endpoints return 503 instead of
silently degrading; invalidation helpers never raise after a
committed write.
The bundled redis container joins the core profile only via
docker-compose.dev.yml (production points REDIS_* at an external
instance, same no-shadowing pattern as postgres), the backend suite
runs against real Redis (test-redis on 127.0.0.1:55433, CI redis:8
service container, per-xdist-worker key prefixes), and operator docs
describe the required-dependency setup (upgrading.md gains the
v0.20.0 section).
Marker clustering changed the map DOM contract the spec relied on:
clustered and off-viewport markers are absent, so raw .map-marker counts
undercount and first-marker clicks can hit-test against the sticky
navbar/banners. Count nodes as individual markers plus cluster badge
totals (new data-testid on the badge), zoom into a cluster before the
label and popup interactions so targets sit mid-map, and assert cluster
clicks spread children into individual markers.
Wrap MapPage node markers in react-leaflet-cluster (leaflet.markercluster)
with spiderfy-at-max-zoom, zoom-to-bounds on click, clustering disabled at
zoom >= 15, and viewport culling so off-screen markers stay out of the DOM.
Cluster bubbles render a plain themed count badge. Marker label HTML is now
only built when the show-labels toggle is on, halving per-marker DOM weight;
hover-to-reveal no longer applies while labels are toggled off.
- channels: post-delete card count is 6 (6 seeded + 1 created - 1), not 5
- empty-states/pagination: the empty text renders twice (mobile
EmptyState + desktop EmptyRow); scope assertions to the table cell
- mobile: list-row is desktop-only; assert the mobile sort select and
card text instead
- not-found: '/$' is a glob, not a regex; assert the exact URL '/'
- packets-detail: the newest seeded packet is the ce01 message, not an
advert; assert the event type returned by the API
Grow the suite from 41 to 77 specs covering previously untested UI
surfaces: channels admin CRUD with QR codes and keyboard nav, node
detail adoption/tag editing with the role permission matrix, SPA 404
page, sortable tables with URL-synced sort state, pagination with
filter-preserving links, mobile viewport (hamburger nav, mobile cards,
sort select), anonymous/member auth gating, raw packet detail, and
empty states. Route cards now assert history chart, recent-match links
and owner profile links.
Supporting changes: purposeful data-testids on channels/node-detail/
pagination/sort headers (plus aria-sort) and mobile nav; seed_data adds
30 trace_data filler groups for pagination, channels at every
visibility tier incl. one disabled, and a pw-operator profile.
Add a summary job that runs after all CI jobs (even on failure),
parses the pytest/vitest/Playwright JUnit reports plus coverage.xml,
and renders a consolidated Markdown summary to $GITHUB_STEP_SUMMARY:
job results, test counts, durations, backend coverage, failure
details, and the artifact inventory. Enable a Playwright JUnit
reporter and include it in the playwright-report artifact.
Starting core with docker-compose.prod.yml also started the bundled
postgres container, whose 'postgres' network alias shadowed a shared
instance of the same name on the same Docker network — the live site
silently talked to an empty local database instead of DATABASE_HOST.
- Base file: postgres profiles are now [all, postgres] (core dropped);
the dev override unions 'core' back in, so the documented dev
workflow (make up, --profile core) is unchanged
- migrate's postgres dependency is required: false — still health-gated
when the bundled container is active, skipped when profile-inactive
- prod override documents the behavior; opt in explicitly with
--profile postgres (or --profile all) for a bundled production DB
- Docs: database.md/deployment.md/configuration.md/README/.env.example
updated; upgrading.md gains a one-time cleanup runbook (full down to
remove the previously started hub-postgres container)
- /health/ready returns 503 (not 200) when not ready; body no longer leaks
raw backend error text (api + web tiers)
- SYSTEM_MAINTENANCE now gates the web /api/* proxy (503 MAINTENANCE)
instead of only hiding the SPA UI
- channel mutations invalidate messages/packets/packet_groups/dashboard
caches so visibility/key changes apply immediately (was stale up to 1h)
- async DB engine: pool_size/max_overflow/pre_ping matching the sync
engine (survives Postgres restarts); adispose() awaited from the API
lifespan (async connections were leaked by the sync dispose)
- webhooks: permanent 4xx (except 408/429) fail fast instead of retrying;
dispatch queue bounded at 1000 with drop-oldest + warning; shutdown
logs dropped events; matches_event evaluated once
- SPA: apiGet/apiPostForm redirect to OIDC login on 401; chart date
labels and route timestamps rendered in UTC/configured timezone
- e2e compose collector healthcheck uses 'meshcore-hub health collector'
(image has no pgrep)
- docs/upgrading.md: fold unreleased notes into v0.18.0 and document the
above
- api CLI single-process path now passes spam_detection_enabled /
spam_score_threshold to create_app() so the messages hide-filter
actually runs in the default (Docker) deployment
- OIDC role names (oidc_role_admin/operator/member/test) moved from
WebSettings to CommonSettings and wired onto API app.state; the API
tier previously matched hardcoded role names, ignoring customized
OIDC_ROLE_* config in authz, channel visibility, and tag/adoption
checks (oidc_roles_claim stays web-only)
- /metrics handler is now sync def so collection runs in the
threadpool instead of blocking the event loop; hoisted fragile
in-loop imports
- evaluate_route_history gates max_path_length on the packet's true
hop count (RawPacket.path_len) instead of the prefix-filtered hop
list, so history day buckets agree with snapshot evaluation
- reload_keys clears the decode cache: negatively-cached packets for
a newly added channel key were permanently dropped by the normalizer
Also documents 'sudo npx playwright install-deps chromium' in the
AGENTS.md e2e setup.
- /map/data: resolve viewer OIDC session and apply the endpoint access
mapping before fetching profiles; serve Cache-Control: private when
OIDC is enabled (payload varies by viewer)
- /user/profile/me + /user/profile/{id}: trust X-User-Id only with a
valid API key (new optional_user_identity dependency); closes the
OIDC subject-identifier leak via forged headers
- OIDC_SESSION_SECRET is now mandatory when OIDC is enabled (hard fail,
no insecure-dev-secret fallback)
- /metrics denies unauthenticated access when no API read key is set;
new METRICS_PUBLIC opt-in (Settings, CLI, compose, env factory)
- CORS: unset CORS_ORIGINS keeps wildcard compat but drops
allow_credentials (invalid combination in browsers) + warns
- Web tier: SecurityHeadersMiddleware (nosniff, X-Frame-Options DENY,
Referrer-Policy, Permissions-Policy) + nonce-based CSP; nonce applied
to spa.html inline scripts; WEB_SECURITY_HEADERS kill switch and
WEB_CSP_EXTRA for custom tile hosts; API emits nosniff
Breaking changes documented in docs/upgrading.md.
The map endpoint fetched a single limit=500 page (the API's hard max,
sorted last_seen desc), so networks with more than 500 nodes silently
hid stale/offline nodes from the unfiltered map — including adopted
infrastructure, which still appeared under the adopted_by filter
because that filter narrows the query before the limit applies.
Follow offset/total pagination via a new _fetch_all_pages helper (40
pages / 20k nodes safety cap, partial results on later-page failure)
for both /api/v1/nodes and /api/v1/user/profiles, and report the
API-reported total in debug.total_nodes.
The meshcore-mqtt-broker destroys subscriber connections (keep-alive
timeout) without sending FIN/RST, leaving a half-open socket that paho
never sees as disconnected: the collector stays "healthy" but silently
stops ingesting forever.
- Enable TCP keepalive (idle 60s / intvl 30s / count 3) on every
connect, unwrapping paho's _WebsocketWrapper to reach the raw socket;
kernel now fails zombie sockets so paho reconnects and resubscribes
- Verify SUBACKs via on_subscribe; log broker refusals (0x80) as ERROR
- Resubscribe with the registered QoS instead of paho's QoS 0 default
- Drop the optimistic _mqtt_connected flag; health now reports
mqtt.is_connected, driven by real CONNACK/disconnect callbacks
PR #348 changed several defaults but left a handful of docs, the
docker-compose fallback, example seed files, and code docstrings stale:
- docker-compose.yml: RAW_PACKET_RETENTION_DAYS fallback 7->2 (the
:-7 override was the most consequential — operators without the env
var set got 7 days, contradicting the code default and .env.example)
- docs/configuration.md: route-history backfill now references the
300 s evaluator cadence (was 60 s)
- docs/seeding.md + example/seed/routes.yaml: window_hours 48/24 -> 6
(matches the new default; both exceeded the 12 h max)
- routes.py docstring: evaluator tick 60 s -> 300 s
- subscriber.py: raw_packet_retention_days signature defaults 7 -> 2
(overridden at runtime by effective_raw_packet_retention_days, but
kept consistent with the authoritative config default)
PR #348 lowered the RAW_PACKET_RETENTION_DAYS default 7->2, but the
dashboard packet count, activity chart, and breakdown chart still
advertised "Last 7 days" — a label that lied once only 2 days of data
remained. Make every raw-packet-derived window track the effective
retention so labels stay honest:
- get_dashboard_stats: packets_7d window = min(7, retention); expose
packets_window_days on DashboardStats so the UI renders the real window
- packet-activity / packet-breakdown: clamp days = min(days, 90, retention)
so responses never advertise days whose data has been purged
- Home StatCard + Dashboard chart subtitles: dynamic "Last N days" /
"Per day (last N days)" via new time.last_n_days / time.per_day_last_n_days
i18n keys ({{n}} interpolation, matching routes_over_last_n_days)
- ActivitySeries.days made optional (type shared across activity charts);
PacketBreakdown.days added (backend always serializes it)
Messages/ads 7-day displays are unchanged — those draw on event data
which retains the full 30-day DATA_RETENTION_DAYS window.
Tests: autouse retention fixture (->90) on the three packet test classes
so existing days-handling tests stay valid; added retention-clamp tests
proving the window tracks retention=2 and relaxes to 7 at retention>=7.
The route evaluator's fetch_candidate_paths triggered a 10-30s full-table
scan of packet_path_hops per route per sweep, flooding the slow-query
log and blocking route saves on the same scan. EXPLAIN analysis (Merge
Join vs forced Nested Loop, across query rewrites, ANALYZE, CTE
MATERIALIZED) confirmed both candidate plans cost ~10-15s at 6.4M rows;
no SQL rewrite or stats change could avoid the scan. The fix applies
four orthogonal data-volume pressures:
- ROUTE_EVALUATOR_INTERVAL_SECONDS default 60->300 (5x less frequent)
- Remove synchronous _reevaluate_route on POST/PUT; route_result and
recent_matches now refresh on the next background sweep instead of
inline (kills save latency)
- Route.window_hours default 48->6, max 12 (bounds the candidate set);
migration clamps pre-existing routes >12 to 12
- Covering index ix_packet_path_hops_raw_packet_id_position INCLUDE
(node_hash, packet_hash, event_hash, received_at, observer_node_id)
so the Merge Join outer scan becomes index-only
- RAW_PACKET_RETENTION_DAYS default 7->2 (hops cascade-delete with raw
packets; smaller table cuts the dominant outer scan)
Migration a59611449e2a clamps window_hours and rebuilds the index
CONCURRENTLY on PostgreSQL (SQLite is a no-op; no INCLUDE support).
DESTRUCTIVE on upgrade: the retention default change purges ~5 days of
raw packets + cascaded hops on the next cleanup run. See
docs/upgrading.md for the full change notes and the new save/sweep
contract.
Playwright browser binaries on this NixOS system are managed by
nixpkgs playwright.browsers (currently 1.61.1, browser revision
1228). Renovate bumped @playwright/test to 1.62.0 (revision 1234)
which has no matching browser binary in the Nix store.
Pin to exact 1.61.1 (no caret) and disable Renovate updates for
this package via renovate.json packageRules.
Vite 8 replaces Rollup with Rolldown as the bundler engine. This
requires migrating build.rollupOptions to build.rolldownOptions and
the removed object-form manualChunks to the new codeSplitting.groups
API.
Migrate vite.config.ts:
- rollupOptions -> rolldownOptions
- manualChunks object -> codeSplitting.groups with RegExp test patterns
(vendor: react/react-dom/react-router, i18n: i18next packages)
@vitejs/plugin-react@6 requires vite@^8 as peer dependency, so both
upgrades must land together.
Verified: npm run build succeeds (chunks: vendor 230KB, i18n 55KB),
tsc --noEmit clean, 339/339 vitest tests passing.
TypeScript 7 removes the 'baseUrl' compiler option and requires
'paths' entries to be relative (leading './'). Update tsconfig.json
accordingly. The '@/alias is also resolved by Vite's resolve.alias
in vite.config.ts, so only tsc --noEmit was affected.
Verified: tsc --noEmit clean, e2e typecheck clean, 339/339 vitest
tests passing.
Replace arrow with dash in the matched/threshold/degraded stat
(0/2→6 becomes 0/2-6) to save horizontal space and prevent the
stats row from wrapping on mobile.
Change the last history chart label from 'Last XXh' to 'Now' for
clarity and brevity.
Add 'now' key to en.json ('Now') and nl.json ('Nu'); remove the
now-dead 'last_n_hours' key from both locale files.
Add tests for the dash separator format and the 'Now' label.
Previous min-w-0 on the card grid item prevented the card from growing
but did not clip content that overflowed the card boundary (Chart.js
canvas, flex items with min-width:auto inside card-body). The overflow
was still visible beyond the card edge, causing page-level scroll.
Add overflow-hidden to card-body — this both resolves card-body's own
min-width:auto to 0 (per CSS spec, overflow != visible on a flex/grid
item zeroes its automatic minimum size) and clips any residual content
overflow from children. Add overflow-x-hidden to the page root as a
safety net for any non-card overflow.
Dashboard Recent Adverts widget overflowed because its card (a CSS
grid item with default min-width: auto) grew to the table's intrinsic
min-content width, preventing the overflow-x-auto wrapper from
engaging. Add min-w-0 to both bottom-grid cards and constrain the
Node name cell with max-w + truncate to reduce the table's min-width.
Routes had page-level horizontal scroll with no visible scrollbar due
to three triggers and no overflow containment in the ancestor chain:
(1) the history day-label flex row whose flex-1 spans had no min-w-0,
(2) -mx-1 on every clickable MatchRow, and (3) a Chart.js responsive
canvas in an unconstrained parent. Add min-w-0 to the card grid item,
chart wrapper, and label spans (with truncate); add overflow-hidden
to the MatchRow container to clip the negative-margin hover-bg.
Add a typed icon prop to PageHeader that renders the section icon at
h-8 w-8 inside the heading, replacing the ad-hoc span wrapper used by
Channels and Routes. Migrate all 10 list pages to pass their section
icon, and add section icons to the 3 detail page headers (PacketDetail,
PacketGroupDetail, NodeDetail).
Normalize Channels/Routes title keys to the entities.* namespace
(used everywhere else) and add the missing entities.channels key to
nl.json ('Kanalen'). Visible text is unchanged.
Update PageHeader tests for the icon contract and add a focused icon
assertion to each affected page test.
The update_profile endpoint used RequireUserOwner which returns only the
caller's user_id — no role information. The ownership check blocked ALL
non-owner edits with 403, including admins. This regressed in d37b30a
when the old Member model (RequireAdmin) was replaced with UserProfile.
Backend: read X-User-Roles header directly in update_profile (same
pattern as node_tags.py / routes.py) and bypass the ownership check when
the admin role is present. Regular members editing their own profiles
are unaffected — RequireUserOwner stays as the dependency.
Frontend: extract ProfileEditForm component (with data-testids) from
OwnProfileView. PublicProfileView now shows an inline edit form when an
admin views another user's profile. Owner still gets the existing edit
link; non-admins see nothing.
Tests:
- Backend: test_update_profile_admin_can_edit_other (admin edits other
user's profile, asserts 200 + all fields updated)
- Vitest: 4 new tests — admin button visibility, non-admin hidden,
owner link vs admin button, form submission to correct endpoint
- E2E: admin-profile-edit.spec.ts (admin edits Mem South's profile,
verifies persistence; admin on own profile sees no admin button);
members.spec.ts negative assertion (member sees no admin button)
Adds a checkbox filter to the Routes page that narrows the list to
routes owned by the current user. The filter is URL-driven (?mine=true),
cached server-side automatically via the existing key builder, and only
shown to operators/admins (members can't own routes).
Backend: mine query param on GET /api/v1/routes filters by created_by
matching the caller's X-User-Id. Legacy NULL routes are excluded.
Tests:
- Backend: 6 new TestRouteMineFilter tests (own/other/null/admin/default)
- Vitest: 6 new tests (param passing, role gating, checkbox state)
- E2E: new seed route owned by pw-operator + mine filter spec
Admins editing an operator-created route no longer steal ownership.
Ownership transfer now happens only for legacy (NULL created_by) routes.
This ensures operators retain edit access to their routes after an admin
makes a small tweak.
Replace visibility-tier-based write permissions with ownership-based:
operators can only edit/delete routes they created; admins can modify
any route and take ownership on edit. Each route stores the creator's
OIDC user_id (created_by column). Legacy routes with NULL created_by
are admin-only.
The creator's friendly name is resolved from UserProfile and displayed
on the route card with a profile link. Edit/delete buttons are hidden
per-route based on ownership rather than a flat role check.
- Migration: add nullable routes.created_by (batch mode, SQLite-safe)
- _assert_route_modifiable: dual-check visibility (404) + ownership (403)
- create_route: stamps created_by, ensures profile exists
- update_route: admin edits transfer ownership (logged)
- RouteOwner schema mirrors AdoptedByUser pattern
- Batch owner resolution in list endpoint (avoids N+1)
- Frontend: per-route canEdit gate, owner badge with profile link
- Tests: 20+ backend ownership tests, 5 frontend gating tests, e2e assertions
- Docs: routes.md + auth.md updated for ownership model
Operators can now create, edit, and delete routes (previously admin-only).
A user may never scope a route above their own role tier: an operator
creating/editing an admin-visibility route is rejected (403 on the
visibility value, 404 on touching an existing higher-visibility route),
preventing them from creating routes they could then never see or modify.
- routes.py: RequireAdmin -> RequireOperatorOrAdmin on create/update/delete;
add visibility-cap enforcement helpers reusing the existing
resolve_user_role / VISIBILITY_LEVELS ladder
- web/app.py: proxy access map admits operator for routes POST/PUT/DELETE
- Routes.tsx: canManage gate (admin||operator) on Add/Edit/Delete; visibility
<select> filters options by caller tier so operators never see 'admin'
- tests: operator-tier coverage (create/update/delete at/below/above level),
proxy access-map assertion, vitest role-gating + filtered select
- e2e: mint operator session + routes-operator spec
- docs: routes.md + auth.md operator/visibility-cap notes
Both Dashboard.tsx and Channels.tsx used the same React Query key
qk.channels.list({}) but returned different data shapes: Dashboard
returned the raw {items, total} object while Channels returned just
the items array. When Dashboard loaded first, React Query cached the
raw object, and Channels' for...of on the cached object threw
'g is not iterable'.
Fix: Channels queryFn now returns the raw API response (matching
Dashboard), and channels extraction uses data?.items ?? []. Added
regression test that pre-seeds the cache with the Dashboard shape.
JingleManSweep