SQLAlchemy 2.1 no longer installs greenlet by default, so the async
engine fails to import (Test, E2E api container), and its sources use
the `untyped-decorator` mypy code that the pinned mypy 1.9 hook rejects
(Lint).
- depend on sqlalchemy[asyncio] so greenlet is installed again
- bump the mirrors-mypy pre-commit hook to v2.4.0
- guard nullable tag values / channel_idx flagged by the newer mypy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UFsp2i9E98EzovBaurmtEb
- Add IconRss component and feedHref prop to ListToolbar
- Wire feed links on Nodes, Advertisements, and Messages pages
- Messages feed link respects channel filter, defaults to idx 17
- Simplify advert/node feed item bodies, link adverts to packet hashes
- Add unit tests, page tests, and e2e specs for feed link behavior
Four feeds (public messages, node adverts deduped per node, new nodes,
per-community-channel messages), each in RSS 2.0 and Atom, built with
stdlib ElementTree. Feeds are hard-pinned to the logged-out view
(anonymous visibility, spam always filtered, identity headers ignored)
and served at clean public web URLs (/feeds/*) via a shared proxy helper
that forwards If-None-Match + X-Forwarded-* and gates on maintenance
mode. Responses are Redis-cached (ETag/304, public max-age) and
invalidated with messages/adverts/channels mutations. Item/self links
always target the public web endpoint; WEB_PUBLIC_URL pins the canonical
origin. Includes a FEATURE_FEEDS kill switch, SPA autodiscovery <link>
tags, and env/docs updates (v0.21.0 upgrading notes).
Remove the upgrade-compatibility items scheduled for v0.20 in the v0.19
release notes:
- the `meshcore-hub db migrate-to-postgres` command and its
`common/db_migrate.py` module (with its dedicated tests)
- the `database_backend` settings field and its legacy-backend validator
(a leftover DATABASE_BACKEND is now silently ignored)
- the `[postgres]` pip extra alias (Dockerfile installs `.` now)
`tests/test_main.py` is rewritten as a minimal CLI smoke test to keep
entry-point coverage. Docs updated: v0.20.0 upgrade section notes the
removals (SQLite deployments must upgrade through v0.19 first), README
and database.md pointers reworded, stale .env.example comment dropped.
Redis becomes mandatory infrastructure (PostgreSQL-parity): the
REDIS_ENABLED flag, NullCache backend, and every conditional caching
code path are removed. /health/ready reports not_ready (503) when
Redis is unreachable and cached endpoints return 503 instead of
silently degrading; invalidation helpers never raise after a
committed write.
The bundled redis container joins the core profile only via
docker-compose.dev.yml (production points REDIS_* at an external
instance, same no-shadowing pattern as postgres), the backend suite
runs against real Redis (test-redis on 127.0.0.1:55433, CI redis:8
service container, per-xdist-worker key prefixes), and operator docs
describe the required-dependency setup (upgrading.md gains the
v0.20.0 section).
Marker clustering changed the map DOM contract the spec relied on:
clustered and off-viewport markers are absent, so raw .map-marker counts
undercount and first-marker clicks can hit-test against the sticky
navbar/banners. Count nodes as individual markers plus cluster badge
totals (new data-testid on the badge), zoom into a cluster before the
label and popup interactions so targets sit mid-map, and assert cluster
clicks spread children into individual markers.
Wrap MapPage node markers in react-leaflet-cluster (leaflet.markercluster)
with spiderfy-at-max-zoom, zoom-to-bounds on click, clustering disabled at
zoom >= 15, and viewport culling so off-screen markers stay out of the DOM.
Cluster bubbles render a plain themed count badge. Marker label HTML is now
only built when the show-labels toggle is on, halving per-marker DOM weight;
hover-to-reveal no longer applies while labels are toggled off.
- channels: post-delete card count is 6 (6 seeded + 1 created - 1), not 5
- empty-states/pagination: the empty text renders twice (mobile
EmptyState + desktop EmptyRow); scope assertions to the table cell
- mobile: list-row is desktop-only; assert the mobile sort select and
card text instead
- not-found: '/$' is a glob, not a regex; assert the exact URL '/'
- packets-detail: the newest seeded packet is the ce01 message, not an
advert; assert the event type returned by the API
Grow the suite from 41 to 77 specs covering previously untested UI
surfaces: channels admin CRUD with QR codes and keyboard nav, node
detail adoption/tag editing with the role permission matrix, SPA 404
page, sortable tables with URL-synced sort state, pagination with
filter-preserving links, mobile viewport (hamburger nav, mobile cards,
sort select), anonymous/member auth gating, raw packet detail, and
empty states. Route cards now assert history chart, recent-match links
and owner profile links.
Supporting changes: purposeful data-testids on channels/node-detail/
pagination/sort headers (plus aria-sort) and mobile nav; seed_data adds
30 trace_data filler groups for pagination, channels at every
visibility tier incl. one disabled, and a pw-operator profile.
Add a summary job that runs after all CI jobs (even on failure),
parses the pytest/vitest/Playwright JUnit reports plus coverage.xml,
and renders a consolidated Markdown summary to $GITHUB_STEP_SUMMARY:
job results, test counts, durations, backend coverage, failure
details, and the artifact inventory. Enable a Playwright JUnit
reporter and include it in the playwright-report artifact.
Starting core with docker-compose.prod.yml also started the bundled
postgres container, whose 'postgres' network alias shadowed a shared
instance of the same name on the same Docker network — the live site
silently talked to an empty local database instead of DATABASE_HOST.
- Base file: postgres profiles are now [all, postgres] (core dropped);
the dev override unions 'core' back in, so the documented dev
workflow (make up, --profile core) is unchanged
- migrate's postgres dependency is required: false — still health-gated
when the bundled container is active, skipped when profile-inactive
- prod override documents the behavior; opt in explicitly with
--profile postgres (or --profile all) for a bundled production DB
- Docs: database.md/deployment.md/configuration.md/README/.env.example
updated; upgrading.md gains a one-time cleanup runbook (full down to
remove the previously started hub-postgres container)
- /health/ready returns 503 (not 200) when not ready; body no longer leaks
raw backend error text (api + web tiers)
- SYSTEM_MAINTENANCE now gates the web /api/* proxy (503 MAINTENANCE)
instead of only hiding the SPA UI
- channel mutations invalidate messages/packets/packet_groups/dashboard
caches so visibility/key changes apply immediately (was stale up to 1h)
- async DB engine: pool_size/max_overflow/pre_ping matching the sync
engine (survives Postgres restarts); adispose() awaited from the API
lifespan (async connections were leaked by the sync dispose)
- webhooks: permanent 4xx (except 408/429) fail fast instead of retrying;
dispatch queue bounded at 1000 with drop-oldest + warning; shutdown
logs dropped events; matches_event evaluated once
- SPA: apiGet/apiPostForm redirect to OIDC login on 401; chart date
labels and route timestamps rendered in UTC/configured timezone
- e2e compose collector healthcheck uses 'meshcore-hub health collector'
(image has no pgrep)
- docs/upgrading.md: fold unreleased notes into v0.18.0 and document the
above
- api CLI single-process path now passes spam_detection_enabled /
spam_score_threshold to create_app() so the messages hide-filter
actually runs in the default (Docker) deployment
- OIDC role names (oidc_role_admin/operator/member/test) moved from
WebSettings to CommonSettings and wired onto API app.state; the API
tier previously matched hardcoded role names, ignoring customized
OIDC_ROLE_* config in authz, channel visibility, and tag/adoption
checks (oidc_roles_claim stays web-only)
- /metrics handler is now sync def so collection runs in the
threadpool instead of blocking the event loop; hoisted fragile
in-loop imports
- evaluate_route_history gates max_path_length on the packet's true
hop count (RawPacket.path_len) instead of the prefix-filtered hop
list, so history day buckets agree with snapshot evaluation
- reload_keys clears the decode cache: negatively-cached packets for
a newly added channel key were permanently dropped by the normalizer
Also documents 'sudo npx playwright install-deps chromium' in the
AGENTS.md e2e setup.
- /map/data: resolve viewer OIDC session and apply the endpoint access
mapping before fetching profiles; serve Cache-Control: private when
OIDC is enabled (payload varies by viewer)
- /user/profile/me + /user/profile/{id}: trust X-User-Id only with a
valid API key (new optional_user_identity dependency); closes the
OIDC subject-identifier leak via forged headers
- OIDC_SESSION_SECRET is now mandatory when OIDC is enabled (hard fail,
no insecure-dev-secret fallback)
- /metrics denies unauthenticated access when no API read key is set;
new METRICS_PUBLIC opt-in (Settings, CLI, compose, env factory)
- CORS: unset CORS_ORIGINS keeps wildcard compat but drops
allow_credentials (invalid combination in browsers) + warns
- Web tier: SecurityHeadersMiddleware (nosniff, X-Frame-Options DENY,
Referrer-Policy, Permissions-Policy) + nonce-based CSP; nonce applied
to spa.html inline scripts; WEB_SECURITY_HEADERS kill switch and
WEB_CSP_EXTRA for custom tile hosts; API emits nosniff
Breaking changes documented in docs/upgrading.md.
The map endpoint fetched a single limit=500 page (the API's hard max,
sorted last_seen desc), so networks with more than 500 nodes silently
hid stale/offline nodes from the unfiltered map — including adopted
infrastructure, which still appeared under the adopted_by filter
because that filter narrows the query before the limit applies.
Follow offset/total pagination via a new _fetch_all_pages helper (40
pages / 20k nodes safety cap, partial results on later-page failure)
for both /api/v1/nodes and /api/v1/user/profiles, and report the
API-reported total in debug.total_nodes.
The meshcore-mqtt-broker destroys subscriber connections (keep-alive
timeout) without sending FIN/RST, leaving a half-open socket that paho
never sees as disconnected: the collector stays "healthy" but silently
stops ingesting forever.
- Enable TCP keepalive (idle 60s / intvl 30s / count 3) on every
connect, unwrapping paho's _WebsocketWrapper to reach the raw socket;
kernel now fails zombie sockets so paho reconnects and resubscribes
- Verify SUBACKs via on_subscribe; log broker refusals (0x80) as ERROR
- Resubscribe with the registered QoS instead of paho's QoS 0 default
- Drop the optimistic _mqtt_connected flag; health now reports
mqtt.is_connected, driven by real CONNACK/disconnect callbacks
PR #348 changed several defaults but left a handful of docs, the
docker-compose fallback, example seed files, and code docstrings stale:
- docker-compose.yml: RAW_PACKET_RETENTION_DAYS fallback 7->2 (the
:-7 override was the most consequential — operators without the env
var set got 7 days, contradicting the code default and .env.example)
- docs/configuration.md: route-history backfill now references the
300 s evaluator cadence (was 60 s)
- docs/seeding.md + example/seed/routes.yaml: window_hours 48/24 -> 6
(matches the new default; both exceeded the 12 h max)
- routes.py docstring: evaluator tick 60 s -> 300 s
- subscriber.py: raw_packet_retention_days signature defaults 7 -> 2
(overridden at runtime by effective_raw_packet_retention_days, but
kept consistent with the authoritative config default)
PR #348 lowered the RAW_PACKET_RETENTION_DAYS default 7->2, but the
dashboard packet count, activity chart, and breakdown chart still
advertised "Last 7 days" — a label that lied once only 2 days of data
remained. Make every raw-packet-derived window track the effective
retention so labels stay honest:
- get_dashboard_stats: packets_7d window = min(7, retention); expose
packets_window_days on DashboardStats so the UI renders the real window
- packet-activity / packet-breakdown: clamp days = min(days, 90, retention)
so responses never advertise days whose data has been purged
- Home StatCard + Dashboard chart subtitles: dynamic "Last N days" /
"Per day (last N days)" via new time.last_n_days / time.per_day_last_n_days
i18n keys ({{n}} interpolation, matching routes_over_last_n_days)
- ActivitySeries.days made optional (type shared across activity charts);
PacketBreakdown.days added (backend always serializes it)
Messages/ads 7-day displays are unchanged — those draw on event data
which retains the full 30-day DATA_RETENTION_DAYS window.
Tests: autouse retention fixture (->90) on the three packet test classes
so existing days-handling tests stay valid; added retention-clamp tests
proving the window tracks retention=2 and relaxes to 7 at retention>=7.
The route evaluator's fetch_candidate_paths triggered a 10-30s full-table
scan of packet_path_hops per route per sweep, flooding the slow-query
log and blocking route saves on the same scan. EXPLAIN analysis (Merge
Join vs forced Nested Loop, across query rewrites, ANALYZE, CTE
MATERIALIZED) confirmed both candidate plans cost ~10-15s at 6.4M rows;
no SQL rewrite or stats change could avoid the scan. The fix applies
four orthogonal data-volume pressures:
- ROUTE_EVALUATOR_INTERVAL_SECONDS default 60->300 (5x less frequent)
- Remove synchronous _reevaluate_route on POST/PUT; route_result and
recent_matches now refresh on the next background sweep instead of
inline (kills save latency)
- Route.window_hours default 48->6, max 12 (bounds the candidate set);
migration clamps pre-existing routes >12 to 12
- Covering index ix_packet_path_hops_raw_packet_id_position INCLUDE
(node_hash, packet_hash, event_hash, received_at, observer_node_id)
so the Merge Join outer scan becomes index-only
- RAW_PACKET_RETENTION_DAYS default 7->2 (hops cascade-delete with raw
packets; smaller table cuts the dominant outer scan)
Migration a59611449e2a clamps window_hours and rebuilds the index
CONCURRENTLY on PostgreSQL (SQLite is a no-op; no INCLUDE support).
DESTRUCTIVE on upgrade: the retention default change purges ~5 days of
raw packets + cascaded hops on the next cleanup run. See
docs/upgrading.md for the full change notes and the new save/sweep
contract.
Playwright browser binaries on this NixOS system are managed by
nixpkgs playwright.browsers (currently 1.61.1, browser revision
1228). Renovate bumped @playwright/test to 1.62.0 (revision 1234)
which has no matching browser binary in the Nix store.
Pin to exact 1.61.1 (no caret) and disable Renovate updates for
this package via renovate.json packageRules.
JingleManSweep