Planning doc for adding Postgres support: code compatibility fixes,
a postgres container, component-based connection config, and a
SQLAlchemy ORM-based migration command for existing SQLite databases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The observed_by filter on messages, advertisements, telemetry, and
trace_paths matched only the first observer (stored in observer_node_id),
silently excluding events whose secondary observers appear only in the
event_observers junction table. This caused filtered lists to appear
'several hours behind' when a dominant observer consistently won the
first-insert race for recent events.
Replace the ObserverNode.public_key predicate with an IN subquery against
the event_observers junction table (the canonical multi-observer source
already used for display). Add a shared observed_by_filter_clause() helper
in observer_utils.py to avoid duplication across all four routes.
Add regression tests proving a secondary observer (present only in
event_observers) sees events via the filter. Update existing fixtures and
inline test data to seed event_hash and EventObserver rows.
Fixes#239
Adverts/Messages rows now link directly to the deduplicated packet-detail
page (/packets/hash/:hash). Each path hop renders as a clickable badge
opening a popover that looks up nodes by public-key prefix via the new
pubkey_prefix query param on GET /api/v1/nodes (case-insensitive
startswith). Adds a derived path_hash_bytes field on GroupedPacketRead.
Defaults changed: FEATURE_PACKETS now defaults to true and
RAW_PACKET_RETENTION_DAYS to 7 (independent of DATA_RETENTION_DAYS).
Fixes a mypy arg-type error by explicitly annotating the packet_hash
list as list[str].
The packet-group detail observer table only ever showed the hop count
because _extract_path_hashes looked at decoded.payload.decoded.pathHashes,
but normal packets carry the routing path at the top level as decoded.path.
Read decoded.path first (falling back to the old pathHashes location).
Frontend: render each hop as its own badge joined by arrows, wrapping on
narrow screens, with centre-truncation for paths over 16 hops. Badges carry
a path-hash-badge class + data-path-hash hook for a future node lookup.
Also expose v1/packet-groups as an open endpoint in the web proxy mapping
(it is not a prefix of v1/packets) and rebase the packet_hash index
migration onto the current head.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Fix black formatting on packet_groups.py and raw_packets.py
- Fix mypy error: annotate sort_exprs dict as dict[str, Any] so
asc()/desc() receive a typed expression, not object
- Add test_packet_groups.py with 33 tests covering list/detail
endpoints, filters, redaction, observer hydration, path_hashes
extraction, and the _extract_path_hashes helper (98% line coverage)
https://claude.ai/code/session_01NH2rZzuHzasJj12SZeRhbJ
- packets.js: switch to /api/v1/packet-groups, replace observer cell
with reception badge (paths × observers), remove SNR/hops columns
and filters, navigate to /packets/hash/:hash on click
- packet-group-detail.js: new detail page showing all (observer, path)
receptions grouped by observer with path hash sequences
- app.js: register packetGroupDetail page and /packets/hash/:hash route
https://claude.ai/code/session_01NH2rZzuHzasJj12SZeRhbJ
- New GroupedPacketRead/PacketReceptionInfo/GroupedPacketList schemas
- GET /api/v1/packet-groups: two-phase GROUP BY query, 7-day default
window, lightweight Phase 2 (no raw_hex/decoded), role-aware cache
- GET /api/v1/packet-groups/{hash}: full reception list with path_hashes
extracted from decoded.payload.decoded.pathHashes
- New (packet_hash, received_at) composite index via Alembic migration
- i18n keys for receptions, path, observer counts
https://claude.ai/code/session_01NH2rZzuHzasJj12SZeRhbJ
Add tests covering the previously-uncovered new lines flagged by Codecov:
- /packets route: search, packet_type, channel_idx, route_type, observed_by,
decryptable (both), max_snr, path-len ranges, since/until, observer-tag
hydration, detail 404, ascending sort, and the role-aware cache key builder.
- store_raw_packet: existing-observer update branch, senderPublicKey source
fallback, and the no-source-prefix case.
- normalizer: advertisement payload carries the wire packet_hash.
- CollectorSettings: raw-packet retention default/override and capture default.
Also stub the test decoder via setattr so mypy's method-assign analysis is
deterministic across incremental-cache states.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a first-class Raw Packets feature that captures every inbound MeshCore
packet from the LetsMesh `packets` feed exactly as received, independent of
how the collector later classifies it.
Capture & storage
- New `RawPacket` model + migration (raw_packets table) with single and
composite indexes for the dominant filter-then-sort-by-newest queries.
- Collector-side `RAW_PACKET_CAPTURE_ENABLED` flag (default off); capture hook
reuses the decoder's per-hex cache (no second decode), one row per observer
reception, never blocks event dispatch.
- Separate `RAW_PACKET_RETENTION_DAYS` (falls back to DATA_RETENTION_DAYS);
cleanup runs regardless of capture so disabling drains the table. Raw-packet
observers retained in the is_observer recompute union.
API
- `GET /packets` and `/packets/{id}` with rich filtering, role-aware Redis
cache key, and channel-visibility redaction (restricted-channel packets are
returned metadata-only, not hidden, so pagination counts stay stable).
Web
- `FEATURE_PACKETS` flag (default off). Responsive Packets page (table desktop,
cards mobile) plus a Packet Detail page (breadcrumb nav, raw hex + decoded).
- Nav entry after Messages on all three surfaces; home.js reordered so Map
precedes Members; new packets icon + colour.
Finer-grained classification
- Replace the single `letsmesh_packet` catch-all with per-payload-type event
types (req, ack, encrypted_direct, encrypted_channel, grp_data, multipart,
control, raw_custom, ...); letsmesh_packet kept only as the unresolved-type
safety net.
Link from structured tables
- Add `packet_hash` to advertisements and messages (populated at ingest);
exact `packet_hash` filter on /packets; cube-icon link on the Adverts and
Messages lists -> /packets?packet_hash=<hash>, shown only when the feature is
on and the row has a stored hash.
Docs/config: .env.example, docker-compose (collector + web), AGENTS.md,
SCHEMAS.md, docs/letsmesh.md, docs/upgrading.md (## v0.13.0), en/nl i18n, and a
plan/tasks doc under docs/plans/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The profile page failed to render with "Cannot access 'h' before
initialization". The /profile/me branch destructured `signal` from
params and used it in the apiGet call, but later redeclared
`const signal = ac.signal` in the same block scope. Block-scoped const
hoisting put the earlier reference in the temporal dead zone.
Drop the redundant inner declaration and pass `ac.signal` directly to
the form submit listener.
Fixes#233
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The User Profile page rendered each adopted node's relative time from
adopted_at (the adoption date) rather than the node's most recent
activity, so it always showed the time since adoption (e.g. "35 days
ago") even when the node had advertised today.
Expose last_seen on AdoptedNodeRead and render it on the profile page,
falling back to "-" when null (matching the nodes/node-detail pages).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix dashboard pages stalling under rapid navigation, plus reduce the cost
of the heaviest dashboard endpoint.
SPA request cancellation: apiGet never passed an AbortSignal, so navigating
away left a page's in-flight requests running — the homepage alone fires
three (/stats + two charts), the slowest being /stats. Under rapid
navigation these piled up, holding browser connections and API threadpool
threads, so the page actually wanted queued behind stale work; a late
resolver could also clobber the new page's DOM.
- api.js: apiGet accepts an optional { signal } and forwards it to fetch;
export isAbortError().
- router.js: each navigation gets an AbortController; the previous one is
aborted at the start of _handleRoute and its signal is passed to the page
handler. A navigation-generation guard stops a superseded route from
hiding the loader for the page that replaced it.
- app.js: pageHandler swallows AbortError (an intentional cancel is not an
error).
- all 11 page modules: thread params.signal into on-load apiGet calls and
guard their catch blocks with isAbortError.
dashboard/stats consolidation: collapse the 11 sequential COUNT(*) queries
into 4 using portable conditional aggregation (func.sum(case(...))) for
nodes, messages, advertisements, and user profiles. Responses are
unchanged.
Docs: extend the v0.12 "Read-Path Query Optimisations" note and add a
"Dashboard Navigation Responsiveness" note (front-end only, no action
required).
Tests: add test_stats_time_bucket_counts asserting the active/today/24h/7d
buckets. SPA bundles are gitignored and rebuilt by the Docker/CI build, so
only committed source changed; the esbuild build was run locally to
validate the JS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Raise diff coverage above target by exercising the previously untested
lines:
- test_cli.py: invoke the `api` command with uvicorn.run mocked to assert
the default path passes the app object (no workers/factory) and that
--workers / API_WORKERS launches the env factory by import string with
the requested worker count and factory=True.
- test_app_factory.py: add METRICS_ENABLED true/false cases that toggle
the /metrics route, covering the env-bool parsing branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a --workers / API_WORKERS option so the API can run multiple worker
processes in a single container for multi-core concurrency, without
needing to scale containers (which would conflict with the api service's
fixed container_name and complicate per-stack ops/monitoring).
The existing create_app() carries hardcoded defaults (sqlite:///./
meshcore.db, Redis off), so forked workers cannot use it — they would
open the wrong database and run without caching. Add an env-driven
factory, create_app_from_env(), that rebuilds the app from APISettings
plus the CLI-only env vars (CORS_ORIGINS, METRICS_ENABLED,
METRICS_CACHE_TTL), mirroring the single-process resolution. workers > 1
runs uvicorn against this factory via an import string; workers == 1
keeps the single-process object path so local CLI flags still apply.
Wire API_WORKERS into the api compose service (default 1, unchanged
behaviour) and document it in the README (new "Scaling the API"
section + env table) and the v0.12 upgrading notes, including the
SQLite single-host caveat and the env-vs-CLI-flag note for workers.
Tests: create_app_from_env reads DB/Redis/MQTT/metrics from env,
honours explicit overrides, and derives the collector DB path from
DATA_HOME rather than the bare create_app default.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two read-only query optimisations, no schema changes.
node-count history: replace the per-day COUNT(*) loop (up to 90 full
scans of the unindexed created_at column) with two queries — a baseline
count of nodes created before the window plus one GROUP BY date()
aggregate, accumulated into the running total in Python. Results are
identical; the baseline seed keeps pre-window nodes counted from day 0.
sender-name resolution: add resolve_sender_names() to observer_utils,
batching all pubkey prefixes into two queries (names + name tags) via an
OR of indexable LIKE 'prefix%' terms instead of two queries per prefix.
Wire it into list_messages (was ~2xN per page) and the dashboard
channel-messages loop (nested per channel x per prefix). The dashboard
recent-ads block already batches on full public keys via IN(), so it is
left as-is.
Tests: add cumulative+baseline correctness for node-count and a
multi-sender batched-resolution case for messages.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three related performance fixes for slow (>500ms) API responses.
1. Stop blocking the event loop. Route handlers were declared `async def`
but ran synchronous SQLAlchemy queries (and synchronous Redis calls via
the cache decorator) directly on the event loop, serializing requests.
Convert all handlers to sync `def` so FastAPI runs them in its
threadpool, and make the `@cached` decorator dual-mode (sync wrapper for
sync handlers, async wrapper preserved for a future async/Postgres path).
2. Tune SQLite for concurrency. Enable WAL, busy_timeout and
synchronous=NORMAL on every connection, and size the pool above the
threadpool so handlers don't wait on connections. In-memory SQLite is
guarded (no overflow-pool kwargs).
3. Precompute an indexed `nodes.is_observer` flag. The `observer=true`
filter scanned ~68k+ event rows to find a handful of observers (the
advertisements page calls it with limit=500). Replace the 5-way OR of
subqueries with `WHERE nodes.is_observer = ?`. The collector sets the
flag on first observation (in add_event_observer); the cleanup job
clears it once a node's events are all pruned; a migration adds the
column/index and backfills from the existing union.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Replace single NETWORK_RADIO_CONFIG comma-delimited string with six
individual environment variables: NETWORK_RADIO_PROFILE, _FREQUENCY,
_BANDWIDTH, _SPREADING_FACTOR, _CODING_RATE, _TX_POWER
- Radio config fields now use raw numeric types (float/int) with units
applied dynamically via RadioConfig.format_for_display()
- Add FEATURE_RADIO_CONFIG feature flag to control radio config panel
visibility on the home page (default: enabled)
- Remove from_config_string class method (no backwards compatibility)
- Update Click CLI options, create_app() signature, and _build_config_json()
- Update docker-compose.yml, .env.example, README.md, AGENTS.md
- Add upgrading.md v0.12.0 section with migration instructions
- Add test coverage for schema, config, and feature flag
Louis King