Add two operator-controlled, startup-time settings:
- SYSTEM_ANNOUNCEMENT: non-dismissable Markdown banner rendered above the
network announcement on every page (navbar -> system -> network).
- SYSTEM_MAINTENANCE: when enabled, forces all feature flags off so the nav
collapses to Home, hides the profile menu, and the SPA renders a maintenance
page for every route. The maintenance page makes no API calls, so the API
and database can be offline while the web component keeps running.
CLI exposes --system-announcement and tri-state --system-maintenance; the bool
falls back to pydantic settings to parse SYSTEM_MAINTENANCE reliably from env.
Adds i18n strings (en/nl), tests, and docs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the multi-select Observer dropdown buried in the filter panel with
a row of clickable observer badges rendered between the filter panel and the
data list (and below the Sorting dropdown on mobile).
- Selection is stored in localStorage (shared across Adverts and Messages)
as the disabled set, so new observers default to enabled.
- Badge style reflects enabled (filled) vs disabled (muted) state; the last
enabled observer cannot be toggled off.
- Observer filter is sourced from localStorage instead of the URL query;
a two-phase fetch resolves the enabled include-list (the API filters by
inclusion only) before fetching data, with no flash of unfiltered results.
- Toggling re-scopes data and resets to page 1.
- Add enable/disable tooltip strings (en/nl) and the previously-missing
Dutch observer label.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PostgreSQL support ships in v0.14.0; v0.13.0 covers the packet-handling
enhancements. Reparent the "Optional PostgreSQL Backend" notes accordingly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The column was declared String(8) but stores 16-char hex signatures (and can
hold the up-to-32-char packet_hash fallback from the LetsMesh normalizer).
SQLite never enforced the length, so the undersized definition went unnoticed
until db migrate-to-postgres rejected the data with
StringDataRightTruncation (varchar(8)). Widen the model column and add an
Alembic migration (batch_alter_table, so it applies on both SQLite and
Postgres). Verified end-to-end migrating a 1.2GB live SQLite DB into Postgres
(all 12 tables reconcile).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Settings classes use env_file=".env" for deployments, which also caused tests
to pick up a developer's local .env (e.g. DATABASE_BACKEND=postgres without
DATABASE_HOST), failing unrelated API/CLI tests. Add an autouse fixture that
sets env_file=None on CommonSettings and every subclass, so tests depend only
on defaults and explicit monkeypatch.setenv overrides.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a Database Backend section to the README (config vars, compose profile,
schema-per-instance) and an "Optional PostgreSQL Backend" section to
docs/upgrading.md covering enablement, search_path isolation, role/db
provisioning, and the SQLite -> Postgres data-migration runbook.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The runtime image installed the package without extras, so psycopg2/asyncpg
were missing and the migrate/collector/api services failed with
ModuleNotFoundError when DATABASE_BACKEND=postgres. Install ".[postgres]" so
the one image serves both SQLite and Postgres.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copies an existing SQLite database into Postgres at the SQLAlchemy Core level,
iterating Base.metadata.sorted_tables (parent-first; excludes alembic_version)
and round-tripping each row through the typed columns so booleans, JSON, and
timestamptz convert correctly with no per-model code.
- streams large tables (stream_results + partitions) in batches
- stamps UTC on naive datetimes for tz-aware columns before insert
- single target transaction (all-or-nothing); refuses a non-empty target
unless --truncate; --dry-run previews per-table counts
- disables FK triggers via session_replication_role only when the target role
is a superuser, else relies on parent-first order (--no-replication-role to
force; managed Postgres). Defaults: source = SQLite under DATA_HOME,
target = configured DATABASE_* (schema-scoped).
- prints a per-table source->target reconciliation and fails on mismatch
Validated end-to-end against live postgres:17 (nodes/observers/raw_packets/
channels): counts reconcile, dedup preserved, is_observer->boolean,
decoded->json, received_at->timestamptz (UTC). SQLite suite green (1064).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- compose: add optional postgres service (postgres:17-alpine, profile
'postgres', healthcheck, postgres_data volume); POSTGRES_* derive from
DATABASE_* (single source of truth). DATABASE_* env added to migrate/
collector/api; migrate depends_on postgres with required:false so SQLite
deployments are unaffected.
- alembic/env: resolve the URL via CommonSettings.effective_database_url so
DATABASE_BACKEND=postgres is honoured (previously DATABASE_URL/DATA_HOME
only -> would silently migrate SQLite).
- migrations: normalize_public_key uses STRING_AGG + HAVING COUNT(*) on
Postgres (was SQLite GROUP_CONCAT + alias); raw_packets uses sa.JSON()
not the sqlite dialect type.
- database: fix _to_async_url to map postgresql+psycopg2:// (what the config
assembles) to asyncpg, so API async sessions work on Postgres; resolve the
search_path schema from DATABASE_SCHEMA env when not passed explicitly.
Validated against a live postgres:17: db upgrade builds all 13 tables in the
meshcorehub schema with correct native types (is_observer boolean, decoded
json) and alembic_version stamped; the upsert, JSON/timestamptz round-trip,
and asyncpg async sessions all work. SQLite suite still green (1061 passed).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- config: add DatabaseBackend enum + DATABASE_* component vars on
CommonSettings (host/port/name/schema/user/password, defaults
'meshcorehub'); single effective_database_url resolver with precedence
DATABASE_URL > postgres (assembled, fail-fast on missing vars) > SQLite
default. effective_database_schema returns the schema only for Postgres.
Collapses the duplicated resolver/field out of Collector/API settings.
- database: create_database_engine/DatabaseManager accept a schema arg and
scope Postgres connections via search_path (psycopg2 -c options; asyncpg
server_settings). No-op for SQLite.
- alembic/env: migrate into the instance schema (version_table_schema +
CREATE SCHEMA + search_path) for Postgres; fix online render_as_batch to
be SQLite-only (matching the offline path).
- .env.example: document the DATABASE_* block.
SQLite behaviour unchanged: default no-env path resolves to the same URL
(new regression tests), full suite green (1051 passed), fresh db upgrade OK.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dialect-neutral fixes that keep SQLite behaviour unchanged:
- event_observer: choose INSERT construct by bind dialect so the
on_conflict upsert works on both SQLite and Postgres
- database: map postgresql:// -> postgresql+asyncpg:// for async sessions
(was sqlite-only), via a _to_async_url helper
- models: import JSON from sqlalchemy (generic) not the sqlite dialect
- alembic/env: render_as_batch only for SQLite (its ALTER TABLE workaround)
Full SQLite test suite green (1045 passed) and fresh db upgrade builds all
tables; black/flake8/mypy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Records the DATABASE_BACKEND explicit switch, schema-per-instance isolation
via search_path for shared-cluster prod/stg, the no-admin-credentials
provisioning decision, the migrate-to-postgres command internals, the
operator runbook, and the phased implementation order with SQLite gates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Planning doc for adding Postgres support: code compatibility fixes,
a postgres container, component-based connection config, and a
SQLAlchemy ORM-based migration command for existing SQLite databases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The observed_by filter on messages, advertisements, telemetry, and
trace_paths matched only the first observer (stored in observer_node_id),
silently excluding events whose secondary observers appear only in the
event_observers junction table. This caused filtered lists to appear
'several hours behind' when a dominant observer consistently won the
first-insert race for recent events.
Replace the ObserverNode.public_key predicate with an IN subquery against
the event_observers junction table (the canonical multi-observer source
already used for display). Add a shared observed_by_filter_clause() helper
in observer_utils.py to avoid duplication across all four routes.
Add regression tests proving a secondary observer (present only in
event_observers) sees events via the filter. Update existing fixtures and
inline test data to seed event_hash and EventObserver rows.
Fixes#239
Adverts/Messages rows now link directly to the deduplicated packet-detail
page (/packets/hash/:hash). Each path hop renders as a clickable badge
opening a popover that looks up nodes by public-key prefix via the new
pubkey_prefix query param on GET /api/v1/nodes (case-insensitive
startswith). Adds a derived path_hash_bytes field on GroupedPacketRead.
Defaults changed: FEATURE_PACKETS now defaults to true and
RAW_PACKET_RETENTION_DAYS to 7 (independent of DATA_RETENTION_DAYS).
Fixes a mypy arg-type error by explicitly annotating the packet_hash
list as list[str].
The packet-group detail observer table only ever showed the hop count
because _extract_path_hashes looked at decoded.payload.decoded.pathHashes,
but normal packets carry the routing path at the top level as decoded.path.
Read decoded.path first (falling back to the old pathHashes location).
Frontend: render each hop as its own badge joined by arrows, wrapping on
narrow screens, with centre-truncation for paths over 16 hops. Badges carry
a path-hash-badge class + data-path-hash hook for a future node lookup.
Also expose v1/packet-groups as an open endpoint in the web proxy mapping
(it is not a prefix of v1/packets) and rebase the packet_hash index
migration onto the current head.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Fix black formatting on packet_groups.py and raw_packets.py
- Fix mypy error: annotate sort_exprs dict as dict[str, Any] so
asc()/desc() receive a typed expression, not object
- Add test_packet_groups.py with 33 tests covering list/detail
endpoints, filters, redaction, observer hydration, path_hashes
extraction, and the _extract_path_hashes helper (98% line coverage)
https://claude.ai/code/session_01NH2rZzuHzasJj12SZeRhbJ
- packets.js: switch to /api/v1/packet-groups, replace observer cell
with reception badge (paths × observers), remove SNR/hops columns
and filters, navigate to /packets/hash/:hash on click
- packet-group-detail.js: new detail page showing all (observer, path)
receptions grouped by observer with path hash sequences
- app.js: register packetGroupDetail page and /packets/hash/:hash route
https://claude.ai/code/session_01NH2rZzuHzasJj12SZeRhbJ
- New GroupedPacketRead/PacketReceptionInfo/GroupedPacketList schemas
- GET /api/v1/packet-groups: two-phase GROUP BY query, 7-day default
window, lightweight Phase 2 (no raw_hex/decoded), role-aware cache
- GET /api/v1/packet-groups/{hash}: full reception list with path_hashes
extracted from decoded.payload.decoded.pathHashes
- New (packet_hash, received_at) composite index via Alembic migration
- i18n keys for receptions, path, observer counts
https://claude.ai/code/session_01NH2rZzuHzasJj12SZeRhbJ
Add tests covering the previously-uncovered new lines flagged by Codecov:
- /packets route: search, packet_type, channel_idx, route_type, observed_by,
decryptable (both), max_snr, path-len ranges, since/until, observer-tag
hydration, detail 404, ascending sort, and the role-aware cache key builder.
- store_raw_packet: existing-observer update branch, senderPublicKey source
fallback, and the no-source-prefix case.
- normalizer: advertisement payload carries the wire packet_hash.
- CollectorSettings: raw-packet retention default/override and capture default.
Also stub the test decoder via setattr so mypy's method-assign analysis is
deterministic across incremental-cache states.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a first-class Raw Packets feature that captures every inbound MeshCore
packet from the LetsMesh `packets` feed exactly as received, independent of
how the collector later classifies it.
Capture & storage
- New `RawPacket` model + migration (raw_packets table) with single and
composite indexes for the dominant filter-then-sort-by-newest queries.
- Collector-side `RAW_PACKET_CAPTURE_ENABLED` flag (default off); capture hook
reuses the decoder's per-hex cache (no second decode), one row per observer
reception, never blocks event dispatch.
- Separate `RAW_PACKET_RETENTION_DAYS` (falls back to DATA_RETENTION_DAYS);
cleanup runs regardless of capture so disabling drains the table. Raw-packet
observers retained in the is_observer recompute union.
API
- `GET /packets` and `/packets/{id}` with rich filtering, role-aware Redis
cache key, and channel-visibility redaction (restricted-channel packets are
returned metadata-only, not hidden, so pagination counts stay stable).
Web
- `FEATURE_PACKETS` flag (default off). Responsive Packets page (table desktop,
cards mobile) plus a Packet Detail page (breadcrumb nav, raw hex + decoded).
- Nav entry after Messages on all three surfaces; home.js reordered so Map
precedes Members; new packets icon + colour.
Finer-grained classification
- Replace the single `letsmesh_packet` catch-all with per-payload-type event
types (req, ack, encrypted_direct, encrypted_channel, grp_data, multipart,
control, raw_custom, ...); letsmesh_packet kept only as the unresolved-type
safety net.
Link from structured tables
- Add `packet_hash` to advertisements and messages (populated at ingest);
exact `packet_hash` filter on /packets; cube-icon link on the Adverts and
Messages lists -> /packets?packet_hash=<hash>, shown only when the feature is
on and the row has a stored hash.
Docs/config: .env.example, docker-compose (collector + web), AGENTS.md,
SCHEMAS.md, docs/letsmesh.md, docs/upgrading.md (## v0.13.0), en/nl i18n, and a
plan/tasks doc under docs/plans/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The profile page failed to render with "Cannot access 'h' before
initialization". The /profile/me branch destructured `signal` from
params and used it in the apiGet call, but later redeclared
`const signal = ac.signal` in the same block scope. Block-scoped const
hoisting put the earlier reference in the temporal dead zone.
Drop the redundant inner declaration and pass `ac.signal` directly to
the form submit listener.
Fixes#233
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The User Profile page rendered each adopted node's relative time from
adopted_at (the adoption date) rather than the node's most recent
activity, so it always showed the time since adoption (e.g. "35 days
ago") even when the node had advertised today.
Expose last_seen on AdoptedNodeRead and render it on the profile page,
falling back to "-" when null (matching the nodes/node-detail pages).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix dashboard pages stalling under rapid navigation, plus reduce the cost
of the heaviest dashboard endpoint.
SPA request cancellation: apiGet never passed an AbortSignal, so navigating
away left a page's in-flight requests running — the homepage alone fires
three (/stats + two charts), the slowest being /stats. Under rapid
navigation these piled up, holding browser connections and API threadpool
threads, so the page actually wanted queued behind stale work; a late
resolver could also clobber the new page's DOM.
- api.js: apiGet accepts an optional { signal } and forwards it to fetch;
export isAbortError().
- router.js: each navigation gets an AbortController; the previous one is
aborted at the start of _handleRoute and its signal is passed to the page
handler. A navigation-generation guard stops a superseded route from
hiding the loader for the page that replaced it.
- app.js: pageHandler swallows AbortError (an intentional cancel is not an
error).
- all 11 page modules: thread params.signal into on-load apiGet calls and
guard their catch blocks with isAbortError.
dashboard/stats consolidation: collapse the 11 sequential COUNT(*) queries
into 4 using portable conditional aggregation (func.sum(case(...))) for
nodes, messages, advertisements, and user profiles. Responses are
unchanged.
Docs: extend the v0.12 "Read-Path Query Optimisations" note and add a
"Dashboard Navigation Responsiveness" note (front-end only, no action
required).
Tests: add test_stats_time_bucket_counts asserting the active/today/24h/7d
buckets. SPA bundles are gitignored and rebuilt by the Docker/CI build, so
only committed source changed; the esbuild build was run locally to
validate the JS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Raise diff coverage above target by exercising the previously untested
lines:
- test_cli.py: invoke the `api` command with uvicorn.run mocked to assert
the default path passes the app object (no workers/factory) and that
--workers / API_WORKERS launches the env factory by import string with
the requested worker count and factory=True.
- test_app_factory.py: add METRICS_ENABLED true/false cases that toggle
the /metrics route, covering the env-bool parsing branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a --workers / API_WORKERS option so the API can run multiple worker
processes in a single container for multi-core concurrency, without
needing to scale containers (which would conflict with the api service's
fixed container_name and complicate per-stack ops/monitoring).
The existing create_app() carries hardcoded defaults (sqlite:///./
meshcore.db, Redis off), so forked workers cannot use it — they would
open the wrong database and run without caching. Add an env-driven
factory, create_app_from_env(), that rebuilds the app from APISettings
plus the CLI-only env vars (CORS_ORIGINS, METRICS_ENABLED,
METRICS_CACHE_TTL), mirroring the single-process resolution. workers > 1
runs uvicorn against this factory via an import string; workers == 1
keeps the single-process object path so local CLI flags still apply.
Wire API_WORKERS into the api compose service (default 1, unchanged
behaviour) and document it in the README (new "Scaling the API"
section + env table) and the v0.12 upgrading notes, including the
SQLite single-host caveat and the env-vs-CLI-flag note for workers.
Tests: create_app_from_env reads DB/Redis/MQTT/metrics from env,
honours explicit overrides, and derives the collector DB path from
DATA_HOME rather than the bare create_app default.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two read-only query optimisations, no schema changes.
node-count history: replace the per-day COUNT(*) loop (up to 90 full
scans of the unindexed created_at column) with two queries — a baseline
count of nodes created before the window plus one GROUP BY date()
aggregate, accumulated into the running total in Python. Results are
identical; the baseline seed keeps pre-window nodes counted from day 0.
sender-name resolution: add resolve_sender_names() to observer_utils,
batching all pubkey prefixes into two queries (names + name tags) via an
OR of indexable LIKE 'prefix%' terms instead of two queries per prefix.
Wire it into list_messages (was ~2xN per page) and the dashboard
channel-messages loop (nested per channel x per prefix). The dashboard
recent-ads block already batches on full public keys via IN(), so it is
left as-is.
Tests: add cumulative+baseline correctness for node-count and a
multi-sender batched-resolution case for messages.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Louis King