When an IdP name claim contains trailing whitespace (e.g. 'Matt '), every
proxied API request failed with 502 'Illegal header value' because httpx
enforces RFC 7230 which forbids leading/trailing OWS in header values.
Defense in depth:
- strip_userinfo() trims the IdP name at ingress (oidc.py)
- _sanitize_header_value() removes CTL/DEL chars at both header-injection
sites (API proxy + auth-callback bootstrap) in app.py
- update_profile() trims user-supplied names in PUT handler
X-User-Name is informational only (seeds non-unique UserProfile.name);
identity/auth are keyed on X-User-Id and X-User-Roles, so sanitization is
safe.
Add two horizontal 100% stacked-bar charts to the Dashboard, gated
behind the existing packets feature flag:
- Packet Types: top 6 event_type values + 'other' rollup
- Path Bytes: 1b/2b/3b path-hash width distribution (NULL excluded)
Driven by a single new cached endpoint
GET /api/v1/dashboard/packet-breakdown?days=7 returning pre-bucketed
counts. Frontend normalizes to percentages via a new
createStackedBarChart helper in charts.js with a 7-hue oklch palette.
Persist the path-hash byte width (1/2/3) as a nullable Integer column on
RawPacket, computed at ingest by the collector and backfilled for historical
rows via a self-contained Python migration. Replace the per-request Python
decode loop in the grouped-list route with a SQL MAX() aggregate + HAVING
filter, and add a discrete <select> filter (Any/1B/2B/3B) to the /packets
SPA page.
- Model: add path_hash_bytes column to RawPacket (nullable Integer)
- Migration: batch_alter_table add_column + keyset-paginated Python backfill
reading decoded via Core select() on sa.JSON column (portable across
SQLite and Postgres)
- Collector: compute path_hash_bytes at ingest via two-tier path-hash
extraction (decoded.path -> payload.decoded.pathHashes)
- API: add func.max() aggregate to group query, HAVING filter on
?path_hash_bytes=1|2|3 param; delete Phase 3 decode loop and dead helper
- Frontend: add path-width select filter wired through query/apiParams/
pagination/headerParams; add i18n keys (en/nl)
- Tests: 1186 passed, 22 skipped; collector + API + model coverage
Recent adverts card now shows route-type badges, an observer column
with three-way fallback (observers -> observed_by -> dash), and renders
all 10 rows instead of 5. The Type column hides on mobile to prevent
overflow, and the misleading cursor-help on observer badges is removed.
Also adds a Packets chart card to the dashboard and a packets stat to
the homepage, backed by the new dashboard packet-activity endpoint.
Promotes routeTypeBadge to a shared export in components.js (previously
local to advertisements.js), and enriches RecentAdvertisement with
route_type, observers, and observed_by fields.
Backend resolves observers via fetch_observers_for_events and
observer_node_id -> public_key in batched queries.
Replace the OS system font stack with IBM Plex: the variable-weight
sans (100-700, one file per subset) for UI and headings, and Plex Mono
400 for the public keys, packet hashes, and hex that font-mono renders
across the app. Latin + latin-ext subsets only (shipped locales are
en/nl); no italics; mono is never rendered bold here.
Fonts are self-hosted from @fontsource packages via the existing
build.js vendor pipeline (no CDN), copied to static/vendor/fonts/ with
a hard build failure on wrong filenames. Wiring:
- input.css: @theme --font-sans/--font-mono + @font-face rules with
unicode-ranges taken verbatim from the package CSS; Tailwind v4
derives the document default from --font-sans, re-fonting daisyUI
components with no other changes.
- spa.html: preload the latin sans variable woff2 (crossorigin, URL
identical to the @font-face src) to minimize FOUT.
- charts.js: Chart.defaults.font.family to match (Chart.js otherwise
uses Helvetica/Arial).
- error.html: name-prepend only; the page stays dependency-free.
- middleware.py: long-term immutable cache for /static/vendor/fonts/
(stable names referenced from CSS, so ?v= versioning can't apply),
with a matching test.
- app.css: slight hero-title letter-spacing tightening for Plex at
display sizes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FastAPI 0.137.0 refactored include_router to keep included routers as
nested objects rather than flattening their routes into app.routes, so
test_app_factory's `{route.path for route in app.routes}` no longer found
the /metrics route (the endpoint still serves; only this introspection
broke). FastAPI now treats router.routes as an internal implementation
detail.
Switch the metrics route checks to the public OpenAPI schema
(app.openapi()["paths"]), which is stable across versions and resolves
router prefixes correctly, and drop the <0.137.0 pin that was blocking
the upgrade.
Verified: app factory tests pass on both 0.136.3 and 0.137.2; full
tests/test_api suite (462 tests) passes on 0.137.2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Restrict which remote observers may ingest events, keyed on the observer's
public key (the <public_key> segment of its LetsMesh upload topic). Anyone
with broker access can publish as an observer via JWT auth, so operators can
now gate ingestion.
- New ObserverFilter (case-insensitive prefix matching, allowlist overrides
denylist, accept-all when both empty)
- New OBSERVER_ALLOWLIST / OBSERVER_DENYLIST collector settings, wired through
the CLI, run_collector, create_subscriber, and Subscriber
- Filter applied at the top of _handle_mqtt_message: blocked observers' packets
are dropped before any decode, raw-packet capture, or DB write; zero added
work on the default accept-all path
- Tests: ObserverFilter unit tests, subscriber drop/allow integration tests,
config parsing tests
- Docs: configuration.md, observer.md, upgrading.md (v0.16.0), .env.example,
docker-compose.yml
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Messages page badged a row as spam at a hardcoded score >= 0.6 while the
API hides rows at SPAM_SCORE_THRESHOLD (now 0.65), so messages scored in
[0.6, 0.65) appeared flagged but were never hidden by the "show potential
spam" toggle.
Expose spam_score_threshold in the SPA config and use it for the badge so a
row is badged exactly when the API would hide it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Guard the optional `_spam_rescore_thread` with an `is not None` assert before
calling `.is_alive()`, matching the sibling test. CI runs `pre-commit run
--all-files`, so mypy checks the test files too.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Raise diff/patch coverage above the Codecov gate (was 79.8%, target 80.29%)
by exercising the previously-untested spam paths:
- subscriber: TestSpamRescoreScheduler covers the disabled early-return, the
enabled thread spawning + one sweep + clean stop, and the swallowed-error
branch of the background re-scoring loop.
- handler: a contact-message scoring test covers the contact log branch.
- spam: get/reset_spam_config caching, the zero-weight combine path, the
default-`now` path, and the null-sender reset in rescore_recent.
Patch coverage for the change is now ~97%.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an optional, off-by-default spam-detection feature that scores each
message's spam likelihood at ingest, stores the score on the row, and lets
the display layer hide likely-spam by default behind a "show potential spam"
toggle. Nothing is ever dropped at ingest, so the threshold can be retuned
without reprocessing.
Scoring (collector/spam.py): windowed COUNT(*) over new
(path_prefix, received_at) and (sender_normalized, received_at) indexes —
joint path+sender signal plus a sender-name signal (trailing-digit suffix
stripped so bob1/bob2 collapse to bob). When the path is short/zero-hop or
absent, the name signal stands alone at full weight so local spam is still
flaggable. A background sweep re-scores recent rows with hindsight to catch
the leading edge of bursts. The collector logs each score (WARNING at/above
the threshold).
Display: the messages API gains include_spam and a master-switch-aware
hide-filter; the SPA shows the toggle + a badge only when the feature is on.
Config: FEATURE_SPAM_DETECTION is the single operator switch, bridged in
Compose to the backend SPAM_DETECTION_ENABLED for collector + api (mirrors
the FEATURE_PACKETS / RAW_PACKET_CAPTURE_ENABLED pattern). Both default off.
Works on SQLite and Postgres: DB-agnostic queries, an Alembic batch migration
for the three new columns + two indexes, and backend-aware collector test
fixtures (lifted db_backend/db_url into the shared conftest).
Also: move the meshcore-hub image pull_policy out of the base compose file.
It lived in docker-compose.yml as pull_policy: daily and made `make up` pull
the published image over a freshly built local one. Base is now policy-neutral
(default missing); dev sets pull_policy: build on the hub services so it only
ever uses local builds. Prod refreshes images via a manual `docker compose
... pull`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dashboard charts (activity, message-activity, node-count) rendered as
flat zeros on Postgres because func.date() returns a str on SQLite but
a datetime.date on Postgres — the dict lookup by string key always
missed. Fixed with a dialect-neutral _date_bucket_key() helper and
pinned the Postgres session timezone to UTC at the engine level.
Also adds dual-backend test infrastructure (TEST_DATABASE_BACKEND env
var), per-worker Postgres databases for pytest-xdist isolation, and
strengthened regression tests asserting non-zero date buckets.
After the Postgres migration, nodes with no last_seen timestamp floated
to the top of the default list because Postgres sorts NULLs first under
ORDER BY ... DESC, whereas SQLite (the previous backend) sorts them last.
Wrap the last_seen ORDER BY with SQLAlchemy nullslast() so NULL last_seen
nodes always sink to the end regardless of database backend or sort
direction. Adds three regression tests covering DESC, ASC, and all-NULL
cases.
The SPA reaches the backend via the web api_proxy, which forwarded query
params using dict(request.query_params). dict() on Starlette's QueryParams
multidict keeps only the last value of a repeated key, so a multi-valued
filter like ?observed_by=A&observed_by=B was forwarded to the backend as
observed_by=B only. The backend then filtered to B's events, making a
message observed only by A disappear as soon as B was also selected — the
reported "filters act like AND" symptom. This happens independently of the
Redis response cache.
Forward request.query_params.multi_items() (a list of (key, value) tuples)
so all repeated values reach the backend. Add web proxy regression tests
asserting both observed_by values are forwarded, and capture forwarded
params in the MockHttpClient.
This is the primary fix; the earlier cache-key change (multi_items in
sorted_query_string) addressed the same collapse pattern at the cache layer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The API response cache key was built from request.query_params.items(),
which in Starlette keeps only the last value of a repeated key. Repeated
params like observed_by (?observed_by=A&observed_by=B) therefore collapsed
to observed_by=B in the key, colliding with any other filter set sharing the
same last value. The first response cached under that key was then served
for the whole TTL, making observer-filtered Messages/Adverts return stale,
wrong results (e.g. an A-only message vanishing when B was also enabled).
Use multi_items() so all repeated values are included, sorted by the full
(key, value) tuple so order-independent filter sets map to one key. Add
regression tests covering preservation, order-independence, and the
{A,B} vs {B} collision case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Raise patch coverage on the v0.14.0 Postgres backend:
- test_db_migrate: _is_superuser plus the full migrate_sqlite_to_postgres
flow (dry-run, copy, non-empty-target refusal, --truncate, missing schema),
routing create_database_engine to SQLite files so a postgresql:// target
satisfies the guard while the run executes SQLite -> SQLite in CI.
- test_main: the `db migrate-to-postgres` CLI command via CliRunner
(success, dry-run, row-count mismatch, ValueError -> ClickException).
- conftest: neutralise dotenv.load_dotenv before collection. Importing the
CLI entrypoint runs load_dotenv() at import time, which leaked a local .env
into os.environ and broke unrelated config/redis tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The web_app fixture already neutralizes NETWORK_ANNOUNCEMENT so a developer's
local .env does not leak into tests. Do the same for the two new system
settings, fixing test_system_banner_absent_when_none when SYSTEM_ANNOUNCEMENT
is set in the environment.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add two operator-controlled, startup-time settings:
- SYSTEM_ANNOUNCEMENT: non-dismissable Markdown banner rendered above the
network announcement on every page (navbar -> system -> network).
- SYSTEM_MAINTENANCE: when enabled, forces all feature flags off so the nav
collapses to Home, hides the profile menu, and the SPA renders a maintenance
page for every route. The maintenance page makes no API calls, so the API
and database can be offline while the web component keeps running.
CLI exposes --system-announcement and tri-state --system-maintenance; the bool
falls back to pydantic settings to parse SYSTEM_MAINTENANCE reliably from env.
Adds i18n strings (en/nl), tests, and docs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Settings classes use env_file=".env" for deployments, which also caused tests
to pick up a developer's local .env (e.g. DATABASE_BACKEND=postgres without
DATABASE_HOST), failing unrelated API/CLI tests. Add an autouse fixture that
sets env_file=None on CommonSettings and every subclass, so tests depend only
on defaults and explicit monkeypatch.setenv overrides.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copies an existing SQLite database into Postgres at the SQLAlchemy Core level,
iterating Base.metadata.sorted_tables (parent-first; excludes alembic_version)
and round-tripping each row through the typed columns so booleans, JSON, and
timestamptz convert correctly with no per-model code.
- streams large tables (stream_results + partitions) in batches
- stamps UTC on naive datetimes for tz-aware columns before insert
- single target transaction (all-or-nothing); refuses a non-empty target
unless --truncate; --dry-run previews per-table counts
- disables FK triggers via session_replication_role only when the target role
is a superuser, else relies on parent-first order (--no-replication-role to
force; managed Postgres). Defaults: source = SQLite under DATA_HOME,
target = configured DATABASE_* (schema-scoped).
- prints a per-table source->target reconciliation and fails on mismatch
Validated end-to-end against live postgres:17 (nodes/observers/raw_packets/
channels): counts reconcile, dedup preserved, is_observer->boolean,
decoded->json, received_at->timestamptz (UTC). SQLite suite green (1064).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- compose: add optional postgres service (postgres:17-alpine, profile
'postgres', healthcheck, postgres_data volume); POSTGRES_* derive from
DATABASE_* (single source of truth). DATABASE_* env added to migrate/
collector/api; migrate depends_on postgres with required:false so SQLite
deployments are unaffected.
- alembic/env: resolve the URL via CommonSettings.effective_database_url so
DATABASE_BACKEND=postgres is honoured (previously DATABASE_URL/DATA_HOME
only -> would silently migrate SQLite).
- migrations: normalize_public_key uses STRING_AGG + HAVING COUNT(*) on
Postgres (was SQLite GROUP_CONCAT + alias); raw_packets uses sa.JSON()
not the sqlite dialect type.
- database: fix _to_async_url to map postgresql+psycopg2:// (what the config
assembles) to asyncpg, so API async sessions work on Postgres; resolve the
search_path schema from DATABASE_SCHEMA env when not passed explicitly.
Validated against a live postgres:17: db upgrade builds all 13 tables in the
meshcorehub schema with correct native types (is_observer boolean, decoded
json) and alembic_version stamped; the upsert, JSON/timestamptz round-trip,
and asyncpg async sessions all work. SQLite suite still green (1061 passed).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- config: add DatabaseBackend enum + DATABASE_* component vars on
CommonSettings (host/port/name/schema/user/password, defaults
'meshcorehub'); single effective_database_url resolver with precedence
DATABASE_URL > postgres (assembled, fail-fast on missing vars) > SQLite
default. effective_database_schema returns the schema only for Postgres.
Collapses the duplicated resolver/field out of Collector/API settings.
- database: create_database_engine/DatabaseManager accept a schema arg and
scope Postgres connections via search_path (psycopg2 -c options; asyncpg
server_settings). No-op for SQLite.
- alembic/env: migrate into the instance schema (version_table_schema +
CREATE SCHEMA + search_path) for Postgres; fix online render_as_batch to
be SQLite-only (matching the offline path).
- .env.example: document the DATABASE_* block.
SQLite behaviour unchanged: default no-env path resolves to the same URL
(new regression tests), full suite green (1051 passed), fresh db upgrade OK.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The observed_by filter on messages, advertisements, telemetry, and
trace_paths matched only the first observer (stored in observer_node_id),
silently excluding events whose secondary observers appear only in the
event_observers junction table. This caused filtered lists to appear
'several hours behind' when a dominant observer consistently won the
first-insert race for recent events.
Replace the ObserverNode.public_key predicate with an IN subquery against
the event_observers junction table (the canonical multi-observer source
already used for display). Add a shared observed_by_filter_clause() helper
in observer_utils.py to avoid duplication across all four routes.
Add regression tests proving a secondary observer (present only in
event_observers) sees events via the filter. Update existing fixtures and
inline test data to seed event_hash and EventObserver rows.
Fixes#239
Adverts/Messages rows now link directly to the deduplicated packet-detail
page (/packets/hash/:hash). Each path hop renders as a clickable badge
opening a popover that looks up nodes by public-key prefix via the new
pubkey_prefix query param on GET /api/v1/nodes (case-insensitive
startswith). Adds a derived path_hash_bytes field on GroupedPacketRead.
Defaults changed: FEATURE_PACKETS now defaults to true and
RAW_PACKET_RETENTION_DAYS to 7 (independent of DATA_RETENTION_DAYS).
Fixes a mypy arg-type error by explicitly annotating the packet_hash
list as list[str].
The packet-group detail observer table only ever showed the hop count
because _extract_path_hashes looked at decoded.payload.decoded.pathHashes,
but normal packets carry the routing path at the top level as decoded.path.
Read decoded.path first (falling back to the old pathHashes location).
Frontend: render each hop as its own badge joined by arrows, wrapping on
narrow screens, with centre-truncation for paths over 16 hops. Badges carry
a path-hash-badge class + data-path-hash hook for a future node lookup.
Also expose v1/packet-groups as an open endpoint in the web proxy mapping
(it is not a prefix of v1/packets) and rebase the packet_hash index
migration onto the current head.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Fix black formatting on packet_groups.py and raw_packets.py
- Fix mypy error: annotate sort_exprs dict as dict[str, Any] so
asc()/desc() receive a typed expression, not object
- Add test_packet_groups.py with 33 tests covering list/detail
endpoints, filters, redaction, observer hydration, path_hashes
extraction, and the _extract_path_hashes helper (98% line coverage)
https://claude.ai/code/session_01NH2rZzuHzasJj12SZeRhbJ
Add tests covering the previously-uncovered new lines flagged by Codecov:
- /packets route: search, packet_type, channel_idx, route_type, observed_by,
decryptable (both), max_snr, path-len ranges, since/until, observer-tag
hydration, detail 404, ascending sort, and the role-aware cache key builder.
- store_raw_packet: existing-observer update branch, senderPublicKey source
fallback, and the no-source-prefix case.
- normalizer: advertisement payload carries the wire packet_hash.
- CollectorSettings: raw-packet retention default/override and capture default.
Also stub the test decoder via setattr so mypy's method-assign analysis is
deterministic across incremental-cache states.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a first-class Raw Packets feature that captures every inbound MeshCore
packet from the LetsMesh `packets` feed exactly as received, independent of
how the collector later classifies it.
Capture & storage
- New `RawPacket` model + migration (raw_packets table) with single and
composite indexes for the dominant filter-then-sort-by-newest queries.
- Collector-side `RAW_PACKET_CAPTURE_ENABLED` flag (default off); capture hook
reuses the decoder's per-hex cache (no second decode), one row per observer
reception, never blocks event dispatch.
- Separate `RAW_PACKET_RETENTION_DAYS` (falls back to DATA_RETENTION_DAYS);
cleanup runs regardless of capture so disabling drains the table. Raw-packet
observers retained in the is_observer recompute union.
API
- `GET /packets` and `/packets/{id}` with rich filtering, role-aware Redis
cache key, and channel-visibility redaction (restricted-channel packets are
returned metadata-only, not hidden, so pagination counts stay stable).
Web
- `FEATURE_PACKETS` flag (default off). Responsive Packets page (table desktop,
cards mobile) plus a Packet Detail page (breadcrumb nav, raw hex + decoded).
- Nav entry after Messages on all three surfaces; home.js reordered so Map
precedes Members; new packets icon + colour.
Finer-grained classification
- Replace the single `letsmesh_packet` catch-all with per-payload-type event
types (req, ack, encrypted_direct, encrypted_channel, grp_data, multipart,
control, raw_custom, ...); letsmesh_packet kept only as the unresolved-type
safety net.
Link from structured tables
- Add `packet_hash` to advertisements and messages (populated at ingest);
exact `packet_hash` filter on /packets; cube-icon link on the Adverts and
Messages lists -> /packets?packet_hash=<hash>, shown only when the feature is
on and the row has a stored hash.
Docs/config: .env.example, docker-compose (collector + web), AGENTS.md,
SCHEMAS.md, docs/letsmesh.md, docs/upgrading.md (## v0.13.0), en/nl i18n, and a
plan/tasks doc under docs/plans/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The User Profile page rendered each adopted node's relative time from
adopted_at (the adoption date) rather than the node's most recent
activity, so it always showed the time since adoption (e.g. "35 days
ago") even when the node had advertised today.
Expose last_seen on AdoptedNodeRead and render it on the profile page,
falling back to "-" when null (matching the nodes/node-detail pages).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix dashboard pages stalling under rapid navigation, plus reduce the cost
of the heaviest dashboard endpoint.
SPA request cancellation: apiGet never passed an AbortSignal, so navigating
away left a page's in-flight requests running — the homepage alone fires
three (/stats + two charts), the slowest being /stats. Under rapid
navigation these piled up, holding browser connections and API threadpool
threads, so the page actually wanted queued behind stale work; a late
resolver could also clobber the new page's DOM.
- api.js: apiGet accepts an optional { signal } and forwards it to fetch;
export isAbortError().
- router.js: each navigation gets an AbortController; the previous one is
aborted at the start of _handleRoute and its signal is passed to the page
handler. A navigation-generation guard stops a superseded route from
hiding the loader for the page that replaced it.
- app.js: pageHandler swallows AbortError (an intentional cancel is not an
error).
- all 11 page modules: thread params.signal into on-load apiGet calls and
guard their catch blocks with isAbortError.
dashboard/stats consolidation: collapse the 11 sequential COUNT(*) queries
into 4 using portable conditional aggregation (func.sum(case(...))) for
nodes, messages, advertisements, and user profiles. Responses are
unchanged.
Docs: extend the v0.12 "Read-Path Query Optimisations" note and add a
"Dashboard Navigation Responsiveness" note (front-end only, no action
required).
Tests: add test_stats_time_bucket_counts asserting the active/today/24h/7d
buckets. SPA bundles are gitignored and rebuilt by the Docker/CI build, so
only committed source changed; the esbuild build was run locally to
validate the JS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Raise diff coverage above target by exercising the previously untested
lines:
- test_cli.py: invoke the `api` command with uvicorn.run mocked to assert
the default path passes the app object (no workers/factory) and that
--workers / API_WORKERS launches the env factory by import string with
the requested worker count and factory=True.
- test_app_factory.py: add METRICS_ENABLED true/false cases that toggle
the /metrics route, covering the env-bool parsing branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a --workers / API_WORKERS option so the API can run multiple worker
processes in a single container for multi-core concurrency, without
needing to scale containers (which would conflict with the api service's
fixed container_name and complicate per-stack ops/monitoring).
The existing create_app() carries hardcoded defaults (sqlite:///./
meshcore.db, Redis off), so forked workers cannot use it — they would
open the wrong database and run without caching. Add an env-driven
factory, create_app_from_env(), that rebuilds the app from APISettings
plus the CLI-only env vars (CORS_ORIGINS, METRICS_ENABLED,
METRICS_CACHE_TTL), mirroring the single-process resolution. workers > 1
runs uvicorn against this factory via an import string; workers == 1
keeps the single-process object path so local CLI flags still apply.
Wire API_WORKERS into the api compose service (default 1, unchanged
behaviour) and document it in the README (new "Scaling the API"
section + env table) and the v0.12 upgrading notes, including the
SQLite single-host caveat and the env-vs-CLI-flag note for workers.
Tests: create_app_from_env reads DB/Redis/MQTT/metrics from env,
honours explicit overrides, and derives the collector DB path from
DATA_HOME rather than the bare create_app default.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two read-only query optimisations, no schema changes.
node-count history: replace the per-day COUNT(*) loop (up to 90 full
scans of the unindexed created_at column) with two queries — a baseline
count of nodes created before the window plus one GROUP BY date()
aggregate, accumulated into the running total in Python. Results are
identical; the baseline seed keeps pre-window nodes counted from day 0.
sender-name resolution: add resolve_sender_names() to observer_utils,
batching all pubkey prefixes into two queries (names + name tags) via an
OR of indexable LIKE 'prefix%' terms instead of two queries per prefix.
Wire it into list_messages (was ~2xN per page) and the dashboard
channel-messages loop (nested per channel x per prefix). The dashboard
recent-ads block already batches on full public keys via IN(), so it is
left as-is.
Tests: add cumulative+baseline correctness for node-count and a
multi-sender batched-resolution case for messages.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three related performance fixes for slow (>500ms) API responses.
1. Stop blocking the event loop. Route handlers were declared `async def`
but ran synchronous SQLAlchemy queries (and synchronous Redis calls via
the cache decorator) directly on the event loop, serializing requests.
Convert all handlers to sync `def` so FastAPI runs them in its
threadpool, and make the `@cached` decorator dual-mode (sync wrapper for
sync handlers, async wrapper preserved for a future async/Postgres path).
2. Tune SQLite for concurrency. Enable WAL, busy_timeout and
synchronous=NORMAL on every connection, and size the pool above the
threadpool so handlers don't wait on connections. In-memory SQLite is
guarded (no overflow-pool kwargs).
3. Precompute an indexed `nodes.is_observer` flag. The `observer=true`
filter scanned ~68k+ event rows to find a handful of observers (the
advertisements page calls it with limit=500). Replace the 5-way OR of
subqueries with `WHERE nodes.is_observer = ?`. The collector sets the
flag on first observation (in add_event_observer); the cleanup job
clears it once a node's events are all pruned; a migration adds the
column/index and backfills from the existing union.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Replace single NETWORK_RADIO_CONFIG comma-delimited string with six
individual environment variables: NETWORK_RADIO_PROFILE, _FREQUENCY,
_BANDWIDTH, _SPREADING_FACTOR, _CODING_RATE, _TX_POWER
- Radio config fields now use raw numeric types (float/int) with units
applied dynamically via RadioConfig.format_for_display()
- Add FEATURE_RADIO_CONFIG feature flag to control radio config panel
visibility on the home page (default: enabled)
- Remove from_config_string class method (no backwards compatibility)
- Update Click CLI options, create_app() signature, and _build_config_json()
- Update docker-compose.yml, .env.example, README.md, AGENTS.md
- Add upgrading.md v0.12.0 section with migration instructions
- Add test coverage for schema, config, and feature flag
- Rename ChannelVisibility.PUBLIC to ChannelVisibility.COMMUNITY
- Update stored value from 'public' to 'community' across model, schema, API, CLI, and frontend
- Add Alembic migration to update existing database rows
- Consolidate upgrade docs: merge v0.11.0, v0.12.0, v0.13.0 into single v0.11.0 section
- Add i18n visibility level translation keys (en, nl)
- Update section headings on channels page to use t() for i18n
- Keep visibility badges lowercase per UI design
- Group channel cards by visibility with section headings
- Move channels before messages in all nav menus for logical grouping
- Add optgroup labels (Standard/Custom) to message channel filter
- Capitalize built-in "Test" channel name for consistency
- Shorten "Advertisements" to "Adverts" in UI labels
- Lay out channel cards with side-by-side QR codes
- Shrink homepage nav cards for better fit
Replaces env-var channel keys with a Channel database model and periodic
DB refresh in the collector. Adds Channels dashboard page with QR codes,
channel visibility filtering on messages/dashboard APIs, and channel card
navigation to filtered messages view.
Track advertisement route type (flood/transport_flood/direct/transport_direct)
and node advert timestamp to distinguish zero-hop from flood adverts, improve
deduplication with 300s buckets, and default all dashboard/ad-API queries to
flood-only (including NULL for historical records).
The node detail page sends a public_key query param when fetching
recent advertisements, but the API silently ignored it (FastAPI
ignores unknown query params). This returned the 10 most recent ads
network-wide instead of for the specific node, making all dates
appear as today.
Louis King