Dashboard charts (activity, message-activity, node-count) rendered as
flat zeros on Postgres because func.date() returns a str on SQLite but
a datetime.date on Postgres — the dict lookup by string key always
missed. Fixed with a dialect-neutral _date_bucket_key() helper and
pinned the Postgres session timezone to UTC at the engine level.
Also adds dual-backend test infrastructure (TEST_DATABASE_BACKEND env
var), per-worker Postgres databases for pytest-xdist isolation, and
strengthened regression tests asserting non-zero date buckets.
- compose: add optional postgres service (postgres:17-alpine, profile
'postgres', healthcheck, postgres_data volume); POSTGRES_* derive from
DATABASE_* (single source of truth). DATABASE_* env added to migrate/
collector/api; migrate depends_on postgres with required:false so SQLite
deployments are unaffected.
- alembic/env: resolve the URL via CommonSettings.effective_database_url so
DATABASE_BACKEND=postgres is honoured (previously DATABASE_URL/DATA_HOME
only -> would silently migrate SQLite).
- migrations: normalize_public_key uses STRING_AGG + HAVING COUNT(*) on
Postgres (was SQLite GROUP_CONCAT + alias); raw_packets uses sa.JSON()
not the sqlite dialect type.
- database: fix _to_async_url to map postgresql+psycopg2:// (what the config
assembles) to asyncpg, so API async sessions work on Postgres; resolve the
search_path schema from DATABASE_SCHEMA env when not passed explicitly.
Validated against a live postgres:17: db upgrade builds all 13 tables in the
meshcorehub schema with correct native types (is_observer boolean, decoded
json) and alembic_version stamped; the upsert, JSON/timestamptz round-trip,
and asyncpg async sessions all work. SQLite suite still green (1061 passed).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three related performance fixes for slow (>500ms) API responses.
1. Stop blocking the event loop. Route handlers were declared `async def`
but ran synchronous SQLAlchemy queries (and synchronous Redis calls via
the cache decorator) directly on the event loop, serializing requests.
Convert all handlers to sync `def` so FastAPI runs them in its
threadpool, and make the `@cached` decorator dual-mode (sync wrapper for
sync handlers, async wrapper preserved for a future async/Postgres path).
2. Tune SQLite for concurrency. Enable WAL, busy_timeout and
synchronous=NORMAL on every connection, and size the pool above the
threadpool so handlers don't wait on connections. In-memory SQLite is
guarded (no overflow-pool kwargs).
3. Precompute an indexed `nodes.is_observer` flag. The `observer=true`
filter scanned ~68k+ event rows to find a handful of observers (the
advertisements page calls it with limit=500). Replace the 5-way OR of
subqueries with `WHERE nodes.is_observer = ?`. The collector sets the
flag on first observation (in add_event_observer); the cleanup job
clears it once a node's events are all pruned; a migration adds the
column/index and backfills from the existing union.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Louis King