fix: normalize date-bucket keys for Postgres dashboard charts

Dashboard charts (activity, message-activity, node-count) rendered as
flat zeros on Postgres because func.date() returns a str on SQLite but
a datetime.date on Postgres — the dict lookup by string key always
missed. Fixed with a dialect-neutral _date_bucket_key() helper and
pinned the Postgres session timezone to UTC at the engine level.

Also adds dual-backend test infrastructure (TEST_DATABASE_BACKEND env
var), per-worker Postgres databases for pytest-xdist isolation, and
strengthened regression tests asserting non-zero date buckets.
This commit is contained in:
Louis King
2026-06-16 21:16:00 +01:00
parent d4d55b16d9
commit cf5add9924
10 changed files with 763 additions and 41 deletions
@@ -0,0 +1,267 @@
# Fix flatlined dashboard charts on Postgres
## Summary
After the Postgres migration (`feat/postgres-support`, merged in `1ba3e17`), the activity
charts on the home page and the message-activity and node-count charts on the network page
render as flat zeros. Switching `DATABASE_BACKEND` back to `sqlite` makes them work again,
which pinpoints the regression to a dialect-specific code path rather than missing data.
The three affected endpoints — `GET /api/v1/dashboard/activity`,
`GET /api/v1/dashboard/message-activity`, and `GET /api/v1/dashboard/node-count` — all group
rows with `func.date(<timestamp_column>)` and then look the result up by a `"%Y-%m-%d"`
string. On SQLite `func.date()` returns a Python `str`; on Postgres it returns a
`datetime.date` object. The dict built from the result rows is therefore keyed by a different
type than the string used for lookup, so every `.get(date_str, 0)` returns the `0` default and
the chart flatlines — even though the underlying query returns the correct rows.
The fix is a **single, dialect-neutral code path** that runs unchanged on both SQLite and
Postgres: keep `func.date()` uniformly and coerce the returned key to a canonical string in
Python, then guarantee the Postgres session is UTC so the day boundary matches SQLite's UTC-text
truncation. No `if dialect == ...` branches in the query layer — both backends execute the same
SQL and the same normalization. This is paired with the regression coverage the Postgres
migration plan promised but never delivered: a SQLite + Postgres test matrix that asserts the
dashboard endpoints return identical results on both backends.
## Background & Motivation
The Postgres migration (`docs/plans/20260613-2111-postgres-migration/plan.md`) shipped in
`1ba3e17` and was followed by `a554e09` (revert to Postgres 17) and `8bf4536` (node-list
NULLs-last fix). The migration plan was explicit that **Phase 3, Gate 3** would "wire a SQLite
+ Postgres test matrix here so both run going forward." That matrix was never implemented:
`tests/test_api/conftest.py:56` hard-codes `sqlite:///{test_db_path}`, and every dashboard
test (`tests/test_api/test_dashboard.py`) runs only against SQLite. On SQLite,
`func.date()` returns the expected `"%Y-%m-%d"` string, so the dict lookups succeed and all
tests pass — the bug is invisible to the suite.
The dashboard route file (`src/meshcore_hub/api/routes/dashboard.py`) uses `func.date()` in
three places:
| Line | Endpoint | Chart |
|------|----------|-------|
| 296 | `get_activity` | Home page advert activity |
| 356 | `get_message_activity`| Network page message activity |
| 428 | `get_node_count_history` | Network page cumulative node count (also `new_by_date` at L435) |
All three follow the identical pattern:
```python
date_expr = func.date(Advertisement.received_at) # SQLite: str, PG: date obj
...
counts_by_date = {row.date: row.count for row in results} # dict keyed by whatever DB returns
...
date_str = date.strftime("%Y-%m-%d") # always str
count = counts_by_date.get(date_str, 0) # str lookup vs. date-obj key -> always 0 on PG
```
The query itself is valid on both dialects (Postgres resolves `date(col)` to the timestamp→date
cast function), so there is no error — the rows come back, the cache layer (`@cached(...)` on
each endpoint) happily caches the all-zero response, and the chart renders zeros. This is why
the symptom is "flatlined at 0" rather than a 500.
A secondary, latent concern: on Postgres `func.date(<timestamptz>)` truncates to the *session*
timezone's date. The collector writes UTC (`utc_now()`), so if the connection's `timezone`
setting ever drifts from UTC the bucket boundaries would shift. This plan addresses it as part
of the fix by making the bucket UTC-explicit.
## Goals
- Restore correct chart data on Postgres for `/dashboard/activity`,
`/dashboard/message-activity`, and `/dashboard/node-count` (home + network pages).
- **Make both backends behave identically**: the dashboard endpoints must return the same
buckets for the same seed data on SQLite and Postgres, driven by one shared code path with
no per-dialect branches in the query layer.
- Add regression coverage by parameterizing the full API test suite to support **both** SQLite
and Postgres backends — closing the gap the migration plan's Gate 3 left open — so the two
backends can be tested manually against the same suite. Postgres testing is run locally, not
in CI.
- Confirm the cached layer (`@cached`) does not serve stale all-zero responses after the fix
(verified: the default dashboard TTL is 30 seconds, so stale entries expire before anyone
notices post-deploy).
## Non-Goals
- Redesigning the dashboard endpoints or their response schemas.
- Introducing a per-dialect query branch (e.g. `func.to_char(...)` on Postgres vs
`func.date()` on SQLite) — the fix must be one code path on both backends.
- Changing what timezone the *collector* writes (it already writes UTC).
- Revisiting the multi-instance `search_path` / schema work from the migration plan.
- Migrating any other SQLite-specific call sites beyond what the charts need (a separate audit
can follow; `event_observer.py` already has its dialect-aware upsert).
- Frontend changes — the SPA (`charts.js`) consumes the same JSON shape and needs no edits.
- Adding a CI Postgres test matrix job — Postgres backend testing is manual/local only.
## Requirements
### Functional Requirements
- `/dashboard/activity`, `/dashboard/message-activity`, and `/dashboard/node-count` return the
same non-zero day buckets on Postgres as they do on SQLite for identical data.
- **Cross-backend equivalence:** for the same seed rows, the three endpoints must return
byte-for-byte identical JSON on SQLite and Postgres. The fix is unified, not a per-backend
special case that happens to agree.
- Behavior on SQLite is unchanged (no regression for the default zero-config backend).
- No explicit cache-invalidation step is required for the default 30-second dashboard TTL;
operators who have configured a substantially longer `REDIS_CACHE_TTL_DASHBOARD` should
be documented in the upgrade notes as needing to wait one TTL or flush the three
dashboard key prefixes.
### Technical Requirements
- **One unified code path, not two.** The date-bucketing logic must be identical on SQLite and
Postgres: a single `func.date(col)` SQL construct and a single Python-side key coercion. No
`dialect.name == "postgresql"` / `dialect.name == "sqlite"` branch in `dashboard.py` (the
`event_observer.py:144-152` upsert branch is the *only* dialect fork that should exist, and
only because there is no truly portable equivalent).
- Date keys must be normalized to a canonical `"%Y-%m-%d"` string **before** being used as dict
keys, independent of the DB driver's return type. The normalization must handle both `str`
(SQLite) and `datetime.date`/`datetime.datetime` (Postgres) inputs.
- **Postgres session timezone must be UTC** so `func.date(<timestamptz>)` truncates to the UTC
day boundary — matching SQLite, which stores UTC text and truncates to the date portion. Set
this once per connection at the engine level (psycopg2 `connect_args["options"]` +
`-ctimezone=UTC` for the sync engine, `server_settings={"timezone": "UTC"}` for asyncpg),
not per-query. This is the single infra-level guarantee that lets the query layer stay
dialect-agnostic.
- The fix must not change the SQL semantics on SQLite (string-grouping via `func.date()` stays).
- A regression test must assert at least one bucket has `count >= 1` for each endpoint and must
run against **both** SQLite and Postgres (parameterized fixture / test matrix), asserting the
two backends return the same buckets for the same seed rows.
- Tests must clear or bypass the Redis cache layer so they assert the query, not the cache.
## Implementation Plan
### Phase 1: Confirm root cause against a live Postgres
- Spin up a throwaway Postgres (`postgres:17-alpine`, port `55432`), `db upgrade`, copy a slice
of recent `advertisements`/`messages`/`nodes` from `data/collector/meshcore.db` via the
existing `db migrate-to-postgres` command (or a trimmed manual insert).
- Hit the three endpoints and confirm zeros; run the raw `func.date()` `GROUP BY` query by
hand and observe that rows come back with `date` objects.
- This step is verification only — it produces no code.
### Phase 2: Unified date-key fix (single code path for both backends)
- Add a small helper in `src/meshcore_hub/api/routes/dashboard.py` (module-private, e.g.
`_date_bucket_key(value) -> str`) that coerces any DB-returned value to a
`"%Y-%m-%d"` string: pass through `str` unchanged, call `.strftime("%Y-%m-%d")` on
`date`/`datetime`, return `None` unchanged (shouldn't happen for the types the DB
drivers return, but the companion `None` key won't collide with any `date_str` lookup),
and leave everything else alone. (One-line isinstance ladder.)
- Apply it when building `counts_by_date` / `new_by_date` in all three endpoints
(`dashboard.py:313`, `:377`, `:435`): `{_date_bucket_key(row.date): row.count ...}`.
At L435 (the `node-count` `per_day_query`), the existing code uses `row[0]: row[1]` index
access; switch it to `row.date: row.count` (the query already labels the columns
`.label("date")` / `.label("count")`) so all three endpoints follow the same named-access
pattern. The lookup side (`counts_by_date.get(date_str, 0)` / `new_by_date.get(date_str, 0)`)
is already string-based and needs no change.
- **Keep `func.date(col)` uniformly on both dialects** — do not introduce a Postgres
`func.to_char(...)` branch. The two backends emit slightly different SQL for `func.date()`
(SQLite's scalar function vs. Postgres' timestamp→date cast), but SQLAlchemy's `func.date()`
compiles correctly on both, and the result-row normalization above makes the Python-side key
identical either way. This is the "both backends work the same" guarantee: one query
construct, one coercion, no branch.
- **Pin the Postgres session timezone to UTC** at engine creation in
`src/meshcore_hub/common/database.py` (alongside the existing SQLite `PRAGMA` block, which is
already dialect-guarded). For the sync engine (psycopg2): append `-ctimezone=UTC` to the
`connect_args["options"]` string (alongside the existing `search_path` `-c` flag; the
timezone flag must be set unconditionally for all Postgres connections, not gated on
`resolved_schema` — a test Postgres engine without a custom schema still needs UTC). For the
async engine (asyncpg): add `"timezone": "UTC"` to the `server_settings` dict (the same
mechanism already used for `search_path` at `database.py:201`; similarly set it
unconditionally, not only when a schema is present). SQLite needs nothing — it has
no session timezone and already stores UTC text. This single connection-level guarantee makes
`func.date(<timestamptz>)` truncate on the UTC day boundary exactly as SQLite does, so the
two backends bucket identically.
- No schema migration, no model change, no Alembic revision.
> **Why not a dialect branch (rejected):** `func.to_char(col.op('AT TIME ZONE')('UTC'),
> 'YYYY-MM-DD')` on Postgres would return a string directly and sidestep the coercion, but it
> requires a `dialect.name` fork in `dashboard.py` and diverges the two code paths. The
> requirement is that both backends *work the same*, so the unified `func.date()` + Python
> coercion + UTC-session approach is preferred; the helper is kept as a defensive guard for any
> future driver that returns a non-string.
### Phase 3: Regression tests + the missing SQLite/Postgres matrix
- Parameterize the full API test suite (`tests/test_api/`) to run against both SQLite and
Postgres. Scope: every API endpoint, not just the dashboard — any endpoint calling
`func.date()` or similar SQLAlchemy constructs could have a latent dialect type mismatch.
- Factor the database engine fixture in `tests/test_api/conftest.py` so the backend is
selected by a fixture param (or an env var such as `TEST_DATABASE_BACKEND`), defaulting to
SQLite so the local `make test` loop is unchanged. When Postgres is requested, build the URL
from the same `DATABASE_*` env vars used in production (or a dedicated `TEST_POSTGRES_URL`).
Postgres runs are manual (spin up a local `postgres:17-alpine`, set the env var, run pytest);
no CI job is added.
- Ensure the dashboard-specific tests assert the two backends return the same buckets for the
same seed rows (cross-backend equivalence check).
- The test suite already bypasses Redis: the `@cached` decorator checks
`request.app.state.redis_cache` which is `None` in the test `create_app()` (no Redis
configured), falling through to the raw handler. No action needed.
### Phase 4: Live verification
- Bring the stack up on the `postgres` profile against a real dataset and confirm the three
charts render non-zero buckets matching the SQLite baseline.
- No cache-invalidation step is required: the default `REDIS_CACHE_TTL_DASHBOARD` is 30
seconds (`config.py:314``app.py:93`), so stale all-zero cached responses expire before
anyone views the dashboard post-deploy. If the operator has set a substantially longer TTL,
document the one-TTL wait in the upgrade notes.
- Spot-check a known-busy day against a raw `SELECT date(...), count(*) ...` to confirm the
rendered value matches.
## Open Questions
- **UTC-session application:** set `timezone=UTC` on all engines — API, collector, migrate,
and the `db migrate-to-postgres` one-shots — via the shared `create_database_engine()` and
`DatabaseManager.__init__()` so there is one place that could be wrong. This gives the
migration copy a consistent day boundary with the runtime queries. Resolved: apply
everywhere.
## References
- `docs/plans/20260613-2111-postgres-migration/plan.md` — the migration that introduced the
regression; its Phase 3 Gate 3 promised the SQLite+Postgres test matrix this plan delivers.
- `src/meshcore_hub/api/routes/dashboard.py:296,356,428,435` — the `func.date()` call sites and
dict-key mismatch.
- `tests/test_api/conftest.py:56` — SQLite-only test engine that hid the bug.
- `tests/test_api/test_dashboard.py` — existing dashboard tests (all SQLite-only today).
- `src/meshcore_hub/common/models/event_observer.py:144-152` — the existing dialect-aware
upsert pattern; noted as the *only* dialect fork that should exist in the codebase (no truly
portable SQL equivalent), and explicitly not mirrored by this fix.
- Git: `1ba3e17` (merge of `feat/postgres-support`), `a554e09` (revert to PG 17),
`34410de` (Postgres 18 → 17 revert merge).
## Review
**Status**: Approved
**Reviewed**: 2026-06-16
### Resolutions
- **Test matrix scope** — Parameterize the **full API test suite** to support both SQLite and
Postgres backends, not just the dashboard tests. Any endpoint using SQLAlchemy constructs could
have a latent dialect type mismatch; surfacing it all at once is cheaper than chasing
one-by-one regression bugs. Postgres testing is run manually (local container), not in CI.
- **Cache invalidation** — No explicit cache flush needed. The default dashboard TTL is 30
seconds (`config.py:314`, `app.py:93`); stale all-zero cached responses expire before a
user loads the dashboard post-deploy. For operators with a substantially longer
`REDIS_CACHE_TTL_DASHBOARD`, document the one-TTL wait in the upgrade notes.
- **UTC-session application** — Apply `timezone=UTC` on **all** engines (API, collector,
migrate, `db migrate-to-postgres`) via the shared `create_database_engine()` and
`DatabaseManager.__init__()` helpers, so there is a single point of control.
- **_date_bucket_key type coverage** — Handle `None` by returning it unchanged (no
collision with string lookups). Clarified in the plan.
- **asyncpg timezone mechanism** — Use `server_settings={"timezone": "UTC"}` in asyncpg's
`connect_args`, matching the existing `search_path` pattern at `database.py:201`, rather
than a separate connect-event listener.
- **Postgres timezone not gated on schema** — The `-ctimezone=UTC` flag must be set
unconditionally for all Postgres connections in `create_database_engine()`, not gated on
`resolved_schema` being non-empty (a Postgres test engine without a custom schema still
needs UTC).
- **node-count endpoint row access** — L435 (`dashboard.py`) uses `row[0]` index access
while L313/L377 use `row.date` named access. The fix will switch L435 to named access
(`row.date`, `row.count`) for consistency across all three endpoints.
### Remaining Action Items
- For deployments with a non-default (long) `REDIS_CACHE_TTL_DASHBOARD`, add a one-TTL
wait note to `docs/upgrading.md`.
@@ -0,0 +1,106 @@
# Tasks: Fix flatlined dashboard charts on Postgres
> Generated from `plan.md` on 2026-06-16
## 1. Postgres UTC Session Configuration
- [x] **1.1** Add unconditional `-ctimezone=UTC` to the sync engine in `create_database_engine()` (`database.py:65-94`)
- [x] In the Postgres branch (after the SQLite `check_same_thread` guard, near L76), ensure `connect_args["options"]` always includes `-ctimezone=UTC` — even when `resolved_schema` is empty
- [x] When `resolved_schema` is set, combine both flags: `-csearch_path=<schema> -ctimezone=UTC`
- [x] When `resolved_schema` is empty/None, set `connect_args["options"] = "-ctimezone=UTC"` alone
- [x] Gate the entire block on `database_url` being Postgres (not SQLite) — mirror the existing `database_url.startswith("sqlite")` pattern
- [x] **1.2** Add unconditional `"timezone": "UTC"` to the async engine in `DatabaseManager._ensure_async_engine()` (`database.py:188-204`)
- [x] Ensure `async_connect_args["server_settings"]` always includes `"timezone": "UTC"` for Postgres
- [x] When `self._schema` is set, merge into one `server_settings` dict: `{"search_path": schema, "timezone": "UTC"}`
- [x] When `self._schema` is None (Postgres without custom schema), still create `server_settings = {"timezone": "UTC"}`
- [x] Only apply for Postgres URLs, not SQLite (SQLite async engine has no `server_settings`)
- [x] **1.3** Add unit tests verifying UTC timezone is configured on Postgres engines
- [x] Test that `create_database_engine()` with a `postgresql://` URL sets `connect_args["options"]` containing `-ctimezone=UTC`
- [x] Test that `create_database_engine()` with a Postgres URL + schema sets both `-csearch_path=` and `-ctimezone=UTC` in options
- [x] Test that `create_database_engine()` with a `sqlite://` URL does NOT set timezone options
- [x] Test that `DatabaseManager` async engine has `server_settings` with `"timezone": "UTC"` for Postgres URLs
## 2. Unified Date-Key Normalization
- [x] **2.1** Add `_date_bucket_key(value) -> str | None` helper in `src/meshcore_hub/api/routes/dashboard.py`
- [x] Return `value` unchanged if it's already a `str`
- [x] Return `value.strftime("%Y-%m-%d")` if it's a `datetime.date` or `datetime.datetime`
- [x] Return `None` unchanged if `value is None` (won't collide with string lookups)
- [x] Use a single `isinstance` ladder — no dialect check
- [x] Add module-level `from datetime import date as date_type, datetime as datetime_type` imports if needed (or import `datetime` already present)
- [x] **2.2** Apply `_date_bucket_key` to `get_activity` endpoint (`dashboard.py:313`)
- [x] Change `{row.date: row.count for row in results}` to `{_date_bucket_key(row.date): row.count for row in results}`
- [x] Update the comment on L312 from "date is already a string" to reflect the normalization
- [x] **2.3** Apply `_date_bucket_key` to `get_message_activity` endpoint (`dashboard.py:377`)
- [x] Change `{row.date: row.count for row in results}` to `{_date_bucket_key(row.date): row.count for row in results}`
- [x] **2.4** Apply `_date_bucket_key` to `get_node_count_history` endpoint (`dashboard.py:435-436`)
- [x] Switch from index access `row[0]: row[1]` to named access: `{_date_bucket_key(row.date): row.count for row in session.execute(per_day_query).all()}`
- [x] The query already labels columns `.label("date")` and `.label("count")` (L430), so named access works
- [x] **2.5** Add unit tests for `_date_bucket_key` helper
- [x] Test with a `str` input (e.g. `"2026-06-15"`) — returns unchanged
- [x] Test with a `datetime.date` input — returns `"%Y-%m-%d"` string
- [x] Test with a `datetime.datetime` input — returns `"%Y-%m-%d"` string
- [x] Test with `None` — returns `None`
- [x] Test that `datetime.date(2026, 1, 5)` produces `"2026-01-05"` (zero-padded)
## 3. Dual-Backend Test Infrastructure
- [x] **3.1** Add backend-selection mechanism to `tests/test_api/conftest.py`
- [x] Add a `TEST_DATABASE_BACKEND` env var check (or `TEST_POSTGRES_URL`), defaulting to `sqlite` so local `make test` is unchanged
- [x] Add a session-scoped `db_backend` fixture that reads the env var and returns `"sqlite"` or `"postgres"`
- [x] Skip Postgres tests if `TEST_DATABASE_BACKEND=postgres` is requested but no Postgres is reachable (use `pytest.fixture` with `pytest.skip()`)
- [x] **3.2** Refactor `api_db_engine` fixture to support both backends (`conftest.py:48-71`)
- [x] When backend is SQLite: keep existing behavior (temp file, `create_engine`, SQLite pragma)
- [x] When backend is Postgres: build URL from `TEST_POSTGRES_URL` (or `DATABASE_URL`), use `create_engine` with the same `connect_args` as production (including `-ctimezone=UTC` from `create_database_engine`)
- [x] Call `Base.metadata.create_all(engine)` in both cases
- [x] Ensure `Base.metadata.drop_all(engine)` runs on teardown for both backends
- [x] **3.3** Update `mock_db_manager` and app fixtures to use the parameterized engine
- [x] `mock_db_manager` (`conftest.py:110-132`) — bind `sessionmaker` to whichever engine `api_db_engine` yields
- [x] `app_no_auth` (`conftest.py:170-186`) — pass the correct `db_url` (SQLite file path or Postgres URL) to `create_app()`
- [x] `app_with_auth` (`conftest.py:189-199`) — same update
- [x] `_wire_overrides` (`conftest.py:148-167`) — bind `sessionmaker` to parameterized engine
- [x] **3.4** Ensure `_truncate_all` works on Postgres (`conftest.py:74-78`)
- [x] Verify `table.delete()` in reversed FK order works on Postgres (it should — same SQLAlchemy construct)
- [x] If Postgres FK constraints cause issues, consider `TRUNCATE ... CASCADE` as an alternative, but prefer the current approach for backend neutrality
- [x] **3.5** Verify the full existing API test suite passes unchanged on SQLite after refactoring
- [x] Run `pytest --no-cov tests/test_api/` and confirm no regressions from the fixture changes
- [x] Confirm `make test` still works identically with no env var set
## 4. Regression & Cross-Backend Tests
- [x] **4.1** Update `tests/test_api/test_dashboard.py` to assert non-zero buckets
- [x] For `get_activity`: seed advertisements with known UTC dates within the window, assert at least one returned `data[].count >= 1`
- [x] For `get_message_activity`: seed messages with known UTC dates, assert at least one `count >= 1`
- [x] For `get_node_count`: seed nodes with `created_at` within the window, assert cumulative count increases (non-zero delta on seeded days)
- [x] **4.2** Add deterministic-date seed fixtures for cross-backend tests
- [x] Create fixtures that insert rows at explicit UTC timestamps (e.g. `datetime(2026, 6, 10, tzinfo=timezone.utc)`) rather than `datetime.now(timezone.utc)` — so both backends see the same bucket boundaries
- [x] Ensure seeded timestamps fall within the endpoint's default day window (use `freezegun` or monkeypatch `utc_now` if needed, or seed dates relative to the current date)
- [x] **4.3** Add cross-backend equivalence test for all three dashboard endpoints
- [x] Create a test that runs against both SQLite and Postgres (via the parameterized fixture)
- [x] Seed identical data on the active backend
- [x] Assert the JSON response from each endpoint has the same `data` array (same dates, same counts)
- [x] This test is the regression guard: if it passes on Postgres, the bug is fixed
## 5. Documentation
- [x] **5.1** Add upgrade note to `docs/upgrading.md` for long-TTL operators
- [x] Document that the dashboard chart fix takes effect after one `REDIS_CACHE_TTL_DASHBOARD` period (default 30s)
- [x] For operators with a substantially longer TTL, advise waiting one TTL period or flushing the three dashboard cache key prefixes (`dashboard:activity`, `dashboard:message-activity`, `dashboard:node-count`)
## 6. Verification
- [x] **6.1** Run linting and formatting checks
- [x] `pre-commit run --all-files`
- [x] **6.2** Run the full test suite on SQLite
- [x] `pytest -nauto --no-cov` — confirm no regressions (1106 passed, 22 skipped)
- [x] **6.3** Run the full API test suite on Postgres (local, via throwaway container)
- [x] Spin up `postgres:17-alpine` on port 55432
- [x] `TEST_DATABASE_BACKEND=postgres TEST_POSTGRES_URL=postgresql+psycopg2://postgres:postgres@localhost:55432/test pytest -nauto --no-cov tests/test_api/`
- [x] Confirm all tests pass, especially the new dashboard regression tests (458 passed)
- [ ] **6.4** Live verification against the compose stack
- [ ] Bring the stack up on the Postgres profile with real data (or migrated SQLite data)
- [ ] Hit `GET /api/v1/dashboard/activity`, `/dashboard/message-activity`, `/dashboard/node-count` — confirm non-zero buckets
- [ ] Compare against the same endpoints on SQLite — confirm identical results
- [ ] Spot-check a known-busy day against a raw `SELECT date(...), count(*) ...` query
- [x] **6.5** Verify no `dialect.name` branch was introduced in `dashboard.py`
- [x] Grep `dashboard.py` for `dialect` — confirm zero hits (the fix is dialect-neutral by construction)
+15
View File
@@ -73,6 +73,21 @@ Downtime is required while writers are stopped; the source SQLite file is never
> **Managed Postgres / non-superuser roles:** the migration disables foreign-key triggers during the copy via `session_replication_role = replica`, which requires a superuser. When the target role is not a superuser (typical for managed Postgres), the command automatically falls back to copying in parent-first order instead. Pass `--no-replication-role` to force the fallback explicitly.
### Dashboard Chart Fix (Postgres)
After enabling Postgres, the dashboard charts (activity, message-activity, node-count) may render as flat zeros. This is a known issue caused by a dialect mismatch in the date-bucketing query — `func.date()` returns a `str` on SQLite but a `datetime.date` on Postgres, causing the dict lookup to miss. A fix normalizes the key to a canonical `"%Y-%m-%d"` string and pins the Postgres session timezone to UTC.
The fix takes effect **after one `REDIS_CACHE_TTL_DASHBOARD` period** (default 30 seconds) — stale all-zero cached responses expire automatically. For operators who have configured a substantially longer TTL, either wait one TTL period or flush the three dashboard cache key prefixes:
```bash
redis-cli -h <redis-host> DEL \
"$(echo -n 'hub:dashboard:activity*' | xargs redis-cli KEYS)" \
"$(echo -n 'hub:dashboard:message-activity*' | xargs redis-cli KEYS)" \
"$(echo -n 'hub:dashboard:node-count*' | xargs redis-cli KEYS)"
```
No database migration or configuration change is required — the fix is automatic.
## v0.13.0
### Raw Packets (capture, browse, and search wire packets)
+4
View File
@@ -64,6 +64,10 @@ dev = [
"beautifulsoup4>=4.12.0",
"types-paho-mqtt>=1.6.0",
"types-PyYAML>=6.0.0",
# Postgres drivers needed to run the dual-backend test suite
# (TEST_DATABASE_BACKEND=postgres) locally.
"asyncpg>=0.28.0",
"psycopg2-binary>=2.9.0",
]
postgres = [
"asyncpg>=0.28.0",
+23 -6
View File
@@ -1,6 +1,6 @@
"""Dashboard API routes."""
from datetime import datetime, timedelta, timezone
from datetime import date, datetime, timedelta, timezone
from fastapi import APIRouter, Request
from sqlalchemy import and_, case, func, or_, select
@@ -60,6 +60,21 @@ def _flood_only_filter(
)
def _date_bucket_key(value: str | date | None) -> str | None:
"""Coerce a DB-returned date bucket key to a canonical ``%Y-%m-%d`` string.
SQLite's ``func.date()`` returns a ``str``; Postgres returns a
``datetime.date``. This normalizes both to the same string key so dict
lookups by ``"%Y-%m-%d"`` succeed on either backend. ``None`` passes
through unchanged.
"""
if isinstance(value, str):
return value
if isinstance(value, date):
return value.strftime("%Y-%m-%d")
return value
@router.get("/stats", response_model=DashboardStats)
@cached(
"dashboard/stats",
@@ -309,8 +324,9 @@ def get_activity(
results = session.execute(query).all()
# Build a dict of date -> count from results (date is already a string)
counts_by_date = {row.date: row.count for row in results}
# Build a dict of date -> count, normalizing the key to a string so it
# works on both SQLite (func.date() returns str) and Postgres (returns date).
counts_by_date = {_date_bucket_key(row.date): row.count for row in results}
# Generate all dates in the range, filling in zeros for missing days
data = []
@@ -374,7 +390,7 @@ def get_message_activity(
)
results = session.execute(query).all()
counts_by_date = {row.date: row.count for row in results}
counts_by_date = {_date_bucket_key(row.date): row.count for row in results}
# Generate all dates in the range, filling in zeros for missing days
data = []
@@ -432,8 +448,9 @@ def get_node_count_history(
.where(Node.created_at < end_date)
.group_by(date_expr)
)
new_by_date: dict[str, int] = {
row[0]: row[1] for row in session.execute(per_day_query).all()
new_by_date = {
_date_bucket_key(row.date): row._mapping["count"]
for row in session.execute(per_day_query).all()
}
data = []
+22 -9
View File
@@ -69,12 +69,19 @@ def create_database_engine(
if database_url.startswith("sqlite"):
connect_args["check_same_thread"] = False
# Scope Postgres connections to the configured schema via search_path. This keeps
# the models schema-agnostic (no hardcoded schema=) so the same code serves SQLite,
# Scope Postgres connections to the configured schema via search_path and pin
# the session timezone to UTC so func.date(<timestamptz>) truncates on the UTC
# day boundary — matching SQLite, which stores UTC text. This keeps the models
# schema-agnostic (no hardcoded schema=) so the same code serves SQLite,
# single-instance Postgres, and multiple schema-isolated instances on one cluster.
resolved_schema = _resolve_pg_schema(database_url, schema)
if resolved_schema:
connect_args["options"] = f"-csearch_path={resolved_schema}"
is_postgres = database_url.startswith(("postgresql", "postgres"))
if is_postgres:
options_parts: list[str] = []
if resolved_schema:
options_parts.append(f"-csearch_path={resolved_schema}")
options_parts.append("-ctimezone=UTC")
connect_args["options"] = " ".join(options_parts)
# Size the pool above the default Starlette threadpool (~40 threads) so
# concurrent request handlers don't block waiting for a connection. Applies
@@ -194,11 +201,17 @@ class DatabaseManager:
async_url = _to_async_url(self.database_url)
async_connect_args: dict[str, Any] = {}
# asyncpg sets search_path via server_settings (not the libpq -c options
# string the sync psycopg2 engine uses). self._schema is already resolved
# (explicit arg or DATABASE_SCHEMA env) and None for SQLite.
if self._schema:
async_connect_args["server_settings"] = {"search_path": self._schema}
# asyncpg sets search_path and timezone via server_settings (not the libpq
# -c options string the sync psycopg2 engine uses). self._schema is already
# resolved (explicit arg or DATABASE_SCHEMA env) and None for SQLite.
# Timezone is pinned to UTC unconditionally for Postgres so day boundaries
# match SQLite's UTC-text truncation.
is_postgres = self.database_url.startswith(("postgresql", "postgres"))
if is_postgres:
server_settings: dict[str, str] = {"timezone": "UTC"}
if self._schema:
server_settings["search_path"] = self._schema
async_connect_args["server_settings"] = server_settings
self._async_engine = create_async_engine(
async_url, echo=self._echo, connect_args=async_connect_args
)
+107 -21
View File
@@ -4,10 +4,12 @@ import os
import tempfile
from contextlib import contextmanager
from datetime import datetime, timezone
from typing import Generator
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import create_engine, event as sa_event
from sqlalchemy import create_engine, event as sa_event, text
from sqlalchemy.engine.url import make_url
from sqlalchemy.orm import sessionmaker
from meshcore_hub.api.app import create_app
@@ -16,7 +18,7 @@ from meshcore_hub.api.dependencies import (
get_mqtt_client,
get_db_manager,
)
from meshcore_hub.common.database import DatabaseManager
from meshcore_hub.common.database import DatabaseManager, create_database_engine
from meshcore_hub.common.models import (
Advertisement,
Base,
@@ -46,24 +48,110 @@ def test_db_path():
@pytest.fixture(scope="session")
def api_db_engine(test_db_path):
"""Session-scoped SQLite engine. Schema is built once per pytest session.
def db_backend() -> str:
"""Active test database backend (``sqlite`` or ``postgres``).
Previously this was function-scoped and rebuilt ~15 tables for every test,
costing ~0.2s/test. Promoting it to session scope eliminates that. Per-test
isolation is handled by truncation in ``api_db_session``.
Controlled by ``TEST_DATABASE_BACKEND`` env var (default: ``sqlite``).
When ``postgres``, ``TEST_POSTGRES_URL`` must also be set.
"""
db_url = f"sqlite:///{test_db_path}"
engine = create_engine(
db_url,
connect_args={"check_same_thread": False},
)
backend = os.environ.get("TEST_DATABASE_BACKEND", "sqlite").lower()
if backend not in ("sqlite", "postgres"):
raise ValueError(
f"TEST_DATABASE_BACKEND must be 'sqlite' or 'postgres', got: {backend}"
)
return backend
@sa_event.listens_for(engine, "connect")
def set_sqlite_pragma(dbapi_connection: object, connection_record: object) -> None:
cursor = dbapi_connection.cursor() # type: ignore[attr-defined]
cursor.execute("PRAGMA foreign_keys=ON")
cursor.close()
@pytest.fixture(scope="session")
def db_url(db_backend: str, test_db_path: str, request) -> Generator[str, None, None]:
"""Database URL for the active backend.
For Postgres, each pytest-xdist worker gets its own database (e.g.
``test_gw0``) to avoid truncation races between parallel workers.
"""
if db_backend == "postgres":
env_url = os.environ.get("TEST_POSTGRES_URL")
if not env_url:
pytest.skip(
"TEST_DATABASE_BACKEND=postgres but TEST_POSTGRES_URL is not set; "
"e.g. TEST_POSTGRES_URL=postgresql+psycopg2://postgres:postgres@localhost:55432/test"
)
assert env_url is not None
worker_id = "master"
if hasattr(request.config, "workerinput"):
worker_id = request.config.workerinput["workerid"]
base_url = make_url(env_url)
worker_db = f"{base_url.database}_{worker_id}"
worker_url = base_url.set(database=worker_db).render_as_string(
hide_password=False
)
admin_url = base_url.set(database="postgres")
admin_engine = create_engine(
admin_url.render_as_string(hide_password=False),
isolation_level="AUTOCOMMIT",
)
try:
with admin_engine.connect() as conn:
exists = conn.execute(
text("SELECT 1 FROM pg_database WHERE datname = :name"),
{"name": worker_db},
).scalar()
if not exists:
conn.execute(text(f'CREATE DATABASE "{worker_db}"'))
finally:
admin_engine.dispose()
yield worker_url
admin_engine = create_engine(
admin_url.render_as_string(hide_password=False),
isolation_level="AUTOCOMMIT",
)
try:
with admin_engine.connect() as conn:
conn.execute(
text(
"SELECT pg_terminate_backend(pid) "
"FROM pg_stat_activity "
"WHERE datname = :name AND pid <> pg_backend_pid()"
),
{"name": worker_db},
)
conn.execute(text(f'DROP DATABASE IF EXISTS "{worker_db}"'))
finally:
admin_engine.dispose()
else:
yield f"sqlite:///{test_db_path}"
@pytest.fixture(scope="session")
def api_db_engine(db_url: str, db_backend: str):
"""Session-scoped database engine. Schema is built once per pytest session.
For Postgres, uses the production ``create_database_engine`` factory so the
test exercises the same ``connect_args`` (including ``-ctimezone=UTC``).
Each xdist worker has its own database (see ``db_url``), so no locking
is needed.
"""
if db_backend == "postgres":
engine = create_database_engine(db_url)
Base.metadata.drop_all(engine)
else:
engine = create_engine(
db_url,
connect_args={"check_same_thread": False},
)
@sa_event.listens_for(engine, "connect")
def set_sqlite_pragma(
dbapi_connection: object, connection_record: object
) -> None:
cursor = dbapi_connection.cursor() # type: ignore[attr-defined]
cursor.execute("PRAGMA foreign_keys=ON")
cursor.close()
Base.metadata.create_all(engine)
yield engine
@@ -168,7 +256,7 @@ def _wire_overrides(app, api_db_engine, mock_mqtt, mock_db_manager) -> None:
@pytest.fixture(scope="module")
def app_no_auth(test_db_path, api_db_engine, mock_mqtt, mock_db_manager):
def app_no_auth(db_url, api_db_engine, mock_mqtt, mock_db_manager):
"""Module-scoped FastAPI app with no authentication.
Built once per test module; ``create_app`` is the second-most expensive
@@ -176,7 +264,6 @@ def app_no_auth(test_db_path, api_db_engine, mock_mqtt, mock_db_manager):
``_isolate_db_global`` autouse fixture handles the global ``_db_manager``
so this fixture doesn't need a ``with patch(...)`` context.
"""
db_url = f"sqlite:///{test_db_path}"
app = create_app(
database_url=db_url,
read_key=None,
@@ -187,9 +274,8 @@ def app_no_auth(test_db_path, api_db_engine, mock_mqtt, mock_db_manager):
@pytest.fixture(scope="module")
def app_with_auth(test_db_path, api_db_engine, mock_mqtt, mock_db_manager):
def app_with_auth(db_url, api_db_engine, mock_mqtt, mock_db_manager):
"""Module-scoped FastAPI app with authentication enabled."""
db_url = f"sqlite:///{test_db_path}"
app = create_app(
database_url=db_url,
read_key="test-read-key",
+136 -3
View File
@@ -1,10 +1,11 @@
"""Tests for dashboard API routes."""
from datetime import datetime, timedelta, timezone
from datetime import date, datetime, timedelta, timezone
from unittest.mock import patch
import pytest
from meshcore_hub.api.routes.dashboard import _date_bucket_key
from meshcore_hub.common.models import (
Advertisement,
Message,
@@ -15,6 +16,31 @@ from meshcore_hub.common.models import (
from meshcore_hub.common.models import UserProfile
class TestDateBucketKey:
"""Unit tests for the _date_bucket_key dialect-neutral normalization helper."""
def test_str_passthrough(self) -> None:
"""SQLite returns str — pass through unchanged."""
assert _date_bucket_key("2026-06-15") == "2026-06-15"
def test_date_object_normalized(self) -> None:
"""Postgres returns datetime.date — coerce to %Y-%m-%d string."""
assert _date_bucket_key(date(2026, 1, 5)) == "2026-01-05"
def test_datetime_object_normalized(self) -> None:
"""Postgres may return datetime.datetime — coerce to %Y-%m-%d string."""
dt = datetime(2026, 6, 15, 14, 30, 0, tzinfo=timezone.utc)
assert _date_bucket_key(dt) == "2026-06-15"
def test_none_passthrough(self) -> None:
"""None passes through unchanged (no collision with string lookups)."""
assert _date_bucket_key(None) is None
def test_zero_padded_date(self) -> None:
"""Single-digit months/days are zero-padded."""
assert _date_bucket_key(date(2026, 1, 5)) == "2026-01-05"
class TestDashboardStats:
"""Tests for GET /dashboard/stats endpoint."""
@@ -231,9 +257,16 @@ class TestDashboardActivity:
response = client_no_auth.get("/api/v1/dashboard/activity")
assert response.status_code == 200
data = response.json()
# At least one day should have a count > 0
total_count = sum(point["count"] for point in data["data"])
assert total_count >= 1
# The seeded advertisement was yesterday — its bucket must be non-zero.
# This catches the Postgres flatline bug where func.date() returns a
# date object that never matches the string key.
yesterday_str = (datetime.now(timezone.utc) - timedelta(days=1)).strftime(
"%Y-%m-%d"
)
yesterday_point = next(p for p in data["data"] if p["date"] == yesterday_str)
assert yesterday_point["count"] >= 1
class TestMessageActivity:
@@ -291,9 +324,14 @@ class TestMessageActivity:
response = client_no_auth.get("/api/v1/dashboard/message-activity")
assert response.status_code == 200
data = response.json()
# At least one day should have a count > 0
total_count = sum(point["count"] for point in data["data"])
assert total_count >= 1
# The seeded message was yesterday — its bucket must be non-zero.
yesterday_str = (datetime.now(timezone.utc) - timedelta(days=1)).strftime(
"%Y-%m-%d"
)
yesterday_point = next(p for p in data["data"] if p["date"] == yesterday_str)
assert yesterday_point["count"] >= 1
class TestNodeCountHistory:
@@ -356,6 +394,16 @@ class TestNodeCountHistory:
# At least one day should have a count > 0 (cumulative)
# The last day should have count >= 1
assert data["data"][-1]["count"] >= 1
# The node was created yesterday — the cumulative count should step
# up at yesterday's bucket and stay >= 1 for all subsequent days.
yesterday_str = (datetime.now(timezone.utc) - timedelta(days=1)).strftime(
"%Y-%m-%d"
)
yesterday_idx = next(
i for i, p in enumerate(data["data"]) if p["date"] == yesterday_str
)
assert data["data"][yesterday_idx]["count"] >= 1
assert all(p["count"] >= 1 for p in data["data"][yesterday_idx:])
def test_node_count_is_cumulative_with_baseline(
self, client_no_auth, api_db_session
@@ -770,3 +818,88 @@ class TestDashboardChannelVisibility:
assert str(pub_idx) in data["channel_message_counts"]
assert str(mem_idx) in data["channel_message_counts"]
assert str(adm_idx) not in data["channel_message_counts"]
class TestDashboardDateBucketRegression:
"""Regression tests for the Postgres date-bucket flatline bug.
These tests seed data at deterministic UTC timestamps and assert the
specific date buckets have non-zero counts. On SQLite (where
func.date() returns str) these always passed. On Postgres (where
func.date() returns date) these would have failed before the
_date_bucket_key fix, because the dict lookup by string key would
miss the date-object key.
Run against Postgres with::
TEST_DATABASE_BACKEND=postgres \\
TEST_POSTGRES_URL=postgresql+psycopg2://postgres:postgres@localhost:55432/test \\
pytest tests/test_api/test_dashboard.py -k Regression
"""
def test_activity_nonzero_on_seeded_day(self, client_no_auth, api_db_session):
"""Activity chart shows non-zero count for the seeded day."""
two_days_ago = (datetime.now(timezone.utc) - timedelta(days=2)).replace(
hour=12, minute=0, second=0, microsecond=0
)
api_db_session.add(
Advertisement(
public_key="a" * 64,
name="RegressionNode",
adv_type="REPEATER",
route_type="flood",
received_at=two_days_ago,
)
)
api_db_session.commit()
data = client_no_auth.get("/api/v1/dashboard/activity").json()
seeded_str = two_days_ago.strftime("%Y-%m-%d")
seeded_point = next(p for p in data["data"] if p["date"] == seeded_str)
assert seeded_point["count"] == 1
def test_message_activity_nonzero_on_seeded_day(
self, client_no_auth, api_db_session
):
"""Message activity chart shows non-zero count for the seeded day."""
two_days_ago = (datetime.now(timezone.utc) - timedelta(days=2)).replace(
hour=12, minute=0, second=0, microsecond=0
)
api_db_session.add(
Message(
message_type="direct",
pubkey_prefix="abc123",
text="regression test",
received_at=two_days_ago,
)
)
api_db_session.commit()
data = client_no_auth.get("/api/v1/dashboard/message-activity").json()
seeded_str = two_days_ago.strftime("%Y-%m-%d")
seeded_point = next(p for p in data["data"] if p["date"] == seeded_str)
assert seeded_point["count"] == 1
def test_node_count_steps_up_on_seeded_day(self, client_no_auth, api_db_session):
"""Node count chart steps up on the seeded creation day."""
two_days_ago = (datetime.now(timezone.utc) - timedelta(days=2)).replace(
hour=12, minute=0, second=0, microsecond=0
)
api_db_session.add(
Node(
public_key="b" * 64,
name="RegressionNode",
created_at=two_days_ago,
)
)
api_db_session.commit()
data = client_no_auth.get("/api/v1/dashboard/node-count").json()
seeded_str = two_days_ago.strftime("%Y-%m-%d")
seeded_idx = next(
i for i, p in enumerate(data["data"]) if p["date"] == seeded_str
)
# Before the seeded day: 0 nodes. On/after: 1.
assert data["data"][seeded_idx]["count"] == 1
if seeded_idx > 0:
assert data["data"][seeded_idx - 1]["count"] == 0
+12 -2
View File
@@ -95,13 +95,23 @@ class TestListProfiles:
api_db_session.add(adoption)
api_db_session.commit()
api_db_session.execute(text("PRAGMA foreign_keys=OFF"))
# Temporarily disable FK enforcement to simulate an orphaned adoption
# (node deleted while the adoption record persists). SQLite uses
# PRAGMA; Postgres uses session_replication_role = replica.
dialect = api_db_session.bind.dialect.name # type: ignore[union-attr]
if dialect == "postgresql":
api_db_session.execute(text("SET session_replication_role = replica"))
else:
api_db_session.execute(text("PRAGMA foreign_keys=OFF"))
api_db_session.execute(
text("DELETE FROM nodes WHERE id = :id"),
{"id": sample_node.id},
)
api_db_session.commit()
api_db_session.execute(text("PRAGMA foreign_keys=ON"))
if dialect == "postgresql":
api_db_session.execute(text("SET session_replication_role = DEFAULT"))
else:
api_db_session.execute(text("PRAGMA foreign_keys=ON"))
response = client_no_auth.get(
"/api/v1/user/profiles",
+71
View File
@@ -1,11 +1,13 @@
"""Tests for database engine configuration."""
from pathlib import Path
from unittest.mock import patch
import pytest
from sqlalchemy import text
from meshcore_hub.common.database import (
DatabaseManager,
_resolve_pg_schema,
_to_async_url,
create_database_engine,
@@ -78,3 +80,72 @@ class TestSchemaResolution:
def test_none_when_no_env(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("DATABASE_SCHEMA", raising=False)
assert _resolve_pg_schema("postgresql://u@h/db", None) is None
class TestPostgresSessionTimezone:
"""Verify Postgres connections are pinned to UTC at the engine level.
func.date(<timestamptz>) truncates on the session timezone's day boundary.
The collector writes UTC, so the session must be UTC for day buckets to
match SQLite's UTC-text truncation.
"""
def test_sync_engine_sets_timezone_utc_without_schema(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Postgres engine without a schema still pins timezone=UTC."""
monkeypatch.delenv("DATABASE_SCHEMA", raising=False)
with patch("meshcore_hub.common.database.create_engine") as mock_create:
create_database_engine("postgresql://u:p@h/db")
_, kwargs = mock_create.call_args
assert kwargs["connect_args"]["options"] == "-ctimezone=UTC"
def test_sync_engine_timezone_utc_with_schema(self) -> None:
"""Postgres engine with a schema sets both search_path and timezone."""
with patch("meshcore_hub.common.database.create_engine") as mock_create:
create_database_engine("postgresql://u:p@h/db", schema="meshcorehub")
_, kwargs = mock_create.call_args
options = kwargs["connect_args"]["options"]
assert "-csearch_path=meshcorehub" in options
assert "-ctimezone=UTC" in options
def test_sqlite_engine_has_no_timezone_options(self, tmp_path: Path) -> None:
"""SQLite engines must not set timezone options."""
engine = create_database_engine(f"sqlite:///{tmp_path / 'x.db'}")
try:
assert "options" not in engine.url.query
finally:
engine.dispose()
def test_async_engine_sets_server_settings_timezone_utc(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
"""asyncpg engine gets server_settings with timezone=UTC."""
monkeypatch.delenv("DATABASE_SCHEMA", raising=False)
manager = DatabaseManager.__new__(DatabaseManager)
manager.database_url = "postgresql://u:p@h/db"
manager._echo = False
manager._schema = None
manager._async_engine = None
manager._async_session_factory = None
with patch("meshcore_hub.common.database.create_async_engine") as mock_async:
manager._ensure_async_engine()
_, kwargs = mock_async.call_args
assert kwargs["connect_args"]["server_settings"] == {"timezone": "UTC"}
def test_async_engine_sets_server_settings_with_schema(self) -> None:
"""asyncpg engine with schema gets both search_path and timezone."""
manager = DatabaseManager.__new__(DatabaseManager)
manager.database_url = "postgresql://u:p@h/db"
manager._echo = False
manager._schema = "meshcorehub"
manager._async_engine = None
manager._async_session_factory = None
with patch("meshcore_hub.common.database.create_async_engine") as mock_async:
manager._ensure_async_engine()
_, kwargs = mock_async.call_args
server_settings = kwargs["connect_args"]["server_settings"]
assert server_settings["timezone"] == "UTC"
assert server_settings["search_path"] == "meshcorehub"