mirror of
https://github.com/ipnet-mesh/meshcore-hub.git
synced 2026-08-06 17:02:59 +02:00
fix: normalize date-bucket keys for Postgres dashboard charts
Dashboard charts (activity, message-activity, node-count) rendered as flat zeros on Postgres because func.date() returns a str on SQLite but a datetime.date on Postgres — the dict lookup by string key always missed. Fixed with a dialect-neutral _date_bucket_key() helper and pinned the Postgres session timezone to UTC at the engine level. Also adds dual-backend test infrastructure (TEST_DATABASE_BACKEND env var), per-worker Postgres databases for pytest-xdist isolation, and strengthened regression tests asserting non-zero date buckets.
This commit is contained in:
@@ -0,0 +1,267 @@
|
||||
# Fix flatlined dashboard charts on Postgres
|
||||
|
||||
## Summary
|
||||
|
||||
After the Postgres migration (`feat/postgres-support`, merged in `1ba3e17`), the activity
|
||||
charts on the home page and the message-activity and node-count charts on the network page
|
||||
render as flat zeros. Switching `DATABASE_BACKEND` back to `sqlite` makes them work again,
|
||||
which pinpoints the regression to a dialect-specific code path rather than missing data.
|
||||
|
||||
The three affected endpoints — `GET /api/v1/dashboard/activity`,
|
||||
`GET /api/v1/dashboard/message-activity`, and `GET /api/v1/dashboard/node-count` — all group
|
||||
rows with `func.date(<timestamp_column>)` and then look the result up by a `"%Y-%m-%d"`
|
||||
string. On SQLite `func.date()` returns a Python `str`; on Postgres it returns a
|
||||
`datetime.date` object. The dict built from the result rows is therefore keyed by a different
|
||||
type than the string used for lookup, so every `.get(date_str, 0)` returns the `0` default and
|
||||
the chart flatlines — even though the underlying query returns the correct rows.
|
||||
|
||||
The fix is a **single, dialect-neutral code path** that runs unchanged on both SQLite and
|
||||
Postgres: keep `func.date()` uniformly and coerce the returned key to a canonical string in
|
||||
Python, then guarantee the Postgres session is UTC so the day boundary matches SQLite's UTC-text
|
||||
truncation. No `if dialect == ...` branches in the query layer — both backends execute the same
|
||||
SQL and the same normalization. This is paired with the regression coverage the Postgres
|
||||
migration plan promised but never delivered: a SQLite + Postgres test matrix that asserts the
|
||||
dashboard endpoints return identical results on both backends.
|
||||
|
||||
## Background & Motivation
|
||||
|
||||
The Postgres migration (`docs/plans/20260613-2111-postgres-migration/plan.md`) shipped in
|
||||
`1ba3e17` and was followed by `a554e09` (revert to Postgres 17) and `8bf4536` (node-list
|
||||
NULLs-last fix). The migration plan was explicit that **Phase 3, Gate 3** would "wire a SQLite
|
||||
+ Postgres test matrix here so both run going forward." That matrix was never implemented:
|
||||
`tests/test_api/conftest.py:56` hard-codes `sqlite:///{test_db_path}`, and every dashboard
|
||||
test (`tests/test_api/test_dashboard.py`) runs only against SQLite. On SQLite,
|
||||
`func.date()` returns the expected `"%Y-%m-%d"` string, so the dict lookups succeed and all
|
||||
tests pass — the bug is invisible to the suite.
|
||||
|
||||
The dashboard route file (`src/meshcore_hub/api/routes/dashboard.py`) uses `func.date()` in
|
||||
three places:
|
||||
|
||||
| Line | Endpoint | Chart |
|
||||
|------|----------|-------|
|
||||
| 296 | `get_activity` | Home page advert activity |
|
||||
| 356 | `get_message_activity`| Network page message activity |
|
||||
| 428 | `get_node_count_history` | Network page cumulative node count (also `new_by_date` at L435) |
|
||||
|
||||
All three follow the identical pattern:
|
||||
|
||||
```python
|
||||
date_expr = func.date(Advertisement.received_at) # SQLite: str, PG: date obj
|
||||
...
|
||||
counts_by_date = {row.date: row.count for row in results} # dict keyed by whatever DB returns
|
||||
...
|
||||
date_str = date.strftime("%Y-%m-%d") # always str
|
||||
count = counts_by_date.get(date_str, 0) # str lookup vs. date-obj key -> always 0 on PG
|
||||
```
|
||||
|
||||
The query itself is valid on both dialects (Postgres resolves `date(col)` to the timestamp→date
|
||||
cast function), so there is no error — the rows come back, the cache layer (`@cached(...)` on
|
||||
each endpoint) happily caches the all-zero response, and the chart renders zeros. This is why
|
||||
the symptom is "flatlined at 0" rather than a 500.
|
||||
|
||||
A secondary, latent concern: on Postgres `func.date(<timestamptz>)` truncates to the *session*
|
||||
timezone's date. The collector writes UTC (`utc_now()`), so if the connection's `timezone`
|
||||
setting ever drifts from UTC the bucket boundaries would shift. This plan addresses it as part
|
||||
of the fix by making the bucket UTC-explicit.
|
||||
|
||||
## Goals
|
||||
- Restore correct chart data on Postgres for `/dashboard/activity`,
|
||||
`/dashboard/message-activity`, and `/dashboard/node-count` (home + network pages).
|
||||
- **Make both backends behave identically**: the dashboard endpoints must return the same
|
||||
buckets for the same seed data on SQLite and Postgres, driven by one shared code path with
|
||||
no per-dialect branches in the query layer.
|
||||
- Add regression coverage by parameterizing the full API test suite to support **both** SQLite
|
||||
and Postgres backends — closing the gap the migration plan's Gate 3 left open — so the two
|
||||
backends can be tested manually against the same suite. Postgres testing is run locally, not
|
||||
in CI.
|
||||
- Confirm the cached layer (`@cached`) does not serve stale all-zero responses after the fix
|
||||
(verified: the default dashboard TTL is 30 seconds, so stale entries expire before anyone
|
||||
notices post-deploy).
|
||||
|
||||
## Non-Goals
|
||||
- Redesigning the dashboard endpoints or their response schemas.
|
||||
- Introducing a per-dialect query branch (e.g. `func.to_char(...)` on Postgres vs
|
||||
`func.date()` on SQLite) — the fix must be one code path on both backends.
|
||||
- Changing what timezone the *collector* writes (it already writes UTC).
|
||||
- Revisiting the multi-instance `search_path` / schema work from the migration plan.
|
||||
- Migrating any other SQLite-specific call sites beyond what the charts need (a separate audit
|
||||
can follow; `event_observer.py` already has its dialect-aware upsert).
|
||||
- Frontend changes — the SPA (`charts.js`) consumes the same JSON shape and needs no edits.
|
||||
- Adding a CI Postgres test matrix job — Postgres backend testing is manual/local only.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- `/dashboard/activity`, `/dashboard/message-activity`, and `/dashboard/node-count` return the
|
||||
same non-zero day buckets on Postgres as they do on SQLite for identical data.
|
||||
- **Cross-backend equivalence:** for the same seed rows, the three endpoints must return
|
||||
byte-for-byte identical JSON on SQLite and Postgres. The fix is unified, not a per-backend
|
||||
special case that happens to agree.
|
||||
- Behavior on SQLite is unchanged (no regression for the default zero-config backend).
|
||||
- No explicit cache-invalidation step is required for the default 30-second dashboard TTL;
|
||||
operators who have configured a substantially longer `REDIS_CACHE_TTL_DASHBOARD` should
|
||||
be documented in the upgrade notes as needing to wait one TTL or flush the three
|
||||
dashboard key prefixes.
|
||||
|
||||
### Technical Requirements
|
||||
- **One unified code path, not two.** The date-bucketing logic must be identical on SQLite and
|
||||
Postgres: a single `func.date(col)` SQL construct and a single Python-side key coercion. No
|
||||
`dialect.name == "postgresql"` / `dialect.name == "sqlite"` branch in `dashboard.py` (the
|
||||
`event_observer.py:144-152` upsert branch is the *only* dialect fork that should exist, and
|
||||
only because there is no truly portable equivalent).
|
||||
- Date keys must be normalized to a canonical `"%Y-%m-%d"` string **before** being used as dict
|
||||
keys, independent of the DB driver's return type. The normalization must handle both `str`
|
||||
(SQLite) and `datetime.date`/`datetime.datetime` (Postgres) inputs.
|
||||
- **Postgres session timezone must be UTC** so `func.date(<timestamptz>)` truncates to the UTC
|
||||
day boundary — matching SQLite, which stores UTC text and truncates to the date portion. Set
|
||||
this once per connection at the engine level (psycopg2 `connect_args["options"]` +
|
||||
`-ctimezone=UTC` for the sync engine, `server_settings={"timezone": "UTC"}` for asyncpg),
|
||||
not per-query. This is the single infra-level guarantee that lets the query layer stay
|
||||
dialect-agnostic.
|
||||
- The fix must not change the SQL semantics on SQLite (string-grouping via `func.date()` stays).
|
||||
- A regression test must assert at least one bucket has `count >= 1` for each endpoint and must
|
||||
run against **both** SQLite and Postgres (parameterized fixture / test matrix), asserting the
|
||||
two backends return the same buckets for the same seed rows.
|
||||
- Tests must clear or bypass the Redis cache layer so they assert the query, not the cache.
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Confirm root cause against a live Postgres
|
||||
- Spin up a throwaway Postgres (`postgres:17-alpine`, port `55432`), `db upgrade`, copy a slice
|
||||
of recent `advertisements`/`messages`/`nodes` from `data/collector/meshcore.db` via the
|
||||
existing `db migrate-to-postgres` command (or a trimmed manual insert).
|
||||
- Hit the three endpoints and confirm zeros; run the raw `func.date()` `GROUP BY` query by
|
||||
hand and observe that rows come back with `date` objects.
|
||||
- This step is verification only — it produces no code.
|
||||
|
||||
### Phase 2: Unified date-key fix (single code path for both backends)
|
||||
- Add a small helper in `src/meshcore_hub/api/routes/dashboard.py` (module-private, e.g.
|
||||
`_date_bucket_key(value) -> str`) that coerces any DB-returned value to a
|
||||
`"%Y-%m-%d"` string: pass through `str` unchanged, call `.strftime("%Y-%m-%d")` on
|
||||
`date`/`datetime`, return `None` unchanged (shouldn't happen for the types the DB
|
||||
drivers return, but the companion `None` key won't collide with any `date_str` lookup),
|
||||
and leave everything else alone. (One-line isinstance ladder.)
|
||||
- Apply it when building `counts_by_date` / `new_by_date` in all three endpoints
|
||||
(`dashboard.py:313`, `:377`, `:435`): `{_date_bucket_key(row.date): row.count ...}`.
|
||||
At L435 (the `node-count` `per_day_query`), the existing code uses `row[0]: row[1]` index
|
||||
access; switch it to `row.date: row.count` (the query already labels the columns
|
||||
`.label("date")` / `.label("count")`) so all three endpoints follow the same named-access
|
||||
pattern. The lookup side (`counts_by_date.get(date_str, 0)` / `new_by_date.get(date_str, 0)`)
|
||||
is already string-based and needs no change.
|
||||
- **Keep `func.date(col)` uniformly on both dialects** — do not introduce a Postgres
|
||||
`func.to_char(...)` branch. The two backends emit slightly different SQL for `func.date()`
|
||||
(SQLite's scalar function vs. Postgres' timestamp→date cast), but SQLAlchemy's `func.date()`
|
||||
compiles correctly on both, and the result-row normalization above makes the Python-side key
|
||||
identical either way. This is the "both backends work the same" guarantee: one query
|
||||
construct, one coercion, no branch.
|
||||
- **Pin the Postgres session timezone to UTC** at engine creation in
|
||||
`src/meshcore_hub/common/database.py` (alongside the existing SQLite `PRAGMA` block, which is
|
||||
already dialect-guarded). For the sync engine (psycopg2): append `-ctimezone=UTC` to the
|
||||
`connect_args["options"]` string (alongside the existing `search_path` `-c` flag; the
|
||||
timezone flag must be set unconditionally for all Postgres connections, not gated on
|
||||
`resolved_schema` — a test Postgres engine without a custom schema still needs UTC). For the
|
||||
async engine (asyncpg): add `"timezone": "UTC"` to the `server_settings` dict (the same
|
||||
mechanism already used for `search_path` at `database.py:201`; similarly set it
|
||||
unconditionally, not only when a schema is present). SQLite needs nothing — it has
|
||||
no session timezone and already stores UTC text. This single connection-level guarantee makes
|
||||
`func.date(<timestamptz>)` truncate on the UTC day boundary exactly as SQLite does, so the
|
||||
two backends bucket identically.
|
||||
- No schema migration, no model change, no Alembic revision.
|
||||
|
||||
> **Why not a dialect branch (rejected):** `func.to_char(col.op('AT TIME ZONE')('UTC'),
|
||||
> 'YYYY-MM-DD')` on Postgres would return a string directly and sidestep the coercion, but it
|
||||
> requires a `dialect.name` fork in `dashboard.py` and diverges the two code paths. The
|
||||
> requirement is that both backends *work the same*, so the unified `func.date()` + Python
|
||||
> coercion + UTC-session approach is preferred; the helper is kept as a defensive guard for any
|
||||
> future driver that returns a non-string.
|
||||
|
||||
### Phase 3: Regression tests + the missing SQLite/Postgres matrix
|
||||
- Parameterize the full API test suite (`tests/test_api/`) to run against both SQLite and
|
||||
Postgres. Scope: every API endpoint, not just the dashboard — any endpoint calling
|
||||
`func.date()` or similar SQLAlchemy constructs could have a latent dialect type mismatch.
|
||||
- Factor the database engine fixture in `tests/test_api/conftest.py` so the backend is
|
||||
selected by a fixture param (or an env var such as `TEST_DATABASE_BACKEND`), defaulting to
|
||||
SQLite so the local `make test` loop is unchanged. When Postgres is requested, build the URL
|
||||
from the same `DATABASE_*` env vars used in production (or a dedicated `TEST_POSTGRES_URL`).
|
||||
Postgres runs are manual (spin up a local `postgres:17-alpine`, set the env var, run pytest);
|
||||
no CI job is added.
|
||||
- Ensure the dashboard-specific tests assert the two backends return the same buckets for the
|
||||
same seed rows (cross-backend equivalence check).
|
||||
- The test suite already bypasses Redis: the `@cached` decorator checks
|
||||
`request.app.state.redis_cache` which is `None` in the test `create_app()` (no Redis
|
||||
configured), falling through to the raw handler. No action needed.
|
||||
|
||||
### Phase 4: Live verification
|
||||
|
||||
- Bring the stack up on the `postgres` profile against a real dataset and confirm the three
|
||||
charts render non-zero buckets matching the SQLite baseline.
|
||||
- No cache-invalidation step is required: the default `REDIS_CACHE_TTL_DASHBOARD` is 30
|
||||
seconds (`config.py:314` → `app.py:93`), so stale all-zero cached responses expire before
|
||||
anyone views the dashboard post-deploy. If the operator has set a substantially longer TTL,
|
||||
document the one-TTL wait in the upgrade notes.
|
||||
- Spot-check a known-busy day against a raw `SELECT date(...), count(*) ...` to confirm the
|
||||
rendered value matches.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **UTC-session application:** set `timezone=UTC` on all engines — API, collector, migrate,
|
||||
and the `db migrate-to-postgres` one-shots — via the shared `create_database_engine()` and
|
||||
`DatabaseManager.__init__()` so there is one place that could be wrong. This gives the
|
||||
migration copy a consistent day boundary with the runtime queries. Resolved: apply
|
||||
everywhere.
|
||||
|
||||
## References
|
||||
- `docs/plans/20260613-2111-postgres-migration/plan.md` — the migration that introduced the
|
||||
regression; its Phase 3 Gate 3 promised the SQLite+Postgres test matrix this plan delivers.
|
||||
- `src/meshcore_hub/api/routes/dashboard.py:296,356,428,435` — the `func.date()` call sites and
|
||||
dict-key mismatch.
|
||||
- `tests/test_api/conftest.py:56` — SQLite-only test engine that hid the bug.
|
||||
- `tests/test_api/test_dashboard.py` — existing dashboard tests (all SQLite-only today).
|
||||
- `src/meshcore_hub/common/models/event_observer.py:144-152` — the existing dialect-aware
|
||||
upsert pattern; noted as the *only* dialect fork that should exist in the codebase (no truly
|
||||
portable SQL equivalent), and explicitly not mirrored by this fix.
|
||||
- Git: `1ba3e17` (merge of `feat/postgres-support`), `a554e09` (revert to PG 17),
|
||||
`34410de` (Postgres 18 → 17 revert merge).
|
||||
|
||||
## Review
|
||||
|
||||
**Status**: Approved
|
||||
|
||||
**Reviewed**: 2026-06-16
|
||||
|
||||
### Resolutions
|
||||
|
||||
- **Test matrix scope** — Parameterize the **full API test suite** to support both SQLite and
|
||||
Postgres backends, not just the dashboard tests. Any endpoint using SQLAlchemy constructs could
|
||||
have a latent dialect type mismatch; surfacing it all at once is cheaper than chasing
|
||||
one-by-one regression bugs. Postgres testing is run manually (local container), not in CI.
|
||||
|
||||
- **Cache invalidation** — No explicit cache flush needed. The default dashboard TTL is 30
|
||||
seconds (`config.py:314`, `app.py:93`); stale all-zero cached responses expire before a
|
||||
user loads the dashboard post-deploy. For operators with a substantially longer
|
||||
`REDIS_CACHE_TTL_DASHBOARD`, document the one-TTL wait in the upgrade notes.
|
||||
|
||||
- **UTC-session application** — Apply `timezone=UTC` on **all** engines (API, collector,
|
||||
migrate, `db migrate-to-postgres`) via the shared `create_database_engine()` and
|
||||
`DatabaseManager.__init__()` helpers, so there is a single point of control.
|
||||
|
||||
- **_date_bucket_key type coverage** — Handle `None` by returning it unchanged (no
|
||||
collision with string lookups). Clarified in the plan.
|
||||
|
||||
- **asyncpg timezone mechanism** — Use `server_settings={"timezone": "UTC"}` in asyncpg's
|
||||
`connect_args`, matching the existing `search_path` pattern at `database.py:201`, rather
|
||||
than a separate connect-event listener.
|
||||
|
||||
- **Postgres timezone not gated on schema** — The `-ctimezone=UTC` flag must be set
|
||||
unconditionally for all Postgres connections in `create_database_engine()`, not gated on
|
||||
`resolved_schema` being non-empty (a Postgres test engine without a custom schema still
|
||||
needs UTC).
|
||||
|
||||
- **node-count endpoint row access** — L435 (`dashboard.py`) uses `row[0]` index access
|
||||
while L313/L377 use `row.date` named access. The fix will switch L435 to named access
|
||||
(`row.date`, `row.count`) for consistency across all three endpoints.
|
||||
|
||||
### Remaining Action Items
|
||||
|
||||
- For deployments with a non-default (long) `REDIS_CACHE_TTL_DASHBOARD`, add a one-TTL
|
||||
wait note to `docs/upgrading.md`.
|
||||
@@ -0,0 +1,106 @@
|
||||
# Tasks: Fix flatlined dashboard charts on Postgres
|
||||
|
||||
> Generated from `plan.md` on 2026-06-16
|
||||
|
||||
## 1. Postgres UTC Session Configuration
|
||||
|
||||
- [x] **1.1** Add unconditional `-ctimezone=UTC` to the sync engine in `create_database_engine()` (`database.py:65-94`)
|
||||
- [x] In the Postgres branch (after the SQLite `check_same_thread` guard, near L76), ensure `connect_args["options"]` always includes `-ctimezone=UTC` — even when `resolved_schema` is empty
|
||||
- [x] When `resolved_schema` is set, combine both flags: `-csearch_path=<schema> -ctimezone=UTC`
|
||||
- [x] When `resolved_schema` is empty/None, set `connect_args["options"] = "-ctimezone=UTC"` alone
|
||||
- [x] Gate the entire block on `database_url` being Postgres (not SQLite) — mirror the existing `database_url.startswith("sqlite")` pattern
|
||||
- [x] **1.2** Add unconditional `"timezone": "UTC"` to the async engine in `DatabaseManager._ensure_async_engine()` (`database.py:188-204`)
|
||||
- [x] Ensure `async_connect_args["server_settings"]` always includes `"timezone": "UTC"` for Postgres
|
||||
- [x] When `self._schema` is set, merge into one `server_settings` dict: `{"search_path": schema, "timezone": "UTC"}`
|
||||
- [x] When `self._schema` is None (Postgres without custom schema), still create `server_settings = {"timezone": "UTC"}`
|
||||
- [x] Only apply for Postgres URLs, not SQLite (SQLite async engine has no `server_settings`)
|
||||
- [x] **1.3** Add unit tests verifying UTC timezone is configured on Postgres engines
|
||||
- [x] Test that `create_database_engine()` with a `postgresql://` URL sets `connect_args["options"]` containing `-ctimezone=UTC`
|
||||
- [x] Test that `create_database_engine()` with a Postgres URL + schema sets both `-csearch_path=` and `-ctimezone=UTC` in options
|
||||
- [x] Test that `create_database_engine()` with a `sqlite://` URL does NOT set timezone options
|
||||
- [x] Test that `DatabaseManager` async engine has `server_settings` with `"timezone": "UTC"` for Postgres URLs
|
||||
|
||||
## 2. Unified Date-Key Normalization
|
||||
|
||||
- [x] **2.1** Add `_date_bucket_key(value) -> str | None` helper in `src/meshcore_hub/api/routes/dashboard.py`
|
||||
- [x] Return `value` unchanged if it's already a `str`
|
||||
- [x] Return `value.strftime("%Y-%m-%d")` if it's a `datetime.date` or `datetime.datetime`
|
||||
- [x] Return `None` unchanged if `value is None` (won't collide with string lookups)
|
||||
- [x] Use a single `isinstance` ladder — no dialect check
|
||||
- [x] Add module-level `from datetime import date as date_type, datetime as datetime_type` imports if needed (or import `datetime` already present)
|
||||
- [x] **2.2** Apply `_date_bucket_key` to `get_activity` endpoint (`dashboard.py:313`)
|
||||
- [x] Change `{row.date: row.count for row in results}` to `{_date_bucket_key(row.date): row.count for row in results}`
|
||||
- [x] Update the comment on L312 from "date is already a string" to reflect the normalization
|
||||
- [x] **2.3** Apply `_date_bucket_key` to `get_message_activity` endpoint (`dashboard.py:377`)
|
||||
- [x] Change `{row.date: row.count for row in results}` to `{_date_bucket_key(row.date): row.count for row in results}`
|
||||
- [x] **2.4** Apply `_date_bucket_key` to `get_node_count_history` endpoint (`dashboard.py:435-436`)
|
||||
- [x] Switch from index access `row[0]: row[1]` to named access: `{_date_bucket_key(row.date): row.count for row in session.execute(per_day_query).all()}`
|
||||
- [x] The query already labels columns `.label("date")` and `.label("count")` (L430), so named access works
|
||||
- [x] **2.5** Add unit tests for `_date_bucket_key` helper
|
||||
- [x] Test with a `str` input (e.g. `"2026-06-15"`) — returns unchanged
|
||||
- [x] Test with a `datetime.date` input — returns `"%Y-%m-%d"` string
|
||||
- [x] Test with a `datetime.datetime` input — returns `"%Y-%m-%d"` string
|
||||
- [x] Test with `None` — returns `None`
|
||||
- [x] Test that `datetime.date(2026, 1, 5)` produces `"2026-01-05"` (zero-padded)
|
||||
|
||||
## 3. Dual-Backend Test Infrastructure
|
||||
|
||||
- [x] **3.1** Add backend-selection mechanism to `tests/test_api/conftest.py`
|
||||
- [x] Add a `TEST_DATABASE_BACKEND` env var check (or `TEST_POSTGRES_URL`), defaulting to `sqlite` so local `make test` is unchanged
|
||||
- [x] Add a session-scoped `db_backend` fixture that reads the env var and returns `"sqlite"` or `"postgres"`
|
||||
- [x] Skip Postgres tests if `TEST_DATABASE_BACKEND=postgres` is requested but no Postgres is reachable (use `pytest.fixture` with `pytest.skip()`)
|
||||
- [x] **3.2** Refactor `api_db_engine` fixture to support both backends (`conftest.py:48-71`)
|
||||
- [x] When backend is SQLite: keep existing behavior (temp file, `create_engine`, SQLite pragma)
|
||||
- [x] When backend is Postgres: build URL from `TEST_POSTGRES_URL` (or `DATABASE_URL`), use `create_engine` with the same `connect_args` as production (including `-ctimezone=UTC` from `create_database_engine`)
|
||||
- [x] Call `Base.metadata.create_all(engine)` in both cases
|
||||
- [x] Ensure `Base.metadata.drop_all(engine)` runs on teardown for both backends
|
||||
- [x] **3.3** Update `mock_db_manager` and app fixtures to use the parameterized engine
|
||||
- [x] `mock_db_manager` (`conftest.py:110-132`) — bind `sessionmaker` to whichever engine `api_db_engine` yields
|
||||
- [x] `app_no_auth` (`conftest.py:170-186`) — pass the correct `db_url` (SQLite file path or Postgres URL) to `create_app()`
|
||||
- [x] `app_with_auth` (`conftest.py:189-199`) — same update
|
||||
- [x] `_wire_overrides` (`conftest.py:148-167`) — bind `sessionmaker` to parameterized engine
|
||||
- [x] **3.4** Ensure `_truncate_all` works on Postgres (`conftest.py:74-78`)
|
||||
- [x] Verify `table.delete()` in reversed FK order works on Postgres (it should — same SQLAlchemy construct)
|
||||
- [x] If Postgres FK constraints cause issues, consider `TRUNCATE ... CASCADE` as an alternative, but prefer the current approach for backend neutrality
|
||||
- [x] **3.5** Verify the full existing API test suite passes unchanged on SQLite after refactoring
|
||||
- [x] Run `pytest --no-cov tests/test_api/` and confirm no regressions from the fixture changes
|
||||
- [x] Confirm `make test` still works identically with no env var set
|
||||
|
||||
## 4. Regression & Cross-Backend Tests
|
||||
|
||||
- [x] **4.1** Update `tests/test_api/test_dashboard.py` to assert non-zero buckets
|
||||
- [x] For `get_activity`: seed advertisements with known UTC dates within the window, assert at least one returned `data[].count >= 1`
|
||||
- [x] For `get_message_activity`: seed messages with known UTC dates, assert at least one `count >= 1`
|
||||
- [x] For `get_node_count`: seed nodes with `created_at` within the window, assert cumulative count increases (non-zero delta on seeded days)
|
||||
- [x] **4.2** Add deterministic-date seed fixtures for cross-backend tests
|
||||
- [x] Create fixtures that insert rows at explicit UTC timestamps (e.g. `datetime(2026, 6, 10, tzinfo=timezone.utc)`) rather than `datetime.now(timezone.utc)` — so both backends see the same bucket boundaries
|
||||
- [x] Ensure seeded timestamps fall within the endpoint's default day window (use `freezegun` or monkeypatch `utc_now` if needed, or seed dates relative to the current date)
|
||||
- [x] **4.3** Add cross-backend equivalence test for all three dashboard endpoints
|
||||
- [x] Create a test that runs against both SQLite and Postgres (via the parameterized fixture)
|
||||
- [x] Seed identical data on the active backend
|
||||
- [x] Assert the JSON response from each endpoint has the same `data` array (same dates, same counts)
|
||||
- [x] This test is the regression guard: if it passes on Postgres, the bug is fixed
|
||||
|
||||
## 5. Documentation
|
||||
|
||||
- [x] **5.1** Add upgrade note to `docs/upgrading.md` for long-TTL operators
|
||||
- [x] Document that the dashboard chart fix takes effect after one `REDIS_CACHE_TTL_DASHBOARD` period (default 30s)
|
||||
- [x] For operators with a substantially longer TTL, advise waiting one TTL period or flushing the three dashboard cache key prefixes (`dashboard:activity`, `dashboard:message-activity`, `dashboard:node-count`)
|
||||
|
||||
## 6. Verification
|
||||
|
||||
- [x] **6.1** Run linting and formatting checks
|
||||
- [x] `pre-commit run --all-files`
|
||||
- [x] **6.2** Run the full test suite on SQLite
|
||||
- [x] `pytest -nauto --no-cov` — confirm no regressions (1106 passed, 22 skipped)
|
||||
- [x] **6.3** Run the full API test suite on Postgres (local, via throwaway container)
|
||||
- [x] Spin up `postgres:17-alpine` on port 55432
|
||||
- [x] `TEST_DATABASE_BACKEND=postgres TEST_POSTGRES_URL=postgresql+psycopg2://postgres:postgres@localhost:55432/test pytest -nauto --no-cov tests/test_api/`
|
||||
- [x] Confirm all tests pass, especially the new dashboard regression tests (458 passed)
|
||||
- [ ] **6.4** Live verification against the compose stack
|
||||
- [ ] Bring the stack up on the Postgres profile with real data (or migrated SQLite data)
|
||||
- [ ] Hit `GET /api/v1/dashboard/activity`, `/dashboard/message-activity`, `/dashboard/node-count` — confirm non-zero buckets
|
||||
- [ ] Compare against the same endpoints on SQLite — confirm identical results
|
||||
- [ ] Spot-check a known-busy day against a raw `SELECT date(...), count(*) ...` query
|
||||
- [x] **6.5** Verify no `dialect.name` branch was introduced in `dashboard.py`
|
||||
- [x] Grep `dashboard.py` for `dialect` — confirm zero hits (the fix is dialect-neutral by construction)
|
||||
@@ -73,6 +73,21 @@ Downtime is required while writers are stopped; the source SQLite file is never
|
||||
|
||||
> **Managed Postgres / non-superuser roles:** the migration disables foreign-key triggers during the copy via `session_replication_role = replica`, which requires a superuser. When the target role is not a superuser (typical for managed Postgres), the command automatically falls back to copying in parent-first order instead. Pass `--no-replication-role` to force the fallback explicitly.
|
||||
|
||||
### Dashboard Chart Fix (Postgres)
|
||||
|
||||
After enabling Postgres, the dashboard charts (activity, message-activity, node-count) may render as flat zeros. This is a known issue caused by a dialect mismatch in the date-bucketing query — `func.date()` returns a `str` on SQLite but a `datetime.date` on Postgres, causing the dict lookup to miss. A fix normalizes the key to a canonical `"%Y-%m-%d"` string and pins the Postgres session timezone to UTC.
|
||||
|
||||
The fix takes effect **after one `REDIS_CACHE_TTL_DASHBOARD` period** (default 30 seconds) — stale all-zero cached responses expire automatically. For operators who have configured a substantially longer TTL, either wait one TTL period or flush the three dashboard cache key prefixes:
|
||||
|
||||
```bash
|
||||
redis-cli -h <redis-host> DEL \
|
||||
"$(echo -n 'hub:dashboard:activity*' | xargs redis-cli KEYS)" \
|
||||
"$(echo -n 'hub:dashboard:message-activity*' | xargs redis-cli KEYS)" \
|
||||
"$(echo -n 'hub:dashboard:node-count*' | xargs redis-cli KEYS)"
|
||||
```
|
||||
|
||||
No database migration or configuration change is required — the fix is automatic.
|
||||
|
||||
## v0.13.0
|
||||
|
||||
### Raw Packets (capture, browse, and search wire packets)
|
||||
|
||||
@@ -64,6 +64,10 @@ dev = [
|
||||
"beautifulsoup4>=4.12.0",
|
||||
"types-paho-mqtt>=1.6.0",
|
||||
"types-PyYAML>=6.0.0",
|
||||
# Postgres drivers needed to run the dual-backend test suite
|
||||
# (TEST_DATABASE_BACKEND=postgres) locally.
|
||||
"asyncpg>=0.28.0",
|
||||
"psycopg2-binary>=2.9.0",
|
||||
]
|
||||
postgres = [
|
||||
"asyncpg>=0.28.0",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
"""Dashboard API routes."""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from sqlalchemy import and_, case, func, or_, select
|
||||
@@ -60,6 +60,21 @@ def _flood_only_filter(
|
||||
)
|
||||
|
||||
|
||||
def _date_bucket_key(value: str | date | None) -> str | None:
|
||||
"""Coerce a DB-returned date bucket key to a canonical ``%Y-%m-%d`` string.
|
||||
|
||||
SQLite's ``func.date()`` returns a ``str``; Postgres returns a
|
||||
``datetime.date``. This normalizes both to the same string key so dict
|
||||
lookups by ``"%Y-%m-%d"`` succeed on either backend. ``None`` passes
|
||||
through unchanged.
|
||||
"""
|
||||
if isinstance(value, str):
|
||||
return value
|
||||
if isinstance(value, date):
|
||||
return value.strftime("%Y-%m-%d")
|
||||
return value
|
||||
|
||||
|
||||
@router.get("/stats", response_model=DashboardStats)
|
||||
@cached(
|
||||
"dashboard/stats",
|
||||
@@ -309,8 +324,9 @@ def get_activity(
|
||||
|
||||
results = session.execute(query).all()
|
||||
|
||||
# Build a dict of date -> count from results (date is already a string)
|
||||
counts_by_date = {row.date: row.count for row in results}
|
||||
# Build a dict of date -> count, normalizing the key to a string so it
|
||||
# works on both SQLite (func.date() returns str) and Postgres (returns date).
|
||||
counts_by_date = {_date_bucket_key(row.date): row.count for row in results}
|
||||
|
||||
# Generate all dates in the range, filling in zeros for missing days
|
||||
data = []
|
||||
@@ -374,7 +390,7 @@ def get_message_activity(
|
||||
)
|
||||
|
||||
results = session.execute(query).all()
|
||||
counts_by_date = {row.date: row.count for row in results}
|
||||
counts_by_date = {_date_bucket_key(row.date): row.count for row in results}
|
||||
|
||||
# Generate all dates in the range, filling in zeros for missing days
|
||||
data = []
|
||||
@@ -432,8 +448,9 @@ def get_node_count_history(
|
||||
.where(Node.created_at < end_date)
|
||||
.group_by(date_expr)
|
||||
)
|
||||
new_by_date: dict[str, int] = {
|
||||
row[0]: row[1] for row in session.execute(per_day_query).all()
|
||||
new_by_date = {
|
||||
_date_bucket_key(row.date): row._mapping["count"]
|
||||
for row in session.execute(per_day_query).all()
|
||||
}
|
||||
|
||||
data = []
|
||||
|
||||
@@ -69,12 +69,19 @@ def create_database_engine(
|
||||
if database_url.startswith("sqlite"):
|
||||
connect_args["check_same_thread"] = False
|
||||
|
||||
# Scope Postgres connections to the configured schema via search_path. This keeps
|
||||
# the models schema-agnostic (no hardcoded schema=) so the same code serves SQLite,
|
||||
# Scope Postgres connections to the configured schema via search_path and pin
|
||||
# the session timezone to UTC so func.date(<timestamptz>) truncates on the UTC
|
||||
# day boundary — matching SQLite, which stores UTC text. This keeps the models
|
||||
# schema-agnostic (no hardcoded schema=) so the same code serves SQLite,
|
||||
# single-instance Postgres, and multiple schema-isolated instances on one cluster.
|
||||
resolved_schema = _resolve_pg_schema(database_url, schema)
|
||||
if resolved_schema:
|
||||
connect_args["options"] = f"-csearch_path={resolved_schema}"
|
||||
is_postgres = database_url.startswith(("postgresql", "postgres"))
|
||||
if is_postgres:
|
||||
options_parts: list[str] = []
|
||||
if resolved_schema:
|
||||
options_parts.append(f"-csearch_path={resolved_schema}")
|
||||
options_parts.append("-ctimezone=UTC")
|
||||
connect_args["options"] = " ".join(options_parts)
|
||||
|
||||
# Size the pool above the default Starlette threadpool (~40 threads) so
|
||||
# concurrent request handlers don't block waiting for a connection. Applies
|
||||
@@ -194,11 +201,17 @@ class DatabaseManager:
|
||||
|
||||
async_url = _to_async_url(self.database_url)
|
||||
async_connect_args: dict[str, Any] = {}
|
||||
# asyncpg sets search_path via server_settings (not the libpq -c options
|
||||
# string the sync psycopg2 engine uses). self._schema is already resolved
|
||||
# (explicit arg or DATABASE_SCHEMA env) and None for SQLite.
|
||||
if self._schema:
|
||||
async_connect_args["server_settings"] = {"search_path": self._schema}
|
||||
# asyncpg sets search_path and timezone via server_settings (not the libpq
|
||||
# -c options string the sync psycopg2 engine uses). self._schema is already
|
||||
# resolved (explicit arg or DATABASE_SCHEMA env) and None for SQLite.
|
||||
# Timezone is pinned to UTC unconditionally for Postgres so day boundaries
|
||||
# match SQLite's UTC-text truncation.
|
||||
is_postgres = self.database_url.startswith(("postgresql", "postgres"))
|
||||
if is_postgres:
|
||||
server_settings: dict[str, str] = {"timezone": "UTC"}
|
||||
if self._schema:
|
||||
server_settings["search_path"] = self._schema
|
||||
async_connect_args["server_settings"] = server_settings
|
||||
self._async_engine = create_async_engine(
|
||||
async_url, echo=self._echo, connect_args=async_connect_args
|
||||
)
|
||||
|
||||
+107
-21
@@ -4,10 +4,12 @@ import os
|
||||
import tempfile
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timezone
|
||||
from typing import Generator
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import create_engine, event as sa_event
|
||||
from sqlalchemy import create_engine, event as sa_event, text
|
||||
from sqlalchemy.engine.url import make_url
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from meshcore_hub.api.app import create_app
|
||||
@@ -16,7 +18,7 @@ from meshcore_hub.api.dependencies import (
|
||||
get_mqtt_client,
|
||||
get_db_manager,
|
||||
)
|
||||
from meshcore_hub.common.database import DatabaseManager
|
||||
from meshcore_hub.common.database import DatabaseManager, create_database_engine
|
||||
from meshcore_hub.common.models import (
|
||||
Advertisement,
|
||||
Base,
|
||||
@@ -46,24 +48,110 @@ def test_db_path():
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def api_db_engine(test_db_path):
|
||||
"""Session-scoped SQLite engine. Schema is built once per pytest session.
|
||||
def db_backend() -> str:
|
||||
"""Active test database backend (``sqlite`` or ``postgres``).
|
||||
|
||||
Previously this was function-scoped and rebuilt ~15 tables for every test,
|
||||
costing ~0.2s/test. Promoting it to session scope eliminates that. Per-test
|
||||
isolation is handled by truncation in ``api_db_session``.
|
||||
Controlled by ``TEST_DATABASE_BACKEND`` env var (default: ``sqlite``).
|
||||
When ``postgres``, ``TEST_POSTGRES_URL`` must also be set.
|
||||
"""
|
||||
db_url = f"sqlite:///{test_db_path}"
|
||||
engine = create_engine(
|
||||
db_url,
|
||||
connect_args={"check_same_thread": False},
|
||||
)
|
||||
backend = os.environ.get("TEST_DATABASE_BACKEND", "sqlite").lower()
|
||||
if backend not in ("sqlite", "postgres"):
|
||||
raise ValueError(
|
||||
f"TEST_DATABASE_BACKEND must be 'sqlite' or 'postgres', got: {backend}"
|
||||
)
|
||||
return backend
|
||||
|
||||
@sa_event.listens_for(engine, "connect")
|
||||
def set_sqlite_pragma(dbapi_connection: object, connection_record: object) -> None:
|
||||
cursor = dbapi_connection.cursor() # type: ignore[attr-defined]
|
||||
cursor.execute("PRAGMA foreign_keys=ON")
|
||||
cursor.close()
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def db_url(db_backend: str, test_db_path: str, request) -> Generator[str, None, None]:
|
||||
"""Database URL for the active backend.
|
||||
|
||||
For Postgres, each pytest-xdist worker gets its own database (e.g.
|
||||
``test_gw0``) to avoid truncation races between parallel workers.
|
||||
"""
|
||||
if db_backend == "postgres":
|
||||
env_url = os.environ.get("TEST_POSTGRES_URL")
|
||||
if not env_url:
|
||||
pytest.skip(
|
||||
"TEST_DATABASE_BACKEND=postgres but TEST_POSTGRES_URL is not set; "
|
||||
"e.g. TEST_POSTGRES_URL=postgresql+psycopg2://postgres:postgres@localhost:55432/test"
|
||||
)
|
||||
assert env_url is not None
|
||||
|
||||
worker_id = "master"
|
||||
if hasattr(request.config, "workerinput"):
|
||||
worker_id = request.config.workerinput["workerid"]
|
||||
|
||||
base_url = make_url(env_url)
|
||||
worker_db = f"{base_url.database}_{worker_id}"
|
||||
worker_url = base_url.set(database=worker_db).render_as_string(
|
||||
hide_password=False
|
||||
)
|
||||
|
||||
admin_url = base_url.set(database="postgres")
|
||||
admin_engine = create_engine(
|
||||
admin_url.render_as_string(hide_password=False),
|
||||
isolation_level="AUTOCOMMIT",
|
||||
)
|
||||
try:
|
||||
with admin_engine.connect() as conn:
|
||||
exists = conn.execute(
|
||||
text("SELECT 1 FROM pg_database WHERE datname = :name"),
|
||||
{"name": worker_db},
|
||||
).scalar()
|
||||
if not exists:
|
||||
conn.execute(text(f'CREATE DATABASE "{worker_db}"'))
|
||||
finally:
|
||||
admin_engine.dispose()
|
||||
|
||||
yield worker_url
|
||||
|
||||
admin_engine = create_engine(
|
||||
admin_url.render_as_string(hide_password=False),
|
||||
isolation_level="AUTOCOMMIT",
|
||||
)
|
||||
try:
|
||||
with admin_engine.connect() as conn:
|
||||
conn.execute(
|
||||
text(
|
||||
"SELECT pg_terminate_backend(pid) "
|
||||
"FROM pg_stat_activity "
|
||||
"WHERE datname = :name AND pid <> pg_backend_pid()"
|
||||
),
|
||||
{"name": worker_db},
|
||||
)
|
||||
conn.execute(text(f'DROP DATABASE IF EXISTS "{worker_db}"'))
|
||||
finally:
|
||||
admin_engine.dispose()
|
||||
else:
|
||||
yield f"sqlite:///{test_db_path}"
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def api_db_engine(db_url: str, db_backend: str):
|
||||
"""Session-scoped database engine. Schema is built once per pytest session.
|
||||
|
||||
For Postgres, uses the production ``create_database_engine`` factory so the
|
||||
test exercises the same ``connect_args`` (including ``-ctimezone=UTC``).
|
||||
Each xdist worker has its own database (see ``db_url``), so no locking
|
||||
is needed.
|
||||
"""
|
||||
if db_backend == "postgres":
|
||||
engine = create_database_engine(db_url)
|
||||
Base.metadata.drop_all(engine)
|
||||
else:
|
||||
engine = create_engine(
|
||||
db_url,
|
||||
connect_args={"check_same_thread": False},
|
||||
)
|
||||
|
||||
@sa_event.listens_for(engine, "connect")
|
||||
def set_sqlite_pragma(
|
||||
dbapi_connection: object, connection_record: object
|
||||
) -> None:
|
||||
cursor = dbapi_connection.cursor() # type: ignore[attr-defined]
|
||||
cursor.execute("PRAGMA foreign_keys=ON")
|
||||
cursor.close()
|
||||
|
||||
Base.metadata.create_all(engine)
|
||||
yield engine
|
||||
@@ -168,7 +256,7 @@ def _wire_overrides(app, api_db_engine, mock_mqtt, mock_db_manager) -> None:
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_no_auth(test_db_path, api_db_engine, mock_mqtt, mock_db_manager):
|
||||
def app_no_auth(db_url, api_db_engine, mock_mqtt, mock_db_manager):
|
||||
"""Module-scoped FastAPI app with no authentication.
|
||||
|
||||
Built once per test module; ``create_app`` is the second-most expensive
|
||||
@@ -176,7 +264,6 @@ def app_no_auth(test_db_path, api_db_engine, mock_mqtt, mock_db_manager):
|
||||
``_isolate_db_global`` autouse fixture handles the global ``_db_manager``
|
||||
so this fixture doesn't need a ``with patch(...)`` context.
|
||||
"""
|
||||
db_url = f"sqlite:///{test_db_path}"
|
||||
app = create_app(
|
||||
database_url=db_url,
|
||||
read_key=None,
|
||||
@@ -187,9 +274,8 @@ def app_no_auth(test_db_path, api_db_engine, mock_mqtt, mock_db_manager):
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_with_auth(test_db_path, api_db_engine, mock_mqtt, mock_db_manager):
|
||||
def app_with_auth(db_url, api_db_engine, mock_mqtt, mock_db_manager):
|
||||
"""Module-scoped FastAPI app with authentication enabled."""
|
||||
db_url = f"sqlite:///{test_db_path}"
|
||||
app = create_app(
|
||||
database_url=db_url,
|
||||
read_key="test-read-key",
|
||||
|
||||
@@ -1,10 +1,11 @@
|
||||
"""Tests for dashboard API routes."""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
from meshcore_hub.api.routes.dashboard import _date_bucket_key
|
||||
from meshcore_hub.common.models import (
|
||||
Advertisement,
|
||||
Message,
|
||||
@@ -15,6 +16,31 @@ from meshcore_hub.common.models import (
|
||||
from meshcore_hub.common.models import UserProfile
|
||||
|
||||
|
||||
class TestDateBucketKey:
|
||||
"""Unit tests for the _date_bucket_key dialect-neutral normalization helper."""
|
||||
|
||||
def test_str_passthrough(self) -> None:
|
||||
"""SQLite returns str — pass through unchanged."""
|
||||
assert _date_bucket_key("2026-06-15") == "2026-06-15"
|
||||
|
||||
def test_date_object_normalized(self) -> None:
|
||||
"""Postgres returns datetime.date — coerce to %Y-%m-%d string."""
|
||||
assert _date_bucket_key(date(2026, 1, 5)) == "2026-01-05"
|
||||
|
||||
def test_datetime_object_normalized(self) -> None:
|
||||
"""Postgres may return datetime.datetime — coerce to %Y-%m-%d string."""
|
||||
dt = datetime(2026, 6, 15, 14, 30, 0, tzinfo=timezone.utc)
|
||||
assert _date_bucket_key(dt) == "2026-06-15"
|
||||
|
||||
def test_none_passthrough(self) -> None:
|
||||
"""None passes through unchanged (no collision with string lookups)."""
|
||||
assert _date_bucket_key(None) is None
|
||||
|
||||
def test_zero_padded_date(self) -> None:
|
||||
"""Single-digit months/days are zero-padded."""
|
||||
assert _date_bucket_key(date(2026, 1, 5)) == "2026-01-05"
|
||||
|
||||
|
||||
class TestDashboardStats:
|
||||
"""Tests for GET /dashboard/stats endpoint."""
|
||||
|
||||
@@ -231,9 +257,16 @@ class TestDashboardActivity:
|
||||
response = client_no_auth.get("/api/v1/dashboard/activity")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
# At least one day should have a count > 0
|
||||
total_count = sum(point["count"] for point in data["data"])
|
||||
assert total_count >= 1
|
||||
# The seeded advertisement was yesterday — its bucket must be non-zero.
|
||||
# This catches the Postgres flatline bug where func.date() returns a
|
||||
# date object that never matches the string key.
|
||||
yesterday_str = (datetime.now(timezone.utc) - timedelta(days=1)).strftime(
|
||||
"%Y-%m-%d"
|
||||
)
|
||||
yesterday_point = next(p for p in data["data"] if p["date"] == yesterday_str)
|
||||
assert yesterday_point["count"] >= 1
|
||||
|
||||
|
||||
class TestMessageActivity:
|
||||
@@ -291,9 +324,14 @@ class TestMessageActivity:
|
||||
response = client_no_auth.get("/api/v1/dashboard/message-activity")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
# At least one day should have a count > 0
|
||||
total_count = sum(point["count"] for point in data["data"])
|
||||
assert total_count >= 1
|
||||
# The seeded message was yesterday — its bucket must be non-zero.
|
||||
yesterday_str = (datetime.now(timezone.utc) - timedelta(days=1)).strftime(
|
||||
"%Y-%m-%d"
|
||||
)
|
||||
yesterday_point = next(p for p in data["data"] if p["date"] == yesterday_str)
|
||||
assert yesterday_point["count"] >= 1
|
||||
|
||||
|
||||
class TestNodeCountHistory:
|
||||
@@ -356,6 +394,16 @@ class TestNodeCountHistory:
|
||||
# At least one day should have a count > 0 (cumulative)
|
||||
# The last day should have count >= 1
|
||||
assert data["data"][-1]["count"] >= 1
|
||||
# The node was created yesterday — the cumulative count should step
|
||||
# up at yesterday's bucket and stay >= 1 for all subsequent days.
|
||||
yesterday_str = (datetime.now(timezone.utc) - timedelta(days=1)).strftime(
|
||||
"%Y-%m-%d"
|
||||
)
|
||||
yesterday_idx = next(
|
||||
i for i, p in enumerate(data["data"]) if p["date"] == yesterday_str
|
||||
)
|
||||
assert data["data"][yesterday_idx]["count"] >= 1
|
||||
assert all(p["count"] >= 1 for p in data["data"][yesterday_idx:])
|
||||
|
||||
def test_node_count_is_cumulative_with_baseline(
|
||||
self, client_no_auth, api_db_session
|
||||
@@ -770,3 +818,88 @@ class TestDashboardChannelVisibility:
|
||||
assert str(pub_idx) in data["channel_message_counts"]
|
||||
assert str(mem_idx) in data["channel_message_counts"]
|
||||
assert str(adm_idx) not in data["channel_message_counts"]
|
||||
|
||||
|
||||
class TestDashboardDateBucketRegression:
|
||||
"""Regression tests for the Postgres date-bucket flatline bug.
|
||||
|
||||
These tests seed data at deterministic UTC timestamps and assert the
|
||||
specific date buckets have non-zero counts. On SQLite (where
|
||||
func.date() returns str) these always passed. On Postgres (where
|
||||
func.date() returns date) these would have failed before the
|
||||
_date_bucket_key fix, because the dict lookup by string key would
|
||||
miss the date-object key.
|
||||
|
||||
Run against Postgres with::
|
||||
|
||||
TEST_DATABASE_BACKEND=postgres \\
|
||||
TEST_POSTGRES_URL=postgresql+psycopg2://postgres:postgres@localhost:55432/test \\
|
||||
pytest tests/test_api/test_dashboard.py -k Regression
|
||||
"""
|
||||
|
||||
def test_activity_nonzero_on_seeded_day(self, client_no_auth, api_db_session):
|
||||
"""Activity chart shows non-zero count for the seeded day."""
|
||||
two_days_ago = (datetime.now(timezone.utc) - timedelta(days=2)).replace(
|
||||
hour=12, minute=0, second=0, microsecond=0
|
||||
)
|
||||
api_db_session.add(
|
||||
Advertisement(
|
||||
public_key="a" * 64,
|
||||
name="RegressionNode",
|
||||
adv_type="REPEATER",
|
||||
route_type="flood",
|
||||
received_at=two_days_ago,
|
||||
)
|
||||
)
|
||||
api_db_session.commit()
|
||||
|
||||
data = client_no_auth.get("/api/v1/dashboard/activity").json()
|
||||
seeded_str = two_days_ago.strftime("%Y-%m-%d")
|
||||
seeded_point = next(p for p in data["data"] if p["date"] == seeded_str)
|
||||
assert seeded_point["count"] == 1
|
||||
|
||||
def test_message_activity_nonzero_on_seeded_day(
|
||||
self, client_no_auth, api_db_session
|
||||
):
|
||||
"""Message activity chart shows non-zero count for the seeded day."""
|
||||
two_days_ago = (datetime.now(timezone.utc) - timedelta(days=2)).replace(
|
||||
hour=12, minute=0, second=0, microsecond=0
|
||||
)
|
||||
api_db_session.add(
|
||||
Message(
|
||||
message_type="direct",
|
||||
pubkey_prefix="abc123",
|
||||
text="regression test",
|
||||
received_at=two_days_ago,
|
||||
)
|
||||
)
|
||||
api_db_session.commit()
|
||||
|
||||
data = client_no_auth.get("/api/v1/dashboard/message-activity").json()
|
||||
seeded_str = two_days_ago.strftime("%Y-%m-%d")
|
||||
seeded_point = next(p for p in data["data"] if p["date"] == seeded_str)
|
||||
assert seeded_point["count"] == 1
|
||||
|
||||
def test_node_count_steps_up_on_seeded_day(self, client_no_auth, api_db_session):
|
||||
"""Node count chart steps up on the seeded creation day."""
|
||||
two_days_ago = (datetime.now(timezone.utc) - timedelta(days=2)).replace(
|
||||
hour=12, minute=0, second=0, microsecond=0
|
||||
)
|
||||
api_db_session.add(
|
||||
Node(
|
||||
public_key="b" * 64,
|
||||
name="RegressionNode",
|
||||
created_at=two_days_ago,
|
||||
)
|
||||
)
|
||||
api_db_session.commit()
|
||||
|
||||
data = client_no_auth.get("/api/v1/dashboard/node-count").json()
|
||||
seeded_str = two_days_ago.strftime("%Y-%m-%d")
|
||||
seeded_idx = next(
|
||||
i for i, p in enumerate(data["data"]) if p["date"] == seeded_str
|
||||
)
|
||||
# Before the seeded day: 0 nodes. On/after: 1.
|
||||
assert data["data"][seeded_idx]["count"] == 1
|
||||
if seeded_idx > 0:
|
||||
assert data["data"][seeded_idx - 1]["count"] == 0
|
||||
|
||||
@@ -95,13 +95,23 @@ class TestListProfiles:
|
||||
api_db_session.add(adoption)
|
||||
api_db_session.commit()
|
||||
|
||||
api_db_session.execute(text("PRAGMA foreign_keys=OFF"))
|
||||
# Temporarily disable FK enforcement to simulate an orphaned adoption
|
||||
# (node deleted while the adoption record persists). SQLite uses
|
||||
# PRAGMA; Postgres uses session_replication_role = replica.
|
||||
dialect = api_db_session.bind.dialect.name # type: ignore[union-attr]
|
||||
if dialect == "postgresql":
|
||||
api_db_session.execute(text("SET session_replication_role = replica"))
|
||||
else:
|
||||
api_db_session.execute(text("PRAGMA foreign_keys=OFF"))
|
||||
api_db_session.execute(
|
||||
text("DELETE FROM nodes WHERE id = :id"),
|
||||
{"id": sample_node.id},
|
||||
)
|
||||
api_db_session.commit()
|
||||
api_db_session.execute(text("PRAGMA foreign_keys=ON"))
|
||||
if dialect == "postgresql":
|
||||
api_db_session.execute(text("SET session_replication_role = DEFAULT"))
|
||||
else:
|
||||
api_db_session.execute(text("PRAGMA foreign_keys=ON"))
|
||||
|
||||
response = client_no_auth.get(
|
||||
"/api/v1/user/profiles",
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
"""Tests for database engine configuration."""
|
||||
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import text
|
||||
|
||||
from meshcore_hub.common.database import (
|
||||
DatabaseManager,
|
||||
_resolve_pg_schema,
|
||||
_to_async_url,
|
||||
create_database_engine,
|
||||
@@ -78,3 +80,72 @@ class TestSchemaResolution:
|
||||
def test_none_when_no_env(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.delenv("DATABASE_SCHEMA", raising=False)
|
||||
assert _resolve_pg_schema("postgresql://u@h/db", None) is None
|
||||
|
||||
|
||||
class TestPostgresSessionTimezone:
|
||||
"""Verify Postgres connections are pinned to UTC at the engine level.
|
||||
|
||||
func.date(<timestamptz>) truncates on the session timezone's day boundary.
|
||||
The collector writes UTC, so the session must be UTC for day buckets to
|
||||
match SQLite's UTC-text truncation.
|
||||
"""
|
||||
|
||||
def test_sync_engine_sets_timezone_utc_without_schema(
|
||||
self, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Postgres engine without a schema still pins timezone=UTC."""
|
||||
monkeypatch.delenv("DATABASE_SCHEMA", raising=False)
|
||||
with patch("meshcore_hub.common.database.create_engine") as mock_create:
|
||||
create_database_engine("postgresql://u:p@h/db")
|
||||
_, kwargs = mock_create.call_args
|
||||
assert kwargs["connect_args"]["options"] == "-ctimezone=UTC"
|
||||
|
||||
def test_sync_engine_timezone_utc_with_schema(self) -> None:
|
||||
"""Postgres engine with a schema sets both search_path and timezone."""
|
||||
with patch("meshcore_hub.common.database.create_engine") as mock_create:
|
||||
create_database_engine("postgresql://u:p@h/db", schema="meshcorehub")
|
||||
_, kwargs = mock_create.call_args
|
||||
options = kwargs["connect_args"]["options"]
|
||||
assert "-csearch_path=meshcorehub" in options
|
||||
assert "-ctimezone=UTC" in options
|
||||
|
||||
def test_sqlite_engine_has_no_timezone_options(self, tmp_path: Path) -> None:
|
||||
"""SQLite engines must not set timezone options."""
|
||||
engine = create_database_engine(f"sqlite:///{tmp_path / 'x.db'}")
|
||||
try:
|
||||
assert "options" not in engine.url.query
|
||||
finally:
|
||||
engine.dispose()
|
||||
|
||||
def test_async_engine_sets_server_settings_timezone_utc(
|
||||
self, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""asyncpg engine gets server_settings with timezone=UTC."""
|
||||
monkeypatch.delenv("DATABASE_SCHEMA", raising=False)
|
||||
manager = DatabaseManager.__new__(DatabaseManager)
|
||||
manager.database_url = "postgresql://u:p@h/db"
|
||||
manager._echo = False
|
||||
manager._schema = None
|
||||
manager._async_engine = None
|
||||
manager._async_session_factory = None
|
||||
|
||||
with patch("meshcore_hub.common.database.create_async_engine") as mock_async:
|
||||
manager._ensure_async_engine()
|
||||
_, kwargs = mock_async.call_args
|
||||
assert kwargs["connect_args"]["server_settings"] == {"timezone": "UTC"}
|
||||
|
||||
def test_async_engine_sets_server_settings_with_schema(self) -> None:
|
||||
"""asyncpg engine with schema gets both search_path and timezone."""
|
||||
manager = DatabaseManager.__new__(DatabaseManager)
|
||||
manager.database_url = "postgresql://u:p@h/db"
|
||||
manager._echo = False
|
||||
manager._schema = "meshcorehub"
|
||||
manager._async_engine = None
|
||||
manager._async_session_factory = None
|
||||
|
||||
with patch("meshcore_hub.common.database.create_async_engine") as mock_async:
|
||||
manager._ensure_async_engine()
|
||||
_, kwargs = mock_async.call_args
|
||||
server_settings = kwargs["connect_args"]["server_settings"]
|
||||
assert server_settings["timezone"] == "UTC"
|
||||
assert server_settings["search_path"] == "meshcorehub"
|
||||
|
||||
Reference in New Issue
Block a user