From 30e8e88ee9b8c5c79219df70f45632fc66044b96 Mon Sep 17 00:00:00 2001 From: Louis King Date: Wed, 17 Jun 2026 15:14:53 +0100 Subject: [PATCH 1/2] docs: add SQLite deprecation notice and consolidate DB docs Replace the lingering v0.9 'Breaking Changes' alert with a concise v0.14 'DEPRECATION NOTICE' for SQLite (dual compatibility for ~3 months, then PostgreSQL-only). Move all database-specific instructions (SQLite + PostgreSQL) out of the README into a new canonical docs/database.md covering: - SQLite zero-config default (DATA_HOME / meshcore.db, WAL/single-host) - PostgreSQL: DATABASE_* env vars, bundled Docker profile, production role/database provisioning (mirrors the ipnet-mesh/infrastructure init script), managed/external Postgres and DATABASE_URL - Schema-per-instance (search_path) isolation for multiple instances on a shared cluster - Pointer to the SQLite->PostgreSQL migration runbook in upgrading.md Update the README Multi-Instance Deployments and Scaling the API sections to link to docs/database.md, and add the new doc to the docs list and project tree. Add a pointer in .env.example and a Postgres note in AGENTS.md. Consolidate docs/upgrading.md v0.14: move the env-var/schema/provisioning reference to docs/database.md (single source of truth) and keep only the upgrade-time migration runbook and dashboard chart fix. --- .env.example | 4 ++ AGENTS.md | 3 ++ README.md | 34 ++++---------- docs/database.md | 110 ++++++++++++++++++++++++++++++++++++++++++++++ docs/upgrading.md | 43 +++--------------- 5 files changed, 131 insertions(+), 63 deletions(-) create mode 100644 docs/database.md diff --git a/.env.example b/.env.example index f73caf4..683cfc0 100644 --- a/.env.example +++ b/.env.example @@ -77,6 +77,10 @@ DATA_HOME=./data # POSTGRES_USER/PASSWORD/DB from DATABASE_USER/PASSWORD/NAME). You must also # activate the compose 'postgres' profile, e.g. `docker compose --profile postgres up`. # +# See docs/database.md for the full backend reference: production role/database +# provisioning, managed/external Postgres, and schema-per-instance (search_path) +# isolation for multiple instances sharing one cluster. +# # DATABASE_BACKEND=postgres # DATABASE_HOST=postgres # DATABASE_PORT=5432 diff --git a/AGENTS.md b/AGENTS.md index b88e5db..e0b4ef6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -66,9 +66,12 @@ pre-commit run --all-files ## Database & Ops +The default backend is **SQLite** (zero-config, file at `${DATA_HOME}/collector/meshcore.db`). **PostgreSQL** is also supported via `DATABASE_BACKEND=postgres` — see `docs/database.md` for the full backend reference, production provisioning, and schema-per-instance setup. Migrations are backend-agnostic; the commands below work for both. + ```bash # --- LOCAL (venv): sync the volume DB to ./meshcore.db, then author a migration # Volume name is ${COMPOSE_PROJECT_NAME:-hub}_data (default: hub_data) +# (SQLite only — for Postgres, point the migration env at the cluster directly) docker run -it --rm -v hub_data:/data -v "$PWD":/pwd ubuntu cp /data/collector/meshcore.db /pwd/meshcore.db meshcore-hub db revision --autogenerate -m "description" diff --git a/README.md b/README.md index 1430cfb..90ba122 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ Python 3.14+ platform for managing and orchestrating MeshCore mesh networks. > [!WARNING] -> **BREAKING CHANGES** - The latest release replaces Mosquitto with a JWT-based MQTT broker, removes the proprietary receiver service in favor of [meshcore-packet-capture](https://github.com/agessaman/meshcore-packet-capture), and renames `receiver_node_id` to `observer_node_id` in the database. If upgrading from a previous version, see [docs/upgrading.md](docs/upgrading.md) for migration steps. +> **DEPRECATION NOTICE** — v0.14 adds PostgreSQL support (`DATABASE_BACKEND=postgres`); SQLite remains the zero-config default. SQLite support will be maintained for at least the next few releases (~3 months), then removed in favour of PostgreSQL-only. See [docs/database.md](docs/database.md) to switch backends and [docs/upgrading.md](docs/upgrading.md) to migrate. ![MeshCore Hub Web Dashboard](docs/images/web.png) @@ -231,6 +231,8 @@ TRAEFIK_PRIORITY=20 This ensures `beta.example.com` (priority 20) is matched before the production wildcard `*.example.com` (priority 10). For other services on the same network (e.g., an MQTT broker at `mqtt.example.com`), use an even higher priority (e.g., 30). +> **Shared Postgres cluster:** the setup above runs each instance in its own directory with its own volumes (the default SQLite path). To instead run several instances (e.g. `prod` + `stg`) against **one** PostgreSQL cluster — isolated via a per-instance schema (`search_path`) — see [docs/database.md](docs/database.md#schema-per-instance-search_path). + #### Scaling the API The API is read-mostly and holds no per-process state — the response cache lives in Redis and authentication is stateless — so it scales across multiple worker processes. Set `API_WORKERS` to run more than one worker in a single container: @@ -244,7 +246,7 @@ Each worker is an independent process sharing one listening socket, so the kerne Pick a worker count around the number of CPU cores available to the container; start with `2`–`4` and measure under realistic load. -**SQLite caveat:** all workers share the same SQLite file on the same host. WAL mode (enabled automatically) allows concurrent readers alongside the single writer (the collector), so reads scale — but **writes do not**, and this does not extend across multiple hosts (a network filesystem breaks SQLite locking). To scale the API across hosts, switch to PostgreSQL (`DATABASE_BACKEND=postgres`); the API requires no code changes for this. See [Database Backend](#database-backend). +**SQLite caveat:** all workers share one SQLite file on the same host (WAL mode lets concurrent readers coexist with the single writer), but writes do not scale and this does not extend across hosts. To scale the API across hosts, switch to PostgreSQL (`DATABASE_BACKEND=postgres`) — the API requires no code changes. See [docs/database.md](docs/database.md) for backend setup and the SQLite → Postgres migration runbook. > Prefer `API_WORKERS` over running multiple `api` containers (`--scale api=N`): the `api` service uses a fixed `container_name`, and one process-managed container per stack keeps logs, health checks, and monitoring simple. @@ -346,31 +348,11 @@ All components are configured via environment variables. Create a `.env` file or > **Note:** `MQTT_PREFIX` also accepts the legacy alias `MQTT_TOPIC_PREFIX` for backward compatibility. -### Database Backend +### Database -MeshCore Hub defaults to **SQLite** (zero-config, single host). Set `DATABASE_BACKEND=postgres` to switch to **PostgreSQL** for write scaling and multi-host deployments. Postgres is opt-in — leave these unset to keep using SQLite. +MeshCore Hub defaults to **SQLite** (zero-config, single host). Set `DATABASE_BACKEND=postgres` to switch to **PostgreSQL** for write scaling, multi-host deployments, and multiple instances sharing one cluster via schema-per-instance. Postgres is opt-in — leave the `DATABASE_*` variables unset to keep using SQLite. -| Variable | Default | Description | -| ------------------- | ------------- | --------------------------------------------------------------------------------------- | -| `DATABASE_BACKEND` | `sqlite` | `sqlite` or `postgres`. Explicit switch — Postgres is never selected implicitly. | -| `DATABASE_HOST` | `postgres` | Postgres hostname (`postgres` = bundled container service name) | -| `DATABASE_PORT` | `5432` | Postgres port | -| `DATABASE_NAME` | `meshcorehub` | Database name | -| `DATABASE_SCHEMA` | `meshcorehub` | Schema (search_path). Set a distinct value per instance on a shared cluster | -| `DATABASE_USER` | `meshcorehub` | Role name | -| `DATABASE_PASSWORD` | _(none)_ | **Required** for Postgres | -| `DATABASE_URL` | _(none)_ | Advanced: full SQLAlchemy URL; overrides all of the above | - -**Docker:** Postgres is bundled behind the `postgres` profile. The container's credentials/name are derived from the `DATABASE_*` values (single source of truth). - -```bash -docker compose --profile postgres --profile core up # Start on Postgres -docker compose --profile core up # Start on SQLite (default) -``` - -**Schema-per-instance:** several instances (e.g. `prod`, `stg`) can share one Postgres cluster, each isolated to its own schema via `search_path` — give each a distinct `DATABASE_SCHEMA`. The schema is created automatically on `db upgrade`. - -See [docs/upgrading.md](docs/upgrading.md#optional-postgresql-backend) for the setup reference and the SQLite → Postgres data-migration runbook. +See [docs/database.md](docs/database.md) for the full backend reference: environment variables, the bundled Docker profile, production role/database provisioning, schema-per-instance isolation, and the SQLite → PostgreSQL migration runbook. ### Collector Settings @@ -680,6 +662,7 @@ meshcore-hub/ │ ├── images/ # Screenshots and images │ ├── hosting/ # Reverse proxy hosting guides │ ├── content.md # Custom content setup guide +│ ├── database.md # Database backends (SQLite/PostgreSQL) reference │ ├── i18n.md # Translation reference guide │ ├── letsmesh.md # LetsMesh packet decoding details │ ├── seeding.md # Seed data format and import guide @@ -692,6 +675,7 @@ meshcore-hub/ ## Documentation - [SCHEMAS.md](SCHEMAS.md) - MeshCore event schemas +- [docs/database.md](docs/database.md) - Database backends (SQLite/PostgreSQL) and migration - [docs/upgrading.md](docs/upgrading.md) - Upgrade guide for breaking changes - [docs/letsmesh.md](docs/letsmesh.md) - LetsMesh packet decoding details - [docs/seeding.md](docs/seeding.md) - Seed data format and import guide diff --git a/docs/database.md b/docs/database.md new file mode 100644 index 0000000..fa49736 --- /dev/null +++ b/docs/database.md @@ -0,0 +1,110 @@ +# Database + +MeshCore Hub supports two database backends: **SQLite** (the zero-config default) and **PostgreSQL** (optional, for write scaling and multi-host deployments). Postgres is opt-in — leave the `DATABASE_*` variables unset to keep using SQLite. + +> [!NOTE] +> As of v0.14, SQLite is **deprecated** in favour of PostgreSQL. SQLite remains the default and continues to work unchanged; support will be removed in a future release (at least 3 months out). New deployments that need to scale across hosts should pick Postgres. Existing SQLite deployments can move to Postgres with a one-command migration — see [Migrating from SQLite to PostgreSQL](#migrating-from-sqlite-to-postgresql) and [upgrading.md](upgrading.md). + +## SQLite (default) + +SQLite needs no configuration. The database file is created automatically on first run and lives under [`DATA_HOME`](../README.md): + +``` +${DATA_HOME}/collector/meshcore.db +``` + +In Docker Compose this is the `hub_data` volume (`${COMPOSE_PROJECT_NAME:-hub}_data`). WAL mode is enabled automatically, allowing concurrent readers alongside the collector's single writer. + +**Limitations:** writes are serialised to one process, and SQLite's file locking does **not** work over network filesystems — it caps you at a single host. To scale writes or run the stack across multiple hosts, switch to PostgreSQL. + +## PostgreSQL + +Set `DATABASE_BACKEND=postgres` and fill in the `DATABASE_*` connection variables. Postgres is never selected implicitly — the explicit switch avoids a silent backend change. + +### Environment variables + +| Variable | Default | Description | +| ------------------- | ------------- | --------------------------------------------------------------------------------------- | +| `DATABASE_BACKEND` | `sqlite` | `sqlite` or `postgres`. Explicit switch — Postgres is never selected implicitly. | +| `DATABASE_HOST` | `postgres` | Postgres hostname (`postgres` = bundled container service name) | +| `DATABASE_PORT` | `5432` | Postgres port | +| `DATABASE_NAME` | `meshcorehub` | Database name | +| `DATABASE_SCHEMA` | `meshcorehub` | Schema (search_path). Set a distinct value per instance on a shared cluster | +| `DATABASE_USER` | `meshcorehub` | Role name | +| `DATABASE_PASSWORD` | _(none)_ | **Required** for Postgres. Generate one, e.g. `openssl rand -base64 32` | +| `DATABASE_URL` | _(none)_ | Advanced: full SQLAlchemy URL; overrides all of the above | + +The bundled `postgres` container derives its `POSTGRES_USER` / `POSTGRES_PASSWORD` / `POSTGRES_DB` from the same `DATABASE_USER` / `DATABASE_PASSWORD` / `DATABASE_NAME` values — one source of truth. + +### Docker (bundled container) + +Postgres is bundled behind the `postgres` compose profile: + +```bash +# Start the stack on Postgres (bundled container) +docker compose -f docker-compose.yml -f docker-compose.dev.yml \ + --profile postgres --profile core up -d + +# Start on SQLite (default — no postgres profile) +docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core up -d +``` + +### Production provisioning (role and database) + +The bundled container provisions the role and database for you on first start from the `DATABASE_*` values. For a **managed or external** Postgres, create them once before pointing Hub at it. This mirrors the init script used in the [ipnet-mesh/infrastructure](https://github.com/ipnet-mesh/infrastructure/blob/main/etc/postgres/init/02_meshcorehub_db.sh) cluster: + +```sql +-- Run once as a superuser/admin role on the target cluster +CREATE DATABASE meshcorehub; +CREATE ROLE meshcorehub LOGIN PASSWORD 'your-password'; +GRANT ALL PRIVILEGES ON DATABASE meshcorehub TO meshcorehub; +``` + +The application **schema** and tables are created automatically by `db upgrade` (run by the `migrate` service on startup); the role just needs `CREATE` privilege on the database. Hub only ever connects as `DATABASE_USER` — no admin or bootstrap credentials are needed at runtime. + +### Managed or external Postgres + +To point Hub at an already-running Postgres (e.g. a managed cloud instance), set `DATABASE_HOST` at it and **do not** activate the `postgres` profile: + +```bash +DATABASE_BACKEND=postgres +DATABASE_HOST=your-managed-postgres.example.com +DATABASE_PORT=5432 +DATABASE_NAME=meshcorehub +DATABASE_USER=meshcorehub +DATABASE_PASSWORD=your-password +``` + +For advanced cases (custom driver, extra query params), set a full SQLAlchemy URL instead — it takes precedence over all the component variables: + +```bash +DATABASE_URL=postgresql+psycopg2://meshcorehub:your-password@host:5432/meshcorehub +``` + +## Schema-per-instance (`search_path`) + +Each Hub instance is isolated to its own Postgres **schema** via the connection's `search_path`, rather than its own database. This lets several instances (e.g. `prod`, `stg`) share **one** Postgres cluster without colliding — each gets its own tables and its own `alembic_version`. + +Give every instance a distinct `DATABASE_SCHEMA`: + +```bash +# Production (.env) +COMPOSE_PROJECT_NAME=hub +DATABASE_BACKEND=postgres +DATABASE_SCHEMA=meshcorehub_prod + +# Staging (.env, separate directory) +COMPOSE_PROJECT_NAME=hub-beta +DATABASE_BACKEND=postgres +DATABASE_SCHEMA=meshcorehub_stg +``` + +The schema is created automatically on `db upgrade` if it does not exist, so no manual `CREATE SCHEMA` is required. Connect both instances to the same `DATABASE_HOST` / `DATABASE_NAME` / `DATABASE_USER`; only `DATABASE_SCHEMA` (and `COMPOSE_PROJECT_NAME`) differ. + +> **Note:** This is the database-level isolation for instances sharing a Postgres cluster. For running multiple instances on the same Docker host (separate volumes, Traefik routing), see [Multi-Instance Deployments](../README.md#multi-instance-deployments) in the README. + +## Migrating from SQLite to PostgreSQL + +Existing SQLite deployments can be moved to Postgres with a single built-in command (`meshcore-hub db migrate-to-postgres`), which copies every table in foreign-key order through the ORM and prints a per-table row-count reconciliation. Downtime is required while writers are stopped; the source SQLite file is never modified. + +See the **v0.14 upgrade guide** in [upgrading.md](upgrading.md#migrating-an-existing-sqlite-database-to-postgres) for the full step-by-step runbook (backup, stop writers, bring up Postgres, run the migration, restart on Postgres). diff --git a/docs/upgrading.md b/docs/upgrading.md index 49f5b32..e68a324 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -6,45 +6,12 @@ This guide covers upgrading from a previous MeshCore Hub release to the current ### Optional PostgreSQL Backend -MeshCore Hub can now run on **PostgreSQL** as an alternative to the default SQLite database. SQLite remains the zero-config default — Postgres is entirely opt-in and **no action is required** to keep using SQLite. Switch to Postgres to scale writes and run the stack across multiple hosts (SQLite's file locking does not work over network filesystems and caps you at a single host). Existing operators can migrate their live SQLite data into Postgres with a single command (downtime required while writers are stopped). +MeshCore Hub can now run on **PostgreSQL** as an alternative to the default SQLite database. SQLite remains the zero-config default — Postgres is entirely opt-in and **no action is required** to keep using SQLite. Switch to Postgres to scale writes and run the stack across multiple hosts (SQLite's file locking does not work over network filesystems and caps you at a single host). -#### Enabling Postgres +> [!NOTE] +> As of v0.14, SQLite is **deprecated** in favour of PostgreSQL. SQLite continues to work as the default, but support will be removed in a future release (at least 3 months out). Existing SQLite deployments can migrate with the command below. -Set `DATABASE_BACKEND=postgres` and the `DATABASE_*` connection variables, then activate the compose `postgres` profile: - -| Variable | Default | Description | -| ------------------- | ------------- | -------------------------------------------------------------------------------------------- | -| `DATABASE_BACKEND` | `sqlite` | `sqlite` (default) or `postgres`. Explicit switch — Postgres is never used implicitly. | -| `DATABASE_HOST` | `postgres` | Postgres hostname (`postgres` is the bundled container's service name). | -| `DATABASE_PORT` | `5432` | Postgres port. | -| `DATABASE_NAME` | `meshcorehub` | Database name. The bundled container is initialised with this name. | -| `DATABASE_SCHEMA` | `meshcorehub` | Postgres schema (search_path). **Set a distinct value per instance** on a shared cluster. | -| `DATABASE_USER` | `meshcorehub` | Role name. The bundled container is initialised with this user. | -| `DATABASE_PASSWORD` | _(none)_ | **Required** for Postgres. Generate one, e.g. `openssl rand -base64 32`. | - -```bash -# Start the stack on Postgres (bundled container) -docker compose -f docker-compose.yml -f docker-compose.dev.yml \ - --profile postgres --profile core up -d -``` - -The bundled `postgres` container derives its `POSTGRES_USER` / `POSTGRES_PASSWORD` / `POSTGRES_DB` from the same `DATABASE_USER` / `DATABASE_PASSWORD` / `DATABASE_NAME` values — one source of truth. For a **managed/external** Postgres, point `DATABASE_HOST` at it (and skip the `postgres` profile). Advanced users can instead set a full `DATABASE_URL` (e.g. `postgresql+psycopg2://user:pass@host:5432/db`), which takes precedence over the component variables. - -#### Schema-per-instance (`search_path`) - -Each Hub instance is isolated to its own Postgres **schema** via the connection's `search_path`, not its own database. This lets several instances (e.g. `prod`, `stg`) share **one** Postgres cluster without colliding — each gets its own tables and its own `alembic_version`. Give every instance a distinct `DATABASE_SCHEMA` (e.g. `meshcorehub_prod`, `meshcorehub_stg`). The schema is created automatically on `db upgrade` if it does not exist. - -#### Provisioning the role and database - -The bundled container provisions the role and database for you on first start. For a managed/external Postgres, create them once before pointing Hub at it: - -```sql -CREATE ROLE meshcorehub LOGIN PASSWORD 'your-password'; -CREATE DATABASE meshcorehub OWNER meshcorehub; --- The schema is created by `db upgrade`; the role just needs CREATE on the database. -``` - -No admin/bootstrap credentials are needed at runtime — Hub only ever connects as `DATABASE_USER`. +For the **backend setup reference** — the `DATABASE_*` environment variables, the bundled Docker `postgres` profile, production role/database provisioning, managed/external Postgres (`DATABASE_URL`), and schema-per-instance (`search_path`) isolation for multiple instances sharing one cluster — see [database.md](database.md). The remainder of this section covers the upgrade-time migration of live SQLite data into Postgres. #### Migrating an existing SQLite database to Postgres @@ -67,7 +34,7 @@ Downtime is required while writers are stopped; the source SQLite file is never run --rm migrate meshcore-hub db migrate-to-postgres ``` It defaults the source to `sqlite:///{DATA_HOME}/collector/meshcore.db` and the target to your configured `DATABASE_*` connection. It copies every table in foreign-key order through the ORM (so SQLite's dynamically typed values are converted correctly — `0/1` → `boolean`, JSON text → `json`, naive datetimes → UTC `timestamptz`), then prints a per-table source-vs-target row-count reconciliation and fails on any mismatch. Use `--dry-run` to preview counts first, and `--truncate` to overwrite a non-empty target. -5. **Start the stack on Postgres** with `DATABASE_BACKEND=postgres` set (see *Enabling Postgres* above). +5. **Start the stack on Postgres** with `DATABASE_BACKEND=postgres` set (see [database.md](database.md) for the env vars and `postgres` compose profile). > **Why not pgloader?** pgloader infers the target schema from SQLite's *dynamic* typing and produces wrong Postgres types (e.g. `is_observer` as `bigint` not `boolean`, JSON columns as `text`, no `timestamptz`), and no `alembic_version` consistent with the migration history. The built-in command reuses the ORM models, so types convert correctly and the schema is created by `db upgrade`. From 0bc66d987181c5e3db5b65c54aa27cd8a9b7db99 Mon Sep 17 00:00:00 2001 From: Louis King Date: Wed, 17 Jun 2026 15:20:11 +0100 Subject: [PATCH 2/2] docs: drop pgloader explanation from upgrade guide --- docs/upgrading.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/upgrading.md b/docs/upgrading.md index e68a324..ea22f33 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -36,8 +36,6 @@ Downtime is required while writers are stopped; the source SQLite file is never It defaults the source to `sqlite:///{DATA_HOME}/collector/meshcore.db` and the target to your configured `DATABASE_*` connection. It copies every table in foreign-key order through the ORM (so SQLite's dynamically typed values are converted correctly — `0/1` → `boolean`, JSON text → `json`, naive datetimes → UTC `timestamptz`), then prints a per-table source-vs-target row-count reconciliation and fails on any mismatch. Use `--dry-run` to preview counts first, and `--truncate` to overwrite a non-empty target. 5. **Start the stack on Postgres** with `DATABASE_BACKEND=postgres` set (see [database.md](database.md) for the env vars and `postgres` compose profile). -> **Why not pgloader?** pgloader infers the target schema from SQLite's *dynamic* typing and produces wrong Postgres types (e.g. `is_observer` as `bigint` not `boolean`, JSON columns as `text`, no `timestamptz`), and no `alembic_version` consistent with the migration history. The built-in command reuses the ORM models, so types convert correctly and the schema is created by `db upgrade`. - > **Managed Postgres / non-superuser roles:** the migration disables foreign-key triggers during the copy via `session_replication_role = replica`, which requires a superuser. When the target role is not a superuser (typical for managed Postgres), the command automatically falls back to copying in parent-first order instead. Pass `--no-replication-role` to force the fallback explicitly. ### Dashboard Chart Fix (Postgres)