feat(api): configurable worker processes via API_WORKERS

Add a --workers / API_WORKERS option so the API can run multiple worker
processes in a single container for multi-core concurrency, without
needing to scale containers (which would conflict with the api service's
fixed container_name and complicate per-stack ops/monitoring).

The existing create_app() carries hardcoded defaults (sqlite:///./
meshcore.db, Redis off), so forked workers cannot use it — they would
open the wrong database and run without caching. Add an env-driven
factory, create_app_from_env(), that rebuilds the app from APISettings
plus the CLI-only env vars (CORS_ORIGINS, METRICS_ENABLED,
METRICS_CACHE_TTL), mirroring the single-process resolution. workers > 1
runs uvicorn against this factory via an import string; workers == 1
keeps the single-process object path so local CLI flags still apply.

Wire API_WORKERS into the api compose service (default 1, unchanged
behaviour) and document it in the README (new "Scaling the API"
section + env table) and the v0.12 upgrading notes, including the
SQLite single-host caveat and the env-vs-CLI-flag note for workers.

Tests: create_app_from_env reads DB/Redis/MQTT/metrics from env,
honours explicit overrides, and derives the collector DB path from
DATA_HOME rather than the bare create_app default.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

.env.example

@@ -258,6 +258,11 @@ NODE_CLEANUP_DAYS=30
 # External API port
 API_PORT=8000
 
+# Number of API worker processes (default 1). Increase to use multiple CPU
+# cores under load — each worker is an independent process sharing the same
+# listening socket. All workers read configuration from these env vars.
+# API_WORKERS=1
+
 # API Keys for authentication
 # Generate secure keys for production: openssl rand -hex 32
 # Leave empty to disable authentication (not recommended for production)

README.md

@@ -230,6 +230,23 @@ TRAEFIK_PRIORITY=20
 
 This ensures `beta.example.com` (priority 20) is matched before the production wildcard `*.example.com` (priority 10). For other services on the same network (e.g., an MQTT broker at `mqtt.example.com`), use an even higher priority (e.g., 30).
 
+#### Scaling the API
+
+The API is read-mostly and holds no per-process state — the response cache lives in Redis and authentication is stateless — so it scales across multiple worker processes. Set `API_WORKERS` to run more than one worker in a single container:
+
+```bash
+# .env
+API_WORKERS=4
+```
+
+Each worker is an independent process sharing one listening socket, so the kernel balances connections across them and CPU-bound work (JSON serialisation, validation) spreads over multiple cores. Workers read their configuration from **environment variables** (CLI flags are not propagated to forked workers), which is how Docker Compose already supplies config. Enabling Redis (`REDIS_ENABLED=true`) is recommended so all workers share one cache.
+
+Pick a worker count around the number of CPU cores available to the container; start with `2`–`4` and measure under realistic load.
+
+**SQLite caveat:** all workers share the same SQLite file on the same host. WAL mode (enabled automatically) allows concurrent readers alongside the single writer (the collector), so reads scale — but **writes do not**, and this does not extend across multiple hosts (a network filesystem breaks SQLite locking). To scale the API across hosts, switch `DATABASE_URL` to PostgreSQL; the API requires no code changes for this.
+
+> Prefer `API_WORKERS` over running multiple `api` containers (`--scale api=N`): the `api` service uses a fixed `container_name`, and one process-managed container per stack keeps logs, health checks, and monitoring simple.
+
 ### Adding Remote Observers
 
 Other operators can run their own [meshcore-packet-capture](https://github.com/agessaman/meshcore-packet-capture) instance and publish decoded packets to your MeshCore Hub. They can also optionally contribute to the LetsMesh and MeshRank networks.
@@ -364,6 +381,7 @@ The collector automatically cleans up old event data and inactive nodes:
 | ------------------- | --------- | ------------------------------------------------------- |
 | `API_HOST`          | `0.0.0.0` | API bind address                                        |
 | `API_PORT`          | `8000`    | API port                                                |
+| `API_WORKERS`       | `1`       | Number of worker processes (increase for multi-core concurrency; see [Scaling the API](#scaling-the-api)) |
 | `API_READ_KEY`      | _(none)_  | Read-only API key                                       |
 | `API_ADMIN_KEY`     | _(none)_  | Admin API key                                           |
 | `METRICS_ENABLED`   | `true`    | Enable Prometheus metrics endpoint at `/metrics`        |

docker-compose.yml

@@ -238,6 +238,8 @@ services:
       - DATA_HOME=/data
       - API_HOST=0.0.0.0
       - API_PORT=8000
+      # Worker processes for multi-core concurrency (1 = single process).
+      - API_WORKERS=${API_WORKERS:-1}
       - API_READ_KEY=${API_READ_KEY:-}
       - API_ADMIN_KEY=${API_ADMIN_KEY:-}
       - METRICS_ENABLED=${METRICS_ENABLED:-true}

docs/upgrading.md

@@ -4,6 +4,26 @@ This guide covers upgrading from a previous MeshCore Hub release to the current
 
 ## v0.12.0
 
+### Multi-Worker API (`API_WORKERS`)
+
+The API can now run multiple worker processes in a single container for multi-core concurrency, controlled by a new `API_WORKERS` environment variable (default `1`, unchanged behaviour). Each worker is an independent process sharing one listening socket.
+
+**New environment variable:**
+
+| Variable      | Default | Description                                                         |
+| ------------- | ------- | ------------------------------------------------------------------- |
+| `API_WORKERS` | `1`     | Number of API worker processes (increase for multi-core concurrency) |
+
+**No action required to upgrade** — the default of `1` preserves the previous single-process behaviour. To use it, set `API_WORKERS` in your `.env` and recreate the `api` service.
+
+**Important:** with more than one worker, configuration must come from **environment variables** — CLI flags passed to `meshcore-hub api` are not propagated to forked worker processes. Docker Compose deployments already configure everything via env, so they are unaffected. Enabling Redis (`REDIS_ENABLED=true`) is recommended so all workers share one response cache.
+
+While on SQLite, all workers share the same database file on the same host (WAL mode allows concurrent reads alongside the collector's single writer). Writes do not scale and this does not extend across multiple hosts; switch `DATABASE_URL` to PostgreSQL to scale beyond a single host. See [Scaling the API](../README.md#scaling-the-api) for details.
+
+### Read-Path Query Optimisations
+
+Several read-heavy endpoints had their query patterns optimised (node `is_observer` filtering, dashboard node-count history, and message/dashboard sender-name resolution). These are internal performance improvements with no API or configuration changes — responses are unchanged. The `is_observer` change ships an Alembic migration that is applied automatically on startup (Docker) or via `meshcore-hub db upgrade`.
+
 ### Optional Redis API Cache
 
 A new optional Redis-backed caching layer reduces database load for read-heavy API endpoints (nodes, advertisements, messages, channels, dashboard). Redis is entirely optional — the API works identically without it.

src/meshcore_hub/api/app.py

@@ -212,3 +212,60 @@ def create_app(
         return result
 
     return app
+
+
+def create_app_from_env() -> FastAPI:
+    """Build the application purely from environment configuration.
+
+    This factory is used when running multiple uvicorn workers: each forked
+    worker re-imports and calls it with no arguments, so all configuration
+    must come from the environment (CLI flags do not propagate to workers).
+    It mirrors the resolution the ``api`` CLI command performs for its
+    single-process path, drawing structured config from ``APISettings`` plus
+    the few env vars the CLI handles directly (CORS / metrics).
+    """
+    import os
+
+    from meshcore_hub.common.config import get_api_settings
+
+    settings = get_api_settings()
+
+    cors_env = os.environ.get("CORS_ORIGINS")
+    cors_origins = [o.strip() for o in cors_env.split(",")] if cors_env else None
+
+    def _env_bool(name: str, default: bool) -> bool:
+        raw = os.environ.get(name)
+        if raw is None:
+            return default
+        return raw.strip().lower() in ("1", "true", "yes", "on")
+
+    metrics_enabled = _env_bool("METRICS_ENABLED", True)
+    metrics_cache_ttl = int(os.environ.get("METRICS_CACHE_TTL", "60"))
+
+    # mqtt_transport is an enum on the settings object; create_app wants a str.
+    mqtt_transport = getattr(settings.mqtt_transport, "value", settings.mqtt_transport)
+
+    return create_app(
+        database_url=settings.effective_database_url,
+        read_key=settings.api_read_key,
+        admin_key=settings.api_admin_key,
+        mqtt_host=settings.mqtt_host,
+        mqtt_port=settings.mqtt_port,
+        mqtt_username=settings.mqtt_username,
+        mqtt_password=settings.mqtt_password,
+        mqtt_prefix=settings.mqtt_prefix,
+        mqtt_tls=settings.mqtt_tls,
+        mqtt_transport=mqtt_transport,
+        mqtt_ws_path=settings.mqtt_ws_path,
+        cors_origins=cors_origins,
+        metrics_enabled=metrics_enabled,
+        metrics_cache_ttl=metrics_cache_ttl,
+        redis_enabled=settings.redis_enabled,
+        redis_host=settings.redis_host,
+        redis_port=settings.redis_port,
+        redis_db=settings.redis_db,
+        redis_password=settings.redis_password,
+        redis_key_prefix=settings.redis_key_prefix,
+        redis_cache_ttl=settings.redis_cache_ttl,
+        redis_cache_ttl_dashboard=settings.redis_cache_ttl_dashboard,
+    )

src/meshcore_hub/api/cli.py

@@ -183,6 +183,18 @@ import click
     default=False,
     help="Enable auto-reload for development",
 )
+@click.option(
+    "--workers",
+    type=int,
+    default=1,
+    envvar="API_WORKERS",
+    help=(
+        "Number of worker processes (default: 1). Values >1 run multiple "
+        "processes for multi-core concurrency; workers are built from the "
+        "environment via the app factory, so configure via env vars (not "
+        "CLI flags) when scaling. Ignored in --reload mode."
+    ),
+)
 @click.pass_context
 def api(
     ctx: click.Context,
@@ -212,6 +224,7 @@ def api(
     redis_cache_ttl: int,
     redis_cache_ttl_dashboard: int,
     reload: bool,
+    workers: int,
 ) -> None:
     """Run the REST API server.
 
@@ -270,6 +283,7 @@ def api(
             f"Redis cache TTL: {redis_cache_ttl}s (dashboard: {redis_cache_ttl_dashboard}s)"
         )
     click.echo(f"Reload mode: {reload}")
+    click.echo(f"Workers: {workers}")
     click.echo("=" * 50)
 
     # Parse CORS origins
@@ -291,8 +305,21 @@ def api(
             reload=True,
             factory=True,
         )
+    elif workers > 1:
+        # Multiple worker processes require an import string so uvicorn can
+        # re-import the app in each forked worker. Workers rebuild the app from
+        # the environment via the factory, so configuration must come from env
+        # vars — CLI flags do not propagate to the workers.
+        click.echo(f"\nStarting API server with {workers} workers...")
+        uvicorn.run(
+            "meshcore_hub.api.app:create_app_from_env",
+            host=host,
+            port=port,
+            workers=workers,
+            factory=True,
+        )
     else:
-        # For production, create app directly
+        # Single process: build the app directly so CLI flags apply.
         app = create_app(
             database_url=effective_db_url,
             read_key=read_key,

tests/test_api/test_app_factory.py

@@ -0,0 +1,67 @@
+"""Tests for the environment-driven app factory used by multi-worker runs."""
+
+import pytest
+
+from meshcore_hub.api.app import create_app_from_env
+
+# Env vars the factory reads, cleared before each test so the host
+# environment can't leak into assertions.
+_FACTORY_ENV = [
+    "DATABASE_URL",
+    "DATA_HOME",
+    "REDIS_ENABLED",
+    "REDIS_HOST",
+    "REDIS_PORT",
+    "REDIS_CACHE_TTL",
+    "MQTT_HOST",
+    "CORS_ORIGINS",
+    "METRICS_ENABLED",
+    "METRICS_CACHE_TTL",
+]
+
+
+@pytest.fixture
+def clean_env(monkeypatch):
+    for var in _FACTORY_ENV:
+        monkeypatch.delenv(var, raising=False)
+    return monkeypatch
+
+
+def test_factory_reads_database_and_redis_from_env(clean_env):
+    """Workers must pick up the real DB/Redis config from env, not the
+    hardcoded create_app defaults."""
+    clean_env.setenv("DATABASE_URL", "sqlite:////tmp/workers-test.db")
+    clean_env.setenv("REDIS_ENABLED", "true")
+    clean_env.setenv("REDIS_HOST", "redis-test")
+    clean_env.setenv("REDIS_PORT", "6390")
+    clean_env.setenv("MQTT_HOST", "mqtt-test")
+    clean_env.setenv("METRICS_CACHE_TTL", "99")
+
+    app = create_app_from_env()
+
+    assert app.state.database_url == "sqlite:////tmp/workers-test.db"
+    assert app.state.redis_enabled is True
+    assert app.state.redis_host == "redis-test"
+    assert app.state.redis_port == 6390
+    assert app.state.mqtt_host == "mqtt-test"
+    assert app.state.metrics_cache_ttl == 99
+
+
+def test_factory_honours_explicit_disable_and_data_home(clean_env):
+    """Env values override anything else, and a data-home (no explicit
+    DATABASE_URL) resolves to the collector DB path — never the bare
+    create_app default that would point workers at ./meshcore.db."""
+    clean_env.setenv("REDIS_ENABLED", "false")
+    clean_env.setenv("DATA_HOME", "/srv/hubdata")
+
+    app = create_app_from_env()
+
+    assert app.state.redis_enabled is False
+    assert app.state.database_url == "sqlite:////srv/hubdata/collector/meshcore.db"
+
+
+def test_factory_redis_enabled_accepts_truthy_values(clean_env):
+    """REDIS_ENABLED / METRICS_ENABLED parse common truthy spellings."""
+    clean_env.setenv("REDIS_ENABLED", "1")
+    app = create_app_from_env()
+    assert app.state.redis_enabled is True