Merge pull request #230 from ipnet-mesh/feat/api-workers

feat(api): configurable worker processes via API_WORKERS

.env.example

@@ -258,6 +258,11 @@ NODE_CLEANUP_DAYS=30
 # External API port
 API_PORT=8000
 
+# Number of API worker processes (default 1). Increase to use multiple CPU
+# cores under load — each worker is an independent process sharing the same
+# listening socket. All workers read configuration from these env vars.
+# API_WORKERS=1
+
 # API Keys for authentication
 # Generate secure keys for production: openssl rand -hex 32
 # Leave empty to disable authentication (not recommended for production)

README.md

@@ -230,6 +230,23 @@ TRAEFIK_PRIORITY=20
 
 This ensures `beta.example.com` (priority 20) is matched before the production wildcard `*.example.com` (priority 10). For other services on the same network (e.g., an MQTT broker at `mqtt.example.com`), use an even higher priority (e.g., 30).
 
+#### Scaling the API
+
+The API is read-mostly and holds no per-process state — the response cache lives in Redis and authentication is stateless — so it scales across multiple worker processes. Set `API_WORKERS` to run more than one worker in a single container:
+
+```bash
+# .env
+API_WORKERS=4
+```
+
+Each worker is an independent process sharing one listening socket, so the kernel balances connections across them and CPU-bound work (JSON serialisation, validation) spreads over multiple cores. Workers read their configuration from **environment variables** (CLI flags are not propagated to forked workers), which is how Docker Compose already supplies config. Enabling Redis (`REDIS_ENABLED=true`) is recommended so all workers share one cache.
+
+Pick a worker count around the number of CPU cores available to the container; start with `2`–`4` and measure under realistic load.
+
+**SQLite caveat:** all workers share the same SQLite file on the same host. WAL mode (enabled automatically) allows concurrent readers alongside the single writer (the collector), so reads scale — but **writes do not**, and this does not extend across multiple hosts (a network filesystem breaks SQLite locking). To scale the API across hosts, switch `DATABASE_URL` to PostgreSQL; the API requires no code changes for this.
+
+> Prefer `API_WORKERS` over running multiple `api` containers (`--scale api=N`): the `api` service uses a fixed `container_name`, and one process-managed container per stack keeps logs, health checks, and monitoring simple.
+
 ### Adding Remote Observers
 
 Other operators can run their own [meshcore-packet-capture](https://github.com/agessaman/meshcore-packet-capture) instance and publish decoded packets to your MeshCore Hub. They can also optionally contribute to the LetsMesh and MeshRank networks.
@@ -364,6 +381,7 @@ The collector automatically cleans up old event data and inactive nodes:
 | ------------------- | --------- | ------------------------------------------------------- |
 | `API_HOST`          | `0.0.0.0` | API bind address                                        |
 | `API_PORT`          | `8000`    | API port                                                |
+| `API_WORKERS`       | `1`       | Number of worker processes (increase for multi-core concurrency; see [Scaling the API](#scaling-the-api)) |
 | `API_READ_KEY`      | _(none)_  | Read-only API key                                       |
 | `API_ADMIN_KEY`     | _(none)_  | Admin API key                                           |
 | `METRICS_ENABLED`   | `true`    | Enable Prometheus metrics endpoint at `/metrics`        |

docker-compose.yml

@@ -238,6 +238,8 @@ services:
       - DATA_HOME=/data
       - API_HOST=0.0.0.0
       - API_PORT=8000
+      # Worker processes for multi-core concurrency (1 = single process).
+      - API_WORKERS=${API_WORKERS:-1}
       - API_READ_KEY=${API_READ_KEY:-}
       - API_ADMIN_KEY=${API_ADMIN_KEY:-}
       - METRICS_ENABLED=${METRICS_ENABLED:-true}

docs/upgrading.md

@@ -4,6 +4,26 @@ This guide covers upgrading from a previous MeshCore Hub release to the current
 
 ## v0.12.0
 
+### Multi-Worker API (`API_WORKERS`)
+
+The API can now run multiple worker processes in a single container for multi-core concurrency, controlled by a new `API_WORKERS` environment variable (default `1`, unchanged behaviour). Each worker is an independent process sharing one listening socket.
+
+**New environment variable:**
+
+| Variable      | Default | Description                                                         |
+| ------------- | ------- | ------------------------------------------------------------------- |
+| `API_WORKERS` | `1`     | Number of API worker processes (increase for multi-core concurrency) |
+
+**No action required to upgrade** — the default of `1` preserves the previous single-process behaviour. To use it, set `API_WORKERS` in your `.env` and recreate the `api` service.
+
+**Important:** with more than one worker, configuration must come from **environment variables** — CLI flags passed to `meshcore-hub api` are not propagated to forked worker processes. Docker Compose deployments already configure everything via env, so they are unaffected. Enabling Redis (`REDIS_ENABLED=true`) is recommended so all workers share one response cache.
+
+While on SQLite, all workers share the same database file on the same host (WAL mode allows concurrent reads alongside the collector's single writer). Writes do not scale and this does not extend across multiple hosts; switch `DATABASE_URL` to PostgreSQL to scale beyond a single host. See [Scaling the API](../README.md#scaling-the-api) for details.
+
+### Read-Path Query Optimisations
+
+Several read-heavy endpoints had their query patterns optimised (node `is_observer` filtering, dashboard node-count history, and message/dashboard sender-name resolution). These are internal performance improvements with no API or configuration changes — responses are unchanged. The `is_observer` change ships an Alembic migration that is applied automatically on startup (Docker) or via `meshcore-hub db upgrade`.
+
 ### Optional Redis API Cache
 
 A new optional Redis-backed caching layer reduces database load for read-heavy API endpoints (nodes, advertisements, messages, channels, dashboard). Redis is entirely optional — the API works identically without it.

src/meshcore_hub/api/app.py

@@ -212,3 +212,60 @@ def create_app(
         return result
 
     return app
+
+
+def create_app_from_env() -> FastAPI:
+    """Build the application purely from environment configuration.
+
+    This factory is used when running multiple uvicorn workers: each forked
+    worker re-imports and calls it with no arguments, so all configuration
+    must come from the environment (CLI flags do not propagate to workers).
+    It mirrors the resolution the ``api`` CLI command performs for its
+    single-process path, drawing structured config from ``APISettings`` plus
+    the few env vars the CLI handles directly (CORS / metrics).
+    """
+    import os
+
+    from meshcore_hub.common.config import get_api_settings
+
+    settings = get_api_settings()
+
+    cors_env = os.environ.get("CORS_ORIGINS")
+    cors_origins = [o.strip() for o in cors_env.split(",")] if cors_env else None
+
+    def _env_bool(name: str, default: bool) -> bool:
+        raw = os.environ.get(name)
+        if raw is None:
+            return default
+        return raw.strip().lower() in ("1", "true", "yes", "on")
+
+    metrics_enabled = _env_bool("METRICS_ENABLED", True)
+    metrics_cache_ttl = int(os.environ.get("METRICS_CACHE_TTL", "60"))
+
+    # mqtt_transport is an enum on the settings object; create_app wants a str.
+    mqtt_transport = getattr(settings.mqtt_transport, "value", settings.mqtt_transport)
+
+    return create_app(
+        database_url=settings.effective_database_url,
+        read_key=settings.api_read_key,
+        admin_key=settings.api_admin_key,
+        mqtt_host=settings.mqtt_host,
+        mqtt_port=settings.mqtt_port,
+        mqtt_username=settings.mqtt_username,
+        mqtt_password=settings.mqtt_password,
+        mqtt_prefix=settings.mqtt_prefix,
+        mqtt_tls=settings.mqtt_tls,
+        mqtt_transport=mqtt_transport,
+        mqtt_ws_path=settings.mqtt_ws_path,
+        cors_origins=cors_origins,
+        metrics_enabled=metrics_enabled,
+        metrics_cache_ttl=metrics_cache_ttl,
+        redis_enabled=settings.redis_enabled,
+        redis_host=settings.redis_host,
+        redis_port=settings.redis_port,
+        redis_db=settings.redis_db,
+        redis_password=settings.redis_password,
+        redis_key_prefix=settings.redis_key_prefix,
+        redis_cache_ttl=settings.redis_cache_ttl,
+        redis_cache_ttl_dashboard=settings.redis_cache_ttl_dashboard,
+    )

src/meshcore_hub/api/cli.py

@@ -183,6 +183,18 @@ import click
     default=False,
     help="Enable auto-reload for development",
 )
+@click.option(
+    "--workers",
+    type=int,
+    default=1,
+    envvar="API_WORKERS",
+    help=(
+        "Number of worker processes (default: 1). Values >1 run multiple "
+        "processes for multi-core concurrency; workers are built from the "
+        "environment via the app factory, so configure via env vars (not "
+        "CLI flags) when scaling. Ignored in --reload mode."
+    ),
+)
 @click.pass_context
 def api(
     ctx: click.Context,
@@ -212,6 +224,7 @@ def api(
     redis_cache_ttl: int,
     redis_cache_ttl_dashboard: int,
     reload: bool,
+    workers: int,
 ) -> None:
     """Run the REST API server.
 
@@ -270,6 +283,7 @@ def api(
             f"Redis cache TTL: {redis_cache_ttl}s (dashboard: {redis_cache_ttl_dashboard}s)"
         )
     click.echo(f"Reload mode: {reload}")
+    click.echo(f"Workers: {workers}")
     click.echo("=" * 50)
 
     # Parse CORS origins
@@ -291,8 +305,21 @@ def api(
             reload=True,
             factory=True,
         )
+    elif workers > 1:
+        # Multiple worker processes require an import string so uvicorn can
+        # re-import the app in each forked worker. Workers rebuild the app from
+        # the environment via the factory, so configuration must come from env
+        # vars — CLI flags do not propagate to the workers.
+        click.echo(f"\nStarting API server with {workers} workers...")
+        uvicorn.run(
+            "meshcore_hub.api.app:create_app_from_env",
+            host=host,
+            port=port,
+            workers=workers,
+            factory=True,
+        )
     else:
-        # For production, create app directly
+        # Single process: build the app directly so CLI flags apply.
         app = create_app(
             database_url=effective_db_url,
             read_key=read_key,

tests/test_api/test_app_factory.py

@@ -0,0 +1,83 @@
+"""Tests for the environment-driven app factory used by multi-worker runs."""
+
+import pytest
+
+from meshcore_hub.api.app import create_app_from_env
+
+# Env vars the factory reads, cleared before each test so the host
+# environment can't leak into assertions.
+_FACTORY_ENV = [
+    "DATABASE_URL",
+    "DATA_HOME",
+    "REDIS_ENABLED",
+    "REDIS_HOST",
+    "REDIS_PORT",
+    "REDIS_CACHE_TTL",
+    "MQTT_HOST",
+    "CORS_ORIGINS",
+    "METRICS_ENABLED",
+    "METRICS_CACHE_TTL",
+]
+
+
+@pytest.fixture
+def clean_env(monkeypatch):
+    for var in _FACTORY_ENV:
+        monkeypatch.delenv(var, raising=False)
+    return monkeypatch
+
+
+def test_factory_reads_database_and_redis_from_env(clean_env):
+    """Workers must pick up the real DB/Redis config from env, not the
+    hardcoded create_app defaults."""
+    clean_env.setenv("DATABASE_URL", "sqlite:////tmp/workers-test.db")
+    clean_env.setenv("REDIS_ENABLED", "true")
+    clean_env.setenv("REDIS_HOST", "redis-test")
+    clean_env.setenv("REDIS_PORT", "6390")
+    clean_env.setenv("MQTT_HOST", "mqtt-test")
+    clean_env.setenv("METRICS_CACHE_TTL", "99")
+
+    app = create_app_from_env()
+
+    assert app.state.database_url == "sqlite:////tmp/workers-test.db"
+    assert app.state.redis_enabled is True
+    assert app.state.redis_host == "redis-test"
+    assert app.state.redis_port == 6390
+    assert app.state.mqtt_host == "mqtt-test"
+    assert app.state.metrics_cache_ttl == 99
+
+
+def test_factory_honours_explicit_disable_and_data_home(clean_env):
+    """Env values override anything else, and a data-home (no explicit
+    DATABASE_URL) resolves to the collector DB path — never the bare
+    create_app default that would point workers at ./meshcore.db."""
+    clean_env.setenv("REDIS_ENABLED", "false")
+    clean_env.setenv("DATA_HOME", "/srv/hubdata")
+
+    app = create_app_from_env()
+
+    assert app.state.redis_enabled is False
+    assert app.state.database_url == "sqlite:////srv/hubdata/collector/meshcore.db"
+
+
+def test_factory_redis_enabled_accepts_truthy_values(clean_env):
+    """REDIS_ENABLED / METRICS_ENABLED parse common truthy spellings."""
+    clean_env.setenv("REDIS_ENABLED", "1")
+    app = create_app_from_env()
+    assert app.state.redis_enabled is True
+
+
+def test_factory_metrics_enabled_via_env(clean_env):
+    """METRICS_ENABLED=true mounts the /metrics endpoint."""
+    clean_env.setenv("METRICS_ENABLED", "true")
+    app = create_app_from_env()
+    paths = {getattr(route, "path", None) for route in app.routes}
+    assert "/metrics" in paths
+
+
+def test_factory_metrics_disabled_via_env(clean_env):
+    """METRICS_ENABLED=false omits the /metrics endpoint."""
+    clean_env.setenv("METRICS_ENABLED", "false")
+    app = create_app_from_env()
+    paths = {getattr(route, "path", None) for route in app.routes}
+    assert "/metrics" not in paths

tests/test_api/test_cli.py

@@ -0,0 +1,49 @@
+"""Tests for the API CLI command (server launch wiring)."""
+
+from unittest.mock import patch
+
+from click.testing import CliRunner
+
+from meshcore_hub.api.cli import api
+
+
+def test_api_default_runs_single_process():
+    """With the default worker count, the app object is passed directly and no
+    worker/factory options are used."""
+    runner = CliRunner()
+    with patch("uvicorn.run") as mock_run:
+        result = runner.invoke(api, [], catch_exceptions=False)
+
+    assert result.exit_code == 0
+    assert mock_run.call_count == 1
+    args, kwargs = mock_run.call_args
+    # Single-process path passes the built app object, not an import string.
+    assert not isinstance(args[0], str)
+    assert "workers" not in kwargs
+
+
+def test_api_workers_uses_env_factory_import_string():
+    """workers > 1 launches uvicorn against the env-driven factory by import
+    string with the requested worker count."""
+    runner = CliRunner()
+    with patch("uvicorn.run") as mock_run:
+        result = runner.invoke(api, ["--workers", "3"], catch_exceptions=False)
+
+    assert result.exit_code == 0
+    args, kwargs = mock_run.call_args
+    assert args[0] == "meshcore_hub.api.app:create_app_from_env"
+    assert kwargs["workers"] == 3
+    assert kwargs["factory"] is True
+
+
+def test_api_workers_from_env_var():
+    """API_WORKERS env var drives the worker count (the Docker path)."""
+    runner = CliRunner()
+    with patch("uvicorn.run") as mock_run:
+        result = runner.invoke(
+            api, [], env={"API_WORKERS": "2"}, catch_exceptions=False
+        )
+
+    assert result.exit_code == 0
+    _, kwargs = mock_run.call_args
+    assert kwargs["workers"] == 2