meshcore-hub/AGENTS.md

AGENTS.md - AI Coding Assistant Guidelines

Critical Rules (MUST follow)

• Always use parenthesized exception tuples — except (ValueError, TypeError): not except ValueError, TypeError:. The comma form is Python 2 syntax and fails at import time in Python 3. The most common error that passes visual review but breaks the app.
• To run tests and get results, use:

  pytest --no-cov 2>&1 | grep -iE "passed|failed" | tail -3
  

  --no-cov skips coverage for speed; the pipe surfaces only the pass/fail summary.
• Use Python (version in .python-version); activate a venv in .venv before running pytest, pre-commit, or alembic locally.
• Application operations run inside the compose stack — never invoke meshcore-hub directly on the host; build/run/exec via docker compose (see Development). The frontend npm/vite/tsc toolchain is the exception — it runs on the host (see Frontend).
• Never git push without explicit confirmation — staging and committing discrete changes is fine.
• Never build the Docker images or run make build / make up — the user builds manually to test. Stop after code changes + tests + pre-commit pass.
• Always generate random Alembic revision IDs — use python -c "import secrets; print(secrets.token_hex(6))" or let alembic revision auto-generate. Never hand-pick sequential or guessable IDs like a1b2c3d4e5f6 — they collide with existing migrations and cause cycle errors at upgrade time.
• Before committing: run targeted pytest --no-cov tests/test_<component>/ then pre-commit run --all-files.

Setup

ls ./.venv || python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pre-commit install

This venv is only for local testing, linting, and migration authoring. Frontend assets and runtime deps build into the Docker image — there is no local npm step.

Development

# Build / start / stop the stack (core = collector + api + web + migrate)
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core build
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core up -d
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core down

# Run a command inside a running service
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core exec <service> <command>

# Shorthands (Makefile, mqtt+core profiles): make build | make up | make down | make logs

Frontend (React)

The web UI is a React 19 + TypeScript + Vite SPA in src/meshcore_hub/web/static/js/spa-react/ (alias @/ → that dir). The Jinja2 shell (web/templates/spa.html) renders only SEO/window.__APP_CONFIG__/footer; React renders the navbar, banners, and routed pages into <div id="app">. Frontend tooling runs on the host (not in Docker): npm install, npm run build (Tailwind → vendor fonts → vite build → static/dist/ + assets.json), npx tsc --noEmit (the TS gate — also run by the frontend-typecheck pre-commit hook; there is no JS linter in pre-commit), and npm run test:frontend (vitest). The Vite build is required to serve the UI; there is no fallback bundle.

npm install            # host: install frontend deps
npm run build          # host: produce static/dist/ + assets.json
npx tsc --noEmit       # host: typecheck (must be clean)
npm run test:frontend  # host: vitest unit + component tests

• Charts: react-chartjs-2 — typed config builders in utils/charts.ts, wrappers in components/charts/Charts.tsx (imports chart.js/auto).
• Maps: react-leaflet (MapPage.tsx, NodeDetail.tsx); both import "leaflet/dist/leaflet.css". That CSS ships in the Vite bundle, which spa.html loads in <head> before app.css so the dark-mode map overrides win — don't reorder those <link>s.
• QR codes: react-qr-code.
• Navbar/shell: React (components/Navbar.tsx, ThemeToggle.tsx, Announcements.tsx, hooks/useNavItems.tsx); nav uses react-router NavLink (client-side nav). Feature flags, custom pages, and announcements all come from window.__APP_CONFIG__.
• Page conventions: useSearchParams() for filters/pagination/sort, typed apiGet<T>() with an AbortController in useEffect, usePageTitle('entities.x'), shared components (Pagination, FilterForm, StatCard, NodeDisplay, etc.).
• Tests: vitest + @testing-library/react (*.test.ts(x) next to code; setup in spa-react/test/). Python web tests assert the embedded __APP_CONFIG__ (tests/test_web/conftest.py::get_app_config), not server-rendered nav HTML.
• Only fonts are vendored (build.js copies them); chart/map/QR libs are bundled by Vite.

Tests & Quality

Coverage is opt-in; add --cov=meshcore_hub (or make test-cov) when you want it. The dev loop defaults to no coverage and parallel across CPU cores. Backend tests require PostgreSQL: make test / make test-cov / make test-unit start and stop the throwaway test Postgres automatically (a postgres:17-alpine on 127.0.0.1:55432; the management is skipped when TEST_POSTGRES_URL points at your own instance). For direct pytest runs, make test-db-up first — pytest fails fast with a hint when the database is unreachable.

# Canonical: run tests in parallel, no coverage, surface the pass/fail summary
pytest -nauto --no-cov 2>&1 | grep -iE "passed|failed" | tail -3

# Makefile shorthands
make test          # backend (pytest -nauto --no-cov) then frontend vitest
make test-cov      # full backend run with coverage report
make test-unit     # parallel, fast unit suites only (skips e2e)
make test-frontend # frontend vitest only (npm run test:frontend)

# Targeted by component (run only what you changed)
pytest --no-cov tests/test_web/        # templates, static JS, web routes
pytest --no-cov tests/test_api/        # API changes
pytest --no-cov tests/test_collector/  # collector changes
pytest --no-cov tests/test_common/     # common models/schemas/config

# Full suite only if changes span multiple components
pytest --no-cov

# Quality checks
pre-commit run --all-files

Browser E2E lives in e2e/ (Playwright, headless Chromium) and replaces the old Python e2e suite. It runs against a throwaway stack (e2e/docker-compose.test.yml) with its own ephemeral Postgres and isolated volumes — it never touches the dev database. Like the rest of the stack, the assistant never builds/runs these images; the user does.

npx playwright install chromium        # one-time browser binary (host)
sudo npx playwright install-deps chromium  # one-time system libs (Debian: nspr/nss/alsa/atk/gbm/X11)
make e2e-build && make e2e-up          # user: build + start mqtt/pg/migrate/collector/api/web
make e2e-test                          # user: seeds via e2e/seed_data.py, then runs the suite
make e2e-down                          # user: tear down (destroys the throwaway DB)
npm run typecheck:e2e                  # assistant: typecheck the e2e TS (safe to run)
npx playwright test --config=e2e/playwright.config.ts --list   # assistant: verify collection

Design notes when extending the suite:

• Auth is forged, not logged in. No mock IdP exists; the web tier fully trusts the signed meshcore-session cookie. e2e/mint_session.py (itsdangerous, run with .venv python) mints admin/member cookies using the stack's OIDC_SESSION_SECRET=test-session-secret; global setup writes them to e2e/.auth/*.json and specs opt in via test.use({ storageState }). OIDC is enabled in the test stack (which also unlocks the Members feature).
• Data is deterministic. e2e/seed_data.py clears + recreates fixed rows (nodes/observers with area tags, adverts, messages on channel idx 17 + the "E2E General" custom channel, raw packets + path hops keyed to node prefixes, a route + health, profiles + adoptions) using recent timestamps (7-day windows).
• Single shared backend: workers: 1, fullyParallel: false; routes/profile specs are describe.serial. WEB_AUTO_REFRESH_SECONDS=2 makes polling assertable.
• Selectors rely on purposeful data-testids (theme/auto-refresh toggles, observer area badges, path-hop badge + popover, route modal fields, nav/hero/member/list rows) added to the React components.

Database & Ops

The database backend is PostgreSQL (the only backend since v0.19). The bundled container is dev-only: it joins the core compose profile via docker-compose.dev.yml (profiles union across override files), so the documented dev commands start it automatically — production (docker-compose.prod.yml) never starts it with core/migrate, so it cannot shadow a shared postgres host on the same network. See docs/database.md for the full reference, production provisioning, and schema-per-instance setup. Backend tests also require Postgres: make test-db-up starts a throwaway instance on 127.0.0.1:55432 (or point TEST_POSTGRES_URL at your own).

# --- LOCAL (venv): author a migration against a local schema (NOT the dev DB)
# Use a scratch schema on the test DB so autogenerate diffs against a clean
# schema; never point revision tooling at the running dev database.
TEST_POSTGRES_URL=postgresql+psycopg2://meshcorehub:meshcorehub-test@localhost:55432/meshcorehub_test \
DATABASE_URL=postgresql+psycopg2://meshcorehub:meshcorehub-test@localhost:55432/meshcorehub_test \
DATABASE_SCHEMA=scratch meshcore-hub db revision --autogenerate -m "description"

# --- CONTAINER: apply migrations (the DB lives in the postgres_data volume)
# Migrations auto-apply on `up` via the migrate service. Manual one-off:
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core run --rm migrate db upgrade

# --- CONTAINER: seed node tags from SEED_HOME (NOT automatic)
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile seed run --rm seed

# --- CONTAINER: data retention / node cleanup (exec into the running collector)
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core exec collector meshcore-hub collector cleanup --retention-days 30 --dry-run
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core exec collector meshcore-hub collector cleanup --retention-days 30

Conventions

Cache invalidation on writes

Every mutation handler (POST/PUT/DELETE) on a user/admin-mutable entity MUST call the matching invalidate_* helper from meshcore_hub.api.cache_invalidation after session.commit() succeeds, so the UI reflects the change on the next page load instead of waiting for the Redis TTL. The helper is a no-op when Redis is disabled and swallows backend errors, so it's always safe to call.

The HTTP-layer cache policy on /api/v1/* GETs is private, no-cache (i.e. must-revalidate) precisely so this works: the browser always sends If-None-Match on navigation, the server answers 304 when Redis is warm and unchanged (cheap — no body) or 200 after an invalidation. Do NOT change this back to max-age>0 — server-side cache invalidation cannot reach the browser's HTTP cache, so any freshness window would let stale responses survive a mutation until expiry.

from meshcore_hub.api.cache_invalidation import invalidate_channels

@router.put("/{channel_id}")
def update_channel(__: RequireAdmin, session: DbSession, channel_id: str,
                   body: ChannelUpdate, request: Request) -> ChannelRead:
    # ... mutate ...
    session.commit()
    session.refresh(channel)
    invalidate_channels(request)   # after commit, before return
    return _channel_to_read(channel)

Mapping (see api/cache_invalidation.py for the canonical prefix knowledge):

Mutation	Helper(s)
POST/PUT/DELETE /channels	invalidate_channels
POST/PUT/DELETE /routes	invalidate_routes (covers list, detail, history)
PUT /user/profile/{id}	invalidate_profiles + invalidate_dashboard
POST/PUT/DELETE /nodes/{pk}/tags	invalidate_nodes + invalidate_messages + invalidate_advertisements + invalidate_dashboard (tags drive names/filters across these)
POST/DELETE /adoptions	invalidate_nodes + invalidate_profiles + invalidate_advertisements + invalidate_dashboard (adopted_by embedded across these)

When adding a new @cached read endpoint, decide whether its key namespace belongs in an existing invalidate helper, and add a test in tests/test_api/test_cache.py::TestMutationInvalidationIntegration. Cache keys split across two formats (endpoint-name keys like nodes: vs URL-path keys like /api/v1/channels:) — the helper module encapsulates that, don't hand-roll prefixes.

# Imports: stdlib, third-party, local
import os
from datetime import datetime
from typing import Optional

from fastapi import FastAPI, Depends
from pydantic import BaseModel
from sqlalchemy import select

from meshcore_hub.common.config import Settings
from meshcore_hub.common.models import Node

# Pydantic model
from pydantic import BaseModel, Field
from datetime import datetime
from typing import Optional

class NodeRead(BaseModel):
    """Schema for reading node data from API."""

    id: str
    public_key: str = Field(..., min_length=64, max_length=64)
    name: Optional[str] = None
    adv_type: Optional[str] = None
    last_seen: Optional[datetime] = None

    model_config = {"from_attributes": True}

# SQLAlchemy model
from sqlalchemy import String, DateTime, Text
from sqlalchemy.orm import Mapped, mapped_column, relationship
from typing import Optional

from meshcore_hub.common.models.base import Base, TimestampMixin, UUIDMixin

class Node(Base, UUIDMixin, TimestampMixin):
    __tablename__ = "nodes"

    public_key: Mapped[str] = mapped_column(String(64), unique=True, index=True)
    name: Mapped[str | None] = mapped_column(String(255), nullable=True)
    adv_type: Mapped[str | None] = mapped_column(String(20), nullable=True)
    last_seen: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)

    tags: Mapped[list["NodeTag"]] = relationship(back_populates="node", cascade="all, delete-orphan")


class UserProfile(Base, UUIDMixin, TimestampMixin):
    """UserProfile model for authenticated OIDC users."""
    __tablename__ = "user_profiles"

    user_id: Mapped[str] = mapped_column(String(255), unique=True, nullable=False, index=True)
    name: Mapped[Optional[str]] = mapped_column(String(255), nullable=True)
    callsign: Mapped[Optional[str]] = mapped_column(String(20), nullable=True)
    roles: Mapped[Optional[str]] = mapped_column(Text, nullable=True)

# FastAPI route
from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy.ext.asyncio import AsyncSession
from typing import Annotated

from meshcore_hub.api.dependencies import get_db, require_read
from meshcore_hub.common.schemas import NodeRead, NodeList

router = APIRouter(prefix="/nodes", tags=["nodes"])

@router.get("", response_model=NodeList)
async def list_nodes(
    db: Annotated[AsyncSession, Depends(get_db)],
    _: Annotated[None, Depends(require_read)],
    limit: int = Query(default=50, le=100),
    offset: int = Query(default=0, ge=0),
) -> NodeList:
    """List all nodes with pagination."""
    pass

# Click CLI command
import click

@click.group()
@click.pass_context
def cli(ctx: click.Context) -> None:
    """MeshCore Hub CLI."""
    ctx.ensure_object(dict)

@cli.command()
@click.option("--host", default="0.0.0.0", help="Bind host")
@click.option("--port", default=8000, type=int, help="Bind port")
@click.pass_context
def api(ctx: click.Context, host: str, port: int) -> None:
    """Start the API server."""
    import uvicorn
    from meshcore_hub.api.app import create_app

    app = create_app()
    uvicorn.run(app, host=host, port=port)

# Async lifespan
from contextlib import asynccontextmanager
from typing import AsyncGenerator
from fastapi import FastAPI

@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
    """Application lifespan handler."""
    await setup_database()
    await connect_mqtt()

    yield

    await disconnect_mqtt()
    await close_database()

# Error handling
from fastapi import HTTPException, status
import logging

logger = logging.getLogger(__name__)

raise HTTPException(
    status_code=status.HTTP_404_NOT_FOUND,
    detail=f"Node with public_key '{public_key}' not found"
)

try:
    result = await risky_operation()
except SomeException as e:
    logger.exception("Failed to perform operation: %s", e)
    raise

Test patterns

# Unit test
import pytest
from unittest.mock import AsyncMock, patch

@pytest.mark.asyncio
async def test_collector_handles_advertisement():
    """Test that collector handler processes advertisement events."""
    handler = AdvertisementHandler(db_session=AsyncMock())

    await handler.handle(event_data)

    handler.db_session.add.assert_called_once()
    node = handler.db_session.add.call_args[0][0]
    assert node.public_key == event_data["public_key"]

# Integration test
import pytest
from httpx import AsyncClient
from sqlalchemy.ext.asyncio import AsyncSession

@pytest.fixture
async def db_session(db_url, db_schema):
    """Async session on the per-worker Postgres schema (see tests/conftest.py)."""
    manager = DatabaseManager(db_url, schema=db_schema)
    manager.create_tables()
    async with manager.async_session() as session:
        yield session

@pytest.fixture
async def client(db_session):
    """Create test client with database session."""
    app = create_app()
    app.dependency_overrides[get_db] = lambda: db_session

    async with AsyncClient(app=app, base_url="http://test") as client:
        yield client

Where to find the rest

• README.md — project overview, deployment, MQTT topics, node tags, data retention, troubleshooting
• SCHEMAS.md — API/data schemas
• .env.example — all environment variables with defaults and comments
• docs/auth.md — OIDC authentication and roles
• docs/content.md — custom content (CONTENT_HOME)
• docs/i18n.md — translation reference
• docs/letsmesh.md — packet decoding and MQTT feed details
• docs/seeding.md — seed data
• docs/upgrading.md — upgrade notes
• docs/webhooks.md — webhook configuration and payloads