meshcore-hub/AGENTS.md

AGENTS.md - AI Coding Assistant Guidelines

Critical Rules (MUST follow)

• Always use parenthesized exception tuples — except (ValueError, TypeError): not except ValueError, TypeError:. The comma form is Python 2 syntax and fails at import time in Python 3. The most common error that passes visual review but breaks the app.
• To run tests and get results, use:

  pytest --no-cov 2>&1 | grep -iE "passed|failed" | tail -3
  

  --no-cov skips coverage for speed; the pipe surfaces only the pass/fail summary.
• Use Python (version in .python-version); activate a venv in .venv before running pytest, pre-commit, or alembic locally.
• Application operations run inside the compose stack — never invoke meshcore-hub directly on the host; build/run/exec via docker compose (see Development). The frontend npm/vite/tsc toolchain is the exception — it runs on the host (see Frontend).
• Never git push without explicit confirmation — staging and committing discrete changes is fine.
• Never build the Docker images or run make build / make up — the user builds manually to test. Stop after code changes + tests + pre-commit pass.
• Always generate random Alembic revision IDs — use python -c "import secrets; print(secrets.token_hex(6))" or let alembic revision auto-generate. Never hand-pick sequential or guessable IDs like a1b2c3d4e5f6 — they collide with existing migrations and cause cycle errors at upgrade time.
• Before committing: run targeted pytest --no-cov tests/test_<component>/ then pre-commit run --all-files.

Setup

ls ./.venv || python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pre-commit install

This venv is only for local testing, linting, and migration authoring. Frontend assets and runtime deps build into the Docker image — there is no local npm step.

Development

# Build / start / stop the stack (core = collector + api + web + migrate + bundled postgres/redis via the dev override)
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core build
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core up -d
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core down

# Run a command inside a running service
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core exec <service> <command>

# Shorthands (Makefile, mqtt+core profiles): make build | make up | make down | make logs

Frontend (React)

The web UI is a React 19 + TypeScript + Vite SPA in src/meshcore_hub/web/static/js/spa-react/ (alias @/ → that dir). The Jinja2 shell (web/templates/spa.html) renders only SEO/window.__APP_CONFIG__/footer; React renders the navbar, banners, and routed pages into <div id="app">. Frontend tooling runs on the host (not in Docker): npm install, npm run build (Tailwind → vendor fonts → vite build → static/dist/ + assets.json), npx tsc --noEmit (the TS gate — also run by the frontend-typecheck pre-commit hook; there is no JS linter in pre-commit), and npm run test:frontend (vitest). The Vite build is required to serve the UI; there is no fallback bundle.

npm install            # host: install frontend deps
npm run build          # host: produce static/dist/ + assets.json
npx tsc --noEmit       # host: typecheck (must be clean)
npm run test:frontend  # host: vitest unit + component tests

• Charts: react-chartjs-2 — typed config builders in utils/charts.ts, wrappers in components/charts/Charts.tsx (imports chart.js/auto).
• Maps: react-leaflet (MapPage.tsx, NodeDetail.tsx); both import "leaflet/dist/leaflet.css". That CSS ships in the Vite bundle, which spa.html loads in <head> before app.css so the dark-mode map overrides win — don't reorder those <link>s.
• QR codes: react-qr-code.
• Navbar/shell: React (components/Navbar.tsx, ThemeToggle.tsx, Announcements.tsx, hooks/useNavItems.tsx); nav uses react-router NavLink (client-side nav). Feature flags, custom pages, and announcements all come from window.__APP_CONFIG__.
• Page conventions: useSearchParams() for filters/pagination/sort, typed apiGet<T>() with an AbortController in useEffect, usePageTitle('entities.x'), shared components (Pagination, FilterForm, StatCard, NodeDisplay, etc.).
• Tests: vitest + @testing-library/react (*.test.ts(x) next to code; setup in spa-react/test/). Python web tests assert the embedded __APP_CONFIG__ (tests/test_web/conftest.py::get_app_config), not server-rendered nav HTML.
• Only fonts are vendored (build.js copies them); chart/map/QR libs are bundled by Vite.

Tests & Quality

Coverage is opt-in; add --cov=meshcore_hub (or make test-cov) when you want it. The dev loop defaults to no coverage and parallel across CPU cores. Backend tests require PostgreSQL and Redis (both mandatory infrastructure): make test / make test-cov / make test-unit start and stop the throwaway test stack automatically (a postgres:17-alpine on 127.0.0.1:55432 and a redis:8-alpine on 127.0.0.1:55433; the management is skipped when TEST_POSTGRES_URL points at your own instance). For direct pytest runs, make test-db-up first — pytest fails fast with a hint when either is unreachable (TEST_REDIS_URL overrides the Redis target).

# Canonical: run tests in parallel, no coverage, surface the pass/fail summary
pytest -nauto --no-cov 2>&1 | grep -iE "passed|failed" | tail -3

# Makefile shorthands
make test          # backend (pytest -nauto --no-cov) then frontend vitest
make test-cov      # full backend run with coverage report
make test-unit     # parallel, fast unit suites only (skips e2e)
make test-frontend # frontend vitest only (npm run test:frontend)

# Targeted by component (run only what you changed)
pytest --no-cov tests/test_web/        # templates, static JS, web routes
pytest --no-cov tests/test_api/        # API changes
pytest --no-cov tests/test_collector/  # collector changes
pytest --no-cov tests/test_common/     # common models/schemas/config

# Full suite only if changes span multiple components
pytest --no-cov

# Quality checks
pre-commit run --all-files

Browser E2E lives in e2e/ (Playwright, headless Chromium) and replaces the old Python e2e suite. It runs against a throwaway stack (e2e/docker-compose.test.yml) with its own ephemeral Postgres and isolated volumes — it never touches the dev database. Like the rest of the stack, the assistant never builds/runs these images; the user does.

npx playwright install chromium        # one-time browser binary (host)
sudo npx playwright install-deps chromium  # one-time system libs (Debian: nspr/nss/alsa/atk/gbm/X11)
make e2e-build && make e2e-up          # user: build + start mqtt/pg/migrate/collector/api/web
make e2e-test                          # user: seeds via e2e/seed_data.py, then runs the suite
make e2e-down                          # user: tear down (destroys the throwaway DB)
npm run typecheck:e2e                  # assistant: typecheck the e2e TS (safe to run)
npx playwright test --config=e2e/playwright.config.ts --list   # assistant: verify collection

Design notes when extending the suite:

• Auth is forged, not logged in. No mock IdP exists; the web tier fully trusts the signed meshcore-session cookie. e2e/mint_session.py (itsdangerous, run with .venv python) mints admin/member cookies using the stack's OIDC_SESSION_SECRET=test-session-secret; global setup writes them to e2e/.auth/*.json and specs opt in via test.use({ storageState }). OIDC is enabled in the test stack (which also unlocks the Members feature).
• Data is deterministic. e2e/seed_data.py clears + recreates fixed rows (nodes/observers with area tags, adverts, messages on channel idx 17 + the "E2E General" custom channel, raw packets + path hops keyed to node prefixes, 30 trace_data filler groups for pagination tests, channels at every visibility tier (community/member/operator/admin + one disabled), a route + health, profiles incl. pw-operator + adoptions) using recent timestamps (7-day windows).
• Single shared backend: workers: 1, fullyParallel: false; routes/profile specs are describe.serial. WEB_AUTO_REFRESH_SECONDS=2 makes polling assertable.
• Selectors rely on purposeful data-testids (theme/auto-refresh toggles, observer area badges, path-hop badge + popover, route modal fields, nav/hero/member/list rows, channel cards/modal, node adoption/tag forms, pagination, sort headers, mobile nav) added to the React components.

Database & Ops

The database backend is PostgreSQL (the only backend since v0.19) and the cache backend is Redis (mandatory since v0.20 — same compose-profile pattern: the bundled containers join the core profile only via docker-compose.dev.yml, so the documented dev commands start them automatically; production (docker-compose.prod.yml) never starts them with core, so they cannot shadow shared postgres/redis hosts on the same network). See docs/database.md for the full database reference and docs/deployment.md → Redis Caching for the Redis one. Backend tests require both: make test-db-up starts throwaway instances on 127.0.0.1:55432 (Postgres) and 127.0.0.1:55433 (Redis) — or point TEST_POSTGRES_URL / TEST_REDIS_URL at your own.

# --- LOCAL (venv): author a migration against a local schema (NOT the dev DB)
# Use a scratch schema on the test DB so autogenerate diffs against a clean
# schema; never point revision tooling at the running dev database.
TEST_POSTGRES_URL=postgresql+psycopg2://meshcorehub:meshcorehub-test@localhost:55432/meshcorehub_test \
DATABASE_URL=postgresql+psycopg2://meshcorehub:meshcorehub-test@localhost:55432/meshcorehub_test \
DATABASE_SCHEMA=scratch meshcore-hub db revision --autogenerate -m "description"

# --- CONTAINER: apply migrations (the DB lives in the postgres_data volume)
# Migrations auto-apply on `up` via the migrate service. Manual one-off:
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core run --rm migrate db upgrade

# --- CONTAINER: seed node tags from SEED_HOME (NOT automatic)
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile seed run --rm seed

# --- CONTAINER: data retention / node cleanup (exec into the running collector)
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core exec collector meshcore-hub collector cleanup --retention-days 30 --dry-run
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core exec collector meshcore-hub collector cleanup --retention-days 30

Conventions

Cache invalidation on writes

Every mutation handler (POST/PUT/DELETE) on a user/admin-mutable entity MUST call the matching invalidate_* helper from meshcore_hub.api.cache_invalidation after session.commit() succeeds, so the UI reflects the change on the next page load instead of waiting for the Redis TTL. The helper never raises (backend errors log at ERROR after a committed write), so it's always safe to call. Redis is a mandatory backend — only read/store paths hard-fail (503) during an outage.

The HTTP-layer cache policy on /api/v1/* GETs is private, no-cache (i.e. must-revalidate) precisely so this works: the browser always sends If-None-Match on navigation, the server answers 304 when Redis is warm and unchanged (cheap — no body) or 200 after an invalidation. Do NOT change this back to max-age>0 — server-side cache invalidation cannot reach the browser's HTTP cache, so any freshness window would let stale responses survive a mutation until expiry.

from meshcore_hub.api.cache_invalidation import invalidate_channels

@router.put("/{channel_id}")
def update_channel(__: RequireAdmin, session: DbSession, channel_id: str,
                   body: ChannelUpdate, request: Request) -> ChannelRead:
    # ... mutate ...
    session.commit()
    session.refresh(channel)
    invalidate_channels(request)   # after commit, before return
    return _channel_to_read(channel)

Mapping (see api/cache_invalidation.py for the canonical prefix knowledge):

Mutation	Helper(s)
POST/PUT/DELETE /channels	invalidate_channels
POST/PUT/DELETE /routes	invalidate_routes (covers list, detail, history)
PUT /user/profile/{id}	invalidate_profiles + invalidate_dashboard
POST/PUT/DELETE /nodes/{pk}/tags	invalidate_nodes + invalidate_messages + invalidate_advertisements + invalidate_dashboard (tags drive names/filters across these)
POST/DELETE /adoptions	invalidate_nodes + invalidate_profiles + invalidate_advertisements + invalidate_dashboard (adopted_by embedded across these)
Feeds read caching	invalidate_messages / invalidate_advertisements / invalidate_channels each also drop the feeds namespace (message, per-channel, and advert feeds embed those rows) — new feed key namespaces must join the matching helper

When adding a new @cached read endpoint, decide whether its key namespace belongs in an existing invalidate helper, and add a test in tests/test_api/test_cache.py::TestMutationInvalidationIntegration. Cache keys split across two formats (endpoint-name keys like nodes: vs URL-path keys like /api/v1/channels:) — the helper module encapsulates that, don't hand-roll prefixes.

# Imports: stdlib, third-party, local
import os
from datetime import datetime
from typing import Optional

from fastapi import FastAPI, Depends
from pydantic import BaseModel
from sqlalchemy import select

from meshcore_hub.common.config import Settings
from meshcore_hub.common.models import Node

# Pydantic model
from pydantic import BaseModel, Field
from datetime import datetime
from typing import Optional

class NodeRead(BaseModel):
    """Schema for reading node data from API."""

    id: str
    public_key: str = Field(..., min_length=64, max_length=64)
    name: Optional[str] = None
    adv_type: Optional[str] = None
    last_seen: Optional[datetime] = None

    model_config = {"from_attributes": True}

# SQLAlchemy model
from sqlalchemy import String, DateTime, Text
from sqlalchemy.orm import Mapped, mapped_column, relationship
from typing import Optional

from meshcore_hub.common.models.base import Base, TimestampMixin, UUIDMixin

class Node(Base, UUIDMixin, TimestampMixin):
    __tablename__ = "nodes"

    public_key: Mapped[str] = mapped_column(String(64), unique=True, index=True)
    name: Mapped[str | None] = mapped_column(String(255), nullable=True)
    adv_type: Mapped[str | None] = mapped_column(String(20), nullable=True)
    last_seen: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)

    tags: Mapped[list["NodeTag"]] = relationship(back_populates="node", cascade="all, delete-orphan")


class UserProfile(Base, UUIDMixin, TimestampMixin):
    """UserProfile model for authenticated OIDC users."""
    __tablename__ = "user_profiles"

    user_id: Mapped[str] = mapped_column(String(255), unique=True, nullable=False, index=True)
    name: Mapped[Optional[str]] = mapped_column(String(255), nullable=True)
    callsign: Mapped[Optional[str]] = mapped_column(String(20), nullable=True)
    roles: Mapped[Optional[str]] = mapped_column(Text, nullable=True)

# FastAPI route
from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy.ext.asyncio import AsyncSession
from typing import Annotated

from meshcore_hub.api.dependencies import get_db, require_read
from meshcore_hub.common.schemas import NodeRead, NodeList

router = APIRouter(prefix="/nodes", tags=["nodes"])

@router.get("", response_model=NodeList)
async def list_nodes(
    db: Annotated[AsyncSession, Depends(get_db)],
    _: Annotated[None, Depends(require_read)],
    limit: int = Query(default=50, le=100),
    offset: int = Query(default=0, ge=0),
) -> NodeList:
    """List all nodes with pagination."""
    pass

# Click CLI command
import click

@click.group()
@click.pass_context
def cli(ctx: click.Context) -> None:
    """MeshCore Hub CLI."""
    ctx.ensure_object(dict)

@cli.command()
@click.option("--host", default="0.0.0.0", help="Bind host")
@click.option("--port", default=8000, type=int, help="Bind port")
@click.pass_context
def api(ctx: click.Context, host: str, port: int) -> None:
    """Start the API server."""
    import uvicorn
    from meshcore_hub.api.app import create_app

    app = create_app()
    uvicorn.run(app, host=host, port=port)

# Async lifespan
from contextlib import asynccontextmanager
from typing import AsyncGenerator
from fastapi import FastAPI

@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
    """Application lifespan handler."""
    await setup_database()
    await connect_mqtt()

    yield

    await disconnect_mqtt()
    await close_database()

# Error handling
from fastapi import HTTPException, status
import logging

logger = logging.getLogger(__name__)

raise HTTPException(
    status_code=status.HTTP_404_NOT_FOUND,
    detail=f"Node with public_key '{public_key}' not found"
)

try:
    result = await risky_operation()
except SomeException as e:
    logger.exception("Failed to perform operation: %s", e)
    raise

Test patterns

# Unit test
import pytest
from unittest.mock import AsyncMock, patch

@pytest.mark.asyncio
async def test_collector_handles_advertisement():
    """Test that collector handler processes advertisement events."""
    handler = AdvertisementHandler(db_session=AsyncMock())

    await handler.handle(event_data)

    handler.db_session.add.assert_called_once()
    node = handler.db_session.add.call_args[0][0]
    assert node.public_key == event_data["public_key"]

# Integration test
import pytest
from httpx import AsyncClient
from sqlalchemy.ext.asyncio import AsyncSession

@pytest.fixture
async def db_session(db_url, db_schema):
    """Async session on the per-worker Postgres schema (see tests/conftest.py)."""
    manager = DatabaseManager(db_url, schema=db_schema)
    manager.create_tables()
    async with manager.async_session() as session:
        yield session

@pytest.fixture
async def client(db_session):
    """Create test client with database session."""
    app = create_app()
    app.dependency_overrides[get_db] = lambda: db_session

    async with AsyncClient(app=app, base_url="http://test") as client:
        yield client

Where to find the rest

• README.md — project overview, deployment, MQTT topics, node tags, data retention, troubleshooting
• SCHEMAS.md — API/data schemas
• .env.example — all environment variables with defaults and comments
• docs/auth.md — OIDC authentication and roles
• docs/content.md — custom content (CONTENT_HOME)
• docs/i18n.md — translation reference
• docs/letsmesh.md — packet decoding and MQTT feed details
• docs/seeding.md — seed data
• docs/upgrading.md — upgrade notes
• docs/webhooks.md — webhook configuration and payloads