mirror of https://github.com/ipnet-mesh/meshcore-hub.git synced 2026-07-28 12:32:28 +02:00

Files

T

Louis King 94dd4c9b42 test(e2e): replace Python e2e suite with Playwright (headless Chromium)

Replaces the httpx-based smoke tests in tests/e2e/ with a browser E2E suite
under e2e/, running against a self-contained throwaway stack that never
touches the local development database.

Stack & data isolation (e2e/docker-compose.test.yml):
- Own ephemeral Postgres 17 (schema via the `migrate` service / Alembic),
  distinct project name + named volumes, no host DB port, no \${VAR}
  interpolation. `make e2e-down` destroys everything.
- OIDC enabled with a known session secret; WEB_AUTO_REFRESH_SECONDS=2 so
  polling is assertable; CONTENT_HOME mounts a test markdown page.
- Deterministic seeder (e2e/seed_data.py) run via global setup inside the
  collector container: nodes/observers with `area` tags, adverts, messages
  on the public (17) + a custom channel, raw packets + path hops keyed to
  node prefixes, a route + health/history, profiles + adoptions, and
  event_observers rows so the observer filter/badges resolve.

Auth & tests:
- Forged signed `meshcore-session` cookies (e2e/mint_session.py,
  itsdangerous) for admin/member identities -> storageState; no real IdP
  needed. 31 specs across 12 files cover global nav/theme, profile
  menu+edit, home hero, dashboard widgets, list filters/auto-refresh/row
  actions, observer toggles, the path-node overlay, map filters +
  show-labels, members, markdown pages, and routes add/edit/delete
  (persistence, validation, confirm dialog). workers:1 / fullyParallel:false
  against the single shared backend; targeted data-testids added to React
  components for stable selectors.

Fixes surfaced while running the suite on fresh Postgres:
- Migration 5e3b712ccf10 aborted on Postgres: its route-health backfill
  queries the live Route model (which now has max_path_length) before that
  column exists. The swallowed error left the transaction aborted, killing
  the subsequent alembic_version stamp. Wrapped the backfill in a SAVEPOINT
  so a failure rolls back cleanly without blocking the migration.
- Restored the full ABUSE_* env set required by the MQTT broker, and set
  the web service's API_KEY to the admin key (admin writes go through the
  proxy as a Bearer token).

31/31 passing; tsc (frontend+e2e), pytest (1460), and pre-commit green.

2026-07-22 10:41:53 +01:00

17 KiB

Raw Blame History

AGENTS.md - AI Coding Assistant Guidelines

Critical Rules (MUST follow)

Always use parenthesized exception tuples — except (ValueError, TypeError): not except ValueError, TypeError:. The comma form is Python 2 syntax and fails at import time in Python 3. The most common error that passes visual review but breaks the app.
To run tests and get results, use:
```
pytest --no-cov 2>&1 | grep -iE "passed|failed" | tail -3
```
--no-cov skips coverage for speed; the pipe surfaces only the pass/fail summary.
Use Python (version in .python-version); activate a venv in .venv before running pytest, pre-commit, or alembic locally.
Application operations run inside the compose stack — never invoke meshcore-hub directly on the host; build/run/exec via docker compose (see Development). The frontend npm/vite/tsc toolchain is the exception — it runs on the host (see Frontend).
Never git push without explicit confirmation — staging and committing discrete changes is fine.
Never build the Docker images or run make build / make up — the user builds manually to test. Stop after code changes + tests + pre-commit pass.
Always generate random Alembic revision IDs — use python -c "import secrets; print(secrets.token_hex(6))" or let alembic revision auto-generate. Never hand-pick sequential or guessable IDs like a1b2c3d4e5f6 — they collide with existing migrations and cause cycle errors at upgrade time.
Before committing: run targeted pytest --no-cov tests/test_<component>/ then pre-commit run --all-files.

Setup

ls ./.venv || python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pre-commit install

This venv is only for local testing, linting, and migration authoring. Frontend assets and runtime deps build into the Docker image — there is no local npm step.

Development

# Build / start / stop the stack (core = collector + api + web + migrate)
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core build
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core up -d
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core down

# Run a command inside a running service
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core exec <service> <command>

# Shorthands (Makefile, mqtt+core profiles): make build | make up | make down | make logs

Frontend (React)

The web UI is a React 19 + TypeScript + Vite SPA in src/meshcore_hub/web/static/js/spa-react/ (alias @/ → that dir). The Jinja2 shell (web/templates/spa.html) renders only SEO/window.__APP_CONFIG__/footer; React renders the navbar, banners, and routed pages into <div id="app">. Frontend tooling runs on the host (not in Docker): npm install, npm run build (Tailwind → vendor fonts → vite build → static/dist/ + assets.json), npx tsc --noEmit (the TS gate — also run by the frontend-typecheck pre-commit hook; there is no JS linter in pre-commit), and npm run test:frontend (vitest). The Vite build is required to serve the UI; there is no fallback bundle.

npm install            # host: install frontend deps
npm run build          # host: produce static/dist/ + assets.json
npx tsc --noEmit       # host: typecheck (must be clean)
npm run test:frontend  # host: vitest unit + component tests

Charts: react-chartjs-2 — typed config builders in utils/charts.ts, wrappers in components/charts/Charts.tsx (imports chart.js/auto).
Maps: react-leaflet (MapPage.tsx, NodeDetail.tsx); both import "leaflet/dist/leaflet.css". That CSS ships in the Vite bundle, which spa.html loads in <head> before app.css so the dark-mode map overrides win — don't reorder those <link>s.
QR codes: react-qr-code.
Navbar/shell: React (components/Navbar.tsx, ThemeToggle.tsx, Announcements.tsx, hooks/useNavItems.tsx); nav uses react-router NavLink (client-side nav). Feature flags, custom pages, and announcements all come from window.__APP_CONFIG__.
Page conventions: useSearchParams() for filters/pagination/sort, typed apiGet<T>() with an AbortController in useEffect, usePageTitle('entities.x'), shared components (Pagination, FilterForm, StatCard, NodeDisplay, etc.).
Tests: vitest + @testing-library/react (*.test.ts(x) next to code; setup in spa-react/test/). Python web tests assert the embedded __APP_CONFIG__ (tests/test_web/conftest.py::get_app_config), not server-rendered nav HTML.
Only fonts are vendored (build.js copies them); chart/map/QR libs are bundled by Vite.

Tests & Quality

Coverage is opt-in; add --cov=meshcore_hub (or make test-cov) when you want it. The dev loop defaults to no coverage and parallel across CPU cores.

# Canonical: run tests in parallel, no coverage, surface the pass/fail summary
pytest -nauto --no-cov 2>&1 | grep -iE "passed|failed" | tail -3

# Makefile shorthands
make test          # backend (pytest -nauto --no-cov) then frontend vitest
make test-cov      # full backend run with coverage report
make test-unit     # parallel, fast unit suites only (skips e2e)
make test-frontend # frontend vitest only (npm run test:frontend)

# Targeted by component (run only what you changed)
pytest --no-cov tests/test_web/        # templates, static JS, web routes
pytest --no-cov tests/test_api/        # API changes
pytest --no-cov tests/test_collector/  # collector changes
pytest --no-cov tests/test_common/     # common models/schemas/config

# Full suite only if changes span multiple components
pytest --no-cov

# Quality checks
pre-commit run --all-files

Browser E2E lives in e2e/ (Playwright, headless Chromium) and replaces the old Python e2e suite. It runs against a throwaway stack (e2e/docker-compose.test.yml) with its own ephemeral Postgres and isolated volumes — it never touches the dev database. Like the rest of the stack, the assistant never builds/runs these images; the user does.

npx playwright install chromium        # one-time browser binary (host)
make e2e-build && make e2e-up          # user: build + start mqtt/pg/migrate/collector/api/web
make e2e-test                          # user: seeds via e2e/seed_data.py, then runs the suite
make e2e-down                          # user: tear down (destroys the throwaway DB)
npm run typecheck:e2e                  # assistant: typecheck the e2e TS (safe to run)
npx playwright test --config=e2e/playwright.config.ts --list   # assistant: verify collection

Design notes when extending the suite:

Auth is forged, not logged in. No mock IdP exists; the web tier fully trusts the signed meshcore-session cookie. e2e/mint_session.py (itsdangerous, run with .venv python) mints admin/member cookies using the stack's OIDC_SESSION_SECRET=test-session-secret; global setup writes them to e2e/.auth/*.json and specs opt in via test.use({ storageState }). OIDC is enabled in the test stack (which also unlocks the Members feature).
Data is deterministic. e2e/seed_data.py clears + recreates fixed rows (nodes/observers with area tags, adverts, messages on channel idx 17 + the "E2E General" custom channel, raw packets + path hops keyed to node prefixes, a route + health, profiles + adoptions) using recent timestamps (7-day windows).
Single shared backend: workers: 1, fullyParallel: false; routes/profile specs are describe.serial. WEB_AUTO_REFRESH_SECONDS=2 makes polling assertable.
Selectors rely on purposeful data-testids (theme/auto-refresh toggles, observer area badges, path-hop badge + popover, route modal fields, nav/hero/member/list rows) added to the React components.

Database & Ops

The default backend is SQLite (zero-config, file at ${DATA_HOME}/collector/meshcore.db). PostgreSQL is also supported via DATABASE_BACKEND=postgres — see docs/database.md for the full backend reference, production provisioning, and schema-per-instance setup. Migrations are backend-agnostic; the commands below work for both.

# --- LOCAL (venv): sync the volume DB to ./meshcore.db, then author a migration
# Volume name is ${COMPOSE_PROJECT_NAME:-hub}_data (default: hub_data)
# (SQLite only — for Postgres, point the migration env at the cluster directly)
docker run -it --rm -v hub_data:/data -v "$PWD":/pwd ubuntu cp /data/collector/meshcore.db /pwd/meshcore.db
meshcore-hub db revision --autogenerate -m "description"

# --- CONTAINER: apply migrations (the DB lives in the data volume)
# Migrations auto-apply on `up` via the migrate service. Manual one-off:
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core run --rm migrate db upgrade

# --- CONTAINER: seed node tags from SEED_HOME (NOT automatic)
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile seed run --rm seed

# --- CONTAINER: data retention / node cleanup (exec into the running collector)
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core exec collector meshcore-hub collector cleanup --retention-days 30 --dry-run
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile core exec collector meshcore-hub collector cleanup --retention-days 30

Conventions

Cache invalidation on writes

Every mutation handler (POST/PUT/DELETE) on a user/admin-mutable entity MUST call the matching invalidate_* helper from meshcore_hub.api.cache_invalidation after session.commit() succeeds, so the UI reflects the change on the next page load instead of waiting for the Redis TTL. The helper is a no-op when Redis is disabled and swallows backend errors, so it's always safe to call.

The HTTP-layer cache policy on /api/v1/* GETs is private, no-cache (i.e. must-revalidate) precisely so this works: the browser always sends If-None-Match on navigation, the server answers 304 when Redis is warm and unchanged (cheap — no body) or 200 after an invalidation. Do NOT change this back to max-age>0 — server-side cache invalidation cannot reach the browser's HTTP cache, so any freshness window would let stale responses survive a mutation until expiry.

from meshcore_hub.api.cache_invalidation import invalidate_channels

@router.put("/{channel_id}")
def update_channel(__: RequireAdmin, session: DbSession, channel_id: str,
                   body: ChannelUpdate, request: Request) -> ChannelRead:
    # ... mutate ...
    session.commit()
    session.refresh(channel)
    invalidate_channels(request)   # after commit, before return
    return _channel_to_read(channel)

Mapping (see api/cache_invalidation.py for the canonical prefix knowledge):

Mutation	Helper(s)
`POST/PUT/DELETE /channels`	`invalidate_channels`
`POST/PUT/DELETE /routes`	`invalidate_routes` (covers list, detail, history)
`PUT /user/profile/{id}`	`invalidate_profiles` + `invalidate_dashboard`
`POST/PUT/DELETE /nodes/{pk}/tags`	`invalidate_nodes` + `invalidate_messages` + `invalidate_advertisements` + `invalidate_dashboard` (tags drive names/filters across these)
`POST/DELETE /adoptions`	`invalidate_nodes` + `invalidate_profiles` + `invalidate_advertisements` + `invalidate_dashboard` (`adopted_by` embedded across these)

When adding a new @cached read endpoint, decide whether its key namespace belongs in an existing invalidate helper, and add a test in tests/test_api/test_cache.py::TestMutationInvalidationIntegration. Cache keys split across two formats (endpoint-name keys like nodes: vs URL-path keys like /api/v1/channels:) — the helper module encapsulates that, don't hand-roll prefixes.

# Imports: stdlib, third-party, local
import os
from datetime import datetime
from typing import Optional

from fastapi import FastAPI, Depends
from pydantic import BaseModel
from sqlalchemy import select

from meshcore_hub.common.config import Settings
from meshcore_hub.common.models import Node

# Pydantic model
from pydantic import BaseModel, Field
from datetime import datetime
from typing import Optional

class NodeRead(BaseModel):
    """Schema for reading node data from API."""

    id: str
    public_key: str = Field(..., min_length=64, max_length=64)
    name: Optional[str] = None
    adv_type: Optional[str] = None
    last_seen: Optional[datetime] = None

    model_config = {"from_attributes": True}

# SQLAlchemy model
from sqlalchemy import String, DateTime, Text
from sqlalchemy.orm import Mapped, mapped_column, relationship
from typing import Optional

from meshcore_hub.common.models.base import Base, TimestampMixin, UUIDMixin

class Node(Base, UUIDMixin, TimestampMixin):
    __tablename__ = "nodes"

    public_key: Mapped[str] = mapped_column(String(64), unique=True, index=True)
    name: Mapped[str | None] = mapped_column(String(255), nullable=True)
    adv_type: Mapped[str | None] = mapped_column(String(20), nullable=True)
    last_seen: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)

    tags: Mapped[list["NodeTag"]] = relationship(back_populates="node", cascade="all, delete-orphan")


class UserProfile(Base, UUIDMixin, TimestampMixin):
    """UserProfile model for authenticated OIDC users."""
    __tablename__ = "user_profiles"

    user_id: Mapped[str] = mapped_column(String(255), unique=True, nullable=False, index=True)
    name: Mapped[Optional[str]] = mapped_column(String(255), nullable=True)
    callsign: Mapped[Optional[str]] = mapped_column(String(20), nullable=True)
    roles: Mapped[Optional[str]] = mapped_column(Text, nullable=True)

# FastAPI route
from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy.ext.asyncio import AsyncSession
from typing import Annotated

from meshcore_hub.api.dependencies import get_db, require_read
from meshcore_hub.common.schemas import NodeRead, NodeList

router = APIRouter(prefix="/nodes", tags=["nodes"])

@router.get("", response_model=NodeList)
async def list_nodes(
    db: Annotated[AsyncSession, Depends(get_db)],
    _: Annotated[None, Depends(require_read)],
    limit: int = Query(default=50, le=100),
    offset: int = Query(default=0, ge=0),
) -> NodeList:
    """List all nodes with pagination."""
    pass

# Click CLI command
import click

@click.group()
@click.pass_context
def cli(ctx: click.Context) -> None:
    """MeshCore Hub CLI."""
    ctx.ensure_object(dict)

@cli.command()
@click.option("--host", default="0.0.0.0", help="Bind host")
@click.option("--port", default=8000, type=int, help="Bind port")
@click.pass_context
def api(ctx: click.Context, host: str, port: int) -> None:
    """Start the API server."""
    import uvicorn
    from meshcore_hub.api.app import create_app

    app = create_app()
    uvicorn.run(app, host=host, port=port)

# Async lifespan
from contextlib import asynccontextmanager
from typing import AsyncGenerator
from fastapi import FastAPI

@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
    """Application lifespan handler."""
    await setup_database()
    await connect_mqtt()

    yield

    await disconnect_mqtt()
    await close_database()

# Error handling
from fastapi import HTTPException, status
import logging

logger = logging.getLogger(__name__)

raise HTTPException(
    status_code=status.HTTP_404_NOT_FOUND,
    detail=f"Node with public_key '{public_key}' not found"
)

try:
    result = await risky_operation()
except SomeException as e:
    logger.exception("Failed to perform operation: %s", e)
    raise

Test patterns

# Unit test
import pytest
from unittest.mock import AsyncMock, patch

@pytest.mark.asyncio
async def test_collector_handles_advertisement():
    """Test that collector handler processes advertisement events."""
    handler = AdvertisementHandler(db_session=AsyncMock())

    await handler.handle(event_data)

    handler.db_session.add.assert_called_once()
    node = handler.db_session.add.call_args[0][0]
    assert node.public_key == event_data["public_key"]

# Integration test
import pytest
from httpx import AsyncClient
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession

@pytest.fixture
async def db_session():
    """Create in-memory SQLite database for testing."""
    engine = create_async_engine("sqlite+aiosqlite:///:memory:")
    async with engine.begin() as conn:
        await conn.run_sync(Base.metadata.create_all)

    async with AsyncSession(engine) as session:
        yield session

@pytest.fixture
async def client(db_session):
    """Create test client with database session."""
    app = create_app()
    app.dependency_overrides[get_db] = lambda: db_session

    async with AsyncClient(app=app, base_url="http://test") as client:
        yield client

Where to find the rest

README.md — project overview, deployment, MQTT topics, node tags, data retention, troubleshooting
SCHEMAS.md — API/data schemas
.env.example — all environment variables with defaults and comments
docs/auth.md — OIDC authentication and roles
docs/content.md — custom content (CONTENT_HOME)
docs/i18n.md — translation reference
docs/letsmesh.md — packet decoding and MQTT feed details
docs/seeding.md — seed data
docs/upgrading.md — upgrade notes
docs/webhooks.md — webhook configuration and payloads

17 KiB Raw Blame History