diff --git a/.agents/skills/docs-sync/SKILL.md b/.agents/skills/docs-sync/SKILL.md new file mode 100644 index 0000000..31fcd61 --- /dev/null +++ b/.agents/skills/docs-sync/SKILL.md @@ -0,0 +1,169 @@ +--- +name: docs-sync +description: "Audits and fixes discrepancies between project source code (Python config, Docker Compose files) and primary documentation files (README.md, AGENTS.md, UPGRADING.md, .env.example, SCHEMAS.md). Extracts environment variables from Pydantic Settings, Click CLI options, and os.getenv calls; parses Docker Compose services, profiles, volumes, and env passthroughs; verifies feature flags, CLI commands, and file paths referenced in documentation. Produces a structured audit report and applies fixes to keep documentation accurate and up-to-date. Invoke after any config change, env var addition/removal, Docker service modification, feature flag change, or when documentation drift is suspected." +license: MIT +compatibility: opencode +metadata: + author: https://github.com/agessaman + version: "0.1.0" + domain: quality + triggers: documentation sync, docs audit, env vars, config drift, .env.example, README, AGENTS.md, UPGRADING.md, SCHEMAS.md, docker compose docs, feature flags, documentation update, keep docs in sync, documentation accuracy + role: specialist + scope: review + output-format: report + related-skills: code-review, docs-writer +--- + +# Docs Sync + +Documentation accuracy specialist that keeps project docs in sync with source code and Docker configuration. + +## When to Use This Skill + +- After adding, removing, or renaming environment variables in Python config +- After modifying Docker Compose services, profiles, volumes, or port mappings +- After adding or removing feature flags +- After adding or removing CLI commands or subcommands +- After changing Pydantic Settings defaults or types +- When documentation drift is suspected +- Before releases to ensure docs are accurate +- When AGENTS.md or README.md references stale files or commands + +## Primary Documentation Files + +The following files are the documentation targets. All must be kept in sync: + +| File | Role | +|------|------| +| `README.md` | User-facing reference: env var tables, Docker instructions, feature list | +| `AGENTS.md` | AI agent instructions: env var list, project structure, conventions | +| `UPGRADING.md` | Upgrade guide: deprecated vars, new vars, migration steps | +| `.env.example` | Example environment file with comments and defaults | +| `SCHEMAS.md` | Event JSON schemas and database column mappings | + +## Core Workflow + +1. **Extract config from Python source** — Parse all environment variables from three sources: Pydantic Settings classes in `common/config.py`, Click `envvar=` parameters in CLI modules, and direct `os.getenv()`/`os.environ` calls. Build a complete inventory with field names, defaults, types, and descriptions. See `references/config-source-guide.md`. + +2. **Extract Docker configuration** — Parse all `docker-compose*.yml` files for services, compose profiles, volumes, port mappings, environment variable references (with defaults), and device mappings. Distinguish hub-consumed vars from passthrough vars (e.g., `PACKETCAPTURE_*`). See `references/docker-source-guide.md`. + +3. **Extract features and commands** — Verify feature flags have corresponding UI routes and config fields. Verify CLI commands documented in README.md and AGENTS.md still exist. Verify file paths and directory structures referenced in docs actually exist. + +4. **Cross-reference against documentation** — For each of the 5 primary doc files, check every env var, Docker service, feature, command, and path reference against the source-of-truth inventories from steps 1-3. See `references/documentation-checklist.md`. + +5. **Verify inline comments** — Check that all comments in `.env.example` and `docker-compose*.yml` accurately describe the values they annotate. Verify default values in comments match actual defaults from source code. + +6. **Produce report and apply fixes** — Generate a structured discrepancy report. For each discrepancy, apply the fix to the relevant documentation file. Summarize all changes made. + +## Reference Guide + +Load detailed guidance based on context: + +| Topic | Reference | Load When | +|-------|-----------|-----------| +| Config Source Extraction | `references/config-source-guide.md` | Extracting env vars from Python source | +| Docker Source Extraction | `references/docker-source-guide.md` | Parsing Docker Compose files | +| Documentation Checklist | `references/documentation-checklist.md` | Cross-referencing against each doc file | + +## Discrepancy Categories + +| Category | Severity | Description | +|----------|----------|-------------| +| Missing env var in docs | High | Variable exists in source but not in a doc file that should list it | +| Stale env var in docs | High | Variable documented but no longer exists in source | +| Wrong default value | High | Documented default doesn't match actual default | +| Wrong type or description | Medium | Documented type or description doesn't match source | +| Missing Docker service/profile | High | Service or profile exists in compose but not documented | +| Stale Docker service/profile | High | Documented service or profile no longer exists | +| Stale file path reference | Medium | Referenced file or directory doesn't exist | +| Stale CLI command | High | Documented command no longer exists | +| Stale feature reference | Medium | Documented feature flag doesn't exist in config | +| Inaccurate inline comment | Low | Comment doesn't accurately describe the value | +| Missing comment | Low | Value lacks a descriptive comment | +| Stale doc cross-reference | Medium | Reference to removed file (e.g., PLAN.md, TASKS.md) | + +## Report Template + +``` +# Docs Sync Audit Report + +## Summary +- Total discrepancies found: N +- High: N | Medium: N | Low: N +- Files modified: N + +## Environment Variables +### Missing from documentation +| Variable | Default | Missing from | +|----------|---------|-------------| +| ... | ... | README.md, .env.example | + +### Stale (removed from source) +| Variable | Still in | +|----------|---------| +| ... | AGENTS.md, README.md | + +### Wrong defaults +| Variable | Documented | Actual | +|----------|-----------|--------| +| ... | ... | ... | + +## Docker Configuration +### Services/Profiles +[Discrepancies between compose files and docs] + +### Environment Passthroughs +[Missing or stale env vars in .env.example for Docker] + +## Features & Commands +### Stale features +[Documented features that don't exist] + +### Missing features +[Features in source but not documented] + +### Stale commands +[Documented CLI commands that don't exist] + +### Stale file references +[Paths referenced in docs that don't exist] + +## Inline Comments +### Inaccurate comments +| File | Line | Current | Correct | +|------|------|---------|---------| +| ... | ... | ... | ... | + +## Changes Applied +[List of edits made to each file] +``` + +## Constraints + +### MUST DO + +- Treat Python source code (`common/config.py`, CLI modules, `os.getenv` calls) as the single source of truth for environment variables +- Treat `docker-compose*.yml` files as the source of truth for Docker configuration +- Check ALL five documentation files on every audit +- Include `.env.example` comment verification +- Include `docker-compose*.yml` inline comment verification +- Verify default values match exactly (type-aware: `true` vs `"true"`, port numbers as strings vs ints) +- Flag references to removed files (PLAN.md, TASKS.md) in AGENTS.md and README.md +- Apply fixes to documentation files after reporting +- Preserve existing formatting and section structure in doc files +- For AGENTS.md env var sections, maintain the existing table format and grouping + +### MUST NOT DO + +- Modify Python source code or Docker Compose files (only documentation files) +- Add documentation for variables that don't exist in source +- Remove content from documentation without confirming it's stale in source +- Change the formatting style of existing documentation (match surrounding content) +- Modify UPGRADING.md historical content (deprecated var lists, old instructions) +- Skip any of the 5 documentation files +- Guess at defaults — always verify against actual source code +- Treat test compose files (`tests/e2e/`) as documentation targets (they are test fixtures) + +## Knowledge Reference + +Pydantic Settings (BaseSettings, env_file, field defaults), Click (envvar parameter), Docker Compose (profiles, volumes, environment, depends_on), YAML parsing, environment variable naming conventions (UPPER_SNAKE_CASE), MeshCore Hub architecture (collector, API, web, MQTT broker, packet capture observer) diff --git a/.agents/skills/docs-sync/references/config-source-guide.md b/.agents/skills/docs-sync/references/config-source-guide.md new file mode 100644 index 0000000..e2e7076 --- /dev/null +++ b/.agents/skills/docs-sync/references/config-source-guide.md @@ -0,0 +1,139 @@ +# Config Source Guide + +How to extract the complete environment variable inventory from Python source code. + +## Source Files + +These files contain the authoritative definition of all environment variables consumed by the MeshCore Hub application. Check them in this order: + +### 1. Pydantic Settings — `src/meshcore_hub/common/config.py` + +**Primary source of truth.** Contains four Settings classes that define env vars via Pydantic fields: + +| Class | Inherits | Component | Env Vars | +|-------|----------|-----------|----------| +| `CommonSettings` | `BaseSettings` | All services | Base config (MQTT, logging, paths) | +| `CollectorSettings` | `CommonSettings` | Collector | Database, webhooks, retention, cleanup, channel keys | +| `APISettings` | `CommonSettings` | API server | Host, port, auth keys, metrics | +| `WebSettings` | `CommonSettings` | Web dashboard | Host, port, theme, locale, features, network info | + +#### Extraction Method + +Read each class and extract: +- **Field name** — the Python attribute name (e.g., `mqtt_host`) +- **Env var name** — the uppercased field name (e.g., `MQTT_HOST`), unless explicitly overridden via `Field(alias=...)` or `alias` in `model_config` +- **Default value** — the `= value` in the field declaration, or `Field(default=...)` +- **Type** — the type annotation (e.g., `str`, `int`, `bool`, `Optional[str]`, constrained types) +- **Description** — the `Field(description=...)` or docstring, if any +- **Constraints** — `Field(ge=1, le=100)`, `min_length`, `max_length`, etc. + +#### Pydantic-to-Env Mapping Rules + +- Field `mqtt_host` → env var `MQTT_HOST` (automatic uppercasing) +- Field `api_base_url` → env var `API_BASE_URL` +- `Optional[str]` fields with `None` default → env var is optional, no default +- `bool` fields use Pydantic's built-in coercion: `"true"`, `"1"`, `"yes"` → `True`; `"false"`, `"0"`, `"no"` → `False` +- Enum fields (e.g., `LogLevel`, `MQTTTransport`) accept their member values as strings + +#### Computed Properties + +Some settings have computed properties (e.g., `database_url` falls back to `sqlite:///{DATA_HOME}/collector/meshcore.db`). These should be documented as having a computed default. + +#### Settings Inheritance + +`CollectorSettings`, `APISettings`, and `WebSettings` all inherit from `CommonSettings`. The shared env vars (`MQTT_*`, `LOG_LEVEL`, `DATA_HOME`, etc.) should appear once under "Common Settings" in documentation, not repeated per component. + +### 2. Click CLI `envvar` — CLI Entry Points + +These files define Click commands that accept env var overrides via `envvar=` or `envvar=[...]`: + +| File | CLI Group | Key Env Vars | +|------|-----------|--------------| +| `src/meshcore_hub/__main__.py` | Root CLI | `DATA_HOME`, `DATABASE_URL` (also sets into `os.environ` for Alembic) | +| `src/meshcore_hub/collector/cli.py` | `collector` | `DATA_HOME`, `DATABASE_URL`, `SEED_HOME` | +| `src/meshcore_hub/api/cli.py` | `api` | `DATA_HOME`, `DATABASE_URL`, `CORS_ORIGINS`, `MQTT_PREFIX`/`MQTT_TOPIC_PREFIX` | +| `src/meshcore_hub/web/cli.py` | `web` | `DATA_HOME`, `API_BASE_URL` | + +#### Extraction Method + +Grep for `envvar=` in each file. Each `envvar=` parameter maps a Click option to an environment variable. + +#### Key Patterns + +- **Single envvar:** `envvar="CORS_ORIGINS"` — maps `--cors-origins` flag to `CORS_ORIGINS` env var +- **Multiple envvars (alias):** `envvar=["MQTT_PREFIX", "MQTT_TOPIC_PREFIX"]` — first is primary, rest are backward-compat aliases +- **Setting os.environ:** `__main__.py` sets `os.environ["DATABASE_URL"]` from CLI args so Alembic can pick it up. This is a pass-through, not a new env var. + +#### Edge Case: `CORS_ORIGINS` + +`CORS_ORIGINS` exists only as a Click `envvar` in `api/cli.py` — it has **no** Pydantic Settings field. It should still be documented as an env var consumed by the API component, but note that it only works when launched via the CLI (not when running with `--reload` which bypasses Click). + +### 3. Direct `os.getenv()` / `os.environ` Access + +Some code bypasses Pydantic Settings entirely and reads env vars directly: + +| File | Env Var | Default | Purpose | +|------|---------|---------|---------| +| `src/meshcore_hub/web/app.py` | `COLLECTOR_CHANNEL_KEYS` | `None` | Reads channel keys for web UI label building | +| `src/meshcore_hub/web/app.py` | `COLLECTOR_INCLUDE_TEST_CHANNEL` | `"false"` | Reads test channel flag for web UI | +| `src/meshcore_hub/common/health.py` | `HEALTH_DIR` | `/tmp/meshcore-hub` | Health status file directory | +| `src/meshcore_hub/alembic/env.py` | `DATABASE_URL` | Falls to config | Alembic migration DB URL | +| `src/meshcore_hub/alembic/env.py` | `DATA_HOME` | Falls to config | Alembic fallback for computing DB URL | + +#### Extraction Method + +Grep for these patterns across `src/meshcore_hub/`: +- `os.getenv(` +- `os.environ.get(` +- `os.environ[` +- `os.environ.setdefault(` + +#### Key Observation + +`HEALTH_DIR` is defined in `common/health.py` but has **no** Pydantic Settings field. It should be documented in AGENTS.md and .env.example if it's user-configurable, or noted as an internal variable if not. + +## Classification: Hub Vars vs. Passthrough Vars + +Not all env vars in `.env.example` are consumed by the Hub's Python code. Some are passed through to external containers via Docker Compose: + +### Hub-Consumed Variables +Read by `meshcore-hub` Python code. Source: `config.py`, CLI modules, direct `os.getenv`. +These MUST be documented in all 5 doc files. + +### Docker Passthrough Variables +Read by external containers (packet-capture, MQTT broker). Source: `docker-compose.yml` `environment:` blocks. +These MUST be in `.env.example` and `README.md` Docker sections, but NOT in AGENTS.md env var sections (since the Hub doesn't consume them). + +Passthrough prefixes: +- `PACKETCAPTURE_*` — consumed by `ghcr.io/agessaman/meshcore-packet-capture` +- `MQTT_TOKEN_AUDIENCE` — consumed by `ghcr.io/ipnet-mesh/meshcore-mqtt-broker` +- `COMPOSE_PROJECT_NAME` — consumed by Docker Compose itself +- `IMAGE_VERSION`, `PACKETCAPTURE_IMAGE_VERSION` — Docker image tags +- `TRAEFIK_DOMAIN` — consumed by Traefik labels in `docker-compose.traefik.yml` +- `SERIAL_PORT` — device mapping for packet capture container +- `PROMETHEUS_PORT`, `ALERTMANAGER_PORT` — port mappings for monitoring stack + +## Complete Extraction Checklist + +For a full audit, run these extractions: + +1. **Read `common/config.py`** — extract all fields from `CommonSettings`, `CollectorSettings`, `APISettings`, `WebSettings` +2. **Read `__main__.py`** — extract Click `envvar=` parameters and `os.environ` writes +3. **Read `collector/cli.py`** — extract Click `envvar=` parameters +4. **Read `api/cli.py`** — extract Click `envvar=` parameters (especially `CORS_ORIGINS`) +5. **Read `web/cli.py`** — extract Click `envvar=` parameters +6. **Read `web/app.py`** — extract `os.getenv()` calls (channel keys, test channel) +7. **Read `common/health.py`** — extract `os.environ.get()` calls (HEALTH_DIR) +8. **Read `alembic/env.py`** — extract `os.environ.get()` calls (DATABASE_URL, DATA_HOME) +9. **Deduplicate** — merge all sources, noting which vars appear in multiple places +10. **Classify** — tag each var as hub-consumed vs. Docker passthrough + +## Three-Layer Config Precedence + +When documenting defaults, understand the precedence: + +1. **Click `envvar=` CLI option** — highest priority, overrides everything +2. **Pydantic Settings field default** — used when no CLI option or env var is set +3. **`.env` file** — loaded by `python-dotenv` at `__main__.py` startup, feeds into Pydantic Settings + +The documented default should be the Pydantic Settings field default, since that's what applies in the general case. diff --git a/.agents/skills/docs-sync/references/docker-source-guide.md b/.agents/skills/docs-sync/references/docker-source-guide.md new file mode 100644 index 0000000..7eb8746 --- /dev/null +++ b/.agents/skills/docs-sync/references/docker-source-guide.md @@ -0,0 +1,275 @@ +# Docker Source Guide + +How to extract Docker configuration from all Compose files and verify documentation accuracy. + +## Compose Files + +| File | Purpose | Scope | +|------|---------|-------| +| `docker-compose.yml` | Base shared config | All services, profiles, volumes | +| `docker-compose.dev.yml` | Development overrides | Port mappings, dev dependencies | +| `docker-compose.prod.yml` | Production overrides | External proxy network | +| `docker-compose.traefik.yml` | Traefik auto-discovery labels | HTTP routing, TLS | + +**Out of scope:** `tests/e2e/docker-compose.test.yml` is a test fixture with hardcoded values. Do NOT audit it against documentation. + +## Services + +### Service Inventory + +Extract from `docker-compose.yml`: + +| Service | Image | Profiles | Command | +|---------|-------|----------|---------| +| `mqtt` | `ghcr.io/ipnet-mesh/meshcore-mqtt-broker:latest` | `all`, `mqtt` | (default) | +| `observer` | `ghcr.io/agessaman/meshcore-packet-capture:${PACKETCAPTURE_IMAGE_VERSION}` | `all`, `observer` | (default) | +| `collector` | `ghcr.io/ipnet-mesh/meshcore-hub:${IMAGE_VERSION}` | `all`, `core` | `["collector"]` | +| `api` | `ghcr.io/ipnet-mesh/meshcore-hub:${IMAGE_VERSION}` | `all`, `core` | `["api"]` | +| `web` | `ghcr.io/ipnet-mesh/meshcore-hub:${IMAGE_VERSION}` | `all`, `core` | `["web"]` | +| `migrate` | `ghcr.io/ipnet-mesh/meshcore-hub:${IMAGE_VERSION}` | `all`, `core`, `migrate` | `["db", "upgrade"]` | +| `seed` | `ghcr.io/ipnet-mesh/meshcore-hub:${IMAGE_VERSION}` | `seed` | `["collector", "seed"]` | + +### Verification Checklist + +For each service, verify documentation includes: +- [ ] Service name and purpose +- [ ] Correct compose profile membership +- [ ] Dependencies (`depends_on`) +- [ ] Volumes mounted +- [ ] Key environment variables + +## Compose Profiles + +Extract from `docker-compose.yml` `profiles:` keys: + +| Profile | Services | Use Case | +|---------|----------|----------| +| `all` | mqtt, observer, collector, api, web, migrate | Everything on one host | +| `core` | collector, api, web, migrate | Central server (no local MQTT or observer) | +| `mqtt` | mqtt | Local MQTT broker only | +| `observer` | observer | Packet capture observer only | +| `migrate` | migrate | Database migration only | +| `seed` | seed | Seed data import only | + +### Verification + +- [ ] Profile table in README.md matches compose file +- [ ] Profile table in AGENTS.md matches compose file +- [ ] All profile names documented +- [ ] All services within each profile listed +- [ ] Use cases described accurately + +## Volumes + +### Named Volumes + +| Volume | Services | Purpose | +|--------|----------|---------| +| `data` | collector, api, migrate, seed | SQLite database + runtime data | +| `mqtt_data` | mqtt | MQTT broker persistence | +| `observer_data` | observer | Packet capture data | + +Volume names are prefixed: `${COMPOSE_PROJECT_NAME:-hub}_data`, `${COMPOSE_PROJECT_NAME:-hub}_mqtt_data`, etc. + +### Bind Mounts + +| Host Path | Container Path | Service | Mode | +|-----------|---------------|---------|------| +| `${SEED_HOME:-./seed}` | `/seed` | collector | rw | +| `${SEED_HOME:-./seed}` | `/seed` | seed | ro | +| `${CONTENT_HOME:-./content}` | `/content` | web | ro | + +### Verification + +- [ ] All named volumes documented in README.md +- [ ] Bind mount paths match `SEED_HOME` and `CONTENT_HOME` env var defaults +- [ ] Volume naming convention (COMPOSE_PROJECT_NAME prefix) documented + +## Port Mappings + +### Development (docker-compose.dev.yml) + +| Service | Host Port | Container Port | Variable | +|---------|-----------|----------------|----------| +| mqtt | `${MQTT_PORT:-1883}` | `${MQTT_PORT:-1883}` | `MQTT_PORT` | +| api | `${API_PORT:-8000}` | `8000` | `API_PORT` | +| web | `${WEB_PORT:-8080}` | `8080` | `WEB_PORT` | + +### Production (docker-compose.prod.yml) + +No ports exposed. Services connect to external `proxy-net` Docker network. + +### Traefik (docker-compose.traefik.yml) + +Routing via labels. API handles `/api`, `/metrics`, `/health`. Web handles everything else. + +### Verification + +- [ ] Port mappings in README.md match `docker-compose.dev.yml` +- [ ] Production routing description matches `docker-compose.traefik.yml` +- [ ] Container ports match `Dockerfile` exposed ports (8000, 8080) + +## Environment Variables in Docker + +### Per-Service Env Var Extraction + +For each service's `environment:` block, extract every variable. Classify each: + +**Hardcoded (container-internal):** Set to a fixed value in compose file, not configurable via `.env`. These should NOT appear in `.env.example` but SHOULD be noted in README.md service descriptions. +- Example: `DATA_HOME=/data` (collector), `API_HOST=0.0.0.0` (api), `CONTENT_HOME=/content` (web) + +**Variable substitution (user-configurable):** Use `${VAR:-default}` syntax. These MUST appear in `.env.example` with the same default. +- Example: `MQTT_HOST=${MQTT_HOST:-mqtt}`, `LOG_LEVEL=${LOG_LEVEL:-INFO}` + +**Passthrough (no default):** Reference `${VAR}` without a default. These MUST appear in `.env.example` (typically commented out or with empty value). +- Example: `API_READ_KEY`, `API_ADMIN_KEY`, `WEBHOOK_ADVERTISEMENT_URL` + +### Collector Service Env Vars + +Complete list from `docker-compose.yml` collector `environment:` block: + +| Variable | Default in Compose | Category | +|----------|--------------------|----------| +| `LOG_LEVEL` | `INFO` | Common | +| `MQTT_HOST` | `mqtt` | MQTT | +| `MQTT_PORT` | `1883` | MQTT | +| `MQTT_USERNAME` | (empty) | MQTT | +| `MQTT_PASSWORD` | (empty) | MQTT | +| `MQTT_PREFIX` | `meshcore` | MQTT | +| `MQTT_TLS` | `false` | MQTT | +| `MQTT_TRANSPORT` | `websockets` | MQTT | +| `MQTT_WS_PATH` | `/` | MQTT | +| `DATA_HOME` | `/data` (hardcoded) | Path | +| `SEED_HOME` | `/seed` (hardcoded) | Path | +| `COLLECTOR_CHANNEL_KEYS` | (empty) | Collector | +| `COLLECTOR_INCLUDE_TEST_CHANNEL` | `false` | Collector | +| `WEBHOOK_ADVERTISEMENT_URL` | (passthrough) | Webhook | +| `WEBHOOK_ADVERTISEMENT_SECRET` | (passthrough) | Webhook | +| `WEBHOOK_MESSAGE_URL` | (passthrough) | Webhook | +| `WEBHOOK_MESSAGE_SECRET` | (passthrough) | Webhook | +| `WEBHOOK_CHANNEL_MESSAGE_URL` | (passthrough) | Webhook | +| `WEBHOOK_CHANNEL_MESSAGE_SECRET` | (passthrough) | Webhook | +| `WEBHOOK_DIRECT_MESSAGE_URL` | (passthrough) | Webhook | +| `WEBHOOK_DIRECT_MESSAGE_SECRET` | (passthrough) | Webhook | +| `WEBHOOK_TIMEOUT` | `10.0` | Webhook | +| `WEBHOOK_MAX_RETRIES` | `3` | Webhook | +| `WEBHOOK_RETRY_BACKOFF` | `2.0` | Webhook | +| `DATA_RETENTION_ENABLED` | `true` | Retention | +| `DATA_RETENTION_DAYS` | `30` | Retention | +| `DATA_RETENTION_INTERVAL_HOURS` | `24` | Retention | +| `NODE_CLEANUP_ENABLED` | `true` | Node Cleanup | +| `NODE_CLEANUP_DAYS` | `7` | Node Cleanup | + +### API Service Env Vars + +| Variable | Default in Compose | Category | +|----------|--------------------|----------| +| `LOG_LEVEL` | `INFO` | Common | +| `MQTT_HOST` | `mqtt` | MQTT | +| `MQTT_PORT` | `1883` | MQTT | +| `MQTT_USERNAME` | (empty) | MQTT | +| `MQTT_PASSWORD` | (empty) | MQTT | +| `MQTT_PREFIX` | `meshcore` | MQTT | +| `MQTT_TLS` | `false` | MQTT | +| `MQTT_TRANSPORT` | `websockets` | MQTT | +| `MQTT_WS_PATH` | `/` | MQTT | +| `DATA_HOME` | `/data` (hardcoded) | Path | +| `API_HOST` | `0.0.0.0` (hardcoded) | API | +| `API_PORT` | `8000` (hardcoded) | API | +| `API_READ_KEY` | (passthrough) | Auth | +| `API_ADMIN_KEY` | (passthrough) | Auth | +| `METRICS_ENABLED` | `true` | Metrics | +| `METRICS_CACHE_TTL` | `60` | Metrics | + +### Web Service Env Vars + +| Variable | Default in Compose | Category | +|----------|--------------------|----------| +| `LOG_LEVEL` | `INFO` | Common | +| `API_BASE_URL` | `http://api:8000` (hardcoded) | API | +| `API_ADMIN_KEY` / `API_READ_KEY` | (cascading passthrough) | Auth | +| `WEB_HOST` | `0.0.0.0` (hardcoded) | Web | +| `WEB_PORT` | `8080` (hardcoded) | Web | +| `WEB_THEME` | `dark` | Theme | +| `WEB_LOCALE` | `en` | Locale | +| `WEB_DATETIME_LOCALE` | `en-US` | Locale | +| `WEB_ADMIN_ENABLED` | `false` | Admin | +| `NETWORK_NAME` | `MeshCore Network` | Network | +| `NETWORK_CITY` | (empty) | Network | +| `NETWORK_COUNTRY` | (empty) | Network | +| `NETWORK_RADIO_CONFIG` | (empty) | Network | +| `NETWORK_CONTACT_EMAIL` | (empty) | Network | +| `NETWORK_CONTACT_DISCORD` | (empty) | Network | +| `NETWORK_CONTACT_GITHUB` | (empty) | Network | +| `NETWORK_CONTACT_YOUTUBE` | (empty) | Network | +| `NETWORK_WELCOME_TEXT` | (empty) | Network | +| `CONTENT_HOME` | `/content` (hardcoded) | Path | +| `TZ` | `UTC` | Display | +| `COLLECTOR_CHANNEL_KEYS` | (empty) | Display | +| `COLLECTOR_INCLUDE_TEST_CHANNEL` | `false` | Display | +| `FEATURE_DASHBOARD` | `true` | Feature | +| `FEATURE_NODES` | `true` | Feature | +| `FEATURE_ADVERTISEMENTS` | `true` | Feature | +| `FEATURE_MESSAGES` | `true` | Feature | +| `FEATURE_MAP` | `true` | Feature | +| `FEATURE_MEMBERS` | `true` | Feature | +| `FEATURE_PAGES` | `true` | Feature | + +### Observer (Packet Capture) Env Vars + +These are ALL passthrough vars consumed by the external packet capture image. None are read by Hub Python code. + +Grouped by function: +- **Connection:** `SERIAL_PORT`, `PACKETCAPTURE_TIMEOUT`, `PACKETCAPTURE_MAX_CONNECTION_RETRIES`, `PACKETCAPTURE_CONNECTION_RETRY_DELAY`, `PACKETCAPTURE_HEALTH_CHECK_INTERVAL` +- **Identity:** `PACKETCAPTURE_IATA`, `PACKETCAPTURE_ORIGIN` +- **Behavior:** `PACKETCAPTURE_ADVERT_INTERVAL_HOURS`, `PACKETCAPTURE_RF_DATA_TIMEOUT` +- **MQTT Broker 1 (Let's Mesh US):** `PACKETCAPTURE_MQTT1_ENABLED`, `PACKETCAPTURE_MQTT1_SERVER`, `PACKETCAPTURE_MQTT1_PORT`, `PACKETCAPTURE_MQTT1_USE_TLS`, `PACKETCAPTURE_MQTT1_USE_AUTH_TOKEN`, `PACKETCAPTURE_MQTT1_TOKEN_AUDIENCE`, `PACKETCAPTURE_MQTT1_KEEPALIVE` +- **MQTT Broker 2 (Let's Mesh EU):** `PACKETCAPTURE_MQTT2_ENABLED`, `PACKETCAPTURE_MQTT2_SERVER`, `PACKETCAPTURE_MQTT2_PORT`, `PACKETCAPTURE_MQTT2_USE_TLS`, `PACKETCAPTURE_MQTT2_USE_AUTH_TOKEN`, `PACKETCAPTURE_MQTT2_TOKEN_AUDIENCE`, `PACKETCAPTURE_MQTT2_KEEPALIVE` +- **MQTT Broker 3 (Local):** `PACKETCAPTURE_MQTT3_ENABLED`, `PACKETCAPTURE_MQTT3_KEEPALIVE` +- **MQTT Reconnection:** `PACKETCAPTURE_MAX_MQTT_RETRIES`, `PACKETCAPTURE_MQTT_RETRY_DELAY`, `PACKETCAPTURE_EXIT_ON_RECONNECT_FAIL` + +Note: Broker 3 is wired to hub's MQTT vars: `MQTT_HOST`, `MQTT_PORT`, `MQTT_USERNAME`, `MQTT_PASSWORD`, `MQTT_TLS`, `MQTT_TOKEN_AUDIENCE`. + +## Infrastructure-Only Variables + +These appear in `.env.example` and compose files but are NOT consumed by any Python code: + +| Variable | Consumer | Default | +|----------|----------|---------| +| `COMPOSE_PROJECT_NAME` | Docker Compose | `hub` | +| `IMAGE_VERSION` | Docker Compose | `latest` | +| `PACKETCAPTURE_IMAGE_VERSION` | Docker Compose | `latest` | +| `TRAEFIK_DOMAIN` | Traefik labels | (required when using traefik compose) | +| `SERIAL_PORT` | Observer container + device mapping | `/dev/ttyUSB0` | +| `MQTT_TOKEN_AUDIENCE` | MQTT broker container | `mqtt.localhost` | +| `PROMETHEUS_PORT` | Docker port mapping | `9090` | +| `ALERTMANAGER_PORT` | Docker port mapping | `9093` | + +These MUST be in `.env.example` and README.md but NOT in AGENTS.md "Environment Variables" section (since AGENTS.md documents Hub-consumed vars). + +## Dockerfile Verification + +The `Dockerfile` sets ENV defaults that should match compose file defaults: + +| Dockerfile ENV | Value | Must Match | +|----------------|-------|------------| +| `LOG_LEVEL` | `INFO` | Compose `LOG_LEVEL` default | +| `MQTT_HOST` | `mqtt` | Compose `MQTT_HOST` default | +| `MQTT_PORT` | `1883` | Compose `MQTT_PORT` default | +| `MQTT_PREFIX` | `meshcore` | Compose `MQTT_PREFIX` default | +| `DATA_HOME` | `/data` | Compose hardcoded `DATA_HOME` | +| `API_HOST` | `0.0.0.0` | Compose hardcoded `API_HOST` | +| `API_PORT` | `8000` | Compose hardcoded `API_PORT` | +| `WEB_HOST` | `0.0.0.0` | Compose hardcoded `WEB_HOST` | +| `WEB_PORT` | `8080` | Compose hardcoded `WEB_PORT` | +| `API_BASE_URL` | `http://api:8000` | Compose hardcoded `API_BASE_URL` | + +## Inline Comment Verification + +For `docker-compose.yml` and `.env.example`, verify every `# comment` accurately describes the value it annotates. Check: + +1. **Default values in comments** match actual `${VAR:-default}` values +2. **Descriptions** accurately describe what the variable does +3. **Section headers** correctly group related variables +4. **References** to other files or sections are still valid (e.g., "see README.md" references) +5. **Examples** use current, valid values (not outdated formats) diff --git a/.agents/skills/docs-sync/references/documentation-checklist.md b/.agents/skills/docs-sync/references/documentation-checklist.md new file mode 100644 index 0000000..3b9f236 --- /dev/null +++ b/.agents/skills/docs-sync/references/documentation-checklist.md @@ -0,0 +1,260 @@ +# Documentation Checklist + +Per-file verification checklists for each of the 5 primary documentation files. + +## 1. README.md + +### Environment Variable Tables + +README.md contains env var tables grouped by component. For each table: + +- [ ] **Common Settings table** — every var from `CommonSettings` is listed with correct default +- [ ] **Collector Settings table** — every collector-specific var from `CollectorSettings` is listed +- [ ] **Webhook table** — all 11 webhook vars listed (6 URL/secret pairs + timeout + retries + backoff) +- [ ] **Data Retention table** — all retention and node cleanup vars listed +- [ ] **API Settings table** — all API-specific vars from `APISettings` listed +- [ ] **Web Dashboard Settings table** — all web-specific vars from `WebSettings` listed +- [ ] **Feature Flags table** — all 7 feature flags listed +- [ ] **Network Info vars** — all `NETWORK_*` vars listed +- [ ] **Contact Info vars** — all `NETWORK_CONTACT_*` vars listed + +For each variable in each table: +- [ ] Default value matches Pydantic Settings field default exactly +- [ ] Description is accurate and matches field purpose +- [ ] No stale/removed variables remain +- [ ] No variables are duplicated across tables + +### Docker Section + +- [ ] Compose files listed: base, dev, prod, traefik (4 files) +- [ ] Service profiles table matches `docker-compose.yml` exactly +- [ ] All 7 services documented: mqtt, observer, collector, api, web, migrate, seed +- [ ] Port mappings match `docker-compose.dev.yml` +- [ ] Volume names documented with `COMPOSE_PROJECT_NAME` prefix convention +- [ ] Bind mounts documented (SEED_HOME, CONTENT_HOME) +- [ ] Traefik integration instructions reference `TRAEFIK_DOMAIN` +- [ ] Production network setup (`proxy-net`) documented +- [ ] Quick start examples use correct current commands + +### Features Section + +- [ ] Each listed feature corresponds to actual code +- [ ] Feature flag dependency rules documented correctly (Dashboard auto-disables when Nodes/Ads/Messages all off; Map auto-disables when Nodes off) + +### CLI Commands + +- [ ] `meshcore-hub collector` — verify still exists +- [ ] `meshcore-hub api` — verify still exists +- [ ] `meshcore-hub web` — verify still exists +- [ ] `meshcore-hub db upgrade` — verify still exists +- [ ] `meshcore-hub collector seed` — verify still exists +- [ ] `meshcore-hub collector cleanup` — verify still exists +- [ ] All example Docker Compose commands use valid profile names + +### File Paths + +- [ ] `src/meshcore_hub/` structure matches actual layout +- [ ] Seed data directory structure (`node_tags.yaml`, `members.yaml`) documented +- [ ] Custom content directory structure (`pages/`, `media/`) documented +- [ ] Translation files location (`src/meshcore_hub/web/static/locales/`) documented +- [ ] No references to removed files (PLAN.md, TASKS.md) + +## 2. AGENTS.md + +### Environment Variables Section + +AGENTS.md has a "Key variables" subsection under "Environment Variables". Verify: + +- [ ] All hub-consumed env vars listed (from Pydantic Settings + Click + os.getenv) +- [ ] Passthrough vars (`PACKETCAPTURE_*`, `COMPOSE_PROJECT_NAME`, etc.) NOT listed as Hub vars +- [ ] Defaults match Pydantic Settings defaults +- [ ] Descriptions match field purpose +- [ ] Grouping matches the Settings class hierarchy (Common, Collector, API, Web) + +### Project Structure + +- [ ] Directory tree matches actual layout +- [ ] All listed files exist +- [ ] No removed files listed (e.g., PLAN.md, TASKS.md) +- [ ] All new directories/files included + +### Features Documented + +- [ ] Feature flags listed in "Environment Variables" match `WebSettings` fields +- [ ] Dependency rules documented (Dashboard/Map auto-disable logic) +- [ ] Admin auth mechanism documented accurately + +### Code Examples + +- [ ] Import paths use current module structure +- [ ] Class names match current models +- [ ] CLI commands match current Click definitions +- [ ] Async patterns match current codebase conventions + +### Cross-References + +- [ ] References to `PLAN.md` — should be removed (file deleted) +- [ ] References to `TASKS.md` — should be removed (file deleted) +- [ ] References to `SCHEMAS.md` — should remain (file exists) +- [ ] References to `UPGRADING.md` — should remain (file exists) + +## 3. UPGRADING.md + +### Deprecated Variables + +UPGRADING.md lists variables to remove during upgrade. Verify: + +- [ ] Each deprecated var truly no longer exists in `config.py` or any CLI module +- [ ] Removal instructions are clear +- [ ] No currently-active vars are listed as deprecated + +### New Variables + +- [ ] Each new var listed actually exists in current `config.py` +- [ ] Defaults for new vars match Pydantic Settings defaults +- [ ] Migration instructions for adding new vars are correct + +### Renamed Variables + +- [ ] Old name no longer exists anywhere in codebase +- [ ] New name matches current `config.py` field name + +### Docker Migration + +- [ ] Volume rename instructions accurate +- [ ] Old service names truly removed from compose files +- [ ] New compose file structure documented correctly +- [ ] Migration commands still valid for current Docker versions + +### Database Migration + +- [ ] Column renames documented accurately (e.g., `receiver_node_id` → `observer_node_id`) +- [ ] Table renames documented accurately (e.g., `event_receivers` → `event_observers`) + +## 4. .env.example + +### Section Structure + +Verify sections exist and are correctly ordered: + +1. [ ] Quick Start header with observer node example +2. [ ] Common Settings (`COMPOSE_PROJECT_NAME`, `TRAEFIK_DOMAIN`, `IMAGE_VERSION`, `LOG_LEVEL`, `DATA_HOME`, `SEED_HOME`) +3. [ ] MQTT Settings (`MQTT_HOST`, `MQTT_PORT`, `MQTT_USERNAME`, `MQTT_PASSWORD`, `MQTT_PREFIX`, `MQTT_TLS`, `MQTT_TRANSPORT`, `MQTT_WS_PATH`, `MQTT_TOKEN_AUDIENCE`) +4. [ ] Packet Capture Settings (all `PACKETCAPTURE_*` vars + `SERIAL_PORT`) +5. [ ] Collector Settings (`COLLECTOR_CHANNEL_KEYS`, `COLLECTOR_INCLUDE_TEST_CHANNEL`, webhooks, retention, cleanup) +6. [ ] API Settings (`API_PORT`, `API_READ_KEY`, `API_ADMIN_KEY`, metrics) +7. [ ] Web Dashboard Settings (`WEB_PORT`, `API_BASE_URL`, `API_KEY`, theme, locale, auto-refresh, admin, TZ, content home, network info, feature flags, contact info) + +### Per-Variable Checks + +For every variable in `.env.example`: + +- [ ] Variable name matches the env var name exactly (UPPER_SNAKE_CASE) +- [ ] Default value matches Pydantic Settings default OR compose file default (for passthrough vars) +- [ ] Comment above the variable accurately describes its purpose +- [ ] Comment includes valid range/options where applicable (e.g., "DEBUG, INFO, WARNING, ERROR, CRITICAL") +- [ ] Optional vars are commented out (`# VAR=`) with a note about the default +- [ ] Required vars have an uncommented assignment with the default value +- [ ] No duplicate variable entries +- [ ] No removed/stale variables + +### Comment Accuracy + +- [ ] `COMPOSE_PROJECT_NAME` comment mentions container/volume prefix +- [ ] `MQTT_TRANSPORT` comment states WebSocket is required by MeshCore broker +- [ ] `MQTT_WS_PATH` comment notes default `/` vs production `/mqtt` +- [ ] `MQTT_TOKEN_AUDIENCE` comment explains it must match broker config +- [ ] `PACKETCAPTURE_*` comments reference the external packet capture image +- [ ] `COLLECTOR_CHANNEL_KEYS` comment explains label=hex format +- [ ] `WEB_*` comments reference web dashboard behavior +- [ ] `FEATURE_*` comments explain what each flag controls +- [ ] `NETWORK_*` comments explain where values appear in UI +- [ ] `PROMETHEUS_PORT` / `ALERTMANAGER_PORT` comments reference the monitoring profile + +### Missing Variables Check + +- [ ] Every hub-consumed var from `config.py` appears (or is commented out) in `.env.example` +- [ ] Every passthrough var from compose files appears in `.env.example` +- [ ] No extra variables that don't exist in any compose file or Python source + +## 5. SCHEMAS.md + +### Event Schema Verification + +SCHEMAS.md documents the JSON schemas for events stored in the database. Verify: + +- [ ] Each event type documented has a corresponding handler in `src/meshcore_hub/collector/handlers/` +- [ ] Each documented field exists in the corresponding Pydantic schema or SQLAlchemy model +- [ ] Field types match current code (e.g., `str`, `int`, `Optional[str]`) +- [ ] Required vs optional fields match current schema definitions +- [ ] Database column names match current SQLAlchemy model definitions + +### Database Table Verification + +For each table documented in SCHEMAS.md: + +- [ ] Table name matches `__tablename__` in the corresponding SQLAlchemy model +- [ ] Column names match `mapped_column()` field names +- [ ] Column types match (String length, DateTime, Text, Integer, etc.) +- [ ] Foreign key relationships documented correctly +- [ ] Indexes and unique constraints documented correctly +- [ ] New columns from recent migrations are included +- [ ] Removed columns are not documented + +### MQTT Topic Schema + +- [ ] Topic structure documented matches what the collector subscribes to +- [ ] Upload topic format (`///`) is correct +- [ ] Subscriber subscriptions listed match `subscriber.py` topic patterns + +## Cross-File Consistency Checks + +These checks ensure all 5 files are consistent with each other: + +### Env Var Coverage Matrix + +Every hub-consumed env var should appear in: + +| File | Required | Format | +|------|----------|--------| +| `config.py` | Yes (source of truth) | Pydantic field | +| `README.md` | Yes | Table row with default + description | +| `AGENTS.md` | Yes | Mentioned in env vars section | +| `.env.example` | Yes | Entry with default + comment | +| `UPGRADING.md` | Only if new/renamed/deprecated | Migration instruction | + +Every passthrough env var should appear in: + +| File | Required | +|------|----------| +| `docker-compose.yml` | Yes (source of truth) | +| `README.md` | Yes | +| `.env.example` | Yes | +| `AGENTS.md` | No | + +### Default Value Consistency + +- [ ] Same default in README.md tables, .env.example values, and config.py +- [ ] No contradictions between files +- [ ] Type representations consistent (e.g., don't mix `true`/`True`/`1` for booleans) + +### Stale Reference Sweep + +Check all 5 files for references to removed items: + +- [ ] `PLAN.md` — removed, references should be deleted from AGENTS.md +- [ ] `TASKS.md` — removed, references should be deleted from AGENTS.md +- [ ] Old compose profiles (`receiver`, `sender`, `mock`) — should only exist in UPGRADING.md as deprecated +- [ ] Old service names (`interface-receiver`, `interface-sender`) — should only exist in UPGRADING.md as deprecated +- [ ] Old env var names (`COLLECTOR_LETSMESH_DECODER_*`, `SERIAL_BAUD`, etc.) — should only exist in UPGRADING.md as deprecated + +## Applying Fixes + +When discrepancies are found: + +1. **Identify the source of truth** — config.py for env vars, docker-compose.yml for Docker config +2. **Determine scope** — which files need updating +3. **Make minimal edits** — only change what's wrong, preserve surrounding formatting +4. **Match existing style** — tables use same column order, comments use same format, sections use same headers +5. **Preserve historical content** — UPGRADING.md deprecated var lists are historical, do not remove them +6. **Verify after editing** — re-read the changed section to confirm accuracy