diff --git a/AGENTS.md b/AGENTS.md index 1c5a23d..1ea9b8e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,10 +16,12 @@ This runs all linting, formatting, type checking, tests, and builds for both bac A web interface for MeshCore mesh radio networks. The backend connects to a MeshCore-compatible radio over Serial, TCP, or BLE and exposes REST/WebSocket APIs. The React frontend provides real-time messaging and radio configuration. -**For detailed component documentation, see:** +**For detailed component documentation, see these primary AGENTS.md files:** - `app/AGENTS.md` - Backend (FastAPI, database, radio connection, packet decryption) - `frontend/AGENTS.md` - Frontend (React, state management, WebSocket, components) -- `frontend/src/components/AGENTS.md` - Frontend visualizer feature (a particularly complex and long force-directed graph visualizer component; can skip this file unless you're working on that feature) +Ancillary AGENTS.md files which should generally not be reviewed unless specific work is being performed on those features include: +- `app/AGENTS_MQTT.md` - MQTT architecture (private broker, community analytics, JWT auth, packet format protocol) +- `frontend/src/components/AGENTS_packet_visualizer.md` - Packet visualizer (force-directed graph, advert-path identity, layout engine) ## Architecture Overview @@ -111,7 +113,7 @@ To improve repeater disambiguation in the network visualizer, the backend stores - This is independent of raw-packet payload deduplication. - Paths are keyed per contact + path, with `heard_count`, `first_seen`, and `last_seen`. - Only the N most recent unique paths are retained per contact (currently 10). -- See `frontend/src/components/AGENTS.md` § "Advert-Path Identity Hints" for how the visualizer consumes this data. +- See `frontend/src/components/AGENTS_packet_visualizer.md` § "Advert-Path Identity Hints" for how the visualizer consumes this data. ## Data Flow diff --git a/app/AGENTS_MQTT.md b/app/AGENTS_MQTT.md new file mode 100644 index 0000000..5667504 --- /dev/null +++ b/app/AGENTS_MQTT.md @@ -0,0 +1,340 @@ +# MQTT Architecture + +RemoteTerm implements two independent MQTT publishing systems that share a common base class: + +1. **Private MQTT** — forwards mesh events to a user-configured broker (home automation, logging, alerting) +2. **Community MQTT** — shares raw RF packets with the MeshCore community aggregator for coverage mapping + +Both are optional, configured entirely through the Settings UI, and require no server restart. + +## File Map + +``` +app/ +├── mqtt_base.py # BaseMqttPublisher — shared lifecycle, connection loop, reconnect +├── mqtt.py # MqttPublisher — private broker forwarding +├── community_mqtt.py # CommunityMqttPublisher — community aggregator integration +├── keystore.py # In-memory Ed25519 key storage (community auth) +├── models.py # AppSettings — all MQTT fields (14 total) +├── repository/settings.py # Database CRUD for MQTT settings +├── routers/settings.py # PATCH /api/settings — validates + restarts publishers +├── routers/health.py # GET /api/health — mqtt_status, community_mqtt_status +├── websocket.py # broadcast_event() — fans out to WS + both MQTT publishers +└── migrations.py # Migration 031 (private fields), 032 (community fields) + +frontend/src/ +├── components/settings/SettingsMqttSection.tsx # Dual collapsible settings UI +└── types.ts # AppSettings, AppSettingsUpdate, HealthStatus + +tests/ +├── test_mqtt.py # Topic routing, lifecycle +├── test_community_mqtt.py # JWT generation, packet format, hash, broadcast +└── test_health_mqtt_status.py # Health endpoint status reporting +``` + +## Base Publisher (`app/mqtt_base.py`) + +`BaseMqttPublisher` is an abstract class that manages the full MQTT client lifecycle for both publishers. Subclasses implement hooks; the base class owns the connection loop. + +### Connection Loop + +The `_connection_loop()` runs as a background `asyncio.Task` and never exits unless cancelled: + +``` +loop: + ├─ _is_configured()? No → call _on_not_configured(), wait for settings change, loop + ├─ _pre_connect()? False → wait and retry + ├─ Build client via _build_client_kwargs() + ├─ Connect with aiomqtt.Client + ├─ Set connected=True, broadcast success toast via _on_connected() + ├─ Wait in 60s intervals: + │ ├─ Settings version changed? → break, reconnect with new settings + │ ├─ _should_break_wait()? → break (e.g., JWT expiry) + │ └─ Otherwise keep waiting (paho-mqtt handles keepalive internally) + ├─ On error: set connected=False, broadcast error toast, exponential backoff + └─ On cancel: cleanup and exit +``` + +### Abstract Hooks + +| Hook | Returns | Purpose | +|------|---------|---------| +| `_is_configured()` | `bool` | Should the publisher attempt to connect? | +| `_build_client_kwargs(settings)` | `dict` | Arguments for `aiomqtt.Client(...)` | +| `_on_connected(settings)` | `(title, detail)` | Success toast content | +| `_on_error()` | `(title, detail)` | Error toast content | + +### Optional Hooks + +| Hook | Default | Purpose | +|------|---------|---------| +| `_pre_connect(settings)` | `return True` | Async setup before connect; return `False` to retry | +| `_should_break_wait(elapsed)` | `return False` | Force reconnect while connected (e.g., token renewal) | +| `_on_not_configured()` | no-op | Called repeatedly while waiting for configuration | + +### Lifecycle Methods + +- `start(settings)` — stores settings, starts the background loop task +- `stop()` — cancels the task, disconnects the client +- `restart(settings)` — `stop()` then `start()` (called when settings change) +- `publish(topic, payload)` — JSON-serializes and publishes; silently drops if disconnected + +### Backoff + +Reconnect delay: 5 seconds minimum, exponential growth, capped at `_backoff_max` (30s for private, 60s for community). Resets on successful connect. + +### QoS + +All publishing uses QoS 0 (at-most-once delivery), the aiomqtt default. + +## Private MQTT (`app/mqtt.py`) + +### When It Connects + +`_is_configured()` returns `True` when all of: +- `mqtt_broker_host` is non-empty +- At least one of `mqtt_publish_messages` or `mqtt_publish_raw_packets` is enabled + +If the user unchecks both publish toggles and saves, the publisher disconnects and the health status shows "Disabled". + +### Client Configuration + +```python +hostname: settings.mqtt_broker_host +port: settings.mqtt_broker_port (default 1883) +username: settings.mqtt_username or None +password: settings.mqtt_password or None +tls_context: ssl.create_default_context() if mqtt_use_tls, else None + # mqtt_tls_insecure=True disables hostname check + cert verification +``` + +TLS is opt-in. When enabled with `mqtt_tls_insecure`, both `check_hostname` and `verify_mode` are relaxed for self-signed certificates. + +### Topic Structure + +Default prefix: `meshcore` (configurable via `mqtt_topic_prefix`). + +**Decrypted messages** (when `mqtt_publish_messages` is on): +- `{prefix}/dm:{contact_key}` — private DM +- `{prefix}/gm:{channel_key}` — channel message +- `{prefix}/message:{conversation_key}` — fallback for unknown type + +**Raw packets** (when `mqtt_publish_raw_packets` is on): +- `{prefix}/raw/dm:{contact_key}` — attributed to a DM contact +- `{prefix}/raw/gm:{channel_key}` — attributed to a channel +- `{prefix}/raw/unrouted` — unattributed + +Topic routing uses `decrypted_info.contact_key` and `decrypted_info.channel_key` from the raw packet data. + +### Fire-and-Forget Pattern + +`mqtt_broadcast(event_type, data)` is called synchronously from `broadcast_event()` in `websocket.py`. It filters to only `"message"` and `"raw_packet"` events, then creates an `asyncio.Task` for the actual publish. No awaiting — failures are logged at WARNING level and silently dropped. + +## Community MQTT (`app/community_mqtt.py`) + +Implements the [meshcore-packet-capture](https://github.com/agessaman/meshcore-packet-capture) protocol for sharing raw RF packets with the MeshCore community aggregator. + +### When It Connects + +`_is_configured()` returns `True` when all of: +- `community_mqtt_enabled` is `True` +- The radio's private key is available in the keystore (`has_private_key()`) + +The private key is exported from the radio firmware on startup via `export_and_store_private_key()` in `app/keystore.py`. This requires `ENABLE_PRIVATE_KEY_EXPORT` to be enabled in the radio firmware. If unavailable, the publisher broadcasts a warning and waits. + +### Client Configuration + +```python +hostname: community_mqtt_broker_host or "mqtt-us-v1.letsmesh.net" +port: community_mqtt_broker_port or 443 +transport: "websockets" +tls_context: ssl.create_default_context() # always enforced, not user-configurable +websocket_path: "/" +username: "v1_{pubkey_hex}" +password: {jwt_token} +``` + +TLS is always on — the community connection uses WebSocket Secure (WSS) with full certificate verification. There is no option to disable it. + +### JWT Authentication + +The community broker authenticates via Ed25519-signed JWT tokens. + +**Token format:** `header_b64url.payload_b64url.signature_hex` + +**Header:** +```json +{"alg": "Ed25519", "typ": "JWT"} +``` + +**Payload:** +```json +{ + "publicKey": "{PUBKEY_HEX_UPPER}", + "iat": 1234567890, + "exp": 1234654290, + "aud": "{broker_host}", + "owner": "{PUBKEY_HEX_UPPER}", + "client": "RemoteTerm (github.com/jkingsman/Remote-Terminal-for-MeshCore)", + "email": "user@example.com" // optional, only if configured +} +``` + +**Signing:** MeshCore uses an "expanded" 64-byte Ed25519 key format (`scalar[32] || prefix[32]`, the "orlp" format). Standard Ed25519 libraries expect seed format and would re-hash the key. The `_ed25519_sign_expanded()` function performs signing manually using `nacl.bindings.crypto_scalarmult_ed25519_base_noclamp()` — a direct port of meshcore-packet-capture's `ed25519_sign_with_expanded_key()`. + +**Token lifetime:** 24 hours. The `_should_break_wait()` hook forces a reconnect at the 23-hour mark to renew before expiry. + +### Packet Formatting + +`_format_raw_packet()` converts raw packet broadcast data into the meshcore-packet-capture JSON format: + +```json +{ + "origin": "NodeName", + "origin_id": "PUBKEY_HEX_UPPER", + "timestamp": "2024-01-15T10:30:00.000000", + "type": "PACKET", + "direction": "rx", + "time": "10:30:00", + "date": "15/01/2024", + "len": "42", + "packet_type": "5", + "route": "F", + "payload_len": "30", + "raw": "AABBCCDD...", + "SNR": "10.5", + "RSSI": "-85", + "hash": "A1B2C3D4E5F6G7H8", + "path": "ab,cd,ef" +} +``` + +- `origin` is the radio's device name from `meshcore.self_info` +- `route` is derived from the header's bottom 2 bits: `0,1→"F"` (Flood), `2→"D"` (Direct), `3→"T"` (Trace) +- `path` is only present when `route=="D"` +- `hash` matches MeshCore's C++ `Packet::calculatePacketHash()`: SHA-256 of `payload_type[1 byte] + [path_len as uint16 LE, TRACE only] + payload_data`, truncated to first 16 hex characters + +### Topic Structure + +``` +meshcore/{IATA}/{PUBKEY_HEX}/packets +``` + +IATA must be exactly 3 uppercase letters (e.g., `DEN`, `LAX`). Validated both client-side (input maxLength + uppercase conversion) and server-side (regex `^[A-Z]{3}$`, returns HTTP 400 on failure). + +### Only Raw Packets + +The community publisher only handles `"raw_packet"` events. Decrypted messages are never shared with the community — `community_mqtt_broadcast()` explicitly filters `event_type != "raw_packet"`. + +## Event Flow + +``` +Radio RF event + ↓ +meshcore_py library callback + ↓ +app/event_handlers.py (on_contact_message, on_rx_log_data, etc.) + ↓ +Store to SQLite database + ↓ +broadcast_event(event_type, data) ← app/websocket.py + ├─ WebSocket → browser clients + ├─ mqtt_broadcast() ← app/mqtt.py (messages + raw packets) + │ └─ asyncio.create_task(_mqtt_maybe_publish()) + └─ community_mqtt_broadcast() ← app/community_mqtt.py (raw packets only) + └─ asyncio.create_task(_community_maybe_publish()) +``` + +## Settings & Persistence + +### Database Fields (`app_settings` table) + +**Private MQTT** (Migration 031): + +| Column | Type | Default | +|--------|------|---------| +| `mqtt_broker_host` | TEXT | `''` | +| `mqtt_broker_port` | INTEGER | `1883` | +| `mqtt_username` | TEXT | `''` | +| `mqtt_password` | TEXT | `''` | +| `mqtt_use_tls` | INTEGER | `0` | +| `mqtt_tls_insecure` | INTEGER | `0` | +| `mqtt_topic_prefix` | TEXT | `'meshcore'` | +| `mqtt_publish_messages` | INTEGER | `0` | +| `mqtt_publish_raw_packets` | INTEGER | `0` | + +**Community MQTT** (Migration 032): + +| Column | Type | Default | +|--------|------|---------| +| `community_mqtt_enabled` | INTEGER | `0` | +| `community_mqtt_iata` | TEXT | `''` | +| `community_mqtt_broker_host` | TEXT | `'mqtt-us-v1.letsmesh.net'` | +| `community_mqtt_broker_port` | INTEGER | `443` | +| `community_mqtt_email` | TEXT | `''` | + +### Settings API + +`PATCH /api/settings` accepts any subset of MQTT fields. The router tracks whether private or community fields changed independently: + +- If any private MQTT field changed → `await mqtt_publisher.restart(result)` +- If any community MQTT field changed → `await community_publisher.restart(result)` + +This means toggling a publish checkbox triggers a full disconnect/reconnect cycle. + +### Health API + +`GET /api/health` reports both statuses: + +```json +{ + "mqtt_status": "connected | disconnected | disabled", + "community_mqtt_status": "connected | disconnected | disabled" +} +``` + +Status logic for each publisher: +- `_is_configured()` returns `True` → report `"connected"` or `"disconnected"` based on `publisher.connected` +- `_is_configured()` returns `False` → report `"disabled"` + +## App Lifecycle + +**Startup** (in `app/main.py` lifespan): +1. Database connects, radio connects +2. `export_and_store_private_key()` — export Ed25519 key from radio (needed for community auth) +3. Load `AppSettings` from database +4. `mqtt_publisher.start(settings)` — spawns background connection loop +5. `community_publisher.start(settings)` — spawns background connection loop + +**Shutdown:** +1. `community_publisher.stop()` +2. `mqtt_publisher.stop()` +3. Radio and database cleanup + +## Frontend (`SettingsMqttSection.tsx`) + +The MQTT settings UI is a single React component with two collapsible sections (both collapsed by default): + +### Private MQTT Broker Section +- Header shows connection status indicator (green/red/gray dot + label) +- Always visible when expanded: Publish Messages and Publish Raw Packets checkboxes +- Broker configuration (host, port, username, password, TLS, topic prefix) only revealed when at least one publish checkbox is checked +- Responsive grid layout (`grid-cols-1 sm:grid-cols-2`) for host+port and username+password pairs + +### Community Analytics Section +- Header shows connection status indicator +- Enable Community Analytics checkbox +- When enabled: broker host/port, IATA code input (3 chars, auto-uppercase), owner email +- Broker host shows "MQTT over TLS (WebSocket Secure) only" note + +### Shared +- Beta warning banner at the top (links to GitHub issues) +- Single "Save MQTT Settings" button outside both collapsibles +- Save constructs an `AppSettingsUpdate` and calls `PATCH /api/settings` +- Success/error feedback via toast notifications + +## Security Notes + +- **Private MQTT password** is stored in plaintext in SQLite, consistent with the project's trusted-network design. +- **Community MQTT** always uses TLS with full certificate verification. The Ed25519 private key is held in memory only (never persisted to disk) and is used solely for JWT signing. +- **Community data** is limited to raw RF packets — decrypted message content is never shared. diff --git a/frontend/src/components/AGENTS.md b/frontend/src/components/AGENTS_packet_visualizer.md similarity index 100% rename from frontend/src/components/AGENTS.md rename to frontend/src/components/AGENTS_packet_visualizer.md diff --git a/frontend/src/components/settings/SettingsMqttSection.tsx b/frontend/src/components/settings/SettingsMqttSection.tsx index 1b9d643..2f3aba3 100644 --- a/frontend/src/components/settings/SettingsMqttSection.tsx +++ b/frontend/src/components/settings/SettingsMqttSection.tsx @@ -341,6 +341,9 @@ export function SettingsMqttSection({ value={communityMqttBrokerHost} onChange={(e) => setCommunityMqttBrokerHost(e.target.value)} /> +
+ MQTT over TLS (WebSocket Secure) only +