Add some additional documentation notes

2026-03-28 17:43:05 +01:00 · 2026-03-02 15:28:32 -08:00
parent f73d10328b
commit b7976206fc
4 changed files with 348 additions and 3 deletions
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -16,10 +16,12 @@ This runs all linting, formatting, type checking, tests, and builds for both bac

 A web interface for MeshCore mesh radio networks. The backend connects to a MeshCore-compatible radio over Serial, TCP, or BLE and exposes REST/WebSocket APIs. The React frontend provides real-time messaging and radio configuration.

-**For detailed component documentation, see:**
+**For detailed component documentation, see these primary AGENTS.md files:**
 - `app/AGENTS.md` - Backend (FastAPI, database, radio connection, packet decryption)
 - `frontend/AGENTS.md` - Frontend (React, state management, WebSocket, components)
- `frontend/src/components/AGENTS.md` - Frontend visualizer feature (a particularly complex and long force-directed graph visualizer component; can skip this file unless you're working on that feature)
+Ancillary AGENTS.md files which should generally not be reviewed unless specific work is being performed on those features include:
+- `app/AGENTS_MQTT.md` - MQTT architecture (private broker, community analytics, JWT auth, packet format protocol)
+- `frontend/src/components/AGENTS_packet_visualizer.md` - Packet visualizer (force-directed graph, advert-path identity, layout engine)

 ## Architecture Overview

@@ -111,7 +113,7 @@ To improve repeater disambiguation in the network visualizer, the backend stores
 - This is independent of raw-packet payload deduplication.
 - Paths are keyed per contact + path, with `heard_count`, `first_seen`, and `last_seen`.
 - Only the N most recent unique paths are retained per contact (currently 10).
- See `frontend/src/components/AGENTS.md` § "Advert-Path Identity Hints" for how the visualizer consumes this data.
+- See `frontend/src/components/AGENTS_packet_visualizer.md` § "Advert-Path Identity Hints" for how the visualizer consumes this data.

 ## Data Flow

--- a/app/AGENTS_MQTT.md
+++ b/app/AGENTS_MQTT.md
@@ -0,0 +1,340 @@
+# MQTT Architecture
+
+RemoteTerm implements two independent MQTT publishing systems that share a common base class:
+
+1. **Private MQTT** — forwards mesh events to a user-configured broker (home automation, logging, alerting)
+2. **Community MQTT** — shares raw RF packets with the MeshCore community aggregator for coverage mapping
+
+Both are optional, configured entirely through the Settings UI, and require no server restart.
+
+## File Map
+
+```
+app/
+├── mqtt_base.py           # BaseMqttPublisher — shared lifecycle, connection loop, reconnect
+├── mqtt.py                # MqttPublisher — private broker forwarding
+├── community_mqtt.py      # CommunityMqttPublisher — community aggregator integration
+├── keystore.py            # In-memory Ed25519 key storage (community auth)
+├── models.py              # AppSettings — all MQTT fields (14 total)
+├── repository/settings.py # Database CRUD for MQTT settings
+├── routers/settings.py    # PATCH /api/settings — validates + restarts publishers
+├── routers/health.py      # GET /api/health — mqtt_status, community_mqtt_status
+├── websocket.py           # broadcast_event() — fans out to WS + both MQTT publishers
+└── migrations.py          # Migration 031 (private fields), 032 (community fields)
+
+frontend/src/
+├── components/settings/SettingsMqttSection.tsx  # Dual collapsible settings UI
+└── types.ts                                     # AppSettings, AppSettingsUpdate, HealthStatus
+
+tests/
+├── test_mqtt.py                  # Topic routing, lifecycle
+├── test_community_mqtt.py        # JWT generation, packet format, hash, broadcast
+└── test_health_mqtt_status.py    # Health endpoint status reporting
+```
+
+## Base Publisher (`app/mqtt_base.py`)
+
+`BaseMqttPublisher` is an abstract class that manages the full MQTT client lifecycle for both publishers. Subclasses implement hooks; the base class owns the connection loop.
+
+### Connection Loop
+
+The `_connection_loop()` runs as a background `asyncio.Task` and never exits unless cancelled:
+
+```
+loop:
+  ├─ _is_configured()? No → call _on_not_configured(), wait for settings change, loop
+  ├─ _pre_connect()? False → wait and retry
+  ├─ Build client via _build_client_kwargs()
+  ├─ Connect with aiomqtt.Client
+  ├─ Set connected=True, broadcast success toast via _on_connected()
+  ├─ Wait in 60s intervals:
+  │   ├─ Settings version changed? → break, reconnect with new settings
+  │   ├─ _should_break_wait()? → break (e.g., JWT expiry)
+  │   └─ Otherwise keep waiting (paho-mqtt handles keepalive internally)
+  ├─ On error: set connected=False, broadcast error toast, exponential backoff
+  └─ On cancel: cleanup and exit
+```
+
+### Abstract Hooks
+
+| Hook | Returns | Purpose |
+|------|---------|---------|
+| `_is_configured()` | `bool` | Should the publisher attempt to connect? |
+| `_build_client_kwargs(settings)` | `dict` | Arguments for `aiomqtt.Client(...)` |
+| `_on_connected(settings)` | `(title, detail)` | Success toast content |
+| `_on_error()` | `(title, detail)` | Error toast content |
+
+### Optional Hooks
+
+| Hook | Default | Purpose |
+|------|---------|---------|
+| `_pre_connect(settings)` | `return True` | Async setup before connect; return `False` to retry |
+| `_should_break_wait(elapsed)` | `return False` | Force reconnect while connected (e.g., token renewal) |
+| `_on_not_configured()` | no-op | Called repeatedly while waiting for configuration |
+
+### Lifecycle Methods
+
+- `start(settings)` — stores settings, starts the background loop task
+- `stop()` — cancels the task, disconnects the client
+- `restart(settings)` — `stop()` then `start()` (called when settings change)
+- `publish(topic, payload)` — JSON-serializes and publishes; silently drops if disconnected
+
+### Backoff
+
+Reconnect delay: 5 seconds minimum, exponential growth, capped at `_backoff_max` (30s for private, 60s for community). Resets on successful connect.
+
+### QoS
+
+All publishing uses QoS 0 (at-most-once delivery), the aiomqtt default.
+
+## Private MQTT (`app/mqtt.py`)
+
+### When It Connects
+
+`_is_configured()` returns `True` when all of:
+- `mqtt_broker_host` is non-empty
+- At least one of `mqtt_publish_messages` or `mqtt_publish_raw_packets` is enabled
+
+If the user unchecks both publish toggles and saves, the publisher disconnects and the health status shows "Disabled".
+
+### Client Configuration
+
+```python
+hostname:    settings.mqtt_broker_host
+port:        settings.mqtt_broker_port (default 1883)
+username:    settings.mqtt_username or None
+password:    settings.mqtt_password or None
+tls_context: ssl.create_default_context() if mqtt_use_tls, else None
+             # mqtt_tls_insecure=True disables hostname check + cert verification
+```
+
+TLS is opt-in. When enabled with `mqtt_tls_insecure`, both `check_hostname` and `verify_mode` are relaxed for self-signed certificates.
+
+### Topic Structure
+
+Default prefix: `meshcore` (configurable via `mqtt_topic_prefix`).
+
+**Decrypted messages** (when `mqtt_publish_messages` is on):
+- `{prefix}/dm:{contact_key}` — private DM
+- `{prefix}/gm:{channel_key}` — channel message
+- `{prefix}/message:{conversation_key}` — fallback for unknown type
+
+**Raw packets** (when `mqtt_publish_raw_packets` is on):
+- `{prefix}/raw/dm:{contact_key}` — attributed to a DM contact
+- `{prefix}/raw/gm:{channel_key}` — attributed to a channel
+- `{prefix}/raw/unrouted` — unattributed
+
+Topic routing uses `decrypted_info.contact_key` and `decrypted_info.channel_key` from the raw packet data.
+
+### Fire-and-Forget Pattern
+
+`mqtt_broadcast(event_type, data)` is called synchronously from `broadcast_event()` in `websocket.py`. It filters to only `"message"` and `"raw_packet"` events, then creates an `asyncio.Task` for the actual publish. No awaiting — failures are logged at WARNING level and silently dropped.
+
+## Community MQTT (`app/community_mqtt.py`)
+
+Implements the [meshcore-packet-capture](https://github.com/agessaman/meshcore-packet-capture) protocol for sharing raw RF packets with the MeshCore community aggregator.
+
+### When It Connects
+
+`_is_configured()` returns `True` when all of:
+- `community_mqtt_enabled` is `True`
+- The radio's private key is available in the keystore (`has_private_key()`)
+
+The private key is exported from the radio firmware on startup via `export_and_store_private_key()` in `app/keystore.py`. This requires `ENABLE_PRIVATE_KEY_EXPORT` to be enabled in the radio firmware. If unavailable, the publisher broadcasts a warning and waits.
+
+### Client Configuration
+
+```python
+hostname:       community_mqtt_broker_host or "mqtt-us-v1.letsmesh.net"
+port:           community_mqtt_broker_port or 443
+transport:      "websockets"
+tls_context:    ssl.create_default_context()  # always enforced, not user-configurable
+websocket_path: "/"
+username:       "v1_{pubkey_hex}"
+password:       {jwt_token}
+```
+
+TLS is always on — the community connection uses WebSocket Secure (WSS) with full certificate verification. There is no option to disable it.
+
+### JWT Authentication
+
+The community broker authenticates via Ed25519-signed JWT tokens.
+
+**Token format:** `header_b64url.payload_b64url.signature_hex`
+
+**Header:**
+```json
+{"alg": "Ed25519", "typ": "JWT"}
+```
+
+**Payload:**
+```json
+{
+  "publicKey": "{PUBKEY_HEX_UPPER}",
+  "iat": 1234567890,
+  "exp": 1234654290,
+  "aud": "{broker_host}",
+  "owner": "{PUBKEY_HEX_UPPER}",
+  "client": "RemoteTerm (github.com/jkingsman/Remote-Terminal-for-MeshCore)",
+  "email": "user@example.com"  // optional, only if configured
+}
+```
+
+**Signing:** MeshCore uses an "expanded" 64-byte Ed25519 key format (`scalar[32] || prefix[32]`, the "orlp" format). Standard Ed25519 libraries expect seed format and would re-hash the key. The `_ed25519_sign_expanded()` function performs signing manually using `nacl.bindings.crypto_scalarmult_ed25519_base_noclamp()` — a direct port of meshcore-packet-capture's `ed25519_sign_with_expanded_key()`.
+
+**Token lifetime:** 24 hours. The `_should_break_wait()` hook forces a reconnect at the 23-hour mark to renew before expiry.
+
+### Packet Formatting
+
+`_format_raw_packet()` converts raw packet broadcast data into the meshcore-packet-capture JSON format:
+
+```json
+{
+  "origin":      "NodeName",
+  "origin_id":   "PUBKEY_HEX_UPPER",
+  "timestamp":   "2024-01-15T10:30:00.000000",
+  "type":        "PACKET",
+  "direction":   "rx",
+  "time":        "10:30:00",
+  "date":        "15/01/2024",
+  "len":         "42",
+  "packet_type": "5",
+  "route":       "F",
+  "payload_len": "30",
+  "raw":         "AABBCCDD...",
+  "SNR":         "10.5",
+  "RSSI":        "-85",
+  "hash":        "A1B2C3D4E5F6G7H8",
+  "path":        "ab,cd,ef"
+}
+```
+
+- `origin` is the radio's device name from `meshcore.self_info`
+- `route` is derived from the header's bottom 2 bits: `0,1→"F"` (Flood), `2→"D"` (Direct), `3→"T"` (Trace)
+- `path` is only present when `route=="D"`
+- `hash` matches MeshCore's C++ `Packet::calculatePacketHash()`: SHA-256 of `payload_type[1 byte] + [path_len as uint16 LE, TRACE only] + payload_data`, truncated to first 16 hex characters
+
+### Topic Structure
+
+```
+meshcore/{IATA}/{PUBKEY_HEX}/packets
+```
+
+IATA must be exactly 3 uppercase letters (e.g., `DEN`, `LAX`). Validated both client-side (input maxLength + uppercase conversion) and server-side (regex `^[A-Z]{3}$`, returns HTTP 400 on failure).
+
+### Only Raw Packets
+
+The community publisher only handles `"raw_packet"` events. Decrypted messages are never shared with the community — `community_mqtt_broadcast()` explicitly filters `event_type != "raw_packet"`.
+
+## Event Flow
+
+```
+Radio RF event
+  ↓
+meshcore_py library callback
+  ↓
+app/event_handlers.py (on_contact_message, on_rx_log_data, etc.)
+  ↓
+Store to SQLite database
+  ↓
+broadcast_event(event_type, data)   ← app/websocket.py
+  ├─ WebSocket → browser clients
+  ├─ mqtt_broadcast()               ← app/mqtt.py (messages + raw packets)
+  │   └─ asyncio.create_task(_mqtt_maybe_publish())
+  └─ community_mqtt_broadcast()     ← app/community_mqtt.py (raw packets only)
+      └─ asyncio.create_task(_community_maybe_publish())
+```
+
+## Settings & Persistence
+
+### Database Fields (`app_settings` table)
+
+**Private MQTT** (Migration 031):
+
+| Column | Type | Default |
+|--------|------|---------|
+| `mqtt_broker_host` | TEXT | `''` |
+| `mqtt_broker_port` | INTEGER | `1883` |
+| `mqtt_username` | TEXT | `''` |
+| `mqtt_password` | TEXT | `''` |
+| `mqtt_use_tls` | INTEGER | `0` |
+| `mqtt_tls_insecure` | INTEGER | `0` |
+| `mqtt_topic_prefix` | TEXT | `'meshcore'` |
+| `mqtt_publish_messages` | INTEGER | `0` |
+| `mqtt_publish_raw_packets` | INTEGER | `0` |
+
+**Community MQTT** (Migration 032):
+
+| Column | Type | Default |
+|--------|------|---------|
+| `community_mqtt_enabled` | INTEGER | `0` |
+| `community_mqtt_iata` | TEXT | `''` |
+| `community_mqtt_broker_host` | TEXT | `'mqtt-us-v1.letsmesh.net'` |
+| `community_mqtt_broker_port` | INTEGER | `443` |
+| `community_mqtt_email` | TEXT | `''` |
+
+### Settings API
+
+`PATCH /api/settings` accepts any subset of MQTT fields. The router tracks whether private or community fields changed independently:
+
+- If any private MQTT field changed → `await mqtt_publisher.restart(result)`
+- If any community MQTT field changed → `await community_publisher.restart(result)`
+
+This means toggling a publish checkbox triggers a full disconnect/reconnect cycle.
+
+### Health API
+
+`GET /api/health` reports both statuses:
+
+```json
+{
+  "mqtt_status": "connected | disconnected | disabled",
+  "community_mqtt_status": "connected | disconnected | disabled"
+}
+```
+
+Status logic for each publisher:
+- `_is_configured()` returns `True` → report `"connected"` or `"disconnected"` based on `publisher.connected`
+- `_is_configured()` returns `False` → report `"disabled"`
+
+## App Lifecycle
+
+**Startup** (in `app/main.py` lifespan):
+1. Database connects, radio connects
+2. `export_and_store_private_key()` — export Ed25519 key from radio (needed for community auth)
+3. Load `AppSettings` from database
+4. `mqtt_publisher.start(settings)` — spawns background connection loop
+5. `community_publisher.start(settings)` — spawns background connection loop
+
+**Shutdown:**
+1. `community_publisher.stop()`
+2. `mqtt_publisher.stop()`
+3. Radio and database cleanup
+
+## Frontend (`SettingsMqttSection.tsx`)
+
+The MQTT settings UI is a single React component with two collapsible sections (both collapsed by default):
+
+### Private MQTT Broker Section
+- Header shows connection status indicator (green/red/gray dot + label)
+- Always visible when expanded: Publish Messages and Publish Raw Packets checkboxes
+- Broker configuration (host, port, username, password, TLS, topic prefix) only revealed when at least one publish checkbox is checked
+- Responsive grid layout (`grid-cols-1 sm:grid-cols-2`) for host+port and username+password pairs
+
+### Community Analytics Section
+- Header shows connection status indicator
+- Enable Community Analytics checkbox
+- When enabled: broker host/port, IATA code input (3 chars, auto-uppercase), owner email
+- Broker host shows "MQTT over TLS (WebSocket Secure) only" note
+
+### Shared
+- Beta warning banner at the top (links to GitHub issues)
+- Single "Save MQTT Settings" button outside both collapsibles
+- Save constructs an `AppSettingsUpdate` and calls `PATCH /api/settings`
+- Success/error feedback via toast notifications
+
+## Security Notes
+
+- **Private MQTT password** is stored in plaintext in SQLite, consistent with the project's trusted-network design.
+- **Community MQTT** always uses TLS with full certificate verification. The Ed25519 private key is held in memory only (never persisted to disk) and is used solely for JWT signing.
+- **Community data** is limited to raw RF packets — decrypted message content is never shared.
--- a/frontend/src/components/AGENTS_packet_visualizer.md
+++ b/frontend/src/components/AGENTS_packet_visualizer.md
--- a/frontend/src/components/settings/SettingsMqttSection.tsx
+++ b/frontend/src/components/settings/SettingsMqttSection.tsx
@@ -341,6 +341,9 @@ export function SettingsMqttSection({
                      value={communityMqttBrokerHost}
                      onChange={(e) => setCommunityMqttBrokerHost(e.target.value)}
                    />
+                    <p className="text-xs text-muted-foreground">
+                      MQTT over TLS (WebSocket Secure) only
+                    </p>
                  </div>
                  <div className="space-y-2">
                    <Label htmlFor="community-broker-port">Broker Port</Label>