mirror of
https://github.com/MarekWo/mc-webui.git
synced 2026-07-28 12:33:13 +02:00
docs: cover the Observer (MQTT packet capture) feature
user-guide.md gains the Observer tab section (master switch, IATA code, scheduled flood adverts, broker management, live badges) and the updated Settings tab list. architecture.md documents the ObserverManager design (hot path, per-broker paho clients, wire-format fidelity, reload without restart), the observer_brokers table, the /api/observer/* endpoints and the observer_status socket event. whatsnew.md announces the feature and notes the paho-mqtt dependency (image rebuild via standard mcupdate) and plaintext broker credentials. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -83,6 +83,17 @@ The `DeviceManager` handles the connection to the MeshCore device via a direct s
|
||||
- **TCP self-heal** - A `_liveness_watcher_loop` task on the DM event loop calls `force_reconnect()` when no RX event has arrived for `HEALTH_STRICT_MAX_RX_STALE_SEC` (5 min). `send_channel_message` also detects empty-string `concurrent.futures.TimeoutError` from `set_flood_scope_key` (the symptom of a degraded long-lived TCP) and runs an in-place reconnect + one retry before failing. A 30 s cooldown and `_reconnect_lock` prevent churn; `_intentional_disconnect` keeps the DISCONNECTED handler from racing the reconnect. The watcher keeps re-checking staleness even after `_connected` has gone False, so a single failed reconnect (e.g. an empty `self_info`) no longer silently stops all further healing for non-BLE transports
|
||||
- **Raw packet resend** - Own channel sends capture a full hex wire snapshot (`header + transport_codes + path_len + encrypted payload`) into `channel_messages.raw_packet`, rebuilt from the actual `pkt_payload` once echo correlation resolves it and honouring the device's cached `path_hash_mode`. `resend_channel_message()` re-broadcasts that snapshot verbatim via `CMD_SEND_RAW_PACKET` (0x41) so repeaters dedupe by packet hash (`Mesh::hasSeen`) and only previously-unreached nodes pick it up. Requires companion firmware ≥1.16 (`fw_ver_code` ≥ 13), gated via the cached `supports_raw_resend` flag captured from the connect-time `DEVICE_INFO` event. Self-echoes of a resend (the firmware seen-table can evict the hash within minutes on a busy mesh) are detected by recomputing the expected `pkt_payload` and matching an existing own row, so a resend never reappears as an inbound message from yourself
|
||||
|
||||
### Observer (MQTT packet capture)
|
||||
|
||||
`app/observer.py` (`ObserverManager`) implements a meshcore-packet-capture-compatible observer on top of the existing device session — no second connection and no device-side enable command (the firmware pushes `RX_LOG_DATA` for every overheard frame):
|
||||
|
||||
- **Hot path** — `_on_rx_log_data` hands every packet (all payload types, before the GRP_TXT echo gate) to `handle_raw_packet()`: pure CPU work (header/path decode, firmware-exact packet hash, JSON build) plus a non-blocking paho `publish()`, wrapped in its own try/except so observer failures can never affect chat
|
||||
- **One paho-mqtt client per enabled broker**, each with its own network thread (`loop_start`), auto-reconnect with backoff, QoS 0 (no backlog during outages), retained `online`/`offline` status plus an LWT on `meshcore[/IATA/PUBKEY]/status`; packets go to `.../packets` (flat `meshcore/packets|status` when IATA is unset)
|
||||
- **Wire-format fidelity** — the decode/hash/format functions are verbatim ports of upstream meshcore-packet-capture (including legacy path-length fallbacks and the publish-with-defaults quirk for unknown payload versions), validated byte-identical against live packets from the reference observer network
|
||||
- **Config** — `observer_brokers` table plus the `observer_settings` JSON key (`enabled`, `iata`, `advert_interval_hours`) in `app_settings`; every mutating API call triggers `reload()` on a daemon thread (the client list is swapped atomically and the hot path reads it lock-free), so changes apply without restart
|
||||
- **Advert scheduler** — an observer-owned daemon thread checks every 10 minutes and sends a flood advert once `advert_interval_hours` has elapsed since the `observer_last_advert_at` timestamp persisted in `app_settings` (restart-safe)
|
||||
- **Live status** — `observer_status` events on the `/chat` namespace (throttled to one per 2 s on the packet path) drive the Settings-tab badges and counters; `GET /api/observer/status` returns the full merged view
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
@@ -97,6 +108,7 @@ mc-webui/
|
||||
│ ├── config.py # Configuration from env vars
|
||||
│ ├── database.py # SQLite database models and CRUD operations
|
||||
│ ├── device_manager.py # Core logic for meshcore communication
|
||||
│ ├── observer.py # Observer: MQTT packet-capture publishing
|
||||
│ ├── contacts_cache.py # Persistent contacts cache (DB-backed)
|
||||
│ ├── read_status.py # Server-side read status manager (DB-backed)
|
||||
│ ├── version.py # Git-based version management
|
||||
@@ -143,6 +155,7 @@ Key tables:
|
||||
- `channel_scopes` - Per-channel region mapping (`channel_idx` → `region_id`, CASCADE on region delete; absent row = no override → firmware default applies)
|
||||
- `read_status` - Per-channel read counters and favorites (`is_favorite` column; used to pin channels in the sidebar/dropdown sort order)
|
||||
- `analyzers` - User-configured MeshCore Analyzer services (`name`, `url_template` with `{packetHash}` placeholder, `is_default`, `is_disabled`; partial unique index enforces a single default)
|
||||
- `observer_brokers` - MQTT brokers for the Observer packet-capture feature (`name`, `host`, `port`, `username`, `password` — stored plaintext, `use_tls`, `tls_verify`, `is_disabled`)
|
||||
|
||||
`direct_messages` gained a `delivery_path_hash_size` column (auto-migrated, defaults to 1) so reloaded DM bubbles render multi-byte routes correctly. The `path_len` column on `channel_messages`, `direct_messages`, and `paths` now stores the raw firmware byte (masked hop count plus path_hash_mode in the upper bits), recombined at write time via `pack_path_len()`; the API endpoints decode it back into `path_hash_size` on read. `channel_messages` also gained a `raw_packet` column (the full hex wire snapshot captured at send time, indexed by `idx_cm_pkt` on `pkt_payload` for fast self-echo lookups) that powers raw resend; it is `NULL` for received and pre-migration rows, so the resend button stays disabled there.
|
||||
|
||||
@@ -253,6 +266,20 @@ The `PUT /api/channels/<index>/scope` endpoint accepts any `index` in `[0, devic
|
||||
|
||||
The backend no longer ships a pre-built `analyzer_url` per message — channel-message payloads include `packet_hash` instead, and the frontend substitutes `{packetHash}` in the chosen URL template at click time. The Letsmesh Analyzer is not a hardcoded pseudo-row: `seed_default_analyzers()` inserts it as an ordinary `analyzers` row exactly once per install (guarded by the `analyzer_letsmesh_seeded` flag in `app_settings`), so it can be renamed, disabled, or deleted like any user entry. The row stores `is_disabled` (not `is_enabled`); the UI inverts it on read/write so the switch reads "Enabled", which is why no data migration was needed.
|
||||
|
||||
### Observer
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| GET | `/api/observer/settings` | Get observer settings (`enabled`, `iata`, `advert_interval_hours`) |
|
||||
| POST | `/api/observer/settings` | Partial update; IATA must be empty or exactly 3 letters, interval 0-8760 h |
|
||||
| GET | `/api/observer/brokers` | List MQTT brokers (passwords never returned; a `has_password` flag instead) |
|
||||
| POST | `/api/observer/brokers` | Create broker (`{name, host, port?, username?, password?, use_tls?, tls_verify?}`) |
|
||||
| PUT | `/api/observer/brokers/<id>` | Partial update; omitted `password` keeps the stored one, empty string clears it |
|
||||
| DELETE | `/api/observer/brokers/<id>` | Delete broker |
|
||||
| GET | `/api/observer/status` | Settings + broker rows merged with live connection state and packet counters |
|
||||
|
||||
Every mutating endpoint hot-reloads the `ObserverManager`, so broker and setting changes take effect without an app restart.
|
||||
|
||||
### Direct Messages
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
@@ -356,6 +383,7 @@ Real-time message delivery via Socket.IO.
|
||||
- `dm_retry_failed` - All retry attempts exhausted (`dm_id`)
|
||||
- `dm_delivered_info` - Delivery details after ACK (`dm_id`, `attempt`, `max_attempts`, `path`, `hash_size`)
|
||||
- `path_changed` - Contact path discovered/updated (`public_key`)
|
||||
- `observer_status` - Observer live state (`enabled`, `running`, `reason`, packet counters, per-broker `connected`/`last_error`)
|
||||
|
||||
### Logs Namespace (`/logs`)
|
||||
|
||||
|
||||
+20
-1
@@ -602,7 +602,7 @@ Access the Settings modal to configure application behavior:
|
||||
1. Click the menu icon (☰) in the navbar (or tap the gear FAB button)
|
||||
2. Select "Settings" from the menu
|
||||
|
||||
The modal is organized into tabs: **Device**, **Messages**, **Group Chat**, **Interface**, **Appearance**, **Contacts**, **Regions**, **Analyzer**, and **Notifications**. A global **Close** button at the bottom of the modal dismisses Settings from any tab.
|
||||
The modal is organized into tabs: **Device**, **Messages**, **Group Chat**, **Interface**, **Appearance**, **Contacts**, **Regions**, **Analyzer**, **Observer**, and **Notifications**. A global **Close** button at the bottom of the modal dismisses Settings from any tab.
|
||||
|
||||
### Device Tab
|
||||
|
||||
@@ -703,6 +703,25 @@ Each row supports:
|
||||
|
||||
When adding or editing, the URL template must contain the placeholder `{packetHash}` — it is substituted with the message's packet hash at click time.
|
||||
|
||||
### Observer Tab
|
||||
|
||||
Turn your node into a MeshCore *observer*: every packet the device overhears on the radio is forwarded to one or more MQTT brokers, in the same format used by the standard `meshcore-packet-capture` observer script. Packet analyzer services (a self-hosted Corescope, letsmesh-style maps) then see your local mesh traffic — without running a separate capture script or dedicating a second node. Capture is completely passive: chat and direct messages are unaffected.
|
||||
|
||||
- **Enable observer** — master switch for the whole feature
|
||||
- **Location code (IATA)** — a 3-letter code (e.g. `KRK`) included in the MQTT topics so analyzers can group observers by area; leave it empty to publish on the plain `meshcore/...` topics
|
||||
- **Flood advert interval (h)** — when set above 0, the app automatically sends a flood advert every N hours so your observer stays visible on analyzer maps; 0 turns scheduled adverts off. The schedule survives restarts, so a redeploy won't send an extra advert early
|
||||
- **Status line** — shows whether the observer is running (it waits until the device is connected) plus live counters of packets captured and published
|
||||
- **MQTT Brokers** — the servers your packets are sent to. Each row shows a live connection badge (green **connected**, gray **offline**, red **error** — hover it for the reason), an Enabled switch, and Edit/Delete buttons
|
||||
|
||||
Click **Add broker** to configure a server: name, host, port (1883 is the common plain port, 8883 with TLS), optional username and password, a TLS switch, and a certificate-verification switch (turn verification off only for brokers with self-signed certificates). When editing later, leave the password field blank to keep the stored password.
|
||||
|
||||
Notes:
|
||||
|
||||
- Settings and broker changes apply immediately — no restart needed
|
||||
- If a broker is temporarily unreachable, packets during the outage are skipped for that broker (there is no backlog); the observer reconnects automatically
|
||||
- Broker passwords are stored in plain text in the app's local database — use dedicated MQTT credentials, not ones shared with other systems
|
||||
- LetsMesh brokers that require token authentication are not supported yet
|
||||
|
||||
### Notifications Tab
|
||||
|
||||
Enable or disable browser push notifications for new messages received while the app is hidden or in the background.
|
||||
|
||||
+8
-1
@@ -8,7 +8,14 @@ For deep technical notes, see [architecture.md](architecture.md). For the full g
|
||||
|
||||
## Unreleased (since debb711)
|
||||
|
||||
_Nothing yet — the next change to land on `dev` goes here._
|
||||
### New features
|
||||
|
||||
- **Observer mode — feed packet analyzers straight from mc-webui.** The new **Settings → Observer** tab turns your node into a MeshCore observer: every packet the device overhears is published to one or more MQTT brokers in the standard `meshcore-packet-capture` format, so analyzer services (a self-hosted Corescope, letsmesh-style maps) see your local mesh traffic without a separate capture script or a dedicated second node. Configure brokers with host/port, optional username/password and TLS; each row shows a live connected/error badge, and the tab counts packets captured vs published in real time. Capture is completely passive — chat and direct messages are unaffected — and all changes apply immediately, no restart needed. (LetsMesh token-authenticated brokers are not supported yet.)
|
||||
- **Scheduled flood adverts.** The Observer tab includes an optional advert interval in hours: the app sends a flood advert on that schedule so your observer stays visible on analyzer maps. The timer survives restarts, so a redeploy won't send an extra advert early. Set it to 0 to keep adverts fully manual.
|
||||
|
||||
### Deploy notes
|
||||
|
||||
- This update adds a new Python dependency (`paho-mqtt`), so the Docker image must be rebuilt — the standard `mcupdate` flow does this automatically. Broker passwords entered in the Observer tab are stored in plain text in the app database; use dedicated MQTT credentials.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user