# mc-webui Architecture Technical documentation for mc-webui, covering system architecture, project structure, and internal APIs. ## Table of Contents - [Tech Stack](#tech-stack) - [Container Architecture](#container-architecture) - [DeviceManager Architecture](#devicemanager-architecture) - [Project Structure](#project-structure) - [Database Architecture](#database-architecture) - [API Reference](#api-reference) - [WebSocket API](#websocket-api) - [Versioning & Releases](#versioning--releases) - [Offline Support](#offline-support) - [Share Tokens & Cards](#share-tokens--cards) --- ## Tech Stack - **Backend:** Python 3.11+, Flask, Flask-SocketIO (gevent), SQLite - **Frontend:** HTML5, Bootstrap 5, vanilla JavaScript, Socket.IO client - **Deployment:** Docker / Docker Compose (Single-container architecture) - **Communication:** Direct hardware access (USB, BLE, or TCP) via `meshcore` library - **Data source:** SQLite Database (`./data/meshcore/.db`) --- ## Container Architecture mc-webui uses a **single-container architecture** for simplified deployment and direct hardware communication: ```text ┌─────────────────────────────────────────────────────────────┐ │ Docker Network │ │ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ mc-webui │ │ │ │ │ │ │ │ - Flask web app (Port 5000) │ │ │ │ - DeviceManager (Direct USB/BLE/TCP access) │ │ │ │ - Database (SQLite) │ │ │ │ │ │ │ └─────────┬─────────────────────────────────────────────┘ │ │ │ │ └────────────┼─────────────────────────────────────────────────┘ │ ▼ ┌──────────────┐ │ USB/BLE/TCP │ │ Device │ └──────────────┘ ``` Three transport options are supported with the following priority: **BLE > TCP > Serial (USB)**. Set the `MC_BLE_ADDRESS` or `MC_TCP_HOST` environment variable to activate BLE or TCP transport respectively; otherwise, USB serial is used by default. This v2 architecture eliminates the need for a separate bridge container and relies on the native `meshcore` Python library for direct communication, ensuring lower latency and greater stability. ### Docker Entrypoint (BLE cleanup) `scripts/docker-entrypoint.sh` runs before the Flask app starts. When `MC_BLE_ADDRESS` is set, it uses D-Bus to check if BlueZ has an active session to the device and disconnects it. BlueZ auto-reconnects trusted devices, which leaves stale GATT notification handles that block `bleak` from establishing a new session. A clean disconnect at startup ensures the app starts with a fresh BLE state. ### Multi-architecture Images Official images are built via GitHub Actions for `linux/amd64`, `linux/arm64`, and `linux/arm/v7` (Raspberry Pi 2/3/4/5 supported). Build dependencies (`gcc`, `python3-dev`, `libjpeg-dev`, `zlib1g-dev`) are installed and then purged to keep the final image size small while still compiling `Pillow` and `pycryptodome` from source when wheels are unavailable (notably on `arm/v7`). GHA layer cache (`cache-from` / `cache-to`) speeds up subsequent rebuilds. Images are pushed to both Docker Hub (`mawoj/mc-webui`) and GitHub Container Registry (`ghcr.io/marekwo/mc-webui`), with `latest` tag on `main` and `dev` tag on the `dev` branch. --- ## DeviceManager Architecture The `DeviceManager` handles the connection to the MeshCore device via a direct session: - **Single persistent session** - One long-lived connection utilizing the `meshcore` library - **Event-driven** - Subscribes to device events (e.g., incoming messages, advert receptions, ACKs) and triggers appropriate handlers - **Direct Database integration** - Seamlessly syncs contacts, messages, and device settings to the SQLite database - **Real-time messages** - Instant message processing via callback events without polling - **Thread-safe queue** - Commands are serialized to prevent device lockups - **Auto-restart watchdog** - Monitors connection health and restarts the session on crash - **BLE keepalive & reconnect** - When using Bluetooth transport, a 60s keepalive loop detects "zombie" connections (reads still succeed but writes silently fail). On disconnect or keepalive failure, the manager marks the session as permanently failed and the `/health` endpoint returns 503, letting the Docker healthcheck trigger a fast container restart (~5s) to get a clean BLE state rather than attempting unreliable in-process reconnects - **Echo correlation** - Sent channel messages pre-compute their expected `pkt_payload` using the channel secret and send timestamp (±3s for clock drift), so incoming echoes are matched exactly instead of only by 1-byte channel hash (prevents misattribution when two messages go out simultaneously on the same channel). Every recent send stays armed in `_pending_echoes` (one entry per message, capped at 32) rather than a single slot, so a resend or a back-to-back send can no longer swallow another message's echo; exact matching stays valid for 5 min because repeats do arrive minutes late, while the loose channel-hash fallback (no secret available) keeps its 60s window since it can grab a foreign message. A resend re-arms correlation from `_payload_from_raw_packet(raw_packet)` when the message has no `pkt_payload` yet, which is what lets it recover the badge of a message that missed its own echo - **Per-channel region scope** - Before each channel send, the channel's mapped region scope key (16 bytes) is pushed to the firmware via `CMD_SET_FLOOD_SCOPE_KEY` (54). The scope-set + send pair is serialised under a `_send_lock` so concurrent sends on different channels can't swap each other's scope. Channels without a mapping get an all-zero key so a previously-set scope doesn't leak across channels - **Per-send channel-secret refresh** - Channel indices on the device compact down after a deletion, so the boot-time `_load_channel_secrets()` cache can drift. `send_channel_message` calls `_refresh_channel_secret(idx)` first (one extra `get_channel(idx)` round-trip) to fetch the current secret straight from firmware, update the in-memory cache and DB if they had drifted, and use it for the `pkt_payload` echo correlation - **Liveness telemetry** - Tracks `_last_rx_at` (bumped on every `RX_LOG_DATA` event) and `_consecutive_stats_failures` (incremented on `get_stats_*` / `get_bat` exceptions, cleared on success). Surfaced via `/health/strict` for the external watchdog - **TCP self-heal** - A `_liveness_watcher_loop` task on the DM event loop calls `force_reconnect()` when no RX event has arrived for `HEALTH_STRICT_MAX_RX_STALE_SEC` (5 min). `send_channel_message` also detects empty-string `concurrent.futures.TimeoutError` from `set_flood_scope_key` (the symptom of a degraded long-lived TCP) and runs an in-place reconnect + one retry before failing. A 30 s cooldown and `_reconnect_lock` prevent churn; `_intentional_disconnect` keeps the DISCONNECTED handler from racing the reconnect. The watcher keeps re-checking staleness even after `_connected` has gone False, so a single failed reconnect (e.g. an empty `self_info`) no longer silently stops all further healing for non-BLE transports - **Raw packet resend** - Own channel sends capture a full hex wire snapshot (`header + transport_codes + path_len + encrypted payload`) into `channel_messages.raw_packet`, rebuilt from the actual `pkt_payload` once echo correlation resolves it and honouring the device's cached `path_hash_mode`. `resend_channel_message()` re-broadcasts that snapshot verbatim via `CMD_SEND_RAW_PACKET` (0x41) so repeaters dedupe by packet hash (`Mesh::hasSeen`) and only previously-unreached nodes pick it up. Requires companion firmware ≥1.16 (`fw_ver_code` ≥ 13), gated via the cached `supports_raw_resend` flag captured from the connect-time `DEVICE_INFO` event. Self-echoes of a resend (the firmware seen-table can evict the hash within minutes on a busy mesh) are detected by recomputing the expected `pkt_payload` and matching an existing own row, so a resend never reappears as an inbound message from yourself ### Observer (MQTT packet capture) `app/observer.py` (`ObserverManager`) implements a meshcore-packet-capture-compatible observer on top of the existing device session — no second connection and no device-side enable command (the firmware pushes `RX_LOG_DATA` for every overheard frame): - **Hot path** — `_on_rx_log_data` hands every packet (all payload types, before the GRP_TXT echo gate) to `handle_raw_packet()`: pure CPU work (header/path decode, firmware-exact packet hash, JSON build) plus a non-blocking paho `publish()`, wrapped in its own try/except so observer failures can never affect chat - **One paho-mqtt client per enabled broker**, each with its own network thread (`loop_start`), auto-reconnect with backoff, QoS 0 (no backlog during outages), retained `online`/`offline` status plus an LWT on `meshcore[/IATA/PUBKEY]/status`; packets go to `.../packets` (flat `meshcore/packets|status` when IATA is unset) - **Wire-format fidelity** — the decode/hash/format functions are verbatim ports of upstream meshcore-packet-capture (including legacy path-length fallbacks and the publish-with-defaults quirk for unknown payload versions), validated byte-identical against live packets from the reference observer network - **Config** — `observer_brokers` table plus the `observer_settings` JSON key (`enabled`, `iata`, `advert_interval_hours`) in `app_settings`; every mutating API call triggers `reload()` on a daemon thread (the client list is swapped atomically and the hot path reads it lock-free), so changes apply without restart - **Advert scheduler** — an observer-owned daemon thread checks every 10 minutes and sends a flood advert once `advert_interval_hours` has elapsed since the `observer_last_advert_at` timestamp persisted in `app_settings` (restart-safe) - **Live status** — `observer_status` events on the `/chat` namespace (throttled to one per 2 s on the packet path) drive the Settings-tab badges and counters; `GET /api/observer/status` returns the full merged view ### Diagnostic capture (support bundle) `app/diagnostics.py` (`DiagnosticsManager`) records a bounded window of device traffic into a zip the user can hand to a maintainer who has no access to their machine. It answers a question the database cannot: whether a missing route badge means "nobody repeated it" or "the repeat never crossed the companion link". - **Why the device counter matters** — `logRxRaw()` in the firmware pushes every overheard packet through `writeFrame()`, which drops the frame when its 4-entry queue is full and ignores the return value, so the loss is invisible to the host (no log, no counter). Drain rates differ hugely: USB writes with no queue, WiFi/TCP sends one frame per main-loop pass, BLE at most one per 60 ms (`BLE_WRITE_MIN_INTERVAL`). The capture brackets its window with `get_device_stats()` calls and samples again every 60 s, so `stats.packets.recv` delta versus the number of RX-log frames actually captured turns that invisible loss into a measurement - **Hot path** — the same threading contract as the Observer, for the same reason. `record()` builds one dict and appends to a `deque`; no disk, no network, no locks. `threading.Event.set()` is deliberately avoided (it takes a lock), so the writer polls at 0.4 s instead of being signalled. Call sites test the plain `recording` attribute before building arguments, so an idle capture costs one attribute load - **Taps** — `_on_rx_log_data` (**before** the empty-payload guard, so the frame count matches what the device counted), `_process_echo` (match rule that fired, plus a snapshot of the pending-send list — that is what explains a *non*-match), and `send_channel_message` (expected payloads, `raw_packet`, region scope). A `logging.Handler` mirrors log records through the same queue; it uses a no-op lock object rather than `lock = None`, because Python 3.13's `Handler.handle()` uses `with self.lock` - **Log level** — a capture raises the root logger to DEBUG for its duration and restores it on stop. Without that, a server at the default INFO emits none of the lines the echo correlator writes about its own decisions, and the log file is worthless - **Writer thread** — one per session, owns every file handle, demultiplexes log records into `log.txt` and everything else into `events.jsonl`, enforces the caps, and performs the whole finalize-and-zip sequence. Stopping never runs on the caller's thread: a request thread raises a flag and joins, so both a user stop and an auto-stop finalise through the same path - **Caps and retention** — 5/15/30/60 min and 25 MB (hard limit 100 MB), whichever fills first; at most 10 captures kept, oldest pruned on start. `/data` is the user's config directory, so these are not optional - **Contents** — `meta.json` (versions, transport, device, options, counters, stop reason), `stats_before.json` / `stats_after.json`, `events.jsonl`, `log.txt`. Every timestamp is **UTC epoch seconds**, stated in `meta.json`, because the DB writes UTC and container logs are local time - **Privacy** — the capture contains the plaintext of every message received while recording, plus contact names and public keys. This is stated in the UI and confirmed before upload. Channel secrets and broker credentials are never included; ciphertext payloads are enough for correlation analysis - **Upload** — `POST /api/upload` against a Zipline instance: bare `authorization` token (no `Bearer`), one multipart `file` field, `x-zipline-original-name: true`, link read from `.files[0].url` (the older `files: ["url"]` shape is accepted too). The default URL ships in the code; **the token never does** — this repository is public, so a baked-in write token would let anyone upload anything. The user pastes a token handed to them out-of-band, and the API is write-only for it - **Offline analysis** — `scripts/diag_report.py ` prints the frame-loss comparison, per-interval detail, inter-frame gap distribution, and a per-sent-message verdict (echo heard / heard but not correlated, with the pending list / no echo at all). It reads nothing but the capture ### Repeater administration (My Repeaters) The `/repeaters` (list) and `/repeaters/manage` (per-repeater tools) panels are standalone iframe pages built on the companion protocol's remote-request commands plus the repeater text CLI: - **Serialization** — the companion firmware tracks a single pending remote request (each new send silently clears the previous one), so every repeater operation runs under `DeviceManager._repeater_lock` (180 s acquire timeout; a held lock returns `{'busy': True}` → HTTP 429). This also guards against two browser tabs operating at once - **Login sessions** — `repeater_login()` waits for `LOGIN_SUCCESS` filtered by the contact's `pubkey_prefix` and stores `{is_admin, permissions, logged_in_at}` in the in-memory `_repeater_sessions` dict (cleared on logout/app restart; the UI auto-relogs with the saved password). All remote endpoints fail fast with 401 `need_login` when no session exists — a repeater only answers pubkeys in its ACL, so requesting without a login would just burn a multi-minute timeout. A wrong password is indistinguishable from an unreachable repeater (the firmware never replies to failed logins), so timeout messages always name both causes - **CLI reply correlation** — `repeater_cmd_wait()` sends one text command and blocks for its reply. Replies arrive as `CONTACT_MSG_RECV` with `txt_type=1` (CLI_DATA) and carry no protocol-level correlation, so correlation = the repeater lock (single command in flight) + a single-slot waiter matched against the sender's 12-hex pubkey prefix at the top of `_on_dm_received`. Matched replies are consumed there and never stored as chat DMs; unmatched CLI replies keep the legacy behavior (stored as a DM) so the Console's fire-and-forget `cmd` flow is unchanged. Wait time derives from the device-suggested timeout (clamped 10–45 s) - **Settings batches** — reads run sequential `get ` commands per section and parse the firmware's `> value` replies, reporting per-field errors without failing the batch; writes send only dirty fields and classify each reply as `ok` (starts with `ok`/`password now`), `reboot_required` (reply mentions reboot — e.g. `set radio`), or `failed` (anything else, surfaced verbatim). Connection-level failures abort the rest of the batch - **Role gating** — the firmware silently drops text CLI from non-admin logins (which would surface as a timeout), so the CLI/Settings/Actions endpoints reject guest sessions with 403 up front instead - **Actions whitelist** — `advert.zerohop`, `advert` (flood), `clock sync`, `reboot`. `reboot` never replies (the firmware restarts immediately without building one), so a clean send followed by silence is reported as success. Text `erase` is firmware-gated to the USB serial console (`sender_timestamp == 0`), hence no erase in the UI ### Demo mode (MC_DEMO) `app/demo_guard.py` turns a publicly shared instance read-only. User-facing guide: [demo-mode.md](demo-mode.md). - **Default deny, not a blacklist** — an `@api_bp.before_request` hook refuses every non-GET on `/api/*` unless the path is on `WRITE_ALLOWLIST` (sending, resending, read markers, and the two unlock endpoints). An endpoint added later is therefore locked without anyone touching this module, which matters because six mutating endpoints already had no frontend caller and would never have made a hand-written blacklist. A short `READ_DENYLIST` covers the GETs that hand out secrets (repeater passwords, backup list/download, capture download, `/api/logs`) since the write guard never sees them - **Two bypasses of the HTTP layer, both closed separately** — the console is a Socket.IO event (`send_command` on `/console`), not a route, so it is guarded in `handle_send_command` against `CONSOLE_READONLY`; the log viewer is guarded at the `/logs` connect handshake by returning `False` - **Unlock is stateless** — the cookie holds `HMAC(MC_DEMO_UNLOCK_CODE, "mc-webui-demo-unlock-v1")`, compared with `hmac.compare_digest`. No session, and deliberately not Flask's `session`, whose `SECRET_KEY` is a hardcoded public constant. Changing the code invalidates every issued cookie - **Trusted networks refuse to match through a proxy** — when a request carries `X-Forwarded-For`/`X-Real-IP`/`Forwarded`/`CF-Connecting-IP` and `MC_TRUST_PROXY` is off, `remote_addr` belongs to the proxy or tunnel rather than the visitor, so it is not matched against `MC_DEMO_TRUSTED_NETS`. Without this, publishing through a LAN-hosted tunnel while trusting the LAN would unlock the instance for every visitor at once. `log_startup_state()` reports the resulting configuration at boot - **Frontend is advisory** — `demo-lock.js` (loaded from `_head_i18n.html`, so it reaches all eight entry points including the six iframes) disables controls inside `[data-demo-lock]`, removes `[data-demo-lock="hide"]`, re-applies through a `MutationObserver` for JS-rendered buttons, and turns any `403 demo_locked` response into a toast. The flags reach templates through the `inject_globals` context processor and JS through `/api/status` ### Path Analyzer The `/path-analyzer` panel (standalone iframe page, `path-analyzer.js`) is a read-only analysis view over data the app already collects — no new tables, no background work: - **Data source** — `GET /api/path-analyzer/messages?days=N` joins `channel_messages` with `echoes` (path hex + SNR + per-echo `hash_size`, keyed by `pkt_payload`). Echoes are fetched with `db.get_echoes_for_payloads()` — chunked `IN` queries (≤500 params, under SQLite's host-parameter limit) via `idx_echoes_pkt` — deliberately avoiding the per-message echo query the older `/api/messages` path still does. The pkt_payload reconstruction (raw_json text → channel-secret AES/HMAC compute) is shared with `/api/messages` via the `_get_row_pkt_payload()` helper - **All filtering/stats/map/routes logic is client-side** over the bulk payload (hundreds of KB for 7 days — fine on a LAN): filters operate on per-hop tokens split with each echo's own `hash_size` (mixed 1/2/3-byte networks are real), so SQL-side token filtering was rejected. Every view always reflects the active filters for free - **Four views** share the one payload: Messages (hop-by-hop echo detail), Repeaters (per-hash relay/SNR stats), Routes (consecutive hop-segment n-grams — user-selectable length 2–4, counted anywhere in a path), and Map (Leaflet path drawing). The repeater filter accepts a `>`-chained sequence (each element a hash prefix or contact name) matched as consecutive hops; Routes rows write such a sequence into that filter on click - **Deep link** — `GET /path-analyzer?hash=&path=` opens straight on the Map view with that message selected and that exact echo drawn. Used by the chat path popup (`app.js` → `openPathInAnalyzer` stashes `{hash, path}` in `window.paDeepLink`; the modal's `show.bs.modal` handler builds the iframe URL). On load the analyzer resolves the message by `packet_hash`, widening the time range once to 7 days if it isn't in the current window, and matches the echo by raw path hex (fallback: shortest routed echo) - **Map layers** — three `L.layerGroup`s added in draw order (base repeater markers → alternative echoes → selected path), both extras gated by opt-in checkboxes in a `topright` `L.Control` (the shared filter bar is wrong for view-specific state). Alternative echoes are coloured by their index in `echoView`, so a hue survives changing which echo is primary; segments are deduplicated against a `Set` of endpoint-pair keys that the primary path fills first, so alternatives render only where they diverge instead of underneath the primary line. Toggling calls `paRenderMapView(false)` — a full re-render (the sidebar swatches must follow) with `fitBounds` suppressed, so overlay changes never discard the user's viewport - **Filter persistence** — toolbar controls plus the Routes segment length are mirrored into `localStorage` under `mc-webui-pa-filters` (browser-local working set, deliberately not device state in SQLite). Only user-driven handlers write, via `paApplyAndSaveFilters()`, so programmatic changes — notably the deep link forcing 7 days — never overwrite the stored set; restore is skipped entirely when `paDeepLink` is present, and stored `