From 653d8d8646e79834c93a0b11df9ff290c1e21fcf Mon Sep 17 00:00:00 2001 From: MarekWo Date: Wed, 11 Mar 2026 08:07:22 +0100 Subject: [PATCH] docs: update public documentation to reflect v2 architecture and features --- README.md | 15 +-- docs/architecture.md | 275 +++++++++---------------------------------- docs/user-guide.md | 13 ++ docs/watchdog.md | 49 ++------ 4 files changed, 86 insertions(+), 266 deletions(-) diff --git a/README.md b/README.md index 79c48bb..c81ea4a 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,13 @@ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) # mc-webui -A lightweight web interface for meshcore-cli, providing browser-based access to MeshCore mesh network. +A lightweight web interface providing browser-based access to MeshCore mesh network. [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/MarekWo/mc-webui) ## Overview -**mc-webui** is a Flask-based web application that wraps `meshcore-cli`, eliminating the need for SSH/terminal access when using MeshCore chat on a LoRa device connected to a Debian VM via BLE or USB. Tested on Heltec V3 and Heltec V4. +**mc-webui** is a Flask-based web application that wraps `meshcore-cli`, eliminating the need for SSH/terminal access when using MeshCore chat on a LoRa device connected to a Debian VM via USB, BLE, or TCP. Tested on Heltec V3 and Heltec V4. ![Diagram](images/diagram.jpeg) @@ -17,7 +17,8 @@ A lightweight web interface for meshcore-cli, providing browser-based access to - **Channel management** - Create, join, share (QR code), and switch between encrypted channels - **Direct Messages (DM)** - Private messaging with delivery status tracking - **Smart notifications** - Unread message counters per channel with cross-device sync -- **Contact management** - Manual approval mode, filtering, protection, cleanup tools +- **Contact management** - Manual approval mode, filtering, protection, ignoring, blocking, and cleanup tools +- **Database** - Fast and reliable SQLite storage for messages, contacts, and configurations - **Contact map** - View contacts with GPS coordinates on OpenStreetMap (Leaflet) - **Message archives** - Automatic daily archiving with browse-by-date selector - **Interactive Console** - Direct meshcli command execution via WebSocket @@ -43,9 +44,9 @@ For detailed feature documentation, see the [User Guide](docs/user-guide.md). - Docker and Docker Compose installed ([installation guide](docs/docker-install.md)) **Important Notes:** -- No meshcore-cli installation required on host - automatically installed inside Docker container +- Powered by direct meshcore library integration (v2 architecture) - No manual directory setup needed - all data stored in `./data/` inside the project directory -- meshcore-cli version 1.3.12+ is automatically installed for proper DM functionality +- Uses a single-container architecture with a fast SQLite database --- @@ -100,7 +101,7 @@ For detailed feature documentation, see the [User Guide](docs/user-guide.md). - Download base images (Python, Alpine Linux) - Install meshcore-cli inside containers - Create `./data/` directory structure automatically - - Start both containers (meshcore-bridge and mc-webui) + - Start the mc-webui container 5. **Verify installation** ```bash @@ -307,7 +308,7 @@ sudo ~/mc-webui/scripts/updater/install.sh --uninstall - [x] @Mentions Autocomplete - Type @ to get contact suggestions with fuzzy search - [x] PWA Notifications (Experimental) - Browser notifications and app badge counters - [x] Full Offline Support - Local Bootstrap libraries and Service Worker caching -- [x] Interactive Console - Direct meshcli access via WebSocket with command history +- [x] Interactive Console - Direct MeshCore commands access via WebSocket with command history - [x] Contact Map - View contacts with GPS coordinates on OpenStreetMap (Leaflet) - [x] Echo Tracking - "Heard X repeats" badge for sent channel messages - [x] MeshCore Analyzer - Packet analysis links on channel messages (analyzer.letsmesh.net) diff --git a/docs/architecture.md b/docs/architecture.md index 42516c2..8fe2275 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -6,9 +6,9 @@ Technical documentation for mc-webui, covering system architecture, project stru - [Tech Stack](#tech-stack) - [Container Architecture](#container-architecture) -- [Bridge Session Architecture](#bridge-session-architecture) +- [DeviceManager Architecture](#devicemanager-architecture) - [Project Structure](#project-structure) -- [Message File Format](#message-file-format) +- [Database Architecture](#database-architecture) - [API Reference](#api-reference) - [Offline Support](#offline-support) @@ -16,159 +16,104 @@ Technical documentation for mc-webui, covering system architecture, project stru ## Tech Stack -- **Backend:** Python 3.11+, Flask, Flask-SocketIO (gevent) +- **Backend:** Python 3.11+, Flask, Flask-SocketIO (gevent), SQLite - **Frontend:** HTML5, Bootstrap 5, vanilla JavaScript, Socket.IO client -- **Deployment:** Docker / Docker Compose (2-container architecture) -- **Communication:** HTTP bridge to meshcore-cli, WebSocket for interactive console -- **Data source:** `~/.config/meshcore/.msgs` (JSON Lines) +- **Deployment:** Docker / Docker Compose (Single-container architecture) +- **Communication:** Direct hardware access (USB, BLE, or TCP) via `meshcore` library +- **Data source:** SQLite Database (`./data/meshcore/.db`) --- ## Container Architecture -mc-webui uses a **2-container architecture** for improved USB stability: +mc-webui uses a **single-container architecture** for simplified deployment and direct hardware communication: -``` +```text ┌─────────────────────────────────────────────────────────────┐ │ Docker Network │ │ │ -│ ┌─────────────────────┐ ┌─────────────────────────┐ │ -│ │ meshcore-bridge │ │ mc-webui │ │ -│ │ │ │ │ │ -│ │ - USB device access│ HTTP │ - Flask web app │ │ -│ │ - meshcli process │◄────►│ - User interface │ │ -│ │ - Port 5001 │ │ - Port 5000 │ │ -│ │ │ │ │ │ -│ └─────────┬───────────┘ └─────────────────────────┘ │ +│ ┌───────────────────────────────────────────────────────┐ │ +│ │ mc-webui │ │ +│ │ │ │ +│ │ - Flask web app (Port 5000) │ │ +│ │ - DeviceManager (Direct USB/TCP access) │ │ +│ │ - Database (SQLite) │ │ +│ │ │ │ +│ └─────────┬─────────────────────────────────────────────┘ │ │ │ │ └────────────┼─────────────────────────────────────────────────┘ │ ▼ ┌──────────────┐ - │ USB Device │ - │ (Heltec V4) │ + │ USB/TCP │ + │ Device │ └──────────────┘ ``` -### meshcore-bridge (Port 5001 - internal) - -Lightweight service with exclusive USB device access: - -- Maintains a **persistent meshcli session** (single long-lived process) -- Multiplexes stdout: JSON adverts → `.adverts.jsonl` log, CLI commands → HTTP responses -- Real-time message reception via `msgs_subscribe` (no polling) -- Thread-safe command queue with event-based synchronization -- Watchdog thread for automatic crash recovery -- Exposes HTTP API on port 5001 (internal only) - -### mc-webui (Port 5000 - external) - -Main web application: - -- Flask-based web interface with Flask-SocketIO -- Communicates with bridge via HTTP API -- WebSocket support for interactive Console (`/console` namespace) -- No direct USB access (prevents device locking) - -This separation solves USB timeout/deadlock issues common in Docker + VM environments. +This v2 architecture eliminates the need for a separate bridge container and relies on the native `meshcore` Python library for direct communication, ensuring lower latency and greater stability. --- -## Bridge Session Architecture +## DeviceManager Architecture -The meshcore-bridge maintains a **single persistent meshcli session** instead of spawning new processes per request: +The `DeviceManager` handles the connection to the MeshCore device via a direct session: -- **Single subprocess.Popen** - One long-lived meshcli process with stdin/stdout pipes -- **Multiplexing** - Intelligently routes output: - - JSON adverts (with `payload_typename: "ADVERT"`) → logged to `{device_name}.adverts.jsonl` - - CLI command responses → returned via HTTP API -- **Real-time messages** - `msgs_subscribe` command enables instant message reception without polling -- **Thread-safe queue** - Commands are serialized through a queue.Queue for FIFO execution -- **Timeout-based detection** - Response completion detected when no new lines arrive for 300ms -- **Auto-restart watchdog** - Monitors process health and restarts on crash - -This architecture enables advanced features like pending contact management (`manual_add_contacts`) and provides better stability and performance. +- **Single persistent session** - One long-lived connection utilizing the `meshcore` library +- **Event-driven** - Subscribes to device events (e.g., incoming messages, advert receptions, ACKs) and triggers appropriate handlers +- **Direct Database integration** - Seamlessly syncs contacts, messages, and device settings to the SQLite database +- **Real-time messages** - Instant message processing via callback events without polling +- **Thread-safe queue** - Commands are serialized to prevent device lockups +- **Auto-restart watchdog** - Monitors connection health and restarts the session on crash --- ## Project Structure -``` +```text mc-webui/ ├── Dockerfile # Main app Docker image -├── docker-compose.yml # Multi-container orchestration -├── meshcore-bridge/ -│ ├── Dockerfile # Bridge service image -│ ├── bridge.py # HTTP API wrapper for meshcli -│ └── requirements.txt # Bridge dependencies (Flask only) +├── docker-compose.yml # Single-container orchestration ├── app/ │ ├── __init__.py -│ ├── main.py # Flask entry point +│ ├── main.py # Flask entry point + Socket.IO │ ├── config.py # Configuration from env vars +│ ├── database.py # SQLite database models and CRUD operations +│ ├── device_manager.py # Core logic for meshcore communication │ ├── read_status.py # Server-side read status manager +│ ├── migrate_v1.py # Migration script from v1 flat files to v2 SQLite │ ├── meshcore/ │ │ ├── __init__.py -│ │ ├── cli.py # HTTP client for bridge API -│ │ └── parser.py # .msgs file parser +│ │ ├── cli.py # Meshcore library wrapper interface +│ │ └── parser.py # Data parsers │ ├── archiver/ │ │ └── manager.py # Archive scheduler and management │ ├── routes/ │ │ ├── __init__.py │ │ ├── api.py # REST API endpoints │ │ └── views.py # HTML views -│ ├── static/ -│ │ ├── css/ -│ │ │ └── style.css # Custom styles -│ │ ├── js/ -│ │ │ ├── app.js # Main page frontend logic -│ │ │ ├── dm.js # Direct Messages page logic -│ │ │ ├── contacts.js # Contact Management logic -│ │ │ ├── console.js # Interactive console WebSocket client -│ │ │ ├── message-utils.js # Message content processing -│ │ │ └── sw.js # Service Worker for PWA -│ │ ├── vendor/ # Local vendor libraries (offline) -│ │ │ ├── bootstrap/ # Bootstrap CSS/JS -│ │ │ ├── bootstrap-icons/ # Icon fonts -│ │ │ ├── socket.io/ # Socket.IO client library -│ │ │ └── emoji-picker-element/ -│ │ └── manifest.json # PWA manifest -│ └── templates/ -│ ├── base.html # Base template -│ ├── index.html # Main chat view -│ ├── dm.html # Direct Messages view -│ ├── console.html # Interactive meshcli console -│ ├── contacts_base.html # Contact pages base template -│ ├── contacts-manage.html # Contact Management settings -│ ├── contacts-pending.html # Pending contacts view -│ └── contacts-existing.html # Existing contacts view +│ ├── static/ # Frontend assets (CSS, JS, images, vendors) +│ └── templates/ # HTML templates ├── docs/ # Documentation -├── images/ # Screenshots and diagrams -├── requirements.txt # Python dependencies -├── .env.example # Example environment config +├── scripts/ # Utility scripts (update, watchdog, etc.) └── README.md ``` --- -## Message File Format +## Database Architecture -Location: `~/.config/meshcore/.msgs` (JSON Lines) +mc-webui v2 uses a robust **SQLite Database** with WAL (Write-Ahead Logging) enabled. -### Message Types +Location: `./data/meshcore/.db` -**Channel messages:** -```json -{"type": "CHAN", "text": "User: message", "timestamp": 1766300846} -{"type": "SENT_CHAN", "text": "my message", "name": "DeviceName", "timestamp": 1766309432} -``` +Key tables: +- `messages` - All channel and direct messages +- `contacts` - Contact list with sync status, types, block/ignore flags +- `channels` - Channel configuration and keys +- `echoes` - Sent message tracking and repeater paths +- `acks` - DM delivery status -**Private messages:** -```json -{"type": "PRIV", "text": "message", "sender_timestamp": 1766300846, "pubkey_prefix": "abc123", "sender": "User"} -{"type": "SENT_MSG", "text": "message", "recipient": "User", "expected_ack": "xyz", "suggested_timeout": 30000} -``` - -**Note on SENT_MSG:** Requires meshcore-cli >= 1.3.12 for correct format with both `recipient` and `sender` fields. +The use of SQLite allows for fast queries, reliable data storage, and complex filtering (such as contact ignoring/blocking) without the risk of file corruption inherent to flat JSON files. --- @@ -183,141 +128,27 @@ Location: `~/.config/meshcore/.msgs` (JSON Lines) | GET | `/api/messages/updates` | Check for new messages (smart refresh) | | GET | `/api/status` | Connection status | | GET | `/api/contacts` | List contacts | -| GET | `/api/contacts/detailed` | Full contact_info data | -| POST | `/api/contacts/delete` | Delete contact by name | -| GET | `/api/contacts/pending` | List pending contacts | -| POST | `/api/contacts/pending/approve` | Approve pending contact | -| POST | `/api/contacts/preview-cleanup` | Preview cleanup matches | -| POST | `/api/contacts/cleanup` | Execute contact cleanup | +| GET | `/api/contacts/detailed` | Full contact data | +| POST | `/api/contacts/delete` | Soft-delete contact by name | +| POST | `/api/contacts/update` | Update contact properties (ignore, block) | | GET | `/api/channels` | List all channels | | POST | `/api/channels` | Create new channel | | POST | `/api/channels/join` | Join existing channel | | DELETE | `/api/channels/` | Remove channel | -| GET | `/api/channels//qr` | Generate QR code | | GET | `/api/dm/conversations` | List DM conversations | | GET | `/api/dm/messages` | Get messages for conversation | | POST | `/api/dm/messages` | Send DM | -| GET | `/api/dm/updates` | Check for new DMs | | GET | `/api/device/info` | Device information | -| GET | `/api/device/settings` | Get device settings | -| POST | `/api/device/settings` | Update device settings | -| POST | `/api/device/command` | Execute special command | | GET | `/api/read_status` | Get server-side read status | -| POST | `/api/read_status/mark_read` | Mark messages as read | -| GET | `/api/archives` | List available archives | -| POST | `/api/archive/trigger` | Manually trigger archiving | ### WebSocket API (Console) -Interactive meshcli console via Socket.IO WebSocket connection. +Interactive console via Socket.IO WebSocket connection. **Namespace:** `/console` -| Event | Direction | Description | -|-------|-----------|-------------| -| `send_command` | Client → Server | Execute command (`{command: "infos"}`) | -| `console_status` | Server → Client | Connection status message | -| `command_response` | Server → Client | Command result (`{success, command, output}`) | - -**Features:** -- Command history navigation (up/down arrows) -- Auto-reconnection on disconnect -- Output cleaning (removes prompts like "DeviceName|*") -- Slow command timeouts: `node_discover` (15s), `recv` (60s), `send` (15s) - -### Bridge Internal API (Port 5001) - -| Method | Endpoint | Description | -|--------|----------|-------------| -| POST | `/cli` | Execute meshcli command (`{args: ["cmd"], timeout?}`) | -| GET | `/health` | Bridge health check | -| GET | `/pending_contacts` | List pending contacts | -| POST | `/add_pending` | Approve pending contact | -| GET | `/device/settings` | Get device settings | -| POST | `/device/settings` | Update device settings | - -**Base URL:** `http://meshcore-bridge:5001` (internal Docker network only) - --- ## Offline Support -The application works completely offline without internet connection - perfect for mesh networks in remote or emergency scenarios. - -### Local Vendor Libraries - -| Library | Size | Location | -|---------|------|----------| -| Bootstrap 5.3.2 CSS | ~227 KB | `static/vendor/bootstrap/css/` | -| Bootstrap 5.3.2 JS | ~80 KB | `static/vendor/bootstrap/js/` | -| Bootstrap Icons 1.11.2 | ~398 KB | `static/vendor/bootstrap-icons/` | -| Emoji Picker Element | ~529 KB | `static/vendor/emoji-picker-element/` | - -**Total offline package size:** ~1.2 MB - -### Service Worker Caching - -- **Cache version:** `mc-webui-v3` -- **Strategy:** Hybrid caching - - **Cache-first** for vendor libraries (static, unchanging) - - **Network-first** for app code (dynamic, needs updates) - -### How It Works - -1. On first visit (online), Service Worker installs and caches all assets -2. Vendor libraries (Bootstrap, Icons, Emoji Picker) loaded from cache instantly -3. App code checks network first, falls back to cache if offline -4. Complete UI functionality available offline -5. Only API calls (messages, channels, contacts) require connectivity - ---- - -## Environment Variables - -| Variable | Description | Default | -|----------|-------------|---------| -| `MC_SERIAL_PORT` | Serial device path | - | -| `MC_DEVICE_NAME` | Device name (for files) | - | -| `MC_CONFIG_DIR` | Configuration directory | `./data/meshcore` | -| `MC_ARCHIVE_DIR` | Archive directory | `./data/archive` | -| `MC_ARCHIVE_ENABLED` | Enable automatic archiving | `true` | -| `MC_ARCHIVE_RETENTION_DAYS` | Days to show in live view | `7` | -| `FLASK_HOST` | Listen address | `0.0.0.0` | -| `FLASK_PORT` | Web server port | `5000` | -| `FLASK_DEBUG` | Debug mode | `false` | -| `TZ` | Timezone for logs | `UTC` | - ---- - -## Persistent Settings - -### Settings File - -**Location:** `MC_CONFIG_DIR/.webui_settings.json` - -```json -{ - "manual_add_contacts": false -} -``` - -### Read Status File - -**Location:** `MC_CONFIG_DIR/.read_status.json` - -Stores per-channel and per-conversation read timestamps for cross-device synchronization. - -```json -{ - "channels": {"0": 1735900000, "1": 1735900100}, - "dm": {"name_User1": 1735900200} -} -``` - ---- - -## Related Documentation - -- [User Guide](user-guide.md) - How to use all features -- [Troubleshooting](troubleshooting.md) - Common issues and solutions -- [Docker Installation](docker-install.md) - How to install Docker +The application works completely offline without internet connection. Vendor libraries (Bootstrap, Bootstrap Icons, Socket.IO, Emoji Picker) are bundled locally. A Service Worker provides hybrid caching to ensure functionality without connectivity. diff --git a/docs/user-guide.md b/docs/user-guide.md index aaf1f78..abe8c67 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -257,6 +257,12 @@ The Existing Contacts section displays all contacts currently stored on your dev 3. **Copy public key:** Click "Copy Key" button to copy the public key prefix to clipboard 4. **Delete a contact:** Click the "Delete" button (red trash icon) and confirm +**Ignoring and Blocking Contacts:** +- **Ignore**: The contact is hidden from the main view and their messages do not trigger notifications. +- **Block**: The contact is completely blocked. Their messages are dropped and will not appear anywhere. + +To ignore or block a contact, click the "Ignore" or "Block" button on their contact card. To restore them, switch the type filter to "Ignored" or "Blocked" and click the "Restore" button. + **Contact capacity monitoring:** - MeshCore devices have a limit of 350 contacts - The counter badge changes color as you approach the limit: @@ -264,6 +270,13 @@ The Existing Contacts section displays all contacts currently stored on your dev - **300-339**: Yellow warning (nearing limit) - **340-350**: Red alarm (critical - delete some contacts soon) +### Contact Map + +Access the map from the main menu to view the GPS locations of your contacts. +- Contacts with known GPS coordinates will be displayed as markers on OpenStreetMap. +- Click a marker to see the contact name and details. +- Use the **Cached** switch to toggle the display of cache-only contacts (contacts that are saved in your database but no longer present in the device's internal memory). + ### Contact Cleanup Tool The advanced cleanup tool allows you to filter and remove contacts based on multiple criteria: diff --git a/docs/watchdog.md b/docs/watchdog.md index 2bd5e42..d779101 100644 --- a/docs/watchdog.md +++ b/docs/watchdog.md @@ -1,15 +1,16 @@ # Container Watchdog -The Container Watchdog is a systemd service that monitors Docker containers and automatically restarts unhealthy or stopped ones. This is useful for ensuring reliability, especially on resource-constrained systems. +The Container Watchdog is a systemd service that monitors the `mc-webui` Docker container and automatically restarts it if it becomes unhealthy or if the LoRa device becomes unresponsive. This is useful for ensuring reliability, especially on resource-constrained systems or when the LoRa hardware hangs. ## Features - **Health monitoring** - Checks container status every 30 seconds -- **Automatic restart** - Restarts containers that become unhealthy -- **Auto-start stopped containers** - Starts containers that have stopped (configurable) -- **Hardware USB reset** - Performs a low-level USB bus reset if the LoRa device freezes (detected after 3 failed container restarts within 8 minutes) +- **Log monitoring** - Monitors `mc-webui` logs for specific "unresponsive LoRa device" errors +- **Automatic restart** - Restarts the container when issues are detected +- **Auto-start stopped container** - Starts the container if it has stopped (configurable) +- **Hardware USB reset** - Performs a low-level USB bus reset (unbind/bind or DTR/RTS) if the LoRa device freezes. *Note: USB reset is automatically skipped if a TCP connection is used.* - **Diagnostic logging** - Captures container logs before restart for troubleshooting -- **HTTP status endpoint** - Query container status via HTTP API +- **HTTP status endpoint** - Query watchdog status via HTTP API - **Restart history** - Tracks all automatic restarts with timestamps ## Installation @@ -21,9 +22,9 @@ sudo ./scripts/watchdog/install.sh The installer will: - Create a systemd service `mc-webui-watchdog` -- Start monitoring containers immediately +- Start monitoring the container immediately - Enable automatic startup on boot -- Create log file at `/var/log/mc-webui-watchdog.log` +- Create a log file at `/var/log/mc-webui-watchdog.log` ## Usage @@ -60,9 +61,9 @@ curl http://localhost:5051/history ### Diagnostic Files -When a container is restarted, diagnostic information is saved to: +When the container is restarted, diagnostic information is saved to: ``` -/tmp/mc-webui-watchdog-{container}-{timestamp}.log +/tmp/mc-webui-watchdog-mc-webui-{timestamp}.log ``` These files contain: @@ -82,8 +83,8 @@ If you need to customize the behavior, the service supports these environment va | `CHECK_INTERVAL` | `30` | Seconds between health checks | | `LOG_FILE` | `/var/log/mc-webui-watchdog.log` | Path to log file | | `HTTP_PORT` | `5051` | HTTP status port (0 to disable) | -| `AUTO_START` | `true` | Start stopped containers (set to `false` to disable) | -| `USB_DEVICE_PATH` | *(auto-detected)* | Path to the LoRa device (e.g., `/dev/bus/usb/001/002`) for hardware USB bus reset | +| `AUTO_START` | `true` | Start stopped container (set to `false` to disable) | +| `USB_DEVICE_PATH` | *(auto-detected)* | Path to the LoRa device for hardware USB bus reset | To modify defaults, create an override file: ```bash @@ -107,29 +108,3 @@ Note: The log file is preserved after uninstall. Remove manually if needed: ```bash sudo rm /var/log/mc-webui-watchdog.log ``` - -## Troubleshooting - -### Service won't start - -Check the logs: -```bash -journalctl -u mc-webui-watchdog -n 50 -``` - -Common issues: -- Docker not running -- Python 3 not installed -- Permission issues - -### Containers keep restarting - -Check the diagnostic files in `/tmp/mc-webui-watchdog-*.log` to see what's causing the containers to become unhealthy. - -### HTTP endpoint not responding - -Verify the service is running and check if port 5051 is available: -```bash -systemctl status mc-webui-watchdog -ss -tlnp | grep 5051 -```