meshcore-gui/OBSERVER.md

MeshCore Observer — Read-Only Archive Monitor

Multi-source aggregation dashboard with optional MQTT uplink to LetsMesh.

A standalone daemon that reads JSON archive files produced by meshcore_gui and meshcore_bridge, aggregates them from all sources, and presents a unified live dashboard. It never connects to a device and never writes to the archive — it only watches and displays.

Table of Contents

• 1. Why This Project Exists
• 2. Features
• 3. Requirements
• 4. Installation
  • 4.1. Base Installation (dashboard only)
  • 4.2. MQTT Uplink Dependencies
  • 4.3. Verify Installation
• 5. Quick Start
• 6. Command-Line Options
• 7. Configuration
  • 7.1. Observer Settings
  • 7.2. Port Allocation
• 8. MQTT Uplink to LetsMesh
  • 8.1. Prerequisites
  • 8.2. Automatic Key Setup (same machine)
  • 8.3. Manual Key Setup (remote GUI)
  • 8.4. MQTT Configuration
  • 8.5. Test and Go Live
  • 8.6. Privacy Controls
  • 8.7. Multiple Brokers
  • 8.8. How Authentication Works
  • 8.9. MQTT Topics
• 9. systemd Installation
• 10. How It Works
• 11. Dashboard Panels
• 12. Running Alongside Other Daemons
• 13. Troubleshooting
  • 13.1. Dashboard
  • 13.2. MQTT
• 14. Version History
• 15. License
• 16. Author

---

1. Why This Project Exists

When running multiple MeshCore devices — a GUI instance on 869 MHz, a bridge between 869 and 868 MHz, perhaps another GUI on a different frequency — each writes its own archive files. There is no single place to see all traffic at once.

The Observer solves this by watching all archive files from all sources, merging them into one live dashboard. It requires no device, no serial port and no meshcore library. Just point it at the archive directory and it works.

With MQTT uplink enabled, the Observer can also contribute your node's received packets to the global LetsMesh analyzer, helping map the mesh network's reach and signal quality.

[meshcore_gui]  ──writes──►  ~/.meshcore-gui/archive/*.json  ◄──reads──  [Observer]
[meshcore_bridge] ──writes──►                                                 │
                                                                         ┌────┴────┐
                                                                         ▼         ▼
                                                                   NiceGUI    MQTT Uplink
                                                                   Dashboard  (optional)
                                                                   :9093      │
                                                                              ▼
                                                                    analyzer.letsmesh.net

2. Features

• Multi-source aggregation — Automatically detects and merges archives from all GUI and Bridge instances
• Live message feed — Channel messages from all sources, sorted by timestamp, filterable by source and channel
• Live RX log — Packet log with SNR, RSSI, type, hops, and decoded path
• Source overview — Table of all detected archive files with entry counts
• Statistics — Uptime, totals, per-source breakdown
• MQTT uplink to LetsMesh — Publishes RX log packets to analyzer.letsmesh.net via MQTT over WebSocket+TLS with Ed25519 JWT authentication. Privacy-configurable: choose which packet types to share
• DOMCA theme — Dark and light mode, consistent with meshcore_gui and meshcore_bridge
• Zero device access — No serial port, no BLE, no meshcore library required

3. Requirements

Dashboard (always required):

• Python 3.10+
• nicegui
• pyyaml

MQTT uplink (optional):

• paho-mqtt >= 2.0
• Node.js 18+ with @michaelhart/meshcore-decoder — required for signing JWT tokens with the orlp/ed25519 algorithm used by LetsMesh

Note on PyNaCl: PyNaCl can be used as a fallback for token signing, but only with legacy 64-char seed keys. The current device_identity.json format uses 128-char orlp/ed25519 expanded keys which are not compatible with PyNaCl. For new installations, use Node.js + meshcore-decoder.

Without the MQTT packages the Observer runs fine — only the LetsMesh uplink is disabled.

---

4. Installation

4.1. Base Installation (dashboard only)

cd ~/meshcore-gui
source venv/bin/activate

pip install nicegui pyyaml

Verify:

python meshcore_observer.py --help

4.2. MQTT Uplink Dependencies

Step 1 — paho-mqtt:

pip install paho-mqtt

Step 2 — Node.js (if not already installed):

# Check if Node.js is available
node --version

# If not installed (Debian/Ubuntu/Raspberry Pi OS):
sudo apt update && sudo apt install -y nodejs npm

Step 3 — meshcore-decoder:

sudo npm install -g @michaelhart/meshcore-decoder

Step 4 — Verify Node.js can find meshcore-decoder:

node -e "require('@michaelhart/meshcore-decoder').createAuthToken; console.log('OK')"

If this prints OK, you're good. If it says Cannot find module, Node.js can't find the global install. Fix with:

# Check where npm installs global packages:
npm root -g

# If it's /usr/local/lib/node_modules, set NODE_PATH:
export NODE_PATH=/usr/local/lib/node_modules
node -e "require('@michaelhart/meshcore-decoder').createAuthToken; console.log('OK')"

If you need NODE_PATH, add it to your shell profile or systemd service (see section 9).

Note: The Observer auto-detects NODE_PATH from /usr/lib/node_modules, /usr/local/lib/node_modules, and ~/.npm-global/lib/node_modules. You only need to set it manually if npm uses a non-standard location.

4.3. Verify Installation

cd ~/meshcore-gui
source venv/bin/activate

# Dashboard only:
python -c "import nicegui, yaml; print('Dashboard deps OK')"

# MQTT uplink:
python -c "import paho.mqtt; print('paho-mqtt OK')"
node -e "require('@michaelhart/meshcore-decoder'); console.log('meshcore-decoder OK')"

---

5. Quick Start

cd ~/meshcore-gui
source venv/bin/activate

# Run with defaults (dashboard on port 9093, reads ~/.meshcore-gui/archive/)
python meshcore_observer.py

The dashboard opens at http://localhost:9093. The Observer immediately starts scanning for archive JSON files. If meshcore_gui or meshcore_bridge is running and writing archives, they will appear within seconds.

---

6. Command-Line Options

Flag	Description	Default
--config=PATH	Path to YAML configuration file	./observer_config.yaml
--port=PORT	Override dashboard port	9093
--debug-on	Enable verbose debug logging	Off
--mqtt-dry-run	Log MQTT payloads without publishing (also enables MQTT)	Off
--help	Show usage information	—

Examples:

python meshcore_observer.py
python meshcore_observer.py --config=/etc/meshcore/observer_config.yaml
python meshcore_observer.py --port=9094 --debug-on
python meshcore_observer.py --mqtt-dry-run --debug-on

---

7. Configuration

All settings are optional. The Observer works with sensible defaults and no config file.

7.1. Observer Settings

observer_config.yaml:

observer:
  archive_dir: "~/.meshcore-gui/archive"
  poll_interval_s: 2.0
  max_messages_display: 100
  max_rxlog_display: 50

gui:
  port: 9093
  title: "MeshCore Observer"

7.2. Port Allocation

Daemon	Default Port
meshcore_gui	8081 / 9090
meshcore_bridge	9092
meshcore_observer	9093

---

8. MQTT Uplink to LetsMesh

The Observer can publish RX log packets to the LetsMesh network analyzer at analyzer.letsmesh.net. MQTT is disabled by default.

[Observer]
    │
    │ RX log entries from archive
    │
    ├── filter by packet type (privacy)
    ├── transform to LetsMesh JSON format
    ├── sign JWT with Ed25519 private key
    │
    └──► mqtt-eu-v1.letsmesh.net:443 (WebSocket+TLS)
              │
              ▼
         analyzer.letsmesh.net

8.1. Prerequisites

Before enabling MQTT, ensure:

1. MQTT dependencies are installed (see section 4.2)

2. meshcore_gui has the fixed device_identity.py — version ≥ 1.2.0 that writes the full 128-char orlp/ed25519 private key. Without this fix, device_identity.json contains only a 64-char key that is not the Ed25519 seed but a clamped scalar, which causes "Not authorized" errors.

   Verify your identity file:

   cat ~/.meshcore-gui/device_identity.json | python -c "
   import json, sys
   d = json.load(sys.stdin)
   pub = d.get('public_key', '')
   priv = d.get('private_key', '')
   print(f'Public key:  {len(pub)} chars — {pub[:16]}...')
   print(f'Private key: {len(priv)} chars')
   if len(priv) == 128:
       print('✅ Correct format (128-char orlp expanded key)')
   elif len(priv) == 64:
       print('❌ Legacy format (64 chars) — update device_identity.py and restart meshcore_gui')
   else:
       print(f'❌ Unexpected length: {len(priv)}')
   "
   

3. Your IATA airport code — 3-letter code for your nearest airport (e.g. AMS, JFK, LHR)

8.2. Automatic Key Setup (same machine)

When meshcore_gui and the Observer run on the same machine, keys are shared automatically via ~/.meshcore-gui/device_identity.json. No manual key configuration needed — just add the MQTT section to your config and the Observer reads the keys at startup.

8.3. Manual Key Setup (remote GUI)

When meshcore_gui runs on a different machine, you need to transfer the keys.

Option A — Copy the identity file (recommended):

Copy ~/.meshcore-gui/device_identity.json from the GUI machine to the Observer machine, then:

cd ~/meshcore-gui
source venv/bin/activate
python -m meshcore_observer.setup_mqtt_keys --identity /path/to/device_identity.json

This saves the private key to ~/.meshcore-observer-key (chmod 600) and writes the public key to observer_config.yaml.

Option B — Interactive setup:

cd ~/meshcore-gui
source venv/bin/activate
python -m meshcore_observer.setup_mqtt_keys

You will need:

• The public key (64 hex chars) — visible in meshcore_gui device info
• The private key (128 hex chars) — from device_identity.json on the GUI machine

Option C — Environment variable:

export MESHCORE_PRIVATE_KEY="<128-char hex private key>"
export MESHCORE_PUBLIC_KEY="<64-char hex public key>"

8.4. MQTT Configuration

Edit observer_config.yaml:

mqtt:
  enabled: true
  iata: "AMS"                          # Your nearest airport code
  device_name: "PE1HVH Observer"       # Name shown on analyzer.letsmesh.net

  # Keys — only needed if meshcore_gui runs on a different machine.
  # On the same machine, keys are read from device_identity.json automatically.
  # public_key: "D955E72C..."
  # private_key_file: "~/.meshcore-observer-key"

  brokers:
    - name: "letsmesh-eu"
      server: "mqtt-eu-v1.letsmesh.net"
      port: 443
      transport: "websockets"
      tls: true
      enabled: true

  upload_packet_types: []               # [] = all types
  status_interval_s: 300
  reconnect_delay_s: 10

8.5. Test and Go Live

Step 1 — Dry run (log payloads without publishing):

python meshcore_observer.py --mqtt-dry-run --debug-on

Check the output for:

• "Loaded device identity from ..." — keys found
• "Using Node.js meshcore-decoder for MQTT auth tokens" — signing works
• "[DRY RUN] Would connect to ..." — broker config OK

Step 2 — Live:

python meshcore_observer.py

The dashboard MQTT panel shows connection status, packet counters, and any errors. Within minutes your packets should appear on analyzer.letsmesh.net.

8.6. Privacy Controls

Control which packet types are shared:

# Only advertisements (network discovery, no message content)
upload_packet_types: [4]

# Advertisements and group text metadata
upload_packet_types: [4, 5]

# Everything (default)
upload_packet_types: []

Packet types: 0=REQ, 1=RESPONSE, 2=TXT_MSG, 3=ACK, 4=ADVERT, 5=GRP_TXT, 6=GRP_DATA, 7=ANON_REQ, 8=PATH, 9=TRACE.

The raw packet payload (hex bytes) is always included for shared types. If you do not want to share message content, use [4] (ADVERT only).

8.7. Multiple Brokers

Publish to EU and US brokers simultaneously for redundancy:

brokers:
  - name: "letsmesh-eu"
    server: "mqtt-eu-v1.letsmesh.net"
    port: 443
    transport: "websockets"
    tls: true
    enabled: true

  - name: "letsmesh-us"
    server: "mqtt-us-v1.letsmesh.net"
    port: 443
    transport: "websockets"
    tls: true
    enabled: true

8.8. How Authentication Works

LetsMesh uses Ed25519 JWT tokens — no registration required. Your device key pair is your identity:

1. meshcore_gui exports the device's Ed25519 keypair to device_identity.json
2. The Observer generates a JWT signed with the 128-char orlp/ed25519 private key via Node.js meshcore-decoder
3. MQTT username: v1_{PUBLIC_KEY} (uppercase, 64 hex chars)
4. MQTT password: the signed JWT token
5. The broker verifies the signature against the public key from the username
6. Topics are scoped to meshcore/{IATA}/{PUBLIC_KEY}/
7. Tokens auto-refresh before expiry (default 1 hour)

Key format: The private key in device_identity.json is 128 hex chars (64 bytes) in orlp/ed25519 expanded format: [clamped_scalar(32)][nonce_prefix(32)]. This is not a seed+pubkey concatenation — it is the output of SHA-512(seed) with clamping. The public key comes from send_appstart() and is stored separately.

8.9. MQTT Topics

Topic	Content	QoS	Retained
meshcore/{IATA}/{KEY}/packets	RX log entries (JSON)	0	No
meshcore/{IATA}/{KEY}/status	Online/offline status	1	Yes

The status topic uses MQTT Last Will and Testament (LWT): if the Observer disconnects unexpectedly, the broker automatically publishes an offline status.

---

9. systemd Installation

For running the Observer as a background service on Linux.

Install:

cd ~/meshcore-gui
source venv/bin/activate

bash install_observer.sh

The installer detects the venv and current user automatically, creates a systemd service, and offers to start it immediately.

If you need a custom NODE_PATH (see section 4.2), edit the service file after installation:

sudo systemctl edit meshcore-observer

Add:

[Service]
Environment="NODE_PATH=/usr/local/lib/node_modules"

Then reload:

sudo systemctl daemon-reload
sudo systemctl restart meshcore-observer

Service commands:

Command	Description
sudo systemctl start meshcore-observer	Start the service
sudo systemctl stop meshcore-observer	Stop the service
sudo systemctl restart meshcore-observer	Restart after config change
sudo systemctl status meshcore-observer	Check status
sudo journalctl -u meshcore-observer -f	Follow live logs

Uninstall:

cd ~/meshcore-gui
bash install_observer.sh --uninstall

---

10. How It Works

The Observer uses a polling-based file watcher (ArchiveWatcher) that:

1. Scans the archive directory for *_messages.json and *_rxlog.json files
2. Checks each file's mtime (modification timestamp)
3. If unchanged since last poll → skip (no disk I/O)
4. If changed → read, parse, extract only new entries (delta detection)
5. Feeds new entries to the dashboard panels (and optionally to MQTT uplink)

This is efficient and safe:

• No file locking conflicts — meshcore_gui uses atomic writes (temp file + rename)
• No race conditions — Observer only reads completed files
• No crash on corruption — Malformed JSON is logged and skipped
• No crash on missing files — Vanished files are removed from tracking

---

11. Dashboard Panels

Sources

Table of all detected archive files with source name, message count, and RX log count.

Messages

Aggregated message feed from all sources. Newest messages on top. Filterable by source and channel.

RX Log

Aggregated packet log from all sources. Columns: Time, Source, SNR, RSSI, Type, Hops, Path, Hash.

Statistics

Observer uptime, total messages and RX log entries seen, number of active sources, per-source breakdown.

MQTT Uplink

Connection status per broker (green/red dot), total packets published, filtered count, skipped count, last publish timestamp, and any errors.

---

12. Running Alongside Other Daemons

The Observer is designed to coexist with meshcore_gui and meshcore_bridge:

┌──────────────────┐     ┌──────────────────┐
│  meshcore_gui    │     │ meshcore_bridge   │
│  :8081           │     │  :9092            │
│  writes archive  │     │  writes archive   │
└────────┬─────────┘     └────────┬──────────┘
         │                        │
         ▼                        ▼
    ~/.meshcore-gui/archive/
         │
         ▼
┌──────────────────┐
│ meshcore_observer │
│  :9093            │──────► mqtt-eu-v1.letsmesh.net
│  reads archive    │        (optional MQTT uplink)
└──────────────────┘

All three can run simultaneously. The Observer only reads atomically-written files and never interferes with the other daemons.

---

13. Troubleshooting

13.1. Dashboard

"Waiting for archive files..."

• Verify meshcore_gui or meshcore_bridge is running and has received at least one message
• Check: ls ~/.meshcore-gui/archive/
• If using a custom path, verify archive_dir in your config

No messages despite archive files existing

• Check file permissions
• Run with --debug-on
• Verify archive files have "version": 1 in their JSON

Port conflict

• Change with --port=9094 or in observer_config.yaml

13.2. MQTT

"Not authorized" (rc=5)

This is almost always a key format issue. Check in order:

1. Is the private key 128 chars?

   python -c "
   import json
   d = json.load(open('$HOME/.meshcore-gui/device_identity.json'))
   print(f\"private_key length: {len(d.get('private_key',''))}\")" 
   

   If 64: you need the fixed device_identity.py in meshcore_gui. Deploy it and restart meshcore_gui.

2. Does the public key match the GUI? The public_key in device_identity.json must match what meshcore_gui shows under device info (uppercase hex). If it doesn't, the identity file was written by a buggy version — deploy the fix and restart.

3. Is meshcore-decoder available?

   node -e "require('@michaelhart/meshcore-decoder').createAuthToken; console.log('OK')"
   

   If this fails, see section 4.2.

4. Run with debug:

   python meshcore_observer.py --mqtt-dry-run --debug-on
   

   Look for "Node.js token generation failed" or "PyNaCl" fallback messages.

"Connecting..." but never connects

• Check firewall allows outbound connections to port 443
• Try the US broker: change server to mqtt-us-v1.letsmesh.net
• Check DNS resolution: nslookup mqtt-eu-v1.letsmesh.net

Packets not appearing on analyzer.letsmesh.net

• Use --mqtt-dry-run to verify payload format
• Check upload_packet_types is not filtering everything
• Verify archive files contain raw_payload data
• The analyzer may take a few minutes to index new nodes

"PyNaCl fallback requires a 64-char Ed25519 seed"

• Node.js meshcore-decoder is not available, and the private key is 128 chars
• Solution: install meshcore-decoder (see section 4.2)

---

14. Version History

Version	Date	Description
1.2.0	2026-02-26	Fix: 128-char orlp/ed25519 private key support, NODE_PATH auto-detection, improved key validation
1.1.0	2026-02-26	Fase 2: MQTT uplink to LetsMesh (WebSocket+TLS, Ed25519 JWT, privacy filter)
1.0.0	2026-02-26	Fase 1: Read-only archive monitor dashboard

---

15. License

MIT License — see LICENSE file

16. Author

PE1HVH — GitHub