mirror of
https://github.com/ajvpot/meshexplorer.git
synced 2026-08-06 08:32:45 +02:00
Alex Vanderpot
281f4ab25b
The meshcore handler did a synchronous per-message ClickHouse insert on paho's single inbound goroutine. At ~86ms/insert (single-row inserts + async_insert wait + materialized views) the goroutine couldn't keep up with the high-volume letsmesh feed, so it stalled past PingTimeout and paho declared "pingresp not received" and reconnected — ~847 cycles in 19.5h, ~45% downtime, ~50% of letsmesh packets lost. The low-volume davekeogh broker never saturated the goroutine and was unaffected. Decouple receipt from insertion: the handler now enqueues decoded rows onto a buffered channel and a single background writer flushes them to meshcore_packets in batched native inserts (every MESHCORE_BATCH_FLUSH_ SECONDS or MESHCORE_BATCH_MAX_ROWS rows). The inbound goroutine never blocks, so PINGRESP is always processed in time. - New batch writer with env-configurable flush interval / max rows / buffer size (MESHCORE_BATCH_* ), wired in docker-compose. - Drop server-side async_insert (redundant once we batch app-side). - Bump PingTimeout 10s -> 20s (env MQTT_PING_TIMEOUT_SECONDS) for margin against Cloudflare WebSocket buffering jitter. - Enqueue is non-blocking; rows are dropped+counted only if the buffer fills (ClickHouse unavailable). A failed batch is dropped and retried by the next flush (native blocks commit atomically). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MeshExplorer
A real-time map, chat client, and packet-analysis tool for MeshCore mesh
networks. This repository ships the whole stack so it can be brought up with a
single docker compose up:
| Component | Path | Description |
|---|---|---|
| Web app | meshexplorer/ |
Next.js UI + API (map, chat, stats, packet analysis) |
| Ingest + DB | ingest/ |
Go MeshCore MQTT→ClickHouse ingest, ClickHouse image, and SQL migrations |
| Discord relay | meshexplorer/ (Dockerfile.bot) |
Optional bot that relays MeshCore channel messages to Discord |
| Grafana | grafana/ |
Dashboards with a pre-provisioned ClickHouse datasource (read-only user), on 127.0.0.1:3000 |
Architecture
MQTT brokers (you configure)
│
▼
┌──────────────┐ ┌──────────────┐
│ meshcoreingest│──▶│ ClickHouse │◀── migrate (one-shot, applies schema)
└──────────────┘ └──────┬───────┘
│ (readonly user)
┌──────────┴──────────┐
▼ ▼ ▼
meshexplorer discord-bot grafana
(web UI :3001) (--profile bot) (:3000)
Quick start
Requirements: Docker + Docker Compose.
cp .env.example .env
# Edit .env — at minimum set:
# CLICKHOUSE_PASSWORD (read/write user, used by ingest + migrations)
# MQTT_BROKERS (JSON array of meshcore MQTT brokers to ingest from)
# Optional, for the Discord relay: DISCORD_WEBHOOK_URL (+ run with --profile bot)
docker compose up --build
Then open http://localhost:3001.
Startup order is handled automatically: ClickHouse becomes healthy → migrate
applies the schema and exits → meshcoreingest and meshexplorer start.
To also run the Discord relay:
docker compose --profile bot up --build
Configuration
All configuration is via environment variables in .env (see
.env.example for the full list and defaults). Highlights:
- ClickHouse — two accounts. The read/write
defaultuser (CLICKHOUSE_PASSWORD) is used by the ingest daemon and the migration runner; thereadonlyuser (CLICKHOUSE_READONLY_PASSWORD) is used by the web app and the Discord bot. ClickHouse is only published to127.0.0.1for debugging and is otherwise reachable only on the internalmeshnetnetwork. - MQTT_BROKERS — a JSON array; each entry is
{ "url", "username", "password", "topics" }(topicsdefaults to["meshcore/#"]). The ingest daemon exits with a clear error if this is unset, so configure at least one broker.
Development
Each component can be run on its own:
- Web app: see
meshexplorer/README.md(npm install && npm run dev). - Ingest: see
ingest/README.md(go build ./...).
Security notes
.envis gitignored — keep real credentials out of version control.- If you previously used the bundled defaults, rotate any secrets before going to production.