Alex Vanderpot 7cea182c6d ingest: batch ClickHouse inserts to stop MQTT flapping & packet loss (#41)
* ingest: batch ClickHouse inserts to stop MQTT flapping & packet loss

The meshcore handler did a synchronous per-message ClickHouse insert on
paho's single inbound goroutine. At ~86ms/insert (single-row inserts +
async_insert wait + materialized views) the goroutine couldn't keep up
with the high-volume letsmesh feed, so it stalled past PingTimeout and
paho declared "pingresp not received" and reconnected — ~847 cycles in
19.5h, ~45% downtime, ~50% of letsmesh packets lost. The low-volume
davekeogh broker never saturated the goroutine and was unaffected.

Decouple receipt from insertion: the handler now enqueues decoded rows
onto a buffered channel and a single background writer flushes them to
meshcore_packets in batched native inserts (every MESHCORE_BATCH_FLUSH_
SECONDS or MESHCORE_BATCH_MAX_ROWS rows). The inbound goroutine never
blocks, so PINGRESP is always processed in time.

- New batch writer with env-configurable flush interval / max rows /
  buffer size (MESHCORE_BATCH_* ), wired in docker-compose.
- Drop server-side async_insert (redundant once we batch app-side).
- Bump PingTimeout 10s -> 20s (env MQTT_PING_TIMEOUT_SECONDS) for margin
  against Cloudflare WebSocket buffering jitter.
- Enqueue is non-blocking; rows are dropped+counted only if the buffer
  fills (ClickHouse unavailable). A failed batch is dropped and retried
  by the next flush (native blocks commit atomically).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ingest: make MQTT KeepAlive configurable (MQTT_KEEPALIVE_SECONDS)

As a near-silent subscriber, paho emits a PINGREQ roughly every KeepAlive
seconds; lowering it sends client->server frames more often to keep the
Cloudflare-proxied WebSocket path warm in both directions, a lever for the
residual mid-stream "pingresp not received" stalls on the letsmesh broker.
Default unchanged (30s); wired through docker-compose.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ingest: add configurable MQTT write timeout (MQTT_WRITE_TIMEOUT_SECONDS)

Bounds PINGREQ/SUBSCRIBE writes so a stalled write through the Cloudflare
WebSocket proxy can't hang the client. Default 0 (paho's existing no-timeout
behavior); wired through docker-compose. Recommended ~20s when behind a
buffering reverse proxy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Alex Vanderpot <alex@Alexs-MacBook-Pro-2.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 22:42:46 -04:00

MeshExplorer

A real-time map, chat client, and packet-analysis tool for MeshCore mesh networks. This repository ships the whole stack so it can be brought up with a single docker compose up:

Component Path Description
Web app meshexplorer/ Next.js UI + API (map, chat, stats, packet analysis)
Ingest + DB ingest/ Go MeshCore MQTT→ClickHouse ingest, ClickHouse image, and SQL migrations
Discord relay meshexplorer/ (Dockerfile.bot) Optional bot that relays MeshCore channel messages to Discord
Grafana grafana/ Dashboards with a pre-provisioned ClickHouse datasource (read-only user), on 127.0.0.1:3000

Architecture

                MQTT brokers (you configure)
                          │
                          ▼
   ┌──────────────┐   ┌──────────────┐
   │ meshcoreingest│──▶│  ClickHouse  │◀── migrate (one-shot, applies schema)
   └──────────────┘   └──────┬───────┘
                             │ (readonly user)
                  ┌──────────┴──────────┐
                  ▼              ▼              ▼
            meshexplorer     discord-bot      grafana
            (web UI :3001)   (--profile bot)  (:3000)

Quick start

Requirements: Docker + Docker Compose.

cp .env.example .env
# Edit .env — at minimum set:
#   CLICKHOUSE_PASSWORD   (read/write user, used by ingest + migrations)
#   MQTT_BROKERS          (JSON array of meshcore MQTT brokers to ingest from)
# Optional, for the Discord relay: DISCORD_WEBHOOK_URL (+ run with --profile bot)

docker compose up --build

Then open http://localhost:3001.

Startup order is handled automatically: ClickHouse becomes healthy → migrate applies the schema and exits → meshcoreingest and meshexplorer start.

To also run the Discord relay:

docker compose --profile bot up --build

Configuration

All configuration is via environment variables in .env (see .env.example for the full list and defaults). Highlights:

  • ClickHouse — two accounts. The read/write default user (CLICKHOUSE_PASSWORD) is used by the ingest daemon and the migration runner; the readonly user (CLICKHOUSE_READONLY_PASSWORD) is used by the web app and the Discord bot. ClickHouse is only published to 127.0.0.1 for debugging and is otherwise reachable only on the internal meshnet network.
  • MQTT_BROKERS — a JSON array; each entry is { "url", "username", "password", "topics" } (topics defaults to ["meshcore/#"]). The ingest daemon exits with a clear error if this is unset, so configure at least one broker.

Development

Each component can be run on its own:

Security notes

  • .env is gitignored — keep real credentials out of version control.
  • If you previously used the bundled defaults, rotate any secrets before going to production.
S
Description
No description provided
Readme 1.2 MiB
Languages
TypeScript 83.6%
Go 10.1%
JavaScript 3.7%
CSS 1.6%
Dockerfile 0.8%
Other 0.2%