mirror of
https://github.com/ajvpot/meshexplorer.git
synced 2026-06-10 17:24:52 +02:00
Alex Vanderpot
a0ab900da3
Fixes a 25.x global memory-tracker drift where the tracker pinned at the max-memory cap (RSS far below it), causing the OvercommitTracker to kill every query (map/stats/neighbors all 500ing). Deployed in-place on prod over the existing data dir after a cold backup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MeshCore Ingest
A Go service that ingests MeshCore MQTT messages into ClickHouse, plus the ClickHouse image and SQL migrations for the schema.
This directory is normally run as part of the full stack via the
root docker compose. The notes below cover running and
developing it on its own.
Components
cmd/meshcoreingest— the ingest daemon. Subscribes to MeshCore MQTT topics and writes raw packets into themeshcore_packetstable.internal/ingestcommon— shared MQTT + ClickHouse connection/daemon logic.internal/migrate— a goose based migration runner (ClickHouse dialect).migrations/— the ClickHouse schema: themeshcore_packetstable, the decodedmeshcore_adverts/meshcore_adverts_latest/meshcore_public_channel_messagesviews, and theunified_latest_nodeinfoview consumed by the web app.clickhouse/— a thin ClickHouse server image plus the read-only user used by the web app.
Configuration
All configuration is via environment variables (no credentials are baked into the source):
| Variable | Description |
|---|---|
MQTT_BROKERS |
JSON array of brokers: [{"url","username","password","topics"}]. topics defaults to ["meshcore/#"]. Required; the daemon exits if unset. |
MQTT_CLIENT_ID |
MQTT client id prefix (default meshcore-ingest). |
CLICKHOUSE_HOST / CLICKHOUSE_PORT |
ClickHouse address (native protocol, default 127.0.0.1:9000). |
CLICKHOUSE_DB / CLICKHOUSE_USER / CLICKHOUSE_PASSWORD |
ClickHouse database and read/write credentials. |
Building
go build ./...
go test ./...
Running migrations
go run ./internal/migrate \
-host localhost -port 9000 \
-username default -password "$CLICKHOUSE_PASSWORD" \
-path migrations -action up
Actions: up, down, reset, status, version.
Running the ingest daemon
export MQTT_BROKERS='[{"url":"tcp://mqtt.example.com:1883","username":"u","password":"p","topics":["meshcore/#"]}]'
export CLICKHOUSE_HOST=localhost CLICKHOUSE_PORT=9000 CLICKHOUSE_PASSWORD=...
go run ./cmd/meshcoreingest