mirror of https://github.com/ajvpot/meshexplorer.git synced 2026-06-11 09:44:52 +02:00

Files

T

Alex Vanderpot 83978609f0 ingest: meshcore MQTT->ClickHouse stack

Vendor the ingest service under ingest/ and move the web app under meshexplorer/.
The ingest builds the meshcoreingest daemon and the goose migration runner,
applies the meshcore ClickHouse schema (packets, adverts, unified node view),
and loads its MQTT broker list and ClickHouse settings entirely from environment
variables (MQTT_BROKERS as a JSON array, CLICKHOUSE_*). No credentials are baked
into the source.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-05-29 01:01:33 -04:00

2.1 KiB

Raw Blame History

MeshCore Ingest

A Go service that ingests MeshCore MQTT messages into ClickHouse, plus the ClickHouse image and SQL migrations for the schema.

This directory is normally run as part of the full stack via the root docker compose. The notes below cover running and developing it on its own.

Components

cmd/meshcoreingest — the ingest daemon. Subscribes to MeshCore MQTT topics and writes raw packets into the meshcore_packets table.
internal/ingestcommon — shared MQTT + ClickHouse connection/daemon logic.
internal/migrate — a goose based migration runner (ClickHouse dialect).
migrations/ — the ClickHouse schema: the meshcore_packets table, the decoded meshcore_adverts / meshcore_adverts_latest / meshcore_public_channel_messages views, and the unified_latest_nodeinfo view consumed by the web app.
clickhouse/ — a thin ClickHouse server image plus the read-only user used by the web app.

Configuration

All configuration is via environment variables (no credentials are baked into the source):

Variable	Description
`MQTT_BROKERS`	JSON array of brokers: `[{"url","username","password","topics"}]`. `topics` defaults to `["meshcore/#"]`. Required; the daemon exits if unset.
`MQTT_CLIENT_ID`	MQTT client id prefix (default `meshcore-ingest`).
`CLICKHOUSE_HOST` / `CLICKHOUSE_PORT`	ClickHouse address (native protocol, default `127.0.0.1:9000`).
`CLICKHOUSE_DB` / `CLICKHOUSE_USER` / `CLICKHOUSE_PASSWORD`	ClickHouse database and read/write credentials.

Building

go build ./...
go test ./...

Running migrations

go run ./internal/migrate \
  -host localhost -port 9000 \
  -username default -password "$CLICKHOUSE_PASSWORD" \
  -path migrations -action up

Actions: up, down, reset, status, version.

Running the ingest daemon

export MQTT_BROKERS='[{"url":"tcp://mqtt.example.com:1883","username":"u","password":"p","topics":["meshcore/#"]}]'
export CLICKHOUSE_HOST=localhost CLICKHOUSE_PORT=9000 CLICKHOUSE_PASSWORD=...
go run ./cmd/meshcoreingest

2.1 KiB Raw Blame History