mirror of
https://github.com/ajvpot/meshexplorer.git
synced 2026-06-29 14:20:59 +02:00
Alex Vanderpot
ad4f660c38
The MeshCore path_length byte encodes hop_count in its low 6 bits and a hash_size_code in its high 2 bits (0/1/2 -> 1/2/3 bytes per hop), and transport route types (TRANSPORT_FLOOD/TRANSPORT_DIRECT) insert a 4-byte transport_codes field between the header and path_length. The packet decoder previously assumed every hop was a single byte and that path_length always sat at byte 2, so it only handled 1-byte-hash, non-transport packets; anything else decoded to an over-long path and an empty payload. Migration 007 reworks the meshcore_packets read-time aliases to honor the transport_codes offset and compute the path as hop_count * hash_size bytes, and exposes hop_count / hash_size_code / hash_size (bytes per hop) as columns. payload, path and packet_hash now decode correctly for every route type and hash size; the adverts and public-channel derived tables are rebuilt from the corrected decode (invalid hash_size_code 3 packets are skipped per spec). hash_size is carried through the chat and advert APIs so the path visualization splits a path into hops of the correct width (pathUtils/PathVisualization), instead of always slicing one byte per hop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MeshCore Ingest
A Go service that ingests MeshCore MQTT messages into ClickHouse, plus the ClickHouse image and SQL migrations for the schema.
This directory is normally run as part of the full stack via the
root docker compose. The notes below cover running and
developing it on its own.
Components
cmd/meshcoreingest— the ingest daemon. Subscribes to MeshCore MQTT topics and writes raw packets into themeshcore_packetstable.internal/ingestcommon— shared MQTT + ClickHouse connection/daemon logic.internal/migrate— a goose based migration runner (ClickHouse dialect).migrations/— the ClickHouse schema: themeshcore_packetstable, the decodedmeshcore_adverts/meshcore_adverts_latest/meshcore_public_channel_messagesviews, and theunified_latest_nodeinfoview consumed by the web app.clickhouse/— a thin ClickHouse server image plus the read-only user used by the web app.
Configuration
All configuration is via environment variables (no credentials are baked into the source):
| Variable | Description |
|---|---|
MQTT_BROKERS |
JSON array of brokers: [{"url","username","password","topics"}]. topics defaults to ["meshcore/#"]. Required; the daemon exits if unset. |
MQTT_CLIENT_ID |
MQTT client id prefix (default meshcore-ingest). |
CLICKHOUSE_HOST / CLICKHOUSE_PORT |
ClickHouse address (native protocol, default 127.0.0.1:9000). |
CLICKHOUSE_DB / CLICKHOUSE_USER / CLICKHOUSE_PASSWORD |
ClickHouse database and read/write credentials. |
Building
go build ./...
go test ./...
Running migrations
go run ./internal/migrate \
-host localhost -port 9000 \
-username default -password "$CLICKHOUSE_PASSWORD" \
-path migrations -action up
Actions: up, down, reset, status, version.
Running the ingest daemon
export MQTT_BROKERS='[{"url":"tcp://mqtt.example.com:1883","username":"u","password":"p","topics":["meshcore/#"]}]'
export CLICKHOUSE_HOST=localhost CLICKHOUSE_PORT=9000 CLICKHOUSE_PASSWORD=...
go run ./cmd/meshcoreingest