mirror of https://github.com/ajvpot/meshexplorer.git synced 2026-06-11 09:44:52 +02:00

Files

T

Alex Vanderpot 02623e5559 neighbors: seattle->letsmesh region + fix path-edge prefix extraction

- Point the seattle region at the letsmesh broker (wss://mqtt-us-v1.letsmesh.net:443,
  topic meshcore/SEA) where Seattle traffic now lives.
- Fix a pre-existing bug in the path-edge extraction: `path` is a hex string of
  1-byte hop prefixes, so use substring(path, 2*i-1, 2) instead of
  hex(substring(path, i, 1)) (which re-hexed a single hex char and never matched
  the 2-char repeater prefixes -> path edges were always empty). Seattle now yields
  path edges again.

Verified on a full prod snapshot: the MV-backed "show all neighbors" query drops
from ~1.6s / 145M rows / 11.8 GiB to ~1ms / 108 rows / 3.8 KiB.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-05-29 03:02:09 -04:00

clickhouse

clickhouse: lift readonly read-size caps

2026-05-29 01:48:56 -04:00

cmd/meshcoreingest

ingest: fix go module path to match repo

2026-05-29 01:19:42 -04:00

internal

release-prep: docs, migration dialect fix, drop unused UDF

2026-05-29 01:01:33 -04:00

migrations

neighbors: seattle->letsmesh region + fix path-edge prefix extraction

2026-05-29 03:02:09 -04:00

.gitignore

ingest: meshcore MQTT->ClickHouse stack

2026-05-29 01:01:33 -04:00

Dockerfile

ingest: meshcore MQTT->ClickHouse stack

2026-05-29 01:01:33 -04:00

go.mod

ingest: fix go module path to match repo

2026-05-29 01:19:42 -04:00

go.sum

ingest: meshcore MQTT->ClickHouse stack

2026-05-29 01:01:33 -04:00

README.md

ingest: meshcore MQTT->ClickHouse stack

2026-05-29 01:01:33 -04:00

README.md

MeshCore Ingest

A Go service that ingests MeshCore MQTT messages into ClickHouse, plus the ClickHouse image and SQL migrations for the schema.

This directory is normally run as part of the full stack via the root docker compose. The notes below cover running and developing it on its own.

Components

cmd/meshcoreingest — the ingest daemon. Subscribes to MeshCore MQTT topics and writes raw packets into the meshcore_packets table.
internal/ingestcommon — shared MQTT + ClickHouse connection/daemon logic.
internal/migrate — a goose based migration runner (ClickHouse dialect).
migrations/ — the ClickHouse schema: the meshcore_packets table, the decoded meshcore_adverts / meshcore_adverts_latest / meshcore_public_channel_messages views, and the unified_latest_nodeinfo view consumed by the web app.
clickhouse/ — a thin ClickHouse server image plus the read-only user used by the web app.

Configuration

All configuration is via environment variables (no credentials are baked into the source):

Variable	Description
`MQTT_BROKERS`	JSON array of brokers: `[{"url","username","password","topics"}]`. `topics` defaults to `["meshcore/#"]`. Required; the daemon exits if unset.
`MQTT_CLIENT_ID`	MQTT client id prefix (default `meshcore-ingest`).
`CLICKHOUSE_HOST` / `CLICKHOUSE_PORT`	ClickHouse address (native protocol, default `127.0.0.1:9000`).
`CLICKHOUSE_DB` / `CLICKHOUSE_USER` / `CLICKHOUSE_PASSWORD`	ClickHouse database and read/write credentials.

Building

go build ./...
go test ./...

Running migrations

go run ./internal/migrate \
  -host localhost -port 9000 \
  -username default -password "$CLICKHOUSE_PASSWORD" \
  -path migrations -action up

Actions: up, down, reset, status, version.

Running the ingest daemon

export MQTT_BROKERS='[{"url":"tcp://mqtt.example.com:1883","username":"u","password":"p","topics":["meshcore/#"]}]'
export CLICKHOUSE_HOST=localhost CLICKHOUSE_PORT=9000 CLICKHOUSE_PASSWORD=...
go run ./cmd/meshcoreingest