mirror of
https://github.com/ajvpot/meshexplorer.git
synced 2026-07-06 17:50:58 +02:00
Alex Vanderpot
c5ee493d8c
Replace the hand-curated region groups with data-driven ones and make the region_groups ClickHouse table the single source of truth. - scripts/generate-region-groups.ts: offline generator — clusters regions by cross-region packet co-occurrence (min-share single-linkage) at two levels (broad "region" + tight "metro"), names clusters via `claude -p`, reconciles codes by member overlap so permalinks stay stable, and emits the region_groups seed. Migration 004 reseeded with the resulting 39 groups. - Groups are DB-sourced: getRegionGroups() (cached) feeds /api/regions and the dropdown/labels; filtering resolves a selector in SQL to a region (region = 'X') or a group (region IN / hasAny ... SELECT region_code FROM region_groups WHERE group_code = ...). No hardcoded membership in TS; resolveSelector removed. - Drop the TS<->SQL parity script (no membership left to sync); regionSql and the migration ALIAS are kept in sync by hand. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MeshCore Ingest
A Go service that ingests MeshCore MQTT messages into ClickHouse, plus the ClickHouse image and SQL migrations for the schema.
This directory is normally run as part of the full stack via the
root docker compose. The notes below cover running and
developing it on its own.
Components
cmd/meshcoreingest— the ingest daemon. Subscribes to MeshCore MQTT topics and writes raw packets into themeshcore_packetstable.internal/ingestcommon— shared MQTT + ClickHouse connection/daemon logic.internal/migrate— a goose based migration runner (ClickHouse dialect).migrations/— the ClickHouse schema: themeshcore_packetstable, the decodedmeshcore_adverts/meshcore_adverts_latest/meshcore_public_channel_messagesviews, and theunified_latest_nodeinfoview consumed by the web app.clickhouse/— a thin ClickHouse server image plus the read-only user used by the web app.
Configuration
All configuration is via environment variables (no credentials are baked into the source):
| Variable | Description |
|---|---|
MQTT_BROKERS |
JSON array of brokers: [{"url","username","password","topics"}]. topics defaults to ["meshcore/#"]. Required; the daemon exits if unset. |
MQTT_CLIENT_ID |
MQTT client id prefix (default meshcore-ingest). |
CLICKHOUSE_HOST / CLICKHOUSE_PORT |
ClickHouse address (native protocol, default 127.0.0.1:9000). |
CLICKHOUSE_DB / CLICKHOUSE_USER / CLICKHOUSE_PASSWORD |
ClickHouse database and read/write credentials. |
Building
go build ./...
go test ./...
Running migrations
go run ./internal/migrate \
-host localhost -port 9000 \
-username default -password "$CLICKHOUSE_PASSWORD" \
-path migrations -action up
Actions: up, down, reset, status, version.
Running the ingest daemon
export MQTT_BROKERS='[{"url":"tcp://mqtt.example.com:1883","username":"u","password":"p","topics":["meshcore/#"]}]'
export CLICKHOUSE_HOST=localhost CLICKHOUSE_PORT=9000 CLICKHOUSE_PASSWORD=...
go run ./cmd/meshcoreingest