mirror of https://github.com/ajvpot/meshexplorer.git synced 2026-06-30 06:40:56 +02:00

Files

T

Alex Vanderpot c689ca1b3d clickhouse: back map, node search, and chat with live materialized views

Replace the per-request argMax/GROUP-BY views with insert-triggered
(incremental) materialized views so map node positions, node search, and
public-channel chat read pre-aggregated state instead of re-scanning all of
meshcore_packets on every query.

- 005: meshcore_adverts_latest_state (AggregatingMergeTree of argMaxState/
  min/maxState) + incremental MV + backfill; meshcore_adverts_latest becomes a
  -Merge view with the identical column contract. Node search reads it directly;
  map (unified_latest_nodeinfo) is unchanged.
- 006: meshcore_public_channel_messages_raw, a decoded payload_type=5 MergeTree
  keyed (channel_hash, ingest_timestamp); chat dedups by message_id at read time
  over a timestamp-bounded scan. Streaming/pagination push channel+cursor onto
  the primary key.
- Neighbor-edge MVs stay hourly REFRESH (they read the preserved view).

Verified against full prod data (14.5M rows): exact parity (0 mismatches) and
5-9x faster reads with no regressions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-15 00:13:00 -04:00

clickhouse

clickhouse: upgrade image 25.6.2.5 -> 26.5.1.882

2026-05-29 10:16:22 -04:00

cmd/meshcoreingest

ingest: fix go module path to match repo

2026-05-29 01:19:42 -04:00

internal

ingest: resubscribe on reconnect + staleness watchdog for zombie MQTT conns (#38 )

2026-06-14 11:55:33 -04:00

migrations

clickhouse: back map, node search, and chat with live materialized views

2026-06-15 00:13:00 -04:00

.gitignore

ingest: meshcore MQTT->ClickHouse stack

2026-05-29 01:01:33 -04:00

Dockerfile

ingest: meshcore MQTT->ClickHouse stack

2026-05-29 01:01:33 -04:00

go.mod

ingest: fix go module path to match repo

2026-05-29 01:19:42 -04:00

go.sum

ingest: meshcore MQTT->ClickHouse stack

2026-05-29 01:01:33 -04:00

README.md

ingest: meshcore MQTT->ClickHouse stack

2026-05-29 01:01:33 -04:00

README.md

MeshCore Ingest

A Go service that ingests MeshCore MQTT messages into ClickHouse, plus the ClickHouse image and SQL migrations for the schema.

This directory is normally run as part of the full stack via the root docker compose. The notes below cover running and developing it on its own.

Components

cmd/meshcoreingest — the ingest daemon. Subscribes to MeshCore MQTT topics and writes raw packets into the meshcore_packets table.
internal/ingestcommon — shared MQTT + ClickHouse connection/daemon logic.
internal/migrate — a goose based migration runner (ClickHouse dialect).
migrations/ — the ClickHouse schema: the meshcore_packets table, the decoded meshcore_adverts / meshcore_adverts_latest / meshcore_public_channel_messages views, and the unified_latest_nodeinfo view consumed by the web app.
clickhouse/ — a thin ClickHouse server image plus the read-only user used by the web app.

Configuration

All configuration is via environment variables (no credentials are baked into the source):

Variable	Description
`MQTT_BROKERS`	JSON array of brokers: `[{"url","username","password","topics"}]`. `topics` defaults to `["meshcore/#"]`. Required; the daemon exits if unset.
`MQTT_CLIENT_ID`	MQTT client id prefix (default `meshcore-ingest`).
`CLICKHOUSE_HOST` / `CLICKHOUSE_PORT`	ClickHouse address (native protocol, default `127.0.0.1:9000`).
`CLICKHOUSE_DB` / `CLICKHOUSE_USER` / `CLICKHOUSE_PASSWORD`	ClickHouse database and read/write credentials.

Building

go build ./...
go test ./...

Running migrations

go run ./internal/migrate \
  -host localhost -port 9000 \
  -username default -password "$CLICKHOUSE_PASSWORD" \
  -path migrations -action up

Actions: up, down, reset, status, version.

Running the ingest daemon

export MQTT_BROKERS='[{"url":"tcp://mqtt.example.com:1883","username":"u","password":"p","topics":["meshcore/#"]}]'
export CLICKHOUSE_HOST=localhost CLICKHOUSE_PORT=9000 CLICKHOUSE_PASSWORD=...
go run ./cmd/meshcoreingest