From b7bcca6bf3d2fe0ca2235635e21b078e30ff0a58 Mon Sep 17 00:00:00 2001 From: Alex Vanderpot Date: Fri, 29 May 2026 00:52:54 -0400 Subject: [PATCH] release-prep: docs, migration dialect fix, drop unused UDF - Add a deploy-focused root README; update the web app README (meshcore-only, point Docker usage at the unified root compose). - Fix the migration runner: set the goose clickhouse dialect (it defaulted to postgres and failed to create its version table). Migrations now apply cleanly. - Remove the unused meshcore decrypt UDF (meshcore_try_decrypt was never called by any view/query/code) and simplify the ClickHouse image to a single stage. Verified end-to-end: `docker compose up` brings up clickhouse -> migrate -> meshcoreingest + meshexplorer; live ingestion from the real MQTT brokers lands packets in ClickHouse and the web API serves decoded meshcore nodes via the readonly user. Co-Authored-By: Claude Opus 4.8 --- README.md | 81 ++++++++++++++++++ ingest/clickhouse/Dockerfile | 23 +---- ingest/clickhouse/meshcore_decrypt_udf.c | 99 ---------------------- ingest/clickhouse/meshcore_decrypt_udf.xml | 30 ------- ingest/clickhouse/udf-config.xml | 4 - ingest/internal/migrate/main.go | 5 ++ meshexplorer/README.md | 74 +++------------- 7 files changed, 102 insertions(+), 214 deletions(-) create mode 100644 README.md delete mode 100644 ingest/clickhouse/meshcore_decrypt_udf.c delete mode 100644 ingest/clickhouse/meshcore_decrypt_udf.xml delete mode 100644 ingest/clickhouse/udf-config.xml diff --git a/README.md b/README.md new file mode 100644 index 0000000..a249fb9 --- /dev/null +++ b/README.md @@ -0,0 +1,81 @@ +# MeshExplorer + +A real-time map, chat client, and packet-analysis tool for **MeshCore** mesh +networks. This repository ships the whole stack so it can be brought up with a +single `docker compose up`: + +| Component | Path | Description | +|-----------|------|-------------| +| Web app | [`meshexplorer/`](./meshexplorer) | Next.js UI + API (map, chat, stats, packet analysis) | +| Ingest + DB | [`ingest/`](./ingest) | Go MeshCore MQTT→ClickHouse ingest, ClickHouse image, and SQL migrations | +| Discord relay | [`meshexplorer/`](./meshexplorer) (`Dockerfile.bot`) | Optional bot that relays MeshCore channel messages to Discord | + +## Architecture + +``` + MQTT brokers (you configure) + │ + ▼ + ┌──────────────┐ ┌──────────────┐ + │ meshcoreingest│──▶│ ClickHouse │◀── migrate (one-shot, applies schema) + └──────────────┘ └──────┬───────┘ + │ (readonly user) + ┌──────────┴──────────┐ + ▼ ▼ + meshexplorer discord-bot + (web UI :3001) (optional --profile bot) +``` + +## Quick start + +Requirements: Docker + Docker Compose. + +```bash +cp .env.example .env +# Edit .env — at minimum set: +# CLICKHOUSE_PASSWORD (read/write user, used by ingest + migrations) +# MQTT_BROKERS (JSON array of meshcore MQTT brokers to ingest from) +# Optional, for the Discord relay: DISCORD_WEBHOOK_URL (+ run with --profile bot) + +docker compose up --build +``` + +Then open . + +Startup order is handled automatically: ClickHouse becomes healthy → `migrate` +applies the schema and exits → `meshcoreingest` and `meshexplorer` start. + +To also run the Discord relay: + +```bash +docker compose --profile bot up --build +``` + +## Configuration + +All configuration is via environment variables in `.env` (see +[`.env.example`](./.env.example) for the full list and defaults). Highlights: + +- **ClickHouse** — two accounts. The read/write `default` user + (`CLICKHOUSE_PASSWORD`) is used by the ingest daemon and the migration runner; + the `readonly` user (`CLICKHOUSE_READONLY_PASSWORD`) is used by the web app and + the Discord bot. ClickHouse is only published to `127.0.0.1` for debugging and + is otherwise reachable only on the internal `meshnet` network. +- **MQTT_BROKERS** — a JSON array; each entry is + `{ "url", "username", "password", "topics" }` (`topics` defaults to + `["meshcore/#"]`). The ingest daemon exits with a clear error if this is unset, + so configure at least one broker. + +## Development + +Each component can be run on its own: + +- Web app: see [`meshexplorer/README.md`](./meshexplorer/README.md) + (`npm install && npm run dev`). +- Ingest: see [`ingest/README.md`](./ingest/README.md) (`go build ./...`). + +## Security notes + +- `.env` is gitignored — keep real credentials out of version control. +- If you previously used the bundled defaults, rotate any secrets before going + to production. diff --git a/ingest/clickhouse/Dockerfile b/ingest/clickhouse/Dockerfile index a26050c..ae21d14 100644 --- a/ingest/clickhouse/Dockerfile +++ b/ingest/clickhouse/Dockerfile @@ -1,26 +1,7 @@ -FROM clickhouse/clickhouse-server:latest as builder - -RUN apt-get update && \ - apt-get install -y build-essential libssl-dev libmsgpack-dev && \ - rm -rf /var/lib/apt/lists/* - -WORKDIR /build -COPY meshcore_decrypt_udf.c ./ -RUN gcc -O2 -o meshcore_decrypt_udf meshcore_decrypt_udf.c -lssl -lcrypto -lmsgpackc - FROM clickhouse/clickhouse-server:latest -# Copy UDF binary to /usr/local/share/clickhouse-udf-bins -COPY --from=builder /build/meshcore_decrypt_udf /usr/local/share/clickhouse-udf-bins/meshcore_decrypt_udf - -# Copy UDF XML config directly to the location and name matching ClickHouse config pattern -COPY meshcore_decrypt_udf.xml /etc/clickhouse-server/meshcore_decrypt_function.xml - -# Copy users configuration file +# Read-only user used by the web app / discord bot COPY users.xml /etc/clickhouse-server/users.d/readonly_user.xml -# Copy access control configuration +# Access control configuration COPY config.xml /etc/clickhouse-server/config.d/access_control.xml - -RUN chmod +x /usr/local/share/clickhouse-udf-bins/meshcore_decrypt_udf -# Remove custom entrypoint, use default \ No newline at end of file diff --git a/ingest/clickhouse/meshcore_decrypt_udf.c b/ingest/clickhouse/meshcore_decrypt_udf.c deleted file mode 100644 index 2a77a14..0000000 --- a/ingest/clickhouse/meshcore_decrypt_udf.c +++ /dev/null @@ -1,99 +0,0 @@ -#include -#include -#include -#include -#include -#include - -#define CIPHER_MAC_SIZE 2 -#define CIPHER_KEY_SIZE 16 -#define MAX_PAYLOAD 184 - -// Default "public" channel key -static const unsigned char public_channel_private_key[16] = { - 0x8B, 0x33, 0x87, 0xE5, 0xCD, 0xEA, 0x6A, 0xC9, - 0xE5, 0xED, 0xBA, 0xA1, 0x15, 0xCD, 0x72, 0x0F -}; - -static int decrypt_msg(const unsigned char *key, const unsigned char *data, size_t len, unsigned char *out, size_t *out_len) { - if (len <= CIPHER_MAC_SIZE) return 0; - - unsigned char mac[SHA256_DIGEST_LENGTH]; - HMAC(EVP_sha256(), key, CIPHER_KEY_SIZE, data + CIPHER_MAC_SIZE, - len - CIPHER_MAC_SIZE, mac, NULL); - - if (memcmp(mac, data, CIPHER_MAC_SIZE) != 0) return 0; - - AES_KEY aes_key; - AES_set_decrypt_key(key, 128, &aes_key); - - size_t blocks = (len - CIPHER_MAC_SIZE) / 16; - for (size_t i = 0; i < blocks; i++) { - AES_decrypt(data + CIPHER_MAC_SIZE + (i * 16), out + (i * 16), &aes_key); - } - *out_len = blocks * 16; - return 1; -} - -int main() { - unsigned char buf[8192]; - size_t len = fread(buf, 1, sizeof(buf), stdin); - if (len == 0) return 0; - - msgpack_unpacked result; - msgpack_unpacked_init(&result); - - size_t off = 0; - msgpack_unpack_return ret = msgpack_unpack_next(&result, (char*)buf, len, &off); - - if (ret == MSGPACK_UNPACK_SUCCESS && result.data.type == MSGPACK_OBJECT_ARRAY) { - msgpack_object_array arr = result.data.via.array; - msgpack_sbuffer sbuf; - msgpack_sbuffer_init(&sbuf); - msgpack_packer pk; - msgpack_packer_init(&pk, &sbuf, msgpack_sbuffer_write); - - msgpack_pack_array(&pk, arr.size); - - for (size_t i = 0; i < arr.size; i++) { - msgpack_object row = arr.ptr[i]; - if (row.type == MSGPACK_OBJECT_ARRAY && row.via.array.size >= 3) { - msgpack_object mac_obj = row.via.array.ptr[0]; - msgpack_object enc_obj = row.via.array.ptr[1]; - msgpack_object hash_obj = row.via.array.ptr[2]; - if (mac_obj.type == MSGPACK_OBJECT_STR && enc_obj.type == MSGPACK_OBJECT_STR && hash_obj.type == MSGPACK_OBJECT_STR && hash_obj.via.str.size > 0) { - // Concatenate mac + encrypted_data - size_t mac_len = mac_obj.via.str.size; - size_t enc_len = enc_obj.via.str.size; - unsigned char mac_and_enc[MAX_PAYLOAD]; - if (mac_len + enc_len > MAX_PAYLOAD) { msgpack_pack_nil(&pk); continue; } - memcpy(mac_and_enc, mac_obj.via.str.ptr, mac_len); - memcpy(mac_and_enc + mac_len, enc_obj.via.str.ptr, enc_len); - unsigned char decrypted[MAX_PAYLOAD]; - size_t decrypted_len; - if (decrypt_msg(public_channel_private_key, mac_and_enc, mac_len + enc_len, decrypted, &decrypted_len)) { - msgpack_pack_array(&pk, 2); - msgpack_pack_str(&pk, decrypted_len); - msgpack_pack_str_body(&pk, (char*)decrypted, decrypted_len); - unsigned char full_hash[SHA256_DIGEST_LENGTH]; - SHA256(public_channel_private_key, CIPHER_KEY_SIZE, full_hash); - msgpack_pack_str(&pk, SHA256_DIGEST_LENGTH); - msgpack_pack_str_body(&pk, (char*)full_hash, SHA256_DIGEST_LENGTH); - } else { - msgpack_pack_nil(&pk); - } - } else { - msgpack_pack_nil(&pk); - } - } else { - msgpack_pack_nil(&pk); - } - } - - fwrite(sbuf.data, 1, sbuf.size, stdout); - msgpack_sbuffer_destroy(&sbuf); - } - - msgpack_unpacked_destroy(&result); - return 0; -} \ No newline at end of file diff --git a/ingest/clickhouse/meshcore_decrypt_udf.xml b/ingest/clickhouse/meshcore_decrypt_udf.xml deleted file mode 100644 index 05a95ce..0000000 --- a/ingest/clickhouse/meshcore_decrypt_udf.xml +++ /dev/null @@ -1,30 +0,0 @@ - - - - meshcore_try_decrypt - executable - /usr/local/share/clickhouse-udf-bins/meshcore_decrypt_udf - - String - mac - - - String - encrypted_data - - - String - channel_hash_raw - - MsgPack - Nullable(Array(String)) - result - 1 - 30 - 10 - 10000 - 10000 - false - true - - \ No newline at end of file diff --git a/ingest/clickhouse/udf-config.xml b/ingest/clickhouse/udf-config.xml deleted file mode 100644 index 54c59b1..0000000 --- a/ingest/clickhouse/udf-config.xml +++ /dev/null @@ -1,4 +0,0 @@ - - /usr/local/share/clickhouse-udfs/*.xml - /usr/local/share/clickhouse-udf-bins/ - \ No newline at end of file diff --git a/ingest/internal/migrate/main.go b/ingest/internal/migrate/main.go index 621e554..a7ac3c0 100644 --- a/ingest/internal/migrate/main.go +++ b/ingest/internal/migrate/main.go @@ -47,6 +47,11 @@ func main() { log.Fatalf("Failed to connect to ClickHouse: %v", err) } + // ClickHouse is the target database; goose defaults to postgres otherwise. + if err := goose.SetDialect("clickhouse"); err != nil { + log.Fatalf("Failed to set goose dialect: %v", err) + } + // Set goose database goose.SetBaseFS(os.DirFS(*migrationsPath)) diff --git a/meshexplorer/README.md b/meshexplorer/README.md index 5a4ec1a..92133b0 100644 --- a/meshexplorer/README.md +++ b/meshexplorer/README.md @@ -1,9 +1,13 @@ # MeshExplorer -MeshExplorer is a real-time map, chat client, and packet analysis tool for mesh networks using MeshCore and Meshtastic. It enables users to visualize mesh nodes on a map, communicate via chat, and analyze packet data in real time. +MeshExplorer is a real-time map, chat client, and packet analysis tool for MeshCore mesh networks. It enables users to visualize mesh nodes on a map, communicate via chat, and analyze packet data in real time. + +> This directory contains the web app only. For the full stack (ClickHouse, +> migrations, ingest, and this app) see the [root README](../README.md) and the +> unified `docker compose` setup. ## Features -- Real-time map of mesh network nodes (MeshCore and Meshtastic) +- Real-time map of mesh network nodes (MeshCore) - Integrated chat client for mesh channels - Packet analysis and inspection tools - Customizable map layers and clustering @@ -70,71 +74,21 @@ The middleware applies the following CORS headers to all `/api/*` routes: ## Learn More - [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API. -- [MeshCore](https://github.com/your-org/meshcore) - mesh network backend -- [Meshtastic](https://meshtastic.org/) - open source mesh communication project +- [MeshCore](https://meshcore.co.uk/) - mesh network firmware ## Docker Deployment -The application includes Docker support for easy deployment. The Docker configuration is set up to connect to ClickHouse running on the Docker host. - -### Prerequisites - -- Docker and Docker Compose installed -- ClickHouse running on the Docker host (default port 8123) -- External Docker network `shared-network` must exist (see setup instructions below) - -### External Network Setup - -The application requires an external Docker network called `shared-network` to communicate with ClickHouse. You must create this network before running the application: +This app is deployed as part of the unified stack at the repository root. From +the repo root: ```bash -docker network create shared-network +cp .env.example .env # then edit +docker compose up --build ``` -**Note**: If the network already exists, this command will show an error but can be safely ignored. - -### Running with Docker Compose - -1. **Create the required external network (if not already created):** - ```bash - docker network create shared-network - ``` - -2. **Build and start the application:** - ```bash - docker-compose up --build - ``` - -3. **Access the application:** - Open [http://localhost:3001](http://localhost:3001) in your browser. - -### Docker Configuration - -The `docker-compose.yml` file is configured with: -- **Port mapping**: Container port 3000 → Host port 3001 -- **ClickHouse connection**: Uses `clickhouse` hostname to connect to ClickHouse via the shared network -- **External network**: Requires `shared-network` to be created externally -- **Environment variables**: Pre-configured for typical ClickHouse setup - -### Customizing ClickHouse Connection - -You can customize the ClickHouse connection by modifying the environment variables in `docker-compose.yml`: - -```yaml -environment: - - CLICKHOUSE_HOST=your-clickhouse-host - - CLICKHOUSE_PORT=8123 - - CLICKHOUSE_USER=your-username - - CLICKHOUSE_PASSWORD=your-password -``` - -### Building with BuildKit - -For faster builds with caching, enable BuildKit: - -```bash -DOCKER_BUILDKIT=1 docker-compose up --build -``` +The web app is served on . ClickHouse, migrations, and the +MeshCore ingest are started for you. See the [root README](../README.md) for the +full list of services and environment variables. ## Deploy