adam/meshexplorer
1
0
Fork 0
mirror of https://github.com/ajvpot/meshexplorer.git synced 2026-08-06 08:32:45 +02:00
Code Issues Packages Projects Releases Wiki Activity

release-prep: docs, migration dialect fix, drop unused UDF

Browse Source 
- Add a deploy-focused root README; update the web app README (meshcore-only,
  point Docker usage at the unified root compose).
- Fix the migration runner: set the goose clickhouse dialect (it defaulted to
  postgres and failed to create its version table). Migrations now apply cleanly.
- Remove the unused meshcore decrypt UDF (meshcore_try_decrypt was never called
  by any view/query/code) and simplify the ClickHouse image to a single stage.

Verified end-to-end: `docker compose up` brings up clickhouse -> migrate ->
meshcoreingest + meshexplorer; live ingestion from the real MQTT brokers lands
packets in ClickHouse and the web API serves decoded meshcore nodes via the
readonly user.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Alex Vanderpot
2026-05-29 00:52:54 -04:00
parent 6a1536410c
commit b7bcca6bf3
7 changed files with 102 additions and 214 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+81
View File
@@ -0,0 +1,81 @@
# MeshExplorer
A real-time map, chat client, and packet-analysis tool for **MeshCore** mesh
networks. This repository ships the whole stack so it can be brought up with a
single `docker compose up`:
| Component | Path | Description |
|-----------|------|-------------|
| Web app | [`meshexplorer/`](./meshexplorer) | Next.js UI + API (map, chat, stats, packet analysis) |
| Ingest + DB | [`ingest/`](./ingest) | Go MeshCore MQTT→ClickHouse ingest, ClickHouse image, and SQL migrations |
| Discord relay | [`meshexplorer/`](./meshexplorer) (`Dockerfile.bot`) | Optional bot that relays MeshCore channel messages to Discord |
## Architecture
```
MQTT brokers (you configure)
│
▼
┌──────────────┐ ┌──────────────┐
│ meshcoreingest│──▶│ ClickHouse │◀── migrate (one-shot, applies schema)
└──────────────┘ └──────┬───────┘
│ (readonly user)
┌──────────┴──────────┐
▼ ▼
meshexplorer discord-bot
(web UI :3001) (optional --profile bot)
```
## Quick start
Requirements: Docker + Docker Compose.
```bash
cp .env.example .env
# Edit .env — at minimum set:
# CLICKHOUSE_PASSWORD (read/write user, used by ingest + migrations)
# MQTT_BROKERS (JSON array of meshcore MQTT brokers to ingest from)
# Optional, for the Discord relay: DISCORD_WEBHOOK_URL (+ run with --profile bot)
docker compose up --build
```
Then open <http://localhost:3001>.
Startup order is handled automatically: ClickHouse becomes healthy → `migrate`
applies the schema and exits → `meshcoreingest` and `meshexplorer` start.
To also run the Discord relay:
```bash
docker compose --profile bot up --build
```
## Configuration
All configuration is via environment variables in `.env` (see
[`.env.example`](./.env.example) for the full list and defaults). Highlights:
- **ClickHouse** — two accounts. The read/write `default` user
(`CLICKHOUSE_PASSWORD`) is used by the ingest daemon and the migration runner;
the `readonly` user (`CLICKHOUSE_READONLY_PASSWORD`) is used by the web app and
the Discord bot. ClickHouse is only published to `127.0.0.1` for debugging and
is otherwise reachable only on the internal `meshnet` network.
- **MQTT_BROKERS** — a JSON array; each entry is
`{ "url", "username", "password", "topics" }` (`topics` defaults to
`["meshcore/#"]`). The ingest daemon exits with a clear error if this is unset,
so configure at least one broker.
## Development
Each component can be run on its own:
- Web app: see [`meshexplorer/README.md`](./meshexplorer/README.md)
(`npm install && npm run dev`).
- Ingest: see [`ingest/README.md`](./ingest/README.md) (`go build ./...`).
## Security notes
- `.env` is gitignored — keep real credentials out of version control.
- If you previously used the bundled defaults, rotate any secrets before going
to production.
ingest/clickhouse/Dockerfile
+2 -21
View File
@@ -1,26 +1,7 @@
FROM clickhouse/clickhouse-server:latest as builder
RUN apt-get update && \
apt-get install -y build-essential libssl-dev libmsgpack-dev && \
rm -rf /var/lib/apt/lists/*
WORKDIR /build
COPY meshcore_decrypt_udf.c ./
RUN gcc -O2 -o meshcore_decrypt_udf meshcore_decrypt_udf.c -lssl -lcrypto -lmsgpackc
FROM clickhouse/clickhouse-server:latest
# Copy UDF binary to /usr/local/share/clickhouse-udf-bins
COPY --from=builder /build/meshcore_decrypt_udf /usr/local/share/clickhouse-udf-bins/meshcore_decrypt_udf
# Copy UDF XML config directly to the location and name matching ClickHouse config pattern
COPY meshcore_decrypt_udf.xml /etc/clickhouse-server/meshcore_decrypt_function.xml
# Copy users configuration file
# Read-only user used by the web app / discord bot
COPY users.xml /etc/clickhouse-server/users.d/readonly_user.xml
# Copy access control configuration
# Access control configuration
COPY config.xml /etc/clickhouse-server/config.d/access_control.xml
RUN chmod +x /usr/local/share/clickhouse-udf-bins/meshcore_decrypt_udf
# Remove custom entrypoint, use default
ingest/clickhouse/meshcore_decrypt_udf.c
-99
View File
@@ -1,99 +0,0 @@
#include <stdio.h>
#include <string.h>
#include <openssl/aes.h>
#include <openssl/hmac.h>
#include <openssl/sha.h>
#include <msgpack.h>
#define CIPHER_MAC_SIZE 2
#define CIPHER_KEY_SIZE 16
#define MAX_PAYLOAD 184
// Default "public" channel key
static const unsigned char public_channel_private_key[16] = {
0x8B, 0x33, 0x87, 0xE5, 0xCD, 0xEA, 0x6A, 0xC9,
0xE5, 0xED, 0xBA, 0xA1, 0x15, 0xCD, 0x72, 0x0F
};
static int decrypt_msg(const unsigned char *key, const unsigned char *data, size_t len, unsigned char *out, size_t *out_len) {
if (len <= CIPHER_MAC_SIZE) return 0;
unsigned char mac[SHA256_DIGEST_LENGTH];
HMAC(EVP_sha256(), key, CIPHER_KEY_SIZE, data + CIPHER_MAC_SIZE,
len - CIPHER_MAC_SIZE, mac, NULL);
if (memcmp(mac, data, CIPHER_MAC_SIZE) != 0) return 0;
AES_KEY aes_key;
AES_set_decrypt_key(key, 128, &aes_key);
size_t blocks = (len - CIPHER_MAC_SIZE) / 16;
for (size_t i = 0; i < blocks; i++) {
AES_decrypt(data + CIPHER_MAC_SIZE + (i * 16), out + (i * 16), &aes_key);
}
*out_len = blocks * 16;
return 1;
}
int main() {
unsigned char buf[8192];
size_t len = fread(buf, 1, sizeof(buf), stdin);
if (len == 0) return 0;
msgpack_unpacked result;
msgpack_unpacked_init(&result);
size_t off = 0;
msgpack_unpack_return ret = msgpack_unpack_next(&result, (char*)buf, len, &off);
if (ret == MSGPACK_UNPACK_SUCCESS && result.data.type == MSGPACK_OBJECT_ARRAY) {
msgpack_object_array arr = result.data.via.array;
msgpack_sbuffer sbuf;
msgpack_sbuffer_init(&sbuf);
msgpack_packer pk;
msgpack_packer_init(&pk, &sbuf, msgpack_sbuffer_write);
msgpack_pack_array(&pk, arr.size);
for (size_t i = 0; i < arr.size; i++) {
msgpack_object row = arr.ptr[i];
if (row.type == MSGPACK_OBJECT_ARRAY && row.via.array.size >= 3) {
msgpack_object mac_obj = row.via.array.ptr[0];
msgpack_object enc_obj = row.via.array.ptr[1];
msgpack_object hash_obj = row.via.array.ptr[2];
if (mac_obj.type == MSGPACK_OBJECT_STR && enc_obj.type == MSGPACK_OBJECT_STR && hash_obj.type == MSGPACK_OBJECT_STR && hash_obj.via.str.size > 0) {
// Concatenate mac + encrypted_data
size_t mac_len = mac_obj.via.str.size;
size_t enc_len = enc_obj.via.str.size;
unsigned char mac_and_enc[MAX_PAYLOAD];
if (mac_len + enc_len > MAX_PAYLOAD) { msgpack_pack_nil(&pk); continue; }
memcpy(mac_and_enc, mac_obj.via.str.ptr, mac_len);
memcpy(mac_and_enc + mac_len, enc_obj.via.str.ptr, enc_len);
unsigned char decrypted[MAX_PAYLOAD];
size_t decrypted_len;
if (decrypt_msg(public_channel_private_key, mac_and_enc, mac_len + enc_len, decrypted, &decrypted_len)) {
msgpack_pack_array(&pk, 2);
msgpack_pack_str(&pk, decrypted_len);
msgpack_pack_str_body(&pk, (char*)decrypted, decrypted_len);
unsigned char full_hash[SHA256_DIGEST_LENGTH];
SHA256(public_channel_private_key, CIPHER_KEY_SIZE, full_hash);
msgpack_pack_str(&pk, SHA256_DIGEST_LENGTH);
msgpack_pack_str_body(&pk, (char*)full_hash, SHA256_DIGEST_LENGTH);
} else {
msgpack_pack_nil(&pk);
}
} else {
msgpack_pack_nil(&pk);
}
} else {
msgpack_pack_nil(&pk);
}
}
fwrite(sbuf.data, 1, sbuf.size, stdout);
msgpack_sbuffer_destroy(&sbuf);
}
msgpack_unpacked_destroy(&result);
return 0;
}
ingest/clickhouse/meshcore_decrypt_udf.xml
-30
View File
@@ -1,30 +0,0 @@
<?xml version="1.0"?>
<clickhouse>
<function>
<name>meshcore_try_decrypt</name>
<type>executable</type>
<command>/usr/local/share/clickhouse-udf-bins/meshcore_decrypt_udf</command>
<argument>
<type>String</type>
<name>mac</name>
</argument>
<argument>
<type>String</type>
<name>encrypted_data</name>
</argument>
<argument>
<type>String</type>
<name>channel_hash_raw</name>
</argument>
<format>MsgPack</format>
<return_type>Nullable(Array(String))</return_type>
<return_name>result</return_name>
<execute_direct>1</execute_direct>
<max_command_execution_time>30</max_command_execution_time>
<command_termination_timeout>10</command_termination_timeout>
<command_read_timeout>10000</command_read_timeout>
<command_write_timeout>10000</command_write_timeout>
<send_chunk_header>false</send_chunk_header>
<deterministic>true</deterministic>
</function>
</clickhouse>
ingest/clickhouse/udf-config.xml
-4
View File
@@ -1,4 +0,0 @@
<clickhouse>
<user_defined_executable_functions_config>/usr/local/share/clickhouse-udfs/*.xml</user_defined_executable_functions_config>
<user_scripts_path>/usr/local/share/clickhouse-udf-bins/</user_scripts_path>
</clickhouse>
ingest/internal/migrate/main.go
+5
View File
@@ -47,6 +47,11 @@ func main() {
log.Fatalf("Failed to connect to ClickHouse: %v", err)
}
// ClickHouse is the target database; goose defaults to postgres otherwise.
if err := goose.SetDialect("clickhouse"); err != nil {
log.Fatalf("Failed to set goose dialect: %v", err)
}
// Set goose database
goose.SetBaseFS(os.DirFS(*migrationsPath))
meshexplorer/README.md
+14 -60
View File
@@ -1,9 +1,13 @@
# MeshExplorer
MeshExplorer is a real-time map, chat client, and packet analysis tool for mesh networks using MeshCore and Meshtastic. It enables users to visualize mesh nodes on a map, communicate via chat, and analyze packet data in real time.
MeshExplorer is a real-time map, chat client, and packet analysis tool for MeshCore mesh networks. It enables users to visualize mesh nodes on a map, communicate via chat, and analyze packet data in real time.
> This directory contains the web app only. For the full stack (ClickHouse,
> migrations, ingest, and this app) see the [root README](../README.md) and the
> unified `docker compose` setup.
## Features
- Real-time map of mesh network nodes (MeshCore and Meshtastic)
- Real-time map of mesh network nodes (MeshCore)
- Integrated chat client for mesh channels
- Packet analysis and inspection tools
- Customizable map layers and clustering
@@ -70,71 +74,21 @@ The middleware applies the following CORS headers to all `/api/*` routes:
## Learn More
- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
- [MeshCore](https://github.com/your-org/meshcore) - mesh network backend
- [Meshtastic](https://meshtastic.org/) - open source mesh communication project
- [MeshCore](https://meshcore.co.uk/) - mesh network firmware
## Docker Deployment
The application includes Docker support for easy deployment. The Docker configuration is set up to connect to ClickHouse running on the Docker host.
### Prerequisites
- Docker and Docker Compose installed
- ClickHouse running on the Docker host (default port 8123)
- External Docker network `shared-network` must exist (see setup instructions below)
### External Network Setup
The application requires an external Docker network called `shared-network` to communicate with ClickHouse. You must create this network before running the application:
This app is deployed as part of the unified stack at the repository root. From
the repo root:
```bash
docker network create shared-network
cp .env.example .env # then edit
docker compose up --build
```
**Note**: If the network already exists, this command will show an error but can be safely ignored.
### Running with Docker Compose
1. **Create the required external network (if not already created):**
```bash
docker network create shared-network
```
2. **Build and start the application:**
```bash
docker-compose up --build
```
3. **Access the application:**
Open [http://localhost:3001](http://localhost:3001) in your browser.
### Docker Configuration
The `docker-compose.yml` file is configured with:
- **Port mapping**: Container port 3000 → Host port 3001
- **ClickHouse connection**: Uses `clickhouse` hostname to connect to ClickHouse via the shared network
- **External network**: Requires `shared-network` to be created externally
- **Environment variables**: Pre-configured for typical ClickHouse setup
### Customizing ClickHouse Connection
You can customize the ClickHouse connection by modifying the environment variables in `docker-compose.yml`:
```yaml
environment:
- CLICKHOUSE_HOST=your-clickhouse-host
- CLICKHOUSE_PORT=8123
- CLICKHOUSE_USER=your-username
- CLICKHOUSE_PASSWORD=your-password
```
### Building with BuildKit
For faster builds with caching, enable BuildKit:
```bash
DOCKER_BUILDKIT=1 docker-compose up --build
```
The web app is served on <http://localhost:3001>. ClickHouse, migrations, and the
MeshCore ingest are started for you. See the [root README](../README.md) for the
full list of services and environment variables.
## Deploy