iarv/meshcore-hub
1
0
Fork 1
mirror of https://github.com/ipnet-mesh/meshcore-hub.git synced 2026-03-28 17:42:56 +01:00
Code Issues Packages Projects Releases Wiki Activity

Updates

Browse Source
This commit is contained in:
Louis King
2025-12-02 22:43:19 +00:00
parent 66d3a9298f
commit 0ccf7619d1
2 changed files with 601 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

108
PROMPT.md Normal file
View File

@@ -0,0 +1,108 @@
# MeshCore Hub
Python 3.11+ project for managing and orchestrating MeshCore networks.
## Repo
Monorepo structure with the following packages:
- `meshcore_interface`: Component to interface with MeshCore companion nodes over Serial/USB and publish/subscribe to MQTT broker
- `meshcore_collector`: Component to collect and store MeshCore events from MQTT broker into a database
- `meshcore_api`: REST API to query collected data and send commands to the MeshCore network via MQTT broker
- `meshcore_web`: Frontend web dashboard for visualizing MeshCore network status and statistics
- `meshcore_common`: Shared utilities, models and configurations used by all components
Project configuration should use Pydantic for settings management, with all options available via environment variables for easy Docker deployment as well as command-line arguments.
Docker image contains all components, with entrypoint to select which component to run. Docker volumes for persistent storage of database.
MeshCore events and schemas are defined in [SCHEMAS.md](SCHEMAS.md) for reference.
## Dependencies
- Python 3.11+ (pyproject configured for 3.11)
- All development in virtual environments (`.venv`)
- Use `pip` for package management
- Use `black` for code formatting
- Use `flake8` for linting
- Use `mypy` for type checking
- Use `pytest` for testing
- Pre-commit hooks for code quality
- Click for CLI interfaces
- Pydantic for data validation and settings management
- SQLAlchemy for database interactions
- Alembic for database migrations
- FastAPI for REST API
- MQTT client library (e.g. `paho-mqtt` or `gmqtt`)
- meshcore_py for MeshCore device interactions
- Docker for containerization
- Single Dockerfile for all components
- Docker Compose for multi-container orchestration
## Testing
`meshcore_interface` should also include a mocked MeshCore device for testing purposes, allowing simulation of various events and conditions without requiring physical hardware. This should be enabled by a command-line flag or equivalent and will facilitate unit and integration testing without the need for actual MeshCore devices.
## Project Components
### Interface
This component interacts with a MeshCore companion node over Serial/USB. It should subscribe to all events provided by the [meshcore_py](https://github.com/meshcore-dev/meshcore_py) Python library. It should also support sending a selection of commands to the companion node, primarily sending messages to contacts or channels, and also sending node advertisments.
This component should run in one of two modes:
- RECEIVER: The component subscribes to all MeshCore events and then publishes them to an MQTT message broker
- SENDER: The component subscribes to an MQTT message broker and sends commands to the MeshCore companion node
The `meshcore_py` library provides a method of querying the connected MeshCore devices "public address" (64 character hex string) which uniquely identifies any MeshCore node on the network. The public key should be the primary identifier for the node in all communications.
Both sender and receiver modes should use a common MQTT topic structure including a customisable prefix, the nodes public address, and the event/command name. For example:
```
<prefix>/<public_address>/event/<event_name>
<prefix>/<public_address>/command/<command
```
The service can only be started in one mode at a time, either SENDER or RECEIVER. A typical MeshCore network might have several receiver nodes distributed around a location, all publishing events to a central MQTT broker, and then a single sender node which subscribes to the broker and sends commands to the network. There should only be one sender node in a network at any time to avoid message duplication.
### Collector
This service subscribes to the MQTT broker and stores all received events in a database for later retrieval. All relevant MeshCore events should be persisted including messages, node advertisments, trace data responses and include the node that provided them (public address in MQTT topic). The data should be persisted using SQLAlchemy to allow for flexibility in database backend (SQLite, Postgres, MySQL etc), but use SQLite as the default backend for simplicity. We should also support database migrations using Alembic from the outset.
Nodes should also be tracked in the database, with their latest known information (name, last seen timestamp etc) updated whenever a node advertisment is received. There should also be a Node Tag model to allow users to assign custom tags/labels to nodes for easier identification, using the public key as a foreign key. This will allow users to add arbitrary metadata to nodes without modifying the core node data.
### API
This service should provide a REST API (using FastAPI) to allow clients to retrieve stored data from the database used by the Collector. The API would need to have access to the same database as the Collector.
The API should support OpenAPI/Swagger documentation for easy exploration and testing. The API should also support optional HTTP bearer token authentication to restrict access to authorized users only. There should be two API keys, one that only allows query access, and another that allows full query and command access.
The API should support retrieving messages by various filters including sender, receiver, channel, date range etc. It should also support retrieving node advertisments and trace data. The API should also provide endpoints to manage Node Tags, allowing users to create, read, update and delete tags associated with nodes.
The API should also provide endpoints to send commands to the network, such as sending messages or advertisements. These commands would be published to the MQTT broker for the SENDER Interface component to pick up and execute. The API should support the same "prefix" configuration as the Interface component to ensure it publishes to the correct topics. The API would publish to a wildcard MQTT topic to allow for multiple sender nodes if required, e.g. `<prefix>/+/command/<command>`, but ideally there should only be one sender node active at any time.
The API should also provide a basic dashboard endpoint that provides summary statistics about the MeshCore network, such as number of nodes, number of messages sent/received, active channels etc. This would provide a quick overview of the network status without needing to query individual endpoints. This should use simple HTML templates served by FastAPI for easy access via web browsers. Styling should use simple CSS, no JavaScript required.
### Web Dashboard
This component provides a more user-friendly web interface for visualizing the MeshCore network status and statistics. It should connect to the API component to retrieve data and display it in an intuitive manner. The dashboard should include the following views:
- Front Page: Customisable welcome page with network name, details and radio configuration.
- Members List: List of all network member profiles (read from static JSON file)
- Network Overview: Summary statistics about the MeshCore network, including number of nodes, messages, advertisements etc.
- Node List: A list of all known nodes with their details and tags. Ability to filter and search nodes.
- Node Map: A visual map showing the locations of nodes based on latitude/longitude from node tags.
- Message Log: A log of all messages sent/received on the network with filtering options.
The following should be configurable via environment variables or command-line arguments:
- NETWORK_DOMAIN: Domain name for the web dashboard (e.g. "meshcore.example.com")
- NETWORK_NAME: Name of the MeshCore network
- NETWORK_CITY: Town/City where the network is located (e.g. "Ipswich")
- NETWORK_COUNTRY: Country where the network is located (ISO 3166-1 alpha-2 code)
- NETWORK_LOCATION: Latitude/Longitude of the network area location
- NETWORK_RADIO_CONFIG: Details about the radio configuration (frequency, power etc)
- NETWORK_CONTACT_EMAIL: Contact email address for network enquiries
- NETWORK_CONTACT_DISCORD: Discord server link for network community
The web dashboard should be a multi-page application using server-side rendering with FastAPI templates. It should use Tailwind CSS for styling and a modern UI component library (DaisyUI or similar) for consistent design. No JavaScript frameworks are required, any JS should be minimal and only for enhancing user experience (e.g. form validation, interactivity).

493
SCHEMAS.md Normal file
View File

@@ -0,0 +1,493 @@
# MeshCore Event Schemas
This document defines the complete JSON payload schemas for all MeshCore events supported by the API.
## Event Categories
Events are categorized by how they're handled:
- **Persisted Events**: Stored in database tables and available via REST API
- **Webhook Events**: Trigger HTTP POST notifications when configured
- **Informational Events**: Logged but not persisted to separate tables
## Table of Contents
- [Persisted & Webhook Events](#persisted--webhook-events)
- [ADVERTISEMENT / NEW_ADVERT](#advertisement--new_advert)
- [CONTACT_MSG_RECV](#contact_msg_recv)
- [CHANNEL_MSG_RECV](#channel_msg_recv)
- [Persisted Events (Non-Webhook)](#persisted-events-non-webhook)
- [TRACE_DATA](#trace_data)
- [TELEMETRY_RESPONSE](#telemetry_response)
- [CONTACTS](#contacts)
- [Informational Events](#informational-events)
- [SEND_CONFIRMED](#send_confirmed)
- [STATUS_RESPONSE](#status_response)
- [BATTERY](#battery)
- [PATH_UPDATED](#path_updated)
- [Webhook Payload Format](#webhook-payload-format)
---
## Persisted & Webhook Events
These events are both stored in the database and trigger webhooks when configured.
### ADVERTISEMENT / NEW_ADVERT
Node advertisements announcing presence and metadata.
**Database Table**: `advertisements`
**Payload Schema**:
```json
{
"public_key": "string (64 hex chars)",
"name": "string (optional)",
"adv_type": "string (optional)",
"flags": "integer (optional)"
}
```
**Field Descriptions**:
- `public_key`: Node's full 64-character hexadecimal public key (required)
- `name`: Node name/alias (e.g., "Gateway-01", "Alice")
- `adv_type`: Node type - one of: `"chat"`, `"repeater"`, `"room"`, `"none"`
- `flags`: Node capability/status flags (bitmask)
**Example**:
```json
{
"public_key": "4767c2897c256df8d85a5fa090574284bfd15b92d47359741b0abd5098ed30c4",
"name": "Gateway-01",
"adv_type": "repeater",
"flags": 218
}
```
**Webhook Trigger**: Yes
**REST API**: `GET /api/v1/advertisements`
---
### CONTACT_MSG_RECV
Direct/private messages between two nodes.
**Database Table**: `messages`
**Payload Schema**:
```json
{
"pubkey_prefix": "string (12 chars)",
"path_len": "integer (optional)",
"txt_type": "integer (optional)",
"signature": "string (optional)",
"text": "string",
"SNR": "number (optional)",
"sender_timestamp": "integer (optional)"
}
```
**Field Descriptions**:
- `pubkey_prefix`: First 12 characters of sender's public key
- `path_len`: Number of hops message traveled
- `txt_type`: Message type indicator (0=plain, 2=signed, etc.)
- `signature`: Message signature (8 hex chars) when `txt_type=2`
- `text`: Message content (required)
- `SNR`: Signal-to-Noise Ratio in dB
- `sender_timestamp`: Unix timestamp when message was sent
**Example**:
```json
{
"pubkey_prefix": "01ab2186c4d5",
"path_len": 3,
"txt_type": 0,
"signature": null,
"text": "Hello Bob!",
"SNR": 15.5,
"sender_timestamp": 1732820498
}
```
**Webhook Trigger**: Yes
**REST API**: `GET /api/v1/messages`
**Webhook JSONPath Examples**:
- Send only text: `$.data.text`
- Send text + SNR: `$.data.[text,SNR]`
---
### CHANNEL_MSG_RECV
Group/broadcast messages on specific channels.
**Database Table**: `messages`
**Payload Schema**:
```json
{
"channel_idx": "integer",
"path_len": "integer (optional)",
"txt_type": "integer (optional)",
"signature": "string (optional)",
"text": "string",
"SNR": "number (optional)",
"sender_timestamp": "integer (optional)"
}
```
**Field Descriptions**:
- `channel_idx`: Channel number (0-255)
- `path_len`: Number of hops message traveled
- `txt_type`: Message type indicator (0=plain, 2=signed, etc.)
- `signature`: Message signature (8 hex chars) when `txt_type=2`
- `text`: Message content (required)
- `SNR`: Signal-to-Noise Ratio in dB
- `sender_timestamp`: Unix timestamp when message was sent
**Example**:
```json
{
"channel_idx": 4,
"path_len": 10,
"txt_type": 0,
"signature": null,
"text": "Hello from the mesh!",
"SNR": 8.5,
"sender_timestamp": 1732820498
}
```
**Webhook Trigger**: Yes
**REST API**: `GET /api/v1/messages`
**Webhook JSONPath Examples**:
- Send only text: `$.data.text`
- Send channel + text: `$.data.[channel_idx,text]`
---
## Persisted Events (Non-Webhook)
These events are stored in the database but do not trigger webhooks.
### TRACE_DATA
Network trace path results showing route and signal strength.
**Database Table**: `trace_paths`
**Payload Schema**:
```json
{
"initiator_tag": "integer",
"path_len": "integer (optional)",
"flags": "integer (optional)",
"auth": "integer (optional)",
"path_hashes": "array of strings",
"snr_values": "array of numbers",
"hop_count": "integer (optional)"
}
```
**Field Descriptions**:
- `initiator_tag`: Unique trace identifier (0-4294967295)
- `path_len`: Length of the path
- `flags`: Trace flags/options
- `auth`: Authentication/validation data
- `path_hashes`: Array of 2-character node hash identifiers (ordered by hops)
- `snr_values`: Array of SNR values corresponding to each hop
- `hop_count`: Total number of hops
**Example**:
```json
{
"initiator_tag": 123456789,
"path_len": 3,
"flags": 0,
"auth": 1,
"path_hashes": ["4a", "b3", "fa"],
"snr_values": [25.3, 18.7, 12.4],
"hop_count": 3
}
```
**Webhook Trigger**: No
**REST API**: `GET /api/v1/trace-paths`
---
### TELEMETRY_RESPONSE
Sensor data from network nodes (temperature, humidity, battery, etc.).
**Database Table**: `telemetry`
**Payload Schema**:
```json
{
"node_public_key": "string (64 hex chars)",
"lpp_data": "bytes (optional)",
"parsed_data": "object (optional)"
}
```
**Field Descriptions**:
- `node_public_key`: Full public key of the reporting node
- `lpp_data`: Raw LPP-encoded sensor data (CayenneLPP format)
- `parsed_data`: Decoded sensor readings as key-value pairs
**Example**:
```json
{
"node_public_key": "4767c2897c256df8d85a5fa090574284bfd15b92d47359741b0abd5098ed30c4",
"lpp_data": null,
"parsed_data": {
"temperature": 22.5,
"humidity": 65,
"battery": 3.8,
"pressure": 1013.25
}
}
```
**Webhook Trigger**: No
**REST API**: `GET /api/v1/telemetry`
---
### CONTACTS
Contact sync event containing all known nodes.
**Database Table**: Updates `nodes` table
**Payload Schema**:
```json
{
"contacts": [
{
"public_key": "string (64 hex chars)",
"name": "string (optional)",
"node_type": "string (optional)"
}
]
}
```
**Field Descriptions**:
- `contacts`: Array of contact objects
- `public_key`: Node's full public key
- `name`: Node name/alias
- `node_type`: One of: `"chat"`, `"repeater"`, `"room"`, `"none"`
**Example**:
```json
{
"contacts": [
{
"public_key": "01ab2186c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1",
"name": "Alice",
"node_type": "chat"
},
{
"public_key": "b3f4e5d6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4",
"name": "Bob",
"node_type": "chat"
}
]
}
```
**Webhook Trigger**: No
**REST API**: `GET /api/v1/nodes`
---
## Informational Events
These events are logged to `events_log` table but not persisted to separate tables.
### SEND_CONFIRMED
Confirmation that a sent message was delivered.
**Payload Schema**:
```json
{
"destination_public_key": "string (64 hex chars)",
"round_trip_ms": "integer"
}
```
**Field Descriptions**:
- `destination_public_key`: Recipient's full public key
- `round_trip_ms`: Round-trip time in milliseconds
**Example**:
```json
{
"destination_public_key": "4767c2897c256df8d85a5fa090574284bfd15b92d47359741b0abd5098ed30c4",
"round_trip_ms": 2500
}
```
---
### STATUS_RESPONSE
Device status information.
**Payload Schema**:
```json
{
"node_public_key": "string (64 hex chars)",
"status": "string (optional)",
"uptime": "integer (optional)",
"message_count": "integer (optional)"
}
```
**Field Descriptions**:
- `node_public_key`: Node's full public key
- `status`: Status description
- `uptime`: Uptime in seconds
- `message_count`: Total messages processed
**Example**:
```json
{
"node_public_key": "4767c2897c256df8d85a5fa090574284bfd15b92d47359741b0abd5098ed30c4",
"status": "operational",
"uptime": 86400,
"message_count": 1523
}
```
---
### BATTERY
Battery status information.
**Payload Schema**:
```json
{
"battery_voltage": "number",
"battery_percentage": "integer"
}
```
**Field Descriptions**:
- `battery_voltage`: Battery voltage (e.g., 3.7V)
- `battery_percentage`: Battery level 0-100%
**Example**:
```json
{
"battery_voltage": 3.8,
"battery_percentage": 75
}
```
---
### PATH_UPDATED
Notification that routing path to a node has changed.
**Payload Schema**:
```json
{
"node_public_key": "string (64 hex chars)",
"hop_count": "integer"
}
```
**Field Descriptions**:
- `node_public_key`: Target node's full public key
- `hop_count`: Number of hops in new path
**Example**:
```json
{
"node_public_key": "4767c2897c256df8d85a5fa090574284bfd15b92d47359741b0abd5098ed30c4",
"hop_count": 3
}
```
---
### Other Informational Events
The following events are logged but have varying or device-specific payloads:
- **STATISTICS**: Network statistics (varies by implementation)
- **DEVICE_INFO**: Device hardware/firmware information
- **BINARY_RESPONSE**: Binary data responses
- **CONTROL_DATA**: Control/command responses
- **RAW_DATA**: Raw protocol data
- **NEXT_CONTACT**: Contact enumeration progress
- **ERROR**: Error messages with description
- **MESSAGES_WAITING**: Notification of queued messages
- **NO_MORE_MSGS**: End of message queue
- **RX_LOG_DATA**: Receive log data
---
## Webhook Payload Format
All webhook events are wrapped in a standard envelope before being sent:
```json
{
"event_type": "string",
"timestamp": "string (ISO 8601)",
"data": {
// Event-specific payload (see schemas above)
}
}
```
**Example (Channel Message)**:
```json
{
"event_type": "CHANNEL_MSG_RECV",
"timestamp": "2025-11-28T19:41:38.748379Z",
"data": {
"channel_idx": 4,
"path_len": 10,
"txt_type": 0,
"text": "Hello from the mesh!",
"SNR": 8.5,
"sender_timestamp": 1732820498
}
}
```
### JSONPath Filtering
You can filter webhook payloads using JSONPath expressions:
| JSONPath | Result |
|----------|--------|
| `$` | Full payload (default) |
| `$.data` | Event data only |
| `$.data.text` | Message text only |
| `$.data.[text,SNR]` | Multiple fields as array |
| `$.event_type` | Event type string |
| `$.timestamp` | Timestamp string |
See [AGENTS.md](AGENTS.md) for webhook configuration details.
---
## Event Flow
1. **Hardware/Mock MeshCore** → Generates raw events
2. **Event Handler** → Processes and persists to database
3. **Webhook Handler** → Sends HTTP POST to configured URLs (if enabled)
4. **REST API** → Query historical data from database
Most events are logged to the `events_log` table with full payloads for debugging and audit purposes. Some high-frequency informational events (e.g., `NEXT_CONTACT`) are intentionally excluded from logging to reduce database size.