meshcore-hub/PLAN.md

MeshCore Hub - Implementation Plan

Project Overview

MeshCore Hub is a Python 3.11+ monorepo for managing and orchestrating MeshCore mesh networks. It consists of five main components that work together to receive, store, query, and visualize mesh network data.

---

Questions Requiring Clarification

Before implementation, the following questions need answers:

Architecture & Design

1. MQTT Broker Selection: Should we include an embedded MQTT broker (like hbmqtt) or assume an external broker (Mosquitto) is always used? The spec mentions Docker but doesn't specify broker deployment.

2. Database Deployment: For production, should we support PostgreSQL/MySQL from the start, or focus on SQLite initially and add other backends later? This affects connection pooling and async driver choices.

3. Web Dashboard Separation: Should meshcore_web be a separate FastAPI app or integrated into meshcore_api? Running two FastAPI apps adds deployment complexity.

4. Member Profiles JSON Location: Where should the static JSON file for member profiles be stored? In the config directory, as a mounted volume, or embedded in the package?

Interface Component

5. Multiple Serial Devices: Should a single Interface instance support multiple serial devices simultaneously, or should users run multiple instances (one per device)?

6. Reconnection Strategy: What should happen when the serial connection is lost? Automatic reconnect with backoff, or exit and let the container orchestrator restart?

7. Mock Device Scope: How comprehensive should the mock MeshCore device be? Should it simulate realistic timing, packet loss, and network topology?

Collector Component

8. Event Deduplication: How should we handle duplicate events from multiple receiver nodes? By message signature/hash, or accept all duplicates?

9. Data Retention Policy: Should we implement automatic data pruning (e.g., delete messages older than X days)? This affects long-running deployments.

10. Webhook Configuration: The SCHEMAS.md mentions webhooks but PROMPT.md doesn't detail webhook management. Should webhooks be configured via API, config file, or environment variables?

API Component

11. API Key Management: How should API keys be generated and managed? Static config, database-stored, or runtime generation? What about key rotation?

12. Rate Limiting: Should the API implement rate limiting? If so, what defaults?

13. CORS Configuration: Should CORS be configurable for web dashboard access from different domains?

Web Dashboard

14. Authentication: Should the web dashboard have its own authentication, or rely on API bearer tokens? What about session management?

15. Real-time Updates: Should the dashboard support real-time updates (WebSocket/SSE), or polling-based refresh?

16. Map Provider: Which map provider for the Node Map view? OpenStreetMap/Leaflet (free) or allow configuration for commercial providers?

Node Tags System

17. Tag Value Types: Should node tags support typed values (string, number, boolean, coordinates) or just strings? The spec mentions lat/lon for the map feature.

18. Reserved Tag Names: Should certain tag names be reserved for system use (e.g., location, description)?

DevOps & Operations

19. Health Checks: What health check endpoints should each component expose for Docker/Kubernetes?

20. Metrics/Observability: Should we include Prometheus metrics endpoints or structured logging for observability?

21. Log Level Configuration: Per-component or global log level configuration?

---

Proposed Architecture

                                    +------------------+
                                    |   MQTT Broker    |
                                    |   (Mosquitto)    |
                                    +--------+---------+
                                             |
              +------------------------------+------------------------------+
              |                              |                              |
              v                              v                              v
    +---------+---------+          +---------+---------+          +---------+---------+
    | meshcore_interface|          | meshcore_interface|          | meshcore_interface|
    |    (RECEIVER)     |          |    (RECEIVER)     |          |    (SENDER)       |
    +-------------------+          +-------------------+          +-------------------+
              |                              |                              ^
              |  Publishes events            |                              |
              v                              v                              |
    +----------------------------------------------------------+           |
    |                      MQTT Topics                          |           |
    |  <prefix>/<pubkey>/event/<event_name>                    |           |
    |  <prefix>/+/command/<command_name>  <--------------------+-----------+
    +----------------------------------------------------------+
                              |
                              v
                    +---------+---------+
                    | meshcore_collector|
                    |   (Subscriber)    |
                    +---------+---------+
                              |
                              v
                    +---------+---------+
                    |     Database      |
                    |  (SQLite/Postgres)|
                    +---------+---------+
                              ^
                              |
              +---------------+---------------+
              |                               |
    +---------+---------+           +---------+---------+
    |   meshcore_api    |           |   meshcore_web    |
    |    (REST API)     |           |   (Dashboard)     |
    +-------------------+           +-------------------+

---

Package Structure

meshcore-hub/
├── pyproject.toml                 # Root project configuration
├── README.md
├── PROMPT.md
├── SCHEMAS.md
├── PLAN.md
├── .env.example                   # Example environment variables
├── .pre-commit-config.yaml        # Pre-commit hooks
├── alembic.ini                    # Alembic configuration
├── alembic/                       # Database migrations
│   ├── env.py
│   ├── versions/
│   └── script.py.mako
├── docker/
│   ├── Dockerfile
│   └── docker-compose.yml
├── tests/
│   ├── conftest.py
│   ├── test_interface/
│   ├── test_collector/
│   ├── test_api/
│   ├── test_web/
│   └── test_common/
└── src/
    └── meshcore_hub/
        ├── __init__.py
        ├── __main__.py            # CLI entrypoint
        ├── common/
        │   ├── __init__.py
        │   ├── config.py          # Pydantic settings
        │   ├── models/            # SQLAlchemy models
        │   │   ├── __init__.py
        │   │   ├── base.py
        │   │   ├── node.py
        │   │   ├── message.py
        │   │   ├── advertisement.py
        │   │   ├── trace_path.py
        │   │   ├── telemetry.py
        │   │   ├── node_tag.py
        │   │   └── event_log.py
        │   ├── schemas/           # Pydantic schemas (API request/response)
        │   │   ├── __init__.py
        │   │   ├── events.py
        │   │   ├── nodes.py
        │   │   ├── messages.py
        │   │   └── commands.py
        │   ├── mqtt.py            # MQTT client utilities
        │   ├── database.py        # Database session management
        │   └── logging.py         # Logging configuration
        ├── interface/
        │   ├── __init__.py
        │   ├── cli.py             # Click CLI for interface
        │   ├── receiver.py        # RECEIVER mode implementation
        │   ├── sender.py          # SENDER mode implementation
        │   ├── device.py          # MeshCore device wrapper
        │   └── mock_device.py     # Mock device for testing
        ├── collector/
        │   ├── __init__.py
        │   ├── cli.py             # Click CLI for collector
        │   ├── subscriber.py      # MQTT subscriber
        │   ├── handlers/          # Event handlers
        │   │   ├── __init__.py
        │   │   ├── message.py
        │   │   ├── advertisement.py
        │   │   ├── trace.py
        │   │   ├── telemetry.py
        │   │   └── contacts.py
        │   └── webhook.py         # Webhook dispatcher
        ├── api/
        │   ├── __init__.py
        │   ├── cli.py             # Click CLI for API
        │   ├── app.py             # FastAPI application
        │   ├── dependencies.py    # FastAPI dependencies
        │   ├── auth.py            # Bearer token authentication
        │   ├── routes/
        │   │   ├── __init__.py
        │   │   ├── messages.py
        │   │   ├── nodes.py
        │   │   ├── advertisements.py
        │   │   ├── trace_paths.py
        │   │   ├── telemetry.py
        │   │   ├── node_tags.py
        │   │   ├── commands.py
        │   │   └── dashboard.py   # Simple HTML dashboard
        │   └── templates/
        │       └── dashboard.html
        └── web/
            ├── __init__.py
            ├── cli.py             # Click CLI for web dashboard
            ├── app.py             # FastAPI application
            ├── routes/
            │   ├── __init__.py
            │   ├── home.py
            │   ├── members.py
            │   ├── network.py
            │   ├── nodes.py
            │   ├── map.py
            │   └── messages.py
            ├── templates/
            │   ├── base.html
            │   ├── home.html
            │   ├── members.html
            │   ├── network.html
            │   ├── nodes.html
            │   ├── map.html
            │   └── messages.html
            └── static/
                ├── css/
                └── js/

---

Database Schema

Core Tables

nodes

Column	Type	Description
id	UUID	Primary key
public_key	VARCHAR(64)	Unique, indexed
name	VARCHAR(255)	Node display name
adv_type	VARCHAR(20)	chat, repeater, room, none
flags	INTEGER	Capability flags
first_seen	TIMESTAMP	First advertisement
last_seen	TIMESTAMP	Most recent activity
created_at	TIMESTAMP	Record creation
updated_at	TIMESTAMP	Record update

node_tags

Column	Type	Description
id	UUID	Primary key
node_id	UUID	FK to nodes
key	VARCHAR(100)	Tag name
value	TEXT	Tag value (JSON for typed values)
value_type	VARCHAR(20)	string, number, boolean, coordinate
created_at	TIMESTAMP	Record creation
updated_at	TIMESTAMP	Record update

Unique constraint on (node_id, key)

messages

Column	Type	Description
id	UUID	Primary key
receiver_node_id	UUID	FK to nodes (receiving interface)
message_type	VARCHAR(20)	contact, channel
pubkey_prefix	VARCHAR(12)	Sender prefix (contact msgs)
channel_idx	INTEGER	Channel index (channel msgs)
text	TEXT	Message content
path_len	INTEGER	Hop count
txt_type	INTEGER	Message type indicator
signature	VARCHAR(8)	Message signature
snr	FLOAT	Signal-to-noise ratio
sender_timestamp	TIMESTAMP	Sender's timestamp
received_at	TIMESTAMP	When received by interface
created_at	TIMESTAMP	Record creation

advertisements

Column	Type	Description
id	UUID	Primary key
receiver_node_id	UUID	FK to nodes (receiving interface)
node_id	UUID	FK to nodes (advertised node)
public_key	VARCHAR(64)	Advertised public key
name	VARCHAR(255)	Advertised name
adv_type	VARCHAR(20)	Node type
flags	INTEGER	Capability flags
received_at	TIMESTAMP	When received
created_at	TIMESTAMP	Record creation

trace_paths

Column	Type	Description
id	UUID	Primary key
receiver_node_id	UUID	FK to nodes
initiator_tag	BIGINT	Trace identifier
path_len	INTEGER	Path length
flags	INTEGER	Trace flags
auth	INTEGER	Auth data
path_hashes	JSON	Array of node hashes
snr_values	JSON	Array of SNR values
hop_count	INTEGER	Total hops
received_at	TIMESTAMP	When received
created_at	TIMESTAMP	Record creation

telemetry

Column	Type	Description
id	UUID	Primary key
receiver_node_id	UUID	FK to nodes
node_id	UUID	FK to nodes (reporting node)
node_public_key	VARCHAR(64)	Reporting node key
lpp_data	BYTEA	Raw LPP data
parsed_data	JSON	Decoded sensor readings
received_at	TIMESTAMP	When received
created_at	TIMESTAMP	Record creation

events_log

Column	Type	Description
id	UUID	Primary key
receiver_node_id	UUID	FK to nodes
event_type	VARCHAR(50)	Event type name
payload	JSON	Full event payload
received_at	TIMESTAMP	When received
created_at	TIMESTAMP	Record creation

---

MQTT Topic Structure

Event Topics (Published by RECEIVER)

meshcore/<public_key>/event/advertisement
meshcore/<public_key>/event/contact_msg_recv
meshcore/<public_key>/event/channel_msg_recv
meshcore/<public_key>/event/trace_data
meshcore/<public_key>/event/telemetry_response
meshcore/<public_key>/event/contacts
meshcore/<public_key>/event/send_confirmed
meshcore/<public_key>/event/status_response
meshcore/<public_key>/event/battery
meshcore/<public_key>/event/path_updated

Command Topics (Subscribed by SENDER)

meshcore/+/command/send_msg
meshcore/+/command/send_channel_msg
meshcore/+/command/send_advert
meshcore/+/command/request_status
meshcore/+/command/request_telemetry

Command Payloads

send_msg

{
  "destination": "public_key or pubkey_prefix",
  "text": "message content",
  "timestamp": 1732820498
}

send_channel_msg

{
  "channel_idx": 4,
  "text": "message content",
  "timestamp": 1732820498
}

send_advert

{
  "flood": true
}

---

API Endpoints

Authentication

• Bearer token in Authorization header
• Two token levels: read (query only) and admin (query + commands)

Nodes

Method	Endpoint	Auth	Description
GET	/api/v1/nodes	read	List all nodes with pagination/filtering
GET	/api/v1/nodes/{public_key}	read	Get single node details

Node Tags

Method	Endpoint	Auth	Description
GET	/api/v1/nodes/{public_key}/tags	read	List node's tags
POST	/api/v1/nodes/{public_key}/tags	admin	Create tag
PUT	/api/v1/nodes/{public_key}/tags/{key}	admin	Update tag
DELETE	/api/v1/nodes/{public_key}/tags/{key}	admin	Delete tag

Messages

Method	Endpoint	Auth	Description
GET	/api/v1/messages	read	List messages with filters
GET	/api/v1/messages/{id}	read	Get single message

Query Parameters:

• type: contact, channel
• pubkey_prefix: Filter by sender
• channel_idx: Filter by channel
• since: Start timestamp
• until: End timestamp
• limit, offset: Pagination

Advertisements

Method	Endpoint	Auth	Description
GET	/api/v1/advertisements	read	List advertisements
GET	/api/v1/advertisements/{id}	read	Get single advertisement

Trace Paths

Method	Endpoint	Auth	Description
GET	/api/v1/trace-paths	read	List trace paths
GET	/api/v1/trace-paths/{id}	read	Get single trace path

Telemetry

Method	Endpoint	Auth	Description
GET	/api/v1/telemetry	read	List telemetry data
GET	/api/v1/telemetry/{id}	read	Get single telemetry record

Commands

Method	Endpoint	Auth	Description
POST	/api/v1/commands/send-message	admin	Send direct message
POST	/api/v1/commands/send-channel-message	admin	Send channel message
POST	/api/v1/commands/send-advertisement	admin	Send advertisement

Dashboard

Method	Endpoint	Auth	Description
GET	/api/v1/dashboard	read	HTML dashboard page
GET	/api/v1/stats	read	JSON statistics

---

Configuration (Environment Variables)

Common

Variable	Default	Description
DATA_HOME	./data	Base directory for service data
LOG_LEVEL	INFO	Logging level
MQTT_HOST	localhost	MQTT broker host
MQTT_PORT	1883	MQTT broker port
MQTT_USERNAME		MQTT username (optional)
MQTT_PASSWORD		MQTT password (optional)
MQTT_PREFIX	meshcore	Topic prefix

Data Directory Structure

The DATA_HOME environment variable controls where all service data is stored:

${DATA_HOME}/
├── collector/
│   ├── meshcore.db    # SQLite database
│   └── tags.json      # Node tags for import
└── web/
    └── members.json   # Network members list

Interface

Variable	Default	Description
INTERFACE_MODE	RECEIVER	RECEIVER or SENDER
SERIAL_PORT	/dev/ttyUSB0	Serial port path
SERIAL_BAUD	115200	Baud rate
MESHCORE_DEVICE_NAME	(none)	Device/node name set on startup
MOCK_DEVICE	false	Use mock device

Collector

Variable	Default	Description
DATABASE_URL	sqlite:///{DATA_HOME}/collector/meshcore.db	SQLAlchemy URL
TAGS_FILE	{DATA_HOME}/collector/tags.json	Path to tags JSON file

API

Variable	Default	Description
API_HOST	0.0.0.0	API bind host
API_PORT	8000	API bind port
API_READ_KEY		Read-only API key
API_ADMIN_KEY		Admin API key
DATABASE_URL	sqlite:///{DATA_HOME}/collector/meshcore.db	SQLAlchemy URL

Web Dashboard

Variable	Default	Description
WEB_HOST	0.0.0.0	Web bind host
WEB_PORT	8080	Web bind port
API_BASE_URL	http://localhost:8000	API endpoint
API_KEY		API key for queries
WEB_LOCALE	en	UI translation locale
WEB_DATETIME_LOCALE	en-US	Date formatting locale for UI timestamps
TZ	UTC	Timezone used for UI timestamp rendering
NETWORK_DOMAIN		Network domain
NETWORK_NAME	MeshCore Network	Network name
NETWORK_CITY		City location
NETWORK_COUNTRY		Country code
NETWORK_LOCATION		Lat,Lon
NETWORK_RADIO_CONFIG		Radio config details
NETWORK_CONTACT_EMAIL		Contact email
NETWORK_CONTACT_DISCORD		Discord link
MEMBERS_FILE	{DATA_HOME}/web/members.json	Path to members JSON

---

Implementation Phases

Phase 1: Foundation

1. Set up project structure with pyproject.toml
2. Configure development tools (black, flake8, mypy, pytest)
3. Set up pre-commit hooks
4. Implement meshcore_common:
   • Pydantic settings/config
   • SQLAlchemy models
   • Database connection management
   • MQTT client utilities
   • Logging configuration
5. Set up Alembic for migrations
6. Create initial migration

Phase 2: Interface Component

1. Implement MeshCore device wrapper
2. Implement RECEIVER mode:
   • Event subscription
   • MQTT publishing
3. Implement SENDER mode:
   • MQTT subscription
   • Command dispatching
4. Implement mock device for testing
5. Create Click CLI
6. Write unit tests

Phase 3: Collector Component

1. Implement MQTT subscriber
2. Implement event handlers for each event type
3. Implement database persistence
4. Create Click CLI
5. Write unit tests

Phase 4: API Component

1. Set up FastAPI application
2. Implement authentication middleware
3. Implement all REST endpoints
4. Implement MQTT command publishing
5. Implement simple HTML dashboard
6. Add OpenAPI documentation
7. Create Click CLI
8. Write unit and integration tests

Phase 5: Web Dashboard

1. Set up FastAPI with Jinja2 templates
2. Configure Tailwind CSS / DaisyUI
3. Implement all dashboard views
4. Add map functionality (Leaflet.js)
5. Create Click CLI
6. Write tests

Phase 6: Docker & Deployment

1. Create multi-stage Dockerfile
2. Create docker-compose.yml with all services
3. Add health check endpoints
4. Document deployment procedures
5. End-to-end testing

---

Dependencies

[project]
requires-python = ">=3.11"
dependencies = [
    "click>=8.1.0",
    "pydantic>=2.0.0",
    "pydantic-settings>=2.0.0",
    "sqlalchemy>=2.0.0",
    "alembic>=1.12.0",
    "fastapi>=0.100.0",
    "uvicorn[standard]>=0.23.0",
    "paho-mqtt>=2.0.0",
    "meshcore-py>=0.1.0",
    "jinja2>=3.1.0",
    "python-multipart>=0.0.6",
    "httpx>=0.25.0",
    "aiosqlite>=0.19.0",
]

[project.optional-dependencies]
dev = [
    "pytest>=7.4.0",
    "pytest-asyncio>=0.21.0",
    "pytest-cov>=4.1.0",
    "black>=23.0.0",
    "flake8>=6.1.0",
    "mypy>=1.5.0",
    "pre-commit>=3.4.0",
]
postgres = [
    "asyncpg>=0.28.0",
    "psycopg2-binary>=2.9.0",
]

---

CLI Interface

# Main entrypoint
meshcore-hub <component> [options]

# Interface component
meshcore-hub interface --mode receiver --port /dev/ttyUSB0
meshcore-hub interface --mode sender --mock

# Collector component
meshcore-hub collector

# API component
meshcore-hub api --host 0.0.0.0 --port 8000

# Web dashboard
meshcore-hub web --host 0.0.0.0 --port 8080

# Database migrations
meshcore-hub db upgrade
meshcore-hub db downgrade
meshcore-hub db revision --message "description"

---

Testing Strategy

Unit Tests

• Test each module in isolation
• Mock external dependencies (MQTT, database, serial)
• Target 80%+ code coverage

Integration Tests

• Test component interactions
• Use SQLite in-memory database
• Use mock MQTT broker

End-to-End Tests

• Full system tests with Docker Compose
• Test complete event flow from mock device to API query

---

Security Considerations

1. API Authentication: Bearer tokens with two permission levels
2. Input Validation: Pydantic validation on all inputs
3. SQL Injection: SQLAlchemy ORM prevents SQL injection
4. MQTT Security: Support for username/password authentication
5. Secret Management: Environment variables for secrets, never in code
6. Rate Limiting: Consider implementing for production deployments

---

Future Enhancements (Out of Scope)

• WebSocket/SSE for real-time updates
• User management and role-based access
• Multi-tenant support
• Prometheus metrics
• Alerting system
• Mobile-responsive dashboard optimization
• Message encryption/decryption display
• Network topology visualization