mirror of
https://github.com/ipnet-mesh/meshcore-hub.git
synced 2026-03-28 17:42:56 +01:00
Updates
This commit is contained in:
Louis King
441
AGENTS.md
Normal file
441
AGENTS.md
Normal file
@@ -0,0 +1,441 @@
|
||||
# AGENTS.md - AI Coding Assistant Guidelines
|
||||
|
||||
This document provides context and guidelines for AI coding assistants working on the MeshCore Hub project.
|
||||
|
||||
## Project Overview
|
||||
|
||||
MeshCore Hub is a Python 3.11+ monorepo for managing and orchestrating MeshCore mesh networks. It consists of five main components:
|
||||
|
||||
- **meshcore_interface**: Serial/USB interface to MeshCore companion nodes, publishes/subscribes to MQTT
|
||||
- **meshcore_collector**: Collects MeshCore events from MQTT and stores them in a database
|
||||
- **meshcore_api**: REST API for querying data and sending commands via MQTT
|
||||
- **meshcore_web**: Web dashboard for visualizing network status
|
||||
- **meshcore_common**: Shared utilities, models, and configurations
|
||||
|
||||
## Key Documentation
|
||||
|
||||
- [PROMPT.md](PROMPT.md) - Original project specification and requirements
|
||||
- [SCHEMAS.md](SCHEMAS.md) - MeshCore event JSON schemas and database mappings
|
||||
- [PLAN.md](PLAN.md) - Implementation plan and architecture decisions
|
||||
- [TASKS.md](TASKS.md) - Detailed task breakdown with checkboxes for progress tracking
|
||||
|
||||
## Technology Stack
|
||||
|
||||
| Category | Technology |
|
||||
|----------|------------|
|
||||
| Language | Python 3.11+ |
|
||||
| Package Management | pip with pyproject.toml |
|
||||
| CLI Framework | Click |
|
||||
| Configuration | Pydantic Settings |
|
||||
| Database ORM | SQLAlchemy 2.0 (async) |
|
||||
| Migrations | Alembic |
|
||||
| REST API | FastAPI |
|
||||
| MQTT Client | paho-mqtt |
|
||||
| MeshCore Interface | meshcore-py |
|
||||
| Templates | Jinja2 |
|
||||
| CSS Framework | Tailwind CSS + DaisyUI |
|
||||
| Testing | pytest, pytest-asyncio |
|
||||
| Formatting | black |
|
||||
| Linting | flake8 |
|
||||
| Type Checking | mypy |
|
||||
|
||||
## Code Style Guidelines
|
||||
|
||||
### General
|
||||
|
||||
- Follow PEP 8 style guidelines
|
||||
- Use `black` for code formatting (line length 88)
|
||||
- Use type hints for all function signatures
|
||||
- Write docstrings for public modules, classes, and functions
|
||||
- Keep functions focused and under 50 lines where possible
|
||||
|
||||
### Imports
|
||||
|
||||
```python
|
||||
# Standard library
|
||||
import os
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
|
||||
# Third-party
|
||||
from fastapi import FastAPI, Depends
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy import select
|
||||
|
||||
# Local
|
||||
from meshcore_hub.common.config import Settings
|
||||
from meshcore_hub.common.models import Node
|
||||
```
|
||||
|
||||
### Naming Conventions
|
||||
|
||||
| Type | Convention | Example |
|
||||
|------|------------|---------|
|
||||
| Modules | snake_case | `node_tags.py` |
|
||||
| Classes | PascalCase | `NodeTagCreate` |
|
||||
| Functions | snake_case | `get_node_by_key()` |
|
||||
| Constants | UPPER_SNAKE_CASE | `DEFAULT_MQTT_PORT` |
|
||||
| Variables | snake_case | `public_key` |
|
||||
| Type Variables | PascalCase | `T`, `NodeT` |
|
||||
|
||||
### Pydantic Models
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel, Field
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
|
||||
class NodeRead(BaseModel):
|
||||
"""Schema for reading node data from API."""
|
||||
|
||||
id: str
|
||||
public_key: str = Field(..., min_length=64, max_length=64)
|
||||
name: Optional[str] = None
|
||||
adv_type: Optional[str] = None
|
||||
last_seen: Optional[datetime] = None
|
||||
|
||||
model_config = {"from_attributes": True}
|
||||
```
|
||||
|
||||
### SQLAlchemy Models
|
||||
|
||||
```python
|
||||
from sqlalchemy import String, DateTime, ForeignKey
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
from datetime import datetime
|
||||
from uuid import uuid4
|
||||
|
||||
from meshcore_hub.common.models.base import Base
|
||||
|
||||
class Node(Base):
|
||||
__tablename__ = "nodes"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=lambda: str(uuid4()))
|
||||
public_key: Mapped[str] = mapped_column(String(64), unique=True, index=True)
|
||||
name: Mapped[str | None] = mapped_column(String(255), nullable=True)
|
||||
adv_type: Mapped[str | None] = mapped_column(String(20), nullable=True)
|
||||
last_seen: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)
|
||||
|
||||
# Relationships
|
||||
tags: Mapped[list["NodeTag"]] = relationship(back_populates="node", cascade="all, delete-orphan")
|
||||
```
|
||||
|
||||
### FastAPI Routes
|
||||
|
||||
```python
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from typing import Annotated
|
||||
|
||||
from meshcore_hub.api.dependencies import get_db, require_read
|
||||
from meshcore_hub.common.schemas import NodeRead, NodeList
|
||||
|
||||
router = APIRouter(prefix="/nodes", tags=["nodes"])
|
||||
|
||||
@router.get("", response_model=NodeList)
|
||||
async def list_nodes(
|
||||
db: Annotated[AsyncSession, Depends(get_db)],
|
||||
_: Annotated[None, Depends(require_read)],
|
||||
limit: int = Query(default=50, le=100),
|
||||
offset: int = Query(default=0, ge=0),
|
||||
) -> NodeList:
|
||||
"""List all nodes with pagination."""
|
||||
# Implementation
|
||||
pass
|
||||
```
|
||||
|
||||
### Click CLI Commands
|
||||
|
||||
```python
|
||||
import click
|
||||
from meshcore_hub.common.config import Settings
|
||||
|
||||
@click.group()
|
||||
@click.pass_context
|
||||
def cli(ctx: click.Context) -> None:
|
||||
"""MeshCore Hub CLI."""
|
||||
ctx.ensure_object(dict)
|
||||
|
||||
@cli.command()
|
||||
@click.option("--host", default="0.0.0.0", help="Bind host")
|
||||
@click.option("--port", default=8000, type=int, help="Bind port")
|
||||
@click.pass_context
|
||||
def api(ctx: click.Context, host: str, port: int) -> None:
|
||||
"""Start the API server."""
|
||||
import uvicorn
|
||||
from meshcore_hub.api.app import create_app
|
||||
|
||||
app = create_app()
|
||||
uvicorn.run(app, host=host, port=port)
|
||||
```
|
||||
|
||||
### Async Patterns
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from contextlib import asynccontextmanager
|
||||
from typing import AsyncGenerator
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
"""Application lifespan handler."""
|
||||
# Startup
|
||||
await setup_database()
|
||||
await connect_mqtt()
|
||||
|
||||
yield
|
||||
|
||||
# Shutdown
|
||||
await disconnect_mqtt()
|
||||
await close_database()
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
```python
|
||||
from fastapi import HTTPException, status
|
||||
|
||||
# Use specific HTTP exceptions
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail=f"Node with public_key '{public_key}' not found"
|
||||
)
|
||||
|
||||
# Log exceptions with context
|
||||
import logging
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
try:
|
||||
result = await risky_operation()
|
||||
except SomeException as e:
|
||||
logger.exception("Failed to perform operation: %s", e)
|
||||
raise
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
meshcore-hub/
|
||||
├── src/meshcore_hub/
|
||||
│ ├── __init__.py
|
||||
│ ├── __main__.py # CLI entry point
|
||||
│ ├── common/
|
||||
│ │ ├── config.py # Pydantic settings
|
||||
│ │ ├── database.py # DB session management
|
||||
│ │ ├── mqtt.py # MQTT utilities
|
||||
│ │ ├── logging.py # Logging config
|
||||
│ │ ├── models/ # SQLAlchemy models
|
||||
│ │ └── schemas/ # Pydantic schemas
|
||||
│ ├── interface/
|
||||
│ │ ├── cli.py
|
||||
│ │ ├── device.py # MeshCore device wrapper
|
||||
│ │ ├── mock_device.py # Mock for testing
|
||||
│ │ ├── receiver.py # RECEIVER mode
|
||||
│ │ └── sender.py # SENDER mode
|
||||
│ ├── collector/
|
||||
│ │ ├── cli.py
|
||||
│ │ ├── subscriber.py # MQTT subscriber
|
||||
│ │ ├── handlers/ # Event handlers
|
||||
│ │ └── webhook.py # Webhook dispatcher
|
||||
│ ├── api/
|
||||
│ │ ├── cli.py
|
||||
│ │ ├── app.py # FastAPI app
|
||||
│ │ ├── auth.py # Authentication
|
||||
│ │ ├── dependencies.py
|
||||
│ │ ├── routes/ # API routes
|
||||
│ │ └── templates/ # Dashboard HTML
|
||||
│ └── web/
|
||||
│ ├── cli.py
|
||||
│ ├── app.py # FastAPI app
|
||||
│ ├── routes/ # Page routes
|
||||
│ ├── templates/ # Jinja2 templates
|
||||
│ └── static/ # CSS, JS
|
||||
├── tests/
|
||||
│ ├── conftest.py
|
||||
│ ├── test_common/
|
||||
│ ├── test_interface/
|
||||
│ ├── test_collector/
|
||||
│ ├── test_api/
|
||||
│ └── test_web/
|
||||
├── alembic/
|
||||
│ ├── env.py
|
||||
│ └── versions/
|
||||
└── docker/
|
||||
├── Dockerfile
|
||||
└── docker-compose.yml
|
||||
```
|
||||
|
||||
## MQTT Topic Structure
|
||||
|
||||
### Events (Published by Interface RECEIVER)
|
||||
```
|
||||
<prefix>/<public_key>/event/<event_name>
|
||||
```
|
||||
|
||||
Examples:
|
||||
- `meshcore/abc123.../event/advertisement`
|
||||
- `meshcore/abc123.../event/contact_msg_recv`
|
||||
- `meshcore/abc123.../event/channel_msg_recv`
|
||||
|
||||
### Commands (Subscribed by Interface SENDER)
|
||||
```
|
||||
<prefix>/+/command/<command_name>
|
||||
```
|
||||
|
||||
Examples:
|
||||
- `meshcore/+/command/send_msg`
|
||||
- `meshcore/+/command/send_channel_msg`
|
||||
- `meshcore/+/command/send_advert`
|
||||
|
||||
## Database Conventions
|
||||
|
||||
- Use UUIDs for primary keys (stored as VARCHAR(36))
|
||||
- Use `public_key` (64-char hex) as the canonical node identifier
|
||||
- All timestamps stored as UTC
|
||||
- JSON columns for flexible data (path_hashes, parsed_data, etc.)
|
||||
- Foreign keys reference nodes by UUID, not public_key
|
||||
|
||||
## Testing Guidelines
|
||||
|
||||
### Unit Tests
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
@pytest.fixture
|
||||
def mock_mqtt_client():
|
||||
client = AsyncMock()
|
||||
client.publish = AsyncMock()
|
||||
return client
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_receiver_publishes_event(mock_mqtt_client):
|
||||
"""Test that receiver publishes events to correct MQTT topic."""
|
||||
# Arrange
|
||||
receiver = Receiver(mqtt_client=mock_mqtt_client, prefix="test")
|
||||
|
||||
# Act
|
||||
await receiver.handle_advertisement(event_data)
|
||||
|
||||
# Assert
|
||||
mock_mqtt_client.publish.assert_called_once()
|
||||
call_args = mock_mqtt_client.publish.call_args
|
||||
assert "test/" in call_args[0][0]
|
||||
assert "/event/advertisement" in call_args[0][0]
|
||||
```
|
||||
|
||||
### Integration Tests
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from httpx import AsyncClient
|
||||
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession
|
||||
|
||||
@pytest.fixture
|
||||
async def db_session():
|
||||
"""Create in-memory SQLite database for testing."""
|
||||
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
|
||||
async with AsyncSession(engine) as session:
|
||||
yield session
|
||||
|
||||
@pytest.fixture
|
||||
async def client(db_session):
|
||||
"""Create test client with database session."""
|
||||
app = create_app()
|
||||
app.dependency_overrides[get_db] = lambda: db_session
|
||||
|
||||
async with AsyncClient(app=app, base_url="http://test") as client:
|
||||
yield client
|
||||
```
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### Adding a New API Endpoint
|
||||
|
||||
1. Create/update Pydantic schema in `common/schemas/`
|
||||
2. Add route function in appropriate `api/routes/` module
|
||||
3. Include router in `api/routes/__init__.py` if new module
|
||||
4. Add tests in `tests/test_api/`
|
||||
5. Update OpenAPI documentation if needed
|
||||
|
||||
### Adding a New Event Handler
|
||||
|
||||
1. Create handler in `collector/handlers/`
|
||||
2. Register handler in `collector/handlers/__init__.py`
|
||||
3. Add corresponding Pydantic schema if needed
|
||||
4. Create/update database model if persisted
|
||||
5. Add Alembic migration if schema changed
|
||||
6. Add tests in `tests/test_collector/`
|
||||
|
||||
### Adding a New Database Model
|
||||
|
||||
1. Create model in `common/models/`
|
||||
2. Export in `common/models/__init__.py`
|
||||
3. Create Alembic migration: `alembic revision --autogenerate -m "description"`
|
||||
4. Review and adjust migration file
|
||||
5. Test migration: `alembic upgrade head`
|
||||
|
||||
### Running the Development Environment
|
||||
|
||||
```bash
|
||||
# Create virtual environment
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate
|
||||
|
||||
# Install dependencies
|
||||
pip install -e ".[dev]"
|
||||
|
||||
# Run pre-commit hooks
|
||||
pre-commit install
|
||||
pre-commit run --all-files
|
||||
|
||||
# Run tests
|
||||
pytest
|
||||
|
||||
# Run specific component
|
||||
meshcore-hub api --reload
|
||||
meshcore-hub collector
|
||||
meshcore-hub interface --mode receiver --mock
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
See [PLAN.md](PLAN.md#configuration-environment-variables) for complete list.
|
||||
|
||||
Key variables:
|
||||
- `MQTT_HOST`, `MQTT_PORT`, `MQTT_PREFIX` - MQTT broker connection
|
||||
- `DATABASE_URL` - SQLAlchemy database URL
|
||||
- `API_READ_KEY`, `API_ADMIN_KEY` - API authentication keys
|
||||
- `LOG_LEVEL` - Logging verbosity
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **MQTT Connection Failed**: Check broker is running and `MQTT_HOST`/`MQTT_PORT` are correct
|
||||
2. **Database Migration Errors**: Ensure `DATABASE_URL` is correct, run `alembic upgrade head`
|
||||
3. **Import Errors**: Ensure package is installed with `pip install -e .`
|
||||
4. **Type Errors**: Run `mypy src/` to check type annotations
|
||||
|
||||
### Debugging
|
||||
|
||||
```python
|
||||
# Enable debug logging
|
||||
import logging
|
||||
logging.basicConfig(level=logging.DEBUG)
|
||||
|
||||
# Or via environment
|
||||
export LOG_LEVEL=DEBUG
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [meshcore_py Documentation](https://github.com/meshcore-dev/meshcore_py)
|
||||
- [FastAPI Documentation](https://fastapi.tiangolo.com/)
|
||||
- [SQLAlchemy 2.0 Documentation](https://docs.sqlalchemy.org/en/20/)
|
||||
- [Pydantic Documentation](https://docs.pydantic.dev/)
|
||||
- [Click Documentation](https://click.palletsprojects.com/)
|
||||
672
PLAN.md
Normal file
672
PLAN.md
Normal file
@@ -0,0 +1,672 @@
|
||||
# MeshCore Hub - Implementation Plan
|
||||
|
||||
## Project Overview
|
||||
|
||||
MeshCore Hub is a Python 3.11+ monorepo for managing and orchestrating MeshCore mesh networks. It consists of five main components that work together to receive, store, query, and visualize mesh network data.
|
||||
|
||||
---
|
||||
|
||||
## Questions Requiring Clarification
|
||||
|
||||
Before implementation, the following questions need answers:
|
||||
|
||||
### Architecture & Design
|
||||
|
||||
1. **MQTT Broker Selection**: Should we include an embedded MQTT broker (like `hbmqtt`) or assume an external broker (Mosquitto) is always used? The spec mentions Docker but doesn't specify broker deployment.
|
||||
|
||||
2. **Database Deployment**: For production, should we support PostgreSQL/MySQL from the start, or focus on SQLite initially and add other backends later? This affects connection pooling and async driver choices.
|
||||
|
||||
3. **Web Dashboard Separation**: Should `meshcore_web` be a separate FastAPI app or integrated into `meshcore_api`? Running two FastAPI apps adds deployment complexity.
|
||||
|
||||
4. **Member Profiles JSON Location**: Where should the static JSON file for member profiles be stored? In the config directory, as a mounted volume, or embedded in the package?
|
||||
|
||||
### Interface Component
|
||||
|
||||
5. **Multiple Serial Devices**: Should a single Interface instance support multiple serial devices simultaneously, or should users run multiple instances (one per device)?
|
||||
|
||||
6. **Reconnection Strategy**: What should happen when the serial connection is lost? Automatic reconnect with backoff, or exit and let the container orchestrator restart?
|
||||
|
||||
7. **Mock Device Scope**: How comprehensive should the mock MeshCore device be? Should it simulate realistic timing, packet loss, and network topology?
|
||||
|
||||
### Collector Component
|
||||
|
||||
8. **Event Deduplication**: How should we handle duplicate events from multiple receiver nodes? By message signature/hash, or accept all duplicates?
|
||||
|
||||
9. **Data Retention Policy**: Should we implement automatic data pruning (e.g., delete messages older than X days)? This affects long-running deployments.
|
||||
|
||||
10. **Webhook Configuration**: The SCHEMAS.md mentions webhooks but PROMPT.md doesn't detail webhook management. Should webhooks be configured via API, config file, or environment variables?
|
||||
|
||||
### API Component
|
||||
|
||||
11. **API Key Management**: How should API keys be generated and managed? Static config, database-stored, or runtime generation? What about key rotation?
|
||||
|
||||
12. **Rate Limiting**: Should the API implement rate limiting? If so, what defaults?
|
||||
|
||||
13. **CORS Configuration**: Should CORS be configurable for web dashboard access from different domains?
|
||||
|
||||
### Web Dashboard
|
||||
|
||||
14. **Authentication**: Should the web dashboard have its own authentication, or rely on API bearer tokens? What about session management?
|
||||
|
||||
15. **Real-time Updates**: Should the dashboard support real-time updates (WebSocket/SSE), or polling-based refresh?
|
||||
|
||||
16. **Map Provider**: Which map provider for the Node Map view? OpenStreetMap/Leaflet (free) or allow configuration for commercial providers?
|
||||
|
||||
### Node Tags System
|
||||
|
||||
17. **Tag Value Types**: Should node tags support typed values (string, number, boolean, coordinates) or just strings? The spec mentions lat/lon for the map feature.
|
||||
|
||||
18. **Reserved Tag Names**: Should certain tag names be reserved for system use (e.g., `location`, `description`)?
|
||||
|
||||
### DevOps & Operations
|
||||
|
||||
19. **Health Checks**: What health check endpoints should each component expose for Docker/Kubernetes?
|
||||
|
||||
20. **Metrics/Observability**: Should we include Prometheus metrics endpoints or structured logging for observability?
|
||||
|
||||
21. **Log Level Configuration**: Per-component or global log level configuration?
|
||||
|
||||
---
|
||||
|
||||
## Proposed Architecture
|
||||
|
||||
```
|
||||
+------------------+
|
||||
| MQTT Broker |
|
||||
| (Mosquitto) |
|
||||
+--------+---------+
|
||||
|
|
||||
+------------------------------+------------------------------+
|
||||
| | |
|
||||
v v v
|
||||
+---------+---------+ +---------+---------+ +---------+---------+
|
||||
| meshcore_interface| | meshcore_interface| | meshcore_interface|
|
||||
| (RECEIVER) | | (RECEIVER) | | (SENDER) |
|
||||
+-------------------+ +-------------------+ +-------------------+
|
||||
| | ^
|
||||
| Publishes events | |
|
||||
v v |
|
||||
+----------------------------------------------------------+ |
|
||||
| MQTT Topics | |
|
||||
| <prefix>/<pubkey>/event/<event_name> | |
|
||||
| <prefix>/+/command/<command_name> <--------------------+-----------+
|
||||
+----------------------------------------------------------+
|
||||
|
|
||||
v
|
||||
+---------+---------+
|
||||
| meshcore_collector|
|
||||
| (Subscriber) |
|
||||
+---------+---------+
|
||||
|
|
||||
v
|
||||
+---------+---------+
|
||||
| Database |
|
||||
| (SQLite/Postgres)|
|
||||
+---------+---------+
|
||||
^
|
||||
|
|
||||
+---------------+---------------+
|
||||
| |
|
||||
+---------+---------+ +---------+---------+
|
||||
| meshcore_api | | meshcore_web |
|
||||
| (REST API) | | (Dashboard) |
|
||||
+-------------------+ +-------------------+
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Package Structure
|
||||
|
||||
```
|
||||
meshcore-hub/
|
||||
├── pyproject.toml # Root project configuration
|
||||
├── README.md
|
||||
├── PROMPT.md
|
||||
├── SCHEMAS.md
|
||||
├── PLAN.md
|
||||
├── .env.example # Example environment variables
|
||||
├── .pre-commit-config.yaml # Pre-commit hooks
|
||||
├── alembic.ini # Alembic configuration
|
||||
├── alembic/ # Database migrations
|
||||
│ ├── env.py
|
||||
│ ├── versions/
|
||||
│ └── script.py.mako
|
||||
├── docker/
|
||||
│ ├── Dockerfile
|
||||
│ └── docker-compose.yml
|
||||
├── tests/
|
||||
│ ├── conftest.py
|
||||
│ ├── test_interface/
|
||||
│ ├── test_collector/
|
||||
│ ├── test_api/
|
||||
│ ├── test_web/
|
||||
│ └── test_common/
|
||||
└── src/
|
||||
└── meshcore_hub/
|
||||
├── __init__.py
|
||||
├── __main__.py # CLI entrypoint
|
||||
├── common/
|
||||
│ ├── __init__.py
|
||||
│ ├── config.py # Pydantic settings
|
||||
│ ├── models/ # SQLAlchemy models
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── base.py
|
||||
│ │ ├── node.py
|
||||
│ │ ├── message.py
|
||||
│ │ ├── advertisement.py
|
||||
│ │ ├── trace_path.py
|
||||
│ │ ├── telemetry.py
|
||||
│ │ ├── node_tag.py
|
||||
│ │ └── event_log.py
|
||||
│ ├── schemas/ # Pydantic schemas (API request/response)
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── events.py
|
||||
│ │ ├── nodes.py
|
||||
│ │ ├── messages.py
|
||||
│ │ └── commands.py
|
||||
│ ├── mqtt.py # MQTT client utilities
|
||||
│ ├── database.py # Database session management
|
||||
│ └── logging.py # Logging configuration
|
||||
├── interface/
|
||||
│ ├── __init__.py
|
||||
│ ├── cli.py # Click CLI for interface
|
||||
│ ├── receiver.py # RECEIVER mode implementation
|
||||
│ ├── sender.py # SENDER mode implementation
|
||||
│ ├── device.py # MeshCore device wrapper
|
||||
│ └── mock_device.py # Mock device for testing
|
||||
├── collector/
|
||||
│ ├── __init__.py
|
||||
│ ├── cli.py # Click CLI for collector
|
||||
│ ├── subscriber.py # MQTT subscriber
|
||||
│ ├── handlers/ # Event handlers
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── message.py
|
||||
│ │ ├── advertisement.py
|
||||
│ │ ├── trace.py
|
||||
│ │ ├── telemetry.py
|
||||
│ │ └── contacts.py
|
||||
│ └── webhook.py # Webhook dispatcher
|
||||
├── api/
|
||||
│ ├── __init__.py
|
||||
│ ├── cli.py # Click CLI for API
|
||||
│ ├── app.py # FastAPI application
|
||||
│ ├── dependencies.py # FastAPI dependencies
|
||||
│ ├── auth.py # Bearer token authentication
|
||||
│ ├── routes/
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── messages.py
|
||||
│ │ ├── nodes.py
|
||||
│ │ ├── advertisements.py
|
||||
│ │ ├── trace_paths.py
|
||||
│ │ ├── telemetry.py
|
||||
│ │ ├── node_tags.py
|
||||
│ │ ├── commands.py
|
||||
│ │ └── dashboard.py # Simple HTML dashboard
|
||||
│ └── templates/
|
||||
│ └── dashboard.html
|
||||
└── web/
|
||||
├── __init__.py
|
||||
├── cli.py # Click CLI for web dashboard
|
||||
├── app.py # FastAPI application
|
||||
├── routes/
|
||||
│ ├── __init__.py
|
||||
│ ├── home.py
|
||||
│ ├── members.py
|
||||
│ ├── network.py
|
||||
│ ├── nodes.py
|
||||
│ ├── map.py
|
||||
│ └── messages.py
|
||||
├── templates/
|
||||
│ ├── base.html
|
||||
│ ├── home.html
|
||||
│ ├── members.html
|
||||
│ ├── network.html
|
||||
│ ├── nodes.html
|
||||
│ ├── map.html
|
||||
│ └── messages.html
|
||||
└── static/
|
||||
├── css/
|
||||
└── js/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Database Schema
|
||||
|
||||
### Core Tables
|
||||
|
||||
#### `nodes`
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| id | UUID | Primary key |
|
||||
| public_key | VARCHAR(64) | Unique, indexed |
|
||||
| name | VARCHAR(255) | Node display name |
|
||||
| adv_type | VARCHAR(20) | chat, repeater, room, none |
|
||||
| flags | INTEGER | Capability flags |
|
||||
| first_seen | TIMESTAMP | First advertisement |
|
||||
| last_seen | TIMESTAMP | Most recent activity |
|
||||
| created_at | TIMESTAMP | Record creation |
|
||||
| updated_at | TIMESTAMP | Record update |
|
||||
|
||||
#### `node_tags`
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| id | UUID | Primary key |
|
||||
| node_id | UUID | FK to nodes |
|
||||
| key | VARCHAR(100) | Tag name |
|
||||
| value | TEXT | Tag value (JSON for typed values) |
|
||||
| value_type | VARCHAR(20) | string, number, boolean, coordinate |
|
||||
| created_at | TIMESTAMP | Record creation |
|
||||
| updated_at | TIMESTAMP | Record update |
|
||||
|
||||
*Unique constraint on (node_id, key)*
|
||||
|
||||
#### `messages`
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| id | UUID | Primary key |
|
||||
| receiver_node_id | UUID | FK to nodes (receiving interface) |
|
||||
| message_type | VARCHAR(20) | contact, channel |
|
||||
| pubkey_prefix | VARCHAR(12) | Sender prefix (contact msgs) |
|
||||
| channel_idx | INTEGER | Channel index (channel msgs) |
|
||||
| text | TEXT | Message content |
|
||||
| path_len | INTEGER | Hop count |
|
||||
| txt_type | INTEGER | Message type indicator |
|
||||
| signature | VARCHAR(8) | Message signature |
|
||||
| snr | FLOAT | Signal-to-noise ratio |
|
||||
| sender_timestamp | TIMESTAMP | Sender's timestamp |
|
||||
| received_at | TIMESTAMP | When received by interface |
|
||||
| created_at | TIMESTAMP | Record creation |
|
||||
|
||||
#### `advertisements`
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| id | UUID | Primary key |
|
||||
| receiver_node_id | UUID | FK to nodes (receiving interface) |
|
||||
| node_id | UUID | FK to nodes (advertised node) |
|
||||
| public_key | VARCHAR(64) | Advertised public key |
|
||||
| name | VARCHAR(255) | Advertised name |
|
||||
| adv_type | VARCHAR(20) | Node type |
|
||||
| flags | INTEGER | Capability flags |
|
||||
| received_at | TIMESTAMP | When received |
|
||||
| created_at | TIMESTAMP | Record creation |
|
||||
|
||||
#### `trace_paths`
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| id | UUID | Primary key |
|
||||
| receiver_node_id | UUID | FK to nodes |
|
||||
| initiator_tag | BIGINT | Trace identifier |
|
||||
| path_len | INTEGER | Path length |
|
||||
| flags | INTEGER | Trace flags |
|
||||
| auth | INTEGER | Auth data |
|
||||
| path_hashes | JSON | Array of node hashes |
|
||||
| snr_values | JSON | Array of SNR values |
|
||||
| hop_count | INTEGER | Total hops |
|
||||
| received_at | TIMESTAMP | When received |
|
||||
| created_at | TIMESTAMP | Record creation |
|
||||
|
||||
#### `telemetry`
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| id | UUID | Primary key |
|
||||
| receiver_node_id | UUID | FK to nodes |
|
||||
| node_id | UUID | FK to nodes (reporting node) |
|
||||
| node_public_key | VARCHAR(64) | Reporting node key |
|
||||
| lpp_data | BYTEA | Raw LPP data |
|
||||
| parsed_data | JSON | Decoded sensor readings |
|
||||
| received_at | TIMESTAMP | When received |
|
||||
| created_at | TIMESTAMP | Record creation |
|
||||
|
||||
#### `events_log`
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| id | UUID | Primary key |
|
||||
| receiver_node_id | UUID | FK to nodes |
|
||||
| event_type | VARCHAR(50) | Event type name |
|
||||
| payload | JSON | Full event payload |
|
||||
| received_at | TIMESTAMP | When received |
|
||||
| created_at | TIMESTAMP | Record creation |
|
||||
|
||||
---
|
||||
|
||||
## MQTT Topic Structure
|
||||
|
||||
### Event Topics (Published by RECEIVER)
|
||||
```
|
||||
meshcore/<public_key>/event/advertisement
|
||||
meshcore/<public_key>/event/contact_msg_recv
|
||||
meshcore/<public_key>/event/channel_msg_recv
|
||||
meshcore/<public_key>/event/trace_data
|
||||
meshcore/<public_key>/event/telemetry_response
|
||||
meshcore/<public_key>/event/contacts
|
||||
meshcore/<public_key>/event/send_confirmed
|
||||
meshcore/<public_key>/event/status_response
|
||||
meshcore/<public_key>/event/battery
|
||||
meshcore/<public_key>/event/path_updated
|
||||
```
|
||||
|
||||
### Command Topics (Subscribed by SENDER)
|
||||
```
|
||||
meshcore/+/command/send_msg
|
||||
meshcore/+/command/send_channel_msg
|
||||
meshcore/+/command/send_advert
|
||||
meshcore/+/command/request_status
|
||||
meshcore/+/command/request_telemetry
|
||||
```
|
||||
|
||||
### Command Payloads
|
||||
|
||||
#### send_msg
|
||||
```json
|
||||
{
|
||||
"destination": "public_key or pubkey_prefix",
|
||||
"text": "message content",
|
||||
"timestamp": 1732820498
|
||||
}
|
||||
```
|
||||
|
||||
#### send_channel_msg
|
||||
```json
|
||||
{
|
||||
"channel_idx": 4,
|
||||
"text": "message content",
|
||||
"timestamp": 1732820498
|
||||
}
|
||||
```
|
||||
|
||||
#### send_advert
|
||||
```json
|
||||
{
|
||||
"flood": true
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Authentication
|
||||
- Bearer token in `Authorization` header
|
||||
- Two token levels: `read` (query only) and `admin` (query + commands)
|
||||
|
||||
### Nodes
|
||||
| Method | Endpoint | Auth | Description |
|
||||
|--------|----------|------|-------------|
|
||||
| GET | /api/v1/nodes | read | List all nodes with pagination/filtering |
|
||||
| GET | /api/v1/nodes/{public_key} | read | Get single node details |
|
||||
|
||||
### Node Tags
|
||||
| Method | Endpoint | Auth | Description |
|
||||
|--------|----------|------|-------------|
|
||||
| GET | /api/v1/nodes/{public_key}/tags | read | List node's tags |
|
||||
| POST | /api/v1/nodes/{public_key}/tags | admin | Create tag |
|
||||
| PUT | /api/v1/nodes/{public_key}/tags/{key} | admin | Update tag |
|
||||
| DELETE | /api/v1/nodes/{public_key}/tags/{key} | admin | Delete tag |
|
||||
|
||||
### Messages
|
||||
| Method | Endpoint | Auth | Description |
|
||||
|--------|----------|------|-------------|
|
||||
| GET | /api/v1/messages | read | List messages with filters |
|
||||
| GET | /api/v1/messages/{id} | read | Get single message |
|
||||
|
||||
**Query Parameters:**
|
||||
- `type`: contact, channel
|
||||
- `pubkey_prefix`: Filter by sender
|
||||
- `channel_idx`: Filter by channel
|
||||
- `since`: Start timestamp
|
||||
- `until`: End timestamp
|
||||
- `limit`, `offset`: Pagination
|
||||
|
||||
### Advertisements
|
||||
| Method | Endpoint | Auth | Description |
|
||||
|--------|----------|------|-------------|
|
||||
| GET | /api/v1/advertisements | read | List advertisements |
|
||||
| GET | /api/v1/advertisements/{id} | read | Get single advertisement |
|
||||
|
||||
### Trace Paths
|
||||
| Method | Endpoint | Auth | Description |
|
||||
|--------|----------|------|-------------|
|
||||
| GET | /api/v1/trace-paths | read | List trace paths |
|
||||
| GET | /api/v1/trace-paths/{id} | read | Get single trace path |
|
||||
|
||||
### Telemetry
|
||||
| Method | Endpoint | Auth | Description |
|
||||
|--------|----------|------|-------------|
|
||||
| GET | /api/v1/telemetry | read | List telemetry data |
|
||||
| GET | /api/v1/telemetry/{id} | read | Get single telemetry record |
|
||||
|
||||
### Commands
|
||||
| Method | Endpoint | Auth | Description |
|
||||
|--------|----------|------|-------------|
|
||||
| POST | /api/v1/commands/send-message | admin | Send direct message |
|
||||
| POST | /api/v1/commands/send-channel-message | admin | Send channel message |
|
||||
| POST | /api/v1/commands/send-advertisement | admin | Send advertisement |
|
||||
|
||||
### Dashboard
|
||||
| Method | Endpoint | Auth | Description |
|
||||
|--------|----------|------|-------------|
|
||||
| GET | /api/v1/dashboard | read | HTML dashboard page |
|
||||
| GET | /api/v1/stats | read | JSON statistics |
|
||||
|
||||
---
|
||||
|
||||
## Configuration (Environment Variables)
|
||||
|
||||
### Common
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| LOG_LEVEL | INFO | Logging level |
|
||||
| MQTT_HOST | localhost | MQTT broker host |
|
||||
| MQTT_PORT | 1883 | MQTT broker port |
|
||||
| MQTT_USERNAME | | MQTT username (optional) |
|
||||
| MQTT_PASSWORD | | MQTT password (optional) |
|
||||
| MQTT_PREFIX | meshcore | Topic prefix |
|
||||
|
||||
### Interface
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| INTERFACE_MODE | RECEIVER | RECEIVER or SENDER |
|
||||
| SERIAL_PORT | /dev/ttyUSB0 | Serial port path |
|
||||
| SERIAL_BAUD | 115200 | Baud rate |
|
||||
| MOCK_DEVICE | false | Use mock device |
|
||||
|
||||
### Collector
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| DATABASE_URL | sqlite:///./meshcore.db | SQLAlchemy URL |
|
||||
|
||||
### API
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| API_HOST | 0.0.0.0 | API bind host |
|
||||
| API_PORT | 8000 | API bind port |
|
||||
| API_READ_KEY | | Read-only API key |
|
||||
| API_ADMIN_KEY | | Admin API key |
|
||||
| DATABASE_URL | sqlite:///./meshcore.db | SQLAlchemy URL |
|
||||
|
||||
### Web Dashboard
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| WEB_HOST | 0.0.0.0 | Web bind host |
|
||||
| WEB_PORT | 8080 | Web bind port |
|
||||
| API_BASE_URL | http://localhost:8000 | API endpoint |
|
||||
| API_KEY | | API key for queries |
|
||||
| NETWORK_DOMAIN | | Network domain |
|
||||
| NETWORK_NAME | MeshCore Network | Network name |
|
||||
| NETWORK_CITY | | City location |
|
||||
| NETWORK_COUNTRY | | Country code |
|
||||
| NETWORK_LOCATION | | Lat,Lon |
|
||||
| NETWORK_RADIO_CONFIG | | Radio config details |
|
||||
| NETWORK_CONTACT_EMAIL | | Contact email |
|
||||
| NETWORK_CONTACT_DISCORD | | Discord link |
|
||||
| MEMBERS_FILE | members.json | Path to members JSON |
|
||||
|
||||
---
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 1: Foundation
|
||||
1. Set up project structure with pyproject.toml
|
||||
2. Configure development tools (black, flake8, mypy, pytest)
|
||||
3. Set up pre-commit hooks
|
||||
4. Implement `meshcore_common`:
|
||||
- Pydantic settings/config
|
||||
- SQLAlchemy models
|
||||
- Database connection management
|
||||
- MQTT client utilities
|
||||
- Logging configuration
|
||||
5. Set up Alembic for migrations
|
||||
6. Create initial migration
|
||||
|
||||
### Phase 2: Interface Component
|
||||
1. Implement MeshCore device wrapper
|
||||
2. Implement RECEIVER mode:
|
||||
- Event subscription
|
||||
- MQTT publishing
|
||||
3. Implement SENDER mode:
|
||||
- MQTT subscription
|
||||
- Command dispatching
|
||||
4. Implement mock device for testing
|
||||
5. Create Click CLI
|
||||
6. Write unit tests
|
||||
|
||||
### Phase 3: Collector Component
|
||||
1. Implement MQTT subscriber
|
||||
2. Implement event handlers for each event type
|
||||
3. Implement database persistence
|
||||
4. Create Click CLI
|
||||
5. Write unit tests
|
||||
|
||||
### Phase 4: API Component
|
||||
1. Set up FastAPI application
|
||||
2. Implement authentication middleware
|
||||
3. Implement all REST endpoints
|
||||
4. Implement MQTT command publishing
|
||||
5. Implement simple HTML dashboard
|
||||
6. Add OpenAPI documentation
|
||||
7. Create Click CLI
|
||||
8. Write unit and integration tests
|
||||
|
||||
### Phase 5: Web Dashboard
|
||||
1. Set up FastAPI with Jinja2 templates
|
||||
2. Configure Tailwind CSS / DaisyUI
|
||||
3. Implement all dashboard views
|
||||
4. Add map functionality (Leaflet.js)
|
||||
5. Create Click CLI
|
||||
6. Write tests
|
||||
|
||||
### Phase 6: Docker & Deployment
|
||||
1. Create multi-stage Dockerfile
|
||||
2. Create docker-compose.yml with all services
|
||||
3. Add health check endpoints
|
||||
4. Document deployment procedures
|
||||
5. End-to-end testing
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
```toml
|
||||
[project]
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"click>=8.1.0",
|
||||
"pydantic>=2.0.0",
|
||||
"pydantic-settings>=2.0.0",
|
||||
"sqlalchemy>=2.0.0",
|
||||
"alembic>=1.12.0",
|
||||
"fastapi>=0.100.0",
|
||||
"uvicorn[standard]>=0.23.0",
|
||||
"paho-mqtt>=2.0.0",
|
||||
"meshcore-py>=0.1.0",
|
||||
"jinja2>=3.1.0",
|
||||
"python-multipart>=0.0.6",
|
||||
"httpx>=0.25.0",
|
||||
"aiosqlite>=0.19.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=7.4.0",
|
||||
"pytest-asyncio>=0.21.0",
|
||||
"pytest-cov>=4.1.0",
|
||||
"black>=23.0.0",
|
||||
"flake8>=6.1.0",
|
||||
"mypy>=1.5.0",
|
||||
"pre-commit>=3.4.0",
|
||||
]
|
||||
postgres = [
|
||||
"asyncpg>=0.28.0",
|
||||
"psycopg2-binary>=2.9.0",
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CLI Interface
|
||||
|
||||
```bash
|
||||
# Main entrypoint
|
||||
meshcore-hub <component> [options]
|
||||
|
||||
# Interface component
|
||||
meshcore-hub interface --mode receiver --port /dev/ttyUSB0
|
||||
meshcore-hub interface --mode sender --mock
|
||||
|
||||
# Collector component
|
||||
meshcore-hub collector
|
||||
|
||||
# API component
|
||||
meshcore-hub api --host 0.0.0.0 --port 8000
|
||||
|
||||
# Web dashboard
|
||||
meshcore-hub web --host 0.0.0.0 --port 8080
|
||||
|
||||
# Database migrations
|
||||
meshcore-hub db upgrade
|
||||
meshcore-hub db downgrade
|
||||
meshcore-hub db revision --message "description"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- Test each module in isolation
|
||||
- Mock external dependencies (MQTT, database, serial)
|
||||
- Target 80%+ code coverage
|
||||
|
||||
### Integration Tests
|
||||
- Test component interactions
|
||||
- Use SQLite in-memory database
|
||||
- Use mock MQTT broker
|
||||
|
||||
### End-to-End Tests
|
||||
- Full system tests with Docker Compose
|
||||
- Test complete event flow from mock device to API query
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **API Authentication**: Bearer tokens with two permission levels
|
||||
2. **Input Validation**: Pydantic validation on all inputs
|
||||
3. **SQL Injection**: SQLAlchemy ORM prevents SQL injection
|
||||
4. **MQTT Security**: Support for username/password authentication
|
||||
5. **Secret Management**: Environment variables for secrets, never in code
|
||||
6. **Rate Limiting**: Consider implementing for production deployments
|
||||
|
||||
---
|
||||
|
||||
## Future Enhancements (Out of Scope)
|
||||
|
||||
- WebSocket/SSE for real-time updates
|
||||
- User management and role-based access
|
||||
- Multi-tenant support
|
||||
- Prometheus metrics
|
||||
- Alerting system
|
||||
- Mobile-responsive dashboard optimization
|
||||
- Message encryption/decryption display
|
||||
- Network topology visualization
|
||||
321
README.md
321
README.md
@@ -1,2 +1,319 @@
|
||||
# meshcore-hub
|
||||
Experiment
|
||||
# MeshCore Hub
|
||||
|
||||
Python 3.11+ platform for managing and orchestrating MeshCore mesh networks.
|
||||
|
||||
## Overview
|
||||
|
||||
MeshCore Hub provides a complete solution for monitoring, collecting, and interacting with MeshCore mesh networks. It consists of multiple components that work together:
|
||||
|
||||
| Component | Description |
|
||||
|-----------|-------------|
|
||||
| **Interface** | Connects to MeshCore companion nodes via Serial/USB, bridges events to/from MQTT |
|
||||
| **Collector** | Subscribes to MQTT events and persists them to a database |
|
||||
| **API** | REST API for querying data and sending commands to the network |
|
||||
| **Web Dashboard** | User-friendly web interface for visualizing network status |
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ MeshCore │ │ MeshCore │ │ MeshCore │
|
||||
│ Device 1 │ │ Device 2 │ │ Device 3 │
|
||||
└────────┬────────┘ └────────┬────────┘ └────────┬────────┘
|
||||
│ │ │
|
||||
│ Serial/USB │ Serial/USB │ Serial/USB
|
||||
│ │ │
|
||||
┌────────▼────────┐ ┌────────▼────────┐ ┌────────▼────────┐
|
||||
│ Interface │ │ Interface │ │ Interface │
|
||||
│ (RECEIVER) │ │ (RECEIVER) │ │ (SENDER) │
|
||||
└────────┬────────┘ └────────┬────────┘ └────────▲────────┘
|
||||
│ │ │
|
||||
│ Publish │ Publish │ Subscribe
|
||||
│ │ │
|
||||
└───────────┬───────────┴───────────────────────┘
|
||||
│
|
||||
┌──────▼──────┐
|
||||
│ MQTT │
|
||||
│ Broker │
|
||||
└──────┬──────┘
|
||||
│
|
||||
┌──────▼──────┐
|
||||
│ Collector │
|
||||
└──────┬──────┘
|
||||
│
|
||||
┌──────▼──────┐
|
||||
│ Database │
|
||||
└──────┬──────┘
|
||||
│
|
||||
┌───────────┴───────────┐
|
||||
│ │
|
||||
┌──────▼──────┐ ┌───────▼───────┐
|
||||
│ API │◄──────│ Web Dashboard │
|
||||
└─────────────┘ └───────────────┘
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- **Multi-node Support**: Connect multiple receiver nodes for better network coverage
|
||||
- **Event Persistence**: Store messages, advertisements, telemetry, and trace data
|
||||
- **REST API**: Query historical data with filtering and pagination
|
||||
- **Command Dispatch**: Send messages and advertisements via the API
|
||||
- **Node Tagging**: Add custom metadata to nodes for organization
|
||||
- **Web Dashboard**: Visualize network status, node locations, and message history
|
||||
- **Docker Ready**: Single image with all components, easy deployment
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Using Docker Compose (Recommended)
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/your-org/meshcore-hub.git
|
||||
cd meshcore-hub
|
||||
|
||||
# Start all services
|
||||
docker compose -f docker/docker-compose.yml up -d
|
||||
|
||||
# View logs
|
||||
docker compose -f docker/docker-compose.yml logs -f
|
||||
```
|
||||
|
||||
### Manual Installation
|
||||
|
||||
```bash
|
||||
# Create virtual environment
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate # Linux/macOS
|
||||
# .venv\Scripts\activate # Windows
|
||||
|
||||
# Install the package
|
||||
pip install -e ".[dev]"
|
||||
|
||||
# Run database migrations
|
||||
meshcore-hub db upgrade
|
||||
|
||||
# Start components (in separate terminals)
|
||||
meshcore-hub interface --mode receiver --port /dev/ttyUSB0
|
||||
meshcore-hub collector
|
||||
meshcore-hub api
|
||||
meshcore-hub web
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
All components are configured via environment variables. Create a `.env` file or export variables:
|
||||
|
||||
### Common Settings
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `LOG_LEVEL` | `INFO` | Logging level (DEBUG, INFO, WARNING, ERROR) |
|
||||
| `MQTT_HOST` | `localhost` | MQTT broker hostname |
|
||||
| `MQTT_PORT` | `1883` | MQTT broker port |
|
||||
| `MQTT_PREFIX` | `meshcore` | Topic prefix for all MQTT messages |
|
||||
|
||||
### Interface Settings
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `INTERFACE_MODE` | `RECEIVER` | Operating mode (RECEIVER or SENDER) |
|
||||
| `SERIAL_PORT` | `/dev/ttyUSB0` | Serial port for MeshCore device |
|
||||
| `SERIAL_BAUD` | `115200` | Serial baud rate |
|
||||
| `MOCK_DEVICE` | `false` | Use mock device for testing |
|
||||
|
||||
### Collector Settings
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `DATABASE_URL` | `sqlite:///./meshcore.db` | SQLAlchemy database URL |
|
||||
|
||||
### API Settings
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `API_HOST` | `0.0.0.0` | API bind address |
|
||||
| `API_PORT` | `8000` | API port |
|
||||
| `API_READ_KEY` | *(none)* | Read-only API key |
|
||||
| `API_ADMIN_KEY` | *(none)* | Admin API key (required for commands) |
|
||||
|
||||
### Web Dashboard Settings
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `WEB_HOST` | `0.0.0.0` | Web server bind address |
|
||||
| `WEB_PORT` | `8080` | Web server port |
|
||||
| `API_BASE_URL` | `http://localhost:8000` | API endpoint URL |
|
||||
| `NETWORK_NAME` | `MeshCore Network` | Display name for the network |
|
||||
| `NETWORK_CITY` | *(none)* | City where network is located |
|
||||
| `NETWORK_COUNTRY` | *(none)* | Country code (ISO 3166-1 alpha-2) |
|
||||
| `NETWORK_LOCATION` | *(none)* | Center coordinates (lat,lon) |
|
||||
|
||||
## CLI Reference
|
||||
|
||||
```bash
|
||||
# Show help
|
||||
meshcore-hub --help
|
||||
|
||||
# Interface component
|
||||
meshcore-hub interface --mode receiver --port /dev/ttyUSB0
|
||||
meshcore-hub interface --mode sender --mock # Use mock device
|
||||
|
||||
# Collector component
|
||||
meshcore-hub collector --database-url sqlite:///./data.db
|
||||
|
||||
# API component
|
||||
meshcore-hub api --host 0.0.0.0 --port 8000
|
||||
|
||||
# Web dashboard
|
||||
meshcore-hub web --port 8080 --network-name "My Network"
|
||||
|
||||
# Database management
|
||||
meshcore-hub db upgrade # Run migrations
|
||||
meshcore-hub db downgrade # Rollback one migration
|
||||
meshcore-hub db current # Show current revision
|
||||
```
|
||||
|
||||
## API Documentation
|
||||
|
||||
When running, the API provides interactive documentation at:
|
||||
|
||||
- **Swagger UI**: http://localhost:8000/docs
|
||||
- **ReDoc**: http://localhost:8000/redoc
|
||||
- **OpenAPI JSON**: http://localhost:8000/openapi.json
|
||||
|
||||
### Authentication
|
||||
|
||||
The API supports optional bearer token authentication:
|
||||
|
||||
```bash
|
||||
# Read-only access
|
||||
curl -H "Authorization: Bearer <API_READ_KEY>" http://localhost:8000/api/v1/nodes
|
||||
|
||||
# Admin access (required for commands)
|
||||
curl -X POST \
|
||||
-H "Authorization: Bearer <API_ADMIN_KEY>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"destination": "abc123...", "text": "Hello!"}' \
|
||||
http://localhost:8000/api/v1/commands/send-message
|
||||
```
|
||||
|
||||
### Example Endpoints
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| GET | `/api/v1/nodes` | List all known nodes |
|
||||
| GET | `/api/v1/nodes/{public_key}` | Get node details |
|
||||
| GET | `/api/v1/nodes/{public_key}/tags` | Get node tags |
|
||||
| POST | `/api/v1/nodes/{public_key}/tags` | Create node tag |
|
||||
| GET | `/api/v1/messages` | List messages with filters |
|
||||
| GET | `/api/v1/advertisements` | List advertisements |
|
||||
| GET | `/api/v1/telemetry` | List telemetry data |
|
||||
| GET | `/api/v1/trace-paths` | List trace paths |
|
||||
| POST | `/api/v1/commands/send-message` | Send direct message |
|
||||
| POST | `/api/v1/commands/send-channel-message` | Send channel message |
|
||||
| GET | `/api/v1/stats` | Get network statistics |
|
||||
|
||||
## Development
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
# Clone and setup
|
||||
git clone https://github.com/your-org/meshcore-hub.git
|
||||
cd meshcore-hub
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -e ".[dev]"
|
||||
|
||||
# Install pre-commit hooks
|
||||
pre-commit install
|
||||
```
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
pytest
|
||||
|
||||
# Run with coverage
|
||||
pytest --cov=meshcore_hub --cov-report=html
|
||||
|
||||
# Run specific test file
|
||||
pytest tests/test_api/test_nodes.py
|
||||
|
||||
# Run tests matching pattern
|
||||
pytest -k "test_list"
|
||||
```
|
||||
|
||||
### Code Quality
|
||||
|
||||
```bash
|
||||
# Format code
|
||||
black src/ tests/
|
||||
|
||||
# Lint
|
||||
flake8 src/ tests/
|
||||
|
||||
# Type check
|
||||
mypy src/
|
||||
```
|
||||
|
||||
### Creating Database Migrations
|
||||
|
||||
```bash
|
||||
# Auto-generate migration from model changes
|
||||
meshcore-hub db revision --autogenerate -m "Add new field to nodes"
|
||||
|
||||
# Create empty migration
|
||||
meshcore-hub db revision -m "Custom migration"
|
||||
|
||||
# Apply migrations
|
||||
meshcore-hub db upgrade
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
meshcore-hub/
|
||||
├── src/meshcore_hub/ # Main package
|
||||
│ ├── common/ # Shared code (models, schemas, config)
|
||||
│ ├── interface/ # MeshCore device interface
|
||||
│ ├── collector/ # MQTT event collector
|
||||
│ ├── api/ # REST API
|
||||
│ └── web/ # Web dashboard
|
||||
├── tests/ # Test suite
|
||||
├── alembic/ # Database migrations
|
||||
├── docker/ # Docker configuration
|
||||
├── PROMPT.md # Project specification
|
||||
├── SCHEMAS.md # Event schema documentation
|
||||
├── PLAN.md # Implementation plan
|
||||
├── TASKS.md # Task tracker
|
||||
└── AGENTS.md # AI assistant guidelines
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
- [PROMPT.md](PROMPT.md) - Original project specification
|
||||
- [SCHEMAS.md](SCHEMAS.md) - MeshCore event schemas
|
||||
- [PLAN.md](PLAN.md) - Architecture and implementation plan
|
||||
- [TASKS.md](TASKS.md) - Development task tracker
|
||||
- [AGENTS.md](AGENTS.md) - Guidelines for AI coding assistants
|
||||
|
||||
## Contributing
|
||||
|
||||
1. Fork the repository
|
||||
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
|
||||
3. Make your changes
|
||||
4. Run tests and linting (`pytest && black . && flake8`)
|
||||
5. Commit your changes (`git commit -m 'Add amazing feature'`)
|
||||
6. Push to the branch (`git push origin feature/amazing-feature`)
|
||||
7. Open a Pull Request
|
||||
|
||||
## License
|
||||
|
||||
See [LICENSE](LICENSE) for details.
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
- [MeshCore](https://meshcore.dev/) - The mesh networking protocol
|
||||
- [meshcore_py](https://github.com/meshcore-dev/meshcore_py) - Python library for MeshCore devices
|
||||
|
||||
787
TASKS.md
Normal file
787
TASKS.md
Normal file
@@ -0,0 +1,787 @@
|
||||
# MeshCore Hub - Task Tracker
|
||||
|
||||
This document tracks implementation progress for the MeshCore Hub project. Each task can be checked off as completed.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Foundation
|
||||
|
||||
### 1.1 Project Setup
|
||||
|
||||
- [ ] Create `pyproject.toml` with project metadata and dependencies
|
||||
- [ ] Configure Python 3.11+ requirement
|
||||
- [ ] Set up `src/meshcore_hub/` package structure
|
||||
- [ ] Create `__init__.py` files for all packages
|
||||
- [ ] Create `__main__.py` entry point
|
||||
|
||||
### 1.2 Development Tools
|
||||
|
||||
- [ ] Configure `black` formatter settings in pyproject.toml
|
||||
- [ ] Configure `flake8` linting (create `.flake8` or add to pyproject.toml)
|
||||
- [ ] Configure `mypy` type checking settings
|
||||
- [ ] Configure `pytest` settings and test directory
|
||||
- [ ] Create `.pre-commit-config.yaml` with hooks:
|
||||
- [ ] black
|
||||
- [ ] flake8
|
||||
- [ ] mypy
|
||||
- [ ] trailing whitespace
|
||||
- [ ] end-of-file-fixer
|
||||
- [ ] Create `.env.example` with all environment variables
|
||||
|
||||
### 1.3 Common Package - Configuration
|
||||
|
||||
- [ ] Create `common/config.py` with Pydantic Settings:
|
||||
- [ ] `CommonSettings` (logging, MQTT connection)
|
||||
- [ ] `InterfaceSettings` (mode, serial port, mock device)
|
||||
- [ ] `CollectorSettings` (database URL, webhook settings)
|
||||
- [ ] `APISettings` (host, port, API keys)
|
||||
- [ ] `WebSettings` (host, port, network info)
|
||||
- [ ] Implement environment variable loading
|
||||
- [ ] Implement CLI argument override support
|
||||
- [ ] Add configuration validation
|
||||
|
||||
### 1.4 Common Package - Database Models
|
||||
|
||||
- [ ] Create `common/database.py`:
|
||||
- [ ] Database engine factory
|
||||
- [ ] Session management
|
||||
- [ ] Async session support
|
||||
- [ ] Create `common/models/base.py`:
|
||||
- [ ] Base model with UUID primary key
|
||||
- [ ] Timestamp mixins (created_at, updated_at)
|
||||
- [ ] Create `common/models/node.py`:
|
||||
- [ ] Node model (public_key, name, adv_type, flags, first_seen, last_seen)
|
||||
- [ ] Indexes on public_key
|
||||
- [ ] Create `common/models/node_tag.py`:
|
||||
- [ ] NodeTag model (node_id FK, key, value, value_type)
|
||||
- [ ] Unique constraint on (node_id, key)
|
||||
- [ ] Create `common/models/message.py`:
|
||||
- [ ] Message model (receiver_node_id, message_type, pubkey_prefix, channel_idx, text, etc.)
|
||||
- [ ] Indexes for common query patterns
|
||||
- [ ] Create `common/models/advertisement.py`:
|
||||
- [ ] Advertisement model (receiver_node_id, node_id, public_key, name, adv_type, flags)
|
||||
- [ ] Create `common/models/trace_path.py`:
|
||||
- [ ] TracePath model (receiver_node_id, initiator_tag, path_hashes JSON, snr_values JSON)
|
||||
- [ ] Create `common/models/telemetry.py`:
|
||||
- [ ] Telemetry model (receiver_node_id, node_id, node_public_key, lpp_data, parsed_data JSON)
|
||||
- [ ] Create `common/models/event_log.py`:
|
||||
- [ ] EventLog model (receiver_node_id, event_type, payload JSON)
|
||||
- [ ] Create `common/models/__init__.py` exporting all models
|
||||
|
||||
### 1.5 Common Package - Pydantic Schemas
|
||||
|
||||
- [ ] Create `common/schemas/events.py`:
|
||||
- [ ] AdvertisementEvent schema
|
||||
- [ ] ContactMessageEvent schema
|
||||
- [ ] ChannelMessageEvent schema
|
||||
- [ ] TraceDataEvent schema
|
||||
- [ ] TelemetryResponseEvent schema
|
||||
- [ ] ContactsEvent schema
|
||||
- [ ] SendConfirmedEvent schema
|
||||
- [ ] StatusResponseEvent schema
|
||||
- [ ] BatteryEvent schema
|
||||
- [ ] PathUpdatedEvent schema
|
||||
- [ ] Create `common/schemas/nodes.py`:
|
||||
- [ ] NodeCreate, NodeRead, NodeList schemas
|
||||
- [ ] NodeTagCreate, NodeTagUpdate, NodeTagRead schemas
|
||||
- [ ] Create `common/schemas/messages.py`:
|
||||
- [ ] MessageRead, MessageList schemas
|
||||
- [ ] MessageFilters schema
|
||||
- [ ] Create `common/schemas/commands.py`:
|
||||
- [ ] SendMessageCommand schema
|
||||
- [ ] SendChannelMessageCommand schema
|
||||
- [ ] SendAdvertCommand schema
|
||||
- [ ] Create `common/schemas/__init__.py` exporting all schemas
|
||||
|
||||
### 1.6 Common Package - Utilities
|
||||
|
||||
- [ ] Create `common/mqtt.py`:
|
||||
- [ ] MQTT client factory function
|
||||
- [ ] Topic builder utilities
|
||||
- [ ] Message serialization helpers
|
||||
- [ ] Async publish/subscribe wrappers
|
||||
- [ ] Create `common/logging.py`:
|
||||
- [ ] Logging configuration function
|
||||
- [ ] Structured logging format
|
||||
- [ ] Log level configuration from settings
|
||||
|
||||
### 1.7 Database Migrations
|
||||
|
||||
- [ ] Create `alembic.ini` configuration
|
||||
- [ ] Create `alembic/env.py` with async support
|
||||
- [ ] Create `alembic/script.py.mako` template
|
||||
- [ ] Create initial migration with all tables:
|
||||
- [ ] nodes table
|
||||
- [ ] node_tags table
|
||||
- [ ] messages table
|
||||
- [ ] advertisements table
|
||||
- [ ] trace_paths table
|
||||
- [ ] telemetry table
|
||||
- [ ] events_log table
|
||||
- [ ] Test migration upgrade/downgrade
|
||||
|
||||
### 1.8 Main CLI Entry Point
|
||||
|
||||
- [ ] Create root Click group in `__main__.py`
|
||||
- [ ] Add `--version` option
|
||||
- [ ] Add `--config` option for config file path
|
||||
- [ ] Add subcommand placeholders for: interface, collector, api, web, db
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Interface Component
|
||||
|
||||
### 2.1 Device Abstraction
|
||||
|
||||
- [ ] Create `interface/device.py`:
|
||||
- [ ] `MeshCoreDevice` class wrapping meshcore_py
|
||||
- [ ] Connection management (connect, disconnect, reconnect)
|
||||
- [ ] Get device public key via `send_appstart()`
|
||||
- [ ] Event subscription registration
|
||||
- [ ] Command sending methods
|
||||
- [ ] Create `interface/mock_device.py`:
|
||||
- [ ] `MockMeshCoreDevice` class
|
||||
- [ ] Configurable event generation
|
||||
- [ ] Simulated message sending
|
||||
- [ ] Simulated network topology (optional)
|
||||
- [ ] Configurable delays and error rates
|
||||
|
||||
### 2.2 Receiver Mode
|
||||
|
||||
- [ ] Create `interface/receiver.py`:
|
||||
- [ ] `Receiver` class
|
||||
- [ ] Initialize MQTT client
|
||||
- [ ] Initialize MeshCore device
|
||||
- [ ] Subscribe to all relevant MeshCore events:
|
||||
- [ ] ADVERTISEMENT
|
||||
- [ ] CONTACT_MSG_RECV
|
||||
- [ ] CHANNEL_MSG_RECV
|
||||
- [ ] TRACE_DATA
|
||||
- [ ] TELEMETRY_RESPONSE
|
||||
- [ ] CONTACTS
|
||||
- [ ] SEND_CONFIRMED
|
||||
- [ ] STATUS_RESPONSE
|
||||
- [ ] BATTERY
|
||||
- [ ] PATH_UPDATED
|
||||
- [ ] Event handler that publishes to MQTT
|
||||
- [ ] Topic construction: `<prefix>/<pubkey>/event/<event_name>`
|
||||
- [ ] JSON serialization of event payloads
|
||||
- [ ] Graceful shutdown handling
|
||||
|
||||
### 2.3 Sender Mode
|
||||
|
||||
- [ ] Create `interface/sender.py`:
|
||||
- [ ] `Sender` class
|
||||
- [ ] Initialize MQTT client
|
||||
- [ ] Initialize MeshCore device
|
||||
- [ ] Subscribe to command topics:
|
||||
- [ ] `<prefix>/+/command/send_msg`
|
||||
- [ ] `<prefix>/+/command/send_channel_msg`
|
||||
- [ ] `<prefix>/+/command/send_advert`
|
||||
- [ ] `<prefix>/+/command/request_status`
|
||||
- [ ] `<prefix>/+/command/request_telemetry`
|
||||
- [ ] Command handlers:
|
||||
- [ ] `handle_send_msg` - send direct message
|
||||
- [ ] `handle_send_channel_msg` - send channel message
|
||||
- [ ] `handle_send_advert` - send advertisement
|
||||
- [ ] `handle_request_status` - request node status
|
||||
- [ ] `handle_request_telemetry` - request telemetry
|
||||
- [ ] Error handling and logging
|
||||
- [ ] Graceful shutdown handling
|
||||
|
||||
### 2.4 Interface CLI
|
||||
|
||||
- [ ] Create `interface/cli.py`:
|
||||
- [ ] `interface` Click command group
|
||||
- [ ] `--mode` option (receiver/sender, required)
|
||||
- [ ] `--port` option for serial port
|
||||
- [ ] `--baud` option for baud rate
|
||||
- [ ] `--mock` flag to use mock device
|
||||
- [ ] `--mqtt-host`, `--mqtt-port` options
|
||||
- [ ] `--prefix` option for MQTT topic prefix
|
||||
- [ ] Signal handlers for graceful shutdown
|
||||
- [ ] Register CLI with main entry point
|
||||
|
||||
### 2.5 Interface Tests
|
||||
|
||||
- [ ] Create `tests/test_interface/conftest.py`:
|
||||
- [ ] Mock MQTT client fixture
|
||||
- [ ] Mock device fixture
|
||||
- [ ] Create `tests/test_interface/test_device.py`:
|
||||
- [ ] Test connection/disconnection
|
||||
- [ ] Test event subscription
|
||||
- [ ] Test command sending
|
||||
- [ ] Create `tests/test_interface/test_mock_device.py`:
|
||||
- [ ] Test mock event generation
|
||||
- [ ] Test mock command handling
|
||||
- [ ] Create `tests/test_interface/test_receiver.py`:
|
||||
- [ ] Test event to MQTT publishing
|
||||
- [ ] Test topic construction
|
||||
- [ ] Test payload serialization
|
||||
- [ ] Create `tests/test_interface/test_sender.py`:
|
||||
- [ ] Test MQTT to command dispatching
|
||||
- [ ] Test command payload parsing
|
||||
- [ ] Test error handling
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Collector Component
|
||||
|
||||
### 3.1 MQTT Subscriber
|
||||
|
||||
- [ ] Create `collector/subscriber.py`:
|
||||
- [ ] `Subscriber` class
|
||||
- [ ] Initialize MQTT client
|
||||
- [ ] Subscribe to all event topics: `<prefix>/+/event/#`
|
||||
- [ ] Parse topic to extract public_key and event_type
|
||||
- [ ] Route events to appropriate handlers
|
||||
- [ ] Handle connection/disconnection
|
||||
- [ ] Graceful shutdown
|
||||
|
||||
### 3.2 Event Handlers
|
||||
|
||||
- [ ] Create `collector/handlers/__init__.py`:
|
||||
- [ ] Handler registry pattern
|
||||
- [ ] Create `collector/handlers/advertisement.py`:
|
||||
- [ ] Parse advertisement payload
|
||||
- [ ] Upsert node in nodes table
|
||||
- [ ] Insert advertisement record
|
||||
- [ ] Update node last_seen timestamp
|
||||
- [ ] Create `collector/handlers/message.py`:
|
||||
- [ ] Parse contact/channel message payload
|
||||
- [ ] Insert message record
|
||||
- [ ] Handle both CONTACT_MSG_RECV and CHANNEL_MSG_RECV
|
||||
- [ ] Create `collector/handlers/trace.py`:
|
||||
- [ ] Parse trace data payload
|
||||
- [ ] Insert trace_path record
|
||||
- [ ] Create `collector/handlers/telemetry.py`:
|
||||
- [ ] Parse telemetry payload
|
||||
- [ ] Insert telemetry record
|
||||
- [ ] Optionally upsert node
|
||||
- [ ] Create `collector/handlers/contacts.py`:
|
||||
- [ ] Parse contacts sync payload
|
||||
- [ ] Upsert multiple nodes
|
||||
- [ ] Create `collector/handlers/event_log.py`:
|
||||
- [ ] Generic handler for events_log table
|
||||
- [ ] Handle informational events (SEND_CONFIRMED, STATUS_RESPONSE, BATTERY, PATH_UPDATED)
|
||||
|
||||
### 3.3 Webhook Dispatcher (Optional based on Q10)
|
||||
|
||||
- [ ] Create `collector/webhook.py`:
|
||||
- [ ] `WebhookDispatcher` class
|
||||
- [ ] Webhook configuration loading
|
||||
- [ ] JSONPath filtering support
|
||||
- [ ] Async HTTP POST sending
|
||||
- [ ] Retry logic with backoff
|
||||
- [ ] Error logging
|
||||
|
||||
### 3.4 Collector CLI
|
||||
|
||||
- [ ] Create `collector/cli.py`:
|
||||
- [ ] `collector` Click command
|
||||
- [ ] `--mqtt-host`, `--mqtt-port` options
|
||||
- [ ] `--prefix` option
|
||||
- [ ] `--database-url` option
|
||||
- [ ] Signal handlers for graceful shutdown
|
||||
- [ ] Register CLI with main entry point
|
||||
|
||||
### 3.5 Collector Tests
|
||||
|
||||
- [ ] Create `tests/test_collector/conftest.py`:
|
||||
- [ ] In-memory SQLite database fixture
|
||||
- [ ] Mock MQTT client fixture
|
||||
- [ ] Create `tests/test_collector/test_subscriber.py`:
|
||||
- [ ] Test topic parsing
|
||||
- [ ] Test event routing
|
||||
- [ ] Create `tests/test_collector/test_handlers/`:
|
||||
- [ ] `test_advertisement.py`
|
||||
- [ ] `test_message.py`
|
||||
- [ ] `test_trace.py`
|
||||
- [ ] `test_telemetry.py`
|
||||
- [ ] `test_contacts.py`
|
||||
- [ ] Create `tests/test_collector/test_webhook.py`:
|
||||
- [ ] Test webhook dispatching
|
||||
- [ ] Test JSONPath filtering
|
||||
- [ ] Test retry logic
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: API Component
|
||||
|
||||
### 4.1 FastAPI Application Setup
|
||||
|
||||
- [ ] Create `api/app.py`:
|
||||
- [ ] FastAPI application instance
|
||||
- [ ] Lifespan handler for startup/shutdown
|
||||
- [ ] Include all routers
|
||||
- [ ] Exception handlers
|
||||
- [ ] CORS middleware configuration
|
||||
- [ ] Create `api/dependencies.py`:
|
||||
- [ ] Database session dependency
|
||||
- [ ] MQTT client dependency
|
||||
- [ ] Settings dependency
|
||||
|
||||
### 4.2 Authentication
|
||||
|
||||
- [ ] Create `api/auth.py`:
|
||||
- [ ] Bearer token extraction
|
||||
- [ ] `require_read` dependency (read or admin key)
|
||||
- [ ] `require_admin` dependency (admin key only)
|
||||
- [ ] 401/403 error responses
|
||||
|
||||
### 4.3 Node Routes
|
||||
|
||||
- [ ] Create `api/routes/nodes.py`:
|
||||
- [ ] `GET /api/v1/nodes` - list nodes with pagination
|
||||
- [ ] Query params: limit, offset, search, adv_type
|
||||
- [ ] `GET /api/v1/nodes/{public_key}` - get single node
|
||||
- [ ] Include related tags in response (optional)
|
||||
|
||||
### 4.4 Node Tag Routes
|
||||
|
||||
- [ ] Create `api/routes/node_tags.py`:
|
||||
- [ ] `GET /api/v1/nodes/{public_key}/tags` - list tags
|
||||
- [ ] `POST /api/v1/nodes/{public_key}/tags` - create tag (admin)
|
||||
- [ ] `PUT /api/v1/nodes/{public_key}/tags/{key}` - update tag (admin)
|
||||
- [ ] `DELETE /api/v1/nodes/{public_key}/tags/{key}` - delete tag (admin)
|
||||
|
||||
### 4.5 Message Routes
|
||||
|
||||
- [ ] Create `api/routes/messages.py`:
|
||||
- [ ] `GET /api/v1/messages` - list messages with filters
|
||||
- [ ] Query params: type, pubkey_prefix, channel_idx, since, until, limit, offset
|
||||
- [ ] `GET /api/v1/messages/{id}` - get single message
|
||||
|
||||
### 4.6 Advertisement Routes
|
||||
|
||||
- [ ] Create `api/routes/advertisements.py`:
|
||||
- [ ] `GET /api/v1/advertisements` - list advertisements
|
||||
- [ ] Query params: public_key, since, until, limit, offset
|
||||
- [ ] `GET /api/v1/advertisements/{id}` - get single advertisement
|
||||
|
||||
### 4.7 Trace Path Routes
|
||||
|
||||
- [ ] Create `api/routes/trace_paths.py`:
|
||||
- [ ] `GET /api/v1/trace-paths` - list trace paths
|
||||
- [ ] Query params: since, until, limit, offset
|
||||
- [ ] `GET /api/v1/trace-paths/{id}` - get single trace path
|
||||
|
||||
### 4.8 Telemetry Routes
|
||||
|
||||
- [ ] Create `api/routes/telemetry.py`:
|
||||
- [ ] `GET /api/v1/telemetry` - list telemetry records
|
||||
- [ ] Query params: node_public_key, since, until, limit, offset
|
||||
- [ ] `GET /api/v1/telemetry/{id}` - get single telemetry record
|
||||
|
||||
### 4.9 Command Routes
|
||||
|
||||
- [ ] Create `api/routes/commands.py`:
|
||||
- [ ] `POST /api/v1/commands/send-message` (admin)
|
||||
- [ ] Request body: destination, text, timestamp (optional)
|
||||
- [ ] Publish to MQTT command topic
|
||||
- [ ] `POST /api/v1/commands/send-channel-message` (admin)
|
||||
- [ ] Request body: channel_idx, text, timestamp (optional)
|
||||
- [ ] Publish to MQTT command topic
|
||||
- [ ] `POST /api/v1/commands/send-advertisement` (admin)
|
||||
- [ ] Request body: flood (boolean)
|
||||
- [ ] Publish to MQTT command topic
|
||||
|
||||
### 4.10 Dashboard Routes
|
||||
|
||||
- [ ] Create `api/routes/dashboard.py`:
|
||||
- [ ] `GET /api/v1/stats` - JSON statistics
|
||||
- [ ] Total nodes count
|
||||
- [ ] Active nodes (last 24h)
|
||||
- [ ] Total messages count
|
||||
- [ ] Messages today
|
||||
- [ ] Total advertisements
|
||||
- [ ] Channel message counts
|
||||
- [ ] `GET /api/v1/dashboard` - HTML dashboard
|
||||
- [ ] Create `api/templates/dashboard.html`:
|
||||
- [ ] Simple HTML template
|
||||
- [ ] Display statistics
|
||||
- [ ] Basic CSS styling
|
||||
- [ ] Auto-refresh meta tag (optional)
|
||||
|
||||
### 4.11 API Router Registration
|
||||
|
||||
- [ ] Create `api/routes/__init__.py`:
|
||||
- [ ] Create main API router
|
||||
- [ ] Include all sub-routers with prefixes
|
||||
- [ ] Add OpenAPI tags
|
||||
|
||||
### 4.12 API CLI
|
||||
|
||||
- [ ] Create `api/cli.py`:
|
||||
- [ ] `api` Click command
|
||||
- [ ] `--host` option
|
||||
- [ ] `--port` option
|
||||
- [ ] `--database-url` option
|
||||
- [ ] `--read-key` option
|
||||
- [ ] `--admin-key` option
|
||||
- [ ] `--mqtt-host`, `--mqtt-port` options
|
||||
- [ ] `--reload` flag for development
|
||||
- [ ] Register CLI with main entry point
|
||||
|
||||
### 4.13 API Tests
|
||||
|
||||
- [ ] Create `tests/test_api/conftest.py`:
|
||||
- [ ] Test client fixture
|
||||
- [ ] In-memory database fixture
|
||||
- [ ] Test API keys
|
||||
- [ ] Create `tests/test_api/test_auth.py`:
|
||||
- [ ] Test missing token
|
||||
- [ ] Test invalid token
|
||||
- [ ] Test read-only access
|
||||
- [ ] Test admin access
|
||||
- [ ] Create `tests/test_api/test_nodes.py`:
|
||||
- [ ] Test list nodes
|
||||
- [ ] Test get node
|
||||
- [ ] Test pagination
|
||||
- [ ] Test filtering
|
||||
- [ ] Create `tests/test_api/test_node_tags.py`:
|
||||
- [ ] Test CRUD operations
|
||||
- [ ] Test permission checks
|
||||
- [ ] Create `tests/test_api/test_messages.py`:
|
||||
- [ ] Test list messages
|
||||
- [ ] Test filtering
|
||||
- [ ] Create `tests/test_api/test_commands.py`:
|
||||
- [ ] Test send message command
|
||||
- [ ] Test permission checks
|
||||
- [ ] Test MQTT publishing
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Web Dashboard
|
||||
|
||||
### 5.1 FastAPI Application Setup
|
||||
|
||||
- [ ] Create `web/app.py`:
|
||||
- [ ] FastAPI application instance
|
||||
- [ ] Jinja2 templates configuration
|
||||
- [ ] Static files mounting
|
||||
- [ ] Lifespan handler
|
||||
- [ ] Include all routers
|
||||
|
||||
### 5.2 Frontend Assets
|
||||
|
||||
- [ ] Create `web/static/css/` directory
|
||||
- [ ] Set up Tailwind CSS:
|
||||
- [ ] Create `tailwind.config.js`
|
||||
- [ ] Create source CSS with Tailwind directives
|
||||
- [ ] Configure DaisyUI plugin
|
||||
- [ ] Build pipeline (npm script or standalone CLI)
|
||||
- [ ] Create `web/static/js/` directory:
|
||||
- [ ] Minimal JS for interactivity (if needed)
|
||||
|
||||
### 5.3 Base Template
|
||||
|
||||
- [ ] Create `web/templates/base.html`:
|
||||
- [ ] HTML5 doctype and structure
|
||||
- [ ] Meta tags (viewport, charset)
|
||||
- [ ] Tailwind CSS inclusion
|
||||
- [ ] Navigation header:
|
||||
- [ ] Network name
|
||||
- [ ] Links to all pages
|
||||
- [ ] Footer with contact info
|
||||
- [ ] Content block for page content
|
||||
|
||||
### 5.4 Home Page
|
||||
|
||||
- [ ] Create `web/routes/home.py`:
|
||||
- [ ] `GET /` - home page route
|
||||
- [ ] Load network configuration
|
||||
- [ ] Create `web/templates/home.html`:
|
||||
- [ ] Welcome message with network name
|
||||
- [ ] Network description/details
|
||||
- [ ] Radio configuration display
|
||||
- [ ] Location information
|
||||
- [ ] Contact information (email, Discord)
|
||||
- [ ] Quick links to other sections
|
||||
|
||||
### 5.5 Members Page
|
||||
|
||||
- [ ] Create `web/routes/members.py`:
|
||||
- [ ] `GET /members` - members list route
|
||||
- [ ] Load members from JSON file
|
||||
- [ ] Create `web/templates/members.html`:
|
||||
- [ ] Members list/grid
|
||||
- [ ] Member cards with:
|
||||
- [ ] Name
|
||||
- [ ] Callsign (if applicable)
|
||||
- [ ] Role/description
|
||||
- [ ] Contact info (optional)
|
||||
|
||||
### 5.6 Network Overview Page
|
||||
|
||||
- [ ] Create `web/routes/network.py`:
|
||||
- [ ] `GET /network` - network stats route
|
||||
- [ ] Fetch stats from API
|
||||
- [ ] Create `web/templates/network.html`:
|
||||
- [ ] Statistics cards:
|
||||
- [ ] Total nodes
|
||||
- [ ] Active nodes
|
||||
- [ ] Total messages
|
||||
- [ ] Messages today
|
||||
- [ ] Channel statistics
|
||||
- [ ] Recent activity summary
|
||||
|
||||
### 5.7 Nodes Page
|
||||
|
||||
- [ ] Create `web/routes/nodes.py`:
|
||||
- [ ] `GET /nodes` - nodes list route
|
||||
- [ ] `GET /nodes/{public_key}` - node detail route
|
||||
- [ ] Fetch from API with pagination
|
||||
- [ ] Create `web/templates/nodes.html`:
|
||||
- [ ] Search/filter form
|
||||
- [ ] Nodes table:
|
||||
- [ ] Name
|
||||
- [ ] Public key (truncated)
|
||||
- [ ] Type
|
||||
- [ ] Last seen
|
||||
- [ ] Tags
|
||||
- [ ] Pagination controls
|
||||
- [ ] Create `web/templates/node_detail.html`:
|
||||
- [ ] Full node information
|
||||
- [ ] All tags
|
||||
- [ ] Recent messages (if any)
|
||||
- [ ] Recent advertisements
|
||||
|
||||
### 5.8 Node Map Page
|
||||
|
||||
- [ ] Create `web/routes/map.py`:
|
||||
- [ ] `GET /map` - map page route
|
||||
- [ ] `GET /map/data` - JSON endpoint for node locations
|
||||
- [ ] Filter nodes with location tags
|
||||
- [ ] Create `web/templates/map.html`:
|
||||
- [ ] Leaflet.js map container
|
||||
- [ ] Leaflet CSS/JS includes
|
||||
- [ ] JavaScript for:
|
||||
- [ ] Initialize map centered on NETWORK_LOCATION
|
||||
- [ ] Fetch node location data
|
||||
- [ ] Add markers for each node
|
||||
- [ ] Popup with node info on click
|
||||
|
||||
### 5.9 Messages Page
|
||||
|
||||
- [ ] Create `web/routes/messages.py`:
|
||||
- [ ] `GET /messages` - messages list route
|
||||
- [ ] Fetch from API with filters
|
||||
- [ ] Create `web/templates/messages.html`:
|
||||
- [ ] Filter form:
|
||||
- [ ] Message type (contact/channel)
|
||||
- [ ] Channel selector
|
||||
- [ ] Date range
|
||||
- [ ] Search text
|
||||
- [ ] Messages table:
|
||||
- [ ] Timestamp
|
||||
- [ ] Type
|
||||
- [ ] Sender/Channel
|
||||
- [ ] Text (truncated)
|
||||
- [ ] SNR
|
||||
- [ ] Hops
|
||||
- [ ] Pagination controls
|
||||
|
||||
### 5.10 Web CLI
|
||||
|
||||
- [ ] Create `web/cli.py`:
|
||||
- [ ] `web` Click command
|
||||
- [ ] `--host` option
|
||||
- [ ] `--port` option
|
||||
- [ ] `--api-url` option
|
||||
- [ ] `--api-key` option
|
||||
- [ ] Network configuration options:
|
||||
- [ ] `--network-name`
|
||||
- [ ] `--network-city`
|
||||
- [ ] `--network-country`
|
||||
- [ ] `--network-location`
|
||||
- [ ] `--network-radio-config`
|
||||
- [ ] `--network-contact-email`
|
||||
- [ ] `--network-contact-discord`
|
||||
- [ ] `--members-file` option
|
||||
- [ ] `--reload` flag for development
|
||||
- [ ] Register CLI with main entry point
|
||||
|
||||
### 5.11 Web Tests
|
||||
|
||||
- [ ] Create `tests/test_web/conftest.py`:
|
||||
- [ ] Test client fixture
|
||||
- [ ] Mock API responses
|
||||
- [ ] Create `tests/test_web/test_home.py`
|
||||
- [ ] Create `tests/test_web/test_members.py`
|
||||
- [ ] Create `tests/test_web/test_network.py`
|
||||
- [ ] Create `tests/test_web/test_nodes.py`
|
||||
- [ ] Create `tests/test_web/test_map.py`
|
||||
- [ ] Create `tests/test_web/test_messages.py`
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Docker & Deployment
|
||||
|
||||
### 6.1 Dockerfile
|
||||
|
||||
- [ ] Create `docker/Dockerfile`:
|
||||
- [ ] Multi-stage build:
|
||||
- [ ] Stage 1: Build frontend assets (Tailwind)
|
||||
- [ ] Stage 2: Python dependencies
|
||||
- [ ] Stage 3: Final runtime image
|
||||
- [ ] Base image: python:3.11-slim
|
||||
- [ ] Install system dependencies
|
||||
- [ ] Copy and install Python package
|
||||
- [ ] Copy frontend assets
|
||||
- [ ] Set entrypoint to `meshcore-hub`
|
||||
- [ ] Default CMD (show help)
|
||||
- [ ] Health check instruction
|
||||
|
||||
### 6.2 Docker Compose
|
||||
|
||||
- [ ] Create `docker/docker-compose.yml`:
|
||||
- [ ] MQTT broker service (Eclipse Mosquitto):
|
||||
- [ ] Port mapping (1883, 9001)
|
||||
- [ ] Volume for persistence
|
||||
- [ ] Configuration file
|
||||
- [ ] Interface Receiver service:
|
||||
- [ ] Depends on MQTT
|
||||
- [ ] Device passthrough (/dev/ttyUSB0)
|
||||
- [ ] Environment variables
|
||||
- [ ] Interface Sender service:
|
||||
- [ ] Depends on MQTT
|
||||
- [ ] Device passthrough
|
||||
- [ ] Environment variables
|
||||
- [ ] Collector service:
|
||||
- [ ] Depends on MQTT
|
||||
- [ ] Database volume
|
||||
- [ ] Environment variables
|
||||
- [ ] API service:
|
||||
- [ ] Depends on Collector (for DB)
|
||||
- [ ] Port mapping (8000)
|
||||
- [ ] Database volume (shared)
|
||||
- [ ] Environment variables
|
||||
- [ ] Web service:
|
||||
- [ ] Depends on API
|
||||
- [ ] Port mapping (8080)
|
||||
- [ ] Environment variables
|
||||
- [ ] Create `docker/mosquitto.conf`:
|
||||
- [ ] Listener configuration
|
||||
- [ ] Anonymous access (or auth)
|
||||
- [ ] Persistence settings
|
||||
|
||||
### 6.3 Health Checks
|
||||
|
||||
- [ ] Add health check endpoint to API:
|
||||
- [ ] `GET /health` - basic health
|
||||
- [ ] `GET /health/ready` - includes DB check
|
||||
- [ ] Add health check endpoint to Web:
|
||||
- [ ] `GET /health` - basic health
|
||||
- [ ] `GET /health/ready` - includes API connectivity
|
||||
- [ ] Add health check to Interface:
|
||||
- [ ] Device connection status
|
||||
- [ ] MQTT connection status
|
||||
- [ ] Add health check to Collector:
|
||||
- [ ] MQTT connection status
|
||||
- [ ] Database connection status
|
||||
|
||||
### 6.4 Database CLI Commands
|
||||
|
||||
- [ ] Create `db` Click command group:
|
||||
- [ ] `meshcore-hub db upgrade` - run migrations
|
||||
- [ ] `meshcore-hub db downgrade` - rollback migration
|
||||
- [ ] `meshcore-hub db revision -m "message"` - create migration
|
||||
- [ ] `meshcore-hub db current` - show current revision
|
||||
- [ ] `meshcore-hub db history` - show migration history
|
||||
|
||||
### 6.5 Documentation
|
||||
|
||||
- [ ] Update `README.md`:
|
||||
- [ ] Project description
|
||||
- [ ] Quick start guide
|
||||
- [ ] Docker deployment instructions
|
||||
- [ ] Manual installation instructions
|
||||
- [ ] Configuration reference
|
||||
- [ ] CLI reference
|
||||
- [ ] Create `docs/` directory (optional):
|
||||
- [ ] Architecture overview
|
||||
- [ ] API documentation link
|
||||
- [ ] Deployment guides
|
||||
|
||||
### 6.6 CI/CD (Optional)
|
||||
|
||||
- [ ] Create `.github/workflows/ci.yml`:
|
||||
- [ ] Run on push/PR
|
||||
- [ ] Set up Python
|
||||
- [ ] Install dependencies
|
||||
- [ ] Run linting (black, flake8)
|
||||
- [ ] Run type checking (mypy)
|
||||
- [ ] Run tests (pytest)
|
||||
- [ ] Upload coverage report
|
||||
- [ ] Create `.github/workflows/docker.yml`:
|
||||
- [ ] Build Docker image
|
||||
- [ ] Push to registry (on release)
|
||||
|
||||
### 6.7 End-to-End Testing
|
||||
|
||||
- [ ] Create `tests/e2e/` directory
|
||||
- [ ] Create `tests/e2e/docker-compose.test.yml`:
|
||||
- [ ] All services with mock device
|
||||
- [ ] Test database
|
||||
- [ ] Create `tests/e2e/test_full_flow.py`:
|
||||
- [ ] Start all services
|
||||
- [ ] Generate mock events
|
||||
- [ ] Verify events stored in database
|
||||
- [ ] Verify API returns events
|
||||
- [ ] Verify web dashboard displays data
|
||||
- [ ] Test command flow (API -> MQTT -> Sender)
|
||||
|
||||
---
|
||||
|
||||
## Progress Summary
|
||||
|
||||
| Phase | Total Tasks | Completed | Progress |
|
||||
|-------|-------------|-----------|----------|
|
||||
| Phase 1: Foundation | 47 | 0 | 0% |
|
||||
| Phase 2: Interface | 35 | 0 | 0% |
|
||||
| Phase 3: Collector | 27 | 0 | 0% |
|
||||
| Phase 4: API | 44 | 0 | 0% |
|
||||
| Phase 5: Web Dashboard | 40 | 0 | 0% |
|
||||
| Phase 6: Docker & Deployment | 28 | 0 | 0% |
|
||||
| **Total** | **221** | **0** | **0%** |
|
||||
|
||||
---
|
||||
|
||||
## Notes & Decisions
|
||||
|
||||
### Decisions Made
|
||||
*(Record architectural decisions and answers to clarifying questions here)*
|
||||
|
||||
- [ ] Q1 (MQTT Broker):
|
||||
- [ ] Q2 (Database):
|
||||
- [ ] Q3 (Web Dashboard Separation):
|
||||
- [ ] Q4 (Members JSON Location):
|
||||
- [ ] Q5 (Multiple Serial Devices):
|
||||
- [ ] Q6 (Reconnection Strategy):
|
||||
- [ ] Q7 (Mock Device Scope):
|
||||
- [ ] Q8 (Event Deduplication):
|
||||
- [ ] Q9 (Data Retention):
|
||||
- [ ] Q10 (Webhook Configuration):
|
||||
- [ ] Q11 (API Key Management):
|
||||
- [ ] Q12 (Rate Limiting):
|
||||
- [ ] Q13 (CORS):
|
||||
- [ ] Q14 (Dashboard Authentication):
|
||||
- [ ] Q15 (Real-time Updates):
|
||||
- [ ] Q16 (Map Provider):
|
||||
- [ ] Q17 (Tag Value Types):
|
||||
- [ ] Q18 (Reserved Tag Names):
|
||||
- [ ] Q19 (Health Checks):
|
||||
- [ ] Q20 (Metrics/Observability):
|
||||
- [ ] Q21 (Log Level Configuration):
|
||||
|
||||
### Blockers
|
||||
*(Track any blockers or dependencies here)*
|
||||
|
||||
### Session Log
|
||||
*(Track what was accomplished in each session)*
|
||||
|
||||
| Date | Session | Tasks Completed | Notes |
|
||||
|------|---------|-----------------|-------|
|
||||
| | | | |
|
||||
|
||||
Reference in New Issue
Block a user