Complete Phase 6: Docker deployment and CI/CD

Health Checks (6.3):
- Add is_healthy property and get_health_status() to Receiver/Sender
- Add is_healthy property and get_health_status() to Collector Subscriber
- Track device, MQTT, and database connection status

Documentation (6.5):
- Update README with Docker Compose profiles documentation
- Add serial device access instructions
- Update API documentation URLs and add health check info

CI/CD (6.6):
- Add .github/workflows/ci.yml for linting, testing, and building
- Add .github/workflows/docker.yml for Docker image builds
- Support Python 3.11 and 3.12 in CI matrix
- Configure Codecov for coverage reporting

End-to-End Testing (6.7):
- Add tests/e2e/ directory with Docker Compose test configuration
- Add e2e test fixtures with service health waiting
- Add comprehensive e2e tests for API, Web, and auth flows
This commit is contained in:
Claude
2025-12-03 15:38:02 +00:00
parent 84b57a211f
commit 50a3b5be19
10 changed files with 758 additions and 8 deletions

93
.github/workflows/ci.yml vendored Normal file
View File

@@ -0,0 +1,93 @@
name: CI
on:
push:
branches: [main, master]
pull_request:
branches: [main, master]
jobs:
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install black flake8 mypy
pip install -e ".[dev]"
- name: Check formatting with black
run: black --check src/ tests/
- name: Lint with flake8
run: flake8 src/ tests/
- name: Type check with mypy
run: mypy src/
test:
name: Test (Python ${{ matrix.python-version }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.11", "3.12"]
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e ".[dev]"
- name: Run tests with pytest
run: |
pytest --cov=meshcore_hub --cov-report=xml --cov-report=term-missing
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
if: matrix.python-version == '3.11'
with:
files: ./coverage.xml
fail_ci_if_error: false
verbose: true
build:
name: Build Package
runs-on: ubuntu-latest
needs: [lint, test]
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install build tools
run: |
python -m pip install --upgrade pip
pip install build
- name: Build package
run: python -m build
- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
name: dist
path: dist/

65
.github/workflows/docker.yml vendored Normal file
View File

@@ -0,0 +1,65 @@
name: Docker
on:
push:
branches: [main, master]
tags:
- "v*"
pull_request:
branches: [main, master]
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
build:
name: Build Docker Image
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to Container Registry
if: github.event_name != 'pull_request'
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata for Docker
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=ref,event=branch
type=ref,event=pr
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=sha
- name: Build and push Docker image
uses: docker/build-push-action@v5
with:
context: .
file: docker/Dockerfile
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Test Docker image
if: github.event_name == 'pull_request'
run: |
docker build -t meshcore-hub-test -f docker/Dockerfile .
docker run --rm meshcore-hub-test --version
docker run --rm meshcore-hub-test --help

View File

@@ -66,16 +66,61 @@ MeshCore Hub provides a complete solution for monitoring, collecting, and intera
### Using Docker Compose (Recommended)
Docker Compose supports **profiles** to selectively enable/disable components:
| Profile | Services |
|---------|----------|
| `mqtt` | Eclipse Mosquitto MQTT broker |
| `interface-receiver` | MeshCore device receiver (events to MQTT) |
| `interface-sender` | MeshCore device sender (MQTT to device) |
| `collector` | MQTT subscriber + database storage |
| `api` | REST API server |
| `web` | Web dashboard |
| `mock` | All services with mock device (for testing) |
| `all` | All production services |
```bash
# Clone the repository
git clone https://github.com/your-org/meshcore-hub.git
cd meshcore-hub
cd meshcore-hub/docker
# Start all services
docker compose -f docker/docker-compose.yml up -d
# Copy and configure environment
cp .env.example .env
# Edit .env with your settings (API keys, serial port, network info)
# Option 1: Start all services with mock device (for testing)
docker compose --profile mock up -d
# Option 2: Start specific services for production
docker compose --profile mqtt --profile collector --profile api --profile web up -d
# Option 3: Start all production services (requires real MeshCore device)
docker compose --profile all up -d
# View logs
docker compose -f docker/docker-compose.yml logs -f
docker compose logs -f
# Run database migrations
docker compose --profile migrate up
# Stop services
docker compose --profile mock down
```
#### Serial Device Access
For production with real MeshCore devices, ensure the serial port is accessible:
```bash
# Check device path
ls -la /dev/ttyUSB*
# Add user to dialout group (Linux)
sudo usermod -aG dialout $USER
# Configure in .env
SERIAL_PORT=/dev/ttyUSB0
SERIAL_PORT_SENDER=/dev/ttyUSB1 # If using separate sender device
```
### Manual Installation
@@ -177,9 +222,14 @@ meshcore-hub db current # Show current revision
When running, the API provides interactive documentation at:
- **Swagger UI**: http://localhost:8000/docs
- **ReDoc**: http://localhost:8000/redoc
- **OpenAPI JSON**: http://localhost:8000/openapi.json
- **Swagger UI**: http://localhost:8000/api/docs
- **ReDoc**: http://localhost:8000/api/redoc
- **OpenAPI JSON**: http://localhost:8000/api/openapi.json
Health check endpoints are also available:
- **Health**: http://localhost:8000/health
- **Ready**: http://localhost:8000/health/ready (includes database check)
### Authentication

View File

@@ -42,6 +42,30 @@ class Subscriber:
self._running = False
self._shutdown_event = threading.Event()
self._handlers: dict[str, EventHandler] = {}
self._mqtt_connected = False
self._db_connected = False
@property
def is_healthy(self) -> bool:
"""Check if the subscriber is healthy.
Returns:
True if MQTT and database are connected
"""
return self._running and self._mqtt_connected and self._db_connected
def get_health_status(self) -> dict[str, Any]:
"""Get detailed health status.
Returns:
Dictionary with health status details
"""
return {
"healthy": self.is_healthy,
"running": self._running,
"mqtt_connected": self._mqtt_connected,
"database_connected": self._db_connected,
}
def register_handler(self, event_type: str, handler: EventHandler) -> None:
"""Register a handler for an event type.
@@ -96,14 +120,23 @@ class Subscriber:
logger.info("Starting collector subscriber")
# Create database tables if needed
self.db.create_tables()
try:
self.db.create_tables()
self._db_connected = True
logger.info("Database initialized")
except Exception as e:
self._db_connected = False
logger.error(f"Failed to initialize database: {e}")
raise
# Connect to MQTT broker
try:
self.mqtt.connect()
self.mqtt.start_background()
self._mqtt_connected = True
logger.info("Connected to MQTT broker")
except Exception as e:
self._mqtt_connected = False
logger.error(f"Failed to connect to MQTT broker: {e}")
raise
@@ -141,6 +174,7 @@ class Subscriber:
# Stop MQTT
self.mqtt.stop()
self.mqtt.disconnect()
self._mqtt_connected = False
logger.info("Collector subscriber stopped")

View File

@@ -43,6 +43,31 @@ class Receiver:
self.mqtt = mqtt_client
self._running = False
self._shutdown_event = threading.Event()
self._device_connected = False
self._mqtt_connected = False
@property
def is_healthy(self) -> bool:
"""Check if the receiver is healthy.
Returns:
True if device and MQTT are connected
"""
return self._running and self._device_connected and self._mqtt_connected
def get_health_status(self) -> dict[str, Any]:
"""Get detailed health status.
Returns:
Dictionary with health status details
"""
return {
"healthy": self.is_healthy,
"running": self._running,
"device_connected": self._device_connected,
"mqtt_connected": self._mqtt_connected,
"device_public_key": self.device.public_key,
}
def _initialize_device(self) -> None:
"""Initialize device after connection.
@@ -108,18 +133,23 @@ class Receiver:
try:
self.mqtt.connect()
self.mqtt.start_background()
self._mqtt_connected = True
logger.info("Connected to MQTT broker")
except Exception as e:
self._mqtt_connected = False
logger.error(f"Failed to connect to MQTT broker: {e}")
raise
# Connect to device
if not self.device.connect():
self._device_connected = False
logger.error("Failed to connect to MeshCore device")
self.mqtt.stop()
self.mqtt.disconnect()
self._mqtt_connected = False
raise RuntimeError("Failed to connect to MeshCore device")
self._device_connected = True
logger.info(f"Connected to MeshCore device: {self.device.public_key}")
# Initialize device: set time and send local advertisement
@@ -154,10 +184,12 @@ class Receiver:
# Stop device
self.device.stop()
self.device.disconnect()
self._device_connected = False
# Stop MQTT
self.mqtt.stop()
self.mqtt.disconnect()
self._mqtt_connected = False
logger.info("Receiver stopped")

View File

@@ -42,6 +42,31 @@ class Sender:
self.mqtt = mqtt_client
self._running = False
self._shutdown_event = threading.Event()
self._device_connected = False
self._mqtt_connected = False
@property
def is_healthy(self) -> bool:
"""Check if the sender is healthy.
Returns:
True if device and MQTT are connected
"""
return self._running and self._device_connected and self._mqtt_connected
def get_health_status(self) -> dict[str, Any]:
"""Get detailed health status.
Returns:
Dictionary with health status details
"""
return {
"healthy": self.is_healthy,
"running": self._running,
"device_connected": self._device_connected,
"mqtt_connected": self._mqtt_connected,
"device_public_key": self.device.public_key,
}
def _handle_mqtt_message(
self,
@@ -175,19 +200,24 @@ class Sender:
# Connect to device first
if not self.device.connect():
self._device_connected = False
logger.error("Failed to connect to MeshCore device")
raise RuntimeError("Failed to connect to MeshCore device")
self._device_connected = True
logger.info(f"Connected to MeshCore device: {self.device.public_key}")
# Connect to MQTT broker
try:
self.mqtt.connect()
self.mqtt.start_background()
self._mqtt_connected = True
logger.info("Connected to MQTT broker")
except Exception as e:
self._mqtt_connected = False
logger.error(f"Failed to connect to MQTT broker: {e}")
self.device.disconnect()
self._device_connected = False
raise
# Subscribe to command topics
@@ -225,10 +255,12 @@ class Sender:
# Stop MQTT
self.mqtt.stop()
self.mqtt.disconnect()
self._mqtt_connected = False
# Stop device
self.device.stop()
self.device.disconnect()
self._device_connected = False
logger.info("Sender stopped")

1
tests/e2e/__init__.py Normal file
View File

@@ -0,0 +1 @@
"""End-to-end tests for MeshCore Hub."""

109
tests/e2e/conftest.py Normal file
View File

@@ -0,0 +1,109 @@
"""Fixtures for end-to-end tests."""
import os
import time
from typing import Generator
import httpx
import pytest
# E2E test configuration
E2E_API_URL = os.environ.get("E2E_API_URL", "http://localhost:18000")
E2E_WEB_URL = os.environ.get("E2E_WEB_URL", "http://localhost:18080")
E2E_MQTT_HOST = os.environ.get("E2E_MQTT_HOST", "localhost")
E2E_MQTT_PORT = int(os.environ.get("E2E_MQTT_PORT", "11883"))
E2E_READ_KEY = os.environ.get("E2E_READ_KEY", "test-read-key")
E2E_ADMIN_KEY = os.environ.get("E2E_ADMIN_KEY", "test-admin-key")
def wait_for_service(url: str, timeout: int = 60) -> bool:
"""Wait for a service to become available.
Args:
url: Health check URL
timeout: Maximum seconds to wait
Returns:
True if service is available, False if timeout
"""
start = time.time()
while time.time() - start < timeout:
try:
response = httpx.get(url, timeout=5.0)
if response.status_code == 200:
return True
except httpx.RequestError:
pass
time.sleep(1)
return False
@pytest.fixture(scope="session")
def api_url() -> str:
"""Get API base URL."""
return E2E_API_URL
@pytest.fixture(scope="session")
def web_url() -> str:
"""Get Web dashboard URL."""
return E2E_WEB_URL
@pytest.fixture(scope="session")
def read_key() -> str:
"""Get read API key."""
return E2E_READ_KEY
@pytest.fixture(scope="session")
def admin_key() -> str:
"""Get admin API key."""
return E2E_ADMIN_KEY
@pytest.fixture(scope="session")
def api_client(api_url: str, read_key: str) -> Generator[httpx.Client, None, None]:
"""Create an API client with read access.
This fixture waits for the API to be available before returning.
"""
health_url = f"{api_url}/health"
if not wait_for_service(health_url):
pytest.skip(f"API not available at {api_url}")
with httpx.Client(
base_url=api_url,
headers={"Authorization": f"Bearer {read_key}"},
timeout=30.0,
) as client:
yield client
@pytest.fixture(scope="session")
def admin_client(api_url: str, admin_key: str) -> Generator[httpx.Client, None, None]:
"""Create an API client with admin access."""
health_url = f"{api_url}/health"
if not wait_for_service(health_url):
pytest.skip(f"API not available at {api_url}")
with httpx.Client(
base_url=api_url,
headers={"Authorization": f"Bearer {admin_key}"},
timeout=30.0,
) as client:
yield client
@pytest.fixture(scope="session")
def web_client(web_url: str) -> Generator[httpx.Client, None, None]:
"""Create a web dashboard client."""
health_url = f"{web_url}/health"
if not wait_for_service(health_url):
pytest.skip(f"Web dashboard not available at {web_url}")
with httpx.Client(
base_url=web_url,
timeout=30.0,
) as client:
yield client

View File

@@ -0,0 +1,136 @@
# MeshCore Hub - End-to-End Test Docker Compose
#
# This configuration runs all services with a mock device for integration testing.
#
# Usage:
# docker compose -f tests/e2e/docker-compose.test.yml up -d
# pytest tests/e2e/
# docker compose -f tests/e2e/docker-compose.test.yml down -v
services:
# MQTT Broker
mqtt:
image: eclipse-mosquitto:2
container_name: meshcore-test-mqtt
ports:
- "11883:1883"
volumes:
- ../../docker/mosquitto.conf:/mosquitto/config/mosquitto.conf:ro
healthcheck:
test: ["CMD", "mosquitto_sub", "-t", "$$SYS/#", "-C", "1", "-i", "healthcheck", "-W", "3"]
interval: 5s
timeout: 5s
retries: 3
start_period: 5s
# Interface with mock device
interface-mock:
build:
context: ../..
dockerfile: docker/Dockerfile
container_name: meshcore-test-interface
depends_on:
mqtt:
condition: service_healthy
environment:
- LOG_LEVEL=DEBUG
- MQTT_HOST=mqtt
- MQTT_PORT=1883
- MQTT_PREFIX=test
- MOCK_DEVICE=true
- NODE_ADDRESS=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
command: ["interface", "receiver", "--mock"]
healthcheck:
test: ["CMD", "pgrep", "-f", "meshcore-hub"]
interval: 5s
timeout: 5s
retries: 3
start_period: 10s
# Collector
collector:
build:
context: ../..
dockerfile: docker/Dockerfile
container_name: meshcore-test-collector
depends_on:
mqtt:
condition: service_healthy
volumes:
- test_data:/data
environment:
- LOG_LEVEL=DEBUG
- MQTT_HOST=mqtt
- MQTT_PORT=1883
- MQTT_PREFIX=test
- DATABASE_URL=sqlite:////data/test.db
command: ["collector"]
healthcheck:
test: ["CMD", "pgrep", "-f", "meshcore-hub"]
interval: 5s
timeout: 5s
retries: 3
start_period: 10s
# API Server
api:
build:
context: ../..
dockerfile: docker/Dockerfile
container_name: meshcore-test-api
depends_on:
mqtt:
condition: service_healthy
collector:
condition: service_started
ports:
- "18000:8000"
volumes:
- test_data:/data
environment:
- LOG_LEVEL=DEBUG
- MQTT_HOST=mqtt
- MQTT_PORT=1883
- MQTT_PREFIX=test
- DATABASE_URL=sqlite:////data/test.db
- API_HOST=0.0.0.0
- API_PORT=8000
- API_READ_KEY=test-read-key
- API_ADMIN_KEY=test-admin-key
command: ["api"]
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
interval: 5s
timeout: 5s
retries: 3
start_period: 10s
# Web Dashboard
web:
build:
context: ../..
dockerfile: docker/Dockerfile
container_name: meshcore-test-web
depends_on:
api:
condition: service_healthy
ports:
- "18080:8080"
environment:
- LOG_LEVEL=DEBUG
- API_BASE_URL=http://api:8000
- API_KEY=test-read-key
- WEB_HOST=0.0.0.0
- WEB_PORT=8080
- NETWORK_NAME=Test Network
command: ["web"]
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/health')"]
interval: 5s
timeout: 5s
retries: 3
start_period: 10s
volumes:
test_data:
name: meshcore_test_data

198
tests/e2e/test_full_flow.py Normal file
View File

@@ -0,0 +1,198 @@
"""End-to-end tests for the full MeshCore Hub flow.
These tests require Docker Compose services to be running:
docker compose -f tests/e2e/docker-compose.test.yml up -d
pytest tests/e2e/
docker compose -f tests/e2e/docker-compose.test.yml down -v
The tests verify:
1. API health endpoints
2. Web dashboard health endpoints
3. Node listing and retrieval
4. Message listing
5. Statistics endpoint
6. Command sending (admin only)
"""
import httpx
class TestHealthEndpoints:
"""Test health check endpoints."""
def test_api_health(self, api_client: httpx.Client) -> None:
"""Test API basic health endpoint."""
response = api_client.get("/health")
assert response.status_code == 200
data = response.json()
assert data["status"] == "healthy"
assert "version" in data
def test_api_ready(self, api_client: httpx.Client) -> None:
"""Test API readiness endpoint with database check."""
response = api_client.get("/health/ready")
assert response.status_code == 200
data = response.json()
assert data["status"] == "ready"
assert data["database"] == "connected"
def test_web_health(self, web_client: httpx.Client) -> None:
"""Test Web dashboard health endpoint."""
response = web_client.get("/health")
assert response.status_code == 200
data = response.json()
assert data["status"] == "healthy"
def test_web_ready(self, web_client: httpx.Client) -> None:
"""Test Web dashboard readiness with API connectivity."""
response = web_client.get("/health/ready")
assert response.status_code == 200
data = response.json()
assert data["status"] == "ready"
assert data["api"] == "connected"
class TestAPIEndpoints:
"""Test API data endpoints."""
def test_list_nodes(self, api_client: httpx.Client) -> None:
"""Test listing nodes."""
response = api_client.get("/api/v1/nodes")
assert response.status_code == 200
data = response.json()
assert "items" in data
assert "total" in data
assert isinstance(data["items"], list)
def test_list_messages(self, api_client: httpx.Client) -> None:
"""Test listing messages."""
response = api_client.get("/api/v1/messages")
assert response.status_code == 200
data = response.json()
assert "items" in data
assert "total" in data
def test_list_advertisements(self, api_client: httpx.Client) -> None:
"""Test listing advertisements."""
response = api_client.get("/api/v1/advertisements")
assert response.status_code == 200
data = response.json()
assert "items" in data
assert "total" in data
def test_get_stats(self, api_client: httpx.Client) -> None:
"""Test getting network statistics."""
response = api_client.get("/api/v1/stats")
assert response.status_code == 200
data = response.json()
assert "total_nodes" in data
assert "active_nodes" in data
assert "total_messages" in data
def test_list_telemetry(self, api_client: httpx.Client) -> None:
"""Test listing telemetry records."""
response = api_client.get("/api/v1/telemetry")
assert response.status_code == 200
data = response.json()
assert "items" in data
def test_list_trace_paths(self, api_client: httpx.Client) -> None:
"""Test listing trace paths."""
response = api_client.get("/api/v1/trace-paths")
assert response.status_code == 200
data = response.json()
assert "items" in data
class TestWebDashboard:
"""Test web dashboard pages."""
def test_home_page(self, web_client: httpx.Client) -> None:
"""Test home page loads."""
response = web_client.get("/")
assert response.status_code == 200
assert "text/html" in response.headers.get("content-type", "")
def test_network_page(self, web_client: httpx.Client) -> None:
"""Test network overview page loads."""
response = web_client.get("/network")
assert response.status_code == 200
assert "text/html" in response.headers.get("content-type", "")
def test_nodes_page(self, web_client: httpx.Client) -> None:
"""Test nodes listing page loads."""
response = web_client.get("/nodes")
assert response.status_code == 200
assert "text/html" in response.headers.get("content-type", "")
def test_messages_page(self, web_client: httpx.Client) -> None:
"""Test messages page loads."""
response = web_client.get("/messages")
assert response.status_code == 200
assert "text/html" in response.headers.get("content-type", "")
def test_map_page(self, web_client: httpx.Client) -> None:
"""Test map page loads."""
response = web_client.get("/map")
assert response.status_code == 200
assert "text/html" in response.headers.get("content-type", "")
def test_members_page(self, web_client: httpx.Client) -> None:
"""Test members page loads."""
response = web_client.get("/members")
assert response.status_code == 200
assert "text/html" in response.headers.get("content-type", "")
class TestAuthentication:
"""Test API authentication."""
def test_read_access_with_read_key(self, api_client: httpx.Client) -> None:
"""Test read access works with read key."""
response = api_client.get("/api/v1/nodes")
assert response.status_code == 200
def test_admin_access_with_admin_key(self, admin_client: httpx.Client) -> None:
"""Test admin access works with admin key."""
# Admin key should have read access
response = admin_client.get("/api/v1/nodes")
assert response.status_code == 200
class TestCommands:
"""Test command endpoints (requires admin access)."""
def test_send_message_requires_admin(self, api_client: httpx.Client) -> None:
"""Test that send message requires admin key."""
response = api_client.post(
"/api/v1/commands/send-message",
json={
"destination": "0" * 64,
"text": "Test message",
},
)
# Read key should not have admin access
assert response.status_code == 403
def test_send_channel_message_admin(self, admin_client: httpx.Client) -> None:
"""Test sending channel message with admin key."""
response = admin_client.post(
"/api/v1/commands/send-channel-message",
json={
"channel_idx": 0,
"text": "Test channel message",
},
)
# Should succeed with admin key (202 = accepted for processing)
assert response.status_code == 202
def test_send_advertisement_admin(self, admin_client: httpx.Client) -> None:
"""Test sending advertisement with admin key."""
response = admin_client.post(
"/api/v1/commands/send-advertisement",
json={
"flood": False,
},
)
assert response.status_code == 202