diff --git a/CLAUDE.md b/CLAUDE.md index 0b2824e..e14342b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -16,10 +16,12 @@ This project is a **MeshCore MQTT Bridge** - a robust Python application that br - **Flexible configuration**: JSON/YAML files, environment variables, CLI arguments - **Configurable event system**: Subscribe to specific MeshCore event types - **Robust MQTT integration**: Authentication, QoS, retention, auto-reconnection +- **Message rate limiting**: Configurable rate limiting to prevent network flooding and ensure reliable message delivery - **Health monitoring**: Built-in health checks and automatic recovery for both workers - **Async architecture**: Built with Python asyncio for high performance - **Type safety**: Full type annotations with mypy support -- **Comprehensive testing**: 59+ tests with pytest and pytest-asyncio +- **Comprehensive testing**: 70+ tests with pytest and pytest-asyncio including rate limiting tests +- **Streamlined CI/CD**: Single pre-commit based CI workflow with automated quality checks ## Architecture @@ -41,10 +43,11 @@ The bridge uses an **Inbox/Outbox Architecture** with independent workers coordi 3. **MeshCore Worker** (`meshcore_mqtt/meshcore_worker.py`) - Independent worker managing MeshCore device connection - - Handles device commands forwarded from MQTT worker + - Handles device commands forwarded from MQTT worker with rate limiting - Auto-reconnection with exponential backoff and health monitoring - Forwards MeshCore events to MQTT worker via message bus - Manages auto-fetch restart after NO_MORE_MSGS events + - Configurable message rate limiting to prevent network flooding 4. **MQTT Worker** (`meshcore_mqtt/mqtt_worker.py`) - Independent worker managing MQTT broker connection @@ -90,6 +93,21 @@ The bridge automatically handles `NO_MORE_MSGS` events from MeshCore by restarti - **Environment Variable**: `MESHCORE_AUTO_FETCH_RESTART_DELAY=10` - **CLI Argument**: `--meshcore-auto-fetch-restart-delay 10` +### Message Rate Limiting Feature + +The bridge includes configurable message rate limiting to prevent network flooding and ensure reliable message delivery: + +- **Purpose**: Prevents overwhelming the MeshCore device with rapid message sending commands +- **Initial Delay**: `message_initial_delay` (0.0-60.0 seconds, default: 5.0) - delay before sending the first message +- **Send Delay**: `message_send_delay` (0.0-60.0 seconds, default: 10.0) - delay between consecutive message sends +- **Behavior**: Messages are queued and sent with appropriate delays using an async rate-limiting system +- **Environment Variables**: + - `MESHCORE_MESSAGE_INITIAL_DELAY=5.0` + - `MESHCORE_MESSAGE_SEND_DELAY=10.0` +- **CLI Arguments**: + - `--meshcore-message-initial-delay 5.0` + - `--meshcore-message-send-delay 10.0` + ### MQTT Topics The bridge publishes to structured MQTT topics: @@ -203,12 +221,14 @@ mosquitto_pub -h localhost -t "meshcore/command/send_telemetry_req" \ ### Code Quality Tools -The project uses these tools (configured in `pyproject.toml`): +The project uses these tools (configured in `pyproject.toml` and `.pre-commit-config.yaml`): - **Black**: Code formatting (line length: 88) - **Flake8**: Linting with custom rules - **MyPy**: Type checking with strict settings +- **isort**: Import sorting and organization - **Pytest**: Testing with asyncio support -- **Pre-commit**: Automated code quality checks +- **Pre-commit**: Automated code quality checks and hooks +- **Streamlined CI**: Single pre-commit based CI workflow replacing multiple separate workflows ### Testing Strategy @@ -218,6 +238,7 @@ The project uses these tools (configured in `pyproject.toml`): - `tests/test_configurable_events.py` - Event configuration (13 tests) - `tests/test_json_serialization.py` - JSON handling (10 tests) - `tests/test_logging.py` - Logging configuration (6 tests) +- `tests/test_rate_limiting.py` - Message rate limiting functionality (9 tests) **Key Test Areas**: - Configuration validation and loading @@ -225,6 +246,7 @@ The project uses these tools (configured in `pyproject.toml`): - Event handler mapping and subscription - JSON serialization edge cases - MQTT topic generation and command handling +- Message rate limiting and queue management - Health monitoring and recovery mechanisms - Logging setup and third-party library control @@ -289,28 +311,35 @@ python -m meshcore_mqtt.main --env ### Major Features Implemented -1. **Inbox/Outbox Architecture (Latest)** +1. **Message Rate Limiting (Latest)** + - Configurable rate limiting for message sending commands + - Initial delay and send delay configuration options + - Async message queue with proper timing controls + - Comprehensive test coverage for rate limiting functionality + +2. **Streamlined CI/CD Pipeline** + - Consolidated multiple CI workflows into single pre-commit based pipeline + - Reduced from 3 separate workflows (ci.yml, test.yml, code-quality.yml) to 1 + - Improved maintainability and faster CI execution + - Full integration with existing pre-commit configuration + +3. **Inbox/Outbox Architecture** - Complete restructure to independent worker pattern - Message bus system for inter-worker communication - Enhanced health monitoring and recovery - Improved resilience and testability -2. **TLS/SSL Support** +4. **TLS/SSL Support** - Secure MQTT connections with configurable certificates - Support for custom CA, client certificates, and private keys - Optional certificate verification bypass for testing -3. **Enhanced Command System** - - Full bidirectional MQTT ↔ MeshCore command support - - Structured command handling with proper error reporting - - Support for all major MeshCore operations - -4. **JSON Serialization (6b492db)** +6. **JSON Serialization** - Robust JSON serialization handling all Python data types - Fallback mechanisms for complex objects - Comprehensive error handling and validation -5. **Configurable Events (f7c10cc)** +7. **Configurable Events** - Made MeshCore event types configurable - Support for config files, env vars, and CLI args - Case-insensitive event parsing with validation @@ -341,6 +370,22 @@ def _on_meshcore_event(self, event_data: Any) -> None: asyncio.create_task(self.message_bus.send_message(message)) ``` +**Rate Limited Command Execution**: +```python +async def _queue_rate_limited_command( + self, command_type: str, command_data: dict +) -> Any: + """Queue a command for rate-limited execution.""" + future: asyncio.Future[Any] = asyncio.Future() + message_data = { + "command_type": command_type, + "future": future, + **command_data, + } + await self._message_queue.put(message_data) + return await future +``` + **Configuration Validation**: ```python @field_validator("field_name") diff --git a/README.md b/README.md index c137990..6538956 100644 --- a/README.md +++ b/README.md @@ -3,8 +3,6 @@ [![License](https://img.shields.io/badge/License-GPL_v3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0) [![Python](https://img.shields.io/badge/Python-3.11+-blue.svg)](https://www.python.org/) [![CI](https://github.com/ipnet-mesh/meshcore-mqtt/actions/workflows/ci.yml/badge.svg)](https://github.com/ipnet-mesh/meshcore-mqtt/actions/workflows/ci.yml) -[![Code Quality](https://github.com/ipnet-mesh/meshcore-mqtt/actions/workflows/code-quality.yml/badge.svg)](https://github.com/ipnet-mesh/meshcore-mqtt/actions/workflows/code-quality.yml) -[![Tests](https://github.com/ipnet-mesh/meshcore-mqtt/actions/workflows/test.yml/badge.svg)](https://github.com/ipnet-mesh/meshcore-mqtt/actions/workflows/test.yml) [![Docker Build and Push](https://github.com/ipnet-mesh/meshcore-mqtt/actions/workflows/docker-build.yml/badge.svg)](https://github.com/ipnet-mesh/meshcore-mqtt/actions/workflows/docker-build.yml) [![Code style: black](https://img.shields.io/badge/Code%20style-black-000000.svg)](https://github.com/psf/black) [![Typing: mypy](https://img.shields.io/badge/Typing-mypy-blue.svg)](https://mypy.readthedocs.io/) @@ -20,11 +18,12 @@ A robust bridge service that connects MeshCore devices to MQTT brokers, enabling - **Flexible Configuration**: JSON, YAML, environment variables, and command-line configuration options - **MQTT Integration**: Full MQTT client with authentication, QoS, retention, and auto-reconnection - **Configurable Event Monitoring**: Subscribe to specific MeshCore event types for optimal performance +- **Message Rate Limiting**: Configurable rate limiting to prevent network flooding and ensure reliable message delivery - **Health Monitoring**: Built-in health checks and automatic recovery for both workers - **Async Architecture**: Built with Python asyncio for high performance and concurrent operations - **Type Safety**: Full type annotations with mypy support -- **Comprehensive Testing**: 59+ unit tests with pytest and pytest-asyncio -- **Code Quality**: Pre-commit hooks with black formatting, flake8 linting, and automated testing +- **Comprehensive Testing**: 70+ unit tests with pytest and pytest-asyncio including rate limiting tests +- **Code Quality**: Streamlined CI/CD with pre-commit hooks for black formatting, flake8 linting, mypy type checking, and automated testing ## Installation @@ -70,6 +69,8 @@ The bridge supports multiple configuration methods with the following precedence - `meshcore_baudrate`: Baudrate for serial connections (default: 115200) - `meshcore_timeout`: Operation timeout in seconds (default: 5) - `meshcore_auto_fetch_restart_delay`: Delay in seconds before restarting auto-fetch after NO_MORE_MSGS (default: 5, range: 1-60) +- `meshcore_message_initial_delay`: Initial delay in seconds before sending the first message (default: 5.0, range: 0.0-60.0) +- `meshcore_message_send_delay`: Delay in seconds between consecutive message sends (default: 10.0, range: 0.0-60.0) - `meshcore_events`: List of MeshCore event types to subscribe to (see [Event Types](#event-types)) #### General Settings @@ -96,6 +97,8 @@ The bridge supports multiple configuration methods with the following precedence "baudrate": 115200, "timeout": 10, "auto_fetch_restart_delay": 10, + "message_initial_delay": 5.0, + "message_send_delay": 10.0, "events": [ "CONTACT_MSG_RECV", "CHANNEL_MSG_RECV", @@ -127,6 +130,8 @@ meshcore: baudrate: 115200 timeout: 10 auto_fetch_restart_delay: 10 + message_initial_delay: 5.0 + message_send_delay: 10.0 events: - CONTACT_MSG_RECV - CHANNEL_MSG_RECV @@ -149,6 +154,8 @@ export MESHCORE_ADDRESS=192.168.1.100 export MESHCORE_PORT=12345 export MESHCORE_BAUDRATE=115200 export MESHCORE_AUTO_FETCH_RESTART_DELAY=10 +export MESHCORE_MESSAGE_INITIAL_DELAY=5.0 +export MESHCORE_MESSAGE_SEND_DELAY=10.0 export MESHCORE_EVENTS="CONNECTED,DISCONNECTED,BATTERY,DEVICE_INFO" export LOG_LEVEL=INFO ``` @@ -548,6 +555,8 @@ MESHCORE_BAUDRATE=115200 # For serial connections MESHCORE_PORT=4403 # Only for TCP connections MESHCORE_TIMEOUT=30 MESHCORE_AUTO_FETCH_RESTART_DELAY=10 # Restart delay after NO_MORE_MSGS (1-60 seconds) +MESHCORE_MESSAGE_INITIAL_DELAY=5.0 # Initial delay before first message (0.0-60.0 seconds) +MESHCORE_MESSAGE_SEND_DELAY=10.0 # Delay between consecutive messages (0.0-60.0 seconds) # Event Configuration (comma-separated) MESHCORE_EVENTS=CONNECTED,DISCONNECTED,BATTERY,DEVICE_INFO