- Introduced a dedicated thread for broadcasting aggregate statistics to WebSocket clients, improving performance by offloading heavy queries from the packet write path.
- Updated the _publish_packet_sync method to ensure only fast, per-packet operations are executed, while the stats are handled separately.
- Added methods for broadcasting stats at intervals and managing the lifecycle of the stats thread.
- Enhanced unit tests to validate the new stats broadcasting behavior and ensure correct functionality under various conditions.
- Added caching for packet type stats and cumulative counts in SQLiteHandler to reduce database load.
- Implemented a dedicated writer thread in StorageCollector to handle blocking storage operations, preventing asyncio event loop stalls.
- Updated record_packet method to utilize the new writer thread for efficient packet processing.
- Introduced `enforce_companion_contact_capacity` to manage contact limits during companion loading, with an option to trim non-favourite contacts when exceeding capacity.
- Updated `SQLiteHandler` to support message retention limits, allowing for automatic trimming of older messages based on `offline_queue_size`.
- Enhanced API endpoints to handle contact trimming on overflow, providing feedback on trimmed contacts during updates.
- Added utility functions for selecting and trimming contacts while preserving favourites.
- Improved logging for contact management actions and errors related to capacity.
- Updated LogBuffer to support log entry IDs, enhanced log entry structure with additional metadata, and implemented subscriber management for real-time log streaming.
- Added OpenAPI specifications for new endpoints related to policy management, including retrieval, updating, validation, and group management for network policies.
- Implemented comprehensive tests for new policy endpoints, ensuring correct behavior for creating, updating, validating, and deleting policy groups and entries.
- Introduced policy evaluation tests to validate the functionality of the PolicyEngine, including various scenarios for action decisions based on defined rules.
- Enhanced packet routing tests to ensure proper handling of policy decisions in packet processing.
Integrate latest dev while preserving per-companion bridge settings
and contact capacity validation. Resolve import conflicts in main.py
and api_endpoints.py.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Introduced `CompanionContactCapacityError` to handle cases where persisted contacts exceed configured limits.
- Added utility functions for parsing companion bridge settings and validating contact capacity.
- Updated `RepeaterDaemon` to check contact capacity during companion loading and initialization.
- Enhanced API endpoints to validate companion settings and manage contact limits effectively.
- Implemented logging for bridge limits and errors related to contact capacity.
- Updated byte representations in tests to use lowercase hex format for consistency.
- Reformatted code for better readability, including line breaks and indentation adjustments.
- Consolidated multiple lines into single lines where appropriate to enhance clarity.
- Ensured that all test cases maintain consistent formatting and style across the test suite.
Replace the try/except UTC shim in mqtt_handler.py with timezone.utc
directly, consistent with the api_endpoints.py fix. Add a static AST
scan that fails if datetime.UTC is reintroduced anywhere in the codebase,
guarding against future regressions on Python 3.10 (LuckFox Pico Ultra).
Co-Authored-By: Zindello <josh@zindello.com.au>
- Introduced tests for TraceHelper and DiscoveryHelper to validate packet forwarding and discovery request handling.
- Implemented tests for LoginHelper to ensure identity registration and login packet processing.
- Added tests for IdentityManager to cover identity registration, lookup, and filtering.
- Created tests for MeshCLI to verify command handling, configuration setting, and error paths.
Every MQTT-published packet has shipped with duration="0" since the
PacketRecord factory was introduced. The repeater already computes LoRa
time-on-air via AirtimeManager.calculate_airtime() (the canonical
Semtech reference formula) for duty-cycle gating and TX delay, but the
result was thrown away after each packet - never stored on the
packet_record dict that flows to MQTT/SQLite/Glass/websocket.
What changes
- engine.py: RepeaterHandler._build_packet_record() now computes
airtime_ms once per packet (Semtech formula via AirtimeManager) and
stores it as packet_record['airtime_ms']. Single source of truth for
every downstream consumer.
- storage_utils.py: PacketRecord.from_packet_record() reads the new
airtime_ms field and serializes it as a rounded integer in the
'duration' field of the published JSON. Falls back to 0 if the field
is missing (backward compatibility for any older code path).
- storage_collector.py: _publish_packet_to_mqtt() simplified - no
recomputation, no helper. The publish path is now a passthrough.
Why
MQTT consumers (firmware-compatible analyzers, dashboards, the upstream
meshcoretomqtt project) expect the same time-on-air value the firmware
emits. Hard-coded "0" makes airtime/utilization charts derived from the
mqtt stream useless and silently diverges from firmware behavior.
Plumbing the value through packet_record (instead of recomputing in the
publish path) means any future consumer - SQLite schema, web UI charts,
Glass telemetry - reads the same number without separate calculations.
Tests
tests/test_packet_duration.py - 5 tests covering:
- backward compat (legacy packet_record without airtime_ms => '0')
- airtime_ms field flows through to duration as rounded integer string
- explicit zero stays '0'
- AirtimeManager output matches an independently-implemented Semtech
reference for typical MeshCore EU settings (SF8/62.5kHz/CR4-8)
- low-data-rate optimization branch (SF12/125kHz triggers DE=1)
Co-Authored-By: Oz <oz-agent@warp.dev>
Introduces a 'set format and forget' workflow for MQTT brokers. Users
reference a bundled preset by name inside the existing brokers: list,
and the package supplies the endpoints, audiences, and TLS settings.
Endpoint changes ship via 'pip install -U' instead of manual edits.
What changes
- New repeater/presets/ package with a tiny lazy YAML loader and two
bundled presets: waev (mqtt-{a,b}.waev.app) and letsmesh (EU + US).
- New format-family constant MC2MQTT_FORMATS = ('meshcoretomqtt',
'letsmesh', 'waev') replaces the inline tuple in topic resolution.
The legacy 'mqtt' format keeps its custom-topic semantics unchanged.
- Two-pass broker assembly in mqtt_handler.py: pass 1 expands every
{preset: <name>} entry inline; pass 2 collapses duplicates by name
with later-wins semantics. Place override entries AFTER preset
entries.
- Hard-coded LETSMESH_BROKERS constant deleted; its data now lives in
repeater/presets/letsmesh.yaml.
- convert_letsmesh_to_broker_config() collapsed from ~70 to ~25 lines
by emitting {preset: letsmesh} plus disable overrides for unwanted
brokers. Honors broker_index in (-1, 0, 1), additional_brokers, and
enabled flag exactly as before.
- update_mqtt_config API endpoint accepts {preset: <name>} entries and
passes them through unchanged so the web UI can author them when the
frontend is updated.
- config.yaml.example documents the preset entry shape, the override
rule, and the format family hierarchy.
- pyproject.toml ships presets/*.yaml as package data.
How to use
mqtt_brokers:
iata_code: "LAX"
brokers:
- preset: waev
# Override a single preset broker:
brokers:
- preset: waev
- name: waev-b
enabled: false
Tests
- tests/test_presets.py: 9 tests covering loader, expand/merge,
MC2MQTT topic-family parity, and parametrized legacy migration.
Co-Authored-By: Oz <oz-agent@warp.dev>
- Introduced options for using GPS coordinates for repeater location fields in config.
- Implemented precision control for GPS coordinates.
- Added a new API endpoint for a Server-Sent Events stream of GPS diagnostics.
- Updated GPSService to handle new configuration options and fallback logic.
- Enhanced unit tests for GPS location handling.
Problem
-------
update_packet_metrics() called rrdtool.info() (cached for 5 s) to get the
RRD's last_update timestamp. rrdtool.info() returns a massive Python dict:
17 data sources × 5 RRAs × ~8 fields each = ~700+ dict entries per call.
tracemalloc showed +10696 new allocations / +251 KB at this exact line,
flagged as "Investigate" in the memory diagnostics dashboard.
The rrdtool.info() approach was also unnecessarily complex: it required a
5-second secondary cache, a _pending_rrd_update buffer, and two extra
instance attributes — all to answer one question ("did we already write
this period?") that we can answer ourselves with a single integer.
Fix
---
Replace _last_rrd_info_cache / _last_rrd_info_time / _pending_rrd_update
with a single self._last_rrd_update: int = 0 that stores the timestamp of
the last successful rrdtool.update() call. The throttle check becomes:
if timestamp <= self._last_rrd_update:
return
On success: self._last_rrd_update = timestamp
Zero dict allocations per call. The only downside vs rrdtool.info() is
that _last_rrd_update resets to 0 on process restart, meaning the first
packet after a restart always triggers a write — correct behaviour.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
yellowcooln