- Introduced `enforce_companion_contact_capacity` to manage contact limits during companion loading, with an option to trim non-favourite contacts when exceeding capacity.
- Updated `SQLiteHandler` to support message retention limits, allowing for automatic trimming of older messages based on `offline_queue_size`.
- Enhanced API endpoints to handle contact trimming on overflow, providing feedback on trimmed contacts during updates.
- Added utility functions for selecting and trimming contacts while preserving favourites.
- Improved logging for contact management actions and errors related to capacity.
- Updated LogBuffer to support log entry IDs, enhanced log entry structure with additional metadata, and implemented subscriber management for real-time log streaming.
- Added OpenAPI specifications for new endpoints related to policy management, including retrieval, updating, validation, and group management for network policies.
- Implemented comprehensive tests for new policy endpoints, ensuring correct behavior for creating, updating, validating, and deleting policy groups and entries.
- Introduced policy evaluation tests to validate the functionality of the PolicyEngine, including various scenarios for action decisions based on defined rules.
- Enhanced packet routing tests to ensure proper handling of policy decisions in packet processing.
Integrate latest dev while preserving per-companion bridge settings
and contact capacity validation. Resolve import conflicts in main.py
and api_endpoints.py.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Introduced `CompanionContactCapacityError` to handle cases where persisted contacts exceed configured limits.
- Added utility functions for parsing companion bridge settings and validating contact capacity.
- Updated `RepeaterDaemon` to check contact capacity during companion loading and initialization.
- Enhanced API endpoints to validate companion settings and manage contact limits effectively.
- Implemented logging for bridge limits and errors related to contact capacity.
- Updated byte representations in tests to use lowercase hex format for consistency.
- Reformatted code for better readability, including line breaks and indentation adjustments.
- Consolidated multiple lines into single lines where appropriate to enhance clarity.
- Ensured that all test cases maintain consistent formatting and style across the test suite.
Replace the try/except UTC shim in mqtt_handler.py with timezone.utc
directly, consistent with the api_endpoints.py fix. Add a static AST
scan that fails if datetime.UTC is reintroduced anywhere in the codebase,
guarding against future regressions on Python 3.10 (LuckFox Pico Ultra).
Co-Authored-By: Zindello <josh@zindello.com.au>
- Introduced tests for TraceHelper and DiscoveryHelper to validate packet forwarding and discovery request handling.
- Implemented tests for LoginHelper to ensure identity registration and login packet processing.
- Added tests for IdentityManager to cover identity registration, lookup, and filtering.
- Created tests for MeshCLI to verify command handling, configuration setting, and error paths.
Every MQTT-published packet has shipped with duration="0" since the
PacketRecord factory was introduced. The repeater already computes LoRa
time-on-air via AirtimeManager.calculate_airtime() (the canonical
Semtech reference formula) for duty-cycle gating and TX delay, but the
result was thrown away after each packet - never stored on the
packet_record dict that flows to MQTT/SQLite/Glass/websocket.
What changes
- engine.py: RepeaterHandler._build_packet_record() now computes
airtime_ms once per packet (Semtech formula via AirtimeManager) and
stores it as packet_record['airtime_ms']. Single source of truth for
every downstream consumer.
- storage_utils.py: PacketRecord.from_packet_record() reads the new
airtime_ms field and serializes it as a rounded integer in the
'duration' field of the published JSON. Falls back to 0 if the field
is missing (backward compatibility for any older code path).
- storage_collector.py: _publish_packet_to_mqtt() simplified - no
recomputation, no helper. The publish path is now a passthrough.
Why
MQTT consumers (firmware-compatible analyzers, dashboards, the upstream
meshcoretomqtt project) expect the same time-on-air value the firmware
emits. Hard-coded "0" makes airtime/utilization charts derived from the
mqtt stream useless and silently diverges from firmware behavior.
Plumbing the value through packet_record (instead of recomputing in the
publish path) means any future consumer - SQLite schema, web UI charts,
Glass telemetry - reads the same number without separate calculations.
Tests
tests/test_packet_duration.py - 5 tests covering:
- backward compat (legacy packet_record without airtime_ms => '0')
- airtime_ms field flows through to duration as rounded integer string
- explicit zero stays '0'
- AirtimeManager output matches an independently-implemented Semtech
reference for typical MeshCore EU settings (SF8/62.5kHz/CR4-8)
- low-data-rate optimization branch (SF12/125kHz triggers DE=1)
Co-Authored-By: Oz <oz-agent@warp.dev>
Introduces a 'set format and forget' workflow for MQTT brokers. Users
reference a bundled preset by name inside the existing brokers: list,
and the package supplies the endpoints, audiences, and TLS settings.
Endpoint changes ship via 'pip install -U' instead of manual edits.
What changes
- New repeater/presets/ package with a tiny lazy YAML loader and two
bundled presets: waev (mqtt-{a,b}.waev.app) and letsmesh (EU + US).
- New format-family constant MC2MQTT_FORMATS = ('meshcoretomqtt',
'letsmesh', 'waev') replaces the inline tuple in topic resolution.
The legacy 'mqtt' format keeps its custom-topic semantics unchanged.
- Two-pass broker assembly in mqtt_handler.py: pass 1 expands every
{preset: <name>} entry inline; pass 2 collapses duplicates by name
with later-wins semantics. Place override entries AFTER preset
entries.
- Hard-coded LETSMESH_BROKERS constant deleted; its data now lives in
repeater/presets/letsmesh.yaml.
- convert_letsmesh_to_broker_config() collapsed from ~70 to ~25 lines
by emitting {preset: letsmesh} plus disable overrides for unwanted
brokers. Honors broker_index in (-1, 0, 1), additional_brokers, and
enabled flag exactly as before.
- update_mqtt_config API endpoint accepts {preset: <name>} entries and
passes them through unchanged so the web UI can author them when the
frontend is updated.
- config.yaml.example documents the preset entry shape, the override
rule, and the format family hierarchy.
- pyproject.toml ships presets/*.yaml as package data.
How to use
mqtt_brokers:
iata_code: "LAX"
brokers:
- preset: waev
# Override a single preset broker:
brokers:
- preset: waev
- name: waev-b
enabled: false
Tests
- tests/test_presets.py: 9 tests covering loader, expand/merge,
MC2MQTT topic-family parity, and parametrized legacy migration.
Co-Authored-By: Oz <oz-agent@warp.dev>
- Introduced options for using GPS coordinates for repeater location fields in config.
- Implemented precision control for GPS coordinates.
- Added a new API endpoint for a Server-Sent Events stream of GPS diagnostics.
- Updated GPSService to handle new configuration options and fallback logic.
- Enhanced unit tests for GPS location handling.
Problem
-------
update_packet_metrics() called rrdtool.info() (cached for 5 s) to get the
RRD's last_update timestamp. rrdtool.info() returns a massive Python dict:
17 data sources × 5 RRAs × ~8 fields each = ~700+ dict entries per call.
tracemalloc showed +10696 new allocations / +251 KB at this exact line,
flagged as "Investigate" in the memory diagnostics dashboard.
The rrdtool.info() approach was also unnecessarily complex: it required a
5-second secondary cache, a _pending_rrd_update buffer, and two extra
instance attributes — all to answer one question ("did we already write
this period?") that we can answer ourselves with a single integer.
Fix
---
Replace _last_rrd_info_cache / _last_rrd_info_time / _pending_rrd_update
with a single self._last_rrd_update: int = 0 that stores the timestamp of
the last successful rrdtool.update() call. The throttle check becomes:
if timestamp <= self._last_rrd_update:
return
On success: self._last_rrd_update = timestamp
Zero dict allocations per call. The only downside vs rrdtool.info() is
that _last_rrd_update resets to 0 on process restart, meaning the first
packet after a restart always triggers a write — correct behaviour.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Problem
-------
rrdtool.fetch() is a blocking C library call that reads 24 hours of RRD
data from disk. The dashboard can call get_data() on every page refresh.
On an SD card each fetch can cost several milliseconds of I/O, and because
the RRD step is 60 seconds the data cannot change more often than that —
any fetch within the same 60-second window returns identical data.
The combined-optimizations branch had a 60-second read cache; rightup's
batching refactor inadvertently removed it. This PR restores it.
Solution
--------
* Add self._get_data_cache: tuple = (0.0, None) to __init__
* In get_data(): set use_cache = (start_time is None and end_time is None)
- if use_cache and cache is < 60 s old: return cached result immediately
- after a successful live fetch with use_cache: store (now, result)
* Explicit start_time / end_time callers always bypass the cache so
fine-grained or historical queries are never stale
Why 60 s TTL?
The RRD step is 60 s, so the database cannot hold a newer sample until
the next step boundary. A 60-second cache is tight enough that the
dashboard always shows data ≤ one step stale, and loose enough that
a burst of refreshes costs one disk read instead of N.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Five targeted changes to sqlite_handler.py, all in the same file.
1. Thread-local persistent connections
_connect() previously opened a new sqlite3.connect() on every DB call and
ran journal_mode + busy_timeout PRAGMAs each time. On SD-card storage each
connection open involves file-system operations; each PRAGMA is a round-trip.
threading.local() now caches one connection per thread (write executor thread
+ event-loop/HTTP threads), eliminating per-call setup overhead.
2. PRAGMA synchronous=NORMAL
Default synchronous=FULL flushes WAL frames to disk after every transaction.
NORMAL flushes only at WAL checkpoints — safe for this workload (no data loss
beyond the current transaction on power failure) and significantly faster on
SD cards, which have slow fsync (5-20ms per flush).
3. Migration 8: UNIQUE index on companion_messages(companion_hash, packet_hash)
companion_push_message previously deduped via SELECT + INSERT (two statements,
two SD-card reads per message). The new UNIQUE index enables INSERT OR IGNORE,
replacing the round-trip with a single atomic statement.
4. Migration 9: UNIQUE index on adverts(pubkey)
Without this index store_advert's ON CONFLICT clause cannot fire and each
advert inserts a new row instead of updating the existing one — unbounded
table growth on busy meshes. The migration deduplicates existing rows
(keeping the most-recently-seen per pubkey) before adding the index.
5. Remove duplicate get_unsynced_count definition
The method was defined twice with the same signature. Python silently uses
the last definition; the first was dead code with reversed SQL parameter
binding order. Removed the first; added a note to the surviving definition.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Lloyd