Added a new method to the AirtimeManager class to refresh modulation parameters without clearing transmission history. Updated ConfigManager to call this method after applying live radio configurations. Enhanced tests to verify that modulation updates occur correctly during live updates.
Keep both migration 12 (companion message signal/channel data) and
migration 13 (packet upstream hash for neighbour links), and retain
ACK/MULTIPART plus TRACE imports in engine tests.
The companion queue's capacity path ran a Python loop of single-row
SELECT+DELETE pairs ordered by created_at. Wall-clock ordering meant a
backwards clock step (NTP correction) could make the just-inserted
channel row sort as oldest and wrongly reject the incoming message while
older channel rows remained evictable.
Replace the loop with one set-based DELETE ordered by id (AUTOINCREMENT,
i.e. insertion order, immune to clock steps), with an evictable-count
pre-check preserving the all-or-nothing rejection rule: never displace a
direct message, never evict the incoming row to make room for itself,
roll back the insert entirely when channel rows cannot make room. The
queue load and pop queries move to id ordering for the same reason.
- Implemented `lbt_diagnostics` API endpoint to return aggregated Listen Before Talk (LBT) diagnostics aligned with RF metrics.
- Introduced methods for calculating Pearson correlation coefficients and auto-bucket sizing for diagnostics.
- Enhanced data aggregation logic in `StorageCollector` for LBT diagnostics.
- Updated OpenAPI specification to include new endpoint and response schemas.
- Added comprehensive unit tests for LBT diagnostics, including validation of correlation calculations and data integrity.
standardize default region on mesh.default_region (remove legacy region_default_scope usage)
add/align CLI support for owner.info, path.hash.mode, and loop.detect with validation
wire UI terminal get/set + autocomplete/help for owner.info, path.hash.mode, loop.detect
extend update_radio_config to persist owner_info for UI set owner.info
add and use shared packet utility for advert creation + default-region transport scoping
refactor repeater and room-server advert paths to reuse shared packet logic
The GROUP BY type and GROUP BY drop_reason sub-queries in
get_packet_stats (and get_packet_type_stats) filter by a time window but
let the planner pick idx_packets_type / idx_packets_transmitted to get
grouping for free. It then heap-checks the timestamp filter across the
entire table, turning a bounded window into a full scan: on a 1.5M-row
packets table the 24h stats load spent ~4.8s (type) + ~2.3s
(drop_reason) instead of ~0.1s each.
Pin these to idx_packets_timestamp with INDEXED BY so they range-scan the
window (~50k rows) and group via a small temp b-tree. Verified on the
live DB: 4.80s -> 0.10s and 2.32s -> 0.10s. Unlike a covering index this
adds no write-path cost on the packet-insert hot path.
The airtime/utilization chart queries (get_airtime_data and
get_airtime_buckets) range-scan and order packets by timestamp,
selecting only timestamp/length/payload_length/transmitted. On a large
packets table this forced a full scan of the row heap, saturating slow
storage (e.g. a Pi SD card): each dashboard poll took longer than the
client timeout, aborted polls stacked, and sustained I/O starved the
transmit queue.
Add a covering index on packets(timestamp, length, payload_length,
transmitted) so these queries run index-only, dropping the read from the
full row heap to just the index range. Verified via EXPLAIN QUERY PLAN
(COVERING INDEX idx_packets_airtime). Additive and idempotent.
Merge the maintainer's fix-general-tidy branch (neighbor discovery,
keygen, API endpoints, web-asset rebuild, HTTP server config/control
commands, and an independent #286 room-server push/ACK/guest fix) into
the companion cleanup branch.
Both branches fixed#286 in parallel with byte-identical push-ACK CRC
logic. In the six overlapping files the maintainer's implementation is
kept (ACL replay-detection/session helpers, encoded path-len with legacy
fallback, expected_crc/ack_timeout_s injector API, dispatcher ACK
helpers); this branch's unique companion work is preserved on top
(sender_prefix persistence + migration, boot-state hardening /
CompanionStateLoadError, MessageQueue.max_size, older-core fallbacks).
Conflict resolution took the maintainer's side across the overlap, then
fixed two integration seams the merge introduced and updated this
branch's tests to the maintainer's API:
- room_server: timeout used undefined `hops`; aligned to `path_len`.
- packet_router: PATH helper was invoked twice (maintainer's
unconditional call plus this branch's conditional local-identity
call); dropped the now-redundant conditional block.
Pin openhop_core to @dev (was @feature/publish-workflow-message-handling)
so this can merge to the repeater's dev; core dev carries the required
sender_prefix and PathUtils.is_valid_path_len APIs.
Full suite green (1040 passed) against openhop_core dev and
refactor/companion-housekeeping; ruff clean.
A transient SQLite error during boot made companion_load_channels/
contacts/messages swallow the exception and return [], which is
indistinguishable from "no data". The Public-channel backfill then ran
over the empty store, so clients saw their channels wiped and later
saves could overwrite the persisted state.
- companion_load_{contacts,channels,messages} now return None on error
vs [] for genuinely empty, and log the companion hash
- add companion_count_{channels,messages} helpers
- extract shared _restore_companion_state used by both the boot and
hot-reload companion paths; each load is cross-checked against the
table's row count for the hash, retried once after a short delay,
and raises CompanionStateLoadError if it still cannot load, aborting
companion init (and the Public backfill) instead of starting empty
- log restored row counts per companion; log when the channel store
rejects a persisted channel (unchecked channels.set return)
- trim_companion_contacts_to_fit refuses to trim on a failed load
TXT_TYPE_SIGNED_PLAIN room posts carry a 4-byte author pubkey prefix
(QueuedMessage.sender_prefix, added in openhop_core). The SQLite
persistence path dropped it, so posts replayed from persistence showed
a zero-padded author prefix in the app frame while posts synced from
the live in-memory queue were correct.
- add sender_prefix column (hex text, default '') to companion_messages
with an ALTER TABLE migration for existing databases
- store/return the prefix in companion_push/pop/load_messages
- pass sender_prefix when rebuilding QueuedMessage from persistence in
the frame server and the startup preload paths
- add round-trip, default, migration, and rebuild tests
- Added caching for packet type stats and cumulative counts in SQLiteHandler to reduce database load.
- Implemented a dedicated writer thread in StorageCollector to handle blocking storage operations, preventing asyncio event loop stalls.
- Updated record_packet method to utilize the new writer thread for efficient packet processing.
- Introduced `enforce_companion_contact_capacity` to manage contact limits during companion loading, with an option to trim non-favourite contacts when exceeding capacity.
- Updated `SQLiteHandler` to support message retention limits, allowing for automatic trimming of older messages based on `offline_queue_size`.
- Enhanced API endpoints to handle contact trimming on overflow, providing feedback on trimmed contacts during updates.
- Added utility functions for selecting and trimming contacts while preserving favourites.
- Improved logging for contact management actions and errors related to capacity.
- Updated LogBuffer to support log entry IDs, enhanced log entry structure with additional metadata, and implemented subscriber management for real-time log streaming.
- Added OpenAPI specifications for new endpoints related to policy management, including retrieval, updating, validation, and group management for network policies.
- Implemented comprehensive tests for new policy endpoints, ensuring correct behavior for creating, updating, validating, and deleting policy groups and entries.
- Introduced policy evaluation tests to validate the functionality of the PolicyEngine, including various scenarios for action decisions based on defined rules.
- Enhanced packet routing tests to ensure proper handling of policy decisions in packet processing.
Integrate latest dev while preserving per-companion bridge settings
and contact capacity validation. Resolve import conflicts in main.py
and api_endpoints.py.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Introduced `CompanionContactCapacityError` to handle cases where persisted contacts exceed configured limits.
- Added utility functions for parsing companion bridge settings and validating contact capacity.
- Updated `RepeaterDaemon` to check contact capacity during companion loading and initialization.
- Enhanced API endpoints to validate companion settings and manage contact limits effectively.
- Implemented logging for bridge limits and errors related to contact capacity.
- Updated byte representations in tests to use lowercase hex format for consistency.
- Reformatted code for better readability, including line breaks and indentation adjustments.
- Consolidated multiple lines into single lines where appropriate to enhance clarity.
- Ensured that all test cases maintain consistent formatting and style across the test suite.
Five targeted changes to sqlite_handler.py, all in the same file.
1. Thread-local persistent connections
_connect() previously opened a new sqlite3.connect() on every DB call and
ran journal_mode + busy_timeout PRAGMAs each time. On SD-card storage each
connection open involves file-system operations; each PRAGMA is a round-trip.
threading.local() now caches one connection per thread (write executor thread
+ event-loop/HTTP threads), eliminating per-call setup overhead.
2. PRAGMA synchronous=NORMAL
Default synchronous=FULL flushes WAL frames to disk after every transaction.
NORMAL flushes only at WAL checkpoints — safe for this workload (no data loss
beyond the current transaction on power failure) and significantly faster on
SD cards, which have slow fsync (5-20ms per flush).
3. Migration 8: UNIQUE index on companion_messages(companion_hash, packet_hash)
companion_push_message previously deduped via SELECT + INSERT (two statements,
two SD-card reads per message). The new UNIQUE index enables INSERT OR IGNORE,
replacing the round-trip with a single atomic statement.
4. Migration 9: UNIQUE index on adverts(pubkey)
Without this index store_advert's ON CONFLICT clause cannot fire and each
advert inserts a new row instead of updating the existing one — unbounded
table growth on busy meshes. The migration deduplicates existing rows
(keeping the most-recently-seen per pubkey) before adding the index.
5. Remove duplicate get_unsynced_count definition
The method was defined twice with the same signature. Python silently uses
the last definition; the first was dead code with reversed SQL parameter
binding order. Removed the first; added a note to the surviving definition.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Added `companion_import_repeater_contacts` method in `SQLiteHandler` to import repeater adverts into a companion's contact store, with options for filtering by contact types, last seen hours, and import limits.
- Introduced `_get_sqlite_handler` method in `CompanionAPIEndpoints` to ensure the SQLite handler is available for contact import operations.
- Created `import_repeater_contacts` endpoint to handle POST requests for importing contacts, validating input parameters, and returning the count of successfully imported contacts.
- Updated the frontend to reflect changes in the contact import process, ensuring a seamless user experience.
- Introduced a new utility function, _to_json_safe, to ensure companion preferences are JSON-serializable, handling various data types including enums and dataclasses.
- Updated the RepeaterCompanionBridge to use the new serialization method when saving preferences to SQLite.
- Modified SQLiteHandler to ensure companion_hash is consistently converted to a string before database operations.
- Enhanced error handling for preference persistence in the RepeaterCompanionBridge.
- Added new methods to SQLiteHandler for loading and saving companion preferences as JSON, improving data persistence.
- Introduced a migration to create a companion_prefs table for storing preferences, ensuring compatibility with existing data.
- Refactored main.py to utilize RepeaterCompanionBridge instead of CompanionBridge, aligning with the new architecture.
- Create a new table for storing CRC errors in SQLite.
- Implement methods to store and retrieve CRC error counts and history.
- Update StorageCollector to record CRC errors and expose relevant methods.
- Enhance RepeaterHandler to track and record CRC error deltas from the radio hardware.
- Add API endpoints to fetch CRC error count and history.
- Updated the SQLiteHandler to apply a migration that prefixes companion_hash values with '0x' for consistency with room_hash patterns.
- Adjusted the main.py file to reflect the new formatting for companion_hash when generating its string representation.
- Enhanced readability by adding descriptive suffixes to packet type labels in both rrdtool_handler.py and sqlite_handler.py.
- Aligned packet type definitions with the new naming conventions from pyMC_core, ensuring consistency across the codebase.
- Updated PacketRouter to deliver packets to all companion bridges when the destination is not recognized, ensuring better handling of ephemeral destinations.
- Refactored CompanionFrameServer to separate contact serialization and persistence logic, allowing for non-blocking database operations.
- Introduced a unique index for companion contacts in SQLite to support upsert functionality, enhancing data integrity and performance.
- Improved AdvertHelper to run database operations in a separate thread, preventing event loop blocking and maintaining responsiveness.
- Updated `_persist_companion_message` to clarify deduplication of messages in SQLite.
- Modified `on_message_received` and `on_channel_message_received` to include `packet_hash` for message identification.
- Enhanced `SQLiteHandler` to support deduplication by `packet_hash` when pushing messages, preventing duplicates in the database.
- Added `packet_hash` column to the `companion_messages` table and created an index for efficient lookups.
- Add repeater/companion with frame server and constants
- Extend config, sqlite_handler, mesh_cli, packet_router for companion
- Update api_endpoints and auth_endpoints; adjust main entry
Co-authored-by: Cursor <cursoragent@cursor.com>
Features
Neighbour details modal with full info and map view
WebSocket support with heartbeat and automatic reconnection
Improved signal quality calculations (SNR-based RSSI)
Route-based pagination for faster initial loads
UI
Mobile sidebar tweaks (logout, version info, lazy-loaded charts)
Sorting added to the neighbour table
Packet view now shows multi-hop paths
CAD calibration charts respect light/dark themes
Statistics charts now show the full requested time range
Performance
Reduced polling when WebSocket is active
Lazy loading for heavier components
Noise floor data capped to keep charts responsive
Technical
Improved type safety across API responses
Contrast improvements for accessibility
Cleaner WebSocket and MQTT reconnection handling
Additional metrics added to heartbeat stats
Bug fixes
Corrected noise floor history query
Fixed authentication for CAD calibration streams
Nothing major required from users — just update and carry on.
As always, shout if something looks off.
agessaman