Build a core RegionMap from the node's served regions and wire it into
the dispatcher and every companion bridge, so a flood reply is re-scoped
to the region its request arrived under (or left plain for a wildcard /
direct request) -- matching firmware simple_repeater::sendFloodReply.
Previously replies went out plain, so a reply to a request in region B
was dropped by B-only repeaters.
The map is built once from the node-wide transport_keys table (each named
region -> RegionEntry, flags=REGION_DENY_FLOOD for deny-flood regions; the
'*' wildcard is deliberately not an entry so plain floods reply plain). A
single shared instance reaches the dispatcher and all bridges. Public
regions rely on name-hashing for their key; a stored key is carried only
when it is genuinely custom material the name would not reproduce, keeping
reply-matching aligned with the forwarding transport-code check.
Region edits at runtime (CLI, web API, Glass sync) all funnel through the
transport_keys CRUD methods, which now fire a post-commit change callback;
the daemon rebuilds the map and reassigns a fresh instance to the
dispatcher and every live bridge (atomic rebind, safe against an in-flight
find_match on the RX thread).
Requires openhop_core with Dispatcher.region_map / CompanionBridge.region_map.
Added a new method to the AirtimeManager class to refresh modulation parameters without clearing transmission history. Updated ConfigManager to call this method after applying live radio configurations. Enhanced tests to verify that modulation updates occur correctly during live updates.
Keep both migration 12 (companion message signal/channel data) and
migration 13 (packet upstream hash for neighbour links), and retain
ACK/MULTIPART plus TRACE imports in engine tests.
The companion queue's capacity path ran a Python loop of single-row
SELECT+DELETE pairs ordered by created_at. Wall-clock ordering meant a
backwards clock step (NTP correction) could make the just-inserted
channel row sort as oldest and wrongly reject the incoming message while
older channel rows remained evictable.
Replace the loop with one set-based DELETE ordered by id (AUTOINCREMENT,
i.e. insertion order, immune to clock steps), with an evictable-count
pre-check preserving the all-or-nothing rejection rule: never displace a
direct message, never evict the incoming row to make room for itself,
roll back the insert entirely when channel rows cannot make room. The
queue load and pop queries move to id ordering for the same reason.
- Implemented `lbt_diagnostics` API endpoint to return aggregated Listen Before Talk (LBT) diagnostics aligned with RF metrics.
- Introduced methods for calculating Pearson correlation coefficients and auto-bucket sizing for diagnostics.
- Enhanced data aggregation logic in `StorageCollector` for LBT diagnostics.
- Updated OpenAPI specification to include new endpoint and response schemas.
- Added comprehensive unit tests for LBT diagnostics, including validation of correlation calculations and data integrity.
standardize default region on mesh.default_region (remove legacy region_default_scope usage)
add/align CLI support for owner.info, path.hash.mode, and loop.detect with validation
wire UI terminal get/set + autocomplete/help for owner.info, path.hash.mode, loop.detect
extend update_radio_config to persist owner_info for UI set owner.info
add and use shared packet utility for advert creation + default-region transport scoping
refactor repeater and room-server advert paths to reuse shared packet logic
The GROUP BY type and GROUP BY drop_reason sub-queries in
get_packet_stats (and get_packet_type_stats) filter by a time window but
let the planner pick idx_packets_type / idx_packets_transmitted to get
grouping for free. It then heap-checks the timestamp filter across the
entire table, turning a bounded window into a full scan: on a 1.5M-row
packets table the 24h stats load spent ~4.8s (type) + ~2.3s
(drop_reason) instead of ~0.1s each.
Pin these to idx_packets_timestamp with INDEXED BY so they range-scan the
window (~50k rows) and group via a small temp b-tree. Verified on the
live DB: 4.80s -> 0.10s and 2.32s -> 0.10s. Unlike a covering index this
adds no write-path cost on the packet-insert hot path.
The airtime/utilization chart queries (get_airtime_data and
get_airtime_buckets) range-scan and order packets by timestamp,
selecting only timestamp/length/payload_length/transmitted. On a large
packets table this forced a full scan of the row heap, saturating slow
storage (e.g. a Pi SD card): each dashboard poll took longer than the
client timeout, aborted polls stacked, and sustained I/O starved the
transmit queue.
Add a covering index on packets(timestamp, length, payload_length,
transmitted) so these queries run index-only, dropping the read from the
full row heap to just the index range. Verified via EXPLAIN QUERY PLAN
(COVERING INDEX idx_packets_airtime). Additive and idempotent.
The PacketWebSocket.opened() method only accepts JWT tokens for
authentication. API keys (used via X-API-Key header for REST endpoints)
are silently rejected, forcing WS clients to obtain a JWT through a
separate login flow.
Add fallback to token_manager.verify_token() when JWT verification
fails or when an X-API-Key header is present. This lets API token
holders connect directly to the WebSocket without needing a JWT.
Auth priority:
1. JWT in ?token= query parameter (existing behavior)
2. API key in X-API-Key header
3. API key in ?token= query parameter (falls through from JWT reject)
Merge the maintainer's fix-general-tidy branch (neighbor discovery,
keygen, API endpoints, web-asset rebuild, HTTP server config/control
commands, and an independent #286 room-server push/ACK/guest fix) into
the companion cleanup branch.
Both branches fixed#286 in parallel with byte-identical push-ACK CRC
logic. In the six overlapping files the maintainer's implementation is
kept (ACL replay-detection/session helpers, encoded path-len with legacy
fallback, expected_crc/ack_timeout_s injector API, dispatcher ACK
helpers); this branch's unique companion work is preserved on top
(sender_prefix persistence + migration, boot-state hardening /
CompanionStateLoadError, MessageQueue.max_size, older-core fallbacks).
Conflict resolution took the maintainer's side across the overlap, then
fixed two integration seams the merge introduced and updated this
branch's tests to the maintainer's API:
- room_server: timeout used undefined `hops`; aligned to `path_len`.
- packet_router: PATH helper was invoked twice (maintainer's
unconditional call plus this branch's conditional local-identity
call); dropped the now-redundant conditional block.
Pin openhop_core to @dev (was @feature/publish-workflow-message-handling)
so this can merge to the repeater's dev; core dev carries the required
sender_prefix and PathUtils.is_valid_path_len APIs.
Full suite green (1040 passed) against openhop_core dev and
refactor/companion-housekeeping; ruff clean.
A transient SQLite error during boot made companion_load_channels/
contacts/messages swallow the exception and return [], which is
indistinguishable from "no data". The Public-channel backfill then ran
over the empty store, so clients saw their channels wiped and later
saves could overwrite the persisted state.
- companion_load_{contacts,channels,messages} now return None on error
vs [] for genuinely empty, and log the companion hash
- add companion_count_{channels,messages} helpers
- extract shared _restore_companion_state used by both the boot and
hot-reload companion paths; each load is cross-checked against the
table's row count for the hash, retried once after a short delay,
and raises CompanionStateLoadError if it still cannot load, aborting
companion init (and the Public backfill) instead of starting empty
- log restored row counts per companion; log when the channel store
rejects a persisted channel (unchecked channels.set return)
- trim_companion_contacts_to_fit refuses to trim on a failed load
TXT_TYPE_SIGNED_PLAIN room posts carry a 4-byte author pubkey prefix
(QueuedMessage.sender_prefix, added in openhop_core). The SQLite
persistence path dropped it, so posts replayed from persistence showed
a zero-padded author prefix in the app frame while posts synced from
the live in-memory queue were correct.
- add sender_prefix column (hex text, default '') to companion_messages
with an ALTER TABLE migration for existing databases
- store/return the prefix in companion_push/pop/load_messages
- pass sender_prefix when rebuilding QueuedMessage from persistence in
the frame server and the startup preload paths
- add round-trip, default, migration, and rebuild tests
- Introduced a dedicated thread for broadcasting aggregate statistics to WebSocket clients, improving performance by offloading heavy queries from the packet write path.
- Updated the _publish_packet_sync method to ensure only fast, per-packet operations are executed, while the stats are handled separately.
- Added methods for broadcasting stats at intervals and managing the lifecycle of the stats thread.
- Enhanced unit tests to validate the new stats broadcasting behavior and ensure correct functionality under various conditions.
- Added caching for packet type stats and cumulative counts in SQLiteHandler to reduce database load.
- Implemented a dedicated writer thread in StorageCollector to handle blocking storage operations, preventing asyncio event loop stalls.
- Updated record_packet method to utilize the new writer thread for efficient packet processing.
- Introduced `enforce_companion_contact_capacity` to manage contact limits during companion loading, with an option to trim non-favourite contacts when exceeding capacity.
- Updated `SQLiteHandler` to support message retention limits, allowing for automatic trimming of older messages based on `offline_queue_size`.
- Enhanced API endpoints to handle contact trimming on overflow, providing feedback on trimmed contacts during updates.
- Added utility functions for selecting and trimming contacts while preserving favourites.
- Improved logging for contact management actions and errors related to capacity.
- Updated LogBuffer to support log entry IDs, enhanced log entry structure with additional metadata, and implemented subscriber management for real-time log streaming.
- Added OpenAPI specifications for new endpoints related to policy management, including retrieval, updating, validation, and group management for network policies.
- Implemented comprehensive tests for new policy endpoints, ensuring correct behavior for creating, updating, validating, and deleting policy groups and entries.
- Introduced policy evaluation tests to validate the functionality of the PolicyEngine, including various scenarios for action decisions based on defined rules.
- Enhanced packet routing tests to ensure proper handling of policy decisions in packet processing.
Integrate latest dev while preserving per-companion bridge settings
and contact capacity validation. Resolve import conflicts in main.py
and api_endpoints.py.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Introduced `CompanionContactCapacityError` to handle cases where persisted contacts exceed configured limits.
- Added utility functions for parsing companion bridge settings and validating contact capacity.
- Updated `RepeaterDaemon` to check contact capacity during companion loading and initialization.
- Enhanced API endpoints to validate companion settings and manage contact limits effectively.
- Implemented logging for bridge limits and errors related to contact capacity.
- Updated byte representations in tests to use lowercase hex format for consistency.
- Reformatted code for better readability, including line breaks and indentation adjustments.
- Consolidated multiple lines into single lines where appropriate to enhance clarity.
- Ensured that all test cases maintain consistent formatting and style across the test suite.
Replace the try/except UTC shim in mqtt_handler.py with timezone.utc
directly, consistent with the api_endpoints.py fix. Add a static AST
scan that fails if datetime.UTC is reintroduced anywhere in the codebase,
guarding against future regressions on Python 3.10 (LuckFox Pico Ultra).
Co-Authored-By: Zindello <josh@zindello.com.au>
- Introduced tests for TraceHelper and DiscoveryHelper to validate packet forwarding and discovery request handling.
- Implemented tests for LoginHelper to ensure identity registration and login packet processing.
- Added tests for IdentityManager to cover identity registration, lookup, and filtering.
- Created tests for MeshCLI to verify command handling, configuration setting, and error paths.
agessaman