The 5s WebSocket broadcast used to carry uptime_seconds alone — while
the dashboard sidebar starved: its duty-cycle gauge, noise floor and
mode never travel over WS, and the heartbeat marks stats fresh, which
suppresses the UI's HTTP fallback poll. The vitals now ride every
beat, all cheap in-memory scalars (~200 bytes): utilization_percent,
noise_floor_dbm (kept on record, no per-beat SQL), the repeater mode
(so a change made over RF or CLI reaches the web badge), and the
advert tier counters the sidebar's OK/Drop line shows, injected as a
callback from the daemon like the other collector hooks.
In exchange the heavy get_packet_stats(24h) SQL aggregate is decimated
to one beat in six (30s): nobody reads a 24h cumulative at 5s
resolution, and the UI already treats both payload fields as optional,
so older dashboards keep working. Net wire traffic goes down.
Signed-off-by: Thibault VINCENT <root@devcat.org>
Scope answers were built into the MQTT payload and then discarded, so the web UI
had nothing to show between cycles and no way to ask a single repeater. Adds a
store for them, a read endpoint, and a single-target query.
Migration 15 adds neighbor_scopes: one row per queried neighbour holding the last
answer (`scopes`, `responded_at`) alongside the last query's outcome
(`status`, `queried_at`). The two are kept apart deliberately -- a failed query
updates the outcome but leaves the answer in place, because the responder
rate-limits anonymous replies to 4 every 3 minutes and one timeout is weak
evidence that a neighbour's scopes changed. An empty answer is stored as a real
answer: it means the neighbour serves unscoped traffic only.
A row is written for any query the node attempted. `timeout` alone cannot carry
that: the sweep reports it both for a neighbour that was asked and stayed silent
and for one it never reached, which is what ScopeResult.transmitted separates.
`send_failed` is recorded too even though nothing reached the air -- it was
attempted and the duty cycle refused, and skipping it left a repeater that keeps
refusing reading as "never queried" however often it was asked.
Scope rows follow their neighbour out of the database, on all four paths that
delete adverts: the two explicit deletes, the 6-hourly retention cleanup, and a
purge of the adverts table. Without the last two the table grew without bound and
a purge left scope counts on screen for repeaters no longer listed.
GET /api/neighbor_scopes serves the table; it stays separate from the paginated
advert queries, which are read per contact type on every page load.
POST /api/query_neighbor_scopes asks one neighbour now. Unlike publish_neighbors
it holds the request open for the reply, since the response window is normally its
5 s floor; past 45 s it returns and leaves the query running so a late answer
still lands rather than throwing away spent airtime. It returns the stored view,
not the raw result, so a failed query cannot tell the client to forget scopes the
database still holds. Nothing is published -- the periodic cycle owns the topic.
A query and a cycle must not collide over the scope helper. A cycle holds it for
its whole run including the discovery window, so a query refuses while one is
active; a cycle defers while a query is in flight; and if the two still race, the
cycle abandons the pass on the short retry delay instead of publishing a table
with every scope missing or dying and re-spending its discovery broadcast. Queries
are tracked so shutdown cancels them rather than transmitting through teardown.
The endpoint needs json_in: it is not on globally for /api and cherrypy's Request
has no `json` attribute without it, so reading the body would have failed on every
real request. A test asserts the decorator rather than trusting a fabricated
cherrypy.request, which is how it went unnoticed.
Follow-up work on the neighbours feature, plus one crash found while reviewing it.
The crash: `mqtt_brokers.neighbors` is a settings block, but the per-broker key of
the same name is a boolean and config.yaml.example documents both, so
`neighbors: true` under mqtt_brokers is an easy hand-edit to make. Every reader
called .get() on it directly, raising AttributeError inside
NeighborScopeHelper.refresh_config -- which is built during daemon init, so the
daemon would not start at all. A shared neighbors_config_block() accessor now
ignores a non-mapping (with one startup warning instead of per-tick noise) and
the API save rewrites it as a proper block.
Payload counters, mirroring firmware buildNeighborsMessage:
- total_neighbors and queried_neighbors. The latter cannot be derived from
`status`: a neighbour the sweep never reached reports `timeout` exactly like one
that was asked and stayed silent. ScopeResult therefore carries a `transmitted`
flag, set only past the point where the injector confirms the request is on air.
- Firmware's third field, `truncated`, is deliberately not emitted -- it reports a
fixed PSRAM JSON buffer overflowing, which openhop does not have. total_neighbors
here consequently always equals the published row count.
- self.default_scope: the region this node stamps on outgoing floods, or `*` when
unset. Read from live config so `region default <name>` applies without a
restart, and `#`-stripped like the scopes string beside it. Firmware tracks this
internally but does not publish it, so this field is openhop-only.
POST /api/publish_neighbors runs a cycle immediately, the HTTP twin of the
`discover.scopes` mesh CLI command. Authenticated by default along with the rest
of /api, and it schedules the cycle on the event loop rather than holding the
request open for the minutes a cycle takes.
Schedule persistence: _next_publish_at is monotonic and _last_publish_at was
in-memory only, so every restart read as "due" and spent a discovery broadcast
plus one serialized scope query per neighbour. A generic daemon_state table
(migration 14) now records the last publish and start() resumes from it, never
sooner than a five-minute grace window and never later than one interval. Two
details this had to get right:
- The schedule keys off the last *successful* publish. A failed publish also sets
_last_publish_at and reschedules on the short retry delay; persisting that would
silently turn the retry into a full interval.
- _tick's disabled branch no longer clears the schedule before the feature has
been enabled in this process. At boot the MQTT connections may not be up, so
enabled() can briefly be false, and clearing there would discard the restored
schedule and re-run the sweep anyway -- the exact thing this prevents.
Added a new method to the AirtimeManager class to refresh modulation parameters without clearing transmission history. Updated ConfigManager to call this method after applying live radio configurations. Enhanced tests to verify that modulation updates occur correctly during live updates.
- Implemented `lbt_diagnostics` API endpoint to return aggregated Listen Before Talk (LBT) diagnostics aligned with RF metrics.
- Introduced methods for calculating Pearson correlation coefficients and auto-bucket sizing for diagnostics.
- Enhanced data aggregation logic in `StorageCollector` for LBT diagnostics.
- Updated OpenAPI specification to include new endpoint and response schemas.
- Added comprehensive unit tests for LBT diagnostics, including validation of correlation calculations and data integrity.
standardize default region on mesh.default_region (remove legacy region_default_scope usage)
add/align CLI support for owner.info, path.hash.mode, and loop.detect with validation
wire UI terminal get/set + autocomplete/help for owner.info, path.hash.mode, loop.detect
extend update_radio_config to persist owner_info for UI set owner.info
add and use shared packet utility for advert creation + default-region transport scoping
refactor repeater and room-server advert paths to reuse shared packet logic
- Introduced a dedicated thread for broadcasting aggregate statistics to WebSocket clients, improving performance by offloading heavy queries from the packet write path.
- Updated the _publish_packet_sync method to ensure only fast, per-packet operations are executed, while the stats are handled separately.
- Added methods for broadcasting stats at intervals and managing the lifecycle of the stats thread.
- Enhanced unit tests to validate the new stats broadcasting behavior and ensure correct functionality under various conditions.
- Added caching for packet type stats and cumulative counts in SQLiteHandler to reduce database load.
- Implemented a dedicated writer thread in StorageCollector to handle blocking storage operations, preventing asyncio event loop stalls.
- Updated record_packet method to utilize the new writer thread for efficient packet processing.
- Updated LogBuffer to support log entry IDs, enhanced log entry structure with additional metadata, and implemented subscriber management for real-time log streaming.
- Added OpenAPI specifications for new endpoints related to policy management, including retrieval, updating, validation, and group management for network policies.
- Implemented comprehensive tests for new policy endpoints, ensuring correct behavior for creating, updating, validating, and deleting policy groups and entries.
- Introduced policy evaluation tests to validate the functionality of the PolicyEngine, including various scenarios for action decisions based on defined rules.
- Enhanced packet routing tests to ensure proper handling of policy decisions in packet processing.
- Updated byte representations in tests to use lowercase hex format for consistency.
- Reformatted code for better readability, including line breaks and indentation adjustments.
- Consolidated multiple lines into single lines where appropriate to enhance clarity.
- Ensured that all test cases maintain consistent formatting and style across the test suite.
Every MQTT-published packet has shipped with duration="0" since the
PacketRecord factory was introduced. The repeater already computes LoRa
time-on-air via AirtimeManager.calculate_airtime() (the canonical
Semtech reference formula) for duty-cycle gating and TX delay, but the
result was thrown away after each packet - never stored on the
packet_record dict that flows to MQTT/SQLite/Glass/websocket.
What changes
- engine.py: RepeaterHandler._build_packet_record() now computes
airtime_ms once per packet (Semtech formula via AirtimeManager) and
stores it as packet_record['airtime_ms']. Single source of truth for
every downstream consumer.
- storage_utils.py: PacketRecord.from_packet_record() reads the new
airtime_ms field and serializes it as a rounded integer in the
'duration' field of the published JSON. Falls back to 0 if the field
is missing (backward compatibility for any older code path).
- storage_collector.py: _publish_packet_to_mqtt() simplified - no
recomputation, no helper. The publish path is now a passthrough.
Why
MQTT consumers (firmware-compatible analyzers, dashboards, the upstream
meshcoretomqtt project) expect the same time-on-air value the firmware
emits. Hard-coded "0" makes airtime/utilization charts derived from the
mqtt stream useless and silently diverges from firmware behavior.
Plumbing the value through packet_record (instead of recomputing in the
publish path) means any future consumer - SQLite schema, web UI charts,
Glass telemetry - reads the same number without separate calculations.
Tests
tests/test_packet_duration.py - 5 tests covering:
- backward compat (legacy packet_record without airtime_ms => '0')
- airtime_ms field flows through to duration as rounded integer string
- explicit zero stays '0'
- AirtimeManager output matches an independently-implemented Semtech
reference for typical MeshCore EU settings (SF8/62.5kHz/CR4-8)
- low-data-rate optimization branch (SF12/125kHz triggers DE=1)
Co-Authored-By: Oz <oz-agent@warp.dev>
- Create a new table for storing CRC errors in SQLite.
- Implement methods to store and retrieve CRC error counts and history.
- Update StorageCollector to record CRC errors and expose relevant methods.
- Enhance RepeaterHandler to track and record CRC error deltas from the radio hardware.
- Add API endpoints to fetch CRC error count and history.
- Added support for Meshcore KISS modem firmware in configuration, allowing users to set `radio_type: kiss` and configure serial port and baud rate.
- Updated `config.yaml.example` to include KISS modem settings.
- Modified `manage.sh` to install with hardware extras for KISS support.
- Enhanced `setup-radio-config.sh` to prompt for radio type and KISS modem settings.
- Updated API endpoints to handle KISS modem configurations and hardware options.
- Improved error handling for missing configuration sections.
This update improves flexibility for users utilizing KISS modems alongside SX1262 hardware.
Features
Neighbour details modal with full info and map view
WebSocket support with heartbeat and automatic reconnection
Improved signal quality calculations (SNR-based RSSI)
Route-based pagination for faster initial loads
UI
Mobile sidebar tweaks (logout, version info, lazy-loaded charts)
Sorting added to the neighbour table
Packet view now shows multi-hop paths
CAD calibration charts respect light/dark themes
Statistics charts now show the full requested time range
Performance
Reduced polling when WebSocket is active
Lazy loading for heavier components
Noise floor data capped to keep charts responsive
Technical
Improved type safety across API responses
Contrast improvements for accessibility
Cleaner WebSocket and MQTT reconnection handling
Additional metrics added to heartbeat stats
Bug fixes
Corrected noise floor history query
Fixed authentication for CAD calibration streams
Nothing major required from users — just update and carry on.
As always, shout if something looks off.
Thibault VINCENT