55 Commits

Author SHA1 Message Date
agessaman cd763e2cac feat: implement threaded stats broadcasting in StorageCollector
- Introduced a dedicated thread for broadcasting aggregate statistics to WebSocket clients, improving performance by offloading heavy queries from the packet write path.
- Updated the _publish_packet_sync method to ensure only fast, per-packet operations are executed, while the stats are handled separately.
- Added methods for broadcasting stats at intervals and managing the lifecycle of the stats thread.
- Enhanced unit tests to validate the new stats broadcasting behavior and ensure correct functionality under various conditions.
2026-06-18 21:38:20 -07:00
agessaman 2435757197 feat: enhance SQLiteHandler and StorageCollector for improved caching and async performance
- Added caching for packet type stats and cumulative counts in SQLiteHandler to reduce database load.
- Implemented a dedicated writer thread in StorageCollector to handle blocking storage operations, preventing asyncio event loop stalls.
- Updated record_packet method to utilize the new writer thread for efficient packet processing.
2026-06-18 21:25:19 -07:00
Rightup 14b4804c26 feat: Enhance logging system and introduce policy management endpoints
- Updated LogBuffer to support log entry IDs, enhanced log entry structure with additional metadata, and implemented subscriber management for real-time log streaming.
- Added OpenAPI specifications for new endpoints related to policy management, including retrieval, updating, validation, and group management for network policies.
- Implemented comprehensive tests for new policy endpoints, ensuring correct behavior for creating, updating, validating, and deleting policy groups and entries.
- Introduced policy evaluation tests to validate the functionality of the PolicyEngine, including various scenarios for action decisions based on defined rules.
- Enhanced packet routing tests to ensure proper handling of policy decisions in packet processing.
2026-06-04 15:53:17 +01:00
Lloyd 45a44eb47b Refactor test cases and base code for consistency and readability
- Updated byte representations in tests to use lowercase hex format for consistency.
- Reformatted code for better readability, including line breaks and indentation adjustments.
- Consolidated multiple lines into single lines where appropriate to enhance clarity.
- Ensured that all test cases maintain consistent formatting and style across the test suite.
2026-05-27 20:15:10 +01:00
Lloyd e4efc8045d feat: implement resolve_storage_dir for consistent storage paths 2026-05-05 17:17:12 +01:00
Daniel Duran 0251304407 fix(mqtt): publish Semtech-derived packet duration instead of hard-coded "0"
Every MQTT-published packet has shipped with duration="0" since the
PacketRecord factory was introduced. The repeater already computes LoRa
time-on-air via AirtimeManager.calculate_airtime() (the canonical
Semtech reference formula) for duty-cycle gating and TX delay, but the
result was thrown away after each packet - never stored on the
packet_record dict that flows to MQTT/SQLite/Glass/websocket.

What changes
- engine.py: RepeaterHandler._build_packet_record() now computes
  airtime_ms once per packet (Semtech formula via AirtimeManager) and
  stores it as packet_record['airtime_ms']. Single source of truth for
  every downstream consumer.
- storage_utils.py: PacketRecord.from_packet_record() reads the new
  airtime_ms field and serializes it as a rounded integer in the
  'duration' field of the published JSON. Falls back to 0 if the field
  is missing (backward compatibility for any older code path).
- storage_collector.py: _publish_packet_to_mqtt() simplified - no
  recomputation, no helper. The publish path is now a passthrough.

Why
MQTT consumers (firmware-compatible analyzers, dashboards, the upstream
meshcoretomqtt project) expect the same time-on-air value the firmware
emits. Hard-coded "0" makes airtime/utilization charts derived from the
mqtt stream useless and silently diverges from firmware behavior.

Plumbing the value through packet_record (instead of recomputing in the
publish path) means any future consumer - SQLite schema, web UI charts,
Glass telemetry - reads the same number without separate calculations.

Tests
tests/test_packet_duration.py - 5 tests covering:
- backward compat (legacy packet_record without airtime_ms => '0')
- airtime_ms field flows through to duration as rounded integer string
- explicit zero stays '0'
- AirtimeManager output matches an independently-implemented Semtech
  reference for typical MeshCore EU settings (SF8/62.5kHz/CR4-8)
- low-data-rate optimization branch (SF12/125kHz triggers DE=1)

Co-Authored-By: Oz <oz-agent@warp.dev>
2026-05-03 12:48:02 -07:00
Lloyd 5c947e6c2e feat: enhance WebSocket handling and add throttling for stats broadcasting 2026-04-22 09:48:27 +01:00
Rigear c7b2b02316 fix: Fixed extra topic publishing to letsmesh 2026-04-21 21:21:13 -07:00
Rigear d318334288 Merge remote-tracking branch 'origin/fix-perform-speed' into feat/mqtt_merge 2026-04-21 20:59:42 -07:00
Lloyd be56e919fd feat: add server-side airtime bucket aggregation for optimized chart rendering 2026-04-21 14:46:30 +01:00
Lloyd 9797e08421 feat: implement background scheduling for deferred network publishing tasks, tidy shutdown process 2026-04-21 10:07:15 +01:00
Lloyd 3df4b03fd9 feat: implement deferred network publishing for packets, adverts, and noise floor records 2026-04-21 09:49:12 +01:00
Rigear 11f749e0e9 fix: Initialize tls_verified and properly handle when mqtt_broker is None 2026-04-19 18:04:36 -07:00
Rigear 2e1d19ab80 Merge remote-tracking branch 'origin/dev' into feat/mqtt_merge
# Conflicts:
#	config.yaml.example
#	repeater/data_acquisition/__init__.py
#	repeater/data_acquisition/storage_collector.py
#	repeater/web/html/assets/CADCalibration-319vQEzv.js
#	repeater/web/html/assets/CADCalibration-Cwr0Kq49.js
#	repeater/web/html/assets/CADCalibration-DWusgblB.js
#	repeater/web/html/assets/Companions-DU19yZyB.js
#	repeater/web/html/assets/Companions-cufpceKN.js
#	repeater/web/html/assets/Companions-zmTexa6a.js
#	repeater/web/html/assets/Configuration-BmDpq7bV.js
#	repeater/web/html/assets/ConfirmDialog-BafURQpE.js
#	repeater/web/html/assets/ConfirmDialog-C9Yf394V.js
#	repeater/web/html/assets/ConfirmDialog-h2bJ_WKJ.js
#	repeater/web/html/assets/Dashboard-CnQfG826.js
#	repeater/web/html/assets/Login-BDsVY-me.js
#	repeater/web/html/assets/Logs-BpG7T8_d.js
#	repeater/web/html/assets/Logs-CVZ1ZqH8.js
#	repeater/web/html/assets/Logs-sxcWuUjs.js
#	repeater/web/html/assets/MessageDialog-B-qWtO0z.js
#	repeater/web/html/assets/MessageDialog-Cp4W1enq.js
#	repeater/web/html/assets/MessageDialog-D2OlpbZ7.js
#	repeater/web/html/assets/Neighbors-BAwKrJdF.js
#	repeater/web/html/assets/Neighbors-BamkiPcU.js
#	repeater/web/html/assets/Neighbors-WHAK_7hU.js
#	repeater/web/html/assets/RoomServers-DbCgmJ6x.js
#	repeater/web/html/assets/RoomServers-i32N0iwv.js
#	repeater/web/html/assets/RoomServers-o3kDed-S.js
#	repeater/web/html/assets/Sessions-B8ZVRIGt.js
#	repeater/web/html/assets/Sessions-B9uqWGaO.js
#	repeater/web/html/assets/Sessions-O3vBapMM.js
#	repeater/web/html/assets/Setup-DyJMgh0L.js
#	repeater/web/html/assets/Statistics-BbiQtXdu.js
#	repeater/web/html/assets/Statistics-CeTg6NYy.js
#	repeater/web/html/assets/Statistics-QSH8GjMX.js
#	repeater/web/html/assets/SystemStats-B7qxcRYp.js
#	repeater/web/html/assets/SystemStats-BmXJQonl.js
#	repeater/web/html/assets/SystemStats-DVaA1ybj.js
#	repeater/web/html/assets/Terminal-CUqcF84y.js
#	repeater/web/html/assets/Terminal-D1kRkrmc.js
#	repeater/web/html/assets/Terminal-Dq6FyjMj.js
#	repeater/web/html/assets/api-CiSov_eM.js
#	repeater/web/html/assets/api-DegLD39Y.js
#	repeater/web/html/assets/api-DjLVJkR1.js
#	repeater/web/html/assets/index-cutq4vvY.js
#	repeater/web/html/assets/packets-Bg0pkGLO.js
#	repeater/web/html/assets/packets-CPLd89q8.js
#	repeater/web/html/assets/packets-DmoWuBlc.js
#	repeater/web/html/assets/system-Bocs8bSU.js
#	repeater/web/html/assets/system-CsY7_jKa.js
#	repeater/web/html/assets/system-qCwV23PE.js
#	repeater/web/html/assets/useSignalQuality-DQTATYAm.js
#	repeater/web/html/assets/useSignalQuality-DlXA7j0p.js
#	repeater/web/html/assets/useSignalQuality-u0_rDpC6.js
#	repeater/web/html/index.html
2026-04-18 20:25:30 -07:00
Rightup ffaaa76ea0 feat: add glass to repeater. 2026-04-17 23:51:04 +01:00
Rigear 4569ff8653 feat: publish crc_records to mqtt 2026-04-15 22:05:54 -07:00
Rigear 27fa2381ea feat:
* Added retain status message bool
* Added back old templates
* Added migration path from old mqtt and letsmesh configs to new mqtt_broker config
2026-04-15 21:20:11 -07:00
Rigear f18e5909fb refactor: Clear out dead code 2026-04-11 22:27:01 -07:00
Rigear 64530a623e refactor: Updated letsmesh references 2026-04-11 21:44:26 -07:00
Rigear 7256807fdd feat: Bring back disallowed types 2026-04-11 20:46:42 -07:00
Rigear ba942ca1b7 Merge remote-tracking branch 'origin/dev' into feat/mqtt_merge 2026-04-11 16:09:20 -07:00
Rigear acf8079761 feat: Merge mqtt handler and letsmesh handlers 2026-04-11 16:09:14 -07:00
Lloyd 110d7c2aec feat: add airtime data retrieval functionality with API endpoint 2026-04-11 20:42:04 +01:00
agessaman c150b9a9bf Merge upstream/feat/newRadios into dev-companion-v2-cleanup
- Keep our Vite-built assets and index.html script (index-DyUIpN7m.js)
- Remove upstream-only asset chunks and RoomServers-BxQ-0q-x.js
- README: keep two-backend intro, add upstream CAUTION/compatibility table
- manage.sh: keep dialog/gauge UX and .[hardware]; add CH341 udev, sudoers, libusb, polkit, silent upgrade
- sqlite_handler: add crc_errors table, index, and cleanup from upstream
- engine: add validate_packet and mark_seen in direct_forward; keep our path hash_size/hop_count logic
- advert: keep comment, use current_time = now
- api_endpoints: use restart_service() from service_utils
- config merge: strip user config comments before yq merge (upstream)

Made-with: Cursor
2026-03-05 16:43:14 -08:00
Lloyd 4a05e20172 Add CRC error tracking and API endpoints for error count and history
- Create a new table for storing CRC errors in SQLite.
- Implement methods to store and retrieve CRC error counts and history.
- Update StorageCollector to record CRC errors and expose relevant methods.
- Enhance RepeaterHandler to track and record CRC error deltas from the radio hardware.
- Add API endpoints to fetch CRC error count and history.
2026-03-02 12:36:08 +00:00
agessaman e9841adeff Merge upstream feat/newRadios (radios, MQTT, SPI)
- Take upstream radio-presets.json and radio-settings.json (Femtofox, AIO v2, CH341).
- config.py: upstream get_radio_for_board (_parse_int, sx1262_ch341, GPIO) + re-add KISS branch.
- config.yaml.example: upstream radio/ch341 + our identities.companions.
- manage.sh: upstream SPI warning prompt.
- main.py: single try/finally with companion shutdown and _shutdown().
- letsmesh_handler: upstream UTC fallback and MQTT v5 reason code handling.
- storage_collector, api_endpoints: upstream storage_dir_cfg; apply_setup_wizard supports KISS and sx1262_ch341/ch341.
- airtime.py: upstream bw_hz fix for symbol time.
- pyproject.toml: keep pymc_core dependency (no git pin).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-21 15:59:01 -08:00
agessaman c2f8a2e3cd refactor: companion FrameServer and related (substantive only, no Black)
Reapply refactor from ce8381a (replace monolithic FrameServer with thin
pymc_core subclass, re-export constants, SQLite persistence hooks) while
preserving pre-refactor whitespace where patch applied cleanly. Remaining
files match refactor commit exactly. Diff vs ce8381a is whitespace-only.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-21 15:35:47 -08:00
agessaman 878ff8dc51 Enhance configuration and setup for KISS modem support
- Added support for Meshcore KISS modem firmware in configuration, allowing users to set `radio_type: kiss` and configure serial port and baud rate.
- Updated `config.yaml.example` to include KISS modem settings.
- Modified `manage.sh` to install with hardware extras for KISS support.
- Enhanced `setup-radio-config.sh` to prompt for radio type and KISS modem settings.
- Updated API endpoints to handle KISS modem configurations and hardware options.
- Improved error handling for missing configuration sections.

This update improves flexibility for users utilizing KISS modems alongside SX1262 hardware.
2026-02-03 16:20:51 -08:00
Lloyd f7c4e2b4a8 Add support for new radio hardware configurations and CH341 USB adapter 2026-02-03 22:28:53 +00:00
Lloyd b0e19b13af Add bulk packet retrieval API with gzip compression and pagination support. 2026-02-03 10:21:59 +00:00
Lloyd 8b95bfb8ba feat: enhance MQTT error handling and add packet stats broadcasting 2026-01-20 20:31:45 +00:00
Lloyd 599e4628d9 Changes include:
Features

Neighbour details modal with full info and map view

WebSocket support with heartbeat and automatic reconnection

Improved signal quality calculations (SNR-based RSSI)

Route-based pagination for faster initial loads

UI

Mobile sidebar tweaks (logout, version info, lazy-loaded charts)

Sorting added to the neighbour table

Packet view now shows multi-hop paths

CAD calibration charts respect light/dark themes

Statistics charts now show the full requested time range

Performance

Reduced polling when WebSocket is active

Lazy loading for heavier components

Noise floor data capped to keep charts responsive

Technical

Improved type safety across API responses

Contrast improvements for accessibility

Cleaner WebSocket and MQTT reconnection handling

Additional metrics added to heartbeat stats

Bug fixes

Corrected noise floor history query

Fixed authentication for CAD calibration streams

Nothing major required from users — just update and carry on.
As always, shout if something looks off.
2026-01-18 20:15:50 +00:00
Lloyd 1a527c761f feat: Refactor LetsMesh handler to use LocalIdentity for key management 2026-01-08 09:11:45 +00:00
Lloyd 1b45060e13 Update key retrieval method in StorageCollector to support both standard and firmware keys 2026-01-08 08:52:02 +00:00
Lloyd 4e1eb888e5 Refactor database connection handling in StorageCollector and reduce log verbosity in RoomServer to minimize spam during idle periods 2025-12-18 22:12:25 +00:00
Lloyd d321612670 Add node name lookup by public key and enhance CORS handling in API endpoints 2025-12-18 21:26:31 +00:00
Lloyd 9b53c5b6de Update packet handling to skip invalid advert packets and update record_packet method to conditionally publish to LetsMesh 2025-12-02 16:09:43 +00:00
Lloyd 52e4f6bef5 add hardware statistics collection to StorageCollector and API endpoints 2025-11-29 23:03:15 +00:00
Lloyd 788db09b59 feat: add delete advert functionality to SQLiteHandler and API endpoints 2025-11-24 21:27:12 +00:00
Lloyd 3f3771d600 feat: implement PacketRecord class for standardized packet handling and update MQTTHandler and StorageCollector to utilize it 2025-11-21 15:35:54 +00:00
Lloyd 08d4e054b0 clean up code formatting and improve readability in letsmesh_handler and storage_collector 2025-11-20 10:11:06 +00:00
Lloyd f0ccf161c9 add disallowed packet types configuration for LetsMesh and implement filtering in storage collector 2025-11-19 15:33:39 +00:00
Lloyd 3a2466f953 refactor LetsMesh handler initialization and add live stats support 2025-11-19 11:28:00 +00:00
Lloyd 1793973a84 tidy configuration handling by adding get_node_info function 2025-11-19 10:39:14 +00:00
Lloyd 007182deb1 MeshCoreToMqttJwtPusher initialization and packet publishing with node name and radio configuration 2025-11-18 23:36:10 +00:00
Lloyd 8ef2d3997d feat: update raw packet handling in StorageCollector and RepeaterHandler 2025-11-18 23:05:34 +00:00
Lloyd 785cdd60f7 feat: update packet publishing logic in StorageCollector to use 'raw_packet' 2025-11-18 22:45:30 +00:00
Lloyd 57db8d8831 feat: fix indentation in LetsMesh handler initialization in StorageCollector 2025-11-18 22:30:44 +00:00
Lloyd 1660f1f8b3 fix typo 2025-11-18 22:27:09 +00:00
Lloyd eabd5245fd feat: fix indentation and improve LetsMesh handler initialization in StorageCollector 2025-11-18 22:26:04 +00:00