diff --git a/CHANGELOG.md b/CHANGELOG.md index f9e4d11..0821c96 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,136 @@ Format follows [Keep a Changelog](https://keepachangelog.com/) and [Semantic Ver --- +## [1.22.3] - 2026-08-16 β€” Reply defaults to the message's own channel + dated message timestamps + +### Fixed +- πŸ›  **Route page reply select ignored the message's channel** + (`gui/route_page.py`): `_render_send_panel` derived its default from + the *position* in the channel list (`data['channels'][0]['idx']`, + normally Public) instead of from the message the page was opened for. + Opening the route of a `#noodnet-ov` message and pressing Send without + touching the select therefore published the reply on the wrong + channel. The default now comes from `msg.channel` via the new + `_default_reply_channel()` helper, which falls back to the first + available channel for DMs (`channel is None`) and for channel indices + that are no longer present on the device β€” a slot can be vacated by + the post-discovery cache sync added in 1.22.2. + +### Added +- ✨ **`Message.date` field** (`core/models.py`): local date as + `YYYY-MM-DD`, populated by `Message.incoming()` and + `Message.outgoing()`. Declared last in the dataclass so positional + construction stays backward-compatible, and defaulted to `""` so + existing callers are unaffected. +- ✨ **`Message.now_date()` and `Message.display_timestamp()`** + (`core/models.py`): `display_timestamp()` is the single formatting + authority for message timestamps β€” it returns + `YYYY-MM-DD HH:MM:SS` when the date is known and degrades to the bare + `HH:MM:SS` when it is not. +- ✨ **`date` persisted in the message archive** + (`services/message_archive.py`): `add_message` writes the field + alongside `time` and `timestamp_utc`. + +### Changed +- πŸ”„ **Message lines now show the date before the time** + (`core/models.py`, `gui/route_page.py`, + `gui/panels/room_server_panel.py`): `Message.format_line()` opens + with `display_timestamp()`, so the dashboard messages panel and the + archive page pick the change up automatically. The route page + message-info header and the room-server message list, which format + their own lines, were switched to the same helper for consistency. + +### Impact +- Archive JSON gains one field; the schema stays additive, so older + archives load unchanged and the domca.nl collector is unaffected. + Entries written before this release carry no `date`, so + `Message.from_dict()` derives one from `timestamp_utc`, converted to + local time so it matches the local `time` field across midnight. + When `timestamp_utc` is absent or unparseable the date stays empty and + the line renders exactly as it did in 1.22.2. +- Message display lines are ~11 characters wider. No public method + signature changed; `Message.incoming()` gained an optional + keyword-only `date` parameter. +- `VERSION` bumped `1.22.2` β†’ `1.22.3` (PATCH: backwards-compatible + bugfix plus an additive, defaulted model field). + +### Rationale +The reply select is on a page that exists to show *one specific* +message, so the message β€” not the channel list ordering β€” is the correct +source for the default; deriving it from list position was a latent +mis-send waiting to happen. For the timestamp, the date could not be +shown by editing a label: `Message` simply had no date, because +`now_timestamp()` produces `HH:MM:SS` only. The date existed solely as +`timestamp_utc` in the archive layer and was not read back by +`from_dict()`. Adding a first-class `date` field fixes the cause rather +than papering over it at each render site, and routing every render site +through `display_timestamp()` keeps the format in one place. + +--- + + +## [1.22.2] - 2026-05-04 β€” Backup honors live channel set + post-discovery cache cleanup + +### Fixed +- πŸ›  **Channel backup exported empty-name slots** (`services/channel_backup_store.py`): + `export_from_cache` built its index list from the union of cached + channel names and cached PSKs. On a device with `MAX_GROUP_CHANNELS=100` + firmware-initialised slots, the cache could hold up to 100 PSKs while + only 3 slots were actually named β€” producing a backup file with 97 + empty-name entries that contradicted the dialog's promise ("This + device reports 3 active channels"). The export now uses the + named-slot set (`name_by_idx`) as the sole authority for which slots + to write; PSKs continue to be looked up from cache, but only for + named slots. +- πŸ›  **PacketDecoder accumulated stale keys across sessions** + (`ble/worker.py`, `ble/packet_decoder.py`): `_discover_channels` now + performs a post-discovery sync that drops any cache entry and any + decoder mapping for slots that did not appear in the discovered + channel set. Without this, brute-force key matching ran across all + 100 keys per packet and could mis-route messages to vacated slot + indices, which in turn caused the bot's channel guard to silently + reject legitimate replies (the symptom previously workaround-fixed + by manually wiping `~/.meshcore-gui/cache/`). A safety net skips + the sync when fewer than two channels were discovered, so a + transient discovery failure (which falls back to the Public-only + list) does not wipe the cache. + +### Added +- **`PacketDecoder.remove_channel_key(channel_idx)`** + (`ble/packet_decoder.py`): removes every registered secret whose + mapping points at the given index. Used by the post-discovery sync; + also available for future delete/move flows that want to evict keys + from the in-memory decoder without waiting for the next discovery. + +### Changed +- `VERSION` bumped `1.22.1` β†’ `1.22.2` (PATCH: backwards-compatible + bug fix). + +### Impact +- Existing on-disk cache files are not migrated automatically; the + first successful discovery after upgrading will prune stale entries + in place. +- Backup files written by 1.22.1 that contain empty-name entries + remain readable by Restore β€” `diff_against_device` still classifies + empty-PSK entries as `skipped`, so no destructive change happens on + re-import. +- No public API or BLE protocol changes. No GUI panel changes. The + bot, BBS, channel-add/move/delete flows, REST API, and `domca.nl` + ingest are untouched. + +### Rationale +The dialog text already advertised "every channel currently known to +this GUI" but the implementation broadcast PSK presence as proof of +existence, leading to silent divergence between what the dialog +promised and what the file contained. The cache-cleanup half of the +fix removes the underlying source of that divergence and, as a side +effect, eliminates the brute-force decoder pollution that was the +root cause of the bot-not-replying behaviour observed before manual +cache wipes. + +--- + + ## [1.22.1] - 2026-04-27 ### Added diff --git a/docs/DEV_RULES.md b/docs/DEV_RULES.md deleted file mode 100644 index 4d1e757..0000000 --- a/docs/DEV_RULES.md +++ /dev/null @@ -1,32 +0,0 @@ -### Map developer rules - -To avoid regressions in the map subsystem, follow these rules: - -**Do** - -- Keep the Leaflet map lifecycle **inside the browser runtime** -- Initialize Leaflet **exactly once per DOM container** -- Send **compact snapshots only** from Python -- Update markers **incrementally by node id** -- Keep theme handling in a **dedicated theme channel** -- Allow the browser runtime to maintain **viewport state** -- Define map/tile `maxZoom` **before** attaching clustering layers - -**Do NOT** - -- Recreate the map inside the 500 ms dashboard update loop -- Call `L.map(...)` from snapshot handlers, timers or retry loops -- Use `ui.leaflet()` or any NiceGUI map wrapper -- Embed theme state inside snapshot payloads -- Force map center/zoom during normal refresh cycles -- Call Leaflet APIs directly from Python -- Place the device marker inside the contact cluster layer - -Breaking these rules will reintroduce: - -- disappearing maps -- marker flicker -- viewport resets -- theme resets -- cluster bootstrap failures -- `Map container is already initialized` errors diff --git a/docs/FEATURE_MESSAGE_PERSISTENCE.md b/docs/FEATURE_MESSAGE_PERSISTENCE.md index 7b08c1b..dd893d6 100644 --- a/docs/FEATURE_MESSAGE_PERSISTENCE.md +++ b/docs/FEATURE_MESSAGE_PERSISTENCE.md @@ -8,6 +8,8 @@ This feature implements persistent storage for all incoming messages, RX log entries, and contacts with configurable retention periods. The system uses a dual-layer architecture to balance real-time UI performance with comprehensive data retention. +The architectural decision (why two layers, why independent locks, lock-ordering rules, schema-evolution policy) is recorded in **`docs/adr/ADR-005-dual-layer-persistence.md`**. This document is the feature reference: storage format, API, performance, testing. + ## Architecture ``` @@ -29,14 +31,6 @@ This feature implements persistent storage for all incoming messages, RX log ent β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` -### Design Principles - -1. **Separation of Concerns**: SharedData handles real-time UI updates, MessageArchive handles persistence -2. **Thread Safety**: Independent locks prevent contention between UI and archiving -3. **Batch Writes**: Buffered writes reduce disk I/O (flushes every 10 items or 60 seconds) -4. **Configurable Retention**: Automatic cleanup based on configurable periods -5. **Backward Compatibility**: SharedData API unchanged, archive is optional - ## Storage Format ### Messages Archive @@ -207,17 +201,14 @@ Cleanup is non-blocking and runs in the background worker thread. ## Thread Safety -### Lock Ordering -1. SharedData acquires its lock -2. SharedData calls MessageArchive methods -3. MessageArchive acquires its own lock +Lock-ordering rules (SharedData β†’ MessageArchive) and the rationale for +two independent locks are documented in +**`docs/adr/ADR-005-dual-layer-persistence.md`**. -This ordering prevents deadlocks. - -### Concurrent Access -- SharedData lock: Protects in-memory buffers -- MessageArchive lock: Protects file writes and batch buffers -- Independent locks prevent contention +In short: +- SharedData lock protects in-memory buffers +- MessageArchive lock protects file writes and batch buffers +- Locks are independent β€” no contention between UI and archiving ## Error Handling diff --git a/docs/INSTALLATIE.md b/docs/INSTALLATIE.md deleted file mode 100644 index 2523b57..0000000 --- a/docs/INSTALLATIE.md +++ /dev/null @@ -1,165 +0,0 @@ -# MeshCore GUI β€” BLE Stabiliteit: Installatie-instructies (Legacy) - -> **Let op:** Dit document is BLE-specifiek en wordt bewaard als referentie. De huidige GUI gebruikt USB-serieel; gebruik het handmatige systeemd-voorbeeld in de README. - -## Wat is gewijzigd - -### Nieuwe bestanden -| Bestand | Doel | -|---------|------| -| `meshcore_gui/ble/ble_agent.py` | Ingebouwde BlueZ D-Bus PIN agent (vervangt `bt-agent.service`) | -| `meshcore_gui/ble/ble_reconnect.py` | Bond-opruiming + automatische reconnect logica | -| `install_ble_stable.sh` | Generiek installatiescript (detecteert paden/user automatisch) | - -### Gewijzigde bestanden -| Bestand | Wijziging | -|---------|-----------| -| `meshcore_gui/ble/worker.py` | Agent startup, disconnect detectie, auto-reconnect loop | -| `meshcore_gui/config.py` | Nieuwe constanten: `BLE_PIN`, `RECONNECT_MAX_RETRIES`, `RECONNECT_BASE_DELAY` | - ---- - -## Snelle installatie (aanbevolen) - -```bash -# 1. Verwijder eerst een eventuele kapotte service -sudo systemctl stop meshcore-gui 2>/dev/null -sudo systemctl disable meshcore-gui 2>/dev/null -sudo rm -f /etc/systemd/system/meshcore-gui.service -sudo systemctl daemon-reload -sudo systemctl reset-failed 2>/dev/null - -# 2. Kopieer de nieuwe/gewijzigde bestanden naar je project -cp ble_agent.py ~/meshcore-gui/meshcore_gui/ble/ -cp ble_reconnect.py ~/meshcore-gui/meshcore_gui/ble/ -cp worker.py ~/meshcore-gui/meshcore_gui/ble/ -cp config.py ~/meshcore-gui/meshcore_gui/ - -# 3. Ga naar je project directory en voer het installatiescript uit -cd ~/meshcore-gui -BLE_ADDRESS=FF:05:D6:71:83:8D bash install_ble_stable.sh -``` - -Het script detecteert automatisch: -- De juiste project directory (waar je het uitvoert) -- De huidige user -- Het pad naar de venv Python -- Het correcte entry point - ---- - -## Handmatige installatie - -Als je het script niet wilt gebruiken: - -### 1. Kopieer Python bestanden -```bash -# Pas het pad aan naar jouw project directory -PROJECT=~/meshcore-gui - -cp ble_agent.py $PROJECT/meshcore_gui/ble/ -cp ble_reconnect.py $PROJECT/meshcore_gui/ble/ -cp worker.py $PROJECT/meshcore_gui/ble/ -cp config.py $PROJECT/meshcore_gui/ -``` - -### 2. Upgrade meshcore library -```bash -cd $PROJECT -source venv/bin/activate -pip install --upgrade meshcore -``` - -### 3. D-Bus policy installeren -Maak `/etc/dbus-1/system.d/meshcore-ble.conf` met je eigen username: -```bash -sudo tee /etc/dbus-1/system.d/meshcore-ble.conf << 'EOF' - - - - - - - - -EOF -``` - -### 4. Systemd service installeren -Maak `/etc/systemd/system/meshcore-gui.service` met je eigen paden: -```bash -sudo tee /etc/systemd/system/meshcore-gui.service << EOF -[Unit] -Description=MeshCore GUI (BLE) -After=bluetooth.target -Wants=bluetooth.target - -[Service] -Type=simple -User=$(whoami) -WorkingDirectory=$PROJECT -ExecStart=$PROJECT/venv/bin/python meshcore_gui.py JOUW_BLE_ADRES --debug-on -Restart=on-failure -RestartSec=30 -Environment=DBUS_SYSTEM_BUS_ADDRESS=unix:path=/var/run/dbus/system_bus_socket - -[Install] -WantedBy=multi-user.target -EOF - -sudo systemctl daemon-reload -sudo systemctl enable meshcore-gui -sudo systemctl start meshcore-gui -``` - ---- - -## Verwijderen - -### Via het script -```bash -cd ~/meshcore-gui -bash install_ble_stable.sh --uninstall -``` - -### Handmatig -```bash -sudo systemctl stop meshcore-gui -sudo systemctl disable meshcore-gui -sudo rm -f /etc/systemd/system/meshcore-gui.service -sudo rm -f /etc/dbus-1/system.d/meshcore-ble.conf -sudo systemctl daemon-reload -sudo systemctl reset-failed -``` - ---- - -## Verificatie - -```bash -# Service status -sudo systemctl status meshcore-gui - -# Live logs -journalctl -u meshcore-gui -f - -# Test PIN pairing (vanuit een andere terminal) -bluetoothctl remove -sudo systemctl restart meshcore-gui - -# Test disconnect recovery -# Zet device uit β†’ wacht 30s β†’ zet weer aan β†’ check logs -``` - ---- - -## Configuratie (config.py) - -```python -BLE_PIN = "123456" # T1000e pairing PIN -RECONNECT_MAX_RETRIES = 5 # Max pogingen per disconnect -RECONNECT_BASE_DELAY = 5.0 # Wachttijd Γ— poging nummer (5s, 10s, 15s...) -``` - -Pas deze waarden aan in `meshcore_gui/config.py` als je een ander device of andere timing nodig hebt. diff --git a/docs/INTEGRATION_GUIDE.md b/docs/INTEGRATION_GUIDE.md index 089fbc1..1c6cc9a 100644 --- a/docs/INTEGRATION_GUIDE.md +++ b/docs/INTEGRATION_GUIDE.md @@ -6,6 +6,10 @@ Deze wijziging lost het BlueZ 5.82 probleem op door `meshcore-ble-connect` als **persistent subprocess** te draaien dat de D-Bus/BLE connectie openhoudt, terwijl `BleakClient` er alleen GATT service discovery overheen doet. +De architectuur, de flow en de risico-mitigatie staan in +**`docs/adr/ADR-003-subprocess-ble-connect.md`**. Dit document bevat alleen +de installatie- en delivery-stappen. + --- ## Gewijzigde bestanden @@ -28,43 +32,6 @@ terwijl `BleakClient` er alleen GATT service discovery overheen doet. --- -## Architectuur - -``` -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ meshcore-gui (worker.py) β”‚ -β”‚ β”‚ -β”‚ 1. ensure_bond() β†’ bond OK β”‚ -β”‚ 2. start subprocess: β”‚ -β”‚ meshcore-ble-connect MAC --pin X β”‚ -β”‚ --connect β”‚ -β”‚ 3. wait for "READY" op stdout β”‚ -β”‚ 4. BleakScanner.find_device_by_address() β”‚ -β”‚ β†’ populeert bleak's interne cache β”‚ -β”‚ 5. BleakClient(addr).connect() β”‚ -β”‚ β†’ bleak ziet Connected=True in BlueZ β”‚ -β”‚ β†’ slaat Device1.Connect() over β”‚ -β”‚ β†’ doet alleen GATT service discovery β”‚ -β”‚ 6. MeshCore.create_ble(client=client) β”‚ -β”‚ 7. Main loop draait β”‚ -β”‚ 8. subprocess health check elke 100ms β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - β”‚ subprocess (stdout PIPE) - β”‚ -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ meshcore-ble-connect --connect β”‚ -β”‚ β”‚ -β”‚ 1. Bond flow (ensure/pair/trust) β”‚ -β”‚ 2. Device1.Connect() via D-Bus β”‚ -β”‚ 3. Poll ServicesResolved tot True β”‚ -β”‚ 4. print("READY") β†’ stdout β”‚ -β”‚ 5. Monitor Connected property β”‚ -β”‚ β†’ print("DISCONNECTED") bij verlies β”‚ -β”‚ 6. Wacht op SIGTERM of disconnect β”‚ -β”‚ 7. Device1.Disconnect() bij shutdown β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -``` - ## Installatie ```bash @@ -90,19 +57,3 @@ cd ~/meshcore-gui && python -m meshcore_gui Als het subprocess faalt (bijv. op BlueZ < 5.78 waar het niet nodig is), valt `_connect()` automatisch terug op de directe `MeshCore.create_ble(address)` aanroep. Dit garandeert backwards compatibility. - -## Risico-mitigatie - -Het document noemde het risico dat bleak mogelijk opnieuw `Device1.Connect()` -aanroept. Dit is opgelost door: - -1. **`BleakScanner.find_device_by_address()`** vΓ³Γ³r `BleakClient.connect()` β€” - dit triggert bleak's `BlueZManager` singleton om bestaande BlueZ device - objecten te ontdekken via `GetManagedObjects()`. -2. De manager ziet `Connected=True` op het device (gezet door het subprocess) - β†’ `BleakClient.connect()` slaat `Device1.Connect()` over. -3. Bleak doet alleen GATT service resolution over de bestaande connectie. - -Als de scanner het device niet vindt (het adverteert mogelijk niet terwijl -het connected is), probeert bleak alsnog. De `GetManagedObjects` call bij -manager initialisatie vangt dit op in de meeste gevallen. diff --git a/docs/MAP_ARCHITECTURE.md b/docs/MAP_ARCHITECTURE.md index ac62404..65bddca 100644 --- a/docs/MAP_ARCHITECTURE.md +++ b/docs/MAP_ARCHITECTURE.md @@ -10,6 +10,8 @@ NiceGUI acts only as a container and data provider. This architecture prevents map resets, marker flicker, and viewport jumps during the 500 ms dashboard refresh cycle. +The rationale and binding rules for this approach are recorded in **`docs/adr/ADR-001-browser-managed-leaflet.md`**. + --- # Architecture @@ -174,26 +176,6 @@ This prevents marker flicker during the refresh loop. --- -# Important Constraints - -Developers must **not**: - -* recreate the Leaflet map inside the dashboard refresh loop -* call `L.map(...)` from snapshot handlers, retry loops or timer callbacks -* embed theme state in snapshots -* call Leaflet APIs directly from Python -* force viewport resets during normal snapshot updates -* place the device marker inside the contact cluster layer - -Violating these rules will reintroduce: - -* disappearing maps -* marker flicker -* viewport resets -* theme resets - ---- - # Reconnect Behaviour When the NiceGUI connection temporarily drops: @@ -229,3 +211,4 @@ Browser β†’ map lifecycle This prevents UI refresh cycles from interfering with map state and ensures smooth rendering even with frequent dashboard updates. +Binding constraints (what developers must and must not do) are listed in **`docs/adr/ADR-001-browser-managed-leaflet.md`** under "Consequences β†’ Bindende uitvloeisels". diff --git a/docs/MULTI_INSTANCE.md b/docs/MULTI_INSTANCE.md index 1c8edd0..a825a1b 100644 --- a/docs/MULTI_INSTANCE.md +++ b/docs/MULTI_INSTANCE.md @@ -129,3 +129,48 @@ rm ~/.meshcore-gui/logs/_dev_ttyUSB1_meshcore_gui.log rm ~/.meshcore-gui/pins/_dev_ttyUSB1_pins.json rm ~/.meshcore-gui/room_passwords/_dev_ttyUSB1_rooms.json ``` + +## Troubleshooting + +### Port already in use + +If meshcore-gui fails to start with: + +``` +ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8081): address already in use +``` + +a previous instance is still running, or the port hasn't been released yet. + +**Quick fix β€” kill previous instance and free the port:** + +```bash +pkill -9 -f meshcore_gui +sleep 3 +lsof -i :8081 +``` + +If `lsof` returns nothing, the port is free. Start the service again: + +```bash +sudo systemctl restart meshcore-gui-device1 +``` + +**If the port is still in use after killing**, sometimes TCP sockets linger +in `TIME_WAIT` state. Wait 30 seconds: + +```bash +sleep 30 +lsof -i :8081 +``` + +**Running in background with nohup (for foreground tests, not the systemd +flow above):** always redirect output to a log file so you can diagnose: + +```bash +# Good - keeps logs for debugging +nohup python meshcore_gui.py /dev/ttyUSB0 --debug-on --port=8081 > ~/meshcore.log 2>&1 & + +# Bad - hides all errors +nohup python meshcore_gui.py /dev/ttyUSB0 --debug-on --port=8081 > /dev/null 2>&1 & +``` diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md deleted file mode 100644 index c2b9605..0000000 --- a/docs/TROUBLESHOOTING.md +++ /dev/null @@ -1,424 +0,0 @@ -# MeshCore GUI - Legacy BLE Troubleshooting Guide - -> **Note:** This guide applies to BLE connections only and is kept for historical reference. The current GUI uses USB serial; for serial issues, verify the correct port (e.g. `/dev/ttyUSB0`) and user permissions (e.g. `dialout` on Linux). - -## Problem 1: EOFError during start_notify - -BLE connection to MeshCore device fails with `EOFError` during `start_notify` on the UART TX characteristic. The error originates in `dbus_fast` (the D-Bus library used by `bleak`) and looks like this: - -``` -File "src/dbus_fast/_private/unmarshaller.py", line 395, in dbus_fast._private.unmarshaller.Unmarshaller._read_sock_with_fds -EOFError -``` - -Basic BLE connect works fine, but subscribing to notifications (`start_notify`) crashes. - -## Problem 2: PIN or Key Missing / Authentication Failure - -BLE connection fails immediately after connecting with `failed to discover services, device disconnected` or `le-connection-abort-by-local`. In `btmon`, the trace shows: - -``` -Encryption Change - Status: PIN or Key Missing (0x06) -Disconnect - Reason: Authentication Failure (0x05) -``` - -This happens when the MeshCore device requires BLE PIN pairing (e.g., PIN `123456`) but no BlueZ agent is running to handle the passkey exchange. Bleak cannot provide a PIN by itself β€” it relies on a BlueZ agent to handle pairing. - -**Symptoms:** -- `bluetoothctl connect` fails with `le-connection-abort-by-local` -- `bluetoothctl pair` asks for a passkey and succeeds -- meshcore-gui still fails because bleak creates its own connection without an agent -- btmon shows repeated connect β†’ encrypt β†’ `PIN or Key Missing` β†’ disconnect cycles - -## Problem 3: Port already in use - -meshcore-gui fails to start with: - -``` -ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8081): address already in use -``` - -This means a previous meshcore-gui instance is still running (or the port hasn't been released yet). - ---- - -## Diagnostic Steps - -### 1. Check adapter status - -```bash -hciconfig -a -``` - -Expected: `UP RUNNING`. If it shows `DOWN`, reset with: - -```bash -sudo hciconfig hci0 down -sudo hciconfig hci0 up -``` - -### 2. Check if adapter is detected - -```bash -lsusb | grep -i blue -``` - -### 3. Check power supply (Raspberry Pi) - -```bash -vcgencmd get_throttled -``` - -Expected: `throttled=0x0`. Any other value indicates power issues that can cause BLE instability. - -### 4. Test basic BLE connection (without notify) - -```bash -python -c " -import asyncio -from bleak import BleakClient -async def test(): - async with BleakClient('AA:BB:CC:DD:EE:FF') as c: - print('Connected:', c.is_connected) -asyncio.run(test()) -" -``` - -If this works but meshcli/meshcore_gui fails, the problem is specifically `start_notify`. - -### 5. Test start_notify in isolation - -```bash -python -c " -import asyncio -from bleak import BleakClient -UART_TX = '6e400003-b5a3-f393-e0a9-e50e24dcca9e' -async def test(): - async with BleakClient('AA:BB:CC:DD:EE:FF') as c: - def cb(s, d): print(f'RX: {d.hex()}') - await c.start_notify(UART_TX, cb) - print('Notify OK!') - await asyncio.sleep(2) -asyncio.run(test()) -" -``` - -If this also fails with `EOFError`, the issue is confirmed at the BlueZ/D-Bus level. - -### 6. Test notifications via bluetoothctl (outside Python) - -```bash -bluetoothctl -scan on -# Wait for device to appear -connect AA:BB:CC:DD:EE:FF -# Wait for "Connection successful" -menu gatt -select-attribute 6e400003-b5a3-f393-e0a9-e50e24dcca9e -notify on -``` - -If `connect` fails with `le-connection-abort-by-local`, the problem is at the BlueZ or device level. No Python fix will help. - -### 7. Check if pairing is required (PIN or Key Missing) - -If `bluetoothctl connect` fails with `le-connection-abort-by-local`, try pairing instead: - -```bash -bluetoothctl -scan on -pair AA:BB:CC:DD:EE:FF -# If it asks for a passkey, the device requires PIN pairing -``` - -If pairing succeeds but meshcore-gui still fails, the issue is a missing BlueZ agent (see Solution 2). - -### 8. Use btmon for HCI-level debugging - -```bash -sudo btmon -``` - -In another terminal, start meshcore-gui. Look for: -- `Encryption Change - Status: PIN or Key Missing (0x06)` β†’ pairing/agent issue (Solution 2) -- Successful encryption but no service discovery β†’ stale bond (Solution 1) - -### 9. Check what is using port 8081 - -```bash -lsof -i :8081 -``` - -If another process holds the port, see Solution 3. - ---- - -## Solution 1: Stale BLE Pairing State (EOFError) - -The root cause is a stale BLE pairing state between the Linux adapter and the MeshCore device. The fix requires a clean reconnect sequence: - -### Step 1 - Remove the device from BlueZ - -```bash -bluetoothctl -remove AA:BB:CC:DD:EE:FF -exit -``` - -### Step 2 - Hard power cycle the MeshCore device - -Physically power off the T1000-e (not just a software reset). Wait 10 seconds, then power it back on. - -### Step 3 - Scan and reconnect from scratch - -```bash -bluetoothctl -scan on -``` - -Wait until the device appears: `[NEW] Device AA:BB:CC:DD:EE:FF MeshCore-...` - -Then immediately connect: - -``` -connect AA:BB:CC:DD:EE:FF -``` - -### Step 4 - Verify notifications work - -``` -menu gatt -select-attribute 6e400003-b5a3-f393-e0a9-e50e24dcca9e -notify on -``` - -If this succeeds, disconnect cleanly: - -``` -notify off -back -disconnect AA:BB:CC:DD:EE:FF -exit -``` - -### Step 5 - Verify channels with meshcli - -```bash -meshcli -d AA:BB:CC:DD:EE:FF -> get_channels -``` - -Confirm output matches `CHANNELS_CONFIG` in `meshcore_gui.py`, then: - -``` -> exit -``` - -### Step 6 - Start the GUI - -```bash -cd ~/meshcore-gui -source venv/bin/activate -python meshcore_gui.py AA:BB:CC:DD:EE:FF -``` - ---- - -## Solution 2: Missing BlueZ Agent for PIN Pairing - -When the MeshCore device requires BLE PIN pairing, bleak cannot provide the PIN by itself. BlueZ needs a running agent that responds to pairing requests with the correct passkey. - -**Why this happens:** `bluetoothctl` acts as its own agent (which is why manual pairing works), but when bleak connects independently, there is no agent to handle the passkey exchange. Even if the device was previously paired via `bluetoothctl`, the bond can become invalid when: -- The MeshCore device is reset or firmware-updated -- Another device (e.g., companion app) pairs with the MeshCore device and overwrites its bond slot -- The bond keys get out of sync for any reason - -### Step 1 - Install bluez-tools - -```bash -sudo apt install bluez-tools -``` - -### Step 2 - Create a PIN file - -```bash -echo "* 123456" > ~/.meshcore-ble-pin -chmod 600 ~/.meshcore-ble-pin -``` - -The format is ` `. Use `*` to match any device, or specify a specific address: - -``` -FF:05:D6:71:83:8D 123456 -``` - -### Step 3 - Remove any existing (corrupt) bond - -```bash -bluetoothctl remove AA:BB:CC:DD:EE:FF -``` - -### Step 4 - Start the agent and meshcore-gui - -```bash -bt-agent -c KeyboardOnly -p ~/.meshcore-ble-pin & -python meshcore_gui.py AA:BB:CC:DD:EE:FF -``` - -### Step 5 - Make the agent permanent (systemd service) - -Create the service file: - -```bash -sudo tee /etc/systemd/system/bt-agent.service << 'EOF' -[Unit] -Description=Bluetooth PIN Agent for MeshCore -After=bluetooth.service -Requires=bluetooth.service - -[Service] -ExecStart=/usr/bin/bt-agent -c KeyboardOnly -p /home/hans/.meshcore-ble-pin -Restart=always -User=hans - -[Install] -WantedBy=multi-user.target -EOF -``` - -Enable and start: - -```bash -sudo systemctl enable bt-agent -sudo systemctl start bt-agent -``` - -Verify it is running: - -```bash -sudo systemctl status bt-agent -``` - -Now meshcore-gui can connect at any time without manual pairing. The agent survives reboots. - -**Important:** Only run ONE bt-agent instance. Multiple agents conflict with each other. If you have both a manual `bt-agent &` process and the systemd service running, kill the manual one: - -```bash -pkill -f bt-agent -sudo systemctl start bt-agent -``` - ---- - -## Solution 3: Port 8081 Already in Use - -This happens when a previous meshcore-gui instance is still running or hasn't fully released the port. - -### Quick fix - Kill previous instance and free the port - -```bash -pkill -9 -f meshcore_gui -sleep 3 -``` - -Verify the port is free: - -```bash -lsof -i :8081 -``` - -If nothing shows up, the port is free. Start meshcore-gui: - -```bash -nohup python meshcore_gui.py AA:BB:CC:DD:EE:FF --debug-on > ~/meshcore.log 2>&1 & -``` - -### If the port is still in use after killing - -Sometimes TCP sockets linger in `TIME_WAIT` state. Wait 30 seconds or force it: - -```bash -sleep 30 -lsof -i :8081 -``` - -### Running in background with nohup - -To run meshcore-gui in the background (survives terminal close): - -```bash -nohup python meshcore_gui.py AA:BB:CC:DD:EE:FF --debug-on > ~/meshcore.log 2>&1 & -``` - -Check if it started successfully: - -```bash -sleep 5 -tail -30 ~/meshcore.log -``` - -**Tip:** Always redirect output to a log file (not `/dev/null`) so you can diagnose problems: - -```bash -# Good - keeps logs for debugging -nohup python meshcore_gui.py AA:BB:CC:DD:EE:FF --debug-on > ~/meshcore.log 2>&1 & - -# Bad - hides all errors -nohup python meshcore_gui.py AA:BB:CC:DD:EE:FF --debug-on > /dev/null 2>&1 & -``` - ---- - -## Things That Did NOT Help - -| Action | Result | -|---|---| -| `sudo systemctl restart bluetooth` | No effect | -| `sudo hciconfig hci0 down/up` | No effect | -| `sudo rmmod btusb && sudo modprobe btusb` | No effect | -| `sudo usbreset "8087:0026"` | No effect | -| `sudo reboot` | No effect | -| Clearing BlueZ cache (`/var/lib/bluetooth/*/cache`) | No effect | -| Recreating Python venv | No effect | -| Downgrading `dbus_fast` / `bleak` | No effect | -| Downgrading `linux-firmware` | No effect | -| Adding `pin="123456"` to `MeshCore.create_ble()` | Pairing fails β€” bleak's `pair()` cannot provide a passkey without a BlueZ agent | -| Pre-connecting via `bluetoothctl connect` before meshcore-gui | Bleak creates its own connection and doesn't reuse the existing one | - ---- - -## Key Takeaways - -### EOFError / stale bond -When `start_notify` fails with `EOFError` but basic BLE connect works, the issue is almost always a stale BLE state between the host adapter and the peripheral device. The fix is: - -1. **Remove** the device from bluetoothctl -2. **Hard power cycle** the peripheral device -3. **Re-scan** and reconnect from scratch - -### PIN or Key Missing / Authentication Failure -When btmon shows `PIN or Key Missing (0x06)` and connections drop immediately after encryption negotiation, the fix is: - -1. **Remove** the corrupt bond from bluetoothctl -2. **Run `bt-agent`** with the correct PIN file so BlueZ can handle pairing requests -3. **Install as systemd service** for persistence across reboots - -### Port already in use -When meshcore-gui fails with `[Errno 98] address already in use`: - -1. **Kill** any existing meshcore-gui process: `pkill -9 -f meshcore_gui` -2. **Wait** a few seconds for the port to be released -3. **Verify** the port is free: `lsof -i :8081` - ---- - -## Recommended Startup Sequence - -For the most reliable BLE connection, always follow this order: - -1. Ensure `bt-agent` is running (if device requires PIN pairing): `sudo systemctl status bt-agent` -2. Ensure no other meshcore-gui instance is running: `pkill -f meshcore_gui` and `lsof -i :8081` -3. Ensure no other application holds the BLE connection (BT manager, bluetoothctl, meshcli, companion app) -4. Verify the device is visible: `bluetoothctl scan on` -5. Check channels: `meshcli -d ` β†’ `get_channels` β†’ `exit` -6. Start the GUI: `python meshcore_gui.py ` diff --git a/docs/adr/ADR-001-browser-managed-leaflet.md b/docs/adr/ADR-001-browser-managed-leaflet.md new file mode 100644 index 0000000..5b46700 --- /dev/null +++ b/docs/adr/ADR-001-browser-managed-leaflet.md @@ -0,0 +1,79 @@ +# ADR-001: Browser-managed Leaflet runtime + +**Status:** Accepted β€” 2026-02 + +## Context + +De MeshCore GUI toont contacten en eigen positie op een Leaflet-kaart in +het centrum-paneel van de dashboard. De dashboard-update-loop draait elke +500 ms en pusht een snapshot van device- en contact-data naar de UI. + +Eerdere implementaties stuurden de map-lifecycle vanuit Python aan +(NiceGUI-wrappers, JS-injection vanuit snapshot-handlers). Dat +veroorzaakte: + +- Verdwijnende kaarten omdat `L.map(...)` in de refresh-loop opnieuw + werd aangeroepen op een al geΓ―nitialiseerd container-element + (`Map container is already initialized`). +- Marker-flicker omdat de hele marker-laag elke 500 ms werd vervangen + in plaats van incrementeel bijgewerkt. +- Viewport-resets en thema-resets omdat thema-state in snapshots zat + en queued snapshots gebruikersacties overschreven. +- Cluster-bootstrap-fouten wanneer de cluster-laag werd aangehecht + voordat `maxZoom` op map of tile-layer bekend was. + +## Decision + +**De Leaflet-map-lifecycle is volledig in handen van de browser.** +Python is data-leverancier en nooit aanroeper van Leaflet-API's. + +Concreet: + +- Leaflet wordt **één keer** geΓ―nitialiseerd per DOM-container in + `static/leaflet_map_panel.js`. +- Python (`MapPanel`, `MapSnapshotService`) stuurt **compacte JSON- + snapshots**; de browser past deze incrementeel toe per stabiele + node-id. +- **Thema-state loopt over een eigen kanaal**, niet door snapshots. +- **Viewport-state blijft bij de browser**; snapshots forceren geen + center/zoom tijdens normale refresh-cycles. +- De **eigen-device-marker zit buiten** de contact-cluster-layer. +- `maxZoom` van map/tile-layer wordt gezet **vΓ³Γ³r** clustering wordt + aangehecht. + +NiceGUI-map-wrappers (`ui.leaflet()` of vergelijkbaar) worden niet +gebruikt. + +## Consequences + +**Plus** + +- Geen flicker, geen viewport-resets, geen thema-resets tijdens de + refresh-cyclus. +- Reconnect na korte NiceGUI-disconnect blijft de map en lokale state + intact. +- Map-state overleeft snapshot-coalescing: alleen het nieuwste payload + wordt toegepast. + +**Min** + +- Leaflet-API's zijn niet vanuit Python aanroepbaar; uitbreidingen + (heatmap, route-overlays, tile-switching) moeten als browser-runtime- + uitbreiding worden geΓ―mplementeerd. +- Debuggen vereist DevTools-inspectie van de runtime, niet alleen + Python-logs. + +**Bindende uitvloeisels** (zie `DEV_RULES.md`): + +- Recreate map in 500 ms loop β†’ verboden +- `L.map(...)` vanuit Python β†’ verboden +- Thema in snapshots β†’ verboden +- Device-marker in cluster-layer β†’ verboden + +## References + +- `MAP_ARCHITECTURE.md` +- `DEV_RULES.md` +- `meshcore_gui/static/leaflet_map_panel.js` +- `meshcore_gui/services/map_snapshot_service.py` +- `meshcore_gui/gui/panels/map_panel.py` diff --git a/docs/adr/ADR-002-protocol-interfaces.md b/docs/adr/ADR-002-protocol-interfaces.md new file mode 100644 index 0000000..6bb0e9e --- /dev/null +++ b/docs/adr/ADR-002-protocol-interfaces.md @@ -0,0 +1,86 @@ +# ADR-002: Protocol-interfaces (`typing.Protocol`) i.p.v. `abc.ABC` + +**Status:** Accepted β€” 2026-02 + +## Context + +De applicatie heeft één centrale, thread-safe data-store (`SharedData`) +die door meerdere consumenten wordt gebruikt: + +- `BLEWorker` schrijft device-info, contacten, channels, messages en + RX-log entries. +- `DashboardPage` leest snapshots en plaatst commands. +- `RouteBuilder` zoekt contacten op prefix. +- `RoutePage` en `ArchivePage` doen beide read en lookup. +- Widget-klassen ontvangen alleen plain dicts en callbacks. + +VΓ³Γ³r de SOLID-refactor importeerden al deze consumenten direct +`shared_data.SharedData`. Dat: + +- Verstopte dat de meeste consumenten maar een fractie van de 15+ + publieke methoden gebruiken (ISP-violation). +- Maakte testen lastig β€” een test-stub moest alle methoden bieden, + ook de niet-gebruikte. +- Bond consumenten aan de concrete implementatie i.p.v. aan het + contract dat ze nodig hebben (DIP-violation). + +## Decision + +**Definieer per consument een smal Protocol-contract in `protocols.py`;** +consumenten typen tegen het contract, niet tegen `SharedData`. + +GeΓ―ntroduceerde Protocols: + +| Protocol | Consument | Methoden | +|----------|-----------|----------| +| `SharedDataWriter` | `BLEWorker` | 10–15 (write + lookup) | +| `SharedDataReader` | `DashboardPage` | 4–6 (snapshot + commands) | +| `ContactLookup` | `RouteBuilder` | 1 | +| `SharedDataReadAndLookup` | `RoutePage`, `ArchivePage` | Reader + Lookup | +| `CommandSink` | `MeshBot`, GUI-pages | `put_command` | + +`SharedData` blijft één concrete klasse die alle Protocols +**structureel** implementeert (zonder `from typing import Protocol`- +inheritance). De **composition root** (`meshcore_gui.py`) is de enige +plek die de concrete klasse kent. + +`typing.Protocol` is gekozen **boven `abc.ABC`** omdat: + +| Aspect | `abc.ABC` (nominal) | `typing.Protocol` (structural) | +|--------|---------------------|-------------------------------| +| Subclassing nodig | ja | nee | +| Duck typing | nee | ja | +| Test-stubs | moeten erven | hoeven alleen methoden te hebben | + +## Consequences + +**Plus** + +- Elke consument ziet alleen wat hij nodig heeft (ISP). +- Test-stubs zijn lichtgewicht: alleen de methoden van het smalle + contract. +- Refactoring van `SharedData`-internen raakt consumenten niet zolang + de Protocol-signatures stabiel blijven (DIP). +- Widget-klassen hebben **nul** kennis van `SharedData` β€” ze krijgen + plain dicts en callbacks. + +**Min** + +- `typing.Protocol` werkt vanaf Python 3.8 β€” geen probleem in deze + codebase (3.10+). +- Vijf interface-classes zijn extra onderhoud bij echt brede signature- + wijzigingen. Drempel voor nieuwe Protocols: alleen wanneer een + consument een meetbaar smaller contract heeft, niet preventief. + +**Niet introduceren** + +- Geen `abc.ABC` met `@abstractmethod` voor deze use-case. De Protocol- + variant levert hetzelfde type-veiligheidsniveau zonder verplichte + subclassing. + +## References + +- `SOLID_ANALYSIS.md` +- `meshcore_gui/core/protocols.py` +- `meshcore_gui/core/shared_data.py` +- PEP 544 β€” Protocols: Structural subtyping diff --git a/docs/adr/ADR-003-subprocess-ble-connect.md b/docs/adr/ADR-003-subprocess-ble-connect.md new file mode 100644 index 0000000..cc2a6c2 --- /dev/null +++ b/docs/adr/ADR-003-subprocess-ble-connect.md @@ -0,0 +1,80 @@ +# ADR-003: Persistente BLE-connectie via subprocess (`meshcore-ble-connect`) + +**Status:** Accepted β€” 2026-02 (v5.5) + +## Context + +Vanaf BlueZ 5.82 (en deels 5.78+) lukt het bleak niet meer betrouwbaar +om zelf de `Device1.Connect()` D-Bus-aanroep te doen vΓ³Γ³r GATT-service- +discovery. Resultaat: `failed to discover services, device disconnected`, +`le-connection-abort-by-local`, en in `btmon` herhaalde +`PIN or Key Missing (0x06) β†’ Authentication Failure (0x05)`-cycli. + +`bluetoothctl` werkt wel, maar zodra bleak een eigen connectie opzet +ziet BlueZ deze als een nieuwe sessie zonder geldig agent-context. + +De directe BLE-connect-flow vanuit `BLEWorker` werd daarmee +onbetrouwbaar op recente Linux-distributies. + +## Decision + +**Gebruik `meshcore-ble-connect` als persistent subprocess** dat de +D-Bus/BLE-connectie openhoudt; bleak doet alleen GATT-service-discovery +over de bestaande connectie. + +Flow in `BLEWorker._connect()`: + +1. `ensure_bond()` (subprocess pairing-flow met PIN-agent) β€” bond OK. +2. Start subprocess: `meshcore-ble-connect --pin --connect`. +3. Wacht op `READY` op subprocess-stdout. +4. `BleakScanner.find_device_by_address()` β€” populeert bleak's + `BlueZManager`-singleton zodat hij `Connected=True` ziet via + `GetManagedObjects()`. +5. `BleakClient(addr).connect()` β€” bleak slaat `Device1.Connect()` + over, doet alleen GATT-discovery. +6. `MeshCore.create_ble(client=client)` β€” normale event-loop. +7. Subprocess-health-check elke 100 ms in main loop; cleanup in + `finally`-blok. + +Het subprocess monitort `Connected`-property en print `DISCONNECTED` +op stdout bij verlies; `BLEWorker` triggert dan reconnect. + +**Fallback:** als het subprocess faalt te starten (bijv. op BlueZ < 5.78 +waar het niet nodig is), valt `_connect()` terug op directe +`MeshCore.create_ble(address)`. + +## Consequences + +**Plus** + +- Werkt op BlueZ 5.82+ zonder workarounds in `bleak`. +- Strikte separation of concerns: `meshcore-ble-connect` heeft één + taak (D-Bus-connectie openhouden), GUI hoeft geen D-Bus-code te + bevatten. +- Backwards compatible via fallback-pad. + +**Min** + +- Extra externe dependency (`meshcore-ble-connect` als geΓ―nstalleerd + CLI-tool in de venv). +- Subprocess-lifecycle moet bewaakt worden (signal handling, cleanup + in `finally`, kill bij worker-shutdown). +- Potentieel risico dat bleak alsnog `Device1.Connect()` aanroept + als de scanner het device niet ziet adverteren terwijl het connected + is. In de praktijk vangt `GetManagedObjects()` dit op. + +**Bindende uitvloeisels** + +- Geen pairing- of D-Bus-code in `meshcore_gui/`. Bond-management en + agent-registratie horen in de subprocess of in `BleAgentManager` + (zie ADR-004). +- `meshcore-ble-connect` is een **separaat project** met eigen + versionering. Geen systemd-files of install-docs van dat project + in deze repo. + +## References + +- `INTEGRATION_GUIDE.md` +- ADR-004 β€” Built-in D-Bus PIN-agent (complementair) +- `meshcore_ble_connect/app.py`, `device.py`, `__main__.py` +- `meshcore_gui/ble/worker.py` diff --git a/docs/adr/ADR-004-builtin-dbus-pin-agent.md b/docs/adr/ADR-004-builtin-dbus-pin-agent.md new file mode 100644 index 0000000..046ec8c --- /dev/null +++ b/docs/adr/ADR-004-builtin-dbus-pin-agent.md @@ -0,0 +1,75 @@ +# ADR-004: Ingebouwde D-Bus PIN-agent (vervangt `bt-agent.service`) + +**Status:** Accepted β€” 2026-02 (v5.11) + +## Context + +MeshCore-devices vereisen BLE PIN-pairing (default `123456`). Bleak kan +zelf geen passkey leveren β€” BlueZ verwacht een geregistreerde +`org.bluez.Agent1`-implementatie die op pairing-callbacks reageert. + +De eerste werkende oplossing was het `bluez-tools`-pakket met +`bt-agent` als systemd-service, gevoed door een PIN-bestand +(`~/.meshcore-ble-pin`). Dat werkte, maar: + +- Vereiste extra OS-pakket (`apt install bluez-tools`). +- Vereiste een aparte systemd-unit en D-Bus-policy per host. +- Was gevoelig voor "twee agents tegelijk"-conflicten (manueel gestarte + `bt-agent &` plus systemd-service). +- Splitste pairing-state tussen GUI-config en een los config-bestand. + +## Decision + +**Implementeer de pairing-agent intern**, in `meshcore_gui/ble/ble_agent.py`, +gebruikmakend van `dbus_fast` (al een bleak-dependency). + +Componenten: + +- `BluezAgent(ServiceInterface)` β€” implementeert + `org.bluez.Agent1` met callbacks voor `RequestPinCode`, + `RequestPasskey`, `DisplayPasskey`, `RequestConfirmation` en + `AuthorizeService`. Antwoordt op alle PIN-requests met de geconfigureerde + `BLE_PIN` (default `"123456"`). +- `BleAgentManager` β€” `start(pin)` exporteert de agent op D-Bus-pad + `/meshcore/ble_agent` en registreert hem als default agent met + capability `KeyboardOnly`. `stop()` deregistreert en sluit de + D-Bus-verbinding. + +`BLEWorker` start de agent **vΓ³Γ³r** elke connect-poging en cleanupt +in `finally`. PIN staat in `config.py` (`BLE_PIN`). + +## Consequences + +**Plus** + +- Geen externe dependency op `bluez-tools`. +- Geen aparte systemd-service of D-Bus-policy meer nodig voor het + agent-pad (system-bus policy voor `org.bluez.Agent1` blijft). +- EΓ©n bron voor de PIN: `config.py`. +- Lifecycle gebonden aan de Worker β€” geen verweesde agents. + +**Min** + +- Vereist `dbus_fast` als runtime-dependency (was er al via bleak, + maar nu expliciet). +- D-Bus-policy in `/etc/dbus-1/system.d/` blijft per-host nodig zodat + de niet-root user mag praten met `org.bluez`. +- EΓ©n PIN per device-fleet (de agent matcht op `*`). Voor multi-PIN + scenario's moet de agent uitgebreid worden. + +**Bindende uitvloeisels** + +- Nieuwe code voor pairing/agent-registratie hoort in + `meshcore_gui/ble/ble_agent.py` of in `meshcore-ble-connect`. Niet + in `worker.py` of GUI-code. +- `INSTALLATIE.md` is daarmee historisch β€” de daar beschreven + `bt-agent`-stappen zijn **niet** meer aanbevolen. + +## References + +- `MeshCore_GUI_Design.docx` Β§3.18 β€” `BleAgentManager` +- `INSTALLATIE.md` (legacy) +- `TROUBLESHOOTING.md` β€” Solution 2 (legacy bt-agent flow) +- ADR-003 β€” Subprocess-BLE-connect (complementair) +- `meshcore_gui/ble/ble_agent.py` +- `meshcore_gui/config.py` β€” `BLE_PIN` diff --git a/docs/adr/ADR-005-dual-layer-persistence.md b/docs/adr/ADR-005-dual-layer-persistence.md new file mode 100644 index 0000000..45d3a96 --- /dev/null +++ b/docs/adr/ADR-005-dual-layer-persistence.md @@ -0,0 +1,105 @@ +# ADR-005: Dual-layer persistence β€” `SharedData` + `MessageArchive` met onafhankelijke locks + +**Status:** Accepted β€” 2026-02 (v5.1) + +## Context + +De GUI moet twee verschillende dingen tegelijk aankunnen: + +1. **Real-time UI** β€” laatste 100 messages, laatste 50 RX-log-entries, + met sub-second responsetijd in de 500 ms refresh-loop. +2. **Lange-termijn archief** β€” alle inkomende messages en RX-log-entries + bewaren over dagen tot maanden, met retentie en query-mogelijkheden + (filter, search, paginatie, hash-lookup voor route-pagina). + +Deze twee belangen botsen in één datastructuur: + +- EΓ©n onbegrensde lijst geeft geheugendruk en trage UI-updates. +- Disk-I/O in de UI-thread bevriest NiceGUI. +- Gedeelde lock tussen UI-buffer en archive-flush geeft contention β€” + archive-write blokkeert dan een dashboard-refresh. + +## Decision + +**Twee gescheiden lagen met eigen verantwoordelijkheden en eigen locks.** + +``` +SharedData (in-memory, UI) + β”œβ”€β”€ messages: laatste 100 + β”œβ”€β”€ rx_log: laatste 50 + β”œβ”€β”€ threading.Lock() + └── on add β†’ MessageArchive + +MessageArchive (persistent) + β”œβ”€β”€ alle messages / rx-log β†’ JSON + β”œβ”€β”€ batch-buffer (10 items / 60 s) + β”œβ”€β”€ retentie-cleanup (dagelijks) + β”œβ”€β”€ threading.Lock() ← onafhankelijk + └── atomic write (temp + rename) +``` + +**Lock-volgorde** (om deadlock te voorkomen): + +1. `SharedData.lock` acquired +2. `SharedData` roept `MessageArchive.add_*()` +3. `MessageArchive.lock` acquired binnen die call + +Deze volgorde is consistent in Γ‘lle write-paden. + +**Storage-format:** één JSON-bestand per data-type per device-identifier: +- `~/.meshcore-gui/archive/
_messages.json` +- `~/.meshcore-gui/archive/
_rxlog.json` + +Velden mogen alleen worden **toegevoegd** (default-waarde), +niet verwijderd of hernoemd. Schema-versie staat in elk bestand. + +**Retentie** is configureerbaar in `config.py` (`MESSAGE_RETENTION_DAYS`, +`RXLOG_RETENTION_DAYS`, `CONTACT_RETENTION_DAYS`). Cleanup draait +dagelijks in de Worker-thread. + +**Backward-compat:** `SharedData()` zonder device-identifier werkt +zonder archive (`archive=None`). + +## Consequences + +**Plus** + +- UI blijft responsief: archive-flush blokkeert nooit een snapshot. +- Buffered batch-writes (drempel `BATCH_SIZE` of timer) houden disk-I/O + laag; ~10 ms voor 1000 messages. +- Retentie-cleanup is een eigen periodieke taak, niet verweven met + refresh-loop. +- Atomic writes (`tempfile + os.replace`) voorkomen halve archives bij + crash. +- Per-device-isolatie maakt multi-instance triviaal (zie + `MULTI_INSTANCE.md`). + +**Min** + +- Twee locks vereisen disciplinaire lock-volgorde. Afwijken + introduceert deadlock-risico. +- Bij crash tijdens een nog-niet-geflushte batch verlies je tot + `BATCH_SIZE` items of de laatste 60 s. Acceptabel voor mesh- + observatie; niet acceptabel als dit financiΓ«le data was. +- JSON-array-append vereist read-modify-write in plaats van streaming + append. Bestandsgrootte is voorlopig acceptabel (<10 MB / maand bij + 100 msg/dag); als het structureel groter wordt, is migratie naar + JSONL of SQLite een logische vervolgstap. + +**Bindende uitvloeisels** + +- `MessageArchive` heeft een **eigen lock**. Niet kruislings vergrendelen + met `SharedData.lock` buiten de gedefinieerde volgorde. +- Schema-velden: alleen toevoegen met default. Verwijderen/hernoemen + vereist nieuwe ADR + migratie. +- Cleanup-interval staat **niet** afgestemd op de refresh-loop maar + draait dagelijks in de Worker. + +## References + +- `FEATURE_MESSAGE_PERSISTENCE.md` +- `MeshCore_GUI_Design.docx` Β§3.13 β€” `MessageArchive` +- `MULTI_INSTANCE.md` β€” per-device data-isolatie +- `meshcore_gui/core/shared_data.py` +- `meshcore_gui/persistence/message_archive.py` +- `meshcore_gui/config.py` β€” `MESSAGE_RETENTION_DAYS` etc. diff --git a/install_scripts/install_serial.sh b/install_scripts/install_serial.sh index 0b4c765..2e5d34d 100644 --- a/install_scripts/install_serial.sh +++ b/install_scripts/install_serial.sh @@ -19,11 +19,11 @@ # DEBUG_ON=yes|no Enable debug logging (will prompt if omitted) # # Examples β€” two instances on the same machine: -# SERIAL_PORT=/dev/ttyUSB0 WEB_PORT=8081 bash install_scripts/install_serial.sh -# SERIAL_PORT=/dev/ttyUSB1 WEB_PORT=8082 bash install_scripts/install_serial.sh +# SERIAL_PORT=/dev/ttyUSB1 WEB_PORT=8081 bash install_scripts/install_serial.sh +# SERIAL_PORT=/dev/ttyUSB2 WEB_PORT=8082 bash install_scripts/install_serial.sh # # Uninstall a specific instance: -# SERIAL_PORT=/dev/ttyUSB0 bash install_scripts/install_serial.sh --uninstall +# SERIAL_PORT=/dev/ttyUSB1 bash install_scripts/install_serial.sh --uninstall # # List all installed instances: # bash install_scripts/install_serial.sh --list diff --git a/meshcore_gui/ble/packet_decoder.py b/meshcore_gui/ble/packet_decoder.py index 44580bd..fce4f1d 100644 --- a/meshcore_gui/ble/packet_decoder.py +++ b/meshcore_gui/ble/packet_decoder.py @@ -143,6 +143,28 @@ class PacketDecoder: secret_bytes = sha256(channel_name.encode("utf-8")).digest()[:16] self.add_channel_key(channel_idx, secret_bytes, source=f"name '{channel_name}'") + def remove_channel_key(self, channel_idx: int) -> None: + """Remove every registered key whose mapping points at *channel_idx*. + + Used by the BLE worker after channel discovery to drop stale entries + for slots that are no longer active on the device (e.g. after a + delete or move). Removing all mappings for the index β€” instead of a + specific secret β€” keeps the decoder consistent even when the same + slot was previously registered with multiple secrets across the + session (cache load + device confirmation may both succeed and leave + two entries for the same idx if the secret bytes differ). + + Args: + channel_idx: Channel index to evict from the decoder. + """ + stale = [s for s, idx in self._secret_to_idx.items() if idx == channel_idx] + for secret_hex in stale: + del self._secret_to_idx[secret_hex] + if stale: + debug_print( + f"PacketDecoder: removed {len(stale)} key(s) for ch{channel_idx}" + ) + @property def has_keys(self) -> bool: """True if at least one channel key has been registered.""" diff --git a/meshcore_gui/ble/worker.py b/meshcore_gui/ble/worker.py index e56e58b..585dc38 100644 --- a/meshcore_gui/ble/worker.py +++ b/meshcore_gui/ble/worker.py @@ -618,6 +618,33 @@ class _BaseWorker(abc.ABC): self._cache.set_channels(discovered) debug_print("Channel list cached to disk") + # Post-discovery sync: drop stale cache + decoder entries for slots + # that are no longer active (no name) on the device. Without this, + # firmware-initialised slots and historical entries accumulate + # forever β€” leading to backups containing 100 channels when the + # device only has 3, and to brute-force decoder pollution that has + # been observed to mis-route messages to vacated slot indices. + # Safety net: only sync when at least one non-Public channel was + # discovered, so a transient discovery failure (which falls back to + # ``[{"idx": 0, "name": "Public"}]`` above) does not wipe the cache. + if len(discovered) >= 2: + valid_indices = {int(ch["idx"]) for ch in discovered} + removed = 0 + for idx_str in list(self._cache.get_channel_keys()): + try: + idx = int(idx_str) + except (ValueError, TypeError): + continue + if idx not in valid_indices: + self._cache.remove_channel_key(idx) + self._decoder.remove_channel_key(idx) + removed += 1 + if removed: + print( + f"{pfx}: 🧹 Cleaned {removed} stale channel key(s) " + f"from cache and decoder" + ) + print(f"{pfx}: Channels discovered: {[c['name'] for c in discovered]}") print(f"{pfx}: PacketDecoder ready β€” has_keys={self._decoder.has_keys}") if confirmed: diff --git a/meshcore_gui/config.py b/meshcore_gui/config.py index 733e70f..e10769b 100644 --- a/meshcore_gui/config.py +++ b/meshcore_gui/config.py @@ -25,7 +25,7 @@ from typing import Any, Dict, List # ============================================================================== -VERSION: str = "1.22.1" +VERSION: str = "1.22.3" # ============================================================================== @@ -402,7 +402,7 @@ MESSAGE_RETENTION_DAYS: int = 30 # Retention period for RX log entries (in days). # RX log entries older than this are automatically removed during cleanup. -RXLOG_RETENTION_DAYS: int = 7 +RXLOG_RETENTION_DAYS: int = 14 # Retention period for contacts (in days). # Contacts not seen for longer than this are removed from cache. diff --git a/meshcore_gui/core/models.py b/meshcore_gui/core/models.py index 27e0e3f..22b1c06 100644 --- a/meshcore_gui/core/models.py +++ b/meshcore_gui/core/models.py @@ -29,7 +29,7 @@ class Message: """A channel message or direct message (DM). Attributes: - time: Formatted timestamp (HH:MM:SS). + time: Formatted local time (HH:MM:SS). sender: Display name of the sender. text: Message body. channel: Channel index, or ``None`` for a DM. @@ -42,6 +42,11 @@ class Message: captured at receive time so the archive is self-contained. message_hash: Deterministic packet identifier (hex string). channel_name: Human-readable channel name (resolved at add time). + date: Formatted local date (YYYY-MM-DD). Added in 1.22.3; + empty for messages archived by older versions, in which + case :meth:`from_dict` derives it from ``timestamp_utc``. + Declared last so positional construction stays + backward-compatible. """ time: str @@ -56,6 +61,7 @@ class Message: path_names: List[str] = field(default_factory=list) message_hash: str = "" channel_name: str = "" + date: str = "" @staticmethod def from_dict(d: dict) -> "Message": @@ -80,15 +86,64 @@ class Message: path_names=d.get("path_names", []), message_hash=d.get("message_hash", ""), channel_name=d.get("channel_name", ""), + date=d.get("date") or Message._date_from_timestamp( + d.get("timestamp_utc") + ), ) - # -- Timestamp helper ------------------------------------------------ + # -- Timestamp helpers ----------------------------------------------- @staticmethod def now_timestamp() -> str: - """Current time formatted as ``HH:MM:SS``.""" + """Current local time formatted as ``HH:MM:SS``.""" return datetime.now().strftime('%H:%M:%S') + @staticmethod + def now_date() -> str: + """Current local date formatted as ``YYYY-MM-DD``.""" + return datetime.now().strftime('%Y-%m-%d') + + @staticmethod + def _date_from_timestamp(timestamp_utc: Optional[str]) -> str: + """Derive a local ``YYYY-MM-DD`` date from an ISO UTC timestamp. + + Backward-compatibility helper for archive entries written before + the ``date`` field existed. The archive stores ``timestamp_utc`` + for every message, so the date can be recovered; it is converted + to local time first so it matches the local ``time`` field around + midnight. + + Args: + timestamp_utc: ISO-8601 timestamp string, or ``None``. + + Returns: + Local date as ``YYYY-MM-DD``, or an empty string when the + timestamp is missing or unparseable. + """ + if not timestamp_utc: + return "" + try: + parsed = datetime.fromisoformat(timestamp_utc) + except (ValueError, TypeError): + return "" + if parsed.tzinfo is not None: + parsed = parsed.astimezone() + return parsed.strftime('%Y-%m-%d') + + def display_timestamp(self) -> str: + """Return the display timestamp for this message. + + Produces ``YYYY-MM-DD HH:MM:SS`` when the date is known, and + falls back to the bare ``HH:MM:SS`` time for legacy messages + that carry no date information. + + Returns: + Formatted timestamp string. + """ + if self.date: + return f"{self.date} {self.time}" + return self.time + # -- Factory methods ------------------------------------------------- @classmethod @@ -99,6 +154,7 @@ class Message: channel: Optional[int], *, time: str = "", + date: str = "", snr: Optional[float] = None, path_len: int = 0, sender_pubkey: str = "", @@ -112,7 +168,8 @@ class Message: sender: Display name of the sender. text: Message body. channel: Channel index, or ``None`` for a DM. - time: Optional pre-generated timestamp (default: now). + time: Optional pre-generated time (default: now). + date: Optional pre-generated date (default: today). snr: Signal-to-noise ratio (dB). path_len: Hop count from the LoRa frame header. sender_pubkey: Full public key of the sender (hex string). @@ -122,6 +179,7 @@ class Message: """ return cls( time=time or cls.now_timestamp(), + date=date or cls.now_date(), sender=sender, text=text, channel=channel, @@ -151,6 +209,7 @@ class Message: """ return cls( time=cls.now_timestamp(), + date=cls.now_date(), sender='Me', text=text, channel=channel, @@ -168,10 +227,11 @@ class Message: ) -> str: """Format as a single display line for the messages panel. - Produces the same output as the original ``messages_panel.py`` - inline formatting, e.g.:: + The line opens with :meth:`display_timestamp`, which prefixes the + local date when it is known (legacy archive entries without a + date fall back to time only), e.g.:: - 12:34:56 ← [Public] [2hβœ“] PE1ABC: Hello mesh! + 2026-08-16 12:34:56 ← [Public] [2hβœ“] PE1ABC: Hello mesh! When *show_channel* is ``False`` the ``[channel]`` / ``[DM]`` tag is omitted (useful when the panel header already indicates @@ -210,9 +270,11 @@ class Message: sender_display = f"{sender_prefix}{self.sender}" if self.sender else '' + stamp = self.display_timestamp() + if self.sender: - return f"{self.time} {direction} {ch_label}{hop_tag}{sender_display}: {self.text}" - return f"{self.time} {direction} {ch_label}{hop_tag}{self.text}" + return f"{stamp} {direction} {ch_label}{hop_tag}{sender_display}: {self.text}" + return f"{stamp} {direction} {ch_label}{hop_tag}{self.text}" # --------------------------------------------------------------------------- diff --git a/meshcore_gui/gui/panels/room_server_panel.py b/meshcore_gui/gui/panels/room_server_panel.py index ab326a7..0fe156a 100644 --- a/meshcore_gui/gui/panels/room_server_panel.py +++ b/meshcore_gui/gui/panels/room_server_panel.py @@ -495,7 +495,7 @@ class RoomServerPanel: for msg in display: direction = 'β†’' if msg.direction == 'out' else '←' sender = self._resolve_sender_name(msg.sender or '', contacts) - line = f"{msg.time} {direction} {sender}: {msg.text}" + line = f"{msg.display_timestamp()} {direction} {sender}: {msg.text}" ui.label(line).classes( 'text-xs leading-tight px-1' diff --git a/meshcore_gui/gui/route_page.py b/meshcore_gui/gui/route_page.py index d56374c..b0a82ea 100644 --- a/meshcore_gui/gui/route_page.py +++ b/meshcore_gui/gui/route_page.py @@ -172,7 +172,7 @@ class RoutePage: f'Message Route β€” {sender_icon} {sender} ({direction})' ).classes('font-bold text-lg') ui.label( - f"{msg.time} {sender_icon} {sender}: {msg.text[:120]}" + f"{msg.display_timestamp()} {sender_icon} {sender}: {msg.text[:120]}" ).classes('text-sm text-gray-600') @staticmethod @@ -453,7 +453,7 @@ class RoutePage: prefilled = ''.join(parts) ch_options = {ch['idx']: f"[{ch['idx']}] {ch['name']}" for ch in data['channels']} - default_ch = data['channels'][0]['idx'] if data['channels'] else 0 + default_ch = self._default_reply_channel(msg, ch_options) with ui.card().classes('w-full'): ui.label('πŸ“€ Reply').classes('font-bold text-gray-600') @@ -473,6 +473,30 @@ class RoutePage: ui.button('Send', on_click=send).classes('bg-blue-500 text-white') + @staticmethod + def _default_reply_channel(msg: Message, ch_options: Dict[int, str]) -> int: + """Determine the channel the reply select should default to. + + The reply belongs on the channel the message arrived on, so that + opening the route page from e.g. ``#noodnet-ov`` pre-selects that + channel instead of the first entry in the channel list. + + Falls back to the first available channel when the message is a + DM (``channel is None``) or when its channel index is no longer + present on the device β€” a slot can be vacated by the + post-discovery cache sync introduced in 1.22.2. + + Args: + msg: The message this route page was opened for. + ch_options: ``{channel_idx: label}`` options for the select. + + Returns: + Channel index to pre-select. + """ + if msg.channel is not None and msg.channel in ch_options: + return msg.channel + return next(iter(ch_options), 0) + @staticmethod def _find_sender_contact(msg: Message, contacts: Dict) -> Optional[tuple]: """Defensive fallback: find sender contact data in snapshot.""" diff --git a/meshcore_gui/services/channel_backup_store.py b/meshcore_gui/services/channel_backup_store.py index 62c800d..66229ec 100644 --- a/meshcore_gui/services/channel_backup_store.py +++ b/meshcore_gui/services/channel_backup_store.py @@ -202,8 +202,15 @@ class ChannelBackupStore: except (ValueError, TypeError): continue - # Union of all indices we know about - all_indices = sorted(set(name_by_idx) | set(psk_by_idx)) + # Active-only export: the dialog promises "every channel currently + # known to this GUI", i.e. the named (active) slots. Empty-name + # slots that still happen to carry a PSK in the cache (firmware- + # initialised slots, or stale entries from older runs) must NOT be + # exported, otherwise a backup of a 3-channel device would contain + # 100 entries and decoder pollution would persist across restores. + # PSKs are still looked up from cache, but only for slots that + # have a name in name_by_idx. + all_indices = sorted(name_by_idx) entries: List[ChannelBackupEntry] = [] for idx in all_indices: diff --git a/meshcore_gui/services/message_archive.py b/meshcore_gui/services/message_archive.py index 061f117..68801d7 100644 --- a/meshcore_gui/services/message_archive.py +++ b/meshcore_gui/services/message_archive.py @@ -127,6 +127,7 @@ class MessageArchive: # Convert to dict and add UTC timestamp msg_dict = { "time": msg.time, + "date": msg.date, "timestamp_utc": datetime.now(timezone.utc).isoformat(), "sender": msg.sender, "text": msg.text,