an0n.eu/meshcore-gui1
1
0
Fork 0
forked from iarv/meshcore-gui
Code Pull Requests Activity

feature: persistent storage for all incoming messages

Browse Source 
This feature implements persistent storage for all incoming messages, RX log entries, and contacts with configurable retention periods. The system uses a dual-layer architecture to balance real-time UI performance with comprehensive data retention.
This commit is contained in:
pe1hvh
2026-02-08 09:12:31 +01:00
parent ac4d10dece
commit 6ec90b57fd
6 changed files with 109 additions and 212 deletions
Download Patch File Download Diff File Expand all files Collapse all files

121
CHANGELOG.md
View File

@@ -1,17 +1,50 @@
# CHANGELOG: Message & Metadata Persistence
# CHANGELOG
## v1.0.4 (2026-02-07) - Archive Viewer Feature
<!-- CHANGED: Titel gewijzigd van "CHANGELOG: Message & Metadata Persistence" naar "CHANGELOG" —
een root-level CHANGELOG.md hoort project-breed te zijn, niet feature-specifiek. -->
All notable changes to MeshCore GUI are documented in this file.
Format follows [Keep a Changelog](https://keepachangelog.com/) and [Semantic Versioning](https://semver.org/).
---
<!-- ADDED: Nieuw v5.3.0 entry bovenaan -->
## [5.3.0] - 2026-02-08 — Documentation Review & Route Table Fix
### Fixed
- 🐛 **Route table names and IDs not displayed** — Route tables in both current messages (RoutePage) and archive messages (ArchivePage) now correctly show node names and public key IDs for sender, repeaters and receiver
### Changed
- 🔄 **CHANGELOG.md**: Corrected version numbering (v1.0.x → v5.x), fixed inaccurate references (archive button location, filter state persistence)
- 🔄 **README.md**: Added Message Archive feature, updated project structure, configuration table and architecture diagram
- 🔄 **MeshCore_GUI_Design.docx**: Added ArchivePage, MessageArchive, Models components; updated project structure, protocols, configuration and version history
---
## [5.2.0] - 2026-02-07 — Archive Viewer Feature
<!-- CHANGED: Versienummer gewijzigd van v1.0.4 naar v5.2.0 — past bij applicatieversie (v5.x).
De __main__.py vermeldt Version: 5.0, het Design Document is v5.2. -->
### Added
- ✅ **Archive Viewer Page** (`/archive`) - Full-featured message archive browser
- ✅ **Archive Viewer Page** (`/archive`) — Full-featured message archive browser
- Pagination (50 messages per page, configurable)
- Channel filter dropdown (All + configured channels)
- Time range filter (24h, 7d, 30d, 90d, All time)
- Text search (case-insensitive)
- Filter state persistence (app.storage.user)
- Filter state stored in instance variables (reset on page reload)
- Message cards with same styling as main messages panel
- Clickable messages for route visualization (where available)
- **💬 Reply functionality** - Expandable reply panel per message
- **💬 Reply functionality** — Expandable reply panel per message
- **🗺️ Inline route table** — Expandable route display per archive message with sender, repeaters and receiver (names, IDs, node types)
<!-- CHANGED: "Filter state persistence (app.storage.user)" vervangen door "Filter state stored in
instance variables" — de code (archive_page.py:36-40) gebruikt self._current_page etc.,
niet app.storage.user. Het commentaar in de code is misleidend. -->
<!-- ADDED: "Inline route table" entry — _render_archive_route() in archive_page.py:333-407
was niet gedocumenteerd. -->
- ✅ **MessageArchive.query_messages()** method
- Filter by: time range, channel, text search, sender
@@ -20,11 +53,14 @@
- Sorting: Newest first
- ✅ **UI Integration**
- "📚 View Archive" button in Actions panel
- Opens in new tab
- "📚 Archive" button in Messages panel header (opens in new tab)
- Back to Dashboard button in archive page
- ✅ **Reply Panel** (NEW!)
<!-- CHANGED: "📚 View Archive button in Actions panel" gecorrigeerd — de knop zit in
MessagesPanel (messages_panel.py:25), niet in ActionsPanel (actions_panel.py).
ActionsPanel bevat alleen Refresh en Advert knoppen. -->
- ✅ **Reply Panel**
- Expandable reply per message (💬 Reply button)
- Pre-filled with @sender mention
- Channel selector
@@ -33,23 +69,10 @@
### Changed
- 🔄 `SharedData.get_snapshot()`: Now includes `'archive'` field
- 🔄 `ActionsPanel`: Added archive button and open handler
- 🔄 `MessagesPanel`: Added archive button in header row
- 🔄 Both entry points (`__main__.py` and `meshcore_gui.py`): Register `/archive` route
### Features
- **Pagination**: Navigate large archives efficiently
- **Filters**: Time range + channel + text search
- **Persistent State**: Filters remembered across sessions
- **Consistent UI**: Same message styling as dashboard
- **Route Integration**: Click messages to view route (if in recent buffer)
- **Reply from Archive**: Direct reply capability for any archived message
### UI/UX
- **Message Cards**: Expandable reply panel integrated
- **Pre-filled Reply**: Auto-mention sender (@sender)
- **Channel Selection**: Choose reply channel
- **Feedback**: Success notification after sending
- **Smart Collapse**: Reply panel closes after send
<!-- CHANGED: "ActionsPanel: Added archive button" gecorrigeerd naar "MessagesPanel" -->
### Performance
- Query: ~10ms for 10k messages with filters
@@ -62,16 +85,11 @@
- Text search is linear scan (no indexing yet)
- Sender filter exists in API but not in UI yet
### Future Improvements
- Archive-based route visualization (use message_hash)
- Sender filter UI component
- Export to CSV/JSON
- Advanced filters (SNR, hop count)
- Full-text search indexing
---
## v1.0.3 (2026-02-07) - Critical Bugfix: Archive Overwrite Prevention
## [5.1.3] - 2026-02-07 — Critical Bugfix: Archive Overwrite Prevention
<!-- CHANGED: Versienummer gewijzigd van v1.0.3 naar v5.1.3 -->
### Fixed
- 🐛 **CRITICAL**: Fixed bug where archive was overwritten instead of appended on restart
@@ -103,7 +121,9 @@ overwriting all historical data with only the new buffered messages.
---
## v1.0.2 (2026-02-07) - RxLog message_hash Enhancement
## [5.1.2] - 2026-02-07 — RxLog message_hash Enhancement
<!-- CHANGED: Versienummer gewijzigd van v1.0.2 naar v5.1.2 -->
### Added
- ✅ `message_hash` field added to `RxLogEntry` model
@@ -120,36 +140,11 @@ overwriting all historical data with only the new buffered messages.
- **Analysis**: Track which packets resulted in messages
- **Debugging**: Better troubleshooting of packet processing
### Example RxLog Entry (Before)
```json
{
"time": "12:34:56",
"timestamp_utc": "2026-02-07T12:34:56Z",
"snr": 8.5,
"rssi": -95.0,
"payload_type": "MSG",
"hops": 2
}
```
### Example RxLog Entry (After)
```json
{
"time": "12:34:56",
"timestamp_utc": "2026-02-07T12:34:56Z",
"snr": 8.5,
"rssi": -95.0,
"payload_type": "MSG",
"hops": 2,
"message_hash": "def456..."
}
```
**Note:** For non-message packets (announcements, broadcasts), `message_hash` will be an empty string.
---
## v1.0.1 (2026-02-07) - Entry Point Fix
## [5.1.1] - 2026-02-07 — Entry Point Fix
<!-- CHANGED: Versienummer gewijzigd van v1.0.1 naar v5.1.1 -->
### Fixed
- ✅ `meshcore_gui.py` (root entry point) now passes ble_address to SharedData
@@ -160,7 +155,9 @@ overwriting all historical data with only the new buffered messages.
---
## v1.0.0 (2026-02-07) - Initial Release
## [5.1.0] - 2026-02-07 — Message & Metadata Persistence
<!-- CHANGED: Versienummer gewijzigd van v1.0.0 naar v5.1.0 -->
### Added
- ✅ MessageArchive class for persistent storage

58
README.md
View File

@@ -28,6 +28,8 @@ Under the hood it uses `bleak` for Bluetooth Low Energy (which talks to BlueZ on
- **Direct Messages** — Click on a contact to send a DM
- **Message Filtering** — Filter messages per channel via checkboxes
- **Message Route Visualization** — Click any message to open a detailed route page showing the path (hops) through the mesh network on an interactive map, with a hop summary, route table and reply panel
- **Message Archive** — All messages and RX log entries are persisted to disk with configurable retention. Browse archived messages via the archive viewer with filters (channel, time range, text search), pagination and inline route tables
<!-- ADDED: Message Archive feature was missing from features list -->
- **Keyword Bot** — Built-in auto-reply bot that responds to configurable keywords on selected channels, with cooldown and loop prevention
- **Packet Decoding** — Raw LoRa packets from RX log are decoded and decrypted using channel keys, providing message hashes, path hashes and hop data
- **Message Deduplication** — Dual-strategy dedup (hash-based and content-based) prevents duplicate messages from appearing
@@ -187,6 +189,10 @@ The GUI opens automatically in your browser at `http://localhost:8080`
| `DEBUG` | `meshcore_gui/config.py` | Set to `True` for verbose logging (or use `--debug-on`) |
| `CHANNELS_CONFIG` | `meshcore_gui/config.py` | List of channels (hardcoded due to BLE timing issues) |
| `CONTACT_REFRESH_SECONDS` | `meshcore_gui/config.py` | Interval between periodic contact refreshes (default: 300s / 5 minutes) |
| `MESSAGE_RETENTION_DAYS` | `meshcore_gui/config.py` | Retention period for archived messages (default: 30 days) |
| `RXLOG_RETENTION_DAYS` | `meshcore_gui/config.py` | Retention period for archived RX log entries (default: 7 days) |
| `CONTACT_RETENTION_DAYS` | `meshcore_gui/config.py` | Retention period for cached contacts (default: 90 days) |
<!-- ADDED: Three retention settings above were missing from config table -->
| `KEY_RETRY_INTERVAL` | `meshcore_gui/ble/worker.py` | Interval between background retry attempts for missing channel keys (default: 30s) |
| `BOT_CHANNELS` | `meshcore_gui/services/bot.py` | Channel indices the bot listens on |
| `BOT_NAME` | `meshcore_gui/services/bot.py` | Display name prepended to bot replies |
@@ -232,6 +238,22 @@ Route data is resolved from two sources (in priority order):
1. **RX log packet decode** — Path hashes extracted from the raw LoRa packet via `meshcoredecoder`
2. **Contact out_path** — Stored route from the sender's contact record (fallback)
<!-- ADDED: Message Archive section was missing from Functionality -->
### Message Archive
All incoming messages and RX log entries are automatically persisted to disk in `~/.meshcore-gui/archive/`. One JSON file per data type per BLE device address.
Click the **📚 Archive** button in the Messages panel header to open the archive viewer in a new tab. The archive viewer provides:
- **Pagination** — 50 messages per page, with Previous/Next navigation
- **Channel filter** — Filter by specific channel or view all
- **Time range filter** — Last 24 hours, 7 days, 30 days, 90 days, or all time
- **Text search** — Case-insensitive search in message text
- **Inline route tables** — Expandable route display per message (sender, repeaters, receiver with names and IDs)
- **Reply from archive** — Expandable reply panel per message with pre-filled @sender mention
Old data is automatically cleaned up based on configurable retention periods (`MESSAGE_RETENTION_DAYS`, `RXLOG_RETENTION_DAYS` in `config.py`).
### Local Cache
Device info, contacts and channel keys are automatically cached to disk in `~/.meshcore-gui/cache/`. One JSON file is created per BLE device address.
@@ -285,6 +307,8 @@ The built-in bot automatically replies to messages containing recognised keyword
## Architecture
<!-- CHANGED: Architecture diagram updated — added MessageArchive component -->
```
┌─────────────────┐ ┌─────────────────┐
│ Main Thread │ │ BLE Thread │
@@ -303,15 +327,22 @@ The built-in bot automatically replies to messages containing recognised keyword
│ ┌─────┴─────┐ │ │ │ ┌────┴────┐ │
│ │ Panels │ │ │ │ │ Bot │ │
│ │ RoutePage│ │ │ │ │ Dedup │ │
│ └───────────┘ │ │ │ │ Cache │ │
└─────────────────┘ │ │ └─────────┘ │
│ └─────────────────┘
│ │ ArchivePg │ │ │ │ │ Cache │ │
│ └───────────┘ │ │ │ └─────────┘ │
└─────────────────┘ │ └─────────────────┘
┌──────┴──────┐
│ SharedData │ ┌───────────────┐
│ (thread- │ │ DeviceCache │
│ safe) │ │ (~/.meshcore- │
└─────────────┘ │ gui/cache/) │
└───────────────┘
└──────┬──────┘ │ gui/cache/) │
│ └───────────────┘
┌──────┴──────┐
│ Message │
│ Archive │
│ (~/.meshcore│
│ -gui/ │
│ archive/) │
└─────────────┘
```
- **BLEWorker**: Runs in separate thread with its own asyncio loop, with background retry for missing channel keys
@@ -321,9 +352,13 @@ The built-in bot automatically replies to messages containing recognised keyword
- **MeshBot**: Keyword-triggered auto-reply on configured channels
- **DualDeduplicator**: Prevents duplicate messages (hash-based + content-based)
- **DeviceCache**: Local JSON cache per device for instant startup and offline resilience
- **MessageArchive**: Persistent storage for messages and RX log with configurable retention and automatic cleanup
<!-- ADDED: MessageArchive component description -->
- **SharedData**: Thread-safe data sharing between BLE and GUI via Protocol interfaces
- **DashboardPage**: Main GUI with modular panels (device, contacts, map, messages, etc.)
- **RoutePage**: Standalone route visualization page opened per message
- **ArchivePage**: Archive viewer with filters, pagination and inline route tables
<!-- ADDED: ArchivePage component description -->
- **Communication**: Via command queue (GUI→BLE) and shared state with flags (BLE→GUI)
## Known Limitations
@@ -331,6 +366,8 @@ The built-in bot automatically replies to messages containing recognised keyword
1. **Channels hardcoded** — The `get_channel()` function in meshcore-py is unreliable via BLE (mitigated by background retry and disk caching of channel keys)
2. **BLE command unreliability** — `send_appstart()`, `send_device_query()` and `get_channel()` can all fail intermittently. The application uses aggressive retries (10 attempts for device info, background retry every 30s for channel keys) and disk caching to compensate
3. **Initial load time** — GUI waits for BLE data before the first render is complete (mitigated by cache: if cached data exists, the GUI populates instantly)
4. **Archive route visualization** — Route data for archived messages depends on contacts currently in memory; archived-only messages without recent contact data may show incomplete routes
<!-- ADDED: Archive-related limitation -->
## Troubleshooting
@@ -408,13 +445,15 @@ Or set `DEBUG = True` in `meshcore_gui/config.py`.
### Project structure
<!-- CHANGED: Project structure updated — added archive_page.py and message_archive.py -->
```
meshcore-gui/
├── meshcore_gui.py # Entry point
├── meshcore_gui/ # Application package
│ ├── __init__.py
│ ├── __main__.py # Alternative entry: python -m meshcore_gui
│ ├── config.py # DEBUG flag, channel configuration, refresh interval
│ ├── config.py # DEBUG flag, channel configuration, refresh interval, retention settings
│ ├── ble/ # BLE communication layer
│ │ ├── __init__.py
│ │ ├── worker.py # BLE thread, connection lifecycle, cache-first startup, background key retry
@@ -423,7 +462,7 @@ meshcore-gui/
│ │ └── packet_decoder.py # Raw LoRa packet decoding via meshcoredecoder
│ ├── core/ # Domain models and shared state
│ │ ├── __init__.py
│ │ ├── models.py # Dataclasses: Message, Contact, RouteNode, etc.
│ │ ├── models.py # Dataclasses: Message, Contact, DeviceInfo, RxLogEntry, RouteNode
│ │ ├── shared_data.py # Thread-safe shared data store
│ │ └── protocols.py # Protocol interfaces (ISP/DIP)
│ ├── gui/ # NiceGUI web interface
@@ -431,6 +470,7 @@ meshcore-gui/
│ │ ├── constants.py # UI display constants
│ │ ├── dashboard.py # Main dashboard page orchestrator
│ │ ├── route_page.py # Message route visualization page
│ │ ├── archive_page.py # Message archive viewer with filters and pagination
│ │ └── panels/ # Modular UI panels
│ │ ├── __init__.py
│ │ ├── device_panel.py # Device info display
@@ -438,7 +478,7 @@ meshcore-gui/
│ │ ├── map_panel.py # Leaflet map
│ │ ├── input_panel.py # Message input and channel select
│ │ ├── filter_panel.py # Channel filters and bot toggle
│ │ ├── messages_panel.py # Filtered message display
│ │ ├── messages_panel.py # Filtered message display with archive button
│ │ ├── actions_panel.py # Refresh and advert buttons
│ │ └── rxlog_panel.py # RX log table
│ └── services/ # Business logic
@@ -446,6 +486,7 @@ meshcore-gui/
│ ├── bot.py # Keyword-triggered auto-reply bot
│ ├── cache.py # Local JSON cache per BLE device
│ ├── dedup.py # Message deduplication
│ ├── message_archive.py # Persistent message and RX log archive
│ └── route_builder.py # Route data construction
├── docs/
│ ├── TROUBLESHOOTING.md # BLE troubleshooting guide (Linux)
@@ -455,6 +496,7 @@ meshcore-gui/
├── .gitattributes
├── .gitignore
├── LICENSE
├── CHANGELOG.md
└── README.md
```

142
RELEASE.md
View File

@@ -1,142 +0,0 @@
# Release Notes — MeshCore GUI
**Date:** 4 February 2026
---
## Summary
This release replaces the single-file monolith (`meshcore_gui.py`, 1,395 lines, 3 classes, 51 methods) with a modular package of 16 files (1,955 lines, 10 classes, 90 methods). The refactoring introduces a `meshcore_gui/` package with Protocol-based dependency inversion, a `widgets/` subpackage with six independent UI components, a message route visualisation page, and full type coverage.
---
## Starting point
The repository contained one file with everything in it:
**`meshcore_gui.py`** — 1,395 lines, 3 classes, 51 methods
| Section | Lines | Methods | Responsibility |
|---------|-------|---------|----------------|
| Config + `debug_print` | 80 | 1 | Constants, debug helper |
| `SharedData` | 225 | 12 | Thread-safe data store |
| `BLEWorker` | 268 | 11 | BLE communication thread |
| `MeshCoreGUI` | 740 | 24 | All GUI: rendering, data updates, user actions |
| Main entry | 74 | 3 | Page handler, `main()` |
All three classes lived in one file. BLEWorker and MeshCoreGUI both depended directly on the concrete SharedData class. MeshCoreGUI handled everything: 8 render methods, 7 data-update methods, 5 user-action methods, the 500ms update timer, and the DM dialog.
---
## Current state
16 files across a package with a `widgets/` subpackage:
| File | Lines | Class | Depends on |
|------|-------|-------|------------|
| `meshcore_gui.py` | 101 | *(entry point)* | concrete SharedData (composition root) |
| `meshcore_gui/__init__.py` | 8 | — | — |
| `meshcore_gui/config.py` | 54 | — | — |
| `meshcore_gui/protocols.py` | 83 | 4 Protocol classes | — |
| `meshcore_gui/shared_data.py` | 263 | SharedData | config |
| `meshcore_gui/ble_worker.py` | 252 | BLEWorker | SharedDataWriter protocol |
| `meshcore_gui/main_page.py` | 148 | DashboardPage | SharedDataReader protocol |
| `meshcore_gui/route_builder.py` | 174 | RouteBuilder | ContactLookup protocol |
| `meshcore_gui/route_page.py` | 258 | RoutePage | SharedDataReadAndLookup protocol |
| `meshcore_gui/widgets/__init__.py` | 22 | — | — |
| `meshcore_gui/widgets/device_panel.py` | 100 | DevicePanel | config |
| `meshcore_gui/widgets/map_panel.py` | 80 | MapPanel | — |
| `meshcore_gui/widgets/contacts_panel.py` | 114 | ContactsPanel | config |
| `meshcore_gui/widgets/message_input.py` | 83 | MessageInput | — |
| `meshcore_gui/widgets/message_list.py` | 156 | MessageList | — |
| `meshcore_gui/widgets/rx_log_panel.py` | 59 | RxLogPanel | — |
| **Total** | **1,955** | **10 classes** | |
---
## What changed
### 1. Monolith → package
The single file was split into a `meshcore_gui/` package. Each class got its own module. Constants and `debug_print` moved to `config.py`. The original `meshcore_gui.py` became a thin entry point (101 lines) that wires components and starts the server.
### 2. Protocol-based dependency inversion
Four `typing.Protocol` interfaces were introduced in `protocols.py`:
| Protocol | Consumer | Methods |
|----------|----------|---------|
| SharedDataWriter | BLEWorker | 10 |
| SharedDataReader | DashboardPage | 4 |
| ContactLookup | RouteBuilder | 1 |
| SharedDataReadAndLookup | RoutePage | 5 |
No consumer imports `shared_data.py` directly. Only the entry point knows the concrete class.
### 3. MeshCoreGUI decomposed into DashboardPage + 6 widgets
The 740-line MeshCoreGUI class was split:
| Old (MeshCoreGUI) | New | Lines |
|--------------------|-----|-------|
| 8 `_render_*` methods | 6 widget classes in `widgets/` | 592 total |
| 7 `_update_*` methods | Widget `update()` methods | *(inside widgets)* |
| 5 user-action methods | Widget `on_command` callbacks | *(inside widgets)* |
| `render()` + `_update_ui()` | DashboardPage (orchestrator) | 148 |
DashboardPage now has 4 methods. It composes widgets and drives the timer. Widgets have zero knowledge of SharedData — they receive plain `Dict` snapshots and callbacks.
### 4. Route visualisation (new feature)
Two new modules that did not exist in the monolith:
| Module | Lines | Purpose |
|--------|-------|---------|
| `route_builder.py` | 174 | Constructs route data from message metadata (pure logic) |
| `route_page.py` | 258 | Renders route on a Leaflet map in a separate browser tab |
Clicking a message in the message list opens `/route/{msg_index}` showing sender → repeater hops → receiver on a map.
### 5. SharedData extended
SharedData gained 4 new methods to support the protocol interfaces and route feature:
| New method | Purpose |
|------------|---------|
| `set_connected()` | Explicit setter (was direct attribute access) |
| `put_command()` | Queue command from GUI (was `cmd_queue.put()` directly) |
| `get_next_command()` | Dequeue command for BLE worker (was `cmd_queue.get_nowait()` directly) |
| `get_contact_by_prefix()` | Contact lookup for route building |
| `get_contact_name_by_prefix()` | Contact name lookup for DM display |
The direct `self.shared.lock` and `self.shared.cmd_queue` access from BLEWorker and MeshCoreGUI was replaced with proper method calls through protocol interfaces.
### 6. Full type coverage
All 90 methods now have complete type annotations (parameters and return types). The old monolith had 51 methods with partial coverage.
---
## Metrics
| Metric | Old | Current |
|--------|-----|---------|
| Files | 1 | 16 |
| Lines | 1,395 | 1,955 |
| Classes | 3 | 10 |
| Methods | 51 | 90 |
| Largest class (lines) | MeshCoreGUI (740) | SharedData (263) |
| Protocol interfaces | 0 | 4 |
| Type-annotated methods | partial | 90/90 |
| Widget classes | 0 | 6 |
---
## Documentation
| Document | Status |
|----------|--------|
| `README.md` | Updated: architecture diagram, project structure, features |
| `docs/MeshCore_GUI_Design.docx` | Updated: widget tables, component descriptions, version history |
| `docs/SOLID_ANALYSIS.md` | Updated: widget SRP, dependency tree, metrics |
| `docs/RELEASE.md` | New (this document) |

0
FEATURE_MESSAGE_PERSISTENCE.md → docs/FEATURE_MESSAGE_PERSISTENCE.md
View File

BIN
docs/MeshCore_GUI_Design.docx
View File

Binary file not shown.

BIN
meshcore_gui_bugfix.zip
View File

Binary file not shown.