Problem
-------
update_packet_metrics() called rrdtool.info() (cached for 5 s) to get the
RRD's last_update timestamp. rrdtool.info() returns a massive Python dict:
17 data sources × 5 RRAs × ~8 fields each = ~700+ dict entries per call.
tracemalloc showed +10696 new allocations / +251 KB at this exact line,
flagged as "Investigate" in the memory diagnostics dashboard.
The rrdtool.info() approach was also unnecessarily complex: it required a
5-second secondary cache, a _pending_rrd_update buffer, and two extra
instance attributes — all to answer one question ("did we already write
this period?") that we can answer ourselves with a single integer.
Fix
---
Replace _last_rrd_info_cache / _last_rrd_info_time / _pending_rrd_update
with a single self._last_rrd_update: int = 0 that stores the timestamp of
the last successful rrdtool.update() call. The throttle check becomes:
if timestamp <= self._last_rrd_update:
return
On success: self._last_rrd_update = timestamp
Zero dict allocations per call. The only downside vs rrdtool.info() is
that _last_rrd_update resets to 0 on process restart, meaning the first
packet after a restart always triggers a write — correct behaviour.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Problem
-------
rrdtool.fetch() is a blocking C library call that reads 24 hours of RRD
data from disk. The dashboard can call get_data() on every page refresh.
On an SD card each fetch can cost several milliseconds of I/O, and because
the RRD step is 60 seconds the data cannot change more often than that —
any fetch within the same 60-second window returns identical data.
The combined-optimizations branch had a 60-second read cache; rightup's
batching refactor inadvertently removed it. This PR restores it.
Solution
--------
* Add self._get_data_cache: tuple = (0.0, None) to __init__
* In get_data(): set use_cache = (start_time is None and end_time is None)
- if use_cache and cache is < 60 s old: return cached result immediately
- after a successful live fetch with use_cache: store (now, result)
* Explicit start_time / end_time callers always bypass the cache so
fine-grained or historical queries are never stale
Why 60 s TTL?
The RRD step is 60 s, so the database cannot hold a newer sample until
the next step boundary. A 60-second cache is tight enough that the
dashboard always shows data ≤ one step stale, and loose enough that
a burst of refreshes costs one disk read instead of N.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Five targeted changes to sqlite_handler.py, all in the same file.
1. Thread-local persistent connections
_connect() previously opened a new sqlite3.connect() on every DB call and
ran journal_mode + busy_timeout PRAGMAs each time. On SD-card storage each
connection open involves file-system operations; each PRAGMA is a round-trip.
threading.local() now caches one connection per thread (write executor thread
+ event-loop/HTTP threads), eliminating per-call setup overhead.
2. PRAGMA synchronous=NORMAL
Default synchronous=FULL flushes WAL frames to disk after every transaction.
NORMAL flushes only at WAL checkpoints — safe for this workload (no data loss
beyond the current transaction on power failure) and significantly faster on
SD cards, which have slow fsync (5-20ms per flush).
3. Migration 8: UNIQUE index on companion_messages(companion_hash, packet_hash)
companion_push_message previously deduped via SELECT + INSERT (two statements,
two SD-card reads per message). The new UNIQUE index enables INSERT OR IGNORE,
replacing the round-trip with a single atomic statement.
4. Migration 9: UNIQUE index on adverts(pubkey)
Without this index store_advert's ON CONFLICT clause cannot fire and each
advert inserts a new row instead of updating the existing one — unbounded
table growth on busy meshes. The migration deduplicates existing rows
(keeping the most-recently-seen per pubkey) before adding the index.
5. Remove duplicate get_unsynced_count definition
The method was defined twice with the same signature. Python silently uses
the last definition; the first was dead code with reversed SQL parameter
binding order. Removed the first; added a note to the surviving definition.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Added `companion_import_repeater_contacts` method in `SQLiteHandler` to import repeater adverts into a companion's contact store, with options for filtering by contact types, last seen hours, and import limits.
- Introduced `_get_sqlite_handler` method in `CompanionAPIEndpoints` to ensure the SQLite handler is available for contact import operations.
- Created `import_repeater_contacts` endpoint to handle POST requests for importing contacts, validating input parameters, and returning the count of successfully imported contacts.
- Updated the frontend to reflect changes in the contact import process, ensuring a seamless user experience.
- Introduced a new utility function, _to_json_safe, to ensure companion preferences are JSON-serializable, handling various data types including enums and dataclasses.
- Updated the RepeaterCompanionBridge to use the new serialization method when saving preferences to SQLite.
- Modified SQLiteHandler to ensure companion_hash is consistently converted to a string before database operations.
- Enhanced error handling for preference persistence in the RepeaterCompanionBridge.
- Added new methods to SQLiteHandler for loading and saving companion preferences as JSON, improving data persistence.
- Introduced a migration to create a companion_prefs table for storing preferences, ensuring compatibility with existing data.
- Refactored main.py to utilize RepeaterCompanionBridge instead of CompanionBridge, aligning with the new architecture.
- Create a new table for storing CRC errors in SQLite.
- Implement methods to store and retrieve CRC error counts and history.
- Update StorageCollector to record CRC errors and expose relevant methods.
- Enhance RepeaterHandler to track and record CRC error deltas from the radio hardware.
- Add API endpoints to fetch CRC error count and history.
- Updated the SQLiteHandler to apply a migration that prefixes companion_hash values with '0x' for consistency with room_hash patterns.
- Adjusted the main.py file to reflect the new formatting for companion_hash when generating its string representation.
- Enhanced readability by adding descriptive suffixes to packet type labels in both rrdtool_handler.py and sqlite_handler.py.
- Aligned packet type definitions with the new naming conventions from pyMC_core, ensuring consistency across the codebase.
- Updated PacketRouter to deliver packets to all companion bridges when the destination is not recognized, ensuring better handling of ephemeral destinations.
- Refactored CompanionFrameServer to separate contact serialization and persistence logic, allowing for non-blocking database operations.
- Introduced a unique index for companion contacts in SQLite to support upsert functionality, enhancing data integrity and performance.
- Improved AdvertHelper to run database operations in a separate thread, preventing event loop blocking and maintaining responsiveness.
- Updated `_persist_companion_message` to clarify deduplication of messages in SQLite.
- Modified `on_message_received` and `on_channel_message_received` to include `packet_hash` for message identification.
- Enhanced `SQLiteHandler` to support deduplication by `packet_hash` when pushing messages, preventing duplicates in the database.
- Added `packet_hash` column to the `companion_messages` table and created an index for efficient lookups.
Rigear