A bridge sends one logical packet through both radios, and the two do not fare
alike: 500 kHz and 62.5 kHz are different channels with different neighbours and
different contention. The packet row holds one set of LBT figures, from the
primary egress, so the other radio's attempts and backoffs were dropped on the
floor -- and an egress that failed was never stored at all, leaving the half of
the story an operator most wants to see only in the log.
A new packet_egress table keeps one row per physical send: packet id, timestamp,
radio, whether it went out, and its LBT attempts, total backoff and channel-busy
flag. Written only on a node with two or more radios; with one radio the packet
row already says all of this, and nothing about single-radio behaviour changes.
The rows are written on the storage writer thread beside the packet, addressed
by the id the insert returns, and are kept off packet_record itself so they do
not ride along on every websocket, Glass and MQTT publish. Retention ages them
out on the packet cutoff, and purging packets takes them along.
/lbt_diagnostics gains radios[]: the same summary and bucket shape per radio,
read from the new table over a covering index, so one radio's contention is
never read as the other's. The combined figures above it are unchanged and still
count each packet once. Per-radio entries carry no rf block -- the RRD series
behind it is node-wide, and hanging it off one radio would read as that radio's
own RSSI and loss -- and no packet-type breakdown, which would cost a heap
lookup per egress.
The fan-out now reads a send's LBT metadata whether or not it succeeded. The
single-egress path has always done so, and the figures on a failed send are
exactly the ones that explain it -- as far as they reach us: a radio that raises
instead of returning metadata (an SX1262 giving up on a busy channel) reports
nothing to read, so that row records the failure with no attempts rather than
inventing them.
Redundancy copies are recorded too. A multi-ack burst is real airtime on both
radios, and leaving it out would undercount exactly the traffic multi-ack adds
to a channel already busy enough to need it.
/db_stats lists packet_egress only once something has been written to it, so a
single-radio node reports the tables it always reported and never pays for
counting rows it does not have.
The bucket and summary maths moved out of get_lbt_diagnostics into three module
functions so the per-radio series is the same code as the combined one rather
than a second copy of 150 lines. The combined answer is unchanged.
A bridge has one receiver per band, and until now one of them stood for the
node: the background timer read the default radio and stored the answer with no
radio attached. On the 910.100/910.525 pair that reported the wide radio's noise
floor as the whole node's, and charged its CRC counter with both radios' errors.
Every physical radio is now sampled, one after another, and stored against its
own id. Migration 19 adds a nullable radio_id to noise_floor and crc_errors with
a covering index each, so a per-radio history reads only its own rows. Rows
written before this stay NULL: they were never measured per radio, and
labelling them would claim an attribution nobody made.
CRC baselines are per radio. A shared one charges every radio's errors to
whichever was sampled last, which reports swings that never happened.
Reads take an optional radio_id: /noise_floor_history, /noise_floor_stats,
/crc_error_count and /crc_error_history. Naming a radio also makes offset paging
stable, which the RF Health page depends on -- paging the merged series shifts a
page's contents every time the other radio samples. History rows name their
radio only on a node with two or more radios, so a single-radio response is
byte-for-byte what it was.
The default radio is sampled first and is the only one published. /stats, the
MQTT status message and companion stats read its cached value, and publishing
per radio doubles the observer-feed cadence -- its own change, not this one.
The StorageCollector facade carries the new arguments too. The web endpoints
reach the database through it, so the tests that drive them autospec it: a bare
mock accepts any keyword and would have let a facade that had not kept up
answer happily in tests and raise TypeError on a real node.
Adding the radio filter and per-radio split by concatenating SQL fragments tripped bandit B608, failing the pre-commit check. The fragments were constants and every value was bound, but the queries are now three fixed strings: a NULL radio parameter disables the rx_radio_id filter, and the per-radio split selects its own query. Results are unchanged.
On a Fabric node the analytics endpoints merged every radio together, with
one transmission per logical packet and nothing to tell a reception on one
radio from the other. Every packet row already records its ingress radio
and each radio that sent it, so the split needs no new data.
The attribution rules get_airtime_buckets introduced move into shared
helpers: packet_carriers() says which stored radio ids a row occupied, and
RadioResolver maps them onto the configured radios (one radio owns every
row; with two, an unknown id is unattributed, never guessed).
get_airtime_buckets now uses them with no change in output.
With two or more radio profiles:
- /packet_stats adds radios[] (received, duplicates, dropped, physical
transmissions, average RSSI/SNR) and unattributed counts. The legacy
totals come from the same grouped pass, so the 24 h aggregate the
WebSocket broadcasts does not gain a second scan. Totals still count
one transmission per packet.
- /route_stats adds the route mix of each radio's receptions, plus
originated for packets no radio received.
- /radio_packet_rates (new) returns receptions and transmissions per
radio per bucket, grouped in SQL on columns idx_packets_airtime covers.
A week-long window is an index-only scan instead of a Python loop over
every row: 0.17 s against 1.11 s for the airtime loop on a synthetic
1.55M-row table.
- /neighbor_links gives each link radios[], its stats as each radio heard
it. The tracker keeps them beside the merged figures without changing
its keys or entry count.
- /neighbor_link_history rows carry rx_radio_id, and the endpoint takes a
radio_id filter and a by_radio bucket split.
A single-radio node gets exactly the answers it had: the legacy queries
still run for it and every new field is omitted.
get_airtime_buckets selects rx_radio_id, tx_radio_id and tx_radio_ids to
charge airtime per radio, but idx_packets_airtime only covers timestamp,
length, payload_length and transmitted. SQLite therefore fell back to
idx_packets_timestamp with a heap lookup for every row in the window:
the access pattern the covering index was added to remove, and the one
that pushes the chart past the client timeout on an SD card. The test
meant to guard this asserted the plain timestamp index.
Migration 18 drops idx_packets_airtime and rebuilds it with the three
radio id columns. CREATE INDEX IF NOT EXISTS never reshapes an existing
index, so the base schema cannot do this. It keeps creating the original
four-column shape, which is safe on a database that predates the radio
id columns, and the migration widens it once migrations 16 and 17 have
added them. The rebuild reads the packets table once, at startup.
The bucket query is now a module constant shared with the test, which
asserts COVERING INDEX idx_packets_airtime for both airtime queries, so
a column added to the query cannot slip out of the index unnoticed.
A two-radio Fabric bridge runs each side at its own SF, so one combined
utilization line computed from a single caller-supplied profile describes
neither radio. Report airtime per radio instead.
build_radio_stack now records the effective air settings of every radio it
builds, in configured order, as radio_stack_meta["radio_profiles"]. The
same builder is exposed as build_radio_profiles() so /stats and the airtime
endpoint can read current settings from the live config rather than a boot
snapshot. Air settings are read the way get_radio_for_board reads them, and
a field that cannot be read is reported as null rather than defaulted, so
nothing is charted from settings that were never on the air.
get_airtime_buckets takes those profiles and charges each stored packet to
the radios that actually carried it: its ingress rx_radio_id and every
successful egress in tx_radio_ids, falling back to the scalar tx_radio_id
for pre-fan-out rows. A relayed packet therefore appears as RX on the radio
that heard it and TX on each radio that sent it, each at that radio's own
time-on-air. Duplicate receptions are included; they occupied the receiver.
With one profile every packet belongs to that radio, so single-radio and
pre-Fabric databases are attributed exactly as before. With two or more, a
packet whose radio cannot be identified is counted as unattributed rather
than guessed onto the default radio.
The response is additive: legacy buckets/rx_total/tx_total remain as the
combined series, and radios[] carries one series per radio alongside
unattributed_rx_count/unattributed_tx_count. Airtime comes from the shared
core estimator instead of a second inline Semtech formula, memoised per
(radio, length). The sf/bw_hz/cr/preamble query parameters are kept only as
a fallback; server profiles are authoritative and are never applied across
both radios of a bridge.
Add two Fabric options for an exactly-two-radio node:
- fabric.repeat_on_ingress (tx_mode: bridge): a relayed packet is sent
on the other radio first, then back out on the radio that heard it.
- fabric.local_tx_mode: all: locally originated packets (companions,
adverts, room/protocol replies) go out once on every radio, default
radio first.
MeshCore processing (policy, dedupe, path mutation) still runs once per
logical packet; only the physical TX is duplicated, each egress with its
own Packet, serialised through the TX lock and gated independently for
duty cycle and link health. One successful egress is a successful
forward. Multi-ack extras share the egress set.
inject_packet registers its single ACK waiter before the first TX so an
ACK heard mid-fan-out outlives the dispatcher's ACK cache. Fan-out fixes
flood scope and path-hash mode once before copying and metering, so all
radios carry identical bytes and the duty-cycle gates see them.
Packet records gain tx_radio_ids (JSON column + migration); tx_radio_id
remains the primary successful egress. Invalid option combinations are
rejected before hardware is opened. Defaults preserve existing behaviour.
`pre-commit run --all-files` fails on `dev` at the `ruff-format` hook, so every
open pull request is red on a check that has nothing to do with its own diff.
Both files arrived with 72da7ee (pr-436) formatted by an earlier ruff.
Purely cosmetic, and exactly what `ruff format` produces:
- `sqlite_handler.py` — a generator expression and an f-string concatenation
that both now fit inside the line limit unsplit.
- `test_companion_import_repeater_contacts.py` — the blank line ruff requires
after a module docstring.
"Import repeater contacts" wrote nothing whenever a contact-type filter was
supplied, and reported success having imported zero rows.
`adverts.contact_type` holds the *display* name written through
`handler_helpers.discovery.NODE_TYPE_NAMES` — "Chat Node", "Repeater",
"Room Server", "Sensor". The import API accepts MeshCore's names — "companion",
"repeater", "room_server", "sensor", validated in
`companion_endpoints.import_repeater_contacts`. `companion_import_repeater_contacts`
compared them directly:
query += f" AND contact_type IN ({placeholders})"
so "repeater" never matched "Repeater", "room_server" never matched
"Room Server", and "companion" never matched "Chat Node" at all. On a live
repeater with 285 adverts, every one of the four permitted values selected 0
rows, as did all four together; only an unfiltered import worked.
A second defect had the same cause. The adv_type lookup normalised the stored
name (`lower()`, spaces to underscores) but mapped it with the API-name table,
where "chat_node" is absent — so chat contacts imported as adv_type 0 rather
than 1, even on an unfiltered import. That is silently wrong rather than empty,
and needs a manual `UPDATE companion_contacts SET adv_type=1 WHERE adv_type=0`
to repair.
Both sites derived the mapping independently, which is how they drifted. They
now share one `_ADVERT_TYPE_BY_STORED` table keyed on the normalised stored
form: the filter reverse-maps the requested API names onto the stored keys and
compares `LOWER(REPLACE(TRIM(contact_type), ' ', '_'))`, and the adv_type comes
from the same table. Filtering and limiting stay in SQL. An unrecognised type
still selects nothing rather than silently widening the query.
Tested against the real stored forms: 6 of the 8 new tests fail on current dev
(each of the four type filters, all four combined, and the adv_type mapping)
and all 8 pass with this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`SQLiteHandler.__init__` runs `_init_database()` and then `_run_migrations()`.
`_init_database` created `idx_packets_upstream_time`, which spans
`upstream_hash` and `upstream_hash_size` -- columns that, on a database
predating migration 13, only appear once `add_upstream_hash_to_packets` has
added them in the *later* migration step.
Fresh installs were fine, because the `CREATE TABLE` carries both columns. Any
existing database raised `no such column: upstream_hash` part-way through
`_init_database`, which caught and logged it. Every statement after the failing
index was therefore skipped -- including the `room_messages` and
`room_client_sync` tables -- and the daemon started with a partially-created
schema while reporting itself healthy: service active, radio initialised,
companions serving, HTTP up, and no packets recorded.
The `add_upstream_hash_to_packets` migration already creates this index, right
after adding the columns it depends on, so the copy in `_init_database` was
redundant as well as too early. Remove it and leave the index to the migration
that owns those columns. All three paths still end up with it: a fresh database
gets the columns from `CREATE TABLE` and the index when the migration runs, an
un-migrated database gets both from the migration, and an already-migrated one
kept the index from its earlier run.
Observed on a Raspberry Pi 4 upgrading a ~13 MB production database from 1.1.1;
the repeater was off-air for four minutes before the cause was found. It
self-heals on a second start, because the migration still runs after the failed
init, which makes it easy to misread as harmless.
`tests/test_sqlite_upgrade_existing_db.py` builds a database with the
pre-migration-13 schema and asserts `_init_database` runs to completion, keyed
on tables created after the old failure point. Asserting only on the upstream
columns is not sufficient -- the migration adds those either way.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Added tests for multi-radio stack functionality, including merging radio entries and building radio stacks.
- Implemented policy context tests to ensure correct handling of radio IDs in multi-radio scenarios.
Scope answers were built into the MQTT payload and then discarded, so the web UI
had nothing to show between cycles and no way to ask a single repeater. Adds a
store for them, a read endpoint, and a single-target query.
Migration 15 adds neighbor_scopes: one row per queried neighbour holding the last
answer (`scopes`, `responded_at`) alongside the last query's outcome
(`status`, `queried_at`). The two are kept apart deliberately -- a failed query
updates the outcome but leaves the answer in place, because the responder
rate-limits anonymous replies to 4 every 3 minutes and one timeout is weak
evidence that a neighbour's scopes changed. An empty answer is stored as a real
answer: it means the neighbour serves unscoped traffic only.
A row is written for any query the node attempted. `timeout` alone cannot carry
that: the sweep reports it both for a neighbour that was asked and stayed silent
and for one it never reached, which is what ScopeResult.transmitted separates.
`send_failed` is recorded too even though nothing reached the air -- it was
attempted and the duty cycle refused, and skipping it left a repeater that keeps
refusing reading as "never queried" however often it was asked.
Scope rows follow their neighbour out of the database, on all four paths that
delete adverts: the two explicit deletes, the 6-hourly retention cleanup, and a
purge of the adverts table. Without the last two the table grew without bound and
a purge left scope counts on screen for repeaters no longer listed.
GET /api/neighbor_scopes serves the table; it stays separate from the paginated
advert queries, which are read per contact type on every page load.
POST /api/query_neighbor_scopes asks one neighbour now. Unlike publish_neighbors
it holds the request open for the reply, since the response window is normally its
5 s floor; past 45 s it returns and leaves the query running so a late answer
still lands rather than throwing away spent airtime. It returns the stored view,
not the raw result, so a failed query cannot tell the client to forget scopes the
database still holds. Nothing is published -- the periodic cycle owns the topic.
A query and a cycle must not collide over the scope helper. A cycle holds it for
its whole run including the discovery window, so a query refuses while one is
active; a cycle defers while a query is in flight; and if the two still race, the
cycle abandons the pass on the short retry delay instead of publishing a table
with every scope missing or dying and re-spending its discovery broadcast. Queries
are tracked so shutdown cancels them rather than transmitting through teardown.
The endpoint needs json_in: it is not on globally for /api and cherrypy's Request
has no `json` attribute without it, so reading the body would have failed on every
real request. A test asserts the decorator rather than trusting a fabricated
cherrypy.request, which is how it went unnoticed.
Follow-up work on the neighbours feature, plus one crash found while reviewing it.
The crash: `mqtt_brokers.neighbors` is a settings block, but the per-broker key of
the same name is a boolean and config.yaml.example documents both, so
`neighbors: true` under mqtt_brokers is an easy hand-edit to make. Every reader
called .get() on it directly, raising AttributeError inside
NeighborScopeHelper.refresh_config -- which is built during daemon init, so the
daemon would not start at all. A shared neighbors_config_block() accessor now
ignores a non-mapping (with one startup warning instead of per-tick noise) and
the API save rewrites it as a proper block.
Payload counters, mirroring firmware buildNeighborsMessage:
- total_neighbors and queried_neighbors. The latter cannot be derived from
`status`: a neighbour the sweep never reached reports `timeout` exactly like one
that was asked and stayed silent. ScopeResult therefore carries a `transmitted`
flag, set only past the point where the injector confirms the request is on air.
- Firmware's third field, `truncated`, is deliberately not emitted -- it reports a
fixed PSRAM JSON buffer overflowing, which openhop does not have. total_neighbors
here consequently always equals the published row count.
- self.default_scope: the region this node stamps on outgoing floods, or `*` when
unset. Read from live config so `region default <name>` applies without a
restart, and `#`-stripped like the scopes string beside it. Firmware tracks this
internally but does not publish it, so this field is openhop-only.
POST /api/publish_neighbors runs a cycle immediately, the HTTP twin of the
`discover.scopes` mesh CLI command. Authenticated by default along with the rest
of /api, and it schedules the cycle on the event loop rather than holding the
request open for the minutes a cycle takes.
Schedule persistence: _next_publish_at is monotonic and _last_publish_at was
in-memory only, so every restart read as "due" and spent a discovery broadcast
plus one serialized scope query per neighbour. A generic daemon_state table
(migration 14) now records the last publish and start() resumes from it, never
sooner than a five-minute grace window and never later than one interval. Two
details this had to get right:
- The schedule keys off the last *successful* publish. A failed publish also sets
_last_publish_at and reschedules on the short retry delay; persisting that would
silently turn the retry into a full interval.
- _tick's disabled branch no longer clears the schedule before the feature has
been enabled in this process. At boot the MQTT connections may not be up, so
enabled() can briefly be false, and clearing there would discard the restored
schedule and re-run the sweep anyway -- the exact thing this prevents.
Build a core RegionMap from the node's served regions and wire it into
the dispatcher and every companion bridge, so a flood reply is re-scoped
to the region its request arrived under (or left plain for a wildcard /
direct request) -- matching firmware simple_repeater::sendFloodReply.
Previously replies went out plain, so a reply to a request in region B
was dropped by B-only repeaters.
The map is built once from the node-wide transport_keys table (each named
region -> RegionEntry, flags=REGION_DENY_FLOOD for deny-flood regions; the
'*' wildcard is deliberately not an entry so plain floods reply plain). A
single shared instance reaches the dispatcher and all bridges. Public
regions rely on name-hashing for their key; a stored key is carried only
when it is genuinely custom material the name would not reproduce, keeping
reply-matching aligned with the forwarding transport-code check.
Region edits at runtime (CLI, web API, Glass sync) all funnel through the
transport_keys CRUD methods, which now fire a post-commit change callback;
the daemon rebuilds the map and reassigns a fresh instance to the
dispatcher and every live bridge (atomic rebind, safe against an in-flight
find_match on the RX thread).
Requires openhop_core with Dispatcher.region_map / CompanionBridge.region_map.
Added a new method to the AirtimeManager class to refresh modulation parameters without clearing transmission history. Updated ConfigManager to call this method after applying live radio configurations. Enhanced tests to verify that modulation updates occur correctly during live updates.
Keep both migration 12 (companion message signal/channel data) and
migration 13 (packet upstream hash for neighbour links), and retain
ACK/MULTIPART plus TRACE imports in engine tests.
The companion queue's capacity path ran a Python loop of single-row
SELECT+DELETE pairs ordered by created_at. Wall-clock ordering meant a
backwards clock step (NTP correction) could make the just-inserted
channel row sort as oldest and wrongly reject the incoming message while
older channel rows remained evictable.
Replace the loop with one set-based DELETE ordered by id (AUTOINCREMENT,
i.e. insertion order, immune to clock steps), with an evictable-count
pre-check preserving the all-or-nothing rejection rule: never displace a
direct message, never evict the incoming row to make room for itself,
roll back the insert entirely when channel rows cannot make room. The
queue load and pop queries move to id ordering for the same reason.
- Implemented `lbt_diagnostics` API endpoint to return aggregated Listen Before Talk (LBT) diagnostics aligned with RF metrics.
- Introduced methods for calculating Pearson correlation coefficients and auto-bucket sizing for diagnostics.
- Enhanced data aggregation logic in `StorageCollector` for LBT diagnostics.
- Updated OpenAPI specification to include new endpoint and response schemas.
- Added comprehensive unit tests for LBT diagnostics, including validation of correlation calculations and data integrity.
standardize default region on mesh.default_region (remove legacy region_default_scope usage)
add/align CLI support for owner.info, path.hash.mode, and loop.detect with validation
wire UI terminal get/set + autocomplete/help for owner.info, path.hash.mode, loop.detect
extend update_radio_config to persist owner_info for UI set owner.info
add and use shared packet utility for advert creation + default-region transport scoping
refactor repeater and room-server advert paths to reuse shared packet logic
The GROUP BY type and GROUP BY drop_reason sub-queries in
get_packet_stats (and get_packet_type_stats) filter by a time window but
let the planner pick idx_packets_type / idx_packets_transmitted to get
grouping for free. It then heap-checks the timestamp filter across the
entire table, turning a bounded window into a full scan: on a 1.5M-row
packets table the 24h stats load spent ~4.8s (type) + ~2.3s
(drop_reason) instead of ~0.1s each.
Pin these to idx_packets_timestamp with INDEXED BY so they range-scan the
window (~50k rows) and group via a small temp b-tree. Verified on the
live DB: 4.80s -> 0.10s and 2.32s -> 0.10s. Unlike a covering index this
adds no write-path cost on the packet-insert hot path.
The airtime/utilization chart queries (get_airtime_data and
get_airtime_buckets) range-scan and order packets by timestamp,
selecting only timestamp/length/payload_length/transmitted. On a large
packets table this forced a full scan of the row heap, saturating slow
storage (e.g. a Pi SD card): each dashboard poll took longer than the
client timeout, aborted polls stacked, and sustained I/O starved the
transmit queue.
Add a covering index on packets(timestamp, length, payload_length,
transmitted) so these queries run index-only, dropping the read from the
full row heap to just the index range. Verified via EXPLAIN QUERY PLAN
(COVERING INDEX idx_packets_airtime). Additive and idempotent.
Merge the maintainer's fix-general-tidy branch (neighbor discovery,
keygen, API endpoints, web-asset rebuild, HTTP server config/control
commands, and an independent #286 room-server push/ACK/guest fix) into
the companion cleanup branch.
Both branches fixed#286 in parallel with byte-identical push-ACK CRC
logic. In the six overlapping files the maintainer's implementation is
kept (ACL replay-detection/session helpers, encoded path-len with legacy
fallback, expected_crc/ack_timeout_s injector API, dispatcher ACK
helpers); this branch's unique companion work is preserved on top
(sender_prefix persistence + migration, boot-state hardening /
CompanionStateLoadError, MessageQueue.max_size, older-core fallbacks).
Conflict resolution took the maintainer's side across the overlap, then
fixed two integration seams the merge introduced and updated this
branch's tests to the maintainer's API:
- room_server: timeout used undefined `hops`; aligned to `path_len`.
- packet_router: PATH helper was invoked twice (maintainer's
unconditional call plus this branch's conditional local-identity
call); dropped the now-redundant conditional block.
Pin openhop_core to @dev (was @feature/publish-workflow-message-handling)
so this can merge to the repeater's dev; core dev carries the required
sender_prefix and PathUtils.is_valid_path_len APIs.
Full suite green (1040 passed) against openhop_core dev and
refactor/companion-housekeeping; ruff clean.
A transient SQLite error during boot made companion_load_channels/
contacts/messages swallow the exception and return [], which is
indistinguishable from "no data". The Public-channel backfill then ran
over the empty store, so clients saw their channels wiped and later
saves could overwrite the persisted state.
- companion_load_{contacts,channels,messages} now return None on error
vs [] for genuinely empty, and log the companion hash
- add companion_count_{channels,messages} helpers
- extract shared _restore_companion_state used by both the boot and
hot-reload companion paths; each load is cross-checked against the
table's row count for the hash, retried once after a short delay,
and raises CompanionStateLoadError if it still cannot load, aborting
companion init (and the Public backfill) instead of starting empty
- log restored row counts per companion; log when the channel store
rejects a persisted channel (unchecked channels.set return)
- trim_companion_contacts_to_fit refuses to trim on a failed load
TXT_TYPE_SIGNED_PLAIN room posts carry a 4-byte author pubkey prefix
(QueuedMessage.sender_prefix, added in openhop_core). The SQLite
persistence path dropped it, so posts replayed from persistence showed
a zero-padded author prefix in the app frame while posts synced from
the live in-memory queue were correct.
- add sender_prefix column (hex text, default '') to companion_messages
with an ALTER TABLE migration for existing databases
- store/return the prefix in companion_push/pop/load_messages
- pass sender_prefix when rebuilding QueuedMessage from persistence in
the frame server and the startup preload paths
- add round-trip, default, migration, and rebuild tests
- Added caching for packet type stats and cumulative counts in SQLiteHandler to reduce database load.
- Implemented a dedicated writer thread in StorageCollector to handle blocking storage operations, preventing asyncio event loop stalls.
- Updated record_packet method to utilize the new writer thread for efficient packet processing.
- Introduced `enforce_companion_contact_capacity` to manage contact limits during companion loading, with an option to trim non-favourite contacts when exceeding capacity.
- Updated `SQLiteHandler` to support message retention limits, allowing for automatic trimming of older messages based on `offline_queue_size`.
- Enhanced API endpoints to handle contact trimming on overflow, providing feedback on trimmed contacts during updates.
- Added utility functions for selecting and trimming contacts while preserving favourites.
- Improved logging for contact management actions and errors related to capacity.
- Updated LogBuffer to support log entry IDs, enhanced log entry structure with additional metadata, and implemented subscriber management for real-time log streaming.
- Added OpenAPI specifications for new endpoints related to policy management, including retrieval, updating, validation, and group management for network policies.
- Implemented comprehensive tests for new policy endpoints, ensuring correct behavior for creating, updating, validating, and deleting policy groups and entries.
- Introduced policy evaluation tests to validate the functionality of the PolicyEngine, including various scenarios for action decisions based on defined rules.
- Enhanced packet routing tests to ensure proper handling of policy decisions in packet processing.
Integrate latest dev while preserving per-companion bridge settings
and contact capacity validation. Resolve import conflicts in main.py
and api_endpoints.py.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Introduced `CompanionContactCapacityError` to handle cases where persisted contacts exceed configured limits.
- Added utility functions for parsing companion bridge settings and validating contact capacity.
- Updated `RepeaterDaemon` to check contact capacity during companion loading and initialization.
- Enhanced API endpoints to validate companion settings and manage contact limits effectively.
- Implemented logging for bridge limits and errors related to contact capacity.
- Updated byte representations in tests to use lowercase hex format for consistency.
- Reformatted code for better readability, including line breaks and indentation adjustments.
- Consolidated multiple lines into single lines where appropriate to enhance clarity.
- Ensured that all test cases maintain consistent formatting and style across the test suite.
Five targeted changes to sqlite_handler.py, all in the same file.
1. Thread-local persistent connections
_connect() previously opened a new sqlite3.connect() on every DB call and
ran journal_mode + busy_timeout PRAGMAs each time. On SD-card storage each
connection open involves file-system operations; each PRAGMA is a round-trip.
threading.local() now caches one connection per thread (write executor thread
+ event-loop/HTTP threads), eliminating per-call setup overhead.
2. PRAGMA synchronous=NORMAL
Default synchronous=FULL flushes WAL frames to disk after every transaction.
NORMAL flushes only at WAL checkpoints — safe for this workload (no data loss
beyond the current transaction on power failure) and significantly faster on
SD cards, which have slow fsync (5-20ms per flush).
3. Migration 8: UNIQUE index on companion_messages(companion_hash, packet_hash)
companion_push_message previously deduped via SELECT + INSERT (two statements,
two SD-card reads per message). The new UNIQUE index enables INSERT OR IGNORE,
replacing the round-trip with a single atomic statement.
4. Migration 9: UNIQUE index on adverts(pubkey)
Without this index store_advert's ON CONFLICT clause cannot fire and each
advert inserts a new row instead of updating the existing one — unbounded
table growth on busy meshes. The migration deduplicates existing rows
(keeping the most-recently-seen per pubkey) before adding the index.
5. Remove duplicate get_unsynced_count definition
The method was defined twice with the same signature. Python silently uses
the last definition; the first was dead code with reversed SQL parameter
binding order. Removed the first; added a note to the surviving definition.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
agessaman