diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index f6a21e2..83e31bd 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -450,6 +450,22 @@ existing codebase, not to the change under review. the failures reproduce identically with and without any change under review and do not occur where the test domains resolve normally (CI). Attribute to the environment, not the codebase or the change. +- **C2 — `tests/test_mesh.py` fails in isolation on Python 3.14.** Running + the C2 command alone (`pytest -q tests/test_mesh.py`) fails 3 daemon + reconnect-loop tests (`test_main_retries_interface_creation`, + `test_main_reconnects_when_connection_event_clears`, + `test_main_recreates_interface_after_snapshot_error`): their local + `DummyEvent` helpers monkeypatch the global `threading.Event` with a + `wait(self, timeout)` signature that requires an argument, and Python + 3.14's `Thread.start()` calls `self._started.wait()` with none → + `TypeError`. Pre-existing and independent of any change under review + (reproduces on a clean tree), and **the full suite (`pytest -q tests/`) + passes** — the failure is an isolation/collection-order artifact of the + global patch. A naive `timeout=None` default is *not* a fix: it lets the + patched-Event path run further and breaks thread startup in the full suite + too ("cannot join thread before it is started"). Needs a proper follow-up + that stops monkeypatching the global `threading.Event` in those three + tests; until then, judge C2 by the full-suite run. --- @@ -2911,3 +2927,114 @@ locals, never the accumulators), the node detail page and chart suites position / neighbor aggregation keep their existing `SNAPSHOT_WINDOW` semantics — only telemetry aggregation changes. No Ruby/Python surface is touched (**C2** and the Python suite unaffected). + +--- + +## Feature: MeshCore RF metrics (RSSI/SNR/hops/path) & roster-eviction assertion + +Maps to SPEC decisions **RF1–RF8**. Ingestor-side logic lives in +`data/mesh_ingestor/protocols/meshcore/` (runner, handlers, decode) and the +Meshtastic hops computation in the packet store path; web-side, one additive +migration adds `messages.hops`, `messages.path`, and `nodes.rssi`, mapped in +`data_processing/` and serialized by the existing GET routes. Store + API only — +no dashboard rendering (RF7). Unless a check says otherwise, Python commands +assume the repo venv (`. .venv/bin/activate`). + +### RF-A1 — hops-travelled stored on messages, both protocols — RF1 +```bash +( . .venv/bin/activate && pytest -q tests/ -k "hops" ) +( cd web && bundle exec rspec spec -e "message hops" ) +``` +**Expected:** pass. MeshCore: a `CHANNEL_MSG_RECV`/`CONTACT_MSG_RECV` payload +with `path_len: N` (N ≤ 63) yields a stored packet with `hops == N`; the `255` +"direct" sentinel yields `hops == 0`; an absent `path_len` omits the field. +Meshtastic: a packet carrying both `hopStart` and `hopLimit` yields +`hops == hopStart − hopLimit`; either absent → field omitted. Web: the +`messages` table has an additive `hops INTEGER` column (NULL for legacy rows), +`POST /api/messages` accepts it, `GET /api/messages` serializes it, and the +existing `hop_limit` column/semantics are untouched. + +### RF-A2 — channel-message RSSI + path via the decrypt_channels join — RF2 +```bash +( . .venv/bin/activate && pytest -q tests/test_provider_unit.py -k "decrypt or path or rssi" ) +( cd web && bundle exec rspec spec -e "message path" ) +``` +**Expected:** pass. `_run_meshcore` sets `mc.decrypt_channels = True` before +`mc.connect()` returns. A channel-message payload carrying joined `RSSI`/`path` +stores both (`rssi` → existing column; `path` → additive `messages.path TEXT`, +lowercase hex, hashes in travel order); a payload **without** them (join miss, +RX-log-less firmware) stores the message identically with the fields absent — +never an error. DMs never carry `path`/`rssi` (E2E, no join — RF2's documented +boundary). The message id (`_derive_message_id` inputs) is byte-identical with +and without the new fields. + +### RF-A3 — RX-log ADVERT frames upsert full node identity + signal — RF3 +```bash +( . .venv/bin/activate && pytest -q tests/test_provider_unit.py -k "rx_log or advert" ) +( cd web && bundle exec rspec spec -e "node rssi" ) +``` +**Expected:** pass. An `RX_LOG_DATA` event with `payload_typename == "ADVERT"` +upserts a node keyed by the canonical id derived from the full `adv_key` +(`_meshcore_node_id`), carrying `adv_name` (long name), the +`_MESHCORE_ADV_TYPE_ROLE` role for `adv_type`, a position when +`adv_lat`/`adv_lon` are present, and per-reception `snr` → `nodes.snr`, +`path_len` → `nodes.hops_away`, `rssi` → the additive `nodes.rssi INTEGER` +column. A malformed advert (missing/short `adv_key`, absent parse fields) is +tolerated without raising. Non-`ADVERT` RX-log frames produce **no** upsert and +remain in the `DEBUG`-only capture; `RX_LOG_DATA` itself no longer lands in +`ignored-meshcore.txt`. With **zero** RX-log frames the provider still passes +RF-A1/RF-A4 behavior (graceful degradation). Web: `POST /api/nodes` accepts +`rssi`, `GET /api/nodes` serializes it, and it stays `NULL` for Meshtastic +nodes (no source). + +### RF-A4 — roster-eviction assertion: read-modify-write, skip, tolerate — RF4 +```bash +( . .venv/bin/activate && pytest -q tests/test_provider_unit.py -k "autoadd" ) +``` +**Expected:** pass. After connect the runner calls `get_autoadd_config`: when +bit `0x01` is already set → **no** `set_autoadd_config` call (no flash write); +when unset → exactly one `set_autoadd_config(config | 0x01)` (type-filter bits +1–4 preserved, one-byte payload so `autoadd_max_hops` is untouched); when the +query/set errors or times out (pre-1.16 firmware) → a warning is logged and +startup **continues** (the connection still succeeds, mirroring +`_ensure_channel_names` tolerance). No env/config knob gates the behavior +(RF4: always-on, README-documented). + +### RF-A5 — CONTACT_DELETED is an explicit debug no-op — RF5 +```bash +( . .venv/bin/activate && pytest -q tests/test_provider_unit.py -k "contact_deleted" ) +``` +**Expected:** pass. `CONTACT_DELETED` appears in the subscribed handler map; on +event it debug-logs and performs **no** node deletion, no POST, and no ignored- +file write — the web DB retains evicted nodes (`retention.rb` remains the only +data-expiry authority). + +### RF-A6 — contract documented; migration additive; dedup frozen — RF6 +```bash +git grep -nE 'hops|path|rssi' -- data/mesh_ingestor/CONTRACTS.md | head +grep -nE 'ALTER TABLE (messages|nodes) ADD COLUMN' data/migrations/*rf_metric*.sql +grep -nE 'hops|path' data/messages.sql; grep -n 'rssi' data/nodes.sql +( . .venv/bin/activate && pytest -q tests/ -k "derive_message_id or dedup" ) +``` +**Expected:** `CONTRACTS.md` documents `messages.hops`/`messages.path` (with +the `255`→direct rule and the path hex format) and `nodes.rssi` (advert→node +mapping). The migration contains only additive `ALTER TABLE … ADD COLUMN` +statements (no drops/rewrites); the base schema files carry the new columns for +fresh databases. The dedup tests pass unchanged — the fingerprint inputs are +byte-identical to pre-feature (MD-A1/MW-A1 hold). + +### RF-R1 — Regression: prior acceptance still holds +```bash +( . .venv/bin/activate && pytest -q tests/ ) +( cd web && bundle exec rspec ) && ( cd web && npm test ) +``` +**Expected:** every prior check still passes. At risk and explicitly required +to remain green: **A4e** (the MeshCore adverts-gap checks — the +bare-`ADVERTISEMENT` minimal-upsert fallback must keep working alongside the +new RX-log enrichment; its assertions are **updated**, not removed), **C2** +(`test_mesh.py` POST shapes — all field additions are additive), **MD-A1 / +MW-A1** (MeshCore dedup — id derivation byte-identical), **MC-A1 / LH-A1 / +GH-A1** (MeshCore message/contact machinery — naming, `last_heard`, and +stale-contact behavior unchanged), and **B1/B4/B5** (all suites, headers, +formatters). The JS suite is exercised for regression only — RF7 adds no +frontend behavior. diff --git a/README.md b/README.md index 1866f13..ab77629 100644 --- a/README.md +++ b/README.md @@ -274,6 +274,24 @@ example `ALLOWED_CHANNELS="Chat,Ops"`); packets on other channels are discarded. Use `HIDDEN_CHANNELS` to block specific channels from the web UI even when they appear in the allowlist. +### MeshCore + +Set `PROTOCOL=meshcore` to ingest from a MeshCore companion-firmware node +instead (serial, TCP, or BLE via the same `CONNECTION` formats). Alongside +contacts and messages, the ingestor captures RF metrics: per-message SNR and +hop counts, per-channel-message RSSI and repeater path (decoded from the +radio's RX log), and per-advert SNR/RSSI/hops for every node heard on air — +including nodes the radio's contact roster has no room for. + +**Note — the ingestor writes one radio setting.** At startup it enables the +firmware's *overwrite-oldest-contact* flag (`autoadd_config` bit `0x01`, +firmware ≥ 1.16) so a full contact roster evicts its oldest non-favourite +entry instead of rejecting new nodes. The write happens only when the bit is +not already set, preserves all other auto-add settings, persists on the +device, and never evicts favourites. Older firmware without the command is +left untouched. Evicted contacts remain in the dashboard's database — only +the radio's local roster rotates. + ### Passive UDP transport The Meshtastic node radio accepts only **one** API client at a time (serial or diff --git a/SPEC.md b/SPEC.md index 26d130e..38595c8 100644 --- a/SPEC.md +++ b/SPEC.md @@ -642,3 +642,56 @@ it rare and, when it occurs, near-invisible. | **BL3** | **Shared dark filter on the fallback tile (amends HT2).** `.map-tiles-fallback` no longer renders `filter:none`; it carries the **same** `grayscale(1) invert(1) brightness(0.9) contrast(1.08)` filter as `.map-tiles-hot`, expressed as one comma-grouped `base.css` rule (the single source of truth for the value). Both providers therefore converge to one coherent dark look, so a viewport mixing HOT and CARTO tiles is no longer a checkerboard — the second half of the fix. The per-tile class swap in `fallback-tile-layer.js` is **unchanged**; only what `.map-tiles-fallback` *does* in CSS changes. Offline placeholder tiles still carry neither class and stay unfiltered. | interview | | **BL4** | **Both maps, one rule; posture preserved (reaffirms HT5 / HT7).** The federation map shares the dashboard's `#map` container, so the single `base.css` filter rule and the shared `createBasemapLayer` factory land the blend on **both** maps identically (parity, Invariant IV). The filter value and the timeout stay **frontend constants** — no Ruby `tile_filters`, no `/version` / `data-app-config` key, no `/api/*` change, no version bump (HT7 / D8 unchanged). | proposed | +--- + +## Feature: MeshCore RF metrics (RSSI/SNR/hops/path) & roster-eviction assertion + +Closes the MeshCore↔Meshtastic RF-metrics parity gap (issue #762 for MeshCore): +messages gain RSSI/SNR/hops-travelled/path, and node records gain per-advert +signal metrics, all sourced from data the `meshcore` library already delivers +but the ingestor currently drops. Three mechanisms: (1) native `path_len`/`SNR` +fields on the message sync events; (2) the library's built-in RX-log⇆message +join enabled via `decrypt_channels`; (3) a new `RX_LOG_DATA` subscription +handling on-air `ADVERT` frames (full node identity + signal metrics, +roster-independent). Additionally the ingestor asserts the firmware's +`AUTO_ADD_OVERWRITE_OLDEST` roster-eviction bit at startup so a full contact +roster keeps rotating instead of rejecting new contacts. Integrates with +`data/mesh_ingestor/protocols/meshcore/{handlers,runner,decode,interface,_constants}.py`, +the packet store path in `data/mesh_ingestor/handlers/`, `CONTRACTS.md`, one +additive SQLite migration (`messages.hops`, `messages.path`, `nodes.rssi`), the +message/node mappings in `web/lib/potato_mesh/application/data_processing/`, +and the serializers in `queries/`. + +**Conflict check against existing decisions.** *Apex I (local LoRa)* — +**consistent**: every new datum arrives over the existing local serial/BLE/TCP +radio link; `decrypt_channels` is local crypto using channel keys the radio +already holds; no broker, dependency, or egress (`guard-edits.py` untriggered). +*Invariant II (privacy & consent)* — **consistent**: no new data class (channel +messages already arrive decrypted via companion sync; adverts are public +broadcasts already stored via the contact path); server-side opt-out, +`PRIVATE`, and retention gates are untouched. The startup config write (RF4) is +a **device** mutation, not a data exposure — treated as a §4.2 scope extension, +documented operator-visibly in the README rather than silent. *Invariant IV +(parity)* — **extends**: fills MeshCore's missing `rxSnr`/`rxRssi`/hops +equivalents using the columns Meshtastic already populates; the new `hops` +column is computed for **both** protocols. *D8/§3.4 (stable contract)* — +**extends**: strictly additive fields (`hops`/`path` on messages, `rssi` on +nodes; adverts otherwise reuse the already-accepted `snr`/`hops_away` node +fields), no version bump. *A4e (MeshCore adverts gap)* — **extends**: RX-log +adverts enrich non-roster nodes with full identity; the bare-`ADVERTISEMENT` +minimal upsert stays as the fallback for builds without RX-log frames (A4e +criteria updated, not removed). *MD-A1/MW-A1 (MeshCore dedup)* — +**consistent**: `_derive_message_id` inputs stay byte-identical (RF6). No +decision is contradicted. + +| # | Decision | Source | +| --- | --- | --- | +| **RF1** | **Hops-travelled on messages, both protocols.** Message packets gain a `hops` field = repeater relays actually travelled: MeshCore reads the native `path_len` on `CONTACT_MSG_RECV`/`CHANNEL_MSG_RECV` (the `255` "direct" sentinel normalizes to `0`; the 2-bit `path_hash_mode` prefix is masked off); Meshtastic computes `hopStart − hopLimit` when both are present (else omits). Stored in a new **additive** `messages.hops INTEGER` column — deliberately distinct from `hop_limit` (remaining budget, a different semantic, left untouched). MeshCore V3 sync frames' native `SNR` continues to flow into the existing `messages.snr` column. | interview + code | +| **RF2** | **Channel-message RSSI/path via the library's RX-log join.** The runner sets `mc.decrypt_channels = True`; the `meshcore` lib then matches each `CHANNEL_MSG_RECV` to its on-air frame by `SHA256(sender_timestamp + text)` and injects `RSSI`, `path`, `recv_time` (channel secrets are already registered by `_ensure_channel_names`, so no new key handling). The handler forwards `RSSI` → the existing `messages.rssi` column and the hop-hash route → a new **additive** `messages.path TEXT` column (lowercase hex, `path_hash_size`-byte hashes concatenated in travel order, last = the repeater heard directly; raw material for a future topology view, no hash→node resolution attempted now). **DM RSSI is explicitly out of scope** — direct messages are E2E-encrypted so no RX-log join exists; DMs still get native SNR + hops via RF1. | interview + code | +| **RF3** | **RX-log `ADVERT` frames become full node upserts (roster-independent).** A new `RX_LOG_DATA` subscription handles frames with `payload_typename == "ADVERT"`: `adv_key` (full 32-byte pubkey) → canonical `!%08x` id, `adv_name` → long name, `adv_type` → role via `_MESHCORE_ADV_TYPE_ROLE`, `adv_lat`/`adv_lon` → position store, and per-reception `snr` → `nodes.snr`, `path_len` → `nodes.hops_away` (existing, already-accepted node fields), `rssi` → a new **additive** `nodes.rssi INTEGER` column (Meshtastic has no per-node RSSI source — `NodeInfo` carries only SNR — so it stays `NULL` there; the column itself is protocol-neutral). This restores full node identity even when the radio's roster is full, closing the anonymous-placeholder gap. **Only `ADVERT` frames are handled**; other RX-log payload types stay in the DEBUG-only catch-all, and `RX_LOG_DATA` no longer falls through to `ignored-meshcore.txt`. Degrades gracefully: companion firmware ≥ 1.16 pushes RX-log frames unconditionally while a client is connected, but if none arrive (other builds), RF1 metrics and the existing bare-`ADVERTISEMENT` fallback (A4e) still function — absent frames are never an error. | interview + code | +| **RF4** | **Always-on roster-eviction assertion (extends §4.2 ingestor scope).** At startup (after connect, `_ensure_channel_names`-style error tolerance) the ingestor reads `get_autoadd_config` and, **only if** bit `0x01` (`AUTO_ADD_OVERWRITE_OLDEST`) is unset, writes `config \| 0x01` back — a read-modify-write that preserves the type-filter bits 1–4 and never touches `autoadd_max_hops`; when the bit is already set no write occurs (the firmware `savePrefs()`es every set — skipping avoids flash wear). Unconditional by deliberate choice — **no env/config knob**; the behavior is documented in the README (deliberate and operator-visible, not silent), favourites are never evicted (firmware guarantee), and the write persists in device flash. `ERROR`/timeout from pre-1.16 firmware logs a warning and continues. This is the ingestor's **first and only** radio-config write, bounded to this single bit. | interview | +| **RF5** | **`CONTACT_DELETED` becomes an explicit no-op handler.** Roster eviction (RF4) makes the firmware emit `PUSH_CODE_CONTACT_DELETED` per evicted contact; the event moves from the DEBUG catch-all to an explicit debug-logged no-op in the handler map — the web DB **intentionally retains** evicted nodes (dashboard history is independent of roster capacity; `retention.rb` remains the only data-expiry authority). | interview | +| **RF6** | **Additive contract; dedup fingerprint frozen.** `CONTRACTS.md` documents the new fields (`messages.hops`/`messages.path`, `nodes.rssi`), the `255`→direct rule, the path hex format, and the advert→node field mapping. All changes are **additive** (D8: no version bump): the POST routes accept and the GET routes serialize the new fields; absent fields stay `NULL`. The MeshCore dedup fingerprint (`_derive_message_id` inputs) and the `v1:` content-dedup scheme are **byte-identical** — new fields ride alongside, never inside, the id derivation (MD-A1/MW-A1 preserved). | interview + code | +| **RF7** | **Store + API only; UI display deferred.** v1 ends at the JSON API — no dashboard rendering of the new metrics (chat hover, node popup, topology view from `path` are tracked follow-ups). Closes #762 for MeshCore at the data layer. | interview | +| **RF8** | **Engineering bar (D9).** All new/changed units ship with 100% unit tests (including: RF1 hops normalization edge cases, RF2 join-absent fallback, RF3 malformed-advert tolerance, RF4 read-modify-write/skip/error paths, RF5 no-op), full PDoc/RDoc, the exact Apache header, `black`/`rufo` clean; every existing suite stays green. | CLAUDE.md | + diff --git a/data/mesh_ingestor/CONTRACTS.md b/data/mesh_ingestor/CONTRACTS.md index d6fa671..0978686 100644 --- a/data/mesh_ingestor/CONTRACTS.md +++ b/data/mesh_ingestor/CONTRACTS.md @@ -44,6 +44,7 @@ remains accepted. Per-field acceptance is nil-aware, so a camelCase value of - `num` (int node number) - `lastHeard` (int unix seconds) - `snr` (float) +- `rssi` (int|nil) — per-advert reception RSSI (SPEC RF3). Sourced from MeshCore RX-log adverts; Meshtastic reports no per-node RSSI, so the field stays absent/NULL there. The web upsert keeps the last stored value when an update omits it (`COALESCE`), so contact-roster refreshes never wipe a per-advert reading. - `hopsAway` (int) - `isFavorite` (bool) - `user` (mapping; e.g. `shortName`, `longName`, `macaddr`, `hwModel`, `publicKey`, `isUnmessagable`) @@ -62,12 +63,16 @@ The web application applies the same normalisation as a safety net so legacy ing **Wire-format note for federation peers (issue #782).** Position time is exposed **only** as `position_time` (unix seconds) on GET responses (`/api/nodes`, `/api/positions`); the redundant ISO twin (`pos_time_iso` on `/api/nodes`, `position_time_iso` on `/api/positions`) was **removed in 0.7.0** — clients format `position_time` themselves. Sentinel rows are compacted by **omitting** `position_time` rather than emitting `0` or `"1970-01-01T00:00:00Z"`. Federation peers consuming this API and any third-party clients SHOULD treat an *absent* `position_time` as "no GPS lock recorded" and not synthesise a zero or epoch value when re-serialising. Older peers that key on `position_time == 0` may need a small adjustment. -**MeshCore advert sourcing (capturing adverts from other nodes).** A MeshCore node announces itself by broadcasting an *advert* (public key + type + name + optional lat/lon). The ingestor surfaces heard adverts to `POST /api/nodes` through three complementary paths so coverage does not depend on the radio's auto-add setting: +**MeshCore advert sourcing (capturing adverts from other nodes).** A MeshCore node announces itself by broadcasting an *advert* (public key + type + name + optional lat/lon). The ingestor surfaces heard adverts to `POST /api/nodes` through four complementary paths so coverage does not depend on the radio's auto-add setting or roster capacity: - *Contact roster (rich).* The startup `ensure_contacts()` fetch plus live `NEW_CONTACT` / `NEXT_CONTACT` pushes carry the full advert (name, role, position) and upsert complete node rows. This covers every node the radio has added to its contact book. - *Auto-update re-fetch (freshness).* The provider sets `mc.auto_update_contacts = True`, so the meshcore library re-fetches **changed** contacts (incrementally, by `lastmod`) whenever an `ADVERTISEMENT` / `PATH_UPDATE` push arrives. A re-advert from a known node therefore refreshes its `last_advert` / position without waiting for a reconnect. - *Bare advert (reach).* The `ADVERTISEMENT` (pubkey-only) push is also handled directly: for a public key **not** in the contact roster it upserts a minimal "heard now" node (`lastHeard`, `protocol`, `user.shortName`/`publicKey` only — no name/type/position), so radios running with auto-add off still register the advertiser. Known keys are skipped (the auto-update path keeps them fresh). The Ruby web app preserves an existing long name on conflict, so this placeholder never clobbers a richer record, and a later full contact advertisement reconciles it. Reconciliation does not depend on timestamp ordering: the contact record carries `lastHeard = last_advert` (the **sender-stamped** advert-creation time), which is always older than the placeholder's wall-clock stamp — the web app's node upsert therefore fills identity fields (name, role, public key, …) that are still NULL even from an older-stamped record, while timestamps/telemetry stay freshness-guarded (ACCEPTANCE GH-A1). +- *RX-log advert (full identity + signal, roster-independent — SPEC RF3).* Companion firmware ≥ 1.16 pushes every received RF frame (`RX_LOG_DATA`) while a client is connected; the library parses `ADVERT` frames completely (full public key, name, type, optional lat/lon). The ingestor converts these to full node upserts carrying per-reception `snr` / `rssi` / `hopsAway`, so node identity and signal metrics no longer depend on the radio's contact roster at all — including when the roster is full. Absent RX-log frames (older/other builds) are never an error; the three paths above still function. Non-`ADVERT` RX-log frames are not ingested (DEBUG-only capture). + +**MeshCore roster-eviction assertion (SPEC RF4).** At startup the provider asserts the firmware's `AUTO_ADD_OVERWRITE_OLDEST` bit (`autoadd_config` bit `0x01`): it reads the current config and, only when the bit is unset, writes `config | 0x01` back — preserving the type-filter bits and `autoadd_max_hops`, and skipping the write (and its flash `savePrefs()`) when already set. With the bit set, a full contact roster evicts its oldest non-favourite entry instead of rejecting new contacts, so `NEW_CONTACT` coverage keeps rotating; favourites are never evicted (firmware guarantee) and the resulting `CONTACT_DELETED` pushes are deliberately ignored (the web DB retains evicted nodes; server-side retention remains the only data-expiry authority). Unconditional, no configuration knob; pre-1.16 firmware answers `ERROR`/timeout, which logs a warning and never blocks startup. + New protocols SHOULD likewise treat "node was heard" as a first-class, name-optional upsert so peer discovery does not hinge on a roster being populated. #### `POST /api/messages` @@ -77,7 +82,10 @@ Single message payload: - Required: `id` (int), `rx_time` (int), `rx_iso` (string) - Identity: `from_id` (string/int), `to_id` (string/int), `channel` (int), `portnum` (string|nil) - Payload: `text` (string|nil), `encrypted` (string|nil), `reply_id` (int|nil), `emoji` (string|nil) -- RF: `snr` (float|nil), `rssi` (int|nil), `hop_limit` (int|nil) +- RF: `snr` (float|nil), `rssi` (int|nil), `hop_limit` (int|nil), `hops` (int|nil), `path` (string|nil) + - `hops` (SPEC RF1) — repeater relays actually travelled, distinct from `hop_limit`'s remaining-budget semantic. MeshCore: the native `path_len` with the `255` "direct" sentinel normalised to `0`. Meshtastic: `hopStart − hopLimit` when both are present, else absent. Additive; absent for legacy senders. + - `path` (SPEC RF2) — MeshCore hop-hash route from the library's RX-log⇆message join (`decrypt_channels`): lowercase hex, `path_hash_size`-byte repeater hashes concatenated in travel order (last hash = the repeater heard directly). Absent on a join miss, on RX-log-less firmware, and on direct messages (E2E-encrypted, no join). Stored verbatim; no hash→node resolution is attempted. Additive. + - Both fields are serialised back on `GET /api/messages`; neither participates in the dedup fingerprint below (the id derivation is byte-identical to pre-RF releases). - Meta: `channel_name` (string; only when not encrypted and known), `ingestor` (canonical host id), `lora_freq`, `modem_preset` - `protocol` (optional string; `"meshtastic"` or `"meshcore"`) — explicit per-record protocol stamp. Takes precedence over the value inherited from the registered ingestor; values outside the whitelist fall back to the ingestor lookup, then to `"meshtastic"`. Ingestors SHOULD stamp this on every message so the web app classifies senders correctly even before the ingestor heartbeat is processed. diff --git a/data/mesh_ingestor/events.py b/data/mesh_ingestor/events.py index 56f2da1..19058d4 100644 --- a/data/mesh_ingestor/events.py +++ b/data/mesh_ingestor/events.py @@ -52,6 +52,8 @@ class MessageEvent(_MessageEventRequired, total=False): snr: float | None rssi: int | None hop_limit: int | None + hops: int | None + path: str | None reply_id: int | None emoji: str | None channel_name: str diff --git a/data/mesh_ingestor/handlers/generic.py b/data/mesh_ingestor/handlers/generic.py index 87a7c2b..6cb78c6 100644 --- a/data/mesh_ingestor/handlers/generic.py +++ b/data/mesh_ingestor/handlers/generic.py @@ -243,6 +243,42 @@ def _is_encrypted_flag(value: object) -> bool: return bool(value) +def _hops_travelled(packet: Mapping, hop_limit: object) -> int | None: + """Return the number of repeater hops a message actually travelled. + + Prefers an explicit ``hops`` value stamped by a protocol handler (MeshCore + normalizes its native ``path_len`` field this way, including the + 255-means-direct sentinel). Falls back to the Meshtastic derivation + ``hopStart - hopLimit``: ``hopStart`` is the sender's initial hop budget + and ``hopLimit`` the budget remaining on receipt, so their difference is + the relay count (SPEC RF1). Distinct from the stored ``hop_limit`` field, + which keeps its remaining-budget semantic unchanged. + + Parameters: + packet: Normalized packet mapping possibly carrying ``hops`` and/or + ``hopStart``/``hop_start``. + hop_limit: The already-extracted ``hopLimit``/``hop_limit`` value + (passed in so the caller's single extraction is reused). + + Returns: + The hop count as an ``int``, or ``None`` when neither source is + present or a value cannot be coerced to an integer. + """ + explicit = _first(packet, "hops", default=None) + if explicit is not None: + try: + return int(explicit) + except (TypeError, ValueError): + return None + hop_start = _first(packet, "hopStart", "hop_start", default=None) + if hop_start is None or hop_limit is None: + return None + try: + return int(hop_start) - int(hop_limit) + except (TypeError, ValueError): + return None + + def upsert_node(node_id: object, node: object) -> None: """Schedule an upsert for a single node. @@ -481,6 +517,13 @@ def store_packet_dict(packet: Mapping) -> None: snr = _first(packet, "snr", "rx_snr", "rxSnr", default=None) rssi = _first(packet, "rssi", "rx_rssi", "rxRssi", default=None) hop = _first(packet, "hopLimit", "hop_limit", default=None) + hops = _hops_travelled(packet, hop) + # Hop-hash route stamped by the MeshCore handler (RF2); Meshtastic packets + # never carry it. Guarded to strings so a malformed value is dropped + # rather than serialized as garbage. + path = _first(packet, "path", default=None) + if not isinstance(path, str): + path = None to_id_normalized = str(to_id).strip() if to_id is not None else "" @@ -538,6 +581,8 @@ def store_packet_dict(packet: Mapping) -> None: "snr": float(snr) if snr is not None else None, "rssi": int(rssi) if rssi is not None else None, "hop_limit": int(hop) if hop is not None else None, + "hops": hops, + "path": path, "reply_id": reply_id, "emoji": emoji, "ingestor": _state.host_node_id(), diff --git a/data/mesh_ingestor/protocols/meshcore/__init__.py b/data/mesh_ingestor/protocols/meshcore/__init__.py index 83612d5..9264836 100644 --- a/data/mesh_ingestor/protocols/meshcore/__init__.py +++ b/data/mesh_ingestor/protocols/meshcore/__init__.py @@ -78,6 +78,7 @@ from ... import queue as _queue # noqa: E402 from ...connection import default_serial_targets # noqa: E402 from ._constants import ( # noqa: E402 - keep grouped with sibling re-exports. + _AUTO_ADD_OVERWRITE_OLDEST, _CHANNEL_PROBE_FALLBACK_MAX, _CONNECT_TIMEOUT_SECS, _DEFAULT_BAUDRATE, @@ -101,6 +102,7 @@ from .decode import ( # noqa: E402 _advert_to_node_dict, _contact_to_node_dict, _derive_modem_preset, + _rx_advert_to_node_dict, _self_info_to_node_dict, ) from .handlers import ( # noqa: E402 @@ -120,12 +122,14 @@ from .interface import ClosedBeforeConnectedError, _MeshcoreInterface # noqa: E from .messages import ( # noqa: E402 _derive_message_id, _extract_mention_names, + _normalize_hops, + _normalize_path, _parse_sender_name, _synthetic_node_dict, ) from .position import _store_meshcore_position # noqa: E402 from .provider import MeshcoreProvider # noqa: E402 -from .runner import _run_meshcore # noqa: E402 +from .runner import _ensure_autoadd_eviction, _run_meshcore # noqa: E402 __all__ = [ "BLEConnection", @@ -135,6 +139,7 @@ __all__ = [ "MeshcoreProvider", "SerialConnection", "TCPConnection", + "_AUTO_ADD_OVERWRITE_OLDEST", "_CHANNEL_PROBE_FALLBACK_MAX", "_CONNECT_TIMEOUT_SECS", "_DEFAULT_BAUDRATE", @@ -150,6 +155,7 @@ __all__ = [ "_derive_message_id", "_derive_modem_preset", "_derive_synthetic_node_id", + "_ensure_autoadd_eviction", "_ensure_channel_names", "_extract_mention_names", "_log_unhandled_loop_exception", @@ -158,6 +164,8 @@ __all__ = [ "_meshcore_adv_type_to_role", "_meshcore_node_id", "_meshcore_short_name", + "_normalize_hops", + "_normalize_path", "_parse_sender_name", "_process_contact_update", "_process_contacts", @@ -165,6 +173,7 @@ __all__ = [ "_pubkey_prefix_to_node_id", "_record_meshcore_message", "_run_meshcore", + "_rx_advert_to_node_dict", "_self_info_to_node_dict", "_store_meshcore_position", "_synthetic_node_dict", diff --git a/data/mesh_ingestor/protocols/meshcore/_constants.py b/data/mesh_ingestor/protocols/meshcore/_constants.py index c849a7e..a1ce379 100644 --- a/data/mesh_ingestor/protocols/meshcore/_constants.py +++ b/data/mesh_ingestor/protocols/meshcore/_constants.py @@ -62,5 +62,23 @@ _MESHCORE_ID_MASK = (1 << _MESHCORE_ID_BITS) - 1 # or returns an older firmware version that omits ``max_channels``. _CHANNEL_PROBE_FALLBACK_MAX = 32 +_AUTO_ADD_OVERWRITE_OLDEST = 0x01 +"""Firmware ``autoadd_config`` bit 0: evict the oldest non-favourite contact +when the roster is full (``MyMesh::shouldOverwriteWhenFull``). + +Bits 1–4 of the same byte are per-type auto-add filters (chat / repeater / +room-server / sensor) that only apply in manual-add mode; the startup +assertion (SPEC RF4) must preserve them via read-modify-write. +""" + +_DIRECT_PATH_LEN = 255 +"""Sentinel ``path_len`` value meaning the packet was received directly. + +The companion protocol encodes a direct (zero-hop) reception as ``0xFF`` +instead of ``0``; any other value is the masked 6-bit hop count (the +``meshcore`` library strips the 2-bit ``path_hash_mode`` prefix before +dispatching, so handlers only ever see ``0``–``63`` or this sentinel). +""" + # Matches @[Name] mention patterns in MeshCore message bodies. _MENTION_RE = re.compile(r"@\[([^\]]+)\]") diff --git a/data/mesh_ingestor/protocols/meshcore/decode.py b/data/mesh_ingestor/protocols/meshcore/decode.py index 3d76912..7a1fbac 100644 --- a/data/mesh_ingestor/protocols/meshcore/decode.py +++ b/data/mesh_ingestor/protocols/meshcore/decode.py @@ -94,6 +94,61 @@ def _advert_to_node_dict(pub_key: str) -> dict: } +def _rx_advert_to_node_dict(frame: dict) -> dict: + """Convert a parsed RX-log ``ADVERT`` frame into a ``POST /api/nodes`` dict. + + Unlike the bare ``ADVERTISEMENT`` push (:func:`_advert_to_node_dict`), an + on-air advert heard via ``RX_LOG_DATA`` is fully self-describing: the + library's packet parser extracts the advertiser's full public key + (``adv_key``), display name, node type, optional position, and the + reception-side signal metrics. This restores full node identity — and + per-advert SNR / RSSI / hop metrics — independently of the radio's contact + roster (SPEC RF3). + + Parameters: + frame: Parsed RX-log frame from the ``meshcore`` library. Relevant + keys: ``adv_key`` (64-hex public key), ``adv_name``, ``adv_type`` + (``ADV_TYPE_*``), ``adv_lat``/``adv_lon``, ``recv_time``, ``snr``, + ``rssi``, and ``path_len`` (hops travelled; RX-log frames carry a + plain count, never the 255 message-sync sentinel). + + Returns: + Node dict compatible with the ``POST /api/nodes`` payload format, + carrying top-level ``snr`` / ``rssi`` / ``hopsAway`` signal fields. + ``longName`` is included only when the advert carried a name, so a + name-less advert never churns an existing richer record. + """ + pub_key = frame.get("adv_key", "") + node_id = _meshcore_node_id(pub_key) + name = (frame.get("adv_name") or "").strip() + role = _meshcore_adv_type_to_role(frame.get("adv_type")) + heard = frame.get("recv_time") or int(time.time()) + node: dict = { + "lastHeard": heard, + "protocol": "meshcore", + "user": { + **({"longName": name} if name else {}), + "shortName": _meshcore_short_name(node_id), + "publicKey": pub_key, + **({"role": role} if role is not None else {}), + }, + } + # Reception-side signal metrics: top-level node fields, matching the keys + # the web app already accepts (``snr``, ``hopsAway``) plus the additive + # ``rssi`` column (RF3). + if frame.get("snr") is not None: + node["snr"] = frame["snr"] + if frame.get("rssi") is not None: + node["rssi"] = frame["rssi"] + if frame.get("path_len") is not None: + node["hopsAway"] = frame["path_len"] + lat = frame.get("adv_lat") + lon = frame.get("adv_lon") + if lat is not None and lon is not None and (lat or lon): + node["position"] = {"latitude": lat, "longitude": lon, "time": heard} + return node + + def _derive_modem_preset(sf: object, bw: object, cr: object) -> str | None: """Return a compact radio-parameter string from spreading factor, bandwidth, and coding rate. diff --git a/data/mesh_ingestor/protocols/meshcore/handlers.py b/data/mesh_ingestor/protocols/meshcore/handlers.py index 3381dc5..12dfad3 100644 --- a/data/mesh_ingestor/protocols/meshcore/handlers.py +++ b/data/mesh_ingestor/protocols/meshcore/handlers.py @@ -16,6 +16,7 @@ from __future__ import annotations +import sys import time from ... import config, ingestors as _ingestors @@ -23,6 +24,7 @@ from .decode import ( _advert_to_node_dict, _contact_to_node_dict, _derive_modem_preset, + _rx_advert_to_node_dict, _self_info_to_node_dict, ) from .identity import _derive_synthetic_node_id, _meshcore_node_id @@ -30,6 +32,8 @@ from .interface import _MeshcoreInterface from .messages import ( _derive_message_id, _extract_mention_names, + _normalize_hops, + _normalize_path, _parse_sender_name, _synthetic_node_dict, ) @@ -281,6 +285,10 @@ def _make_event_handlers(iface: _MeshcoreInterface, target: str | None) -> dict: "channel": channel_idx, "snr": payload.get("SNR"), "rssi": payload.get("RSSI"), + "hops": _normalize_hops(payload.get("path_len")), + # Injected by the decrypt_channels RX-log join (RF2); absent on a + # join miss or RX-log-less firmware, never required. + "path": _normalize_path(payload.get("path")), "protocol": "meshcore", "decoded": { "portnum": "TEXT_MESSAGE_APP", @@ -320,6 +328,7 @@ def _make_event_handlers(iface: _MeshcoreInterface, target: str | None) -> dict: "to_id": iface.host_node_id, "channel": 0, "snr": payload.get("SNR"), + "hops": _normalize_hops(payload.get("path_len")), "protocol": "meshcore", "decoded": { "portnum": "TEXT_MESSAGE_APP", @@ -330,6 +339,65 @@ def _make_event_handlers(iface: _MeshcoreInterface, target: str | None) -> dict: _handlers._mark_packet_seen() _handlers.store_packet_dict(packet) + async def on_rx_log_data(evt) -> None: + payload = evt.payload or {} + if payload.get("payload_typename") != "ADVERT": + # Non-ADVERT RF frames keep their DEBUG-only observability, routed + # explicitly now that RX_LOG_DATA is a handled event and no longer + # reaches the unhandled catch-all (RF3). Resolved via the parent + # package so test fakes installed with monkeypatch apply. + pkg = sys.modules["data.mesh_ingestor.protocols.meshcore"] + pkg._record_meshcore_message( + payload, source=f"{target or 'auto'}:RX_LOG_DATA" + ) + return + + pub_key = payload.get("adv_key", "") + node_id = _meshcore_node_id(pub_key) + if node_id is None: + # Malformed advert (absent/short key, parse failure upstream) — + # tolerated without raising (RF3). + config._debug_log( + "Malformed RX-log advert skipped", + context="meshcore.rx_advert", + severity="warning", + ) + return + + _handlers.upsert_node(node_id, _rx_advert_to_node_dict(payload)) + lat = payload.get("adv_lat") + lon = payload.get("adv_lon") + if lat is not None and lon is not None and (lat or lon): + _store_meshcore_position( + node_id, + lat, + lon, + payload.get("recv_time"), + _handlers.host_node_id(), + ) + _handlers._mark_packet_seen() + config._debug_log( + "MeshCore RX-log advert", + context="meshcore.rx_advert", + node_id=node_id, + name=payload.get("adv_name"), + snr=payload.get("snr"), + rssi=payload.get("rssi"), + hops=payload.get("path_len"), + ) + + async def on_contact_deleted(evt) -> None: + # Deliberate no-op (SPEC RF5): the radio evicting a contact from its + # roster (AUTO_ADD_OVERWRITE_OLDEST, RF4) must not delete anything + # from the dashboard — the web DB intentionally retains evicted nodes + # and ``retention.rb`` stays the only data-expiry authority. + payload = evt.payload or {} + config._debug_log( + "MeshCore contact evicted from radio roster", + context="meshcore.contact_deleted", + node_id=_meshcore_node_id(payload.get("pubkey", "")), + ) + async def on_disconnected(evt) -> None: iface.isConnected = False config._debug_log( @@ -349,5 +417,7 @@ def _make_event_handlers(iface: _MeshcoreInterface, target: str | None) -> dict: "ADVERTISEMENT": on_advertisement, "CHANNEL_MSG_RECV": on_channel_msg, "CONTACT_MSG_RECV": on_contact_msg, + "CONTACT_DELETED": on_contact_deleted, + "RX_LOG_DATA": on_rx_log_data, "DISCONNECTED": on_disconnected, } diff --git a/data/mesh_ingestor/protocols/meshcore/messages.py b/data/mesh_ingestor/protocols/meshcore/messages.py index eda7a34..1190292 100644 --- a/data/mesh_ingestor/protocols/meshcore/messages.py +++ b/data/mesh_ingestor/protocols/meshcore/messages.py @@ -19,7 +19,58 @@ from __future__ import annotations import hashlib import time -from ._constants import _MENTION_RE, _MESHCORE_ID_MASK +from ._constants import _DIRECT_PATH_LEN, _MENTION_RE, _MESHCORE_ID_MASK + + +def _normalize_hops(path_len: object) -> int | None: + """Convert a MeshCore ``path_len`` event field to a hops-travelled count. + + ``CONTACT_MSG_RECV`` / ``CHANNEL_MSG_RECV`` payloads carry ``path_len`` as + the number of repeater relays the message travelled, except that a direct + (zero-hop) reception is encoded as the :data:`~._constants._DIRECT_PATH_LEN` + sentinel (``255``) rather than ``0`` (SPEC RF1). + + Parameters: + path_len: Raw ``path_len`` value from the event payload, or ``None`` + when the firmware frame omitted it. + + Returns: + ``0`` for the direct sentinel, the non-negative hop count otherwise, + or ``None`` when the value is absent or unparseable (defensive: the + library masks the field to 0–63 or 255, but payloads are untyped). + """ + if path_len is None: + return None + try: + value = int(path_len) + except (TypeError, ValueError): + return None + if value == _DIRECT_PATH_LEN: + return 0 + if value < 0: + return None + return value + + +def _normalize_path(path: object) -> str | None: + """Normalize a MeshCore hop-hash route string for storage. + + The ``decrypt_channels`` RX-log join injects ``path`` into + ``CHANNEL_MSG_RECV`` payloads as a hex string of concatenated + ``path_hash_size``-byte repeater hashes in travel order (last hash = the + repeater heard directly). Stored verbatim as raw material for a future + topology view — no hash→node resolution is attempted (SPEC RF2). + + Parameters: + path: Raw ``path`` value from the event payload. + + Returns: + The lowercased hex string, or ``None`` when the value is absent, + empty, or not a string (defensive: payloads are untyped). + """ + if not isinstance(path, str) or not path: + return None + return path.lower() def _derive_message_id( diff --git a/data/mesh_ingestor/protocols/meshcore/runner.py b/data/mesh_ingestor/protocols/meshcore/runner.py index d5c06bb..f7ab568 100644 --- a/data/mesh_ingestor/protocols/meshcore/runner.py +++ b/data/mesh_ingestor/protocols/meshcore/runner.py @@ -21,13 +21,77 @@ import sys import threading from ... import config -from ._constants import _DEFAULT_BAUDRATE +from ._constants import _AUTO_ADD_OVERWRITE_OLDEST, _DEFAULT_BAUDRATE from .channels import _ensure_channel_names from .connection import _make_connection from .handlers import _make_event_handlers from .interface import ClosedBeforeConnectedError, _MeshcoreInterface +async def _ensure_autoadd_eviction(mc) -> None: + """Assert the firmware's roster-eviction bit at startup (SPEC RF4). + + Reads the device's ``autoadd_config`` and, **only when** bit ``0x01`` + (:data:`~._constants._AUTO_ADD_OVERWRITE_OLDEST`) is unset, writes + ``config | 0x01`` back — a read-modify-write that preserves the + type-filter bits 1–4, and a one-byte set so the firmware leaves + ``autoadd_max_hops`` untouched. When the bit is already set no write is + issued: the firmware runs ``savePrefs()`` on every set, so skipping the + no-op write avoids a flash write per ingestor restart. + + Unconditional by design (no env/config knob); favourites are never + evicted (firmware guarantee) and the setting persists in device flash. + A device that does not support commands 58/59 (pre-1.16 firmware) + answers with an ``ERROR`` event or times out — both are tolerated by the + caller's ``try``/``except`` warning path, mirroring + :func:`~.channels._ensure_channel_names`. + + Parameters: + mc: Connected ``MeshCore`` instance. + """ + evt = await mc.commands.get_autoadd_config() + payload = getattr(evt, "payload", None) or {} + current = payload.get("config") + if current is None: + # ERROR reply (unsupported command) or malformed payload — leave the + # device untouched and surface a warning; startup continues. + config._debug_log( + "MeshCore autoadd config unavailable; eviction bit not asserted", + context="meshcore.autoadd", + severity="warning", + always=True, + ) + return + + current = int(current) + if current & _AUTO_ADD_OVERWRITE_OLDEST: + config._debug_log( + "MeshCore roster-eviction bit already set", + context="meshcore.autoadd", + autoadd_config=current, + ) + return + + desired = current | _AUTO_ADD_OVERWRITE_OLDEST + set_evt = await mc.commands.set_autoadd_config(desired) + if getattr(getattr(set_evt, "type", None), "name", "") == "ERROR": + config._debug_log( + "MeshCore rejected autoadd eviction config write", + context="meshcore.autoadd", + severity="warning", + always=True, + autoadd_config=desired, + ) + return + config._debug_log( + "MeshCore roster-eviction bit asserted", + context="meshcore.autoadd", + severity="info", + always=True, + autoadd_config=desired, + ) + + async def _run_meshcore( iface: _MeshcoreInterface, target: str, @@ -68,6 +132,15 @@ async def _run_meshcore( mc = MeshCore(cx) iface._mc = mc + # Enable the library's RX-log⇆message join (SPEC RF2): with channel + # secrets registered (``_ensure_channel_names`` fetches every channel, + # and the reader auto-registers each secret into its packet parser), + # the lib matches each CHANNEL_MSG_RECV to its on-air frame and injects + # RSSI / path / recv_time. Purely local decryption with keys already + # on the radio; a miss (no RX-log frame) simply leaves those fields + # absent, so this degrades gracefully on firmware without RX logging. + mc.decrypt_channels = True + handlers_map = _make_event_handlers(iface, target) for event_name, callback in handlers_map.items(): mc.subscribe(EventType[event_name], callback) @@ -149,6 +222,20 @@ async def _run_meshcore( error=str(exc), ) + # Assert the roster-eviction bit (RF4) after the readiness signal so a + # slow or unsupported command never delays startup; errors and + # timeouts are tolerated exactly like the channel-name fetch above. + try: + await _ensure_autoadd_eviction(mc) + except Exception as exc: + config._debug_log( + "Failed to assert autoadd eviction config", + context="meshcore.autoadd", + severity="warning", + always=True, + error=str(exc), + ) + await mc.start_auto_message_fetching() await stop_event.wait() diff --git a/data/messages.sql b/data/messages.sql index 2d45c47..7e1b62f 100644 --- a/data/messages.sql +++ b/data/messages.sql @@ -25,6 +25,8 @@ CREATE TABLE IF NOT EXISTS messages ( snr REAL, rssi INTEGER, hop_limit INTEGER, + hops INTEGER, + path TEXT, lora_freq INTEGER, modem_preset TEXT, channel_name TEXT, diff --git a/data/migrations/20260722_add_rf_metric_columns.sql b/data/migrations/20260722_add_rf_metric_columns.sql new file mode 100644 index 0000000..ff45bdf --- /dev/null +++ b/data/migrations/20260722_add_rf_metric_columns.sql @@ -0,0 +1,30 @@ +-- Copyright © 2025-26 l5yth & contributors +-- +-- Licensed under the Apache License, Version 2.0 (the "License"); +-- you may not use this file except in compliance with the License. +-- You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, software +-- distributed under the License is distributed on an "AS IS" BASIS, +-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +-- See the License for the specific language governing permissions and +-- limitations under the License. + +-- MeshCore RF metrics (SPEC RF1-RF3, RF6): additive columns only. +-- +-- messages.hops: repeater relays actually travelled (MeshCore path_len with +-- the 255-direct sentinel normalized to 0; Meshtastic hopStart - hopLimit). +-- Distinct from hop_limit, which keeps its remaining-budget semantic. +-- messages.path: MeshCore hop-hash route (lowercase hex, path_hash_size-byte +-- hashes in travel order; last hash = repeater heard directly). +-- nodes.rssi: per-advert reception RSSI (MeshCore RX-log adverts; NULL for +-- Meshtastic, which reports no per-node RSSI). +-- +-- The web app applies these conditionally at boot (database.rb); this file is +-- the standalone mirror for CLI/manual migration of older installations. + +ALTER TABLE messages ADD COLUMN hops INTEGER; +ALTER TABLE messages ADD COLUMN path TEXT; +ALTER TABLE nodes ADD COLUMN rssi INTEGER; diff --git a/data/nodes.sql b/data/nodes.sql index d3d4435..f3fd2d7 100644 --- a/data/nodes.sql +++ b/data/nodes.sql @@ -27,6 +27,7 @@ CREATE TABLE IF NOT EXISTS nodes ( is_favorite BOOLEAN, hops_away INTEGER, snr REAL, + rssi INTEGER, last_heard INTEGER, first_heard INTEGER, battery_level REAL, diff --git a/tests/test_events_unit.py b/tests/test_events_unit.py index 323a770..e23905e 100644 --- a/tests/test_events_unit.py +++ b/tests/test_events_unit.py @@ -41,6 +41,8 @@ def test_message_event_schema(): assert "from_id" in MessageEvent.__optional_keys__ assert "snr" in MessageEvent.__optional_keys__ assert "rssi" in MessageEvent.__optional_keys__ + assert "hops" in MessageEvent.__optional_keys__ + assert "path" in MessageEvent.__optional_keys__ def test_message_event_requires_id_rx_time_rx_iso(): diff --git a/tests/test_handlers_unit.py b/tests/test_handlers_unit.py index 57b5082..4e292f9 100644 --- a/tests/test_handlers_unit.py +++ b/tests/test_handlers_unit.py @@ -1476,3 +1476,133 @@ class TestStorePacketDictPrimaryChannelGuard: assert any(path == "/api/messages" for path, _ in sent) assert "non-primary-channel" not in ignored + + +# --------------------------------------------------------------------------- +# _hops_travelled — hops-travelled derivation (SPEC RF1) +# --------------------------------------------------------------------------- + + +class TestHopsTravelled: + """Unit tests for :func:`generic._hops_travelled`. + + The helper prefers an explicit handler-stamped ``hops`` value (MeshCore's + normalized ``path_len``) and falls back to the Meshtastic derivation + ``hopStart - hopLimit``; anything unparseable yields ``None``. + """ + + def test_explicit_hops_wins_over_derivation(self): + """A handler-stamped hops value beats the hopStart/hopLimit fallback.""" + assert generic_mod._hops_travelled({"hops": 4, "hopStart": 7}, 2) == 4 + + def test_explicit_zero_hops_is_preserved(self): + """hops == 0 (MeshCore direct sentinel already normalized) survives.""" + assert generic_mod._hops_travelled({"hops": 0, "hopStart": 7}, 2) == 0 + + def test_explicit_hops_uncoercible_returns_none(self): + """A non-integer explicit hops yields None, never a crash.""" + assert generic_mod._hops_travelled({"hops": "bogus"}, 2) is None + + def test_meshtastic_derivation_camel_case(self): + """hopStart 7 with hopLimit 2 remaining means 5 relays travelled.""" + assert generic_mod._hops_travelled({"hopStart": 7}, 2) == 5 + + def test_meshtastic_derivation_snake_case(self): + """The snake_case hop_start alias is accepted too.""" + assert generic_mod._hops_travelled({"hop_start": 5}, 1) == 4 + + def test_missing_hop_start_returns_none(self): + """Without hopStart the derivation cannot run.""" + assert generic_mod._hops_travelled({}, 2) is None + + def test_missing_hop_limit_returns_none(self): + """Without hopLimit the derivation cannot run.""" + assert generic_mod._hops_travelled({"hopStart": 3}, None) is None + + def test_uncoercible_fallback_values_return_none(self): + """Unparseable hopStart/hopLimit values yield None, never a crash.""" + assert generic_mod._hops_travelled({"hopStart": "x"}, 2) is None + assert generic_mod._hops_travelled({"hopStart": 7}, "y") is None + + +class TestStorePacketDictHops: + """``store_packet_dict`` stamps the message payload with ``hops`` (RF1). + + Mirrors the primary-channel-guard harness: capture the queued POST payload + and assert the ``hops`` field alongside the untouched ``hop_limit``. + """ + + def _store(self, monkeypatch, packet: dict) -> dict: + """Run ``store_packet_dict`` and return the queued message payload.""" + import data.mesh_ingestor.queue as q + + monkeypatch.setattr(config, "PRIMARY_CHANNEL_ONLY", False) + monkeypatch.setattr(config, "ALLOWED_CHANNELS", ()) + monkeypatch.setattr(config, "HIDDEN_CHANNELS", ()) + monkeypatch.setattr(config, "DEBUG", False) + + sent = [] + original = q._queue_post_json + q._queue_post_json = lambda path, payload, *, priority, **kw: sent.append( + (path, payload) + ) + try: + handlers.store_packet_dict(packet) + finally: + q._queue_post_json = original + + assert len(sent) == 1 and sent[0][0] == "/api/messages" + return sent[0][1] + + def _make_packet(self, **extra) -> dict: + packet = { + "id": 555, + "rxTime": 1_700_000_100, + "from": "!sender", + "to": "^all", + "channel": 0, + "decoded": {"text": "hops probe", "portnum": 1}, + } + packet.update(extra) + return packet + + def test_meshtastic_hops_derived_from_hop_start_and_limit(self, monkeypatch): + """hopStart 7 / hopLimit 2 stores hops 5 with hop_limit untouched.""" + payload = self._store(monkeypatch, self._make_packet(hopStart=7, hopLimit=2)) + assert payload["hops"] == 5 + assert payload["hop_limit"] == 2 + + def test_explicit_hops_field_is_stored_verbatim(self, monkeypatch): + """A protocol-handler-stamped hops value (MeshCore) is stored as-is.""" + payload = self._store(monkeypatch, self._make_packet(hops=0)) + assert payload["hops"] == 0 + + def test_hops_none_when_no_source_present(self, monkeypatch): + """Neither hops nor hopStart present -> hops is None (legacy shape).""" + payload = self._store(monkeypatch, self._make_packet(hopLimit=3)) + assert payload["hops"] is None + assert payload["hop_limit"] == 3 + + +class TestStorePacketDictPath: + """``store_packet_dict`` forwards the MeshCore hop-hash route (RF2).""" + + def test_meshcore_path_is_forwarded(self, monkeypatch): + """A handler-stamped path string reaches the queued payload.""" + harness = TestStorePacketDictHops() + payload = harness._store(monkeypatch, harness._make_packet(path="f0bf44b53377")) + assert payload["path"] == "f0bf44b53377" + + def test_non_string_path_is_dropped(self, monkeypatch): + """A malformed (non-string) path value is dropped, not serialized.""" + harness = TestStorePacketDictHops() + payload = harness._store( + monkeypatch, harness._make_packet(path=["f0bf", "4453"]) + ) + assert payload["path"] is None + + def test_path_none_when_absent(self, monkeypatch): + """Meshtastic packets carry no path -> payload path is None.""" + harness = TestStorePacketDictHops() + payload = harness._store(monkeypatch, harness._make_packet()) + assert payload["path"] is None diff --git a/tests/test_provider_unit.py b/tests/test_provider_unit.py index 16a2c16..a1c23e0 100644 --- a/tests/test_provider_unit.py +++ b/tests/test_provider_unit.py @@ -70,6 +70,8 @@ from data.mesh_ingestor.protocols.meshcore import ( # noqa: E402 - path setup _make_connection, _make_event_handlers, _meshcore_adv_type_to_role, + _normalize_hops, + _normalize_path, _meshcore_node_id, _meshcore_short_name, _parse_sender_name, @@ -1726,6 +1728,382 @@ def test_on_contact_msg_queues_packet_with_from_id(monkeypatch): ) +def test_normalize_hops_maps_path_len_to_hops_travelled(): + """_normalize_hops passes counts through, maps the 255 sentinel to 0, and + rejects absent/unparseable/negative values (SPEC RF1).""" + assert _normalize_hops(None) is None + assert _normalize_hops(0) == 0 + assert _normalize_hops(3) == 3 + assert _normalize_hops("2") == 2 + assert _normalize_hops(255) == 0 + assert _normalize_hops("bogus") is None + assert _normalize_hops(-1) is None + + +def test_on_channel_msg_includes_hops_from_path_len(monkeypatch): + """A channel message with path_len carries the hop count on the packet.""" + import asyncio + + captured, _upserted, _iface, hmap = _setup_channel_msg_handlers(monkeypatch) + asyncio.run( + hmap["CHANNEL_MSG_RECV"]( + _FakeEvt( + { + "sender_timestamp": 1_758_000_010, + "text": "routed message", + "channel_idx": 1, + "path_len": 3, + } + ) + ) + ) + + assert len(captured) == 1 + assert captured[0]["hops"] == 3 + + +def test_on_channel_msg_direct_sentinel_yields_zero_hops(monkeypatch): + """The 255 'direct' path_len sentinel normalizes to hops == 0.""" + import asyncio + + captured, _upserted, _iface, hmap = _setup_channel_msg_handlers(monkeypatch) + asyncio.run( + hmap["CHANNEL_MSG_RECV"]( + _FakeEvt( + { + "sender_timestamp": 1_758_000_011, + "text": "direct channel message", + "channel_idx": 0, + "path_len": 255, + } + ) + ) + ) + + assert len(captured) == 1 + assert captured[0]["hops"] == 0 + + +def test_on_channel_msg_hops_none_when_path_len_absent(monkeypatch): + """A payload without path_len (older firmware) leaves hops unset.""" + import asyncio + + captured, _upserted, _iface, hmap = _setup_channel_msg_handlers(monkeypatch) + asyncio.run( + hmap["CHANNEL_MSG_RECV"]( + _FakeEvt( + { + "sender_timestamp": 1_758_000_012, + "text": "legacy message", + "channel_idx": 0, + } + ) + ) + ) + + assert len(captured) == 1 + assert captured[0]["hops"] is None + + +def test_on_contact_msg_includes_hops_from_path_len(monkeypatch): + """Direct messages carry the normalized hop count too (native field, RF1).""" + import asyncio + import data.mesh_ingestor as _mesh_pkg + import data.mesh_ingestor.protocols.meshcore as _mod + + captured: list = [] + stub = _make_stub_handlers_module() + stub.store_packet_dict = lambda pkt: captured.append(pkt) + monkeypatch.setattr(_mod.config, "_debug_log", lambda *_a, **_k: None) + monkeypatch.setattr(_mesh_pkg, "handlers", stub) + + iface = _MeshcoreInterface(target=None) + iface.host_node_id = "!deadbeef" + + hmap = _make_event_handlers(iface, "/dev/ttyUSB0") + asyncio.run( + hmap["CONTACT_MSG_RECV"]( + _FakeEvt( + { + "sender_timestamp": 1_758_000_013, + "text": "routed dm", + "pubkey_prefix": "aabbccddee11", + "path_len": 2, + } + ) + ) + ) + + assert len(captured) == 1 + assert captured[0]["hops"] == 2 + + +def test_normalize_path_values(): + """_normalize_path lowercases hex strings and rejects non-string/empty.""" + assert _normalize_path("F0BF44B53377") == "f0bf44b53377" + assert _normalize_path("aabb") == "aabb" + assert _normalize_path("") is None + assert _normalize_path(None) is None + assert _normalize_path(123) is None + + +def test_on_channel_msg_includes_path_from_rx_log_join(monkeypatch): + """A channel message with a joined RX-log path stores it lowercased.""" + import asyncio + + captured, _upserted, _iface, hmap = _setup_channel_msg_handlers(monkeypatch) + asyncio.run( + hmap["CHANNEL_MSG_RECV"]( + _FakeEvt( + { + "sender_timestamp": 1_758_000_014, + "text": "joined message", + "channel_idx": 0, + "path": "F0BF44B53377", + "path_len": 3, + } + ) + ) + ) + + assert len(captured) == 1 + assert captured[0]["path"] == "f0bf44b53377" + assert captured[0]["hops"] == 3 + + +def test_on_channel_msg_path_none_on_join_miss_or_invalid(monkeypatch): + """A join miss (absent path) or malformed path leaves the field None.""" + import asyncio + + captured, _upserted, _iface, hmap = _setup_channel_msg_handlers(monkeypatch) + asyncio.run( + hmap["CHANNEL_MSG_RECV"]( + _FakeEvt( + { + "sender_timestamp": 1_758_000_015, + "text": "no join", + "channel_idx": 0, + } + ) + ) + ) + asyncio.run( + hmap["CHANNEL_MSG_RECV"]( + _FakeEvt( + { + "sender_timestamp": 1_758_000_016, + "text": "bad join", + "channel_idx": 0, + "path": 4711, + } + ) + ) + ) + + assert len(captured) == 2 + assert captured[0]["path"] is None + assert captured[1]["path"] is None + + +def test_on_contact_deleted_is_debug_logged_no_op(monkeypatch): + """CONTACT_DELETED must log the evicted node and touch nothing else (RF5).""" + import asyncio + import data.mesh_ingestor.protocols.meshcore as _mod + + captured, upserted, _iface, hmap = _setup_channel_msg_handlers(monkeypatch) + logs: list = [] + monkeypatch.setattr( + _mod.config, "_debug_log", lambda msg, **kw: logs.append((msg, kw)) + ) + + assert "CONTACT_DELETED" in hmap + asyncio.run(hmap["CONTACT_DELETED"](_FakeEvt({"pubkey": "aabbccdd" + "00" * 28}))) + + # No packet stored, no node upserted, no exception — just the debug line. + assert captured == [] + assert upserted == [] + assert any( + kw.get("context") == "meshcore.contact_deleted" + and kw.get("node_id") == "!aabbccdd" + for _msg, kw in logs + ) + + +def test_on_contact_deleted_tolerates_empty_payload(monkeypatch): + """A CONTACT_DELETED push with no payload must not raise.""" + import asyncio + + captured, upserted, _iface, hmap = _setup_channel_msg_handlers(monkeypatch) + asyncio.run(hmap["CONTACT_DELETED"](_FakeEvt(None))) + + assert captured == [] + assert upserted == [] + + +def test_rx_advert_to_node_dict_full_frame(): + """A fully-populated RX-log ADVERT frame maps every field (RF3).""" + from data.mesh_ingestor.protocols.meshcore import _rx_advert_to_node_dict + + pub_key = "511617e3" + "00" * 28 + node = _rx_advert_to_node_dict( + { + "adv_key": pub_key, + "adv_name": "BER Drachentoeter", + "adv_type": 2, + "adv_lat": 52.516274, + "adv_lon": 13.405612, + "recv_time": 1_758_000_020, + "snr": 12.0, + "rssi": -69, + "path_len": 2, + } + ) + + assert node["lastHeard"] == 1_758_000_020 + assert node["protocol"] == "meshcore" + assert node["user"]["longName"] == "BER Drachentoeter" + assert node["user"]["publicKey"] == pub_key + assert node["user"]["role"] == "REPEATER" + assert node["snr"] == 12.0 + assert node["rssi"] == -69 + assert node["hopsAway"] == 2 + assert node["position"] == { + "latitude": 52.516274, + "longitude": 13.405612, + "time": 1_758_000_020, + } + + +def test_rx_advert_to_node_dict_minimal_frame(): + """A name-less, position-less advert omits those keys instead of churning.""" + from data.mesh_ingestor.protocols.meshcore import _rx_advert_to_node_dict + + pub_key = "aabbccdd" + "00" * 28 + node = _rx_advert_to_node_dict({"adv_key": pub_key, "path_len": 0}) + + assert node["user"]["publicKey"] == pub_key + assert "longName" not in node["user"] + assert "role" not in node["user"] + assert "position" not in node + assert "snr" not in node and "rssi" not in node + # A zero-hop (directly heard) advert still records hopsAway == 0. + assert node["hopsAway"] == 0 + assert isinstance(node["lastHeard"], int) + + +def test_on_rx_log_data_advert_upserts_node_and_position(monkeypatch): + """An RX-log ADVERT frame upserts the full node and stores its position.""" + import asyncio + import data.mesh_ingestor.protocols.meshcore.handlers as _handlers_mod + + captured, upserted, _iface, hmap = _setup_channel_msg_handlers(monkeypatch) + positions: list = [] + monkeypatch.setattr( + _handlers_mod, + "_store_meshcore_position", + lambda *args: positions.append(args), + ) + + pub_key = "511617e3" + "00" * 28 + asyncio.run( + hmap["RX_LOG_DATA"]( + _FakeEvt( + { + "payload_typename": "ADVERT", + "adv_key": pub_key, + "adv_name": "BER Drachentoeter", + "adv_type": 2, + "adv_lat": 52.516274, + "adv_lon": 13.405612, + "recv_time": 1_758_000_021, + "snr": 11.5, + "rssi": -70, + "path_len": 3, + } + ) + ) + ) + + assert captured == [] # adverts never produce message packets + assert len(upserted) == 1 + node_id, node = upserted[0] + assert node_id == "!511617e3" + assert node["user"]["longName"] == "BER Drachentoeter" + assert node["snr"] == 11.5 and node["rssi"] == -70 and node["hopsAway"] == 3 + assert len(positions) == 1 + assert positions[0][0] == "!511617e3" + assert positions[0][1] == 52.516274 and positions[0][2] == 13.405612 + + +def test_on_rx_log_data_advert_without_position_skips_position_store(monkeypatch): + """No adv_lat/adv_lon on the advert -> node upsert only, no position POST.""" + import asyncio + import data.mesh_ingestor.protocols.meshcore.handlers as _handlers_mod + + _captured, upserted, _iface, hmap = _setup_channel_msg_handlers(monkeypatch) + positions: list = [] + monkeypatch.setattr( + _handlers_mod, + "_store_meshcore_position", + lambda *args: positions.append(args), + ) + + asyncio.run( + hmap["RX_LOG_DATA"]( + _FakeEvt( + { + "payload_typename": "ADVERT", + "adv_key": "aabbccdd" + "00" * 28, + "snr": 4.0, + } + ) + ) + ) + + assert len(upserted) == 1 + assert positions == [] + + +def test_on_rx_log_data_non_advert_routes_to_debug_capture(monkeypatch): + """Non-ADVERT RF frames go to the DEBUG-only capture, never upsert (RF3).""" + import asyncio + import data.mesh_ingestor.protocols.meshcore as _mod + + captured, upserted, _iface, hmap = _setup_channel_msg_handlers(monkeypatch) + recorded: list = [] + monkeypatch.setattr( + _mod, + "_record_meshcore_message", + lambda message, *, source: recorded.append((message, source)), + ) + + asyncio.run( + hmap["RX_LOG_DATA"]( + _FakeEvt({"payload_typename": "GRP_TXT", "snr": 1.0, "rssi": -90}) + ) + ) + + assert captured == [] and upserted == [] + assert len(recorded) == 1 + message, source = recorded[0] + assert message["payload_typename"] == "GRP_TXT" + assert source.endswith(":RX_LOG_DATA") + + +def test_on_rx_log_data_malformed_advert_tolerated(monkeypatch): + """A short/absent adv_key is skipped without raising (RF3).""" + import asyncio + + captured, upserted, _iface, hmap = _setup_channel_msg_handlers(monkeypatch) + + asyncio.run( + hmap["RX_LOG_DATA"](_FakeEvt({"payload_typename": "ADVERT", "adv_key": "ab"})) + ) + asyncio.run(hmap["RX_LOG_DATA"](_FakeEvt({"payload_typename": "ADVERT"}))) + + assert captured == [] and upserted == [] + + def test_on_channel_msg_id_identical_across_ingestors_with_different_rosters( monkeypatch, ): @@ -2917,6 +3295,9 @@ def _make_fake_meshcore_mod( disconnect_raises: bool = False, connect_stall_event=None, on_ensure_contacts=None, + autoadd_config: int | None = 0x1F, + autoadd_get_raises: bool = False, + autoadd_set_error: bool = False, ): """Build a minimal fake ``meshcore`` module for testing :func:`_run_meshcore`. @@ -2947,6 +3328,8 @@ def _make_fake_meshcore_mod( "CHANNEL_MSG_RECV", "CONTACT_MSG_RECV", "ADVERTISEMENT", + "CONTACT_DELETED", + "RX_LOG_DATA", "DISCONNECTED", "CONNECTED", "ACK", @@ -2961,6 +3344,11 @@ def _make_fake_meshcore_mod( ) class _FakeCommands: + def __init__(self): + # Records every set_autoadd_config write so RF4 tests can assert + # the read-modify-write / skip-when-set behavior. + self.autoadd_set_calls: list[int] = [] + async def send_device_query(self): # Return minimal DEVICE_INFO — channel probing is not under test here. return types.SimpleNamespace( @@ -2971,6 +3359,24 @@ def _make_fake_meshcore_mod( # Return ERROR for all channels — channel probing is not under test here. return types.SimpleNamespace(type=EventType.ERROR, payload={}) + async def get_autoadd_config(self): + # ``autoadd_config=None`` simulates pre-1.16 firmware: an ERROR + # reply whose payload carries no ``config`` key. + if autoadd_get_raises: + raise TimeoutError("autoadd query timed out") + if autoadd_config is None: + return types.SimpleNamespace(type=EventType.ERROR, payload={}) + return types.SimpleNamespace( + type=EventType.CHANNEL_INFO, # any non-ERROR type + payload={"config": autoadd_config}, + ) + + async def set_autoadd_config(self, flag): + self.autoadd_set_calls.append(flag) + if autoadd_set_error: + return types.SimpleNamespace(type=EventType.ERROR, payload={}) + return types.SimpleNamespace(type=EventType.OK, payload={}) + class _FakeMeshCore: def __init__(self, cx): self._catch_all = None @@ -2978,6 +3384,9 @@ def _make_fake_meshcore_mod( # Mirrors the upstream property the runner flips on to keep the # contact roster live across re-adverts (meshcore adverts gap). self.auto_update_contacts = False + # Mirrors the upstream property enabling the RX-log⇆message join + # (SPEC RF2); the runner must flip it on before connecting. + self.decrypt_channels = False # Records every non-catch-all subscription so tests can assert the # runner wires the ADVERTISEMENT handler. self.subscribed_events = [] @@ -3511,7 +3920,9 @@ def test_on_advertisement_ignores_unmappable_pubkey(monkeypatch): def test_run_meshcore_enables_auto_update_and_subscribes_advert(monkeypatch): - """_run_meshcore must enable contact auto-update and subscribe the advert handler.""" + """_run_meshcore must enable contact auto-update, enable the RX-log join + (decrypt_channels, RF2), and subscribe the advert + contact-deleted + handlers.""" import asyncio import data.mesh_ingestor.protocols.meshcore as _mod @@ -3526,4 +3937,101 @@ def test_run_meshcore_enables_auto_update_and_subscribes_advert(monkeypatch): assert error_holder[0] is None assert iface._mc.auto_update_contacts is True + assert iface._mc.decrypt_channels is True assert fake_mod.EventType.ADVERTISEMENT in iface._mc.subscribed_events + assert fake_mod.EventType.CONTACT_DELETED in iface._mc.subscribed_events + assert fake_mod.EventType.RX_LOG_DATA in iface._mc.subscribed_events + + +def _run_meshcore_with_autoadd(monkeypatch, **factory_kwargs): + """Drive ``_run_meshcore`` with a fake lib and return ``(iface, error_holder, logs)``. + + Shared harness for the RF4 roster-eviction assertion tests: captures + ``config._debug_log`` calls so tests can assert the warning/info paths. + """ + import asyncio + import data.mesh_ingestor.protocols.meshcore as _mod + + logs: list = [] + monkeypatch.setattr( + _mod.config, "_debug_log", lambda msg, **kw: logs.append((msg, kw)) + ) + fake_mod = _make_fake_meshcore_mod(**factory_kwargs) + _patch_meshcore_mod(monkeypatch, _mod, fake_mod) + + iface = _MeshcoreInterface(target=None) + connected, error_holder = asyncio.run( + _run_until_connected(iface, "/dev/ttyUSB0", fake_mod, _mod) + ) + assert connected.is_set() + return iface, error_holder, logs + + +def test_run_meshcore_asserts_eviction_bit_when_unset(monkeypatch): + """Bit 0x01 unset -> exactly one read-modify-write set preserving bits 1-4.""" + iface, error_holder, logs = _run_meshcore_with_autoadd( + monkeypatch, autoadd_config=0x1E + ) + + assert error_holder[0] is None + assert iface._mc.commands.autoadd_set_calls == [0x1F] + assert any( + kw.get("context") == "meshcore.autoadd" and kw.get("autoadd_config") == 0x1F + for _msg, kw in logs + ) + + +def test_run_meshcore_skips_autoadd_write_when_bit_already_set(monkeypatch): + """Bit 0x01 already set -> no set call (no savePrefs flash write).""" + iface, error_holder, _logs = _run_meshcore_with_autoadd( + monkeypatch, autoadd_config=0x1F + ) + + assert error_holder[0] is None + assert iface._mc.commands.autoadd_set_calls == [] + + +def test_run_meshcore_autoadd_unsupported_firmware_continues(monkeypatch): + """Pre-1.16 firmware (ERROR reply, no config) -> warning, startup continues.""" + iface, error_holder, logs = _run_meshcore_with_autoadd( + monkeypatch, autoadd_config=None + ) + + assert error_holder[0] is None + assert iface._mc.commands.autoadd_set_calls == [] + assert any( + kw.get("context") == "meshcore.autoadd" and kw.get("severity") == "warning" + for _msg, kw in logs + ) + + +def test_run_meshcore_autoadd_query_timeout_continues(monkeypatch): + """A raising/timing-out query is swallowed with a warning; startup continues.""" + iface, error_holder, logs = _run_meshcore_with_autoadd( + monkeypatch, autoadd_get_raises=True + ) + + assert error_holder[0] is None + assert iface._mc.commands.autoadd_set_calls == [] + assert any( + kw.get("context") == "meshcore.autoadd" + and kw.get("severity") == "warning" + and "timed out" in str(kw.get("error", "")) + for _msg, kw in logs + ) + + +def test_run_meshcore_autoadd_set_rejected_logs_warning(monkeypatch): + """An ERROR reply to the set is logged as a warning; startup continues.""" + iface, error_holder, logs = _run_meshcore_with_autoadd( + monkeypatch, autoadd_config=0x00, autoadd_set_error=True + ) + + assert error_holder[0] is None + assert iface._mc.commands.autoadd_set_calls == [0x01] + assert any( + kw.get("context") == "meshcore.autoadd" + and kw.get("severity") == "warning" + and kw.get("autoadd_config") == 0x01 + for _msg, kw in logs + ) diff --git a/web/lib/potato_mesh/application/data_processing/messages.rb b/web/lib/potato_mesh/application/data_processing/messages.rb index 17782ea..9221b31 100644 --- a/web/lib/potato_mesh/application/data_processing/messages.rb +++ b/web/lib/potato_mesh/application/data_processing/messages.rb @@ -181,6 +181,10 @@ module PotatoMesh emoji = string_or_nil(message["emoji"]) ingestor = string_or_nil(message["ingestor"]) protocol = resolve_record_protocol(db, message, ingestor, cache: protocol_cache) + # RF metrics (SPEC RF1/RF2): hops actually travelled and the MeshCore + # hop-hash route; both additive and absent for legacy senders. + hops = coerce_integer(message["hops"]) + path = string_or_nil(message["path"]) row = [ msg_id, @@ -195,6 +199,8 @@ module PotatoMesh message["snr"], message["rssi"], message["hop_limit"], + hops, + path, lora_freq, modem_preset, channel_name, @@ -312,6 +318,8 @@ module PotatoMesh updates["snr"] = message["snr"] if message.key?("snr") updates["rssi"] = message["rssi"] if message.key?("rssi") updates["hop_limit"] = message["hop_limit"] if message.key?("hop_limit") + updates["hops"] = hops unless hops.nil? + updates["path"] = path if path updates["lora_freq"] = lora_freq unless lora_freq.nil? updates["modem_preset"] = modem_preset if modem_preset updates["channel_name"] = channel_name if channel_name @@ -381,8 +389,8 @@ module PotatoMesh begin db.execute <<~SQL, row - INSERT INTO messages(id,rx_time,rx_iso,from_id,to_id,channel,portnum,text,encrypted,snr,rssi,hop_limit,lora_freq,modem_preset,channel_name,reply_id,emoji,ingestor,protocol) - VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) + INSERT INTO messages(id,rx_time,rx_iso,from_id,to_id,channel,portnum,text,encrypted,snr,rssi,hop_limit,hops,path,lora_freq,modem_preset,channel_name,reply_id,emoji,ingestor,protocol) + VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) SQL rescue SQLite3::ConstraintException existing_row = db.get_first_row( @@ -413,6 +421,8 @@ module PotatoMesh fallback_updates["snr"] = message["snr"] if message.key?("snr") fallback_updates["rssi"] = message["rssi"] if message.key?("rssi") fallback_updates["hop_limit"] = message["hop_limit"] if message.key?("hop_limit") + fallback_updates["hops"] = hops unless hops.nil? + fallback_updates["path"] = path if path fallback_updates["portnum"] = portnum if portnum fallback_updates["lora_freq"] = lora_freq unless lora_freq.nil? fallback_updates["modem_preset"] = modem_preset if modem_preset diff --git a/web/lib/potato_mesh/application/data_processing/node_writes.rb b/web/lib/potato_mesh/application/data_processing/node_writes.rb index 195e607..3bcd31d 100644 --- a/web/lib/potato_mesh/application/data_processing/node_writes.rb +++ b/web/lib/potato_mesh/application/data_processing/node_writes.rb @@ -268,6 +268,7 @@ module PotatoMesh coerce_bool(pick_alias(n, "isFavorite", "is_favorite")), pick_alias(n, "hopsAway", "hops_away"), n["snr"], + n["rssi"], lh, lh, pick_alias(met, "batteryLevel", "battery_level"), @@ -294,9 +295,9 @@ module PotatoMesh db.transaction do db.execute(<<~SQL, row) INSERT INTO nodes(node_id,num,short_name,long_name,macaddr,hw_model,role,public_key,is_unmessagable,is_favorite, - hops_away,snr,last_heard,first_heard,battery_level,voltage,channel_utilization,air_util_tx,uptime_seconds, + hops_away,snr,rssi,last_heard,first_heard,battery_level,voltage,channel_utilization,air_util_tx,uptime_seconds, position_time,location_source,precision_bits,latitude,longitude,altitude,lora_freq,modem_preset,protocol,synthetic) - VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) + VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) ON CONFLICT(node_id) DO UPDATE SET num=COALESCE(excluded.num, nodes.num), short_name=COALESCE(excluded.short_name, nodes.short_name), @@ -307,6 +308,7 @@ module PotatoMesh public_key=COALESCE(excluded.public_key, nodes.public_key), is_unmessagable=COALESCE(excluded.is_unmessagable, nodes.is_unmessagable), is_favorite=excluded.is_favorite, hops_away=excluded.hops_away, snr=excluded.snr, last_heard=excluded.last_heard, + rssi=COALESCE(excluded.rssi, nodes.rssi), first_heard=COALESCE(nodes.first_heard, excluded.first_heard, excluded.last_heard), battery_level=excluded.battery_level, voltage=excluded.voltage, channel_utilization=excluded.channel_utilization, air_util_tx=excluded.air_util_tx, uptime_seconds=excluded.uptime_seconds, diff --git a/web/lib/potato_mesh/application/database.rb b/web/lib/potato_mesh/application/database.rb index 10f8b5e..48d9086 100644 --- a/web/lib/potato_mesh/application/database.rb +++ b/web/lib/potato_mesh/application/database.rb @@ -152,6 +152,12 @@ module PotatoMesh db.execute("ALTER TABLE nodes ADD COLUMN synthetic BOOLEAN NOT NULL DEFAULT 0") end + # RF metrics (SPEC RF3/RF6): per-advert reception RSSI. NULL for + # Meshtastic nodes, which report no per-node RSSI. + unless node_columns.include?("rssi") + db.execute("ALTER TABLE nodes ADD COLUMN rssi INTEGER") + end + if node_columns.include?("long_name") existing_indexes = db.execute("SELECT name FROM sqlite_master WHERE type='index' AND tbl_name='nodes'").flatten unless existing_indexes.include?("idx_nodes_long_name") @@ -259,6 +265,17 @@ module PotatoMesh db.execute("UPDATE messages SET protocol = 'meshtastic' WHERE protocol IS NULL OR TRIM(protocol) = ''") end + # RF metrics (SPEC RF1/RF2/RF6): hops actually travelled (distinct + # from hop_limit's remaining-budget semantic) and the MeshCore + # hop-hash route. Both additive, NULL for legacy rows. + unless message_columns.include?("hops") + db.execute("ALTER TABLE messages ADD COLUMN hops INTEGER") + end + + unless message_columns.include?("path") + db.execute("ALTER TABLE messages ADD COLUMN path TEXT") + end + reply_index_exists = db.get_first_value( "SELECT COUNT(*) FROM sqlite_master WHERE type='index' AND name='idx_messages_reply_id'", diff --git a/web/lib/potato_mesh/application/queries/chat_queries.rb b/web/lib/potato_mesh/application/queries/chat_queries.rb index f3d5c00..1b1dbdf 100644 --- a/web/lib/potato_mesh/application/queries/chat_queries.rb +++ b/web/lib/potato_mesh/application/queries/chat_queries.rb @@ -71,6 +71,7 @@ module PotatoMesh sql = <<~SQL SELECT m.id, m.rx_time, m.rx_iso, m.from_id, m.to_id, m.channel, m.portnum, m.text, m.encrypted, m.rssi, m.hop_limit, + m.hops, m.path, m.lora_freq, m.modem_preset, m.channel_name, m.snr, m.reply_id, m.emoji, m.ingestor, m.protocol FROM messages m diff --git a/web/lib/potato_mesh/application/queries/node_queries.rb b/web/lib/potato_mesh/application/queries/node_queries.rb index 9d0d004..254fffd 100644 --- a/web/lib/potato_mesh/application/queries/node_queries.rb +++ b/web/lib/potato_mesh/application/queries/node_queries.rb @@ -177,6 +177,7 @@ module PotatoMesh sql = <<~SQL SELECT node_id, short_name, long_name, hw_model, role, snr, + rssi, hops_away, battery_level, voltage, last_heard, first_heard, uptime_seconds, channel_utilization, air_util_tx, position_time, location_source, precision_bits, diff --git a/web/spec/rf_metrics_spec.rb b/web/spec/rf_metrics_spec.rb new file mode 100644 index 0000000..0af8f34 --- /dev/null +++ b/web/spec/rf_metrics_spec.rb @@ -0,0 +1,212 @@ +# Copyright © 2025-26 l5yth & contributors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# frozen_string_literal: true + +require "spec_helper" +require "sqlite3" +require "json" + +# RF-metrics ingest/read-side coverage (SPEC RF1-RF3, RF6; ACCEPTANCE RF-A1 - +# RF-A3): the additive `messages.hops` / `messages.path` / `nodes.rssi` +# columns round-trip through the authenticated POST ingest routes and surface +# on the GET collection responses, while legacy payloads without the fields +# keep storing NULL. +RSpec.describe "RF metrics (hops/path/rssi) ingest and read-side" do + let(:app) { Sinatra::Application } + let(:api_token) { "spec-token" } + let(:auth_headers) do + { + "CONTENT_TYPE" => "application/json", + "HTTP_AUTHORIZATION" => "Bearer #{api_token}", + } + end + + # Execute the provided block with a configured SQLite connection. + # + # @yieldparam db [SQLite3::Database] open database handle. + # @return [void] + def with_db + db = SQLite3::Database.new(PotatoMesh::Config.db_path) + db.busy_timeout = PotatoMesh::Config.db_busy_timeout_ms + yield db + ensure + db&.close + end + + # Remove the rows this spec writes so examples stay independent. + # + # @return [void] + def clear_rf_tables + with_db do |db| + db.execute("DELETE FROM messages") + db.execute("DELETE FROM nodes") + db.execute("DELETE FROM positions") + end + end + + before do + @original_token = ENV["API_TOKEN"] + ENV["API_TOKEN"] = api_token + clear_rf_tables + PotatoMesh::App::ApiCache.invalidate_all + end + + after do + if @original_token.nil? + ENV.delete("API_TOKEN") + else + ENV["API_TOKEN"] = @original_token + end + end + + # Build a minimal valid message payload the ingest route accepts. + # + # @param overrides [Hash] extra/overriding message fields. + # @return [Hash] POST /api/messages payload. + def message_payload(overrides = {}) + now = Time.now.to_i + { + "id" => 424_242, + "rx_time" => now, + "rx_iso" => Time.at(now).utc.iso8601, + "from_id" => "!aabbccdd", + "to_id" => "^all", + "channel" => 0, + "portnum" => "TEXT_MESSAGE_APP", + "text" => "rf metrics probe", + "snr" => 5.5, + "rssi" => -80, + "hop_limit" => 2, + }.merge(overrides) + end + + describe "message hops" do + it "stores hops on POST /api/messages and serves it on GET /api/messages" do + post "/api/messages", message_payload("hops" => 5).to_json, auth_headers + expect(last_response.status).to eq(201) + + get "/api/messages" + expect(last_response).to be_ok + row = JSON.parse(last_response.body).find { |m| m["id"] == 424_242 } + expect(row).not_to be_nil + expect(row["hops"]).to eq(5) + # hop_limit keeps its remaining-budget semantic untouched (RF1). + expect(row["hop_limit"]).to eq(2) + end + + it "stores NULL hops for legacy messages without the field" do + post "/api/messages", message_payload.to_json, auth_headers + expect(last_response.status).to eq(201) + + get "/api/messages" + row = JSON.parse(last_response.body).find { |m| m["id"] == 424_242 } + expect(row).not_to be_nil + expect(row["hops"]).to be_nil + end + end + + describe "message path" do + it "stores the meshcore hop-hash path and serves it on GET /api/messages" do + payload = message_payload( + "id" => 424_243, + "protocol" => "meshcore", + "hops" => 3, + "path" => "f0bf44b53377", + ) + post "/api/messages", payload.to_json, auth_headers + expect(last_response.status).to eq(201) + + get "/api/messages" + row = JSON.parse(last_response.body).find { |m| m["id"] == 424_243 } + expect(row).not_to be_nil + expect(row["path"]).to eq("f0bf44b53377") + expect(row["hops"]).to eq(3) + end + + it "stores NULL path when the field is absent (join miss / Meshtastic)" do + post "/api/messages", message_payload("id" => 424_244).to_json, auth_headers + expect(last_response.status).to eq(201) + + get "/api/messages" + row = JSON.parse(last_response.body).find { |m| m["id"] == 424_244 } + expect(row).not_to be_nil + expect(row["path"]).to be_nil + end + end + + describe "node rssi" do + # Build a minimal MeshCore advert-style node payload (RF3). + # + # @param overrides [Hash] extra/overriding node fields. + # @return [Hash] node dict for POST /api/nodes. + def node_payload(overrides = {}) + { + "lastHeard" => Time.now.to_i, + "protocol" => "meshcore", + "snr" => 12.0, + "rssi" => -69, + "hopsAway" => 2, + "user" => { + "longName" => "BER Drachentoeter", + "shortName" => "5116", + "publicKey" => "511617e3" + "00" * 28, + "role" => "REPEATER", + }, + }.merge(overrides) + end + + it "stores advert rssi and serves rssi and hops_away on GET /api/nodes" do + post "/api/nodes", { "!511617e3" => node_payload }.to_json, auth_headers + expect(last_response.status).to eq(201) + + get "/api/nodes" + expect(last_response).to be_ok + row = JSON.parse(last_response.body).find { |n| n["node_id"] == "!511617e3" } + expect(row).not_to be_nil + expect(row["rssi"]).to eq(-69) + expect(row["hops_away"]).to eq(2) + expect(row["snr"]).to eq(12.0) + end + + it "preserves a stored rssi when a later upsert omits it (COALESCE)" do + post "/api/nodes", { "!511617e3" => node_payload }.to_json, auth_headers + expect(last_response.status).to eq(201) + + # A contact-roster refresh carries no rssi; the stored per-advert value + # must survive (nodes.rssi COALESCE upsert, RF3/RF6). + refresh = node_payload("rssi" => nil, "lastHeard" => Time.now.to_i + 60) + refresh.delete("rssi") + post "/api/nodes", { "!511617e3" => refresh }.to_json, auth_headers + expect(last_response.status).to eq(201) + + get "/api/nodes" + row = JSON.parse(last_response.body).find { |n| n["node_id"] == "!511617e3" } + expect(row).not_to be_nil + expect(row["rssi"]).to eq(-69) + end + + it "leaves rssi NULL for meshtastic nodes (no source)" do + payload = node_payload("protocol" => "meshtastic") + payload.delete("rssi") + post "/api/nodes", { "!511617e3" => payload }.to_json, auth_headers + expect(last_response.status).to eq(201) + + get "/api/nodes" + row = JSON.parse(last_response.body).find { |n| n["node_id"] == "!511617e3" } + expect(row).not_to be_nil + expect(row["rssi"]).to be_nil + end + end +end