Merge pull request #174 from ipnet-mesh/feature/improve-snr-path-visibility

Add observer detail rows with SNR/path data and UI polish
This commit is contained in:
JingleManSweep
2026-04-26 13:05:10 +01:00
committed by GitHub
30 changed files with 1208 additions and 203 deletions
+21
View File
@@ -189,6 +189,11 @@ Group/broadcast messages on specific channels.
- Payload type `4` location metadata (`appData.location.latitude/longitude`) is mapped to node `lat/lon` for map rendering.
- This keeps advertisement persistence aligned with meshcore-packet-capture expectations (advertisement traffic only).
**Compatibility ingest note (envelope fields)**:
- The LetsMesh upload envelope carries `SNR` and `path` fields alongside the decoded packet payload. These are available on all packet types (messages, advertisements, traces, telemetry).
- The normalizer extracts `SNR` (normalized to lowercase `snr`) and `path` (converted to `path_len` via hop count) from the envelope and includes them in the normalized payload.
- Per-observer `snr` and `path_len` are stored in the `event_observers` junction table, allowing each observer to record its own signal strength and hop count.
**Compatibility ingest note (non-message structured events)**:
- Decoded payload type `9` is normalized to `TRACE_DATA` (`traceTag`, flags, auth, path hashes, and SNR values).
- Decoded payload type `11` (`Control/NodeDiscoverResp`) is normalized to `contact` events for node upsert parity.
@@ -514,6 +519,22 @@ See [AGENTS.md](AGENTS.md) for webhook configuration details.
---
## API Response: Observer Info
Events that support multi-observer tracking (messages, advertisements, trace paths, telemetry) include an `observers` array in API responses. Each observer entry contains:
| Field | Type | Description |
|-------|------|-------------|
| `node_id` | string (UUID) | Observer node UUID |
| `public_key` | string (64 hex chars) | Observer node public key |
| `name` | string or null | Observer node advertised name |
| `tag_name` | string or null | Observer name from node tags |
| `snr` | number or null | Signal-to-noise ratio at this observer (dB) |
| `path_len` | integer or null | Hop count at this observer |
| `observed_at` | string (ISO 8601) | When this observer captured the event |
---
## Event Flow
1. **Hardware/Mock MeshCore** → Generates raw events
@@ -0,0 +1,45 @@
"""add path_len to event_observers
Revision ID: a10dbca883a2
Revises: b1c2d3e4f5a6
Create Date: 2026-04-26 10:52:09.664958+00:00
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = "a10dbca883a2"
down_revision: Union[str, None] = "b1c2d3e4f5a6"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("event_observers", schema=None) as batch_op:
batch_op.add_column(sa.Column("path_len", sa.Integer(), nullable=True))
batch_op.drop_constraint(
batch_op.f("uq_event_receivers_hash_node"), type_="unique"
)
batch_op.create_unique_constraint(
"uq_event_observers_hash_node", ["event_hash", "observer_node_id"]
)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table("event_observers", schema=None) as batch_op:
batch_op.drop_constraint("uq_event_observers_hash_node", type_="unique")
batch_op.create_unique_constraint(
batch_op.f("uq_event_receivers_hash_node"),
["event_hash", "observer_node_id"],
)
batch_op.drop_column("path_len")
# ### end Alembic commands ###
@@ -0,0 +1,342 @@
# Observer Detail Rows — Implementation Plan
**Date:** 2026-04-26
**Status:** Approved
## Decisions
1. **No backfill** — Historical data unchanged. Per-observer `path_len` only captured for new events.
2. **Canonical case: lowercase `snr`** — All code normalizes SNR references to lowercase.
3. **Trace observer pattern** — Per-observer `path_len` tracks the observer's hop count to the event source.
## Terminology
The original research used outdated names. This plan uses the correct codebase terminology:
| Old Reference | Actual Codebase |
|---------------|-----------------|
| `event_receivers` | `event_observers` |
| `add_event_receiver()` | `add_event_observer()` |
| `_fetch_receivers_for_events()` | `_fetch_observers_for_events()` |
| `ReceiverInfo` | `ObserverInfo` |
| `receivers` (API field) | `observers` |
## Verified Current State
### Database (`event_observers` table)
```
event_observers
├── id UUID PK
├── event_type String(20)
├── event_hash String(32)
├── observer_node_id FK → nodes.id
├── snr Float (nullable)
├── observed_at DateTime
├── created_at DateTime
└── updated_at DateTime
```
**Missing:** `path_len` column.
### Per-Event vs Per-Observer Fields
| Field | Scope | Rationale |
|-------|-------|-----------|
| `snr` | **Per-observer** | Signal strength differs by observer location |
| `path_len` | **Per-observer** | Hop count differs by observer position in mesh topology |
| `observed_at` | **Per-observer** | Each observer sees the event at a different time |
| `snr_values` (trace) | **Per-event only** | Per-hop SNR along the trace path |
| `hop_count` (trace) | **Per-event only** | Total hops in the trace |
### Handler Payload Extraction
| Handler | SNR extraction | path_len extraction |
|---------|---------------|-------------------|
| `message.py` | `payload.get("SNR") or payload.get("snr")` | `payload.get("path_len")` |
| `advertisement.py` | None | None |
| `trace.py` | None | `payload.get("path_len")` |
| `telemetry.py` | None | None |
Each handler has exactly 3 `add_event_observer()` call sites:
1. Duplicate path (existing event)
2. First observer (new event)
3. Race condition recovery
All call sites need `path_len` parameter.
### LetsMesh Normalizer
`_build_letsmesh_advertisement_payload()` (handles decoded packet type 4) does NOT extract envelope `SNR` or `path`. The message payload method already does (lines 143-160). **Note:** The message method outputs `normalized_payload["SNR"]` (uppercase) at line 160, which contradicts Decision #2. This must be changed to lowercase `"snr"` alongside the advertisement fix.
### API Routes
| Route | Populates `observers`? | Notes |
|-------|----------------------|-------|
| `api/routes/messages.py` | Yes | Uses `_fetch_observers_for_events()` |
| `api/routes/advertisements.py` | Yes | Uses `_fetch_observers_for_events()` |
| `api/routes/trace_paths.py` | No — returns `[]` | Schema has field but never queries |
| `api/routes/telemetry.py` | No — returns `[]` | Same issue |
`_fetch_observers_for_events()` is duplicated identically in `messages.py` and `advertisements.py`.
### Frontend
- **`components.js:444` `receiverIcons()`** — Dead code. Uses wrong property names (`receiver_node_name` / `receiver_node_public_key`). No page imports it.
- **`messages.js` and `advertisements.js`** — Render satellite dish icons with correct property names (`recv.tag_name`, `recv.name`, `recv.public_key`) but display name-only tooltips. Per-observer SNR/path_len is returned by API but never displayed.
- **No trace_paths.js or telemetry.js frontend pages exist.**
- **No expandable/collapsible row patterns** exist anywhere in the SPA.
### Schema (`ObserverInfo`)
```python
class ObserverInfo(BaseModel):
node_id: str
public_key: str
name: Optional[str]
tag_name: Optional[str]
snr: Optional[float]
observed_at: datetime
```
**Missing:** `path_len` field.
---
## Implementation Plan
### Phase 1: Database Schema — Add `path_len` to `event_observers`
**File:** `src/meshcore_hub/common/models/event_observer.py`
- Add column: `path_len: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)`
- Update `add_event_observer()` signature to accept `path_len: Optional[int] = None`
- Include `path_len` in `sqlite_insert().values()` call
**Migration:**
```bash
source .venv/bin/activate
meshcore-hub db revision --autogenerate -m "add path_len to event_observers"
meshcore-hub db upgrade
```
### Phase 2: API Utility & Schema
**New file:** `src/meshcore_hub/api/observer_utils.py`
Move `_fetch_observers_for_events()` into a shared module. Update the query to also select `EventObserver.path_len`.
```python
def _fetch_observers_for_events(
session: DbSession,
event_type: str,
event_hashes: list[str],
) -> dict[str, list[ObserverInfo]]:
```
The query selects: `EventObserver.event_hash`, `EventObserver.snr`, `EventObserver.path_len`, `EventObserver.observed_at`, `Node.id`, `Node.public_key`, `Node.name`. Also fetches `NodeTag` name tags.
**Schema update:** `src/meshcore_hub/common/schemas/messages.py`
Add to `ObserverInfo`:
```python
path_len: Optional[int] = Field(default=None, description="Hop count at this observer")
```
**Route updates:**
| File | Change |
|------|--------|
| `api/routes/messages.py` | Remove local `_fetch_observers_for_events`, import from `observer_utils` |
| `api/routes/advertisements.py` | Same |
| `api/routes/trace_paths.py` | Import and call `_fetch_observers_for_events(session, "trace", event_hashes)` to populate `observers` |
| `api/routes/telemetry.py` | Import and call `_fetch_observers_for_events(session, "telemetry", event_hashes)` to populate `observers` |
### Phase 3: LetsMesh Normalizer — Extract Envelope SNR & Path for Advertisements
**File:** `src/meshcore_hub/collector/letsmesh_normalizer.py`
In `_build_letsmesh_advertisement_payload()`, add after the `normalized_payload` dict initialization (line 574):
```python
snr = self._parse_float(payload.get("SNR"))
if snr is None:
snr = self._parse_float(payload.get("snr"))
if snr is not None:
normalized_payload["snr"] = snr
path_len = self._parse_path_length(payload.get("path"))
if path_len is not None:
normalized_payload["path_len"] = path_len
```
This follows the same pattern used in `_build_letsmesh_message_payload()` (lines 143-160), with the key difference being lowercase `"snr"` output.
> **Inaccuracy corrected (review 2026-04-26):** The message normalizer at line 160 outputs `normalized_payload["SNR"]` (uppercase). Phase 4 below removes the handler's uppercase fallback (`payload.get("SNR") or ...`). Without also fixing the normalizer output, LetsMesh-routed messages would lose SNR data. The change below is added to enforce Decision #2 (canonical lowercase `snr`) consistently.
In `_build_letsmesh_message_payload()`, change line 160:
```python
# Before:
normalized_payload["SNR"] = snr
# After:
normalized_payload["snr"] = snr
```
### Phase 4: Collector Handlers — Pass SNR & path_len
**File:** `src/meshcore_hub/collector/handlers/message.py`
- Change line 78: `payload.get("SNR") or payload.get("snr")``payload.get("snr")`
- `path_len` is already extracted (line 75) — just pass it to all 3 `add_event_observer()` call sites (lines 124, 158, 178)
**File:** `src/meshcore_hub/collector/handlers/advertisement.py`
- Add extraction: `snr = payload.get("snr")`
- Add extraction: `path_len = payload.get("path_len")`
- Pass both to all 3 `add_event_observer()` call sites (lines 120, 182, 203)
**File:** `src/meshcore_hub/collector/handlers/trace.py`
- Add extraction: `snr = payload.get("snr")`
- `path_len` is already extracted (line 38) — pass both to all 3 `add_event_observer()` call sites (lines 74, 106, 126)
**File:** `src/meshcore_hub/collector/handlers/telemetry.py`
- Add extraction: `snr = payload.get("snr")`
- Add extraction: `path_len = payload.get("path_len")`
- Pass both to all 3 `add_event_observer()` call sites (lines 87, 133, 154)
### Phase 5: Frontend Components
**File:** `src/meshcore_hub/web/static/js/spa/components.js`
- **Remove** dead `receiverIcons()` function (lines 444-452) — uses wrong property names, no page imports it
- **Add** `observerDetailRow(observers, eventProperties)` component:
- Renders an expandable sub-table below the event row
- Observer columns:
- **Observer** — `tag_name || name || truncateKey(public_key, 12)`, linked to `/nodes/${public_key}`
- **SNR** — Formatted as "X.X dB" or "—" if null
- **Path** — Formatted as "N hops" or "—" if null
- **Received** — Relative time via `formatRelativeTime(observed_at)`
- `eventProperties` parameter for event-level context (e.g., trace `snr_values`)
- Toggle helper: click event row to show/hide `.observer-detail` row below it
- **Add** `observerIcons(observers)` — count badge with tooltip listing observer names
**File:** `src/meshcore_hub/web/static/css/app.css`
- `.observer-detail` expandable row styles (indented, compact sub-table)
- CSS transition for smooth expand/collapse (max-height animation)
- Responsive: desktop table vs mobile card layout
### Phase 6: Frontend Pages
**File:** `src/meshcore_hub/web/static/js/spa/pages/messages.js`
Replace current satellite dish icon rendering with:
- Observer count badge in the Receivers column (clickable to expand)
- Expandable detail row showing per-observer: name, SNR, path_len, observed_at
- Both desktop table (~line 255) and mobile card (~line 206) views
**File:** `src/meshcore_hub/web/static/js/spa/pages/advertisements.js`
Same pattern as messages.
- Desktop table (~line 135)
- Mobile card (~line 99)
**Note:** No `trace_paths.js` frontend page exists. No frontend changes needed for trace paths — API changes in Phase 2 will surface observer data for future pages or API consumers.
### Phase 7: Tests
**`tests/test_collector/test_letsmesh_normalizer.py`**
- Test type 4 packet extracts `snr` (lowercase) and `path_len` from envelope
- Test both `"SNR"` and `"snr"` input casing normalizes to lowercase `"snr"` output
- Test message payload also outputs lowercase `"snr"` (verifies line 160 casing fix)
**`tests/test_collector/test_handlers/test_advertisement.py`**
- Test handler with `snr` and `path_len` in payload → stored in `event_observers`
**`tests/test_collector/test_handlers/test_message.py`**
- Test handler passes `path_len` to `add_event_observer()`
- Fix casing: change `"SNR": 15.5``"snr": 15.5` (line 21) and `"SNR": 8.5``"snr": 8.5` (line 102)
**`tests/test_collector/test_handlers/test_trace.py`**
- Test handler with envelope `snr`/`path_len` → stored in `event_observers`
**`tests/test_collector/test_handlers/test_telemetry.py`**
- Test handler with envelope `snr`/`path_len` → stored in `event_observers`
**`tests/test_common/test_models.py`**
- Test `add_event_observer()` accepts and stores `path_len`
- Test backwards compatibility (`path_len=None` is default)
**`tests/test_api/test_trace_paths.py`**
- Test `observers` list is populated (query returns data)
**`tests/test_api/test_telemetry.py`**
- Test `observers` list is populated (query returns data)
**Run commands:**
```bash
source .venv/bin/activate
pytest tests/test_collector/ -v
pytest tests/test_api/ -v
pytest tests/test_common/ -v
pre-commit run --all-files
```
### Phase 8: Documentation
**`SCHEMAS.md`:**
- Document that `SNR` and `path` are LetsMesh envelope fields available on all packet types
- Update `ObserverInfo` description to include `path_len`
**`AGENTS.md`:**
- Update `event_observers` table description to include `path_len` column
---
## File Change Summary
| # | File | Action | Phase |
|---|------|--------|-------|
| 1 | `common/models/event_observer.py` | Modify | 1 |
| 2 | `alembic/versions/*.py` | Create | 1 |
| 3 | `common/schemas/messages.py` | Modify | 2 |
| 4 | `api/observer_utils.py` | Create | 2 |
| 5 | `api/routes/messages.py` | Modify | 2 |
| 6 | `api/routes/advertisements.py` | Modify | 2 |
| 7 | `api/routes/trace_paths.py` | Modify | 2 |
| 8 | `api/routes/telemetry.py` | Modify | 2 |
| 9 | `collector/letsmesh_normalizer.py` | Modify | 3 |
| 10 | `collector/handlers/message.py` | Modify | 4 |
| 11 | `collector/handlers/advertisement.py` | Modify | 4 |
| 12 | `collector/handlers/trace.py` | Modify | 4 |
| 13 | `collector/handlers/telemetry.py` | Modify | 4 |
| 14 | `web/static/js/spa/components.js` | Modify | 5 |
| 15 | `web/static/css/app.css` | Modify | 5 |
| 16 | `web/static/js/spa/pages/messages.js` | Modify | 6 |
| 17 | `web/static/js/spa/pages/advertisements.js` | Modify | 6 |
| 18 | `SCHEMAS.md` | Modify | 8 |
| 19 | `AGENTS.md` | Modify | 8 |
---
## Source Files Reference
| File | Key Locations |
|------|--------------|
| `common/models/event_observer.py` | `EventObserver` model, `add_event_observer()` helper |
| `common/hash_utils.py` | Hash computation for deduplication |
| `common/schemas/messages.py` | `ObserverInfo`, `MessageRead`, `AdvertisementRead`, `TracePathRead`, `TelemetryRead` |
| `collector/letsmesh_normalizer.py` | `_build_letsmesh_advertisement_payload()` (line 544), `_build_letsmesh_message_payload()` (line 84) |
| `collector/handlers/advertisement.py` | 3x `add_event_observer()` at lines 120, 182, 203 |
| `collector/handlers/message.py` | 3x `add_event_observer()` at lines 124, 158, 178 |
| `collector/handlers/trace.py` | 3x `add_event_observer()` at lines 74, 106, 126 |
| `collector/handlers/telemetry.py` | 3x `add_event_observer()` at lines 87, 133, 154 |
| `api/routes/messages.py` | `_fetch_observers_for_events()` at line 28 |
| `api/routes/advertisements.py` | `_fetch_observers_for_events()` at line 42 |
| `api/routes/trace_paths.py` | Returns `[]` for observers |
| `api/routes/telemetry.py` | Returns `[]` for observers |
| `web/static/js/spa/components.js` | Dead `receiverIcons()` at line 444 |
| `web/static/js/spa/pages/messages.js` | Observer rendering at lines 206-216 (mobile), 255-267 (desktop) |
| `web/static/js/spa/pages/advertisements.js` | Observer rendering at lines 99-109 (mobile), 135-147 (desktop) |
@@ -0,0 +1,127 @@
# Observer Detail Rows — Task Checklist
**Plan:** `docs/plans/20260426-1137-improve-snr-path-visibility/plan.md`
**Status:** Complete
## Review Notes
> **Plan accuracy issue found:** Phase 4 changes `message.py` handler to read only `payload.get("snr")` (lowercase), but the LetsMesh **message** normalizer (`letsmesh_normalizer.py:160`) still outputs `normalized_payload["SNR"]` (uppercase). Phase 3 only fixes the **advertisement** normalizer. An additional task (Phase 3.5) is included below to normalize the message normalizer output to lowercase `"snr"`, consistent with Decision #2.
---
## Phase 1: Database Schema
- [x] **1.1** Add `path_len: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)` to `EventObserver` model in `src/meshcore_hub/common/models/event_observer.py`
- [x] **1.2** Add `path_len: Optional[int] = None` parameter to `add_event_observer()` signature
- [x] **1.3** Include `path_len=path_len` in `sqlite_insert().values()` call
- [x] **1.4** Generate Alembic migration: `meshcore-hub db revision --autogenerate -m "add path_len to event_observers"`
- [x] **1.5** Review and adjust migration file
- [x] **1.6** Run migration: `meshcore-hub db upgrade`
## Phase 2: API Utility & Schema
- [x] **2.1** Add `path_len: Optional[int] = Field(default=None, description="Hop count at this observer")` to `ObserverInfo` schema in `src/meshcore_hub/common/schemas/messages.py`
- [x] **2.2** Create `src/meshcore_hub/api/observer_utils.py` — move shared `_fetch_observers_for_events()` function
- [x] **2.3** Update shared query to also select `EventObserver.path_len`
- [x] **2.4** Include `path_len=row.path_len` in `ObserverInfo()` construction within the shared function
- [x] **2.5** Update `api/routes/messages.py` — remove local `_fetch_observers_for_events`, import from `observer_utils`; also remove local `_get_tag_name` if it becomes unused
- [x] **2.6** Update `api/routes/advertisements.py` — same as 2.5
- [x] **2.7** Update `api/routes/trace_paths.py` — import and call `_fetch_observers_for_events(session, "trace", event_hashes)` for both list and detail endpoints; include `"observers"` key in response dicts
- [x] **2.8** Update `api/routes/telemetry.py` — same pattern as 2.7 with `"telemetry"` event type
## Phase 3: LetsMesh Normalizer — Advertisement SNR & Path
- [x] **3.1** In `src/meshcore_hub/collector/letsmesh_normalizer.py`, in `_build_letsmesh_advertisement_payload()`, add SNR extraction after `normalized_payload` dict initialization (~line 574): `snr = self._parse_float(payload.get("SNR"))`, fallback to `payload.get("snr")`, store as `normalized_payload["snr"]` (lowercase)
- [x] **3.2** Add `path_len = self._parse_path_length(payload.get("path"))` extraction, store as `normalized_payload["path_len"]`
- [x] **3.3** Return the updated `normalized_payload` (already returned at line 627)
## Phase 3.5: LetsMesh Normalizer — Message SNR Casing Fix
> **Accuracy fix:** Not in original plan. Required to prevent SNR data loss for LetsMesh-routed messages after Phase 4 removes the uppercase fallback.
- [x] **3.5.1** In `src/meshcore_hub/collector/letsmesh_normalizer.py`, change line 160: `normalized_payload["SNR"]``normalized_payload["snr"]` (lowercase, per Decision #2)
## Phase 4: Collector Handlers
- [x] **4.1** `src/meshcore_hub/collector/handlers/message.py` — change line 78: `payload.get("SNR") or payload.get("snr")``payload.get("snr")`
- [x] **4.2** `message.py` — pass `path_len=path_len` to all 3 `add_event_observer()` call sites (lines 124, 158, 178)
- [x] **4.3** `src/meshcore_hub/collector/handlers/advertisement.py` — add extraction: `snr = payload.get("snr")` and `path_len = payload.get("path_len")`
- [x] **4.4** `advertisement.py` — pass `snr=snr, path_len=path_len` to all 3 `add_event_observer()` call sites (lines 120, 182, 203); replace current `snr=None`
- [x] **4.5** `src/meshcore_hub/collector/handlers/trace.py` — add extraction: `snr = payload.get("snr")`
- [x] **4.6** `trace.py` — pass `snr=snr, path_len=path_len` to all 3 `add_event_observer()` call sites (lines 74, 106, 126); replace current `snr=None`
- [x] **4.7** `src/meshcore_hub/collector/handlers/telemetry.py` — add extraction: `snr = payload.get("snr")` and `path_len = payload.get("path_len")`
- [x] **4.8** `telemetry.py` — pass `snr=snr, path_len=path_len` to all 3 `add_event_observer()` call sites (lines 87, 133, 154); replace current `snr=None`
## Phase 5: Frontend Components
- [x] **5.1** Remove dead `receiverIcons()` function from `src/meshcore_hub/web/static/js/spa/components.js` (lines 444-452)
- [x] **5.2** Add `observerDetailRow(observers, eventProperties)` component to `components.js`:
- Renders expandable sub-table with columns: Observer (name/link), SNR (formatted dB), Path (formatted hops), Received (relative time)
- `eventProperties` param for event-level context (e.g., trace `snr_values`)
- Toggle helper: click event row to show/hide `.observer-detail` row
- [x] **5.3** Add `observerIcons(observers)` component — count badge with tooltip listing observer names
- [x] **5.4** Add `.observer-detail` expandable row styles to `src/meshcore_hub/web/static/css/app.css`:
- Indented, compact sub-table styling
- CSS transition for smooth expand/collapse (max-height animation)
- Responsive: desktop table vs mobile card layout
## Phase 6: Frontend Pages
- [x] **6.1** Update `src/meshcore_hub/web/static/js/spa/pages/messages.js`:
- Replace satellite dish icon rendering with observer count badge (clickable to expand)
- Add expandable detail row showing per-observer: name, SNR, path_len, observed_at
- Update both mobile card view (~line 206) and desktop table view (~line 255)
- [x] **6.2** Update `src/meshcore_hub/web/static/js/spa/pages/advertisements.js`:
- Same pattern as messages
- Update both mobile card view (~line 99) and desktop table view (~line 135)
## Phase 7: Tests
- [x] **7.1** `tests/test_collector/test_letsmesh_normalizer.py` — test type 4 packet extracts `snr` (lowercase) and `path_len` from envelope
- [x] **7.2** `tests/test_collector/test_letsmesh_normalizer.py` — test both `"SNR"` and `"snr"` input casing normalizes to lowercase `"snr"` output
- [x] **7.3** `tests/test_collector/test_letsmesh_normalizer.py` — test message payload also outputs lowercase `"snr"` (verifies Phase 3.5 fix)
- [x] **7.4** `tests/test_collector/test_handlers/test_advertisement.py` — test handler with `snr` and `path_len` in payload → stored in `event_observers`
- [x] **7.5** `tests/test_collector/test_handlers/test_message.py` — test handler passes `path_len` to `add_event_observer()`
- [x] **7.6** `tests/test_collector/test_handlers/test_message.py` — fix casing: `"SNR": 15.5``"snr": 15.5` (line 21) and `"SNR": 8.5``"snr": 8.5` (line 102)
- [x] **7.7** `tests/test_collector/test_handlers/test_trace.py` — test handler with envelope `snr`/`path_len` → stored in `event_observers`
- [x] **7.8** `tests/test_collector/test_handlers/test_telemetry.py` — test handler with envelope `snr`/`path_len` → stored in `event_observers`
- [x] **7.9** `tests/test_common/test_models.py` — test `add_event_observer()` accepts and stores `path_len`
- [x] **7.10** `tests/test_common/test_models.py` — test backwards compatibility (`path_len=None` is default)
- [x] **7.11** `tests/test_api/test_trace_paths.py` — test `observers` list is populated (query returns data)
- [x] **7.12** `tests/test_api/test_telemetry.py` — test `observers` list is populated (query returns data)
- [x] **7.13** Run targeted tests: `pytest tests/test_collector/ -v`
- [x] **7.14** Run targeted tests: `pytest tests/test_api/ -v`
- [x] **7.15** Run targeted tests: `pytest tests/test_common/ -v`
- [x] **7.16** Run quality checks: `pre-commit run --all-files`
## Phase 8: Documentation
- [x] **8.1** Update `SCHEMAS.md` — document that `SNR` and `path` are LetsMesh envelope fields available on all packet types
- [x] **8.2** Update `SCHEMAS.md` — update `ObserverInfo` description to include `path_len`
- [x] **8.3** Update `AGENTS.md` — update `event_observers` table description to include `path_len` column
---
## File Change Summary
| # | File | Action | Phase(s) |
|---|------|--------|----------|
| 1 | `common/models/event_observer.py` | Modify | 1 |
| 2 | `alembic/versions/*.py` | Create | 1 |
| 3 | `common/schemas/messages.py` | Modify | 2 |
| 4 | `api/observer_utils.py` | Create | 2 |
| 5 | `api/routes/messages.py` | Modify | 2 |
| 6 | `api/routes/advertisements.py` | Modify | 2 |
| 7 | `api/routes/trace_paths.py` | Modify | 2 |
| 8 | `api/routes/telemetry.py` | Modify | 2 |
| 9 | `collector/letsmesh_normalizer.py` | Modify | 3, 3.5 |
| 10 | `collector/handlers/message.py` | Modify | 4 |
| 11 | `collector/handlers/advertisement.py` | Modify | 4 |
| 12 | `collector/handlers/trace.py` | Modify | 4 |
| 13 | `collector/handlers/telemetry.py` | Modify | 4 |
| 14 | `web/static/js/spa/components.js` | Modify | 5 |
| 15 | `web/static/css/app.css` | Modify | 5 |
| 16 | `web/static/js/spa/pages/messages.js` | Modify | 6 |
| 17 | `web/static/js/spa/pages/advertisements.js` | Modify | 6 |
| 18 | `SCHEMAS.md` | Modify | 8 |
| 19 | `AGENTS.md` | Modify | 8 |
+75
View File
@@ -0,0 +1,75 @@
"""Shared utilities for fetching event observer data."""
from sqlalchemy import select
from meshcore_hub.api.dependencies import DbSession
from meshcore_hub.common.models import EventObserver, Node, NodeTag
from meshcore_hub.common.schemas.messages import ObserverInfo
def fetch_observers_for_events(
session: DbSession,
event_type: str,
event_hashes: list[str],
) -> dict[str, list[ObserverInfo]]:
"""Fetch observer info for a list of events by their hashes.
Args:
session: Database session
event_type: Type of event ('message', 'advertisement', etc.)
event_hashes: List of event hashes to fetch observers for
Returns:
Dict mapping event_hash to list of ObserverInfo objects
"""
if not event_hashes:
return {}
query = (
select(
EventObserver.event_hash,
EventObserver.snr,
EventObserver.path_len,
EventObserver.observed_at,
Node.id.label("node_id"),
Node.public_key,
Node.name,
)
.join(Node, EventObserver.observer_node_id == Node.id)
.where(EventObserver.event_type == event_type)
.where(EventObserver.event_hash.in_(event_hashes))
.order_by(EventObserver.observed_at)
)
results = session.execute(query).all()
observers_by_hash: dict[str, list[ObserverInfo]] = {}
node_ids = [r.node_id for r in results]
tag_names: dict[str, str] = {}
if node_ids:
tag_query = (
select(NodeTag.node_id, NodeTag.value)
.where(NodeTag.node_id.in_(node_ids))
.where(NodeTag.key == "name")
)
for node_id, value in session.execute(tag_query).all():
tag_names[node_id] = value
for row in results:
if row.event_hash not in observers_by_hash:
observers_by_hash[row.event_hash] = []
observers_by_hash[row.event_hash].append(
ObserverInfo(
node_id=row.node_id,
public_key=row.public_key,
name=row.name,
tag_name=tag_names.get(row.node_id),
snr=row.snr,
path_len=row.path_len,
observed_at=row.observed_at,
)
)
return observers_by_hash
+4 -60
View File
@@ -9,11 +9,11 @@ from sqlalchemy.orm import aliased, selectinload
from meshcore_hub.api.auth import RequireRead
from meshcore_hub.api.dependencies import DbSession
from meshcore_hub.common.models import Advertisement, EventObserver, Node, NodeTag
from meshcore_hub.api.observer_utils import fetch_observers_for_events
from meshcore_hub.common.models import Advertisement, Node, NodeTag
from meshcore_hub.common.schemas.messages import (
AdvertisementList,
AdvertisementRead,
ObserverInfo,
)
router = APIRouter()
@@ -39,62 +39,6 @@ def _get_tag_description(node: Optional[Node]) -> Optional[str]:
return None
def _fetch_observers_for_events(
session: DbSession,
event_type: str,
event_hashes: list[str],
) -> dict[str, list[ObserverInfo]]:
"""Fetch receiver info for a list of events by their hashes."""
if not event_hashes:
return {}
query = (
select(
EventObserver.event_hash,
EventObserver.snr,
EventObserver.observed_at,
Node.id.label("node_id"),
Node.public_key,
Node.name,
)
.join(Node, EventObserver.observer_node_id == Node.id)
.where(EventObserver.event_type == event_type)
.where(EventObserver.event_hash.in_(event_hashes))
.order_by(EventObserver.observed_at)
)
results = session.execute(query).all()
observers_by_hash: dict[str, list[ObserverInfo]] = {}
node_ids = [r.node_id for r in results]
tag_names: dict[str, str] = {}
if node_ids:
tag_query = (
select(NodeTag.node_id, NodeTag.value)
.where(NodeTag.node_id.in_(node_ids))
.where(NodeTag.key == "name")
)
for node_id, value in session.execute(tag_query).all():
tag_names[node_id] = value
for row in results:
if row.event_hash not in observers_by_hash:
observers_by_hash[row.event_hash] = []
observers_by_hash[row.event_hash].append(
ObserverInfo(
node_id=row.node_id,
public_key=row.public_key,
name=row.name,
tag_name=tag_names.get(row.node_id),
snr=row.snr,
observed_at=row.observed_at,
)
)
return observers_by_hash
@router.get("", response_model=AdvertisementList)
async def list_advertisements(
_: RequireRead,
@@ -201,7 +145,7 @@ async def list_advertisements(
# Fetch all observers for these advertisements
event_hashes = [r[0].event_hash for r in results if r[0].event_hash]
observers_by_hash = _fetch_observers_for_events(
observers_by_hash = fetch_observers_for_events(
session, "advertisement", event_hashes
)
@@ -290,7 +234,7 @@ async def get_advertisement(
# Fetch observers for this advertisement
observers = []
if adv.event_hash:
observers_by_hash = _fetch_observers_for_events(
observers_by_hash = fetch_observers_for_events(
session, "advertisement", [adv.event_hash]
)
observers = observers_by_hash.get(adv.event_hash, [])
+5 -73
View File
@@ -9,8 +9,9 @@ from sqlalchemy.orm import aliased, selectinload
from meshcore_hub.api.auth import RequireRead
from meshcore_hub.api.dependencies import DbSession
from meshcore_hub.common.models import EventObserver, Message, Node, NodeTag
from meshcore_hub.common.schemas.messages import MessageList, MessageRead, ObserverInfo
from meshcore_hub.api.observer_utils import fetch_observers_for_events
from meshcore_hub.common.models import Message, Node, NodeTag
from meshcore_hub.common.schemas.messages import MessageList, MessageRead
router = APIRouter()
@@ -25,75 +26,6 @@ def _get_tag_name(node: Optional[Node]) -> Optional[str]:
return None
def _fetch_observers_for_events(
session: DbSession,
event_type: str,
event_hashes: list[str],
) -> dict[str, list[ObserverInfo]]:
"""Fetch receiver info for a list of events by their hashes.
Args:
session: Database session
event_type: Type of event ('message', 'advertisement', etc.)
event_hashes: List of event hashes to fetch observers for
Returns:
Dict mapping event_hash to list of ObserverInfo objects
"""
if not event_hashes:
return {}
# Query event_observers with receiver node info
query = (
select(
EventObserver.event_hash,
EventObserver.snr,
EventObserver.observed_at,
Node.id.label("node_id"),
Node.public_key,
Node.name,
)
.join(Node, EventObserver.observer_node_id == Node.id)
.where(EventObserver.event_type == event_type)
.where(EventObserver.event_hash.in_(event_hashes))
.order_by(EventObserver.observed_at)
)
results = session.execute(query).all()
# Group by event_hash
observers_by_hash: dict[str, list[ObserverInfo]] = {}
# Get tag names for receiver nodes
node_ids = [r.node_id for r in results]
tag_names: dict[str, str] = {}
if node_ids:
tag_query = (
select(NodeTag.node_id, NodeTag.value)
.where(NodeTag.node_id.in_(node_ids))
.where(NodeTag.key == "name")
)
for node_id, value in session.execute(tag_query).all():
tag_names[node_id] = value
for row in results:
if row.event_hash not in observers_by_hash:
observers_by_hash[row.event_hash] = []
observers_by_hash[row.event_hash].append(
ObserverInfo(
node_id=row.node_id,
public_key=row.public_key,
name=row.name,
tag_name=tag_names.get(row.node_id),
snr=row.snr,
observed_at=row.observed_at,
)
)
return observers_by_hash
@router.get("", response_model=MessageList)
async def list_messages(
_: RequireRead,
@@ -197,7 +129,7 @@ async def list_messages(
# Fetch all observers for these messages
event_hashes = [r[0].event_hash for r in results if r[0].event_hash]
observers_by_hash = _fetch_observers_for_events(session, "message", event_hashes)
observers_by_hash = fetch_observers_for_events(session, "message", event_hashes)
# Build response with sender info and observed_by
items = []
@@ -269,7 +201,7 @@ async def get_message(
# Fetch observers for this message
observers = []
if message.event_hash:
observers_by_hash = _fetch_observers_for_events(
observers_by_hash = fetch_observers_for_events(
session, "message", [message.event_hash]
)
observers = observers_by_hash.get(message.event_hash, [])
+17
View File
@@ -9,6 +9,7 @@ from sqlalchemy.orm import aliased
from meshcore_hub.api.auth import RequireRead
from meshcore_hub.api.dependencies import DbSession
from meshcore_hub.api.observer_utils import fetch_observers_for_events
from meshcore_hub.common.models import Node, Telemetry
from meshcore_hub.common.schemas.messages import TelemetryList, TelemetryRead
@@ -59,6 +60,10 @@ async def list_telemetry(
# Execute
results = session.execute(query).all()
# Fetch observers for these telemetry records
event_hashes = [tel.event_hash for tel, _ in results if tel.event_hash]
observers_by_hash = fetch_observers_for_events(session, "telemetry", event_hashes)
# Build response with observed_by
items = []
for tel, observer_pk in results:
@@ -71,6 +76,9 @@ async def list_telemetry(
"parsed_data": tel.parsed_data,
"received_at": tel.received_at,
"created_at": tel.created_at,
"observers": (
observers_by_hash.get(tel.event_hash, []) if tel.event_hash else []
),
}
items.append(TelemetryRead(**data))
@@ -101,6 +109,14 @@ async def get_telemetry(
raise HTTPException(status_code=404, detail="Telemetry record not found")
tel, observer_pk = result
observers = []
if tel.event_hash:
observers_by_hash = fetch_observers_for_events(
session, "telemetry", [tel.event_hash]
)
observers = observers_by_hash.get(tel.event_hash, [])
data = {
"id": tel.id,
"observer_node_id": tel.observer_node_id,
@@ -110,5 +126,6 @@ async def get_telemetry(
"parsed_data": tel.parsed_data,
"received_at": tel.received_at,
"created_at": tel.created_at,
"observers": observers,
}
return TelemetryRead(**data)
@@ -9,6 +9,7 @@ from sqlalchemy.orm import aliased
from meshcore_hub.api.auth import RequireRead
from meshcore_hub.api.dependencies import DbSession
from meshcore_hub.api.observer_utils import fetch_observers_for_events
from meshcore_hub.common.models import Node, TracePath
from meshcore_hub.common.schemas.messages import TracePathList, TracePathRead
@@ -55,6 +56,10 @@ async def list_trace_paths(
# Execute
results = session.execute(query).all()
# Fetch observers for these trace paths
event_hashes = [tp.event_hash for tp, _ in results if tp.event_hash]
observers_by_hash = fetch_observers_for_events(session, "trace", event_hashes)
# Build response with observed_by
items = []
for tp, observer_pk in results:
@@ -71,6 +76,9 @@ async def list_trace_paths(
"hop_count": tp.hop_count,
"received_at": tp.received_at,
"created_at": tp.created_at,
"observers": (
observers_by_hash.get(tp.event_hash, []) if tp.event_hash else []
),
}
items.append(TracePathRead(**data))
@@ -101,6 +109,14 @@ async def get_trace_path(
raise HTTPException(status_code=404, detail="Trace path not found")
tp, observer_pk = result
observers = []
if tp.event_hash:
observers_by_hash = fetch_observers_for_events(
session, "trace", [tp.event_hash]
)
observers = observers_by_hash.get(tp.event_hash, [])
data = {
"id": tp.id,
"observer_node_id": tp.observer_node_id,
@@ -114,5 +130,6 @@ async def get_trace_path(
"hop_count": tp.hop_count,
"received_at": tp.received_at,
"created_at": tp.created_at,
"observers": observers,
}
return TracePathRead(**data)
@@ -72,6 +72,9 @@ def handle_advertisement(
lon = _coerce_float(lon)
now = datetime.now(timezone.utc)
snr = payload.get("snr")
path_len = payload.get("path_len")
# Compute event hash for deduplication (30-second time bucket)
event_hash = compute_advertisement_hash(
public_key=adv_public_key,
@@ -122,7 +125,8 @@ def handle_advertisement(
event_type="advertisement",
event_hash=event_hash,
observer_node_id=receiver_node.id,
snr=None, # Advertisements don't have SNR
snr=snr,
path_len=path_len,
observed_at=now,
)
if added:
@@ -184,7 +188,8 @@ def handle_advertisement(
event_type="advertisement",
event_hash=event_hash,
observer_node_id=receiver_node.id,
snr=None,
snr=snr,
path_len=path_len,
observed_at=now,
)
@@ -205,7 +210,8 @@ def handle_advertisement(
event_type="advertisement",
event_hash=event_hash,
observer_node_id=receiver_node.id,
snr=None,
snr=snr,
path_len=path_len,
observed_at=now,
)
return
@@ -75,7 +75,7 @@ def _handle_message(
path_len = payload.get("path_len")
txt_type = payload.get("txt_type")
signature = payload.get("signature")
snr = payload.get("SNR") or payload.get("snr")
snr = payload.get("snr")
# Parse sender timestamp
sender_ts = payload.get("sender_timestamp")
@@ -127,6 +127,7 @@ def _handle_message(
event_hash=event_hash,
observer_node_id=receiver_node.id,
snr=snr,
path_len=path_len,
observed_at=now,
)
if added:
@@ -161,6 +162,7 @@ def _handle_message(
event_hash=event_hash,
observer_node_id=receiver_node.id,
snr=snr,
path_len=path_len,
observed_at=now,
)
@@ -181,6 +183,7 @@ def _handle_message(
event_hash=event_hash,
observer_node_id=receiver_node.id,
snr=snr,
path_len=path_len,
observed_at=now,
)
return
@@ -37,6 +37,8 @@ def handle_telemetry(
lpp_data = payload.get("lpp_data")
parsed_data = payload.get("parsed_data")
snr = payload.get("snr")
path_len = payload.get("path_len")
# Convert lpp_data to bytes if it's a string or list
lpp_bytes = None
@@ -89,7 +91,8 @@ def handle_telemetry(
event_type="telemetry",
event_hash=event_hash,
observer_node_id=receiver_node.id,
snr=None,
snr=snr,
path_len=path_len,
observed_at=now,
)
if added:
@@ -135,7 +138,8 @@ def handle_telemetry(
event_type="telemetry",
event_hash=event_hash,
observer_node_id=receiver_node.id,
snr=None,
snr=snr,
path_len=path_len,
observed_at=now,
)
@@ -156,7 +160,8 @@ def handle_telemetry(
event_type="telemetry",
event_hash=event_hash,
observer_node_id=receiver_node.id,
snr=None,
snr=snr,
path_len=path_len,
observed_at=now,
)
return
+7 -3
View File
@@ -41,6 +41,7 @@ def handle_trace_data(
path_hashes = payload.get("path_hashes")
snr_values = payload.get("snr_values")
hop_count = payload.get("hop_count")
snr = payload.get("snr")
# Compute event hash for deduplication (initiator_tag is unique per trace)
event_hash = compute_trace_hash(initiator_tag=initiator_tag)
@@ -76,7 +77,8 @@ def handle_trace_data(
event_type="trace",
event_hash=event_hash,
observer_node_id=receiver_node.id,
snr=None, # Trace events don't have a single SNR value
snr=snr,
path_len=path_len,
observed_at=now,
)
if added:
@@ -108,7 +110,8 @@ def handle_trace_data(
event_type="trace",
event_hash=event_hash,
observer_node_id=receiver_node.id,
snr=None,
snr=snr,
path_len=path_len,
observed_at=now,
)
@@ -128,7 +131,8 @@ def handle_trace_data(
event_type="trace",
event_hash=event_hash,
observer_node_id=receiver_node.id,
snr=None,
snr=snr,
path_len=path_len,
observed_at=now,
)
return
@@ -157,7 +157,8 @@ class LetsMeshNormalizer:
if snr is None:
snr = self._parse_float(payload.get("snr"))
if snr is not None:
normalized_payload["SNR"] = snr
normalized_payload["snr"] = snr
normalized_payload.pop("SNR", None)
decoded_sender = self._extract_letsmesh_decoder_sender(
decoded_packet,
@@ -575,6 +576,16 @@ class LetsMeshNormalizer:
"public_key": public_key,
}
snr = self._parse_float(payload.get("SNR"))
if snr is None:
snr = self._parse_float(payload.get("snr"))
if snr is not None:
normalized_payload["snr"] = snr
path_len = self._parse_path_length(payload.get("path"))
if path_len is not None:
normalized_payload["path_len"] = path_len
app_data = decoded_payload.get("appData")
if isinstance(app_data, dict):
name = app_data.get("name")
@@ -4,7 +4,15 @@ from datetime import datetime
from typing import TYPE_CHECKING, Optional
from uuid import uuid4
from sqlalchemy import DateTime, Float, ForeignKey, Index, String, UniqueConstraint
from sqlalchemy import (
DateTime,
Float,
ForeignKey,
Integer,
Index,
String,
UniqueConstraint,
)
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
from sqlalchemy.orm import Mapped, Session, mapped_column, relationship
@@ -27,6 +35,7 @@ class EventObserver(Base, UUIDMixin, TimestampMixin):
event_hash: Hash identifying the unique event (links to event tables)
observer_node_id: FK to the node that observed this event
snr: Signal-to-noise ratio at this observer (if available)
path_len: Hop count at this observer (if available)
observed_at: When this specific observer captured the event
created_at: Record creation timestamp
updated_at: Record update timestamp
@@ -53,6 +62,10 @@ class EventObserver(Base, UUIDMixin, TimestampMixin):
Float,
nullable=True,
)
path_len: Mapped[Optional[int]] = mapped_column(
Integer,
nullable=True,
)
observed_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
@@ -85,6 +98,7 @@ def add_event_observer(
event_hash: str,
observer_node_id: str,
snr: Optional[float] = None,
path_len: Optional[int] = None,
observed_at: Optional[datetime] = None,
) -> bool:
"""Add an observer to an event, handling duplicates gracefully.
@@ -97,6 +111,7 @@ def add_event_observer(
event_hash: Hash identifying the unique event
observer_node_id: UUID of the observer node
snr: Signal-to-noise ratio at this observer (optional)
path_len: Hop count at this observer (optional)
observed_at: When this observer captured the event (defaults to now)
Returns:
@@ -114,6 +129,7 @@ def add_event_observer(
event_hash=event_hash,
observer_node_id=observer_node_id,
snr=snr,
path_len=path_len,
observed_at=now,
created_at=now,
updated_at=now,
@@ -16,6 +16,9 @@ class ObserverInfo(BaseModel):
snr: Optional[float] = Field(
default=None, description="Signal-to-noise ratio at this observer"
)
path_len: Optional[int] = Field(
default=None, description="Hop count at this observer"
)
observed_at: datetime = Field(
..., description="When this observer captured the event"
)
+38
View File
@@ -348,3 +348,41 @@
#header-map .leaflet-control {
z-index: auto !important;
}
/* ==========================================================================
Observer Detail Rows
Expandable sub-table showing per-observer signal/path data.
========================================================================== */
.observer-detail .observer-detail-content {
padding: 0.5rem 1rem 0.5rem 2rem;
}
.observer-detail table {
margin: 0;
}
.observer-badge {
transition: background-color 0.15s ease;
}
.observer-badge:hover {
background-color: color-mix(in oklch, var(--color-base-content) 15%, transparent);
}
.observer-badge-group {
display: inline-flex;
align-items: center;
gap: 0.25rem;
}
/* Mobile card observer details */
.observer-detail-card {
padding: 0.5rem 0.75rem;
margin-top: 0.25rem;
}
.observer-detail-card table {
margin: 0;
font-size: 0.8rem;
}
@@ -437,18 +437,83 @@ export function timezoneIndicator() {
}
/**
* Render receiver node icons with tooltips.
* @param {Array} receivers
* Render an observer count badge with tooltip listing observer names.
* @param {Array} observers - Array of observer objects
* @returns {TemplateResult|nothing}
*/
export function receiverIcons(receivers) {
if (!receivers || receivers.length === 0) return nothing;
return html`${receivers.map(r => {
const name = r.receiver_node_name || truncateKey(r.receiver_node_public_key || '', 8);
const time = formatRelativeTime(r.received_at);
const tooltip = time ? `${name} (${time})` : name;
return html`<span class="cursor-help" title=${tooltip}>\u{1F4E1}</span>`;
})}`;
export function observerIcons(observers) {
if (!observers || observers.length === 0) return nothing;
const names = observers.map(o => o.tag_name || o.name || truncateKey(o.public_key, 8));
const tooltip = names.join(', ');
return html`<span class="observer-badge-group">\u{1F4E1}<span class="badge badge-sm badge-ghost cursor-help observer-badge" title=${tooltip}>${observers.length}</span></span>`;
}
/**
* Render an expandable observer detail row.
* Shows per-observer: name, SNR, path_len, observed_at.
* @param {Array} observers - Array of observer objects
* @param {Object} [eventProperties] - Event-level context (unused, for future use)
* @returns {TemplateResult|nothing}
*/
export function observerDetailRow(observers, eventProperties, options = {}) {
if (!observers || observers.length === 0) return nothing;
const showPath = !options.hidePath;
return html`
<tr class="observer-detail hidden">
<td colspan="100" class="p-0">
<div class="observer-detail-content">
<table class="table table-xs w-full">
<thead>
<tr>
<th>Observer</th>
<th>SNR</th>
${showPath ? html`<th>Path</th>` : nothing}
<th>Received</th>
</tr>
</thead>
<tbody>
${observers.map(o => {
const displayName = o.tag_name || o.name || truncateKey(o.public_key, 12);
const snrDisplay = o.snr != null ? `${Number(o.snr).toFixed(1)} dB` : '\u2014';
const pathDisplay = o.path_len != null ? `${o.path_len} hop${o.path_len !== 1 ? 's' : ''}` : '\u2014';
const timeDisplay = formatRelativeTime(o.observed_at);
return html`
<tr>
<td>\u{1F4E1} <a href="/nodes/${o.public_key}" class="link link-hover">${displayName}</a></td>
<td>${snrDisplay}</td>
${showPath ? html`<td>${pathDisplay}</td>` : nothing}
<td><span title=${formatDateTime(o.observed_at)}>${timeDisplay}</span></td>
</tr>
`;
})}
</tbody>
</table>
</div>
</td>
</tr>
`;
}
/**
* Toggle observer detail row visibility when clicking an event row.
* @param {Event} event - Click event
*/
export function toggleObserverDetail(event) {
const row = event.currentTarget;
const detailRow = row.nextElementSibling;
if (detailRow && detailRow.classList.contains('observer-detail')) {
detailRow.classList.toggle('hidden');
}
}
export function toggleCardObserverDetail(event) {
event.stopPropagation();
event.preventDefault();
const card = event.currentTarget.closest('.card');
if (card) {
const detail = card.querySelector('.observer-detail-card');
if (detail) detail.classList.toggle('hidden');
}
}
// --- Form Helpers ---
@@ -1,9 +1,10 @@
import { apiGet } from '../api.js';
import {
html, litRender, nothing, t,
getConfig, formatDateTime, formatDateTimeShort,
getConfig, formatDateTime, formatDateTimeShort, formatRelativeTime,
truncateKey, errorAlert,
pagination, createFilterHandler, autoSubmit, submitOnEnter, copyToClipboard, renderNodeDisplay
pagination, createFilterHandler, autoSubmit, submitOnEnter, copyToClipboard, renderNodeDisplay,
observerIcons, observerDetailRow, toggleObserverDetail, toggleCardObserverDetail
} from '../components.js';
import { createAutoRefresh } from '../auto-refresh.js';
@@ -98,15 +99,9 @@ ${content}`, container);
const adDescription = ad.node_tag_description;
let receiversBlock = nothing;
if (ad.observers && ad.observers.length >= 1) {
receiversBlock = html`<div class="flex gap-0.5 justify-end mt-1">
${ad.observers.map(recv => {
const recvName = recv.tag_name || recv.name || truncateKey(recv.public_key, 12);
return html`<span class="text-sm" title=${recvName}>\u{1F4E1}</span>`;
})}
</div>`;
receiversBlock = html`<span @click=${toggleCardObserverDetail} class="cursor-pointer">${observerIcons(ad.observers)}</span>`;
} else if (ad.observed_by) {
const recvTitle = ad.observer_tag_name || ad.observer_name || truncateKey(ad.observed_by, 12);
receiversBlock = html`<span class="text-sm" title=${recvTitle}>\u{1F4E1}</span>`;
receiversBlock = html`<span class="opacity-50 text-xs">\u{1F4E1}</span>`;
}
return html`<a href="/nodes/${ad.public_key}" class="card bg-base-100 shadow-sm block">
<div class="card-body p-3">
@@ -123,6 +118,25 @@ ${content}`, container);
${receiversBlock}
</div>
</div>
${ad.observers && ad.observers.length > 0 ? html`
<div class="observer-detail-card hidden mt-2">
<table class="table table-xs w-full">
<thead><tr><th>Observer</th><th>SNR</th><th>Received</th></tr></thead>
<tbody>
${ad.observers.map(o => {
const dn = o.tag_name || o.name || truncateKey(o.public_key, 12);
const snrD = o.snr != null ? `${Number(o.snr).toFixed(1)} dB` : '\u2014';
const timeD = formatRelativeTime(o.observed_at);
return html`<tr>
<td>\u{1F4E1} <a href="/nodes/${o.public_key}" class="link link-hover">${dn}</a></td>
<td>${snrD}</td>
<td><span title=${formatDateTime(o.observed_at)}>${timeD}</span></td>
</tr>`;
})}
</tbody>
</table>
</div>
` : nothing}
</div>
</a>`;
});
@@ -134,19 +148,13 @@ ${content}`, container);
const adDescription = ad.node_tag_description;
let receiversBlock;
if (ad.observers && ad.observers.length >= 1) {
receiversBlock = html`<div class="flex gap-1">
${ad.observers.map(recv => {
const recvName = recv.tag_name || recv.name || truncateKey(recv.public_key, 12);
return html`<a href="/nodes/${recv.public_key}" class="text-lg hover:opacity-70" title=${recvName}>\u{1F4E1}</a>`;
})}
</div>`;
receiversBlock = html`${observerIcons(ad.observers)}`;
} else if (ad.observed_by) {
const recvTitle = ad.observer_tag_name || ad.observer_name || truncateKey(ad.observed_by, 12);
receiversBlock = html`<a href="/nodes/${ad.observed_by}" class="text-lg hover:opacity-70" title=${recvTitle}>\u{1F4E1}</a>`;
receiversBlock = html`<span class="opacity-50">\u{1F4E1}</span>`;
} else {
receiversBlock = html`<span class="opacity-50">-</span>`;
}
return html`<tr class="hover">
return html`<tr class="hover cursor-pointer" @click=${toggleObserverDetail}>
<td>
<a href="/nodes/${ad.public_key}" class="link link-hover">
${renderNodeDisplay({
@@ -165,7 +173,7 @@ ${content}`, container);
</td>
<td class="text-sm whitespace-nowrap">${formatDateTime(ad.received_at)}</td>
<td>${receiversBlock}</td>
</tr>`;
</tr>${observerDetailRow(ad.observers || [], null, { hidePath: true })}`;
});
const paginationBlock = pagination(page, totalPages, '/advertisements', {
@@ -203,7 +211,7 @@ ${content}`, container);
<th>${t('entities.node')}</th>
<th>${t('common.public_key')}</th>
<th>${t('common.time')}</th>
<th>${t('common.receivers')}</th>
<th>${t('common.observers')}</th>
</tr>
</thead>
<tbody>
@@ -1,11 +1,12 @@
import { apiGet } from '../api.js';
import {
html, litRender, nothing, t,
getConfig, formatDateTime, formatDateTimeShort,
getConfig, formatDateTime, formatDateTimeShort, formatRelativeTime,
getChannelLabelsMap, resolveChannelLabel,
truncateKey, errorAlert,
pagination, timezoneIndicator,
createFilterHandler, autoSubmit, submitOnEnter
createFilterHandler, autoSubmit, submitOnEnter,
observerIcons, observerDetailRow, toggleObserverDetail, toggleCardObserverDetail
} from '../components.js';
import { createAutoRefresh } from '../auto-refresh.js';
@@ -205,15 +206,9 @@ ${content}`, container);
: sender;
let receiversBlock = nothing;
if (msg.observers && msg.observers.length >= 1) {
receiversBlock = html`<div class="flex gap-0.5">
${msg.observers.map(recv => {
const recvName = recv.tag_name || recv.name || truncateKey(recv.public_key, 12);
return html`<a href="/nodes/${recv.public_key}" class="text-sm hover:opacity-70" title=${recvName}>\u{1F4E1}</a>`;
})}
</div>`;
receiversBlock = html`<span @click=${toggleCardObserverDetail} class="cursor-pointer">${observerIcons(msg.observers)}</span>`;
} else if (msg.observed_by) {
const recvTitle = msg.observer_tag_name || msg.observer_name || truncateKey(msg.observed_by, 12);
receiversBlock = html`<a href="/nodes/${msg.observed_by}" class="text-sm hover:opacity-70" title=${recvTitle}>\u{1F4E1}</a>`;
receiversBlock = html`<span class="opacity-50 text-xs">\u{1F4E1}</span>`;
}
return html`<div class="card bg-base-100 shadow-sm">
<div class="card-body p-3">
@@ -236,6 +231,27 @@ ${content}`, container);
</div>
</div>
<p class="text-sm mt-2 break-words whitespace-pre-wrap">${displayMessage}</p>
${msg.observers && msg.observers.length > 0 ? html`
<div class="observer-detail-card hidden mt-2">
<table class="table table-xs w-full">
<thead><tr><th>Observer</th><th>SNR</th><th>Path</th><th>Received</th></tr></thead>
<tbody>
${msg.observers.map(o => {
const dn = o.tag_name || o.name || truncateKey(o.public_key, 12);
const snrD = o.snr != null ? `${Number(o.snr).toFixed(1)} dB` : '\u2014';
const pathD = o.path_len != null ? `${o.path_len} hop${o.path_len !== 1 ? 's' : ''}` : '\u2014';
const timeD = formatRelativeTime(o.observed_at);
return html`<tr>
<td>\u{1F4E1} <a href="/nodes/${o.public_key}" class="link link-hover">${dn}</a></td>
<td>${snrD}</td>
<td>${pathD}</td>
<td><span title=${formatDateTime(o.observed_at)}>${timeD}</span></td>
</tr>`;
})}
</tbody>
</table>
</div>
` : nothing}
</div>
</div>`;
});
@@ -254,19 +270,13 @@ ${content}`, container);
: sender;
let receiversBlock;
if (msg.observers && msg.observers.length >= 1) {
receiversBlock = html`<div class="flex gap-1">
${msg.observers.map(recv => {
const recvName = recv.tag_name || recv.name || truncateKey(recv.public_key, 12);
return html`<a href="/nodes/${recv.public_key}" class="text-lg hover:opacity-70" title=${recvName}>\u{1F4E1}</a>`;
})}
</div>`;
receiversBlock = html`${observerIcons(msg.observers)}`;
} else if (msg.observed_by) {
const recvTitle = msg.observer_tag_name || msg.observer_name || truncateKey(msg.observed_by, 12);
receiversBlock = html`<a href="/nodes/${msg.observed_by}" class="text-lg hover:opacity-70" title=${recvTitle}>\u{1F4E1}</a>`;
receiversBlock = html`<span class="opacity-50">\u{1F4E1}</span>`;
} else {
receiversBlock = html`<span class="opacity-50">-</span>`;
}
return html`<tr class="hover align-top">
return html`<tr class="hover cursor-pointer" @click=${toggleObserverDetail}>
<td class="text-lg" title=${typeTitle}>${typeIcon}</td>
<td class="text-sm whitespace-nowrap">${formatDateTime(msg.received_at)}</td>
<td class="text-sm whitespace-nowrap">
@@ -274,7 +284,7 @@ ${content}`, container);
</td>
<td class="break-words max-w-md" style="white-space: pre-wrap;">${displayMessage}</td>
<td>${receiversBlock}</td>
</tr>`;
</tr>${observerDetailRow(msg.observers || [])}`;
});
const paginationBlock = pagination(page, totalPages, '/messages', {
@@ -326,7 +336,7 @@ ${content}`, container);
<th>${t('common.time')}</th>
<th>${t('common.from')}</th>
<th>${t('entities.message')}</th>
<th>${t('common.receivers')}</th>
<th>${t('common.observers')}</th>
</tr>
</thead>
<tbody>
+1 -1
View File
@@ -94,7 +94,7 @@
"public_key": "Public Key",
"received": "Received",
"received_by": "Received By",
"receivers": "Receivers",
"observers": "Observers",
"from": "From",
"close": "close",
"unnamed": "Unnamed",
+58
View File
@@ -118,3 +118,61 @@ class TestListTelemetryFilters:
assert response.status_code == 200
data = response.json()
assert len(data["items"]) == 1
class TestTelemetryObservers:
"""Tests for observer data in telemetry API responses."""
def test_telemetry_observers_populated(self, client_no_auth, api_db_session):
"""Test that telemetry list returns observers with data."""
from meshcore_hub.common.hash_utils import compute_telemetry_hash
from meshcore_hub.common.models import EventObserver, Node, Telemetry
observer_node = Node(
public_key="z" * 64,
name="TelObserver",
first_seen=datetime.now(timezone.utc),
)
api_db_session.add(observer_node)
api_db_session.flush()
now = datetime.now(timezone.utc)
event_hash = compute_telemetry_hash(
node_public_key="tel123tel123tel123tel123tel123tel",
parsed_data={"temperature": 22.0},
received_at=now,
)
telemetry = Telemetry(
node_public_key="tel123tel123tel123tel123tel123tel",
parsed_data={"temperature": 22.0},
received_at=now,
observer_node_id=observer_node.id,
event_hash=event_hash,
)
api_db_session.add(telemetry)
api_db_session.flush()
ev_obs = EventObserver(
event_type="telemetry",
event_hash=event_hash,
observer_node_id=observer_node.id,
snr=8.5,
path_len=2,
observed_at=now,
)
api_db_session.add(ev_obs)
api_db_session.commit()
response = client_no_auth.get("/api/v1/telemetry")
assert response.status_code == 200
data = response.json()
items = data["items"]
assert len(items) >= 1
tel_item = next(
i
for i in items
if i["node_public_key"] == "tel123tel123tel123tel123tel123tel"
)
assert len(tel_item["observers"]) == 1
assert tel_item["observers"][0]["snr"] == 8.5
assert tel_item["observers"][0]["path_len"] == 2
+50
View File
@@ -153,3 +153,53 @@ class TestListTracePathsFilters:
assert response.status_code == 200
data = response.json()
assert len(data["items"]) == 1
class TestTracePathObservers:
"""Tests for observer data in trace path API responses."""
def test_trace_path_observers_populated(self, client_no_auth, api_db_session):
"""Test that trace path list returns observers with data."""
from meshcore_hub.common.hash_utils import compute_trace_hash
from meshcore_hub.common.models import EventObserver, Node, TracePath
observer_node = Node(
public_key="z" * 64,
name="TraceObserver",
first_seen=datetime.now(timezone.utc),
)
api_db_session.add(observer_node)
api_db_session.flush()
event_hash = compute_trace_hash(initiator_tag=44444)
trace = TracePath(
initiator_tag=44444,
path_hashes=["aa", "bb"],
hop_count=2,
received_at=datetime.now(timezone.utc),
observer_node_id=observer_node.id,
event_hash=event_hash,
)
api_db_session.add(trace)
api_db_session.flush()
ev_obs = EventObserver(
event_type="trace",
event_hash=event_hash,
observer_node_id=observer_node.id,
snr=12.0,
path_len=3,
observed_at=datetime.now(timezone.utc),
)
api_db_session.add(ev_obs)
api_db_session.commit()
response = client_no_auth.get("/api/v1/trace-paths")
assert response.status_code == 200
data = response.json()
items = data["items"]
assert len(items) >= 1
trace_item = next(i for i in items if i["initiator_tag"] == 44444)
assert len(trace_item["observers"]) == 1
assert trace_item["observers"][0]["snr"] == 12.0
assert trace_item["observers"][0]["path_len"] == 3
@@ -199,3 +199,20 @@ class TestHandleAdvertisement:
select(Node).where(Node.public_key == receiver_pk)
).scalar_one_or_none()
assert receiver is not None
def test_advertisement_stores_snr_and_path_len(self, db_manager, db_session):
"""Advertisement handler stores snr and path_len in event_observers."""
payload = {
"public_key": "a" * 64,
"name": "TestNode",
"adv_type": "chat",
"snr": 12.5,
"path_len": 3,
}
handle_advertisement("b" * 64, "advertisement", payload, db_manager)
observer = db_session.execute(select(EventObserver)).scalar_one_or_none()
assert observer is not None
assert observer.snr == 12.5
assert observer.path_len == 3
@@ -18,7 +18,7 @@ class TestHandleContactMessage:
"pubkey_prefix": "01ab2186c4d5",
"text": "Hello World!",
"path_len": 3,
"SNR": 15.5,
"snr": 15.5,
}
handle_contact_message("a" * 64, "contact_msg_recv", payload, db_manager)
@@ -99,7 +99,7 @@ class TestHandleChannelMessage:
"channel_idx": 4,
"text": "Channel broadcast",
"path_len": 10,
"SNR": 8.5,
"snr": 8.5,
}
handle_channel_message("a" * 64, "channel_msg_recv", payload, db_manager)
@@ -158,3 +158,19 @@ class TestHandleChannelMessage:
msg = db_session.execute(select(Message)).scalar_one()
assert msg.signature == "abcdef1234567890"
def test_message_handler_passes_path_len_to_observer(self, db_manager, db_session):
"""Message handler passes path_len to add_event_observer."""
payload = {
"pubkey_prefix": "01ab2186c4d5",
"text": "Path test",
"path_len": 5,
"snr": 10.0,
}
handle_contact_message("a" * 64, "contact_msg_recv", payload, db_manager)
observer = db_session.execute(select(EventObserver)).scalar_one_or_none()
assert observer is not None
assert observer.path_len == 5
assert observer.snr == 10.0
@@ -126,3 +126,18 @@ class TestHandleTelemetry:
telemetry = db_session.execute(select(Telemetry)).scalar_one()
assert telemetry.node_public_key == "b" * 64
def test_telemetry_stores_snr_and_path_len(self, db_manager, db_session):
"""Telemetry handler stores snr and path_len in event_observers."""
payload = {
"node_public_key": "b" * 64,
"parsed_data": {"temperature": 20.0},
"snr": 7.5,
"path_len": 2,
}
handle_telemetry("a" * 64, "telemetry_response", payload, db_manager)
observer = db_session.execute(select(EventObserver)).scalar_one()
assert observer.snr == 7.5
assert observer.path_len == 2
@@ -88,3 +88,18 @@ class TestHandleTraceData:
observers = db_session.execute(select(EventObserver)).scalars().all()
assert len(observers) == 1
assert observers[0].event_type == "trace"
def test_trace_stores_snr_and_path_len(self, db_manager, db_session):
"""Trace handler stores snr and path_len in event_observers."""
payload = {
"initiator_tag": 88888,
"path_len": 4,
"snr": 11.5,
"hop_count": 2,
}
handle_trace_data("a" * 64, "trace_data", payload, db_manager)
observer = db_session.execute(select(EventObserver)).scalar_one()
assert observer.snr == 11.5
assert observer.path_len == 4
@@ -1,4 +1,6 @@
"""Tests for LetsMesh normalizer _normalize_hash_list method."""
"""Tests for LetsMesh normalizer."""
from unittest.mock import MagicMock
from meshcore_hub.collector.letsmesh_normalizer import LetsMeshNormalizer
@@ -66,3 +68,102 @@ class TestNormalizeHashList:
"""Single-character strings are rejected (minimum is 2)."""
result = LetsMeshNormalizer._normalize_hash_list(["a", "4a"])
assert result == ["4A"]
class TestAdvertisementSnrAndPath:
"""Tests for SNR and path_len extraction in advertisement payloads."""
def _make_normalizer(self) -> LetsMeshNormalizer:
norm = LetsMeshNormalizer()
norm._letsmesh_decoder = MagicMock()
norm._include_test_channel = False
return norm
def _make_decoded_type4(self) -> dict:
return {
"payloadType": 4,
"payload": {
"decoded": {
"publicKey": "b" * 64,
},
},
}
def test_advertisement_extracts_snr_from_uppercase_key(self) -> None:
"""Advertisement payload with uppercase SNR normalizes to lowercase snr."""
norm = self._make_normalizer()
result = norm._build_letsmesh_advertisement_payload(
{"SNR": 12.5, "path": "91CBC3"},
decoded_packet=self._make_decoded_type4(),
)
assert result is not None
assert result["snr"] == 12.5
def test_advertisement_extracts_snr_from_lowercase_key(self) -> None:
"""Advertisement payload with lowercase snr extracts it."""
norm = self._make_normalizer()
result = norm._build_letsmesh_advertisement_payload(
{"snr": 9.0},
decoded_packet=self._make_decoded_type4(),
)
assert result is not None
assert result["snr"] == 9.0
def test_advertisement_extracts_path_len(self) -> None:
"""Advertisement payload with path extracts path_len."""
norm = self._make_normalizer()
result = norm._build_letsmesh_advertisement_payload(
{"path": "91CBC3"},
decoded_packet=self._make_decoded_type4(),
)
assert result is not None
assert result.get("path_len") is not None
def test_output_has_lowercase_snr_key_only(self) -> None:
"""Output contains lowercase snr key, never uppercase SNR."""
norm = self._make_normalizer()
result = norm._build_letsmesh_advertisement_payload(
{"SNR": 8.0},
decoded_packet=self._make_decoded_type4(),
)
assert result is not None
assert "snr" in result
assert "SNR" not in result
class TestMessageSnrCasing:
"""Tests for message payload SNR casing (lowercase output)."""
def _make_normalizer(self) -> LetsMeshNormalizer:
norm = LetsMeshNormalizer()
norm._letsmesh_decoder = MagicMock()
norm._include_test_channel = True
return norm
def _make_decoded_type5(self) -> dict:
return {
"payloadType": 5,
"payload": {
"decoded": {
"text": "hello channel",
"channel": {"hash": "A" * 24},
},
},
}
def test_message_outputs_lowercase_snr(self) -> None:
"""Message payload outputs lowercase snr key, not uppercase SNR."""
norm = self._make_normalizer()
result = norm._build_letsmesh_message_payload(
{
"packet_type": "5",
"hash": "ABCDEF1234",
"SNR": "12.5",
"path": "91CBC3",
},
decoded_packet=self._make_decoded_type5(),
)
assert result is not None
_, payload = result
assert "snr" in payload
assert "SNR" not in payload
+1 -1
View File
@@ -240,7 +240,7 @@ class TestSubscriber:
assert payload["text"] == "hello channel"
assert payload["txt_type"] == 5
assert "sender_timestamp" not in payload
assert payload["SNR"] == 12.5
assert payload["snr"] == 12.5
assert payload["path_len"] == 3
def test_letsmesh_packet_without_decrypted_text_is_not_shown_as_message(
+45 -1
View File
@@ -1,7 +1,7 @@
"""Tests for database models."""
import pytest
from sqlalchemy import create_engine
from sqlalchemy import create_engine, select
from sqlalchemy.orm import sessionmaker
from meshcore_hub.common.models import (
@@ -13,6 +13,8 @@ from meshcore_hub.common.models import (
TracePath,
Telemetry,
EventLog,
EventObserver,
add_event_observer,
)
@@ -216,3 +218,45 @@ class TestEventLogModel:
assert event.event_type == "BATTERY"
assert event.payload is not None
assert event.payload["battery_percentage"] == 75
class TestEventObserverModel:
"""Tests for EventObserver model and add_event_observer helper."""
def test_add_event_observer_stores_path_len(self, db_session) -> None:
"""add_event_observer accepts and stores path_len."""
node = Node(public_key="a" * 64, name="Observer")
db_session.add(node)
db_session.commit()
result = add_event_observer(
session=db_session,
event_type="message",
event_hash="abcdef12",
observer_node_id=node.id,
snr=10.5,
path_len=3,
)
assert result is True
observer = db_session.execute(select(EventObserver)).scalar_one()
assert observer.snr == 10.5
assert observer.path_len == 3
def test_add_event_observer_path_len_defaults_none(self, db_session) -> None:
"""add_event_observer defaults path_len to None when not provided."""
node = Node(public_key="b" * 64, name="Observer2")
db_session.add(node)
db_session.commit()
result = add_event_observer(
session=db_session,
event_type="trace",
event_hash="12345678",
observer_node_id=node.id,
)
assert result is True
observer = db_session.execute(select(EventObserver)).scalar_one()
assert observer.path_len is None
assert observer.snr is None